I have not tested how to prevent it yet, but I have tested the so-called workaround being passed around that recommends moving the invite to a temporary calendar, then deleting that calendar, and answering "Delete and Don't Notify" -- it STILL sends a DECLINE notice.
Folks passing it around and up voting it here and elsewhere did not actually test it out, they just assumed the "Delete and Don't Notify" button did what it promised to do.
Try it with another iCloud user. Your status on their invite will go from NO REPLY to DECLINED shortly after you click "Delete and Don't Reply."
This is repeatable on macOS Sierra 10.12.1 and Calendar 9.0 (2155.8) both are latest versions as of this writing.