Is there a way to not only mute a call, but stop it hijacking the entire screen? I don't want the person to know I'm not answering their calls, so don't want to decline it and have it get sent to voicemail early.
The best I've come up with is to hit the Volume Down button on the side of the phone during a situation like this. That causes the phone to stop ringing/vibrating, but doesn't decline the call right then and there (maybe this is what you referred to as "mute a call"?). The incoming call will last the normal amount of time before being sent over to voicemail. However, this leaves the incoming call screen active and blocking whatever you were previously looking at.
Press ‘Remind Me’! The phone keeps ringing on the other end, and you get to move on with your life.
Edit: No longer works as of iOS 12.2 The best solution now is to take a screenshot of the call dialog, hit share, then use the various options there (including shortcuts) to continue doing whatever you wanted to do. Email, and I think messaging, is still fully accessible, and I suspect there's a shortcut pattern that could give you web access, although I haven't found it yet.
A qualified 'yes' (for iOS)
Prior to iOS 14, to the best of my knowledge, no.
The call is either able to be answered (i.e. displayed on your screen,) or ignored (i.e. sent to voicemail,) thereby freeing-up your phone for other uses.
What purpose would be served for the carrier to allow the caller to continue to hear ringing if there was no way for you to answer?
Update for iOS version >=14
However, since the introduction of iOS 14, you can now choose to have incoming calls display as a banner at the top of the screen instead of the full-screen display.
In the Settings app, under Phone there is, in the top section, an item called Incoming Calls. If you select this, you may choose between Full Screen and Banner. See this article for more details.
While all of these nuances are academically interesting, I personally recommend using a call blocker/scanner tool. These are widely available and can intercept your incoming calls and screen them for spam calls fairly effectively. The one I use can even play custom recordings. I chose the 'disconnected tone' message in case some call bots might identify that sound and log the number as being out-of-service.
Pressing the power button once mutes a call but does not decline; it will stop ringing on your phone but will keep ringing on the caller's phone and not get sent to voicemail immediately. Two presses of the power button declines the call and sends it to voicemail.
Source is my personal testing and this Apple Support forum question.