Until today I was able to silent a call (both from ringing and on silent/vibrate modes) on my iPhone 7 plus with the lock button on the right. When I press it now, it just declines the call on both modes.
Under iOS 10 and earlier, the side/power button was responsible for silencing a call, just as you'd experienced. But now, under iOS 11, the volume keys — maybe just the Volume Down button? — are responsible for silencing a call. The side/power button remains the way to decline a call.
If you've recently installed iOS 11 on your iPhone, that may explain the change in functionality you've observed.
So, the next time you get a call, try pressing the Volume Down button and see if that achieves the effect you want.