I’ve been living with this annoying issue for a year or so now and I am delighted to say it appears the most recent iOS update, version 13.5.1, has fixed this issue! 

I no longer see AppleTV as an option when toggling speakerphone. It simply puts the caller on speakerphone.