Update June 15, 2020: I’ve been living with this annoying issue for a year or so now and I am delighted to say it appears the most recent iOS update, version 13.5.1, has fixed this issue!
I no longer see AppleTV as an option when toggling speakerphone. It simply puts the caller on speakerphone. Hallelujah. Has anyone else that has updated to 13.5.1 still experiencing this issue?