I don't know that there's anything you can do about this—speakers just use up a lot of Bluetooth's available bandwidth. While I've never had trouble with JUST a speaker and mouse, I've found that if I connect a Bluetooth mouse, keyboard, and speaker at once, the speaker becomes horribly stuttery until either the mouse or keyboard is disconnected.
You say you've already tried different speakers; I might try different mice as well. You could try a USB Bluetooth adapter, but there's no guarantee that it will be any better. Also, if you have any additional Bluetooth devices you didn't tell us about, definitely disconnect those as well.
(At one point, I set up Eventscripts to automatically disconnect my keyboard whenever a Bluetooth speaker was connected. It worked, but was somewhat fragile, and I don't recommend it.)
You could also try lowering macOS's Bluetooth audio quality in the terminal so it uses less bandwidth. I have never been able to get acceptable results this way, but it could be worth a try. https://lifehacker.com/fix-your-bluetooth-audio-in-yosemite-with-this-terminal-1670380974
You're not going to like this solution, but I would recommend replacing either the mouse or speaker with a non-Bluetooth alternative, such as a wireless mouse with a USB dongle or a wifi speaker that works over airplay.
Sorry this isn't the answer you wanted.