I have a new Magic Keyboard 2 and I was wondering if keeping it connected all the time to the USB port can somehow damage the battery
In general, keeping it connected all the time will not damage the battery. Apple has designed these peripherals to only accept a charge when it's needed. When it doesn't need the power, it simply will not use it.
However, it's a good practice to periodically, every month or so, unplug the keyboard and let it run on its battery until it gets very low (below 10%) or stops working. Then you can plug it back in and leave it alone while it charges. Apple has a nice section on batteries and their maintenance here.