Yes, you need the updates one way or another.
Since you don’t have Xcode installed, you could simplify things by uninstalling the command line tools and then install one clean package.
rm /Library/Developer/CommandLineTools
#optionally restart and check for updates before reinstalling
xcode-select --install
Running all the updates is the other path forward that’s easy. This is expected when each update is a delta even when Xcode does the install (which adds complexity beyond what you require) and not “the whole install whether things change each version or not” type installer.
If the goal is minimizing downloads you can set up a local caching service in the future so you only download each update once for the entire network.
Also, this is often worse on betas where Apple will test updates so you might be part of testing things or the beta packages don’t get seeded globally so you have to do more downloads than once things slow down after general release.