So I symlinked my Downloads folder from my boot drive to an external HDD I always keep connected to my mac. I deleted the original downloads folder from my home folder and replaced it with a symlink to a new folder on my HDD.
This works perfectly, except that in Finder, the sidebar icon is now a folder:
When it should look like this:
Operating system is High Sierra 10.13.6