I freshly installed Big Sur with Windows 10 Boot Camp on a late-2013 MacBook Pro Retina with a replaced SSD. When I hold down the option button at boot, the options are listed as "EFI Boot" (Windows 10) and "Macintosh HD", but both are using external drive icons instead of internal. Under MacOS, the icons for both drives appear correctly as internal drives. I tried changing drive names in diskutil, renaming the drives in the MacOS Disk Utility and under Windows, placing .VolumeIcon.icns files in various places, and various bless commands.
I'm asking this to answer my own question, I couldn't quite find a correct solution in any one spot. Pay close attention to the folders involved, this was the important part.