Font Book has a few international fonts that need to be downloaded before use. After downloading one there is no immediate way to remove it again. Can this be worked around?
You can replace them with themselves, but manually installed, with the shell command:
cp /System/Library/Assets/com_apple_MobileAsset_Font5/*/AssetData/* ~/Library/Fonts/
Then, from the “User” section of the sidebar, select all of the fonts you want to remove, and the system-installed version should be removed as well. An alternative way is to resolve the duplicates by keeping the manual version, and then removing that as well.