Why does installing a package create a directory on my desktop?
That's not a directory on your Desktop, but the icon for mounted disk image.
As the other answers have aptly mentioned, a lot of apps for macOS are distributed packaged in a disk image (typically a DMG file). Executing (by double-clicking or opening) a DMG file mounts the disk image, and it appears to users just how a mounted disk (external/internal hard drive, thumb-drive etc.) would appear in macOS.
A mounted disk (which could be Mac's built-in hard drive, a connected external USB drive, network storage, or a DMG file) can appear in two places:
Topmost level in Finder, under the Computer.
The visibility in the former case can be toggled to show/hide the mounted disk(s). You can do so by opening Finder Preferences (either from the Finder Menu → Preferences... or using the keyboard shortcut Command + ,).
Now in the Finder Preferences window, select the General tab and check/uncheck the External disks item to show/hide the mounted DMG(s) from the Desktop.
Irrespective of whether the External disk visibility is enabled on Desktop, they would always show under the topmost level in Finder under your computer. You can access the topmost level in Finder by using either one of the following three techniques:
Open a Finder window/tab, and:
in the Sidebar, under the Locations item, click on your computer name (16" MacBook Pro in the screenshot below).
use the keyboard shortcut, Command + Shift + C to directly access the Computer.
keep invoking the keyboard shortcut Command + ↑ until you reach the topmost level and can see the mounted disk. (Once you reach the top-level) the keyboard shortcut won't work anymore.)
After you have installed the app by copying the .app bundle from the disk, you can and should unmount it. You can unmount a DMG by using either one of the following techniques:
Drag and dropping the mounted disk icon from any of the two locations discussed above to the Trash in the Dock.
Clicking on the eject icon shown alongside the disk name in the Finder Sidebar (as shown in the screenshot above).
Clciking to select the mounted disk image icon and using the keyboard shortcut Command + E to eject.
To answer your specific questions:
When I install a package, it creates a directory with the name of the package on my desktop. In Finder it looks like a new external drive:
Why is that?
The installer you downloaded comes in the form of a DMG file which gets mounted as an External drive. It gets shown on your Desktop, as Finder is by default configured to shown mounted External drives on your Desktop.
Should I always remove those drives after installation?
Yes. Once you are done copying the .app bundle out of the disk image, you should ideally unmount the disk image and either backup the original DMG file for later re-installation of the app, or trash it.
Or did I do something wrong and this should not show up?
You have not done anything wrong and this is by design. The answer above should give you clear explanation about the observed behavior and possible customizations.