What you're seeing is the mounted disk image, you need to eject them.
These are no different than a USB disk or flash drive that you attach or a CD/DVD media that you load into a drive. It will show up on your Desktop so you have an easy way to access them. Once you're done installing them, you can simply right click and then select Eject.
For example, I double clicked a (random) DMG file (Western Digital macOS Apps) that was in a different directory (from my Desktop) and the mounted file shows up on the Desktop. If you double click, you will see the contents (usually install scripts or the App to be copied to the /Applications folder). When you're done, right click and select Eject. Note, many newer apps (VirtualBox I know does this) will, as part of it's script eject and move the DMG to Trash for you.
If using Terminal, you can see them by listing the contents of
% ls /Volumes
You cannot delete the item; if you drag it to the Trash, the trash icon will turn into the Eject Symbol.