Resolving this question involves having guest additions on a MacOS guest. If, instead, you want to install guest additions from a MacOS host onto a non-MacOS guest:
You can copy the guest additions iso from your machine.
First locate VirtualBox in the Application folder, right-click it and select show package contents. Then browse to Contents/MacOS/VBoxGuestAdditions.iso. Copy it to a location that accessible when browsing for files.
After that, you can insert the device by clicking Devices > Optical Drives > Choose disk image... and then choose your copied iso file.
Then you can run your iso from within your vm and install the guest additions.