The command you mention in your comments
hdiutil create -o /tmp/Catalina -size 8500m -volname Catalina -layout SPUD -fs HFS+J appears to come from this tutorial which in full does this:
hdiutil create -o /tmp/Catalina -size 8500m -volname Catalina -layout SPUD -fs HFS+J
hdiutil attach /tmp/Catalina.dmg -noverify -mountpoint /Volumes/Catalina
sudo /Applications/Install\ macOS\ Catalina.app/Contents/Resources/createinstallmedia --volume /Volumes/Catalina --nointeraction
hdiutil detach /volumes/Install\ macOS\ Catalina
hdiutil convert /tmp/Catalina.sparseimage -format UDTO -o ~/Desktop/Catalina.cdr
mv ~/Desktop/Catalina.cdr ~/Desktop/Catalina.iso
These steps work absolutely fine to create an ISO but you have 2 issues:
You must tell VirtualBox what to boot
The ISO produced will work fine with VirtualBox but will not run automatically as Apple have put
boot.efi in a non-standard place. Unlike a real mac the firmware doesn't know where to look.
In the EFI shell, if you enter
FS0:\System\Library\Coreservices\boot.efi (it may be
FS2: &c depending on the order of your virtual disks) it will run. This can be automated by putting the command in a startup.nsh script which in fact is contained in the VirtualBox Boot.vdmk mentioned in the tutorial.
If you would rather create your own than download from a random google drive you can do it like this:
hdiutil create -o /tmp/CatalinaBoot -size 50m -volname MACOS_BOOT -fs FAT32
hdiutil attach /tmp/CatalinaBoot.dmg -noverify -mountpoint /Volumes/CatalinaBoot
echo 'FS0:\System\Library\Coreservices\\boot.efi' > /Volumes/CatalinaBoot/startup.nsh
echo 'FS1:\System\Library\Coreservices\\boot.efi' >> /Volumes/CatalinaBoot/startup.nsh
echo 'FS2:\System\Library\Coreservices\\boot.efi' >> /Volumes/CatalinaBoot/startup.nsh
echo 'FS3:\System\Library\Coreservices\\boot.efi' >> /Volumes/CatalinaBoot/startup.nsh
echo 'FS4:\System\Library\Coreservices\\boot.efi' >> /Volumes/CatalinaBoot/startup.nsh
hdiutil detach /Volumes/CatalinaBoot
VBoxManage convertfromraw --format VDI /tmp/CatalinaBoot.dmg ~/Desktop/CatalinaBoot.vdi
Attach the created CatalinaBoot.vdi disk first in your VM and it will automatically run the boot.efi from the ISO created as they describe.
Note when I tested it on VirtualBox 6.1.2 the ISO generated by the tutorial would fail when using a SATA controller giving the error: "BdsDxe: loading Boot0001 “UEFI VBOX CD-ROM VB1-1a2b3c4d ”". When connected to an IDE controller it worked fine.
VirtualBox 6.1.2 will not run Catalina 10.15.2 and later guests
Virtualbox 6.1.0 added support for booting APFS. Unfortunately there is a problem if you do not have 10.15.1 version of Install macOS Catalina.app.
There is a bug in VirtualBox which is exposed by 10.15.2 and later versions of
boot.efi. See this VirtualBox forum thread Catalina 10.15.2 does not start. As mentioned in the relevant VirtualBox Ticket #19188
The source of the boot hangs has been identified, although fixing it correctly isn't entirely trivial. It's triggered by a very, very old problem in VirtualBox.
At any rate it will be fixed for the next maintenance release.
If you want to run a macOS 10.15.2 or later guest (on any VirtualBox 6.0.2 host) you must use
boot.efi from Catalina 10.15.1 or Mojave. You could do this yourself following the steps in the VirtualBox forum thread or use this script on GitHub macos-guest-virtualbox
As 10.15.1 is no longer available on Apple servers you may need to download and unpack Mojave to get
boot.efi and then copy this first to unpacked Catalina installer and then afterwards to Catalina itself. As they say in their linked workaround
The real easy way to run Catalina is to use the script to install Mojave on VirtualBox and then run it in VMWare and upgrade it to Catalina.