autoOpen feature is a simple flag that needs to be set within the header of the file.
The file format details is there:
It is explained that the offset 0x58 of the "header" section should be set to an Int32 representing the ID of the directory that needs to be opened (0 for none, 1 for root's parent, 2 for root, 0xXXXXXXXX for the directory with ID 0xXXXXXXXX).
Also don't forget that the header itself is starting at offset 1024 (0x400) so the total offset should be 0x0458.
It tested the following command successfully:
printf '\x00\x00\x00\x02' | dd conv=notrunc of=$dmg_file bs=1 seek=$((0x0458))
where $dmg_file is your dmg file.