The reason testdisk fails is because it needs to write to the device, but your dmg is probably stored in read-only format, so it is not able to write to the virtual device that represents the dmg.
Because it's not smart enough to check if the device is a read-only media (like read-only dmg), it assumes that it cannot write to the disk because a partition is mounted. When a partition is mounted, it's locked/write protected so that you don't lose data.
To solve this, you can either "restore" the dmg to an actual physical drive then run testdisk on it, or you can mount this dmg as read-write using a shadow file. See man hdiutil for more details.
-shadow [shadowfile]
Use a shadow file in conjunction with the data in the pri-
mary image file. This option prevents modification of the
original image and allows read-only images to be attached
read/write. When blocks are being read from the image,
blocks present in the shadow file override blocks in the
base image. All data written to an attached device will
be redirected to the shadow file. If not specified,
shadowfile defaults to image.shadow. If the shadow file
does not exist, it is created. hdiutil verbs taking
images as input accept -shadow, -cacert, and
-insecurehttp.
So your command would probably look something like: hdiutil attach -nomount -noverify /path/to/dmg -shadow (/path/to/shadow/file (optional))