I often can hear that an external HDD has spun down, and figure it would be safe to unplug since it won't be writing any data.
Am I wrong in this assumption?
Your best bet is to always eject (or unmount) the drive.
Basically, failure to unmount before disconnecting the device can result in loss of data and/or a corrupted file system.
If the external hard drive has in fact spun down, then unplugging it (accidentally or otherwise) is less risky than it would be if it hadn't spun down, but this won't remove the risk entirely. For example, in your question you refer to the fact that often you can hear that an external HDD has spun down, but this is prone to errors in judgment.
Also, not all external drives are the same. Some are USB powered, some have their own dedicated power source, some don't even use USB. What I'm getting at here, is that the only rule that can safely be applied to all external drives is to eject/unmount them first prior to unplugging them!