As noted by Allan, just removing the HDD effectively means removing the integrated temperature sensor. In case one wants to replace a defective HDD, his solution is definitely the way to go.
However, I didn’t want to replace the HDD but simply remove it (the HDD was never used, the internal SSD has enough capacity as a system/working drive for me, and I prefer a computer with as little spinning parts as possible).
Some more in-depth research brought me to this article (on archive.org). Towards the end, it shows what Apple did on iMacs which were delivered with just an SSD and no HDD:
The article says:
It’s just a plug that goes into the HDD power connector that shorts pins 2 and 7. Apparently tying the temperature-sense pin in the connector to ground signals to the BMC to just leave the fan at its base speed. Careful modification of the existing hard drive power cable should allow one to perform this hack while installing any drive of his or her choice — albeit while running the risk of having the drive overheat.
One can even find some online shops which list the mentioned jumper plug (however, either out of stock or for ridiculous prices). The Apple part no. is 922-9877.
What I did in the end:
Open the iMac and remove the HDD following iFixit’s instructions
Plug a 15-Pin-SATA-to-4-Pin power connector on the free SATA power cable. The other end is not connected, its only purpose is to connect pin 11 to ground which is effectively the same what Apple’s jumper plug would do when directly connected on the logic board.
Keep the spare screws in a plastic bag which I hooked in the HDD mount -- just in case.
HDD fan now spins at 2,500 rpm in idle mode.
Warning: Again, this only makes sense when removing the HDD. Do not trick the temperature control when you add an HDD or SSD into the slot!