Is it possible to replace the GPU in this 2006 iMac? If yes what kind of models fit in there under the cooler?
It is not possible to upgrade the GPU on the iMac you described because it is integrated into the Main Logic board (pictured in your post).
In theory it might be possible to buy the GPU from another iMac Logic Board that is higher specification, however you'll likely have to buy it with the Logic Board from that computer and so would just be swapping the logic boards.
Short answer: No
EDIT: Confirmation that it is upgradable but the only options are those that were available when the product was released: https://discussions.apple.com/thread/2268642