I have a mid-2011 iMac with 16gb of RAM and an SSD drive installed. Currently I'm using macOS High Sierra (version 10.13.6) which is the last supported version on my Mac. Is there any way to force the installation of a newer version of macOS, Catalina for example? If the answer is yes, how it can damage my mac?
For Catalina, there is dosdude1's macOS Catalina Patcher. While dosdude1's website states compatibility with 2011 iMac models (iMac12,x), the website also states the following.
iMac12,x (systems with AMD Radeon HD 5xxx and 6xxx series GPUs will be almost unstable when running Catalina)
All 2011 iMacs were shipped from Apple with Radeon HD 6750M, 6770M, 6970M or 6750M GPUs. I assume this means the video would need to be upgraded before installing Catalina. While there are websites which claim this is possible, but I do not think this should be attempted by the average Mac user.
For Big Sur, there is a big-sur-micropatcher. Here the instructions basically say a 2011 iMac would need a GPU upgrade before a Big Sur install would be practical endeavor.
The only instance that I have heard of where software can harm hardware on a Mac is when upgrading the firmware. Evidently, the only way to downgrade from a firmware upgrade on many (if not all) Macs is to replace the logic board. So if you encounter any instructions which state non-Apple firmware is to be installed, then I would have cause to worry.
There are hacks to force-install newer OS versions onto unsupported Macs.
Software can almost never 'damage' hardware -- not unless you're instructing something mechanical to start/stop over and over, or similar.
However, 'worst case' is unpredictable kernel panics, and slow speeds. Your Mac doesn't support the Metal graphics framework, so graphics acceleration will be disabled, making scrolling and other interactions slow.