You can disable cache in developer toolbar (right click-> inspect element), "network" tab, on the right side of the opened pane. When you hover the element it says "Disable/Ignore cached ressource while loading resource".
Once it is red, velvet cache are deactivated and memory consumption is far lower after that.
You also have to empty the cache by activating developer mode, open menu "developer" -> "empty cache" and you will see a huge drop in RAM consumed by Safari.
There is also another major factor of performance drop : Virtual Memory (swap file). If a process start using it heavily (WindowServer), you will see a huge drop in computer performance because read/write access to disk is monopolized by these processes, leaving little room to other processes for disk access operations.
I suspect Catalina to be buggy in that regard as I saw a huge drop in performance after updating due to excessive Virtual Memory usage, reaching as high as 250Mo/s of writing access and 200MB/s of reading access for a SSD.
You can try to disable Virtual Memory, if (and only if) you have enough RAM (>=16GB) with this command :
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist
If you run into issues after that, like sudden self-awareness making your mac unresponsive or excessive RAM usage, enable Virtual Memory again with :
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist