This functionality is referred to by Apple as “Transparency, Consent, and Control” (TCC), Access Control, and Privacy Preferences Policy Control (PPPC). It is designed to give a user control over Apps to protect their privacy. In some cases an App will request access to something that App really doesn't need and the user can block the App from accessing that data or filesystem path. It is annoying, but it's typically a one-time event per App. So unless you are frequently clean installing macOS it wouldn't be super annoying.
Apple provides a way to build a Configuration Profile payload to whitelist applications so the user approval prompts do not appear. An Mobile Device Management (MDM) server would be the best way to deploy the payload.
You might be able to build a custom XML Plist Configuration Profile and manually load it on macOS Catalina without an MDM and it might work to whitelist the Apps you specify. But it's a lot of work and as of macOS 11 (10.16) Big Sur will break. Big Sur simply won't trust a Configuration Profile unless it comes from a trusted MDM.
If you want to try whitelisting the Apps and manually installing a custom profile you can review that sample here:
https://support.apple.com/guide/mdm/privacy-preferences-policy-control-custom-mdm9ddb7e0b5/1/web/1 You can use Apple Configurator to create the Configuration profile with this payload and double-clicking the .mobileconfig should install the profile. There's a command line profiles command as well.
Those who use MDM typically deploy a bunch of Apps and configurations and they whitelist kernel extensions and PPPC/TCC entries via Configuration Profiles. They can lockdown a great many things on macOS/iPadOS/iOS. Admins would whitelist Apps so the users are not flooded with a bunch of user approval prompts but also seeing fewer of them will help a user be surprised when they see one and hopefully make an appropriate choice or at least call the Help Desk. You don't want users clicking through frequent prompts without thinking about the question being asked. Most Macs managed by an MDM wouldn't even grant administrator rights to the users and they would provide a company specific App Store where pre-packaged and prepared Apps are provided. Those Apps would all be whitelisted on PPPC/TCC approvals. The Mac App Store may be blocked to the user. Those Apps can be deployed by the MDM via VPP (Volume Purchase Pricing) integration with the companies procurement department.
So unless you setup your own MDM server and manually create a Configuration Profile to whitelist all the Apps and update that list over time. Then deploy it. There isn't really a good answer here. It's still a lot of work to manually specify every app in an XML file and only really useful if you are doing it across many Macs.
There is a Python tccutil.py utility on Github that can whitelist individual apps to the tcc.db but access to the tcc.db is blocked by SIP (System Integrity Protection) since Sierra. TCC was updated since Mojave to add the user approval to Desktop, Documents, Downloads, etc. This tool won't work unless you disable SIP. Disabling SIP is NOT RECOMMENDED. Since Catalina, the System volume is Read Only. So you would have to not only disable SIP but also get around the Read Only System APFS volume which is possible. Again, NOT RECOMMENDED. It is a lot of work that is frankly, not worth the effort to get around a one time prompt per App.
There is a published exploit where a malicious App could impersonate a trusted App identifier and signatures to bypass the PPPC/TCC protections.
The X11 based apps such as Fontforge, Gimp, Inkscape, etc. do not run as true macOS applications. They run a wrapper around a command line binary then load the X11 resources into the wrapper. Therefore you must grant permissions to the Terminal App where these applications actually run.
Try going to System Preferences -> Security & Privacy -> Privacy -> Full Disk Access -> Unlock the panel and click + and add the Terminal App. This is fairly dangerous and has security implications but according to the Github issues regarding Gimp, Fontforge, etc. This may resolve the problem as a work around. It is unlikely these applications will fully resolve this issue. They have made some recent changes to help alleviate the problem on macOS.