There are some types of apps that are not allowed in the App Store but are still useful or actually needed to get a job done. The accepted answer lists some of the reasons. Sometimes, we app developers need access to certain APIs to provide the intended functionality, but Apple does not allow apps in the App Store which access those. As a specific example, my company provides a VPN client and we need to access low-level APIs to be able to make VPN connections. But access to these APIs is forbidden for apps in the App Store.
Apple has recognized that there are apps which simply cannot be distributed via the App Store, for various reasons, and has provided an alternative to ensure at least some safety for users: Developers can (and now sometimes must) notarize apps. For example, we must notarize when distributing Kernel Extensions or System Extensions. As a user, you actually need to actively disable System Integrity Protection to allow un-notarized apps with Kernel or System Extensions on macOS 10.15 Catalina and 11.0 Big Sur.
What Notarization means is that we developers build an app, then pass it to Apple which scans it and if Apple thinks the app is free of malware, a "ticket" is saved with Apple that macOS can query. (Developers can then "staple" the ticket to the app so macOS doesn't need to query it via the Internet. It's cryptographically signed by Apple so you can't fake it.) The app is not reviewed like for the App Store, it's just an automated malware check and that's basically it.
When you start such a notarized app, you get a dialog that warns you that the app was provided by a third-party developer, but also informs you that Apple has scanned it and deemed it safe for execution:
When you see this dialog (wording may change, but will contain a phrase that Apple has checked it), it's reasonably safe for you as a user to start that third-party notarized app. The notarization ensures the app has not been tampered with (or else the cryptographic sign would break and macOS would refuse to start it). Apple vouches that it checked the app for "malicious content".
This does not guarantee that the app does not contain malicious content (developers can hide stuff from Apple if they really want to). Only that Apple wasn't able to find any. However, the notarization process allows Apple to retro-actively revoke a ticket if the app turned out to be malicious, thus preventing the app from running on your Mac. Apple was somewhat able to do so before with signed apps, but only by revoking the certificate of the developer, thus blocking all apps of said developer. Notarization allows Apple to block on a per-app-and-version level. So Apple is able to specifically block version 1.2.3 of
Foo.app but allow version 1.2.4 to run.