For a free solution to add a compressor/limiter to the system-wide audio, you can use Soundflower along with Apple's AU Lab:
https://github.com/mattingalls/Soundflower/releases
https://www.apple.com/itunes/mastered-for-itunes/
Following these steps:
- Install Soundflower and AU Lab.
- Open System Preferences > Sound, and set both the Sound Effects and the Output to play through Soundflower (2ch).
- Open AU Lab, and in the Document Configuration window, under Factory Configurations, select Stereo In/Stereo Out.
- Set the Input to Soundflower (2ch), and the output to Built-in Output.
- Click Create Document.
- In the Output 1 channel strip under Effects, select Apple > AUDynamicsProcessor.
- Adjust the settings in the Dynamics Processor window as desired.
A similar setup using the AUDynamicsProcessor is possible with the commercial Audio Hijack:
https://rogueamoeba.com/audiohijack/
The Hear app also has a limiter, and seems to use less CPU resources than the above. However, it's no longer in developoment and has limited support:
https://www.prosofteng.com/download/?hearmac
While these are answers to the question of how to add a compressor/limiter to the overall system audio, there are some drawbacks. One is that the above solutions may use significant CPU resources. Another is that you'll most likely need to remember to turn it on and off when watching a movie. Otherwise, having the compressor on all the time may reduce the overall dynamic range and sound quality of your computer, for example when playing music.
Rather than compressing or limiting the overall system audio, another approach is to compress then boost the quiet parts of movies so that you don't have to turn up the volume level on your speakers. I use mpv ( https://mpv.io ) to play video, which has several options for precise dynamics control including the "lavfi compander" or the "acompressor.lua" script, with very little impact on CPU usage. VLC also has a built-in compressor found in Preferences (Show all) Audio > Filters > Compressor.
Audio Hijack can be used to route the audio of an app that has no native compression options through dynamics processing. There are also other options with Audio Hijack, such as routing your movie's audio to external speakers, while routing everything else to internal speakers with a separate volume control.