Q How do I enable transparent compression?
You don't. You can't. But keep reading.
This is not "enabled" – for the whole filesystem – and then constantly applied, for example for newly created files.
Using filesystem compression features
What you can do is compress transparently individual files and folders.
(One option for this would be using ditto or like in jksoegaard's answer, afsctool, alternative fork)
This distinction is important as it sets limits on what you can compress, effectively.
What to compress, and what not
You can compress your entire home-folder with this.
But that makes not much sense.
In part because many file types are already compressed. JPG, PNG, PDF, zip etc are all formats that usually do have some compression. Compressing them again doesn't save much space but adds a redundant de-compression penalty on accessing them.
In part because how Apple implements its filesystems. Files do not get created and then added to if you amend the content. They get newly created and the old version destroyed. Newly created files don't get and constantly changing files lose all benefit of transparent compression the next time they are written to/committed to the filesystem. This includes all temp-files, cache files, log-files, many preferences etc.
Thus, you should only apply this to files that do not or rarely change.
Those that are read but not written. Application files are the prime candidate for this. And often a great candidate for space saving, as some developers have really atrocious packaging and deployment habits of waste and bloat included.
Tools to use
On older systems: Apart from afsctool and ditto, GUI tools to achieve this would be MoreSpace Folder Compression (seems to b ediscontinued) or Clusters (v1.7.2, 13$, last updated 2014, also with problems on newer systems).
That means only the commandline options are currently working:
ditto --hfsCompression will compress files, despite the option's name, on APFS.
afsctool will do that as well, but only an updated version/fork https://github.com/RJVB/afsctool does so without much problems.
(As referenced in @jksoegaard's answer, only that the original works on 10.14, but often not with files on APFS).
Note that the version homebrew offers is only the outdated version 1.6.4. That version has problems with APFS and is not recommended.
To get the newer/forked version that works well with APFS you need to install from source or resort to Macports for easier installation.
On Macports however the binary installed for this fork is called
port install afsctool you'll get the outdated 1.6.4 version!
brew install afsctool.
In other words: Macports has a 'working fine on APFS' version called afscompress, and the outdated afsctool, homebrew has no good version, as only the outdated afsctool is available.)
Precautions for unexpected consequences
Further, you have to be a bit careful in using this.
Unexpected results are very rare, but exist.
For example, a binary of
grep seizes to function and compressing
git results in your git-repos becoming unreadable unless you revert that compression (the repositories are still fine).
Some GUI applications do perform some homebrewed tamper detection that might fail if they are compressed in this way. You might never run into one of those, but it's still best to have a backup ready and proceed in smaller steps, not by compressing all at once.
All subquestions answered:
Q How do I enable transparent filesystem compression on any APFS volume?
Tool of choice is now afscompress commandline tool.
To compress your Documents folder:
afscompress -c ~/Documents
Q Or check if it is already enabled?
afscompress FILE would do this.
Output to expect:
File is HFS+/APFS compressed.
Q I am expecting something similiar to NTFS compressed attribute. According to Wiki it should be supported.
It is. Since OS 10.6. But Apple has no GUI tools for a user to use it. If a compressed file is re-saved again, exported etc, the new file will be uncompressed again. So the compression used has to be re-applied.
Slightly outdated info here:
Unix commands for managing HFS+ compression in OS X
Q Is it per file or volume?
Q I am using Mojave 10.14.0, volumes without encryption.
As long as you don't use the outdated afsctool, 10.14 is fine. APFS is fine. Encryption doesn't matter.