24

When you copy a file on APFS, it doesn't actually duplicate the file blocks, instead it just clones the metadata. This is known as copy on write (COW). That prevents identical files from using any more space until you make an edit to the duplicate file, and then , only the changed blocks take up space so substantially identical files share most of their storage in common even after a change.

I'm hoping there is some tool to find existing duplicates and replace them with clones, reclaiming the space used to store the duplicate.

My research shows questions on Apple Communities and the MacRumors Forums don't propose any solutions.

Is there a way to use the copy-on-write cloning feature of APFS to reclaim space from existing duplicate files?

Improve this question
1
  • 3
    Hard to believe that nobody wrote an app to do this yet. On my home folder alone, there's 4.41 GB to be saved from doing this. That's far from a negligible amount. Take note: app developers, I'd gladly pay a few dollars for this!
    – swineone
    Commented Feb 14, 2019 at 12:28

5 Answers 5

Reset to default
12

As of June 2020, Jdupes can do this.

For example, this will find all duplicates in your Applications folder:

jdupes --recurse --dedupe /Applications/

WARNING: Versions before 1.19.0 will overwrite metadata (date, owner, permission), more recent versions preserve all metadata when using jdupes --dedupe on APFS filesystems.

Improve this answer
4
  • 1
    metadata is declared fixed in 1.19.0
    – leavez
    Commented Dec 6, 2020 at 16:48
  • This is incredible and almost certainly will work on iOS jailbroken APFS devices too (iOS 10.3+). I'm going to compile jdupes on my iPad and see if I can dedupe!
    – Syclone0044
    Commented Mar 17, 2021 at 10:56
  • Update: jdupes 1.19 and 1.20 compile perfectly on my iOS 12.4 and 13.3 jailbroken systems using clang 10 and iOS SDK 12.4 and 13.4. The APFS clonefile(2) CoW dedupe is incredible!!! Test files I experimented with have worked great. I stand to gain at least 10GB on my (full) 256GB iPad Pro from deduping, but I'm still testing to gain more confidence before I turn jdupes loose on my entire disk.
    – Syclone0044
    Commented Aug 23, 2021 at 8:00
  • 1
    jdupes 1.2.0 can corrupt files having APFS compression - github.com/jbruchon/jdupes/issues/189
    – adib
    Commented Dec 1, 2021 at 4:34
5

I had the same problem and wrote a small shell script to do this using native cp, mv, jdupes (for speed), and gcp (for metadata preservation). Hope this helps!

Advantage over other solutions:

Improve this answer
2
  • 1
    How do you use this script? Is there documentation somewhere?
    – whoKnows
    Commented Jul 4, 2020 at 20:27
  • 2
    jdupes now supports APFS clonefile() natively as of v1.17.1 - use jdupes -B just like with BTRFS/XFS on Linux. github.com/jbruchon/jdupes/releases
    – Jody Bruchon
    Commented Jul 17, 2020 at 15:06
5

Check out diskDedupe http://www.diskdedupe.com It does exactly what you are looking for.

It just has two buttons (scan and deduplicate) and maintains all metadata, when replacing a duplicate with a clone of the the original.

I used it on all my archives.

Improve this answer
3
  • 3
    Welcome to Ask Different. What's your experience using this software? You appear to have just joined this site today so we'd like to know why you're recommending this specific software?
    – fsb
    Commented Feb 18, 2020 at 14:23
  • 3
    Hi fsb, I have waited for APFS since Apple decided to write their own file system APFS, instead of using ZFS. Then I was disappointed that APFS does not provide any deduplication functionality like ZFS does. Therefore I was searching for a simple app that provides at least some offline deduplication. I found diskDedupe via Google search for APFS and deduplication and tried it out. I also found the this page and wanted to share my experience. I use diskDedupe for more than 4 months now, it is not expensive and it saved me a lot of disk space.
    – peter1999
    Commented Feb 18, 2020 at 16:31
  • Thanks both @fsb for being skeptical and scrutinizing claims and also peter1999 for then clearly stating your motives & use-cases which seem plausible and believable.
    – porg
    Commented Oct 20, 2021 at 8:57
4

There are some alpha-quality open source scripts that try to do this - https://github.com/ranvel/clonefile-dedup

Improve this answer
2
  • Yeah, I found this one too. Unfortunately there are a couple of reasons I don't find it usable: ultimately, it is just using cp -c, which discards all the metadata of the origin file (owner, permission, extended attributes, etc.). And there's no reason it couldn't just be done with a shell script, calling sqlite3 directly—instead, you have to install Python 3, which means its off-limits to non-admins who can't change settings to run unsigned/non-App-Store signed code.
    – Geoff Nixon
    Commented Dec 19, 2018 at 7:30
  • 2
    I submitted some changes (and they were approved) to github.com/ranvel/clonefile-dedup Now it will preserve file date and time, as well as chown/chmod attributes.
    – George Yohng
    Commented Aug 3, 2019 at 8:38
2

See also https://github.com/deckarep/apfs-compactor, which is also just a prototype.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .