2

When I move a large file between folders, I notice that Time Machine takes a long time on the next hourly backup. Apparently, it is quite stupid and considers a file move just like a deletion and an addition of a completely new file...

Since my backups are getting quite large, I'd like to shrink them by removing duplicate files across the backup sets. Would it be safe to simply run fdupes or a similar program on the backup sets? If not, would it be reasonable to instead use some custom script that replaces each found dupe with a hardlink to the duped file?

Improve this question
4
  • Do you have a compelling reason to mess with the storage algorithm chosen to make backups reliable? Other than knowing if you could do it, I don't see a practical use case where this sort of micro management of storage makes any sense. How much space do you calculate is wasted? Are you really losing months or longer time on the backup since intermediate snapshots are being pruned due to out of space considerations? The question is clear, but I wonder if it's of use to anyone else at this point.
    – bmike ♦
    Commented Apr 21, 2013 at 1:48
  • 1
    @bmike e.g. I've moved around 100GB of videos to a different folder. Now those same 100GB are stored as 200GB on the backup drive. Unless I dedupe these, it's only a matter of time before such redundancy comes at the expense of having older snapshots auto-deleted by TM
    – GJ.
    Commented Apr 21, 2013 at 11:56
  • Why not just use the time machine feature to delete all copies of the old folder at that point. Problem solved?
    – bmike ♦
    Commented Apr 21, 2013 at 13:01
  • 1
    One reason would be that this will cause me to delete all of the older snapshots on that folder.. if I didn't care about old snapshots I also wouldn't mind about the drive filling up...
    – GJ.
    Commented Apr 21, 2013 at 16:59

1 Answer 1

Reset to default
6

No, you cannot run most deduplicator applications on a time-machine backup.

Time machine makes extensive use of hard-links internally. Any application that is not aware of this will find hundreds or thousands of false duplicates (which are really hard-links to the same file), and proceed to hose your time machine backup. Apple also seems to do some filesystem-specific stuff in the backup too, which is why it needs to be on a HFS+ volume (either a entire drive, or a .sparseimage on a network share).

A lot of the complexity within the time machine mechanism is poorly or undocumented entirely, and if you break something, time machine will simply not work, without any meaningful documentation. If your deduplicator is hard-link aware, it may work, but it has equal chances of hosing something, from which you cannot recover.

Really, there does not seem to be any decent tools for managing time-machine images, unfortunately.

Improve this answer
3
  • TM doesn't need to be on HFS+ (it can even be on ZFS github.com/jollyjinx/ZFS-TimeMachine). The dedpulicator I was contemplating would replace real dupes with hard links, and my question about what specifically may get "hosed" and why.
    – GJ.
    Commented Apr 21, 2013 at 11:59
  • @GJ. - Multiple filesystems do support snapshotting. The link your provide is a completely different filesystem, with a completely different piece of software named "Time Machine". It behaves similarly to Apple's time-machine, but it's not the same thing. That implementation is simply a script that copies the data from ZFS's internal snapshot mechanism to a remote host.
    – Fake Name
    Commented Apr 22, 2013 at 3:11
  • As for how your software would work if applied to apple's time-machine backup - The answer is no one knows. There is very little to no information on modifying or the effects of modifying the internal time-machine backup image.
    – Fake Name
    Commented Apr 22, 2013 at 3:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .