0

I use a cryptocurrency and it's blockchain is represented on disk by a single large lmdb database file.

The file is 72GB or so.

Each day the crypto node updates the chain with new transactions so the file is always getting added to.

I'm running Time Machine and seeing huge backups.

I thought Time Machine only backed up deltas of files? (i.e the difference between the old file and new file).

Does Time Machine instead backup a file completely, EVERY time it changes by a bit?

It seems that way.

Is time machine so that it will backup a full 70~ gigs a day each time my blockchain db changes slightly?

This seems wrong to me.

What do video editors do?? Aren't they always hitting this issue with large video files, 4K and whatnot?

  • Video files don't change - the edits are soft & held inside the editing software, partly to avoid precisely the issue you're facing & partly just because it makes a whole lot more sense to do it that way. Backup software cannot backup only changes in a single file, especially if that file is encrypted, which I would imagine a blockchain is. – Tetsujin Apr 4 at 17:08
  • @Tetsujin so you think it is making a full copy of the chain each time it changes slightly? – Woodstock Apr 4 at 17:09
  • What happens if you loose the disk between two backup runs? Will you loose all transactions recorded in the meantime? – nohillside Apr 4 at 17:12
  • @nohillside the node keeps the blockchain in sync from any point in history if that's what you mean. – Woodstock Apr 4 at 17:15
3

Time Machine works on file level, so every time a file changes the full file will be backed up. As you‘ve found out it‘s not well suited for large files with frequent small changes.

One way out would be to exclude the directory containing your blockchain file from Time Machine and use rsync to do backups.

  • Cool thanks. I honestly thought Time Machine diffed files like Git and took the deltas. I can't believe it just takes the ENTIRE file if there is a ANY change. Any backup software work like I thought Time Machine works? – Woodstock Apr 4 at 17:14
  • 1
    You get similar with VM 'drive' files. It's the way of the world, I'm afraid. For most use-cases it's probably not efficient to run a dif. It would also affect how the entire chain of difs could be saved. – Tetsujin Apr 4 at 17:19
  • 1
    @Woodstock Running a diff implies that the file is guaranteed to remain unchanged while the backup is running (which in your case would require you to close/quit the crypto currency application for each backup). – nohillside Apr 4 at 17:22
  • 1
    Ok cool, at least its running as expected. thanks for the help gents. I will exclude the chain. – Woodstock Apr 4 at 17:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .