0

I use a cryptocurrency and it's blockchain is represented on disk by a single large lmdb database file.

The file is 72GB or so.

Each day the crypto node updates the chain with new transactions so the file is always getting added to.

I'm running Time Machine and seeing huge backups.

I thought Time Machine only backed up deltas of files? (i.e the difference between the old file and new file).

Does Time Machine instead backup a file completely, EVERY time it changes by a bit?

It seems that way.

Is time machine so that it will backup a full 70~ gigs a day each time my blockchain db changes slightly?

This seems wrong to me.

What do video editors do?? Aren't they always hitting this issue with large video files, 4K and whatnot?

Improve this question
4
  • Video files don't change - the edits are soft & held inside the editing software, partly to avoid precisely the issue you're facing & partly just because it makes a whole lot more sense to do it that way. Backup software cannot backup only changes in a single file, especially if that file is encrypted, which I would imagine a blockchain is.
    – Tetsujin
    Commented Apr 4, 2019 at 17:08
  • @Tetsujin so you think it is making a full copy of the chain each time it changes slightly?
    – Woodstock
    Commented Apr 4, 2019 at 17:09
  • What happens if you loose the disk between two backup runs? Will you loose all transactions recorded in the meantime?
    – nohillside ♦
    Commented Apr 4, 2019 at 17:12
  • @nohillside the node keeps the blockchain in sync from any point in history if that's what you mean.
    – Woodstock
    Commented Apr 4, 2019 at 17:15

2 Answers 2

Reset to default
4

Time Machine works on file level, so every time a file changes the full file will be backed up. As you‘ve found out it‘s not well suited for large files with frequent small changes.

One way out would be to exclude the directory containing your blockchain file from Time Machine and use rsync to do backups.

Improve this answer
4
  • Cool thanks. I honestly thought Time Machine diffed files like Git and took the deltas. I can't believe it just takes the ENTIRE file if there is a ANY change. Any backup software work like I thought Time Machine works?
    – Woodstock
    Commented Apr 4, 2019 at 17:14
  • 1
    You get similar with VM 'drive' files. It's the way of the world, I'm afraid. For most use-cases it's probably not efficient to run a dif. It would also affect how the entire chain of difs could be saved.
    – Tetsujin
    Commented Apr 4, 2019 at 17:19
  • 1
    @Woodstock Running a diff implies that the file is guaranteed to remain unchanged while the backup is running (which in your case would require you to close/quit the crypto currency application for each backup).
    – nohillside ♦
    Commented Apr 4, 2019 at 17:22
  • 2
    Ok cool, at least its running as expected. thanks for the help gents. I will exclude the chain.
    – Woodstock
    Commented Apr 4, 2019 at 17:30
0

I know this question is 3+ years old, but I would just point out that locally backing up a blockchain data file is completely pointless. If the working copy ever got corrupted (unlikely in the first place, since a well designed blockchain is constantly syncing and checking its own integrity anyway), the blockchain software should simply re-download and sync the current blockchain from the blockchain node network. As the OP concluded, simply taking that directory out of the Timeshare backup rotation is the correct solution.

Improve this answer
2
  • Having a local backup of a >70 GB file has its value, even if it can also be rebuilt over the network. Not everybody has fast Internet.
    – nohillside ♦
    Commented Nov 22, 2022 at 18:19
  • op here. Exactly that. Also for the blockchain in question, the validation process from scratch is computationally expensive.
    – Woodstock
    Commented Nov 22, 2022 at 19:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .