I have several hard drives I've used throughout the years to backup my data. Each one has a different folder structure and fileset. I now have a massive NAS that I want to consolidate all my data onto. I've created a new folder structure on the NAS and have moved over most of my data.

Does anyone know of a tool that could index my NAS and then compare any other hard drive, folder, or other media and list what folders/files are not on the NAS.

I believe I could write a script that would md5/sha1 each file on the NAS and put that data into a database. Then the script could compare using those hashes and list differences. I'm hoping for a better solution if one exists.

  • Do you mind githubbing/sharing your script? I have about 20TB of photos etc and I had thought of the similar solution keeping all the various names and dates for the files in a text file for reference or including it in the meta data for each file. At one point, I imagined it would be possible to use Dropbox's autoupload feature to consolidate data as it does not upload photos it already has uploaded but you need a few TB's on your pc to transfer each drive for DP. Also their system slows down beyond 300k files.
    – Goodcy
    Oct 11 '18 at 1:43

To close out this question, I didn't really find anything out there already.

I ended up writing a python script that read all the files on the drive and saved an MD5 hash of each one in a database.

Then I created another script that would read the contents of the drive to compare and lookup the MD5 in the database to see if the file already existed. It's not perfect, but it worked.


It is possible to transfer photos say in 20-100 GB chunks at a time to an SD card or a USB flash card and use Dropbox uploader to scan the thumb drive and upload only the new files automatically to Dropbox. Once the unique files are uploaded to Dropbox, it is possible to download them all to a new hard drive. This was even if you dont use Dropbox regularly, you can use them for a few days or weeks just to get the job done. It is also possible to write scripts in unix shell, Ruby, PHP or Python to create a database and transfer the unique files which you seem to have done. The dropbox solution is for the layman. Hope this helps others.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .