Search file duplicates in OSX by hash

Question

I am looking for a way to search for a determinate file in OSX (Maverick but more generally OSX). In particular I would like to do the following:
given a File_001 I'd like to search if in the filesystem exists a copy of this file.
Not just with the same name, I would like the comparison method to be an hashing algorithm like MD5, SHA etc..

Most of the "duplicate file finder" I have tried just search for all the duplicates in a drive/system. I would, instead, be interested in submitting one file and search for its duplicates.

Does anyone know if such a program exists? Maybe some obscure function of Spotlight?

Answer 1

You might also use fdupes. It doesn't have an option to search for duplicates of a specific file, but you can just grep the output for the filename:

fdupes -r1 .|grep filename

-r recurses into directories and -1 prints each group of duplicate files on a single line.

Other useful examples:

fdupes -r . finds all duplicate files under the current directory;

fdupes -r . -dN deletes all except the first duplicate from each group of duplicates;

fdupes -r dir1 dir2|grep dir1/|xargs rm removes duplicates in dir1.

You can install fdupes with brew install fdupes.

Answer 2

You can easily build this yourself with some shell commands:

• find ~ -type f -exec md5 -r '{}' \; > /tmp/md5.list

  will build a list of md5 hashes over all your files.

• grep $(md5 -q FILE-TO-SEARCH) /tmp/md5.list

  will search for the md5 hash of FILE-TO-SEARCH

Running the first command (especially if you run it across the whole disc) will take a long time though.

---

If you only want to search for one file, you can also use

SIZE=$(stat -f '%z' FILE-TO-SEARCH)
MD5=$(md5 -q FILE-TO-SEARCH)
find ~ -type f -size ${SIZE}c | while read f; do
    [[ $MD5 = $(md5 -q "$f") ]] && echo $f
done
    

Answer 3

This should work if you substitute the size and hash for FILE_001 into the command.

198452 bytes is the file size I used and the file md5 hash is 3915dc84b4f464d0d550113287c8273b

find . -type f -size 198452c -exec md5 -r {} \; |
    grep -o "3915dc84b4f464d0d550113287c8273b\ \(.*\)" | awk '{print $2}'

The output will be a list of files with path names relative to the directory sent to the find command.

This approach has the advantage that it will only hash files that match the size of your original and will only output file names that match the hash.

Answer 4

If you don't want to mess with scripts, you can get close to the behavior you want with Araxis Find Duplicate Files $10 in the Mac App Store. There is also a 7 day demo on their web site. Find Duplicate Files searches for dupes by computing the hash for each file.

You can approximate the behavior you want you would set up a folder with the single file you are concerned about, then add the folders you want to search in. This will also report other dupes, if there are any, in the search paths.

This app has many nice sorting features making the results very easy to understand.

Answer 5

Suggesting some changes to Lri's answer. xargs runs into issues when path names have special characters. Since fdupes doesn't have a -print0 flag to pass to xargs like find does - it trips up.

The following command results in an error because of a single quote in a filename:

fdupes -r dir1 dir2|grep dir1/|xargs rm

xargs: unterminated quote

The UNIX way around this is to add another command between the grep and xarg commands and adding additional flags to xargs:

... | tr '\n' '\0' | xargs -0 -n1 ...

This addition comes from an excellent explanation at Make xargs execute the command once for each line of input

The full command would then be:

fdupes -r dir1 dir2 |grep "dir2/" |tr '\n' '\0' |xargs -0 -n1 rm -v

Adding the -v flag allows you to see what has been removed.

Check this command first using echo or another non-destructive command before using rm.

An example of a non-destructive option is to use the tag command (install with homebrew). Add a red Finder tag to files that are duplicates so you can manually select and drag to trash :)

fdupes -r dir1 dir2 | grep  "dir2" |  tr '\n' '\0' | xargs -0 -n1 -I % tag -a red %