13

I am looking for a way to search for a determinate file in OSX (Maverick but more generally OSX). In particular I would like to do the following:
given a File_001 I'd like to search if in the filesystem exists a copy of this file.
Not just with the same name, I would like the comparison method to be an hashing algorithm like MD5, SHA etc..

Most of the "duplicate file finder" I have tried just search for all the duplicates in a drive/system. I would, instead, be interested in submitting one file and search for its duplicates.

Does anyone know if such a program exists? Maybe some obscure function of Spotlight?

  • 1
    Be wary of a hash determining whether a given file is a copy. This approach may fail with .emlx files (Apple Mail's file format), for example. As an aid to Spotlight, OSX appends metadata to mail files. The same email in two different paths may have different metadata even though the Message-id is the same. Different hash for two files containing the exact same raw email. – John D. Jan 22 '15 at 22:31
13

You might also use fdupes. It doesn't have an option to search for duplicates of a specific file, but you can just grep the output for the filename:

fdupes -r1 .|grep filename

-r recurses into directories and -1 prints each group of duplicate files on a single line.

Other useful examples:

fdupes -r . finds all duplicate files under the current directory;

fdupes -r . -dN deletes all except the first duplicate from each group of duplicates;

fdupes -r dir1 dir2|grep dir1/|xargs rm removes duplicates in dir1.

You can install fdupes with brew install fdupes.

7

You can easily build this yourself with some shell commands:

  • find ~ -type f -exec md5 -r '{}' \; > /tmp/md5.list

    will build a list of md5 hashes over all your files.

  • grep $(md5 -q FILE-TO-SEARCH) /tmp/md5.list

    will search for the md5 hash of FILE-TO-SEARCH

Running the first command (especially if you run it across the whole disc) will take a long time though.


If you only want to search for one file, you can also use

SIZE=$(stat -f '%z' FILE-TO-SEARCH)
MD5=$(md5 -q FILE-TO-SEARCH)
find ~ -type f -size ${SIZE}c | while read f; do
    [[ $MD5 = $(md5 -q "$f") ]] && echo $f
done
  • The very first pass should be a find by exact size operation. – biziclop Jan 26 '14 at 11:17
  • @biziclop If you only want to search for one file, yes. If you want to search for several it's faster to build the index once and just search through the index file afterwards. – nohillside Jan 26 '14 at 11:58
  • 1
    It's true of course, I just noticed this sentence in the question: "I would, instead, be interested in submitting one file and search for its duplicates." – biziclop Jan 26 '14 at 12:26
1

This should work if you substitute the size and hash for FILE_001 into the command.

198452 bytes is the file size I used and the file md5 hash is 3915dc84b4f464d0d550113287c8273b

find . -type f -size 198452c -exec md5 -r {} \; |
    grep -o "3915dc84b4f464d0d550113287c8273b\ \(.*\)" | awk '{print $2}'

The output will be a list of files with path names relative to the directory sent to the find command.

This approach has the advantage that it will only hash files that match the size of your original and will only output file names that match the hash.

0

Here is the list of the utils for this: list of dupes finders. Some of them are highly optimized and will be much faster than whatever you'll come up with any script.

0

If you don't want to mess with scripts, you can get close to the behavior you want with Araxis Find Duplicate Files $10 in the Mac App Store. There is also a 7 day demo on their web site. Find Duplicate Files searches for dupes by computing the hash for each file.

You can approximate the behavior you want you would set up a folder with the single file you are concerned about, then add the folders you want to search in. This will also report other dupes, if there are any, in the search paths.

This app has many nice sorting features making the results very easy to understand.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .