Comparing files within a folder

Question

I have a folder with many book files in different formats e.g. .txt, .epub, .rtf, etc.

Many of them are duplicates but have different names e.g.:

Brooks, Terry - Shannara 1 - The Sword of Shannara.txt

Brooks, Terry - Shannara 01 - The Sword of Shannara.txt

If I check these files with filemerge they are exactly the same file.

I have tried running several programs like Gemini on the folder in question with no results.

Every suggestion online seems to involve comparing two folders. I do NOT want to compare two folders. I only want to find the duplicate files within a directory or its sub-directories.

I was looking for a script or terminal command to loop through the folder(s) and show which files have the same content even though they may have different filenames, creation dates, modification dates, etc.

Any help would be much appreciated.

[edit] - I should clarify that I want to find a way to compare two files even though they may be named differently but are of the same type. e.g. text files, or epub files. I do not wish to compare them between types a text file and an epub, of couse, would be different.

Answer 1

fdupes may be your solution. It can be installed via homebrew with

brew install fdupes

If you don’t have homebrew, you can install homebrew with

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Running fdupes -r <directory> will list duplicate files within the directory, even if they’re not in the same folder. fdupes compares based on file contents, not names, so the names being different will not be a problem.

You can use the -d option to be prompted which files to keep (deleting all others), but I would suggest doing the deletions manually.