Find and delete duplicate mails in Apple Mail (Big Sur/M1)

Question

How can I find and delete duplicate mails in Apple Mail (on Big Sur/M1)? Is there a (free) app or a script for that?

Problem was discussed here long ago: remove duplicate emails. However, the Araxis App and the bohemianboomer-link mentioned there, are not available anymore.

Also JollyRoger’s free Remove Duplicate Messages AppleScript (Direct Download Link), which worked fine with Apple Mail on macOS Mojave, does not seem to work with Apple Mail on Big Sur (on my M1 MacBook Air at least).

Answer 1

For a little faster and perhaps bit easier method, you can skip the EXPORT. I processed ~96k emails, removing ~37K duplicates in under 10 minutes. {helps to have 4-2TB nvme (x4) m.2s on tb3/4}

1. Close Mail

2. Go to ~/library/mail/v9

3. Run Duplicate Finder Pro (DFP) on all folders you want to dedupe except "MailData"

4. At this point, you are ready to re-index(es) via one of the two methods.
   A.) Start Mail and re-index each folder you added to DFP. Simply select the folder and go to Mail's menu Mailbox->Rebuild. You are done!

   B.) Go into ~/library/mail/v9/MailData and delete the three Index files with names ending in Index, Index-shm and Index-wal. Next, open Mail it will appear you are starting mail for the first time, but will only rebuild all of your mail indexes. You are done!

   a. Reindex inside mail app by selecting each email box or folder (local storage)

Answer 2

The only way I've found, using macOS' standard Mail app and a simple duplicate file software, is:

• Export all your emails as .eml (select, and drag-drop emails into a Finder folder, from the Apple Mail app). See answer https://apple.stackexchange.com/a/350900/278784 for more details on how to export, but also import back after deletions are done. Note that when selecting a lot of emails and dragging to a folder, in my case it took a very long time with seemingly nothing happening until the .eml files were actually exported.
• Use a duplicate file software, I use Duplicate File Finder from the App Store, which is free for exact duplicate file deletion. Delete all duplicates that it finds. This worked great for me, it found all duplicates (I think, because I have >100 000 emails so...).
• Import back the remaining .eml files, also using the details from the answer linked above. You could then delete all the original emails, or do it before you import back the cleaned up emails, but make sure you have a back up.

As a benchmark, exporting ~10 000 emails into .eml files in a folder took me over 2 hours. During that time, I could check if something was actually going on by looking in Activity Monitor: Mail.app was using around 80% of a processor core, even dropping later to below 30% after a while. Otherwise Mail.app when not doing anything was using only way under 1%.