1

I am sorting a large number of archived picture files where some are duplicated, and I want to remove the duplicates. I use the following shell script to get a list of dupes.

fdupes -r -A /Users/yves/Pictures >> "/Users/yves/Desktop/Dupes-new-$(date +%Y-%m-%-d-%Hh%M).txt"

However, there are a number of cases where the number of duplicated files is above 2, such as 3 or more, as shown below:

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_*2702.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_2702 2.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_2702.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK2702.NEF

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_FF1629 2.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_FF1629.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/DSC_1629.NEF

When the number of dupes is consistently exactly 2, I have used an awk command to retrieve the first line of each group: awk '{if(NR%3==1) print $0}' filepath > result and created a file "TO-DELETE". And then I deleted the files identified in the file by using the following AppleScript script:

set srcFile to ((path to desktop) as text) & "TO-DELETE.txt"

set lns to paragraphs of (read file srcFile as «class utf8»)
repeat with i from 1 to count lns
    if (item i of lns) is not in {missing value, ""} then
        set f to POSIX file (item i of lns)
        tell application "Finder"
            try
                move f to trash
            on error
                display dialog "Error with file" & space & (item i of lns as string)
            end try
        end tell
    end if
end repeat

But I cannot see an easy and reliable path to handle the dupes whose number is above 2. I have now several thousand duplicated files, and therefore manual editing is not an option. Any suggestion and help?

2
  • Slightly off-topic, but you can delete the found duplicates directly from your shell by running rm $(< /path/to/TO-DELETE.txt).
    – nohillside
    Commented Jul 29, 2023 at 11:24
  • Thanks, but I am obviously more inclined to be able to review what was deleted and to recover the fiiles if soemthing unexpected occured in the proposed deletion.
    – Yves
    Commented Jul 30, 2023 at 12:16

2 Answers 2

0

Assuming a text that resembles this, the script below would delete each file after the first of every clump. So in this example, four files would be retained, seven would be trashed.

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_*2702.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_2702 2.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_2702.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK2702.NEF

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_FF4629.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/DSC_4629.NEF

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_FF1629 2.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_FF1629.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/DSC_1629.NEF

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_3702 2.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_3702.NEF

No spaces at end of lines; empty line between each clump of duplicates;

What it does is split the text on the double paragraphs and create a list of filepath clumps — four in this example. It then cycles through each clump and puts the first item into the keeps list (optional) and the rest into the loses list. Each action type is logged to make the files' fates traceable but this logging can be commented out.

Note that the loses list is not flat. It will contain four lists, each of which will contain however many files requiring deleting.

It then cycles through each batch of files in loses and then through each file in each batch, converting each posix path to a file url for the finder to move to the trash.

If testing is successful then you can change the beginning so that dupeList becomes the list of files in your 'TO-DELETE.txt' file. If for some reason, it is imperative that you keep the last file in each clump, then you could simply reverse each list before the rest of text items operation.

set dupeList to "/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_*2702.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_2702 2.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_2702.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK2702.NEF

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_FF4629.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/DSC_4629.NEF

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_FF1629 2.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_FF1629.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/DSC_1629.NEF

/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_3702 2.NEF
/Volumes/Archives-photos-new/Fichiers-RAW/_NIK_3702.NEF"

set AppleScript's text item delimiters to (linefeed & linefeed)
set fileClumps to text items of dupeList -- each clump of files between empty lines

set keeps to {} -- files to keep
set loses to {} -- files to trash
set AppleScript's text item delimiters to linefeed
repeat with x from 1 to count fileClumps
    set end of keeps to first text item of contents of item x of fileClumps
    log keeps -- optional
    set end of loses to rest of text items of contents of item x of fileClumps
    log loses -- optional
end repeat

repeat with y in loses -- every batch of files to trash
    repeat with z in y -- each file in the batch
        set w to POSIX file (contents of z) -- each file as «class furl»
        tell application "Finder"
            try
                move w to trash
            on error
                display dialog "Error with file" & space & w
            end try
        end tell
    end repeat
end repeat
1
  • Thanks to all. It turns out that while visiting my son proposed an alternate option, based on Java, that sorts the proper files to delete in the file generated by fdupes. Below is his solution: I know nothing about Java, but with his help I was able to process successfully my task.
    – Yves
    Commented Jul 30, 2023 at 12:22
3

fdupes can delete duplicates automatically:

fdupes -r -A -D --noprompt --delete /Users/yves/Pictures

Leave out the --noprompt option if you want to confirm first that it works as expected.

If you don't trust it at all, you can also use fdupes -r -A --omitfirst /Users/yves/Pictures >> ~/TO-DELETE.txt to list all found duplicates without the first one.

3
  • Thanks, but for obvious safety concerns I want to be able to review what is going to be deleted and to retrieve files deleted by mistake.
    – Yves
    Commented Jul 30, 2023 at 12:18
  • @yves you can use the second version then.
    – nohillside
    Commented Jul 30, 2023 at 18:03
  • Thanks, I am learning a lot ans will test the various options.
    – Yves
    Commented Jul 31, 2023 at 19:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .