10

I have a folder containing many folders containing many files. Thousands.

I can do find . -type f > ./FILE-LISTING.TXT to create a file containing many thousands of file paths that looks like this:

./Anders/Letters/20190101 Rent.pdf
./Anders/Letters/20190103 Appeal.pdf
./Anders/Letters/20190107 Decision.pdf
./Beeker/Letters/20180101 Rent.pdf

How would I feed that list of filepaths into md5 to produce an output formatted like this:

9cf14e4d666dcb6aab17763b02429a19 ./Anders/Letters/20190101 Rent.pdf
d1bb70baa31f1df69628c00632b65eab ./Anders/Letters/20190103 Appeal.pdf
7a0f5bc18688fe8ba32f43aa6ec53fb1 ./Anders/Letters/20190107 Decision.pdf
a0c96a79cf3b1847025d9f073151519d ./Beeker/Letters/20180101 Rent.pdf

NB: I want the md5 hashes of the referenced files, not the md5 of the list of files, nor the md5 hashes of the strings in the file-listing.txt.

Also, would it be faster to do it all in one command line, or do it in two passes (find to create file-listing.txt, then md5 to create file-listing-md5.txt)?

  • 3
    This is a superb question - clear, poses a few challenges, but is going to be very doable since every tool for automation on MacOS needs to handle spaces in file names, loops and variables to handle the changing file being processed. Well done - I hope we get some great answers in python, bash, swift and other options for scripting. – bmike Jul 10 at 11:23
  • 3
    What is your use case for this file? mtree is a tool already available to monitor file hashes and detect changes to filenames, file contents, permissions or datestamps. man mtree for details. mtree -c -K md5digest – Jim L. Jul 11 at 6:14
  • The use case is to hand a file of hashes and filepaths to a third party RDBMS which tracks a lot of extra detail not present in the file system. If the files get moved around, they can be re-linked. If the file is edited in place it can be re-linked. – Erics Jul 11 at 11:04
11
find . -type f -exec /sbin/md5 -r {} +
       ^^^^^^^ ^^^^^ ^^^^^^^^^^^^ ^^ ^
          |      |        |       |  |
          |      |        |       |  +- add as many file names as possible per call
          |      |        |       +---- replace with names of found files
          |      |        +------------ command to run
          |      +--------------------- execute following command
          +---------------------------- find any "normal" file

should do the trick (and take care of the usual issues with spaces etc within filenames).

As for faster: one pass is almost always faster than two passes. In the specific case the MD5 calculation takes so much time that other factors most probably can be ignored.

PS: Tip of the hat to @lhf for reminding me of -r

  • 4
    Both @nohillside and @lhf provided good, valid answers. On a whim, I decided to see if one is substantially better than the other. I ran both on a directory containing more than 64,000 files under time. The find -exec version was about 3 seconds faster than find | xargs. However, the run time for both was around 45 seconds, meaning that (a) the difference is less than 10% and (b) the time is probably I/O bound (printing to the console). – Craig S. Cottingham Jul 10 at 19:09
  • This is almost certainly I/O bound (but not for printing to the console, it has to digest all those files, that's gonna take time) – Thilo Jul 11 at 9:51
  • @CraigS.Cottingham There are about that many files, but in deeply nested directories - not just one directory - which might explain why the command line I inherited takes about 15 minutes to run. Next time I'm on site I'll do a comparison too. – Erics Jul 11 at 11:08
  • @Erics Simple find commands (as the one you have in the question) are purely I/O bound. When calculating MD5 hashes as well, it could be both I/O (for reading all data) or CPU (for calculating the hash), but this then depends on the hardware used. – nohillside Jul 11 at 11:15
9

Try this:

find . -type f -print0 | xargs -0 md5 -r

Note -print0 and -0 to handle spaces in filenames.

Compared to find . -type f -exec, this solution runs md5 much less frequently, although this might not have a measurable impact.

  • 2
    find's -exec {} can also handle spaces in filenames. – fd0 Jul 10 at 11:30
  • 2
    Both @nohillside and @lhf provided good, valid answers. On a whim, I decided to see if one is substantially better than the other. I ran both on a directory containing more than 64,000 files under time. The find -exec version was about 3 seconds faster than find | xargs. However, the run time for both was around 45 seconds, meaning that (a) the difference is less than 10% and (b) the time is probably I/O bound (printing to the console). – Craig S. Cottingham Jul 10 at 19:09
  • Thanks for testing it out! – lhf Jul 10 at 20:10
  • What shell are you using? – fd0 Jul 10 at 20:13
  • @fd0, I'm using bash. – lhf Jul 10 at 21:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .