I have a million+ files spread across four fiber-connected arrays. I need to get the path, filesize, and last_modified date to insert into a mysql database. What would be the best way to do this?

Here is what I am currently doing:

sudo find "$FULFILLMENT" "$ARCH1" "$ARCH2" "$MASTERING" -type f -exec ls -lT {} + > $FILE

And then I parse that file (using Python) to insert it into my db.

What would be the best way to accomplish the above?

  • Is there anything wrong with what you are currently doing? Also, I don't see how we can answer this properly without some insight on which files and on what location you need to search. – Gerry Feb 1 '13 at 22:52
  • There are a millions files spread across four volumes. They're all types of files (really, it's four different computers with 100's of TB storage each). What I'm currently doing takes about 3 hours, I was wondering if there's something that is faster. – David542 Feb 1 '13 at 22:53
  • And how much of those 3 hours does the find command take to complete? In any case, I do believe find is the most appropriate tool for listing the files. – Gerry Feb 1 '13 at 22:55
  • The find command takes about 99% of the time. – David542 Feb 1 '13 at 23:00
  • You could try find -type f -ls, though I'm not sure that will make any significant difference. It sounds like there just is a lot of data to process. – Gerry Feb 1 '13 at 23:04

You must log in to answer this question.

Browse other questions tagged .