Tell me more ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.
up vote 1 down vote favorite

I have a million+ files spread across four fiber-connected arrays. I need to get the path, filesize, and last_modified date to insert into a mysql database. What would be the best way to do this?

Here is what I am currently doing:

sudo find "$FULFILLMENT" "$ARCH1" "$ARCH2" "$MASTERING" -type f -exec ls -lT {} + > $FILE

And then I parse that file (using Python) to insert it into my db.

What would be the best way to accomplish the above?

share|improve this question
Is there anything wrong with what you are currently doing? Also, I don't see how we can answer this properly without some insight on which files and on what location you need to search. – Gerry Feb 1 at 22:52
There are a millions files spread across four volumes. They're all types of files (really, it's four different computers with 100's of TB storage each). What I'm currently doing takes about 3 hours, I was wondering if there's something that is faster. – David542 Feb 1 at 22:53
And how much of those 3 hours does the find command take to complete? In any case, I do believe find is the most appropriate tool for listing the files. – Gerry Feb 1 at 22:55
The find command takes about 99% of the time. – David542 Feb 1 at 23:00
You could try find -type f -ls, though I'm not sure that will make any significant difference. It sounds like there just is a lot of data to process. – Gerry Feb 1 at 23:04
show 1 more comment

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.