Take the 2-minute tour ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.

I have a million+ files spread across four fiber-connected arrays. I need to get the path, filesize, and last_modified date to insert into a mysql database. What would be the best way to do this?

Here is what I am currently doing:

sudo find "$FULFILLMENT" "$ARCH1" "$ARCH2" "$MASTERING" -type f -exec ls -lT {} + > $FILE

And then I parse that file (using Python) to insert it into my db.

What would be the best way to accomplish the above?

share|improve this question
    
Is there anything wrong with what you are currently doing? Also, I don't see how we can answer this properly without some insight on which files and on what location you need to search. –  Gerry Feb 1 '13 at 22:52
    
There are a millions files spread across four volumes. They're all types of files (really, it's four different computers with 100's of TB storage each). What I'm currently doing takes about 3 hours, I was wondering if there's something that is faster. –  David542 Feb 1 '13 at 22:53
    
And how much of those 3 hours does the find command take to complete? In any case, I do believe find is the most appropriate tool for listing the files. –  Gerry Feb 1 '13 at 22:55
    
The find command takes about 99% of the time. –  David542 Feb 1 '13 at 23:00
    
You could try find -type f -ls, though I'm not sure that will make any significant difference. It sounds like there just is a lot of data to process. –  Gerry Feb 1 '13 at 23:04
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.