13

Many years ago I vaguely recall that the Finder in Mac OS X would start having problems if a folder contained 2,000 - 3,000 items or more.

Apple doc says that the HFS Plus file system has a theoretical limit of a 2 billion files per folder all Mac OS X versions.

What is the practical limit?

Will having 10,000 photos in a folder be a problem?

Improve this question
7
  • 2
    In 10.6 which I still use at work. the 2000-3000 problem still exists. I have not had the opportunity to use this sort of mass of files on later versions. But I suspect if the docs still say the same thing then the same limit may be a problem. I should add that I never ever have the problem of 2000-3000 on the Local drive. But only across a network drive.
    – markhunte
    Dec 14 '13 at 10:25
  • 1
    I don't know exactly where the limit is drawn, but millions of files will definitely put you in a very big problem (good luck deleting such a folder even with rm -rf).
    – Ohad Schneider
    Feb 24 '17 at 17:20
  • In early versions of macOS X 10.6 at least you would experience performance issues when you placed a "large", 2k to 3k, amount of files in you desktop folder. Something to do with drawing all the icons on the desktop. I didn't hear about a problem with other folders.
    – historystamp
    Jun 17 '19 at 20:21
  • I'm going to put this on hold. If we want more answers, we'll want to explain what sort of timing / requirement makes a wait "practical" - the answer on making 10,000 files show that on most systems, we will run out of patience to look at files in Finder before the system slows down or cannot handle the files.
    – bmike ♦
    Jun 17 '19 at 21:20
  • @bmike I do agree with you except that the new Apple File System (APFS) that replaces HFS+ puts a new spin on this topic. Perhaps I should post a new Question on APFS specifically if not yet existing?
    – Basil Bourque
    Jun 17 '19 at 22:28
5

Seems that around 10,000 is safe. However, I've found that if you go a lot higher like 50,000 Finder will never even list the files in the directory when you try to browse it. I suspect this is why a lot of data recovery software will create a new folder every 10,000 files if you're doing file carving in RAW.

Improve this answer
8

You can easily try this yourself by running the following in Terminal

mkdir ~/t
cd ~/t
dd if=/dev/random of=test bs=1024 count=16
for i in {1..10000}; do cp test test.$i; done

to create a folder containing 10'000 files with 16kB each (replace the 16 in the third line with another number for differently sized files).

Improve this answer
5
  • 2
    A late word of caution: Do not do that within your home directory unless you want to grow your fame with your local system administrator. Depending on the backup tool of his choice, he might not like the effects of his tool trying to archive your precious experiment. Experiments like these are best restricted to media that is not part of a backup plan. /tmp would be a better choice than ~/.
    – Tatjana Heuser
    Jun 17 '19 at 18:13
  • @TatjanaHeuser Not sure where your interest in this topic comes from, but on macOS a user typically is the local admin (even if they may not be aware of that) and /tmp is a symlink to /private/tmp which by default resides on the main (only) partition.
    – nohillside ♦
    Jun 17 '19 at 18:19
  • True for desktop Macs - and it doesn't change the backup problem. Even on MacOS, /tmp gets cleared between reboots. (Unlike /var/tmp). #alt.follkore.computers aside: I have known administrators to curse the curious user who created one tiny file with a large hole to give lseek a try when the number of tapes for their backup went way beyond the capacity of the file system being backed up...
    – Tatjana Heuser
    Jun 17 '19 at 18:31
  • @TatjanaHeuser I've come back from photo trips and uploaded hundreds of RAW files between 30 and 40 GB each to my Mac. TimeMachine handled that without any problems at all :-)
    – nohillside ♦
    Jun 17 '19 at 19:04
  • @nohillside 40GB RAW files isn't a challenging scenario at all. Filesystems usually fall over when you have terabytes of tiny files. Big files are almost never a problem.
    – Abhi Beckert
    Oct 21 '20 at 6:11
5

Answering considering a practical example: I have now 326.000 files in a folder, created by an application that download bits from a server. The files are zipped XML files, and my application extracts XML data from it and store it on a local database.

The application runs from the command line. Everything works fine without any issue but rm * or ls * does not work due to the expansion of the wildcard (error message Argument list too long). Since the files are stored in a temporary folder I can just remove the folder after processing the files.

I didn't try to open the folder with Finder, though. I suspect that could be very slow if possible at all.

Improve this answer
1
  • 2
    Same use case here—I downloaded over 290,000 files into a directory from a site I was archiving... and made the mistake of trying to open it in the Finder. After letting it spin for like 20 minutes I force restarted the Finder, then closed that folder and did everything else in the Terminal (lots of find | xargs!)
    – geerlingguy
    Jan 25 '18 at 3:25
4

There are several limits to consider that have been touched upon by some of the comments:

  • argument length and shell expansion - a simple echo * will bail if the concatenated length of filenames the asterisk expands to hits upon that limit. If running into this snare, often find will be your friend. find . -depth 1 -type f | exec echo {} \; would be a working replacement for the innocent echo * mentioned above, limited to listing files only. (echo to be replaced with the action of your choice)

  • per-program limits to the size of internal data structures used to hold directory contents (finder, all kinds of tools trying to read directory listings).

  • directory lookup cache size. While the filesystem may be able to hold 2.1 billion files within the on-disk structure of a directory, it won't be pleasant to work with that number, and you'd be well advised to introduce some strategy of sorting files into subdirectories if you're dealing in structures of that size. (Hint: those people designing web caching structures did have to deal with that - see Maltzahn/Richardson, Reducing the Disk I/O of Web Proxy Server Caches, Usenix 1999.

To speed up access to frequently used disk structures, file systems are using (memory) cache, and the size of these caches is limited. This is where the sudden penalty for large and less-than-optimal structured directorie starts to hit. Depending on the frequency and intensity of access to these directories, the penalty can be significant.

The 2015 article by Tsai et al., How to get more value from your file system directory cache would probably be one of the easier introductions to the subject.

Improve this answer
2

Apple has a support document related to that:

Maximum number of files (or files and folders) in a folder (all Mac OS X versions)

Up to 2.1 billion (2)

Improve this answer
2
  • Yes, indeed, I did include that link and fact in my question (2nd paragraph). I'm asking about practical real-world limits.
    – Basil Bourque
    Dec 14 '13 at 8:30
  • 2
    Thanks for this, I had a good laugh. Try having 4M files in a single folder and let me know how it goes (hint - you can't view it, can't ls, can't find, can't delete it and so forth).
    – Ohad Schneider
    Feb 24 '17 at 17:18

Not the answer you're looking for? Browse other questions tagged .