Take the 2-minute tour ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.

I'm a tech for my local school district and we are having some problems with our iMac Multimedia Lab. Over the past nine months, 5 of the 22 iMacs have suffered from filesystem corruption. The only recourse for these machines has been to completely wipe out the OS X partition and start fresh (with appropriate restore from Time Machine.)

Here's the list for why I know it's file system corruption:

  • The iMac will not boot OS X. I have seen it stop at the "throbber", the progress bar, or just the Apple logo.

  • Mounting the iMac disk via Target Disk Mode (love that feature) succeeds, but only for the Bootcamp partition. The OS X partition fails to mount.

  • A verify of the disk reveals the OS X partition needs to be repaired (I've seen invalid sibling entries, orphaned children etc.). Attempting to repair the disk fails. This latest attempt (just yesterday) said that the catalog B trees could not be rebuilt. I should have made more complete notes on what was said each time, but each time until this last one I assumed it was an odd one-in-a-million kind of event. A fluke

  • Attempting to repair or rebuild the drives with Drive Genius 3 also fails So far 98% of the user's files have been recoverable via Data Rescue 3 The physical hard drive does not appear to be failing (retrieving files from the drive does not hang or "beachball", the drive does not appear and disappear in Disk Utility, Finder remains responsive, etc.)

  • Normally I'd chalk it up to a series of bad drives. Just happened to be the unlucky guy who purchased a bad run of iMacs, right? Here's where things start to get interesting. I submit to you, the list of oddities:

  • The drives verify as good via SMART

  • The RAM checks out
  • After deleting and re-creating the partition (and re-installing OS X) all problems disappear.
  • The corruption has not happened to the same Mac twice
  • Bootcamp is installed on the same drive and functions before, after, and during the corruption on the Mac side.
  • The Bootcamp partition has not had this issue on ANY of the iMacs

Also, to rule out the obvious:

  • There have been no brownouts or surges

  • We seriously doubt a virus, as the malfunctions appear anywhere from simultaneously (two machines went down at the same time about a month ago) to months apart. Plus, the user's documents are restored after reformat, so one would surmise that if it were a malicious program the Mac would keep failing again and again.

  • The machines have been in a climate-controlled area

  • It has not been the same user affected

  • Sometimes the problem occurs after an unavoidable hard shutdown (which occurs only infrequently. These machines are not being excessively powered down improperly. Only what you would expect with a Mac Lab running multimedia five days a week), other times it is completely out-of-the-blue

Frequently used software includes:

  • iPhoto
  • iDVD
  • iMovie
  • Safari

The machines are also loaded with Parallels 5, which loads the Bootcamp partition into a VM. Parallels was setup via the standard wizard, no oddball configuration or hacks.

And last but not least, the specs:

  • iMac 10,1 (21.5 inch)
  • Stock drives
  • OS X Snow Leopard (latest updates)
  • Stock memory
  • Joined to our Active Directory infrastructure
  • HFS+ file system (not case sensitive, the default for OS X Snow Leopard)
  • No out-of-the-ordinary drive maint. programs. Drive Genius was loaded yesterday afternoon (AFTER recovering from the latest failure) to run a verification on all iMacs, but was not installed prior. All Macs, both those that have failed in the past and those that have never failed, passed with flying colors.

TL;DR: The OS X partition has become corrupted on five different iMacs, but the physical drives are fine. WHY!?!?!

share|improve this question
    
Off Topic: 8¼b: You can use Shift+Enter to enter linebreaks into comment boxes. I've also heard that there are browser plugins (userscripts?) out there that reverses the role of enter & shift-enter, so enter will insert a line break, and shift-enter will submit the comment. –  Jason Salaz Apr 29 '11 at 16:59
    
+1 well documented question. Have you ever considered that it could be an act by students? –  user6124 Jun 28 '11 at 14:37
    
How does iMac 10,1 map to iMac {early|mid|late} 20YX for ease in looking into the hardware side of things? If you replace the #desktopcomputers with one of the serial numbers, support.apple.com/manuals/#desktopcomputers will get the descriptive name for your iMacs. –  bmike Jul 28 '11 at 16:52

4 Answers 4

HFS Plus (HFS+) is a fragile and a little outdated filesystem. If you google it, you'll find many reports of filesystem corruption.

Rebooting without unmounting the filesystem is the best way to corrupt it. This happens when the mac freezes for some reason (in my case it's the nvidia video card) or power failures.

Here are some tips, that IMHO should lower the chance of filesystem corruption:

  1. When system freezes, try rebooting from ssh. When the graphics subsystem of my mac freeses, it's still accessible via SSH - try opening ssh connection from your network and reboot it. You could use Apple Remote Desktop (€62) for this task. You should enable ssh access first.

  2. Do diskutil verifyVolume / periodically. Yes, even if HFS+ is a journaled filesystem, corruption is possible. You could use Apple Remote Desktop to run this on all classroom computers at once.

  3. Use multiple volumes. Using multiple volumes should reduce chance for corruption. Splitting / from /Users/ should make restoration easier (either / or /Users will be corrupted). Note that this probably could complicate things with Bootcamp.

  4. Mount partition(s) with options, which reduce writing. Mounting partitions with noatime option should reduce writing on it. By default every time a file is accessed, it's access timestamp is "touched".

  5. Make sure there are no attempts to mount HFS+ partition from other os-es. Is it possible that someone is starting a linux distro from usb/dvd and mounting / in rw mode or playing with journal settings?

Hope my answer is helpful.

PS: corruption usually is gradual not sudden. There is a possibility that something specific is causing this, software or workflow. My mind is at Parallels 5, but it should corrupt the bootcamp volume, not the MacOS one. Searching their KB does not reveal anything useful.

PPS: it is fragile because it has no actual system to correct corruption within a file. A journal records transfers and attempts to recopy data in order to return the filesystem to a consistent state but if the file lost is vital (like actual filesystem structure data) then there is no recourse. In fact, because the Catalog File (which is lists all the logical data information) is stored as a file, if it corrupted in certain places your entire filesystem is rendered useless garbage data, or partially tended garbage in the event that it is corrupted and the a journal replay occurs which causes it to restructure the filesystem in a way that is not consistent with the data (e.g. file a and b are 1MB and 2MB respectively but the replay changes them to be 2MB and 1MB resulting in half of the contents of B being inside A).

share|improve this answer
2  
Any filesystem will have problems considering circumstances. But "Fragile" and "A Little Outdated"? Really? A little outdated in my mind is no Journaling. And that's not the case for HFS+. –  Jason Salaz Apr 29 '11 at 16:58
    
If the filesystem were fragile, wouldn't all of the macs have issues? Installations with hundreds of macs that audit for abrupt power disconnects and a rolling schedule of safe boots, file system repairs will nip this sort of thing in the bud. –  bmike Jul 28 '11 at 16:42

I would consider student activity. If you are running bootcamp and OS X, then I bet the students are crashing the Mac side to run boot camp, because they aren't patient. This can also be the case if there is a screen locker present.

I suggest:

  1. Remove boot camp. Instead run a VM either under Parallels or Virtual Box. (I have found that Windows XP under VirtualBox runs quite well.) Configure this so that the Virtual Machine is immutable, stored locally on the old bootcamp partition. This will prevent kids from screwing around with the windows install. Redirect their home directory to read off of their Mac home directory on your server. (This is a PITA to set up the first time, as it encompasses about a zillion regedits)

  2. Establish some form of login monitoring so you know which students have used which machines. This has the advantage that you can spot the same kid logging in on multiple machines, usually doing this as a favour for someone who is banned. I set it up that when the same login was on two machines, both machines were rebooted.

share|improve this answer

Have you considered a periodic check of the machines? You could easily schedule weekly fsck verification passes (until you figure out why the corruption is happening) and then monthly to keep a tab on things.

With a journaled file system, it takes some repeated bad treatment for macs to degrade to the point of not booting. Even bad software doesn't write to the system side of booting, so I would suspect something is clearly amiss. On macs that get shut down cleanly and get attention the whenever minor filesystem errors are repaired (any time a mac restarts and fsck isn't running in preen mode is a sign of trouble on the horizon).

With a deployment of 25 macs, you can easily spend some time being proactive about file system checks and seeing which are not powering down cleanly by setting up a syslog server or other centralized auditing system.

share|improve this answer

Things that could do it off the top of my head...

you said you haven't had power surges or brownouts. How are you confirming it? We had a classroom where PC power supplies were blowing seemingly at random. We had to have maintenance staff connect a monitoring meter to the circuit and discovered that outlet is having huge voltage spikes.

Memory isn't seated properly and corrupting data.

Drive cables loose.

marginal hard disks that have a bad set of sectors but not bad enough that it's triggering alerts or scans for bad sectors.

Something in the Windows side via bootcamp is modifying the drive in a way the drive doesn't like. Copy protection? Drive utilities?

You said it's in a lab. What are the students running? Are you monitoring or locking down what can be executed that could be doing it?

You've said this seems to be random, no two machines having this happen in a row. This would lead me to suspect that either a student or group of students are causing it or there is a random power issue in the lab causing it. Is there a way of tracking who last used the machines to see if this issue seems to magically follow one of your users?

share|improve this answer
    
We have not been using any power monitoring equipment, no. Simply going off the report from the teachers at the high school on that one. edit GAH! Note to self: the enter key does not create a new line in comment mode. –  EightQuarterBit Mar 17 '11 at 13:37
    
Might want to try that. There were no indications of such an issue in the classroom, either, but the IT director was an electrical engineering major in college and had them check it. He did the math and found that they were having periodic spikes of thousands of volts... –  Bart Silverstrim Mar 17 '11 at 13:40
    
We've done a RAM test on each machine that has had this problem, they all passed. You'd think that if the drive cables were loose the problem would have manifested itself on the same machine a few times on the same iMac, rather than on multiple iMacs. Also, if it were a loose drive cable it's odd that it hasn't occurred to the Bootcamp partitions. –  EightQuarterBit Mar 17 '11 at 13:56
    
It's possible that the drives are indeed marginal, but I have a hard time buying that one. I've checked, re-checked, and triple checked these drives. I've used Apple's Disk Utility as well as Drive Genius 3 to scan the snot out of the disks. Also, I've never seen a marginal drive without having at least a few indicators in SMART. I've seen "bad" hard disks that passed SMART, but they still had high relocated sector counts and whatnot when using a SMART utility. These drives? Nada. –  EightQuarterBit Mar 17 '11 at 14:01
    
I've thought of Windows screwing with the Mac partition, and I still haven't ruled it out. However, the fact that Apple's bootcamp driver will ONLY mount HFS+ partitions as read-only seems to preclude that. We aren't using any disk tools or "deep freeze" type utilities on the Windows side. –  EightQuarterBit Mar 17 '11 at 14:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.