How to debug the cause of an enormous Spotlight index?

Question

My Spotlight index (MacBook Air 13, 10.13.6) has been growing out of control requiring me to delete and rebuild at least twice per day.

I have about 70 Gb available on my HDD, and the Spotlight database located in /.Spotlight-V100/ slowly increases until it takes up the entirety of the free space.

At that point I delete the index as follows:

#!/bin/bash

sudo mdutil -a -i off
sudo rm -rf /.Spotlight-V100/*
sudo mdutil -a -i on

and allow it to rebuild itself.

This post indicates that others have seen this behavior, but gives no help in debugging, as the poster found an issue with a device driver that I don't have. Spotlight Massive (out of control at 100GB+)

Any tips as to how to debug this issue? I've poked around the Console.app but haven't found a clear pointer to anything going out of control.

One possibility is that there are a large number of mdworker processes created during the rebuild; I don't know if this is unexpected but it seems greater than what others have reported.

At this point I'm out of ideas of what to check. I rely on Spotlight pretty thoroughly for mail search within Outlook so I can't just turn it off entirely. Thanks!

Answer 1

Working under the assumption that it is a specific file, folder or file type that triggers some kind of bug that makes the index size explode, I would take a look at the actual files that Spotlight builds up (i.e. the database itself). Tryk looking at them with any text editor or even a hex-editor.

If this is indeed caused by a bug that caused the same content to be indexed over and over again, then you might be able to see which file or file type it is. Then you can narrow down which folder to exclude from Spotlight.

Another more time consuming method is simply by trial and error doing a kind of "binary search", where you exclude half of your folders from the Spotlight index. If the problem persists, the problem is in this half - otherwise it is in the other. Repeat that process until you've found the specific folder containing the problematic file.

If you're comfortable with the Terminal, you can also take a look at which files are actually opened and read by the mdworker processes. This could again give you a hint as to why the index is so big.

First use Activity Monitor like you have already done to find the mdworker processes. Make note of the "PID" column that contains the process ID.

Open Terminal and run the following command:

lsof -n -p <pid>

You need to replace with the actual process ID. This will give you a list of the files that are currently in use by that process.

You could also run:

fs_usage -p <pid>

Again replace with the actual process ID. This command will show you file accesses for that process as they occur. You can stop the display by pressing Ctrl-C.