mds processes are constantly crashing

Question

Mojave 10.14.4 here -- mds_worker processes constantly crashing and re-spawning as per the logs. The 'indexing' spotlight bar stops at various different points, always getting stuck, though in different places apparently. I've rebuilt the database many times (using mdutil -i on/off, -a -E, etc., as well as the GUI method). The database is clearly not complete - searching filenames that I can literally see turns up nothing in some cases and finds it in others. Mail isn't returning good search results either.

In-place OS reinstall has been done; didn't clear up the issue.

Apple is now recommending I install Mojave on a separate partition, boot into it and see if it has the same issue - don't see how that's going to help (yet), but I have to do it in order to move the process forward. I'll update with any findings once I've done that.

It seems like it's choking on some piece of data I have, and I need to figure out what that is so I can remove it.

It seems like many md_worker processes get spawned from a parent -- and the process that crashes does so, so quickly that I can't don't have time to do an lsof -p on it.

Is there another way to do this, or to get more information about the file(s) it's choking on from log?

Edit1: Ok, I figured out that I can get significantly more information about what mds* is doing with the following commands. I have yet to run this on the affected machine.

sudo log config --mode "private_data:on"

log stream --style compact --predicate '(process contains[c] "mdworker") && (subsystem == [c] "com.apple.spotlightserver")' --info --debug

The first command turns off obfuscation, otherwise file paths are obscured (and replaced with "<private>") - I suppose to make it safe to share logs. See here: https://superuser.com/questions/1311578/in-console-app-how-can-i-reveal-to-what-private-tags-are-actually-referring

Answer 1

Okay, I think the issue is resolved.

The culprit turned out to be an old piece of software from 2011 - Tags, for Mail (http://www.caseapps.com/tags/).

It installed an mdimporter into /Library/Spotlight/.

After reading up on the new logging system and figuring out how to disable obfuscation in order to not get those <private> entries in the logs since those don't help me see which files are being accessed by the md processes,

I ran log stream --style compact --predicate '(process contains[c] "mdworker") && (subsystem ==[c] "com.apple.spotlightserver")' --info --debug and watched it.

It kept hanging up in a particular directory, in ~/Mail/V6. I went looking through the directory tree looking for permissions/uid issues -- nothing.

I re-ran the command with tee, so that I could tail it and grep out interesting parts to investigate.

Then I noticed that it seemed like _TagsMail.mdimporter was being mentioned in each line where mdworker was being called to import an email.

An excerpt: 2019-05-08 21:36:07.002 Db mdworker[53808:1463e7e] [com.apple.spotlightserver:Server] sandbox profile activated time:0.01667 uid:501 profile:mdworker cache:/var/folders/xb/25w333f48xl1ns0059bd0kxh0000gn/C/ temp:/var/folders/xb/25w333f48xl1ns0059bd0kxh0000gn/T/ home:/Users/B**** lib:/Users/B****/Library M:(null) MV2:(null) typeExtension:0 bundleId:com.gravityapps.TagsMail

I looked up tagsmail.mdimporter and the first web page hit is "_TagsMail.mdimporter prevents Spotlight in Mojave to finish": http://www.mnott.de/_tagsmail-mdimporter-prevents-spotlight-in-mojave-to-finish/

Interesting. I moved the mdimporter plugin out and immediately saw progress being made, rather than the same roughly 10x files being attempted to index continually.

Rebooted the machine, created a test file on the desktop, searched for it and saw it pop up almost immediately. Good sign that the indexer is working properly.