2

I try to test GNU version of locate command. First, I have to create the database like this :

sudo gupdatedb --prunepaths=/Volumes --output=$HOME/locatedb_gupdatedb

Unfortunately, 1 minute after launching the command, I still get the following error :

gfind: failed to read file names from file system at or below '/': No such file or directory

I don't understand where this error could come from ?

UPDATE 1: I replaced gfind by find commad and correct path of find which is /usr/bin/. Unfortunately, I get embarassing error messages like this when I launch the gupdatedb command like this :

sudo gupdatedb --prunepaths='/private/tmp /private/var/folders /private/var/tmp */Backups.backupdb /Volumes /System' --output=$HOME/locatedb_gupdatedb

Here the error messages :

find: /System/Volumes/Data/.Spotlight-V100: No such file or directory
find: /System/Volumes/Data/.PKInstallSandboxManager: No such file or directory
find: /System/Volumes/Data/.PKInstallSandboxManager-SystemSoftware: No such file or directory
find: /System/Volumes/Data/.cleverfiles: No such file or directory
find: /System/Volumes/Data/mnt: No such file or directory
find: /System/Volumes/Data/.DocumentRevisions-V100: No such file or directory
etc ...

I tried to modify into /usr/local/Cellar/findutils/4.7.0/libexec/bin/gupdatedb file, the option :

: ${FINDOPTIONS="2 > /dev/null"}

But the issue is this option is set in front of the command find, not at the end, so it is not correct in the following of the file.

Given the fact there a lot of find commands after in the script, I can't add manually each time the 2 > /dev/null terminal option.

Anyone could see how to suppress all these error messages from find command when I launch a gupdatedb command ?

UPDATE 2: I finally managed to create a database with gupdatedb (GNU version of MacOS updatedb command) by doing :

sudo gupdatedb --prunepaths='/private/tmp /private/var/folders /private/var/tmp */Backups.backupdb /System /Volumes' --output=$HOME/locatedb_gupdatedb

The issue now is, when I do a research on a substring of a file or directory, the informations seems to be duplicated in results (sub_string is simply the part of a file or directory name) :

For example, if I do a : glocate -d ~/locatedb_gupdatedb sub_string

Then, I have duplicates results like :

/System/Volumes/Data/Users/fab/sub_string.dat
/Users/fab/sub_string.dat

I don't know how to exclude '/System/Volumes/Data/' from these results : however, I have well specified in --prunepaths option the directory System, why isn't it taken into account in database created by gupdatedb ?

Or maybe I should perform a :

sudo gupdatedb --prunepaths='/private/tmp /private/var/folders /private/var/tmp */Backups.backupdb /System/Volumes/Data /Volumes' --output=$HOME/locatedb_gupdatedb

??

Any help is welcome to exclude this directory

/System/Volumes/Data

from indexing database.

UPDATE 3: Here is an example of quickly generating a database with updatedb on Debian 10 Buster. Few modifications of normal using have been done between the timestamps of the 2 commands updatedb.

enter image description here

So I conclude there is a really a difference betweeen GNU/MacOS and GNU/Linux implementation.

Any explanation is welcome.

1
  • Not sure we are the right place to ask about Linux , it also makes your question rather broad.
    – nohillside
    Jul 30 '20 at 4:42
2
+50

The problem here is that you are running GNU's updatedb with the macOS's find. gupdatedb is a shell script that expects to be running a GNU-compatible find. In particular, it converts --prunepaths to a GNU-compatible basic regular expression.

--prunepaths='/private/tmp /private/var/folders /private/var/tmp */Backups.backupdb /Volumes /System'

converts to

PRUNEREGEX="\(^/private/tmp$\)\|\(^/private/var/folders$\)\|\(^/private/var/tmp$\)\|\(^*/Backups.backupdb$\)\|\(^/System$\)\|\(^/Volumes$\)"

GNU BRE (basic regular expressions) treat \| as an "alternative" operator:

‘foo\|bar’ matches either ‘foo’ or ‘bar’

This is a GNU extension to the BRE syntax, and the macOS find operator does not accept it. So the whole prune string fails to match anything.

The macOS (and POSIX) BRE does not have an alternative operator, so it is not a super easy fix. The macOS updatedb script converts the alternative into explicit -or operators in the find command. You can modify the GNU script to do that. Or get gfind to work.

By the way, GNU's updatedb script builds a completely new database from scratch on every run, just like macOS's.

Your Debian installation is using mlocate which is a completely different implementation than GNU locate, not as widely ported, and not available on macOS AFAIK. And even though mlocate runs quickly on your Debian installation, that does not mean it runs a lot faster than GNU locate. Both run in under a second building the entire database from scratch on my Debian installation when all the disk metadata is in RAM (which is usually the case).

Apple provides mdfind, which uses the incremental database created by mds, which is largely triggered by file system events. This makes it (theoretically) much more efficient than even mlocate, which still has to traverse the entire directory structure looking for changed directories. The problem is that incremental builds accumulate errors. That is why locate, with its full rebuild every time, is still around and preferred by many.

7
  • "By the way, GNU's updatedb script builds a completely new database from scratch on every run, just like macOS's. " : This is not the case with Linux, isn't it ?
    – youpilat13
    Jul 30 '20 at 2:28
  • @youpilat13 I am looking at the code from GNU findutils 4.7.0. It is not specific to macOS, so I expect it is the same on Linux. If you look at the code in general, updatedb just runs a find command to find all the files and then processes the output into a database file. Since find has to scan the entire directory tree to find new files anyway, there is no performance benefit for an incremental update (at least not one worth the effort).
    – Old Pro
    Jul 30 '20 at 2:53
  • @OlfPro It"s weird, on my Debian, I have not this behavior, updatedb updates only just what has been modified between a last of utilization, typically a few hours or minutes. I conclude the implementation is not exactly the same.
    – youpilat13
    Jul 30 '20 at 2:59
  • Do you know a tool that doesn't any to buidd at each time all the hard disk ? mdfind is pretty good but instable (sometimes results appear and sometimes not).
    – youpilat13
    Jul 30 '20 at 3:01
  • @youpilat13 Let me know what version of updatedb you are running on your Debian system. When I run apt install locate on Debian Buster I get the one from GNU findutils 4.6.0 which creates a new $LOCATE_DB.n and if successful, replaces the old database with the new one. AFAIK, mds does incremental updates for mdfind. If mdfind is missing something not excluded on purpose (by privacy settings or turning of indexing of a drive, for example), I recommend using locate :-) It does not try to do incremental updates so is much more reliable, if somewhat out of date.
    – Old Pro
    Jul 30 '20 at 3:13
1

The Apple version is run via /usr/libexec/locate.updatedb which is a shell script. Some additional paths are excluded from the search there. In Mojave these are

: ${PRUNEPATHS:="/private/tmp /private/var/folders /private/var/tmp */Backups.backupdb"} # unwanted directories

PS: I don't have Catalina so you may need to check whether other paths need to be excluded as well.

5
  • Thanks for your quick answer but the issue is about the root directory "/", the message is : gfind: failed to read file names from file system at or below '/' .I tried to do, with your sugeestion : sudo gupdatedb --prunepaths='/private/tmp /private/var/folders /private/var/tmp */Backups.backupdb /Volumes' --output=$HOME/locatedb_gupdatedband I get the error message above. Do you think it's a sudo problem ?
    – youpilat13
    Jul 27 '20 at 10:44
  • 1
    @youpilat13 It's probably more complex than it seems, the shell script runs find and then feeds the result into the locatedb updater. Don't know how the GNU version works but maybe you can just use a copy of the shell script to feed the GNU version.
    – nohillside
    Jul 27 '20 at 11:20
  • Why do you need to use GNU locate when there's already one installed? Jul 28 '20 at 22:48
  • @MarcWilson . Because I expect the GNU version to have the same behavior than on Linux systems, i.e when we have already a database, if we launch another updatedb command after few hours of using, the current database is updated and not erased to build entirely another again. This is a waste of time to rebuild all the database and unfortunately, this is the dafaut behavior of classical updatedb command of MacOS.
    – youpilat13
    Jul 29 '20 at 22:30
  • Documentation showing that this is the behavior of BSD locate? Jul 29 '20 at 23:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .