I connect to a remote drive via WebDAV protocol, whereon I have a folder full of about 143 .tar.gz files containing data dumps, whereamong I'm trying to find my email.

Herefor, I use this command:

zgrep -a "my@email.com" /Volumes/storage.address.name/home/folder/Collection\ \#2-\#5\ \&\ Antipublic/*

Once the command is running, I can see my free space on my local MacBook, wherefrom I run the above mentioned command, disappear swiftly, like 0,5 GB per minute. If I let the command run, I would totally run out of space.

Could someone please tell me why is this happening or how to prevent it?

  • 1
    Most likely the files need to be uncompressed locally before they can be searched, so your free disk space goes down because temporary files get written. – nohillside Jul 29 '20 at 12:41
  • @nohillside If it is so, is it somehow possible to make they files get uncompressed remotely? – yssup Jul 29 '20 at 13:01
  • Do you have write access via WebDAV? Can you log into the remote server via ssh or similar? – nohillside Jul 29 '20 at 13:07
  • @nohillside I don't know, maybe so. How do I find out? – yssup Jul 29 '20 at 13:40
  • Copy a file to the WebDAV folder maybe? – nohillside Jul 29 '20 at 14:11

Once the command is running, I can see my free space on my local MacBook...disappear swiftly... Could someone please tell me why is this happening or how to prevent it?

Operating Locally while connected remotely

The command is acting upon those files as if they were local, even when they are not. *nix systems have this wonderful feature where you can mount almost anything as if it were a local file/directory. This "feature" is where you're running into this problem.

To operate on the file, it has to "download" it from the remote, hold it in memory and if necessary, resort to swap space. If it needs to unzip it (bzip, gzip, compress, etc.) it will likely utilize a temporary file (i.e. mktemp). Since your command is operating locally, all the resources it will utilize will be local as well - it doesn't know to use your remote for temp space.

You can do a simple experiment to show how this works - move a file from one remote directory to another. Using a simple mount on a NAS (/Volumes/NAS/) if you try to move /Volumes/NAS/folder1/* to /Volumes/NAS/folder2/, this simple operation will need to "download" the files to local machine, then upload them to the destination folder. However, if you login directly to the server and issue the move command there - for example, mv /home/username/folder1/* /home/username/folder2, it will happen almost instantly.

In your case, it has to extract a number of files from each of those .tar.gz files, download them, store them in a temp location, then load it to memory for as the ASCII text is searched for.

Cleaning up Temp files...

An important consideration to keep in mind is that temp files are not cleaned up until the process has completed or terminated abnormally; if the developer was good about it. If not, those temp files will be cleared out upon reboot.

If you have 200GB of files to search and a MacBook Air with 128GB or 256GB of storage, you will run out of space very quickly filling it up with nothing but temp files.

How to prevent it

Do you operation on the server (remote machine) itself. Instead of mounting the remote file/directory locally and operating on it locally, connect via SSH (for example) and issue your command on the remote itself.

Can you specify a different location for the temp directory like the remote?

Technically yes. For example, gzip uses /tmp and you can set your environment variable to a remote temporary directory (i.e. /Volumes/NAS/tmp). The problem is, you're still operating on the file locally, so it would "download" the file, and then while extracting them, "upload" all the extracted bits to the remote.

  • Wow, thank you for an elaborate answer, mister! – yssup Jul 30 '20 at 9:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .