Is there an easy way to automatically schedule a move of files matching a certain pattern from a user directory to a USB drive?

Question

I'm looking to automate a backup process that involves transporting backup files (with names in the form "*.savf") first onto a Mac Mini (running Yosemite), and then, once they're on the Mini, moving them from where they were initially placed onto external USB drives. This is a process that was formerly running on a very large, very noisy, and very worn-out Linux box.

So far, I have the files arriving on the Mini. But can anybody point me towards an easy way to automate moving them to the external drives?

You want details, here they are:

The two external boxes being backed up (AS/400s, hence the "SAVF" extensions), either every evening at a specified time, or when they are told to shut themselves down, will run a program that checks specified libraries to see if there is anything in them that needs backing up. If a library needs backing up, it will be saved to an AS/400-specific construct called a "save file," and then FTP'd to the backup server (each has its own account on the backup server, with its own home directory).

On the Linux backup server, because I could not work out a way for it to FTP directly to the external drives, there was a shell script, running as a cron job, under root authority, which copied those save files to the external drives, then deleted them from the home directories.

I have the first part working very nicely on the Mac. Not sure how to make the second part work.

5/8/2023: I've just successfully reverse-engineered my own shell script:

#!
cd /Users/foo
cp *.savf /Volumes/BACKUP/foo
mv *.savf /Volumes/BACKUP2/foo

cd /Users/bar
cp *.savf /Volumes/BACKUP/bar
mv *.savf /Volumes/BACKUP2/bar

Now, to make it execute at a specific time of day, under root authority, and ignore all errors . . .

**** May 10th ****

I now have a plist in /Library/LaunchDaemons (put in late yesterday afternoon), BUT IT DIDN'T WORK! It's called "sweep.plist," and it is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>sweep</string>
<key>Program</key>
<string>/Users/europa/sweep.sh</string>
<key>StartCalendarInterval</key>
<dict>
<key>Hour</key>
<integer>8</integer>
<key>Minute</key>
<integer>0</integer>
</dict>
</dict>
</plist>

And I did a launchctl load /Library/LaunchDaemons/sweep.plist (either sudo'd or in a sudo bash) once it was in place. I'm absolutely certain that after I did so yesterday, "sweep" showed up in a launchctl list, but it didn't execute, and it no longer shows up when I do a launchctl list.

Any idea where to look for the problem?

New: I just checked the system log, and found this: May 10 08:00:05 Europas-Mac-mini com.apple.xpc.launchd[1] (sweep[1299]): Program specified by service is not a Mach-O executable file.

Looking at the shell script with a ls -l, it shows permissions as -rwxr-xr-x@

Answer 1

launchd

Apple provide launchd with macOS for managing daemons and other background processes, including scripts:

Script management with launchd in Terminal on Mac

The launchd process is used by macOS to manage daemons and agents, and you can use it to run your shell scripts. You don’t interact with launchd directly; instead you use the launchctl command to load or unload launchd daemons and agents.

cron

cron remains available on macOS and is a valid option for running your script to a schedule. Apple have deprecated cron in favour of launchd. The tool is deprecated because cron is not considered energy efficient.

However, cron works and is reliable.

Third Party

If you are happy to use a third party tool for scheduling your script, many options are available, including:

• Hazel
• Keyboard Maestro
• Power Manager

Power Manager

I am an engineer behind Power Manager, which is a tool that can run your shell script as root.

The following two links show two different approaches – one with an external script, the other with an inline script:

• How to Schedule an External Shell Script
• How to Run a Shell Script Every Day on a Mac

Answer 2

Earlier today, I found something: https://stackoverflow.com/a/39062525/3654526

It seems that once I changed the generic shebang to one specifying a shell (from #! to #!/bin/bash) and changed the time the script's supposed to execute, it worked.

Unfortunately, I also discovered that the spontaneously disconnecting USB drive wasn't just from the box going to sleep: it seems that the drive is starting to flake out, and the box going to sleep was just a trigger. Time to retire the Seagate and get a new drive. Then again, that's why I'm backing up to two separate USB drives.

Answer 3

For something as simple as this, I just stick with the old-fashioned way.

First, since you want it done as root, sudo sh or sudo bash or sudo zsh according to your preference.

Then, crontab -e to edit a crontab with vim.  man 5 crontab tells the format you need.  Basically,

17  4  *  *  *  /usr/local/bin/rsync -av /Users/foo/ /Volumes/BACKUP/foo
23  4  *  *  *  /usr/local/bin/rsync -av /Users/foo/ /Volumes/BACKUP2/foo

will, ar 04:17 and 04:23, move everything in /Users/foo to the other unless it is already there.  The slash on /Users/foo is important. Without it, you'll create /Volumes/BACKUP/foo/foo  In other words,

/usr/local/bin/rsync -av /Users/foo/ /Volumes/BACKUP/foo
and
/usr/local/bin/rsync -av /Users/foo /Volumes/BACKUP

do the exact same thing.  man rsync for details.

launchd has a lot more capabilities than cron, but when I don't need the enhancements, I revert to the simpler “archaic” method.

rsync copies only what is missing or changed, so if you suspect any glitches, you can repeat it an hour later and it won't waste bandwidth repeating what already got copied:

17 4,5 * * * /usr/local/bin/rsync -av /Users/foo/ /Volumes/BACKUP/foo