5

Okay, quite simply I have a shell script that needs to wait for something to happen, but it has a lock-file and some child-processes that I need to ensure are tidied up if the script is interrupted.

I've achieved this without issue by using the trap command to set some appropriate actions, and have come up with a script that looks a bit like this:

#!/bin/sh
LOG="$0.log"

# Create a lock-file to prevent simultaneous access
lockfile -l 86400 "$LOG.lock" || $(echo 'Locking failed' >&2 && exit 3)

# Create trap for interrupt and cleanup
on_complete() {
    echo $(date +%R)' Ended.' >> "$LOG"
    kill $(jobs -p)
    rm -f "$LOG.lock"
    exit
}
trap 'on_complete 2> /dev/null' SIGTERM SIGINT SIGHUP EXIT

# Do nothing
echo $(date +%R)' Running…' >> "$LOG"
sleep 86400 &
while wait; do sleep 86400 &; done

This can be run just fine in a terminal via sh Example.sh, and terminating it with Ctrl + C, causing it to remove its lock-file without any fuss.

I then tried creating a launchd job for this script like so:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.example</string>
    <key>ProgramArguments</key>
    <array>
        <string>sh</string>
        <string>~/Downloads/Example.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>EnableGlobbing</key>
    <true/>
</dict>
</plist>

Creating Example.sh and Example.plist from the above in the ~/Downloads folder allows me to run the launchd job via launchd load ~/Downloads/Example.plist and end it via launchd unload ~/Downloads/Example.plist. However, ending the job doesn't cause a SIGTERM to reach the script, which is instead SIGKILL'd after the 20 second timeout.

So what I'd like to know is; why isn't my script receiving SIGTERM, and how I can ensure it does?

2
  • 1
    Does this behaviour continue if you remove 'sh' and call the script directly? I am assuming the script has its executable flag set. Mar 31, 2014 at 9:46
  • 1
    Rather than 'launchd unload', have you tried 'launchctl stop ~/Downloads/Example.plist'?
    – audiomason
    Mar 31, 2014 at 20:39

3 Answers 3

6
+25

The ultimate problem here is that Bash does not normally kill its non-builtin children.

If bash is waiting for a command to complete and receives a signal for which a
trap has been set, the trap will not be executed until the command completes.
When bash is waiting for an asynchronous command  via  the  wait  builtin, the
reception of a signal for which a trap has been set will cause the wait builtin
to return immediately with an exit status greater than 128, immediately after
which  the trap is executed.

When you hit <CTRL>+<C> you're killing the shell script, which behaves normally -- but the sleep lives on. Use ps to see.

When try to stop things externally, via kill, then Bash as above. After some time-out period (I'm guessing 20 seconds) launchd then issues a kill -9 which the script cannot trap.

The solution is to issue a wait after the sleep, to indicate to Bash that it can interrupt itself:

sleep 86400 & wait

This will allow the script to be interrupted, but the sleep will still survive. I'm sure there's a way to kill the children, but I didn't bother looking it up...

2
  • Thanks for the explanation, but use of wait doesn't help (it's what I'm doing in the actual script I'm trying to debug, I've tweaked the example to match a bit more closely), so I'm not sure what's going on.
    – Haravikk
    Apr 1, 2014 at 10:54
  • 1
    Turn out that as of Yosemite this is now the correct answer; a launch agent or launch daemon that is sleeping (asynchronously) with a suitable trap (must be SIGINT, not INT) will receive the signal prior to being unloaded. Of course this is no good for Mavericks, Mountain Lion etc., but it's great that this finally works as it should, so I'm marking this is the correct answer, but it might be worth editing in that it only works correctly under 10.10.
    – Haravikk
    Oct 19, 2014 at 14:09
4

Realizing you've just shared with us a essentially a code fragment and it's not clear what more you're daemon is looking to actually achieve other than to perform some action every so many seconds. So I'm going to make some assumptions just based on what you've written.

  1. It seems like you're using the lockfile to prevent duplicate launch.
  2. It then seems that you need the trap to clean up the lock file used to implement your test to assure singularity.
  3. Additionally it appears that your deamon is doing a sleep loop to wake periodically and perform some action. (Just sleep more, in your example.)

These are all issues that launchd is meant to resolve in better ways under Darwin (and hence OS X).

As for the question(s) with the unload and SIGTERM, specifically, when you unload your launchdeamon is sent a SIGKILL instead of a SIGTERM. If you just wanted to stop the job or send it a SIGTERM then use stop instead of unload.

If you want a SIGTERM sent on unload you may need to set EnableTransactions. Likewise if you have cleanup tasks and you want your deamon to received signals for cleanup and SIGTERM then you should set EnableTransactions as part of the launchd plist for your script. <key>EnableTransactions</key><true/>. This is described in the docs at https://developer.apple.com/library/mac/documentation/Darwin/Reference/Manpages/man5/launchd.plist.5.html

But the three mechanisms above are unnecessary given launchd...

Under Darwin / OS X using launchdaemons the appropriate method for implementing a sleep loop daemon is to use StartInterval to run on an interval or StartCalendarInterval to run based at specific times. Using StartCalendarInterval additionally gives the advantage that when the system is asleep it will execute a missed interval time instead of having to wait for the next interval, and is generally what you want in these situations. If you have a job you just want to stay invoked, also consider using KeepAlive as part of the plist.

So it looks like -- from the code sample you've provided -- you just want to execute something every 86400 seconds. If this is the case then launchd has a mechanism for doing this that you should be using instead and obviates the need for your lock file and trap altogether as launchd is designed to handle all this for you automagically. That mechanism is StartInterval and when defined it will launch your deamon every N seconds. Launchd also makes sure it hasn't launched multiple copies of your daemon.

This mechanism is described in the launchd docs at https://developer.apple.com/library/mac/documentation/Darwin/Reference/Manpages/man5/launchd.plist.5.html where it states:

StartInterval <integer>
This optional key causes the job to be started every N seconds.  If the system is
asleep, the job will be started the next time the computer wakes up.  If multiple
intervals transpire before the computer is woken, those events will be coalesced 
into one event upon wake from sleep.

So your Darwin-ized script ~/Downloads/Example.sh would look something very simply now like this:

#!/bin/sh
echo $(date +%R)' Running…' # or whatever it is you wanted to do on the interval

And your plist would look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.example</string>
    <key>ProgramArguments</key>
    <array>
        <string>sh</string>
        <string>~/Downloads/Example.sh</string>
    </array>
    <key>EnableGlobbing</key>
    <true/>
    <key>StartInterval</key>
    <integer>86400</integer>
    <key>StandardOutPath</key>
    <string>/mypathtolog/myjob.log</string>
    <key>StandardErrorPath</key>
    <string>/mypathtolog/myjob.log</string>
</dict>
</plist>

Note I've also adjusted this to set the logging files here in a Darwin/launchd like manner rather than in the script itself. (You could of course remove them and handle them in your script but it's not necessary given launchd.)

I'd note that you could also implement this using Program like so:

<key>Program</key>
<string>sh</string>
<key>ProgramArguments</key>
<array>
    <string>~/Downloads/Example.sh</string>
</array>

You may also find http://launchd.info a useful reference as well along with the Apple docs for how launchd operates at https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/Introduction.html

Information about daemons run periodically can be found at https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html#//apple_ref/doc/uid/10000172i-CH1-SW2

6
  • 1
    Sorry, I should have said that actually my script is waiting for logout/shutdown, which is when launchd will send SIGINT to all running agents (or rather, should). EnableTransactions true won't work as shell scripts can't call the vproc commands, and set to false is supposed to be the default. I've tried including it as false to be sure but it doesn't seem to help.
    – Haravikk
    Apr 3, 2014 at 10:30
  • OK, that's a completely different situation than what your script was apparently trying to do, and one in which the solution you've crafted is a complex solution to a trivial non-problem. There's a completely better way to await for logout and shutdown in OS X. Also EnableTransactions works for signals accordingly regardless if the process is actually using vproc or not, that documentation commentary is basically for actual applications, but it's functionally the same regardless. Apr 4, 2014 at 4:18
  • Sorry, that saved before I was finished. Apr 4, 2014 at 4:25
  • You can create a LaunchAgent await for logout or use a logouthook (sudo defaults write com.apple.loginwindow LogoutHook /Users/Shared/logoutHook.sh). Apr 4, 2014 at 4:25
  • Does the logout hook require sudo or can it be done per-user? Correct way to watch for logout or not, I'm still at a loss as to why I'm not getting SIGINT in my script as-is; were you able to reproduce the issue by running the example code I've provided? Neither setting for EnableTransactions seems to make a difference for me.
    – Haravikk
    Apr 7, 2014 at 10:27
0

You should not set EnableTransactions unless you are going to actively call vproc_transaction_begin at startup (which you cannot do directly from a shell script) followed by vproc_transaction_end at shutdown; enabling this setting and then not making the first call marks your script as suitable for sudden termination.

You also want to implement logging, as you are quite constrained in what you can do at launchd shutdown, and you need to see what is failing and why!

Combining these, you get this as /Library/LaunchDaemons/org.example.shutdownhook.plist (in this location, it will be automatically launched on boot):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.example.shutdownhook</string>
    <key>ProgramArguments</key>
    <array>
        <string>sh</string>
        <string>/Library/PrivilegedHelperTools/org.example.shutdownhook.helper</string>
    </array>
    <key>StandardOutPath</key>
    <string>/var/log/org.example.shutdownhook.launchd/launchd.log</string>
    <key>StandardErrorPath</key>
    <string>/var/log/org.example.shutdownhook.launchd/launchd.log</string>
    <key>RunAtLoad</key>
    <true/>
    <key>EnableGlobbing</key>
    <false/>
    <key>EnableTransactions</key>
    <false/>
    <key>KeepAlive</key>
    <true/>
</dict>
</plist>

and this as /Library/PrivilegedHelperTools/org.example.shutdownhook.helper:

#!/bin/sh

dolog() {
  echo "$(date) ${1}"
}

on_complete() {
  dolog "Trap ${1}"

  # do stuff
  # ...

  dolog "I did some stuff."

  dolog "Ended."

  # kill child (i.e. sleep) processes - launchd does this
  # automatically, but useful if running this script directly
  kill $(jobs -p)

  exit
}

# useful for debugging to see what signals you get;
# better to only trap specific signals in production
for s in {1..31} ;do trap "on_complete $s" $s ;done

while true
do
    dolog "Running…"
    sleep $(expr $RANDOM / 3277) & wait
done

The above writes every 0-9 seconds to the log for debugging. For non-debugging use, use sleep $RANDOM & wait (0-32767 seconds delay), and move potentially move dolog "Running…" above the while loop (depending on whether you want periodic pings to the log).

You can use sudo launchctl list | grep -v com.apple to see non-Apple sudo launchd processes including this one, and you can tail -f the log file to see what is going on (e.g. if you kill the process, you can see it restart because of KeepAlive).

REFS:
https://www.unix.com/man-page/osx/5/launchd.plist/
https://developer.apple.com/forums/thread/44221?answerId=622576022#622576022
https://stackoverflow.com/a/61909029/795690
https://apple.stackexchange.com/a/284652/113758

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .