1

I have a service configured via launchd to automatically restart when it crashes. Unfortunately it will sometimes hang, and stop working, without actually crashing. I can detect when this has happens because it stops responding to HTTP on a specific port.

What's the best way to automatically restart the service when this happens?

I've thought about creating a separate launchd agent whose only purpose is to periodically poll the HTTP port and kill the service when it stops responding. But I feel like there must be a better way to do it, perhaps built into launchd?

0

2 Answers 2

1

Heartbeat / Healthcheck

The act of polling a URL to monitor a service is often referred to as checking for a heartbeat. Popular tools like docker promote this polling approach, see docker's healthcheck configuration entry for an example:

The HEALTHCHECK instruction tells Docker how to test a container to check that it's still working. This can detect cases such as a web server stuck in an infinite loop and unable to handle new connections, even though the server process is still running.

You could replicate this approach with a cronjob and shell script to restart your HTTP service. See health check of web page using curl for a related question.

1

launchd can’t detect if a launched process gets stuck, so you need to handle this yourself:

  • Debug/fix the code to avoid the hangs,
  • Redesign the daemon to detect the situation itself (e.g. by having launchd start a wrapper which both starts the actual daemon and periodically checks for hangs,
  • Have a separate job to periodically check for hangs.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .