I want to run a script every 24 hours, that scrapes a website and fetches data, save it to my local MySQL database, and push its DB to my remote Heroku or IBM Bluemix MySQL server (or ClearDB). I did this set of procedures in my Mac Pro in the past using launchd, but since I now have MacBook Pro only, I cannot ensure that this is always done properly, because I occasionally close the laptop.

I know how to set the launchd to execute the script once the laptop wakes up, but I still must make sure that this script is executed in the correct time, because the remote database must be set up to be updated every day to ensure that users in my iOS app always see the updated datasets.

How can I set it up so that the script is always executed in the exactly same time every day, such as 23:15, even in the laptop is being sleep? I have never shut down the laptop; I just close the tab and go into sleep.

  • I'd stick something like that on a digitalocean droplet it's faster for one and for two you never have to worry about it not being able to run. I don't know how much scraping you're doing but for a fairly complex daily data collection task we live comfortably on the $5 plan (unless your scraping gigs a day or video or something the $5 plan goes far) – Tyson Apr 11 '16 at 3:10
  • @Tyson The dataset consists of only a few MB and I want to push it to my remote MySQL (or ClearDB) database. I will check it out, thanks. – Blaszard Apr 11 '16 at 7:19

The only thing I can think of that will wake a Mac, do maintenance, and put it immediately back to sleep is power nap. But PowerNap is not available to 3rd party developers. https://www.quora.com/OS-X-Mountain-Lion-10-8-Can-third-party-apps-take-advantage-of-Power-Nap

Why not just scrape from the server?

  • Right, I know about Power Nap but it is so sad that this is not available to devs. As to the server, it is not mine. – Blaszard Apr 11 '16 at 2:18
  • Well, it seems that the scheduler can be used in Heroku via Add-On, but not in Bluemix. – Blaszard Apr 11 '16 at 8:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .