How to make an installation stay up 4evr (Mac OS X version)

22 Nov How to make an installation stay up 4evr (Mac OS X version)

Posted at 14:17h in Examples, openframeworks by laser 22 Comments

THIS ARTICLE IS OUT OF DATE – The most up to date version of this article for OSX 10.10 and beyond will be kept on Github now – feel free to add things or comment there!: 

https://github.com/laserpilot/Installation_Up_4evr

 

It is also part of the fabulous ofBook – a guide to openFrameworks, C++ and other useful creative code tips

 

 

NEW: Part 2 of How to Keep an installation up Forever – how to do process logging and emailing yourself on crashes

UPDATED: Added launchd boot option and some other system preference tips

(if you’re looking for help with this task with Windows, check out this awesome script StayUp from Stephen Schieberl. And check out this great step by step from EVSC: http://www.evsc.net/home/prep-windows-machine-for-fulltime-exhibition-setup ).

At work I recently had to set up a four installations of different configurations that would need to run all day, every day, 24 hours a day for a couple months with as few crashes or glitches as possible and without anyone going to check on them. This is something that a lot of media artists need to do all the time, and there are a bunch of different tricks and tips to keeping things up for an extended period, I figured I’d share my findings. There are alternate ways to do many of these tasks and this is only one road so please share some tips you’ve picked up out in the field down in the comments box below.

I had to do several searches in a couple different places to find all the information I needed to keep everything consistently up and bug free. Luckily most of the installations I was dealing with this time were fairly light in terms of resources and complications, but it’s always best practices to have a safety net.

I usually run these off brand new, unboxed computers so this is sort of starting from scratch. Most of the things I point out are set to the opposite by default.

Tip: if you’re doing multiple computers, do these prep steps on one of them and just boot the others into target disk mode and use something like Carbon Copy Cloner to mirror the first one on the next so everything is as consistent as possible.

Step 1: Prep your software and the computer

When building your software or whatever it might be, always keep the long running installation in mind. Plan which things will need to be adjusted by whoever is watching over the installation from the beginning (or at least don’t save it for the end). In my experience, keep it as simple as possible, so that it’s easy for the caretaker to get in there to fix or adjust what they need without opening Xcode and compiling or even exiting out of your app. Time you spend now to make things simple will save you hours of remote debugging when something breaks.

You’ll need to go through and turn off or disable several different automatic settings to keep things from popping up over top of your application. This can differ depending on whether you’re running 10.6, 10.7, 10.8 etc etc.

In System Preferences:

•••Desktop and Screensaver: Disable Screensaver. Set it’s time to “Never”

•••Energy Saver: Turn Display Sleep and Computer Sleep to Never. Enable “Start up automatically after power failure” and “Restart automatically if the computer freezes” (these are only available in 10.7 and later…not exactly sure how reliable they are, have not used them myself yet)

•••Accounts: ->Login Options: Enable Automatic Login

•••Software update: Disable automatic updates.

•••Sharing: (optional if you’re concerned about security). If you are running these without a monitor or in an inaccessible area, don’t forget to turn on file sharing and screen sharing so that you can just plug an Ethernet cable into the install computer and your laptop and be able to use screen sharing to control the computer without a mouse and keyboard but with low latency.

•••Network: If you don’t need remote access or don’t need Internet access for the installation, it’s not a bad idea to disable the wifi so the “please select a wireless network” window doesn’t pop up when you least expect it.

•••Bluetooth :If running without a mouse or keyboard plugged in, sometimes you can get the annoying  “Bluetooth keyboard/mouse setup” pop up over your application. You can temporality disable these by going to the advanced settings within the Bluetooth Preferences. See below for it’s location in 10.6.

•••Security: If you’re really paranoid, you can even disable things like the IR remote receiver that still exists on some macs and definitely on Macbooks. This would keep pranksters with Apple TV remotes from “Front Rowing” your installation. To disable, go to your Security->General tab (Security->General->Advanced (at the bottom) on 10.8) and “Disable remote control IR receiver”.

 

(Click the image for the full size version)

Step 2: Boot into your software

Things get unplugged, power goes out, not everyone has budget or space for a battery backup etc etc. Above, I covered how to have everything reboot automatically after power failures or freezes, but you’ll also need your app to be ready to go from the start and not leave the desktop open to prying eyes. In the System Preferences “Accounts” panel, select “Login Items” and drag your application into there to have it open automatically on launch and strip out whatever you don’t need.

(Click the image for the full size version)

Step 3: Keep it up (champ!)

There are several ways to make sure your application goes up and stays up

•••Launchd

Alternate method that is a little simpler using a Launch Agent plist to have things be launched automatically on boot and continuously if they go down. Launchd plists are very useful alternatives to cron jobs and can be used to run things on a periodic basis or on calendar days. You could achieve similar results with a combination of automator and iCal, but it depends on what you’re comfortable with.

Here is an Apple Doc on using Launch Agents and Launch Daemons in various ways.

The difference between a Launch Daemon and a Launch Agent (Basically whether you need it to run when a user is logged in or not…for most simple options like launching a regular app, you’ll just want a Launch Agent)

A launchd example from admsyn: https://gist.github.com/4140204

(Tip from James George below in comments): Of course you could make the launchd plist yourself for free from a template like above. One quick method to setting up Launchd is to use Lingon ($4.99 in the App Store)

In Lingon, hit the + to create a new launchd plist. Just make it a standard launch agent. Now Set up your plist like so:

One additional/optional thing you can add to this is to put an additional key in the plist for a “Successful Exit”. By adding this, your app won’t re-open when it has detected that it closed normally (ie You just hit escape intentionally, it didn’t crash). Can be useful if you’re trying to check something and OS X won’t stop re-opening the app on you. To easily add this to the key, just hit “expert mode” on the bottom of the Lingon window after selecting your newly made script on the left. Then modify the relevant bits highlighted in the screenshot:

•••Shell script+Cron Job method

(I got the following super helpful tip from Kyle McDonald)

In this step, shell scripting is your best friend. With the help of the script below and an application called CronniX , you will be able to use a cronjob to check the system’s list of currently running processes. If your app does not appear on the list, then the script will open it again, otherwise it won’t do anything. Either download the script or type the following into a text editor, replacing Twitter.app with your app’s name and filepath. Don’t forget the “.app” extension in the if statement!:

#!/bin/sh
if [ $(ps ax | grep -v grep | grep "Twitter.app" | wc -l) -eq 0 ]
then
       echo "Twitter not running. opening..."
       open /Applications/Twitter.app
else 
	echo "Twitter running"
fi

Save that file as something like “KeepOpen.sh” and keep it next to your application or somewhere convenient. Download the script here: [Download not found]

After creating that file, you’ll need to make it executable. To do this, open the Terminal and in a new window type “chmod +x ” and then enter the path to the shell script you just created (you can either drag the shell script into the terminal window or manually type it). It would look something like this:

Laser-MacBook-Pro:~ laser$ chmod +x /Users/laser/Desktop/KeepOpen.sh

After you have made it executable, you’re now ready to set it up as a cronjob. Tip: to test the script, you can change the extension at the end to KeepOpen.command as an alternative to opening it with Terminal, but the same thing gets done.

Cronjobs are just low level system tasks that are set to run on a timer. The syntax for cronjobs is outside of the scope of this walkthrough, but there are many sites available for that. Instead, the application CronniX can do a lot of the heavy lifting for you.

After downloading CronniX, open it up and create a new cronjob. In the window that opens,  in the command window, point it to your KeepOpen.sh file and  check all of the boxes in the simple tab for minute, hour, month, etc. This tells the job to run every minute, every hour, every day, every month. If you want it to run less frequently or at a different frequency, play around with the sliders.

Now just hit “New” and then make sure to hit “Save” to save it into the system’s crontab. Now if you just wait a minute then it should open your app every minute on the minute. Maybe save this one for the very end if you have more to do 🙂

This is a great tool if there is an unintended crash because the app will never be down longer than a minute.

•••Non Cronjob Shell method

From comment below by Kris Meeusen:

#!/bin/bash

while true
do
#using open to get focus
echo "Trying to open empty example"
open -a emptyExample
sleep 10
done

Just type this into a plaintext document and save it as something like “KeepMyAppAlivePlz.command” and then use chmod as above to make the file executable  and then drop this in your login items as  above. This one will just continuously try and open your app every 10ms, but if it is already open, the OS knows to not try opening it a second, third, fourth time.

Make sure to check the Console.app for any errors that may have come through when no one caught them, whenever you check the installation in person or remotely. This is not a fix-all for buggy programming, just a helper to keep things running smooth. The more things you can do to leave yourself notes about why the crash happened, the faster you can address the core issue.

Step 4: Reboot periodically

This one is a little more preventative, or maybe superstitious so hopefully someone can point out a concrete reason why this is a good idea. Depending on your app and the amount of stuff it reaches into, there could be some memory leaks or other OS bugs that you haven’t accounted for. Rebooting every day or week is a good idea to keep everything tidy, system wise.

The simplest option by far would be to go to System Preferences->Energy Saver and then click “Schedule…” and enter in some values if you need to turn the computer off to rest for a longer period of time to save it some stress when it might not be used at night time or something. Heat can do funny things sometimes, so if you have a chance to get your computer to rest and the time to test it, definitely give this a shot…saves some energy too which is nice.

You could also set up another shell script with a crontab as above with CronniX that reboots the system with as often as you specify.

Another option (if you don’t want to deal with the terminal and shell scripting) is to use iCal to call an Automator iCal event. This method is perhaps a little easier to schedule and visualize when you will reboot. Within Automator, create a new file with the iCal event template to do something like this:

(Click the image for the full size version)

Run it to see if it does what you expect and then save it out. When you save,it will open in iCal as an action that gets opened. Just set it to repeat as often as you’d like.

If you’d like to just close your programs and re-open them and there is a background and foreground do something like this (pauses are so the quitting and re-opening stuff has time to actually execute):

(Click the image for the full size version)

Step 5: Check in on it from afar. 

There are a bunch of options here from various paid web services (like Logmein or Teamviewer), to VNC (many options for this RealVNC tends to come up a bunch) to SSHing. The choice here depends on your comfort level and how much you need to do to perform maintenance from far away.

Leaving a Dropbox connected to the computer and your own is super useful for file swaps between the two computers. Although most remote screensharing services have file sharing built in, Dropbox is just a nice, fairly transparent option.

Step 6: Test, test, test. 

You’ve already tested and perfected your software for the installation, so make sure to test all of the above methods and automatic scripts in as realistic manner as you can before you leave it alone for the first day at school.

You can’t account for everything, so don’t beat yourself up if something does eventually happen, but this list will hopefully alleviate a little bit of frustration. Good luck!

 

Other resources people have sent me:

http://vormplus.be/blog/article/configuring-mac-os-x-for-interactive-installations

http://www.evsc.net/home/prep-windows-machine-for-fulltime-exhibition-setup