November 22nd, 2012

How to make an installation stay up 4evr (Mac OS X version)

The most up to date version of this article will be kept on Github now: https://github.com/laserpilot/Installation_Up_4evr

 

NEW: Part 2 of How to Keep an installation up Forever – how to do process logging and emailing yourself on crashes

UPDATED: Added launchd boot option and some other system preference tips

(if you’re looking for help with this task with Windows, check out this awesome script StayUp from Stephen Schieberl. And check out this great step by step from EVSC: http://www.evsc.net/home/prep-windows-machine-for-fulltime-exhibition-setup ).

At work I recently had to set up a four installations of different configurations that would need to run all day, every day, 24 hours a day for a couple months with as few crashes or glitches as possible and without anyone going to check on them. This is something that a lot of media artists need to do all the time, and there are a bunch of different tricks and tips to keeping things up for an extended period, I figured I’d share my findings. There are alternate ways to do many of these tasks and this is only one road so please share some tips you’ve picked up out in the field down in the comments box below.

I had to do several searches in a couple different places to find all the information I needed to keep everything consistently up and bug free. Luckily most of the installations I was dealing with this time were fairly light in terms of resources and complications, but it’s always best practices to have a safety net.

I usually run these off brand new, unboxed computers so this is sort of starting from scratch. Most of the things I point out are set to the opposite by default.

Tip: if you’re doing multiple computers, do these prep steps on one of them and just boot the others into target disk mode and use something like Carbon Copy Cloner to mirror the first one on the next so everything is as consistent as possible.


Step 1: Prep your software and the computer

When building your software or whatever it might be, always keep the long running installation in mind. Plan which things will need to be adjusted by whoever is watching over the installation from the beginning (or at least don’t save it for the end). In my experience, keep it as simple as possible, so that it’s easy for the caretaker to get in there to fix or adjust what they need without opening Xcode and compiling or even exiting out of your app. Time you spend now to make things simple will save you hours of remote debugging when something breaks.

You’ll need to go through and turn off or disable several different automatic settings to keep things from popping up over top of your application. This can differ depending on whether you’re running 10.6, 10.7, 10.8 etc etc.

In System Preferences:

•••Desktop and Screensaver: Disable Screensaver. Set it’s time to “Never”

•••Energy Saver: Turn Display Sleep and Computer Sleep to Never. Enable “Start up automatically after power failure” and “Restart automatically if the computer freezes” (these are only available in 10.7 and later…not exactly sure how reliable they are, have not used them myself yet)

•••Accounts: ->Login Options: Enable Automatic Login

•••Software update: Disable automatic updates.

•••Sharing: (optional if you’re concerned about security). If you are running these without a monitor or in an inaccessible area, don’t forget to turn on file sharing and screen sharing so that you can just plug an Ethernet cable into the install computer and your laptop and be able to use screen sharing to control the computer without a mouse and keyboard but with low latency.

•••Network: If you don’t need remote access or don’t need Internet access for the installation, it’s not a bad idea to disable the wifi so the “please select a wireless network” window doesn’t pop up when you least expect it.

•••Bluetooth :If running without a mouse or keyboard plugged in, sometimes you can get the annoying  “Bluetooth keyboard/mouse setup” pop up over your application. You can temporality disable these by going to the advanced settings within the Bluetooth Preferences. See below for it’s location in 10.6.

•••Security: If you’re really paranoid, you can even disable things like the IR remote receiver that still exists on some macs and definitely on Macbooks. This would keep pranksters with Apple TV remotes from “Front Rowing” your installation. To disable, go to your Security->General tab (Security->General->Advanced (at the bottom) on 10.8) and “Disable remote control IR receiver”.

 

(Click the image for the full size version)


Step 2: Boot into your software

Things get unplugged, power goes out, not everyone has budget or space for a battery backup etc etc. Above, I covered how to have everything reboot automatically after power failures or freezes, but you’ll also need your app to be ready to go from the start and not leave the desktop open to prying eyes. In the System Preferences “Accounts” panel, select “Login Items” and drag your application into there to have it open automatically on launch and strip out whatever you don’t need.

(Click the image for the full size version)


Step 3: Keep it up (champ!)

There are several ways to make sure your application goes up and stays up

•••Launchd

Alternate method that is a little simpler using a Launch Agent plist to have things be launched automatically on boot and continuously if they go down. Launchd plists are very useful alternatives to cron jobs and can be used to run things on a periodic basis or on calendar days. You could achieve similar results with a combination of automator and iCal, but it depends on what you’re comfortable with.

Here is an Apple Doc on using Launch Agents and Launch Daemons in various ways.

The difference between a Launch Daemon and a Launch Agent (Basically whether you need it to run when a user is logged in or not…for most simple options like launching a regular app, you’ll just want a Launch Agent)

A launchd example from admsynhttps://gist.github.com/4140204

(Tip from James George below in comments): Of course you could make the launchd plist yourself for free from a template like above. One quick method to setting up Launchd is to use Lingon ($4.99 in the App Store)

In Lingon, hit the + to create a new launchd plist. Just make it a standard launch agent. Now Set up your plist like so:

One additional/optional thing you can add to this is to put an additional key in the plist for a “Successful Exit”. By adding this, your app won’t re-open when it has detected that it closed normally (ie You just hit escape intentionally, it didn’t crash). Can be useful if you’re trying to check something and OS X won’t stop re-opening the app on you. To easily add this to the key, just hit “expert mode” on the bottom of the Lingon window after selecting your newly made script on the left. Then modify the relevant bits highlighted in the screenshot:

•••Shell script+Cron Job method

(I got the following super helpful tip from Kyle McDonald)

In this step, shell scripting is your best friend. With the help of the script below and an application called CronniX , you will be able to use a cronjob to check the system’s list of currently running processes. If your app does not appear on the list, then the script will open it again, otherwise it won’t do anything. Either download the script or type the following into a text editor, replacing Twitter.app with your app’s name and filepath. Don’t forget the “.app” extension in the if statement!:

#!/bin/sh
if [ $(ps ax | grep -v grep | grep "Twitter.app" | wc -l) -eq 0 ]
then
       echo "Twitter not running. opening..."
       open /Applications/Twitter.app
else 
	echo "Twitter running"
fi

Save that file as something like “KeepOpen.sh” and keep it next to your application or somewhere convenient. Download the script here: Keep Open Script (372 downloads)

After creating that file, you’ll need to make it executable. To do this, open the Terminal and in a new window type “chmod +x ” and then enter the path to the shell script you just created (you can either drag the shell script into the terminal window or manually type it). It would look something like this:

Laser-MacBook-Pro:~ laser$ chmod +x /Users/laser/Desktop/KeepOpen.sh

After you have made it executable, you’re now ready to set it up as a cronjob. Tip: to test the script, you can change the extension at the end to KeepOpen.command as an alternative to opening it with Terminal, but the same thing gets done.

Cronjobs are just low level system tasks that are set to run on a timer. The syntax for cronjobs is outside of the scope of this walkthrough, but there are many sites available for that. Instead, the application CronniX can do a lot of the heavy lifting for you.

After downloading CronniX, open it up and create a new cronjob. In the window that opens,  in the command window, point it to your KeepOpen.sh file and  check all of the boxes in the simple tab for minute, hour, month, etc. This tells the job to run every minute, every hour, every day, every month. If you want it to run less frequently or at a different frequency, play around with the sliders.

Now just hit “New” and then make sure to hit “Save” to save it into the system’s crontab. Now if you just wait a minute then it should open your app every minute on the minute. Maybe save this one for the very end if you have more to do :)

This is a great tool if there is an unintended crash because the app will never be down longer than a minute.

•••Non Cronjob Shell method

From comment below by Kris Meeusen:

#!/bin/bash

while true
do
#using open to get focus
echo "Trying to open empty example"
open -a emptyExample
sleep 10
done

Just type this into a plaintext document and save it as something like “KeepMyAppAlivePlz.command” and then use chmod as above to make the file executable  and then drop this in your login items as  above. This one will just continuously try and open your app every 10ms, but if it is already open, the OS knows to not try opening it a second, third, fourth time.

Make sure to check the Console.app for any errors that may have come through when no one caught them, whenever you check the installation in person or remotely. This is not a fix-all for buggy programming, just a helper to keep things running smooth. The more things you can do to leave yourself notes about why the crash happened, the faster you can address the core issue.


Step 4: Reboot periodically

This one is a little more preventative, or maybe superstitious so hopefully someone can point out a concrete reason why this is a good idea. Depending on your app and the amount of stuff it reaches into, there could be some memory leaks or other OS bugs that you haven’t accounted for. Rebooting every day or week is a good idea to keep everything tidy, system wise.

The simplest option by far would be to go to System Preferences->Energy Saver and then click “Schedule…” and enter in some values if you need to turn the computer off to rest for a longer period of time to save it some stress when it might not be used at night time or something. Heat can do funny things sometimes, so if you have a chance to get your computer to rest and the time to test it, definitely give this a shot…saves some energy too which is nice.

You could also set up another shell script with a crontab as above with CronniX that reboots the system with as often as you specify.

Another option (if you don’t want to deal with the terminal and shell scripting) is to use iCal to call an Automator iCal event. This method is perhaps a little easier to schedule and visualize when you will reboot. Within Automator, create a new file with the iCal event template to do something like this:

(Click the image for the full size version)

Run it to see if it does what you expect and then save it out. When you save,it will open in iCal as an action that gets opened. Just set it to repeat as often as you’d like.

If you’d like to just close your programs and re-open them and there is a background and foreground do something like this (pauses are so the quitting and re-opening stuff has time to actually execute):

(Click the image for the full size version)


Step 5: Check in on it from afar. 

There are a bunch of options here from various paid web services (like Logmein or Teamviewer), to VNC (many options for this RealVNC tends to come up a bunch) to SSHing. The choice here depends on your comfort level and how much you need to do to perform maintenance from far away.

Leaving a Dropbox connected to the computer and your own is super useful for file swaps between the two computers. Although most remote screensharing services have file sharing built in, Dropbox is just a nice, fairly transparent option.


Step 6: Test, test, test. 

You’ve already tested and perfected your software for the installation, so make sure to test all of the above methods and automatic scripts in as realistic manner as you can before you leave it alone for the first day at school.

You can’t account for everything, so don’t beat yourself up if something does eventually happen, but this list will hopefully alleviate a little bit of frustration. Good luck!

 

Other resources people have sent me:

http://vormplus.be/blog/article/configuring-mac-os-x-for-interactive-installations

http://www.evsc.net/home/prep-windows-machine-for-fulltime-exhibition-setup

  • Ryan Jenkins

    IN power saver, you can also set a time to have the computer cycle on and off, so you can restart your installation computer and get back to your automated boot setting without having to do any ical stuff /anything. It is all built into the OS, at least in 10.6

    • laserpilot

      Ah yes…i forgot about this too…the scheduler in energy saver is much simpler…oh well..at least the automator example is there in case they need to do something more complicated

      • Julien-Robert Legault Salvail

        I used that script to keep the installation opened. The problem is when using the scheduler in energy saver it tries to shut down but the opened terminal asks me if I want to stop the script before the computer shuts down… then the script re-opens the application and the computer never shuts down… do you have any workaround for that?

        #!/bin/bash

        while true
        do
        #using open to get focus
        open -a emptyExample
        sleep 10
        done

  • Jason Van Cleave

    Nice write up! another good tool is TinkerTool http://www.bresink.com/osx/TinkerTool.html

    I use it for a lot of UI stuff but it has easy access to turn off the Crash reporter so if the app crashes you don’t end up with the infamous crash dialog stuck on the screen trying to notify Apple

    • laserpilot

      Ah great tip..thanks! Forgot about that one..

  • Kris Meeusen

    I use a simpler bash script without a cron check.

    #!/bin/bash

    while true
    do
    #using open to get focus
    open -a emptyExample
    sleep 10
    done

    And on Windows

    :begin

    start /WAIT emptyExample.exe

    timeout /T 10

    Goto begin

    • laserpilot

      Ah awesome..thats much simpler. I guess “my” version maybe gives you a better idea of if/when it crashed whereas this would just keep opening it ad infinitum but not using cron definitely makes things simpler

  • laserpilot

    I will add this as an addendum soon but on twitter this was just posted by @admsyn as an alternative to bash/shell: https://gist.github.com/4140204

  • http://www.memo.tv Memo Akten

    Great list, thanks! A few extras:

    – The ‘no keyboard’ or ‘no mouse’ warning can actually be fixed from System Preferences. It’s under the ‘Bluetooth’ panel, click the ‘Advanced’ button and then there are two tickboxes, one for ‘Open if no keyboard found’, and one for ‘Open if no mouse/trackpad found’. (So you don’t need to rename the applications).

    – If you’re not using wifi, make sure to disable Airport. Otherwise you’re likely to get a system popup if no known wifi network is found. Or if you are using wifi, disable the option to ‘Ask to join new networks’ (under Network settings). So if your wifi network goes down for whatever reason, you don’t get the OSX Wifi popup appearing over your app (instead your app should handle it somehow).

    • laserpilot

      Ah ha…i knew I was forgetting something with the bluetooth one, thanks for that…I updated it above. Good suggestion for the network one as well

  • http://www.facebook.com/profile.php?id=10713449 James George

    This is awesome, Blair! Very useful info — and this comment thread is bound to become a priceless smattering of other tricks-of-the-trade.

    One thing i’ve found useful is this app called Lingon, which is a GUI for managing launchd

    http://osxdaily.com/2007/03/12/lingon-gui-configuration-of-launchd/

    You can get it off the app store.

    Also, launchd has some nice special configurations for restarting apps if they crash but not if they exit intentionally, using the KeepAlive + SuccessfulExit, a la: http://hints.macworld.com/article.php?story=20071220095645567

  • laserpilot

    For a leaky application, check out this script that emails you if an app starts using an “excessive” amount of resources: http://hintsforums.macworld.com/showthread.php?p=592991

  • Pingback: Prep Windows machine for fulltime exhibition setup | evsc

  • http://robray.net Rob Ray

    I have a penchant for taking photos of “field glitches.” This is one from a big show earlier this fall. Growl is not your art friend.

    • http://robray.net Rob Ray

      image blurred to protect the innocent.

  • Graham Thorne

    Brilliant thank you.

    I have a few systmes running in a museum that run for 9 hours a day 7 days a week for 99% of the year. I had to make my own automator script to make sure software was open. I made a few posts here, might be of use to some.

    http://vjskulpture.wordpress.com/tag/museum-2/

  • Pingback: Blair Neal | Applescript to automatically fullscreen Madmapper for installations

  • Pingback: » Computers at an ExhibitionUCLA ARTS S0FTWARE STUDI0

  • Pingback: Blair Neal | How to keep an installation up forever – Part 2

  • Pingback: How to Build an Indie Arcade Cabinet - Studio Mercato

  • Pingback: How to Build an Indie Arcade Cabinet – Gamasutra (blog) | Top Kitchen Shops

  • Elie Zananiri

    Super useful post, thanks for this! We just used it for our last installation. One thing I would add is to disable the crash reporter dialog when your app crashes (it happens): http://osxdaily.com/2010/02/15/disable-the-crash-reporter-dialog-in-mac-os-x/