Skinny daemons

For the past several months, we’ve been having a lot of fun writing a series of small, self-contained web apps – branston, enigmamachine, octopus, and adhd, among others. One of the things I’ve been most enjoying is how easy we’ve got it. Ruby gives us so many libraries that taking an idea for a simple, focused app and turning it into working software is ridiculously easy – thin, eventmachine, sinatra, padrino, rails give us loads of support for whatever weird thing we want to do. It’s a great environment to work in.

When we’re building these kinds of applications, which are often meant as low-ceremony apps targeted at a very specific purpose, or as service utilities, a lot of the time we don’t want to go through the hassle associated with a “normal” web app. Passenger and Capistrano make deployment really easy, but nothing can beat
gem install fooapp
and then
fooapp start
Essentially, all we want to do is daemonize a process with an integrated HTTP server – using Apache and Passenger as a daemonization container here would be big-time overkill.

Likewise, there are already a lot of great options for asynchronous processing in Ruby. There are lots of ways to spin off a new worker thread if your webapp needs to do something that’s going to take a while – backgroundrb, dj, bj, starling, workling, daemons – and in general they work pretty well. One thing about them, though, is that they attempt to couple the resulting service implementation very tightly into your Ruby webapp. There might be occasions when you don’t want that, or when you want to run a whole bank of service machines, or when you want to make your services accept input from things other than your Rails app, or…well, maybe you just want to expose a service via HTTP rather than via Drb, because it’s a more generic way to do things.

So, let’s say you’d like to fire up a persistent RESTful web service. It shouldn’t be part of any other app, and you want it to be able to accept HTTP POST requests for service control. Maybe you need a skinny daemon – a daemonized Ruby process running inside a thin webserver.

The basic idea here is to use the thin webserver as a container for whatever app or service you want to run inside it. The whole thing can then be packaged as a rubygem, and you end up with an easily installable service which can be used by any programmer who can send an HTTP request – not just Rubyists.

Let’s install jeweler and generate an example gem:

gem install jeweler
jeweler skinny_daemon_example

Note: I am keenly aware of all the current controversies surrounding gems, gemspecs, etc. The point of this post is to demonstrate the creation of a skinny daemon, not to take a position in the Gem Wars.

This gives us a skeletal gem structure to play with:

- lib/
  - skinny_daemon_example.rb
- test/
  - helper.rb
  - test_skinny_daemon_example.rb
LICENSE
Rakefile
README.rdoc

In the gem root folder, add a “bin” folder with a file called “skinny_daemon_example”, then paste this code into the new file:

# bin/skinny_daemon_example

#!/usr/bin/env ruby

# Skinny daemon command line interface script.
# Run skinny_daemon_example -h to get more usage.
require File.dirname(__FILE__) + '/../lib/skinny_daemon_example'
require 'thin'

rackup_file = "#{File.dirname(__FILE__)}/../lib/skinny_daemon_example/config.ru"

argv = ARGV
argv << ["-R", rackup_file] unless ARGV.include?("-R")
argv << ["-p", "2003"] unless ARGV.include?("-p")
argv << ["-e", "production"] unless ARGV.include?("-e")
Thin::Runner.new(argv.flatten).run!

This is the startup script for our gem - and when you install the gem, it'll be installed on your system's PATH so you can run it to start our new app on the command line.

This thing is going to run as a rack application, so let's add a rackup file. Create a new folder in your gem's "lib" folder, and call it, once again, "skinny_daemon_example". Your file structure should now look like this:

The rackup file should contain this code:

# lib/skinny_daemon_example/config.ru:

require File.dirname(__FILE__) + '/../skinny_daemon_example'
SkinnyDaemonExample.run! :port => 2003

Lastly, you can set up the basis of a RESTful web service. I find that the easiest way to do this is by using Sinatra, which gives us all the HTTP goodness we might need. It also provides the ability for us to build in any handy status or configuration screens we might want.

# lib/skinny_daemon_example.rb

require 'rubygems'
require 'sinatra/base'

class SkinnyDaemonExample < Sinatra::Base

  # This can display a nice status message.
  #
  get "/" do
    "Your skinny daemon is up and running."
  end

  # This POST allows your other apps to control the service.
  #
  post "/do-something/:great" do
    # something great could happen here
  end  

end

Let's build and install our gem. Add a gem.summary and gem.description in the Rakefile, and commit all the files to your git repo so that jeweler picks them up. Then go for it with the installation:

rake version:write
rake build
sudo rake install

That builds your gem. Once it's installed, you'll have a new command installed on your system. You should be able to type:

skinny_daemon_example start

and get a web service fired up and running on http://localhost:2003/.

If you wanted to run it as its own standalone process, rather than in a terminal you had to keep open, you could daemonize it like this:

skinny_daemon_example start -d

To stop it, you can issue:

skinny_daemon_example stop

from the same directory you started it in.

It might not seem like much, but think about what you've got now, in only a few lines of code:

• A RESTful web service which can take POST requests
• A daemon process which you can start and stop using its own command
• A miniature web site which you can use to display status messages or do configuration
• A packaged application which can be easily installed by anyone who's got Ruby

Because thin is based on eventmachine, you've also got all the event-driven programming power of eventmachine at your fingertips, opening up the possibilities of easy multi-threaded programming, highly concurrent applications using the reactor pattern, and things like timers and queueing systems. This is what we've done for enigmamachine - it's basically a skinny daemon which applies long-running ffmpeg commands to video files as a web service. Enigmamachine is a more detailed and advanced example of a skinny daemon, it might be worth checking out the sources and README if you're interested in learning more. At the very least, you'll get a feel for bundling sqlite so you've got a datastore, and starting an eventmachine thread pool to go multi-threaded.

If you'd like to see just the skinny_daemon_example code, take a look at the github repo.