Ever find yourself loading your application in a browser after a deploy just to make sure you didn’t break it? Ever forget to do that and have a panicked coworker call you a few minutes later wondering if you know you took the whole site down?

Thing is, you can have the best automated test suite ever, and you can have a great set of passive monitoring tools, but nothing beats an active check of the application after a deploy. After yet another time of making an insignificant tweak that turned out to work just ever so slightly differently in production in such a way as to take one of our apps down post-deploy, I finally said, “Enough!” and made Duff code us up a solution.

That’s right folks, this one isn’t about tooting my own horn, but rather about tooting the horn of fellow Spreedly developer Duff’s since he doesn’t do it nearly enough. After we fixed my latest gaffe (which thankfully “only” affected the UI), Duff tackled the problem of adding an active sanity check to our deploy processes that would just do a basic check (or checks) and tell us as part of the deployment itself if the check failed.

I love the term smoke test for this check; originally used in plumbing, the smoke test was when they’d force smoke through newly installed pipes to see if any smoke seeped out. It was later adopted by the electronics industry as well: “You plug in a new board and turn on the power. If you see smoke coming from the board, turn off the power. You don’t have to do any more testing.” ¹ The smoke test is that first test you run on a newly assembled system just to make sure it’s not immediately and obviously broken.

So Duff dug in and added a new recipe to our internal deployment gem:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Capistrano::Configuration.instance(:must_exist).load do

  def all_smoke_tests_passed
    send_to_hipchat("All smoke tests were successful.", color: 'green')
  end

  def smoke_tests_failed(messages)
    puts "\n\n******************\n\nSMOKE TEST FAILURE!!!\n#{messages}\n\n******************\n\n"
    send_to_hipchat("http://f.cl.ly/items/2T3c2B2m3H2l3R2Y3o2z/Image%202013.02.15%201:50:04%20PM.jpeg", message_format: 'text', color: 'red')
    messages.each do |message|
      send_to_hipchat(message, color: 'red')
    end
  end

  task :smoke_test do
  end

  after "deploy", "smoke_test"

end

It’s pretty simple, it just hooks in after deploy and runs a (by default empty) task. It also provides convenience methods for taking action upon success or failure. Usage is simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

require 'potluck/recipes/smoke'

task :smoke_test do
  failed_tests = []

  curl_result = `curl -s -w "%{http_code}" https://id.spreedly.com/signin -o /dev/null`
  failed_tests << "Id smoke test FAILURE." unless (curl_result == "200")

  curl_result = `curl -s -w "%{http_code}" https://spreedly.com/ -o /dev/null`
  failed_tests << "Public smoke test FAILURE." unless (curl_result == "200")

  if failed_tests.empty?
    all_smoke_tests_passed
  else
    smoke_tests_failed(failed_tests)
  end
end

All we have to do for a project is override the :smoke_test task, do whatever makes sense for that particular app, and use the convenience methods to report the result. Now if a smoke test fails, the whole team knows right away as it gets blasted into our Hipchat room:

We of course also see if the smoke test passed (though it’s much less “in your face”):

It’s amazing how comforting it is to see that line show up after a deploy.

Of course, there are plenty of other ways we test our apps, both pre- and post-deploy, but the smoke test is a great addition to the overall lineup and I’m grateful to Duff for getting them going for Spreedly. And I’d encourage other developers to consider adding an active check to the deploy process to see if anything starts smoking once everything’s live.

¹ Smoke Testing on Wikipedia back