In Progress
Unit 1, Lesson 21
In Progress

FFI Part 2: Smoke Test One

Video transcript & code

Back in Episode 26, we investigated how to interface with a C library using the FFI gem. We wrote a script that would list audio output devices on the local system using the PulseAudio library for Linux. Now that we have a proof of concept, it's time to turn our attention to building a reusable library that can do more than just list output modules at the command line.

We hesitate, though, as we set out to rewrite our one-off script into a library. Right now we have working code. It's somewhat hacky, dicidedly fragile, and it took a lot of trial and error to get it right, but now it works. We wonder: could we somehow get this script under test as it stands right now? Then we could start to pull parts of it out into a library while always being just one test run away from knowing if we accidentally broke something in the process.

Testing this code is tricky. It integrates tightly with a system service, the PulseAudio subsystem. Not only that, its output is nondeterministic. The audio devices our script lists may vary depending on the machine it us run on, as well as on how that machine happens to be configured. So we can't rely on the output our script will produce. But we also can't just fake out the parts of our code that talk to the C library, because that would defeat the purpose of checking that everything still works.

avdi@hazel:~/dev/pulse-ffi$ ruby list-sources.rb
0 alsa_output.pci-0000_00_1b.0.analog-stereo.monitor Monitor of Built-in Audio Analog Stereo
1 alsa_input.pci-0000_00_1b.0.analog-stereo Built-in Audio Analog Stereo

After giving it some thought, we decide to create a reliable test by using the PulseAudio command-line interface to insert a known output source into the list of active audio modules. Then we'll search for that known module in our program's output. We'll ignore any other output which might vary based on where the script is run.

We start writing a test using Ruby's built-in MiniTest framework. We call it "SmokeTest". A "smoke test" is a high-level test that simply verifies that nothing is badly broken. It's so named because it's just enough to verify that running the program doesn't cause black smoke to start billowing out.

require 'minitest/autorun'
require 'rake'                  # for FileUtils::RUBY

class SmokeTest < MiniTest::Unit::TestCase

We define a helper method in our test suite which will return a full shell command to run the list-sources.rb script. It uses File.expand_path to locate the script relative to the current file. It also uses a constant called FileUtils::RUBY to locate the path of the current Ruby executable. This constant actually isn't provided by Ruby itself; in order to use it we first have to require the Rake library. It's a bit of a sneaky trick to use a constant that Rake defines for its own internal use, but it gives us assurance that we'll be executing the script under test in the same Ruby environment that the test itself is running in.

require 'rake'

class SmokeTest < MiniTest::Unit::TestCase
  # ...

  def list_sources_command
    script_path= File.expand_path("../../list-sources.rb", __FILE__)
    "#{FileUtils::RUBY} #{script_path}"
  end
end

Then we begin to write the test itself. We start by establishing a baseline. We run the list-sources script and collect its output, and verify that the output does not contain the test source. This ensures that the test won't pass based on an improperly torn-down test fixture from a previous run.

Once we've checked that we're starting with a clean slate, we load our test source into the PulseAudio subsystem using the pactl command, executed in a subshell. The source we add is a simple sine-wave generator which should be available on any system that uses PulseAudio. We give it an easily recognizable name, the word TEST_SOURCE in all caps.

def test_list_sources
  # Check the output before loading test module
  output = `#{list_sources_command}`
  refute_match /TEST_SOURCE Sine source at 440 Hz/, output

Now for the meat of the test. We once again run the list-sources script and collect its output. This time, we verify that the output does contain the TEST_SOURCE.

# Now load a test source module
module_id = 
  `pactl load-module module-sine-source source_name=TEST_SOURCE`.strip
output = `#{list_sources_command}`

# This time the test module should be listed
assert_match /TEST_SOURCE Sine source at 440 Hz/, output

Finally, we clean up after ourselves by executing another pactl command to remove the test source from the PulseAudio server.

ensure
  # Clean up the test module and verify the cleanup succeeded
  `pactl unload-module #{module_id}` if module_id
end

We run the test, and it passes.

You might be wondering why we run the code being tested as a separate process instead of running the code inside the test process. The simplest answer is that our our original proof-of-concept was written in the form of a command-line script, and we want to leave it in its pristine, working state until we have some kind of test over it. But another reason is that running the system under test as a subprocess makes our smoke test a lot more robust:

  • There is little danger that the code will fail when run under test, while still succeeding when run by itself, due to some difference in how it behaves when loaded as a library. If the script works by itself, it will almost certainly still work under the test.
  • And if we need to exercise the code ourselves, to gather information about a failure, all we have to do is run the script and we will see exactly the same output that the test sees.

Remember, this code is here only to verify that our FFI code can at least match the functionality of the original proof-of-concept. Other tests, which we will write later, will test that it behaves well as a reusable library.

Well, here we are at the end of episode 2 of this miniseries, and we still haven't written a library! But we've gained something very important: peace of mind. As we refactor our proof-of-concept into something more general, this smoke test will act as a safety net, always there to ensure that as we reorganize code, we aren't breaking it in the process.

That's all for today. Happy hacking!

Responses