In Progress
Unit 1, Lesson 21
In Progress

Re-Run Just the Tests That Failed

A test-driven development workflow is only effective when it gives you quick, meaningful feedback. One of the obstacles to this kind of feedback is waiting for lots of slow tests that have nothing to do with the problem at hand. In this video, you’ll learn about power-user facilities for both RSpec and MiniTest that help you skip the noise and focus in on the tests that are currently failing.

Video transcript & code

Today is a big day! It's our first day working on this project, and we're excited! We've followed all the project setup directions, and we're about to run the test suite for the first time!

OK, here we go...

$ rspec --format doc 

a sloth
  gets out of bed slowly
  blinks slowly
  makes coffee slowly (FAILED - 1)
  reads the newspaper slowly

Oh boy. Oh man. This is gonna take a while isn't it.

Time to go make some coffee.

Right, let's use the magic of video fast-forward to the end here...

OK, it looks like we've got some failed tests to address. We know these are passing in CI, so we probably just have some configuration or dependencies missing from our development environment. But what's our workflow going to be to address these?

Well we know one thing we don't want to do: we don't want to re-run that entire test suite each time we try something!

Ideally, we'd try a fix and then re-run only the failing tests. How do we do that? Well, we could take note of which tests failed last time and manually tell RSpec to run those. But that's going to be tedious.

Fortunately, we have another option! We can run rspec with the --only-failures flag! This flag does exactly what we want: it just re-runs the examples which failed last time. Running only these tests will shorten our feedback loop considerably!

$ rspec --format doc --only-failures
Run options: include {:last_run_status=>"failed"}

a sloth
  makes coffee slowly (FAILED - 1)
  orders food slowly (FAILED - 2)
  waves to neighbors slowly (FAILED - 3)
  writes a shopping list slowly (FAILED - 4)

Failures:

  1) a sloth makes coffee slowly
     Failure/Error: expect(false).to be_truthy

       expected: truthy value
            got: false
     # ./spec/sloth_spec.rb:13:in `block (2 levels) in <top (required)>'

  2) a sloth orders food slowly
     Failure/Error: expect(false).to be_truthy

       expected: truthy value
            got: false
     # ./spec/sloth_spec.rb:37:in `block (2 levels) in <top (required)>'

  3) a sloth waves to neighbors slowly
     Failure/Error: expect(false).to be_truthy

       expected: truthy value
            got: false
     # ./spec/sloth_spec.rb:57:in `block (2 levels) in <top (required)>'

  4) a sloth writes a shopping list slowly
     Failure/Error: expect(false).to be_truthy

       expected: truthy value
            got: false
     # ./spec/sloth_spec.rb:81:in `block (2 levels) in <top (required)>'

Finished in 0.14792 seconds (files took 0.46399 seconds to load)
4 examples, 4 failures

Failed examples:

rspec ./spec/sloth_spec.rb:12 # a sloth makes coffee slowly
rspec ./spec/sloth_spec.rb:36 # a sloth orders food slowly
rspec ./spec/sloth_spec.rb:56 # a sloth waves to neighbors slowly
rspec ./spec/sloth_spec.rb:80 # a sloth writes a shopping list slowly

If we want to tighten it up even more, we can run rspec with the --next-failure flag. This runs the tests that failed last time, and stops at the first failure this time around.

$ rspec --format doc --next-failure
Run options: include {:last_run_status=>"failed"}

a sloth
  makes coffee slowly (FAILED - 1)

Failures:

  1) a sloth makes coffee slowly
     Failure/Error: expect(false).to be_truthy

       expected: truthy value
            got: false
     # ./spec/sloth_spec.rb:13:in `block (2 levels) in <top (required)>'

Finished in 0.11254 seconds (files took 0.50506 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/sloth_spec.rb:12 # a sloth makes coffee slowly

The result is a very focused and efficient workflow for getting to the bottom of failed tests, one problem at a time.

You might be wondering how this works under the hood.

The answer is: RSpec keeps a record of the last test run in a file, including whether the examples succeeded or failed, and how long they took.

$ head examples.txt 
example_id                 | status | run_time        |
-------------------------- | ------ | --------------- |
./spec/sloth_spec.rb[1:1]  | passed | 3.03 seconds    |
./spec/sloth_spec.rb[1:2]  | passed | 3 seconds       |
./spec/sloth_spec.rb[1:3]  | failed | 0.11051 seconds |
./spec/sloth_spec.rb[1:4]  | passed | 3 seconds       |
./spec/sloth_spec.rb[1:5]  | passed | 3 seconds       |
./spec/sloth_spec.rb[1:6]  | passed | 3 seconds       |
./spec/sloth_spec.rb[1:7]  | passed | 3 seconds       |
./spec/sloth_spec.rb[1:8]  | passed | 3 seconds       |

Now, be aware that this is a feature that needs to be configured to turn it on. If we take a look at our spec_helper.rb, we can see the that the key example_status_persistence_file_path is being set. However, in a Rails project you may not need to configure it.

RSpec.configure do |c|
  c.example_status_persistence_file_path = "examples.txt"
end

The default configuration created by the rspec-rails integration includes this key out of the box,

# The settings below are suggested to provide a good initial experience
# with RSpec, but feel free to customize to your heart's content.
=begin
  ...
  # Allows RSpec to persist some state between runs in order to support
  # the `--only-failures` and `--next-failure` CLI options. We recommend
  # you configure your source control system to ignore this file.
  config.example_status_persistence_file_path = "spec/examples.txt"
  ...
=end

although you may need to uncomment the relevant section.

So this is all great for people who are already using RSpec for test.

But what if you're running the Ruby and Rails default testing library, MiniTest?

By design, MiniTest is a lot more stripped-down, and doesn't provide as many test-runner conveniences as RSpec. Can we still re-run just failed tests?

Well, we can't do it out of the box, and the only projects I found to extend MiniTest with this feature is defunct. But! All hope is not lost.

There's this gem called mrspec, by Josh Cheek. It which enables us to use the rspec test runner to run a MiniTest suite!

We just have to substitute mrspec for the rspec command, and voila, the tests are being run as if they were Rspec examples!

This gives us access to all the features of the rspec runner, including the --only-failures flag.

$ mrspec --format doc --only-failures
Run options: include {:last_run_status=>"failed"}

Sloth
  gets out of bed slowly (FAILED - 1)
  orders food slowly (FAILED - 2)
  waves to neighbors slowly (FAILED - 3)

...

Finished in 0.00677 seconds (files took 1.1 seconds to load)
3 examples, 3 failures

Failed examples:

rspec ./test/test_sloth.rb:5 # Sloth gets out of bed slowly
rspec ./test/test_sloth.rb:45 # Sloth orders food slowly
rspec ./test/test_sloth.rb:70 # Sloth waves to neighbors slowly

One little catch here: we still need to configure Rspec to generate that test state file, and we have to do it using a MiniTest helper rather than using a spec_helper.rb.

Here's how I did it: a test_helper.rb containing a block that only runs Rspec configuration if the Rspec module is loaded.

if defined?(RSpec)
  RSpec.configure do |c|
    c.example_status_persistence_file_path = "examples.txt"
  end
end

One last thing: you should not commit the test state file to your version control system! It is volatile and very specific to your development environment.

So make sure you put it in your '.gitignore' or equivalent file.

And there you have it: a workflow for working through a list of failing tests one at a time, without the delay of waiting for passing tests to complete. (slowly) Happy hacking!

Responses