In Progress
Unit 1, Lesson 1
In Progress

Better Tests with Mutant

Hello hackers. Look at your tests. Now back at me. Now back at your tests. Now back to me.

Do your tests comprehensively cover your code’s intended functionality? Sadly, they probably don’t. But they could, if only you had the time to try every conceivable change to your code and see which changes still resulted in passing tests.

Look down. Back up. Where are you? You’re in a canoe, with guest chefs Daniel Gollahon and John Backus. Today, Daniel is going to show you how you can use mutation testing tools to discover holes in your test coverage and learn to write better tests.

You’re welcome.

Video transcript & code

Writing better tests with mutant


Imagine you have a simple FizzBuzz implementation:

module FizzBuzz
  FIZZ_FACTOR      = 3
  BUZZ_FACTOR      = 5
  FIZZ_BUZZ_FACTOR = 15

  def self.upto(n)
    (1..n).each do |i|
      output =
        if (i % FIZZ_BUZZ_FACTOR).zero?
          'FIZZBUZZ'
        elsif (i % BUZZ_FACTOR).zero?
          'BUZZ'
        elsif (i % FIZZ_FACTOR).zero?
          'FIZZ'
        else
          i
        end

      puts output
    end
  end
end

This looks okay, but how do we know we haven't made a mistake somewhere? Generally, we write tests--but how do we know those tests are doing their job?

RSpec.describe FizzBuzz do
  it 'is fully covered' do
    FizzBuzz.upto(500)
  end
end

Here's a test that covers 100% of our code and executes every statement dozens of times. Even though we didn't assert anything, a code-coverage tool like SimpleCov reports full coverage.

module FizzBuzz
  def self.upto(_)
  end
end

I could actually delete the entire method body...

...and my test still passes.

Our tests should fail if we make meaningful changes to our code, but ours didn't. So how can you tell if your tests are trustworthy?

Mutation testing title slide Mutation testing tools like the mutant gem modify or delete parts of your code and then check to see that your tests fail. If your tests still pass after it has mutated your code, you either have code you don't need or tests that you do need but don't have.

Put another way,

Did this line get run by your tests? (bullet point) Line coverage asks: Did this line get run by your tests?

If the meaning of your code changes, will your tests catch it? (bullet point) Mutation coverage asks: If the meaning of your code changes, will your tests catch it?

If we run mutant on my FizzBuzz example, it will output 66 ways it can change the code and have the tests still pass, including deleting the method body, like we tried by hand.

RSpec.describe FizzBuzz do
  it 'is fully covered' do
    expect { FizzBuzz.upto(15) }.to output(<<~OUT).to_stdout
      1
      2
      Fizz
      4
      Buzz
      Fizz
      7
      8
      Fizz
      Buzz
      11
      Fizz
      13
      14
      FizzBuzz
    OUT
  end
end

When we write a test that fully specifies the behavior,

mutant will tell us we've "killed" every mutation.

Let's take a look at some more realistic examples.

class Comment < ApplicationModel
  has_one :author

  def editable_by?(editor)
    case editor.role
    when :admin     then true
    when :moderator then !author.admin?
    when :user
      editor == author && !locked?
    when :guest, :muted then false
    end
  end
end

Imagine we're working on a comment system where users are allowed to edit comments, but they can only edit in certain cases:

RSpec.describe Comment do
  it 'allows admins to edit the comment' do
    expect(Comment.new(author: user).editable_by?(admin)).to be(true)
  end

  it 'allows moderators to edit the comment' do
    expect(Comment.new(author: user).editable_by?(moderator)).to be(true)
  end

  it 'users to edit their own comment' do
    expect(Comment.new(author: user).editable_by?(user)).to be(true)
  end

  it 'does not allow guests to edit' do
    expect(Comment.new(author: user).editable_by?(guest)).to be(false)
  end
end

So we write a few tests to make sure

  • Admins and moderators can edit comments
  • Users can edit their own comments
  • Guests can't edit anything

This is a good start, but before we run our tests, let's look back at the original code again.

class Comment < ApplicationModel
  has_one :author

  def editable_by?(editor)
    case editor.role
    when :admin     then true
    when :moderator then !author.admin?
    when :user
      editor == author && !locked?
    when :guest, :muted then false
    end
  end
end

Ruby is compact and readable, but sometimes we forget how much complexity this can disguise. Try pausing the video and counting how many different conditions this method handles.

Comment code as a slide

I count 8 separate cases we need to consider:

Comment annotation 1 1. Is the editor an admin? Comment annotation 2 2. Is the editor a moderator AND the comment is NOT authored by an admin? Comment annotation 3 3. Is the editor a moderator AND the comment IS authored by an admin? Comment annotation 4 4. Is the editor a normal user AND they authored the comment AND the comment is not locked? Comment annotation 5 5. Is the editor a normal user BUT they are not the author? Comment annotation 6 6. Is the editor a normal user BUT the comment is locked? Comment annotation 7 7. Is the editor a guest? Comment annotation 8 8. Is the editor muted?

Let's see what mutant reports:

Mutant is able to delete parts of the case statement, showing we've missed some of our conditions.

  # ...

  it 'does not allow moderators to edit admin comments' do
    expect(Comment.new(author: admin).editable_by?(moderator)).to be(false)
  end

  it 'does not allow muted users to edit a comment' do
    expect(Comment.new(author: muted).editable_by?(muted)).to be(false)
  end

  it 'does not allow regular users to edit comments they did not author' do
    expect(Comment.new(author: user).editable_by?(another_user)).to be(false)
  end

  it 'does not allow regular users to edit locked comments' do
    expect(Comment.new(author: user, locked: true).editable_by?(user)).to be(false)
  end
end

For full coverage, we also need to test that

  • moderators can't edit admin comments
  • muted users can't edit any comment
  • normal users can't modify a locked comment
  • normal users can't modify another user's comment.

When we run mutant again, it confirms we've covered the relevant behavior.

Let's look at one final example:

class User < ApplicationModel
  def change_password(current_password, new_password)
    return unless correct_password?(current_password)

    send_email(:password_update)
    self.password_digest = BCrypt.create(new_password)
    save!
  end
end

When a user changes their password, we check that they supplied their current password, send them an email, update their password, and save the record. Let's look at our tests:

context 'when the current password is correct' do
  it "updates the user's password" do
    user.change_password('oldpassword', 'newpassword')

    expect(user.password_match?('newpassword')).to be(true)
  end
end

context 'when the current password is incorrect' do
  it "does not update the user's password" do
   user.change_password('wrong!!11!ONE!1', 'new_password')

   expect(user.password_match?('oldpassword')).to be(true)
  end
end

We test the two obvious cases: when the user knows their password and when they don't. But will this satisfy mutant?

Mutant discovers two issues:

1. It can remove send_email, showing we didn't assert the email was sent. 2. It can also remove the save! call. We checked that the password changed in memory, but not that the change was persisted.

In other words, mutant makes sure we test the side effects of our method.

The immediate benefit of mutant is that it improves your test coverage. With time, however, it reveals your blind spots and actually teaches you how to write better tests.

Be sure to give mutation testing a try. Next week we'll go beyond test coverage and discuss how mutant can help you write better code.

Responses