In Progress
Unit 1, Lesson 21
In Progress

Test Substitute

When testing code that makes use of third-party libraries, we can mock out the dependencies for fast but incomplete tests. Or we can do full integration tests which are slow and potentially flaky. These are our two options, right?

Well, maybe not. In today’s episode, guest chef Nathan Ladd makes a case for a third way: test substitutes that are maintained alongside a production API. He’ll show you an example of how to implement such a substitute, inspired by real-world experience. Enjoy!

Video transcript & code


class Checkout
  attr_reader :order
  attr_reader :payment_source

  def initialize(order, payment_source)
    @order, @payment_source = order, payment_source
  end

  def call(errors=nil)
    errors ||= []

    amount = order.total_amount

    customer_name = payment_source.customer_name
    card_number = payment_source.card_number
    expiration_date = payment_source.expiration_date
    cvv = payment_source.cvv

    response = ChargeCustomer.(customer_name, amount, card_number, expiration_date, cvv)

    if response.success?
      true
    else
      errors << response.message
      false
    end
  end
end

Often times when learning to write tests, it is common to feel forced to make a difficult choice.

On one hand, exercising a broad constellation of objects that depend on the thing we're testing.


RSpec.describe Checkout do
  let(:checkout) { Checkout.new(order, payment_source) }

  let(:order) { Controls::Order.example }
  let(:payment_source) { Controls::PaymentSource.example }

  before do
    expect(checkout.charge_customer).to receive(:call).with(
      payment_source.customer_name,
      order.total_amount,
      payment_source.card_number,
      payment_source.expiration_date,
      payment_source.cvv
    ).and_return(response)
  end

  context "Approval" do
    let(:response) do
      OpenStruct.new({
        :success? => true
      })
    end

    specify "Returns true" do
      return_value = checkout.()

      expect(return_value).to be true
    end
  end

  context "Decline" do
    let(:response) do
      OpenStruct.new({
        :success? => false,
        :message => decline_message
      })
    end

    let(:decline_message) { "Card was declined" }

    specify "Returns false" do
      return_value = checkout.()

      expect(return_value).to be false
    end

    specify "Appends decline message" do
      errors = []

      checkout.(errors)

      expect(errors).to eq [decline_message]
    end
  end
end

On the other, using a mocking library to impose isolation between our test subject and its dependencies.

When we exercise the test subject's full dependency graph, we may appear to gain more formidable test coverage,...

screenshot

... but it often comes at the expense of extremely slow execution, and additional brittleness, particularly when IO bound dependencies like third party HTTP APIs are unavailable.

For instance, this ChargeCustomer dependency is posting a payment to TrustCommerce via the ActiveMerchant gem. Let's run the test with the network offline and see what happens.

screenshot

Ooof!

... ...

On the bright side, if you're getting charged money for each of those web API calls, at least you're saving some when you're running your test suite on an airplane.

hand of architect altering drawing

There's also a deeper software design issue revealed by tests that inadvertently trigger distant nodes in our object graphs -- if we can't exercise our objects in isolation, they can't really be understood on their own, can they? Indeed, they can't even be considered designs at all...

photo of automobile engine

... think of real world examples of designs. For instance, can a car manufacturer exercise, say, a controller chip independently of having to house it within a fully finished car?

technician working on circuit board

Of course they can!


require_relative './spec_helper'

RSpec.describe Checkout do
  let(:checkout) { Checkout.new(order, payment_source) }

  let(:order) { Controls::Order.example }

  context "Approval" do
    let(:payment_source) { Controls::PaymentSource::Approve.example }

    specify "Returns true" do
      return_value = checkout.()

      expect(return_value).to be true
    end
  end

  context "Decline" do
    let(:payment_source) { Controls::PaymentSource::Decline.example }

    specify "Returns false" do
      return_value = checkout.()

      expect(return_value).to be false
    end
  end
end

Here's our end to end test that caused our full object graph to be exercised when executed. It's quite short!


RSpec.describe Checkout do
  let(:checkout) { Checkout.new(order, payment_source) }

  let(:order) { Controls::Order.example }
  let(:payment_source) { Controls::PaymentSource.example }

  before do
    expect(checkout.charge_customer).to receive(:call).with(
      payment_source.customer_name,
      order.total_amount,
      payment_source.card_number,
      payment_source.expiration_date,
      payment_source.cvv
    ).and_return(response)
  end

  context "Approval" do
    let(:response) do
      OpenStruct.new({
        :success? => true
      })
    end

    specify "Returns true" do
      return_value = checkout.()

      expect(return_value).to be true
    end
  end

  context "Decline" do
    let(:response) do
      OpenStruct.new({
        :success? => false,
        :message => decline_message
      })
    end

    let(:decline_message) { "Card was declined" }

    specify "Returns false" do
      return_value = checkout.()

      expect(return_value).to be false
    end

    specify "Appends decline message" do
      errors = []

      checkout.(errors)

      expect(errors).to eq [decline_message]
    end
  end
end

Let's take another look at that RSpec test that uses mocks to eliminate the calls out to the ChargeCustomer dependency.

As you can see, the test has grown quite a bit more verbose and it's had to couple to implementation details of the ChargeCustomer class, like the name of the method (call) and its rather lengthy parameter signature.

screenshot

On the bright side, when we run the tests, they certainly execute much faster. But faster tests are only valuable when they don't come at the expense of other qualities.

So besides the details of the ChargeCustomer class leaking into this test of an adjacent object, there are a few other issues I'm going to raise.

First, mocks cause your tests to depend on a mocking library. This may seem like small potatoes, but if you've ever tried changing test frameworks or mocking libraries, you've experienced how substantial this can be.


class ChargeCustomer
  def call(customer_name, amount, card_number, expiration_date, cvv)
    if amount.is_a?(Float)
      amount *= 100
      amount = amount.round
    end

    # ...
  end
end

Second, and more significant, when the ChargeCustomer interface is changed, our mocks don't necessarily cause our tests to fail!

screenshot

Now, some mocking libraries can raise errors if the stubbed methods aren't on the interface being mocked, but regardless, if the user who is running the Checkout tests did not introduce the change to the ChargeCustomer interface, they may not be able to immediately recognize whether the fault lies in the test's mocks or in the dependency.

In other words, mocks in tests do not change congruently with corresponding changes to the underlying interface. And the Single Responsibility Principle dictates that what changes together stays together.

screenshot

So. Is there a third way?


class ChargeCustomer
  class Substitute
  end
end

Instead of actually charging the customer, or mocking the ChargeCustomer dependency, let's build something called a substitute. In this case, we will write a substitute for the ChargeCustomer dependency. It will live inside ChargeCustomer's namespace, but it will be a separate class.

It's going to comport to the same interface as ChargeCustomer, so it will accept five arguments ...

  • customer name
  • amount to be charged
  • card number
  • expiration date
  • three or four-digit CVV code

screenshot

That's a lot of arguments, but after all the credit card fraud that's happened in the last two decades, the days of accepting just the card number and expiration date are long gone.

Since the operational ChargeCustomer class returns an ActiveMerchant response object, which can have a lot of unrelated data attached to it, for the sake of this example, we're going to return an ...

screenshot

... OpenStruct that just has the fields we need, success and message.

Our substitute needs to be able to approve or decline charges (just like its operational counterpart). So, we'll add an ...

screenshot

... extra method to activate a mode where the substitute declines all charges instead of approving them. Note that even though this additional activation method is not present on the operational dependency, this does not break substitutability, so long as our substitute's interface is a strict superset of our operational counterpart.


class ChargeCustomer
  class Substitute
    attr_accessor :decline_message

    def call(customer_name, amount, card_number, expiration_date, cvv)
      success = decline_message.nil?

      OpenStruct.new(
        :success? => success,
        :message => decline_message
      )
    end

    def decline!(message=nil)
      message ||= ''

      self.decline_message = message
    end
  end
end

In other words, every method on our "real" class needs to be accounted for in our substitute ...

... but not necessarily the other way around.


RSpec.describe Checkout do
  let(:checkout) { Checkout.new(order, payment_source) }

  let(:order) { Controls::Order.example }
  let(:payment_source) { Controls::PaymentSource.example }

  before do
    expect(checkout.charge_customer).to receive(:call).with(
      payment_source.customer_name,
      order.total_amount,
      payment_source.card_number,
      payment_source.expiration_date,
      payment_source.cvv
    ).and_return(response)
  end

  context "Approval" do
    let(:response) do
      OpenStruct.new({
        :success? => true
      })
    end

    specify "Returns true" do
      return_value = checkout.()

      expect(return_value).to be true
    end
  end

  context "Decline" do
    let(:response) do
      OpenStruct.new({
        :success? => false,
        :message => decline_message
      })
    end

    let(:decline_message) { "Card was declined" }

    specify "Returns false" do
      return_value = checkout.()

      expect(return_value).to be false
    end

    specify "Appends decline message" do
      errors = []

      checkout.(errors)

      expect(errors).to eq [decline_message]
    end
  end
end

Now, back in our test.

We can replace all the code we brought in with the use of the mocking library by passing in a substitute instead.


RSpec.describe Checkout do
  let(:checkout) do
    checkout = Checkout.new(order, payment_source)
    checkout.charge_customer = charge_customer
    checkout
  end

  let(:charge_customer) { ChargeCustomer::Substitute.new }

  let(:order) { Controls::Order.example }
  let(:payment_source) { Controls::PaymentSource.example }

  context "Approval" do
    specify "Returns true" do
      return_value = checkout.()

      expect(return_value).to be true
    end
  end

  context "Decline" do
    before do
      charge_customer.decline!(decline_message)
    end

    let(:decline_message) { "Card was declined" }

    specify "Returns false" do
      return_value = checkout.()

      expect(return_value).to be false
    end

    specify "Appends decline message" do
      errors = []

      checkout.(errors)

      expect(errors).to eq [decline_message]
    end
  end
end

Our tests still run quite fast, but we no longer depend on mocks! However, an astute observer will notice that our test no longer verifies that Checkout supplies the correct values for the customer name, amount, and card info. To solve that problem, let's revisit our substitute.


class ChargeCustomer
  class Substitute
    attr_accessor :decline_message

    def call(customer_name, amount, card_number, expiration_date, cvv)
      success = decline_message.nil?

      OpenStruct.new(
        :success? => success,
        :message => decline_message
      )
    end

    def decline!(message=nil)
      message ||= ''

      self.decline_message = message
    end
  end
end

In order to verify that our dependency is invoked correctly, we're going to record telemetry in the substitute, and use it as the basis for a query (or, more specifically, predicate) method that determines if the invocation was correct.

First, we'll define a Charge data structure, using ruby's Struct class, which will contain the customer name, amount, and card info for each charge.

Then, inside the actuator (the call method), we'll record a Charge instance.

Finally, we add a predicate method, charged?, which accepts any number of optional arguments that correspond to the required arguments of the call method. If no arguments are given, the predicate method returns true if and only if any charge has been made at all. If any arguments are given, a recorded Charge must additionally have fields that match those arguments for the predicate to return true.


class ChargeCustomer
  class Substitute
    attr_accessor :decline_message

    def charges
      @charges ||= []
    end

    def call(customer_name, amount, card_number, expiration_date, cvv)
      success = decline_message.nil?

      charge = Charge.new(customer_name, amount, card_number, expiration_date, cvv)
      charges << charge OpenStruct.new( :success? => success,
        :message => decline_message
      )
    end

    def decline!(message=nil)
      message ||= ''

      self.decline_message = message
    end

    def charged?(customer_name: nil, amount: nil, card_number: nil, expiration_date: nil, cvv: nil)
      values = {}

      values[:customer_name] = customer_name unless customer_name.nil?
      values[:amount] = amount unless amount.nil?
      values[:card_number] = card_number unless card_number.nil?
      values[:expiration_date] = expiration_date unless expiration_date.nil?
      values[:cvv] = cvv unless cvv.nil?

      charges.any? do |charge|
        values.all? do |key, value|
          charge[key] == value
        end
      end
    end

    Charge = Struct.new(
      :customer_name,
      :amount,
      :card_number,
      :expiration_date,
      :cvv
    )
  end
end

Armed with our new predicate method, we can add a test that ensures the ChargeCustomer interface is being actuated correctly.


RSpec.describe Checkout do
  let(:checkout) do
    checkout = Checkout.new(order, payment_source)
    checkout.charge_customer = charge_customer
    checkout
  end

  let(:charge_customer) { ChargeCustomer::Substitute.new }

  let(:order) { Controls::Order.example }
  let(:payment_source) { Controls::PaymentSource.example }

  specify "Charges customer" do
    checkout.()

    expect(charge_customer).to be_charged(
      customer_name: payment_source.name,
      amount: order.total_amount,
      card_number: payment_source.card_number,
      expiration_date: payment_source.expiration_date,
      cvv: payment_source.cvv
    )
  end
  
  # etc
end

So far, it might seem like we've merely re-implemented by hand what our mocking libraries already do for us. Let's revisit the test file.


RSpec.describe Checkout do
  let(:checkout) do
    checkout = Checkout.new(order, payment_source)
    checkout.charge_customer = charge_customer
    checkout
  end

  let(:charge_customer) { ChargeCustomer::Substitute.new }

  let(:order) { Controls::Order.example }
  let(:payment_source) { Controls::PaymentSource.example }

  specify "Charges customer" do
    checkout.()

    expect(charge_customer).to be_charged(
      customer_name: payment_source.name,
      amount: order.total_amount,
      card_number: payment_source.card_number,
      expiration_date: payment_source.expiration_date,
      cvv: payment_source.cvv
    )
  end

  context "Approval" do
    specify "Returns true" do
      return_value = checkout.()

      expect(return_value).to be true
    end
  end

  context "Decline" do
    before do
      charge_customer.decline!(decline_message)
    end

    let(:decline_message) { "Card was declined" }

    specify "Returns false" do
      return_value = checkout.()

      expect(return_value).to be false
    end

    specify "Appends decline message" do
      errors = []

      checkout.(errors)

      expect(errors).to eq [decline_message]
    end
  end
end

The test file now leverages a substitute, which is implemented within the namespace of the ChargeCustomer dependency.

This means that if and when our ChargeCustomer dependency's interface is changed, our substitute's interface will change with it, and there will be no ambiguity about the cause of any test failures.

screenshot

Also, any test of any object that depends on ChargeCustomer will be substantially less entangled with our test tooling. Our approach to testing is now to control and observe our test subject solely through its own interface.

We've perhaps increased the amount of work required to implement ChargeCustomer marginally, but we've also greatly decreased the amount of work needed to leverage it elsewhere. Since almost all the code we write ends up being used elsewhere, this goes in the books as a win.

Responses