In Progress
Unit 1, Lesson 1
In Progress

Testing Retry

Video transcript & code

Back in episode #257, we talked about Ruby's retry keyword, which makes it easy to construct workflows where an unreliable operation is retried until it is successful.

For instance, take the common example of a Client class whose job is to encapsulate interactions with a web service.

require "net/http"

class Client
  def make_request
    Net::HTTP.get_response("www.rubytapas.com", "/").code
  end
end

If we apply the lessons of episode #257 to this code, it comes out looking something like this. There's a new keyword argument for the number of tries, and a new rescue clause that retries the HTTP request so long as the number of tries has not been exhausted.

require "net/http"

class Client
  def make_request(tries: 3)
    Net::HTTP.get_response("www.rubytapas.com", "/").code
  rescue
    tries -= 1
    retry if tries > 0
    raise
  end
end

One viewer of that episode asked a very reasonable question: how would we test this method?

This is an excellent question. But it's also a slightly dangerous one if we don't think critically about it. A large percentage of problems I see programmers running into with unit testing result from asking the question: "how would we test this method?".

Instead, I want to turn the question on its head. Today, let's answer the question: "How would we add limited retries to client requests, using test-driven development?"

Let's get started.

As we look at our original method, and contemplate adding the ability to retry requests, a thought strikes us: retrying requests sounds a lot like a separate responsibility from actually making the requests.

require "net/http"

class Client
  def make_request
    Net::HTTP.get_response("www.rubytapas.com", "/").code
  end
end

Let us assume that the functionality of the make_request method is already thoroughly covered by tests. We have confidence that, so long as the server on the other end is responding correctly, this method makes the right kinds of requests.

If we take that part as read, we can start immediately describing the behavior of a RetryingClient that adds request retries into the mix.

Our first test for this new class specifies that it can forward requests on to an original-flavor client. We set up a test double for the original client, which we call inner because our RetryingClient object will act as a decorator that wraps around the original client. It will respond to a single message, #make_request, and return a readily-identifiable string.

We instantiate a RetryingClient object that we can test, feeding it our stand-in inner client.

Then we assert that when we send the make_request message to RetryingClient, and all goes well, we get back the return value from the inner client.

This fails, because we have no RetryingClient class yet. When we fix this failure, we get another failure because the new class doesn't have an initializer that can take an inner client as an argument.

Instead of writing this method, we simply make our new class extend a DelegateClass customized for the Client class. We'll probably talk about this in more detail another time, but the upshot is that when we run the test again, it passes. Inheriting from a delegater class gives us an initializer which accepts an inner object, and it arranges for the #make_request method to be forwarded along to the inner object.

require "./client"
require "rspec/autorun"
require "delegate"

class RetryingClient < DelegateClass(Client)
end

RSpec.describe RetryingClient do
  it "delegates requests to an inner client object" do
    inner = double(make_request: "THE_RESULT")
    client = RetryingClient.new(inner)
    expect(client.make_request).to eq("THE_RESULT")
  end
end

# >> .
# >>
# >> Finished in 0.00078 seconds (files took 0.07492 seconds to load)
# >> 1 example, 0 failures

Now that we have message pass-through working, it's time to start adding retry capabilities. Our next example will show that a RetryingClient retries a failed request.

To show this, we once again declare the inner object as a test double, and feed that inner object into our RetryingClient object under test.

Now we need to simulate a failure of the request. The obvious way to do this is to have the inner object generate exceptions every time it is messaged.

But we can foresee a problem with this. A common failure mode of code that performs retries is that it is over-eager and retries indefinitely, never returning. We don't mind getting test failures while we are developing this class. But we prefer to have tests actually fail and show us an error message. We don't like it when tests simply hang infinitely.

To head off that possibility, we define a count of times that our fake request will fail. Then we tell the inner client double that it may receive a make_request message. If it does, it should raise an exception while at the same time decrementing the times counter. Once the counter is at 0, it should desist from generating exceptions and instead return a flag value.

We are making use, here, of the fact that RSpec method stubs can take an optional custom implementation block which determines their behavior when invoked.

Now it's just a matter of setting up the expectation that when we send make_request to our proxy client, it will still return that eventual flag value, despite any exceptions raised along the way. This fails, because right now class is just a pure delegator.

To make the test pass, we have to add an explicit method definition. We ignore any arguments that are sent, and delegate to the original method. Then we diverge from pure delegator by rescuing exceptions and retrying.

This is enough to make the current tests pass.

Before we move on, it's worth noting what our test-driven perspective has bought us. By asking "how do we drive this out with tests" instead of "how do we test this", we had a chance to step back and realize that we were dealing with two separate responsibilities. As a result, we aren't fiddling around with WebMock or some similar HTTP testing framework, trying to simulate network failure exceptions. We are able to drop in a test double for the object that does the actual web requests, and make it do whatever we need. This way we can keep our focus firmly on the logic that we are presently adding.

require "./client"
require "rspec/autorun"
require "delegate"

class RetryingClient < DelegateClass(Client)
  def make_request(*)
    super
  rescue
    retry
  end
end

RSpec.describe RetryingClient do
  it "delegates requests to an inner client object" do
    inner = double(make_request: "THE_RESULT")
    client = RetryingClient.new(inner)
    expect(client.make_request).to eq("THE_RESULT")
  end

  it "retries if a request fails" do
    inner = double
    client = RetryingClient.new(inner)
    times = 2
    allow(inner).to receive(:make_request) do
      raise "Hiccup" if (times -= 1) > 0
      "THE_RESULT"
    end
    expect(client.make_request).to eq("THE_RESULT")
  end
end

# >> ..
# >>
# >> Finished in 0.00563 seconds (files took 0.07996 seconds to load)
# >> 2 examples, 0 failures

Back to the task at hand, we have more work to do. We don't want a method that retries forever; we want one that gives up after three tries. Accordingly, we add a third example. This one specifies that the RetryingClient gives up after making three attempts.

This example starts similar to the last one. We have our inner double and our client object. This time we set our counter to 4 instead of 2. Our stub implementation of the inner make_request method is different as well. This time we don't care about return value. We just raise exceptions until the count is exhausted.

We then expect that when we make the request, the result will be that an exception is raised. But we need to go a step beyond this. We want to verify that before this happens, the inner request method is invoked exactly three times - no more, and no less. Fortunately, this is an easy assertion to make in RSpec, since it supports "spy"-style after-the-fact message expectations.

Predictably, this fails. Fortunately, our counter-limited stub method prevents it from failing by looping forever.

At this point, we introduce the rest of the pattern shown in episode #257. We add a parameter for the number of tries, defaulting to 3. We make the retry conditional on a decrementing count being greater than 0. And if we run out of tries, we re-raise the current exception.

With these changes made, the tests pass.

require "./client"
require "rspec/autorun"
require "delegate"

class RetryingClient < DelegateClass(Client)
  def make_request(*, times: 3)
    super
  rescue
    retry if (times -= 1) > 0
    raise
  end
end

RSpec.describe RetryingClient do
  it "delegates requests to an inner client object" do
    inner = double(make_request: "THE_RESULT")
    client = RetryingClient.new(inner)
    expect(client.make_request).to eq("THE_RESULT")
  end

  it "retries if a request fails" do
    inner = double
    client = RetryingClient.new(inner)
    times = 2
    allow(inner).to receive(:make_request) do
      raise "Hiccup" if (times -= 1) > 0
      "THE_RESULT"
    end
    expect(client.make_request).to eq("THE_RESULT")
  end

  it "gives up after 3 retries" do
    inner = double
    client = RetryingClient.new(inner)
    times = 4
    allow(inner).to receive(:make_request) do
      raise "Hiccup" if (times -= 1) > 0
    end
    expect{client.make_request}.to raise_error("Hiccup")
    expect(inner).to have_received(:make_request).exactly(3).times
  end

end

# >> ...
# >>
# >> Finished in 0.00762 seconds (files took 0.08391 seconds to load)
# >> 3 examples, 0 failures

We now have what we set out to build: a client class which can retry requests, for a limited period of time. We also have an answer to the question "how would we test this". It turns out that part of the answer is to let the tests guide the class we end up building.

And that's it for today. Happy hacking!

Responses