In Progress
Unit 1, Lesson 1
In Progress

Consequences

Video transcript & code

Proponents of test-driven development often argue that TDD is primarily a design discipline. Writing code to pass tests, especially when they are small, focused tests, forces units to be small and decoupled. It encourages injectable dependencies that can easily be switched out for test doubles.

Skeptics of TDD point out that this kind of design emphasis can seem circular: we write tests first in order to encourage decoupled designs, and we encourage decoupled designs in order to make our tests easier to write. Understandably, they question whether this style of design is actually accomplishing anything other than making us feel better about ourselves.

Today I want to take a look at what happens when we write code without any intent to make it "testable".

As an example, we'll use some code I wrote recently. Here's a selection from a Sinatra application. This code is not in any way contrived; it's code I wrote to solve a very real problem.

post "/ipn" do
  demand_basic_auth
  logger.info "Received IPN post with params:"
  logger.info params.pretty_print_inspect

  redemption_url_template = Addressable::Template.new(
      "#{request.base_url}/redeem?token={token}")

  Perkolator.process_ipn(
      params,
      redemption_url_template: redemption_url_template,
      email_options:           settings.email_options,
      logger:                  logger)

  [202, "Accepted"]
end

The problem is this. I recently launched a new book project, and I want to add early buyers to the Github project so they can participate in the book writing process. In order to make this happen, I need to invite new buyers as soon as they make a purchase.

The storefront service I use is able to notify other services by hitting a webhook whenever a purchase is made. The Sinatra action we're looking at is the handler for that webhook. It receives an HTTP POST with set of parameters, known as IPN data.

I wrote this application in a single sitting, with a lot of manual testing but no automated tests. Since then I added a high-level smoke test, but it interacts with the application as a black box, so it had no influence on the application's design (or lack thereof).

And indeed, this little app has little design to speak of. There's only one nod to decoupling here: instead of handling all of the logic inline, this Sinatra action delegates its logic to a transaction script called Perkolator.process_ipn.

Perkolator.process_ipn(
    params,
    redemption_url_template: redemption_url_template,
    email_options:           settings.email_options,
    logger:                  logger)

Recently this app experienced some downtime, and missed one or two IPN messages from the storefront. This means I need to register the purchases manually. Let's explore what this task entails.

In the spirit of keeping things simple, we won't worry about anything as ambitious as building an admin UI for the app. Instead, we can just use an IRB console on the application server to update the appropriate tables and send the requisite invitation email. Let's first spend some time locally, figuring out the commands we'll need to execute on the server.

Approaching this problem, it seems like we're in a good position to simulate an IPN callback from the console. After all, the logic is already inside a single transaction script method. We just need to set up our environment and then invoke that method.

First off, we'll set up the app's environment. This is made easy by an environment.rb file that exists for this purpose. Next, we require the file that contains the Perkolator module.

Dir.chdir "/home/avdi/dev/perkolator"
require "~/dev/perkolator/environment"
require "perkolator"

Now, let's take another look at the method we need to invoke. It takes several parameters. The first is a params hash of IPN data.

A full set of IPN data looks like this:

{
    "payer_email"          => "johndoe@example.org",
    "txn_id"               => "3664317",
    "mc_gross"             => "25.00",
    "discount"             => "0.00",
    "mc_currency"          => "USD",
    "tax"                  => "0.00",
    "mc_shipping"          => "0.00",
    "coupon_code"          => "",
    "shipping_method"      => "",
    "first_name"           => "John",
    "last_name"            => "Doe",
    "address_street"       => "",
    "address_city"         => "",
    "address_state"        => "",
    "address_zip"          => "",
    "address_country"      => "",
    "address_country_code" => "",
    "ip_address"           => "174.54.224.242",
    "item_name1"           => "The Rake Field Manual",
    "item_number1"         => "89064",
    "quantity1"            => "1",
    "mc_gross_1"           => "25.00",
    "discount_1"           => "0.00",
    "mc_currency_1"        => "USD",
    "sku1"                 => "",
    "product_key1"         => "",
    "verify_sign"          => "eQhmv/Lavtlo46jBtiYUUJF8q82OeQznnfjjYZeLoxZ5AQtTIWwK+mckfkLsh9Y+1qgUet0VcRg/vjrvCeJgXg=="
}

But when we look at the source code for the .process_ipn method, we see that the only field it cares about is payer_email. So we create a params hash that only contains that property.

params = {
  "payer_email" => "tomservo@example.com"
}

The next argument we need to supply is called redemption_url_template. This is expected to be a URL template which, with the addition of a redemption token, will be expanded to the URL the user must click in their email in order to accept their invitation.

Why do we have to pass this in? Well, in the live app, we use the current request in order to figure out the base URL of the application. This information is only available from inside the Sinatra action, so we have to pass it in from outside the .process_ipn method.

redemption_url_template = Addressable::Template.new(
      "#{request.base_url}/redeem?token={token}")

We have to require the addressable/template gem in order to provide a URL template. We might talk about this gem more in another episode.

For our version of the argument, we'll just hardcode the base URL.

require "addressable/template"
url_template = Addressable::Template.new(
  "http://localhost:5678/redeem?token={token}")

The next argument we need to provide is a set of email options. This is Hash, which we fill in mostly from various environment variables.

email_options = {
  :address        => ENV.fetch("SMTP_HOST"),
  :port           => ENV.fetch("SMTP_PORT"),
  :user_name      => ENV.fetch("MANDRILL_USERNAME"),
  :password       => ENV.fetch("MANDRILL_PASSWORD"),
  :domain         => "heroku.com",
  :authentication => :plain
}

Again, you might wonder why we have to pass this information in. It's because the canonical source for it is in Sinatra settings which we don't have direct access to in the Perkolator module.

Next up, we need to provide a logger argument. For this we just need to require the logger library and construct a basic Logger object.

require "logger"
logger = Logger.new($stdout)

At least we're ready to actually invoke the method. We fill in all of the parameters with the arguments we've carefully constructed, and then execute. Unfortunately, we run into some problems with undefined constants. It turns out we need to require a few more files before the method will run to completion.

Perkolator.process_ipn(params, 
  redemption_url_template: url_template, 
  email_options: email_options,
  logger: logger)
# ~> /home/avdi/dev/perkolator/lib/perkolator.rb:11:in `process_ipn': uninitialized constant Perkolator::DB (NameError)
# ~>    from -:25:in `<main>'
# >> I, [2014-07-21T14:47:34.703725 #5384]  INFO -- : Storing IPN for tomservo@example.com

Finally, we have successfully made use of the .process_ipn transaction script. Let's take a look back at how much code we needed to write to get to this point.

Dir.chdir "/home/avdi/dev/perkolator"
require "~/dev/perkolator/environment"
require "perkolator"
 # !> File.exists? is a deprecated name, use File.exist? instead
params = {
  "payer_email" => "tomservo@example.com"
}

require "addressable/template"
url_template = Addressable::Template.new(
  "http://localhost:5678/redeem?token={token}")

email_options = {
  :address        => ENV.fetch("SMTP_HOST"),
  :port           => ENV.fetch("SMTP_PORT"),
  :user_name      => ENV.fetch("MANDRILL_USERNAME"),
  :password       => ENV.fetch("MANDRILL_PASSWORD"),
  :domain         => "heroku.com",
  :authentication => :plain
}

require "logger"
logger = Logger.new($stdout)

require "db"
require "tilt"

Perkolator.process_ipn(params, 
  redemption_url_template: url_template, 
  email_options: email_options,
  logger: logger)
# >> I, [2014-07-21T14:48:13.844691 #5422]  INFO -- : Storing IPN for tomservo@example.com
# >> I, [2014-07-21T14:48:13.871850 #5422]  INFO -- : Storing token 3da9ee781d8ac6bfffeed0eea7a98eb8 for tomservo@example.com
# >> I, [2014-07-21T14:48:13.935373 #5422]  INFO -- : Sending welcome email to tomservo@example.com
# >> I, [2014-07-21T14:48:13.935435 #5422]  INFO -- : Redemption URL for tomservo@example.com is http://localhost:5678/redeem?token=3da9ee781d8ac6bfffeed0eea7a98eb8

Let's not mince words: this was a stupidly large amount of code needed to get this working. And it's not just about the number of arguments we needed to provide. For some of the parameters we had to carefully read through the source code to see what the needed data was supposed to look like. And we also discovered some implicit dependencies on gems or other project files along the way.

And yet nothing about this is unusual. In various projects I've worked on that lacked a strong test-first discipline, I've regularly had to write this much code (or more) in order to get existing code working in some new context. To write the code we see here I had the advantage of still remembering how I'd written the app. I hesitate to think how long it would have taken me to write this code if I were returning to the project after 6 months away.

Remember, the problem at hand isn't about getting the .process_ipn method working under test. This is about using it in the very practical context of an IRB console in order to perform some app administration, while customers impatiently wait for the promised invitation to show up in their email inbox. So for now let's put aside the question of whether it's important to write code that's easy to test. Let's instead ask: is it important to write code that's easy to use from the console?

To this, my answer is an emphatic yes. Without ease of use in the REPL, we lose one of the greatest advantages of coding our applications in a dynamic language. Setting aside its usefulness for quick-and-dirty administrative tasks, the console is also a wonderful tool for exploring a codebase and prototyping code. And the ease or difficulty of using an interface in the console provides a great indication of how hard that code will be to use in other contexts, such as a background job.

If I had written the Perkolator module test-first, I would have discovered or preempted all of these interface usability issues in the process of driving it out. On the other hand, I have no doubt that if I'd written it this way it also would have taken me a little longer to complete it. I chose to prioritize initial time to market over the design of the code, and that decision had consequences.

A key value in OO design is how easy it is to use objects and methods from contexts they were not originally written for. The best way to promote this aspect is to give objects a second client apart their original call site. Using an object from the console is one example of a second client. Unit tests are another example, one that has the added benefit of being automated and cumulative. Whether in the form of unit tests or the console, the choice to make objects easy to use outside of their original context has important consequences for our design, and ultimately for our ability to quickly adapt to new needs.

And that's all for today. Happy hacking!

Responses