OpenAPI and Apivore – with Ariel Caplan

Have you ever integrated an API into your application, only to find out that the real-life behavior of the API wasn't what you'd expect from the documentation?

Maybe you've been on the other side, developing an API with documentation that always seemed to fall out of date.

Let's relive that pain for just a moment, and then we'll see what we can do as API developers to make things better for ourselves and for our users.

In this made-up case, we have a Rails API for managing widgets.

$ curl -i http://localhost:3000/widgets/blue.json

When we send a GET request to /widgets/blue, where "blue" is the slug of the widget, we expect the API to return either a 200 status code and a JSON representation of a widget, or, if the widget we're looking for is not found, a 404 status code and a descriptive error.

Instead, we get back a 500 status code and a vague error message.

$ curl -i http://localhost:3000/widgets/blue.json
HTTP/1.1 500 Internal Server Error
Content-Type: application/json; charset=utf-8
X-Request-Id: 95bf9d6f-3d68-4dc0-8600-275e06082a17
X-Runtime: 0.021941
Content-Length: 46

{"status":500,"error":"Internal Server Error"}

Digging into the application code, it seems that the problem starts in the controller. If the widget with the requested slug is not found, the @widget instance variable is set to nil. That nil is passed to the view , which tries to render attributes of the object. Since nil doesn't have an id, name, slug, etc., we end up with an ambiguous NoMethodError.

# app/controllers/widgets_controller.rb
def set_widget
  @widget = Widget.find_by(slug: params[:id])
end

# app/views/widgets/show.json.jbuilder
json.partial! "widgets/widget", widget: @widget

# app/views/widgets/_widget.json.jbuilder
json.extract! widget, :id, :name, :slug, :description, :price, :created_at, :updated_at
json.url widget_url(widget, format: :json)

This case is probably covered in our documentation, and the scary part is, we probably told our users explicitly to expect a 404. Even if we correct the problem in our code, there's nothing programmatically linking our docs and our code to make sure there aren't tons of other little issues like this lurking all over our API.

Today we're going to talk about a way to generate friendly documentation using YAML or JSON, and more importantly, to make sure it's always accurate.

OpenAPI, more popularly known as Swagger, is an open JSON specification for describing the structure and semantics of JSON APIs.

Because it's a standard designed to be understood easily by computers, there is a vast array of open-source tooling around Swagger, including auto-generated human-friendly documentation with built-in functionality to try out web requests right from your browser.

There's also an online editor where you can build up and edit your documentation in YAML format and see in realtime how your changes impact your docs.  However, I'd like to focus this episode on a particular Swagger-based tool, which is all about solving the problem of inaccurate documentation. I'd like you to meet Apivore.  Apivore is a library that lets you write RSpec tests for Rails apps to ensure that all your Swagger docs are actually implemented in the code you've written.

What's really neat is that with Apivore, you start thinking of your docs as the source of truth about your application, and you write code until your application does exactly what your documentation says it does.That sounds a little abstract, so let's look at a concrete example.Before we do, though, I'll mention that there's a lot more to say about these tools than I can cover in an episode. If you do want to follow up and learn more, I've set up a page with information and resources on Swagger, Apivore, and documentation testing in general. You can follow the link in the show notes. And with that, let's hop into the code!

We'll jump a bit into the future, and assume we've already integrated Swagger docs into our project, available at the /docs route.

As you can see,...

...we've established an expectation that the /widgets/{slug} endpoint will return either a 200 or a 404.

I've also taken the liberty of including the Apivore gem in our Gemfile, and I've written a bit of boilerplate that will let us write extremely concise tests.

Note that the subject of all tests is an instance of Apivore::SwaggerChecker. This will actually be the same object across all your tests, which comes in handy, as you'll see in just a moment. The rest of the boilerplate just sets up hooks to let us specify headers, query params, and request body more easily in any test where we need to.

# Gemfile
gem 'apivore'

# spec/requests/api_spec.rb
require 'rails_helper'

RSpec.describe 'the API', type: :apivore, order: :defined do
  subject { Apivore::SwaggerChecker.instance_for('/docs.json') }
  let(:url_params) {{}}
  let(:headers) {{}}
  let(:query_string_params) {{}}
  let(:data_params) {{}}
  let(:params) {
    url_params.merge(
      '_headers' => headers.merge('Accept' => 'application/json'),
      '_query_string' => query_string_params.to_query,
      '_data' => data_params
    )
  }
end

The first thing we'll do is take advantage of a really nifty feature of Apivore, namely writing a spec for our specs. In this case, we're going to assert that every endpoint in the documentation is covered by a passing test.

  context 'tying it all together' do
    it 'tests all documented routes' do
      expect(subject).to validate_all_paths
    end
  end

It's important that you have order: :defined in the context of your tests, and you put this test last.

This way, as you run your tests, the SwaggerChecker instance you use - remember how I said it's the same instance every time? - that SwaggerChecker instance will keep track of which endpoints and statuses you've tested, and it'll give you a big red failure if you haven't yet tested every endpoint and status in your documentation.

For example, if we run our test now, looks like the /widgets/{slug} endpoint is untested for both 200 and 404 response codes. Having this sort of to-do list is a huge help both in backfilling a documented API with tests, and in making sure your documentation will never leave your tests behind.

$ rspec
F

Failures:

  1) the API tying it all together tests all documented routes
     Failure/Error: expect(subject).to validate_all_paths
     
       get /widgets/{slug} is untested for response code 200
       get /widgets/{slug} is untested for response code 404
     # ./spec/requests/api_spec.rb:19:in `block (3 levels) in '

Finished in 0.14786 seconds (files took 1.57 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/requests/api_spec.rb:18 # the API tying it all together tests all documented routes

The first test is going to be fairly straightforward. We'll test the 404 case for the /widgets/{slug} endpoint, since it doesn't require any database setup. We use our handy url_params from that boilerplate to specify what we want to use in the dynamic portion of the URL, in this case the slug "blue".

Then, we'll use RSpec's implicit subject to call validate on our SwaggerChecker with the request method, GET, the URL under test, the 404 response code, and the request parameters. Apivore will then make a request to our application, filling in the dynamic {slug} in the URL, and check that the response matches the documentation for the 404 case.

  describe "GET '/widgets/{slug}'" do
    let(:url_params) {{ 'slug' => 'blue' }}

    context 'the widget does not exist' do
      it { is_expected.to validate(:get, '/widgets/{slug}', 404, params) }
    end
  end

As expected, our test fails because of our NoMethodError we saw earlier.

$ rspec
FF

Failures:

  1) the API GET '/widgets/{slug}' the widget does not exist should validate that get /widgets/{slug} returns 404
     Failure/Error: json.extract! widget, :id, :name, :slug, :description, :price, :created_at, :updated_at
     
     ActionView::Template::Error:
       undefined method `id' for nil:NilClass
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder.rb:260:in `public_send'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder.rb:260:in `block in _extract_method_values'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder.rb:260:in `each'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder.rb:260:in `_extract_method_values'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder.rb:219:in `extract!'
     # ./app/views/widgets/_widget.json.jbuilder:1:in `_app_views_widgets__widget_json_jbuilder__1560270032508288162_70267474213860'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder/jbuilder_template.rb:128:in `_render_partial'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder/jbuilder_template.rb:122:in `_render_partial_with_options'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder/jbuilder_template.rb:215:in `_render_explicit_partial'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder/jbuilder_template.rb:22:in `partial!'
     # ./app/views/widgets/show.json.jbuilder:1:in `_app_views_widgets_show_json_jbuilder___4056245668939668145_70267448600180'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/rack-2.0.4/lib/rack/etag.rb:25:in `call'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/rack-2.0.4/lib/rack/conditional_get.rb:25:in `call'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/rack-2.0.4/lib/rack/head.rb:12:in `call'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/railties-5.1.5/lib/rails/rack/logger.rb:36:in `call_app'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/railties-5.1.5/lib/rails/rack/logger.rb:24:in `block in call'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/railties-5.1.5/lib/rails/rack/logger.rb:24:in `call'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/rack-2.0.4/lib/rack/runtime.rb:22:in `call'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/rack-2.0.4/lib/rack/sendfile.rb:111:in `call'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/railties-5.1.5/lib/rails/engine.rb:522:in `call'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/rack-test-0.8.3/lib/rack/mock_session.rb:29:in `request'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/rack-test-0.8.3/lib/rack/test.rb:251:in `process_request'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/rack-test-0.8.3/lib/rack/test.rb:119:in `request'
     # /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/apivore-1.6.2/lib/apivore/validator.rb:22:in `matches?'
     # ./spec/requests/api_spec.rb:21:in `block (4 levels) in '
     # ------------------
     # --- Caused by: ---
     # NoMethodError:
     #   undefined method `id' for nil:NilClass
     #   /Users/acaplan/.rvm/gems/ruby-2.5.0/gems/jbuilder-2.7.0/lib/jbuilder.rb:260:in `public_send'

  2) the API tying it all together tests all documented routes
     Failure/Error: expect(subject).to validate_all_paths
     
       get /widgets/{slug} is untested for response code 200
       get /widgets/{slug} is untested for response code 404
     # ./spec/requests/api_spec.rb:27:in `block (3 levels) in '

Finished in 0.13557 seconds (files took 2.36 seconds to load)
2 examples, 2 failures

Failed examples:

rspec ./spec/requests/api_spec.rb:21 # the API GET '/widgets/{slug}' the widget does not exist should validate that get /widgets/{slug} returns 404
rspec ./spec/requests/api_spec.rb:26 # the API tying it all together tests all documented routes

Luckily, it's an easy fix! First, we'll modify our set_widget method to use the bang version of find_by so it raises a more descriptive error when the requested record doesn't exist in our system. Running the test again verifies we now have an ActiveRecord::RecordNotFound error.

# app/controllers/widgets_controller.rb
def set_widget
  @widget = Widget.find_by!(slug: params[:id])
end

$ rspec
FF

Failures:

  1) the API GET '/widgets/{slug}' the widget does not exist should validate that get /widgets/{slug} returns 404
     Failure/Error: @widget = Widget.find_by!(slug: params[:id])
     
     ActiveRecord::RecordNotFound:
       Couldn't find Widget
# etc...

Finally, we'll add some code to rescue that type of error, and render an error message with a 404.

# app/controllers/application_controller.rb
class ApplicationController < ActionController::API
  include ActionView::Rendering
  rescue_from ActiveRecord::RecordNotFound, with: :not_found

  def not_found(error)
    render json: { error: 'Requested record not found.' }, status: 404
  end
end

Run the test again, and check it out! We have one passing test, and just one response code, 200, left to test.

$ rspec
.F

Failures:

  1) the API tying it all together tests all documented routes
     Failure/Error: expect(subject).to validate_all_paths
       get /widgets/{slug} is untested for response code 200
     # ./spec/requests/api_spec.rb:27:in `block (3 levels) in '

Finished in 0.12543 seconds (files took 1.46 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./spec/requests/api_spec.rb:26 # the API tying it all together tests all documented routes
$ 
 

Let's look back at our documentation to remember how this is supposed to look. We expect to send a request which specifies a slug, and receive a bunch of attributes in the response: id as an integer, name as a string, price as a number... you get the idea.

Again, we'll start by writing the test first. This time, we'll create an example model with a full set of attributes, and verify that it's returned in the documented format for a 200 response code. Notice that our test is just a quick one-liner, because Apivore is using our documentation to figure out what to expect in the response.

# spec/requests/api_spec.rb
describe "GET '/widgets/{slug}'" do
  let(:url_params) {{ 'slug' => 'blue' }}

  context 'the widget does not exist' do
    it { is_expected.to validate(:get, '/widgets/{slug}', 404, params) }
  end

  context 'the widget exists' do
    before do
      Widget.create(name: 'blue', slug: 'blue', price: 14.90, description: 'A blue widget!')
    end

    it { is_expected.to validate(:get, '/widgets/{slug}', 200, params) }
  end
end

Now when we run our test, something interesting happens. Our documentation said to expect price as a number, but it turns out we're actually returning a String. And now we have to ask: Is that actually such a bad thing?

We could return price as a floating point number, but perhaps the complexity of floating point will mean that the number of decimal places may be more or less than 2.

$ rspec
.F.

Failures:

  1) the API GET '/widgets/{slug}' the widget exists should validate that get /widgets/{slug} returns 200
     Failure/Error: it { is_expected.to validate(:get, '/widgets/{slug}', 200, params) }
     
        '/widgets/blue?#/price' of type string did not match the following type: number  
       Response body:
        {
         "id": 14,
         "name": "blue",
         "slug": "blue",
         "description": "A blue widget!",
         "price": "14.9",
         "created_at": "2018-03-07T00:05:22.537Z",
         "updated_at": "2018-03-07T00:05:22.537Z",
         "url": "http://www.example.com/widgets/14.json"
       }
     # ./spec/requests/api_spec.rb:29:in `block (4 levels) in '

Finished in 0.13562 seconds (files took 1.66 seconds to load)
3 examples, 1 failure

Failed examples:

rspec ./spec/requests/api_spec.rb:29 # the API GET '/widgets/{slug}' the widget exists should validate that get /widgets/{slug} returns 200

Thinking through things more carefully, returning as a String with 2 decimal places seems to be the best idea for our application, so we'll update our documentation to say so. We modify the price attribute of the widget to be of the "string" type, using the pattern of one or more digits, followed by a period and exactly 2 more digits.

# in the middle of config/swagger.yml
price:
  type: "string"
  pattern: "^\\d+\\.\\d{2}$"

And just based on that change to documentation, our test now expects a price-formatted String as the price.

$ rspec
.F.

Failures:

  1) the API GET '/widgets/{slug}' the widget exists should validate that get /widgets/{slug} returns 200
     Failure/Error: it { is_expected.to validate(:get, '/widgets/{slug}', 200, params) }
     
        '/widgets/blue?#/price' value "14.9" did not match the regex '^\d+\.\d{2}$'  
       Response body:
        {
         "id": 15,
         "name": "blue",
         "slug": "blue",
         "description": "A blue widget!",
         "price": "14.9",
         "created_at": "2018-03-07T00:06:19.377Z",
         "updated_at": "2018-03-07T00:06:19.377Z",
         "url": "http://www.example.com/widgets/15.json"
       }
     # ./spec/requests/api_spec.rb:29:in `block (4 levels) in '

Finished in 0.1509 seconds (files took 1.7 seconds to load)
3 examples, 1 failure

Failed examples:

rspec ./spec/requests/api_spec.rb:29 # the API GET '/widgets/{slug}' the widget exists should validate that get /widgets/{slug} returns 200

At this point, all that's left is to actually implement converting price to a properly formatted String.

# app/views/widgets/_widget.json.jbuilder
json.extract! widget, :id, :name, :slug, :description, :created_at, :updated_at
json.price sprintf('%.2f', widget.price)
json.url widget_url(widget, format: :json)

And we're done! All the documented paths are tested, and those tests are passing.

$ rspec
...

Finished in 0.1504 seconds (files took 2.23 seconds to load)
3 examples, 0 failures

I want to make sure you noticed something important here: We didn't just fix our application to match our documentation, we also revised our documentation because the exercise of testing it put us into our users' shoes and helped us think about what would be most useful to them.

Having a guarantee that your documentation is accurate is really, incredibly valuable, and the exercise is worthwhile for that value alone. But I've found that the way I build APIs changes when I drive testing from documentation in this way. I think a lot more about the user's perspective, and I become more aware of bad design decisions, or areas that may cause confusion or frustration for users.

I hope you'll find that documentation testing will also help you apply user-centered design principles automatically, just by virtue of the tools you've chosen to use.

That's all for today. Happy hacking!