In Progress
Unit 1, Lesson 1
In Progress

Cache Rot

Video transcript & code

Did you ever put some food aside, then pull it out of the fridge later, only to find that it had grown a fresh coat of green fur? Sometimes data in a cache can give us a similarly rude surprise. In this episode I want to show you an example that bit me just the other day, and then discuss how to avoid it.

Here's a class I've been using in my episode publishing scripts.

module Tapub
  class WistiaGateway
    # ...
  end
end

It's a "gateway" object, a pattern we've explored in a number of past episodes. This particular gateway provides an insulation layer between my business logic and a third-party video API gem.

There's a method in this class which fetches metadata for a whole project's worth of videos.

def get_video_list(project_id)
  logger.info "Fetch Wistia video list for project #{project_id}"
  videos = []
  number = 1
  while (page = get_video_list_page(number, project_id)).any?
    videos.concat(page)
    number += 1
  end
  videos
end

In order to do so, it has to request multiple pages of results. It takes a long time to complete all these requests. We'd like to cache this data and only re-request it every day or so.

Fortunately, we already have a cache object injected into this class..

def initialize(
    password:,
    collection_name:,
    public_base_uri:,
    logger: Logger.new($stderr),
    cache: Moneta.new(:Null))
  # ...
  @cache           = cache
  # ...
end

It's a Moneta object, which you might recall from episode #67, "Moneta".

So, let's do the obvious. We'll go down to our get_video_list method.

We'll surround the whole body of the method in a cache fetch block.

As a cache key, we'll combine a prefix with the id of the project being listed.

And at the bottom of the block, we'll update the cache value with the new results before returning them.

def get_video_list(project_id)
  logger.info "Fetch Wistia video list for project #{project_id}"
  key = "wistia_video_list:#{project_id}"
  cache.fetch(key) {
    videos = []
    number = 1
    while (page = get_video_list_page(number, project_id)).any?
      videos.concat(page)
      number += 1
    end
    cache[key] = videos
  }
end

The first time we run the code with this change, it works great.

$ tapub wistia_upload
Fetch Wistia video list for project 570689
Write .tapub/video-info.yaml

The second time we run it, it blows up.

$ rm .tapub/video-info.yaml
$ tapub wistia_upload
Fetch Wistia video list for project 570689
/home/avdi/.gem/ruby/2.3.0/gems/moneta-0.8.0/lib/moneta/transformer.rb:132:in `load': undefined class/module Wistia::Media::Thumbnail (ArgumentError)

The error we see is that there's an undefined class or module named Wistia::Media::Thumbnail.

So, OK. This is annoying, but we know how to fix it, right? We just have to make sure that this class is loaded before we try to reconstruct any cached objects. All we need to do is find out where the class is defined, and ensure that the file is loaded before any of our business logic runs.

And this is why I'm doing this demonstration in RubyMine today. RubyMine is really good at finding out where things are defined, even in third-party gems. Let's ask it where the Thumbnail class comes from.

Hmmm… that's weird. RubyMine doesn't seem to be able to find it.

Now, at this point I'm going to spare you the investigation I did when I first ran into this mystery. I'll just cut to the chase and tell you what I discovered: the Wistia::Media::Thumbnail class isn't defined anywhere. Apparently, it is auto-generated at runtime as part of some ActiveResource magic.

So now we have a problem. We need to preload this class in order to avoid errors when the cache is read. But it's not possible to preload the class!

The solution to this problem is to solve a slightly different problem instead.

First, let's get rid of the offending caching code.

def get_video_list(project_id)
  logger.info "Fetch Wistia video list for project #{project_id}"
  videos = []
  number = 1
  while (page = get_video_list_page(number, project_id)).any?
    videos.concat(page)
    number += 1
  end
  videos
end

Next, let's look around this class a little more. This is a Gateway class, and one of the rules I try to observe for Gateways is that they don't return any third-party data-types. Instead, they map information to a simplified data representation first.

There's a method which does exactly this for video objects, called info_from_video.

def info_from_video(video)
  public_url     = Addressable::URI.join(
    @public_base_uri,
    "/medias/#{video.hashed_id}").to_s
  download_asset = video.assets.detect { |a| a.type == "OriginalFile" }
  oembed_url     = public_url +
                   "?embedType=async&videoFoam=true&videoWidth=1280"
  result         = {
    id:            video.id,
    hashed_id:     video.hashed_id,
    name:          video.name,
    thumbnail_url: video.thumbnail.url,
    public_url:    public_url,
    oembed_url:    oembed_url,
    still_url:     video.still(1280),
    downloads:     [
      {
        name:         "Original Video",
        content_type: download_asset.contentType,
        file_size:    download_asset.fileSize,
        url:          download_asset.url,
        width:        download_asset.attributes["width"],
        height:       download_asset.attributes["height"],
      }
    ]
  }
  result
end

Even with just a quick once-over, we can see that this method is mapping data into a simple structure of nested hashes and arrays.

A little further up is a method which calls the info_from_video method in order to convert a list of fetched Video objects to this simplified, application-specific format.

def videos_in_project(project_name)
  project_id = find_project_id(project_name)
  videos     = get_video_list(project_id)
  videos.map { |v|
    info_from_video(v)
  }
end

Let's put our caching at this level instead of at the point where the specialized video objects are returned.

Once again, we'll surround the method with a cache fetch.

Once again, we'll construct a cache key from a prefix and, in this case, the video project name.

Once again, we'll update the cache contents when the data has been fetched.

def videos_in_project(project_name)
  key = "wistia_video_list:#{project_name}"
  cache.fetch(key) {
    project_id = find_project_id(project_name)
    videos     = get_video_list(project_id)
    cache[key] = videos.map { |v|
      info_from_video(v)
    }
  }
end

Let's try this code out again. We blow away the cache and then run the command again. We see that it fetches the video list.

Then we run it a second time. This time, we see no output, because the video list has been cached. What we don't see is a crash caused by a missing class.

$ tapub wistia_upload
Fetch Wistia video list for project 570689
Write .tapub/video-info.yaml
$ tapub wistia_upload
$

The problem we ran into today is part of a larger class of caching bugs that can be caused by serializing objects to a persistent store. In this case, the cache couldn't be reloaded because of a missing dynamic class definition. Another, sneakier problem that sometimes crops up is when there are objects from an older version of a library still left serialized and dormant in a cache.

There's a simple and reliable rule for avoiding this whole family of bugs. It's this: ensure that we only ever put plain old data into caches. When I say "plain old data", I'm talking about fundamental data types that are part of core Ruby and are always loaded. Things like strings, numerics, arrays, hashes, and Time objects.

This is one of several reasons I like my Gateway objects to return "plain old data" from their AP Is: It makes those methods a lot easier to cache.

I know I already mentioned this issue, and its solution, in passing way back in episode #169. But after I was bitten by it again the other day, I thought that it might be helpful to have a concrete example of the problem.

The lesson here is as simple as the problem is tricky: Caches work best when they are full of simple, universally reconstitutable data. If we only keep non-perishable objects in our cache, and we'll always be able to get back what we put in. Happy hacking!

Responses