In Progress
Unit 1, Lesson 1
In Progress

Filesystem Testing

Video transcript & code

Back in episode #312, we cleaned up some brittle unit tests by extracting out responsibilities from a class. One of the responsibilities we pulled out was that of a "persistor"; a role responsible for persisting a buffer's contents to disk.

Let's drop in at the point where we have the skeleton of a class and the outline of some tests. For the sake of making these tests a little more interesting, we're going to expand on this role's responsibilities. We'll say that not only must it be able to persist some text to a file, it must also create the path to the specified file if any directories do not yet exist.

class FilePersistor
  def initialize(path)
    @path = path
  end

  def save(buffer)
    # ...
  end
end

require "rspec/autorun"
RSpec.describe FilePersistor do
  it "writes the contents of a buffer to a file"
  it "creates any missing directories"
end

In episode #312, we saw the dangers of trying to mock out every system call involved in writing to the filesystem. We saw that there are so many different, equally valid ways to interact with the filesystem in Ruby that erecting mocks and stubs to mimic the filesystem means our tests break every time we adjust our implementation.

Not to mention that when we mock a service we don't own like Ruby's file I/O libraries, we lose any assurance that our code actually does what it needs to do.

RSpec.describe RevPad do
  # ...
  it "can save buffer contents to a file" do
    pad = RevPad.new
    pad.buffer = "It was a dark and stormy night"
    allow(File).to receive(:open).and_return(file = spy("file"))
    pad.save
    expect(file).to have_received(:write).with("It was a dark and stormy night")
    expect(file).to have_received(:close)
  end
  # ...
end

There are a few different libraries in the Ruby ecosystem which seek to mitigate these problems with mocking out the filesystem. Both the MockFS and FakeFS projects set out to completely mimic every single Ruby File I/O method, erecting a "virtual filesystem" for these faked methods to interact with. The idea is to make it possible to test file I/O without ever actually writing or reading anything to or from the disk.

As impressive as these projects are in their scope, there are a couple of big problems with this approach. The first is simply that no matter how completely you mimic Ruby's built-in File I/O APIs, you can never fully imitate the operating system that underlies those calls. There are always going to be scenarios where the tests pass when using the faked filesystem, but the code fails when hooked up to the real thing. This is especially true when code has to work across platforms. There are operations that will work on Linux but fail on Windows, or vice-versa. In the end, you're testing against a hypothetical world.

Second, there is the problem that no matter how comprehensively we fake out the Ruby File I/O systems, if we change the code to sidestep Ruby I/O, the tests will break even while the code continues to work.

Here's a super simple example of what I'm talking about. Let's say that we are shelling out to the Silver Searcher executable to do some grepping through files, then writing the results to a new file.

We can test this using the RSpec and the FakeFS gem. Our example invokes the method and then checks for the existence of the results file. This code will run and the test will show green, even though no file is ever actually written to the filesystem. FakeFS "catches" the write and simulates it within its virtual filesystem.

def do_search
  File.write("results.txt", `ag DelegateClass ..`)
end

require "rspec/autorun"
require "fakefs/spec_helpers"
RSpec.describe "do_search" do
  include FakeFS::SpecHelpers
  it "records search results" do
    do_search
    expect(File.exist?("results.txt")).to be_truthy
  end
end

# >> .
# >>
# >> Finished in 1.53 seconds (files took 0.1173 seconds to load)
# >> 1 example, 0 failures
# >>

However, what if we decide to rewrite this code so that instead of slurping the results into Ruby and then writing them out again, it uses shell redirection to pipe the results directly into a file? In that case, the FakeFS tests fail, because FakeFS can't "see into" the shell command.

def do_search
  `ag DelegateClass .. > results.txt`
end

require "rspec/autorun"
require "fakefs/spec_helpers"
RSpec.describe "do_search" do
  include FakeFS::SpecHelpers
  it "records search results" do
    do_search
    expect(File.exist?("results.txt")).to be_truthy
  end
end

# >> F
# >>
# >> Failures:
# >>
# >>   1) do_search records search results
# >>      Failure/Error: expect(File.exist?("results.txt")).to be_truthy
# >>        expected: truthy value
# >>             got: false
# >>      # xmptmp-in9696Atz.rb:11:in `block (2 levels) in <main>'
# >>
# >> Finished in 0.3352 seconds (files took 0.09023 seconds to load)
# >> 1 example, 1 failure
# >>
# >> Failed examples:
# >>
# >> rspec  # do_search records search results
# >>

But this failure is a false positive. If we remove FakeFS, we can see that the test once again passes. Now that we are writing real bytes to real files on disk, we see that the code is still working perfectly.

def do_search
  `ag DelegateClass .. > results.txt`
end

require "rspec/autorun"
RSpec.describe "do_search" do
  it "records search results" do
    do_search
    expect(File.exist?("results.txt")).to be_truthy
  end
end

# >> .
# >>
# >> Finished in 0.37812 seconds (files took 0.07633 seconds to load)
# >> 1 example, 0 failures
# >>

For these reasons, I find it difficult to think of a situation in which I would recommend the use of a gem like MockFS or FakeFS.

So what do we do? I think the only practical way to test code that interacts with the filesystem is to have it interact with an actual filesystem. Here's how we might test our FilePersistor class.

Using the tools we learned about in episode #313, we construct a temporary directory. We create a FilePersistor object which should write inside this temp directory. We tell the persistor to #save some text. Then we do a file read to verify that the text has been written as we expect.

The next test isn't far different. This time, though, we specify that the file to be written is a few levels lower down the directory tree. Then we verify that after saving, the intermediate directories now exist.

require "fileutils"
class FilePersistor
  def initialize(path)
    @path = path
  end

  def save(buffer)
    FileUtils.mkpath(File.dirname(@path))
    File.write(@path, buffer)
  end
end

require "rspec/autorun"
require "tmpdir"
RSpec.describe FilePersistor do
  it "writes the contents of a buffer to a file" do
    Dir.mktmpdir do |d|
      fp = FilePersistor.new("#{d}/jabberwocky.txt")
      fp.save("'twas brillig and the slithy toves")
      expect(File.read("#{d}/jabberwocky.txt"))
        .to eq("'twas brillig and the slithy toves")
    end
  end
  it "creates any missing directories" do
    Dir.mktmpdir do |d|
      fp = FilePersistor.new("#{d}/foo/bar/jabberwocky.txt")
      fp.save("'twas brillig and the slithy toves")
      expect(File.directory?("#{d}/foo/bar")).to be_truthy
    end
  end
end

# >> ..
# >>
# >> Finished in 0.0025 seconds (files took 0.09675 seconds to load)
# >> 2 examples, 0 failures
# >>

These are robust tests that say absolutely nothing about how the class under test should be implemented. As long as it writes the correct text in the correct place in the filesystem, we have infinite leeway to update the internals of this object anyway we see fit, without breaking the tests.

Of course, there is one obvious objection to this approach: speed. Won't this strategy give us slow tests?

Yes, to some degree this is unavoidable. Tests that deal with the real filesystem are always going to be slower than tests that deal in mere simulacra.

But that's where the importance of limiting class responsibility enters in. We originally arrived at this concept of a "persistor" role as we were extracting responsibilities from another class. Because this object is only responsible for writing text to a file, we don't have much to test. We can probably get by with a half a dozen tests or fewer for this class.

The key to test suites that are fast but still robust isn't faking out everything under the sun. Rather, it's a matter of squeezing system interactions down to their most essential parts at the edges of the application, and then limiting our integration tests to those "adapter" classes. The rest of our tests can be isolated and test only that our code makes the right decisions in the right circumstances.

And that's it for today. Happy hacking!

Responses