Redirecting Output

Ruby programs, like pretty much any kind of program, start out with two output channels: standard out and standard error. As you know, we typically communicate with standard out by calling puts.

puts "This goes to standard out"
# >> This goes to standard out

While writing to the standard streams is sufficient in command-line applications and in demonstration scripts, in larger applications it's often useful to capture a program's output. We may want to ship the output off to another machine, store it for later examination, or present it to the user in GUI or web browser.

Today we'll look at how to redirect output and capture it as a string. But first, let's dig into how the standard streams are handled in Ruby.

When we write puts, that's actually short for $stdout.puts:

$stdout.puts "This goes to standard out"
# >> This goes to standard out

$stdout is a special global variable that points to an IO object for the standard output stream. There is a similar $stderr variable for the standard error stream.

$stderr.puts << "This goes to standard error"

One simple way to capture output is to reassign the $stdout variable to some other object which can store up the output. Let's write a method to do that. We need some kind of object which behaves like an IO object but stores the output into a string instead of writing it to the console. Ruby gives us just such an object, in the form of the StringIO class. We require the 'stringio' library, and instantiate a new StringIO object. We save the old value of $stdout, set it to point to our StringIO, and execute the given block. Once the block is finished, we restore the original $stdout. We do this inside an ensure clause to make sure it happens even if an exception was raised from within the block. Then we return the String buffer from inside the StringIO.

When we wrap this around a puts statement, the output is captured to a string.

require 'stringio'

def capture_output
  fake_stdout = StringIO.new
  old_stdout = $stdout
  $stdout = fake_stdout
  yield
ensure
  $stdout = old_stdout
  return fake_stdout.string
end

output = capture_output do
  puts "This goes to standard out"
end
output # => "This goes to standard out\n"

Ruby has another name for standard out: a constant named STDOUT (in all caps). This constant is unaffected by changes to the $stdout global, so we can bypass the capture by explicitly referencing the STDOUT constant inside our capture block.

output = capture_output do
  puts "This goes to standard out"
  STDOUT.puts "This bypasses capture"
end
output # => "This goes to standard out\n"
# >> This bypasses capture

If we are seeking a more complete capture, we could reassign this constant too. But that results in Ruby warnings, and it's kind of against the spirit of having STDOUT be a constant. So I wouldn't advise it.

require 'stringio'

def capture_output
  fake_stdout = StringIO.new
  old_stdout = $stdout
  $stdout = fake_stdout
  Object.const_set(:STDOUT,fake_stdout) # !> already initialized constant STDOUT
  yield
ensure
  $stdout = old_stdout
  Object.const_set(:STDOUT,old_stdout) # !> already initialized constant STDOUT
  return fake_stdout.string
end

output = capture_output do
  puts "This goes to standard out"
  STDOUT.puts "This tries to bypass capture"
end
output
# => "This goes to standard out\nThis tries to bypass capture\n"

And in any case, it's still an imperfect capture strategy. Let's look at what happens when we output some text to standard out from within a subprocess:

require 'stringio'

def capture_output
  fake_stdout = StringIO.new
  old_stdout = $stdout
  $stdout = fake_stdout
  Object.const_set(:STDOUT,fake_stdout) # !> already initialized constant STDOUT
  yield
ensure
  $stdout = old_stdout
  Object.const_set(:STDOUT,old_stdout) # !> already initialized constant STDOUT
  return fake_stdout.string
end

output = capture_output do
  puts "This goes to standard out"
  STDOUT.puts "This tries to bypass capture"
  system 'echo "output from a subprocess"'
end
output
# => "This goes to standard out\nThis tries to bypass capture\n"
# >> output from a subprocess

Subprocesses have no knowledge of Ruby variables. They just use the original process standard output and error streams. There are ways to explicitly redirect subprocess output in Ruby, but that means every subprocess invocation has to be modified. That doesn't really help us construct a general purpose output-capture method.

Instead, let's try a different tactic. This time, instead of constructing a StringIO object, we'll construct an operating system-level pipe. The IO.pipe method returns two objects: the read end and the write end of the newly-created pipe. We turn on sync mode on the read end of the pipe: this will ensure that data is immediately available from it, rather than having to wait until some operating system buffer is full.

We save a clone of $stdout, so that we can restore it again later. Then we use the #reopen method on the $stdout stream to change $stdout in-place to point to the pipe's write end. Now anything written to standard output, regardless of whether it uses the $stdout global or the STDOUT constant, will go into our new pipe instead.

Now we need to arrange get data back out of the pipe. First, we define a string to act as an output buffer. Then we kick off a very simple thread. Its job will be to continually try to read data from the pipe, and append that data to our output buffer. readpartial will read as many bytes as it can, up to the given maximum. If there is no data available it will block until there is.

We then yield to the given block. Once it has finished, we close the read end of the pipe. This will cause an EOFError to be raised in our reader thread, terminating it. We wait until the thread is done shutting down, then restore the $stdout stream to its original state using another #reopen. Finally, we returned the captured buffer.

When we test this new version out, we can see that it captures even output generated from within a subprocess.

def capture_output
  old_stdout = STDOUT.clone
  pipe_r, pipe_w = IO.pipe
  pipe_r.sync    = true
  output         = ""
  reader = Thread.new do
    begin
      loop do
        output << pipe_r.readpartial(1024)
      end
    rescue EOFError
    end
  end
  STDOUT.reopen(pipe_w)
  yield
ensure
  STDOUT.reopen(old_stdout)
  pipe_w.close
  reader.join
  return output
end

output = capture_output do
  puts "This goes to standard out"
  STDOUT.puts "This tries to bypass capture"
  system 'echo "output from a subprocess"'
end
output.split("\n")
# => ["This goes to standard out",
#     "This tries to bypass capture",
#     "output from a subprocess"]

It's easy to make this method capture standard error as well; easy enough that I'll leave it as an exercise to the viewer.

We now have a truly robust method for capturing program output to a string, as well as, hopefully, a better understanding of how Ruby handles standard output streams. Until next time, happy hacking!