Ruby Spelunking

I often find myself working with code that surprises me. Whether it be an complex gem or my own code from six months ago, one of my favorite approaches to getting unstuck is to use method introspection to go spelunking.

Spelunking is the act of jumping from method to method using method introspection, reading the source code as you go. Much like real spelunking, spelunking through Ruby code can lead us to dark and exciting places. While documentation may be inaccurate or incomplete, source code reveals exactly what happens when you call a method.

Before we get started, let's learn a few things about method introspection to make the trip a success.

We have a single method called foo that returns the string "Hello".

Using the method method, we can get the method object for foo.

If we call source_location, we can see the file and line number where the method was defined.

A natural next step is call source but when we do, we get a NoMethodError. Thankfully, a gem called method_source provides the exact functionality we want. Let's include it and try again.

Nice! It returns a string of the method definition. To make it more readable, let's add a call to Object#display, which prints the string to STDOUT and returns nil. Now we see the method with the same formatting as we used in the code.

require "method_source"

def foo
  "Hello!"
end

method(:foo).source.display # => nil

# >> def foo
# >>   "Hello!"
# >> end

There's one other trick we need to learn before going spelunking. Let's wrap foo in a class and add a subclass that overrides the foo method and calls super.

Now, let's update our code to get the foo method object from the child. It returns the source of the Child#foo method but how do we follow the code path up into the super method? We can use the super_method method.

Going back to the foo method object, we can see it is defined on Child.

If we add super_method, it now returns a method defined on Parent.

And now we can get the source code from the super method.

require "method_source"

class Parent
  def foo
    "Hello!"
  end
end

class Child < Parent def foo super.reverse end end Child.new.method(:foo).super_method.source.display # => nil

# >>   def foo
# >>     "Hello!"
# >>   end

Okay, now we're ready to go. Now, it's not very interesting to go spelunking in a one-room cave, so let's step into some code that is deeply layered. Let's look at activerecord.

We have an example Rails app with a Book model and while writing code, we noticed some oddities with the count, size, and length methods.

All three return the same value but the underlying queries are different. count and size run SELECT COUNT(*) while length runs a SELECT *.

To make things more confusing, the queries change when we reorder the method calls, moving size last. Now we only see two queries. Why is that?

require_relative "./example-app"

books = Book.all
books.count # => 2
books.length # => 2
books.size # => 2

# >>    (0.1ms)  SELECT COUNT(*) FROM "books"
# >>   Book Load (0.1ms)  SELECT "books".* FROM "books"

Let's investigate count first by getting the count method. Here we can see it is defined by the ActiveRecord::Calculations module.

require_relative "./example-app"

books = Book.all
books.method(:count) # => #

Displaying the source, we can see it takes a column name and when there is a block, it does something special, otherwise it calls calculate.

require_relative "./example-app"

books = Book.all
books.method(:count).source.display # => nil

# >>     def count(column_name = nil)
# >>       if block_given?
# >>         unless column_name.nil?
# >>           ActiveSupport::Deprecation.warn \
# >>             "When `count' is called with a block, it ignores other arguments. " \
# >>             "This behavior is now deprecated and will result in an ArgumentError in Rails 6.0."
# >>         end
# >>
# >>         return super()
# >>       end
# >>
# >>       calculate(:count, column_name)
# >>     end

Looking at calculate next, it takes an operation and a column name and does some work. In our case, we're not using any includes so it calls perform_calculation.

require_relative "./example-app"

books = Book.all
books.method(:calculate).source.display # => nil

# >>     def calculate(operation, column_name)
# >>       if has_include?(column_name)
# >>         relation = apply_join_dependency
# >>
# >>         if operation.to_s.downcase == "count"
# >>           relation.distinct!
# >>           # PostgreSQL: ORDER BY expressions must appear in SELECT list when using DISTINCT
# >>           if (column_name == :all || column_name.nil?) && select_values.empty?
# >>             relation.order_values = []
# >>           end
# >>         end
# >>
# >>         relation.calculate(operation, column_name)
# >>       else
# >>         perform_calculation(operation, column_name)
# >>       end
# >>     end

Here's where things start to get complex, with some special cases for count and distinct operations. Ignoring the bulk of the method, let's dive one level deeper into execute_simple_calculation.

require_relative "./example-app"

books = Book.all
books.method(:perform_calculation).source.display # => nil

# >>       def perform_calculation(operation, column_name)
# >>         operation = operation.to_s.downcase
# >>
# >>         # If #count is used with #distinct (i.e. `relation.distinct.count`) it is
# >>         # considered distinct.
# >>         distinct = distinct_value
# >>
# >>         if operation == "count"
# >>           column_name ||= select_for_count
# >>           if column_name == :all
# >>             if distinct && (group_values.any? || select_values.empty? && order_values.empty?)
# >>               column_name = primary_key
# >>             end
# >>           elsif column_name =~ /\s*DISTINCT[\s(]+/i
# >>             distinct = nil
# >>           end
# >>         end
# >>
# >>         if group_values.any?
# >>           execute_grouped_calculation(operation, column_name, distinct)
# >>         else
# >>           execute_simple_calculation(operation, column_name, distinct)
# >>         end
# >>       end

It's a big one but, finally, four levels down we've found the code that generates and executes the query using select_all. We might not know the ins-and-outs of these code paths but that's okay, we now understand enough to go back and look at how length is different.

require_relative "./example-app"

books = Book.all
books.method(:execute_simple_calculation).source.display # => nil

# >>       def execute_simple_calculation(operation, column_name, distinct) #:nodoc:
# >>         column_alias = column_name
# >>
# >>         if operation == "count" && (column_name == :all && distinct || has_limit_or_offset?)
# >>           # Shortcut when limit is zero.
# >>           return 0 if limit_value == 0
# >>
# >>           query_builder = build_count_subquery(spawn, column_name, distinct)
# >>         else
# >>           # PostgreSQL doesn't like ORDER BY when there are no GROUP BY
# >>           relation = unscope(:order).distinct!(false)
# >>
# >>           column = aggregate_column(column_name)
# >>
# >>           select_value = operation_over_aggregate_column(column, operation, distinct)
# >>           if operation == "sum" && distinct
# >>             select_value.distinct = true
# >>           end
# >>
# >>           column_alias = select_value.alias
# >>           column_alias ||= @klass.connection.column_name_for_operation(operation, select_value)
# >>           relation.select_values = [select_value]
# >>
# >>           query_builder = relation.arel
# >>         end
# >>
# >>         result = skip_query_cache_if_necessary { @klass.connection.select_all(query_builder, nil) }
# >>         row    = result.first
# >>         value  = row && row.values.first
# >>         type   = result.column_types.fetch(column_alias) do
# >>           type_for(column_name)
# >>         end
# >>
# >>         type_cast_calculated_value(value, type, operation)
# >>       end

So, let's grab the length method source. Immediately, we can see a difference -- instead of a normal def keyword, we see a call to delegate, which forwards a method to another object, in this case the records object.

require_relative "./example-app"

books = Book.all
books.method(:length).source.display # => nil

# >>     delegate :to_xml, :encode_with, :length, :each, :uniq, :join,
# >>              :[], :&, :|, :+, :-, :sample, :reverse, :rotate, :compact, :in_groups, :in_groups_of,
# >>              :to_sentence, :to_formatted_s, :as_json,
# >>              :shuffle, :split, :slice, :index, :rindex, to: :records

If we look at the records method, we can see it loads data from the database then returns the records.

require_relative "./example-app"

books = Book.all
books.method(:records).source.display # => nil

# >>     def records # :nodoc:
# >>       load
# >>       @records
# >>     end

So books.length is equivalent to calling books.records.length. Okay, makes sense! Even though the method names are similar between count and length, the code paths are wildly different.

require_relative "./example-app"

books = Book.all
books.records.length # => 2

# >>   Book Load (0.1ms)  SELECT "books".* FROM "books"

Now what about the oddity with size? Why did the query disappear if size was called after length? Once we take a peek at the source, it becomes obvious: when the records are loaded, return the length, otherwise execute a count. Because calling length loads the records, it also has the side effect of changing how size behaves.

require_relative "./example-app"

books = Book.all
books.method(:size).source.display # => nil

# >>     def size
# >>       loaded? ? @records.length : count(:all)
# >>     end

So that's a quick look at how method introspection, or spelunking, can help you debug problems or explore libraries you're unfamiliar with. Thanks for watching!