Explore and Extend – Part 2
In the previous episode, we learned how to extend ActiveJob serialization
to add a trace UUID to every serialized job using method introspection techniques to peek under the surface and work with the code beneath. Today we’ll be looking at how to work with code that wasn’t designed to be extended by adding the trace UUID to deserialization.
Video transcript & code
(Part 1)
Part 2: Responsible Monkey-Patching
This time, we don’t have a natural starting place because we don’t ever pull things from the queue—that’s Sidekiq’s job! Instead of using the console, let’s go read some code on GitHub and see if we can find some clues.
ActiveJob is a gem that lives on GitHub at rails/rails
. Let’s start by looking at the queue adapter for Sidekiq.
Some of this code looks familiar—we already saw the #enqueue
method while we were spelunking. Aside from the enqueue methods, the only other bit is a method called perform
, which makes a call to ActiveJob::Base.execute
. That looks promising so let’s hop back to our console and take a look.
ActiveJob::Base.method(:execute).source.display
# def execute(job_data) #:nodoc:
# job = deserialize(job_data)
# job.perform_now
# end
# => nil
It does two things: pass job_data
into a method called deserialize
and then call perform_now
on the job that’s returned.
Looking at deserialize
, we can see the serialized values being assigned to attributes on the newly created job instance.
ActiveJob::Base.method(:deserialize).source.display
# def deserialize(job_data)
# job = job_data['job_class'].constantize.new
# job.job_id = job_data['job_id']
# job.queue_name = job_data['queue_name']
# job.serialized_arguments = job_data['arguments']
# job.locale = job_data['locale'] || I18n.locale
# job
# end
# => nil
So, we’ve made some progress but there’s a problem. Deserialization is done by calling ActiveJob::Base.deserialize
so in order to override it in our jobs, we’d need to monkey patch ActiveJob::Base
. It would be much easier to extend if it worked like serialization and we could simply implement ApplicationJob#deserialize
but it looks like we’ll need to take a more aggressive approach.
Before we do, now is a good time to pause, take a breathe, and check Rails master. Remember, we’re running Rails 4.2 app and there’s a chance that someone else hit the same issue as us. It may have even been solved in a future version of Rails so let’s take a look at the deserialize
method in Rails master.
Using source_location
, we can see that it's defined in lib/active_job/core.rb
so let’s start by looking there.
ActiveJob::Base.method(:deserialize).source_location
# => ["activejob-4.2.10/lib/active_job/core.rb", 27]
def deserialize(job_data)
job = job_data["job_class"].constantize.new
job.deserialize(job_data)
job
end
In a wonderful bit of luck, it looks like someone solved this for us! Now, ActiveJob::Base.deserialize
delegates to the job via the #deserialize
instance method. Even though we’re using Rails 4.2, we can backport this behaviour using a monkey patch and start using it immediately. This helps prepare us for the future so when it comes time to upgrade, our code will already work with the newer version of Rails.
Let’s add an initializer that re-opens ActiveJob::Base
and redefine the deserialize
method. This is the most crude form of monkey patching but it gets the job done. Let's copy/paste the original Rails 4.2 implementation into the method and then, after the original deserialization happens, we’ll add a call to #deserialize
.
class ActiveJob::Base
def self.deserialize(job_data)
job = job_data['job_class'].constantize.new
job.job_id = job_data['job_id']
job.queue_name = job_data['queue_name']
job.serialized_arguments = job_data['arguments']
job.locale = job_data['locale'] || I18n.locale
job.deserialize(job_data)
job
end
end
Let's try running the job, first turning on inline processing to ensure Sidekiq serializes and deserialized the job completely. Oops! There's a problem. Our jobs don't implement deserialize
yet so...
require 'sidekiq/testing'; Sidekiq::Testing.inline!
MyJob.perform_later
# NoMethodError: undefined method `deserialize' for #
# Did you mean? serialize
# config/initializers/extend_active_job.rb:9:in `deserialize'
...let's tweak our patch to only call deserialize
if the job responds to it.
class ActiveJob::Base
def self.deserialize(job_data)
job = job_data['job_class'].constantize.new
job.job_id = job_data['job_id']
job.queue_name = job_data['queue_name']
job.serialized_arguments = job_data['arguments']
job.locale = job_data['locale'] || I18n.locale
job.deserialize(job_data) if job.respond_to?(:deserialize)
job
end
end
require 'sidekiq/testing'; Sidekiq::Testing.inline!
MyJob.perform_later
# TRACE UUID: nil
# => nil
Better! Now we have a working backport of the Rails master behaviour...
...and we can go ahead and implement ApplicationJob#deserialize
to take job_data
and assign trace_uuid
.
class ApplicationJob < ActiveJob::Base attr_accessor :trace_uuid def serialize super.merge("trace_uuid" => trace_uuid || Trace.uuid)
end
def deserialize(job_data)
self.trace_uuid = job_data["trace_uuid"]
end
end
require 'sidekiq/testing'; Sidekiq::Testing.inline!
Trace.uuid = "abcdef"
MyJob.perform_later
# TRACE UUID: "abcdef"
# => #
This time, it works! Our job serializes the trace UUID and then restores it during deserialization. We’re done!
Or, are we?
Remember, when we copied and pasted code into the patch? Aside from making our patch longer and more complex, it also means we take full responsibility of implementing the original behaviour. If we upgrade Rails and the implementation changes, our code won’t adapt.
What if we swap it out for a call to super
instead?
class ActiveJob::Base
def self.deserialize(job_data)
job = super
job.deserialize(job_data) if job.respond_to?(:deserialize)
job
end
end
Let’s check to make sure everything works as expected—and it does.
require 'sidekiq/testing'; Sidekiq::Testing.inline!
Trace.uuid = "abcdef"
MyJob.perform_later
# TRACE UUID: abcdef
# => #
This looks a lot better: it's small and delegates to the original implementation.
But before we call it a day, there’s one more important piece: this monkey patch has an expiry date. We only need it so long as we’re using an old version of Rails. To make sure we remember to remove it, let's raise an exception when running a new version of Rails.
First, let’s find out when the new behaviour was added by going back to GitHub and looking at the git blame for the specific commit that added the change. Once on the commit, we can see all of the tags associated with this commit and see it was first added to the Rails 5.0 beta.
And now, if we're running Rails 5.0 or newer, we can raise an exception by using the Gem::Version
object to do a version comparison.
if Gem::Version.new(Rails.version) >= Gem::Version.new("5.0")
raise "Remove this file! You don't need it anymore!"
end
class ActiveJob::Base
def self.deserialize(job_data)
job = super
job.deserialize(job_data) if job.respond_to?(:deserialize)
job
end
end
That’s a look at how you can use spelunking along with GitHub to learn more about the gems you use and how to carefully and responsibly patch and extend code.
Responses