In Progress
Unit 1, Lesson 1
In Progress

Using Sidekiq Middleware

Sidekiq is one of the most popular background job queueing systems for Ruby and Rails projects. Today, Sidekiq’s creator Mike Perham is joining us in the RubyTapas kitchen. He’s going to show us how to use Sidekiq middlewares to capture and recreate job context between the point where a job is *queued* and the point where it is *executed*. Enjoy!

Video transcript & code

Using Sidekiq Middleware

Hi, I’m Mike Perham, author of Sidekiq, the most popular background job system for Ruby and Rails. In this video, I want to show you a Sidekiq API you can use to “level up” your Sidekiq jobs. If you like Sidekiq, be sure to check out Sidekiq Pro and Sidekiq Enterprise for even more useful features for your background jobs.

class SomeWorker
include Sidekiq::Worker

def perform(an_argument, another)
# do some work here
end
end

In Sidekiq, a background job is a Ruby class which includes Sidekiq::Worker and has a perform method which takes a set of arguments.

Within the perform method, your worker can call any code it wants but often times, Rails applications will require some global state to execute, for instance a current user.

class User < ActiveRecord::Base
def self.current_id=(val)
Thread.current[:user_id] = val
Thread.current[:user] = nil
end
def self.current_id
Thread.current[:user_id]
end
def self.current
Thread.current[:user] ||= begin
raise "No user set" unless current_id
User.find(current_id)
end
end
end

Assume we have a thread-safe API that tracks the current user, with methods for getting and setting the current User. We’ll use it in our app for authorization and auditing purposes.

class ApplicationController < ActionController::Base
before_action :set_current_user
def set_current_user
User.current_id = cookies.signed[:user_id]
end
end

In a Rails app, you’d typically use a global before_action callback to set the current user based on a signed cookie that is set by your login page. But what if you create a background job on behalf of that user?

class SomeWorker
include Sidekiq::Worker

def perform(string)
if User.current.admin?
# do some admin thing
puts "ADMIN: #{string}"
else
# do some normal user thing
puts "NORMAL: #{string}"
end
end
end

Here we have a Sidekiq job which behaves differently for an administrator user.

For security or auditing purposes we will want to know who this code is running on behalf of. Remember, a background job can run in a different process or even different machine. How do we transfer and set up that context so the current user is set when the job executes in Sidekiq? In essence, we want to record the current user when the job is created and restore that current user before the job is executed.

That’s where Sidekiq middleware comes in. Middleware provides two extension points: one right before the Sidekiq client API enqueues a job and the other when the Sidekiq server process executes a job.

def create
# client-side middleware will execute “inside” this call
SomeWorker.perform_async("abc")
end

Our Rails controller action creates a job by calling perform_async on the worker class.

Client-side middleware is a Ruby class with a call method.

Sidekiq will invoke the call method with a few parameters, one of which is a hash of data which represents the job.

We can add a user_id element to the hash before it is enqueued.

Just remember all elements of the job hash must be legal JSON types, string, integer, boolean, etc and not complex Ruby objects. Our user_id is an integer so it’s a legal JSON type.

The call method must yield to continue the job enqueue, not yielding will stop the enqueue process.

class UserClient
def call(worker, job, queue, redis_pool=nil)
job["user_id"] = User.current_id
yield # enqueue the job
end
end

Server-side middleware is almost identical to client-side middleware: a Ruby class with a call method.

The middleware will establish some thread-local data...

yield to execute the job...

and then clean up that data. After yielding, we use an ensure block to unset the user so future jobs don’t accidentally reuse this user, even if the job raises an exception.

class UserServer
def call(worker, job, queue)
User.current_id = job["user_id"]
yield # execute the job
ensure
User.current_id = nil
end
end

Finally we’ll use Sidekiq’s configuration APIs to register our middleware. Normally this snippet will go in your Rails initializer. The configure_client block executes in any non-server process, like rake, puma, or passenger.

We’ll call client_middleware and add our new class to the middleware chain.

# any process that is not a Sidekiq process
Sidekiq.configure_client do |config|
config.client_middleware do |chain|
chain.add UserClient
end
end

Likewise, the configure_server block configures the Sidekiq process itself.

We’ll call server_middleware and add our class to the middleware chain.

We configure the client middleware in configure_server also.

# the Sidekiq process
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add UserServer
end
config.client_middleware do |chain|
chain.add UserClient
end
end

This is because jobs can create other jobs — in that case the server process also acts as a client; we want the user_id to transfer to those jobs also.

That’s it. Sidekiq middleware is often only a line of two of actual code but it’s a really powerful tool to handle concerns that span across all your application’s background jobs like authorization, logging, auditing, profiling, and caching. Have more questions about Sidekiq? Visit the Sidekiq wiki for lots of documentation.

Good luck!

Responses