Enqueue Multiple Jobs Together with perform_all

Sometimes, you have a large number of jobs to dispatch. Maybe you're sending hundreds of thousands of emails at once, or processing a large file containing thousands of records and dispatching a job for each.

Agreed, you're already using a background job, so your end users don't have to wait for the whole job to finish. However, enqueuing each job still takes some work, right? You loop over each job and enqueue it individually. That means your application has to talk to the job backend (database, Redis, etc.) once for each job.

Wouldn't it be nice if there was a method that could let you enqueue all the jobs together, all in one go, without the overhead of multiple round-trips to the queue datastore?

Turns out, there is!

Rails 7.1 shipped with a handy perform_all_later method that lets you enqueue multiple jobs to be executed at once.

At this time, as far as I know, it's only properly supported by Sidekiq and GoodJob.

Sidekiq's push_bulk method cuts down on Redis round-trip latency by pushing multiple jobs with Redis' lpush command.
GoodJob's bulk-enqueue feature buffers and enqueues multiple jobs at once using a single INSERT statement on the PostgreSQL database.

For other backends like Delayed Job and Resque it still loops over the jobs.

However, the nice thing about using perform_all_later is, as other job backends implement this functionality (here is Ben Sheldon, creator of GoogJob trying to implement it in recently released Solid Queue ), your code won't have to change if you're using this method.

Note: This method won't trigger any callbacks for each job. However, it fires a enqueue_all.active_job event which you can subscribe to and get the adapter name and the list of jobs to be enqueued.

To learn more about the instrumentation API in Rails, check out the following article.

Queue adapters can communicate the enqueue status of each job by setting successfully_enqueued and/or enqueue_error on the passed-in job instances.

Since you have access to the job classes in the event subscriber payload, you can query which jobs were enqueued successfully using the successfully_enqueued? method on the job.

enqueued_jobs = jobs.select(&:successfully_enqueued?)

Sidekiq has already supported this functionality for a while now, first with Sidekiq::Client.push_bulk method and later with a high-level wrapper method called perform_bulk, which you can directly call on your job class.

How It Works

The perform_all_later method accepts an array of job instances. You can use it as follows:

reminder_jobs = users.map do |user|
  ReminderJob.new(user).set(wait: 1.day)
end

ActiveJob.perform_all_later(reminder_jobs)

Here's a highly simplified implementation of the perform_all_later method.

# lib/active_job/enqueuing.rb

def perform_all_later(*jobs)
  if queue_adapter.respond_to?(:enqueue_all)
    queue_adapter.enqueue_all(jobs)
  else
    jobs.each do |job|
      queue_adapter.enqueue(job)
    end        
  end
end

As you can see, it first checks if the queue adapter, i.e. Resque, Sidekiq, etc. responds to the enqueue_all method. If yes, then it calls enqueue_all passing the list of jobs. Otherwise, it simply loops over each job and enqueues it individually.

Now let's see what a sample enqueue_all implementation does. We'll look at the Sidekiq adapter.

# lib/active_job/queue_adapters/sidekiq_adapter.rb

def def enqueue_all(jobs)
  Sidekiq::Client.push_bulk(
    "class" => JobWrapper,
    "wrapped" => job_class,
    "queue" => queue,
    "args" => jobs.map { |job| [job.serialize] },
  )
end

So behind the scenes, the enqueue_all uses the push_bulk feature in Sidekiq to dispatch all the jobs at once.

I couldn't stop myself from going one step further and taking a peek at the Sidekiq's implementation of the push_bulk method, which is quite interesting. But that's a topic for a separate post. So stay tuned!

P.S. If you'd like to learn more about how Active Job is implemented, check out the following post 👇

That's a wrap. I hope you found this article helpful and you learned something new.

As always, if you have any questions or feedback, didn't understand something, or found a mistake, please leave a comment below or send me an email. I reply to all emails I get from developers, and I look forward to hearing from you.

If you'd like to receive future articles directly in your email, please subscribe to my blog. If you're already a subscriber, thank you.