Background Processing With Resque – –

View Github Repository Open presentation in a new window

vongrippen

See all presentation from vongrippen

Background Processing With Resque – –

0 0

memrb-2013-04-resque

On Github vongrippen / memrb-2013-04-resque

Background Processing With Resque

Mike Cochran

@vongrippen

http://vongrippen.com/memrb-2013-04-resque

Quick Info on Resque

  • Built by Github
    • https://github.com/blog/542-introducing-resque
  • Uses Redis
    • Fast key-value store
  • Pure Ruby, Rails not required
  • Named queues
  • Queued data is pure JSON
  • Simple, easy to use
    • Create a class with a "perform" method.That's pretty much it.
  • Uses process forks for workers
    • Avoids the GIL in MRI Ruby
  • Bundled Web UI

the Problem

Simulating an investment portfolio

(Average Case)
  • Invest for 30 years
  • Rebalance investments after each year
  • Simulate 205 times, incrementing the starting quarter
    • Start in Dec 1932, invest for 30 years
    • Start in Mar 1933, invest for 30 years
    • Start in Jun 1933, invest for 30 years
    • ... etc
  • Save each simulation (called a "period")
  • Collect the summary of each as a "Case Result"

Without Resque

Controller
                def create
  client_case.input = filtered_params
  client_case.calculate
  # 3 minutes later...
  client.save
  # Render HTML
end

The result?

Timeout Error

Here's the Slow down

Model 
                    def calculate
  periods = 0..(periods_to_run-1)
  summaries = [ ]
  periods.each do |period|
    # Do a bunch of fun math
    years_in_period = calculate_period(input, period)
    self.save_period(period, years_in_period)
    summaries << years_in_period.last
  end
  self.summary = summarize(summaries)
end

Let's Add Resque

With Resque

Controller
                    def create
  client_case.input = filtered_params
  client_case.save
  Resque.enqueue(CaseWorker, client_case.id)
  # Render HTML
end
CaseWorker
                class CaseWorker
  @queue = :cases

  def self.perform(case_id)
    client_case = Case.find(case_id)
    client_case.calculate
    client_case.save
  end
end

The result?

No Timeout!
(But it still takes 3 minutes to calculate)

Here's the Slow down (Still)

Model 
                    def calculate
  periods = 0..(periods_to_run-1)
  summaries = [ ]
  periods.each do |period|
    # Do a bunch of fun math
    years_in_period = calculate_period(input, period)
    self.save_period(period, years_in_period)
    summaries << years_in_period.last
  end
  self.summary = summarize(summaries)
end

Let's Add (More) Resque

With (More) Resque

Controller
                        def create
  client_case.input = filtered_params
  client_case.save
  (0..client_case.periods_to_run-1).each do |period|
    Resque.enqueue(PeriodWorker, case_id, period)
  end
  # Render HTML
end
CaseWorker
                    class CaseWorker
  @queue = :cases
  def self.perform(case_id)
    client_case = Case.find(case_id)
    client_case.summary = summarize(client_case.summaries)
    client_case.save
  end
end

With (More) Resque

PeriodWorker
                    class PeriodWorker
  @queue = :periods

  def self.perform(case_id, period)
    client_case = Case.find(case_id)
    years_in_period = calculate_period(input, period)
    client_case.save_period(period, years_in_period)
    client_case.summaries << years_in_period.last
    client_case.save
    if client_case.periods == client_case.periods_in_case
      Resque.enqueue(CaseWorker, case_id)
    end
  end 
end

The result?

No Timeout!
Plus, I can now throw more machines at it.
With 30 "workers" running, 25 seconds.

What did we just do exactly?

  • We separated the calculation from our web app itself
  • We broke down our code into smaller bits
  • We ensured that our code could be run in parallel on multiple machines

Workers run

Anywhere,Anytime

Run as manyas you want

Q & A

Resources

Project:
https://github.com/resque/resque
Screencast:
http://railscasts.com/episodes/271-resque

Sidekiq

sidekiq.org
  • Uses threads instead of forks
  • Easily extensible
  • Has a lot of extra functionality out of the box
  • Only loads a single copy of Rails in memory (if using Rails)
  • Smaller footprint

CAVEATS

  • You code must be thread-safe
  • Any gem you use must be thread-safe
  • More processor intensive