Task Management with Celery And Django

Claire Gowler

About Me

• @kitation
• Django developer for 2 years at Zoo Digital in Sheffield
• We make configurable workflow management web applications

Outline

• Why Celery?
• What is Celery?
• Getting Started
• Tips and Tricks

The Problem

What is Celery?

• Distributed task management system written in Python
• Django integration as standard
• Supports many messaging brokers and result backends
• Wide range of features inc. queues, grouping and webhooks
• Easy to check and get results

Setup

Making a Task

#celery.py
app = Celery("celery_test")
app.config_from_object("django.conf:settings")
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

#tasks.py
from models import Thing, SubThing
from celery_test.celery import app

@app.task
def make_things(number):
    s = SubThing.objects.create(number=SubThing.objects.count() + 1)
    t = Thing(name=str(number), subthing=s)
    t.save()
    return t.id

Calling a Task

from my_app.tasks import make_things

result = make_things.delay(1)

result = make_things.apply_async(
    args=(1,),
    task_id="myrandomtaskid"
)

result = make_things.apply_async(
    (1,),
    link=my_callback.s()
)

Calling a Group of Tasks

from my_app.tasks import make_things
from celery import group

tasks = [make_things.s(i) for i in xrange(5)]
group_result = group(tasks).apply_async()

Result!

result.status
>>> "PENDING"
result.status
>>> "SUCCESS"
result.ready()
>>> True

result.get()
>>> 1

group_result.completed_count()
>>> 3
group_result.waiting()
>>> True

Tips and Tricks

• Check out Flower, a 3rd party monitoring webapp
• Serialization defaults to Pickle, change to JSON
• --autoreload is unreliable
• Everything is pending, but you know your IDs
• Result retrieval is inconsistent
• Point Celery to the app with settings.py

Thank You

@kitation