Introduce

Sooner or later, every Django project hits the same moment. A client asks you to recalculate prices for every product, backfill a new column, send a notification to every user, or re-index an entire table. You write a simple loop, run it on your laptop, and it works. Then you run it on production data with 5 million rows, and everything falls apart.

Memory climbs, the request times out, the database slows down for real users, and when the process finally dies halfway through, you have no idea which records were handled and which were not.

This guide walks through a practical, production-safe approach to processing millions of records with Django and Celery. The focus is on the design decisions that matter, not on tricks that only look fast in a demo.

Why the Naive Approach Fails

Here is the code most of us write first:

def recalculate_all():
    for order in Order.objects.all():
        order.total = calculate_total(order)
        order.save()

At small scale this is fine. At 5 million rows it breaks for several reasons:

• Memory: Order.objects.all() can load huge result sets into memory, and every model instance carries overhead.
• One query per save: 5 million save() calls means 5 million round trips to the database.
• A single point of failure: If the process crashes at record 3,200,000, you start over or guess where it stopped.
• No parallelism: One process handles everything while your other CPU cores and servers sit idle.
• Long transactions and locks: Large operations can hold locks and hurt the rest of your application.

The fix is not "make the loop faster". The fix is to change the shape of the work.

The Core Idea: Split the Work into Small, Independent Chunks

Instead of one giant job, create thousands of small jobs. Each Celery task handles a slice of the data, for example 2,000 records, and any task can be retried on its own without affecting the others.

The architecture has three parts:

1. A dispatcher task that figures out the ranges of work and queues the chunk tasks.
2. Chunk worker tasks that each process one range.
3. Tracking and monitoring so you know what finished, what failed, and what needs a retry.

Step 1: Chunk by ID Range, Not by OFFSET

The most common mistake is paginating with OFFSET. On a large table, OFFSET 4000000 forces PostgreSQL to scan and throw away four million rows before returning anything. Every later page gets slower.

Use keyset (range) pagination on an indexed column, usually the primary key. Each chunk becomes a simple, fast range query:

# tasks.py
from celery import shared_task
from django.db.models import Min, Max
from .models import Order

CHUNK_SIZE = 2000

@shared_task
def dispatch_order_recalculation():
    bounds = Order.objects.aggregate(lo=Min("id"), hi=Max("id"))
    if bounds["lo"] is None:
        return 0

    start = bounds["lo"]
    queued = 0
    while start <= bounds["hi"]:
        end = start + CHUNK_SIZE - 1
        process_order_chunk.delay(start, end)
        start = end + 1
        queued += 1
    return queued

The dispatcher does almost no work. It queues small messages containing two integers each, which is cheap even for thousands of tasks.

If your IDs have large gaps because of deletions, some chunks will be nearly empty. That is harmless. If the gaps are extreme, you can instead fetch the next batch of IDs with filter(id__gt=last_id).order_by("id")[:CHUNK_SIZE] and chain tasks.

Step 2: Process Each Chunk with Bulk Operations

Inside each task, avoid per-row queries. Fetch only the fields you need, calculate in memory, and write back in a single batch:

@shared_task(bind=True, max_retries=3, acks_late=True)
def process_order_chunk(self, start_id, end_id):
    try:
        orders = list(
            Order.objects.filter(id__gte=start_id, id__lte=end_id)
            .only("id", "subtotal", "tax_rate", "discount")
        )

        for order in orders:
            order.total = calculate_total(order)

        Order.objects.bulk_update(orders, ["total"], batch_size=500)
        return len(orders)

    except Exception as exc:
        raise self.retry(exc=exc, countdown=30 * (self.request.retries + 1))

A few details matter here:

• .only() reduces the data transferred and the memory used per object.
• bulk_update() replaces thousands of individual UPDATE statements with a few batched ones. Use bulk_create() the same way for inserts.
• batch_size keeps each SQL statement a reasonable size.
• .iterator() is also worth knowing when a single chunk is large, because it streams rows instead of loading everything at once.
• Remember what bulk_update() skips: it does not call save(), so signals and auto_now fields are not triggered. Set updated_at yourself if you need it.

Step 3: Make Every Task Idempotent

This is the rule that separates reliable pipelines from fragile ones: running a task twice must produce the same result as running it once.

Celery gives you at-least-once delivery in many setups, so a task can run again after a worker crash, a retry, or a network hiccup. If your task does balance += 10, a duplicate run corrupts data. If it does total = calculate_total(order), a duplicate run is harmless.

Practical ways to stay idempotent:

• Compute values from source data instead of incrementing existing values.
• Use update_or_create() or bulk_create(..., update_conflicts=True, unique_fields=[...], update_fields=[...]) for upserts.
• For external side effects like emails or payments, store a "processed" flag or an idempotency key and check it before acting.

Step 4: Tune Celery for Long, Heavy Workloads

Default Celery settings are tuned for many small, quick tasks. Bulk processing usually needs a few adjustments:

# settings.py
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_TASK_TIME_LIMIT = 600
CELERY_TASK_SOFT_TIME_LIMIT = 540

• acks_late acknowledges a task only after it finishes, so a crashed worker's task goes back to the queue.
• prefetch_multiplier = 1 stops one worker from hoarding a pile of tasks while others sit idle.
• Time limits prevent a stuck task from blocking a worker forever.

Also use a dedicated queue for bulk work so it never starves your time-sensitive tasks such as password reset emails:

@shared_task(queue="bulk")
def process_order_chunk(...):
    ...

celery -A config worker -Q bulk -c 8 --loglevel=info

Start workers on this queue separately from your default workers. That way a 5 million row backfill does not delay anything a customer is waiting for.

Step 5: Protect Your Database

Parallelism is powerful, and it can also overwhelm PostgreSQL. Twenty workers each running heavy updates can saturate connections, I/O, or locks.

• Limit concurrency on the bulk queue. More workers is not always faster, and the database is usually the bottleneck.
• Keep transactions short. One chunk per transaction is a good default. Avoid wrapping the entire job in one transaction.
• Watch connection counts. Each Celery process opens its own connections. A connection pooler such as PgBouncer helps when you scale workers.
• Run during quiet hours when possible, or throttle by dispatching chunks gradually.
• Check indexes. The columns used in your filter() must be indexed, or each chunk query becomes a slow scan.

Step 6: Track Progress and Handle Failures

For a job that runs for hours, "it seems to be working" is not good enough. Track it:

class BulkJob(models.Model):
    name = models.CharField(max_length=100)
    total_chunks = models.PositiveIntegerField(default=0)
    completed_chunks = models.PositiveIntegerField(default=0)
    failed_chunks = models.PositiveIntegerField(default=0)
    created_at = models.DateTimeField(auto_now_add=True)

Increment the counters atomically with F() expressions at the end of each chunk:

from django.db.models import F

BulkJob.objects.filter(pk=job_id).update(completed_chunks=F("completed_chunks") + 1)

Now you can build a simple admin view or dashboard showing "1,840 of 2,500 chunks complete". If a chunk fails after all retries, log its ID range so you can requeue just that slice instead of the entire job.

For visibility into the workers themselves, Flower gives you a live view of queues, task rates, and failures. Pair it with error tracking such as Sentry so exceptions do not disappear silently.

Common Mistakes to Avoid

1. Passing model objects to tasks. Pass IDs or ranges, never instances. Serialized objects can be stale or huge.
2. Using OFFSET pagination on big tables.
3. Calling .save() in a loop when bulk_update() or bulk_create() would do.
4. Skipping idempotency, then discovering duplicates after the first retry.
5. Running everything on the default queue and blocking user-facing tasks.
6. Chunks that are too small or too large. Tiny chunks create broker overhead; huge chunks lose the benefit of retries and parallelism. Start around 1,000 to 5,000 records and measure with your own data.
7. Skipping a dry run. Test the full pipeline on a copy or a small ID range before touching production.

Choosing a Chunk Size

There is no universal number. Chunk size depends on row width, the cost of your per-record logic, and database capacity. A reasonable approach is to run a few chunks at different sizes, such as 500, 2,000, and 5,000, record the time per chunk and the database load, and pick the size that keeps tasks in the range of a few seconds to a minute. Measure on your real data instead of trusting a number from a blog post, including this one.

A Production Checklist

Before running a large job in production, confirm that:

• The filter columns are indexed.
• Tasks are idempotent and safe to retry.
• Bulk work runs on its own queue with limited concurrency.
• Time limits and retries are configured.
• You have progress tracking and error alerts.
• You have tested on a small range and verified the results.
• You have a rollback or backup plan for data changes.

Conclusion

Processing 5 million records with Django and Celery is not about finding one magic optimization. It comes from a few consistent habits: split the work into small independent chunks, paginate by key instead of offset, write with bulk operations, make every task idempotent, tune Celery for long-running work, and protect the database that everything depends on.

Follow these steps and a job that once crashed your server becomes a predictable, restartable pipeline you can run with confidence, whether the dataset is 5 million rows or 50 million.

Frequently Asked Questions

How long does it take to process 5 million records with Celery? It depends on your per-record logic, row size, number of workers, and database capacity. Simple updates can finish in minutes with good bulk operations and parallel workers, while heavy logic or external API calls can take hours. Benchmark a sample chunk and extrapolate.

Is bulk_update faster than save() in Django? Yes, in most cases. bulk_update() groups many rows into a few SQL statements instead of one query per object. The trade-off is that it does not call save() or send signals.

What is the best chunk size for Celery tasks? A common starting range is 1,000 to 5,000 records per task. Adjust based on measured task duration and database load.

Should I use Celery or a management command for large jobs? A management command is fine for one-off, single-machine jobs. Celery is better when you need parallelism, automatic retries, progress tracking, or the ability to resume after a failure.

How do I avoid duplicate processing in Celery? Design tasks to be idempotent: compute values from source data, use upserts, and store processed flags or idempotency keys for side effects like emails or payments.