Reiteach Aigean Logo
Reiteach Aigeanblog.reiteachaigean.com
Admin
HomeAI UpdatesCodingCloudApps & ToolsTech NewsArticles
CodingCeleryDjangoRedis9 min read·9 reads

Celery: How to Prevent Duplicate Task Execution (Guide)

Celery tasks run twice and how to prevent duplicate task execution with idempotency, unique constraints, Redis locks, and idempotency keys in Django.

S
Sizan Mahmud
Published on November 16, 2026
Celery: How to Prevent Duplicate Task Execution (Guide)

Celery: How to Prevent Duplicate Task Execution

Few bugs are as frustrating as this one: a customer gets two confirmation emails, an invoice is generated twice, or a payout is sent two times. You look at your code and it seems correct. The task is only queued once. Yet somehow it ran twice.

The uncomfortable truth is that Celery does not promise that a task runs exactly once. In many setups, it promises closer to "at least once". Duplicates are a normal part of distributed systems, and the right response is to design for them instead of hoping they never happen.

This guide explains why Celery tasks run more than once and shows the practical techniques to prevent duplicate task execution, from the most reliable (idempotent design) to the most targeted (locks and idempotency keys).

Why Celery Tasks Run Twice

Before fixing duplicates, it helps to know where they come from.

1. Late acknowledgement and worker crashes. With acks_late=True, a task is acknowledged only after it finishes. If the worker dies halfway, the broker redelivers the task to another worker. This is a good feature for reliability, but it means a task can start twice, even if the first attempt had already done part of the work.

2. The Redis visibility timeout. When you use Redis as a broker, an unacknowledged message becomes visible again after the visibility_timeout (one hour by default). A task that runs longer than that, or one scheduled with a long countdown or eta, can be redelivered while the original is still running.

3. Retries. If a task raises an exception after sending an email but before finishing, the retry sends the email again.

4. Double submissions. A user double-clicks a button, a webhook is delivered twice, or a client retries a request after a timeout. Your view queues the same task two times, correctly from its point of view.

5. Overlapping periodic tasks. A Celery Beat task scheduled every minute can start a new run while the previous one is still going. If you run more than one Beat process by mistake, every scheduled task is queued multiple times.

Free tech newsletter

Free

Free to join — no card, no paywall. Weekly AI, cloud & tooling insights.

6. Queuing before the transaction commits. If you call .delay() inside a database transaction, the worker may start before the data is committed, fail, and retry, or the surrounding code may queue the task again.

Knowing the cause tells you which fix to use.

Fix 1: Make Tasks Idempotent (The Most Important Fix)

An idempotent task produces the same result whether it runs once or five times. This is the strongest protection, because it works no matter why the duplicate happened.

Compare these two tasks:

## Not idempotent: a duplicate run adds the bonus twice
@shared_task
def give_bonus(user_id):
    Wallet.objects.filter(user_id=user_id).update(balance=F("balance") + 10)

## Idempotent: a duplicate run produces the same final state
@shared_task
def recalculate_total(order_id):
    order = Order.objects.get(pk=order_id)
    order.total = calculate_total(order)
    order.save(update_fields=["total"])

Practical ways to get there:

  • Compute values from source data instead of incrementing existing values.
  • Use update_or_create() or get_or_create() instead of blind inserts.
  • Check the current state at the start of the task and exit early if the work is already done.
@shared_task
def send_welcome_email(user_id):
    user = User.objects.get(pk=user_id)
    if user.welcome_email_sent_at:
        return  # already done
    send_email(user)
    user.welcome_email_sent_at = timezone.now()
    user.save(update_fields=["welcome_email_sent_at"])

The check-then-act pattern above is not enough on its own when two copies run at the same moment, because both can pass the check before either saves. That is where the next fixes come in.

Fix 2: Let the Database Enforce Uniqueness

The database is the best referee, because it handles concurrent writers correctly. A unique constraint makes a duplicate insert fail instead of succeed silently.

class Payout(models.Model):
    order = models.OneToOneField("Order", on_delete=models.PROTECT)
    amount = models.DecimalField(max_digits=10, decimal_places=2)
    created_at = models.DateTimeField(auto_now_add=True)
from django.db import IntegrityError, transaction

@shared_task
def create_payout(order_id):
    order = Order.objects.get(pk=order_id)
    try:
        with transaction.atomic():
            Payout.objects.create(order=order, amount=order.total)
    except IntegrityError:
        return  # a payout already exists, so this run is a duplicate

For work that is not naturally tied to one row, create a small table of processed keys with a unique column and insert into it first. If the insert fails, someone else already did the work. For concurrent updates to the same row, select_for_update() inside transaction.atomic() makes the second task wait for the first to finish, and then it can see the updated state and exit.

Fix 3: Use a Distributed Lock for "Only One at a Time"

When you need to guarantee that only one worker runs a given piece of work at once, use a lock in Redis. This is useful for report generation, imports, and anything expensive or non-idempotent.

Free tech newsletter

Free

Free to join — no card, no paywall. Weekly AI, cloud & tooling insights.

import redis
from celery import shared_task
from django.conf import settings

redis_client = redis.Redis.from_url(settings.REDIS_URL)

@shared_task(bind=True)
def generate_invoice(self, order_id):
    lock = redis_client.lock(f"lock:invoice:{order_id}", timeout=300)
    if not lock.acquire(blocking=False):
        return "skipped: already running"
    try:
        build_invoice(order_id)
    finally:
        try:
            lock.release()
        except redis.exceptions.LockError:
            pass  # lock expired before we finished

Things to get right with locks:

  • Always set a timeout. Without one, a crashed worker leaves the lock behind forever and the task never runs again.
  • Set the timeout longer than the task's normal duration. If the lock expires while the task is still running, a second copy can start. Locks reduce duplicates but are not a perfect guarantee.
  • Release only your own lock. The redis-py lock object uses a unique token, so one task cannot release another task's lock. Avoid hand-rolled SET and DELETE logic that skips this check.
  • Use a shared backend. Django's cache.add() is atomic on Redis and Memcached backends and works as a simple lock, but not with the local-memory cache, which is separate for each process.
  • A lock does not make a task idempotent. If a task finishes and a duplicate arrives later, the lock is gone and the duplicate runs. Combine locks with an idempotency check.

Fix 4: Use Idempotency Keys for External Events

When the duplicate comes from outside, such as webhooks or API retries, use the event's own unique ID as the idempotency key.

class ProcessedEvent(models.Model):
    event_id = models.CharField(max_length=200, unique=True)
    processed_at = models.DateTimeField(auto_now_add=True)

@shared_task
def handle_payment_webhook(event_id, payload):
    try:
        with transaction.atomic():
            ProcessedEvent.objects.create(event_id=event_id)
            apply_payment(payload)  # rolls back with the key if it fails
    except IntegrityError:
        return  # event already handled

Because the key and the business change are in the same transaction, a failed run leaves no key behind, so a retry can still succeed. If you call external APIs that support idempotency keys, such as many payment providers, pass your own key there too.

Fix 5: Stop Overlapping Periodic Tasks

For Celery Beat tasks, there are two separate problems.

Overlapping runs. If a run can take longer than its interval, wrap it in a lock like the one above, and skip the run when the lock is already held.

Multiple Beat processes. Run exactly one Beat instance. Two Beat processes will each queue every scheduled task, which looks exactly like random duplicate execution. Check your deployment, especially after scaling or when using autoscaling containers.

Fix 6: Queue Tasks After the Transaction Commits

Queueing inside a transaction can cause both missing data and duplicate attempts. Use transaction.on_commit() so the task is sent only after the data is safely saved:

from django.db import transaction

def place_order(request):
    with transaction.atomic():
        order = Order.objects.create(...)
        transaction.on_commit(lambda: process_order.delay(order.id))

Pass IDs, not model objects, so the worker always reads fresh data.

Fix 7: Configure Celery with Duplicates in Mind

A few settings are worth reviewing:

Free tech newsletter

Free

Free to join — no card, no paywall. Weekly AI, cloud & tooling insights.

CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 43200}  # 12 hours
  • acks_late improves reliability but increases the chance of a rerun after a crash, so it makes idempotency non-negotiable.
  • Set visibility_timeout higher than your longest task runtime and your longest countdown or eta.
  • Avoid very long ETAs on Redis. For tasks scheduled far in the future, store the schedule in the database and queue the task closer to the time.

Which Fix Should You Use?

Situation Best approach
Any task that can be retried or redelivered Idempotent design
Creating a record only once Unique constraint plus IntegrityError handling
Concurrent updates to the same row select_for_update()
Expensive job that must not overlap Redis lock with a timeout
Webhooks and API retries Idempotency key stored in the database
Periodic tasks Lock inside the task, and a single Beat instance

In practice, combine them: idempotent logic as the foundation, a database constraint as the safety net, and a lock only where you need to avoid wasted work.

Related Reading: Build Reliable Pipelines Around Celery

Duplicate execution is one part of building reliable background processing. These guides cover the rest:

A Quick Checklist

Before shipping a Celery task that has side effects, ask:

  • Is the task safe to run twice?
  • Is there a database constraint that blocks the duplicate result?
  • Do external events carry an idempotency key that I store?
  • Do expensive tasks use a lock with a timeout?
  • Is only one Celery Beat instance running?
  • Is the task queued with transaction.on_commit()?
  • Is visibility_timeout longer than my longest task and countdown?
  • Have I tested what happens when the task runs twice at the same time?

Conclusion

You cannot fully stop Celery from delivering a task twice, but you can make duplicates harmless. Design tasks to be idempotent, let the database enforce uniqueness, use locks with timeouts where you need exclusivity, and use idempotency keys for outside events. Then tune Celery's acknowledgement and timeout settings with duplicates in mind.

Build it this way and a worker crash, a retry, or a double-click stops being a production incident and becomes just another event your system handles correctly.

Frequently Asked Questions

Why does my Celery task run twice? Common causes are worker crashes with acks_late enabled, the Redis visibility timeout redelivering long tasks, retries after partial success, double submissions from users or webhooks, and multiple Celery Beat instances. Identify which one applies before choosing a fix.

Does Celery guarantee exactly-once execution? No. Celery provides at-least-once delivery in many configurations, so tasks should be written to be safe when run more than once.

What is an idempotent Celery task? It is a task that produces the same final result whether it runs once or multiple times. For example, setting a value from source data is idempotent, while incrementing a counter is not.

Is a Redis lock enough to prevent duplicates? Not on its own. A lock prevents concurrent runs, but if the lock expires or the task finishes and a duplicate arrives later, it can run again. Combine locks with idempotent logic and database constraints.

How do I stop Celery Beat from running tasks twice? Run only one Beat process, and wrap long periodic tasks in a lock so a new run skips when the previous one is still active.

Share this article

Pass this technical dispatch along to fellow engineers

Did you enjoy this article?

Rate it so we know what content to bring next

Comments (0)