Using PostgreSQL SKIP LOCKED to safely process concurrent jobs in Django

Processing database-backed jobs looks simple until you add a second worker.

Imagine we have a table of pending jobs:

class Job(models.Model):
    class Status(models.TextChoices):
        PENDING = "pending"
        PROCESSING = "processing"
        DONE = "done"

    status = models.CharField(
        max_length=20,
        choices=Status.choices,
        default=Status.PENDING,
    )
    created_at = models.DateTimeField(auto_now_add=True)

A worker could fetch the next available job like this:

job = (
    Job.objects
    .filter(status=Job.Status.PENDING)
    .order_by("created_at")
    .first()
)

This is fine with one worker.

With multiple workers, however, two processes can read the same row before either one updates it:

Worker A → reads Job #1
Worker B → reads Job #1
Worker A → processes Job #1
Worker B → processes Job #1 again

For jobs that send emails, call external APIs, generate invoices, or publish events, processing the same record twice can be a serious problem. Processing should be idempotent anyway, but duplicates like this still slip through.

Locking the job with select_for_update()

Django exposes PostgreSQL row-level locks through select_for_update().

We can claim a job inside a transaction:

from django.db import transaction


def claim_job():
    with transaction.atomic():
        job = (
            Job.objects
            .select_for_update()
            .filter(status=Job.Status.PENDING)
            .order_by("created_at")
            .first()
        )

        if job is None:
            return None

        job.status = Job.Status.PROCESSING
        job.save(update_fields=["status"])

        return job

The generated SQL uses SELECT ... FOR UPDATE.

Once Worker A locks Job #1, another transaction trying to acquire the same lock must wait until Worker A finishes its transaction.

There is an important detail here: the lock alone does not prevent duplicates. It only makes the second worker wait.

What prevents the duplicate is the combination of:

• the row lock;
• code that changes the job’s state inside the same transaction (PENDING → PROCESSING);
• a query that always filters on the claimable state (status=PENDING).

When Worker A commits, the lock is released and PostgreSQL re-checks Worker B’s query against the updated row. Job #1 is no longer PENDING, so it no longer matches.

Without that state change, the waiting worker simply gets the same job as soon as the lock is released:

with transaction.atomic():
    job = (
        Job.objects
        .select_for_update()
        .filter(status=Job.Status.PENDING)
        .first()
    )

process_job(job)  # status is still pending

Worker B waits, Worker A commits, the lock is gone, Job #1 still matches the filter, and Worker B processes it again.

With the state change in place, this prevents duplicate claims, but it introduces another problem.

Row locking can serialize your workers

Suppose we have three pending jobs:

Job #1
Job #2
Job #3

Worker A locks Job #1.

At almost the same time, Worker B asks PostgreSQL for the oldest pending job.

The oldest row is still Job #1, so Worker B encounters the lock and waits.

We added multiple workers to increase throughput, but they can now end up waiting for each other.

This is where SKIP LOCKED becomes useful.

Skip the busy row instead of waiting

Django exposes PostgreSQL’s SKIP LOCKED behavior with:

select_for_update(skip_locked=True)

Our query becomes:

def claim_job():
    with transaction.atomic():
        job = (
            Job.objects
            .select_for_update(skip_locked=True)
            .filter(status=Job.Status.PENDING)
            .order_by("created_at")
            .first()
        )

        if job is None:
            return None

        job.status = Job.Status.PROCESSING
        job.save(update_fields=["status"])

        return job

Now PostgreSQL behaves differently:

Worker A
└── locks Job #1

Worker B
├── sees Job #1 is locked
└── skips it and locks Job #2

Worker C
├── skips Job #1
├── skips Job #2
└── locks Job #3

Instead of turning lock contention into waiting, we turn it into parallel work.

The state change still matters here. Once Worker A commits, Job #1 is no longer locked, so SKIP LOCKED would not skip it anymore. What keeps Workers B and C away from it after that point is its status: it is no longer PENDING.

PostgreSQL explicitly describes SKIP LOCKED as useful for multiple consumers accessing a queue-like table because locked rows are skipped rather than waited on.

Keep the transaction small

There is an important detail in the example above.

We should use the transaction to claim the job, not necessarily to perform the whole job.

Avoid this:

with transaction.atomic():
    job = claim_job()

    call_slow_external_api(job)
    generate_large_report(job)
    upload_file(job)

That keeps the database lock open while potentially slow operations happen. claim_job() opens its own transaction.atomic(), but inside an outer atomic() block it becomes a savepoint, so the lock is only released when the outer block commits.

A better approach is:

job = claim_job()

if job:
    process_job(job)

The transaction only protects the state transition:

pending → processing

Once committed, the worker owns the job and can perform expensive work without keeping the row locked.

What SKIP LOCKED does not guarantee

SKIP LOCKED solves concurrent claiming.

It does not automatically solve job reliability.

A worker can still:

1. claim the job;
2. mark it as processing;
3. crash before finishing.

Now the job may remain stuck.

Production systems usually need additional mechanisms such as:

• a started_at timestamp;
• a lease or timeout;
• retry counters;
• idempotent processing;
• recovery of jobs stuck in processing;
• monitoring for old jobs.

For example:

class Job(models.Model):
    status = models.CharField(...)
    started_at = models.DateTimeField(null=True)
    attempts = models.PositiveIntegerField(default=0)

Then a recovery process can return abandoned jobs to the queue. Two cautions:

• the recovery process should claim stuck jobs with the same select_for_update(skip_locked=True) pattern, so it does not race with workers or with another recovery run;
• a lease can expire while a slow worker is still alive, so two workers can end up running the same job. That is why idempotent processing stays on the list.

SKIP LOCKED is one piece of the coordination strategy, not a replacement for one.

There is also a fairness trade-off

Skipping locked rows means consumers do not necessarily process rows in strict order.

If Job #1 remains locked for a long time, Workers B and C can continue processing Jobs #2, #3, and beyond.

That makes SKIP LOCKED a good fit when:

throughput > strict ordering

It may not be appropriate when jobs must be processed in an exact sequence.

PostgreSQL also warns that skipping locked rows gives an inconsistent view of the data, which is why this behavior is intended for queue-like workloads rather than normal application queries.

When this pattern is useful

We usually reach for a dedicated queue such as Celery when background processing becomes an important part of the architecture.

But database-backed claiming with SKIP LOCKED can be surprisingly useful for smaller coordination problems, including:

• transactional outbox consumers;
• webhook delivery;
• import pipelines;
• reconciliation jobs;
• notification processing;
• scheduled batch work.

It is especially convenient when the work already depends on state stored in PostgreSQL and we want the claiming decision to participate in the same transactional model.

Django makes the PostgreSQL primitive available with a very small API:

.select_for_update(skip_locked=True)

But the effect is significant.

Without it, competing workers can either process the same record or wait behind one another.

With it, PostgreSQL can coordinate them:

lock what I am working on,
skip what somebody else already claimed,
and keep processing the rest.

Sometimes a small database feature is enough to remove an entire layer of application-side coordination.