Django Database Connection Pooling with PostgreSQL Guide
Learn Django database connection pooling with PostgreSQL: CONN_MAX_AGE, Django 5.1 native pooling, PgBouncer setup, pool sizing, Celery tips, and common pitfalls.
Sooner or later, a growing Django application meets this error:
FATAL: sorry, too many clients already
or its cousin:
FATAL: remaining connection slots are reserved for non-replication superuser connections
Nothing is wrong with your queries. You have simply run out of PostgreSQL connections. Every web worker, every Celery process, every cron script, and every admin tool wants its own connection, and PostgreSQL only allows so many.
Connection pooling is the standard solution. This guide explains why PostgreSQL connections are expensive, how Django handles connections by default, and the three main ways to pool them: persistent connections with CONN_MAX_AGE, Django's native pool (added in Django 5.1), and an external pooler such as PgBouncer. It also covers how to size a pool, what to watch out for with Celery, and the pitfalls that cause the most production incidents.
Why PostgreSQL Connections Are Expensive
Unlike some databases that use lightweight threads, PostgreSQL starts a separate operating system process for every client connection. Each backend process uses memory, and creating one takes time, because of the TCP handshake, authentication, and TLS negotiation if you use it.
This has two consequences:
- Opening a connection per request is slow. The setup cost can be several milliseconds or more, which is a lot when the query itself takes one millisecond.
- Having many open connections is costly too. The default
max_connectionsis 100. Raising it to thousands increases memory use and can hurt performance, because many active backends compete for CPU, locks, and I/O.
The goal of pooling is simple: keep a small number of well-used connections open, and share them among many requests.
How Django Handles Connections by Default
Out of the box, Django opens a new database connection at the start of each request and closes it when the request ends. That is controlled by CONN_MAX_AGE, which defaults to 0.
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "mydb",
"USER": "myuser",
"PASSWORD": "secret",
"HOST": "127.0.0.1",
"PORT": "5432",
"CONN_MAX_AGE": 0, # default: new connection per request
}
}
This is safe and simple, but it pays the connection setup cost on every request. On a busy site, that adds latency and puts churn on PostgreSQL.
Option 1: Persistent Connections with CONN_MAX_AGE
The simplest improvement is to let each Django process keep its connection open between requests:
DATABASES = {
"default": {
# ...
"CONN_MAX_AGE": 60, # seconds; None means unlimited
"CONN_HEALTH_CHECKS": True, # verify the connection before reuse
}
}
With this, each worker thread reuses its connection for up to 60 seconds. CONN_HEALTH_CHECKS makes Django check that a reused connection is still alive at the start of a request, which avoids errors after a database restart or a dropped connection.
What this is, and what it is not. This is connection reuse, not a true pool. Each thread or process holds its own connection, whether it is busy or idle. With 8 Gunicorn workers and 4 threads each, you can hold 32 connections open all the time. It does not share connections between workers.
Free tech newsletter
FreeFree to join — no card, no paywall. Weekly AI, cloud & tooling insights.
A warning about threads and ASGI. Persistent connections are tied to threads. Under ASGI, Django runs sync code in thread pools, and persistent connections can leak or accumulate. Django's documentation recommends CONN_MAX_AGE = 0 for ASGI deployments, and suggests using a real pool instead.
Persistent connections are a good first step for small and medium sites with a modest number of workers. They stop helping when the number of processes grows.
Option 2: Django's Native Connection Pool (Django 5.1+)
Django 5.1 added built-in connection pooling for PostgreSQL. It requires the psycopg (version 3) driver with its pool extra:
pip install "psycopg[binary,pool]"
Then enable it in OPTIONS:
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "mydb",
"USER": "myuser",
"PASSWORD": "secret",
"HOST": "127.0.0.1",
"PORT": "5432",
"CONN_MAX_AGE": 0, # must be 0 when using the native pool
"OPTIONS": {
"pool": {
"min_size": 2,
"max_size": 10,
"timeout": 10,
"max_lifetime": 1800,
"max_idle": 300,
},
},
}
}
Setting "pool": True uses the driver's defaults, and a dictionary lets you pass options to psycopg's ConnectionPool. The ones that matter most:
min_size: connections kept open even when idle.max_size: the most connections this pool will open.timeout: how long a request waits for a free connection before failing.max_lifetime: how long a connection lives before being replaced, which helps with load balancers and long-lived resources.max_idle: how long an unused connection stays open before the pool shrinks.
Important points about the native pool:
- It is per process. Each Gunicorn worker process creates its own pool. If you run 8 workers with
max_size: 10, your application can open up to 80 connections. Size the pool with that multiplication in mind. - It replaces
CONN_MAX_AGE. Django refuses to start if you combine the pool with persistent connections, so keepCONN_MAX_AGEat 0. - It uses psycopg 3 only. Projects still on
psycopg2need to migrate the driver first. That is usually a small change, but test your app because some behaviors differ. - It does not reduce connections across processes. Because each process has its own pool, it improves reuse inside a process but does not cap the total across your whole fleet. For that, you need an external pooler.
The native pool is a great fit for a single application server or a modest number of processes, with no extra infrastructure to run.
Option 3: An External Pooler Like PgBouncer
When you have many application processes, such as multiple web servers, Celery workers, and cron jobs, you need a pooler that sits between all of them and PostgreSQL. The most widely used one is PgBouncer.
Django web workers ─┐
Celery workers ─────┼──► PgBouncer ──► PostgreSQL (few real connections)
Cron / scripts ─────┘
Thousands of client connections can share a few dozen real PostgreSQL connections. This is how large Django deployments keep the database stable.
PgBouncer pool modes
The pool mode decides when a real server connection is handed back to the pool:
| Mode | Connection returned to the pool | Notes |
|---|---|---|
| Session | When the client disconnects | Safest, but saves the least |
| Transaction | When each transaction finishes | Best sharing, the usual choice for web apps |
| Statement | After every statement | Breaks multi-statement transactions, not suitable for Django |
Transaction pooling is what most Django teams use, because a connection is only held while a transaction is actually running.
A minimal pgbouncer.ini:
[databases]
mydb = host=127.0.0.1 port=5432 dbname=mydb
[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20
reserve_pool_size = 5
server_idle_timeout = 600
Here, up to 1,000 clients can connect to PgBouncer, but at most about 20 server connections per database and user pair are used against PostgreSQL.
Free tech newsletter
FreeFree to join — no card, no paywall. Weekly AI, cloud & tooling insights.
Django settings for transaction pooling
Point Django at PgBouncer's port, and turn off a feature that does not work in transaction mode:
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "mydb",
"USER": "myuser",
"PASSWORD": "secret",
"HOST": "127.0.0.1",
"PORT": "6432", # PgBouncer, not PostgreSQL
"CONN_MAX_AGE": 0, # or a modest value; see below
"DISABLE_SERVER_SIDE_CURSORS": True, # required for transaction pooling
}
}
Why disable server-side cursors? Django uses server-side cursors for QuerySet.iterator() on PostgreSQL. A cursor lives on one specific server connection, but in transaction mode your next statement may land on a different one. The result is errors or missing cursors. Setting DISABLE_SERVER_SIDE_CURSORS = True makes iterator() fetch results in chunks without them, which still keeps memory use low.
What about CONN_MAX_AGE? With PgBouncer, a client connection to the pooler is cheap, so CONN_MAX_AGE = 0 is perfectly fine. A modest persistent value also works, since the real server connections are managed by PgBouncer. Do not rely on long-lived Django connections for correctness.
Transaction Pooling Pitfalls
Transaction pooling is powerful but changes the rules. Anything that depends on session state can break, because consecutive transactions may run on different server connections.
- Server-side cursors: disable them, as above.
- Session-level advisory locks:
pg_advisory_lock()held across transactions will not behave as expected. Use transaction-level locks (pg_advisory_xact_lock) instead. LISTEN/NOTIFY: needs a stable session. Use a direct connection for listeners.- Session-level
SETcommands: aSETsuch as a customsearch_pathorstatement_timeoutmay not stick. UseSET LOCALinside a transaction, or configure the setting per role or database. - Prepared statements: historically a problem in transaction mode. Recent PgBouncer versions (1.21 and later) can support protocol-level prepared statements, but check your version and settings. With psycopg 3, you can also disable automatic server-side preparation by setting
prepare_thresholdtoNoneinOPTIONSif you see "prepared statement does not exist" errors. - Temporary tables: they belong to a session, so they may vanish between transactions.
If you depend on any of these, either avoid them, use session pooling, or send those specific workloads through a direct connection that bypasses the pooler.
How to Size Your Pool
More connections does not mean more speed. PostgreSQL performs best when the number of active queries is close to what its CPU and disk can handle. A common rule of thumb for the number of active connections is a small multiple of the CPU core count, but the right number depends on your workload, so treat any formula as a starting point and measure.
Do the arithmetic across your whole system:
total possible app connections =
(web processes × pool max_size)
+ (celery processes × connections each)
+ cron jobs, admin tools, migrations, monitoring
This total must stay below PostgreSQL's max_connections, with headroom for superuser access and maintenance. If it does not, either reduce pool sizes or put PgBouncer in front.
A few practical guidelines:
- Start small. A pool of 5 to 20 connections per application server is often enough.
- Watch for requests waiting on the pool (
timeouterrors). That is a sign of slow queries or too few connections, not always a reason to add more. - Slow queries hold connections longer, so fixing them often reduces the connections you need.
- Keep
max_connectionsmodest on PostgreSQL, and let the pooler absorb client spikes.
Celery and Connection Pooling
Celery deserves special attention, because it multiplies your process count.
- Each Celery prefork worker process opens its own database connection. A worker with
--concurrency=16can use 16 connections, and ten such workers use 160. - Long-running tasks hold connections for a long time, even while waiting on external APIs.
- Django closes stale connections between requests, but Celery tasks do not go through the request cycle, so old or broken connections can linger. Django's
close_old_connections()is called around Celery tasks by Django's Celery integration, but it is still worth checking how your setup behaves, especially withCONN_MAX_AGEset.
Practical advice: route Celery workers through PgBouncer too, keep concurrency reasonable, and avoid holding a transaction open while doing slow network calls. For bulk jobs, small chunks with short transactions use connections much more efficiently than one long task.
Monitoring Your Connections
Do not guess; look at what PostgreSQL is doing.
Free tech newsletter
FreeFree to join — no card, no paywall. Weekly AI, cloud & tooling insights.
-- How many connections, by state
SELECT state, count(*)
FROM pg_stat_activity
GROUP BY state;
-- Who is using them
SELECT usename, application_name, state, count(*)
FROM pg_stat_activity
GROUP BY usename, application_name, state
ORDER BY count(*) DESC;
-- The limit
SHOW max_connections;
Watch for a large number of connections in the idle in transaction state. These hold resources and locks while doing nothing, and are often caused by code that opens a transaction and then makes a slow external call. Setting idle_in_transaction_session_timeout in PostgreSQL helps catch them.
With PgBouncer, connect to its admin console and run SHOW POOLS; and SHOW STATS; to see active clients, waiting clients, and server connections.
Which Option Should You Choose?
| Situation | Recommended approach |
|---|---|
| Small app, few workers | CONN_MAX_AGE with CONN_HEALTH_CHECKS |
| Single server, Django 5.1+, psycopg 3 | Django's native pool |
| ASGI deployment | Native pool or PgBouncer, not persistent connections |
| Many web servers and Celery workers | PgBouncer in transaction mode |
| Managed cloud database | Provider's pooler or proxy, if available |
| Need session features (LISTEN, advisory locks) | Direct connection or session pooling for those workloads |
You can also combine layers. For example, use PgBouncer for the fleet and keep Django's settings simple, instead of stacking multiple pools on top of each other, which makes behavior harder to reason about.
Related Reading: Keep Your Database Healthy Under Load
Connection pooling solves the "too many connections" problem, but a healthy database also needs efficient queries and well-behaved background jobs:
- If your workers hold connections for a long time: How to Process 5 Million Records with Django and Celery shows how to split big jobs into short, chunked tasks, which keeps connections and transactions short.
- If slow queries are eating your pool: Django ORM Optimization for Millions of Records covers indexes,
EXPLAIN ANALYZE, and pagination, the fixes that free connections faster. - If a page runs hundreds of queries: select_related vs prefetch_related: Practical Examples helps you cut the query count that keeps each request holding a connection.
- If Celery tasks might run twice: Celery: How to Prevent Duplicate Task Execution explains idempotency and locks, which matter when retries and redeliveries add extra database load.
- If the admin is slow on big tables: How to Fix Slow Django Admin Pages covers the query habits that often hold admin connections open for too long.
A Quick Checklist
Before going to production, confirm that:
- You know your PostgreSQL
max_connectionsand your total possible app connections - Each process type (web, Celery, cron) has a known connection budget
- You use persistent connections, the native pool, or PgBouncer, and not several layers by accident
- With PgBouncer in transaction mode,
DISABLE_SERVER_SIDE_CURSORS = Trueis set - You do not depend on session state, such as session-level advisory locks or
SET CONN_MAX_AGEis0when using Django's native pool- Slow queries and long transactions are monitored
idle in transactionconnections are tracked and timed out- You load-tested with realistic concurrency
Conclusion
Database connection pooling is not an optimization you add at the end. It is part of how a Django and PostgreSQL system survives growth. Start with persistent connections if your app is small, move to Django's native pool when you want real pooling inside a process, and put PgBouncer in front of PostgreSQL when many processes share the same database.
Whichever you choose, keep the total number of connections low, keep transactions short, and measure. A small, busy pool backed by fast queries will beat a large pool of idle connections almost every time.
Frequently Asked Questions
What is connection pooling in Django?
It is the practice of keeping a limited set of database connections open and reusing them across requests, instead of opening and closing a new connection each time. Django supports persistent connections through CONN_MAX_AGE, a native pool for PostgreSQL since Django 5.1, and external poolers like PgBouncer.
Does Django have built-in connection pooling?
Yes, starting with Django 5.1, for PostgreSQL only, using psycopg 3 with the pool extra. You enable it through the pool option inside OPTIONS. Earlier versions only offer persistent connections through CONN_MAX_AGE.
Is CONN_MAX_AGE the same as connection pooling?
Not exactly. CONN_MAX_AGE keeps one connection open per thread or process for reuse, but those connections are not shared between workers. A real pool manages a set of connections that requests borrow and return.
Do I need PgBouncer if I use Django's native pool? Not necessarily. The native pool is per process, so it works well for a single server or a small number of processes. If many web servers and Celery workers all connect to one database, PgBouncer gives you a single place to cap total connections.
Why do I need DISABLE_SERVER_SIDE_CURSORS with PgBouncer?
In transaction pooling mode, consecutive statements can run on different server connections, but a server-side cursor exists on only one. Disabling server-side cursors avoids errors when using QuerySet.iterator().
How many database connections should my Django app use?
There is no universal number. Start with a small pool, add up connections across all web, Celery, and background processes, keep the total comfortably under max_connections, and adjust based on measured wait times and database load.
