How to Export Large CSV Files in Django Without Memory Errors

A CSV export button looks like the simplest feature in any admin panel. It works perfectly in development with a few hundred rows. Then it reaches production, a customer clicks "Export all orders", and the server runs out of memory, the request times out, or the worker gets killed.

The reason is almost always the same: the code builds the entire file in memory before sending a single byte. With a few thousand rows, that is fine. With a few million, it is a problem.

This guide shows how to export large CSV files in Django safely, starting with the common mistake and moving through streaming, memory-safe querying, background exports, and a PostgreSQL-native option for very large datasets.

Why Large CSV Exports Fail

Here is the version most developers write first:

import csv
from django.http import HttpResponse

def export_orders(request):
    response = HttpResponse(content_type="text/csv")
    response["Content-Disposition"] = 'attachment; filename="orders.csv"'

    writer = csv.writer(response)
    for order in Order.objects.all():
        writer.writerow([order.id, order.customer.name, order.total])

    return response

This has three separate problems:

• The queryset is cached in memory. Order.objects.all() loads every row into Python objects as you iterate, and Django keeps them in the queryset's result cache.
• The response is built in memory. HttpResponse holds the entire CSV as one string before sending it.
• N+1 queries. Accessing order.customer.name inside the loop runs an extra query per row.

On top of that, the browser and your web server wait the whole time. A request that takes longer than your gateway timeout, often 30 to 60 seconds, gets cut off before the file finishes.

Fix 1: Stream the Response with StreamingHttpResponse

Django's StreamingHttpResponse sends data to the client in pieces as it is generated, instead of building the whole body first. Combined with the csv module, it lets you start the download immediately and keep memory flat.

The csv writer needs something with a write() method. A small helper that simply returns the value turns each written row into a string you can yield:

import csv
from django.http import StreamingHttpResponse

class Echo:
    """An object that implements just the write method of the file-like interface."""
    def write(self, value):
        return value

def export_orders(request):
    writer = csv.writer(Echo())

    def rows():
        yield ["ID", "Customer", "Total", "Created"]
        queryset = (
            Order.objects
            .values_list("id", "customer__name", "total", "created_at")
            .order_by("id")
        )
        for row in queryset.iterator(chunk_size=2000):
            yield row

    response = StreamingHttpResponse(
        (writer.writerow(row) for row in rows()),
        content_type="text/csv",
    )
    response["Content-Disposition"] = 'attachment; filename="orders.csv"'
    return response

Each row is converted to a CSV line, sent to the client, and then discarded. Memory use stays roughly constant whether the export has 10,000 rows or 10 million.

Fix 2: Query Memory-Safely with iterator() and values_list()

Streaming the response is only half the job. If the queryset still loads everything, you have only moved the problem.

• iterator(chunk_size=2000) fetches rows in batches and does not cache them in the queryset.
• values_list() returns plain tuples instead of full model instances, which is much lighter and faster when you only need a few columns.
• only() is the alternative when you do need model instances or methods, because it limits which columns are loaded.
• Joined fields like customer__name inside values_list() are fetched with a JOIN, so you avoid the N+1 problem without needing select_related().

Two details are worth knowing. With PostgreSQL, iterator() uses server-side cursors by default, which can misbehave behind a connection pooler such as PgBouncer in transaction pooling mode. If you use one, check your Django DISABLE_SERVER_SIDE_CURSORS setting. And if you do need related objects through iterator(), pass a chunk_size so Django can prefetch per batch.

For more on the query side of this, including indexes and avoiding N+1 queries, see my guide on Django ORM optimization for millions of records: https://blog.reiteachaigean.com/blog/django-orm-optimize-millions-of-records

Fix 3: Make Sure Your Server Does Not Buffer the Stream

Streaming in Django does not help if something in front of it collects the whole response first. A few things to check:

• Nginx: proxy buffering can hold the response. You can send an X-Accel-Buffering: no header from Django to disable buffering for that response.
• Gunicorn: a request that streams for a long time can hit the worker timeout. Increase the timeout for export endpoints, or better, move big exports to the background (see Fix 4).
• Compression middleware: some middleware tries to read the whole body. Test with a real large export, not just a small one.
• ASGI: if you run Django under ASGI, a synchronous iterator in StreamingHttpResponse makes Django load the whole thing to preserve async behavior and logs a warning. Use an async iterator for ASGI deployments.

Fix 4: Move Very Large Exports to a Background Task

Streaming works well up to a point. If an export takes several minutes, ties up a web worker, and fails if the user closes the tab, a background job is the better design.

The pattern:

1. The user clicks "Export".
2. Django queues a Celery task and responds immediately.
3. The task writes the CSV to a file or cloud storage.
4. The user gets a download link, by email or a notification.

import csv
import tempfile
from celery import shared_task

@shared_task
def export_orders_task(user_id):
    with tempfile.NamedTemporaryFile(
        mode="w", newline="", suffix=".csv", delete=False, encoding="utf-8"
    ) as f:
        writer = csv.writer(f)
        writer.writerow(["ID", "Customer", "Total", "Created"])

        queryset = (
            Order.objects
            .values_list("id", "customer__name", "total", "created_at")
            .order_by("id")
        )
        for row in queryset.iterator(chunk_size=2000):
            writer.writerow(row)

    # Upload f.name to S3 or default storage, then notify the user with a link
    return f.name

This also gives you room to add progress tracking, retries, and file expiry. For tasks that process huge volumes, the chunking and idempotency ideas from my guide on processing 5 million records with Django and Celery apply directly: https://blog.reiteachaigean.com/blog/process-5-million-records-django-celery

Fix 5: Use PostgreSQL COPY for Extreme Sizes

When the export is tens of millions of rows, the Python loop itself becomes the bottleneck. PostgreSQL can generate CSV directly with COPY ... TO STDOUT, which is usually much faster than formatting each row in Python.

With psycopg 3, you can stream the result straight through:

from django.db import connection
from django.http import StreamingHttpResponse

def export_orders_copy(request):
    sql = """
        COPY (
            SELECT o.id, c.name, o.total, o.created_at
            FROM orders_order o
            JOIN customers_customer c ON c.id = o.customer_id
            ORDER BY o.id
        ) TO STDOUT WITH CSV HEADER
    """

    def stream():
        with connection.cursor() as cursor:
            with cursor.cursor.copy(sql) as copy:
                for chunk in copy:
                    yield bytes(chunk)

    response = StreamingHttpResponse(stream(), content_type="text/csv")
    response["Content-Disposition"] = 'attachment; filename="orders.csv"'
    return response

The trade-offs are real: it is PostgreSQL-specific, needs raw SQL, and bypasses model logic such as custom field conversion. If you add filters from user input, pass them as parameters and never build the SQL with string formatting. Treat it as an advanced option for when the standard approach is not fast enough.

Don't Forget CSV Quality and Security

Memory is not the only thing that can go wrong with an export.

CSV injection. If a cell starts with =, +, -, or @, spreadsheet apps may treat it as a formula. When exports contain user-supplied text, sanitize those cells:

def safe_cell(value):
    if isinstance(value, str) and value.startswith(("=", "+", "-", "@")):
        return "'" + value
    return value

Excel and encoding. Excel often misreads UTF-8 files without a byte order mark, which garbles non-English characters. If your users open files in Excel, write a UTF-8 BOM ("\ufeff") at the start of the file.

Permissions and limits. Make sure the export respects the same access rules as the page it comes from, and consider rate-limiting export endpoints so a single user cannot trigger dozens at once.

Filters. Apply the user's filters in the queryset, so they export what they see and not the entire table.

Which Approach Should You Choose?

These are starting points, not hard limits. Measure memory, duration, and database load on your own data.

A Quick Checklist

Before shipping a CSV export feature, confirm that you:

• Use StreamingHttpResponse instead of building the file in memory
• Use iterator(chunk_size=...) and values_list() or only()
• Avoid N+1 queries inside the export loop
• Have indexes on the columns used to filter and order
• Checked that Nginx, Gunicorn, and middleware do not buffer or time out the stream
• Moved long exports to a background task
• Sanitize cells against CSV injection and handle encoding for Excel
• Tested with a realistic, large dataset

Conclusion

Large CSV exports fail when the code holds everything in memory at once. The fix is to stream: read rows in batches, write them out as you go, and never keep the full dataset or the full file in memory. Start with StreamingHttpResponse and iterator(), move heavy jobs to Celery, and reach for PostgreSQL COPY only when the dataset truly demands it.

Do that, and your export button will work the same whether the table has a thousand rows or a hundred million.

Frequently Asked Questions

Why does my Django CSV export run out of memory? Usually because the queryset is loaded and cached in memory, and the HttpResponse builds the entire CSV as one string before sending it. Use iterator() with a StreamingHttpResponse so rows are processed and sent in small batches.

What is the difference between HttpResponse and StreamingHttpResponse? HttpResponse holds the full body in memory and sends it at once. StreamingHttpResponse sends the body in pieces from an iterator, so memory use stays low and the download starts immediately.

Does iterator() work with prefetch_related? Only when you pass a chunk_size, in which case Django prefetches related objects for each batch. For simple exports, values_list() with joined fields is usually lighter.

How do I export a CSV that takes several minutes to generate? Run it as a background task with Celery, write the file to storage such as S3, and send the user a download link when it is ready. This avoids request timeouts and frees web workers.

Is PostgreSQL COPY faster than a Python loop? Generally yes, because PostgreSQL formats the CSV itself and Python only passes the bytes through. It is PostgreSQL-specific and requires raw SQL, so use it when the standard streaming approach is not fast enough.