The Fork Bug and the Deploy Bug

The Consequence of Forking

The previous post solved the memory problem by running tasks in subprocesses. Memory went from 500 MB to 50 MB. Problem solved.

Then a new bug appeared — a direct consequence of how process creation works on Linux, subtle enough to pass testing but fail in production. And while chasing it, we found a second bug that had nothing to do with fork at all: it was hiding in our deploy script, waiting for a long-running process to expose it. This post covers both.

Quick background: When Python’s multiprocessing creates a child process on Linux, it uses a system call called fork(). This creates an almost-exact copy of the parent process — same memory contents, same open network connections, same everything. The child is a clone that starts running from the point of the fork. This is different from starting a fresh Python interpreter; the child inherits whatever state the parent had at that moment.

Bug 1: The Shared Database Connection

The symptom

Intermittent database errors in production. Not every task — maybe 1 in 20. Errors like:

django.db.utils.OperationalError: server closed the connection unexpectedly
psycopg2.InterfaceError: connection already closed

Tasks that worked fine in-process started failing randomly after we switched to subprocesses.

The cause

Under the hood, a Django database connection is a network socket — a live communication channel between your Python process and PostgreSQL. When fork() copies the parent process to create a child, it copies the references to all open connections too. But it doesn’t create new connections. Both parent and child now point to the same underlying network channel to PostgreSQL.

flowchart TB
    subgraph "Before fork()"
        P1["Parent Process"]
        FD1["DB Connection"]
        P1 --- FD1
    end
    subgraph "After fork()"
        P2["Parent Process"]
        C1["Child Process"]
        FD2["DB Connection (SHARED!)"]
        P2 -.- FD2
        C1 -.- FD2
    end
    FD2 -->|"same network socket"| DB["PostgreSQL"]

This is like two people trying to talk on the same phone call simultaneously. When the child finishes a task and exits, Python’s cleanup hangs up the connection. Now the parent’s connection object points to a dead line. The next time the parent tries to query the database — error.

Even worse: if both parent and child try to query the database simultaneously, their messages get interleaved on the same connection. PostgreSQL sees garbled data and drops the connection entirely.

The fix

Close all database connections in the child process before doing any work. The _child_worker function from the previous post gains exactly one line at the top:

from django.db import connections

def _child_worker(task_name, task_params):
    # CRITICAL: close inherited connections before doing anything
    connections.close_all()
    ...  # import and execute the task as before

connections.close_all() tells Django to close every database connection it currently holds. This drops the inherited (shared) connections. When the task code subsequently accesses the database, Django automatically opens a new connection — one that belongs exclusively to the child process. No sharing, no conflict.

Why close in the child, not the parent? The child is short-lived — it runs one task and exits. The parent is long-lived and polls the database constantly. Closing the parent’s connections would force it to reconnect on every poll cycle. Closing in the child is the right place.

ctx = multiprocessing.get_context("spawn")
process = ctx.Process(target=_child_worker, args=(...))

Bug 2: The Vanishing Working Directory

To be clear up front: this one is not a fork bug. It’s a deploy-script bug that predates the subprocess work — any long-running process would eventually have hit it. We just found it in the same debugging session, because the subprocess worker was the long-running process that finally tripped over it.

The symptom

After a deployment, the worker stops processing tasks. No error. No crash. The management command just… hangs. Restarting the worker fixes it.

The cause

Our deployment script looked like this:

# deploy.sh
rm -rf /opt/myapp/current
cp -r /opt/myapp/release /opt/myapp/current
systemctl restart myapp-web
# Note: worker was NOT restarted

The problem: rm -rf deletes the directory, and cp -r creates a new directory at the same path. To a human looking at the filesystem, /opt/myapp/current looks identical before and after. But to the operating system, it’s a completely different directory.

Here’s why: every file and directory on disk has an internal ID number (called an inode). When a process opens a directory or sets its working directory, it remembers that internal ID — not the path string. rm -rf destroys the old directory (and its ID). cp -r creates a new one with the same name but a different internal ID.

The running worker process is still holding onto the old ID. It’s now pointing at a directory that no longer exists.

flowchart TB
    subgraph "Before deploy"
        W1["Worker (remembers directory #12345)"]
        D1["/opt/myapp/current (is directory #12345)"]
        W1 --> D1
    end
    subgraph "After rm + cp"
        W2["Worker (still looking for #12345 — GONE)"]
        D2["/opt/myapp/current (is directory #67890 — NEW)"]
        W2 -->|"points at nothing"| X["deleted directory"]
    end

The worker was running in the old directory. When it spawns a child process, the child inherits this broken reference. The child can’t find files relative to its working directory, module imports may fail, and the whole thing silently breaks.

The fix

Switch from “delete and copy” to rsync:

# deploy.sh (fixed)
rsync -a --delete /opt/myapp/release/ /opt/myapp/current/
systemctl restart myapp-web

rsync updates files in place inside the existing directory. The directory itself — its internal ID — stays the same. Running processes that reference this directory continue to work because the directory they’re pointing at still exists; only its contents changed.

The tradeoff: rsync is slightly slower than cp for a full fresh copy, and it leaves the directory in a mixed state during sync (some files old, some new). For atomic deployments, some teams use symlink swapping instead. But symlink swapping has its own issues — the process may resolve the symlink target at startup and still end up with a stale reference.

The Design Lesson

The two bugs are not the same kind of bug, and it took me a while to see that. The connection bug is a genuine consequence of fork(): the child inherits the parent’s state, and some of that state — live network sockets — must not be shared. The deploy bug was latent in the deploy script all along; the subprocess work didn’t cause it, it just made us look closely enough to find it.

flowchart TB
    A["In-process execution"] -->|"memory leak"| B["Subprocess isolation"]
    B -->|"shared connections after fork"| C["connections.close_all()"]
    F["rm + cp deploy script"] -->|"inode invalidation"| D["rsync deployment"]
    C --> E["Working system"]
    D --> E

What they share: every architectural decision constrains the decisions around it. The subprocess model is the right choice for memory management, but it means you need to think about what the child process inherits from the parent — connections, working directory, and more. And a long-running worker means your deploy script can no longer treat the filesystem as if nothing is watching it.

Checklist for Subprocess Workers

If you’re spawning child processes from a long-running Django command, verify:

1. Database connections — Call connections.close_all() in the child before any DB access, or the parent and child will fight over the same connection
2. Open connections — Any network connections, open files, or sockets in the parent are shared with the child after fork
3. Working directory — Deployment must update files in-place (rsync) or restart the worker; deleting and recreating the directory breaks running processes
4. Signal handling — The child inherits the parent’s signal handlers, which may not be appropriate for task code
5. Logging — If both parent and child write to the same log file, their output can get mixed together

Key Takeaways

1. Forking shares database connections — parent and child end up talking over the same network socket, causing corruption
2. connections.close_all() before work — let Django create fresh connections in the child
3. Directories have internal IDs — rm + cp destroys and recreates the directory; rsync updates it in place
4. Long-running workers constrain deployment — deleting and recreating their directory breaks them; update in place or restart them
5. Fixes surface neighboring bugs — solving memory created the fork bug, and debugging the fork bug exposed a deploy bug that had been there all along

Next in this series: Pessimistic Locking: One Task, One Worker