FastAPI Backpressure: Protecting Your Service from Unbounded Fan-Out

Async changes waiting, not capacity

An async FastAPI process can keep many sockets in flight while tasks wait for I/O. That is useful, but it can also make overload arrive faster: a burst of requests can fan out into hundreds of database queries or outbound HTTP calls. The event loop is responsive while every downstream pool is exhausted, and users still see latency climb.

Think in terms of a capacity budget. If each request can make six parallel service calls, then 100 active requests may create 600 downstream operations. A semaphore around a scarce operation, a bounded worker queue, or a carefully sized client pool limits that amplification. The correct limit depends on downstream capacity and latency, not on a desire to maximize local concurrency.

Put limits at resource boundaries

Create long-lived async clients and database engines in the application lifespan so their connection pools can be reused. Configure pool sizes, acquisition timeouts, and per-request deadlines deliberately. Avoid creating a new client in every endpoint call: that discards pooling benefits and can create connection churn.

import asyncio

partner_limit = asyncio.Semaphore(20)

async def fetch_partner(client, url: str):
    async with partner_limit:
        return await client.get(url, timeout=2.0)

This limits concurrent calls in one process. If the service runs ten replicas, the partner may still see up to 200 calls, so local limits must be considered together with replica scaling and provider quotas.

Design an end-to-end deadline

Independent five-second timeouts at three layers can turn one request into fifteen seconds of waiting. Pass a remaining deadline through the call chain, reserve time for response serialization and cleanup, and cancel work that no longer has a useful caller. Retries need the same budget and should be bounded; retrying into an already overloaded dependency compounds the incident.

Fail deliberately under load

When a queue or pool is full, choose a clear behavior: reject with a retryable response, shed low-priority work, or enqueue it durably if asynchronous completion is part of the product contract. Do not create an unbounded in-memory task queue. It converts a visible overload into memory pressure and delayed failure.

Instrument active requests, pool wait time, queue depth, timeout counts, and downstream latency. Those measurements tell you whether to add capacity, reduce fan-out, or change the product's overload policy. Async is a concurrency tool; backpressure is what keeps that concurrency within the system's real limits.