An event-driven provider queue built on Celery

When a customer requests a Letter of Medical Necessity, that request has to reach a licensed practitioner, get reviewed against the customer's stated condition, and come back with a decision — all within a time window customers actually notice. Our original implementation was a polling loop: a worker checked a database table every few seconds for pending requests and assigned them to available practitioners. It worked at low volume and fell over the moment request volume became bursty.

Why the provider queue needed to be event-driven

Polling has a fundamental tension: poll too slowly and requests sit idle waiting for the next tick; poll too fast and you spend most of your capacity scanning a table that hasn't changed. Neither problem is solvable by tuning the interval — you need the system to react when something happens, not on a fixed schedule. We rebuilt the queue around Celery so that a new LMN request, a practitioner coming online, or a review being submitted all emit an event that immediately triggers the next step, rather than waiting to be discovered.

The move also let us decouple two things that had been awkwardly coupled: request intake and practitioner assignment. Intake now just publishes an event. Assignment is a separate consumer that can scale independently and be reasoned about on its own.

Architecture

The queue has three logical stages: intake, assignment, and review, each backed by its own Celery task and routed through a dedicated queue rather than a shared default queue.

Task routing with Celery

Each stage gets its own named queue so that a spike in one stage — say, a burst of new requests — doesn't starve practitioner review tasks of worker capacity:

app = Celery("provider_queue")

app.conf.task_routes = {
    "provider_queue.intake.*": {"queue": "intake"},
    "provider_queue.assignment.*": {"queue": "assignment"},
    "provider_queue.review.*": {"queue": "review"},
}

@app.task(bind=True, max_retries=5, queue="assignment")
def assign_practitioner(self, request_id: str):
    request = LmnRequest.objects.get(id=request_id)
    practitioner = find_available_practitioner(request.condition)
    if practitioner is None:
        raise self.retry(countdown=exponential_backoff(self.request.retries))
    request.assign(practitioner)

Routing tasks by stage rather than letting Celery's default queue handle everything means we can scale worker pools independently — assignment workers need to be fast and numerous during business hours, while review-adjacent tasks are steadier throughout the day.

Backpressure and retries

The trickiest part of an event-driven queue is handling the case where an event fires but the downstream system genuinely can't act on it yet — for example, no practitioner is currently available for a given condition. Rather than treating that as a failure, we treat it as a retry with backoff, capped at a maximum wait before the request escalates to a broader practitioner pool:

• If no practitioner is available, the task retries with exponential backoff instead of failing immediately, so a temporary capacity gap doesn't require a customer to resubmit.
• If retries are exhausted, the request escalates automatically to an on-call practitioner pool rather than sitting unassigned.
• Every retry increments a counter on the request record, which feeds our alerting so a sustained spike in retries pages the on-call engineer before customers notice a delay.

We also cap queue depth per stage. If the assignment queue backs up past a threshold, intake pauses accepting new requests into that queue and returns a "still processing" status to the client instead of accepting requests it can't service in a reasonable time.

Results and operational notes

The event-driven redesign cut median time-to-assignment substantially, but the bigger win was operational clarity: because each stage is its own queue with its own metrics, an on-call engineer can look at queue depth and retry counts and immediately tell whether the bottleneck is intake volume, practitioner availability, or a downstream review delay. That kind of legibility was effectively impossible with a single polling loop, where every slowdown looked the same from the outside regardless of its actual cause.