Most LLM serving tutorials demonstrate the happy path: one request, one response, warm cache, no concurrency. Production behavior is different. The failures that take a service down are cold starts after a rolling deploy, cache stampede when a key expires, and memory growth under sustained load.

This article compares two stack shapes that teams commonly reach for:

• A compiled service: a Rust inference core compiled to WebAssembly, hosted in a small runtime, behind a hand-written reverse proxy. The appeal is a flat memory profile and fast cold starts.
• A Python async API: an ASGI framework serving an inference engine, with a Redis cache and a CDN in front. The appeal is iteration speed and a mature ecosystem.

The comparison is organized around the metrics that actually break: warm-up latency, cache stampede resilience, memory footprint, and the operational cost of each failure mode. Every number below is either a documented default, arithmetic shown from stated assumptions, or explicitly labeled illustrative.

Why this comparison matters now

Two things dominate LLM serving cost and reliability: time-to-first-token under load, and whether the prompt cache stays warm without hammering the backing store.

Teams that treat inference caching as a configuration toggle usually discover the problem during a traffic spike. A cache stampede — many concurrent requests missing the same key and all recomputing — converts a cache miss into a compute surge. The database or cache layer absorbs the retry storm, latency climbs, and the bill follows.

The stack choice affects how easily each failure mode is contained. A compiled service tends to have predictable memory and fast process start. A Python async service tends to have faster iteration and richer libraries but a larger baseline footprint and more dependency surface.

Option A: the compiled Rust service

The shape: a Rust crate compiled to a WebAssembly module, loaded by a host runtime, fronted by a small proxy that handles connection pooling to Redis and enforces timeouts.

The build is deliberately explicit. You compile the crate to wasm32-unknown-unknown, bundle it with the runtime, and run it behind the proxy. The proxy owns keep-alive pooling and per-stage timeouts, for example a 500 ms inference budget and a 25 ms cache-lookup budget.

Where this stack shines:

• Constrained VMs. A small droplet can serve modest traffic without swapping, provided the cache hit rate stays high.
• Flat memory. Once the module is stable, resident memory tends to plateau rather than climb.
• Fast cold start. Process start is dominated by module load, not interpreter and library import.

The trade-offs:

• Build time. Compiling to WebAssembly is slower than installing a wheel. On a 4-core laptop this is minutes, not seconds.
• Runtime pinning. The host runtime version matters. Version drift between the runtime and the compiled module is a common source of memory leaks under load.
• No native WebSocket story. Real-time streaming requires a custom proxy or a separate service.

The allocator trap

A frequent failure mode with WebAssembly modules in a managed runtime is forgetting that the module needs a tuned allocator. Without one, the binary grows and cold-start latency climbs sharply.

[features]
default = ["wee_alloc"]

The fix is one line in Cargo.toml and a rebuild. Without it, the runtime is effectively unusable for production workloads — the module's memory footprint grows unbounded under sustained request volume.

How to measure the compiled stack

Instrument three things:

1. Cold-start latency. Restart the process and time the first request end to end. Repeat at least 20 times; report the median and p99, not a single sample.
2. Warm-start latency. Issue the same request twice with a warm cache and time the second call.
3. Resident memory under load. Sample RSS every 10 seconds for an hour at your target RPS. A flat line is the goal; a monotonic climb means the allocator or the host runtime is leaking.

A simple load generator with a fixed concurrency and a fixed duration is enough. The point is to hold concurrency constant and watch memory and p99 together.

Option B: the Python async API

The shape: an ASGI framework serving an inference engine, with a Redis cache and a CDN in front. The framework generates OpenAPI docs for free, the CDN absorbs static traffic, and Redis holds the prompt cache.

Where this stack shines:

• Iteration speed. A prompt template change is picked up by the reloader in well under a second.
• Streaming. Native support for Server-Sent Events and WebSockets removes most of the boilerplate for chat UX.
• Observability. Auto-instrumentation for tracing and metrics is mature; cache client libraries propagate trace context out of the box.

The trade-offs:

• Larger baseline footprint. The interpreter, the inference engine, and the numerical libraries add up.
• Dependency churn. Inference engines pin specific framework and driver versions. Upgrading the OS can break the wheel.
• The GIL. CPU-bound work does not parallelize across threads; you scale with processes, which multiplies memory.

The eviction-policy trap

A frequent failure mode is leaving Redis at the default eviction policy. Under memory pressure with noeviction, writes fail and reads miss; the cache effectively becomes random-access garbage and the inference engine recomputes on every request.

redis-cli config set maxmemory-policy allkeys-lru

The fix is one config line and a restart. Without it, latency spikes on every eviction cycle and the stack becomes unusable under load.

How to measure the Python stack

The same three instruments apply, plus one more:

1. Cold-start latency. Restart the worker and time the first request. Repeat 20 times.
2. Warm-start latency. Time the second identical request.
3. Resident memory per worker. Sample RSS per worker process, not just the parent.
4. Cache hit rate and eviction rate. redis-cli info stats gives keyspace_hits, keyspace_misses, and evicted_keys. A rising evicted_keys with a falling hit rate is the signature of the eviction-policy trap.

The dependency-pin trap

A common failure mode is a wheel mismatch after an OS upgrade. The inference engine pins a specific framework and driver combination; a fresh instance pulls a wheel built against a different driver and fails at import time.

pip install torch==2.4.0+cu124 --extra-index-url https://download.pytorch.org/whl/cu124

Pinning the exact wheel avoids the mismatch. Without the pin, the deploy fails and the on-call engineer debugs a driver error that only appears under load.

Head-to-head: what actually differs

The table is a qualitative comparison. The quantitative answer depends on your hardware, your cache hit rate, and your concurrency. Measure all three.

A worked example: cache stampede arithmetic

Assume the following, all stated as assumptions rather than measured facts:

• Steady-state traffic: 1,000 requests per second.
• Cache hit rate: 94%, so 60 requests per second miss.
• Compute cost per miss: 1.2 seconds of inference.
• Cache TTL: 300 seconds, with no jitter.

With a uniform TTL, all keys written in the same window expire together. The number of keys expiring per second is the write rate divided by the TTL. If the write rate equals the miss rate, that is 60 / 300 = 0.2 keys per second expiring — but each expiration triggers a burst of concurrent misses from every in-flight request for that key.

The fix is TTL jitter. Spreading TTLs uniformly over 300 ± 60 seconds reduces the probability that two keys expire in the same second by roughly the ratio of the jitter window to the TTL window. The arithmetic here is illustrative: the point is that a uniform TTL concentrates expirations, and jitter spreads them.

The second fix is single-flight: on a miss, one request computes and the rest wait on the same future. This converts N concurrent misses into one compute and N-1 waits. The compiled stack typically implements single-flight in the proxy; the Python stack typically implements it with a cache client lock. Both are worth measuring.

Failure-mode analysis

Three failure modes recur across both stacks:

Cold-start amplification. After a rolling deploy, every instance is cold at once. If the load balancer sends traffic before the first request completes, p99 spikes for the duration of the warm-up. Mitigation: pre-warm with synthetic requests before adding the instance to the pool, and use a readiness probe that only passes after a warm-up request succeeds.

Cache stampede. Covered above. Mitigation: TTL jitter plus single-flight.

Memory creep. The compiled stack creeps when the allocator is untuned or the host runtime version drifts. The Python stack creeps when worker processes accumulate state or when the inference engine caches without bounds. Mitigation: sample RSS per process on a fixed interval and alert on a monotonic trend, not a threshold.

Decision checklist

Use this to decide, in order:

1. Is your traffic under roughly 50 RPS? Build time and maintenance overhead dominate. Pick the stack your team already ships in.
2. Do you need real-time streaming? If yes, the Python async stack wins by default unless you are prepared to write a proxy.
3. Is your cache hit rate stable above 95%? If not, fix the cache before choosing a stack. A stampede will hurt either one.
4. Can you tolerate a multi-minute build cycle? If not, the compiled stack's iteration loop will slow you down.
5. Do you have a dedicated person for tracing and export? If not, the Python stack's auto-instrumentation is worth more than the compiled stack's memory profile.
6. Is your traffic spiky but predictable? Pre-warm before the spike. This matters more than the stack choice.

A common blind spot is assuming the compiled stack always saves money. It does not at low traffic, where build-time and maintenance overhead outweigh the savings. Conversely, the Python stack becomes expensive when the cache hit rate drops below roughly 85% under load, because every miss recomputes at full inference cost.

Verdict

For a production LLM feature where cold-start latency and memory footprint are the binding constraints, and the team can maintain a build pipeline, the compiled Rust service is the stronger choice. For a team that ships Python daily, needs streaming, and has predictable traffic under a few hundred RPS, the Python async stack is faster to ship and cheaper to run.

The deciding factor is rarely the language. It is whether the cache is configured correctly, whether single-flight is implemented, and whether cold starts are pre-warmed. Fix those three things first; then the stack choice becomes a matter of team fit.

Do this in the next 30 minutes

Run redis-cli info memory and check the maxmemory_policy field. If it is not allkeys-lru (or another eviction policy appropriate to your workload), set it with redis-cli config set maxmemory-policy allkeys-lru and restart Redis. Then run redis-cli info stats and record keyspace_hits, keyspace_misses, and evicted_keys as your baseline. If evicted_keys is climbing, you have a cache sizing problem to fix before you ship another feature.

Related reading

• AI-native apps: forget the old stack
• Serverless Cold Starts: Cost Levers and How to Measure Them
• AI features as tech debt or golden paths

Official documentation

• Redis documentation
• WebAssembly
• The Rust Programming Language