A payment agent that works on a laptop and fails in the cloud after a few minutes of runtime is a well-documented pattern. The cause is usually not the payment logic. It is a mismatch between the agent's connection lifecycle and the provider's session contract.

The error and why it's confusing

An agent submits USSD requests to a mobile-money provider on behalf of a user paying from a phone wallet in local currency. It runs correctly against a local test harness. Deployed to a cloud runtime, it starts returning ERR_PAYMENT_FAILED after two to three minutes. The provider's payment logs show successful transactions. The agent's own logs show no exceptions. The error string is generic, and the provider documentation often says nothing about session timeouts.

The mismatch is between a stateless HTTP client loop and a stateful USSD session contract. USSD gateways were built for feature phones, where a session is a short-lived, interactive exchange. Many gateways enforce a hard idle timeout — commonly around 90 seconds — and release session state when it expires. A modern HTTP agent that keeps a long-lived connection pool reuses the same TCP socket across requests. If the provider maps a socket to a session, an idle gap longer than the timeout invalidates that session. The next request over the same socket is rejected, and the rejection surfaces as a payment error rather than a session error. This is silent session invalidation disguised as a payment failure.

The confusion compounds because health checks and balance queries may keep returning 200 OK on the same client. Only the transaction path fails, and only after an idle period.

What's actually causing it

Three mechanisms are responsible, and they can overlap.

Socket-to-session binding. Some gateways treat an open TCP connection as a session. Connection pooling — for example an HTTP client configured with limits=10 — keeps sockets open between requests. After the idle timeout, the gateway discards session state but the socket remains open on the client side. The next write goes to a socket the provider no longer recognises.

Keep-alive that does not reset the timer. Some providers expose a keep-alive or ping endpoint. A common trap is assuming a ping resets the session timer. On legacy SS7-backed gateways, keep-alive traffic can be advisory: the gateway still enforces the idle timeout independently. The documentation rarely states this, and teams typically discover it only from traffic captures.

Egress-layer idle timers. Cloud NAT gateways and load balancers have their own TCP idle timeouts. A NAT gateway may hold a connection open for several minutes. If the provider's session timeout is shorter, the socket survives past the point where the provider has forgotten the session. The next request is then routed over a socket the provider considers new or invalid. This is environment-specific: it depends on the egress timeout of the runtime, and it is more likely to appear on platforms with short default NAT timeouts and cold-start behaviour.

A typical failure sequence: an agent drains a queue of local-currency payment requests. It reuses one HTTP client to avoid TLS handshake overhead. A gap between requests exceeds the gateway idle timeout. The provider drops session state. The agent sends the next request on the same socket and receives ERR_PAYMENT_FAILED. The agent classifies this as a payment failure and retries, which can double-charge or exhaust the user's balance if the retry lands on a fresh session.

Diagnosing before fixing

Do not apply fixes blind. Establish which mechanism is active.

  1. Capture traffic. Run tcpdump -i any -w momo.pcap port 443 on the agent host (or a sidecar) for five minutes, then inspect in Wireshark. Look for whether the client reuses one TCP stream across requests, and whether the server sends FIN or RST after an idle period.
  2. Log connection identity. In Python with httpx, log id(response) and the underlying transport's connection info per request. In Node.js, log the socket's local port. If the local port is identical across requests separated by more than the gateway timeout, socket reuse is confirmed.
  3. Correlate timing. Plot request timestamps against failures. A cluster of failures at a fixed offset after the last successful request is strong evidence of an idle timeout.
  4. Check the provider's response headers. Some gateways return X-Session-Timeout, X-Idle-Timeout, or a session identifier in headers. If present, record the value; it is the authoritative number, not a guess.
  5. Compare environments. Run the same agent against the same provider from a local machine and from the cloud runtime. If only the cloud path fails, an egress timer is involved.

Only after this step does it make sense to choose a fix.

Fix 1 — session per transaction

The most direct fix is to stop sharing connections across transactions. Create a fresh HTTP client per request, or per transaction, so each payment gets a new socket and therefore a new session.

# Before — shared client, session reuse, eventual ERR_PAYMENT_FAILED
import httpx

client = httpx.Client(timeout=30.0, limits=httpx.Limits(max_connections=10))

# After — new client per request, matching the per-session contract
import uuid
import httpx

def send_payment(payload: dict) -> str:
    with httpx.Client(timeout=30.0) as client:
        r = client.post(
            "https://example-provider.invalid/v2/ussd/send",
            json=payload,
            headers={"X-Reference-Id": str(uuid.uuid4())}
        )
        r.raise_for_status()
        return r.json()["status"]

The tradeoff is a TLS handshake per request: added latency, more ephemeral source ports, and possibly more NAT gateway traffic. To measure whether this is acceptable, instrument the following before and after the change:

  • Median and p99 request latency, from the agent's own timing, with the client creation included.
  • Count of ERR_PAYMENT_FAILED responses per thousand requests.
  • Number of distinct source ports observed by the provider, if it exposes that, or locally via ss -tn sampling.
  • NAT gateway or egress cost line item, if the platform bills per connection or per GB processed.

A pool of two or three shared clients does not fix this. If the provider binds sessions to sockets and enforces an idle timeout, any socket that idles past the timeout is invalid, regardless of pool size.

Fix 2 — explicit session identifiers

Newer REST/JSON gateways expose an explicit session identifier that the client must echo back. Where that is available, use it. The session identifier, not the socket, becomes the unit of continuity, and the provider's timer resets on each request that carries a valid identifier.

import uuid
import httpx

async def send_with_session(payload: dict, session_id: str | None = None):
    headers = {"X-Reference-Id": str(uuid.uuid4())}
    if session_id:
        headers["X-Session-Id"] = session_id
    async with httpx.AsyncClient(timeout=30.0) as client:
        r = await client.post(
            "https://example-provider.invalid/v2/ussd/send",
            json=payload,
            headers=headers
        )
        data = r.json()
        return data["status"], data.get("sessionId")

Two cautions. First, confirm in the provider's documentation that the session identifier is intended to be reused across requests within a transaction, and whether it has its own expiry. Second, propagate the identifier across retries and async workers. If a retry starts without the identifier, it creates a new session, which may duplicate a payment.

To measure the effect, track the same error rate and latency metrics as in Fix 1, plus the rate of ERR_SESSION_INVALID responses. A rise in that error alongside a fall in ERR_PAYMENT_FAILED indicates the session model is now explicit and observable rather than silent.

Fix 3 — egress timers

When the provider's session timeout is shorter than the cloud egress idle timeout, the socket can outlive the session. The fix is to make the client's connection lifetime shorter than the provider's session timeout.

Options, in order of preference:

  • Disable connection reuse for the transaction path. This is Fix 1, and it removes the egress timer from the equation entirely.
  • Set the socket keepalive and user timeout. On Linux, TCP_USER_TIMEOUT bounds how long unacknowledged data is retried before the socket errors. This is set per socket and typically requires a container image or runtime that exposes socket options. It is not a portable fix.
  • Reduce the NAT idle timeout. Some cloud NAT services allow the TCP idle timeout to be configured. Check the platform's documented minimum. Note that the tag-based configuration shown in some examples is not a real AWS feature: NAT gateway idle timeout is a property of the NAT gateway resource, not an arbitrary tag. Verify the current API surface before relying on it.
  • Send traffic more often than the shortest timer. A periodic lightweight request on the same connection can keep both the egress socket and the provider session alive, provided the provider's keep-alive semantics actually reset the timer. Confirm this from a traffic capture rather than assuming it.

The decision rule is simple: make the shortest timer in the chain the one you control, and make the client's connection lifetime shorter than it.

How to verify the fix

Pick three metrics and watch them for at least a week, because mobile-money traffic is bursty and a single day can mislead.

  • ERR_PAYMENT_FAILED count per thousand transactions, segmented by provider and by endpoint path.
  • Median and p99 latency, including client construction time.
  • Egress cost per thousand transactions.

A fix is working when the error rate falls and stays low across at least one full weekly traffic cycle, latency stays within an acceptable band, and cost does not rise disproportionately to the error reduction. There is no universal "good" error rate: the correct baseline is the provider's own documented success rate for the endpoint, plus your agent's own historical floor under known-good conditions.

For log-based checking, a query such as the following surfaces bursts:

filter @message like /ERR_PAYMENT_FAILED/
| stats count(*) as errors by bin(5m)
| filter errors > 5

If failures persist, capture the provider's response headers for a failing request and compare them against a passing request. Differences in session or timeout headers are the fastest way to confirm the contract.

Preventing recurrence

Add a contract test that exercises the idle boundary. The test does not need to run on every commit; a nightly job is usually enough.

# tests/test_provider_session.py
import time
import uuid
import httpx
import pytest

@pytest.mark.asyncio
async def test_session_timeout_recovery():
    payload = {"amount": "100", "currency": "GHS", "msisdn": "233551234567"}
    url = "https://example-provider.invalid/v2/ussd/send"

    async with httpx.AsyncClient(timeout=30.0) as client:
        r = await client.post(url, json=payload, headers={"X-Reference-Id": str(uuid.uuid4())})
        assert r.status_code == 200

    # Idle past the provider's documented session timeout.
    time.sleep(95)

    async with httpx.AsyncClient(timeout=30.0) as client:
        r = await client.post(url, json=payload, headers={"X-Reference-Id": str(uuid.uuid4())})
        if r.status_code == 200:
            return
        data = r.json()
        assert "ERR_SESSION_INVALID" in data.get("status", "")

Two design notes. The test asserts that a post-idle failure is reported as a session error, not a payment error — that is the contract you actually care about. And the idle duration should come from configuration, not a hard-coded 95, so it can be tuned to the provider's documented timeout.

Run this in staging against the provider's sandbox. Running it against production is a cost and risk decision that belongs to the team operating the agent.

  • ERR_SESSION_INVALID — the provider explicitly invalidated the session. This is the well-behaved version of the failure. Fix: use explicit session identifiers and refresh them on expiry.
  • ERR_RATE_LIMIT — throttling, often caused by retry storms after session failures. Fix: reduce concurrency and apply backoff before increasing retry count.
  • ERR_INVALID_SIGNATURE — a signature or HMAC mismatch, sometimes because the agent reused a key tied to an expired session. Fix: regenerate the signature per request.
  • ERR_TIMEOUT — a TCP-level timeout before an HTTP response. Fix: distinguish this from session invalidation in your metrics, then adjust client timeouts and add a circuit breaker.

Retry policy that does not burn money

Retries are where session confusion becomes expensive. Two rules reduce most of the damage.

First, classify the error before retrying. A session invalidation is not a payment failure. Retrying a session invalidation on a fresh connection is usually safe; retrying an ambiguous payment failure is not, because the original may have succeeded.

Second, bound the retries and the window.

import random

def retry_delay(attempt: int) -> int | None:
    if attempt >= 3:
        return None
    return int(min(2 ** attempt + random.uniform(0, 1), 10))

for attempt in range(3):
    try:
        send_payment(payload)
        break
    except PaymentFailed:
        delay = retry_delay(attempt)
        if delay is None:
            raise
        time.sleep(delay)

The cap of three attempts and the maximum delay of ten seconds are illustrative choices, not provider requirements. The important properties are: bounded attempts, jitter to avoid synchronised retries, and a hard stop that escalates to a human or a reconciliation job rather than looping.

Escalation checklist

If the failure survives all of the above, escalate with evidence rather than a description.

  • A short packet capture (tcpdump -w momo.pcap) covering at least one failure, showing socket reuse and the provider's close behaviour.
  • Request and response logs for one failing transaction and one passing transaction, including reference identifiers and any session identifier.
  • A minimal reproduction script that runs in staging.
  • The provider's documented session timeout, if any, and the observed timeout from the capture, if they differ.

The gap between documented and observed behaviour is usually the fastest thing a provider's support team can explain, and it is the detail most likely to be missing from a first escalation.

FAQ

Why does the agent fail after roughly the same interval each time in production but not locally?

Locally, connections are typically short-lived and each request looks like a fresh session. In production, a pooled connection survives across requests, and an egress layer may keep it open longer than the provider's session timeout. The fixed interval is the provider's idle timeout, not a property of the payment logic.

Does reducing the connection pool size help?

Not on its own. If the provider binds sessions to sockets and enforces an idle timeout, any socket that idles past the timeout is invalid. Pool size changes how many sockets exist, not how long each one lives.

How can latency be reduced after switching to per-request clients?

Use protocol features the provider supports, such as HTTP/2, for endpoints that do not require fresh sessions. Keep the per-request pattern only for the transaction path. Measure the split before assuming it helps: the handshake cost and the protocol benefit may cancel out on short requests.

Can sessions be shared across async workers?

Only if the provider exposes an explicit session identifier and the agent propagates it correctly. A shared store with a TTL shorter than the provider's session timeout is the usual pattern. The failure mode to avoid is a worker picking up an identifier that has already expired, which produces the same silent invalidation the fix was meant to remove.

What is the safest retry strategy?

Classify errors first, cap attempts, add jitter, and stop rather than loop. Treat an ambiguous payment failure as potentially successful until reconciled.

Do this in the next 30 minutes

Run one packet capture against your staging environment while sending two requests separated by 120 seconds, then open the capture and check whether the client reused the same TCP stream and whether the server closed it in between. That single observation tells you whether your provider binds sessions to sockets, which determines which of the three fixes above applies to your agent.

Official documentation