Build Logs21 October 20268 min read

Retry a Call Only When the Call Is Safe

A retry helps when the first attempt did nothing. It hurts when the first attempt committed and the response was lost. Know which call you are making.

VikasSoftware Engineer
FIGURE 1.0
EDITORIAL REFERENCE
Network cables carrying traffic between systems
CONTEXT:Network cables carrying traffic between systems
REFERENCE:Compare Fibre
ON THIS PAGE (8 SECTIONS)
EXECUTIVE BRIEFING · KEY TAKEAWAYS
  • ✓HTTP calls GET, PUT, and DELETE idempotent. POST is not, unless you add your own key.
  • ✓A timeout means you do not know whether the server finished. That is the dangerous case.
  • ✓Cap attempts and wait between them. An immediate loop turns a short outage into a stampede.
  • ✓Inbound webhook deduplication is a different problem from outbound retries.

The payment provider took four seconds and your client gave up at three. The charge may have been created. The code retries because the error handler says "network errors are retryable." A second charge lands. The bug is not the timeout. The bug is retrying a write that was not safe to repeat.

RFC 9110 defines idempotent methods as those for which the intended effect of several identical requests is the same as one. GET and HEAD are safe. PUT and DELETE are idempotent: repeating them leaves the same state, even if the first response was lost. POST is not. A retry policy that ignores that distinction will duplicate the one operation you most wanted to happen once.

Decide from the operation, not from the exception type

What you calledIf the response is lostWhat to do
ReadNo state changedRetry with a limit and a pause
PUT of a full resource, or DELETEThe intended end state is the sameRetry the same request
POST that creates a recordA record may already existRetry only with an idempotency key the server stores
POST that sends a message or takes moneyThe side effect may have happenedDo not blind-retry. Reconcile, or send the same key

A pause matters. Ten clients retrying immediately will hit a dependency that is already failing. Wait, increase the wait, and stop after a small number of attempts. After the last attempt, record the uncertainty: the call may have succeeded. A person or a reconciliation job should be able to see that state. Hiding it inside a catch block is how duplicate rows are discovered a week later.

This is not the webhook article

Webhook Idempotency at Scale is about someone else calling you twice. This article is about you calling someone else and not knowing whether they finished. The same idea shows up in both: a key that makes a second delivery a no-op. On the outbound side, you generate the key and the server must honor it. If the server will not, you cannot safely retry a create. You can only read back whether the first one landed.

Illustrative scenario

This scenario is illustrative. An order service creates a shipment by POST. The carrier times out. The order stays in shipment_unknown, with the key that was sent. A worker asks the carrier for that key before it sends another request. If the carrier has the shipment, the order moves to shipped. If the carrier has never seen the key, the worker sends one request. It does not loop from the user request that already returned.

Checklist

  • Classify each outbound call as read, idempotent write, or create.
  • Retry reads and idempotent writes with a cap and a pause.
  • Give creates a key the receiver stores, or do not retry them automatically.
  • Treat timeout as unknown, not as failure.
  • Surface the unknown state. Do not leave it only in a log line.
  • Load-test the pause. A retry storm is a second incident.

Questions that usually come next

Is a 500 always safe to retry?

No. The server may have committed and then failed while writing the response. Retry a 500 on a create only when the idempotency key makes the second call a no-op.

Where should the retry live?

Not in a loop inside the user request if the user is waiting and the dependency is down. Record the attempt and continue from a worker that can pause. The user request should return a state, not hold a connection open for a chain of retries.

Direct answers

Is a 500 always safe to retry?

No. The server may have committed before the response was lost. Retry a create only when an idempotency key makes the second call a no-op.

Where should the retry live?

In a worker that can pause, once the user request has recorded the attempt. A loop inside the waiting request turns one timeout into many.

ACADEMIC & INDUSTRY CITATIONS

References & Prior Art

REF 01Internet standard

RFC 9110: HTTP Semantics

R. Fielding, M. Nottingham, J. Reschke • IETF (2022)

Source
VERIFIED CONTRIBUTOR

Vikas

Software Engineer

Vikas focuses on event architectures, webhook idempotency, queue resilience, and sequential product development workflows at Vasiva.

CURATED ARCHIVE

Continue Reading

Contextually relevant engineering logs and production architecture notes.

SYSTEM ADVISORY

Have an AI workflow or complex system worth exploring?

Work directly with the senior engineers who designed and shipped these systems. Zero vendor lock-in, deterministic architectures, and first-party audit integrity.

Evaluate the opportunity