Idempotency in practice: webhooks, scheduled jobs, and agent tools
Idempotency is easy to define and easy to get wrong. An operation is idempotent if running it twice has the same effect as running it once, which matters the moment anything can retry: a webhook redelivered, a scheduled job that fired twice, a network call whose response was lost. Here are three places we enforce it, and how the mechanism differs in each.
Webhooks: a natural key and a unique index
Payment providers retry webhooks, sometimes minutes apart, sometimes the same event twice in a second. The defense is a natural key drawn from the payload, in our case the pair of payment source and the provider's transaction reference, with a unique index on it. On each delivery we look up that key before writing. If the record exists we return the existing result and do nothing else; if it does not, we insert and process. The provider can deliver the same event any number of times and the ledger gains exactly one row. The index is what keeps that lookup cheap enough to run on every delivery at webhook volume.
The subtle part is what you return on a duplicate. The caller should get a success, not an error, because from the provider's side the delivery did succeed. It simply was not the first.
Scheduled jobs: idempotency by natural key, not by run
A monthly settlement job retries for a different reason: running the same period twice must not produce two settlements. We key the settlement on its natural identity, the account and the period, and upsert, so a re-run recomputes and overwrites rather than appends. The job is safe to repeat because its output is a function of its inputs for that period, independent of how many times it executed.
Two edge cases bite here. The schedule has to clamp the run day to the days the month actually has, or a job set for the 30th silently skips February. And a crash partway through has to leave the period either fully settled or untouched, never half, which means the recompute is transactional or fully overwriting.
Agent tools: the idempotency you usually do not have
Agent tool calls are the case where idempotency is often missing, and sometimes that is fine. A retried read (re-list the alarms, re-fetch the inverter data) just repeats a query and costs nothing. The danger is tools that write. A tool that creates a work order or sends a command needs the same treatment as a webhook: a caller-supplied idempotency key on the call, checked before the side effect, so a retried step does not double-act. We give write-capable tools an explicit operation id and dedupe on it, and we keep read tools cheap to repeat.
The pattern under all three
Each case is the same move in different clothing. Find the natural identity of the operation, make the side effect conditional on not having seen that identity before, and decide what a repeat should return. Reach for a unique constraint when there is a natural key, an idempotency key when the caller can supply one, and a transactional upsert when the operation recomputes a known slot. The failure you are preventing never changes: a duplicated effect in an at-least-once world.
Where AgentKick fits
We build the reliability layer for systems where a duplicated effect costs real money or real trust: payments, settlement, and agents that take actions. If retries, webhooks, or scheduled jobs are producing duplicates, or could, that is the work we do, usually as a short scoping engagement into a phased build.