Orchestrate the saga; compensate, never two-phase commit
- When
- A money movement spans three internal services plus an external gateway, and any step can fail independently — but a classic 2PC coordinator that dies after phase-1 commit (Robinhood's 72-hour ghost transfer) leaves funds in limbo with no recovery path.
- AWS
- Model the flow as a Step Functions Express Workflow: each forward step is an idempotent Lambda task, and each has an explicit compensating task (ReleaseBalance, VoidAuth) wired through a Catch into a reverse path. Every compensation is a guarded state transition (UPDATE ... WHERE status = :expected_status AND version = :v), not a relative delta, so an at-least-once replay matches zero rows and is a safe no-op; VoidAuth reuses the forward call's gateway idempotency-key. Sync execution returns the result in under the 800 ms checkout budget; the 5-minute Express ceiling is ample for synchronous auth.
- Trade-off
- You give up the illusion of a single atomic transaction and accept windows where the system is in a known-intermediate state (reserved-but-not-captured) that compensation must unwind; you must make every step AND every compensation idempotent — guarded transitions, never relative deltas — and reason about compensations that themselves fail, which a DLQ for human review backstops. Express does not persist execution history to the service (only CloudWatch Logs), so the authoritative state is the DynamoDB idempotency slot, not the workflow console.
from: Real-time payment processing with distributed sagas
Gate every payment with a DynamoDB conditional put
- When
- A client retry races the original request to completion and both reach the gateway — the Stripe-2013 double-charge — so the very first thing on the path must collapse all copies of one logical request onto one outcome.
- AWS
- On entry, PutItem to a DynamoDB idempotency table with ConditionExpression attribute_not_exists(pk), pk = sha256(merchant_id + idempotency_key). First writer wins the slot (status PROCESSING, 24 h TTL); a ConditionalCheckFailed means the request is already in flight or done, so return the cached response, never re-execute.
- Trade-off
- You add ~5 ms and one strongly-consistent write to every payment, and you inherit a stuck-PROCESSING edge case (caller crashed mid-saga) that needs a lease timeout to reclaim — in exchange for a hard exactly-once boundary that volatile caches like Redis cannot guarantee.
from: Real-time payment processing with distributed sagas
Reconcile against the gateway before settlement, on a heartbeat
- When
- Even with idempotency, sagas, and locks, a dropped async event or a partition can leave internal ledger and the gateway's view divergent — a captured charge with no ledger row, or a Monzo-style card reserve the merchant never captures and the bank silently releases at T+7d.
- AWS
- Push, not pull: subscribe to the gateway's real-time settlement webhook feed and land it via a second Kinesis Firehose stream into S3 Parquet, alongside ledger_entries snapshots — pulling 36M records/hour through a paginated API would not fit a Lambda timeout. An Athena join flags divergence to SNS. An EventBridge Scheduler heartbeat re-checks only pending reservations at T+7d, T+14d, T+30d (small volume) so a dropped release event cannot drift the ledger past settlement.
- Trade-off
- Reconciliation is eventually-consistent (hourly, not per-transaction) and is a detective control, not a preventive one — it catches and surfaces divergence rather than stopping it, which is acceptable because settlement is T+1 and gives the window time to resolve before money actually moves.
from: Real-time payment processing with distributed sagas
Idempotency-key dedup gate with atomic conditional insert
- When
- A payment command can be retried — by a flaky client, an API Gateway retry, an SQS redrive — and two retries can arrive concurrently before the first has committed. A naive read-then-write check ('does this key exist yet?') lets both retries pass and double-charges the account.
- AWS
- A DynamoDB idempotency table keyed on tenant_id#sha256(client_id + request_id + amount + currency + timestamp_bucket) — the tenant_id prefix lets dynamodb:LeadingKeys clamp each tenant's credential to its own partition space. The first thing every command does is a conditional PutItem with attribute_not_exists — an atomic compare-and-set that claims the key with status=PENDING. Exactly one writer wins the partition; the loser catches ConditionalCheckFailedException and returns the stored prior result. The Aurora journal commits as a SEPARATE phase (DynamoDB and Aurora are distinct transactional domains — there is no 2PC between them), then the claim flips to status=COMPLETE. A sweeper Lambda (EventBridge every 5 minutes) resolves any claim stranded in PENDING against Aurora — completing it or releasing it for retry. A 30-day TTL reaps the keyspace.
- Trade-off
- This is safe eventual consistency, not a single atomic unit: there is an observable window between the claim and the journal commit, bounded to ~5 minutes by the sweeper rather than to the 30-day TTL. The conditional write adds one synchronous round-trip on the hot path, and the timestamp_bucket bounds the dedup window. The idempotency store is a correctness-critical dependency, not a cache: if it is unavailable you must fail closed.
from: Financial ledger and double-entry accounting at scale
Deferred double-entry invariant (SUM of entries = 0 per transaction)
- When
- Every money movement must conserve value: for each transaction the debits and credits must net to exactly zero, or money was created or destroyed. But the debit row and the credit row cannot both be inserted in the same instant, so a row-level check would reject the half-written transaction.
- AWS
- Aurora PostgreSQL holds immutable journal entries (journal_entry_id, transaction_id, account_id, amount_cents BIGINT, direction). A DEFERRABLE INITIALLY DEFERRED constraint (or a constraint trigger) evaluates SUM(signed_amount) GROUP BY transaction_id = 0 at COMMIT, not per row — so a transaction can write its debit and credit legs and is only validated when complete. Amounts are integer cents (BIGINT), never floats, so the sum is exact. Entries are append-only: no UPDATE, no DELETE; a reversal is a new compensating transaction.
- Trade-off
- Deferred constraints defer the failure to commit time, so the application must handle a late rollback of an otherwise-accepted transaction. Integer cents means currency precision is fixed at the minor unit — sub-cent intermediate math (FX, interest accrual) must round explicitly and book the rounding remainder somewhere.
from: Financial ledger and double-entry accounting at scale
Saga with compensating entries for cross-account transfer
- When
- A transfer debits account A and credits account B. If the process dies after the debit but before the credit, money is stranded — destroyed from A, never created in B. A naive two-step write has no atomicity across the legs.
- AWS
- Model the transfer as a single Aurora transaction when both legs share a database (the deferred SUM=0 constraint then enforces atomicity for free). When legs cross service boundaries, use a Step Functions Express saga: each step is idempotent on transaction_id#step_name, and any failure after the debit triggers a compensating reversal transaction (a new credit back to A) rather than a destructive rollback. Express (at-least-once, ~$8k/month at 2,000 sagas/s) is chosen over Standard (exactly-once, ~$648k/month) because exactly-once belongs on the per-step idempotency key, not on the orchestrator — which the design already provides.
- Trade-off
- A saga is eventually consistent and money is briefly in an intermediate state (debited from A, not yet credited to B) visible to reconciliation. Express gives at-least-once execution, so each step must be idempotent — a burden the design already carries. Compensation is itself a journal entry, so the audit trail shows the failed-and-reversed path rather than hiding it — the ledger records attempts, not just successes.
from: Financial ledger and double-entry accounting at scale
Idempotency key with cached response replay
- When
- A mutating operation has an external side effect (a charge) that must run exactly once even when the response is lost to a timeout and the client retries, and concurrent retries can race in. You need an explicit identity for the operation - not a guess from request fields - and a way to return the original answer on every retry.
- AWS
- Client generates a V4 UUID once per logical operation and sends it as an Idempotency-Key header, reusing it across retries. The server persists ((tenant_id, idempotency_uuid), request_hash over a canonical form, status, response, expires_at, lease_expires_at) in Aurora PostgreSQL via INSERT ... ON CONFLICT DO NOTHING. First caller executes and stores the outcome; later callers replay the cached response on terminal status, get 409 while PROCESSING, and 422 when the same key arrives with a different request hash. Reads filter on expires_at in application code so best-effort TTL deletion is never a correctness boundary; expiry is jittered to avoid a stampede.
- Trade-off
- Every mutating request now pays a key-store round trip before doing work, and clients must persist and correctly reuse the key across retries (a new key means a new charge). Binding the key to a canonicalised request hash means a legitimate retry with any payload drift is rejected with 422 rather than silently replayed. The PROCESSING lease must be tuned: too short and a slow saga gets reclaimed mid-flight, too long and a crashed saga stays stuck until the sweeper finds it.
from: Idempotent payment gateway
Append-only ledger with streamed tamper-evident audit
- When
- A financial system must record every state transition (created, charged, settled, voided) as an immutable system of record, prove to an auditor that no record was altered, and fan the terminal outcome out to downstream consumers without coupling them to the charge path.
- AWS
- Each saga transition is appended to a DynamoDB ledger that is the system of record for SETTLED and VOIDED states. DynamoDB Streams feeds those transitions to S3 with Object Lock (WORM) in a dedicated logging account, so records cannot be altered or deleted within the retention window even by an admin - satisfying PCI Req. 10 and SOC 2 CC7 / NIST AU-9. Terminal outcomes also emit payment.completed / payment.failed to EventBridge for decoupled downstream fan-out.
- Trade-off
- Append-only means the ledger only grows; you pay storage and need a retention/archival strategy, and a correction is a new compensating entry rather than an update, so reads must fold the event history to get current state. The WORM immutability that satisfies auditors also means a genuinely wrong record cannot be deleted within retention - only annotated.
from: Idempotent payment gateway
Gate every payment on a conditional write to the primary
- When
- A client may retry a payment request after a lost response, a timeout, or a load-balancer hiccup, and a second execution would move money twice.
- AWS
- Client mints a UUID idempotency key before the first attempt and resends it on every retry; the server does a conditional PutItem with attribute_not_exists(pk) on a DynamoDB table (key sharded tenant#id#shard#mod_N to avoid a hot partition) to claim the key at PENDING, drives it PENDING to PROCESSING to COMPLETE with a leaseExpiry on the PROCESSING row so a crashed worker self-heals in ~30 s instead of orphaning the key for the TTL, and on a ConditionalCheckFailed reads the winner's stored response with ConsistentRead true and returns it verbatim.
- Trade-off
- Every payment pays for one strongly consistent DynamoDB write plus a consistent read on the retry path, and you must fail closed (reject) when the gate is unavailable — single-region and strongly consistent on purpose, since DynamoDB Global Tables (last-writer-wins, eventually consistent) would reopen the replica-lag double-charge hole. The gate's availability becomes a hard dependency of accepting any payment.
from: Distributed payment ledger with idempotent settlement
Model money as append-only double-entry pairs that sum to zero
- When
- You need an auditable, regulator-grade record of fund flows where balances are derivable and never silently corrupted by a partial write.
- AWS
- Write exactly one DEBIT and one CREDIT row per transaction in a single Aurora PostgreSQL ACID transaction, with a UNIQUE constraint on idempotency_key as a second line of defense behind the DynamoDB gate; never UPDATE or DELETE a row, only append reversing entries.
- Trade-off
- The ledger grows monotonically forever (7-year retention, no compaction) and corrections cost two extra reversing rows instead of an edit, in exchange for an immutable audit trail and a database-enforced no-duplicate guarantee that survives application bugs.
from: Distributed payment ledger with idempotent settlement
Orchestrate multi-step settlement with compensating transactions
- When
- A settlement spans steps that cannot share one ACID transaction (hold, external KYC verify, release, collect fee) and a mid-flight failure must reverse only the steps that already committed.
- AWS
- Step Functions runs the saga (Express for short settlements, Standard for long-running limbo cases); each state persists, steps retry with full-jitter backoff and a TimeoutSeconds on the external KYC call guarded by a DynamoDB circuit breaker; a Catch block runs a compensating state machine that reverses committed steps using derived idempotency keys ({key}:compensate:{step}); a stuck compensation flips the account to LIMBO and pages via EventBridge to SNS to AWS Systems Manager Incident Manager.
- Trade-off
- You accept eventual consistency across the settlement and the operational burden of compensation logic plus a manual-resolution path for stuck reversals, in exchange for a durable, restartable workflow with no distributed-transaction coordinator across external APIs.
from: Distributed payment ledger with idempotent settlement
Read money state in-band and fail closed
- When
- Stateless nodes spend a shared budget and a lost connection to the spend counter tempts a fail-open default - the exact gap that cost Meta advertisers $100K-$500K overnight during a DB failover.
- AWS
- Distribute the daily budget as token-chunk LEASES via a control-plane Lambda against a PID pacing curve; nodes spend locally with no hot-path hop and emit a fast HTTP 204 within an 8 ms internal deadline if they cannot read their allowance or the lease expires (2 missed reconciliation intervals); reconcile from a Kinesis burl stream every 1-5 s; a circuit-breaker Lambda - fed by both Kinesis AND the CloudWatch EMF spend-velocity metric so a Kinesis stall cannot disable it - pauses any campaign over 3x target for 60 s.
- Trade-off
- Accept 1-2 percent overspend and a 1-5 s reconciliation blind window in exchange for an off-hot-path budget check and a hard ceiling on runaway spend; fail-closed means a fast 204 (never a timeout, which Google throttles on), and the lease bounds even a stalled reconciliation.
from: Real-time bidding engine at scale
Account spend effectively-once on the billing notice
- When
- An auction win does not equal a charge - the impression may never render - and the same impression can arrive via multiple supply paths, so naively billing on the win notice double-charges or over-charges.
- AWS
- Bill on the OpenRTB billing notice (burl), not the win notice (nurl); stream events through Kinesis Data Streams (BatchWindow 1s, bisectBatchOnFunctionError, DLQ) deduplicated by the SSP-generated burl transaction id (trid); fold into a Snowflake-keyed ledger and archive to S3 under Object Lock as a tamper-evident audit trail.
- Trade-off
- This is effectively-once, not exactly-once: the burl trid is SSP-generated per auction and stays stable, but Prebid Aug-2025 trid fragmentation breaks real-time dedup on the request path - so real-time request dedup is abandoned in favor of structural Supply Path Optimization, at the cost of bidding into fewer paths.
from: Real-time bidding engine at scale