An image CDN looks like the easy part of the stack: put CloudFront in front of S3, append ?w=300&fmt=webp when the frontend needs a thumbnail, transform on the fly. That sketch survives a demo and dies in production for one reason nobody draws on the whiteboard: every distinct query string is a distinct cache object. A CDN whose job is to absorb reads becomes a pass-through that forwards almost everything to a Lambda, and a design whose egress you thought was the whole bill quietly grows a transform line of the same order of magnitude on top. The interesting version delivers a billion transformed images a day, survives a product launch and a DMCA takedown at 3am, isolates tenants who each bring their own signing keys, and never leaves AWS.
The problem, and the numbers we design to
Image delivery. Before any boxes — what are we actually building, and at what scale?
A multi-tenant image origin-and-delivery platform. Tenants upload originals — product photos, user avatars, marketing assets — and our clients request derivatives: a specific width, height, format, quality, crop. The frontend says "give me this image at 300 px wide, WebP, quality 80" and we return it from the nearest edge in tens of milliseconds. Scale target: 1 billion image requests/day, average delivered object 40 KB, served globally. SLO: p95 edge latency ≤ 50 ms on a cache hit, and a cache-hit ratio we can actually defend — I'm going to argue the whole design is a fight to keep that ratio above 90%.
Why is hit ratio the headline number? Egress is the same bytes either way.
Because a miss isn't just "fetch from S3" here — it's "fetch the original, run an image transform, re-encode." A hit costs us a cached byte. A miss costs us a Lambda invocation plus CPU plus an S3 GET. The egress bill is roughly fixed by traffic; the miss bill is the variable we control, and it's set almost entirely by how many distinct cache objects we let exist. Get the cache key wrong and hit ratio collapses toward zero — at which point CloudFront is an expensive reverse proxy and the real system is 200 million Lambda runs a day. So yes, hit ratio is the headline.
Traffic: $1\text{B/day} \div 86400\text{ s} \approx 11{,}600\text{ req/s}$ average, call it ~35k req/s at peak (3× diurnal). Egress at 40 KB/object:
$$ 10^{9} \times 40\ \text{KB} = 4 \times 10^{13}\ \text{B/day} = 40\ \text{TB/day} \approx 1{,}200\ \text{TB/month} $$
CloudFront egress is tiered, not flat — you can't multiply 1,200 TB by the headline rate. Blended across the tiers (US/EU pricing):
$$ 10\,\text{TB}\!\times\!\$0.085 + 40\,\text{TB}\!\times\!\$0.080 + 100\,\text{TB}\!\times\!\$0.060 + 350\,\text{TB}\!\times\!\$0.040 + 700\,\text{TB}\!\times\!\$0.030 \approx \$45{,}000/\text{month}, $$
a blended ~$0.038/GB, not $0.06. (An earlier draft used $0.06 flat and overstated egress by ~$27k.) And bytes aren't the only CloudFront charge — there's a per-request HTTPS fee that a 1B/day workload cannot ignore:
$$ 3\times10^{10}\ \text{req/month} \div 10^{4} \times \$0.01 = \$30{,}000/\text{month}. $$
That request line is 29% of egress and was missing entirely from the first cut. At this volume the CloudFront flat-rate/committed plan beats on-demand on both bytes and requests; we assume it. Egress + requests is the fixed floor — everything else in this transcript is about not adding a second number of that magnitude on top.
Cache-key explosion — the failure nobody draws
Naive version. CloudFront in front of S3, transform on the query string. Walk me to where it breaks.
The naive instinct is to forward the full query string to the origin and let CloudFront cache per-URL. It's tempting because it Just Works in a demo — every distinct request gets its own cache entry, correctness is trivial. The failure is that the cache key is now an unbounded namespace. CloudFront's default, if you forward all query strings, is to treat ?w=300&h=200 and ?h=200&w=300 as two different objects. Same pixels, two cache entries. Add a tracking param — ?utm_source=email — and now every marketing campaign mints a fresh, uncacheable copy of every image. Add a malicious client appending ?cachebust=<random> and they can drive your hit ratio to zero on demand.
Suppose a single original is legitimately requested at 5 widths × 3 formats × 4 quality levels = 60 valid derivatives. Fine. Now let one junk param leak into the key — say a session id with $10^{6}$ distinct values. The cache namespace for that one image becomes
$$ 60 \times 10^{6} = 6 \times 10^{7}\ \text{distinct keys}, $$
of which 60 are real. Effective hit ratio for that object $\approx 60 / (6\times10^{7}) \approx 10^{-6}$. Across the catalogue you've converted a CDN into a Lambda trigger. At 1B req/day and a 0% hit ratio you'd be paying for $10^{9}$ origin transforms/day instead of $2\times10^{8}$ — a 5× blow-up of the most expensive line item, caused by one stray query parameter.
So you allowlist params. Where does that logic run? Not in Lambda, I hope — you can't afford a function call on a hit.
Correct, and that's the crux. Two mechanisms, layered. First, a CloudFront cache policy that allowlists exactly the four params that affect the bytes — w, h, fmt, q — and ignores everything else for cache-key purposes. That alone kills utm_* and session junk for free; it's pure configuration, no compute. Second, a CloudFront Function on the viewer-request event that normalizes what's left: sort the params into canonical order, clamp w and h to an allowed set of breakpoints (snap 287 px up to 320), lowercase the format, and reject anything outside policy. CloudFront Functions run in ~1 ms at the edge with no cold start, at $0.10 per million — so I can afford to run this on the viewer side of every request, hit or miss.
The thundering herd on a cold object
Product launch. A new hero image goes live and 10,000 edge requests hit it in the same second, all cold. What does your origin see?
Without mitigation, something close to 10,000 simultaneous misses, each independently deciding "not in cache, go to origin." That's 10,000 concurrent S3 GETs and 10,000 concurrent transform invocations for the same output. The transform is the dangerous part — it's CPU, and 10,000 cold Lambda@Edge invocations for one image is a synchronized spike that does no useful work, because 9,999 of them produce the identical bytes.
So you want request coalescing. CloudFront doesn't collapse misses across edge locations on its own. What's the AWS answer?
Origin Shield. It's a designated regional cache layer that sits between all the edge POPs and your origin. Every edge miss for a given object funnels through one Shield region, and Shield collapses concurrent misses for the same key into a single origin fetch — the rest wait on that in-flight request. Ten thousand edge misses become one transform. That's the textbook fix.
Then turn it on everywhere and move on?
No — and this is where people cargo-cult it. Origin Shield is not free, and for a plain S3 origin it can cost more than it saves. Let me show the arithmetic, because it flips the obvious answer.
At 1B req/day and 80% hit ratio, misses $= 2\times10^{8}/\text{day}$. Shield charges per request routed through it (~$0.0075 per 10k in the US):
$$ 2\times10^{8} \div 10^{4} \times \$0.0075 = \$150/\text{day} \approx \$4{,}500/\text{month}. $$
What does it save? If the origin were just S3, those 200M GETs cost $2\times10^{8} \div 10^{3} \times \$0.0004 = \$80/\text{day}$. So Shield would cost $150/day to save $80/day — a net loss of $70/day. Shield only pays for itself when the per-miss origin work is expensive. Here it is: each miss is a Lambda@Edge transform on the order of $\$8\times10^{-6}$ each, plus the herd-collapse benefit. Coalescing 10,000-deep launch spikes into one transform is worth far more than the per-request fee, because it removes the correlated CPU spike that would otherwise blow your Lambda concurrency limits.
Careful — you're conflating two cases. One hero image, 10k requests, same key: Shield coalesces beautifully. But a launch of 50,000 new images going live at embargo-lift is 50,000×8 = 400,000 distinct cold keys. Shield only collapses concurrent misses for the same key. It does nothing for 400k different keys.
You're right, and that's the important distinction. Shield is a same-key coalescer, not a launch shock absorber. For the 400k-distinct-key launch, the structural answer isn't Shield — it's pre-compute completing before the embargo lifts (beat 08). The Step Functions render job materializes all 8 variants per image into S3 ahead of go-live, and I add a warm-prefetch step: after rendering, the workflow issues a HEAD for each variant through CloudFront so the object is cache-resident before the first real viewer arrives. Embargo lift then sees hits, not 400k cold transforms. If pre-compute hasn't finished, the launch is not allowed to flip live — that's a release gate, not a hope.
And Shield itself — single region, us-east-1. That's a SPOF and a cross-Pacific tax. What happens when it degrades, and why is APAC paying 150 ms?
Two real problems, both accepted. First, the SPOF: when the Shield region degrades, CloudFront doesn't fail the request — it bypasses Shield and sends edge misses straight to origin, which removes coalescing at exactly the worst moment. I won't pretend Shield is HA; the structural protection is again pre-compute (misses are plain S3 GETs, no transform to stampede) plus aggressive TTLs that keep the miss rate low so a bypass doesn't become a herd. Second, the cross-Pacific tax: a single us-east-1 Shield adds ~150–180 ms RTT to every APAC cold miss. So I deploy Origin Shield in three regions matching the CRR set — us-east-1, eu-west-1, ap-northeast-1 — and route each POP to its nearest Shield. APAC misses coalesce locally. The hit-ratio cost of three Shields instead of one is small in practice because APAC, EMEA and the Americas request largely different catalogue slices, so the caches barely overlap.
CloudFront Functions vs Lambda@Edge — the 1ms/50ms split
You've used both already. Make the boundary explicit. Why not one compute primitive for everything?
Because they're built for different physics. CloudFront Functions are a sandboxed JS runtime that runs on the POP itself: ~1 ms CPU budget, no cold start, no network, no filesystem, ~2 MB memory. Lambda@Edge is real Lambda running in regional edge caches: up to 5 s (viewer) or 30 s (origin) timeout, up to 3008 MB, full network and a real runtime — but with cold starts of 50–500 ms. The rule I apply: per-request, no-I/O string work goes to Functions; anything that needs the image bytes, a library, or a network call goes to Lambda@Edge, and only on the origin-request event so it runs on misses only.
So the topology is: viewer-request → CloudFront Function (normalize key, validate signed URL claims, hotlink check) on every request. Origin-request → Lambda@Edge (fetch original from S3, transform with sharp/libvips, re-encode) on misses only. The image-decoding library and the CPU-heavy resize physically cannot run in a 1 ms Function — that alone forces the split.
One sizing correction worth making loudly: don't run the transform at 128 MB. libvips decoding a 2–5 MB JPEG comfortably uses 200–400 MB, and Lambda's vCPU scales with memory — 128 MB is ~0.07 vCPU, which turns a resize into an 800 ms–2 s slog (and can OOM on large originals). We start at 512 MB (~1 vCPU), benchmark the catalogue's actual originals, and treat memory as a latency/throughput knob, not a cost-minimization one — more memory finishes faster, so the GB-second product barely moves while p95 transform latency drops sharply.
Viewer layer, all requests, on CloudFront Functions:
$$ 10^{9}/\text{day} \times \$0.10/10^{6} = \$100/\text{day} = \$3{,}000/\text{month}. $$
Origin transform, misses only, on Lambda@Edge (512 MB, ~300 ms at 1 vCPU — faster than the 128 MB straw man):
$$ \text{compute} = 2\times10^{8} \times 0.3\,\text{s} \times \tfrac{512}{1024}\,\text{GB} \times \$0.00005001 = \$1{,}500/\text{day}, $$
$$ \text{invocations} = 2\times10^{8}/10^{6} \times \$0.60 = \$120/\text{day}. $$
Total transform ≈$1.6k/day ≈ $49k/month at the steady 80%-hit assumption. More memory costs more GB-seconds here than the 128 MB fantasy, but it's the honest number and it makes the pre-compute argument (beat 08) stronger, not weaker. If I'd done the viewer-layer normalization in Lambda@Edge instead of Functions, I'd add another $22k+/month for work a 1 ms Function does for $3k.
Geo routing and multi-region origin failover
CloudFront is global, but your origin and your transform Lambdas live in regions. A user in Tokyo misses cache — where does the transform happen, and what if that region is down?
Lambda@Edge origin-request executes in the regional edge cache nearest the POP, so the Tokyo miss runs the transform in or near ap-northeast-1 — close to the user. The originals live in S3, and I replicate the original bucket across three regions — us-east-1, eu-west-1, ap-northeast-1 — with S3 Cross-Region Replication, so the transform's source fetch is regional too. For failover I use CloudFront Origin Groups: each cache behavior has a primary origin and a secondary, and CloudFront fails over on configured status codes automatically. Primary is the in-region S3 bucket; secondary is the next-nearest replica.
Two failover-codes details that bite if you skip them. I configure the Origin Group failover set as 500/502/503/504 and 403 and 404. The 403 is for an IAM regression — a bad bucket policy returns 403, not 5xx, and without it in the set a misconfig silently returns 403 to every user with no failover. The 404 is for the CRR race: a freshly uploaded original may exist in us-east-1 but not yet in the eu-west-1 replica, so a failover fetch returns 404, not 5xx, and the default set wouldn't retry the primary. (404 in the failover set needs care — a genuinely deleted object will now try both origins — but for an image origin where 404 means "not replicated yet" far more often than "gone," it's the right trade.)
Why not Route 53 latency-based routing for that instead of Origin Groups?
I use both, for different jobs. Route 53 latency-based routing is great for steering to a regional endpoint when the origin is a dynamic service behind a load balancer. But S3 origins and the failover decision are best handled inside CloudFront with Origin Groups, because CloudFront makes the retry decision per-request on the actual response code, with no DNS-TTL lag — a Route 53 health-check flip can take tens of seconds and is cached by resolvers. Origin Groups fail over on the very request that saw the failure. So: Origin Groups for primary/secondary S3 failover; Route 53 latency routing reserved for any dynamic control-plane endpoints (uploads, admin API). Note that Route 53 is DNS resolution — it is not a hop in the image data path; the bytes go client → POP → (Shield) → origin.
One thing Origin Groups don't catch: a Lambda@Edge that times out, throttles, or OOMs returns a 502 to the viewer — but that's a function error, not an origin 5xx. Origin Group failover never triggers on it. So when your transform Lambda is sick, what does the viewer get?
Correct, and it's the failure mode people miss. Origin Group failover keys on the origin's HTTP status; a Lambda@Edge crash short-circuits before the origin is even reached, so failover doesn't fire. Two mitigations, in order. First and structural: pre-compute means standard variants never invoke the transform Lambda — they're plain S3 GETs, so the most common requests have no function on the critical path to fail. Second, for the long-tail non-standard derivatives that do hit Lambda: a CloudFront custom error response that maps any 5xx to a fallback placeholder image with a short TTL. The user sees a graceful placeholder, not a broken-image icon, and the next pre-compute or retry fills the real variant. I also note the asymmetry honestly: if us-east-1 has a combined Shield-plus-Lambda degradation, the Origin Group fails the S3 fetch over to eu-west-1, but the Lambda still executes in the degraded region for that request — so Lambda fallback is best-effort during a regional incident. Pre-compute is the thing that actually holds.
And the S3 read path itself at 1B/day. Misses hit S3 hard. Have you partitioned the keyspace?
Yes — S3 scales request rate per prefix, at 5,500 GET/s each. At 20% miss that's ~2,300 GET/s average, spiking to ~7,000 GET/s, which exceeds a single prefix. So all object keys carry a 2-hex-char shard prefix derived from a hash of the key — 256 prefixes × 5,500 = ~1.4M GET/s of headroom, which we will never approach. The shard is deterministic from the key, so the transform Lambda and the pre-compute job both compute the same path. This is free; it's just a key-naming convention applied from day one rather than retrofitted after the first 503 SlowDown.
Cache invalidation at scale — launches and DMCA
3am. Legal sends a DMCA takedown — a copyrighted image must be gone from every edge in minutes. Also, a tenant just re-uploaded 50,000 product photos for a launch. Walk me through invalidation.
These are two different problems and conflating them is the classic mistake. For the launch re-upload, I do not invalidate — I use versioned URLs. The path is /img/v2/product-123.jpg; bumping the version to v3 mints a fresh key that's simply never been cached, so it's a guaranteed miss with no purge needed and no race between "new bytes uploaded" and "old bytes still cached." CloudFront's invalidation limits are brutal — 3,000 in-progress invalidation paths and ~15 wildcard invalidations/sec — so trying to invalidate 50,000 objects is both slow and a quota violation. Versioning sidesteps it entirely.
Versioning works when you control the change. DMCA is the opposite — the URL must stop serving the same bytes. You can't version your way out of that.
Right. For takedowns I do three things at once: (1) delete the original from S3 so no future transform can produce it; (2) issue a wildcard CloudFront invalidation for that image's derivatives (/img/*/product-123*); and (3) rely on a short TTL bucket for legally-sensitive content. Anything flagged takedown-eligible is served under a path with a 60 s max-TTL behavior, so even if an invalidation is slow, the blast radius is one minute. To fan out a batch of takedowns without tripping the per-second wildcard limit, I drive invalidations through Step Functions Express. The wildcard limit is ~15/sec, so the Map state runs MaxConcurrency: 1 with a 100 ms Wait between iterations — a steady ~10/sec that stays safely under the throttle, with built-in retry, rather than a naive loop that hits the limit and silently drops paths.
Security: signed URLs, hotlinking, SSRF, tenant isolation
Multi-tenant. Tenant A must never serve from tenant B's keys, and nobody should be able to make your transformer fetch an arbitrary URL. Convince me.
Start with the control that's easy to get subtly wrong: signed URLs with Trusted Key Groups. Each tenant gets its own key group, and the viewer Function validates the signature before anything else runs. But a signature only proves "signed by a trusted key" — it does not by itself prove the signed URL is for the right tenant's content. If tenant A's key group can sign a URL whose Resource is cdn/B/..., A can read B's images. So the binding has to be explicit at three layers: (1) the signed policy's Resource must be scoped to https://cdn/<tenantId>/*, not a wildcard; (2) behaviors are keyed on /<tenantId>/* path patterns, never a shared /img/*; and (3) the viewer Function asserts that the requested path prefix matches the key-group identity that validated the signature. Signature-valid but prefix-mismatched is a hard 403. Cross-tenant signing is closed by construction, not convention.
Second, the related trap: path traversal in key derivation. "Server-derived S3 key = tenantId + '/' + requestPath" is not safe on its own — a requestPath of ../otherTenant/secret.jpg (or encoded %2e%2e%2f, or a null byte, or a tenantId containing /) escapes the prefix. So the Function rejects any path containing .., //, their percent-encoded variants, or null bytes; allowlists the charset [a-zA-Z0-9/_.-]; and after normalization asserts the final key startsWith(tenantPrefix + "/") or fails closed. Tenant IDs are opaque UUIDs with no / or .. The S3 key is built from the authenticated tenant ID (from the validated key group), never echoed from the request.
Third, SSRF — the transformer must never fetch a user-supplied URL. It only ever does GetObject against a known bucket with the server-constructed key above; no request body or query param ever becomes a URL it dials. A hard architectural invariant, not a filter.
Fourth, key rotation and revocation. A key group holds up to 5 public keys; we keep ≤2 active normally so 3 slots stay free for emergency rotation. Removing a compromised key stops new signature validation, but already-issued URLs stay valid until they expire — so short signed-URL expiry (minutes, not hours) is load-bearing for incident response, not a nicety. Rotation is automated: Secrets Manager rotation drives a Lambda that adds the new CloudFront public key and retires the old after an overlap window.
Fifth, origin/IAM isolation, not just signing isolation. Distribution sharding isolates signing, but a single shared transform role with broad s3:GetObject means one key-derivation bug or compromised Lambda reads every tenant prefix on the shard. So the transform role assumes a session scoped with the validated tenant ID as a session tag, and the IAM policy puts an s3:prefix Condition on that tag — the role can only read the one tenant's prefix per request. Same discipline on the control plane: cloudfront:CreateInvalidation and s3:DeleteObject are granted only to the Step Functions / takedown execution roles, scoped to specific distribution and bucket ARNs with path conditions — so a compromised serving component can't purge or delete another tenant's content (an availability and cost attack), and every invalidation and delete lands in CloudTrail.
Sixth, upload-time content validation. Before pre-compute fans an upload into 8 variants, the first Step Functions step validates it: magic-byte check against a format allowlist (no polyglots), max decoded-dimension and max file-size caps (no pixel bombs / decompression bombs), EXIF strip, and a re-encode to a canonical form before it's stored as the "original." A Rekognition Content Moderation call runs in the same gate — a moderation hit quarantines the asset and renders nothing. The libvips version is pinned and scanned. This maps to SOC 2 CC6.8/CC7.1, ISO 27001 A.8.7/A.8.8, NIST SI-3.
Seventh, encryption and data residency. All buckets are SSE-KMS; large/regulated tenants get a per-tenant CMK (which makes crypto-shredding a valid erasure path), the long tail shares a CMK, and CRR uses multi-region keys. Residency is a per-tenant policy, not a global topology: an EU-resident tenant replicates only within the EEA (eu-west-1 + eu-central-1) — geo-restricting requests does nothing if CRR has already shipped the bytes to us-east-1, which would be a GDPR Chapter V violation. So residency-zoned tenants get residency-zoned buckets and distributions.
Finally, hotlink + monitoring: a Referer match in the viewer Function is deterrence only (Referer spoofs trivially), backed by a WAF rate-based rule per source IP. Real detection comes from CloudFront real-time logs to S3 plus CloudWatch anomaly-detection alarms on per-tenant request rate and cache-miss-ratio spikes (NIST DE.AE/DE.CM). The audit trail itself is tamper-evident: CloudTrail log-file integrity validation on, S3 Object Lock (compliance mode) on a dedicated cross-account log bucket, and CloudTrail data events enabled on the originals and variants buckets.
GDPR erasure — a user invokes right-to-be-forgotten on an avatar. What's the deletion path and how do you prove it's gone?
Provably complete erasure is more than "delete the original." The runbook deletes the original and all 8 pre-computed variants and all CRR replicas, then invalidates the derivatives at CloudFront. The completeness depends on two things being true ahead of time: PII-bearing paths (avatars) must have been tagged at creation onto a private, no-store short-TTL behavior — not left on the default long-TTL behavior — and CRR must run with RTC so replica deletion has a bounded 15-minute SLA rather than "minutes to hours." For per-CMK tenants, crypto-shredding the key is an additional belt-and-suspenders. We log every S3 delete (CloudTrail data events) and the invalidation id as the audit artifact; the residual browser-cache copy is handled by data-minimization argument and the short max-age. This maps to GDPR Art. 17 (erasure) and Art. 32 (security), SOC 2 CC6.1, and ISO 27001 A.8.24; the per-tenant key-group, session-tag IAM scoping, and least-privilege control-plane roles are the NIST AC-3/AC-6 story.
Cost, and the pre-compute escape hatch
Put the whole bill on the table. Then tell me the one change that could halve it.
| Line | $/month | Basis |
|---|---|---|
| Egress (blended tiers) | ~$45,000 | 1,200 TB, blended ~$0.038/GB |
| CloudFront HTTPS requests | ~$30,000 | 30B req × $0.01/10k |
| Lambda@Edge transform | ~$49,000 | 200M miss/day, 512 MB, ~300 ms |
| AWS WAF | ~$18,000 | 30B req × $0.60/M + WebACL |
| CloudFront Functions (viewer) | ~$3,000 | 1B/day × $0.10/M |
| Origin Shield (3 regions) | ~$4,500 | 200M miss/day × ~$0.0075/10k |
| S3 GETs on misses | ~$2,400 | 200M/day × $0.0004/1k |
| CloudWatch Logs (7-day, via Firehose) | ~$600 | 1,200 GB/month × $0.50/GB |
| Invalidations + CRR + Route 53 | ~$500 | takedown batches + replication + DNS |
$$ \approx \$153{,}000/\text{month}. $$
The first draft showed ~$104k and was wrong in two directions: it overstated egress (flat $0.06 instead of blended $0.038) and omitted three large lines — the per-request HTTPS fee, WAF, and CloudWatch Logs ingestion. The corrected picture is bigger and more honest. Crucially, the two biggest attackable lines are the $49k transform and, indirectly through it, the WAF/request volume on misses. Egress and requests are a floor. Everything below is about collapsing that transform line.
The transform bill exists because we re-derive variants on demand. For a product catalogue with a bounded variant set, the right default is to make pre-compute the primary path, not a fallback. On upload, an S3 event triggers Step Functions to render all 8 standard variants and write them to S3 as static objects. Now the overwhelming majority of requests are plain cache-fillable S3 GETs with no Lambda on the critical path — which is also what protects us from the concurrency cliff and the Lambda-vs-Origin-Group failover gap from earlier beats. On-the-fly Lambda@Edge transform stays only for the genuine long tail of non-standard derivatives.
Whether pre-compute wins is a formula, not a vibe. Pre-compute renders every variant once; on-the-fly renders each variant once per cache-miss. So pre-compute wins when
$$ V \times P_\text{render} \;<\; F_\text{avg} \times P_\text{transform}, $$
where $V$ is variants per image, $P_\text{render}$ the one-time render cost, $F_\text{avg}$ the average miss-driven transforms per variant over its lifetime, and $P_\text{transform}$ the per-miss transform cost. For an 8-variant catalogue with 100k products requested for months, $F_\text{avg} \gg 1$ and the inequality is trivially satisfied. For 60-variant UGC where each variant is requested ~twice a month, $F_\text{avg} \approx 2$ and pre-computing 60 variants you'll mostly never serve loses — that case stays on-the-fly. The crossover is the whole decision.
100k products × 8 variants × 50 KB $= 40\ \text{GB}$ stored. S3 Standard at $0.023/GB:
$$ 40\ \text{GB} \times \$0.023 \approx \$0.92/\text{month}\ \text{storage}. $$
Render cost is a one-time $100\text{k} \times 8 = 8\times10^{5}$ Lambda runs per full catalogue refresh — negligible amortized. Net: trade $22.4k/month of on-the-fly transform for under $1/month of storage plus a one-time render. The on-the-fly Lambda@Edge path stays only as the fallback for the long-tail of non-standard requests.
Failure and recovery
3am, your transform Lambda's region (us-east-1) is degraded and error rates spike. What does a user in New York experience?
For the 80%+ of requests that are cache hits: nothing — they're served from the POP, the origin is irrelevant. For misses where the S3 fetch fails: CloudFront Origin Groups fail the source fetch over to the secondary region's replica. If transforms are broadly failing, the pre-computed static variants save us — standard sizes are already materialized in S3 and served as plain GETs with no Lambda in the path at all (recall Origin Groups don't fail over on a Lambda function error, so keeping Lambda off the common path is the actual protection). The only requests that fail are cold misses for non-standard derivatives during the incident — a small slice, caught by the custom-error placeholder. I'm precise about stale-while-revalidate here because it's easy to overstate: the Lambda@Edge transform sets Cache-Control: public, max-age=86400, stale-while-revalidate=3600, stale-if-error=86400, so an expiring-but-present object is served while a background refresh runs, and a present object is served on an origin 5xx. What SWR does not cover is a cold miss during a Lambda outage — there's no stale copy to serve, so that request genuinely fails to the placeholder. SWR converts incidents into staleness only where something is already cached; pre-compute is what shrinks the cold-and-unprotected set to near zero.
Monitoring and observability
You've stated a p95 ≤ 50 ms SLO and a "keep hit ratio above 90%" thesis. How do you actually measure either on a 1B req/day platform — and how do you know before the customer does?
Three feeds, all AWS-native. First, CloudFront real-time logs → Kinesis Data Firehose → S3, queried with Athena / CloudWatch Logs Insights for cache-hit-ratio trending per tenant and per behavior — that's how we watch the 90% thesis hold or slip. Second, CloudWatch metrics: CloudFront publishes CacheHitRate and edge latency, and Lambda@Edge publishes errors, throttles, and duration. Third, alarms that page: CacheHitRate < 90% (sustained), Lambda@Edge Throttles > 0 and error-rate breach, and CloudWatch anomaly-detection bands on per-tenant request rate and miss-ratio to catch scraping or a cache-busting attack before it becomes a Lambda bill. p95 edge latency is read straight off the CloudFront percentile metric, alarmed against the 50 ms SLO. The throttle alarm is the one that matters most — it's the early-warning for the concurrency cliff from beat 04.
Did we ever leave AWS?
Final question. Anything in here that isn't AWS-native — or did you stay on the platform the whole way?
We never left. Delivery is CloudFront; storage and originals are S3 with Cross-Region Replication; edge compute is CloudFront Functions (viewer) and Lambda@Edge (origin); coalescing is Origin Shield; failover is Origin Groups; geo and dynamic-endpoint routing is Route 53; auth is CloudFront Trusted Key Groups; the WAF is AWS WAF; pre-compute orchestration is Step Functions with S3 events; audit is CloudTrail. The one component people assume forces you off-platform — the image transform itself — runs inside Lambda@Edge using an open-source library (sharp/libvips) bundled into the function. That's still "on AWS": it's a library in a managed runtime, not a service we operate. The only requirement that would push us off would be a transform AWS's runtimes can't host — say a GPU-bound ML upscaler exceeding Lambda's resources — at which point I'd reach for an ECS/Fargate GPU task on the origin-request path, still on AWS (and if we ever do, that task runs with IMDSv2 enforced, no public IP, and egress locked to S3 and the originating service — a transformer should never be able to reach the internet or the instance-metadata credentials path). There is no hard requirement in this design that leaves the platform.