Cold start p50, arm64 vs x86_64
Client-measured request→response on a fresh environment, minus the median warm round-trip overhead. n = 10 per cell. Axis clipped at 2.6 s; clipped values printed at right.
Init / restore / first invoke: REPORT-line p50. Warm: billed-duration p50 of 3 warm invokes after each cold start. $: eu-west-2 on-demand (arm $0.0000133334, x86 $0.0000166667 per GB-s, $0.20 per 1M requests); SnapStart adds the $0.0001397998/GB restore charge (US East list); snapshot cache (~$0.0000015046 per GB-s per live version) excluded. Accuracy: 12 headlines vs PyTorch FP32; flips = topic decisions crossing 0.20 out of 120; entities = matching spans / union.
arm64 vs x86_64 by variant
| Variant | Cold arm64 | Cold x86 | Δ cold | Warm arm64 | Warm x86 | Δ warm | $/1M warm arm64 | $/1M warm x86 |
|---|
Δ = arm64 relative to x86_64; negative is faster on arm64.
| Memory | vCPU seen | arm cold | arm p90 | arm warm | arm $/1M cold | arm $/1M warm | x86 cold | x86 p90 | x86 warm | x86 $/1M cold | x86 $/1M warm |
|---|
Zip vs container image (arm64, 1769 MB)
| Payload | Size | Zip cold | Image cold | Ratio | Zip platform | Image platform |
|---|
Platform = client cold − init − first invoke (sandbox + code/image setup, not in REPORT). Zip limit is 250 MB unzipped regardless of S3 upload, so FP32 is image-only.
SnapStart (arm64, 1769 MB)
| Variant | Plain cold | Snap cold | Restore | 1st invoke | Plain $/1M | Snap $/1M |
|---|
Snap cold starts: burst of 10 concurrent requests per freshly published version. Snapshot taken after a warm-up forward pass.
Precision (container image, arm64, 1769 MB)
| Engine · precision | Payload | Cold p50 | Init | Engine load | Warm p50 | Max RSS | Topic flips | Entities | Max Δ prob |
|---|
BF16 / FP4 / F16-stored (candle) are storage formats upcast to FP32 at load: no FP4 arithmetic on Lambda CPUs, no BF16 matmul in candle's CPU backend. INT8 ONNX uses reduce_range (7-bit weights); INT4 ONNX is MatMulNBits block-32 with int8 compute.
First cold start after deploy
| Function | Payload | First start | Repeat cold p50 (same mem) |
|---|
First invoke after publishing new code (image chunk cache empty). One sample per function. "init timeout" = INIT hit the 10 s limit and Lambda re-ran init inside the invoke.
CPU fleet seen at cold start
Failures and fixes
Definitions
- Cold start
- Env-var change on the function (retires all environments), wait for update, invoke once. First sample per memory size discarded. No SDK retries.
- Rust
- lambda_runtime on current-thread Tokio, mimalloc, target-cpu neoverse-n1 / x86-64-v3. candle 0.11 custom DistilBERT (fused QKV, exact GELU); ONNX Runtime 1.28.0 load-dynamic, graph fused offline, intra-op threads = nproc, no spinning.
- Python
- 3.13 managed runtime / base image. onnxruntime 1.28.0 + tokenizers on the same ONNX files, or transformers 5 + torch (CPU) AutoModel with trust_remote_code.
- Packaging
- zip → provided.al2023 (Rust) / python3.13, uploaded via S3. img → public.ecr.aws/lambda/provided:al2023 or python:3.13; distroless → gcr.io/distroless/cc-debian12. snap → SnapStart on the al2023 / python image or python zip.
- Precision keys
- fp32 · fp16 (computed in FP16) · f16s (stored FP16, FP32 compute) · bf16 (stored) · int8 (candle Q8_0 / ORT dynamic) · int4 (candle Q4_K / ORT MatMulNBits) · fp4 (MXFP4 stored)
- Limits
- One region, one day, n = 10 per cell; p90 indicative. Client on a corporate network (overhead-corrected). 12-headline accuracy set.