Postmortem: Full API outage due to storage exhaustion (Oct 28–29, 2026)

Severity: Full outage
Impact window: Oct 28, 2026, 20:40 UTC → Oct 29, 2026, 00:16 UTC (~3 hours)
Affected components: API, SMS sending, Inbound SMS processing, Push notifications
Status: Resolved

What users experienced

For approximately 3 hours, api.sms-gate.app was unreachable — API requests
failed with connection timeouts, so outgoing SMS could not be submitted and
devices could not sync. From 23:50 UTC a standby instance served part of the
traffic while full service was being restored. All failed requests were
connection-level rejections: nothing was partially accepted or lost.

Impact

• Outgoing SMS submissions failed or were delayed until clients retried after restoration.
• Inbound SMS upload and push notifications paused and resumed automatically; all backlogs were delivered within minutes of recovery.
• No data loss and no data corruption. The database and all message history were unaffected throughout.

Timeline (UTC)

Time	Event
Oct 28, 20:30	Storage on the primary edge host reaches 100%. Internal cluster services begin failing; users are not yet affected.
Oct 28, 20:35	Container runtime fails under disk pressure — edge proxy, API and worker instances stop. Public outage begins.
Oct 28, 20:37	Uptime monitoring alerts the team; incident response starts with read-only diagnosis.
Oct 28, 23:47	Root cause confirmed; ~4.5 GB of accumulated log data is reclaimed and the host is stable again.
Oct 28, 23:50	Standby API instance activated via temporary routing.
Oct 29, 00:05	Cluster control plane rebuilt from healthy state; API, workers, edge proxy and cache all restored and passing health checks.
Oct 29, 00:16	Traffic returned to the primary path; end-to-end health checks pass. Service fully restored.
Oct 29, 01:25	Prevention measures live (see below).

Root cause

The incident was caused by unbounded growth of access logs on the primary
edge host: every HTTP request was written as a verbose structured log line to
the system log, which was only rotated once per day with no size limit. Over
time this filled the 15 GB disk to 100%.

A full disk then cascaded: the write-ahead log of our container orchestration
control plane could no longer be written and was corrupted, which took down the
edge proxy (the public entry point) and the ability to manage the cluster —
even though the database and application data were intact. During the failure,
the log subsystem's own "no space left" errors amplified the log volume,
accelerating the crash.

Resolution

1. Reclaimed disk space immediately (log truncation), stabilizing the host.
2. Activated a standby API instance to restore partial service.
3. Rebuilt the orchestration control plane from the surviving healthy node — service definitions, configurations and secrets were fully preserved.
4. Restored all services, verified health end-to-end, then returned traffic to the primary path and confirmed clients were syncing again.

Prevention

Completed:

• Log rotation is now size-capped: rotations trigger at 50 MB (checked hourly, compressed, 6 retained) instead of once a day — log growth is bounded at all times.
• Root cause of the log volume identified (per-request structured access logs on the highest-traffic endpoints).

Planned:

• Reduce access-log volume at the source (format/retention policy for high-frequency request logs).
• Increase disk headroom on edge hosts and review control-plane redundancy so a single-host failure cannot repeat this cascade.

Certificate Authority recovered.

API recovered.

API and Certificate Authority went down.