Incident Report — API Availability Issues on July 24, 2026

On July 24, 2026, the public API of sms-gate.app experienced a period of degraded availability.

Incident window: 07:02–08:50 UTC

During this time, most API requests failed or timed out, although some requests continued to succeed intermittently.

Fortunately, no user data was lost or corrupted.

What happened?

At the time of the incident, our cloud infrastructure consisted of three virtual servers.

Server #1 was responsible for several critical roles at once:

• Docker Swarm manager
• Traefik reverse proxy
• Prometheus
• Grafana
• gateway to the private network

It was also the smallest server in the cluster (1 vCPU, 1 GB RAM).

Beginning around 00:00 UTC, CPU utilization on Server #1 climbed to 100% and remained there. The first API issues appeared at 03:50 UTC, eventually escalating until only occasional requests were making it through by 07:02 UTC.

During the investigation we found that most CPU time was being consumed by Traefik and Prometheus.

Interestingly, our monitoring didn't show any unusual traffic volume or request patterns. We found no evidence of a traffic spike, attack, or recent deployment that could explain the sudden increase in CPU usage.

As a precaution, we enabled Cloudflare proxying in front of the API while continuing the investigation.

By 08:50 UTC, the API had recovered and request processing returned to normal. However, the gateway server continued running under unusually high load, with a load average reaching 5–6.

Root cause

We were not able to identify the exact trigger that caused the CPU saturation.

What the incident did reveal, however, was an infrastructure design issue.

The gateway server had historically accumulated several responsibilities over time: routing public traffic, managing the Docker Swarm cluster, and running the monitoring stack. When CPU resources became exhausted, these components began competing with each other, ultimately affecting API availability.

While the original trigger remains unknown, this architecture significantly reduced our margin for handling unexpected load.

What we changed

Over the following two days (July 25–26), we reworked the infrastructure to better separate responsibilities.

The changes included:

• Promoted Server #2 to the Docker Swarm manager.
• Moved Prometheus and Grafana from Server #1 to Server #2.
• Reduced Server #1 to primarily serving as the gateway into the private network.
• Kept Cloudflare enabled in front of the public API.

These changes immediately reduced resource contention.

Today, the load average on both primary servers remains below 1.0 under normal operation.

Lessons learned

This incident reminded us that infrastructure components are easy to colocate when a project is small—but those shortcuts eventually become bottlenecks.

A few takeaways:

• Gateway services should stay lightweight whenever possible.
• Monitoring infrastructure should not compete with request processing for CPU time.
• Even when traffic looks normal, additional edge protection and traffic management are valuable.
• Better observability is needed to determine why CPU saturation occurred in the first place.

What's next?

We're continuing to monitor the updated infrastructure and are adding more visibility into CPU usage and request processing so that similar issues can be diagnosed much faster if they occur again.

Although we still don't know exactly what triggered the initial CPU spike, we're confident that the new architecture is considerably more resilient than the one that existed before the incident.

Thanks to everyone who reported the outage and for your patience while we worked through the issue.

API recovered.

Certificate Authority recovered.

API went down.

Certificate Authority went down.

Certificate Authority recovered.

Certificate Authority went down.

Certificate Authority recovered.

API recovered.

Certificate Authority went down.

API went down.