Back to overview
Resolved

API is down

Jul 4, 2026 at 2:50pm UTC
Affected services
API

Resolved
Jul 4, 2026 at 11:32pm UTC

SMSGate API Incident Post-Mortem (July 3–4, 2026)

On July 4, 2026, SMSGate API experienced an extended service outage affecting Public Cloud Server deployments.

Thank you for your patience while the service was being restored. We appreciate everyone who reported the issue and shared diagnostic information.

Summary

Following scheduled network maintenance performed by our hosting provider, a network configuration issue caused the database server to lose connectivity. As a result, SMSGate API was unavailable for approximately 7 hours.

Customer data remained safe throughout the incident. No data was lost or corrupted.

Impact

The incident affected deployments as follows:

  • Public Cloud Server — SMSGate API was unavailable from 14:55 UTC to 22:05 UTC on July 4.
  • Private Server — Message delivery notifications using Firebase Cloud Messaging (FCM) as the primary notification channel could be delayed.
  • Local Server — No impact.

Timeline (UTC)

  • July 3, 16:00 — Scheduled network maintenance began at the hosting provider.
  • July 3, 22:00 — Maintenance completed. During the migration, SMSGate API experienced approximately 15 minutes of expected service interruption.
  • July 4, 14:55 — SMSGate API lost connectivity to the database server over the private network, causing the Public Cloud service to become unavailable.
  • July 4, 22:05 — Network connectivity to the database server was restored manually, and all services returned to normal operation.

Root Cause

During the migration to the hosting provider's new private network, DHCP was disabled. Because the server relied on DHCP to obtain its private IP address, it became unreachable from the application servers after its network lease expired.

Without access to the database, SMSGate API could no longer process requests, resulting in the outage.

Resolution

The database server's network configuration was restored manually, re-establishing connectivity between the application and database servers. Once connectivity was restored, SMSGate API resumed normal operation, and the integrity of customer data was verified.

Lessons Learned

This incident exposed an incorrect assumption in our infrastructure that critical services would continue to receive their network configuration through DHCP after the provider's network migration. While the maintenance itself completed successfully, this configuration change caused a delayed failure that was not detected during post-maintenance validation.

Preventive Actions

To reduce the likelihood of similar incidents, the following improvements are being implemented:

  • Replace DHCP with static IP addressing for critical infrastructure components.
  • Add automated monitoring to detect loss of connectivity between application and database servers.
  • Expand post-maintenance validation to verify end-to-end service connectivity after infrastructure changes.
  • Review infrastructure configuration to eliminate similar single points of failure.

Closing

We understand that reliable message delivery is critical for SMSGate API users. Although this incident affected only Public Cloud Server deployments and no customer data was lost, the extended outage did not meet our reliability expectations.

We are committed to learning from this incident and improving the resilience of the service to reduce the risk of similar failures in the future.

Thank you for your continued support of SMSGate API.

Updated
Jul 4, 2026 at 10:13pm UTC

API recovered.

Created
Jul 4, 2026 at 2:50pm UTC

API went down.