Platform: US - Intermittent Login Failures

Incident Overview

Users in the US region experienced intermittent login failures while accessing Platform tenants. Some users also experienced session logouts during this period. Affected users were generally able to log in successfully after retrying.

Tenants outside the US region remained fully operational. The issue was detected promptly through our internal monitoring, and engineering teams were immediately engaged.

Impact started: Oct 9, 2025, 2:21 PM (GMT-05:00)
Impact ended: Oct 9, 2025, 2:34 PM (GMT-05:00)

Root Cause

Multiple Platform services rely on a shared message queueing service to publish and consume messages. One of the services did not have sufficient consumers to process queued messages. A sudden spike in message volume caused the underlying storage for the message queue to approach its capacity limits, resulting in intermittent login failures and session timeouts.

The issue was identified quickly through proactive alerting, and engineers mitigated it by expanding the available storage, restoring normal operations.

 Preventive Actions

1. We are adding alerts at lower disk space utilization thresholds to detect and act before reaching critical limits.
2. Implement new monitors to identify and alert on orphaned or under-consumed queues that may accumulate unprocessed messages.
3. Continue efforts to migrate message queue hosting in-house to improve control over resource scaling, performance tuning, and monitoring.