Impact

A subset of Platform customers within the South East Asia (SEA) data boundary experienced login failures when accessing their tenant. This issue affected only tenants configured with identity federation for user logins in the SEA region.

Start of Impact (UTC): Oct 19, 2024, 12:36 PM

End of Impact (UTC): Oct 19, 2024, 03:01 PM

Incident Overview

Users with federated identities encountered an error message, “HTTP failure response” and “504 OK,” which prevented access to the platform.

Root cause

The incident was triggered by resource overload in the messaging system, which caused it to restart. This disrupted communication between services and resulted in the login failures.

Preventative Actions

To prevent a recurrence of this issue, we are taking the following actions:

• Enhance monitoring of session encoding services to identify issues early.
• Update the messaging system to allow for automated and faster recovery, reducing the risk of future disruptions.
• Reassess the architecture of the messaging system to prevent future resource conflicts. 

These improvements will be deployed within the next two weeks. We apologize for the disruption and appreciate your patience as we work to enhance platform reliability.