Failures logging into Delinea Platform tenants on Friday October 4th, 2024
Impact
A subset of Delinea Platform customers within the EU data boundary experienced login failures when accessing their tenant. The issue affected only EU tenants configured with identity federation for user logins.
Start of Impact (UTC): Oct 4, 2024, 9:14 AM
End of Impact (UTC): Oct 4, 2024, 12:50 PM
Incident Overview
We apologize for the recent login issues affecting your tenant. Users with federated identities encountered an error message, "This page isn’t working," accompanied by "HTTP ERROR 500" message, preventing them from accessing their platform tenant.
Root cause
The outage was triggered by a bug in the latest audit release, causing contention in the messaging broker's queues. After identifying the issue, we rolled back the faulty release, reducing the load and stabilizing the system, allowing the federated login service to resume normal operations.
Preventative Actions
To prevent similar incidents in the future, we are taking the following steps:
- Enhance monitoring with synthetic tests for end-to-end coverage.
- Strengthen backend monitoring of the messaging broker.
- Reassess messaging queue architecture to prevent resource contention.
- Implement automated tests to improve the QA process for the Audit service release.
Once again, we apologize for the disruption this caused and appreciate your patience as we work to improve our platform's reliability.