Platform: Global - Privileged Remote Access sessions could not be launched
Postmortem
Impact
Some Platform users were unable to launch Privileged Remote Access (PRA) sessions.
- Start of Impact: March 19, 2025, at 14:27 UTC
- End of Impact: March 19, 2025, at 17:07 UTC
Incident Overview
Some Platform users experienced intermittent failures when attempting to launch Privileged Remote Access (PRA) sessions, receiving the error message:
"Unable to launch secret - an unexpected error has occurred. Contact your administrator or try again."
The issue was introduced in a recent platform update that modified how site data is queried within the PRA service. These changes inadvertently affected users with a legacy PRA configuration that lacked certain platform dependencies. While the issue was not observed in pre-production environments, some production tenants encountered failures due to missing dependencies.
To mitigate the impact, we rolled back to the previous version, fully restoring functionality by March 19, 17:07 UTC.
Root cause
The failure was caused by a dependency mismatch between the PRA APIs and a supporting platform service responsible for managing site configurations. The update introduced changes to how site data is retrieved, but did not account for scenarios where customers were using PRA as a standalone application without certain platform dependencies. This resulted in failed queries for a subset of users. Additionally, our pre-release testing did not include environments without these dependencies, allowing the issue to go undetected before deployment.
Preventative Actions & Next Steps
To prevent similar issues in the future, we are taking the following actions:
- A fix has been developed and is currently undergoing testing in the pre-production environment. It will be incorporated into the next release.
- Additional test cases will be introduced to cover scenarios where PRA is used as a standalone application without certain platform dependencies. We are refining our processes on identifying service dependencies and potential impact areas before releasing changes.
- More automated integration tests will be implemented to ensure broader coverage of different platform configurations.
Resolved
End Time: March 19, 17:07 UTC
Impact: A subset of users intermittently encountered errors when attempting to start Privileged Remote Access (PRA) sessions on a secret.
Incident Details:
Beginning at 14:27 UTC on March 19, some users experienced failures when attempting to launch a PRA session, receiving the error:
"Unable to launch secret - an unexpected error has occurred. Contact your administrator or try again."
The issue was intermittent and did not affect all users. Upon investigation, we identified a problem introduced in our latest release. To mitigate the impact, we performed a rollback to the previous version.
Resolution:
The rollback was completed, and by 17:07 UTC, the issue was fully resolved. No further failures have been observed since.
Next Steps:
We will conduct a root cause analysis (RCA) and provide an update within seven business days.