Secret Server Cloud: EU - Intermittent failures with API calls and Launching Secrets

Incident Overview

On May 11, 2026, starting at 07:53 UTC, Secret Server Cloud customers in the EU region experienced intermittent failures when launching secrets, initiating proxied RDP/SSH sessions, and making API calls requiring distributed engine communication. The incident was traced to a degradation in the underlying cloud messaging infrastructure in the West Central Europe region. At 13:37 UTC, the degraded performance affecting our services was fully resolved and normal operations were restored.

The impact was limited to SSC customers with Distributed Engines. Secret viewing, management, and Web UI availability remained unaffected.

Root Cause

A degradation in the cloud messaging infrastructure in the West Central Europe region caused message subscription management operations to return HTTP 504 Gateway Timeout errors, preventing Distributed Engines from completing initialization and taking them offline. This resulted in timeouts across all distributed engine-routed operations, most visibly secret launches and proxied session initiations.

The failure was isolated to the control plane layer of the messaging infrastructure. TCP-level connectivity remained healthy throughout the incident, and the issue was not attributed to any network or configuration change on our side.

The issue was mitigated by our Cloud provider rolling back a recent release on the messaging infrastructure that had contributed to the control plane failures.

Preventive Actions

• Expand monitoring coverage for cloud messaging exception rates and Distributed Engine subscription failure patterns to enable proactive detection ahead of customer impact.
• Review integration of Cloud provider health notifications into our on-call alerting pipeline to improve visibility into infrastructure events affecting Secret Server Cloud regions.
• Assess improvements to Distributed Engine startup and reconnection logic to introduce retry handling with exponential back-off on transient messaging failures, reducing the risk of short-lived disruptions escalating into sustained engine outages.

Lessons Learned

The duration of customer impact during this incident was extended by gaps in our operational response. Specifically:

• Limited visibility into cloud provider health events delayed our awareness of the underlying infrastructure degradation, and we did not follow our standard operating procedure to escalate with our vendor in a timely manner.
• Acknowledgment of the incident on our status page was delayed, deviating from our standard incident communication process.
• This incident reinforced the importance of continual improvements in both our monitoring and situational awareness of our infrastructure, as well as in our engineer training and development.

We apologize for the extended impact our handling of this incident had on our customers and on their operations. We continue to take our responsibilities to our customers seriously, and have taken lessons from the handling of this incident to strengthen our processes going forward.