Secret Server Cloud: Remote Password Changing (RPC) and Heartbeat failures

Incident Overview

Summary

Secret Server Cloud customers using Distributed Engine (DE) for Remote Password Changing (RPC) operations experienced RPC requests remaining in a “Pending” state. In some cases, RPC functionality was partially restored before full service was recovered. A follow-on regression occurred during remediation, temporarily impacting a subset of customers again.

First Occurrence

• Start Time: December 6, 2025, 5:44 PM ET
• End Time: December 7, 2025, 2:35 AM ET

Second Occurrence (Follow-on Regression)

• Start Time: December 14, 2025, 10:53 AM ET
• End Time: December 14, 2025, 12:00 PM ET

Impact

• RPC operations initiated through Distributed Engine remained in a pending state and did not complete successfully.
• Other Secret Server Cloud functionality was not impacted.

Root Cause

The issue was caused by enhancements introduced as part of the RPC pre-run validation feature, which is tightly integrated with the RPC execution flow.

The enhancement required changes to how RPC requests are structured and processed. These changes introduced compatibility issues between the updated Secret Server Cloud service and certain previously installed Distributed Engine versions, preventing RPC requests from being processed correctly.

During the initial rollback, additional infrastructure changes included in the same release prevented an automatic rollback from fully completing, which extended the time required to restore full functionality.

During the subsequent remediation rollout on Dec 13, 2025, a configuration issue related to message routing caused a temporary regression, impacting a subset of Distributed Engines until the configuration was corrected.

Preventive Actions

To prevent recurrence, we have taken and are taking the following actions:

• Improved QA validation to explicitly test upgrades across all supported Distributed Engine versions, not just incremental or phase-based upgrades.
• Ensured large, foundational features such as RPC pre-run validation are integrated into QA environments earlier in the release cycle to better reflect production conditions.
• Added broader cross-team technical reviews for complex, highly integrated features before release.
• Implemented backward compatibility safeguards to ensure newer Secret Server Cloud releases can safely interoperate with older Distributed Engine versions.
• Enhanced deployment validation and monitoring to more quickly detect configuration regressions during remediation rollouts.