Incident Overview

On July 27, 2026, Secret Server Cloud (US region) customers experienced widespread failures with RDP sessions, remote password rotation, and folder creation. The root cause was a critical internal caching layer that filled up to capacity because session entries were being stored in it without expiration. As the cache filled, it began rejecting write operations, causing a backend processing service to fail. Messages continually accumulated in an internal message queue which degraded performance for all affected tenants.

• Start: July 27, 2026, approximately 2:00 PM EDT
• End: July 27, 2026, approximately 4:10 PM EDT
• Region impacted: US

Root Cause and Remediation

The internal caching layer had been accumulating session data with no expiration for over 20 days. At the time of investigation, approximately 86% of cache entries had no expiration set. The root cause is a code-level bug that was not setting expirations properly on session entries. Once the cache reached capacity, new writes were rejected, breaking the log pipeline and cascading into broader service failures across tenants.

To remediate, the caching layer was manually flushed at approximately 4:04 PM EDT, restoring write capacity and allowing the message queue to drain. A code fix to add TTL (time-to-live) to the affected keys has been identified and is in progress.

Preventative Actions

• Apply a code fix to add TTL to session entries and other unbounded growth keys in the caching layer, preventing recurrence regardless of deployment cadence.
• Add memory utilization alerting for the caching layer at defined thresholds, so memory pressure is detected proactively before capacity is reached.
• We are looking into adding a post-release action for flushing the cache as a standard step in the deployment process

We sincerely apologize for the disruption this caused and the inconvenience to your operations. We are committed to preventing recurrence through the above actions.