Secret Server Cloud - Discovery Import Rule Issues
Incident Report for Delinea
Postmortem

Impact

Duplicate Secrets were created for Cloud customers with certain specific Discovery configurations.
The impact varied with the specific configuration.

Start of Impact (EST): Jan 23, 2025, 12:00 AM

End of Impact (EST): Jan 26, 2025, 6:30 PM

Incident Overview

Discovery Import Rules configured in a particular way created duplicate Secrets.
Dependencies on the duplicated Secrets (Windows Services, Scheduled Tasks, etc.) may have stopped working.

Duplicate Secrets were created only for customers with the following configuration:

  1. Cloud customers only.
  2. Non-computer accounts (Active Directory, Entra Id) scanning enabled on the Discovery Source.
  3. “Discover Specific OUs” enabled on the Discovery Source.
  4. One or more Discovery Import Rules created on the source for the non-computer accounts.

Customers in any of the following situations are not impacted:

  1. On-premises customers.
  2. Customers who are only using computer-associated account (Local Windows, Unix, etc.) scanning.
  3. Customers who are not using “Discover Specific OUs”.
  4. Customers who are not using Discovery Import Rules.

Dependencies (Windows Services, Scheduled Tasks, etc.) may have stopped working when:

  1. A pre-existing Secret exists for non-computer account and has dependencies (Windows Service, Scheduled Task, etc.)
  2. A Discovery Import Rule exists which covers the non-computer account and is configured to take over the account during import.

In this configuration, the bug would cause the pre-existing Secret to be disassociated from the non-computer account.
The Discovery Import Rule would run and generate a duplicate Secret, and change the password of the Secret. This could make
any dependencies using the original Secret's password fail, as the password they had would be out of date.

Root cause

A code change to address a reported issue in Discovery caused a side effect which disassociated Secrets
from discovered non-computer accounts when "Discover Specific OUs" was enabled.
The Discovery Import Rule process uses this association to
prevent importing Secrets multiple times. Since the Secrets became disassociated, the Discovery Import Rule
process followed its configured actions and created the Secrets again. Depending on the configured interval,
this may have happened multiple times within the incident window.

Preventative Actions

To prevent a recurrence of this issue, we are taking the following actions:

  • Extend our test coverage of the functionality which associates Secrets to accounts and imports new Secrets from Discovery.
  • Extend our pre-release and continuous automation testing of Discovery Rules.

Customer link to our knowledge base articles providing more information on this issue as well as diagnostic and remediation tools:
: https://support.delinea.com/s/article/1738106421935

Posted Jan 31, 2025 - 10:39 EST

Resolved
The Discovery Import Rule issue has been resolved by the rollback.
Posted Jan 26, 2025 - 18:36 EST
Monitoring
The rollback has been completed in all regions.
Posted Jan 26, 2025 - 16:12 EST
Identified
An issue has been identified in the Thursday, January 23rd patch. Discovery Import Rules will create duplicate Secrets for active directory accounts when Discover Specific OUs is enabled. A rollback is in process to mitigate the issue.
Posted Jan 26, 2025 - 15:36 EST
This incident affected: US (Secret Server Cloud), UK (Secret Server Cloud), EU (Secret Server Cloud), SEA (Secret Server Cloud), AU (Secret Server Cloud), and CA (Secret Server Cloud).