Incident De-Duplication

De-duplicate incidents in Cortex XSOAR 6.12.

In the lifecycle of incident management, there are cases when incidents are duplicated. Cortex XSOAR provides the following de-duplication capabilities:

• Manual De-Duplication: You can manually de-duplicate incidents from the Incidents page or the Related Incidents page. To de-duplicate incidents manually, see Manually De-Duplicate Incidents.
• Automatic De-Duplication: You can automate de-duplicate incidents by using Pre-Process Rules and Scripts.
• Automations: You can create an automation that creates child incidents from duplicates.
• Playbooks: Identify, review or close duplicate incidents using playbooks.

  There are several out-of-the-box playbooks you can run to identify and close duplicate incidents. Alternatively, you can use these playbooks as the basis for customized de-duplication playbooks. For example, instead of automatically closing the duplicate incidents, include a manual review of the duplicate incidents.

  Playbook	Description
  Dedup - Generic v4	Identifies duplicate incidents using the machine learning model (used mainly for phishing).
  DeDup - Generic v3	Identifies duplicate incidents using one of the supported methods, such as rules, text, and machine learning.

Manually De-Duplicate Incidents

You can manually de-duplicate incidents from the Incidents page or the Related Incidents tab.

• To de-duplicate incidents from the Incidents page:

  1. Select the incident checkboxes that are duplicates.
  2. Click Mark as Duplicate.
  3. Type the ID of the incident that you want to retain.
  4. Click Validate.
  5. Click Close others as duplicate.

     The duplicated incident is removed from the table.
• To de-duplicate incidents from the Related Incidents tab:

  1. From the Incidents page, select the relevant incident checkbox.
  2. In the Related Incidents tab, click the related incident you want to de-duplicate.
  3. Click Close as duplicate.

     The duplicated incident is closed and marked as duplicate.

Automatic De-Duplication Using Scripts

There are various scripts you can use in automations and playbooks to identify and close duplicate incidents:

• FindSimilarIncidentsByText
• FindSimilarIncidents
• GetDuplicatesMI

FindSimilarIncidentsByText

• Identifies similar incidents based on text similarity. For this script you specify incident keys, labels, or custom fields.
• The comparison is based on the TF-IDF method.
• A score is calculated for each candidate (0-1), and incidents are considered duplicates when exceeding the threshold. The default threshold is 98%.

!FindSimilarIncidentsByText textFields=name,details maximumNumberOfIncidents=1000 threshold=0.95 timeFrameHours=24 ignoreClosedIncidents=no

This command example checks for duplicate incidents using the following methodology:

1. Query for duplicate candidates:

   • Incidents created in the previous 24 hours [timeFrameHours=24].
   • Includes closed incidents [ignoreClosedIncidents].
   • Maximum number of incidents to check is 1,000 [maximumNumberOfIncidents=1000].
2. For each candidate, concatenate name and details incident fields [textFields=name,details] into a text document.
3. Compare the current incident text with all candidates using the TF-IDF method
4. Check if there is at least one similar candidate:

   • Candidates with a TF-IDF score of 95% [threshold=0.95]. If there is at least one candidate, announce duplicate.

FindSimilarIncidents

• Rule-based script that identifies similar incidents based on common incident keys, labels, custom fields, or context keys.
• We recommend using incident keys, for example, "type" for same incident type.
• Due to performance considerations, we recommend not using context keys, for example, if the value also appears in the label key. Each duplicate candidate creates an additional server query.

!FindSimilarIncidents similarIncidentKeys="type,severity" similarLabelsKeys="Email/from,Email/subject:*,Email/text:5" ignoreClosedIncidents="yes" maxNumberOfIncidents="1000" hoursBack="48" timeField="created" maxResults="10"

This command example checks for duplicate incidents using the following methodology:

1. Query for duplicate candidates:

   • Incidents created in the 48 hours [hoursBack="48", timeField=created] before the original incidents
   • Excludes closed incidents [ignoreClosedIncidents=yes]
   • Maximum number of incidents to check is 1,000 [maxNumberOfIncidents=1000]
   • Filters by the same incident type and severity [similarIncidentKeys=type,severity]
2. Check for candidate with the same Email/from label, or similar Email/subject label:

   • Contains, or contained, the original incident Email/subject label, and similar Email/text label
   • Equal or a maximum difference of 5 words from the original Email/text label [similarLabelsKeys="Email/from,Email/subject:*,Email/text:5"]
3. If duplicate incidents are found, store the results in the context:

   • Maximum of 10 [maxResults="10"]

GetDuplicatesMI

• Identifies duplicate incidents based on a machine learning (ML) algorithm, which uses ML techniques with predefined data. Alternatively, you can use data from the local environment.
• This script takes several features into consideration: labels comparison, email labels (relevant for phishing scenarios), incident time difference, and shared indicators (which you can customize with arguments).

!GetDuplicatesMl maxNumberOfIncidents="1000" timeFrameDays="7" ignoreClosedIncidents="yes" threshold="0.5" compareIndicators="Email, IP, Domain, File SHA256, File MD5, URL" compareEmailLabels="Email/headers/From, Email/headers/Subject, Email/text, Email/html, Email/attachments" compareOtherLabels="yes" compareIncidentTimeDiff="yes" UseLocalEnvDuplicatesInLastDays="0" ipComparisonSubnetMask="32" maxCandidates="10"

This command example checks for duplicate incidents using the following methodology.

1. Query for duplicate candidates:

   • Incidents created in the 7 days [timeFrameDays="7"] before the original incidents
   • Excludes closed incidents [ignoreClosedIncidents=yes]
   • Maximum number of incidents to check is 1,000 [maxNumberOfIncidents=1000]
2. For each candidate calculate features based on similar email labels and other labels:

   • Email labels [compareEmailLabels, compareOtherLabels]
   • Indicators [compareIndicators, ipComparisonSubnetMask]
   • Time difference between the incidents [compareIncidentTimeDiff]
3. Build machine learning model based on pre-defined data set:

   • Does not take into account local environment data set - linked and duplicate incidents in the system [UseLocalEnvDuplicatesInLastDays=0]
4. Predict whether each candidate is a duplicate:

   • Prediction is based on a score (probability) between 0-1