> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/xsoar-6-administrator-guide/6.12/configure-cortex-xsoar/customize-and-configure-cortex-xsoar/incidents/incident-de-duplication.md).

# Incident De-Duplication

In the lifecycle of incident management, there are cases when incidents are duplicated. Cortex XSOAR provides the following de-duplication capabilities:

* **Manual De-Duplication**: You can manually de-duplicate incidents from the **Incidents** page or the **Related Incidents** page. To de-duplicate incidents manually, see [Manually De-Duplicate Incidents](#UUID-4df07e5e-08bb-1348-938d-1e9ac018da75).
* **Automatic De-Duplication**: You can automate de-duplicate incidents by using [Pre-Process Rules](/xsoar-6-administrator-guide/6.12/configure-cortex-xsoar/customize-and-configure-cortex-xsoar/incidents/pre-process-rules.md) and [Scripts](#UUID-f47fef00-753a-54d3-b6e3-b89201fa9c68).
* **Automations**: You can create an automation that creates child incidents from duplicates.
* **Playbooks**: Identify, review or close duplicate incidents using playbooks.

  There are several out-of-the-box playbooks you can run to identify and close duplicate incidents. Alternatively, you can use these playbooks as the basis for customized de-duplication playbooks. For example, instead of automatically closing the duplicate incidents, include a manual review of the duplicate incidents.

  | Playbook                                                                                | Description                                                                                                   |
  | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
  | [Dedup - Generic v4](https://xsoar.pan.dev/docs/reference/playbooks/dedup---generic-v4) | Identifies duplicate incidents using the machine learning model (used mainly for phishing).                   |
  | [DeDup - Generic v3](https://xsoar.pan.dev/docs/reference/playbooks/dedup---generic-v3) | Identifies duplicate incidents using one of the supported methods, such as rules, text, and machine learning. |

### **Manually De-Duplicate Incidents**

You can manually de-duplicate incidents from the **Incidents** page or the **Related Incidents** tab.

* To de-duplicate incidents from the **Incidents** page:
  1. Select the incident checkboxes that are duplicates.
  2. Click **Mark as Duplicate**.
  3. Type the ID of the incident that you want to retain.
  4. Click **Validate**.
  5. Click **Close others as duplicate**.

     The duplicated incident is removed from the table.
* To de-duplicate incidents from the **Related Incidents** tab:
  1. From the **Incidents** page, select the relevant incident checkbox.
  2. In the **Related Incidents** tab, click the related incident you want to de-duplicate.
  3. Click **Close as duplicate**.

     The duplicated incident is closed and marked as duplicate.

### **Automatic De-Duplication Using Scripts**

There are various scripts you can use in automations and playbooks to identify and close duplicate incidents:

* FindSimilarIncidentsByText
* FindSimilarIncidents
* GetDuplicatesMI

**FindSimilarIncidentsByText**

* Identifies similar incidents based on text similarity. For this script you specify incident keys, labels, or custom fields.
* The comparison is based on the [TF-IDF method](https://en.wikipedia.org/wiki/Tf-idf).
* A score is calculated for each candidate (0-1), and incidents are considered duplicates when exceeding the threshold. The default threshold is 98%.

**`!FindSimilarIncidentsByText textFields=name,details maximumNumberOfIncidents=1000 threshold=0.95 timeFrameHours=24 ignoreClosedIncidents=no`**

This command example checks for duplicate incidents using the following methodology:

1. Query for duplicate candidates:
   * Incidents created in the previous 24 hours \[timeFrameHours=24].
   * Includes closed incidents \[ignoreClosedIncidents].
   * Maximum number of incidents to check is 1,000 \[maximumNumberOfIncidents=1000].
2. For each candidate, concatenate name and details incident fields \[textFields=name,details] into a text document.
3. Compare the current incident text with all candidates using the TF-IDF method
4. Check if there is at least one similar candidate:
   * Candidates with a TF-IDF score of 95% \[threshold=0.95]. If there is at least one candidate, announce duplicate.

**FindSimilarIncidents**

* Rule-based script that identifies similar incidents based on common incident keys, labels, custom fields, or context keys.
* We recommend using incident keys, for example, "type" for same incident type.
* Due to performance considerations, we recommend not using context keys, for example, if the value also appears in the label key. Each duplicate candidate creates an additional server query.

**`!FindSimilarIncidents similarIncidentKeys="type,severity" similarLabelsKeys="Email/from,Email/subject:*,Email/text:5" ignoreClosedIncidents="yes" maxNumberOfIncidents="1000" hoursBack="48" timeField="created" maxResults="10"`**

This command example checks for duplicate incidents using the following methodology:

1. Query for duplicate candidates:
   * Incidents created in the 48 hours \[hoursBack="48", timeField=created] before the original incidents
   * Excludes closed incidents \[ignoreClosedIncidents=yes]
   * Maximum number of incidents to check is 1,000 \[maxNumberOfIncidents=1000]
   * Filters by the same incident type and severity \[similarIncidentKeys=type,severity]
2. Check for candidate with the same Email/from label, or similar Email/subject label:
   * Contains, or contained, the original incident Email/subject label, and similar Email/text label
   * Equal or a maximum difference of 5 words from the original Email/text label \[similarLabelsKeys="Email/from,Email/subject:\*,Email/text:5"]
3. If duplicate incidents are found, store the results in the context:
   * Maximum of 10 \[maxResults="10"]

**GetDuplicatesMI**

* Identifies duplicate incidents based on a machine learning (ML) algorithm, which uses ML techniques with predefined data. Alternatively, you can use data from the local environment.
* This script takes several features into consideration: labels comparison, email labels (relevant for phishing scenarios), incident time difference, and shared indicators (which you can customize with arguments).

**`!GetDuplicatesMl maxNumberOfIncidents="1000" timeFrameDays="7" ignoreClosedIncidents="yes" threshold="0.5" compareIndicators="Email, IP, Domain, File SHA256, File MD5, URL" compareEmailLabels="Email/headers/From, Email/headers/Subject, Email/text, Email/html, Email/attachments" compareOtherLabels="yes" compareIncidentTimeDiff="yes" UseLocalEnvDuplicatesInLastDays="0" ipComparisonSubnetMask="32" maxCandidates="10"`**

This command example checks for duplicate incidents using the following methodology.

1. Query for duplicate candidates:
   * Incidents created in the 7 days \[timeFrameDays="7"] before the original incidents
   * Excludes closed incidents \[ignoreClosedIncidents=yes]
   * Maximum number of incidents to check is 1,000 \[maxNumberOfIncidents=1000]
2. For each candidate calculate features based on similar email labels and other labels:
   * Email labels \[compareEmailLabels, compareOtherLabels]
   * Indicators \[compareIndicators, ipComparisonSubnetMask]
   * Time difference between the incidents \[compareIncidentTimeDiff]
3. Build machine learning model based on pre-defined data set:
   * Does not take into account local environment data set - linked and duplicate incidents in the system \[UseLocalEnvDuplicatesInLastDays=0]
4. Predict whether each candidate is a duplicate:
   * Prediction is based on a score (probability) between 0-1


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://cortex-docs.paloaltonetworks.com/xsoar-6-administrator-guide/6.12/configure-cortex-xsoar/customize-and-configure-cortex-xsoar/incidents/incident-de-duplication.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.