> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xsoar-8-on-prem/8.7/investigate-and-respond-to-threats/threat-intel-management/indicator-configuration/indicator-extraction.md).

# Indicator extraction

Indicator extraction identifies indicators from different text sources in the system (such as War Room entries, email content, etc.), extracts them (usually based on regex) and creates indicators in Cortex XSOAR. After extraction, the indicator can be enriched.

After indicators are extracted, they are enriched using commands and scripts defined for the indicator type. Indicator enrichment provides detailed information about the indicator, based on enrichment feeds such as VirusTotal and IPinfo.

To extract indicators from incoming feeds without enrichment or to prevent enrichment for existing indicators, see [Exclude indicators from enrichment](/cortex-xsoar-8-on-prem/8.8/detect-and-respond-to-threats/threat-intel-management/indicator-configuration/exclude-indicators-from-enrichment.md).

{% hint style="info" %}

### Note

Reputation commands, such as **`!ip`** and **`!domain`**, can only be used after you configure and enable a reputation integration instance, such as Virus Total and Whois.
{% endhint %}

Some content packs include a dashboard and widget that track API rate limit errors. You can use this information for troubleshooting and to make decisions about indicator enrichment.

### Indicator extraction methods

You can customize indicator extraction using the following methods:

* Incident types

  You can extract indicators from incident fields when an incident is created and when an incident field changes. Indicator extraction rules for content pack incident types are determined by the content pack. For example, in a Phishing incident type, by default, in the Destination IP field, IPv6 and IP indicators are extracted. For the Detection URL field, the URL indicator field is extracted.

  If enabled, indicator extraction is automatic. For example, in a Phishing incident, indicator extraction is set to extract the IP indicator (in the incident type). When the incident field updates, the IP indicator field is extracted automatically. In the War Room, you can check that the IP indicator field has been extracted by typing **`1.1.1.1`**. Cortex XSOAR recognizes the indicator as an IP indicator by matching it to the IP indicator’s regex. It then extracts and enriches the indicator using an integration that includes the IP command (such as IPinfo).

  <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><h3>Note</h3><p>To change the indicator extraction rules for an incident type installed with a content pack, including an incident type propagated to a tenant in a multi-tenant environment, you need to detach the incident type. Once detached, the incident type does not receive new content from Cortex XSOAR. If you want to receive content updates reattach the incident type. If you want to instead receive content updates and save the content, duplicate the incident type and edit the duplicate type. For more information, see <a href="/pages/EDiMhGLGwmvYyoez91la">Incident layout customization</a>.</p></div>

  <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><h3>Caution</h3><p>Extracting indicators can adversely affect system performance. We recommend that you define extraction settings for each incident type, as needed.</p><p>For example, for Malware you may want to extract all IP addresses, for Phishing you may only want to extract IP addresses from specific email headers. For attachments, you may want to disable indicator extraction to reduce external API usage and protect restricted data (the hash) from being sent.</p></div>
* Playbook tasks. For more information, see [Set the indicator extraction mode for a playbook task](#UUID-2d8b6e14-5ca3-4b40-daf4-dc5cb2839243).
* Commands: Run a command using the command line in Cortex XSOAR during an investigation. For more information, see [Extract and enrich an indicator](/cortex-xsoar-8-on-prem/8.7/investigate-and-respond-to-threats/threat-intel-management/indicator-investigation/extract-and-enrich-an-indicator.md).

### Indicator extraction mode options

Indicator Extraction supports the following modes:

* None
* Inline
* Out of band
* Use system default

For detailed information about the modes and how to set them up, see [Indicator extraction modes](#UUID-e069cc0c-da62-1d0f-44d7-78f90e6b8b9f).

### Indicator scripts

When creating or editing an indicator type, you can add the following scripts:

* [Formatting scripts](/cortex-xsoar-8-on-prem/8.7/investigate-and-respond-to-threats/threat-intel-management/indicator-configuration/customize-indicator-types-fields-and-layouts.md#UUID-82c0e5e1-e4d5-1e43-b22b-b06e1cdc9be0)
* [Enhancement scripts](/cortex-xsoar-8-on-prem/8.7/investigate-and-respond-to-threats/threat-intel-management/indicator-configuration/customize-indicator-types-fields-and-layouts.md#UUID-e5abe7b4-8aab-e436-f085-8611fddd1fbc)
* [Reputation scripts](/cortex-xsoar-8-on-prem/8.7/investigate-and-respond-to-threats/threat-intel-management/indicator-configuration/customize-indicator-types-fields-and-layouts.md#UUID-7acbe9d4-ed88-4c5e-c185-fab11c8d95f4)

During the indicator extraction and extraction flow, the order of execution is regex, formatting script, and reputation command, reputation script. Enhancement scripts are not part of the flow.

Indicators are identified using regex, and then the formatting script transforms the regex into a usable indicator for use in Cortex XSOAR in the War Room, reports, dashboards, etc. Reputation commands and scripts enable you to change the reputation of the indicator.

Enhancement scripts enable you to gather additional data about the highlighted entry in the War Room.

### Indicator extraction and enrichment in the CLI

You can run commands in the CLI, such as **`!extractIndicators`**, **`!enrichindicators`**, **`!ip`** , **`!domain`**, and reputation script commands such as **`!1URLReputation`**, **`!IPReputation`**. For more information, see [Extract and enrich an indicator](/cortex-xsoar-8-on-prem/8.7/investigate-and-respond-to-threats/threat-intel-management/indicator-investigation/extract-and-enrich-an-indicator.md).

### Indicator extraction modes

Indicator extraction supports the following modes:

* **None**: Indicators are not extracted automatically. Use this option when you do not want to extract and enrich the indicators.
* **Inline**: Indicators are extracted within the context that the indicator extraction runs (synchronously). The findings are added to the context data. For example, if you define indicator extraction for the phishing incident type as inline:
  * For incident creation, by default, the playbook you defined to run does not run until the indicators have been extracted.
  * For an on field change, extraction occurs before the next playbook tasks run. Use this option when you need to have the most robust information available per indicator.

    <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><h3>Note</h3><p>This configuration may delay playbook execution. While indicator creation using the command createIndicator is asynchronous, automatic indicator extraction and enrichment is run synchronously. Data is placed into the incident context and is available via the context for subsequent tasks.</p></div>
* **Out of band**: Indicators are extracted in parallel (asynchronously) to other actions. The extracted data is available within the incident, but it is not available for immediate use in task inputs, or outputs, since the information is not available in real time.

  For incident creation, out of band is used in rare cases where you do not need the indicators extracted for the playbook flow. You still want to extract them and save them in the system as indicators, so that they can be reviewed at a later stage for manual review. System performance may be better as the playbook flow does not stop to extract, but if the incident contains indicators that are needed or expected in the playbook execution flow, inline should be used, as it will not execute the playbook before all indicators are extracted from the incident.

  <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><h3>Note</h3><p>When using Out of band, the extracted indicators do not appear in the context. If you want the extracted indicators to appear select Inline.</p></div>
* **Use system default**: Indicators are extracted according to the following defaults:

  | Component             | Description                                                                                                                                                                                 | Default     |
  | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
  | Incident creation     | Sets the indicator extraction mode for incident creation. It extracts from all associated fields at the point of incident creation. You can change the value when editing an incident type. | Inline      |
  | Incident field change | Sets the indicator extraction mode for incident field change. You can change the value when editing an incident type.                                                                       | Out of band |
  | Tasks                 | Applies to the result of the task. You can change the value when editing a task.                                                                                                            | None        |
  | Manual                | Applies to commands triggered from the CLI. You can change the value when using the indicator extraction parameter.                                                                         | Out of band |

### Create indicator extraction rules for an incident type

You can extract indicators from incident fields on creation of an incident and when a field changes. For example, you might want to extract the IP address upon incident creation and again when the field changes.

![indicator-extract-new-2.png](/files/aCwZpkvlWK4HfHo5ezeZ)

The indicator extraction feature extracts indicators from incident fields and enriches them using commands and scripts defined for the indicator type.

1. Go to Settings & Info → Settings → Object Setup → Incidents → **Types**.
2. For a content pack installed incident type, detach or duplicate the incident type, and then click the detached or duplicated incident type. For custom incident types, click the incident type.
3. From the **Indicators Extraction Rules** tab, in the **On incident creation** and the **On field change** fields, select the required indicator extraction mode.

   If you select **Out of band**, the extracted indicators do not appear in the context. If you want the extracted indicators to appear, select **Inline**. For more information, see [Indicator extraction modes](#UUID-e069cc0c-da62-1d0f-44d7-78f90e6b8b9f).
4. In the **What to Extract** section, if you want to extract all incident fields, select **Extract all indicators from all fields**.
5. If you want to choose which indicators are extracted according to each field, select **Extract specific indicators**.

   You can search and filter the incident fields. For each field, use the dropdown menu to control the indicator types to extract:

   (Optional) You can select all indicators, set all indicators to none, or copy settings from an incident type by clicking ![gear-icon.png](/files/VLdQeqEPaMbmPLWqbWUN) (to the right of the table’s column headers).

   | Indicator type to extract          | Description                                                                                                                                                                          |
   | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
   | **None**                           | No indicators are extracted.                                                                                                                                                         |
   | **All indicator types with regex** | <p>Some indicator types are associated with a regex (such as IP), and some are not (such as Registry Key).</p><p>Only indicators that are associated with a regex are extracted.</p> |
   | **Specific indicator types**       | You can choose one or more indicator types based on regex. The system extracts values that match the regex from this incident field.                                                 |

   Select the **Use field value** checkbox, to use any indicator based on the field value (not regex based). This creates an indicator out of the entire value of the field, regardless whether the indicator type has a configured regex. This can be used in cases such as extracting hostnames.

   ![extract-fig1\_2.png](/files/rY4QZtGdQn5FVRMoxpRU)

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><h3>Note</h3><ul><li>We recommend turning off (setting to <strong>None</strong>) incident extraction for the <strong>Labels</strong> incident field. When an incident JSON is received from an integration, the JSON members are mapped to incident fields (based on the mapping configuration). Every member in the JSON that was not mapped to a field, will be written to the <strong>Labels</strong> field. If the <strong>Labels</strong> field extracts indicators, it can expose unmapped or unknown data to external sources. You should only map the relevant data to fields and set their extraction settings.</li><li>If you want to extract attachments, select the <strong>attachment</strong> field and then select <strong>File</strong> as the indicator type to extract. The <strong>File</strong> extracts a hash (usually SHA-256), which can be viewed in the War Room. You may want to disable indicator extraction for attachments to reduce external API usage and protect restricted data (the hash) from being sent.</li></ul></div>
6. Click **Save**.
7. (Optional) If you want to configure which scripts and commands the indicator type executes, go to **Settings & Info** → **Settings** → **Object Setup** → **Indicators** → **Types** and edit or [Create an indicator type](/cortex-xsoar-8-on-prem/8.7/investigate-and-respond-to-threats/threat-intel-management/indicator-configuration/customize-indicator-types-fields-and-layouts.md#UUID-88723ccc-3ff9-b905-e57a-d9b13bdcc72f).

   Add scripts and reputation commands for the indicator type. When indicator extraction occurs, indicators are extracted as defined in an indicator type, and enriched using the commands and scripts associated with the indicator type. For example, the URL indicator is enriched using the **`!url`** command.

In this example, if an email is forwarded that potentially includes phishing, we want to extract at incident creation (inline) and upon a field change (out of band):

* **Campaign Email Subject**: Extract all indicators.
* **Campaign Email Body**: Extract all indicators.
* **Email Delete Result**: Extract email only.
* **Email Delete Reason**: Extract email only.

![extract-fig-2-2.png](/files/ws2e1ZRtx0qoUjy2qGG7)

### Set the indicator extraction mode for a playbook task

You can set the indicator extraction mode for specific playbook tasks.

1. Select the playbook where you want to add indicator extraction to a task, and click **Edit**.
2. In the playbook, click a task to open the **Edit Task** window.
3. Click the **Advanced** tab.
4. In the indicator extraction drop-down menu, select the mode you want to use.
5. Click **OK**.

### Disable indicator extraction for scripts or integrations

This procedure describes how to disable indicator extraction for a specific script or an integration.

* To disable indicator extraction for a script, add the **`IgnoreAutoExtract`** entry with the value of **`true`**, when returning an entry.

  For example:

  ```programlisting
  entry = {
      'Type': entryTypes['note'],
      'Contents': {
      'Echo' : demisto.args()['echo']
          },
      'ContentsFormat': formats['json'],
      'ReadableContentsFormat': formats['markdown'],
      'HumanReadable': hr,
      'IgnoreAutoExtract' : True
     }
  ```
* To disable indicator extraction for an integration, add the **`'IgnoreAutoExtract'`** entry with the value of **`true`**, when returning an entry.

  For example in the ServiceNow integration:

  ```programlisting
  entry = {
          'Type': entryTypes['note'],
          'Contents': result,
          'ContentsFormat': formats['json'],
          'ReadableContentsFormat': formats['markdown'],
          'HumanReadable': tableToMarkdown('ServiceNow ticket', hr, headers=headers, removeNull=True),
          'EntryContext': {
              'Ticket(val.ID===obj.ID)': context,
              'ServiceNow.Ticket(val.ID===obj.ID)': context
          },
          'IgnoreAutoExtract': True
      }
      entries.append(entry)
      return entries
  ```

For more information about command results in Python, see [Python code conventions for CommandResults](https://xsoar.pan.dev/docs/integrations/code-conventions#commandresults).

### Troubleshoot indicator extraction

If indicators are not extracting, check whether the indicator mode is set to none. Even if you select the relevant incident fields and the indicators to extract, if the mode is set to none, indicators do not extract.

When creating new incident types, if you select **Extract all indicators from all fields**, all fields are extracted including custom fields. If you select **Extract specific indicators by default**, indicator extraction for new custom fields is set to none.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://cortex-docs.paloaltonetworks.com/cortex-xsoar-8-on-prem/8.7/investigate-and-respond-to-threats/threat-intel-management/indicator-configuration/indicator-extraction.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
