Indicator extraction

Extract indicators in Cortex XSOAR On-prem 8.7 (EoL).

Indicator extraction identifies indicators from different text sources in the system (such as War Room entries, email content, etc.), extracts them (usually based on regex) and creates indicators in Cortex XSOAR. After extraction, the indicator can be enriched.

After indicators are extracted, they are enriched using commands and scripts defined for the indicator type. Indicator enrichment provides detailed information about the indicator, based on enrichment feeds such as VirusTotal and IPinfo.

To extract indicators from incoming feeds without enrichment or to prevent enrichment for existing indicators, see Exclude indicators from enrichment.

Note

Reputation commands, such as !ip and !domain, can only be used after you configure and enable a reputation integration instance, such as Virus Total and Whois.

Some content packs include a dashboard and widget that track API rate limit errors. You can use this information for troubleshooting and to make decisions about indicator enrichment.

Indicator extraction methods

You can customize indicator extraction using the following methods:

• Incident types

  You can extract indicators from incident fields when an incident is created and when an incident field changes. Indicator extraction rules for content pack incident types are determined by the content pack. For example, in a Phishing incident type, by default, in the Destination IP field, IPv6 and IP indicators are extracted. For the Detection URL field, the URL indicator field is extracted.

  If enabled, indicator extraction is automatic. For example, in a Phishing incident, indicator extraction is set to extract the IP indicator (in the incident type). When the incident field updates, the IP indicator field is extracted automatically. In the War Room, you can check that the IP indicator field has been extracted by typing 1.1.1.1. Cortex XSOAR recognizes the indicator as an IP indicator by matching it to the IP indicator’s regex. It then extracts and enriches the indicator using an integration that includes the IP command (such as IPinfo).

  Note

  To change the indicator extraction rules for an incident type installed with a content pack, including an incident type propagated to a tenant in a multi-tenant environment, you need to detach the incident type. Once detached, the incident type does not receive new content from Cortex XSOAR. If you want to receive content updates reattach the incident type. If you want to instead receive content updates and save the content, duplicate the incident type and edit the duplicate type. For more information, see Incident layout customization.

  Caution

  Extracting indicators can adversely affect system performance. We recommend that you define extraction settings for each incident type, as needed.

  For example, for Malware you may want to extract all IP addresses, for Phishing you may only want to extract IP addresses from specific email headers. For attachments, you may want to disable indicator extraction to reduce external API usage and protect restricted data (the hash) from being sent.
• Playbook tasks. For more information, see Set the indicator extraction mode for a playbook task.
• Commands: Run a command using the command line in Cortex XSOAR during an investigation. For more information, see Extract and enrich an indicator.

Indicator extraction mode options

Indicator Extraction supports the following modes:

• None
• Inline
• Out of band
• Use system default

For detailed information about the modes and how to set them up, see Indicator extraction modes.

Indicator scripts

When creating or editing an indicator type, you can add the following scripts:

• Formatting scripts
• Enhancement scripts
• Reputation scripts

During the indicator extraction and extraction flow, the order of execution is regex, formatting script, and reputation command, reputation script. Enhancement scripts are not part of the flow.

Indicators are identified using regex, and then the formatting script transforms the regex into a usable indicator for use in Cortex XSOAR in the War Room, reports, dashboards, etc. Reputation commands and scripts enable you to change the reputation of the indicator.

Enhancement scripts enable you to gather additional data about the highlighted entry in the War Room.

Indicator extraction and enrichment in the CLI

You can run commands in the CLI, such as !extractIndicators, !enrichindicators, !ip , !domain, and reputation script commands such as !1URLReputation, !IPReputation. For more information, see Extract and enrich an indicator.

Indicator extraction modes

Indicator extraction supports the following modes:

• None: Indicators are not extracted automatically. Use this option when you do not want to extract and enrich the indicators.
• Inline: Indicators are extracted within the context that the indicator extraction runs (synchronously). The findings are added to the context data. For example, if you define indicator extraction for the phishing incident type as inline:

  • For incident creation, by default, the playbook you defined to run does not run until the indicators have been extracted.
  • For an on field change, extraction occurs before the next playbook tasks run. Use this option when you need to have the most robust information available per indicator.

    Note

    This configuration may delay playbook execution. While indicator creation using the command createIndicator is asynchronous, automatic indicator extraction and enrichment is run synchronously. Data is placed into the incident context and is available via the context for subsequent tasks.
• Out of band: Indicators are extracted in parallel (asynchronously) to other actions. The extracted data is available within the incident, but it is not available for immediate use in task inputs, or outputs, since the information is not available in real time.

  For incident creation, out of band is used in rare cases where you do not need the indicators extracted for the playbook flow. You still want to extract them and save them in the system as indicators, so that they can be reviewed at a later stage for manual review. System performance may be better as the playbook flow does not stop to extract, but if the incident contains indicators that are needed or expected in the playbook execution flow, inline should be used, as it will not execute the playbook before all indicators are extracted from the incident.

  Note

  When using Out of band, the extracted indicators do not appear in the context. If you want the extracted indicators to appear select Inline.
• Use system default: Indicators are extracted according to the following defaults:

  Component	Description	Default
  Incident creation	Sets the indicator extraction mode for incident creation. It extracts from all associated fields at the point of incident creation. You can change the value when editing an incident type.	Inline
  Incident field change	Sets the indicator extraction mode for incident field change. You can change the value when editing an incident type.	Out of band
  Tasks	Applies to the result of the task. You can change the value when editing a task.	None
  Manual	Applies to commands triggered from the CLI. You can change the value when using the indicator extraction parameter.	Out of band

Create indicator extraction rules for an incident type

You can extract indicators from incident fields on creation of an incident and when a field changes. For example, you might want to extract the IP address upon incident creation and again when the field changes.

indicator-extract-new-2.png

The indicator extraction feature extracts indicators from incident fields and enriches them using commands and scripts defined for the indicator type.

1. Go to Settings & Info → Settings → Object Setup → Incidents → Types.
2. For a content pack installed incident type, detach or duplicate the incident type, and then click the detached or duplicated incident type. For custom incident types, click the incident type.
3. From the Indicators Extraction Rules tab, in the On incident creation and the On field change fields, select the required indicator extraction mode.

   If you select Out of band, the extracted indicators do not appear in the context. If you want the extracted indicators to appear, select Inline. For more information, see Indicator extraction modes.
4. In the What to Extract section, if you want to extract all incident fields, select Extract all indicators from all fields.
5. If you want to choose which indicators are extracted according to each field, select Extract specific indicators.

   You can search and filter the incident fields. For each field, use the dropdown menu to control the indicator types to extract:

   (Optional) You can select all indicators, set all indicators to none, or copy settings from an incident type by clicking (to the right of the table’s column headers).

   Indicator type to extract	Description
   None	No indicators are extracted.
   All indicator types with regex	Some indicator types are associated with a regex (such as IP), and some are not (such as Registry Key). Only indicators that are associated with a regex are extracted.
   Specific indicator types	You can choose one or more indicator types based on regex. The system extracts values that match the regex from this incident field.

   Select the Use field value checkbox, to use any indicator based on the field value (not regex based). This creates an indicator out of the entire value of the field, regardless whether the indicator type has a configured regex. This can be used in cases such as extracting hostnames.

   

   extract-fig1_2.png

   Note

   • We recommend turning off (setting to None) incident extraction for the Labels incident field. When an incident JSON is received from an integration, the JSON members are mapped to incident fields (based on the mapping configuration). Every member in the JSON that was not mapped to a field, will be written to the Labels field. If the Labels field extracts indicators, it can expose unmapped or unknown data to external sources. You should only map the relevant data to fields and set their extraction settings.
   • If you want to extract attachments, select the attachment field and then select File as the indicator type to extract. The File extracts a hash (usually SHA-256), which can be viewed in the War Room. You may want to disable indicator extraction for attachments to reduce external API usage and protect restricted data (the hash) from being sent.
6. Click Save.
7. (Optional) If you want to configure which scripts and commands the indicator type executes, go to Settings & Info → Settings → Object Setup → Indicators → Types and edit or Create an indicator type.

   Add scripts and reputation commands for the indicator type. When indicator extraction occurs, indicators are extracted as defined in an indicator type, and enriched using the commands and scripts associated with the indicator type. For example, the URL indicator is enriched using the !url command.

In this example, if an email is forwarded that potentially includes phishing, we want to extract at incident creation (inline) and upon a field change (out of band):

• Campaign Email Subject: Extract all indicators.
• Campaign Email Body: Extract all indicators.
• Email Delete Result: Extract email only.
• Email Delete Reason: Extract email only.

extract-fig-2-2.png

Set the indicator extraction mode for a playbook task

You can set the indicator extraction mode for specific playbook tasks.

1. Select the playbook where you want to add indicator extraction to a task, and click Edit.
2. In the playbook, click a task to open the Edit Task window.
3. Click the Advanced tab.
4. In the indicator extraction drop-down menu, select the mode you want to use.
5. Click OK.

Disable indicator extraction for scripts or integrations

This procedure describes how to disable indicator extraction for a specific script or an integration.

• To disable indicator extraction for a script, add the IgnoreAutoExtract entry with the value of true, when returning an entry.

  For example:

  entry = {
      'Type': entryTypes['note'],
      'Contents': {
      'Echo' : demisto.args()['echo']
          },
      'ContentsFormat': formats['json'],
      'ReadableContentsFormat': formats['markdown'],
      'HumanReadable': hr,
      'IgnoreAutoExtract' : True
     }

• To disable indicator extraction for an integration, add the 'IgnoreAutoExtract' entry with the value of true, when returning an entry.

  For example in the ServiceNow integration:

  entry = {
          'Type': entryTypes['note'],
          'Contents': result,
          'ContentsFormat': formats['json'],
          'ReadableContentsFormat': formats['markdown'],
          'HumanReadable': tableToMarkdown('ServiceNow ticket', hr, headers=headers, removeNull=True),
          'EntryContext': {
              'Ticket(val.ID===obj.ID)': context,
              'ServiceNow.Ticket(val.ID===obj.ID)': context
          },
          'IgnoreAutoExtract': True
      }
      entries.append(entry)
      return entries

For more information about command results in Python, see Python code conventions for CommandResults.

Troubleshoot indicator extraction

If indicators are not extracting, check whether the indicator mode is set to none. Even if you select the relevant incident fields and the indicators to extract, if the mode is set to none, indicators do not extract.

When creating new incident types, if you select Extract all indicators from all fields, all fields are extracted including custom fields. If you select Extract specific indicators by default, indicator extraction for new custom fields is set to none.