> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xsiam/configure-cortex-xsiam/data-management/configure-data-pipelines.md).

# Configure Data Pipelines

Configure Data Pipelines  in Cortex XSIAM to control ingestion costs by deciding what data enters Cortex XSIAM and where it lands.

{% hint style="info" %}

#### Licenses

The Data Pipeline is available as an optional add-on license that allows organizations to optimize data collection and maximize GB savings.
{% endhint %}

{% hint style="warning" %}

#### Prerequisite

* **Permissions**: Requires **View/Edit** RBAC permissions for **Data Management** (under **Configurations** → **Data Management**).
* **Data source or connector**: Must have a data source or connector already configured on the **Data sources & Integrations** page that is currently collecting data. For more information, see [Cortex XSIAM Data Sources and Connectors](https://cortex-docs.paloaltonetworks.com/cortex-xsiam/configure-cortex-xsiam/cortex-xsiam-data-sources.md).
  {% endhint %}

The Data Pipeline gives you a visual canvas to build pipelines that filter, parse, model, and ingest events across Cortex XSIAM tiers. This capability provides a powerful, visual, and granular mechanism to control ingestion costs and optimize storage.

A Pipeline defines how data is delivered into the system and its destination. Each pipeline must include a data source or connector, a single vendor-product, parsing rule, and a destination. Filtering and data-modeling rules (read-only) are optional transformation stages, applied in order when defined.

### Key terms

The following key terms are essential to understanding the Data Pipeline architecture:

* **Pipeline**: A reusable, named processing program composed of an ordered chain of transformation rules (**Filtering** → **Parsing** → **Modeling**), which has one source and can have multiple destinations.
* **Data source or connector**: A Vendor-Product pair streaming logs or events into the system, such as `{cisco-asa}`. A data source or connector can contain one or many instances of the same Vendor-Product pair.
* **Vendor-Product**: The specific Vendor-Product pair that constitutes the data source or connector. Each pipeline is built from exactly one Vendor-Product.
* **Destination**: The target storage where data resides after processing. This includes Cortex XSIAM (Analytics or Data Lake tiers).
* **Filtering Rule**: Criteria for excluding specific rows and columns before ingestion to reduce overall data volume and costs, including statistical sampling.
* **Parsing Rule**: The logic layer that structures raw incoming data into a standardized format and defines its target dataset. Any changes made to rule logic within the Data Pipeline canvas are automatically synchronized with the existing **Parsing Rules** page.

### Data Pipeline architecture and capabilities

Cortex XSIAM utilizes a foundational pipeline model to ensure data is processed efficiently before storage:

#### Visual pipeline canvas

Cortex XSIAM provides an intuitive, step-by-step visual canvas to construct and manage your pipelines. The data pipeline canvas remains empty until you create a data pipeline.

The visual canvas interface represents "Streams" flowing from left to right:

* **Visual node types**:
  * **Source containers**: Expandable nodes, such as AWS and Okta, displaying connection points for individual Vendor-Products.
  * **Rule nodes**: Pill-shaped nodes representing Filter, Parsing, and Data Model rules.
  * **Destination containers**: Terminal connection points grouped by provider (Cortex XSIAM).
* **Interactive controls**: Navigate the canvas using standard zoom and fit-to-screen controls. The default **Graph View** provides a comprehensive map of your data streams.
* **Inline configuration**: Clicking a rule node expands into an inline configuration panel, allowing you to edit logic without leaving the canvas.

Through this canvas, you can:

1. **Add additional pipelines for the Vendor-Products listed**: On a source container for the Vendor-Products, you can add another data pipeline using the pipeline wizard.
2. **Add/edit/delete filter rules or parsing rules**: To your rule nodes.
3. **Configure processing stages**: Drop in processors for filtering, sampling, or transformation.
4. **Define outputs**: Select the target storage tier (Analytics or Data Lake).

#### Pipeline model

The processing sequence is strictly ordered as: **Source** → **Filter** → **Parse** → **Model** → **Destination**.

* **Multiple pipelines**: A single Vendor-Product can feed multiple pipelines, allowing you to split a data source or connector into distinct filtered streams for different datasets.
* **Granular destinations**: Choose your XSIAM tier (Analytics or Data Lake) independently for every dataset ingested to Cortex XSIAM.

#### Filtering an incoming source

Filtering an incoming source runs first to reduce overall ingestion volume before the data is ingested into Cortex XSIAM. Supported functions include:

* **Drop**: Filters out events matching specific criteria.
* **Fields removal**: Filters out specific columns (except for mandatory system fields).
* **Sampling**: Ingests a representative percentage, such as 1 of every 10 events.
* **Syntax support**: Supports limited Cortex Query Language (XQL) expressions and regex-based lookups related to filtering. For a list of the functions and stages supported for filtering an incoming source, see [Supported XQL syntax for ingestion filters](#supported-xql-syntax-for-ingestion-filters).

#### Out-of-the-box analytics-detector warnings

Configuring a filter may remove data required by built-in detectors for security alerts.

* **Warning trigger**: A non-blocking warning appears if a filter impacts required fields for out-of-the-box detectors.

### Comparison with other tiers

The following table highlights how the Data Pipeline relates to other data management options in Cortex XSIAM:

| Point of Comparison    | Analytics Tier                             | Cortex Data Lake Tier                       | Data Pipelines                                                  |
| ---------------------- | ------------------------------------------ | ------------------------------------------- | --------------------------------------------------------------- |
| **Primary Goal**       | Real-time Detection and Response           | Cost-effective Investigation and Compliance | Filtering and sampling data before ingestion to optimize costs. |
| **Data Storage**       | Ingested and stored in Cortex XSIAM        | Ingested and stored in Cortex XSIAM         | None (filters data)                                             |
| **AI/ML & Detection**  | Full AI/ML support and real-time detection | None                                        | Only when ingested by Analytics tier                            |
| **Configuration Tool** | Parsing Rules                              | Parsing Rules                               | Visual Pipeline Canvas                                          |
| **Licensing**          | Mandatory minimum (100 GB/day)             | Optional add-on (minimum 50 GB/day)         | Optional add-on (based on GB savings)                           |

### Licensing and usage monitoring

* GB savings metrics: In-product licensing and metrics focus on the volume of data dropped and the resulting GB savings achieved by filtering and routing before ingestion.
* Technical enforcement: Exceeding the licensed quota does not throttle, block, or disable any processing or platform capability. Overages are surfaced in metrics and handled commercially.
* Visibility: The Licenses page tracks pipeline volume alongside Analytics and Data Lake consumption. Detailed GB savings (data dropped/filtered) are reported on the Ingestion Dashboard.

### How to configure a Data Pipeline

Follow these steps to configure a data pipeline for your target dataset:

1. Navigate to the Data Pipeline editor.\
   Select **Settings** → **Data Pipelines**.
2. Open the wizard.\
   Click **Create Pipeline** to open the 4-step configuration wizard.
   1. Step 1: Define the Pipeline details. Enter the general information and data source details.

      * **General**:
        * **Pipeline name**: Enter a unique name for the pipeline.
        * **Details**: Enter a description to help identify the pipeline's purpose.
      * **Data source details**:
        * **Vendor**: Select or start typing the name of the log vendor. You can only select vendors for data sources and connectors that are already configured and actively collecting data.
        * **Product**: Select the specific product associated with the vendor.

      Click **Next**.
   2. Step 2: Define the Filter rule.\
      Configure the filtering and sampling ratio to determine which data is ingested. You can choose to optimize data using **Rule Details**, a **Data sampling ratio**, or both.

      * **Select a filter mode**:
        * **Use filter rule**: Select this to define specific filtering logic and/or a sampling ratio.
        * **Do not use filter rule**: Select this if you do not want to optimize the data stream at this stage.
      * **Rule Details**: (If using a filter rule) Use the editor to define your XQL-based filtering logic.
        * **Simulate**: Click to test your rule logic against raw logs before proceeding. The editor includes the following sections:

          | Section      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
          | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
          | Rule content | The current **User defined** rules on the left side of the window.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
          | XQL Samples  | A table of the existing Cortex Query Language (XQL) raw data samples on the right side of the window, which contain sample logs listing the **Vendor**, **Product**, **Raw Log**, and **Sample Time**. For each **Vendor** and **Product**, up to 5 different samples are available to choose from. From this list, you can select the logs used to simulate the rule.                                                                                                                                                                                                                                                                                            |
          | Logs Output  | <p>Displays the Vendor and Product associated with this dataset, and in a table format the following columns per dataset at the bottom of the window:</p><ul><li><strong>Dataset</strong>: Displays the applicable dataset name.</li><li><strong>Output Logs</strong>: Displays the output logs that are available based on your <strong>Rule content</strong> and <strong>XQL Samples</strong> selected after simulating the results.</li><li><strong>Input Logs</strong>: Displays the relevant input log with a right-click pivot to <strong>Show diff</strong> between the <strong>Output Logs</strong> and <strong>Input Logs</strong>.</li></ul><p><br></p> |

          <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><h4>Important</h4><p>If you drop specific fields (columns) in your filter rule, the data schema is lost. As a result, Cortex XSIAM cannot apply built-in analytics to the ingested data.</p></div>
        * **Data sampling ratio**: Set the percentage of available data to process (1-99). Higher percentages allow more data to be analyzed. This option can be used with or without a custom filter rule defined above.

          <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><h4>Note</h4><p>If you configure both <strong>Rule Details</strong> and a <strong>Data sampling ratio</strong>, Cortex XSIAM processes the filtering logic first. It then applies the data sampling ratio.</p></div>

      Click **Next**.
   3. Step 3: Configure data routing.\
      Structure your data and send it to the relevant storage destinations.

      * **Rules configuration**: Structure your data by selecting or adding **Parsing rules**. You can choose from **Default rules**, **User-defined rules**, or rules from **Marketplace**.
      * **Default rules**: This tab lists existing parsing rules for your selected vendor and product. Since data pipelines are built using active data sources or connectors, a default parsing rule is always available, automatically created based on the first event received by Cortex XSIAM.
      * **User-defined**: In this tab, you can add or edit custom parsing rules.
        * **Add rule**: Click **+ Add rule** to create a new parsing rule for the stream.
        * **Simulate**: Just as in the filter rule step, you can click Simulate to test your custom rule logic against raw logs.

          <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><h4>Note</h4><p>When using the simulation tool, the XQL samples in the data pipeline are automatically filtered by your selected vendor and product.</p></div>
      * **Select a rule**: Click on a specific rule in the list to view its configuration details in the side panel.
      * **Destinations**: This area displays the target storage tier (Analytics or Data Lake) for the selected parsing rule. This section is read-only; the destination is determined automatically based on how the parsing rule is configured.
      * **Manage rules list**: When you are finished configuring a parsing rule, click Done. This takes you to the **Rule configuration** overview list.\
        From this list, you can:
        * View the parsing rule ID, such as **Parsing rule 212**.
        * Add as many parsing rules as needed by clicking **+ Add rule**.
        * **Delete** or **Edit** existing parsing rules using the icons on the rule row.

      Click **Next**.
   4. Step 4: Summary.\
      Review your configuration to ensure the pipeline details, filter rules, and parsing rules are correct. If everything looks accurate, click **Create pipeline**.\
      The configuration is applied immediately to incoming data and the pipeline is added to the **Data Pipelines** visual canvas. For more information on the visual interface, see [Data Pipeline architecture and capabilities](#data-pipeline-architecture-and-capabilities).

### Managing existing pipelines

Once created, you can monitor and manage your pipelines directly from the Data Pipelines visual canvas page.

From this page, you can:

* **Create new pipelines**: Click **Create pipeline** at the top right of the canvas to open the configuration wizard and build new pipelines as needed.
* **Isolate a pipeline**: Since the visual canvas can list many pipelines, use the **Search** box at the top of the page to find a specific pipeline by name.
* **Expand to expose rule nodes**: Click the **Expand pipelines** icon (two arrows) on a pipeline to expand it. This exposes the nodes for your configured rules:
  * **Filter rule**: Labeled **Filter rule**.
  * **Parsing rule**: Labeled with names starting with `pr-<name>`.
  * **Data Model rule**: If a data model rule exists for the vendor-product, it is displayed as a node between the parsing rule and the destination.

    <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><h4>Note</h4><p>Data Model rules are read-only on the canvas. To modify a data model rule, you must navigate to the dedicated <strong>Data Model Rules</strong> page.</p></div>
* **Modify pipeline logic**:
  * **Edit existing logic**: To modify an existing pipeline, click the corresponding **Rule Node** (Filter or Parsing) on the canvas. This opens an **inline configuration** **panel** where you can update the logic without leaving the canvas.
  * **Modify or swap parsing rules**:
    * **Swap rules**: In the expanded view of a parsing rule, click the **Edit** icon to open the parsing rule editing screen and select a different rule.
    * **Edit User-Defined rule**: To edit the logic of a User-Defined parsing rule, hover over the rule and click the **ellipse** icon to open the text editor.
* **Add a new filter rule**: Hover on the data pipeline and click the **+ Add** icon, then select the **Add filter** icon. The **Filter rule** screen opens where you can configure the filtering and sampling ratio to determine which data is ingested.
* **Add a new parsing rule**: Hover on the data pipeline and click the **+ Add** icon. Select the **Add parsing rule** icon. A new line is added to the pipeline representing this rule; hover on this line, click the **+ Add** icon, and then select the **Add parsing info** icon to open the configuration screen for the new rule.
* **More actions**: Hover on the data pipeline and click the **+ Add** icon, then select the **ellipse** icon to open more actions. From here, you can **Edit details** of the pipeline (Pipeline name and Details) or **Delete pipeline**.
* **Minimize pipeline**: Click the minimize icon on an expanded pipeline to collapse the view. The configured rule nodes will no longer be displayed, leaving only the pipeline node visible.

## Supported XQL syntax for ingestion filters

When creating filters for an incoming source, you can use a specific subset of Cortex Query Language (XQL) stages and functions to reduce overall ingestion volume before the data is ingested into Cortex XSIAM.

{% hint style="info" %}

#### Note

For more information on any of these stages and functions, see [Cortex XQL Command Reference](https://cortex-docs.paloaltonetworks.com/xql-command-reference-guide).
{% endhint %}

#### Supported stages

The following XQL stages are supported for use in ingestion filters:

* filter
* switch
* fields

#### Supported functions

The following functions are supported within ingestion filter stages:

* add
* subtract
* multiply
* divide
* ceil
* floor
* pow
* round
* concat
* format\_string
* len
* lowercase
* uppercase
* ltrim
* rtrim
* trim
* split
* string\_count
* replace
* wildcard\_match
* regextract
* replex
* if
* coalesce
* to\_boolean
* to\_float
* to\_integer
* to\_number
* to\_string
* to\_timestamp
* to\_epoch
* to\_json\_string
* current\_time
* extract\_time
* format\_timestamp
* parse\_timestamp
* parse\_epoch
* timestamp\_diff
* timestamp\_seconds
* incidr
* incidr6
* incidrlist
* ip\_to\_int
* safe\_ip\_to\_int
* int\_to\_ip
* is\_ipv4
* is\_ipv4\_public
* is\_known\_private\_ipv4
* is\_ipv6
* is\_ipv6\_public
* is\_known\_private\_ipv6
* json\_extract
* json\_extract\_array
* json\_extract\_scalar
* json\_extract\_scalar\_array
* object\_create
* object\_merge
* md5
* sha1
* sha256
* sha512
* convert\_to\_base\_64
* convert\_from\_base\_64


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://cortex-docs.paloaltonetworks.com/cortex-xsiam/configure-cortex-xsiam/data-management/configure-data-pipelines.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
