> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax/ingest.md). # INGEST {% hint style="info" %} ### License Requires a Data Collection add-on. {% endhint %} {% hint style="warning" %} ### Prerequisite Parsing Rules requires **View/Edit** RBAC permissions for **Data Management** (under **Configurations** → **Data Management**), which are the same permissions required for Dataset Management, Data Model Rules, and Event Forwarding. {% endhint %} An `INGEST` section is used to define the resulting dataset. The `COLLECT`, `CONST`, and `RULE` sections are only add-ons, used to help organize the `INGEST` sections, and are optional to configure. Yet, a Parsing Rules file that contains no `INGEST` sections, generates no Parsing Rules. Therefore, the `INGEST` section is mandatory to configure. `INGEST` syntax is derived from Cortex Query Language (XQL) with a few modifications as explained in [Parsing Rules file structure and syntax](/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax.md). In addition, `INGEST` sections contain the following syntax add-ons: * `INGEST` sections can have more than one XQLp statement, separated by a semicolon (`;`). Each statement creates a different Parsing Rule. * The following XQL functions and stages are also supported in the `INGEST` section: * Functions: [arrayfilter](/xql-command-reference-guide/readme/functions/arrayfilter.md), [arraycreate](/xql-command-reference-guide/readme/functions/arraycreate.md), [arraymerge](/xql-command-reference-guide/readme/functions/arraymerge.md), [object\_create](/xql-command-reference-guide/readme/functions/object_create.md), [parse\_cef](/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax/ingest/parse_cef.md), [parse\_cisco](/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax/ingest/parse_cisco.md), and [parse\_json](/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax/ingest/parse_json.md). * Stages: [iploc](/xql-command-reference-guide/readme/stages/iploc.md) and [arrayexpand](/xql-command-reference-guide/readme/stages/arrayexpand.md). * [fields](/xql-command-reference-guide/readme/stages/fields.md): Using the `fields` stage in the `[INGEST]` section of the parsing rule explicitly controls the schema creation of the raw dataset. If you explicitly define only to ingest a few fields, then only these fields will be stored in Cortex XDR and available to query. Defining `fields` is a good way to only ingest clean data without including any corrupt data. * Another new stage is available called `drop`. * `drop` takes a condition similar to the XQL `filter` stage (same syntax), but drops every log entry that passes that condition. One can think of it as a negative filter, so `drop ` is not equivalent to `filter not `. * `drop` can only appear last in a statement. No other XQLp rules can follow. * `INGEST` sections take parameters, and not names as `RULE` sections use, where some are mandatory and others optional. ```programlisting [ingest:vendor=, product=, target_dataset=, no_hit=, ingestnull=] filter raw_log not contains "issue"; ``` The parameter descriptions are explained in the following table: | Parameter | Description | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `vendor` | The vendor that the specified Parsing Rules apply to (mandatory). | | `product` | The product that the specified Parsing Rules apply to (mandatory). | | `target_dataset` | The name of the dataset to insert every row with the results after applying any of the specified Parsing Rules (mandatory). | | `no_hit` |

No-match strategy to use for the entire specified group of rules (optional). The default is keep.

  • If no\_hit = drop, then in a scenario where none of the rules in the group generates output for a given log record, that record is discarded.
  • If no\_hit = keep, then in a scenario where none of the rules in the group generates output for a given log record, that record is kept in the \_raw\_log field. This record is inserted into the group's dataset once, but every column holds NULL except for \_raw\_log, which holds the original JSON log record.
| | `ingestnull` | Defines whether null value fields are ingested (optional). By default this is set to `true`, so you only need to set this parameter when you want to overwrite the default definition. | Each statement represents a different Parsing Rule in the same group as depicted in the following example: Example ```programlisting [CONST] DEVICE_NAME = "ngfw"; [rule:use_two_rules] filter severity = "medium" | call basic_rule | call use_xql_and_another_rule; [rule:basic_rule] fields log_type, severity | filter log_type="eal" and severity="HIGH" and type="something"; [rule:use_xql_and_another_rule]call multiline_statement | filter severity = "medium"; [rule:multiline_statement] alter url = json_extract(_raw_log, "$.url") | join type = inner conflict_strategy = both (dataset=my_lookup) as inn url=inn.url |filter severity = "medium"; [ingest:vendor=panw, product=ngfw, target_dataset=panw_ngfw_ds, no_hit=drop] filter log_type="traffic" | alter url = json_extract(_raw_log, "$.url"); call use_two_rules | join type = inner conflict_strategy = both (dataset=my_lookup) as inn severity=inn.severity | fields severity, log_type | drop device_name = $DEVICE_NAME; ``` This generates 1 group of 2 Parsing Rules for panw/ngfw, where all the ingested data into `panw_ngfw_ds` dataset. The following represents the syntax for the rules: ```programlisting Rule #1: filter log_type="traffic" | alter url = json_extract(_raw_log, "$.url"); Rule #2: filter severity = "medium" | fields log_type, severity | filter log_type="eal" and severity="HIGH" and type="something" | alter url = json_extract(_raw_log, "$.url") | join type = inner conflict_strategy = both (dataset=my_lookup) as inn url=inn.url | filter severity = "medium" | filter severity = "medium" | join type = inner conflict_strategy = both (dataset=my_lookup) as inn severity=inn.severity | fields severity, log_type | drop device_name = $DEVICE_NAME ``` A few more points to keep in mind when writing `INGEST` sections: * `INGEST` parameter names are not case-sensitive. Therefore, `vendor=PANW` and `vendor=panw` are the same. * Since section order is unimportant, you do not have to declare a `RULE` or a `CONST` before using it in an `INGEST` section. * You can have multiple `INGEST` sections with the same `vendor`, `product`, `dataset` , and `no_hit` values. Yet, this can lead to unexpected results. Consider the following example: Example ```programlisting [ingest:vendor=panw, product=ngfw, tartget_dataset=panw_ngfw_ds, no_hit=keep] filter raw_log not contains "issue"; [ingest:vendor=panw, product=ngfw, target_dataset=panw_ngfw_ds, no_hit=keep] filter device_type not contains "agent"; ``` Let `lw` be a log row. If `lw.raw_log` doesn't contain an `issue` and `lw.device_type` doesn't contain an `agent`, then `lw` is inserted twice into the `pan_ngfw_ds` dataset as every section is standalone. * To eliminate these kind of errors and misunderstandings, it is highly advised to group all rules having the same `vendor`, `product`, `dataset` , and `no_hit` values in a single `INGEST` section. * Logs that were discarded by a `drop` stage are considered ingested with a no-match policy. This means they are not kept even if `no_hit = keep`. * Keep in mind that all rules inside a group get evaluated independently. This is in contrast to firewall-like rules, which stop evaluating the first rule that is able to make a decision. Therefore, without proper filtering, it is possible to ingest the same log more than once. * You can override the default raw dataset in `INGEST` sections. For more information, see [Parsing Rules Raw Dataset](/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-raw-dataset.md). * Cortex XDR supports configuring case sensitivity in Parsing Rules only within the `INGEST` section using the following configuration stage: ```programlisting config case_sensitive = true | false ``` * You can add a single tag or list of tags to the ingested data as part of the ingestion flow that you can easily query. You can add tags as part of the `INGEST` section or use both the `INGEST` and `RULE` sections.

Note

You can't add tags to parsing rules using the Next-Generation Firewall (NGFW) datasets that are in the format panw_ngfw_<text>_raw

The following are examples of each: * `INGEST` section: Example Adding a single tag: ```programlisting [INGEST:vendor="MSFT", product="Azure AD Audit", target_dataset="msft_ad_audit_tagging", no_hit=drop, ingestnull = false ] tag add "New Event" ``` Adding a list of tags: ```programlisting [INGEST:vendor="MSFT", product="Azure AD Audit", target_dataset="msft_ad_audit_tagging", no_hit=drop, ingestnull = false ] tag add "New Event1", "New Event2", "New Event3" ``` * `INGEST` and `RULE` sections: Example Adding a single tag: ```programlisting [INGEST:vendor="Check Point", product="Anti Malware", target_dataset="malware_test", no_hit= drop , ingestnull = true ] alter xx = call new_tag_rule; ``` ```programlisting [RULE:new_tag_rule] tag add "test"; ``` Adding a list of tags: ```programlisting [INGEST:vendor="Check Point", product="Anti Malware", target_dataset="malware_test", no_hit= drop , ingestnull = true ] alter xx = call new_tag_rule; ``` ```programlisting [RULE:new_tag_rule] tag add "test1", "test2", "test3"; ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax/ingest.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.