> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax/collect.md). # COLLECT {% hint style="info" %} ### License Requires a Data Collection add-on. {% endhint %} {% hint style="warning" %} ### Prerequisite Parsing Rules requires **View/Edit** RBAC permissions for **Data Management** (under **Configurations** → **Data Management**), which are the same permissions required for Dataset Management, Data Model Rules, and Event Forwarding. {% endhint %} A `COLLECT` section defines a rule that enables data reduction and data manipulation at the Broker VM to help avoid sending unnecessary data to the Cortex XDR server and reduces traffic, storage, and computing costs. In addition, the `COLLECT` section is used to manipulate, alter, and enrich the data before it’s passed to the Cortex XDR server. While this rule is optional to configure, once added, this rule runs before the `INGEST` section. {% hint style="info" %} ### Note The [CSV Collector applet](/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/generic-on-premise-data-collectors/broker-vm-data-collector-applets/activate-csv-collector.md) is not affected by the `COLLECT` rules applied to a Broker VM. {% endhint %} To avoid performance issues on the Broker VM, Cortex XDR does not permit all Parsing Rules to run on the Broker VM by default, but only the Parsing Rules that you designate. The Broker VM is directly affected by the `[COLLECT]` rules you create, so depending on the complexity of the rules more hardware resources on the Broker VM may be required. As a result, ensure that your Broker VM meets the following minimum hardware requirements to run `[COLLECT]` rules: * 8-core processor * 8GB RAM * 512GB disk * Plan for a max of 10K eps (events per second) per core. `COLLECT` syntax is derived from Cortex Query Language (XQL) with a few modifications as explained in [Parsing Rules file structure and syntax](/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax.md). In addition, `COLLECT` rules contain the following syntax add-ons: * `COLLECT` rules can have more than one XQLp statement, separated by a semicolon (`;`). Each statement creates a different data reduction and manipulation at the Broker VM for a different vendor and product. * While the XQL stages [alter](/xql-command-reference-guide/readme/stages/alter.md) and [fields](/xql-command-reference-guide/readme/stages/fields.md) are permitted in `COLLECT` rules for various vendors and products, you should avoid using them for supported vendors that can be used for Analytics as these stages can disrupt the operation of the Analytics Engine. For a list of these vendors, see the [Complete data source catalog](/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/complete-data-source-catalog.md). * Another new stage is available called `drop`. * `drop` takes a condition similar to the XQL `filter` stage (same syntax), but drops every log entry that passes that condition. One can think of it as a negative filter, so `drop ` is not equivalent to `filter not `. * `drop` can only appear last in a statement. No other XQLp syntax can follow. * `COLLECT` sections take parameters, where some are mandatory and others optional. ```programlisting [COLLECT:vendor=, product=, target_brokers = (), no_hit = ]; ``` Example 27. Here's an example of how to define the `COLLECT` section with a single `broker_ID`: ```programlisting [COLLECT:vendor="PANW", product="NGFW_CEF", target_brokers=(BROKER_ID), no_hit=drop] ``` Here's an example of how to define the `COLLECT` section with multiple `broker_ID`s: ```programlisting [COLLECT:vendor="PANW", product="NGFW_CEF", target_brokers=(BROKER_ID1, BROKER_ID2, BROKERID3), no_hit=drop] ``` The parameter descriptions are explained in the following table: | Parameter | Description | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `vendor` | The vendor that the specified `COLLECT` rule for data reduction and data manipulation at the Broker VM applies to (mandatory). | | `product` | The product that the specified `COLLECT` rule for data reduction and data manipulation at the Broker VM applies to (mandatory). | | `target_brokers` |

Specifies the list of Brokers to run the COLLECT rule for data reduction and data manipulation based on the vendor and product configured (mandatory). When target\_brokers=\*, the COLLECT rule applies to all the data collected by the Broker VM applets.

Note

The CSV Collector applet is not affected by the COLLECT rules applied to a Broker VM.

| | `no_hit` |

No-match strategy to use for the entire specified group of COLLECT rules (optional). The default is keep.

  • If no\_hit = drop, then in a scenario where none of the COLLECT rules in the group generates output for a given event, that event is discarded.
  • If no\_hit = keep, then in a scenario where none of the COLLECT rules in the group generates output for a given event, that event is passed to the Cortex XDR server.
| The following is an example of using a `COLLECT` rule to filter data for a specific vendor and product that will run before the `INGEST` section. Example ```programlisting [COLLECT:vendor="Apache", product="ApacheServer", target_brokers = (bvm1, bvm2, bvm3), no_hit = drop] alter source_log = json_extract_scalar(_raw_log, "$.source") | filter source_log = "WebApp-Logs" | fields source_log, _raw_log; [INGEST:vendor="Apache", product="ApacheServer", target_dataset = "dvwa_application_log"] alter log_timestamp = json_extract_scalar(_raw_log, "$.timestamp") | alter log_msg = json_extract_scalar(_raw_log, "$.msg") | alter log_remote_ip = json_extract_scalar(_raw_log, "$.Remote_IP") | alter scanned_ip = json_extract_scalar(_raw_log, "$.Scanned_IP") | fields log_msg ,log_remote_ip ,log_timestamp ,source_log ,scanned_ip , _raw_log; ``` A few more points to keep in mind when writing `COLLECT` rules: * There are no `COLLECT` rules by default, so all collected events are forwarded by the Broker VM to the Cortex XDR server.

Tip

To reduce the amount of data transmitted to Cortex XDR from the broker, use filters to drop logs. Yet, be aware that once the logs are modified using alter or fields stages, the Broker VM will convert the original log into a JSON format, which could increase the data size being sent from the broker to Cortex XDR.

* When `COLLECT` rules are defined, the designated Broker VMs check every collected event versus each rule. When there is a match for a given product or vendor, the Broker VM checks if it meets the filter criteria. * If it meets the criteria, the event is passed to the Cortex XDR server. * If it doesn’t meet the criteria, it depends on the `no_hit` parameter. -If `no_hit=drop`, then this `COLLECT` rule will not pass the event. Yet, the event still goes through other rules on this Broker VM. -If `no_hit=keep`, the event is passed to the Cortex XDR server, and goes through other rules on this Broker VM. * When the evaluated event, doesn’t match any product or vendor for a defined `COLLECT` rule, the event is passed to the Cortex XDR server. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/data-management/parsing-rules/parsing-rules-file-structure-and-syntax/collect.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.