> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/dataset-management/what-are-datasets.md). # What are datasets? {% hint style="info" %} ### Note This feature requires a Cortex XDR Pro per GB license. {% endhint %} Cortex XDR runs every Cortex Query Language (XQL) query against a dataset. A dataset is a collection of column:value sets. If you do not specify a dataset in your query, Cortex XDR runs the query against the default datasets configured, which is by default `xdr_data`. The `xdr_data` dataset contains all of the endpoint and network data that Cortex XDR collects. You can always change the default datasets using the set to default option. You can also upload datasets as a CSV, TSV, or JSON file that contains the data you are interested in querying. These uploaded datasets are called lookup datasets. It's also possible to create dataset views, which provide a virtual representation of data from one or more datasets, based on the Cortex Query Language (XQL) query defined. Dataset views enhance data efficiency and security. For example, by segregating data for specific user needs or access privileges through the Role-based access control (RBAC) settings. For more information, see [Dataset views](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/dataset-management.md). To query other datasets, you have the following options: * Set a dataset as default, which enables you to query the datasets without specifying them in the query. * Name a specific dataset at the beginning of your query with the `dataset` stage command.
Dataset types The type of dataset is based on the method used to upload the data. The possible types include: * **Correlation**: A dataset containing data saved from a correlation rule. * **Lookup**: A dataset containing key-value pairs that can be used as a reference to correlate to events. For example, a user list with corresponding access privileges. You can import or create a lookup dataset, and then reference the values for a certain key, run queries and take action. For more information, see [Lookup datasets](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/dataset-management/lookup-datasets.md). * **Raw**: Every dataset where PANW data is ingested out-of-the-box or third-party data is ingested using a configured dedicated collector. The schema for the raw dataset is automatically generated based on the log data collected by Cortex XDR and the data format sent, such as JSON, CEF, and LEEF. * **Snapshot**: A dataset that contains only the last successful snapshot of the data, such as Workday or ServiceNow CMDB tables. * **System**: Cortex XDR datasets that are created out-of-the-box. * **User**: If saved by a query using the **`target`** command, the **Type** can be either **User** or **Lookup**.
Dataset parsers To ensure accurate visibility and threat detection within Cortex XDR, the platform utilizes a variety of specialized dataset parsers. The selection and effectiveness of these parsers are directly dependent on the specific data sources ingested, encompassing both the source data (the original log format generated by the device) and the integration configuration. By identifying whether logs arrive as structured JSON, standardized security formats like CEF and LEEF, or unstructured raw text, Cortex XDR can properly parse the information for analysis. **Supported parser types** * CEF (Common Event Format): A parser for logs formatted in the ArcSight CEF standard. CEF logs contain a pipe-delimited header with vendor, product, version, event class, name, and severity fields, followed by key-value pair extensions. Commonly used by vendors like Check Point, Fortinet, and Zscaler. * Cisco ASA: A parser for specific Cisco Adaptive Security Appliance (ASA) syslog messages. Supports logs like connection-related messages (Built/Teardown) and AnyConnect VPN events. Only a subset of Cisco ASA message types is supported. * Corelight: A parser for network traffic logs generated by Corelight sensors (based on Zeek/Bro). Processes structured JSON logs containing network connection metadata such as connection records, DNS queries, and HTTP transactions. * Filebeat: A parser for logs collected and forwarded using Elastic Filebeat agents. Extracts the log payload from the Filebeat JSON envelope, handling metadata fields such as timestamps, agent information, and host details. * JSON: A general-purpose parser for logs sent in standard JSON format. This is the most common format for third-party data sources that send structured data, including cloud services, SaaS applications, and API-based integrations. * LEEF (Log Event Extended Format): A parser for the IBM QRadar LEEF standard. LEEF logs contain a tab-delimited or custom-delimited header with version, vendor, product, product version, and event ID fields, followed by key-value pair attributes. Typically used by IBM security products and QRadar-integrated vendors.

Note

CEF and LEEF logs are often wrapped in a Syslog envelope (RFC 3164/5424). Cortex XDR automatically detects and strips the Syslog header to extract the reporting device IP and hostname before parsing the inner CEEF or LEEF payload.

* Raw Text: A parser for unstructured, plain-text log data that does not conform to any specific structured format. The raw text is ingested as-is and can be processed using parsing rules. This is also the default parser used when the log format cannot be identified or is not explicitly defined. * WEC (Windows Event Collection): A parser for Windows Event logs collected via Windows Event Forwarding (WEF/WEC). Parses XML-formatted Windows Event logs and converts them into a structured format. Supports Windows Security, System, and Application event logs. * Windows DNS Debug: A parser for Microsoft Windows DNS Server debug log files. Extracts DNS query and response details including query type, remote IP, protocol, response code, and question name from the Windows DNS debug log format. * Winlogbeat: A parser for Windows Event logs collected using Elastic Winlogbeat agents. Similar to the WEC parser but handles the Winlogbeat-specific JSON envelope format, extracting Windows Event data along with associated metadata.
Datasets in XQL {% hint style="info" %} ### Important By default, forensic datasets are not included in XQL query results, unless the dataset query is explicitly defined to use a forensic dataset. {% endhint %} Cortex Query Language (XQL) supports using different languages for dataset and field names. In addition, when setting up your XQL query, it is important to keep in mind the following: * The dataset formats supported are dependent on the data retention offerings available in Cortex XDR according to whether you want to query hot storage or cold storage. * Hot Storage queries are performed on a dataset using the format **`dataset = `**. This is the default option. ```programlisting dataset = xdr_data ``` * Cold Storage queries are performed using the format **`cold_dataset = `**. ```programlisting cold_dataset = xdr_data ``` * Dataset refresh times: While most out-of-the-box system datasets are ingested in near real-time, the following datasets have specific refresh schedules: * `endpoints`: Refreshed every hour. * `pan_dss_raw`: Refreshed daily. * Forensics datasets: Data collection behavior depends on your **Agent Settings** profile. * Default: Data is collected as a one-time snapshot and does not update. * Scheduled: If you specify a collection interval, the value represents the number of hours between updates, such as an interval of 24 equals once per day. * Minimum: The shortest allowable interval is 12 hours. * Query against a dataset by selecting it with the `dataset` command when you create an XQL query. For more information, see [Create XQL query](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/investigate-and-respond-to-incidents/build-xql-queries/how-to-build-xql-queries/create-xql-query.md). * After you query runs, you can always save your query results as a dataset. You can use the [target](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/cortex-xdr-xql/stages/target.md) stage command to save query results as a dataset. * Schema changes to datasets may not be reflected in the autocomplete suggestions and definitions as you type in real time the XQL query and can appear with a slight delay.
Managing datasets and dataset views You can manage your datasets and dataset views in Cortex XDR from the **Settings** → **Configurations** → **Data Management** → **Dataset Management** page. Below are some of the main tasks available for all dataset types by right-clicking a particular dataset or dataset view listed in either the **Datasets** or **Dataset Views** table. Only tasks that need further explanation are explained below. Datasets and dataset views can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset or dataset view, you wouldn't be able to delete the dataset or dataset view until you removed the dataset view from the XQL query of the Correlation Rule. {% hint style="info" %} ### Note For more information on tasks specific to lookup datasets, see [Lookup datasets](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/dataset-management/lookup-datasets.md). {% endhint %} **View Schema** Select **View Schema** to view the schema information for every field found in the dataset or dataset view result set in the **Schema** tab after running the query in XQL. Each system field in the schema is written with an underscore (`_`) before the name of the field in the **FIELD NAME** column in the table. {% hint style="info" %} ### Note Schema changes to datasets may not be reflected in the autocomplete suggestions and definitions as you type in real time the XQL query and can appear with a slight delay. {% endhint %} **Set as default** Select **Set as default** to query the dataset without having to specify it in your queries in XQL by typing `dataset = `. Once configured, the **DEFAULT QUERY TARGET** column entry for this dataset is set to **Yes** in the **Datasets** table. By default, this option is not available when right-clicking the `xdr_data` dataset as this dataset is the only dataset configured as the **DEFAULT QUERY TARGET** as it contains all of the endpoint and network data that Cortex XDR collects. Once you **Set as default** another dataset, you can always remove it by right-clicking the dataset and selecting **Remove from defaults**. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XDR queries the default datasets using a `join`. This option is only relevant for datasets. **Copy text to clipboard** Select **Copy text to clipboard** to copy the name of the dataset or dataset view to your clipboard.
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/dataset-management/what-are-datasets.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.