Dataset management

Learn more about managing your datasets and understanding your overall data storage, period-based retention.

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

The Dataset Management page enables you to manage your datasets and understand your overall data storage duration for different retention periods and datasets based on your hot and cold storage licenses, and retention add-ons that extend your storage. You can view details about your Cortex XDR licenses and retention add-ons by selecting Settings → Cortex XDR License. For more information on license retention and the defaults provided per license, see Data retention in Cortex XDR.

Important

Cortex XDR enforces retention on all log-type datasets excluding Host Inventory, Vulnerability Assessment, Metrics, and Users.

Hot and cold storage

Your current hot and cold storage licenses, including the default license retention and any additional retention add-ons to extend storage, are listed within the Hot Storage License and Cold Storage License sections of the Dataset Management page. Whenever you extend your license retention, depending on your requirements and license add-ons for both hot storage and cold storage, the add-ons are listed.

Note

Cold storage, in addition to a cold storage license, requires compute units (CU) to run cold storage queries. For more information on CU, see Manage compute units.

For information on the CU add-on license, see Cortex XDR license plan.

Additional hot storage

You can expand your license retention to include flexible Hot Storage based retention to help accommodate varying storage requirements for different retention periods and datasets. This add-on license is available to purchase based on your storage requirements for a minimum of 1,000 GB. If this license is purchased, an Additional Storage subheading in the Hot Storage License section is displayed on the Dataset Management page with a bar indicating how much of the storage is used.

Note

Only datasets that are already handled as part of the GB license are supported for this license. In addition, the retention configuration is only available in Cortex XDR, as opposed to the public APIs.

Edit the retention plan

On any dataset configured to use Additional Hot Storage, you can edit the retention period. This enables you to view the current retention details and configure the retention. This includes setting the amount of flexible hot storage-based retention designated for a dataset and the priority for the dataset's hot storage.

Datasets table

For each dataset listed in the table, the following information is available:

Note

• Certain fields are exposed and hidden by default. An asterisk (*) is beside every field that is exposed by default.
• Datasets include dataset permission enforcements in the Cortex Query Language(XQL), Query Center, and XQL Widgets. For example, to view or access any of the endpoints and host_inventory datasets, you need role-based access control (RBAC) permissions to the Endpoint Administration and Host Inventory views. Managed Security Services Providers (MSSP) administration permissions are not enforced on child tenants, but only on the MSSP tenant.

Field	Description
*TYPE	Displays the type of dataset based on the method used to upload the data. The possible values include: Correlation, Lookup, Raw, Snapshot, System, and User. For more information on each dataset type, see What are datasets?.
*LOG UPDATE TYPE	Event logs are updated either continuously (Logs) or the current state is updated periodically (State) as detailed in the Last Updated column.
*LAST UPDATED	Last time the data in the dataset logs were updated. Important This column is updated once a day. Therefore, if the dataset was created or updated by the target or lookup flows, it's possible that the Last Updated value is a day behind when the queries or reports were run as it was before this column was updated.
*ADDITIONAL STORAGE	Amount of flexible hot storage-based retention designated for this dataset in months, where a month is calculated as 31 days.
*TOTAL DAYS STORED	Actual number of days that the data is stored in the Cortex XDR tenant, which is comprised of the HOT RANGE + the COLD RANGE.
*HOT RANGE	Details the exact period of the Hot Storage from the start date to the end date.
*COLD RANGE	Details the exact period of the Cold Storage from the start date to the end date.
*TOTAL SIZE STORED	Actual size of the data that is stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage. For the xdr_data dataset, where the first 31 days of storage are included with your license, the first 31 days are not included in the TOTAL SIZE STORED number.
*ADDITIONAL SIZE STORED	Actual size of the additional flexible hot storage data that is stored in the Cortex XDR tenant in GB. This number is dependent on the events stored in the hot storage.
*AVERAGE DAILY SIZE	Average daily amount stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage.
*HOT STORAGE PRIORITY	Indicates the priority set for the dataset's hot storage as either Low, Medium, or High.
*TOTAL EVENTS	Number of total events/logs that are stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage.
*AVERAGE EVENT SIZE	Average size of a single event in the dataset (TOTAL SIZE STORED divided by the TOTAL EVENTS). This number is dependent on the events stored in the hot storage.
*TTL	For lookup datasets, displays the value of the time to live (TTL) configured for when lookup entries expire and are removed automatically from the dataset. The possible values are: Forever: Lookup entries never expire (default). Custom: Lookup entries expire according to a set number of days, hours, and minutes. The maximum number of days is 99999. For more information, see Set time to live for lookup datasets.
DEFAULT QUERY TARGET	Details whether the dataset is configured to use as your default query target in XQL Search, so when you write your queries you do not need to define a dataset. By default, only the xdr_data dataset is configured as the DEFAULT QUERY TARGET and this field is set to Yes. All other datasets have this field set to No. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XDR queries the default datasets using a join.
TOTAL HOT RETENTION	Total hot storage retention configured for the dataset in months, where a month is calculated as 31 days.
TOTAL COLD RETENTION	Total cold storage retention configured for the dataset in months, where a month is calculated as 31 days.

Dataset views

Cortex XDR supports creating dataset views in the Dataset Management page to enhance data efficiency and security. Dataset views provide a virtual representation of data from one or more datasets, based on the Cortex Query Language (XQL) query defined, and provide multiple benefits, such as joining datasets into logical subsets through defined queries, manipulating data without altering underlying datasets, and segregating data for specific user needs or access privileges through the Role-based access control (RBAC) settings.

Once a dataset view is created, you can edit or delete the dataset view by right-clicking the dataset view in the Dataset Views table. A dataset view can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset view, you wouldn't be able to delete the dataset view until you removed the dataset view from the XQL query of the Correlation Rule.

Cortex XDR logs entries for events related to creating, editing, and deleting datasets or dataset views. These monitored activities are available to view in the datasets and dataset views audit logs in the Management Audit Logs. For more information, see Monitor datasets and dataset views activity.

What are datasets?

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

Cortex XDR runs every Cortex Query Language (XQL) query against a dataset. A dataset is a collection of column:value sets. If you do not specify a dataset in your query, Cortex XDR runs the query against the default datasets configured, which is by default xdr_data. The xdr_data dataset contains all of the endpoint and network data that Cortex XDR collects. You can always change the default datasets using the set to default option. You can also upload datasets as a CSV, TSV, or JSON file that contains the data you are interested in querying. These uploaded datasets are called lookup datasets.

It's also possible to create dataset views, which provide a virtual representation of data from one or more datasets, based on the Cortex Query Language (XQL) query defined. Dataset views enhance data efficiency and security. For example, by segregating data for specific user needs or access privileges through the Role-based access control (RBAC) settings.

To query other datasets, you have the following options:

• Set a dataset as default, which enables you to query the datasets without specifying them in the query.
• Name a specific dataset at the beginning of your query with the dataset stage command.

Dataset types

The type of dataset is based on the method used to upload the data. The possible types include:

• Correlation: A dataset containing data saved from a correlation rule.
• Lookup: A dataset containing key-value pairs that can be used as a reference to correlate to events. For example, a user list with corresponding access privileges. You can import or create a lookup dataset, and then reference the values for a certain key, run queries and take action. For more information, see Lookup datasets.
• Raw: Every dataset where PANW data is ingested out-of-the-box or third-party data is ingested using a configured dedicated collector. The schema for the raw dataset is automatically generated based on the log data collected by Cortex XDR and the data format sent, such as JSON, CEF, and LEEF.
• Snapshot: A dataset that contains only the last successful snapshot of the data, such as Workday or ServiceNow CMDB tables.
• System: Cortex XDR datasets that are created out-of-the-box.
• User: If saved by a query using the target command, the Type can be either User or Lookup.

Dataset parsers

To ensure accurate visibility and threat detection within Cortex XDR, the platform utilizes a variety of specialized dataset parsers. The selection and effectiveness of these parsers are directly dependent on the specific data sources ingested, encompassing both the source data (the original log format generated by the device) and the integration configuration. By identifying whether logs arrive as structured JSON, standardized security formats like CEF and LEEF, or unstructured raw text, Cortex XDR can properly parse the information for analysis.

Supported parser types

• CEF (Common Event Format): A parser for logs formatted in the ArcSight CEF standard. CEF logs contain a pipe-delimited header with vendor, product, version, event class, name, and severity fields, followed by key-value pair extensions. Commonly used by vendors like Check Point, Fortinet, and Zscaler.
• Cisco ASA: A parser for specific Cisco Adaptive Security Appliance (ASA) syslog messages. Supports logs like connection-related messages (Built/Teardown) and AnyConnect VPN events. Only a subset of Cisco ASA message types is supported.
• Corelight: A parser for network traffic logs generated by Corelight sensors (based on Zeek/Bro). Processes structured JSON logs containing network connection metadata such as connection records, DNS queries, and HTTP transactions.
• Filebeat: A parser for logs collected and forwarded using Elastic Filebeat agents. Extracts the log payload from the Filebeat JSON envelope, handling metadata fields such as timestamps, agent information, and host details.
• JSON: A general-purpose parser for logs sent in standard JSON format. This is the most common format for third-party data sources that send structured data, including cloud services, SaaS applications, and API-based integrations.
• LEEF (Log Event Extended Format): A parser for the IBM QRadar LEEF standard. LEEF logs contain a tab-delimited or custom-delimited header with version, vendor, product, product version, and event ID fields, followed by key-value pair attributes. Typically used by IBM security products and QRadar-integrated vendors.

  Note

  CEF and LEEF logs are often wrapped in a Syslog envelope (RFC 3164/5424). Cortex XDR automatically detects and strips the Syslog header to extract the reporting device IP and hostname before parsing the inner CEEF or LEEF payload.
• Raw Text: A parser for unstructured, plain-text log data that does not conform to any specific structured format. The raw text is ingested as-is and can be processed using parsing rules. This is also the default parser used when the log format cannot be identified or is not explicitly defined.
• WEC (Windows Event Collection): A parser for Windows Event logs collected via Windows Event Forwarding (WEF/WEC). Parses XML-formatted Windows Event logs and converts them into a structured format. Supports Windows Security, System, and Application event logs.
• Windows DNS Debug: A parser for Microsoft Windows DNS Server debug log files. Extracts DNS query and response details including query type, remote IP, protocol, response code, and question name from the Windows DNS debug log format.
• Winlogbeat: A parser for Windows Event logs collected using Elastic Winlogbeat agents. Similar to the WEC parser but handles the Winlogbeat-specific JSON envelope format, extracting Windows Event data along with associated metadata.

Datasets in XQL

Important

By default, forensic datasets are not included in XQL query results, unless the dataset query is explicitly defined to use a forensic dataset.

Cortex Query Language (XQL) supports using different languages for dataset and field names. In addition, when setting up your XQL query, it is important to keep in mind the following:

• The dataset formats supported are dependent on the data retention offerings available in Cortex XDR according to whether you want to query hot storage or cold storage.

  • Hot Storage queries are performed on a dataset using the format dataset = <dataset name>. This is the default option.

    dataset = xdr_data

  • Cold Storage queries are performed using the format cold_dataset = <dataset name>.

    cold_dataset = xdr_data

• Dataset refresh times: While most out-of-the-box system datasets are ingested in near real-time, the following datasets have specific refresh schedules.

  • endpoints: Refreshed every hour.
  • pan_dss_raw: Refreshed daily.
  • Forensics datasets: Data collection behavior depends on your Agent Settings profile.

    • Default: Data is collected as a one-time snapshot and does not update.
    • Scheduled: If you specify a collection interval, the value represents the number of hours between updates, such as an interval of 24 equals once per day.
    • Minimum: The shortest allowable interval is 12 hours.
• Query against a dataset by selecting it with the dataset command when you create an XQL query. For more information, see Create XQL query.
• After your query runs, you can always save your query results as a dataset. You can use the target stage command to save query results as a dataset.
• Schema changes to datasets may not be reflected in the autocomplete suggestions and definitions as you type in real time the XQL query and can appear with a slight delay.

Managing datasets and dataset views

You can manage your datasets and dataset views in Cortex XDR from the Settings → Configurations → Data Management → Dataset Management page.

Below are some of the main tasks available for all dataset types by right-clicking a particular dataset or dataset view listed in either the Datasets or Dataset Views table. Only tasks that need further explanation are explained below. Datasets and dataset views can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset or dataset view or dataset view, you wouldn't be able to delete the dataset or dataset view until you removed the dataset view from the XQL query of the Correlation Rule.

Note

For more information on tasks specific to lookup datasets, see Lookup datasets.