> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr/configure-cortex-xdr/data-management/dataset-management.md). # Dataset management Learn more about managing your datasets and understanding your overall data storage, period-based retention. {% hint style="warning" %} ### Prerequisite Dataset Management requires the RBAC permissions for **Dataset Management** to be **Enabled** (under **Configurations** → **Data Management**). * Users must have the **Create Datasets** capability to create new user, lookup, dataset views, or correlation datasets. * Users must have the **Edit Public Datasets** capability to modify datasets marked as **Public** that they do not own. {% endhint %} The **Dataset Management** page enables you to manage your datasets and understand your overall data storage duration for different retention periods and datasets based on your hot and cold storage licenses, and retention add-ons that extend your storage. You can view details about your Cortex XDR licenses and retention add-ons by selecting **Settings** → **Cortex XDR License**. For more information on license retention and the defaults provided per license, see [Data retention in Cortex XDR](/cortex-xdr/learn-about-cortex-xdr/cortex-xdr-license-plan/data-retention-in-cortex-xdr.md). {% hint style="info" %} ### Important Cortex XDR enforces retention on all log-type datasets excluding Host Inventory, Vulnerability Assessment, Metrics, and Users. {% endhint %}
Hot and cold storage Your current hot and cold storage licenses, including the default license retention and any additional retention add-ons to extend storage, are listed within the **Hot Storage License** and **Cold Storage License** sections of the **Dataset Management** page. Whenever you extend your license retention, depending on your requirements and license add-ons for both hot storage and cold storage, the add-ons are listed. {% hint style="info" %} ### Note Cold storage, in addition to a cold storage license, requires compute units (CU) to run cold storage queries. For more information on CU, see [Manage compute units](/cortex-xdr/configure-cortex-xdr/data-management/manage-compute-units.md). For information on the CU add-on license, see [Cortex XDR license plan](/cortex-xdr/learn-about-cortex-xdr/cortex-xdr-license-plan.md). {% endhint %}
Additional hot storage You can expand your license retention to include flexible Hot Storage based retention to help accommodate varying storage requirements for different retention periods and datasets. This add-on license is available to purchase based on your storage requirements for a minimum of 1,000 GB. If this license is purchased, an **Additional Storage** subheading in the **Hot Storage License** section is displayed on the **Dataset Management** page with a bar indicating how much of the storage is used. {% hint style="info" %} ### Note Only datasets that are already handled as part of the GB license are supported for this license. In addition, the retention configuration is only available in Cortex XDR, as opposed to the public APIs. {% endhint %}
Edit the retention plan On any dataset configured to use Additional Hot Storage, you can edit the retention period. This enables you to view the current retention details and configure the retention. This includes setting the amount of flexible hot storage-based retention designated for a dataset and the priority for the dataset's hot storage.
Datasets table For each dataset listed in the table, the following information is available: {% hint style="info" %} ### Note * Certain fields are exposed and hidden by default. An asterisk (\*) is beside every field that is exposed by default. * **RBAC and SBAC enforcement**: Datasets include dataset permission enforcements in the Cortex Query Language(XQL), Query Center, and XQL Widgets. For example, to view or access any of the **`endpoints`** and **`host_inventory`** datasets, you need role-based access control (RBAC) permissions to the **Endpoint Administration** and **Host Inventory** views. Managed Security Services Providers (MSSP) administration permissions are not enforced on child tenants, but only on the MSSP tenant. While per-object access controls visibility of the dataset itself, underlying data remains governed by Scope-Based Access Control (SBAC). * **Access control**: User-managed datasets (User, Lookup, Dataset Views, and Correlation) enforce object-level permissions. You only see datasets that you own, that are shared with you, or that are marked as **Public**. Actions such as **Share** and **Change Owner** are only available for these user-managed types. * **Target command enforcement**: To prevent unauthorized or accidental modifications to data collections, the XQL `target` command is strictly enforced based on object-level permissions. Users must hold an **Editor** or **Owner** role for a specific dataset to append or overwrite its data; users with **Viewer** access are restricted from these actions. {% endhint %} | Field | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | \*TYPE | Displays the type of dataset based on the method used to upload the data. The possible values include: Correlation, Lookup, Raw, Snapshot, System, and User. For more information on each dataset type, see [What are datasets?](#what-are-datasets). | | \*LOG UPDATE TYPE | Event logs are updated either continuously (**Logs**) or the current state is updated periodically (**State**) as detailed in the **Last Updated** column. | | \*CREATED BY | Indicates the user who created the dataset. For system-provided datasets, this displays **Palo Alto Networks**. | | \*LAST UPDATED |

Last time the data in the dataset logs were updated.

Important

This column is updated once a day. Therefore, if the dataset was created or updated by the target or lookup flows, it's possible that the Last Updated value is a day behind when the queries or reports were run as it was before this column was updated.

| | \*SOURCE | Displays the source principal or process that created or last updated the dataset entries. For system-provided datasets, this displays **Palo Alto Networks**. | | \*ACCESS | Details the access level of the dataset. For user-managed datasets, this is either **Restricted** or **Public**. For system-managed datasets, this displays **System Default**. | | \*ADDITIONAL STORAGE | Amount of flexible hot storage-based retention designated for this dataset in months, where a month is calculated as 31 days. | | \*TOTAL DAYS STORED | Actual number of days that the data is stored in the Cortex XDR tenant, which is comprised of the **HOT RANGE** + the **COLD RANGE**. | | \*HOT RANGE | Details the exact period of the Hot Storage from the start date to the end date. | | \*COLD RANGE | Details the exact period of the Cold Storage from the start date to the end date. | | \*TOTAL SIZE STORED | Actual size of the data that is stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage. For the **`xdr_data`** dataset, where the first 31 days of storage are included with your license, the first 31 days are not included in the **TOTAL SIZE STORED** number. | | \*ADDITIONAL SIZE STORED | Actual size of the additional flexible hot storage data that is stored in the Cortex XDR tenant in GB. This number is dependent on the events stored in the hot storage. | | \*AVERAGE DAILY SIZE | Average daily amount stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage. | | \*HOT STORAGE PRIORITY | Indicates the priority set for the dataset's hot storage as either **Low**, **Medium**, or **High**. | | \*TOTAL EVENTS | Number of total events/logs that are stored in the Cortex XDR tenant. This number is dependent on the events stored in the hot storage. | | \*AVERAGE EVENT SIZE | Average size of a single event in the dataset (**TOTAL SIZE STORED** divided by the **TOTAL EVENTS**). This number is dependent on the events stored in the hot storage. | | \*TTL |

For lookup datasets, displays the value of the time to live (TTL) configured for when lookup entries expire and are removed automatically from the dataset. The possible values are:

For more information, see Set time to live for lookup datasets.

| | DEFAULT QUERY TARGET | Details whether the dataset is configured to use as your default query target in XQL Search, so when you write your queries you do not need to define a dataset. By default, only the **`xdr_data`** dataset is configured as the **DEFAULT QUERY TARGET** and this field is set to **Yes**. All other datasets have this field set to **No**. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XDR queries the default datasets using a **`join`**. | | TOTAL HOT RETENTION | Total hot storage retention configured for the dataset in months, where a month is calculated as 31 days. | | TOTAL COLD RETENTION | Total cold storage retention configured for the dataset in months, where a month is calculated as 31 days. |
Dataset views Cortex XDR supports creating dataset views in the `Dataset Management` page to enhance data efficiency and security. Dataset views provide a virtual representation of data from one or more datasets, based on the Cortex Query Language (XQL) query defined, and provide multiple benefits, such as joining datasets into logical subsets through defined queries, manipulating data without altering underlying datasets, and segregating data for specific user needs or access privileges through the per-object access settings. Once a dataset view is created, you can edit or delete the dataset view by right-clicking the dataset view in the **Dataset Views** table. A dataset view can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset view, you wouldn't be able to delete the dataset view until you removed the dataset view from the XQL query of the Correlation Rule. Cortex XDR logs entries for events related to creating, editing, and deleting datasets or dataset views. These monitored activities are available to view in the datasets and dataset views audit logs in the Management Audit Logs. For more information, see [Monitor datasets and dataset views activity](/cortex-xdr/configure-cortex-xdr/data-management/dataset-management/monitor-datasets-and-dataset-views-activity.md). **Building XQL dataset view queries** When building an XQL query to define a dataset view, the query is built in the same way as creating a query through XQL Search. Yet, it's important to be aware of the following points that are specific for dataset view queries: * The following features are unsupported in dataset view queries: * RT Correlation Rules * Query Library * Presets * Cold storage queries (`cold_dataset = `) * Only the following XQL stages are supported when building a dataset view query: * `alter` * `dedup` * `fields` * `filter` * `join` * `replacenull` * `union` * Once the dataset view is created, it is listed as an available `dataset` when building your XQL queries as long as you are the owner or have been granted access. **How to create a dataset view** 1. Select Settings → Configurations → Data Management → Dataset Management → Dataset Views. 2. Click New Datset View. 3. Enter a Name and Description (optional) for the dataset view. 4. Create your XQL query for the dataset view by typing in the query box. 5. (Optional) Click Run to view the query results. The query must contain no errors, including using only supported commands, to run; otherwise, the Run button remain disabled. 6. Click Save. You'll only be able to save the dataset view if the query contains no errors; otherwise, the Save button is disabled. By default, newly created dataset views are **Restricted**.\ Once the dataset view is created, you can now control user access permissions through the **Share** action. **Dataset views access permissions** Access for dataset views is managed through the same per-object access framework as other datasets and custom objects. By using object-level access, you can ensure that specific dataset views are only accessible to authorized users and user groups. Once the dataset view is shared with the designated principals, they can access it according to their assigned **Viewer** or **Editor** role. For more information, see [Manage access to datasets](/cortex-xdr/onboard-cortex-xdr/post-deployment-steps/manage-user-roles-and-access-management/manage-access-to-objects/manage-access-to-datasets.md). {% hint style="info" %} #### Note Only Account Admins and Instance Administrators have the authority to change the owner of a dataset view. {% endhint %} How to set access permissions for dataset views 1. Select **Settings** → **Configurations** → **Access Management**. 2. Under **Dataset Views**, right-click the dataset view and select **Share**. * (If sharing is enabled) Search for the **User** or **User Group**. 3. Assign the access level: **Viewer** (read/query) or **Editor** (modify/append). 4. Set the **General access** state: * **Restricted**: Private to the Owner and explicitly added principals. * **Public**: Visible to all users with the **Dataset Management** RBAC permission enabled. 5. Click **Save**. For more information on user roles, see [Manage access to datasets](/cortex-xdr/onboard-cortex-xdr/post-deployment-steps/manage-user-roles-and-access-management/manage-access-to-objects/manage-access-to-datasets.md). **Dataset Views table** For each dataset view listed in the table, information is available. Here are descriptions on the columns that may require further explanation: | Field | Description | | -------------- | --------------------------------------------------------------------------- | | \*CREATED BY | The username of the creator or **Palo Alto Networks**. | | \*SOURCE | Displays the source principal or process for the dataset view. | | \*ACCESS | The access level, either **Restricted**, **Public**, or **System Default**. | | SOURCE QUERY | Displays the query used to create the dataset view. | | IS VALID | Details whether the query for the dataset view is still valid or not. | | RELATED TABLES | Details the other datasets that are related to this dataset view. |
Change owner of a dataset or dataset view To ensure continuity when personnel changes occur or to hand off management of a resource, administrators can change the ownership of a user-managed dataset (User, Lookup, Dataset Views, or Correlation) or dataset view. {% hint style="info" %} #### Note Only Account Admins and Instance Administrators have the authority to change the owner of an object. {% endhint %} 1. Select **Settings** → **Configurations** → **Data Management** → **Dataset Management**. 2. In the **Datasets** or **Dataset Views** table, right-click the user-managed item and select **Change Owner**. 3. Select the new owner from the list of users, and click **Change**.
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xdr/configure-cortex-xdr/data-management/dataset-management.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.