Dataset management

Learn more about managing your datasets and understanding your overall data storage, period-based retention.

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

The Dataset Management page enables you to manage your datasets and understand your overall data storage duration for different retention periods and datasets based on your hot and cold storage licenses, and retention add-ons that extend your storage. You can view details about your Cortex Xpanse licenses and retention add-ons by selecting Settings → Cortex Xpanse License.

Important

Cortex Xpanse enforces retention on all log-type datasets excluding Host Inventory, Vulnerability Assessment, Metrics, and Users.

Datasets table

For each dataset listed in the table, the following information is available:

Note

• Certain fields are exposed and hidden by default. An asterisk (*) is beside every field that is exposed by default.
• Datasets include dataset permission enforcements in the Cortex Query Language(XQL), Query Center, and XQL Widgets. For example, to view or access any of the endpoints and host_inventory datasets, you need role-based access control (RBAC) permissions to the Endpoint Administration and Host Inventory views. Managed Security Services Providers (MSSP) administration permissions are not enforced on child tenants, but only on the MSSP tenant.

Field	Description
*TYPE	Displays the type of dataset based on the method used to upload the data. The possible values include: Correlation, Lookup, Raw, Snapshot, System, and User. For more information on each dataset type, see What are datasets?.
*LOG UPDATE TYPE	Event logs are updated either continuously (Logs) or the current state is updated periodically (State) as detailed in the Last Updated column.
*LAST UPDATED	Last time the data in the dataset logs were updated. Important This column is updated once a day. Therefore, if the dataset was created or updated by the target or lookup flows, it's possible that the Last Updated value is a day behind when the queries or reports were run as it was before this column was updated.
*ADDITIONAL STORAGE	Amount of flexible hot storage-based retention designated for this dataset in months, where a month is calculated as 31 days.
*TOTAL DAYS STORED	Actual number of days that the data is stored in the Cortex Xpanse tenant, which is comprised of the HOT RANGE + the COLD RANGE.
*HOT RANGE	Details the exact period of the Hot Storage from the start date to the end date.
*COLD RANGE	Details the exact period of the Cold Storage from the start date to the end date.
*TOTAL SIZE STORED	Actual size of the data that is stored in the Cortex Xpanse tenant. This number is dependent on the events stored in the hot storage. For the xdr_data dataset, where the first 31 days of storage are included with your license, the first 31 days are not included in the TOTAL SIZE STORED number.
*ADDITIONAL SIZE STORED	Actual size of the additional flexible hot storage data that is stored in the Cortex Xpanse tenant in GB. This number is dependent on the events stored in the hot storage.
*AVERAGE DAILY SIZE	Average daily amount stored in the Cortex Xpanse tenant. This number is dependent on the events stored in the hot storage.
*HOT STORAGE PRIORITY	Indicates the priority set for the dataset's hot storage as either Low, Medium, or High.
*TOTAL EVENTS	Number of total events/logs that are stored in the Cortex Xpanse tenant. This number is dependent on the events stored in the hot storage.
*AVERAGE EVENT SIZE	Average size of a single event in the dataset (TOTAL SIZE STORED divided by the TOTAL EVENTS). This number is dependent on the events stored in the hot storage.
*TTL	For lookup datasets, displays the value of the time to live (TTL) configured for when lookup entries expire and are removed automatically from the dataset. The possible values are: Forever: Lookup entries never expire (default). Custom: Lookup entries expire according to a set number of days, hours, and minutes. The maximum number of days is 99999.
DEFAULT QUERY TARGET	Details whether the dataset is configured to use as your default query target in XQL Search, so when you write your queries you do not need to define a dataset. By default, only the xdr_data dataset is configured as the DEFAULT QUERY TARGET and this field is set to Yes. All other datasets have this field set to No. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex XSIAM queries the default datasets using a join.
TOTAL HOT RETENTION	Total hot storage retention configured for the dataset in months, where a month is calculated as 31 days.
TOTAL COLD RETENTION	Total cold storage retention configured for the dataset in months, where a month is calculated as 31 days.

Dataset views

Cortex Xpanse supports creating dataset views in the Dataset Management page to enhance data efficiency and security. Dataset views provide a virtual representation of data from one or more datasets, based on the Cortex Query Language (XQL) query defined, and provide multiple benefits, such as joining datasets into logical subsets through defined queries, manipulating data without altering underlying datasets, and segregating data for specific user needs or access privileges through the Role-based access control (RBAC) settings.

Once a dataset view is created, you can edit or delete the dataset view by right-clicking the dataset view in the Dataset Views table. A dataset view can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset view, you wouldn't be able to delete the dataset view until you removed the dataset view from the XQL query of the Correlation Rule.

Cortex Xpanse logs entries for events related to creating, editing, and deleting datasets or dataset views. These monitored activities are available to view in the datasets and dataset views audit logs in the Management Audit Logs. For more information, see Monitor datasets and dataset views activity.

Building XQL dataset view queries

When building an XQL query to define a dataset view, the query is built in the same way as creating a query through the Query Builder. Yet, it's important to be aware of the following points that are specific for dataset view queries:

• The following features are unsupported in dataset view queries:

  • RT Correlation Rules
  • Cortex Data Model (XDM)
  • Query Library
  • Presets
  • Cold storage queries (cold_dataset = <dataset name>)
• Only the following XQL stages are supported when building a dataset view query:

  • alter
  • dedup
  • fields
  • filter
  • join
  • replacenull
  • union
• Once the dataset view is created, it is listed as an available dataset when building your XQL queries as long as you have the necessary permissions to access the dataset view in the Role-based access control (RBAC) settings.

How to create a dataset view

1. Select Settings > Configurations > Data Management > Dataset Management > Dataset Views.
2. Click New Dataset View.
3. Enter a Name and Description (optional) for the dataset view.
4. Create your XQL query for the dataset view by typing in the query box.
5. (Optional) Click Run to view the query results.

• The query must contain no errors, including using only supported commands, to run; otherwise, the Run button remain disabled.

1. Click Save.

• Note
• You'll only be able to save the dataset view if the query contains no errors; otherwise, the Save button is disabled.
• Once the dataset view is created, you can now control user access permissions through Role-based access control (RBAC).

Dataset views access permissions

Note

Managing Roles requires an Account Admin or Instance Administrator role. For more information, see Predefined user roles.

Access permissions for dataset views are configured in the same way that you set dataset access permissions for any dataset through user roles in Cortex Xpanse Access Management. Cortex Xpanse uses role-based access control (RBAC) to manage roles with specific permissions for controlling user access. RBAC helps manage access to Cortex Xpanse components and datasets, so that users, based on their roles, are granted minimal access required to accomplish their tasks. Once the user role is configured to access these dataset views, you can now assign the user role to the designated users or user groups, who you want to access these dataset views.

How to set access permissions for dataset views

1. Select Settings > Configurations > Access Management.
2. Configure a user role with the dataset views that you want users to access.

   1. Select Roles.
   2. You can perform one of the following:

      • To create a new role to assign the dataset views, click New Role, and set a Role Name and Description (optional).
      • To edit an existing user role with these dataset views, right-click the relevant user role, and select Edit Role.
      • To create a new role based on an existing role, right-click the relevant user role, select Save As New Role, and set a Role Name and Description (optional).
   3. Under Datasets, you have two options for setting the Cortex Query Language (XQL) dataset access permissions for the user role:

      • Set the user role with access to all XQL datasets by disabling the Enable dataset access management toggle.
      • Set the user role with limited access to certain XQL datasets by selecting the Enable dataset access management toggle and selecting the datasets under the different dataset category headings.
   4. Scroll down to Dataset View and select the particular dataset views that you want assigned to this user role.
   5. Click Save.

   • For more information on user roles, see Manage Roles.
3. Assign the user role with the dataset views configured to the designated users or user groups. For more information, see Assign a user to a role.

Dataset Views table

For each dataset view listed in the table, information is available. Here are descriptions on the columns that may require further explanation:

Field	Description
SOURCE QUERY	Displays the query used to create the dataset view.
IS VALID	Details whether the query for the dataset view is still valid or not.
RELATED TABLES	Details the other datasets that are related to this dataset view.

What are datasets?

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

Cortex Xpanse runs every Cortex Query Language (XQL) query against a dataset. A dataset is a collection of column:value sets. If you do not specify a dataset in your query, Cortex Xpanse runs the query against the default datasets configured, which is by default xdr_data for a dataset query. The xdr_data dataset contains all of the endpoint and network data that Cortex Xpanse collects. For a Cortex Data Model (XDM) query, unless specific datasets are specified, a query will run against all mapped datasets. You can always change the default datasets using the set to default option. You can also upload datasets as a CSV, TSV, or JSON file that contains the data you are interested in querying. These uploaded datasets are called lookup datasets.

It's also possible to create dataset views, which provide a virtual representation of data from one or more datasets, based on the Cortex Query Language (XQL) query defined. Dataset views enhance data efficiency and security. For example, by segregating data for specific user needs or access privileges through the Role-based access control (RBAC) settings. For more information, see Dataset views.

To query other datasets, you have the following options:

• Set a dataset as default, which enables you to query the datasets without specifying them in the query.
• Name a specific dataset at the beginning of your query with the dataset stage command.

Dataset types

The type of dataset is based on the method used to upload the data. The possible types include:

• Correlation: A dataset containing data saved from a correlation rule.
• Lookup: A dataset containing key-value pairs that can be used as a reference to correlate to events. For example, a user list with corresponding access privileges. You can import or create a lookup dataset, and then reference the values for a certain key, run queries and take action. For more information, see Lookup datasets.
• Raw: Every dataset where PANW data is ingested out-of-the-box or third-party data is ingested using a configured dedicated collector. The schema for the raw dataset is automatically generated based on the log data collected by Cortex Xpanse and the data format sent, such as JSON, CEF, and LEEF.
• Snapshot: A dataset that contains only the last successful snapshot of the data, such as Workday or ServiceNow CMDB tables.
• System: Cortex Xpanse datasets that are created out-of-the-box.
• User: If saved by a query using the target command, the Type can be either User or Lookup.

Datasets in XQL

Important

By default, forensic datasets are not included in XQL query results, unless the dataset query is explicitly defined to use a forensic dataset.

Cortex Query Language (XQL) supports using different languages for dataset and field names. In addition, when setting up your XQL query, it is important to keep in mind the following:

• The dataset formats supported are dependent on the data retention offerings available in Cortex Xpanse according to whether you want to query hot storage or cold storage.

  • Hot Storage queries are performed on a dataset using the format dataset = <dataset name>. This is the default option.

    dataset = xdr_data

  • Cold Storage queries are performed using the format cold_dataset = <dataset name>.

    cold_dataset = xdr_data

• Dataset refresh times: While most out-of-the-box system datasets are ingested in near real-time, the following datasets have specific refresh schedules.

  • endpoints: Refreshed every hour.
  • pan_dss_raw: Refreshed daily.
  • Forensics datasets: Data collection behavior depends on your Agent Settings profile.

    • Default: Data is collected as a one-time snapshot and does not update.
    • Scheduled: If you specify a collection interval, the value represents the number of hours between updates, such as an interval of 24 equals once per day.
    • Minimum: The shortest allowable interval is 12 hours.
• Query against a dataset by selecting it with the dataset command when you create an XQL query. For more information, see Create XQL query.
• After your query runs, you can always save your query results as a dataset. You can use the target stage command to save query results as a dataset.
• Schema changes to datasets may not be reflected in the autocomplete suggestions and definitions as you type in real time the XQL query and can appear with a slight delay.

Managing datasets and dataset views

You can manage your datasets and dataset views in Cortex Xpanse from the Settings → Configurations → Data Management → Dataset Management page.

Below are some of the main tasks available for all dataset types by right-clicking a particular dataset or dataset view listed in either the Datasets or Dataset Views table. Only tasks that need further explanation are explained below. Datasets and dataset views can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a dataset or dataset view or dataset view, you wouldn't be able to delete the dataset or dataset view until you removed the dataset view from the XQL query of the Correlation Rule.

For more information on tasks specific to lookup datasets, see Lookup datasets.

View Schema

Select View Schema to view the schema information for every field found in the dataset or dataset view result set in the Schema tab after running the query in XQL. Each system field in the schema is written with an underscore (_) before the name of the field in the FIELD NAME column in the table.

Schema changes to datasets may not be reflected in the autocomplete suggestions and definitions as you type in real time the XQL query and can appear with a slight delay.

Set as default

Select Set as default to query the dataset without having to specify it in your queries in XQL by typing dataset = <name of dataset>. Once configured, the DEFAULT QUERY TARGET column entry for this dataset is set to Yes in the Datasets table. By default, this option is not available when right-clicking the xdr_data dataset as this dataset is the only dataset configured as the DEFAULT QUERY TARGET as it contains all of the endpoint and network data that Cortex Xpanse collects.

Once you Set as default another dataset, you can always remove it by right-clicking the dataset and selecting Remove from defaults. When setting multiple default datasets, your query does not need to mention any of the dataset names, and Cortex Xpanse queries the default datasets using a join. This option is only relevant for datasets.

Copy text to clipboard

Select Copy text to clipboard to copy the name of the dataset or dataset view to your clipboard.

Lookup datasets

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

Lookup datasets enable you to correlate data from a data source you provide with the events in your environment. For example, you can create a lookup with a list of high-value assets, terminated employees, or service accounts in your environment. Use lookups in your search, detection rules, threat hunting, and response playbooks. Lookups are stored as name-value pairs and are cached for optimal query performance and low latency.

Lookup tables support low-frequency changes of up to 1200 modifications per day. Changes are implemented whenever a lookup dataset is edited, where only one person or user can edit the file at a given time. Concurrent users editing the file are not supported.

Use case scenarios

• Investigate threats and respond to cases quickly with the rapid import of IP addresses, file hashes, and other data from CSV files. After you import the data, use lookup name-value pairs for joins and filters in threat hunting and general queries.
• Import business data as a lookup. For example, import user lists with privileged system access, or terminated employees. Then, use the lookup to create allow lists and blocklists to detect or prevent those users from logging in to the network.
• Create allow lists to suppress issues from a group of users, such as users from authorized IP addresses that perform tasks that would normally trigger the issue. Prevent benign events from becoming issues.
• Enrich event data. Use lookups to enrich your event data with name-value combinations derived from external data sources.

How are lookup datasets created?

You can import or create a lookup dataset, and then reference the values for a certain key, run queries, and take action. Lookup datasets are created by any of the following methods:

• Manual upload from a CSV, TSV, or JSON file to Cortex Xpanse from the Dataset Management page. For more information, see Import a lookup dataset.
• Automatic upload by the Files and Folders Collector.
• Query results are saved to a lookup dataset. If saved using the target stage, the Type can be either User or Lookup. For more information, see the target stage.

  Important

  When you create or add data to a lookup dataset using the target stage, the _time field won't be included by default unless you explicitly add it with the fields stage.

After a lookup, a dataset is imported, you can always edit the dataset to update the data manually by right-clicking the dataset and selecting Edit.

Note

A lookup dataset can only be deleted if there are no other dependencies. For example, if a Correlation Rule is based on a lookup dataset, you wouldn't be able to delete the lookup dataset until you removed the dataset from the XQL query of the Correlation Rule.

Import a lookup dataset

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

You can import data from CSV, TSV, or JSON files into Cortex Xpanse to create or update lookup datasets.

Prerequisite

When uploading a CSV, TSV, or JSON file, ensure that the file meets the following requirements:

• The maximum size for the total data to be imported into a lookup dataset is 30 MB from the Dataset Management page. Otherwise, the limit is 50 MB using Cortex Query Language (XQL) or APIs.
• Field names can contain characters from different languages, special characters, numbers (0-9), and underscores (_).
• Field names can't exceed 128 characters.
• Field names can't contain duplicate names, white spaces, or carriage returns.
• The file doesn't contain a byte array (binary data) as it can't be uploaded.
• Each line in the JSON file must represent one JSON object. Ensure no brackets enclose the objects at the top-level.

Example 34.

Here's an example of a JSON file in the correct format for upload:

{"firstName": "NAME_1", "SurName": "NAME_11", "employeeID": {"id": "ID_AAAAA_2"}}
{"firstName": "NAME_2", "SurName": "NAME_22", "employeeID": {"id": "ID_AAAAA_3"}}
{"firstName": "NAME_3", "SurName": "NAME_32", "employeeID": {"id": "ID_AAAAA_4"}}

1. Select Settings → Configurations → Data Management → Dataset Management → + Lookup.
2. Browse to your CSV, TSV, or JSON file. You can only upload a TSV file if it contains a .tsv file extension.
3. (Optional) Under Name, type a new name for the target dataset.

   By default, Cortex Xpanse uses the name of the original file as the dataset name. You can change this name to something that will be more meaningful for your users when they query the dataset. For example, if the original file name is mrkdptusrsnov23.json, you can save the dataset as marketing_dept_users_Nov_2023.

   Dataset names can contain special characters from different languages, numbers (0-9) and underscores (_). You can create dataset names using uppercase characters, but in queries, dataset names are always treated as if they are lowercase.

   Important

   The name of a dataset created from a TSV file must always include the extension. For example, if the original file name is mrkdptusrsnov23.tsv, you can save the dataset with the name marketing_dept_users_Nov_2023.tsv.
4. Replace the existing data in the dataset overwrites the data in an existing lookup dataset with the contents of the new file.
5. Click Add to add the file as a lookup.
6. After receiving a notification reporting that the upload succeeded, Refresh to view it in your list of datasets.

   If the upload fails for any reason, you'll receive a notification in the Notification Center.

Download JSON file of lookup dataset

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

You can only download a JSON file for a lookup dataset, where the Type set to Lookup on the Dataset Management page. This option is not available for any other dataset type.

When you download a lookup dataset with field names in a foreign language, the downloaded JSON file displays the fields as COL_<randomstring> as opposed to returning the fields in the foreign language as expected.

1. Open the Settings → Configurations → Data Management → Dataset Management page.
2. In the Datasets table, right-click the lookup dataset that you want to download as a JSON file, and select Download.

Set time to live for lookup datasets

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

You can specify when lookup entries expire and are removed automatically from the lookup dataset by configuring the time to live (TTL). The time period of the TTL interval is based on when the data was last updated. The default is forever and the entries never expire. You can also configure a specific time according to the days, hours, and minutes. Expired elements are removed from the lookup dataset by a scheduled job that runs every five minutes.

1. Open the Settings → Configurations → Data Management → Dataset Management page.
2. In the Datasets table, right-click the lookup dataset, and select Set TTL.
3. Select one of the following to configure when lookup dataset entries expire and are removed:

   • Forever: Lookup entries never expire (default).
   • Custom: Lookup entries expire according to a set number of days, hours, and minutes. The maximum number of days is 99999.
4. Click Save.

   The TTL column in the Datasets table is updated with the changes and these changes are applied immediately on all existing lookup entries.

Monitor datasets and dataset views activity

Prerequisite

Dataset Management requires View/Edit RBAC permissions for Data Management (under Configurations → Data Management), which are the same permissions required for Parsing Rules, Data Model Rules, and Event Forwarding.

Cortex Xpanse logs entries for events related to datasets and dataset views monitored activities. Cortex Xpanse stores the logs for 365 days. To view the datasets and dataset views audit logs, select Settings → Management Audit Logs.

You can customize your view of the logs by adding or removing filters to the Management Audit Logs table. You can also filter the page result to narrow down your search. The following table describes the default and optional fields that you can view in the Cortex XSIAM Management Audit Logs table:

Note

Certain fields are exposed and hidden by default. An asterisk (*) is beside every field that is exposed by default.

Field	Description
Description*	Log message that describes the action.
Email	Email of the user who performed the action.
Host Name*	This field is not applicable for datasets and dataset views logs.
ID	Unique ID of the action.
Reason	This field is not applicable for datasets and dataset views logs.
Result*	The result of the action ( Success, Fail, or N/A)
Severity*	Severity associated with the log: Critical High Medium Low Informational
Timestamp*	Date and time when the action occurred.
Type* and Sub-Type*	Additional classifications of dataset and dataset view logs (Type and Sub-Type): Datasets: Create Dataset Delete Dataset Update Dataset Dataset Views: Create Dataset View Delete Dataset View Update Dataset View
User Name*	Name of the user who performed the action.