> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xsiam/detect-investigate-and-respond-to-threats/investigation-and-response/notebooks/manage-datasets-in-notebooks.md). # Manage datasets in Notebooks Create, edit, and delete datasets directly in Notebooks and use them in rules. You can create datasets in BigQuery through Notebooks using custom Cortex XSIAM APIs. You can then bring the insights and enriched data through machine learning into Cortex XSIAM to use them inside rules. For example, you can run a query in Cortex XSIAM that searches a case and correlates it to a sensitive users list you've created in Notebooks to trigger an issue. To use the Cortex XSIAM APIs inside Notebooks, in **Investigation & Response** → **Notebooks**, import them from the Cortex SDK. ```programlisting from cortex.dataset import define_dataset, create_dataset_from_dataframe, delete_dataset, get_created_datasets. from cortex.xql import start_query, get_query_results. ``` The created datasets are available for querying in the **Query Builder** and can be used when defining rules. You can view them under **Dataset Management**, and they can be selected for access when creating a user role. Creating and deleting datasets are recorded in the **Management Audit Logs**. To change the schema of a dataset created using the Notebooks API, delete the dataset and create a new dataset with the updated schema. You can use all the Google BigQuery functions to update the data in a dataset created using the Notebooks API. The functions that are available for creating and editing datasets in Notebooks are listed below.
Define dataset Creates an XQL dataset based on an existing BQ table. ```programlisting define_dataset(table_name: str, client: Optional[Client] = None) ``` | | | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | Arguments | |
Create a dataset from a dataframe Creates an XQL dataset and the table at the same time, where you supply the data and the schema of the table in the API. ```programlisting create_dataset_from_dataframe( table_name: str, dataframe: DataFrame, schema: Optional[Sequence[Union[SchemaField, Mapping[str, Any]]]] = None, client: Optional[Client] = None, bq_client: Optional[BqClient] = None, ) ```
Arguments
  • table_name: Dataset name.

  • schema: Schema of the table - a list of dicts or SchemaField objects defining the structure of the table.

    For example:

    schema = [ SchemaField("project_name", "STRING"), SchemaField("project_id", "INT64"), SchemaField("users", "STRING", mode="REPEATED"), SchemaField("assets", "RECORD", mode="REPEATED", fields=[ SchemaField("ip", "STRING"), SchemaField("created_time", "TIMESTAMP") ]) ]

  • client: Cortex HTTP client.
  • bq_client: Cortex BigQuery client.
{% hint style="info" %} If a schema is not provided, the function automatically detects the schema. {% endhint %}
Get created datasets Retrieves a list of all XQL datasets generated using the Cortex SDK. ```programlisting get_created_datasets(client: Optional[Client] = None) ```
Argumentsclient: Cortex HTTP client.
Delete dataset Deletes the XQL dataset that was created by the Cortex SDK. {% hint style="info" %} * Using this function, you can only delete datasets created using the Notebooks APIs. * When you delete a dataset, the rules that use the dataset return an error. {% endhint %} ```programlisting delete_dataset(dataset_name: str, delete_underlying_bq_table: Optional[bool] = False, client: Optional[Client] = None) ```
Arguments
  • dataset_name: Name of the dataset to be deleted.
  • delete_underlying_bq_table: When True, deletes the BQ table related to the dataset. Default value is False.
  • client: Cortex HTTP client.
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xsiam/detect-investigate-and-respond-to-threats/investigation-and-response/notebooks/manage-datasets-in-notebooks.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.