> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xsiam/configure-cortex-xsiam/data-management/optimize-data-management-in-cortex-xsiam.md).
# Optimize data management in Cortex XSIAM
{% hint style="info" %}
The Cortex Data Lake tier is only available with an active Cortex XSIAM NG SIEM, Cortex XSIAM Enterprise, or Cortex XSIAM Premium license.
Federated Search is not enabled by default. To enable it in your tenant, contact your [Customer Support Team](https://support.paloaltonetworks.com/Support/Index).
{% endhint %}
In modern security operations, balancing the need for comprehensive visibility with the reality of high data volumes and ingestion costs is a constant challenge. Cortex XSIAM addresses this by offering flexible data management solutions designed to align with the specific security value, ingestion requirements, and compliance needs of different log types.
By categorizing data into Analytics or Data Lake tiers, organizations can ensure that high-value logs receive real-time AI/ML processing and detection while supplementary logs are stored cost-effectively. Furthermore, Cortex XSIAM provides Federated Search, a query mechanism designed to provide unified access to distributed data sources without requiring pre-ingestion or centralization. This capability enables you to query data in place, significantly reducing the complexity and operational costs associated with the ingestion process and long-term data retention.
**Comparison of data management solutions**
The following table breaks down the differences between the internal ingestion tiers and the federated query mechanism to help you determine the best approach for your data.
| Point of Comparison | Analytics Tier (Regular cost ingestion) | Data Lake Tier (Cost-Optimized ingestion) | Federated Search (Query mechanism with no ingestion) |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Primary Goal** | Detection and Response | Compliance, Investigation, and Threat Hunting | Compliance and Historical Search |
| **Best For** | High-value security logs needed for real-time AI/ML detection. | High data volume, but low value in terms of real-time security, needed for compliance or as investigation supporting data | Massive historical archives or data you want to manage in your own cloud storage (AWS/GCP/Azure) |
| **Data Location** | Ingested and stored in Cortex XSIAM | Ingested and stored in Cortex XSIAM | Not Ingested. Stays in your Azure Blob/S3/GCS buckets. |
| **AI/ML Analytics** | Full AI/ML analytics support | None | None |
| **Correlation Rules** | Full support | Full Support (consumes Compute Units) | None (Search only) |
| **Usage Model** | Usage included (subject to limits) | Consumes Compute Units | Consumes Compute Units |
| **Key Restriction** | None | - Cannot store PANW Firewall logs in Data Lake Tier
- No Cortex Data Model (XDM) data modeling
- No OOTB Analytics
| - Queries take longer to return results as dependent on external cloud
- Supports specific formats and structure
- Search only
|
| **Decision Criteria** | 1. Real time AI/ML detection
2. Heavy query and access usage
| 1. High data volume, but low value in terms of real-time security; you rarely find threats solely in these, but you need them for context.
2. Data is required primarily for compliance or hunting. You need to keep it for 12 months for auditors, but you only search it occasionally
3. Supports correlation rules. You can write XQL rules to trigger alerts on this data, even though it doesn't get the full AI/ML treatment.
| 1. No ingestion required, data remains in its original location
2. Strict data sovereignty or ownership
3. This tier is Search Only. You cannot run, for example, correlation rules, scheduled queries, widgets, and dashboards, on Federated Search data. It is strictly for ad-hoc investigations via the XQL query builder.
|
| **Examples** | Any of the primary data sources like Firewall, Identity, Endpoint, Cloud audit log, and network logs (such as, Cloud Flow, WAF, Load Balancer, and NetFlow) for critical or sensitive environments. | Non-critical Application logs and high-volume Network logs (Cloud Flow, WAF, Load Balancer, raw NetFlow) from non-critical environments.
Note
If a data source is heavily used for real-time Detection and Response (whether out-of-the-box Cortex XSIAM detections or user-defined rules), we recommend ingesting it via the Analytics tier, which includes all real-time detection capabilities.
Given they follow the supported formats (CSV, Parquet, JSONL) and Hive structure:
1. Raw application logs
2. CDN Logs, such as Akamai and Cloudflare: High-volume edge traffic logs stored in S3/Blob storage.
3. Database Audit Logs, such as RDS and SQL Audit: Compliance-heavy logs that prove "who accessed what table" years ago.
4. Any structured data you export from other systems, such as HR data dumps and old SIEM archives, into a "Hive" folder structure, such as ds=2024-01-01/, in your cloud bucket. Cortex XSIAM can query these directly.
|
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://cortex-docs.paloaltonetworks.com/cortex-xsiam/configure-cortex-xsiam/data-management/optimize-data-management-in-cortex-xsiam.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.