> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr/configure-cortex-xdr/cortex-xdr-data-sources/administration-and-troubleshooting/overview-of-data-ingestion-metrics.md).

# Overview of data ingestion metrics

Manage and troubleshoot Cortex XDR data integrations.

{% hint style="warning" %}

### Prerequisite

For Cortex XDR to monitor data ingestion health and create health issues, you must enable **Cortex - Analytics.** Go to Configurations → **Cortex - Analytics**. For more information, see [Enable the Analytics Engine and Identity Analytics](https://cortex-docs.paloaltonetworks.com/cortex-xdr/configure-cortex-xdr/configure-and-deploy-cortex-xdr/enable-the-analytics-engine-and-identity-analytics.md).
{% endhint %}

The data ingestion metrics are calculated in 5-minute aggregation periods and saved to the `metrics_source` dataset. These metrics measure the amount, size, and rate at which logs are ingested by a data source:

| Metric              | Description                                                                             |
| ------------------- | --------------------------------------------------------------------------------------- |
| total\_size\_bytes  | Total size (in bytes) of the logs collected during the aggregation period.              |
| total\_size\_rate   | Average size (in bytes per second) of the logs collected during the aggregation period. |
| total\_event\_count | Total number of logs collected during the aggregation period                            |
| total\_event\_rate  | Average number (in count per second) of logs collected during the aggregation period.   |

In the `metrics_source` dataset, the data ingestion metrics are saved alongside additional fields that describe the data source associated with the metrics. Only entries with ingestion metric values greater than zero are saved in the dataset. Entries with zero values are not saved in this dataset.

In addition, you can create your own custom logic for data ingestion health monitoring by setting up correlation rules that monitor the data ingestion metrics. For more information, see [Creating correlation rules to monitor data ingestion health](https://cortex-docs.paloaltonetworks.com/cortex-xdr/configure-cortex-xdr/cortex-xdr-data-sources/administration-and-troubleshooting/overview-of-data-ingestion-metrics/creating-correlation-rules-to-monitor-data-ingestion-health.md).

The following table describes all the fields in the `metrics_source` dataset:

<details>

<summary>Read more...</summary>

| Field                                  | Type     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| -------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| total\_size\_bytes                     | Integer  | Total size (in bytes) of the logs collected during the aggregation period.                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| total\_size\_rate                      | Integer  | Average size (in bytes per second) of the logs collected during the aggregation period.                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| total\_event\_count                    | Integer  | Total number of logs collected during the aggregation period                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| total\_event\_rate                     | Integer  | Average number (in count per second) of logs collected during the aggregation period.                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| data\_freshness\_max\_delay            | Float    | The highest delay bucket reached during the 5-minute aggregation period for logs from the data source (in seconds). This value represents the upper boundary of coarse predefined buckets (`0`, `30`, `60`, `300`, `3600`, `86400`, `604800` seconds) and reflects end-to-end elapsed time (`_insert_time − _time`), including source device clock drift, buffering, network latency, and intermediate collection hops, not Cortex ingestion processing alone. It should not be interpreted as an exact delay measurement. |
| data\_freshness\_median                | Float    | Estimated median delay (p50) across logs collected during the aggregation period (in seconds), calculated from aggregated delay buckets. Measures end-to-end elapsed time (`_insert_time − _time`).                                                                                                                                                                                                                                                                                                                        |
| data\_freshness\_ninetieth\_percentile | Float    | Estimated 90th percentile delay (p90) across logs collected during the aggregation period (in seconds), calculated from aggregated delay buckets. For delays exceeding 1 hour, values are estimated across a wide bucket range (1h to 24h). Measures end-to-end elapsed time (`_insert_time − _time`).                                                                                                                                                                                                                     |
| last\_seen                             | Datetime | Timestamp when the last real log was collected from the data source (grouped by `_vendor`, `_product`, `_device_id`, and `_collector_type`).                                                                                                                                                                                                                                                                                                                                                                               |
| \_vendor                               | String   | Vendor of the observing data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| \_product                              | String   | Product name of the observing data source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| \_device\_id                           | String   | (For firewall devices) Device ID                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| \_log\_type                            | String   | (For firewall devices) Log type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| \_collector\_type                      | String   | (Event Metadata) Type of collector that provided the log.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| \_collector\_name                      | String   | (Event Metadata) Name of the collector instance.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| \_collector\_id                        | String   | (Event Metadata) ID of the XDR Collector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| \_collector\_ip                        | String   | (Event Metadata) IP address of the XDR Collector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| \_reporting\_device\_name              | String   | (Event Metadata) Host name of the device where the log originated.                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| \_reporting\_device\_ip                | String   | (Event Metadata) IP Address of the device where the log originated.                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| \_final\_reporting\_device\_name       | String   | (Event Metadata) Hostname of the device that the log was extracted from.                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| \_final\_reporting\_device\_ip         | String   | (Event Metadata) IP of the device that the log was extracted from.                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| \_broker\_device\_name                 | String   | (Event Metadata) Host name of the Broker VM.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| \_broker\_device\_ip                   | String   | (Event Metadata) IP address of the Broker VM.                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| \_broker\_device\_id                   | String   | (Event Metadata) ID of the Broker VM.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| \_time                                 | Datetime | Timestamp of the interval.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| \_insert\_timestamp                    | Datetime | Recorded time of the entry.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://cortex-docs.paloaltonetworks.com/cortex-xdr/configure-cortex-xdr/cortex-xdr-data-sources/administration-and-troubleshooting/overview-of-data-ingestion-metrics.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
