> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-cloud-runtime-security/cortex-cloud-data-sources-and-connectors/administration-and-troubleshooting/overview-of-data-ingestion-metrics.md).

# Overview of data ingestion metrics

{% hint style="warning" %}

### Prerequisite

For Cortex Cloud to monitor data ingestion health and create health issues, you must enable **Cortex - Analytics.** Go to Configurations → **Cortex - Analytics**. For more information, see [Enable the Analytics Engine and Identity Analytics](/cortex-cloud-runtime-security/threat-management/analytics/identity-analytics.md).
{% endhint %}

The data ingestion metrics are calculated in 5-minute aggregation periods and saved to the `metrics_source` dataset and `metrics_view` preset. These metrics measure the amount, size, and rate at which logs are ingested by a data source:

| Metric              | Description                                                                             |
| ------------------- | --------------------------------------------------------------------------------------- |
| total\_size\_bytes  | Total size (in bytes) of the logs collected during the aggregation period.              |
| total\_size\_rate   | Average size (in bytes per second) of the logs collected during the aggregation period. |
| total\_event\_count | Total number of logs collected during the aggregation period                            |
| total\_event\_rate  | Average number (in count per second) of logs collected during the aggregation period.   |

In the `metrics_source` dataset, the data ingestion metrics are saved alongside additional fields that describe the data source associated with the metrics. Only entries with ingestion metric values greater than zero are saved in the dataset. Entries with zero values are not saved in this dataset.

`metrics_view` is a preset for data in the `metrics_source` dataset. The preset also simulates completion of entries with zero values in data ingestion metrics at runtime, which allows effective use of metrics. Therefore, when investigating disruptions in data collection, we recommend using the `metrics_view` preset in XQL queries and correlation rules.

In addition, you can create your own custom logic for data ingestion health monitoring by setting up correlation rules that monitor the data ingestion metrics. For more information, see [Creating correlation rules to monitor data ingestion health](/cortex-cloud-runtime-security/cortex-cloud-data-sources-and-connectors/administration-and-troubleshooting/overview-of-data-ingestion-metrics/creating-correlation-rules-to-monitor-data-ingestion-health.md).

The following table describes all the fields in the `metrics_source` dataset and `metrics_view` preset:

<details>

<summary>Read more...</summary>

| Field                                  | Type     | Description                                                                                                                                            |
| -------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| total\_size\_bytes                     | Integer  | Total size (in bytes) of the logs collected during the aggregation period.                                                                             |
| total\_size\_rate                      | Integer  | Average size (in bytes per second) of the logs collected during the aggregation period.                                                                |
| total\_event\_count                    | Integer  | Total number of logs collected during the aggregation period                                                                                           |
| total\_event\_rate                     | Integer  | Average number (in count per second) of logs collected during the aggregation period.                                                                  |
| data\_freshness\_max\_delay            | Float    | Maximum delay value from all log entries in a record between log creation at the source and ingestion into Cortex Cloud (in seconds).                  |
| data\_freshness\_median                | Float    | Median delay value from all log entries in a record between log creation at the source and ingestion into Cortex Cloud (in seconds).                   |
| data\_freshness\_ninetieth\_percentile | Float    | Ninetieth percentile of delay values from all log entries in a record between log creation at the source and ingestion into Cortex Cloud (in seconds). |
| last\_seen                             | Datetime | Time that the last logs were collected.                                                                                                                |
| \_vendor                               | String   | Vendor of the observing data source.                                                                                                                   |
| \_product                              | String   | Product name of the observing data source.                                                                                                             |
| \_device\_id                           | String   | (For firewall devices) Device ID                                                                                                                       |
| \_log\_type                            | String   | (For firewall devices) Log type                                                                                                                        |
| \_collector\_type                      | String   | (Event Metadata) Type of collector that provided the log.                                                                                              |
| \_collector\_name                      | String   | (Event Metadata) Name of the collector instance.                                                                                                       |
| \_collector\_id                        | String   | (Event Metadata) ID of the XDR Collector.                                                                                                              |
| \_collector\_ip                        | String   | (Event Metadata) IP address of the XDR Collector.                                                                                                      |
| \_reporting\_device\_name              | String   | (Event Metadata) Host name of the device where the log originated.                                                                                     |
| \_reporting\_device\_ip                | String   | (Event Metadata) IP Address of the device where the log originated.                                                                                    |
| \_final\_reporting\_device\_name       | String   | (Event Metadata) Hostname of the device that the log was extracted from.                                                                               |
| \_final\_reporting\_device\_ip         | String   | (Event Metadata) IP of the device that the log was extracted from.                                                                                     |
| \_broker\_device\_name                 | String   | (Event Metadata) Host name of the Broker VM.                                                                                                           |
| \_broker\_device\_ip                   | String   | (Event Metadata) IP address of the Broker VM.                                                                                                          |
| \_broker\_device\_id                   | String   | (Event Metadata) ID of the Broker VM.                                                                                                                  |
| \_time                                 | Datetime | Timestamp of the interval.                                                                                                                             |
| \_insert\_timestamp                    | Datetime | Recorded time of the entry.                                                                                                                            |

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://cortex-docs.paloaltonetworks.com/cortex-cloud-runtime-security/cortex-cloud-data-sources-and-connectors/administration-and-troubleshooting/overview-of-data-ingestion-metrics.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.