> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr/configure-cortex-xdr/cortex-xdr-data-sources/administration-and-troubleshooting/overview-of-data-ingestion-metrics/understanding-data-freshness-metrics-vs.-ingestion-latency.md).

# Understanding data freshness metrics vs. ingestion latency

Understand the difference between data freshness and ingestion latency in Cortex XDR.

The `data_freshness_*` fields in `dataset = metrics_source` provide high-level health signals to detect silent or disrupted data sources, but they do not measure isolated Cortex XDR processing latency.

### Key differences

* **End-to-end measurement** (`_insert_time − _time`): Freshness metrics calculate total elapsed time from when an event is stamped by the originating device (`_time`) to when it becomes searchable in Cortex XDR (`_insert_time`). This includes device clock drift (NTP skew), local buffering, network transit, and upstream services (such as Strata Logging Service), not solely Cortex XDR processing.
* **Aggregated buckets and maximums**: Delays are categorized into coarse duration buckets. Queries that aggregate maximums surface only the single worst 5-minute window or bucket ceiling, even when over 99% of logs arrived on time.
* **Precision and skews**: The `data_freshness_max_delay` metric shows whole numbers based on bucket ceilings. The `data_freshness_median` and `data_freshness_ninetieth_percentile` are statistical approximations shown as decimal numbers. Timezone differences between `_TIME` and `_INSERT_TIME` might cause negative differences, which are rounded to zero.

### How freshness buckets work

The recorded freshness delay value is the top value in the range of the bucket. If even a single log falls into a higher bucket, the `data_freshness_max_delay` will reflect that bucket ceiling:

* 0 to 30 seconds → 30 seconds
* 30 to 60 seconds → 60 seconds
* 60 seconds to 5 minutes → 300 seconds
* 5 minutes to 1 hour → 3,600 seconds
* 1 hour to 24 hours → 86,400 seconds
* 24 hours to week → 604,800 seconds

### What to run instead: Latency queries on raw datasets

To accurately evaluate ingestion latency and verify service levels, query the raw dataset of the data source (for example, `panw_ngfw_traffic_raw`, `panw_ngfw_threat_raw`, or other `<vendor>_<product>_raw` tables) using `_insert_time − _time`. Set the time range in the time picker (start with 24 hours).

#### **A. What share of logs arrived within X? (Best single view)**

This query calculates the exact latency for each event (`_insert_time − _time`) and groups logs into duration brackets:

```
| dataset = <dataset>_raw
| alter lag_sec = timestamp_diff(_insert_time, _time, "SECOND")
| alter lag_bucket = if(lag_sec <= 60, "1) under 1 min",
    if(lag_sec <= 300, "2) 1-5 min",
    if(lag_sec <= 1800, "3) 5-30 min",
    if(lag_sec <= 3600, "4) 30-60 min", "5) over 1 hour"))))
| comp count() as events by lag_bucket
| sort asc lag_bucket
```

Divide each row count by the total event count to calculate percentages (for example, "99.6% under 30 minutes").

#### **B. Median and p90 delay per hour, per device**

This query tracks delay trends per device in 1-hour intervals:

```
| dataset = <dataset>_raw
| alter lag_sec = timestamp_diff(_insert_time, _time, "SECOND")
| filter lag_sec >= 0
| bin _time span = 1h
| comp median(lag_sec) as p50_sec, approx_quantiles(lag_sec, 100) as q, count() as events by _time, _device_id
| alter p90_sec = arrayindex(q, 90)
| fields _time, _device_id, events, p50_sec, p90_sec
```

Display the results as a line chart with `_time` on the X-axis and `p50_sec` / `p90_sec` on the Y-axis. The overall trend matters, not an isolated spike.

#### **C. Who was late: one device or all of them?**

This query identifies which specific devices are reporting delayed logs (delayed by more than 30 minutes):

```
| dataset = <dataset>_raw
| alter lag_min = timestamp_diff(_insert_time, _time, "MINUTE")
| filter lag_min > 30
| comp count() as late_events, min(_time) as first_late, max(_time) as last_late by _device_id
| sort desc late_events
```

* **One or a few devices**: The delay is typically localized to customer-side factors such as device clock drift, local buffering, or network connectivity.
* **Many devices at the same time**: Indicates a shared upstream bottleneck, collector issue, or platform delay.

#### **D. Customer-side vs. after-collector split (only if `_collection_timestamp` is populated)**

For ingestion paths where the collector records `_collection_timestamp` (such as supported Broker VM paths), this query separates the delay occurring before the collector from the processing time after collection:

```
| dataset = <dataset>_raw
| filter _collection_timestamp != null
| alter coll_ts = to_timestamp(to_integer(_collection_timestamp), "MILLIS")
| alter before_collector_sec = timestamp_diff(coll_ts, _time, "SECOND"),
        after_collector_sec = timestamp_diff(_insert_time, coll_ts, "SECOND")
| comp median(before_collector_sec) as p50_before, median(after_collector_sec) as p50_after by _device_id
```

{% hint style="info" %}

#### Note

`_collection_timestamp` is only populated on raw datasets for supported collection mechanisms (it is not populated for XDR Collector). If no rows return, alter the query (for example, remove `_device_id`) or use queries A–C instead.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://cortex-docs.paloaltonetworks.com/cortex-xdr/configure-cortex-xdr/cortex-xdr-data-sources/administration-and-troubleshooting/overview-of-data-ingestion-metrics/understanding-data-freshness-metrics-vs.-ingestion-latency.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
