Overview of data ingestion metrics

Learn more about the data ingestion health metrics in the metrics_source dataset and the metrics_view preset.

Prerequisite

For Cortex XDR to monitor data ingestion health and create health issues, you must enable the following settings under Configurations:

• Cortex - Analytics: Go to Configurations → Cortex - Analytics. For more information, see Enable the Analytics Engine and Identity Analytics.
• Data Ingestion Monitoring: Go to Configurations → General → Server Settings → Data Ingestion Monitoring. For more information, see Set up your environment.

The data ingestion metrics are calculated in 5-minute aggregation periods and saved to the metrics_source dataset and metrics_view preset. These metrics measure the amount, size, and rate in which logs are ingested by a data source:

Metric	Description
total_size_bytes	Total size (in bytes) of the logs collected during the aggregation period.
total_size_rate	Average size (in bytes per second) of the logs collected during the aggregation period.
total_event_count	Total number of logs collected during the aggregation period
total_event_rate	Average number (in count per second) of logs collected during the aggregation period.

In the metrics_source dataset, the data ingestion metrics are saved alongside additional fields that describe the data source associated with the metrics. Only entries with ingestion metric values greater than zero are saved in the dataset. Entries with zero values are not saved in this dataset.

metrics_view is a preset for data in the metrics_source dataset. The preset also simulates completion of entries with zero values in data ingestion metrics at runtime, which allows effective use of metrics. Therefore, when investigating disruptions in data collection, we recommend using the metrics_view preset in XQL queries and correlation rules.

In addition, you can create your own custom logic for data ingestion health monitoring by setting up correlation rules that monitor the data ingestion metrics. For more information, see Creating correlation rules to monitor data ingestion health.

The following table describes all the fields in the metrics_source dataset and metrics_view preset:

Read more...

Field	Type	Description
total_size_bytes	Integer	Total size (in bytes) of the logs collected during the aggregation period.
total_size_rate	Integer	Average size (in bytes per second) of the logs collected during the aggregation period.
total_event_count	Integer	Total number of logs collected during the aggregation period.
total_event_rate	Integer	Average number (in count per second) of logs collected during the aggregation period.
data_freshness_max_delay	Float	Maximum delay value from all log entries in a record between log creation at the source and ingestion into Cortex XDR (in seconds).
data_freshness_median	Float	Median delay value from all log entries in a record between log creation at the source and ingestion into Cortex XDR (in seconds).
data_freshness_ninetieth_percentile	Float	Ninetieth percentile of delay values from all log entries in a record between log creation at the source and ingestion into Cortex XDR (in seconds).
last_seen	Datetime	Time that the last logs were collected.
_vendor	String	Vendor of the observing data source.
_product	String	Product name of the observing data source.
_device_id	String	(For firewall devices) Device ID
_log_type	String	(For firewall devices) Log type
_collector_name	String	(Event Metadata) Name of the collector instance.
_collector_id	String	(Event Metadata) ID of the XDR Collector.
_collector_ip	String	(Event Metadata) IP address of the XDR Collector.
_reporting_device_name	String	(Event Metadata) Host name of the device where the log originated.
_reporting_device_ip	String	(Event Metadata) IP Address of the device where the log originated.
_final_reporting_device_name	String	(Event Metadata) Hostname of the device that the log was extracted from.
_final_reporting_device_ip	String	(Event Metadata) IP of the device that the log was extracted from.
_broker_device_name	String	(Event Metadata) Host name of the Broker VM.
_broker_device_ip	String	(Event Metadata) IP address of the Broker VM.
_broker_device_id	String	(Event Metadata) ID of the Broker VM.
_time	Datetime	Timestamp of the interval.