Activate Apache Kafka Collector

Learn more about activating the Broker VM with an Apache Kafka Collector applet.

Notice

Ingesting logs and data from external sources requires a Cortex XDR Pro per GB license.

Note

This data source is only available in your tenant if the tenant was activated before October 1, 2025 with an active Cortex XDR Pro per GB license.

Apache Kafka is an open-source distributed event streaming platform for high-performance data pipelines, streaming analytics and data integration. Kafka records are organized into Topics. The partitions for each Topic are spread across the bootstrap servers in the Kafka cluster. The bootstrap servers are responsible for transferring data from Producers to Consumer Groups, which enable the Kafka server to save offsets of each partition in the Topic consumed by each group.

The Broker VM provides a Kafka Collector applet that enables you to monitor and collect events from Topics on self-managed on-prem Kafka clusters directly to your log repository for query and visualization purposes. The applet supports Kafka setups with no authentication, with SSL authentication, and SASL SSL authentication.

After you activate the Kafka Collector applet, you can collect events as datasets (<Vendor>_<Product>_raw) by defining the following.

• Kafka connection details including the Bootstrap Server List and Authentication Method.
• Topics Collection configuration for the Kafka topics that you want to collect.

Prerequisite

• Apache Kafka version 2.5.1 and above.
• Kafka cluster set up on premises, from which the data will be ingested.
• Privileges to manage Broker Service configuration, such as Instance Administrator privileges.
• Create a user in the Kafka cluster with the necessary permissions and the following authentication details:

  • Broker Certificate and Private Key for an SSL connection.
  • Username and Password for an SASL SSL connection.
• Set up and configure Broker VM

How to activate the Kafka Collector

1. Select Settings → Configurations → Data Broker → Broker VMs.
2. Do one of the following:

   • On the Brokers tab, find the Broker VM, and in the APPS column, left-click Add → Kafka Collector.
   • On the Clusters tab, find the Broker VM, and in the APPS column, left-click Add → Kafka Collector.
3. Configure the Kafka Connection.

   1. Specify the Bootstrap Server List, which is the <hostname/ip>:<port> of the bootstrap server (or servers). You can specify multiple servers, separated with a comma. For example, hostname1:9092,1.1.1.1:9092.
   2. Select one of the Authentication Methods below.
   3. Test Connection to verify that you can connect to the Kafka server.

No Authentication

Default connection method for a new Kafka setup, which doesn’t require authentication. With a standard Kafka setup, any user or application can write messages to any topic, as well as read data from any topic.

SSL Authentication

Authenticate your connection to Kafka using an SSL certificate. Use this authentication method when the connection to the Kafka server is a secure TCP, and upload the following:

• Broker Certificate: Signed certificate used for the applet to authenticate to the Kafka server.
• Private Key: Private key for the applet used for decrypting the SSL messages coming from the Kafka server.
• (Optional) CA Certificate: CA certificate that was used to sign the server and private certificates. This CA certificate is also used to authenticate the Kafka server identity.

SASL SSL (SCRAM-SHA-256)

Authenticate your connection to the Kafka server with your Username, Password, and optionally, your CA Certificate.

4. Configure the Topics Collection parameters.

Topic Subscription Method

Select the Topic Subscription Method for subscribing to Kafka topics. Use List Topics to specify a list of topics. Use Regex Pattern Matching to specify a regular expression to search available topics.

Topic(s)

Specify Topic(s) from the Kafka server. For the List Topics subscription method, use a comma separated list of topics to subscribe to. For the Regex Pattern Matching subscription method, use a regular expression to match the Topic(s) to subscribe to. We do not recommend mixing log/event types in a topic.

(optional) Consumer Group

Specify a Consumer Group, a unique string or label that identifies the consumer group this log source belongs to. Each record that is published to a Kafka topic is delivered to one consumer instance within each subscribing consumer group. Kafka uses these labels to load balance the records over all consumer instances in a group. When specified, the Kafka collector uses the given consumer group. When not specified, Cortex XDR assigns the Kafka applet collector to a new automatically generated consumer group which is automatically generated for this log source with the name PAN-<Broker VM device name>-<topic name>.

Log Format

Select the Log Format from the list as either RAW (default), JSON, CEF, LEEF, CISCO, or CORELIGHT. This setting defines the log type, which represents the type of logs the collector will receive from the configured Kafka topics.

Vendor and Product

Specify the Vendor and Product which will be associated with each entry in the dataset. The vendor and product are used to define the name of your Cortex Query Language (XQL) dataset (<Vendor>_<Product>_raw).

Note

For CEF and LEEF logs, Cortex XDR takes the vendor and product names from the log itself, regardless of what you configure on this page.

(optional) Add Query

Click Add Query to create another Topic Collection. Each topic can be added for a server only once.

(optional) Other available options for Topic Collection

As needed, you can manage your Topic Collection settings. Here are the actions available to you.

• Edit the Topics Collection details.
• Disable/Enable a Topics Collection by hovering over the top area of the Topics Collection section, on the opposite side of the Topics Collection name, and selecting the applicable button.
• Rename a Topics Collection by hovering over the top area of the Topics Collection section, on the opposite side of the Topics Collection name, and selecting the pen icon.
• Delete a Topics Collection by hovering over the top area of the Topics Collection section, on the opposite side of the Topics Collection name, and selecting the delete icon.

5. (Optional) Click Add Connection to create another Kafka Connection for collecting data.

1. (Optional) Other available options for Connections.

   As needed, you can return to your Kafka Collector settings to manage your connections.

   Here are the actions available to you.

   • Edit the Connection details.
   • Rename a connection by hovering over the default Collection name, and selecting the edit icon to edit the text.
   • Delete a connection by hovering over the top area of the connection section, on the opposite side of the connection name, and selecting the delete icon. You can only delete a connection when you have more than one connection configured. Otherwise, this icon is not displayed.
2. Activate the Kafka Collector applet. The Activate button is enabled when all the mandatory fields are filled in.

   After a successful activation, the APPS field displays Kafka with a green dot indicating a successful connection.
3. (Optional) To view metrics about the Kafka Collector, in the Broker VMs page, left-click the Kafka connection displayed in the APPS field for your Broker VM.

   Cortex XDR displays Resources, including the amount of CPU, Memory, and Disk space the applet is using.
4. Manage the Kafka Collector.

   After you activate the Kafka Collector, you can make additional changes as needed. To modify a configuration, left-click the Kafka connection in the APPS column to display the Kafka Collector settings, and select the following:

   • Configure to redefine the Kafka Collector configurations.
   • Deactivate to disable the Kafka Collector.

   Ensure that you Save your changes, which is enabled when all mandatory fields are filled in.