> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/generic-on-premise-data-collectors/broker-vm-data-collector-applets/activate-apache-kafka-collector.md). # Activate Apache Kafka Collector {% hint style="info" %} **License** * Requires a Data Collection add-on. * This data source is only available in your tenant if the tenant was activated before October 1, 2025 with an active Data Collection add-on. {% endhint %} Apache Kafka is an open-source distributed event streaming platform for high-performance data pipelines, streaming analytics and data integration. Kafka records are organized into Topics. The partitions for each Topic are spread across the bootstrap servers in the Kafka cluster. The bootstrap servers are responsible for transferring data from Producers to Consumer Groups, which enable the Kafka server to save offsets of each partition in the Topic consumed by each group. The Broker VM provides a Kafka Collector applet that enables you to monitor and collect events from Topics on self-managed on-prem Kafka clusters directly to your log repository for query and visualization purposes. The applet supports Kafka setups with no authentication, with SSL authentication, and SASL SSL authentication. After you activate the Kafka Collector applet, you can collect events as datasets (`__raw`) by defining the following. * Kafka connection details including the Bootstrap Server List and Authentication Method. * Topics Collection configuration for the Kafka topics that you want to collect. ### Prerequisite * Apache Kafka version 2.5.1 and above. * Kafka cluster set up on premises, from which the data will be ingested. * Privileges to manage Broker Service configuration, such as Instance Administrator privileges. * Create a user in the Kafka cluster with the necessary permissions and the following authentication details: * Broker Certificate and Private Key for an SSL connection. * Username and Password for an SASL SSL connection. * [Set up and configure Broker VM](/cortex-xdr-5.x/configure-cortex-xdr/data-management/broker-vm/set-up-and-configure-broker-vm.md) ### How to activate the Kafka Collector 1. Select Settings → Configurations → Data Broker → Broker VMs. 2. Do one of the following: * On the Brokers tab, find the Broker VM, and in the APPS column, left-click Add → Kafka Collector. * On the Clusters tab, find the Broker VM, and in the APPS column, left-click Add → Kafka Collector. 3. Configure the Kafka Connection. 1. Specify the Bootstrap Server List, which is the `:` of the bootstrap server (or servers). You can specify multiple servers, separated with a comma. For example, **`hostname1:9092,1.1.1.1:9092`**. 2. Select one of the Authentication Methods: * **No Authentication:** Default connection method for a new Kafka setup, which doesn’t require authentication. With a standard Kafka setup, any user or application can write messages to any topic, as well as read data from any topic. * **SSL Authentication:** Authenticate your connection to Kafka using an SSL certificate. Use this authentication method when the connection to the Kafka server is a secure TCP, and upload the following: * Broker Certificate: Signed certificate used for the applet to authenticate to the Kafka server. * Private Key: Private key for the applet used for decrypting the SSL messages coming from the Kafka server. * (Optional) CA Certificate: CA certificate that was used to sign the server and private certificates. This CA certificate is also used to authenticate the Kafka server identity. * **SASL SSL (SCRAM-SHA-256):** Authenticate your connection to the Kafka server with your Username, Password, and optionally, your CA Certificate. 3. Test Connection to verify that you can connect to the Kafka server. An error message is displayed for each server connection test that fails. 4. Configure the Topics Collection parameters. * **Topic Subscription Method** Select the Topic Subscription Method for subscribing to Kafka topics. Use List Topics to specify a list of topics. Use Regex Pattern Matching to specify a regular expression to search available topics. * **Topic(s):** Specify Topic(s) from the Kafka server. For the List Topics subscription method, use a comma separated list of topics to subscribe to. For the Regex Pattern Matching subscription method, use a regular expression to match the Topic(s) to subscribe to. We do not recommend mixing log/event types in a topic. * **(optional) Consumer Group:** Specify a Consumer Group, a unique string or label that identifies the consumer group this log source belongs to. Each record that is published to a Kafka topic is delivered to one consumer instance within each subscribing consumer group. Kafka uses these labels to load balance the records over all consumer instances in a group. When specified, the Kafka collector uses the given consumer group. When not specified, Cortex XDR assigns the Kafka applet collector to a new automatically generated consumer group which is automatically generated for this log source with the name `PAN--`. * **Log Format:** Select the Log Format from the list as either RAW (default), JSON, CEF, LEEF, CISCO, or CORELIGHT. This setting defines the log type, which represents the type of logs the collector will receive from the configured Kafka topics. * **Vendor and Product:** Specify the Vendor and Product which will be associated with each entry in the dataset. The vendor and product are used to define the name of your Cortex Query Language (XQL) dataset (`__raw`).

Note

For CEF and LEEF logs, Cortex XDR takes the vendor and product names from the log itself, regardless of what you configure on this page.

* **(optional) Add Query:** Click Add Query to create another Topic Collection. Each topic can be added for a server only once. * **(optional) Other available options for Topic Collection:** As needed, you can manage your Topic Collection settings. Here are the actions available to you. * Edit the Topics Collection details. * Disable/Enable a Topics Collection by hovering over the top area of the Topics Collection section, on the opposite side of the Topics Collection name, and selecting the applicable button. * Rename a Topics Collection by hovering over the top area of the Topics Collection section, on the opposite side of the Topics Collection name, and selecting the pen icon. * Delete a Topics Collection by hovering over the top area of the Topics Collection section, on the opposite side of the Topics Collection name, and selecting the delete icon. 5. (Optional) Click Add Connection to create another Kafka Connection for collecting data. 6. (Optional) Other available options for Connections. As needed, you can return to your Kafka Collector settings to manage your connections. Here are the actions available to you. * Edit the Connection details. * Rename a connection by hovering over the default Collection name, and selecting the edit icon to edit the text. * Delete a connection by hovering over the top area of the connection section, on the opposite side of the connection name, and selecting the delete icon. You can only delete a connection when you have more than one connection configured. Otherwise, this icon is not displayed. 7. Activate the Kafka Collector applet. The Activate button is enabled when all the mandatory fields are filled in. After a successful activation, the APPS field displays Kafka with a green dot indicating a successful connection. 8. (Optional) To view metrics about the Kafka Collector, in the Broker VMs page, left-click the Kafka connection displayed in the APPS field for your Broker VM.Cortex XDR displays Resources, including the amount of CPU, Memory, and Disk space the applet is using. 9. Manage the Kafka Collector. After you activate the Kafka Collector, you can make additional changes as needed. To modify a configuration, left-click the Kafka connection in the APPS column to display the Kafka Collector settings, and select the following: * Configure to redefine the Kafka Collector configurations. * Deactivate to disable the Kafka Collector. Ensure that you Save your changes, which is enabled when all mandatory fields are filled in. You can also ingest Apache Kafka events as datasets. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/generic-on-premise-data-collectors/broker-vm-data-collector-applets/activate-apache-kafka-collector.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.