> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/vendor-specific-data-sources/google/google-cloud-platform/ingest-logs-and-data-from-a-gcp-pub-sub.md). # Ingest logs and data from a GCP Pub/Sub {% hint style="info" %} **License** Requires the Data Collection add-on. {% endhint %} If you use the Pub/Sub messaging service from Global Cloud Platform (GCP), you can send logs and data from your GCP instance to Cortex XDR. Data from GCP is then searchable in Cortex XDR to provide additional information and context to your investigations using the GCP Cortex Query Language (XQL) dataset, which is dependent on the type of GCP logs collected. For example queries, refer to the in-app XQL Library. You can configure a Google Cloud Platform collector to receive generic, flow, audit, or Google Cloud DNS logs. When configuring generic logs, you can receive logs in a Raw, JSON, CEF, LEEF, Cisco, or Corelight format. You can also configure Cortex XDR to normalize different GCP logs as part of the enhanced cloud protection, which you can query with XQL Search using the applicable dataset. Cortex XDR can also generate Cortex XDR issues (Analytics, IOC, BIOC, and Correlation Rules), when relevant, from GCP logs. While Correlation Rules isssues are generated on non-normalized and normalized logs, Analytics, IOC, and BIOC issues are only raised on normalized logs. Enhanced cloud protection provides the following: * Normalization of cloud logs * Cloud logs stitching * Enrichment with cloud data * Detection based on cloud analytics * Cloud-tailored investigations The following table lists the various GCP log types the XQL datasets you can use to query in XQL Search: | GCP log type | Dataset | Dataset with normalized data | | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Audit logs, including Google Kubernetes Engine (GKE) audit logs | `google_cloud_logging_raw` | `cloud_audit_logs` | | Generic logs |

Log Format types:

| N/A | | Google Cloud DNS logs | `google_dns_raw` | `xdr_data`: Once configured, Cortex XDR ingests Google Cloud DNS logs as XDR network connection stories, which you can query with XQL Search using the `xdr_data` dataset with the preset called `network_story`. | | Network flow logs | `google_cloud_logging_raw` | `xdr_data`: Once configured, Cortex XDR ingests network flow logs as XDR network connection stories, which you can query with XQL Search using the `xdr_data` dataset with the preset called `network_story`. | {% hint style="info" %} **Note** When collecting flow logs, we recommend that you include GKE annotations in your logs, which enable you to view the names of the containers that communicated with each other. GKE annotations are only included in logs if appended manually using the custom metadata configuration in GCP. For more information, see [VPC Flow Logs Overview](https://cloud.google.com/vpc/docs/flow-logs#customizing_metadata_fields). In addition, to customize metadata fields, you must use the gcloud command-line interface or the API. For more information, see [Using VPC Flow Logs](https://cloud.google.com/vpc/docs/using-flow-logs#enabling_vpc_flow_logging_for_an_existing_subnet). {% endhint %} To receive logs and data from GCP, you must first set up log forwarding using a Pub/Sub topic in GCP. You can configure GCP settings using either the GCP web interface or a GCP cloud shell terminal. After you set up your service account in GCP, you configure the Data Collection settings in Cortex XDR. The setup process requires the subscription name and authentication key from your GCP instance. After you set up log collection, Cortex XDR immediately begins receiving new logs and data from GCP. ### Set up log forwarding using the GCP web interface * In Cortex XDR, set up Data Collection. a. Navigate to **Settings → Data Sources & Integrations**. b. On the **Data Sources & Integrations** page, click **+ Add New**, search for **Google Cloud Platform**, then hover over it and click **Add**. c. Specify the **Subscription Name** that you previously noted or copied. d. Browse to the **JSON** file containing your authentication key for the service account. e. Select the **Log Type** as one of the following, where your selection changes the options displayed. * **Flow or Audit Logs:** When selecting this log type, you can decide whether to normalize and enrich the logs as part of the enhanced cloud protection. * **(Optional)** You can Normalize and enrich flow and audit logs by selecting the checkbox (default). If selected, Cortex XDR ingests the network flow logs as Cortex XDR network connection stories, which you can query using XQL Search from the `xdr_dataset` dataset with the preset called `network_story`. In addition, you can configure Cortex XDR to normalize GCP audit logs, which you can query with XQL Search using the `cloud_audit_logs` dataset. * The **Vendor** is automatically set to **Google** and **Product** to **Cloud Logging**, which is not configurable. This means that all GCP data for the flow and audit logs, whether it's normalized or not, can be queried in XQL Search using the `google_cloud_logging_raw` dataset. * **Generic:** When selecting this log type, you can configure the following settings. * **Log Format:** Select the log format type as **Raw, JSON, CEF, LEEF, Cisco, or Corelight.** * **CEF or LEEF:** The **Vendor** and **Product** defaults to **Auto-Detect.**

Note

For a Log Format set to CEF or LEEF, Cortex XDR reads events row by row to look for the Vendor and Product configured in the logs. When the values are populated in the event log row, Cortex XDR uses these values even if you specified a value in the Vendor and Product fields in the GCP data collector settings. Yet, when the values are blank in the event log row, Cortex XDR uses the Vendor and Product that you specified in the GCP data collector settings. If you did not specify a Vendor or Product in the GCP data collector settings, and the values are blank in the event log row, the values for both fields are set to unknown.

* **Cisco:** The following fields are automatically set and not configurable. * **Vendor:** Cisco * **Product:** ASA Cisco data can be queried in XQL Search using the `cisco_asa_raw` dataset. * **Corelight:** The following fields are automatically set and not configurable. * **Vendor:** Corelight * **Product:** Zeek Corelight data can be queried in XQL Search using the `corelight_zeek_raw` dataset. * **Raw or JSON:** The following fields are automatically set and are configurable. * **Vendor:** Google * **Product:** Cloud Logging Raw or JSON data can be queried in XQL Search using the `google_cloud_logging_raw` dataset.Cortex XDR supports logs in single line format or multiline format. For a JSON format, multiline logs are collected automatically when the Log Format is configured as JSON. When configuring a Raw format, you must also define the Multiline Parsing Regex as explained below. * **Vendor:** (Optional) Specify a particular vendor name for the GCP generic data collection, which is used in the GCP XQL dataset `__raw` that Cortex XDR creates as soon as it begins receiving logs. * **Product:** (Optional) Specify a particular product name for the GCP generic data collection, which is used in the GCP XQL dataset name `__raw` that Cortex XDR creates as soon as it begins receiving logs. * **Multiline Parsing Regex:** (Optional) This option is only displayed when the Log Format is set to Raw, where you can set the regular expression that identifies when the multiline event starts in logs with multilines. It is assumed that when a new event begins, the previous one has ended. * **Google Cloud DNS:** When selecting this log type, you can configure whether to normalize the logs as part of the enhanced cloud protection. * **Optional)** You can Normalize DNS logs by selecting the checkbox (default). If selected, Cortex XDR ingests the Google Cloud DNS logs as Cortex XDR network connection stories, which you can query using XQL Search from the `xdr_dataset` dataset with the preset called `network_story`. * The **Vendor** is automatically set to **Google** and **Product** to **DNS** , which is not configurable. This means that all Google Cloud DNS logs, whether it's normalized or not, can be queried in XQL Search using the `google_dns_raw` dataset. f. Test the provided settings and, if successful, proceed to **Enable** log collection. 1. Log in to your GCP account. 2. Set up log forwarding from GCP to Cortex XDR. a. Select **Logging → Logs Router**. b. Select **Create Sink → Cloud Pub/Sub topic**, and then click **Next**. c. To filter only specific types of data, select the filter or desired resource. d. In the **Edit Sink** configuration, define a descriptive **Sink Name**. e. Select **Sink Destination → Create new Cloud Pub/Sub topic**. f. Enter a descriptive **Name** that identifies the sink purpose for Cortex XDR, and then **Create**. g. **Create Sink** and then **Close** when finished. 3. Create a subscription for your Pub/Sub topic. a. Select the menu icon in G Cloud, and then select **Pub/Sub → Topics**. b. Select the name of the topic you created in the previous steps. Use the filters if necessary. c. Select **Create Subscription → Create subscription**. d. Enter a unique **Subscription ID**. e. Choose **Pull** as the **Delivery Type**. f. Create the subscription. After the subscription is set up, G Cloud displays statistics and settings for the service. g. In the subscription details, identify and note your **Subscription Name**. Optionally, use the copy button to copy the name to the clipboard. You will need the name when you configure Collection in Cortex XDR. 4. Create a service account and authentication key. You will use the key to enable Cortex XDR to authenticate with the subscription service. a. Select the menu icon, and then select **IAM & Admin → Service Accounts**. b. **Create Service Account**. c. Enter a **Service account name** and then **Create**. d. Select a role for the account: **Pub/Sub → Pub/Sub Subscriber**. e. Click **Continue → Done**. f. Locate the service account by name, using the filters to refine the results, if needed. g. Click the **Actions** menu identified by the three dots in the row for the service account and then **Create Key**. h. Select **JSON** as the key type, and then **Create**. After you create the service account key, G Cloud automatically downloads it. 5. After Cortex XDR begins receiving information from the GCP Pub/Sub service, you can use the XQL Query language to search for specific data. {% hint style="info" %} **Note** If you encounter errors while enabling this integration due to VPC Service Controls (VPC SC) in your Google Cloud organization, configure the required ingress rules. For detailed technical guidance, contact our [Customer Support team](https://support.paloaltonetworks.com/Support/Index). {% endhint %} ### Set up log forwarding using the GCP cloud shell terminal In Cortex XDR, set up Data Collection. 1. Navigate to **Settings → Data Sources & Integrations**. 2. On the **Data Sources & Integrations** page, click **+ Add New**, search for **Google Cloud Platform**, then hover over it and click **Add**. 3. Specify the **Subscription Name** that you previously noted or copied. 4. Browse to the **JSON** file containing your authentication key for the service account. 5. Select the **Log Type** as one of the following, where your selection changes the options displayed. * **Flow or Audit Logs:** When selecting this log type, you can decide whether to normalize and enrich the logs as part of the enhanced cloud protection. * **(Optional)** You can Normalize and enrich flow and audit logs by selecting the checkbox (default). If selected, Cortex XDR ingests the network flow logs as Cortex XDR network connection stories, which you can query using XQL Search from the `xdr_dataset` dataset with the preset called `network_story`. In addition, you can configure Cortex XDR to normalize GCP audit logs, which you can query with XQL Search using the `cloud_audit_logs` dataset. * The **Vendor** is automatically set to **Google** and **Product** to **Cloud Logging**, which is not configurable. This means that all GCP data for the flow and audit logs, whether it's normalized or not, can be queried in XQL Search using the `google_cloud_logging_raw` dataset. * **Generic:** When selecting this log type, you can configure the following settings. * **Log Format:** Select the log format type as **Raw, JSON, CEF, LEEF, Cisco, or Corelight.** * **CEF or LEEF:** The **Vendor** and **Product** defaults to **Auto-Detect.** For a Log Format set to CEF or LEEF, Cortex XDR reads events row by row to look for the Vendor and Product configured in the logs. When the values are populated in the event log row, Cortex XDR uses these values even if you specified a value in the Vendor and Product fields in the GCP data collector settings. Yet, when the values are blank in the event log row, Cortex XDR uses the Vendor and Product that you specified in the GCP data collector settings. If you did not specify a Vendor or Product in the GCP data collector settings, and the values are blank in the event log row, the values for both fields are set to unknown. * **Cisco:** The following fields are automatically set and not configurable. * **Vendor:** Cisco * **Product:** ASA Cisco data can be queried in XQL Search using the `cisco_asa_raw` dataset. * **Corelight:** The following fields are automatically set and not configurable. * **Vendor:** Corelight * **Product:** Zeek Corelight data can be queried in XQL Search using the `corelight_zeek_raw` dataset. * **Raw or JSON:** The following fields are automatically set and are configurable. * **Vendor:** Google * **Product:** Cloud Logging Raw or JSON data can be queried in XQL Search using the `google_cloud_logging_raw` dataset. Cortex XDR supports logs in single line format or multiline format. For a JSON format, multiline logs are collected automatically when the Log Format is configured as JSON. When configuring a Raw format, you must also define the Multiline Parsing Regex as explained below. * **Vendor:** (Optional) Specify a particular vendor name for the GCP generic data collection, which is used in the GCP XQL dataset `__raw` that Cortex XDR creates as soon as it begins receiving logs. * **Product:** (Optional) Specify a particular product name for the GCP generic data collection, which is used in the GCP XQL dataset name `__raw` that Cortex XDR creates as soon as it begins receiving logs. * **Multiline Parsing Regex:** (Optional) This option is only displayed when the Log Format is set to Raw, where you can set the regular expression that identifies when the multiline event starts in logs with multilines. It is assumed that when a new event begins, the previous one has ended. * **Google Cloud DNS:** When selecting this log type, you can configure whether to normalize the logs as part of the enhanced cloud protection. * Optional) You can Normalize DNS logs by selecting the checkbox (default). If selected, Cortex XDR ingests the Google Cloud DNS logs as Cortex XDR network connection stories, which you can query using XQL Search from the `xdr_dataset` dataset with the preset called `network_story`. * The **Vendor** is automatically set to **Google** and **Product** to **DNS** , which is not configurable. This means that all Google Cloud DNS logs, whether it's normalized or not, can be queried in XQL Search using the `google_dns_raw` dataset. 6. Test the provided settings and, if successful, proceed to **Enable** log collection. 7. Launch the GCP cloud shell terminal or use your preferred shell with gcloud installed. [![gcp-cli.png](https://docs-cortex.paloaltonetworks.com/api/khub/maps/GD6sG6FlxDWxAn13_eZuUQ/resources/bZCGuan1equcIHdGD~icMA-GD6sG6FlxDWxAn13_eZuUQ/content?v=ef2817efc42b48d1\&Ft-Calling-App=ft/turnkey-portal)](https://docs-cortex.paloaltonetworks.com/viewer/attachment/GD6sG6FlxDWxAn13_eZuUQ/bZCGuan1equcIHdGD~icMA-GD6sG6FlxDWxAn13_eZuUQ) 8. Define your project ID. ``` gcloud config set project ``` 9. Create a Pub/Sub topic. ``` gcloud pubsub topics create ``` 10. Create a subscription for this topic. ``` gcloud pubsub subscriptions create --topic= ``` Note the subscription name you define in this step as you will need it to set up log ingestion from Cortex XDR. 11. Create a logging sink. During the logging sink creation, you can also define additional log filters to exclude specific logs. To filter logs, supply the optional parameter `--log-filter=` ``` gcloud logging sinks create pubsub.googleapis.com/projects//topics/ --log-filter= ``` If setup is successful, the console displays a summary of your log sink settings: ``` Created [https://logging.googleapis.com/v2/projects/PROJECT_ID/sinks/SINK_NAME]. Please remember to grant `serviceAccount:LOGS_SINK_SERVICE_ACCOUNT` \ the Pub/Sub Publisher role on the topic. More information about sinks can be found at /logging/docs/export/configure_export ``` 12. Grant log sink service account to publish to the new topic. Note the `serviceAccount` name from the previous step and use it to define the service for which you want to grant publish access. ``` gcloud pubsub topics add-iam-policy-binding --member serviceAccount: --role=roles/pubsub.publisher ``` 13. Create a service account. For example, use cortex-xdr-sa as the service account name and Cortex XDR Service Account as the display name. ``` gcloud iam service-accounts create --description="" --display-name="" ``` 14. Grant the IAM role to the service account. ``` gcloud pubsub subscriptions add-iam-policy-binding --member serviceAccount:@.iam.gserviceaccount.com --role=roles/pubsub.subscriber ``` 15. Create a JSON key for the service account. You will need the JSON file to enable Cortex XDR to authenticate with the GCP service. Specify the file destination and filename using a .json extension. ``` gcloud iam service-accounts keys create --iam-account @.iam.gserviceaccount.com ``` 16. After Cortex XDR begins receiving information from the GCP Pub/Sub service, you can use the XQL Query language to search for specific data. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/vendor-specific-data-sources/google/google-cloud-platform/ingest-logs-and-data-from-a-gcp-pub-sub.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.