> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/data-ingestion/external-data-ingestion/ingest-network-connection-logs/ingest-network-flow-logs-from-amazon-s3.md). # Ingest network flow logs from Amazon S3 {% hint style="info" %} ### Notice Ingestion of logs and data requires a Cortex XDR Pro per GB license. {% endhint %} You can forward network flow logs to Cortex XDR from Amazon Simple Storage Service (Amazon S3). To receive network flow logs from Amazon S3, you must first configure data collection from Amazon S3. You can then configure the Collection Integrations settings in Cortex XDR for Amazon S3. After you set up collection integration, Cortex XDR begins receiving new logs and data from the source. You can either configure Amazon S3 with SQS notification manually on your own or use the AWS CloudFormation Script that we have created for you to make the process easier. The instructions below explain how to configure Cortex XDR to receive network flow logs from Amazon S3 using SQS. To perform these steps manually, see Configure Data Collection from Amazon S3 Manually. {% hint style="info" %} ### Note For more information on configuring data collection from Amazon S3, see the Amazon S3 Documentation. {% endhint %} As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 Cortex Query Language (XQL) dataset (`aws_s3_raw`). This enables you to search the logs with XQL Search using the dataset. For example, queries refer to the in-app XQL Library. For enhanced cloud protection, you can also configure Cortex XDR to ingest network flow logs as Cortex XDR network connection stories, which you can query with XQL Search using the `xdr_data` dataset with the preset called `network_story`. Cortex XDR can also raise Cortex XDR alerts (Analytics, Correlation Rules, IOC, and BIOC) when relevant from Amazon S3 logs. While Correlation Rules alerts are raised on non-normalized and normalized logs, Analytics, IOC, and BIOC alerts are only raised on normalized logs. Enhanced cloud protection provides the following: * Normalization of cloud logs * Cloud logs stitching * Enrichment with cloud data * Detection based on cloud analytics * Cloud-tailored investigations Be sure you do the following tasks before you begin configuring data collection from Amazon S3 using the AWS CloudFormation Script. * Ensure that you have the proper permissions to run AWS CloudFormation with the script provided in Cortex XDR. You need at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS): * **Amazon S3 bucket**: `GetObject` * **SQS**: `ChangeMessageVisibility`, `ReceiveMessage`, and `DeleteMessage`. * Ensure that you can access your Amazon Virtual Private Cloud (VPC) and have the necessary permissions to create flow logs. * Determine how you want to provide access to Cortex XDR to your logs and perform API operations. You have the following options: * Designate an AWS IAM user, where you will need to know the Account ID for the user and have the relevant permissions to create an access key/id for the relevant IAM user. This is the default option as explained in Configure the Amazon S3 Collection in Cortex XDR by selecting **Access Key**. * Create an assumed role in AWS to delegate permissions to a Cortex XDR AWS service. This role grants Cortex XDR access to your flow logs. For more information, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html). This is the **Assumed Role** option as described in the Configure the Amazon S3 collection in Cortex XDR. For more information on creating an assumed role for Cortex XDR, see [Create an assumed role](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/data-ingestion/external-data-ingestion/ingest-network-connection-logs/ingest-network-flow-logs-from-amazon-s3/create-an-assumed-role.md). To collect Amazon S3 logs that use server-side encryption (SSE), the user role must have an IAM policy that states that Cortex XDR has kms:Decrypt permissions. With this permission, Amazon S3 automatically detects if a bucket is encrypted and decrypts it. If you want to collect encrypted logs from different accounts, you must have the decrypt permissions for the user role also in the key policy for the master account Key Management Service (KMS). For more information, see [Allowing users in other accounts to use a KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying-external-accounts.html). Configure Cortex XDR to receive network flow logs from Amazon S3 using the CloudFormation Script. 1. Download the CloudFormation Script in Cortex XDR . 1. Select **Settings** → **Configurations** → **Data Collection** → **Collection Integrations**. 2. In the **Amazon S3** configuration, click **Add Instance**. 3. To provide access to Cortex XDR to your logs and to perform API operations using a designated AWS IAM user, leave the **Access Key** option selected. Otherwise, select **Assumed Role**, and ensure that you Create an Assumed Role for before continuing with these instructions. 4. For the **Log Type**, select **Flow Logs** to configure your log collection to receive network flow logs from Amazon S3, and the following text is displayed under the field **Download CloudFormation Script. See instructions here.** 5. Click the **Download CloudFormation Script.** link to download the script to your computer. 2. Create a new Stack in the CloudFormation Console with the script you downloaded from Cortex XDR. For more information on creating a Stack, see [Creating a stack on the AWS CloudFormation console](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html). 1. Log in to the [CloudFormation Console](https://console.aws.amazon.com/cloudformation/). 2. From the CloudFormation → **Stacks** page, ensure that you have selected the correct region for your configuration. 3. Select Create Stack → **With new resources (standard)**. 4. Specify the template that you want AWS CloudFormation to use to create your stack. This template is the script that you downloaded from Cortex XDR , which will create an Amazon S3 bucket, Amazon Simple Queue Service (SQS) queue, and Queue Policy. Configure the following settings in the **Specify template** page. * **Prerequisite - Prepare template** → **Prepare template**: Select **Template is ready**. * **Specify Template** * **Template source**: Select **Upload a template file**. * **Upload a template file**: **Choose file**, and select the `cortex-xdr-create-s3-with-sqs-flow-logs.json` file that you downloaded from Cortex XDR. ![create-stack.png](/files/3eg2f4xIUOsgT5qvrhJo) 5. Click **Next**. 6. In the **Specify stack details** page, configure the following stack details. * **Stack name**: Specify a descriptive name for your stack. * Parameters → **Cortex XDR Flow Logs Integration** * **Bucket Name**: Specify the name of the S3 bucket to create, where you can leave the default populated name as **xdr-flow-logs** or create a new one. The name must be unique. * **Publisher Account ID**: Specify the AWS IAM user account ID with whom you are sharing access. * **Queue Name**: Specify the name for your Amazon SQS queue to create, where you can leave the default populated name as **xdr-flow** or create a new one. The name must be unique. ![specify-stack-details.png](/files/43rk4a3GLNTWxdaG9s7t) 7. Click **Next**. 8. In the **Configure stack options** page, there is nothing to configure, so click **Next**. 9. In the **Review** page, look over the stack configurations settings that you have configured and if they are correct, click **Create stack**. If you need to make a change, click **Edit** beside the particular step that you want to update. The stack is created and is opened with the **Events** tab displayed. It can take a few minutes for the new Amazon S3 bucket, SQS queue, and Queue Policy to be created. Click **Refresh** to get updates. Once everything is created, leave the stack opened in the current browser, because you will need to access information in the stack for other steps detailed below.

Note

For the Amazon S3 bucket created using CloudFormation, it is the customer’s responsibility to define a retention policy by creating a Lifecycle rule in the Management tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.

3. Configure your Amazon Virtual Private Cloud (VPC) with flow logs: 1. Open the [Amazon VPC Console](https://console.aws.amazon.com/vpc/.), and in the **Resources by Region** listed, select **VPCs** to view the VPCs configured for the current region selected. To select another VPC from another region, select **See all regions**, and select one of them.

Note

To create a new VPC, click Launch VPC Wizard. For more information, see AWS VPC Flow Logs.

2. From the list of **Your VPCs**, select the checkbox beside the VPC that you want to configure to create flow logs, and then select Actions → **Create flow log**. ![your-vpcs.png](/files/ElVoWRFZyFnkdRAedziL) 3. Configure the following **Flow log settings**: * **Name - optional**: (Optional) Specify a descriptive name for your VPC flow log. * **Filter**: Select **All** types of traffic to capture. * **Maximum aggregation interval**: If you anticipate a heavy flow of traffic, select **1 minute**. Otherwise, leave the default setting as **10 minutes**. * **Destination**: Select **Send to an Amazon S3 bucket** as the destination to publish the flow log data. * **S3 bucket ARN**:Specify the Amazon Resource Name (ARN) for your Amazon S3 bucket. You can retrieve your bucket’s ARN by opening another instance of the AWS Management Console in a browser window and opening the [Amazon S3 console](https://console.aws.amazon.com/s3/). In the **Buckets** section, select the bucket that you created for collecting the Amazon S3 flow logs when you created your stack, click **Copy ARN**, and paste the ARN in this field. ![bucket-copy-arn.png](/files/DB0VxSjG81tuI2MtKeR2) * **Log record format**: Select **Custom Format**, and in the **Log Format** field, specify the following fields to include in the flow log record, which you can select from the list displayed: * **account-id** * **action** * **az-id** * **bytes** * **dstaddr** * **dstport** * **end** * **flow-direction** * **instance-id** * **interface-id** * **packets** * **log-status** * **pkt-srcaddr** * **pkt-dstaddr** * **protocol** * **region** * **srcaddr** * **srcport** * **start** * **sublocation-id** * **sublocation-type** * **subnet-id** * **tcp-flags** * **type** * **vpc-id** * **version** 4. Click **Create flow log**. Once the flow log is created, a message indicating that the flow log was successfully created is displayed at the top of the **Your VPCs** page. In addition, if you open your Amazon S3 bucket configurations, by selecting the bucket from the [Amazon S3 console](https://console.aws.amazon.com/s3/), the **Objects** tab contains a folder called `AWSLogs/` to collect the flow logs. 4. Configure access keys for the AWS IAM user that Cortex XDR uses for API operations.

Note

1. Open the [AWS IAM Console](https://console.aws.amazon.com/iam/), and in the navigation pane, select **Access management** → **Users**. 2. Select the **User name** of the AWS IAM user. 3. Select the **Security credentials** tab, scroll down to the **Access keys** section, and click **Create access key**. 4. Click the copy icon next to the **Access key ID** and **Secret access key** keys, where you must click **Show secret access key** to see the secret key and record them somewhere safe before closing the window. You will need to provide these keys when you edit the Access policy of the SQS queue and when setting the **AWS Client ID** and **AWS Client Secret** in Cortex XDR. If you forget to record the keys and close the window, you will need to generate new keys and repeat this process.

Note

For more information, see Managing access keys for IAM users.

5. When you create an Assumed Role in Cortex XDR, ensure that you edit the policy that defines the permissions for the role with the S3 Bucket ARN and **SQS ARN**, which is taken from the Stack you created.

Note

Skip this step if you are using an Access Key to provide access to Cortex XDR.

6. Configure the Amazon S3 collection in Cortex XDR. 1. Select **Settings** → **Configurations** → **Data Collection** → **Collection Integrations**. 2. In the **Amazon S3** configuration, click **Add Instance** to begin a new configuration. 3. Set these parameters, where the parameters change depending on whether you configured an **Access Key** or **Assumed Role**. * **SQS URL**: Specify the **SQS URL**, which is taken from the stack you created. In the browser you left open after creating the stack, open the **Outputs** tab, copy the **Value** of the **QueueURL** and paste it in this field. * **Name**: Specify a descriptive name for your log collection configuration. * When setting an **Access Key**, set these parameters. * **AWS Client ID**: Specify the **Access key ID**, which you received when you created access keys for the AWS IAM user in AWS. * **AWS Client Secret**: Specify the **Secret access key** you received when you created access keys for the AWS IAM user in AWS. * When setting an **Assumed Role**, set these parameters. * **Role ARN**: Specify the **Role ARN** for the Assumed Role you created for in AWS. * **External Id**:Specify the **External Id** for the Assumed Role you created for in AWS. * **Log Type**: Select **Flow Logs** to configure your log collection to receive network flow logs from Amazon S3. When configuring network flow log collection, the following additional field is displayed for **Enhanced Cloud Protection**. You can **Normalize and enrich flow logs** by selecting the checkbox. If selected, Cortex XDR ingests the network flow logs as XDR network connection stories, which you can query using XQL Search from the `xdr_data` dataset using the preset called `network_story`. 4. Click **Test** to validate access, and then click **Enable**. Once events start to come in, a green check mark appears underneath the **Amazon S3** configuration with the number of logs received. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/data-ingestion/external-data-ingestion/ingest-network-connection-logs/ingest-network-flow-logs-from-amazon-s3.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.