> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/vendor-specific-data-sources/amazon/amazon-s3/configure-data-collection-from-amazon-s3-manually.md).

# Configure data collection from Amazon S3 manually

Collect data from Amazon S3 manually in Cortex XDR.

{% hint style="info" %}
**License**

Requires the Data Collection add-on.
{% endhint %}

There are various reasons why you may need to configure data collection from Amazon S3 manually, as opposed to using the CloudFormation Script provided in Cortex XDR. For example, if your organization does not use CloudFormation scripts, you will need to follow the instructions below, which explain at a high-level how to perform these steps manually with a link to the relevant topic in the Amazon S3 documentation with the detailed steps to follow.

As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 Cortex Query Language (XQL) dataset (`aws_s3_raw`). This enables you to search the logs with XQL Search using the dataset. For example queries, refer to the in-app XQL Library. For enhanced cloud protection, you can also configure Cortex XDR to ingest network flow logs as Cortex XDR network connection stories, which you can query with XQL Search using the `xdr_dataset` dataset with the preset called `network_story`. Cortex XDR can also generate Cortex XDR issues (Analytics, Correlations, IOC, and BIOC) when relevant from Amazon S3 logs. While Correlation Rules issues are generated on non-normalized and normalized logs, Analytics, IOC, and BIOC issues are only generated on normalized logs.

Enhanced cloud protection provides:

* Normalization of cloud logs
* Cloud logs stitching
* Enrichment with cloud data
* Detection based on cloud analytics
* Cloud-tailored investigations

Be sure you do the following tasks before you begin configuring data collection manually from Amazon CloudWatch to Amazon S3.

{% hint style="info" %}
**Note:** If you already have an Amazon S3 bucket configured with VPC flow logs that you want to use for this configuration, you do not need to perform the prerequisite steps detailed in the first two bullets.
{% endhint %}

* Ensure that you have at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS).
  * **Amazon S3 bucket**: `GetObject`
  * **SQS**: `ChangeMessageVisibility`, `ReceiveMessage`, `GetQueueAttributes`, and `DeleteMessage`.
* Create a dedicated Amazon S3 bucket for collecting network flow logs with the default settings. For more information, see [Creating a bucket using the Amazon S3 Console](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). ### Note It is your responsibility to define a retention policy for your Amazon S3 bucket by creating a Lifecycle rule in the Management tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.
* Ensure that you can access your Amazon Virtual Private Cloud (VPC) and have the necessary permissions to create flow logs.
* Determine how you want to provide access to Cortex XDR to your logs and perform API operations. You have the following options.
  * Use Workload Federated Identity to allow the Cortex XDR log collector service account to assume an IAM role in your AWS account with a short-lived, Google-signed OIDC token instead of stored AWS access keys. This is the **Workload Federated Identity** option in the Amazon S3 collection configuration, which is selected by default when available. For setup instructions, see [Create a Workload Federated Identity role](/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/vendor-specific-data-sources/amazon/amazon-s3/create-a-workload-federated-identity-role.md).
  * Designate an AWS IAM user, where you will need to know the Account ID for the user and have the relevant permissions to create an access key/id for the relevant IAM user. This is the **Access Key** option in the Amazon S3 collection configuration.
  * Create an assumed role in AWS to delegate permissions to a Cortex XDR AWS service. This role grants Cortex XDR access to your flow logs. For more information, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html). This is the **STS AssumeRole** option in the Amazon S3 collection configuration. For more information on creating an assumed role for Cortex XDR, see [Create an assumed role](/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/vendor-specific-data-sources/amazon/amazon-s3/create-an-assumed-role.md). To collect Amazon S3 logs that use server-side encryption (SSE), the user role must have an IAM policy that states that Cortex XDR has kms:Decrypt permissions. With this permission, Amazon S3 automatically detects if a bucket is encrypted and decrypts it. If you want to collect encrypted logs from different accounts, you must have the decrypt permissions for the user role also in the key policy for the master account Key Management Service (KMS). For more information, see [Allowing users in other accounts to use a KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying-external-accounts.html).
* If you use **Workload Federated Identity**, ensure that you have IAM permissions to create identity providers, policies, and roles in your AWS account. Google (`accounts.google.com`) must be registered as an OIDC identity provider in your AWS account with `sts.amazonaws.com` as an audience. If it isn't registered yet, you register it when you create the role. You also need the Cortex XDR service account identifier, which you copy from the Amazon S3 collection configuration in Cortex XDR before you save the configuration. To collect logs that use server-side encryption with AWS KMS, add a statement that allows `kms:Decrypt` on the key ARN to the permissions policy of the IAM role.

{% hint style="info" %}
**Note:** For resources such as Amazon S3, SQS, VPC flow logs, and event notifications, you may use the default configurations; yet, these settings remain fully customizable to meet your specific environment requirements.
{% endhint %}

Configure Cortex XDR to receive network flow logs from Amazon S3 manually.

1. Log in to the [AWS Management Console](https://console.aws.amazon.com/).
2. From the menu bar, ensure that you have selected the correct region for your configuration.
3. Configure your Amazon Virtual Private Cloud (VPC) with flow logs. For more information, see [AWS VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html).

   If you already have an Amazon S3 bucket configured with VPC flow logs, skip this step and go to Configure an Amazon Simple Queue Service (SQS).
4. Configure an Amazon Simple Queue Service (SQS). For more information, see [Configuring Amazon SQS queues (console)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configuring.html).

   Ensure that you create your Amazon S3 bucket and Amazon SQS queue in the same region.
5. Configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket. For more information, see [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html).
6. Configure access keys for the AWS IAM user that Cortex XDR uses for API operations. For more information, see [Managing access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
   * It is the responsibility of the customer’s organization to ensure that the user who performs this task of creating the access key is designated with the relevant permissions. Otherwise, this can cause the process to fail with errors.
   * Skip this step if you are using an Assumed Role or Workload Federated Identity for Cortex XDR.
7. Update the Access Policy of your SQS queue and grant the required permissions mentioned above to the relevant IAM user. For more information, see [Granting permissions to publish event notification messages to a destination](https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html).

   Skip this step if you are using an Assumed Role or Workload Federated Identity for Cortex XDR.
8. Configure the Amazon S3 collection in Cortex XDR.
   1. Navigate to **Settings → Data Sources & Integrations**.
   2. On the **Data Sources & Integrations** page, click **+ Add New**, search for Amazon S3, then hover over it and click **Add**.
   3. Select the authentication method you configured and enter the following values:

{% tabs %}
{% tab title="Workload Federated Identity" %}
Uses a short-lived OIDC token to assume an IAM role in your AWS account, without stored AWS access keys. Before you enter these values, set up the IAM role and its trust policy as described in [Create a Workload Federated Identity role](/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/vendor-specific-data-sources/amazon/amazon-s3/create-a-workload-federated-identity-role.md).

<table><thead><tr><th width="169.71484375">Field</th><th>Input</th></tr></thead><tbody><tr><td>SQS URL</td><td>Specify the <strong>SQS URL</strong>, which is the URL of the Amazon SQS that you configured in the AWS Management Console.</td></tr><tr><td>Name</td><td>Specify a descriptive name for your log collection configuration.</td></tr><tr><td>Role ARN</td><td>Specify the ARN of the IAM role that you created for Workload Federated Identity, in the format <code>arn:aws:iam::&#x3C;ACCOUNT_ID>:role/&#x3C;ROLE_NAME></code>. You can copy it from the role <strong>Summary</strong> page.</td></tr><tr><td>Audience</td><td>Enter <code>sts.amazonaws.com</code>. This value must match the <code>accounts.google.com:aud</code> condition in the IAM role trust policy.</td></tr><tr><td>Log Type</td><td>Select <strong>Flow Logs</strong> to configure your log collection to receive network flow logs from Amazon S3.</td></tr></tbody></table>

In the **Copy service account identifier** section, click **Identifier** to copy the Cortex XDR service account identifier. Use this value in the `accounts.google.com:sub` condition of the IAM role trust policy. The identifier is available as soon as you select **Workload Federated Identity**, so you can copy it before you create the IAM role.
{% endtab %}

{% tab title="Access Key" %}
Uses a designated AWS IAM user's static access key and secret to authenticate.

<table><thead><tr><th width="188.20703125">Field</th><th>Input</th></tr></thead><tbody><tr><td>SQS URL</td><td>Specify the <strong>SQS URL</strong>, which is the URL of the Amazon SQS that you configured in the AWS Management Console.</td></tr><tr><td>Name</td><td>Specify a descriptive name for your log collection configuration.</td></tr><tr><td>AWS Client ID</td><td>Specify the Access key ID, which you received when you configured access keys for the AWS IAM user in AWS.</td></tr><tr><td>AWS Client Secret</td><td>Specify the Secret access key you received when you configured access keys for the AWS IAM user in AWS.</td></tr><tr><td>Log Type</td><td>Select <strong>Flow Logs</strong> to configure your log collection to receive network flow logs from Amazon S3.</td></tr></tbody></table>
{% endtab %}

{% tab title="STS AssumeRole" %}
Delegates permissions to a Cortex XDR AWS service using an IAM assumed role. For more information, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html).

<table><thead><tr><th width="168.859375">Field</th><th>Input</th></tr></thead><tbody><tr><td>SQS URL</td><td>Specify the <strong>SQS URL</strong>, which is the URL of the Amazon SQS that you configured in the AWS Management Console.</td></tr><tr><td>Name</td><td>Specify a descriptive name for your log collection configuration.</td></tr><tr><td>Role ARN</td><td>Specify the Role ARN for the Assumed Role you created in AWS.</td></tr><tr><td>External ID</td><td>Specify the External ID for the Assumed Role you created in AWS.</td></tr><tr><td>Log Type</td><td>Select <strong>Flow Logs</strong> to configure your log collection to receive network flow logs from Amazon S3.</td></tr></tbody></table>
{% endtab %}
{% endtabs %}

9. Click **Test** to validate access, and then click **Enable**. For Workload Federated Identity, the **Test** button isn't available, so click **Enable**.

Once events start to come in, a green check mark appears underneath the Amazon S3 configuration with the number of logs received.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://cortex-docs.paloaltonetworks.com/cortex-xdr-5.x/configure-cortex-xdr/cortex-xdr-data-sources/vendor-specific-data-sources/amazon/amazon-s3/configure-data-collection-from-amazon-s3-manually.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.