> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/data-ingestion/external-data-ingestion/ingest-operation-and-system-logs-from-cloud-providers/ingest-generic-logs-from-amazon-s3.md). # Ingest generic logs from Amazon S3 {% hint style="info" %} **LICENSE TYPE:** Ingestion of logs and data requires a Cortex XDR Pro per GB license. {% endhint %} You can forward generic logs for the relative service to Cortex XDR from Amazon S3. To receive generic data from Amazon Simple Storage Service (Amazon S3), you must first configure data collection from Amazon S3. You can then configure the Collection Integrations settings in Cortex XDR for Amazon S3. After you set up collection integration, Cortex XDR begins receiving new logs and data from the source. {% hint style="info" %} **Note:** For more information on configuring data collection from Amazon S3, see the Amazon S3 Documentation. {% endhint %} As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 Cortex Query Language (XQL) dataset (`__raw`). This enables you to search the logs using XQL Search with the dataset. For example queries, refer to the in-app XQL Library. Cortex XDR can also raise Cortex XDR alerts (Correlation Rules only) when relevant from Amazon S3 logs. {% hint style="info" %} **Note:** You need to set up an Amazon S3 data collector to receive generic logs when collecting logs from BeyondTrust Privilege Management Cloud. For more information, see [Ingest logs from BeyondTrust Privilege Management Cloud.](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/data-ingestion/external-data-ingestion/additional-log-ingestion-methods/ingest-logs-from-beyondtrust-privilege-management-cloud.md) {% endhint %} ### Prerequisites Perform the following tasks before you begin configuring data collection from Amazon S3: * Create a dedicated Amazon S3 bucket, which collects the generic logs that you want capture. For more information, see [Creating a bucket using the Amazon S3 Console](#prerequisites).

Note:

It is the customer’s responsibility to define a retention policy for your Amazon S3 bucket by creating a Lifecycle rule in the Management tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.

* The logs collected by your dedicated Amazon S3 bucket must adhere to the following guidelines. * Each log file must use the 1 log per line format as multi-line format is not supported. * The log format must be compressed as gzip or uncompressed. * For best performance, we recommend limiting each file size to up to 50 MB (compressed). * Ensure that you have at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS). * **Amazon S3 bucket**: `GetObject` * **SQS**: `ChangeMessageVisibility`, `ReceiveMessage`, and `DeleteMessage`. * Determine how you want to provide access to Cortex XDR to your logs and perform API operations. You have the following options: * Designate an AWS IAM user, where you will need to know the Account ID for the user and have the relevant permissions to create an access key/id for the relevant IAM user. * Create an assumed role in AWS to delegate permissions to a Cortex XDR AWS service. This role grants Cortex XDR access to your flow logs. For more information, see [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html). This is the **Assumed Role** option described in the configure the Amazon S3 collection in Cortex XDR. For more information on creating an assumed role for Cortex XDR, see [Create an assumed role](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/data-ingestion/external-data-ingestion/ingest-network-connection-logs/ingest-network-flow-logs-from-amazon-s3/create-an-assumed-role.md). To collect Amazon S3 logs that use server-side encryption (SSE), the user role must have an IAM policy that states that Cortex XDR has kms:Decrypt permissions. With this permission, Amazon S3 automatically detects if a bucket is encrypted and decrypts it. If you want to collect encrypted logs from different accounts, you must have the decrypt permissions for the user role also in the key policy for the master account Key Management Service (KMS). For more information, see [Allowing users in other accounts to use a KMS key](https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying-external-accounts.html). ### Configure Cortex XDR to receive generic logs from Amazon S3 1. Log in to the [AWS Management Console](https://eu-north-1.signin.aws.amazon.com/oauth?client_id=arn%3Aaws%3Asignin%3A%3A%3Aconsole%2Fcanvas\&code_challenge=SqCV301kLZ86ajJtPRmME0LYXLumlJThtViqF75o2TM\&code_challenge_method=SHA-256\&response_type=code\&redirect_uri=https%3A%2F%2Fconsole.aws.amazon.com%2Fconsole%2Fhome%3Fca-oauth-flow-id%3D3741%26hashArgs%3D%2523%26isauthcode%3Dtrue%26oauthStart%3D1785839429334%26state%3DhashArgsFromTB_eu-north-1_668aa99d379e7027). 2. From the menu bar, ensure that you have selected the correct region for your configuration. 3. Configure an Amazon Simple Queue Service (SQS).

Note: Ensure that you create your Amazon S3 bucket and Amazon SQS queue in the same region.

1. Open the [Amazon SQS console](https://console.aws.amazon.com/sqs/) and select **Create queue**. 2. Configure the following settings, where the default settings should be configured unless otherwise indicated. * **Type:** Select **Standard** queue (default). * **Name:** Specify a descriptive name for your SQS queue. * **Configuration** section: Leave the default settings for the various fields. * **Access policy** → **Choose method**: Select **Advanced** and update the Access policy code in the editor window to enable your Amazon S3 bucket to publish event notification messages to your SQS queue. Use this sample code as a guide for defining the `“Statement”` with the following definitions: * **`“Resource”`**: Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format `“arn:sns:Region:account-id:topic-name”`. You can retrieve your bucket’s ARN by opening the [Amazon S3 Console](https://console.aws.amazon.com/s3/) in a browser window. In the **Buckets** section, select the bucket that you created for collecting the Amazon S3 flow logs, click **Copy ARN**, and paste the ARN in the field.

Note:

For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.

``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "[SQS queue ARN]", "Condition": { "ArnLike": { "aws:SourceArn": "[S3 bucket ARN]" } } } ] } ``` * **Dead-letter queue** section: We recommend that you configure a queue for sending undeliverable messages by selecting **Enabled**, and then in the **Choose queue** field selecting the queue to send the messages. You may need to create a new queue for this, if you do not already have one set up. For more information, see [Amazon SQS dead-letter queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html). 3. Select **Create queue**. Once the SQS is created, a message indicating that the queue was successfully configured is displayed at the top of the page. 4. Configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket. a. Open the [Amazon S3 Console](https://console.aws.amazon.com/s3/) and in the **Properties** tab of your Amazon S3 bucket, scroll down to the **Event notifications** section, and click **Create event notification**. b. Configure the following settings: * **Event name:** Specify a descriptive name for your event notification containing up to 255 characters. * **Prefix:** Do not set a prefix as the Amazon S3 bucket is meant to be a dedicated bucket for collecting only network flow logs. * **Event types:** Select **All object create events** for the type of event notifications that you want to receive. * **Destination:** Select **SQS queue** to send notifications to an SQS queue to be read by a server. * **Specify SQS queue:** You can either select **Choose from your SQS queues** and then select the **SQS queue**, or select **Enter SQS queue ARN** and specify the ARN in the **SQS queue** field. You can retrieve your SQS queue ARN by opening another instance of the AWS Management Console in a browser window, and opening the [Amazon SQS Console](https://console.aws.amazon.com/sqs/), and selecting the Amazon SQS that you created. In the **Details** section, under **ARN**, click the copy icon (⧉), and paste the ARN in the field.
c. Click **Save changes**. Once the event notification is created, a message indicating that the event notification was successfully created is displayed at the top of the page.

Note:

If your receive an error when trying to save your changes, you should ensure that the permissions are set up correctly.

5. Configure access keys for the AWS IAM user.

Note:

  • It is the responsibility of your organization to ensure that the user who performs this task of creating the access key is assigned the relevant permissions. Otherwise, this can cause the process to fail with errors.
  • Skip this step if you are using an Assumed Role for Cortex XDR.
1. Open the [AWS IAM Console](https://console.aws.amazon.com/iam/), and in the navigation pane, select **Access management** → **Users**. 2. Select the **User name** of the AWS IAM user. 3. Select the **Security credentials** tab, and scroll down to the **Access keys** section, and click **Create access key**. 4. Click the copy icon () next to the **Access key ID** and **Secret access key** keys, where you must click **Show secret access key** to see the secret key, and record them somewhere safe before closing the window. You will need to provide these keys when you edit the Access policy of the SQS queue and when setting the **AWS Client ID** and **AWS Client Secret** in Cortex XDR. If you forget to record the keys and close the window, you will need to generate new keys and repeat this process.

Note: For more information, see Managing access keys for IAM users.

6. Update the Access policy of your Amazon SQS queue.

Note: Skip this step if you are using an Assumed Role for Cortex XDR.

1. In the [Amazon SQS Console](https://console.aws.amazon.com/sqs/), select the SQS queue that you created when you configured an Amazon Simple Queue Service (SQS). 2. Select the **Access policy** tab, and **Edit** the Access policy code in the editor window to enable the IAM user to perform operations on the Amazon SQS with permissions to `SQS:ChangeMessageVisibility`, `SQS:DeleteMessage`, and `SQS:ReceiveMessage`. Use this sample code as a guide for defining the `“Sid”: “__receiver_statement”` with the following definitions. * `“aws:SourceArn”`: Specify the ARN of the AWS IAM user. You can retrieve the **User ARN** from the **Security credentials** tab, which you accessed when you configured access keys for the AWS API user. * `“Resource”`: Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format `“arn:sns:Region:account-id:topic-name”`.

Note: For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.

``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]", "Condition": { "ArnLike": { "aws:SourceArn": "[ARN of your Amazon S3 bucket]" } } }, { "Sid": "__receiver_statement", "Effect": "Allow", "Principal": { "AWS": "[Add the ARN for the AWS IAM user]" }, "Action": [ "SQS:ChangeMessageVisibility", "SQS:DeleteMessage", "SQS:ReceiveMessage" ], "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]" } ] } ``` 7. Configure the Amazon S3 collection in Cortex XDR. a. Select **Settings** → **Configurations** → **Data Collection** → **Collection Integrations**. b. In the **Amazon S3** configuration, click **Add Instance**. c. Set these parameters, where the parameters change depending on whether you configured an **Access Key** or **Assumed Role**. * To provide access to Cortex XDR to your logs and perform API operations using a designated AWS IAM user, leave the **Access Key** option selected. Otherwise, select **Assumed Role**, and ensure that you create an Assumed Role for Cortex XDR before continuing with these instructions. In addition, when you create an Assumed Role for Cortex XDR, ensure that you edit the policy that defines the permissions for the role with the Amazon S3 Bucket ARN and SQS ARN. * **SQS URL:** Specify the **SQS URL**, which is the ARN of the Amazon SQS that you configured in the AWS Management Console. * **Name:** Specify a descriptive name for your log collection configuration. * When setting an **Access Key**, set these parameters. * **AWS Client ID:** Specify the **Access key ID**, which you received when you configured access keys for the AWS IAM user in AWS. * **AWS Client Secret:** Specify the **Secret access key** you received when you configured access keys for the AWS IAM user in AWS. * When setting an **Assumed Role**, set these parameters. * **Role ARN:** Specify the **Role ARN** for the Assumed Role you created for Cortex XDR in AWS. * **External Id:** Specify the **External Id** for the Assumed Role you created for Cortex XDR in AWS. * **Log Type:** Select **Generic** to configure your log collection to receive generic logs from Amazon S3, which can include different types of data, such as file and metadata. When selecting this option, the following additional fields are displayed. * **Log Format:** Select the log format type as **Raw**, **JSON**, **CEF**, **LEEF**, **Cisco**, **Corelight**, or **Beyondtrust Cloud ECS**.

Note:

  • The Vendor and Product defaults to Auto-Detect when the Log Format is set to CEF or LEEF.
  • For a Log Format set to CEF or LEEF, Cortex XDR reads events row by row to look for the Vendor and Product configured in the logs. When the values are populated in the event log row, Cortex XDR uses these values even if you specified a value in the Vendor and Product fields in the Amazon S3 data collector settings. Yet, when the values are blank in the event log row, Cortex XDR uses the Vendor and Product that you specified in these fields in the Amazon S3 data collector settings. If you did not specify a Vendor or Product in the Amazon S3 data collector settings, and the values are blank in the event log row, the values for both fields are set to unknown.
For a **Log Format** set to **Beyondtrust Cloud ECS**, the following fields are automatically set and are not configurable: * **Vendor: Beyondtrust** * **Product: Privilege Management** * **Compression: Uncompressed** For more information, see [Ingest logs from BeyondTrust Privilege Management Cloud.](/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/data-ingestion/external-data-ingestion/additional-log-ingestion-methods/ingest-logs-from-beyondtrust-privilege-management-cloud.md) For a **Log Format** set to **Cisco**, the following fields are automatically set and not configurable. * **Vendor: Cisco** * **Product: ASA** For a **Log Format** set to **Corelight**, the following fields are automatically set and not configurable: * **Vendor: Corelight** * **Product: Zeek** For a **Log Format** set to **Raw** or **JSON**, the following fields are automatically set and are configurable. * **Vendor: AMAZON** * **Product: AWS** Cortex XDR supports logs in single line format or multiline format. For a **JSON** format, multiline logs are collected automatically when the **Log Format** is configured as **JSON**. When configuring a **Raw** format, you must also define the **Multiline Parsing Regex** as explained below. * **Vendor:** (Optional) Specify a particular vendor name for the Amazon S3 generic data collection, which is used in the Amazon S3 XQL dataset `__raw` that Cortex XDR creates as soon as it begins receiving logs. * **Product:** (Optional) Specify a particular product name for the Amazon S3 generic data collection, which is used in the Amazon S3 XQL dataset name `__raw` that Cortex XDR creates as soon as it begins receiving logs. * **Compression:** Select whether the logs are compressed into a **gzip** file or are **uncompressed**. * **Multiline Parsing Regex:** (Optional) This option is only displayed when the **Log Format** is set to **Raw**, where you can set the regular expression that identifies when the multiline event starts in logs with multilines. It is assumed that when a new event begins, the previous one has ended. d. Click **Test** to validate access, and then click **Enable**. Once events start to come in, a green check mark appears underneath the **Amazon S3** configuration with the number of logs received. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/cortex-xdr-3.x/cortex-xdr-3.x-documentation/data-management/data-ingestion/external-data-ingestion/ingest-operation-and-system-logs-from-cloud-providers/ingest-generic-logs-from-amazon-s3.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.