Ingest generic logs from Amazon S3

Collect generic logs from Amazon S3 in Cortex XSIAM.

You can forward generic logs for the relevant service to Cortex XSIAM from Amazon S3.

To receive generic data from Amazon Simple Storage Service (Amazon S3), you must first configure data collection from Amazon S3. You can then configure the Data Sources & Integrations settings in Cortex XSIAM for Amazon S3. After you set up collection integration, Cortex XSIAM begins receiving new logs and data from the source.

Note: For more information on configuring data collection from Amazon S3, see the Amazon S3 Documentation.

When Cortex XSIAM begins receiving logs, the app automatically creates an Amazon S3 Cortex Query Language (XQL) dataset (<Vendor>_<Product>_raw). This enables you to search the logs using XQL Search with the dataset. For example queries, refer to the in-app XQL Library. Cortex XSIAM can also generate Cortex XSIAM issues (Correlation Rules only), when relevant, from Amazon S3 logs.

Note: You need to set up an Amazon S3 data collector to receive generic logs when collecting logs from BeyondTrust Privilege Management Cloud. For more information, see Ingest logs from BeyondTrust Privilege Management Cloud.

Prerequisites

Perform the following tasks before you begin configuring data collection from Amazon S3:

• Create a dedicated Amazon S3 bucket, which collects the generic logs that you want to capture. For more information, see Creating a bucket using the Amazon S3 Console.

  Note: It is the customer’s responsibility to define a retention policy for your Amazon S3 bucket by creating a Lifecycle rule in the Management tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.

• The logs collected by your dedicated Amazon S3 bucket must adhere to the following guidelines.

  • Each log file must use the 1 log per line format. By default, multi-line format is not supported. It can only be used for raw format when you specifically configure your environment for that use case.

  • The log format must be compressed as gzip or uncompressed.

  • For best performance, we recommend limiting each file size to up to 50 MB (compressed).

• Ensure that you have at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS).

  • Amazon S3 bucket: GetObject

  • SQS: ChangeMessageVisibility, ReceiveMessage, GetQueueAttributes, and DeleteMessage.

• Determine how you want to provide access to Cortex XSIAM to your logs and perform API operations. You have the following options:

  • Use Workload Federated Identity to allow the Cortex XSIAM log collector service account to assume an IAM role in your AWS account with a short-lived, Google-signed OIDC token instead of stored AWS access keys. This is the Workload Federated Identity option in the Amazon S3 collection configuration, which is selected by default when available. For setup instructions, see Create a Workload Federated Identity role.

  • Designate an AWS IAM user, where you will need to know the Account ID for the user and have the relevant permissions to create an access key/id for the relevant IAM user. This is the Access Key option in the Amazon S3 collection configuration.

  • Create an assumed role in AWS to delegate permissions to a Cortex XSIAM AWS service. This role grants Cortex XSIAM access to your logs. For more information, see Creating a role to delegate permissions to an AWS service. This is the STS AssumeRole option in the Amazon S3 collection configuration. For more information on creating an assumed role for Cortex XSIAM, see Create an assumed role.

• If you use Workload Federated Identity, ensure that you have IAM permissions to create identity providers, policies, and roles in your AWS account. Google (accounts.google.com) must be registered as an OIDC identity provider in your AWS account with sts.amazonaws.com as an audience. If it isn't registered yet, you register it when you create the role. You also need the Cortex XSIAM service account identifier, which you copy from the Amazon S3 collection configuration in Cortex XSIAM before you save the configuration.

• To collect Amazon S3 logs that use server-side encryption (SSE), the user role must have an IAM policy that states that Cortex XSIAM has kms:Decrypt permissions. With this permission, Amazon S3 automatically detects if a bucket is encrypted and decrypts it. If you want to collect encrypted logs from different accounts, you must have the decrypt permissions for the user role also in the key policy for the master account Key Management Service (KMS). For more information, see Allowing users in other accounts to use a KMS key. If you use Workload Federated Identity, add a statement that allows kms:Decrypt on the key ARN to the permissions policy of the IAM role. For cross-account keys, also allow the role in the KMS key policy.

Configure Cortex XSIAM to receive generic logs from Amazon S3

1. Log in to the AWS Management Console.

2. From the menu bar, ensure that you have selected the correct region for your configuration.

3. Configure an Amazon Simple Queue Service (SQS).

   Note: Ensure that you create your Amazon S3 bucket and Amazon SQS queue in the same region.

   1. In the Amazon SQS Console, click Create Queue.

   2. Configure the following settings, where the default settings should be used unless otherwise indicated.

      • Type: Select Standard queue (default).

      • Name: Specify a descriptive name for your SQS queue.

      • Configuration section: Leave the default settings for the various fields.

      • Access policy → Choose method: Select Advanced and update the Access policy code in the editor window to enable your Amazon S3 bucket to publish event notification messages to your SQS queue. Use this sample code as a guide for defining the “Statement” with the following definitions.

        -“Resource”: Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format arn:aws:sqs:<region>:<account-id>:<queue-name>.

        You can retrieve your bucket’s ARN by opening the Amazon S3 Console in a browser window. In the Buckets section, select the bucket that you created for collecting the generic logs, click Copy ARN, and paste the ARN in the field.

        

        Note: For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.

        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "s3.amazonaws.com"
              },
              "Action": "SQS:SendMessage",
              "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]",
              "Condition": {
                "ArnLike": {
                  "aws:SourceArn": "[ARN of your Amazon S3 bucket]"
                }
              }
            }
          ]
        }

      • Dead-letter queue section: We recommend that you configure a queue for sending undeliverable messages by selecting Enabled, and then in the Choose queue field selecting the queue to send the messages. You may need to create a new queue for this, if you do not already have one set up. For more information, see Amazon SQS dead-letter queues.

   3. Click Create queue.

      Once the SQS is created, a message indicating that the queue was successfully configured is displayed at the top of the page.

4. Configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket.

   1. Open the Amazon S3 Console and in the Properties tab of your Amazon S3 bucket, scroll down to the Event notifications section, and click Create event notification.

   2. Configure the following settings:

      • Event name: Specify a descriptive name for your event notification containing up to 255 characters.

      • Prefix: Do not set a prefix, as the Amazon S3 bucket is meant to be a dedicated bucket for collecting only these logs.

      • Event types: Select All object create events for the type of event notifications that you want to receive.

      • Destination: Select SQS queue to send notifications to an SQS queue to be read by a server.

      • Specify SQS queue: You can either select Choose from your SQS queues and then select the SQS queue, or select Enter SQS queue ARN and specify the ARN in the SQS queue field.

        You can retrieve your SQS queue ARN by opening another instance of the AWS Management Console in a browser window, and opening the Amazon SQS Console, and selecting the Amazon SQS that you created. In the Details section, under ARN, click the copy icon (), and paste the ARN in the field.

        
   3. Click Save changes.

      Once the event notification is created, a message indicating that the event notification was successfully created is displayed at the top of the page.

      Note: If you receive an error when trying to save your changes, you should ensure that the permissions are set up correctly.

5. Configure access keys for the AWS IAM user.

   Note:

   • It is the responsibility of your organization to ensure that the user who performs this task of creating the access key is assigned the relevant permissions. Otherwise, this can cause the process to fail with errors.

   • Skip this step if you are using an Assumed Role or Workload Federated Identity for Cortex XSIAM.

   1. Open the AWS IAM Console, and in the navigation pane, select Access management → Users.

   2. Select the User name of the AWS IAM user.

   3. Select the Security credentials tab, scroll down to the Access keys section, and click Create access key.

   4. Click the copy icon next to the Access key ID and Secret access key. You must click Show secret access key to see the secret key, and record them somewhere safe before closing the window. You will need to provide these keys when you edit the Access policy of the SQS queue and when setting the AWS Client ID and AWS Client Secret in Cortex XSIAM. If you forget to record the keys and close the window, you will need to generate new keys and repeat this process.

      For more information, see Managing access keys for IAM users.

6. Update the Access policy of your Amazon SQS queue.

   Note: Skip this step if you are using an Assumed Role or Workload Federated Identity for Cortex XSIAM.

   1. In the Amazon SQS Console, select the SQS queue that you created when you configured an Amazon Simple Queue Service (SQS).

   2. Select the Access policy tab, and edit the Access policy code in the editor window to enable the IAM user to perform operations on the Amazon SQS with permissions to SQS:ChangeMessageVisibility, SQS:DeleteMessage, SQS:ReceiveMessage, and SQS:GetQueueAttributes. Use this sample code as a guide for defining the “Sid”: “__receiver_statement” with the following definitions.

      • “Principal”: {“AWS”: …}: Specify the ARN of the AWS IAM user. You can retrieve the User ARN from the Security credentials tab, which you accessed when you configured access keys for the AWS API user.

      • “Resource”: Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format arn:aws:sqs:<region>:<account-id>:<queue-name>.

        Note: For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.

        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "s3.amazonaws.com"
              },
              "Action": "SQS:SendMessage",
              "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]",
              "Condition": {
                "ArnLike": {
                  "aws:SourceArn": "[ARN of your Amazon S3 bucket]"
                }
              }
            },
           {
              "Sid": "__receiver_statement",
              "Effect": "Allow",
              "Principal": {
                "AWS": "[Add the ARN for the AWS IAM user]"
              },
              "Action": [
                "SQS:ChangeMessageVisibility",
                "SQS:DeleteMessage",
                "SQS:ReceiveMessage",
                "SQS:GetQueueAttributes"
              ],
              "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]"
            }
          ]
        }

7. Configure the Amazon S3 collection in Cortex XSIAM.

   1. Navigate to Settings → Data Sources & Integrations.

   2. On the Data Sources & Integrations page, click + Add New, search for Amazon S3, then hover over it and click Add.

   3. Select the authentication method you configured (Workload Federated Identity, which is selected by default when available, Access Key, or STS AssumeRole), and enter the values for that method. For Workload Federated Identity, first create a Workload Federated Identity role.

Field	Input
SQS URL	Specify the SQS URL, which is the URL of the Amazon SQS that you configured in the AWS Management Console.
Name	Specify a descriptive name for your log collection configuration.
Role ARN	(Workload Federated Identity and STS AssumeRole) Specify the ARN of the IAM role that you created in AWS, in the format arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>. For STS AssumeRole, ensure that the permissions policy of the role includes your Amazon S3 bucket ARN and SQS queue ARN.
Audience	(Workload Federated Identity only) Enter sts.amazonaws.com. This value must match the accounts.google.com:aud condition in the IAM role trust policy.
Identifier	(Workload Federated Identity only) In the Copy service account identifier section, click Identifier to copy the Cortex XSIAM service account identifier for the accounts.google.com:sub condition in the IAM role trust policy.
AWS Client ID	(Access Key only) Specify the Access key ID, which you received when you configured access keys for the AWS IAM user in AWS.
AWS Client Secret	(Access Key only) Specify the Secret access key you received when you configured access keys for the AWS IAM user in AWS.
External ID	(STS AssumeRole only) Specify the External ID for the Assumed Role you created in AWS.
Log Type	Select Generic to configure your log collection to receive generic logs from Amazon S3, which can include different types of data, such as file and metadata.
Log Format	Select the log format type as Raw, JSON, CEF, LEEF, Cisco, Corelight, or Beyondtrust Cloud ECS. Note: The Vendor and Product defaults to Auto-Detect when the Log Format is set to CEF or LEEF. For CEF or LEEF log formats, Cortex XSIAM parses events row by row for Vendor and Product values. Populated row values override any Vendor and Product specified in the Amazon S3 data collector settings. If the row values are blank, Cortex XSIAM falls back to your S3 collector settings; if those are also left blank, both fields default to unknown.
Vendor	(Optional) Specify a particular vendor name for the Amazon S3 generic data collection, which is used in the Amazon S3 XQL dataset <Vendor>_<Product>_raw that Cortex XSIAM creates as soon as it begins receiving logs.
Product	(Optional) Specify a particular product name for the Amazon S3 generic data collection, which is used in the Amazon S3 XQL dataset name <Vendor>_<Product>_raw that Cortex XSIAM creates as soon as it begins receiving logs.
Compression	Select whether the logs are compressed into a gzip file or are uncompressed.
Multiline Parsing Regex	(Optional, Raw format only) Enter a regular expression to identify the start of a new log event. Cortex XSIAM assumes each new event start indicates the previous event has ended.

When Log Format is set to one of the following, the fields are pre-populated as shown:

Log Format	Vendor	Product	Compression	Configurable?
Beyondtrust Cloud ECS	Beyondtrust	Privilege Management	Uncompressed	No
Cisco	Cisco	ASA	N/A	No
Corelight	Corelight	Zeek	N/A	No
Raw or JSON	AMAZON	AWS	N/A	Yes

• Cortex XSIAM supports logs in single-line format or multiline format. For a JSON format, multiline logs are collected automatically when the Log Format is configured as JSON. When configuring a Raw format, you must also define the Multiline Parsing Regex field described in the table.

1. Click Test to validate access, and then click Enable. For Workload Federated Identity, the Test button isn't available, so click Enable.

Once events start to come in, a green check mark appears underneath the Amazon S3 configuration with the number of logs received.