Pull Data from Your Self-Managed Amazon S3 Bucket

Before you start the Amazon S3 integration, please make sure the following requirements are met:

1. You have an active Red Canary Security Data Lake license.

2. You have configured your external data source to store its logs in an S3 bucket in your AWS account.

   1. Ensure that your data source is configured to emit logs as either gzip, zstd, or uncompressed files.

   2. When possible, we recommend configuring your external data source to emit logs as newline-delimited JSON to maximize your visibility into the data, but any line-delimited text format can be ingested.

3. You have an AWS Console admin account with permissions to:

   1. Create Simple Notification Service (SNS) topics

   2. Adjust resource policies on SNS topics

   3. Set notifications on S3 buckets

   4. Adjust resource policies on S3 buckets

   5. Create IAM roles

This step assumes that you have already configured your external data source to write its logs to an Amazon S3 bucket in your AWS account. In order for Red Canary to ingest those logs, you’ll need to provide the Amazon Resource Name (ARN) that uniquely identifies the bucket.

1. Navigate to Amazon S3 in your AWS Console.

2. Search for the desired bucket, select it, and click Copy ARN.
   

3. Save this bucket ARN — you will need to provide it during configuration of the Red Canary integration.

You need to set up an SNS topic in AWS to receive event notifications when data is added to the S3 bucket. The SNS topic will require a resource policy which permits S3 to publish notifications.

1. Navigate to the Create Topic form in your AWS Console.
   

2. Choose Standard as the topic type and enter a Name.

   Be sure to give your topic a meaningful name: for instance, you might name it pdx-campus-juniper-to-red-canary if you were setting up an integration to ingest security data from Juniper appliances at your Portland office.

3. Expand Access policy and select Advanced to edit the access policy document.
   

4. Modify the following policy fragment to match your configuration, replacing <REGION> with the correct AWS region, <ACCOUNT ID> with your AWS account ID, <TOPIC NAME> with the topic name entered earlier, and <BUCKET NAME> with the source bucket name. Note that there is a comma after the last curly brace.

       {
         "Sid": "S3",
         "Effect": "Allow",
         "Principal": {
           "Service": "s3.amazonaws.com"
         },
         "Action": "SNS:Publish",
         "Resource": "arn:aws:sns:<REGION>:<ACCOUNT ID>:<TOPIC NAME>",
         "Condition": {
           "StringEquals": {
             "aws:SourceAccount": "<ACCOUNT ID>"
           },
           "ArnEquals": {
             "aws:SourceArn": "arn:aws:s3:::<BUCKET NAME>"
           }
         }
       },

5. Once you have done this, insert the policy fragment into the beginning of the “Statement” array in the JSON editor (do not replace the entire policy statement).
   

6. Click Create topic in the bottom right corner to save the new topic.

7. Note the topic ARN — you will need to provide it during configuration of the Red Canary integration.
   

You need to configure the S3 bucket to send notifications to the SNS topic whenever new data is available.

The following instructions assume that your bucket does not already have any event notifications configured. If your bucket has preexisting event notifications configured, skip to the next section for instructions specific to that scenario.

1. Navigate to the Amazon S3 bucket list in your AWS Console.

2. Select the desired bucket, and navigate to the Properties tab.
   

3. Scroll down to find the Event Notifications section and click Create event notification.
   

4. Under General configuration, enter an Event name. These event names are unique only within the bucket, and are strictly informational.
   

5. Optionally, specify a Prefix and/or Suffix to restrict what data is sent to Red Canary.

   A prefix is often used to specify a specific folder in the bucket that should be included in the data ingest, but it does not have to contain slashes. For example, if your bucket receives files from multiple sources, and the ones which you wish to send to Red Canary all start with “security-vendor-product-name-”, you would use that as the prefix.

6. Under Event types, select All object create events.
   

7. Scroll down to find the Destination section, select SNS topic, and choose the topic you created earlier under Specify SNS topic.
   

8. Click Save changes.

   If the topic is correctly set up, you should see it populate with two subscriptions after you finish and activate the integration.

You need to configure the S3 bucket to send notifications to the SNS topic whenever new data is available, but your bucket may already have event notifications configured. If so, read this section.

Depending on how these notifications were configured, you may be able to follow the steps in the previous section without issue. Before reading further, you should attempt the steps in the previous section — if they succeed, you can skip this section. If you get an error, continue below.

1. If you receive an error message stating “Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type”,
   it is because there is an existing event notification rule that overlaps in some way with the Event type, Prefix, and Suffix you specified. If this happens, review your object notifications and see if you can remove that overlap. Each event notification must flow unambiguously to a single topic.

2. If, as above, you have an existing event notification rule that matches the data you wish to send to Red Canary, and which sends notifications to an SNS topic which is not already in use by another Red Canary data source, consider reusing the existing topic rather than trying to create a new topic. Filtering by prefix or suffix can be added to the Red Canary integration instead.

3. If the topic is already in use by another Red Canary data source, or notifications are not published to an SNS topic (i.e.: the Destination is a Lambda function or SQS queue), please speak with your account team. At this time, you may need to copy data to Red Canary using the Data Source via S3 (Managed by Red Canary) ingest method, or implement a solution to replicate the notifications to an additional topic with a Lambda function.

Now that you have prepared your AWS resources, you can create a new data lake integration in the Red Canary portal.

1. From your Red Canary dashboard navigate to Integrations, click the split button to the right of Add Integration, and click Add Data Lake Integration.
   

2. Next to Add Integration, enter a name for your integration.
   

3. Choose how Red Canary will receive this data:

   1. Under Ingest Format / Method, select Data Source via S3 (Self Managed).
      

   2. Click the Next button.

4. Configure Red Canary to retrieve data from this integration:

   1. Specify the ARN of the S3 bucket containing your data and the ARN of the SNS topic with S3 bucket notifications you created earlier.
      

You’ll use a Red Canary-provided template to provision an IAM role in your AWS environment for Red Canary access.

1. In the Provision an AWS IAM role that Red Canary will assume to read from the above AWS S3 bucket section on the Red Canary configuration page, select CloudFormation or Terraform to generate the appropriate template.
   

2. Copy and paste your required template into a new file/document and save it. You’ll upload this file to AWS later.

   If you have not already done so for a previous data lake source, use the supplied CloudFormation or Terraform documents to configure the role and policy that Red Canary will use to access the SNS topic and files in your bucket. Please note that this is not the same role that is used for other non-data-lake integrations like AWS or SentinelOne. If your organizational policy requires the use of a custom role name, or you have already created this role but it is used in another Red Canary subdomain, please contact your account team for assistance in configuring this.

You’ll now pivot back to AWS to apply the saved template.

If you have already created a data source that uses the Data Source via S3 (Self Managed) ingest method to ingest data from the same AWS account, you do not need to apply the CloudFormation or Terraform again, as the necessary role will already exist. Instead, you must grant the existing role permission to perform the actions listed in the CloudFormation or Terraform document on the applicable resources.

For CloudFormation

1. Navigate to the CloudFormation Stacks page in your AWS Console.

2. Click Create stack and select With new resources (standard).
   

3. Under Prerequisite - Prepare template, select Choose an existing template.

4. Under Specify template, select Upload template file.

5. Click Choose file and upload the template file you created earlier.
   

6. Click Next.

7. Enter a name for your new stack.
   

8. Click Next.

9. Under Capabilities, accept the acknowledgment message.
   

10. Click Next.

11. Click Submit.

For Terraform

Terraform usage is dependent on your environment. If you need assistance with the Terraform template, please contact Red Canary Support.

1. When you’ve finished provisioning the IAM role in AWS, check I’ve configured this integration to send data to Red Canary on the Red Canary configuration page.
   

For efficiency and maximum control, we recommend that you implement filtering as part of the event notifications you set up within AWS. However, there are cases when this is not possible — especially if you are reusing an SNS topic for multiple destinations. In those cases, you can optionally specify the filters in Red Canary.

1. If needed, specify a folder name or prefix under Only ingest S3 objects with the key prefix.
   

2. If needed, specify a file extension or suffix under Only ingest S3 objects with the key suffix.

3. If this prefix/suffix was already specified as part of the AWS event notification, there is no need to add it here.

4. Click Next.

Customize how data from this integration is handled:

1. Specify your desired data retention period in days.
   

2. Click Save in the bottom right corner.

   The Amazon S3 integration is now live!

   You should see data start appearing in the Security Data Lake within one hour.