---
title: "Pull Data from Your Self-Managed Amazon S3 Bucket"
slug: "pull-data-from-your-self-managed-amazon-s3-bucket"
updated: 2026-10-07T17:15:14Z
published: 2026-10-07T17:15:14Z
canonical: "docs.redcanary.com/pull-data-from-your-self-managed-amazon-s3-bucket"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.redcanary.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Pull Data from Your Self-Managed Amazon S3 Bucket

Any external data source that can be configured to forward logs to an Amazon S3 bucket in your AWS account can forward data to the Zscaler Security Data Lake. All data forwarded in this way is [storable and exportable](/doc/docs/search-and-export-data-from-the-security-data-lake) from the Security Data Lake, and if it is newline-delimited JSON, it can be queried via the [Search](/doc/docs/searching-your-data) page and [used as context](/doc/docs/view-investigations#related-data-by-query-tab) in threat investigations..

## How does it work?

This ingest method works by listening for S3 object creation notifications, and ingesting any of the objects related to those notifications. Notifications are produced via the [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html) system, and are delivered to Zscaler MDR via the Simple Notification Service (SNS). At this time, a unique SNS topic is required for each distinct data source that uses this ingest method. Zscaler MDR will require limited access to your AWS account via an assumed role in order to retrieve files from the bucket and subscribe to notifications. If it is not feasible to grant this access, consider using the [Data Source via S3 (Managed by Zscaler MDR)](/doc/docs/integrate-your-aws-s3-data-source-with-the-security-data-lake) ingest method instead.

By integrating your security logs with the Zscaler Security Data Lake, you can meet data retention requirements, export logs when needed for investigation or reporting, and ensure greater visibility into your security infrastructure for your team and Zscaler MDR. To integrate an external data source with Zscaler MDR through your self-managed Amazon S3 bucket, follow the procedure below from beginning to end.

## Prerequisites

Before you start the Amazon S3 integration, please make sure the following requirements are met:

1. You have an active Zscaler Security Data Lake license.
2. You have configured your external data source to store its logs in an S3 bucket in your AWS account.
  1. Ensure that your data source is configured to emit logs as either gzip, zstd, or uncompressed files.
  2. When possible, we recommend configuring your external data source to emit logs as newline-delimited JSON to maximize your visibility into the data, but any line-delimited text format can be ingested.
3. You have an AWS Console admin account with permissions to:
  1. Create Simple Notification Service (SNS) topics
  2. Adjust resource policies on SNS topics
  3. Set notifications on S3 buckets
  4. Adjust resource policies on S3 buckets
  5. Create IAM roles

## 1 | AWS | Identify the source bucket

This step assumes that you have already configured your external data source to write its logs to an Amazon S3 bucket in your AWS account. In order for Zscaler MDR to ingest those logs, you’ll need to provide the Amazon Resource Name (ARN) that uniquely identifies the bucket.

1. Navigate to **Amazon S3** in your **AWS Console**.
2. Search for the desired bucket, select it, and click **Copy ARN**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(444).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
3. Save this bucket ARN — you will need to provide it during configuration of the Zscaler MDR integration.

## 2 | AWS | Create an SNS topic

You need to set up an SNS topic in AWS to receive event notifications when data is added to the S3 bucket. The SNS topic will require a resource policy which permits S3 to publish notifications.

1. Navigate to the [Create Topic](https://console.aws.amazon.com/sns/v3/home#/create-topic) form in your **AWS Console**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(447).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
2. Choose **Standard** as the topic type and enter a **Name**.

> [!NOTE]
> Be sure to give your topic a meaningful name: for instance, you might name it **pdx-campus-juniper-to-red-canary** if you were setting up an integration to ingest security data from Juniper appliances at your Portland office.
3. Expand **Access policy** and select **Advanced** to edit the access policy document. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(449).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
4. Modify the following policy fragment to match your configuration, replacing `<REGION>` with the correct AWS region, `<ACCOUNT ID>` with your AWS account ID, `<TOPIC NAME>` with the topic name entered earlier, and `<BUCKET NAME>` with the source bucket name. Note that there is a comma after the last curly brace.

```json
    {
      "Sid": "S3",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:<REGION>:<ACCOUNT ID>:<TOPIC NAME>",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "<ACCOUNT ID>"
        },
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::<BUCKET NAME>"
        }
      }
    },
```
5. Once you have done this, **insert** the policy fragment into the beginning of the `“Statement”` array in the **JSON editor** (do not replace the entire policy statement). ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(450).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
6. Click **Create topic** in the bottom right corner to save the new topic.
7. Note the topic ARN — you will need to provide it during configuration of the Zscaler MDR integration. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(451).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)

## 3 | AWS | Send S3 event notifications to the SNS topic

You need to configure the S3 bucket to send notifications to the SNS topic whenever new data is available.

> [!NOTE]
> The following instructions assume that your bucket does not already have any event notifications configured. If your bucket has preexisting event notifications configured, skip to the next section for instructions specific to that scenario.

1. Navigate to the [Amazon S3 bucket list](https://console.aws.amazon.com/s3/home) in your **AWS Console**.
2. Select the desired bucket, and navigate to the **Properties** tab. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(452).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
3. Scroll down to find the **Event Notifications** section and click **Create event notification**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(453).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
4. Under **General configuration**, enter an **Event name**. These event names are unique only within the bucket, and are strictly informational. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(454).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
5. Optionally, specify a **Prefix** and/or **Suffix** to restrict what data is sent to Zscaler MDR.

> [!NOTE]
> A prefix is often used to specify a specific folder in the bucket that should be included in the data ingest, but it does not have to contain slashes. For example, if your bucket receives files from multiple sources, and the ones which you wish to send to Zscaler MDR all start with “security-vendor-product-name-”, you would use that as the prefix.
6. Under **Event types**, select **All object create events**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(455).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
7. Scroll down to find the **Destination** section, select **SNS topic**, and choose the topic you created earlier under **Specify SNS topic**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(457).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
8. Click **Save changes**.

> [!NOTE]
> If the topic is correctly set up, you should see it populate with two subscriptions after you finish and activate the integration.

## 4 | AWS | Working around existing event notifications

You need to configure the S3 bucket to send notifications to the SNS topic whenever new data is available, but your bucket may already have event notifications configured. If so, read this section.

> [!NOTE]
> Depending on how these notifications were configured, you may be able to follow the steps in the previous section without issue. Before reading further, you should attempt the steps in the previous section — if they succeed, you can skip this section. If you get an error, continue below.

1. If you receive an error message stating “*Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type*”, it is because there is an existing event notification rule that overlaps in some way with the **Event type**, **Prefix**, and **Suffix** you specified. If this happens, review your object notifications and see if you can remove that overlap. Each event notification must flow unambiguously to a single topic.
2. If, as above, you have an existing event notification rule that matches the data you wish to send to Zscaler MDR, and which sends notifications to an SNS topic which is not already in use by another Zscaler MDR data source, consider reusing the existing topic rather than trying to create a new topic. Filtering by prefix or suffix can be added to the Zscaler MDR integration instead.
3. If the topic is already in use by another Zscaler MDR data source, or notifications are not published to an SNS topic (i.e.: the **Destination** is a Lambda function or SQS queue), please speak with your account team. At this time, you may need to copy data to Zscaler MDR using the [Data Source via S3 (Managed by Zscaler MDR)](/doc/docs/integrate-your-aws-s3-data-source-with-the-security-data-lake) ingest method, or implement a solution to replicate the notifications to an additional topic with a Lambda function.

## 5 | Zscaler MDR | Add a new data lake integration

Now that you have prepared your AWS resources, you can create a new data lake integration in the Zscaler MDR portal.

1. In the Zscaler MDR portal navigate to **Integrations**, click the split button to the right of **Add Integration**, and click **Add Data Lake Integration**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(317).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
2. Next to **Add Integration**, enter a name for your integration. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(441).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
3. Choose how Zscaler MDR will receive this data:
  1. Under **Ingest Format / Method**, select **Data Source via S3 (Self Managed)**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(442).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
  2. Click the **Next** button.
4. Configure Zscaler MDR to retrieve data from this integration:
  1. Specify the **ARN of the S3 bucket containing your data** and the **ARN of the SNS topic with S3 bucket notifications** you created earlier. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(459).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)

## 6 | Zscaler MDR | Generate an IAM role template

You’ll use a Zscaler MDR-provided template to provision an IAM role in your AWS environment for Zscaler MDR access.

1. In the **Provision an AWS IAM role that Zscaler MDR will assume to read from the above AWS S3 bucket** section on the Zscaler MDR configuration page, select **CloudFormation** or **Terraform** to generate the appropriate template. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(464).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
2. Copy and paste your required template into a new file/document and save it. You’ll upload this file to AWS later.

> [!NOTE]
> If you have not already done so for a previous **data lake** source, use the supplied CloudFormation or Terraform documents to configure the role and policy that Zscaler MDR will use to access the SNS topic and files in your bucket. Please note that this is not the same role that is used for other non-data-lake integrations like AWS or SentinelOne. If your organizational policy requires the use of a custom role name, or you have already created this role but it is used in another Zscaler MDR subdomain, please contact your account team for assistance in configuring this.

## 7 | AWS | Provision the IAM role

You’ll now pivot back to AWS to apply the saved template.

> [!NOTE]
> If you have already created a data source that uses the **Data Source via S3 (Self Managed)** ingest method to ingest data from the same AWS account, you do not need to apply the CloudFormation or Terraform again, as the necessary role will already exist. Instead, you must grant the existing role permission to perform the actions listed in the CloudFormation or Terraform document on the applicable resources.

### For CloudFormation

1. Navigate to the [CloudFormation](https://console.aws.amazon.com/cloudformation/home) **Stacks** page in your **AWS Console**.
2. Click **Create stack** and select **With new resources (standard)**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(466).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
3. Under **Prerequisite - Prepare template**, select **Choose an existing template**.
4. Under **Specify template**, select **Upload template file**.
5. Click **Choose file** and upload the template file you created earlier. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(467).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
6. Click **Next**.
7. Enter a name for your new stack. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(469).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
8. Click **Next**.
9. Under **Capabilities**, accept the acknowledgment message. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(470).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
10. Click **Next**.
11. Click **Submit**.

### For Terraform

Terraform usage is dependent on your environment. If you need assistance with the Terraform template, please contact Zscaler MDR Support.

## 8 | Zscaler MDR | Confirm the IAM role in Zscaler MDR

1. When you’ve finished provisioning the IAM role in AWS, check **I’ve configured this integration to send data to Zscaler MDR** on the Zscaler MDR configuration page. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(472).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)

## 9 | Zscaler MDR | Add filtering by prefix/suffix

For efficiency and maximum control, we recommend that you implement filtering as part of the event notifications you set up within AWS. However, there are cases when this is not possible — especially if you are reusing an SNS topic for multiple destinations. In those cases, you can optionally specify the filters in Zscaler MDR.

1. If needed, specify a folder name or prefix under **Only ingest S3 objects with the key prefix**. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(473).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
2. If needed, specify a file extension or suffix under **Only ingest S3 objects with the key suffix**.
3. If this prefix/suffix was already specified as part of the AWS event notification, there is no need to add it here.
4. Click **Next**.

## 10 | Zscaler MDR | Specify the data retention period and save

Customize how data from this integration is handled:

1. Specify your desired data retention period in days. ![](https://cdn.document360.io/75147bee-bf0d-444c-8afb-6e83b11ec0ef/Images/Documentation/image(410).png?sv=2026-02-06&spr=https&st=2026-11-04T15%3A16%3A11Z&se=2026-11-04T16%3A16%3A11Z&sr=c&sp=r&sig=bIrBU7m4Qg48n%2BKKk2aWePMpDP9rKYFnR5S8ojxTj78%3D)
2. Click **Save** in the bottom right corner.

> [!TIP]
> **The Amazon S3 integration is now live!**
> 
> You should see data start appearing in the Security Data Lake within one hour.