Login
        Contents x
        Push Data to a Red Canary-Managed Amazon S3 Bucket
        • 30 May 2025
        • 2 Minutes to read
        • PDF
        Contents

        Push Data to a Red Canary-Managed Amazon S3 Bucket

        • Updated on 30 May 2025
        • 2 Minutes to read
        • PDF
        Article summary

        Any external data source that can be configured to forward logs to an Amazon S3 bucket using access keys can forward data to the Red Canary Security Data Lake. All data forwarded in this way is storable and exportable from the Security Data Lake, and if it is newline-delimited JSON, it can be queried via the Search page.

        How does it work?

        This ingest method works by creating a Red Canary-managed Amazon S3 bucket/folder that you can use to receive logs from your external data source. You will be provided a fully qualified URL containing the bucket name, folder name (i.e.: prefix), and region to which you will point your external data source. Authentication is handled via Amazon’s long-term access keys. If your data source only supports role assumption for authentication, consider using the Data Source via S3 (Self-Managed) ingest method instead.

        By integrating your security logs with the Red Canary Security Data Lake, you can meet data retention requirements, export logs when needed for investigation or reporting, and ensure greater visibility into your security infrastructure for your team and Red Canary. To integrate an external data source with Red Canary through a Red Canary-managed Amazon S3 bucket, follow the procedure below from beginning to end.

        Prerequisites

        Before you start the Amazon S3 integration, please make sure the following requirements are met:

        1. You have an active Red Canary Security Data Lake license.

        2. You have validated that your external data source supports log forwarding to Amazon S3 using access keys.

        3. You have validated that your external data source emit logs in a supported format.

          1. Ensure that your data source is configured to emit logs as either gzip, zstd, or uncompressed files.

          2. When possible, we recommend configuring your external data source to emit logs as newline-delimited JSON to maximize your visibility into the data, but any line-delimited text format can be ingested.

        4. You have appropriate admin permissions to make configuration changes to your external data source.

        1 | Red Canary | Add a new data lake integration

        1. From your Red Canary dashboard navigate to Integrations, click the split button to the right of Add Integration, and click Add Data Lake Integration.

        2. Next to Add Integration, enter a name for your integration.

        3. Choose how Red Canary will receive this data:

          1. Under Ingest Format / Method, select Data Source via S3 (Managed by Red Canary).

          2. Click the Next button.

        4. Configure Red Canary to retrieve data from this integration:

          1. Click the Provision button.

          2. This will save and activate your integration. If successful, you should get a “User provisioned successfully” notification.

          3. Under Set up log forwarding in your external log source, there will be an S3 Folder URL, AWS Access ID, and AWS Secret Key that you can use to send data to Red Canary. Copy and save these values. You will use them in a later step.

            The AWS Access ID and AWS Secret Key can take time to generate. If they say “(pending…)”, wait 10 minutes and reload the page.

          4. Click the Next button.

        5. Customize how data from this integration is handled:

          1. Specify your desired data retention period in days.

        6. Click Save in the bottom right corner.

        2 | External Data Source | Configure log forwarding

        1. From your external data source, set up log forwarding using the S3 Folder URL, AWS Access ID, and AWS Secret Key values noted in the previous section.

        2. Ensure that the data source is configured to emit logs as either gzip, zstd, or uncompressed files.

        3. If possible, ensure that the data source is configured to emit logs as newline-delimited JSON. Other line-delimited formats are supported for data retention only.

          Examples of line-delimited file formats: newline separated JSON, CSV, TSV, CEF, CLF, etc.

        Was this article helpful?
        What's Next
        Change password!
        Changing your password will log you out immediately. Use the new password to log back in.
        Change profile
        Success!
        First name must have atleast 2 characters. Numbers and special characters are not allowed.
        Last name must have atleast 1 characters. Numbers and special characters are not allowed.
        Enter a valid email
        Enter a valid password
        Your profile has been successfully updated.
        Logout