Integrate Your AWS S3 Data Source with the Security Data Lake
- 12 Nov 2024
- 1 Minute to read
- PDF
Integrate Your AWS S3 Data Source with the Security Data Lake
- Updated on 12 Nov 2024
- 1 Minute to read
- PDF
Article summary
Did you find this summary helpful?
Thank you for your feedback
Any external data source that can be configured to write logs to an AWS S3 bucket can forward data to the Red Canary Security Data Lake.
By integrating your security logs with the Red Canary Security Data Lake, you can meet data retention requirements, export logs when needed for investigation or reporting, and ensure greater visibility into your security infrastructure for your team and Red Canary. To integrate an external data source with Red Canary through AWS S3, follow the procedure below from beginning to end.
Step 1: Red Canary–Create your Red Canary generated URL
From your Red Canary dashboard navigate to Integrations, click the split button to the right of Add Integration, and click Add Data Lake Integration.
.png?sv=2022-11-02&spr=https&st=2024-11-23T10%3A45%3A40Z&se=2024-11-23T10%3A55%3A40Z&sr=c&sp=r&sig=PIPYiAMeZvaGj7HYmunPeo2p2w9bBFiQv49ayAkA0lE%3D "image(317).png")
Enter a name for your integration.
Under Ingest Format / Method, select Data Source via S3 (Security Data Lake).
Select the desired data retention period in days (default: 90).
Click Save.
Click Edit Configuration.
Click Activate.
After a few minutes, Red Canary will generate an S3 Folder URL, AWS Access ID, and AWS Secret Key that you will use to set up log forwarding in your external data source. Copy and then save these values. You will use them in a later step.
.png?sv=2022-11-02&spr=https&st=2024-11-23T10%3A45%3A40Z&se=2024-11-23T10%3A55%3A40Z&sr=c&sp=r&sig=PIPYiAMeZvaGj7HYmunPeo2p2w9bBFiQv49ayAkA0lE%3D "image(343).png")
These configuration settings will not be generated until the Red Canary integration is saved and activated.
.png?sv=2022-11-02&spr=https&st=2024-11-23T10%3A45%3A40Z&se=2024-11-23T10%3A55%3A40Z&sr=c&sp=r&sig=PIPYiAMeZvaGj7HYmunPeo2p2w9bBFiQv49ayAkA0lE%3D "image(317).png")
.png?sv=2022-11-02&spr=https&st=2024-11-23T10%3A45%3A40Z&se=2024-11-23T10%3A55%3A40Z&sr=c&sp=r&sig=PIPYiAMeZvaGj7HYmunPeo2p2w9bBFiQv49ayAkA0lE%3D "image(343).png")
Step 2: External Data Source–Configure log forwarding
From your external data source, set up log forwarding using the S3 Folder URL, AWS Access ID, and AWS Secret Key values noted in the previous section.
Ensure that the data source is configured to emit logs in a line-delimited format.
Examples of line-delimited file formats: newline separated JSON, CSV, TSV, CEF, CLF, etc.
Files can be sent in a raw, uncompressed format, or compressed. As long as the compression uses a common compression algorithm, the Data Lake Adapter should be able to ingest the data.
Examples of supported compression algorithms: ZIP, GZIP, etc.
Was this article helpful?
