How Amazon S3 Works with Red Canary

Any external data source that can be configured to write logs to an Amazon S3 bucket can forward data to the Red Canary Security Data Lake. All data forwarded in this way is storable and exportable from the Security Data Lake, and if it is newline-delimited JSON, it can be queried via the Search page.

There are two methods of integration:

1. Configure your external data source to push logs directly to a Red Canary-managed S3 bucket.

2. Configure Red Canary to pull logs from an S3 bucket in your AWS account.

What kinds of file formats are supported?

Any line-delimited text-based format can be stored in the data lake for long-term retention and export on-demand (CSV, TSV, CEF, CLF, etc.), but files containing newline-delimited JSON (NDJSON) will automatically support SQL searches as well. When possible, we recommend configuring your external data source to emit logs as JSON / NDJSON to maximize your visibility into the data.

Can I forward compressed files?

Yes! Ensure that your data source is configured to emit logs as either gzip, zstd, or uncompressed files.

What fields are available when querying JSON sources?

If the data you are forwarding to S3 is in JSON format, the Security Data Lake will automatically parse the file structure and generate corresponding tables that can be queried from the Search page.

All Security Data Lake sources include a set of metadata columns — data generated by Red Canary at time of ingest. These always begin with rc_:

JSON sources will also include a set of columns automatically parsed from the original JSON logs. These will vary, as they are based on the schema/structure of the data ingested.