Linux FAQ

Is there a lifecycle for the files, and will they be updated/upgraded independently of the RPM binary?

Yes, the plugins are not distributed in the RPM and they are always downloaded if enabled.

Is it possible to disable plugin updates?

The customer can disable the updates by not incrementing the version in the remote config when we update the plugins. We can introduce new versions on specific endpoints via the exception mechanism, but currently we not have any UI or automation for this.

Can you introduce new plugins into our production environment without our approval or testing?

Technically the sensor can introduce new plugins, but we have not done so. The Response Action plugin is an example of a new plugin we released following the normal plugin mechanism, however it is not enabled unless requested (and paid for).

Red Canary Linux Endpoint Detection and Response (EDR) does not currently support prevention or response capabilities.

Common pain points with EDR/EPP products for Linux are:

• Limited Linux distribution and version support

• Limited Cloud support and integrations

• Limited telemetry collection

• Limited detection capabilities due to a lack of investment in threat research and detection writing for Linux

• Limited visibility and guarantees into sensor safety, performance and reliability

These issues are the largest reason why many businesses do not deploy traditional EDR or EPP products into their Linux Datacenter or Cloud environments. For these businesses, detection expectations are low, and the risk of things going wrong are perceived as high.

Red Canary Linux EDR addresses these issues

• We provide the broadest support for Linux distributions and versions.

• It is written in the safe, performant Rust language and does not require the installation of a kernel module.

• It supports AWS, Azure and Google Cloud.

• It provides sensor performance reporting in the platform, giving you, your team and your internal stakeholders confidence that the sensor is performant and not causing system degradation. This includes CPU and memory utilization, with p50, p90 and p99 percentiles.

• We are 100 percent focused on Linux and have dedicated engineers consistently delivering on fixes and new capabilities.

• We have dedicated threat researchers and detection engineering consistently tuning and delivering on Linux threat detection.

Prevention Capabilities

Red Canary Linux EDR does not yet provide Prevention/Anti-Virus/Next Generation Anti-Virus capabilities for the following reasons:

• Prevention capabilities are limited in their value for traditional EDR/EPP products. There are few or zero threats to block due to the vendors lack of investment in Linux capabilities and detection. We're changing this.

• Traditional EDR/EPP prevention capabilities require the use of low-level code that halts execution until the product gives a “yes/no”, “is this safe?” answer, which introduces performance degradation. This is a deal breaker for many companies as Linux systems are designed to perform workloads, and slowing them down can result in resource fatigue and, as a result, customer facing impact (internal, or external). The stakes of impacting a server are often higher than impacting an employee's laptop.

• Most vendors do not support Prevention for Linux, and for those that do, very few customers enable it due to perceived risk.

• Only once we have completely delivered on detection outcomes for our customers can we examine prevention capabilities.

Response (Containment/Eradication) Capabilities

Red Canary Linux EDR has a response action plugin but actions are limited.

The Rapid7 Insight Agent collects Linux telemetry data and requires AuditD to be present but disabled. Since the Red Canary Linux Endpoint Detection and Response (EDR) agent can consume data from AuditD, this leads to challenges for running both simultaneously.

Configure the Linux EDR sensor to use eBPF

Because the Rapid7 Insight Agent doesn’t collect telemetry using eBPF, you can configure the Linux EDR sensor to use eBPF, and then run both the Linux EDR sensor and Insight Agent simultaneously.

For more information about configuring eBPF as the primary telemetry source, see Use eBPF as the default telemetry source.

Use the Insight Agent compatibility mode

Rapid7 provides a guide for enabling Insight Agent compatibility mode with Linux assets requiring AuditD to be enabled. However, this workaround is not supported by Red Canary Linux EDR, and is not a recommended solution owing to concerns around stability and degraded functionality.

What is the local backlog limit?

1GB (1024MB).

What happens when the backlog limit is reached?

Oldest data is removed until we’re below the local backlog limit.

When backlog exists, what order is the data uploaded?

The data is uploaded in order, from oldest to newest.

What is the maximum size of a payload sent to Red Canary?

The total size of 5000 discrete events.

What is the maximum time until a telemetry payload upload attempt occurs?

Five minutes (endpoint telemetry is recorded continuously).

What is the maximum time until a health payload upload attempt occurs?

Five minutes (health is recorded every 30s).

What determines when a payload upload attempt occurs?

Whichever occurs first, the maximum time duration of five minutes, or the maximum size of 5000 discrete events buffered in memory. This is not currently configurable.

If network connectivity does not exist, what is the retry logic algorithm?

The first time a payload fails to offload, the agent enters offline mode. For each payload file ready to offload:

• The agent will first check if there’s internet connectivity.

  • If there is internet connectivity, it will re-enter online mode and begin uploading again.

  • If there is no internet connectivity, the agent will return prematurely from any attempt to offload.

The time between offload attempts is 100ms.

Retry is indefinite but only one payload attempt per 100ms interval. If you fail once with "offline mode" then it bails on attempting any other payloads.

What compression is used for local storage?

GZIP.

What compression is used when sending data to Red Canary?

GZIP.

Do we support multi-CPU/multi-core?

Yes - the agent is multi-threaded by default. There is no option for single-threaded operation.

Tamper protection ensures that an Endpoint Detection & Response (EDR) sensor cannot be stopped or manipulated by a malicious actor with access to an endpoint. Linux EDR sensors don’t usually contain tamper protection, as this is primarily a Windows functionality. This is complicated to implement on Linux systems for a variety of reasons.

A specific technical example to show the complexities around this is the <KILL> command in Linux, which sends a signal to a process running on an endpoint. Used with the < -9> or <SIGKILL> flag, the kill command can immediately terminate a given process on an endpoint.

<SIGKILL> signals are incredibly destructive and will terminate a process regardless of what the process wishes to do with the signal. There is no way to intercept these situations on Linux endpoints.

You would need root access on a Linux endpoint to interfere with services. Anyone with this access on Linux is essentially “super admin” and can perform any action on the system. Red Canary is designed to detect behavioral activity considerably earlier in the attack chain, well before an attacker gains root access to an endpoint.

We recommend that customers focus on hardening their endpoints based on industry standards to protect root/sudo privileges on their production Linux endpoints.

For more information regarding Linux security, please download: 15 critical tactics for protecting Linux from cyber attacks

Keeping containers secure in Amazon Web Services (AWS) poses intriguing issues and questions about impact and lateral movement. It’s tempting to off-load the entire responsibility of securing the architecture to AWS, but it is always wise to implement best practices and consider potential vulnerabilities. It is also critical to remember what tools AWS already provides to accomplish this before examining service offerings you may not require.

Fargate is a popular solution in the AWS ecosystem that provides serverless computing for containers while also providing secure application isolation and observability via current AWS services such as CloudWatch Container Insights.

When using eBPF as a telemetry source, the number of file descriptors required is directly proportional to the number of CPUs. As a result, on endpoints with a large number of CPUs, file descriptors may run out.

The sensor will restart, and there will be a message in the cf_system_log.csv file that states No file descriptors available.

To fix this, raise the file ulimit value on the affected endpoint(s). For more information, please read Increase set open file limits.