Pocket Guide: Machine and Container Security

Idea

Any IT solution can be viewed as a system of interconnected elements, typically machines or containers. That is why machine and container security is fundamentally important.

Let’s start with Anti-virus (AV), which is a special software on the machine, which monitors and prevents infections. With time, it has grown into Endpoint Detection and Response (EDR), which includes real-time advanced monitoring, behaviour analysis, threat hunting, and incident response.

Regular patching is critical for machine and container security. Unfortunately, regular and frequent patching is in a conflict of interests with operational stability. Every change, including minor patching, comes with the risk of service disruption, and we want to avoid it. Good systems are designed in such a way that frequent patching and service recovery are both quick and easy. Automation is the key.

Another technique is hardening, which is application of known good secure configuration settings (baseline) to the machine or container. For example, “Do not generally permit containers to be run as the root user” is a Kubernetes hardening setting. Hardening also includes minimisation of the installed software packages and monitoring for configuration drift. Centre of Internet Security is known for producing security baselines.

Container base image sourcing is an important security consideration. The images should be carefully selected and sourced from the trusted repositories.

Close to hardening is application whitelisting. It is a technique to allow execution only explicitly approved applications. Both hardening and whitelisting serve to minimise the software packages which could run on the machine, but application whitelisting does it in explicit, deny-by-default mode. Maintaining the whitelist is challenging due to frequent software changes. Blacklisting is the alternative to whitelisting, but is not as effective.

Another interesting technology is Runtime Application Self-Protection (RASP). It detects attacks and protects applications in real-time by analyzing information from inside the application. This can complement other security measures, as RASP has the best knowledge of the application context.

Last, but not least, is vulnerability scanning. We scan the list of installed software packages on the machines or containers, then compare it with the list of known vulnerabilities in a database like NVD. The resulting report is then used to determine the patching action. Such scanning should be performed frequently.

Application examples

Organisation should have a policy of prohibiting the deployment of containers with known high or critical vulnerabilities.

Risks

The frequency and volume of known vulnerabilities increases every year. Organisations have to up the patching game every year or risk falling behind. Too many critical and high vulnerabilities may also cause dangerous desensitisation. “There cannot be so many critical vulnerabilities, security people just crying wolves” – is an example of such thinking.

Opposite to desensitisation is patch overanalysis. Neither of the extremes are good.

Container images are layered – they are build by pulling software from different sources, which emphasices the supply chain attack risk.

Quirks

While It is considered best practice to use “distroless” container images, there is no definition of what it distroless. All images contain some number of packages from different distributions. In my opinion, the quality of image maintenance is more important than the number of packages included. Check this article for similar opinion.

While AV/EDR is highly desired on user machines, its application in containers is questionable, as containers are mostly stateless in the modern architectures. Exising containers are typically not patched, they are just replaced with the newer, patched version of the containers.

Further reading

Please check out the map of cyber security controls below. It is simple yet awesome.