Going beyond pod in Kubernetes via logging

Note perev. : This note was written by an IT security researcher from Aqua Security, a company specializing in DevSecOps. She is an excellent illustration of the subtleties in Kubernetes configuration that it is important to always keep in mind when serving clusters in production. Of course, if you think about their safety ...

Kubernetes consists of many components, and sometimes combining them in a certain way leads to unexpected results. In this article I will show how a pod launched with root privileges and a mounted /var/log directory of a node can expand the contents of the entire host file system to a user with access to his logs. We will also discuss solutions to this problem.

How Kubernetes sees logs

Have you ever wondered how kubectl logs <pod_name> extracts logs from pod? Who is responsible for collecting logs from containers? And how do they get to your computer?

The following diagram illustrates the process:

Kubelet creates a structure inside the /var/log directory on the host that represents the pods on the host. There is a file 0.log (1) in the directory for our 0.log , but in fact it is a symlink to the container log located in /var/lib/docker/containers . This is all from the point of view of the host.

Kubelet opens endpoint /logs/ (2), which simply works with the HTTP file server in directory (3), making logs available for requests coming from the API server.

Now imagine that we deployed pod with hostPath mounted in /var/log . Such a pod will have access to all the log files on the host. Although this in itself is a potential problem, we can take the next logical step. What if we replace 0.log with a symlink to ... say, /etc/shadow ?

 │ ├── var │ ├── logs │ │ ├── pods │ │ │ ├── default_mypod_e7869b14-abca-11e8-9888-42010a8e020e │ │ │ │ ├── mypod │ │ │ │ │ ├── 0.log -> /etc/shadow │ │ │ │ │ │

Now, trying to download the logs using kubectl logs on the client machine, we get:

 $ kubectl logs mypod failed to get parse function: unsupported log format: "root:*:18033:0:99999:7:::\n"

Kubelet follows the link and reads the contents of the file that it points to (it can be any file on the node).

Since JSON was expected, kubectl crashed after the first line, however, we can easily read the specific lines of the shadow file by running the command with the –-tail=-<line_number> .

This is amazing. Since kubelet follows the symlink, you can use its root privileges to read any file on the node, simply creating a symbolic link inside the pod.

Escape from the pod

Let's go even further. We know that when a pod is launched in Kubernetes, the ServiceAccount token is installed in it. Thus, if the service account allows access to the logs, we can directly access the kubelet and root privileges on the node.

I wrote a proof of concept (POC) demonstrating this attack vector:

deployment of pod with mount point /var/log ;
creating a symbolic link to the root directory of the host;
reading the user's ssh private key on the host.

The following video shows two specific commands that run inside a pod:

lsh == ls (on the host file system);
cath == cat (on the host file system).

Note perev. : Unfortunately, the insertion of content with asciinema was never repaired on the hub, although we have already addressed this problem, therefore we are forced to “embed” the video with the simple link above.

All files involved in this POC can be found in the corresponding GitHub repository . There is another POC script that automatically collects private keys and ServiceAccount tokens from the host file system.

Mounting directories can be dangerous

So is this a vulnerability or just bad practice?

Deploying a pod with a write-open hostPath in /var/log is rare (in addition, there are other ways to abuse the mounting of host secret directories in pod). But even if you knew that mounting /var/log was a dubious practice, you probably did not expect it to allow you to take over the node with such ease.

Before publishing, we contacted the Kubernetes security team to find out if they consider this a vulnerability. They concluded that this was just a sad consequence of mounting a private host directory with write permissions: the risks involved are well documented . However, this vulnerability is quite easy to exploit. In the world there are many projects that use this mount. If you are using one of these projects, remember that your deployment will be vulnerable to this way of hijacking the host.

This method has been tested on Kubernetes 1.15 and 1.13, but most likely affects other versions.

Elimination

Such an “escape” is only possible if pod is running as root. This should generally be avoided . Aqua CSP allows you to set a policy with a minimum of effort that prevents containers from running as root or grants permissions only to a specific group of images that really require root.

Another way is to simply not deploy pods with hostPath with write permissions in /var/log . This approach is not set by default and is not a usual practice, therefore it is necessary to consciously determine it (however, the possibility still remains). But how to check?

We added a new script (hunter) to kube-hunter - our lightweight Open Source tool for testing Kubernetes - which checks the cluster for pods with such dangerous mount points. ( Note : Kube-hunter was present in a recent review of K8s security utilities that we posted on our blog.)

Aqua users can protect themselves from this risk by using the runtime policy to prevent certain volumes from being mounted:

Note perev. : Part of this problem can be solved using Pod Security Policies , namely AllowedHostPaths . However, this is also not protection against symlinks. Finally, as the comments suggest, we can simply limit the launch as root, again guided by the PSP .

Total

Kubernetes is a complex system with a lot of subtleties in security settings, which are not always obvious to the average and even experienced user. In this article, I have shown how, under certain circumstances, innocent logging can lead to potential vulnerability. In most cases, this is not possible, but Kubernetes offers users greater freedom of action that may affect security. It is important to keep this in mind and implement appropriate controls to prevent such errors.