Know what you don’t know (KWYDK)
when Should I deploy a single-tenant or multi-tenant K8 ?
That depends. How risky are the the other applications and tenants running on the the shared platform? What is the legal landscape?
Sometimes, vendor products come wildly insecure or they simply require high level of permissions for the type of automation they perform. Sometimes you don’t want to mix those riskier application stacks with your HR, Finance or Marketing systems. (Eggs in one basket vs. Diversification of risk). Sometimes, the business operating model has a greater degree of autonomy and resists centralied services.
What about the public cloud?
The public cloud platform security is often overlooked in the threat models and K8 guides. Remember that IAM admins, K8 admins, Network admins all have high levels of inside permissions. More importantly, who you grant these roles to controls the propagation of the platform across the enterprise and it’s exposure throughout the business and internet.
Meaning, the better IAM operating model you have the more or less clusters being deployed and secured. The cloud IAM operating models ultimately controls your CICD pipeline and you ability to enforce standards.
- What is the RACI and how does it support the cloud IAM model ?
- What is the VPC strategy and where is the K8 hosted within the vpc ?
- How well is the K8 isolated within the VPC ?
- What are your K8 instances role permissions, are they too broad ?
- How do you provisioning, deploy, update the K8 infrastructure? Jenkins, Code-Pipeline, Travis etc ?
- Are you protecting the cloud meta-data API ?
Should I block Docker.io?
That depends. Some engineering teams will likely still need access to this service to bring in well known common images and build “golden images”.
However, you likely already have teams bringing in untrusted binaries via docker and using them as part of their day to day SDLC. Are you willing to disrupt development and deal with the blow back? Most enterprise organization’s scale and impact are too large.
In an ideal world, you would be to lock down egress to docker.io and restrict egress via an engineering teams build pipeline (Jenkins/Travis/Spinnaker) so they can make trusted images and push them into your internal repo. However in the real world the pushback may be too large and not align with the businesses risk appetite, you may need to compensate with other security controls such as Desktop EDR and monitoring for data exfiltrations on the network to Docker.io or even worse assume the containers already have egress to command and control servers.
What about vulnerability management?
There are plenty of open source and enterprise tools for this. Check on the k8 tools page on my site. A few things to consider when tackling this problem ….
- Kubernetes misconfiguration
- Container misconfiguration
- Network policy misconfiguration
- Container application / libraries / kernel vulnerabilities
- Cloud IAM and VPC misconfiguration
Regardless of whether you build a centralized or de-centralized operating model, you should build this capabilities into the infrastructure and application pipelines. The security tooling should be available to build teams to call and see their results at will to accelerate development and reduce friction.
The policies should be owned by the Security Team and the ability to limit or remove policies should kept to only admins of the tools and not the App-teams. The results of the security scans should always be piped back to a centralized monitoring platform managed by the Security Team, so they can get the “Full picture” of the landscape and not just a single tree.
How do I build an inventory of Docker/K8?
That depends. Containers can be running on-premise, in-the public cloud and on Desktops. Here’s a couple of suggestions….
- Script out Public Cloud APIs that “List” / “Describe” all the “native/managed” Kubernetes that are running
- Use MDM or other device management to search for Docker running on the local Desktops
- Monitor Netflow and proxy logs for use of Docker.io, HELM etc. etc.
- “Run it yourself” type of solutions you’ll need to scan for the well known ports, use a network scanner.. Here’s an example of K8 but Open Shift and Rancher and Pivotal may use other ports
What about detection of IOC in K8?
Traditional tools are still catching up because K8 runs in an overlay network and sometimes packets never need to leave the host at all. This means that there is an entirely hidden network world within the container runtime that most people don’t see and the containers operate at a level just above the operating system which limits monitoring to syscal activity from the container daemon. This all means that local commands to do bad stuff or network attacks may go unnoticed by traditional tools until the attack tries to go outside of the k8 runtime and over your network IDS/IPS. A few suggestions … Focus on putting sensors on the platform itself which specialize in this type of attack chain
- Falco open-source project
- Crowdstrike Falcon
The above can be used as advanced sensors with enhanced visibility which point back to your Splunk or other SIEMs and integrated to your existing CIRT. Here are a few areas to focus on,
- Privilege containers spawning more containers (There is legitimate use-cases for this but they are rare, once you hammer this out then false positive are removed and everything else is suspicious)
- Attempts to mount local-volumes to root directories on the local system
- Attempts to read or overwrite /etc/ or /bin/ directories (Remember most containers are meant to be READ only file systems)
- Attempts to access to pods default service account credentials from within a container and query the kube API
- Attempts to access the public cloud meta-data API
- Attempts to netcat,ssh,sudo,curl,wget from within a container (Although there may be legitimate needs to use these tools within non-prod or build pipelines it’s unlikely that a Web App or Database in production needs to implement these capabilities)
what are they other important things to learn?
What’s the architecture of k8
What does OWASP have to say?
What is K8 Security Best Practices?
How do you authenticate to k8 ?
How do get authorized on k8 ?
What are my storage options and how to I secure them ?
How to limit privileged access & priv esc?
How to limit lateral movement within the virtual network?
How can I lock down my K8 to a specific registry?
How can I mange secrets in K8?