Kubernetes Security Guidelines

Following are the best practices:

  1. Securing Kubernetes Hosts

    • Host administrators and maintainers should interact with cluster nodes through privileged access management systems (or bastion hosts).

    • It is recommended configure the OS and software following the baseline and standards (CIS, NIST).

    • It is recommended regularly scan packages and configuration for vulnerabilities.

    • Implementing essential firewall rules.

    • It is recommended regularly update the OS kernel version.

  2. Securing Kubernetes Components

    • Control Plane

      Perform integrity monitoring for the most critical Kubernetes files.

      File or directory Recommended ownership Permissions
      /etc/kubernetes/manifests/kube-apiserver.yaml root:root 644
      /etc/kubernetes/manifests/kube-controller-manager.yaml root:root 644
      /etc/kubernetes/manifests/kube-scheduler.yaml root:root 644
      /etc/kubernetes/manifests/etcd.yaml root:root 644
      Container Network Interface file root:root 644
      /var/lib/etcd root:root 644
      /etc/kubernetes/admin.conf root:root 644
      /etc/kubernetes/scheduler.conf root:root 644
      /etc/kubernetes/controller-manager.conf root:root 644
      /etc/kubernetes/pki/ root:root ---*
      Kubernetes PKI certificate file: /etc/kubernetes/pki/*.crt root:root 644
      Kubernetes PKI key file: /etc/kubernetes/pki/*.key root:root 600

    • API Server

      Kubeapi server should not be exposed to internet.

      Name Value Default
      Ensure that the –basic-auth-file argument is not set Not set Y
      Ensure that the –token-auth-file parameter is not set Not set Y
      Ensure that the –kubelet-https argument is set to true or doesn't exist Not set Y
      Ensure that the –kubelet-client-certificate and –kubelet-client-key arguments are set as appropriate /etc/kubernetes/pki/apiserver-kubelet-client.crt Y
        /etc/kubernetes/pki/apiserver-kubelet-client.key Y
      Ensure that the –kubelet-certificate-authority argument is set as appropriate N/S Y
      Ensure that the –authorization-mode argument is not set to AlwaysAllow Node, RBAC Y
      Ensure that the –authorization-mode argument includes Node Node, RBAC Y
      Ensure that the –authorization-mode argument includes RBAC Node, RBAC Y
      Ensure that the admission control plugin EventRateLimit is set Not set N
      Ensure that the admission control plugin AlwaysAdmit is not set Not set N
      Ensure that the admission control plugin AlwaysPullImages is set Not set N
      Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used Not set N
      Ensure that the admission control plugin ServiceAccount is set Not set N
      Ensure that the admission control plugin NamespaceLifecycle is set Not set N
      Ensure that the admission control plugin PodSecurityPolicy is set Not set N
      Ensure that the admission control plugin NodeRestriction is set NodeRestriction Y
      Ensure that the –insecure-bind-address argument is not set Not set Y
      Ensure that the –insecure-port argument is set to 0 0  
      Ensure that the –secure-port argument is not set to 0 6443  
      Ensure that the –profiling argument is set to false Not set N
      Ensure that the –audit-log-path argument is set Not set N
      Ensure that the –audit-log-maxage argument is set to 30 or as appropriate Not set N
      Ensure that the –audit-log-maxbackup argument is set to 10 or as appropriate Not set N
      Ensure that the –audit-log-maxsize argument is set to 100 or as appropriate Not set N
      Ensure that the –request-timeout argument is set as appropriate Not set N
      Ensure that the –service-account-lookup argument is set to true Not set N
      Ensure that the –service-account-key-file argument is set as appropriate /etc/kubernetes/pki/sa.pub  
      Ensure that the –etcd-certfile and –etcd-keyfile arguments are set as appropriate /etc/kubernetes/pki/apiserver-kubelet-client.crt Y
        /etc/kubernetes/pki/apiserver-kubelet-client.key Y
      Ensure that the –tls-cert-file and –tls-private-key-file arguments are set as appropriate /etc/kubernetes/pki/apiserver.crt Y
        /etc/kubernetes/pki/apiserver.key Y
      Ensure that the –client-ca-file argument is set as appropriate /etc/kubernetes/pki/ca.crt Y
      Ensure that the –etcd-cafile argument is set as appropriate /etc/kubernetes/pki/ca.crt Y
      Ensure that the –encryption-provider-config argument is set as appropriate Not set N
      Ensure that encryption providers are appropriately configured Not set N
      Ensure that the API server only makes use of strong cryptographic ciphers Not set N

    • Control Network Access to Sensitive Ports

      • Master Node

        Protocol and Port Range Purpose
        TCP 6443- Kubernetes API Server
        TCP 2379-2380 etcd server client API
        TCP 10250 Kubelet API
        TCP 10251 kube-scheduler
        TCP 10252 kube-controller-manager
        TCP 10255 Read-Only Kubelet API

      • Worker Node

        Protocol and Port Range Purpose
        TCP 10250 Kubelet API
        TCP 10255 Read-Only Kubelet API
        TCP 30000-32767 NodePort Services

    • Limit Direct Access to Kubernetes Nodes

      • Limit SSH access to Kubernetes nodes.

      • Use "kubectl exec".

    • Authentication

      • It is recommended to use an IdP server as a provider for user authentication to the Kubernetes API (for example, using OIDC). Cluster administrators are advised not to use service account tokens for authentication.

      • User accounts should be personalized. The names of the service accounts should reflect the purpose access rights of the accounts.

    • Authorization

      • For each cluster, a role-based access model should be developed.

      • Role-Based Access Control (RBAC) should be configured for the Kubernetes cluster. Rights need to be assigned within the project namespace based on least privilege and separation of duties (RBAC-tool).

      • All services should have a unique service account with configured RBAC rights.

      • Cluster administrators and maintainers should interact with the cluster API and infrastructure services through privileged access management systems (Teleport, Boundary).

      • All information systems should be divided in separate namespaces. It is recommended to avoid the situation when the same maintainer team is responsible for different namespaces.

    • Cluster Configuration Security

      • Communication in the cluster between services should be handled using TLS.

      • Note that some components and installation methods may enable local ports over HTTP.

      • It is recommended to use only the latest versions of cluster components.

      • For services with increased security requirements, it is recommended to use a low-level run-time with a high degree of isolation (gVisor, Kata-runtime).

      • Cluster Configuration should be audited regularly (Kube-bench, Kube-hunter, Kubestriker).

    • Restrict access to etcd

      • Secrets should be stored in third-party storage (HashiCorp Vault, Conjur), or in etcd in encrypted form.

      • Secure the etcd datastore with authentication and RBAC policies.

      • Protect kubeconfig files from unauthorized modifications.

      • By defalut etcd data is encrypted with base64 but we need encrypty with strongs algorithms.

    • Controlling access to the Kubelet

      • Production clusters should enable Kubelet authentication and authorization.

    • Securing Kubernetes Dashboard

      • Turn on RBAC, so you can limit the service account the dashboard uses.

      • Do not grant the service account of the dashboard high privileges.

      • Deploy the dashboard with an authenticating reverse proxy, with multi-factor authentication enabled.

  3. Kubernetes Security Best Practices: Build Phase

    • Use latest image/ensure images are up to date.

    • Container registry and the use of an image scanner to identify known vulnerabilities.

    • Ensure That Only Authorized Images are used in your Environment.

    • Use private registries to store your approved images - make sure you only push approved images to these registries.

  4. Kubernetes Security Best Practices: Deploy Phase

    • Kubernetes infrastructure should be configured securely prior to workloads being deployed.

      • What is being deployed - including information about the image being used, such as components or vulnerabilities, and the pods that will be deployed.

      • Where it is going to be deployed - which clusters, namespaces, and nodes.

      • How it is deployed - whether it runs privileged, what other deployments it can communicate with, the pod security context that is applied, if any.

      • What it can access - including secrets, volumes, and other infrastructure components such as the host or orchestrator API.

      • Is it compliant - whether it complies with your policies and security requirements?

    • Use Kubernetes namespaces to properly isolate your Kubernetes resources.

    • Create policies to govern image provenance using the ImagePolicyWebhook.

      • Prevent unapproved images from being used with the admission controller ImagePolicyWebhook to reject pods that use unapproved images including:

        • Images that haven’t been scanned recently.

        • Images that use a base image that’s not explicitly allowed.

    • Implement continuous security vulnerability scanning.

    • Regularly apply security updates to your environment.

      • Upgrading containers is extremely easy with the Kubernetes rolling updates feature - this allows gradually updating a running application by upgrading its images to the latest version.

    • Assess the privileges used by containers.

      • Pod Security Policies are one way to control the security-related attributes of pods, including container privilege levels. These can allow an operator to specify the following:

        • Do not run application processes as root.

        • Use a read-only root filesystem.

        • Use the default (masked) /proc filesystem mount.

        • Do not use the host network or process space - using ""hostNetwork:true"" will cause NetworkPolicies to be ignored since the pod will use its host network.

        • Drop unused and unnecessary linux capabilities.

        • Use SELinux options for more fine-grained process controls.

        • Give each application its own kubernetes service account.

        • Do not mount the service account credentials in a container if it does not need to access the kubernetes API.

    • Apply security context to your pods and containers.

      • Security context setting.

        • SecurityContext->runAsNonRoot

          Indicates that containers should run as non-root user.

        • SecurityContext->capabilities

          Controls the linux capabilities assigned to the container.

        • SecurityContext->readOnlyRootFilesystem

          Controls whether a container will be able to write into the root filesystem.

        • PodSecurityContext->runAsNonRoot

          Prevents running a container with 'root' user as part of the pod.

        • Implementing Open Policy Agent (OPA) for a centralized policy management.

          Implement RBAC policies:

          • Application authorization

          • Kubernetes admission control

          • Service mesh authorization

        • Limiting resource usage on a cluster.

        • Use Kubernetes network policies to control traffic between pods and clusters.

          • Network segmentation is important to ensure that containers can communicate only with those they are supposed.

          • A new network policy API should address the need to create firewall rules around pods, limiting the network access that a containerized can have.

        • Keeps secrets as secrets.

          Even though a pod is not able to access the secrets of another pod, it is crucial to keep the secret separate from an image or pod.

        • Encrypt secrets at rest.

          Always encrypt your backups using a well reviewed backup and encryption solution, and consider using full disk encryption where possible.

          Secret Mode Best Practices
          Storage Encryption at Rest
          Retrival Encryption in Transit
          Rotation Update at reguler intervals in alignment with ABB policy or with NIST standard
          Revocation Ensure the ability to remove secret values entirely in the event of an incident

  5. Kubernetes Security Best Practices: Runtime Phase

    • Use pod security policies to prevent risky containers/pods from being used.

      Pod security policies address several critical security use cases, including:

      • Preventing containers from running with privileged flag - this type of container will have most of the capabilities available to the underlying host. This flag also overwrites any rules you set using CAP DROP or CAP ADD.

      • Preventing sharing of host PID/IPC namespace, networking, and ports - this step ensures proper isolation between Docker containers and the underlying host.

      • Limiting use of volume types - writable hostPath directory volumes, for example, allow containers to write to the filesystem in a manner that allows them to traverse the host filesystem outside the pathPrefix, so readOnly: true must be used.

      • Putting limits on host filesystem use.

      • Enforcing read only for root file system via the ReadOnlyRootFilesystem.

      • Preventing privilege escalation to root privileges.

      • Rejecting containers with root privileges.

      • Restricting Linux capabilities to bare minimum in adherence with least privilege principles.

      Control Aspect Field Names
      Running of privileged containers privileged
      Usage of host namespaces hostPID, hostIPC
      Usage of host networking and ports hostNetwork, hostPorts
      Usage of volume types volumes
      Usage of the host filesystem allowedHostPaths
      Allow specific FlexVolume drivers allowedFlexVolumes
      Allocating an FSGroup that owns the pod's volumes fsGroup
      Requiring the use of a read only root file system readOnlyRootFilesystem
      The user and group IDs of the container runAsUser, runAsGroup, supplementalGroups
      Restricting escalation to root privileges allowPrivilegeEscalation, defaultAllowPrivilegeEscalation
      Linux capabilities defaultAddCapabilities, requiredDropCapabilities, allowedCapabilitieslities
      The SELinux context of the container seLinux
      The Allowed Proc Mount types for the container allowedProcMountTypes
      The AppArmor profile used by containers annotations
      The seccomp profile used by containers annotations
      The sysctl profile used by containers forbiddenSysctls,allowedUnsafeSysctl

    • Container runtime security

      Events that should trigger an alert would include:

      • A shell is run inside a container.

      • A container mounts a sensitive path from the host such as /proc.

      • A sensitive file is unexpectedly read in a running container such as /etc/shadow.

      • An outbound network connection is established.

    • Container sandboxing

      • Preventing containers from loading unwanted kernel modules.

      • Compare and analyze different runtime activity in pods of the same deployments.

      • Monitor network traffic to limit unnecessary or insecure communication.

      • If breached, scale suspicious pods to zero.

    • Container security

      • Scan containers and Pods for vulnerabilities or misconfigurations.

      • Run containers and Pods with the least privileges possible.

      • Use network separation to control the amount of damage a compromise can cause.

      • Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality.

      • Use strong authentication and authorization to limit user and administrator access as well as to limit the attack surface.

    • Logging

      • Log all cases of changing access rights in the cluster.

      • Log all operations with secrets (including unauthorized access tsecrets).

      • Log all cases of changing parameters, system settings, or configuration of the entire cluster (including OS level).

      • The audit logging system should be located outside the Kubernetes cluster.

      • Use third-party security monitoring tool on all cluster nodes (Falco, Sysdig, Aqua Enterpise, NeuVector, Prisma Cloud Compute).