Network communication

Before using a confidential VM, the cloud customer should always certify it first. Pod-to-pod communication needs to be secured in Kubernetes. Especially when a pod is running on a different node. But there is a very simple solution. Earlier in the SGX and Knative architecture, we briefly mentioned Istio, which is used here to secure pod-to-pod communication. One feature of Istio is that it provides mTLS (mutual TLS) for all pods. With mTLS, both parties authenticate themselves with a certificate. Each pod is injected with a sidecar container called an Envoy, which acts as a proxy for all incoming and outgoing traffic of a pod and implements mTLS. Envoy is a standalone open source project designed specifically to act as a proxy in a cloud environment. In the pod itself, traffic can be sent unencrypted. In a confidential VM, this is not a problem because communication within a pod goes through localhost and thus never leaves the confidential VM. This has the huge advantage that applications do not need to be aware of the mTLS connection, so no code changes are required.

To issue the required certificates, Istio integrates a CA into the Kubernetes control plane. There are two ways to have the CA issue certificates. The first option is that the CA watches for newly created pods, creates a certificate and key for them, and stores them in the etcd store. The other is that when the sidecar container envoy is created, it sends a certificate and key request to the node-agent. The node-agent runs on each node, creates the private key, and sends a CSR with it's credentials to the Istio CA. After receiving the certificate, it forwards it, along with the private key, back to the requesting envoy for use. Both options use authentication and authorization policies defined by a cloud customer to block traffic. These policies are stored within the etcd and can be manipulated by an adversary on disk without integrity protection. This potentially bypasses Istio's authentication and authorization mechanisms. Therefore, additional integrity protection of the file system is a must. It is better to use the node-agent method because the private keys are not stored in etcd. Kubernetes secrets are accessible to all containers in the same namespace. A cloud customer has to put all containers in different namespaces, which is prone to misconfigurations. The node agent only caches the keys, providing a smaller attack surface by leaking only a small amount of keys if compromised. The certificate issued includes the namespace and name of the pod's associated Kubernetes service. Other pods are reached through their services, so the certificate can be used to verify that the correct pod has been reached.