Design Decision

In Knative's ConfigMap feature, you can enable the "runtimeClassName" specification within a Knative service, which is disabled by default. This feature allows Knative to work with Kata containers. QEMU/KVM supports starting confidential virtual machines protected by AMD SEV-SNP, and because kata containers support QEMU/KVM as a hypervisor, confidential microVMs can be started. With this setting, each serverless container can be launched inside it's own private microVM, making this architecture comparable to Intel SGX. However, unlike SGX, this has some serious security risks.

To actually be able to manage containers inside a microVM, kata-containers runs a kata agent inside it. On the one hand, the kata agent needs to be trusted, since it's inside the trusted environment with the serverless container and thus able to read its secrets. However, this is not the biggest problem because, as with Knative, trusting the kata code is a valid assumption due to the fact that it is open source and has a medium sized community behind it. Therefore, the code of the Kata agent can be verified before using it. Kata containers allow multiple containers to be deployed within a microVM. Therefore, anyone with access to the container runtime can deploy malicious containers inside the confidential VM. Since the isolation provided by containers is not sufficient, this is a huge security risk. To fix this, the kata agent must be rewritten to allow only one container per microVM, and once the container is shut down, the microVM must be destroyed. This is not the only problem. Containers are loaded/unloaded into the microVM after it is started and thus are not covered by the measurement of the confidential virtual machine. Using the secure storage (confidential KMS) would not work because all microVMs that have Kata enabled will have the same measurement. A different solution would have to be used here. Also, to be able to deploy confidential microVMs, a bare-metal machine is a requirement because most cloud providers only allow you to rent a confidential virtual machine, not a virtual machine that can launch SEV VMs.

In addition, AMD does not allow you to deploy a cVM inside a cVM. Therefore, the whole thing is associated with very high costs due to the bare-metal machine. For these reasons, the idea was discarded, but it was worth mentioning since this is the architecture that is close to SGX.