Memory Layout

All of an enclave's data is stored in a special area of DRAM called Processor Reserved Memory (PRM). Any software other than enclaves that access their own memory is not allowed to access the PRM and will return the abort page if attempted (see the figure below). Software requires the involvement of a CPU core to perform memory accesses. The CPU is able to deny memory access to any software, even privileged software such as the hypervisor and OS kernel. In addition, any peripheral device accesses memory via Direct Memory Access (DMA). Unlike running software, DMA does not require the involvement of any execution core of the CPU, thus bypassing the CPU. Fortunately, the memory controllers reside on the CPU and are programmed to deny DMA to the PRM.

Intel SGX Memory Layout

Inside the PRM is a region of memory called the Enclave Page Cache (EPC). The EPC consists of multiple 4KB pages, with each page associated with only a single enclave. The page contains either the contents of a single enclave or it's associated data structures. The allocation and initialization of EPC pages is performed by SGX-specific CPU instructions from the hypervisor if the enclave is running in a virtualized environment, or from the OS kernel on bare metal. This requires the system software to implement SGX functionality and expose it to the application that wants to run inside an enclave. These features have been implemented in the Linux kernel since 5.11. The advantage of using the system software for address translation is that it does not need to be significantly modified to work with SGX. To initialize an EPC page, data is copied from a non-PRM memory page to an EPC page. Note that the EPC and regular pages are the same size to simplify copying.

Because untrusted software indirectly allocates and initializes EPC pages, the CPU must verify the legitimacy of the instructions it receives. To accomplish this, SGX introduces an Enclave Page Cache Map (EPCM) that has an entry for each EPC page. It is used to store additional data that is tightly associated with a single EPC page. This additional data includes a valid bit, a page type, and the enclave identity. A zero in the valid bit indicates an uninitialized EPC page and a one indicates an initialized page. The page type indicates the content type (enclave content or associated data structures) and the enclave identity is used to identify the enclave to which this page belongs. This allows the CPU, through the EPCM, to track the ownership of each EPC and deny enclaves access to EPC pages that they do not own. An enclave's code and data are stored with the page type set to PT_REG and are accessible by the owning enclave.

In addition, a dedicated EPC page for each enclave contains metadata known as the SGX Enclave Control Structure (SECS) and is identified by the page type PT_SECS. The SECS contains the measurements and attributes of the enclave. Therefore, it is associated with the enclave identity. The enclave identity in the EPCM points to an enclave's SECS page. Note that the SECS does not map to an enclave's address space and is accessible only by the CPU. Thus, even the enclave associated with the SECS can't access its SECS. The reason for this is to prevent inadvertent modification and access to the information contained there. Among the attributes is the init flag, which indicates the initialization status of the enclave and is an important flag for the enclave lifecycle.

Because the system software is responsible for address translation in SGX, each enclave receives a virtual address space from it. This address space is shared with the host application. Within its virtual address space, each enclave selects a virtual address space to map to it's code and private data in the EPC. This is provided by the CPU and is called the Enclave Linear Address Range (ELRANGE). Virtual memory accesses outside the ELRANGE access the memory of the host process.

Another control structure, the Thread Control Structure (TCS), allows concurrent thread execution of the same code within an enclave. They are located in the EPC with the page type set to PT_TCS. TCS, like SECS, is only accessible through SGX instructions. In addition, special debugging instructions allow access to the TCS, which must be enabled on enclave launch and are normally disabled. Context switching, in and out of the enclave, is done with the TCS. The TCS contains an instruction pointer (RIP) that acts as a well-defined entry point to the enclave, called OENTRY. If a hardware exception occurs while enclave code is being executed in a thread, the thread's execution context must be securely stored to prevent information leakage to untrusted software. It is stored in EPC pages, with the EPCM page type set to PT_REG, and is called a State Save Area (SSA). SSAs are included in the TCS. Inside the SECS there is an additional field called SSAFRAMESIZE that specifies the maximum size of an SSA.