Part_2: Enabling Intel VMX in a Rust Kernel Module

May 01, 2025 • by Benjamin Gallois

🧠 Part_2: Enabling Intel VMX in a Rust Kernel Module

Introduction

In this post, we continue building our hypervisor atop a minimal Rust-based kernel module by implementing the first essential step: enabling VMX operation on an Intel processor. This step involves issuing the VMXON instruction, which transitions the CPU into VMX root mode, allowing the hypervisor to begin managing virtual machines.

There are many acronyms in this post, and their definitions are summarized in the glossary at the bottom.

Rust in the Linux kernel is still experimental, and key functionality, for example, converting virtual to physical memory addresses, is missing in the mainline kernel. As of this writing, such functionality was recently added to Asahi Linux but remains unavailable upstream. We use the same approach as already seen in the previous post to work around this: write a small C helper function to perform the physical address translation, expose it as a symbol, and link it into the Rust module using the kernel build system. For the moment, this part is the only one not written in Rust, but it will be available shortly so we can say that our hypervisor is still 100% Rust-based.

In this post, we will focus on the conceptual and architectural flow of enabling VMX. A working documented reference implementation is available in the companion repository. Although undocumented, we rely on the x86 crate for interaction with processor registers and instructions and refer to Intel® 64 and IA-32 Architectures Software Developer's Manuals.

Understanding VMX

Intel's Virtual Machine Extensions (VMX) provide hardware-level support for virtualization by introducing a dual-mode CPU execution model. VMX defines two main operational modes:

VMX root mode where the hypervisor executes.
VMX non-root mode where guest operating systems run.

These modes allow a hypervisor to retain full control of system resources while guest systems are isolated and prevented from accessing privileged features directly.

VMX introduces new instructions like VMXON, VMLAUNCH, VMRESUME, and VMXOFF, and a key data structure known as the VMCS (Virtual Machine Control Structure), which stores all state and control fields for a virtual machine. Before any guest execution can occur, VMX operation must be activated via the VMXON instruction, which is the subject of this article.

Architecture Overview

Our implementation is separated into two main components:

The Rust-based kernel module allocates memory and triggers hypervisor setup logic.
A Rust crate for hypervisor logic, compiled separately and exposed as C-compatible functions to be linked with the kernel module.

The kernel module performs platform-specific work, such as allocating a physically contiguous 4KB region for VMX use. At the same time, the hypervisor crate takes care of low-level operations such as writing to model-specific registers (MSRs), manipulating control registers, and invoking instructions like VMXON.

All error handling is centralized within the hypervisor crate to keep the code clean and minimal. Errors encountered during hypervisor initialization are captured using a dedicated error enum. Each variant maps to a unique integer code, exposing these as C-compatible symbols straightforwardly. This design enables the kernel module to interface with the hypervisor logic in a minimal, low-overhead manner, receiving structured error feedback without embedding complex error-handling logic.

Memory Preparation for VMXON

To enter VMX operation, the processor requires a VMXON region, a special memory region with strict constraints:

The region must be exactly 4KB and aligned on a 4KB boundary.
It must be allocated from non-paged, physically contiguous memory.
The first 4 bytes must contain the VMCS revision identifier obtained from the IA32_VMX_BASIC MSR.
The rest of the region should be zeroed to ensure a clean and predictable starting state.

We allocate the VMX memory region in the kernel module using the kmalloc function with the appropriate flags to ensure alignment and zeroing. Since malloc returns a virtual address, we use our helper C function to retrieve the corresponding physical address (in waiting for the Rust implementation), which the VMXON instruction requires.

Enabling VMXON

Activating VMXON is a multi-step process. The steps below detail each requirement and how it fits into the logic.

Verifying CPU Capabilities

We begin by confirming that the CPU supports VMX. It can be done using the CPUID instruction, checking whether bit 5 of the ECX register is set. If this bit is unset, the processor does not support VMX, and the process must be aborted.

We also check whether the system is already running a hypervisor. Bit 31 of the same ECX register indicates the presence of another hypervisor, which could interfere with our setup.

It's also important to note that virtualization support must be enabled in the BIOS. If disabled, VMX instructions will fail regardless of software configuration.

Enabling VMX in Control Register CR4

Once VMX capabilities are confirmed, we must enable the VMX instruction by setting bit 13 (VMXE) in control register CR4. This bit allows the CPU to recognize and execute VMX-specific instructions such as VMXON and VMXOFF.

Adjusting CR0 and CR4 Fixed Bits

Before issuing VMXON, both CR0 and CR4 must comply with fixed bit constraints defined by the CPU. Intel enforces these via four MSRs:

IA32_VMX_CR0_FIXED0: Bits in CR0 that must be set.
IA32_VMX_CR0_FIXED1: Bits in CR0 that must be cleared or left unchanged.
IA32_VMX_CR4_FIXED0: Bits in CR4 that must be set.
IA32_VMX_CR4_FIXED1: Bits in CR4 that must be cleared or left unchanged.

The values of CR0 and CR4 must be updated so that all bits required by the FIXED0 MSRs are set to 1, and no bits disallowed by the FIXED1 MSRs are left set to 1.

These MSRs are model-specific and may vary slightly across processor generations, so the hypervisor logic must dynamically query and adjust CR0/CR4 during initialization.

Configuring `IA32_FEATURE_CONTROL` MSR

The IA32_FEATURE_CONTROL MSR is a VMX instruction gatekeeper. This register must be configured appropriately before VMXON will succeed. This MSR includes three essential bits:

Bit 0 (Lock bit): Once set, this register becomes read-only until the next reboot.
Bit 1: Enables VMX inside SMX (Safer Mode Extensions).
Bit 2: Enables VMX outside SMX, which is the mode we need.

If the lock bit is not yet set, we must set both bits 0 and 2 and then write the result back to the MSR. If the register is locked and bit 2 is not set, the CPU will reject VMXON until the system is rebooted.

Initializing the VMXON Region

We next prepare the memory region that will be passed to VMXON:

We write the VMCS revision identifier, obtained from the IA32_VMX_BASIC MSR, into the first 4 bytes of the region.
The remaining 4092 bytes are cleared to zero.

The VMXON region must remain allocated and unchanged for the duration of VMX operation.

Executing the VMXON Instruction

With the VMXON region prepared and the CPU properly configured, we issue the VMXON instruction, passing on the region's physical address. If successful, the processor transitions into VMX root mode, and the hypervisor is officially active.

From this point forward, we can initialize a VMCS, configure guest execution, and begin launching virtual machines. If VMXON fails, we must re-examine every step, particularly MSR configuration, memory alignment, CR0/CR4 compliance, and CPU capability checks. Logging the exact error code returned can significantly aid in identifying the issue.

Conclusion

Enabling VMX via the VMXON instruction is the critical first step in launching a hardware-accelerated hypervisor. It requires precise setup of control registers (especially CR0 and CR4 fixed bits), correct MSR configuration, and carefully aligned physical memory.

In the next post, we will initialize and configure a VMCS, define the guest execution environment, and enter VMX non-root mode to begin running guest code.

Glossary of Abbreviations and Terms

VMX: Virtual Machine Extensions – A set of Intel CPU features providing hardware-level virtualization support.
VMXON: An instruction that enables VMX operation and transitions the processor into VMX root mode.
VMCS: Virtual Machine Control Structure – Defines the execution state and control fields for a virtual machine.
CR0 / CR4: Control Registers 0 and 4 – Used to configure low-level processor features, including VMX enablement.
MSR: Model-Specific Register – Special-purpose CPU registers used to control and report processor features.
IA32_FEATURE_CONTROL: An MSR (index 0x3A) that controls VMX instruction availability and locks VMX configuration on the first write.
IA32_VMX_BASIC: An MSR that provides the VMCS revision ID and information about VMX capabilities.
IA32_VMX_CR0_FIXED0 / FIXED1: MSRs specifying required and allowed bit states in CR0 for VMX to operate correctly.
IA32_VMX_CR4_FIXED0 / FIXED1: MSRs specifying required and allowed bit states in CR4 for VMX to operate correctly.
CPUID: An x86 instruction to query CPU features, such as VMX support and hypervisor presence.
SMX: Safer Mode Extensions – Intel's trusted execution technology for secure and measured booting.
ECX: A general-purpose CPU register used to hold feature flags returned by the CPUID instruction.
BIOS: Basic Input/Output System – Firmware interface that initializes hardware; must enable virtualization support.
Hypervisor: Software or firmware that manages virtual machines. It operates in VMX root mode while guests run in non-root mode.
VMM: Virtual Machine Monitor – A type of hypervisor or software layer that manages virtual machine execution and resource allocation.
kmalloc: A Linux kernel memory allocator for allocating physically contiguous, kernel-resident memory.
Physical Address: The actual memory location used by hardware; required for VMX instructions like VMXON.
Virtual Address: An abstract memory address software uses must be translated to a physical address for hardware operations.