Enable virtualization - Qualcomm Dragonwing Documentation

Qualcomm^® Linux supports hardware-assisted virtualization through KVM (Kernel-based Virtual Machine), which integrates directly into the Linux kernel and runs guest operating systems at near-native performance. Virtual Host Extensions (VHE) are enabled by default on Qualcomm SoCs that implement Armv8.1 or later, allowing the host kernel to run at EL2 while guest kernels and user space run at EL1 and EL0.

Supported targets

All Qualcomm^® Linux development kits support KVM-based virtualization. Each target uses either KVM or Gunyah as its default boot mode. Table: Virtualization support by target

Hardware SoC	Development kit	Default boot mode
IQ-9075 (Lemans)	Qualcomm Dragonwing^™ IQ-9075	KVM
IQ-8275 (Monaco)	Qualcomm Dragonwing^™ IQ-8275	KVM
IQ-615 (Talos)	Qualcomm Dragonwing^™ IQ-615	KVM
QCS6490 (Kodiak)	QCS6490 Qualcomm Dragonwing^™ RB3 Gen 2 Core Development Kit, QCS6490 Dragonwing^™ RB3 Gen 2 optional mezzanine boards	Gunyah
QCS5430 (Kodiak)	QCS5430 Dragonwing^™ RB3 Gen 2 Core Development Kit	Gunyah
IQ-X7181 (Hamoa)	Qualcomm Dragonwing^™ IQ-X7181 EVK	Gunyah
IQ-X5121 (Purwa)	Qualcomm Dragonwing^™ IQ-X5121 EVK	Gunyah

Refer release notes to know about platform capability and support for hypervisor solution.

Prerequisites and Kconfig

Enable the following in the host kernel configuration:

CONFIG_KVM=y
CONFIG_VIRTUALIZATION=y

KVM requires the kernel to boot in exception level 2 (EL2). Verify KVM is available on the device after boot:

ls -l /dev/kvm

Boot the kernel in EL2

Skip this section if the target uses KVM as its default boot mode (see Supported targets), the host boots in EL2. If the target uses Gunyah as its default boot mode, follow these instructions to switch to KVM. Complete the steps in the order presented.

Boot using the default images.

Update the EFI variable to select the KVM device tree overlay:

echo -n "el2kvm" > /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -w -f /var/data
efivar -n 882f8c2b-9646-435f-8de5-f208ff80c1bd-VendorDtbOverlays -p
sync

Reboot into fastboot mode. Run the following command on the device shell:
reboot bootloader
Flash the KVM XBL config image from the host:
fastboot flash xbl_config_a xbl_config_kvm.elf
The xbl_config_kvm.elf file is located under build/tmp/deploy/images/<machine-name> in the Yocto build output. Replace the following:
- <machine-name> by the actual Yocto machine configuration name, for example iq-9075-evk.
Reboot the device.

Build the guest VM artifacts

Before you launch a guest VM, build a guest kernel image and a guest root file system. The host image doesn’t include either component.

Build the guest kernel

The guest runs at EL1 and doesn’t require Qualcomm board device trees or EL2 configuration. Build the guest kernel by using the upstream arm64 defconfig. Use the same kernel source tree that you use for the host:

Build the kernel with the standalone kmake workflow. For host setup, see Build the kernel without Yocto.

kmake O=../kobj-guest ARCH=arm64 defconfig
kmake O=../kobj-guest ARCH=arm64 -j"$(nproc)" Image

Add the virtio front-end drivers the guest needs. The upstream defconfig already enables the common ones; confirm and add any that are missing using a configuration fragment:

cat >> ../kobj-guest/.config <<'EOF'
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_VSOCKETS=y
EOF
kmake O=../kobj-guest ARCH=arm64 olddefconfig
kmake O=../kobj-guest ARCH=arm64 -j"$(nproc)" Image

For the full list of virtio Kconfig symbols by device, see Supported virtio devices.

The build process writes the guest kernel image to ../kobj-guest/arch/arm64/boot/Image.

Build the guest root file system

Use Yocto to build a minimal guest root file system. The same recipe generates both the CPIO ramdisk and the ext4 disk image.

Open a kas shell in your workspace:

kas shell meta-qcom/ci/<SoC>-<board>.yml:meta-qcom/ci/qcom-distro.yml

Add the required image types to build/conf/local.conf so the build produces both a ramdisk and a disk image:
IMAGE_FSTYPES:append = " cpio.gz ext4"
Build the console image, which is the smallest supported image and adequate for a guest:
bitbake qcom-console-image

Collect the artifacts from the deploy directory:

build/tmp/deploy/images/<machine>/qcom-console-image-<machine>.rootfs.cpio.gz
build/tmp/deploy/images/<machine>/qcom-console-image-<machine>.rootfs.ext4

You can use any arm64 root file system as a guest. If you don’t need a Yocto-built guest image, use an aarch64 distribution cloud image with the -drive option instead of rootfs.ext4.

Stage the artifacts on the device

Copy the guest kernel and root file system to the host device and place them in the directory that the following examples use:

# On the build host
mkdir -p guest-artifacts
cp ../kobj-guest/arch/arm64/boot/Image guest-artifacts/
cp build/tmp/deploy/images/<machine>/qcom-console-image-<machine>.rootfs.cpio.gz \
    guest-artifacts/rootfs.cpio.gz
cp build/tmp/deploy/images/<machine>/qcom-console-image-<machine>.rootfs.ext4 \
    guest-artifacts/rootfs.ext4

# Push to the device
adb shell mkdir -p /mnt/overlay/guest
adb push guest-artifacts/. /mnt/overlay/guest/

Confirm the guest kernel is the raw image that QEMU expects:

# On the device
file /mnt/overlay/guest/Image
# Expected: Linux kernel ARM64 boot executable Image, little-endian

Launch a guest VM

Before launching a guest VM, ensure the guest kernel image (Image), root file system CPIO (rootfs.cpio.gz), and root file system image (rootfs.ext4) are present in the /mnt/overlay/guest/ directory on the host. To produce them, see Build the guest VM artifacts.

Using QEMU

Boot with a ramdisk:

qemu-system-aarch64 \
    -M virt -m 2G \
    -kernel /mnt/overlay/guest/Image \
    -initrd /mnt/overlay/guest/rootfs.cpio.gz \
    -cpu host --enable-kvm -smp 4 -nographic

Boot with a root file system image:

qemu-system-aarch64 \
    -M virt -m 2G \
    -kernel /mnt/overlay/guest/Image \
    -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \
    -append "root=/dev/vda" \
    -cpu host --enable-kvm -smp 4 -nographic

Using libvirt

Libvirt manages VMs through the virsh command-line utility and the libvirtd daemon. Define a VM from an XML domain file, then control it with the following commands: Table: Common virsh VM management commands

Command	Purpose
`virsh define <xml-file>`	Register a new VM domain from an XML definition file
`virsh list --all`	List all defined VM domains and their state
`virsh start <domain>`	Start a stopped VM
`virsh console <domain>`	Connect to the VM serial console (Ctrl+] to disconnect)
`virsh shutdown <domain>`	Gracefully shut down a running VM
`virsh undefine <domain>`	Remove a VM definition

Replace the following:

<xml-file> by the path to the libvirt XML domain definition file.
<domain> by the VM domain name as defined in the XML <name> element.

For libvirt XML domain definition examples, see the libvirt domain format documentation.

Virtio device support

Virtio provides a paravirtualized I/O framework for high-performance device emulation between guest VMs and the host. Front-end drivers run in the guest OS; back-end drivers run in QEMU or the kernel. Communication uses virtqueues (ring buffers) to minimize guest-to-host transitions. Table: Supported virtio devices

Device	Host Kconfig	Guest Kconfig	Purpose
virtio-block	—	`CONFIG_VIRTIO_BLK`	Block storage device for guest VMs
virtio-net	—	`CONFIG_VIRTIO_NET`	Network interface for guest VMs
virtio-serial	—	`CONFIG_VIRTIO_CONSOLE`	Serial communication channels between host and guest
virtio-9p	`CONFIG_NET_9P`, `CONFIG_NET_9P_VIRTIO`	`CONFIG_NET_9P`, `CONFIG_NET_9P_VIRTIO`, `CONFIG_9P_FS`	Host-to-guest directory sharing over the 9P protocol
VSOCK	`CONFIG_VSOCKETS`, `CONFIG_VHOST_VSOCK`	`CONFIG_VSOCKETS`, `CONFIG_VIRTIO_VSOCKETS`	Host-to-guest socket communication
virtio-IOMMU	—	`CONFIG_VIRTIO_IOMMU`	Paravirtualized IOMMU for DMA management
virtio-balloon	—	`CONFIG_VIRTIO_BALLOON`	Dynamic guest memory reclaim and allocation

Pass a host directory to a guest VM using the 9P file system:

qemu-system-aarch64 \
    -M virt -m 2G \
    -kernel /mnt/overlay/guest/Image \
    -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \
    -append "root=/dev/vda" \
    -cpu host --enable-kvm -smp 4 -nographic \
    -fsdev local,id=fsdev0,path=/mnt/overlay/test_dir,security_model=passthrough \
    -device virtio-9p-pci,fsdev=fsdev0,mount_tag=hostshare

Mount the shared directory inside the guest:

mount -t 9p -o trans=virtio hostshare <mount-point>

Replace the following:

<mount-point> by the directory inside the guest to mount the shared folder.

Virtual sockets (VSOCK)

VSOCK enables socket communication between guest VMs and the host using a context identifier (CID). The host CID is always 2; guest CIDs start from 3. Table: Reserved CID values

CID	Description
-1	Any address (for binding)
0	Hypervisor
1	Loopback
2	Host

Add VSOCK to a QEMU invocation:

-device vhost-vsock-pci,guest-cid=<cid>

Replace the following:

<cid> by the CID to assign to the guest VM, for example 73.

Device passthrough

Physical devices can be passed through to a guest VM using VFIO (PCI), libusb (USB), or a chardev backend (UART). Identify USB devices on the host:

lsusb

Identify PCI devices on the host:

lspci

Add USB passthrough to a QEMU invocation using the vendor and product IDs from lsusb:

-device qemu-xhci -device usb-host,vendorid=<vid>,productid=<pid>

Add PCI passthrough using the domain, bus, slot, and function from lspci:

-device vfio-pci,host=<bus>:<slot>.<function>

Replace the following:

<vid> and <pid> by the USB vendor and product ID in hex, for example 0x0781 and 0x5567.
<bus>, <slot>, <function> by the PCI address components from lspci output.

For UART passthrough, configure a chardev backend pointing to the host TTY device and expose it to the guest as a virtserialport.

Observability and maintenance

KVM traces

Enable KVM event tracing via tracefs:

echo 1 > /sys/kernel/tracing/events/kvm/enable
cat /sys/kernel/tracing/trace

QEMU trace events can be redirected to the kernel ftrace buffer. To launch a guest VM with virtio traces enabled:

qemu-system-aarch64 \
    -M virt -m 2G \
    -kernel /mnt/overlay/guest/Image \
    -drive file=/mnt/overlay/guest/rootfs.ext4,if=virtio,format=raw \
    -append "root=/dev/vda" \
    -cpu host --enable-kvm -smp 4 -nographic \
    -trace "virtio*"

Watchdog

QEMU emulates an I6300 ESB watchdog device, exposed to the guest as a standard watchdog character device. Enable it in the guest kernel:

CONFIG_I6300ESB_WDT=y

Add the watchdog to the libvirt domain XML:

<devices>
  <watchdog model='i6300esb' action='reset'/>
</devices>

The action attribute controls behavior on timeout: reset restarts the guest, poweroff shuts it down. For details, see the libvirt watchdog documentation.

Remote command execution

The QEMU guest agent (qemu-ga) allows commands to be run on a guest VM from the host without a network connection. Enable qemu-ga in the guest OS user space and configure it through a virtio-serial interface. Use virsh qemu-agent-command with the guest-exec subcommand to execute commands remotely:

virsh qemu-agent-command <domain> '{"execute":"guest-exec","arguments":{"path":"cat","arg":["/proc/meminfo"],"capture-output":true}}'

Replace the following:

<domain> by the VM domain name.

For more information, see QEMU Guest Agent.

​Supported targets

​Prerequisites and Kconfig

​Boot the kernel in EL2

​Build the guest VM artifacts

​Build the guest kernel

​Build the guest root file system

​Stage the artifacts on the device

​Launch a guest VM

​Using QEMU

​Using libvirt

​Virtio device support

​Host-to-guest file sharing (virtio-9p)

​Virtual sockets (VSOCK)

​Device passthrough

​Observability and maintenance

​KVM traces

​Watchdog

​Remote command execution

Supported targets

Prerequisites and Kconfig

Boot the kernel in EL2

Build the guest VM artifacts

Build the guest kernel

Build the guest root file system

Stage the artifacts on the device

Launch a guest VM

Using QEMU

Using libvirt

Virtio device support

Host-to-guest file sharing (virtio-9p)

Virtual sockets (VSOCK)

Device passthrough

Observability and maintenance

KVM traces

Watchdog

Remote command execution