IVSHMEM with the KVMFR module (Recommended)

The kernel module implements a basic interface to the IVSHMEM device for Looking Glass allowing DMA GPU transfers.

Prerequisites

The Linux kernel headers for your kernel version are required for building along with dkms to manage the module when you upgrade your kernel.

apt-get install linux-headers-$(uname -r) dkms

Then switch to the module/ directory

cd module/

Installing

To install the module into DKMS, run

dkms install "."

Loading

Using the value you should have already calculated as per Determining Memory, simply use modprobe with the parameter static_size_mb, for example:

modprobe kvmfr static_size_mb=32

Alternatively you can make this setting permanent by creating the file /etc/modprobe.d/kvmfr.conf with the following content.

options kvmfr static_size_mb=32

After this has been done, simply running modprobe kvmfr is all that is required.

Note

Don’t forget to adjust static_size_mb to your needs.

systemd-modules-load

For convenience, you may load the KVMFR module when starting your computer. We can use the systemd-modules-load.service(8) service for this task.

Create the file /etc/modules-load.d/kvmfr.conf with the following contents:

# KVMFR Looking Glass module
kvmfr

This will now run the next time you start your machine.

Verification

If everything has been done correctly you should see the following output in dmesg:

kvmfr: creating 1 static devices

You should now also have the character device /dev/kvmfr0

$ ls -l /dev/kvmfr0
crw------- 1 root root 242, 0 Mar  5 05:53 /dev/kvmfr0

Warning

If you start the VM prior to loading the module, QEMU will create the file /dev/kvmfr0 as a regular file. You can confirm if this has happened by running ls -l /dev/kvmfr0 and checking if the file size is greater then zero, or the permissions do not start with c. If this has occurred, you must delete the file and reload the module.

Permissions

The module will create the /dev/kvmfr0 node, which represents the KVMFR interface. To use the interface, you need permission to access it by either creating a udev rule to ensure your user can read and write to it, or simply change its ownership manually, i.e.:

sudo chown user:kvm /dev/kvmfr0

As an example, you can create a new file in /etc/udev/rules.d/99-kvmfr.rules with the following contents:

SUBSYSTEM=="kvmfr", OWNER="user", GROUP="kvm", MODE="0660"

(replace user with your username)

libvirt

Using the module in libvirt requires adding a <qemu:commandline> block to your libvirt XML configuration. That block, in turn, requires modifying the XML domain namespace. To modify the namespace edit the <domain> tag at the top of your XML config to:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

then add one of the <qemu:commandline> blocks below based on your QEMU/libvirt versions.

Note

Make sure to add both the block and the domain namespace change in a single editing session prior to saving it. Failure to do so will cause libvirt to reject the changes.

Starting with QEMU 6.2 and libvirt 7.9, JSON style QEMU configuration is the default syntax. Users running QEMU 6.2 or later and libvirt 7.9 or later, should use this XML block to configure their VM for kvmfr:

<qemu:commandline>
  <qemu:arg value="-device"/>
  <qemu:arg value="{'driver':'ivshmem-plain','id':'shmem0','memdev':'looking-glass'}"/>
  <qemu:arg value="-object"/>
  <qemu:arg value="{'qom-type':'memory-backend-file','id':'looking-glass','mem-path':'/dev/kvmfr0','size':33554432,'share':true}"/>
</qemu:commandline>

Note

The 'size' tag represents the size of the shared memory device in bytes. Once you determine the proper size of the device as per Determining Memory, use the figure you got to calculate the size in bytes:

size_in_MB x 1024 x 1024 = size_in_bytes

If you are running QEMU older than 6.2 or libvirt older than 7.9, please use legacy syntax for IVSHMEM setup:

<qemu:commandline>
  <qemu:arg value="-device"/>
  <qemu:arg value="ivshmem-plain,id=shmem0,memdev=looking-glass"/>
  <qemu:arg value="-object"/>
  <qemu:arg value="memory-backend-file,id=looking-glass,mem-path=/dev/kvmfr0,size=32M,share=yes"/>
</qemu:commandline>

Note

Using the legacy syntax on QEMU 6.2/libvirt 7.9 may cause QEMU to abort with the following error message: “error: internal error: ... PCI: slot 1 function 0 not available for pcie-root-port, in use by ivshmem-plain”

Running libvirt this way violates AppArmor and cgroups policies, which will block the VM from running. These policies must be amended to allow the VM to start.

Tip

If you are not sure, you likely have cgroups also as this is usually deployed and configured by default by most distributions when you install libvirt.

AppArmor

Create /etc/apparmor.d/local/abstractions/libvirt-qemu if it doesn’t exist and add the following:

# Looking Glass
/dev/kvmfr0 rw,

cgroups

Edit the file /etc/libvirt/qemu.conf and uncomment the cgroup_device_acl block, adding /dev/kvmfr0 to the list. To make this change active you then must restart libvirtd

sudo systemctl restart libvirtd.service

QEMU

If you are using QEMU directly without libvirt, add the following arguments to your qemu command line:

-device ivshmem-plain,id=shmem0,memdev=looking-glass
-object memory-backend-file,id=looking-glass,mem-path=/dev/kvmfr0,size=32M,share=yes

Note

The size argument must be the same size you passed to the static_size_mb argument when loading the kernel module.