Skip to content

pcie io space multiplexing for bootable pass through HVM domain

Isaku Yamahata xen-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest0 of 5 patches 4 repliesIn review

Patches in v1 No applies recorded yet

None of the 5 patches the cover letter announces are in this archive.

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/5] pcie io space multiplexing for bootable pass through HVM domain

This patch series is for PCIe IO space multiplexing patch for commit.

It is not uncommon that a big iron for server consolidation has many (e.g. > 16) PCIe slots. It will hold many domains, and the administrator wants them to boot from pass through devices. But currently up to 16 hvm domains can boot from pass through device. This patch series address it by multiplexing PCI IO space access.

Usage:
Add the following options to dom0 kernel command line

guestdev=<device path>+iomul
(append "+iomul")
or
guestiomuldev=[<segment>:]<bus>:<dev>[,[<segment:><bus>:dev]][,...] (In this case, don't forget to add related options. pciback.hide,
reassign_resources or guestdev.)

Then dom0 Linux will allocate IO ports which are shared by specified devices. And ioemu will automatically recognize IO-port-shared devices. The unspecified devices will be treated same as before.
Note: Specifying unit to share IO port is PCI slot (device), on the other
hand the unit of guestdev is function.
If you specify a function to guestdev with "+iomul", all the functions of the given slot will share IO port even if you specify some of functions.

This patch series addresses the issue by multiplexing IO space access. The patches are composed of
Linux part: backport: preliminary patch
Linux part: IO space ressignment code and multiplexing driver Linux part: guestdev kernel parameter support. Linux part: add kernel command line to reserve io/memory space
xen part: udev script for the driver
ioemu part: make use of the PCIe io space multiplexing driver

Details:
PCI expansion ROM BIOS often uses IO port access to boot from its device and Linux as dom0 exclusively assigns IO space to downstream PCI bridges and the assignment unit of PCI bridge IO space is 4K. So the only up to 16 PCIe device can be accessed via IO space within 64K IO ports. So on virtualized environment, it means only up to 16 guest domains can boot from such pass-through devices.

The solution is to assign the same IO port region to pci devices under same PCIe switch and disable IO bit in command register. When accessing to one of IO port shared devices, the IO bit of the device is enabled, and then issues IOIO.

Limitation:
- PCI devices or root complex integrated endpoints aren't supported.
- IO port of IO shared devices can't be accessed from dom0 Linux device driver.
But those wouldn't be big issues because PCIe specification discourages the use of IO space and recommends that IO space should be used only for bootable device with ROM code. OS device driver should work without IO space access.

Test:
I don't have a machine with complicated PCIe topology nor many PCIe cards.
- PCI hotplug was tested with Linux fakephp.
- Only pci device (not bridge) hot plug/remove was tested.
- I have tested with only single multifunction PCIe

Changes from take2:
- rebased
- add kernel command line to reserve memory/io space for unused pci slot

change take 2:
- support PCI hotplug
- guestdev kernel paremeter.

thanks