Skip to content

PCI: liveupdate: Keep preserved devices accessible across Live Update

David Matlack kexec linux-mm linux-pci lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest10 patches 2 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/10PCI: Factor out BAR sizing from pci_read_bases()drivers/pci/probe.c+21 -140 replies
2/10PCI: liveupdate: Avoid sizing BARs of incoming devicesdrivers/pci/liveupdate.c drivers/pci/liveupdate.h drivers/pci/probe.c +1 more+178 -40 replies
3/10PCI: Factor out bridge window probing from pci_read_bridge_windows()drivers/pci/probe.c+28 -161 reply
4/10PCI: liveupdate: Avoid probing windows of incoming bridgesdrivers/pci/liveupdate.c drivers/pci/liveupdate.h drivers/pci/probe.c +1 more+131 -10 replies
5/10PCI: Skip programming bridge windows that are already programmeddrivers/pci/setup-bus.c+41 -110 replies
6/10PCI: liveupdate: Claim resources of incoming devices during scanningdrivers/pci/liveupdate.c drivers/pci/liveupdate.h drivers/pci/probe.c+112 -00 replies
7/10PCI: liveupdate: Never assign resources of incoming devicesdrivers/pci/setup-bus.c+4 -00 replies
8/10PCI: liveupdate: Prevent releasing resource windows of incoming bridgesdrivers/pci/setup-bus.c+6 -00 replies
9/10PCI: liveupdate: Don't realign resources of incoming devicesdrivers/pci/pci.c+10 -00 replies
10/10PCI: liveupdate: Avoid enabling INTx on incoming devicesdrivers/pci/liveupdate.c drivers/pci/liveupdate.h drivers/pci/pci.c +2 more+63 -11 reply

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 00/10] PCI: liveupdate: Keep preserved devices accessible across Live Update

This series can be found on GitHub:

https://github.com/dmatlack/linux/tree/liveupdate/pci/p2pdma/v1

This series builds on the PCI core support for Live Update [1] to allow memory transactions targeting preserved PCI devices to proceed uninterrupted across a Live Update. The base series allows preserved devices to perform DMA to system memory across a Live Update, but the next kernel still sizes, probes, and may reassign the BARs and bridge windows of preserved devices during enumeration, which disables decoding or moves address ranges out from under anything that is still accessing them.

Use cases include:

- P2PDMA between preserved devices, e.g. between devices assigned to the same VM via VFIO.

- MMIO from Orphaned VMs [2], whose vCPUs keep running in guest mode on dedicated CPUs during kexec, and therefore keep accessing the BARs of devices assigned to them throughout the Live Update.

To support this, the BAR addresses and bridge windows of preserved devices must not change across a Live Update, and the next kernel must not disable decoding or forwarding on preserved devices or on the bridges above them while enumerating them and assigning resources.

[1] https://lore.kernel.org/linux-pci/[email protected]/
[2] https://lore.kernel.org/kvm/[email protected]/

Series Overview
---------------

This series implements the following:

1. Avoid sizing the BARs of incoming preserved devices. Sizing a BAR requires disabling decoding. Instead, have the previous kernel pass the BAR masks (the values the BARs return when read after writing all 1s) and decode the BARs from those (patches 1-2).

2. Avoid probing which windows incoming preserved bridges implement, which temporarily disables the prefetchable window. Instead, add a generic mechanism for passing the results of probes that require writing to the device, and use it to pass which windows each bridge implements (patches 3-4).

3. Skip programming bridge windows that already contain the values being written, so that pci_setup_bridge() does not temporarily disable the windows of preserved bridges. This applies to all bridges, not only preserved ones (patch 5).

4. Claim the BARs and bridge windows of incoming preserved devices, as programmed by the previous kernel, while scanning and before anything sizes or assigns resources, regardless of how the host bridge driver sets up resources. Panic if they cannot be claimed, since that means something else may be decoding the same address range (patch 6).

5. Prevent the PCI core from assigning, releasing, or realigning the resources of incoming preserved devices (patches 7-9).

6. Avoid enabling INTx on incoming preserved devices, both while probing whether INTx masking works and when the device is enabled (patch 10).

All new behavior applies only to incoming preserved devices, i.e. only after a Live Update and only until pci_liveupdate_finish(), except for patch 5, which avoids redundant writes for all bridges.

ABI Changes
-----------

This series uses the optional feature flags added to struct pci_ser in v10 of the base series to append new fields to struct pci_dev_ser:

- PCI_SER_OPTIONAL_BAR_MASKS: @bar_masks and @rom_mask.

- PCI_SER_OPTIONAL_PROBED: @probed_supported and @probed, a bitmask of probe results (which windows a bridge implements, and whether INTx masking is broken).

The ABI only contains hardware state (register values and the results of probing hardware), not kernel-internal state such as IORESOURCE_* flags. If the previous kernel did not pass a field, the next kernel falls back to sizing or probing the device as before.

Merge Plan
----------

This series applies on top of v10 of the PCI core support for Live Update series [1], which is targeting Linux 7.5. This series is targeting Linux 7.6, or 7.5 if feasible.

Both series will be merged through liveupdate.git, matching the agreed-upon tree in the MAINTAINERS entry for PCI LIVE UPDATE.

Testing
-------

This series was tested in conjunction with v5 of the VFIO PCI driver series:

https://lore.kernel.org/kvm/[email protected]/

The full set of patches used for testing can be found on GitHub.

https://github.com/dmatlack/linux/tree/liveupdate/pci/p2pdma/v1-with-vfio

Testing was performed using vfio_pci_liveupdate_kexec_test with the igb selftest driver, which keeps the device performing DMA while kexec happens. QEMU traced every PCI config write, and each run checked that the next kernel did not write to the BAR, ROM, bridge window, or bus number registers of the preserved device or any bridge upstream of it, did not change their decode or Bus Master Enable bits, and did not enable INTx on them, and that their BARs and windows had the same values before and after kexec. The following configurations were tested:

- x86 (q35): igb behind a root port.

- x86 (q35): igb behind a PCIe switch (root port, upstream port, downstream port), with a non-preserved device on another downstream port of the same switch.

- arm64 (virt, SMMUv3, generic ECAM host bridge): igb behind a root port. This exercises the path where bridge windows are not read from hardware during scanning.

Future Work
-----------

Please see the PCI core Live Update Roadmap:

https://lore.kernel.org/linux-pci/[email protected]/

David Matlack (10):
PCI: Factor out BAR sizing from pci_read_bases()
PCI: liveupdate: Avoid sizing BARs of incoming devices
PCI: Factor out bridge window probing from pci_read_bridge_windows()
PCI: liveupdate: Avoid probing windows of incoming bridges
PCI: Skip programming bridge windows that are already programmed
PCI: liveupdate: Claim resources of incoming devices during scanning
PCI: liveupdate: Never assign resources of incoming devices
PCI: liveupdate: Prevent releasing resource windows of incoming
bridges
PCI: liveupdate: Don't realign resources of incoming devices
PCI: liveupdate: Avoid enabling INTx on incoming devices

 drivers/pci/liveupdate.c    | 363 +++++++++++++++++++++++++++++++++++-
 drivers/pci/liveupdate.h    |  26 +++
 drivers/pci/pci.c           |  18 ++
 drivers/pci/probe.c         | 110 ++++++++---
 drivers/pci/setup-bus.c     |  62 ++++--
 include/linux/kho/abi/pci.h |  52 +++++-
 6 files changed, 589 insertions(+), 42 deletions(-)

base-commit: 312200e9211ede3f9a03cc17d094805e06e4674e

Signature · 1 line
2.56.0.212.g129ede20d9-goog