PCI: liveupdate: PCI core support for Live Update
Subsystems (5)
v11 cover PCI: liveupdate: PCI core support for Live Update
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/liveupdate/pci/base/v11
This series introduces support in the PCI core for Live Update, enabling drivers to preserve PCI devices across a kexec-based kernel update without interrupting the device. This lets devices, such as ones assigned to VMs through VFIO, keep running across a host kernel upgrade.
Preserved devices can keep doing DMA to system memory throughout the Live Update. Devices may be behind bridges, but VFs, devices that need device-specific ACS quirks, and CardBus are not supported. Support for P2P and VF preservation will be addressed in future series.
Series Overview
---------------
This series implements the following to support PCI device preservation across Live Update:
1. Set up a File-Lifecycle-Bound (FLB) handler to track and preserve PCI-specific state (struct pci_ser) across Live Update using Kexec Handover (KHO).
2. Add APIs for drivers to register devices for preservation (outgoing) and for the PCI core to identify preserved devices during enumeration (incoming).
3. Automatically preserve all upstream bridges for any preserved endpoint. Use reference counting to ensure bridges remain preserved as long as any downstream device is preserved.
4. Keep the Requester ID (bus, device, function) of preserved devices stable for the duration of their preservation by adopting the secondary bus numbers, subordinate bus numbers, and ARI Forwarding Enable of preserved bridges. After a Live Update, pci=assign-busses and pci=noari do not override this adopted state.
5. Save and restore the ACS Control register and Egress Control Vector across resets for all devices (patch 8), so that adopted ACS controls survive a reset.
6. Keep memory transactions to/from preserved devices routed consistently by adopting Access Control Services (ACS) controls across a Live Update, from the endpoint up to the root port.
7. Freeze each device's preservation status when it is shut down, and avoid disabling bus mastering on preserved devices in the PCI shutdown path, so preserved devices can keep doing DMA during kexec.
8. Add Documentation/PCI/liveupdate.rst, built from the kernel-doc in drivers/pci/liveupdate.c and include/linux/kho/abi/pci.h, describing the FLB API, device tracking, and the division of responsibilities between the PCI core, drivers, and userspace.
Merge Plan
----------
This series is intended to go through liveupdate.git, the tree listed in the new PCI LIVE UPDATE MAINTAINERS entry.
v10 and a follow-up patch were applied to the lu-pci-preservation branch of liveupdate.git. This version is meant to replace them, and applies on v7.4-rc1, the base of that branch.
Testing
-------
VFIO
~~~~
This series was tested with v5 of the vfio-pci Live Update series, rebased onto this series and the luo-internal-api branch of liveupdate.git (which the VFIO series depends on):
https://lore.kernel.org/kvm/[email protected]/
The full set of patches used for testing can be found on GitHub.
https://github.com/dmatlack/linux/tree/liveupdate/pci/base/v11-with-vfio
Testing was performed using the VFIO selftests, including the new Live Update selftests:
- vfio_pci_liveupdate_uapi_test
- vfio_pci_liveupdate_kexec_test
The tests were run in a QEMU-based VM environment (using an emulated Intel IGB device connected to a PCIe Root Port to exercise bridge support) and on bare metal using an Intel Emerald Rapids server with 8x Intel DSA PCIe devices and 1x NVMe device, all of which were preserved across a Live Update by vfio_pci_liveupdate_kexec_test.
PCI Selftests
~~~~~~~~~~~~~
This series was also tested with a new set of PCI Live Update selftests:
https://github.com/dmatlack/linux/tree/liveupdate/pci/base/v11-with-selftests
Each test preserves a PCI device across a Live Update in a QEMU VM and checks that the PCI core keeps the device, and every bridge above it,
running: same BDF and bus numbers, Bus Master Enable left set, ARI Forwarding Enable and ACS controls adopted from the previous kernel, and no PCI config writes that disrupt the device during the Live Update. The tests cover endpoints on the root bus, below a Root Port, below a switch, and below two levels of switches, as well as booting the next kernel with command lines that would otherwise renumber buses or change ARI and ACS. All tests pass on x86_64 (q35) and arm64 (virt), except for the pci=assign-busses test, which is skipped on arm64 (the option is x86-only).
Future Work
-----------
Please see the PCI core Live Update Roadmap:
https://lore.kernel.org/linux-pci/[email protected]/
Changelog
---------
v11:
- Rebase onto v7.4-rc1, the base of lu-pci-preservation in liveupdate.git
- Fold in "PCI: liveupdate: Factor out pci_dev_ser allocation and release", which was posted and applied as a follow-up to v10 [1]:
introduce the struct pci_dev_ser allocation and release helpers in "Track outgoing preserved PCI devices", and split pci_liveupdate_preserve_device() into separate paths for new and already-preserved devices in "Auto-preserve upstream bridges across Live Update" (suggested by Bjorn; the follow-up was reviewed and tested by Chris Down)
- Don't allocate an ACS save buffer for devices that rely on device-specific ACS quirks (Alex)
- Adopt the ACS controls of preserved devices even if they match a device-specific ACS quirk, logging a warning instead of programming ACS from scratch, and panic if a preserved device has no ACS save buffer (Alex)
- Clean up commit messages, comments and documentation: fix inaccuracies, remove filler and duplicated text, document that pci=noari does not override adopted ARI Forwarding Enable, and note in the documentation that devices cannot be preserved or unpreserved once they are shut down
- Panic if any outgoing device was never unpreserved when the outgoing FLB is freed, instead of leaving dangling dev->liveupdate.outgoing pointers, and share the check with the incoming side (Sashiko)
[1] https://lore.kernel.org/linux-pci/[email protected]/
v10: https://lore.kernel.org/linux-pci/[email protected]/
v9: https://lore.kernel.org/linux-pci/[email protected]/
v8: https://lore.kernel.org/linux-pci/[email protected]/
v7: https://lore.kernel.org/linux-pci/[email protected]/
v6: https://lore.kernel.org/linux-pci/[email protected]/
v5: https://lore.kernel.org/linux-pci/[email protected]/
v4: https://lore.kernel.org/linux-pci/[email protected]/
v3: https://lore.kernel.org/kvm/[email protected]/
v2: https://lore.kernel.org/kvm/[email protected]/
v1: https://lore.kernel.org/kvm/[email protected]/
rfc: https://lore.kernel.org/kvm/[email protected]/
David Matlack (13):
PCI: liveupdate: Set up FLB handler for the PCI core
PCI: liveupdate: Track outgoing preserved PCI devices
PCI: liveupdate: Track incoming preserved PCI devices
PCI: liveupdate: Document driver binding responsibilities
PCI: liveupdate: Auto-preserve upstream bridges across Live Update
PCI: liveupdate: Preserve bus numbers during Live Update
PCI: Refactor matching logic for pci_dev_acs_ops
PCI: Save and restore the ACS Control register and Egress Control
Vector
PCI: liveupdate: Adopt ACS controls in incoming preserved devices
PCI: liveupdate: Adopt ARI Forwarding Enable on preserved bridges
PCI: liveupdate: Freeze preservation status during shutdown
PCI: liveupdate: Do not disable bus mastering on preserved devices
during kexec
Documentation: PCI: Add documentation for Live Update
Documentation/PCI/index.rst | 1 + Documentation/PCI/liveupdate.rst | 35 + .../admin-guide/kernel-parameters.txt | 11 +- Documentation/core-api/liveupdate.rst | 5 + MAINTAINERS | 15 + drivers/pci/Kconfig | 14 + drivers/pci/Makefile | 1 + drivers/pci/liveupdate.c | 1011 +++++++++++++++++ drivers/pci/liveupdate.h | 72 ++ drivers/pci/pci-driver.c | 8 +- drivers/pci/pci.c | 169 ++- drivers/pci/pci.h | 5 + drivers/pci/probe.c | 16 +- drivers/pci/quirks.c | 58 +- include/linux/kho/abi/pci.h | 96 ++ include/linux/pci.h | 7 + include/linux/pci_liveupdate.h | 72 ++ 17 files changed, 1556 insertions(+), 40 deletions(-) create mode 100644 Documentation/PCI/liveupdate.rst create mode 100644 drivers/pci/liveupdate.c create mode 100644 drivers/pci/liveupdate.h create mode 100644 include/linux/kho/abi/pci.h create mode 100644 include/linux/pci_liveupdate.h
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 1/13 PCI: liveupdate: Set up FLB handler for the PCI core
Set up a File-Lifecycle-Bound (FLB) handler so that the PCI core can preserve its own state across a Live Update kexec.
Preserving a PCI device across kexec requires preserving two independent sets of state:
- Driver state, e.g. everything vfio-pci needs so that userspace can keep using the device in the new kernel. The driver preserves this itself and the PCI core is not involved.
- PCI core state, e.g. which devices are preserved, so that the new kernel knows not to disturb them while they are still running and doing DMA. That is what this commit adds, serialized into struct pci_ser.
Userspace, not the kernel, decides which devices are preserved, and it does so through the Live Update Orchestrator's (LUO) support for file
preservation: a driver exposes a file that represents a single PCI device, and userspace preserves that device with ioctl(LIVEUPDATE_SESSION_PRESERVE_FD) on that file. Binding preservation to a file gives it proper lifecycle management, e.g. the preservation is undone if userspace cancels it or goes away. How a driver exposes that file is up to the driver and invisible to the PCI core (vfio-pci variant drivers, the first intended use-case, use their per-device cdev).
LUO only knows that a file was preserved; it does not know that it represents a PCI device, or which one. Bridging that gap, drivers register their liveupdate_file_handler with the PCI core:
pci_liveupdate_register_flb(driver_file_handler); pci_liveupdate_unregister_flb(driver_file_handler);
LUO then keeps a reference count on the PCI core's FLB for the files preserved by that handler, and that reference count drives the lifetime of struct pci_ser:
- On the first preserved file, luo_flb_file_preserve_one() calls pci_flb_preserve(), which allocates struct pci_ser and preserves it with KHO.
- On the last unpreserved file (i.e. preservation cancelled), liveupdate_flb_put_outgoing() calls pci_flb_unpreserve(), which unpreserves and frees struct pci_ser.
- In the next kernel, pci_flb_retrieve() hands the PCI core the struct pci_ser built by the previous kernel, whenever the PCI core asks for it (e.g. during enumeration), and pci_flb_finish() frees it once the PCI core is done with it.
So the flow for preserving a device, once a driver has registered, looks like this:
ioctl(LIVEUPDATE_SESSION_PRESERVE_FD)
luo_session_preserve_fd()
luo_preserve_file()
luo_flb_file_preserve()
luo_flb_file_preserve_one() # only on the first preserved file
pci_flb_preserve() # alloc + KHO-preserve pci_ser
fh->ops->preserve() # driver callback, e.g. vfio-pci
Note that struct pci_ser is deliberately not allocated when a driver calls pci_liveupdate_register_flb(). A driver can be loaded for the lifetime of the machine without ever preserving a device, and there is no reason to allocate memory and hand it to the next kernel in that case. Letting LUO own the lifetime also means the PCI core does not have to duplicate LUO's reference counting and unwind logic for preservation failures, session aborts and fd close, and the incoming side (retrieve/finish) comes from the same object rather than requiring a separate KHO FDT entry owned by the PCI core.
struct pci_ser begins with required_flags and optional_flags, rather than a version number, so that the ABI can be extended without breaking compatibility between kernels. A kernel that sets a bit in required_flags requires the next kernel to understand the corresponding feature, while bits in optional_flags can be safely ignored. The size of each struct pci_dev_ser is recorded in struct pci_ser as well, so that new fields can be appended to it.
Note: This commit only allocates struct pci_ser and preserves it across Live Update. A subsequent commit adds pci_liveupdate_preserve(), the API drivers call from their fh->ops->preserve() callback to tell the PCI core exactly which devices are being preserved.
Note: There is no reason to check for kho_is_enabled() since it can be assumed to return true. If KHO was not enabled then Live Update would not be enabled and these routines would never run.
Note: PCI_LIVEUPDATE depends on 64BIT so that a device's domain and BDF always fit in an unsigned long, which a subsequent commit uses as the key when looking up incoming preserved devices.
--- Documentation/core-api/liveupdate.rst | 4 + MAINTAINERS | 13 ++ drivers/pci/Kconfig | 14 ++ drivers/pci/Makefile | 1 + drivers/pci/liveupdate.c | 204 ++++++++++++++++++++++++++ include/linux/kho/abi/pci.h | 96 ++++++++++++ include/linux/pci.h | 1 + include/linux/pci_liveupdate.h | 30 ++++ 8 files changed, 363 insertions(+) create mode 100644 drivers/pci/liveupdate.c create mode 100644 include/linux/kho/abi/pci.h create mode 100644 include/linux/pci_liveupdate.h
Documentation/core-api/liveupdate.rst +4 -0
@@ -1,5 +1,7 @@ .. SPDX-License-Identifier: GPL-2.0 +.. _luo: + ======================== Live Update Orchestrator ========================
@@ -18,6 +20,8 @@ LUO Preserving File Descriptors .. kernel-doc:: kernel/liveupdate/luo_file.c :doc: LUO File Descriptors +.. _flb: + LUO File Lifecycle Bound Global Data ==================================== .. kernel-doc:: kernel/liveupdate/luo_flb.c
MAINTAINERS +13 -0
@@ -21545,6 +21545,19 @@ L: [email protected] S: Supported F: Documentation/PCI/pci-error-recovery.rst +PCI LIVE UPDATE +M: David Matlack <[email protected]> +R: Pasha Tatashin <[email protected]> +R: Mike Rapoport <[email protected]> +R: Pratyush Yadav <[email protected]> +L: [email protected] +L: [email protected] +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git +F: drivers/pci/liveupdate.c +F: include/linux/kho/abi/pci.h +F: include/linux/pci_liveupdate.h + PCI MSI DRIVER FOR ALTERA MSI IP L: [email protected] S: Orphan
drivers/pci/Kconfig +14 -0
@@ -271,6 +271,20 @@ config VGA_ARB_MAX_GPUS Reserves space in the kernel to maintain resource locking for multiple GPUS. The overhead for each GPU is very small. +config PCI_LIVEUPDATE + bool "PCI Live Update Support" + depends on PCI && LIVEUPDATE && 64BIT + help + Enable PCI core support for preserving PCI devices across Live + Update. This, in combination with support in a device's driver, + enables PCI devices to run and perform memory transactions + uninterrupted during a kexec for Live Update. + + Say Y if you use Live Update and need to preserve PCI devices + across it. + + If unsure, say N. + source "drivers/pci/hotplug/Kconfig" source "drivers/pci/controller/Kconfig" source "drivers/pci/endpoint/Kconfig"
drivers/pci/Makefile +1 -0
@@ -16,6 +16,7 @@ obj-$(CONFIG_PROC_FS) += proc.o obj-$(CONFIG_SYSFS) += pci-sysfs.o slot.o obj-$(CONFIG_ACPI) += pci-acpi.o obj-$(CONFIG_GENERIC_PCI_IOMAP) += iomap.o +obj-$(CONFIG_PCI_LIVEUPDATE) += liveupdate.o endif obj-$(CONFIG_OF) += of.o
drivers/pci/liveupdate.c new file +204 -0
204 lines not on this page · raw message
include/linux/kho/abi/pci.h new file +96 -0
96 lines not on this page · raw message
include/linux/pci.h +1 -0
@@ -43,6 +43,7 @@#include <uapi/linux/pci.h> #include <linux/pci_ids.h> +#include <linux/pci_liveupdate.h> #define PCI_STATUS_ERROR_BITS (PCI_STATUS_DETECTED_PARITY | \ PCI_STATUS_SIG_SYSTEM_ERROR | \
include/linux/pci_liveupdate.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* PCI Live Update support (Public/Driver API) +* +* Copyright (c) 2026, Google LLC. +* David Matlack <[email protected]> +*/ +#ifndef LINUX_PCI_LIVEUPDATE_H +#define LINUX_PCI_LIVEUPDATE_H + +#include <linux/liveupdate.h> +#include <linux/types.h> + +struct pci_dev ; + +#ifdef CONFIG_PCI_LIVEUPDATE +int pci_liveupdate_register_flb (struct liveupdate_file_handler *fh); +void pci_liveupdate_unregister_flb (struct liveupdate_file_handler *fh); +#else +static inline int pci_liveupdate_register_flb (struct liveupdate_file_handler *fh) +{ +return -EOPNOTSUPP ; +} + +static inline void pci_liveupdate_unregister_flb (struct liveupdate_file_handler *fh) +{ +} +#endif + +#endif /* LINUX_PCI_LIVEUPDATE_H */
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 2/13 PCI: liveupdate: Track outgoing preserved PCI devices
In reply to David MatlackAdd APIs to allow drivers to notify the PCI core of which devices are being preserved across a Live Update for the next kernel, i.e. "outgoing" devices.
Drivers must notify the PCI core when devices are preserved so that the PCI core can update its FLB data (struct pci_ser) and track the list of outgoing devices. pci_liveupdate_preserve() notifies the PCI core that a device must be preserved across Live Update. pci_liveupdate_unpreserve() reverses this (cancels the preservation of the device).
Later patches use this to leave preserved devices running during shutdown and kexec, and the list of preserved devices is handed off to the next kernel.
Warn if a device is released while it is still preserved.
Panic if any device is still preserved when the outgoing FLB is freed, i.e. some driver failed to call pci_liveupdate_unpreserve(). Freeing the outgoing state would otherwise leave dangling dev->liveupdate.outgoing pointers behind.
For now, the API only supports preserving non-VF devices on a root bus (not behind a PCI-to-PCI bridge).
--- MAINTAINERS | 1 + drivers/pci/liveupdate.c | 267 ++++++++++++++++++++++++++++++++- drivers/pci/liveupdate.h | 21 +++ drivers/pci/probe.c | 2 + include/linux/pci.h | 3 + include/linux/pci_liveupdate.h | 21 +++ 6 files changed, 314 insertions(+), 1 deletion(-) create mode 100644 drivers/pci/liveupdate.h
MAINTAINERS +1 -0
@@ -21555,6 +21555,7 @@ L: [email protected] S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git F: drivers/pci/liveupdate.c +F: drivers/pci/liveupdate.h F: include/linux/kho/abi/pci.h F: include/linux/pci_liveupdate.h
drivers/pci/liveupdate.c +266 -1
303 lines not on this page · raw message
drivers/pci/liveupdate.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* PCI Live Update support (core API) +* +* Copyright (c) 2026, Google LLC. +* David Matlack <[email protected]> +*/ +#ifndef DRIVERS_PCI_LIVEUPDATE_H +#define DRIVERS_PCI_LIVEUPDATE_H + +#include <linux/pci.h> + +#ifdef CONFIG_PCI_LIVEUPDATE +void pci_liveupdate_cleanup_device (struct pci_dev *dev); +#else +static inline void pci_liveupdate_cleanup_device (struct pci_dev *dev) +{ +} +#endif + +#endif /* DRIVERS_PCI_LIVEUPDATE_H */
drivers/pci/probe.c +2 -0
@@ -24,6 +24,7 @@#include <linux/pm_runtime.h> #include <linux/bitfield.h> #include <trace/events/pci.h> +#include "liveupdate.h" #include "pci.h" static struct resource busn_resource = {
@@ -2492,6 +2493,7 @@ static void pci_release_dev(struct device *dev) pci_dev =to_pci_dev (dev);pci_release_capabilities (pci_dev); +pci_liveupdate_cleanup_device (pci_dev);pci_release_of_node (pci_dev);pcibios_release_device (pci_dev);pci_bus_put (pci_dev->bus);
include/linux/pci.h +3 -0
@@ -600,6 +600,9 @@ struct pci_dev {u8 tph_mode;/* TPH mode */ u8 tph_req_type;/* TPH requester type */ #endif +#ifdef CONFIG_PCI_LIVEUPDATE +struct pci_liveupdate liveupdate; +#endif };static inline struct pci_dev *pci_physfn (struct pci_dev *dev)
include/linux/pci_liveupdate.h +21 -0
@@ -8,14 +8,26 @@ #ifndef LINUX_PCI_LIVEUPDATE_H #define LINUX_PCI_LIVEUPDATE_H +#include <linux/kho/abi/pci.h> #include <linux/liveupdate.h> +#include <linux/spinlock_types.h> #include <linux/types.h> +/** + * struct pci_liveupdate - PCI Live Update state for a struct pci_dev + * @outgoing: State preserved for the next kernel. + */ +struct pci_liveupdate { + struct pci_dev_ser *outgoing; +}; + struct pci_dev; #ifdef CONFIG_PCI_LIVEUPDATE int pci_liveupdate_register_flb(struct liveupdate_file_handler *fh); void pci_liveupdate_unregister_flb(struct liveupdate_file_handler *fh); +int pci_liveupdate_preserve(struct pci_dev *dev); +void pci_liveupdate_unpreserve(struct pci_dev *dev); #else static inline int pci_liveupdate_register_flb(struct liveupdate_file_handler *fh) {
@@ -25,6 +37,15 @@ static inline int pci_liveupdate_register_flb(struct liveupdate_file_handler *fhstatic inline void pci_liveupdate_unregister_flb (struct liveupdate_file_handler *fh) { } + +static inline int pci_liveupdate_preserve (struct pci_dev *dev) +{ +return -EOPNOTSUPP ; +} + +static inline void pci_liveupdate_unpreserve (struct pci_dev *dev) +{ +}#endif #endif /* LINUX_PCI_LIVEUPDATE_H */
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 3/13 PCI: liveupdate: Track incoming preserved PCI devices
In reply to David MatlackIdentify devices that the previous kernel preserved across Live Update ("incoming" devices) when they are enumerated, and add pci_liveupdate_finish() and pci_liveupdate_is_incoming() for drivers.
The previous kernel passes state about the devices it preserved across kexec, and the PCI core needs this state to know which devices require special handling.
Add pci_liveupdate_setup_device(), which is called during device setup to fetch the serialized state (struct pci_ser) from the Live Update Orchestrator. The first time this happens, pci_flb_retrieve() runs and loads the struct pci_dev_ser entries into an xarray so that they can be looked up efficiently.
If a device is found in the xarray, the PCI core stores a pointer to its state in dev->liveupdate.incoming until the driver calls pci_liveupdate_finish(). This pointer allows the PCI core and drivers to apply Live Update-specific logic to incoming devices in subsequent commits.
Drivers can check if a device is an incoming preserved device (e.g. during probe) by calling pci_liveupdate_is_incoming().
Any error in pci_flb_retrieve() is fatal, since the previous kernel handed off devices that are still performing DMA and this kernel cannot safely take them over without their state. This includes the previous kernel setting a bit in required_flags that this kernel does not know how to handle, and struct pci_dev_ser entries that are smaller than PCI_DEV_SER_MIN_SIZE. Fields appended to struct pci_dev_ser by the previous kernel are ignored.
pci_flb_finish() panics if any preserved device was never finished, since that means a driver failed to call pci_liveupdate_finish().
--- drivers/pci/liveupdate.c | 261 +++++++++++++++++++++++++++++++-- drivers/pci/liveupdate.h | 5 + drivers/pci/probe.c | 3 + include/linux/pci_liveupdate.h | 13 ++ 4 files changed, 273 insertions(+), 9 deletions(-)
drivers/pci/liveupdate.c +252 -9
@@ -45,6 +45,11 @@* it (e.g. during enumeration), and pci_flb_finish() frees it once the PCI * core is done with it. * +* State handed over by the previous kernel is trusted. The PCI core validates +* it only far enough to detect an incompatible or corrupt handover, and makes +* no attempt to defend against deliberate modification, since a previous kernel +* able to corrupt preserved state is able to corrupt arbitrary memory anyway. +* * Call Flow * --------- *
@@ -69,14 +74,17 @@* * # ---------------- kexec ---------------- * -* # New kernel: the PCI core asks for the previous kernel's state -* liveupdate_flb_get_incoming() -* luo_flb_retrieve_one() # first request only -* pci_flb_retrieve() # previous kernel's struct pci_ser +* # New kernel: PCI enumeration +* pci_setup_device() +* pci_liveupdate_setup_device() +* liveupdate_flb_get_incoming() +* luo_flb_retrieve_one() # first request only +* pci_flb_retrieve() # previous kernel's struct pci_ser * * # Userspace: ioctl(LIVEUPDATE_SESSION_FINISH) * luo_file_finish_one() * fh->ops->finish() # driver callback +* pci_liveupdate_finish(dev) # release this device's pci_dev_ser * luo_flb_file_finish() * liveupdate_flb_put_incoming() # last incoming file only * pci_flb_finish() # free struct pci_ser
@@ -93,6 +101,20 @@* This allows the PCI core to keep its FLB data (struct pci_ser) up to date * with the list of **outgoing** preserved devices for the next kernel. * +* After kexec, whenever a device is enumerated, the PCI core looks it up in the +* incoming FLB data (struct pci_ser) to see if it is an **incoming** preserved +* device (i.e. preserved by the previous kernel). +* +* Drivers must notify the PCI core when an **incoming** device is done +* participating in the incoming Live Update with the following API: +* +* * ``pci_liveupdate_finish(pci_dev)`` +* +* The PCI core does not enforce any ordering of ``pci_liveupdate_finish()`` and +* ``pci_liveupdate_preserve()``, i.e., a PCI device can be **outgoing** +* (preserved for next kernel) and **incoming** (preserved by previous kernel) +* at the same time. +* * Restrictions * ============ *
@@ -140,13 +162,39 @@ struct pci_flb_outgoing {struct kho_block_set block_set; }; +/** +* struct pci_flb_incoming - Incoming PCI FLB object +* @ser: The incoming struct pci_ser from the previous kernel. +* @xa: Xarray used to quickly lookup devices in @ser. +* @block_set: The KHO block set holding the incoming devices. +*/ +struct pci_flb_incoming { +struct pci_ser *ser; +struct xarray xa; +struct kho_block_set block_set; +}; + +/* +* Bits in struct pci_ser::required_flags that this kernel knows how to handle. +* No flags are currently defined. +*/ +#define PCI_SER_REQUIRED_FLAGS_SUPPORTED 0ULL + +/* PCI_LIVEUPDATE depends on 64BIT, so domain and bdf always fit. */ +static unsigned long pci_ser_xa_key (u32 domain,u16 bdf) +{ +return (unsigned long )domain <<16 | bdf; +} +/* * Panic if any struct pci_dev_ser in @block_set is still in use, i.e. some -* driver failed to call pci_liveupdate_unpreserve(). This runs once the last -* reference to the FLB is dropped, so no unpreserve can be in flight. +* driver failed to call pci_liveupdate_unpreserve() (outgoing) or +* pci_liveupdate_finish() (incoming). This runs once the last reference to the +* FLB is dropped, so no unpreserve or finish can be in flight. * * Report the devices from the FLB rather than by walking struct pci_dev, so -* that devices that were destroyed while preserved are identified as well. +* that devices that never showed up after kexec, or that were destroyed while +* preserved, are identified as well. */ static void pci_check_all_devices_released (struct pci_ser *ser,struct kho_block_set *block_set)
@@ -211,17 +259,78 @@ static void pci_flb_unpreserve(struct liveupdate_flb_op_args *args)kfree (outgoing); } +/* +* Any failure here is fatal. The previous kernel handed over devices that are +* still performing DMA, and this kernel cannot identify them without this +* state. Continuing would let the PCI core reassign bus numbers and rebind +* drivers underneath live devices, so fail loudly instead of unwinding. +*/ static int pci_flb_retrieve (struct liveupdate_flb_op_args *args) { +struct pci_ser *ser =phys_to_virt (args->data); +struct pci_flb_incoming *incoming; +struct pci_dev_ser *dev_ser; +struct kho_block_set_it it; +int ret; +pr_debug ("Retrieving struct pci_ser (0x%llx)\n" , args->data); - args->obj =phys_to_virt (args->data); + +if (ser->required_flags & ~PCI_SER_REQUIRED_FLAGS_SUPPORTED ) +panic ("Incoming PCI FLB requires unsupported features (0x%llx)\n" , + ser->required_flags & ~PCI_SER_REQUIRED_FLAGS_SUPPORTED ); + +/* +* The previous kernel may have appended fields to struct pci_dev_ser, +* which this kernel ignores, but every entry must be at least the +* minimum size. +*/ +if (ser->dev_ser_size <PCI_DEV_SER_MIN_SIZE ) +panic ("Incoming PCI FLB has unsupported struct pci_dev_ser size (%u)\n" , + ser->dev_ser_size); + + incoming =kzalloc_obj (*incoming); +if (!incoming) +panic ("Failed to allocate struct pci_flb_incoming\n" ); + + incoming->ser = ser; +xa_init (&incoming->xa); + +kho_block_set_init (&incoming->block_set, ser->dev_ser_size); + ret =kho_block_set_restore (&incoming->block_set, ser->devices); +if (ret) +panic ("Failed to restore devices KHO block set (%d)\n" , ret); + +kho_block_set_it_init (&it, &incoming->block_set); +while ((dev_ser =kho_block_set_it_read_entry (&it))) { +unsigned long key; + +if (!dev_ser->refcount) +continue ; + + key =pci_ser_xa_key (dev_ser->domain, dev_ser->bdf); + ret =xa_insert (&incoming->xa, key, dev_ser,GFP_KERNEL ); +if (ret) +panic ("Failed to insert PCI device %04x:%02x:%02x.%d into xarray (%d)\n" , + dev_ser->domain,PCI_BUS_NUM (dev_ser->bdf), +PCI_SLOT (dev_ser->bdf),PCI_FUNC (dev_ser->bdf), + ret); + } + + args->obj = incoming;return 0 ; }static void pci_flb_finish (struct liveupdate_flb_op_args *args) { +struct pci_flb_incoming *incoming = args->obj; +pr_debug ("Finished struct pci_ser (0x%llx)\n" , args->data); -kho_restore_free (args->obj); +pci_check_all_devices_released (incoming->ser, &incoming->block_set); + +xa_destroy (&incoming->xa); +kho_block_set_destroy (&incoming->block_set); +kho_restore_free (incoming->ser); +kfree (incoming); }static struct liveupdate_flb_ops pci_liveupdate_flb_ops = {
@@ -423,6 +532,67 @@ void pci_liveupdate_unpreserve(struct pci_dev *dev) }EXPORT_SYMBOL_GPL (pci_liveupdate_unpreserve); +static struct pci_flb_incoming *pci_liveupdate_flb_get_incoming (void ) +{ +struct pci_flb_incoming *incoming =NULL ; +int ret; + + ret =liveupdate_flb_get_incoming (&pci_liveupdate_flb, (void **)&incoming); + +/* +* -EOPNOTSUPP (Live Update is not enabled), -ENODATA (no incoming FLB +* data) and -ENOENT (no incoming FLB data matches +* pci_liveupdate_flb.compatible) mean there is no incoming PCI state. +* +* Any other error means there is incoming PCI state but it could not be +* retrieved (pci_flb_retrieve() returned an error or LUO failed to +* acquire a reference to pci_liveupdate_flb_ops.owner), which is fatal. +*/ +if (ret == -EOPNOTSUPP || ret == -ENODATA || ret == -ENOENT ) +return NULL ; + +if (ret) +panic ("Failed to retrieve incoming FLB data (%d)\n" , ret); + +return incoming; +} + +static void pci_liveupdate_flb_put_incoming (void ) +{ +liveupdate_flb_put_incoming (&pci_liveupdate_flb); +} + +void pci_liveupdate_setup_device (struct pci_dev *dev) +{ +struct pci_flb_incoming *incoming; +struct pci_dev_ser *dev_ser; +unsigned long key; + +guard (rwsem_write)(&pci_liveupdate.rwsem); + + incoming =pci_liveupdate_flb_get_incoming (); +if (!incoming) +return ; + + key =pci_ser_xa_key (pci_domain_nr (dev->bus),pci_dev_id (dev)); + dev_ser =xa_load (&incoming->xa, key); + +/* +* This device was not preserved across Live Update, or it was preserved +* but has already been probed and gone through pci_liveupdate_finish(), +* e.g. due to removing and re-adding the device. Either way, it's not +* treated as incoming-preserved. +*/ +if (!dev_ser || !dev_ser->refcount) { +pci_liveupdate_flb_put_incoming (); +return ; + } + +pci_info (dev,"Device was preserved by previous kernel across Live Update\n" ); + dev->liveupdate.incoming = dev_ser; +pci_liveupdate_flb_put_incoming (); +} +void pci_liveupdate_cleanup_device (struct pci_dev *dev) {/*
@@ -432,7 +602,80 @@ void pci_liveupdate_cleanup_device(struct pci_dev *dev)*/ if (READ_ONCE (dev->liveupdate.outgoing))pci_WARN (dev,1 ,"Destroying outgoing-preserved device!\n" ); + +if (READ_ONCE (dev->liveupdate.incoming)) +pci_WARN (dev,1 ,"Destroying incoming-preserved device!\n" ); +} + +static void pci_liveupdate_finish_device (struct pci_ser *ser,struct pci_dev *dev) +{ +if (!dev->liveupdate.incoming) { +pci_warn (dev,"Cannot finish preserving an unpreserved device\n" ); +return ; + } + +if (dev->liveupdate.incoming->refcount !=1 ) { +pci_WARN (dev,1 ,"Preserved device has a corrupted refcount!\n" ); +return ; + } + +/* +* Drop the refcount so this device does not get treated as an incoming +* device again, e.g. in case pci_liveupdate_setup_device() gets called +* again because the device is hot-plugged. +*/ + dev->liveupdate.incoming->refcount =0 ; + +pci_info (dev,"Device is finished participating in Live Update\n" ); + dev->liveupdate.incoming =NULL ; + ser->nr_devices--; +} + +/** +* pci_liveupdate_finish() - Finish the preservation of a PCI device +* @dev: The PCI device +* +* pci_liveupdate_finish() notifies the PCI core that a PCI device that was +* preserved across the previous Live Update has finished participating in Live +* Update. Drivers must call pci_liveupdate_finish() from their struct +* liveupdate_file_handler finish() callback, while the incoming struct pci_ser +* is still available. +*/ +void pci_liveupdate_finish (struct pci_dev *dev) +{ +struct pci_flb_incoming *incoming; + +guard (rwsem_write)(&pci_liveupdate.rwsem); + + incoming =pci_liveupdate_flb_get_incoming (); +if (!incoming) { +pci_warn (dev,"Cannot finish preserving device without incoming FLB\n" ); +return ; + } + +pci_liveupdate_finish_device (incoming->ser, dev); +pci_liveupdate_flb_put_incoming (); +} +EXPORT_SYMBOL_GPL (pci_liveupdate_finish); + +/** +* pci_liveupdate_is_incoming() - Check if a device is incoming-preserved +* @dev: The PCI device to check +* +* Check if a device was preserved across Live Update by the previous kernel, +* i.e. the device is incoming-preserved. Note that a device is only considered +* incoming-preserved prior to pci_liveupdate_finish(). It is up to drivers to +* synchronize usage of pci_liveupdate_is_incoming() with their own call to +* pci_liveupdate_finish() to avoid acting on stale data. +* +* Return: True if the device is incoming-preserved, false otherwise. +*/ +bool pci_liveupdate_is_incoming (struct pci_dev *dev) +{ +guard (rwsem_read)(&pci_liveupdate.rwsem); +return dev->liveupdate.incoming; } +EXPORT_SYMBOL_GPL (pci_liveupdate_is_incoming);/** * pci_liveupdate_register_flb() - Register a file handler with the PCI core
drivers/pci/liveupdate.h +5 -0
@@ -11,8 +11,13 @@#include <linux/pci.h> #ifdef CONFIG_PCI_LIVEUPDATE +void pci_liveupdate_setup_device (struct pci_dev *dev);void pci_liveupdate_cleanup_device (struct pci_dev *dev);#else +static inline void pci_liveupdate_setup_device (struct pci_dev *dev) +{ +} +static inline void pci_liveupdate_cleanup_device (struct pci_dev *dev) { }
drivers/pci/probe.c +3 -0
@@ -2075,6 +2075,8 @@ int pci_setup_device(struct pci_dev *dev)if (pci_early_dump)early_dump_pci_device (dev); +pci_liveupdate_setup_device (dev); +/* Need to have dev->class ready */ dev->cfg_size =pci_cfg_space_size (dev);
@@ -2195,6 +2197,7 @@ int pci_setup_device(struct pci_dev *dev)default :/* unknown header */ pci_err (dev,"unknown header type %02x, ignoring device\n" , dev->hdr_type); +pci_liveupdate_cleanup_device (dev);pci_release_of_node (dev);return -EIO ;
include/linux/pci_liveupdate.h +13 -0
@@ -16,9 +16,11 @@/** * struct pci_liveupdate - PCI Live Update state for a struct pci_dev * @outgoing: State preserved for the next kernel. +* @incoming: State preserved by the previous kernel. */ struct pci_liveupdate {struct pci_dev_ser *outgoing; +struct pci_dev_ser *incoming; };struct pci_dev ;
@@ -28,6 +30,8 @@ int pci_liveupdate_register_flb(struct liveupdate_file_handler *fh);void pci_liveupdate_unregister_flb (struct liveupdate_file_handler *fh);int pci_liveupdate_preserve (struct pci_dev *dev);void pci_liveupdate_unpreserve (struct pci_dev *dev); +void pci_liveupdate_finish (struct pci_dev *dev); +bool pci_liveupdate_is_incoming (struct pci_dev *dev);#else static inline int pci_liveupdate_register_flb (struct liveupdate_file_handler *fh) {
@@ -46,6 +50,15 @@ static inline int pci_liveupdate_preserve(struct pci_dev *dev)static inline void pci_liveupdate_unpreserve (struct pci_dev *dev) { } + +static inline void pci_liveupdate_finish (struct pci_dev *dev) +{ +} + +static inline bool pci_liveupdate_is_incoming (struct pci_dev *dev) +{ +return false; +}#endif #endif /* LINUX_PCI_LIVEUPDATE_H */
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 4/13 PCI: liveupdate: Document driver binding responsibilities
In reply to David MatlackDocument how driver binding works during a Live Update and what the PCI core expects of drivers and users.
--- drivers/pci/liveupdate.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
drivers/pci/liveupdate.c +14 -0
@@ -123,6 +123,20 @@* * * The device cannot be a Virtual Function (VF). * * The device cannot be behind a PCI-to-PCI bridge. +* +* Driver Binding +* ============== +* +* In the outgoing kernel, the driver must ensure that it does not unbind from a +* device between pci_liveupdate_preserve() and pci_liveupdate_unpreserve(). +* +* In the incoming kernel, the driver must ensure that it does not unbind from a +* preserved device between probe() and pci_liveupdate_finish(). +* +* It is the user's responsibility to ensure that incoming preserved devices are +* bound to the correct driver. The PCI core does not protect against a device +* getting preserved by driver A in the outgoing kernel and then getting bound +* to driver B in the incoming kernel. */ #define pr_fmt (fmt) "PCI: liveupdate: " fmt
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 5/13 PCI: liveupdate: Auto-preserve upstream bridges across Live Update
In reply to David MatlackAutomatically preserve every bridge between a preserved device and its root port when the device is preserved.
Subsequent commits use this to ensure that preserved devices can continue performing memory transactions without a disruption or change in routing.
Track the number of preserved devices at or below each bridge with a reference count in struct pci_dev_ser. A bridge stays preserved until all of its downstream preserved devices are unpreserved or finish their participation in the Live Update.
struct pci_ser::nr_devices counts the struct pci_dev_ser entries that are in use, so it only changes when a device's entry is allocated or released, i.e. when its reference count goes from 0 to 1 or from 1 to 0, not on every preserve and unpreserve.
--- drivers/pci/liveupdate.c | 125 +++++++++++++++++++++++++++++---------- include/linux/pci.h | 3 + 2 files changed, 97 insertions(+), 31 deletions(-)
drivers/pci/liveupdate.c +94 -31
@@ -122,7 +122,6 @@* preserved. These may be relaxed in the future: * * * The device cannot be a Virtual Function (VF). -* * The device cannot be behind a PCI-to-PCI bridge. * * Driver Binding * ==============
@@ -137,6 +136,18 @@* bound to the correct driver. The PCI core does not protect against a device * getting preserved by driver A in the outgoing kernel and then getting bound * to driver B in the incoming kernel. +* +* PCI-to-PCI Bridges +* ================== +* +* Any PCI-to-PCI bridges upstream of a preserved device are automatically +* preserved when the device is preserved. The PCI core keeps track of the +* number of downstream devices that are preserved under a bridge so that the +* bridge is only unpreserved once all downstream devices are unpreserved. +* +* This lets the PCI core keep the bridges above preserved devices forwarding +* their memory transactions, without changes in routing, throughout the Live +* Update. */ #define pr_fmt (fmt) "PCI: liveupdate: " fmt
@@ -437,7 +448,8 @@ static struct pci_dev_ser *pci_liveupdate_alloc_dev_ser( }/* -* Stop preserving @dev by releasing its struct pci_dev_ser. +* Stop preserving @dev by releasing its struct pci_dev_ser. The caller must +* have dropped the last reference on it. */ static void pci_liveupdate_free_dev_ser (struct pci_flb_outgoing *outgoing,struct pci_dev *dev)
@@ -450,43 +462,71 @@ static void pci_liveupdate_free_dev_ser(struct pci_flb_outgoing *outgoing, dev->liveupdate.outgoing =NULL ; } -static void pci_liveupdate_unpreserve_device (struct pci_flb_outgoing *outgoing, -struct pci_dev *dev) +static int pci_liveupdate_unpreserve_device (struct pci_flb_outgoing *outgoing, +struct pci_dev *dev) {struct pci_dev_ser *dev_ser = dev->liveupdate.outgoing;if (!dev_ser) {pci_warn (dev,"Cannot unpreserve device that is not preserved\n" ); -return ; +return -EINVAL ; } +if (!dev_ser->refcount) { +pci_WARN (dev,1 ,"Preserved device has a 0 refcount!\n" ); +return -EINVAL ; + } + +if (--dev_ser->refcount) +return 0 ; +pci_liveupdate_free_dev_ser (outgoing, dev); +return 0 ; +} + +static void pci_liveupdate_unpreserve_path (struct pci_flb_outgoing *outgoing, +struct pci_dev *dev, +struct pci_dev *end) +{ +for_each_pci_dev_in_path (dev) { +if (dev == end) +break ; + +if (pci_liveupdate_unpreserve_device (outgoing, dev)) +return ; + } }static int pci_liveupdate_preserve_device (struct pci_flb_outgoing *outgoing,struct pci_dev *dev) { -struct pci_dev_ser *dev_ser; +struct pci_dev_ser *dev_ser = dev->liveupdate.outgoing;if (dev->is_virtfn) {pci_warn (dev,"Cannot preserve Virtual Functions\n" );return -EINVAL ; } -if (dev->liveupdate.outgoing) { -pci_warn (dev,"Device is already preserved\n" ); -return -EBUSY ; +if (dev_ser) { +/* +* Endpoint devices can only be preserved once. Bridges are +* preserved once for every preserved device below them. +*/ +if (!dev->subordinate) { +pci_warn (dev,"Device is already preserved\n" ); +return -EBUSY ; + } + +if (!dev_ser->refcount) { +pci_WARN (dev,1 ,"Preserved device with 0 refcount!\n" ); +return -EINVAL ; + } + }else { + dev_ser =pci_liveupdate_alloc_dev_ser (outgoing, dev); +if (IS_ERR (dev_ser)) +return PTR_ERR (dev_ser); } -if (!pci_is_root_bus (dev->bus)) { -pci_warn (dev,"Cannot preserve devices behind bridges\n" ); -return -EINVAL ; - } - - dev_ser =pci_liveupdate_alloc_dev_ser (outgoing, dev); -if (IS_ERR (dev_ser)) -return PTR_ERR (dev_ser); - dev_ser->refcount++;return 0 ; }
@@ -500,12 +540,16 @@ static int pci_liveupdate_preserve_device(struct pci_flb_outgoing *outgoing,* pci_liveupdate_preserve() from their struct liveupdate_file_handler * preserve() callback to ensure the outgoing struct pci_ser is already set up. * +* pci_liveupdate_preserve() automatically preserves all bridges upstream of +* @dev. +* * Return: 0 on success, <0 on failure. */ int pci_liveupdate_preserve (struct pci_dev *dev) {struct pci_flb_outgoing *outgoing =NULL ; -int ret; +struct pci_dev *start = dev; +int ret = -ENODEV ;guard (rwsem_write)(&pci_liveupdate.rwsem);
@@ -513,7 +557,13 @@ int pci_liveupdate_preserve(struct pci_dev *dev)if (IS_ERR (outgoing))return PTR_ERR (outgoing); - ret =pci_liveupdate_preserve_device (outgoing, dev); +for_each_pci_dev_in_path (dev) { + ret =pci_liveupdate_preserve_device (outgoing, dev); +if (ret) { +pci_liveupdate_unpreserve_path (outgoing, start, dev); +break ; + } + }pci_liveupdate_flb_put_outgoing ();return ret;
@@ -528,6 +578,9 @@ EXPORT_SYMBOL_GPL(pci_liveupdate_preserve);* longer be preserved across the next Live Update. Drivers are expected to call * pci_liveupdate_unpreserve() from their struct liveupdate_file_handler * unpreserve() callback, while the outgoing struct pci_ser still exists. +* +* pci_liveupdate_unpreserve() drops @dev's reference on each bridge upstream of +* @dev, unpreserving bridges that have no other preserved devices below them. */ void pci_liveupdate_unpreserve (struct pci_dev *dev) {
@@ -541,7 +594,7 @@ void pci_liveupdate_unpreserve(struct pci_dev *dev)return ; } -pci_liveupdate_unpreserve_device (outgoing, dev); +pci_liveupdate_unpreserve_path (outgoing, dev,/*end=*/ NULL );pci_liveupdate_flb_put_outgoing (); }EXPORT_SYMBOL_GPL (pci_liveupdate_unpreserve);
@@ -621,28 +674,30 @@ void pci_liveupdate_cleanup_device(struct pci_dev *dev)pci_WARN (dev,1 ,"Destroying incoming-preserved device!\n" ); } -static void pci_liveupdate_finish_device (struct pci_ser *ser,struct pci_dev *dev) +static int pci_liveupdate_finish_device (struct pci_ser *ser,struct pci_dev *dev) {if (!dev->liveupdate.incoming) {pci_warn (dev,"Cannot finish preserving an unpreserved device\n" ); -return ; +return -EINVAL ; } -if (dev->liveupdate.incoming->refcount !=1 ) { -pci_WARN (dev,1 ,"Preserved device has a corrupted refcount!\n" ); -return ; +if (!dev->liveupdate.incoming->refcount) { +pci_WARN (dev,1 ,"Preserved device has a 0 refcount!\n" ); +return -EINVAL ; }/* -* Drop the refcount so this device does not get treated as an incoming -* device again, e.g. in case pci_liveupdate_setup_device() gets called -* again because the device is hot-plugged. +* Once the refcount reaches 0 the device is no longer treated as +* incoming, e.g. if pci_liveupdate_setup_device() gets called again +* because the device is hot-plugged. */ - dev->liveupdate.incoming->refcount =0 ; +if (--dev->liveupdate.incoming->refcount) +return 0 ;pci_info (dev,"Device is finished participating in Live Update\n" ); dev->liveupdate.incoming =NULL ; ser->nr_devices--; +return 0 ; }/**
@@ -654,6 +709,10 @@ static void pci_liveupdate_finish_device(struct pci_ser *ser, struct pci_dev *de* Update. Drivers must call pci_liveupdate_finish() from their struct * liveupdate_file_handler finish() callback, while the incoming struct pci_ser * is still available. +* +* pci_liveupdate_finish() drops @dev's reference on each bridge upstream of +* @dev, finishing bridges that have no other incoming preserved devices below +* them. */ void pci_liveupdate_finish (struct pci_dev *dev) {
@@ -667,7 +726,11 @@ void pci_liveupdate_finish(struct pci_dev *dev)return ; } -pci_liveupdate_finish_device (incoming->ser, dev); +for_each_pci_dev_in_path (dev) { +if (pci_liveupdate_finish_device (incoming->ser, dev)) +break ; + } +pci_liveupdate_flb_put_incoming (); }EXPORT_SYMBOL_GPL (pci_liveupdate_finish);
include/linux/pci.h +3 -0
@@ -840,6 +840,9 @@ static inline struct pci_dev *pci_upstream_bridge(struct pci_dev *dev)return dev->bus->self; } +#define for_each_pci_dev_in_path (dev) \ + for (; dev; dev = pci_upstream_bridge(dev)) +#ifdef CONFIG_PCI_MSI static inline bool pci_dev_msi_enabled (struct pci_dev *pci_dev) {
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 6/13 PCI: liveupdate: Preserve bus numbers during Live Update
In reply to David MatlackKeep the secondary and subordinate bus numbers that the previous kernel programmed into bridges, rather than assigning new ones, if the previous kernel preserved any device across a Live Update. Do this even when pcibios_assign_all_busses() is true, e.g. with pci=assign-busses.
Preserved devices must be allowed to continue performing memory transactions across a Live Update, so the kernel cannot change the fabric topology. Changing the bus numbers of a bridge changes the Requester IDs of the devices below it, which would require disabling and flushing any in-flight memory transactions first.
Apply the policy globally rather than only to the paths that contain preserved devices. Bus numbers have to be preserved above a preserved device anyway, since an upstream bridge cannot expand its window. A global policy matches the scope of pcibios_assign_all_busses(), and gives an answer that cannot change part way through the two passes of a bridge scan.
Bridges that do not have bus numbers are still assigned new ones, so hot-adding a bridge keeps working, both during and after a Live Update. The two-pass bridge scan guarantees such bridges are only assigned bus numbers above those already claimed by preserved bridges.
The exception is a bridge that was preserved but comes up without a valid bus number configuration, e.g. because it was reset during kexec. Refuse to assign it new bus numbers, since that would silently change the BDF of every preserved device in its hierarchy. Also refuse to assign bus numbers to the other bridges on the same bus, since the bus numbers of the failed bridge can no longer be read from hardware and handing them out would let an unrelated device inherit the BDF of a preserved device.
Make PCI_LIVEUPDATE depend on !CARDBUS, since preserving bus numbers on CardBus bridges is not implemented.
--- .../admin-guide/kernel-parameters.txt | 6 +- drivers/pci/Kconfig | 2 +- drivers/pci/liveupdate.c | 81 +++++++++++++++++++ drivers/pci/liveupdate.h | 13 +++ drivers/pci/probe.c | 11 ++- include/linux/pci_liveupdate.h | 5 ++ 6 files changed, 112 insertions(+), 6 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +5 -1
@@ -5129,7 +5129,11 @@ Kernel parameters explicitly which ones they are. assign-busses [X86] Always assign all PCI bus numbers ourselves, overriding - whatever the firmware may have done. + whatever the firmware may have done. Ignored + if the previous kernel preserved any PCI + device across a Live Update, since bus + numbers must not change while those devices + are doing DMA. usepirqmask [X86] Honor the possible IRQ mask stored in the BIOS $PIR table. This is needed on some systems with broken BIOSes, notably
drivers/pci/Kconfig +1 -1
@@ -273,7 +273,7 @@ config VGA_ARB_MAX_GPUS config PCI_LIVEUPDATE bool "PCI Live Update Support" - depends on PCI && LIVEUPDATE && 64BIT + depends on PCI && LIVEUPDATE && 64BIT && !CARDBUS help Enable PCI core support for preserving PCI devices across Live Update. This, in combination with support in a device's driver,
drivers/pci/liveupdate.c +81 -0
114 lines not on this page · raw message
drivers/pci/liveupdate.h +13 -0
@@ -13,6 +13,8 @@#ifdef CONFIG_PCI_LIVEUPDATE void pci_liveupdate_setup_device (struct pci_dev *dev);void pci_liveupdate_cleanup_device (struct pci_dev *dev); +bool pci_liveupdate_preserve_bus_numbers (void ); +bool pci_liveupdate_refuse_bus_numbers (struct pci_bus *bus,struct pci_dev *dev);#else static inline void pci_liveupdate_setup_device (struct pci_dev *dev) {
@@ -21,6 +23,17 @@ static inline void pci_liveupdate_setup_device(struct pci_dev *dev)static inline void pci_liveupdate_cleanup_device (struct pci_dev *dev) { } + +static inline bool pci_liveupdate_preserve_bus_numbers (void ) +{ +return false; +} + +static inline bool pci_liveupdate_refuse_bus_numbers (struct pci_bus *bus, +struct pci_dev *dev) +{ +return false; +}#endif #endif /* DRIVERS_PCI_LIVEUPDATE_H */
drivers/pci/probe.c +7 -4
@@ -1400,6 +1400,8 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev, int max, unsigned int available_buses, int pass) { + bool preserve_bus_numbers = !pcibios_assign_all_busses() || +pci_liveupdate_preserve_bus_numbers ();struct pci_bus *child;u32 buses;u16 bctl;
@@ -1452,8 +1454,7 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev, goto out; } -if ((secondary || subordinate) && - !pcibios_assign_all_busses () && !broken) { +if ((secondary || subordinate) && preserve_bus_numbers && !broken) {unsigned int cmax, buses;/*
@@ -1495,8 +1496,7 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev,* do in the second pass. */ if (!pass) { -if (pcibios_assign_all_busses () || broken) - +if (!preserve_bus_numbers || broken)/* * Temporarily disable forwarding of the * configuration cycles on all bridges in
@@ -1510,6 +1510,9 @@ static int pci_scan_bridge_extend(struct pci_bus *bus, struct pci_dev *dev, goto out; } +if (pci_liveupdate_refuse_bus_numbers (bus, dev)) + goto out; +/* Clear errors */ pci_write_config_word (dev,PCI_STATUS ,0xffff );
include/linux/pci_liveupdate.h +5 -0
@@ -17,10 +17,15 @@* struct pci_liveupdate - PCI Live Update state for a struct pci_dev * @outgoing: State preserved for the next kernel. * @incoming: State preserved by the previous kernel. +* @was_incoming: True if this struct pci_dev was incoming-preserved when it was +* set up, i.e. it was matched to state preserved by the previous +* kernel. Unlike @incoming, this is never cleared, so it stays +* true after the device finishes participating in Live Update. */ struct pci_liveupdate {struct pci_dev_ser *outgoing;struct pci_dev_ser *incoming; +bool was_incoming; };struct pci_dev ;
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 7/13 PCI: Refactor matching logic for pci_dev_acs_ops
In reply to David MatlackRefactor the logic to match devices to pci_dev_acs_ops by factoring out the device matching loop into its own routine, pci_dev_acs_ops_get(). Move the per-quirk device checks into a new .match callback so that pci_dev_acs_ops_get() can return the single matching entry.
This eliminates duplicate code between pci_dev_specific_enable_acs() and pci_dev_specific_disable_acs_redir(), and will also be used in a subsequent commit to check if a device requires device-specific enable_acs() during a Live Update.
No functional change intended.
--- drivers/pci/quirks.c | 51 ++++++++++++++++++-------------------------- 1 file changed, 21 insertions(+), 30 deletions(-)
drivers/pci/quirks.c +21 -30
@@ -5530,9 +5530,6 @@ static void pci_quirk_enable_intel_rp_mpc_acs(struct pci_dev *dev)*/ static int pci_quirk_enable_intel_pch_acs (struct pci_dev *dev) { -if (!pci_quirk_intel_pch_acs_match (dev)) -return -ENOTTY ; -if (pci_quirk_enable_intel_lpc_acs (dev)) {pci_warn (dev,"Failed to enable Intel PCH ACS quirk\n" );return 0 ;
@@ -5552,9 +5549,6 @@ static int pci_quirk_enable_intel_spt_pch_acs(struct pci_dev *dev)int pos;u32 cap, ctrl; -if (!pci_quirk_intel_spt_pch_acs_match (dev)) -return -ENOTTY ; - pos = dev->acs_cap;if (!pos)return -ENOTTY ;
@@ -5582,9 +5576,6 @@ static int pci_quirk_disable_intel_spt_pch_acs_redir(struct pci_dev *dev)int pos;u32 cap, ctrl; -if (!pci_quirk_intel_spt_pch_acs_match (dev)) -return -ENOTTY ; - pos = dev->acs_cap;if (!pos)return -ENOTTY ;
@@ -5604,56 +5595,56 @@ static int pci_quirk_disable_intel_spt_pch_acs_redir(struct pci_dev *dev)static const struct pci_dev_acs_ops {u16 vendor;u16 device; +bool (*match)(struct pci_dev *dev);int (*enable_acs)(struct pci_dev *dev);int (*disable_acs_redir)(struct pci_dev *dev); } pci_dev_acs_ops[] = { {PCI_VENDOR_ID_INTEL ,PCI_ANY_ID , + .match = pci_quirk_intel_pch_acs_match, .enable_acs = pci_quirk_enable_intel_pch_acs, }, {PCI_VENDOR_ID_INTEL ,PCI_ANY_ID , + .match = pci_quirk_intel_spt_pch_acs_match, .enable_acs = pci_quirk_enable_intel_spt_pch_acs, .disable_acs_redir = pci_quirk_disable_intel_spt_pch_acs_redir, }, }; -int pci_dev_specific_enable_acs (struct pci_dev *dev) +static const struct pci_dev_acs_ops *pci_dev_acs_ops_get (struct pci_dev *dev) {const struct pci_dev_acs_ops *p; -int i, ret; +int i;for (i =0 ; i <ARRAY_SIZE (pci_dev_acs_ops); i++) { p = &pci_dev_acs_ops[i];if ((p->vendor == dev->vendor || p->vendor == (u16 )PCI_ANY_ID ) && (p->device == dev->device || - p->device == (u16 )PCI_ANY_ID ) && - p->enable_acs) { - ret = p->enable_acs (dev); -if (ret >=0 ) -return ret; + p->device == (u16 )PCI_ANY_ID )) { +if (!p->match || p->match (dev)) +return p; } } +return NULL ; +} + +int pci_dev_specific_enable_acs (struct pci_dev *dev) +{ +const struct pci_dev_acs_ops *p =pci_dev_acs_ops_get (dev); + +if (p && p->enable_acs) +return p->enable_acs (dev); +return -ENOTTY ; }int pci_dev_specific_disable_acs_redir (struct pci_dev *dev) { -const struct pci_dev_acs_ops *p; -int i, ret; +const struct pci_dev_acs_ops *p =pci_dev_acs_ops_get (dev); -for (i =0 ; i <ARRAY_SIZE (pci_dev_acs_ops); i++) { - p = &pci_dev_acs_ops[i]; -if ((p->vendor == dev->vendor || - p->vendor == (u16 )PCI_ANY_ID ) && - (p->device == dev->device || - p->device == (u16 )PCI_ANY_ID ) && - p->disable_acs_redir) { - ret = p->disable_acs_redir (dev); -if (ret >=0 ) -return ret; - } - } +if (p && p->disable_acs_redir) +return p->disable_acs_redir (dev);return -ENOTTY ; }
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 8/13 PCI: Save and restore the ACS Control register and Egress Control Vector
In reply to David MatlackSave the ACS Control register and the ACS Egress Control Vector in pci_save_state() and write them back in pci_restore_state(), instead of recomputing the ACS controls from scratch with pci_enable_acs().
This makes ACS symmetric with the rest of a device's saved state. Today pci_save_state() ignores ACS entirely and pci_restore_state() re-enables the ACS controls from the kernel's current ACS policy. As a result, a device can come out of a reset with different ACS controls than it went in with, e.g. any controls programmed outside of pci_enable_acs() are silently dropped.
This also lets a later patch adopt the ACS controls established by a previous kernel across a Live Update rather than assigning new ones through pci_enable_acs().
Save and restore the Egress Control Vector as well to keep it in sync with the Egress Control Enable bit in the ACS Control register, which is now restored from saved state. The kernel never enables Egress Controls or programs the vector, but the firmware could have and they need to be kept in sync to avoid changing how P2P traffic is routed.
The Egress Control Vector Size is a HwInit field of the ACS Capability register (PCIe r7.1, sec 7.7.12.2), so read it directly from the device to size the save buffer in pci_allocate_cap_save_buffers(), similar to how the VC capability relies on its HwInit fields.
pci_enable_acs() runs when a driver binds to a device (pci_dma_configure()), i.e. after pci_bus_add_device() has already saved the device's state. Refresh the saved ACS state there as well, otherwise a subsequent reset would revert ACS back to the configuration left behind by firmware.
Devices that rely on device-specific quirks to enable an ACS equivalent keep that configuration outside of the ACS Control register, so don't allocate an ACS save buffer for them. Keep configuring ACS from scratch on restore for any device that has no saved ACS state.
--- drivers/pci/pci.c | 152 ++++++++++++++++++++++++++++++++++++++++++- drivers/pci/pci.h | 5 ++ drivers/pci/quirks.c | 7 ++ 3 files changed, 163 insertions(+), 1 deletion(-)
drivers/pci/pci.c +151 -1
@@ -1021,6 +1021,145 @@ static void pci_std_enable_acs(struct pci_dev *dev, struct pci_acs *caps) caps->ctrl |= (dev->acs_capabilities &PCI_ACS_TB ); } +/* +* Layout of the ACS save buffer. @ecv holds the ACS Egress Control Vector, if +* implemented, and is sized by pci_allocate_acs_save_buffer(). +*/ +struct pci_acs_saved_state { +u16 ctrl; +u32 ecv[]; +}; + +/* +* Return the number of DWORDs in the ACS Egress Control Vector, or 0 if the +* device does not implement P2P Egress Control. The Egress Control Vector +* Size is HwInit, so it is read directly from the ACS Capability register. +*/ +static unsigned int pci_acs_ecv_dwords (struct pci_dev *dev,int pos) +{ +unsigned int bits; +u16 cap; +u8 size; + +pci_read_config_word (dev, pos +PCI_ACS_CAP , &cap); +if (!(cap &PCI_ACS_EC )) +return 0 ; + +/* An Egress Control Vector Size of 0 means 256 bits */ +pci_read_config_byte (dev, pos +PCI_ACS_EGRESS_BITS , &size); + bits = size ?:256 ; + +return DIV_ROUND_UP (bits,32 ); +} + +/* +* Return the number of Egress Control Vector DWORDs held in @save_state, as +* sized by pci_allocate_acs_save_buffer(). +*/ +static unsigned int +pci_acs_saved_ecv_dwords (struct pci_cap_saved_state *save_state) +{ +return (save_state->cap.size - + offsetof(struct pci_acs_saved_state , ecv)) /sizeof (u32); +} + +static void pci_allocate_acs_save_buffer (struct pci_dev *dev) +{ +size_t size; +int pos; + +/* +* Devices that rely on device-specific quirks to enable an ACS +* equivalent keep that configuration outside of the ACS Control +* register, so they are configured from scratch on restore. +*/ +if (pci_need_dev_specific_enable_acs (dev)) +return ; + +/* dev->acs_cap is not set yet, pci_acs_init() runs later */ + pos =pci_find_ext_capability (dev,PCI_EXT_CAP_ID_ACS ); +if (!pos) +return ; + + size = struct_size_t(struct pci_acs_saved_state, ecv, +pci_acs_ecv_dwords (dev, pos)); +if (pci_add_ext_cap_save_buffer (dev,PCI_EXT_CAP_ID_ACS , size)) +pci_err (dev,"unable to allocate suspend buffer for ACS\n" ); +} + +/** +* pci_save_acs_state - save the ACS Control register and Egress Control Vector +* @dev: the PCI device +* +* Record the ACS configuration currently programmed in hardware so that +* pci_restore_acs_state() can reapply it after a reset. +*/ +static void pci_save_acs_state (struct pci_dev *dev) +{ +struct pci_cap_saved_state *save_state; +struct pci_acs_saved_state *acs; +unsigned int i, dwords; +int ecv_pos; + +if (!dev->acs_cap) +return ; + + save_state =pci_find_saved_ext_cap (dev,PCI_EXT_CAP_ID_ACS ); +if (!save_state) +return ; + + acs = (struct pci_acs_saved_state *)save_state->cap.data; + +pci_read_config_word (dev, dev->acs_cap +PCI_ACS_CTRL , &acs->ctrl); + + ecv_pos = dev->acs_cap +PCI_ACS_EGRESS_CTL_V ; + dwords =pci_acs_saved_ecv_dwords (save_state); +for (i =0 ; i < dwords; i++) +pci_read_config_dword (dev, ecv_pos + i *sizeof (u32), + &acs->ecv[i]); +} + +/** +* pci_restore_acs_state - restore the ACS Control register and Egress Control +* Vector +* @dev: the PCI device +*/ +static void pci_restore_acs_state (struct pci_dev *dev) +{ +struct pci_cap_saved_state *save_state =NULL ; +struct pci_acs_saved_state *acs; +unsigned int i, dwords; +int ecv_pos; + +if (dev->acs_cap) + save_state =pci_find_saved_ext_cap (dev,PCI_EXT_CAP_ID_ACS ); + +/* +* Configure ACS from scratch for devices that have no saved ACS state, +* e.g. devices that rely on device-specific quirks to enable an ACS +* equivalent (see pci_allocate_acs_save_buffer()). +*/ +if (!save_state) { +pci_enable_acs (dev); +return ; + } + + acs = (struct pci_acs_saved_state *)save_state->cap.data; + +/* +* Restore the Egress Control Vector before the ACS Control register. +* The vector resets to zero, so enabling P2P Egress Control first +* would briefly apply the reset vector instead of the saved one. +*/ + ecv_pos = dev->acs_cap +PCI_ACS_EGRESS_CTL_V ; + dwords =pci_acs_saved_ecv_dwords (save_state); +for (i =0 ; i < dwords; i++) +pci_write_config_dword (dev, ecv_pos + i *sizeof (u32), + acs->ecv[i]); + +pci_write_config_word (dev, dev->acs_cap +PCI_ACS_CTRL , acs->ctrl); +} +/** * pci_enable_acs - enable ACS if hardware support it * @dev: the PCI device
@@ -1057,6 +1196,15 @@ void pci_enable_acs(struct pci_dev *dev)__pci_config_acs (dev, &caps, config_acs_param,0 ,0 );pci_write_config_word (dev, pos +PCI_ACS_CTRL , caps.ctrl); + +/* +* pci_enable_acs() runs when a driver binds to the device, i.e. after +* pci_bus_add_device() has already saved the device's state. Refresh +* the saved ACS state so that a subsequent reset restores the +* configuration programmed here rather than the one left behind by +* firmware. +*/ +pci_save_acs_state (dev); }/**
@@ -1825,6 +1973,7 @@ int pci_save_state(struct pci_dev *dev)pci_save_aer_state (dev);pci_save_ptm_state (dev);pci_save_tph_state (dev); +pci_save_acs_state (dev);return pci_save_vc_state (dev); }EXPORT_SYMBOL (pci_save_state);
@@ -1902,7 +2051,7 @@ void pci_restore_state(struct pci_dev *dev)pci_restore_msi_state (dev);/* Restore ACS and IOV configuration state */ -pci_enable_acs (dev); +pci_restore_acs_state (dev);pci_restore_iov_state (dev); dev->state_saved = false;
@@ -3565,6 +3714,7 @@ void pci_allocate_cap_save_buffers(struct pci_dev *dev)if (error)pci_err (dev,"unable to allocate suspend buffer for LTR\n" ); +pci_allocate_acs_save_buffer (dev);pci_allocate_vc_save_buffers (dev); }
drivers/pci/pci.h +5 -0
@@ -1101,6 +1101,7 @@ void pci_acs_init(struct pci_dev *dev);void pci_enable_acs (struct pci_dev *dev);#ifdef CONFIG_PCI_QUIRKS int pci_dev_specific_acs_enabled (struct pci_dev *dev,u16 acs_flags); +bool pci_need_dev_specific_enable_acs (struct pci_dev *dev);int pci_dev_specific_enable_acs (struct pci_dev *dev);int pci_dev_specific_disable_acs_redir (struct pci_dev *dev);void pci_disable_broken_acs_cap (struct pci_dev *pdev);
@@ -1111,6 +1112,10 @@ static inline int pci_dev_specific_acs_enabled(struct pci_dev *dev, {return -ENOTTY ; } +static inline bool pci_need_dev_specific_enable_acs (struct pci_dev *dev) +{ +return false; +}static inline int pci_dev_specific_enable_acs (struct pci_dev *dev) {return -ENOTTY ;
drivers/pci/quirks.c +7 -0
@@ -5629,6 +5629,13 @@ static const struct pci_dev_acs_ops *pci_dev_acs_ops_get(struct pci_dev *dev)return NULL ; } +bool pci_need_dev_specific_enable_acs (struct pci_dev *dev) +{ +const struct pci_dev_acs_ops *p =pci_dev_acs_ops_get (dev); + +return p && p->enable_acs; +} +int pci_dev_specific_enable_acs (struct pci_dev *dev) {const struct pci_dev_acs_ops *p =pci_dev_acs_ops_get (dev);
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 9/13 PCI: liveupdate: Adopt ACS controls in incoming preserved devices
In reply to David MatlackAdopt Access Control Services (ACS) controls on all incoming preserved devices (endpoints and upstream bridges) during a Live Update.
Adopting ACS controls avoids changing routing rules while memory transactions are in flight from preserved devices. It also keeps IOMMU groups from changing during or after Live Update.
The adopted controls are recorded by the pci_save_state() call in pci_bus_add_device(), which runs before drivers bind, so they are reapplied by pci_restore_state() if the device is reset. Allocate the ACS save buffer for incoming preserved devices even if they match a device-specific ACS quirk, since their adopted controls live in the ACS capability, and panic if it could not be allocated.
To simplify ACS adoption, reject preserving any devices that require quirks to enable ACS as those quirks would also have to take Live Update into account. If such a quirk matches an incoming preserved device anyway, e.g. because it was added in the new kernel, log a warning and adopt the device's ACS controls regardless, rather than reprogram a device in the path of a preserved device's memory transactions.
--- drivers/pci/liveupdate.c | 75 ++++++++++++++++++++++++++++++++++++++++ drivers/pci/liveupdate.h | 16 +++++++++ drivers/pci/pci.c | 13 +++++-- 3 files changed, 102 insertions(+), 2 deletions(-)
drivers/pci/liveupdate.c +75 -0
@@ -123,6 +123,9 @@* * * The device cannot be a Virtual Function (VF). * +* * The device cannot require device-specific quirks to enable Access +* Control Services (ACS). +* * Driver Binding * ============== *
@@ -174,6 +177,17 @@* bus, since the bus numbers of the failed bridge can no longer be read from * hardware and handing them to another bridge would let an unrelated device * inherit the BDF of a preserved device. +* +* Handling Preserved Devices +* ========================== +* +* The PCI core handles preserved devices differently in the following ways: +* +* * The PCI core adopts all ACS controls enabled on incoming preserved devices +* rather than assigning new ones, so TLP routing and IOMMU groups stay the +* same after Live Update. Note that a device will use its adopted ACS +* controls for the lifetime of its struct pci_dev (i.e. even after +* pci_liveupdate_finish()). */ #define pr_fmt (fmt) "PCI: liveupdate: " fmt
@@ -189,6 +203,7 @@#include <linux/slab.h> #include "liveupdate.h" +#include "pci.h" /** * struct pci_liveupdate_global - Global state for PCI Live Update support
@@ -540,6 +555,15 @@ static int pci_liveupdate_preserve_device(struct pci_flb_outgoing *outgoing,return -EINVAL ; } +/* +* Do not preserve devices that rely on device-specific ACS equivalents +* since that would complicate keeping ACS constant across Live Update. +*/ +if (pci_need_dev_specific_enable_acs (dev)) { +pci_warn (dev,"Refusing to preserve device that relies on ACS quirks\n" ); +return -EINVAL ; + } +if (dev_ser) {/* * Endpoint devices can only be preserved once. Bridges are
@@ -816,6 +840,57 @@ void pci_liveupdate_finish(struct pci_dev *dev) }EXPORT_SYMBOL_GPL (pci_liveupdate_finish); +/** +* pci_liveupdate_adopt_acs() - Adopt ACS controls +* @dev: The PCI device to adopt ACS controls for +* +* For devices preserved across a Live Update, leave the ACS controls +* established by the previous kernel alone instead of programming new ones. +* The adopted controls are recorded by the pci_save_state() call in +* pci_bus_add_device(), so they are reapplied by pci_restore_state() if the +* device is subsequently reset. +* +* Panics if the device has an ACS capability but no ACS save buffer, since the +* adopted controls would be lost on the first reset. +* +* Return: 0 if the ACS controls were adopted, or -EINVAL if the device was not +* preserved. +*/ +int pci_liveupdate_adopt_acs (struct pci_dev *dev) +{ +/* +* Check if the device was preserved over a previous Live Update (even +* if it has already gone through pci_liveupdate_finish()). This ensures +* that the device continues to use the ACS controls established by the +* previous kernel. +*/ +if (!dev->liveupdate.was_incoming) +return -EINVAL ; + +/* +* The previous kernel does not preserve devices that require +* device-specific quirks to enable ACS, so the controls being adopted +* live entirely in the ACS capability. If such a quirk matches anyway +* (e.g. it was added in the current kernel), ignore it rather than +* reprogram a device in the path of a preserved device's memory +* transactions. +*/ +if (pci_need_dev_specific_enable_acs (dev)) +pci_warn (dev,"Ignoring device-specific ACS quirk, adopting ACS controls from previous kernel\n" ); + +/* +* The adopted controls must be captured in the ACS save buffer so that +* they are reapplied if the device is later reset. Without it, e.g. +* because it could not be allocated, the first reset would silently +* drop them. +*/ +if (dev->acs_cap && !pci_find_saved_ext_cap (dev,PCI_EXT_CAP_ID_ACS )) +panic ("%s: No ACS save buffer for preserved device\n" , +pci_name (dev)); + +return 0 ; +} +/** * pci_liveupdate_is_incoming() - Check if a device is incoming-preserved * @dev: The PCI device to check
drivers/pci/liveupdate.h +16 -0
@@ -15,6 +15,12 @@ void pci_liveupdate_setup_device(struct pci_dev *dev);void pci_liveupdate_cleanup_device (struct pci_dev *dev);bool pci_liveupdate_preserve_bus_numbers (void );bool pci_liveupdate_refuse_bus_numbers (struct pci_bus *bus,struct pci_dev *dev); +int pci_liveupdate_adopt_acs (struct pci_dev *dev); + +static inline bool pci_liveupdate_was_incoming (struct pci_dev *dev) +{ +return dev->liveupdate.was_incoming; +}#else static inline void pci_liveupdate_setup_device (struct pci_dev *dev) {
@@ -34,6 +40,16 @@ static inline bool pci_liveupdate_refuse_bus_numbers(struct pci_bus *bus, {return false; } + +static inline int pci_liveupdate_adopt_acs (struct pci_dev *dev) +{ +return -EINVAL ; +} + +static inline bool pci_liveupdate_was_incoming (struct pci_dev *dev) +{ +return false; +}#endif #endif /* DRIVERS_PCI_LIVEUPDATE_H */
drivers/pci/pci.c +11 -2
@@ -35,6 +35,8 @@#include <linux/aer.h> #include <linux/bitfield.h> #include <linux/suspend.h> + +#include "liveupdate.h" #include "pci.h" DEFINE_MUTEX (pci_slot_mutex);
@@ -1071,9 +1073,13 @@ static void pci_allocate_acs_save_buffer(struct pci_dev *dev)/* * Devices that rely on device-specific quirks to enable an ACS * equivalent keep that configuration outside of the ACS Control -* register, so they are configured from scratch on restore. +* register, so they are configured from scratch on restore. The +* exception is devices preserved across a Live Update, which always +* adopt the ACS controls established by the previous kernel and need +* the buffer to keep them across resets. */ -if (pci_need_dev_specific_enable_acs (dev)) +if (pci_need_dev_specific_enable_acs (dev) && + !pci_liveupdate_was_incoming (dev))return ;/* dev->acs_cap is not set yet, pci_acs_init() runs later */
@@ -1170,6 +1176,9 @@ void pci_enable_acs(struct pci_dev *dev)bool enable_acs = false;int pos; +if (!pci_liveupdate_adopt_acs (dev)) +return ; +/* If an iommu is present we start with kernel default caps */ if (pci_acs_enable) {if (pci_dev_specific_enable_acs (dev))
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 10/13 PCI: liveupdate: Adopt ARI Forwarding Enable on preserved bridges
In reply to David MatlackAdopt the ARI Forwarding Enable bit on preserved bridges and update pci_dev->ari_enabled accordingly during a Live Update. This ensures that the preserved devices on the bridge's secondary bus can be identified with the same expanded 8-bit function number after a Live Update.
Adopt the bit even if pci=noari is passed, and document this in kernel-parameters.txt. The previous kernel may have enabled ARI Forwarding, and the preserved devices below the bridge rely on it.
pci_liveupdate_adopt_ari() updates the bridge's ari_enabled the same way, and from the same context, as pci_configure_ari() already does when it enables or disables ARI Forwarding on the bridge.
--- .../admin-guide/kernel-parameters.txt | 5 ++- drivers/pci/liveupdate.c | 36 +++++++++++++++++++ drivers/pci/liveupdate.h | 6 ++++ drivers/pci/pci.c | 8 ++++- 4 files changed, 53 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +4 -1
@@ -5239,7 +5239,10 @@ Kernel parameters off: Turn realloc off on: Turn realloc on realloc same as realloc=on - noari do not use PCIe ARI. + noari do not use PCIe ARI. Bridges above devices + preserved across a Live Update keep the ARI + Forwarding Enable setting from the previous + kernel. noats [PCIE, Intel-IOMMU, AMD-IOMMU] do not use PCIe ATS (and IOMMU device IOTLB). pcie_scan_all Scan all possible PCIe devices. Otherwise we
drivers/pci/liveupdate.c +36 -0
@@ -188,6 +188,11 @@* same after Live Update. Note that a device will use its adopted ACS * controls for the lifetime of its struct pci_dev (i.e. even after * pci_liveupdate_finish()). +* +* * The PCI core adopts ARI Forwarding Enable on bridges above incoming +* preserved devices, even with pci=noari, until all preserved devices below +* the bridge have finished. This keeps the preserved devices on the bridge's +* secondary bus addressable with the same function numbers. */ #define pr_fmt (fmt) "PCI: liveupdate: " fmt
@@ -891,6 +896,37 @@ int pci_liveupdate_adopt_acs(struct pci_dev *dev)return 0 ; } +/** +* pci_liveupdate_adopt_ari() - Adopt ARI configuration +* @dev: The PCI device to adopt ARI configuration for +* +* For devices preserved across a Live Update, read the ARI state from +* hardware and adopt it. This ensures the device continues to use the ARI +* configuration established by the previous kernel. +* +* Return: 0 on success, or -EINVAL if the device was not preserved. +*/ +int pci_liveupdate_adopt_ari (struct pci_dev *dev) +{ +u16 val; + +guard (rwsem_read)(&pci_liveupdate.rwsem); + +/* +* Unlike ACS, only adopt ARI while the bridge is incoming-preserved. +* Once all devices below the bridge have finished, recomputing ARI +* Forwarding Enable yields the same result for the same device, and +* must be allowed to change if a different device is hot-plugged. +*/ +if (!dev->liveupdate.incoming) +return -EINVAL ; + +pcie_capability_read_word (dev,PCI_EXP_DEVCTL2 , &val); + + dev->ari_enabled = !!(val &PCI_EXP_DEVCTL2_ARI ); +return 0 ; +} +/** * pci_liveupdate_is_incoming() - Check if a device is incoming-preserved * @dev: The PCI device to check
drivers/pci/liveupdate.h +6 -0
@@ -16,6 +16,7 @@ void pci_liveupdate_cleanup_device(struct pci_dev *dev);bool pci_liveupdate_preserve_bus_numbers (void );bool pci_liveupdate_refuse_bus_numbers (struct pci_bus *bus,struct pci_dev *dev);int pci_liveupdate_adopt_acs (struct pci_dev *dev); +int pci_liveupdate_adopt_ari (struct pci_dev *dev);static inline bool pci_liveupdate_was_incoming (struct pci_dev *dev) {
@@ -46,6 +47,11 @@ static inline int pci_liveupdate_adopt_acs(struct pci_dev *dev)return -EINVAL ; } +static inline int pci_liveupdate_adopt_ari (struct pci_dev *dev) +{ +return -EINVAL ; +} +static inline bool pci_liveupdate_was_incoming (struct pci_dev *dev) {return false;
drivers/pci/pci.c +7 -1
@@ -3748,7 +3748,7 @@ void pci_configure_ari(struct pci_dev *dev)u32 cap;struct pci_dev *bridge; -if (pcie_ari_disabled || !pci_is_pcie (dev) || dev->devfn) +if (!pci_is_pcie (dev) || dev->devfn)return ; bridge = dev->bus->self;
@@ -3759,6 +3759,12 @@ void pci_configure_ari(struct pci_dev *dev)if (!(cap &PCI_EXP_DEVCAP2_ARI ))return ; +if (!pci_liveupdate_adopt_ari (bridge)) +return ; + +if (pcie_ari_disabled) +return ; +if (pci_find_ext_capability (dev,PCI_EXT_CAP_ID_ARI )) {pcie_capability_set_word (bridge,PCI_EXP_DEVCTL2 ,PCI_EXP_DEVCTL2_ARI );
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 11/13 PCI: liveupdate: Freeze preservation status during shutdown
In reply to David MatlackFreeze a device's outgoing preservation status (preserved or not preserved) during shutdown. This enables the PCI core and drivers to safely make decisions based on the device's preservation status during shutdown.
The PCI core freezes every device in pci_device_shutdown(), not just preserved ones, so that no device can be preserved or unpreserved after its shutdown starts.
--- drivers/pci/liveupdate.c | 19 +++++++++++++++++++ drivers/pci/liveupdate.h | 5 +++++ drivers/pci/pci-driver.c | 2 ++ include/linux/pci_liveupdate.h | 3 +++ 4 files changed, 29 insertions(+)
drivers/pci/liveupdate.c +19 -0
@@ -126,6 +126,9 @@* * The device cannot require device-specific quirks to enable Access * Control Services (ACS). * +* In addition, a device cannot be preserved or unpreserved once the PCI core +* has started shutting it down (pci_device_shutdown()). +* * Driver Binding * ============== *
@@ -520,6 +523,11 @@ static int pci_liveupdate_unpreserve_device(struct pci_flb_outgoing *outgoing, {struct pci_dev_ser *dev_ser = dev->liveupdate.outgoing; +if (dev->liveupdate.frozen) { +pci_warn (dev,"Cannot unpreserve device after it is frozen!\n" ); +return -EINVAL ; + } +if (!dev_ser) {pci_warn (dev,"Cannot unpreserve device that is not preserved\n" );return -EINVAL ;
@@ -569,6 +577,11 @@ static int pci_liveupdate_preserve_device(struct pci_flb_outgoing *outgoing,return -EINVAL ; } +if (dev->liveupdate.frozen) { +pci_warn (dev,"Cannot preserve device after it is frozen!\n" ); +return -EINVAL ; + } +if (dev_ser) {/* * Endpoint devices can only be preserved once. Bridges are
@@ -784,6 +797,12 @@ void pci_liveupdate_cleanup_device(struct pci_dev *dev)pci_WARN (dev,1 ,"Destroying incoming-preserved device!\n" ); } +void pci_liveupdate_freeze (struct pci_dev *dev) +{ +guard (rwsem_write)(&pci_liveupdate.rwsem); + dev->liveupdate.frozen = true; +} +static int pci_liveupdate_finish_device (struct pci_ser *ser,struct pci_dev *dev) {if (!dev->liveupdate.incoming) {
drivers/pci/liveupdate.h +5 -0
@@ -13,6 +13,7 @@#ifdef CONFIG_PCI_LIVEUPDATE void pci_liveupdate_setup_device (struct pci_dev *dev);void pci_liveupdate_cleanup_device (struct pci_dev *dev); +void pci_liveupdate_freeze (struct pci_dev *dev);bool pci_liveupdate_preserve_bus_numbers (void );bool pci_liveupdate_refuse_bus_numbers (struct pci_bus *bus,struct pci_dev *dev);int pci_liveupdate_adopt_acs (struct pci_dev *dev);
@@ -31,6 +32,10 @@ static inline void pci_liveupdate_cleanup_device(struct pci_dev *dev) { } +static inline void pci_liveupdate_freeze (struct pci_dev *dev) +{ +} +static inline bool pci_liveupdate_preserve_bus_numbers (void ) {return false;
drivers/pci/pci-driver.c +2 -0
@@ -21,6 +21,7 @@#include <linux/acpi.h> #include <linux/dma-map-ops.h> #include <linux/iommu.h> +#include "liveupdate.h" #include "pci.h" #include "pcie/portdrv.h"
@@ -559,6 +560,7 @@ static void pci_device_shutdown(struct device *dev)struct pci_dev *pci_dev =to_pci_dev (dev);struct pci_driver *drv = pci_dev->driver; +pci_liveupdate_freeze (pci_dev);pm_runtime_resume (dev);if (drv && drv->shutdown)
include/linux/pci_liveupdate.h +3 -0
@@ -21,11 +21,14 @@* set up, i.e. it was matched to state preserved by the previous * kernel. Unlike @incoming, this is never cleared, so it stays * true after the device finishes participating in Live Update. +* @frozen: True if the outgoing preservation status of this device is frozen +* and thus cannot be changed. */ struct pci_liveupdate {struct pci_dev_ser *outgoing;struct pci_dev_ser *incoming;bool was_incoming; +bool frozen; };struct pci_dev ;
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 12/13 PCI: liveupdate: Do not disable bus mastering on preserved devices during kexec
In reply to David MatlackDo not disable bus mastering on outgoing preserved devices during pci_device_shutdown() for kexec.
Preserved devices must keep performing memory transactions during a Live Update. Clearing the Bus Master Enable bit would prevent these devices from issuing any memory requests while the new kernel boots.
Because bridges upstream of preserved endpoint devices are also automatically preserved, this change also avoids clearing bus mastering on them. Clearing Bus Master Enable on an upstream bridge prevents the bridge from forwarding memory requests upstream (i.e. it would prevent the endpoint device from accessing system RAM and doing peer-to-peer transactions with devices not downstream of the bridge).
--- drivers/pci/liveupdate.c | 11 +++++++++++ drivers/pci/liveupdate.h | 6 ++++++ drivers/pci/pci-driver.c | 6 ++++-- 3 files changed, 21 insertions(+), 2 deletions(-)
drivers/pci/liveupdate.c +11 -0
@@ -196,6 +196,10 @@* preserved devices, even with pci=noari, until all preserved devices below * the bridge have finished. This keeps the preserved devices on the bridge's * secondary bus addressable with the same function numbers. +* +* * The PCI core does not disable bus mastering on outgoing preserved devices +* during kexec. This allows preserved devices to issue memory transactions +* throughout the Live Update. */ #define pr_fmt (fmt) "PCI: liveupdate: " fmt
@@ -946,6 +950,13 @@ int pci_liveupdate_adopt_ari(struct pci_dev *dev)return 0 ; } +bool pci_liveupdate_is_outgoing (struct pci_dev *dev) +{ +guard (rwsem_read)(&pci_liveupdate.rwsem); +pci_WARN_ONCE (dev, !dev->liveupdate.frozen,"Preservation status is unstable!\n" ); +return dev->liveupdate.outgoing; +} +/** * pci_liveupdate_is_incoming() - Check if a device is incoming-preserved * @dev: The PCI device to check
drivers/pci/liveupdate.h +6 -0
@@ -18,6 +18,7 @@ bool pci_liveupdate_preserve_bus_numbers(void);bool pci_liveupdate_refuse_bus_numbers (struct pci_bus *bus,struct pci_dev *dev);int pci_liveupdate_adopt_acs (struct pci_dev *dev);int pci_liveupdate_adopt_ari (struct pci_dev *dev); +bool pci_liveupdate_is_outgoing (struct pci_dev *dev);static inline bool pci_liveupdate_was_incoming (struct pci_dev *dev) {
@@ -57,6 +58,11 @@ static inline int pci_liveupdate_adopt_ari(struct pci_dev *dev)return -EINVAL ; } +static inline bool pci_liveupdate_is_outgoing (struct pci_dev *dev) +{ +return false; +} +static inline bool pci_liveupdate_was_incoming (struct pci_dev *dev) {return false;
drivers/pci/pci-driver.c +4 -2
@@ -569,11 +569,13 @@ static void pci_device_shutdown(struct device *dev)/* * If this is a kexec reboot, turn off Bus Master bit on the * device to tell it to not continue to do DMA. Don't touch -* devices in D3cold or unknown states. +* devices being preserved for Live Update or in D3cold or +* unknown states. * If it is not a kexec reboot, firmware will hit the PCI * devices with big hammer and stop their DMA any way. */ -if (kexec_in_progress && (pci_dev->current_state <= PCI_D3hot)) +if (kexec_in_progress && !pci_liveupdate_is_outgoing (pci_dev) && + pci_dev->current_state <= PCI_D3hot)pci_clear_master (pci_dev); }
Signature · 1 line
2.56.1.703.gf848ed807c-goog
PATCH v11 13/13 Documentation: PCI: Add documentation for Live Update
In reply to David MatlackAdd Documentation/PCI/liveupdate.rst, which pulls in the kernel-doc from drivers/pci/liveupdate.c and include/linux/kho/abi/pci.h, and link it from the PCI index and the Live Update Orchestrator documentation.
--- Documentation/PCI/index.rst | 1 + Documentation/PCI/liveupdate.rst | 35 +++++++++++++++++++++++++++ Documentation/core-api/liveupdate.rst | 1 + MAINTAINERS | 1 + 4 files changed, 38 insertions(+) create mode 100644 Documentation/PCI/liveupdate.rst
Documentation/PCI/index.rst +1 -0
@@ -20,3 +20,4 @@ PCI Bus Subsystem controller/index boot-interrupts tph + liveupdate
Documentation/PCI/liveupdate.rst new file +35 -0
@@ -0,0 +1,35 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later + +=========================== +PCI Support for Live Update +=========================== + +.. kernel-doc:: drivers/pci/liveupdate.c + :doc: PCI Live Update + +Driver API +========== + +.. kernel-doc:: drivers/pci/liveupdate.c + :export: + +Internal API +============ + +.. kernel-doc:: drivers/pci/liveupdate.c + :internal: + +Live Update ABI +=============== + +.. kernel-doc:: include/linux/kho/abi/pci.h + :doc: PCI File-Lifecycle Bound (FLB) Live Update ABI + +.. kernel-doc:: include/linux/kho/abi/pci.h + :internal: + +See Also +======== + + * :doc:`/core-api/liveupdate` + * :doc:`/core-api/kho/index`
Documentation/core-api/liveupdate.rst +1 -0
@@ -74,3 +74,4 @@ See Also - :doc:`Live Update uAPI </userspace-api/liveupdate>` - :doc:`/core-api/kho/index` +- :doc:`PCI </PCI/liveupdate>`
MAINTAINERS +1 -0
@@ -21554,6 +21554,7 @@ L: [email protected] L: [email protected] S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux.git +F: Documentation/PCI/liveupdate.rst F: drivers/pci/liveupdate.c F: drivers/pci/liveupdate.h F: include/linux/kho/abi/pci.h
Signature · 1 line
2.56.1.703.gf848ed807c-goog
From: Mike Rapoport (Microsoft) <[email protected]>
Quoted 10 lines · David Matlack
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/liveupdate/pci/base/v11
This series introduces support in the PCI core for Live Update, enabling drivers to preserve PCI devices across a kexec-based kernel update without interrupting the device. This lets devices, such as ones assigned to VMs through VFIO, keep running across a host kernel upgrade.
[...]
I updated the lu-pci-preservation branch with this version.
I'm waiting for Alex and Bjorn to confirm they are happy with it before moving it forward.
Applied to lu-pci-preservation of liveupdate/linux.git tree, thanks!
[01/13] PCI: liveupdate: Set up FLB handler for the PCI core
(no commit info)
[02/13] PCI: liveupdate: Track outgoing preserved PCI devices
(no commit info)
[03/13] PCI: liveupdate: Track incoming preserved PCI devices
(no commit info)
[04/13] PCI: liveupdate: Document driver binding responsibilities
(no commit info)
[05/13] PCI: liveupdate: Auto-preserve upstream bridges across Live Update
(no commit info)
[06/13] PCI: liveupdate: Preserve bus numbers during Live Update
(no commit info)
[07/13] PCI: Refactor matching logic for pci_dev_acs_ops
(no commit info)
[08/13] PCI: Save and restore the ACS Control register and Egress Control Vector
(no commit info)
[09/13] PCI: liveupdate: Adopt ACS controls in incoming preserved devices
(no commit info)
[10/13] PCI: liveupdate: Adopt ARI Forwarding Enable on preserved bridges
(no commit info)
[11/13] PCI: liveupdate: Freeze preservation status during shutdown
(no commit info)
[12/13] PCI: liveupdate: Do not disable bus mastering on preserved devices during kexec
(no commit info)
[13/13] Documentation: PCI: Add documentation for Live Update
(no commit info)
tree: https://git.kernel.org/pub/scm/linux/kernel/git/liveupdate/linux
branch: liveupdate-7.5
--
Sincerely yours,
Mike.