Skip to content

s390x/pci: Fixing s390 vfio-pci ISM support

Matthew Rosato qemu-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest8 patches 18 repliesIn review

Changes from RFC -> v1 from the cover letter

- Refresh the header sync (built using Eric's 'update-linux-headers:
Include const.h' + manually removed pvrdma_ring.h again)
- Remove s390x/pci: fix pcistb length (already merged)
- Remove s390x/pci: Fix memory_region_access_valid call (already merged)
- Fix bug: s390_pci_vfio_pcistb should use the pre-allocated PCISTB buffer pcistb_buf rather than allocating/freeing its own.
- New patch: track the PFT (PCI Function Type) separately from guest CLP response data -- we tell the guest '0' for now due to limitations in measurement block support, but we can still use the real value provided via the vfio CLP capabilities to make decisions.
- Use the PFT (pci function type) to determine when to use the region for PCISTB/PCILG (only for ISM), rather than using the relaxed alignment bit.
- As a result, the pcistb_default is now updated to also handle the possibility of relaxed alignment via 2 new functions, pcistb_validate_write and pcistb_write, which serve as wrappers to the memory_region calls.
- New patch, which partially restores the MSI-X fence for passthrough devices... Could potentially be squashed with 's390x/pci: MSI-X isn't strictly required for passthrough' but left separately for now as I felt it needed a clear commit description of why we should still fence this case.

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8linux-headers: update against 5.11-rc4include/standard-headers/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h include/standard-headers/drm/drm_fourcc.h include/standard-headers/linux/ethtool.h +24 more+458 -410 replies
2/8s390x/pci: Keep track of the PCI Function typehw/s390x/s390-pci-vfio.c include/hw/s390x/s390-pci-bus.h+2 -00 replies
3/8s390x/pci: MSI-X isn't strictly required for passthroughhw/s390x/s390-pci-bus.c+8 -60 replies
4/8s390x/pci: Introduce the ZpciOps structurehw/s390x/s390-pci-bus.c hw/s390x/s390-pci-inst.c include/hw/s390x/s390-pci-bus.h +1 more+189 -830 replies
5/8s390x/pci: Handle devices that support relaxed alignmenthw/s390x/s390-pci-inst.c hw/s390x/s390-pci-vfio.c include/hw/s390x/s390-pci-clp.h+132 -240 replies
6/8s390x/pci: PCISTB via the vfio zPCI I/O regionhw/s390x/s390-pci-bus.c hw/s390x/s390-pci-inst.c hw/s390x/s390-pci-vfio.c +3 more+139 -00 replies
7/8s390x/pci: PCILG via the vfio zPCI I/O regionhw/s390x/s390-pci-inst.c hw/s390x/s390-pci-vfio.c include/hw/s390x/s390-pci-inst.h +1 more+64 -10 replies
8/8s390x/pci: Prevent ISM device passthrough on older host kernelshw/s390x/s390-pci-bus.c include/hw/s390x/s390-pci-bus.h+14 -00 replies

Reviews trailers per patch

1 of 8 patches has a Reviewed-by or Acked-by.

PatchPierre Morel
1/8
2/8
3/8R-b
4/8
5/8
6/8
7/8
8/8

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/8] s390x/pci: Fixing s390 vfio-pci ISM support

Today, ISM devices are completely disallowed for vfio-pci passthrough as QEMU rejects the device due to an (inappropriate) MSI-X check. Removing this fence, however, reveals additional deficiencies in the s390x PCI interception layer that prevent ISM devices from working correctly. Namely, ISM block write operations have particular requirements in regards to the alignment, size and order of writes performed that cannot be guaranteed when breaking up write operations through the typical vfio_pci_bar_rw paths. Furthermore, ISM requires that legacy/non-MIO s390 PCI instructions are used, which is also not guaranteed when the I/O is passed through the typical userspace channels.

This patchset provides a set of fixes related to enabling ISM device passthrough and includes patches to enable use of a new vfio region that will allow s390x PCI pass-through devices to perform s390 PCI instructions in such a way that the same instruction issued on the guest is re-issued on the host.

Associated kernel patchset:
https://lkml.org/lkml/2021/1/19/874

Changes from RFC -> v1:
- Refresh the header sync (built using Eric's 'update-linux-headers:
Include const.h' + manually removed pvrdma_ring.h again)
- Remove s390x/pci: fix pcistb length (already merged)
- Remove s390x/pci: Fix memory_region_access_valid call (already merged)
- Fix bug: s390_pci_vfio_pcistb should use the pre-allocated PCISTB buffer pcistb_buf rather than allocating/freeing its own.
- New patch: track the PFT (PCI Function Type) separately from guest CLP response data -- we tell the guest '0' for now due to limitations in measurement block support, but we can still use the real value provided via the vfio CLP capabilities to make decisions.
- Use the PFT (pci function type) to determine when to use the region for PCISTB/PCILG (only for ISM), rather than using the relaxed alignment bit.
- As a result, the pcistb_default is now updated to also handle the possibility of relaxed alignment via 2 new functions, pcistb_validate_write and pcistb_write, which serve as wrappers to the memory_region calls.
- New patch, which partially restores the MSI-X fence for passthrough devices... Could potentially be squashed with 's390x/pci: MSI-X isn't strictly required for passthrough' but left separately for now as I felt it needed a clear commit description of why we should still fence this case.

Matthew Rosato (8):
linux-headers: update against 5.11-rc4
s390x/pci: Keep track of the PCI Function type s390x/pci: MSI-X isn't strictly required for passthrough s390x/pci: Introduce the ZpciOps structure
s390x/pci: Handle devices that support relaxed alignment s390x/pci: PCISTB via the vfio zPCI I/O region s390x/pci: PCILG via the vfio zPCI I/O region s390x/pci: Prevent ISM device passthrough on older host kernels

 hw/s390x/s390-pci-bus.c                            |  45 ++-
 hw/s390x/s390-pci-inst.c                           | 389 +++++++++++++++------
 hw/s390x/s390-pci-vfio.c                           | 152 ++++++++
 include/hw/s390x/s390-pci-bus.h                    |  29 ++
 include/hw/s390x/s390-pci-clp.h                    |   1 +
 include/hw/s390x/s390-pci-inst.h                   |   3 +
 include/hw/s390x/s390-pci-vfio.h                   |  23 ++
 .../infiniband/hw/vmw_pvrdma/pvrdma_verbs.h        |   2 +-
 include/standard-headers/drm/drm_fourcc.h          | 175 ++++++++-
 include/standard-headers/linux/ethtool.h           |   2 +-
 include/standard-headers/linux/fuse.h              |  30 +-
 include/standard-headers/linux/kernel.h            |   9 +-
 include/standard-headers/linux/pci_regs.h          |  16 +
 include/standard-headers/linux/vhost_types.h       |   9 +
 include/standard-headers/linux/virtio_gpu.h        |  82 +++++
 include/standard-headers/linux/virtio_ids.h        |  44 ++-
 linux-headers/asm-arm64/kvm.h                      |   3 -
 linux-headers/asm-generic/unistd.h                 |   6 +-
 linux-headers/asm-mips/unistd_n32.h                |   1 +
 linux-headers/asm-mips/unistd_n64.h                |   1 +
 linux-headers/asm-mips/unistd_o32.h                |   1 +
 linux-headers/asm-powerpc/unistd_32.h              |   1 +
 linux-headers/asm-powerpc/unistd_64.h              |   1 +
 linux-headers/asm-s390/unistd_32.h                 |   1 +
 linux-headers/asm-s390/unistd_64.h                 |   1 +
 linux-headers/asm-x86/kvm.h                        |   1 +
 linux-headers/asm-x86/unistd_32.h                  |   1 +
 linux-headers/asm-x86/unistd_64.h                  |   1 +
 linux-headers/asm-x86/unistd_x32.h                 |   1 +
 linux-headers/linux/kvm.h                          |  58 ++-
 linux-headers/linux/userfaultfd.h                  |   9 +
 linux-headers/linux/vfio.h                         |   5 +
 linux-headers/linux/vfio_zdev.h                    |  34 ++
 linux-headers/linux/vhost.h                        |   4 +
 34 files changed, 996 insertions(+), 145 deletions(-)
Signature · 1 line
1.8.3.1