Skip to contentSkip to conversation

vfio_pci_nvlink2: Do not attempt NPU2 setup on old P8's NPU

PATCH In review Alexey Kardashevskiy 3 participants 4 messages Nov 13, 2020 – Nov 16, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vfio_pci_nvlink2: Do not attempt NPU2 setup on old P8's NPU

We execute certain NPU2 setup code (such as mapping an LPID to a device in NPU2) unconditionally if an Nvlink bridge is detected. However this cannot succeed on P8+ machines as the init helpers return an error other than ENODEV which means the device is there is and setup failed so vfio_pci_enable() fails and pass through is not possible.

This changes the two NPU2 related init helpers to return -ENODEV if there is no "memory-region" device tree property as this is the distinction between NPU and NPU2.

Fixes 7f92891778df ("vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver")
---
 drivers/vfio/pci/vfio_pci_nvlink2.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci_nvlink2.c +5 -2
index 65c61710c0e9..9adcf6a8f888 100644
@@ -231,7 +231,7 @@ int vfio_pci_nvdia_v100_nvlink2_init(struct vfio_pci_device *vdev)
 		return -EINVAL;
 
 	if (of_property_read_u32(npu_node, "memory-region", &mem_phandle))
-		return -EINVAL;
+		return -ENODEV;
 
 	mem_node = of_find_node_by_phandle(mem_phandle);
 	if (!mem_node)
@@ -393,7 +393,7 @@ int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
 	int ret;
 	struct vfio_pci_npu2_data *data;
 	struct device_node *nvlink_dn;
-	u32 nvlink_index = 0;
+	u32 nvlink_index = 0, mem_phandle = 0;
 	struct pci_dev *npdev = vdev->pdev;
 	struct device_node *npu_node = pci_device_to_OF_node(npdev);
 	struct pci_controller *hose = pci_bus_to_host(npdev->bus);
@@ -408,6 +408,9 @@ int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
 	if (!pnv_pci_get_gpu_dev(vdev->pdev))
 		return -ENODEV;
 
+	if (of_property_read_u32(npu_node, "memory-region", &mem_phandle))
+		return -ENODEV;
+
 	/*
 	 * NPU2 normally has 8 ATSD registers (for concurrency) and 6 links
 	 * so we can allocate one register per link, using nvlink index as
Signature · 1 line
2.17.1
#1 lore raw
Quoted 6 lines · Alexey Kardashevskiy

We execute certain NPU2 setup code (such as mapping an LPID to a device in NPU2) unconditionally if an Nvlink bridge is detected. However this cannot succeed on P8+ machines as the init helpers return an error other than ENODEV which means the device is there is and setup failed so vfio_pci_enable() fails and pass through is not possible.

Quoted 6 lines

This changes the two NPU2 related init helpers to return -ENODEV if there is no "memory-region" device tree property as this is the distinction between NPU and NPU2.

Fixes 7f92891778df ("vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver")

Should this be Cc: stable?

Andrew

Signature · 2 lines
Andrew Donnellan              OzLabs, ADL Canberra
[email protected]             IBM Australia Limited
#2 lore raw
Quoted 7 lines · Andrew Donnellan
Quoted 6 lines · Alexey Kardashevskiy

We execute certain NPU2 setup code (such as mapping an LPID to a device in NPU2) unconditionally if an Nvlink bridge is detected. However this cannot succeed on P8+ machines as the init helpers return an error other than ENODEV which means the device is there is and setup failed so vfio_pci_enable() fails and pass through is not possible.

Quoted 8 lines
Quoted 6 lines

This changes the two NPU2 related init helpers to return -ENODEV if there is no "memory-region" device tree property as this is the distinction between NPU and NPU2.

Fixes 7f92891778df ("vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver")

Should this be Cc: stable?

This depends on whether P8+ + NVLink was ever a product (hi Leonardo) and had actual customers who still rely on upstream kernels to work as after many years only the last week I heard form some Redhat test engineer that it does not work. May be cc: stable...

Signature · 1 line
Alexey
#3 lore raw
Quoted 16 lines · Alexey Kardashevskiy
Quoted 15 lines · Andrew Donnellan
Quoted 13 lines · Alexey Kardashevskiy

We execute certain NPU2 setup code (such as mapping an LPID to a device in NPU2) unconditionally if an Nvlink bridge is detected. However this cannot succeed on P8+ machines as the init helpers return an error other than ENODEV which means the device is there is and setup failed so vfio_pci_enable() fails and pass through is not possible.

This changes the two NPU2 related init helpers to return -ENODEV if there is no "memory-region" device tree property as this is the distinction between NPU and NPU2.

Fixes 7f92891778df ("vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2]

subdriver")

Quoted 6 lines
Quoted 1 line

Should this be Cc: stable?

This depends on whether P8+ + NVLink was ever a product (hi Leonardo) and had actual customers who still rely on upstream kernels to work as after many years only the last week I heard form some Redhat test engineer that it does not work. May be cc: stable...

I don't think it really matters if it was a product or not. Upstream is never a product anyway.

If the fix is simple and unlikely to introduce a regression, and would potentially save someone having to debug the problem again, then it should get backported to stable.

You should also clarify what you mean by "P8+", it won't be clear to most readers if you mean "Power 8 and/or later" or specifically Naples / Power8 NVL.

cheers