Skip to contentSkip to conversation

PCI: Fix isolated function probing and enable ARI for s390

PATCH In review Niklas Schnelle 2 participants 15 messages Oct 29, 2025 – Dec 24, 2025

Subsystems (4)
lore
Share & tools
lore

Niklas Schnelle

author maintainer
#0 lore raw

v5 cover PCI: Fix isolated function probing and enable ARI for s390

Hi Bjorn,

This series originally aimed to correctly detect ARI as being used on s390. I had missed however that this, in a pretty obious manner, breaks the isolated function probing resulting devices not getting probed. For example if a partition/system only has a PF with devfn 1 passed-through. Additionally the fix may (TBD) help with an issue that LoongArch has encountered when using isolated function probing and tried to fix by limiting it to bus 0 ([0]). If it does fix this it may make sense to apply this separately from the second patch.

Besides the effect on s390 the second patch should also ensure that VFs on a virtual bus created via virtfn_add_bus() correctly present ARI as enabled. Sadly I don't have access to any device to test this.

Speaking of testing. For the first patch I can reproduce the AER scenario described by only applying the second patch. The SR-IOV scenario I encountered in the past before commit 25f39d3dcb48 ("s390/pci:
Ignore RID for isolated VFs") and tested the fix now with a partial revert hack of that commit.

I also tested the series on x86_64 both on a Ryzen AI 340 based laptop and a Xeon Sapphire Rapids based system (including with SR-IOV on a ConnectX-6 DX).

Thanks,
Niklas

[0] https://lore.kernel.org/linux-pci/[email protected]/

---
Changes in v5:
- Rename variables to match "fn" abbreviation
- Add Cc stable
- Rebase on v6.18-rc3
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- Add fix of isolated function probing with ARI enabled and in certain
cases with SR-IOV devices.
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- Move setting of ari_enabled on s390 to bus creation and clear it in pcibios_add_device() if the capability is not available.
- Rebase on v6.15-rc2
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- Rebased on v6.11
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Niklas Schnelle (2):
PCI: Fix isolated PCI function probing with ARI and SR-IOV
PCI: s390: Handle ARI on bus without associated struct pci_dev

 arch/s390/pci/pci.c     |  7 +++++++
 arch/s390/pci/pci_bus.c | 10 ++++++++++
 drivers/pci/pci.c       |  4 ++--
 drivers/pci/probe.c     | 22 ++++++++++++++--------
 include/linux/pci.h     |  4 ++--
 5 files changed, 35 insertions(+), 12 deletions(-)

---
base-commit: dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa
change-id: 20240724-ari_no_bus_dev-52b2a27f3466

Best regards,

Signature · 1 line
Niklas Schnelle
author maintainer
#1 lore raw

PATCH v5 1/2 PCI: Fix isolated PCI function probing with ARI and SR-IOV

When the isolated PCI function probing mechanism is used in conjunction with ARI or SR-IOV it may not find all available PCI functions. In the case of ARI the problem is that next_ari_fn() always returns -ENODEV if dev is NULL and thus if fn 0 is missing the scan stops.

For SR-IOV things are more complex. Here the problem is that the check for multifunction may fail. One example where this can occur is if the first passed-through function is a VF with devfn 8. Now in pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it remains unset and probing stops even if there is a devfn 9.

Now at the moment both of these issues are hidden on s390. The first one because ARI is detected as disabled as struct pci_bus's self is NULL even though firmware does enable and use ARI. The second issue is hidden as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for isolated VFs"). This is because VFs are either put on their own virtual bus if the parent PF is not passed-through to the same instance or VFs are hotplugged once SR-IOV is enabled on the parent PF and then pci_scan_single_device() is used.

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

Fixes 189c6c33ff42 ("PCI: Extend isolated function probing to s390")
---
 drivers/pci/probe.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
drivers/pci/probe.c +13 -8
index 0ce98e18b5a876afe72af35a9f4a44d598e8d500..41dd1a339a994956a6bc7e1fea0fe0d55452a963 100644
@@ -2808,16 +2808,19 @@ static int next_ari_fn(struct pci_bus *bus, struct pci_dev *dev, int fn)
 	return next_fn;
 }
 
-static int next_fn(struct pci_bus *bus, struct pci_dev *dev, int fn)
+static int next_fn(struct pci_bus *bus, struct pci_dev *dev, int fn,
+		   bool isolated_fns)
 {
-	if (pci_ari_enabled(bus))
-		return next_ari_fn(bus, dev, fn);
+	if (!isolated_fns) {
+		if (pci_ari_enabled(bus))
+			return next_ari_fn(bus, dev, fn);
 
+		/* only multifunction devices may have more functions */
+		if (dev && !dev->multifunction)
+			return -ENODEV;
+	}
 	if (fn >= 7)
 		return -ENODEV;
-	/* only multifunction devices may have more functions */
-	if (dev && !dev->multifunction)
-		return -ENODEV;
 
 	return fn + 1;
 }
@@ -2859,10 +2862,12 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
 {
 	struct pci_dev *dev;
 	int fn = 0, nr = 0;
+	bool isolated_fns;
 
 	if (only_one_child(bus) && (devfn > 0))
 		return 0; /* Already scanned the entire slot */
 
+	isolated_fns = hypervisor_isolated_pci_functions();
 	do {
 		dev = pci_scan_single_device(bus, devfn + fn);
 		if (dev) {
@@ -2876,10 +2881,10 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
 			 * a hypervisor that passes through individual PCI
 			 * functions.
 			 */
-			if (!hypervisor_isolated_pci_functions())
+			if (!isolated_fns)
 				break;
 		}
-		fn = next_fn(bus, dev, fn);
+		fn = next_fn(bus, dev, fn, isolated_fns);
 	} while (fn >= 0);
 
 	/* Only one slot has PCIe device */
Signature · 1 line
2.48.1
maintainer
#2 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Hi, Niklas,

Quoted 22 lines · Niklas Schnelle

When the isolated PCI function probing mechanism is used in conjunction with ARI or SR-IOV it may not find all available PCI functions. In the case of ARI the problem is that next_ari_fn() always returns -ENODEV if dev is NULL and thus if fn 0 is missing the scan stops.

For SR-IOV things are more complex. Here the problem is that the check for multifunction may fail. One example where this can occur is if the first passed-through function is a VF with devfn 8. Now in pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it remains unset and probing stops even if there is a devfn 9.

Now at the moment both of these issues are hidden on s390. The first one because ARI is detected as disabled as struct pci_bus's self is NULL even though firmware does enable and use ARI. The second issue is hidden as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for isolated VFs"). This is because VFs are either put on their own virtual bus if the parent PF is not passed-through to the same instance or VFs are hotplugged once SR-IOV is enabled on the parent PF and then pci_scan_single_device() is used.

Quoted 4 lines

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

I'm very sorry, but applying this patch on top of commit a02fd05661d7 ("PCI: Extend isolated function probing to LoongArch") we fail to boot.

Boot log:

[   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
00:01:03 EST 2006)
[   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[   10.379564] megasas: 07.734.00.00-rc1
[   10.383222] mpt3sas version 54.100.00.00 loaded
[   10.388304] nvme nvme0: pci function 0000:08:00.0
[   10.395088] Freeing initrd memory: 45632K
[   10.469822] ------------[ cut here ]------------
[   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
ahci_enable_ahci+0x64/0xb8
[   10.482804] Modules linked in:
[   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.508139] Workqueue: events work_for_cpu_fn
[   10.512468] pc 900000000103be2c ra 900000000103be28 tp
900000010ae44000 sp 900000010ae47be0
[   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
0000000000000001 a3 9000000001810e0c
[   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
0000000000000010 a7 0000000000000000
[   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
0000000000001280 t3 000000000674c000
[   10.545673] t4 0000000000000000 t5 0000000000000000 t6
9000000008002480 t7 00000000000000b4
[   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
9000000002221a50 s0 0000000000000000
[   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
ffffffff80000000 s4 9000000002221570
[   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
90000000023dd000 s8 900000010ae47d08
[   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
[   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
[   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.621878] Workqueue: events work_for_cpu_fn
[   10.621881] Stack : 900000010ae47848 0000000000000000
90000000002436bc 900000010ae44000
[   10.621884]         900000010ae47820 900000010ae47828
0000000000000000 900000010ae47968
[   10.621887]         900000010ae47960 900000010ae47960
900000010ae47630 0000000000000001
[   10.621890]         0000000000000001 900000010ae47828
d10951fa66920f31 9000000100414300
[   10.621893]         80000000ffffe34d fffffffffffffffe
000000000000034f 000000000000002f
[   10.621896]         0000000000000063 0000000000000001
000000000674c000 9000000002221a50
[   10.621899]         0000000000000000 0000000000000000
90000000020b6500 90000000023dd000
[   10.621902]         00000000000000e9 0000000000000009
0000000000000002 90000000023dd000
[   10.621905]         900000010ae47d08 0000000000000000
90000000002436d4 0000000000000000
[   10.621908]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   10.621910]         ...
[   10.621912] Call Trace:
[   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
[   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
[   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
[   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
[   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
[   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
[   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
[   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
[   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
[   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
[   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
[   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
[   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
[   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
[   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
[   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
[   10.621973]

Huacai

Quoted 69 lines
Fixes 189c6c33ff42 ("PCI: Extend isolated function probing to s390")
---
 drivers/pci/probe.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
drivers/pci/probe.c +13 -9
index 0ce98e18b5a876afe72af35a9f4a44d598e8d500..41dd1a339a994956a6bc7e1fea0fe0d55452a963 100644
@@ -2808,16 +2808,19 @@ static int next_ari_fn(struct pci_bus *bus, struct pci_dev *dev, int fn)
        return next_fn;
 }
 
-static int next_fn(struct pci_bus *bus, struct pci_dev *dev, int fn)
+static int next_fn(struct pci_bus *bus, struct pci_dev *dev, int fn,
+                  bool isolated_fns)
 {
-       if (pci_ari_enabled(bus))
-               return next_ari_fn(bus, dev, fn);
+       if (!isolated_fns) {
+               if (pci_ari_enabled(bus))
+                       return next_ari_fn(bus, dev, fn);
 
+               /* only multifunction devices may have more functions */
+               if (dev && !dev->multifunction)
+                       return -ENODEV;
+       }
        if (fn >= 7)
                return -ENODEV;
-       /* only multifunction devices may have more functions */
-       if (dev && !dev->multifunction)
-               return -ENODEV;
 
        return fn + 1;
 }
@@ -2859,10 +2862,12 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
 {
        struct pci_dev *dev;
        int fn = 0, nr = 0;
+       bool isolated_fns;
 
        if (only_one_child(bus) && (devfn > 0))
                return 0; /* Already scanned the entire slot */
 
+       isolated_fns = hypervisor_isolated_pci_functions();
        do {
                dev = pci_scan_single_device(bus, devfn + fn);
                if (dev) {
@@ -2876,10 +2881,10 @@ int pci_scan_slot(struct pci_bus *bus, int devfn)
                         * a hypervisor that passes through individual PCI
                         * functions.
                         */
-                       if (!hypervisor_isolated_pci_functions())
+                       if (!isolated_fns)
                                break;
                }
-               fn = next_fn(bus, dev, fn);
+               fn = next_fn(bus, dev, fn, isolated_fns);
        } while (fn >= 0);
 
        /* Only one slot has PCIe device */
 
--

2.48.1

author maintainer
#3 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 112 lines · Huacai Chen

Hi, Niklas,

Quoted 22 lines · Niklas Schnelle

When the isolated PCI function probing mechanism is used in conjunction with ARI or SR-IOV it may not find all available PCI functions. In the case of ARI the problem is that next_ari_fn() always returns -ENODEV if dev is NULL and thus if fn 0 is missing the scan stops.

For SR-IOV things are more complex. Here the problem is that the check for multifunction may fail. One example where this can occur is if the first passed-through function is a VF with devfn 8. Now in pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it remains unset and probing stops even if there is a devfn 9.

Now at the moment both of these issues are hidden on s390. The first one because ARI is detected as disabled as struct pci_bus's self is NULL even though firmware does enable and use ARI. The second issue is hidden as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for isolated VFs"). This is because VFs are either put on their own virtual bus if the parent PF is not passed-through to the same instance or VFs are hotplugged once SR-IOV is enabled on the parent PF and then pci_scan_single_device() is used.

Quoted 4 lines

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

I'm very sorry, but applying this patch on top of commit a02fd05661d7 ("PCI: Extend isolated function probing to LoongArch") we fail to boot.

Boot log:

[   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
00:01:03 EST 2006)
[   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[   10.379564] megasas: 07.734.00.00-rc1
[   10.383222] mpt3sas version 54.100.00.00 loaded
[   10.388304] nvme nvme0: pci function 0000:08:00.0
[   10.395088] Freeing initrd memory: 45632K
[   10.469822] ------------[ cut here ]------------
[   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
ahci_enable_ahci+0x64/0xb8
[   10.482804] Modules linked in:
[   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.508139] Workqueue: events work_for_cpu_fn
[   10.512468] pc 900000000103be2c ra 900000000103be28 tp
900000010ae44000 sp 900000010ae47be0
[   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
0000000000000001 a3 9000000001810e0c
[   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
0000000000000010 a7 0000000000000000
[   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
0000000000001280 t3 000000000674c000
[   10.545673] t4 0000000000000000 t5 0000000000000000 t6
9000000008002480 t7 00000000000000b4
[   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
9000000002221a50 s0 0000000000000000
[   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
ffffffff80000000 s4 9000000002221570
[   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
90000000023dd000 s8 900000010ae47d08
[   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
[   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
[   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.621878] Workqueue: events work_for_cpu_fn
[   10.621881] Stack : 900000010ae47848 0000000000000000
90000000002436bc 900000010ae44000
[   10.621884]         900000010ae47820 900000010ae47828
0000000000000000 900000010ae47968
[   10.621887]         900000010ae47960 900000010ae47960
900000010ae47630 0000000000000001
[   10.621890]         0000000000000001 900000010ae47828
d10951fa66920f31 9000000100414300
[   10.621893]         80000000ffffe34d fffffffffffffffe
000000000000034f 000000000000002f
[   10.621896]         0000000000000063 0000000000000001
000000000674c000 9000000002221a50
[   10.621899]         0000000000000000 0000000000000000
90000000020b6500 90000000023dd000
[   10.621902]         00000000000000e9 0000000000000009
0000000000000002 90000000023dd000
[   10.621905]         900000010ae47d08 0000000000000000
90000000002436d4 0000000000000000
[   10.621908]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   10.621910]         ...
[   10.621912] Call Trace:
[   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
[   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
[   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
[   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
[   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
[   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
[   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
[   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
[   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
Quoted 10 lines
[   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
[   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
[   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
[   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
[   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
[   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
[   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
[   10.621973]

This looks like a warning telling us that AHCI enable failed / timed out. Do you have Panic on Warn on that this directly causes a boot failure? The only relation I can see with my patch is that maybe this AHCI device wasn't probed before and somehow isn't working?

Thanks,
Niklas

maintainer
#4 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 125 lines · Niklas Schnelle
Quoted 112 lines · Huacai Chen

Hi, Niklas,

Quoted 22 lines · Niklas Schnelle

When the isolated PCI function probing mechanism is used in conjunction with ARI or SR-IOV it may not find all available PCI functions. In the case of ARI the problem is that next_ari_fn() always returns -ENODEV if dev is NULL and thus if fn 0 is missing the scan stops.

For SR-IOV things are more complex. Here the problem is that the check for multifunction may fail. One example where this can occur is if the first passed-through function is a VF with devfn 8. Now in pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it remains unset and probing stops even if there is a devfn 9.

Now at the moment both of these issues are hidden on s390. The first one because ARI is detected as disabled as struct pci_bus's self is NULL even though firmware does enable and use ARI. The second issue is hidden as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for isolated VFs"). This is because VFs are either put on their own virtual bus if the parent PF is not passed-through to the same instance or VFs are hotplugged once SR-IOV is enabled on the parent PF and then pci_scan_single_device() is used.

Quoted 4 lines

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

I'm very sorry, but applying this patch on top of commit a02fd05661d7 ("PCI: Extend isolated function probing to LoongArch") we fail to boot.

Boot log:

[   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
00:01:03 EST 2006)
[   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[   10.379564] megasas: 07.734.00.00-rc1
[   10.383222] mpt3sas version 54.100.00.00 loaded
[   10.388304] nvme nvme0: pci function 0000:08:00.0
[   10.395088] Freeing initrd memory: 45632K
[   10.469822] ------------[ cut here ]------------
[   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
ahci_enable_ahci+0x64/0xb8
[   10.482804] Modules linked in:
[   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.508139] Workqueue: events work_for_cpu_fn
[   10.512468] pc 900000000103be2c ra 900000000103be28 tp
900000010ae44000 sp 900000010ae47be0
[   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
0000000000000001 a3 9000000001810e0c
[   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
0000000000000010 a7 0000000000000000
[   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
0000000000001280 t3 000000000674c000
[   10.545673] t4 0000000000000000 t5 0000000000000000 t6
9000000008002480 t7 00000000000000b4
[   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
9000000002221a50 s0 0000000000000000
[   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
ffffffff80000000 s4 9000000002221570
[   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
90000000023dd000 s8 900000010ae47d08
[   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
[   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
[   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.621878] Workqueue: events work_for_cpu_fn
[   10.621881] Stack : 900000010ae47848 0000000000000000
90000000002436bc 900000010ae44000
[   10.621884]         900000010ae47820 900000010ae47828
0000000000000000 900000010ae47968
[   10.621887]         900000010ae47960 900000010ae47960
900000010ae47630 0000000000000001
[   10.621890]         0000000000000001 900000010ae47828
d10951fa66920f31 9000000100414300
[   10.621893]         80000000ffffe34d fffffffffffffffe
000000000000034f 000000000000002f
[   10.621896]         0000000000000063 0000000000000001
000000000674c000 9000000002221a50
[   10.621899]         0000000000000000 0000000000000000
90000000020b6500 90000000023dd000
[   10.621902]         00000000000000e9 0000000000000009
0000000000000002 90000000023dd000
[   10.621905]         900000010ae47d08 0000000000000000
90000000002436d4 0000000000000000
[   10.621908]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   10.621910]         ...
[   10.621912] Call Trace:
[   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
[   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
[   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
[   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
[   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
[   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
[   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
[   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
[   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
Quoted 10 lines
[   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
[   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
[   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
[   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
[   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
[   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
[   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
[   10.621973]
Quoted 4 lines

This looks like a warning telling us that AHCI enable failed / timed out. Do you have Panic on Warn on that this directly causes a boot failure? The only relation I can see with my patch is that maybe this AHCI device wasn't probed before and somehow isn't working?

The rootfs is on the AHCI controller, so AHCI failure causes the boot failure, without this patch no boot problems.

Huacai

Quoted 3 lines

Thanks,
Niklas

author maintainer
#5 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 126 lines · Huacai Chen
Quoted 125 lines · Niklas Schnelle
Quoted 122 lines · Huacai Chen

Hi, Niklas,

Quoted 22 lines · Niklas Schnelle

When the isolated PCI function probing mechanism is used in conjunction with ARI or SR-IOV it may not find all available PCI functions. In the case of ARI the problem is that next_ari_fn() always returns -ENODEV if dev is NULL and thus if fn 0 is missing the scan stops.

For SR-IOV things are more complex. Here the problem is that the check for multifunction may fail. One example where this can occur is if the first passed-through function is a VF with devfn 8. Now in pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it remains unset and probing stops even if there is a devfn 9.

Now at the moment both of these issues are hidden on s390. The first one because ARI is detected as disabled as struct pci_bus's self is NULL even though firmware does enable and use ARI. The second issue is hidden as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for isolated VFs"). This is because VFs are either put on their own virtual bus if the parent PF is not passed-through to the same instance or VFs are hotplugged once SR-IOV is enabled on the parent PF and then pci_scan_single_device() is used.

Quoted 4 lines

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

I'm very sorry, but applying this patch on top of commit a02fd05661d7 ("PCI: Extend isolated function probing to LoongArch") we fail to boot.

Boot log:

[   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
00:01:03 EST 2006)
[   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[   10.379564] megasas: 07.734.00.00-rc1
[   10.383222] mpt3sas version 54.100.00.00 loaded
[   10.388304] nvme nvme0: pci function 0000:08:00.0
[   10.395088] Freeing initrd memory: 45632K
[   10.469822] ------------[ cut here ]------------
[   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
ahci_enable_ahci+0x64/0xb8
[   10.482804] Modules linked in:
[   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.508139] Workqueue: events work_for_cpu_fn
[   10.512468] pc 900000000103be2c ra 900000000103be28 tp
900000010ae44000 sp 900000010ae47be0
[   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
0000000000000001 a3 9000000001810e0c
[   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
0000000000000010 a7 0000000000000000
[   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
0000000000001280 t3 000000000674c000
[   10.545673] t4 0000000000000000 t5 0000000000000000 t6
9000000008002480 t7 00000000000000b4
[   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
9000000002221a50 s0 0000000000000000
[   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
ffffffff80000000 s4 9000000002221570
[   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
90000000023dd000 s8 900000010ae47d08
[   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
[   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
[   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.621878] Workqueue: events work_for_cpu_fn
[   10.621881] Stack : 900000010ae47848 0000000000000000
90000000002436bc 900000010ae44000
[   10.621884]         900000010ae47820 900000010ae47828
0000000000000000 900000010ae47968
[   10.621887]         900000010ae47960 900000010ae47960
900000010ae47630 0000000000000001
[   10.621890]         0000000000000001 900000010ae47828
d10951fa66920f31 9000000100414300
[   10.621893]         80000000ffffe34d fffffffffffffffe
000000000000034f 000000000000002f
[   10.621896]         0000000000000063 0000000000000001
000000000674c000 9000000002221a50
[   10.621899]         0000000000000000 0000000000000000
90000000020b6500 90000000023dd000
[   10.621902]         00000000000000e9 0000000000000009
0000000000000002 90000000023dd000
[   10.621905]         900000010ae47d08 0000000000000000
90000000002436d4 0000000000000000
[   10.621908]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   10.621910]         ...
[   10.621912] Call Trace:
[   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
[   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
[   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
[   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
[   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
[   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
[   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
[   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
[   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
[   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
[   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
[   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
[   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
[   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
[   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
[   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
[   10.621973]
Quoted 9 lines
Quoted 4 lines

This looks like a warning telling us that AHCI enable failed / timed out. Do you have Panic on Warn on that this directly causes a boot failure? The only relation I can see with my patch is that maybe this AHCI device wasn't probed before and somehow isn't working?

The rootfs is on the AHCI controller, so AHCI failure causes the boot failure, without this patch no boot problems.

Huacai

Ok, I'm going to need more details to make sense of this. Can you tell me if ARI is enabled for that bus? Did you test with both patches or just this one? Could you provide lspci -vv from a good boot and can you tell which AHCI device the root device is on? Also could you clarify why you set hypervisor_isolated_pci_functions() in particular this seems like a bare metal boot, right? When running in KVM do you pass- through individual PCI functions with the guest seeing a devfn other than 0 alone, i.e. a missing devfn 0? Or do you need this for bare metal for some reason? If you don't need it for bare metal, does the boot work if you return 0 from hypervisor_isolated_pci_functions() with this patch?

Thanks,
Niklas

maintainer
#6 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 138 lines · Niklas Schnelle
Quoted 126 lines · Huacai Chen
Quoted 125 lines · Niklas Schnelle
Quoted 122 lines · Huacai Chen

Hi, Niklas,

Quoted 22 lines · Niklas Schnelle

When the isolated PCI function probing mechanism is used in conjunction with ARI or SR-IOV it may not find all available PCI functions. In the case of ARI the problem is that next_ari_fn() always returns -ENODEV if dev is NULL and thus if fn 0 is missing the scan stops.

For SR-IOV things are more complex. Here the problem is that the check for multifunction may fail. One example where this can occur is if the first passed-through function is a VF with devfn 8. Now in pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it remains unset and probing stops even if there is a devfn 9.

Now at the moment both of these issues are hidden on s390. The first one because ARI is detected as disabled as struct pci_bus's self is NULL even though firmware does enable and use ARI. The second issue is hidden as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for isolated VFs"). This is because VFs are either put on their own virtual bus if the parent PF is not passed-through to the same instance or VFs are hotplugged once SR-IOV is enabled on the parent PF and then pci_scan_single_device() is used.

Quoted 4 lines

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

I'm very sorry, but applying this patch on top of commit a02fd05661d7 ("PCI: Extend isolated function probing to LoongArch") we fail to boot.

Boot log:

[   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
00:01:03 EST 2006)
[   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[   10.379564] megasas: 07.734.00.00-rc1
[   10.383222] mpt3sas version 54.100.00.00 loaded
[   10.388304] nvme nvme0: pci function 0000:08:00.0
[   10.395088] Freeing initrd memory: 45632K
[   10.469822] ------------[ cut here ]------------
[   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
ahci_enable_ahci+0x64/0xb8
[   10.482804] Modules linked in:
[   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.508139] Workqueue: events work_for_cpu_fn
[   10.512468] pc 900000000103be2c ra 900000000103be28 tp
900000010ae44000 sp 900000010ae47be0
[   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
0000000000000001 a3 9000000001810e0c
[   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
0000000000000010 a7 0000000000000000
[   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
0000000000001280 t3 000000000674c000
[   10.545673] t4 0000000000000000 t5 0000000000000000 t6
9000000008002480 t7 00000000000000b4
[   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
9000000002221a50 s0 0000000000000000
[   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
ffffffff80000000 s4 9000000002221570
[   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
90000000023dd000 s8 900000010ae47d08
[   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
[   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
[   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.621878] Workqueue: events work_for_cpu_fn
[   10.621881] Stack : 900000010ae47848 0000000000000000
90000000002436bc 900000010ae44000
[   10.621884]         900000010ae47820 900000010ae47828
0000000000000000 900000010ae47968
[   10.621887]         900000010ae47960 900000010ae47960
900000010ae47630 0000000000000001
[   10.621890]         0000000000000001 900000010ae47828
d10951fa66920f31 9000000100414300
[   10.621893]         80000000ffffe34d fffffffffffffffe
000000000000034f 000000000000002f
[   10.621896]         0000000000000063 0000000000000001
000000000674c000 9000000002221a50
[   10.621899]         0000000000000000 0000000000000000
90000000020b6500 90000000023dd000
[   10.621902]         00000000000000e9 0000000000000009
0000000000000002 90000000023dd000
[   10.621905]         900000010ae47d08 0000000000000000
90000000002436d4 0000000000000000
[   10.621908]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   10.621910]         ...
[   10.621912] Call Trace:
[   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
[   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
[   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
[   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
[   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
[   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
[   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
[   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
[   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
[   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
[   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
[   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
[   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
[   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
[   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
[   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
[   10.621973]
Quoted 9 lines
Quoted 4 lines

This looks like a warning telling us that AHCI enable failed / timed out. Do you have Panic on Warn on that this directly causes a boot failure? The only relation I can see with my patch is that maybe this AHCI device wasn't probed before and somehow isn't working?

The rootfs is on the AHCI controller, so AHCI failure causes the boot failure, without this patch no boot problems.

Huacai

Quoted 11 lines

Ok, I'm going to need more details to make sense of this. Can you tell me if ARI is enabled for that bus? Did you test with both patches or just this one? Could you provide lspci -vv from a good boot and can you tell which AHCI device the root device is on? Also could you clarify why you set hypervisor_isolated_pci_functions() in particular this seems like a bare metal boot, right? When running in KVM do you pass- through individual PCI functions with the guest seeing a devfn other than 0 alone, i.e. a missing devfn 0? Or do you need this for bare metal for some reason? If you don't need it for bare metal, does the boot work if you return 0 from hypervisor_isolated_pci_functions() with this patch?

1. ARI isn't enabled.
2. Only test the first patch.
3. This is a bare metal boot.
4. If hypervisor_isolated_pci_functions() return 0 then boot is OK
5. PCI information please see the attachment.

Huacai

Quoted 3 lines

Thanks,
Niklas

author maintainer
#7 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 150 lines · Huacai Chen
Quoted 138 lines · Niklas Schnelle
Quoted 126 lines · Huacai Chen
Quoted 125 lines · Niklas Schnelle
Quoted 122 lines · Huacai Chen

Hi, Niklas,

Quoted 22 lines · Niklas Schnelle

When the isolated PCI function probing mechanism is used in conjunction with ARI or SR-IOV it may not find all available PCI functions. In the case of ARI the problem is that next_ari_fn() always returns -ENODEV if dev is NULL and thus if fn 0 is missing the scan stops.

For SR-IOV things are more complex. Here the problem is that the check for multifunction may fail. One example where this can occur is if the first passed-through function is a VF with devfn 8. Now in pci_scan_slot() this means it is fn 0 and thus multifunction doesn't get set. Since VFs don't get multifunction set via PCI_HEADER_TYPE_MFD it remains unset and probing stops even if there is a devfn 9.

Now at the moment both of these issues are hidden on s390. The first one because ARI is detected as disabled as struct pci_bus's self is NULL even though firmware does enable and use ARI. The second issue is hidden as a side effect of commit 25f39d3dcb48 ("s390/pci: Ignore RID for isolated VFs"). This is because VFs are either put on their own virtual bus if the parent PF is not passed-through to the same instance or VFs are hotplugged once SR-IOV is enabled on the parent PF and then pci_scan_single_device() is used.

Quoted 4 lines

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

I'm very sorry, but applying this patch on top of commit a02fd05661d7 ("PCI: Extend isolated function probing to LoongArch") we fail to boot.

Boot log:

[   10.365340] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16
00:01:03 EST 2006)
[   10.372628] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[   10.379564] megasas: 07.734.00.00-rc1
[   10.383222] mpt3sas version 54.100.00.00 loaded
[   10.388304] nvme nvme0: pci function 0000:08:00.0
[   10.395088] Freeing initrd memory: 45632K
[   10.469822] ------------[ cut here ]------------
[   10.474409] WARNING: CPU: 0 PID: 247 at drivers/ata/libahci.c:233
ahci_enable_ahci+0x64/0xb8
[   10.482804] Modules linked in:
[   10.485838] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.494397] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.508139] Workqueue: events work_for_cpu_fn
[   10.512468] pc 900000000103be2c ra 900000000103be28 tp
900000010ae44000 sp 900000010ae47be0
[   10.520769] a0 0000000000000000 a1 00000000000000b0 a2
0000000000000001 a3 9000000001810e0c
[   10.529069] a4 9000000002343e20 a5 0000000000000001 a6
0000000000000010 a7 0000000000000000
[   10.537373] t0 d10951fa66920f31 t1 d10951fa66920f31 t2
0000000000001280 t3 000000000674c000
[   10.545673] t4 0000000000000000 t5 0000000000000000 t6
9000000008002480 t7 00000000000000b4
[   10.553972] t8 90000001055eab90 u0 900000010ae47b68 s9
9000000002221a50 s0 0000000000000000
[   10.562272] s1 ffff800032435800 s2 0000000000000000 s3
ffffffff80000000 s4 9000000002221570
[   10.570571] s5 0000000000000005 s6 9000000101ccf0b8 s7
90000000023dd000 s8 900000010ae47d08
[   10.578869]    ra: 900000000103be28 ahci_enable_ahci+0x60/0xb8
[   10.584665]   ERA: 900000000103be2c ahci_enable_ahci+0x64/0xb8
[   10.590461]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   10.596609]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   10.600937]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   10.605698]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   10.610458] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   10.615994]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   10.621875] CPU: 0 UID: 0 PID: 247 Comm: kworker/0:11 Not tainted
6.18.0-rc3 #1 PREEMPT(full)
[   10.621877] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.621878] Workqueue: events work_for_cpu_fn
[   10.621881] Stack : 900000010ae47848 0000000000000000
90000000002436bc 900000010ae44000
[   10.621884]         900000010ae47820 900000010ae47828
0000000000000000 900000010ae47968
[   10.621887]         900000010ae47960 900000010ae47960
900000010ae47630 0000000000000001
[   10.621890]         0000000000000001 900000010ae47828
d10951fa66920f31 9000000100414300
[   10.621893]         80000000ffffe34d fffffffffffffffe
000000000000034f 000000000000002f
[   10.621896]         0000000000000063 0000000000000001
000000000674c000 9000000002221a50
[   10.621899]         0000000000000000 0000000000000000
90000000020b6500 90000000023dd000
[   10.621902]         00000000000000e9 0000000000000009
0000000000000002 90000000023dd000
[   10.621905]         900000010ae47d08 0000000000000000
90000000002436d4 0000000000000000
[   10.621908]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   10.621910]         ...
[   10.621912] Call Trace:
[   10.621913] [<90000000002436d4>] show_stack+0x5c/0x180
[   10.621918] [<900000000023f328>] dump_stack_lvl+0x6c/0x9c
[   10.621923] [<9000000000266eb8>] __warn+0x80/0x108
[   10.621927] [<90000000017d1910>] report_bug+0x158/0x2a8
[   10.621932] [<900000000180b610>] do_bp+0x2d0/0x340
[   10.621938] [<9000000000241da0>] handle_bp+0x120/0x1c0
[   10.621940] [<900000000103be2c>] ahci_enable_ahci+0x64/0xb8
[   10.621943] [<900000000103beb8>] ahci_save_initial_config+0x38/0x4d8
[   10.621946] [<90000000010391b4>] ahci_init_one+0x354/0x1088
[   10.621950] [<9000000000d16cdc>] local_pci_probe+0x44/0xb8
[   10.621953] [<9000000000286f78>] work_for_cpu_fn+0x18/0x30
[   10.621956] [<900000000028a840>] process_one_work+0x160/0x330
[   10.621961] [<900000000028b208>] worker_thread+0x330/0x460
[   10.621964] [<9000000000295fdc>] kthread+0x11c/0x138
[   10.621968] [<900000000180b740>] ret_from_kernel_thread+0x28/0xa8
[   10.621971] [<9000000000241484>] ret_from_kernel_thread_asm+0xc/0x88
[   10.621973]
Quoted 9 lines
Quoted 4 lines

This looks like a warning telling us that AHCI enable failed / timed out. Do you have Panic on Warn on that this directly causes a boot failure? The only relation I can see with my patch is that maybe this AHCI device wasn't probed before and somehow isn't working?

The rootfs is on the AHCI controller, so AHCI failure causes the boot failure, without this patch no boot problems.

Huacai

Quoted 11 lines

Ok, I'm going to need more details to make sense of this. Can you tell me if ARI is enabled for that bus? Did you test with both patches or just this one? Could you provide lspci -vv from a good boot and can you tell which AHCI device the root device is on? Also could you clarify why you set hypervisor_isolated_pci_functions() in particular this seems like a bare metal boot, right? When running in KVM do you pass- through individual PCI functions with the guest seeing a devfn other than 0 alone, i.e. a missing devfn 0? Or do you need this for bare metal for some reason? If you don't need it for bare metal, does the boot work if you return 0 from hypervisor_isolated_pci_functions() with this patch?

Quoted 7 lines

1. ARI isn't enabled.
2. Only test the first patch.
3. This is a bare metal boot.
4. If hypervisor_isolated_pci_functions() return 0 then boot is OK
5. PCI information please see the attachment.

Huacai

Thanks for the input. As far as I can see the lspci from a good boot shows no holes in your devfn space so this particular system doesn't seem to need the isolated function probing at all. But even then using it should only try out all devfns and thus never skip one that is found without isolated function probing.

To sanity check this, I just booted my personal AMD Ryzen 3900X system with this series plus a two-liner to unconditionally enable isolated function probing also on x86_64 and it came up fine including AMD graphics and my Intel NIC with enabled SR-IOV.

So I'm really perplexed and coming back to the thought that a device on your system is misbehaving when probing is attempted and maybe due to a similar issue as what I saw with SR-IOV it wasn't probed so far but really should be probed if isolated function probing is enabled. I also still don't understand your use-case. If it is for VMs then maybe you could limit it to those? Otherwise it feels like this is just a hack to probe an odd topology and I wonder if you should rather set PCI_SCAN_ALL_PCIE_DEVS to find those?

Thanks,
Niklas

author maintainer
#8 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 10 lines · Niklas Schnelle
Quoted 9 lines · Huacai Chen
Quoted 8 lines · Niklas Schnelle
Quoted 6 lines · Huacai Chen
Quoted 5 lines · Niklas Schnelle
Quoted 3 lines · Huacai Chen

Hi, Niklas,

On Wed, Oct 29, 2025 at 5:42 PM Niklas Schnelle <[email protected]> wrote:

--- snip ---

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

I'm very sorry, but applying this patch on top of commit a02fd05661d7 ("PCI: Extend isolated function probing to LoongArch") we fail to boot.

Boot log:

--- snip ---

Quoted 51 lines
Quoted 23 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

This looks like a warning telling us that AHCI enable failed / timed out. Do you have Panic on Warn on that this directly causes a boot failure? The only relation I can see with my patch is that maybe this AHCI device wasn't probed before and somehow isn't working?

The rootfs is on the AHCI controller, so AHCI failure causes the boot failure, without this patch no boot problems.

Huacai

Quoted 11 lines

Ok, I'm going to need more details to make sense of this. Can you tell me if ARI is enabled for that bus? Did you test with both patches or just this one? Could you provide lspci -vv from a good boot and can you tell which AHCI device the root device is on? Also could you clarify why you set hypervisor_isolated_pci_functions() in particular this seems like a bare metal boot, right? When running in KVM do you pass- through individual PCI functions with the guest seeing a devfn other than 0 alone, i.e. a missing devfn 0? Or do you need this for bare metal for some reason? If you don't need it for bare metal, does the boot work if you return 0 from hypervisor_isolated_pci_functions() with this patch?

Quoted 7 lines

1. ARI isn't enabled.
2. Only test the first patch.
3. This is a bare metal boot.
4. If hypervisor_isolated_pci_functions() return 0 then boot is OK
5. PCI information please see the attachment.

Huacai

Thanks for the input. As far as I can see the lspci from a good boot shows no holes in your devfn space so this particular system doesn't seem to need the isolated function probing at all. But even then using it should only try out all devfns and thus never skip one that is found without isolated function probing.

To sanity check this, I just booted my personal AMD Ryzen 3900X system with this series plus a two-liner to unconditionally enable isolated function probing also on x86_64 and it came up fine including AMD graphics and my Intel NIC with enabled SR-IOV.

So I'm really perplexed and coming back to the thought that a device on your system is misbehaving when probing is attempted and maybe due to a similar issue as what I saw with SR-IOV it wasn't probed so far but really should be probed if isolated function probing is enabled. I also still don't understand your use-case. If it is for VMs then maybe you could limit it to those? Otherwise it feels like this is just a hack to probe an odd topology and I wonder if you should rather set PCI_SCAN_ALL_PCIE_DEVS to find those?

Quoted 2 lines

Thanks,
Niklas

Hi LoongArch Maintainers, Hi Bjorn,

Sorry for the ping but I'd really like to somehow get this unstuck and I haven't heard back on my previous questions. From my testing on s390 this patch fixes a real logic error which prevents the scanning of some devfns which I believe should be scanned if isolated functions are possible. And in all my testing, including on x86 as stated in the previous mail, the code does exactly what I think it is supposed to do. So to me it really looks like something goes wrong with your use of hypervisor_isolated_pci_functions() on your specific hardware.

One idea I had is if maybe you need to somehow exclude known empty slots in you config space accessors?

And just in general I'd really like to better understand your use-case for the isolated PCI functions. And speaking of that, I'm sorry for having been so blunt in my last mail saying that it felt like a hack. I'm just worried, that we've run into incompatible interpretations or uses of this feature that now prevent us from fixing actual bugs.

Thanks in advance,
Niklas Schnelle

maintainer
#9 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 90 lines · Niklas Schnelle
Quoted 10 lines · Niklas Schnelle
Quoted 9 lines · Huacai Chen
Quoted 8 lines · Niklas Schnelle
Quoted 6 lines · Huacai Chen
Quoted 5 lines · Niklas Schnelle
Quoted 3 lines · Huacai Chen

Hi, Niklas,

On Wed, Oct 29, 2025 at 5:42 PM Niklas Schnelle <[email protected]> wrote:

--- snip ---

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

Still especially the first issue prevents correct detection of ARI and the second might be a problem for other users of isolated function probing. Fix both issues by keeping things as simple as possible. If isolated function probing is enabled simply scan every possible devfn.

I'm very sorry, but applying this patch on top of commit a02fd05661d7 ("PCI: Extend isolated function probing to LoongArch") we fail to boot.

Boot log:

--- snip ---

Quoted 51 lines
Quoted 23 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

This looks like a warning telling us that AHCI enable failed / timed out. Do you have Panic on Warn on that this directly causes a boot failure? The only relation I can see with my patch is that maybe this AHCI device wasn't probed before and somehow isn't working?

The rootfs is on the AHCI controller, so AHCI failure causes the boot failure, without this patch no boot problems.

Huacai

Quoted 11 lines

Ok, I'm going to need more details to make sense of this. Can you tell me if ARI is enabled for that bus? Did you test with both patches or just this one? Could you provide lspci -vv from a good boot and can you tell which AHCI device the root device is on? Also could you clarify why you set hypervisor_isolated_pci_functions() in particular this seems like a bare metal boot, right? When running in KVM do you pass- through individual PCI functions with the guest seeing a devfn other than 0 alone, i.e. a missing devfn 0? Or do you need this for bare metal for some reason? If you don't need it for bare metal, does the boot work if you return 0 from hypervisor_isolated_pci_functions() with this patch?

Quoted 7 lines

1. ARI isn't enabled.
2. Only test the first patch.
3. This is a bare metal boot.
4. If hypervisor_isolated_pci_functions() return 0 then boot is OK
5. PCI information please see the attachment.

Huacai

Thanks for the input. As far as I can see the lspci from a good boot shows no holes in your devfn space so this particular system doesn't seem to need the isolated function probing at all. But even then using it should only try out all devfns and thus never skip one that is found without isolated function probing.

To sanity check this, I just booted my personal AMD Ryzen 3900X system with this series plus a two-liner to unconditionally enable isolated function probing also on x86_64 and it came up fine including AMD graphics and my Intel NIC with enabled SR-IOV.

So I'm really perplexed and coming back to the thought that a device on your system is misbehaving when probing is attempted and maybe due to a similar issue as what I saw with SR-IOV it wasn't probed so far but really should be probed if isolated function probing is enabled. I also still don't understand your use-case. If it is for VMs then maybe you could limit it to those? Otherwise it feels like this is just a hack to probe an odd topology and I wonder if you should rather set PCI_SCAN_ALL_PCIE_DEVS to find those?

Quoted 2 lines

Thanks,
Niklas

Hi LoongArch Maintainers, Hi Bjorn,

Sorry for the ping but I'd really like to somehow get this unstuck and I haven't heard back on my previous questions. From my testing on s390 this patch fixes a real logic error which prevents the scanning of some devfns which I believe should be scanned if isolated functions are possible. And in all my testing, including on x86 as stated in the previous mail, the code does exactly what I think it is supposed to do. So to me it really looks like something goes wrong with your use of hypervisor_isolated_pci_functions() on your specific hardware.

Quoted 8 lines

One idea I had is if maybe you need to somehow exclude known empty slots in you config space accessors?

And just in general I'd really like to better understand your use-case for the isolated PCI functions. And speaking of that, I'm sorry for having been so blunt in my last mail saying that it felt like a hack. I'm just worried, that we've run into incompatible interpretations or uses of this feature that now prevent us from fixing actual bugs.

Sorry for the late reply, Let me describe what problem LoongArch has.

You said that "it feels like this is just a hack to probe an odd topology". Yes, to some extent you are right.

1, One of our SoC (LS2K3000) has a special device which has func1 but without func0. To let the PCI core scan func1 we can only make hypervisor_isolated_pci_functions() return true.
2, In the above case, PCI_SCAN_ALL_PCIE_DEVS has no help. 3, Though we change hypervisor_isolated_pci_functions() to resolve the above problem, it also lets us pass isolated PCI functions to a guest OS instance.

As a summary, for real machines commit a02fd05661d73a850 is a hack to probe an odd device, for virtual machines it allows passing isolated PCI functions.

Huacai

Quoted 3 lines

Thanks in advance,
Niklas Schnelle

author maintainer
#10 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 1 line · Huacai Chen

--- snip ---

Quoted 11 lines

You said that "it feels like this is just a hack to probe an odd topology". Yes, to some extent you are right.

1, One of our SoC (LS2K3000) has a special device which has func1 but without func0. To let the PCI core scan func1 we can only make hypervisor_isolated_pci_functions() return true.
2, In the above case, PCI_SCAN_ALL_PCIE_DEVS has no help. 3, Though we change hypervisor_isolated_pci_functions() to resolve the above problem, it also lets us pass isolated PCI functions to a guest OS instance.

Quoted 3 lines

As a summary, for real machines commit a02fd05661d73a850 is a hack to probe an odd device, for virtual machines it allows passing isolated PCI functions.

Ok, thanks for the answer. So let's see how we can debug this and get to a solution that works for both of us. Looking around a bit I see that your pci_loongson_map_bus() has some special handling for trying not to access non-existent devices added by your commit 2410e3301fcc ("PCI: loongson: Don't access non-existent devices"). I wonder if with this patch applied we're running into this same issue but with a devfn that was previously not tried and is not covered by your checks? And maybe since your root complex doesn't return 0xff for these non- existent devices we could end up trying to probe AHCI on such an empty slot misinterpreting whatever it returns as matching device/vendor?

And looking at pdev_may_exist() does the "device <= 20" really make sense with 20 in decimal? If I pull in the negation I get "device > 19" But if it was 0x20 I'd get "device > 0x1f" which would match the maximum value of PCI_SLOT(), though then the check would be redundant since the device value already comes out of PCI_SLOT().

Could you try redoing the test with the AHCI hang but add a print of the affected bus/device/function that AHCI thinks it is probing? Then if the above theory applies we should see it trying to probe on a device that is missing in the correctly booted case and got past your existing checks.

Thanks,
Niklas Schnelle

maintainer
#11 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 19 lines · Niklas Schnelle
Quoted 1 line · Huacai Chen

--- snip ---

Quoted 11 lines

You said that "it feels like this is just a hack to probe an odd topology". Yes, to some extent you are right.

1, One of our SoC (LS2K3000) has a special device which has func1 but without func0. To let the PCI core scan func1 we can only make hypervisor_isolated_pci_functions() return true.
2, In the above case, PCI_SCAN_ALL_PCIE_DEVS has no help. 3, Though we change hypervisor_isolated_pci_functions() to resolve the above problem, it also lets us pass isolated PCI functions to a guest OS instance.

Quoted 3 lines

As a summary, for real machines commit a02fd05661d73a850 is a hack to probe an odd device, for virtual machines it allows passing isolated PCI functions.

Quoted 10 lines

Ok, thanks for the answer. So let's see how we can debug this and get to a solution that works for both of us. Looking around a bit I see that your pci_loongson_map_bus() has some special handling for trying not to access non-existent devices added by your commit 2410e3301fcc ("PCI: loongson: Don't access non-existent devices"). I wonder if with this patch applied we're running into this same issue but with a devfn that was previously not tried and is not covered by your checks? And maybe since your root complex doesn't return 0xff for these non- existent devices we could end up trying to probe AHCI on such an empty slot misinterpreting whatever it returns as matching device/vendor?

Commit 2410e3301fcc seems to have no relationship with current problems.

The LS2K3000 problem that commit a02fd05661d73 want to resolve is we have such a device:

bus=0, device=6,  fun=0,  GPU
bus=0, device=6,  fun=1,  DC

The GPU part can be disabled and in which case there is a device without func0, so need a hack.

Quoted 6 lines

And looking at pdev_may_exist() does the "device <= 20" really make sense with 20 in decimal? If I pull in the negation I get "device > 19" But if it was 0x20 I'd get "device > 0x1f" which would match the maximum value of PCI_SLOT(), though then the check would be redundant since the device value already comes out of PCI_SLOT().

I have confirmed that 20 in decimal is right. Devices between dev-20 and dev-0x20 are completely sane.

Quoted 6 lines

Could you try redoing the test with the AHCI hang but add a print of the affected bus/device/function that AHCI thinks it is probing? Then if the above theory applies we should see it trying to probe on a device that is missing in the correctly booted case and got past your existing checks.

By redoing this test we found there is only one AHCI detected, and the BDF is: bus=0, device=8, fun=0.

With or without this patch, only one AHCI. But without this patch, the AHCI initialization doesn't hang.

Huacai

Quoted 3 lines

Thanks,
Niklas Schnelle

author maintainer
#12 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 30 lines · Huacai Chen
Quoted 19 lines · Niklas Schnelle
Quoted 1 line · Huacai Chen

--- snip ---

Quoted 11 lines

You said that "it feels like this is just a hack to probe an odd topology". Yes, to some extent you are right.

1, One of our SoC (LS2K3000) has a special device which has func1 but without func0. To let the PCI core scan func1 we can only make hypervisor_isolated_pci_functions() return true.
2, In the above case, PCI_SCAN_ALL_PCIE_DEVS has no help. 3, Though we change hypervisor_isolated_pci_functions() to resolve the above problem, it also lets us pass isolated PCI functions to a guest OS instance.

Quoted 3 lines

As a summary, for real machines commit a02fd05661d73a850 is a hack to probe an odd device, for virtual machines it allows passing isolated PCI functions.

Quoted 10 lines

Ok, thanks for the answer. So let's see how we can debug this and get to a solution that works for both of us. Looking around a bit I see that your pci_loongson_map_bus() has some special handling for trying not to access non-existent devices added by your commit 2410e3301fcc ("PCI: loongson: Don't access non-existent devices"). I wonder if with this patch applied we're running into this same issue but with a devfn that was previously not tried and is not covered by your checks? And maybe since your root complex doesn't return 0xff for these non- existent devices we could end up trying to probe AHCI on such an empty slot misinterpreting whatever it returns as matching device/vendor?

Quoted 1 line

Commit 2410e3301fcc seems to have no relationship with current problems.

I'm not so sure. The only thing this patch is potentially supposed to change is which devfns get enumerated and thus config space accessed looking for a device. And that commit talks about accessing non existent devices causing a system hang so that does seem fitting in principle.

Quoted 1 line
Quoted 1 line

--- snip ---

Quoted 11 lines
Quoted 5 lines

Could you try redoing the test with the AHCI hang but add a print of the affected bus/device/function that AHCI thinks it is probing? Then if the above theory applies we should see it trying to probe on a device that is missing in the correctly booted case and got past your existing checks.

By redoing this test we found there is only one AHCI detected, and the BDF is: bus=0, device=8, fun=0.

With or without this patch, only one AHCI. But without this patch, the AHCI initialization doesn't hang.

This is all very odd. Just so there is no chance of misunderstanding. You did check the BDF that the ahci driver is trying to probe directly? I.e. something like what I added as the top commit here:
https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/log/?h=loongarch_debug

This is because with the AHCI controller having a devfn 08.0 devfn and likely dev->multifunction not set this patch would make a difference in that it would try to enumerate 08.1 and so on while without this patch these would be skipped because of the dev && !dev->multifunction condition even though isolated function probing should look at all functions. And I was thinking maybe this causes us to end up trying to probe an AHCI controller where there is none.

Another thing I could imagine, especially with commit 2410e3301fcc ("PCI: loongson: Don't access non-existent devices") in mind is that accessing the device/vendor config space for some non existent devices leaves your PCIe controller in some bad state and then the MMIOs for the AHCI enable go lost or something. Maybe you could add debug code in the relevant parts of drivers/pci/controller/pci-loongson.c to check which devices get accessed with this patch vs without it? Would it help if I provided a debug patch for that? Though I really don't know what part is relevant for the system you're seeing the problem with.

Thanks,
Niklas

maintainer
#13 lore raw

Re: [PATCH v5 1/2] PCI: Fix isolated PCI function probing with ARI and SR-IOV

Quoted 55 lines · Niklas Schnelle
Quoted 30 lines · Huacai Chen
Quoted 19 lines · Niklas Schnelle
Quoted 1 line · Huacai Chen

--- snip ---

Quoted 11 lines

You said that "it feels like this is just a hack to probe an odd topology". Yes, to some extent you are right.

1, One of our SoC (LS2K3000) has a special device which has func1 but without func0. To let the PCI core scan func1 we can only make hypervisor_isolated_pci_functions() return true.
2, In the above case, PCI_SCAN_ALL_PCIE_DEVS has no help. 3, Though we change hypervisor_isolated_pci_functions() to resolve the above problem, it also lets us pass isolated PCI functions to a guest OS instance.

Quoted 3 lines

As a summary, for real machines commit a02fd05661d73a850 is a hack to probe an odd device, for virtual machines it allows passing isolated PCI functions.

Quoted 10 lines

Ok, thanks for the answer. So let's see how we can debug this and get to a solution that works for both of us. Looking around a bit I see that your pci_loongson_map_bus() has some special handling for trying not to access non-existent devices added by your commit 2410e3301fcc ("PCI: loongson: Don't access non-existent devices"). I wonder if with this patch applied we're running into this same issue but with a devfn that was previously not tried and is not covered by your checks? And maybe since your root complex doesn't return 0xff for these non- existent devices we could end up trying to probe AHCI on such an empty slot misinterpreting whatever it returns as matching device/vendor?

Quoted 1 line

Commit 2410e3301fcc seems to have no relationship with current problems.

I'm not so sure. The only thing this patch is potentially supposed to change is which devfns get enumerated and thus config space accessed looking for a device. And that commit talks about accessing non existent devices causing a system hang so that does seem fitting in principle.

Quoted 1 line
Quoted 1 line

--- snip ---

Quoted 11 lines
Quoted 5 lines

Could you try redoing the test with the AHCI hang but add a print of the affected bus/device/function that AHCI thinks it is probing? Then if the above theory applies we should see it trying to probe on a device that is missing in the correctly booted case and got past your existing checks.

By redoing this test we found there is only one AHCI detected, and the BDF is: bus=0, device=8, fun=0.

With or without this patch, only one AHCI. But without this patch, the AHCI initialization doesn't hang.

Quoted 4 lines

This is all very odd. Just so there is no chance of misunderstanding. You did check the BDF that the ahci driver is trying to probe directly? I.e. something like what I added as the top commit here:
https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/log/?h=loongarch_debug

We check the BDF in ahci_init_one().

And with your repo, the boot log is like this:

[   10.454172] ahci 0000:00:08.0: ahci_enable_ahci() hung
[   10.459292] ------------[ cut here ]------------
[   10.463876] WARNING: drivers/ata/libahci.c:459 at
ahci_save_initial_config+0x3d8/0x448, CPU#0: kworker/0:2/253
[   10.473824] Modules linked in:
[   10.476856] CPU: 0 UID: 0 PID: 253 Comm: kworker/0:2 Not tainted
6.19.0-rc1+ #1 PREEMPT(full)
[   10.485416] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.499160] Workqueue: events work_for_cpu_fn
[   10.503489] pc 900000000104e3f8 ra 900000000104e3f8 tp
900000010fb74000 sp 900000010fb77c20
[   10.511788] a0 000000000000002a a1 90000000027223c0 a2
900000010fb77968 a3 90000000027223c8
[   10.520089] a4 90000000027223c0 a5 900000010fb77960 a6
0000000000000001 a7 0000000000000001
[   10.528390] t0 09c296afbf53694b t1 09c296afbf53694b t2
ffffffffffffffff t3 0000000000000001
[   10.536693] t4 fffffffffffffffe t5 0000000000000332 t6
0000000000000005 t7 00000000000011bd
[   10.544993] t8 0000000000000000 u0 9000000000232db0 s9
900000000226a4a8 s0 9000000100e7ea30
[   10.553293] s1 ffff80003243d800 s2 90000001012af0b8 s3
900000010fb77cc8 s4 9000000002269fb8
[   10.561592] s5 0000000000000005 s6 9000000002429000 s7
90000001012af0b8 s8 900000010fb77d08
[   10.569890]    ra: 900000000104e3f8 ahci_save_initial_config+0x3d8/0x448
[   10.576551]   ERA: 900000000104e3f8 ahci_save_initial_config+0x3d8/0x448
[   10.583212]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   10.589359]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   10.593687]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   10.598447]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   10.603207] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   10.608744]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   10.614625] CPU: 0 UID: 0 PID: 253 Comm: kworker/0:2 Not tainted
6.19.0-rc1+ #1 PREEMPT(full)
[   10.614628] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.614630] Workqueue: events work_for_cpu_fn
[   10.614633] Stack : 900000010fb77858 0000000000000000
900000000024371c 900000010fb74000
[   10.614637]         900000010fb77830 900000010fb77838
0000000000000000 900000010fb77978
[   10.614639]         900000010fb77970 900000010fb77970
900000010fb77650 0000000000000001
[   10.614642]         0000000000000001 900000010fb77838
09c296afbf53694b 9000000100414300
[   10.614645]         80000000ffffe349 fffffffffffffffe
000000000000034b 000000000000002f
[   10.614648]         0000000000000063 0000000000000001
000000000672c000 900000000226a4a8
[   10.614651]         0000000000000000 0000000000000000
90000000020fd980 9000000002429000
[   10.614654]         00000000000001cb 0000000000000009
0000000000000002 90000001012af0b8
[   10.614657]         900000010fb77d08 0000000000000000
9000000000243734 0000000000000000
[   10.614659]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   10.614662]         ...
[   10.614664] Call Trace:
[   10.614665] [<9000000000243734>] show_stack+0x5c/0x180
[   10.614669] [<900000000023f3c4>] dump_stack_lvl+0x6c/0x9c
[   10.614674] [<9000000000267168>] __warn+0x90/0x128
[   10.614679] [<90000000017e983c>] __report_bug+0x84/0x198
[   10.614683] [<90000000017e9a4c>] report_bug+0x3c/0xc0
[   10.614686] [<9000000001823910>] do_bp+0x2d0/0x340
[   10.614690] [<9000000000241e00>] handle_bp+0x120/0x1c0
[   10.614692] [<900000000104e3f8>] ahci_save_initial_config+0x3d8/0x448
[   10.614695] [<900000000104b374>] ahci_init_one+0x354/0x1068
[   10.614699] [<9000000000d2343c>] local_pci_probe+0x44/0xb8
[   10.614703] [<9000000000286bf8>] work_for_cpu_fn+0x18/0x30
[   10.614706] [<900000000028a4e0>] process_one_work+0x160/0x330
[   10.614709] [<900000000028aee8>] worker_thread+0x338/0x468
[   10.614712] [<9000000000295ef4>] kthread+0x11c/0x138
[   10.614716] [<9000000001823a58>] ret_from_kernel_thread+0x28/0xd0
[   10.614717] [<90000000002414e4>] ret_from_kernel_thread_asm+0xc/0x88
[   10.614720]
[   10.614720] ---[ end trace 0000000000000000 ]---
[   10.818394] ahci 0000:00:08.0: forcing PORTS_IMPL to 0x1
[   10.902172] ahci 0000:00:08.0: ahci_enable_ahci() hung
[   10.907282] ------------[ cut here ]------------
[   10.911866] WARNING: drivers/ata/libahci.c:994 at
ahci_reset_controller+0x88/0x1d8, CPU#0: kworker/0:2/253
[   10.921466] Modules linked in:
[   10.924496] CPU: 0 UID: 0 PID: 253 Comm: kworker/0:2 Tainted: G
   W           6.19.0-rc1+ #1 PREEMPT(full)
[   10.934611] Tainted: [W]=WARN
[   10.937551] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   10.951294] Workqueue: events work_for_cpu_fn
[   10.955622] pc 900000000104e4f0 ra 900000000104e4f0 tp
900000010fb74000 sp 900000010fb77c10
[   10.963922] a0 000000000000002a a1 90000000027223c0 a2
900000010fb77958 a3 90000000027223c8
[   10.972221] a4 90000000027223c0 a5 900000010fb77950 a6
0000000000000001 a7 0000000000000001
[   10.980519] t0 09c296afbf53694b t1 09c296afbf53694b t2
ffffffffffffffff t3 0000000000000001
[   10.988818] t4 fffffffffffffffe t5 000000000000036b t6
0000000000000005 t7 000000000000233d
[   10.997117] t8 0000000000000000 u0 9000000000232db0 s9
90000001001f0000 s0 900000010000dc80
[   11.005416] s1 ffff80003243d800 s2 9000000100e7ea30 s3
9000000100e7ea30 s4 9000000002269fb8
[   11.013715] s5 90000001012af0b8 s6 9000000002429000 s7
90000001012af0b8 s8 0000000000000001
[   11.022014]    ra: 900000000104e4f0 ahci_reset_controller+0x88/0x1d8
[   11.028328]   ERA: 900000000104e4f0 ahci_reset_controller+0x88/0x1d8
[   11.034641]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   11.040787]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   11.045116]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   11.049877]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   11.054636] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   11.060173]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   11.066053] CPU: 0 UID: 0 PID: 253 Comm: kworker/0:2 Tainted: G
   W           6.19.0-rc1+ #1 PREEMPT(full)
[   11.066055] Tainted: [W]=WARN
[   11.066056] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   11.066057] Workqueue: events work_for_cpu_fn
[   11.066060] Stack : 900000010fb77848 0000000000000000
900000000024371c 900000010fb74000
[   11.066063]         900000010fb77820 900000010fb77828
0000000000000000 900000010fb77968
[   11.066066]         900000010fb77960 900000010fb77960
900000010fb77640 0000000000000001
[   11.066068]         0000000000000001 900000010fb77828
09c296afbf53694b 9000000100414300
[   11.066071]         80000000ffffe384 fffffffffffffffe
0000000000000386 000000000000002f
[   11.066074]         0000000000000063 0000000000000001
000000000672c000 90000001001f0000
[   11.066077]         0000000000000000 0000000000000000
90000000020fd980 9000000002429000
[   11.066079]         00000000000003e2 0000000000000009
0000000000000002 90000001012af0b8
[   11.066082]         0000000000000001 0000000000000000
9000000000243734 0000000000000000
[   11.066085]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   11.066088]         ...
[   11.066089] Call Trace:
[   11.066089] [<9000000000243734>] show_stack+0x5c/0x180
[   11.066091] [<900000000023f3c4>] dump_stack_lvl+0x6c/0x9c
[   11.066095] [<9000000000267168>] __warn+0x90/0x128
[   11.066098] [<90000000017e983c>] __report_bug+0x84/0x198
[   11.066102] [<90000000017e9a4c>] report_bug+0x3c/0xc0
[   11.066105] [<9000000001823910>] do_bp+0x2d0/0x340
[   11.066106] [<9000000000241e00>] handle_bp+0x120/0x1c0
[   11.066108] [<900000000104e4f0>] ahci_reset_controller+0x88/0x1d8
[   11.066111] [<900000000104a58c>] ahci_pci_reset_controller+0x2c/0xd8
[   11.066114] [<900000000104bc10>] ahci_init_one+0xbf0/0x1068
[   11.066116] [<9000000000d2343c>] local_pci_probe+0x44/0xb8
[   11.066119] [<9000000000286bf8>] work_for_cpu_fn+0x18/0x30
[   11.066122] [<900000000028a4e0>] process_one_work+0x160/0x330
[   11.066125] [<900000000028aee8>] worker_thread+0x338/0x468
[   11.066128] [<9000000000295ef4>] kthread+0x11c/0x138
[   11.066131] [<9000000001823a58>] ret_from_kernel_thread+0x28/0xd0
[   11.066132] [<90000000002414e4>] ret_from_kernel_thread_asm+0xc/0x88
[   11.066134]
[   11.066135] ---[ end trace 0000000000000000 ]---
[   11.358172] ahci 0000:00:08.0: ahci_enable_ahci() hung
[   11.363283] ------------[ cut here ]------------
[   11.367867] WARNING: drivers/ata/libahci.c:1028 at
ahci_reset_controller+0x1cc/0x1d8, CPU#0: kworker/0:2/253
[   11.377638] Modules linked in:
[   11.380668] CPU: 0 UID: 0 PID: 253 Comm: kworker/0:2 Tainted: G
   W           6.19.0-rc1+ #1 PREEMPT(full)
[   11.390783] Tainted: [W]=WARN
[   11.393723] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   11.407466] Workqueue: events work_for_cpu_fn
[   11.411792] pc 900000000104e634 ra 900000000104e634 tp
900000010fb74000 sp 900000010fb77c10
[   11.420092] a0 000000000000002a a1 90000000027223c0 a2
900000010fb77958 a3 90000000027223c8
[   11.428391] a4 90000000027223c0 a5 900000010fb77950 a6
0000000000000001 a7 0000000000000001
[   11.436691] t0 09c296afbf53694b t1 09c296afbf53694b t2
ffffffffffffffff t3 0000000000000001
[   11.444990] t4 fffffffffffffffe t5 00000000000003a6 t6
0000000000000005 t7 0000000000000dfd
[   11.453288] t8 0000000000000000 u0 9000000000232db0 s9
90000001001f0000 s0 900000010000dc80
[   11.461587] s1 ffff80003243d800 s2 9000000100e7ea30 s3
9000000100e7ea30 s4 9000000002269fb8
[   11.469885] s5 90000001012af0b8 s6 9000000002429000 s7
90000001012af0b8 s8 0000000000000001
[   11.478185]    ra: 900000000104e634 ahci_reset_controller+0x1cc/0x1d8
[   11.484586]   ERA: 900000000104e634 ahci_reset_controller+0x1cc/0x1d8
[   11.490987]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[   11.497131]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[   11.501458]  EUEN: 00000000 (-FPE -SXE -ASXE -BTE)
[   11.506218]  ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
[   11.510977] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[   11.516512]  PRID: 0014d010 (Loongson-64bit, Loongson-3C6000/S)
[   11.522393] CPU: 0 UID: 0 PID: 253 Comm: kworker/0:2 Tainted: G
   W           6.19.0-rc1+ #1 PREEMPT(full)
[   11.522395] Tainted: [W]=WARN
[   11.522396] Hardware name: To be filled by O.E.M.To be fill To be
filled by O.E.M.To be fill/To be filled by O.E.M.To be fill, BIOS
Loongson-UDK2018-V4.0.
[   11.522397] Workqueue: events work_for_cpu_fn
[   11.522400] Stack : 900000010fb77848 0000000000000000
900000000024371c 900000010fb74000
[   11.522403]         900000010fb77820 900000010fb77828
0000000000000000 900000010fb77968
[   11.522406]         900000010fb77960 900000010fb77960
900000010fb77640 0000000000000001
[   11.522409]         0000000000000001 900000010fb77828
09c296afbf53694b 9000000100414300
[   11.522412]         80000000ffffe3bf fffffffffffffffe
00000000000003c1 000000000000002f
[   11.522415]         0000000000000063 0000000000000001
000000000672c000 90000001001f0000
[   11.522418]         0000000000000000 0000000000000000
90000000020fd980 9000000002429000
[   11.522420]         0000000000000404 0000000000000009
0000000000000002 90000001012af0b8
[   11.522423]         0000000000000001 0000000000000000
9000000000243734 0000000000000000
[   11.522426]         00000000000000b0 0000000000000004
0000000000000000 0000000000071c1d
[   11.522428]         ...
[   11.522430] Call Trace:
[   11.522430] [<9000000000243734>] show_stack+0x5c/0x180
[   11.522432] [<900000000023f3c4>] dump_stack_lvl+0x6c/0x9c
[   11.522436] [<9000000000267168>] __warn+0x90/0x128
[   11.522439] [<90000000017e983c>] __report_bug+0x84/0x198
[   11.522442] [<90000000017e9a4c>] report_bug+0x3c/0xc0
[   11.522445] [<9000000001823910>] do_bp+0x2d0/0x340
[   11.522447] [<9000000000241e00>] handle_bp+0x120/0x1c0
[   11.522448] [<900000000104e634>] ahci_reset_controller+0x1cc/0x1d8
[   11.522451] [<900000000104a58c>] ahci_pci_reset_controller+0x2c/0xd8
[   11.522454] [<900000000104bc10>] ahci_init_one+0xbf0/0x1068
[   11.522456] [<9000000000d2343c>] local_pci_probe+0x44/0xb8
[   11.522459] [<9000000000286bf8>] work_for_cpu_fn+0x18/0x30
[   11.522462] [<900000000028a4e0>] process_one_work+0x160/0x330
[   11.522465] [<900000000028aee8>] worker_thread+0x338/0x468
[   11.522467] [<9000000000295ef4>] kthread+0x11c/0x138
[   11.522470] [<9000000001823a58>] ret_from_kernel_thread+0x28/0xd0
[   11.522471] [<90000000002414e4>] ret_from_kernel_thread_asm+0xc/0x88
[   11.522473]
[   11.522474] ---[ end trace 0000000000000000 ]---
[   11.736718] ahci 0000:00:08.0: AHCI vers 0000.0000, 1 command
slots, ? Gbps, unknown mode
[   11.744847] ahci 0000:00:08.0: 1/1 ports implemented (port mask 0x1)
[   11.751159] ahci 0000:00:08.0: flags:
[   11.755115] scsi host0: ahci
[   11.758048] ata1: SATA max UDMA/133 abar m1024@0xe0030401800 port
0xe0030401900 irq 44 lpm-pol 1
[   11.767294] e1000: Intel(R) PRO/1000 Network Driver
[   11.772145] e1000: Copyright (c) 1999-2006 Intel Corporation.
[   11.777870] e1000e: Intel(R) PRO/1000 Network Driver
[   11.782800] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.

Huacai

Quoted 21 lines

This is because with the AHCI controller having a devfn 08.0 devfn and likely dev->multifunction not set this patch would make a difference in that it would try to enumerate 08.1 and so on while without this patch these would be skipped because of the dev && !dev->multifunction condition even though isolated function probing should look at all functions. And I was thinking maybe this causes us to end up trying to probe an AHCI controller where there is none.

Another thing I could imagine, especially with commit 2410e3301fcc ("PCI: loongson: Don't access non-existent devices") in mind is that accessing the device/vendor config space for some non existent devices leaves your PCIe controller in some bad state and then the MMIOs for the AHCI enable go lost or something. Maybe you could add debug code in the relevant parts of drivers/pci/controller/pci-loongson.c to check which devices get accessed with this patch vs without it? Would it help if I provided a debug patch for that? Though I really don't know what part is relevant for the system you're seeing the problem with.

Thanks,
Niklas

author maintainer
#14 lore raw

PATCH v5 2/2 PCI: s390: Handle ARI on bus without associated struct pci_dev

In reply to Niklas Schnelle

On s390 PCI busses are virtualized and the downstream ports are invisible to the OS and self in struct pci_bus is NULL. This associated struct pci_dev is however relied upon in pci_ari_enabled() to check whether ARI is enabled for the bus. ARI is therefor always detected as disabled. At the same time, firmware on s390 always enables and relies upon ARI thus causing a mismatch.

Despite simply being a mismatch this causes problems as some PCI devices present a different SR-IOV topology depending on PCI_SRIOV_CTRL_ARI.

A similar mismatch may occur with SR-IOV when virtfn_add_bus() creates new busses with no associated struct pci_dev. Here too pci_ari_enabled() on these busses returns false even if ARI is actually used.

Prevent both mismatches by moving the ari_enabled flag from struct pci_dev to struct pci_bus making it independent from self in struct pci_bus. Let the bus inherit the ari_enabled state from its parent bus when there is no bridge device such that busses added by virtfn_add_bus() match their parent. For s390 set ari_enabled when the device supports ARI in the awareness that all PCIe ports on s390 systems are ARI capable.

---
 arch/s390/pci/pci.c     |  7 +++++++
 arch/s390/pci/pci_bus.c | 10 ++++++++++
 drivers/pci/pci.c       |  4 ++--
 drivers/pci/probe.c     |  1 +
 include/linux/pci.h     |  4 ++--
 5 files changed, 22 insertions(+), 4 deletions(-)
arch/s390/pci/pci.c +7 -0
index c82c577db2bcd2143476cb8189fd89b9a4dc9836..773c0cbfc313ea1a6419a44d6158397dd13f6e76 100644
@@ -609,6 +609,13 @@ int pcibios_device_add(struct pci_dev *pdev)
 			continue;
 		pci_claim_resource(pdev, i);
 	}
+	/*
+	 * The below is the s390 equivalent of pci_configure_ari()
+	 * which we can't use directly because the bridge devices
+	 * are hidden in firmware.
+	 */
+	if (!pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_ARI))
+		zdev->zbus->bus->ari_enabled = 0;
 
 	return 0;
 }
arch/s390/pci/pci_bus.c +10 -0
index 45a1c36c5a54e3a841e61cc365d3f36e9a94ba50..c887e61eb384ca98ff27d4f8af69e58c715b5002 100644
@@ -207,6 +207,16 @@ static int zpci_bus_create_pci_bus(struct zpci_bus *zbus, struct zpci_dev *fr, s
 		return -EFAULT;
 	}
 
+	/*
+	 * On s390 PCI busses are virtualized and the bridge
+	 * devices are invisible to the OS. Furthermore busses
+	 * may exist without a devfn 0 function. Thus the normal
+	 * ARI detection does not work. At the same time fw/hw
+	 * has always enabled ARI when possible. Reflect the actual
+	 * state by setting ari_enabled whenever a device on the bus
+	 * supports it.
+	 */
+	bus->ari_enabled = 1;
 	zbus->bus = bus;
 
 	return 0;
drivers/pci/pci.c +2 -2
index b14dd064006cca80ec5275e45a35d6dc2b4d0bbc..8ef3c68280a629449e0a2176d938bf987c68dddf 100644
@@ -3532,11 +3532,11 @@ void pci_configure_ari(struct pci_dev *dev)
 	if (pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ARI)) {
 		pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
 					 PCI_EXP_DEVCTL2_ARI);
-		bridge->ari_enabled = 1;
+		dev->bus->ari_enabled = 1;
 	} else {
 		pcie_capability_clear_word(bridge, PCI_EXP_DEVCTL2,
 					   PCI_EXP_DEVCTL2_ARI);
-		bridge->ari_enabled = 0;
+		dev->bus->ari_enabled = 0;
 	}
 }
 
drivers/pci/probe.c +1 -0
index 41dd1a339a994956a6bc7e1fea0fe0d55452a963..6fc03328a62f54f33f47309f8300516e8f52b5eb 100644
@@ -1216,6 +1216,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 
 	if (!bridge) {
 		child->dev.parent = parent->bridge;
+		child->ari_enabled = parent->ari_enabled;
 		goto add_dev;
 	}
 
include/linux/pci.h +2 -2
index d1fdf81fbe1e427aecbc951fa3fdf65c20450b05..a9c3dbf17339e523362bd179ad3c7c8c91293cf0 100644
@@ -445,7 +445,6 @@ struct pci_dev {
 	unsigned int	irq_reroute_variant:2;	/* Needs IRQ rerouting variant */
 	unsigned int	msi_enabled:1;
 	unsigned int	msix_enabled:1;
-	unsigned int	ari_enabled:1;		/* ARI forwarding */
 	unsigned int	ats_enabled:1;		/* Address Translation Svc */
 	unsigned int	pasid_enabled:1;	/* Process Address Space ID */
 	unsigned int	pri_enabled:1;		/* Page Request Interface */
@@ -691,6 +690,7 @@ struct pci_bus {
 	unsigned int		is_added:1;
 	unsigned int		unsafe_warn:1;	/* warned about RW1C config write */
 	unsigned int		flit_mode:1;	/* Link in Flit mode */
+	unsigned int		ari_enabled:1;	/* ARI forwarding enabled */
 };
 
 #define to_pci_bus(n)	container_of(n, struct pci_bus, dev)
@@ -2740,7 +2740,7 @@ static inline bool pci_is_dev_assigned(struct pci_dev *pdev)
  */
 static inline bool pci_ari_enabled(struct pci_bus *bus)
 {
-	return bus->self && bus->self->ari_enabled;
+	return bus->ari_enabled;
 }
 
 /**
Signature · 1 line
2.48.1