Config space access to Mediatek MT7922 doesn't work after device reset in Xen PV dom0 (regression, Linux 6.12)

After updating PV dom0 to Linux 6.12, The Mediatek MT7922 device reports all 0xff when accessing its config space. This happens only after device reset (which is also triggered when binding the device to the xen-pciback driver).

Reproducer:

    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: c3 14 16 06 00 00 10 00 00 00 80 02 10 00 00 00
    ...
    # echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

The same operation done on Linux 6.12 running without Xen works fine.

git bisect points at:

commit d591f6804e7e1310881c9224d72247a2b65039af
Author: Bjorn Helgaas <[email protected]>
Date: Tue Aug 27 18:48:46 2024 -0500

PCI: Wait for device readiness with Configuration RRS

part of that commit:

(quoted diff) +9 -3

@@ -1311,9 +1320,15 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
                        return -ENOTTY;
                }
 
-               pci_read_config_dword(dev, PCI_COMMAND, &id);
-               if (!PCI_POSSIBLE_ERROR(id))
-                       break;
+               if (root && root->config_crs_sv) {
+                       pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+                       if (!pci_bus_crs_vendor_id(id))
+                               break;
+               } else {
+                       pci_read_config_dword(dev, PCI_COMMAND, &id);
+                       if (!PCI_POSSIBLE_ERROR(id))
+                               break;
+               }
 
 

Adding some debugging, the PCI_VENDOR_ID read in pci_dev_wait() returns initially 0xffffffff. If I extend the condition with
"&& !PCI_POSSIBLE_ERROR(id)", then the issue disappear. But reading the patch description, it would break VF.
I'm not sure where the issue is, but given it breaks only when running with Xen, I guess something is wrong with "Configuration RRS Software Visibility" in that case.

BTW, shouldn't PCI_VENDOR_ID be accessed via pci_read_config_word() instead of pci_read_config_dword()?

Quoted 4 lines · Marek Marczykowski-Górecki

After updating PV dom0 to Linux 6.12, The Mediatek MT7922 device reports all 0xff when accessing its config space. This happens only after device reset (which is also triggered when binding the device to the xen-pciback driver).

Thanks for the report and for all the debugging you've already done!

Quoted 40 lines

Reproducer:

    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: c3 14 16 06 00 00 10 00 00 00 80 02 10 00 00 00
    ...
    # echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

The same operation done on Linux 6.12 running without Xen works fine.

git bisect points at:

commit d591f6804e7e1310881c9224d72247a2b65039af
Author: Bjorn Helgaas <[email protected]>
Date: Tue Aug 27 18:48:46 2024 -0500

PCI: Wait for device readiness with Configuration RRS

part of that commit:

(quoted diff) +9 -3

@@ -1311,9 +1320,15 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
                        return -ENOTTY;
                }
 
-               pci_read_config_dword(dev, PCI_COMMAND, &id);
-               if (!PCI_POSSIBLE_ERROR(id))
-                       break;
+               if (root && root->config_crs_sv) {
+                       pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+                       if (!pci_bus_crs_vendor_id(id))
+                               break;
+               } else {
+                       pci_read_config_dword(dev, PCI_COMMAND, &id);
+                       if (!PCI_POSSIBLE_ERROR(id))
+                               break;
+               }
 
 

Quoted 7 lines

Adding some debugging, the PCI_VENDOR_ID read in pci_dev_wait() returns initially 0xffffffff. If I extend the condition with
"&& !PCI_POSSIBLE_ERROR(id)", then the issue disappear. But reading the patch description, it would break VF.
I'm not sure where the issue is, but given it breaks only when running with Xen, I guess something is wrong with "Configuration RRS Software Visibility" in that case.

I'm missing something. If you get 0xffffffff, that is not the 0x0001 Vendor ID, so pci_dev_wait() should exit immediately.

But the log at
https://github.com/QubesOS/qubes-issues/issues/9689#issuecomment-2582927149
says it *doesn't* exit and eventually times out.

And the lspci above shows ~0 data for much of the header, even though the device must be ready by then.

I don't have any good ideas, but since the problem only happens with Xen, and it seems to affect more than just the Vendor ID, maybe you could instrument xen_pcibk_config_read() and see if there's something wonky going on there?

Quoted 48 lines · Bjorn Helgaas

Quoted 4 lines · Marek Marczykowski-Górecki

After updating PV dom0 to Linux 6.12, The Mediatek MT7922 device reports all 0xff when accessing its config space. This happens only after device reset (which is also triggered when binding the device to the xen-pciback driver).

Thanks for the report and for all the debugging you've already done!

Quoted 40 lines

Reproducer:

    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: c3 14 16 06 00 00 10 00 00 00 80 02 10 00 00 00
    ...
    # echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

The same operation done on Linux 6.12 running without Xen works fine.

git bisect points at:

commit d591f6804e7e1310881c9224d72247a2b65039af
Author: Bjorn Helgaas <[email protected]>
Date: Tue Aug 27 18:48:46 2024 -0500

PCI: Wait for device readiness with Configuration RRS

part of that commit:

(quoted diff) +9 -3

@@ -1311,9 +1320,15 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
                        return -ENOTTY;
                }
 
-               pci_read_config_dword(dev, PCI_COMMAND, &id);
-               if (!PCI_POSSIBLE_ERROR(id))
-                       break;
+               if (root && root->config_crs_sv) {
+                       pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+                       if (!pci_bus_crs_vendor_id(id))
+                               break;
+               } else {
+                       pci_read_config_dword(dev, PCI_COMMAND, &id);
+                       if (!PCI_POSSIBLE_ERROR(id))
+                               break;
+               }
 
 

Quoted 10 lines

Quoted 7 lines

Adding some debugging, the PCI_VENDOR_ID read in pci_dev_wait() returns initially 0xffffffff. If I extend the condition with
"&& !PCI_POSSIBLE_ERROR(id)", then the issue disappear. But reading the patch description, it would break VF.
I'm not sure where the issue is, but given it breaks only when running with Xen, I guess something is wrong with "Configuration RRS Software Visibility" in that case.

I'm missing something. If you get 0xffffffff, that is not the 0x0001 Vendor ID, so pci_dev_wait() should exit immediately.

I'm not sure what is going on there either, but my _guess_ is that the loop exits too early due to the above. And it makes some further actions to fail.

Quoted 3 lines

But the log at
https://github.com/QubesOS/qubes-issues/issues/9689#issuecomment-2582927149
says it *doesn't* exit and eventually times out.

Note this log is from "working" kernel, so that timeout must be something else.

Quoted 7 lines

And the lspci above shows ~0 data for much of the header, even though the device must be ready by then.

I don't have any good ideas, but since the problem only happens with Xen, and it seems to affect more than just the Vendor ID, maybe you could instrument xen_pcibk_config_read() and see if there's something wonky going on there?

This one is used when pcifront (from a different PV VM) is asking pciback to read something. I see the issue even before starting any other VM and not even attaching the device to the xen-pciback driver...

[+cc linux-pci]

Quoted 60 lines · Marek Marczykowski-Górecki

Quoted 58 lines · Bjorn Helgaas

Quoted 4 lines · Marek Marczykowski-Górecki

After updating PV dom0 to Linux 6.12, The Mediatek MT7922 device reports all 0xff when accessing its config space. This happens only after device reset (which is also triggered when binding the device to the xen-pciback driver).

Thanks for the report and for all the debugging you've already done!

Quoted 40 lines

Reproducer:

    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: c3 14 16 06 00 00 10 00 00 00 80 02 10 00 00 00
    ...
    # echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

The same operation done on Linux 6.12 running without Xen works fine.

git bisect points at:

commit d591f6804e7e1310881c9224d72247a2b65039af
Author: Bjorn Helgaas <[email protected]>
Date: Tue Aug 27 18:48:46 2024 -0500

PCI: Wait for device readiness with Configuration RRS

part of that commit:

(quoted diff) +9 -3

@@ -1311,9 +1320,15 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
                        return -ENOTTY;
                }
 
-               pci_read_config_dword(dev, PCI_COMMAND, &id);
-               if (!PCI_POSSIBLE_ERROR(id))
-                       break;
+               if (root && root->config_crs_sv) {
+                       pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+                       if (!pci_bus_crs_vendor_id(id))
+                               break;
+               } else {
+                       pci_read_config_dword(dev, PCI_COMMAND, &id);
+                       if (!PCI_POSSIBLE_ERROR(id))
+                               break;
+               }
 
 

Quoted 7 lines

Adding some debugging, the PCI_VENDOR_ID read in pci_dev_wait() returns initially 0xffffffff. If I extend the condition with
"&& !PCI_POSSIBLE_ERROR(id)", then the issue disappear. But reading the patch description, it would break VF.
I'm not sure where the issue is, but given it breaks only when running with Xen, I guess something is wrong with "Configuration RRS Software Visibility" in that case.

I'm missing something. If you get 0xffffffff, that is not the 0x0001 Vendor ID, so pci_dev_wait() should exit immediately.

Quoted 3 lines

I'm not sure what is going on there either, but my _guess_ is that the loop exits too early due to the above. And it makes some further actions to fail.

When RRS SV is enabled, reading PCI_VENDOR_ID should always return 0x0001 (if the device isn't ready and responds with RRS status) or the valid Vendor ID. I don't think it should ever return 0xffff (unless the device is powered off, unplugged, or broken, of course).

Quoted 7 lines

Quoted 4 lines

But the log at
https://github.com/QubesOS/qubes-issues/issues/9689#issuecomment-2582927149
says it *doesn't* exit and eventually times out.

Note this log is from "working" kernel, so that timeout must be something else.

I saw it was labeled "NO BUG" but I'm not sure it's labeled correctly since there are no interesting messages from the "BUG PRESENT" part. Awfully funny coincidence if it's unrelated.

Quoted 11 lines

Quoted 7 lines

And the lspci above shows ~0 data for much of the header, even though the device must be ready by then.

I don't have any good ideas, but since the problem only happens with Xen, and it seems to affect more than just the Vendor ID, maybe you could instrument xen_pcibk_config_read() and see if there's something wonky going on there?

This one is used when pcifront (from a different PV VM) is asking pciback to read something. I see the issue even before starting any other VM and not even attaching the device to the xen-pciback driver...

The report claims the problem only happens with Xen. I'm not a Xen person, and I don't know how to find the relevant config accessors. The snippets of kernel messages I see at [1] all mention pciback, so that's my only clue of where to look. Bottom line, I have no idea what the config accessor path is, and maybe we could learn something by looking at whatever it is.

[1] https://github.com/QubesOS/qubes-issues/issues/9689#issuecomment-2582927149

Quoted 65 lines · Bjorn Helgaas

Quoted 63 lines · Marek Marczykowski-Górecki

Quoted 48 lines · Bjorn Helgaas

Quoted 4 lines · Marek Marczykowski-Górecki

After updating PV dom0 to Linux 6.12, The Mediatek MT7922 device reports all 0xff when accessing its config space. This happens only after device reset (which is also triggered when binding the device to the xen-pciback driver).

Thanks for the report and for all the debugging you've already done!

Quoted 40 lines

Reproducer:

    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: c3 14 16 06 00 00 10 00 00 00 80 02 10 00 00 00
    ...
    # echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

The same operation done on Linux 6.12 running without Xen works fine.

git bisect points at:

commit d591f6804e7e1310881c9224d72247a2b65039af
Author: Bjorn Helgaas <[email protected]>
Date: Tue Aug 27 18:48:46 2024 -0500

PCI: Wait for device readiness with Configuration RRS

part of that commit:

(quoted diff) +9 -3

@@ -1311,9 +1320,15 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
                        return -ENOTTY;
                }
 
-               pci_read_config_dword(dev, PCI_COMMAND, &id);
-               if (!PCI_POSSIBLE_ERROR(id))
-                       break;
+               if (root && root->config_crs_sv) {
+                       pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+                       if (!pci_bus_crs_vendor_id(id))
+                               break;
+               } else {
+                       pci_read_config_dword(dev, PCI_COMMAND, &id);
+                       if (!PCI_POSSIBLE_ERROR(id))
+                               break;
+               }
 
 

Quoted 10 lines

Quoted 7 lines

Adding some debugging, the PCI_VENDOR_ID read in pci_dev_wait() returns initially 0xffffffff. If I extend the condition with
"&& !PCI_POSSIBLE_ERROR(id)", then the issue disappear. But reading the patch description, it would break VF.
I'm not sure where the issue is, but given it breaks only when running with Xen, I guess something is wrong with "Configuration RRS Software Visibility" in that case.

I'm missing something. If you get 0xffffffff, that is not the 0x0001 Vendor ID, so pci_dev_wait() should exit immediately.

I'm not sure what is going on there either, but my _guess_ is that the loop exits too early due to the above. And it makes some further actions to fail.

Quoted 4 lines

When RRS SV is enabled, reading PCI_VENDOR_ID should always return 0x0001 (if the device isn't ready and responds with RRS status) or the valid Vendor ID. I don't think it should ever return 0xffff (unless the device is powered off, unplugged, or broken, of course).

Maybe it isn't really enabled when Xen is involved?
By looking at lspci of the bridge for this device, I do see RootCtl: ... CRSVisible+, but maybe there is something else needed too?

Just in case, full lspci -vvvs 2.2 (the bridge):

00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Phoenix GPP Bridge (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 102
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000f000-00000fff [disabled] [32-bit]
Memory behind bridge: 90b00000-90bfffff [size=1M] [32-bit]
Prefetchable memory behind bridge: 8010900000-80109fffff [size=1M] [32-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Express (v2) Root Port (Slot+), IntMsgNum 0
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag+ RBE+ TEE-IO-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #4, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
ExtSynch+ ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1
TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 75W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState+
RootCap: CRSVisible+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- LN System CLS Not Supported, TPHComp+ ExtTPHComp- ARIFwd+
AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS-
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis- ARIFwd-
AtomicOpsCtl: ReqEn- EgressBlck-
IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee08000 Data: 4000
Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [270 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
Capabilities: [370 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=10us PortTPowerOnTime=150us
L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
T_CommonMode=10us LTR1.2_Threshold=166912ns
L1SubCtl2: T_PwrOn=150us
Capabilities: [400 v1] Data Link Feature <?>
Capabilities: [410 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [440 v1] Lane Margining at the Receiver
PortCap: Uses Driver-
PortSta: MargReady- MargSoftReady-
Kernel driver in use: pcieport

Quoted 10 lines

Quoted 6 lines

Quoted 3 lines

But the log at
https://github.com/QubesOS/qubes-issues/issues/9689#issuecomment-2582927149
says it *doesn't* exit and eventually times out.

Note this log is from "working" kernel, so that timeout must be something else.

I saw it was labeled "NO BUG" but I'm not sure it's labeled correctly since there are no interesting messages from the "BUG PRESENT" part. Awfully funny coincidence if it's unrelated.

The timeout thing I have seen before, possibly also on a different hardware (although I'm not 100% sure), I think it's a different issue.

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

And the lspci above shows ~0 data for much of the header, even though the device must be ready by then.

I don't have any good ideas, but since the problem only happens with Xen, and it seems to affect more than just the Vendor ID, maybe you could instrument xen_pcibk_config_read() and see if there's something wonky going on there?

This one is used when pcifront (from a different PV VM) is asking pciback to read something. I see the issue even before starting any other VM and not even attaching the device to the xen-pciback driver...

Quoted 6 lines

The report claims the problem only happens with Xen. I'm not a Xen person, and I don't know how to find the relevant config accessors. The snippets of kernel messages I see at [1] all mention pciback, so that's my only clue of where to look. Bottom line, I have no idea what the config accessor path is, and maybe we could learn something by looking at whatever it is.

AFAIK there are no separate config accessors under Xen dom0, the default ones are used. xen-pcifront takes over PCI config space access (and few more) only in a domU (and only for PV), when PCI passthrough is used. Here, it didn't went that far...

But then, Xen may intercept such access [2]. If I read it right, it should allow all access (is_hardware_domain(dom0)==true, and also the device is not on ro_map - otherwise reset wouldn't work at all).

Quoted 1 line

[1] https://github.com/QubesOS/qubes-issues/issues/9689#issuecomment-2582927149

[2] https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/pv/emul-priv-op.c;h=70150c27227661baa253af8693ff00f2ab640a98;hb=HEAD#l295

I guess the code at [2] is running in user mode and uses Linux syscalls for config access? Is it straceable?

Can you reproduce this without Xen at all? If so, can you post a complete dmesg and complete lspci -vv somewhere?

Quoted 3 lines

Quoted 1 line

[1] https://github.com/QubesOS/qubes-issues/issues/9689#issuecomment-2582927149

[2] https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/pv/emul-priv-op.c;h=70150c27227661baa253af8693ff00f2ab640a98;hb=HEAD#l295

[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/probe.c?id=v6.13#n1208

Quoted 2 lines · Bjorn Helgaas

I guess the code at [2] is running in user mode and uses Linux syscalls for config access? Is it straceable?

Nope, it's running as the hypervisor and mediates Linux's access to the hardware. In fact, Linux PV kernel (which dom0 is by default under Xen) is running in ring 3...

But I can add some more logging there.

Quoted 1 line

Quoted 1 line

[2] https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/pv/emul-priv-op.c;h=70150c27227661baa253af8693ff00f2ab640a98;hb=HEAD#l295

Quoted 8 lines · Marek Marczykowski-Górecki

Quoted 2 lines · Bjorn Helgaas

I guess the code at [2] is running in user mode and uses Linux syscalls for config access? Is it straceable?

Nope, it's running as the hypervisor and mediates Linux's access to the hardware. In fact, Linux PV kernel (which dom0 is by default under Xen) is running in ring 3...

Quoted 1 line

But I can add some more logging there.

So I guess the hypervisor performs the config access on behalf of the Linux PV kernel? Obviously Linux thinks CRS/RRS SV is enabled, but I suppose all the lspci output is similarly based on whatever the hypervisor does, so how do we know the actual hardware config?

Quoted 11 lines · Bjorn Helgaas

Quoted 9 lines · Marek Marczykowski-Górecki

Quoted 2 lines · Bjorn Helgaas

I guess the code at [2] is running in user mode and uses Linux syscalls for config access? Is it straceable?

Nope, it's running as the hypervisor and mediates Linux's access to the hardware. In fact, Linux PV kernel (which dom0 is by default under Xen) is running in ring 3...

But I can add some more logging there.

Quoted 4 lines

So I guess the hypervisor performs the config access on behalf of the Linux PV kernel? Obviously Linux thinks CRS/RRS SV is enabled, but I suppose all the lspci output is similarly based on whatever the hypervisor does, so how do we know the actual hardware config?

The hypervisor only intercepts config space writes; reads, particularly when going via mmcfg, ought to be unaffected, and hence what lspci shows should be "the actual hardware config".

Quoted 6 lines · Bjorn Helgaas

The report claims the problem only happens with Xen. I'm not a Xen person, and I don't know how to find the relevant config accessors. The snippets of kernel messages I see at [1] all mention pciback, so that's my only clue of where to look. Bottom line, I have no idea what the config accessor path is, and maybe we could learn something by looking at whatever it is.

AFAIK there are no separate config accessors under Xen dom0, the default ones are used. xen-pcifront takes over PCI config space access (and few more) only in a domU (and only for PV), when PCI passthrough is used. Here, it didn't went that far...

But then, Xen may intercept such access [2]. If I read it right, it should allow all access (is_hardware_domain(dom0)==true, and also the device is not on ro_map - otherwise reset wouldn't work at all).

Quoted 13 lines · Marek Marczykowski-Górecki

Quoted 6 lines · Bjorn Helgaas

The report claims the problem only happens with Xen. I'm not a Xen person, and I don't know how to find the relevant config accessors. The snippets of kernel messages I see at [1] all mention pciback, so that's my only clue of where to look. Bottom line, I have no idea what the config accessor path is, and maybe we could learn something by looking at whatever it is.

AFAIK there are no separate config accessors under Xen dom0, the default ones are used. xen-pcifront takes over PCI config space access (and few more) only in a domU (and only for PV), when PCI passthrough is used. Here, it didn't went that far...

Quoted 3 lines

But then, Xen may intercept such access [2]. If I read it right, it should allow all access (is_hardware_domain(dom0)==true, and also the device is not on ro_map - otherwise reset wouldn't work at all).

The other day you mentioned (on Matrix I think) that you observe mmcfg not being used on that system. Am I misremembering? (Since the capability where the control bit lives is an extended one, that capability would neither be read nor modified when mmcfg is unavailable.)

Quoted 14 lines · Jan Beulich

Quoted 13 lines · Marek Marczykowski-Górecki

Quoted 6 lines · Bjorn Helgaas

The report claims the problem only happens with Xen. I'm not a Xen person, and I don't know how to find the relevant config accessors. The snippets of kernel messages I see at [1] all mention pciback, so that's my only clue of where to look. Bottom line, I have no idea what the config accessor path is, and maybe we could learn something by looking at whatever it is.

AFAIK there are no separate config accessors under Xen dom0, the default ones are used. xen-pcifront takes over PCI config space access (and few more) only in a domU (and only for PV), when PCI passthrough is used. Here, it didn't went that far...

Quoted 8 lines

Quoted 3 lines

But then, Xen may intercept such access [2]. If I read it right, it should allow all access (is_hardware_domain(dom0)==true, and also the device is not on ro_map - otherwise reset wouldn't work at all).

The other day you mentioned (on Matrix I think) that you observe mmcfg not being used on that system. Am I misremembering? (Since the capability where the control bit lives is an extended one, that capability would neither be read nor modified when mmcfg is unavailable.)

Yes, but later (once dom0 starts) it switched back to mmcfg. Now I see this:
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff (XEN) PCI: Using MCFG for segment 0000 bus 00-ff

Another thing I noticed in the bug report - the reporter says warm reboot from 6.11 (where it works) to 6.12 avoids the issue (not sure about further reboots). Cold boot directly to 6.12 results in this buggy behavior.

Quoted 13 lines · Marek Marczykowski-Górecki

Quoted 6 lines · Bjorn Helgaas

The report claims the problem only happens with Xen. I'm not a Xen person, and I don't know how to find the relevant config accessors. The snippets of kernel messages I see at [1] all mention pciback, so that's my only clue of where to look. Bottom line, I have no idea what the config accessor path is, and maybe we could learn something by looking at whatever it is.

AFAIK there are no separate config accessors under Xen dom0, the default ones are used. xen-pcifront takes over PCI config space access (and few more) only in a domU (and only for PV), when PCI passthrough is used. Here, it didn't went that far...

Quoted 3 lines

But then, Xen may intercept such access [2]. If I read it right, it should allow all access (is_hardware_domain(dom0)==true, and also the device is not on ro_map - otherwise reset wouldn't work at all).

The other day you mentioned (on Matrix I think) that you observe mmcfg not being used on that system. Am I misremembering? (Since the capability where the control bit lives is an extended one, that capability would neither be read nor modified when mmcfg is unavailable.)

Quoted 14 lines · Jan Beulich

Quoted 13 lines · Marek Marczykowski-Górecki

Quoted 6 lines · Bjorn Helgaas

The report claims the problem only happens with Xen. I'm not a Xen person, and I don't know how to find the relevant config accessors. The snippets of kernel messages I see at [1] all mention pciback, so that's my only clue of where to look. Bottom line, I have no idea what the config accessor path is, and maybe we could learn something by looking at whatever it is.

AFAIK there are no separate config accessors under Xen dom0, the default ones are used. xen-pcifront takes over PCI config space access (and few more) only in a domU (and only for PV), when PCI passthrough is used. Here, it didn't went that far...

Quoted 8 lines

Quoted 3 lines

But then, Xen may intercept such access [2]. If I read it right, it should allow all access (is_hardware_domain(dom0)==true, and also the device is not on ro_map - otherwise reset wouldn't work at all).

The other day you mentioned (on Matrix I think) that you observe mmcfg not being used on that system. Am I misremembering? (Since the capability where the control bit lives is an extended one, that capability would neither be read nor modified when mmcfg is unavailable.)

If you're referring to the Configuration RRS Software Visibility Enable bit, that's in the PCIe Capability Root Control register, which is in the PCI-compatible config space (the first 256 bytes), not the extended config space.

Quoted 48 lines · Bjorn Helgaas

Quoted 4 lines · Marek Marczykowski-Górecki

After updating PV dom0 to Linux 6.12, The Mediatek MT7922 device reports all 0xff when accessing its config space. This happens only after device reset (which is also triggered when binding the device to the xen-pciback driver).

Thanks for the report and for all the debugging you've already done!

Quoted 40 lines

Reproducer:

    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: c3 14 16 06 00 00 10 00 00 00 80 02 10 00 00 00
    ...
    # echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

The same operation done on Linux 6.12 running without Xen works fine.

git bisect points at:

commit d591f6804e7e1310881c9224d72247a2b65039af
Author: Bjorn Helgaas <[email protected]>
Date: Tue Aug 27 18:48:46 2024 -0500

PCI: Wait for device readiness with Configuration RRS

part of that commit:

(quoted diff) +9 -3

@@ -1311,9 +1320,15 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
                        return -ENOTTY;
                }
 
-               pci_read_config_dword(dev, PCI_COMMAND, &id);
-               if (!PCI_POSSIBLE_ERROR(id))
-                       break;
+               if (root && root->config_crs_sv) {
+                       pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+                       if (!pci_bus_crs_vendor_id(id))
+                               break;
+               } else {
+                       pci_read_config_dword(dev, PCI_COMMAND, &id);
+                       if (!PCI_POSSIBLE_ERROR(id))
+                               break;
+               }
 
 

Quoted 10 lines

Quoted 7 lines

Adding some debugging, the PCI_VENDOR_ID read in pci_dev_wait() returns initially 0xffffffff. If I extend the condition with
"&& !PCI_POSSIBLE_ERROR(id)", then the issue disappear. But reading the patch description, it would break VF.
I'm not sure where the issue is, but given it breaks only when running with Xen, I guess something is wrong with "Configuration RRS Software Visibility" in that case.

I'm missing something. If you get 0xffffffff, that is not the 0x0001 Vendor ID, so pci_dev_wait() should exit immediately.

I'm not sure what is going on there either, but my _guess_ is that the loop exits too early due to the above. And it makes some further actions to fail.

Quoted 60 lines · Marek Marczykowski-Górecki

Quoted 58 lines · Bjorn Helgaas

Quoted 4 lines · Marek Marczykowski-Górecki

After updating PV dom0 to Linux 6.12, The Mediatek MT7922 device reports all 0xff when accessing its config space. This happens only after device reset (which is also triggered when binding the device to the xen-pciback driver).

Thanks for the report and for all the debugging you've already done!

Quoted 40 lines

Reproducer:

    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: c3 14 16 06 00 00 10 00 00 00 80 02 10 00 00 00
    ...
    # echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
    # lspci -xs 01:00.0
    01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
    00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

The same operation done on Linux 6.12 running without Xen works fine.

git bisect points at:

commit d591f6804e7e1310881c9224d72247a2b65039af
Author: Bjorn Helgaas <[email protected]>
Date: Tue Aug 27 18:48:46 2024 -0500

PCI: Wait for device readiness with Configuration RRS

part of that commit:

(quoted diff) +9 -3

@@ -1311,9 +1320,15 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
                        return -ENOTTY;
                }
 
-               pci_read_config_dword(dev, PCI_COMMAND, &id);
-               if (!PCI_POSSIBLE_ERROR(id))
-                       break;
+               if (root && root->config_crs_sv) {
+                       pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
+                       if (!pci_bus_crs_vendor_id(id))
+                               break;
+               } else {
+                       pci_read_config_dword(dev, PCI_COMMAND, &id);
+                       if (!PCI_POSSIBLE_ERROR(id))
+                               break;
+               }
 
 

Quoted 7 lines

Adding some debugging, the PCI_VENDOR_ID read in pci_dev_wait() returns initially 0xffffffff. If I extend the condition with
"&& !PCI_POSSIBLE_ERROR(id)", then the issue disappear. But reading the patch description, it would break VF.
I'm not sure where the issue is, but given it breaks only when running with Xen, I guess something is wrong with "Configuration RRS Software Visibility" in that case.

I'm missing something. If you get 0xffffffff, that is not the 0x0001 Vendor ID, so pci_dev_wait() should exit immediately.

Quoted 3 lines

I'm not sure what is going on there either, but my _guess_ is that the loop exits too early due to the above. And it makes some further actions to fail.

Seems like a good guess worth investigating. Maybe log all config accesses to this device after the FLR and see what we're doing?

I've added logging of all config read/write to this device. Full log at [1].

A little explanation:
- it's done in pci_conf_read/pci_conf_write in https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/pci.c;h=97b792e578f1093194466081ad3651ade21cae7d;hb=HEAD
- cf8 means cf8 port value (BDF + register)
- bytes is read/write size (1/2/4)
- offset is the offset in the register (on top of cf8), but not in data
- data is either retrieved value, or written value, depending on
function
- it's logging only accesses to 01:00.0

interesting part:

lspci before reset:
(XEN) d0v3 conf read cf8 0x80010000 bytes 4 offset 0 data 0x61614c3 (XEN) d0v3 conf read cf8 0x80010004 bytes 4 offset 0 data 0x100000 (XEN) d0v3 conf read cf8 0x80010008 bytes 4 offset 0 data 0x2800000 (XEN) d0v3 conf read cf8 0x8001000c bytes 4 offset 0 data 0x10 (XEN) d0v3 conf read cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v3 conf read cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v3 conf read cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v3 conf read cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v3 conf read cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v3 conf read cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v3 conf read cf8 0x80010028 bytes 4 offset 0 data 0 (XEN) d0v3 conf read cf8 0x8001002c bytes 4 offset 0 data 0xe61614c3 (XEN) d0v3 conf read cf8 0x80010030 bytes 4 offset 0 data 0 (XEN) d0v3 conf read cf8 0x80010034 bytes 4 offset 0 data 0x80 (XEN) d0v3 conf read cf8 0x80010038 bytes 4 offset 0 data 0 (XEN) d0v3 conf read cf8 0x8001003c bytes 4 offset 0 data 0x1ff (XEN) d0v3 conf read cf8 0x80010080 bytes 4 offset 0 data 0x2e010 (XEN) d0v3 conf read cf8 0x800100e0 bytes 4 offset 0 data 0x18af805 (XEN) d0v3 conf read cf8 0x800100f8 bytes 4 offset 0 data 0xc8030001

reset:
(XEN) d0v1 conf read cf8 0x800100fc bytes 2 offset 0 data 0x8 (XEN) d0v1 conf read cf8 0x800100fc bytes 2 offset 0 data 0x8 (XEN) d0v1 conf read cf8 0x8001008c bytes 4 offset 0 data 0x145dc12 (XEN) d0v1 conf read cf8 0x80010000 bytes 4 offset 0 data 0x61614c3 (XEN) d0v1 conf read cf8 0x80010004 bytes 4 offset 0 data 0x100000 (XEN) d0v1 conf read cf8 0x80010008 bytes 4 offset 0 data 0x2800000 (XEN) d0v1 conf read cf8 0x8001000c bytes 4 offset 0 data 0x10 (XEN) d0v1 conf read cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v1 conf read cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v1 conf read cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v1 conf read cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v1 conf read cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v1 conf read cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v1 conf read cf8 0x80010028 bytes 4 offset 0 data 0 (XEN) d0v1 conf read cf8 0x8001002c bytes 4 offset 0 data 0xe61614c3 (XEN) d0v1 conf read cf8 0x80010030 bytes 4 offset 0 data 0 (XEN) d0v1 conf read cf8 0x80010034 bytes 4 offset 0 data 0x80 (XEN) d0v1 conf read cf8 0x80010038 bytes 4 offset 0 data 0 (XEN) d0v1 conf read cf8 0x8001003c bytes 4 offset 0 data 0x1ff (XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 0 data 0x2910 (XEN) d0v1 conf read cf8 0x80010090 bytes 2 offset 0 data 0x1c2 (XEN) d0v1 conf read cf8 0x800100a8 bytes 2 offset 0 data 0x400 (XEN) d0v1 conf read cf8 0x800100b0 bytes 2 offset 0 data 0x2 (XEN) d0v1 conf read cf8 0x80010004 bytes 2 offset 2 data 0x10

(XEN) d0v1 conf read cf8 0x80010034 bytes 1 offset 0 data 0x80 (XEN) d0v1 conf read cf8 0x80010080 bytes 2 offset 0 data 0xe010 (XEN) d0v1 conf read cf8 0x800100e0 bytes 2 offset 0 data 0xf805 (XEN) d0v1 conf read cf8 0x800100f8 bytes 2 offset 0 data 0x1 (XEN) d0v1 conf write cf8 0x80010004 bytes 2 offset 0 data 0x400 (XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 2 data 0x9 (XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 0 data 0x2910 (XEN) d0v1 conf write cf8 0x80010088 bytes 2 offset 0 data 0xa910

(XEN) d0v2 conf read cf8 0x80010000 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf read cf8 0x80010090 bytes 2 offset 0 data 0xffff (XEN) d0v2 conf write cf8 0x80010090 bytes 2 offset 0 data 0xfffc (XEN) d0v2 conf write cf8 0x80010090 bytes 2 offset 0 data 0xffff (XEN) d0v2 conf write cf8 0x80010088 bytes 2 offset 0 data 0x2910 (XEN) d0v2 conf write cf8 0x80010090 bytes 2 offset 0 data 0x1c2 (XEN) d0v2 conf write cf8 0x800100a8 bytes 2 offset 0 data 0x400 (XEN) d0v2 conf write cf8 0x800100b0 bytes 2 offset 0 data 0x2 (XEN) d0v2 conf read cf8 0x8001003c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001003c bytes 4 offset 0 data 0x1ff (XEN) d0v2 conf read cf8 0x80010038 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010038 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010034 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010034 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010030 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010030 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001002c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001002c bytes 4 offset 0 data 0xe61614c3 (XEN) d0v2 conf read cf8 0x80010028 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010028 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010024 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010024 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010020 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010020 bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x8001001c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001001c bytes 4 offset 0 data 0 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010018 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010018 bytes 4 offset 0 data 0x90b00004 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010014 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010014 bytes 4 offset 0 data 0x80 (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x80010010 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010010 bytes 4 offset 0 data 0x1090000c (XEN) d0v2 conf read cf8 0x8001000c bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x8001000c bytes 4 offset 0 data 0x10 (XEN) d0v2 conf read cf8 0x80010008 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010008 bytes 4 offset 0 data 0x2800000 (XEN) d0v2 conf read cf8 0x80010004 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010004 bytes 4 offset 0 data 0x100000 (XEN) d0v2 conf read cf8 0x80010000 bytes 4 offset 0 data 0xffffffff (XEN) d0v2 conf write cf8 0x80010000 bytes 4 offset 0 data 0x61614c3 (XEN) d0v2 conf read cf8 0x80010004 bytes 2 offset 2 data 0xffff (XEN) d0v2 conf read cf8 0x80010034 bytes 1 offset 0 data 0xff (XEN) d0v2 conf read cf8 0x800100fc bytes 2 offset 0 data 0xffff

[1] https://gist.github.com/marmarek/b4391c71801145e52590e877c559c5e0

Quoted 2 lines · Marek Marczykowski-Górecki

I've added logging of all config read/write to this device. Full log at [1].

...

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010034 bytes 1 offset 0 data 0x80

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010080 bytes 2 offset 0 data 0xe010

Quoted 1 line

(XEN) d0v1 conf read cf8 0x800100e0 bytes 2 offset 0 data 0xf805

Quoted 1 line

(XEN) d0v1 conf read cf8 0x800100f8 bytes 2 offset 0 data 0x1

Quoted 1 line

(XEN) d0v1 conf write cf8 0x80010004 bytes 2 offset 0 data 0x400

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 2 data 0x9

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 0 data 0x2910

Quoted 1 line

(XEN) d0v1 conf write cf8 0x80010088 bytes 2 offset 0 data 0xa910

This is the express capability's Link Control 2 Register afaict.

Quoted 1 line

(XEN) d0v2 conf read cf8 0x80010000 bytes 4 offset 0 data 0xffffffff

Quoted 1 line

[1] https://gist.github.com/marmarek/b4391c71801145e52590e877c559c5e0

Quoted 2 lines · Jan Beulich

Quoted 1 line · Marek Marczykowski-Górecki

(XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 2 data 0x9

PCIe Cap at 0x80, PCI_EXP_DEVCTL is 0x08, PCI_EXP_DEVSTA is 0x0a.

0x80010088 would be PCI_EXP_DEVCTL (a 2-byte register), maybe offset 2 gets us to PCI_EXP_DEVSTA? Not sure.

0x0001 PCI_EXP_DEVSTA_CED /* Correctable Error Detected */ 0x0008 PCI_EXP_DEVSTA_URD /* Unsupported Request Detected */

Not impossible that these would be set. Lots of URs happen during enumeration and we're not very good about cleaning these up. Correctable errors are common for some devices. lspci -vv would decode the PCIe cap registers, including this.

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 0 data 0x2910

PCI_EXP_DEVCTL:
0x2000 PCI_EXP_DEVCTL_READRQ_512B
0x0800 PCI_EXP_DEVCTL_NOSNOOP_EN
0x0100 PCI_EXP_DEVCTL_EXT_TAG
0x0010 PCI_EXP_DEVCTL_RELAX_EN

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf write cf8 0x80010088 bytes 2 offset 0 data 0xa910

PCI_EXP_DEVCTL:
set 0x8000 PCI_EXP_DEVCTL_BCR_FLR

This looks like the actual FLR being initiated.

Quoted 1 line

This is the express capability's Link Control 2 Register afaict.

Unless I'm missing something this is actually Device Control. So far I think this all looks OK. The next part:

Quoted 32 lines · Bjorn Helgaas

Quoted 2 lines · Jan Beulich

Quoted 1 line · Marek Marczykowski-Górecki

(XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 2 data 0x9

PCIe Cap at 0x80, PCI_EXP_DEVCTL is 0x08, PCI_EXP_DEVSTA is 0x0a.

0x80010088 would be PCI_EXP_DEVCTL (a 2-byte register), maybe offset 2 gets us to PCI_EXP_DEVSTA? Not sure.

0x0001 PCI_EXP_DEVSTA_CED /* Correctable Error Detected */ 0x0008 PCI_EXP_DEVSTA_URD /* Unsupported Request Detected */

Not impossible that these would be set. Lots of URs happen during enumeration and we're not very good about cleaning these up. Correctable errors are common for some devices. lspci -vv would decode the PCIe cap registers, including this.

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 0 data 0x2910

PCI_EXP_DEVCTL:
0x2000 PCI_EXP_DEVCTL_READRQ_512B
0x0800 PCI_EXP_DEVCTL_NOSNOOP_EN
0x0100 PCI_EXP_DEVCTL_EXT_TAG
0x0010 PCI_EXP_DEVCTL_RELAX_EN

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf write cf8 0x80010088 bytes 2 offset 0 data 0xa910

PCI_EXP_DEVCTL:
set 0x8000 PCI_EXP_DEVCTL_BCR_FLR

This looks like the actual FLR being initiated.

Quoted 4 lines

Quoted 1 line

This is the express capability's Link Control 2 Register afaict.

Unless I'm missing something this is actually Device Control. So far I think this all looks OK. The next part:

What you say is very plausible as far as the observed behavior goes,
but: According to the lspci output provided earlier the express capability is at 58 (hex).

Hence here we're 30 (hex) into the
capability, which according to the spec I'm looking at is Link Control 2. Yet as said - with what you say being plausible, likely I'm simply getting something very wrong.

Jan

Quoted 4 lines · Jan Beulich

Quoted 2 lines · Marek Marczykowski-Górecki

I've added logging of all config read/write to this device. Full log at [1].

...

I suspect there's something wrong with the Root Port RRS SV configuration.

Can you add the two patches below?

I don't *think* either should make a difference. The first enables RRS SV earlier and maybe in a cleaner place; the second should avoid some pointless capability searches that clutter the logs.

What does d0v1/d0v2/d0v3 mean?

Can you add 00:02.2, the Root Port leading to bus 01, so we can see the RRS SV configuration? Maybe also lspci -vv for both 00:02.2 and 01:00.0?

Maybe include timestamps and try an FLR without Xen (which I assume works correctly) so we can see how long the device typically takes to become ready?

Notes below on the snippet that you commented on, Jan. I think it makes sense until the read after FLR returns 0xffffffff.

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010034 bytes 1 offset 0 data 0x80

PCI_CAPABILITY_LIST, first cap at 0x80

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010080 bytes 2 offset 0 data 0xe010

PCI_CAP_ID_EXP (0x10) at 0x80, next cap at 0xe0

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf read cf8 0x800100e0 bytes 2 offset 0 data 0xf805

PCI_CAP_ID_MSI (0x05) at 0xe0, next cap at 0xf8

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf read cf8 0x800100f8 bytes 2 offset 0 data 0x1

PCI_CAP_ID_PM (0x01) at 0xf8, end of cap list

These caps match the offsets from the lspci output in the full log:

1:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
Subsystem: MEDIATEK Corp. Device e616
Flags: fast devsel, IRQ 255
Memory at 8010900000 (64-bit, prefetchable) [disabled] [size=1M] Memory at 90b00000 (64-bit, non-prefetchable) [disabled] [size=32K]
Capabilities: [80] Express Endpoint, IntMsgNum 0
Capabilities: [e0] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [f8] Power Management version 3

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf write cf8 0x80010004 bytes 2 offset 0 data 0x400

Set PCI_COMMAND_INTX_DISABLE, disable BARs, Bus Master. Looks like the end of pci_dev_save_and_disable().

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 2 data 0x9

PCIe Cap at 0x80, PCI_EXP_DEVCTL is 0x08, PCI_EXP_DEVSTA is 0x0a.

0x80010088 would be PCI_EXP_DEVCTL (a 2-byte register), maybe offset 2 gets us to PCI_EXP_DEVSTA? Not sure.

0x0001 PCI_EXP_DEVSTA_CED /* Correctable Error Detected */ 0x0008 PCI_EXP_DEVSTA_URD /* Unsupported Request Detected */

Not impossible that these would be set. Lots of URs happen during enumeration and we're not very good about cleaning these up. Correctable errors are common for some devices. lspci -vv would decode the PCIe cap registers, including this.

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf read cf8 0x80010088 bytes 2 offset 0 data 0x2910

PCI_EXP_DEVCTL:
0x2000 PCI_EXP_DEVCTL_READRQ_512B
0x0800 PCI_EXP_DEVCTL_NOSNOOP_EN
0x0100 PCI_EXP_DEVCTL_EXT_TAG
0x0010 PCI_EXP_DEVCTL_RELAX_EN

Quoted 1 line

Quoted 1 line

(XEN) d0v1 conf write cf8 0x80010088 bytes 2 offset 0 data 0xa910

PCI_EXP_DEVCTL:
set 0x8000 PCI_EXP_DEVCTL_BCR_FLR

This looks like the actual FLR being initiated.

Quoted 1 line

This is the express capability's Link Control 2 Register afaict.

Unless I'm missing something this is actually Device Control. So far I think this all looks OK. The next part:

Quoted 1 line

Quoted 1 line

(XEN) d0v2 conf read cf8 0x80010000 bytes 4 offset 0 data 0xffffffff

looks like a problem. The normal successful read gets 0x061614c3. After the FLR, if RRS SV is enabled, we should get either 0x0001ffff or 0x061614c3.

Here we would exit the loop in pci_dev_wait() because we didn't see 0x0001 and we expect that the device is ready and we got a valid Vendor ID.

So we proceed to restoring config space via pci_restore_state(), where we restore some PCIe registers and the header (first 64 bytes). My *guess* is the device isn't ready (or at least not responding) since all the reads return ~0.

Quoted 5 lines · Bjorn Helgaas

Quoted 4 lines · Jan Beulich

Quoted 2 lines · Marek Marczykowski-Górecki

I've added logging of all config read/write to this device. Full log at [1].

...

... Generally it looks like this device has broken FLR, and the reset works due to the fallback to the secondary bus reset on timeout. I repeated the test with my additional "&& !PCI_POSSIBLE_ERROR(id)" and I got this:
[2] https://gist.github.com/marmarek/db0808702131b69ea2f66f339a55d71b

The first log is with xen, and the second with native linux (and added PCI config space logging in drivers/pci/access.c).

Quoted 13 lines · Marek Marczykowski-Górecki

Quoted 5 lines · Bjorn Helgaas

Quoted 4 lines · Jan Beulich

Quoted 2 lines · Marek Marczykowski-Górecki

I've added logging of all config read/write to this device. Full log at [1].

...

... Generally it looks like this device has broken FLR, and the reset works due to the fallback to the secondary bus reset on timeout. I repeated the test with my additional "&& !PCI_POSSIBLE_ERROR(id)" and I got this:
[2] https://gist.github.com/marmarek/db0808702131b69ea2f66f339a55d71b

Quoted 2 lines

The first log is with xen, and the second with native linux (and added PCI config space logging in drivers/pci/access.c).

This is just to annotate these logs. Correct me if you see something wrong.

Both logs include this patch:

@@ -1297,7 +1297,8 @@ static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)

                  if (root && root->config_rrs_sv) {
                          pci_read_config_dword(dev, PCI_VENDOR_ID, &id);
  -                     if (!pci_bus_rrs_vendor_id(id))
  +                     if (!pci_bus_rrs_vendor_id(id) &&
  +                         !PCI_POSSIBLE_ERROR(id))
                                  break;

I think both logs show this sequence:

- Initiate FLR on 01:00.0

- In pci_dev_wait(), poll PCI_VENDOR_ID, looking for something other

    than 0x0001 (which would indicate RRS response) or 0xffff (from
    patch above).

- Time out after ~70 seconds and return -ENOTTY.

- Attempt Secondary Bus Reset using 00:02.2, the Root Port leading

    to 01:00.0.

- Successfully read PCI_VENDOR_ID.

- Looks the same, whether linux is running natively or on top of Xen.

Relevant devices (from mediatek-debug-6.12-patch2+bridgelog.log):

00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Phoenix GPP Bridge

    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0

01:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
Capabilities: [80] Express (v2) Endpoint, IntMsgNum 0

From mediatek-debug-6.12-patch2+bridgelog.log (from [2] above):

[anaconda root@test-12 /]# time echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset
(XEN) d0v3 conf write cf8 0x80010088 bytes 2 offset 0 data 0xa910 <-- set 01:00.0 FLR
(XEN) d0v3 conf read cf8 0x80010000 bytes 4 offset 0 data 0xffffffff
...
(XEN) d0v4 conf read cf8 0x80010000 bytes 4 offset 0 data 0xffffffff
...
(XEN) d0v4 conf read cf8 0x8000123c bytes 2 offset 2 data 0x2 (0x3c + offset 2 = 0x3e)
(XEN) d0v4 conf write cf8 0x8000123c bytes 2 offset 2 data 0x42 <-- set 00:02.2 SBR
(XEN) d0v4 conf write cf8 0x8000123c bytes 2 offset 2 data 0x2
...
(XEN) d0v4 conf read cf8 0x80010000 bytes 4 offset 0 data 0x61614c3 <-- 01:00.0 VID/DID
...
real 1m10.825s

From mediatek-debug-native-6.12-patch2+bridgelog.log (also from [2] above):

[anaconda root@test-12 ~]# time echo 1 > /sys/bus/pci/devices/0000:01:00.0/reset

  [  240.449215] pciback 0000:01:00.0: resetting
  [  240.450709] PCI: write bus 0x1 devfn 0x0 pos 0x88 size 2 value 0xa910   <-- set 01:00.0 FLR
  [  240.553264] PCI: read bus 0x1 devfn 0x0 pos 0x0 size 4 value 0xffffffff
  ...
  [  309.481728] PCI: read bus 0x1 devfn 0x0 pos 0x0 size 4 value 0xffffffff
  [  309.481747] pciback 0000:01:00.0: not ready 65535ms after FLR; giving up
  ...
  [  309.482667] PCI: read bus 0x0 devfn 0x12 pos 0x3e size 2 value 0x2      PCI_BRIDGE_CONTROL
  [  309.482670] PCI: write bus 0x0 devfn 0x12 pos 0x3e size 2 value 0x42    <-- set 00:02.2 SBR
  [  309.485184] PCI: write bus 0x0 devfn 0x12 pos 0x3e size 2 value 0x2

...

  [  309.617782] PCI: read bus 0x1 devfn 0x0 pos 0x0 size 4 value 0x61614c3  <-- 01:00.0 VID/DID
  [  309.629234] pciback 0000:01:00.0: reset done

Quoted 5 lines · Bjorn Helgaas

Quoted 4 lines · Jan Beulich

Quoted 2 lines · Marek Marczykowski-Górecki

I've added logging of all config read/write to this device. Full log at [1].

...

... Generally it looks like this device has broken FLR, and the reset works due to the fallback to the secondary bus reset on timeout.

I repeated the test with my additional "&& !PCI_POSSIBLE_ERROR(id)" and I got this:
https://gist.github.com/marmarek/db0808702131b69ea2f66f339a55d71b