pci: quirks: Disable native PCIe hotplug on MSI Claw A8 root bridge
Subsystems (3)
PATCH v2 1/1 pci: quirks: Disable native PCIe hotplug on MSI Claw A8 root bridge
The MSI Claw A8 (MS-1T8K) hard-locks on resume from s2idle whenever an SD/MMC card is present in the onboard Realtek RTS525A card reader (10ec:525a), which sits behind root port 0000:00:02.2. The reader's PCIe endpoint is fixed/soldered and not user-hotpluggable.
Firmware advertises this root port as PCIe hotplug-capable, and _OSC grants the OS native hotplug control. When native hotplug is OS-owned, resuming from s2idle races the port's hotplug/PME resume handling against the RTS525A's own resume sequence when a card is present, hard-locking the system.
Two kernel command line parameters were confirmed to prevent the hang:
pcie_aspm=off and pcie_ports=compat. Both stop the OS from running its own hotplug/PME resume path against this port. The former does so by dropping _OSC negotiation during boot, skipping the path that enables PCIeHotPlug. The latter prevents the PME service resume handling by aborting registration of the ports native services. Testing different kernel boot commands, including pciehp=off and pcie_pme=off did not resolve the issue. AER, DPC, SHPC, and LTR flags were not tested because the hardware reports that the platform doesn't support those features.
This quirk implements as narrow a fix as possible, clearing is_hotplug_bridge and is_pciehp on the root bridge, scoped by DMI board name and this root port's bus/device/function, so the port driver never registers a native hotplug service against it.
SD card insertion is handled entirely by rtsx_pci's own card-detect logic, so is unaffected by this quirk.
---
v2:
- Switch to pci_info vice dev_info.
- use is_hotplug_bridge and is_pciehp on the root port instead of native_pcie_hotplug on the root bridge.
v1: https://lore.kernel.org/linux-pci/[email protected]/
--- drivers/pci/quirks.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
drivers/pci/quirks.c +23 -0
@@ -6420,3 +6420,26 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_GLI ,0x9750 , pci_mask_replay_timer_timeout);DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_GLI ,0x9755 , pci_mask_replay_timer_timeout);#endif + +/* +* The MSI Claw A8 firmware advertises native PCIe hotplug support for +* this root port, but native hotplug handling causes resume failures. +* Prevent the PCIe port driver from claiming native hotplug ownership +* of this port. +*/ +static void quirk_claw_a8_no_native_hotplug (struct pci_dev *pdev) +{ +if (!dmi_match (DMI_BOARD_NAME ,"MS-1T8K" )) +return ; + +if (pdev->bus->number !=0 || +PCI_SLOT (pdev->devfn) !=2 || +PCI_FUNC (pdev->devfn) !=2 ) +return ; + +pci_info (pdev,"disabling native PCIe hotplug\n" ); + pdev->is_hotplug_bridge =0 ; + pdev->is_pciehp =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_AMD ,0x150b , + quirk_claw_a8_no_native_hotplug);
Signature · 1 line
2.55.0
[email protected]
AI reviewSashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines
The MSI Claw A8 (MS-1T8K) hard-locks on resume from s2idle whenever an SD/MMC card is present in the onboard Realtek RTS525A card reader (10ec:525a), which sits behind root port 0000:00:02.2. The reader's PCIe endpoint is fixed/soldered and not user-hotpluggable.
Firmware advertises this root port as PCIe hotplug-capable, and
_OSCgrants the OS native hotplug control. When native hotplug is OS-owned, resuming from s2idle races the port's hotplug/PME resume handling against the RTS525A's own resume sequence when a card is present, hard-locking the system.Two kernel command line parameters were confirmed to prevent the hang:
pcie_aspm=off and pcie_ports=compat. Both stop the OS from running its own hotplug/PME resume path against this port. The former does so by dropping_OSCnegotiation during boot, skipping the path that [ ... ]
Quoted 4 lines
SD card insertion is handled entirely by rtsx_pci's own card-detect logic, so is unaffected by this quirk.
Signed-off-by Derek J. Clark <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 3 lines · Derek J. Clark
The MSI Claw A8 (MS-1T8K) hard-locks on resume from s2idle whenever an SD/MMC card is present in the onboard Realtek RTS525A card reader (10ec:525a), which sits behind root port 0000:00:02.2.
The RTS525A is known to falsely claim that it supports ASPM L0s, even though it only supports L1. There is a patch pending to work around this hardware erratum, I'm wondering whether it makes the issue go away on the MSI Claw A8?
https://lore.kernel.org/r/[email protected]/
Quoted 2 lines
The reader's PCIe endpoint is fixed/soldered and not user-hotpluggable. Firmware advertises this root port as PCIe hotplug-capable
That looks like an oversight on the part of MSI. Is there a BIOS update available which clears the Hot-Plug Capable bit in the Slot Capabilities Register? (I'm *assuming* that bit is set -- I haven't seen any dmesg or lspci -vvv output in your e-mails, perhaps you could open a bug at bugzilla.kernel.org and upload it there.)
Quoted 4 lines
When native hotplug is OS-owned,
resuming from s2idle races the port's hotplug/PME resume handling against the RTS525A's own resume sequence when a card is present, hard-locking the system.
This sounds like a driver bug in rtsx_pci which needs to be root-caused and fixed, rather than worked around.
pciehp (the OS-native PCIe hotplug driver) normally doesn't de-enumerate and re-enumerate devices on resume from system sleep. It uses a heuristic to determine whether the device in the slot was replaced during system sleep and synthesizes a hotplug event if it believes there's now a different device in the slot. Otherwise it just leaves the device in the slot alone. The replacement detection is in pciehp_device_replaced(), drivers/pci/hotplug/pciehp_hpc.c line 567. You could try hacking that function to return false and see if it helps.
The heuristic looks at the Vendor ID and Device ID in config space. I do recall that a different type of card readers, RTS5261, trigger a hotplug event when a regular UHS card is replaced with an SD Express card. The card reader is hot-unplugged and the SD Express card appears as an NVMe drive with a different Device ID in config space:
https://lore.kernel.org/r/[email protected]/
That said, I'm not even sure RTS525A supports SD Express.
What kind of card do you have inserted in the card reader, is it UHS or SD Express? Does the issue go away if you eject the card before going to system sleep?
Quoted 1 line
+++
b/drivers/pci/quirks.c
[...]
Quoted 12 lines
drivers/pci/quirks.c +12
-0+/* +* The MSI Claw A8 firmware advertises native PCIe hotplug support for +* this root port, but native hotplug handling causes resume failures. +* Prevent the PCIe port driver from claiming native hotplug ownership +* of this port. +*/ +static void quirk_claw_a8_no_native_hotplug (struct pci_dev *pdev) +{ +if (!dmi_match (DMI_BOARD_NAME ,"MS-1T8K" )) +return ; + + if (pdev->bus->number != 0 ||
drivers/pci/quirks.c static void pci_mask_replay_timer_timeout(struct pci_dev *pdev) · line 6445
Quoted 10 lines
drivers/pci/quirks.c +10
-0+PCI_SLOT (pdev->devfn) !=2 || + PCI_FUNC(pdev->devfn) != 2) + return; + +pci_info (pdev,"disabling native PCIe hotplug\n" ); + pdev->is_hotplug_bridge =0 ; + pdev->is_pciehp =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_AMD ,0x150b , + quirk_claw_a8_no_native_hotplug);
Unfortunately this is just a workaround that happens to make the issue disappear, but we'd really want to identify the root cause first.
If you boot with no_console_suspend, are you able to see any messages on suspend or resume that could help us understand what is going on?
Thanks,
Lukas
Quoted 6 lines · Lukas Wunner
Quoted 3 lines · Derek J. Clark
The MSI Claw A8 (MS-1T8K) hard-locks on resume from s2idle whenever an SD/MMC card is present in the onboard Realtek RTS525A card reader (10ec:525a), which sits behind root port 0000:00:02.2.
Quoted 7 lines
The RTS525A is known to falsely claim that it supports ASPM L0s, even though it only supports L1. There is a patch pending to work around this hardware erratum, I'm wondering whether it makes the issue go away on the MSI Claw A8?
Hi Lukas,
Sending again as i inadvertently hit reply instead of reply all.
The freeze still occurs with that patch applied, yes.
Quoted 9 lines
Quoted 2 lines
The reader's PCIe endpoint is fixed/soldered and not user-hotpluggable. Firmware advertises this root port as PCIe hotplug-capable
That looks like an oversight on the part of MSI. Is there a BIOS update available which clears the Hot-Plug Capable bit in the Slot Capabilities Register? (I'm *assuming* that bit is set -- I haven't seen any dmesg or lspci -vvv output in your e-mails, perhaps you could open a bug at bugzilla.kernel.org and upload it there.)
I'm on the latest firmware they have as of abuut a month ago. They don't participate in LVFS AFAIK but I made sure to update before switching to Linux.
I posted them here for the entire device tree for the SD card reader:
https://github.com/ValveSoftware/SteamOS/issues/2473#issuecomment-5222972815
Quoted 8 lines
Quoted 4 lines
When native hotplug is OS-owned,
resuming from s2idle races the port's hotplug/PME resume handling against the RTS525A's own resume sequence when a card is present, hard-locking the system.This sounds like a driver bug in
rtsx_pciwhich needs to be root-caused and fixed, rather than worked around.
Quoted 9 lines
pciehp (the OS-native PCIe hotplug driver) normally doesn't de-enumerate and re-enumerate devices on resume from system sleep. It uses a heuristic to determine whether the device in the slot was replaced during system sleep and synthesizes a hotplug event if it believes there's now a different device in the slot. Otherwise it just leaves the device in the slot alone. The replacement detection is in
pciehp_device_replaced(),drivers/pci/hotplug/pciehp_hpc.cline 567. You could try hacking that function to return false and see if it helps.
That worked! That strikes me as odd since the card is never replaced.
Quoted 6 lines
The heuristic looks at the Vendor ID and Device ID in config space. I do recall that a different type of card readers, RTS5261, trigger a hotplug event when a regular UHS card is replaced with an SD Express card. The card reader is hot-unplugged and the SD Express card appears as an NVMe drive with a different Device ID in config space:
Quoted 8 lines
https://lore.kernel.org/r/[email protected]/
That said, I'm not even sure RTS525A supports SD Express.
What kind of card do you have inserted in the card reader, is it UHS or SD Express? Does the issue go away if you eject the card before going to system sleep?
I've been using a SanDisk Ultra 32GB microSDHC UHS-I Card. And yes, it only happens when the card is present.
Quoted 29 lines
Quoted 1 line
+++
b/drivers/pci/quirks.c[...]
Quoted 12 lines
drivers/pci/quirks.c +12
-0+/* +* The MSI Claw A8 firmware advertises native PCIe hotplug support for +* this root port, but native hotplug handling causes resume failures. +* Prevent the PCIe port driver from claiming native hotplug ownership +* of this port. +*/ +static void quirk_claw_a8_no_native_hotplug (struct pci_dev *pdev) +{ +if (!dmi_match (DMI_BOARD_NAME ,"MS-1T8K" )) +return ; + + if (pdev->bus->number != 0 ||drivers/pci/quirks.c static void pci_mask_replay_timer_timeout(struct pci_dev *pdev) · line 6445
Quoted 10 lines
drivers/pci/quirks.c +10
-0+PCI_SLOT (pdev->devfn) !=2 || + PCI_FUNC(pdev->devfn) != 2) + return; + +pci_info (pdev,"disabling native PCIe hotplug\n" ); + pdev->is_hotplug_bridge =0 ; + pdev->is_pciehp =0 ; +} +DECLARE_PCI_FIXUP_HEADER (PCI_VENDOR_ID_AMD ,0x150b , + quirk_claw_a8_no_native_hotplug);Unfortunately this is just a workaround that happens to make the issue disappear, but we'd really want to identify the root cause first.
Quoted 3 lines
If you boot with
no_console_suspend, are you able to see any messages on suspend or resume that could help us understand what is going on?
Nothing of note. After amdgpu finishes its smu resume sequence the console _ flashes and doesn't progress.
I did try adding some debug printing to pciehp_device_replaced() but saw no output for that either. Since that didn't work I tried replacing each true result in the function one at a time and found the offending case
if (pdev->hdr_type == PCI_HEADER_TYPE_NORMAL &&
(pci_read_config_dword(pdev, PCI_SUBSYSTEM_VENDOR_ID, ®) ||- reg != (pdev->subsystem_vendor | (pdev->subsystem_device << 16))))
- return true;
+ reg != (pdev->subsystem_vendor | (pdev->subsystem_device
<< 16)))) {
+ pci_info(pdev, "pdev subsystem vendor ID mismatch");
+ return false;
+
+ }
[ 25.686128] rtsx_pci 0000:c2:00.0: pciehp: pdev subsystem vendor ID mismatch
Any insight on why that might be occurring and what my next troubleshooting steps should be are appreciated.
** Update: after my initial email & before I sent it back to the entire ML I did another test and printed the reg vs expected values. It seems the SD card reader ID is reflected when it is enumerated during resume.
reg=0x525a10ec expected=0x14af1462
Thanks,
Derek
Quoted 3 lines
Thanks,
Lukas
Quoted 1 line · Derek John Clark
[ 25.686128] rtsx_pci 0000:c2:00.0: pciehp: pdev subsystem vendor ID mismatch
[...]
Quoted 1 line
reg=0x525a10ec expected=0x14af1462
The subsystem vendor ID was 0x1462 (MSI) when the card reader was enumerated on boot and it is 0x10ec (Realtek) on resume.
Likewise the subystem device ID was 0x14af on boot, 0x525a on resume.
It's another bug in the MSI BIOS (in addition to marking the Root Port hotplug-capable): The BIOS neglected to re-initialize the subsystem vendor/device ID registers in config space on resume to the same values that it set on boot. The card reader was likely reset while going through the system sleep transition and its config space thus needs to be re-initialized.
What happens is that pciehp marks the card reader disconnected because it assumes that it was replaced during system sleep. It then synthesizes a Presence Detect Changed event:
https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/pci/hotplug/pciehp_core.c#L300
So the pciehp interrupt thread pciehp_ist() will remove the rtsx_pci driver and de-enumerate the device.
Replacing or removing PCIe devices during system sleep happens all the time (e.g. with Thunderbolt) and we haven't seen issues with regards to that. The likelihood is higher that the lockup is caused by rtsx_pci.
Maybe rtsx_pci_remove() blocks somewhere and prevents forward progress of the resume transition. In particular, it calls pm_runtime_get_sync() and cancel_delayed_work_sync(), which are blocking calls, i.e. they wait for something else to happen. You could try commenting those calls out and see if it avoids the lockup:
https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/misc/cardreader/rtsx_pcr.c#L1608
Another possibility is a crash due to a NULL pointer deref or GPF somewhere in that driver. This is pretty difficult to debug if there's no possibility to see any dmesg output.
Even if we manage to find the root cause of the lockup, as long as the BIOS isn't fixed, the card reader will always be de-enumerated and re-enumerated on resume and consequently the card will briefly disappear. This will make it impossible to use the card as root filesystem.
Quoted 2 lines
I've been using a SanDisk Ultra 32GB microSDHC UHS-I Card. And yes, it only happens when the card is present.
If you go to sleep and resume without SD card, do you then see in dmesg that the card reader was de-enumerated and re-enumerated? Normally there should be at least a "Card not present" / "Card present" message from pciehp, if it sensed a replaced device. ("Card" refers to PCIe card, not SD card.)
Also, what's the subsystem vendor/device ID as seen with lspci after a successful system sleep transition (successful = without SD card)?
Thanks,
Lukas
Quoted 32 lines · Lukas Wunner
Quoted 1 line · Derek John Clark
[ 25.686128] rtsx_pci 0000:c2:00.0: pciehp: pdev subsystem vendor ID mismatch[...]
Quoted 1 line
reg=0x525a10ec expected=0x14af1462
The subsystem vendor ID was
0x1462(MSI) when the card reader was enumerated on boot and it is0x10ec(Realtek) on resume.Likewise the subystem device ID was
0x14afon boot,0x525aon resume.It's another bug in the MSI BIOS (in addition to marking the Root Port hotplug-capable): The BIOS neglected to re-initialize the subsystem vendor/device ID registers in config space on resume to the same values that it set on boot. The card reader was likely reset while going through the system sleep transition and its config space thus needs to be re-initialized.
What happens is that pciehp marks the card reader disconnected because it assumes that it was replaced during system sleep. It then synthesizes a Presence Detect Changed event:
https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/pci/hotplug/pciehp_core.c#L300
So the pciehp interrupt thread
pciehp_ist()will remove thertsx_pcidriver and de-enumerate the device.Replacing or removing PCIe devices during system sleep happens all the time (e.g. with Thunderbolt) and we haven't seen issues with regards to that. The likelihood is higher that the lockup is caused by
rtsx_pci.
Quoted 8 lines
Maybe
rtsx_pci_remove()blocks somewhere and prevents forward progress of the resume transition. In particular, it callspm_runtime_get_sync()andcancel_delayed_work_sync(), which are blocking calls, i.e. they wait for something else to happen. You could try commenting those calls out and see if it avoids the lockup:https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/misc/cardreader/rtsx_pcr.c#L1608
This was good instinct. I was able to drill down and find approximately where it is hanging.
rtsx_pci_remove() ->mfd_remove_devices() -> mfd_remove_devices_fn() -> platform_device_unregister() -> platform_device_del() -> device_del() -> bus_remove_device() -> device_release_driver() -> device_release_driver_internal() -> __device_release_driver() -> device_remove()
In device_remove() (drivers/base/dd.c) it gets past device_remove_groups() and hangs before completing this if block:
if (dev->bus && dev->bus->remove)
dev->bus->remove(dev);
else if (dev->driver->remove)
dev->driver->remove(dev);After this logging stops.
- Derek
Quoted 9 lines
Another possibility is a crash due to a NULL pointer deref or GPF somewhere in that driver. This is pretty difficult to debug if there's no possibility to see any dmesg output.
Even if we manage to find the root cause of the lockup, as long as the BIOS isn't fixed, the card reader will always be de-enumerated and re-enumerated on resume and consequently the card will briefly disappear. This will make it impossible to use the card as root filesystem.
I agree. Even if we were to convince them to fix it, there are plenty of devices in the wild that will likely never get the BIOS update since MSI isn't using LFVS for this device. (I'm not sure if they do that for any device)
Quoted 8 lines
Quoted 2 lines
I've been using a SanDisk Ultra 32GB microSDHC UHS-I Card. And yes, it only happens when the card is present.
If you go to sleep and resume without SD card, do you then see in dmesg that the card reader was de-enumerated and re-enumerated? Normally there should be at least a "Card not present" / "Card present" message from pciehp, if it sensed a replaced device. ("Card" refers to PCIe card, not SD card.)
i get card not present, then card present, then link up.
Quoted 3 lines
Also, what's the subsystem vendor/device ID as seen with lspci after a successful system sleep transition (successful = without SD card)?
it reflects the Realtek device VID/PID. I also tested resume after the 1 cycle with it re-inserted and it works fine after the first cycle, presumably because the device id matches on the second run and beyond
Thanks,
Derek.
Quoted 3 lines
Thanks,
Lukas
Quoted 9 lines · Derek John Clark
This was good instinct. I was able to drill down and find approximately where it is hanging.
rtsx_pci_remove()->mfd_remove_devices() ->mfd_remove_devices_fn()->platform_device_unregister()->platform_device_del()->device_del()->bus_remove_device()->device_release_driver()->device_release_driver_internal()->__device_release_driver()->device_remove()
Quoted 7 lines
In
device_remove()(drivers/base/dd.c) it gets pastdevice_remove_groups()and hangs before completing this if block:if (
dev->bus&&dev->bus->remove)dev->bus->remove(dev); else if (dev->driver->remove) dev->driver->remove(dev);
Okay, that ->remove callback should be rtsx_pci_sdmmc_drv_remove() in drivers/mmc/host/rtsx_pci_sdmmc.c. Can you maybe identify where execution stops in that function?
Basically the rtsx_pci driver creates a child device of the PCI device and the mmc host driver binds to that child device. Removing the mmc host driver fails here for some reason.
Quoted 6 lines
Quoted 5 lines
If you go to sleep and resume without SD card, do you then see in dmesg that the card reader was de-enumerated and re-enumerated? Normally there should be at least a "Card not present" / "Card present" message from pciehp, if it sensed a replaced device. ("Card" refers to PCIe card, not SD card.)
Quoted 8 lines
i get card not present, then card present, then link up.
Quoted 2 lines
Also, what's the subsystem vendor/device ID as seen with lspci after a successful system sleep transition (successful = without SD card)?
it reflects the Realtek device VID/PID. I also tested resume after the 1 cycle with it re-inserted and it works fine after the first cycle, presumably because the device id matches on the second run and beyond
Right, when the device is re-enumerated after the first system sleep transition, the Realtek subdevice vendor/device ID is cached and that matches on all subsequent system sleep transitions. So if we manage to find and eliminate the cause of the hang, there'll only be the inconvenience of the removal/re-enumeration on first sleep.
Thanks,
Lukas
Quoted 19 lines · Lukas Wunner
Quoted 9 lines · Derek John Clark
This was good instinct. I was able to drill down and find approximately where it is hanging.
rtsx_pci_remove()->mfd_remove_devices() ->mfd_remove_devices_fn()->platform_device_unregister()->platform_device_del()->device_del()->bus_remove_device()->device_release_driver()->device_release_driver_internal()->__device_release_driver()->device_remove()Quoted 7 lines
In
device_remove()(drivers/base/dd.c) it gets pastdevice_remove_groups()and hangs before completing this if block:if (
dev->bus&&dev->bus->remove)dev->bus->remove(dev); else if (dev->driver->remove) dev->driver->remove(dev);
Quoted 7 lines
Okay, that ->remove callback should be
rtsx_pci_sdmmc_drv_remove()indrivers/mmc/host/rtsx_pci_sdmmc.c. Can you maybe identify where execution stops in that function?Basically the
rtsx_pcidriver creates a child device of the PCI device and the mmc host driver binds to that child device. Removing the mmc host driver fails here for some reason.
Hi Lukas,
I was able to drill down further. When the mmc device gets to blk_report_disk_dead() in block/genhd.c there is an xa_for_each loop. On the second loop of that it seems to hang in bdev_mark_dead(). That sets a callback that runs fs_bdev_mark_dead() which then runs sync_filesystem(). This is all hit because the "surprise" bool is set to false unconditionally in __del_gendisk().
Commenting out this from __del_gendisk():
if (!test_bit(GD_DEAD, &disk->state))
blk_report_disk_dead(disk, false);
Avoids the hang.
I'm not sure how much further we want to go down into this as we've moved quite far out of the driver space and are now well into the block subsystem. I'm working on a proof of concept that would allow the driver to inform the mmc core that this was a surprise, but I'm not sure how that would land. From what I've read the mmc core was intentionally detached from higher level drivers, though a few notification methods do exist. If I get that fully working I'll post it here but it will probably be a few days as I've run into some other obligations this week.
Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
If you go to sleep and resume without SD card, do you then see in dmesg that the card reader was de-enumerated and re-enumerated? Normally there should be at least a "Card not present" / "Card present" message from pciehp, if it sensed a replaced device. ("Card" refers to PCIe card, not SD card.)
Quoted 8 lines
i get card not present, then card present, then link up.
Quoted 2 lines
Also, what's the subsystem vendor/device ID as seen with lspci after a successful system sleep transition (successful = without SD card)?
it reflects the Realtek device VID/PID. I also tested resume after the 1 cycle with it re-inserted and it works fine after the first cycle, presumably because the device id matches on the second run and beyond
Quoted 5 lines
Right, when the device is re-enumerated after the first system sleep transition, the Realtek subdevice vendor/device ID is cached and that matches on all subsequent system sleep transitions. So if we manage to find and eliminate the cause of the hang, there'll only be the inconvenience of the removal/re-enumeration on first sleep.
I suppose that is true, but in that case we still wouldn't be able to use it as the root fs since it gets removed/re-added during a suspend. Would the quirk I submitted not have this compromise?
Quoted 3 lines
Thanks,
Lukas
Quoted 7 lines · Derek John Clark
I was able to drill down further. When the mmc device gets to
blk_report_disk_dead()inblock/genhd.cthere is anxa_for_eachloop. On the second loop of that it seems to hang inbdev_mark_dead(). That sets a callback that runsfs_bdev_mark_dead()which then runssync_filesystem(). This is all hit because the "surprise" bool is set to false unconditionally in__del_gendisk().
Quoted 5 lines
Commenting out this from
__del_gendisk():
if (!test_bit(GD_DEAD,&disk->state))
blk_report_disk_dead(disk, false);Avoids the hang.
Thank you so much, you've root-caused the issue: We're missing a call to blk_mark_disk_dead() at the top of rtsx_pci_sdmmc_drv_remove() if the underlying pci_dev is marked disconnected. Let me get back to you with a fix in a bit.
Quoted 9 lines
I'm not sure how much further we want to go down into this as we've moved quite far out of the driver space and are now well into the block subsystem. I'm working on a proof of concept that would allow the driver to inform the mmc core that this was a surprise, but I'm not sure how that would land. From what I've read the mmc core was intentionally detached from higher level drivers, though a few notification methods do exist. If I get that fully working I'll post it here but it will probably be a few days as I've run into some other obligations this week.
It's fine, you don't need to invest any more time into that and I understand your frustration about how much effort is necessary to fix this.
The issue you've found affects anyone removing e.g. a Thunderbolt- attached dock with an MMC card reader during system sleep. So it's useful and important to have root-caused and fix that. Unfortunately kernel development often means getting sidetracked like this.
Quoted 3 lines
I suppose that is true, but in that case we still wouldn't be able to use it as the root fs since it gets removed/re-added during a suspend. Would the quirk I submitted not have this compromise?
Okay I wasn't sure that using the MMC card as root filesystem is even possible on this product. If that's a potential use case, it needs a separate fix. I'll come up with a proposal for that one as well.
Thanks,
Lukas
[cc += Ulf, start of thread is here:
https://lore.kernel.org/all/[email protected]/
]
Quoted 12 lines · Lukas Wunner
Quoted 11 lines · Derek John Clark
I was able to drill down further. When the mmc device gets to
blk_report_disk_dead()inblock/genhd.cthere is anxa_for_eachloop. On the second loop of that it seems to hang inbdev_mark_dead(). That sets a callback that runsfs_bdev_mark_dead()which then runssync_filesystem(). This is all hit because the "surprise" bool is set to false unconditionally in__del_gendisk().Commenting out this from
__del_gendisk():
if (!test_bit(GD_DEAD,&disk->state))
blk_report_disk_dead(disk, false);
Quoted 6 lines
Quoted 1 line
Avoids the hang.
Thank you so much, you've root-caused the issue: We're missing a call to
blk_mark_disk_dead()at the top ofrtsx_pci_sdmmc_drv_remove()if the underlyingpci_devis marked disconnected. Let me get back to you with a fix in a bit.
So the completely untested patch below might be an upstreamable approach. I'm adding MMC maintainer Ulf to cc in case he has early feedback.
If this works, feel free to submit a proper patch and claim authorship if you want. I'll gladly let you have that given the amount of time you've already sunk into it. If you'd rather have me submit a patch (and deal with any regressions caused by it), I'll be happy to do that as well. Thanks!
-- >8 --
drivers/mmc/core/block.c +7 -4
@@ -2978,8 +2978,11 @@ static int mmc_blk_alloc_parts(struct mmc_card *card, struct mmc_blk_data *md)return 0 ; } -static void mmc_blk_remove_req (struct mmc_blk_data *md) +static void mmc_blk_remove_req (struct mmc_card *card,struct mmc_blk_data *md) { +if (mmc_card_removed (card)) +blk_mark_disk_dead (md->disk); +/* * Flush remaining requests and free queues. It is freeing the queue * that stops new requests from being accepted.
@@ -3006,7 +3009,7 @@ static void mmc_blk_remove_parts(struct mmc_card *card,list_for_each_safe (pos, q, &md->part) { part_md = list_entry(pos, struct mmc_blk_data, part);list_del (pos); -mmc_blk_remove_req (part_md); +mmc_blk_remove_req (card, part_md); } }
@@ -3252,7 +3255,7 @@ static int mmc_blk_probe(struct mmc_card *card) out:mmc_blk_remove_parts (card, md); -mmc_blk_remove_req (md); +mmc_blk_remove_req (card, md); out_free:destroy_workqueue (card->complete_wq);return ret;
@@ -3273,7 +3276,7 @@ static void mmc_blk_remove(struct mmc_card *card)if (!mmc_card_sd_combo (card))pm_runtime_disable (&card->dev);pm_runtime_put_noidle (&card->dev); -mmc_blk_remove_req (md); +mmc_blk_remove_req (card, md);destroy_workqueue (card->complete_wq); }
drivers/mmc/core/host.c +14 -0
@@ -703,3 +703,17 @@ void mmc_free_host(struct mmc_host *host) }EXPORT_SYMBOL (mmc_free_host); + +/** +* mmc_host_set_removed - declare host removed +* @host: mmc host +* +* Declare the host (and any inserted card) removed and inaccessible. +*/ +void mmc_host_set_removed (struct mmc_host *host) +{ +if (host->card) +mmc_card_set_removed (host->card); +} + +EXPORT_SYMBOL (mmc_host_set_removed);
drivers/mmc/host/rtsx_pci_sdmmc.c +3 -0
@@ -1511,6 +1511,9 @@ static void rtsx_pci_sdmmc_drv_remove(struct platform_device *pdev) pcr->slots[RTSX_SD_CARD ].card_event =NULL ; mmc = host->mmc; +if (pci_dev_is_disconnected (pcr->pci)) +mmc_host_set_removed (mmc); +cancel_work_sync (&host->work);mutex_lock (&host->host_mutex);
include/linux/mmc/host.h +1 -0
@@ -588,6 +588,7 @@ struct mmc_host {int mmc_add_host (struct mmc_host *);void mmc_remove_host (struct mmc_host *);void mmc_free_host (struct mmc_host *); +void mmc_host_set_removed (struct mmc_host *host);void mmc_of_parse_clk_phase (struct device *dev,struct mmc_clk_phase_map *map);int mmc_of_parse (struct mmc_host *host);
Quoted 19 lines · Lukas Wunner
[cc += Ulf, start of thread is here:
https://lore.kernel.org/all/[email protected]/
]Quoted 14 lines · Lukas Wunner
Quoted 12 lines · Derek John Clark
I was able to drill down further. When the mmc device gets to
blk_report_disk_dead()inblock/genhd.cthere is anxa_for_eachloop. On the second loop of that it seems to hang inbdev_mark_dead(). That sets a callback that runsfs_bdev_mark_dead()which then runssync_filesystem(). This is all hit because the "surprise" bool is set to false unconditionally in__del_gendisk().Commenting out this from
__del_gendisk():
if (!test_bit(GD_DEAD,&disk->state))
blk_report_disk_dead(disk, false);Avoids the hang.
Quoted 8 lines
Quoted 4 lines
Thank you so much, you've root-caused the issue: We're missing a call to
blk_mark_disk_dead()at the top ofrtsx_pci_sdmmc_drv_remove()if the underlyingpci_devis marked disconnected. Let me get back to you with a fix in a bit.So the completely untested patch below might be an upstreamable approach. I'm adding MMC maintainer Ulf to cc in case he has early feedback.
Great, I'll give it a test soon and follow up.
Quoted 6 lines
If this works, feel free to submit a proper patch and claim authorship if you want. I'll gladly let you have that given the amount of time you've already sunk into it. If you'd rather have me submit a patch (and deal with any regressions caused by it), I'll be happy to do that as well. Thanks!
I think tested by tags will be sufficient once I've completed that. The shape of your patch is a bit different than what I was working on and is a better approach. I appreciate your help on this so far.
Thanks,
Derek
Quoted 94 lines
-- >8 --
drivers/mmc/core/block.c +5
-2@@ -2978,8 +2978,11 @@ static int mmc_blk_alloc_parts(struct mmc_card *card, struct mmc_blk_data *md)return 0 ; } -static void mmc_blk_remove_req (struct mmc_blk_data *md) +static void mmc_blk_remove_req (struct mmc_card *card,struct mmc_blk_data *md) { +if (mmc_card_removed (card)) +blk_mark_disk_dead (md->disk); +/* * Flush remaining requests and free queues. It is freeing the queue * that stops new requests from being accepted. @@ -3006,7 +3009,7 @@ static void mmc_blk_remove_parts(struct mmc_card *card,list_for_each_safe (pos, q, &md->part) { part_md = list_entry(pos, struct mmc_blk_data, part);list_del (pos); -mmc_blk_remove_req (part_md); +mmc_blk_remove_req (card, part_md); } }@@ -3252,7 +3255,7 @@ static int mmc_blk_probe(struct mmc_card *card)out:
mmc_blk_remove_parts(card, md);drivers/mmc/core/block.c +1
-1-mmc_blk_remove_req (md); +mmc_blk_remove_req (card, md);
out_free:destroy_workqueue(card->complete_wq); return ret;drivers/mmc/core/block.c +1
-1@@ -3273,7 +3276,7 @@ static void mmc_blk_remove(struct mmc_card *card)if (!mmc_card_sd_combo (card))pm_runtime_disable (&card->dev);pm_runtime_put_noidle (&card->dev); -mmc_blk_remove_req (md); +mmc_blk_remove_req (card, md);destroy_workqueue (card->complete_wq); }drivers/mmc/core/host.c +14
-0@@ -703,3 +703,17 @@ void mmc_free_host(struct mmc_host *host) }EXPORT_SYMBOL (mmc_free_host); + +/** +* mmc_host_set_removed - declare host removed +* @host: mmc host +* +* Declare the host (and any inserted card) removed and inaccessible. +*/ +void mmc_host_set_removed (struct mmc_host *host) +{ +if (host->card) +mmc_card_set_removed (host->card); +} + +EXPORT_SYMBOL (mmc_host_set_removed);drivers/mmc/host/rtsx_pci_sdmmc.c +3
-0@@ -1511,6 +1511,9 @@ static void rtsx_pci_sdmmc_drv_remove(struct platform_device *pdev) pcr->slots[RTSX_SD_CARD ].card_event =NULL ; mmc = host->mmc; +if (pci_dev_is_disconnected (pcr->pci)) +mmc_host_set_removed (mmc); +cancel_work_sync (&host->work);mutex_lock (&host->host_mutex);include/linux/mmc/host.h +1
-0@@ -588,6 +588,7 @@ struct mmc_host {int mmc_add_host (struct mmc_host *);void mmc_remove_host (struct mmc_host *);void mmc_free_host (struct mmc_host *); +void mmc_host_set_removed (struct mmc_host *host);void mmc_of_parse_clk_phase (struct device *dev,struct mmc_clk_phase_map *map);int mmc_of_parse (struct mmc_host *host);
Quoted 1 line · Lukas Wunner
So the completely untested patch below might be an upstreamable approach.
Sorry, I was OOO for a day and sent out the patch without compile-testing right before hitting the road. I'm back now and it turns out that an #include statement was mising. Rectified patch below.
-- >8 --
drivers/mmc/core/block.c +7 -4
@@ -2978,8 +2978,11 @@ static int mmc_blk_alloc_parts(struct mmc_card *card, struct mmc_blk_data *md)return 0 ; } -static void mmc_blk_remove_req (struct mmc_blk_data *md) +static void mmc_blk_remove_req (struct mmc_card *card,struct mmc_blk_data *md) { +if (mmc_card_removed (card)) +blk_mark_disk_dead (md->disk); +/* * Flush remaining requests and free queues. It is freeing the queue * that stops new requests from being accepted.
@@ -3006,7 +3009,7 @@ static void mmc_blk_remove_parts(struct mmc_card *card,list_for_each_safe (pos, q, &md->part) { part_md = list_entry(pos, struct mmc_blk_data, part);list_del (pos); -mmc_blk_remove_req (part_md); +mmc_blk_remove_req (card, part_md); } }
@@ -3252,7 +3255,7 @@ static int mmc_blk_probe(struct mmc_card *card) out:mmc_blk_remove_parts (card, md); -mmc_blk_remove_req (md); +mmc_blk_remove_req (card, md); out_free:destroy_workqueue (card->complete_wq);return ret;
@@ -3273,7 +3276,7 @@ static void mmc_blk_remove(struct mmc_card *card)if (!mmc_card_sd_combo (card))pm_runtime_disable (&card->dev);pm_runtime_put_noidle (&card->dev); -mmc_blk_remove_req (md); +mmc_blk_remove_req (card, md);destroy_workqueue (card->complete_wq); }
drivers/mmc/core/host.c +15 -0
@@ -22,6 +22,7 @@#include <linux/mmc/card.h> #include <linux/mmc/slot-gpio.h> +#include "card.h" #include "core.h" #include "crypto.h" #include "host.h"
@@ -703,3 +704,17 @@ void mmc_free_host(struct mmc_host *host) }EXPORT_SYMBOL (mmc_free_host); + +/** +* mmc_host_set_removed - declare host removed +* @host: mmc host +* +* Declare the host (and any inserted card) removed and inaccessible. +*/ +void mmc_host_set_removed (struct mmc_host *host) +{ +if (host->card) +mmc_card_set_removed (host->card); +} + +EXPORT_SYMBOL (mmc_host_set_removed);
drivers/mmc/host/rtsx_pci_sdmmc.c +3 -0
@@ -1511,6 +1511,9 @@ static void rtsx_pci_sdmmc_drv_remove(struct platform_device *pdev) pcr->slots[RTSX_SD_CARD ].card_event =NULL ; mmc = host->mmc; +if (pci_dev_is_disconnected (pcr->pci)) +mmc_host_set_removed (mmc); +cancel_work_sync (&host->work);mutex_lock (&host->host_mutex);
include/linux/mmc/host.h +1 -0
@@ -588,6 +588,7 @@ struct mmc_host *devm_mmc_alloc_host(struct device *dev, int extra);int mmc_add_host (struct mmc_host *);void mmc_remove_host (struct mmc_host *);void mmc_free_host (struct mmc_host *); +void mmc_host_set_removed (struct mmc_host *host);void mmc_of_parse_clk_phase (struct device *dev,struct mmc_clk_phase_map *map);int mmc_of_parse (struct mmc_host *host);
Derek John Clark
Tested-byQuoted 8 lines · Lukas Wunner
Quoted 1 line · Lukas Wunner
So the completely untested patch below might be an upstreamable approach.
Sorry, I was OOO for a day and sent out the patch without compile-testing right before hitting the road. I'm back now and it turns out that an #include statement was mising. Rectified patch below.
No problem. Thanks for catching it after. I only got around to testing today. It works well.
Thanks,
Derek
Quoted 102 lines
-- >8 --
drivers/mmc/core/block.c +7
-4@@ -2978,8 +2978,11 @@ static int mmc_blk_alloc_parts(struct mmc_card *card, struct mmc_blk_data *md)return 0 ; } -static void mmc_blk_remove_req (struct mmc_blk_data *md) +static void mmc_blk_remove_req (struct mmc_card *card,struct mmc_blk_data *md) { +if (mmc_card_removed (card)) +blk_mark_disk_dead (md->disk); +/* * Flush remaining requests and free queues. It is freeing the queue * that stops new requests from being accepted. @@ -3006,7 +3009,7 @@ static void mmc_blk_remove_parts(struct mmc_card *card,list_for_each_safe (pos, q, &md->part) { part_md = list_entry(pos, struct mmc_blk_data, part);list_del (pos); -mmc_blk_remove_req (part_md); +mmc_blk_remove_req (card, part_md); } }@@ -3252,7 +3255,7 @@ static int mmc_blk_probe(struct mmc_card *card) out:mmc_blk_remove_parts (card, md); -mmc_blk_remove_req (md); +mmc_blk_remove_req (card, md); out_free:destroy_workqueue (card->complete_wq);return ret;@@ -3273,7 +3276,7 @@ static void mmc_blk_remove(struct mmc_card *card)if (!mmc_card_sd_combo (card))pm_runtime_disable (&card->dev);pm_runtime_put_noidle (&card->dev); -mmc_blk_remove_req (md); +mmc_blk_remove_req (card, md);destroy_workqueue (card->complete_wq); }drivers/mmc/core/host.c +15
-0@@ -22,6 +22,7 @@#include <linux/mmc/card.h> #include <linux/mmc/slot-gpio.h> +#include "card.h" #include "core.h" #include "crypto.h" #include "host.h" @@ -703,3 +704,17 @@ void mmc_free_host(struct mmc_host *host) }EXPORT_SYMBOL (mmc_free_host); + +/** +* mmc_host_set_removed - declare host removed +* @host: mmc host +* +* Declare the host (and any inserted card) removed and inaccessible. +*/ +void mmc_host_set_removed (struct mmc_host *host) +{ +if (host->card) +mmc_card_set_removed (host->card); +} + +EXPORT_SYMBOL (mmc_host_set_removed);drivers/mmc/host/rtsx_pci_sdmmc.c +3
-0@@ -1511,6 +1511,9 @@ static void rtsx_pci_sdmmc_drv_remove(struct platform_device *pdev) pcr->slots[RTSX_SD_CARD ].card_event =NULL ; mmc = host->mmc; +if (pci_dev_is_disconnected (pcr->pci)) +mmc_host_set_removed (mmc); +cancel_work_sync (&host->work);mutex_lock (&host->host_mutex);include/linux/mmc/host.h +1
-0@@ -588,6 +588,7 @@ struct mmc_host *devm_mmc_alloc_host(struct device *dev, int extra);int mmc_add_host (struct mmc_host *);void mmc_remove_host (struct mmc_host *);void mmc_free_host (struct mmc_host *); +void mmc_host_set_removed (struct mmc_host *host);void mmc_of_parse_clk_phase (struct device *dev,struct mmc_clk_phase_map *map);int mmc_of_parse (struct mmc_host *host);
Quoted 9 lines · Lukas Wunner
Quoted 3 lines · Derek John Clark
I suppose that is true, but in that case we still wouldn't be able to use it as the root fs since it gets removed/re-added during a suspend. Would the quirk I submitted not have this compromise?
Okay I wasn't sure that using the MMC card as root filesystem is even possible on this product. If that's a potential use case, it needs a separate fix. I'll come up with a proposal for that one as well.
Here's my proposal for allowing root on an MMC card.
Both this and the other patch I just sent (or something like it) need to be submitted upstream. The other one is necessary so others don't see a lockup when disconnecting e.g. a Thunderbolt-attached card reader during system sleep.
But either one of the patches should make the issue go away on the MSI Claw, so each patch needs to be tested by itself, without applying the other one as well. Thanks!
-- >8 --
arch/x86/pci/fixup.c +16 -0
@@ -1080,4 +1080,20 @@ static void quirk_tuxeo_rp_d3(struct pci_dev *pdev) } }DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_AMD ,0x1502 , quirk_tuxeo_rp_d3); + +/* +* MSI Claw A8 firmware initializes the onboard Realtek RTS525A cardreader +* with an MSI-specific Subsystem Vendor and Device ID on boot, but neglects +* to re-initialize it on resume. Avoid hot-removal of the cardreader due to +* the spurious ID change. +*/ +static void quirk_msi_claw_cardreader (struct pci_dev *pdev) +{ +if (pdev->subsystem_vendor ==0x1462 && + pdev->subsystem_device ==0x14af ) { + pdev->subsystem_vendor = pdev->vendor; + pdev->subsystem_device = pdev->device; + } +} +DECLARE_PCI_FIXUP_SUSPEND_LATE (PCI_VENDOR_ID_REALTEK ,0x525a , quirk_msi_claw_cardreader);#endif /* CONFIG_SUSPEND */
Derek John Clark
Tested-byQuoted 19 lines · Lukas Wunner
Quoted 9 lines · Lukas Wunner
Quoted 3 lines · Derek John Clark
I suppose that is true, but in that case we still wouldn't be able to use it as the root fs since it gets removed/re-added during a suspend. Would the quirk I submitted not have this compromise?
Okay I wasn't sure that using the MMC card as root filesystem is even possible on this product. If that's a potential use case, it needs a separate fix. I'll come up with a proposal for that one as well.
Here's my proposal for allowing root on an MMC card.
Both this and the other patch I just sent (or something like it) need to be submitted upstream. The other one is necessary so others don't see a lockup when disconnecting e.g. a Thunderbolt-attached card reader during system sleep.
Quoted 5 lines
But either one of the patches should make the issue go away on the MSI Claw, so each patch needs to be tested by itself, without applying the other one as well. Thanks!
-- >8 --
This also works independently of the previous fix.
- Derek
Quoted 25 lines
arch/x86/pci/fixup.c +16
-0@@ -1080,4 +1080,20 @@ static void quirk_tuxeo_rp_d3(struct pci_dev *pdev) } }DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_AMD ,0x1502 , quirk_tuxeo_rp_d3); + +/* +* MSI Claw A8 firmware initializes the onboard Realtek RTS525A cardreader +* with an MSI-specific Subsystem Vendor and Device ID on boot, but neglects +* to re-initialize it on resume. Avoid hot-removal of the cardreader due to +* the spurious ID change. +*/ +static void quirk_msi_claw_cardreader (struct pci_dev *pdev) +{ +if (pdev->subsystem_vendor ==0x1462 && + pdev->subsystem_device ==0x14af ) { + pdev->subsystem_vendor = pdev->vendor; + pdev->subsystem_device = pdev->device; + } +} +DECLARE_PCI_FIXUP_SUSPEND_LATE (PCI_VENDOR_ID_REALTEK ,0x525a , quirk_msi_claw_cardreader);#endif /* CONFIG_SUSPEND */