Skip to contentSkip to conversation

pci: quirks: Disable native PCIe hotplug on MSI Claw A8 root bridge

PATCH In review Derek J. Clark 3 participants 15 messages Aug 6 – Aug 12

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 pci: quirks: Disable native PCIe hotplug on MSI Claw A8 root bridge

The MSI Claw A8 (MS-1T8K) hard-locks on resume from s2idle whenever an SD/MMC card is present in the onboard Realtek RTS525A card reader (10ec:525a), which sits behind root port 0000:00:02.2. The reader's PCIe endpoint is fixed/soldered and not user-hotpluggable.

Firmware advertises this root port as PCIe hotplug-capable, and _OSC grants the OS native hotplug control. When native hotplug is OS-owned, resuming from s2idle races the port's hotplug/PME resume handling against the RTS525A's own resume sequence when a card is present, hard-locking the system.

Two kernel command line parameters were confirmed to prevent the hang:
pcie_aspm=off and pcie_ports=compat. Both stop the OS from running its own hotplug/PME resume path against this port. The former does so by dropping _OSC negotiation during boot, skipping the path that enables PCIeHotPlug. The latter prevents the PME service resume handling by aborting registration of the ports native services. Testing different kernel boot commands, including pciehp=off and pcie_pme=off did not resolve the issue. AER, DPC, SHPC, and LTR flags were not tested because the hardware reports that the platform doesn't support those features.

This quirk implements as narrow a fix as possible, clearing is_hotplug_bridge and is_pciehp on the root bridge, scoped by DMI board name and this root port's bus/device/function, so the port driver never registers a native hotplug service against it.

SD card insertion is handled entirely by rtsx_pci's own card-detect logic, so is unaffected by this quirk.

---
v2:
- Switch to pci_info vice dev_info.
- use is_hotplug_bridge and is_pciehp on the root port instead of native_pcie_hotplug on the root bridge.
v1: https://lore.kernel.org/linux-pci/[email protected]/

---
 drivers/pci/quirks.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
drivers/pci/quirks.c +23 -0
index 6501c949c5b7..62480fe10bcd 100644
@@ -6420,3 +6420,26 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9750, pci_mask_replay_timer_timeout);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9755, pci_mask_replay_timer_timeout);
 #endif
+
+/*
+ * The MSI Claw A8 firmware advertises native PCIe hotplug support for
+ * this root port, but native hotplug handling causes resume failures.
+ * Prevent the PCIe port driver from claiming native hotplug ownership
+ * of this port.
+ */
+static void quirk_claw_a8_no_native_hotplug(struct pci_dev *pdev)
+{
+	if (!dmi_match(DMI_BOARD_NAME, "MS-1T8K"))
+		return;
+
+	if (pdev->bus->number != 0 ||
+	    PCI_SLOT(pdev->devfn) != 2 ||
+	    PCI_FUNC(pdev->devfn) != 2)
+		return;
+
+	pci_info(pdev, "disabling native PCIe hotplug\n");
+	pdev->is_hotplug_bridge = 0;
+	pdev->is_pciehp = 0;
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_AMD, 0x150b,
+			 quirk_claw_a8_no_native_hotplug);
Signature · 1 line
2.55.0
#1 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines

The MSI Claw A8 (MS-1T8K) hard-locks on resume from s2idle whenever an SD/MMC card is present in the onboard Realtek RTS525A card reader (10ec:525a), which sits behind root port 0000:00:02.2. The reader's PCIe endpoint is fixed/soldered and not user-hotpluggable.

Firmware advertises this root port as PCIe hotplug-capable, and _OSC grants the OS native hotplug control. When native hotplug is OS-owned, resuming from s2idle races the port's hotplug/PME resume handling against the RTS525A's own resume sequence when a card is present, hard-locking the system.

Two kernel command line parameters were confirmed to prevent the hang:
pcie_aspm=off and pcie_ports=compat. Both stop the OS from running its own hotplug/PME resume path against this port. The former does so by dropping _OSC negotiation during boot, skipping the path that [ ... ]

Quoted 4 lines

SD card insertion is handled entirely by rtsx_pci's own card-detect logic, so is unaffected by this quirk.

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Derek J. Clark
Quoted 3 lines · Derek J. Clark

The MSI Claw A8 (MS-1T8K) hard-locks on resume from s2idle whenever an SD/MMC card is present in the onboard Realtek RTS525A card reader (10ec:525a), which sits behind root port 0000:00:02.2.

The RTS525A is known to falsely claim that it supports ASPM L0s, even though it only supports L1. There is a patch pending to work around this hardware erratum, I'm wondering whether it makes the issue go away on the MSI Claw A8?

https://lore.kernel.org/r/[email protected]/

Quoted 2 lines

The reader's PCIe endpoint is fixed/soldered and not user-hotpluggable. Firmware advertises this root port as PCIe hotplug-capable

That looks like an oversight on the part of MSI. Is there a BIOS update available which clears the Hot-Plug Capable bit in the Slot Capabilities Register? (I'm *assuming* that bit is set -- I haven't seen any dmesg or lspci -vvv output in your e-mails, perhaps you could open a bug at bugzilla.kernel.org and upload it there.)

Quoted 4 lines

When native hotplug is OS-owned,
resuming from s2idle races the port's hotplug/PME resume handling against the RTS525A's own resume sequence when a card is present, hard-locking the system.

This sounds like a driver bug in rtsx_pci which needs to be root-caused and fixed, rather than worked around.

pciehp (the OS-native PCIe hotplug driver) normally doesn't de-enumerate and re-enumerate devices on resume from system sleep. It uses a heuristic to determine whether the device in the slot was replaced during system sleep and synthesizes a hotplug event if it believes there's now a different device in the slot. Otherwise it just leaves the device in the slot alone. The replacement detection is in pciehp_device_replaced(), drivers/pci/hotplug/pciehp_hpc.c line 567. You could try hacking that function to return false and see if it helps.

The heuristic looks at the Vendor ID and Device ID in config space. I do recall that a different type of card readers, RTS5261, trigger a hotplug event when a regular UHS card is replaced with an SD Express card. The card reader is hot-unplugged and the SD Express card appears as an NVMe drive with a different Device ID in config space:

https://lore.kernel.org/r/[email protected]/

That said, I'm not even sure RTS525A supports SD Express.

What kind of card do you have inserted in the card reader, is it UHS or SD Express? Does the issue go away if you eject the card before going to system sleep?

Quoted 1 line

+++ b/drivers/pci/quirks.c

[...]

Quoted 12 lines
drivers/pci/quirks.c +12 -0
+/*
+ * The MSI Claw A8 firmware advertises native PCIe hotplug support for
+ * this root port, but native hotplug handling causes resume failures.
+ * Prevent the PCIe port driver from claiming native hotplug ownership
+ * of this port.
+ */
+static void quirk_claw_a8_no_native_hotplug(struct pci_dev *pdev)
+{
+	if (!dmi_match(DMI_BOARD_NAME, "MS-1T8K"))
+		return;
+
+	if (pdev->bus->number != 0 ||

drivers/pci/quirks.c static void pci_mask_replay_timer_timeout(struct pci_dev *pdev) · line 6445

Quoted 10 lines
drivers/pci/quirks.c +10 -0
+	    PCI_SLOT(pdev->devfn) != 2 ||
+	    PCI_FUNC(pdev->devfn) != 2)
+		return;
+
+	pci_info(pdev, "disabling native PCIe hotplug\n");
+	pdev->is_hotplug_bridge = 0;
+	pdev->is_pciehp = 0;
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_AMD, 0x150b,
+			 quirk_claw_a8_no_native_hotplug);

Unfortunately this is just a workaround that happens to make the issue disappear, but we'd really want to identify the root cause first.

If you boot with no_console_suspend, are you able to see any messages on suspend or resume that could help us understand what is going on?

Thanks,

Lukas

#3 lore raw
Quoted 6 lines · Lukas Wunner
Quoted 3 lines · Derek J. Clark

The MSI Claw A8 (MS-1T8K) hard-locks on resume from s2idle whenever an SD/MMC card is present in the onboard Realtek RTS525A card reader (10ec:525a), which sits behind root port 0000:00:02.2.

Quoted 7 lines

The RTS525A is known to falsely claim that it supports ASPM L0s, even though it only supports L1. There is a patch pending to work around this hardware erratum, I'm wondering whether it makes the issue go away on the MSI Claw A8?

https://lore.kernel.org/r/[email protected]/

Hi Lukas,

Sending again as i inadvertently hit reply instead of reply all.

The freeze still occurs with that patch applied, yes.

Quoted 9 lines
Quoted 2 lines

The reader's PCIe endpoint is fixed/soldered and not user-hotpluggable. Firmware advertises this root port as PCIe hotplug-capable

That looks like an oversight on the part of MSI. Is there a BIOS update available which clears the Hot-Plug Capable bit in the Slot Capabilities Register? (I'm *assuming* that bit is set -- I haven't seen any dmesg or lspci -vvv output in your e-mails, perhaps you could open a bug at bugzilla.kernel.org and upload it there.)

I'm on the latest firmware they have as of abuut a month ago. They don't participate in LVFS AFAIK but I made sure to update before switching to Linux.

I posted them here for the entire device tree for the SD card reader:
https://github.com/ValveSoftware/SteamOS/issues/2473#issuecomment-5222972815

Quoted 8 lines
Quoted 4 lines

When native hotplug is OS-owned,
resuming from s2idle races the port's hotplug/PME resume handling against the RTS525A's own resume sequence when a card is present, hard-locking the system.

This sounds like a driver bug in rtsx_pci which needs to be root-caused and fixed, rather than worked around.

Quoted 9 lines

pciehp (the OS-native PCIe hotplug driver) normally doesn't de-enumerate and re-enumerate devices on resume from system sleep. It uses a heuristic to determine whether the device in the slot was replaced during system sleep and synthesizes a hotplug event if it believes there's now a different device in the slot. Otherwise it just leaves the device in the slot alone. The replacement detection is in pciehp_device_replaced(), drivers/pci/hotplug/pciehp_hpc.c line 567. You could try hacking that function to return false and see if it helps.

That worked! That strikes me as odd since the card is never replaced.

Quoted 6 lines

The heuristic looks at the Vendor ID and Device ID in config space. I do recall that a different type of card readers, RTS5261, trigger a hotplug event when a regular UHS card is replaced with an SD Express card. The card reader is hot-unplugged and the SD Express card appears as an NVMe drive with a different Device ID in config space:

Quoted 8 lines

https://lore.kernel.org/r/[email protected]/

That said, I'm not even sure RTS525A supports SD Express.

What kind of card do you have inserted in the card reader, is it UHS or SD Express? Does the issue go away if you eject the card before going to system sleep?

I've been using a SanDisk Ultra 32GB microSDHC UHS-I Card. And yes, it only happens when the card is present.

Quoted 29 lines
Quoted 1 line

+++ b/drivers/pci/quirks.c

[...]

Quoted 12 lines
drivers/pci/quirks.c +12 -0
+/*
+ * The MSI Claw A8 firmware advertises native PCIe hotplug support for
+ * this root port, but native hotplug handling causes resume failures.
+ * Prevent the PCIe port driver from claiming native hotplug ownership
+ * of this port.
+ */
+static void quirk_claw_a8_no_native_hotplug(struct pci_dev *pdev)
+{
+     if (!dmi_match(DMI_BOARD_NAME, "MS-1T8K"))
+             return;
+
+     if (pdev->bus->number != 0 ||

drivers/pci/quirks.c static void pci_mask_replay_timer_timeout(struct pci_dev *pdev) · line 6445

Quoted 10 lines
drivers/pci/quirks.c +10 -0
+         PCI_SLOT(pdev->devfn) != 2 ||
+         PCI_FUNC(pdev->devfn) != 2)
+             return;
+
+     pci_info(pdev, "disabling native PCIe hotplug\n");
+     pdev->is_hotplug_bridge = 0;
+     pdev->is_pciehp = 0;
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_AMD, 0x150b,
+                      quirk_claw_a8_no_native_hotplug);

Unfortunately this is just a workaround that happens to make the issue disappear, but we'd really want to identify the root cause first.

Quoted 3 lines

If you boot with no_console_suspend, are you able to see any messages on suspend or resume that could help us understand what is going on?

Nothing of note. After amdgpu finishes its smu resume sequence the console _ flashes and doesn't progress.

I did try adding some debug printing to pciehp_device_replaced() but saw no output for that either. Since that didn't work I tried replacing each true result in the function one at a time and found the offending case

        if (pdev->hdr_type == PCI_HEADER_TYPE_NORMAL &&
            (pci_read_config_dword(pdev, PCI_SUBSYSTEM_VENDOR_ID, &reg) ||

- reg != (pdev->subsystem_vendor | (pdev->subsystem_device << 16))))
- return true;
+ reg != (pdev->subsystem_vendor | (pdev->subsystem_device
<< 16)))) {
+ pci_info(pdev, "pdev subsystem vendor ID mismatch");
+ return false;
+
+ }

[   25.686128] rtsx_pci 0000:c2:00.0: pciehp: pdev subsystem vendor ID mismatch

Any insight on why that might be occurring and what my next troubleshooting steps should be are appreciated.

** Update: after my initial email & before I sent it back to the entire ML I did another test and printed the reg vs expected values. It seems the SD card reader ID is reflected when it is enumerated during resume.

reg=0x525a10ec expected=0x14af1462

Thanks,

Derek

Quoted 3 lines

Thanks,

Lukas

#4 lore raw
Quoted 1 line · Derek John Clark
[   25.686128] rtsx_pci 0000:c2:00.0: pciehp: pdev subsystem vendor ID mismatch

[...]

Quoted 1 line

reg=0x525a10ec expected=0x14af1462

The subsystem vendor ID was 0x1462 (MSI) when the card reader was enumerated on boot and it is 0x10ec (Realtek) on resume.

Likewise the subystem device ID was 0x14af on boot, 0x525a on resume.

It's another bug in the MSI BIOS (in addition to marking the Root Port hotplug-capable): The BIOS neglected to re-initialize the subsystem vendor/device ID registers in config space on resume to the same values that it set on boot. The card reader was likely reset while going through the system sleep transition and its config space thus needs to be re-initialized.

What happens is that pciehp marks the card reader disconnected because it assumes that it was replaced during system sleep. It then synthesizes a Presence Detect Changed event:

https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/pci/hotplug/pciehp_core.c#L300

So the pciehp interrupt thread pciehp_ist() will remove the rtsx_pci driver and de-enumerate the device.

Replacing or removing PCIe devices during system sleep happens all the time (e.g. with Thunderbolt) and we haven't seen issues with regards to that. The likelihood is higher that the lockup is caused by rtsx_pci.

Maybe rtsx_pci_remove() blocks somewhere and prevents forward progress of the resume transition. In particular, it calls pm_runtime_get_sync() and cancel_delayed_work_sync(), which are blocking calls, i.e. they wait for something else to happen. You could try commenting those calls out and see if it avoids the lockup:

https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/misc/cardreader/rtsx_pcr.c#L1608

Another possibility is a crash due to a NULL pointer deref or GPF somewhere in that driver. This is pretty difficult to debug if there's no possibility to see any dmesg output.

Even if we manage to find the root cause of the lockup, as long as the BIOS isn't fixed, the card reader will always be de-enumerated and re-enumerated on resume and consequently the card will briefly disappear. This will make it impossible to use the card as root filesystem.

Quoted 2 lines

I've been using a SanDisk Ultra 32GB microSDHC UHS-I Card. And yes, it only happens when the card is present.

If you go to sleep and resume without SD card, do you then see in dmesg that the card reader was de-enumerated and re-enumerated? Normally there should be at least a "Card not present" / "Card present" message from pciehp, if it sensed a replaced device. ("Card" refers to PCIe card, not SD card.)

Also, what's the subsystem vendor/device ID as seen with lspci after a successful system sleep transition (successful = without SD card)?

Thanks,

Lukas

#5 lore raw
Quoted 32 lines · Lukas Wunner
Quoted 1 line · Derek John Clark
[   25.686128] rtsx_pci 0000:c2:00.0: pciehp: pdev subsystem vendor ID mismatch

[...]

Quoted 1 line

reg=0x525a10ec expected=0x14af1462

The subsystem vendor ID was 0x1462 (MSI) when the card reader was enumerated on boot and it is 0x10ec (Realtek) on resume.

Likewise the subystem device ID was 0x14af on boot, 0x525a on resume.

It's another bug in the MSI BIOS (in addition to marking the Root Port hotplug-capable): The BIOS neglected to re-initialize the subsystem vendor/device ID registers in config space on resume to the same values that it set on boot. The card reader was likely reset while going through the system sleep transition and its config space thus needs to be re-initialized.

What happens is that pciehp marks the card reader disconnected because it assumes that it was replaced during system sleep. It then synthesizes a Presence Detect Changed event:

https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/pci/hotplug/pciehp_core.c#L300

So the pciehp interrupt thread pciehp_ist() will remove the rtsx_pci driver and de-enumerate the device.

Replacing or removing PCIe devices during system sleep happens all the time (e.g. with Thunderbolt) and we haven't seen issues with regards to that. The likelihood is higher that the lockup is caused by rtsx_pci.

Quoted 8 lines

Maybe rtsx_pci_remove() blocks somewhere and prevents forward progress of the resume transition. In particular, it calls pm_runtime_get_sync() and cancel_delayed_work_sync(), which are blocking calls, i.e. they wait for something else to happen. You could try commenting those calls out and see if it avoids the lockup:

https://elixir.bootlin.com/linux/v7.2-rc6/source/drivers/misc/cardreader/rtsx_pcr.c#L1608

This was good instinct. I was able to drill down and find approximately where it is hanging.

rtsx_pci_remove() ->mfd_remove_devices() -> mfd_remove_devices_fn() -> platform_device_unregister() -> platform_device_del() -> device_del() -> bus_remove_device() -> device_release_driver() -> device_release_driver_internal() -> __device_release_driver() -> device_remove()

In device_remove() (drivers/base/dd.c) it gets past device_remove_groups() and hangs before completing this if block:

if (dev->bus && dev->bus->remove)

    dev->bus->remove(dev);
else if (dev->driver->remove)
    dev->driver->remove(dev);

After this logging stops.

- Derek

Quoted 9 lines

Another possibility is a crash due to a NULL pointer deref or GPF somewhere in that driver. This is pretty difficult to debug if there's no possibility to see any dmesg output.

Even if we manage to find the root cause of the lockup, as long as the BIOS isn't fixed, the card reader will always be de-enumerated and re-enumerated on resume and consequently the card will briefly disappear. This will make it impossible to use the card as root filesystem.

I agree. Even if we were to convince them to fix it, there are plenty of devices in the wild that will likely never get the BIOS update since MSI isn't using LFVS for this device. (I'm not sure if they do that for any device)

Quoted 8 lines
Quoted 2 lines

I've been using a SanDisk Ultra 32GB microSDHC UHS-I Card. And yes, it only happens when the card is present.

If you go to sleep and resume without SD card, do you then see in dmesg that the card reader was de-enumerated and re-enumerated? Normally there should be at least a "Card not present" / "Card present" message from pciehp, if it sensed a replaced device. ("Card" refers to PCIe card, not SD card.)

i get card not present, then card present, then link up.

Quoted 3 lines

Also, what's the subsystem vendor/device ID as seen with lspci after a successful system sleep transition (successful = without SD card)?

it reflects the Realtek device VID/PID. I also tested resume after the 1 cycle with it re-inserted and it works fine after the first cycle, presumably because the device id matches on the second run and beyond

Thanks,
Derek.

Quoted 3 lines

Thanks,

Lukas

#6 lore raw
Quoted 9 lines · Derek John Clark

This was good instinct. I was able to drill down and find approximately where it is hanging.

rtsx_pci_remove() ->mfd_remove_devices() -> mfd_remove_devices_fn() -> platform_device_unregister() -> platform_device_del() -> device_del() -> bus_remove_device() -> device_release_driver() -> device_release_driver_internal() -> __device_release_driver() -> device_remove()

Quoted 7 lines

In device_remove() (drivers/base/dd.c) it gets past device_remove_groups() and hangs before completing this if block:

if (dev->bus && dev->bus->remove)

    dev->bus->remove(dev);
else if (dev->driver->remove)
    dev->driver->remove(dev);

Okay, that ->remove callback should be rtsx_pci_sdmmc_drv_remove() in drivers/mmc/host/rtsx_pci_sdmmc.c. Can you maybe identify where execution stops in that function?

Basically the rtsx_pci driver creates a child device of the PCI device and the mmc host driver binds to that child device. Removing the mmc host driver fails here for some reason.

Quoted 6 lines
Quoted 5 lines

If you go to sleep and resume without SD card, do you then see in dmesg that the card reader was de-enumerated and re-enumerated? Normally there should be at least a "Card not present" / "Card present" message from pciehp, if it sensed a replaced device. ("Card" refers to PCIe card, not SD card.)

Quoted 8 lines

i get card not present, then card present, then link up.

Quoted 2 lines

Also, what's the subsystem vendor/device ID as seen with lspci after a successful system sleep transition (successful = without SD card)?

it reflects the Realtek device VID/PID. I also tested resume after the 1 cycle with it re-inserted and it works fine after the first cycle, presumably because the device id matches on the second run and beyond

Right, when the device is re-enumerated after the first system sleep transition, the Realtek subdevice vendor/device ID is cached and that matches on all subsequent system sleep transitions. So if we manage to find and eliminate the cause of the hang, there'll only be the inconvenience of the removal/re-enumeration on first sleep.

Thanks,

Lukas

#7 lore raw
Quoted 19 lines · Lukas Wunner
Quoted 9 lines · Derek John Clark

This was good instinct. I was able to drill down and find approximately where it is hanging.

rtsx_pci_remove() ->mfd_remove_devices() -> mfd_remove_devices_fn() -> platform_device_unregister() -> platform_device_del() -> device_del() -> bus_remove_device() -> device_release_driver() -> device_release_driver_internal() -> __device_release_driver() -> device_remove()

Quoted 7 lines

In device_remove() (drivers/base/dd.c) it gets past device_remove_groups() and hangs before completing this if block:

if (dev->bus && dev->bus->remove)

    dev->bus->remove(dev);
else if (dev->driver->remove)
    dev->driver->remove(dev);
Quoted 7 lines

Okay, that ->remove callback should be rtsx_pci_sdmmc_drv_remove() in drivers/mmc/host/rtsx_pci_sdmmc.c. Can you maybe identify where execution stops in that function?

Basically the rtsx_pci driver creates a child device of the PCI device and the mmc host driver binds to that child device. Removing the mmc host driver fails here for some reason.

Hi Lukas,

I was able to drill down further. When the mmc device gets to blk_report_disk_dead() in block/genhd.c there is an xa_for_each loop. On the second loop of that it seems to hang in bdev_mark_dead(). That sets a callback that runs fs_bdev_mark_dead() which then runs sync_filesystem(). This is all hit because the "surprise" bool is set to false unconditionally in __del_gendisk().

Commenting out this from __del_gendisk():
if (!test_bit(GD_DEAD, &disk->state))
blk_report_disk_dead(disk, false);

Avoids the hang.

I'm not sure how much further we want to go down into this as we've moved quite far out of the driver space and are now well into the block subsystem. I'm working on a proof of concept that would allow the driver to inform the mmc core that this was a surprise, but I'm not sure how that would land. From what I've read the mmc core was intentionally detached from higher level drivers, though a few notification methods do exist. If I get that fully working I'll post it here but it will probably be a few days as I've run into some other obligations this week.

Quoted 15 lines
Quoted 6 lines
Quoted 5 lines

If you go to sleep and resume without SD card, do you then see in dmesg that the card reader was de-enumerated and re-enumerated? Normally there should be at least a "Card not present" / "Card present" message from pciehp, if it sensed a replaced device. ("Card" refers to PCIe card, not SD card.)

Quoted 8 lines

i get card not present, then card present, then link up.

Quoted 2 lines

Also, what's the subsystem vendor/device ID as seen with lspci after a successful system sleep transition (successful = without SD card)?

it reflects the Realtek device VID/PID. I also tested resume after the 1 cycle with it re-inserted and it works fine after the first cycle, presumably because the device id matches on the second run and beyond

Quoted 5 lines

Right, when the device is re-enumerated after the first system sleep transition, the Realtek subdevice vendor/device ID is cached and that matches on all subsequent system sleep transitions. So if we manage to find and eliminate the cause of the hang, there'll only be the inconvenience of the removal/re-enumeration on first sleep.

I suppose that is true, but in that case we still wouldn't be able to use it as the root fs since it gets removed/re-added during a suspend. Would the quirk I submitted not have this compromise?

Quoted 3 lines

Thanks,

Lukas

#8 lore raw
Quoted 7 lines · Derek John Clark

I was able to drill down further. When the mmc device gets to blk_report_disk_dead() in block/genhd.c there is an xa_for_each loop. On the second loop of that it seems to hang in bdev_mark_dead(). That sets a callback that runs fs_bdev_mark_dead() which then runs sync_filesystem(). This is all hit because the "surprise" bool is set to false unconditionally in __del_gendisk().

Quoted 5 lines

Commenting out this from __del_gendisk():
if (!test_bit(GD_DEAD, &disk->state))
blk_report_disk_dead(disk, false);

Avoids the hang.

Thank you so much, you've root-caused the issue: We're missing a call to blk_mark_disk_dead() at the top of rtsx_pci_sdmmc_drv_remove() if the underlying pci_dev is marked disconnected. Let me get back to you with a fix in a bit.

Quoted 9 lines

I'm not sure how much further we want to go down into this as we've moved quite far out of the driver space and are now well into the block subsystem. I'm working on a proof of concept that would allow the driver to inform the mmc core that this was a surprise, but I'm not sure how that would land. From what I've read the mmc core was intentionally detached from higher level drivers, though a few notification methods do exist. If I get that fully working I'll post it here but it will probably be a few days as I've run into some other obligations this week.

It's fine, you don't need to invest any more time into that and I understand your frustration about how much effort is necessary to fix this.

The issue you've found affects anyone removing e.g. a Thunderbolt- attached dock with an MMC card reader during system sleep. So it's useful and important to have root-caused and fix that. Unfortunately kernel development often means getting sidetracked like this.

Quoted 3 lines

I suppose that is true, but in that case we still wouldn't be able to use it as the root fs since it gets removed/re-added during a suspend. Would the quirk I submitted not have this compromise?

Okay I wasn't sure that using the MMC card as root filesystem is even possible on this product. If that's a potential use case, it needs a separate fix. I'll come up with a proposal for that one as well.

Thanks,

Lukas

#9 lore raw

[cc += Ulf, start of thread is here:
https://lore.kernel.org/all/[email protected]/
]

Quoted 12 lines · Lukas Wunner
Quoted 11 lines · Derek John Clark

I was able to drill down further. When the mmc device gets to blk_report_disk_dead() in block/genhd.c there is an xa_for_each loop. On the second loop of that it seems to hang in bdev_mark_dead(). That sets a callback that runs fs_bdev_mark_dead() which then runs sync_filesystem(). This is all hit because the "surprise" bool is set to false unconditionally in __del_gendisk().

Commenting out this from __del_gendisk():
if (!test_bit(GD_DEAD, &disk->state))
blk_report_disk_dead(disk, false);

Quoted 6 lines
Quoted 1 line

Avoids the hang.

Thank you so much, you've root-caused the issue: We're missing a call to blk_mark_disk_dead() at the top of rtsx_pci_sdmmc_drv_remove() if the underlying pci_dev is marked disconnected. Let me get back to you with a fix in a bit.

So the completely untested patch below might be an upstreamable approach. I'm adding MMC maintainer Ulf to cc in case he has early feedback.

If this works, feel free to submit a proper patch and claim authorship if you want. I'll gladly let you have that given the amount of time you've already sunk into it. If you'd rather have me submit a patch (and deal with any regressions caused by it), I'll be happy to do that as well. Thanks!

-- >8 --

drivers/mmc/core/block.c +7 -4
index 0274e8d..1a82233 100644
@@ -2978,8 +2978,11 @@ static int mmc_blk_alloc_parts(struct mmc_card *card, struct mmc_blk_data *md)
 	return 0;
 }
 
-static void mmc_blk_remove_req(struct mmc_blk_data *md)
+static void mmc_blk_remove_req(struct mmc_card *card, struct mmc_blk_data *md)
 {
+	if (mmc_card_removed(card))
+		blk_mark_disk_dead(md->disk);
+
 	/*
 	 * Flush remaining requests and free queues. It is freeing the queue
 	 * that stops new requests from being accepted.
@@ -3006,7 +3009,7 @@ static void mmc_blk_remove_parts(struct mmc_card *card,
 	list_for_each_safe(pos, q, &md->part) {
 		part_md = list_entry(pos, struct mmc_blk_data, part);
 		list_del(pos);
-		mmc_blk_remove_req(part_md);
+		mmc_blk_remove_req(card, part_md);
 	}
 }
 
@@ -3252,7 +3255,7 @@ static int mmc_blk_probe(struct mmc_card *card)
 
 out:
 	mmc_blk_remove_parts(card, md);
-	mmc_blk_remove_req(md);
+	mmc_blk_remove_req(card, md);
 out_free:
 	destroy_workqueue(card->complete_wq);
 	return ret;
@@ -3273,7 +3276,7 @@ static void mmc_blk_remove(struct mmc_card *card)
 	if (!mmc_card_sd_combo(card))
 		pm_runtime_disable(&card->dev);
 	pm_runtime_put_noidle(&card->dev);
-	mmc_blk_remove_req(md);
+	mmc_blk_remove_req(card, md);
 	destroy_workqueue(card->complete_wq);
 }
 
drivers/mmc/core/host.c +14 -0
index b7ce313..658b241 100644
@@ -703,3 +703,17 @@ void mmc_free_host(struct mmc_host *host)
 }
 
 EXPORT_SYMBOL(mmc_free_host);
+
+/**
+ *	mmc_host_set_removed - declare host removed
+ *	@host: mmc host
+ *
+ *	Declare the host (and any inserted card) removed and inaccessible.
+ */
+void mmc_host_set_removed(struct mmc_host *host)
+{
+	if (host->card)
+		mmc_card_set_removed(host->card);
+}
+
+EXPORT_SYMBOL(mmc_host_set_removed);
drivers/mmc/host/rtsx_pci_sdmmc.c +3 -0
index 8dfbc62..3f97659f 100644
@@ -1511,6 +1511,9 @@ static void rtsx_pci_sdmmc_drv_remove(struct platform_device *pdev)
 	pcr->slots[RTSX_SD_CARD].card_event = NULL;
 	mmc = host->mmc;
 
+	if (pci_dev_is_disconnected(pcr->pci))
+		mmc_host_set_removed(mmc);
+
 	cancel_work_sync(&host->work);
 
 	mutex_lock(&host->host_mutex);
include/linux/mmc/host.h +1 -0
index ba84f02..a240306 100644
@@ -588,6 +588,7 @@ struct mmc_host {
 int mmc_add_host(struct mmc_host *);
 void mmc_remove_host(struct mmc_host *);
 void mmc_free_host(struct mmc_host *);
+void mmc_host_set_removed(struct mmc_host *host);
 void mmc_of_parse_clk_phase(struct device *dev,
 			    struct mmc_clk_phase_map *map);
 int mmc_of_parse(struct mmc_host *host);
#10 lore raw
Quoted 19 lines · Lukas Wunner

[cc += Ulf, start of thread is here:
https://lore.kernel.org/all/[email protected]/
]

Quoted 14 lines · Lukas Wunner
Quoted 12 lines · Derek John Clark

I was able to drill down further. When the mmc device gets to blk_report_disk_dead() in block/genhd.c there is an xa_for_each loop. On the second loop of that it seems to hang in bdev_mark_dead(). That sets a callback that runs fs_bdev_mark_dead() which then runs sync_filesystem(). This is all hit because the "surprise" bool is set to false unconditionally in __del_gendisk().

Commenting out this from __del_gendisk():
if (!test_bit(GD_DEAD, &disk->state))
blk_report_disk_dead(disk, false);

Avoids the hang.

Quoted 8 lines
Quoted 4 lines

Thank you so much, you've root-caused the issue: We're missing a call to blk_mark_disk_dead() at the top of rtsx_pci_sdmmc_drv_remove() if the underlying pci_dev is marked disconnected. Let me get back to you with a fix in a bit.

So the completely untested patch below might be an upstreamable approach. I'm adding MMC maintainer Ulf to cc in case he has early feedback.

Great, I'll give it a test soon and follow up.

Quoted 6 lines

If this works, feel free to submit a proper patch and claim authorship if you want. I'll gladly let you have that given the amount of time you've already sunk into it. If you'd rather have me submit a patch (and deal with any regressions caused by it), I'll be happy to do that as well. Thanks!

I think tested by tags will be sufficient once I've completed that. The shape of your patch is a bit different than what I was working on and is a better approach. I appreciate your help on this so far.

Thanks,
Derek

Quoted 94 lines

-- >8 --

drivers/mmc/core/block.c +5 -2
index 0274e8d..1a82233 100644
@@ -2978,8 +2978,11 @@ static int mmc_blk_alloc_parts(struct mmc_card *card, struct mmc_blk_data *md)
 	return 0;
 }
 
-static void mmc_blk_remove_req(struct mmc_blk_data *md)
+static void mmc_blk_remove_req(struct mmc_card *card, struct mmc_blk_data *md)
 {
+	if (mmc_card_removed(card))
+		blk_mark_disk_dead(md->disk);
+
 	/*
 	 * Flush remaining requests and free queues. It is freeing the queue
 	 * that stops new requests from being accepted.
@@ -3006,7 +3009,7 @@ static void mmc_blk_remove_parts(struct mmc_card *card,
 	list_for_each_safe(pos, q, &md->part) {
 		part_md = list_entry(pos, struct mmc_blk_data, part);
 		list_del(pos);
-		mmc_blk_remove_req(part_md);
+		mmc_blk_remove_req(card, part_md);
 	}
 }
 
@@ -3252,7 +3255,7 @@ static int mmc_blk_probe(struct mmc_card *card)
 

out:

	mmc_blk_remove_parts(card, md);
drivers/mmc/core/block.c +1 -1
-	mmc_blk_remove_req(md);
+	mmc_blk_remove_req(card, md);

out_free:

	destroy_workqueue(card->complete_wq);
	return ret;
drivers/mmc/core/block.c +1 -1
@@ -3273,7 +3276,7 @@ static void mmc_blk_remove(struct mmc_card *card)
 	if (!mmc_card_sd_combo(card))
 		pm_runtime_disable(&card->dev);
 	pm_runtime_put_noidle(&card->dev);
-	mmc_blk_remove_req(md);
+	mmc_blk_remove_req(card, md);
 	destroy_workqueue(card->complete_wq);
 }
 
drivers/mmc/core/host.c +14 -0
index b7ce313..658b241 100644
@@ -703,3 +703,17 @@ void mmc_free_host(struct mmc_host *host)
 }
 
 EXPORT_SYMBOL(mmc_free_host);
+
+/**
+ *	mmc_host_set_removed - declare host removed
+ *	@host: mmc host
+ *
+ *	Declare the host (and any inserted card) removed and inaccessible.
+ */
+void mmc_host_set_removed(struct mmc_host *host)
+{
+	if (host->card)
+		mmc_card_set_removed(host->card);
+}
+
+EXPORT_SYMBOL(mmc_host_set_removed);
drivers/mmc/host/rtsx_pci_sdmmc.c +3 -0
index 8dfbc62..3f97659f 100644
@@ -1511,6 +1511,9 @@ static void rtsx_pci_sdmmc_drv_remove(struct platform_device *pdev)
 	pcr->slots[RTSX_SD_CARD].card_event = NULL;
 	mmc = host->mmc;
 
+	if (pci_dev_is_disconnected(pcr->pci))
+		mmc_host_set_removed(mmc);
+
 	cancel_work_sync(&host->work);
 
 	mutex_lock(&host->host_mutex);
include/linux/mmc/host.h +1 -0
index ba84f02..a240306 100644
@@ -588,6 +588,7 @@ struct mmc_host {
 int mmc_add_host(struct mmc_host *);
 void mmc_remove_host(struct mmc_host *);
 void mmc_free_host(struct mmc_host *);
+void mmc_host_set_removed(struct mmc_host *host);
 void mmc_of_parse_clk_phase(struct device *dev,
 			    struct mmc_clk_phase_map *map);
 int mmc_of_parse(struct mmc_host *host);
#11 lore raw
In reply to Lukas Wunner
Quoted 1 line · Lukas Wunner

So the completely untested patch below might be an upstreamable approach.

Sorry, I was OOO for a day and sent out the patch without compile-testing right before hitting the road. I'm back now and it turns out that an #include statement was mising. Rectified patch below.

-- >8 --

drivers/mmc/core/block.c +7 -4
index 0274e8d07660..1a82233a30d0 100644
@@ -2978,8 +2978,11 @@ static int mmc_blk_alloc_parts(struct mmc_card *card, struct mmc_blk_data *md)
 	return 0;
 }
 
-static void mmc_blk_remove_req(struct mmc_blk_data *md)
+static void mmc_blk_remove_req(struct mmc_card *card, struct mmc_blk_data *md)
 {
+	if (mmc_card_removed(card))
+		blk_mark_disk_dead(md->disk);
+
 	/*
 	 * Flush remaining requests and free queues. It is freeing the queue
 	 * that stops new requests from being accepted.
@@ -3006,7 +3009,7 @@ static void mmc_blk_remove_parts(struct mmc_card *card,
 	list_for_each_safe(pos, q, &md->part) {
 		part_md = list_entry(pos, struct mmc_blk_data, part);
 		list_del(pos);
-		mmc_blk_remove_req(part_md);
+		mmc_blk_remove_req(card, part_md);
 	}
 }
 
@@ -3252,7 +3255,7 @@ static int mmc_blk_probe(struct mmc_card *card)
 
 out:
 	mmc_blk_remove_parts(card, md);
-	mmc_blk_remove_req(md);
+	mmc_blk_remove_req(card, md);
 out_free:
 	destroy_workqueue(card->complete_wq);
 	return ret;
@@ -3273,7 +3276,7 @@ static void mmc_blk_remove(struct mmc_card *card)
 	if (!mmc_card_sd_combo(card))
 		pm_runtime_disable(&card->dev);
 	pm_runtime_put_noidle(&card->dev);
-	mmc_blk_remove_req(md);
+	mmc_blk_remove_req(card, md);
 	destroy_workqueue(card->complete_wq);
 }
 
drivers/mmc/core/host.c +15 -0
index b7ce3137d452..2625d918081b 100644
@@ -22,6 +22,7 @@
 #include <linux/mmc/card.h>
 #include <linux/mmc/slot-gpio.h>
 
+#include "card.h"
 #include "core.h"
 #include "crypto.h"
 #include "host.h"
@@ -703,3 +704,17 @@ void mmc_free_host(struct mmc_host *host)
 }
 
 EXPORT_SYMBOL(mmc_free_host);
+
+/**
+ *	mmc_host_set_removed - declare host removed
+ *	@host: mmc host
+ *
+ *	Declare the host (and any inserted card) removed and inaccessible.
+ */
+void mmc_host_set_removed(struct mmc_host *host)
+{
+	if (host->card)
+		mmc_card_set_removed(host->card);
+}
+
+EXPORT_SYMBOL(mmc_host_set_removed);
drivers/mmc/host/rtsx_pci_sdmmc.c +3 -0
index 8dfbc62f165b..3f97659fba7c 100644
@@ -1511,6 +1511,9 @@ static void rtsx_pci_sdmmc_drv_remove(struct platform_device *pdev)
 	pcr->slots[RTSX_SD_CARD].card_event = NULL;
 	mmc = host->mmc;
 
+	if (pci_dev_is_disconnected(pcr->pci))
+		mmc_host_set_removed(mmc);
+
 	cancel_work_sync(&host->work);
 
 	mutex_lock(&host->host_mutex);
include/linux/mmc/host.h +1 -0
index ba84f02c2a10..a2403062d770 100644
@@ -588,6 +588,7 @@ struct mmc_host *devm_mmc_alloc_host(struct device *dev, int extra);
 int mmc_add_host(struct mmc_host *);
 void mmc_remove_host(struct mmc_host *);
 void mmc_free_host(struct mmc_host *);
+void mmc_host_set_removed(struct mmc_host *host);
 void mmc_of_parse_clk_phase(struct device *dev,
 			    struct mmc_clk_phase_map *map);
 int mmc_of_parse(struct mmc_host *host);
author Tested-by
#12 lore raw
Quoted 8 lines · Lukas Wunner
Quoted 1 line · Lukas Wunner

So the completely untested patch below might be an upstreamable approach.

Sorry, I was OOO for a day and sent out the patch without compile-testing right before hitting the road. I'm back now and it turns out that an #include statement was mising. Rectified patch below.

No problem. Thanks for catching it after. I only got around to testing today. It works well.

Thanks,
Derek

Quoted 102 lines

-- >8 --

drivers/mmc/core/block.c +7 -4
index 0274e8d07660..1a82233a30d0 100644
@@ -2978,8 +2978,11 @@ static int mmc_blk_alloc_parts(struct mmc_card *card, struct mmc_blk_data *md)
        return 0;
 }
 
-static void mmc_blk_remove_req(struct mmc_blk_data *md)
+static void mmc_blk_remove_req(struct mmc_card *card, struct mmc_blk_data *md)
 {
+       if (mmc_card_removed(card))
+               blk_mark_disk_dead(md->disk);
+
        /*
         * Flush remaining requests and free queues. It is freeing the queue
         * that stops new requests from being accepted.
@@ -3006,7 +3009,7 @@ static void mmc_blk_remove_parts(struct mmc_card *card,
        list_for_each_safe(pos, q, &md->part) {
                part_md = list_entry(pos, struct mmc_blk_data, part);
                list_del(pos);
-               mmc_blk_remove_req(part_md);
+               mmc_blk_remove_req(card, part_md);
        }
 }
 
@@ -3252,7 +3255,7 @@ static int mmc_blk_probe(struct mmc_card *card)
 
 out:
        mmc_blk_remove_parts(card, md);
-       mmc_blk_remove_req(md);
+       mmc_blk_remove_req(card, md);
 out_free:
        destroy_workqueue(card->complete_wq);
        return ret;
@@ -3273,7 +3276,7 @@ static void mmc_blk_remove(struct mmc_card *card)
        if (!mmc_card_sd_combo(card))
                pm_runtime_disable(&card->dev);
        pm_runtime_put_noidle(&card->dev);
-       mmc_blk_remove_req(md);
+       mmc_blk_remove_req(card, md);
        destroy_workqueue(card->complete_wq);
 }
 
drivers/mmc/core/host.c +15 -0
index b7ce3137d452..2625d918081b 100644
@@ -22,6 +22,7 @@
 #include <linux/mmc/card.h>
 #include <linux/mmc/slot-gpio.h>
 
+#include "card.h"
 #include "core.h"
 #include "crypto.h"
 #include "host.h"
@@ -703,3 +704,17 @@ void mmc_free_host(struct mmc_host *host)
 }
 
 EXPORT_SYMBOL(mmc_free_host);
+
+/**
+ *     mmc_host_set_removed - declare host removed
+ *     @host: mmc host
+ *
+ *     Declare the host (and any inserted card) removed and inaccessible.
+ */
+void mmc_host_set_removed(struct mmc_host *host)
+{
+       if (host->card)
+               mmc_card_set_removed(host->card);
+}
+
+EXPORT_SYMBOL(mmc_host_set_removed);
drivers/mmc/host/rtsx_pci_sdmmc.c +3 -0
index 8dfbc62f165b..3f97659fba7c 100644
@@ -1511,6 +1511,9 @@ static void rtsx_pci_sdmmc_drv_remove(struct platform_device *pdev)
        pcr->slots[RTSX_SD_CARD].card_event = NULL;
        mmc = host->mmc;
 
+       if (pci_dev_is_disconnected(pcr->pci))
+               mmc_host_set_removed(mmc);
+
        cancel_work_sync(&host->work);
 
        mutex_lock(&host->host_mutex);
include/linux/mmc/host.h +1 -0
index ba84f02c2a10..a2403062d770 100644
@@ -588,6 +588,7 @@ struct mmc_host *devm_mmc_alloc_host(struct device *dev, int extra);
 int mmc_add_host(struct mmc_host *);
 void mmc_remove_host(struct mmc_host *);
 void mmc_free_host(struct mmc_host *);
+void mmc_host_set_removed(struct mmc_host *host);
 void mmc_of_parse_clk_phase(struct device *dev,
                            struct mmc_clk_phase_map *map);
 int mmc_of_parse(struct mmc_host *host);
#13 lore raw
In reply to Lukas Wunner
Quoted 9 lines · Lukas Wunner
Quoted 3 lines · Derek John Clark

I suppose that is true, but in that case we still wouldn't be able to use it as the root fs since it gets removed/re-added during a suspend. Would the quirk I submitted not have this compromise?

Okay I wasn't sure that using the MMC card as root filesystem is even possible on this product. If that's a potential use case, it needs a separate fix. I'll come up with a proposal for that one as well.

Here's my proposal for allowing root on an MMC card.

Both this and the other patch I just sent (or something like it) need to be submitted upstream. The other one is necessary so others don't see a lockup when disconnecting e.g. a Thunderbolt-attached card reader during system sleep.

But either one of the patches should make the issue go away on the MSI Claw, so each patch needs to be tested by itself, without applying the other one as well. Thanks!

-- >8 --

arch/x86/pci/fixup.c +16 -0
index b301c6c8df75..9da7ae0b2437 100644
@@ -1080,4 +1080,20 @@ static void quirk_tuxeo_rp_d3(struct pci_dev *pdev)
 	}
 }
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1502, quirk_tuxeo_rp_d3);
+
+/*
+ * MSI Claw A8 firmware initializes the onboard Realtek RTS525A cardreader
+ * with an MSI-specific Subsystem Vendor and Device ID on boot, but neglects
+ * to re-initialize it on resume.  Avoid hot-removal of the cardreader due to
+ * the spurious ID change.
+ */
+static void quirk_msi_claw_cardreader(struct pci_dev *pdev)
+{
+	if (pdev->subsystem_vendor == 0x1462 &&
+	    pdev->subsystem_device == 0x14af) {
+		pdev->subsystem_vendor = pdev->vendor;
+		pdev->subsystem_device = pdev->device;
+	}
+}
+DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_REALTEK, 0x525a, quirk_msi_claw_cardreader);
 #endif /* CONFIG_SUSPEND */
author Tested-by
#14 lore raw
Quoted 19 lines · Lukas Wunner
Quoted 9 lines · Lukas Wunner
Quoted 3 lines · Derek John Clark

I suppose that is true, but in that case we still wouldn't be able to use it as the root fs since it gets removed/re-added during a suspend. Would the quirk I submitted not have this compromise?

Okay I wasn't sure that using the MMC card as root filesystem is even possible on this product. If that's a potential use case, it needs a separate fix. I'll come up with a proposal for that one as well.

Here's my proposal for allowing root on an MMC card.

Both this and the other patch I just sent (or something like it) need to be submitted upstream. The other one is necessary so others don't see a lockup when disconnecting e.g. a Thunderbolt-attached card reader during system sleep.

Quoted 5 lines

But either one of the patches should make the issue go away on the MSI Claw, so each patch needs to be tested by itself, without applying the other one as well. Thanks!

-- >8 --

This also works independently of the previous fix.

- Derek

Quoted 25 lines
arch/x86/pci/fixup.c +16 -0
index b301c6c8df75..9da7ae0b2437 100644
@@ -1080,4 +1080,20 @@ static void quirk_tuxeo_rp_d3(struct pci_dev *pdev)
        }
 }
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1502, quirk_tuxeo_rp_d3);
+
+/*
+ * MSI Claw A8 firmware initializes the onboard Realtek RTS525A cardreader
+ * with an MSI-specific Subsystem Vendor and Device ID on boot, but neglects
+ * to re-initialize it on resume.  Avoid hot-removal of the cardreader due to
+ * the spurious ID change.
+ */
+static void quirk_msi_claw_cardreader(struct pci_dev *pdev)
+{
+       if (pdev->subsystem_vendor == 0x1462 &&
+           pdev->subsystem_device == 0x14af) {
+               pdev->subsystem_vendor = pdev->vendor;
+               pdev->subsystem_device = pdev->device;
+       }
+}
+DECLARE_PCI_FIXUP_SUSPEND_LATE(PCI_VENDOR_ID_REALTEK, 0x525a, quirk_msi_claw_cardreader);
 #endif /* CONFIG_SUSPEND */