PCI: Enable Bus Master in pci_power_up()
PATCH 1/1 PCI: Enable Bus Master in pci_power_up()
commit 4d4c10f763d78 ("PCI: Explicitly put devices into D0 when initializing") addressed the issue of devices not being explicitly initialized to D0 during system startup, resolving mismatches between firmware and OS states.
However, this change affected devices lacking runtime PM, as noted in commit 907a7a2e5bf40 ("PCI/PM: Set up runtime PM even for devices without PCI PM").
Matthew however reports that there is additional problems specifically on AWS NVME hardware that can't handle a kexec since these changes were introduced.
During a kexec reboot ever since commit 4fc9bbf98fd66 ("PCI: Disable Bus Master only on kexec reboot") bus mastering will be turned off, and this is a different flow than is observed for shutdown/reboot. The problem appears to be that because the device is actually in D0 during the startup routine, clearing bus mastering as part of pci_device_shutdown() leads to a mismatch during the next kernel boot.
I'd hypothesize that the firmware on this platform normally sets bus mastering as part of startup and the difference in kexec behavior lead to an incongruity.
Set bus mastering when the device powers up to fix the mismatch.
---
NOTE:
This could also be addressed by disabling the clearing of bus mastering across a kexec reboot if that is preferred.
--- drivers/pci/pci.c | 1 + 1 file changed, 1 insertion(+)
drivers/pci/pci.c +1 -0
@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)return -EIO ; } +pci_set_master (dev);pci_read_config_word (dev, dev->pm_cap +PCI_PM_CTRL , &pmcsr);if (PCI_POSSIBLE_ERROR (pmcsr)) { pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",
Signature · 1 line
2.43.0
Quoted 53 lines · Mario Limonciello (AMD)
commit
4d4c10f763d78("PCI: Explicitly put devices into D0 when initializing") addressed the issue of devices not being explicitly initialized to D0 during system startup, resolving mismatches between firmware and OS states.However, this change affected devices lacking runtime PM, as noted in commit
907a7a2e5bf40("PCI/PM: Set up runtime PM even for devices without PCI PM").Matthew however reports that there is additional problems specifically on AWS NVME hardware that can't handle a kexec since these changes were introduced.
During a kexec reboot ever since commit
4fc9bbf98fd66("PCI: Disable Bus Master only on kexec reboot") bus mastering will be turned off, and this is a different flow than is observed for shutdown/reboot. The problem appears to be that because the device is actually in D0 during the startup routine, clearing bus mastering as part ofpci_device_shutdown()leads to a mismatch during the next kernel boot.I'd hypothesize that the firmware on this platform normally sets bus mastering as part of startup and the difference in kexec behavior lead to an incongruity.
Set bus mastering when the device powers up to fix the mismatch.
Reported-by Matthew Ruffell <[email protected]>Closes https://lore.kernel.org/linux-pci/CAKAwkKvmdKxRRA4cR=jJEdyadon6uKXe+aFXaGSe=PNSgwDf9g@mail.gmail.com/ open hereFixes 4d4c10f763d78 ("PCI: Explicitly put devices into D0 when initializing")Signed-off-by Mario Limonciello (AMD) <[email protected]>---
NOTE:
This could also be addressed by disabling the clearing of bus mastering across a kexec reboot if that is preferred.--- drivers/pci/pci.c | 1 + 1 file changed, 1 insertion(+)drivers/pci/pci.c +1
-1@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)return -EIO ; } +pci_set_master (dev);pci_read_config_word (dev, dev->pm_cap +PCI_PM_CTRL , &pmcsr);if (PCI_POSSIBLE_ERROR (pmcsr)) { pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n", --
Quoted 2 lines
2.43.0
Tested on a AWS c5.metal system, with git-tag v6.19-rc5 and this patch applied. Kexec works great, and the system comes up properly.
Thank you Mario.
drivers/pci/pci.c int pci_power_up(struct pci_dev *dev) · line 1329
Quoted 9 lines · Mario Limonciello (AMD)
+++
b/drivers/pci/pci.cdrivers/pci/pci.c +1
-0@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)return -EIO ; } +pci_set_master (dev);pci_read_config_word (dev, dev->pm_cap +PCI_PM_CTRL , &pmcsr);if (PCI_POSSIBLE_ERROR (pmcsr)) { pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",
So any device will be allowed to write to memory from the get-go? That sounds like a very bad idea. For security reasons alone, we only want to enable bus mastering when needed. It's up to the driver to enable it, not up to the PCI core. We've had cases in the past where devices corrupted memory because BIOS left bus mastering enabled, see abb2bafd295f. Enabling bus mastering for everything anytime will exacerbate such problems or uncover new ones.
Thanks,
Lukas
Quoted 6 lines · Lukas Wunner
Quoted 5 lines · Mario Limonciello (AMD)
+++
b/drivers/pci/pci.cdrivers/pci/pci.c +0
-0@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)return -EIO ; }
Quoted 14 lines
drivers/pci/pci.c int pci_power_up(struct pci_dev *dev) · line 1329
Quoted 4 lines
drivers/pci/pci.c +1
-0+pci_set_master (dev);pci_read_config_word (dev, dev->pm_cap +PCI_PM_CTRL , &pmcsr);if (PCI_POSSIBLE_ERROR (pmcsr)) { pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",So any device will be allowed to write to memory from the get-go? That sounds like a very bad idea. For security reasons alone, we only want to enable bus mastering when needed. It's up to the driver to enable it, not up to the PCI core. We've had cases in the past where devices corrupted memory because BIOS left bus mastering enabled, see
abb2bafd295f. Enabling bus mastering for everything anytime will exacerbate such problems or uncover new ones.
Indeed and it will pave the way for a big security hole as the PCI core will then allow the device to perform DMA without setting up the address translation and so on.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Quoted 6 lines · Lukas Wunner
Quoted 5 lines · Mario Limonciello (AMD)
+++
b/drivers/pci/pci.c(quoted diff) +0
-0@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev) return -EIO; }
Quoted 6 lines
Review int pci_power_up(struct pci_dev *dev) · line 1329
Quoted 4 lines
(quoted diff) +1
-0+ pci_set_master(dev); pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr); if (PCI_POSSIBLE_ERROR(pmcsr)) { pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",So any device will be allowed to write to memory from the get-go?
Well so I did a quick check on a modern production Strix laptop with 6.19-rc5 on my desk with this:
drivers/pci/pci.c +5 -0
@@ -1305,6 +1305,7 @@ int pci_power_up(struct pci_dev *dev)bool need_restore;pci_power_t state;u16 pmcsr; +u16 old_cmd;platform_pci_set_power_state (dev,PCI_D0 );
@@ -1352,6 +1353,10 @@ int pci_power_up(struct pci_dev *dev)udelay (PCI_PM_D2_DELAY ); end: +pci_read_config_word (dev,PCI_COMMAND , &old_cmd); +pci_info (dev,"Bus mastering bit is %sabled in D0\n" , + (old_cmd &PCI_COMMAND_MASTER ) ?"en" :"dis" ); + dev->current_state =PCI_D0 ;if (need_restore)return 1 ;
Here's what I observe.
$ sudo dmesg | grep mastering [ 2.560916] pci 0000:00:01.1: Bus mastering bit is enabled in D0 [ 2.580832] pci 0000:00:01.2: Bus mastering bit is enabled in D0 [ 2.594468] pci 0000:00:02.1: Bus mastering bit is enabled in D0 [ 2.595439] pci 0000:00:02.2: Bus mastering bit is enabled in D0 [ 2.596292] pci 0000:00:02.3: Bus mastering bit is enabled in D0 [ 2.597235] pci 0000:00:02.4: Bus mastering bit is enabled in D0 [ 2.600120] pci 0000:00:08.1: Bus mastering bit is enabled in D0 [ 2.600899] pci 0000:00:08.2: Bus mastering bit is enabled in D0 [ 2.601535] pci 0000:00:08.3: Bus mastering bit is enabled in D0 [ 2.610333] pci 0000:c1:00.0: Bus mastering bit is enabled in D0 [ 2.611717] pci 0000:c2:00.0: Bus mastering bit is disabled in D0 [ 2.612795] pci 0000:c3:00.0: Bus mastering bit is disabled in D0 [ 2.613959] pci 0000:c4:00.0: Bus mastering bit is enabled in D0 [ 2.621622] pci 0000:c5:00.0: Bus mastering bit is enabled in D0 [ 2.624022] pci 0000:c5:00.1: Bus mastering bit is disabled in D0 [ 2.624697] pci 0000:c5:00.2: Bus mastering bit is disabled in D0 [ 2.629270] pci 0000:c5:00.4: Bus mastering bit is enabled in D0 [ 2.634864] pci 0000:c5:00.5: Bus mastering bit is disabled in D0 [ 2.635657] pci 0000:c5:00.7: Bus mastering bit is disabled in D0 [ 2.636505] pci 0000:c6:00.0: Bus mastering bit is disabled in D0 [ 2.636859] pci 0000:c6:00.1: Bus mastering bit is disabled in D0 [ 2.641870] pci 0000:c7:00.0: Bus mastering bit is enabled in D0 [ 2.649681] pci 0000:c7:00.3: Bus mastering bit is enabled in D0 [ 2.657457] pci 0000:c7:00.4: Bus mastering bit is enabled in D0 [ 2.665211] pci 0000:c7:00.5: Bus mastering bit is enabled in D0 [ 2.669811] pci 0000:c7:00.6: Bus mastering bit is enabled in D0 [ 5.114596] pcieport 0000:00:01.1: Bus mastering bit is enabled in D0 [ 5.170824] pcieport 0000:00:01.2: Bus mastering bit is enabled in D0 [ 8.025850] snd_hda_intel 0000:c5:00.1: Bus mastering bit is disabled in D0 [ 22.314165] pcieport 0000:00:02.4: Bus mastering bit is enabled in D0 [ 22.330193] r8169 0000:c4:00.0: Bus mastering bit is enabled in D0 [ 22.336309] xhci_hcd 0000:c7:00.0: Bus mastering bit is disabled in D0 [ 34.587505] snd_hda_intel 0000:c5:00.1: Bus mastering bit is enabled in D0 [ 35.771050] xhci_hcd 0000:c5:00.4: Bus mastering bit is disabled in D0
So doesn't BIOS appear to have set bus mastering on a majority of devices already?
Quoted 2 lines
That sounds like a very bad idea. For security reasons alone, we only want to enable bus mastering when needed.
Should actually be doing the reverse of my proposed patch and explicitly disabling bus mastering in the PCI core at startup then require drivers to set policy?
Quoted 7 lines
It's up to
the driver to enable it, not up to the PCI core. We've had cases in the past where devices corrupted memory because BIOS left bus mastering enabled, seeabb2bafd295f. Enabling bus mastering for everything anytime will exacerbate such problems or uncover new ones.
How do you feel about the other proposal I mentioned, not clearing it on kexec? Matthew confirmed this will help this issue too.