Skip to contentSkip to conversation

PCI: Enable Bus Master in pci_power_up()

PATCH In review Mario Limonciello (AMD) 4 participants 5 messages Jan 13 – Jan 14 PCI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: Enable Bus Master in pci_power_up()

commit 4d4c10f763d78 ("PCI: Explicitly put devices into D0 when initializing") addressed the issue of devices not being explicitly initialized to D0 during system startup, resolving mismatches between firmware and OS states.

However, this change affected devices lacking runtime PM, as noted in commit 907a7a2e5bf40 ("PCI/PM: Set up runtime PM even for devices without PCI PM").

Matthew however reports that there is additional problems specifically on AWS NVME hardware that can't handle a kexec since these changes were introduced.

During a kexec reboot ever since commit 4fc9bbf98fd66 ("PCI: Disable Bus Master only on kexec reboot") bus mastering will be turned off, and this is a different flow than is observed for shutdown/reboot. The problem appears to be that because the device is actually in D0 during the startup routine, clearing bus mastering as part of pci_device_shutdown() leads to a mismatch during the next kernel boot.

I'd hypothesize that the firmware on this platform normally sets bus mastering as part of startup and the difference in kexec behavior lead to an incongruity.

Set bus mastering when the device powers up to fix the mismatch.

---
NOTE:
This could also be addressed by disabling the clearing of bus mastering across a kexec reboot if that is preferred.

---
 drivers/pci/pci.c | 1 +
 1 file changed, 1 insertion(+)
drivers/pci/pci.c +1 -0
index 13dbb405dc31f..c0c0b5c9bf838 100644
@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)
 		return -EIO;
 	}
 
+	pci_set_master(dev);
 	pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);
 	if (PCI_POSSIBLE_ERROR(pmcsr)) {
 		pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",
Signature · 1 line
2.43.0
Tested-by on v1 1/1
#1 lore raw
Quoted 53 lines · Mario Limonciello (AMD)

commit 4d4c10f763d78 ("PCI: Explicitly put devices into D0 when initializing") addressed the issue of devices not being explicitly initialized to D0 during system startup, resolving mismatches between firmware and OS states.

However, this change affected devices lacking runtime PM, as noted in commit 907a7a2e5bf40 ("PCI/PM: Set up runtime PM even for devices without PCI PM").

Matthew however reports that there is additional problems specifically on AWS NVME hardware that can't handle a kexec since these changes were introduced.

During a kexec reboot ever since commit 4fc9bbf98fd66 ("PCI: Disable Bus Master only on kexec reboot") bus mastering will be turned off, and this is a different flow than is observed for shutdown/reboot. The problem appears to be that because the device is actually in D0 during the startup routine, clearing bus mastering as part of pci_device_shutdown() leads to a mismatch during the next kernel boot.

I'd hypothesize that the firmware on this platform normally sets bus mastering as part of startup and the difference in kexec behavior lead to an incongruity.

Set bus mastering when the device powers up to fix the mismatch.

---
NOTE:
This could also be addressed by disabling the clearing of bus mastering across a kexec reboot if that is preferred.

---
 drivers/pci/pci.c | 1 +
 1 file changed, 1 insertion(+)
drivers/pci/pci.c +1 -1
index 13dbb405dc31f..c0c0b5c9bf838 100644
@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)
                return -EIO;
        }
 
+       pci_set_master(dev);
        pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);
        if (PCI_POSSIBLE_ERROR(pmcsr)) {
                pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",
--
Quoted 2 lines

2.43.0

Tested on a AWS c5.metal system, with git-tag v6.19-rc5 and this patch applied. Kexec works great, and the system comes up properly.

Thank you Mario.

#2 lore raw
In reply to Mario Limonciello (AMD)

drivers/pci/pci.c int pci_power_up(struct pci_dev *dev) · line 1329

Quoted 9 lines · Mario Limonciello (AMD)

+++ b/drivers/pci/pci.c

drivers/pci/pci.c +1 -0
@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)
 		return -EIO;
 	}
 
+	pci_set_master(dev);
 	pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);
 	if (PCI_POSSIBLE_ERROR(pmcsr)) {
 		pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",

So any device will be allowed to write to memory from the get-go? That sounds like a very bad idea. For security reasons alone, we only want to enable bus mastering when needed. It's up to the driver to enable it, not up to the PCI core. We've had cases in the past where devices corrupted memory because BIOS left bus mastering enabled, see abb2bafd295f. Enabling bus mastering for everything anytime will exacerbate such problems or uncover new ones.

Thanks,

Lukas

#3 lore raw
Quoted 6 lines · Lukas Wunner
Quoted 5 lines · Mario Limonciello (AMD)

+++ b/drivers/pci/pci.c

drivers/pci/pci.c +0 -0
@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)
 		return -EIO;
 	}
 
Quoted 14 lines

drivers/pci/pci.c int pci_power_up(struct pci_dev *dev) · line 1329

Quoted 4 lines
drivers/pci/pci.c +1 -0
+	pci_set_master(dev);
 	pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);
 	if (PCI_POSSIBLE_ERROR(pmcsr)) {
 		pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",

So any device will be allowed to write to memory from the get-go? That sounds like a very bad idea. For security reasons alone, we only want to enable bus mastering when needed. It's up to the driver to enable it, not up to the PCI core. We've had cases in the past where devices corrupted memory because BIOS left bus mastering enabled, see abb2bafd295f. Enabling bus mastering for everything anytime will exacerbate such problems or uncover new ones.

Indeed and it will pave the way for a big security hole as the PCI core will then allow the device to perform DMA without setting up the address translation and so on.

- Mani

Signature · 1 line
மணிவண்ணன் சதாசிவம்
#4 lore raw
In reply to Lukas Wunner
Quoted 6 lines · Lukas Wunner
Quoted 5 lines · Mario Limonciello (AMD)

+++ b/drivers/pci/pci.c

(quoted diff) +0 -0
@@ -1323,6 +1323,7 @@ int pci_power_up(struct pci_dev *dev)
  		return -EIO;
  	}
 
Quoted 6 lines

Review int pci_power_up(struct pci_dev *dev) · line 1329

Quoted 4 lines
(quoted diff) +1 -0
+	pci_set_master(dev);
  	pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);
  	if (PCI_POSSIBLE_ERROR(pmcsr)) {
  		pci_err(dev, "Unable to change power state from %s to D0, device inaccessible\n",

So any device will be allowed to write to memory from the get-go?

Well so I did a quick check on a modern production Strix laptop with 6.19-rc5 on my desk with this:

drivers/pci/pci.c +5 -0
index 13dbb405dc31..74d7745c185c 100644
@@ -1305,6 +1305,7 @@ int pci_power_up(struct pci_dev *dev)
        bool need_restore;
        pci_power_t state;
        u16 pmcsr;
+       u16 old_cmd;
 
        platform_pci_set_power_state(dev, PCI_D0);
 
@@ -1352,6 +1353,10 @@ int pci_power_up(struct pci_dev *dev)
                udelay(PCI_PM_D2_DELAY);
 
 end:
+       pci_read_config_word(dev, PCI_COMMAND, &old_cmd);
+       pci_info(dev, "Bus mastering bit is %sabled in D0\n",
+                (old_cmd & PCI_COMMAND_MASTER) ? "en" : "dis");
+
        dev->current_state = PCI_D0;
        if (need_restore)
                return 1;

Here's what I observe.

$ sudo dmesg | grep mastering
[    2.560916] pci 0000:00:01.1: Bus mastering bit is enabled in D0
[    2.580832] pci 0000:00:01.2: Bus mastering bit is enabled in D0
[    2.594468] pci 0000:00:02.1: Bus mastering bit is enabled in D0
[    2.595439] pci 0000:00:02.2: Bus mastering bit is enabled in D0
[    2.596292] pci 0000:00:02.3: Bus mastering bit is enabled in D0
[    2.597235] pci 0000:00:02.4: Bus mastering bit is enabled in D0
[    2.600120] pci 0000:00:08.1: Bus mastering bit is enabled in D0
[    2.600899] pci 0000:00:08.2: Bus mastering bit is enabled in D0
[    2.601535] pci 0000:00:08.3: Bus mastering bit is enabled in D0
[    2.610333] pci 0000:c1:00.0: Bus mastering bit is enabled in D0
[    2.611717] pci 0000:c2:00.0: Bus mastering bit is disabled in D0
[    2.612795] pci 0000:c3:00.0: Bus mastering bit is disabled in D0
[    2.613959] pci 0000:c4:00.0: Bus mastering bit is enabled in D0
[    2.621622] pci 0000:c5:00.0: Bus mastering bit is enabled in D0
[    2.624022] pci 0000:c5:00.1: Bus mastering bit is disabled in D0
[    2.624697] pci 0000:c5:00.2: Bus mastering bit is disabled in D0
[    2.629270] pci 0000:c5:00.4: Bus mastering bit is enabled in D0
[    2.634864] pci 0000:c5:00.5: Bus mastering bit is disabled in D0
[    2.635657] pci 0000:c5:00.7: Bus mastering bit is disabled in D0
[    2.636505] pci 0000:c6:00.0: Bus mastering bit is disabled in D0
[    2.636859] pci 0000:c6:00.1: Bus mastering bit is disabled in D0
[    2.641870] pci 0000:c7:00.0: Bus mastering bit is enabled in D0
[    2.649681] pci 0000:c7:00.3: Bus mastering bit is enabled in D0
[    2.657457] pci 0000:c7:00.4: Bus mastering bit is enabled in D0
[    2.665211] pci 0000:c7:00.5: Bus mastering bit is enabled in D0
[    2.669811] pci 0000:c7:00.6: Bus mastering bit is enabled in D0
[    5.114596] pcieport 0000:00:01.1: Bus mastering bit is enabled in D0
[    5.170824] pcieport 0000:00:01.2: Bus mastering bit is enabled in D0
[    8.025850] snd_hda_intel 0000:c5:00.1: Bus mastering bit is disabled in D0
[   22.314165] pcieport 0000:00:02.4: Bus mastering bit is enabled in D0
[   22.330193] r8169 0000:c4:00.0: Bus mastering bit is enabled in D0
[   22.336309] xhci_hcd 0000:c7:00.0: Bus mastering bit is disabled in D0
[   34.587505] snd_hda_intel 0000:c5:00.1: Bus mastering bit is enabled in D0
[   35.771050] xhci_hcd 0000:c5:00.4: Bus mastering bit is disabled in D0

So doesn't BIOS appear to have set bus mastering on a majority of devices already?

Quoted 2 lines

That sounds like a very bad idea. For security reasons alone, we only want to enable bus mastering when needed.

Should actually be doing the reverse of my proposed patch and explicitly disabling bus mastering in the PCI core at startup then require drivers to set policy?

Quoted 7 lines

It's up to
the driver to enable it, not up to the PCI core. We've had cases in the past where devices corrupted memory because BIOS left bus mastering enabled, see abb2bafd295f. Enabling bus mastering for everything anytime will exacerbate such problems or uncover new ones.

How do you feel about the other proposal I mentioned, not clearing it on kexec? Matthew confirmed this will help this issue too.