Skip to contentSkip to conversation

accel/amdxdna: Keep PCI power state in sync with runtime PM

PATCH In review [email protected] 3 participants 3 messages Aug 14 – Aug 17

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 accel/amdxdna: Keep PCI power state in sync with runtime PM

From: Reza Jelveh <[email protected]>

The platform powers the NPU down as soon as the driver stops its firmware, but aie2_hw_stop() never recorded that transition. The PCI core kept tracking the device as D0, so on the next system suspend the noirq phase attempted a D0-to-D3hot transition on a device whose config space returns 0xffffffff:

Unable to change power state from D0 to D3hot, device inaccessible

The platform's power state was then out of sync with the kernel's, the SMU idle condition was never met, and suspend-to-idle did not reach the deepest state. On resume, aie2_hw_start() re-enabled the device without requesting D0 first and failed reading the SMU mailbox (0xffffffff, -EINVAL), leaving the NPU dead until reboot.

Record D3hot at the end of aie2_hw_stop(), while the device is still reachable, and request D0 at the beginning of aie2_hw_start(), so the kernel's PCI power state always matches the platform state. Skip the firmware stop in aie2_hw_suspend() when runtime PM has already stopped the device, and demote the "device is already stopped" message to debug level.

---
 drivers/accel/amdxdna/aie2_pci.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
drivers/accel/amdxdna/aie2_pci.c +18 -2
index a7b923005..ad14e4fc6 100644
@@ -288,7 +288,7 @@ static void aie2_hw_stop(struct amdxdna_dev *xdna)
 	struct amdxdna_dev_hdl *ndev = xdna->dev_handle;
 
 	if (ndev->dev_status <= AIE2_DEV_INIT) {
-		XDNA_ERR(xdna, "device is already stopped");
+		XDNA_DBG(xdna, "device is already stopped");
 		return;
 	}
 
@@ -301,6 +301,12 @@ static void aie2_hw_stop(struct amdxdna_dev *xdna)
 	aie2_smu_fini(ndev);
 	aie2_error_async_events_free(ndev);
 	pci_disable_device(pdev);
+	/*
+	 * The platform powers the NPU down once the firmware is stopped.
+	 * Record D3hot while the device is still reachable, so the noirq
+	 * suspend path does not attempt a stale D0 transition.
+	 */
+	pci_set_power_state(pdev, PCI_D3hot);
 
 	ndev->dev_status = AIE2_DEV_INIT;
 }
@@ -318,6 +324,13 @@ static int aie2_hw_start(struct amdxdna_dev *xdna)
 		return 0;
 	}
 
+	/* The platform powers the device down when the firmware is stopped. */
+	ret = pci_set_power_state(pdev, PCI_D0);
+	if (ret) {
+		XDNA_ERR(xdna, "failed to power up device, ret %d", ret);
+		return ret;
+	}
+
 	ret = pci_enable_device(pdev);
 	if (ret) {
 		XDNA_ERR(xdna, "failed to enable device, ret %d", ret);
@@ -427,12 +440,15 @@ static int aie2_hw_start(struct amdxdna_dev *xdna)
 
 static int aie2_hw_suspend(struct amdxdna_dev *xdna)
 {
+	struct amdxdna_dev_hdl *ndev = xdna->dev_handle;
 	struct amdxdna_client *client;
 
 	list_for_each_entry(client, &xdna->client_list, node)
 		aie2_hwctx_suspend(client);
 
-	aie2_hw_stop(xdna);
+	/* Runtime PM may already have stopped the device. */
+	if (ndev->dev_status > AIE2_DEV_INIT)
+		aie2_hw_stop(xdna);
 
 	return 0;
 }
Signature · 1 line
2.55.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Error paths in aie2_hw_start() leave the PCI power state out of sync

Pre-existing issues:
- [High] Missing PCI state save before D3hot transition causes kernel warnings and configuration corruption --

--- Patch [1]: [PATCH] accel/amdxdna: Keep PCI power state in sync with runtime PM --- commit f75387302cbbf4db35ad9c9ef5066d43bf266934
Author: Reza Jelveh <[email protected]>

accel/amdxdna: Keep PCI power state in sync with runtime PM

This patch attempts to keep the PCI core's power state tracking in sync with the NPU hardware by explicitly setting D3hot in aie2_hw_stop() and D0 in aie2_hw_start(). It aims to prevent errors during noirq suspend and resume due to reading unreachable config space.

Quoted 4 lines
drivers/accel/amdxdna/aie2_pci.c +0 -0
index daec1f6b49079..831277c1eb744 100644

[ ... ]

drivers/accel/amdxdna/aie2_pci.c static void aie2_hw_stop(struct amdxdna_dev *xdna) · line 324

Quoted 10 lines
drivers/accel/amdxdna/aie2_pci.c +6 -0
@@ -316,6 +316,12 @@ static void aie2_hw_stop(struct amdxdna_dev *xdna)
 	aie2_smu_fini(ndev);
 	aie2_error_async_events_free(ndev);
 	pci_disable_device(pdev);
+	/*
+	 * The platform powers the NPU down once the firmware is stopped.
+	 * Record D3hot while the device is still reachable, so the noirq
+	 * suspend path does not attempt a stale D0 transition.
+	 */
+	pci_set_power_state(pdev, PCI_D3hot);

[Severity: High]
This is a pre-existing issue, but will setting the power state to D3hot here without first saving the PCI state trigger a PM warning and force a read of corrupted config state?

If aie2_hw_stop() calls pci_set_power_state() without a preceding call to pci_save_state(), the suspend paths detect that the power state changed but the state is not saved, which triggers a "PCI PM: State of device not saved" warning.

During pci_pm_suspend(), does this warning proceed to the noirq phase where pci_save_state() reads from the already-powered-off device? This could save 0xffffffff and restore it on resume, which would permanently break the device until a cold reboot.

[ ... ]

drivers/accel/amdxdna/aie2_pci.c static int aie2_hw_start(struct amdxdna_dev *xdna) · line 348

Quoted 11 lines
drivers/accel/amdxdna/aie2_pci.c +7 -0
@@ -333,6 +339,13 @@ static int aie2_hw_start(struct amdxdna_dev *xdna)
 		return 0;
 	}
 
+	/* The platform powers the device down when the firmware is stopped. */
+	ret = pci_set_power_state(pdev, PCI_D0);
+	if (ret) {
+		XDNA_ERR(xdna, "failed to power up device, ret %d", ret);
+		return ret;
+	}
+

[Severity: High]
If hardware initialization fails later in aie2_hw_start(), do the error paths leave the PCI power state out of sync?

For example, if aie2_mgmt_fw_init() fails, the code jumps to stop_fw and eventually disable_dev:

stop_fw:

    aie2_suspend_fw(ndev);
    xdna_mailbox_stop_channel(ndev->aie.mgmt_chann);
stop_psp:
    aie_psp_stop(ndev->aie.psp_hdl);
fini_smu:
    aie2_smu_fini(ndev);
free_channel:
    xdna_mailbox_free_channel(ndev->aie.mgmt_chann);
    ndev->aie.mgmt_chann = NULL;
disable_dev:
    pci_disable_device(pdev);

These error paths power down the hardware by stopping the firmware but fail to roll back the kernel's state tracking with pci_set_power_state(pdev, PCI_D3hot).

Since the PM core continues to track the device as D0, won't the next suspend attempt try a D0-to-D3hot transition on the unreachable device and trigger the exact inaccessible warning this patch is meant to fix?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#2 lore raw
In reply to [email protected]
Quoted 16 lines · On 8/14/26 06:27, [email protected] wrote:

From: Reza Jelveh <[email protected]>

The platform powers the NPU down as soon as the driver stops its firmware, but aie2_hw_stop() never recorded that transition. The PCI core kept tracking the device as D0, so on the next system suspend the noirq phase attempted a D0-to-D3hot transition on a device whose config space returns 0xffffffff:

Unable to change power state from D0 to D3hot, device inaccessible

The platform's power state was then out of sync with the kernel's, the SMU idle condition was never met, and suspend-to-idle did not reach the deepest state. On resume, aie2_hw_start() re-enabled the device without requesting D0 first and failed reading the SMU mailbox (0xffffffff, -EINVAL), leaving the NPU dead until reboot.

Quoted 8 lines

Record D3hot at the end of aie2_hw_stop(), while the device is still reachable, and request D0 at the beginning of aie2_hw_start(), so the kernel's PCI power state always matches the platform state. Skip the firmware stop in aie2_hw_suspend() when runtime PM has already stopped the device, and demote the "device is already stopped" message to debug level.

Thanks for providing the fix. Could you add a 'Fixes' tag?

Quoted 18 lines
---
  drivers/accel/amdxdna/aie2_pci.c | 20 ++++++++++++++++++--
  1 file changed, 18 insertions(+), 2 deletions(-)
drivers/accel/amdxdna/aie2_pci.c +1 -1
index a7b923005..ad14e4fc6 100644
@@ -288,7 +288,7 @@ static void aie2_hw_stop(struct amdxdna_dev *xdna)
  	struct amdxdna_dev_hdl *ndev = xdna->dev_handle;
 
  	if (ndev->dev_status <= AIE2_DEV_INIT) {
-		XDNA_ERR(xdna, "device is already stopped");
+		XDNA_DBG(xdna, "device is already stopped");
  		return;
  	}
 

drivers/accel/amdxdna/aie2_pci.c static void aie2_hw_stop(struct amdxdna_dev *xdna) · line 309

Quoted 9 lines
drivers/accel/amdxdna/aie2_pci.c +6 -0
@@ -301,6 +301,12 @@ static void aie2_hw_stop(struct amdxdna_dev *xdna)
  	aie2_smu_fini(ndev);
  	aie2_error_async_events_free(ndev);
  	pci_disable_device(pdev);
+	/*
+	 * The platform powers the NPU down once the firmware is stopped.
+	 * Record D3hot while the device is still reachable, so the noirq
+	 * suspend path does not attempt a stale D0 transition.
+	 */
+	pci_set_power_state(pdev, PCI_D3hot);

Need to add

    pci_save_state(pdev);

before pci_disable_device(pdev) ? (and pci_restore_state after recovering to D0)

Quoted 21 lines
drivers/accel/amdxdna/aie2_pci.c +7 -0
  	ndev->dev_status = AIE2_DEV_INIT;
  }
@@ -318,6 +324,13 @@ static int aie2_hw_start(struct amdxdna_dev *xdna)
  		return 0;
  	}
 
+	/* The platform powers the device down when the firmware is stopped. */
+	ret = pci_set_power_state(pdev, PCI_D0);
+	if (ret) {
+		XDNA_ERR(xdna, "failed to power up device, ret %d", ret);
+		return ret;
+	}
+
  	ret = pci_enable_device(pdev);
  	if (ret) {
  		XDNA_ERR(xdna, "failed to enable device, ret %d", ret);
@@ -427,12 +440,15 @@ static int aie2_hw_start(struct amdxdna_dev *xdna)
 
  static int aie2_hw_suspend(struct amdxdna_dev *xdna)
  {

drivers/accel/amdxdna/aie2_pci.c static int aie2_hw_start(struct amdxdna_dev *xdna) · line 451

Quoted 10 lines
drivers/accel/amdxdna/aie2_pci.c +4 -1
+	struct amdxdna_dev_hdl *ndev = xdna->dev_handle;
  	struct amdxdna_client *client;
 
  	list_for_each_entry(client, &xdna->client_list, node)
  		aie2_hwctx_suspend(client);
 
-	aie2_hw_stop(xdna);
+	/* Runtime PM may already have stopped the device. */
+	if (ndev->dev_status > AIE2_DEV_INIT)
+		aie2_hw_stop(xdna);

This is redundant check. aie2_hw_stop() already checks it and returns early.

Thanks,

Lizhi

Quoted 3 lines
drivers/accel/amdxdna/aie2_pci.c +0 -0
  	return 0;
  }