Date: Thu, 20 Mar 2025 17:27:16 +0000
From: "Russell King (Oracle)" <linux@armlinux.org.uk>
To: "Rafael J. Wysocki" <rafael@kernel.org>,
	Len Brown <len.brown@intel.com>, Pavel Machek <pavel@kernel.org>
Cc: Jon Hunter <jonathanh@nvidia.com>, Thierry Reding <treding@nvidia.com>,
	linux-pm@vger.kernel.org, linux-arm-kernel@lists.infradead.org
Subject: suspend-to-ram resume fails on Tegra Jetson Xavier NX
Message-ID: <Z9xP9IpLD04O5KPY@shell.armlinux.org.uk>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Sender: Russell King (Oracle) <linux@armlinux.org.uk>

Hi,

I'm struggling with a deadlock in the _noirq paths when resuming from
suspend-to-ram. with 6.14-rc6 (specifically net-next).

Having thrown lots of debug into kernel/power/suspend.c and
drivers/base/power/main.c, what I observe is:

1. /sys/power/pm_async is 1. If this is set to 0, then the problem does
   not manifest.

2. it seems the sync/async resume doesn't order correctly. With my own
   debugging in place, I see:

CPU5 is up
resume: suspend_enter:471
resume: dpm_resume_noirq:794
resume: dpm_noirq_resume_devices:742
 faux: PM: trying async resume (async=0)
 faux: PM:   -- will resume sync
 platform: PM: trying async resume (async=0)
 platform: PM:   -- will resume sync
 cpu: PM: trying async resume (async=0)
 cpu: PM:   -- will resume sync
cpu cpu0: PM: trying async resume (async=0)
cpu cpu0: PM:   -- will resume sync
...
tegra-bpmp bpmp: PM: trying async resume (async=0)
tegra-bpmp bpmp: PM:   -- will resume sync
platform 17000000.gpu: PM: trying async resume (async=0)
platform 17000000.gpu: PM:   -- will resume sync
platform 13e00000.host1x: PM: trying async resume (async=0)
platform 13e00000.host1x: PM:   -- will resume sync
...
tegra-bpmp-i2c bpmp:i2c: PM: trying async resume (async=0)
tegra-bpmp-i2c bpmp:i2c: PM:   -- will resume sync

*** Note - bpmp:i2c is on the dpm_noirq_list after 13e00000.host1x
and is being resumed synchronously.

i2c i2c-0: PM: trying async resume (async=1)
i2c-dev i2c-0: PM: trying async resume (async=0)
i2c i2c-0: PM: device_resume_noirq:650
i2c-dev i2c-0: PM:   -- will resume sync
max77620 0-003c: PM: trying async resume (async=1)
max77620-pinctrl max20024-pinctrl: PM: trying async resume (async=0)
max77620 0-003c: PM: device_resume_noirq:650
...
i2c i2c-0: PM: waiting for parent (bpmp:i2c)
max77620 0-003c: PM: waiting for parent (i2c-0)
...
tegra-bpmp bpmp: PM: device_resume_noirq:650
tegra-bpmp bpmp: PM: waiting for parent (platform)
tegra-bpmp bpmp: PM: parent wait finished, waiting for suppliers
tegra-bpmp bpmp: PM: waiting for supplier 2c00000.memory-controller
tegra-bpmp bpmp: PM: supplier finished
tegra-bpmp bpmp: PM: waiting for supplier 3c00000.hsp
tegra-bpmp bpmp: PM: supplier finished
tegra-bpmp bpmp: PM: suppliers finished
tegra-bpmp bpmp: PM: -- resume_noirq: tegra_bpmp_resume (sync)
tegra-bpmp bpmp: PM: -- complete
...
platform 13e00000.host1x: PM: device_resume_noirq:650
platform 13e00000.host1x: PM: waiting for parent (bus@0)
platform 13e00000.host1x: PM: parent wait finished, waiting for suppliers
platform 13e00000.host1x: PM: waiting for supplier 2c60000.external-memory-controller
platform 13e00000.host1x: PM: supplier finished
platform 13e00000.host1x: PM: waiting for supplier 2c00000.memory-controller
platform 13e00000.host1x: PM: supplier finished
platform 13e00000.host1x: PM: waiting for supplier 2200000.gpio
platform 13e00000.host1x: PM: supplier finished
platform 13e00000.host1x: PM: waiting for supplier bpmp
platform 13e00000.host1x: PM: supplier finished
platform 13e00000.host1x: PM: waiting for supplier regulator-vdd-hdmi
platform 13e00000.host1x: PM: supplier finished
platform 13e00000.host1x: PM: waiting for supplier bus@0
platform 13e00000.host1x: PM: supplier finished
platform 13e00000.host1x: PM: waiting for supplier 0-003c

Here we die - with interrupts off, so none of the kernel lockup
detectors are functional, so without debug the console is completely
silent.

13e00000.host1x is not bound to a driver (it failed its probe), but
is being synchronously resumed. It is waiting for 0-003c to be resumed.

0-003c is being resumed asynchronously, and is waiting for i2c-0 to
be resumed.

i2c-0 is being resumed asynchronously, and is waiting for bpmp:i2c to
be resumed.

bpmp:i2c is not *yet* being resumed, because it is to be resumed
synchronously, and is on the dpm_noirq_list after 13e00000.host1x.
This results in a silently dead system on resume.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!