pmdomain: ti_sci: re-sync TIFS with genpd on resume
Subsystems (3)
PATCH 1/1 pmdomain: ti_sci: re-sync TIFS with genpd on resume
From: Vitor Soares <[email protected]>
When a device in a TI SCI power domain is on the wakeup path of a wakeup-capable child, the suspend path skips genpd_sync_power_off(). No put_device is sent to TIFS and the domain's genpd status remains ON.
TIFS powers off the hardware during deep sleep regardless, since it was never informed to keep the domain active. On resume, because the domain's genpd status is ON, no get_device is issued. The driver then accesses registers of a powered-off domain, causing a synchronous external abort (AXI bus error, ESR 0x96000010).
Commit 0b5fe1c4ab3c ("pmdomain: ti-sci: Set PD on/off state according to the HW state") exposed this. Before, domain status was initialized to OFF, so get_device was always issued on resume.
Add a .resume hook that queries the domain's state from TIFS and re-syncs TIFS with get_device when genpd has it ON but TIFS has it OFF. The hook is only registered when the is_on op is available, since detection depends on it.
Move ti_sci_pm_pd_is_on() earlier in the file so it is available to the resume hook.
--- drivers/pmdomain/ti/ti_sci_pm_domains.c | 66 ++++++++++++++++++------- 1 file changed, 49 insertions(+), 17 deletions(-)
drivers/pmdomain/ti/ti_sci_pm_domains.c +49 -17
@@ -131,6 +131,23 @@ static int ti_sci_pd_power_on(struct generic_pm_domain *domain)return ti_sci->ops.dev_ops.get_device (ti_sci, pd->idx); } +static bool ti_sci_pm_pd_is_on (struct ti_sci_genpd_provider *pd_provider, +int pd_idx) +{ +bool is_on; +int ret; + +if (!pd_provider->ti_sci->ops.dev_ops.is_on) +return false; + + ret = pd_provider->ti_sci->ops.dev_ops.is_on (pd_provider->ti_sci, + pd_idx,NULL , &is_on); +if (ret) +return false; + +return is_on; +} +#ifdef CONFIG_PM_SLEEP static int ti_sci_pd_suspend (struct device *dev) {
@@ -149,8 +166,37 @@ static int ti_sci_pd_suspend(struct device *dev)return 0 ; } + +static int ti_sci_pd_resume (struct device *dev) +{ +struct generic_pm_domain *genpd =pd_to_genpd (dev->pm_domain); +struct ti_sci_pm_domain *pd =genpd_to_ti_sci_pd (genpd); +const struct ti_sci_handle *ti_sci = pd->parent->ti_sci; +int ret; + +/* +* If genpd's domain state is ON but TIFS powered it OFF during +* suspend, re-sync by issuing get_device before the driver resumes. +*/ +if (genpd->status ==GENPD_STATE_ON && + !ti_sci_pm_pd_is_on (pd->parent, pd->idx)) { +dev_dbg (dev,"ti_sci_pd: ID:%d genpd/TIFS out of sync on resume, re-syncing\n" , + pd->idx); +if (pd->exclusive) + ret = ti_sci->ops.dev_ops.get_device_exclusive (ti_sci, + pd->idx); +else + ret = ti_sci->ops.dev_ops.get_device (ti_sci, pd->idx); +if (ret) +return ret; + } + +return pm_generic_resume (dev); +} +#else #define ti_sci_pd_suspend NULL +#define ti_sci_pd_resume NULL#endif /*
@@ -200,23 +246,6 @@ static bool ti_sci_pm_idx_exists(struct ti_sci_genpd_provider *pd_provider, u32return false; } -static bool ti_sci_pm_pd_is_on (struct ti_sci_genpd_provider *pd_provider, -int pd_idx) -{ -bool is_on; -int ret; - -if (!pd_provider->ti_sci->ops.dev_ops.is_on) -return false; - - ret = pd_provider->ti_sci->ops.dev_ops.is_on (pd_provider->ti_sci, - pd_idx,NULL , &is_on); -if (ret) -return false; - -return is_on; -} -static int ti_sci_pm_domain_probe (struct platform_device *pdev) {struct device *dev = &pdev->dev;
@@ -283,6 +312,9 @@ static int ti_sci_pm_domain_probe(struct platform_device *pdev) pd_provider->ti_sci->ops.pm_ops.set_latency_constraint) pd->pd.domain.ops.suspend = ti_sci_pd_suspend; +if (pd_provider->ti_sci->ops.dev_ops.is_on) + pd->pd.domain.ops.resume = ti_sci_pd_resume; + is_on =ti_sci_pm_pd_is_on (pd_provider, pd->idx);
Signature · 1 line
2.53.0
Hi Vitor
Quoted 6 lines · Vitor Soares
From: Vitor Soares <[email protected]>
When a device in a TI SCI power domain is on the wakeup path of a wakeup-capable child, the suspend path skips
genpd_sync_power_off(). Noput_deviceis sent to TIFS and the domain's genpd status remains ON.
Correction of terminologies: TIFS is Root of trust component and is not usually involved in power management, that would be DM (Device Manager)
But to be really sure who is doing what, Could you provide an example and the platform on which you see the issue / external abort?
Quoted 6 lines
TIFS powers off the hardware during deep sleep regardless, since it was never informed to keep the domain active. On resume, because the domain's genpd status is ON, no
get_deviceis issued. The driver then accesses registers of a powered-off domain, causing a synchronous external abort (AXI bus error, ESR0x96000010).
Hmm, if something is wakeup source, I would expect even TIFS/DM not to turn if off, else module wakeup wouldn't work.
Quoted 17 lines
Commit
0b5fe1c4ab3c("pmdomain: ti-sci: Set PD on/off state according to the HW state") exposed this. Before, domain status was initialized to OFF, soget_devicewas always issued on resume.Add a .resume hook that queries the domain's state from TIFS and re-syncs TIFS with
get_devicewhen genpd has it ON but TIFS has it OFF. The hook is only registered when theis_onop is available, since detection depends on it.Move
ti_sci_pm_pd_is_on()earlier in the file so it is available to the resume hook.Fixes 0b5fe1c4ab3c ("pmdomain: ti-sci: Set PD on/off state according to the HW state")Cc [email protected] # 6.18+Signed-off-by Vitor Soares <[email protected]>---
[...]
Signature · 3 lines
Regards Vignesh https://ti.com/opensource
Hi Vignesh
Thank you for the review.
Quoted 5 lines · Vignesh Raghavendra
Hi Vitor
Quoted 2 lines · Vitor Soares
From: Vitor Soares <[email protected]>
Quoted 8 lines
Quoted 4 lines
When a device in a TI SCI power domain is on the wakeup path of a wakeup-capable child, the suspend path skips
genpd_sync_power_off(). Noput_deviceis sent to TIFS and the domain's genpd status remains ON.Correction of terminologies: TIFS is Root of trust component and is not usually involved in power management, that would be DM (Device Manager)
Thank you for the clarification. I will address this on v2. Also, I was thinking to replace put_device/get_device with ti_sci_pd_power_off/ti_sci_pd_power_on if that makes more clear the content.
Quoted 3 lines
But to be really sure who is doing what, Could you provide an example and the platform on which you see the issue / external abort?
This was reproduced on our Toradex Verdin AM62P WB and the driver for our Wi-Fi module on the SDIO bus calls device_init_wakeup() during the initialization.
After enter in suspend, it show the following error resume path:
[ 41.759341] Internal error: synchronous external abort: 0000000096000010 [#1] SMP [ 41.843286] CPU: 0 UID: 0 PID: 933 Comm: rtcwake Tainted: G M O 6.18.21-dirty #3 PREEMPT [ 41.852762] Tainted: [M]=MACHINE_CHECK, [O]=OOT_MODULE [ 41.857891] Hardware name: Toradex Verdin AM62P WB on Verdin Development Board (DT) [ 41.865537] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 41.872492] pc : regmap_mmio_read32le+0x8/0x20 [ 41.876941] lr : regmap_mmio_read+0x44/0x70 [ 41.881120] sp : ffff800081fdb8e0 [ 41.884428] x29: ffff800081fdb8e0 x28: 0000000000000000 x27: ffffa95bb64aa9c8 [ 41.891563] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 [ 41.898697] x23: 0000000080000000 x22: ffff000002df5c00 x21: ffff800081fdb9b4 [ 41.905831] x20: 0000000000000100 x19: ffff000001286400 x18: 0000000000000000 [ 41.912965] x17: 2d69696d67722f79 x16: 687020726f662067 x15: ffff00007fb74f40 [ 41.920100] x14: 00000000000002ea x13: 000000000000031f x12: 0000000000000000 [ 41.927234] x11: 00000000000000c0 x10: 00000000000009e0 x9 : ffff800081fdb7a0 [ 41.934368] x8 : ffff00007fb6ce00 x7 : 0000000000000000 x6 : 0000000000000000 [ 41.941502] x5 : ffffa95bb57948d8 x4 : 0000000000000100 x3 : 0000000000000100 [ 41.948636] x2 : ffffa95bb5795034 x1 : 0000000000000100 x0 : ffff80008025d100 [ 41.955770] Call trace: [ 41.958211] regmap_mmio_read32le+0x8/0x20 (P) [ 41.962655] _regmap_bus_reg_read+0x70/0xb0 [ 41.966839] _regmap_read+0x64/0xdc [ 41.970327] _regmap_update_bits+0xf4/0x140 [ 41.974509] regmap_update_bits_base+0x64/0x98 [ 41.978952] sdhci_am654_runtime_resume+0x138/0x208 [ 41.983830] pm_generic_runtime_resume+0x2c/0x44 [ 41.988445] __genpd_runtime_resume+0x30/0x7c [ 41.992804] genpd_runtime_resume+0xdc/0x2e8 [ 41.997073] pm_runtime_force_resume+0x68/0xf4 [ 42.001517] dpm_run_callback+0x8c/0x14c [ 42.005439] device_resume+0x11c/0x34c [ 42.009188] dpm_resume+0x178/0x1f0 [ 42.012673] dpm_resume_end+0x18/0x34 [ 42.016332] suspend_devices_and_enter+0x4a4/0x668 [ 42.021123] pm_suspend+0x170/0x2dc [ 42.024610] state_store+0x80/0x104 [ 42.028096] kobj_attr_store+0x18/0x2c [ 42.031845] sysfs_kf_write+0x7c/0x94 [ 42.035508] kernfs_fop_write_iter+0x130/0x1fc [ 42.039949] vfs_write+0x200/0x370 [ 42.043351] ksys_write+0x6c/0x100 [ 42.046752] __arm64_sys_write+0x1c/0x28 [ 42.050673] invoke_syscall.constprop.0+0x50/0xe4 [ 42.055378] do_el0_svc+0x40/0xc4 [ 42.058691] el0_svc+0x40/0x15c [ 42.061834] el0t_64_sync_handler+0xa0/0xe4 [ 42.066015] el0t_64_sync+0x198/0x19c [ 42.069680] Code: aa0603e0 d65f03c0 f9400000 8b214000 (b9400000)
Quoted 11 lines
Quoted 7 lines
TIFS powers off the hardware during deep sleep regardless, since it was never informed to keep the domain active. On resume, because the domain's genpd status is ON, no
get_deviceis issued. The driver then accesses registers of a powered-off domain, causing a synchronous external abort (AXI bus error, ESR0x96000010).Hmm, if something is wakeup source, I would expect even TIFS/DM not to turn if off, else module wakeup wouldn't work.
I tested UART as a wakeup source and I couldn't reproduce this issue. My understanding is that UART has its own TI SCI domain and device_may_wakeup() is true directly on that domain device, so the set_device_constraint fires correctly and DM keeps it powered.
Here is my tracking of the issue:
Wi-Fi driver registers as wakeup source:device_init_wakeup(mmc0:0001)
During suspend/resume.dpm_suspend()
->genpd_suspend_dev(fa20000.mmc)
->ti_sci_pd_suspend(fa20000.mmc)
->ti_sci_pd_set_wkup_constraint(fa20000.mmc)
device_may_wakeup(fa20000.mmc) = false
set_device_constraint never sent to DMdpm_suspend_noirq()
->genpd_finish_suspend(fa20000.mmc)
->device_awake_path(fa20000.mmc) = true
->GENPD_FLAG_ACTIVE_WAKEUP = true
genpd status = GENPD_STATE_ON
skip power_off (ti_sci_pd_power_off)
On deep sleep entry, DM powers off fa20000.mmc independently. It received no set_device_constraint nor ti_sci_pd_power_off.
I attempted to fix this by calling set_device_constraint when device_wakeup_path() is true but it prevented the system from entering deep sleep entirely.
[...]
Best regards,
Vitor Soares
Hi Vitor,
Quoted 123 lines · Vitor Soares
Hi Vignesh
Thank you for the review.
Quoted 5 lines · Vignesh Raghavendra
Hi Vitor
Quoted 2 lines · Vitor Soares
From: Vitor Soares <[email protected]>
Quoted 8 lines
Quoted 4 lines
When a device in a TI SCI power domain is on the wakeup path of a wakeup-capable child, the suspend path skips
genpd_sync_power_off(). Noput_deviceis sent to TIFS and the domain's genpd status remains ON.Correction of terminologies: TIFS is Root of trust component and is not usually involved in power management, that would be DM (Device Manager)
Thank you for the clarification. I will address this on v2. Also, I was thinking to replace put_device/get_device with ti_sci_pd_power_off/ti_sci_pd_power_on if that makes more clear the content.
Quoted 3 lines
But to be really sure who is doing what, Could you provide an example and the platform on which you see the issue / external abort?
This was reproduced on our Toradex Verdin AM62P WB and the driver for our Wi-Fi module on the SDIO bus calls
device_init_wakeup()during the initialization.After enter in suspend, it show the following error resume path:
[ 41.759341] Internal error: synchronous external abort: 0000000096000010 [#1] SMP [ 41.843286] CPU: 0 UID: 0 PID: 933 Comm: rtcwake Tainted: G M O 6.18.21-dirty #3 PREEMPT [ 41.852762] Tainted: [M]=MACHINE_CHECK, [O]=OOT_MODULE [ 41.857891] Hardware name: Toradex Verdin AM62P WB on Verdin Development Board (DT) [ 41.865537] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 41.872492] pc : regmap_mmio_read32le+0x8/0x20 [ 41.876941] lr : regmap_mmio_read+0x44/0x70 [ 41.881120] sp : ffff800081fdb8e0 [ 41.884428] x29: ffff800081fdb8e0 x28: 0000000000000000 x27: ffffa95bb64aa9c8 [ 41.891563] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 [ 41.898697] x23: 0000000080000000 x22: ffff000002df5c00 x21: ffff800081fdb9b4 [ 41.905831] x20: 0000000000000100 x19: ffff000001286400 x18: 0000000000000000 [ 41.912965] x17: 2d69696d67722f79 x16: 687020726f662067 x15: ffff00007fb74f40 [ 41.920100] x14: 00000000000002ea x13: 000000000000031f x12: 0000000000000000 [ 41.927234] x11: 00000000000000c0 x10: 00000000000009e0 x9 : ffff800081fdb7a0 [ 41.934368] x8 : ffff00007fb6ce00 x7 : 0000000000000000 x6 : 0000000000000000 [ 41.941502] x5 : ffffa95bb57948d8 x4 : 0000000000000100 x3 : 0000000000000100 [ 41.948636] x2 : ffffa95bb5795034 x1 : 0000000000000100 x0 : ffff80008025d100 [ 41.955770] Call trace: [ 41.958211] regmap_mmio_read32le+0x8/0x20 (P) [ 41.962655] _regmap_bus_reg_read+0x70/0xb0 [ 41.966839] _regmap_read+0x64/0xdc [ 41.970327] _regmap_update_bits+0xf4/0x140 [ 41.974509] regmap_update_bits_base+0x64/0x98 [ 41.978952] sdhci_am654_runtime_resume+0x138/0x208 [ 41.983830] pm_generic_runtime_resume+0x2c/0x44 [ 41.988445] __genpd_runtime_resume+0x30/0x7c [ 41.992804] genpd_runtime_resume+0xdc/0x2e8 [ 41.997073] pm_runtime_force_resume+0x68/0xf4 [ 42.001517] dpm_run_callback+0x8c/0x14c [ 42.005439] device_resume+0x11c/0x34c [ 42.009188] dpm_resume+0x178/0x1f0 [ 42.012673] dpm_resume_end+0x18/0x34 [ 42.016332] suspend_devices_and_enter+0x4a4/0x668 [ 42.021123] pm_suspend+0x170/0x2dc [ 42.024610] state_store+0x80/0x104 [ 42.028096] kobj_attr_store+0x18/0x2c [ 42.031845] sysfs_kf_write+0x7c/0x94 [ 42.035508] kernfs_fop_write_iter+0x130/0x1fc [ 42.039949] vfs_write+0x200/0x370 [ 42.043351] ksys_write+0x6c/0x100 [ 42.046752] __arm64_sys_write+0x1c/0x28 [ 42.050673] invoke_syscall.constprop.0+0x50/0xe4 [ 42.055378] do_el0_svc+0x40/0xc4 [ 42.058691] el0_svc+0x40/0x15c [ 42.061834] el0t_64_sync_handler+0xa0/0xe4 [ 42.066015] el0t_64_sync+0x198/0x19c [ 42.069680] Code: aa0603e0 d65f03c0 f9400000 8b214000 (b9400000)Quoted 11 lines
Quoted 7 lines
TIFS powers off the hardware during deep sleep regardless, since it was never informed to keep the domain active. On resume, because the domain's genpd status is ON, no
get_deviceis issued. The driver then accesses registers of a powered-off domain, causing a synchronous external abort (AXI bus error, ESR0x96000010).Hmm, if something is wakeup source, I would expect even TIFS/DM not to turn if off, else module wakeup wouldn't work.
I tested UART as a wakeup source and I couldn't reproduce this issue. My understanding is that UART has its own TI SCI domain and
device_may_wakeup()is true directly on that domain device, so theset_device_constraintfires correctly and DM keeps it powered.Here is my tracking of the issue:
Wi-Fi driver registers as wakeup source:
device_init_wakeup(mmc0:0001)During suspend/resume.
dpm_suspend()
->genpd_suspend_dev(fa20000.mmc)->ti_sci_pd_suspend(fa20000.mmc) ->ti_sci_pd_set_wkup_constraint(fa20000.mmc) device_may_wakeup(fa20000.mmc) = false set_device_constraint never sent to DM
dpm_suspend_noirq()
->genpd_finish_suspend(fa20000.mmc)
->device_awake_path(fa20000.mmc) = true
->GENPD_FLAG_ACTIVE_WAKEUP = true
genpd status =GENPD_STATE_ON
skippower_off(ti_sci_pd_power_off)
Quoted 2 lines
On deep sleep entry, DM powers off fa20000.mmc independently. It received no
set_device_constraintnorti_sci_pd_power_off.
In AM62P fa20000.mmc is part of main domain. During deepsleep the entire main domain is turned off by the DM, that is why you see the failures.
In-order to debug this we need to check why pd off and pd on call is not getting called for fa20000.mmc during suspend and resume.
Quoted 4 lines
I attempted to fix this by calling
set_device_constraintwhendevice_wakeup_path()is true but it prevented the system from entering deep sleep entirely.
In AM62P the DM manager selects the low power mode to enter based on the constrains set. The mode selection logic will ensure that if a constraint is set on the device, it will select a low power mode in which the device is kept on or can wake the system up. the MMC is part of main domain and there is no low power mode in which the MMC can stay alive or generate a wake up interrupt. so when a constraint is set of MMC, we cannot enter any low power mode. that why you see a failure.
Regards
Sebin
Quoted 5 lines
[...]
Best regards,
Vitor Soares
Hello Sebin
Quoted 142 lines · Sebin Francis
Hi Vitor,
Quoted 139 lines · Vitor Soares
Hi Vignesh
Thank you for the review.
Quoted 5 lines · Vignesh Raghavendra
Hi Vitor
Quoted 2 lines · Vitor Soares
From: Vitor Soares <[email protected]>
Quoted 8 lines
Quoted 4 lines
When a device in a TI SCI power domain is on the wakeup path of a wakeup-capable child, the suspend path skips
genpd_sync_power_off(). Noput_deviceis sent to TIFS and the domain's genpd status remains ON.Correction of terminologies: TIFS is Root of trust component and is not usually involved in power management, that would be DM (Device Manager)
Thank you for the clarification. I will address this on v2. Also, I was thinking
to replace put_device/get_device with ti_sci_pd_power_off/ti_sci_pd_power_on if
that makes more clear the content.Quoted 3 lines
But to be really sure who is doing what, Could you provide an example and the platform on which you see the issue / external abort?
This was reproduced on our Toradex Verdin AM62P WB and the driver for our Wi-Fi
module on the SDIO bus callsdevice_init_wakeup()during the initialization.After enter in suspend, it show the following error resume path:
[ 41.759341] Internal error: synchronous external abort: 0000000096000010 [#1]
SMP
[ 41.843286] CPU: 0 UID: 0 PID: 933 Comm: rtcwake Tainted: G M O 6.18.21-dirty #3 PREEMPT
[ 41.852762] Tainted: [M]=MACHINE_CHECK, [O]=OOT_MODULE
[ 41.857891] Hardware name: Toradex Verdin AM62P WB on Verdin Development Board (DT)
[ 41.865537] pstate:200000c5(nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=- -)
[ 41.872492] pc :regmap_mmio_read32le+0x8/0x20
[ 41.876941] lr :regmap_mmio_read+0x44/0x70
[ 41.881120] sp :ffff800081fdb8e0
[ 41.884428] x29:ffff800081fdb8e0x28: 0000000000000000 x27:ffffa95bb64aa9c8
[ 41.891563] x26: 0000000000000000 x25: 0000000000000000 x24:
0000000000000000
[ 41.898697] x23: 0000000080000000 x22:ffff000002df5c00x21:ffff800081fdb9b4
[ 41.905831] x20: 0000000000000100 x19:ffff000001286400x18:
0000000000000000
[ 41.912965] x17:2d69696d67722f79x16:687020726f662067x15:ffff00007fb74f40
[ 41.920100] x14:00000000000002eax13:000000000000031fx12:
0000000000000000
[ 41.927234] x11:00000000000000c0x10:00000000000009e0x9 :ffff800081fdb7a0
[ 41.934368] x8 :ffff00007fb6ce00x7 : 0000000000000000 x6 :
0000000000000000
[ 41.941502] x5 :ffffa95bb57948d8x4 : 0000000000000100 x3 :
0000000000000100
[ 41.948636] x2 :ffffa95bb5795034x1 : 0000000000000100 x0 :ffff80008025d100
[ 41.955770] Call trace:
[ 41.958211]regmap_mmio_read32le+0x8/0x20(P)
[ 41.962655]_regmap_bus_reg_read+0x70/0xb0
[ 41.966839]_regmap_read+0x64/0xdc
[ 41.970327]_regmap_update_bits+0xf4/0x140
[ 41.974509]regmap_update_bits_base+0x64/0x98
[ 41.978952]sdhci_am654_runtime_resume+0x138/0x208
[ 41.983830]pm_generic_runtime_resume+0x2c/0x44
[ 41.988445]__genpd_runtime_resume+0x30/0x7c
[ 41.992804]genpd_runtime_resume+0xdc/0x2e8
[ 41.997073]pm_runtime_force_resume+0x68/0xf4
[ 42.001517]dpm_run_callback+0x8c/0x14c
[ 42.005439]device_resume+0x11c/0x34c
[ 42.009188]dpm_resume+0x178/0x1f0
[ 42.012673]dpm_resume_end+0x18/0x34
[ 42.016332]suspend_devices_and_enter+0x4a4/0x668
[ 42.021123]pm_suspend+0x170/0x2dc
[ 42.024610]state_store+0x80/0x104
[ 42.028096]kobj_attr_store+0x18/0x2c
[ 42.031845]sysfs_kf_write+0x7c/0x94
[ 42.035508]kernfs_fop_write_iter+0x130/0x1fc
[ 42.039949]vfs_write+0x200/0x370
[ 42.043351]ksys_write+0x6c/0x100
[ 42.046752]__arm64_sys_write+0x1c/0x28
[ 42.050673] invoke_syscall.constprop.0+0x50/0xe4
[ 42.055378]do_el0_svc+0x40/0xc4
[ 42.058691]el0_svc+0x40/0x15c
[ 42.061834]el0t_64_sync_handler+0xa0/0xe4
[ 42.066015]el0t_64_sync+0x198/0x19c
[ 42.069680] Code:aa0603e0d65f03c0f94000008b214000(b9400000)Quoted 11 lines
Quoted 7 lines
TIFS powers off the hardware during deep sleep regardless, since it was never informed to keep the domain active. On resume, because the domain's genpd status is ON, no
get_deviceis issued. The driver then accesses registers of a powered-off domain, causing a synchronous external abort (AXI bus error, ESR0x96000010).Hmm, if something is wakeup source, I would expect even TIFS/DM not to turn if off, else module wakeup wouldn't work.
I tested UART as a wakeup source and I couldn't reproduce this issue. My understanding is that UART has its own TI SCI domain and
device_may_wakeup()is
true directly on that domain device, so theset_device_constraintfires correctly and DM keeps it powered.Here is my tracking of the issue:
Wi-Fi driver registers as wakeup source:
device_init_wakeup(mmc0:0001)During suspend/resume.
dpm_suspend()->genpd_suspend_dev(fa20000.mmc) ->ti_sci_pd_suspend(fa20000.mmc) ->ti_sci_pd_set_wkup_constraint(fa20000.mmc) device_may_wakeup(fa20000.mmc) = falseset_device_constraintnever sent to DM
dpm_suspend_noirq()
->genpd_finish_suspend(fa20000.mmc)
->device_awake_path(fa20000.mmc) = true
->GENPD_FLAG_ACTIVE_WAKEUP = true
genpd status =GENPD_STATE_ON
skippower_off(ti_sci_pd_power_off)
Quoted 8 lines
Quoted 2 lines
On deep sleep entry, DM powers off fa20000.mmc independently. It received no
set_device_constraintnorti_sci_pd_power_off.In AM62P fa20000.mmc is part of main domain. During deepsleep the entire main domain is turned off by the DM, that is why you see the failures.
In-order to debug this we need to check why pd off and pd on call is not getting called for fa20000.mmc during suspend and resume.
This is an expected behavior from genpd. On suspend, ti_sci_pd_power_off is not called because genpd_finish_suspend() takes an early return when both device_awake_path() and GENPD_FLAG_ACTIVE_WAKEUP are true.
On resume, ti_sci_pd_power_on is not called because genpd sees the domain status as GENPD_STATE_ON (it was never cleared) and skips the power-on entirely.
Quoted 6 lines
Quoted 5 lines
I attempted to fix this by calling
set_device_constraintwhendevice_wakeup_path()is true but it prevented the system from entering deep sleep entirely.
Quoted 8 lines
In AM62P the DM manager selects the low power mode to enter based on the constrains set. The mode selection logic will ensure that if a constraint is set on the device, it will select a low power mode in which the device is kept on or can wake the system up. the MMC is part of main domain and there is no low power mode in which the MMC can stay alive or generate a wake up interrupt. so when a constraint is set of MMC, we cannot enter any low power mode. that why you see a failure.
This is consistent with what we observed. I am open to suggestions if there is a better way to handle this.
Thanks,
Vitor Soares
Hi Vitor
Quoted 176 lines · Vitor Soares
Hello Sebin
Quoted 142 lines · Sebin Francis
Hi Vitor,
Quoted 139 lines · Vitor Soares
Hi Vignesh
Thank you for the review.
Quoted 5 lines · Vignesh Raghavendra
Hi Vitor
Quoted 2 lines · Vitor Soares
From: Vitor Soares <[email protected]>
Quoted 8 lines
Quoted 4 lines
When a device in a TI SCI power domain is on the wakeup path of a wakeup-capable child, the suspend path skips
genpd_sync_power_off(). Noput_deviceis sent to TIFS and the domain's genpd status remains ON.Correction of terminologies: TIFS is Root of trust component and is not usually involved in power management, that would be DM (Device Manager)
Thank you for the clarification. I will address this on v2. Also, I was thinking
to replace put_device/get_device with ti_sci_pd_power_off/ti_sci_pd_power_on if
that makes more clear the content.Quoted 3 lines
But to be really sure who is doing what, Could you provide an example and the platform on which you see the issue / external abort?
This was reproduced on our Toradex Verdin AM62P WB and the driver for our Wi-Fi
module on the SDIO bus callsdevice_init_wakeup()during the initialization.After enter in suspend, it show the following error resume path:
[ 41.759341] Internal error: synchronous external abort: 0000000096000010 [#1]
SMP
[ 41.843286] CPU: 0 UID: 0 PID: 933 Comm: rtcwake Tainted: G M O 6.18.21-dirty #3 PREEMPT
[ 41.852762] Tainted: [M]=MACHINE_CHECK, [O]=OOT_MODULE
[ 41.857891] Hardware name: Toradex Verdin AM62P WB on Verdin Development Board (DT)
[ 41.865537] pstate:200000c5(nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=- -)
[ 41.872492] pc :regmap_mmio_read32le+0x8/0x20
[ 41.876941] lr :regmap_mmio_read+0x44/0x70
[ 41.881120] sp :ffff800081fdb8e0
[ 41.884428] x29:ffff800081fdb8e0x28: 0000000000000000 x27:ffffa95bb64aa9c8
[ 41.891563] x26: 0000000000000000 x25: 0000000000000000 x24:
0000000000000000
[ 41.898697] x23: 0000000080000000 x22:ffff000002df5c00x21:ffff800081fdb9b4
[ 41.905831] x20: 0000000000000100 x19:ffff000001286400x18:
0000000000000000
[ 41.912965] x17:2d69696d67722f79x16:687020726f662067x15:ffff00007fb74f40
[ 41.920100] x14:00000000000002eax13:000000000000031fx12:
0000000000000000
[ 41.927234] x11:00000000000000c0x10:00000000000009e0x9 :ffff800081fdb7a0
[ 41.934368] x8 :ffff00007fb6ce00x7 : 0000000000000000 x6 :
0000000000000000
[ 41.941502] x5 :ffffa95bb57948d8x4 : 0000000000000100 x3 :
0000000000000100
[ 41.948636] x2 :ffffa95bb5795034x1 : 0000000000000100 x0 :ffff80008025d100
[ 41.955770] Call trace:
[ 41.958211]regmap_mmio_read32le+0x8/0x20(P)
[ 41.962655]_regmap_bus_reg_read+0x70/0xb0
[ 41.966839]_regmap_read+0x64/0xdc
[ 41.970327]_regmap_update_bits+0xf4/0x140
[ 41.974509]regmap_update_bits_base+0x64/0x98
[ 41.978952]sdhci_am654_runtime_resume+0x138/0x208
[ 41.983830]pm_generic_runtime_resume+0x2c/0x44
[ 41.988445]__genpd_runtime_resume+0x30/0x7c
[ 41.992804]genpd_runtime_resume+0xdc/0x2e8
[ 41.997073]pm_runtime_force_resume+0x68/0xf4
[ 42.001517]dpm_run_callback+0x8c/0x14c
[ 42.005439]device_resume+0x11c/0x34c
[ 42.009188]dpm_resume+0x178/0x1f0
[ 42.012673]dpm_resume_end+0x18/0x34
[ 42.016332]suspend_devices_and_enter+0x4a4/0x668
[ 42.021123]pm_suspend+0x170/0x2dc
[ 42.024610]state_store+0x80/0x104
[ 42.028096]kobj_attr_store+0x18/0x2c
[ 42.031845]sysfs_kf_write+0x7c/0x94
[ 42.035508]kernfs_fop_write_iter+0x130/0x1fc
[ 42.039949]vfs_write+0x200/0x370
[ 42.043351]ksys_write+0x6c/0x100
[ 42.046752]__arm64_sys_write+0x1c/0x28
[ 42.050673] invoke_syscall.constprop.0+0x50/0xe4
[ 42.055378]do_el0_svc+0x40/0xc4
[ 42.058691]el0_svc+0x40/0x15c
[ 42.061834]el0t_64_sync_handler+0xa0/0xe4
[ 42.066015]el0t_64_sync+0x198/0x19c
[ 42.069680] Code:aa0603e0d65f03c0f94000008b214000(b9400000)Quoted 11 lines
Quoted 7 lines
TIFS powers off the hardware during deep sleep regardless, since it was never informed to keep the domain active. On resume, because the domain's genpd status is ON, no
get_deviceis issued. The driver then accesses registers of a powered-off domain, causing a synchronous external abort (AXI bus error, ESR0x96000010).Hmm, if something is wakeup source, I would expect even TIFS/DM not to turn if off, else module wakeup wouldn't work.
I tested UART as a wakeup source and I couldn't reproduce this issue. My understanding is that UART has its own TI SCI domain and
device_may_wakeup()is
true directly on that domain device, so theset_device_constraintfires correctly and DM keeps it powered.Here is my tracking of the issue:
Wi-Fi driver registers as wakeup source:
device_init_wakeup(mmc0:0001)During suspend/resume.
dpm_suspend()->genpd_suspend_dev(fa20000.mmc)
->ti_sci_pd_suspend(fa20000.mmc) ->ti_sci_pd_set_wkup_constraint(fa20000.mmc) device_may_wakeup(fa20000.mmc) = falseset_device_constraintnever sent to DM
dpm_suspend_noirq()
->genpd_finish_suspend(fa20000.mmc)
->device_awake_path(fa20000.mmc) = true
->GENPD_FLAG_ACTIVE_WAKEUP = true
genpd status =GENPD_STATE_ON
skippower_off(ti_sci_pd_power_off)Quoted 8 lines
Quoted 2 lines
On deep sleep entry, DM powers off fa20000.mmc independently. It received no
set_device_constraintnorti_sci_pd_power_off.In AM62P fa20000.mmc is part of main domain. During deepsleep the entire main domain is turned off by the DM, that is why you see the failures.
In-order to debug this we need to check why pd off and pd on call is not getting called for fa20000.mmc during suspend and resume.
This is an expected behavior from genpd. On suspend,
ti_sci_pd_power_offis not called becausegenpd_finish_suspend()takes an early return when bothdevice_awake_path()andGENPD_FLAG_ACTIVE_WAKEUPare true.On resume,
ti_sci_pd_power_onis not called because genpd sees the domain status asGENPD_STATE_ON(it was never cleared) and skips the power-on entirely.Quoted 6 lines
Quoted 5 lines
I attempted to fix this by calling
set_device_constraintwhendevice_wakeup_path()is true but it prevented the system from entering deep sleep entirely.Quoted 8 lines
In AM62P the DM manager selects the low power mode to enter based on the constrains set. The mode selection logic will ensure that if a constraint is set on the device, it will select a low power mode in which the device is kept on or can wake the system up. the MMC is part of main domain and there is no low power mode in which the MMC can stay alive or generate a wake up interrupt. so when a constraint is set of MMC, we cannot enter any low power mode. that why you see a failure.
Quoted 2 lines
This is consistent with what we observed. I am open to suggestions if there is a better way to handle this.
We looked into this issue and found the root-cause. What we have identified is that, there was bug in our ti_sci driver. In ti_sci driver we were not handling the devices in which the child device can act as a wake up source, we only handled the cases in which the device can act as a wake up source. Kendall is working on fix for this. soon she will post the fix.
With the fix, if you suspend it will fail because the wake up is enabled for the WiFi, as a constraint will be sent to the DM firmware and DM will NACK the suspend because it cannot suspend with WiFi acting as a wake up source. But you can disable the wake up of WiFi from the sysfs entry and try suspend then suspend and resume will work.
Thankyou very much for digging deep into this issue and proposing a solution.
Thanks
Sebin
Quoted 4 lines
Thanks,
Vitor Soares
Quoted 204 lines · Sebin Francis
Hi Vitor
Quoted 190 lines · Vitor Soares
Hello Sebin
Quoted 153 lines · Sebin Francis
Hi Vitor,
Quoted 150 lines · Vitor Soares
Hi Vignesh
Thank you for the review.
Quoted 10 lines · Vignesh Raghavendra
Hi Vitor
Quoted 6 lines · Vitor Soares
From: Vitor Soares <[email protected]>
When a device in a TI SCI power domain is on the wakeup path of a wakeup-capable child, the suspend path skips
genpd_sync_power_off(). Noput_deviceis sent to TIFS and the domain's genpd status remains ON.Quoted 5 lines
Correction of terminologies: TIFS is Root of trust component and is not
usually involved in power management, that would be DM (Device Manager)Thank you for the clarification. I will address this on v2. Also, I was thinking
to replace put_device/get_device with ti_sci_pd_power_off/ti_sci_pd_power_on
if
that makes more clear the content.Quoted 3 lines
But to be really sure who is doing what, Could you provide an example and the platform on which you see the issue / external abort?
This was reproduced on our Toradex Verdin AM62P WB and the driver for our
Wi-Fi
module on the SDIO bus callsdevice_init_wakeup()during the initialization.After enter in suspend, it show the following error resume path:
[ 41.759341] Internal error: synchronous external abort:
0000000096000010
[#1]
SMP
[ 41.843286] CPU: 0 UID: 0 PID: 933 Comm: rtcwake Tainted: G M O
6.18.21-dirty #3 PREEMPT
[ 41.852762] Tainted: [M]=MACHINE_CHECK, [O]=OOT_MODULE [ 41.857891] Hardware name: Toradex Verdin AM62P WB on Verdin Development
Board (DT)
[ 41.865537] pstate:200000c5(nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=-
-)
[ 41.872492] pc :regmap_mmio_read32le+0x8/0x20
[ 41.876941] lr :regmap_mmio_read+0x44/0x70
[ 41.881120] sp :ffff800081fdb8e0
[ 41.884428] x29:ffff800081fdb8e0x28: 0000000000000000 x27:ffffa95bb64aa9c8
[ 41.891563] x26: 0000000000000000 x25: 0000000000000000 x24:
0000000000000000
[ 41.898697] x23: 0000000080000000 x22:ffff000002df5c00x21:ffff800081fdb9b4
[ 41.905831] x20: 0000000000000100 x19:ffff000001286400x18:
0000000000000000
[ 41.912965] x17:2d69696d67722f79x16:687020726f662067x15:ffff00007fb74f40
[ 41.920100] x14:00000000000002eax13:000000000000031fx12:
0000000000000000
[ 41.927234] x11:00000000000000c0x10:00000000000009e0x9 :ffff800081fdb7a0
[ 41.934368] x8 :ffff00007fb6ce00x7 : 0000000000000000 x6 :
0000000000000000
[ 41.941502] x5 :ffffa95bb57948d8x4 : 0000000000000100 x3 :
0000000000000100
[ 41.948636] x2 :ffffa95bb5795034x1 : 0000000000000100 x0 :ffff80008025d100
[ 41.955770] Call trace:
[ 41.958211]regmap_mmio_read32le+0x8/0x20(P)
[ 41.962655]_regmap_bus_reg_read+0x70/0xb0
[ 41.966839]_regmap_read+0x64/0xdc
[ 41.970327]_regmap_update_bits+0xf4/0x140
[ 41.974509]regmap_update_bits_base+0x64/0x98
[ 41.978952]sdhci_am654_runtime_resume+0x138/0x208
[ 41.983830]pm_generic_runtime_resume+0x2c/0x44
[ 41.988445]__genpd_runtime_resume+0x30/0x7c
[ 41.992804]genpd_runtime_resume+0xdc/0x2e8
[ 41.997073]pm_runtime_force_resume+0x68/0xf4
[ 42.001517]dpm_run_callback+0x8c/0x14c
[ 42.005439]device_resume+0x11c/0x34c
[ 42.009188]dpm_resume+0x178/0x1f0
[ 42.012673]dpm_resume_end+0x18/0x34
[ 42.016332]suspend_devices_and_enter+0x4a4/0x668
[ 42.021123]pm_suspend+0x170/0x2dc
[ 42.024610]state_store+0x80/0x104
[ 42.028096]kobj_attr_store+0x18/0x2c
[ 42.031845]sysfs_kf_write+0x7c/0x94
[ 42.035508]kernfs_fop_write_iter+0x130/0x1fc
[ 42.039949]vfs_write+0x200/0x370
[ 42.043351]ksys_write+0x6c/0x100
[ 42.046752]__arm64_sys_write+0x1c/0x28
[ 42.050673] invoke_syscall.constprop.0+0x50/0xe4
[ 42.055378]do_el0_svc+0x40/0xc4
[ 42.058691]el0_svc+0x40/0x15c
[ 42.061834]el0t_64_sync_handler+0xa0/0xe4
[ 42.066015]el0t_64_sync+0x198/0x19c
[ 42.069680] Code:aa0603e0d65f03c0f94000008b214000(b9400000)Quoted 11 lines
Quoted 7 lines
TIFS powers off the hardware during deep sleep regardless, since it was never informed to keep the domain active. On resume, because the domain's genpd status is ON, no
get_deviceis issued. The driver then accesses registers of a powered-off domain, causing a synchronous external abort (AXI bus error, ESR0x96000010).Hmm, if something is wakeup source, I would expect even TIFS/DM not to turn if off, else module wakeup wouldn't work.
I tested UART as a wakeup source and I couldn't reproduce this issue. My
understanding is that UART has its own TI SCI domain anddevice_may_wakeup()
is
true directly on that domain device, so theset_device_constraintfires correctly and DM keeps it powered.Here is my tracking of the issue:
Wi-Fi driver registers as wakeup source:
device_init_wakeup(mmc0:0001)During suspend/resume.
dpm_suspend()->genpd_suspend_dev(fa20000.mmc) ->ti_sci_pd_suspend(fa20000.mmc) ->ti_sci_pd_set_wkup_constraint(fa20000.mmc) device_may_wakeup(fa20000.mmc) = falseset_device_constraintnever sent to DM
dpm_suspend_noirq()
->genpd_finish_suspend(fa20000.mmc)
->device_awake_path(fa20000.mmc) = true
->GENPD_FLAG_ACTIVE_WAKEUP = true
genpd status =GENPD_STATE_ON
skippower_off(ti_sci_pd_power_off)Quoted 8 lines
Quoted 2 lines
On deep sleep entry, DM powers off fa20000.mmc independently. It received no
set_device_constraintnorti_sci_pd_power_off.In AM62P fa20000.mmc is part of main domain. During deepsleep the entire main domain is turned off by the DM, that is why you see the failures.
In-order to debug this we need to check why pd off and pd on call is not getting called for fa20000.mmc during suspend and resume.
This is an expected behavior from genpd. On suspend,
ti_sci_pd_power_offis not
called becausegenpd_finish_suspend()takes an early return when bothdevice_awake_path()andGENPD_FLAG_ACTIVE_WAKEUPare true.
On resume,ti_sci_pd_power_onis not called because genpd sees the domain status
asGENPD_STATE_ON(it was never cleared) and skips the power-on entirely.Quoted 7 lines
Quoted 6 lines
I attempted to fix this by calling
set_device_constraintwhendevice_wakeup_path()is true but it prevented the system from entering deep
sleep entirely.Quoted 8 lines
In AM62P the DM manager selects the low power mode to enter based on the constrains set. The mode selection logic will ensure that if a constraint is set on the device, it will select a low power mode in which the device is kept on or can wake the system up. the MMC is part of main domain and there is no low power mode in which the MMC can stay alive or generate a wake up interrupt. so when a constraint is set of MMC, we cannot enter any low power mode. that why you see a failure.
Quoted 3 lines
This is consistent with what we observed. I am open to suggestions if there is a
better way to handle this.We looked into this issue and found the root-cause. What we have identified is that, there was bug in our
ti_scidriver. Inti_scidriver we were not handling the devices in which the child device can act as a wake up source, we only handled the cases in which the device can act as a wake up source. Kendall is working on fix for this. soon she will post the fix.
Quoted 6 lines
With the fix, if you suspend it will fail because the wake up is enabled for the WiFi, as a constraint will be sent to the DM firmware and DM will NACK the suspend because it cannot suspend with WiFi acting as a wake up source. But you can disable the wake up of WiFi from the sysfs entry and try suspend then suspend and resume will work.
To avoid above and still have wakeup from WiFi supported (if needed), this should use IO Daisy chain as the wakeup source and not SDIO directly (MMC IP isnt capable of waking up the system but any activity on the IOPAD can wakeup the system (similar to Main UART)
[...]
Signature · 3 lines
Regards Vignesh https://ti.com/opensource