Skip to contentSkip to conversation

clk: Properly update prepare/enable count on orphan clock reparent

PATCH Superseded Marek Szyprowski 3 participants 7 messages Jan 26, 2018 – Feb 15, 2018 1 message not in the archive Common clk

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): CGME20180126091902eucas1p2b3ad9978e3ab09bd7a2bd9123c9eecaa@eucas1p2.samsung.com

#1 lore raw

PATCH v2 1/1 clk: Properly update prepare/enable count on orphan clock reparent

If orphaned clock has been already prepared/enabled (for example if it or one of its children has CLK_IS_CRITICAL flag), then the prepare/enable counters of the newly assigned parent are not updated correctly. This might later cause warnings during changing clock parents.

This patch fixes following warning on Exynos5422-based boards:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:811 clk_core_disable_lock+0x18/0x24
Modules linked in:
CPU: 0 PID: 59 Comm: kworker/0:1 Not tainted 4.15.0-rc7-next-20180115 #106
Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
Workqueue: pm genpd_power_off_work_fn
[<c0111504>] (unwind_backtrace) from [<c010dbec>] (show_stack+0x10/0x14)
[<c010dbec>] (show_stack) from [<c09b3f34>] (dump_stack+0x90/0xc8)
[<c09b3f34>] (dump_stack) from [<c0125060>] (__warn+0xe4/0x110)
[<c0125060>] (__warn) from [<c01250cc>] (warn_slowpath_null+0x40/0x48)
[<c01250cc>] (warn_slowpath_null) from [<c048d584>] (clk_core_disable_lock+0x18/0x24)
[<c048d584>] (clk_core_disable_lock) from [<c048e490>] (clk_core_disable_unprepare+0xc/0x20)
[<c048e490>] (clk_core_disable_unprepare) from [<c048ea60>] (__clk_set_parent_after+0x48/0x4c)
[<c048ea60>] (__clk_set_parent_after) from [<c048ec88>] (clk_core_set_parent_nolock+0x224/0x5b4)
[<c048ec88>] (clk_core_set_parent_nolock) from [<c048f23c>] (clk_set_parent+0x38/0x6c)
[<c048f23c>] (clk_set_parent) from [<c049c41c>] (exynos_pd_power+0x74/0x1cc)
[<c049c41c>] (exynos_pd_power) from [<c0550768>] (genpd_power_off+0x164/0x264)
[<c0550768>] (genpd_power_off) from [<c0550b34>] (genpd_power_off_work_fn+0x2c/0x40)
[<c0550b34>] (genpd_power_off_work_fn) from [<c01432e4>] (process_one_work+0x1d0/0x7bc)
[<c01432e4>] (process_one_work) from [<c014393c>] (worker_thread+0x34/0x4dc)
[<c014393c>] (worker_thread) from [<c014a0d0>] (kthread+0x128/0x164)
[<c014a0d0>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20)
Exception stack(0xeeb01fb0 to 0xeeb01ff8)
1fa0:                                     00000000 00000000 00000000 00000000
1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
---[ end trace 503c239fb760f17a ]---
------------[ cut here ]------------
WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:684 clk_core_disable_unprepare+0x18/0x20
Modules linked in:
CPU: 0 PID: 59 Comm: kworker/0:1 Tainted: G        W        4.15.0-rc7-next-20180115 #106
Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
Workqueue: pm genpd_power_off_work_fn
[<c0111504>] (unwind_backtrace) from [<c010dbec>] (show_stack+0x10/0x14)
[<c010dbec>] (show_stack) from [<c09b3f34>] (dump_stack+0x90/0xc8)
[<c09b3f34>] (dump_stack) from [<c0125060>] (__warn+0xe4/0x110)
[<c0125060>] (__warn) from [<c01250cc>] (warn_slowpath_null+0x40/0x48)
[<c01250cc>] (warn_slowpath_null) from [<c048e49c>] (clk_core_disable_unprepare+0x18/0x20)
[<c048e49c>] (clk_core_disable_unprepare) from [<c048ea60>] (__clk_set_parent_after+0x48/0x4c)
[<c048ea60>] (__clk_set_parent_after) from [<c048ec88>] (clk_core_set_parent_nolock+0x224/0x5b4)
[<c048ec88>] (clk_core_set_parent_nolock) from [<c048f23c>] (clk_set_parent+0x38/0x6c)
[<c048f23c>] (clk_set_parent) from [<c049c41c>] (exynos_pd_power+0x74/0x1cc)
[<c049c41c>] (exynos_pd_power) from [<c0550768>] (genpd_power_off+0x164/0x264)
[<c0550768>] (genpd_power_off) from [<c0550b34>] (genpd_power_off_work_fn+0x2c/0x40)
[<c0550b34>] (genpd_power_off_work_fn) from [<c01432e4>] (process_one_work+0x1d0/0x7bc)
[<c01432e4>] (process_one_work) from [<c014393c>] (worker_thread+0x34/0x4dc)
[<c014393c>] (worker_thread) from [<c014a0d0>] (kthread+0x128/0x164)
[<c014a0d0>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20)
Exception stack(0xeeb01fb0 to 0xeeb01ff8)
1fa0:                                     00000000 00000000 00000000 00000000
1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
---[ end trace 503c239fb760f17b ]---
------------[ cut here ]------------
Fixes f8f8f1d04494 ("clk: Don't touch hardware when reparenting during registration")

---
v2:
- moved clk_core_prepare(parent) before clk_enable_lock() to avoid calling it under a spinlock

---
 drivers/clk/clk.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/clk/clk.c +4 -0
index 0f686a9dac3e..a14d64fa6a28 100644
@@ -2981,7 +2981,11 @@ static int __clk_core_init(struct clk_core *core)
 		 */
 		if (parent) {
 			/* update the clk tree topology */
+			if (orphan->prepare_count)
+				clk_core_prepare(parent);
 			flags = clk_enable_lock();
+			if (orphan->enable_count)
+				clk_core_enable(parent);
 			clk_reparent(orphan, parent);
 			clk_enable_unlock(flags);
 			__clk_recalc_accuracies(orphan);
Signature · 1 line
2.15.0
#2 lore raw

PATCH v2 1/1 clk: Properly update prepare/enable count on orphan clock reparent

Quoted 4 lines · Marek Szyprowski

If orphaned clock has been already prepared/enabled (for example if it or one of its children has CLK_IS_CRITICAL flag), then the prepare/enable counters of the newly assigned parent are not updated correctly. This might later cause warnings during changing clock parents.

This doesn't feel right. Perhaps we should delay enabling a clk if it's CRITICAL until we adopt an orphaned clk. Good news is we have orphan status tracking now so this should be pretty simple. Otherwise migrating the count up is complicated and requires us to call the prepare/enable ops on a critical clk and then keep doing that each time it gets re-parented. Do you have this case, where some clk is marked as CRITICAL, and then we need to migrate that enable/prepare count to the parent?

Hopefully it isn't the worser case, where the clk is handed out to some consumer but it's still orphaned at that point, and then we have little control over the migration of state to the parent.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#3 lore raw

PATCH v2 1/1 clk: Properly update prepare/enable count on orphan clock reparent

Hi Stephen

Quoted 8 lines · Stephen Boyd
Quoted 4 lines · Marek Szyprowski

If orphaned clock has been already prepared/enabled (for example if it or one of its children has CLK_IS_CRITICAL flag), then the prepare/enable counters of the newly assigned parent are not updated correctly. This might later cause warnings during changing clock parents.

This doesn't feel right. Perhaps we should delay enabling a clk if it's CRITICAL until we adopt an orphaned clk. Good news is we have orphan status tracking now so this should be pretty simple.

Not really. It won't be so simple, because we would need to track the status of the whole clock sub-tree for enabling critical clocks. Even then we would need to delay enabling them until the real top clock is registered.

Quoted 5 lines

Otherwise migrating the count up is complicated and requires us to call the prepare/enable ops on a critical clk and then keep doing that each time it gets re-parented. Do you have this case, where some clk is marked as CRITICAL, and then we need to migrate that enable/prepare count to the parent?

Yes, this is the case of Exynos5422-based boards, where we have a bunch of critical clocks, which are reparented from internal PLL to external oscillator when respective power domain is suspended.

It took me a while to gather all the needed information from the debug logs. Here is one example of such clock tree:

fin_pll (clk1, top clock, external pll)
??? fout_dpll (clk2)
??? ??? mout_sclk_dpll (clk3)
??? ??? ??? mout_aclk300_disp1 (clk4)
??? ??? ??? ??? dout_aclk300_disp1 (clk5)
??? ??? ??? ??? ??? mout_sw_aclk300_disp1 (clk6) ??? ??? ??? ??? ??? ??? mout_user_aclk300_disp1 (clk7) ??? ??? ??? ??? ??? ??? ??? aclk300_disp1 (clk8, critical)

Before turning off power domains, aclk300_disp1 is reparented directly to fin_pll (this is a hardware requirement). When power domain is turned on again, aclk300_disp1 is reparented back to mout_user_aclk300_disp1.

Clocks are registered in the following order:
1, 2, 7, 3, 6, 5, 8, 4.

In this case when critical clock is registered, clk4 is not yet available, so clocks 1-3 won't get proper prepare/enable count. Then this causes a warning during reparenting clk8 to clk1:

?------------[ cut here ]------------ ?WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:811

clk_core_disable_lock+0x18/0x24
 ?Modules linked in:
 ?CPU: 0 PID: 59 Comm: kworker/0:1 Not tainted 

4.15.0-next-20180130-dirty #131
?Hardware name: SAMSUNG EXYNOS (Flattened Device Tree) ?Workqueue: pm genpd_power_off_work_fn
?[<c01114e4>] (unwind_backtrace) from [<c010dbc0>] (show_stack+0x10/0x14) ?[<c010dbc0>] (show_stack) from [<c09b7b94>] (dump_stack+0x90/0xc8) ?[<c09b7b94>] (dump_stack) from [<c0124ff8>] (__warn+0xe4/0x110) ?[<c0124ff8>] (__warn) from [<c0125064>] (warn_slowpath_null+0x40/0x48) ?[<c0125064>] (warn_slowpath_null) from [<c048e650>]
(clk_core_disable_lock+0x18/0x24)
?[<c048e650>] (clk_core_disable_lock) from [<c048f51c>]
(clk_core_disable_unprepare+0xc/0x20)
?[<c048f51c>] (clk_core_disable_unprepare) from [<c048faec>]
(__clk_set_parent_after+0x48/0x4c)
?[<c048faec>] (__clk_set_parent_after) from [<c048fd28>]
(clk_core_set_parent_nolock+0x238/0x5d0)
?[<c048fd28>] (clk_core_set_parent_nolock) from [<c04902e4>]
(clk_set_parent+0x38/0x6c)
?[<c04902e4>] (clk_set_parent) from [<c049d760>]
(exynos_pd_power+0x74/0x1cc)
?[<c049d760>] (exynos_pd_power) from [<c0552064>]
(genpd_power_off+0x164/0x264)
?[<c0552064>] (genpd_power_off) from [<c0552430>]
(genpd_power_off_work_fn+0x2c/0x40)
?[<c0552430>] (genpd_power_off_work_fn) from [<c014355c>]
(process_one_work+0x1d0/0x7bc)
?[<c014355c>] (process_one_work) from [<c0143bb4>]
(worker_thread+0x34/0x4dc)
?[<c0143bb4>] (worker_thread) from [<c014a348>] (kthread+0x128/0x164) ?[<c014a348>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20) ?Exception stack(0xeeb01fb0 to 0xeeb01ff8) ?1fa0:???????????????????????????????????? 00000000 00000000 00000000
00000000
?1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
?1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 ?---[ end trace 48eea511a44c78ef ]---
?------------[ cut here ]------------
?WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:684

clk_core_disable_unprepare+0x18/0x20
 ?Modules linked in:
 ?CPU: 0 PID: 59 Comm: kworker/0:1 Tainted: G??????? W 

4.15.0-next-20180130-dirty #131
?Hardware name: SAMSUNG EXYNOS (Flattened Device Tree) ?Workqueue: pm genpd_power_off_work_fn
?[<c01114e4>] (unwind_backtrace) from [<c010dbc0>] (show_stack+0x10/0x14) ?[<c010dbc0>] (show_stack) from [<c09b7b94>] (dump_stack+0x90/0xc8) ?[<c09b7b94>] (dump_stack) from [<c0124ff8>] (__warn+0xe4/0x110) ?[<c0124ff8>] (__warn) from [<c0125064>] (warn_slowpath_null+0x40/0x48) ?[<c0125064>] (warn_slowpath_null) from [<c048f528>]
(clk_core_disable_unprepare+0x18/0x20)
?[<c048f528>] (clk_core_disable_unprepare) from [<c048faec>]
(__clk_set_parent_after+0x48/0x4c)
?[<c048faec>] (__clk_set_parent_after) from [<c048fd28>]
(clk_core_set_parent_nolock+0x238/0x5d0)
?[<c048fd28>] (clk_core_set_parent_nolock) from [<c04902e4>]
(clk_set_parent+0x38/0x6c)
?[<c04902e4>] (clk_set_parent) from [<c049d760>]
(exynos_pd_power+0x74/0x1cc)
?[<c049d760>] (exynos_pd_power) from [<c0552064>]
(genpd_power_off+0x164/0x264)
?[<c0552064>] (genpd_power_off) from [<c0552430>]
(genpd_power_off_work_fn+0x2c/0x40)
?[<c0552430>] (genpd_power_off_work_fn) from [<c014355c>]
(process_one_work+0x1d0/0x7bc)
?[<c014355c>] (process_one_work) from [<c0143bb4>]
(worker_thread+0x34/0x4dc)
?[<c0143bb4>] (worker_thread) from [<c014a348>] (kthread+0x128/0x164) ?[<c014a348>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20) ?Exception stack(0xeeb01fb0 to 0xeeb01ff8) ?1fa0:???????????????????????????????????? 00000000 00000000 00000000
00000000
?1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
?1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 ?---[ end trace 48eea511a44c78f0 ]---

Please note that this pattern happens for other critical clocks too.

Quoted 3 lines

Hopefully it isn't the worser case, where the clk is handed out to some consumer but it's still orphaned at that point, and then we have little control over the migration of state to the parent.

Well, in my opinion my patch is simplest fix for the regression introduced by commit f8f8f1d04494 ("clk: Don't touch hardware when reparenting during registration"). It also doesn't have any side-effects as it affects only the situation when orphaned clock has been already prepared/enabled, what in practice means it or one of its children has critical flag. Reworking critical clock handling will be much more complicated.

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#4 lore raw

PATCH v2 1/1 clk: Properly update prepare/enable count on orphan clock reparent

Hi Stephen,

Quoted 139 lines · Marek Szyprowski

Hi Stephen

Quoted 9 lines · Stephen Boyd
Quoted 5 lines · Marek Szyprowski

If orphaned clock has been already prepared/enabled (for example if it or
one of its children has CLK_IS_CRITICAL flag), then the prepare/enable counters of the newly assigned parent are not updated correctly. This might later cause warnings during changing clock parents.

This doesn't feel right. Perhaps we should delay enabling a clk if it's CRITICAL until we adopt an orphaned clk. Good news is we have orphan status tracking now so this should be pretty simple.

Not really. It won't be so simple, because we would need to track the status of the whole clock sub-tree for enabling critical clocks. Even then we would need to delay enabling them until the real top clock is registered.

Quoted 5 lines

Otherwise migrating the count up is complicated and requires us to call the prepare/enable ops on a critical clk and then keep doing that each time it gets re-parented. Do you have this case, where some clk is marked as CRITICAL, and then we need to migrate that enable/prepare count to the parent?

Yes, this is the case of Exynos5422-based boards, where we have a bunch of critical clocks, which are reparented from internal PLL to external oscillator when respective power domain is suspended.

It took me a while to gather all the needed information from the debug logs. Here is one example of such clock tree:

fin_pll (clk1, top clock, external pll)
??? fout_dpll (clk2)
??? ??? mout_sclk_dpll (clk3)
??? ??? ??? mout_aclk300_disp1 (clk4)
??? ??? ??? ??? dout_aclk300_disp1 (clk5)
??? ??? ??? ??? ??? mout_sw_aclk300_disp1 (clk6) ??? ??? ??? ??? ??? ??? mout_user_aclk300_disp1 (clk7) ??? ??? ??? ??? ??? ??? ??? aclk300_disp1 (clk8, critical)

Before turning off power domains, aclk300_disp1 is reparented directly to fin_pll (this is a hardware requirement). When power domain is turned on again, aclk300_disp1 is reparented back to mout_user_aclk300_disp1.

Clocks are registered in the following order:
1, 2, 7, 3, 6, 5, 8, 4.

In this case when critical clock is registered, clk4 is not yet available, so clocks 1-3 won't get proper prepare/enable count. Then this causes a warning during reparenting clk8 to clk1:

?------------[ cut here ]------------
?WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:811 clk_core_disable_lock+0x18/0x24
?Modules linked in:
?CPU: 0 PID: 59 Comm: kworker/0:1 Not tainted 4.15.0-next-20180130-dirty #131
?Hardware name: SAMSUNG EXYNOS (Flattened Device Tree) ?Workqueue: pm genpd_power_off_work_fn
?[<c01114e4>] (unwind_backtrace) from [<c010dbc0>] (show_stack+0x10/0x14) ?[<c010dbc0>] (show_stack) from [<c09b7b94>] (dump_stack+0x90/0xc8) ?[<c09b7b94>] (dump_stack) from [<c0124ff8>] (__warn+0xe4/0x110) ?[<c0124ff8>] (__warn) from [<c0125064>] (warn_slowpath_null+0x40/0x48) ?[<c0125064>] (warn_slowpath_null) from [<c048e650>] (clk_core_disable_lock+0x18/0x24)
?[<c048e650>] (clk_core_disable_lock) from [<c048f51c>] (clk_core_disable_unprepare+0xc/0x20)
?[<c048f51c>] (clk_core_disable_unprepare) from [<c048faec>] (__clk_set_parent_after+0x48/0x4c)
?[<c048faec>] (__clk_set_parent_after) from [<c048fd28>] (clk_core_set_parent_nolock+0x238/0x5d0)
?[<c048fd28>] (clk_core_set_parent_nolock) from [<c04902e4>] (clk_set_parent+0x38/0x6c)
?[<c04902e4>] (clk_set_parent) from [<c049d760>] (exynos_pd_power+0x74/0x1cc)
?[<c049d760>] (exynos_pd_power) from [<c0552064>] (genpd_power_off+0x164/0x264)
?[<c0552064>] (genpd_power_off) from [<c0552430>] (genpd_power_off_work_fn+0x2c/0x40)
?[<c0552430>] (genpd_power_off_work_fn) from [<c014355c>] (process_one_work+0x1d0/0x7bc)
?[<c014355c>] (process_one_work) from [<c0143bb4>] (worker_thread+0x34/0x4dc)
?[<c0143bb4>] (worker_thread) from [<c014a348>] (kthread+0x128/0x164) ?[<c014a348>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20) ?Exception stack(0xeeb01fb0 to 0xeeb01ff8) ?1fa0:???????????????????????????????????? 00000000 00000000 00000000 00000000
?1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
?1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 ?---[ end trace 48eea511a44c78ef ]---
?------------[ cut here ]------------
?WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:684 clk_core_disable_unprepare+0x18/0x20
?Modules linked in:
?CPU: 0 PID: 59 Comm: kworker/0:1 Tainted: G??????? W 4.15.0-next-20180130-dirty #131
?Hardware name: SAMSUNG EXYNOS (Flattened Device Tree) ?Workqueue: pm genpd_power_off_work_fn
?[<c01114e4>] (unwind_backtrace) from [<c010dbc0>] (show_stack+0x10/0x14) ?[<c010dbc0>] (show_stack) from [<c09b7b94>] (dump_stack+0x90/0xc8) ?[<c09b7b94>] (dump_stack) from [<c0124ff8>] (__warn+0xe4/0x110) ?[<c0124ff8>] (__warn) from [<c0125064>] (warn_slowpath_null+0x40/0x48) ?[<c0125064>] (warn_slowpath_null) from [<c048f528>] (clk_core_disable_unprepare+0x18/0x20)
?[<c048f528>] (clk_core_disable_unprepare) from [<c048faec>] (__clk_set_parent_after+0x48/0x4c)
?[<c048faec>] (__clk_set_parent_after) from [<c048fd28>] (clk_core_set_parent_nolock+0x238/0x5d0)
?[<c048fd28>] (clk_core_set_parent_nolock) from [<c04902e4>] (clk_set_parent+0x38/0x6c)
?[<c04902e4>] (clk_set_parent) from [<c049d760>] (exynos_pd_power+0x74/0x1cc)
?[<c049d760>] (exynos_pd_power) from [<c0552064>] (genpd_power_off+0x164/0x264)
?[<c0552064>] (genpd_power_off) from [<c0552430>] (genpd_power_off_work_fn+0x2c/0x40)
?[<c0552430>] (genpd_power_off_work_fn) from [<c014355c>] (process_one_work+0x1d0/0x7bc)
?[<c014355c>] (process_one_work) from [<c0143bb4>] (worker_thread+0x34/0x4dc)
?[<c0143bb4>] (worker_thread) from [<c014a348>] (kthread+0x128/0x164) ?[<c014a348>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20) ?Exception stack(0xeeb01fb0 to 0xeeb01ff8) ?1fa0:???????????????????????????????????? 00000000 00000000 00000000 00000000
?1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
?1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 ?---[ end trace 48eea511a44c78f0 ]---

Please note that this pattern happens for other critical clocks too.

Quoted 3 lines

Hopefully it isn't the worser case, where the clk is handed out to some consumer but it's still orphaned at that point, and then we have little control over the migration of state to the parent.

Quoted 7 lines

Well, in my opinion my patch is simplest fix for the regression introduced by commit f8f8f1d04494 ("clk: Don't touch hardware when reparenting during registration"). It also doesn't have any side-effects as it affects only the situation when orphaned clock has been already prepared/enabled, what in practice means it or one of its children has critical flag. Reworking critical clock handling will be much more complicated.

Stephen, do you have any comments?

The issue is there for almost 2 months... v4.16-rc1 is out and its a good time to get a fix in (this or any other if you propose such).

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#5 lore raw

PATCH v2 1/1 clk: Properly update prepare/enable count on orphan clock reparent

Quoted 151 lines · Marek Szyprowski

Hi Stephen,

Quoted 140 lines · Marek Szyprowski

Hi Stephen

Quoted 10 lines · Stephen Boyd
Quoted 5 lines · Marek Szyprowski

If orphaned clock has been already prepared/enabled (for example if it or
one of its children has CLK_IS_CRITICAL flag), then the prepare/enable counters of the newly assigned parent are not updated correctly. This might later cause warnings during changing clock parents.

This doesn't feel right. Perhaps we should delay enabling a clk if it's CRITICAL until we adopt an orphaned clk. Good news is we have orphan status tracking now so this should be pretty simple.

Not really. It won't be so simple, because we would need to track the status of the whole clock sub-tree for enabling critical clocks. Even then we would need to delay enabling them until the real top clock is registered.

Quoted 5 lines

Otherwise migrating the count up is complicated and requires us to call the prepare/enable ops on a critical clk and then keep doing that each time it gets re-parented. Do you have this case, where some clk is marked as CRITICAL, and then we need to migrate that enable/prepare count to the parent?

Yes, this is the case of Exynos5422-based boards, where we have a bunch of critical clocks, which are reparented from internal PLL to external oscillator when respective power domain is suspended.

It took me a while to gather all the needed information from the debug logs. Here is one example of such clock tree:

fin_pll (clk1, top clock, external pll)

    fout_dpll (clk2)
        mout_sclk_dpll (clk3)
            mout_aclk300_disp1 (clk4)
                dout_aclk300_disp1 (clk5)
                    mout_sw_aclk300_disp1 (clk6)
                        mout_user_aclk300_disp1 (clk7)
                            aclk300_disp1 (clk8, critical)

Before turning off power domains, aclk300_disp1 is reparented directly to fin_pll (this is a hardware requirement). When power domain is turned on again, aclk300_disp1 is reparented back to mout_user_aclk300_disp1.

Clocks are registered in the following order:
1, 2, 7, 3, 6, 5, 8, 4.

In this case when critical clock is registered, clk4 is not yet available, so clocks 1-3 won't get proper prepare/enable count. Then this causes a warning during reparenting clk8 to clk1:

 ------------[ cut here ]------------
 WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:811 
clk_core_disable_lock+0x18/0x24
 Modules linked in:
 CPU: 0 PID: 59 Comm: kworker/0:1 Not tainted 
4.15.0-next-20180130-dirty #131
 Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
 Workqueue: pm genpd_power_off_work_fn
 [<c01114e4>] (unwind_backtrace) from [<c010dbc0>] (show_stack+0x10/0x14)
 [<c010dbc0>] (show_stack) from [<c09b7b94>] (dump_stack+0x90/0xc8)
 [<c09b7b94>] (dump_stack) from [<c0124ff8>] (__warn+0xe4/0x110)
 [<c0124ff8>] (__warn) from [<c0125064>] (warn_slowpath_null+0x40/0x48)
 [<c0125064>] (warn_slowpath_null) from [<c048e650>] 
(clk_core_disable_lock+0x18/0x24)
 [<c048e650>] (clk_core_disable_lock) from [<c048f51c>] 
(clk_core_disable_unprepare+0xc/0x20)
 [<c048f51c>] (clk_core_disable_unprepare) from [<c048faec>] 
(__clk_set_parent_after+0x48/0x4c)
 [<c048faec>] (__clk_set_parent_after) from [<c048fd28>] 
(clk_core_set_parent_nolock+0x238/0x5d0)
 [<c048fd28>] (clk_core_set_parent_nolock) from [<c04902e4>] 
(clk_set_parent+0x38/0x6c)
 [<c04902e4>] (clk_set_parent) from [<c049d760>] 
(exynos_pd_power+0x74/0x1cc)
 [<c049d760>] (exynos_pd_power) from [<c0552064>] 
(genpd_power_off+0x164/0x264)
 [<c0552064>] (genpd_power_off) from [<c0552430>] 
(genpd_power_off_work_fn+0x2c/0x40)
 [<c0552430>] (genpd_power_off_work_fn) from [<c014355c>] 
(process_one_work+0x1d0/0x7bc)
 [<c014355c>] (process_one_work) from [<c0143bb4>] 
(worker_thread+0x34/0x4dc)
 [<c0143bb4>] (worker_thread) from [<c014a348>] (kthread+0x128/0x164)
 [<c014a348>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20)
 Exception stack(0xeeb01fb0 to 0xeeb01ff8)
 1fa0:                                     00000000 00000000 00000000 

00000000
1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
1fe0: 00000000 00000000 00000000 00000000 00000013 00000000

 ---[ end trace 48eea511a44c78ef ]---
 ------------[ cut here ]------------
 WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:684 
clk_core_disable_unprepare+0x18/0x20
 Modules linked in:
 CPU: 0 PID: 59 Comm: kworker/0:1 Tainted: G        W 
4.15.0-next-20180130-dirty #131
 Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
 Workqueue: pm genpd_power_off_work_fn
 [<c01114e4>] (unwind_backtrace) from [<c010dbc0>] (show_stack+0x10/0x14)
 [<c010dbc0>] (show_stack) from [<c09b7b94>] (dump_stack+0x90/0xc8)
 [<c09b7b94>] (dump_stack) from [<c0124ff8>] (__warn+0xe4/0x110)
 [<c0124ff8>] (__warn) from [<c0125064>] (warn_slowpath_null+0x40/0x48)
 [<c0125064>] (warn_slowpath_null) from [<c048f528>] 
(clk_core_disable_unprepare+0x18/0x20)
 [<c048f528>] (clk_core_disable_unprepare) from [<c048faec>] 
(__clk_set_parent_after+0x48/0x4c)
 [<c048faec>] (__clk_set_parent_after) from [<c048fd28>] 
(clk_core_set_parent_nolock+0x238/0x5d0)
 [<c048fd28>] (clk_core_set_parent_nolock) from [<c04902e4>] 
(clk_set_parent+0x38/0x6c)
 [<c04902e4>] (clk_set_parent) from [<c049d760>] 
(exynos_pd_power+0x74/0x1cc)
 [<c049d760>] (exynos_pd_power) from [<c0552064>] 
(genpd_power_off+0x164/0x264)
 [<c0552064>] (genpd_power_off) from [<c0552430>] 
(genpd_power_off_work_fn+0x2c/0x40)
 [<c0552430>] (genpd_power_off_work_fn) from [<c014355c>] 
(process_one_work+0x1d0/0x7bc)
 [<c014355c>] (process_one_work) from [<c0143bb4>] 
(worker_thread+0x34/0x4dc)
 [<c0143bb4>] (worker_thread) from [<c014a348>] (kthread+0x128/0x164)
 [<c014a348>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20)
 Exception stack(0xeeb01fb0 to 0xeeb01ff8)
 1fa0:                                     00000000 00000000 00000000 

00000000
1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
1fe0: 00000000 00000000 00000000 00000000 00000013 00000000

 ---[ end trace 48eea511a44c78f0 ]---

Please note that this pattern happens for other critical clocks too.

Quoted 3 lines

Hopefully it isn't the worser case, where the clk is handed out to some consumer but it's still orphaned at that point, and then we have little control over the migration of state to the parent.

Quoted 7 lines

Well, in my opinion my patch is simplest fix for the regression introduced by commit f8f8f1d04494 ("clk: Don't touch hardware when reparenting during registration"). It also doesn't have any side-effects as it affects only the situation when orphaned clock has been already prepared/enabled, what in practice means it or one of its children has critical flag. Reworking critical clock handling will be much more complicated.

Quoted 6 lines

Stephen, do you have any comments?

The issue is there for almost 2 months... v4.16-rc1 is out and its a good time to get a fix in (this or any other if you propose such).

Best regards

Hi Marek,

I've sent another patch addressing the same issue (sorry, I did not see your thread earlier)

While I agree parent adopting orphan need to take care of any existing count, your solution could be missing a few things, like dealing with CLK_OPS_PARENT_ENABLE clock types

Fortunately, __clk_set_parent_before() and __clk_set_parent_after() already deal with those annoying stuff and does count migration well.

Would you mind trying this change [0] and let me know if works for you as well ?

[0] : https://lkml.kernel.org/r/20180214134340.17242-5-jbrunet at baylibre.com

Cheers
Jerome

#6 lore raw

PATCH v2 1/1 clk: Properly update prepare/enable count on orphan clock reparent

Hi Jerome,

Quoted 158 lines · Jerome Brunet
Quoted 142 lines · Marek Szyprowski
Quoted 134 lines · Marek Szyprowski
Quoted 9 lines · Stephen Boyd
Quoted 5 lines · Marek Szyprowski

If orphaned clock has been already prepared/enabled (for example if it or
one of its children has CLK_IS_CRITICAL flag), then the prepare/enable counters of the newly assigned parent are not updated correctly. This might later cause warnings during changing clock parents.

This doesn't feel right. Perhaps we should delay enabling a clk if it's CRITICAL until we adopt an orphaned clk. Good news is we have orphan status tracking now so this should be pretty simple.

Not really. It won't be so simple, because we would need to track the status of the whole clock sub-tree for enabling critical clocks. Even then we would need to delay enabling them until the real top clock is registered.

Quoted 5 lines

Otherwise migrating the count up is complicated and requires us to call the prepare/enable ops on a critical clk and then keep doing that each time it gets re-parented. Do you have this case, where some clk is marked as CRITICAL, and then we need to migrate that enable/prepare count to the parent?

Yes, this is the case of Exynos5422-based boards, where we have a bunch of critical clocks, which are reparented from internal PLL to external oscillator when respective power domain is suspended.

It took me a while to gather all the needed information from the debug logs. Here is one example of such clock tree:

fin_pll (clk1, top clock, external pll)

     fout_dpll (clk2)
         mout_sclk_dpll (clk3)
             mout_aclk300_disp1 (clk4)
                 dout_aclk300_disp1 (clk5)
                     mout_sw_aclk300_disp1 (clk6)
                         mout_user_aclk300_disp1 (clk7)
                             aclk300_disp1 (clk8, critical)

Before turning off power domains, aclk300_disp1 is reparented directly to fin_pll (this is a hardware requirement). When power domain is turned on again, aclk300_disp1 is reparented back to mout_user_aclk300_disp1.

Clocks are registered in the following order:
1, 2, 7, 3, 6, 5, 8, 4.

In this case when critical clock is registered, clk4 is not yet available, so clocks 1-3 won't get proper prepare/enable count. Then this causes a warning during reparenting clk8 to clk1:

  ------------[ cut here ]------------
  WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:811
clk_core_disable_lock+0x18/0x24
  Modules linked in:
  CPU: 0 PID: 59 Comm: kworker/0:1 Not tainted
4.15.0-next-20180130-dirty #131
  Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
  Workqueue: pm genpd_power_off_work_fn
  [<c01114e4>] (unwind_backtrace) from [<c010dbc0>] (show_stack+0x10/0x14)
  [<c010dbc0>] (show_stack) from [<c09b7b94>] (dump_stack+0x90/0xc8)
  [<c09b7b94>] (dump_stack) from [<c0124ff8>] (__warn+0xe4/0x110)
  [<c0124ff8>] (__warn) from [<c0125064>] (warn_slowpath_null+0x40/0x48)
  [<c0125064>] (warn_slowpath_null) from [<c048e650>]
(clk_core_disable_lock+0x18/0x24)
  [<c048e650>] (clk_core_disable_lock) from [<c048f51c>]
(clk_core_disable_unprepare+0xc/0x20)
  [<c048f51c>] (clk_core_disable_unprepare) from [<c048faec>]
(__clk_set_parent_after+0x48/0x4c)
  [<c048faec>] (__clk_set_parent_after) from [<c048fd28>]
(clk_core_set_parent_nolock+0x238/0x5d0)
  [<c048fd28>] (clk_core_set_parent_nolock) from [<c04902e4>]
(clk_set_parent+0x38/0x6c)
  [<c04902e4>] (clk_set_parent) from [<c049d760>]
(exynos_pd_power+0x74/0x1cc)
  [<c049d760>] (exynos_pd_power) from [<c0552064>]
(genpd_power_off+0x164/0x264)
  [<c0552064>] (genpd_power_off) from [<c0552430>]
(genpd_power_off_work_fn+0x2c/0x40)
  [<c0552430>] (genpd_power_off_work_fn) from [<c014355c>]
(process_one_work+0x1d0/0x7bc)
  [<c014355c>] (process_one_work) from [<c0143bb4>]
(worker_thread+0x34/0x4dc)
  [<c0143bb4>] (worker_thread) from [<c014a348>] (kthread+0x128/0x164)
  [<c014a348>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20)
  Exception stack(0xeeb01fb0 to 0xeeb01ff8)
  1fa0:                                     00000000 00000000 00000000

00000000
1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
1fe0: 00000000 00000000 00000000 00000000 00000013 00000000

  ---[ end trace 48eea511a44c78ef ]---
  ------------[ cut here ]------------
  WARNING: CPU: 0 PID: 59 at drivers/clk/clk.c:684
clk_core_disable_unprepare+0x18/0x20
  Modules linked in:
  CPU: 0 PID: 59 Comm: kworker/0:1 Tainted: G        W
4.15.0-next-20180130-dirty #131
  Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
  Workqueue: pm genpd_power_off_work_fn
  [<c01114e4>] (unwind_backtrace) from [<c010dbc0>] (show_stack+0x10/0x14)
  [<c010dbc0>] (show_stack) from [<c09b7b94>] (dump_stack+0x90/0xc8)
  [<c09b7b94>] (dump_stack) from [<c0124ff8>] (__warn+0xe4/0x110)
  [<c0124ff8>] (__warn) from [<c0125064>] (warn_slowpath_null+0x40/0x48)
  [<c0125064>] (warn_slowpath_null) from [<c048f528>]
(clk_core_disable_unprepare+0x18/0x20)
  [<c048f528>] (clk_core_disable_unprepare) from [<c048faec>]
(__clk_set_parent_after+0x48/0x4c)
  [<c048faec>] (__clk_set_parent_after) from [<c048fd28>]
(clk_core_set_parent_nolock+0x238/0x5d0)
  [<c048fd28>] (clk_core_set_parent_nolock) from [<c04902e4>]
(clk_set_parent+0x38/0x6c)
  [<c04902e4>] (clk_set_parent) from [<c049d760>]
(exynos_pd_power+0x74/0x1cc)
  [<c049d760>] (exynos_pd_power) from [<c0552064>]
(genpd_power_off+0x164/0x264)
  [<c0552064>] (genpd_power_off) from [<c0552430>]
(genpd_power_off_work_fn+0x2c/0x40)
  [<c0552430>] (genpd_power_off_work_fn) from [<c014355c>]
(process_one_work+0x1d0/0x7bc)
  [<c014355c>] (process_one_work) from [<c0143bb4>]
(worker_thread+0x34/0x4dc)
  [<c0143bb4>] (worker_thread) from [<c014a348>] (kthread+0x128/0x164)
  [<c014a348>] (kthread) from [<c01010b4>] (ret_from_fork+0x14/0x20)
  Exception stack(0xeeb01fb0 to 0xeeb01ff8)
  1fa0:                                     00000000 00000000 00000000

00000000
1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
1fe0: 00000000 00000000 00000000 00000000 00000013 00000000

  ---[ end trace 48eea511a44c78f0 ]---

Please note that this pattern happens for other critical clocks too.

Quoted 3 lines

Hopefully it isn't the worser case, where the clk is handed out to some consumer but it's still orphaned at that point, and then we have little control over the migration of state to the parent.

Quoted 7 lines

Well, in my opinion my patch is simplest fix for the regression introduced by commit f8f8f1d04494 ("clk: Don't touch hardware when reparenting during registration"). It also doesn't have any side-effects as it affects only the situation when orphaned clock has been already prepared/enabled, what in practice means it or one of its children has critical flag. Reworking critical clock handling will be much more complicated.

Quoted 6 lines

Stephen, do you have any comments?

The issue is there for almost 2 months... v4.16-rc1 is out and its a good time to get a fix in (this or any other if you propose such).

Best regards

Hi Marek,

I've sent another patch addressing the same issue (sorry, I did not see your thread earlier)

While I agree parent adopting orphan need to take care of any existing count, your solution could be missing a few things, like dealing with CLK_OPS_PARENT_ENABLE clock types

Quoted 6 lines

Fortunately, __clk_set_parent_before() and __clk_set_parent_after() already deal with those annoying stuff and does count migration well.

Would you mind trying this change [0] and let me know if works for you as well ?

[0] : https://lkml.kernel.org/r/20180214134340.17242-5-jbrunet at baylibre.com

I'm fine with your approach. It fixes the issue observed on Exynos5422-based boards.

Best regards

Signature · 2 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland
#7 lore raw

PATCH v2 1/1 clk: Properly update prepare/enable count on orphan clock reparent

In reply to Stephen Boyd
Quoted 8 lines · Stephen Boyd
Quoted 4 lines · Marek Szyprowski

If orphaned clock has been already prepared/enabled (for example if it or one of its children has CLK_IS_CRITICAL flag), then the prepare/enable counters of the newly assigned parent are not updated correctly. This might later cause warnings during changing clock parents.

This doesn't feel right. Perhaps we should delay enabling a clk if it's CRITICAL until we adopt an orphaned clk.

It does not sounds right. A critical clock should be enabled as soon as possible, regardless of current knowledge of the clock tree.

How would you decide when is right time anyway ?
The orphaned critical clock could be reparented to clock that is orphan itself, you would get into the same count issue again.

Quoted 5 lines

Good news is we
have orphan status tracking now so this should be pretty simple. Otherwise migrating the count up is complicated and requires us to call the prepare/enable ops on a critical clk and then keep doing that each time it gets re-parented.

What is the problem with this ? We are going to have call those enable ops anyway. Whether it is done one at a time or all at once, does it really matter ?

Quoted 3 lines

Do you have this case,
where some clk is marked as CRITICAL, and then we need to migrate that enable/prepare count to the parent?

yes.

Quoted 5 lines

Hopefully it isn't the worser case, where the clk is handed out to some consumer but it's still orphaned at that point, and then we have little control over the migration of state to the parent.

This should not be a problem if the clock reparenting and count migration is done properly. To do this, the __clk_set_parent_before() and __clk_set_parent_after() function look good to me, have I missed anything ?