Skip to contentSkip to conversation

clk: rockchip: implement proper GATE_LINK support

PATCH Sebastian Reichel 3 participants 11 messages Mar 8, 2024 – Mar 22, 2024 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

[removed DT people from CC, since this is all about clocks now]

Hello Ilya,

Quoted 3 lines · Ilya K

This change seems to make my Orange Pi 5 (RK3588S) lock up on boot :( Any suggestions on how to debug this? It doesn't really log much...

I suppose with this change you explicitly mean the last patch, which has not yet been applied? That patch effectively allows some clocks to be disabled, that have previously been marked as critical (and thus always-on).

If that results in a boot lockup, I expect you have a peripheral driver, that does not enable a required clock (e.g. because of a missing clock reference).

I assume you have a bunch of other patches applied on top of upstream and not just this one? In that case it might be sensible to first apply this patch, do a test boot (it hopefully boots). Then step-by-step re-add the other patches and check which one breaks when being combined with this patch.

P.S.: I plan to send a v9 of this patch once 6.9-rc1 has been tagged and will split it, so that the removal of CLK_IS_CRITICAL is done in a separate patch.

Greetings,

-- Sebastian

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

Hi,

Quoted 19 lines · Ilya K
Quoted 13 lines
Quoted 4 lines

This change seems to make my Orange Pi 5 (RK3588S) lock up on boot :( Any suggestions on how to debug this? It doesn't really log much...

I suppose with this change you explicitly mean the last patch, which has not yet been applied? That patch effectively allows some clocks to be disabled, that have previously been marked as critical (and thus always-on).

If that results in a boot lockup, I expect you have a peripheral driver, that does not enable a required clock (e.g. because of a missing clock reference).

Quoted 6 lines

I assume you have a bunch of other patches applied on top of upstream and not just this one? In that case it might be sensible to first apply this patch, do a test boot (it hopefully boots). Then step-by-step re-add the other patches and check which one breaks when being combined with this patch.

Quoted 3 lines

Thanks! I think the email bounced from most of the mailing lists anyway... The exact tree I'm running is here: https://github.com/K900/linux/commits/
rk3588-test/ and the last revert commit is what makes things boot again.

Please update your mailer to send plain text mails instead of HTML when working with the kernel mailing lists. Also please do not top-post.

The tree seems to be more or less what is working on Rock 5B and EVB1. Please try a partial revert, which just undos the removal of RK3588_LINKED_CLK (the define and the usage of the define). If that works the next step would be to step-by-step remove usage of RK3588_LINKED_CLK again until finding the clock that needs to be critical.

Thanks,

-- Sebastian

#4 lore raw
In reply to Sebastian Reichel
Quoted 9 lines · Sebastian Reichel

[removed DT people from CC, since this is all about clocks now]

Hello Ilya,

Quoted 3 lines · Ilya K

This change seems to make my Orange Pi 5 (RK3588S) lock up on boot :( Any suggestions on how to debug this? It doesn't really log much...

Quoted 8 lines

I suppose with this change you explicitly mean the last patch, which has not yet been applied? That patch effectively allows some clocks to be disabled, that have previously been marked as critical (and thus always-on).

If that results in a boot lockup, I expect you have a peripheral driver, that does not enable a required clock (e.g. because of a missing clock reference).

Sebastian,

Another Orange Pi 5 owner here... I likewise ran into trouble with this patch. I have been playing with your rk3588 branch from the Collabora gitlab and was also experiencing hangs on boot on my OPi5. I bisected your branch and found that the GATE_LINK support commit was the problem for me.

Digging in, I found that the problem behind the hang was that I was not getting the aclk_nvm_root which in turn caused DMA transactions to the SFC driver to hang.

...

[   2.804519] rockchip-sfc fe2b0000.spi: DMA wait for transfer finish
timeout
[   2.805127] rockchip-sfc fe2b0000.spi: xfer data failed ret -110 dir 1

...

Setting aclk_nvm_root to critical allows the system to boot. However not all is well as I do get errors like the following which also seem to indicate further clock problems:

[   6.296554] rockchip-pm-domain

fd8d8000.power-management:power-controller: failed to set idle on domain 'vo0', val=0

I assume that the con-ops of your patch is analogous to the downstream "clock-link" driver. (ie. you are looking for PM events to cause the pm_clk_suspend()/pm_clk_resume() routines to enable/disable the linked (second parent) clocks). This wasn't happening for me. Unlike in the downstream implementation, the gate_link devices' dev->driver pointer was null. So that when rpm_resume() was being called, it would make its way to pm_generic_runtime_resume() which then would do nothing. The end result is that I would have a prepare count of 1 on aclk_nvm_root, but an enable count of 0 so it would get disabled (orig patch w/o marking this critical).

I did a quick hacky proof of concept where I set up the pm ops on the clk-rk3588 driver to point to the pm_clk_suspend/pm_clk_resume routines like downstream did. I used device_bind_driver() to forceably bind the clk-rk3588 driver to the gate link devices it was setting up.

This worked for me. With those changes I then was able to boot without needing to set aclk_nvm_root as critical. Further, it cleaned up the power-controller error messages on the sdio and vo0 domains.

The force bind I did doesn't feel like a particularly clean solution to me. However, I am not knowledgeable enough in the (platform) device model to know what the right way to do this is without fully doing what downstream did and having DT nodes for the various gate link devices. But it seems to prove out that having the driver bound to the devices with the relevant runtime pm ops is the missing piece in my use case on the OPi5.

Hope this additional data point helps!

--
Chad LeClair

#5 lore raw

Hello Chad,

Quoted 19 lines · Chad LeClair
Quoted 9 lines · Sebastian Reichel

[removed DT people from CC, since this is all about clocks now]

Hello Ilya,

Quoted 3 lines · Ilya K

This change seems to make my Orange Pi 5 (RK3588S) lock up on boot :( Any suggestions on how to debug this? It doesn't really log much...

Quoted 8 lines

I suppose with this change you explicitly mean the last patch, which has not yet been applied? That patch effectively allows some clocks to be disabled, that have previously been marked as critical (and thus always-on).

If that results in a boot lockup, I expect you have a peripheral driver, that does not enable a required clock (e.g. because of a missing clock reference).

Quoted 5 lines

Another Orange Pi 5 owner here... I likewise ran into trouble with this patch. I have been playing with your rk3588 branch from the Collabora gitlab and was also experiencing hangs on boot on my OPi5. I bisected your branch and found that the GATE_LINK support commit was the problem for me.

Ok.

Quoted 2 lines

Digging in, I found that the problem behind the hang was that I was not getting the aclk_nvm_root

That matches what Ilya found (there were some more off-list messages because of HTML mail issues). I suspected, that HCLK_SD (which is sourced from SCMI) might need HCLK_NVM.

Quoted 6 lines

which in turn caused DMA transactions to the SFC driver to hang.

...

[   2.804519] rockchip-sfc fe2b0000.spi: DMA wait for transfer finish
timeout
[   2.805127] rockchip-sfc fe2b0000.spi: xfer data failed ret -110 dir 1
Quoted 8 lines

...

Setting aclk_nvm_root to critical allows the system to boot. However not all is well as I do get errors like the following which also seem to indicate further clock problems:

[   6.296554] rockchip-pm-domain

fd8d8000.power-management:power-controller: failed to set idle on domain 'vo0', val=0

I see the same message for AV1 and I've also seen failed idle message with the rockchip kernel. I have not yet debugged what is going on with that.

Quoted 20 lines

I assume that the con-ops of your patch is analogous to the downstream "clock-link" driver. (ie. you are looking for PM events to cause the pm_clk_suspend()/pm_clk_resume() routines to enable/disable the linked (second parent) clocks). This wasn't happening for me. Unlike in the downstream implementation, the gate_link devices' dev->driver pointer was null. So that when rpm_resume() was being called, it would make its way to pm_generic_runtime_resume() which then would do nothing. The end result is that I would have a prepare count of 1 on aclk_nvm_root, but an enable count of 0 so it would get disabled (orig patch w/o marking this critical).

I did a quick hacky proof of concept where I set up the pm ops on the clk-rk3588 driver to point to the pm_clk_suspend/pm_clk_resume routines like downstream did. I used device_bind_driver() to forceably bind the clk-rk3588 driver to the gate link devices it was setting up.

This worked for me. With those changes I then was able to boot without needing to set aclk_nvm_root as critical. Further, it cleaned up the power-controller error messages on the sdio and vo0 domains.

Quoted 7 lines

The force bind I did doesn't feel like a particularly clean solution to me. However, I am not knowledgeable enough in the (platform) device model to know what the right way to do this is without fully doing what downstream did and having DT nodes for the various gate link devices. But it seems to prove out that having the driver bound to the devices with the relevant runtime pm ops is the missing piece in my use case on the OPi5.

Can you test again using rk3588-test [0] (which is based on today's master branch) from today as base? Ideally it should work with your Orange Pi 5, since it contains "clk: rockchip: rk3588: mark hclk_nvm critical".

I also worked on a cleaner solution for the issue you described and integrated it in the patch adding proper gate clock support. So please also test with HCLK_NVM not being marked as CRITICAL.

[0] https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commits/rk3588-test

If everything is fine I will prepare a v9.

Quoted 1 line

Hope this additional data point helps!

Thanks for the detailed analysis.

Greetings,

-- Sebastian

#6 lore raw
Quoted 5 lines · Sebastian Reichel

Can you test again using rk3588-test [0] (which is based on today's master branch) from today as base? Ideally it should work with your Orange Pi 5, since it contains "clk: rockchip: rk3588: mark hclk_nvm critical".

Quoted 8 lines

I also worked on a cleaner solution for the issue you described and integrated it in the patch adding proper gate clock support. So please also test with HCLK_NVM not being marked as CRITICAL.

[0] https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commits/rk3588-test

If everything is fine I will prepare a v9.

Tested this on my end, and I'm getting the same SFC issue, so there's probably another clock that's missing somewhere. Building with ROCKCHIP_SFC=y (instead of =m) removes the error message, but any access to the flash just locks up, so it's probably placebo, unless there's a second, unrelated issue.

#7 lore raw
In reply to Sebastian Reichel

Sebastian,

Quoted 8 lines · Sebastian Reichel

I also worked on a cleaner solution for the issue you described and integrated it in the patch adding proper gate clock support. So please also test with HCLK_NVM not being marked as CRITICAL.

[0] https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commits/rk3588-test

If everything is fine I will prepare a v9.

Quoted 7 lines
Quoted 1 line

Hope this additional data point helps!

Thanks for the detailed analysis.

Greetings,

-- Sebastian

No luck unfortunately. I still see the SFC dma timeouts. It looks like aclk_nvm_root is still getting disabled. It now has both an enable count of 0 and a prepare count of 0.

Unlike your previous version, I _do_ see the driver bound to the device and the rpm_resume() call finds its way to pm_clk_resume(). So it looks like you resolved the original issue I was seeing.

However, when I reach __pm_clk_enable() it looks like the clock entry (ce) is not in a good state:
(gdb) print *ce
$3 = {node = {next = 0xffff0001f0ce1930, prev = 0xffff0001f0ce1930}, con_id = 0xffff0001f0a214f0 "aclk_nvm_root", clk = 0xfffffffffffffffe, status = PCE_STATUS_ERROR, enabled_when_prepared = false}

The immediate problem at hand is that ce->status is PCE_STATUS_ERROR so the switch statement will take the default case and return without doing anything. Also the ce->clk pointer looks like some sort of error pointer value so I'm wondering if something went wrong in the setup you were doing in clk_gate_link_probe().

Note: I ran to that same __pm_clk_enable() breakpoint for a number of of GATE_LINK clocks. They all looked to be in that same bad state. I put the "aclk_nvm_root" one in the message here since that is the one that is most relevant to the discussion, but they all look to be broken in the same way.

Hopefully this gives you a hint as to what is going on.

BTW, In case it is of interest to you or Ilya, I have my "working" tree available here:
https://github.com/sesca128/linux-rk3588/tree/v6.8-rk3588-test

My quick and dirty hack is at the tip of the branch. The rest is pretty much your rk3588-test branch from a few days ago.

--
Chad LeClair

#8 lore raw

Hi Chad and Ilya,

Quoted 42 lines · Chad LeClair

Sebastian,

Quoted 8 lines · Sebastian Reichel

I also worked on a cleaner solution for the issue you described and integrated it in the patch adding proper gate clock support. So please also test with HCLK_NVM not being marked as CRITICAL.

[0] https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commits/rk3588-test

If everything is fine I will prepare a v9.

Quoted 7 lines
Quoted 1 line

Hope this additional data point helps!

Thanks for the detailed analysis.

Greetings,

-- Sebastian

No luck unfortunately. I still see the SFC dma timeouts. It looks like aclk_nvm_root is still getting disabled. It now has both an enable count of 0 and a prepare count of 0.

Unlike your previous version, I _do_ see the driver bound to the device and the rpm_resume() call finds its way to pm_clk_resume(). So it looks like you resolved the original issue I was seeing.

However, when I reach __pm_clk_enable() it looks like the clock entry (ce) is not in a good state:
(gdb) print *ce
$3 = {node = {next = 0xffff0001f0ce1930, prev = 0xffff0001f0ce1930}, con_id = 0xffff0001f0a214f0 "aclk_nvm_root", clk = 0xfffffffffffffffe, status = PCE_STATUS_ERROR, enabled_when_prepared = false}

The immediate problem at hand is that ce->status is PCE_STATUS_ERROR so the switch statement will take the default case and return without doing anything. Also the ce->clk pointer looks like some sort of error pointer value so I'm wondering if something went wrong in the setup you were doing in clk_gate_link_probe().

Quoted 6 lines

Note: I ran to that same __pm_clk_enable() breakpoint for a number of of GATE_LINK clocks. They all looked to be in that same bad state. I put the "aclk_nvm_root" one in the message here since that is the one that is most relevant to the discussion, but they all look to be broken in the same way.

Hopefully this gives you a hint as to what is going on.

Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the GATE_LINK code into the separate driver now. Please give it another try.

Greetings,

-- Sebastian

#9 lore raw
Quoted 8 lines · Sebastian Reichel

Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the GATE_LINK code into the separate driver now. Please give it another try.

Greetings,

-- Sebastian

Applied this to my 6.8.1: https://github.com/K900/linux/tree/rk3588-test

As far as I can tell, literally everything works now - it boots, runs, and I can read and write the flash even with ROCKCHIP_SFC=m.

Thanks a lot for digging into this, y'all!

#10 lore raw

Hello Ilya,

Quoted 8 lines · Ilya K
Quoted 7 lines · Sebastian Reichel

Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the GATE_LINK code into the separate driver now. Please give it another try.

Greetings,

Quoted 7 lines
Quoted 1 line

-- Sebastian

Applied this to my 6.8.1: https://github.com/K900/linux/tree/rk3588-test

As far as I can tell, literally everything works now - it boots, runs, and I can read and write the flash even with ROCKCHIP_SFC=m.

Thanks a lot for digging into this, y'all!

Great, thanks for testing. Can you check if it still works with ROCKCHIP_SFC=m when applying the following additional change on top of your tree?

drivers/clk/rockchip/clk-rk3588.c +1 -1
index fea7e7fcc4a4..f0eb380b727c 100644
@@ -2413,7 +2413,7 @@ static struct rockchip_clk_branch rk3588_early_clk_branches[] __initdata = {
 static struct rockchip_clk_branch rk3588_clk_branches[] = {
        GATE_LINK(ACLK_ISP1_PRE, "aclk_isp1_pre", "aclk_isp1_root", ACLK_VI_ROOT, 0, RK3588_CLKGATE_CON(26), 6, GFLAGS),
        GATE_LINK(HCLK_ISP1_PRE, "hclk_isp1_pre", "hclk_isp1_root", HCLK_VI_ROOT, 0, RK3588_CLKGATE_CON(26), 8, GFLAGS),
-       GATE_LINK(HCLK_NVM, "hclk_nvm", "hclk_nvm_root", ACLK_NVM_ROOT, CLK_IS_CRITICAL, RK3588_CLKGATE_CON(31), 2, GFLAGS),
+       GATE_LINK(HCLK_NVM, "hclk_nvm", "hclk_nvm_root", ACLK_NVM_ROOT, 0, RK3588_CLKGATE_CON(31), 2, GFLAGS),
        GATE_LINK(ACLK_USB, "aclk_usb", "aclk_usb_root", ACLK_VO1USB_TOP_ROOT, 0, RK3588_CLKGATE_CON(42), 2, GFLAGS),
        GATE_LINK(HCLK_USB, "hclk_usb", "hclk_usb_root", HCLK_VO1USB_TOP_ROOT, 0, RK3588_CLKGATE_CON(42), 3, GFLAGS),
        GATE_LINK(ACLK_JPEG_DECODER_PRE, "aclk_jpeg_decoder_pre", "aclk_jpeg_decoder_root", ACLK_VDPU_ROOT, 0, RK3588_CLKGATE_CON(44), 7, GFLAGS),

Greetings,

-- Sebastian

#11 lore raw

Sebastian,

Quoted 29 lines · Sebastian Reichel

Hello Ilya,

Quoted 8 lines · Ilya K
Quoted 7 lines · Sebastian Reichel

Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the GATE_LINK code into the separate driver now. Please give it another try.

Greetings,

Quoted 7 lines
Quoted 1 line

-- Sebastian

Applied this to my 6.8.1: https://github.com/K900/linux/tree/rk3588-test

As far as I can tell, literally everything works now - it boots, runs, and I can read and write the flash even with ROCKCHIP_SFC=m.

Thanks a lot for digging into this, y'all!

Great, thanks for testing. Can you check if it still works with ROCKCHIP_SFC=m when applying the following additional change on top of your tree?

drivers/clk/rockchip/clk-rk3588.c +0 -0
index fea7e7fcc4a4..f0eb380b727c 100644

drivers/clk/rockchip/clk-rk3588.c static struct rockchip_clk_branch rk3588_early_clk_branches[] __initdata = { · line 2419

Quoted 8 lines
drivers/clk/rockchip/clk-rk3588.c +1 -1
@@ -2413,7 +2413,7 @@ static struct rockchip_clk_branch rk3588_early_clk_branches[] __initdata = {
 static struct rockchip_clk_branch rk3588_clk_branches[] = {
        GATE_LINK(ACLK_ISP1_PRE, "aclk_isp1_pre", "aclk_isp1_root", ACLK_VI_ROOT, 0, RK3588_CLKGATE_CON(26), 6, GFLAGS),
        GATE_LINK(HCLK_ISP1_PRE, "hclk_isp1_pre", "hclk_isp1_root", HCLK_VI_ROOT, 0, RK3588_CLKGATE_CON(26), 8, GFLAGS),
-       GATE_LINK(HCLK_NVM, "hclk_nvm", "hclk_nvm_root", ACLK_NVM_ROOT, CLK_IS_CRITICAL, RK3588_CLKGATE_CON(31), 2, GFLAGS),
+       GATE_LINK(HCLK_NVM, "hclk_nvm", "hclk_nvm_root", ACLK_NVM_ROOT, 0, RK3588_CLKGATE_CON(31), 2, GFLAGS),
        GATE_LINK(ACLK_USB, "aclk_usb", "aclk_usb_root", ACLK_VO1USB_TOP_ROOT, 0, RK3588_CLKGATE_CON(42), 2, GFLAGS),
        GATE_LINK(HCLK_USB, "hclk_usb", "hclk_usb_root", HCLK_VO1USB_TOP_ROOT, 0, RK3588_CLKGATE_CON(42), 3, GFLAGS),
        GATE_LINK(ACLK_JPEG_DECODER_PRE, "aclk_jpeg_decoder_pre", "aclk_jpeg_decoder_root", ACLK_VDPU_ROOT, 0, RK3588_CLKGATE_CON(44), 7, GFLAGS),

I can also report success with your latest tree. It boots successfully and I am able to interact with the SFC flash. Looking at the clk debug info I now see an enable value of 1 on aclk_nvm_root.

It also cleaned up the power-controller error message I had been seeing on vo0. I just get the 'PM: genpd: Disabling unused power domains' message and no further power errors/warnings. So looks good.

I also tried the patch in your most recent message (removing CLK_IS_CRITICAL from hclk_nvm). My kernel is compiled ROCKCHIP_SFC=m. Everything still works as expected.

So overall, looks great! Thanks for all of your work on this!

--
Chad LeClair

#12 lore raw
In reply to Sebastian Reichel
Quoted 37 lines · Sebastian Reichel

Hello Ilya,

Quoted 8 lines · Ilya K
Quoted 7 lines · Sebastian Reichel

Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the GATE_LINK code into the separate driver now. Please give it another try.

Greetings,

Quoted 7 lines
Quoted 1 line

-- Sebastian

Applied this to my 6.8.1: https://github.com/K900/linux/tree/rk3588-test

As far as I can tell, literally everything works now - it boots, runs, and I can read and write the flash even with ROCKCHIP_SFC=m.

Thanks a lot for digging into this, y'all!

Great, thanks for testing. Can you check if it still works with ROCKCHIP_SFC=m when applying the following additional change on top of your tree?

drivers/clk/rockchip/clk-rk3588.c +1 -1
index fea7e7fcc4a4..f0eb380b727c 100644
@@ -2413,7 +2413,7 @@ static struct rockchip_clk_branch rk3588_early_clk_branches[] __initdata = {
 static struct rockchip_clk_branch rk3588_clk_branches[] = {
        GATE_LINK(ACLK_ISP1_PRE, "aclk_isp1_pre", "aclk_isp1_root", ACLK_VI_ROOT, 0, RK3588_CLKGATE_CON(26), 6, GFLAGS),
        GATE_LINK(HCLK_ISP1_PRE, "hclk_isp1_pre", "hclk_isp1_root", HCLK_VI_ROOT, 0, RK3588_CLKGATE_CON(26), 8, GFLAGS),
-       GATE_LINK(HCLK_NVM, "hclk_nvm", "hclk_nvm_root", ACLK_NVM_ROOT, CLK_IS_CRITICAL, RK3588_CLKGATE_CON(31), 2, GFLAGS),
+       GATE_LINK(HCLK_NVM, "hclk_nvm", "hclk_nvm_root", ACLK_NVM_ROOT, 0, RK3588_CLKGATE_CON(31), 2, GFLAGS),
        GATE_LINK(ACLK_USB, "aclk_usb", "aclk_usb_root", ACLK_VO1USB_TOP_ROOT, 0, RK3588_CLKGATE_CON(42), 2, GFLAGS),
        GATE_LINK(HCLK_USB, "hclk_usb", "hclk_usb_root", HCLK_VO1USB_TOP_ROOT, 0, RK3588_CLKGATE_CON(42), 3, GFLAGS),
        GATE_LINK(ACLK_JPEG_DECODER_PRE, "aclk_jpeg_decoder_pre", "aclk_jpeg_decoder_root", ACLK_VDPU_ROOT, 0, RK3588_CLKGATE_CON(44), 7, GFLAGS),
Quoted 3 lines

Greetings,

-- Sebastian

Can confirm everything works with this change too: https://github.com/K900/linux/commit/d11ccce948117aeef39d3a6dbedd17e04555a9cc

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip