clk: rockchip: implement proper GATE_LINK support
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
[removed DT people from CC, since this is all about clocks now]
Hello Ilya,
Quoted 3 lines · Ilya K
This change seems to make my Orange Pi 5 (RK3588S) lock up on boot :( Any suggestions on how to debug this? It doesn't really log much...
I suppose with this change you explicitly mean the last patch, which has not yet been applied? That patch effectively allows some clocks to be disabled, that have previously been marked as critical (and thus always-on).
If that results in a boot lockup, I expect you have a peripheral driver, that does not enable a required clock (e.g. because of a missing clock reference).
I assume you have a bunch of other patches applied on top of upstream and not just this one? In that case it might be sensible to first apply this patch, do a test boot (it hopefully boots). Then step-by-step re-add the other patches and check which one breaks when being combined with this patch.
P.S.: I plan to send a v9 of this patch once 6.9-rc1 has been tagged and will split it, so that the removal of CLK_IS_CRITICAL is done in a separate patch.
Greetings,
-- Sebastian
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 19 lines · Ilya K
Quoted 13 lines
Quoted 4 lines
This change seems to make my Orange Pi 5 (RK3588S) lock up on boot :( Any suggestions on how to debug this? It doesn't really log much...
I suppose with this change you explicitly mean the last patch, which has not yet been applied? That patch effectively allows some clocks to be disabled, that have previously been marked as critical (and thus always-on).
If that results in a boot lockup, I expect you have a peripheral driver, that does not enable a required clock (e.g. because of a missing clock reference).
Quoted 6 lines
I assume you have a bunch of other patches applied on top of upstream and not just this one? In that case it might be sensible to first apply this patch, do a test boot (it hopefully boots). Then step-by-step re-add the other patches and check which one breaks when being combined with this patch.
Quoted 3 lines
Thanks! I think the email bounced from most of the mailing lists anyway... The exact tree I'm running is here: https://github.com/K900/linux/commits/
rk3588-test/ and the last revert commit is what makes things boot again.
Please update your mailer to send plain text mails instead of HTML when working with the kernel mailing lists. Also please do not top-post.
The tree seems to be more or less what is working on Rock 5B and EVB1. Please try a partial revert, which just undos the removal of RK3588_LINKED_CLK (the define and the usage of the define). If that works the next step would be to step-by-step remove usage of RK3588_LINKED_CLK again until finding the clock that needs to be critical.
Thanks,
-- Sebastian
Quoted 9 lines · Sebastian Reichel
[removed DT people from CC, since this is all about clocks now]
Hello Ilya,
Quoted 3 lines · Ilya K
This change seems to make my Orange Pi 5 (RK3588S) lock up on boot :( Any suggestions on how to debug this? It doesn't really log much...
Quoted 8 lines
I suppose with this change you explicitly mean the last patch, which has not yet been applied? That patch effectively allows some clocks to be disabled, that have previously been marked as critical (and thus always-on).
If that results in a boot lockup, I expect you have a peripheral driver, that does not enable a required clock (e.g. because of a missing clock reference).
Sebastian,
Another Orange Pi 5 owner here... I likewise ran into trouble with this patch. I have been playing with your rk3588 branch from the Collabora gitlab and was also experiencing hangs on boot on my OPi5. I bisected your branch and found that the GATE_LINK support commit was the problem for me.
Digging in, I found that the problem behind the hang was that I was not getting the aclk_nvm_root which in turn caused DMA transactions to the SFC driver to hang.
...
[ 2.804519] rockchip-sfc fe2b0000.spi: DMA wait for transfer finish timeout [ 2.805127] rockchip-sfc fe2b0000.spi: xfer data failed ret -110 dir 1
...
Setting aclk_nvm_root to critical allows the system to boot. However not all is well as I do get errors like the following which also seem to indicate further clock problems:
[ 6.296554] rockchip-pm-domain
fd8d8000.power-management:power-controller: failed to set idle on domain 'vo0', val=0
I assume that the con-ops of your patch is analogous to the downstream "clock-link" driver. (ie. you are looking for PM events to cause the pm_clk_suspend()/pm_clk_resume() routines to enable/disable the linked (second parent) clocks). This wasn't happening for me. Unlike in the downstream implementation, the gate_link devices' dev->driver pointer was null. So that when rpm_resume() was being called, it would make its way to pm_generic_runtime_resume() which then would do nothing. The end result is that I would have a prepare count of 1 on aclk_nvm_root, but an enable count of 0 so it would get disabled (orig patch w/o marking this critical).
I did a quick hacky proof of concept where I set up the pm ops on the clk-rk3588 driver to point to the pm_clk_suspend/pm_clk_resume routines like downstream did. I used device_bind_driver() to forceably bind the clk-rk3588 driver to the gate link devices it was setting up.
This worked for me. With those changes I then was able to boot without needing to set aclk_nvm_root as critical. Further, it cleaned up the power-controller error messages on the sdio and vo0 domains.
The force bind I did doesn't feel like a particularly clean solution to me. However, I am not knowledgeable enough in the (platform) device model to know what the right way to do this is without fully doing what downstream did and having DT nodes for the various gate link devices. But it seems to prove out that having the driver bound to the devices with the relevant runtime pm ops is the missing piece in my use case on the OPi5.
Hope this additional data point helps!
--
Chad LeClair
Hello Chad,
Quoted 19 lines · Chad LeClair
Quoted 9 lines · Sebastian Reichel
[removed DT people from CC, since this is all about clocks now]
Hello Ilya,
Quoted 3 lines · Ilya K
This change seems to make my Orange Pi 5 (RK3588S) lock up on boot :( Any suggestions on how to debug this? It doesn't really log much...
Quoted 8 lines
I suppose with this change you explicitly mean the last patch, which has not yet been applied? That patch effectively allows some clocks to be disabled, that have previously been marked as critical (and thus always-on).
If that results in a boot lockup, I expect you have a peripheral driver, that does not enable a required clock (e.g. because of a missing clock reference).
Quoted 5 lines
Another Orange Pi 5 owner here... I likewise ran into trouble with this patch. I have been playing with your rk3588 branch from the Collabora gitlab and was also experiencing hangs on boot on my OPi5. I bisected your branch and found that the
GATE_LINKsupport commit was the problem for me.
Ok.
Quoted 2 lines
Digging in, I found that the problem behind the hang was that I was not getting the
aclk_nvm_root
That matches what Ilya found (there were some more off-list messages because of HTML mail issues). I suspected, that HCLK_SD (which is sourced from SCMI) might need HCLK_NVM.
Quoted 6 lines
which in turn caused DMA transactions to the SFC driver to hang.
...
[ 2.804519] rockchip-sfc fe2b0000.spi: DMA wait for transfer finish timeout [ 2.805127] rockchip-sfc fe2b0000.spi: xfer data failed ret -110 dir 1
Quoted 8 lines
...
Setting
aclk_nvm_rootto critical allows the system to boot. However not all is well as I do get errors like the following which also seem to indicate further clock problems:[ 6.296554] rockchip-pm-domainfd8d8000.power-management:power-controller: failed to set idle on domain 'vo0', val=0
I see the same message for AV1 and I've also seen failed idle message with the rockchip kernel. I have not yet debugged what is going on with that.
Quoted 20 lines
I assume that the con-ops of your patch is analogous to the downstream "clock-link" driver. (ie. you are looking for PM events to cause the pm_clk_suspend()/pm_clk_resume() routines to enable/disable the linked (second parent) clocks). This wasn't happening for me. Unlike in the downstream implementation, the
gate_linkdevices'dev->driverpointer was null. So that whenrpm_resume()was being called, it would make its way topm_generic_runtime_resume()which then would do nothing. The end result is that I would have a prepare count of 1 onaclk_nvm_root, but an enable count of 0 so it would get disabled (orig patch w/o marking this critical).I did a quick hacky proof of concept where I set up the pm ops on the clk-rk3588 driver to point to the pm_clk_suspend/pm_clk_resume routines like downstream did. I used
device_bind_driver()to forceably bind the clk-rk3588 driver to the gate link devices it was setting up.This worked for me. With those changes I then was able to boot without needing to set
aclk_nvm_rootas critical. Further, it cleaned up the power-controller error messages on the sdio and vo0 domains.
Quoted 7 lines
The force bind I did doesn't feel like a particularly clean solution to me. However, I am not knowledgeable enough in the (platform) device model to know what the right way to do this is without fully doing what downstream did and having DT nodes for the various gate link devices. But it seems to prove out that having the driver bound to the devices with the relevant runtime pm ops is the missing piece in my use case on the OPi5.
Can you test again using rk3588-test [0] (which is based on today's master branch) from today as base? Ideally it should work with your Orange Pi 5, since it contains "clk: rockchip: rk3588: mark hclk_nvm critical".
I also worked on a cleaner solution for the issue you described and integrated it in the patch adding proper gate clock support. So please also test with HCLK_NVM not being marked as CRITICAL.
[0] https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commits/rk3588-test
If everything is fine I will prepare a v9.
Quoted 1 line
Hope this additional data point helps!
Thanks for the detailed analysis.
Greetings,
-- Sebastian
Quoted 5 lines · Sebastian Reichel
Can you test again using rk3588-test [0] (which is based on today's master branch) from today as base? Ideally it should work with your Orange Pi 5, since it contains "clk: rockchip: rk3588: mark
hclk_nvmcritical".
Quoted 8 lines
I also worked on a cleaner solution for the issue you described and integrated it in the patch adding proper gate clock support. So please also test with
HCLK_NVMnot being marked as CRITICAL.[0] https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commits/rk3588-test
If everything is fine I will prepare a v9.
Tested this on my end, and I'm getting the same SFC issue, so there's probably another clock that's missing somewhere. Building with ROCKCHIP_SFC=y (instead of =m) removes the error message, but any access to the flash just locks up, so it's probably placebo, unless there's a second, unrelated issue.
Sebastian,
Quoted 8 lines · Sebastian Reichel
I also worked on a cleaner solution for the issue you described and integrated it in the patch adding proper gate clock support. So please also test with
HCLK_NVMnot being marked as CRITICAL.[0] https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commits/rk3588-test
If everything is fine I will prepare a v9.
Quoted 7 lines
Quoted 1 line
Hope this additional data point helps!
Thanks for the detailed analysis.
Greetings,
-- Sebastian
No luck unfortunately. I still see the SFC dma timeouts. It looks like aclk_nvm_root is still getting disabled. It now has both an enable count of 0 and a prepare count of 0.
Unlike your previous version, I _do_ see the driver bound to the device and the rpm_resume() call finds its way to pm_clk_resume(). So it looks like you resolved the original issue I was seeing.
However, when I reach __pm_clk_enable() it looks like the clock entry (ce) is not in a good state:
(gdb) print *ce
$3 = {node = {next = 0xffff0001f0ce1930, prev = 0xffff0001f0ce1930}, con_id = 0xffff0001f0a214f0 "aclk_nvm_root", clk = 0xfffffffffffffffe, status = PCE_STATUS_ERROR, enabled_when_prepared = false}
The immediate problem at hand is that ce->status is PCE_STATUS_ERROR so the switch statement will take the default case and return without doing anything. Also the ce->clk pointer looks like some sort of error pointer value so I'm wondering if something went wrong in the setup you were doing in clk_gate_link_probe().
Note: I ran to that same __pm_clk_enable() breakpoint for a number of of GATE_LINK clocks. They all looked to be in that same bad state. I put the "aclk_nvm_root" one in the message here since that is the one that is most relevant to the discussion, but they all look to be broken in the same way.
Hopefully this gives you a hint as to what is going on.
BTW, In case it is of interest to you or Ilya, I have my "working" tree available here:
https://github.com/sesca128/linux-rk3588/tree/v6.8-rk3588-test
My quick and dirty hack is at the tip of the branch. The rest is pretty much your rk3588-test branch from a few days ago.
--
Chad LeClair
Hi Chad and Ilya,
Quoted 42 lines · Chad LeClair
Sebastian,
Quoted 8 lines · Sebastian Reichel
I also worked on a cleaner solution for the issue you described and integrated it in the patch adding proper gate clock support. So please also test with
HCLK_NVMnot being marked as CRITICAL.[0] https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/commits/rk3588-test
If everything is fine I will prepare a v9.
Quoted 7 lines
Quoted 1 line
Hope this additional data point helps!
Thanks for the detailed analysis.
Greetings,
-- Sebastian
No luck unfortunately. I still see the SFC dma timeouts. It looks like
aclk_nvm_rootis still getting disabled. It now has both an enable count of 0 and a prepare count of 0.Unlike your previous version, I _do_ see the driver bound to the device and the
rpm_resume()call finds its way topm_clk_resume(). So it looks like you resolved the original issue I was seeing.However, when I reach
__pm_clk_enable()it looks like the clock entry (ce) is not in a good state:
(gdb) print *ce
$3 = {node = {next =0xffff0001f0ce1930, prev =0xffff0001f0ce1930},con_id=0xffff0001f0a214f0"aclk_nvm_root", clk =0xfffffffffffffffe, status =PCE_STATUS_ERROR,enabled_when_prepared= false}The immediate problem at hand is that
ce->statusisPCE_STATUS_ERRORso the switch statement will take the default case and return without doing anything. Also thece->clkpointer looks like some sort of error pointer value so I'm wondering if something went wrong in the setup you were doing inclk_gate_link_probe().
Quoted 6 lines
Note: I ran to that same
__pm_clk_enable()breakpoint for a number of ofGATE_LINKclocks. They all looked to be in that same bad state. I put the "aclk_nvm_root" one in the message here since that is the one that is most relevant to the discussion, but they all look to be broken in the same way.Hopefully this gives you a hint as to what is going on.
Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the GATE_LINK code into the separate driver now. Please give it another try.
Greetings,
-- Sebastian
Quoted 8 lines · Sebastian Reichel
Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the
GATE_LINKcode into the separate driver now. Please give it another try.Greetings,
-- Sebastian
Applied this to my 6.8.1: https://github.com/K900/linux/tree/rk3588-test
As far as I can tell, literally everything works now - it boots, runs, and I can read and write the flash even with ROCKCHIP_SFC=m.
Thanks a lot for digging into this, y'all!
Hello Ilya,
Quoted 8 lines · Ilya K
Quoted 7 lines · Sebastian Reichel
Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the
GATE_LINKcode into the separate driver now. Please give it another try.Greetings,
Quoted 7 lines
Quoted 1 line
-- Sebastian
Applied this to my 6.8.1: https://github.com/K900/linux/tree/rk3588-test
As far as I can tell, literally everything works now - it boots, runs, and I can read and write the flash even with ROCKCHIP_SFC=m.
Thanks a lot for digging into this, y'all!
Great, thanks for testing. Can you check if it still works with ROCKCHIP_SFC=m when applying the following additional change on top of your tree?
drivers/clk/rockchip/clk-rk3588.c +1 -1
@@ -2413,7 +2413,7 @@ static struct rockchip_clk_branch rk3588_early_clk_branches[] __initdata = {static struct rockchip_clk_branch rk3588_clk_branches[] = {GATE_LINK (ACLK_ISP1_PRE ,"aclk_isp1_pre" ,"aclk_isp1_root" ,ACLK_VI_ROOT ,0 ,RK3588_CLKGATE_CON (26 ),6 ,GFLAGS ),GATE_LINK (HCLK_ISP1_PRE ,"hclk_isp1_pre" ,"hclk_isp1_root" ,HCLK_VI_ROOT ,0 ,RK3588_CLKGATE_CON (26 ),8 ,GFLAGS ), -GATE_LINK (HCLK_NVM ,"hclk_nvm" ,"hclk_nvm_root" ,ACLK_NVM_ROOT ,CLK_IS_CRITICAL ,RK3588_CLKGATE_CON (31 ),2 ,GFLAGS ), +GATE_LINK (HCLK_NVM ,"hclk_nvm" ,"hclk_nvm_root" ,ACLK_NVM_ROOT ,0 ,RK3588_CLKGATE_CON (31 ),2 ,GFLAGS ),GATE_LINK (ACLK_USB ,"aclk_usb" ,"aclk_usb_root" ,ACLK_VO1USB_TOP_ROOT ,0 ,RK3588_CLKGATE_CON (42 ),2 ,GFLAGS ),GATE_LINK (HCLK_USB ,"hclk_usb" ,"hclk_usb_root" ,HCLK_VO1USB_TOP_ROOT ,0 ,RK3588_CLKGATE_CON (42 ),3 ,GFLAGS ),GATE_LINK (ACLK_JPEG_DECODER_PRE ,"aclk_jpeg_decoder_pre" ,"aclk_jpeg_decoder_root" ,ACLK_VDPU_ROOT ,0 ,RK3588_CLKGATE_CON (44 ),7 ,GFLAGS ),
Greetings,
-- Sebastian
Sebastian,
Quoted 29 lines · Sebastian Reichel
Hello Ilya,
Quoted 8 lines · Ilya K
Quoted 7 lines · Sebastian Reichel
Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the
GATE_LINKcode into the separate driver now. Please give it another try.Greetings,
Quoted 7 lines
Quoted 1 line
-- Sebastian
Applied this to my 6.8.1: https://github.com/K900/linux/tree/rk3588-test
As far as I can tell, literally everything works now - it boots, runs, and I can read and write the flash even with ROCKCHIP_SFC=m.
Thanks a lot for digging into this, y'all!
Great, thanks for testing. Can you check if it still works with ROCKCHIP_SFC=m when applying the following additional change on top of your tree?
drivers/clk/rockchip/clk-rk3588.c +0
-0
drivers/clk/rockchip/clk-rk3588.c static struct rockchip_clk_branch rk3588_early_clk_branches[] __initdata = { · line 2419
Quoted 8 lines
drivers/clk/rockchip/clk-rk3588.c +1
-1@@ -2413,7 +2413,7 @@ static struct rockchip_clk_branch rk3588_early_clk_branches[] __initdata = {static struct rockchip_clk_branch rk3588_clk_branches[] = {GATE_LINK (ACLK_ISP1_PRE ,"aclk_isp1_pre" ,"aclk_isp1_root" ,ACLK_VI_ROOT ,0 ,RK3588_CLKGATE_CON (26 ),6 ,GFLAGS ),GATE_LINK (HCLK_ISP1_PRE ,"hclk_isp1_pre" ,"hclk_isp1_root" ,HCLK_VI_ROOT ,0 ,RK3588_CLKGATE_CON (26 ),8 ,GFLAGS ), -GATE_LINK (HCLK_NVM ,"hclk_nvm" ,"hclk_nvm_root" ,ACLK_NVM_ROOT ,CLK_IS_CRITICAL ,RK3588_CLKGATE_CON (31 ),2 ,GFLAGS ), +GATE_LINK (HCLK_NVM ,"hclk_nvm" ,"hclk_nvm_root" ,ACLK_NVM_ROOT ,0 ,RK3588_CLKGATE_CON (31 ),2 ,GFLAGS ),GATE_LINK (ACLK_USB ,"aclk_usb" ,"aclk_usb_root" ,ACLK_VO1USB_TOP_ROOT ,0 ,RK3588_CLKGATE_CON (42 ),2 ,GFLAGS ),GATE_LINK (HCLK_USB ,"hclk_usb" ,"hclk_usb_root" ,HCLK_VO1USB_TOP_ROOT ,0 ,RK3588_CLKGATE_CON (42 ),3 ,GFLAGS ),GATE_LINK (ACLK_JPEG_DECODER_PRE ,"aclk_jpeg_decoder_pre" ,"aclk_jpeg_decoder_root" ,ACLK_VDPU_ROOT ,0 ,RK3588_CLKGATE_CON (44 ),7 ,GFLAGS ),
I can also report success with your latest tree. It boots successfully and I am able to interact with the SFC flash. Looking at the clk debug info I now see an enable value of 1 on aclk_nvm_root.
It also cleaned up the power-controller error message I had been seeing on vo0. I just get the 'PM: genpd: Disabling unused power domains' message and no further power errors/warnings. So looks good.
I also tried the patch in your most recent message (removing CLK_IS_CRITICAL from hclk_nvm). My kernel is compiled ROCKCHIP_SFC=m. Everything still works as expected.
So overall, looks great! Thanks for all of your work on this!
--
Chad LeClair
Quoted 37 lines · Sebastian Reichel
Hello Ilya,
Quoted 8 lines · Ilya K
Quoted 7 lines · Sebastian Reichel
Ah, that was actually not setting up the clock links at all. Sorry about that. I reworked everything again and moved all the
GATE_LINKcode into the separate driver now. Please give it another try.Greetings,
Quoted 7 lines
Quoted 1 line
-- Sebastian
Applied this to my 6.8.1: https://github.com/K900/linux/tree/rk3588-test
As far as I can tell, literally everything works now - it boots, runs, and I can read and write the flash even with ROCKCHIP_SFC=m.
Thanks a lot for digging into this, y'all!
Great, thanks for testing. Can you check if it still works with ROCKCHIP_SFC=m when applying the following additional change on top of your tree?
drivers/clk/rockchip/clk-rk3588.c +1
-1@@ -2413,7 +2413,7 @@ static struct rockchip_clk_branch rk3588_early_clk_branches[] __initdata = {static struct rockchip_clk_branch rk3588_clk_branches[] = {GATE_LINK (ACLK_ISP1_PRE ,"aclk_isp1_pre" ,"aclk_isp1_root" ,ACLK_VI_ROOT ,0 ,RK3588_CLKGATE_CON (26 ),6 ,GFLAGS ),GATE_LINK (HCLK_ISP1_PRE ,"hclk_isp1_pre" ,"hclk_isp1_root" ,HCLK_VI_ROOT ,0 ,RK3588_CLKGATE_CON (26 ),8 ,GFLAGS ), -GATE_LINK (HCLK_NVM ,"hclk_nvm" ,"hclk_nvm_root" ,ACLK_NVM_ROOT ,CLK_IS_CRITICAL ,RK3588_CLKGATE_CON (31 ),2 ,GFLAGS ), +GATE_LINK (HCLK_NVM ,"hclk_nvm" ,"hclk_nvm_root" ,ACLK_NVM_ROOT ,0 ,RK3588_CLKGATE_CON (31 ),2 ,GFLAGS ),GATE_LINK (ACLK_USB ,"aclk_usb" ,"aclk_usb_root" ,ACLK_VO1USB_TOP_ROOT ,0 ,RK3588_CLKGATE_CON (42 ),2 ,GFLAGS ),GATE_LINK (HCLK_USB ,"hclk_usb" ,"hclk_usb_root" ,HCLK_VO1USB_TOP_ROOT ,0 ,RK3588_CLKGATE_CON (42 ),3 ,GFLAGS ),GATE_LINK (ACLK_JPEG_DECODER_PRE ,"aclk_jpeg_decoder_pre" ,"aclk_jpeg_decoder_root" ,ACLK_VDPU_ROOT ,0 ,RK3588_CLKGATE_CON (44 ),7 ,GFLAGS ),
Quoted 3 lines
Greetings,
-- Sebastian
Can confirm everything works with this change too: https://github.com/K900/linux/commit/d11ccce948117aeef39d3a6dbedd17e04555a9cc
_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip