Porting 3.8-rc4-nohz3 to ARM (PandaBoard)
Hi Frederic,
(Sorry for the re-post, I spelled the mailing list wrong...)
I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.
Are there any caveats I should be aware of?
Has anyone else started any work in this area that I could co-operate with?
Regards
Mats Liljegren
2013/2/4 Mats Liljegren <[email protected]>:
Quoted 7 lines
Hi Frederic,
(Sorry for the re-post, I spelled the mailing list wrong...)
I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.
No that part is handled by the core code. But you need to implement the context tracking arch backend.
Here is what Li Zhong did to port it on Powerpc for example:
https://lkml.org/lkml/2013/2/1/216
Quoted 1 line
Are there any caveats I should be aware of?
The generic cputime accounting only works on 64 bits archs. We'll need to do some safe accessors to read the kernel_stat. In the meantime you can do home tests by removing the CONFIG_64BIT dependency. This shouldn't be dangerous
Quoted 1 line
Has anyone else started any work in this area that I could co-operate with?
Yeah two people have been working on this. But it was several month ago and I haven't hear from them since then. I'm adding Geoff in Cc. I can't retrieve the other guy who has worked on it too.
Hi,
Quoted 6 lines · Frederic Weisbecker
2013/2/4 Mats Liljegren <[email protected]>:
Quoted 1 line
I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
Yeah two people have been working on this. But it was several month ago and I haven't hear from them since then. I'm adding Geoff in Cc. I can't retrieve the other guy who has worked on it too.
I haven't kept the patches up to date, but what I have is here:
http://git.kernel.org/?p=linux/kernel/git/geoff/nohz.git
I have some Panda specific debugging patches in there too. The last version that worked OK with Panda was cpuset-v3-arm. The HEAD of the master branch had some problems at startup which I never had a chance to fix.
-Geoff
Quoted 21 lines · Frederic Weisbecker
2013/2/4 Mats Liljegren <[email protected]>:
Quoted 7 lines
Hi Frederic,
(Sorry for the re-post, I spelled the mailing list wrong...)
I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.
No that part is handled by the core code. But you need to implement the context tracking arch backend.
Here is what Li Zhong did to port it on Powerpc for example:
https://lkml.org/lkml/2013/2/1/216Quoted 1 line
Are there any caveats I should be aware of?
The generic cputime accounting only works on 64 bits archs. We'll need to do some safe accessors to read the
kernel_stat. In the meantime you can do home tests by removing theCONFIG_64BITdependency. This shouldn't be dangerous
Quoted 5 lines
Quoted 1 line
Has anyone else started any work in this area that I could co-operate with?
Yeah two people have been working on this. But it was several month ago and I haven't hear from them since then. I'm adding Geoff in Cc. I can't retrieve the other guy who has worked on it too.
Thanks for the hints! They helped me along the way.
I have now managed to boot Linux without getting RCU complaints with RCU checking enabled. I enabled virtual CPU accounting by ignoring 64BIT dependency, as you suggested. I had to add an __aeabi_uldivmod function to get it to compile though.
It boots without RCU complaints when CONFIG_PROVE_RCU is enabled.
Next step will be to figure out those boot parameters to enable full dynticks mode, and see whether I forgot something... ;-) I'll do so on Monday.
Regards
Mats Liljegren
Quoted 21 lines · Frederic Weisbecker
2013/2/4 Mats Liljegren <[email protected]>:
Quoted 7 lines
Hi Frederic,
(Sorry for the re-post, I spelled the mailing list wrong...)
I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.
No that part is handled by the core code. But you need to implement the context tracking arch backend.
Here is what Li Zhong did to port it on Powerpc for example:
https://lkml.org/lkml/2013/2/1/216Quoted 1 line
Are there any caveats I should be aware of?
The generic cputime accounting only works on 64 bits archs. We'll need to do some safe accessors to read the
kernel_stat. In the meantime you can do home tests by removing theCONFIG_64BITdependency. This shouldn't be dangerous
Quoted 5 lines
Quoted 1 line
Has anyone else started any work in this area that I could co-operate with?
Yeah two people have been working on this. But it was several month ago and I haven't hear from them since then. I'm adding Geoff in Cc. I can't retrieve the other guy who has worked on it too.
Could the other guy be Kevin Hillman? The name has been mentioned to me but I don't have his e-mail address.
I have stumbled across a problem, I get the following error messages during boot:
---- Start snippet ----
[ 41.119873] INFO: task rcuo1:10 blocked for more than 20 seconds. [ 41.126373] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 41.134765] rcuo1 D c045c5d8 0 10 2 0x00000000 [ 41.141662] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 41.150329] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) [ 41.159423] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc) [ 41.168487] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 41.177124] no locks held by rcuo1/10. [ 41.181182] INFO: task rcuo1:12 blocked for more than 20 seconds. [ 41.187683] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 41.196044] rcuo1 D c045c5d8 0 12 2 0x00000000 [ 41.202911] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 41.211578] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) [ 41.220733] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc) [ 41.229705] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 41.238372] no locks held by rcuo1/12. [ 61.239868] INFO: task rcuo1:10 blocked for more than 20 seconds. [ 61.246337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 61.254730] rcuo1 D c045c5d8 0 10 2 0x00000000 [ 61.261596] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 61.270294] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) [ 61.279357] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc) [ 61.288391] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 61.297058] no locks held by rcuo1/10. [ 61.301116] INFO: task rcuo1:12 blocked for more than 20 seconds. [ 61.307617] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 61.315979] rcuo1 D c045c5d8 0 12 2 0x00000000 [ 61.322845] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 61.331512] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) [ 61.340667] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc) [ 61.349639] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 61.358306] no locks held by rcuo1/12. [ 81.359863] INFO: task swapper/0:1 blocked for more than 20 seconds. [ 81.366638] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 81.374999] swapper/0 D c045c5d8 0 1 0 0x00000000 [ 81.381866] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 81.390563] [<c045c880>] (schedule+0x74/0x78) from [<c0459c0c>] (schedule_timeout+0x1e4/0x244) [ 81.399719] [<c0459c0c>] (schedule_timeout+0x1e4/0x244) from [<c0459c9c>] (schedule_timeout_uninterruptible+0x30/0x34) [ 81.411163] [<c0459c9c>] (schedule_timeout_uninterruptible+0x30/0x34) from [<c0057a98>] (msleep+0x24/0x30) [ 81.421508] [<c0057a98>] (msleep+0x24/0x30) from [<c0659d20>] (mount_root+0x74/0xfc) [ 81.429748] [<c0659d20>] (mount_root+0x74/0xfc) from [<c0659ed0>] (prepare_namespace+0x128/0x18c) [ 81.439270] [<c0659ed0>] (prepare_namespace+0x128/0x18c) from [<c044e35c>] (kernel_init+0x1cc/0x338) [ 81.449066] [<c044e35c>] (kernel_init+0x1cc/0x338) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 81.458282] no locks held by swapper/0/1. [ 81.462615] INFO: task rcuo1:10 blocked for more than 20 seconds. [ 81.469085] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 81.477447] rcuo1 D c045c5d8 0 10 2 0x00000000 [ 81.484313] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 81.493011] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) [ 81.502136] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc) [ 81.511169] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 81.519836] no locks held by rcuo1/10. [ 81.523834] INFO: task rcuo1:12 blocked for more than 20 seconds. [ 81.530364] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 81.538696] rcuo1 D c045c5d8 0 12 2 0x00000000 [ 81.545562] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 81.554229] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) [ 81.563354] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc) [ 81.572387] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 81.581054] no locks held by rcuo1/12. [ 101.579895] INFO: task rcuo1:10 blocked for more than 20 seconds. [ 101.586364] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 101.594757] rcuo1 D c045c5d8 0 10 2 0x00000000 [ 101.601623] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 101.610290] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) [ 101.619384] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc) [ 101.628417] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 101.637084] no locks held by rcuo1/10. [ 101.641143] INFO: task rcuo1:12 blocked for more than 20 seconds. [ 101.647613] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 101.656005] rcuo1 D c045c5d8 0 12 2 0x00000000 [ 101.662872] [<c045c5d8>] (__schedule+0x8f4/0xa74) from [<c045c880>] (schedule+0x74/0x78) [ 101.671539] [<c045c880>] (schedule+0x74/0x78) from [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) [ 101.680664] [<c00df138>] (rcu_nocb_kthread+0xdc/0x224) from [<c006d830>] (kthread+0xb0/0xbc) [ 101.689636] [<c006d830>] (kthread+0xb0/0xbc) from [<c00142c8>] (ret_from_fork+0x14/0x20) [ 101.698303] no locks held by rcuo1/12. [ 112.400695] VFS: Unable to mount root fs via NFS, trying floppy. [ 112.407226] List of all partitions: [ 112.411041] No filesystem could mount root, tried: [ 112.416351] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0) [ 112.425262] CPU1: stopping [ 112.428161] [<c001dcb8>] (unwind_backtrace+0x0/0x104) from [<c0453cb8>] (dump_stack+0x20/0x24) [ 112.437316] [<c0453cb8>] (dump_stack+0x20/0x24) from [<c001b16c>] (handle_IPI+0xf4/0x154) [ 112.445983] [<c001b16c>] (handle_IPI+0xf4/0x154) from [<c0008600>] (gic_handle_irq+0x68/0x84) [ 112.455047] [<c0008600>] (gic_handle_irq+0x68/0x84) from [<c045df84>] (__irq_svc+0x44/0x58) [ 112.463897] Exception stack(0xeda29f68 to 0xeda29fb0) [ 112.469238] 9f60: 00000001 00000001 00000000 eda20a80 eda28000 eda28000 [ 112.477905] 9f80: c073d9c8 c06c0574 c0468c60 eda28000 c06c4f90 eda29fdc eda29f80 eda29fb0 [ 112.486572] 9fa0: c00ab854 c00156fc 20000113 ffffffff [ 112.491943] [<c045df84>] (__irq_svc+0x44/0x58) from [<c00156fc>] (cpu_idle+0xac/0x128) [ 112.500366] [<c00156fc>] (cpu_idle+0xac/0x128) from [<c0450a1c>] (secondary_start_kernel+0xec/0x104) [ 112.510040] [<c0450a1c>] (secondary_start_kernel+0xec/0x104) from [<8044ff88>] (0x8044ff88)
PANIC: VFS: Unable to mount root fs on unknown-block(2,0)
---- End snipped ----
Do you have any suggestions for the best way of tracking down the cause?
Regards
Mats Liljegren
Quoted 26 lines · Mats Liljegren
Quoted 25 lines · Frederic Weisbecker
2013/2/4 Mats Liljegren <[email protected]>:
Quoted 11 lines
Hi Frederic,
(Sorry for the re-post, I spelled the mailing list wrong...)
I want to port dynticks patches 3.8-rc4-nohz3 to ARM architecture, and try it out on the PandaBoard. Initially this is for evaluating it, but if it looks promising I'd like to upstream this.
As far as I can see the first thing to do is to implement virtual time accounting. After an initial look at the patches done for ia64, it doesn't look too difficult.
No that part is handled by the core code. But you need to implement the context tracking arch backend.
Here is what Li Zhong did to port it on Powerpc for example:
https://lkml.org/lkml/2013/2/1/216Quoted 1 line
Are there any caveats I should be aware of?
The generic cputime accounting only works on 64 bits archs. We'll need to do some safe accessors to read the
kernel_stat. In the meantime you can do home tests by removing theCONFIG_64BITdependency. This shouldn't be dangerous
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Has anyone else started any work in this area that I could co-operate with?
Yeah two people have been working on this. But it was several month ago and I haven't hear from them since then. I'm adding Geoff in Cc. I can't retrieve the other guy who has worked on it too.
Could the other guy be Kevin Hillman?
Yup, that's him (though my last name has only one 'l' ;)
Quoted 1 line
The name has been mentioned to me but I don't have his e-mail address.
Now you do. :)
After some time away, I'm back to working on getting adaptive nohz working on ARM, but things have changed quite a bit since I last worked on it. Things actually look a bit simpler now with the new context_tracking layer, but that needs a little work for ARM which I'm now looking into.
However, just to get things compiling needs a little work too. I've setup a work-in-progress branch which you can start from to at least get things compiling[1].
I noticed you already ran accross the 64-bit division problem. I've proposed a patch for that[2] which is included in my branch along with a few other things.
Hope that helps,
Kevin
[1] git://git.kernel.org/pub/scm/linux/kernel/git/khilman/linux.git wip/3.8-rc6-nohz4-arm
[2] http://marc.info/?l=linux-kernel&m=136087139214252&w=2
2013/2/11 Mats Liljegren <[email protected]>:
Quoted 2 lines
Could the other guy be Kevin Hillman? The name has been mentioned to me but I don't have his e-mail address.
Exactly :)
Quoted 6 lines
I have stumbled across a problem, I get the following error messages during boot:
---- Start snippet ----
[ 41.119873] INFO: task rcuo1:10 blocked for more than 20 seconds.
This showed up upstream as well, it hshould have been fixed in linus tree now.
[...]
Quoted 4 lines
[ 112.400695] VFS: Unable to mount root fs via NFS, trying floppy. [ 112.407226] List of all partitions: [ 112.411041] No filesystem could mount root, tried: [ 112.416351] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
Ah, that probably not due to my patches :)
Quoted 9 lines
2013/2/11 Mats Liljegren <[email protected]>:
Quoted 2 lines
Could the other guy be Kevin Hillman? The name has been mentioned to me but I don't have his e-mail address.
Exactly :)
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
I have stumbled across a problem, I get the following error messages during boot:
---- Start snippet ----
[ 41.119873] INFO: task rcuo1:10 blocked for more than 20 seconds.This showed up upstream as well, it hshould have been fixed in linus tree now.
Ok.
Quoted 8 lines
[...]
Quoted 4 lines
[ 112.400695] VFS: Unable to mount root fs via NFS, trying floppy. [ 112.407226] List of all partitions: [ 112.411041] No filesystem could mount root, tried: [ 112.416351] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)Ah, that probably not due to my patches :)
By removing some kernel parameters I managed to get further. They were'nt important, just tried to disable watchdog and idle calls. My current problem is that it refuses to enter full nohz mode due to too many runable threads. I need to check were those extra threads come from.
Could it be related to rcu problem above? If so I might need to switch to a newer version of Linus tree.
/Mats
2013/2/20 Mats Liljegren <[email protected]>:
Quoted 26 lines
Quoted 9 lines
2013/2/11 Mats Liljegren <[email protected]>:
Quoted 2 lines
Could the other guy be Kevin Hillman? The name has been mentioned to me but I don't have his e-mail address.
Exactly :)
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
I have stumbled across a problem, I get the following error messages during boot:
---- Start snippet ----
[ 41.119873] INFO: task rcuo1:10 blocked for more than 20 seconds.This showed up upstream as well, it hshould have been fixed in linus tree now.
Ok.
Quoted 7 lines
[...]
Quoted 4 lines
[ 112.400695] VFS: Unable to mount root fs via NFS, trying floppy. [ 112.407226] List of all partitions: [ 112.411041] No filesystem could mount root, tried: [ 112.416351] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
Quoted 6 lines
Quoted 1 line
Ah, that probably not due to my patches :)
By removing some kernel parameters I managed to get further. They were'nt important, just tried to disable watchdog and idle calls. My current problem is that it refuses to enter full nohz mode due to too many runable threads. I need to check were those extra threads come from.
Could it be related to rcu problem above? If so I might need to switch to a newer version of Linus tree.
May be. If so you may need to migrate the rcuoX kthreads. You probably need to use the sched switch tracepoint to check what's going on. And migrate some threads affinity if needed.