Skip to contentSkip to conversation

local_softirq_pending storm

BUG Anant Nitya 7 participants 28 messages May 9, 2007 – May 24, 2007 3 messages not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi,
Ever since I upgrade to 2.6.21/.1, system log is filled with following messages if I enable CONFIG_NO_HZ=y, going through archives it seems ingo sometime back posted some patch and now it is upstream, but its not helping here. If I disable NOHZ by kernel command line nohz=off this problem disappears. This system is P4/2.40GHz/HT with SMP/SMT on in kernel config. One more thing that I noticed is this problem only arises while using X or network otherwise plain command line with no network access don't trigger this with nohz=on.
Please ask if more information is required regarding this setup.

dumdum@hahakaar [~]$ >> grep NOHZ /var/log/messages

May  8 03:38:14 rudra kernel: [  419.271195] NOHZ: local_softirq_pending 02
May  8 03:38:14 rudra kernel: [  419.271663] NOHZ: local_softirq_pending 02
May  8 03:38:14 rudra kernel: [  419.343948] NOHZ: local_softirq_pending 22
May  8 03:38:14 rudra kernel: [  419.344236] NOHZ: local_softirq_pending 22
May  8 03:38:14 rudra kernel: [  419.344397] NOHZ: local_softirq_pending 22
May  8 03:38:14 rudra kernel: [  419.344545] NOHZ: local_softirq_pending 22
May  8 03:38:14 rudra kernel: [  419.344691] NOHZ: local_softirq_pending 22
May  8 03:38:14 rudra kernel: [  419.344842] NOHZ: local_softirq_pending 22
May  8 03:38:14 rudra kernel: [  419.344991] NOHZ: local_softirq_pending 22
May  8 03:38:14 rudra kernel: [  419.345137] NOHZ: local_softirq_pending 22
May  8 03:38:18 rudra kernel: [  423.065780] NOHZ: local_softirq_pending 22
May  8 03:38:18 rudra kernel: [  423.066206] NOHZ: local_softirq_pending 22
May  8 03:38:18 rudra kernel: [  423.066509] NOHZ: local_softirq_pending 22
May  8 03:38:19 rudra kernel: [  424.006549] NOHZ: local_softirq_pending 22
May  8 03:38:19 rudra kernel: [  424.006983] NOHZ: local_softirq_pending 22
May  8 03:38:19 rudra kernel: [  424.007239] NOHZ: local_softirq_pending 22
May  8 03:38:19 rudra kernel: [  424.007473] NOHZ: local_softirq_pending 22
May  8 03:38:19 rudra kernel: [  424.007706] NOHZ: local_softirq_pending 22
May  8 03:38:19 rudra kernel: [  424.007941] NOHZ: local_softirq_pending 22
May  8 03:38:22 rudra kernel: [  426.862456] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.331619] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.331991] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.332192] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.332378] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.332553] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.332740] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.332914] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.333097] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.333271] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.333443] NOHZ: local_softirq_pending 22
May  8 03:38:23 rudra kernel: [  428.333619] NOHZ: local_softirq_pending 22
May  8 03:38:24 rudra kernel: [  429.261574] NOHZ: local_softirq_pending 22
May  8 03:38:24 rudra kernel: [  429.262024] NOHZ: local_softirq_pending 22
May  8 03:38:24 rudra kernel: [  429.262339] NOHZ: local_softirq_pending 22
May  8 03:38:24 rudra kernel: [  429.262610] NOHZ: local_softirq_pending 22
May  8 03:38:24 rudra kernel: [  429.262847] NOHZ: local_softirq_pending 22
May  8 03:38:24 rudra kernel: [  429.263081] NOHZ: local_softirq_pending 22
May  8 03:38:35 rudra kernel: [  440.182998] NOHZ: local_softirq_pending 22
May  8 03:38:35 rudra kernel: [  440.183408] NOHZ: local_softirq_pending 22
May  8 03:38:35 rudra kernel: [  440.183661] NOHZ: local_softirq_pending 22
May  8 03:38:43 rudra kernel: [  448.272087] NOHZ: local_softirq_pending 22
May  8 03:38:43 rudra kernel: [  448.272529] NOHZ: local_softirq_pending 22
May  8 03:38:44 rudra kernel: [  449.223360] NOHZ: local_softirq_pending 22
May  8 03:38:44 rudra kernel: [  449.223887] NOHZ: local_softirq_pending 22
May  8 03:38:44 rudra kernel: [  449.224570] NOHZ: local_softirq_pending 22
May  8 03:38:44 rudra kernel: [  449.225066] NOHZ: local_softirq_pending 22
May  8 03:38:44 rudra kernel: [  449.232989] NOHZ: local_softirq_pending 22
May  8 03:38:47 rudra kernel: [  452.178583] NOHZ: local_softirq_pending a2
May  8 03:38:47 rudra kernel: [  452.179017] NOHZ: local_softirq_pending a2
May  8 03:38:47 rudra kernel: [  452.179257] NOHZ: local_softirq_pending a2
May  8 03:38:51 rudra kernel: [  455.957968] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.958462] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.958741] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.958984] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.959292] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.959540] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.959774] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.960005] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.972368] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.972681] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.972884] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.973069] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.973251] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.973433] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.973631] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.973832] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.974018] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.974234] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.974422] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.974610] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.974791] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.974981] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.975194] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.975390] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.975570] NOHZ: local_softirq_pending 10
May  8 03:38:51 rudra kernel: [  455.976091] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.976376] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.976622] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.976862] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.977099] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.977337] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.977577] NOHZ: local_softirq_pending 22
May  8 03:38:51 rudra kernel: [  455.977815] NOHZ: local_softirq_pending 22
May  8 03:38:52 rudra kernel: [  456.848267] NOHZ: local_softirq_pending 22
May  8 03:38:52 rudra kernel: [  456.848726] NOHZ: local_softirq_pending 22
May  8 03:38:52 rudra kernel: [  456.848982] NOHZ: local_softirq_pending 22
May  8 03:38:52 rudra kernel: [  456.849223] NOHZ: local_softirq_pending 22
May  8 03:38:52 rudra kernel: [  456.849459] NOHZ: local_softirq_pending 22
May  8 03:38:54 rudra kernel: [  459.279942] NOHZ: local_softirq_pending 02
May  8 03:38:54 rudra kernel: [  459.280267] NOHZ: local_softirq_pending 02
May  8 03:38:54 rudra kernel: [  459.280477] NOHZ: local_softirq_pending 02
May  8 03:38:54 rudra kernel: [  459.280667] NOHZ: local_softirq_pending 02
May  8 03:38:54 rudra kernel: [  459.280865] NOHZ: local_softirq_pending 02
May  8 03:38:59 rudra kernel: [  464.046556] NOHZ: local_softirq_pending 22
May  8 03:38:59 rudra kernel: [  464.046999] NOHZ: local_softirq_pending 22
May  8 03:38:59 rudra kernel: [  464.047315] NOHZ: local_softirq_pending 22
May  8 03:38:59 rudra kernel: [  464.047554] NOHZ: local_softirq_pending 22
May  8 03:38:59 rudra kernel: [  464.047790] NOHZ: local_softirq_pending 22
May  8 03:39:08 rudra kernel: [  473.019484] NOHZ: local_softirq_pending a2
May  8 03:39:08 rudra kernel: [  473.019954] NOHZ: local_softirq_pending a2
May  8 03:39:08 rudra kernel: [  473.020215] NOHZ: local_softirq_pending a2
May  8 03:39:08 rudra kernel: [  473.020455] NOHZ: local_softirq_pending a2
May  8 03:39:08 rudra kernel: [  473.020692] NOHZ: local_softirq_pending a2
May  8 03:39:08 rudra kernel: [  473.020925] NOHZ: local_softirq_pending a2
May  8 03:39:11 rudra kernel: [  475.876271] NOHZ: local_softirq_pending 22
May  8 03:39:11 rudra kernel: [  475.876697] NOHZ: local_softirq_pending 22
May  8 03:39:11 rudra kernel: [  475.876947] NOHZ: local_softirq_pending 22
May  8 03:39:11 rudra kernel: [  475.877183] NOHZ: local_softirq_pending 22
May  8 03:39:11 rudra kernel: [  475.877490] NOHZ: local_softirq_pending 22
May  8 03:39:11 rudra kernel: [  475.877736] NOHZ: local_softirq_pending 22
May  8 03:39:11 rudra kernel: [  475.877964] NOHZ: local_softirq_pending 22
May  8 03:39:31 rudra kernel: [  495.878914] NOHZ: local_softirq_pending 22
May  8 03:39:31 rudra kernel: [  495.879262] NOHZ: local_softirq_pending 22
May  8 03:39:31 rudra kernel: [  495.879473] NOHZ: local_softirq_pending 22
May  8 03:39:31 rudra kernel: [  495.879674] NOHZ: local_softirq_pending 22
May  8 03:39:31 rudra kernel: [  495.879849] NOHZ: local_softirq_pending 22
May  8 03:39:31 rudra kernel: [  495.880031] NOHZ: local_softirq_pending 22
May  8 03:39:31 rudra kernel: [  495.880206] NOHZ: local_softirq_pending 22
May  8 03:39:53 rudra kernel: [  517.755212] NOHZ: local_softirq_pending 22
May  8 03:39:53 rudra kernel: [  517.755863] NOHZ: local_softirq_pending 22
May  8 03:39:53 rudra kernel: [  517.756181] NOHZ: local_softirq_pending 22
May  8 03:39:53 rudra kernel: [  517.756434] NOHZ: local_softirq_pending 22
May  8 03:39:53 rudra kernel: [  517.756667] NOHZ: local_softirq_pending 22
May  8 03:39:53 rudra kernel: [  517.756900] NOHZ: local_softirq_pending 22
May  8 03:39:54 rudra kernel: [  518.718220] NOHZ: local_softirq_pending 22
May  8 03:39:54 rudra kernel: [  518.718540] NOHZ: local_softirq_pending 22
May  8 03:39:54 rudra kernel: [  518.718756] NOHZ: local_softirq_pending 22
May  8 03:39:54 rudra kernel: [  518.718944] NOHZ: local_softirq_pending 22
May  8 03:39:54 rudra kernel: [  518.719121] NOHZ: local_softirq_pending 22
May  8 03:39:54 rudra kernel: [  518.719298] NOHZ: local_softirq_pending 22
May  8 03:39:54 rudra kernel: [  518.719484] NOHZ: local_softirq_pending 22
May  8 03:39:54 rudra kernel: [  518.719672] NOHZ: local_softirq_pending 22
May  8 03:40:02 rudra kernel: [  526.814840] NOHZ: local_softirq_pending 22
May  8 03:40:02 rudra kernel: [  526.815354] NOHZ: local_softirq_pending 22
May  8 03:40:02 rudra kernel: [  526.815703] NOHZ: local_softirq_pending 22
May  8 03:40:02 rudra kernel: [  526.816015] NOHZ: local_softirq_pending 22
May  8 03:40:02 rudra kernel: [  526.816255] NOHZ: local_softirq_pending 22
May  8 03:40:11 rudra kernel: [  535.847890] NOHZ: local_softirq_pending a2
May  8 03:40:11 rudra kernel: [  535.848360] NOHZ: local_softirq_pending a2
May  8 03:40:11 rudra kernel: [  535.848618] NOHZ: local_softirq_pending a2
May  8 03:40:11 rudra kernel: [  535.848857] NOHZ: local_softirq_pending a2
May  8 03:40:11 rudra kernel: [  535.849095] NOHZ: local_softirq_pending a2
May  8 03:40:12 rudra kernel: [  536.788182] NOHZ: local_softirq_pending 22
May  8 03:40:12 rudra kernel: [  536.788630] NOHZ: local_softirq_pending 22
May  8 03:40:12 rudra kernel: [  536.788939] NOHZ: local_softirq_pending 22
May  8 03:40:12 rudra kernel: [  536.789240] NOHZ: local_softirq_pending 22
May  8 03:40:12 rudra kernel: [  536.789484] NOHZ: local_softirq_pending 22
May  8 03:40:12 rudra kernel: [  536.789718] NOHZ: local_softirq_pending 22
May  8 03:40:16 rudra kernel: [  541.078474] NOHZ: local_softirq_pending a2
May  8 03:40:16 rudra kernel: [  541.078933] NOHZ: local_softirq_pending a2
May  8 03:40:16 rudra kernel: [  541.079186] NOHZ: local_softirq_pending a2
May  8 03:40:16 rudra kernel: [  541.079425] NOHZ: local_softirq_pending a2
May  8 03:40:16 rudra kernel: [  541.079657] NOHZ: local_softirq_pending a2
May  8 03:40:16 rudra kernel: [  541.079893] NOHZ: local_softirq_pending a2
May  8 03:40:18 rudra kernel: [  542.974969] NOHZ: local_softirq_pending 22
May  8 03:40:18 rudra kernel: [  542.975419] NOHZ: local_softirq_pending 22
May  8 03:40:18 rudra kernel: [  542.975754] NOHZ: local_softirq_pending 22
May  8 03:40:18 rudra kernel: [  542.976018] NOHZ: local_softirq_pending 22
May  8 03:40:18 rudra kernel: [  542.976253] NOHZ: local_softirq_pending 22
May  8 03:40:18 rudra kernel: [  542.976484] NOHZ: local_softirq_pending 22
May  8 03:40:24 rudra kernel: [  548.687200] NOHZ: local_softirq_pending a2
May  8 03:40:24 rudra kernel: [  548.687691] NOHZ: local_softirq_pending a2
May  8 03:40:28 rudra kernel: [  552.977399] NOHZ: local_softirq_pending a2
May  8 03:40:28 rudra kernel: [  552.977730] NOHZ: local_softirq_pending a2
May  8 03:40:28 rudra kernel: [  552.977946] NOHZ: local_softirq_pending a2
May  8 03:40:28 rudra kernel: [  552.978135] NOHZ: local_softirq_pending a2
May  8 03:40:28 rudra kernel: [  552.978318] NOHZ: local_softirq_pending a2
May  8 03:40:28 rudra kernel: [  552.978499] NOHZ: local_softirq_pending a2
May  8 03:40:28 rudra kernel: [  552.978708] NOHZ: local_softirq_pending a2
May  8 03:40:33 rudra kernel: [  557.728762] NOHZ: local_softirq_pending 22
May  8 03:40:33 rudra kernel: [  557.729230] NOHZ: local_softirq_pending 22
May  8 03:40:33 rudra kernel: [  557.729539] NOHZ: local_softirq_pending 22
May  8 03:40:33 rudra kernel: [  557.729821] NOHZ: local_softirq_pending 22
May  8 03:40:33 rudra kernel: [  557.730060] NOHZ: local_softirq_pending 22
May  8 03:40:33 rudra kernel: [  557.730297] NOHZ: local_softirq_pending 22
May  8 03:40:36 rudra kernel: [  561.065192] NOHZ: local_softirq_pending e2
May  8 03:40:36 rudra kernel: [  561.065539] NOHZ: local_softirq_pending e2
May  8 03:40:36 rudra kernel: [  561.065750] NOHZ: local_softirq_pending e2
May  8 03:40:36 rudra kernel: [  561.065942] NOHZ: local_softirq_pending e2
May  8 03:40:36 rudra kernel: [  561.066139] NOHZ: local_softirq_pending e2
May  8 03:40:36 rudra kernel: [  561.066339] NOHZ: local_softirq_pending e2
May  8 03:40:36 rudra kernel: [  561.066515] NOHZ: local_softirq_pending e2
May  8 03:40:42 rudra kernel: [  566.772651] NOHZ: local_softirq_pending 22
May  8 03:40:42 rudra kernel: [  566.773108] NOHZ: local_softirq_pending 22
May  8 03:40:42 rudra kernel: [  566.773314] NOHZ: local_softirq_pending 22
May  8 03:40:42 rudra kernel: [  566.773359] NOHZ: local_softirq_pending 22
May  8 03:40:42 rudra kernel: [  566.787249] NOHZ: local_softirq_pending 62
May  8 03:40:42 rudra kernel: [  566.787536] NOHZ: local_softirq_pending 62
May  8 03:40:51 rudra kernel: [  575.816602] NOHZ: local_softirq_pending 22
May  8 03:40:51 rudra kernel: [  575.817065] NOHZ: local_softirq_pending 22
May  8 03:41:03 rudra kernel: [  587.715470] NOHZ: local_softirq_pending 22
May  8 03:41:03 rudra kernel: [  587.715920] NOHZ: local_softirq_pending 22
May  8 03:41:03 rudra kernel: [  587.716259] NOHZ: local_softirq_pending 22
May  8 03:41:03 rudra kernel: [  587.716528] NOHZ: local_softirq_pending 22
May  8 03:41:03 rudra kernel: [  587.716764] NOHZ: local_softirq_pending 22
May  8 03:41:03 rudra kernel: [  587.717001] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.805445] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.805945] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.806196] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.806430] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.806659] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.806888] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.807116] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.807341] NOHZ: local_softirq_pending 22
May  8 03:41:12 rudra kernel: [  596.807570] NOHZ: local_softirq_pending 22
May  8 03:41:20 rudra kernel: [  604.847182] NOHZ: local_softirq_pending 22
May  8 03:41:20 rudra kernel: [  604.847637] NOHZ: local_softirq_pending 22
May  8 03:41:20 rudra kernel: [  604.847941] NOHZ: local_softirq_pending 22
May  8 03:41:20 rudra kernel: [  604.848225] NOHZ: local_softirq_pending 22
May  8 03:41:20 rudra kernel: [  604.848463] NOHZ: local_softirq_pending 22
May  8 03:41:20 rudra kernel: [  604.848700] NOHZ: local_softirq_pending 22
May  8 03:41:23 rudra kernel: [  607.702270] NOHZ: local_softirq_pending 22
May  8 03:41:23 rudra kernel: [  607.702604] NOHZ: local_softirq_pending 22
May  8 03:41:23 rudra kernel: [  607.702804] NOHZ: local_softirq_pending 22
May  8 03:41:23 rudra kernel: [  607.702982] NOHZ: local_softirq_pending 22
May  8 03:41:23 rudra kernel: [  607.703181] NOHZ: local_softirq_pending 22
May  8 03:41:23 rudra kernel: [  607.703431] NOHZ: local_softirq_pending 22
May  8 03:41:23 rudra kernel: [  607.703621] NOHZ: local_softirq_pending 22
May  8 03:41:29 rudra kernel: [  613.888941] NOHZ: local_softirq_pending 22
May  8 03:41:29 rudra kernel: [  613.889415] NOHZ: local_softirq_pending 22
May  8 03:41:29 rudra kernel: [  613.889712] NOHZ: local_softirq_pending 22
May  8 03:41:29 rudra kernel: [  613.890009] NOHZ: local_softirq_pending 22
May  8 03:41:29 rudra kernel: [  613.890251] NOHZ: local_softirq_pending 22
May  8 03:41:29 rudra kernel: [  613.890487] NOHZ: local_softirq_pending 22
May  8 03:41:32 rudra kernel: [  616.743862] NOHZ: local_softirq_pending 22
May  8 03:41:32 rudra kernel: [  616.744331] NOHZ: local_softirq_pending 22
May  8 03:41:32 rudra kernel: [  616.744640] NOHZ: local_softirq_pending 22
May  8 03:41:32 rudra kernel: [  616.744929] NOHZ: local_softirq_pending 22
May  8 03:41:32 rudra kernel: [  616.745175] NOHZ: local_softirq_pending 22
May  8 03:41:32 rudra kernel: [  616.745409] NOHZ: local_softirq_pending 22
May  8 03:41:34 rudra kernel: [  618.660611] NOHZ: local_softirq_pending a2
May  8 03:41:34 rudra kernel: [  618.660958] NOHZ: local_softirq_pending a2
May  8 03:41:34 rudra kernel: [  618.661166] NOHZ: local_softirq_pending a2
May  8 03:41:34 rudra kernel: [  618.661349] NOHZ: local_softirq_pending a2
May  8 03:41:34 rudra kernel: [  618.661541] NOHZ: local_softirq_pending a2
May  8 03:41:34 rudra kernel: [  618.661729] NOHZ: local_softirq_pending a2
May  8 03:41:34 rudra kernel: [  618.661911] NOHZ: local_softirq_pending a2
May  8 03:41:34 rudra kernel: [  618.662089] NOHZ: local_softirq_pending a2
May  8 03:41:37 rudra kernel: [  621.996786] NOHZ: local_softirq_pending 22
May  8 03:41:50 rudra kernel: [  634.831692] NOHZ: local_softirq_pending 22
May  8 03:41:50 rudra kernel: [  634.832143] NOHZ: local_softirq_pending 22
May  8 03:41:50 rudra kernel: [  634.832453] NOHZ: local_softirq_pending 22
May  8 03:41:50 rudra kernel: [  634.832765] NOHZ: local_softirq_pending 22
May  8 03:41:50 rudra kernel: [  634.833011] NOHZ: local_softirq_pending 22
May  8 03:41:50 rudra kernel: [  634.833241] NOHZ: local_softirq_pending 22
May  8 03:41:51 rudra kernel: [  635.790271] NOHZ: local_softirq_pending a2
May  8 03:41:51 rudra kernel: [  635.790608] NOHZ: local_softirq_pending a2
May  8 03:41:51 rudra kernel: [  635.790823] NOHZ: local_softirq_pending a2
May  8 03:41:51 rudra kernel: [  635.791001] NOHZ: local_softirq_pending a2
May  8 03:41:51 rudra kernel: [  635.791179] NOHZ: local_softirq_pending a2
May  8 03:41:51 rudra kernel: [  635.791354] NOHZ: local_softirq_pending a2
May  8 03:41:51 rudra kernel: [  635.791537] NOHZ: local_softirq_pending a2
May  8 03:41:53 rudra kernel: [  637.686505] NOHZ: local_softirq_pending 22
May  8 03:41:53 rudra kernel: [  637.686981] NOHZ: local_softirq_pending 22
May  8 03:41:53 rudra kernel: [  637.687428] NOHZ: local_softirq_pending 22
May  8 03:42:08 rudra kernel: [  652.941445] NOHZ: local_softirq_pending 22
May  8 03:42:08 rudra kernel: [  652.941976] NOHZ: local_softirq_pending 22
May  8 03:42:08 rudra kernel: [  652.942243] NOHZ: local_softirq_pending 22
May  8 03:42:08 rudra kernel: [  652.942485] NOHZ: local_softirq_pending 22
May  8 03:42:08 rudra kernel: [  652.942717] NOHZ: local_softirq_pending 22
May  8 03:42:08 rudra kernel: [  652.942953] NOHZ: local_softirq_pending 22
May  8 03:42:08 rudra kernel: [  652.943188] NOHZ: local_softirq_pending 22
May  8 03:42:09 rudra kernel: [  653.862586] NOHZ: local_softirq_pending 22
May  8 03:42:09 rudra kernel: [  653.862918] NOHZ: local_softirq_pending 22
May  8 03:42:09 rudra kernel: [  653.863124] NOHZ: local_softirq_pending 22
May  8 03:42:09 rudra kernel: [  653.863316] NOHZ: local_softirq_pending 22
May  8 03:42:09 rudra kernel: [  653.863501] NOHZ: local_softirq_pending 22
May  8 03:42:09 rudra kernel: [  653.863687] NOHZ: local_softirq_pending 22
May  8 03:42:09 rudra kernel: [  653.863863] NOHZ: local_softirq_pending 22
May  8 03:42:12 rudra kernel: [  656.714675] NOHZ: local_softirq_pending 22
May  8 03:42:12 rudra kernel: [  656.715163] NOHZ: local_softirq_pending 22
May  8 03:42:12 rudra kernel: [  656.715423] NOHZ: local_softirq_pending 22
May  8 03:42:12 rudra kernel: [  656.715665] NOHZ: local_softirq_pending 22
May  8 03:42:12 rudra kernel: [  656.715904] NOHZ: local_softirq_pending 22
May  8 03:42:12 rudra kernel: [  656.716140] NOHZ: local_softirq_pending 22
May  8 03:42:12 rudra kernel: [  656.716378] NOHZ: local_softirq_pending 22
May  8 03:42:12 rudra kernel: [  656.716611] NOHZ: local_softirq_pending 22
May  8 03:42:14 rudra kernel: [  658.622576] NOHZ: local_softirq_pending 22
May  8 03:42:15 rudra kernel: [  659.569910] NOHZ: local_softirq_pending 22
May  8 03:42:15 rudra kernel: [  659.570340] NOHZ: local_softirq_pending 22
May  8 03:42:15 rudra kernel: [  659.570650] NOHZ: local_softirq_pending 22
May  8 03:42:15 rudra kernel: [  659.570924] NOHZ: local_softirq_pending 22
May  8 03:42:15 rudra kernel: [  659.571159] NOHZ: local_softirq_pending 22
May  8 03:42:15 rudra kernel: [  659.571393] NOHZ: local_softirq_pending 22
May  9 01:10:43 rudra kernel: [ 4472.241952] NOHZ: local_softirq_pending 22
May  9 01:10:43 rudra kernel: [ 4472.243018] NOHZ: local_softirq_pending 22
May  9 01:10:43 rudra kernel: [ 4472.243383] NOHZ: local_softirq_pending 22
May  9 01:11:15 rudra kernel: [ 4504.194191] NOHZ: local_softirq_pending 22
May  9 01:11:15 rudra kernel: [ 4504.194815] NOHZ: local_softirq_pending 22
May  9 01:11:15 rudra kernel: [ 4504.195140] NOHZ: local_softirq_pending 22
May  9 01:11:35 rudra kernel: [ 4524.158524] NOHZ: local_softirq_pending 22
May  9 01:11:35 rudra kernel: [ 4524.159118] NOHZ: local_softirq_pending 22
May  9 01:11:35 rudra kernel: [ 4524.159439] NOHZ: local_softirq_pending 22
May  9 01:11:45 rudra kernel: [ 4534.140638] NOHZ: local_softirq_pending 22
May  9 01:11:45 rudra kernel: [ 4534.141046] NOHZ: local_softirq_pending 22
May  9 01:11:45 rudra kernel: [ 4534.141456] NOHZ: local_softirq_pending 22
May  9 01:11:45 rudra kernel: [ 4534.141738] NOHZ: local_softirq_pending 22
May  9 01:12:32 rudra kernel: [ 4580.718873] NOHZ: local_softirq_pending 22
May  9 01:12:32 rudra kernel: [ 4580.719297] NOHZ: local_softirq_pending 22
May  9 01:12:32 rudra kernel: [ 4580.719557] NOHZ: local_softirq_pending 22
May  9 01:12:32 rudra kernel: [ 4580.719808] NOHZ: local_softirq_pending 22
May  9 01:12:32 rudra kernel: [ 4580.720047] NOHZ: local_softirq_pending 22
May  9 01:12:32 rudra kernel: [ 4580.720295] NOHZ: local_softirq_pending 22
May  9 01:12:32 rudra kernel: [ 4580.720534] NOHZ: local_softirq_pending 22
May  9 01:12:36 rudra kernel: [ 4585.053403] NOHZ: local_softirq_pending 22
May  9 01:12:36 rudra kernel: [ 4585.054308] NOHZ: local_softirq_pending 22
May  9 01:12:36 rudra kernel: [ 4585.054588] NOHZ: local_softirq_pending 22
May  9 01:12:36 rudra kernel: [ 4585.054796] NOHZ: local_softirq_pending 22
May  9 01:12:36 rudra kernel: [ 4585.054981] NOHZ: local_softirq_pending 22
May  9 01:12:36 rudra kernel: [ 4585.055163] NOHZ: local_softirq_pending 22
May  9 01:12:38 rudra kernel: [ 4586.927035] NOHZ: local_softirq_pending 22
May  9 01:12:51 rudra kernel: [ 4599.749155] NOHZ: local_softirq_pending 22
May  9 01:12:51 rudra kernel: [ 4599.750614] NOHZ: local_softirq_pending 22
May  9 01:12:53 rudra kernel: [ 4601.648012] NOHZ: local_softirq_pending 22
May  9 01:12:53 rudra kernel: [ 4601.648457] NOHZ: local_softirq_pending 22
May  9 01:12:53 rudra kernel: [ 4601.648722] NOHZ: local_softirq_pending 22
May  9 01:12:53 rudra kernel: [ 4601.648971] NOHZ: local_softirq_pending 22
May  9 01:12:53 rudra kernel: [ 4601.649217] NOHZ: local_softirq_pending 22
May  9 01:12:53 rudra kernel: [ 4601.649463] NOHZ: local_softirq_pending 22
May  9 01:12:53 rudra kernel: [ 4601.649705] NOHZ: local_softirq_pending 22
May  9 01:12:53 rudra kernel: [ 4601.649951] NOHZ: local_softirq_pending 22
May  9 01:19:16 rudra kernel: [  325.029390] NOHZ: local_softirq_pending 22

TIA

#1 lore raw
Quoted 9 lines · Anant Nitya

Hi,
Ever since I upgrade to 2.6.21/.1, system log is filled with following messages if I enable CONFIG_NO_HZ=y, going through archives it seems ingo sometime back posted some patch and now it is upstream, but its not helping here. If I disable NOHZ by kernel command line nohz=off this problem disappears. This system is P4/2.40GHz/HT with SMP/SMT on in kernel config. One more thing that I noticed is this problem only arises while using X or network otherwise plain command line with no network access don't trigger this with nohz=on.

Is this independent of the load on the system ? i.e. : What happens if you only use the console and run a kernel compile with -j4 ?

Quoted 3 lines

dumdum@hahakaar [~]$ >> grep NOHZ /var/log/messages

May  8 03:38:14 rudra kernel: [  419.271195] NOHZ: local_softirq_pending 02
May  8 03:38:14 rudra kernel: [  419.271663] NOHZ: local_softirq_pending 02

The patch below ratelimits the printk output, so your syslog is not flooded anymore.

tglx

kernel/time/tick-sched.c +9 -3
Index: linux-2.6.21/kernel/time/tick-sched.c
===================================================================
@@ -167,9 +167,15 @@ void tick_nohz_stop_sched_tick(void)
 		goto end;
 
 	cpu = smp_processor_id();
-	if (unlikely(local_softirq_pending()))
-		printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
-		       local_softirq_pending());
+	if (unlikely(local_softirq_pending())) {
+		static int ratelimit;
+
+		if (ratelimit < 10) {
+			printk(KERN_ERR "NOHZ: local_softirq_pending %02x\n",
+			       local_softirq_pending());
+			ratelimit++;
+		}
+	}
 
 	now = ktime_get();
 	/*
#2 lore raw
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Anant Nitya

Hi,
Ever since I upgrade to 2.6.21/.1, system log is filled with following messages if I enable CONFIG_NO_HZ=y, going through archives it seems ingo sometime back posted some patch and now it is upstream, but its not helping here. If I disable NOHZ by kernel command line nohz=off this problem disappears. This system is P4/2.40GHz/HT with SMP/SMT on in kernel config. One more thing that I noticed is this problem only arises while using X or network otherwise plain command line with no network access don't trigger this with nohz=on.

Quoted 2 lines

Is this independent of the load on the system ? i.e. : What happens if you only use the console and run a kernel compile with -j4 ?

Yep, it seems independent of load on the system, to test I compiled kernel with make -j8 and to throw some more load same time I also used amavisd to clean around 2000 of spam/virus infected mails and same time was listening to few radio stream over net in console only login mode. Even then there was not a single NOHZ: local_softirq_pending message in log. Once kernel got compiled and amavisd finished its job and load drops back to around 0.5/2.0 I started X and within few secs log starts getting filled with local_softirq_pending messages, (sorry, I didn't applied ratelimit patch since I wanted to test is it high load on system that causes this or something else). It seems even network operation is not causing this. It seems either X is doing something terribly wrong or kernel is getting hosed by X. If some more information is needed please feel free to ask

dumdum@hahakaar [~]$ >> grep load top.log

top - 14:04:12 up 15 min,  4 users,  load average: 9.42, 5.12, 2.20
top - 14:04:17 up 15 min,  4 users,  load average: 9.47, 5.20, 2.24
top - 14:04:22 up 16 min,  4 users,  load average: 9.35, 5.25, 2.27
top - 14:04:27 up 16 min,  4 users,  load average: 9.32, 5.31, 2.31
top - 14:04:32 up 16 min,  4 users,  load average: 9.22, 5.36, 2.34
top - 14:04:37 up 16 min,  4 users,  load average: 9.36, 5.45, 2.38
top - 14:04:42 up 16 min,  4 users,  load average: 9.33, 5.51, 2.42
top - 14:04:47 up 16 min,  4 users,  load average: 9.38, 5.58, 2.46
top - 14:04:52 up 16 min,  4 users,  load average: 9.35, 5.64, 2.49
top - 14:04:57 up 16 min,  4 users,  load average: 9.24, 5.68, 2.52
top - 14:05:02 up 16 min,  4 users,  load average: 9.14, 5.72, 2.55
top - 14:05:07 up 16 min,  4 users,  load average: 9.05, 5.75, 2.58
top - 14:05:12 up 16 min,  4 users,  load average: 9.13, 5.83, 2.62
top - 14:05:17 up 16 min,  4 users,  load average: 9.28, 5.91, 2.66
top - 14:05:22 up 17 min,  4 users,  load average: 9.34, 5.98, 2.70
top - 14:05:27 up 17 min,  4 users,  load average: 9.47, 6.06, 2.75
top - 14:05:32 up 17 min,  4 users,  load average: 9.43, 6.11, 2.78
top - 14:05:37 up 17 min,  4 users,  load average: 9.32, 6.14, 2.81
top - 14:05:42 up 17 min,  4 users,  load average: 9.37, 6.20, 2.85
top - 14:05:47 up 17 min,  4 users,  load average: 9.26, 6.23, 2.87
top - 14:05:52 up 17 min,  4 users,  load average: 9.16, 6.26, 2.90
top - 14:05:58 up 17 min,  4 users,  load average: 9.07, 6.29, 2.93
top - 14:06:03 up 17 min,  4 users,  load average: 9.14, 6.35, 2.97
top - 14:06:08 up 17 min,  4 users,  load average: 9.13, 6.40, 3.00
top - 14:06:13 up 17 min,  4 users,  load average: 9.28, 6.47, 3.04
top - 14:06:18 up 17 min,  4 users,  load average: 9.34, 6.53, 3.08
top - 14:06:23 up 18 min,  4 users,  load average: 9.31, 6.57, 3.11
top - 14:06:28 up 18 min,  4 users,  load average: 9.36, 6.63, 3.15
top - 14:06:33 up 18 min,  4 users,  load average: 9.25, 6.65, 3.17
top - 14:06:38 up 18 min,  4 users,  load average: 9.23, 6.69, 3.20
top - 14:06:43 up 18 min,  4 users,  load average: 9.38, 6.76, 3.25
top - 14:06:48 up 18 min,  4 users,  load average: 9.27, 6.78, 3.27
top - 14:06:53 up 18 min,  4 users,  load average: 9.32, 6.84, 3.31
top - 14:06:58 up 18 min,  4 users,  load average: 9.70, 6.95, 3.36
top - 14:07:03 up 18 min,  4 users,  load average: 9.72, 7.00, 3.40
top - 14:07:08 up 18 min,  4 users,  load average: 9.82, 7.07, 3.44
top - 14:07:13 up 18 min,  4 users,  load average: 9.84, 7.12, 3.48
top - 14:07:18 up 18 min,  4 users,  load average: 9.85, 7.17, 3.51
top - 14:07:23 up 19 min,  4 users,  load average: 9.78, 7.20, 3.54
top - 14:07:28 up 19 min,  4 users,  load average: 9.80, 7.24, 3.57
top - 14:07:33 up 19 min,  4 users,  load average: 10.14, 7.40, 3.66
top - 14:07:38 up 19 min,  4 users,  load average: 10.45, 7.51, 3.72
top - 14:07:43 up 19 min,  4 users,  load average: 10.33, 7.53, 3.75
top - 14:07:48 up 19 min,  4 users,  load average: 9.58, 7.42, 3.73
top - 14:07:53 up 19 min,  4 users,  load average: 8.97, 7.33, 3.72
top - 14:07:58 up 19 min,  4 users,  load average: 8.34, 7.23, 3.71
top - 14:08:03 up 19 min,  4 users,  load average: 8.31, 7.24, 3.73
top - 14:08:08 up 19 min,  4 users,  load average: 8.36, 7.27, 3.76
top - 14:08:13 up 19 min,  4 users,  load average: 8.89, 7.40, 3.82
top - 14:08:18 up 20 min,  4 users,  load average: 8.34, 7.31, 3.81
top - 14:08:23 up 20 min,  4 users,  load average: 8.47, 7.35, 3.84
top - 14:08:28 up 20 min,  4 users,  load average: 8.52, 7.38, 3.87
top - 14:08:33 up 20 min,  4 users,  load average: 8.56, 7.41, 3.90
top - 14:08:39 up 20 min,  4 users,  load average: 8.51, 7.42, 3.92
top - 14:08:44 up 20 min,  4 users,  load average: 8.63, 7.46, 3.95
top - 14:08:49 up 20 min,  4 users,  load average: 8.66, 7.48, 3.98
top - 14:08:54 up 20 min,  4 users,  load average: 8.85, 7.54, 4.02
top - 14:08:59 up 20 min,  4 users,  load average: 8.94, 7.58, 4.05
top - 14:09:04 up 20 min,  4 users,  load average: 9.02, 7.62, 4.08
top - 14:09:09 up 20 min,  4 users,  load average: 9.02, 7.64, 4.10
top - 14:09:14 up 20 min,  4 users,  load average: 8.94, 7.65, 4.13
top - 14:09:19 up 21 min,  4 users,  load average: 9.02, 7.69, 4.16
top - 14:09:24 up 21 min,  4 users,  load average: 8.94, 7.69, 4.18
top - 14:09:29 up 21 min,  4 users,  load average: 8.87, 7.70, 4.20
top - 14:09:34 up 21 min,  4 users,  load average: 8.88, 7.72, 4.22
top - 14:09:39 up 21 min,  4 users,  load average: 8.81, 7.72, 4.24
top - 14:09:44 up 21 min,  4 users,  load average: 8.82, 7.75, 4.27
top - 14:09:49 up 21 min,  4 users,  load average: 8.76, 7.75, 4.29
top - 14:09:54 up 21 min,  4 users,  load average: 8.85, 7.79, 4.32
top - 14:09:59 up 21 min,  4 users,  load average: 8.79, 7.79, 4.34
top - 14:10:04 up 21 min,  4 users,  load average: 8.88, 7.83, 4.37
top - 14:10:09 up 21 min,  4 users,  load average: 8.81, 7.83, 4.39
top - 14:10:14 up 21 min,  5 users,  load average: 8.75, 7.83, 4.41
top - 14:10:19 up 22 min,  5 users,  load average: 8.77, 7.85, 4.43
top - 14:10:24 up 22 min,  5 users,  load average: 8.79, 7.87, 4.46
top - 14:10:29 up 22 min,  5 users,  load average: 8.72, 7.87, 4.47
top - 14:10:34 up 22 min,  5 users,  load average: 8.74, 7.89, 4.50
top - 14:10:39 up 22 min,  5 users,  load average: 8.92, 7.94, 4.53
top - 14:10:44 up 22 min,  5 users,  load average: 9.17, 8.01, 4.57
top - 14:10:49 up 22 min,  5 users,  load average: 9.24, 8.04, 4.60
top - 14:10:54 up 22 min,  5 users,  load average: 9.14, 8.04, 4.62
top - 14:10:59 up 22 min,  5 users,  load average: 9.21, 8.07, 4.65
top - 14:11:04 up 22 min,  5 users,  load average: 9.27, 8.10, 4.68
top - 14:11:09 up 22 min,  5 users,  load average: 9.25, 8.12, 4.70
top - 14:11:14 up 22 min,  5 users,  load average: 9.15, 8.12, 4.72
top - 14:11:19 up 23 min,  5 users,  load average: 9.06, 8.11, 4.73
top - 14:11:24 up 23 min,  5 users,  load average: 8.97, 8.11, 4.75
top - 14:11:29 up 23 min,  5 users,  load average: 9.13, 8.16, 4.79
top - 14:11:35 up 23 min,  5 users,  load average: 9.04, 8.16, 4.80
top - 14:11:40 up 23 min,  5 users,  load average: 9.12, 8.19, 4.83
top - 14:11:45 up 23 min,  5 users,  load average: 9.27, 8.23, 4.86
top - 14:11:50 up 23 min,  5 users,  load average: 9.33, 8.26, 4.89
top - 14:11:55 up 23 min,  5 users,  load average: 9.22, 8.26, 4.91
top - 14:12:00 up 23 min,  5 users,  load average: 9.28, 8.29, 4.93
top - 14:12:05 up 23 min,  5 users,  load average: 9.34, 8.32, 4.96
top - 14:12:10 up 23 min,  5 users,  load average: 9.47, 8.36, 4.99
top - 14:12:15 up 23 min,  5 users,  load average: 9.51, 8.39, 5.02
top - 14:12:20 up 24 min,  5 users,  load average: 9.47, 8.40, 5.04
top - 14:12:25 up 24 min,  5 users,  load average: 9.52, 8.42, 5.07
top - 14:12:30 up 24 min,  5 users,  load average: 9.47, 8.43, 5.09
top - 14:12:35 up 24 min,  5 users,  load average: 9.36, 8.43, 5.10
top - 14:12:40 up 24 min,  5 users,  load average: 9.25, 8.42, 5.12
top - 14:12:45 up 24 min,  5 users,  load average: 9.39, 8.46, 5.15
top - 14:12:50 up 24 min,  5 users,  load average: 9.36, 8.47, 5.17
top - 14:12:55 up 24 min,  5 users,  load average: 9.41, 8.50, 5.20
top - 14:13:00 up 24 min,  5 users,  load average: 9.45, 8.52, 5.22
top - 14:13:05 up 24 min,  5 users,  load average: 9.34, 8.51, 5.24
top - 14:13:10 up 24 min,  5 users,  load average: 9.23, 8.50, 5.25
top - 14:13:15 up 24 min,  5 users,  load average: 9.29, 8.53, 5.28
top - 14:13:20 up 25 min,  5 users,  load average: 9.35, 8.55, 5.30
top - 14:13:25 up 25 min,  5 users,  load average: 9.32, 8.56, 5.32
top - 14:13:30 up 25 min,  5 users,  load average: 9.21, 8.55, 5.34
top - 14:13:35 up 25 min,  5 users,  load average: 9.20, 8.56, 5.35
top - 14:13:40 up 25 min,  5 users,  load average: 9.10, 8.55, 5.37
top - 14:13:45 up 25 min,  5 users,  load average: 9.17, 8.57, 5.39
top - 14:13:50 up 25 min,  5 users,  load average: 9.08, 8.56, 5.41
top - 14:13:55 up 25 min,  5 users,  load average: 9.07, 8.57, 5.43
top - 14:14:00 up 25 min,  5 users,  load average: 8.99, 8.56, 5.44
top - 14:14:05 up 25 min,  5 users,  load average: 8.91, 8.55, 5.45
top - 14:14:10 up 25 min,  5 users,  load average: 8.91, 8.56, 5.47
top - 14:14:16 up 25 min,  5 users,  load average: 8.92, 8.56, 5.49
top - 14:14:21 up 26 min,  5 users,  load average: 8.85, 8.55, 5.50
top - 14:14:26 up 26 min,  5 users,  load average: 8.86, 8.56, 5.52
top - 14:14:31 up 26 min,  5 users,  load average: 9.03, 8.60, 5.55
top - 14:14:36 up 26 min,  5 users,  load average: 9.19, 8.64, 5.58
top - 14:14:41 up 26 min,  5 users,  load average: 9.17, 8.65, 5.60
top - 14:14:46 up 26 min,  5 users,  load average: 9.24, 8.67, 5.62
top - 14:14:51 up 26 min,  5 users,  load average: 9.22, 8.67, 5.64
top - 14:14:56 up 26 min,  5 users,  load average: 9.28, 8.70, 5.66
top - 14:15:01 up 26 min,  5 users,  load average: 9.26, 8.70, 5.68
top - 14:15:06 up 26 min,  5 users,  load average: 9.16, 8.69, 5.69
top - 14:15:11 up 26 min,  5 users,  load average: 9.14, 8.69, 5.71
top - 14:15:16 up 26 min,  5 users,  load average: 9.05, 8.68, 5.72
top - 14:15:21 up 27 min,  5 users,  load average: 9.13, 8.70, 5.75
top - 14:15:26 up 27 min,  5 users,  load average: 9.04, 8.69, 5.76
top - 14:15:31 up 27 min,  5 users,  load average: 9.11, 8.71, 5.78
top - 14:15:36 up 27 min,  5 users,  load average: 9.19, 8.74, 5.80
top - 14:15:41 up 27 min,  5 users,  load average: 9.17, 8.74, 5.82
top - 14:15:46 up 27 min,  5 users,  load average: 9.08, 8.73, 5.83
top - 14:15:51 up 27 min,  5 users,  load average: 8.99, 8.71, 5.84
top - 14:15:56 up 27 min,  5 users,  load average: 8.99, 8.72, 5.86
top - 14:16:01 up 27 min,  5 users,  load average: 8.99, 8.72, 5.88
top - 14:16:06 up 27 min,  5 users,  load average: 8.91, 8.71, 5.89
top - 14:16:11 up 27 min,  5 users,  load average: 8.92, 8.72, 5.90
top - 14:16:16 up 27 min,  5 users,  load average: 9.00, 8.74, 5.93
top - 14:16:21 up 28 min,  5 users,  load average: 9.00, 8.74, 5.94
top - 14:16:26 up 28 min,  5 users,  load average: 9.00, 8.74, 5.96
top - 14:16:31 up 28 min,  5 users,  load average: 9.00, 8.75, 5.97
top - 14:16:37 up 28 min,  5 users,  load average: 9.08, 8.77, 6.00
top - 14:16:42 up 28 min,  5 users,  load average: 9.16, 8.79, 6.02
top - 14:16:47 up 28 min,  5 users,  load average: 9.06, 8.78, 6.03
top - 14:16:52 up 28 min,  5 users,  load average: 8.98, 8.76, 6.04
top - 14:16:57 up 28 min,  5 users,  load average: 8.90, 8.75, 6.05
top - 14:17:02 up 28 min,  5 users,  load average: 9.15, 8.80, 6.08
top - 14:17:07 up 28 min,  5 users,  load average: 9.14, 8.81, 6.09
top - 14:17:12 up 28 min,  5 users,  load average: 9.12, 8.81, 6.11
top - 14:17:17 up 28 min,  5 users,  load average: 9.11, 8.81, 6.12
top - 14:17:22 up 29 min,  5 users,  load average: 9.02, 8.80, 6.13
top - 14:17:27 up 29 min,  5 users,  load average: 9.02, 8.80, 6.15
top - 14:17:32 up 29 min,  5 users,  load average: 9.02, 8.80, 6.16
top - 14:17:37 up 29 min,  5 users,  load average: 9.18, 8.84, 6.19
top - 14:17:42 up 29 min,  5 users,  load average: 9.24, 8.86, 6.21
top - 14:17:47 up 29 min,  5 users,  load average: 9.22, 8.86, 6.23
top - 14:17:52 up 29 min,  5 users,  load average: 9.21, 8.86, 6.24
top - 14:17:57 up 29 min,  5 users,  load average: 9.19, 8.87, 6.25
top - 14:18:02 up 29 min,  5 users,  load average: 9.33, 8.90, 6.28
top - 14:18:07 up 29 min,  5 users,  load average: 9.47, 8.93, 6.31
top - 14:18:12 up 29 min,  5 users,  load average: 9.43, 8.94, 6.32
top - 14:18:17 up 29 min,  5 users,  load average: 9.47, 8.95, 6.34
top - 14:18:22 up 30 min,  5 users,  load average: 9.52, 8.97, 6.36
top - 14:18:27 up 30 min,  5 users,  load average: 9.64, 9.00, 6.38
top - 14:18:32 up 30 min,  5 users,  load average: 9.58, 9.00, 6.40
top - 14:18:37 up 30 min,  5 users,  load average: 9.62, 9.02, 6.42
top - 14:18:42 up 30 min,  5 users,  load average: 9.65, 9.04, 6.43
top - 14:18:47 up 30 min,  5 users,  load average: 9.68, 9.05, 6.45
top - 14:18:52 up 30 min,  5 users,  load average: 9.86, 9.10, 6.48
top - 14:18:57 up 30 min,  5 users,  load average: 10.03, 9.15, 6.51
top - 14:19:02 up 30 min,  5 users,  load average: 10.11, 9.18, 6.54
top - 14:19:07 up 30 min,  5 users,  load average: 10.18, 9.21, 6.56
top - 14:19:12 up 30 min,  6 users,  load average: 10.25, 9.24, 6.58
top - 14:19:17 up 30 min,  6 users,  load average: 10.47, 9.30, 6.62
top - 14:19:22 up 31 min,  6 users,  load average: 10.51, 9.33, 6.64
top - 14:19:27 up 31 min,  6 users,  load average: 10.47, 9.34, 6.66
top - 14:19:32 up 31 min,  6 users,  load average: 10.59, 9.38, 6.69
top - 14:19:38 up 31 min,  6 users,  load average: 10.46, 9.38, 6.70
top - 14:19:43 up 31 min,  6 users,  load average: 10.59, 9.42, 6.73
top - 14:19:48 up 31 min,  6 users,  load average: 10.46, 9.41, 6.74
top - 14:19:53 up 31 min,  6 users,  load average: 10.34, 9.40, 6.75
top - 14:19:58 up 31 min,  6 users,  load average: 10.23, 9.40, 6.76
top - 14:20:03 up 31 min,  6 users,  load average: 10.29, 9.42, 6.79
top - 14:20:08 up 31 min,  6 users,  load average: 10.19, 9.42, 6.80
top - 14:20:13 up 31 min,  6 users,  load average: 10.26, 9.44, 6.82
top - 14:20:18 up 31 min,  6 users,  load average: 10.23, 9.45, 6.84
top - 14:20:23 up 32 min,  6 users,  load average: 10.14, 9.44, 6.85
top - 14:20:28 up 32 min,  6 users,  load average: 10.04, 9.44, 6.86
top - 14:20:33 up 32 min,  6 users,  load average: 9.96, 9.43, 6.87
top - 14:20:38 up 32 min,  6 users,  load average: 10.12, 9.47, 6.90
top - 14:20:43 up 32 min,  6 users,  load average: 10.11, 9.48, 6.91
top - 14:20:48 up 32 min,  6 users,  load average: 10.10, 9.49, 6.93
top - 14:20:53 up 32 min,  6 users,  load average: 10.18, 9.51, 6.95
top - 14:20:58 up 32 min,  6 users,  load average: 10.08, 9.50, 6.96
top - 14:21:03 up 32 min,  6 users,  load average: 9.99, 9.50, 6.97
top - 14:21:08 up 32 min,  6 users,  load average: 10.07, 9.52, 7.00
top - 14:21:13 up 32 min,  6 users,  load average: 9.99, 9.51, 7.01
top - 14:21:18 up 32 min,  6 users,  load average: 10.23, 9.57, 7.04
top - 14:21:23 up 33 min,  6 users,  load average: 10.13, 9.56, 7.05
top - 14:21:28 up 33 min,  6 users,  load average: 10.19, 9.59, 7.08
top - 14:21:33 up 33 min,  6 users,  load average: 10.17, 9.60, 7.10
top - 14:21:38 up 33 min,  6 users,  load average: 10.40, 9.65, 7.13
top - 14:21:43 up 33 min,  6 users,  load average: 10.45, 9.67, 7.15
top - 14:21:49 up 33 min,  6 users,  load average: 10.65, 9.73, 7.18
top - 14:21:54 up 33 min,  6 users,  load average: 10.52, 9.72, 7.19
top - 14:21:59 up 33 min,  6 users,  load average: 10.64, 9.75, 7.22
top - 14:22:04 up 33 min,  6 users,  load average: 10.59, 9.76, 7.23
top - 14:22:09 up 33 min,  6 users,  load average: 10.54, 9.76, 7.25
top - 14:22:14 up 33 min,  6 users,  load average: 11.14, 9.90, 7.30
top - 14:22:19 up 34 min,  6 users,  load average: 11.21, 9.93, 7.33
top - 14:22:24 up 34 min,  6 users,  load average: 11.03, 9.92, 7.34
top - 14:22:29 up 34 min,  6 users,  load average: 11.11, 9.95, 7.36
top - 14:22:34 up 34 min,  6 users,  load average: 10.86, 9.92, 7.37
top - 14:22:39 up 34 min,  6 users,  load average: 10.79, 9.92, 7.38
top - 14:22:44 up 34 min,  6 users,  load average: 10.81, 9.94, 7.40
top - 14:22:49 up 34 min,  6 users,  load average: 10.82, 9.96, 7.42
top - 14:22:54 up 34 min,  6 users,  load average: 10.59, 9.92, 7.42
top - 14:22:59 up 34 min,  6 users,  load average: 10.63, 9.94, 7.44
top - 14:23:04 up 34 min,  6 users,  load average: 10.58, 9.94, 7.45
top - 14:23:10 up 34 min,  6 users,  load average: 10.37, 9.91, 7.46
top - 14:23:15 up 34 min,  6 users,  load average: 10.34, 9.91, 7.47
top - 14:23:20 up 35 min,  6 users,  load average: 10.39, 9.93, 7.49
top - 14:23:25 up 35 min,  6 users,  load average: 10.52, 9.96, 7.51
top - 14:23:30 up 35 min,  6 users,  load average: 10.56, 9.98, 7.53
top - 14:23:35 up 35 min,  6 users,  load average: 10.59, 10.00, 7.55
top - 14:23:40 up 35 min,  6 users,  load average: 10.71, 10.03, 7.57
top - 14:23:45 up 35 min,  6 users,  load average: 10.73, 10.05, 7.59
top - 14:23:50 up 35 min,  6 users,  load average: 10.83, 10.08, 7.61
top - 14:23:55 up 35 min,  6 users,  load average: 11.08, 10.14, 7.65
top - 14:24:00 up 35 min,  6 users,  load average: 11.00, 10.14, 7.66
top - 14:24:05 up 35 min,  6 users,  load average: 10.92, 10.14, 7.67
top - 14:24:10 up 35 min,  6 users,  load average: 10.76, 10.12, 7.68
top - 14:24:15 up 35 min,  6 users,  load average: 10.78, 10.13, 7.70
top - 14:24:20 up 36 min,  6 users,  load average: 12.88, 10.58, 7.85
top - 14:24:25 up 36 min,  6 users,  load average: 13.05, 10.65, 7.89
top - 14:24:30 up 36 min,  6 users,  load average: 12.89, 10.66, 7.91
top - 14:24:35 up 36 min,  6 users,  load average: 12.82, 10.68, 7.93
top - 14:24:40 up 36 min,  6 users,  load average: 12.59, 10.67, 7.94
top - 14:24:46 up 36 min,  6 users,  load average: 12.46, 10.67, 7.96
top - 14:24:51 up 36 min,  6 users,  load average: 12.51, 10.71, 7.98
top - 14:24:56 up 36 min,  6 users,  load average: 12.30, 10.70, 8.00
top - 14:25:01 up 36 min,  6 users,  load average: 12.12, 10.69, 8.01
top - 14:25:06 up 36 min,  6 users,  load average: 12.03, 10.69, 8.02
top - 14:25:11 up 36 min,  6 users,  load average: 12.19, 10.75, 8.05
top - 14:25:16 up 36 min,  6 users,  load average: 11.93, 10.72, 8.06
top - 14:25:21 up 37 min,  6 users,  load average: 11.94, 10.74, 8.08
top - 14:25:26 up 37 min,  6 users,  load average: 11.62, 10.69, 8.08
top - 14:25:31 up 37 min,  6 users,  load average: 11.57, 10.70, 8.09
top - 14:25:36 up 37 min,  6 users,  load average: 11.45, 10.69, 8.10
top - 14:25:41 up 37 min,  6 users,  load average: 11.49, 10.71, 8.12
top - 14:25:46 up 37 min,  6 users,  load average: 11.69, 10.76, 8.16
top - 14:25:51 up 37 min,  6 users,  load average: 11.64, 10.77, 8.17
top - 14:25:56 up 37 min,  6 users,  load average: 11.58, 10.77, 8.19
top - 14:26:01 up 37 min,  6 users,  load average: 11.46, 10.76, 8.20
top - 14:26:06 up 37 min,  6 users,  load average: 11.58, 10.79, 8.22
top - 14:26:11 up 37 min,  6 users,  load average: 11.53, 10.80, 8.24
top - 14:26:16 up 37 min,  6 users,  load average: 11.41, 10.78, 8.25
top - 14:26:21 up 38 min,  6 users,  load average: 11.22, 10.75, 8.25
top - 14:26:26 up 38 min,  6 users,  load average: 11.04, 10.72, 8.25
top - 14:26:31 up 38 min,  6 users,  load average: 10.88, 10.69, 8.26
top - 14:26:36 up 38 min,  6 users,  load average: 10.89, 10.70, 8.27
top - 14:26:41 up 38 min,  6 users,  load average: 10.73, 10.67, 8.28
top - 14:26:46 up 38 min,  6 users,  load average: 10.68, 10.66, 8.28
top - 14:26:51 up 38 min,  6 users,  load average: 10.70, 10.67, 8.30
top - 14:26:57 up 38 min,  6 users,  load average: 10.81, 10.69, 8.32
top - 14:27:02 up 38 min,  6 users,  load average: 10.66, 10.66, 8.32
top - 14:27:07 up 38 min,  6 users,  load average: 10.85, 10.70, 8.35
top - 14:27:12 up 38 min,  6 users,  load average: 10.70, 10.67, 8.35
top - 14:27:17 up 38 min,  6 users,  load average: 10.72, 10.67, 8.36
top - 14:27:22 up 39 min,  6 users,  load average: 10.91, 10.71, 8.39
top - 14:27:27 up 39 min,  6 users,  load average: 10.99, 10.73, 8.41
top - 14:27:32 up 39 min,  6 users,  load average: 10.99, 10.74, 8.42
top - 14:27:37 up 39 min,  6 users,  load average: 10.99, 10.74, 8.43
top - 14:27:42 up 39 min,  6 users,  load average: 10.99, 10.75, 8.45
top - 14:27:47 up 39 min,  6 users,  load average: 10.99, 10.75, 8.46
top - 14:27:52 up 39 min,  6 users,  load average: 10.99, 10.75, 8.47
top - 14:27:57 up 39 min,  6 users,  load average: 10.99, 10.76, 8.49
top - 14:28:02 up 39 min,  6 users,  load average: 11.07, 10.78, 8.51
top - 14:28:07 up 39 min,  6 users,  load average: 10.99, 10.77, 8.51
top - 14:28:12 up 39 min,  6 users,  load average: 10.99, 10.77, 8.53
top - 14:28:17 up 39 min,  6 users,  load average: 10.91, 10.76, 8.54
top - 14:28:22 up 40 min,  6 users,  load average: 10.76, 10.73, 8.54
top - 14:28:27 up 40 min,  6 users,  load average: 10.70, 10.71, 8.55
top - 14:28:32 up 40 min,  6 users,  load average: 10.72, 10.72, 8.56
top - 14:28:37 up 40 min,  6 users,  load average: 10.58, 10.69, 8.56
top - 14:28:42 up 40 min,  6 users,  load average: 10.61, 10.69, 8.57
top - 14:28:47 up 40 min,  6 users,  load average: 10.49, 10.67, 8.58
top - 14:28:53 up 40 min,  6 users,  load average: 10.45, 10.66, 8.58
top - 14:28:58 up 40 min,  6 users,  load average: 10.33, 10.63, 8.58
top - 14:29:03 up 40 min,  6 users,  load average: 10.46, 10.65, 8.60
top - 14:29:08 up 40 min,  6 users,  load average: 10.43, 10.64, 8.61
top - 14:29:13 up 40 min,  6 users,  load average: 10.39, 10.63, 8.62
top - 14:29:18 up 40 min,  6 users,  load average: 10.68, 10.68, 8.65
top - 14:29:23 up 41 min,  6 users,  load average: 10.79, 10.70, 8.66
top - 14:29:28 up 41 min,  6 users,  load average: 10.64, 10.68, 8.66
top - 14:29:33 up 41 min,  6 users,  load average: 10.75, 10.70, 8.68
top - 14:29:38 up 41 min,  6 users,  load average: 10.85, 10.72, 8.70
top - 14:29:43 up 41 min,  6 users,  load average: 10.78, 10.71, 8.71
top - 14:29:48 up 41 min,  6 users,  load average: 10.96, 10.75, 8.73
top - 14:29:53 up 41 min,  6 users,  load average: 11.12, 10.78, 8.75
top - 14:29:58 up 41 min,  6 users,  load average: 11.19, 10.80, 8.77
top - 14:30:03 up 41 min,  6 users,  load average: 11.55, 10.89, 8.82
top - 14:30:08 up 41 min,  6 users,  load average: 11.43, 10.88, 8.83
top - 14:30:13 up 41 min,  6 users,  load average: 11.23, 10.84, 8.83
top - 14:30:18 up 42 min,  6 users,  load average: 11.13, 10.83, 8.83
top - 14:30:23 up 42 min,  6 users,  load average: 11.20, 10.85, 8.85
top - 14:30:28 up 42 min,  6 users,  load average: 11.03, 10.82, 8.85
top - 14:30:34 up 42 min,  6 users,  load average: 10.94, 10.80, 8.85
top - 14:30:39 up 42 min,  6 users,  load average: 10.87, 10.79, 8.86
top - 14:30:44 up 42 min,  6 users,  load average: 10.64, 10.74, 8.86
top - 14:30:49 up 42 min,  6 users,  load average: 10.59, 10.73, 8.86
top - 14:30:54 up 42 min,  6 users,  load average: 10.62, 10.74, 8.87
top - 14:30:59 up 42 min,  6 users,  load average: 10.65, 10.74, 8.88
top - 14:31:04 up 42 min,  6 users,  load average: 10.68, 10.74, 8.90
top - 14:31:09 up 42 min,  6 users,  load average: 10.78, 10.76, 8.91
top - 14:31:14 up 42 min,  6 users,  load average: 10.64, 10.74, 8.91
top - 14:31:19 up 43 min,  6 users,  load average: 10.67, 10.74, 8.92
top - 14:31:24 up 43 min,  6 users,  load average: 10.69, 10.74, 8.93
top - 14:31:29 up 43 min,  6 users,  load average: 10.72, 10.75, 8.94
top - 14:31:34 up 43 min,  6 users,  load average: 10.66, 10.73, 8.95
top - 14:31:39 up 43 min,  6 users,  load average: 10.77, 10.76, 8.97
top - 14:31:44 up 43 min,  6 users,  load average: 10.79, 10.76, 8.98
top - 14:31:49 up 43 min,  6 users,  load average: 10.64, 10.73, 8.98
top - 14:31:54 up 43 min,  6 users,  load average: 10.99, 10.80, 9.01
top - 14:31:59 up 43 min,  6 users,  load average: 10.83, 10.77, 9.01
top - 14:32:05 up 43 min,  6 users,  load average: 10.68, 10.74, 9.01
top - 14:32:10 up 43 min,  6 users,  load average: 10.63, 10.73, 9.01
top - 14:32:15 up 43 min,  6 users,  load average: 10.74, 10.75, 9.03
top - 14:32:20 up 44 min,  6 users,  load average: 10.76, 10.75, 9.04
top - 14:32:25 up 44 min,  6 users,  load average: 10.94, 10.79, 9.06
top - 14:32:30 up 44 min,  6 users,  load average: 10.86, 10.78, 9.06
top - 14:32:35 up 44 min,  6 users,  load average: 10.87, 10.78, 9.07
top - 14:32:40 up 44 min,  6 users,  load average: 10.88, 10.78, 9.08
top - 14:32:45 up 44 min,  6 users,  load average: 10.89, 10.79, 9.10
top - 14:32:50 up 44 min,  6 users,  load average: 10.90, 10.79, 9.10
top - 14:32:55 up 44 min,  6 users,  load average: 10.99, 10.81, 9.12
top - 14:33:00 up 44 min,  6 users,  load average: 11.07, 10.83, 9.14
top - 14:33:05 up 44 min,  6 users,  load average: 10.90, 10.80, 9.13
top - 14:33:10 up 44 min,  6 users,  load average: 10.75, 10.77, 9.13
top - 14:33:15 up 44 min,  6 users,  load average: 11.01, 10.82, 9.16
top - 14:33:20 up 45 min,  6 users,  load average: 11.17, 10.86, 9.18
top - 14:33:25 up 45 min,  6 users,  load average: 11.16, 10.86, 9.19
top - 14:33:30 up 45 min,  6 users,  load average: 11.22, 10.88, 9.20
top - 14:33:35 up 45 min,  6 users,  load average: 11.13, 10.86, 9.21
top - 14:33:41 up 45 min,  6 users,  load average: 11.12, 10.87, 9.22
top - 14:33:46 up 45 min,  6 users,  load average: 11.19, 10.88, 9.23
top - 14:33:51 up 45 min,  6 users,  load average: 11.17, 10.89, 9.24
top - 14:33:56 up 45 min,  6 users,  load average: 11.08, 10.87, 9.25
top - 14:34:01 up 45 min,  6 users,  load average: 10.99, 10.86, 9.25
top - 14:34:06 up 45 min,  6 users,  load average: 10.91, 10.84, 9.25
top - 14:34:11 up 45 min,  6 users,  load average: 10.76, 10.81, 9.25
top - 14:34:16 up 45 min,  6 users,  load average: 11.02, 10.86, 9.28
top - 14:34:21 up 46 min,  6 users,  load average: 11.10, 10.88, 9.29
top - 14:34:26 up 46 min,  6 users,  load average: 11.17, 10.90, 9.31
top - 14:34:31 up 46 min,  6 users,  load average: 11.23, 10.92, 9.32
top - 14:34:36 up 46 min,  6 users,  load average: 11.13, 10.90, 9.32
top - 14:34:41 up 46 min,  6 users,  load average: 11.20, 10.92, 9.34
top - 14:34:46 up 46 min,  6 users,  load average: 11.03, 10.89, 9.33
top - 14:34:51 up 46 min,  6 users,  load average: 11.10, 10.91, 9.35
top - 14:34:56 up 46 min,  6 users,  load average: 10.86, 10.86, 9.34
top - 14:35:01 up 46 min,  6 users,  load average: 10.95, 10.88, 9.36
top - 14:35:06 up 46 min,  6 users,  load average: 10.79, 10.85, 9.35
top - 14:35:11 up 46 min,  6 users,  load average: 11.21, 10.93, 9.39
top - 14:35:16 up 46 min,  6 users,  load average: 11.19, 10.93, 9.40
top - 14:35:21 up 47 min,  6 users,  load average: 11.09, 10.92, 9.40
top - 14:35:27 up 47 min,  6 users,  load average: 11.01, 10.90, 9.40
top - 14:35:32 up 47 min,  6 users,  load average: 11.01, 10.90, 9.41
top - 14:35:37 up 47 min,  6 users,  load average: 11.09, 10.92, 9.43
top - 14:35:42 up 47 min,  6 users,  load average: 11.00, 10.91, 9.43
top - 14:35:47 up 47 min,  6 users,  load average: 10.92, 10.89, 9.43
top - 14:35:52 up 47 min,  6 users,  load average: 11.00, 10.91, 9.45
top - 14:35:57 up 47 min,  6 users,  load average: 11.08, 10.93, 9.46
top - 14:36:02 up 47 min,  6 users,  load average: 11.24, 10.96, 9.48
top - 14:36:07 up 47 min,  6 users,  load average: 11.06, 10.93, 9.47
top - 14:36:12 up 47 min,  6 users,  load average: 11.45, 11.01, 9.51
top - 14:36:17 up 47 min,  6 users,  load average: 11.42, 11.01, 9.52
top - 14:36:22 up 48 min,  6 users,  load average: 11.46, 11.03, 9.53
top - 14:36:27 up 48 min,  6 users,  load average: 11.75, 11.09, 9.56
top - 14:36:32 up 48 min,  6 users,  load average: 11.53, 11.06, 9.56
top - 14:36:37 up 48 min,  6 users,  load average: 11.64, 11.09, 9.57
top - 14:36:42 up 48 min,  6 users,  load average: 11.67, 11.10, 9.59
top - 14:36:47 up 48 min,  6 users,  load average: 11.62, 11.10, 9.59
top - 14:36:52 up 48 min,  6 users,  load average: 11.65, 11.12, 9.61
top - 14:36:57 up 48 min,  6 users,  load average: 11.68, 11.13, 9.62
top - 14:37:02 up 48 min,  6 users,  load average: 11.70, 11.15, 9.63
top - 14:37:07 up 48 min,  6 users,  load average: 12.05, 11.23, 9.67
top - 14:37:13 up 48 min,  6 users,  load average: 12.20, 11.27, 9.69
top - 14:37:18 up 48 min,  6 users,  load average: 12.03, 11.25, 9.69
top - 14:37:23 up 49 min,  6 users,  load average: 12.10, 11.28, 9.71
top - 14:37:28 up 49 min,  6 users,  load average: 12.74, 11.42, 9.76
top - 14:37:33 up 49 min,  6 users,  load average: 12.68, 11.43, 9.78
top - 14:37:38 up 49 min,  6 users,  load average: 12.30, 11.38, 9.77
top - 14:37:43 up 49 min,  6 users,  load average: 12.52, 11.44, 9.79
top - 14:37:48 up 49 min,  6 users,  load average: 12.24, 11.40, 9.79
top - 14:37:53 up 49 min,  6 users,  load average: 12.22, 11.41, 9.80
top - 14:37:58 up 49 min,  6 users,  load average: 12.28, 11.43, 9.82
top - 14:38:03 up 49 min,  6 users,  load average: 12.10, 11.41, 9.82
top - 14:38:08 up 49 min,  6 users,  load average: 11.93, 11.38, 9.82
top - 14:38:13 up 49 min,  6 users,  load average: 11.77, 11.36, 9.82
top - 14:38:18 up 49 min,  6 users,  load average: 11.87, 11.39, 9.84
top - 14:38:23 up 50 min,  6 users,  load average: 12.12, 11.45, 9.86
top - 14:38:28 up 50 min,  6 users,  load average: 12.19, 11.47, 9.88
top - 14:38:33 up 50 min,  6 users,  load average: 12.10, 11.46, 9.89
top - 14:38:38 up 50 min,  6 users,  load average: 12.95, 11.68, 9.97
top - 14:38:43 up 50 min,  6 users,  load average: 12.80, 11.67, 9.98
top - 14:38:48 up 50 min,  6 users,  load average: 12.57, 11.64, 9.98
top - 14:38:54 up 50 min,  6 users,  load average: 12.69, 11.68, 10.00
top - 14:38:59 up 50 min,  6 users,  load average: 12.63, 11.68, 10.01
top - 14:39:04 up 50 min,  6 users,  load average: 12.42, 11.65, 10.01
top - 14:39:09 up 50 min,  6 users,  load average: 12.31, 11.64, 10.01
top - 14:39:14 up 50 min,  6 users,  load average: 12.12, 11.61, 10.01
top - 14:39:19 up 51 min,  6 users,  load average: 12.03, 11.60, 10.02
top - 14:39:24 up 51 min,  6 users,  load average: 11.71, 11.54, 10.01
top - 14:39:29 up 51 min,  6 users,  load average: 12.21, 11.65, 10.05
top - 14:39:34 up 51 min,  6 users,  load average: 12.03, 11.62, 10.05
top - 14:39:39 up 51 min,  6 users,  load average: 11.87, 11.60, 10.05
top - 14:39:44 up 51 min,  6 users,  load average: 11.64, 11.55, 10.04
top - 14:39:50 up 51 min,  6 users,  load average: 11.59, 11.54, 10.05
top - 14:39:55 up 51 min,  6 users,  load average: 11.38, 11.50, 10.04
top - 14:40:00 up 51 min,  6 users,  load average: 11.27, 11.48, 10.04
top - 14:40:05 up 51 min,  6 users,  load average: 11.01, 11.42, 10.03
top - 14:40:10 up 51 min,  6 users,  load average: 10.93, 11.39, 10.03
top - 14:40:15 up 51 min,  6 users,  load average: 10.85, 11.37, 10.03
top - 14:40:20 up 52 min,  6 users,  load average: 10.94, 11.38, 10.04
top - 14:40:25 up 52 min,  6 users,  load average: 10.87, 11.36, 10.04
top - 14:40:30 up 52 min,  6 users,  load average: 10.80, 11.33, 10.04
top - 14:40:35 up 52 min,  6 users,  load average: 10.81, 11.33, 10.04
top - 14:40:40 up 52 min,  6 users,  load average: 10.67, 11.29, 10.04
top - 14:40:45 up 52 min,  6 users,  load average: 10.86, 11.32, 10.05
top - 14:40:50 up 52 min,  6 users,  load average: 10.87, 11.31, 10.06
top - 14:40:55 up 52 min,  6 users,  load average: 10.96, 11.32, 10.07
top - 14:41:00 up 52 min,  6 users,  load average: 10.96, 11.32, 10.07
top - 14:41:05 up 52 min,  6 users,  load average: 11.20, 11.36, 10.09
top - 14:41:10 up 52 min,  6 users,  load average: 11.43, 11.41, 10.12
top - 14:41:16 up 52 min,  6 users,  load average: 11.47, 11.42, 10.12
top - 14:41:21 up 53 min,  6 users,  load average: 11.35, 11.39, 10.12
top - 14:41:26 up 53 min,  6 users,  load average: 11.41, 11.40, 10.13
top - 14:41:31 up 53 min,  6 users,  load average: 11.21, 11.36, 10.13
top - 14:41:36 up 53 min,  6 users,  load average: 11.28, 11.37, 10.14
top - 14:41:41 up 53 min,  6 users,  load average: 11.25, 11.37, 10.14
top - 14:41:46 up 53 min,  6 users,  load average: 11.15, 11.34, 10.14
top - 14:41:51 up 53 min,  6 users,  load average: 11.30, 11.37, 10.16
top - 14:41:56 up 53 min,  6 users,  load average: 11.28, 11.36, 10.16
top - 14:42:01 up 53 min,  6 users,  load average: 11.17, 11.34, 10.16
top - 14:42:06 up 53 min,  6 users,  load average: 11.16, 11.33, 10.16
top - 14:42:11 up 53 min,  6 users,  load average: 10.99, 11.30, 10.16
top - 14:42:16 up 53 min,  6 users,  load average: 10.83, 11.26, 10.15
top - 14:42:21 up 54 min,  6 users,  load average: 11.08, 11.30, 10.17
top - 14:42:26 up 54 min,  6 users,  load average: 10.99, 11.28, 10.17
top - 14:42:31 up 54 min,  6 users,  load average: 11.07, 11.29, 10.18
top - 14:42:37 up 54 min,  6 users,  load average: 10.99, 11.27, 10.18
top - 14:42:42 up 54 min,  6 users,  load average: 10.83, 11.23, 10.17
top - 14:42:47 up 54 min,  6 users,  load average: 10.68, 11.20, 10.17
top - 14:42:52 up 54 min,  6 users,  load average: 10.79, 11.21, 10.18
top - 14:42:57 up 54 min,  6 users,  load average: 10.88, 11.22, 10.19
top - 14:43:02 up 54 min,  6 users,  load average: 11.05, 11.25, 10.20
top - 14:43:07 up 54 min,  6 users,  load average: 10.89, 11.21, 10.19
top - 14:43:12 up 54 min,  6 users,  load average: 10.82, 11.19, 10.19
top - 14:43:17 up 54 min,  6 users,  load average: 10.83, 11.19, 10.20
top - 14:43:22 up 55 min,  6 users,  load average: 11.00, 11.22, 10.21
top - 14:43:27 up 55 min,  6 users,  load average: 11.00, 11.21, 10.21
top - 14:43:32 up 55 min,  6 users,  load average: 10.76, 11.16, 10.20
top - 14:43:37 up 55 min,  6 users,  load average: 10.06, 11.01, 10.16
top - 14:43:42 up 55 min,  6 users,  load average: 9.42, 10.86, 10.11
top - 14:43:47 up 55 min,  6 users,  load average: 8.90, 10.73, 10.08
top - 14:43:52 up 55 min,  6 users,  load average: 8.27, 10.57, 10.03
top - 14:43:57 up 55 min,  6 users,  load average: 8.01, 10.47, 10.00
top - 14:44:02 up 55 min,  6 users,  load average: 7.44, 10.32, 9.95
top - 14:44:07 up 55 min,  6 users,  load average: 6.93, 10.16, 9.90
top - 14:44:12 up 55 min,  6 users,  load average: 6.69, 10.06, 9.87
top - 14:44:17 up 55 min,  6 users,  load average: 6.64, 9.99, 9.85
top - 14:44:22 up 56 min,  6 users,  load average: 6.27, 9.86, 9.81
top - 14:44:27 up 56 min,  6 users,  load average: 5.84, 9.71, 9.76
top - 14:44:32 up 56 min,  6 users,  load average: 5.78, 9.63, 9.73
top - 14:44:37 up 56 min,  6 users,  load average: 5.63, 9.54, 9.70
top - 14:44:43 up 56 min,  6 users,  load average: 5.90, 9.53, 9.70
top - 14:44:48 up 56 min,  6 users,  load average: 6.95, 9.69, 9.75
top - 14:44:53 up 56 min,  6 users,  load average: 7.28, 9.71, 9.76
top - 14:44:58 up 56 min,  6 users,  load average: 7.49, 9.71, 9.76
top - 14:45:03 up 56 min,  6 users,  load average: 8.09, 9.80, 9.78
top - 14:45:08 up 56 min,  6 users,  load average: 8.25, 9.80, 9.79
top - 14:45:13 up 56 min,  6 users,  load average: 8.71, 9.87, 9.81
top - 14:45:18 up 56 min,  6 users,  load average: 8.65, 9.84, 9.80
top - 14:45:23 up 57 min,  6 users,  load average: 8.76, 9.84, 9.80
top - 14:45:28 up 57 min,  6 users,  load average: 8.86, 9.85, 9.80
top - 14:45:33 up 57 min,  6 users,  load average: 9.19, 9.90, 9.82
top - 14:45:38 up 57 min,  6 users,  load average: 9.49, 9.95, 9.83
top - 14:45:43 up 57 min,  6 users,  load average: 9.61, 9.97, 9.84
top - 14:45:48 up 57 min,  6 users,  load average: 10.05, 10.05, 9.87
top - 14:45:53 up 57 min,  6 users,  load average: 10.28, 10.10, 9.88
top - 14:45:58 up 57 min,  6 users,  load average: 10.34, 10.11, 9.89
top - 14:46:03 up 57 min,  6 users,  load average: 10.55, 10.16, 9.91
top - 14:46:08 up 57 min,  6 users,  load average: 10.67, 10.19, 9.92
top - 14:46:13 up 57 min,  6 users,  load average: 11.09, 10.30, 9.95
top - 14:46:18 up 58 min,  6 users,  load average: 10.92, 10.28, 9.95
top - 14:46:23 up 58 min,  6 users,  load average: 11.33, 10.37, 9.98
top - 14:46:28 up 58 min,  6 users,  load average: 11.38, 10.40, 9.99
top - 14:46:34 up 58 min,  6 users,  load average: 11.19, 10.38, 9.99
top - 14:46:39 up 58 min,  6 users,  load average: 11.02, 10.35, 9.98
top - 14:46:44 up 58 min,  6 users,  load average: 10.94, 10.35, 9.98
top - 14:46:49 up 58 min,  6 users,  load average: 10.94, 10.36, 9.99
top - 14:46:54 up 58 min,  6 users,  load average: 11.02, 10.38, 10.00
top - 14:46:59 up 58 min,  6 users,  load average: 10.86, 10.36, 9.99
top - 14:47:04 up 58 min,  6 users,  load average: 10.87, 10.37, 10.00
top - 14:47:09 up 58 min,  6 users,  load average: 10.80, 10.36, 10.00
top - 14:47:14 up 58 min,  6 users,  load average: 10.74, 10.36, 10.00
top - 14:47:19 up 59 min,  6 users,  load average: 10.84, 10.39, 10.01
top - 14:47:24 up 59 min,  6 users,  load average: 10.85, 10.39, 10.01
top - 14:47:29 up 59 min,  6 users,  load average: 11.02, 10.44, 10.03
top - 14:47:34 up 59 min,  6 users,  load average: 10.94, 10.43, 10.03
top - 14:47:39 up 59 min,  6 users,  load average: 10.87, 10.42, 10.03
top - 14:47:44 up 59 min,  6 users,  load average: 10.72, 10.40, 10.02
top - 14:47:49 up 59 min,  6 users,  load average: 10.66, 10.39, 10.02
top - 14:47:54 up 59 min,  6 users,  load average: 10.69, 10.40, 10.03
top - 14:47:59 up 59 min,  6 users,  load average: 10.71, 10.41, 10.03
top - 14:48:04 up 59 min,  6 users,  load average: 10.73, 10.42, 10.04
top - 14:48:09 up 59 min,  6 users,  load average: 10.75, 10.43, 10.04
top - 14:48:14 up 59 min,  6 users,  load average: 10.77, 10.44, 10.05
top - 14:48:20 up  1:00,  6 users,  load average: 11.03, 10.50, 10.07
top - 14:48:25 up  1:00,  6 users,  load average: 11.27, 10.56, 10.09
top - 14:48:30 up  1:00,  6 users,  load average: 11.33, 10.58, 10.10
top - 14:48:35 up  1:00,  6 users,  load average: 11.14, 10.55, 10.09
top - 14:48:40 up  1:00,  6 users,  load average: 11.29, 10.60, 10.11
top - 14:48:45 up  1:00,  6 users,  load average: 11.43, 10.63, 10.12
top - 14:48:50 up  1:00,  6 users,  load average: 11.47, 10.66, 10.13
top - 14:48:55 up  1:00,  6 users,  load average: 11.51, 10.68, 10.14
top - 14:49:00 up  1:00,  6 users,  load average: 11.39, 10.67, 10.14
top - 14:49:05 up  1:00,  6 users,  load average: 11.52, 10.71, 10.16
top - 14:49:10 up  1:00,  6 users,  load average: 11.40, 10.69, 10.16
top - 14:49:15 up  1:00,  6 users,  load average: 11.45, 10.72, 10.16
top - 14:49:20 up  1:01,  6 users,  load average: 11.41, 10.72, 10.17
top - 14:49:25 up  1:01,  6 users,  load average: 11.30, 10.71, 10.17
top - 14:49:30 up  1:01,  6 users,  load average: 11.75, 10.81, 10.20
top - 14:49:35 up  1:01,  6 users,  load average: 11.53, 10.78, 10.20
top - 14:49:40 up  1:01,  6 users,  load average: 11.41, 10.77, 10.20
top - 14:49:45 up  1:01,  6 users,  load average: 11.46, 10.79, 10.21
top - 14:49:50 up  1:01,  6 users,  load average: 11.34, 10.77, 10.20
top - 14:49:55 up  1:01,  6 users,  load average: 11.15, 10.75, 10.20
top - 14:50:00 up  1:01,  6 users,  load average: 11.38, 10.80, 10.22
top - 14:50:05 up  1:01,  6 users,  load average: 11.03, 10.74, 10.20
top - 14:50:10 up  1:01,  6 users,  load average: 11.11, 10.76, 10.21
top - 14:50:15 up  1:01,  6 users,  load average: 10.86, 10.71, 10.20
top - 14:50:20 up  1:02,  6 users,  load average: 11.11, 10.76, 10.22
top - 14:50:25 up  1:02,  6 users,  load average: 11.42, 10.83, 10.24
top - 14:50:31 up  1:02,  6 users,  load average: 11.47, 10.85, 10.25
top - 14:50:36 up  1:02,  6 users,  load average: 11.59, 10.89, 10.27
top - 14:50:41 up  1:02,  6 users,  load average: 11.94, 10.97, 10.30
top - 14:50:46 up  1:02,  6 users,  load average: 12.19, 11.04, 10.32
top - 14:50:51 up  1:02,  6 users,  load average: 12.65, 11.15, 10.36
top - 14:50:56 up  1:02,  6 users,  load average: 12.92, 11.23, 10.39
top - 14:51:01 up  1:02,  6 users,  load average: 13.17, 11.31, 10.42
top - 14:51:06 up  1:02,  6 users,  load average: 13.47, 11.41, 10.46
top - 14:51:11 up  1:02,  6 users,  load average: 13.76, 11.50, 10.49
top - 14:51:16 up  1:02,  6 users,  load average: 14.02, 11.59, 10.53
top - 14:51:21 up  1:03,  6 users,  load average: 13.37, 11.50, 10.50
top - 14:51:26 up  1:03,  6 users,  load average: 12.86, 11.42, 10.48
top - 14:51:31 up  1:03,  6 users,  load average: 12.23, 11.32, 10.45
top - 14:51:36 up  1:03,  6 users,  load average: 11.65, 11.21, 10.42
top - 14:51:41 up  1:03,  6 users,  load average: 11.20, 11.12, 10.40
top - 14:51:46 up  1:03,  6 users,  load average: 10.62, 11.01, 10.37
top - 14:51:51 up  1:03,  6 users,  load average: 9.93, 10.86, 10.32
top - 14:51:56 up  1:03,  6 users,  load average: 9.38, 10.73, 10.28
top - 14:52:01 up  1:03,  6 users,  load average: 8.95, 10.61, 10.25
top - 14:52:06 up  1:03,  6 users,  load average: 8.47, 10.49, 10.21
top - 14:52:12 up  1:03,  6 users,  load average: 7.95, 10.35, 10.16
top - 14:52:17 up  1:03,  6 users,  load average: 7.55, 10.22, 10.12
top - 14:52:22 up  1:04,  6 users,  load average: 7.27, 10.12, 10.09
top - 14:52:27 up  1:04,  6 users,  load average: 7.01, 10.02, 10.06
top - 14:52:32 up  1:04,  6 users,  load average: 6.61, 9.89, 10.02
top - 14:52:37 up  1:04,  6 users,  load average: 6.40, 9.79, 9.98
top - 14:52:42 up  1:04,  6 users,  load average: 6.12, 9.67, 9.94
top - 14:52:47 up  1:04,  6 users,  load average: 6.03, 9.60, 9.92
top - 14:52:52 up  1:04,  6 users,  load average: 5.95, 9.52, 9.89
top - 14:52:57 up  1:04,  6 users,  load average: 5.79, 9.43, 9.86
top - 14:53:02 up  1:04,  6 users,  load average: 5.65, 9.34, 9.83
top - 14:53:07 up  1:04,  6 users,  load average: 5.44, 9.23, 9.79
top - 14:53:12 up  1:04,  6 users,  load average: 5.24, 9.13, 9.75
top - 14:53:17 up  1:04,  6 users,  load average: 5.22, 9.06, 9.73
top - 14:53:22 up  1:05,  6 users,  load average: 4.97, 8.94, 9.69
top - 14:53:27 up  1:05,  6 users,  load average: 5.05, 8.89, 9.67
top - 14:53:32 up  1:05,  6 users,  load average: 5.36, 8.89, 9.66
top - 14:53:37 up  1:05,  6 users,  load average: 5.09, 8.78, 9.62
top - 14:53:42 up  1:05,  6 users,  load average: 4.85, 8.67, 9.58
top - 14:53:47 up  1:05,  6 users,  load average: 4.62, 8.56, 9.54
top - 14:53:52 up  1:05,  6 users,  load average: 4.33, 8.43, 9.49
top - 14:53:57 up  1:05,  6 users,  load average: 4.30, 8.36, 9.46
top - 14:54:02 up  1:05,  6 users,  load average: 4.04, 8.23, 9.42
top - 14:54:07 up  1:05,  6 users,  load average: 3.95, 8.15, 9.38
top - 14:54:12 up  1:05,  6 users,  load average: 4.04, 8.09, 9.36
top - 14:54:17 up  1:05,  6 users,  load average: 4.03, 8.03, 9.33
top - 14:54:22 up  1:06,  6 users,  load average: 5.07, 8.18, 9.37
top - 14:54:27 up  1:06,  6 users,  load average: 4.91, 8.09, 9.34
top - 14:54:32 up  1:06,  6 users,  load average: 4.83, 8.02, 9.31
top - 14:54:37 up  1:06,  6 users,  load average: 4.53, 7.90, 9.26
top - 14:54:42 up  1:06,  6 users,  load average: 4.64, 7.87, 9.24
top - 14:54:47 up  1:06,  6 users,  load average: 4.83, 7.86, 9.23
top - 14:54:52 up  1:06,  6 users,  load average: 4.61, 7.76, 9.19
top - 14:54:57 up  1:06,  6 users,  load average: 4.80, 7.75, 9.18
top - 14:55:02 up  1:06,  6 users,  load average: 4.49, 7.64, 9.14
top - 14:55:07 up  1:06,  6 users,  load average: 4.29, 7.54, 9.10
top - 14:55:12 up  1:06,  6 users,  load average: 4.19, 7.47, 9.07
top - 14:55:17 up  1:06,  6 users,  load average: 4.17, 7.41, 9.04
top - 14:55:23 up  1:07,  6 users,  load average: 4.40, 7.40, 9.03
top - 14:55:28 up  1:07,  6 users,  load average: 4.21, 7.31, 8.99
top - 14:55:33 up  1:07,  6 users,  load average: 4.43, 7.31, 8.98
top - 14:55:38 up  1:07,  6 users,  load average: 4.32, 7.23, 8.95
top - 14:55:43 up  1:07,  6 users,  load average: 4.29, 7.18, 8.92
top - 14:55:48 up  1:07,  6 users,  load average: 4.27, 7.13, 8.89
top - 14:55:53 up  1:07,  6 users,  load average: 4.09, 7.04, 8.85
top - 14:55:58 up  1:07,  6 users,  load average: 3.84, 6.94, 8.81
top - 14:56:03 up  1:07,  6 users,  load average: 3.69, 6.86, 8.78
top - 14:56:08 up  1:07,  6 users,  load average: 3.55, 6.78, 8.74
top - 14:56:13 up  1:07,  6 users,  load average: 3.43, 6.70, 8.70
top - 14:56:18 up  1:07,  6 users,  load average: 3.24, 6.60, 8.66
top - 14:56:23 up  1:08,  6 users,  load average: 3.22, 6.54, 8.63
top - 14:56:28 up  1:08,  6 users,  load average: 3.20, 6.48, 8.60
top - 14:56:33 up  1:08,  6 users,  load average: 3.34, 6.46, 8.58
top - 14:56:38 up  1:08,  6 users,  load average: 3.15, 6.37, 8.54
top - 14:56:43 up  1:08,  6 users,  load average: 3.14, 6.31, 8.51
top - 14:56:48 up  1:08,  6 users,  load average: 2.97, 6.22, 8.47
top - 14:56:53 up  1:08,  6 users,  load average: 2.81, 6.14, 8.43
top - 14:56:58 up  1:08,  6 users,  load average: 2.75, 6.07, 8.39
top - 14:57:03 up  1:08,  6 users,  load average: 2.77, 6.02, 8.36
top - 14:57:08 up  1:08,  6 users,  load average: 2.70, 5.95, 8.33
top - 14:57:13 up  1:08,  6 users,  load average: 3.22, 5.95, 8.30
top - 14:57:18 up  1:09,  6 users,  load average: 3.20, 5.90, 8.28
top - 14:57:23 up  1:09,  6 users,  load average: 3.10, 5.84, 8.24
top - 14:57:28 up  1:09,  6 users,  load average: 2.93, 5.76, 8.20
top - 14:57:33 up  1:09,  6 users,  load average: 2.78, 5.68, 8.16
top - 14:57:38 up  1:09,  6 users,  load average: 2.64, 5.60, 8.12
top - 14:57:43 up  1:09,  6 users,  load average: 2.51, 5.52, 8.09
top - 14:57:48 up  1:09,  6 users,  load average: 2.55, 5.48, 8.06
top - 14:57:53 up  1:09,  6 users,  load average: 2.66, 5.46, 8.04
top - 14:57:58 up  1:09,  6 users,  load average: 2.69, 5.41, 8.01
top - 14:58:04 up  1:09,  6 users,  load average: 2.71, 5.37, 7.98
top - 14:58:09 up  1:09,  6 users,  load average: 2.82, 5.35, 7.96
top - 14:58:14 up  1:09,  6 users,  load average: 2.91, 5.33, 7.94
top - 14:58:19 up  1:10,  6 users,  load average: 2.92, 5.29, 7.91
top - 14:58:24 up  1:10,  6 users,  load average: 2.76, 5.22, 7.87
top - 14:58:29 up  1:10,  6 users,  load average: 2.86, 5.20, 7.85
top - 14:58:34 up  1:10,  6 users,  load average: 2.79, 5.14, 7.82
top - 14:58:39 up  1:10,  6 users,  load average: 2.65, 5.08, 7.79
top - 14:58:44 up  1:10,  6 users,  load average: 2.76, 5.06, 7.76
top - 14:58:49 up  1:10,  6 users,  load average: 2.62, 4.99, 7.73
top - 14:58:54 up  1:10,  6 users,  load average: 2.65, 4.96, 7.70
top - 14:58:59 up  1:10,  6 users,  load average: 2.51, 4.89, 7.67
top - 14:59:04 up  1:10,  6 users,  load average: 2.63, 4.88, 7.65
top - 14:59:09 up  1:10,  6 users,  load average: 2.82, 4.88, 7.63
top - 14:59:14 up  1:10,  6 users,  load average: 2.68, 4.81, 7.60
top - 14:59:19 up  1:11,  6 users,  load average: 2.62, 4.77, 7.57
top - 14:59:24 up  1:11,  6 users,  load average: 2.57, 4.72, 7.54
top - 14:59:29 up  1:11,  6 users,  load average: 2.69, 4.71, 7.52
top - 14:59:34 up  1:11,  6 users,  load average: 2.63, 4.66, 7.49
top - 14:59:39 up  1:11,  6 users,  load average: 2.50, 4.60, 7.45
top - 14:59:44 up  1:11,  6 users,  load average: 2.54, 4.58, 7.43
top - 14:59:49 up  1:11,  6 users,  load average: 2.42, 4.52, 7.39
top - 14:59:54 up  1:11,  6 users,  load average: 2.38, 4.47, 7.36
top - 14:59:59 up  1:11,  6 users,  load average: 2.59, 4.48, 7.35
top - 15:00:04 up  1:11,  6 users,  load average: 2.70, 4.47, 7.33
top - 15:00:09 up  1:11,  6 users,  load average: 2.57, 4.42, 7.30
top - 15:00:14 up  1:11,  6 users,  load average: 2.44, 4.36, 7.26
top - 15:00:19 up  1:12,  6 users,  load average: 2.49, 4.34, 7.24
top - 15:00:24 up  1:12,  6 users,  load average: 2.53, 4.31, 7.22
top - 15:00:29 up  1:12,  6 users,  load average: 2.40, 4.26, 7.18
top - 15:00:34 up  1:12,  6 users,  load average: 2.37, 4.22, 7.16
top - 15:00:39 up  1:12,  6 users,  load average: 2.34, 4.18, 7.13
top - 15:00:44 up  1:12,  6 users,  load average: 2.23, 4.13, 7.10
top - 15:00:49 up  1:12,  6 users,  load average: 2.29, 4.11, 7.07
top - 15:00:55 up  1:12,  6 users,  load average: 2.19, 4.06, 7.04
top - 15:01:00 up  1:12,  6 users,  load average: 2.18, 4.02, 7.01
top - 15:01:05 up  1:12,  6 users,  load average: 2.08, 3.97, 6.98
top - 15:01:10 up  1:12,  6 users,  load average: 1.99, 3.92, 6.95
top - 15:01:15 up  1:12,  6 users,  load average: 1.99, 3.89, 6.92
top - 15:01:20 up  1:13,  6 users,  load average: 1.91, 3.84, 6.89
top - 15:01:25 up  1:13,  6 users,  load average: 1.84, 3.80, 6.86
top - 15:01:30 up  1:13,  6 users,  load average: 1.93, 3.78, 6.84
top - 15:01:35 up  1:13,  6 users,  load average: 1.86, 3.74, 6.80
top - 15:01:40 up  1:13,  6 users,  load average: 1.95, 3.72, 6.78
top - 15:01:45 up  1:13,  6 users,  load average: 2.03, 3.71, 6.76
top - 15:01:50 up  1:13,  6 users,  load average: 2.27, 3.73, 6.75
top - 15:01:55 up  1:13,  6 users,  load average: 2.41, 3.74, 6.74
top - 15:02:00 up  1:13,  6 users,  load average: 2.46, 3.72, 6.72
top - 15:02:05 up  1:13,  6 users,  load average: 2.50, 3.71, 6.70
top - 15:02:10 up  1:13,  6 users,  load average: 2.54, 3.70, 6.68
top - 15:02:15 up  1:13,  6 users,  load average: 2.74, 3.72, 6.67
top - 15:02:20 up  1:14,  6 users,  load average: 2.92, 3.74, 6.66
top - 15:02:25 up  1:14,  6 users,  load average: 3.08, 3.76, 6.65
top - 15:02:30 up  1:14,  6 users,  load average: 3.32, 3.80, 6.65
top - 15:02:35 up  1:14,  6 users,  load average: 3.13, 3.75, 6.62
top - 15:02:40 up  1:14,  6 users,  load average: 3.04, 3.72, 6.59
top - 15:02:45 up  1:14,  6 users,  load average: 2.96, 3.69, 6.57
top - 15:02:50 up  1:14,  6 users,  load average: 2.88, 3.67, 6.54
top - 15:02:55 up  1:14,  6 users,  load average: 2.97, 3.67, 6.53
top - 15:03:00 up  1:14,  6 users,  load average: 2.97, 3.66, 6.51
top - 15:03:05 up  1:14,  6 users,  load average: 2.89, 3.63, 6.48
top - 15:03:10 up  1:14,  6 users,  load average: 2.82, 3.61, 6.46
top - 15:03:15 up  1:14,  6 users,  load average: 2.68, 3.56, 6.43
top - 15:03:20 up  1:15,  6 users,  load average: 2.54, 3.52, 6.40
top - 15:03:25 up  1:15,  6 users,  load average: 2.42, 3.48, 6.37
top - 15:03:30 up  1:15,  6 users,  load average: 2.62, 3.50, 6.36
top - 15:03:35 up  1:15,  6 users,  load average: 2.73, 3.51, 6.35
top - 15:03:40 up  1:15,  6 users,  load average: 2.76, 3.50, 6.33
top - 15:03:45 up  1:15,  6 users,  load average: 3.10, 3.56, 6.34
top - 15:03:51 up  1:15,  6 users,  load average: 2.93, 3.52, 6.31
top - 15:03:56 up  1:15,  6 users,  load average: 3.01, 3.52, 6.29
top - 15:04:01 up  1:15,  6 users,  load average: 2.93, 3.50, 6.27
top - 15:04:06 up  1:15,  6 users,  load average: 2.86, 3.47, 6.25
top - 15:04:11 up  1:15,  6 users,  load average: 2.79, 3.45, 6.22
top - 15:04:16 up  1:15,  6 users,  load average: 3.04, 3.49, 6.22
top - 15:04:21 up  1:16,  6 users,  load average: 3.20, 3.52, 6.22
top - 15:04:26 up  1:16,  6 users,  load average: 3.18, 3.51, 6.20
top - 15:04:31 up  1:16,  6 users,  load average: 3.25, 3.51, 6.19
top - 15:04:36 up  1:16,  6 users,  load average: 3.07, 3.47, 6.16
top - 15:04:41 up  1:16,  6 users,  load average: 3.14, 3.48, 6.15
top - 15:04:46 up  1:16,  6 users,  load average: 2.89, 3.42, 6.11
top - 15:04:51 up  1:16,  6 users,  load average: 2.66, 3.37, 6.08
top - 15:04:56 up  1:16,  6 users,  load average: 2.61, 3.34, 6.06
top - 15:05:01 up  1:16,  6 users,  load average: 2.56, 3.32, 6.04
top - 15:05:06 up  1:16,  6 users,  load average: 2.35, 3.27, 6.00
top - 15:05:11 up  1:16,  6 users,  load average: 2.16, 3.21, 5.97
top - 15:05:16 up  1:16,  6 users,  load average: 2.23, 3.21, 5.95
top - 15:05:21 up  1:17,  6 users,  load average: 2.13, 3.17, 5.93
top - 15:05:26 up  1:17,  6 users,  load average: 2.20, 3.17, 5.91
top - 15:05:31 up  1:17,  6 users,  load average: 2.35, 3.18, 5.90
top - 15:05:36 up  1:17,  6 users,  load average: 2.16, 3.13, 5.87
top - 15:05:41 up  1:17,  6 users,  load average: 1.98, 3.07, 5.84
top - 15:05:46 up  1:17,  6 users,  load average: 1.83, 3.02, 5.81
top - 15:05:51 up  1:17,  6 users,  load average: 1.92, 3.02, 5.79
top - 15:05:56 up  1:17,  6 users,  load average: 2.01, 3.02, 5.78
top - 15:06:01 up  1:17,  6 users,  load average: 1.84, 2.97, 5.74
top - 15:06:06 up  1:17,  6 users,  load average: 1.70, 2.92, 5.71
top - 15:06:11 up  1:17,  6 users,  load average: 1.72, 2.91, 5.69
top - 15:06:16 up  1:17,  6 users,  load average: 1.58, 2.86, 5.66
top - 15:06:21 up  1:18,  6 users,  load average: 1.46, 2.81, 5.63
top - 15:06:26 up  1:18,  6 users,  load average: 1.34, 2.76, 5.60
top - 15:06:31 up  1:18,  6 users,  load average: 1.31, 2.73, 5.58
top - 15:06:36 up  1:18,  6 users,  load average: 1.45, 2.74, 5.56
top - 15:06:41 up  1:18,  6 users,  load average: 1.49, 2.73, 5.54
top - 15:06:46 up  1:18,  6 users,  load average: 1.37, 2.68, 5.51
top - 15:06:51 up  1:18,  6 users,  load average: 1.50, 2.69, 5.50
top - 15:06:56 up  1:18,  6 users,  load average: 1.46, 2.66, 5.47
top - 15:07:01 up  1:18,  6 users,  load average: 1.50, 2.65, 5.46
top - 15:07:06 up  1:18,  6 users,  load average: 1.46, 2.62, 5.43
top - 15:07:11 up  1:18,  6 users,  load average: 1.43, 2.59, 5.41
top - 15:07:16 up  1:18,  6 users,  load average: 1.31, 2.55, 5.38
top - 15:07:21 up  1:19,  6 users,  load average: 1.29, 2.52, 5.35
top - 15:07:26 up  1:19,  6 users,  load average: 1.18, 2.48, 5.33
top - 15:07:31 up  1:19,  6 users,  load average: 1.17, 2.45, 5.30
top - 15:07:36 up  1:19,  6 users,  load average: 1.31, 2.46, 5.29
top - 15:07:41 up  1:19,  6 users,  load average: 1.29, 2.44, 5.27
top - 15:07:46 up  1:19,  6 users,  load average: 1.19, 2.40, 5.24
top - 15:07:52 up  1:19,  6 users,  load average: 1.09, 2.36, 5.21
top - 15:07:57 up  1:19,  6 users,  load average: 1.16, 2.35, 5.19
top - 15:08:02 up  1:19,  6 users,  load average: 1.07, 2.31, 5.16
top - 15:08:07 up  1:19,  6 users,  load average: 0.98, 2.27, 5.14
top - 15:08:12 up  1:19,  6 users,  load average: 1.39, 2.34, 5.14
top - 15:08:17 up  1:19,  6 users,  load average: 1.43, 2.33, 5.12
top - 15:08:22 up  1:20,  6 users,  load average: 1.56, 2.34, 5.11
top - 15:08:27 up  1:20,  6 users,  load average: 1.59, 2.34, 5.09
top - 15:08:32 up  1:20,  6 users,  load average: 1.55, 2.31, 5.07
top - 15:08:37 up  1:20,  6 users,  load average: 1.74, 2.34, 5.07
top - 15:08:42 up  1:20,  6 users,  load average: 1.60, 2.30, 5.04
top - 15:08:47 up  1:20,  6 users,  load average: 1.47, 2.26, 5.01
top - 15:08:52 up  1:20,  6 users,  load average: 1.60, 2.28, 5.00
top - 15:08:57 up  1:20,  6 users,  load average: 1.47, 2.24, 4.97
top - 15:09:02 up  1:20,  6 users,  load average: 1.35, 2.20, 4.95
top - 15:09:07 up  1:20,  6 users,  load average: 1.24, 2.16, 4.92
top - 15:09:12 up  1:20,  6 users,  load average: 1.46, 2.19, 4.91
top - 15:09:17 up  1:20,  6 users,  load average: 1.35, 2.16, 4.89
top - 15:09:22 up  1:21,  6 users,  load average: 1.32, 2.14, 4.87
top - 15:09:27 up  1:21,  6 users,  load average: 1.21, 2.10, 4.84
top - 15:09:32 up  1:21,  6 users,  load average: 1.20, 2.08, 4.82
top - 15:09:37 up  1:21,  6 users,  load average: 1.10, 2.05, 4.79
top - 15:09:42 up  1:21,  6 users,  load average: 1.01, 2.01, 4.77
top - 15:09:47 up  1:21,  6 users,  load average: 1.09, 2.01, 4.75
top - 15:09:52 up  1:21,  6 users,  load average: 1.16, 2.01, 4.74
top - 15:09:57 up  1:21,  6 users,  load average: 1.15, 2.00, 4.72
top - 15:10:02 up  1:21,  6 users,  load average: 1.14, 1.98, 4.70
top - 15:10:07 up  1:21,  6 users,  load average: 1.37, 2.01, 4.69
top - 15:10:12 up  1:21,  6 users,  load average: 1.26, 1.98, 4.67
top - 15:10:17 up  1:21,  6 users,  load average: 1.16, 1.95, 4.64
top - 15:10:22 up  1:22,  6 users,  load average: 1.14, 1.93, 4.62
top - 15:10:27 up  1:22,  6 users,  load average: 1.05, 1.90, 4.60
top - 15:10:32 up  1:22,  6 users,  load average: 1.05, 1.88, 4.58
top - 15:10:37 up  1:22,  6 users,  load average: 1.20, 1.90, 4.57
top - 15:10:42 up  1:22,  6 users,  load average: 1.11, 1.87, 4.54
top - 15:10:47 up  1:22,  6 users,  load average: 1.02, 1.84, 4.52
top - 15:10:52 up  1:22,  6 users,  load average: 1.02, 1.82, 4.50
top - 15:10:57 up  1:22,  6 users,  load average: 0.93, 1.79, 4.47
top - 15:11:02 up  1:22,  6 users,  load average: 0.86, 1.76, 4.45
top - 15:11:07 up  1:22,  6 users,  load average: 1.03, 1.78, 4.44
top - 15:11:12 up  1:22,  6 users,  load average: 0.95, 1.75, 4.42
top - 15:11:17 up  1:22,  6 users,  load average: 1.19, 1.79, 4.42
top - 15:11:22 up  1:23,  6 users,  load average: 1.10, 1.76, 4.39
top - 15:11:27 up  1:23,  6 users,  load average: 1.09, 1.75, 4.37
top - 15:11:32 up  1:23,  6 users,  load average: 1.00, 1.72, 4.35
top - 15:11:37 up  1:23,  6 users,  load average: 0.92, 1.69, 4.33
top - 15:11:42 up  1:23,  6 users,  load average: 0.85, 1.66, 4.30
top - 15:11:47 up  1:23,  6 users,  load average: 0.78, 1.63, 4.28
top - 15:11:52 up  1:23,  6 users,  load average: 0.80, 1.62, 4.26
top - 15:11:57 up  1:23,  6 users,  load average: 0.81, 1.61, 4.24
top - 15:12:02 up  1:23,  6 users,  load average: 0.91, 1.62, 4.23
top - 15:12:08 up  1:23,  6 users,  load average: 0.83, 1.59, 4.21
top - 15:12:13 up  1:23,  6 users,  load average: 0.85, 1.58, 4.19
top - 15:12:18 up  1:23,  6 users,  load average: 0.78, 1.56, 4.17
top - 15:12:23 up  1:24,  6 users,  load average: 0.88, 1.56, 4.16
top - 15:12:28 up  1:24,  6 users,  load average: 1.21, 1.62, 4.16
top - 15:12:33 up  1:24,  6 users,  load average: 1.43, 1.66, 4.16
top - 15:12:38 up  1:24,  6 users,  load average: 1.40, 1.65, 4.14
top - 15:12:43 up  1:24,  6 users,  load average: 1.28, 1.62, 4.12
top - 15:12:48 up  1:24,  6 users,  load average: 1.42, 1.64, 4.11
top - 15:12:53 up  1:24,  6 users,  load average: 1.39, 1.63, 4.10
top - 15:12:58 up  1:24,  6 users,  load average: 1.36, 1.62, 4.08
top - 15:13:03 up  1:24,  6 users,  load average: 1.33, 1.61, 4.06
top - 15:13:08 up  1:24,  6 users,  load average: 1.46, 1.63, 4.06
top - 15:13:13 up  1:24,  6 users,  load average: 1.34, 1.61, 4.04
top - 15:13:18 up  1:24,  6 users,  load average: 1.24, 1.58, 4.01
top - 15:13:23 up  1:25,  6 users,  load average: 1.22, 1.57, 4.00
top - 15:13:28 up  1:25,  6 users,  load average: 1.12, 1.54, 3.97
top - 15:13:33 up  1:25,  6 users,  load average: 1.03, 1.52, 3.95
top - 15:13:38 up  1:25,  6 users,  load average: 1.11, 1.52, 3.94
top - 15:13:43 up  1:25,  6 users,  load average: 1.02, 1.50, 3.92
top - 15:13:48 up  1:25,  6 users,  load average: 1.10, 1.51, 3.91
top - 15:13:53 up  1:25,  6 users,  load average: 1.09, 1.50, 3.89
top - 15:13:58 up  1:25,  6 users,  load average: 1.32, 1.54, 3.89
top - 15:14:03 up  1:25,  6 users,  load average: 1.30, 1.53, 3.88
top - 15:14:08 up  1:25,  6 users,  load average: 1.35, 1.54, 3.87
top - 15:14:13 up  1:25,  6 users,  load average: 1.48, 1.56, 3.86
top - 15:14:18 up  1:25,  6 users,  load average: 1.85, 1.64, 3.88
top - 15:14:23 up  1:26,  6 users,  load average: 3.54, 1.99, 3.98
top - 15:14:28 up  1:26,  6 users,  load average: 4.99, 2.35, 4.07
top - 15:14:33 up  1:26,  6 users,  load average: 5.71, 2.55, 4.13
top - 15:14:39 up  1:26,  6 users,  load average: 5.90, 2.64, 4.15
top - 15:14:44 up  1:26,  6 users,  load average: 6.31, 2.77, 4.18
top - 15:14:49 up  1:26,  6 users,  load average: 6.20, 2.81, 4.19
top - 15:14:54 up  1:26,  6 users,  load average: 6.26, 2.88, 4.20
top - 15:14:59 up  1:26,  6 users,  load average: 6.00, 2.88, 4.20
top - 15:15:04 up  1:26,  6 users,  load average: 6.16, 2.97, 4.22
top - 15:15:09 up  1:26,  6 users,  load average: 5.91, 2.97, 4.21
top - 15:15:14 up  1:26,  6 users,  load average: 5.84, 3.00, 4.21
top - 15:15:19 up  1:27,  6 users,  load average: 5.93, 3.07, 4.23
top - 15:15:24 up  1:27,  6 users,  load average: 5.69, 3.07, 4.22
top - 15:15:29 up  1:27,  6 users,  load average: 6.12, 3.20, 4.26
top - 15:15:34 up  1:27,  6 users,  load average: 6.35, 3.29, 4.28
top - 15:15:39 up  1:27,  6 users,  load average: 6.72, 3.42, 4.32
top - 15:15:44 up  1:27,  6 users,  load average: 7.62, 3.66, 4.39
top - 15:15:49 up  1:27,  6 users,  load average: 8.29, 3.87, 4.45
top - 15:15:55 up  1:27,  6 users,  load average: 10.19, 4.33, 4.60
top - 15:16:00 up  1:27,  6 users,  load average: 10.10, 4.41, 4.62
top - 15:16:05 up  1:27,  6 users,  load average: 10.25, 4.54, 4.66
top - 15:16:10 up  1:27,  6 users,  load average: 10.07, 4.59, 4.68
top - 15:16:15 up  1:27,  6 users,  load average: 9.82, 4.63, 4.69
top - 15:16:20 up  1:28,  6 users,  load average: 9.60, 4.67, 4.71
top - 15:16:25 up  1:28,  6 users,  load average: 9.63, 4.76, 4.73
top - 15:16:30 up  1:28,  6 users,  load average: 9.66, 4.85, 4.76
top - 15:16:35 up  1:28,  6 users,  load average: 9.52, 4.90, 4.78
top - 15:16:40 up  1:28,  6 users,  load average: 9.16, 4.90, 4.78
top - 15:16:45 up  1:28,  6 users,  load average: 9.39, 5.02, 4.82
top - 15:16:50 up  1:28,  6 users,  load average: 9.12, 5.04, 4.83
top - 15:16:56 up  1:28,  6 users,  load average: 8.71, 5.02, 4.82
top - 15:17:01 up  1:28,  6 users,  load average: 8.25, 4.98, 4.81
top - 15:17:06 up  1:28,  6 users,  load average: 7.99, 4.98, 4.81
top - 15:17:11 up  1:28,  6 users,  load average: 7.75, 4.98, 4.81
top - 15:17:16 up  1:28,  6 users,  load average: 7.77, 5.03, 4.83
top - 15:17:21 up  1:29,  6 users,  load average: 7.79, 5.08, 4.85
top - 15:17:26 up  1:29,  6 users,  load average: 7.88, 5.15, 4.87
top - 15:17:31 up  1:29,  6 users,  load average: 8.05, 5.23, 4.89
top - 15:17:36 up  1:29,  6 users,  load average: 8.29, 5.32, 4.93
top - 15:17:41 up  1:29,  6 users,  load average: 8.51, 5.42, 4.96
top - 15:17:48 up  1:29,  6 users,  load average: 9.26, 5.68, 5.05
top - 15:17:53 up  1:29,  6 users,  load average: 9.48, 5.79, 5.09
top - 15:17:59 up  1:29,  6 users,  load average: 9.12, 5.77, 5.09
top - 15:18:04 up  1:29,  6 users,  load average: 8.63, 5.73, 5.08
top - 15:18:09 up  1:29,  6 users,  load average: 8.18, 5.68, 5.06
top - 15:18:14 up  1:29,  6 users,  load average: 8.01, 5.69, 5.07
top - 15:18:19 up  1:30,  6 users,  load average: 7.77, 5.68, 5.07
top - 15:18:24 up  1:30,  6 users,  load average: 7.62, 5.68, 5.07
top - 15:18:29 up  1:30,  6 users,  load average: 7.17, 5.62, 5.06
top - 15:18:34 up  1:30,  6 users,  load average: 6.84, 5.58, 5.05
top - 15:18:39 up  1:30,  6 users,  load average: 6.45, 5.52, 5.03
top - 15:18:44 up  1:30,  6 users,  load average: 6.17, 5.47, 5.02
top - 15:18:49 up  1:30,  6 users,  load average: 6.00, 5.45, 5.01
top - 15:18:54 up  1:30,  6 users,  load average: 6.24, 5.51, 5.03
top - 15:18:59 up  1:30,  6 users,  load average: 5.82, 5.43, 5.01
top - 15:19:04 up  1:30,  6 users,  load average: 5.67, 5.41, 5.01
top - 15:19:09 up  1:30,  6 users,  load average: 5.38, 5.35, 4.99
top - 15:19:14 up  1:30,  6 users,  load average: 5.11, 5.30, 4.97
top - 15:19:19 up  1:31,  6 users,  load average: 4.70, 5.21, 4.95
top - 15:19:24 up  1:31,  6 users,  load average: 4.48, 5.15, 4.93
top - 15:19:29 up  1:31,  6 users,  load average: 4.20, 5.08, 4.91
top - 15:19:34 up  1:31,  6 users,  load average: 3.95, 5.02, 4.89
top - 15:19:39 up  1:31,  6 users,  load average: 3.87, 4.98, 4.88
top - 15:19:44 up  1:31,  6 users,  load average: 3.64, 4.92, 4.86
top - 15:19:49 up  1:31,  6 users,  load average: 3.35, 4.83, 4.83
top - 15:19:54 up  1:31,  6 users,  load average: 3.16, 4.77, 4.81
top - 15:19:59 up  1:31,  6 users,  load average: 3.31, 4.77, 4.81
top - 15:20:04 up  1:31,  6 users,  load average: 3.28, 4.74, 4.80
top - 15:20:09 up  1:31,  6 users,  load average: 3.26, 4.72, 4.79
top - 15:20:15 up  1:31,  6 users,  load average: 3.00, 4.64, 4.76
top - 15:20:20 up  1:32,  6 users,  load average: 2.76, 4.56, 4.74
top - 15:20:25 up  1:32,  6 users,  load average: 2.70, 4.52, 4.72
top - 15:20:30 up  1:32,  6 users,  load average: 2.56, 4.46, 4.70
top - 15:20:35 up  1:32,  6 users,  load average: 2.36, 4.38, 4.68
top - 15:20:40 up  1:32,  6 users,  load average: 2.25, 4.33, 4.66
top - 15:20:45 up  1:32,  6 users,  load average: 2.07, 4.25, 4.63
top - 15:20:50 up  1:32,  6 users,  load average: 1.90, 4.18, 4.61
top - 15:20:55 up  1:32,  6 users,  load average: 1.91, 4.15, 4.59
top - 15:21:00 up  1:32,  6 users,  load average: 1.76, 4.08, 4.57
top - 15:21:05 up  1:32,  6 users,  load average: 1.69, 4.03, 4.55
top - 15:21:10 up  1:32,  6 users,  load average: 1.56, 3.96, 4.52
top - 15:21:15 up  1:32,  6 users,  load average: 1.43, 3.89, 4.50
top - 15:21:20 up  1:33,  6 users,  load average: 1.48, 3.86, 4.49
top - 15:21:25 up  1:33,  6 users,  load average: 2.24, 3.98, 4.52
top - 15:21:30 up  1:33,  6 users,  load average: 2.14, 3.93, 4.50
top - 15:21:35 up  1:33,  6 users,  load average: 2.85, 4.05, 4.54
top - 15:21:40 up  1:33,  6 users,  load average: 2.62, 3.98, 4.51
top - 15:21:45 up  1:33,  6 users,  load average: 2.41, 3.91, 4.49
top - 15:21:50 up  1:33,  6 users,  load average: 2.22, 3.85, 4.46
top - 15:21:55 up  1:33,  6 users,  load average: 2.04, 3.78, 4.44
top - 15:22:00 up  1:33,  6 users,  load average: 1.88, 3.72, 4.41
top - 15:22:05 up  1:33,  6 users,  load average: 1.73, 3.66, 4.39
top - 15:22:10 up  1:33,  6 users,  load average: 1.59, 3.60, 4.37
top - 15:22:15 up  1:33,  6 users,  load average: 1.46, 3.54, 4.34
top - 15:22:20 up  1:34,  6 users,  load average: 1.34, 3.48, 4.32
top - 15:22:25 up  1:34,  6 users,  load average: 1.24, 3.42, 4.30
top - 15:22:30 up  1:34,  6 users,  load average: 1.14, 3.36, 4.27
top - 15:22:35 up  1:34,  6 users,  load average: 1.04, 3.31, 4.25
top - 15:22:40 up  1:34,  6 users,  load average: 0.96, 3.25, 4.23
top - 15:22:45 up  1:34,  6 users,  load average: 0.88, 3.20, 4.20
top - 15:22:50 up  1:34,  6 users,  load average: 0.81, 3.15, 4.18
top - 15:22:55 up  1:34,  6 users,  load average: 0.75, 3.09, 4.16
top - 15:23:00 up  1:34,  6 users,  load average: 0.69, 3.04, 4.14
top - 15:23:05 up  1:34,  6 users,  load average: 0.63, 2.99, 4.11
top - 15:23:10 up  1:34,  6 users,  load average: 0.58, 2.94, 4.09
top - 15:23:15 up  1:34,  6 users,  load average: 0.53, 2.89, 4.07
top - 15:23:20 up  1:35,  6 users,  load average: 0.49, 2.84, 4.05
top - 15:23:25 up  1:35,  6 users,  load average: 0.45, 2.80, 4.02
top - 15:23:30 up  1:35,  6 users,  load average: 0.42, 2.75, 4.00
top - 15:23:35 up  1:35,  6 users,  load average: 0.38, 2.70, 3.98
top - 15:23:40 up  1:35,  6 users,  load average: 0.35, 2.66, 3.96
top - 15:23:45 up  1:35,  6 users,  load average: 0.32, 2.61, 3.94
top - 15:23:50 up  1:35,  6 users,  load average: 0.30, 2.57, 3.92
top - 15:23:55 up  1:35,  6 users,  load average: 0.27, 2.53, 3.89
top - 15:24:00 up  1:35,  6 users,  load average: 0.25, 2.49, 3.87
top - 15:24:05 up  1:35,  6 users,  load average: 0.23, 2.44, 3.85
top - 15:24:10 up  1:35,  6 users,  load average: 0.21, 2.40, 3.83
top - 15:24:15 up  1:35,  6 users,  load average: 0.19, 2.36, 3.81
top - 15:24:20 up  1:36,  6 users,  load average: 0.18, 2.32, 3.79
top - 15:24:25 up  1:36,  6 users,  load average: 0.16, 2.28, 3.77
top - 15:24:30 up  1:36,  6 users,  load average: 0.15, 2.25, 3.75
top - 15:24:35 up  1:36,  6 users,  load average: 0.14, 2.21, 3.73
top - 15:24:40 up  1:36,  6 users,  load average: 0.13, 2.17, 3.71
top - 15:24:45 up  1:36,  6 users,  load average: 0.12, 2.14, 3.69
top - 15:24:50 up  1:36,  6 users,  load average: 0.11, 2.10, 3.67
top - 15:24:55 up  1:36,  6 users,  load average: 0.10, 2.06, 3.65
top - 15:25:00 up  1:36,  6 users,  load average: 0.09, 2.03, 3.63
top - 15:25:05 up  1:36,  6 users,  load average: 0.08, 2.00, 3.61
top - 15:25:11 up  1:36,  6 users,  load average: 0.08, 1.96, 3.59
top - 15:25:16 up  1:36,  6 users,  load average: 0.07, 1.93, 3.57
top - 15:25:21 up  1:37,  6 users,  load average: 0.06, 1.90, 3.55
top - 15:25:26 up  1:37,  6 users,  load average: 0.06, 1.87, 3.53
top - 15:25:31 up  1:37,  6 users,  load average: 0.05, 1.83, 3.51
top - 15:25:36 up  1:37,  6 users,  load average: 0.05, 1.80, 3.49
top - 15:25:41 up  1:37,  6 users,  load average: 0.05, 1.77, 3.47
top - 15:25:46 up  1:37,  6 users,  load average: 0.04, 1.74, 3.45
top - 15:25:51 up  1:37,  6 users,  load average: 0.04, 1.71, 3.44
top - 15:25:56 up  1:37,  6 users,  load average: 0.03, 1.69, 3.42
#3 lore raw
Quoted 14 lines · Anant Nitya
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Anant Nitya

Hi,
Ever since I upgrade to 2.6.21/.1, system log is filled with following messages if I enable CONFIG_NO_HZ=y, going through archives it seems ingo sometime back posted some patch and now it is upstream, but its not helping here. If I disable NOHZ by kernel command line nohz=off this problem disappears. This system is P4/2.40GHz/HT with SMP/SMT on in kernel config. One more thing that I noticed is this problem only arises while using X or network otherwise plain command line with no network access don't trigger this with nohz=on.

Quoted 2 lines

Is this independent of the load on the system ? i.e. : What happens if you only use the console and run a kernel compile with -j4 ?

Quoted 13 lines

Yep, it seems independent of load on the system, to test I compiled kernel with make -j8 and to throw some more load same time I also used amavisd to clean around 2000 of spam/virus infected mails and same time was listening to few radio stream over net in console only login mode. Even then there was not a single NOHZ: local_softirq_pending message in log. Once kernel got compiled and amavisd finished its job and load drops back to around 0.5/2.0 I started X and within few secs log starts getting filled with local_softirq_pending messages, (sorry, I didn't applied ratelimit patch since I wanted to test is it high load on system that causes this or something else). It seems even network operation is not causing this. It seems either X is doing something terribly wrong or kernel is getting hosed by X. If some more information is needed please feel free to ask

Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.

When you apply the ratelimit patch, does the softlockup problem persist ?

tglx

#4 lore raw
Quoted 3 lines · On Friday 11 May 2007 03:28:46 Thomas Gleixner wrote:

Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.

Yes with HT disabled in BIOS there is no local_softirq_pending messages. BTW why does this problem persist only with X ?

Quoted 3 lines

When you apply the ratelimit patch, does the softlockup problem persist ?

Yes, though softlockup is rare and mostly hit when system is under high load. Apart of that I am also getting following messages consistently across multiple boot cycles with NOHZ=y and ratelimit patch applied.

May 15 11:51:22 rudra kernel: [ 2594.341068] Clocksource tsc unstable (delta = 
28111260302 ns)
May 15 11:51:22 rudra kernel: [ 2594.343194] Time: acpi_pm clocksource has 

been installed.

#5 lore raw
Quoted 7 lines · Anant Nitya
Quoted 3 lines · On Friday 11 May 2007 03:28:46 Thomas Gleixner wrote:

Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.

Yes with HT disabled in BIOS there is no local_softirq_pending messages. BTW why does this problem persist only with X ?

No idea. I uploaded a debug patch against 2.6.22-rc1 to

http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch

Can you give it a try and report the output ?

Quoted 11 lines
Quoted 3 lines

When you apply the ratelimit patch, does the softlockup problem persist ?

Yes, though softlockup is rare and mostly hit when system is under high load. Apart of that I am also getting following messages consistently across multiple boot cycles with NOHZ=y and ratelimit patch applied.

May 15 11:51:22 rudra kernel: [ 2594.341068] Clocksource tsc unstable (delta = 
28111260302 ns)
May 15 11:51:22 rudra kernel: [ 2594.343194] Time: acpi_pm clocksource has 

been installed.

That's informal. The TSC is detected to be unstable and replaced by the pm timer. Nothing to worry about. It happens with NOHZ=n as well, right ?

tglx

#6 lore raw
Quoted 6 lines · On Friday 18 May 2007 18:31:17 Thomas Gleixner wrote:
Quoted 5 lines · Anant Nitya
Quoted 3 lines · On Friday 11 May 2007 03:28:46 Thomas Gleixner wrote:

Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.

Quoted 8 lines
Quoted 2 lines

Yes with HT disabled in BIOS there is no local_softirq_pending messages. BTW why does this problem persist only with X ?

No idea. I uploaded a debug patch against 2.6.22-rc1 to

http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch

Can you give it a try and report the output ?

I am compiling kernel with above patch applied and will post the results.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

When you apply the ratelimit patch, does the softlockup problem persist ?

Yes, though softlockup is rare and mostly hit when system is under high load. Apart of that I am also getting following messages consistently across multiple boot cycles with NOHZ=y and ratelimit patch applied.

Quoted 8 lines
Quoted 4 lines
May 15 11:51:22 rudra kernel: [ 2594.341068] Clocksource tsc unstable
(delta = 28111260302 ns)
May 15 11:51:22 rudra kernel: [ 2594.343194] Time: acpi_pm clocksource

has been installed.

That's informal. The TSC is detected to be unstable and replaced by the pm timer. Nothing to worry about. It happens with NOHZ=n as well, right ?

Nope it don't appear with nohz=no till now across so many boot cycles, it always appears with nohz!=no.

Regards
Ananitya

#7 lore raw
In reply to Thomas Gleixner
Quoted 6 lines · On Friday 18 May 2007 18:31:17 Thomas Gleixner wrote:
Quoted 5 lines · Anant Nitya
Quoted 3 lines · On Friday 11 May 2007 03:28:46 Thomas Gleixner wrote:

Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.

Quoted 8 lines
Quoted 2 lines

Yes with HT disabled in BIOS there is no local_softirq_pending messages. BTW why does this problem persist only with X ?

No idea. I uploaded a debug patch against 2.6.22-rc1 to

http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch

Can you give it a try and report the output ?

Hi
Here it goes

[  159.646196] NOHZ softirq pending 22 on CPU 0
[  159.646207] .... task state: 1 00000000
[  159.646217] .... last caller: __tasklet_schedule
[  159.646997] NOHZ softirq pending 22 on CPU 0
[  159.647006] .... task state: 1 00000000
[  159.647013] .... last caller: __tasklet_schedule
[  159.647398] NOHZ softirq pending 22 on CPU 0
[  159.647405] .... task state: 1 00000000
[  159.647412] .... last caller: __tasklet_schedule
[  159.647768] NOHZ softirq pending 22 on CPU 0
[  159.647775] .... task state: 1 00000000
[  159.647781] .... last caller: __tasklet_schedule
[  166.285664] NOHZ softirq pending 22 on CPU 0
[  166.285675] .... task state: 1 00000000
[  166.285687] .... last caller: raise_softirq
[  166.286321] NOHZ softirq pending 22 on CPU 0
[  166.286329] .... task state: 1 00000000
[  166.286337] .... last caller: raise_softirq
[  166.286715] NOHZ softirq pending 22 on CPU 0
[  166.286722] .... task state: 1 00000000
[  166.286729] .... last caller: raise_softirq
[  166.287085] NOHZ softirq pending 22 on CPU 0
[  166.287092] .... task state: 1 00000000
[  166.287098] .... last caller: raise_softirq
[  171.512134] NOHZ softirq pending 22 on CPU 0
[  171.512144] .... task state: 1 00000000
[  171.512154] .... last caller: __tasklet_schedule
[  171.512712] NOHZ softirq pending 22 on CPU 0
[  171.512720] .... task state: 1 00000000
[  171.512727] .... last caller: __tasklet_schedule

Regards
Ananitya

#8 lore raw
Quoted 9 lines · Anant Nitya
Quoted 5 lines

No idea. I uploaded a debug patch against 2.6.22-rc1 to

http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch

Can you give it a try and report the output ?

Hi
Here it goes

[  159.646196] NOHZ softirq pending 22 on CPU 0
[  159.646207] .... task state: 1 00000000

1 == TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.

I work out a more complex debug patch and pester you to test once I'm done.

tglx

#9 lore raw
Quoted 12 lines · On Sunday 20 May 2007 00:41:08 Thomas Gleixner wrote:
Quoted 10 lines · Anant Nitya
Quoted 5 lines

No idea. I uploaded a debug patch against 2.6.22-rc1 to

http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch

Can you give it a try and report the output ?

Hi
Here it goes

[  159.646196] NOHZ softirq pending 22 on CPU 0
[  159.646207] .... task state: 1 00000000
Quoted 5 lines

1 == TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.

I work out a more complex debug patch and pester you to test once I'm done.

No problem :)

Regards
Ananitya

Quoted 2 lines

tglx

Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of
those who have achieved perfection, hardly one knows Me in truth.
				-- Gita Sutra Of Mysticism
#10 lore raw
Quoted 6 lines · Anant Nitya
Quoted 5 lines

1 == TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.

I work out a more complex debug patch and pester you to test once I'm done.

No problem :)

You asked for it :)

Please patch 2.6.22-rc2 with

http://tglx.de/projects/hrtimers/2.6.22-rc2/patch-2.6.22-rc2-hrt2.patch
and
http://www.tglx.de/private/tglx/ht-debug/tracer.diff

Compile it with the config

http://www.tglx.de/private/tglx/ht-debug/config.debug

You should find something like:

( swapper-0 |#0): new 67173 us user-latency.

along with the familiar "NOHZ ......" message in your log file.

Once that happened please do:

$ cat /proc/latency_trace >trace.txt

compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.

Michal,

IIRC you encountered the same P4/HT related wreckage. Can you do the same ?

Thanks,

tglx

#11 lore raw
Quoted 25 lines · On Monday 21 May 2007 03:13:08 Thomas Gleixner wrote:
Quoted 7 lines · Anant Nitya
Quoted 5 lines

1 == TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.

I work out a more complex debug patch and pester you to test once I'm done.

No problem :)

You asked for it :)

Please patch 2.6.22-rc2 with

http://tglx.de/projects/hrtimers/2.6.22-rc2/patch-2.6.22-rc2-hrt2.patch
and
http://www.tglx.de/private/tglx/ht-debug/tracer.diff

Compile it with the config

http://www.tglx.de/private/tglx/ht-debug/config.debug

You should find something like:

( swapper-0 |#0): new 67173 us user-latency.

Quoted 8 lines

along with the familiar "NOHZ ......" message in your log file.

Once that happened please do:

$ cat /proc/latency_trace >trace.txt

compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.

Hi Thomas

Here are the links...
http://cybertek.info/taitai/dmesg-2.6.22.rc2.hrt2-1.SMP.DN.LINUX.txt
http://cybertek.info/taitai/trace.txt.bz2

Regards
Ananitya

Quoted 9 lines

Michal,

IIRC you encountered the same P4/HT related wreckage. Can you do the same ?

Thanks,

tglx

Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of
those who have achieved perfection, hardly one knows Me in truth.
				-- Gita Sutra Of Mysticism
#12 lore raw
Quoted 10 lines · Anant Nitya
Quoted 10 lines

You should find something like:

( swapper-0 |#0): new 67173 us user-latency.

along with the familiar "NOHZ ......" message in your log file.

Once that happened please do:

$ cat /proc/latency_trace >trace.txt
Quoted 7 lines
Quoted 2 lines

compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.

Hi Thomas

Here are the links...
http://cybertek.info/taitai/dmesg-2.6.22.rc2.hrt2-1.SMP.DN.LINUX.txt
http://cybertek.info/taitai/trace.txt.bz2

Thanks. Sorry, I need more info. I uploaded a new tracer.diff to

http://www.tglx.de/private/tglx/ht-debug/tracer.diff

Can you please revert the first one and retest with the new one ?

Thanks,

tglx

#13 lore raw
Quoted 20 lines · On Monday 21 May 2007 14:31:57 Thomas Gleixner wrote:
Quoted 10 lines · Anant Nitya
Quoted 10 lines

You should find something like:

( swapper-0 |#0): new 67173 us user-latency.

along with the familiar "NOHZ ......" message in your log file.

Once that happened please do:

$ cat /proc/latency_trace >trace.txt
Quoted 8 lines
Quoted 2 lines

compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.

Hi Thomas

Here are the links...
http://cybertek.info/taitai/dmesg-2.6.22.rc2.hrt2-1.SMP.DN.LINUX.txt
http://cybertek.info/taitai/trace.txt.bz2

Quoted 5 lines

Thanks. Sorry, I need more info. I uploaded a new tracer.diff to

http://www.tglx.de/private/tglx/ht-debug/tracer.diff

Can you please revert the first one and retest with the new one ?

Okay sure, compiling now.

Regards
Ananitya

Quoted 4 lines

Thanks,

tglx

Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of
those who have achieved perfection, hardly one knows Me in truth.
				-- Gita Sutra Of Mysticism
#14 lore raw
In reply to Thomas Gleixner
Quoted 20 lines · On Monday 21 May 2007 14:31:57 Thomas Gleixner wrote:
Quoted 10 lines · Anant Nitya
Quoted 10 lines

You should find something like:

( swapper-0 |#0): new 67173 us user-latency.

along with the familiar "NOHZ ......" message in your log file.

Once that happened please do:

$ cat /proc/latency_trace >trace.txt
Quoted 8 lines
Quoted 2 lines

compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.

Hi Thomas

Here are the links...
http://cybertek.info/taitai/dmesg-2.6.22.rc2.hrt2-1.SMP.DN.LINUX.txt
http://cybertek.info/taitai/trace.txt.bz2

Quoted 7 lines

Thanks. Sorry, I need more info. I uploaded a new tracer.diff to

http://www.tglx.de/private/tglx/ht-debug/tracer.diff

Can you please revert the first one and retest with the new one ?

Thanks,

Sorry for delay, here is link for output from new tacer.
http://cybertek.info/taitai/trace-new.txt.bz2

Regards
Ananitya

Quoted 2 lines

tglx

Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of
those who have achieved perfection, hardly one knows Me in truth.
				-- Gita Sutra Of Mysticism
#15 lore raw
In reply to Thomas Gleixner

Hi Thomas,

Quoted 30 lines · Thomas Gleixner
Quoted 6 lines · Anant Nitya
Quoted 5 lines

1 == TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.

I work out a more complex debug patch and pester you to test once I'm done.

No problem :)

You asked for it :)

Please patch 2.6.22-rc2 with

http://tglx.de/projects/hrtimers/2.6.22-rc2/patch-2.6.22-rc2-hrt2.patch
and
http://www.tglx.de/private/tglx/ht-debug/tracer.diff

Compile it with the config

http://www.tglx.de/private/tglx/ht-debug/config.debug

You should find something like:

( swapper-0 |#0): new 67173 us user-latency.

along with the familiar "NOHZ ......" message in your log file.

Once that happened please do:

$ cat /proc/latency_trace >trace.txt
Quoted 7 lines

compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.

Michal,

IIRC you encountered the same P4/HT related wreckage. Can you do the same ?

Good news - I can't reproduce this bug. It's time to remove
"Subject : 2.6.21-git4 BUG: soft lockup detected on CPU#1!
References : http://lkml.org/lkml/2007/5/2/511
Submitter : Michal Piotrowski <[email protected]>

Status : problem is being debugged"
from KR.

Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq works

http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrt-config
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrt-console.log

Now I'm trying to apply hrtimers debug patch ontop 2.6.22-rc2-hrt3
http://lkml.org/lkml/2007/3/23/106

Regards,
Michal

Signature · 3 lines
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)
#16 lore raw

Michal Piotrowski napisał(a):

Quoted 54 lines

Hi Thomas,

Quoted 31 lines · Thomas Gleixner
Quoted 1 line · Anant Nitya
Quoted 1 line

1 == TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken

up. At

Quoted 5 lines
Quoted 4 lines

least it is not a scheduler problem.

I work out a more complex debug patch and pester you to test once I'm done.

No problem :)

You asked for it :)

Please patch 2.6.22-rc2 with

http://tglx.de/projects/hrtimers/2.6.22-rc2/patch-2.6.22-rc2-hrt2.patch
and
http://www.tglx.de/private/tglx/ht-debug/tracer.diff

Compile it with the config

http://www.tglx.de/private/tglx/ht-debug/config.debug

You should find something like:

( swapper-0 |#0): new 67173 us user-latency.

along with the familiar "NOHZ ......" message in your log file.

Once that happened please do:

$ cat /proc/latency_trace >trace.txt
Quoted 7 lines

compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.

Michal,

IIRC you encountered the same P4/HT related wreckage. Can you do the same ?

Good news - I can't reproduce this bug. It's time to remove
"Subject : 2.6.21-git4 BUG: soft lockup detected on CPU#1!
References : http://lkml.org/lkml/2007/5/2/511
Submitter : Michal Piotrowski <[email protected]>

Status : problem is being debugged"
from KR.

Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq works

Quoted 8 lines

http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrt-config

http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrt-console.log

Now I'm trying to apply hrtimers debug patch ontop 2.6.22-rc2-hrt3
http://lkml.org/lkml/2007/3/23/106

It _almost_ works

http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrtimers_debug.patch

Hmmm..

[  135.206505] SysRq : Show Pending Timers
[  135.210476] Timer List Version: v0.3
[  135.214102] HRTIMER_MAX_CLOCK_BASES: 2
[  135.217899] now at 112955954742 nsecs
[  135.221651]
[  135.221662] cpu: 0
[  135.225246]  clock 0:
[  135.227588]   .index:      0
[  135.230545]   .resolution: 1 nsecs
[  135.234010]   .get_time:   ktime_get_real
[  135.238151]   .offset:     1179862680097808665 nsecs
[  135.243207] active timers:
[  135.245992]  clock 1:
[  135.248316]   .index:      1
[  135.251246]   .resolution: 1 nsecs
[  135.254722]   .get_time:   ktime_get
[  135.258473]   .offset:     0 nsecs
[  135.261923] active timers:
[  135.264683]  #0: hardirq_stack, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
[  135.273913]  # expires at 112952000000 nsecs [in 18446744073705596874 nsecs]
[  135.281194]  #1: hardirq_stack, hrtimer_wakeup, S:01, futex_wait, automount/2293
[  135.289057]  # expires at 113902202172 nsecs [in 946247430 nsecs]
[  135.295233]  #2: hardirq_stack, hrtimer_wakeup, S:01, do_nanosleep, crond/2373
[  135.302973]  # expires at 121028637388 nsecs [in 8072682646 nsecs]
[  135.309279]  #3: hardirq_stack, it_real_fn, S:01, do_setitimer, syslogd/2171
[  135.316701]  # expires at 121465401717 nsecs [in 8509446975 nsecs]
[  135.323035]   .expires_next   : 112952000000 nsecs
[  135.327921]   .exp_prev       : 112951000000 nsecs
[  135.332786]  last expires_next stacktrace:
[  135.336944]   update_cpu_base_expires_next
[  135.341188]   hrtimer_interrupt
[  135.344430]   smp_apic_timer_interrupt
[  135.348310]   apic_timer_interrupt
[  135.351845]   __mcount
[  135.354327]   mcount
[  135.356627]   permission
[  135.359298]   vfs_permission
[  135.362305]   __link_path_walk
[  135.365497]   link_path_walk
[  135.368486]   path_walk
[  135.371044]   do_path_lookup
[  135.374086]   __user_walk_fd
[  135.377122]   vfs_stat_fd
[  135.379905]   vfs_stat
[  135.382385]   sys_stat64
[  135.385065]   syscall_call
[  135.387909]   <3>BUG: sleeping function called from invalid context at kernel/mutex.c:86
[  135.396175] in_atomic():1, irqs_disabled():1
[  135.400541] 2 locks held by readahead/2442:
[  135.404783]  #0:  (&serio->lock){++..}, at: [<c02dbf17>] serio_interrupt+0x22/0x79
[  135.412623]  #1:  (sysrq_key_table_lock){+...}, at: [<c027a4af>] __handle_sysrq+0x1b/0x10c
[  135.421208] irq event stamp: 3249062
[  135.424856] hardirqs last  enabled at (3249061): [<c0181b67>] kmem_cache_free+0x97/0xa1
[  135.433050] hardirqs last disabled at (3249062): [<c0104bd4>] common_interrupt+0x24/0x34
[  135.441350] softirqs last  enabled at (3248980): [<c012a418>] __do_softirq+0xe8/0xef
[  135.449261] softirqs last disabled at (3248975): [<c0106df7>] do_softirq+0x69/0xd6
[  135.457049]  [<c010526c>] show_trace_log_lvl+0x35/0x54
[  135.462312]  [<c0106027>] show_trace+0x2c/0x2e
[  135.466876]  [<c01060d0>] dump_stack+0x29/0x2b
[  135.471423]  [<c011de0c>] __might_sleep+0xe0/0xe2
[  135.476221]  [<c0360257>] mutex_lock+0x22/0x2e
[  135.480767]  [<c014eb96>] lookup_module_symbol_name+0x1a/0xc4
[  135.486739]  [<c014f514>] lookup_symbol_name+0x62/0x69
[  135.492021]  [<c013eb45>] print_name_offset+0x29/0x81
[  135.497200]  [<c013f4c5>] timer_list_show+0x657/0xc55
[  135.502368]  [<c013fae2>] sysrq_timer_list_show+0x1f/0x21
[  135.507859]  [<c027a5fd>] sysrq_handle_show_timers+0xd/0xf
[  135.513451]  [<c027a525>] __handle_sysrq+0x91/0x10c
[  135.518507]  [<c027a79e>] handle_sysrq+0x37/0x39
[  135.523296]  [<c0274b00>] kbd_event+0x335/0x58d
[  135.527921]  [<c02df806>] input_event+0x43e/0x460
[  135.532718]  [<c02e3766>] atkbd_interrupt+0x467/0x58d
[  135.537887]  [<c02dbf37>] serio_interrupt+0x42/0x79
[  135.542865]  [<c02dceed>] i8042_interrupt+0x1ff/0x210
[  135.548130]  [<c0160963>] handle_IRQ_event+0x24/0x54
[  135.553187]  [<c0161b2a>] handle_edge_irq+0xd7/0x11f
[  135.558253]  [<c0106f46>] do_IRQ+0xe2/0x10c
[  135.562544]  [<c0104bde>] common_interrupt+0x2e/0x34
[  135.567675]  [<c0116134>] mcount+0x14/0x18
[  135.571858]  [<c019b9ab>] mntput_no_expire+0xd/0x89
[  135.576845]  [<c018cd6a>] path_release+0x2f/0x33
[  135.581645]  [<c01893f4>] vfs_stat_fd+0x4c/0x55
[  135.586276]  [<c01894c9>] vfs_stat+0x25/0x27
[  135.590660]  [<c01894e9>] sys_stat64+0x1e/0x37
[  135.595215]  [<c01041f5>] syscall_call+0x7/0xb
[  135.599752]  =======================
[  135.603365] ---------------------------
[  135.607229] | preempt count: 00010000 ]
[  135.611146] | 0-level deep critical section nesting:
[  135.616202] ----------------------------------------
[  135.621258]
[  135.795930] (       readahead-2442 |#0): new 112512428 us user-latency.
[  135.802591] stopped custom tracer.
[  135.806058] BUG: at kernel/mutex.c:132 __mutex_lock_common()
[  135.811824]  [<c010526c>] show_trace_log_lvl+0x35/0x54
[  135.817035]  [<c0106027>] show_trace+0x2c/0x2e
[  135.821557]  [<c01060d0>] dump_stack+0x29/0x2b
[  135.826078]  [<c035fff2>] __mutex_lock_slowpath+0x64/0x2a7
[  135.831714]  [<c036025e>] mutex_lock+0x29/0x2e
[  135.836277]  [<c014eb96>] lookup_module_symbol_name+0x1a/0xc4
[  135.842121]  [<c014f514>] lookup_symbol_name+0x62/0x69
[  135.847342]  [<c013eb45>] print_name_offset+0x29/0x81
[  135.852478]  [<c013f4c5>] timer_list_show+0x657/0xc55
[  135.857611]  [<c013fae2>] sysrq_timer_list_show+0x1f/0x21
[  135.863128]  [<c027a5fd>] sysrq_handle_show_timers+0xd/0xf
[  135.868744]  [<c027a525>] __handle_sysrq+0x91/0x10c
[  135.873689]  [<c027a79e>] handle_sysrq+0x37/0x39
[  135.878383]  [<c0274b00>] kbd_event+0x335/0x58d
[  135.882991]  [<c02df806>] input_event+0x43e/0x460
[  135.887806]  [<c02e3766>] atkbd_interrupt+0x467/0x58d
[  135.892948]  [<c02dbf37>] serio_interrupt+0x42/0x79
[  135.897918]  [<c02dceed>] i8042_interrupt+0x1ff/0x210
[  135.903053]  [<c0160963>] handle_IRQ_event+0x24/0x54
[  135.908120]  [<c0161b2a>] handle_edge_irq+0xd7/0x11f
[  135.913176]  [<c0106f46>] do_IRQ+0xe2/0x10c
[  135.917449]  [<c0104bde>] common_interrupt+0x2e/0x34
[  135.922502]  [<c0116134>] mcount+0x14/0x18
[  135.926696]  [<c019b9ab>] mntput_no_expire+0xd/0x89
[  135.931657]  [<c018cd6a>] path_release+0x2f/0x33
[  135.936378]  [<c01893f4>] vfs_stat_fd+0x4c/0x55
[  135.941001]  [<c01894c9>] vfs_stat+0x25/0x27
[  135.945376]  [<c01894e9>] sys_stat64+0x1e/0x37
[  135.949931]  [<c01041f5>] syscall_call+0x7/0xb
[  135.954467]  =======================
[  135.958091] ---------------------------
[  135.962015] | preempt count: 00010000 ]
[  135.965905] | 0-level deep critical section nesting:
[  135.970927] ----------------------------------------
[  135.975931]
[  135.977468] <ffffffff>
[  135.979865]
[  135.981396]   .hres_active    : 1
[  135.984757]   .nr_events      : 91663
[  135.988465]   .nohz_mode      : 2
[  135.991820]   .idle_tick      : 112686000000 nsecs
[  135.996651]   .tick_stopped   : 0
[  136.000016]   .idle_jiffies   : 4294779982
[  136.004190]   .idle_calls     : 30792
[  136.007882]   .idle_sleeps    : 3020
[  136.011486]   .idle_entrytime : 112932690671 nsecs
[  136.016335]   .idle_sleeptime : 18007304053 nsecs
[  136.021063]   .last_jiffies   : 4294780229
[  136.025220]   .next_jiffies   : 4294780556
[  136.029352]   .idle_expires   : 112855000000 nsecs
[  136.034167] jiffies: 4294780248
[  136.037331]
[  136.037336] cpu: 1
[  136.040884]  clock 0:
[  136.043182]   .index:      0
[  136.046087]   .resolution: 1 nsecs
[  136.049535]   .get_time:   ktime_get_real
[  136.053739]   .offset:     1179862680097808665 nsecs
[  136.058751] active timers:
[  136.061501]  clock 1:
[  136.063877]   .index:      1
[  136.066781]   .resolution: 1 nsecs
[  136.070229]   .get_time:   ktime_get
[  136.073922]   .offset:     0 nsecs
[  136.077465] active timers:
[  136.080293]  #0: hardirq_stack, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
[  136.089309]  # expires at 113828250000 nsecs [in 872295258 nsecs]
[  136.095654]  #1: hardirq_stack, hrtimer_wakeup, S:01, futex_wait, automount/2294
[  136.103356]  # expires at 113902196736 nsecs [in 946241994 nsecs]
[  136.109485]  #2: hardirq_stack, hrtimer_wakeup, S:01, do_nanosleep, smartd/2640
[  136.117101]  # expires at 1897330250564 nsecs [in 1784374295822 nsecs]
[  136.123723]   .expires_next   : 113871250000 nsecs
[  136.128726]   .exp_prev       : 113875250000 nsecs
[  136.133558]  last expires_next stacktrace:
[  136.137662]   update_cpu_base_expires_next
[  136.141838]   hrtimer_interrupt
[  136.145071]   smp_apic_timer_interrupt
[  136.148935]   apic_timer_interrupt
[  136.152419]   cpu_idle
[  136.154890]   start_secondary
[  136.157967]   <00000000>
[  136.160639]   <ffffffff>
[  136.163267]
[  136.164789]   .hres_active    : 1
[  136.168143]   .nr_events      : 85123
[  136.171981]   .nohz_mode      : 2
[  136.175344]   .idle_tick      : 112682250000 nsecs
[  136.180281]   .tick_stopped   : 0
[  136.183737]   .idle_jiffies   : 4294779978
[  136.187974]   .idle_calls     : 44643
[  136.191708]   .idle_sleeps    : 1384
[  136.195424]   .idle_entrytime : 113942257992 nsecs
[  136.200421]   .idle_sleeptime : 18596870874 nsecs
[  136.205158]   .last_jiffies   : 4294780248
[  136.209316]   .next_jiffies   : 4294780261
[  136.213595]   .idle_expires   : 112778000000 nsecs
[  136.218451] jiffies: 4294780248
[  136.221650]
[  136.223163]
[  136.223168] Tick Device: mode:     1
[  136.228383] Clock Event Device: pit
[  136.231928]  max_delta_ns:   27461866
[  136.235636]  min_delta_ns:   12571
[  136.239172]  mult:           5124677
[  136.242769]  shift:          32
[  136.245965]  mode:           3
[  136.249095]  next_event:     9223372036854775807 nsecs
[  136.254282]  set_next_event: pit_next_event
[  136.258613]  set_mode:       init_pit_timer
[  136.262908]  event_handler:  tick_handle_oneshot_broadcast
[  136.268510] tick_broadcast_mask: 00000000
[  136.272583] tick_broadcast_oneshot_mask: 00000000
[  136.277310]
[  136.278840]
[  136.278845] Tick Device: mode:     1
[  136.284086] Clock Event Device: lapic
[  136.287803]  max_delta_ns:   671411430
[  136.291615]  min_delta_ns:   1200
[  136.294977]  mult:           53661274
[  136.298668]  shift:          32
[  136.301902]  mode:           3
[  136.304971]  next_event:     112952000000 nsecs
[  136.309638]  set_next_event: lapic_next_event
[  136.314177]  set_mode:       lapic_timer_setup
[  136.318680]  event_handler:  hrtimer_interrupt
[  136.323236]
[  136.323241] Tick Device: mode:     1
[  136.328404] Clock Event Device: lapic
[  136.332191]  max_delta_ns:   671411430
[  136.335977]  min_delta_ns:   1200
[  136.339367]  mult:           53661274
[  136.343075]  shift:          32
[  136.346268]  mode:           3
[  136.349375]  next_event:     114098250000 nsecs
[  136.353914]  set_next_event: lapic_next_event
[  136.358357]  set_mode:       lapic_timer_setup
[  136.362973]  event_handler:  hrtimer_interrupt

Regards,
Michal

Signature · 3 lines
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)
#17 lore raw
In reply to Michal Piotrowski
Quoted 3 lines · Michal Piotrowski

Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq works

please try the patch below! I think we have nailed this bug.

Ingo

kernel/sched.c +1 -3
Index: linux/kernel/sched.c
===================================================================
@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)
 	BUG_ON(!in_softirq());
 
 	if (need_resched() && system_state == SYSTEM_RUNNING) {
-		raw_local_irq_disable();
-		_local_bh_enable();
-		raw_local_irq_enable();
+		local_bh_enable();
 		__cond_resched();
 		local_bh_disable();
 		return 1;
#18 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 3 lines · Michal Piotrowski

Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq works

please try the patch below! I think we have nailed this bug.

I picked it from -hrt5 and so far so good.

Regards,
Michal

Signature · 3 lines
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)
#19 lore raw
In reply to Ingo Molnar
Quoted 15 lines · Ingo Molnar
Quoted 3 lines · Michal Piotrowski

Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq works

please try the patch below! I think we have nailed this bug.

Ingo

Index: linux/kernel/sched.c
===================================================================

kernel/sched.c int __sched cond_resched_softirq(void) · line 4218

Quoted 10 lines
kernel/sched.c +1 -3
@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)
 	BUG_ON(!in_softirq());
 
 	if (need_resched() && system_state == SYSTEM_RUNNING) {
-		raw_local_irq_disable();
-		_local_bh_enable();
-		raw_local_irq_enable();
+		local_bh_enable();
 		__cond_resched();
 		local_bh_disable();
 		return 1;

We may have a problem with that:

 BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted)
  [<c042b2ef>] local_bh_enable+0x45/0x92
  [<c06036b7>] cond_resched_softirq+0x2c/0x42
  [<c059d5d0>] release_sock+0x54/0xa3
  [<c05c9428>] tcp_sendmsg+0x91b/0xa0c
  [<c05e1bb9>] inet_sendmsg+0x3b/0x45
  [<c059af34>] sock_aio_write+0xf9/0x105
  [<c0476035>] do_sync_write+0xc7/0x10a
  [<c0437265>] autoremove_wake_function+0x0/0x35
  [<c047688e>] vfs_write+0xbc/0x154
  [<c0476e8c>] sys_write+0x41/0x67
  [<c0404f70>] syscall_call+0x7/0xb

That's:

        WARN_ON_ONCE(irqs_disabled());

And another, tainted but probably valid:

BUG: warning at kernel/softirq.c:138/local_bh_enable() (Tainted: P      )
 [<c042b2ef>] local_bh_enable+0x45/0x92
 [<c06036b7>] cond_resched_softirq+0x2c/0x42
 [<c059d5d0>] release_sock+0x54/0xa3
 [<c05c9428>] tcp_sendmsg+0x91b/0xa0c
 [<c04e8b30>] copy_to_user+0x3c/0x50
 [<c05a1b71>] memcpy_toiovec+0x27/0x4a
 [<c059d58f>] release_sock+0x13/0xa3
 [<c0604ad5>] _spin_unlock_bh+0x5/0xd
 [<c05e1bb9>] inet_sendmsg+0x3b/0x45
 [<c059af34>] sock_aio_write+0xf9/0x105
 [<c0475f31>] do_sync_readv_writev+0xc1/0xfe
 [<c0437265>] autoremove_wake_function+0x0/0x35
 [<c04e88e0>] copy_from_user+0x3a/0x66
 [<c0475dec>] rw_copy_check_uvector+0x5c/0xb0
 [<c047667a>] do_readv_writev+0xbc/0x187
 [<c059ae3b>] sock_aio_write+0x0/0x105
 [<c0476782>] vfs_writev+0x3d/0x48
 [<c0476bee>] sys_writev+0x41/0x95
 [<c0404f70>] syscall_call+0x7/0xb

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=240982

#20 lore raw
Quoted 2 lines · Chuck Ebbert

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=240982

Another; these started to appear after the below patch was merged:

Quoted 7 lines

Index: linux/kernel/sched.c

Quoted 6 lines

===================================================================

kernel/sched.c +0 -0
@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)
 	BUG_ON(!in_softirq());
 
Quoted 8 lines

kernel/sched.c int __sched cond_resched_softirq(void) · line 4218

Quoted 8 lines
kernel/sched.c +1 -3
 	if (need_resched() && system_state == SYSTEM_RUNNING) {
-		raw_local_irq_disable();
-		_local_bh_enable();
-		raw_local_irq_enable();
+		local_bh_enable();
 		__cond_resched();
 		local_bh_disable();
 		return 1;
May 23 19:26:26 localhost kernel: BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted)
May 23 19:26:26 localhost kernel:  [<c042b2ef>] local_bh_enable+0x45/0x92
May 23 19:26:26 localhost kernel:  [<c06036b7>] cond_resched_softirq+0x2c/0x42
May 23 19:26:26 localhost kernel:  [<c059d5d0>] release_sock+0x54/0xa3
May 23 19:26:26 localhost kernel:  [<c04373af>] prepare_to_wait+0x24/0x3f
May 23 19:26:26 localhost kernel:  [<c05e267f>] inet_stream_connect+0x116/0x1ff
May 23 19:26:26 localhost kernel:  [<c0437265>] autoremove_wake_function+0x0/0x35
May 23 19:26:26 localhost kernel:  [<c059c339>] sys_connect+0x82/0xad
May 23 19:26:26 localhost kernel:  [<c059d58f>] release_sock+0x13/0xa3
May 23 19:26:26 localhost kernel:  [<c0604ad5>] _spin_unlock_bh+0x5/0xd
May 23 19:26:26 localhost kernel:  [<c059e714>] sock_setsockopt+0x4a8/0x4b2
May 23 19:26:26 localhost kernel:  [<c059b6b6>] sock_attach_fd+0x70/0xd2
May 23 19:26:26 localhost kernel:  [<c04774a0>] get_empty_filp+0xfc/0x170
May 23 19:26:26 localhost kernel:  [<c059b54f>] sys_setsockopt+0x9b/0xa7
May 23 19:26:26 localhost kernel:  [<c059cb83>] sys_socketcall+0xac/0x261
May 23 19:26:26 localhost kernel:  [<c0404f70>] syscall_call+0x7/0xb
#21 lore raw
Quoted 31 lines · On Thursday 24 May 2007 00:08:40 Chuck Ebbert wrote:
Quoted 1 line · Chuck Ebbert

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=240982

Another; these started to appear after the below patch was merged:

Quoted 8 lines

Index: linux/kernel/sched.c

Quoted 6 lines

===================================================================

kernel/sched.c +0 -0
@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)
 	BUG_ON(!in_softirq());
 
Quoted 8 lines

kernel/sched.c int __sched cond_resched_softirq(void) · line 4218

Quoted 8 lines
kernel/sched.c +1 -3
 	if (need_resched() && system_state == SYSTEM_RUNNING) {
-		raw_local_irq_disable();
-		_local_bh_enable();
-		raw_local_irq_enable();
+		local_bh_enable();
 		__cond_resched();
 		local_bh_disable();
 		return 1;
May 23 19:26:26 localhost kernel: BUG: warning at
kernel/softirq.c:138/local_bh_enable() (Not tainted) May 23 19:26:26
localhost kernel:  [<c042b2ef>] local_bh_enable+0x45/0x92 May 23 19:26:26
localhost kernel:  [<c06036b7>] cond_resched_softirq+0x2c/0x42 May 23
19:26:26 localhost kernel:  [<c059d5d0>] release_sock+0x54/0xa3 May 23
19:26:26 localhost kernel:  [<c04373af>] prepare_to_wait+0x24/0x3f May 23
19:26:26 localhost kernel:  [<c05e267f>] inet_stream_connect+0x116/0x1ff
May 23 19:26:26 localhost kernel:  [<c0437265>]
autoremove_wake_function+0x0/0x35 May 23 19:26:26 localhost kernel: 
[<c059c339>] sys_connect+0x82/0xad May 23 19:26:26 localhost kernel: 
Quoted 8 lines
[<c059d58f>] release_sock+0x13/0xa3 May 23 19:26:26 localhost kernel: 
[<c0604ad5>] _spin_unlock_bh+0x5/0xd May 23 19:26:26 localhost kernel: 
[<c059e714>] sock_setsockopt+0x4a8/0x4b2 May 23 19:26:26 localhost kernel: 
[<c059b6b6>] sock_attach_fd+0x70/0xd2 May 23 19:26:26 localhost kernel: 
[<c04774a0>] get_empty_filp+0xfc/0x170 May 23 19:26:26 localhost kernel: 
[<c059b54f>] sys_setsockopt+0x9b/0xa7 May 23 19:26:26 localhost kernel: 
[<c059cb83>] sys_socketcall+0xac/0x261 May 23 19:26:26 localhost kernel: 
[<c0404f70>] syscall_call+0x7/0xb

strange, while applying the concerned patch first time I was hand editing __ kernel/sched.c __ and stupidly typed _local_bh_enable() instead of local_bh_enable() and when did a reboot, as soon as I got inside X I was welcomed with following message in system log.

[  152.692609] BUG: at kernel/softirq.c:122 _local_bh_enable()
[  152.692637]  [<b040624d>] show_trace_log_lvl+0x1a/0x2f
[  152.692658]  [<b0406801>] show_trace+0x12/0x14
[  152.692668]  [<b0406885>] dump_stack+0x16/0x18
[  152.692678]  [<b0428d66>] _local_bh_enable+0x8b/0xc3
[  152.692688]  [<b059e497>] cond_resched_softirq+0x2b/0x40
[  152.692700]  [<b05797b6>] established_get_first+0x19/0xad
[  152.692712]  [<b057a831>] tcp_seq_next+0x76/0x8c
[  152.692722]  [<b04875ef>] seq_read+0x17b/0x264
[  152.692733]  [<b047102a>] vfs_read+0xad/0x161
[  152.692745]  [<b04714b6>] sys_read+0x3d/0x61
[  152.692755]  [<b04050d4>] syscall_call+0x7/0xb
[  152.692765]  =======================
[  152.692770] BUG: at kernel/lockdep.c:1937 trace_softirqs_on()
[  152.692777]  [<b040624d>] show_trace_log_lvl+0x1a/0x2f
[  152.692789]  [<b0406801>] show_trace+0x12/0x14
[  152.692800]  [<b0406885>] dump_stack+0x16/0x18
[  152.692810]  [<b043f5a5>] trace_softirqs_on+0x5f/0xa5
[  152.692822]  [<b0428d8e>] _local_bh_enable+0xb3/0xc3
[  152.692831]  [<b059e497>] cond_resched_softirq+0x2b/0x40
[  152.692842]  [<b05797b6>] established_get_first+0x19/0xad
[  152.692852]  [<b057a831>] tcp_seq_next+0x76/0x8c
[  152.692862]  [<b04875ef>] seq_read+0x17b/0x264
[  152.692870]  [<b047102a>] vfs_read+0xad/0x161
[  152.692879]  [<b04714b6>] sys_read+0x3d/0x61
[  152.692889]  [<b04050d4>] syscall_call+0x7/0xb
[  152.692899]  =======================
[  159.257890] NOHZ: local_softirq_pending 22
[  159.266009] NOHZ: local_softirq_pending 22
[  159.273965] NOHZ: local_softirq_pending 22
[  159.281884] NOHZ: local_softirq_pending 22
[  160.712828] NOHZ: local_softirq_pending 22
[  162.609377] NOHZ: local_softirq_pending 22
[  162.609804] NOHZ: local_softirq_pending 22
[  162.610054] NOHZ: local_softirq_pending 22
[  162.610279] NOHZ: local_softirq_pending 22
[  162.610502] NOHZ: local_softirq_pending 22

After realzing my mistake, I changed it to local_bh_enable() as was in patch and since then not a single BUG or local_softirq_pending message occurs in system log, maybe my system waiting for that condition to happen :).

Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of
those who have achieved perfection, hardly one knows Me in truth.
				-- Gita Sutra Of Mysticism
#22 lore raw
In reply to Chuck Ebbert
Quoted 31 lines · On Thursday 24 May 2007 00:08:40 Chuck Ebbert wrote:
Quoted 1 line · Chuck Ebbert

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=240982

Another; these started to appear after the below patch was merged:

Quoted 8 lines

Index: linux/kernel/sched.c

Quoted 6 lines

===================================================================

kernel/sched.c +0 -0
@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)
 	BUG_ON(!in_softirq());
 
Quoted 8 lines

kernel/sched.c int __sched cond_resched_softirq(void) · line 4218

Quoted 8 lines
kernel/sched.c +1 -3
 	if (need_resched() && system_state == SYSTEM_RUNNING) {
-		raw_local_irq_disable();
-		_local_bh_enable();
-		raw_local_irq_enable();
+		local_bh_enable();
 		__cond_resched();
 		local_bh_disable();
 		return 1;
May 23 19:26:26 localhost kernel: BUG: warning at
kernel/softirq.c:138/local_bh_enable() (Not tainted) May 23 19:26:26
localhost kernel:  [<c042b2ef>] local_bh_enable+0x45/0x92 May 23 19:26:26
localhost kernel:  [<c06036b7>] cond_resched_softirq+0x2c/0x42 May 23
19:26:26 localhost kernel:  [<c059d5d0>] release_sock+0x54/0xa3 May 23
19:26:26 localhost kernel:  [<c04373af>] prepare_to_wait+0x24/0x3f May 23
19:26:26 localhost kernel:  [<c05e267f>] inet_stream_connect+0x116/0x1ff
May 23 19:26:26 localhost kernel:  [<c0437265>]
autoremove_wake_function+0x0/0x35 May 23 19:26:26 localhost kernel: 
[<c059c339>] sys_connect+0x82/0xad May 23 19:26:26 localhost kernel: 
Quoted 8 lines
[<c059d58f>] release_sock+0x13/0xa3 May 23 19:26:26 localhost kernel: 
[<c0604ad5>] _spin_unlock_bh+0x5/0xd May 23 19:26:26 localhost kernel: 
[<c059e714>] sock_setsockopt+0x4a8/0x4b2 May 23 19:26:26 localhost kernel: 
[<c059b6b6>] sock_attach_fd+0x70/0xd2 May 23 19:26:26 localhost kernel: 
[<c04774a0>] get_empty_filp+0xfc/0x170 May 23 19:26:26 localhost kernel: 
[<c059b54f>] sys_setsockopt+0x9b/0xa7 May 23 19:26:26 localhost kernel: 
[<c059cb83>] sys_socketcall+0xac/0x261 May 23 19:26:26 localhost kernel: 
[<c0404f70>] syscall_call+0x7/0xb

strange, while applying the concerned patch first time I was hand editing __ kernel/sched.c __ and stupidly typed _local_bh_enable() instead of local_bh_enable() and when did a reboot, as soon as I got inside X I was welcomed with following message in system log.

[  152.692609] BUG: at kernel/softirq.c:122 _local_bh_enable()
[  152.692637]  [<b040624d>] show_trace_log_lvl+0x1a/0x2f
[  152.692658]  [<b0406801>] show_trace+0x12/0x14
[  152.692668]  [<b0406885>] dump_stack+0x16/0x18
[  152.692678]  [<b0428d66>] _local_bh_enable+0x8b/0xc3
[  152.692688]  [<b059e497>] cond_resched_softirq+0x2b/0x40
[  152.692700]  [<b05797b6>] established_get_first+0x19/0xad
[  152.692712]  [<b057a831>] tcp_seq_next+0x76/0x8c
[  152.692722]  [<b04875ef>] seq_read+0x17b/0x264
[  152.692733]  [<b047102a>] vfs_read+0xad/0x161
[  152.692745]  [<b04714b6>] sys_read+0x3d/0x61
[  152.692755]  [<b04050d4>] syscall_call+0x7/0xb
[  152.692765]  =======================
[  152.692770] BUG: at kernel/lockdep.c:1937 trace_softirqs_on()
[  152.692777]  [<b040624d>] show_trace_log_lvl+0x1a/0x2f
[  152.692789]  [<b0406801>] show_trace+0x12/0x14
[  152.692800]  [<b0406885>] dump_stack+0x16/0x18
[  152.692810]  [<b043f5a5>] trace_softirqs_on+0x5f/0xa5
[  152.692822]  [<b0428d8e>] _local_bh_enable+0xb3/0xc3
[  152.692831]  [<b059e497>] cond_resched_softirq+0x2b/0x40
[  152.692842]  [<b05797b6>] established_get_first+0x19/0xad
[  152.692852]  [<b057a831>] tcp_seq_next+0x76/0x8c
[  152.692862]  [<b04875ef>] seq_read+0x17b/0x264
[  152.692870]  [<b047102a>] vfs_read+0xad/0x161
[  152.692879]  [<b04714b6>] sys_read+0x3d/0x61
[  152.692889]  [<b04050d4>] syscall_call+0x7/0xb
[  152.692899]  =======================
[  159.257890] NOHZ: local_softirq_pending 22
[  159.266009] NOHZ: local_softirq_pending 22
[  159.273965] NOHZ: local_softirq_pending 22
[  159.281884] NOHZ: local_softirq_pending 22
[  160.712828] NOHZ: local_softirq_pending 22
[  162.609377] NOHZ: local_softirq_pending 22
[  162.609804] NOHZ: local_softirq_pending 22
[  162.610054] NOHZ: local_softirq_pending 22
[  162.610279] NOHZ: local_softirq_pending 22
[  162.610502] NOHZ: local_softirq_pending 22

After realzing my mistake, I changed it to local_bh_enable() as was in patch and since then not a single BUG or local_softirq_pending message occurs in system log, maybe my system waiting for that condition to happen :).

Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of
those who have achieved perfection, hardly one knows Me in truth.
				-- Gita Sutra Of Mysticism
#23 lore raw
In reply to Chuck Ebbert
Quoted 11 lines · Chuck Ebbert

Review

Quoted 8 lines
(quoted diff) +1 -3
 	if (need_resched() && system_state == SYSTEM_RUNNING) {
-		raw_local_irq_disable();
-		_local_bh_enable();
-		raw_local_irq_enable();
+		local_bh_enable();
 		__cond_resched();
 		local_bh_disable();
 		return 1;

We may have a problem with that:

Quoted 12 lines
 BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted)
  [<c042b2ef>] local_bh_enable+0x45/0x92
  [<c06036b7>] cond_resched_softirq+0x2c/0x42
  [<c059d5d0>] release_sock+0x54/0xa3
  [<c05c9428>] tcp_sendmsg+0x91b/0xa0c
  [<c05e1bb9>] inet_sendmsg+0x3b/0x45
  [<c059af34>] sock_aio_write+0xf9/0x105
  [<c0476035>] do_sync_write+0xc7/0x10a
  [<c0437265>] autoremove_wake_function+0x0/0x35
  [<c047688e>] vfs_write+0xbc/0x154
  [<c0476e8c>] sys_write+0x41/0x67
  [<c0404f70>] syscall_call+0x7/0xb

hm, this place really shouldnt call cond_resched_softirq() with hardirqs disabled.

perhaps a buggy ->sk_backlog_rcv() handler disabled interrupts without restoring them?

could you enable CONFIG_PROVE_LOCKING and apply the patch below - which location is printed as having last disabled hardirqs?

Ingo

--------------------->
Subject: [patch] softirqs: print out irq-trace events
From: Ingo Molnar <[email protected]>

some code is fiddling with softirqs but hardirqs are disabled, so try to figure out who disabled hardirqs.

---
 kernel/softirq.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
kernel/softirq.c +9 -1
Index: linux/kernel/softirq.c
===================================================================
@@ -135,7 +135,15 @@ void local_bh_enable(void)
 
 	WARN_ON_ONCE(in_irq());
 #endif
-	WARN_ON_ONCE(irqs_disabled());
+	if (irqs_disabled()) {
+		static int once = 1;
+
+		if (once) {
+			once = 0;
+			print_irqtrace_events(current);
+			WARN_ON(1);
+		}
+	}
 
 #ifdef CONFIG_TRACE_IRQFLAGS
 	local_irq_save(flags);
#24 lore raw
In reply to Thomas Gleixner
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Anant Nitya
Quoted 5 lines

No idea. I uploaded a debug patch against 2.6.22-rc1 to

http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch

Can you give it a try and report the output ?

Hi
Here it goes

[  159.646196] NOHZ softirq pending 22 on CPU 0
[  159.646207] .... task state: 1 00000000
Quoted 5 lines

1 == TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.

I work out a more complex debug patch and pester you to test once I'm done.

I've also tons of 'NOHZ: local_softirq_pending 08' that disappear with nohz=off. But I'm still running 2.6.21. Are there any patches that should fix this?
Machine is a Lenovo T60p:

i686 Intel(R) Core(TM)2 CPU T7600 @ 2.33GHz GenuineIntel GNU/Linux

Besides that I get a lot of clock skews on 'make headers_check', but these are unrelated to nohz:

CHECK include/asm/dasd.h
make[2]: warning: Clock skew detected. Your build may be incomplete. make[2]: Warning: File `/dev/null' has modification time 5.1e+03 s in the future

#25 lore raw
Quoted 9 lines · Heiko Carstens
Quoted 2 lines

I work out a more complex debug patch and pester you to test once I'm done.

I've also tons of 'NOHZ: local_softirq_pending 08' that disappear with nohz=off. But I'm still running 2.6.21. Are there any patches that should fix this?
Machine is a Lenovo T60p:

i686 Intel(R) Core(TM)2 CPU T7600 @ 2.33GHz GenuineIntel GNU/Linux

Hmm, that's a different problem than the 0x22 which shows up on hyperthreading enabled P4 systems. Are you using plip ?

Quoted 6 lines

Besides that I get a lot of clock skews on 'make headers_check', but these are unrelated to nohz:

CHECK include/asm/dasd.h
make[2]: warning: Clock skew detected. Your build may be incomplete. make[2]: Warning: File `/dev/null' has modification time 5.1e+03 s in the future

Strange.

tglx

#26 lore raw
Quoted 9 lines · Thomas Gleixner
Quoted 8 lines · Heiko Carstens
Quoted 2 lines

I work out a more complex debug patch and pester you to test once I'm done.

I've also tons of 'NOHZ: local_softirq_pending 08' that disappear with nohz=off. But I'm still running 2.6.21. Are there any patches that should fix this?
Machine is a Lenovo T60p:

Quoted 4 lines
Quoted 1 line

i686 Intel(R) Core(TM)2 CPU T7600 @ 2.33GHz GenuineIntel GNU/Linux

Hmm, that's a different problem than the 0x22 which shows up on hyperthreading enabled P4 systems. Are you using plip ?

No, after all it turned out that is caused by an IBM internal module. Just ignore me, sorry for the noise :)

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 1179689389.6570.1.camel@chaos

A message that is not in the archive (sent off-list or not yet received): [email protected]

#30 lore raw
Quoted 12 lines · Heiko Carstens
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Heiko Carstens
Quoted 5 lines
Quoted 2 lines

Hmm, that's a different problem than the 0x22 which shows up on hyperthreading enabled P4 systems. Are you using plip ?

No, after all it turned out that is caused by an IBM internal module. Just ignore me, sorry for the noise :)

Just in case you're interested: I just looked into it and it turned out that this module was calling netif_rx() from process context, just like plip. Changing it to netif_rx_ni() fixes it.

Quoted 5 lines
Quoted 2 lines

Thanks, that's a good pointer. You should have cc'ed Dave Miller and the guy who found this plip thingy. Care to resend ?

No problem. Here we go, even though I doubt that netif_rx_ni() is what people want for plip. Dunno...

netif_rx_ni calls softirq synchronously, am I right? That would probably hurt plip. I can try it if someone will be interested in it, but I doubt it is good solution for any network interfaces except tun.

BTW. does in_interrupt() have high cost on some high-performance architectures? Would it be possible to just add if (!in_interrupt()) wakeup_softirqd(); to netif_rx()?

Mikulas