local_softirq_pending storm
Subsystems (3)
Hi,
Ever since I upgrade to 2.6.21/.1, system log is filled with following messages if I enable CONFIG_NO_HZ=y, going through archives it seems ingo sometime back posted some patch and now it is upstream, but its not helping here. If I disable NOHZ by kernel command line nohz=off this problem disappears. This system is P4/2.40GHz/HT with SMP/SMT on in kernel config. One more thing that I noticed is this problem only arises while using X or network otherwise plain command line with no network access don't trigger this with nohz=on.
Please ask if more information is required regarding this setup.
dumdum@hahakaar [~]$ >> grep NOHZ /var/log/messages
May 8 03:38:14 rudra kernel: [ 419.271195] NOHZ: local_softirq_pending 02 May 8 03:38:14 rudra kernel: [ 419.271663] NOHZ: local_softirq_pending 02 May 8 03:38:14 rudra kernel: [ 419.343948] NOHZ: local_softirq_pending 22 May 8 03:38:14 rudra kernel: [ 419.344236] NOHZ: local_softirq_pending 22 May 8 03:38:14 rudra kernel: [ 419.344397] NOHZ: local_softirq_pending 22 May 8 03:38:14 rudra kernel: [ 419.344545] NOHZ: local_softirq_pending 22 May 8 03:38:14 rudra kernel: [ 419.344691] NOHZ: local_softirq_pending 22 May 8 03:38:14 rudra kernel: [ 419.344842] NOHZ: local_softirq_pending 22 May 8 03:38:14 rudra kernel: [ 419.344991] NOHZ: local_softirq_pending 22 May 8 03:38:14 rudra kernel: [ 419.345137] NOHZ: local_softirq_pending 22 May 8 03:38:18 rudra kernel: [ 423.065780] NOHZ: local_softirq_pending 22 May 8 03:38:18 rudra kernel: [ 423.066206] NOHZ: local_softirq_pending 22 May 8 03:38:18 rudra kernel: [ 423.066509] NOHZ: local_softirq_pending 22 May 8 03:38:19 rudra kernel: [ 424.006549] NOHZ: local_softirq_pending 22 May 8 03:38:19 rudra kernel: [ 424.006983] NOHZ: local_softirq_pending 22 May 8 03:38:19 rudra kernel: [ 424.007239] NOHZ: local_softirq_pending 22 May 8 03:38:19 rudra kernel: [ 424.007473] NOHZ: local_softirq_pending 22 May 8 03:38:19 rudra kernel: [ 424.007706] NOHZ: local_softirq_pending 22 May 8 03:38:19 rudra kernel: [ 424.007941] NOHZ: local_softirq_pending 22 May 8 03:38:22 rudra kernel: [ 426.862456] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.331619] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.331991] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.332192] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.332378] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.332553] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.332740] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.332914] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.333097] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.333271] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.333443] NOHZ: local_softirq_pending 22 May 8 03:38:23 rudra kernel: [ 428.333619] NOHZ: local_softirq_pending 22 May 8 03:38:24 rudra kernel: [ 429.261574] NOHZ: local_softirq_pending 22 May 8 03:38:24 rudra kernel: [ 429.262024] NOHZ: local_softirq_pending 22 May 8 03:38:24 rudra kernel: [ 429.262339] NOHZ: local_softirq_pending 22 May 8 03:38:24 rudra kernel: [ 429.262610] NOHZ: local_softirq_pending 22 May 8 03:38:24 rudra kernel: [ 429.262847] NOHZ: local_softirq_pending 22 May 8 03:38:24 rudra kernel: [ 429.263081] NOHZ: local_softirq_pending 22 May 8 03:38:35 rudra kernel: [ 440.182998] NOHZ: local_softirq_pending 22 May 8 03:38:35 rudra kernel: [ 440.183408] NOHZ: local_softirq_pending 22 May 8 03:38:35 rudra kernel: [ 440.183661] NOHZ: local_softirq_pending 22 May 8 03:38:43 rudra kernel: [ 448.272087] NOHZ: local_softirq_pending 22 May 8 03:38:43 rudra kernel: [ 448.272529] NOHZ: local_softirq_pending 22 May 8 03:38:44 rudra kernel: [ 449.223360] NOHZ: local_softirq_pending 22 May 8 03:38:44 rudra kernel: [ 449.223887] NOHZ: local_softirq_pending 22 May 8 03:38:44 rudra kernel: [ 449.224570] NOHZ: local_softirq_pending 22 May 8 03:38:44 rudra kernel: [ 449.225066] NOHZ: local_softirq_pending 22 May 8 03:38:44 rudra kernel: [ 449.232989] NOHZ: local_softirq_pending 22 May 8 03:38:47 rudra kernel: [ 452.178583] NOHZ: local_softirq_pending a2 May 8 03:38:47 rudra kernel: [ 452.179017] NOHZ: local_softirq_pending a2 May 8 03:38:47 rudra kernel: [ 452.179257] NOHZ: local_softirq_pending a2 May 8 03:38:51 rudra kernel: [ 455.957968] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.958462] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.958741] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.958984] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.959292] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.959540] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.959774] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.960005] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.972368] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.972681] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.972884] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.973069] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.973251] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.973433] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.973631] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.973832] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.974018] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.974234] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.974422] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.974610] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.974791] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.974981] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.975194] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.975390] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.975570] NOHZ: local_softirq_pending 10 May 8 03:38:51 rudra kernel: [ 455.976091] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.976376] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.976622] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.976862] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.977099] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.977337] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.977577] NOHZ: local_softirq_pending 22 May 8 03:38:51 rudra kernel: [ 455.977815] NOHZ: local_softirq_pending 22 May 8 03:38:52 rudra kernel: [ 456.848267] NOHZ: local_softirq_pending 22 May 8 03:38:52 rudra kernel: [ 456.848726] NOHZ: local_softirq_pending 22 May 8 03:38:52 rudra kernel: [ 456.848982] NOHZ: local_softirq_pending 22 May 8 03:38:52 rudra kernel: [ 456.849223] NOHZ: local_softirq_pending 22 May 8 03:38:52 rudra kernel: [ 456.849459] NOHZ: local_softirq_pending 22 May 8 03:38:54 rudra kernel: [ 459.279942] NOHZ: local_softirq_pending 02 May 8 03:38:54 rudra kernel: [ 459.280267] NOHZ: local_softirq_pending 02 May 8 03:38:54 rudra kernel: [ 459.280477] NOHZ: local_softirq_pending 02 May 8 03:38:54 rudra kernel: [ 459.280667] NOHZ: local_softirq_pending 02 May 8 03:38:54 rudra kernel: [ 459.280865] NOHZ: local_softirq_pending 02 May 8 03:38:59 rudra kernel: [ 464.046556] NOHZ: local_softirq_pending 22 May 8 03:38:59 rudra kernel: [ 464.046999] NOHZ: local_softirq_pending 22 May 8 03:38:59 rudra kernel: [ 464.047315] NOHZ: local_softirq_pending 22 May 8 03:38:59 rudra kernel: [ 464.047554] NOHZ: local_softirq_pending 22 May 8 03:38:59 rudra kernel: [ 464.047790] NOHZ: local_softirq_pending 22 May 8 03:39:08 rudra kernel: [ 473.019484] NOHZ: local_softirq_pending a2 May 8 03:39:08 rudra kernel: [ 473.019954] NOHZ: local_softirq_pending a2 May 8 03:39:08 rudra kernel: [ 473.020215] NOHZ: local_softirq_pending a2 May 8 03:39:08 rudra kernel: [ 473.020455] NOHZ: local_softirq_pending a2 May 8 03:39:08 rudra kernel: [ 473.020692] NOHZ: local_softirq_pending a2 May 8 03:39:08 rudra kernel: [ 473.020925] NOHZ: local_softirq_pending a2 May 8 03:39:11 rudra kernel: [ 475.876271] NOHZ: local_softirq_pending 22 May 8 03:39:11 rudra kernel: [ 475.876697] NOHZ: local_softirq_pending 22 May 8 03:39:11 rudra kernel: [ 475.876947] NOHZ: local_softirq_pending 22 May 8 03:39:11 rudra kernel: [ 475.877183] NOHZ: local_softirq_pending 22 May 8 03:39:11 rudra kernel: [ 475.877490] NOHZ: local_softirq_pending 22 May 8 03:39:11 rudra kernel: [ 475.877736] NOHZ: local_softirq_pending 22 May 8 03:39:11 rudra kernel: [ 475.877964] NOHZ: local_softirq_pending 22 May 8 03:39:31 rudra kernel: [ 495.878914] NOHZ: local_softirq_pending 22 May 8 03:39:31 rudra kernel: [ 495.879262] NOHZ: local_softirq_pending 22 May 8 03:39:31 rudra kernel: [ 495.879473] NOHZ: local_softirq_pending 22 May 8 03:39:31 rudra kernel: [ 495.879674] NOHZ: local_softirq_pending 22 May 8 03:39:31 rudra kernel: [ 495.879849] NOHZ: local_softirq_pending 22 May 8 03:39:31 rudra kernel: [ 495.880031] NOHZ: local_softirq_pending 22 May 8 03:39:31 rudra kernel: [ 495.880206] NOHZ: local_softirq_pending 22 May 8 03:39:53 rudra kernel: [ 517.755212] NOHZ: local_softirq_pending 22 May 8 03:39:53 rudra kernel: [ 517.755863] NOHZ: local_softirq_pending 22 May 8 03:39:53 rudra kernel: [ 517.756181] NOHZ: local_softirq_pending 22 May 8 03:39:53 rudra kernel: [ 517.756434] NOHZ: local_softirq_pending 22 May 8 03:39:53 rudra kernel: [ 517.756667] NOHZ: local_softirq_pending 22 May 8 03:39:53 rudra kernel: [ 517.756900] NOHZ: local_softirq_pending 22 May 8 03:39:54 rudra kernel: [ 518.718220] NOHZ: local_softirq_pending 22 May 8 03:39:54 rudra kernel: [ 518.718540] NOHZ: local_softirq_pending 22 May 8 03:39:54 rudra kernel: [ 518.718756] NOHZ: local_softirq_pending 22 May 8 03:39:54 rudra kernel: [ 518.718944] NOHZ: local_softirq_pending 22 May 8 03:39:54 rudra kernel: [ 518.719121] NOHZ: local_softirq_pending 22 May 8 03:39:54 rudra kernel: [ 518.719298] NOHZ: local_softirq_pending 22 May 8 03:39:54 rudra kernel: [ 518.719484] NOHZ: local_softirq_pending 22 May 8 03:39:54 rudra kernel: [ 518.719672] NOHZ: local_softirq_pending 22 May 8 03:40:02 rudra kernel: [ 526.814840] NOHZ: local_softirq_pending 22 May 8 03:40:02 rudra kernel: [ 526.815354] NOHZ: local_softirq_pending 22 May 8 03:40:02 rudra kernel: [ 526.815703] NOHZ: local_softirq_pending 22 May 8 03:40:02 rudra kernel: [ 526.816015] NOHZ: local_softirq_pending 22 May 8 03:40:02 rudra kernel: [ 526.816255] NOHZ: local_softirq_pending 22 May 8 03:40:11 rudra kernel: [ 535.847890] NOHZ: local_softirq_pending a2 May 8 03:40:11 rudra kernel: [ 535.848360] NOHZ: local_softirq_pending a2 May 8 03:40:11 rudra kernel: [ 535.848618] NOHZ: local_softirq_pending a2 May 8 03:40:11 rudra kernel: [ 535.848857] NOHZ: local_softirq_pending a2 May 8 03:40:11 rudra kernel: [ 535.849095] NOHZ: local_softirq_pending a2 May 8 03:40:12 rudra kernel: [ 536.788182] NOHZ: local_softirq_pending 22 May 8 03:40:12 rudra kernel: [ 536.788630] NOHZ: local_softirq_pending 22 May 8 03:40:12 rudra kernel: [ 536.788939] NOHZ: local_softirq_pending 22 May 8 03:40:12 rudra kernel: [ 536.789240] NOHZ: local_softirq_pending 22 May 8 03:40:12 rudra kernel: [ 536.789484] NOHZ: local_softirq_pending 22 May 8 03:40:12 rudra kernel: [ 536.789718] NOHZ: local_softirq_pending 22 May 8 03:40:16 rudra kernel: [ 541.078474] NOHZ: local_softirq_pending a2 May 8 03:40:16 rudra kernel: [ 541.078933] NOHZ: local_softirq_pending a2 May 8 03:40:16 rudra kernel: [ 541.079186] NOHZ: local_softirq_pending a2 May 8 03:40:16 rudra kernel: [ 541.079425] NOHZ: local_softirq_pending a2 May 8 03:40:16 rudra kernel: [ 541.079657] NOHZ: local_softirq_pending a2 May 8 03:40:16 rudra kernel: [ 541.079893] NOHZ: local_softirq_pending a2 May 8 03:40:18 rudra kernel: [ 542.974969] NOHZ: local_softirq_pending 22 May 8 03:40:18 rudra kernel: [ 542.975419] NOHZ: local_softirq_pending 22 May 8 03:40:18 rudra kernel: [ 542.975754] NOHZ: local_softirq_pending 22 May 8 03:40:18 rudra kernel: [ 542.976018] NOHZ: local_softirq_pending 22 May 8 03:40:18 rudra kernel: [ 542.976253] NOHZ: local_softirq_pending 22 May 8 03:40:18 rudra kernel: [ 542.976484] NOHZ: local_softirq_pending 22 May 8 03:40:24 rudra kernel: [ 548.687200] NOHZ: local_softirq_pending a2 May 8 03:40:24 rudra kernel: [ 548.687691] NOHZ: local_softirq_pending a2 May 8 03:40:28 rudra kernel: [ 552.977399] NOHZ: local_softirq_pending a2 May 8 03:40:28 rudra kernel: [ 552.977730] NOHZ: local_softirq_pending a2 May 8 03:40:28 rudra kernel: [ 552.977946] NOHZ: local_softirq_pending a2 May 8 03:40:28 rudra kernel: [ 552.978135] NOHZ: local_softirq_pending a2 May 8 03:40:28 rudra kernel: [ 552.978318] NOHZ: local_softirq_pending a2 May 8 03:40:28 rudra kernel: [ 552.978499] NOHZ: local_softirq_pending a2 May 8 03:40:28 rudra kernel: [ 552.978708] NOHZ: local_softirq_pending a2 May 8 03:40:33 rudra kernel: [ 557.728762] NOHZ: local_softirq_pending 22 May 8 03:40:33 rudra kernel: [ 557.729230] NOHZ: local_softirq_pending 22 May 8 03:40:33 rudra kernel: [ 557.729539] NOHZ: local_softirq_pending 22 May 8 03:40:33 rudra kernel: [ 557.729821] NOHZ: local_softirq_pending 22 May 8 03:40:33 rudra kernel: [ 557.730060] NOHZ: local_softirq_pending 22 May 8 03:40:33 rudra kernel: [ 557.730297] NOHZ: local_softirq_pending 22 May 8 03:40:36 rudra kernel: [ 561.065192] NOHZ: local_softirq_pending e2 May 8 03:40:36 rudra kernel: [ 561.065539] NOHZ: local_softirq_pending e2 May 8 03:40:36 rudra kernel: [ 561.065750] NOHZ: local_softirq_pending e2 May 8 03:40:36 rudra kernel: [ 561.065942] NOHZ: local_softirq_pending e2 May 8 03:40:36 rudra kernel: [ 561.066139] NOHZ: local_softirq_pending e2 May 8 03:40:36 rudra kernel: [ 561.066339] NOHZ: local_softirq_pending e2 May 8 03:40:36 rudra kernel: [ 561.066515] NOHZ: local_softirq_pending e2 May 8 03:40:42 rudra kernel: [ 566.772651] NOHZ: local_softirq_pending 22 May 8 03:40:42 rudra kernel: [ 566.773108] NOHZ: local_softirq_pending 22 May 8 03:40:42 rudra kernel: [ 566.773314] NOHZ: local_softirq_pending 22 May 8 03:40:42 rudra kernel: [ 566.773359] NOHZ: local_softirq_pending 22 May 8 03:40:42 rudra kernel: [ 566.787249] NOHZ: local_softirq_pending 62 May 8 03:40:42 rudra kernel: [ 566.787536] NOHZ: local_softirq_pending 62 May 8 03:40:51 rudra kernel: [ 575.816602] NOHZ: local_softirq_pending 22 May 8 03:40:51 rudra kernel: [ 575.817065] NOHZ: local_softirq_pending 22 May 8 03:41:03 rudra kernel: [ 587.715470] NOHZ: local_softirq_pending 22 May 8 03:41:03 rudra kernel: [ 587.715920] NOHZ: local_softirq_pending 22 May 8 03:41:03 rudra kernel: [ 587.716259] NOHZ: local_softirq_pending 22 May 8 03:41:03 rudra kernel: [ 587.716528] NOHZ: local_softirq_pending 22 May 8 03:41:03 rudra kernel: [ 587.716764] NOHZ: local_softirq_pending 22 May 8 03:41:03 rudra kernel: [ 587.717001] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.805445] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.805945] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.806196] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.806430] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.806659] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.806888] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.807116] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.807341] NOHZ: local_softirq_pending 22 May 8 03:41:12 rudra kernel: [ 596.807570] NOHZ: local_softirq_pending 22 May 8 03:41:20 rudra kernel: [ 604.847182] NOHZ: local_softirq_pending 22 May 8 03:41:20 rudra kernel: [ 604.847637] NOHZ: local_softirq_pending 22 May 8 03:41:20 rudra kernel: [ 604.847941] NOHZ: local_softirq_pending 22 May 8 03:41:20 rudra kernel: [ 604.848225] NOHZ: local_softirq_pending 22 May 8 03:41:20 rudra kernel: [ 604.848463] NOHZ: local_softirq_pending 22 May 8 03:41:20 rudra kernel: [ 604.848700] NOHZ: local_softirq_pending 22 May 8 03:41:23 rudra kernel: [ 607.702270] NOHZ: local_softirq_pending 22 May 8 03:41:23 rudra kernel: [ 607.702604] NOHZ: local_softirq_pending 22 May 8 03:41:23 rudra kernel: [ 607.702804] NOHZ: local_softirq_pending 22 May 8 03:41:23 rudra kernel: [ 607.702982] NOHZ: local_softirq_pending 22 May 8 03:41:23 rudra kernel: [ 607.703181] NOHZ: local_softirq_pending 22 May 8 03:41:23 rudra kernel: [ 607.703431] NOHZ: local_softirq_pending 22 May 8 03:41:23 rudra kernel: [ 607.703621] NOHZ: local_softirq_pending 22 May 8 03:41:29 rudra kernel: [ 613.888941] NOHZ: local_softirq_pending 22 May 8 03:41:29 rudra kernel: [ 613.889415] NOHZ: local_softirq_pending 22 May 8 03:41:29 rudra kernel: [ 613.889712] NOHZ: local_softirq_pending 22 May 8 03:41:29 rudra kernel: [ 613.890009] NOHZ: local_softirq_pending 22 May 8 03:41:29 rudra kernel: [ 613.890251] NOHZ: local_softirq_pending 22 May 8 03:41:29 rudra kernel: [ 613.890487] NOHZ: local_softirq_pending 22 May 8 03:41:32 rudra kernel: [ 616.743862] NOHZ: local_softirq_pending 22 May 8 03:41:32 rudra kernel: [ 616.744331] NOHZ: local_softirq_pending 22 May 8 03:41:32 rudra kernel: [ 616.744640] NOHZ: local_softirq_pending 22 May 8 03:41:32 rudra kernel: [ 616.744929] NOHZ: local_softirq_pending 22 May 8 03:41:32 rudra kernel: [ 616.745175] NOHZ: local_softirq_pending 22 May 8 03:41:32 rudra kernel: [ 616.745409] NOHZ: local_softirq_pending 22 May 8 03:41:34 rudra kernel: [ 618.660611] NOHZ: local_softirq_pending a2 May 8 03:41:34 rudra kernel: [ 618.660958] NOHZ: local_softirq_pending a2 May 8 03:41:34 rudra kernel: [ 618.661166] NOHZ: local_softirq_pending a2 May 8 03:41:34 rudra kernel: [ 618.661349] NOHZ: local_softirq_pending a2 May 8 03:41:34 rudra kernel: [ 618.661541] NOHZ: local_softirq_pending a2 May 8 03:41:34 rudra kernel: [ 618.661729] NOHZ: local_softirq_pending a2 May 8 03:41:34 rudra kernel: [ 618.661911] NOHZ: local_softirq_pending a2 May 8 03:41:34 rudra kernel: [ 618.662089] NOHZ: local_softirq_pending a2 May 8 03:41:37 rudra kernel: [ 621.996786] NOHZ: local_softirq_pending 22 May 8 03:41:50 rudra kernel: [ 634.831692] NOHZ: local_softirq_pending 22 May 8 03:41:50 rudra kernel: [ 634.832143] NOHZ: local_softirq_pending 22 May 8 03:41:50 rudra kernel: [ 634.832453] NOHZ: local_softirq_pending 22 May 8 03:41:50 rudra kernel: [ 634.832765] NOHZ: local_softirq_pending 22 May 8 03:41:50 rudra kernel: [ 634.833011] NOHZ: local_softirq_pending 22 May 8 03:41:50 rudra kernel: [ 634.833241] NOHZ: local_softirq_pending 22 May 8 03:41:51 rudra kernel: [ 635.790271] NOHZ: local_softirq_pending a2 May 8 03:41:51 rudra kernel: [ 635.790608] NOHZ: local_softirq_pending a2 May 8 03:41:51 rudra kernel: [ 635.790823] NOHZ: local_softirq_pending a2 May 8 03:41:51 rudra kernel: [ 635.791001] NOHZ: local_softirq_pending a2 May 8 03:41:51 rudra kernel: [ 635.791179] NOHZ: local_softirq_pending a2 May 8 03:41:51 rudra kernel: [ 635.791354] NOHZ: local_softirq_pending a2 May 8 03:41:51 rudra kernel: [ 635.791537] NOHZ: local_softirq_pending a2 May 8 03:41:53 rudra kernel: [ 637.686505] NOHZ: local_softirq_pending 22 May 8 03:41:53 rudra kernel: [ 637.686981] NOHZ: local_softirq_pending 22 May 8 03:41:53 rudra kernel: [ 637.687428] NOHZ: local_softirq_pending 22 May 8 03:42:08 rudra kernel: [ 652.941445] NOHZ: local_softirq_pending 22 May 8 03:42:08 rudra kernel: [ 652.941976] NOHZ: local_softirq_pending 22 May 8 03:42:08 rudra kernel: [ 652.942243] NOHZ: local_softirq_pending 22 May 8 03:42:08 rudra kernel: [ 652.942485] NOHZ: local_softirq_pending 22 May 8 03:42:08 rudra kernel: [ 652.942717] NOHZ: local_softirq_pending 22 May 8 03:42:08 rudra kernel: [ 652.942953] NOHZ: local_softirq_pending 22 May 8 03:42:08 rudra kernel: [ 652.943188] NOHZ: local_softirq_pending 22 May 8 03:42:09 rudra kernel: [ 653.862586] NOHZ: local_softirq_pending 22 May 8 03:42:09 rudra kernel: [ 653.862918] NOHZ: local_softirq_pending 22 May 8 03:42:09 rudra kernel: [ 653.863124] NOHZ: local_softirq_pending 22 May 8 03:42:09 rudra kernel: [ 653.863316] NOHZ: local_softirq_pending 22 May 8 03:42:09 rudra kernel: [ 653.863501] NOHZ: local_softirq_pending 22 May 8 03:42:09 rudra kernel: [ 653.863687] NOHZ: local_softirq_pending 22 May 8 03:42:09 rudra kernel: [ 653.863863] NOHZ: local_softirq_pending 22 May 8 03:42:12 rudra kernel: [ 656.714675] NOHZ: local_softirq_pending 22 May 8 03:42:12 rudra kernel: [ 656.715163] NOHZ: local_softirq_pending 22 May 8 03:42:12 rudra kernel: [ 656.715423] NOHZ: local_softirq_pending 22 May 8 03:42:12 rudra kernel: [ 656.715665] NOHZ: local_softirq_pending 22 May 8 03:42:12 rudra kernel: [ 656.715904] NOHZ: local_softirq_pending 22 May 8 03:42:12 rudra kernel: [ 656.716140] NOHZ: local_softirq_pending 22 May 8 03:42:12 rudra kernel: [ 656.716378] NOHZ: local_softirq_pending 22 May 8 03:42:12 rudra kernel: [ 656.716611] NOHZ: local_softirq_pending 22 May 8 03:42:14 rudra kernel: [ 658.622576] NOHZ: local_softirq_pending 22 May 8 03:42:15 rudra kernel: [ 659.569910] NOHZ: local_softirq_pending 22 May 8 03:42:15 rudra kernel: [ 659.570340] NOHZ: local_softirq_pending 22 May 8 03:42:15 rudra kernel: [ 659.570650] NOHZ: local_softirq_pending 22 May 8 03:42:15 rudra kernel: [ 659.570924] NOHZ: local_softirq_pending 22 May 8 03:42:15 rudra kernel: [ 659.571159] NOHZ: local_softirq_pending 22 May 8 03:42:15 rudra kernel: [ 659.571393] NOHZ: local_softirq_pending 22 May 9 01:10:43 rudra kernel: [ 4472.241952] NOHZ: local_softirq_pending 22 May 9 01:10:43 rudra kernel: [ 4472.243018] NOHZ: local_softirq_pending 22 May 9 01:10:43 rudra kernel: [ 4472.243383] NOHZ: local_softirq_pending 22 May 9 01:11:15 rudra kernel: [ 4504.194191] NOHZ: local_softirq_pending 22 May 9 01:11:15 rudra kernel: [ 4504.194815] NOHZ: local_softirq_pending 22 May 9 01:11:15 rudra kernel: [ 4504.195140] NOHZ: local_softirq_pending 22 May 9 01:11:35 rudra kernel: [ 4524.158524] NOHZ: local_softirq_pending 22 May 9 01:11:35 rudra kernel: [ 4524.159118] NOHZ: local_softirq_pending 22 May 9 01:11:35 rudra kernel: [ 4524.159439] NOHZ: local_softirq_pending 22 May 9 01:11:45 rudra kernel: [ 4534.140638] NOHZ: local_softirq_pending 22 May 9 01:11:45 rudra kernel: [ 4534.141046] NOHZ: local_softirq_pending 22 May 9 01:11:45 rudra kernel: [ 4534.141456] NOHZ: local_softirq_pending 22 May 9 01:11:45 rudra kernel: [ 4534.141738] NOHZ: local_softirq_pending 22 May 9 01:12:32 rudra kernel: [ 4580.718873] NOHZ: local_softirq_pending 22 May 9 01:12:32 rudra kernel: [ 4580.719297] NOHZ: local_softirq_pending 22 May 9 01:12:32 rudra kernel: [ 4580.719557] NOHZ: local_softirq_pending 22 May 9 01:12:32 rudra kernel: [ 4580.719808] NOHZ: local_softirq_pending 22 May 9 01:12:32 rudra kernel: [ 4580.720047] NOHZ: local_softirq_pending 22 May 9 01:12:32 rudra kernel: [ 4580.720295] NOHZ: local_softirq_pending 22 May 9 01:12:32 rudra kernel: [ 4580.720534] NOHZ: local_softirq_pending 22 May 9 01:12:36 rudra kernel: [ 4585.053403] NOHZ: local_softirq_pending 22 May 9 01:12:36 rudra kernel: [ 4585.054308] NOHZ: local_softirq_pending 22 May 9 01:12:36 rudra kernel: [ 4585.054588] NOHZ: local_softirq_pending 22 May 9 01:12:36 rudra kernel: [ 4585.054796] NOHZ: local_softirq_pending 22 May 9 01:12:36 rudra kernel: [ 4585.054981] NOHZ: local_softirq_pending 22 May 9 01:12:36 rudra kernel: [ 4585.055163] NOHZ: local_softirq_pending 22 May 9 01:12:38 rudra kernel: [ 4586.927035] NOHZ: local_softirq_pending 22 May 9 01:12:51 rudra kernel: [ 4599.749155] NOHZ: local_softirq_pending 22 May 9 01:12:51 rudra kernel: [ 4599.750614] NOHZ: local_softirq_pending 22 May 9 01:12:53 rudra kernel: [ 4601.648012] NOHZ: local_softirq_pending 22 May 9 01:12:53 rudra kernel: [ 4601.648457] NOHZ: local_softirq_pending 22 May 9 01:12:53 rudra kernel: [ 4601.648722] NOHZ: local_softirq_pending 22 May 9 01:12:53 rudra kernel: [ 4601.648971] NOHZ: local_softirq_pending 22 May 9 01:12:53 rudra kernel: [ 4601.649217] NOHZ: local_softirq_pending 22 May 9 01:12:53 rudra kernel: [ 4601.649463] NOHZ: local_softirq_pending 22 May 9 01:12:53 rudra kernel: [ 4601.649705] NOHZ: local_softirq_pending 22 May 9 01:12:53 rudra kernel: [ 4601.649951] NOHZ: local_softirq_pending 22 May 9 01:19:16 rudra kernel: [ 325.029390] NOHZ: local_softirq_pending 22
TIA
Quoted 9 lines · Anant Nitya
Hi,
Ever since I upgrade to 2.6.21/.1, system log is filled with following messages if I enableCONFIG_NO_HZ=y, going through archives it seems ingo sometime back posted some patch and now it is upstream, but its not helping here. If I disable NOHZ by kernel command line nohz=off this problem disappears. This system is P4/2.40GHz/HT with SMP/SMT on in kernel config. One more thing that I noticed is this problem only arises while using X or network otherwise plain command line with no network access don't trigger this with nohz=on.
Is this independent of the load on the system ? i.e. : What happens if you only use the console and run a kernel compile with -j4 ?
Quoted 3 lines
dumdum@hahakaar [~]$ >> grep NOHZ
/var/log/messagesMay 8 03:38:14 rudra kernel: [ 419.271195] NOHZ: local_softirq_pending 02 May 8 03:38:14 rudra kernel: [ 419.271663] NOHZ: local_softirq_pending 02
The patch below ratelimits the printk output, so your syslog is not flooded anymore.
tglx
kernel/time/tick-sched.c +9 -3
@@ -167,9 +167,15 @@ void tick_nohz_stop_sched_tick(void) goto end; cpu =smp_processor_id (); -if (unlikely (local_softirq_pending ())) -printk (KERN_ERR "NOHZ: local_softirq_pending %02x\n" , -local_softirq_pending ()); +if (unlikely (local_softirq_pending ())) { +static int ratelimit; + +if (ratelimit <10 ) { +printk (KERN_ERR "NOHZ: local_softirq_pending %02x\n" , +local_softirq_pending ()); + ratelimit++; + } + } now =ktime_get ();/*
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Anant Nitya
Hi,
Ever since I upgrade to 2.6.21/.1, system log is filled with following messages if I enableCONFIG_NO_HZ=y, going through archives it seems ingo sometime back posted some patch and now it is upstream, but its not helping here. If I disable NOHZ by kernel command line nohz=off this problem disappears. This system is P4/2.40GHz/HT with SMP/SMT on in kernel config. One more thing that I noticed is this problem only arises while using X or network otherwise plain command line with no network access don't trigger this with nohz=on.
Quoted 2 lines
Is this independent of the load on the system ? i.e. : What happens if you only use the console and run a kernel compile with -j4 ?
Yep, it seems independent of load on the system, to test I compiled kernel with make -j8 and to throw some more load same time I also used amavisd to clean around 2000 of spam/virus infected mails and same time was listening to few radio stream over net in console only login mode. Even then there was not a single NOHZ: local_softirq_pending message in log. Once kernel got compiled and amavisd finished its job and load drops back to around 0.5/2.0 I started X and within few secs log starts getting filled with local_softirq_pending messages, (sorry, I didn't applied ratelimit patch since I wanted to test is it high load on system that causes this or something else). It seems even network operation is not causing this. It seems either X is doing something terribly wrong or kernel is getting hosed by X. If some more information is needed please feel free to ask
dumdum@hahakaar [~]$ >> grep load top.log
top - 14:04:12 up 15 min, 4 users, load average: 9.42, 5.12, 2.20 top - 14:04:17 up 15 min, 4 users, load average: 9.47, 5.20, 2.24 top - 14:04:22 up 16 min, 4 users, load average: 9.35, 5.25, 2.27 top - 14:04:27 up 16 min, 4 users, load average: 9.32, 5.31, 2.31 top - 14:04:32 up 16 min, 4 users, load average: 9.22, 5.36, 2.34 top - 14:04:37 up 16 min, 4 users, load average: 9.36, 5.45, 2.38 top - 14:04:42 up 16 min, 4 users, load average: 9.33, 5.51, 2.42 top - 14:04:47 up 16 min, 4 users, load average: 9.38, 5.58, 2.46 top - 14:04:52 up 16 min, 4 users, load average: 9.35, 5.64, 2.49 top - 14:04:57 up 16 min, 4 users, load average: 9.24, 5.68, 2.52 top - 14:05:02 up 16 min, 4 users, load average: 9.14, 5.72, 2.55 top - 14:05:07 up 16 min, 4 users, load average: 9.05, 5.75, 2.58 top - 14:05:12 up 16 min, 4 users, load average: 9.13, 5.83, 2.62 top - 14:05:17 up 16 min, 4 users, load average: 9.28, 5.91, 2.66 top - 14:05:22 up 17 min, 4 users, load average: 9.34, 5.98, 2.70 top - 14:05:27 up 17 min, 4 users, load average: 9.47, 6.06, 2.75 top - 14:05:32 up 17 min, 4 users, load average: 9.43, 6.11, 2.78 top - 14:05:37 up 17 min, 4 users, load average: 9.32, 6.14, 2.81 top - 14:05:42 up 17 min, 4 users, load average: 9.37, 6.20, 2.85 top - 14:05:47 up 17 min, 4 users, load average: 9.26, 6.23, 2.87 top - 14:05:52 up 17 min, 4 users, load average: 9.16, 6.26, 2.90 top - 14:05:58 up 17 min, 4 users, load average: 9.07, 6.29, 2.93 top - 14:06:03 up 17 min, 4 users, load average: 9.14, 6.35, 2.97 top - 14:06:08 up 17 min, 4 users, load average: 9.13, 6.40, 3.00 top - 14:06:13 up 17 min, 4 users, load average: 9.28, 6.47, 3.04 top - 14:06:18 up 17 min, 4 users, load average: 9.34, 6.53, 3.08 top - 14:06:23 up 18 min, 4 users, load average: 9.31, 6.57, 3.11 top - 14:06:28 up 18 min, 4 users, load average: 9.36, 6.63, 3.15 top - 14:06:33 up 18 min, 4 users, load average: 9.25, 6.65, 3.17 top - 14:06:38 up 18 min, 4 users, load average: 9.23, 6.69, 3.20 top - 14:06:43 up 18 min, 4 users, load average: 9.38, 6.76, 3.25 top - 14:06:48 up 18 min, 4 users, load average: 9.27, 6.78, 3.27 top - 14:06:53 up 18 min, 4 users, load average: 9.32, 6.84, 3.31 top - 14:06:58 up 18 min, 4 users, load average: 9.70, 6.95, 3.36 top - 14:07:03 up 18 min, 4 users, load average: 9.72, 7.00, 3.40 top - 14:07:08 up 18 min, 4 users, load average: 9.82, 7.07, 3.44 top - 14:07:13 up 18 min, 4 users, load average: 9.84, 7.12, 3.48 top - 14:07:18 up 18 min, 4 users, load average: 9.85, 7.17, 3.51 top - 14:07:23 up 19 min, 4 users, load average: 9.78, 7.20, 3.54 top - 14:07:28 up 19 min, 4 users, load average: 9.80, 7.24, 3.57 top - 14:07:33 up 19 min, 4 users, load average: 10.14, 7.40, 3.66 top - 14:07:38 up 19 min, 4 users, load average: 10.45, 7.51, 3.72 top - 14:07:43 up 19 min, 4 users, load average: 10.33, 7.53, 3.75 top - 14:07:48 up 19 min, 4 users, load average: 9.58, 7.42, 3.73 top - 14:07:53 up 19 min, 4 users, load average: 8.97, 7.33, 3.72 top - 14:07:58 up 19 min, 4 users, load average: 8.34, 7.23, 3.71 top - 14:08:03 up 19 min, 4 users, load average: 8.31, 7.24, 3.73 top - 14:08:08 up 19 min, 4 users, load average: 8.36, 7.27, 3.76 top - 14:08:13 up 19 min, 4 users, load average: 8.89, 7.40, 3.82 top - 14:08:18 up 20 min, 4 users, load average: 8.34, 7.31, 3.81 top - 14:08:23 up 20 min, 4 users, load average: 8.47, 7.35, 3.84 top - 14:08:28 up 20 min, 4 users, load average: 8.52, 7.38, 3.87 top - 14:08:33 up 20 min, 4 users, load average: 8.56, 7.41, 3.90 top - 14:08:39 up 20 min, 4 users, load average: 8.51, 7.42, 3.92 top - 14:08:44 up 20 min, 4 users, load average: 8.63, 7.46, 3.95 top - 14:08:49 up 20 min, 4 users, load average: 8.66, 7.48, 3.98 top - 14:08:54 up 20 min, 4 users, load average: 8.85, 7.54, 4.02 top - 14:08:59 up 20 min, 4 users, load average: 8.94, 7.58, 4.05 top - 14:09:04 up 20 min, 4 users, load average: 9.02, 7.62, 4.08 top - 14:09:09 up 20 min, 4 users, load average: 9.02, 7.64, 4.10 top - 14:09:14 up 20 min, 4 users, load average: 8.94, 7.65, 4.13 top - 14:09:19 up 21 min, 4 users, load average: 9.02, 7.69, 4.16 top - 14:09:24 up 21 min, 4 users, load average: 8.94, 7.69, 4.18 top - 14:09:29 up 21 min, 4 users, load average: 8.87, 7.70, 4.20 top - 14:09:34 up 21 min, 4 users, load average: 8.88, 7.72, 4.22 top - 14:09:39 up 21 min, 4 users, load average: 8.81, 7.72, 4.24 top - 14:09:44 up 21 min, 4 users, load average: 8.82, 7.75, 4.27 top - 14:09:49 up 21 min, 4 users, load average: 8.76, 7.75, 4.29 top - 14:09:54 up 21 min, 4 users, load average: 8.85, 7.79, 4.32 top - 14:09:59 up 21 min, 4 users, load average: 8.79, 7.79, 4.34 top - 14:10:04 up 21 min, 4 users, load average: 8.88, 7.83, 4.37 top - 14:10:09 up 21 min, 4 users, load average: 8.81, 7.83, 4.39 top - 14:10:14 up 21 min, 5 users, load average: 8.75, 7.83, 4.41 top - 14:10:19 up 22 min, 5 users, load average: 8.77, 7.85, 4.43 top - 14:10:24 up 22 min, 5 users, load average: 8.79, 7.87, 4.46 top - 14:10:29 up 22 min, 5 users, load average: 8.72, 7.87, 4.47 top - 14:10:34 up 22 min, 5 users, load average: 8.74, 7.89, 4.50 top - 14:10:39 up 22 min, 5 users, load average: 8.92, 7.94, 4.53 top - 14:10:44 up 22 min, 5 users, load average: 9.17, 8.01, 4.57 top - 14:10:49 up 22 min, 5 users, load average: 9.24, 8.04, 4.60 top - 14:10:54 up 22 min, 5 users, load average: 9.14, 8.04, 4.62 top - 14:10:59 up 22 min, 5 users, load average: 9.21, 8.07, 4.65 top - 14:11:04 up 22 min, 5 users, load average: 9.27, 8.10, 4.68 top - 14:11:09 up 22 min, 5 users, load average: 9.25, 8.12, 4.70 top - 14:11:14 up 22 min, 5 users, load average: 9.15, 8.12, 4.72 top - 14:11:19 up 23 min, 5 users, load average: 9.06, 8.11, 4.73 top - 14:11:24 up 23 min, 5 users, load average: 8.97, 8.11, 4.75 top - 14:11:29 up 23 min, 5 users, load average: 9.13, 8.16, 4.79 top - 14:11:35 up 23 min, 5 users, load average: 9.04, 8.16, 4.80 top - 14:11:40 up 23 min, 5 users, load average: 9.12, 8.19, 4.83 top - 14:11:45 up 23 min, 5 users, load average: 9.27, 8.23, 4.86 top - 14:11:50 up 23 min, 5 users, load average: 9.33, 8.26, 4.89 top - 14:11:55 up 23 min, 5 users, load average: 9.22, 8.26, 4.91 top - 14:12:00 up 23 min, 5 users, load average: 9.28, 8.29, 4.93 top - 14:12:05 up 23 min, 5 users, load average: 9.34, 8.32, 4.96 top - 14:12:10 up 23 min, 5 users, load average: 9.47, 8.36, 4.99 top - 14:12:15 up 23 min, 5 users, load average: 9.51, 8.39, 5.02 top - 14:12:20 up 24 min, 5 users, load average: 9.47, 8.40, 5.04 top - 14:12:25 up 24 min, 5 users, load average: 9.52, 8.42, 5.07 top - 14:12:30 up 24 min, 5 users, load average: 9.47, 8.43, 5.09 top - 14:12:35 up 24 min, 5 users, load average: 9.36, 8.43, 5.10 top - 14:12:40 up 24 min, 5 users, load average: 9.25, 8.42, 5.12 top - 14:12:45 up 24 min, 5 users, load average: 9.39, 8.46, 5.15 top - 14:12:50 up 24 min, 5 users, load average: 9.36, 8.47, 5.17 top - 14:12:55 up 24 min, 5 users, load average: 9.41, 8.50, 5.20 top - 14:13:00 up 24 min, 5 users, load average: 9.45, 8.52, 5.22 top - 14:13:05 up 24 min, 5 users, load average: 9.34, 8.51, 5.24 top - 14:13:10 up 24 min, 5 users, load average: 9.23, 8.50, 5.25 top - 14:13:15 up 24 min, 5 users, load average: 9.29, 8.53, 5.28 top - 14:13:20 up 25 min, 5 users, load average: 9.35, 8.55, 5.30 top - 14:13:25 up 25 min, 5 users, load average: 9.32, 8.56, 5.32 top - 14:13:30 up 25 min, 5 users, load average: 9.21, 8.55, 5.34 top - 14:13:35 up 25 min, 5 users, load average: 9.20, 8.56, 5.35 top - 14:13:40 up 25 min, 5 users, load average: 9.10, 8.55, 5.37 top - 14:13:45 up 25 min, 5 users, load average: 9.17, 8.57, 5.39 top - 14:13:50 up 25 min, 5 users, load average: 9.08, 8.56, 5.41 top - 14:13:55 up 25 min, 5 users, load average: 9.07, 8.57, 5.43 top - 14:14:00 up 25 min, 5 users, load average: 8.99, 8.56, 5.44 top - 14:14:05 up 25 min, 5 users, load average: 8.91, 8.55, 5.45 top - 14:14:10 up 25 min, 5 users, load average: 8.91, 8.56, 5.47 top - 14:14:16 up 25 min, 5 users, load average: 8.92, 8.56, 5.49 top - 14:14:21 up 26 min, 5 users, load average: 8.85, 8.55, 5.50 top - 14:14:26 up 26 min, 5 users, load average: 8.86, 8.56, 5.52 top - 14:14:31 up 26 min, 5 users, load average: 9.03, 8.60, 5.55 top - 14:14:36 up 26 min, 5 users, load average: 9.19, 8.64, 5.58 top - 14:14:41 up 26 min, 5 users, load average: 9.17, 8.65, 5.60 top - 14:14:46 up 26 min, 5 users, load average: 9.24, 8.67, 5.62 top - 14:14:51 up 26 min, 5 users, load average: 9.22, 8.67, 5.64 top - 14:14:56 up 26 min, 5 users, load average: 9.28, 8.70, 5.66 top - 14:15:01 up 26 min, 5 users, load average: 9.26, 8.70, 5.68 top - 14:15:06 up 26 min, 5 users, load average: 9.16, 8.69, 5.69 top - 14:15:11 up 26 min, 5 users, load average: 9.14, 8.69, 5.71 top - 14:15:16 up 26 min, 5 users, load average: 9.05, 8.68, 5.72 top - 14:15:21 up 27 min, 5 users, load average: 9.13, 8.70, 5.75 top - 14:15:26 up 27 min, 5 users, load average: 9.04, 8.69, 5.76 top - 14:15:31 up 27 min, 5 users, load average: 9.11, 8.71, 5.78 top - 14:15:36 up 27 min, 5 users, load average: 9.19, 8.74, 5.80 top - 14:15:41 up 27 min, 5 users, load average: 9.17, 8.74, 5.82 top - 14:15:46 up 27 min, 5 users, load average: 9.08, 8.73, 5.83 top - 14:15:51 up 27 min, 5 users, load average: 8.99, 8.71, 5.84 top - 14:15:56 up 27 min, 5 users, load average: 8.99, 8.72, 5.86 top - 14:16:01 up 27 min, 5 users, load average: 8.99, 8.72, 5.88 top - 14:16:06 up 27 min, 5 users, load average: 8.91, 8.71, 5.89 top - 14:16:11 up 27 min, 5 users, load average: 8.92, 8.72, 5.90 top - 14:16:16 up 27 min, 5 users, load average: 9.00, 8.74, 5.93 top - 14:16:21 up 28 min, 5 users, load average: 9.00, 8.74, 5.94 top - 14:16:26 up 28 min, 5 users, load average: 9.00, 8.74, 5.96 top - 14:16:31 up 28 min, 5 users, load average: 9.00, 8.75, 5.97 top - 14:16:37 up 28 min, 5 users, load average: 9.08, 8.77, 6.00 top - 14:16:42 up 28 min, 5 users, load average: 9.16, 8.79, 6.02 top - 14:16:47 up 28 min, 5 users, load average: 9.06, 8.78, 6.03 top - 14:16:52 up 28 min, 5 users, load average: 8.98, 8.76, 6.04 top - 14:16:57 up 28 min, 5 users, load average: 8.90, 8.75, 6.05 top - 14:17:02 up 28 min, 5 users, load average: 9.15, 8.80, 6.08 top - 14:17:07 up 28 min, 5 users, load average: 9.14, 8.81, 6.09 top - 14:17:12 up 28 min, 5 users, load average: 9.12, 8.81, 6.11 top - 14:17:17 up 28 min, 5 users, load average: 9.11, 8.81, 6.12 top - 14:17:22 up 29 min, 5 users, load average: 9.02, 8.80, 6.13 top - 14:17:27 up 29 min, 5 users, load average: 9.02, 8.80, 6.15 top - 14:17:32 up 29 min, 5 users, load average: 9.02, 8.80, 6.16 top - 14:17:37 up 29 min, 5 users, load average: 9.18, 8.84, 6.19 top - 14:17:42 up 29 min, 5 users, load average: 9.24, 8.86, 6.21 top - 14:17:47 up 29 min, 5 users, load average: 9.22, 8.86, 6.23 top - 14:17:52 up 29 min, 5 users, load average: 9.21, 8.86, 6.24 top - 14:17:57 up 29 min, 5 users, load average: 9.19, 8.87, 6.25 top - 14:18:02 up 29 min, 5 users, load average: 9.33, 8.90, 6.28 top - 14:18:07 up 29 min, 5 users, load average: 9.47, 8.93, 6.31 top - 14:18:12 up 29 min, 5 users, load average: 9.43, 8.94, 6.32 top - 14:18:17 up 29 min, 5 users, load average: 9.47, 8.95, 6.34 top - 14:18:22 up 30 min, 5 users, load average: 9.52, 8.97, 6.36 top - 14:18:27 up 30 min, 5 users, load average: 9.64, 9.00, 6.38 top - 14:18:32 up 30 min, 5 users, load average: 9.58, 9.00, 6.40 top - 14:18:37 up 30 min, 5 users, load average: 9.62, 9.02, 6.42 top - 14:18:42 up 30 min, 5 users, load average: 9.65, 9.04, 6.43 top - 14:18:47 up 30 min, 5 users, load average: 9.68, 9.05, 6.45 top - 14:18:52 up 30 min, 5 users, load average: 9.86, 9.10, 6.48 top - 14:18:57 up 30 min, 5 users, load average: 10.03, 9.15, 6.51 top - 14:19:02 up 30 min, 5 users, load average: 10.11, 9.18, 6.54 top - 14:19:07 up 30 min, 5 users, load average: 10.18, 9.21, 6.56 top - 14:19:12 up 30 min, 6 users, load average: 10.25, 9.24, 6.58 top - 14:19:17 up 30 min, 6 users, load average: 10.47, 9.30, 6.62 top - 14:19:22 up 31 min, 6 users, load average: 10.51, 9.33, 6.64 top - 14:19:27 up 31 min, 6 users, load average: 10.47, 9.34, 6.66 top - 14:19:32 up 31 min, 6 users, load average: 10.59, 9.38, 6.69 top - 14:19:38 up 31 min, 6 users, load average: 10.46, 9.38, 6.70 top - 14:19:43 up 31 min, 6 users, load average: 10.59, 9.42, 6.73 top - 14:19:48 up 31 min, 6 users, load average: 10.46, 9.41, 6.74 top - 14:19:53 up 31 min, 6 users, load average: 10.34, 9.40, 6.75 top - 14:19:58 up 31 min, 6 users, load average: 10.23, 9.40, 6.76 top - 14:20:03 up 31 min, 6 users, load average: 10.29, 9.42, 6.79 top - 14:20:08 up 31 min, 6 users, load average: 10.19, 9.42, 6.80 top - 14:20:13 up 31 min, 6 users, load average: 10.26, 9.44, 6.82 top - 14:20:18 up 31 min, 6 users, load average: 10.23, 9.45, 6.84 top - 14:20:23 up 32 min, 6 users, load average: 10.14, 9.44, 6.85 top - 14:20:28 up 32 min, 6 users, load average: 10.04, 9.44, 6.86 top - 14:20:33 up 32 min, 6 users, load average: 9.96, 9.43, 6.87 top - 14:20:38 up 32 min, 6 users, load average: 10.12, 9.47, 6.90 top - 14:20:43 up 32 min, 6 users, load average: 10.11, 9.48, 6.91 top - 14:20:48 up 32 min, 6 users, load average: 10.10, 9.49, 6.93 top - 14:20:53 up 32 min, 6 users, load average: 10.18, 9.51, 6.95 top - 14:20:58 up 32 min, 6 users, load average: 10.08, 9.50, 6.96 top - 14:21:03 up 32 min, 6 users, load average: 9.99, 9.50, 6.97 top - 14:21:08 up 32 min, 6 users, load average: 10.07, 9.52, 7.00 top - 14:21:13 up 32 min, 6 users, load average: 9.99, 9.51, 7.01 top - 14:21:18 up 32 min, 6 users, load average: 10.23, 9.57, 7.04 top - 14:21:23 up 33 min, 6 users, load average: 10.13, 9.56, 7.05 top - 14:21:28 up 33 min, 6 users, load average: 10.19, 9.59, 7.08 top - 14:21:33 up 33 min, 6 users, load average: 10.17, 9.60, 7.10 top - 14:21:38 up 33 min, 6 users, load average: 10.40, 9.65, 7.13 top - 14:21:43 up 33 min, 6 users, load average: 10.45, 9.67, 7.15 top - 14:21:49 up 33 min, 6 users, load average: 10.65, 9.73, 7.18 top - 14:21:54 up 33 min, 6 users, load average: 10.52, 9.72, 7.19 top - 14:21:59 up 33 min, 6 users, load average: 10.64, 9.75, 7.22 top - 14:22:04 up 33 min, 6 users, load average: 10.59, 9.76, 7.23 top - 14:22:09 up 33 min, 6 users, load average: 10.54, 9.76, 7.25 top - 14:22:14 up 33 min, 6 users, load average: 11.14, 9.90, 7.30 top - 14:22:19 up 34 min, 6 users, load average: 11.21, 9.93, 7.33 top - 14:22:24 up 34 min, 6 users, load average: 11.03, 9.92, 7.34 top - 14:22:29 up 34 min, 6 users, load average: 11.11, 9.95, 7.36 top - 14:22:34 up 34 min, 6 users, load average: 10.86, 9.92, 7.37 top - 14:22:39 up 34 min, 6 users, load average: 10.79, 9.92, 7.38 top - 14:22:44 up 34 min, 6 users, load average: 10.81, 9.94, 7.40 top - 14:22:49 up 34 min, 6 users, load average: 10.82, 9.96, 7.42 top - 14:22:54 up 34 min, 6 users, load average: 10.59, 9.92, 7.42 top - 14:22:59 up 34 min, 6 users, load average: 10.63, 9.94, 7.44 top - 14:23:04 up 34 min, 6 users, load average: 10.58, 9.94, 7.45 top - 14:23:10 up 34 min, 6 users, load average: 10.37, 9.91, 7.46 top - 14:23:15 up 34 min, 6 users, load average: 10.34, 9.91, 7.47 top - 14:23:20 up 35 min, 6 users, load average: 10.39, 9.93, 7.49 top - 14:23:25 up 35 min, 6 users, load average: 10.52, 9.96, 7.51 top - 14:23:30 up 35 min, 6 users, load average: 10.56, 9.98, 7.53 top - 14:23:35 up 35 min, 6 users, load average: 10.59, 10.00, 7.55 top - 14:23:40 up 35 min, 6 users, load average: 10.71, 10.03, 7.57 top - 14:23:45 up 35 min, 6 users, load average: 10.73, 10.05, 7.59 top - 14:23:50 up 35 min, 6 users, load average: 10.83, 10.08, 7.61 top - 14:23:55 up 35 min, 6 users, load average: 11.08, 10.14, 7.65 top - 14:24:00 up 35 min, 6 users, load average: 11.00, 10.14, 7.66 top - 14:24:05 up 35 min, 6 users, load average: 10.92, 10.14, 7.67 top - 14:24:10 up 35 min, 6 users, load average: 10.76, 10.12, 7.68 top - 14:24:15 up 35 min, 6 users, load average: 10.78, 10.13, 7.70 top - 14:24:20 up 36 min, 6 users, load average: 12.88, 10.58, 7.85 top - 14:24:25 up 36 min, 6 users, load average: 13.05, 10.65, 7.89 top - 14:24:30 up 36 min, 6 users, load average: 12.89, 10.66, 7.91 top - 14:24:35 up 36 min, 6 users, load average: 12.82, 10.68, 7.93 top - 14:24:40 up 36 min, 6 users, load average: 12.59, 10.67, 7.94 top - 14:24:46 up 36 min, 6 users, load average: 12.46, 10.67, 7.96 top - 14:24:51 up 36 min, 6 users, load average: 12.51, 10.71, 7.98 top - 14:24:56 up 36 min, 6 users, load average: 12.30, 10.70, 8.00 top - 14:25:01 up 36 min, 6 users, load average: 12.12, 10.69, 8.01 top - 14:25:06 up 36 min, 6 users, load average: 12.03, 10.69, 8.02 top - 14:25:11 up 36 min, 6 users, load average: 12.19, 10.75, 8.05 top - 14:25:16 up 36 min, 6 users, load average: 11.93, 10.72, 8.06 top - 14:25:21 up 37 min, 6 users, load average: 11.94, 10.74, 8.08 top - 14:25:26 up 37 min, 6 users, load average: 11.62, 10.69, 8.08 top - 14:25:31 up 37 min, 6 users, load average: 11.57, 10.70, 8.09 top - 14:25:36 up 37 min, 6 users, load average: 11.45, 10.69, 8.10 top - 14:25:41 up 37 min, 6 users, load average: 11.49, 10.71, 8.12 top - 14:25:46 up 37 min, 6 users, load average: 11.69, 10.76, 8.16 top - 14:25:51 up 37 min, 6 users, load average: 11.64, 10.77, 8.17 top - 14:25:56 up 37 min, 6 users, load average: 11.58, 10.77, 8.19 top - 14:26:01 up 37 min, 6 users, load average: 11.46, 10.76, 8.20 top - 14:26:06 up 37 min, 6 users, load average: 11.58, 10.79, 8.22 top - 14:26:11 up 37 min, 6 users, load average: 11.53, 10.80, 8.24 top - 14:26:16 up 37 min, 6 users, load average: 11.41, 10.78, 8.25 top - 14:26:21 up 38 min, 6 users, load average: 11.22, 10.75, 8.25 top - 14:26:26 up 38 min, 6 users, load average: 11.04, 10.72, 8.25 top - 14:26:31 up 38 min, 6 users, load average: 10.88, 10.69, 8.26 top - 14:26:36 up 38 min, 6 users, load average: 10.89, 10.70, 8.27 top - 14:26:41 up 38 min, 6 users, load average: 10.73, 10.67, 8.28 top - 14:26:46 up 38 min, 6 users, load average: 10.68, 10.66, 8.28 top - 14:26:51 up 38 min, 6 users, load average: 10.70, 10.67, 8.30 top - 14:26:57 up 38 min, 6 users, load average: 10.81, 10.69, 8.32 top - 14:27:02 up 38 min, 6 users, load average: 10.66, 10.66, 8.32 top - 14:27:07 up 38 min, 6 users, load average: 10.85, 10.70, 8.35 top - 14:27:12 up 38 min, 6 users, load average: 10.70, 10.67, 8.35 top - 14:27:17 up 38 min, 6 users, load average: 10.72, 10.67, 8.36 top - 14:27:22 up 39 min, 6 users, load average: 10.91, 10.71, 8.39 top - 14:27:27 up 39 min, 6 users, load average: 10.99, 10.73, 8.41 top - 14:27:32 up 39 min, 6 users, load average: 10.99, 10.74, 8.42 top - 14:27:37 up 39 min, 6 users, load average: 10.99, 10.74, 8.43 top - 14:27:42 up 39 min, 6 users, load average: 10.99, 10.75, 8.45 top - 14:27:47 up 39 min, 6 users, load average: 10.99, 10.75, 8.46 top - 14:27:52 up 39 min, 6 users, load average: 10.99, 10.75, 8.47 top - 14:27:57 up 39 min, 6 users, load average: 10.99, 10.76, 8.49 top - 14:28:02 up 39 min, 6 users, load average: 11.07, 10.78, 8.51 top - 14:28:07 up 39 min, 6 users, load average: 10.99, 10.77, 8.51 top - 14:28:12 up 39 min, 6 users, load average: 10.99, 10.77, 8.53 top - 14:28:17 up 39 min, 6 users, load average: 10.91, 10.76, 8.54 top - 14:28:22 up 40 min, 6 users, load average: 10.76, 10.73, 8.54 top - 14:28:27 up 40 min, 6 users, load average: 10.70, 10.71, 8.55 top - 14:28:32 up 40 min, 6 users, load average: 10.72, 10.72, 8.56 top - 14:28:37 up 40 min, 6 users, load average: 10.58, 10.69, 8.56 top - 14:28:42 up 40 min, 6 users, load average: 10.61, 10.69, 8.57 top - 14:28:47 up 40 min, 6 users, load average: 10.49, 10.67, 8.58 top - 14:28:53 up 40 min, 6 users, load average: 10.45, 10.66, 8.58 top - 14:28:58 up 40 min, 6 users, load average: 10.33, 10.63, 8.58 top - 14:29:03 up 40 min, 6 users, load average: 10.46, 10.65, 8.60 top - 14:29:08 up 40 min, 6 users, load average: 10.43, 10.64, 8.61 top - 14:29:13 up 40 min, 6 users, load average: 10.39, 10.63, 8.62 top - 14:29:18 up 40 min, 6 users, load average: 10.68, 10.68, 8.65 top - 14:29:23 up 41 min, 6 users, load average: 10.79, 10.70, 8.66 top - 14:29:28 up 41 min, 6 users, load average: 10.64, 10.68, 8.66 top - 14:29:33 up 41 min, 6 users, load average: 10.75, 10.70, 8.68 top - 14:29:38 up 41 min, 6 users, load average: 10.85, 10.72, 8.70 top - 14:29:43 up 41 min, 6 users, load average: 10.78, 10.71, 8.71 top - 14:29:48 up 41 min, 6 users, load average: 10.96, 10.75, 8.73 top - 14:29:53 up 41 min, 6 users, load average: 11.12, 10.78, 8.75 top - 14:29:58 up 41 min, 6 users, load average: 11.19, 10.80, 8.77 top - 14:30:03 up 41 min, 6 users, load average: 11.55, 10.89, 8.82 top - 14:30:08 up 41 min, 6 users, load average: 11.43, 10.88, 8.83 top - 14:30:13 up 41 min, 6 users, load average: 11.23, 10.84, 8.83 top - 14:30:18 up 42 min, 6 users, load average: 11.13, 10.83, 8.83 top - 14:30:23 up 42 min, 6 users, load average: 11.20, 10.85, 8.85 top - 14:30:28 up 42 min, 6 users, load average: 11.03, 10.82, 8.85 top - 14:30:34 up 42 min, 6 users, load average: 10.94, 10.80, 8.85 top - 14:30:39 up 42 min, 6 users, load average: 10.87, 10.79, 8.86 top - 14:30:44 up 42 min, 6 users, load average: 10.64, 10.74, 8.86 top - 14:30:49 up 42 min, 6 users, load average: 10.59, 10.73, 8.86 top - 14:30:54 up 42 min, 6 users, load average: 10.62, 10.74, 8.87 top - 14:30:59 up 42 min, 6 users, load average: 10.65, 10.74, 8.88 top - 14:31:04 up 42 min, 6 users, load average: 10.68, 10.74, 8.90 top - 14:31:09 up 42 min, 6 users, load average: 10.78, 10.76, 8.91 top - 14:31:14 up 42 min, 6 users, load average: 10.64, 10.74, 8.91 top - 14:31:19 up 43 min, 6 users, load average: 10.67, 10.74, 8.92 top - 14:31:24 up 43 min, 6 users, load average: 10.69, 10.74, 8.93 top - 14:31:29 up 43 min, 6 users, load average: 10.72, 10.75, 8.94 top - 14:31:34 up 43 min, 6 users, load average: 10.66, 10.73, 8.95 top - 14:31:39 up 43 min, 6 users, load average: 10.77, 10.76, 8.97 top - 14:31:44 up 43 min, 6 users, load average: 10.79, 10.76, 8.98 top - 14:31:49 up 43 min, 6 users, load average: 10.64, 10.73, 8.98 top - 14:31:54 up 43 min, 6 users, load average: 10.99, 10.80, 9.01 top - 14:31:59 up 43 min, 6 users, load average: 10.83, 10.77, 9.01 top - 14:32:05 up 43 min, 6 users, load average: 10.68, 10.74, 9.01 top - 14:32:10 up 43 min, 6 users, load average: 10.63, 10.73, 9.01 top - 14:32:15 up 43 min, 6 users, load average: 10.74, 10.75, 9.03 top - 14:32:20 up 44 min, 6 users, load average: 10.76, 10.75, 9.04 top - 14:32:25 up 44 min, 6 users, load average: 10.94, 10.79, 9.06 top - 14:32:30 up 44 min, 6 users, load average: 10.86, 10.78, 9.06 top - 14:32:35 up 44 min, 6 users, load average: 10.87, 10.78, 9.07 top - 14:32:40 up 44 min, 6 users, load average: 10.88, 10.78, 9.08 top - 14:32:45 up 44 min, 6 users, load average: 10.89, 10.79, 9.10 top - 14:32:50 up 44 min, 6 users, load average: 10.90, 10.79, 9.10 top - 14:32:55 up 44 min, 6 users, load average: 10.99, 10.81, 9.12 top - 14:33:00 up 44 min, 6 users, load average: 11.07, 10.83, 9.14 top - 14:33:05 up 44 min, 6 users, load average: 10.90, 10.80, 9.13 top - 14:33:10 up 44 min, 6 users, load average: 10.75, 10.77, 9.13 top - 14:33:15 up 44 min, 6 users, load average: 11.01, 10.82, 9.16 top - 14:33:20 up 45 min, 6 users, load average: 11.17, 10.86, 9.18 top - 14:33:25 up 45 min, 6 users, load average: 11.16, 10.86, 9.19 top - 14:33:30 up 45 min, 6 users, load average: 11.22, 10.88, 9.20 top - 14:33:35 up 45 min, 6 users, load average: 11.13, 10.86, 9.21 top - 14:33:41 up 45 min, 6 users, load average: 11.12, 10.87, 9.22 top - 14:33:46 up 45 min, 6 users, load average: 11.19, 10.88, 9.23 top - 14:33:51 up 45 min, 6 users, load average: 11.17, 10.89, 9.24 top - 14:33:56 up 45 min, 6 users, load average: 11.08, 10.87, 9.25 top - 14:34:01 up 45 min, 6 users, load average: 10.99, 10.86, 9.25 top - 14:34:06 up 45 min, 6 users, load average: 10.91, 10.84, 9.25 top - 14:34:11 up 45 min, 6 users, load average: 10.76, 10.81, 9.25 top - 14:34:16 up 45 min, 6 users, load average: 11.02, 10.86, 9.28 top - 14:34:21 up 46 min, 6 users, load average: 11.10, 10.88, 9.29 top - 14:34:26 up 46 min, 6 users, load average: 11.17, 10.90, 9.31 top - 14:34:31 up 46 min, 6 users, load average: 11.23, 10.92, 9.32 top - 14:34:36 up 46 min, 6 users, load average: 11.13, 10.90, 9.32 top - 14:34:41 up 46 min, 6 users, load average: 11.20, 10.92, 9.34 top - 14:34:46 up 46 min, 6 users, load average: 11.03, 10.89, 9.33 top - 14:34:51 up 46 min, 6 users, load average: 11.10, 10.91, 9.35 top - 14:34:56 up 46 min, 6 users, load average: 10.86, 10.86, 9.34 top - 14:35:01 up 46 min, 6 users, load average: 10.95, 10.88, 9.36 top - 14:35:06 up 46 min, 6 users, load average: 10.79, 10.85, 9.35 top - 14:35:11 up 46 min, 6 users, load average: 11.21, 10.93, 9.39 top - 14:35:16 up 46 min, 6 users, load average: 11.19, 10.93, 9.40 top - 14:35:21 up 47 min, 6 users, load average: 11.09, 10.92, 9.40 top - 14:35:27 up 47 min, 6 users, load average: 11.01, 10.90, 9.40 top - 14:35:32 up 47 min, 6 users, load average: 11.01, 10.90, 9.41 top - 14:35:37 up 47 min, 6 users, load average: 11.09, 10.92, 9.43 top - 14:35:42 up 47 min, 6 users, load average: 11.00, 10.91, 9.43 top - 14:35:47 up 47 min, 6 users, load average: 10.92, 10.89, 9.43 top - 14:35:52 up 47 min, 6 users, load average: 11.00, 10.91, 9.45 top - 14:35:57 up 47 min, 6 users, load average: 11.08, 10.93, 9.46 top - 14:36:02 up 47 min, 6 users, load average: 11.24, 10.96, 9.48 top - 14:36:07 up 47 min, 6 users, load average: 11.06, 10.93, 9.47 top - 14:36:12 up 47 min, 6 users, load average: 11.45, 11.01, 9.51 top - 14:36:17 up 47 min, 6 users, load average: 11.42, 11.01, 9.52 top - 14:36:22 up 48 min, 6 users, load average: 11.46, 11.03, 9.53 top - 14:36:27 up 48 min, 6 users, load average: 11.75, 11.09, 9.56 top - 14:36:32 up 48 min, 6 users, load average: 11.53, 11.06, 9.56 top - 14:36:37 up 48 min, 6 users, load average: 11.64, 11.09, 9.57 top - 14:36:42 up 48 min, 6 users, load average: 11.67, 11.10, 9.59 top - 14:36:47 up 48 min, 6 users, load average: 11.62, 11.10, 9.59 top - 14:36:52 up 48 min, 6 users, load average: 11.65, 11.12, 9.61 top - 14:36:57 up 48 min, 6 users, load average: 11.68, 11.13, 9.62 top - 14:37:02 up 48 min, 6 users, load average: 11.70, 11.15, 9.63 top - 14:37:07 up 48 min, 6 users, load average: 12.05, 11.23, 9.67 top - 14:37:13 up 48 min, 6 users, load average: 12.20, 11.27, 9.69 top - 14:37:18 up 48 min, 6 users, load average: 12.03, 11.25, 9.69 top - 14:37:23 up 49 min, 6 users, load average: 12.10, 11.28, 9.71 top - 14:37:28 up 49 min, 6 users, load average: 12.74, 11.42, 9.76 top - 14:37:33 up 49 min, 6 users, load average: 12.68, 11.43, 9.78 top - 14:37:38 up 49 min, 6 users, load average: 12.30, 11.38, 9.77 top - 14:37:43 up 49 min, 6 users, load average: 12.52, 11.44, 9.79 top - 14:37:48 up 49 min, 6 users, load average: 12.24, 11.40, 9.79 top - 14:37:53 up 49 min, 6 users, load average: 12.22, 11.41, 9.80 top - 14:37:58 up 49 min, 6 users, load average: 12.28, 11.43, 9.82 top - 14:38:03 up 49 min, 6 users, load average: 12.10, 11.41, 9.82 top - 14:38:08 up 49 min, 6 users, load average: 11.93, 11.38, 9.82 top - 14:38:13 up 49 min, 6 users, load average: 11.77, 11.36, 9.82 top - 14:38:18 up 49 min, 6 users, load average: 11.87, 11.39, 9.84 top - 14:38:23 up 50 min, 6 users, load average: 12.12, 11.45, 9.86 top - 14:38:28 up 50 min, 6 users, load average: 12.19, 11.47, 9.88 top - 14:38:33 up 50 min, 6 users, load average: 12.10, 11.46, 9.89 top - 14:38:38 up 50 min, 6 users, load average: 12.95, 11.68, 9.97 top - 14:38:43 up 50 min, 6 users, load average: 12.80, 11.67, 9.98 top - 14:38:48 up 50 min, 6 users, load average: 12.57, 11.64, 9.98 top - 14:38:54 up 50 min, 6 users, load average: 12.69, 11.68, 10.00 top - 14:38:59 up 50 min, 6 users, load average: 12.63, 11.68, 10.01 top - 14:39:04 up 50 min, 6 users, load average: 12.42, 11.65, 10.01 top - 14:39:09 up 50 min, 6 users, load average: 12.31, 11.64, 10.01 top - 14:39:14 up 50 min, 6 users, load average: 12.12, 11.61, 10.01 top - 14:39:19 up 51 min, 6 users, load average: 12.03, 11.60, 10.02 top - 14:39:24 up 51 min, 6 users, load average: 11.71, 11.54, 10.01 top - 14:39:29 up 51 min, 6 users, load average: 12.21, 11.65, 10.05 top - 14:39:34 up 51 min, 6 users, load average: 12.03, 11.62, 10.05 top - 14:39:39 up 51 min, 6 users, load average: 11.87, 11.60, 10.05 top - 14:39:44 up 51 min, 6 users, load average: 11.64, 11.55, 10.04 top - 14:39:50 up 51 min, 6 users, load average: 11.59, 11.54, 10.05 top - 14:39:55 up 51 min, 6 users, load average: 11.38, 11.50, 10.04 top - 14:40:00 up 51 min, 6 users, load average: 11.27, 11.48, 10.04 top - 14:40:05 up 51 min, 6 users, load average: 11.01, 11.42, 10.03 top - 14:40:10 up 51 min, 6 users, load average: 10.93, 11.39, 10.03 top - 14:40:15 up 51 min, 6 users, load average: 10.85, 11.37, 10.03 top - 14:40:20 up 52 min, 6 users, load average: 10.94, 11.38, 10.04 top - 14:40:25 up 52 min, 6 users, load average: 10.87, 11.36, 10.04 top - 14:40:30 up 52 min, 6 users, load average: 10.80, 11.33, 10.04 top - 14:40:35 up 52 min, 6 users, load average: 10.81, 11.33, 10.04 top - 14:40:40 up 52 min, 6 users, load average: 10.67, 11.29, 10.04 top - 14:40:45 up 52 min, 6 users, load average: 10.86, 11.32, 10.05 top - 14:40:50 up 52 min, 6 users, load average: 10.87, 11.31, 10.06 top - 14:40:55 up 52 min, 6 users, load average: 10.96, 11.32, 10.07 top - 14:41:00 up 52 min, 6 users, load average: 10.96, 11.32, 10.07 top - 14:41:05 up 52 min, 6 users, load average: 11.20, 11.36, 10.09 top - 14:41:10 up 52 min, 6 users, load average: 11.43, 11.41, 10.12 top - 14:41:16 up 52 min, 6 users, load average: 11.47, 11.42, 10.12 top - 14:41:21 up 53 min, 6 users, load average: 11.35, 11.39, 10.12 top - 14:41:26 up 53 min, 6 users, load average: 11.41, 11.40, 10.13 top - 14:41:31 up 53 min, 6 users, load average: 11.21, 11.36, 10.13 top - 14:41:36 up 53 min, 6 users, load average: 11.28, 11.37, 10.14 top - 14:41:41 up 53 min, 6 users, load average: 11.25, 11.37, 10.14 top - 14:41:46 up 53 min, 6 users, load average: 11.15, 11.34, 10.14 top - 14:41:51 up 53 min, 6 users, load average: 11.30, 11.37, 10.16 top - 14:41:56 up 53 min, 6 users, load average: 11.28, 11.36, 10.16 top - 14:42:01 up 53 min, 6 users, load average: 11.17, 11.34, 10.16 top - 14:42:06 up 53 min, 6 users, load average: 11.16, 11.33, 10.16 top - 14:42:11 up 53 min, 6 users, load average: 10.99, 11.30, 10.16 top - 14:42:16 up 53 min, 6 users, load average: 10.83, 11.26, 10.15 top - 14:42:21 up 54 min, 6 users, load average: 11.08, 11.30, 10.17 top - 14:42:26 up 54 min, 6 users, load average: 10.99, 11.28, 10.17 top - 14:42:31 up 54 min, 6 users, load average: 11.07, 11.29, 10.18 top - 14:42:37 up 54 min, 6 users, load average: 10.99, 11.27, 10.18 top - 14:42:42 up 54 min, 6 users, load average: 10.83, 11.23, 10.17 top - 14:42:47 up 54 min, 6 users, load average: 10.68, 11.20, 10.17 top - 14:42:52 up 54 min, 6 users, load average: 10.79, 11.21, 10.18 top - 14:42:57 up 54 min, 6 users, load average: 10.88, 11.22, 10.19 top - 14:43:02 up 54 min, 6 users, load average: 11.05, 11.25, 10.20 top - 14:43:07 up 54 min, 6 users, load average: 10.89, 11.21, 10.19 top - 14:43:12 up 54 min, 6 users, load average: 10.82, 11.19, 10.19 top - 14:43:17 up 54 min, 6 users, load average: 10.83, 11.19, 10.20 top - 14:43:22 up 55 min, 6 users, load average: 11.00, 11.22, 10.21 top - 14:43:27 up 55 min, 6 users, load average: 11.00, 11.21, 10.21 top - 14:43:32 up 55 min, 6 users, load average: 10.76, 11.16, 10.20 top - 14:43:37 up 55 min, 6 users, load average: 10.06, 11.01, 10.16 top - 14:43:42 up 55 min, 6 users, load average: 9.42, 10.86, 10.11 top - 14:43:47 up 55 min, 6 users, load average: 8.90, 10.73, 10.08 top - 14:43:52 up 55 min, 6 users, load average: 8.27, 10.57, 10.03 top - 14:43:57 up 55 min, 6 users, load average: 8.01, 10.47, 10.00 top - 14:44:02 up 55 min, 6 users, load average: 7.44, 10.32, 9.95 top - 14:44:07 up 55 min, 6 users, load average: 6.93, 10.16, 9.90 top - 14:44:12 up 55 min, 6 users, load average: 6.69, 10.06, 9.87 top - 14:44:17 up 55 min, 6 users, load average: 6.64, 9.99, 9.85 top - 14:44:22 up 56 min, 6 users, load average: 6.27, 9.86, 9.81 top - 14:44:27 up 56 min, 6 users, load average: 5.84, 9.71, 9.76 top - 14:44:32 up 56 min, 6 users, load average: 5.78, 9.63, 9.73 top - 14:44:37 up 56 min, 6 users, load average: 5.63, 9.54, 9.70 top - 14:44:43 up 56 min, 6 users, load average: 5.90, 9.53, 9.70 top - 14:44:48 up 56 min, 6 users, load average: 6.95, 9.69, 9.75 top - 14:44:53 up 56 min, 6 users, load average: 7.28, 9.71, 9.76 top - 14:44:58 up 56 min, 6 users, load average: 7.49, 9.71, 9.76 top - 14:45:03 up 56 min, 6 users, load average: 8.09, 9.80, 9.78 top - 14:45:08 up 56 min, 6 users, load average: 8.25, 9.80, 9.79 top - 14:45:13 up 56 min, 6 users, load average: 8.71, 9.87, 9.81 top - 14:45:18 up 56 min, 6 users, load average: 8.65, 9.84, 9.80 top - 14:45:23 up 57 min, 6 users, load average: 8.76, 9.84, 9.80 top - 14:45:28 up 57 min, 6 users, load average: 8.86, 9.85, 9.80 top - 14:45:33 up 57 min, 6 users, load average: 9.19, 9.90, 9.82 top - 14:45:38 up 57 min, 6 users, load average: 9.49, 9.95, 9.83 top - 14:45:43 up 57 min, 6 users, load average: 9.61, 9.97, 9.84 top - 14:45:48 up 57 min, 6 users, load average: 10.05, 10.05, 9.87 top - 14:45:53 up 57 min, 6 users, load average: 10.28, 10.10, 9.88 top - 14:45:58 up 57 min, 6 users, load average: 10.34, 10.11, 9.89 top - 14:46:03 up 57 min, 6 users, load average: 10.55, 10.16, 9.91 top - 14:46:08 up 57 min, 6 users, load average: 10.67, 10.19, 9.92 top - 14:46:13 up 57 min, 6 users, load average: 11.09, 10.30, 9.95 top - 14:46:18 up 58 min, 6 users, load average: 10.92, 10.28, 9.95 top - 14:46:23 up 58 min, 6 users, load average: 11.33, 10.37, 9.98 top - 14:46:28 up 58 min, 6 users, load average: 11.38, 10.40, 9.99 top - 14:46:34 up 58 min, 6 users, load average: 11.19, 10.38, 9.99 top - 14:46:39 up 58 min, 6 users, load average: 11.02, 10.35, 9.98 top - 14:46:44 up 58 min, 6 users, load average: 10.94, 10.35, 9.98 top - 14:46:49 up 58 min, 6 users, load average: 10.94, 10.36, 9.99 top - 14:46:54 up 58 min, 6 users, load average: 11.02, 10.38, 10.00 top - 14:46:59 up 58 min, 6 users, load average: 10.86, 10.36, 9.99 top - 14:47:04 up 58 min, 6 users, load average: 10.87, 10.37, 10.00 top - 14:47:09 up 58 min, 6 users, load average: 10.80, 10.36, 10.00 top - 14:47:14 up 58 min, 6 users, load average: 10.74, 10.36, 10.00 top - 14:47:19 up 59 min, 6 users, load average: 10.84, 10.39, 10.01 top - 14:47:24 up 59 min, 6 users, load average: 10.85, 10.39, 10.01 top - 14:47:29 up 59 min, 6 users, load average: 11.02, 10.44, 10.03 top - 14:47:34 up 59 min, 6 users, load average: 10.94, 10.43, 10.03 top - 14:47:39 up 59 min, 6 users, load average: 10.87, 10.42, 10.03 top - 14:47:44 up 59 min, 6 users, load average: 10.72, 10.40, 10.02 top - 14:47:49 up 59 min, 6 users, load average: 10.66, 10.39, 10.02 top - 14:47:54 up 59 min, 6 users, load average: 10.69, 10.40, 10.03 top - 14:47:59 up 59 min, 6 users, load average: 10.71, 10.41, 10.03 top - 14:48:04 up 59 min, 6 users, load average: 10.73, 10.42, 10.04 top - 14:48:09 up 59 min, 6 users, load average: 10.75, 10.43, 10.04 top - 14:48:14 up 59 min, 6 users, load average: 10.77, 10.44, 10.05 top - 14:48:20 up 1:00, 6 users, load average: 11.03, 10.50, 10.07 top - 14:48:25 up 1:00, 6 users, load average: 11.27, 10.56, 10.09 top - 14:48:30 up 1:00, 6 users, load average: 11.33, 10.58, 10.10 top - 14:48:35 up 1:00, 6 users, load average: 11.14, 10.55, 10.09 top - 14:48:40 up 1:00, 6 users, load average: 11.29, 10.60, 10.11 top - 14:48:45 up 1:00, 6 users, load average: 11.43, 10.63, 10.12 top - 14:48:50 up 1:00, 6 users, load average: 11.47, 10.66, 10.13 top - 14:48:55 up 1:00, 6 users, load average: 11.51, 10.68, 10.14 top - 14:49:00 up 1:00, 6 users, load average: 11.39, 10.67, 10.14 top - 14:49:05 up 1:00, 6 users, load average: 11.52, 10.71, 10.16 top - 14:49:10 up 1:00, 6 users, load average: 11.40, 10.69, 10.16 top - 14:49:15 up 1:00, 6 users, load average: 11.45, 10.72, 10.16 top - 14:49:20 up 1:01, 6 users, load average: 11.41, 10.72, 10.17 top - 14:49:25 up 1:01, 6 users, load average: 11.30, 10.71, 10.17 top - 14:49:30 up 1:01, 6 users, load average: 11.75, 10.81, 10.20 top - 14:49:35 up 1:01, 6 users, load average: 11.53, 10.78, 10.20 top - 14:49:40 up 1:01, 6 users, load average: 11.41, 10.77, 10.20 top - 14:49:45 up 1:01, 6 users, load average: 11.46, 10.79, 10.21 top - 14:49:50 up 1:01, 6 users, load average: 11.34, 10.77, 10.20 top - 14:49:55 up 1:01, 6 users, load average: 11.15, 10.75, 10.20 top - 14:50:00 up 1:01, 6 users, load average: 11.38, 10.80, 10.22 top - 14:50:05 up 1:01, 6 users, load average: 11.03, 10.74, 10.20 top - 14:50:10 up 1:01, 6 users, load average: 11.11, 10.76, 10.21 top - 14:50:15 up 1:01, 6 users, load average: 10.86, 10.71, 10.20 top - 14:50:20 up 1:02, 6 users, load average: 11.11, 10.76, 10.22 top - 14:50:25 up 1:02, 6 users, load average: 11.42, 10.83, 10.24 top - 14:50:31 up 1:02, 6 users, load average: 11.47, 10.85, 10.25 top - 14:50:36 up 1:02, 6 users, load average: 11.59, 10.89, 10.27 top - 14:50:41 up 1:02, 6 users, load average: 11.94, 10.97, 10.30 top - 14:50:46 up 1:02, 6 users, load average: 12.19, 11.04, 10.32 top - 14:50:51 up 1:02, 6 users, load average: 12.65, 11.15, 10.36 top - 14:50:56 up 1:02, 6 users, load average: 12.92, 11.23, 10.39 top - 14:51:01 up 1:02, 6 users, load average: 13.17, 11.31, 10.42 top - 14:51:06 up 1:02, 6 users, load average: 13.47, 11.41, 10.46 top - 14:51:11 up 1:02, 6 users, load average: 13.76, 11.50, 10.49 top - 14:51:16 up 1:02, 6 users, load average: 14.02, 11.59, 10.53 top - 14:51:21 up 1:03, 6 users, load average: 13.37, 11.50, 10.50 top - 14:51:26 up 1:03, 6 users, load average: 12.86, 11.42, 10.48 top - 14:51:31 up 1:03, 6 users, load average: 12.23, 11.32, 10.45 top - 14:51:36 up 1:03, 6 users, load average: 11.65, 11.21, 10.42 top - 14:51:41 up 1:03, 6 users, load average: 11.20, 11.12, 10.40 top - 14:51:46 up 1:03, 6 users, load average: 10.62, 11.01, 10.37 top - 14:51:51 up 1:03, 6 users, load average: 9.93, 10.86, 10.32 top - 14:51:56 up 1:03, 6 users, load average: 9.38, 10.73, 10.28 top - 14:52:01 up 1:03, 6 users, load average: 8.95, 10.61, 10.25 top - 14:52:06 up 1:03, 6 users, load average: 8.47, 10.49, 10.21 top - 14:52:12 up 1:03, 6 users, load average: 7.95, 10.35, 10.16 top - 14:52:17 up 1:03, 6 users, load average: 7.55, 10.22, 10.12 top - 14:52:22 up 1:04, 6 users, load average: 7.27, 10.12, 10.09 top - 14:52:27 up 1:04, 6 users, load average: 7.01, 10.02, 10.06 top - 14:52:32 up 1:04, 6 users, load average: 6.61, 9.89, 10.02 top - 14:52:37 up 1:04, 6 users, load average: 6.40, 9.79, 9.98 top - 14:52:42 up 1:04, 6 users, load average: 6.12, 9.67, 9.94 top - 14:52:47 up 1:04, 6 users, load average: 6.03, 9.60, 9.92 top - 14:52:52 up 1:04, 6 users, load average: 5.95, 9.52, 9.89 top - 14:52:57 up 1:04, 6 users, load average: 5.79, 9.43, 9.86 top - 14:53:02 up 1:04, 6 users, load average: 5.65, 9.34, 9.83 top - 14:53:07 up 1:04, 6 users, load average: 5.44, 9.23, 9.79 top - 14:53:12 up 1:04, 6 users, load average: 5.24, 9.13, 9.75 top - 14:53:17 up 1:04, 6 users, load average: 5.22, 9.06, 9.73 top - 14:53:22 up 1:05, 6 users, load average: 4.97, 8.94, 9.69 top - 14:53:27 up 1:05, 6 users, load average: 5.05, 8.89, 9.67 top - 14:53:32 up 1:05, 6 users, load average: 5.36, 8.89, 9.66 top - 14:53:37 up 1:05, 6 users, load average: 5.09, 8.78, 9.62 top - 14:53:42 up 1:05, 6 users, load average: 4.85, 8.67, 9.58 top - 14:53:47 up 1:05, 6 users, load average: 4.62, 8.56, 9.54 top - 14:53:52 up 1:05, 6 users, load average: 4.33, 8.43, 9.49 top - 14:53:57 up 1:05, 6 users, load average: 4.30, 8.36, 9.46 top - 14:54:02 up 1:05, 6 users, load average: 4.04, 8.23, 9.42 top - 14:54:07 up 1:05, 6 users, load average: 3.95, 8.15, 9.38 top - 14:54:12 up 1:05, 6 users, load average: 4.04, 8.09, 9.36 top - 14:54:17 up 1:05, 6 users, load average: 4.03, 8.03, 9.33 top - 14:54:22 up 1:06, 6 users, load average: 5.07, 8.18, 9.37 top - 14:54:27 up 1:06, 6 users, load average: 4.91, 8.09, 9.34 top - 14:54:32 up 1:06, 6 users, load average: 4.83, 8.02, 9.31 top - 14:54:37 up 1:06, 6 users, load average: 4.53, 7.90, 9.26 top - 14:54:42 up 1:06, 6 users, load average: 4.64, 7.87, 9.24 top - 14:54:47 up 1:06, 6 users, load average: 4.83, 7.86, 9.23 top - 14:54:52 up 1:06, 6 users, load average: 4.61, 7.76, 9.19 top - 14:54:57 up 1:06, 6 users, load average: 4.80, 7.75, 9.18 top - 14:55:02 up 1:06, 6 users, load average: 4.49, 7.64, 9.14 top - 14:55:07 up 1:06, 6 users, load average: 4.29, 7.54, 9.10 top - 14:55:12 up 1:06, 6 users, load average: 4.19, 7.47, 9.07 top - 14:55:17 up 1:06, 6 users, load average: 4.17, 7.41, 9.04 top - 14:55:23 up 1:07, 6 users, load average: 4.40, 7.40, 9.03 top - 14:55:28 up 1:07, 6 users, load average: 4.21, 7.31, 8.99 top - 14:55:33 up 1:07, 6 users, load average: 4.43, 7.31, 8.98 top - 14:55:38 up 1:07, 6 users, load average: 4.32, 7.23, 8.95 top - 14:55:43 up 1:07, 6 users, load average: 4.29, 7.18, 8.92 top - 14:55:48 up 1:07, 6 users, load average: 4.27, 7.13, 8.89 top - 14:55:53 up 1:07, 6 users, load average: 4.09, 7.04, 8.85 top - 14:55:58 up 1:07, 6 users, load average: 3.84, 6.94, 8.81 top - 14:56:03 up 1:07, 6 users, load average: 3.69, 6.86, 8.78 top - 14:56:08 up 1:07, 6 users, load average: 3.55, 6.78, 8.74 top - 14:56:13 up 1:07, 6 users, load average: 3.43, 6.70, 8.70 top - 14:56:18 up 1:07, 6 users, load average: 3.24, 6.60, 8.66 top - 14:56:23 up 1:08, 6 users, load average: 3.22, 6.54, 8.63 top - 14:56:28 up 1:08, 6 users, load average: 3.20, 6.48, 8.60 top - 14:56:33 up 1:08, 6 users, load average: 3.34, 6.46, 8.58 top - 14:56:38 up 1:08, 6 users, load average: 3.15, 6.37, 8.54 top - 14:56:43 up 1:08, 6 users, load average: 3.14, 6.31, 8.51 top - 14:56:48 up 1:08, 6 users, load average: 2.97, 6.22, 8.47 top - 14:56:53 up 1:08, 6 users, load average: 2.81, 6.14, 8.43 top - 14:56:58 up 1:08, 6 users, load average: 2.75, 6.07, 8.39 top - 14:57:03 up 1:08, 6 users, load average: 2.77, 6.02, 8.36 top - 14:57:08 up 1:08, 6 users, load average: 2.70, 5.95, 8.33 top - 14:57:13 up 1:08, 6 users, load average: 3.22, 5.95, 8.30 top - 14:57:18 up 1:09, 6 users, load average: 3.20, 5.90, 8.28 top - 14:57:23 up 1:09, 6 users, load average: 3.10, 5.84, 8.24 top - 14:57:28 up 1:09, 6 users, load average: 2.93, 5.76, 8.20 top - 14:57:33 up 1:09, 6 users, load average: 2.78, 5.68, 8.16 top - 14:57:38 up 1:09, 6 users, load average: 2.64, 5.60, 8.12 top - 14:57:43 up 1:09, 6 users, load average: 2.51, 5.52, 8.09 top - 14:57:48 up 1:09, 6 users, load average: 2.55, 5.48, 8.06 top - 14:57:53 up 1:09, 6 users, load average: 2.66, 5.46, 8.04 top - 14:57:58 up 1:09, 6 users, load average: 2.69, 5.41, 8.01 top - 14:58:04 up 1:09, 6 users, load average: 2.71, 5.37, 7.98 top - 14:58:09 up 1:09, 6 users, load average: 2.82, 5.35, 7.96 top - 14:58:14 up 1:09, 6 users, load average: 2.91, 5.33, 7.94 top - 14:58:19 up 1:10, 6 users, load average: 2.92, 5.29, 7.91 top - 14:58:24 up 1:10, 6 users, load average: 2.76, 5.22, 7.87 top - 14:58:29 up 1:10, 6 users, load average: 2.86, 5.20, 7.85 top - 14:58:34 up 1:10, 6 users, load average: 2.79, 5.14, 7.82 top - 14:58:39 up 1:10, 6 users, load average: 2.65, 5.08, 7.79 top - 14:58:44 up 1:10, 6 users, load average: 2.76, 5.06, 7.76 top - 14:58:49 up 1:10, 6 users, load average: 2.62, 4.99, 7.73 top - 14:58:54 up 1:10, 6 users, load average: 2.65, 4.96, 7.70 top - 14:58:59 up 1:10, 6 users, load average: 2.51, 4.89, 7.67 top - 14:59:04 up 1:10, 6 users, load average: 2.63, 4.88, 7.65 top - 14:59:09 up 1:10, 6 users, load average: 2.82, 4.88, 7.63 top - 14:59:14 up 1:10, 6 users, load average: 2.68, 4.81, 7.60 top - 14:59:19 up 1:11, 6 users, load average: 2.62, 4.77, 7.57 top - 14:59:24 up 1:11, 6 users, load average: 2.57, 4.72, 7.54 top - 14:59:29 up 1:11, 6 users, load average: 2.69, 4.71, 7.52 top - 14:59:34 up 1:11, 6 users, load average: 2.63, 4.66, 7.49 top - 14:59:39 up 1:11, 6 users, load average: 2.50, 4.60, 7.45 top - 14:59:44 up 1:11, 6 users, load average: 2.54, 4.58, 7.43 top - 14:59:49 up 1:11, 6 users, load average: 2.42, 4.52, 7.39 top - 14:59:54 up 1:11, 6 users, load average: 2.38, 4.47, 7.36 top - 14:59:59 up 1:11, 6 users, load average: 2.59, 4.48, 7.35 top - 15:00:04 up 1:11, 6 users, load average: 2.70, 4.47, 7.33 top - 15:00:09 up 1:11, 6 users, load average: 2.57, 4.42, 7.30 top - 15:00:14 up 1:11, 6 users, load average: 2.44, 4.36, 7.26 top - 15:00:19 up 1:12, 6 users, load average: 2.49, 4.34, 7.24 top - 15:00:24 up 1:12, 6 users, load average: 2.53, 4.31, 7.22 top - 15:00:29 up 1:12, 6 users, load average: 2.40, 4.26, 7.18 top - 15:00:34 up 1:12, 6 users, load average: 2.37, 4.22, 7.16 top - 15:00:39 up 1:12, 6 users, load average: 2.34, 4.18, 7.13 top - 15:00:44 up 1:12, 6 users, load average: 2.23, 4.13, 7.10 top - 15:00:49 up 1:12, 6 users, load average: 2.29, 4.11, 7.07 top - 15:00:55 up 1:12, 6 users, load average: 2.19, 4.06, 7.04 top - 15:01:00 up 1:12, 6 users, load average: 2.18, 4.02, 7.01 top - 15:01:05 up 1:12, 6 users, load average: 2.08, 3.97, 6.98 top - 15:01:10 up 1:12, 6 users, load average: 1.99, 3.92, 6.95 top - 15:01:15 up 1:12, 6 users, load average: 1.99, 3.89, 6.92 top - 15:01:20 up 1:13, 6 users, load average: 1.91, 3.84, 6.89 top - 15:01:25 up 1:13, 6 users, load average: 1.84, 3.80, 6.86 top - 15:01:30 up 1:13, 6 users, load average: 1.93, 3.78, 6.84 top - 15:01:35 up 1:13, 6 users, load average: 1.86, 3.74, 6.80 top - 15:01:40 up 1:13, 6 users, load average: 1.95, 3.72, 6.78 top - 15:01:45 up 1:13, 6 users, load average: 2.03, 3.71, 6.76 top - 15:01:50 up 1:13, 6 users, load average: 2.27, 3.73, 6.75 top - 15:01:55 up 1:13, 6 users, load average: 2.41, 3.74, 6.74 top - 15:02:00 up 1:13, 6 users, load average: 2.46, 3.72, 6.72 top - 15:02:05 up 1:13, 6 users, load average: 2.50, 3.71, 6.70 top - 15:02:10 up 1:13, 6 users, load average: 2.54, 3.70, 6.68 top - 15:02:15 up 1:13, 6 users, load average: 2.74, 3.72, 6.67 top - 15:02:20 up 1:14, 6 users, load average: 2.92, 3.74, 6.66 top - 15:02:25 up 1:14, 6 users, load average: 3.08, 3.76, 6.65 top - 15:02:30 up 1:14, 6 users, load average: 3.32, 3.80, 6.65 top - 15:02:35 up 1:14, 6 users, load average: 3.13, 3.75, 6.62 top - 15:02:40 up 1:14, 6 users, load average: 3.04, 3.72, 6.59 top - 15:02:45 up 1:14, 6 users, load average: 2.96, 3.69, 6.57 top - 15:02:50 up 1:14, 6 users, load average: 2.88, 3.67, 6.54 top - 15:02:55 up 1:14, 6 users, load average: 2.97, 3.67, 6.53 top - 15:03:00 up 1:14, 6 users, load average: 2.97, 3.66, 6.51 top - 15:03:05 up 1:14, 6 users, load average: 2.89, 3.63, 6.48 top - 15:03:10 up 1:14, 6 users, load average: 2.82, 3.61, 6.46 top - 15:03:15 up 1:14, 6 users, load average: 2.68, 3.56, 6.43 top - 15:03:20 up 1:15, 6 users, load average: 2.54, 3.52, 6.40 top - 15:03:25 up 1:15, 6 users, load average: 2.42, 3.48, 6.37 top - 15:03:30 up 1:15, 6 users, load average: 2.62, 3.50, 6.36 top - 15:03:35 up 1:15, 6 users, load average: 2.73, 3.51, 6.35 top - 15:03:40 up 1:15, 6 users, load average: 2.76, 3.50, 6.33 top - 15:03:45 up 1:15, 6 users, load average: 3.10, 3.56, 6.34 top - 15:03:51 up 1:15, 6 users, load average: 2.93, 3.52, 6.31 top - 15:03:56 up 1:15, 6 users, load average: 3.01, 3.52, 6.29 top - 15:04:01 up 1:15, 6 users, load average: 2.93, 3.50, 6.27 top - 15:04:06 up 1:15, 6 users, load average: 2.86, 3.47, 6.25 top - 15:04:11 up 1:15, 6 users, load average: 2.79, 3.45, 6.22 top - 15:04:16 up 1:15, 6 users, load average: 3.04, 3.49, 6.22 top - 15:04:21 up 1:16, 6 users, load average: 3.20, 3.52, 6.22 top - 15:04:26 up 1:16, 6 users, load average: 3.18, 3.51, 6.20 top - 15:04:31 up 1:16, 6 users, load average: 3.25, 3.51, 6.19 top - 15:04:36 up 1:16, 6 users, load average: 3.07, 3.47, 6.16 top - 15:04:41 up 1:16, 6 users, load average: 3.14, 3.48, 6.15 top - 15:04:46 up 1:16, 6 users, load average: 2.89, 3.42, 6.11 top - 15:04:51 up 1:16, 6 users, load average: 2.66, 3.37, 6.08 top - 15:04:56 up 1:16, 6 users, load average: 2.61, 3.34, 6.06 top - 15:05:01 up 1:16, 6 users, load average: 2.56, 3.32, 6.04 top - 15:05:06 up 1:16, 6 users, load average: 2.35, 3.27, 6.00 top - 15:05:11 up 1:16, 6 users, load average: 2.16, 3.21, 5.97 top - 15:05:16 up 1:16, 6 users, load average: 2.23, 3.21, 5.95 top - 15:05:21 up 1:17, 6 users, load average: 2.13, 3.17, 5.93 top - 15:05:26 up 1:17, 6 users, load average: 2.20, 3.17, 5.91 top - 15:05:31 up 1:17, 6 users, load average: 2.35, 3.18, 5.90 top - 15:05:36 up 1:17, 6 users, load average: 2.16, 3.13, 5.87 top - 15:05:41 up 1:17, 6 users, load average: 1.98, 3.07, 5.84 top - 15:05:46 up 1:17, 6 users, load average: 1.83, 3.02, 5.81 top - 15:05:51 up 1:17, 6 users, load average: 1.92, 3.02, 5.79 top - 15:05:56 up 1:17, 6 users, load average: 2.01, 3.02, 5.78 top - 15:06:01 up 1:17, 6 users, load average: 1.84, 2.97, 5.74 top - 15:06:06 up 1:17, 6 users, load average: 1.70, 2.92, 5.71 top - 15:06:11 up 1:17, 6 users, load average: 1.72, 2.91, 5.69 top - 15:06:16 up 1:17, 6 users, load average: 1.58, 2.86, 5.66 top - 15:06:21 up 1:18, 6 users, load average: 1.46, 2.81, 5.63 top - 15:06:26 up 1:18, 6 users, load average: 1.34, 2.76, 5.60 top - 15:06:31 up 1:18, 6 users, load average: 1.31, 2.73, 5.58 top - 15:06:36 up 1:18, 6 users, load average: 1.45, 2.74, 5.56 top - 15:06:41 up 1:18, 6 users, load average: 1.49, 2.73, 5.54 top - 15:06:46 up 1:18, 6 users, load average: 1.37, 2.68, 5.51 top - 15:06:51 up 1:18, 6 users, load average: 1.50, 2.69, 5.50 top - 15:06:56 up 1:18, 6 users, load average: 1.46, 2.66, 5.47 top - 15:07:01 up 1:18, 6 users, load average: 1.50, 2.65, 5.46 top - 15:07:06 up 1:18, 6 users, load average: 1.46, 2.62, 5.43 top - 15:07:11 up 1:18, 6 users, load average: 1.43, 2.59, 5.41 top - 15:07:16 up 1:18, 6 users, load average: 1.31, 2.55, 5.38 top - 15:07:21 up 1:19, 6 users, load average: 1.29, 2.52, 5.35 top - 15:07:26 up 1:19, 6 users, load average: 1.18, 2.48, 5.33 top - 15:07:31 up 1:19, 6 users, load average: 1.17, 2.45, 5.30 top - 15:07:36 up 1:19, 6 users, load average: 1.31, 2.46, 5.29 top - 15:07:41 up 1:19, 6 users, load average: 1.29, 2.44, 5.27 top - 15:07:46 up 1:19, 6 users, load average: 1.19, 2.40, 5.24 top - 15:07:52 up 1:19, 6 users, load average: 1.09, 2.36, 5.21 top - 15:07:57 up 1:19, 6 users, load average: 1.16, 2.35, 5.19 top - 15:08:02 up 1:19, 6 users, load average: 1.07, 2.31, 5.16 top - 15:08:07 up 1:19, 6 users, load average: 0.98, 2.27, 5.14 top - 15:08:12 up 1:19, 6 users, load average: 1.39, 2.34, 5.14 top - 15:08:17 up 1:19, 6 users, load average: 1.43, 2.33, 5.12 top - 15:08:22 up 1:20, 6 users, load average: 1.56, 2.34, 5.11 top - 15:08:27 up 1:20, 6 users, load average: 1.59, 2.34, 5.09 top - 15:08:32 up 1:20, 6 users, load average: 1.55, 2.31, 5.07 top - 15:08:37 up 1:20, 6 users, load average: 1.74, 2.34, 5.07 top - 15:08:42 up 1:20, 6 users, load average: 1.60, 2.30, 5.04 top - 15:08:47 up 1:20, 6 users, load average: 1.47, 2.26, 5.01 top - 15:08:52 up 1:20, 6 users, load average: 1.60, 2.28, 5.00 top - 15:08:57 up 1:20, 6 users, load average: 1.47, 2.24, 4.97 top - 15:09:02 up 1:20, 6 users, load average: 1.35, 2.20, 4.95 top - 15:09:07 up 1:20, 6 users, load average: 1.24, 2.16, 4.92 top - 15:09:12 up 1:20, 6 users, load average: 1.46, 2.19, 4.91 top - 15:09:17 up 1:20, 6 users, load average: 1.35, 2.16, 4.89 top - 15:09:22 up 1:21, 6 users, load average: 1.32, 2.14, 4.87 top - 15:09:27 up 1:21, 6 users, load average: 1.21, 2.10, 4.84 top - 15:09:32 up 1:21, 6 users, load average: 1.20, 2.08, 4.82 top - 15:09:37 up 1:21, 6 users, load average: 1.10, 2.05, 4.79 top - 15:09:42 up 1:21, 6 users, load average: 1.01, 2.01, 4.77 top - 15:09:47 up 1:21, 6 users, load average: 1.09, 2.01, 4.75 top - 15:09:52 up 1:21, 6 users, load average: 1.16, 2.01, 4.74 top - 15:09:57 up 1:21, 6 users, load average: 1.15, 2.00, 4.72 top - 15:10:02 up 1:21, 6 users, load average: 1.14, 1.98, 4.70 top - 15:10:07 up 1:21, 6 users, load average: 1.37, 2.01, 4.69 top - 15:10:12 up 1:21, 6 users, load average: 1.26, 1.98, 4.67 top - 15:10:17 up 1:21, 6 users, load average: 1.16, 1.95, 4.64 top - 15:10:22 up 1:22, 6 users, load average: 1.14, 1.93, 4.62 top - 15:10:27 up 1:22, 6 users, load average: 1.05, 1.90, 4.60 top - 15:10:32 up 1:22, 6 users, load average: 1.05, 1.88, 4.58 top - 15:10:37 up 1:22, 6 users, load average: 1.20, 1.90, 4.57 top - 15:10:42 up 1:22, 6 users, load average: 1.11, 1.87, 4.54 top - 15:10:47 up 1:22, 6 users, load average: 1.02, 1.84, 4.52 top - 15:10:52 up 1:22, 6 users, load average: 1.02, 1.82, 4.50 top - 15:10:57 up 1:22, 6 users, load average: 0.93, 1.79, 4.47 top - 15:11:02 up 1:22, 6 users, load average: 0.86, 1.76, 4.45 top - 15:11:07 up 1:22, 6 users, load average: 1.03, 1.78, 4.44 top - 15:11:12 up 1:22, 6 users, load average: 0.95, 1.75, 4.42 top - 15:11:17 up 1:22, 6 users, load average: 1.19, 1.79, 4.42 top - 15:11:22 up 1:23, 6 users, load average: 1.10, 1.76, 4.39 top - 15:11:27 up 1:23, 6 users, load average: 1.09, 1.75, 4.37 top - 15:11:32 up 1:23, 6 users, load average: 1.00, 1.72, 4.35 top - 15:11:37 up 1:23, 6 users, load average: 0.92, 1.69, 4.33 top - 15:11:42 up 1:23, 6 users, load average: 0.85, 1.66, 4.30 top - 15:11:47 up 1:23, 6 users, load average: 0.78, 1.63, 4.28 top - 15:11:52 up 1:23, 6 users, load average: 0.80, 1.62, 4.26 top - 15:11:57 up 1:23, 6 users, load average: 0.81, 1.61, 4.24 top - 15:12:02 up 1:23, 6 users, load average: 0.91, 1.62, 4.23 top - 15:12:08 up 1:23, 6 users, load average: 0.83, 1.59, 4.21 top - 15:12:13 up 1:23, 6 users, load average: 0.85, 1.58, 4.19 top - 15:12:18 up 1:23, 6 users, load average: 0.78, 1.56, 4.17 top - 15:12:23 up 1:24, 6 users, load average: 0.88, 1.56, 4.16 top - 15:12:28 up 1:24, 6 users, load average: 1.21, 1.62, 4.16 top - 15:12:33 up 1:24, 6 users, load average: 1.43, 1.66, 4.16 top - 15:12:38 up 1:24, 6 users, load average: 1.40, 1.65, 4.14 top - 15:12:43 up 1:24, 6 users, load average: 1.28, 1.62, 4.12 top - 15:12:48 up 1:24, 6 users, load average: 1.42, 1.64, 4.11 top - 15:12:53 up 1:24, 6 users, load average: 1.39, 1.63, 4.10 top - 15:12:58 up 1:24, 6 users, load average: 1.36, 1.62, 4.08 top - 15:13:03 up 1:24, 6 users, load average: 1.33, 1.61, 4.06 top - 15:13:08 up 1:24, 6 users, load average: 1.46, 1.63, 4.06 top - 15:13:13 up 1:24, 6 users, load average: 1.34, 1.61, 4.04 top - 15:13:18 up 1:24, 6 users, load average: 1.24, 1.58, 4.01 top - 15:13:23 up 1:25, 6 users, load average: 1.22, 1.57, 4.00 top - 15:13:28 up 1:25, 6 users, load average: 1.12, 1.54, 3.97 top - 15:13:33 up 1:25, 6 users, load average: 1.03, 1.52, 3.95 top - 15:13:38 up 1:25, 6 users, load average: 1.11, 1.52, 3.94 top - 15:13:43 up 1:25, 6 users, load average: 1.02, 1.50, 3.92 top - 15:13:48 up 1:25, 6 users, load average: 1.10, 1.51, 3.91 top - 15:13:53 up 1:25, 6 users, load average: 1.09, 1.50, 3.89 top - 15:13:58 up 1:25, 6 users, load average: 1.32, 1.54, 3.89 top - 15:14:03 up 1:25, 6 users, load average: 1.30, 1.53, 3.88 top - 15:14:08 up 1:25, 6 users, load average: 1.35, 1.54, 3.87 top - 15:14:13 up 1:25, 6 users, load average: 1.48, 1.56, 3.86 top - 15:14:18 up 1:25, 6 users, load average: 1.85, 1.64, 3.88 top - 15:14:23 up 1:26, 6 users, load average: 3.54, 1.99, 3.98 top - 15:14:28 up 1:26, 6 users, load average: 4.99, 2.35, 4.07 top - 15:14:33 up 1:26, 6 users, load average: 5.71, 2.55, 4.13 top - 15:14:39 up 1:26, 6 users, load average: 5.90, 2.64, 4.15 top - 15:14:44 up 1:26, 6 users, load average: 6.31, 2.77, 4.18 top - 15:14:49 up 1:26, 6 users, load average: 6.20, 2.81, 4.19 top - 15:14:54 up 1:26, 6 users, load average: 6.26, 2.88, 4.20 top - 15:14:59 up 1:26, 6 users, load average: 6.00, 2.88, 4.20 top - 15:15:04 up 1:26, 6 users, load average: 6.16, 2.97, 4.22 top - 15:15:09 up 1:26, 6 users, load average: 5.91, 2.97, 4.21 top - 15:15:14 up 1:26, 6 users, load average: 5.84, 3.00, 4.21 top - 15:15:19 up 1:27, 6 users, load average: 5.93, 3.07, 4.23 top - 15:15:24 up 1:27, 6 users, load average: 5.69, 3.07, 4.22 top - 15:15:29 up 1:27, 6 users, load average: 6.12, 3.20, 4.26 top - 15:15:34 up 1:27, 6 users, load average: 6.35, 3.29, 4.28 top - 15:15:39 up 1:27, 6 users, load average: 6.72, 3.42, 4.32 top - 15:15:44 up 1:27, 6 users, load average: 7.62, 3.66, 4.39 top - 15:15:49 up 1:27, 6 users, load average: 8.29, 3.87, 4.45 top - 15:15:55 up 1:27, 6 users, load average: 10.19, 4.33, 4.60 top - 15:16:00 up 1:27, 6 users, load average: 10.10, 4.41, 4.62 top - 15:16:05 up 1:27, 6 users, load average: 10.25, 4.54, 4.66 top - 15:16:10 up 1:27, 6 users, load average: 10.07, 4.59, 4.68 top - 15:16:15 up 1:27, 6 users, load average: 9.82, 4.63, 4.69 top - 15:16:20 up 1:28, 6 users, load average: 9.60, 4.67, 4.71 top - 15:16:25 up 1:28, 6 users, load average: 9.63, 4.76, 4.73 top - 15:16:30 up 1:28, 6 users, load average: 9.66, 4.85, 4.76 top - 15:16:35 up 1:28, 6 users, load average: 9.52, 4.90, 4.78 top - 15:16:40 up 1:28, 6 users, load average: 9.16, 4.90, 4.78 top - 15:16:45 up 1:28, 6 users, load average: 9.39, 5.02, 4.82 top - 15:16:50 up 1:28, 6 users, load average: 9.12, 5.04, 4.83 top - 15:16:56 up 1:28, 6 users, load average: 8.71, 5.02, 4.82 top - 15:17:01 up 1:28, 6 users, load average: 8.25, 4.98, 4.81 top - 15:17:06 up 1:28, 6 users, load average: 7.99, 4.98, 4.81 top - 15:17:11 up 1:28, 6 users, load average: 7.75, 4.98, 4.81 top - 15:17:16 up 1:28, 6 users, load average: 7.77, 5.03, 4.83 top - 15:17:21 up 1:29, 6 users, load average: 7.79, 5.08, 4.85 top - 15:17:26 up 1:29, 6 users, load average: 7.88, 5.15, 4.87 top - 15:17:31 up 1:29, 6 users, load average: 8.05, 5.23, 4.89 top - 15:17:36 up 1:29, 6 users, load average: 8.29, 5.32, 4.93 top - 15:17:41 up 1:29, 6 users, load average: 8.51, 5.42, 4.96 top - 15:17:48 up 1:29, 6 users, load average: 9.26, 5.68, 5.05 top - 15:17:53 up 1:29, 6 users, load average: 9.48, 5.79, 5.09 top - 15:17:59 up 1:29, 6 users, load average: 9.12, 5.77, 5.09 top - 15:18:04 up 1:29, 6 users, load average: 8.63, 5.73, 5.08 top - 15:18:09 up 1:29, 6 users, load average: 8.18, 5.68, 5.06 top - 15:18:14 up 1:29, 6 users, load average: 8.01, 5.69, 5.07 top - 15:18:19 up 1:30, 6 users, load average: 7.77, 5.68, 5.07 top - 15:18:24 up 1:30, 6 users, load average: 7.62, 5.68, 5.07 top - 15:18:29 up 1:30, 6 users, load average: 7.17, 5.62, 5.06 top - 15:18:34 up 1:30, 6 users, load average: 6.84, 5.58, 5.05 top - 15:18:39 up 1:30, 6 users, load average: 6.45, 5.52, 5.03 top - 15:18:44 up 1:30, 6 users, load average: 6.17, 5.47, 5.02 top - 15:18:49 up 1:30, 6 users, load average: 6.00, 5.45, 5.01 top - 15:18:54 up 1:30, 6 users, load average: 6.24, 5.51, 5.03 top - 15:18:59 up 1:30, 6 users, load average: 5.82, 5.43, 5.01 top - 15:19:04 up 1:30, 6 users, load average: 5.67, 5.41, 5.01 top - 15:19:09 up 1:30, 6 users, load average: 5.38, 5.35, 4.99 top - 15:19:14 up 1:30, 6 users, load average: 5.11, 5.30, 4.97 top - 15:19:19 up 1:31, 6 users, load average: 4.70, 5.21, 4.95 top - 15:19:24 up 1:31, 6 users, load average: 4.48, 5.15, 4.93 top - 15:19:29 up 1:31, 6 users, load average: 4.20, 5.08, 4.91 top - 15:19:34 up 1:31, 6 users, load average: 3.95, 5.02, 4.89 top - 15:19:39 up 1:31, 6 users, load average: 3.87, 4.98, 4.88 top - 15:19:44 up 1:31, 6 users, load average: 3.64, 4.92, 4.86 top - 15:19:49 up 1:31, 6 users, load average: 3.35, 4.83, 4.83 top - 15:19:54 up 1:31, 6 users, load average: 3.16, 4.77, 4.81 top - 15:19:59 up 1:31, 6 users, load average: 3.31, 4.77, 4.81 top - 15:20:04 up 1:31, 6 users, load average: 3.28, 4.74, 4.80 top - 15:20:09 up 1:31, 6 users, load average: 3.26, 4.72, 4.79 top - 15:20:15 up 1:31, 6 users, load average: 3.00, 4.64, 4.76 top - 15:20:20 up 1:32, 6 users, load average: 2.76, 4.56, 4.74 top - 15:20:25 up 1:32, 6 users, load average: 2.70, 4.52, 4.72 top - 15:20:30 up 1:32, 6 users, load average: 2.56, 4.46, 4.70 top - 15:20:35 up 1:32, 6 users, load average: 2.36, 4.38, 4.68 top - 15:20:40 up 1:32, 6 users, load average: 2.25, 4.33, 4.66 top - 15:20:45 up 1:32, 6 users, load average: 2.07, 4.25, 4.63 top - 15:20:50 up 1:32, 6 users, load average: 1.90, 4.18, 4.61 top - 15:20:55 up 1:32, 6 users, load average: 1.91, 4.15, 4.59 top - 15:21:00 up 1:32, 6 users, load average: 1.76, 4.08, 4.57 top - 15:21:05 up 1:32, 6 users, load average: 1.69, 4.03, 4.55 top - 15:21:10 up 1:32, 6 users, load average: 1.56, 3.96, 4.52 top - 15:21:15 up 1:32, 6 users, load average: 1.43, 3.89, 4.50 top - 15:21:20 up 1:33, 6 users, load average: 1.48, 3.86, 4.49 top - 15:21:25 up 1:33, 6 users, load average: 2.24, 3.98, 4.52 top - 15:21:30 up 1:33, 6 users, load average: 2.14, 3.93, 4.50 top - 15:21:35 up 1:33, 6 users, load average: 2.85, 4.05, 4.54 top - 15:21:40 up 1:33, 6 users, load average: 2.62, 3.98, 4.51 top - 15:21:45 up 1:33, 6 users, load average: 2.41, 3.91, 4.49 top - 15:21:50 up 1:33, 6 users, load average: 2.22, 3.85, 4.46 top - 15:21:55 up 1:33, 6 users, load average: 2.04, 3.78, 4.44 top - 15:22:00 up 1:33, 6 users, load average: 1.88, 3.72, 4.41 top - 15:22:05 up 1:33, 6 users, load average: 1.73, 3.66, 4.39 top - 15:22:10 up 1:33, 6 users, load average: 1.59, 3.60, 4.37 top - 15:22:15 up 1:33, 6 users, load average: 1.46, 3.54, 4.34 top - 15:22:20 up 1:34, 6 users, load average: 1.34, 3.48, 4.32 top - 15:22:25 up 1:34, 6 users, load average: 1.24, 3.42, 4.30 top - 15:22:30 up 1:34, 6 users, load average: 1.14, 3.36, 4.27 top - 15:22:35 up 1:34, 6 users, load average: 1.04, 3.31, 4.25 top - 15:22:40 up 1:34, 6 users, load average: 0.96, 3.25, 4.23 top - 15:22:45 up 1:34, 6 users, load average: 0.88, 3.20, 4.20 top - 15:22:50 up 1:34, 6 users, load average: 0.81, 3.15, 4.18 top - 15:22:55 up 1:34, 6 users, load average: 0.75, 3.09, 4.16 top - 15:23:00 up 1:34, 6 users, load average: 0.69, 3.04, 4.14 top - 15:23:05 up 1:34, 6 users, load average: 0.63, 2.99, 4.11 top - 15:23:10 up 1:34, 6 users, load average: 0.58, 2.94, 4.09 top - 15:23:15 up 1:34, 6 users, load average: 0.53, 2.89, 4.07 top - 15:23:20 up 1:35, 6 users, load average: 0.49, 2.84, 4.05 top - 15:23:25 up 1:35, 6 users, load average: 0.45, 2.80, 4.02 top - 15:23:30 up 1:35, 6 users, load average: 0.42, 2.75, 4.00 top - 15:23:35 up 1:35, 6 users, load average: 0.38, 2.70, 3.98 top - 15:23:40 up 1:35, 6 users, load average: 0.35, 2.66, 3.96 top - 15:23:45 up 1:35, 6 users, load average: 0.32, 2.61, 3.94 top - 15:23:50 up 1:35, 6 users, load average: 0.30, 2.57, 3.92 top - 15:23:55 up 1:35, 6 users, load average: 0.27, 2.53, 3.89 top - 15:24:00 up 1:35, 6 users, load average: 0.25, 2.49, 3.87 top - 15:24:05 up 1:35, 6 users, load average: 0.23, 2.44, 3.85 top - 15:24:10 up 1:35, 6 users, load average: 0.21, 2.40, 3.83 top - 15:24:15 up 1:35, 6 users, load average: 0.19, 2.36, 3.81 top - 15:24:20 up 1:36, 6 users, load average: 0.18, 2.32, 3.79 top - 15:24:25 up 1:36, 6 users, load average: 0.16, 2.28, 3.77 top - 15:24:30 up 1:36, 6 users, load average: 0.15, 2.25, 3.75 top - 15:24:35 up 1:36, 6 users, load average: 0.14, 2.21, 3.73 top - 15:24:40 up 1:36, 6 users, load average: 0.13, 2.17, 3.71 top - 15:24:45 up 1:36, 6 users, load average: 0.12, 2.14, 3.69 top - 15:24:50 up 1:36, 6 users, load average: 0.11, 2.10, 3.67 top - 15:24:55 up 1:36, 6 users, load average: 0.10, 2.06, 3.65 top - 15:25:00 up 1:36, 6 users, load average: 0.09, 2.03, 3.63 top - 15:25:05 up 1:36, 6 users, load average: 0.08, 2.00, 3.61 top - 15:25:11 up 1:36, 6 users, load average: 0.08, 1.96, 3.59 top - 15:25:16 up 1:36, 6 users, load average: 0.07, 1.93, 3.57 top - 15:25:21 up 1:37, 6 users, load average: 0.06, 1.90, 3.55 top - 15:25:26 up 1:37, 6 users, load average: 0.06, 1.87, 3.53 top - 15:25:31 up 1:37, 6 users, load average: 0.05, 1.83, 3.51 top - 15:25:36 up 1:37, 6 users, load average: 0.05, 1.80, 3.49 top - 15:25:41 up 1:37, 6 users, load average: 0.05, 1.77, 3.47 top - 15:25:46 up 1:37, 6 users, load average: 0.04, 1.74, 3.45 top - 15:25:51 up 1:37, 6 users, load average: 0.04, 1.71, 3.44 top - 15:25:56 up 1:37, 6 users, load average: 0.03, 1.69, 3.42
Quoted 14 lines · Anant Nitya
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Anant Nitya
Hi,
Ever since I upgrade to 2.6.21/.1, system log is filled with following messages if I enableCONFIG_NO_HZ=y, going through archives it seems ingo sometime back posted some patch and now it is upstream, but its not helping here. If I disable NOHZ by kernel command line nohz=off this problem disappears. This system is P4/2.40GHz/HT with SMP/SMT on in kernel config. One more thing that I noticed is this problem only arises while using X or network otherwise plain command line with no network access don't trigger this with nohz=on.Quoted 2 lines
Is this independent of the load on the system ? i.e. : What happens if you only use the console and run a kernel compile with -j4 ?
Quoted 13 lines
Yep, it seems independent of load on the system, to test I compiled kernel with make -j8 and to throw some more load same time I also used amavisd to clean around 2000 of spam/virus infected mails and same time was listening to few radio stream over net in console only login mode. Even then there was not a single NOHZ:
local_softirq_pendingmessage in log. Once kernel got compiled and amavisd finished its job and load drops back to around 0.5/2.0 I started X and within few secs log starts getting filled withlocal_softirq_pendingmessages, (sorry, I didn't applied ratelimit patch since I wanted to test is it high load on system that causes this or something else). It seems even network operation is not causing this. It seems either X is doing something terribly wrong or kernel is getting hosed by X. If some more information is needed please feel free to ask
Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.
When you apply the ratelimit patch, does the softlockup problem persist ?
tglx
Quoted 3 lines · On Friday 11 May 2007 03:28:46 Thomas Gleixner wrote:
Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.
Yes with HT disabled in BIOS there is no local_softirq_pending messages. BTW why does this problem persist only with X ?
Quoted 3 lines
When you apply the ratelimit patch, does the softlockup problem persist ?
Yes, though softlockup is rare and mostly hit when system is under high load. Apart of that I am also getting following messages consistently across multiple boot cycles with NOHZ=y and ratelimit patch applied.
May 15 11:51:22 rudra kernel: [ 2594.341068] Clocksource tsc unstable (delta = 28111260302 ns) May 15 11:51:22 rudra kernel: [ 2594.343194] Time: acpi_pm clocksource has
been installed.
Quoted 7 lines · Anant Nitya
Quoted 3 lines · On Friday 11 May 2007 03:28:46 Thomas Gleixner wrote:
Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.
Yes with HT disabled in BIOS there is no
local_softirq_pendingmessages. BTW why does this problem persist only with X ?
No idea. I uploaded a debug patch against 2.6.22-rc1 to
http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch
Can you give it a try and report the output ?
Quoted 11 lines
Quoted 3 lines
When you apply the ratelimit patch, does the softlockup problem persist ?
Yes, though softlockup is rare and mostly hit when system is under high load. Apart of that I am also getting following messages consistently across multiple boot cycles with NOHZ=y and ratelimit patch applied.
May 15 11:51:22 rudra kernel: [ 2594.341068] Clocksource tsc unstable (delta = 28111260302 ns) May 15 11:51:22 rudra kernel: [ 2594.343194] Time: acpi_pm clocksource hasbeen installed.
That's informal. The TSC is detected to be unstable and replaced by the pm timer. Nothing to worry about. It happens with NOHZ=n as well, right ?
tglx
Quoted 6 lines · On Friday 18 May 2007 18:31:17 Thomas Gleixner wrote:
Quoted 5 lines · Anant Nitya
Quoted 3 lines · On Friday 11 May 2007 03:28:46 Thomas Gleixner wrote:
Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.
Quoted 8 lines
Quoted 2 lines
Yes with HT disabled in BIOS there is no
local_softirq_pendingmessages. BTW why does this problem persist only with X ?No idea. I uploaded a debug patch against 2.6.22-rc1 to
http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch
Can you give it a try and report the output ?
I am compiling kernel with above patch applied and will post the results.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
When you apply the ratelimit patch, does the softlockup problem persist ?
Yes, though softlockup is rare and mostly hit when system is under high load. Apart of that I am also getting following messages consistently across multiple boot cycles with NOHZ=y and ratelimit patch applied.
Quoted 8 lines
Quoted 4 lines
May 15 11:51:22 rudra kernel: [ 2594.341068] Clocksource tsc unstable (delta = 28111260302 ns) May 15 11:51:22 rudra kernel: [ 2594.343194] Time: acpi_pm clocksourcehas been installed.
That's informal. The TSC is detected to be unstable and replaced by the pm timer. Nothing to worry about. It happens with NOHZ=n as well, right ?
Nope it don't appear with nohz=no till now across so many boot cycles, it always appears with nohz!=no.
Regards
Ananitya
Quoted 6 lines · On Friday 18 May 2007 18:31:17 Thomas Gleixner wrote:
Quoted 5 lines · Anant Nitya
Quoted 3 lines · On Friday 11 May 2007 03:28:46 Thomas Gleixner wrote:
Ok, that's consistent with earlier reports. The problem surfaces when one of the SMT-"cpus" goes idle. The problem goes away when you disable hyperthreading.
Quoted 8 lines
Quoted 2 lines
Yes with HT disabled in BIOS there is no
local_softirq_pendingmessages. BTW why does this problem persist only with X ?No idea. I uploaded a debug patch against 2.6.22-rc1 to
http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch
Can you give it a try and report the output ?
Hi
Here it goes
[ 159.646196] NOHZ softirq pending 22 on CPU 0 [ 159.646207] .... task state: 1 00000000 [ 159.646217] .... last caller: __tasklet_schedule [ 159.646997] NOHZ softirq pending 22 on CPU 0 [ 159.647006] .... task state: 1 00000000 [ 159.647013] .... last caller: __tasklet_schedule [ 159.647398] NOHZ softirq pending 22 on CPU 0 [ 159.647405] .... task state: 1 00000000 [ 159.647412] .... last caller: __tasklet_schedule [ 159.647768] NOHZ softirq pending 22 on CPU 0 [ 159.647775] .... task state: 1 00000000 [ 159.647781] .... last caller: __tasklet_schedule [ 166.285664] NOHZ softirq pending 22 on CPU 0 [ 166.285675] .... task state: 1 00000000 [ 166.285687] .... last caller: raise_softirq [ 166.286321] NOHZ softirq pending 22 on CPU 0 [ 166.286329] .... task state: 1 00000000 [ 166.286337] .... last caller: raise_softirq [ 166.286715] NOHZ softirq pending 22 on CPU 0 [ 166.286722] .... task state: 1 00000000 [ 166.286729] .... last caller: raise_softirq [ 166.287085] NOHZ softirq pending 22 on CPU 0 [ 166.287092] .... task state: 1 00000000 [ 166.287098] .... last caller: raise_softirq [ 171.512134] NOHZ softirq pending 22 on CPU 0 [ 171.512144] .... task state: 1 00000000 [ 171.512154] .... last caller: __tasklet_schedule [ 171.512712] NOHZ softirq pending 22 on CPU 0 [ 171.512720] .... task state: 1 00000000 [ 171.512727] .... last caller: __tasklet_schedule
Regards
Ananitya
Quoted 9 lines · Anant Nitya
Quoted 5 lines
No idea. I uploaded a debug patch against 2.6.22-rc1 to
http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch
Can you give it a try and report the output ?
Hi
Here it goes[ 159.646196] NOHZ softirq pending 22 on CPU 0 [ 159.646207] .... task state: 1 00000000
1 == TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.
I work out a more complex debug patch and pester you to test once I'm done.
tglx
Quoted 12 lines · On Sunday 20 May 2007 00:41:08 Thomas Gleixner wrote:
Quoted 10 lines · Anant Nitya
Quoted 5 lines
No idea. I uploaded a debug patch against 2.6.22-rc1 to
http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch
Can you give it a try and report the output ?
Hi
Here it goes[ 159.646196] NOHZ softirq pending 22 on CPU 0 [ 159.646207] .... task state: 1 00000000
Quoted 5 lines
1 ==
TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.I work out a more complex debug patch and pester you to test once I'm done.
No problem :)
Regards
Ananitya
Quoted 2 lines
tglx
Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of those who have achieved perfection, hardly one knows Me in truth. -- Gita Sutra Of Mysticism
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Anant Nitya
Quoted 5 lines
No idea. I uploaded a debug patch against 2.6.22-rc1 to
http://www.tglx.de/private/tglx/2.6.22-rc1-hrt-debug.patch
Can you give it a try and report the output ?
Hi
Here it goes[ 159.646196] NOHZ softirq pending 22 on CPU 0 [ 159.646207] .... task state: 1 00000000
Quoted 5 lines
1 ==
TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.I work out a more complex debug patch and pester you to test once I'm done.
I've also tons of 'NOHZ: local_softirq_pending 08' that disappear with nohz=off. But I'm still running 2.6.21. Are there any patches that should fix this?
Machine is a Lenovo T60p:
i686 Intel(R) Core(TM)2 CPU T7600 @ 2.33GHz GenuineIntel GNU/Linux
Besides that I get a lot of clock skews on 'make headers_check', but these are unrelated to nohz:
CHECK include/asm/dasd.h
make[2]: warning: Clock skew detected. Your build may be incomplete. make[2]: Warning: File `/dev/null' has modification time 5.1e+03 s in the future
Quoted 9 lines · Heiko Carstens
Quoted 2 lines
I work out a more complex debug patch and pester you to test once I'm done.
I've also tons of 'NOHZ:
local_softirq_pending08' that disappear with nohz=off. But I'm still running 2.6.21. Are there any patches that should fix this?
Machine is a Lenovo T60p:i686 Intel(R) Core(TM)2 CPU T7600 @ 2.33GHz GenuineIntel GNU/Linux
Hmm, that's a different problem than the 0x22 which shows up on hyperthreading enabled P4 systems. Are you using plip ?
Quoted 6 lines
Besides that I get a lot of clock skews on 'make
headers_check', but these are unrelated to nohz:CHECK
include/asm/dasd.h
make[2]: warning: Clock skew detected. Your build may be incomplete. make[2]: Warning: File `/dev/null' has modification time 5.1e+03 s in the future
Strange.
tglx
Quoted 9 lines · Thomas Gleixner
Quoted 8 lines · Heiko Carstens
Quoted 2 lines
I work out a more complex debug patch and pester you to test once I'm done.
I've also tons of 'NOHZ:
local_softirq_pending08' that disappear with nohz=off. But I'm still running 2.6.21. Are there any patches that should fix this?
Machine is a Lenovo T60p:
Quoted 4 lines
Quoted 1 line
i686 Intel(R) Core(TM)2 CPU T7600 @ 2.33GHz GenuineIntel GNU/Linux
Hmm, that's a different problem than the
0x22which shows up on hyperthreading enabled P4 systems. Are you using plip ?
No, after all it turned out that is caused by an IBM internal module. Just ignore me, sorry for the noise :)
Quoted 6 lines · Anant Nitya
Quoted 5 lines
1 ==
TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.I work out a more complex debug patch and pester you to test once I'm done.
No problem :)
You asked for it :)
Please patch 2.6.22-rc2 with
http://tglx.de/projects/hrtimers/2.6.22-rc2/patch-2.6.22-rc2-hrt2.patch
and
http://www.tglx.de/private/tglx/ht-debug/tracer.diff
Compile it with the config
http://www.tglx.de/private/tglx/ht-debug/config.debug
You should find something like:
( swapper-0 |#0): new 67173 us user-latency.
along with the familiar "NOHZ ......" message in your log file.
Once that happened please do:
$ cat /proc/latency_trace >trace.txt
compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.
Michal,
IIRC you encountered the same P4/HT related wreckage. Can you do the same ?
Thanks,
tglx
Quoted 25 lines · On Monday 21 May 2007 03:13:08 Thomas Gleixner wrote:
Quoted 7 lines · Anant Nitya
Quoted 5 lines
1 ==
TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.I work out a more complex debug patch and pester you to test once I'm done.
No problem :)
You asked for it :)
Please patch 2.6.22-rc2 with
http://tglx.de/projects/hrtimers/2.6.22-rc2/patch-2.6.22-rc2-hrt2.patch
and
http://www.tglx.de/private/tglx/ht-debug/tracer.diffCompile it with the config
http://www.tglx.de/private/tglx/ht-debug/config.debug
You should find something like:
( swapper-0 |#0): new 67173 us user-latency.
Quoted 8 lines
along with the familiar "NOHZ ......" message in your log file.
Once that happened please do:
$ cat /proc/latency_trace >trace.txtcompress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.
Hi Thomas
Here are the links...
http://cybertek.info/taitai/dmesg-2.6.22.rc2.hrt2-1.SMP.DN.LINUX.txt
http://cybertek.info/taitai/trace.txt.bz2
Regards
Ananitya
Quoted 9 lines
Michal,
IIRC you encountered the same P4/HT related wreckage. Can you do the same ?
Thanks,
tglx
Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of those who have achieved perfection, hardly one knows Me in truth. -- Gita Sutra Of Mysticism
Quoted 10 lines · Anant Nitya
Quoted 10 lines
You should find something like:
( swapper-0 |#0): new 67173 us user-latency.
along with the familiar "NOHZ ......" message in your log file.
Once that happened please do:
$ cat /proc/latency_trace >trace.txt
Quoted 7 lines
Quoted 2 lines
compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.
Hi Thomas
Here are the links...
http://cybertek.info/taitai/dmesg-2.6.22.rc2.hrt2-1.SMP.DN.LINUX.txt
http://cybertek.info/taitai/trace.txt.bz2
Thanks. Sorry, I need more info. I uploaded a new tracer.diff to
http://www.tglx.de/private/tglx/ht-debug/tracer.diff
Can you please revert the first one and retest with the new one ?
Thanks,
tglx
Quoted 20 lines · On Monday 21 May 2007 14:31:57 Thomas Gleixner wrote:
Quoted 10 lines · Anant Nitya
Quoted 10 lines
You should find something like:
( swapper-0 |#0): new 67173 us user-latency.
along with the familiar "NOHZ ......" message in your log file.
Once that happened please do:
$ cat /proc/latency_trace >trace.txtQuoted 8 lines
Quoted 2 lines
compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.
Hi Thomas
Here are the links...
http://cybertek.info/taitai/dmesg-2.6.22.rc2.hrt2-1.SMP.DN.LINUX.txt
http://cybertek.info/taitai/trace.txt.bz2
Quoted 5 lines
Thanks. Sorry, I need more info. I uploaded a new tracer.diff to
http://www.tglx.de/private/tglx/ht-debug/tracer.diff
Can you please revert the first one and retest with the new one ?
Okay sure, compiling now.
Regards
Ananitya
Quoted 4 lines
Thanks,
tglx
Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of those who have achieved perfection, hardly one knows Me in truth. -- Gita Sutra Of Mysticism
Quoted 20 lines · On Monday 21 May 2007 14:31:57 Thomas Gleixner wrote:
Quoted 10 lines · Anant Nitya
Quoted 10 lines
You should find something like:
( swapper-0 |#0): new 67173 us user-latency.
along with the familiar "NOHZ ......" message in your log file.
Once that happened please do:
$ cat /proc/latency_trace >trace.txtQuoted 8 lines
Quoted 2 lines
compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.
Hi Thomas
Here are the links...
http://cybertek.info/taitai/dmesg-2.6.22.rc2.hrt2-1.SMP.DN.LINUX.txt
http://cybertek.info/taitai/trace.txt.bz2
Quoted 7 lines
Thanks. Sorry, I need more info. I uploaded a new tracer.diff to
http://www.tglx.de/private/tglx/ht-debug/tracer.diff
Can you please revert the first one and retest with the new one ?
Thanks,
Sorry for delay, here is link for output from new tacer.
http://cybertek.info/taitai/trace-new.txt.bz2
Regards
Ananitya
Quoted 2 lines
tglx
Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of those who have achieved perfection, hardly one knows Me in truth. -- Gita Sutra Of Mysticism
Hi Thomas,
Quoted 30 lines · Thomas Gleixner
Quoted 6 lines · Anant Nitya
Quoted 5 lines
1 ==
TASK_INTERRUPTIBLE, so we know that ksoftirqd was not woken up. At least it is not a scheduler problem.I work out a more complex debug patch and pester you to test once I'm done.
No problem :)
You asked for it :)
Please patch 2.6.22-rc2 with
http://tglx.de/projects/hrtimers/2.6.22-rc2/patch-2.6.22-rc2-hrt2.patch
and
http://www.tglx.de/private/tglx/ht-debug/tracer.diffCompile it with the config
http://www.tglx.de/private/tglx/ht-debug/config.debug
You should find something like:
( swapper-0 |#0): new 67173 us user-latency.
along with the familiar "NOHZ ......" message in your log file.
Once that happened please do:
$ cat /proc/latency_trace >trace.txt
Quoted 7 lines
compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.
Michal,
IIRC you encountered the same P4/HT related wreckage. Can you do the same ?
Good news - I can't reproduce this bug. It's time to remove
"Subject : 2.6.21-git4 BUG: soft lockup detected on CPU#1!
References : http://lkml.org/lkml/2007/5/2/511
Submitter : Michal Piotrowski <[email protected]>
Status : problem is being debugged"
from KR.
Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq works
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrt-config
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrt-console.log
Now I'm trying to apply hrtimers debug patch ontop 2.6.22-rc2-hrt3
http://lkml.org/lkml/2007/3/23/106
Regards,
Michal
Signature · 3 lines
Michal K. K. Piotrowski Kernel Monkeys (http://kernel.wikidot.com/start)
Michal Piotrowski napisał(a):
Quoted 54 lines
Hi Thomas,
Quoted 31 lines · Thomas Gleixner
Quoted 1 line · Anant Nitya
Quoted 1 line
1 ==
TASK_INTERRUPTIBLE, so we know that ksoftirqd was not wokenup. At
Quoted 5 lines
Quoted 4 lines
least it is not a scheduler problem.
I work out a more complex debug patch and pester you to test once I'm done.
No problem :)
You asked for it :)
Please patch 2.6.22-rc2 with
http://tglx.de/projects/hrtimers/2.6.22-rc2/patch-2.6.22-rc2-hrt2.patch
and
http://www.tglx.de/private/tglx/ht-debug/tracer.diffCompile it with the config
http://www.tglx.de/private/tglx/ht-debug/config.debug
You should find something like:
( swapper-0 |#0): new 67173 us user-latency.
along with the familiar "NOHZ ......" message in your log file.
Once that happened please do:
$ cat /proc/latency_trace >trace.txtQuoted 7 lines
compress it and send it to me along with the full dmesg output or put both up to some place, where I can download it.
Michal,
IIRC you encountered the same P4/HT related wreckage. Can you do the same ?
Good news - I can't reproduce this bug. It's time to remove
"Subject : 2.6.21-git4 BUG: soft lockup detected on CPU#1!
References : http://lkml.org/lkml/2007/5/2/511
Submitter : Michal Piotrowski <[email protected]>Handled-By Thomas Gleixner <[email protected]>Status : problem is being debugged"
from KR.Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq works
Quoted 8 lines
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrt-config
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrt-console.log
Now I'm trying to apply hrtimers debug patch ontop 2.6.22-rc2-hrt3
http://lkml.org/lkml/2007/3/23/106
It _almost_ works
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.22-rc2-hrt3/hrtimers_debug.patch
Hmmm..
[ 135.206505] SysRq : Show Pending Timers
[ 135.210476] Timer List Version: v0.3
[ 135.214102] HRTIMER_MAX_CLOCK_BASES: 2
[ 135.217899] now at 112955954742 nsecs
[ 135.221651]
[ 135.221662] cpu: 0
[ 135.225246] clock 0:
[ 135.227588] .index: 0
[ 135.230545] .resolution: 1 nsecs
[ 135.234010] .get_time: ktime_get_real
[ 135.238151] .offset: 1179862680097808665 nsecs
[ 135.243207] active timers:
[ 135.245992] clock 1:
[ 135.248316] .index: 1
[ 135.251246] .resolution: 1 nsecs
[ 135.254722] .get_time: ktime_get
[ 135.258473] .offset: 0 nsecs
[ 135.261923] active timers:
[ 135.264683] #0: hardirq_stack, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
[ 135.273913] # expires at 112952000000 nsecs [in 18446744073705596874 nsecs]
[ 135.281194] #1: hardirq_stack, hrtimer_wakeup, S:01, futex_wait, automount/2293
[ 135.289057] # expires at 113902202172 nsecs [in 946247430 nsecs]
[ 135.295233] #2: hardirq_stack, hrtimer_wakeup, S:01, do_nanosleep, crond/2373
[ 135.302973] # expires at 121028637388 nsecs [in 8072682646 nsecs]
[ 135.309279] #3: hardirq_stack, it_real_fn, S:01, do_setitimer, syslogd/2171
[ 135.316701] # expires at 121465401717 nsecs [in 8509446975 nsecs]
[ 135.323035] .expires_next : 112952000000 nsecs
[ 135.327921] .exp_prev : 112951000000 nsecs
[ 135.332786] last expires_next stacktrace:
[ 135.336944] update_cpu_base_expires_next
[ 135.341188] hrtimer_interrupt
[ 135.344430] smp_apic_timer_interrupt
[ 135.348310] apic_timer_interrupt
[ 135.351845] __mcount
[ 135.354327] mcount
[ 135.356627] permission
[ 135.359298] vfs_permission
[ 135.362305] __link_path_walk
[ 135.365497] link_path_walk
[ 135.368486] path_walk
[ 135.371044] do_path_lookup
[ 135.374086] __user_walk_fd
[ 135.377122] vfs_stat_fd
[ 135.379905] vfs_stat
[ 135.382385] sys_stat64
[ 135.385065] syscall_call
[ 135.387909] <3>BUG: sleeping function called from invalid context at kernel/mutex.c:86
[ 135.396175] in_atomic():1, irqs_disabled():1
[ 135.400541] 2 locks held by readahead/2442:
[ 135.404783] #0: (&serio->lock){++..}, at: [<c02dbf17>] serio_interrupt+0x22/0x79
[ 135.412623] #1: (sysrq_key_table_lock){+...}, at: [<c027a4af>] __handle_sysrq+0x1b/0x10c
[ 135.421208] irq event stamp: 3249062
[ 135.424856] hardirqs last enabled at (3249061): [<c0181b67>] kmem_cache_free+0x97/0xa1
[ 135.433050] hardirqs last disabled at (3249062): [<c0104bd4>] common_interrupt+0x24/0x34
[ 135.441350] softirqs last enabled at (3248980): [<c012a418>] __do_softirq+0xe8/0xef
[ 135.449261] softirqs last disabled at (3248975): [<c0106df7>] do_softirq+0x69/0xd6
[ 135.457049] [<c010526c>] show_trace_log_lvl+0x35/0x54
[ 135.462312] [<c0106027>] show_trace+0x2c/0x2e
[ 135.466876] [<c01060d0>] dump_stack+0x29/0x2b
[ 135.471423] [<c011de0c>] __might_sleep+0xe0/0xe2
[ 135.476221] [<c0360257>] mutex_lock+0x22/0x2e
[ 135.480767] [<c014eb96>] lookup_module_symbol_name+0x1a/0xc4
[ 135.486739] [<c014f514>] lookup_symbol_name+0x62/0x69
[ 135.492021] [<c013eb45>] print_name_offset+0x29/0x81
[ 135.497200] [<c013f4c5>] timer_list_show+0x657/0xc55
[ 135.502368] [<c013fae2>] sysrq_timer_list_show+0x1f/0x21
[ 135.507859] [<c027a5fd>] sysrq_handle_show_timers+0xd/0xf
[ 135.513451] [<c027a525>] __handle_sysrq+0x91/0x10c
[ 135.518507] [<c027a79e>] handle_sysrq+0x37/0x39
[ 135.523296] [<c0274b00>] kbd_event+0x335/0x58d
[ 135.527921] [<c02df806>] input_event+0x43e/0x460
[ 135.532718] [<c02e3766>] atkbd_interrupt+0x467/0x58d
[ 135.537887] [<c02dbf37>] serio_interrupt+0x42/0x79
[ 135.542865] [<c02dceed>] i8042_interrupt+0x1ff/0x210
[ 135.548130] [<c0160963>] handle_IRQ_event+0x24/0x54
[ 135.553187] [<c0161b2a>] handle_edge_irq+0xd7/0x11f
[ 135.558253] [<c0106f46>] do_IRQ+0xe2/0x10c
[ 135.562544] [<c0104bde>] common_interrupt+0x2e/0x34
[ 135.567675] [<c0116134>] mcount+0x14/0x18
[ 135.571858] [<c019b9ab>] mntput_no_expire+0xd/0x89
[ 135.576845] [<c018cd6a>] path_release+0x2f/0x33
[ 135.581645] [<c01893f4>] vfs_stat_fd+0x4c/0x55
[ 135.586276] [<c01894c9>] vfs_stat+0x25/0x27
[ 135.590660] [<c01894e9>] sys_stat64+0x1e/0x37
[ 135.595215] [<c01041f5>] syscall_call+0x7/0xb
[ 135.599752] =======================
[ 135.603365] ---------------------------
[ 135.607229] | preempt count: 00010000 ]
[ 135.611146] | 0-level deep critical section nesting:
[ 135.616202] ----------------------------------------
[ 135.621258]
[ 135.795930] ( readahead-2442 |#0): new 112512428 us user-latency.
[ 135.802591] stopped custom tracer.
[ 135.806058] BUG: at kernel/mutex.c:132 __mutex_lock_common()
[ 135.811824] [<c010526c>] show_trace_log_lvl+0x35/0x54
[ 135.817035] [<c0106027>] show_trace+0x2c/0x2e
[ 135.821557] [<c01060d0>] dump_stack+0x29/0x2b
[ 135.826078] [<c035fff2>] __mutex_lock_slowpath+0x64/0x2a7
[ 135.831714] [<c036025e>] mutex_lock+0x29/0x2e
[ 135.836277] [<c014eb96>] lookup_module_symbol_name+0x1a/0xc4
[ 135.842121] [<c014f514>] lookup_symbol_name+0x62/0x69
[ 135.847342] [<c013eb45>] print_name_offset+0x29/0x81
[ 135.852478] [<c013f4c5>] timer_list_show+0x657/0xc55
[ 135.857611] [<c013fae2>] sysrq_timer_list_show+0x1f/0x21
[ 135.863128] [<c027a5fd>] sysrq_handle_show_timers+0xd/0xf
[ 135.868744] [<c027a525>] __handle_sysrq+0x91/0x10c
[ 135.873689] [<c027a79e>] handle_sysrq+0x37/0x39
[ 135.878383] [<c0274b00>] kbd_event+0x335/0x58d
[ 135.882991] [<c02df806>] input_event+0x43e/0x460
[ 135.887806] [<c02e3766>] atkbd_interrupt+0x467/0x58d
[ 135.892948] [<c02dbf37>] serio_interrupt+0x42/0x79
[ 135.897918] [<c02dceed>] i8042_interrupt+0x1ff/0x210
[ 135.903053] [<c0160963>] handle_IRQ_event+0x24/0x54
[ 135.908120] [<c0161b2a>] handle_edge_irq+0xd7/0x11f
[ 135.913176] [<c0106f46>] do_IRQ+0xe2/0x10c
[ 135.917449] [<c0104bde>] common_interrupt+0x2e/0x34
[ 135.922502] [<c0116134>] mcount+0x14/0x18
[ 135.926696] [<c019b9ab>] mntput_no_expire+0xd/0x89
[ 135.931657] [<c018cd6a>] path_release+0x2f/0x33
[ 135.936378] [<c01893f4>] vfs_stat_fd+0x4c/0x55
[ 135.941001] [<c01894c9>] vfs_stat+0x25/0x27
[ 135.945376] [<c01894e9>] sys_stat64+0x1e/0x37
[ 135.949931] [<c01041f5>] syscall_call+0x7/0xb
[ 135.954467] =======================
[ 135.958091] ---------------------------
[ 135.962015] | preempt count: 00010000 ]
[ 135.965905] | 0-level deep critical section nesting:
[ 135.970927] ----------------------------------------
[ 135.975931]
[ 135.977468] <ffffffff>
[ 135.979865]
[ 135.981396] .hres_active : 1
[ 135.984757] .nr_events : 91663
[ 135.988465] .nohz_mode : 2
[ 135.991820] .idle_tick : 112686000000 nsecs
[ 135.996651] .tick_stopped : 0
[ 136.000016] .idle_jiffies : 4294779982
[ 136.004190] .idle_calls : 30792
[ 136.007882] .idle_sleeps : 3020
[ 136.011486] .idle_entrytime : 112932690671 nsecs
[ 136.016335] .idle_sleeptime : 18007304053 nsecs
[ 136.021063] .last_jiffies : 4294780229
[ 136.025220] .next_jiffies : 4294780556
[ 136.029352] .idle_expires : 112855000000 nsecs
[ 136.034167] jiffies: 4294780248
[ 136.037331]
[ 136.037336] cpu: 1
[ 136.040884] clock 0:
[ 136.043182] .index: 0
[ 136.046087] .resolution: 1 nsecs
[ 136.049535] .get_time: ktime_get_real
[ 136.053739] .offset: 1179862680097808665 nsecs
[ 136.058751] active timers:
[ 136.061501] clock 1:
[ 136.063877] .index: 1
[ 136.066781] .resolution: 1 nsecs
[ 136.070229] .get_time: ktime_get
[ 136.073922] .offset: 0 nsecs
[ 136.077465] active timers:
[ 136.080293] #0: hardirq_stack, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
[ 136.089309] # expires at 113828250000 nsecs [in 872295258 nsecs]
[ 136.095654] #1: hardirq_stack, hrtimer_wakeup, S:01, futex_wait, automount/2294
[ 136.103356] # expires at 113902196736 nsecs [in 946241994 nsecs]
[ 136.109485] #2: hardirq_stack, hrtimer_wakeup, S:01, do_nanosleep, smartd/2640
[ 136.117101] # expires at 1897330250564 nsecs [in 1784374295822 nsecs]
[ 136.123723] .expires_next : 113871250000 nsecs
[ 136.128726] .exp_prev : 113875250000 nsecs
[ 136.133558] last expires_next stacktrace:
[ 136.137662] update_cpu_base_expires_next
[ 136.141838] hrtimer_interrupt
[ 136.145071] smp_apic_timer_interrupt
[ 136.148935] apic_timer_interrupt
[ 136.152419] cpu_idle
[ 136.154890] start_secondary
[ 136.157967] <00000000>
[ 136.160639] <ffffffff>
[ 136.163267]
[ 136.164789] .hres_active : 1
[ 136.168143] .nr_events : 85123
[ 136.171981] .nohz_mode : 2
[ 136.175344] .idle_tick : 112682250000 nsecs
[ 136.180281] .tick_stopped : 0
[ 136.183737] .idle_jiffies : 4294779978
[ 136.187974] .idle_calls : 44643
[ 136.191708] .idle_sleeps : 1384
[ 136.195424] .idle_entrytime : 113942257992 nsecs
[ 136.200421] .idle_sleeptime : 18596870874 nsecs
[ 136.205158] .last_jiffies : 4294780248
[ 136.209316] .next_jiffies : 4294780261
[ 136.213595] .idle_expires : 112778000000 nsecs
[ 136.218451] jiffies: 4294780248
[ 136.221650]
[ 136.223163]
[ 136.223168] Tick Device: mode: 1
[ 136.228383] Clock Event Device: pit
[ 136.231928] max_delta_ns: 27461866
[ 136.235636] min_delta_ns: 12571
[ 136.239172] mult: 5124677
[ 136.242769] shift: 32
[ 136.245965] mode: 3
[ 136.249095] next_event: 9223372036854775807 nsecs
[ 136.254282] set_next_event: pit_next_event
[ 136.258613] set_mode: init_pit_timer
[ 136.262908] event_handler: tick_handle_oneshot_broadcast
[ 136.268510] tick_broadcast_mask: 00000000
[ 136.272583] tick_broadcast_oneshot_mask: 00000000
[ 136.277310]
[ 136.278840]
[ 136.278845] Tick Device: mode: 1
[ 136.284086] Clock Event Device: lapic
[ 136.287803] max_delta_ns: 671411430
[ 136.291615] min_delta_ns: 1200
[ 136.294977] mult: 53661274
[ 136.298668] shift: 32
[ 136.301902] mode: 3
[ 136.304971] next_event: 112952000000 nsecs
[ 136.309638] set_next_event: lapic_next_event
[ 136.314177] set_mode: lapic_timer_setup
[ 136.318680] event_handler: hrtimer_interrupt
[ 136.323236]
[ 136.323241] Tick Device: mode: 1
[ 136.328404] Clock Event Device: lapic
[ 136.332191] max_delta_ns: 671411430
[ 136.335977] min_delta_ns: 1200
[ 136.339367] mult: 53661274
[ 136.343075] shift: 32
[ 136.346268] mode: 3
[ 136.349375] next_event: 114098250000 nsecs
[ 136.353914] set_next_event: lapic_next_event
[ 136.358357] set_mode: lapic_timer_setup
[ 136.362973] event_handler: hrtimer_interruptRegards,
Michal
Signature · 3 lines
Michal K. K. Piotrowski Kernel Monkeys (http://kernel.wikidot.com/start)
Quoted 3 lines · Michal Piotrowski
Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq works
please try the patch below! I think we have nailed this bug.
Ingo
kernel/sched.c +1 -3
@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)BUG_ON (!in_softirq ());if (need_resched () && system_state ==SYSTEM_RUNNING ) { -raw_local_irq_disable (); -_local_bh_enable (); -raw_local_irq_enable (); +local_bh_enable ();__cond_resched ();local_bh_disable ();return 1 ;
Quoted 8 lines · Ingo Molnar
Quoted 3 lines · Michal Piotrowski
Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq worksplease try the patch below! I think we have nailed this bug.
I picked it from -hrt5 and so far so good.
Regards,
Michal
Signature · 3 lines
Michal K. K. Piotrowski Kernel Monkeys (http://kernel.wikidot.com/start)
Quoted 15 lines · Ingo Molnar
Quoted 3 lines · Michal Piotrowski
Bad news - I hit a bug in 2.6.22-rc2-hrt3. Bug symptoms:
- X hangs (keyboard, mouse, sound etc.)
- only Magic SysRq worksplease try the patch below! I think we have nailed this bug.
Ingo
Index:
linux/kernel/sched.c
===================================================================
kernel/sched.c int __sched cond_resched_softirq(void) · line 4218
Quoted 10 lines
kernel/sched.c +1
-3@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)BUG_ON (!in_softirq ());if (need_resched () && system_state ==SYSTEM_RUNNING ) { -raw_local_irq_disable (); -_local_bh_enable (); -raw_local_irq_enable (); +local_bh_enable ();__cond_resched ();local_bh_disable ();return 1 ;
We may have a problem with that:
BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted) [<c042b2ef>] local_bh_enable+0x45/0x92 [<c06036b7>] cond_resched_softirq+0x2c/0x42 [<c059d5d0>] release_sock+0x54/0xa3 [<c05c9428>] tcp_sendmsg+0x91b/0xa0c [<c05e1bb9>] inet_sendmsg+0x3b/0x45 [<c059af34>] sock_aio_write+0xf9/0x105 [<c0476035>] do_sync_write+0xc7/0x10a [<c0437265>] autoremove_wake_function+0x0/0x35 [<c047688e>] vfs_write+0xbc/0x154 [<c0476e8c>] sys_write+0x41/0x67 [<c0404f70>] syscall_call+0x7/0xb
That's:
WARN_ON_ONCE(irqs_disabled());
And another, tainted but probably valid:
BUG: warning at kernel/softirq.c:138/local_bh_enable() (Tainted: P ) [<c042b2ef>] local_bh_enable+0x45/0x92 [<c06036b7>] cond_resched_softirq+0x2c/0x42 [<c059d5d0>] release_sock+0x54/0xa3 [<c05c9428>] tcp_sendmsg+0x91b/0xa0c [<c04e8b30>] copy_to_user+0x3c/0x50 [<c05a1b71>] memcpy_toiovec+0x27/0x4a [<c059d58f>] release_sock+0x13/0xa3 [<c0604ad5>] _spin_unlock_bh+0x5/0xd [<c05e1bb9>] inet_sendmsg+0x3b/0x45 [<c059af34>] sock_aio_write+0xf9/0x105 [<c0475f31>] do_sync_readv_writev+0xc1/0xfe [<c0437265>] autoremove_wake_function+0x0/0x35 [<c04e88e0>] copy_from_user+0x3a/0x66 [<c0475dec>] rw_copy_check_uvector+0x5c/0xb0 [<c047667a>] do_readv_writev+0xbc/0x187 [<c059ae3b>] sock_aio_write+0x0/0x105 [<c0476782>] vfs_writev+0x3d/0x48 [<c0476bee>] sys_writev+0x41/0x95 [<c0404f70>] syscall_call+0x7/0xb
Quoted 2 lines · Chuck Ebbert
Another; these started to appear after the below patch was merged:
Quoted 7 lines
Index:
linux/kernel/sched.cQuoted 6 lines
===================================================================
kernel/sched.c +0
-0@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)BUG_ON (!in_softirq ());
Quoted 8 lines
kernel/sched.c int __sched cond_resched_softirq(void) · line 4218
Quoted 8 lines
kernel/sched.c +1
-3if (need_resched () && system_state ==SYSTEM_RUNNING ) { -raw_local_irq_disable (); -_local_bh_enable (); -raw_local_irq_enable (); +local_bh_enable ();__cond_resched ();local_bh_disable ();return 1 ;
May 23 19:26:26 localhost kernel: BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted) May 23 19:26:26 localhost kernel: [<c042b2ef>] local_bh_enable+0x45/0x92 May 23 19:26:26 localhost kernel: [<c06036b7>] cond_resched_softirq+0x2c/0x42 May 23 19:26:26 localhost kernel: [<c059d5d0>] release_sock+0x54/0xa3 May 23 19:26:26 localhost kernel: [<c04373af>] prepare_to_wait+0x24/0x3f May 23 19:26:26 localhost kernel: [<c05e267f>] inet_stream_connect+0x116/0x1ff May 23 19:26:26 localhost kernel: [<c0437265>] autoremove_wake_function+0x0/0x35 May 23 19:26:26 localhost kernel: [<c059c339>] sys_connect+0x82/0xad May 23 19:26:26 localhost kernel: [<c059d58f>] release_sock+0x13/0xa3 May 23 19:26:26 localhost kernel: [<c0604ad5>] _spin_unlock_bh+0x5/0xd May 23 19:26:26 localhost kernel: [<c059e714>] sock_setsockopt+0x4a8/0x4b2 May 23 19:26:26 localhost kernel: [<c059b6b6>] sock_attach_fd+0x70/0xd2 May 23 19:26:26 localhost kernel: [<c04774a0>] get_empty_filp+0xfc/0x170 May 23 19:26:26 localhost kernel: [<c059b54f>] sys_setsockopt+0x9b/0xa7 May 23 19:26:26 localhost kernel: [<c059cb83>] sys_socketcall+0xac/0x261 May 23 19:26:26 localhost kernel: [<c0404f70>] syscall_call+0x7/0xb
Quoted 12 lines · Heiko Carstens
Quoted 11 lines · Thomas Gleixner
Quoted 9 lines · Heiko Carstens
Quoted 5 lines
Quoted 2 lines
Hmm, that's a different problem than the
0x22which shows up on hyperthreading enabled P4 systems. Are you using plip ?No, after all it turned out that is caused by an IBM internal module. Just ignore me, sorry for the noise :)
Just in case you're interested: I just looked into it and it turned out that this module was calling
netif_rx()from process context, just like plip. Changing it tonetif_rx_ni()fixes it.
Quoted 5 lines
Quoted 2 lines
Thanks, that's a good pointer. You should have cc'ed Dave Miller and the guy who found this plip thingy. Care to resend ?
No problem. Here we go, even though I doubt that
netif_rx_ni()is what people want for plip. Dunno...
netif_rx_ni calls softirq synchronously, am I right? That would probably hurt plip. I can try it if someone will be interested in it, but I doubt it is good solution for any network interfaces except tun.
BTW. does in_interrupt() have high cost on some high-performance architectures? Would it be possible to just add if (!in_interrupt()) wakeup_softirqd(); to netif_rx()?
Mikulas
Quoted 31 lines · On Thursday 24 May 2007 00:08:40 Chuck Ebbert wrote:
Quoted 1 line · Chuck Ebbert
Another; these started to appear after the below patch was merged:
Quoted 8 lines
Index:
linux/kernel/sched.cQuoted 6 lines
===================================================================
kernel/sched.c +0
-0@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)BUG_ON (!in_softirq ());Quoted 8 lines
kernel/sched.c int __sched cond_resched_softirq(void) · line 4218
Quoted 8 lines
kernel/sched.c +1
-3if (need_resched () && system_state ==SYSTEM_RUNNING ) { -raw_local_irq_disable (); -_local_bh_enable (); -raw_local_irq_enable (); +local_bh_enable ();__cond_resched ();local_bh_disable ();return 1 ;May 23 19:26:26 localhost kernel: BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted) May 23 19:26:26 localhost kernel: [<c042b2ef>] local_bh_enable+0x45/0x92 May 23 19:26:26 localhost kernel: [<c06036b7>] cond_resched_softirq+0x2c/0x42 May 23 19:26:26 localhost kernel: [<c059d5d0>] release_sock+0x54/0xa3 May 23 19:26:26 localhost kernel: [<c04373af>] prepare_to_wait+0x24/0x3f May 23 19:26:26 localhost kernel: [<c05e267f>] inet_stream_connect+0x116/0x1ff May 23 19:26:26 localhost kernel: [<c0437265>] autoremove_wake_function+0x0/0x35 May 23 19:26:26 localhost kernel: [<c059c339>] sys_connect+0x82/0xad May 23 19:26:26 localhost kernel:
Quoted 8 lines
[<c059d58f>] release_sock+0x13/0xa3 May 23 19:26:26 localhost kernel: [<c0604ad5>] _spin_unlock_bh+0x5/0xd May 23 19:26:26 localhost kernel: [<c059e714>] sock_setsockopt+0x4a8/0x4b2 May 23 19:26:26 localhost kernel: [<c059b6b6>] sock_attach_fd+0x70/0xd2 May 23 19:26:26 localhost kernel: [<c04774a0>] get_empty_filp+0xfc/0x170 May 23 19:26:26 localhost kernel: [<c059b54f>] sys_setsockopt+0x9b/0xa7 May 23 19:26:26 localhost kernel: [<c059cb83>] sys_socketcall+0xac/0x261 May 23 19:26:26 localhost kernel: [<c0404f70>] syscall_call+0x7/0xb
strange, while applying the concerned patch first time I was hand editing __ kernel/sched.c __ and stupidly typed _local_bh_enable() instead of local_bh_enable() and when did a reboot, as soon as I got inside X I was welcomed with following message in system log.
[ 152.692609] BUG: at kernel/softirq.c:122 _local_bh_enable() [ 152.692637] [<b040624d>] show_trace_log_lvl+0x1a/0x2f [ 152.692658] [<b0406801>] show_trace+0x12/0x14 [ 152.692668] [<b0406885>] dump_stack+0x16/0x18 [ 152.692678] [<b0428d66>] _local_bh_enable+0x8b/0xc3 [ 152.692688] [<b059e497>] cond_resched_softirq+0x2b/0x40 [ 152.692700] [<b05797b6>] established_get_first+0x19/0xad [ 152.692712] [<b057a831>] tcp_seq_next+0x76/0x8c [ 152.692722] [<b04875ef>] seq_read+0x17b/0x264 [ 152.692733] [<b047102a>] vfs_read+0xad/0x161 [ 152.692745] [<b04714b6>] sys_read+0x3d/0x61 [ 152.692755] [<b04050d4>] syscall_call+0x7/0xb [ 152.692765] ======================= [ 152.692770] BUG: at kernel/lockdep.c:1937 trace_softirqs_on() [ 152.692777] [<b040624d>] show_trace_log_lvl+0x1a/0x2f [ 152.692789] [<b0406801>] show_trace+0x12/0x14 [ 152.692800] [<b0406885>] dump_stack+0x16/0x18 [ 152.692810] [<b043f5a5>] trace_softirqs_on+0x5f/0xa5 [ 152.692822] [<b0428d8e>] _local_bh_enable+0xb3/0xc3 [ 152.692831] [<b059e497>] cond_resched_softirq+0x2b/0x40 [ 152.692842] [<b05797b6>] established_get_first+0x19/0xad [ 152.692852] [<b057a831>] tcp_seq_next+0x76/0x8c [ 152.692862] [<b04875ef>] seq_read+0x17b/0x264 [ 152.692870] [<b047102a>] vfs_read+0xad/0x161 [ 152.692879] [<b04714b6>] sys_read+0x3d/0x61 [ 152.692889] [<b04050d4>] syscall_call+0x7/0xb [ 152.692899] ======================= [ 159.257890] NOHZ: local_softirq_pending 22 [ 159.266009] NOHZ: local_softirq_pending 22 [ 159.273965] NOHZ: local_softirq_pending 22 [ 159.281884] NOHZ: local_softirq_pending 22 [ 160.712828] NOHZ: local_softirq_pending 22 [ 162.609377] NOHZ: local_softirq_pending 22 [ 162.609804] NOHZ: local_softirq_pending 22 [ 162.610054] NOHZ: local_softirq_pending 22 [ 162.610279] NOHZ: local_softirq_pending 22 [ 162.610502] NOHZ: local_softirq_pending 22
After realzing my mistake, I changed it to local_bh_enable() as was in patch and since then not a single BUG or local_softirq_pending message occurs in system log, maybe my system waiting for that condition to happen :).
Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of those who have achieved perfection, hardly one knows Me in truth. -- Gita Sutra Of Mysticism
Quoted 31 lines · On Thursday 24 May 2007 00:08:40 Chuck Ebbert wrote:
Quoted 1 line · Chuck Ebbert
Another; these started to appear after the below patch was merged:
Quoted 8 lines
Index:
linux/kernel/sched.cQuoted 6 lines
===================================================================
kernel/sched.c +0
-0@@ -4212,9 +4212,7 @@ int __sched cond_resched_softirq(void)BUG_ON (!in_softirq ());Quoted 8 lines
kernel/sched.c int __sched cond_resched_softirq(void) · line 4218
Quoted 8 lines
kernel/sched.c +1
-3if (need_resched () && system_state ==SYSTEM_RUNNING ) { -raw_local_irq_disable (); -_local_bh_enable (); -raw_local_irq_enable (); +local_bh_enable ();__cond_resched ();local_bh_disable ();return 1 ;May 23 19:26:26 localhost kernel: BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted) May 23 19:26:26 localhost kernel: [<c042b2ef>] local_bh_enable+0x45/0x92 May 23 19:26:26 localhost kernel: [<c06036b7>] cond_resched_softirq+0x2c/0x42 May 23 19:26:26 localhost kernel: [<c059d5d0>] release_sock+0x54/0xa3 May 23 19:26:26 localhost kernel: [<c04373af>] prepare_to_wait+0x24/0x3f May 23 19:26:26 localhost kernel: [<c05e267f>] inet_stream_connect+0x116/0x1ff May 23 19:26:26 localhost kernel: [<c0437265>] autoremove_wake_function+0x0/0x35 May 23 19:26:26 localhost kernel: [<c059c339>] sys_connect+0x82/0xad May 23 19:26:26 localhost kernel:
Quoted 8 lines
[<c059d58f>] release_sock+0x13/0xa3 May 23 19:26:26 localhost kernel: [<c0604ad5>] _spin_unlock_bh+0x5/0xd May 23 19:26:26 localhost kernel: [<c059e714>] sock_setsockopt+0x4a8/0x4b2 May 23 19:26:26 localhost kernel: [<c059b6b6>] sock_attach_fd+0x70/0xd2 May 23 19:26:26 localhost kernel: [<c04774a0>] get_empty_filp+0xfc/0x170 May 23 19:26:26 localhost kernel: [<c059b54f>] sys_setsockopt+0x9b/0xa7 May 23 19:26:26 localhost kernel: [<c059cb83>] sys_socketcall+0xac/0x261 May 23 19:26:26 localhost kernel: [<c0404f70>] syscall_call+0x7/0xb
strange, while applying the concerned patch first time I was hand editing __ kernel/sched.c __ and stupidly typed _local_bh_enable() instead of local_bh_enable() and when did a reboot, as soon as I got inside X I was welcomed with following message in system log.
[ 152.692609] BUG: at kernel/softirq.c:122 _local_bh_enable() [ 152.692637] [<b040624d>] show_trace_log_lvl+0x1a/0x2f [ 152.692658] [<b0406801>] show_trace+0x12/0x14 [ 152.692668] [<b0406885>] dump_stack+0x16/0x18 [ 152.692678] [<b0428d66>] _local_bh_enable+0x8b/0xc3 [ 152.692688] [<b059e497>] cond_resched_softirq+0x2b/0x40 [ 152.692700] [<b05797b6>] established_get_first+0x19/0xad [ 152.692712] [<b057a831>] tcp_seq_next+0x76/0x8c [ 152.692722] [<b04875ef>] seq_read+0x17b/0x264 [ 152.692733] [<b047102a>] vfs_read+0xad/0x161 [ 152.692745] [<b04714b6>] sys_read+0x3d/0x61 [ 152.692755] [<b04050d4>] syscall_call+0x7/0xb [ 152.692765] ======================= [ 152.692770] BUG: at kernel/lockdep.c:1937 trace_softirqs_on() [ 152.692777] [<b040624d>] show_trace_log_lvl+0x1a/0x2f [ 152.692789] [<b0406801>] show_trace+0x12/0x14 [ 152.692800] [<b0406885>] dump_stack+0x16/0x18 [ 152.692810] [<b043f5a5>] trace_softirqs_on+0x5f/0xa5 [ 152.692822] [<b0428d8e>] _local_bh_enable+0xb3/0xc3 [ 152.692831] [<b059e497>] cond_resched_softirq+0x2b/0x40 [ 152.692842] [<b05797b6>] established_get_first+0x19/0xad [ 152.692852] [<b057a831>] tcp_seq_next+0x76/0x8c [ 152.692862] [<b04875ef>] seq_read+0x17b/0x264 [ 152.692870] [<b047102a>] vfs_read+0xad/0x161 [ 152.692879] [<b04714b6>] sys_read+0x3d/0x61 [ 152.692889] [<b04050d4>] syscall_call+0x7/0xb [ 152.692899] ======================= [ 159.257890] NOHZ: local_softirq_pending 22 [ 159.266009] NOHZ: local_softirq_pending 22 [ 159.273965] NOHZ: local_softirq_pending 22 [ 159.281884] NOHZ: local_softirq_pending 22 [ 160.712828] NOHZ: local_softirq_pending 22 [ 162.609377] NOHZ: local_softirq_pending 22 [ 162.609804] NOHZ: local_softirq_pending 22 [ 162.610054] NOHZ: local_softirq_pending 22 [ 162.610279] NOHZ: local_softirq_pending 22 [ 162.610502] NOHZ: local_softirq_pending 22
After realzing my mistake, I changed it to local_bh_enable() as was in patch and since then not a single BUG or local_softirq_pending message occurs in system log, maybe my system waiting for that condition to happen :).
Signature · 3 lines
Out of many thousands, one may endeavor for perfection, and of those who have achieved perfection, hardly one knows Me in truth. -- Gita Sutra Of Mysticism
Quoted 11 lines · Chuck Ebbert
Review
Quoted 8 lines
(quoted diff) +1
-3if (need_resched() && system_state == SYSTEM_RUNNING) { - raw_local_irq_disable(); - _local_bh_enable(); - raw_local_irq_enable(); + local_bh_enable(); __cond_resched(); local_bh_disable(); return 1;We may have a problem with that:
Quoted 12 lines
BUG: warning at kernel/softirq.c:138/local_bh_enable() (Not tainted) [<c042b2ef>] local_bh_enable+0x45/0x92 [<c06036b7>] cond_resched_softirq+0x2c/0x42 [<c059d5d0>] release_sock+0x54/0xa3 [<c05c9428>] tcp_sendmsg+0x91b/0xa0c [<c05e1bb9>] inet_sendmsg+0x3b/0x45 [<c059af34>] sock_aio_write+0xf9/0x105 [<c0476035>] do_sync_write+0xc7/0x10a [<c0437265>] autoremove_wake_function+0x0/0x35 [<c047688e>] vfs_write+0xbc/0x154 [<c0476e8c>] sys_write+0x41/0x67 [<c0404f70>] syscall_call+0x7/0xb
hm, this place really shouldnt call cond_resched_softirq() with hardirqs disabled.
perhaps a buggy ->sk_backlog_rcv() handler disabled interrupts without restoring them?
could you enable CONFIG_PROVE_LOCKING and apply the patch below - which location is printed as having last disabled hardirqs?
Ingo
--------------------->
Subject: [patch] softirqs: print out irq-trace events
From: Ingo Molnar <[email protected]>
some code is fiddling with softirqs but hardirqs are disabled, so try to figure out who disabled hardirqs.
--- kernel/softirq.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
kernel/softirq.c +9 -1
@@ -135,7 +135,15 @@ void local_bh_enable(void)WARN_ON_ONCE (in_irq ());#endif -WARN_ON_ONCE (irqs_disabled ()); +if (irqs_disabled ()) { +static int once =1 ; + +if (once) { + once =0 ; +print_irqtrace_events (current); +WARN_ON (1 ); + } + }#ifdef CONFIG_TRACE_IRQFLAGS local_irq_save (flags);