Content-Type: multipart/mixed; boundary="===============3420338793013728423==" MIME-Version: 1.0 From: Fengguang Wu To: lkp@lists.01.org Subject: Re: [rcu] 5057f55e543: -23.5% qperf.udp.recv_bw Date: Wed, 04 Jun 2014 20:33:38 +0800 Message-ID: <20140604123338.GA1263@localhost> In-Reply-To: <20140603141720.GN22231@linux.vnet.ibm.com> List-Id: --===============3420338793013728423== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Paul, On Tue, Jun 03, 2014 at 07:17:20AM -0700, Paul E. McKenney wrote: > On Tue, Jun 03, 2014 at 06:08:41PM +0800, Jet Chen wrote: > > Hi Paul, > > = > > FYI, we noticed the below changes on > > = > > git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git rcu= /fixes > > commit 5057f55e543b7859cfd26bc281291795eac93f8a ("rcu: Bind RCU grace-p= eriod kthreads if NO_HZ_FULL") > = > My guess would be that some of these workloads generated enough callbacks > that binding all the rcuo callback-offloading kthreads to CPU 0 resulted > in a bottleneck. If that was the case, CPU 0 would often hit 100% > CPU utilization, and there would be more wait time on other CPUs because > callback execution was delayed. > = > Does that match what you are seeing? This is the /proc/stat contents on completion of test brickland3/qperf/600s. It shows that CPU 0 is not busy. cpu 1707 0 57716 8569826 33 0 14572 0 0 0 cpu0 2 0 575 69345 29 0 8 0 0 0 cpu1 0 0 86 72203 3 0 1 0 0 0 cpu2 116 0 386 71437 0 0 0 0 0 0 The columns of numbers are: - user: normal processes executing in user mode - nice: niced processes executing in user mode - system: processes executing in kernel mode - idle: twiddling thumbs - iowait: waiting for I/O to complete - irq: servicing interrupts - softirq: servicing softirqs - steal: involuntary wait - guest: running a normal guest - guest_nice: running a niced guest cpu3 32 0 59 72213 0 0 0 0 0 0 cpu4 8 0 35 72272 0 0 0 0 0 0 cpu5 1 0 3 72318 0 0 0 0 0 0 cpu6 1 0 3 72320 0 0 0 0 0 0 cpu7 0 0 2 72327 0 0 0 0 0 0 cpu8 0 0 2 72323 0 0 0 0 0 0 cpu9 0 0 1 72323 0 0 0 0 0 0 cpu10 0 0 2 72314 0 0 0 0 0 0 cpu11 0 0 2 72317 0 0 0 0 0 0 cpu12 0 0 3 72311 0 0 0 0 0 0 cpu13 0 0 2 72313 0 0 0 0 0 0 cpu14 0 0 20 72296 0 0 0 0 0 0 cpu15 0 0 90 72192 0 0 1 0 0 0 cpu16 0 0 13 72288 0 0 1 0 0 0 cpu17 115 0 137 71848 0 0 0 0 0 0 cpu18 28 0 32 72106 0 0 0 0 0 0 cpu19 7 0 5 72256 0 0 0 0 0 0 cpu20 2 0 72 71938 0 0 0 0 0 0 cpu21 0 0 3 72273 0 0 0 0 0 0 cpu22 0 0 2 72284 0 0 0 0 0 0 cpu23 0 0 1 72284 0 0 0 0 0 0 cpu24 0 0 2 72279 0 0 0 0 0 0 cpu25 0 0 1 72281 0 0 0 0 0 0 cpu26 0 0 2 72274 0 0 0 0 0 0 cpu27 0 0 5 72274 0 0 0 0 0 0 cpu28 0 0 22 72255 0 0 0 0 0 0 cpu29 0 0 1 72273 0 0 0 0 0 0 cpu30 24 0 748 70165 0 0 407 0 0 0 cpu31 28 0 906 70251 0 0 465 0 0 0 cpu32 62 0 2276 68376 0 0 604 0 0 0 cpu33 27 0 1100 70059 0 0 453 0 0 0 cpu34 23 0 1359 69933 0 0 418 0 0 0 cpu35 32 0 1663 69109 0 0 775 0 0 0 cpu36 103 0 8569 60806 0 0 2306 0 0 0 cpu37 88 0 7052 63141 0 0 518 0 0 0 cpu38 51 0 2881 67927 0 0 498 0 0 0 cpu39 36 0 1705 69166 0 0 430 0 0 0 cpu40 99 0 8657 60720 0 0 1767 0 0 0 cpu41 103 0 7088 61185 0 0 1749 0 0 0 cpu42 42 0 1567 69244 0 0 525 0 0 0 cpu43 43 0 1661 68865 0 0 457 0 0 0 cpu44 39 0 2610 68046 0 0 660 0 0 0 cpu45 3 0 248 71852 0 0 30 0 0 0 cpu46 4 0 16 72140 0 0 6 0 0 0 cpu47 115 0 297 71435 0 0 159 0 0 0 cpu48 30 0 81 71940 0 0 61 0 0 0 cpu49 14 0 163 71815 0 0 118 0 0 0 cpu50 6 0 144 71832 0 0 129 0 0 0 cpu51 12 0 508 71217 0 0 276 0 0 0 cpu52 18 0 498 71082 0 0 378 0 0 0 cpu53 11 0 440 71236 0 0 282 0 0 0 cpu54 4 0 205 71710 0 0 144 0 0 0 cpu55 4 0 136 71898 0 0 78 0 0 0 cpu56 13 0 400 71305 0 0 288 0 0 0 cpu57 14 0 377 71346 0 0 316 0 0 0 cpu58 2 0 109 71954 0 0 73 0 0 0 cpu59 1 0 8 72174 0 0 3 0 0 0 cpu60 0 0 0 72191 0 0 0 0 0 0 cpu61 86 0 313 71749 0 0 0 0 0 0 cpu62 3 0 9 72173 0 0 0 0 0 0 cpu63 2 0 8 72175 0 0 0 0 0 0 cpu64 0 0 2 72178 0 0 0 0 0 0 cpu65 0 0 2 72177 0 0 0 0 0 0 cpu66 0 0 2 72175 0 0 0 0 0 0 cpu67 0 0 2 72173 0 0 0 0 0 0 cpu68 0 0 4 72169 0 0 0 0 0 0 cpu69 0 0 2 72168 0 0 0 0 0 0 cpu70 0 0 1 72166 0 0 0 0 0 0 cpu71 0 0 1 72164 0 0 0 0 0 0 cpu72 0 0 0 72163 0 0 0 0 0 0 cpu73 0 0 0 72161 0 0 0 0 0 0 cpu74 0 0 1 72158 0 0 0 0 0 0 cpu75 0 0 0 72156 0 0 0 0 0 0 cpu76 73 0 330 71709 0 0 0 0 0 0 cpu77 4 0 10 72137 0 0 0 0 0 0 cpu78 1 0 4 72145 0 0 0 0 0 0 cpu79 0 0 50 72077 0 0 0 0 0 0 cpu80 0 0 5 72138 0 0 0 0 0 0 cpu81 0 0 1 72141 0 0 0 0 0 0 cpu82 0 0 66 72074 0 0 0 0 0 0 cpu83 0 0 1 72137 0 0 0 0 0 0 cpu84 0 0 0 72136 0 0 0 0 0 0 cpu85 0 0 1 72133 0 0 0 0 0 0 cpu86 0 0 0 72131 0 0 0 0 0 0 cpu87 0 0 0 72129 0 0 0 0 0 0 cpu88 0 0 1 72127 0 0 0 0 0 0 cpu89 0 0 0 72124 0 0 0 0 0 0 cpu90 0 0 0 72122 0 0 0 0 0 0 cpu91 63 0 1377 70501 0 0 162 0 0 0 cpu92 2 0 14 72100 0 0 0 0 0 0 cpu93 1 0 5 72109 0 0 0 0 0 0 cpu94 0 0 4 72108 0 0 0 0 0 0 cpu95 1 0 4 72106 0 0 0 0 0 0 cpu96 0 0 1 72106 0 0 0 0 0 0 cpu97 0 0 0 72105 0 0 0 0 0 0 cpu98 0 0 1 72102 0 0 0 0 0 0 cpu99 0 0 2 72099 0 0 0 0 0 0 cpu100 0 0 1 72099 0 0 0 0 0 0 cpu101 0 0 0 72096 0 0 0 0 0 0 cpu102 0 0 0 72094 0 0 0 0 0 0 cpu103 0 0 1 72092 0 0 0 0 0 0 cpu104 0 0 0 72089 0 0 0 0 0 0 cpu105 0 0 1 72086 0 0 0 0 0 0 cpu106 66 0 363 71601 0 0 9 0 0 0 cpu107 2 0 4 72075 0 0 0 0 0 0 cpu108 0 0 2 72077 0 0 0 0 0 0 cpu109 0 0 8 72069 0 0 0 0 0 0 cpu110 0 0 1 72073 0 0 0 0 0 0 cpu111 0 0 9 72063 0 0 0 0 0 0 cpu112 0 0 1 72070 0 0 0 0 0 0 cpu113 0 0 0 72068 0 0 0 0 0 0 cpu114 0 0 1 72064 0 0 0 0 0 0 cpu115 0 0 1 72063 0 0 0 0 0 0 cpu116 0 0 1 72060 0 0 0 0 0 0 cpu117 0 0 0 72059 0 0 0 0 0 0 cpu118 0 0 0 72056 0 0 0 0 0 0 cpu119 0 0 1 72053 0 0 0 0 0 0 intr 5639794 132 0 0 9 856 0 0 0 1 0 0 0 0 0 0 0 31 0 0 0 0 0 0 311 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 = 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 = 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 0 2794 395 4= 11 385 399 378 399 375 396 384 383 396 379 470 386 375 375 386 382 512 375 = 375 375 375 375 375 375 375 375 375 375 375 378 375 375 375 375 375 375 375= 375 375 379 381 388 375 375 375 376 379 375 375 375 375 375 375 375 375 37= 5 375 375 382 375 1 78 64 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0= 0 0 0 0 0 0 0 ctxt 84441175 btime 1401871947 processes 23608 procs_running 2 procs_blocked 0 softirq 84702758 9 1574595 174 82369760 28 0 305 456968 84738 216181 Thanks, Fengguang --===============3420338793013728423==--