thousands of kworker processes with 4.7.x and 4.8-rc*
On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing thousands of kworker processes.
# ps auxf|grep -c kworker 2104
Load average goes into hundreds on a pretty much idle server (biggest CPU and RAM consumers are probably SSHD with one user logged in and rsyslog writing ~1 line per minute):
# uptime 06:58:56 up 26 min, 1 user, load average: 146.11, 215.46, 105.70
# uptime 06:59:48 up 26 min, 1 user, load average: 305.20, 240.84, 120.25
Sometimes seeing lots of them in "D" state:
root 19474 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:208] root 19475 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:209] root 19477 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:211] root 19480 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:214] root 19483 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:217] root 19485 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:219] root 19486 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:220] root 19487 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:221] root 19492 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:226] root 19533 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/4:257]
Is it a known issue?
The server has 8 CPUs and 32 GB RAM.
Tomasz Chmielewski
https://lxadm.com
Quoted 16 lines · Tomasz Chmielewski
On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing thousands of kworker processes.
# ps auxf|grep -c kworker 2104 Load average goes into hundreds on a pretty much idle server (biggest CPU and RAM consumers are probably SSHD with one user logged in and rsyslog writing ~1 line per minute): # uptime 06:58:56 up 26 min, 1 user, load average: 146.11, 215.46, 105.70 # uptime 06:59:48 up 26 min, 1 user, load average: 305.20, 240.84, 120.25 Sometimes seeing lots of them in "D" state: root 19474 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:208] root 19475 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:209]
I did some experiments to see when the problem first appeared. Thousands of kworker processes start to show up in 4.7.0-rc5.
kernel version | kworker count after boot
-------------------------------------------
4.6.3 37 4.6.4 47 4.6.5 46 4.6.6 49 4.6.7 49
4.7.0-rc1 46
4.7.0-rc2 49
4.7.0-rc3 45
4.7.0-rc4 47
4.7.0-rc5 1592
4.7.0-rc6 1714
4.7.0-rc7 1955
4.7.0 2088 4.7.1 1222 4.7.2 1699
4.7.3 1446
4.7.4 1781
4.8-rc1 (not tested)
4.8-rc2 2012
4.8-rc3 1696
4.8-rc4 1210
4.8-rc5 1890
4.8-rc6 1657
4.8-rc7 1647
Tomasz Chmielewski
https://lxadm.com
Quoted 29 lines · Tomasz Chmielewski
Quoted 16 lines · Tomasz Chmielewski
On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing thousands of kworker processes.
# ps auxf|grep -c kworker 2104 Load average goes into hundreds on a pretty much idle server (biggest CPU and RAM consumers are probably SSHD with one user logged in and rsyslog writing ~1 line per minute): # uptime 06:58:56 up 26 min, 1 user, load average: 146.11, 215.46, 105.70 # uptime 06:59:48 up 26 min, 1 user, load average: 305.20, 240.84, 120.25 Sometimes seeing lots of them in "D" state: root 19474 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:208] root 19475 0.0 0.0 0 0 ? D 06:54 0:00 \_ [kworker/0:209]I did some experiments to see when the problem first appeared. Thousands of kworker processes start to show up in 4.7.0-rc5.
kernel version | kworker count after boot
-------------------------------------------4.6.3 > > 37 4.6.4 > > 47 4.6.5 > > 46 4.6.6 > > 49 4.6.7 > > 49
Quoted 5 lines
4.7.0-rc1 > > 46
4.7.0-rc2 > > 49
4.7.0-rc3> > 45
4.7.0-rc4> > 47
4.7.0-rc5> > 1592
Best bet would be to use 'git bisect' to locate the exact commit that caused this, and post the bisection result along with your config.
AFAIK, nobody else is seeing this, is the kernel virgin source?
-Mike
Quoted 16 lines · Mike Galbraith
Quoted 10 lines
I did some experiments to see when the problem first appeared. Thousands of kworker processes start to show up in 4.7.0-rc5.
kernel version | kworker count after boot
-------------------------------------------4.6.3 > > 37 4.6.4 > > 47 4.6.5 > > 46 4.6.6 > > 49 4.6.7 > > 49Quoted 5 lines
4.7.0-rc1 > > 46
4.7.0-rc2 > > 49
4.7.0-rc3> > 45
4.7.0-rc4> > 47
4.7.0-rc5> > 1592
Quoted 4 lines
Best bet would be to use 'git bisect' to locate the exact commit that caused this, and post the bisection result along with your config.
AFAIK, nobody else is seeing this, is the kernel virgin source?
Yes, it's a kernel.org kernel.
I found some similar reports, though without much more info:
https://github.com/zfsonlinux/zfs/issues/5036 - kernel 4.7.2, initially attributed to ZFS on Linux, but then reproduced without ZFS
https://github.com/systemd/systemd/issues/4069 - kernel 4.7.2
I'll try to bisect.
Tomasz Chmielewski
https://lxadm.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Tomasz Chmielewski
Quoted 1 line
I'll try to bisect.
OK, not a kernel regression, but some config change caused it. However, I'm not able to locate which change exactly.
I'm attaching two configs which I've tried with 4.7.3 - one results in thousands of kworkers, and the other doesn't. Also included a diff between them.
Any obvious changes I should try?
The problem is the allocator.
-CONFIG_SLUB=y
+CONFIG_SLAB=y
With SLUB, I'm getting a handful of kworker processes, as expected.
With SLAB, I'm getting thousands of kworker processes.
Not sure if that's expected behaviour or not.
Tomasz Chmielewski
https://lxadm.com
Quoted 6 lines · Tomasz Chmielewski
The problem is the allocator.
(quoted diff) +1
-1-CONFIG_SLUB=y +CONFIG_SLAB=y
Quoted 6 lines
With SLUB, I'm getting a handful of kworker processes, as expected.
With SLAB, I'm getting thousands of kworker processes.
Not sure if that's expected behaviour or not.
I seriously doubt 1500+ kworkers piling up is expected.
4.7.0-rc4 47
4.7.0-rc5 1592
Presuming you didn't switch to SLAB and/or change userspace all around while testing, that would still indicate a kernel regression lurking between rc4 and rc5.
-Mike
Quoted 17 lines · Mike Galbraith
Quoted 6 lines · Tomasz Chmielewski
The problem is the allocator.
(quoted diff) +1
-1-CONFIG_SLUB=y +CONFIG_SLAB=yQuoted 6 lines
With SLUB, I'm getting a handful of kworker processes, as expected.
With SLAB, I'm getting thousands of kworker processes.
Not sure if that's expected behaviour or not.
I seriously doubt 1500+ kworkers piling up is expected.
Quoted 6 lines
4.7.0-rc4 47
4.7.0-rc5 1592Presuming you didn't switch to SLAB and/or change userspace all around while testing, that would still indicate a kernel regression lurking between rc4 and rc5.
Nevermind, seems config change was how you met the thing.
FWIW, I turned on SLAB with a distro config here, and see no such behavior. If I could reproduce, I'd take the busted config back to older kernels.
-Mike
Quoted 19 lines · Tomasz Chmielewski
Quoted 10 lines · Tomasz Chmielewski
Quoted 1 line
I'll try to bisect.
OK, not a kernel regression, but some config change caused it. However, I'm not able to locate which change exactly.
I'm attaching two configs which I've tried with 4.7.3 - one results in thousands of kworkers, and the other doesn't. Also included a diff between them.
Any obvious changes I should try?
The problem is the allocator.
(quoted diff) +1
-1-CONFIG_SLUB=y +CONFIG_SLAB=y
Quoted 6 lines
With SLUB, I'm getting a handful of kworker processes, as expected.
With SLAB, I'm getting thousands of kworker processes.
Not sure if that's expected behaviour or not.
Why don't you sample the stacks of some of those kworker processes to see if they are all executing a parituclar piece of work. That might help you narrow down where they originate from. Cat multiple /proc/$kworker-pid/stack files and see if a pattern emerges.
Regards,
Nikolay
Quoted 6 lines
Tomasz Chmielewski
https://lxadm.com
Quoted 7 lines · Nikolay Borisov
Quoted 1 line
Not sure if that's expected behaviour or not.
Why don't you sample the stacks of some of those kworker processes to see if they are all executing a parituclar piece of work. That might help you narrow down where they originate from. Cat multiple
/proc/$kworker-pid/stackfiles and see if a pattern emerges.
FYI, it was reproduced and bisected here (scroll to the bottom):
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1626564
Tomasz Chmielewski
https://lxadm.com