Skip to contentSkip to conversation

thousands of kworker processes with 4.7.x and 4.8-rc*

DISCUSSION Tomasz Chmielewski 3 participants 9 messages Sep 19, 2016 – Sep 27, 2016 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing thousands of kworker processes.

# ps auxf|grep -c kworker
2104

Load average goes into hundreds on a pretty much idle server (biggest CPU and RAM consumers are probably SSHD with one user logged in and rsyslog writing ~1 line per minute):

# uptime
 06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70
# uptime
 06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25

Sometimes seeing lots of them in "D" state:

root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:208]
root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:209]
root     19477  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:211]
root     19480  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:214]
root     19483  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:217]
root     19485  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:219]
root     19486  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:220]
root     19487  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:221]
root     19492  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/0:226]
root     19533  0.0  0.0      0     0 ?        D    06:54   0:00  \_ [kworker/4:257]

Is it a known issue?

The server has 8 CPUs and 32 GB RAM.

Tomasz Chmielewski
https://lxadm.com

#1 lore raw
Quoted 16 lines · Tomasz Chmielewski

On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing thousands of kworker processes.

# ps auxf|grep -c kworker
2104
Load average goes into hundreds on a pretty much idle server (biggest
CPU and RAM consumers are probably SSHD with one user logged in and
rsyslog writing ~1 line per minute):
# uptime
 06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70
# uptime
 06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25
Sometimes seeing lots of them in "D" state:
root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_
[kworker/0:208]
root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_
[kworker/0:209]

I did some experiments to see when the problem first appeared. Thousands of kworker processes start to show up in 4.7.0-rc5.

kernel version | kworker count after boot
-------------------------------------------

4.6.3 	        37
4.6.4 	        47
4.6.5 	        46
4.6.6 	        49
4.6.7 	        49

4.7.0-rc1 46
4.7.0-rc2 49
4.7.0-rc3 45
4.7.0-rc4 47
4.7.0-rc5 1592
4.7.0-rc6 1714
4.7.0-rc7 1955

4.7.0 	        2088
4.7.1 	        1222
4.7.2 	        1699

4.7.3 1446
4.7.4 1781
4.8-rc1 (not tested)
4.8-rc2 2012
4.8-rc3 1696
4.8-rc4 1210
4.8-rc5 1890
4.8-rc6 1657
4.8-rc7 1647

Tomasz Chmielewski
https://lxadm.com

#2 lore raw
Quoted 29 lines · Tomasz Chmielewski
Quoted 16 lines · Tomasz Chmielewski

On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing thousands of kworker processes.

# ps auxf|grep -c kworker
2104
Load average goes into hundreds on a pretty much idle server (biggest
CPU and RAM consumers are probably SSHD with one user logged in and
rsyslog writing ~1 line per minute):
# uptime
 06:58:56 up 26 min,  1 user,  load average: 146.11, 215.46, 105.70
# uptime
 06:59:48 up 26 min,  1 user,  load average: 305.20, 240.84, 120.25
Sometimes seeing lots of them in "D" state:
root     19474  0.0  0.0      0     0 ?        D    06:54   0:00  \_
[kworker/0:208]
root     19475  0.0  0.0      0     0 ?        D    06:54   0:00  \_
[kworker/0:209]

I did some experiments to see when the problem first appeared. Thousands of kworker processes start to show up in 4.7.0-rc5.

kernel version | kworker count after boot
-------------------------------------------

4.6.3 > 	>         37
4.6.4 > 	>         47
4.6.5 > 	>         46
4.6.6 > 	>         49
4.6.7 > 	>         49
Quoted 5 lines

4.7.0-rc1 > > 46
4.7.0-rc2 > > 49
4.7.0-rc3> > 45
4.7.0-rc4> > 47
4.7.0-rc5> > 1592

Best bet would be to use 'git bisect' to locate the exact commit that caused this, and post the bisection result along with your config.

AFAIK, nobody else is seeing this, is the kernel virgin source?

-Mike

#3 lore raw
Quoted 16 lines · Mike Galbraith
Quoted 10 lines

I did some experiments to see when the problem first appeared. Thousands of kworker processes start to show up in 4.7.0-rc5.

kernel version | kworker count after boot
-------------------------------------------

4.6.3 > 	>         37
4.6.4 > 	>         47
4.6.5 > 	>         46
4.6.6 > 	>         49
4.6.7 > 	>         49
Quoted 5 lines

4.7.0-rc1 > > 46
4.7.0-rc2 > > 49
4.7.0-rc3> > 45
4.7.0-rc4> > 47
4.7.0-rc5> > 1592

Quoted 4 lines

Best bet would be to use 'git bisect' to locate the exact commit that caused this, and post the bisection result along with your config.

AFAIK, nobody else is seeing this, is the kernel virgin source?

Yes, it's a kernel.org kernel.

I found some similar reports, though without much more info:

https://github.com/zfsonlinux/zfs/issues/5036 - kernel 4.7.2, initially attributed to ZFS on Linux, but then reproduced without ZFS

https://github.com/systemd/systemd/issues/4069 - kernel 4.7.2

I'll try to bisect.

Tomasz Chmielewski
https://lxadm.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 10 lines · Tomasz Chmielewski
Quoted 1 line

I'll try to bisect.

OK, not a kernel regression, but some config change caused it. However, I'm not able to locate which change exactly.

I'm attaching two configs which I've tried with 4.7.3 - one results in thousands of kworkers, and the other doesn't. Also included a diff between them.

Any obvious changes I should try?

The problem is the allocator.

-CONFIG_SLUB=y
+CONFIG_SLAB=y

With SLUB, I'm getting a handful of kworker processes, as expected.

With SLAB, I'm getting thousands of kworker processes.

Not sure if that's expected behaviour or not.

Tomasz Chmielewski
https://lxadm.com

#6 lore raw
Quoted 6 lines · Tomasz Chmielewski

The problem is the allocator.

(quoted diff) +1 -1
-CONFIG_SLUB=y
+CONFIG_SLAB=y
Quoted 6 lines

With SLUB, I'm getting a handful of kworker processes, as expected.

With SLAB, I'm getting thousands of kworker processes.

Not sure if that's expected behaviour or not.

I seriously doubt 1500+ kworkers piling up is expected.

4.7.0-rc4 47
4.7.0-rc5 1592

Presuming you didn't switch to SLAB and/or change userspace all around while testing, that would still indicate a kernel regression lurking between rc4 and rc5.

-Mike

#7 lore raw
Quoted 17 lines · Mike Galbraith
Quoted 6 lines · Tomasz Chmielewski

The problem is the allocator.

(quoted diff) +1 -1
-CONFIG_SLUB=y
+CONFIG_SLAB=y
Quoted 6 lines

With SLUB, I'm getting a handful of kworker processes, as expected.

With SLAB, I'm getting thousands of kworker processes.

Not sure if that's expected behaviour or not.

I seriously doubt 1500+ kworkers piling up is expected.

Quoted 6 lines

4.7.0-rc4 47
4.7.0-rc5 1592

Presuming you didn't switch to SLAB and/or change userspace all around while testing, that would still indicate a kernel regression lurking between rc4 and rc5.

Nevermind, seems config change was how you met the thing.

FWIW, I turned on SLAB with a distro config here, and see no such behavior. If I could reproduce, I'd take the busted config back to older kernels.

-Mike

#8 lore raw
In reply to Tomasz Chmielewski
Quoted 19 lines · Tomasz Chmielewski
Quoted 10 lines · Tomasz Chmielewski
Quoted 1 line

I'll try to bisect.

OK, not a kernel regression, but some config change caused it. However, I'm not able to locate which change exactly.

I'm attaching two configs which I've tried with 4.7.3 - one results in thousands of kworkers, and the other doesn't. Also included a diff between them.

Any obvious changes I should try?

The problem is the allocator.

(quoted diff) +1 -1
-CONFIG_SLUB=y
+CONFIG_SLAB=y
Quoted 6 lines

With SLUB, I'm getting a handful of kworker processes, as expected.

With SLAB, I'm getting thousands of kworker processes.

Not sure if that's expected behaviour or not.

Why don't you sample the stacks of some of those kworker processes to see if they are all executing a parituclar piece of work. That might help you narrow down where they originate from. Cat multiple /proc/$kworker-pid/stack files and see if a pattern emerges.

Regards,
Nikolay

Quoted 6 lines

Tomasz Chmielewski
https://lxadm.com

#9 lore raw
Quoted 7 lines · Nikolay Borisov
Quoted 1 line

Not sure if that's expected behaviour or not.

Why don't you sample the stacks of some of those kworker processes to see if they are all executing a parituclar piece of work. That might help you narrow down where they originate from. Cat multiple /proc/$kworker-pid/stack files and see if a pattern emerges.

FYI, it was reproduced and bisected here (scroll to the bottom):

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1626564

Tomasz Chmielewski
https://lxadm.com