Skip to contentSkip to conversation

New: kworker using a lot of cpu

Bug 207817 [email protected] 2 participants 4 messages May 21, 2020

lore
Share & tools
lore
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=207817

            Bug ID: 207817
           Summary: kworker using a lot of cpu
           Product: File System
           Version: 2.5
    Kernel Version: 4.19.69-1.el7.x86_64
          Hardware: Intel
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: XFS
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 289193
--> https://bugzilla.kernel.org/attachment.cgi?id=289193&action=edit
files from description

Hello!

I'm using CentOS Linux release 7.7.1908 (Core) with kernel 4.19.69 under VMWare hypervisor.

During my application stress tests, this problem occurred twice over the last half-year. The last occurrence affected all three VM under the load at once.

# top -sH | head
top - 16:16:14 up  4:49,  1 user,  load average: 1.01, 1.02, 1.00
Threads: 290 total,   2 running, 186 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.5 us, 26.9 sy,  0.0 ni, 71.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8168128 total,  5761540 free,  1267184 used,  1139404 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  6620652 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
360 root 20 0 0 0 0 R 99.9 0.0 283:13.38
kworker/u8:2+fl

    1 root      20   0   43784   5592   3936 S  0.0  0.1   0:08.64 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kthreadd
# cat /proc/360/comm
kworker/u8:2+flush-253:2
# uname -a
Linux hbr01 4.19.69-1.el7.x86_64 #1 SMP Thu Aug 29 11:11:09 UTC 2019 x86_64
x86_64 x86_64 GNU/Linux
# xfs_repair -V
xfs_repair version 4.5.0
Quoted 1 line

number of CPUs

attached as 'cpuinfo' file

Quoted 1 line

contents of /proc/meminfo

attached as 'meminfo' file

Quoted 1 line

contents of /proc/mounts

attached as 'mounts' file
please, not that /var/log is mounted from network storage to hypervisor

Quoted 1 line

contents of /proc/partitions

attached as 'partitions' file

Quoted 1 line

RAID layout (hardware and/or software)

not used

Quoted 1 line

write cache status of drives

# hdparm -W /dev/sd[a-d] | grep "write-caching"
...
 write-caching = not supported
 write-caching = not supported
 write-caching = not supported
 write-caching = not supported
Quoted 1 line

xfs_info output on the filesystem in question

attached as 'xfs_info' file

# echo w > /proc/sysrq-trigger
# dmesg
attached as 'w_sysrq-trigger' file
# echo l > /proc/sysrq-trigger
# dmesg
attached as 'l_sysrq-trigger' file
# perf record -g -a sleep 10
attached as 'perf.data' file
# trace-cmd record -e xfs\*
the trace.dat has the size about 1.4Gb over +-10 second
and
the trace_report.txt has the size more 3+Gb over +-10 second
I guess it's better to share it from some file storage?
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#1 lore raw

[Bug 207817] kworker using a lots of cpu

https://bugzilla.kernel.org/show_bug.cgi?id=207817

--- Comment #1 from Alexander Petrovsky ([email protected]) --- After 1 day, it seems like some internal activity calm down kworker at 00:00 UTC, it could be logrotate or smth else. But now, I'm observe the follow (seems like it has the same root cause):

#df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/mapper/vg_logs-lv_varlog 38G -30G 68G - /var/log
...

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#2 lore raw

Re: [Bug 207817] kworker using a lots of cpu

Quoted 7 lines · On Thu, May 21, 2020 at 04:45:34PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207817

--- Comment #1 from Alexander Petrovsky ([email protected]) --- After 1 day, it seems like some internal activity calm down kworker at 00:00 UTC, it could be logrotate or smth else. But now, I'm observe the follow (seems like it has the same root cause):

Note that you're reporting problems with a distro kernel and proprietary hypervisor to an upstream mailing list (via an upstream bug tracker). The feedback will likely be limited unless you can reproduce on an upstream kernel.

In general, it's not really clear to me what you're reporting beyond the writeback task using more CPU than anticipated. What is that based on? What problematic functional or performance related behavior is observed? If performance related, what exactly is the workload?

Quoted 5 lines

#df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/mapper/vg_logs-lv_varlog 38G -30G 68G - /var/log
...

I think we've had some upstream patches to fix underflows and such in space reporting paths fairly recently, but I'm not sure off hand if those are associated with any functional issues beyond indication of potential corruption. This suggests you should probably run 'xfs_repair -n' on this filesystem if you haven't already.

Brian

Quoted 5 lines
Signature · 3 lines
-- 
You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw

[Bug 207817] kworker using a lots of cpu

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=207817

--- Comment #2 from [email protected] ---

Quoted 8 lines · On Thu, May 21, 2020 at 04:45:34PM +0000, [email protected]…

https://bugzilla.kernel.org/show_bug.cgi?id=207817

--- Comment #1 from Alexander Petrovsky ([email protected]) --- After 1 day, it seems like some internal activity calm down kworker at 00:00 UTC, it could be logrotate or smth else. But now, I'm observe the follow (seems
like it has the same root cause):

Note that you're reporting problems with a distro kernel and proprietary hypervisor to an upstream mailing list (via an upstream bug tracker). The feedback will likely be limited unless you can reproduce on an upstream kernel.

In general, it's not really clear to me what you're reporting beyond the writeback task using more CPU than anticipated. What is that based on? What problematic functional or performance related behavior is observed? If performance related, what exactly is the workload?

Quoted 5 lines

#df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/mapper/vg_logs-lv_varlog 38G -30G 68G - /var/log
...

I think we've had some upstream patches to fix underflows and such in space reporting paths fairly recently, but I'm not sure off hand if those are associated with any functional issues beyond indication of potential corruption. This suggests you should probably run 'xfs_repair -n' on this filesystem if you haven't already.

Brian

Quoted 5 lines
Signature · 3 lines
-- 
You are receiving this mail because:
You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.