New: kworker using a lot of cpu
https://bugzilla.kernel.org/show_bug.cgi?id=207817
Bug ID: 207817
Summary: kworker using a lot of cpu
Product: File System
Version: 2.5
Kernel Version: 4.19.69-1.el7.x86_64
Hardware: Intel
OS: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: XFS
Assignee: [email protected]
Reporter: [email protected]
Regression: NoCreated attachment 289193
--> https://bugzilla.kernel.org/attachment.cgi?id=289193&action=edit
files from description
Hello!
I'm using CentOS Linux release 7.7.1908 (Core) with kernel 4.19.69 under VMWare hypervisor.
During my application stress tests, this problem occurred twice over the last half-year. The last occurrence affected all three VM under the load at once.
# top -sH | head top - 16:16:14 up 4:49, 1 user, load average: 1.01, 1.02, 1.00 Threads: 290 total, 2 running, 186 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.5 us, 26.9 sy, 0.0 ni, 71.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 8168128 total, 5761540 free, 1267184 used, 1139404 buff/cache KiB Swap: 0 total, 0 free, 0 used. 6620652 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
360 root 20 0 0 0 0 R 99.9 0.0 283:13.38
kworker/u8:2+fl
1 root 20 0 43784 5592 3936 S 0.0 0.1 0:08.64 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd# cat /proc/360/comm kworker/u8:2+flush-253:2
# uname -a Linux hbr01 4.19.69-1.el7.x86_64 #1 SMP Thu Aug 29 11:11:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# xfs_repair -V xfs_repair version 4.5.0
Quoted 1 line
number of CPUs
attached as 'cpuinfo' file
Quoted 1 line
contents of
/proc/meminfo
attached as 'meminfo' file
Quoted 1 line
contents of
/proc/mounts
attached as 'mounts' file
please, not that /var/log is mounted from network storage to hypervisor
Quoted 1 line
contents of
/proc/partitions
attached as 'partitions' file
Quoted 1 line
RAID layout (hardware and/or software)
not used
Quoted 1 line
write cache status of drives
# hdparm -W /dev/sd[a-d] | grep "write-caching" ... write-caching = not supported write-caching = not supported write-caching = not supported write-caching = not supported
Quoted 1 line
xfs_infooutput on the filesystem in question
attached as 'xfs_info' file
# echo w > /proc/sysrq-trigger # dmesg attached as 'w_sysrq-trigger' file
# echo l > /proc/sysrq-trigger # dmesg attached as 'l_sysrq-trigger' file
# perf record -g -a sleep 10 attached as 'perf.data' file
# trace-cmd record -e xfs\* the trace.dat has the size about 1.4Gb over +-10 second and the trace_report.txt has the size more 3+Gb over +-10 second I guess it's better to share it from some file storage?
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207817] kworker using a lots of cpu
https://bugzilla.kernel.org/show_bug.cgi?id=207817
--- Comment #1 from Alexander Petrovsky ([email protected]) --- After 1 day, it seems like some internal activity calm down kworker at 00:00 UTC, it could be logrotate or smth else. But now, I'm observe the follow (seems like it has the same root cause):
#df -h
Filesystem Size Used Avail Use% Mounted on
.../dev/mapper/vg_logs-lv_varlog 38G -30G 68G - /var/log
...
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Re: [Bug 207817] kworker using a lots of cpu
Quoted 7 lines · On Thu, May 21, 2020 at 04:45:34PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207817
--- Comment #1 from Alexander Petrovsky ([email protected]) --- After 1 day, it seems like some internal activity calm down kworker at 00:00 UTC, it could be logrotate or smth else. But now, I'm observe the follow (seems like it has the same root cause):
Note that you're reporting problems with a distro kernel and proprietary hypervisor to an upstream mailing list (via an upstream bug tracker). The feedback will likely be limited unless you can reproduce on an upstream kernel.
In general, it's not really clear to me what you're reporting beyond the writeback task using more CPU than anticipated. What is that based on? What problematic functional or performance related behavior is observed? If performance related, what exactly is the workload?
Quoted 5 lines
#df -h
Filesystem Size Used Avail Use% Mounted on
.../dev/mapper/vg_logs-lv_varlog38G -30G 68G -/var/log
...
I think we've had some upstream patches to fix underflows and such in space reporting paths fairly recently, but I'm not sure off hand if those are associated with any functional issues beyond indication of potential corruption. This suggests you should probably run 'xfs_repair -n' on this filesystem if you haven't already.
Brian
Quoted 5 lines
Signature · 3 lines
-- You are receiving this mail because: You are watching the assignee of the bug.
[Bug 207817] kworker using a lots of cpu
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=207817
--- Comment #2 from [email protected] ---
Quoted 8 lines · On Thu, May 21, 2020 at 04:45:34PM +0000, [email protected]…
https://bugzilla.kernel.org/show_bug.cgi?id=207817
--- Comment #1 from Alexander Petrovsky ([email protected]) --- After 1 day, it seems like some internal activity calm down kworker at 00:00 UTC, it could be logrotate or smth else. But now, I'm observe the follow (seems
like it has the same root cause):
Note that you're reporting problems with a distro kernel and proprietary hypervisor to an upstream mailing list (via an upstream bug tracker). The feedback will likely be limited unless you can reproduce on an upstream kernel.
In general, it's not really clear to me what you're reporting beyond the writeback task using more CPU than anticipated. What is that based on? What problematic functional or performance related behavior is observed? If performance related, what exactly is the workload?
Quoted 5 lines
#df -h
Filesystem Size Used Avail Use% Mounted on
.../dev/mapper/vg_logs-lv_varlog38G -30G 68G -/var/log
...
I think we've had some upstream patches to fix underflows and such in space reporting paths fairly recently, but I'm not sure off hand if those are associated with any functional issues beyond indication of potential corruption. This suggests you should probably run 'xfs_repair -n' on this filesystem if you haven't already.
Brian
Quoted 5 lines
Signature · 3 lines
-- You are receiving this mail because: You are watching the assignee of the bug.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.