Hello and a question about high cpu usage on jfsCommit (kernel 2.6.25.1)
Hey guys,
I'm new to the list, but I've been using Linux and fooling around with the kernel for ages. :)
I've been experiencing high CPU usage for jfsCommit on kernel 2.6.25.1 (haven't had a chance to go to 2.6.25.4, but I didn't see any JFS-specific changes between the versions yet). Here's my hardware config, as well:
CPUs: 2x Intel Xeon E5420 2.5GHz Quad-core
RAM: 8GB
RAID Controller: 3Ware 9650SE-24M8
I can't find a mention of what motherboard I have in here, so I'll give a bit of lspci:
00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller Hub (rev b1)
00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) 03:00.0 PCI bridge: Integrated Device Technology, Inc. Unknown device 8018 (rev 04) (prog-if 00 [Normal decode]) 05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 07:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01) 0c:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID (rev 01)
0f:0c.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
There's the salient points there. The NIC is a quad, which is trunked to the switch using LAPC.
I've got a RAID-6 on the 3Ware controller of 20 TB, and when I'm running "top", I see this:
---
top - 15:33:27 up 5:40, 3 users, load average: 4.33, 3.59, 3.98
Tasks: 315 total, 2 running, 313 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 13.2%sy, 0.0%ni, 61.3%id, 24.9%wa, 0.1%hi, 0.2%si, 0.0%st Mem: 8194264k total, 8144500k used, 49764k free, 3884k buffers Swap: 16779884k total, 148k used, 16779736k free, 7667400k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2669 root 15 -5 0 0 0 R 99 0.0 19:45.71 jfsCommit
---
And people are complaining (and I'm seeing) very slow writes to the drives.
Just wondering if anyone has any ideas. :) If you need any information, I'll provide whatever you need.
Thanks in advance!
Dave
I'm sorry, I should have mentioned that this is a x86_64 build.
Quoted 7 lines · David Fix
Hey guys,
I'm new to the list, but I've been using Linux and fooling around with the kernel for ages. :)
I've been experiencing high CPU usage for jfsCommit on kernel 2.6.25.1 (haven't had a chance to go to 2.6.25.4, but I didn't see any JFS-specific changes between the versions yet). Here's my hardware config, as well:
<SNIP>
I'm copying this to jfs-discussion to see if anyone has seen anything like this.
Quoted 8 lines · David Fix
Hey guys,
I'm new to the list, but I've been using Linux and fooling around with the kernel for ages. :)
I've been experiencing high CPU usage for jfsCommit on kernel 2.6.25.1 (haven't had a chance to go to 2.6.25.4, but I didn't see any JFS-specific changes between the versions yet).
In fact, there haven't been a whole lot of non-cosmetic changes to jfs at all recently. Nothing I see that suspicious.
What was the previous kernel you were running before moving to 2.6.25.1?
Quoted 7 lines
Here's my hardware
config, as well:CPUs: 2x Intel Xeon E5420 2.5GHz Quad-core
RAM: 8GB
RAID Controller: 3Ware 9650SE-24M8
For anyone seeing this for the first time on jfs-discussion, Dave followed up stating that this was an x86_64 build.
Quoted 33 lines
I can't find a mention of what motherboard I have in here, so I'll give a bit of lspci:
00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller Hub (rev b1)
00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9) 01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01)
03:00.0 PCI bridge: Integrated Device Technology, Inc. Unknown device 8018 (rev 04) (prog-if 00 [Normal decode])
05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
07:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
0c:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID (rev 01) 0f:0c.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)There's the salient points there. The NIC is a quad, which is trunked to the switch using LAPC.
I've got a RAID-6 on the 3Ware controller of 20 TB, and when I'm running "top", I see this:
---
top - 15:33:27 up 5:40, 3 users, load average: 4.33, 3.59, 3.98
Tasks: 315 total, 2 running, 313 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 13.2%sy, 0.0%ni, 61.3%id, 24.9%wa, 0.1%hi, 0.2%si, 0.0%st
Mem: 8194264k total, 8144500k used, 49764k free, 3884k buffers
Swap: 16779884k total, 148k used, 16779736k free, 7667400k cached
Quoted 2 lines
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2669 root 15 -5 0 0 0 R 99 0.0 19:45.71 jfsCommit
This is highly unusual. Is this thread continually eating cpu at this rate, or does it happen in spurts?
Quoted 4 lines
And people are complaining (and I'm seeing) very slow writes to the drives.
Just wondering if anyone has any ideas. :) If you need any information, I'll provide whatever you need.
Has this happened more than once (have you rebooted and still seen the problem)? I'm not sure if some rare bug has caused some kind of linked list corruption that puts the thread into an infinite loop, or if this is a real regression.
Quoted 3 lines
Thanks in advance!
Dave
Thanks,
Shaggy
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Quoted 5 lines · Dave Kleikamp
Quoted 3 lines
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2669 root 15 -5 0 0 0 R 99 0.0 19:45.71 jfsCommitThis is highly unusual. Is this thread continually eating cpu at this rate, or does it happen in spurts?
Would oprofile[0] help to see what jfsCommit is doing here?
C.
Signature · 1 line
make bzImage, not war
Hey Dave,
Thanks for following up on this... The previous kernel that I was running was 2.6.18.1... From CentOS 5.1.
It appears that the thread eating up that much CPU isn't a continuous happening, only when there's a fair amount of activity going on. It's hard to nail down exactly when it happens, but the next time it does, I'll definitely let you all know!
I haven't been able to reboot this machine, as it's a production unit, but if I do get the chance, I'll do so. It seems to have leveled out now, with there being no high usage at all on there right now.
Dave
Quoted 79 lines · Dave Kleikamp
I'm copying this to jfs-discussion to see if anyone has seen anything like this.
Quoted 6 lines · David Fix
Hey guys,
I'm new to the list, but I've been using Linux and fooling around with the kernel for ages. :)
I've been experiencing high CPU usage for jfsCommit on kernel 2.6.25.1 (haven't had a chance to go to 2.6.25.4, but I didn't see any JFS-specific changes between the versions yet).
In fact, there haven't been a whole lot of non-cosmetic changes to jfs at all recently. Nothing I see that suspicious.
What was the previous kernel you were running before moving to 2.6.25.1?
Quoted 7 lines
Here's my hardware config, as well:
CPUs: 2x Intel Xeon E5420 2.5GHz Quad-core
RAM: 8GB
RAID Controller: 3Ware 9650SE-24M8For anyone seeing this for the first time on jfs-discussion, Dave followed up stating that this was an
x86_64build.Quoted 18 lines
I can't find a mention of what motherboard I have in here, so I'll give a bit of lspci:
00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller Hub (rev b1)
00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) 03:00.0 PCI bridge: Integrated Device Technology, Inc. Unknown device 8018 (rev 04) (prog-if 00 [Normal decode]) 05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 07:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01) 0c:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID (rev 01)
0f:0c.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)There's the salient points there. The NIC is a quad, which is trunked to the switch using LAPC.
I've got a RAID-6 on the 3Ware controller of 20 TB, and when I'm running "top", I see this:
---
top - 15:33:27 up 5:40, 3 users, load average: 4.33, 3.59, 3.98Quoted 8 lines
Tasks: 315 total, 2 running, 313 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 13.2%sy, 0.0%ni, 61.3%id, 24.9%wa, 0.1%hi, 0.2%si, 0.0%st Mem: 8194264k total, 8144500k used, 49764k free, 3884k buffers Swap: 16779884k total, 148k used, 16779736k free, 7667400k cachedPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2669 root 15 -5 0 0 0 R 99 0.0 19:45.71 jfsCommitThis is highly unusual. Is this thread continually eating cpu at this rate, or does it happen in spurts?
Quoted 4 lines
And people are complaining (and I'm seeing) very slow writes to the drives.
Just wondering if anyone has any ideas. :) If you need any information, I'll provide whatever you need.
Has this happened more than once (have you rebooted and still seen the problem)? I'm not sure if some rare bug has caused some kind of linked list corruption that puts the thread into an infinite loop, or if this is a real regression.
Quoted 4 lines
Thanks in advance!
Dave
Thanks,
Shaggy
Quoted 9 lines · David Fix
Hey Dave,
Thanks for following up on this... The previous kernel that I was running was 2.6.18.1... From CentOS 5.1.
It appears that the thread eating up that much CPU isn't a continuous happening, only when there's a fair amount of activity going on. It's hard to nail down exactly when it happens, but the next time it does, I'll definitely let you all know!
Since it's only happening during periods of high activity, it doesn't seem as serious a problem as I suspected it may be. Several threads operating on a lot of files and/or directories at one time could put a lot on the jfsCommit thread's queue, even though 99% cpu utilization still seems high.
Quoted 2 lines
I haven't been able to reboot this machine, as it's a production unit, but if I do get the chance, I'll do so.
Since the thread isn't stuck in some kind of infinite loop, I don't really think a reboot will make any difference. When I asked I wasn't sure if the thread was continuously munching cpu.
Quoted 2 lines
It seems to have leveled out
now, with there being no high usage at all on there right now.
If you see it again, Christian's suggesting of running oprofile to see where the the cpu is spending most of its time might be helpful.
Thanks,
Shaggy
Signature · 2 lines
David Kleikamp IBM Linux Technology Center