2.4.20 instability on bigmem systems?
This started out as my asking about random crashes on 2.4.19, but in the course of me trying to figure out what was going on, I got to experience first-hand the increadibly poor performance of 2.4.x (notably 2.4.20) on a system with >4GB of memory (8GB).
So, I'm looking for a solution, preferrably a set of patches, to help with the problems described below.
On Monday, I installed a 2.4.20 kernel on our Dell PowerEdge 6600. This machine is configured as follows:
4x 1.6Ghz Xeon CPUs
8GB RAM
Built-in:
ATI Rage 128 graphics
dual Broadcomm Gigabit Ethernet serial/parallel/usb
Adaptec SCSI
Dell PERC3/DC (AMI/LSI MegaRAID) dual-channel
The kernel was built from the linux-2.4.20.tar.bz2 from kernel.org, and patched with only the lvm-1.0.7 and linux-2.4.20 VFS locking patches from Sistina's LVM-1.0.7 package.
The primary problem: Whenever any process (or set of processes) initiates intensive disk I/O, the system grinds to a halt, kswapd and kupdated consume upwards of 40% to 60% CPU each, and system load averages can jump upwards of 21.00. The problem can be replicated with a simple find command ("find / -print" seems to do it nicely).
I have had two rather painful nights dealing with this (Monday and Tuesday nights). Luckily, I have a serial null-modem cable rigged up between the troubled server and another server, and was able to capture all the info from the Magic Sysrq commands that I could.
Full details are at http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20
I've included the kernel config, the kernel and initrd images, the system map file, output from "ps auxfww" and a couple screen scrapings from top, and captures from magic sysrq commands from both crashes.
I had problems like this with 2.4.19, and was directed to apply a patch to inode.c, which appears to be part of a patch set for 2.4.19pre9aa2. I've archived it at:
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19/patches/10_inode-highmem-2
For 2.4.19, this solves _most_ of the stability issues, but I still have to work with the LVM people and possibly whomever is responsible for the VM in 2.4.19/2.4.20 to track down some kernel oopses (possibly a seperate problem.)
I will happily provide whatever other information is needed, though my opportunities to test things on the machine in question is limited by the fact that it's a production server.
Thanks in advance,
Gregory
Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]> Sr. Systems Administrator Cast & Crew Entertainment Services, Inc.
Quoted 10 lines · Gregory K. Ruiz-Ade
The primary problem: Whenever any process (or set of processes) initiates intensive disk I/O, the system grinds to a halt, kswapd and kupdated consume upwards of 40% to 60% CPU each, and system load averages can jump upwards of 21.00. The problem can be replicated with a simple find command ("find / -print" seems to do it nicely).
I have had two rather painful nights dealing with this (Monday and Tuesday nights). Luckily, I have a serial null-modem cable rigged up between the troubled server and another server, and was able to capture all the info from the Magic Sysrq commands that I could.
Full details are at http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20
Hmm, slabinfo would be very helpful, as well as meminfo.
Quoted 6 lines · Gregory K. Ruiz-Ade
I've included the kernel config, the kernel and initrd images, the system map file, output from "ps auxfww" and a couple screen scrapings from top, and captures from magic sysrq commands from both crashes.
I had problems like this with 2.4.19, and was directed to apply a patch toinode.c, which appears to be part of a patch set for 2.4.19pre9aa2. I've archived it at:
Quoted 8 lines
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19/patches/10_inode-highmem-2
For 2.4.19, this solves _most_ of the stability issues, but I still have to work with the LVM people and possibly whomever is responsible for the VM in 2.4.19/2.4.20 to track down some kernel oopses (possibly a seperate problem.)
I will happily provide whatever other information is needed, though my opportunities to test things on the machine in question is limited by the fact that it's a production server.
You might need bh stuff (memclass-related or something like it) if it's general disk io. Can't be too sure until slabinfo + meminfo materialize.
-- wli
Quoted 4 lines · William Lee Irwin III
Quoted 2 lines
Full details are at
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20Hmm, slabinfo would be very helpful, as well as meminfo.
I'll have to schedule a reboot into that kernel, but I'll try to get it tonight if at all possible.
Quoted 2 lines
You might need bh stuff (memclass-related or something like it) if it's general disk io. Can't be too sure until slabinfo + meminfo materialize.
I'm not familiar with "bh"... where can I read up on what it is?
Thanks again,
Gregory
Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]> Sr. Systems Administrator Cast & Crew Entertainment Services, Inc.
Quoted 1 line · William Lee Irwin III
Quoted 1 line
Hmm, slabinfo would be very helpful, as well as meminfo.
Quoted 2 lines · Gregory K. Ruiz-Ade
I'll have to schedule a reboot into that kernel, but I'll try to get it tonight if at all possible.
That's fine; I'm not in a hurry. =)
Quoted 2 lines · William Lee Irwin III
Quoted 2 lines
You might need bh stuff (memclass-related or something like it) if it's general disk io. Can't be too sure until slabinfo + meminfo materialize.
Quoted 1 line · Gregory K. Ruiz-Ade
I'm not familiar with "bh"... where can I read up on what it is?
bh == buffer_head. fs/buffer_head.c has the code.
-- wli
Quoted 8 lines · William Lee Irwin III
Quoted 1 line · William Lee Irwin III
Quoted 1 line
Hmm, slabinfo would be very helpful, as well as meminfo.
Quoted 2 lines · Gregory K. Ruiz-Ade
I'll have to schedule a reboot into that kernel, but I'll try to get it tonight if at all possible.
That's fine; I'm not in a hurry. =)
I got an opportunity, so the contents of cpuinfo, slabinfo, and meminfo are
at: http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20
Hopefully they're useful.
Additionally, I managed to generate an Oops (processed with ksymoops, too) while trying to create an LVM snapshot. The oops and it's ksymoops-translated file are up there too. The oops traces back into the VM somewhere.
Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]> Sr. Systems Administrator Cast & Crew Entertainment Services, Inc.
Quoted 7 lines · Gregory K. Ruiz-Ade
I got an opportunity, so the contents of cpuinfo, slabinfo, and meminfo are
at: http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20
Hopefully they're useful.
Additionally, I managed to generate an Oops (processed with ksymoops, too) while trying to create an LVM snapshot. The oops and it's ksymoops-translated file are up there too. The oops traces back into the VM somewhere.
Hmm, neither slabinfo nor meminfo show the machine being under any stress. Were they generated while the problem was happening?
The useful information would be to collect meminfo and slabinfo while kswapd and updated are spinning. Also, cpuinfo doesn't ever change, (at least while being run on the same box) so you can leave that out.
BTW, oopses tracing back into the VM doesn't help. It's usually someone doing something wrong the VM checks for. In this case I'll bet someone (i.e. LVM) called vmalloc() with interrupts off.
-- wli
Quoted 6 lines · William Lee Irwin III
Hmm, neither slabinfo nor meminfo show the machine being under any stress. Were they generated while the problem was happening?
The useful information would be to collect meminfo and slabinfo while kswapd and updated are spinning. Also, cpuinfo doesn't ever change, (at least while being run on the same box) so you can leave that out.
Ahh. I was a bit out of it yesterday, and didn't think to actually stress the machine. :\
I'll be able to give it a good beating this weekend sometime.
Quoted 3 lines
BTW, oopses tracing back into the VM doesn't help. It's usually someone doing something wrong the VM checks for. In this case I'll bet someone (i.e. LVM) called
vmalloc()with interrupts off.
Hmm... Okay, mind if I quote you when I post that oops the the lvm list? :)
Thanks again,
Gregory
Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]> Sr. Systems Administrator Cast & Crew Entertainment Services, Inc.
Quoted 5 lines · William Lee Irwin III
Quoted 5 lines
Hmm, neither slabinfo nor meminfo show the machine being under any stress. Were they generated while the problem was happening? The useful information would be to collect meminfo and slabinfo while kswapd and updated are spinning. Also, cpuinfo doesn't ever change, (at least while being run on the same box) so you can leave that out.
Quoted 3 lines · Gregory K. Ruiz-Ade
Ahh. I was a bit out of it yesterday, and didn't think to actually stress the machine. :\
I'll be able to give it a good beating this weekend sometime.
Quoted 3 lines · William Lee Irwin III
Quoted 3 lines
BTW, oopses tracing back into the VM doesn't help. It's usually someone doing something wrong the VM checks for. In this case I'll bet someone (i.e. LVM) called
vmalloc()with interrupts off.
Quoted 1 line · Gregory K. Ruiz-Ade
Hmm... Okay, mind if I quote you when I post that oops the the lvm list? :)
Understand that was said in the context of finding the VM bug. I'm not interested in LVM bugs, legitimate though they may be, mostly b/c it's not my project and I can't save the world.
-- wli
Quoted 6 lines · William Lee Irwin III
Quoted 3 lines · Gregory K. Ruiz-Ade
Ahh. I was a bit out of it yesterday, and didn't think to actually stress the machine. :\
I'll be able to give it a good beating this weekend sometime.cc me when you post those results.
Okay, I tried to load the system a bit and stress out the disk I/O, running a couple finds across the whole system (find | xargs stat, find | xargs cat
Quoted 1 line
/dev/null, a couple other things) after sucking up free memory by catting
our database disk files to /dev/null. I also had a 'make -j5 clean oldconfig dep bzImage modules' running to try to drive the load up a bit, too.
I've got snapshots of meminfo, slabinfo, and output from 'ps auxfww' at:
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20/loadtest/
It only really starts getting interesting after 20030316.1725, when I started the kernel build. I have a very simple shell script that basically does nothing other than "make clean oldconfig dep && make -j5 bzImage && make -j5 modules". I ran that a couple times in the sources for Red Hat's 2.4.9-e.12 kernel sources.
Surprisingly I wasn't able to grind down the system like I expected. Not sure why it's behaving so wonderfully today.
It crashed again on Friday night, running 2.4.19. The only information I was able to get was a kernel BUG message on the serial console (I ksymoops'ed it after rebooting). From what I could tell after the fact, nothing was really running. Several scripts got fired off by cron, which check various things (mainly to make sure certain services are still running), and around then is when the system locked up. The info I have for that crash is at:
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19
As it is, I'm going to try running on Red Hat's 2.4.9-e.12 sources this week. I'm compiling the kernel right now, and will be rebooting into it shortly.
This has been quite the week of headaches for me.
Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]> Sr. Systems Administrator Cast & Crew Entertainment Services, Inc.
Quoted 15 lines · Gregory K. Ruiz-Ade
Okay, I tried to load the system a bit and stress out the disk I/O, running a couple finds across the whole system (find | xargs stat, find | xargs cat
Quoted 1 line
/dev/null, a couple other things) after sucking up free memory by cattingour database disk files to
/dev/null. I also had a 'make -j5 clean oldconfig dep bzImage modules' running to try to drive the load up a bit, too.
I've got snapshots of meminfo, slabinfo, and output from 'ps auxfww' at:
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20/loadtest/
It only really starts getting interesting after 20030316.1725, when I started the kernel build. I have a very simple shell script that basically does nothing other than "make clean oldconfig dep && make -j5 bzImage && make -j5 modules". I ran that a couple times in the sources for Red Hat's 2.4.9-e.12 kernel sources.
Surprisingly I wasn't able to grind down the system like I expected. Not sure why it's behaving so wonderfully today.
If it didn't behave badly then it won't help to look at the stats.
-- wli
Quoted 1 line · William Lee Irwin III
If it didn't behave badly then it won't help to look at the stats.
I'm seriously at my wits end. (Not because of you! I hate vague problems...) I have no idea why it behaved itself this time, just like i have no idea why it misbehaves.
Right now, I'm back to running 2.4.19 with the inode.c patch from one of the 2.4.19-preXX-aaY kernels (see
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19/patches) as the most stable thing we've gotten so far.
I'm going to write some scripts to run out of cron ever 5 minutes (or maybe even every minute) to collect meminfo, slabinfo, ps output, and whatever else i can think of. What else would be useful to help you track down these problems?
Hopefully, the next time the system goes to hell, I'll have _something_ to give you.
As a side question, is bigmem >2GB? I.e., if I pass "mem=2048m" to the kernel from lilo, will the bigmem stuff for the VM be disabled, or should I instead build a new kernel with high memory support turned off? Also, with highmem support turned off, the max memory is 2GB, right? I may well just ignore the high 6GB out of desperation to get a stable system until 2.6 is released.
Thanks again for taking the time.
Working on a migrain,
Gregory
Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]> Sr. Systems Administrator Cast & Crew Entertainment Services, Inc.
Quoted 4 lines · Gregory K. Ruiz-Ade
Right now, I'm back to running 2.4.19 with the
inode.cpatch from one of the 2.4.19-preXX-aaY kernels (see
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19/patches) as the most stable thing we've gotten so far.
Could you possibly take a look at the kernel BUG that I got Friday night on the 2.4.19 kernel, to see if it (oh god please) hopefully points in a useful direction?
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19
The files are:
burpr-kernel-bug.20030314.2100 (the raw bug) burpr-kernel-bug.20030314.2100.ksymoops (processed through ksymoops)
FYI, the amaivschk program listed in the BUG is a shell script, and also at that URL if you're curious as to what it does.
Thanks again...
Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]> Sr. Systems Administrator Cast & Crew Entertainment Services, Inc.
Quoted 5 lines
The primary problem: Whenever any process (or set of processes) initiates intensive disk I/O, the system grinds to a halt, kswapd and kupdated consume upwards of 40% to 60% CPU each, and system load averages can jump upwards of 21.00. The problem can be replicated with a simple find command ("find / -print" seems to do it nicely).
Well known set of problems.
2.4 vm sucks on big machines. Run 2.5, 2.4-aa, or UL.
Yes, you can spend a few weeks beating your head against a brick wall gathering various bugfixes if you like ... but you'll probably just end up with a sore head ...
M.