Skip to contentSkip to conversation

2.4.20 instability on bigmem systems?

DISCUSSION Gregory K. Ruiz-Ade 3 participants 13 messages Mar 14, 2003 – Mar 17, 2003

lore
Share & tools
lore
#0 lore raw

This started out as my asking about random crashes on 2.4.19, but in the course of me trying to figure out what was going on, I got to experience first-hand the increadibly poor performance of 2.4.x (notably 2.4.20) on a system with >4GB of memory (8GB).

So, I'm looking for a solution, preferrably a set of patches, to help with the problems described below.

On Monday, I installed a 2.4.20 kernel on our Dell PowerEdge 6600. This machine is configured as follows:

4x 1.6Ghz Xeon CPUs
8GB RAM
Built-in:
ATI Rage 128 graphics
dual Broadcomm Gigabit Ethernet serial/parallel/usb
Adaptec SCSI
Dell PERC3/DC (AMI/LSI MegaRAID) dual-channel

The kernel was built from the linux-2.4.20.tar.bz2 from kernel.org, and patched with only the lvm-1.0.7 and linux-2.4.20 VFS locking patches from Sistina's LVM-1.0.7 package.

The primary problem: Whenever any process (or set of processes) initiates intensive disk I/O, the system grinds to a halt, kswapd and kupdated consume upwards of 40% to 60% CPU each, and system load averages can jump upwards of 21.00. The problem can be replicated with a simple find command ("find / -print" seems to do it nicely).

I have had two rather painful nights dealing with this (Monday and Tuesday nights). Luckily, I have a serial null-modem cable rigged up between the troubled server and another server, and was able to capture all the info from the Magic Sysrq commands that I could.

Full details are at http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20

I've included the kernel config, the kernel and initrd images, the system map file, output from "ps auxfww" and a couple screen scrapings from top, and captures from magic sysrq commands from both crashes.

I had problems like this with 2.4.19, and was directed to apply a patch to inode.c, which appears to be part of a patch set for 2.4.19pre9aa2. I've archived it at:

http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19/patches/10_inode-highmem-2

For 2.4.19, this solves _most_ of the stability issues, but I still have to work with the LVM people and possibly whomever is responsible for the VM in 2.4.19/2.4.20 to track down some kernel oopses (possibly a seperate problem.)

I will happily provide whatever other information is needed, though my opportunities to test things on the machine in question is limited by the fact that it's a production server.

Thanks in advance,
Gregory

Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]>
Sr. Systems Administrator
Cast & Crew Entertainment Services, Inc.
#1 lore raw
Quoted 10 lines · Gregory K. Ruiz-Ade

The primary problem: Whenever any process (or set of processes) initiates intensive disk I/O, the system grinds to a halt, kswapd and kupdated consume upwards of 40% to 60% CPU each, and system load averages can jump upwards of 21.00. The problem can be replicated with a simple find command ("find / -print" seems to do it nicely).
I have had two rather painful nights dealing with this (Monday and Tuesday nights). Luckily, I have a serial null-modem cable rigged up between the troubled server and another server, and was able to capture all the info from the Magic Sysrq commands that I could.
Full details are at http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20

Hmm, slabinfo would be very helpful, as well as meminfo.

Quoted 6 lines · Gregory K. Ruiz-Ade

I've included the kernel config, the kernel and initrd images, the system map file, output from "ps auxfww" and a couple screen scrapings from top, and captures from magic sysrq commands from both crashes.
I had problems like this with 2.4.19, and was directed to apply a patch to inode.c, which appears to be part of a patch set for 2.4.19pre9aa2. I've archived it at:

Quoted 8 lines

http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19/patches/10_inode-highmem-2
For 2.4.19, this solves _most_ of the stability issues, but I still have to work with the LVM people and possibly whomever is responsible for the VM in 2.4.19/2.4.20 to track down some kernel oopses (possibly a seperate problem.)
I will happily provide whatever other information is needed, though my opportunities to test things on the machine in question is limited by the fact that it's a production server.

You might need bh stuff (memclass-related or something like it) if it's general disk io. Can't be too sure until slabinfo + meminfo materialize.

-- wli

#2 lore raw
Quoted 4 lines · William Lee Irwin III
Quoted 2 lines

Full details are at
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20

Hmm, slabinfo would be very helpful, as well as meminfo.

I'll have to schedule a reboot into that kernel, but I'll try to get it tonight if at all possible.

Quoted 2 lines

You might need bh stuff (memclass-related or something like it) if it's general disk io. Can't be too sure until slabinfo + meminfo materialize.

I'm not familiar with "bh"... where can I read up on what it is?

Thanks again,
Gregory

Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]>
Sr. Systems Administrator
Cast & Crew Entertainment Services, Inc.
#3 lore raw
Quoted 1 line · William Lee Irwin III
Quoted 1 line

Hmm, slabinfo would be very helpful, as well as meminfo.

Quoted 2 lines · Gregory K. Ruiz-Ade

I'll have to schedule a reboot into that kernel, but I'll try to get it tonight if at all possible.

That's fine; I'm not in a hurry. =)

Quoted 2 lines · William Lee Irwin III
Quoted 2 lines

You might need bh stuff (memclass-related or something like it) if it's general disk io. Can't be too sure until slabinfo + meminfo materialize.

Quoted 1 line · Gregory K. Ruiz-Ade

I'm not familiar with "bh"... where can I read up on what it is?

bh == buffer_head. fs/buffer_head.c has the code.

-- wli

#4 lore raw
Quoted 8 lines · William Lee Irwin III
Quoted 1 line · William Lee Irwin III
Quoted 1 line

Hmm, slabinfo would be very helpful, as well as meminfo.

Quoted 2 lines · Gregory K. Ruiz-Ade

I'll have to schedule a reboot into that kernel, but I'll try to get it tonight if at all possible.

That's fine; I'm not in a hurry. =)

I got an opportunity, so the contents of cpuinfo, slabinfo, and meminfo are
at: http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20

Hopefully they're useful.

Additionally, I managed to generate an Oops (processed with ksymoops, too) while trying to create an LVM snapshot. The oops and it's ksymoops-translated file are up there too. The oops traces back into the VM somewhere.

Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]>
Sr. Systems Administrator
Cast & Crew Entertainment Services, Inc.
#5 lore raw
Quoted 7 lines · Gregory K. Ruiz-Ade

I got an opportunity, so the contents of cpuinfo, slabinfo, and meminfo are
at: http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20
Hopefully they're useful.
Additionally, I managed to generate an Oops (processed with ksymoops, too) while trying to create an LVM snapshot. The oops and it's ksymoops-translated file are up there too. The oops traces back into the VM somewhere.

Hmm, neither slabinfo nor meminfo show the machine being under any stress. Were they generated while the problem was happening?

The useful information would be to collect meminfo and slabinfo while kswapd and updated are spinning. Also, cpuinfo doesn't ever change, (at least while being run on the same box) so you can leave that out.

BTW, oopses tracing back into the VM doesn't help. It's usually someone doing something wrong the VM checks for. In this case I'll bet someone (i.e. LVM) called vmalloc() with interrupts off.

-- wli

#6 lore raw
Quoted 6 lines · William Lee Irwin III

Hmm, neither slabinfo nor meminfo show the machine being under any stress. Were they generated while the problem was happening?

The useful information would be to collect meminfo and slabinfo while kswapd and updated are spinning. Also, cpuinfo doesn't ever change, (at least while being run on the same box) so you can leave that out.

Ahh. I was a bit out of it yesterday, and didn't think to actually stress the machine. :\

I'll be able to give it a good beating this weekend sometime.

Quoted 3 lines

BTW, oopses tracing back into the VM doesn't help. It's usually someone doing something wrong the VM checks for. In this case I'll bet someone (i.e. LVM) called vmalloc() with interrupts off.

Hmm... Okay, mind if I quote you when I post that oops the the lvm list? :)

Thanks again,
Gregory

Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]>
Sr. Systems Administrator
Cast & Crew Entertainment Services, Inc.
#7 lore raw
Quoted 5 lines · William Lee Irwin III
Quoted 5 lines

Hmm, neither slabinfo nor meminfo show the machine being under any stress. Were they generated while the problem was happening? The useful information would be to collect meminfo and slabinfo while kswapd and updated are spinning. Also, cpuinfo doesn't ever change, (at least while being run on the same box) so you can leave that out.

Quoted 3 lines · Gregory K. Ruiz-Ade

Ahh. I was a bit out of it yesterday, and didn't think to actually stress the machine. :\
I'll be able to give it a good beating this weekend sometime.

cc me when you post those results.
Quoted 3 lines · William Lee Irwin III
Quoted 3 lines

BTW, oopses tracing back into the VM doesn't help. It's usually someone doing something wrong the VM checks for. In this case I'll bet someone (i.e. LVM) called vmalloc() with interrupts off.

Quoted 1 line · Gregory K. Ruiz-Ade

Hmm... Okay, mind if I quote you when I post that oops the the lvm list? :)

Understand that was said in the context of finding the VM bug. I'm not interested in LVM bugs, legitimate though they may be, mostly b/c it's not my project and I can't save the world.

-- wli

#8 lore raw
Quoted 6 lines · William Lee Irwin III
Quoted 3 lines · Gregory K. Ruiz-Ade

Ahh. I was a bit out of it yesterday, and didn't think to actually stress the machine. :\
I'll be able to give it a good beating this weekend sometime.

cc me when you post those results.

Okay, I tried to load the system a bit and stress out the disk I/O, running a couple finds across the whole system (find | xargs stat, find | xargs cat

Quoted 1 line

/dev/null, a couple other things) after sucking up free memory by catting

our database disk files to /dev/null. I also had a 'make -j5 clean oldconfig dep bzImage modules' running to try to drive the load up a bit, too.

I've got snapshots of meminfo, slabinfo, and output from 'ps auxfww' at:

http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20/loadtest/

It only really starts getting interesting after 20030316.1725, when I started the kernel build. I have a very simple shell script that basically does nothing other than "make clean oldconfig dep && make -j5 bzImage && make -j5 modules". I ran that a couple times in the sources for Red Hat's 2.4.9-e.12 kernel sources.

Surprisingly I wasn't able to grind down the system like I expected. Not sure why it's behaving so wonderfully today.

It crashed again on Friday night, running 2.4.19. The only information I was able to get was a kernel BUG message on the serial console (I ksymoops'ed it after rebooting). From what I could tell after the fact, nothing was really running. Several scripts got fired off by cron, which check various things (mainly to make sure certain services are still running), and around then is when the system locked up. The info I have for that crash is at:

http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19

As it is, I'm going to try running on Red Hat's 2.4.9-e.12 sources this week. I'm compiling the kernel right now, and will be rebooting into it shortly.

This has been quite the week of headaches for me.

Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]>
Sr. Systems Administrator
Cast & Crew Entertainment Services, Inc.
#9 lore raw
Quoted 15 lines · Gregory K. Ruiz-Ade

Okay, I tried to load the system a bit and stress out the disk I/O, running a couple finds across the whole system (find | xargs stat, find | xargs cat

Quoted 1 line

/dev/null, a couple other things) after sucking up free memory by catting

our database disk files to /dev/null. I also had a 'make -j5 clean oldconfig dep bzImage modules' running to try to drive the load up a bit, too.
I've got snapshots of meminfo, slabinfo, and output from 'ps auxfww' at:
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.20/loadtest/
It only really starts getting interesting after 20030316.1725, when I started the kernel build. I have a very simple shell script that basically does nothing other than "make clean oldconfig dep && make -j5 bzImage && make -j5 modules". I ran that a couple times in the sources for Red Hat's 2.4.9-e.12 kernel sources.
Surprisingly I wasn't able to grind down the system like I expected. Not sure why it's behaving so wonderfully today.

If it didn't behave badly then it won't help to look at the stats.

-- wli

#10 lore raw
Quoted 1 line · William Lee Irwin III

If it didn't behave badly then it won't help to look at the stats.

I'm seriously at my wits end. (Not because of you! I hate vague problems...) I have no idea why it behaved itself this time, just like i have no idea why it misbehaves.

Right now, I'm back to running 2.4.19 with the inode.c patch from one of the 2.4.19-preXX-aaY kernels (see
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19/patches) as the most stable thing we've gotten so far.

I'm going to write some scripts to run out of cron ever 5 minutes (or maybe even every minute) to collect meminfo, slabinfo, ps output, and whatever else i can think of. What else would be useful to help you track down these problems?

Hopefully, the next time the system goes to hell, I'll have _something_ to give you.

As a side question, is bigmem >2GB? I.e., if I pass "mem=2048m" to the kernel from lilo, will the bigmem stuff for the VM be disabled, or should I instead build a new kernel with high memory support turned off? Also, with highmem support turned off, the max memory is 2GB, right? I may well just ignore the high 6GB out of desperation to get a stable system until 2.6 is released.

Thanks again for taking the time.

Working on a migrain,
Gregory

Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]>
Sr. Systems Administrator
Cast & Crew Entertainment Services, Inc.
#11 lore raw
Quoted 4 lines · Gregory K. Ruiz-Ade

Right now, I'm back to running 2.4.19 with the inode.c patch from one of the 2.4.19-preXX-aaY kernels (see
http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19/patches) as the most stable thing we've gotten so far.

Could you possibly take a look at the kernel BUG that I got Friday night on the 2.4.19 kernel, to see if it (oh god please) hopefully points in a useful direction?

http://castandcrew.com/~gregory/lkmlstuff/burpr/2.4.19

The files are:

burpr-kernel-bug.20030314.2100 (the raw bug) burpr-kernel-bug.20030314.2100.ksymoops (processed through ksymoops)

FYI, the amaivschk program listed in the BUG is a shell script, and also at that URL if you're curious as to what it does.

Thanks again...

Signature · 3 lines
Gregory K. Ruiz-Ade <[email protected]>
Sr. Systems Administrator
Cast & Crew Entertainment Services, Inc.
#12 lore raw
In reply to Gregory K. Ruiz-Ade
Quoted 5 lines

The primary problem: Whenever any process (or set of processes) initiates intensive disk I/O, the system grinds to a halt, kswapd and kupdated consume upwards of 40% to 60% CPU each, and system load averages can jump upwards of 21.00. The problem can be replicated with a simple find command ("find / -print" seems to do it nicely).

Well known set of problems.
2.4 vm sucks on big machines. Run 2.5, 2.4-aa, or UL.

Yes, you can spend a few weeks beating your head against a brick wall gathering various bugfixes if you like ... but you'll probably just end up with a sore head ...

M.