Skip to contentSkip to conversation

bizzare slowdowns and hangs w/ 2.4.18, 2.4.19

DISCUSSION Gregory K. Ade 3 participants 3 messages Oct 27, 2002 – Oct 28, 2002

lore
Share & tools
lore

Gregory K. Ade

author
#0 lore raw

I am posting this here somewhat out of desperation, as I'm trying to find a good explaination for some bizzare slowdowns and lockups and have been turning up empty so far.

We have a Dell PowerEdge 6600, 4x 1.6HGz Xeon, 8GB RAM, PERC3/DCL (MegaRAID) controller with 128MB cache, driving two channels each with 3 36GB 15krpm UltraSCSI/LVD 160 drives. I currently have 8GB of swapspace set up (4 2gb swapspaces, in descending priority.) It's a real beast.

When it's been running for a couple hours, doing a find / will drag it to it's knees, as evidenced by this scraping from top during one of it's bad moments:

--->8--[ Cut Here ]--->8--
1:17pm up 14:34, 4 users, load average: 0.83, 0.98, 1.11
78 processes: 76 sleeping, 1 running, 1 zombie, 0 stopped

CPU0 states:  0.0% user, 53.0% system,  0.0% nice, 46.0% idle
CPU1 states:  0.1% user, 41.1% system,  0.0% nice, 57.0% idle
CPU2 states:  0.1% user, 42.0% system,  0.0% nice, 56.1% idle
CPU3 states:  0.0% user, 41.1% system,  0.0% nice, 58.0% idle
Mem:  7762204K av, 7756320K used,    5884K free,       0K shrd,  236704K buff
Swap: 8388544K av,     432K used, 8388112K free                 7260064K cached
  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
 8595 gregory    9   0   664  664   528 D    41.3  0.0   0:02 find /usr
 7195 root       9   0  2080 2080  1476 S    40.2  0.0   0:02 /usr/sbin/sshd
  333 root       9   0 15252  14M 15072 D    35.0  0.1   0:38 /usr/sbin/MegaSer
 7115 root      10   0  2092 2092  1476 S    28.1  0.0   0:25 /usr/sbin/sshd
    7 root      15   0     0    0     0 SW   22.9  0.0   3:06 kswapd
 7300 root      10   0   928  928   720 R    14.9  0.0   1:19 top
    1 root       8   0   224  224   184 S     0.0  0.0   1:21 init [3]
    2 root       9   0     0    0     0 SW    0.0  0.0   0:01 keventd
    3 root      19  19     0    0     0 SWN   0.0  0.0   0:00 ksoftirqd_CPU0
    4 root      18  19     0    0     0 SWN   0.0  0.0   0:00 ksoftirqd_CPU1

--->8--[ Cut Here ]--->8--

This was still on it's way to total oblivion. Once it's _at_ oblivion, all 4 CPU's are at 100% system, with find, kswapd and kupdated usually the top three processes, but always find & kswapd are up at the top.

Something as simple as a find should not do this, and doesn't, even on much whimpier machines (single 300mhz P-III with a single IDE drive & 128MB ram doesn't even break a sweat.)

Software: Kernel v. 2.4.19 (from kernel.org), patched with LVM 1.0.5 and Dell's BroadCom Gigabit Ethernet drivers. Userland LVM tools are from the same 1.0.5 sources. Using the kernel megaraid driver for the RAID controller.

So far, on the advice of one of the Dell developers for the megaraid driver, I've tried flashing the PERC card back to firmware 1.61O (it shipped with 1.72). This didn't seem to have much positive effect. I've since flashed up to 1.73, and am letting the system warm up a bit to see if it'll have any affect.

The problem never manifests itself within an hour so or so of startup, but after a certain amount of time, it becomes progressively more noticable. After 21 days of uptime, the system became so unresponsive on a Thursday morning (our busiest day) that we had to restart.

So, what other information can I provide to make the problem more diagnosable, and where else can I look for an explaination?

I have been unable to replicate this on any other hardware. I have at least three other systems running the same version of SuSE Linux (7.2), all with the same version of the kernel and LVM as mentioned above. Unfortunately, this Dell PE6600 is the only one of its kind that we own...

Thanks in advance for any help. Feel free to reply to the list, as I am subscribed.

Signature · 3 lines
Gregory K. Ade <[email protected]>
Sr. Systems Administrator
Cast & Crew Entertainment Services, Inc.
#2 lore raw

I'm Having a similar problem with Mandrake 2.4.19-16mdksmp kernel...

I'm not into the mood to see applied patches, but I will try a newer Kernel, my only complain is that I have NEVER succesfully compiled a CD-RW capable kernel, any strong advice on that ?? , i have tried several configurations, read tutorials, lots of kernels (vanilla an ac's).

Best Regards
Max