Skip to contentSkip to conversation

bad command responsiveness Proliant DL 585

DISCUSSION David Osojnik 2 participants 8 messages Jun 14, 2006 – Jun 15, 2006 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hello,

We have a problem with four HP Proliant DL 585 servers with 4 AMD Opteron processors and 16Gb of memory and 3x 300Gb U320 SCSI disks and with all the latest firmware. we noticed bad command responsiveness in an production environment and poor performance (web server and mysql) The problem is good reproducible if creating a large file 30Gb in size with:

time dd if=/dev/zero of=test.file bs=3072 count=10240000

on the root partition no matter which one reiserfs, ext3

what happens is I open three more console windows and do random commands
like: "ls, w, route -n, ifconfig" but the commands freezes for random time (this time is from 1 minute to 15minutes!! per command execution time) when the command starts working (after 5minutes) i try it again and the command freezes again for a random time... the strange thing is that if one command freezes all other commands freeze too when one starts to work others work too. (if running top it stops refreshing for the lockup period)

we tried raid0, raid1, raid5 its the same and we even tried some other raid controllers other then default onboard Smart Array 5i we tried Smart Array 6402 and MegaRAID SCSI 320-2X. The problem is still there the only difference is writing speed but the lockups are still there!

and we tried this on four different DL 585 servers with all the same problem!

load when creating 32Gb file was around 10 (starting creation of the file with load 0.05) and the system did not run any services except the default (after a fresh minimum install of more then one distribution) we tried it too with all services shutdown and as much modules unloaded as we could remove.

we tried lots of kernels and cciss modules but all have the problem including 2.6.16.20

only the kernel 2.4 was working better there were still the lockups but were less often and commands executed faster around 50%.

why am i writing to kernel list because it looks like a kernel problem to me since the kernel 2.4 looks much better.

hope someone can help because we are getting really desperate

thanks

#1 lore raw
Quoted 12 lines · David Osojnik

Hello,

We have a problem with four HP Proliant DL 585 servers with 4 AMD Opteron processors and 16Gb of memory and 3x 300Gb U320 SCSI disks and with all the latest firmware. we noticed bad command responsiveness in an production environment and poor performance (web server and mysql) The problem is good reproducible if creating a large file 30Gb in size with:

time dd if=/dev/zero of=test.file bs=3072 count=10240000

on the root partition no matter which one reiserfs, ext3

Quoted 8 lines

what happens is I open three more console windows and do random commands
like: "ls, w, route -n, ifconfig" but the commands freezes for random time (this time is from 1 minute to 15minutes!! per command execution time) when the command starts working (after 5minutes) i try it again and the command freezes again for a random time... the strange thing is that if one command freezes all other commands freeze too when one starts to work others work too. (if running top it stops refreshing for the lockup period)

Does top freeze if started from an mlockall(MCL_PRESENT|MCL_FUTURE) shell running at realtime priority?

Try SysRq-T and SysRq-M during freezes to gather info about VM and task state during freeze.

-Mike

#2 lore raw

here is the output of SysRq-T and SysRq-M:

http://www.dworf.com/sysrq.txt

any ideas?

thanks

Quoted 14 lines · Mike Galbraith

Does top freeze if started from an mlockall(MCL_PRESENT|MCL_FUTURE) shell running at realtime priority?

Try SysRq-T and SysRq-M during freezes to gather info about VM and task state during freeze.

-Mike

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
Quoted 5 lines · David Osojnik

here is the output of SysRq-T and SysRq-M:

http://www.dworf.com/sysrq.txt

any ideas?

Not really.

I see I/O jammed up on reiserfs:.text.lock.journal, but you said reiserfs and ext3 both stall the same way. If the journal is in the raid, I'd try moving it, but I can't really imagine seek troubles leading to 15 minutes of grinding. I noticed that those last two instances of bash got nailed because of atime. Do things get any better if mounted noatime, nodiratime?

-Mike

#4 lore raw

Hello,

IT Works perfect when setting noatime,nodiratime on the partition!!

can i try anything else? what does this actually mean?

thanks!!

David

Quoted 29 lines · Mike Galbraith
Quoted 7 lines · David Osojnik

here is the output of SysRq-T and SysRq-M:

http://www.dworf.com/sysrq.txt

any ideas?

Not really.

I see I/O jammed up on reiserfs:.text.lock.journal, but you said reiserfs and ext3 both stall the same way. If the journal is in the raid, I'd try moving it, but I can't really imagine seek troubles leading to 15 minutes of grinding. I noticed that those last two instances of bash got nailed because of atime. Do things get any better if mounted noatime, nodiratime?

-Mike

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 3 lines · David Osojnik

Hello,

IT Works perfect when setting noatime,nodiratime on the partition!!

That's good to hear... sort of.

Quoted 2 lines

can i try anything else? what does this actually mean?

Besides having a constipated journal sucks rocks? ;-) Dunno. You could try a different elevator as a shot in the dark - eliminate something.

-Mike

#7 lore raw

Well i tired cfq,anticipatory,deadline,no-op schedulers/elevators with atime but none worked the only difference is when I use noatime and nodiratime

could this be an kernel problem?

David

Quoted 28 lines · Mike Galbraith
Quoted 5 lines · David Osojnik

Hello,

IT Works perfect when setting noatime,nodiratime on the partition!!

That's good to hear... sort of.

Quoted 3 lines

can i try anything else? what does this actually mean?

Besides having a constipated journal sucks rocks? ;-) Dunno. You could try a different elevator as a shot in the dark - eliminate something.

-Mike

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#8 lore raw
Quoted 3 lines · David Osojnik

Well i tired cfq,anticipatory,deadline,no-op schedulers/elevators with atime but none worked the only difference is when I use noatime and nodiratime

That's the result I expected.

Quoted 1 line

could this be an kernel problem?

Sure seems like one to me. Perhaps someone with a good understanding of journaling fs will comment. I can only speculate.

-Mike