Understanding I/O behaviour
Hi,
for a customer we are operating a rackful of HP/DL380/G4 boxes that
have given us some problems with system responsiveness under [I/O triggered] system load.
The systems in question have the following HW:
2x Intel/EM64T CPUs
8GB memory
CCISS Raid controller with 4x72GB SCSI disks as RAID5 2x BCM5704 NIC (using tg3)
The distribution is RHEL4. We have tested several kernels including
the original 2.6.9, 2.6.19.2, 2.6.22-rc7 and 2.6.22-rc7+cfs-v18.
One part of the workload is when several processes try to write 5 GB
each to the local filesystem (ext2->LVM->CCISS). When this happens, the load goes up to 12 and responsiveness goes down. This means from one moment to the next things like opening a ssh connection to the host in question, or doing "df" take forever (minutes). Especially bad with the vendor kernel, better (but not perfect) with 2.6.19 and 2.6.22-rc7.
The load basically comes from the writing processes and up to 12
"pdflush" threads all being in "D" state.
So, what I would like to understand is how we can maximize the
responsiveness of the system, while keeping disk throughput at maximum.
During my investiogation I basically performed the following test,
because it represents the kind of trouble situation:
----
$ cat dd3.sh echo "Start 3 dd processes: "`date` dd if=/dev/zero of=/scratch/X1 bs=1M count=5000& dd if=/dev/zero of=/scratch/X2 bs=1M count=5000& dd if=/dev/zero of=/scratch/X3 bs=1M count=5000& wait echo "Finish 3 dd processes: "`date` sync echo "Finish sync: "`date` rm -f /scratch/X? echo "Files removed: "`date` ----
This results in the following timings. All with the anticipatory
scheduler, because it gives the best results:
2.6.19.2, HT: 10m
2.6.19.2, non-HT: 8m45s
2.6.22-rc7, HT: 10m
2.6.22-rc7, non-HT: 6m
2.6.22-rc7+cfs_v18, HT: 10m40s
2.6.22-rc7+cfs_v18, non-HT: 10m45s
The "felt" responsiveness was best with the last two kernels, although
the load profile over time looks identical in all cases.
So, a few questions:
a) any idea why disabling HT improves throughput, except for the cfs kernels? For plain 2.6.22 the difference is quite substantial
b) any ideas how to optimize the settings of the /proc/sys/vm/ parameters? The documentation is a bit thin here.
Thanks in advance
Martin
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
Quoted 56 lines · Martin Knoblauch
Hi,
for a customer we are operating a rackful of HP/DL380/G4 boxes that
have given us some problems with system responsiveness under [I/O triggered] system load.The systems in question have the following HW:
2x Intel/EM64T CPUs
8GB memory
CCISS Raid controller with 4x72GB SCSI disks as RAID5 2x BCM5704 NIC (using tg3)The distribution is RHEL4. We have tested several kernels including
the original 2.6.9, 2.6.19.2, 2.6.22-rc7 and 2.6.22-rc7+cfs-v18.One part of the workload is when several processes try to write 5 GB
each to the local filesystem (ext2->LVM->CCISS). When this happens, the load goes up to 12 and responsiveness goes down. This means from one moment to the next things like opening a ssh connection to the host in question, or doing "df" take forever (minutes). Especially bad with the vendor kernel, better (but not perfect) with 2.6.19 and 2.6.22-rc7.The load basically comes from the writing processes and up to 12
"pdflush" threads all being in "D" state.So, what I would like to understand is how we can maximize the
responsiveness of the system, while keeping disk throughput at maximum.During my investiogation I basically performed the following test,
because it represents the kind of trouble situation:----
$ cat dd3.sh echo "Start 3 dd processes: "`date` dd if=/dev/zero of=/scratch/X1 bs=1M count=5000& dd if=/dev/zero of=/scratch/X2 bs=1M count=5000& dd if=/dev/zero of=/scratch/X3 bs=1M count=5000& wait echo "Finish 3 dd processes: "`date` sync echo "Finish sync: "`date` rm -f /scratch/X? echo "Files removed: "`date` ----This results in the following timings. All with the anticipatory
scheduler, because it gives the best results:2.6.19.2, HT: 10m
2.6.19.2, non-HT: 8m45s
2.6.22-rc7, HT: 10m
2.6.22-rc7, non-HT: 6m
2.6.22-rc7+cfs_v18, HT: 10m40s
2.6.22-rc7+cfs_v18, non-HT: 10m45s
Quoted 7 lines
The "felt" responsiveness was best with the last two kernels, although
the load profile over time looks identical in all cases.So, a few questions:
a) any idea why disabling HT improves throughput, except for the cfs kernels? For plain 2.6.22 the difference is quite substantial
Under certain loads HT can reduce performance, I have had serious performance problems on windows terminal servers with HT enabled, and I now disable it on all servers, no matter what OS they run.
Why? http://blogs.msdn.com/slavao/archive/2005/11/12/492119.aspx
Quoted 2 lines
b) any ideas how to optimize the settings of the
/proc/sys/vm/parameters? The documentation is a bit thin here.
I cant offer any advice there, but is raid-5 really the best choice for your needs? I would not choose raid-5 for a system that is regularly performing lots of large writes at the same time, dont forget that each write can require several reads to recalculate the partity.
Does the raid card have much cache ram?
If you can afford to loose some space raid-10 would probably perform better.
Andy
Quoted 14 lines
Thanks in advance
Martin------------------------------------------------------
Martin Knoblauchemail: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 23 lines · Martin Knoblauch
Hi,
for a customer we are operating a rackful of HP/DL380/G4 boxes that
have given us some problems with system responsiveness under [I/O triggered] system load.The systems in question have the following HW:
2x Intel/EM64T CPUs
8GB memory
CCISS Raid controller with 4x72GB SCSI disks as RAID5 2x BCM5704 NIC (using tg3)The distribution is RHEL4. We have tested several kernels including
the original 2.6.9, 2.6.19.2, 2.6.22-rc7 and 2.6.22-rc7+cfs-v18.One part of the workload is when several processes try to write 5 GB
each to the local filesystem (ext2->LVM->CCISS). When this happens, the load goes up to 12 and responsiveness goes down. This means from one moment to the next things like opening a ssh connection to the host in question, or doing "df" take forever (minutes). Especially bad with the vendor kernel, better (but not perfect) with 2.6.19 and 2.6.22-rc7.
Quoted 6 lines
The load basically comes from the writing processes and up to 12
"pdflush" threads all being in "D" state.So, what I would like to understand is how we can maximize the
responsiveness of the system, while keeping disk throughput at maximum.
I'd suspect you can't get both at 100%.
I'd guess you are probably using a 100Hz no-preempt kernel. Have you
tried a 1000Hz + preempt kernel? Sure, you'll get a bit lower
overall throughput, but interactive responsiveness should be better - if it is, then you could experiment with various combinations of CONFIG_PREEMPT, CONFIG_PREEMPT_VOLUNTARY, CONFIG_PREEMPT_NONE and CONFIG_HZ_1000, CONFIG_HZ_300, CONFIG_HZ_250, CONFIG_HZ_100 to see what gives you the best balance between throughput and interactive responsiveness (you could also throw CONFIG_PREEMPT_BKL and/or CONFIG_NO_HZ, but I don't think the impact will be as significant as with the other options, so to keep things simple I'd leave those out at first) .
I'd guess that something like CONFIG_PREEMPT_VOLUNTARY + CONFIG_HZ_300 would probably be a good compromise for you, but just to see if there's any effect at all, start out with CONFIG_PREEMPT + CONFIG_HZ_1000.
Hope that helps.
(PS. please don't do crap like using that spamtrap@ address and have people manually replace it with the one from your .signature when posting on LKML - it's annoying as hell)
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Quoted 45 lines · Jesper Juhl
Quoted 23 lines · Martin Knoblauch
Hi,
for a customer we are operating a rackful of HP/DL380/G4 boxes that
have given us some problems with system responsiveness under [I/O triggered] system load.The systems in question have the following HW:
2x Intel/EM64T CPUs
8GB memory
CCISS Raid controller with 4x72GB SCSI disks as RAID5 2x BCM5704 NIC (using tg3)The distribution is RHEL4. We have tested several kernels including
the original 2.6.9, 2.6.19.2, 2.6.22-rc7 and 2.6.22-rc7+cfs-v18.One part of the workload is when several processes try to write 5 GB
each to the local filesystem (ext2->LVM->CCISS). When this happens, the load goes up to 12 and responsiveness goes down. This means from one moment to the next things like opening a ssh connection to the host in question, or doing "df" take forever (minutes). Especially bad with the vendor kernel, better (but not perfect) with 2.6.19 and 2.6.22-rc7.Quoted 6 lines
The load basically comes from the writing processes and up to 12
"pdflush" threads all being in "D" state.So, what I would like to understand is how we can maximize the
responsiveness of the system, while keeping disk throughput at maximum.I'd suspect you can't get both at 100%.
I'd guess you are probably using a 100Hz no-preempt kernel. Have you
tried a 1000Hz + preempt kernel? Sure, you'll get a bit lower
overall throughput, but interactive responsiveness should be better - if it is, then you could experiment with various combinations ofCONFIG_PREEMPT,CONFIG_PREEMPT_VOLUNTARY,CONFIG_PREEMPT_NONEandCONFIG_HZ_1000,CONFIG_HZ_300,CONFIG_HZ_250,CONFIG_HZ_100to see what gives you the best balance between throughput and interactive responsiveness (you could also throwCONFIG_PREEMPT_BKLand/orCONFIG_NO_HZ, but I don't think the impact will be as significant as with the other options, so to keep things simple I'd leave those out at first) .
Quoted 5 lines
I'd guess that something like
CONFIG_PREEMPT_VOLUNTARY+CONFIG_HZ_300would probably be a good compromise for you, but just to see if there's any effect at all, start out withCONFIG_PREEMPT+CONFIG_HZ_1000.
I'm currious, did you ever try playing around with CONFIG_PREEMPT* and CONFIG_HZ* to see if that had any noticable impact on interactive performance and stuff like logging into the box via ssh etc...?
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Quoted 30 lines · --- Jesper Juhl
Quoted 7 lines · Jesper Juhl
Quoted 2 lines · Martin Knoblauch
Hi,
I'd suspect you can't get both at 100%.
I'd guess you are probably using a 100Hz no-preempt kernel. Have
you
Quoted 2 lines
tried a 1000Hz + preempt kernel? Sure, you'll get a bit lower
overall throughput, but interactive responsiveness should be better-
Quoted 6 lines
if it is, then you could experiment with various combinations of
CONFIG_PREEMPT,CONFIG_PREEMPT_VOLUNTARY,CONFIG_PREEMPT_NONEandCONFIG_HZ_1000,CONFIG_HZ_300,CONFIG_HZ_250,CONFIG_HZ_100to see what gives you the best balance between throughput and interactive responsiveness (you could also throwCONFIG_PREEMPT_BKLand/orCONFIG_NO_HZ, but I don't think the impact will be as significantas
Quoted 1 line
with the other options, so to keep things simple I'd leave those
out
Quoted 3 lines
at first) .
I'd guess that something like
CONFIG_PREEMPT_VOLUNTARY+
CONFIG_HZ_300Quoted 4 lines
would probably be a good compromise for you, but just to see if there's any effect at all, start out with
CONFIG_PREEMPT+CONFIG_HZ_1000.
Quoted 11 lines
I'm currious, did you ever try playing around with CONFIG_PREEMPT* and
CONFIG_HZ* to see if that had any noticable impact on interactive performance and stuff like logging into the box via ssh etc...?Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Hi Jesper,
my initial kernel was voluntary@100HZ. I have switched to 300HZ, but
have not observed much difference. The config is now:
config-2.6.22-rc7:# CONFIG_PREEMPT_NONE is not set config-2.6.22-rc7:CONFIG_PREEMPT_VOLUNTARY=y config-2.6.22-rc7:# CONFIG_PREEMPT is not set config-2.6.22-rc7:CONFIG_PREEMPT_BKL=y
Cheers
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de