Skip to contentSkip to conversation

Understanding I/O behaviour

DISCUSSION Martin Knoblauch 3 participants 5 messages Jul 5, 2007 – Jul 9, 2007

lore
Share & tools
lore
#0 lore raw

Hi,

for a customer we are operating a rackful of HP/DL380/G4 boxes that
have given us some problems with system responsiveness under [I/O triggered] system load.

The systems in question have the following HW:

2x Intel/EM64T CPUs
8GB memory
CCISS Raid controller with 4x72GB SCSI disks as RAID5 2x BCM5704 NIC (using tg3)

The distribution is RHEL4. We have tested several kernels including
the original 2.6.9, 2.6.19.2, 2.6.22-rc7 and 2.6.22-rc7+cfs-v18.

One part of the workload is when several processes try to write 5 GB
each to the local filesystem (ext2->LVM->CCISS). When this happens, the load goes up to 12 and responsiveness goes down. This means from one moment to the next things like opening a ssh connection to the host in question, or doing "df" take forever (minutes). Especially bad with the vendor kernel, better (but not perfect) with 2.6.19 and 2.6.22-rc7.

The load basically comes from the writing processes and up to 12
"pdflush" threads all being in "D" state.

So, what I would like to understand is how we can maximize the
responsiveness of the system, while keeping disk throughput at maximum.

During my investiogation I basically performed the following test,
because it represents the kind of trouble situation:

----

$ cat dd3.sh
echo "Start 3 dd processes: "`date`
dd if=/dev/zero of=/scratch/X1 bs=1M count=5000&
dd if=/dev/zero of=/scratch/X2 bs=1M count=5000&
dd if=/dev/zero of=/scratch/X3 bs=1M count=5000&
wait
echo "Finish 3 dd processes: "`date`
sync
echo "Finish sync: "`date`
rm -f /scratch/X?
echo "Files removed: "`date`
----

This results in the following timings. All with the anticipatory
scheduler, because it gives the best results:

2.6.19.2, HT: 10m
2.6.19.2, non-HT: 8m45s
2.6.22-rc7, HT: 10m
2.6.22-rc7, non-HT: 6m
2.6.22-rc7+cfs_v18, HT: 10m40s
2.6.22-rc7+cfs_v18, non-HT: 10m45s

The "felt" responsiveness was best with the last two kernels, although
the load profile over time looks identical in all cases.

So, a few questions:

a) any idea why disabling HT improves throughput, except for the cfs kernels? For plain 2.6.22 the difference is quite substantial
b) any ideas how to optimize the settings of the /proc/sys/vm/ parameters? The documentation is a bit thin here.

Thanks in advance
Martin

------------------------------------------------------
Martin Knoblauch

email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de
#1 lore raw
Quoted 56 lines · Martin Knoblauch

Hi,

for a customer we are operating a rackful of HP/DL380/G4 boxes that
have given us some problems with system responsiveness under [I/O triggered] system load.

The systems in question have the following HW:

2x Intel/EM64T CPUs
8GB memory
CCISS Raid controller with 4x72GB SCSI disks as RAID5 2x BCM5704 NIC (using tg3)

The distribution is RHEL4. We have tested several kernels including
the original 2.6.9, 2.6.19.2, 2.6.22-rc7 and 2.6.22-rc7+cfs-v18.

One part of the workload is when several processes try to write 5 GB
each to the local filesystem (ext2->LVM->CCISS). When this happens, the load goes up to 12 and responsiveness goes down. This means from one moment to the next things like opening a ssh connection to the host in question, or doing "df" take forever (minutes). Especially bad with the vendor kernel, better (but not perfect) with 2.6.19 and 2.6.22-rc7.

The load basically comes from the writing processes and up to 12
"pdflush" threads all being in "D" state.

So, what I would like to understand is how we can maximize the
responsiveness of the system, while keeping disk throughput at maximum.

During my investiogation I basically performed the following test,
because it represents the kind of trouble situation:

----

$ cat dd3.sh
echo "Start 3 dd processes: "`date`
dd if=/dev/zero of=/scratch/X1 bs=1M count=5000&
dd if=/dev/zero of=/scratch/X2 bs=1M count=5000&
dd if=/dev/zero of=/scratch/X3 bs=1M count=5000&
wait
echo "Finish 3 dd processes: "`date`
sync
echo "Finish sync: "`date`
rm -f /scratch/X?
echo "Files removed: "`date`
----

This results in the following timings. All with the anticipatory
scheduler, because it gives the best results:

2.6.19.2, HT: 10m
2.6.19.2, non-HT: 8m45s
2.6.22-rc7, HT: 10m
2.6.22-rc7, non-HT: 6m
2.6.22-rc7+cfs_v18, HT: 10m40s
2.6.22-rc7+cfs_v18, non-HT: 10m45s

Quoted 7 lines

The "felt" responsiveness was best with the last two kernels, although
the load profile over time looks identical in all cases.

So, a few questions:

a) any idea why disabling HT improves throughput, except for the cfs kernels? For plain 2.6.22 the difference is quite substantial

Under certain loads HT can reduce performance, I have had serious performance problems on windows terminal servers with HT enabled, and I now disable it on all servers, no matter what OS they run.

Why? http://blogs.msdn.com/slavao/archive/2005/11/12/492119.aspx

Quoted 2 lines

b) any ideas how to optimize the settings of the /proc/sys/vm/ parameters? The documentation is a bit thin here.

I cant offer any advice there, but is raid-5 really the best choice for your needs? I would not choose raid-5 for a system that is regularly performing lots of large writes at the same time, dont forget that each write can require several reads to recalculate the partity.

Does the raid card have much cache ram?

If you can afford to loose some space raid-10 would probably perform better.

Andy

Quoted 14 lines

Thanks in advance
Martin

------------------------------------------------------
Martin Knoblauch

email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
In reply to Martin Knoblauch
Quoted 23 lines · Martin Knoblauch

Hi,

for a customer we are operating a rackful of HP/DL380/G4 boxes that
have given us some problems with system responsiveness under [I/O triggered] system load.

The systems in question have the following HW:

2x Intel/EM64T CPUs
8GB memory
CCISS Raid controller with 4x72GB SCSI disks as RAID5 2x BCM5704 NIC (using tg3)

The distribution is RHEL4. We have tested several kernels including
the original 2.6.9, 2.6.19.2, 2.6.22-rc7 and 2.6.22-rc7+cfs-v18.

One part of the workload is when several processes try to write 5 GB
each to the local filesystem (ext2->LVM->CCISS). When this happens, the load goes up to 12 and responsiveness goes down. This means from one moment to the next things like opening a ssh connection to the host in question, or doing "df" take forever (minutes). Especially bad with the vendor kernel, better (but not perfect) with 2.6.19 and 2.6.22-rc7.

Quoted 6 lines

The load basically comes from the writing processes and up to 12
"pdflush" threads all being in "D" state.

So, what I would like to understand is how we can maximize the
responsiveness of the system, while keeping disk throughput at maximum.

I'd suspect you can't get both at 100%.

I'd guess you are probably using a 100Hz no-preempt kernel. Have you
tried a 1000Hz + preempt kernel? Sure, you'll get a bit lower
overall throughput, but interactive responsiveness should be better - if it is, then you could experiment with various combinations of CONFIG_PREEMPT, CONFIG_PREEMPT_VOLUNTARY, CONFIG_PREEMPT_NONE and CONFIG_HZ_1000, CONFIG_HZ_300, CONFIG_HZ_250, CONFIG_HZ_100 to see what gives you the best balance between throughput and interactive responsiveness (you could also throw CONFIG_PREEMPT_BKL and/or CONFIG_NO_HZ, but I don't think the impact will be as significant as with the other options, so to keep things simple I'd leave those out at first) .

I'd guess that something like CONFIG_PREEMPT_VOLUNTARY + CONFIG_HZ_300 would probably be a good compromise for you, but just to see if there's any effect at all, start out with CONFIG_PREEMPT + CONFIG_HZ_1000.

Hope that helps.

(PS. please don't do crap like using that spamtrap@ address and have people manually replace it with the one from your .signature when posting on LKML - it's annoying as hell)

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#3 lore raw
Quoted 45 lines · Jesper Juhl
Quoted 23 lines · Martin Knoblauch

Hi,

for a customer we are operating a rackful of HP/DL380/G4 boxes that
have given us some problems with system responsiveness under [I/O triggered] system load.

The systems in question have the following HW:

2x Intel/EM64T CPUs
8GB memory
CCISS Raid controller with 4x72GB SCSI disks as RAID5 2x BCM5704 NIC (using tg3)

The distribution is RHEL4. We have tested several kernels including
the original 2.6.9, 2.6.19.2, 2.6.22-rc7 and 2.6.22-rc7+cfs-v18.

One part of the workload is when several processes try to write 5 GB
each to the local filesystem (ext2->LVM->CCISS). When this happens, the load goes up to 12 and responsiveness goes down. This means from one moment to the next things like opening a ssh connection to the host in question, or doing "df" take forever (minutes). Especially bad with the vendor kernel, better (but not perfect) with 2.6.19 and 2.6.22-rc7.

Quoted 6 lines

The load basically comes from the writing processes and up to 12
"pdflush" threads all being in "D" state.

So, what I would like to understand is how we can maximize the
responsiveness of the system, while keeping disk throughput at maximum.

I'd suspect you can't get both at 100%.

I'd guess you are probably using a 100Hz no-preempt kernel. Have you
tried a 1000Hz + preempt kernel? Sure, you'll get a bit lower
overall throughput, but interactive responsiveness should be better - if it is, then you could experiment with various combinations of CONFIG_PREEMPT, CONFIG_PREEMPT_VOLUNTARY, CONFIG_PREEMPT_NONE and CONFIG_HZ_1000, CONFIG_HZ_300, CONFIG_HZ_250, CONFIG_HZ_100 to see what gives you the best balance between throughput and interactive responsiveness (you could also throw CONFIG_PREEMPT_BKL and/or CONFIG_NO_HZ, but I don't think the impact will be as significant as with the other options, so to keep things simple I'd leave those out at first) .

Quoted 5 lines

I'd guess that something like CONFIG_PREEMPT_VOLUNTARY + CONFIG_HZ_300 would probably be a good compromise for you, but just to see if there's any effect at all, start out with CONFIG_PREEMPT + CONFIG_HZ_1000.

I'm currious, did you ever try playing around with CONFIG_PREEMPT* and CONFIG_HZ* to see if that had any noticable impact on interactive performance and stuff like logging into the box via ssh etc...?

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#4 lore raw
Quoted 30 lines · --- Jesper Juhl
Quoted 7 lines · Jesper Juhl
Quoted 2 lines · Martin Knoblauch

Hi,

I'd suspect you can't get both at 100%.

I'd guess you are probably using a 100Hz no-preempt kernel. Have

you

Quoted 2 lines

tried a 1000Hz + preempt kernel? Sure, you'll get a bit lower
overall throughput, but interactive responsiveness should be better

-

Quoted 6 lines

if it is, then you could experiment with various combinations of CONFIG_PREEMPT, CONFIG_PREEMPT_VOLUNTARY, CONFIG_PREEMPT_NONE and CONFIG_HZ_1000, CONFIG_HZ_300, CONFIG_HZ_250, CONFIG_HZ_100 to see what gives you the best balance between throughput and interactive responsiveness (you could also throw CONFIG_PREEMPT_BKL and/or CONFIG_NO_HZ, but I don't think the impact will be as significant

as

Quoted 1 line

with the other options, so to keep things simple I'd leave those

out

Quoted 3 lines

at first) .

I'd guess that something like CONFIG_PREEMPT_VOLUNTARY +

CONFIG_HZ_300

Quoted 4 lines

would probably be a good compromise for you, but just to see if there's any effect at all, start out with CONFIG_PREEMPT + CONFIG_HZ_1000.

Quoted 11 lines

I'm currious, did you ever try playing around with CONFIG_PREEMPT* and
CONFIG_HZ* to see if that had any noticable impact on interactive performance and stuff like logging into the box via ssh etc...?

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Hi Jesper,

my initial kernel was voluntary@100HZ. I have switched to 300HZ, but
have not observed much difference. The config is now:

config-2.6.22-rc7:# CONFIG_PREEMPT_NONE is not set config-2.6.22-rc7:CONFIG_PREEMPT_VOLUNTARY=y config-2.6.22-rc7:# CONFIG_PREEMPT is not set config-2.6.22-rc7:CONFIG_PREEMPT_BKL=y

Cheers

------------------------------------------------------
Martin Knoblauch

email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de