XFS and 2.6.18 -> 2.6.20-rc3
Quoted 4 lines · Mr. Berkley Shands
Dave Lloyd (our in-house Idea Guy) looked at the allocation groups... Non-sequential, random...
What data would you like to see?
xfs_bmap -v on files where you consider the allocation to have changed between kernels, would show exactly how it has changed.
-Eric
My testbench is a 4 core Opteron (dual 275's) into
two LSI8408E SAS controllers, into 16 Seagate 7200.10 320GB satas. Redhat ES4.4 (Centos 4.4). A slightly newer parted is needed than the contemporary of Moses that is shipped with the O/S.
I have a standard burn in script that takes the 4 4-drive raid0's and puts a GPT label on them, aligns the partitions to stripe boundary's. It then proceeds to write 8GB files concurrently onto all 4 raid drives.
Under 2.6.18.1 the write speeds start at 265MB/Sec and decrease mostly monotonically down to ~160MB/Sec, indicating that the files start on the outside (fastest tracks) and work in. All 4 raids are within 7-8MB/Sec of each other (usually they are identical in speed).
By the time of 2.6.20-rc3, the same testbench shows
a 10% across the board decrease in throughput for writes. Reads are unaffected.
But now the allocation order for virgin file systems are random, usually starting at the slow 140MB/Sec, then bouncing up to 220MB/Sec, then around and around. No two raids get the same write speeds at the same time.
Dave Lloyd (our in-house Idea Guy) looked at the allocation groups... Non-sequential, random...
What data would you like to see?
The run logs from 2.6.18.1 and 2.6.20-rc3?
Want the scripts?
The xfs-debug dumps of a few files?
Berkley
Signature · 33 lines
//E. F. Berkley Shands, MSc// **Exegy Inc.** 3668 S. Geyer Road, Suite 300 St. Louis, MO 63127 Direct: (314) 450-5348 Cell: (314) 303-2546 Office: (314) 450-5353 Fax: (314) 450-5354 This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others. [[HTML alternate version deleted]]
Quoted 9 lines · Mr. Berkley Shands
My testbench is a 4 core Opteron (dual 275's) into
two LSI8408E SAS controllers, into 16 Seagate 7200.10 320GB satas. Redhat ES4.4 (Centos 4.4). A slightly newer parted is needed than the contemporary of Moses that is shipped with the O/S.I have a standard burn in script that takes the 4 4-drive raid0's and puts a GPT label on them, aligns the partitions to stripe boundary's. It then proceeds to write 8GB files concurrently onto all 4 raid drives.
How many files are being written at the same time to each filesystem? buffered or direct I/O? I/O size? how much memory in the machine? What size I/Os are actually hitting the disks?
Quoted 3 lines
Under 2.6.18.1 the write speeds start at 265MB/Sec and decrease mostly monotonically down to ~160MB/Sec, indicating that the files start on the outside (fastest tracks) and work in.
So you are filling the entire disk with this test?
Quoted 6 lines
All 4 raids are within 7-8MB/Sec of each other (usually they are identical in speed).
By the time of 2.6.20-rc3, the same testbench shows a 10% across the board decrease in throughput for writes. Reads are unaffected.
Reads being unaffected indicates the files are not being fragmented badly.
Quoted 1 line
But now the allocation order for virgin file systems are random,
How did you determine this?
Quoted 8 lines
usually starting at the slow 140MB/Sec, then bouncing up to 220MB/Sec, then around and around. No two raids get the same write speeds at the same time.
Dave Lloyd (our in-house Idea Guy) looked at the allocation groups... Non-sequential, random...
What data would you like to see?
First thing to do is run a set of write tests to the _raw_ devices, not to the filesystem so we can rule out a driver/hardware problem.
Can you do something as simple as concurrent writes to each raid lun to see if .18 and .20 perform the same?
Quoted 2 lines
The run logs from 2.6.18.1 and 2.6.20-rc3?
Want the scripts?
Yes please.
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 10 lines · David Chinner
Quoted 9 lines · Mr. Berkley Shands
My testbench is a 4 core Opteron (dual 275's) into
two LSI8408E SAS controllers, into 16 Seagate 7200.10 320GB satas. Redhat ES4.4 (Centos 4.4). A slightly newer parted is needed than the contemporary of Moses that is shipped with the O/S.I have a standard burn in script that takes the 4 4-drive raid0's and puts a GPT label on them, aligns the partitions to stripe boundary's. It then proceeds to write 8GB files concurrently onto all 4 raid drives.
I just ran up a similar test - single large file per device on a 4 core Xeon (woodcrest) with 16GB RAM, a single PCI-X SAS HBA and 12x10krpm 300GB SAS disks split into 3x4 disk dm raid zero stripes on 2.6.18 and 2.6.20-rc3.
I see the same thing - 2.6.20-rc3 is more erractic and quite a bit slower than 2.6.18 when going through XFS.
I suggest trying this on 2.6.20-rc3:
# echo 10 > /proc/sys/vm/dirty_ratio
That restored most of the lost performance and consistency in my testing....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
With a fresh install of the O/S on a non-broken motherboard, the change to
/proc/sys/vm/dirty_ratio
restores most of the lost performance from 2.6.18,
as of 2.6.20-rc4. The difference is 10% to 15% without the dirty_ratio change (40 is the default, 10 gives the old performance).
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 33571 MS, Rate: 244.020, to /s0/GigaData.0
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 33480 MS, Rate: 244.683, to /s2/GigaData.0
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 33570 MS, Rate: 244.027, to /s1/GigaData.0
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 33579 MS, Rate: 243.962, to /s3/GigaData.0
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 31486 MS, Rate: 260.179, to /s0/GigaData.1
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 31537 MS, Rate: 259.758, to /s2/GigaData.1
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 31538 MS, Rate: 259.750, to /s3/GigaData.1
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 31549 MS, Rate: 259.660, to /s1/GigaData.1
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 32205 MS, Rate: 254.370, to /s2/GigaData.2
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 32205 MS, Rate: 254.370, to /s1/GigaData.2
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 32218 MS, Rate: 254.268, to /s3/GigaData.2
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 32220 MS, Rate: 254.252, to /s0/GigaData.2
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 33536 MS, Rate: 244.275, to /s0/GigaData.3
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 33583 MS, Rate: 243.933, to /s3/GigaData.3
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 34016 MS, Rate: 240.828, to /s1/GigaData.3
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 34018 MS, Rate: 240.814, to /s2/GigaData.3
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 34121 MS, Rate: 240.087, to /s1/GigaData.4
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 34123 MS, Rate: 240.073, to /s0/GigaData.4
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 34114 MS, Rate: 240.136, to /s2/GigaData.4
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 34122 MS, Rate: 240.080, to /s3/GigaData.4
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 30386 MS, Rate: 269.598, to /s0/GigaData.9
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 30384 MS, Rate: 269.616, to /s1/GigaData.9
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 30717 MS, Rate: 266.693, to /s3/GigaData.9
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 30726 MS, Rate: 266.615, to /s2/GigaData.9
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 30088 MS, Rate: 272.268, to /s0/GigaData.10
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 30079 MS, Rate: 272.349, to /s2/GigaData.10
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 30086 MS, Rate: 272.286, to /s1/GigaData.10
Data: Writing, 8192 MB, Buffer: 128 KB, Time: 29989 MS, Rate: 273.167, to /s3/GigaData.10
So whatever needs to be tweaked in the VM system seems to be the key.
Thanks to all for getting this regression repaired.
Berkley
Quoted 34 lines · David Chinner
Quoted 7 lines · David Chinner
Quoted 5 lines · Mr. Berkley Shands
My testbench is a 4 core Opteron (dual 275's) into
two LSI8408E SAS controllers, into 16 Seagate 7200.10 320GB satas. Redhat ES4.4 (Centos 4.4). A slightly newer parted is needed than the contemporary of Moses that is shipped with the O/S.Quoted 5 lines
Quoted 5 lines
I have a standard burn in script that takes the 4 4-drive raid0's and puts a GPT label on them, aligns the partitions to stripe boundary's. It then proceeds to write 8GB files concurrently onto all 4 raid drives.
I just ran up a similar test - single large file per device on a 4 core Xeon (woodcrest) with 16GB RAM, a single PCI-X SAS HBA and 12x10krpm 300GB SAS disks split into 3x4 disk dm raid zero stripes on 2.6.18 and 2.6.20-rc3.
I see the same thing - 2.6.20-rc3 is more erractic and quite a bit slower than 2.6.18 when going through XFS.
I suggest trying this on 2.6.20-rc3:
# echo 10 > /proc/sys/vm/dirty_ratioThat restored most of the lost performance and consistency in my testing....
Cheers,
Dave.
Signature · 33 lines
//E. F. Berkley Shands, MSc// **Exegy Inc.** 3668 S. Geyer Road, Suite 300 St. Louis, MO 63127 Direct: (314) 450-5348 Cell: (314) 303-2546 Office: (314) 450-5353 Fax: (314) 450-5354 This e-mail and any documents accompanying it may contain legally privileged and/or confidential information belonging to Exegy, Inc. Such information may be protected from disclosure by law. The information is intended for use by only the addressee. If you are not the intended recipient, you are hereby notified that any disclosure or use of the information is strictly prohibited. If you have received this e-mail in error, please immediately contact the sender by e-mail or phone regarding instructions for return or destruction and do not use or disclose the content to others. [[HTML alternate version deleted]]
Quoted 7 lines · Mr. Berkley Shands
With a fresh install of the O/S on a non-broken motherboard, the change to
/proc/sys/vm/dirty_ratiorestores most of the lost performance from 2.6.18,
as of 2.6.20-rc4. The difference is 10% to 15% without thedirty_ratiochange (40 is the default, 10 gives the old performance).
....
Quoted 3 lines
So whatever needs to be tweaked in the VM system seems to be the key.
Thanks to all for getting this regression repaired.
Well, it's not repaired as such - you've got a WAR for the problem. I'll report the problem to lkml so that the VM gurus can try to really fix the problem....
Thanks for confirming that the dirty_ratio tweak also worked for you.
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group