Skip to contentSkip to conversation

Poor performance -- poor config?

DISCUSSION Robert Petkus 4 participants 6 messages Jun 20, 2007 – Jun 25, 2007

lore
Share & tools
lore

Robert Petkus

author
#0 lore raw

Folks,
I'm trying to configure a system (server + DS4700 disk array) that can offer the highest performance for our application. We will be reading and writing multiple threads of 1-2GB files with 1MB block sizes. DS4700 config:
(16) 500 GB SATA disks
(3) 4+1 RAID 5 arrays and (1) hot spare == (3) 2TB LUNs.
(2) RAID arrays are on controller A, (1) RAID array is on controller B.
512k segment size

Server Config:
IBM x3550, 9GB RAM, RHEL 5 x86_64 (2.6.18)
The (3) LUNs are sdb, sdc {both controller A}, sdd {controller B}

My original goal was to use XFS and create a highly optimized config. Here is what I came up with:
Create separate partitions for XFS log files: sdd1, sdd2, sdd3 each 150M -- 128MB is the maximum allowable XFS log size. The XFS "stripe unit" (su) = 512k to match the DS4700 segment size
The "stripe width" ( (n-1)*sunit )= swidth=2048k = sw=4 (a multiple of su)
4k is the max block size allowable on x86_64 since 4k is the max kernel page size

[root@~]# mkfs.xfs -l logdev=/dev/sdd1,size=128m -d su=512k -d sw=4 -f /dev/sdb
[root@~]#  mount -t xfs -o context=system_u:object_r:unconfined_t,noatime,nodiratime,logbufs=8,logdev=/dev/sdd1 /dev/sdb /data0

And the write performance is lousy compared to ext3 built like so:

[root@~]# mke2fs -j -m 1 -b4096 -E stride=128 /dev/sdc
[root@~]# mount -t ext3 -o noatime,nodiratime,context="system_u:object_r:unconfined_t:s0",reservation /dev/sdc /data1

What am I missing?

Thanks!

Signature · 8 lines
Robert Petkus
RHIC/USATLAS Computing Facility
Brookhaven National Laboratory
Physics Dept. - Bldg. 510A
Upton, New York 11973

http://www.bnl.gov/RHIC
http://www.acf.bnl.gov
#1 lore raw
Quoted 21 lines · Robert Petkus

Folks,
I'm trying to configure a system (server + DS4700 disk array) that can offer the highest performance for our application. We will be reading and writing multiple threads of 1-2GB files with 1MB block sizes. DS4700 config:
(16) 500 GB SATA disks
(3) 4+1 RAID 5 arrays and (1) hot spare == (3) 2TB LUNs.
(2) RAID arrays are on controller A, (1) RAID array is on controller B.
512k segment size

Server Config:
IBM x3550, 9GB RAM, RHEL 5 x86_64 (2.6.18)
The (3) LUNs are sdb, sdc {both controller A}, sdd {controller B}

My original goal was to use XFS and create a highly optimized config. Here is what I came up with:
Create separate partitions for XFS log files: sdd1, sdd2, sdd3 each 150M -- 128MB is the maximum allowable XFS log size. The XFS "stripe unit" (su) = 512k to match the DS4700 segment size
The "stripe width" ( (n-1)*sunit )= swidth=2048k = sw=4 (a multiple of su)
4k is the max block size allowable on x86_64 since 4k is the max kernel page size

[root@~]# mkfs.xfs -l logdev=/dev/sdd1,size=128m -d su=512k -d sw=4 -f /dev/sdb
[root@~]#  mount -t xfs -o context=system_u:object_r:unconfined_t,noatime,nodiratime,logbufs=8,logdev=/dev/sdd1 /dev/sdb /data0
Quoted 19 lines

And the write performance is lousy compared to ext3 built like so:

[root@~]# mke2fs -j -m 1 -b4096 -E stride=128 /dev/sdc
[root@~]# mount -t ext3 -o noatime,nodiratime,context="system_u:object_r:unconfined_t:s0",reservation /dev/sdc /data1

What am I missing?

Thanks!

Signature · 8 lines
Robert Petkus
RHIC/USATLAS Computing Facility
Brookhaven National Laboratory
Physics Dept. - Bldg. 510A
Upton, New York 11973

http://www.bnl.gov/RHIC
http://www.acf.bnl.gov

What speeds are you getting?

Have you tried a SW RAID with the 16 drives, if you do that, XFS will auto-optimize per the physical characteristics of the md array.

Also, most of those mount options besides the logdev/noatime don't do much with XFS from my personal benchmarks, you're better off with the defaults+noatime.

What speed are you getting reads/writes, what do you expect? How are the drives attached/what type of controller? PCI?

author
#2 lore raw
Quoted 41 lines · Justin Piszcz
Quoted 37 lines · Robert Petkus

Folks,
I'm trying to configure a system (server + DS4700 disk array) that can offer the highest performance for our application. We will be reading and writing multiple threads of 1-2GB files with 1MB block sizes. DS4700 config:
(16) 500 GB SATA disks
(3) 4+1 RAID 5 arrays and (1) hot spare == (3) 2TB LUNs.
(2) RAID arrays are on controller A, (1) RAID array is on controller B.
512k segment size

Server Config:
IBM x3550, 9GB RAM, RHEL 5 x86_64 (2.6.18)
The (3) LUNs are sdb, sdc {both controller A}, sdd {controller B}

My original goal was to use XFS and create a highly optimized config. Here is what I came up with:
Create separate partitions for XFS log files: sdd1, sdd2, sdd3 each 150M -- 128MB is the maximum allowable XFS log size. The XFS "stripe unit" (su) = 512k to match the DS4700 segment size
The "stripe width" ( (n-1)*sunit )= swidth=2048k = sw=4 (a multiple of su)
4k is the max block size allowable on x86_64 since 4k is the max kernel page size

[root@~]# mkfs.xfs -l logdev=/dev/sdd1,size=128m -d su=512k -d sw=4 
-f /dev/sdb
[root@~]#  mount -t xfs -o context=system_u:object_r:unconfined_t,noatime,nodiratime,logbufs=8,logdev=/dev/sdd1 /dev/sdb /data0

And the write performance is lousy compared to ext3 built like so:

[root@~]# mke2fs -j -m 1 -b4096 -E stride=128 /dev/sdc
[root@~]# mount -t ext3 -o noatime,nodiratime,context="system_u:object_r:unconfined_t:s0",reservation /dev/sdc /data1

What am I missing?

Thanks!

Signature · 5 lines
Robert Petkus
RHIC/USATLAS Computing Facility
Brookhaven National Laboratory
Physics Dept. - Bldg. 510A
Upton, New York 11973
Quoted 6 lines
Quoted 4 lines

http://www.bnl.gov/RHIC
http://www.acf.bnl.gov

What speeds are you getting?

dd if=/dev/zero of=/data0/bigfile bs=1024k count=5000 5242880000 bytes (5.2 GB) copied, 149.296 seconds, 35.1 MB/s

dd if=/data0/bigfile of=/dev/null bs=1024k count=5000 5242880000 bytes (5.2 GB) copied, 26.3148 seconds, 199 MB/s

iozone.linux -w -r 1m -s 1g -i0 -t 4 -e -w -f /data0/test1
Children see throughput for 4 initial writers = 28528.59 KB/sec

       Parent sees throughput for  4 initial writers   =   25212.79 KB/sec
       Min throughput per process                      =    6259.05 KB/sec
       Max throughput per process                      =    7548.29 KB/sec
       Avg throughput per process                      =    7132.15 KB/sec

iozone.linux -w -r 1m -s 1g -i1 -t 4 -e -w -f /data0/test1
Children see throughput for 4 readers = 3059690.19 KB/sec

       Parent sees throughput for  4 readers           = 3055307.71 KB/sec
       Min throughput per process                      =  757151.81 KB/sec
       Max throughput per process                      =  776032.62 KB/sec
       Avg throughput per process                      =  764922.55 KB/sec
Quoted 2 lines

Have you tried a SW RAID with the 16 drives, if you do that, XFS will auto-optimize per the physical characteristics of the md array.

No because this would waste an expensive disk array. I've done this with various JBODs, even a SUN Thumper, with OK results...

Quoted 2 lines

Also, most of those mount options besides the logdev/noatime don't do much with XFS from my personal benchmarks, you're better off with the defaults+noatime.

The security context stuff is in there since I run a strict SELinux policy. Otherwise, I need logdev since it's on a different disk. BTW, the same filesystem w/out a separate log disk made no difference in performance.

Quoted 2 lines

What speed are you getting reads/writes, what do you expect? How are the drives attached/what type of controller? PCI?

I can get ~3x write performance with ext3. I have a dual-port FC-4 PCIe HBA connected to (2) IBM DS4700 FC-4 controllers. There is lots of headroom.

Signature · 8 lines
Robert Petkus
RHIC/USATLAS Computing Facility
Brookhaven National Laboratory
Physics Dept. - Bldg. 510A
Upton, New York 11973

http://www.bnl.gov/RHIC
http://www.acf.bnl.gov
#3 lore raw
Quoted 69 lines · Robert Petkus
Quoted 41 lines · Justin Piszcz
Quoted 37 lines · Robert Petkus

Folks,
I'm trying to configure a system (server + DS4700 disk array) that can offer the highest performance for our application. We will be reading and writing multiple threads of 1-2GB files with 1MB block sizes. DS4700 config:
(16) 500 GB SATA disks
(3) 4+1 RAID 5 arrays and (1) hot spare == (3) 2TB LUNs.
(2) RAID arrays are on controller A, (1) RAID array is on controller B.
512k segment size

Server Config:
IBM x3550, 9GB RAM, RHEL 5 x86_64 (2.6.18)
The (3) LUNs are sdb, sdc {both controller A}, sdd {controller B}

My original goal was to use XFS and create a highly optimized config. Here is what I came up with:
Create separate partitions for XFS log files: sdd1, sdd2, sdd3 each 150M
-- 128MB is the maximum allowable XFS log size.
The XFS "stripe unit" (su) = 512k to match the DS4700 segment size
The "stripe width" ( (n-1)*sunit )= swidth=2048k = sw=4 (a multiple of su)
4k is the max block size allowable on x86_64 since 4k is the max kernel page size

[root@~]# mkfs.xfs -l logdev=/dev/sdd1,size=128m -d su=512k -d sw=4 -f /dev/sdb
[root@~]#  mount -t xfs -o context=system_u:object_r:unconfined_t,noatime,nodiratime,logbufs=8,logdev=/dev/sdd1 /dev/sdb /data0

And the write performance is lousy compared to ext3 built like so:

[root@~]# mke2fs -j -m 1 -b4096 -E stride=128 /dev/sdc
[root@~]# mount -t ext3 -o noatime,nodiratime,context="system_u:object_r:unconfined_t:s0",reservation /dev/sdc /data1

What am I missing?

Thanks!

Signature · 5 lines
Robert Petkus
RHIC/USATLAS Computing Facility
Brookhaven National Laboratory
Physics Dept. - Bldg. 510A
Upton, New York 11973
Quoted 6 lines
Quoted 4 lines

http://www.bnl.gov/RHIC
http://www.acf.bnl.gov

What speeds are you getting?

dd if=/dev/zero of=/data0/bigfile bs=1024k count=5000 5242880000 bytes (5.2 GB) copied, 149.296 seconds, 35.1 MB/s

dd if=/data0/bigfile of=/dev/null bs=1024k count=5000 5242880000 bytes (5.2 GB) copied, 26.3148 seconds, 199 MB/s

iozone.linux -w -r 1m -s 1g -i0 -t 4 -e -w -f /data0/test1
Children see throughput for 4 initial writers = 28528.59 KB/sec

      Parent sees throughput for  4 initial writers   =   25212.79 KB/sec
      Min throughput per process                      =    6259.05 KB/sec
      Max throughput per process                      =    7548.29 KB/sec
      Avg throughput per process                      =    7132.15 KB/sec

iozone.linux -w -r 1m -s 1g -i1 -t 4 -e -w -f /data0/test1
Children see throughput for 4 readers = 3059690.19 KB/sec

      Parent sees throughput for  4 readers           = 3055307.71 KB/sec
      Min throughput per process                      =  757151.81 KB/sec
      Max throughput per process                      =  776032.62 KB/sec
      Avg throughput per process                      =  764922.55 KB/sec
Quoted 1 line
Quoted 20 lines
Quoted 1 line

Have you tried a SW RAID with the 16 drives, if you do that, XFS will auto-optimize per the physical characteristics of the md array.

No because this would waste an expensive disk array. I've done this with various JBODs, even a SUN Thumper, with OK results...

Quoted 2 lines

Also, most of those mount options besides the logdev/noatime don't do much with XFS from my personal benchmarks, you're better off with the defaults+noatime.

The security context stuff is in there since I run a strict SELinux policy. Otherwise, I need logdev since it's on a different disk. BTW, the same filesystem w/out a separate log disk made no difference in performance.

Quoted 2 lines

What speed are you getting reads/writes, what do you expect? How are the drives attached/what type of controller? PCI?

I can get ~3x write performance with ext3. I have a dual-port FC-4 PCIe HBA connected to (2) IBM DS4700 FC-4 controllers. There is lots of headroom.

Signature · 8 lines
Robert Petkus
RHIC/USATLAS Computing Facility
Brookhaven National Laboratory
Physics Dept. - Bldg. 510A
Upton, New York 11973

http://www.bnl.gov/RHIC
http://www.acf.bnl.gov

EXT3 up to 3x fast? Hrm.. Have you tried default mkfs.xfs options [internal journal]? What write speed do you get using the defaults?

What kernel version?

Justin.

#4 lore raw
Quoted 3 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf

Of Justin Piszcz

Quoted 7 lines

Sent: Mittwoch, 20. Juni 2007 23:24
To: Robert Petkus
Cc: [email protected]
Subject: Re: Poor performance -- poor config?

Quoted 9 lines
Quoted 7 lines · Robert Petkus
Quoted 6 lines · Justin Piszcz
Quoted 2 lines · Robert Petkus

Folks,
I'm trying to configure a system (server + DS4700 disk array) that

can

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

offer the highest performance for our application. We will be

reading and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

writing multiple threads of 1-2GB files with 1MB block sizes. DS4700 config:
(16) 500 GB SATA disks
(3) 4+1 RAID 5 arrays and (1) hot spare == (3) 2TB LUNs.
(2) RAID arrays are on controller A, (1) RAID array is on

controller B.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

512k segment size

Server Config:
IBM x3550, 9GB RAM, RHEL 5 x86_64 (2.6.18)
The (3) LUNs are sdb, sdc {both controller A}, sdd {controller B}

My original goal was to use XFS and create a highly optimized

config.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Here is what I came up with:
Create separate partitions for XFS log files: sdd1, sdd2, sdd3

each 150M

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

-- 128MB is the maximum allowable XFS log size.
The XFS "stripe unit" (su) = 512k to match the DS4700 segment size The "stripe width" ( (n-1)*sunit )= swidth=2048k = sw=4 (a

multiple of

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

su)
4k is the max block size allowable on x86_64 since 4k is the max

kernel

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

page size

[root@~]# mkfs.xfs -l logdev=/dev/sdd1,size=128m -d su=512k -d

sw=4 -f

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

/dev/sdb

[root@~]#  mount -t xfs -o

context=system_u:object_r:unconfined_t,noatime,nodiratime,logbufs=8,logd ev=/dev/sdd1

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

/dev/sdb /data0

And the write performance is lousy compared to ext3 built like so:

[root@~]# mke2fs -j -m 1 -b4096 -E stride=128 /dev/sdc
[root@~]# mount -t ext3 -o

noatime,nodiratime,context="system_u:object_r:unconfined_t:s0",reservati on

Quoted 22 lines
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines

/dev/sdc /data1

What am I missing?

Thanks!

--
Robert Petkus
RHIC/USATLAS Computing Facility
Brookhaven National Laboratory
Physics Dept. - Bldg. 510A
Upton, New York 11973

Quoted 6 lines
Quoted 4 lines

http://www.bnl.gov/RHIC
http://www.acf.bnl.gov

What speeds are you getting?

dd if=/dev/zero of=/data0/bigfile bs=1024k count=5000 5242880000 bytes (5.2 GB) copied, 149.296 seconds, 35.1 MB/s

Quoted 6 lines
Quoted 6 lines

dd if=/data0/bigfile of=/dev/null bs=1024k count=5000 5242880000 bytes (5.2 GB) copied, 26.3148 seconds, 199 MB/s

iozone.linux -w -r 1m -s 1g -i0 -t 4 -e -w -f /data0/test1
Children see throughput for 4 initial writers = 28528.59 KB/sec
Parent sees throughput for 4 initial writers = 25212.79

KB/sec

Quoted 1 line
Quoted 1 line

Min throughput per process = 6259.05

KB/sec

Quoted 1 line
Quoted 1 line

Max throughput per process = 7548.29

KB/sec

Quoted 1 line
Quoted 1 line

Avg throughput per process = 7132.15

KB/sec

Quoted 4 lines
Quoted 4 lines

iozone.linux -w -r 1m -s 1g -i1 -t 4 -e -w -f /data0/test1
Children see throughput for 4 readers = 3059690.19 KB/sec
Parent sees throughput for 4 readers = 3055307.71

KB/sec

Quoted 1 line
Quoted 1 line

Min throughput per process = 757151.81

KB/sec

Quoted 1 line
Quoted 1 line

Max throughput per process = 776032.62

KB/sec

Quoted 1 line
Quoted 1 line

Avg throughput per process = 764922.55

KB/sec

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Have you tried a SW RAID with the 16 drives, if you do that, XFS

will

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

auto-optimize per the physical characteristics of the md array.

No because this would waste an expensive disk array. I've done this

with

Quoted 3 lines
Quoted 3 lines

various JBODs, even a SUN Thumper, with OK results...

Quoted 2 lines

Also, most of those mount options besides the logdev/noatime don't

do much

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

with XFS from my personal benchmarks, you're better off with the defaults+noatime.

The security context stuff is in there since I run a strict SELinux

policy.

Quoted 1 line
Quoted 1 line

Otherwise, I need logdev since it's on a different disk. BTW, the

same

Quoted 1 line
Quoted 1 line

filesystem w/out a separate log disk made no difference in

performance.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

What speed are you getting reads/writes, what do you expect? How

are the

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

drives attached/what type of controller? PCI?

I can get ~3x write performance with ext3. I have a dual-port FC-4

PCIe HBA

Quoted 1 line
Quoted 1 line

connected to (2) IBM DS4700 FC-4 controllers. There is lots of

headroom.

Quoted 13 lines
Quoted 8 lines

--
Robert Petkus
RHIC/USATLAS Computing Facility
Brookhaven National Laboratory
Physics Dept. - Bldg. 510A
Upton, New York 11973

Quoted 4 lines

http://www.bnl.gov/RHIC
http://www.acf.bnl.gov

Quoted 7 lines

EXT3 up to 3x fast? Hrm.. Have you tried default mkfs.xfs options [internal journal]? What write speed do you get using the defaults?

What kernel version?

Justin.

Not sure if it makes much sense to set stripe unit and width for a Raid which appears as a single device.
As you state, the "width" of your DS lun is 4 x 512K == 2MB. In case you don't have write cache enabled each of your 1MB writes will cause the DS to write to two out of four disks only, causing heavy overhead to create parity.
Write cache mirroring on the DS also causes limitation in write performance. And finally there is an option in the DS to change the cache segment size from 16k default to 4k IIRC. Make sure it is set to 16k.
But still, 35MB/s for a single sequential write is really poor. Almost looks like you get single spindle performance only.

Sebastian

#5 lore raw
Quoted 2 lines · Sebastian Brings

Not sure if it makes much sense to set stripe unit and width for a Raid which appears as a single device.

Certainly it does.

That way you get stripe aligned allocation and therfore you are much more likely to get full-stripe width writes instead of unaligned writes that force RMW cycles on the RAID controller for parity calculations.

Quoted 4 lines

As you state, the "width" of your DS lun is 4 x 512K == 2MB. In case you don't have write cache enabled each of your 1MB writes will cause the DS to write to two out of four disks only, causing heavy overhead to create parity.

You're assuming stripe aligned I/O there. That 1MB could hit 3 of the 4 data disks - if you don't have a stripe unit set then that will be the common case. i.e. its worse than you think :/

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group