Skip to contentSkip to conversation

Performance problems when writing large files on CCISS hardware

DISCUSSION Martin Knoblauch 2 participants 2 messages Jan 23, 2008 – Feb 5, 2008

lore
Share & tools
lore
#0 lore raw

Please CC me on replies, as I am not subscribed.

Hi,

for a while now I am having problems writing large files sequentially to EXT2 filesystems on CCISS based boxes. The problem is that writing multiple files in parallel is extremely slow compared to a single file in non-DIO mode. When using DIO, the scaling is almost "perfect". The problem manifests itself in RHEL4 kernels (2.6.9-X) and any mainline kernel up to 2.6.24-rc8.

The systems in question are HP/DL380G4 with 2 cpus, 8 GB memory, SmartArray6i (CCISS) with BBWC and 4x72GB@10krpm disks in RAID5 configuration. Environment is 64-bit RHEL4.3.

The problem can be reproduced by running 1, 2 or 3 parallel "dd" processes, or "iozone" with 1, 2 or 3 threads. Curiously, there was a period from 2.6.24-rc1 until 2.6.24-rc5 where the problem went away. It turned out that this was due to a "regression" that was "fixed" by below commit. Unfortunatelly this is not good for my systems, but it might shed some light on the underlying problem:

Quoted 7 lines

#commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d
#Author: Mel Gorman <[email protected]>
#Date: Mon Dec 17 16:20:05 2007 -0800
#
# mm: fix page allocation for larger I/O segments
#
# In some cases the IO subsystem is able to merge requests if the

pages are

Quoted 1 line

# adjacent in physical memory. This was achieved in the allocator

by having

Quoted 1 line

# expand() return pages in physically contiguous order in

situations were a

Quoted 1 line

# large buddy was split. However, list-based anti-fragmentation

changed the

Quoted 1 line

# order pages were returned in to avoid searching in

buffered_rmqueue() for a

Quoted 3 lines

# page of the appropriate migrate type.
#
# This patch restores behaviour of rmqueue_bulk() preserving the

physical

Quoted 1 line

# order of pages returned by the allocator without incurring

increased search

Quoted 9 lines

# costs for anti-fragmentation.
#

#    Signed-off-by: Mel Gorman <[email protected]>
#    Cc: James Bottomley <[email protected]>
#    Cc: Jens Axboe <[email protected]>
#    Cc: Mark Lord <[email protected]
#    Signed-off-by: Andrew Morton <[email protected]>
#    Signed-off-by: Linus Torvalds <[email protected]>

diff -urN linux-2.6.24-rc5/mm/page_alloc.c

linux-2.6.24-rc6/mm/page_alloc.c

Quoted 1 line

--- linux-2.6.24-rc5/mm/page_alloc.c 2007-12-21 04:14:11.305633890

+0000

Quoted 1 line

+++ linux-2.6.24-rc6/mm/page_alloc.c 2007-12-21 04:14:17.746985697

+0000

Review line 847

Quoted 2 lines
(quoted diff) +0 -0
@@ -847,8 +847,19 @@
                struct page *page = __rmqueue(zone, order,

migratetype);

Review line 852

Quoted 5 lines
(quoted diff) +3 -0
                if (unlikely(page == NULL))
                        break;
+
+               /*
+                * Split buddy pages returned by expand() are

received here

Review line 853

Quoted 1 line
(quoted diff) +1 -0
+                * in physical page order. The page is added to the

callers and

Review line 854

Quoted 1 line
(quoted diff) +1 -0
+                * list and the list head then moves forward. From

the callers

Review line 855

Quoted 1 line
(quoted diff) +1 -0
+                * perspective, the linked list is ordered by page

number in

Review line 856

Quoted 1 line
(quoted diff) +1 -0
+                * some conditions. This is useful for IO devices

that can

Review line 857

Quoted 1 line
(quoted diff) +1 -0
+                * merge IO requests if the physical pages are

ordered

Review line 865

Quoted 9 lines
(quoted diff) +3 -0
+                * properly.
+                */
                list_add(&page->lru, list);
                set_page_private(page, migratetype);
+               list = &page->lru;
        }
        spin_unlock(&zone->lock);
        return i;

Reverting this patch from 2.6.24-rc8 gives the good performance reported below (rc8*). So, apparently CCISS is very sensitive to the page ordering.

Here are the numbers (MB/sec) including sync-time. I compare 2.6.24-rc8 (rc8) and 2.6.24-rc8 with abore commit reverted (rc8*). Reported is the combined throughput for 1,2,3 iozone threads, for reference also the DIO numbers. Raw numbers are attached.

Test rc8 rc8*
----------------------------------------
1x3GB 56 90
1x3GB-DIO 86 86
2x1.5GB 9.5 87
2x1.5GB-DIO 80 85
3x1GB 16.5 85
3x1GB-DIO 85 85

One can see that in mainline/rc8 all non-DIO numbers are smaller than the corresponding DIO numbers, or the non-DIO numbers from rc8*. The performance for 2 and 3 threads in mainline/rc8 is just bad.

Of course I have the option to revert commit ....54b6d for my systems, but I think a more general solution would be better. If I can help tracking the real problem down, I am open for suggestions.

Cheers
Martin

------------------------------------------------------
Martin Knoblauch

email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:53:26 2008

Record Size 1024 KB
File size set to 3072000 KB
Include close in write timing
Include fsync in write timing
Command line used: ./iozone -r 1m -s 3000m -c -e -t 1 -F /scratch/X1 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 3072000 Kbyte file in 1024 Kbyte records

Children see throughput for 1 initial writers = 55847.39 KB/sec
Parent sees throughput for 1 initial writers = 55844.87 KB/sec
Min throughput per process = 55847.39 KB/sec
Max throughput per process = 55847.39 KB/sec
Avg throughput per process = 55847.39 KB/sec
Min xfer = 3072000.00 KB

Children see throughput for 1 rewriters = 60760.14 KB/sec
Parent sees throughput for 1 rewriters = 60757.17 KB/sec
Min throughput per process = 60760.14 KB/sec
Max throughput per process = 60760.14 KB/sec
Avg throughput per process = 60760.14 KB/sec
Min xfer = 3072000.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:55:22 2008

Record Size 1024 KB
File size set to 3072000 KB
Include close in write timing
Include fsync in write timing
O_DIRECT feature enabled
Command line used: ./iozone -r 1m -s 3000m -c -e -I -t 1 -F /scratch/X1 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 3072000 Kbyte file in 1024 Kbyte records

Children see throughput for 1 initial writers = 86304.83 KB/sec
Parent sees throughput for 1 initial writers = 86300.38 KB/sec
Min throughput per process = 86304.83 KB/sec
Max throughput per process = 86304.83 KB/sec
Avg throughput per process = 86304.83 KB/sec
Min xfer = 3072000.00 KB

Children see throughput for 1 rewriters = 85388.78 KB/sec
Parent sees throughput for 1 rewriters = 85379.37 KB/sec
Min throughput per process = 85388.78 KB/sec
Max throughput per process = 85388.78 KB/sec
Avg throughput per process = 85388.78 KB/sec
Min xfer = 3072000.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:56:41 2008

Record Size 1024 KB
File size set to 1536000 KB
Include close in write timing
Include fsync in write timing
Command line used: ./iozone -r 1m -s 1500m -c -e -t 2 -F /scratch/X1 /scratch/X2 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 2 processes
Each process writes a 1536000 Kbyte file in 1024 Kbyte records

Children see throughput for 2 initial writers = 9461.15 KB/sec
Parent sees throughput for 2 initial writers = 9455.69 KB/sec
Min throughput per process = 4726.85 KB/sec
Max throughput per process = 4734.29 KB/sec
Avg throughput per process = 4730.57 KB/sec
Min xfer = 1533952.00 KB

Children see throughput for 2 rewriters = 9780.23 KB/sec
Parent sees throughput for 2 rewriters = 9771.18 KB/sec
Min throughput per process = 4885.66 KB/sec
Max throughput per process = 4894.57 KB/sec
Avg throughput per process = 4890.12 KB/sec
Min xfer = 1536000.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 15:07:30 2008

Record Size 1024 KB
File size set to 1536000 KB
Include close in write timing
Include fsync in write timing
O_DIRECT feature enabled
Command line used: ./iozone -r 1m -s 1500m -c -e -I -t 2 -F /scratch/X1 /scratch/X2 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 2 processes
Each process writes a 1536000 Kbyte file in 1024 Kbyte records

Children see throughput for 2 initial writers = 79399.89 KB/sec
Parent sees throughput for 2 initial writers = 48569.16 KB/sec
Min throughput per process = 32239.45 KB/sec
Max throughput per process = 47160.45 KB/sec
Avg throughput per process = 39699.95 KB/sec
Min xfer = 1050624.00 KB

Children see throughput for 2 rewriters = 80684.21 KB/sec
Parent sees throughput for 2 rewriters = 80342.38 KB/sec
Min throughput per process = 27509.90 KB/sec
Max throughput per process = 53174.30 KB/sec
Avg throughput per process = 40342.10 KB/sec
Min xfer = 799744.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 15:09:00 2008

Record Size 1024 KB
File size set to 1024000 KB
Include close in write timing
Include fsync in write timing
Command line used: ./iozone -r 1m -s 1000m -c -e -t 3 -F /scratch/X1 /scratch/X2 /scratch/X3 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 3 processes
Each process writes a 1024000 Kbyte file in 1024 Kbyte records

Children see throughput for 3 initial writers = 16572.10 KB/sec
Parent sees throughput for 3 initial writers = 16228.59 KB/sec
Min throughput per process = 4908.56 KB/sec
Max throughput per process = 5942.16 KB/sec
Avg throughput per process = 5524.03 KB/sec
Min xfer = 815104.00 KB

Children see throughput for 3 rewriters = 17564.65 KB/sec
Parent sees throughput for 3 rewriters = 17144.66 KB/sec
Min throughput per process = 5643.31 KB/sec
Max throughput per process = 6190.94 KB/sec
Avg throughput per process = 5854.88 KB/sec
Min xfer = 995328.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 15:15:00 2008

Record Size 1024 KB
File size set to 1024000 KB
Include close in write timing
Include fsync in write timing
O_DIRECT feature enabled
Command line used: ./iozone -r 1m -s 1000m -c -e -I -t 3 -F /scratch/X1 /scratch/X2 /scratch/X3 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 3 processes
Each process writes a 1024000 Kbyte file in 1024 Kbyte records

Children see throughput for 3 initial writers = 85558.32 KB/sec
Parent sees throughput for 3 initial writers = 85553.00 KB/sec
Min throughput per process = 28509.68 KB/sec
Max throughput per process = 28538.28 KB/sec
Avg throughput per process = 28519.44 KB/sec
Min xfer = 1022976.00 KB

Children see throughput for 3 rewriters = 85329.49 KB/sec
Parent sees throughput for 3 rewriters = 85320.34 KB/sec
Min throughput per process = 28441.95 KB/sec
Max throughput per process = 28444.73 KB/sec
Avg throughput per process = 28443.16 KB/sec
Min xfer = 1024000.00 KB

iozone test complete.

Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:12:42 2008

Record Size 1024 KB
File size set to 3072000 KB
Include close in write timing
Include fsync in write timing
Command line used: ./iozone -r 1m -s 3000m -c -e -t 1 -F /scratch/X1 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 3072000 Kbyte file in 1024 Kbyte records

Children see throughput for 1 initial writers = 90735.91 KB/sec
Parent sees throughput for 1 initial writers = 90727.50 KB/sec
Min throughput per process = 90735.91 KB/sec
Max throughput per process = 90735.91 KB/sec
Avg throughput per process = 90735.91 KB/sec
Min xfer = 3072000.00 KB

Children see throughput for 1 rewriters = 86449.82 KB/sec
Parent sees throughput for 1 rewriters = 86444.85 KB/sec
Min throughput per process = 86449.82 KB/sec
Max throughput per process = 86449.82 KB/sec
Avg throughput per process = 86449.82 KB/sec
Min xfer = 3072000.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:36:53 2008

Record Size 1024 KB
File size set to 1536000 KB
Include close in write timing
Include fsync in write timing
Command line used: ./iozone -r 1m -s 1500m -c -e -t 2 -F /scratch/X1 /scratch/X2 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 2 processes
Each process writes a 1536000 Kbyte file in 1024 Kbyte records

Children see throughput for 2 initial writers = 87352.61 KB/sec
Parent sees throughput for 2 initial writers = 84191.30 KB/sec
Min throughput per process = 42023.50 KB/sec
Max throughput per process = 45329.11 KB/sec
Avg throughput per process = 43676.30 KB/sec
Min xfer = 1497088.00 KB

Children see throughput for 2 rewriters = 83801.86 KB/sec
Parent sees throughput for 2 rewriters = 83674.05 KB/sec
Min throughput per process = 41841.26 KB/sec
Max throughput per process = 41960.60 KB/sec
Avg throughput per process = 41900.93 KB/sec
Min xfer = 1536000.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:38:15 2008

Record Size 1024 KB
File size set to 1024000 KB
Include close in write timing
Include fsync in write timing
Command line used: ./iozone -r 1m -s 1000m -c -e -t 3 -F /scratch/X1 /scratch/X2 /scratch/X3 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 3 processes
Each process writes a 1024000 Kbyte file in 1024 Kbyte records

Children see throughput for 3 initial writers = 85182.48 KB/sec
Parent sees throughput for 3 initial writers = 73354.87 KB/sec
Min throughput per process = 26752.83 KB/sec
Max throughput per process = 30931.31 KB/sec
Avg throughput per process = 28394.16 KB/sec
Min xfer = 802816.00 KB

Children see throughput for 3 rewriters = 86484.59 KB/sec
Parent sees throughput for 3 rewriters = 81781.53 KB/sec
Min throughput per process = 27262.20 KB/sec
Max throughput per process = 31317.65 KB/sec
Avg throughput per process = 28828.20 KB/sec
Min xfer = 1024000.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:40:04 2008

Record Size 1024 KB
File size set to 3072000 KB
Include close in write timing
Include fsync in write timing
O_DIRECT feature enabled
Command line used: ./iozone -r 1m -s 3000m -c -e -I -t 1 -F /scratch/X1 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 1 process
Each process writes a 3072000 Kbyte file in 1024 Kbyte records

Children see throughput for 1 initial writers = 86219.96 KB/sec
Parent sees throughput for 1 initial writers = 86214.89 KB/sec
Min throughput per process = 86219.96 KB/sec
Max throughput per process = 86219.96 KB/sec
Avg throughput per process = 86219.96 KB/sec
Min xfer = 3072000.00 KB

Children see throughput for 1 rewriters = 85718.92 KB/sec
Parent sees throughput for 1 rewriters = 85714.79 KB/sec
Min throughput per process = 85718.92 KB/sec
Max throughput per process = 85718.92 KB/sec
Avg throughput per process = 85718.92 KB/sec
Min xfer = 3072000.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:42:42 2008

Record Size 1024 KB
File size set to 1536000 KB
Include close in write timing
Include fsync in write timing
O_DIRECT feature enabled
Command line used: ./iozone -r 1m -s 1500m -c -e -I -t 2 -F /scratch/X1 /scratch/X2 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 2 processes
Each process writes a 1536000 Kbyte file in 1024 Kbyte records

Children see throughput for 2 initial writers = 85696.29 KB/sec
Parent sees throughput for 2 initial writers = 85689.20 KB/sec
Min throughput per process = 42834.30 KB/sec
Max throughput per process = 42861.98 KB/sec
Avg throughput per process = 42848.14 KB/sec
Min xfer = 1534976.00 KB

Children see throughput for 2 rewriters = 85508.10 KB/sec
Parent sees throughput for 2 rewriters = 85495.34 KB/sec
Min throughput per process = 42738.68 KB/sec
Max throughput per process = 42769.42 KB/sec
Avg throughput per process = 42754.05 KB/sec
Min xfer = 1534976.00 KB

iozone test complete.
Iozone: Performance Test of File I/O
Version $Revision: 3.283 $
Compiled for 64 bit mode.
Build: linux-AMD64

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong.

Run began: Wed Jan 23 14:44:15 2008

Record Size 1024 KB
File size set to 1024000 KB
Include close in write timing
Include fsync in write timing
O_DIRECT feature enabled
Command line used: ./iozone -r 1m -s 1000m -c -e -I -t 3 -F /scratch/X1 /scratch/X2 /scratch/X3 -i 0 Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 3 processes
Each process writes a 1024000 Kbyte file in 1024 Kbyte records

Children see throughput for 3 initial writers = 85836.74 KB/sec
Parent sees throughput for 3 initial writers = 85828.38 KB/sec
Min throughput per process = 28602.75 KB/sec
Max throughput per process = 28630.92 KB/sec
Avg throughput per process = 28612.25 KB/sec
Min xfer = 1022976.00 KB

Children see throughput for 3 rewriters = 85296.01 KB/sec
Parent sees throughput for 3 rewriters = 85285.05 KB/sec
Min throughput per process = 28431.54 KB/sec
Max throughput per process = 28432.71 KB/sec
Avg throughput per process = 28432.00 KB/sec
Min xfer = 1024000.00 KB

iozone test complete.

#1 lore raw

2008/1/23, Martin Knoblauch <[email protected]>:

Quoted 7 lines

Please CC me on replies, as I am not subscribed.

Hi,

for a while now I am having problems writing large files sequentially to EXT2 filesystems on CCISS based boxes. The problem is that writing multiple files in parallel is extremely slow compared to a single file in non-DIO mode. When using DIO, the scaling is almost "perfect". The problem manifests itself in RHEL4 kernels (2.6.9-X) and any mainline kernel up to 2.6.24-rc8.

The systems in question are HP/DL380G4 with 2 cpus, 8 GB memory, SmartArray6i (CCISS) with BBWC and 4x72GB@10krpm disks in RAID5 configuration. Environment is 64-bit RHEL4.3.

I've seen similar problems on HP DL380 G4 (SmartArray 6i) and HP DL385 G5 (SmartArray P400).

RHEL 4, kernel 2.6.9 and reiserfs had siginifcantly worse I/O performance than a Gentoo box running 2.6.18 (iirc) when I did some I/O-test with different distributions on a HP DL380 G4 before deploying the machine in production. Switching I/O-schedulers on RHEL4 didn't help either.

Also, I'm having awful I/O-performance with a DL385 G2/2x2.6GH/4GB running MySQL 5 on Ubuntu 7.10 (kernel 2.6.22). It previously ran Ubuntu 7.04 (kernel 2.6.20) which had the same issue. On this server MySQL stalls for a long time waiting for I/O after SQL updates that causes lots of writes.

I've been trying to mitigate the problems by adding 512MB Batterybacked Writecache and lately also switching from RAID1 to RAID 1+0 (4x72GB 15k SAS disks). It's better but there are still issues.

I think after switching to RAID 1+0 I'm now getting around 50-70 MB/s, which is 1.5-2.0 times the performance I had before.

Running sync on any of the servers while there are dirty pages to be written (according to /proc/meminfo) virtually kills all I/O until the sync completes.

I don't have much experience with other RAID controllers than the SmartArray and naturally don't know what to expect, but I sure think it should be better.

I'm getting much better performance out of an ordinary "home computer" that has 4 standard disks in RAID 1+0 configuration (software; Linux md) and AES encryption with dm-crypt.

Below are some numbers.

HP DL385 G5, 2x2.6GHz/2GB/4x72GB in RAID1+0, kernel 2.6.22 (Ubuntu 7.10 x64)
====
# sync; time sh -c "dd if=/dev/zero of=/data/test bs=1024k count=8192;sync"
dd: writing `/data/test': No space left on device
6187+0 records in
6186+0 records out
6487523328 bytes (6.5 GB) copied, 113.756 seconds, 57.0 MB/s

real    1m56.916s
user    0m0.050s
sys     0m31.040s

HP DL385 G5, 2x2.6GHz/4GB/4x72GB in RAID1+0 512MB BBWC, kernel 2.6.22 (Ubuntu 7.10 x64)
===
# sync; time sh -c "dd if=/dev/zero of=/data/test bs=1024k count=8192;sync" 8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 120.797 seconds, 71.1 MB/s

real    2m1.883s
user    0m0.020s
sys     0m26.530s

-- noah