howto boost write(2) performance?
Hi list,
i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs).
The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.
Maybe mmaping the disk would work?
Cheers,
-Michael
PS. I would like to be cc'd as i usually don't read the list due to high traffic.
Quoted 19 lines · El Tue, 09 Oct 2007 15:50:17 +0200 Michael Stiller <[email protected]> escribió:
Hi list,
i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs).
The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.
Maybe mmaping the disk would work?
Cheers,
-Michael
Quoted 8 lines
PS. I would like to be cc'd as i usually don't read the list due to high traffic.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Create a 1GB ram disk, write data there, and then backup into a hard disk
Signature · 1 line
Gustavo Chaín Dumit
Quoted 4 lines · Michael Stiller
The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.
You could use an O_DIRECT write if the data is suitably aligned and your IO sizes are big enough (O_DIRECT is usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine.
-Andi
Quoted 11 lines · Andi Kleen
Quoted 3 lines · Michael Stiller
The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.
You could use an
O_DIRECTwrite if the data is suitably aligned and your IO sizes are big enough (O_DIRECTis usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine.-Andi
If your target is a SCSI target you can gain up to 15% by using sg. Search on the net for the "sg utils" package. The source code of sg_dd and others are a grate example of how to do it.
Other wise O_DIRECT is your friend. Also look for asynchronous I/O so you have a few pending IO buffers to keep the pipes full.
Boaz
Boaz Harrosh wrote:
[]
Quoted 6 lines
If your target is a SCSI target you can gain up to 15% by using sg. Search on the net for the "sg utils" package. The source code of
sg_ddand others are a grate example of how to do it.Other wise
O_DIRECTis your friend. Also look for asynchronous I/O so you have a few pending IO buffers to keep the pipes full.
What's the advantage of sg_io over O_DIRECT to the block device? I think sg devices are being (slowly?) obsoleted since most stuff which has been in sg now works over normal block device, no? At least, I haven't seen any significant difference between sg_dd and dd with oflag=direct.
Thanks.
/mjt
Quoted 10 lines · Andi Kleen
Quoted 4 lines · Michael Stiller
The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.
You could use an
O_DIRECTwrite if the data is suitably aligned and your IO sizes are big enough (O_DIRECTis usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine.
Thanks to all who answered and especially to Andi. Using O_DIRECT did the trick.
Cheers,
Michael
Quoted 6 lines · Michael Stiller
Hi list,
i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
Quoted 6 lines
I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs).
The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.
The kernel really cannot sustain 125MB/s? I assume the disk array is capable?
Where is the bottleneck? Does it keep all disks busy, or are the CPUs overloaded?
Quoted 4 lines
The kernel really cannot sustain 125MB/s? I assume the disk array is capable?
Yes, the array should be capable, LSI controller U320, SATA disks inside the array.
Quoted 2 lines
Where is the bottleneck? Does it keep all disks busy, or are the CPUs overloaded?
I'm not sure where the bottleneck is. CPU Load goes up with many pdflush processes in D state.
Thanks for the suggestions with sg and O_DIRECT so far, will try this.
-Michael
Quoted 3 lines · Michael Stiller
Hi list,
i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
People regularly report speeds higher than that on the RAID list, and I can get that order of magnitude speed using dd with 1MB buffers to a software RAID-0 array or cheap SATA drives. Are you using a decent controller? Many "RAID" controllers have bandwidth limitations, buffer size issues, etc. I had some chea "SCSI" arrays which were just SCSI controllers in from of a bunch of cheap, slow, non-SCSI drives.
Quoted 3 lines
I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs).
I would limit the write size to a MB and see if that helps, regardless of the buffer size. A circular queue of smaller buffers, like ptbuf, may perform better.
Quoted 6 lines
The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.
Maybe mmaping the disk would work?
I don't think it would help, I'd really try limiting the size of the write() calls first, assuming your hardware is adequate.
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot