Skip to contentSkip to conversation

howto boost write(2) performance?

DISCUSSION Michael Stiller 7 participants 9 messages Oct 9, 2007 – Oct 17, 2007

lore
Share & tools
lore
#0 lore raw

Hi list,

i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.

I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs).

The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.

Maybe mmaping the disk would work?

Cheers,

-Michael

PS. I would like to be cc'd as i usually don't read the list due to high traffic.

#1 lore raw
Quoted 19 lines · El Tue, 09 Oct 2007 15:50:17 +0200 Michael Stiller <[email protected]> escribió:

Hi list,

i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.

I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs).

The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.

Maybe mmaping the disk would work?

Cheers,

-Michael

Quoted 8 lines

PS. I would like to be cc'd as i usually don't read the list due to high traffic.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Create a 1GB ram disk, write data there, and then backup into a hard disk

Signature · 1 line
Gustavo Chaín Dumit
#2 lore raw
In reply to Michael Stiller
Quoted 4 lines · Michael Stiller

The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.

You could use an O_DIRECT write if the data is suitably aligned and your IO sizes are big enough (O_DIRECT is usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine.

-Andi

#3 lore raw
Quoted 11 lines · Andi Kleen
Quoted 3 lines · Michael Stiller

The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.

You could use an O_DIRECT write if the data is suitably aligned and your IO sizes are big enough (O_DIRECT is usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine.

-Andi

If your target is a SCSI target you can gain up to 15% by using sg. Search on the net for the "sg utils" package. The source code of sg_dd and others are a grate example of how to do it.

Other wise O_DIRECT is your friend. Also look for asynchronous I/O so you have a few pending IO buffers to keep the pipes full.

Boaz

#4 lore raw

Boaz Harrosh wrote:
[]

Quoted 6 lines

If your target is a SCSI target you can gain up to 15% by using sg. Search on the net for the "sg utils" package. The source code of sg_dd and others are a grate example of how to do it.

Other wise O_DIRECT is your friend. Also look for asynchronous I/O so you have a few pending IO buffers to keep the pipes full.

What's the advantage of sg_io over O_DIRECT to the block device? I think sg devices are being (slowly?) obsoleted since most stuff which has been in sg now works over normal block device, no? At least, I haven't seen any significant difference between sg_dd and dd with oflag=direct.

Thanks.

/mjt

#5 lore raw
In reply to Andi Kleen
Quoted 10 lines · Andi Kleen
Quoted 4 lines · Michael Stiller

The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.

You could use an O_DIRECT write if the data is suitably aligned and your IO sizes are big enough (O_DIRECT is usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine.

Thanks to all who answered and especially to Andi. Using O_DIRECT did the trick.

Cheers,

Michael

#6 lore raw
In reply to Michael Stiller
Quoted 6 lines · Michael Stiller

Hi list,

i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.

Quoted 6 lines

I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs).

The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.

The kernel really cannot sustain 125MB/s? I assume the disk array is capable?

Where is the bottleneck? Does it keep all disks busy, or are the CPUs overloaded?

#7 lore raw
Quoted 4 lines

The kernel really cannot sustain 125MB/s? I assume the disk array is capable?

Yes, the array should be capable, LSI controller U320, SATA disks inside the array.

Quoted 2 lines

Where is the bottleneck? Does it keep all disks busy, or are the CPUs overloaded?

I'm not sure where the bottleneck is. CPU Load goes up with many pdflush processes in D state.

Thanks for the suggestions with sg and O_DIRECT so far, will try this.

-Michael

#8 lore raw
In reply to Michael Stiller
Quoted 3 lines · Michael Stiller

Hi list,

i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.

People regularly report speeds higher than that on the RAID list, and I can get that order of magnitude speed using dd with 1MB buffers to a software RAID-0 array or cheap SATA drives. Are you using a decent controller? Many "RAID" controllers have bandwidth limitations, buffer size issues, etc. I had some chea "SCSI" arrays which were just SCSI controllers in from of a bunch of cheap, slow, non-SCSI drives.

Quoted 3 lines

I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs).

I would limit the write size to a MB and see if that helps, regardless of the buffer size. A circular queue of smaller buffers, like ptbuf, may perform better.

Quoted 6 lines

The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance.

Maybe mmaping the disk would work?

I don't think it would help, I'd really try limiting the size of the write() calls first, assuming your hardware is adequate.

Signature · 3 lines
Bill Davidsen <[email protected]>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot