Skip to contentSkip to conversation

Possible O_DIRECT problems ?

DISCUSSION Dave Jones 8 participants 9 messages Dec 21, 2001 – Dec 30, 2001

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

Andrea, lk,
I just experimented with O_DIRECT in conjunction with fsx,
and the results aren't pretty.

Over NFS it survives around 921 operations, all local filesystems (ext2,ext3,reiser tested) just 6 operations.
I've put the source to a modified fsx at
http://www.codemonkey.org.uk/cruft/fsx-odirect.c

It's possible I've done something wrong here, so look it over. Just adding O_DIRECT flag to open() should be all thats necessary correct ?

Also note, that by changing the flags on line 988 to have O_DIRECT also, we get different failure type.

So, did I get the usage of O_DIRECT correct and find some bugs, or have I had a little too much xmas spirits already ? 8-)

Dave.

Signature · 2 lines
| Dave Jones.                    http://www.codemonkey.org.uk
| SuSE Labs .
#1 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Dave Jones <[email protected]> writes:

> Andrea, lk,
> I just experimented with O_DIRECT in conjunction with fsx, > and the results aren't pretty.

> Over NFS it survives around 921 operations, all local > filesystems (ext2,ext3,reiser tested) just 6 operations. I've > put the source to a modified fsx at
> http://www.codemonkey.org.uk/cruft/fsx-odirect.c

Dave,

O_DIRECT for NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?

Cheers,
Trond

#2 lore raw
In reply to Dave Jones
Quoted 2 lines · Trond Myklebust

O_DIRECT for NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?

Nope, stock 2.4.17rc2 & 2.5.1.
I thought NFS might just ignore the O_DIRECT flag if it didn't understand it yet, I wasn't expecting such a dramatic failure.

I just got reminded of the bugs Andrew Morton & some others found in O_DIRECT, so this may be hitting the same problems already found.

Dave.

#3 lore raw
In reply to Dave Jones
Quoted 8 lines · GOTO Masanori

At Fri, 21 Dec 2001 00:39:42 +0000,

Quoted 3 lines · Dave Jones
Quoted 2 lines · Trond Myklebust

O_DIRECT for NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?

Where is Chuck's patch ? I searched but didn't find.

I haven't put it up on my own web-site, but it should be available from the CITI NFS client performance project site. See

http://www.citi.umich.edu/projects/nfs-perf/patches/

Quoted 2 lines

Supporting direct_IO with NFS is some meaningful for users who have fast NAS server environment, IMHO.

It can also provide for better data security in some circumstances. Journaling in databases over NFS can for instance benefit greatly, and has been one of Chuck's motivations for doing it.

Cheers,
Trond

#4 lore raw
Quoted 13 lines · Trond Myklebust
Quoted 8 lines · GOTO Masanori

At Fri, 21 Dec 2001 00:39:42 +0000,

Quoted 3 lines · Dave Jones
Quoted 2 lines · Trond Myklebust

O_DIRECT for NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?

Where is Chuck's patch ? I searched but didn't find.

I haven't put it up on my own web-site, but it should be available from the CITI NFS client performance project site. See

Quoted 8 lines

http://www.citi.umich.edu/projects/nfs-perf/patches/

Quoted 2 lines

Supporting direct_IO with NFS is some meaningful for users who have fast NAS server environment, IMHO.

It can also provide for better data security in some circumstances. Journaling in databases over NFS can for instance benefit greatly, and has been one of Chuck's motivations for doing it.

the patch is designed for applications that manage their own data cache, like databases do. but it is also useful for applications that want to move large datasets without blowing the O/S level data cache.

in the NFS case, because O_DIRECT read() and write() always go back to the server, you can more easily build clustered and HA applications that share the data storage backend.

- Chuck Lever
--
corporate: <[email protected]>
personal: <[email protected]>

#5 lore raw
In reply to Dave Jones
Quoted 8 lines · Dave Jones
Quoted 2 lines · Trond Myklebust

O_DIRECT for NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?

Nope, stock 2.4.17rc2 & 2.5.1.
I thought NFS might just ignore the O_DIRECT flag if it didn't understand it yet, I wasn't expecting such a dramatic failure.

The point of O_DIRECT is to do DMA directly into the userspace memory (and to avoid the VM overhead but that's a secondary issue and with data journaling we may need to put an anchor into the VM to serialize the direct I/O with the pagecache I/O in a secondary - slower - direct_IO callback for the data journaling fs).

But to avoid the mem copies you're required to use strict alignment and size of the userspace buffers, just like rawio.

If you don't you will get -EINVAL. This ensures people will use O_DIRECT correctly in their apps. In short every single bugreport like this about this -EINVAL strict behaviour is the proof we need to be strict and to return -EINVAL :)

Andrea

#6 lore raw
In reply to Dave Jones

Shouldn't O_DIRECT's requirements come from the hardware? If we can ASPI or CAM DMA SCSI devices to odd addresses and lengths, why not O_DIRECT? Do ape drives DMA to user buffers? Are O_DIRECT's current limits gratuitous?

Quoted 34 lines · Andrea Arcangeli
Quoted 8 lines · Dave Jones
Quoted 2 lines · Trond Myklebust

O_DIRECT for NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?

Nope, stock 2.4.17rc2 & 2.5.1. I thought NFS might just ignore the O_DIRECT flag if it didn't understand it yet, I wasn't expecting such a dramatic failure.

The point of O_DIRECT is to do DMA directly into the userspace memory (and to avoid the VM overhead but that's a secondary issue and with data journaling we may need to put an anchor into the VM to serialize the direct I/O with the pagecache I/O in a secondary - slower - direct_IO callback for the data journaling fs).

But to avoid the mem copies you're required to use strict alignment and size of the userspace buffers, just like rawio.

If you don't you will get -EINVAL. This ensures people will use O_DIRECT correctly in their apps. In short every single bugreport like this about this -EINVAL strict behaviour is the proof we need to be strict and to return -EINVAL :)

Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

.

#7 lore raw
Quoted 4 lines · CJ

Shouldn't O_DIRECT's requirements come from the hardware? If we can ASPI or CAM DMA SCSI devices to odd addresses and lengths, why not O_DIRECT? Do ape drives DMA to user buffers? Are O_DIRECT's current limits gratuitous?

CAM is a very bad thing and that is why the X3 committees split.

Andre Hedrick
Linux Disk Certification Project Linux ATA Development

#8 lore raw
Quoted 8 lines · Andre Hedrick
Quoted 4 lines · CJ

Shouldn't O_DIRECT's requirements come from the hardware? If we can ASPI or CAM DMA SCSI devices to odd addresses and lengths, why not O_DIRECT? Do ape drives DMA to user buffers? Are O_DIRECT's current limits gratuitous?

CAM is a very bad thing and that is why the X3 committees split.

There were interesting guide-lines in CAM, notably the topology handling and the error recovery scheme. But it was another different wheel in a world where everybody did reinvent its own. It seemed also very DEC tainted.

Btw, given guys like you in X3 committees, I am not surprised that splits occur in this place. :-)

Gérard.

PS: Your various email addresses bounce back claiming some ridiculous text about spammers. Is this still another show of your apparent existential complex.