Possible O_DIRECT problems ?
Andrea, lk,
I just experimented with O_DIRECT in conjunction with fsx,
and the results aren't pretty.
Over NFS it survives around 921 operations, all local filesystems (ext2,ext3,reiser tested) just 6 operations.
I've put the source to a modified fsx at
http://www.codemonkey.org.uk/cruft/fsx-odirect.c
It's possible I've done something wrong here, so look it over. Just adding O_DIRECT flag to open() should be all thats necessary correct ?
Also note, that by changing the flags on line 988 to have O_DIRECT also, we get different failure type.
So, did I get the usage of O_DIRECT correct and find some bugs, or have I had a little too much xmas spirits already ? 8-)
Dave.
Signature · 2 lines
| Dave Jones. http://www.codemonkey.org.uk | SuSE Labs .
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
" " == Dave Jones <[email protected]> writes:
> Andrea, lk,
> I just experimented with O_DIRECT in conjunction with fsx, > and the results aren't pretty.
> Over NFS it survives around 921 operations, all local > filesystems (ext2,ext3,reiser tested) just 6 operations. I've > put the source to a modified fsx at
> http://www.codemonkey.org.uk/cruft/fsx-odirect.c
Dave,
O_DIRECT for NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?
Cheers,
Trond
Quoted 2 lines · Trond Myklebust
O_DIRECTfor NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?
Nope, stock 2.4.17rc2 & 2.5.1.
I thought NFS might just ignore the O_DIRECT flag if it didn't understand it yet, I wasn't expecting such a dramatic failure.
I just got reminded of the bugs Andrew Morton & some others found in O_DIRECT, so this may be hitting the same problems already found.
Dave.
Quoted 8 lines · GOTO Masanori
At Fri, 21 Dec 2001 00:39:42 +0000,
Quoted 3 lines · Dave Jones
Quoted 2 lines · Trond Myklebust
O_DIRECTfor NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?Where is Chuck's patch ? I searched but didn't find.
I haven't put it up on my own web-site, but it should be available from the CITI NFS client performance project site. See
http://www.citi.umich.edu/projects/nfs-perf/patches/
Quoted 2 lines
Supporting
direct_IOwith NFS is some meaningful for users who have fast NAS server environment, IMHO.
It can also provide for better data security in some circumstances. Journaling in databases over NFS can for instance benefit greatly, and has been one of Chuck's motivations for doing it.
Cheers,
Trond
Quoted 13 lines · Trond Myklebust
Quoted 8 lines · GOTO Masanori
At Fri, 21 Dec 2001 00:39:42 +0000,
Quoted 3 lines · Dave Jones
Quoted 2 lines · Trond Myklebust
O_DIRECTfor NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?Where is Chuck's patch ? I searched but didn't find.
I haven't put it up on my own web-site, but it should be available from the CITI NFS client performance project site. See
Quoted 8 lines
http://www.citi.umich.edu/projects/nfs-perf/patches/
Quoted 2 lines
Supporting
direct_IOwith NFS is some meaningful for users who have fast NAS server environment, IMHO.It can also provide for better data security in some circumstances. Journaling in databases over NFS can for instance benefit greatly, and has been one of Chuck's motivations for doing it.
the patch is designed for applications that manage their own data cache, like databases do. but it is also useful for applications that want to move large datasets without blowing the O/S level data cache.
in the NFS case, because O_DIRECT read() and write() always go back to the server, you can more easily build clustered and HA applications that share the data storage backend.
- Chuck Lever
--
corporate: <[email protected]>
personal: <[email protected]>
Quoted 8 lines · Dave Jones
Quoted 2 lines · Trond Myklebust
O_DIRECTfor NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?Nope, stock 2.4.17rc2 & 2.5.1.
I thought NFS might just ignore theO_DIRECTflag if it didn't understand it yet, I wasn't expecting such a dramatic failure.
The point of O_DIRECT is to do DMA directly into the userspace memory (and to avoid the VM overhead but that's a secondary issue and with data journaling we may need to put an anchor into the VM to serialize the direct I/O with the pagecache I/O in a secondary - slower - direct_IO callback for the data journaling fs).
But to avoid the mem copies you're required to use strict alignment and size of the userspace buffers, just like rawio.
If you don't you will get -EINVAL. This ensures people will use O_DIRECT correctly in their apps. In short every single bugreport like this about this -EINVAL strict behaviour is the proof we need to be strict and to return -EINVAL :)
Andrea
Shouldn't O_DIRECT's requirements come from the hardware? If we can ASPI or CAM DMA SCSI devices to odd addresses and lengths, why not O_DIRECT? Do ape drives DMA to user buffers? Are O_DIRECT's current limits gratuitous?
Quoted 34 lines · Andrea Arcangeli
Quoted 8 lines · Dave Jones
Quoted 2 lines · Trond Myklebust
O_DIRECTfor NFS isn't yet merged into the kernel. Are these Chuck
Lever's NFS patches you've been testing?Nope, stock 2.4.17rc2 & 2.5.1. I thought NFS might just ignore the
O_DIRECTflag if it didn't understand it yet, I wasn't expecting such a dramatic failure.The point of
O_DIRECTis to do DMA directly into the userspace memory (and to avoid the VM overhead but that's a secondary issue and with data journaling we may need to put an anchor into the VM to serialize the direct I/O with the pagecache I/O in a secondary - slower -direct_IOcallback for the data journaling fs).But to avoid the mem copies you're required to use strict alignment and size of the userspace buffers, just like rawio.
If you don't you will get
-EINVAL. This ensures people will useO_DIRECTcorrectly in their apps. In short every single bugreport like this about this-EINVALstrict behaviour is the proof we need to be strict and to return-EINVAL:)Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/.
Quoted 4 lines · CJ
Shouldn't O_DIRECT's requirements come from the hardware? If we can ASPI or CAM DMA SCSI devices to odd addresses and lengths, why not
O_DIRECT? Do ape drives DMA to user buffers? Are O_DIRECT's current limits gratuitous?
CAM is a very bad thing and that is why the X3 committees split.
Andre Hedrick
Linux Disk Certification Project Linux ATA Development
Quoted 8 lines · Andre Hedrick
Quoted 4 lines · CJ
Shouldn't O_DIRECT's requirements come from the hardware? If we can ASPI or CAM DMA SCSI devices to odd addresses and lengths, why not
O_DIRECT? Do ape drives DMA to user buffers? Are O_DIRECT's current limits gratuitous?CAM is a very bad thing and that is why the X3 committees split.
There were interesting guide-lines in CAM, notably the topology handling and the error recovery scheme. But it was another different wheel in a world where everybody did reinvent its own. It seemed also very DEC tainted.
Btw, given guys like you in X3 committees, I am not surprised that splits occur in this place. :-)
Gérard.
PS: Your various email addresses bounce back claiming some ridiculous text about spammers. Is this still another show of your apparent existential complex.