open.2: improve description of O_DIRECT
PATCH 1/1 open.2: improve description of O_DIRECT
Against man-pages-2.76. Update the description of the O_DIRECT flag to open(2)
- to document the behaviour of O_DIRECT with NFS, and
- to be clearer about the O_DIRECT alignment restriction mess in Linux, and
- to recommend that application writers exercise caution.
Information from reading NFS & XFS source and talking to XFS folks.
Reviewed-by David Chinner <dgc-sJ/[email protected]>
Reviewed-by Jeremy Higdon <jeremy-sJ/[email protected]>
References: SGI:PV975946
---
man2/open.2 | 81 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 63 insertions(+), 18 deletions(-)
man2/open.2 +63 -18
@@ -197,14 +197,9 @@ The I/O is synchronous, that is, at the .BR read (2) or .BR write (2), -data is guaranteed to have been transferred. -Under Linux 2.4 transfer sizes, and the alignment of user buffer -and file offset must all be multiples of the logical block size -of the file system. -Under Linux 2.6 alignment to 512-byte boundaries -suffices. -.\" Alignment should satisfy requirements for the underlying device -.\" There may be coherency problems. +data is guaranteed to have been transferred. See +.B NOTES +below for further discussion. .sp A semantically similar (but deprecated) interface for block devices is described in
@@ -587,17 +582,67 @@ On many systems the file is actually tru .LP The .B O_DIRECT -flag was introduced in SGI IRIX, where it has alignment restrictions -similar to those of Linux 2.4. -IRIX has also a fcntl(2) call to -query appropriate alignments, and sizes. -FreeBSD 4.x introduced -a flag of same name, but without alignment restrictions. -Support was added under Linux in kernel version 2.4.10. +flag may impose alignment restrictions on the length and address +of userspace buffers and the file offset of I/Os. In Linux alignment +restrictions vary by filesystem and kernel version and might be +absent entirely. However there is currently no filesystem\-independent +interface for an application to discover these restrictions for a given +file or filesystem. Some filesystems provide their own interfaces +for doing so, for example the +.B XFS_IOC_DIOINFO +operation in +.BR xfsctl (3). +.LP +Under Linux 2.4, transfer sizes, and the alignment of user buffer +and file offset must all be multiples of the logical block size +of the file system. Under Linux 2.6, alignment to 512-byte boundaries +suffices. The flag was introduced in SGI IRIX, where it has alignment +restrictions similar to those of Linux 2.4. IRIX has also a fcntl(2) +call to query appropriate alignments, and sizes. FreeBSD 4.x introduced +a flag of the same name, but without alignment restrictions. +.LP +.B O_DIRECT +support was added under Linux in kernel version 2.4.10. Older Linux kernels simply ignore this flag. -One may have to define the -.B _GNU_SOURCE -macro to get its definition. +Some filesystems may not implement the flag and +.B open +will fail with EINVAL if it is used. +.LP +Applications should avoid mixing +.B O_DIRECT +and normal I/O to the same +file, and especially to overlapping byte regions in the same file. +Even when the filesystem correctly handles the coherency issues in +this situation, overall I/O throughput is likely to be slower than +using either mode alone. Likewise, applications should avoid mixing +.BR mmap (2) +of files with direct I/O to the same files. +.LP +The behaviour of +.B O_DIRECT +with NFS will differ from local filesystems. Older kernels, or +kernels configured in certain ways, may not support this combination. +The NFS protocol does not support passing the flag to the server, so +.B O_DIRECT +I/O will only bypass the page cache on the client; the server may +still cache the I/O. The client asks the server to make the I/O +synchronous to preserve the synchronous semantics of +.BR O_DIRECT . +Some servers will perform poorly under these circumstances, especially +if the I/O size is small. Some servers may also be configured to +lie to clients about the I/O having reached stable storage; this +will avoid the performance penalty at some risk to data integrity +in the event of server power failure. The Linux NFS client places +no alignment restrictions on +.B O_DIRECT +I/O. +.PP +In summary, +.B O_DIRECT +is a potentially powerful tool that should be used with caution. It +is recommended that applications treat use of +.B O_DIRECT +as a performance option which is disabled by default. .PP There are many infelicities in the protocol underlying NFS, affecting amongst others
Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group. The cake is *not* a lie. I don't speak for SGI.
A message that is not in the archive (sent off-list or not yet received): 4796E05E.1000508-cP1dWloDopni96+mSzHFpQC/[email protected]
Hi Greg,
Sorry for the delay in following up.
In general the patch looks great -- thanks! I have one small question, noted below.
Quoted 68 lines · Greg Banks
Against man-pages-2.76. Update the description of the
O_DIRECTflag to open(2)- to document the behaviour of
O_DIRECTwith NFS, and- to be clearer about the
O_DIRECTalignment restriction mess in Linux, and- to recommend that application writers exercise caution.
Information from reading NFS & XFS source and talking to XFS folks.
Reviewed-by David Chinner <dgc-sJ/[email protected]>Reviewed-by Jeremy Higdon <jeremy-sJ/[email protected]>References: SGI:PV975946
---man2/open.2 | 81 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 63 insertions(+), 18 deletions(-)Index: man-pages-2.76/man2/open.2
===================================================================man2/open.2 +18
-15@@ -197,14 +197,9 @@ The I/O is synchronous, that is, at the .BR read (2) or .BR write (2), -data is guaranteed to have been transferred. -Under Linux 2.4 transfer sizes, and the alignment of user buffer -and file offset must all be multiples of the logical block size -of the file system. -Under Linux 2.6 alignment to 512-byte boundaries -suffices. -.\" Alignment should satisfy requirements for the underlying device -.\" There may be coherency problems. +data is guaranteed to have been transferred. See +.B NOTES +below for further discussion. .sp A semantically similar (but deprecated) interface for block devices is described in@@ -587,17 +582,67 @@ On many systems the file is actually tru .LP The .B O_DIRECT -flag was introduced in SGI IRIX, where it has alignment restrictions -similar to those of Linux 2.4. -IRIX has also a fcntl(2) call to -query appropriate alignments, and sizes. -FreeBSD 4.x introduced -a flag of same name, but without alignment restrictions. -Support was added under Linux in kernel version 2.4.10. +flag may impose alignment restrictions on the length and address +of userspace buffers and the file offset of I/Os. In Linux alignment +restrictions vary by filesystem and kernel version and might be +absent entirely. However there is currently no filesystem\-independent +interface for an application to discover these restrictions for a given +file or filesystem. Some filesystems provide their own interfaces +for doing so, for example the +.B XFS_IOC_DIOINFO +operation in +.BR xfsctl (3). +.LP +Under Linux 2.4, transfer sizes, and the alignment of user buffer +and file offset must all be multiples of the logical block size +of the file system. Under Linux 2.6, alignment to 512-byte boundaries +suffices. The flag was introduced in SGI IRIX, where it has alignment
man2/open.2 On many systems the file is actually tru · line 600
Quoted 10 lines
man2/open.2 +6
-3+restrictions similar to those of Linux 2.4. IRIX has also a fcntl(2) +call to query appropriate alignments, and sizes. FreeBSD 4.x introduced +a flag of the same name, but without alignment restrictions. +.LP +.B O_DIRECT +support was added under Linux in kernel version 2.4.10. Older Linux kernels simply ignore this flag. -One may have to define the -.B _GNU_SOURCE -macro to get its definition.
I take it that you removed that last sentence because the information is repeated elsewhere on the page?
Quoted 32 lines
man2/open.2 +32
-0+Some filesystems may not implement the flag and +.B open +will fail with EINVAL if it is used. +.LP +Applications should avoid mixing +.B O_DIRECT +and normal I/O to the same +file, and especially to overlapping byte regions in the same file. +Even when the filesystem correctly handles the coherency issues in +this situation, overall I/O throughput is likely to be slower than +using either mode alone. Likewise, applications should avoid mixing +.BR mmap (2) +of files with direct I/O to the same files. +.LP +The behaviour of +.B O_DIRECT +with NFS will differ from local filesystems. Older kernels, or +kernels configured in certain ways, may not support this combination. +The NFS protocol does not support passing the flag to the server, so +.B O_DIRECT +I/O will only bypass the page cache on the client; the server may +still cache the I/O. The client asks the server to make the I/O +synchronous to preserve the synchronous semantics of +.BR O_DIRECT . +Some servers will perform poorly under these circumstances, especially +if the I/O size is small. Some servers may also be configured to +lie to clients about the I/O having reached stable storage; this +will avoid the performance penalty at some risk to data integrity +in the event of server power failure. The Linux NFS client places +no alignment restrictions on +.B O_DIRECT +I/O.
man2/open.2 On many systems the file is actually tru · line 648
Quoted 10 lines
man2/open.2 +7
-0+.PP +In summary, +.B O_DIRECT +is a potentially powerful tool that should be used with caution. It +is recommended that applications treat use of +.B O_DIRECT +as a performance option which is disabled by default. .PP There are many infelicities in the protocol underlying NFS, affecting amongst others
I applied your patch, did some very light (formatting) edits to your changes, and reorganized the NOTES section a little afterwards, so that the O_DIRECT material stands in a subsection of its own. Also, your new material gives much better context to Linus's quote, so I relocated that quote from BUGS into NOTES.
I also added your name to the list of copyright holders for the page, since
you have added a substantial piece to the page.
The changes will be in man-pages-2.78.
Cheers,
Michael
Signature · 5 lines
Michael Kerrisk Maintainer of the Linux man-pages project http://www.kernel.org/doc/man-pages/ Want to report a man-pages bug? Look here: http://www.kernel.org/doc/man-pages/reporting_bugs.html