dd PATCH: add conv=direct
Linux-kernel: is this patch horribly wrong?
On modern Linux, apparently the correct way to bypass the buffer cache when writing to a block device is to open the block device with O_DIRECT. This enables, for example, the user to more easily force a reallocation of a single sector of an IDE disk with a media error (without overwriting anything but the 1k "sector pair" containing the error). dd(1) is convenient for this purpose, but is lacking a method to force O_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.
Alas, I don't do autoconf, so I hope someone else can add the appropriate stanza to autoconfiscate HAS_O_DIRECT.
-andy
doc/coreutils.texi +5 -0
@@ -6373,6 +6373,11 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. When used with @samp{block} or @samp{unblock}, pad with spaces instead of zero bytes. + +@item direct +@opindex direct +Open the output file with O_DIRECT, avoiding (on Linux) using the buffer +cache. @end table @end table
src/dd.c +12 -0
@@ -66,6 +66,9 @@/* Default input and output blocksize. */ #define DEFAULT_BLOCKSIZE 512 +/* XXX hack */ +#define HAVE_O_DIRECT 1 +/* Conversions bit masks. */ #define C_ASCII 01#define C_EBCDIC 02
@@ -78,6 +81,7 @@#define C_NOERROR 0400#define C_NOTRUNC 01000#define C_SYNC 02000 +#define C_DIRECT 010000/* Use separate input and output buffers, and combine partial input blocks. */ #define C_TWOBUFS 04000
@@ -162,6 +166,9 @@ {"noerror", C_NOERROR}, /* Ignore i/o errors. */ {"notrunc", C_NOTRUNC}, /* Do not truncate output file. */ {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ +#ifdef HAVE_O_DIRECT + {"direct", C_DIRECT}, /* open files with O_DIRECT */ +#endif {NULL, 0} };
@@ -1190,6 +1197,11 @@ = (O_CREAT | (seek_records || (conversions_mask &C_NOTRUNC ) ?0 :O_TRUNC )); +#if HAVE_O_DIRECT +if (conversions_mask &C_DIRECT ) + opts |=O_DIRECT ; +#endif +/* Open the output file with *read* access only if we might need to read to satisfy a `seek=' request. If we can't read the file, go ahead with write-only access; it might work. */
Quoted 9 lines · Andy Isaacson
On modern Linux, apparently the correct way to bypass the buffer cache when writing to a block device is to open the block device with
O_DIRECT. This enables, for example, the user to more easily force a reallocation of a single sector of an IDE disk with a media error (without overwriting anything but the 1k "sector pair" containing the error). dd(1) is convenient for this purpose, but is lacking a method to forceO_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.
This would be rather nice to have. You'll need to ensure that the data is page-aligned in memory.
While you're there, please add an fsync-before-closing option.
Quoted 9 lines
Quoted 8 lines
On modern Linux, apparently the correct way to bypass the buffer cache when writing to a block device is to open the block device with
O_DIRECT. This enables, for example, the user to more easily force a reallocation of a single sector of an IDE disk with a media error (without overwriting anything but the 1k "sector pair" containing the error). dd(1) is convenient for this purpose, but is lacking a method to forceO_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.
Quoted 4 lines
This would be rather nice to have. You'll need to ensure that the data is page-aligned in memory.
While you're there, please add an fsync-before-closing option.
Andrew, am I right that this is NOT needed for the proposed O_DIRECT option, since open(2) says:
"The I/O is synchronous, i.e., at the completion of the read(2) or
write(2) system call, data is guaranteed to have been transferred."
so the write will block until data is physically on the disk.
Cheers,
Bruce
Quoted 13 lines · Bruce Allen
Quoted 13 lines
Quoted 9 lines
On modern Linux, apparently the correct way to bypass the buffer cache when writing to a block device is to open the block device with
O_DIRECT. This enables, for example, the user to more easily force a reallocation of a single sector of an IDE disk with a media error (without overwriting anything but the 1k "sector pair" containing the error). dd(1) is convenient for this purpose, but is lacking a method to forceO_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.This would be rather nice to have. You'll need to ensure that the data is page-aligned in memory.
Quoted 7 lines
Quoted 1 line
While you're there, please add an fsync-before-closing option.
Andrew, am I right that this is NOT needed for the proposed
O_DIRECToption, since open(2) says:
"The I/O is synchronous, i.e., at the completion of the read(2) or
write(2) system call, data is guaranteed to have been transferred."
so the write will block until data is physically on the disk.
A conv=fsync option is an irrelevant wishlist item. If you were to provide conv=fsync then dd should still fsync the file after performing the direct-io read or write.
You're correct that an fsync after a direct-io read or write should not need to perform any file data I/O. But it will often need to perform file metadata I/O - file allocation tables, inode, etc. direct-io only syncs the file data, not the file metadata.
Quoted 7 lines · Andrew Morton
Quoted 3 lines · Andy Isaacson
dd(1) is convenient for this purpose, but is lacking a method
to forceO_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.This would be rather nice to have. You'll need to ensure that the data is page-aligned in memory.
So, some confusion on my part about O_DIRECT: I can't get O_DIRECT to work on ext3, at all, on 2.4.25 -- open(O_DIRECT) succeeds, but the write returns EINVAL. Same code works fine when writing to a block device. If the problem is that ext3 can't support O_DIRECT, why does the open succeed?
Quoted 1 line
While you're there, please add an fsync-before-closing option.
Easy enough. How does this look? Note that C_TWOBUFS ensures the output buffer is getpagesize()-aligned.
The next feature to add would be OpenBSD-style "KB/s" reporting. I'm not going there.
-andy
2004-04-07 Andy Isaacson <[email protected]>
* add conv=direct and conv=fsync options.
doc/coreutils.texi +10 -0
@@ -6373,6 +6373,16 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. When used with @samp{block} or @samp{unblock}, pad with spaces instead of zero bytes. + +@item fsync +@opindex fsync +Call @samp{fsync(2)} on the output file before exiting. This ensures +that the file data is written to permanent store. + +@item direct +@opindex direct +Open the output file with O_DIRECT, avoiding (on Linux) using the buffer +cache. @end table @end table
src/dd.c +20 -0
@@ -66,6 +66,9 @@/* Default input and output blocksize. */ #define DEFAULT_BLOCKSIZE 512 +/* XXX hack */ +#define HAVE_O_DIRECT 1 +/* Conversions bit masks. */ #define C_ASCII 01#define C_EBCDIC 02
@@ -78,6 +81,8 @@#define C_NOERROR 0400#define C_NOTRUNC 01000#define C_SYNC 02000 +#define C_FSYNC 010000 +#define C_DIRECT 020000/* Use separate input and output buffers, and combine partial input blocks. */ #define C_TWOBUFS 04000
@@ -162,6 +167,10 @@ {"noerror", C_NOERROR}, /* Ignore i/o errors. */ {"notrunc", C_NOTRUNC}, /* Do not truncate output file. */ {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ + {"fsync", C_FSYNC}, /* call fsync(2) before closing output file */ +#ifdef HAVE_O_DIRECT + {"direct", C_DIRECT | C_TWOBUFS}, /* open files with O_DIRECT */ +#endif {NULL, 0} };
@@ -1190,6 +1199,11 @@ = (O_CREAT | (seek_records || (conversions_mask &C_NOTRUNC ) ?0 :O_TRUNC )); +#if HAVE_O_DIRECT +if (conversions_mask &C_DIRECT ) + opts |=O_DIRECT ; +#endif +/* Open the output file with *read* access only if we might need to read to satisfy a `seek=' request. If we can't read the file, go ahead with write-only access; it might work. */
@@ -1240,5 +1254,11 @@ exit_status =dd_copy (); +if (conversions_mask &C_FSYNC ) + { +if (fsync (STDOUT_FILENO ) !=0 ) +error (0 , errno,_ ("cannot fsync %s" ),quote (output_file)); + } +quit (exit_status); }
Quoted 7 lines · Andy Isaacson
Quoted 5 lines · Andrew Morton
Quoted 3 lines · Andy Isaacson
dd(1) is convenient for this purpose, but is lacking a method
to forceO_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.
Quoted 5 lines
Quoted 2 lines
This would be rather nice to have. You'll need to ensure that the data is page-aligned in memory.
So, some confusion on my part about
O_DIRECT: I can't getO_DIRECTto work on ext3, at all, on 2.4.25
ext3 doesn't support O_DIRECT in 2.4 kernels. I did a patch once and I think it's in 2.4-aa kernels.
ext3 supports O_DIRECT in 2.6 kernels. Quite a number of filesystems do.
Quoted 2 lines
--
open(O_DIRECT)succeeds, but the write
returns EINVAL.
Yup that's a bit silly. In 2.6 we do the check at open() and fcntl() time. In 2.4 we don't fail until the actual I/O attempt.
Quoted 3 lines
Same code works fine when writing to a block device.
If the problem is that ext3 can't supportO_DIRECT, why does the open succeed?
We have been insufficiently assiduous in merging externally-supported patches into the mainline 2.4 tree.
Quoted 4 lines
Quoted 1 line
While you're there, please add an fsync-before-closing option.
Easy enough. How does this look? Note that
C_TWOBUFSensures the output buffer is getpagesize()-aligned.
Looks nice and simple. You'll need an ext2 filesystem to test it under 2.4.
Be aware that it's rather a challenge to actually get the O_DIRECT #define in scope under some glibc versions. I think you need to define _GNU_SOURCE or something like that.
Quoted 7 lines · Andrew Morton
Quoted 4 lines · Andy Isaacson
Quoted 1 line
While you're there, please add an fsync-before-closing option.
Easy enough. How does this look? Note that
C_TWOBUFSensures the output buffer is getpagesize()-aligned.Looks nice and simple. You'll need an ext2 filesystem to test it under 2.4.
I tested against a block device. Easier to just "swapoff /dev/hda2" than to find space to make a new filesystem.
Quoted 3 lines
Be aware that it's rather a challenge to actually get the
O_DIRECT#define in scope under some glibc versions. I think you need to define_GNU_SOURCEor something like that.
Yeah, I had to -D_GNU_SOURCE to build standalone, but coreutils' makefile seems to do OK.
Would there be any reason to allow O_DIRECT on the read side?
-andy
Quoted 2 lines · Andy Isaacson
Would there be any reason to allow
O_DIRECTon the read side?
Sure. It saves CPU, avoids blowing pagecache, just as with O_DIRECT writes.
Quoted 4 lines · Andrew Morton
Quoted 1 line · Andy Isaacson
Would there be any reason to allow
O_DIRECTon the read side?Sure. It saves CPU,
OK, I can see that one. But it seems like a pretty small benefit to me -- CPU utilization is already really low.
Quoted 1 line
avoids blowing pagecache,
Um, that sounds like a bad idea to me. It seems to me it's the kernel's responsibility to figure out "hey, looks like a streaming read - let's not blow out the buffer cache trying to hold 20GB on a 512M system." If you're saying that the kernel guys have given up and the established wisdom is now "you gotta use O_DIRECT if you don't want to throw everything else out due to streaming data", well... I'm disappointed.
Quoted 1 line
just as with
O_DIRECTwrites.
Wouldn't opening both if= and of= with O_DIRECT turn dd into a synchronous copier? That would really suck in the
"dd if=/dev/hda1 of=/dev/hdc1" case. With buffer cache doing readahead, that command can get, say, 40MB/s read and 40MB/s write; with synch read and synch write, it would drop to 40MB/s read+write, assuming that block sizes are big enough to amortize seek overhead.
Having O_DIRECT on just of=, I think you can get back to 40MB/s+40MB/s.
I claim that O_DIRECT on of= is important because you just plain *can not* do the minimal-sized IDE block scrub without it. I don't yet see a similar benefit to O_DIRECT on if= side.
-andy
Quoted 4 lines · Andy Isaacson
I claim that
O_DIRECTon of= is important because you just plain *can not* do the minimal-sized IDE block scrub without it. I don't yet see a similar benefit toO_DIRECTon if= side.
If you want a block scrubber then write a block scrubber.
If you want to add O_DIRECT support to dd then it should be implemented properly, and that means implementing it for both read and write.
In fact the user should be able to specify the read-O_DIRECT and the write-O_DIRECT independently - if for no other reason than that the source and dest filesytems may not both support O_DIRECT.
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Andy Isaacson
I claim that
O_DIRECTon of= is important because you just plain *can not* do the minimal-sized IDE block scrub without it. I don't yet see a similar benefit toO_DIRECTon if= side.If you want a block scrubber then write a block scrubber.
They exist. They're a pain in the ass to find when you need one. dd is almost there; a small patch that might have a chance of getting accepted upstream seemed like a reasonable time investment.
Quoted 6 lines
If you want to add
O_DIRECTsupport to dd then it should be implemented properly, and that means implementing it for both read and write.In fact the user should be able to specify the read-O_DIRECT and the write-O_DIRECT independently - if for no other reason than that the source and dest filesytems may not both support
O_DIRECT.
Of course if both directions are supported they must be independently specifiable. I just don't see a compelling use case for the input side.
Nevertheless, here you go. Still needs some autoconfiscation.
2004-04-07 Andy Isaacson <[email protected]>
* add conv=direct, conv=idirect, and conv=fsync options.
doc/coreutils.texi +14 -0
@@ -6373,6 +6373,20 @@ Pad every input block to size of @samp{ibs} with trailing zero bytes. When used with @samp{block} or @samp{unblock}, pad with spaces instead of zero bytes. + +@item fsync +@opindex fsync +Call @samp{fsync(2)} on the output file before exiting. This ensures +that the file data is written to permanent store. + +@item direct +@opindex direct +Open the output file with O_DIRECT, avoiding (on Linux) using the buffer +cache. + +@item idirect +@opindex idirect +Open the input file with O_DIRECT, avoiding (on Linux) using the buffer @end table @end table
src/dd.c +30 -1
@@ -66,6 +66,9 @@/* Default input and output blocksize. */ #define DEFAULT_BLOCKSIZE 512 +/* XXX hack */ +#define HAVE_O_DIRECT 1 +/* Conversions bit masks. */ #define C_ASCII 01#define C_EBCDIC 02
@@ -78,6 +81,9 @@#define C_NOERROR 0400#define C_NOTRUNC 01000#define C_SYNC 02000 +#define C_FSYNC 010000 +#define C_DIRECT 020000 +#define C_IDIRECT 040000/* Use separate input and output buffers, and combine partial input blocks. */ #define C_TWOBUFS 04000
@@ -162,6 +168,11 @@ {"noerror", C_NOERROR}, /* Ignore i/o errors. */ {"notrunc", C_NOTRUNC}, /* Do not truncate output file. */ {"sync", C_SYNC}, /* Pad input records to ibs with NULs. */ + {"fsync", C_FSYNC}, /* call fsync(2) before closing output file */ +#ifdef HAVE_O_DIRECT + {"direct", C_DIRECT | C_TWOBUFS}, /* open output with O_DIRECT */ + {"idirect", C_IDIRECT | C_TWOBUFS}, /* open input with O_DIRECT */ +#endif {NULL, 0} };
@@ -1177,7 +1188,14 @@if (input_file !=NULL ) { -if (open_fd (STDIN_FILENO , input_file,O_RDONLY ,0 ) <0 ) +int opts =O_RDONLY ; + +#if HAVE_O_DIRECT +if (conversions_mask &C_IDIRECT ) + opts |=O_DIRECT ; +#endif + +if (open_fd (STDIN_FILENO , input_file, opts,0 ) <0 )error (EXIT_FAILURE , errno,_ ("opening %s" ),quote (input_file)); }else
@@ -1190,6 +1208,11 @@ = (O_CREAT | (seek_records || (conversions_mask &C_NOTRUNC ) ?0 :O_TRUNC )); +#if HAVE_O_DIRECT +if (conversions_mask &C_DIRECT ) + opts |=O_DIRECT ; +#endif +/* Open the output file with *read* access only if we might need to read to satisfy a `seek=' request. If we can't read the file, go ahead with write-only access; it might work. */
@@ -1240,5 +1263,11 @@ exit_status =dd_copy (); +if (conversions_mask &C_FSYNC ) + { +if (fsync (STDOUT_FILENO ) !=0 ) +error (0 , errno,_ ("cannot fsync %s" ),quote (output_file)); + } +quit (exit_status); }
Quoted 3 lines · Andy Isaacson
Quoted 1 line · Andrew Morton
If you want a block scrubber then write a block scrubber.
They exist. They're a pain in the ass to find when you need one. dd is
And finding a block scrubber that actually works *properly* is even more of a challenge. I've seen more than a few that do stupid things like truncating the file and then writing the appropriate number of bytes....
Quoted 9 lines
Quoted 6 lines
If you want to add
O_DIRECTsupport to dd then it should be implemented properly, and that means implementing it for both read and write.In fact the user should be able to specify the read-O_DIRECT and the write-O_DIRECT independently - if for no other reason than that the source and dest filesytems may not both support
O_DIRECT.Of course if both directions are supported they must be independently specifiable. I just don't see a compelling use case for the input side.
Andy, since I was the one who suggested adding a conv=odirect flag to the output side, so that we could use a standard tool (dd) to force reallocation of bad sectors by writing to them, let me argue that it ALSO makes sense to allow O_DIRECT (as an independent option) on the input side.
At least one obvious use is to help FIND unreadable disk sectors (or to verify that some sector is indeed unreadable). One would like to be able to do:
dd if=/dev/hda of=/dev/null bs=512 count=1 skip=LBA conv=idirect
and see if the dd suceeds or fails.
If dd fails, then without having an O_DIRECT option (idirect) at the input side, there is no way of telling if the failure was because of an unreadable sector at LBA, or an unreadable sector somewhere else in the block containing LBA. With the O_DIRECT input option, you can be sure that if this command fails it's because the sector at LBA was unreadable, not some nearby sector.
Cheers,
Bruce
Quoted 2 lines · Bruce Allen
let me argue that it ALSO makes sense to allow
O_DIRECT(as an independent option) on the input side.
At the end of this message is a proposed patch to implement everything people other than myself have asked for so far, along with several other things since I was in the neighborhood.
This patch adds the following NEWS entry, which I'll repeat here to make it easy to see what's going on:
dd has new conversions for the conv= option:
nocreat do not create the output file
excl fail if the output file already exists
fdatasync physically write output file data before finishing
fsync likewise, but also write metadata
dd has new iflags= and oflags= options with the following flags:
append append mode (makes sense for output file only)
direct use direct I/O for data
dsync use synchronized I/O for data
sync likewise, but also for metadata
nonblock use non-blocking I/O
nofollow do not follow symlinks
noctty do not assign controlling terminal from fileThis patch is relative to coreutils CVS, so your line numbers may differ. I didn't add hooks for Solaris-style directio, because that turns out to be a systemwide property and perhaps should be in a different command.
2004-04-07 Paul Eggert <[email protected]>
* NEWS: New dd conv= symbols nocreat, excl, fdatasync, fsync, and new dd options iflag= and oflag=.
* doc/coreutils.texi (dd invocation): Likewise.
* src/dd.c (usage): Likewise.
* m4/jm-macros.m4 (jm_MACROS): Check for fdatasync within -lrt and -lposix4, so that it can be used in Solaris 2.5.1 and later.
* src/Makefile.am (dd_LDADD, shred_LDADD): Add fdatasync's lib.
* src/dd.c (fdatasync) [!HAVE_FDATASYNC]: New macro. (C_NOCREAT, C_EXCL, C_FDATASYNC, C_FSYNC): New macros. (input_flags, output_flags): New vars.
(LONGEST_SYMBOL): New macro.
(struct symbol_value): Renamed from struct conversion. Members symbol and value renamed from convname and conversion. The symbol value is now an array instead of a pointer; this saves a bit of space and time in practice. All uses changed. (conversions): Add nocreat, excl, fdatasync, fsync. Now const. (flags): New constant array.
(iflag_error_msgid, oflag_error_msgid): New constants. (parse_symbols): Renamed from parse_conversion and generalized to handle either conversion or flag symbols.
(scanargs): Adjust uses of parse_symbols accodingly. Add support for iflag= and oflag=. Reject attempts to use both excl and nocreat.
(set_fd_flags): New function.
(dd_copy): Just return X rather than calling quit (X), since our caller invokes quit with the returned value. Add support for fdatasync and fsync.
(main): Add support for iflags=, oflags=, and new conv= symbols.
* src/system.h (O_DIRECT, O_DSYNC, O_NDELAY, O_NOFOLLOW, O_RSYNC, O_SYNC): Define to 0 if not already defined.
* NEWS: Remove duplicate mention of BLOCKSIZE.
Index: NEWS
===================================================================
RCS file: /home/meyering/coreutils/cu/NEWS,v
retrieving revision 1.198
NEWS +17 -7
@@ -9,14 +9,24 @@ GNU coreutils NEWS ** New features - stty now provides support (iutf8) for setting UTF-8 input mode. + dd has new conversions for the conv= option: - 'df', 'du', and 'ls' now take the default block size from the - BLOCKSIZE environment variable if the BLOCK_SIZE, DF_BLOCK_SIZE, - DU_BLOCK_SIZE, and LS_BLOCK_SIZE environment variables are not set. - Unlike the other variables, though, BLOCKSIZE does not affect - values like 'ls -l' sizes that are normally displayed as bytes. - This new behavior is for compatibility with BSD. + nocreat do not create the output file + excl fail if the output file already exists + fdatasync physically write output file data before finishing + fsync likewise, but also write metadata + + dd has new iflags= and oflags= options with the following flags: + + append append mode (makes sense for output file only) + direct use direct I/O for data + dsync use synchronized I/O for data + sync likewise, but also for metadata + nonblock use non-blocking I/O + nofollow do not follow symlinks + noctty do not assign controlling terminal from file + + stty now provides support (iutf8) for setting UTF-8 input mode. With stat, a specified format is no longer automatically newline terminated. If you want a newline at the end of your output, append `\n' to the format
Index: doc/coreutils.texi
===================================================================
RCS file: /home/meyering/coreutils/cu/doc/coreutils.texi,v retrieving revision 1.175
coreutils.texi +89 -0
102 lines not on this page · raw message
Index: m4/jm-macros.m4
===================================================================
RCS file: /home/meyering/coreutils/cu/m4/jm-macros.m4,v retrieving revision 1.184
jm-macros.m4 +9 -1
@@ -81,7 +81,6 @@ AC_DEFUN([jm_MACROS], AC_CHECK_FUNCS( \ endgrent \ endpwent \ - fdatasync \ ftruncate \ gethrtime \ hasmntopt \
@@ -110,6 +109,15 @@ AC_DEFUN([jm_MACROS], AC_FUNC_STRTOD AC_REQUIRE([GL_FUNC_GETCWD_PATH_MAX]) AC_REQUIRE([GL_FUNC_READDIR]) + + # for dd.c and shred.c + fetish_saved_libs=$LIBS + AC_SEARCH_LIBS([fdatasync], [rt posix4], + [test "$ac_cv_search_fdatasync" = "none required" || + LIB_FDATASYNC=$ac_cv_search_fdatasync]) + AC_SUBST([LIB_FDATASYNC]) + AC_CHECK_FUNCS(fdatasync) + LIBS=$fetish_saved_libs # See if linking `seq' requires -lm. # It does on nearly every system. The single exception (so far) is
Index: src/Makefile.am
===================================================================
RCS file: /home/meyering/coreutils/cu/src/Makefile.am,v retrieving revision 1.34
Makefile.am +3 -2
@@ -32,10 +32,11 @@ AM_CPPFLAGS = -I.. -I$(srcdir) -I$(top_s # replacement functions defined in libfetish.a. LDADD = ../lib/libfetish.a $(LIBINTL) ../lib/libfetish.a -# for clock_gettime +# for clock_gettime and fdatasync +dd_LDADD = $(LDADD) $(LIB_FDATASYNC) dir_LDADD = $(LDADD) $(LIB_CLOCK_GETTIME) ls_LDADD = $(LDADD) $(LIB_CLOCK_GETTIME) -shred_LDADD = $(LDADD) $(LIB_CLOCK_GETTIME) +shred_LDADD = $(LDADD) $(LIB_CLOCK_GETTIME) $(LIB_FDATASYNC) vdir_LDADD = $(LDADD) $(LIB_CLOCK_GETTIME) ## If necessary, add -lm to resolve use of pow in lib/strtod.c.
Index: src/dd.c
===================================================================
RCS file: /home/meyering/coreutils/cu/src/dd.c,v
retrieving revision 1.154
dd.c +164 -39
356 lines not on this page · raw message
Index: src/system.h
===================================================================
RCS file: /home/meyering/coreutils/cu/src/system.h,v retrieving revision 1.82
system.h +20 -0
@@ -197,6 +197,14 @@ initialize_exit_failure (int status)# define O_TEXT _O_TEXT#endif +#if !definedO_DIRECT +# define O_DIRECT 0 +#endif + +#if !definedO_DSYNC +# define O_DSYNC 0 +#endif +#if !definedO_NDELAY # define O_NDELAY 0#endif
@@ -207,6 +215,18 @@ initialize_exit_failure (int status)#if !definedO_NOCTTY # define O_NOCTTY 0 +#endif + +#if !definedO_NOFOLLOW +# define O_NOFOLLOW 0 +#endif + +#if !definedO_RSYNC +# define O_RSYNC 0 +#endif + +#if !definedO_SYNC +# define O_SYNC 0#endif #ifdef __BEOS__
Quoted 3 lines · Paul Eggert
At the end of this message is a proposed patch to implement everything people other than myself have asked for so far, along with several other things since I was in the neighborhood.
Thanks for that nice patch!
I've applied it, and made these additional changes:
2004-04-08 Jim Meyering <[email protected]>
* src/dd.c (set_fd_flags): Don't OR in -1 when fcntl fails. Rename parameter, flags, to avoid shadowing global.
Index: dd.c
===================================================================
RCS file: /fetish/cu/src/dd.c,v
retrieving revision 1.155
dd.c +3 -3
@@ -1017,12 +1017,12 @@ copy_with_unblock (char const *buf, sizein FLAGS. The file's name is NAME. */ static void -set_fd_flags (int fd,int flags,char const *name) +set_fd_flags (int fd,int add_flags,char const *name) { -if (flags) +if (add_flags) {int old_flags =fcntl (fd,F_GETFL ); -int new_flags = old_flags | flags; +int new_flags = old_flags <0 ? add_flags : (old_flags | add_flags);if (old_flags <0 || (new_flags != old_flags &&fcntl (fd,F_SETFL , new_flags) ==-1 ))error (EXIT_FAILURE , errno,_ ("setting flags for %s" ),quote (name));
Quoted 3 lines · Jim Meyering
2004-04-08 Jim Meyering <[email protected]>
*
src/dd.c(set_fd_flags): Don't OR in -1 when fcntl fails.
Doesn't that fix generate worse code in the usual case, since it causes two conditional branches instead of one?
How about this further patch? It relies on common subexpression elimination, but that's common these days.
2004-04-08 Paul Eggert <[email protected]>
* src/dd.c (set_fd_flags): Don't test old_flags < 0 twice.
Index: src/dd.c
===================================================================
RCS file: /home/meyering/coreutils/cu/src/dd.c,v
retrieving revision 1.157
dd.c +3 -3
@@ -1014,7 +1014,7 @@ copy_with_unblock (char const *buf, size }/* Set the file descriptor flags for FD that correspond to the nonzero bits -in FLAGS. The file's name is NAME. */ +in ADD_FLAGS. The file's name is NAME. */ static void set_fd_flags (int fd,int add_flags,char const *name)
@@ -1022,9 +1022,9 @@ set_fd_flags (int fd, int add_flags, chaif (add_flags) {int old_flags =fcntl (fd,F_GETFL ); -int new_flags = old_flags <0 ? add_flags : (old_flags | add_flags);if (old_flags <0 - || (new_flags != old_flags &&fcntl (fd,F_SETFL , new_flags) ==-1 )) + || (old_flags != (old_flags | add_flags) + &&fcntl (fd,F_SETFL , old_flags | add_flags) ==-1 ))error (EXIT_FAILURE , errno,_ ("setting flags for %s" ),quote (name)); } }
Quoted 7 lines · Paul Eggert
Quoted 3 lines · Jim Meyering
2004-04-08 Jim Meyering <[email protected]>
*
src/dd.c(set_fd_flags): Don't OR in -1 when fcntl fails.Doesn't that fix generate worse code in the usual case, since it causes two conditional branches instead of one?
I think it's unnecessary anyway - if old_flags is -1, then the conditional jumps to error() before looking at new_flags at all.
paul
Quoted 9 lines · Paul Eggert
Quoted 3 lines · Jim Meyering
2004-04-08 Jim Meyering <[email protected]>
*
src/dd.c(set_fd_flags): Don't OR in -1 when fcntl fails.Doesn't that fix generate worse code in the usual case, since it causes two conditional branches instead of one?
Quoted 6 lines
How about this further patch? It relies on common subexpression elimination, but that's common these days.
2004-04-08 Paul Eggert <[email protected]>
*
src/dd.c(set_fd_flags): Don't testold_flags< 0 twice.
I see your point. Thanks for the comment fix. I've gone ahead and reverted my change,
since I think your original code is a little more readable than the more-efficient-when-fcntl-fails code that you're proposing.
(set_fd_flags): Undo part of today's change: it's a little cleaner -- and more efficient in the common case -- to go ahead and OR in the -1 when fcntl fails.
Index: src/dd.c
===================================================================
RCS file: /fetish/cu/src/dd.c,v
retrieving revision 1.158
retrieving revision 1.159
src/dd.c +2 -2
@@ -1014,7 +1014,7 @@ copy_with_unblock (char const *buf, size }/* Set the file descriptor flags for FD that correspond to the nonzero bits -in FLAGS. The file's name is NAME. */ +in ADD_FLAGS. The file's name is NAME. */ static void set_fd_flags (int fd,int add_flags,char const *name)
@@ -1022,7 +1022,7 @@ set_fd_flags (int fd, int add_flags, chaif (add_flags) {int old_flags =fcntl (fd,F_GETFL ); -int new_flags = old_flags <0 ? add_flags : (old_flags | add_flags); +int new_flags = old_flags | add_flags;if (old_flags <0 || (new_flags != old_flags &&fcntl (fd,F_SETFL , new_flags) ==-1 ))error (EXIT_FAILURE , errno,_ ("setting flags for %s" ),quote (name));
Quoted 9 lines · Paul Eggert
dd has new iflags= and oflags= options with the following flags:
append append mode (makes sense for output file only) direct use direct I/O for data dsync use synchronized I/O for data sync likewise, but also for metadata nonblock use non-blocking I/O nofollow do not follow symlinks noctty do not assign controlling terminal from file
noctty definitely seems overkill... I can't see why dd would ever want to open a file without O_NOCTTY. On systems where O_NOCTTY makes sense (SvR4) that is.
Phil.
dd patch to remove noctty
Quoted 2 lines · Philippe Troin
noctty definitely seems overkill... I can't see why dd would ever want to open a file without
O_NOCTTY.
Good point; it's just a confusion for the user. Here's a patch to cause dd to always use O_NOCTTY. If someone ever needs it the other way (not likely) I suppose we can add a "ctty" flag.
2004-04-08 Paul Eggert <[email protected]>
* NEWS: Remove noctty flag from dd. Suggested by Philippe Troin.
* doc/coreutils.texi (dd invocation): Likewise.
* src/shred.c (O_NOCTTY): Remove redundant decl.
* src/dd.c (flags, usage): Remove noctty flag. (main): Always use O_NOCTTY when opening files.
Index: NEWS
===================================================================
RCS file: /home/meyering/coreutils/cu/NEWS,v
retrieving revision 1.199
NEWS +0 -1
@@ -24,7 +24,6 @@ GNU coreutils NEWS sync likewise, but also for metadata nonblock use non-blocking I/O nofollow do not follow symlinks - noctty do not assign controlling terminal from file stty now provides support (iutf8) for setting UTF-8 input mode.
Index: doc/coreutils.texi
===================================================================
RCS file: /home/meyering/coreutils/cu/doc/coreutils.texi,v retrieving revision 1.176
coreutils.texi +2 -8
@@ -6666,18 +6666,12 @@ Use non-blocking I/O. @cindex symbolic links, following Do not follow symbolic links. -@item noctty -@opindex noctty -@cindex controlling terminal -Do not assign the file to be a controlling terminal for @command{dd}. -This has no effect when the file is not a terminal. - @end table These flags are not supported on all systems, and @samp{dd} rejects attempts to use them when they are not supported. When reading from -standard input or writing to standard output, the @samp{nofollow} and -@samp{noctty} flags should not be specified, and the other flags +standard input or writing to standard output, the @samp{nofollow} flag +should not be specified, and the other flags (e.g., @samp{nonblock}) can affect how other processes behave with the affected file descriptors, even after @command{dd} exits.
Index: src/shred.c
===================================================================
RCS file: /home/meyering/coreutils/cu/src/shred.c,v retrieving revision 1.84
shred.c +0 -4
@@ -110,10 +110,6 @@#include "quotearg.h" /* For quotearg_colon */ #include "quote.h" /* For quotearg_colon */ -#ifndef O_NOCTTY -# define O_NOCTTY 0/* This is a very optional frill */ -#endif -#define DEFAULT_PASSES 25/* Default */ /* How many seconds to wait before checking whether to output another
dd.c renamed from dd.c-bak +3 -6
@@ -192,7 +192,6 @@ static struct symbol_value const flags[] {"append", O_APPEND}, {"direct", O_DIRECT}, {"dsync", O_DSYNC}, - {"noctty", O_NOCTTY}, {"nofollow", O_NOFOLLOW}, {"nonblock", O_NONBLOCK}, {"sync", O_SYNC},
@@ -384,9 +383,6 @@ Each FLAG symbol may be:\n\fputs (_ (" nonblock use non-blocking I/O\n" ), stdout);if (O_NOFOLLOW )fputs (_ (" nofollow do not follow symlinks\n" ), stdout); -if (O_NOCTTY ) -fputs (_ (" noctty do not assign controlling terminal from file\n" ), - stdout); fputs (_("\ \n\ Note that sending a SIGUSR1 signal to a running `dd' process makes it\n\
@@ -1300,7 +1296,8 @@ main (int argc, char **argv) }else { -if (open_fd (STDIN_FILENO , input_file,O_RDONLY | input_flags,0 ) <0 ) +int opts = input_flags |O_NOCTTY ; +if (open_fd (STDIN_FILENO , input_file,O_RDONLY | opts,0 ) <0 )error (EXIT_FAILURE , errno,_ ("opening %s" ),quote (input_file)); }
@@ -1313,7 +1310,7 @@ main (int argc, char **argv) {mode_t perms =S_IRUSR |S_IWUSR |S_IRGRP |S_IWGRP |S_IROTH |S_IWOTH ;int opts - = (output_flags + = (output_flags |O_NOCTTY | (conversions_mask &C_NOCREAT ?0 :O_CREAT ) | (conversions_mask &C_EXCL ?O_EXCL :0 ) | (seek_records || (conversions_mask &C_NOTRUNC ) ?0 :O_TRUNC ));
Jim Meyering
AppliedRe: dd patch to remove noctty
Quoted 9 lines · Paul Eggert
Quoted 2 lines · Philippe Troin
noctty definitely seems overkill... I can't see why dd would ever want to open a file without
O_NOCTTY.Good point; it's just a confusion for the user. Here's a patch to cause dd to always use
O_NOCTTY. If someone ever needs it the other way (not likely) I suppose we can add a "ctty" flag.
Quoted 7 lines
2004-04-08 Paul Eggert <[email protected]>
* NEWS: Remove noctty flag from dd. Suggested by Philippe Troin.
* doc/coreutils.texi (dd invocation): Likewise.
*src/shred.c(O_NOCTTY): Remove redundant decl.
*src/dd.c(flags, usage): Remove noctty flag. (main): Always useO_NOCTTYwhen opening files.
Applied. Thanks!
Hi,
Quoted 2 lines
OK, I can see that one. But it seems like a pretty small benefit to me -- CPU utilization is already really low.
Maybe not to you but it does make a big difference on our 500 disk setup. At the moment we use dd to do an initial sniff, then ext3 utils to do O_DIRECT reads/writes. With O_DIRECT read/write in dd we could use it instead. (We are basically interested in IO performance that a database would see).
Quoted 6 lines
Um, that sounds like a bad idea to me. It seems to me it's the kernel's responsibility to figure out "hey, looks like a streaming read - let's not blow out the buffer cache trying to hold 20GB on a 512M system." If you're saying that the kernel guys have given up and the established wisdom is now "you gotta use
O_DIRECTif you don't want to throw everything else out due to streaming data", well... I'm disappointed.
When you start hitting memory bandwidth limits, O_DIRECT will help you. Sure it wont be an issue for your dd copy scenario, but I wanted to point out there are other valid uses for it.
Anton
philip copeland did a whole set of patches for coreutils to allow directio for both read write and mixed sizes even
the rpm is at
http://oss.oracle.com/projects/ocfs/files/source/RHAT/RHAS3/coreutils-4.5.3-33.src.rpm,
I think he took it up with the maintainers but so far had no luck
it makes sense to have this stuff, we use it a lot.
Wim
Quoted 29 lines · Anton Blanchard
Hi,
Quoted 2 lines
OK, I can see that one. But it seems like a pretty small benefit to me -- CPU utilization is already really low.
Maybe not to you but it does make a big difference on our 500 disk setup. At the moment we use dd to do an initial sniff, then ext3 utils to do
O_DIRECTreads/writes. WithO_DIRECTread/write in dd we could use it instead. (We are basically interested in IO performance that a database would see).Quoted 6 lines
Um, that sounds like a bad idea to me. It seems to me it's the kernel's responsibility to figure out "hey, looks like a streaming read - let's not blow out the buffer cache trying to hold 20GB on a 512M system." If you're saying that the kernel guys have given up and the established wisdom is now "you gotta use
O_DIRECTif you don't want to throw everything else out due to streaming data", well... I'm disappointed.When you start hitting memory bandwidth limits,
O_DIRECTwill help you. Sure it wont be an issue for your dd copy scenario, but I wanted to point out there are other valid uses for it.Anton
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines · Wim Coekaerts
philip copeland did a whole set of patches for coreutils to allow directio for both read write and mixed sizes even
the rpm is at
http://oss.oracle.com/projects/ocfs/files/source/RHAT/RHAS3/coreutils-4.5.3-33.src.rpm,
Thanks for the pointer.
FYI, that URL didn't work for me. This one did:
http://oss.oracle.com/projects/ocfs/dist/files/source/RHAT/RHAS3/coreutils-4.5.3-33.src.rpm
Whoever maintains that code should consider merging their changes with something newer. There are over 500 lines of NEWS entries alone for the coreutils releases that have been made since 4.5.3.
Besides, those patches have portability and robustness problems, and they aren't even based on coreutils-4.5.3. Maybe they're based on some vendor's version of coreutils?
Quoted 1 line
I think he took it up with the maintainers but so far had no luck
As far as I know, no one ever submitted such O_DIRECT-based patches to me or any of the bug-*@gnu.org mailing lists.
The only pending O_DIRECT-based patch is one for shred.
Clean, complete, and well-justified patches usually go in pretty quickly.
Quoted 2 lines · Andrew Morton
In 2.6 we do the check at
open()andfcntl()time. In 2.4 we don't fail until the actual I/O attempt.
This raises the issue of what "dd conv=direct" should do in 2.4 kernels. I propose that it should report an error and exit, when the write fails, since conv=direct can't be implemented. The basic idea is that on systems that lack direct I/O, conv=direct should fail.
Another issue with this patch: in Solaris, direct I/O is done by invoking directio(DIRECTIO_ON); see
<http://docs.sun.com/db/doc/816-0213/6m6ne37so?q=directio&a=view>.
Is Solaris direct I/O a direct analog to Linux direct I/O, or are there subtle differences in semantics that should be made visible to the users of GNU "dd"?
Quoted 9 lines · Paul Eggert
Quoted 2 lines · Andrew Morton
In 2.6 we do the check at
open()andfcntl()time. In 2.4 we don't fail until the actual I/O attempt.This raises the issue of what "dd conv=direct" should do in 2.4 kernels. I propose that it should report an error and exit, when the write fails,
And when the read fails.
Quoted 2 lines
since conv=direct can't be implemented. The basic idea is that on systems that lack direct I/O, conv=direct should fail.
I think that's best. It's a bit user-unfriendly, but a silent fallback is misleading.
It might be acceptable to print a warning, then fall back.
Quoted 6 lines
Another issue with this patch: in Solaris, direct I/O is done by invoking
directio(DIRECTIO_ON); see
<http://docs.sun.com/db/doc/816-0213/6m6ne37so?q=directio&a=view>.
Is Solaris direct I/O a direct analog to Linux direct I/O, or are there subtle differences in semantics that should be made visible to the users of GNU "dd"?
solaris directio(DIRECTIO_ON) is a hint only. If the system cannot perform the uncached zerocopy then it will fall back to buffered IO. Solaris will perform direct-IO "when the application's buffer is aligned on a two-byte (short) boundary, the offset into the file is on a device sector boundary, and the size of the operation is a multiple of device sectors."
On Linux you set direct-io with open(O_DIRECT) or fcntl(F_SETFL, O_DIRECT). If the filesystem doesn't support direct-io we will fail the open/fcntl attempt up-front in 2.6 only.
If O_DIRECT was successfully set and the IO is not correctly aligned Linux will fail the relevant I/O attempt. Alignment requirements are:
2.4: page aligned on-disk and in-memory
2.6: device sector aligned on-disk and in-memory.
I'd recomment that dd use getpagesize() alignment on-disk and in-memory.
Quoted 8 lines · Paul Eggert
Quoted 2 lines · Andrew Morton
In 2.6 we do the check at
open()andfcntl()time. In 2.4 we don't fail until the actual I/O attempt.This raises the issue of what "dd conv=direct" should do in 2.4 kernels. I propose that it should report an error and exit, when the write fails, since conv=direct can't be implemented. The basic idea is that on systems that lack direct I/O, conv=direct should fail.
I agree, this is the appropriate behavior. When "dd conv=direct" is used on a fd that cannot support direct I/O, and the kernel doesn't tell us until it's too late, erroring on the I/O is the only sane action.
Fortuitously, that's what happens with the patches I posted.
(Note that it's a function of the kernel and the underlying FS whether direct I/O is going to work. So some kind of heuristic on kernel version is a bad idea.)
Quoted 6 lines
Another issue with this patch: in Solaris, direct I/O is done by invoking
directio(DIRECTIO_ON); see
<http://docs.sun.com/db/doc/816-0213/6m6ne37so?q=directio&a=view>.
Is Solaris direct I/O a direct analog to Linux direct I/O, or are there subtle differences in semantics that should be made visible to the users of GNU "dd"?
The Solaris semantics are more sane than the Linux ones, to be sure -- it always does the I/O, falling back to buffered I/O if conditions are not right for direct.
I don't believe I have a test system for the Solaris case (when was directio(3C) added?) so someone else will have to add that support.
-andy
In article <[email protected]>,
Quoted 5 lines · Andy Isaacson
The next feature to add would be OpenBSD-style "KB/s" reporting. I'm not going there.
diff -ur coreutils-5.0.91/doc/coreutils.texi
coreutils-5.0.91-adi/doc/coreutils.texi
Doesn't it already do that ?
$ dd if=/dev/zero of=/tmp/file bs=4K count=100 100+0 records in 100+0 records out 409600 bytes transferred in 0.005108 seconds (80189583 bytes/sec)
$ dpkg -S /bin/dd coreutils: /bin/dd $ dpkg -s coreutils Package: coreutils Version: 5.0.91-2
Mike.
Quoted 8 lines · Miquel van Smoorenburg
In article <[email protected]>,
Quoted 5 lines · Andy Isaacson
The next feature to add would be OpenBSD-style "KB/s" reporting. I'm not going there.
diff -ur coreutils-5.0.91/doc/coreutils.texi
coreutils-5.0.91-adi/doc/coreutils.texi
Quoted 6 lines
Doesn't it already do that ?
$ dd if=/dev/zero of=/tmp/file bs=4K count=100 100+0 records in 100+0 records out 409600 bytes transferred in 0.005108 seconds (80189583 bytes/sec)
hmmm...
debian/patches/05_dd-performance-counter.patch
Not upstream.
-andy
Quoted 1 line · Andy Isaacson
Linux-kernel: is this patch horribly wrong?
...
Quoted 2 lines
to force
O_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.
Adding the functionality to conv= doesn't seem right to me. conv= is for converting the data in some way. This is just changing the way data is written. Right?
Have you looked at lmdd? It supports O_DIRECT and many other things not in the standard dd.
Signature · 3 lines
Nate Straz [email protected] sgi, inc http://www.sgi.com/ Linux Test Project http://ltp.sf.net/
Quoted 9 lines · Nathan Straz
Quoted 1 line · Andy Isaacson
Linux-kernel: is this patch horribly wrong?
...
Quoted 2 lines
to force
O_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.Adding the functionality to conv= doesn't seem right to me. conv= is for converting the data in some way. This is just changing the way data is written. Right?
There's already conv=notrunc which means "open without O_TRUNC". I agree that it's a nonintuitive interface, but then, the entire dd(1) command line is.
Quoted 2 lines
Have you looked at lmdd? It supports
O_DIRECTand many other things not in the standard dd.
Wasn't aware of that one. Thanks for the pointer.
-andy
In article <[email protected]>,
Quoted 7 lines · Andy Isaacson
Quoted 6 lines · Nathan Straz
Quoted 1 line · Andy Isaacson
Linux-kernel: is this patch horribly wrong?
...
Quoted 2 lines
to force
O_DIRECT. The enclosed patch adds a "conv=direct" flag to enable this usage.
Quoted 7 lines
Quoted 3 lines
Adding the functionality to conv= doesn't seem right to me. conv= is for converting the data in some way. This is just changing the way data is written. Right?
There's already conv=notrunc which means "open without
O_TRUNC". I agree that it's a nonintuitive interface, but then, the entire dd(1) command line is.
Well as you already added iflags and oflags, it makes more sense to use those. oflags=fsync makes more sense to me than conv=fsync. Likewise oflags=notrunc.
Mike.