Update of file offset on write() etc. is non-atomic with I/O
Hello all,
A note from Yongzhi Pan about some of my own code led me to dig deeper and discover behavior that is surprising and also seems to be a fairly clear violation of POSIX requirements.
It appears that write() (and, presumably read() and other similar system calls) are not atomic with respect to performing I/O and updating the file offset behavior.
The problem can be demonstrated using the program below. That program takes three arguments:
$ ./multi_writer num-children num-blocks block-size > somefile
It creates 'num-children' children, each of which writes 'num-blocks' blocks of 'block-size' bytes to standard output; for my experiments, stdout is redirected to a file. After all children have finished, the parent inspects the size of the file written on stdout, calculates the expected size of the file, and displays these two values, and their difference on stderr.
Some observations:
* All children inherit the stdout file descriptor from the parent; thus the FDs refer to the same open file description, and therefore share the file offset.
* When I run this on a multi-CPU BSD systems, I get the expected result:
$ ./multi_writer 10 10000 1000 > g 2> run.log $ ls -l g -rw------- 1 mkerrisk users 100000000 Jan 17 07:34 g
* Someone else tested this code for me on a Solaris system, and also got the expected result.
* On Linux, by contrast, we see behavior such as the following:
$ ./multi_writer 10 10000 1000 > g Expected file size: 100000000 Actual file size: 16323000 Difference: 83677000 $ ls -l g -rw-r--r--. 1 mtk mtk 16323000 Feb 17 16:05 g
Summary of the above output: some children are overwriting the output of other children because output is not atomic with respect to updates to the file offset.
For reference, POSIX.1-2008/SUSv4 Section XSI 2.9.7 says:
[[
2.9.7 Thread Interactions with Regular File Operations
All of the following functions shall be atomic with respect to each other in the effects specified in POSIX.1-2008 when they operate on regular files or symbolic links:
chmod()
...pread()read()
...readv()pwrite()
...write()writev()
If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them. ]]
(POSIX.1-2001 has similar text.)
This text is in one of the Threads sections, but it applies equally to threads in different processes as to threads in the same process.
I've tested the code below on ext4, XFS, and BtrFS, on kernel 3.12 and a number of other recent kernels, all with similar results, which suggests the result is in the VFS layer. (Can it really be so simple as no locking around pieces such as
loff_t pos = file_pos_read(f.file);
ret = vfs_write(f.file, buf, count, &pos);
if (ret >= 0)
file_pos_write(f.file, pos);in fs/read_write.c?)
I discovered this behavior after Yongzhi Pan reported some unexpected behavior in some of my code that forked to create a parent and child that wrote to the same file. In some cases, expected output was not appearing. In other words, after a fork(), and in the absence of any other synchronization technique, a parent and a child cannot safely write to the same file descriptor without risking overwriting each other's output. But POSIX requires this, and other systems seem to guarantee it.
Am I correct to think there's a kernel problem here?
Thanks,
Michael
===
/* multi_writer.c
*/
#include <sys/wait.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/fcntl.h>
#include <sys/stat.h>
#include <string.h>
#include <errno.h>
typedef enum { FALSE, TRUE } Boolean;
#define errExit(msg) do { perror(msg); exit(EXIT_FAILURE); \
} while (0)#define fatal(msg) do { fprintf(stderr, "%s\n", msg); \
exit(EXIT_FAILURE); } while (0)#define usageErr(msg, progName) \
do { fprintf(stderr, "Usage: "); \
fprintf(stderr, msg, progName); \
exit(EXIT_FAILURE); } while (0)intmain(int argc, char *argv[])
{
char *buf;
int j, k, nblocks, nchildren;
size_t blocksize;
struct stat sb;
// int nchanges;
// off_t pos;
long long expected;
if (argc < 4 || strcmp(argv[1], "--help") == 0)
usageErr("%s num-children num-blocks block-size [O_APPEND-flag]\n",
argv[0]);
nblocks = atoi(argv[2]);
blocksize = atoi(argv[3]);
buf = malloc(blocksize + 1);
if (buf == NULL)
errExit("malloc");
/* If a fourth command-line argument is specified, set the O_APPEND
flag on stdout */
if (argc > 4)
if (fcntl(STDOUT_FILENO, F_SETFL, O_APPEND) == -1)
errExit("fcntl-F_SETFL");
nchildren = atoi(argv[1]);
/* Create child processes that write blocks to stdout */
for (j = 0; j < nchildren; j++) {
switch(fork()) {
case -1:
errExit("fork");
case 0: /* Each child writes nblocks * blocksize bytes to stdout */// nchanges = 0;
/* Put something distinctive in each child's buffer (in case
we want to analyze byte sequences in the output) */
for (k = 0; k < blocksize; k++)
buf[k] = 'a' + getpid() % 26;
for (k = 0; k < nblocks; k++) {
// if (k > 0 && pos != lseek(STDOUT_FILENO, 0, SEEK_END))
// nchanges++;
if (write(STDOUT_FILENO, buf, blocksize) != blocksize)
fatal("write");
// pos = lseek(STDOUT_FILENO, 0, SEEK_END);
}// fprintf(stderr, "%ld: nchanges = %d\n",
// (long) getpid(), nchanges);
exit(EXIT_SUCCESS);
default:
break; /* Parent falls through to create next child */
}
}
/* Wait for all children to terminate */
while (wait(NULL) > 0)
continue;
/* Compare final length of file against expected size */
if (fstat(STDOUT_FILENO, &sb) == -1)
errExit("fstat");
expected = blocksize * nblocks * nchildren;
fprintf(stderr, "Expected file size: %10lld\n", expected);
fprintf(stderr, "Actual file size: %10lld\n", (long long) sb.st_size);
fprintf(stderr, "Difference: %10lld\n", expected - sb.st_size);
exit(EXIT_SUCCESS);
}Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
[expanding the CC list a little more to bring in some previously interested parties]
Quoted 210 lines · Michael Kerrisk (man-pages)
Hello all,
A note from Yongzhi Pan about some of my own code led me to dig deeper and discover behavior that is surprising and also seems to be a fairly clear violation of POSIX requirements.
It appears that
write()(and, presumablyread()and other similar system calls) are not atomic with respect to performing I/O and updating the file offset behavior.The problem can be demonstrated using the program below. That program takes three arguments:
$ ./multi_writer num-children num-blocks block-size > somefileIt creates 'num-children' children, each of which writes 'num-blocks' blocks of 'block-size' bytes to standard output; for my experiments, stdout is redirected to a file. After all children have finished, the parent inspects the size of the file written on stdout, calculates the expected size of the file, and displays these two values, and their difference on stderr.
Some observations:
* All children inherit the stdout file descriptor from the parent; thus the FDs refer to the same open file description, and therefore share the file offset.
* When I run this on a multi-CPU BSD systems, I get the expected result:
$ ./multi_writer 10 10000 1000 > g 2> run.log $ ls -l g -rw------- 1 mkerrisk users 100000000 Jan 17 07:34 g* Someone else tested this code for me on a Solaris system, and also got the expected result.
* On Linux, by contrast, we see behavior such as the following:
$ ./multi_writer 10 10000 1000 > g Expected file size: 100000000 Actual file size: 16323000 Difference: 83677000 $ ls -l g -rw-r--r--. 1 mtk mtk 16323000 Feb 17 16:05 gSummary of the above output: some children are overwriting the output of other children because output is not atomic with respect to updates to the file offset.
For reference, POSIX.1-2008/SUSv4 Section XSI 2.9.7 says:
[[
2.9.7 Thread Interactions with Regular File OperationsAll of the following functions shall be atomic with respect to each other in the effects specified in POSIX.1-2008 when they operate on regular files or symbolic links:
chmod()
...pread()read()
...readv()pwrite()
...write()writev()If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them. ]]
(POSIX.1-2001 has similar text.)
This text is in one of the Threads sections, but it applies equally to threads in different processes as to threads in the same process.
I've tested the code below on ext4, XFS, and BtrFS, on kernel 3.12 and a number of other recent kernels, all with similar results, which suggests the result is in the VFS layer. (Can it really be so simple as no locking around pieces such as
loff_t pos = file_pos_read(f.file); ret = vfs_write(f.file, buf, count, &pos); if (ret >= 0) file_pos_write(f.file, pos);in
fs/read_write.c?)I discovered this behavior after Yongzhi Pan reported some unexpected behavior in some of my code that forked to create a parent and child that wrote to the same file. In some cases, expected output was not appearing. In other words, after a
fork(), and in the absence of any other synchronization technique, a parent and a child cannot safely write to the same file descriptor without risking overwriting each other's output. But POSIX requires this, and other systems seem to guarantee it.Am I correct to think there's a kernel problem here?
Thanks,
Michael
===
/*
multi_writer.c
*/#include <
sys/wait.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/fcntl.h>
#include <sys/stat.h>
#include <string.h>
#include <errno.h>typedef enum { FALSE, TRUE } Boolean;
#define errExit(msg) do { perror(msg); exit(EXIT_FAILURE); \ } while (0)#define fatal(msg) do { fprintf(stderr, "%s\n", msg); \ exit(EXIT_FAILURE); } while (0)#define usageErr(msg, progName) \ do { fprintf(stderr, "Usage: "); \ fprintf(stderr, msg, progName); \ exit(EXIT_FAILURE); } while (0)int
main(int argc, char *argv[]){ char *buf; int j, k, nblocks, nchildren; size_t blocksize; struct stat sb; // int nchanges; // off_t pos; long long expected; if (argc < 4 || strcmp(argv[1], "--help") == 0) usageErr("%s num-children num-blocks block-size [O_APPEND-flag]\n", argv[0]); nblocks = atoi(argv[2]); blocksize = atoi(argv[3]); buf = malloc(blocksize + 1); if (buf == NULL) errExit("malloc"); /* If a fourth command-line argument is specified, set the O_APPEND flag on stdout */ if (argc > 4) if (fcntl(STDOUT_FILENO, F_SETFL, O_APPEND) == -1) errExit("fcntl-F_SETFL"); nchildren = atoi(argv[1]); /* Create child processes that write blocks to stdout */ for (j = 0; j < nchildren; j++) { switch(fork()) { case -1: errExit("fork"); case 0: /* Each child writes nblocks * blocksize bytes to stdout */// nchanges = 0;
/* Put something distinctive in each child's buffer (in case we want to analyze byte sequences in the output) */ for (k = 0; k < blocksize; k++) buf[k] = 'a' + getpid() % 26; for (k = 0; k < nblocks; k++) { // if (k > 0 && pos != lseek(STDOUT_FILENO, 0, SEEK_END)) // nchanges++; if (write(STDOUT_FILENO, buf, blocksize) != blocksize) fatal("write"); // pos = lseek(STDOUT_FILENO, 0, SEEK_END); }// fprintf(stderr, "%ld: nchanges = %d\n",
// (long) getpid(), nchanges); exit(EXIT_SUCCESS); default: break; /* Parent falls through to create next child */ } } /* Wait for all children to terminate */ while (wait(NULL) > 0) continue; /* Compare final length of file against expected size */ if (fstat(STDOUT_FILENO, &sb) == -1) errExit("fstat");
Quoted 8 lines
expected = blocksize * nblocks * nchildren;
fprintf(stderr, "Expected file size: %10lld\n", expected);
fprintf(stderr, "Actual file size: %10lld\n", (long long) sb.st_size);
fprintf(stderr, "Difference: %10lld\n", expected - sb.st_size);
exit(EXIT_SUCCESS);
}Offlist, I was pointed to some previous threads on this topic:
http://thread.gmane.org/gmane.linux.kernel.kernelnewbies/43508
http://lwn.net/Articles/180387/
http://lwn.net/Articles/180396/
Notwithstanding the comments of Alan and Linus at
http://thread.gmane.org/gmane.linux.kernel/397980/focus=398248 and
http://thread.gmane.org/gmane.linux.kernel/397980/focus=398281 this *is* a violation of POSIX. When XSI 2.9.7 talks uses the "threads" language, that is to provide the widest possible guarantee--i.e., that the threads within a process also get the same atomicity guarantees as threads in different processes.
I can't comment on the performance implications of adding locking to fix this issue (at least for simultaneous I/O), but there is an argument that it should be done on correctness grounds. Linux isn't conformant with SUSv3 and SUSv4, and isn't consistent with other implementations such as FreeBSD and Solaris. And I'm pretty sure Linux isn't consistent with UNIX since early times. (E.g., page 191 of the 1992 edition of Stevens APUE discusses the sharing of the file offset between the parent and child after fork(). Although Stevens didn't explicitly spell out the atomicity guarantee, the discussion there would be a bit nonsensical without the presumption of that guarantee.)
Thanks,
Michael
PS I tried hacking Linus's untested 2006 patch
(http://lwn.net/Articles/180396/) into the kernel, but perhaps not surprisingly given the age of that patch, I ran into various errors after boot that meant that user-space didn't come up.
Yes, I do think we violate POSIX here because of how we handle f_pos (the earlier thread from 2006 you point to talks about the "thread safe" part, the point here about the actual wording of "atomic" is a separate issue).
Long long ago we used to just pass in the pointer to f_pos directly, and then the low-level write would update it all under the inode semaphore, and all was good.
And then we ended up having tons of problems with non-regular files and drivers accessing f_pos and having nasty races with it because they did *not* have any locking (and very fundamentally didn't want any), and we broke the serialization of f_pos. We still do the *IO* atomically, but yes, the f_pos access itself is outside the lock.
Ho humm.. Al, any ideas of how to fix this?
Linus
Quoted 228 lines · Michael Kerrisk (man-pages)
Hello all,
A note from Yongzhi Pan about some of my own code led me to dig deeper and discover behavior that is surprising and also seems to be a fairly clear violation of POSIX requirements.
It appears that
write()(and, presumablyread()and other similar system calls) are not atomic with respect to performing I/O and updating the file offset behavior.The problem can be demonstrated using the program below. That program takes three arguments:
$ ./multi_writer num-children num-blocks block-size > somefileIt creates 'num-children' children, each of which writes 'num-blocks' blocks of 'block-size' bytes to standard output; for my experiments, stdout is redirected to a file. After all children have finished, the parent inspects the size of the file written on stdout, calculates the expected size of the file, and displays these two values, and their difference on stderr.
Some observations:
* All children inherit the stdout file descriptor from the parent; thus the FDs refer to the same open file description, and therefore share the file offset.
* When I run this on a multi-CPU BSD systems, I get the expected result:
$ ./multi_writer 10 10000 1000 > g 2> run.log $ ls -l g -rw------- 1 mkerrisk users 100000000 Jan 17 07:34 g* Someone else tested this code for me on a Solaris system, and also got the expected result.
* On Linux, by contrast, we see behavior such as the following:
$ ./multi_writer 10 10000 1000 > g Expected file size: 100000000 Actual file size: 16323000 Difference: 83677000 $ ls -l g -rw-r--r--. 1 mtk mtk 16323000 Feb 17 16:05 gSummary of the above output: some children are overwriting the output of other children because output is not atomic with respect to updates to the file offset.
For reference, POSIX.1-2008/SUSv4 Section XSI 2.9.7 says:
[[
2.9.7 Thread Interactions with Regular File OperationsAll of the following functions shall be atomic with respect to each other in the effects specified in POSIX.1-2008 when they operate on regular files or symbolic links:
chmod()
...pread()read()
...readv()pwrite()
...write()writev()If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them. ]]
(POSIX.1-2001 has similar text.)
This text is in one of the Threads sections, but it applies equally to threads in different processes as to threads in the same process.
I've tested the code below on ext4, XFS, and BtrFS, on kernel 3.12 and a number of other recent kernels, all with similar results, which suggests the result is in the VFS layer. (Can it really be so simple as no locking around pieces such as
loff_t pos = file_pos_read(f.file); ret = vfs_write(f.file, buf, count, &pos); if (ret >= 0) file_pos_write(f.file, pos);in
fs/read_write.c?)I discovered this behavior after Yongzhi Pan reported some unexpected behavior in some of my code that forked to create a parent and child that wrote to the same file. In some cases, expected output was not appearing. In other words, after a
fork(), and in the absence of any other synchronization technique, a parent and a child cannot safely write to the same file descriptor without risking overwriting each other's output. But POSIX requires this, and other systems seem to guarantee it.Am I correct to think there's a kernel problem here?
Thanks,
Michael
===
/*
multi_writer.c
*/#include <
sys/wait.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/fcntl.h>
#include <sys/stat.h>
#include <string.h>
#include <errno.h>typedef enum { FALSE, TRUE } Boolean;
#define errExit(msg) do { perror(msg); exit(EXIT_FAILURE); \ } while (0)#define fatal(msg) do { fprintf(stderr, "%s\n", msg); \ exit(EXIT_FAILURE); } while (0)#define usageErr(msg, progName) \ do { fprintf(stderr, "Usage: "); \ fprintf(stderr, msg, progName); \ exit(EXIT_FAILURE); } while (0)int
main(int argc, char *argv[]){ char *buf; int j, k, nblocks, nchildren; size_t blocksize; struct stat sb; // int nchanges; // off_t pos; long long expected; if (argc < 4 || strcmp(argv[1], "--help") == 0) usageErr("%s num-children num-blocks block-size [O_APPEND-flag]\n", argv[0]); nblocks = atoi(argv[2]); blocksize = atoi(argv[3]); buf = malloc(blocksize + 1); if (buf == NULL) errExit("malloc"); /* If a fourth command-line argument is specified, set the O_APPEND flag on stdout */ if (argc > 4) if (fcntl(STDOUT_FILENO, F_SETFL, O_APPEND) == -1) errExit("fcntl-F_SETFL"); nchildren = atoi(argv[1]); /* Create child processes that write blocks to stdout */ for (j = 0; j < nchildren; j++) { switch(fork()) { case -1: errExit("fork"); case 0: /* Each child writes nblocks * blocksize bytes to stdout */// nchanges = 0;
/* Put something distinctive in each child's buffer (in case we want to analyze byte sequences in the output) */ for (k = 0; k < blocksize; k++) buf[k] = 'a' + getpid() % 26; for (k = 0; k < nblocks; k++) { // if (k > 0 && pos != lseek(STDOUT_FILENO, 0, SEEK_END)) // nchanges++; if (write(STDOUT_FILENO, buf, blocksize) != blocksize) fatal("write"); // pos = lseek(STDOUT_FILENO, 0, SEEK_END); }// fprintf(stderr, "%ld: nchanges = %d\n",
// (long) getpid(), nchanges); exit(EXIT_SUCCESS); default: break; /* Parent falls through to create next child */ } } /* Wait for all children to terminate */ while (wait(NULL) > 0) continue; /* Compare final length of file against expected size */ if (fstat(STDOUT_FILENO, &sb) == -1) errExit("fstat"); expected = blocksize * nblocks * nchildren; fprintf(stderr, "Expected file size: %10lld\n", expected); fprintf(stderr, "Actual file size: %10lld\n", (long long) sb.st_size); fprintf(stderr, "Difference: %10lld\n", expected - sb.st_size); exit(EXIT_SUCCESS); }--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Ok, sorry for the long delay, I was distracted (and hoping that Al would come up with a patch).
Anyway, attached is the patch I think we should do for this issue. It is fairly simple:
- it adds a "f_pos_mutex" to the "struct file".
- it adds a new FMODE_ATOMIC_POS flag to the file mode flags to mark
things that need atomic f_pos updates
- it makes the "struct fd" flags be two flags rather than one: the
second flag is for "unlock f_pos_mutex when done"
- it introduces "fd[get,put]_pos()" which gets the f_pos_mutex when required
- it makes read/write/lseek use that.
It's pretty damn straightforward, I think, and is minimally serializing.
Al, comments? Yongzhi Pan, this is pretty much untested, but it's pretty simple and it does fix your test-case.
Linus
Quoted 9 lines · Linus Torvalds
Yes, I do think we violate POSIX here because of how we handle
f_pos(the earlier thread from 2006 you point to talks about the "thread safe" part, the point here about the actual wording of "atomic" is a separate issue).Long long ago we used to just pass in the pointer to
f_posdirectly, and then the low-level write would update it all under the inode semaphore, and all was good.
Quoted 7 lines
And then we ended up having tons of problems with non-regular files and drivers accessing
f_posand having nasty races with it because they did *not* have any locking (and very fundamentally didn't want any), and we broke the serialization off_pos. We still do the *IO* atomically, but yes, thef_posaccess itself is outside the lock.Ho humm.. Al, any ideas of how to fix this?
fs/file_table.c | 1 + fs/open.c | 4 ++++ fs/read_write.c | 46 ++++++++++++++++++++++++++++++++++++---------- include/linux/file.h | 6 +++--- include/linux/fs.h | 6 +++++- 5 files changed, 49 insertions(+), 14 deletions(-)
fs/file_table.c +1 -0
@@ -135,6 +135,7 @@ struct file *get_empty_filp(void)atomic_long_set (&f->f_count,1 );rwlock_init (&f->f_owner.lock);spin_lock_init (&f->f_lock); +mutex_init (&f->f_pos_lock);eventpoll_init_file (f);/* f->f_version: 0 */ return f;
fs/open.c +4 -0
@@ -705,6 +705,10 @@ static int do_dentry_open(struct file *f,return 0 ; } +/* POSIX.1-2008/SUSv4 Section XSI 2.9.7 */ +if (S_ISREG (inode->i_mode)) + f->f_mode |=FMODE_ATOMIC_POS ; + f->f_op =fops_get (inode->i_fop);if (unlikely (WARN_ON (!f->f_op))) { error = -ENODEV ;
fs/read_write.c +36 -10
109 lines not on this page · raw message
include/linux/file.h +3 -3
@@ -28,7 +28,7 @@ static inline void fput_light(struct file *file, int fput_needed)struct fd {struct file *file; -int need_put; +unsigned need_put:1 , need_pos_unlock:1 ; };static inline void fdput (struct fd fd)
@@ -44,7 +44,7 @@ static inline struct fd fdget(unsigned int fd) {int b;struct file *f =fget_light (fd, &b); -return (struct fd ){f,b}; +return (struct fd ){f,b,0 }; }extern struct file *fget_raw (unsigned int fd);
@@ -54,7 +54,7 @@ static inline struct fd fdget_raw(unsigned int fd) {int b;struct file *f =fget_raw_light (fd, &b); -return (struct fd ){f,b}; +return (struct fd ){f,b,0 }; }extern int f_dupfd (unsigned int from,struct file *file,unsigned flags);
include/linux/fs.h +5 -1
@@ -123,6 +123,9 @@ typedef void (dio_iodone_t)(struct kiocb *iocb, loff_t offset,/* File is opened with O_PATH; almost nothing can be done with it */ #define FMODE_PATH ((__force fmode_t)0x4000) +/* File needs atomic accesses to f_pos */ +#define FMODE_ATOMIC_POS ((__force fmode_t)0x8000) +/* File was opened by fanotify and shouldn't generate fanotify events */ #define FMODE_NONOTIFY ((__force fmode_t)0x1000000)
@@ -780,13 +783,14 @@ struct file {const struct file_operations *f_op;/* -* Protects f_ep_links, f_flags, f_pos vs i_size in lseek SEEK_CUR. +* Protects f_ep_links, f_flags. * Must not be taken from IRQ context. */ spinlock_t f_lock;atomic_long_t f_count;unsigned int f_flags;fmode_t f_mode; +struct mutex f_pos_lock;loff_t f_pos;struct fown_struct f_owner;const struct cred *f_cred;
Al Viro
maintainerfs/read_write.c loff_t vfs_llseek(struct file *file, loff_t offset, int whence) · line 278
Quoted 7 lines · Linus Torvalds
fs/read_write.c +7
-0+static struct fd fdget_pos (int fd) +{ +struct fd f =fdget (fd); +struct file *file = f.file; + +if (file && (file->f_mode &FMODE_ATOMIC_POS )) { +if (f.need_put ||file_count (file) >1 ) {
Um... That's odd - we *could* get there with f.need_put and file_count(file) equal to 1, but why would we want to take f_pos_lock in that case?
And it looks like you've missed compat counterparts. Other than that, it'd probably work... Let me check if there are other paths we might need to take care of, though.
PS: apologies for being pretty much MIA for the last two weeks; I'd been ears-deep in interesting scalability crap around vfsmount handling ;-/ Will post the resulting patches for review shortly...
Quoted 4 lines · Al Viro
Um... That's odd - we *could* get there with f.need_put and
file_count(file)equal to 1, but why would we want to takef_pos_lockin that case?
Because that means that the file table is shared among threads. So another thread can access the struct file pointer and do a concurrent read() or write() on it, and so we need to lock f_pos.
Basically, there are two cases:
- duplicated file pointers due to fork(). That's the "file_count()" test.
Yes, this will trigger even if they didn't fork, just dup'ed the
file descriptor. We have no way of telling the difference, though.
- concurrent access due to duplicated "struct file_table" pointers.
This is the "need_put" test, since __fget_light() will have tested the proper files->count already.
Both need f_pos_lock.
Linus
Al Viro
maintainerQuoted 9 lines · Linus Torvalds
Quoted 4 lines · Al Viro
Um... That's odd - we *could* get there with f.need_put and
file_count(file)equal to 1, but why would we want to takef_pos_lockin that case?Because that means that the file table is shared among threads. So another thread can access the struct file pointer and do a concurrent
read()orwrite()on it, and so we need to lockf_pos.
Yes, but in that case fdget() has grabbed a reference to that sucker, so the only way to end with refcount 1 is to have the damn thing gone from descriptor table in between. And AFAICS in that case we are just fine without f_pos_lock.
Quoted 3 lines
- concurrent access due to duplicated "struct
file_table" pointers.
This is the "need_put" test, since__fget_light()will have tested the properfiles->countalready.
Shared descriptor table means that we'd better have a reference grabbed already...
I agree that file_count(file) > 1 for any reason requires locking it; it's just that need_put means exactly that we have grabbed a reference ourselves. So having need_put && file_count <= 1 means that we have grabbed it and then somebody dropped all other references. Including the ones in descriptor table(s), shared or not, etc. In that case they'd better not touch that struct file anymore - after all, once we are done with whatever we are doing, we'll do fput() and it will be ripped from under them with no warning.
Quoted 5 lines · Al Viro
Yes, but in that case
fdget()has grabbed a reference to that sucker, so the only way to end with refcount 1 is to have the damn thing gone from descriptor table in between. And AFAICS in that case we are just fine withoutf_pos_lock.
Ahh. Yes, you're right, just checking the file_count() is sufficient, because as you say, the threaded case will have incremented it for the case we care about./
Linus
Quoted 2 lines · Al Viro
And it looks like you've missed compat counterparts.
Grr. Yes, I missed them for compat_readv/writev - even though I grepped for various users, but I grepped for all the *other* cases (ie vfs_{read|write|llseek}). That means that the locker helper can't be static to fs/read_write.c. Oh well.
Linus
Al Viro
maintainerQuoted 8 lines · Linus Torvalds
Quoted 2 lines · Al Viro
And it looks like you've missed compat counterparts.
Grr. Yes, I missed them for compat_readv/writev - even though I grepped for various users, but I grepped for all the *other* cases (ie vfs_{read|write|llseek}). That means that the locker helper can't be static to
fs/read_write.c. Oh well.
It can - compat read/write/read/writev are also there these days. I think that's the only thing you've missed, but I've several more places to trace before I'll be sure.
Quoted 2 lines · Al Viro
It can - compat read/write/read/writev are also there these days.
There is no "compat_read/write()" - or I would have noticed it when grepping. Regular read/write is fine for the compat case.
It's only readv/writev that have compat versions, because the vectors need to be converted,
Linus
Quoted 14 lines · Linus Torvalds
Ok, sorry for the long delay, I was distracted (and hoping that Al would come up with a patch).
Anyway, attached is the patch I think we should do for this issue. It is fairly simple:
- it adds a "
f_pos_mutex" to the "struct file".- it adds a new
FMODE_ATOMIC_POSflag to the file mode flags to mark
things that need atomicf_posupdates- it makes the "struct fd" flags be two flags rather than one: the
second flag is for "unlockf_pos_mutexwhen done"
Quoted 8 lines
- it introduces "fd[get,put]_pos()" which gets the
f_pos_mutexwhen required- it makes read/write/lseek use that.
It's pretty damn straightforward, I think, and is minimally serializing.
Al, comments? Yongzhi Pan, this is pretty much untested, but it's pretty simple and it does fix your test-case.
The original code (where Yongzhi Pan reported an issue) and the multi_writer.c test code where both mine actually. Anyway, I applied the patch to 3.14-rc5, and I (not withstanding the other points raised by Al about the patch) I confirm that this patch makes the problem that I'm seeing go away.
Thanks for looking at this, Linus.
Cheers,
Michael
Signature · 3 lines
Michael Kerrisk Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/ Linux/UNIX System Programming Training: http://man7.org/training/
2014-03-06 23:03 GMT+08:00 Michael Kerrisk (man-pages) <[email protected]>:
Quoted 7 lines
The original code (where Yongzhi Pan reported an issue) and the
multi_writer.ctest code where both mine actually. Anyway, I applied the patch to 3.14-rc5, and I (not withstanding the other points raised by Al about the patch) I confirm that this patch makes the problem that I'm seeing go away.
Quoted 5 lines
Thanks for looking at this, Linus.
Cheers,
Michael
I tested the original patch on 3.13.5, the multi_writer.c test code and the code in your book both run flawlessly.
Best Regards,
Yongzhi Pan