VFS locking: f_pos thread-safe ?
Quoted 15 lines
Quoted 6 lines · Werner Almesberger
"[...]
read( )[...] shall be atomic with respect to each other
in the effects specified in IEEE Std. 1003.1-200x when they operate on regular files. If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them."Whichever thread finishes its read last gets to update
f_pos.I'm struggling a bit to understand what they're calling for there. If thread A enters a read and then shortly afterwards thread B enters the read, does thread B see an
f_poswhich starts out at the beginning of A's read, or the end of it?
Quoted 4 lines
Similar questions apply as the threads exit their read()s.
Either way, there's no way in which we should serialise concurrent readers. That would really suck for sensible apps which are using
pread64().
Surely, we can just serialise read() (and related) calls that modify f_pos? Since pread() doesn't modify f_pos we shouldn't need to serialise those calls no? Also doesn't spec make the same claims about other calls that modify f_pos such as write()?
Matt
Quoted 18 lines · Matt
Quoted 17 lines
Quoted 6 lines · Werner Almesberger
"[...]
read( )[...] shall be atomic with respect to each other
in the effects specified in IEEE Std. 1003.1-200x when they operate on regular files. If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them."Whichever thread finishes its read last gets to update
f_pos.I'm struggling a bit to understand what they're calling for there. If thread A enters a read and then shortly afterwards thread B enters the read, does thread B see an
f_poswhich starts out at the beginning of A's read, or the end of it?Similar questions apply as the threads exit their read()s.
Quoted 7 lines
Quoted 2 lines
Either way, there's no way in which we should serialise concurrent readers. That would really suck for sensible apps which are using
pread64().Surely, we can just serialise
read()(and related) calls that modifyf_pos? Sincepread()doesn't modifyf_poswe shouldn't need to serialise those calls no? Also doesn't spec make the same claims about other calls that modifyf_possuch aswrite()?
We could do somethnig like that.
But is there any application in which two threads simultaneously perform read() against the same fd which is not already buggy?
Quoted 36 lines · Andrew Morton
Quoted 22 lines · Matt
Quoted 22 lines
Quoted 7 lines · Werner Almesberger
"[...]
read( )[...] shall be atomic with respect to each other
in the effects specified in IEEE Std. 1003.1-200x when they operate on regular files. If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them."Whichever thread finishes its read last gets to update
f_pos.I'm struggling a bit to understand what they're calling for there. If thread A enters a read and then shortly afterwards thread B enters the read, does thread B see an
f_poswhich starts out at the beginning of A's read, or the end of it?Similar questions apply as the threads exit their read()s.
Quoted 10 lines
Quoted 4 lines
Either way, there's no way in which we should serialise concurrent readers. That would really suck for sensible apps which are using
pread64().Surely, we can just serialise
read()(and related) calls that modifyf_pos? Sincepread()doesn't modifyf_poswe shouldn't need to serialise those calls no? Also doesn't spec make the same claims about other calls that modifyf_possuch aswrite()?
Quoted 8 lines
We could do somethnig like that.
But is there any application in which two threads simultaneously perform
read()against the same fd which is not already buggy?
touché :) but still we should do what we can.. want me to make a patch?
Matt
Quoted 8 lines · Matt
Quoted 7 lines
But is there any application in which two threads simultaneously perform
read()against the same fd which is not already buggy?touché :) but still we should do what we can.. want me to make a patch?
Not unless we can think of a way in which it actually matters, thanks.
Quoted 4 lines · Andrew Morton
Quoted 1 line
touché :) but still we should do what we can.. want me to make a patch?
Not unless we can think of a way in which it actually matters, thanks.
I've written one of those... a migration program from an old fixed-record-size database to SQL. Somebody had the brilliant idea to let two threads read the file concurrently, postprocess the thing (which took a variable amount of time depending on what was in the record) and then feed it to the database (ditto).
When that idea didn't work, I used a separate reader thread and coordinated buffer usage with semaphores or whatever, making the thing a whole lot more complicated in the process. :-/
So count me as one of those people who think that it does matter -- if N threads read a file of M bytes, they should collectively get M bytes from it (possibly out of order, of course, but that's a different problem). Anything else would be inconsistent.
Same thing goes for writing (in append mode as well as otherwise).
Signature · 1 line
Matthias Urlichs
Quoted 5 lines · Matthias Urlichs
I've written one of those... a migration program from an old fixed-record-size database to SQL. Somebody had the brilliant idea to let two threads read the file concurrently, postprocess the thing (which took a variable amount of time depending on what was in the record) and then feed it to the database (ditto).
"Somebody made a guess about undefined behaviour"
Quoted 3 lines
When that idea didn't work, I used a separate reader thread and coordinated buffer usage with semaphores or whatever, making the thing a whole lot more complicated in the process. :-/
"Guess what? It didn't work".
Quoted 4 lines
So count me as one of those people who think that it does matter -- if N threads read a file of M bytes, they should collectively get M bytes from it (possibly out of order, of course, but that's a different problem). Anything else would be inconsistent.
"Please, make that undefined behaviour conform to the guess made by that somebody. Whaddya mean, 'non-portable' and 'broken application'?!?!?".
Quoted 1 line
Same thing goes for writing (in append mode as well as otherwise).
Look for ->i_sem use in there.
Quoted 1 line · Andrew Morton
Not unless we can think of a way in which it actually matters, thanks.
What I'm struggling with when reading that POSIX draft is to understand whether CLONE_FILES is appropriate or not for pthread_create.
If this is unspecified, the f_pos issue becomes largely academic, although one might argue that read() should behave either one way or the other.
And no, luckily, I don't have a real-life application for this either :-)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 7 lines · Werner Almesberger
Quoted 1 line · Andrew Morton
Not unless we can think of a way in which it actually matters, thanks.
What I'm struggling with when reading that POSIX draft is to understand whether
CLONE_FILESis appropriate or not forpthread_create.
Quoted 6 lines
If this is unspecified, the
f_posissue becomes largely academic, although one might argue thatread()should behave either one way or the other.And no, luckily, I don't have a real-life application for this either :-)
WTF does that have to CLONE_FILES? Whether you share descriptor table or get an independent copy, pointers to struct file are the same.
Quoted 2 lines · [email protected] wrote:
WTF does that have to
CLONE_FILES? Whether you share descriptor table or get an independent copy, pointers to struct file are the same.
Err, right, of course. Sorry, I shouldn't post before the caffeine breakfast ...
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 1 line · viro
"Somebody made a guess about undefined behaviour"
Guess what? The manpage says that read(2) return N bytes and advances the file pointer by N bytes. It doesn't talk, much less caution, about threads.
YOU may immediately know, based on your kernel knowledge or whatever, that things get somewhat undefined when two threads do that at the same time, but it's NOT AT ALL obvious to a "normal" application programmer. There's plenty of system calls that CAN be done concurrently, after all.
So we tried the simple idea, it didn't work, we grumbled a bit ;-) and then did something else instead. Not a big deal.
Please save your "translations" for stupid ideas that are obviously so without in-depth kernel knowledge or equivalent.
Thank you.
Signature · 1 line
Matthias Urlichs
Hi,
Stephen Hemminger:
Quoted 2 lines
Ever hear of pread/pwrite?
Sure I have, but the general idea had been to leave the work entirely to the file system...
Signature · 1 line
Matthias Urlichs | noris network AG | http://smurf.noris.de/
Quoted 1 line · Matthias Urlichs
but it's NOT AT ALL obvious to a "normal" application programmer.
It gets worse. From the same draft (perhaps someone who has the final version could comment ?), in the rationale for read():
| The standard developers considered adding atomicity requirements to a
| pipe or FIFO, but recognized that due to the nature of pipes and FIFOs
| there could be no guarantee of atomicity of reads of {PIPE_BUF} or any
| other size that would be an aid to applications portability.But then
| I/O is intended to be atomic to ordinary files and pipes and FIFOs.
Now, what exactly does "intended" mean ?
Of course, in this part, they only talk about data staying together, not whether it can get duplicated, or effects on f_pos.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
[email protected] (Matthias Urlichs) wrote on 06.02.04 in <[email protected]>:
Quoted 6 lines
Quoted 1 line · viro
"Somebody made a guess about undefined behaviour"
Guess what? The manpage says that read(2) return N bytes and advances the file pointer by N bytes. It doesn't talk, much less caution, about threads.
That's a defect in the man page, then.
POSIX says on <http://www.opengroup.org/onlinepubs/007904975/functions/
read.html>:
[...]
DESCRIPTION
The read() function shall attempt to read nbyte bytes from the file associated with the open file descriptor, fildes, into the buffer pointed to by buf. The behavior of multiple concurrent reads on the same pipe, FIFO, or terminal device is unspecified.
[...]
That's a pretty explicit warning exactly where one would expect it.
If Linux read(2) doesn't say something like that, I'll consider that a serious bug.
OTOH ...
Quoted 4 lines
YOU may immediately know, based on your kernel knowledge or whatever, that things get somewhat undefined when two threads do that at the same time, but it's NOT AT ALL obvious to a "normal" application programmer. There's plenty of system calls that CAN be done concurrently, after all.
... these days, programmers really *should* know enough to consult a free, current, and easily readable version of the relevant standard!
(If you forget where it is, it's easy to find from <http://www.unix-
systems.org/>.)
Quoted 2 lines
Please save your "translations" for stupid ideas that are obviously so without in-depth kernel knowledge or equivalent.
... such as the current case.
MfG Kai
Hi!
Quoted 10 lines
POSIX says on <http://www.opengroup.org/onlinepubs/007904975/functions/
read.html>:[...]
DESCRIPTIONThe
read()function shall attempt to read nbyte bytes from the file associated with the open file descriptor, fildes, into the buffer pointed to by buf. The behavior of multiple concurrent reads on the same pipe, FIFO, or terminal device is unspecified.
...which implies that concurrent reads on regular files are okay :-P.
Pavel
Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms