Skip to contentSkip to conversation

VFS locking: f_pos thread-safe ?

DISCUSSION Matt 7 participants 14 messages Feb 6, 2004 – Feb 23, 2004 1 message not in the archive

lore
Share & tools
lore

Matt

author
#0 lore raw
Quoted 15 lines
Quoted 6 lines · Werner Almesberger

"[...] read( ) [...] shall be atomic with respect to each other
in the effects specified in IEEE Std. 1003.1-200x when they operate on regular files. If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them."

Whichever thread finishes its read last gets to update f_pos.

I'm struggling a bit to understand what they're calling for there. If thread A enters a read and then shortly afterwards thread B enters the read, does thread B see an f_pos which starts out at the beginning of A's read, or the end of it?

Quoted 4 lines

Similar questions apply as the threads exit their read()s.

Either way, there's no way in which we should serialise concurrent readers. That would really suck for sensible apps which are using pread64().

Surely, we can just serialise read() (and related) calls that modify f_pos? Since pread() doesn't modify f_pos we shouldn't need to serialise those calls no? Also doesn't spec make the same claims about other calls that modify f_pos such as write()?

Matt

#1 lore raw
Quoted 18 lines · Matt
Quoted 17 lines
Quoted 6 lines · Werner Almesberger

"[...] read( ) [...] shall be atomic with respect to each other
in the effects specified in IEEE Std. 1003.1-200x when they operate on regular files. If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them."

Whichever thread finishes its read last gets to update f_pos.

I'm struggling a bit to understand what they're calling for there. If thread A enters a read and then shortly afterwards thread B enters the read, does thread B see an f_pos which starts out at the beginning of A's read, or the end of it?

Similar questions apply as the threads exit their read()s.

Quoted 7 lines
Quoted 2 lines

Either way, there's no way in which we should serialise concurrent readers. That would really suck for sensible apps which are using pread64().

Surely, we can just serialise read() (and related) calls that modify f_pos? Since pread() doesn't modify f_pos we shouldn't need to serialise those calls no? Also doesn't spec make the same claims about other calls that modify f_pos such as write()?

We could do somethnig like that.

But is there any application in which two threads simultaneously perform read() against the same fd which is not already buggy?

author
#2 lore raw
Quoted 36 lines · Andrew Morton
Quoted 22 lines · Matt
Quoted 22 lines
Quoted 7 lines · Werner Almesberger

"[...] read( ) [...] shall be atomic with respect to each other
in the effects specified in IEEE Std. 1003.1-200x when they operate on regular files. If two threads each call one of these functions, each call shall either see all of the specified effects of the other call, or none of them."

Whichever thread finishes its read last gets to update f_pos.

I'm struggling a bit to understand what they're calling for there. If thread A enters a read and then shortly afterwards thread B enters the read, does thread B see an f_pos which starts out at the beginning of A's read, or the end of it?

Similar questions apply as the threads exit their read()s.

Quoted 10 lines
Quoted 4 lines

Either way, there's no way in which we should serialise concurrent readers. That would really suck for sensible apps which are using pread64().

Surely, we can just serialise read() (and related) calls that modify f_pos? Since pread() doesn't modify f_pos we shouldn't need to serialise those calls no? Also doesn't spec make the same claims about other calls that modify f_pos such as write()?

Quoted 8 lines

We could do somethnig like that.

But is there any application in which two threads simultaneously perform read() against the same fd which is not already buggy?

touché :) but still we should do what we can.. want me to make a patch?

Matt

#3 lore raw
Quoted 8 lines · Matt
Quoted 7 lines

But is there any application in which two threads simultaneously perform read() against the same fd which is not already buggy?

touché :) but still we should do what we can.. want me to make a patch?

Not unless we can think of a way in which it actually matters, thanks.

#4 lore raw
Quoted 4 lines · Andrew Morton
Quoted 1 line

touché :) but still we should do what we can.. want me to make a patch?

Not unless we can think of a way in which it actually matters, thanks.

I've written one of those... a migration program from an old fixed-record-size database to SQL. Somebody had the brilliant idea to let two threads read the file concurrently, postprocess the thing (which took a variable amount of time depending on what was in the record) and then feed it to the database (ditto).

When that idea didn't work, I used a separate reader thread and coordinated buffer usage with semaphores or whatever, making the thing a whole lot more complicated in the process. :-/

So count me as one of those people who think that it does matter -- if N threads read a file of M bytes, they should collectively get M bytes from it (possibly out of order, of course, but that's a different problem). Anything else would be inconsistent.

Same thing goes for writing (in append mode as well as otherwise).

Signature · 1 line
Matthias Urlichs
#5 lore raw
Quoted 5 lines · Matthias Urlichs

I've written one of those... a migration program from an old fixed-record-size database to SQL. Somebody had the brilliant idea to let two threads read the file concurrently, postprocess the thing (which took a variable amount of time depending on what was in the record) and then feed it to the database (ditto).

"Somebody made a guess about undefined behaviour"

Quoted 3 lines

When that idea didn't work, I used a separate reader thread and coordinated buffer usage with semaphores or whatever, making the thing a whole lot more complicated in the process. :-/

"Guess what? It didn't work".

Quoted 4 lines

So count me as one of those people who think that it does matter -- if N threads read a file of M bytes, they should collectively get M bytes from it (possibly out of order, of course, but that's a different problem). Anything else would be inconsistent.

"Please, make that undefined behaviour conform to the guess made by that somebody. Whaddya mean, 'non-portable' and 'broken application'?!?!?".

Quoted 1 line

Same thing goes for writing (in append mode as well as otherwise).

Look for ->i_sem use in there.

#12 lore raw
In reply to Andrew Morton
Quoted 1 line · Andrew Morton

Not unless we can think of a way in which it actually matters, thanks.

What I'm struggling with when reading that POSIX draft is to understand whether CLONE_FILES is appropriate or not for pthread_create.

If this is unspecified, the f_pos issue becomes largely academic, although one might argue that read() should behave either one way or the other.

And no, luckily, I don't have a real-life application for this either :-)

- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/
#13 lore raw
Quoted 7 lines · Werner Almesberger
Quoted 1 line · Andrew Morton

Not unless we can think of a way in which it actually matters, thanks.

What I'm struggling with when reading that POSIX draft is to understand whether CLONE_FILES is appropriate or not for pthread_create.

Quoted 6 lines

If this is unspecified, the f_pos issue becomes largely academic, although one might argue that read() should behave either one way or the other.

And no, luckily, I don't have a real-life application for this either :-)

WTF does that have to CLONE_FILES? Whether you share descriptor table or get an independent copy, pointers to struct file are the same.

#14 lore raw
Quoted 2 lines · [email protected] wrote:

WTF does that have to CLONE_FILES? Whether you share descriptor table or get an independent copy, pointers to struct file are the same.

Err, right, of course. Sorry, I shouldn't post before the caffeine breakfast ...

- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/
#6 lore raw
In reply to [email protected]
Quoted 1 line · viro

"Somebody made a guess about undefined behaviour"

Guess what? The manpage says that read(2) return N bytes and advances the file pointer by N bytes. It doesn't talk, much less caution, about threads.

YOU may immediately know, based on your kernel knowledge or whatever, that things get somewhat undefined when two threads do that at the same time, but it's NOT AT ALL obvious to a "normal" application programmer. There's plenty of system calls that CAN be done concurrently, after all.

So we tried the simple idea, it didn't work, we grumbled a bit ;-) and then did something else instead. Not a big deal.

Please save your "translations" for stupid ideas that are obviously so without in-depth kernel knowledge or equivalent.

Thank you.

Signature · 1 line
Matthias Urlichs
#7 lore raw
In reply to Matthias Urlichs
Quoted 1 line · Matthias Urlichs

but it's NOT AT ALL obvious to a "normal" application programmer.

It gets worse. From the same draft (perhaps someone who has the final version could comment ?), in the rationale for read():

| The standard developers considered adding atomicity requirements to a
| pipe or FIFO, but recognized that due to the nature of pipes and FIFOs
| there could be no guarantee of atomicity of reads of {PIPE_BUF} or any
| other size that would be an aid to applications portability.

But then

| I/O is intended to be atomic to ordinary files and pipes and FIFOs.

Now, what exactly does "intended" mean ?

Of course, in this part, they only talk about data staying together, not whether it can get duplicated, or effects on f_pos.

- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         [email protected] /
/_http://www.almesberger.net/____________________________________________/
#8 lore raw
In reply to Matthias Urlichs

[email protected] (Matthias Urlichs) wrote on 06.02.04 in <[email protected]>:

Quoted 6 lines
Quoted 1 line · viro

"Somebody made a guess about undefined behaviour"

Guess what? The manpage says that read(2) return N bytes and advances the file pointer by N bytes. It doesn't talk, much less caution, about threads.

That's a defect in the man page, then.

POSIX says on <http://www.opengroup.org/onlinepubs/007904975/functions/
read.html>:

[...]
DESCRIPTION

The read() function shall attempt to read nbyte bytes from the file associated with the open file descriptor, fildes, into the buffer pointed to by buf. The behavior of multiple concurrent reads on the same pipe, FIFO, or terminal device is unspecified.
[...]

That's a pretty explicit warning exactly where one would expect it.

If Linux read(2) doesn't say something like that, I'll consider that a serious bug.

OTOH ...

Quoted 4 lines

YOU may immediately know, based on your kernel knowledge or whatever, that things get somewhat undefined when two threads do that at the same time, but it's NOT AT ALL obvious to a "normal" application programmer. There's plenty of system calls that CAN be done concurrently, after all.

... these days, programmers really *should* know enough to consult a free, current, and easily readable version of the relevant standard!

(If you forget where it is, it's easy to find from <http://www.unix-
systems.org/>.)

Quoted 2 lines

Please save your "translations" for stupid ideas that are obviously so without in-depth kernel knowledge or equivalent.

... such as the current case.

MfG Kai

#9 lore raw

Hi!

Quoted 10 lines

POSIX says on <http://www.opengroup.org/onlinepubs/007904975/functions/
read.html>:

[...]
DESCRIPTION

The read() function shall attempt to read nbyte bytes from the file associated with the open file descriptor, fildes, into the buffer pointed to by buf. The behavior of multiple concurrent reads on the same pipe, FIFO, or terminal device is unspecified.

...which implies that concurrent reads on regular files are okay :-P.

Pavel

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms