file offset corruption on 32-bit machines?
Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.html
Signature · 1 line
Meelis Roos ([email protected])
[Adding fsdevel list]
Quoted 6 lines · Meelis Roos
Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern.
Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).
I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (f_pos is used at many places in the kernel code).
I think that it is better to live with tiny-race-on-broken-patterns rather than paying for synchronization which is not needed for correct paths.
[*] file_pos_{read,write} (fs/read_write.c) are not called under lock (in sys_read, sys_write, ...), so even if f_pos is written atomically, you will be able to get races when accessing shared descriptor from different threads. I think that POSIX states, that behavior is undefined under these conditions.
Best regards
Signature · 5 lines
Michal Hocko SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 9 lines · Michal Hocko
Quoted 5 lines
Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.htmlAFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).
... but not in cases the caller is an intentionally evil code, right? :)
Quoted 3 lines
I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (
f_posis used at many places in the kernel code).
The f_pos races are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt
Signature · 2 lines
Jiri Kosina SUSE Labs
Jan Kara
reviewerQuoted 7 lines · Michal Hocko
Quoted 6 lines
Quoted 6 lines
Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.html
Quoted 6 lines
Quoted 4 lines
AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).
... but not in cases the caller is an intentionally evil code, right? :)
Yes.
Quoted 6 lines
Quoted 3 lines
I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (
f_posis used at many places in the kernel code).The
f_posraces are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt
Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SuSE CR Labs
Quoted 7 lines · Jan Kara
Quoted 2 lines
The
f_posraces are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txtWell, this race is more subtle - the window is just one instruction
wide (stores tof_posfrom CPU2 must come between the store of lower and upper 32-bits off_poson CPU1). And the only result is thatf_poshas 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...
Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?
I think this is worth fixing.
Signature · 2 lines
Jiri Kosina SUSE Labs
Quoted 6 lines · Jiri Kosina
Quoted 5 lines
Well, this race is more subtle - the window is just one instruction
wide (stores tof_posfrom CPU2 must come between the store of lower and upper 32-bits off_poson CPU1). And the only result is thatf_poshas 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...
Quoted 6 lines
Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?
I think this is worth fixing.
Seems a lot like reading jiffies to me. Is the seqlock the right solution to use for fixing this?
Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Jan Kara
reviewerQuoted 12 lines · Jiri Kosina
Quoted 11 lines
Quoted 5 lines
Well, this race is more subtle - the window is just one instruction
wide (stores tof_posfrom CPU2 must come between the store of lower and upper 32-bits off_poson CPU1). And the only result is thatf_poshas 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?
Quoted 4 lines
Quoted 1 line
I think this is worth fixing.
Seems a lot like reading jiffies to me. Is the seqlock the right solution to use for fixing this?
You can get your inspiration in the implementation of i_size_read()
and i_size_write() functions :). They deal with exactly the same problem. But in the case of f_pos, the number of readers and writers is balanced so maybe a spinlock would be fine as well...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SuSE CR Labs
Quoted 4 lines · Jan Kara
You can get your inspiration in the implementation of
i_size_read()
andi_size_write()functions :). They deal with exactly the same problem. But in the case off_pos, the number of readers and writers is balanced so maybe a spinlock would be fine as well...
It's not quite balanced -- see sys_getdents() for a counterexample.
i_size_read/write use a seqcount rather than a seqlock, but the principle is the same.
Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Jan Kara
reviewerQuoted 10 lines · Jan Kara
Quoted 8 lines
Quoted 3 lines
The
f_posraces are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txtWell, this race is more subtle - the window is just one instruction
wide (stores tof_posfrom CPU2 must come between the store of lower and upper 32-bits off_poson CPU1). And the only result is thatf_poshas 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...
Quoted 4 lines
Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?
Yes, this is theoretically possible.
Quoted 1 line
I think this is worth fixing.
Hmm, maybe it is, although I still don't see how to exploit it :).
Honza
Signature · 2 lines
Jan Kara <[email protected]> SuSE CR Labs
Quoted 16 lines · Jan Kara
Quoted 11 lines · Jan Kara
Quoted 9 lines
Quoted 3 lines
The
f_posraces are in fact exploitable, we've already been there. See for example
http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txtWell, this race is more subtle - the window is just one instruction
wide (stores tof_posfrom CPU2 must come between the store of lower and upper 32-bits off_poson CPU1). And the only result is thatf_poshas 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...Quoted 4 lines
Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?
Quoted 5 lines
Yes, this is theoretically possible.
Quoted 1 line
I think this is worth fixing.
Hmm, maybe it is, although I still don't see how to exploit it :).
Maybe (just guess) some high priority malicious process could try to preempt reading thread to always in the bad moment (when the half of the f_pos is written) and thus forcing it to read bad data (you usually don't check that file position is growing after each read and you wait only for end of the file).
But do agree, I still don't see something with really security implications (privileged processes usually don't work with such a big files).
Quoted 2 lines
Honza
Best regards
Signature · 5 lines
Michal Hocko SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Jan Kara
reviewerQuoted 23 lines · Michal Hocko
Quoted 16 lines · Jan Kara
Quoted 11 lines · Jan Kara
Quoted 9 lines
Quoted 3 lines
The
f_posraces are in fact exploitable, we've already been there. See for example
http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txtWell, this race is more subtle - the window is just one instruction
wide (stores tof_posfrom CPU2 must come between the store of lower and upper 32-bits off_poson CPU1). And the only result is thatf_poshas 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...Quoted 4 lines
Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?
Quoted 5 lines
Yes, this is theoretically possible.
Quoted 1 line
I think this is worth fixing.
Hmm, maybe it is, although I still don't see how to exploit it :).
Quoted 7 lines
Maybe (just guess) some high priority malicious process could try to preempt reading thread to always in the bad moment (when the half of the
f_posis written) and thus forcing it to read bad data (you usually don't check that file position is growing after each read and you wait only for end of the file).
But do agree, I still don't see something with really security implications (privileged processes usually don't work with such a big files).
Well, but for this to work the process you try to attack must access the
file from several threads in parallel without any locking... And I'm not aware of anybody really doing this.
Really the only attack vector I could imagine is that you create several
malitious processes which will try to corrupt f_pos and then use it (like if you could make it negative, I could imagine this could trigger some bug somewhere). But since possible corruptions are quite limited, I don't see how to corrupt f_pos to something at least remotely "useful".
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 1 line · Jiri Kosina
I think this is worth fixing.
This question comes very often, and Linus even wrote a patch
(http://lkml.org/lkml/2006/4/13/124 , http://lkml.org/lkml/2006/4/13/130)
But apparently there's no much interest in fixing it, because it would slow down some workloads...
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Jan Kara
reviewerQuoted 9 lines · Diego Calleja
Quoted 1 line · Jiri Kosina
I think this is worth fixing.
This question comes very often, and Linus even wrote a patch
(http://lkml.org/lkml/2006/4/13/124 , http://lkml.org/lkml/2006/4/13/130)But apparently there's no much interest in fixing it, because it would slow down some workloads...
Well, what Linus writes about is a different issue (and with a more
costly solution). Here we are concerned just with the problem that
file->f_pos = pos;
isn't atomic on some archs.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 19 lines · Jan Kara
Quoted 7 lines · Michal Hocko
Quoted 6 lines
Quoted 6 lines
Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.htmlQuoted 6 lines
Quoted 4 lines
AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).
... but not in cases the caller is an intentionally evil code, right? :)
Yes.
Quoted 4 lines
Quoted 3 lines
I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (
f_posis used at many places in the kernel code).
Quoted 7 lines
Quoted 2 lines
The
f_posraces are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txtWell, this race is more subtle - the window is just one instruction
wide (stores tof_posfrom CPU2 must come between the store of lower and upper 32-bits off_poson CPU1). And the only result is thatf_poshas 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...
Don't we have rlimit on max file size? I'd guess this could work around it?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Jan Kara
reviewerQuoted 22 lines · Pavel Machek
Quoted 21 lines · Jan Kara
Quoted 7 lines · Michal Hocko
Quoted 6 lines
Quoted 6 lines
Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.htmlQuoted 6 lines
Quoted 4 lines
AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).
... but not in cases the caller is an intentionally evil code, right? :)
Yes.
Quoted 6 lines
Quoted 3 lines
I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (
f_posis used at many places in the kernel code).The
f_posraces are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt
Quoted 8 lines
Quoted 5 lines
Well, this race is more subtle - the window is just one instruction
wide (stores tof_posfrom CPU2 must come between the store of lower and upper 32-bits off_poson CPU1). And the only result is thatf_poshas 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...Don't we have rlimit on max file size? I'd guess this could work around it?
There is this limit but AFAIK it limits max size of file you're able to
create. And write/truncate checks already their local variable so the real value used later.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 7 lines · Jiri Kosina
Quoted 6 lines · Michal Hocko
Quoted 5 lines
Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.html
Quoted 6 lines
Quoted 4 lines
AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).
... but not in cases the caller is an intentionally evil code, right? :)
Ok, but evil code needs to have access to your struct file and in such a case he can do worse things ;)
Or do you have some concrete (innocent looking) example?
Quoted 7 lines
Quoted 4 lines
I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (
f_posis used at many places in the kernel code).The
f_posraces are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt
This is different race with file position IMO. If I understand the report correctly, problem was with sleeping copy_to_user while the f_pos has changed.
Best regards
Signature · 5 lines
Michal Hocko SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Quoted 3 lines · Michal Hocko
This is different race with file position IMO. If I understand the report correctly, problem was with sleeping
copy_to_userwhile thef_poshas changed.
Is this really in principle different from obtaining reschedule between the two mov instructions?
Signature · 2 lines
Jiri Kosina SUSE Labs
Hello!
Quoted 3 lines
[*] file_pos_{read,write} (
fs/read_write.c) are not called under lock (insys_read,sys_write, ...), so even iff_posis written atomically, you will be able to get races when accessing shared descriptor from different threads.
There are however cases when such behavior is perfectly valid: For example you can have a file of records of a fixed size, whose order does not matter. Then multiple processes can produce the records in parallel, sharing a single fd.
Quoted 1 line
I think that POSIX states, that behavior is undefined under these conditions.
Do you have a pointer to that?
Have a nice fortnight
Signature · 3 lines
Martin `MJ' Mares <[email protected]> http://mj.ucw.cz/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth Mr. Worf, scan that ship." "Aye, Captain... 600 DPI?
Jan Kara
reviewerQuoted 10 lines
Hello!
Quoted 3 lines
[*] file_pos_{read,write} (
fs/read_write.c) are not called under lock (insys_read,sys_write, ...), so even iff_posis written atomically, you will be able to get races when accessing shared descriptor from different threads.There are however cases when such behavior is perfectly valid: For example you can have a file of records of a fixed size, whose order does not matter. Then multiple processes can produce the records in parallel, sharing a single fd.
Well, but noone guarantees that both processes don't read the same
data.
Quoted 3 lines
Quoted 1 line
I think that POSIX states, that behavior is undefined under these conditions.
Do you have a pointer to that?
SUSv3 says:
On files that support seeking (for example, a regular file), the read() shall start at a position in the file given by the file offset associated with fildes. The file offset shall be incremented by the number of bytes actually read.
But nowhere is specified when this happens so OS is perfectly free to
advance f_pos after read finishes when read from the other process is already running. And Linux does exactly that - actually, we do:
pos = f_pos
do reading which advances pos
f_pos = pos
So it can even in theory happen that one thread reads entries 1,2,3,2
because the other thread in the mean time finished reading entry 1...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SuSE CR Labs
Quoted 9 lines · Martin Mares
Quoted 4 lines
[*] file_pos_{read,write} (
fs/read_write.c) are not called under lock (insys_read,sys_write, ...), so even iff_posis written atomically, you will be able to get races when accessing shared descriptor from different threads.There are however cases when such behavior is perfectly valid: For example you can have a file of records of a fixed size, whose order does not matter. Then multiple processes can produce the records in parallel, sharing a single fd.
A rather more common thing:
Does this problem apply when appending lines or records to a log file, with or without O_APPEND?
Also, can this problem affect programs doing concurrent reads/writes using pread/pwrite (or the AIO equivalents)?
-- Jamie
Quoted 2 lines · Jamie Lokier
Also, can this problem affect programs doing concurrent reads/writes using pread/pwrite (or the AIO equivalents)?
pread/pwrite specify an explicit offset and do not change the file offset, so there's no way they can be affected. See the manpage.
Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Jan Kara
reviewerQuoted 11 lines
Quoted 9 lines · Martin Mares
Quoted 4 lines
[*] file_pos_{read,write} (
fs/read_write.c) are not called under lock (insys_read,sys_write, ...), so even iff_posis written atomically, you will be able to get races when accessing shared descriptor from different threads.There are however cases when such behavior is perfectly valid: For example you can have a file of records of a fixed size, whose order does not matter. Then multiple processes can produce the records in parallel, sharing a single fd.
Quoted 4 lines
A rather more common thing:
Does this problem apply when appending lines or records to a log file, with or without
O_APPEND?
O_APPEND works correctly in all cases (it ignores f_pos in the
descriptor). Without O_APPEND you can hit the race (but I'd like to see a sensible use case of this ;).
Quoted 2 lines
Also, can this problem affect programs doing concurrent reads/writes using pread/pwrite (or the AIO equivalents)?
As Matthew said, pread/pwrite are safe, parallel read can hit the race,
write was described above...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SuSE CR Labs
Quoted 8 lines · Michal Hocko
[Adding fsdevel list]
Quoted 5 lines · Meelis Roos
Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
Quoted 4 lines
Quoted 1 line
AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern.
We discussed this extensively some time ago in
http://thread.gmane.org/gmane.linux.file-systems/20712/focus=20771
No solution so far
-Andi