Skip to contentSkip to conversation

file offset corruption on 32-bit machines?

DISCUSSION Meelis Roos 10 participants 23 messages Apr 8, 2008 – Apr 14, 2008 VFS

lore
Share & tools
lore
#0 lore raw

Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.

http://jeffr-tech.livejournal.com/21014.html

Signature · 1 line
Meelis Roos ([email protected])
#1 lore raw

[Adding fsdevel list]

Quoted 6 lines · Meelis Roos

Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.

http://jeffr-tech.livejournal.com/21014.html

AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern.
Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).

I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (f_pos is used at many places in the kernel code).

I think that it is better to live with tiny-race-on-broken-patterns rather than paying for synchronization which is not needed for correct paths.

[*] file_pos_{read,write} (fs/read_write.c) are not called under lock (in sys_read, sys_write, ...), so even if f_pos is written atomically, you will be able to get races when accessing shared descriptor from different threads. I think that POSIX states, that behavior is undefined under these conditions.

Best regards

Signature · 5 lines
Michal Hocko
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic 
#2 lore raw
Quoted 9 lines · Michal Hocko
Quoted 5 lines

Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.html

AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).

... but not in cases the caller is an intentionally evil code, right? :)

Quoted 3 lines

I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (f_pos is used at many places in the kernel code).

The f_pos races are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

Signature · 2 lines
Jiri Kosina
SUSE Labs
reviewer
#3 lore raw
Quoted 7 lines · Michal Hocko
Quoted 6 lines
Quoted 6 lines

Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.html

Quoted 6 lines
Quoted 4 lines

AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).

... but not in cases the caller is an intentionally evil code, right? :)

Yes.

Quoted 6 lines
Quoted 3 lines

I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (f_pos is used at many places in the kernel code).

The f_pos races are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#4 lore raw
Quoted 7 lines · Jan Kara
Quoted 2 lines

The f_pos races are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?

I think this is worth fixing.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#5 lore raw
Quoted 6 lines · Jiri Kosina
Quoted 5 lines

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Quoted 6 lines

Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?

I think this is worth fixing.

Seems a lot like reading jiffies to me. Is the seqlock the right solution to use for fixing this?

Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
reviewer
#6 lore raw
Quoted 12 lines · Jiri Kosina
Quoted 11 lines
Quoted 5 lines

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?

Quoted 4 lines
Quoted 1 line

I think this is worth fixing.

Seems a lot like reading jiffies to me. Is the seqlock the right solution to use for fixing this?

You can get your inspiration in the implementation of i_size_read()
and i_size_write() functions :). They deal with exactly the same problem. But in the case of f_pos, the number of readers and writers is balanced so maybe a spinlock would be fine as well...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#7 lore raw
Quoted 4 lines · Jan Kara

You can get your inspiration in the implementation of i_size_read()
and i_size_write() functions :). They deal with exactly the same problem. But in the case of f_pos, the number of readers and writers is balanced so maybe a spinlock would be fine as well...

It's not quite balanced -- see sys_getdents() for a counterexample.

i_size_read/write use a seqcount rather than a seqlock, but the principle is the same.

Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
reviewer
#8 lore raw
In reply to Jiri Kosina
Quoted 10 lines · Jan Kara
Quoted 8 lines
Quoted 3 lines

The f_pos races are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Quoted 4 lines

Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?

Yes, this is theoretically possible.

Quoted 1 line

I think this is worth fixing.

Hmm, maybe it is, although I still don't see how to exploit it :).

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#9 lore raw
Quoted 16 lines · Jan Kara
Quoted 11 lines · Jan Kara
Quoted 9 lines
Quoted 3 lines

The f_pos races are in fact exploitable, we've already been there. See for example
http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Quoted 4 lines

Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?

Quoted 5 lines

Yes, this is theoretically possible.

Quoted 1 line

I think this is worth fixing.

Hmm, maybe it is, although I still don't see how to exploit it :).

Maybe (just guess) some high priority malicious process could try to preempt reading thread to always in the bad moment (when the half of the f_pos is written) and thus forcing it to read bad data (you usually don't check that file position is growing after each read and you wait only for end of the file).
But do agree, I still don't see something with really security implications (privileged processes usually don't work with such a big files).

Quoted 2 lines

Honza

Best regards

Signature · 5 lines
Michal Hocko
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic 
reviewer
#10 lore raw
Quoted 23 lines · Michal Hocko
Quoted 16 lines · Jan Kara
Quoted 11 lines · Jan Kara
Quoted 9 lines
Quoted 3 lines

The f_pos races are in fact exploitable, we've already been there. See for example
http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Quoted 4 lines

Supposing you are not holding any spinlock and are running with preemptible kernel (pretty common scenario nowadays), there is nothing that would prevent kernel from rescheduling between the two instructions, enlarging the race window to be more comfortable for attacker, right?

Quoted 5 lines

Yes, this is theoretically possible.

Quoted 1 line

I think this is worth fixing.

Hmm, maybe it is, although I still don't see how to exploit it :).

Quoted 7 lines

Maybe (just guess) some high priority malicious process could try to preempt reading thread to always in the bad moment (when the half of the f_pos is written) and thus forcing it to read bad data (you usually don't check that file position is growing after each read and you wait only for end of the file).
But do agree, I still don't see something with really security implications (privileged processes usually don't work with such a big files).

Well, but for this to work the process you try to attack must access the
file from several threads in parallel without any locking... And I'm not aware of anybody really doing this.
Really the only attack vector I could imagine is that you create several
malitious processes which will try to corrupt f_pos and then use it (like if you could make it negative, I could imagine this could trigger some bug somewhere). But since possible corruptions are quite limited, I don't see how to corrupt f_pos to something at least remotely "useful".

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#11 lore raw
In reply to Jiri Kosina
Quoted 1 line · Jiri Kosina

I think this is worth fixing.

This question comes very often, and Linus even wrote a patch
(http://lkml.org/lkml/2006/4/13/124 , http://lkml.org/lkml/2006/4/13/130)

But apparently there's no much interest in fixing it, because it would slow down some workloads...
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

reviewer
#12 lore raw
Quoted 9 lines · Diego Calleja
Quoted 1 line · Jiri Kosina

I think this is worth fixing.

This question comes very often, and Linus even wrote a patch
(http://lkml.org/lkml/2006/4/13/124 , http://lkml.org/lkml/2006/4/13/130)

But apparently there's no much interest in fixing it, because it would slow down some workloads...

Well, what Linus writes about is a different issue (and with a more
costly solution). Here we are concerned just with the problem that
file->f_pos = pos;
isn't atomic on some archs.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#13 lore raw
In reply to Jan Kara
Quoted 19 lines · Jan Kara
Quoted 7 lines · Michal Hocko
Quoted 6 lines
Quoted 6 lines

Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.html

Quoted 6 lines
Quoted 4 lines

AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).

... but not in cases the caller is an intentionally evil code, right? :)

Yes.

Quoted 4 lines
Quoted 3 lines

I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (f_pos is used at many places in the kernel code).

Quoted 7 lines
Quoted 2 lines

The f_pos races are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Don't we have rlimit on max file size? I'd guess this could work around it?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
reviewer
#14 lore raw
Quoted 22 lines · Pavel Machek
Quoted 21 lines · Jan Kara
Quoted 7 lines · Michal Hocko
Quoted 6 lines
Quoted 6 lines

Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.html

Quoted 6 lines
Quoted 4 lines

AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).

... but not in cases the caller is an intentionally evil code, right? :)

Yes.

Quoted 6 lines
Quoted 3 lines

I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (f_pos is used at many places in the kernel code).

The f_pos races are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

Quoted 8 lines
Quoted 5 lines

Well, this race is more subtle - the window is just one instruction
wide (stores to f_pos from CPU2 must come between the store of lower and upper 32-bits of f_pos on CPU1). And the only result is that f_pos has 32-bits from one file pointer and 32-bits from the other one. So I can hardly imagine this would be exploitable...

Don't we have rlimit on max file size? I'd guess this could work around it?

There is this limit but AFAIK it limits max size of file you're able to
create. And write/truncate checks already their local variable so the real value used later.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#15 lore raw
In reply to Jiri Kosina
Quoted 7 lines · Jiri Kosina
Quoted 6 lines · Michal Hocko
Quoted 5 lines

Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.
http://jeffr-tech.livejournal.com/21014.html

Quoted 6 lines
Quoted 4 lines

AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern. Note that you have to access shared struct file (same file descriptor) in different threads which should be synchronized by caller anyway (*).

... but not in cases the caller is an intentionally evil code, right? :)

Ok, but evil code needs to have access to your struct file and in such a case he can do worse things ;)
Or do you have some concrete (innocent looking) example?

Quoted 7 lines
Quoted 4 lines

I also don't see any security implications from this race, but maybe someone with more knowlage about fs can see (f_pos is used at many places in the kernel code).

The f_pos races are in fact exploitable, we've already been there. See for example http://www.isec.pl/vulnerabilities/isec-0016-procleaks.txt

This is different race with file position IMO. If I understand the report correctly, problem was with sleeping copy_to_user while the f_pos has changed.

Best regards

Signature · 5 lines
Michal Hocko
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9    
Czech Republic 
#16 lore raw
Quoted 3 lines · Michal Hocko

This is different race with file position IMO. If I understand the report correctly, problem was with sleeping copy_to_user while the f_pos has changed.

Is this really in principle different from obtaining reschedule between the two mov instructions?

Signature · 2 lines
Jiri Kosina
SUSE Labs
#17 lore raw
In reply to Michal Hocko

Hello!

Quoted 3 lines

[*] file_pos_{read,write} (fs/read_write.c) are not called under lock (in sys_read, sys_write, ...), so even if f_pos is written atomically, you will be able to get races when accessing shared descriptor from different threads.

There are however cases when such behavior is perfectly valid: For example you can have a file of records of a fixed size, whose order does not matter. Then multiple processes can produce the records in parallel, sharing a single fd.

Quoted 1 line

I think that POSIX states, that behavior is undefined under these conditions.

Do you have a pointer to that?

Have a nice fortnight

Signature · 3 lines
Martin `MJ' Mares                          <[email protected]>   http://mj.ucw.cz/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
Mr. Worf, scan that ship."  "Aye, Captain... 600 DPI?
reviewer
#18 lore raw
Quoted 10 lines

Hello!

Quoted 3 lines

[*] file_pos_{read,write} (fs/read_write.c) are not called under lock (in sys_read, sys_write, ...), so even if f_pos is written atomically, you will be able to get races when accessing shared descriptor from different threads.

There are however cases when such behavior is perfectly valid: For example you can have a file of records of a fixed size, whose order does not matter. Then multiple processes can produce the records in parallel, sharing a single fd.

Well, but noone guarantees that both processes don't read the same
data.

Quoted 3 lines
Quoted 1 line

I think that POSIX states, that behavior is undefined under these conditions.

Do you have a pointer to that?

SUSv3 says:
On files that support seeking (for example, a regular file), the read() shall start at a position in the file given by the file offset associated with fildes. The file offset shall be incremented by the number of bytes actually read.

But nowhere is specified when this happens so OS is perfectly free to
advance f_pos after read finishes when read from the other process is already running. And Linux does exactly that - actually, we do:
pos = f_pos
do reading which advances pos
f_pos = pos

So it can even in theory happen that one thread reads entries 1,2,3,2
because the other thread in the mean time finished reading entry 1...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#19 lore raw
In reply to Martin Mares
Quoted 9 lines · Martin Mares
Quoted 4 lines

[*] file_pos_{read,write} (fs/read_write.c) are not called under lock (in sys_read, sys_write, ...), so even if f_pos is written atomically, you will be able to get races when accessing shared descriptor from different threads.

There are however cases when such behavior is perfectly valid: For example you can have a file of records of a fixed size, whose order does not matter. Then multiple processes can produce the records in parallel, sharing a single fd.

A rather more common thing:

Does this problem apply when appending lines or records to a log file, with or without O_APPEND?

Also, can this problem affect programs doing concurrent reads/writes using pread/pwrite (or the AIO equivalents)?

-- Jamie

#20 lore raw
Quoted 2 lines · Jamie Lokier

Also, can this problem affect programs doing concurrent reads/writes using pread/pwrite (or the AIO equivalents)?

pread/pwrite specify an explicit offset and do not change the file offset, so there's no way they can be affected. See the manpage.

Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
reviewer
#21 lore raw
In reply to Jamie Lokier
Quoted 11 lines
Quoted 9 lines · Martin Mares
Quoted 4 lines

[*] file_pos_{read,write} (fs/read_write.c) are not called under lock (in sys_read, sys_write, ...), so even if f_pos is written atomically, you will be able to get races when accessing shared descriptor from different threads.

There are however cases when such behavior is perfectly valid: For example you can have a file of records of a fixed size, whose order does not matter. Then multiple processes can produce the records in parallel, sharing a single fd.

Quoted 4 lines

A rather more common thing:

Does this problem apply when appending lines or records to a log file, with or without O_APPEND?

O_APPEND works correctly in all cases (it ignores f_pos in the
descriptor). Without O_APPEND you can hit the race (but I'd like to see a sensible use case of this ;).

Quoted 2 lines

Also, can this problem affect programs doing concurrent reads/writes using pread/pwrite (or the AIO equivalents)?

As Matthew said, pread/pwrite are safe, parallel read can hit the race,
write was described above...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#22 lore raw
In reply to Michal Hocko
Quoted 8 lines · Michal Hocko

[Adding fsdevel list]

Quoted 5 lines · Meelis Roos

Jeff Robertson analyzes the behaviour of different operating systems' 64-bit file offset implementation and concludes that on 32-bit machines, Linux and Solaris lack any locking to keep the two 32-bit halves in sync and this could cause rare file offset corruption.

Quoted 4 lines
Quoted 1 line

http://jeffr-tech.livejournal.com/21014.html

AFAICS, this race is theoretically possible, but it is very hard (almost impossible) to trigger with a sane file usage pattern.

We discussed this extensively some time ago in

http://thread.gmane.org/gmane.linux.file-systems/20712/focus=20771

No solution so far

-Andi