DAX data corruption for mmaped and written files
Jan Kara
reviewerHello,
yesterday I have been stress-testing mmap code with my new fault locking patches and I have found a data corruption issue when file is written both via mmap and standard write(2). The problem is following:
CPU1 CPU2dax_io() dax_fault()
get_block() - allocates block
... get_block() - finds allocated block
- zeroes it inside fs
fault completese
if (buffer_unwritten(bh) || buffer_new(bh)) -> new buffer
dax_new_buf() -> zeroes buffer which may
overwrite user data
In some cases the race can also go the other way around and we lose data written by write.
So either we need to do the zeroing inside fs also for write(2) path (but that would essentially mean we would write the block twice for each allocating write) or we would need dax_io() to also use radix tree locking to serialize against page faults (in the same way page cache does this with page lock). Any opinion on what would be better?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 17 lines · Jan Kara
Hello,
yesterday I have been stress-testing mmap code with my new fault locking patches and I have found a data corruption issue when file is written both via mmap and standard write(2). The problem is following:
CPU1 CPU2
dax_io()dax_fault()
get_block()- allocates block
...get_block()- finds allocated block
- zeroes it inside fs
fault completeseif (
buffer_unwritten(bh)||buffer_new(bh)) -> new buffer
dax_new_buf()-> zeroes buffer which may
overwrite user data
Quoted 8 lines
In some cases the race can also go the other way around and we lose data written by write.
So either we need to do the zeroing inside fs also for write(2) path (but that would essentially mean we would write the block twice for each allocating write) or we would need
dax_io()to also use radix tree locking to serialize against page faults (in the same way page cache does this with page lock). Any opinion on what would be better?
If the radix tree locking is essentially trying to be analogous to the page lock, and the page lock is already used to serialize I/O vs page faults for the page cache case, I guess doing that same serialization using our new DAX locking seems like a natural fit.
Quoted 13 lines · Jan Kara
Hello,
yesterday I have been stress-testing mmap code with my new fault locking patches and I have found a data corruption issue when file is written both via mmap and standard write(2). The problem is following:
CPU1 CPU2
dax_io()dax_fault()
get_block()- allocates block
...get_block()- finds allocated block
- zeroes it inside fs
fault completese
Quoted 3 lines
if (
buffer_unwritten(bh)||buffer_new(bh)) -> new buffer
dax_new_buf()-> zeroes buffer which may
overwrite user data
Which filesystem? XFS should be, in both cases, zeroing the block entirely inside get_block() when it is first allocated. i.e we should see:
CPU1 CPU2dax_io() dax_fault()
get_block() - allocates block
- zeroes it inside fs
... get_block() - finds allocated block
fault completes
buffer returned is not new or unwritten.
Quoted 2 lines
In some cases the race can also go the other way around and we lose data written by write.
It shouldn't:
CPU1 CPU2dax_io() dax_fault()
get_block() - allocates block
- zeroes it inside fs
....
get_block() - finds allocated block
buffer returned is not new or unwritten.
fault completes
Quoted 3 lines
So either we need to do the zeroing inside fs also for write(2) path (but that would essentially mean we would write the block twice for each allocating write)
Yup, that's what XFS is supposed to be doing right now...
Quoted 3 lines
or we would need
dax_io()to also use radix tree locking
to serialize against page faults (in the same way page cache does this with page lock). Any opinion on what would be better?
Don't care, as long as data is not corrupted ;)
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jan Kara
reviewerQuoted 22 lines · Dave Chinner
Quoted 13 lines · Jan Kara
Hello,
yesterday I have been stress-testing mmap code with my new fault locking patches and I have found a data corruption issue when file is written both via mmap and standard write(2). The problem is following:
CPU1 CPU2
dax_io()dax_fault()
get_block()- allocates block
...get_block()- finds allocated block
- zeroes it inside fs
fault completeseQuoted 3 lines
if (
buffer_unwritten(bh)||buffer_new(bh)) -> new buffer
dax_new_buf()-> zeroes buffer which may
overwrite user dataWhich filesystem? XFS should be, in both cases, zeroing the block entirely inside
get_block()when it is first allocated. i.e we should see:
Quoted 12 lines
CPU1 CPU2
dax_io()dax_fault()
get_block()- allocates block
- zeroes it inside fs
...get_block()- finds allocated block
fault completesbuffer returned is not new or unwritten.
Quoted 2 lines
In some cases the race can also go the other way around and we lose data written by write.
Yeah, correct. This is ext4 specific issue because ext4 doesn't return prezeroed blocks from get_block() callback used for dax_io().
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR