Skip to contentSkip to conversation

DAX data corruption for mmaped and written files

DISCUSSION Jan Kara 3 participants 4 messages Mar 24, 2016 – Apr 4, 2016 VFS

lore
Share & tools
lore

Jan Kara

author reviewer
#0 lore raw

Hello,

yesterday I have been stress-testing mmap code with my new fault locking patches and I have found a data corruption issue when file is written both via mmap and standard write(2). The problem is following:

CPU1 CPU2
dax_io() dax_fault()
get_block() - allocates block
... get_block() - finds allocated block
- zeroes it inside fs
fault completese

if (buffer_unwritten(bh) || buffer_new(bh)) -> new buffer
dax_new_buf() -> zeroes buffer which may
overwrite user data

In some cases the race can also go the other way around and we lose data written by write.

So either we need to do the zeroing inside fs also for write(2) path (but that would essentially mean we would write the block twice for each allocating write) or we would need dax_io() to also use radix tree locking to serialize against page faults (in the same way page cache does this with page lock). Any opinion on what would be better?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#1 lore raw
Quoted 17 lines · Jan Kara

Hello,

yesterday I have been stress-testing mmap code with my new fault locking patches and I have found a data corruption issue when file is written both via mmap and standard write(2). The problem is following:

CPU1 CPU2
dax_io() dax_fault()
get_block() - allocates block
... get_block() - finds allocated block
- zeroes it inside fs
fault completese

if (buffer_unwritten(bh) || buffer_new(bh)) -> new buffer
dax_new_buf() -> zeroes buffer which may
overwrite user data

Quoted 8 lines

In some cases the race can also go the other way around and we lose data written by write.

So either we need to do the zeroing inside fs also for write(2) path (but that would essentially mean we would write the block twice for each allocating write) or we would need dax_io() to also use radix tree locking to serialize against page faults (in the same way page cache does this with page lock). Any opinion on what would be better?

If the radix tree locking is essentially trying to be analogous to the page lock, and the page lock is already used to serialize I/O vs page faults for the page cache case, I guess doing that same serialization using our new DAX locking seems like a natural fit.

#2 lore raw
In reply to Jan Kara
Quoted 13 lines · Jan Kara

Hello,

yesterday I have been stress-testing mmap code with my new fault locking patches and I have found a data corruption issue when file is written both via mmap and standard write(2). The problem is following:

CPU1 CPU2
dax_io() dax_fault()
get_block() - allocates block
... get_block() - finds allocated block
- zeroes it inside fs
fault completese

Quoted 3 lines

if (buffer_unwritten(bh) || buffer_new(bh)) -> new buffer
dax_new_buf() -> zeroes buffer which may
overwrite user data

Which filesystem? XFS should be, in both cases, zeroing the block entirely inside get_block() when it is first allocated. i.e we should see:

CPU1 CPU2
dax_io() dax_fault()
get_block() - allocates block
- zeroes it inside fs
... get_block() - finds allocated block
fault completes

buffer returned is not new or unwritten.

Quoted 2 lines

In some cases the race can also go the other way around and we lose data written by write.

It shouldn't:

CPU1 CPU2
dax_io() dax_fault()
get_block() - allocates block
- zeroes it inside fs
....
get_block() - finds allocated block
buffer returned is not new or unwritten.
fault completes

Quoted 3 lines

So either we need to do the zeroing inside fs also for write(2) path (but that would essentially mean we would write the block twice for each allocating write)

Yup, that's what XFS is supposed to be doing right now...

Quoted 3 lines

or we would need dax_io() to also use radix tree locking
to serialize against page faults (in the same way page cache does this with page lock). Any opinion on what would be better?

Don't care, as long as data is not corrupted ;)

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author reviewer
#3 lore raw
Quoted 22 lines · Dave Chinner
Quoted 13 lines · Jan Kara

Hello,

yesterday I have been stress-testing mmap code with my new fault locking patches and I have found a data corruption issue when file is written both via mmap and standard write(2). The problem is following:

CPU1 CPU2
dax_io() dax_fault()
get_block() - allocates block
... get_block() - finds allocated block
- zeroes it inside fs
fault completese

Quoted 3 lines

if (buffer_unwritten(bh) || buffer_new(bh)) -> new buffer
dax_new_buf() -> zeroes buffer which may
overwrite user data

Which filesystem? XFS should be, in both cases, zeroing the block entirely inside get_block() when it is first allocated. i.e we should see:

Quoted 12 lines

CPU1 CPU2
dax_io() dax_fault()
get_block() - allocates block
- zeroes it inside fs
... get_block() - finds allocated block
fault completes

buffer returned is not new or unwritten.

Quoted 2 lines

In some cases the race can also go the other way around and we lose data written by write.

Yeah, correct. This is ext4 specific issue because ext4 doesn't return prezeroed blocks from get_block() callback used for dax_io().

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR