Skip to contentSkip to conversation

mark buffer_head mapping preallocate area as new during write_begin with delayed allocation

RFC PATCH In review Aneesh Kumar K.V 5 participants 16 messages Apr 27, 2009 – Apr 29, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 mark buffer_head mapping preallocate area as new during write_begin with delayed allocation

We need to mark the buffer_head mapping prealloc space as new during write_begin. Otherwise we don't zero out the page cache content properly for a partial write. This will cause file corruption with preallocation.

---
 fs/ext4/inode.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
fs/ext4/inode.c +2 -0
index c6bd6ce..c7251ec 100644
@@ -2323,6 +2323,8 @@ static int ext4_da_get_block_prep(struct inode *inode, sector_t iblock,
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
 		bh_result->b_size = (ret << inode->i_blkbits);
+		if (buffer_unwritten(bh_result))
+			set_buffer_new(bh_result);
 		ret = 0;
 	}
 
Signature · 1 line
tg: (51bec03..) preallocate_corruption (depends on: ext4_lock_group_conversion)
#1 lore raw
Quoted 16 lines · Aneesh Kumar K.V

We need to mark the buffer_head mapping prealloc space as new during write_begin. Otherwise we don't zero out the page cache content properly for a partial write. This will cause file corruption with preallocation.

---
 fs/ext4/inode.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
fs/ext4/inode.c +0 -0
index c6bd6ce..c7251ec 100644

fs/ext4/inode.c static int ext4_da_get_block_prep(struct inode *inode, sector_t iblock, · line 2330

Quoted 8 lines
fs/ext4/inode.c +2 -0
@@ -2323,6 +2323,8 @@ static int ext4_da_get_block_prep(struct inode *inode, sector_t iblock,
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
 		bh_result->b_size = (ret << inode->i_blkbits);
+		if (buffer_unwritten(bh_result))
+			set_buffer_new(bh_result);
 		ret = 0;
 	}
 

Hm, yep, that sure does break things. For some future regression test:

[root@inode tmp]# /root/fallocate -l 8k testfile
[root@inode tmp]# dd if=/dev/zero of=testfile bs=1 count=10 conv=notrunc 10+0 records in
10+0 records out
10 bytes (10 B) copied, 5.1491e-05 s, 194 kB/s
[root@inode tmp]# hexdump -C testfile

<much garbage ensues>

This looks pretty reasonable; Aneesh & I talked online and found that xfs has a somewhat similar fix:

commit 549054afadae44889c0b40d4c3bfb0207b98d5a0

Author: David Chinner <[email protected]>
Date:   Sat Feb 10 18:36:35 2007 +1100

[XFS] Fix sub-block zeroing for buffered writes into unwritten extents.

When writing less than a filesystem block of data into an unwritten
extent
via buffered I/O, __xfs_get_blocks fails to set the buffer new flag.
As a
result, the generic code will not zero either edge of the block
resulting
in garbage being written to disk either side of the real data. Set the buffer new state on bufferd writes to unwritten extents to ensure that zeroing occurs.

-Eric

#2 lore raw
In reply to Aneesh Kumar K.V
Quoted 17 lines · 在 2009-04-28二的 00:35 +0530,Aneesh Kumar K.V写道:

We need to mark the buffer_head mapping prealloc space as new during write_begin. Otherwise we don't zero out the page cache content properly for a partial write. This will cause file corruption with preallocation.

---
 fs/ext4/inode.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
fs/ext4/inode.c +0 -0
index c6bd6ce..c7251ec 100644

fs/ext4/inode.c static int ext4_da_get_block_prep(struct inode *inode, sector_t iblock, · line 2330

Quoted 8 lines
fs/ext4/inode.c +2 -0
@@ -2323,6 +2323,8 @@ static int ext4_da_get_block_prep(struct inode *inode, sector_t iblock,
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
 		bh_result->b_size = (ret << inode->i_blkbits);
+		if (buffer_unwritten(bh_result))
+			set_buffer_new(bh_result);
 		ret = 0;
 	}
 

Thanks Aneesh.

Just to share with list, I have seen garbage content show up on a preallocated but later partially written blocks. This only happens with delayed allocation. The test simply preallocate 2blocks to a new file, then write a few bytes to the beginning of file(less than a block), and od shows the first block the written content followed by garbage filled to the end of the first block.

After examing the code, we did set the buffer as new for nondelalloc, as the create flag passed to ext4_ext_get_blocks() is 1, while for delalloc case, ext4_get_blocks_prep() calling ext4_ext_get_block() with create =0, which leads to the code path that forget to set the bh as new if the block is preallocated.

This patch is mostly correct except forget to set the bh_result->bdev, which caused the fs blow out.

The updated patch fixed the problem for me.

fs/ext4/inode.c +3 -0
Index: linux-2.6.28-rc6/fs/ext4/inode.c
===================================================================
@@ -2177,7 +2177,10 @@ static int ext4_da_get_block_prep(struct
 		set_buffer_new(bh_result);
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
+		if (buffer_unwritten(bh_result))
+			set_buffer_new(bh_result);
 		bh_result->b_size = (ret << inode->i_blkbits);
+		bh_result->b_bdev = inode->i_sb->s_bdev;
 		ret = 0;
 	}
 

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 36 lines · Mingming Cao
Quoted 17 lines · 在 2009-04-28二的 00:35 +0530,Aneesh Kumar K.V写道:

We need to mark the buffer_head mapping prealloc space as new during write_begin. Otherwise we don't zero out the page cache content properly for a partial write. This will cause file corruption with preallocation.

---
 fs/ext4/inode.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
fs/ext4/inode.c +0 -0
index c6bd6ce..c7251ec 100644

fs/ext4/inode.c static int ext4_da_get_block_prep(struct inode *inode, sector_t iblock, · line 2330

Quoted 8 lines
fs/ext4/inode.c +2 -0
@@ -2323,6 +2323,8 @@ static int ext4_da_get_block_prep(struct inode *inode, sector_t iblock,
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
 		bh_result->b_size = (ret << inode->i_blkbits);
+		if (buffer_unwritten(bh_result))
+			set_buffer_new(bh_result);
 		ret = 0;
 	}
 

Thanks Aneesh.

Just to share with list, I have seen garbage content show up on a preallocated but later partially written blocks. This only happens with delayed allocation. The test simply preallocate 2blocks to a new file, then write a few bytes to the beginning of file(less than a block), and od shows the first block the written content followed by garbage filled to the end of the first block.

Quoted 8 lines

After examing the code, we did set the buffer as new for nondelalloc, as the create flag passed to ext4_ext_get_blocks() is 1, while for delalloc case, ext4_get_blocks_prep() calling ext4_ext_get_block() with create =0, which leads to the code path that forget to set the bh as new if the block is preallocated.

This patch is mostly correct except forget to set the bh_result->bdev, which caused the fs blow out.

Yep, I saw the oops too.

Quoted 9 lines

The updated patch fixed the problem for me.

Index: linux-2.6.28-rc6/fs/ext4/inode.c
===================================================================

fs/ext4/inode.c static int ext4_da_get_block_prep(struct · line 2185

Quoted 9 lines
fs/ext4/inode.c +3 -0
@@ -2177,7 +2177,10 @@ static int ext4_da_get_block_prep(struct
 		set_buffer_new(bh_result);
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
+		if (buffer_unwritten(bh_result))
+			set_buffer_new(bh_result);
 		bh_result->b_size = (ret << inode->i_blkbits);
+		bh_result->b_bdev = inode->i_sb->s_bdev;
 		ret = 0;
 	}

It may be just me, but I'd like to sort out why we now need to set b_bdev here just because we set it as new, and it wasn't necessary before...? If it's obvious I'm not yet seeing it :)

-Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
In reply to Mingming Cao

On Mon, Apr 27, 2009 at 04:04:54PM -0700, Mingming Cao wrote:
.....

fs/ext4/inode.c static int ext4_da_get_block_prep(struct · line 2183

Quoted 13 lines

Index: linux-2.6.28-rc6/fs/ext4/inode.c
===================================================================

fs/ext4/inode.c +3 -0
@@ -2177,7 +2177,10 @@ static int ext4_da_get_block_prep(struct
 		set_buffer_new(bh_result);
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
+		if (buffer_unwritten(bh_result))
+			set_buffer_new(bh_result);
 		bh_result->b_size = (ret << inode->i_blkbits);
+		bh_result->b_bdev = inode->i_sb->s_bdev;

Updated patch to set bh_result->b_dev. I also added comments in the source to explain whey we need to mark buffer_head new. Also updated single line patch summary. I will send the update (-v2) patch.

-aneesh

#5 lore raw
Quoted 13 lines · Aneesh Kumar K.V

On Mon, Apr 27, 2009 at 04:04:54PM -0700, Mingming Cao wrote:
.....

Quoted 10 lines

Index: linux-2.6.28-rc6/fs/ext4/inode.c
===================================================================

fs/ext4/inode.c +1 -0
@@ -2177,7 +2177,10 @@ static int ext4_da_get_block_prep(struct
 		set_buffer_new(bh_result);
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
+		if (buffer_unwritten(bh_result))
Quoted 8 lines

fs/ext4/inode.c static int ext4_da_get_block_prep(struct · line 2183

Quoted 3 lines
fs/ext4/inode.c +2 -0
+			set_buffer_new(bh_result);
 		bh_result->b_size = (ret << inode->i_blkbits);
+		bh_result->b_bdev = inode->i_sb->s_bdev;

Updated patch to set bh_result->b_dev. I also added comments in the source to explain whey we need to mark buffer_head new. Also updated single line patch summary. I will send the update (-v2) patch.

Looking at the source again i guess setting just b_dev is not enough. unmap_underlying_metadata looks at the mapping block number, which we don't have in case on unwritten buffer_head. How about the below patch ? It involve vfs changes. But i guess it is correct with respect to the meaning of BH_New (Disk mapping was newly created by get_block). I guess BH_New implies BH_Mapped.

I haven't tested the patch yet. Also it should be split into multiple patches. It also a fix a problem where we missed an unamp_underlying_metadata in case of delayed allocated blocks. I guess that can also cause corruption with delayed allocation.

From: Aneesh Kumar K.V <[email protected]>
Subject: [PATCH -V3] ext4: Fix sub-block zeroing for buffered writes into unwritten extents.

We need to mark the buffer_head mapping prealloc space as new during write_begin. Otherwise we don't zero out the page cache content properly for a partial write. This will cause file corruption with preallocation.

---
 fs/buffer.c     |    9 ++++++++-
 fs/ext4/inode.c |    8 +++++---
 2 files changed, 13 insertions(+), 4 deletions(-)
fs/buffer.c +8 -1
index b3e5be7..13f0d52 100644
@@ -1867,15 +1867,22 @@ static int __block_prepare_write(struct inode *inode, struct page *page,
 			err = get_block(inode, block, bh, 1);
 			if (err)
 				break;
-			if (buffer_new(bh)) {
+			if (buffer_new(bh))
 				unmap_underlying_metadata(bh->b_bdev,
 							bh->b_blocknr);
+			if (buffer_new(bh) || buffer_unwritten(bh) ||
+					buffer_delay(bh)) {
 				if (PageUptodate(page)) {
 					clear_buffer_new(bh);
 					set_buffer_uptodate(bh);
 					mark_buffer_dirty(bh);
 					continue;
 				}
+				/*
+				 * sub-block writes into unwritten or
+				 * delayed buffer should result in zero out
+				 * of the rest of the buffer
+				 */
 				if (block_end > to || block_start < from)
 					zero_user_segments(page,
 						to, block_end,
fs/ext4/inode.c +5 -3
index e91f978..504afb7 100644
@@ -1892,13 +1892,17 @@ static void mpage_put_bnr_to_bhs(struct mpage_da_data *mpd, sector_t logical,
 				if (buffer_delay(bh)) {
 					bh->b_blocknr = pblock;
 					clear_buffer_delay(bh);
+					set_buffer_mapped(bh);
 					bh->b_bdev = inode->i_sb->s_bdev;
+					unmap_underlying_metadata(bh->b_bdev,
+								pblock);
 				} else if (buffer_unwritten(bh)) {
 					bh->b_blocknr = pblock;
 					clear_buffer_unwritten(bh);
 					set_buffer_mapped(bh);
-					set_buffer_new(bh);
 					bh->b_bdev = inode->i_sb->s_bdev;
+					unmap_underlying_metadata(bh->b_bdev,
+								pblock);
 				} else if (buffer_mapped(bh))
 					BUG_ON(bh->b_blocknr != pblock);
 
@@ -2318,8 +2322,6 @@ static int ext4_da_get_block_prep(struct inode *inode, sector_t iblock,
 			/* not enough space to reserve */
 			return ret;
 
-		map_bh(bh_result, inode->i_sb, 0);
-		set_buffer_new(bh_result);
 		set_buffer_delay(bh_result);
 	} else if (ret > 0) {
 		bh_result->b_size = (ret << inode->i_blkbits);
Signature · 1 line
tg: (2084a87..) preallocate_corruption (depends on: ext4_lock_group_conversion)
maintainer
#6 lore raw
Quoted 7 lines · Aneesh Kumar K.V

Looking at the source again i guess setting just b_dev is not enough. unmap_underlying_metadata looks at the mapping block number, which we don't have in case on unwritten buffer_head. How about the below patch ? It involve vfs changes. But i guess it is correct with respect to the meaning of BH_New (Disk mapping was newly created by get_block). I guess BH_New implies BH_Mapped.

Argh. So we have multiple problems going on here. One is the original problem, namely that of a partial write into an preallocated block can leave garbage behind in that unitialized block.

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

In theory this could cause problems if we try installing a new bootloader in the filesystem's boot block while there's a delayed writes happening in the background, since we could end up discarding the write to the boot sector. We've lived with this for quite a wihle though.

My concern with making the fs/buffer.c changes is that we need to make sure it doesn't break any of the other filesystems, so that's going to make it hard to try to slip this with 2.6.30-rc4 nearly upon us. (Silly question; why doesn't XFS get caught by this?)

So the question is do we try to fix both bugs with one patch, and very likely have to wait until 2.6.31 before the patch is incorporated? Or do we fix the second bug using an ext4-only fix, with the knowledge that post 2.6.30, we'll need undo most of it and fix it properly with a change that involves fs/buffer.c?

My preference is for the former, unless we belive the 2nd bug is serious enough that we really need to address it ASAP (in which case we have a lot of work ahead of us in terms of coordinating with the
other filesystem developers). What do other folks think?

- Ted

#7 lore raw
Quoted 9 lines · Theodore Tso
Quoted 7 lines · Aneesh Kumar K.V

Looking at the source again i guess setting just b_dev is not enough. unmap_underlying_metadata looks at the mapping block number, which we don't have in case on unwritten buffer_head. How about the below patch ? It involve vfs changes. But i guess it is correct with respect to the meaning of BH_New (Disk mapping was newly created by get_block). I guess BH_New implies BH_Mapped.

Quoted 7 lines

Argh. So we have multiple problems going on here. One is the original problem, namely that of a partial write into an preallocated block can leave garbage behind in that unitialized block.

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

Not just that. On block allocation we are not calling unmap_underlying_metadata(dev, blocknumber) for delayed allocated blocks. That would imply file corruption.

Quoted 18 lines

In theory this could cause problems if we try installing a new bootloader in the filesystem's boot block while there's a delayed writes happening in the background, since we could end up discarding the write to the boot sector. We've lived with this for quite a wihle though.

My concern with making the fs/buffer.c changes is that we need to make sure it doesn't break any of the other filesystems, so that's going to make it hard to try to slip this with 2.6.30-rc4 nearly upon us. (Silly question; why doesn't XFS get caught by this?)

So the question is do we try to fix both bugs with one patch, and very likely have to wait until 2.6.31 before the patch is incorporated? Or do we fix the second bug using an ext4-only fix, with the knowledge that post 2.6.30, we'll need undo most of it and fix it properly with a change that involves fs/buffer.c?

Quoted 4 lines

My preference is for the former, unless we belive the 2nd bug is serious enough that we really need to address it ASAP (in which case we have a lot of work ahead of us in terms of coordinating with the
other filesystem developers). What do other folks think?

The original reported problem is something really easy to reproduce. So i guess if we can have a ext4 local change that would fix the original problem that would be good. Considering that map_bh(bdev, 0) didn't create any issues till now, what we can do is to do a similar update for unwritten_buffer in ext4_da_block_write_prep. That's the v2 version of the patch with the below addition

	bh_result->b_blocknr = 0;

-aneesh

maintainer
#8 lore raw
Quoted 14 lines · Aneesh Kumar K.V
Quoted 13 lines · Theodore Tso
Quoted 7 lines · Aneesh Kumar K.V

Looking at the source again i guess setting just b_dev is not enough. unmap_underlying_metadata looks at the mapping block number, which we don't have in case on unwritten buffer_head. How about the below patch ? It involve vfs changes. But i guess it is correct with respect to the meaning of BH_New (Disk mapping was newly created by get_block). I guess BH_New implies BH_Mapped.

Argh. So we have multiple problems going on here. One is the original problem, namely that of a partial write into an preallocated block can leave garbage behind in that unitialized block.

Quoted 7 lines
Quoted 3 lines

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

Not just that. On block allocation we are not calling unmap_underlying_metadata(dev, blocknumber) for delayed allocated blocks. That would imply file corruption.

I don't think I'm following you . If we write into block that was delayed allocated. Are you saying we might get in trouble of the delayed allocation block is mmap'ed in?

Quoted 7 lines

The original reported problem is something really easy to reproduce. So i guess if we can have a ext4 local change that would fix the original problem that would be good. Considering that map_bh(bdev, 0) didn't create any issues till now, what we can do is to do a similar update for unwritten_buffer in ext4_da_block_write_prep. That's the v2 version of the patch with the below addition

	bh_result->b_blocknr = 0;

OK, I can put togehter a patch to do this. Whatever we do, I think we're going to need a *lot* of testing.

- Ted

#9 lore raw
Quoted 19 lines · Theodore Tso
Quoted 18 lines · Aneesh Kumar K.V
Quoted 16 lines · Theodore Tso
Quoted 7 lines · Aneesh Kumar K.V

Looking at the source again i guess setting just b_dev is not enough. unmap_underlying_metadata looks at the mapping block number, which we don't have in case on unwritten buffer_head. How about the below patch ? It involve vfs changes. But i guess it is correct with respect to the meaning of BH_New (Disk mapping was newly created by get_block). I guess BH_New implies BH_Mapped.

Argh. So we have multiple problems going on here. One is the original problem, namely that of a partial write into an preallocated block can leave garbage behind in that unitialized block.

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

Quoted 7 lines
Quoted 3 lines

Not just that. On block allocation we are not calling unmap_underlying_metadata(dev, blocknumber) for delayed allocated blocks. That would imply file corruption.

I don't think I'm following you . If we write into block that was delayed allocated. Are you saying we might get in trouble of the delayed allocation block is mmap'ed in?

We allocate blocks for delayed buffer during writepage. Now we need to make sure after getting the blocks we drop the old buffer_head mapping that we may have with this particular block attached to the block device. That is done by calling unmap_underlying_metadata. Now the current code doesn't call unmap_underlying_metadata for delayed allocated blocks. That would mean we can see corrupt files if old buffer_head mapping gets synced to disk AFTER we write the new buffer_head mapping.

Quoted 11 lines
Quoted 8 lines

The original reported problem is something really easy to reproduce. So i guess if we can have a ext4 local change that would fix the original problem that would be good. Considering that map_bh(bdev, 0) didn't create any issues till now, what we can do is to do a similar update for unwritten_buffer in ext4_da_block_write_prep. That's the v2 version of the patch with the below addition

	bh_result->b_blocknr = 0;

OK, I can put togehter a patch to do this. Whatever we do, I think we're going to need a *lot* of testing.

I just sent -v3 version

-aneesh

#10 lore raw
Quoted 2 lines · Aneesh Kumar K.V
Quoted 1 line · Theodore Tso

On Tue, Apr 28, 2009 at 10:05:54PM +0530, Aneesh Kumar K.V wrote:

...

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

Not just that. On block allocation we are not calling unmap_underlying_metadata(dev, blocknumber) for delayed allocated blocks. That would imply file corruption.

I don't think I'm following you . If we write into block that was delayed allocated. Are you saying we might get in trouble of the delayed allocation block is mmap'ed in?

Quoted 8 lines

We allocate blocks for delayed buffer during writepage. Now we need to make sure after getting the blocks we drop the old buffer_head mapping that we may have with this particular block attached to the block device. That is done by calling unmap_underlying_metadata. Now the current code doesn't call unmap_underlying_metadata for delayed allocated blocks. That would mean we can see corrupt files if old buffer_head mapping gets synced to disk AFTER we write the new buffer_head mapping.

Talking w/ Aneesh on IRC, I don't see how we can have stray dirty mappings lying around for this block device unless someone is writing directly to the mounted block device, which I don't think is ever considered safe ...

I'm not quite sure what the call to __unmap_underlying_blocks() in mpage_da_map_blocks() is for, I guess?

-Eric

reviewer
#11 lore raw
Quoted 24 lines
Quoted 2 lines · Aneesh Kumar K.V
Quoted 1 line · Theodore Tso

On Tue, Apr 28, 2009 at 10:05:54PM +0530, Aneesh Kumar K.V wrote:

...

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

Not just that. On block allocation we are not calling unmap_underlying_metadata(dev, blocknumber) for delayed allocated blocks. That would imply file corruption.

I don't think I'm following you . If we write into block that was delayed allocated. Are you saying we might get in trouble of the delayed allocation block is mmap'ed in?

Quoted 8 lines

We allocate blocks for delayed buffer during writepage. Now we need to make sure after getting the blocks we drop the old buffer_head mapping that we may have with this particular block attached to the block device. That is done by calling unmap_underlying_metadata. Now the current code doesn't call unmap_underlying_metadata for delayed allocated blocks. That would mean we can see corrupt files if old buffer_head mapping gets synced to disk AFTER we write the new buffer_head mapping.

Quoted 7 lines

Talking w/ Aneesh on IRC, I don't see how we can have stray dirty mappings lying around for this block device unless someone is writing directly to the mounted block device, which I don't think is ever considered safe ...

I'm not quite sure what the call to __unmap_underlying_blocks() in mpage_da_map_blocks() is for, I guess?

For ext3 / ext4 I think we don't need unmap_underlying_blocks() since
before we reallocate a block, we make sure that the transaction freeing the block is committed and clear all dirty bits from freed blocks.
But for more careless filesystems, if they reallocate metadata block
as a data block and don't clear the dirty bit in blockdev mapping, unmap_underlying_blocks() does it for them.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#12 lore raw
Quoted 27 lines · Jan Kara
Quoted 27 lines
Quoted 2 lines · Aneesh Kumar K.V
Quoted 1 line · Theodore Tso

On Tue, Apr 28, 2009 at 10:05:54PM +0530, Aneesh Kumar K.V wrote:

...

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

Not just that. On block allocation we are not calling unmap_underlying_metadata(dev, blocknumber) for delayed allocated blocks. That would imply file corruption.

I don't think I'm following you . If we write into block that was delayed allocated. Are you saying we might get in trouble of the delayed allocation block is mmap'ed in?

Quoted 8 lines

We allocate blocks for delayed buffer during writepage. Now we need to make sure after getting the blocks we drop the old buffer_head mapping that we may have with this particular block attached to the block device. That is done by calling unmap_underlying_metadata. Now the current code doesn't call unmap_underlying_metadata for delayed allocated blocks. That would mean we can see corrupt files if old buffer_head mapping gets synced to disk AFTER we write the new buffer_head mapping.

Talking w/ Aneesh on IRC, I don't see how we can have stray dirty mappings lying around for this block device unless someone is writing directly to the mounted block device, which I don't think is ever considered safe ...

Quoted 8 lines
Quoted 2 lines

I'm not quite sure what the call to __unmap_underlying_blocks() in mpage_da_map_blocks() is for, I guess?

For ext3 / ext4 I think we don't need unmap_underlying_blocks() since
before we reallocate a block, we make sure that the transaction freeing the block is committed and clear all dirty bits from freed blocks.
But for more careless filesystems, if they reallocate metadata block
as a data block and don't clear the dirty bit in blockdev mapping, unmap_underlying_blocks() does it for them.

That's what I thought - so I was wondering why we have specific calls to this in ext4:

mpage_da_map_blocks

	__unmap_underlying_blocks
		for (i = 0; i < blocks; i++)
			unmap_underlying_metadata

-Eric

reviewer
#13 lore raw
Quoted 37 lines · Eric Sandeen
Quoted 27 lines · Jan Kara
Quoted 27 lines
Quoted 2 lines · Aneesh Kumar K.V
Quoted 1 line · Theodore Tso

On Tue, Apr 28, 2009 at 10:05:54PM +0530, Aneesh Kumar K.V wrote:

...

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

Not just that. On block allocation we are not calling unmap_underlying_metadata(dev, blocknumber) for delayed allocated blocks. That would imply file corruption.

I don't think I'm following you . If we write into block that was delayed allocated. Are you saying we might get in trouble of the delayed allocation block is mmap'ed in?

Quoted 8 lines

We allocate blocks for delayed buffer during writepage. Now we need to make sure after getting the blocks we drop the old buffer_head mapping that we may have with this particular block attached to the block device. That is done by calling unmap_underlying_metadata. Now the current code doesn't call unmap_underlying_metadata for delayed allocated blocks. That would mean we can see corrupt files if old buffer_head mapping gets synced to disk AFTER we write the new buffer_head mapping.

Talking w/ Aneesh on IRC, I don't see how we can have stray dirty mappings lying around for this block device unless someone is writing directly to the mounted block device, which I don't think is ever considered safe ...

Quoted 8 lines
Quoted 2 lines

I'm not quite sure what the call to __unmap_underlying_blocks() in mpage_da_map_blocks() is for, I guess?

For ext3 / ext4 I think we don't need unmap_underlying_blocks() since
before we reallocate a block, we make sure that the transaction freeing the block is committed and clear all dirty bits from freed blocks.
But for more careless filesystems, if they reallocate metadata block
as a data block and don't clear the dirty bit in blockdev mapping, unmap_underlying_blocks() does it for them.

Quoted 7 lines

That's what I thought - so I was wondering why we have specific calls to this in ext4:

mpage_da_map_blocks

	__unmap_underlying_blocks
		for (i = 0; i < blocks; i++)
			unmap_underlying_metadata

Hmm, OK. So maybe change it warn on dirty blockdev buffer and if the warning
does not trigger we can believe that our theory is right ;).

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#14 lore raw
In reply to Theodore Tso
Quoted 35 lines · Theodore Tso
Quoted 14 lines · Aneesh Kumar K.V
Quoted 13 lines · Theodore Tso
Quoted 7 lines · Aneesh Kumar K.V

Looking at the source again i guess setting just b_dev is not enough. unmap_underlying_metadata looks at the mapping block number, which we don't have in case on unwritten buffer_head. How about the below patch ? It involve vfs changes. But i guess it is correct with respect to the meaning of BH_New (Disk mapping was newly created by get_block). I guess BH_New implies BH_Mapped.

Argh. So we have multiple problems going on here. One is the original problem, namely that of a partial write into an preallocated block can leave garbage behind in that unitialized block.

Quoted 7 lines
Quoted 3 lines

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

Not just that. On block allocation we are not calling unmap_underlying_metadata(dev, blocknumber) for delayed allocated blocks. That would imply file corruption.

I don't think I'm following you . If we write into block that was delayed allocated. Are you saying we might get in trouble of the delayed allocation block is mmap'ed in?

Quoted 7 lines

The original reported problem is something really easy to reproduce. So i guess if we can have a ext4 local change that would fix the original problem that would be good. Considering that map_bh(bdev, 0) didn't create any issues till now, what we can do is to do a similar update for unwritten_buffer in ext4_da_block_write_prep. That's the v2 version of the patch with the below addition

	bh_result->b_blocknr = 0;
Quoted 4 lines

OK, I can put togehter a patch to do this. Whatever we do, I think we're going to need a *lot* of testing.

- Ted

Aneesh, Eric and I discussed this online today, we find a separate issue, the lookup on the preallocated extent doesn't set the buffer_mapped(), so loop up/write to the same preallocated block multiple times (e.g. write 1 byte at a time, for 10 bytes total) will end up calling ext4_get_blocks_wrap() multiple times.

It seems reasonable to set the buffer mapped for preallocated buffer, with blocknr set to the real mapped block number (rather than faked -1 for the buffer blocknr in the V3 proposed fix for partial write garbage issue), and later reply on unwritten flag to force the writepage()/mpage_da_map_blocks calls get_block() to do the unintialized extent split. But this change seems require more thoughts and heavy auditing, and not as urgency as the data corruption problem.

Mingming

#15 lore raw
In reply to Theodore Tso
Quoted 22 lines · Theodore Tso
Quoted 6 lines · Aneesh Kumar K.V

Looking at the source again i guess setting just b_dev is not enough. unmap_underlying_metadata looks at the mapping block number, which we don't have in case on unwritten buffer_head. How about the below patch ? It involve vfs changes. But i guess it is correct with respect to the meaning of BH_New (Disk mapping was newly created by get_block). I guess BH_New implies BH_Mapped.

Argh. So we have multiple problems going on here. One is the original problem, namely that of a partial write into an preallocated block can leave garbage behind in that unitialized block.

The other problem seems to be in the case of a delayed allocation write, where we return a buffer_head which is marked new, and this causes block_prepare_write() to call unmap_underlying_metadata(dev, 0).

In theory this could cause problems if we try installing a new bootloader in the filesystem's boot block while there's a delayed writes happening in the background, since we could end up discarding the write to the boot sector. We've lived with this for quite a wihle though.

Quoted 4 lines

My concern with making the fs/buffer.c changes is that we need to make sure it doesn't break any of the other filesystems, so that's going to make it hard to try to slip this with 2.6.30-rc4 nearly upon us. (Silly question; why doesn't XFS get caught by this?)

I'm not sure offhand. All xfs does is this in the get_block path:

         * With sub-block writes into unwritten extents we also need to mark
         * the buffer as new so that the unwritten parts of the buffer gets
         * correctly zeroed.
         */
        if (create &&
            ((!buffer_mapped(bh_result) && !buffer_uptodate(bh_result)) ||
             (offset >= i_size_read(inode)) ||
             (iomap.iomap_flags & (IOMAP_NEW|IOMAP_UNWRITTEN))))
                set_buffer_new(bh_result);

so it returns with BH_New as well.

Quoted 5 lines

So the question is do we try to fix both bugs with one patch, and very likely have to wait until 2.6.31 before the patch is incorporated? Or do we fix the second bug using an ext4-only fix, with the knowledge that post 2.6.30, we'll need undo most of it and fix it properly with a change that involves fs/buffer.c?

I have the sense that this might need a bit more digging around, and I finally got stuff out of the way to do so :)

-Eric