Skip to contentSkip to conversation

regression in ext4_ind_remove_space

DISCUSSION Chris J Arges 2 participants 6 messages Jan 30, 2015 – Feb 2, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case.

To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.

In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.

Any ideas for bug hunting here? Commit 4f579ae7 completely re-writes ext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things. I'm happy to continue debugging, or run any tests as necessary.

Thanks,
--chris j arges

#1 lore raw
Quoted 18 lines · Chris J Arges

Hi,

Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case.

To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.

Quoted 7 lines

In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.

Any ideas for bug hunting here? Commit 4f579ae7 completely re-writes ext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.

Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.

I'll run longer tests on 77ea2a4b as well.
--chris

I'm happy to

Quoted 5 lines

continue debugging, or run any tests as necessary.

Thanks,
--chris j arges

#2 lore raw
Quoted 29 lines · Chris J Arges
Quoted 18 lines · Chris J Arges

Hi,

Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case.

To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.

Quoted 7 lines

In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.

Any ideas for bug hunting here? Commit 4f579ae7 completely re-writes ext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.

Quoted 5 lines

Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.

I'll run longer tests on 77ea2a4b as well.
--chris

77ea2a4b eventually fails (~20 iterations) but seems to be a bit more resilient. So this does not seem like a regression; but still ext4_ind_remove_space is at the heart of the issue. If I use the following code snippet below, I can run the test case for days (~400 iterations so far) without failure:

fs/ext4/inode.c +0 -0
index 5653fa4..e14cdfe 100644
@@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t

offset, loff_t length)
unsigned int credits;
int ret = 0;

+ /* EXTENTS required */
+ if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
+ return -EOPNOTSUPP;
+
if (!S_ISREG(inode->i_mode))
return -EOPNOTSUPP;

--chris

Quoted 7 lines

I'm happy to

Quoted 5 lines

continue debugging, or run any tests as necessary.

Thanks,
--chris j arges

#3 lore raw
Quoted 56 lines · Chris J Arges

Date: Sun, 01 Feb 2015 15:46:06 -0600
From: Chris J Arges <[email protected]>
To: Lukas Czerner <[email protected]>, Theodore Ts'o <[email protected]>,
Jan Kara <[email protected]>
Cc: [email protected]
Subject: Re: regression in ext4_ind_remove_space

Quoted 29 lines · Chris J Arges
Quoted 18 lines · Chris J Arges

Hi,

Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case.

To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.

Quoted 7 lines

In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.

Any ideas for bug hunting here? Commit 4f579ae7 completely re-writes ext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.

Quoted 5 lines

Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.

I'll run longer tests on 77ea2a4b as well.
--chris

77ea2a4b eventually fails (~20 iterations) but seems to be a bit more resilient. So this does not seem like a regression; but still ext4_ind_remove_space is at the heart of the issue. If I use the following code snippet below, I can run the test case for days (~400 iterations so far) without failure:

fs/ext4/inode.c +0 -0
index 5653fa4..e14cdfe 100644

fs/ext4/inode.c int ext4_punch_hole(struct inode *inode, loff_t · line 3376

Quoted 10 lines
fs/ext4/inode.c +4 -0
@@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t
 offset, loff_t length)
 	unsigned int credits;
 	int ret = 0;
 
+	/* EXTENTS required */
+	if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
+		return -EOPNOTSUPP;
+
 	if (!S_ISREG(inode->i_mode))
 		return -EOPNOTSUPP;

Hi Chris,

unfortunately you're right. I've noticed this recently as well and I am looking into this, however the indirect block handling code is damn complicated (but maybe it's just me) and I was not able to find the root cause of the problem yet.

Btw, this can also be very easily reproduced by running xfstest generic/270 on 1k file system. That way I am able to reproduce the problem in more than 90% runs.

Now since Al Viro was the one who wrote most of the indirect code and since he is much smarter than me, maybe we can kindly ask him to take a look at the ext4_ind_remove_space() to figure out where the problem is ?

Al, could you take a quick look at this ?

Thanks!
-Lukas

Quoted 12 lines

--chris

Quoted 7 lines

I'm happy to

Quoted 5 lines

continue debugging, or run any tests as necessary.

Thanks,
--chris j arges

#4 lore raw
Quoted 87 lines · Lukáš Czerner

Date: Mon, 2 Feb 2015 13:39:07 +0100 (CET)
From: Lukáš Czerner <[email protected]>
To: Chris J Arges <[email protected]>
Cc: Theodore Ts'o <[email protected]>, Jan Kara <[email protected]>,
[email protected]
Subject: Re: regression in ext4_ind_remove_space

Quoted 56 lines · Chris J Arges

Date: Sun, 01 Feb 2015 15:46:06 -0600
From: Chris J Arges <[email protected]>
To: Lukas Czerner <[email protected]>, Theodore Ts'o <[email protected]>,
Jan Kara <[email protected]>
Cc: [email protected]
Subject: Re: regression in ext4_ind_remove_space

Quoted 29 lines · Chris J Arges
Quoted 18 lines · Chris J Arges

Hi,

Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case.

To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.

Quoted 7 lines

In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.

Any ideas for bug hunting here? Commit 4f579ae7 completely re-writes ext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.

Quoted 5 lines

Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.

I'll run longer tests on 77ea2a4b as well.
--chris

77ea2a4b eventually fails (~20 iterations) but seems to be a bit more resilient. So this does not seem like a regression; but still ext4_ind_remove_space is at the heart of the issue. If I use the following code snippet below, I can run the test case for days (~400 iterations so far) without failure:

fs/ext4/inode.c +0 -0
index 5653fa4..e14cdfe 100644

fs/ext4/inode.c int ext4_punch_hole(struct inode *inode, loff_t · line 3376

Quoted 10 lines
fs/ext4/inode.c +4 -0
@@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t
 offset, loff_t length)
 	unsigned int credits;
 	int ret = 0;
 
+	/* EXTENTS required */
+	if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
+		return -EOPNOTSUPP;
+
 	if (!S_ISREG(inode->i_mode))
 		return -EOPNOTSUPP;

Hi Chris,

unfortunately you're right. I've noticed this recently as well and I am looking into this, however the indirect block handling code is damn complicated (but maybe it's just me) and I was not able to find the root cause of the problem yet.

Btw, this can also be very easily reproduced by running xfstest generic/270 on 1k file system. That way I am able to reproduce the problem in more than 90% runs.

Quoted 6 lines

Now since Al Viro was the one who wrote most of the indirect code and since he is much smarter than me, maybe we can kindly ask him to take a look at the ext4_ind_remove_space() to figure out where the problem is ?

Al, could you take a quick look at this ?

Actually adding Al to CC.

-Lukas

Quoted 21 lines

Thanks!
-Lukas

Quoted 12 lines

--chris

Quoted 7 lines

I'm happy to

Quoted 5 lines

continue debugging, or run any tests as necessary.

Thanks,
--chris j arges

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
Quoted 80 lines · Lukáš Czerner
Quoted 78 lines · Lukáš Czerner

Date: Mon, 2 Feb 2015 13:39:07 +0100 (CET)
From: Lukáš Czerner <[email protected]>
To: Chris J Arges <[email protected]>
Cc: Theodore Ts'o <[email protected]>, Jan Kara <[email protected]>,
[email protected]
Subject: Re: regression in ext4_ind_remove_space

Quoted 56 lines · Chris J Arges

Date: Sun, 01 Feb 2015 15:46:06 -0600
From: Chris J Arges <[email protected]>
To: Lukas Czerner <[email protected]>, Theodore Ts'o <[email protected]>,
Jan Kara <[email protected]>
Cc: [email protected]
Subject: Re: regression in ext4_ind_remove_space

Quoted 29 lines · Chris J Arges
Quoted 18 lines · Chris J Arges

Hi,

Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case.

To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.

Quoted 7 lines

In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.

Any ideas for bug hunting here? Commit 4f579ae7 completely re-writes ext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.

Quoted 5 lines

Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.

I'll run longer tests on 77ea2a4b as well.
--chris

77ea2a4b eventually fails (~20 iterations) but seems to be a bit more resilient. So this does not seem like a regression; but still ext4_ind_remove_space is at the heart of the issue. If I use the following code snippet below, I can run the test case for days (~400 iterations so far) without failure:

fs/ext4/inode.c +0 -0
index 5653fa4..e14cdfe 100644

fs/ext4/inode.c int ext4_punch_hole(struct inode *inode, loff_t · line 3376

Quoted 10 lines
fs/ext4/inode.c +4 -0
@@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t
 offset, loff_t length)
 	unsigned int credits;
 	int ret = 0;
 
+	/* EXTENTS required */
+	if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
+		return -EOPNOTSUPP;
+
 	if (!S_ISREG(inode->i_mode))
 		return -EOPNOTSUPP;

Hi Chris,

Quoted 8 lines
Quoted 8 lines

unfortunately you're right. I've noticed this recently as well and I am looking into this, however the indirect block handling code is damn complicated (but maybe it's just me) and I was not able to find the root cause of the problem yet.

Btw, this can also be very easily reproduced by running xfstest generic/270 on 1k file system. That way I am able to reproduce the problem in more than 90% runs.

Lukáš,
Test case 270 doesn't seem to fail when I run with ext3 and 1024 byte blocks. Are there other steps I need to do to see this issue? Thanks,
--chris

Quoted 6 lines
Quoted 6 lines

Now since Al Viro was the one who wrote most of the indirect code and since he is much smarter than me, maybe we can kindly ask him to take a look at the ext4_ind_remove_space() to figure out where the problem is ?

Quoted 26 lines
Quoted 1 line

Al, could you take a quick look at this ?

Actually adding Al to CC.

-Lukas

Quoted 20 lines

Thanks!
-Lukas

Quoted 12 lines

--chris

Quoted 7 lines

I'm happy to

Quoted 5 lines

continue debugging, or run any tests as necessary.

Thanks,
--chris j arges

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html