regression in ext4_ind_remove_space
Subsystems (2)
Hi,
Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/ CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate with FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE flags. This seems to be a regression in ext4_ind_remove_space introduced in 4f579ae7, whereas commit 77ea2a4b passes the following test case.
To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem with CONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.
In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.
Any ideas for bug hunting here? Commit 4f579ae7 completely re-writes ext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things. I'm happy to continue debugging, or run any tests as necessary.
Thanks,
--chris j arges
Quoted 18 lines · Chris J Arges
Hi,
Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/
CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate withFALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZEflags. This seems to be a regression inext4_ind_remove_spaceintroduced in4f579ae7, whereas commit77ea2a4bpasses the following test case.To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem withCONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.
Quoted 7 lines
In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.
Any ideas for bug hunting here? Commit
4f579ae7completely re-writesext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.
Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.
I'll run longer tests on 77ea2a4b as well.
--chris
I'm happy to
Quoted 5 lines
continue debugging, or run any tests as necessary.
Thanks,
--chris j arges
Quoted 29 lines · Chris J Arges
Quoted 18 lines · Chris J Arges
Hi,
Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/
CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate withFALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZEflags. This seems to be a regression inext4_ind_remove_spaceintroduced in4f579ae7, whereas commit77ea2a4bpasses the following test case.To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem withCONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.Quoted 7 lines
In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.
Any ideas for bug hunting here? Commit
4f579ae7completely re-writesext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.
Quoted 5 lines
Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.
I'll run longer tests on
77ea2a4bas well.
--chris
77ea2a4b eventually fails (~20 iterations) but seems to be a bit more resilient. So this does not seem like a regression; but still ext4_ind_remove_space is at the heart of the issue. If I use the following code snippet below, I can run the test case for days (~400 iterations so far) without failure:
fs/ext4/inode.c +0 -0
@@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t
offset, loff_t length)
unsigned int credits;
int ret = 0;
+ /* EXTENTS required */
+ if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
+ return -EOPNOTSUPP;
+
if (!S_ISREG(inode->i_mode))
return -EOPNOTSUPP;
--chris
Quoted 7 lines
I'm happy to
Quoted 5 lines
continue debugging, or run any tests as necessary.
Thanks,
--chris j arges
Quoted 56 lines · Chris J Arges
Date: Sun, 01 Feb 2015 15:46:06 -0600
From: Chris J Arges <[email protected]>
To: Lukas Czerner <[email protected]>, Theodore Ts'o <[email protected]>,
Jan Kara <[email protected]>
Cc: [email protected]
Subject: Re: regression inext4_ind_remove_spaceQuoted 29 lines · Chris J Arges
Quoted 18 lines · Chris J Arges
Hi,
Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/
CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate withFALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZEflags. This seems to be a regression inext4_ind_remove_spaceintroduced in4f579ae7, whereas commit77ea2a4bpasses the following test case.To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem withCONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.Quoted 7 lines
In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.
Any ideas for bug hunting here? Commit
4f579ae7completely re-writesext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.Quoted 5 lines
Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.
I'll run longer tests on
77ea2a4bas well.
--chris
77ea2a4beventually fails (~20 iterations) but seems to be a bit more resilient. So this does not seem like a regression; but stillext4_ind_remove_spaceis at the heart of the issue. If I use the following code snippet below, I can run the test case for days (~400 iterations so far) without failure:fs/ext4/inode.c +0
-0
fs/ext4/inode.c int ext4_punch_hole(struct inode *inode, loff_t · line 3376
Quoted 10 lines
fs/ext4/inode.c +4
-0@@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t offset, loff_t length)unsigned int credits;int ret =0 ; +/* EXTENTS required */ +if (!(ext4_test_inode_flag (inode,EXT4_INODE_EXTENTS ))) +return -EOPNOTSUPP ; +if (!S_ISREG (inode->i_mode))return -EOPNOTSUPP ;
Hi Chris,
unfortunately you're right. I've noticed this recently as well and I am looking into this, however the indirect block handling code is damn complicated (but maybe it's just me) and I was not able to find the root cause of the problem yet.
Btw, this can also be very easily reproduced by running xfstest generic/270 on 1k file system. That way I am able to reproduce the problem in more than 90% runs.
Now since Al Viro was the one who wrote most of the indirect code and since he is much smarter than me, maybe we can kindly ask him to take a look at the ext4_ind_remove_space() to figure out where the problem is ?
Al, could you take a quick look at this ?
Thanks!
-Lukas
Quoted 12 lines
--chris
Quoted 7 lines
I'm happy to
Quoted 5 lines
continue debugging, or run any tests as necessary.
Thanks,
--chris j arges
Quoted 87 lines · Lukáš Czerner
Date: Mon, 2 Feb 2015 13:39:07 +0100 (CET)
From: Lukáš Czerner <[email protected]>
To: Chris J Arges <[email protected]>
Cc: Theodore Ts'o <[email protected]>, Jan Kara <[email protected]>,
[email protected]
Subject: Re: regression inext4_ind_remove_spaceQuoted 56 lines · Chris J Arges
Date: Sun, 01 Feb 2015 15:46:06 -0600
From: Chris J Arges <[email protected]>
To: Lukas Czerner <[email protected]>, Theodore Ts'o <[email protected]>,
Jan Kara <[email protected]>
Cc: [email protected]
Subject: Re: regression inext4_ind_remove_spaceQuoted 29 lines · Chris J Arges
Quoted 18 lines · Chris J Arges
Hi,
Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/
CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate withFALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZEflags. This seems to be a regression inext4_ind_remove_spaceintroduced in4f579ae7, whereas commit77ea2a4bpasses the following test case.To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem withCONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.Quoted 7 lines
In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.
Any ideas for bug hunting here? Commit
4f579ae7completely re-writesext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.Quoted 5 lines
Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.
I'll run longer tests on
77ea2a4bas well.
--chris
77ea2a4beventually fails (~20 iterations) but seems to be a bit more resilient. So this does not seem like a regression; but stillext4_ind_remove_spaceis at the heart of the issue. If I use the following code snippet below, I can run the test case for days (~400 iterations so far) without failure:fs/ext4/inode.c +0
-0fs/ext4/inode.c int ext4_punch_hole(struct inode *inode, loff_t · line 3376
Quoted 10 lines
fs/ext4/inode.c +4
-0@@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t offset, loff_t length)unsigned int credits;int ret =0 ; +/* EXTENTS required */ +if (!(ext4_test_inode_flag (inode,EXT4_INODE_EXTENTS ))) +return -EOPNOTSUPP ; +if (!S_ISREG (inode->i_mode))return -EOPNOTSUPP ;Hi Chris,
unfortunately you're right. I've noticed this recently as well and I am looking into this, however the indirect block handling code is damn complicated (but maybe it's just me) and I was not able to find the root cause of the problem yet.
Btw, this can also be very easily reproduced by running xfstest generic/270 on 1k file system. That way I am able to reproduce the problem in more than 90% runs.
Quoted 6 lines
Now since Al Viro was the one who wrote most of the indirect code and since he is much smarter than me, maybe we can kindly ask him to take a look at the
ext4_ind_remove_space()to figure out where the problem is ?Al, could you take a quick look at this ?
Actually adding Al to CC.
-Lukas
Quoted 21 lines
Thanks!
-LukasQuoted 12 lines
--chris
Quoted 7 lines
I'm happy to
Quoted 5 lines
continue debugging, or run any tests as necessary.
Thanks,
--chris j arges--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 80 lines · Lukáš Czerner
Quoted 78 lines · Lukáš Czerner
Date: Mon, 2 Feb 2015 13:39:07 +0100 (CET)
From: Lukáš Czerner <[email protected]>
To: Chris J Arges <[email protected]>
Cc: Theodore Ts'o <[email protected]>, Jan Kara <[email protected]>,
[email protected]
Subject: Re: regression inext4_ind_remove_spaceQuoted 56 lines · Chris J Arges
Date: Sun, 01 Feb 2015 15:46:06 -0600
From: Chris J Arges <[email protected]>
To: Lukas Czerner <[email protected]>, Theodore Ts'o <[email protected]>,
Jan Kara <[email protected]>
Cc: [email protected]
Subject: Re: regression inext4_ind_remove_spaceQuoted 29 lines · Chris J Arges
Quoted 18 lines · Chris J Arges
Hi,
Users of non-extent ext4 filesystems (ext4 ^extents, or ext3 w/
CONFIG_EXT4_USE_FOR_EXT23=y) can encounter data corruption when using fallocate withFALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZEflags. This seems to be a regression inext4_ind_remove_spaceintroduced in4f579ae7, whereas commit77ea2a4bpasses the following test case.To reproduce this issue do the following:
1) Setup ext4 ^extents, or ext3 filesystem withCONFIG_EXT4_USE_FOR_EXT23=y
2) Create and install a VM using a qcow2 image and store the file on the filesystem
3) Snapshot the image with qemu-img
4) Boot the image and do some disk operations (fio,etc)
5) Shutdown image and delete snapshot
6) Repeat 3-5 until VM no longer boots due to image corruption, generally this takes a few iterations depending on disk operations.Quoted 7 lines
In addition, I've tested this with a single vCPU and single host CPU and the problem persists. Running the same test on ext4 w/ extents exhibits no failures.
Any ideas for bug hunting here? Commit
4f579ae7completely re-writesext4_ind_remove_space. A revert of that commit from master fixes the issue, but I'm unsure if that un-fixes other things.Quoted 5 lines
Well, after a bit more extensive testing a revert of this patch still has failures. This may just be a novel bug.
I'll run longer tests on
77ea2a4bas well.
--chris
77ea2a4beventually fails (~20 iterations) but seems to be a bit more resilient. So this does not seem like a regression; but stillext4_ind_remove_spaceis at the heart of the issue. If I use the following code snippet below, I can run the test case for days (~400 iterations so far) without failure:fs/ext4/inode.c +0
-0fs/ext4/inode.c int ext4_punch_hole(struct inode *inode, loff_t · line 3376
Quoted 10 lines
fs/ext4/inode.c +4
-0@@ -3367,6 +3367,10 @@ int ext4_punch_hole(struct inode *inode, loff_t offset, loff_t length)unsigned int credits;int ret =0 ; +/* EXTENTS required */ +if (!(ext4_test_inode_flag (inode,EXT4_INODE_EXTENTS ))) +return -EOPNOTSUPP ; +if (!S_ISREG (inode->i_mode))return -EOPNOTSUPP ;Hi Chris,
Quoted 8 lines
Quoted 8 lines
unfortunately you're right. I've noticed this recently as well and I am looking into this, however the indirect block handling code is damn complicated (but maybe it's just me) and I was not able to find the root cause of the problem yet.
Btw, this can also be very easily reproduced by running xfstest generic/270 on 1k file system. That way I am able to reproduce the problem in more than 90% runs.
Lukáš,
Test case 270 doesn't seem to fail when I run with ext3 and 1024 byte blocks. Are there other steps I need to do to see this issue? Thanks,
--chris
Quoted 6 lines
Quoted 6 lines
Now since Al Viro was the one who wrote most of the indirect code and since he is much smarter than me, maybe we can kindly ask him to take a look at the
ext4_ind_remove_space()to figure out where the problem is ?
Quoted 26 lines
Quoted 1 line
Al, could you take a quick look at this ?
Actually adding Al to CC.
-Lukas
Quoted 20 lines
Thanks!
-LukasQuoted 12 lines
--chris
Quoted 7 lines
I'm happy to
Quoted 5 lines
continue debugging, or run any tests as necessary.
Thanks,
--chris j arges--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html