Skip to contentSkip to conversation

generic/347: Fix sporadic test failures

PATCH In review Jan Kara 2 participants 6 messages Jul 30 – Aug 18

lore
Share & tools
lore

Jan Kara

author
#0 lore raw

PATCH 1/1 generic/347: Fix sporadic test failures

generic/347 was occasionally failing on ext4 in our QA due to ext4 aborting its journal when the filesystem on thinp device was overfilled. I have tracked the problem down to journal checkpointing failing to write a metadata block to its final location due to ENOSPC failure from the thinp device. Modify the test to first preallocate blocks for the files and remount the filesystem which practically makes sure all involved metadata blocks were written and so their further modifications will not fail.

---
 tests/generic/347 | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/generic/347 +11 -1
index 06df0cf9eddc..56538c160392 100755
@@ -38,7 +38,17 @@ _setup_thin()
 
 _workout()
 {
-	# Overfill it by a bit
+	# Preallocate space to avoid failure for metadata writeback
+	for I in `seq 1 500`; do
+		$XFS_IO_PROG -f -c "falloc 0 1M" $SCRATCH_MNT/file$I &>/dev/null
+	done
+
+	# Unmount and check the device to make sure all metadata is written
+	_dmthin_check_fs
+	_dmthin_mount
+
+	# Write the data blocks to force thinp space allocation.
+	# Overfill it by a bit.
 	for I in `seq 1 500`; do
 		$XFS_IO_PROG -f -c "pwrite -W 0 1M" $SCRATCH_MNT/file$I &>/dev/null
 	done
Signature · 1 line
2.51.0
#1 lore raw
Quoted 19 lines · Jan Kara

generic/347 was occasionally failing on ext4 in our QA due to ext4 aborting its journal when the filesystem on thinp device was overfilled. I have tracked the problem down to journal checkpointing failing to write a metadata block to its final location due to ENOSPC failure from the thinp device. Modify the test to first preallocate blocks for the files and remount the filesystem which practically makes sure all involved metadata blocks were written and so their further modifications will not fail.

---
 tests/generic/347 | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/generic/347 +0 -0
index 06df0cf9eddc..56538c160392 100755

tests/generic/347 _setup_thin() · line 43

Quoted 7 lines
tests/generic/347 +3 -1
@@ -38,7 +38,17 @@ _setup_thin()
 
 _workout()
 {
-	# Overfill it by a bit
+	# Preallocate space to avoid failure for metadata writeback
+	for I in `seq 1 500`; do
+		$XFS_IO_PROG -f -c "falloc 0 1M" $SCRATCH_MNT/file$I &>/dev/null

Hi Jan,

Thanks for this fix! Adding fallocate introduces an extra dependency via _require_xfs_io_command "falloc", which will limit some filesystems can run this test.

Additionally, since the underlying storage is a thinp device, I doubt falloc 0 1M actually triggers physical block allocation on the thin pool (correct me if I'm wrong). If so, I suspect this 1M * 500 preallocation might exhaust the 500M BACKING_SIZE prematurely (along with file system metadata overhead), similar to the pwrite loop below.

Do you think changing it to something smaller, like fallocate 0 64k, would be better to populate the metadata structures without prematurely running out of thin pool space?

Thanks,
Zorro

Quoted 14 lines
tests/generic/347 +8 -0
+	done
+
+	# Unmount and check the device to make sure all metadata is written
+	_dmthin_check_fs
+	_dmthin_mount
+
+	# Write the data blocks to force thinp space allocation.
+	# Overfill it by a bit.
 	for I in `seq 1 500`; do
 		$XFS_IO_PROG -f -c "pwrite -W 0 1M" $SCRATCH_MNT/file$I &>/dev/null
 	done
Signature · 1 line
2.51.0
author
#2 lore raw

Hi Zorro!

Quoted 21 lines · Zorro Lang
Quoted 20 lines · Jan Kara

generic/347 was occasionally failing on ext4 in our QA due to ext4 aborting its journal when the filesystem on thinp device was overfilled. I have tracked the problem down to journal checkpointing failing to write a metadata block to its final location due to ENOSPC failure from the thinp device. Modify the test to first preallocate blocks for the files and remount the filesystem which practically makes sure all involved metadata blocks were written and so their further modifications will not fail.

---
 tests/generic/347 | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/generic/347 +0 -0
index 06df0cf9eddc..56538c160392 100755
@@ -38,7 +38,17 @@ _setup_thin()
 
Quoted 10 lines

tests/generic/347 _setup_thin() · line 43

Quoted 6 lines
tests/generic/347 +3 -1
 _workout()
 {
-	# Overfill it by a bit
+	# Preallocate space to avoid failure for metadata writeback
+	for I in `seq 1 500`; do
+		$XFS_IO_PROG -f -c "falloc 0 1M" $SCRATCH_MNT/file$I &>/dev/null

Thanks for this fix! Adding fallocate introduces an extra dependency via _require_xfs_io_command "falloc", which will limit some filesystems can run this test.

I agree the fix has some downsides. OTOH when I was thinking about it I've concluded that filesystems where you realistically care about behavior on thinp storage also do support fallocate. So I don't think this it's a serious test coverage limitation but it's up for discussion.

Quoted 5 lines

Additionally, since the underlying storage is a thinp device, I doubt falloc 0 1M actually triggers physical block allocation on the thin pool (correct me if I'm wrong). If so, I suspect this 1M * 500 preallocation might exhaust the 500M BACKING_SIZE prematurely (along with file system metadata overhead), similar to the pwrite loop below.

We do *not* want fallocate to trigger the data block allocation. It will however modify all the relevant metadata blocks and subsequent unmount will writeout these modified metadata which forces the physical block allocation for the metadata which is what we need. Physical space for data blocks is not reserved from thinp during fallocate so we should not run out of backing device space during this loop.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#3 lore raw
Quoted 35 lines · Jan Kara

Hi Zorro!

Quoted 21 lines · Zorro Lang
Quoted 20 lines · Jan Kara

generic/347 was occasionally failing on ext4 in our QA due to ext4 aborting its journal when the filesystem on thinp device was overfilled. I have tracked the problem down to journal checkpointing failing to write a metadata block to its final location due to ENOSPC failure from the thinp device. Modify the test to first preallocate blocks for the files and remount the filesystem which practically makes sure all involved metadata blocks were written and so their further modifications will not fail.

---
 tests/generic/347 | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/generic/347 +0 -0
index 06df0cf9eddc..56538c160392 100755
@@ -38,7 +38,17 @@ _setup_thin()
 
Quoted 10 lines

tests/generic/347 _setup_thin() · line 43

Quoted 6 lines
tests/generic/347 +3 -1
 _workout()
 {
-	# Overfill it by a bit
+	# Preallocate space to avoid failure for metadata writeback
+	for I in `seq 1 500`; do
+		$XFS_IO_PROG -f -c "falloc 0 1M" $SCRATCH_MNT/file$I &>/dev/null

Thanks for this fix! Adding fallocate introduces an extra dependency via _require_xfs_io_command "falloc", which will limit some filesystems can run this test.

Quoted 4 lines

I agree the fix has some downsides. OTOH when I was thinking about it I've concluded that filesystems where you realistically care about behavior on thinp storage also do support fallocate. So I don't think this it's a serious test coverage limitation but it's up for discussion.

Sure, if we accept this patch, we'll need the _require_xfs_io_command "falloc" restriction. This is a basic test for filesystems on thinp, let's try to get more review points, if there are no objections, we can merge it at first.

Quoted 8 lines
Quoted 6 lines

Additionally, since the underlying storage is a thinp device, I doubt falloc 0 1M actually triggers physical block allocation on the thin pool (correct me if I'm wrong). If so, I suspect this 1M * 500 preallocation might exhaust the 500M BACKING_SIZE prematurely (along with file system metadata overhead), similar to the pwrite loop below.

We do *not* want fallocate to trigger the data block allocation. It will

Sorry, I previously thought that fallocate would trigger actual physical block allocation on thinp under certain filesystems or mount options. However, I just tested it out (including exFAT, and Btrfs with nodatacow/compress=zstd), and found that none of them actually allocate physical space.

Thanks,
Zorro

Quoted 11 lines

however modify all the relevant metadata blocks and subsequent unmount will writeout these modified metadata which forces the physical block allocation for the metadata which is what we need. Physical space for data blocks is not reserved from thinp during fallocate so we should not run out of backing device space during this loop.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Reviewed-by on v1 1/1
#4 lore raw
In reply to Jan Kara
Quoted 11 lines · Jan Kara

generic/347 was occasionally failing on ext4 in our QA due to ext4 aborting its journal when the filesystem on thinp device was overfilled. I have tracked the problem down to journal checkpointing failing to write a metadata block to its final location due to ENOSPC failure from the thinp device. Modify the test to first preallocate blocks for the files and remount the filesystem which practically makes sure all involved metadata blocks were written and so their further modifications will not fail.

---

Since there're no further review comments/objections so far, I'll merge this patch with one small addition: "_require_xfs_io_command falloc".

Quoted 29 lines
 tests/generic/347 | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/generic/347 +11 -1
index 06df0cf9eddc..56538c160392 100755
@@ -38,7 +38,17 @@ _setup_thin()
 
 _workout()
 {
-	# Overfill it by a bit
+	# Preallocate space to avoid failure for metadata writeback
+	for I in `seq 1 500`; do
+		$XFS_IO_PROG -f -c "falloc 0 1M" $SCRATCH_MNT/file$I &>/dev/null
+	done
+
+	# Unmount and check the device to make sure all metadata is written
+	_dmthin_check_fs
+	_dmthin_mount
+
+	# Write the data blocks to force thinp space allocation.
+	# Overfill it by a bit.
 	for I in `seq 1 500`; do
 		$XFS_IO_PROG -f -c "pwrite -W 0 1M" $SCRATCH_MNT/file$I &>/dev/null
 	done
Signature · 1 line
2.51.0
author
#5 lore raw
Quoted 10 lines · Zorro Lang
Quoted 9 lines · Jan Kara

generic/347 was occasionally failing on ext4 in our QA due to ext4 aborting its journal when the filesystem on thinp device was overfilled. I have tracked the problem down to journal checkpointing failing to write a metadata block to its final location due to ENOSPC failure from the thinp device. Modify the test to first preallocate blocks for the files and remount the filesystem which practically makes sure all involved metadata blocks were written and so their further modifications will not fail.

Quoted 7 lines
Quoted 2 lines

---

Since there're no further review comments/objections so far, I'll merge this patch with one small addition: "_require_xfs_io_command falloc".

Thanks!

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR