fstests: filter: Introduce filter to filter out offset for xfs_io

PATCH Superseded Qu Wenruo 3 participants 7 messages Jan 11, 2018

lore

Share & tools

lore

Qu Wenruo

author

2018-01-11 06:56 UTC #0 lore raw

PATCH 1/5 fstests: filter: Introduce filter to filter out offset for xfs_io

Some test cases (AFAIK, btrfs RAID recovery test cases) read out certain location to verify its data.

Such read is mostly OK, but the golden output contains the on-disk offset, which can differ due to underlying chunk change. (This time is mkfs chunk layout change for btrfs)

So introduce macro _filter_xfs_io_offset to filter out the offset part wrote 65536/65536 bytes at offset 136708096

                       ^^^^^^^^^^^^^^^^^^^^

And offset from "pread -v"
08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................ ^^^^^^^^^

Reported-by Nikolay Borisov <[email protected]>

Signed-off-by Qu Wenruo <[email protected]>

---
 common/filter | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

common/filter +9 -1

index 9c33efac..77afcbc4 100644
@@ -226,6 +226,15 @@ _filter_xfs_io()
     sed -e "s/[0-9/.]* [GMKiBbytes]*, [0-9]* ops\; [0-9/:. sec]* ([infa0-9/.]* [EPGMKiBbytes]*\/sec and [infa0-9/.]* ops\/sec)/XXX Bytes, X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/"
 }
 
+# Also filter out the offset part of xfs_io output
+# Some test cases may be affected by underlaying extent/chunk layout change,
+# so wipe out this part to avoid golden output difference
+_filter_xfs_io_offset()
+{
+    # filter out " at offset XXX" and offset of "pread -v"
+    sed -e "s/ at offset [0-9]*$//" -e "s/^[0-9a-f]\+:/XXXXXXXX:/"
+}
+
 # stderr filter for xfs_io to handle change of error output format (e.g.
 # pwrite64 -> pwrite).
 _filter_xfs_io_error()

@@ -277,7 +286,6 @@ _filter_xfs_io_pages_modified()
 	_filter_xfs_io_units_modified "Page" $PAGE_SIZE
 }
 
-
 _filter_test_dir()
 {
 	# TEST_DEV may be a prefix of TEST_DIR (e.g. /mnt, /mnt/ovl-mnt)

Signature · 1 line

2.15.1

Qu Wenruo

author

2018-01-11 06:56 UTC #1 lore raw

PATCH 2/5 fstests: btrfs/140: Filter out offset to handle mkfs chunk layout change

Due to recent mkfs.btrfs chunk layout change, offset in golden output is not flex to handle different mkfs chunk layout.

So use _filter_xfs_io_offset to filter out offset so golden output can handle any chunk layout.

Signed-off-by Qu Wenruo <[email protected]>

---
 tests/btrfs/140     |  9 ++++---
 tests/btrfs/140.out | 70 ++++++++++++++++++++++++++---------------------------
 2 files changed, 41 insertions(+), 38 deletions(-)

tests/btrfs/140 +6 -3

index 985327a0..f3ff3500 100755
@@ -78,7 +78,8 @@ _scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
 # chunk
 _scratch_mount -o nospace_cache
 
-$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 # step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
 # one in $SCRATCH_DEV_POOL

@@ -89,7 +90,8 @@ logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | c
 physical_on_scratch=`get_physical ${logical_in_btrfs}`
 
 _scratch_unmount
-$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 _scratch_mount

@@ -109,7 +111,8 @@ done
 _scratch_unmount
 
 # check if the repair works
-$XFS_IO_PROG -d -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
+$XFS_IO_PROG -d -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 _scratch_dev_pool_put
 # success, all done

tests/btrfs/140.out +35 ~~-35~~

74 lines not on this page · raw message

Signature · 1 line

2.15.1

Qu Wenruo

author

2018-01-11 06:56 UTC #2 lore raw

PATCH 3/5 fstests: btrfs/141: Filter out offset to handle mkfs chunk layout change

In reply to Qu Wenruo

Due to recent mkfs.btrfs chunk layout change, offset in golden output is not flex enough to handle different mkfs chunk layout.

So use _filter_xfs_io_offset to filter out offset so golden output can handle any chunk layout.

Signed-off-by Qu Wenruo <[email protected]>

---
 tests/btrfs/141     |  9 ++++---
 tests/btrfs/141.out | 70 ++++++++++++++++++++++++++---------------------------
 2 files changed, 41 insertions(+), 38 deletions(-)

tests/btrfs/141 +6 -3

index 709bb0af..643ce6df 100755
@@ -78,7 +78,8 @@ _scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
 # chunk
 _scratch_mount -o nospace_cache
 
-$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 # step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
 # one in $SCRATCH_DEV_POOL

@@ -89,7 +90,8 @@ logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | c
 physical_on_scratch=`get_physical ${logical_in_btrfs}`
 
 _scratch_unmount
-$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 _scratch_mount

@@ -110,7 +112,8 @@ done
 _scratch_unmount
 
 # check if the repair works
-$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
+$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 _scratch_dev_pool_put
 # success, all done

tests/btrfs/141.out +35 ~~-35~~

74 lines not on this page · raw message

Signature · 1 line

2.15.1

Qu Wenruo

author

2018-01-11 06:56 UTC #3 lore raw

PATCH 4/5 fstests: btrfs/142: Filter out offset to handle mkfs chunk layout change

In reply to Qu Wenruo

Due to recent mkfs.btrfs chunk layout change, offset in golden output is not flex to handle different mkfs chunk layout.

So use _filter_xfs_io_offset to filter out offset so golden output can handle any chunk layout.

Signed-off-by Qu Wenruo <[email protected]>

---
 tests/btrfs/142     |  9 ++++---
 tests/btrfs/142.out | 70 ++++++++++++++++++++++++++---------------------------
 2 files changed, 41 insertions(+), 38 deletions(-)

tests/btrfs/142 +6 -3

index 5bd8b728..267bdfac 100755
@@ -99,7 +99,8 @@ _scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
 # chunk
 _scratch_mount -o nospace_cache,nodatasum
 
-$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 # step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
 # one in $SCRATCH_DEV_POOL

@@ -110,7 +111,8 @@ logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | c
 physical_on_scratch=`get_physical ${logical_in_btrfs}`
 
 _scratch_unmount
-$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 _scratch_mount -o nospace_cache

@@ -135,7 +137,8 @@ stop_fail
 _scratch_unmount
 
 # check if the repair works
-$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
+$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 _scratch_dev_pool_put
 # success, all done

tests/btrfs/142.out +35 ~~-35~~

74 lines not on this page · raw message

Signature · 1 line

2.15.1

Qu Wenruo

author

2018-01-11 06:56 UTC #4 lore raw

PATCH 5/5 fstests: btrfs/143: Filter out offset to handle mkfs chunk layout change

In reply to Qu Wenruo

Due to recent mkfs.btrfs chunk layout change, offset in golden output is not flex to handle different mkfs chunk layout.

So use _filter_xfs_io_offset to filter out offset so golden output can handle any chunk layout.

Signed-off-by Qu Wenruo <[email protected]>

---
 tests/btrfs/143     |  9 ++++---
 tests/btrfs/143.out | 70 ++++++++++++++++++++++++++---------------------------
 2 files changed, 41 insertions(+), 38 deletions(-)

tests/btrfs/143 +6 -3

index 3875b6c4..ca51d35e 100755
@@ -106,7 +106,8 @@ _scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
 # chunk
 _scratch_mount -o nospace_cache,nodatasum
 
-$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 # step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
 # one in $SCRATCH_DEV_POOL

@@ -117,7 +118,8 @@ logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | c
 physical_on_scratch=`get_physical ${logical_in_btrfs}`
 
 _scratch_unmount
-$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 _scratch_mount -o nospace_cache

@@ -142,7 +144,8 @@ done
 _scratch_unmount
 
 # check if the repair works
-$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
+$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV |\
+	_filter_xfs_io | _filter_xfs_io_offset
 
 _scratch_dev_pool_put
 # success, all done

tests/btrfs/143.out +35 ~~-35~~

74 lines not on this page · raw message

Signature · 1 line

2.15.1

Eryu Guan

2018-01-11 07:23 UTC #5 lore raw

In reply to Qu Wenruo

Quoted 27 lines · Qu Wenruo

Some test cases (AFAIK, btrfs RAID recovery test cases) read out certain location to verify its data.
Such read is mostly OK, but the golden output contains the on-disk offset, which can differ due to underlying chunk change. (This time is mkfs chunk layout change for btrfs)
So introduce macro _filter_xfs_io_offset to filter out the offset part wrote 65536/65536 bytes at offset 136708096
                       ^^^^^^^^^^^^^^^^^^^^
And offset from "pread -v"
08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................ ^^^^^^^^^
Reported-by Nikolay Borisov <[email protected]>
Signed-off-by Qu Wenruo <[email protected]>
---
 common/filter | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
common/filter +0 -0
index 9c33efac..77afcbc4 100644
@@ -226,6 +226,15 @@ _filter_xfs_io()
     sed -e "s/[0-9/.]* [GMKiBbytes]*, [0-9]* ops\; [0-9/:. sec]* ([infa0-9/.]* [EPGMKiBbytes]*\/sec and [infa0-9/.]* ops\/sec)/XXX Bytes, X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/"
 }

common/filter _filter_xfs_io() · line 237

Quoted 10 lines

common/filter +9 -0

 
+# Also filter out the offset part of xfs_io output
+# Some test cases may be affected by underlaying extent/chunk layout change,
+# so wipe out this part to avoid golden output difference
+_filter_xfs_io_offset()
+{
+    # filter out " at offset XXX" and offset of "pread -v"
+    sed -e "s/ at offset [0-9]*$//" -e "s/^[0-9a-f]\+:/XXXXXXXX:/"
+}
+

I'd put _filter_xfs_io part of _filter_xfs_io_offset, so the callers don't have to chain the filter themselves, e.g.

- $XFS_IO_PROG ... | _filter_xfs_io | _filter_xfs_io_offset
+ $XFS_IO_PROG ... | _filter_xfs_io_offset

And I perfer folding this 5-patch patchset into a single patch, so the introduction and the usage of the new helper function together provide a complete context in git log.

Thanks,
Eryu

Quoted 18 lines

common/filter +0 -1

 # stderr filter for xfs_io to handle change of error output format (e.g.
 # pwrite64 -> pwrite).
 _filter_xfs_io_error()

@@ -277,7 +286,6 @@ _filter_xfs_io_pages_modified()
 	_filter_xfs_io_units_modified "Page" $PAGE_SIZE
 }
 
-
 _filter_test_dir()
 {
 	# TEST_DEV may be a prefix of TEST_DIR (e.g. /mnt, /mnt/ovl-mnt)

Signature · 6 lines

2.15.1

--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Qu Wenruo

2018-01-11 07:50 UTC #6 lore raw

Quoted 42 lines · Eryu Guan

Quoted 27 lines · Qu Wenruo
Some test cases (AFAIK, btrfs RAID recovery test cases) read out certain location to verify its data.
Such read is mostly OK, but the golden output contains the on-disk offset, which can differ due to underlying chunk change. (This time is mkfs chunk layout change for btrfs)
So introduce macro _filter_xfs_io_offset to filter out the offset part wrote 65536/65536 bytes at offset 136708096
                       ^^^^^^^^^^^^^^^^^^^^
And offset from "pread -v"
08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................ ^^^^^^^^^
Reported-by Nikolay Borisov <[email protected]>
Signed-off-by Qu Wenruo <[email protected]>
---
 common/filter | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
common/filter +0 -0
index 9c33efac..77afcbc4 100644
@@ -226,6 +226,15 @@ _filter_xfs_io()
     sed -e "s/[0-9/.]* [GMKiBbytes]*, [0-9]* ops\; [0-9/:. sec]* ([infa0-9/.]* [EPGMKiBbytes]*\/sec and [infa0-9/.]* ops\/sec)/XXX Bytes, X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/"
 }
common/filter _filter_xfs_io() · line 237
Quoted 10 lines
common/filter +9 -0
 
+# Also filter out the offset part of xfs_io output
+# Some test cases may be affected by underlaying extent/chunk layout change,
+# so wipe out this part to avoid golden output difference
+_filter_xfs_io_offset()
+{
+    # filter out " at offset XXX" and offset of "pread -v"
+    sed -e "s/ at offset [0-9]*$//" -e "s/^[0-9a-f]\+:/XXXXXXXX:/"
+}
+
I'd put _filter_xfs_io part of _filter_xfs_io_offset, so the callers don't have to chain the filter themselves, e.g.

Quoted 6 lines

- $XFS_IO_PROG ... | _filter_xfs_io | _filter_xfs_io_offset
+ $XFS_IO_PROG ... | _filter_xfs_io_offset
And I perfer folding this 5-patch patchset into a single patch, so the introduction and the usage of the new helper function together provide a complete context in git log.

No problem, update on the way.

Thanks,
Qu

Quoted 27 lines

Thanks,
Eryu

Quoted 18 lines

common/filter +0 -1

 # stderr filter for xfs_io to handle change of error output format (e.g.
 # pwrite64 -> pwrite).
 _filter_xfs_io_error()

@@ -277,7 +286,6 @@ _filter_xfs_io_pages_modified()
 	_filter_xfs_io_units_modified "Page" $PAGE_SIZE
 }
 
-
 _filter_test_dir()
 {
 	# TEST_DEV may be a prefix of TEST_DIR (e.g. /mnt, /mnt/ovl-mnt)

Signature · 6 lines

2.15.1

--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html