Skip to contentSkip to conversation

generic: add a test to ensure that page is properly filled before write

PATCH In review Jeff Layton 3 participants 6 messages Jul 2, 2021 – Jul 7, 2021

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 generic: add a test to ensure that page is properly filled before write

We had a broken optimization in cephfs and netfs lib that could cause part of a page to be improperly zeroed-out when writing to an offset that was beyond the EOF but in an existing page.

Add a simple test that would have caught this.

---
 tests/generic/639     | 39 +++++++++++++++++++++++++++++++++++++++
 tests/generic/639.out |  5 +++++
 2 files changed, 44 insertions(+)
 create mode 100755 tests/generic/639
 create mode 100644 tests/generic/639.out

v3: adapt to new test template

tests/generic/639 new file +39 -0
new file mode 100755
index 000000000000..c30f7644a2fc
@@ -0,0 +1,39 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021, Jeff Layton <[email protected]>
+#
+# FS QA Test No. 639
+#
+# Open a file and write a little data to it. Unmount (to clean out the cache)
+# and then mount again. Then write some data to it beyond the EOF and ensure
+# the result is correct.
+#
+# Prompted by a bug in ceph_write_begin that was fixed by commit 827a746f405d.
+#
+. ./common/preamble
+_begin_fstest auto quick rw
+
+# Import common functions.
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+
+testfile="$TEST_DIR/test_write_begin.$$"
+
+# write some data to file and fsync it out
+$XFS_IO_PROG -f -c "pwrite -q 0 32" $testfile
+
+# cycle the mount to clean out the pagecache
+_test_cycle_mount
+
+# now, write to the file (near the end)
+$XFS_IO_PROG -c "pwrite -q 32 32" $testfile
+
+# dump what we think is in there
+echo "The result should be 64 bytes filled with 0xcd:"
+hexdump -C $testfile
+
+status=0
+exit
tests/generic/639.out new file +5 -0
new file mode 100644
index 000000000000..9bf0bac96864
@@ -0,0 +1,5 @@
+QA output created by 639
+The result should be 64 bytes filled with 0xcd:
+00000000  cd cd cd cd cd cd cd cd  cd cd cd cd cd cd cd cd  |................|
+*
+00000040
Signature · 1 line
2.31.1
#1 lore raw
Quoted 38 lines · Jeff Layton

We had a broken optimization in cephfs and netfs lib that could cause part of a page to be improperly zeroed-out when writing to an offset that was beyond the EOF but in an existing page.

Add a simple test that would have caught this.

---
 tests/generic/639     | 39 +++++++++++++++++++++++++++++++++++++++
 tests/generic/639.out |  5 +++++
 2 files changed, 44 insertions(+)
 create mode 100755 tests/generic/639
 create mode 100644 tests/generic/639.out

v3: adapt to new test template

tests/generic/639 new file +16 -0
new file mode 100755
index 000000000000..c30f7644a2fc
@@ -0,0 +1,39 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021, Jeff Layton <[email protected]>
+#
+# FS QA Test No. 639
+#
+# Open a file and write a little data to it. Unmount (to clean out the cache)
+# and then mount again. Then write some data to it beyond the EOF and ensure
+# the result is correct.
+#
+# Prompted by a bug in ceph_write_begin that was fixed by commit 827a746f405d.
+#
+. ./common/preamble
+_begin_fstest auto quick rw
+
+# Import common functions.

tests/generic/639 line 26

Quoted 10 lines
tests/generic/639 new file +10 -0
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+
+testfile="$TEST_DIR/test_write_begin.$$"
+
+# write some data to file and fsync it out
+$XFS_IO_PROG -f -c "pwrite -q 0 32" $testfile

Looks good to me, if you've used this case to reproduce that issue.

Just one question, the comment says "fsync it out", does that mean we need an explicit "fsync" call (e.g. -c "pwrite -q 0 32" -c "fsync") to reproduce this bug? Or there's an implicit "fsync" somewhere?

Thanks,
Zorro

Quoted 27 lines
tests/generic/639 new file +13 -0
+
+# cycle the mount to clean out the pagecache
+_test_cycle_mount
+
+# now, write to the file (near the end)
+$XFS_IO_PROG -c "pwrite -q 32 32" $testfile
+
+# dump what we think is in there
+echo "The result should be 64 bytes filled with 0xcd:"
+hexdump -C $testfile
+
+status=0
+exit
tests/generic/639.out new file +5 -0
new file mode 100644
index 000000000000..9bf0bac96864
@@ -0,0 +1,5 @@
+QA output created by 639
+The result should be 64 bytes filled with 0xcd:
+00000000  cd cd cd cd cd cd cd cd  cd cd cd cd cd cd cd cd  |................|
+*
+00000040
Signature · 1 line
2.31.1
#2 lore raw
Quoted 52 lines · Zorro Lang
Quoted 38 lines · Jeff Layton

We had a broken optimization in cephfs and netfs lib that could cause part of a page to be improperly zeroed-out when writing to an offset that was beyond the EOF but in an existing page.

Add a simple test that would have caught this.

---
 tests/generic/639     | 39 +++++++++++++++++++++++++++++++++++++++
 tests/generic/639.out |  5 +++++
 2 files changed, 44 insertions(+)
 create mode 100755 tests/generic/639
 create mode 100644 tests/generic/639.out

v3: adapt to new test template

tests/generic/639 new file +16 -0
new file mode 100755
index 000000000000..c30f7644a2fc
@@ -0,0 +1,39 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021, Jeff Layton <[email protected]>
+#
+# FS QA Test No. 639
+#
+# Open a file and write a little data to it. Unmount (to clean out the cache)
+# and then mount again. Then write some data to it beyond the EOF and ensure
+# the result is correct.
+#
+# Prompted by a bug in ceph_write_begin that was fixed by commit 827a746f405d.
+#
+. ./common/preamble
+_begin_fstest auto quick rw
+
+# Import common functions.

tests/generic/639 line 26

Quoted 10 lines
tests/generic/639 new file +10 -0
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+
+testfile="$TEST_DIR/test_write_begin.$$"
+
+# write some data to file and fsync it out
+$XFS_IO_PROG -f -c "pwrite -q 0 32" $testfile

Looks good to me, if you've used this case to reproduce that issue.

Quoted 3 lines

Just one question, the comment says "fsync it out", does that mean we need an explicit "fsync" call (e.g. -c "pwrite -q 0 32" -c "fsync") to reproduce this bug? Or there's an implicit "fsync" somewhere?

Oh, this patch has just been merged, sorry for my late review. You can ignore this message.

Thanks,
Zorro

Quoted 31 lines

Thanks,
Zorro

Quoted 27 lines
tests/generic/639 new file +13 -0
+
+# cycle the mount to clean out the pagecache
+_test_cycle_mount
+
+# now, write to the file (near the end)
+$XFS_IO_PROG -c "pwrite -q 32 32" $testfile
+
+# dump what we think is in there
+echo "The result should be 64 bytes filled with 0xcd:"
+hexdump -C $testfile
+
+status=0
+exit
tests/generic/639.out new file +5 -0
new file mode 100644
index 000000000000..9bf0bac96864
@@ -0,0 +1,5 @@
+QA output created by 639
+The result should be 64 bytes filled with 0xcd:
+00000000  cd cd cd cd cd cd cd cd  cd cd cd cd cd cd cd cd  |................|
+*
+00000040
Signature · 1 line
2.31.1
#3 lore raw
In reply to Zorro Lang
Quoted 52 lines · Zorro Lang
Quoted 38 lines · Jeff Layton

We had a broken optimization in cephfs and netfs lib that could cause part of a page to be improperly zeroed-out when writing to an offset that was beyond the EOF but in an existing page.

Add a simple test that would have caught this.

---
 tests/generic/639     | 39 +++++++++++++++++++++++++++++++++++++++
 tests/generic/639.out |  5 +++++
 2 files changed, 44 insertions(+)
 create mode 100755 tests/generic/639
 create mode 100644 tests/generic/639.out

v3: adapt to new test template

tests/generic/639 new file +16 -0
new file mode 100755
index 000000000000..c30f7644a2fc
@@ -0,0 +1,39 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021, Jeff Layton <[email protected]>
+#
+# FS QA Test No. 639
+#
+# Open a file and write a little data to it. Unmount (to clean out the cache)
+# and then mount again. Then write some data to it beyond the EOF and ensure
+# the result is correct.
+#
+# Prompted by a bug in ceph_write_begin that was fixed by commit 827a746f405d.
+#
+. ./common/preamble
+_begin_fstest auto quick rw
+
+# Import common functions.

tests/generic/639 line 26

Quoted 10 lines
tests/generic/639 new file +10 -0
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+
+testfile="$TEST_DIR/test_write_begin.$$"
+
+# write some data to file and fsync it out
+$XFS_IO_PROG -f -c "pwrite -q 0 32" $testfile

Looks good to me, if you've used this case to reproduce that issue.

Quoted 5 lines

Just one question, the comment says "fsync it out", does that mean we need an explicit "fsync" call (e.g. -c "pwrite -q 0 32" -c "fsync") to reproduce this bug? Or there's an implicit "fsync" somewhere?

No, there is no implicit fsync. This is holdover from copy/paste that I did from another test. The patch is already merged now, but if we expand this test to cover other cases, I'll plan to clean up the comment then.

Thanks,
Jeff

Quoted 29 lines
Quoted 28 lines
tests/generic/639 new file +13 -0
+
+# cycle the mount to clean out the pagecache
+_test_cycle_mount
+
+# now, write to the file (near the end)
+$XFS_IO_PROG -c "pwrite -q 32 32" $testfile
+
+# dump what we think is in there
+echo "The result should be 64 bytes filled with 0xcd:"
+hexdump -C $testfile
+
+status=0
+exit
tests/generic/639.out new file +5 -0
new file mode 100644
index 000000000000..9bf0bac96864
@@ -0,0 +1,5 @@
+QA output created by 639
+The result should be 64 bytes filled with 0xcd:
+00000000  cd cd cd cd cd cd cd cd  cd cd cd cd cd cd cd cd  |................|
+*
+00000040
Signature · 1 line
2.31.1
Signature · 1 line
Jeff Layton <[email protected]>
#4 lore raw
Quoted 63 lines · Jeff Layton
Quoted 52 lines · Zorro Lang
Quoted 38 lines · Jeff Layton

We had a broken optimization in cephfs and netfs lib that could cause part of a page to be improperly zeroed-out when writing to an offset that was beyond the EOF but in an existing page.

Add a simple test that would have caught this.

---
 tests/generic/639     | 39 +++++++++++++++++++++++++++++++++++++++
 tests/generic/639.out |  5 +++++
 2 files changed, 44 insertions(+)
 create mode 100755 tests/generic/639
 create mode 100644 tests/generic/639.out

v3: adapt to new test template

tests/generic/639 new file +16 -0
new file mode 100755
index 000000000000..c30f7644a2fc
@@ -0,0 +1,39 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021, Jeff Layton <[email protected]>
+#
+# FS QA Test No. 639
+#
+# Open a file and write a little data to it. Unmount (to clean out the cache)
+# and then mount again. Then write some data to it beyond the EOF and ensure
+# the result is correct.
+#
+# Prompted by a bug in ceph_write_begin that was fixed by commit 827a746f405d.
+#
+. ./common/preamble
+_begin_fstest auto quick rw
+
+# Import common functions.

tests/generic/639 line 26

Quoted 10 lines
tests/generic/639 new file +10 -0
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+
+testfile="$TEST_DIR/test_write_begin.$$"
+
+# write some data to file and fsync it out
+$XFS_IO_PROG -f -c "pwrite -q 0 32" $testfile

Looks good to me, if you've used this case to reproduce that issue.

Quoted 5 lines

Just one question, the comment says "fsync it out", does that mean we need an explicit "fsync" call (e.g. -c "pwrite -q 0 32" -c "fsync") to reproduce this bug? Or there's an implicit "fsync" somewhere?

No, there is no implicit fsync. This is holdover from copy/paste that I did from another test. The patch is already merged now, but if we expand this test to cover other cases, I'll plan to clean up the comment then.

Quoted 7 lines

Thanks,
Jeff

Quoted 4 lines

tests/generic/639 line 29

Quoted 4 lines
tests/generic/639 new file +3 -0
+
+# cycle the mount to clean out the pagecache
+_test_cycle_mount

I think this cycle mount accounts as an implicit fsync?

Thanks,
Eryu

Quoted 28 lines
Quoted 25 lines
Quoted 24 lines
tests/generic/639 new file +10 -0
+
+# now, write to the file (near the end)
+$XFS_IO_PROG -c "pwrite -q 32 32" $testfile
+
+# dump what we think is in there
+echo "The result should be 64 bytes filled with 0xcd:"
+hexdump -C $testfile
+
+status=0
+exit
tests/generic/639.out new file +5 -0
new file mode 100644
index 000000000000..9bf0bac96864
@@ -0,0 +1,5 @@
+QA output created by 639
+The result should be 64 bytes filled with 0xcd:
+00000000  cd cd cd cd cd cd cd cd  cd cd cd cd cd cd cd cd  |................|
+*
+00000040
Signature · 1 line
2.31.1
Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 68 lines · Eryu Guan
Quoted 67 lines · Jeff Layton
Quoted 52 lines · Zorro Lang
Quoted 38 lines · Jeff Layton

We had a broken optimization in cephfs and netfs lib that could cause part of a page to be improperly zeroed-out when writing to an offset that was beyond the EOF but in an existing page.

Add a simple test that would have caught this.

---
 tests/generic/639     | 39 +++++++++++++++++++++++++++++++++++++++
 tests/generic/639.out |  5 +++++
 2 files changed, 44 insertions(+)
 create mode 100755 tests/generic/639
 create mode 100644 tests/generic/639.out

v3: adapt to new test template

tests/generic/639 new file +16 -0
new file mode 100755
index 000000000000..c30f7644a2fc
@@ -0,0 +1,39 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021, Jeff Layton <[email protected]>
+#
+# FS QA Test No. 639
+#
+# Open a file and write a little data to it. Unmount (to clean out the cache)
+# and then mount again. Then write some data to it beyond the EOF and ensure
+# the result is correct.
+#
+# Prompted by a bug in ceph_write_begin that was fixed by commit 827a746f405d.
+#
+. ./common/preamble
+_begin_fstest auto quick rw
+
+# Import common functions.

tests/generic/639 line 26

Quoted 10 lines
tests/generic/639 new file +10 -0
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+
+testfile="$TEST_DIR/test_write_begin.$$"
+
+# write some data to file and fsync it out
+$XFS_IO_PROG -f -c "pwrite -q 0 32" $testfile

Looks good to me, if you've used this case to reproduce that issue.

Quoted 5 lines

Just one question, the comment says "fsync it out", does that mean we need an explicit "fsync" call (e.g. -c "pwrite -q 0 32" -c "fsync") to reproduce this bug? Or there's an implicit "fsync" somewhere?

No, there is no implicit fsync. This is holdover from copy/paste that I did from another test. The patch is already merged now, but if we expand this test to cover other cases, I'll plan to clean up the comment then.

Thanks,
Jeff

Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

tests/generic/639 line 29

Quoted 3 lines
tests/generic/639 new file +3 -0
+
+# cycle the mount to clean out the pagecache
+_test_cycle_mount

I think this cycle mount accounts as an implicit fsync?

Yes, the data should get synced out naturally during a umount. The comment is still a bit misleading though since it may not happen until this point.

Quoted 28 lines
Quoted 28 lines
Quoted 25 lines
Quoted 24 lines
tests/generic/639 new file +10 -0
+
+# now, write to the file (near the end)
+$XFS_IO_PROG -c "pwrite -q 32 32" $testfile
+
+# dump what we think is in there
+echo "The result should be 64 bytes filled with 0xcd:"
+hexdump -C $testfile
+
+status=0
+exit
tests/generic/639.out new file +5 -0
new file mode 100644
index 000000000000..9bf0bac96864
@@ -0,0 +1,5 @@
+QA output created by 639
+The result should be 64 bytes filled with 0xcd:
+00000000  cd cd cd cd cd cd cd cd  cd cd cd cd cd cd cd cd  |................|
+*
+00000040
Signature · 1 line
2.31.1
Signature · 1 line
Jeff Layton <[email protected]>
Signature · 1 line
Jeff Layton <[email protected]>