Skip to contentSkip to conversation

sync_file_range02: remove the toplimit of write back

RFC PATCH New Li Wang 4 participants 14 messages Dec 18, 2019 – Jan 2, 2020

lore
Share & tools
lore

Li Wang

author
#0 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Test tries to sync a range of $BYTES bytes, and then makes sure that between $BYTES and $BYTES+10% was written to disk. But sometimes, more than $BYTES+10% hit the disk: "Synced 39843840, expected 33554432" so it failed as below.

tst_test.c:1179: INFO: Testing on ext4 sync_file_range02.c:74: FAIL: Sync equals write: Synced 39843840,

                          expected 33554432
  sync_file_range02.c:74: FAIL: Sync inside of write: Synced 18612224,
                          expected 16777216

From FS dev's view:

" The test's assumptions are fundamentally false; it thinks it can look at IO counters (tst_dev_bytes_written) for a disk before and after a system call, and attribute all of the IO seen to the system call that was made - this isn't necessarily correct. Other processes may generate IO in the background.

ext4 defers a lot of IO on a freshly made filesystem to the kernel - for example it will initialize the journal and inode tables after the mount, and this will cause extra IO.

Creating ext4 filesystems with the options: "-E lazy_itable_init=0, lazy_journal_init=0" might help.

Another option would be to raise the threshold. Essentially, the report here is that the test is failing because the filesystem wrote "too much" as a result of the sync. How much is "too much?" ..."

Let's remove the toplimit of write back, and think as long as we synced at least the expected amount, the test passes. The +10% limit seems arbitrary.

---
 testcases/kernel/syscalls/sync_file_range/sync_file_range02.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
testcases/kernel/syscalls/sync_file_range/sync_file_range02.c +1 -2
index eb08143c3..643be14fa 100644
@@ -66,8 +66,7 @@ static void verify_sync_file_range(struct testcase *tc)
 
 	SAFE_CLOSE(fd);
 
-	if ((written >= tc->exp_sync_size) &&
-	    (written <= (tc->exp_sync_size + tc->exp_sync_size/10)))
+	if ((written >= tc->exp_sync_size))
 		tst_res(TPASS, "%s", tc->desc);
 	else
 		tst_res(TFAIL, "%s: Synced %li, expected %li", tc->desc,
Signature · 1 line
2.20.1
#1 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

----- Original Message -----

Quoted 5 lines

" The test's assumptions are fundamentally false; it thinks it can look at IO counters (tst_dev_bytes_written) for a disk before and after a system call, and attribute all of the IO seen to the system call that was made - this isn't necessarily correct. Other processes may generate IO in the background.

We create our own block device, so there shouldn't be other processes writing to it.

Quoted 3 lines

ext4 defers a lot of IO on a freshly made filesystem to the kernel - for example it will initialize the journal and inode tables after the mount

Journal was my guess as well.

Quoted 3 lines

Let's remove the toplimit of write back, and think as long as we synced at least the expected amount, the test passes. The +10% limit seems arbitrary.

I think this is reasonable approach until we find better way to measure what was synced.

#2 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

+ Cyril (who originally proposed upper bound check)

Quoted 9 lines · Jan Stancek

----- Original Message -----

Quoted 5 lines

" The test's assumptions are fundamentally false; it thinks it can look at IO counters (tst_dev_bytes_written) for a disk before and after a system call, and attribute all of the IO seen to the system call that was made - this isn't necessarily correct. Other processes may generate IO in the background.

Quoted 8 lines

We create our own block device, so there shouldn't be other processes writing to it.

Quoted 3 lines

ext4 defers a lot of IO on a freshly made filesystem to the kernel - for example it will initialize the journal and inode tables after the mount

Journal was my guess as well.

To avoid similar scenarios, I suggested to add a "sync()" call just prior to test here [1]. And I couldn't reproduce the failure in 1000-times run with 4.19 kernel.

Also, I think having a "sync()" call becomes more important in case we remove the upper bound otherwise test might not provide reliable results.

[1] http://lists.linux.it/pipermail/ltp/2019-October/014157.html

-Sumit

Quoted 13 lines
Quoted 4 lines

Let's remove the toplimit of write back, and think as long as we synced at least the expected amount, the test passes. The +10% limit seems arbitrary.

I think this is reasonable approach until we find better way to measure what was synced.

--
Mailing list info: https://lists.linux.it/listinfo/ltp

author
#3 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Hi Sumit,

Quoted 19 lines · Sumit Garg

+ Cyril (who originally proposed upper bound check)

Quoted 16 lines · Jan Stancek

----- Original Message -----

Quoted 5 lines

" The test's assumptions are fundamentally false; it thinks it can look at IO counters (tst_dev_bytes_written) for a disk before and after a system call, and attribute all of the IO seen to the system call that was made - this isn't necessarily correct. Other processes may generate IO in the background.

We create our own block device, so there shouldn't be other processes writing to it.

Quoted 3 lines

ext4 defers a lot of IO on a freshly made filesystem to the kernel - for example it will initialize the journal and inode tables after the mount

Quoted 6 lines
Quoted 1 line

Journal was my guess as well.

To avoid similar scenarios, I suggested to add a "sync()" call just prior to test here [1]. And I couldn't reproduce the failure in 1000-times run with 4.19 kernel.

Yes, that makes sense to me.

Quoted 24 lines

Also, I think having a "sync()" call becomes more important in case we remove the upper bound otherwise test might not provide reliable results.

[1] http://lists.linux.it/pipermail/ltp/2019-October/014157.html

-Sumit

Quoted 13 lines
Quoted 4 lines

Let's remove the toplimit of write back, and think as long as we synced at least the expected amount, the test passes. The +10% limit seems arbitrary.

I think this is reasonable approach until we find better way to measure what was synced.

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191219/d21b6bec/attachment.htm>
#4 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Hi Li,

Quoted 24 lines · Li Wang

Hi Sumit,

Quoted 20 lines · Sumit Garg

+ Cyril (who originally proposed upper bound check)

Quoted 16 lines · Jan Stancek

----- Original Message -----

Quoted 5 lines

" The test's assumptions are fundamentally false; it thinks it can look at IO counters (tst_dev_bytes_written) for a disk before and after a system call, and attribute all of the IO seen to the system call that was made - this isn't necessarily correct. Other processes may generate IO in the background.

We create our own block device, so there shouldn't be other processes writing to it.

Quoted 3 lines

ext4 defers a lot of IO on a freshly made filesystem to the kernel - for example it will initialize the journal and inode tables after the mount

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

Journal was my guess as well.

To avoid similar scenarios, I suggested to add a "sync()" call just prior to test here [1]. And I couldn't reproduce the failure in 1000-times run with 4.19 kernel.

Yes, that makes sense to me.

Would you mind to give it a try and check if you could reproduce the failure?

-Sumit

Quoted 29 lines
Quoted 24 lines

Also, I think having a "sync()" call becomes more important in case we remove the upper bound otherwise test might not provide reliable results.

[1] http://lists.linux.it/pipermail/ltp/2019-October/014157.html

-Sumit

Quoted 13 lines
Quoted 4 lines

Let's remove the toplimit of write back, and think as long as we synced at least the expected amount, the test passes. The +10% limit seems arbitrary.

I think this is reasonable approach until we find better way to measure what was synced.

--
Mailing list info: https://lists.linux.it/listinfo/ltp

--
Regards,
Li Wang

author
#5 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Quoted 6 lines · Sumit Garg

...

Quoted 5 lines
Quoted 3 lines

To avoid similar scenarios, I suggested to add a "sync()" call just prior to test here [1]. And I couldn't reproduce the failure in 1000-times run with 4.19 kernel.

Quoted 6 lines
Quoted 2 lines

Yes, that makes sense to me.

Would you mind to give it a try and check if you could reproduce the failure?

Actually, I haven't had one time to reproduce it. This failure was reported by CI jobs and very low frequency occurred. The fix is based on a discussion with FS developer analysis.

But I would have a try with this patch(+ sync()).

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191219/b20347a7/attachment.htm>
author
#6 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Hi,

Quoted 17 lines · Li Wang
Quoted 6 lines · Sumit Garg

...

Quoted 5 lines
Quoted 3 lines

To avoid similar scenarios, I suggested to add a "sync()" call just prior to test here [1]. And I couldn't reproduce the failure in 1000-times run with 4.19 kernel.

Quoted 6 lines
Quoted 2 lines

Yes, that makes sense to me.

Would you mind to give it a try and check if you could reproduce the failure?

Quoted 6 lines

Actually, I haven't had one time to reproduce it. This failure was reported by CI jobs and very low frequency occurred. The fix is based on a discussion with FS developer analysis.

But I would have a try with this patch(+ sync()).

I have tried this for many different systems, and it doesn't hit the failure at least one time. So if no more comments, I would merge the patch as below in the next step.

testcases/kernel/syscalls/sync_file_range/sync_file_range02.c +3 -2
@@ -48,6 +48,8 @@ static void verify_sync_file_range(struct testcase *tc)
 
        lseek(fd, tc->write_off, SEEK_SET);
 
+       sync();
+
        tst_dev_bytes_written(tst_device->dev);
 
        tst_fill_fd(fd, 0, TST_MB, tc->write_size_mb);
@@ -66,8 +68,7 @@ static void verify_sync_file_range(struct testcase *tc)
 
        SAFE_CLOSE(fd);
 
-       if ((written >= tc->exp_sync_size) &&
-           (written <= (tc->exp_sync_size + tc->exp_sync_size/10)))
+       if (written >= tc->exp_sync_size)
                tst_res(TPASS, "%s", tc->desc);
        else
                tst_res(TFAIL, "%s: Synced %li, expected %li", tc->desc,
Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191231/886c1803/attachment-0001.htm>
#7 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Quoted 41 lines · Li Wang

Hi,

Quoted 14 lines · Li Wang
Quoted 10 lines · Sumit Garg

...

Quoted 7 lines
Quoted 3 lines

To avoid similar scenarios, I suggested to add a "sync()" call just prior to test here [1]. And I couldn't reproduce the failure in 1000-times run with 4.19 kernel.

Yes, that makes sense to me.

Quoted 6 lines
Quoted 1 line

Would you mind to give it a try and check if you could reproduce the failure?

Actually, I haven't had one time to reproduce it. This failure was reported by CI jobs and very low frequency occurred. The fix is based on a discussion with FS developer analysis.

But I would have a try with this patch(+ sync()).

I have tried this for many different systems, and it doesn't hit the failure at least one time. So if no more comments, I would merge the patch as below in the next step.

testcases/kernel/syscalls/sync_file_range/sync_file_range02.c +2 -0
@@ -48,6 +48,8 @@ static void verify_sync_file_range(struct testcase *tc)
 
        lseek(fd, tc->write_off, SEEK_SET);
 
+       sync();
+
        tst_dev_bytes_written(tst_device->dev);
 
        tst_fill_fd(fd, 0, TST_MB, tc->write_size_mb);

testcases/kernel/syscalls/sync_file_range/sync_file_range02.c static void verify_sync_file_range(struct testcase *tc) · line 74

Quoted 9 lines
testcases/kernel/syscalls/sync_file_range/sync_file_range02.c +1 -2
@@ -66,8 +68,7 @@ static void verify_sync_file_range(struct testcase *tc)
 
        SAFE_CLOSE(fd);
 
-       if ((written >= tc->exp_sync_size) &&
-           (written <= (tc->exp_sync_size + tc->exp_sync_size/10)))
+       if (written >= tc->exp_sync_size)
                tst_res(TPASS, "%s", tc->desc);
        else
                tst_res(TFAIL, "%s: Synced %li, expected %li", tc->desc,
Quoted 3 lines

--
Regards,
Li Wang

author
#8 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Quoted 4 lines · Sumit Garg

...

Pushed.

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20200102/7be7051c/attachment.htm>
#9 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

In reply to Li Wang

Hi Li

Quoted 28 lines · Li Wang

Hi,

On Thu, Dec 19, 2019 at 5:15 PM Li Wang <[email protected] <mailto:[email protected]>> wrote:

On Thu, Dec 19, 2019 at 4:52 PM Sumit Garg <[email protected] <mailto:[email protected]>> wrote:

...
>> To avoid similar scenarios, I suggested to add a "sync()" call just
>> prior to test here [1]. And I couldn't reproduce the failure in >> 1000-times run with 4.19 kernel.
>
>
> Yes, that makes sense to me.
>

Would you mind to give it a try and check if you could reproduce the failure?

Actually, I haven't had one time to reproduce it. This failure was reported by CI jobs and very low frequency occurred. The fix is based on a discussion?with FS developer analysis.

Quoted 6 lines

But I would have a try with this patch(+ sync()).

I have tried this for many different systems, and it doesn't hit the failure at least one time. So if no more comments, I would merge the patch as below in the next step.

I have a question.we must call sync()? I think syncfs is more accurate.

Best Regards
Yang Xu

Quoted 20 lines
testcases/kernel/syscalls/sync_file_range/sync_file_range02.c +3 -2
@@ -48,6 +48,8 @@ static void verify_sync_file_range(struct testcase *tc)
 
 ? ? ? ? lseek(fd, tc->write_off, SEEK_SET);
 
+ ? ? ? sync();
+
 ? ? ? ? tst_dev_bytes_written(tst_device->dev);
 
 ? ? ? ? tst_fill_fd(fd, 0, TST_MB, tc->write_size_mb);
@@ -66,8 +68,7 @@ static void verify_sync_file_range(struct testcase *tc)
 
 ? ? ? ? SAFE_CLOSE(fd);
 
- ? ? ? if ((written >= tc->exp_sync_size) &&
- ? ? ? ? ? (written <= (tc->exp_sync_size + tc->exp_sync_size/10)))
+ ? ? ? if (written >= tc->exp_sync_size)
 ? ? ? ? ? ? ? ? tst_res(TPASS, "%s", tc->desc);
Quoted 7 lines

? ? ? ? else
? ? ? ? ? ? ? ? tst_res(TFAIL, "%s: Synced %li, expected %li", tc->desc,

Signature · 2 lines
Regards,
Li Wang

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191231/eb79f954/attachment.htm>

author
#10 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Hi Yang,

Quoted 10 lines · Yang Xu

...

I have tried this for many different systems, and it doesn't hit the failure at least one time. So if no more comments, I would merge the patch as below in the next step.

I have a question.we must call sync()? I think syncfs is more accurate.

Here we use sync() is to make fs metadata/cache being written back before testing because there is no obtainable file descriptor 'fd' for the ext4 deferred IO (e.g. initialize the journal and inode tables).

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191231/4dd2b6ca/attachment.htm>
#11 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Hi Li

Quoted 12 lines

Hi Yang,

Yang Xu <[email protected]
<mailto:[email protected]>> wrote:

    ...
Quoted 4 lines

I have tried this for many different systems, and it doesn't hit the failure at least one time. So if no more comments, I would merge the patch as below in the next step.

Quoted 6 lines

I have a question.we must call sync()? I think syncfs is more accurate.

Here we use sync() is to make fs metadata/cache being written back before testing because there is no obtainable?file descriptor 'fd' for the ext4 deferred?IO (e.g. initialize the journal and inode tables).

I see. For other test cases using tst_dev_bytes_written api such as fdatasync03.c, we also need to call sync() to make this written value more accurate. What do you think about it?

Best Regards
Yang Xu

Quoted 4 lines
Signature · 3 lines
-- 
Regards,
Li Wang
author
#12 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Quoted 15 lines · Yang Xu

...

Quoted 5 lines
Quoted 3 lines

I have tried this for many different systems, and it doesn't hit the failure at least one time. So if no more comments, I would merge the patch as below in the next step.

I have a question.we must call sync()? I think syncfs is more

accurate.

Quoted 5 lines

Here we use sync() is to make fs metadata/cache being written back before testing because there is no obtainable file descriptor 'fd' for the ext4 deferred IO (e.g. initialize the journal and inode tables).

Quoted 4 lines

I see. For other test cases using tst_dev_bytes_written api such as fdatasync03.c, we also need to call sync() to make this written value more accurate. What do you think about it?

Good point. I think we should take care of them as we do for sync_file_range02 too. It will more easily report fail in the case of a situation that synced data is less than expected.

$ git grep tst_dev_bytes_written
kernel/syscalls/fdatasync/fdatasync03.c:
 tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/fdatasync/fdatasync03.c:        written =
tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/fsync/fsync04.c:
 tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/fsync/fsync04.c:        written =
tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/sync/sync03.c:  tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/sync/sync03.c:  written =
tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/sync_file_range/sync_file_range02.c:
 tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/sync_file_range/sync_file_range02.c:    written =
tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/syncfs/syncfs01.c:
 tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/syncfs/syncfs01.c:      written =
tst_dev_bytes_written(tst_device->dev);
Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20191231/73f7871b/attachment.htm>
#13 lore raw

RFC PATCH 1/1 sync_file_range02: remove the toplimit of write back

Quoted 26 lines · Li Wang
Quoted 15 lines · Yang Xu

...

Quoted 10 lines
Quoted 3 lines

I have tried this for many different systems, and it doesn't hit the failure at least one time. So if no more comments, I would merge the patch as below in the next step.

I have a question.we must call sync()? I think syncfs is more accurate.

Here we use sync() is to make fs metadata/cache being written back before testing because there is no obtainable file descriptor 'fd' for the ext4 deferred IO (e.g. initialize the journal and inode tables).

Quoted 3 lines

I see. For other test cases using tst_dev_bytes_written api such as fdatasync03.c, we also need to call sync() to make this written value more accurate. What do you think about it?

Good point. I think we should take care of them as we do for sync_file_range02 too. It will more easily report fail in the case of a situation that synced data is less than expected.

Quoted 12 lines
$ git grep tst_dev_bytes_written
kernel/syscalls/fdatasync/fdatasync03.c:        tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/fdatasync/fdatasync03.c:        written = tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/fsync/fsync04.c:        tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/fsync/fsync04.c:        written = tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/sync/sync03.c:  tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/sync/sync03.c:  written = tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/sync_file_range/sync_file_range02.c:    tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/sync_file_range/sync_file_range02.c:    written = tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/syncfs/syncfs01.c:      tst_dev_bytes_written(tst_device->dev);
kernel/syscalls/syncfs/syncfs01.c:      written = tst_dev_bytes_written(tst_device->dev);

Agree.

-Sumit

Quoted 3 lines

--
Regards,
Li Wang