Skip to contentSkip to conversation

tst_device: add new tst_dev_sync

PATCH New Li Wang 4 participants 17 messages Jan 7, 2020 – Jan 9, 2020

lore
Share & tools
lore

Li Wang

author
#0 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

To follow up commit: 0e2ec72de49ab48c151a3c30a1e6ea0cdaea321c
tst_device: do sync() before reading test block device stat file

Cc Cyril Hrubis <[email protected]>
Cc Sumit Garg <[email protected]>
Cc Yang Xu <[email protected]>
---
 doc/test-writing-guidelines.txt                        |  5 ++++-
 include/tst_device.h                                   | 10 ++++++++++
 testcases/kernel/syscalls/fdatasync/fdatasync03.c      |  3 +--
 testcases/kernel/syscalls/fsync/fsync04.c              |  3 +--
 testcases/kernel/syscalls/sync/sync03.c                |  3 +--
 .../syscalls/sync_file_range/sync_file_range02.c       |  3 +--
 testcases/kernel/syscalls/syncfs/syncfs01.c            |  3 +--
 7 files changed, 19 insertions(+), 11 deletions(-)
doc/test-writing-guidelines.txt +4 -1
index 88f771823..dfe28ef3f 100644
@@ -1072,7 +1072,10 @@ unsigned long tst_dev_bytes_written(const char *dev);
 -------------------------------------------------------------------------------
 
 This function reads test block device stat file (/sys/block/<device>/stat) and
-returns the bytes written since the last invocation of this function.
+returns the bytes written since the last invocation of this function. To avoid
+FS deferred IO metadata/cache interferes the result, we suggest doing "syncfs"
+before the tst_dev_bytes_written first invocation. And an inline function named
+tst_dev_sync is created for that intention.
 
 2.2.16 Formatting a device with a filesystem
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
include/tst_device.h +10 -0
index 56835e712..992d1383d 100644
@@ -68,6 +68,16 @@ int tst_attach_device(const char *dev_path, const char *file_path);
  */
 int tst_detach_device(const char *dev_path);
 
+/*
+ * To avoid FS deferred IO metadata/cache interferes test result, so we do
+ * syncfs simply before the tst_dev_bytes_written invocation. For easy to
+ * use, we create this inline function tst_dev_sync.
+ */
+static inline void tst_dev_sync(int fd)
+{
+	syncfs(fd);
+}
+
 /*
  * Reads test block device stat file and returns the bytes written since the
  * last call of this function.
testcases/kernel/syscalls/fdatasync/fdatasync03.c +1 -2
index 032ac4b58..263175b85 100644
@@ -32,8 +32,7 @@ static void verify_fdatasync(void)
 
 	fd = SAFE_OPEN(FNAME, O_RDWR|O_CREAT, MODE);
 
-	sync();
-
+	tst_dev_sync(fd);
 	tst_dev_bytes_written(tst_device->dev);
 
 	tst_fill_fd(fd, 0, TST_MB, FILE_SIZE_MB);
testcases/kernel/syscalls/fsync/fsync04.c +1 -2
index 3c1f45e94..1e4b8754c 100644
@@ -32,8 +32,7 @@ static void verify_fsync(void)
 
 	fd = SAFE_OPEN(FNAME, O_RDWR|O_CREAT, MODE);
 
-	sync();
-
+	tst_dev_sync(fd);
 	tst_dev_bytes_written(tst_device->dev);
 
 	tst_fill_fd(fd, 0, TST_MB, FILE_SIZE_MB);
testcases/kernel/syscalls/sync/sync03.c +1 -2
index 085ccfdeb..c5c02f877 100644
@@ -32,8 +32,7 @@ static void verify_sync(void)
 
 	fd = SAFE_OPEN(FNAME, O_RDWR|O_CREAT, MODE);
 
-	sync();
-
+	tst_dev_sync(fd);
 	tst_dev_bytes_written(tst_device->dev);
 
 	tst_fill_fd(fd, 0, TST_MB, FILE_SIZE_MB);
testcases/kernel/syscalls/sync_file_range/sync_file_range02.c +1 -2
index 1a6d84c49..64d069e93 100644
@@ -48,8 +48,7 @@ static void verify_sync_file_range(struct testcase *tc)
 
 	lseek(fd, tc->write_off, SEEK_SET);
 
-	sync();
-
+	tst_dev_sync(fd);
 	tst_dev_bytes_written(tst_device->dev);
 
 	tst_fill_fd(fd, 0, TST_MB, tc->write_size_mb);
testcases/kernel/syscalls/syncfs/syncfs01.c +1 -2
index 3cf404450..333726eaa 100644
@@ -33,8 +33,7 @@ static void verify_syncfs(void)
 
 	fd = SAFE_OPEN(FNAME, O_RDWR|O_CREAT, MODE);
 
-	sync();
-
+	tst_dev_sync(fd);
 	tst_dev_bytes_written(tst_device->dev);
 
 	tst_fill_fd(fd, 0, TST_MB, FILE_SIZE_MB);
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi!

doc/test-writing-guidelines.txt unsigned long tst_dev_bytes_written(const char *dev); · line 1076

Quoted 4 lines

This function reads test block device stat file (/sys/block/<device>/stat) and

doc/test-writing-guidelines.txt +2 -1
-returns the bytes written since the last invocation of this function.
+returns the bytes written since the last invocation of this function. To avoid
+FS deferred IO metadata/cache interferes the result, we suggest doing "syncfs"

| interfecence |

doc/test-writing-guidelines.txt unsigned long tst_dev_bytes_written(const char *dev); · line 1078

Quoted 2 lines
doc/test-writing-guidelines.txt +2 -0
+before the tst_dev_bytes_written first invocation. And an inline function named
+tst_dev_sync is created for that intention.

Other than this it looks good to me, thanks for doing this, acked.

Signature · 2 lines
Cyril Hrubis
[email protected]
author
#2 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Quoted 10 lines · Cyril Hrubis

Hi!

Quoted 1 line

This function reads test block device stat file

(/sys/block/<device>/stat) and

Review

Quoted 2 lines
(quoted diff) +1 -1
-returns the bytes written since the last invocation of this function.
+returns the bytes written since the last invocation of this function.

To avoid

Quoted 1 line

+FS deferred IO metadata/cache interferes the result, we suggest doing

"syncfs"
| interfecence |

Thanks.

Quoted 7 lines
Quoted 2 lines

+before the tst_dev_bytes_written first invocation. And an inline

function named

Quoted 1 line

+tst_dev_sync is created for that intention.

Other than this it looks good to me, thanks for doing this, acked.

Pushed.

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20200108/8350ae6e/attachment.htm>
#3 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi Li, Cyril,

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

+before the tst_dev_bytes_written first invocation. And an inline

function named

Quoted 1 line

+tst_dev_sync is created for that intention.

Other than this it looks good to me, thanks for doing this, acked.

Pushed.

This broke build with -Werror=implicit-function-declaration [1]:

../include/tst_device.h:78:2: error: implicit declaration of function 'syncfs'; did you mean 'sync'? [-Werror=implicit-function-declaration]

2770  syncfs(fd);
2771  ^~~~~~
2772  sync

The problem is that syncfs() is guarded with __USE_GNU (in glibc, in musl directly _GNU_SOURCE), so this requires to use _GNU_SOURCE (before including first header).
Because it's in tst_device.h, we effectively need to build with -D_GNU_SOURCE. Is that what we want? Or should we always use tst_syscall(__NR_syncfs, fd) (without conditional check #ifndef HAVE_SYNCFS) ?

Kind regards,
Petr

[1] https://travis-ci.org/linux-test-project/ltp/jobs/634178861

#4 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi Li, Cyril,

Quoted 7 lines

This broke build with -Werror=implicit-function-declaration [1]:

../include/tst_device.h:78:2: error: implicit declaration of function 'syncfs'; did you mean 'sync'? [-Werror=implicit-function-declaration]

2770  syncfs(fd);
2771  ^~~~~~
2772  sync
Quoted 6 lines

The problem is that syncfs() is guarded with __USE_GNU (in glibc, in musl directly _GNU_SOURCE), so this requires to use _GNU_SOURCE (before including first header).
Because it's in tst_device.h, we effectively need to build with -D_GNU_SOURCE. Is that what we want? Or should we always use tst_syscall(__NR_syncfs, fd) (without conditional check #ifndef HAVE_SYNCFS) ?

Or maybe use tst_syscall(__NR_syncfs, fd) in include/tst_device.h and keep include/lapi/syncfs.h for tests (which would need to define _GNU_SOURCE?).

Kind regards,
Petr

Quoted 1 line

[1] https://travis-ci.org/linux-test-project/ltp/jobs/634178861

#5 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

In reply to Petr Vorel

Hi!

Quoted 7 lines

This broke build with -Werror=implicit-function-declaration [1]:

../include/tst_device.h:78:2: error: implicit declaration of function 'syncfs'; did you mean 'sync'? [-Werror=implicit-function-declaration]

2770  syncfs(fd);
2771  ^~~~~~
2772  sync
Quoted 6 lines

The problem is that syncfs() is guarded with __USE_GNU (in glibc, in musl directly _GNU_SOURCE), so this requires to use _GNU_SOURCE (before including first header).
Because it's in tst_device.h, we effectively need to build with -D_GNU_SOURCE. Is that what we want? Or should we always use tst_syscall(__NR_syncfs, fd) (without conditional check #ifndef HAVE_SYNCFS) ?

I guess calling raw syscall in the tst_device.h would be easiest fix.

Signature · 2 lines
Cyril Hrubis
[email protected]
author
#6 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Quoted 9 lines · Cyril Hrubis

Hi!

Quoted 3 lines

This broke build with -Werror=implicit-function-declaration [1]:

../include/tst_device.h:78:2: error: implicit declaration of function

'syncfs'; did you mean 'sync'? [-Werror=implicit-function-declaration]

Quoted 3 lines
2770  syncfs(fd);
2771  ^~~~~~
2772  sync

Thanks for highlight this.

Quoted 4 lines
Quoted 3 lines

The problem is that syncfs() is guarded with __USE_GNU (in glibc, in musl directly _GNU_SOURCE), so this requires to use _GNU_SOURCE (before

including

Quoted 9 lines
Quoted 2 lines

first header).
Because it's in tst_device.h, we effectively need to build with

-D_GNU_SOURCE.

Quoted 1 line

Is that what we want? Or should we always use tst_syscall(__NR_syncfs,

fd)

Quoted 1 line

(without conditional check #ifndef HAVE_SYNCFS) ?

I guess calling raw syscall in the tst_device.h would be easiest fix.

Yes, we can have a try.

Btw I just pushed a simple fix to include unistd.h, it seems not works.

Quoted 5 lines

--
Cyril Hrubis
[email protected]

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20200108/1a77dcb4/attachment.htm>
#7 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi

Quoted 1 line

Btw I just pushed a simple fix to include unistd.h, it seems not works.

You would need to define _GNU_SOURCE at the top of each test that uses it otherwise the definition is not exposed.

Signature · 2 lines
Cyril Hrubis
[email protected]
#8 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi Li, Cyril,

Quoted 5 lines

Hi

Quoted 1 line

Btw I just pushed a simple fix to include unistd.h, it seems not works.

You would need to define _GNU_SOURCE at the top of each test that uses it otherwise the definition is not exposed.

Yep :(. I join to Cyril's vote for raw syscall.
BTW please test it with travis or locally build.sh script (which adds -Werror=implicit-function-declaration).

I guess we should work on travis CI integration so we don't have to push it to travis manually [1].

Kind regards,
Petr

[1] https://github.com/linux-test-project/ltp/issues/599

author
#9 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Quoted 11 lines · Petr Vorel

Hi Li, Cyril,

Quoted 5 lines

Hi

Quoted 1 line

Btw I just pushed a simple fix to include unistd.h, it seems not works.

You would need to define _GNU_SOURCE at the top of each test that uses it otherwise the definition is not exposed.

Yep :(. I join to Cyril's vote for raw syscall.
BTW please test it with travis or locally build.sh script (which adds -Werror=implicit-function-declaration).

I'd go with syscall(__NR_syncfs, fd) in the tst_device.h and define _GNU_SOURCE in relevant test cases.

The reason why not use tst_syscall() is that involves a new compile error of tst_brk, and it can not get rid of errors only via include tst_test.h file.

Quoted 5 lines

I guess we should work on travis CI integration so we don't have to push it to
travis manually [1].

That's a good proposal. I feel so sorry for pushing without a compiling in Travis this time, because I thought the code is good enough but ...

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20200109/b045fbe0/attachment-0001.htm>
#10 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi Li,

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

You would need to define _GNU_SOURCE at the top of each test that uses it otherwise the definition is not exposed.

Yep :(. I join to Cyril's vote for raw syscall.
BTW please test it with travis or locally build.sh script (which adds -Werror=implicit-function-declaration).

Quoted 6 lines

I'd go with syscall(__NR_syncfs, fd) in the tst_device.h and define _GNU_SOURCE in relevant test cases.

The reason why not use tst_syscall() is that involves a new compile error of tst_brk, and it can not get rid of errors only via include tst_test.h file.

+1. (I already tested it and replied to that patch).

I wonder if this is a pattern (try to avoid _GNU_SOURCE code in the library, if needed use syscall() instead).
I'm asking because I still plan to write v2 for library API writing guidelines docs patch.

BTW you didn't update wiki after merging eca0fa3c3.
I did it :) (we need some maintainer docs as well, we keep with Cyril to update wiki and (sometimes) maintain patchwork for other maintainers).
BTW there is git repository for each git on github (our [2]), I maintain it with shell script.

Quoted 7 lines
Quoted 3 lines

I guess we should work on travis CI integration so we don't have to push it to
travis manually [1].

That's a good proposal. I feel so sorry for pushing without a compiling in Travis this time, because I thought the code is good enough but ...

No problem :).

Kind regards,
Petr

[1] https://patchwork.ozlabs.org/patch/1166786/
[2] https://github.com/linux-test-project/ltp.wiki.git

#11 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

In reply to Li Wang

Hi!

Quoted 3 lines

The reason why not use tst_syscall() is that involves a new compile error of tst_brk, and it can not get rid of errors only via include tst_test.h file.

The problem is that the header is used both for old and new test API, so until we convert the rest of the oldlib testcases we can't use anything from the new library there.

We solve that in other places by ifdefs, such as:

#ifdef TST_TEST_H__
	tst_syscall(...);
#else
	ltp_syscall(...);

#endif

Signature · 2 lines
Cyril Hrubis
[email protected]
#12 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi,

Quoted 7 lines
Quoted 3 lines

The reason why not use tst_syscall() is that involves a new compile error of tst_brk, and it can not get rid of errors only via include tst_test.h file.

The problem is that the header is used both for old and new test API, so until we convert the rest of the oldlib testcases we can't use anything from the new library there.

Good point.

Quoted 7 lines

We solve that in other places by ifdefs, such as:

#ifdef TST_TEST_H__
	tst_syscall(...);
#else
	ltp_syscall(...);

#endif

or something like (less #ifdef/#else branches):
#ifdef TST_TEST_H__

# define TST_SYSCALL(...) tst_syscall(##__VA_ARGS__)
#else
# define TST_SYSCALL(...) ltp_syscall(##__VA_ARGS__)
#endif

Kind regards,
Petr

#13 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

In reply to Petr Vorel

Hi Petr

Quoted 5 lines

Hi Li, Cyril,

Quoted 2 lines

Hi

Quoted 1 line

Btw I just pushed a simple fix to include unistd.h, it seems not works.

Quoted 8 lines
Quoted 2 lines

You would need to define _GNU_SOURCE at the top of each test that uses it otherwise the definition is not exposed.

Yep :(. I join to Cyril's vote for raw syscall.
BTW please test it with travis or locally build.sh script (which adds -Werror=implicit-function-declaration).

I guess we should work on travis CI integration so we don't have to push it to travis manually [1].

Do we have a tool or a function that
when this pushing patch complie succeeded, then merge successfully or reject auto because of fail. Like gerrit.
Or, pushing patch compiler on travis CI first, then merged into ltp master.

If not, it seems that we can only try on our own ltp fork(Usually, I do it when I write a new case or cleanup)

Best Regards
Yang Xu

Quoted 6 lines

Kind regards,
Petr

[1] https://github.com/linux-test-project/ltp/issues/599

#14 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi Xu,

Quoted 5 lines
Quoted 2 lines

I guess we should work on travis CI integration so we don't have to push it to travis manually [1].

Do we have a tool or a function that
when this pushing patch complie succeeded, then merge successfully or reject auto because of fail. Like gerrit.

Well, we have patchwork and his REST API [2] (IMHO lightweight than gerrit, but needs to be setup). More info is at #599 [2].

Quoted 4 lines

Or, pushing patch compiler on travis CI first, then merged into ltp master.

If not, it seems that we can only try on our own ltp fork(Usually, I do it when I write a new case or cleanup)

That's what I want to automate. It will probably needs some special github account, but we have it for travis anyway.

Kind regards,
Petr

[1] https://github.com/linux-test-project/ltp/issues/599
[2] https://patchwork.readthedocs.io/en/latest/api/rest/

#15 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi Petr

Quoted 10 lines

Hi Xu,

Quoted 5 lines
Quoted 2 lines

I guess we should work on travis CI integration so we don't have to push it to travis manually [1].

Do we have a tool or a function that
when this pushing patch complie succeeded, then merge successfully or reject auto because of fail. Like gerrit.

Well, we have patchwork and his REST API [2] (IMHO lightweight than gerrit, but needs to be setup). More info is at #599 [2].

Quoted 6 lines
Quoted 4 lines

Or, pushing patch compiler on travis CI first, then merged into ltp master.

If not, it seems that we can only try on our own ltp fork(Usually, I do it when I write a new case or cleanup)

That's what I want to automate. It will probably needs some special github account, but we have it for travis anyway.

Thanks for your info. If this can realize, it will be great.

ps: When I have free time, I will read them in detail.

Best Regards
Yang Xu

Quoted 8 lines

Kind regards,
Petr

[1] https://github.com/linux-test-project/ltp/issues/599
[2] https://patchwork.readthedocs.io/en/latest/api/rest/

#16 lore raw

PATCH 1/1 tst_device: add new tst_dev_sync

Hi Xu,

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

If not, it seems that we can only try on our own ltp fork(Usually, I do it when I write a new case or cleanup)

That's what I want to automate. It will probably needs some special github account, but we have it for travis anyway.

Thanks for your info. If this can realize, it will be great.

ps: When I have free time, I will read them in detail.

Thanks!
IMHO server will be needed to run these scripts.

Kind regards,
Petr