Skip to contentSkip to conversation

lib: Add known-fail test tag

PATCH Superseded Petr Vorel 2 participants 6 messages Jan 3, 2022 – Jan 14, 2022 Library code

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Petr Vorel

author
#0 lore raw

PATCH 1/2 lib: Add known-fail test tag

to save test reviewers time.

---
 doc/c-test-api.txt | 1 +
 lib/tst_test.c     | 8 ++++++--
 2 files changed, 7 insertions(+), 2 deletions(-)
doc/c-test-api.txt +1 -0
index 6481531cf2..f8ee4de0d1 100644
@@ -2259,6 +2259,7 @@ struct tst_test test = {
 	.tags = (const struct tst_tag[]) {
 		{"linux-git", "9392a27d88b9"},
 		{"linux-git", "ff002b30181d"},
+		{"known-fail", "ustat() is known to fail with EINVAL on Btrfs"},
 		{"linux-stable-git", "c4a23c852e80"},
 		{"CVE", "2020-29373"},
 		{}
lib/tst_test.c +6 -2
index 23fc0ebf48..901b363704 100644
@@ -762,10 +762,13 @@ static void print_failure_hint(const char *tag, const char *hint,
 				hint_printed = 1;
 				fprintf(stderr, "\n");
 				print_colored("HINT: ");
-				fprintf(stderr, "You _MAY_ be %s, see:\n\n", hint);
+				fprintf(stderr, "You _MAY_ be %s:\n\n", hint);
 			}
 
-			fprintf(stderr, "%s%s\n", url, tags[i].value);
+			if (url)
+				fprintf(stderr, "%s%s\n", url, tags[i].value);
+			else
+				fprintf(stderr, "%s\n", tags[i].value);
 		}
 	}
 }
@@ -778,6 +781,7 @@ static void print_failure_hints(void)
 					   LINUX_STABLE_GIT_URL);
 	print_failure_hint("glibc-git", "missing glibc fixes", GLIBC_GIT_URL);
 	print_failure_hint("CVE", "vulnerable to CVE(s)", CVE_DB_URL);
+	print_failure_hint("known-fail", "hit by known kernel failures", NULL);
 }
 
 static void do_exit(int ret)
Signature · 5 lines
2.34.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#1 lore raw

PATCH 2/2 ustat0{1, 2}: Mark failure on Btrfs with known-fail tag

ustat() has been always broken on Btrfs and it's unlikely to be fixed due ustat() being deprecated (glibc doesn't expose this system call anymore).

---
 testcases/kernel/syscalls/ustat/ustat01.c | 4 ++++
 testcases/kernel/syscalls/ustat/ustat02.c | 4 ++++
 2 files changed, 8 insertions(+)
testcases/kernel/syscalls/ustat/ustat01.c +4 -0
index 0252858beb..e314649e8d 100644
@@ -44,6 +44,10 @@ static void setup(void)
 static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
+	.tags = (const struct tst_tag[]) {
+		{"known-fail", "ustat() is known to fail with EINVAL on Btrfs"},
+		{}
+	}
 };
 #else
 TST_TEST_TCONF("testing ustat requires <sys/ustat.h> or <linux/types.h>");
testcases/kernel/syscalls/ustat/ustat02.c +4 -0
index d08446eaa1..67d2918f83 100644
@@ -63,6 +63,10 @@ static struct tst_test test = {
 	.test = run,
 	.setup = setup,
 	.tcnt = ARRAY_SIZE(tc),
+	.tags = (const struct tst_tag[]) {
+		{"known-fail", "ustat() is known to fail with EINVAL on Btrfs"},
+		{}
+	}
 };
 #else
 TST_TEST_TCONF("testing ustat requires <sys/ustat.h> or <linux/types.h>");
Signature · 5 lines
2.34.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw

Re: [LTP] [PATCH 2/2] ustat0{1, 2}: Mark failure on Btrfs with known-fail tag

Hi!

testcases/kernel/syscalls/ustat/ustat01.c static void setup(void) · line 48

Quoted 8 lines
testcases/kernel/syscalls/ustat/ustat01.c +2 -0
@@ -44,6 +44,10 @@ static void setup(void)
 static struct tst_test test = {
 	.test_all = run,
 	.setup = setup,
+	.tags = (const struct tst_tag[]) {
+		{"known-fail", "ustat() is known to fail with EINVAL on Btrfs"},

^
Maybe we should just remove the "known to fail" from these messages.

Since the code in the library will print:

---
You _MAY_ be hit by known kernel failures:

ustat() is known to fail with EINVAL on Btrfs
---

so the 'known to fail' part is redundant here.

What about:

---
You _MAY_ be hit by known kernel failures:

ustat() fails with EINVAL on Btrfs
---

Other than this I think that this is a good way how to put these kind of hints into the testcases.

Also I do wonder if there is a good URL that would contain more verbose description of the problem but I guess there isn't. Well maybe this one:

https://www.mail-archive.com/[email protected]/msg95697.html

Not sure how to encode that, maybe just add it to the test top level comment?

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#3 lore raw

Re: [LTP] [PATCH 2/2] ustat0{1, 2}: Mark failure on Btrfs with known-fail tag

Hi Cyril,

Quoted 16 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+	.tags = (const struct tst_tag[]) {
+		{"known-fail", "ustat() is known to fail with EINVAL on Btrfs"},

^
Maybe we should just remove the "known to fail" from these messages.

Since the code in the library will print:

---
You _MAY_ be hit by known kernel failures:

ustat() is known to fail with EINVAL on Btrfs
---

so the 'known to fail' part is redundant here.

Quoted 7 lines

What about:

---
You _MAY_ be hit by known kernel failures:

ustat() fails with EINVAL on Btrfs
---

+1

Quoted 8 lines

Other than this I think that this is a good way how to put these kind of hints into the testcases.

Also I do wonder if there is a good URL that would contain more verbose description of the problem but I guess there isn't. Well maybe this one:

https://www.mail-archive.com/[email protected]/msg95697.html

Let's use lore.
https://lore.kernel.org/linux-btrfs/CAB=NE6X2-mbZwVFnKUwjRmTGp3auZFHQXJ1h_YTJ2driUeoR+A@mail.gmail.com/

But I'd put Josef Bacik's reply as he's a maintainer:
https://lore.kernel.org/linux-btrfs/[email protected]/

Quoted 2 lines

Not sure how to encode that, maybe just add it to the test top level comment?

I'd prefer to have it in metadata html/pdf and in known-bug section. Thus I'd prefer:

	.tags = (const struct tst_tag[]) {
		{"known-fail", "ustat() is known to fail with EINVAL on Btrfs, see "
			"https://lore.kernel.org/linux-btrfs/[email protected]/"
		},
		{}

But while this is obviously ok for test output:

$ ./ustat01 -h
...
Tags
----
known-fail: ustat() is known to fail with EINVAL on Btrfs, see https://lore.kernel.org/linux-btrfs/[email protected]/

It gets parsed as new line:

  "ustat01": {
   "tags": [
     [
      "known-fail",
      "ustat() is known to fail with EINVAL on Btrfs, see ",
      "https://lore.kernel.org/linux-btrfs/[email protected]/"
     ]
    ],
   "fname": "testcases/kernel/syscalls/ustat/ustat01.c"
  }

Unless you think it's useful to expect that tag values can have longer input and thus testinfo.pl should use all array items (except the first which is key), I'd go the easy way and put the link only to top level docparse section as you suggested.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw

Re: [LTP] [PATCH 2/2] ustat0{1, 2}: Mark failure on Btrfs with known-fail tag

Hi!

Quoted 30 lines

I'd prefer to have it in metadata html/pdf and in known-bug section. Thus I'd prefer:

	.tags = (const struct tst_tag[]) {
		{"known-fail", "ustat() is known to fail with EINVAL on Btrfs, see "
			"https://lore.kernel.org/linux-btrfs/[email protected]/"
		},
		{}

But while this is obviously ok for test output:

$ ./ustat01 -h
...
Tags
----
known-fail: ustat() is known to fail with EINVAL on Btrfs, see https://lore.kernel.org/linux-btrfs/[email protected]/

It gets parsed as new line:

  "ustat01": {
   "tags": [
     [
      "known-fail",
      "ustat() is known to fail with EINVAL on Btrfs, see ",
      "https://lore.kernel.org/linux-btrfs/[email protected]/"
     ]
    ],
   "fname": "testcases/kernel/syscalls/ustat/ustat01.c"
  }
Quoted 4 lines

Unless you think it's useful to expect that tag values can have longer input and thus testinfo.pl should use all array items (except the first which is key), I'd go the easy way and put the link only to top level docparse section as you suggested.

I guess that the URL in the tag field is okay too. It makes the line slightly longer, but at the same time it's more descriptive.

Anyway, can we please still consider this for the release?

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#5 lore raw

Re: [LTP] [PATCH 2/2] ustat0{1, 2}: Mark failure on Btrfs with known-fail tag

Quoted 36 lines

Hi!

Quoted 30 lines

I'd prefer to have it in metadata html/pdf and in known-bug section. Thus I'd prefer:

	.tags = (const struct tst_tag[]) {
		{"known-fail", "ustat() is known to fail with EINVAL on Btrfs, see "
			"https://lore.kernel.org/linux-btrfs/[email protected]/"
		},
		{}

But while this is obviously ok for test output:

$ ./ustat01 -h
...
Tags
----
known-fail: ustat() is known to fail with EINVAL on Btrfs, see https://lore.kernel.org/linux-btrfs/[email protected]/

It gets parsed as new line:

  "ustat01": {
   "tags": [
     [
      "known-fail",
      "ustat() is known to fail with EINVAL on Btrfs, see ",
      "https://lore.kernel.org/linux-btrfs/[email protected]/"
     ]
    ],
   "fname": "testcases/kernel/syscalls/ustat/ustat01.c"
  }
Quoted 4 lines

Unless you think it's useful to expect that tag values can have longer input and thus testinfo.pl should use all array items (except the first which is key), I'd go the easy way and put the link only to top level docparse section as you suggested.

Quoted 4 lines

I guess that the URL in the tag field is okay too. It makes the line slightly longer, but at the same time it's more descriptive.

Anyway, can we please still consider this for the release?

I'll try to post tonight or on Monday with fix for testinfo.pl which use all array items for value (not just the second).

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp