Skip to contentSkip to conversation

lib: Print in summary also tests not run at all

RFC PATCH In review Petr Vorel 4 participants 15 messages Jan 27, 2022 – Feb 10, 2022 Library code

lore
Share & tools
lore

Petr Vorel

author
#0 lore raw

RFC PATCH 1/1 lib: Print in summary also tests not run at all

We count only tests which TCONF. In case tst_brk() some tests might be skipped without even trying to run them), thus print them.

---
Hi,

probably quite confusing to have "skipped" and "not run". Maybe rename s/skipped/cancelled/ and s/not run/skipped/ ? Or just to increase cound of skipped?

E.g. instead of:

bind05.c:132: TCONF: socket(10, 2, 0) failed: EAFNOSUPPORT (97)

Summary:

passed   8
failed   0
broken   0
skipped  1

warnings 0
not run 5

have
bind05.c:132: TCONF: socket(10, 2, 0) failed: EAFNOSUPPORT (97)

Summary:

passed   8
failed   0
broken   0
skipped  6

warnings 0

Kind regards,
Petr

 lib/tst_test.c | 3 +++
 1 file changed, 3 insertions(+)
lib/tst_test.c +3 -0
index 844756fbd7..e5ea9e6165 100644
@@ -811,6 +811,9 @@ static void do_exit(int ret)
 		fprintf(stderr, "broken   %d\n", results->broken);
 		fprintf(stderr, "skipped  %d\n", results->skipped);
 		fprintf(stderr, "warnings %d\n", results->warnings);
+		fprintf(stderr, "not run  %d\n", tst_test->tcnt -
+				results->passed - results->failed - results->broken -
+				results->skipped - results->warnings);
 	}
 
 	do_cleanup();
Signature · 5 lines
2.34.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

Hi Petr

Quoted 44 lines

We count only tests which TCONF. In case tst_brk() some tests might be skipped without even trying to run them), thus print them.

---
Hi,

probably quite confusing to have "skipped" and "not run". Maybe rename s/skipped/cancelled/ and s/not run/skipped/ ? Or just to increase cound of skipped?

E.g. instead of:

bind05.c:132: TCONF: socket(10, 2, 0) failed: EAFNOSUPPORT (97)

Summary:

passed   8
failed   0
broken   0
skipped  1

warnings 0
not run 5

have
bind05.c:132: TCONF: socket(10, 2, 0) failed: EAFNOSUPPORT (97)

Summary:

passed   8
failed   0
broken   0
skipped  6

warnings 0

Kind regards,
Petr

  lib/tst_test.c | 3 +++
  1 file changed, 3 insertions(+)
lib/tst_test.c +0 -0
index 844756fbd7..e5ea9e6165 100644

lib/tst_test.c static void do_exit(int ret) · line 816

Quoted 6 lines
lib/tst_test.c +3 -0
@@ -811,6 +811,9 @@ static void do_exit(int ret)
  		fprintf(stderr, "broken   %d\n", results->broken);
  		fprintf(stderr, "skipped  %d\n", results->skipped);
  		fprintf(stderr, "warnings %d\n", results->warnings);
+		fprintf(stderr, "not run  %d\n", tst_test->tcnt -
+				results->passed - results->failed - results->broken -
+				results->skipped - results->warnings);

IMO, these values are not necessary related. Refer to quotactl01.c( tst_test->tcnt is less than results->passed)

Also, if you want to add a new tag, you should define the situation for it.

Best Regards
Yang Xu

Quoted 3 lines
lib/tst_test.c +0 -0
  	}
 
  	do_cleanup();
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw
Quoted 12 lines · On Sun, Jan 30, 2022 at 11:44 AM [email protected]…

Hi Petr

Quoted 10 lines

We count only tests which TCONF. In case tst_brk() some tests might be skipped without even trying to run them), thus print them.

---
Hi,

probably quite confusing to have "skipped" and "not run". Maybe rename s/skipped/cancelled/ and s/not run/skipped/ ? Or just to increase cound of skipped?

Both fine to me, I slightly think "not-run" might more precise to describe that.

Quoted 29 lines
Quoted 29 lines

E.g. instead of:

bind05.c:132: TCONF: socket(10, 2, 0) failed: EAFNOSUPPORT (97)

Summary:

passed   8
failed   0
broken   0
skipped  1

warnings 0
not run 5

have
bind05.c:132: TCONF: socket(10, 2, 0) failed: EAFNOSUPPORT (97)

Summary:

passed   8
failed   0
broken   0
skipped  6

warnings 0

Kind regards,
Petr

  lib/tst_test.c | 3 +++
  1 file changed, 3 insertions(+)
Quoted 13 lines

lib/tst_test.c static void do_exit(int ret) · line 816

Quoted 11 lines
lib/tst_test.c +3 -0
index 844756fbd7..e5ea9e6165 100644
@@ -811,6 +811,9 @@ static void do_exit(int ret)
              fprintf(stderr, "broken   %d\n", results->broken);
              fprintf(stderr, "skipped  %d\n", results->skipped);
              fprintf(stderr, "warnings %d\n", results->warnings);
+             fprintf(stderr, "not run  %d\n", tst_test->tcnt -
+                             results->passed - results->failed - results->broken -
+                             results->skipped - results->warnings);

IMO, these values are not necessary related. Refer to quotactl01.c( tst_test->tcnt is less than results->passed)

That's because of the test looping 'tst_variant + 1' times. Maybe we can just multiply it eliminate the distractions?

tst_test->tcnt * (tst_variant + 1)

Quoted 2 lines

Also, if you want to add a new tag, you should define the situation for it.

Theoritically yes, but the problem here is hard to count the not-run numbers. Because some of the test items will never be performed if test return by test environment unmatch. I'm sure we have quite lot of test doing like that.

Quoted 11 lines

Best Regards
Yang Xu

lib/tst_test.c static void do_exit(int ret) · line 819

Quoted 3 lines
lib/tst_test.c +0 -0
      }
 
      do_cleanup();

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 6 lines
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw

Yes,but it still be wrong if we use serval TPASS in sub test case.

Best Regards
Yang Xu

| |
xuyang_jy_0410
|
|
邮箱:[email protected]
|

签名由 网易邮箱大师 定制

On 01/30/2022 14:09, Li Wang wrote:

Quoted 12 lines · On Sun, Jan 30, 2022 at 11:44 AM [email protected]…

Hi Petr

Quoted 10 lines

We count only tests which TCONF. In case tst_brk() some tests might be skipped without even trying to run them), thus print them.

---
Hi,

probably quite confusing to have "skipped" and "not run". Maybe rename s/skipped/cancelled/ and s/not run/skipped/ ? Or just to increase cound of skipped?

Both fine to me, I slightly think "not-run" might more precise to describe that.

Quoted 29 lines
Quoted 29 lines

E.g. instead of:

bind05.c:132: TCONF: socket(10, 2, 0) failed: EAFNOSUPPORT (97)

Summary:

passed   8
failed   0
broken   0
skipped  1

warnings 0
not run 5

have
bind05.c:132: TCONF: socket(10, 2, 0) failed: EAFNOSUPPORT (97)

Summary:

passed   8
failed   0
broken   0
skipped  6

warnings 0

Kind regards,
Petr

  lib/tst_test.c | 3 +++
  1 file changed, 3 insertions(+)
Quoted 13 lines

lib/tst_test.c static void do_exit(int ret) · line 816

Quoted 11 lines
lib/tst_test.c +3 -0
index 844756fbd7..e5ea9e6165 100644
@@ -811,6 +811,9 @@ static void do_exit(int ret)
              fprintf(stderr, "broken   %d\n", results->broken);
              fprintf(stderr, "skipped  %d\n", results->skipped);
              fprintf(stderr, "warnings %d\n", results->warnings);
+             fprintf(stderr, "not run  %d\n", tst_test->tcnt -
+                             results->passed - results->failed - results->broken -
+                             results->skipped - results->warnings);

IMO, these values are not necessary related. Refer to quotactl01.c( tst_test->tcnt is less than results->passed)

That's because of the test looping 'tst_variant + 1' times. Maybe we can just multiply it eliminate the distractions?

tst_test->tcnt * (tst_variant + 1)

Quoted 2 lines

Also, if you want to add a new tag, you should define the situation for it.

Theoritically yes, but the problem here is hard to count the not-run numbers. Because some of the test items will never be performed if test return by test environment unmatch. I'm sure we have quite lot of test doing like that.

Quoted 11 lines

Best Regards
Yang Xu

lib/tst_test.c static void do_exit(int ret) · line 819

Quoted 3 lines
lib/tst_test.c +0 -0
      }
 
      do_cleanup();

--
Mailing list info: https://lists.linux.it/listinfo/ltp

--
Regards,
Li Wang

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw
Quoted 2 lines · xuyang

Yes,but it still be wrong if we use serval TPASS in sub test case.

We can simply avoid that by skipping tst_test->forks_child. But in this case limited the scope of the "not-run" uses. TBH, I haven't come up with a perfect
idea to satisfy all situations.

+ if (tst_test->tcnt && !tst_test->forks_child) {
+ fprintf(stderr, "not-run %d\n", (tst_test->tcnt *
test_variants) -
+ results->passed - results->failed -
results->broken -
+ results->skipped -
results->warnings);
+ }

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#5 lore raw
In reply to [email protected]

...

Quoted 10 lines

lib/tst_test.c static void do_exit(int ret) · line 816

Quoted 8 lines

+++ b/lib/tst_test.c

lib/tst_test.c +3 -0
@@ -811,6 +811,9 @@ static void do_exit(int ret)
  		fprintf(stderr, "broken   %d\n", results->broken);
  		fprintf(stderr, "skipped  %d\n", results->skipped);
  		fprintf(stderr, "warnings %d\n", results->warnings);
+		fprintf(stderr, "not run  %d\n", tst_test->tcnt -
+				results->passed - results->failed - results->broken -
+				results->skipped - results->warnings);

IMO, these values are not necessary related. Refer to quotactl01.c( tst_test->tcnt is less than results->passed)

Ah, correct. Also .test_all means tst_test->tcnt to be 0. So my patch is utterly wrong.

Quoted 1 line

Also, if you want to add a new tag, you should define the situation for it.

What is my point: number of the defined tests does not mean they are all run.

Maybe printing number of the tests in the test (tst_test->tcnt or 1 when tst_test->test_all)?

My point is to have an idea without looking into the code know how many tests have been skipped on certain setup.

Kind regards,
Petr

Quoted 2 lines

Best Regards
Yang Xu

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw

.

Quoted 12 lines · Petr Vorel

...

Quoted 10 lines

Review static void do_exit(int ret) · line 816

Quoted 8 lines

+++ b/lib/tst_test.c

(quoted diff) +3 -0
@@ -811,6 +811,9 @@ static void do_exit(int ret)
            fprintf(stderr, "broken   %d\n", results->broken);
            fprintf(stderr, "skipped  %d\n", results->skipped);
            fprintf(stderr, "warnings %d\n", results->warnings);
+           fprintf(stderr, "not run  %d\n", tst_test->tcnt -
+                           results->passed - results->failed - results->broken -
+                           results->skipped - results->warnings);

IMO, these values are not necessary related. Refer to quotactl01.c( tst_test->tcnt is less than results->passed)

Quoted 4 lines

Ah, correct. Also .test_all means tst_test->tcnt to be 0. So my patch is utterly wrong.

Quoted 1 line

Also, if you want to add a new tag, you should define the situation for it.

That sounds like a complicated method, if we go add a new tag, we have to count the times for each arbitrary 'return' of the test. (Because it does not use TCONF)
This obviously brings more work to the author and is not recommended IMHO.

To strictly, every test should follow the principle of new LTP API, to list all test items in the 'struct tcase{ ... }', and with no tst_test->tcnt defined means that is only one test.

Based on these, I think the total test number of a testcase (with 'tst_test->tcnt' defined) should be:

    (tst_test->tcnt * test_variants)
Quoted 2 lines

What is my point: number of the defined tests does not mean they are all run.

Agree, at least it makes sense for those tests explicitly defined tcnt.

Quoted 6 lines

Maybe printing number of the tests in the test (tst_test->tcnt or 1 when tst_test->test_all)?

My point is to have an idea without looking into the code know how many tests have been skipped on certain setup.

Yes, I understand. So how about this way:

lib/tst_test.c +2 -2
@@ -785,6 +785,8 @@ static void print_failure_hints(void)
        print_failure_hint("known-fail", "hit by known kernel failures", NULL);
 }
 
+static unsigned int test_variants = 1;
+
 static void do_exit(int ret)
 {
        if (results) {
@@ -811,9 +813,11 @@ static void do_exit(int ret)
                fprintf(stderr, "broken   %d\n", results->broken);
                fprintf(stderr, "skipped  %d\n", results->skipped);
                fprintf(stderr, "warnings %d\n", results->warnings);
-               fprintf(stderr, "not run  %d\n", tst_test->tcnt -
-                               results->passed - results->failed -

results->broken -
- results->skipped - results->warnings);
+ if (tst_test->tcnt) {
+ fprintf(stderr, "not-run %d\n",
(tst_test->tcnt * test_variants) -
+ results->passed -
results->failed - results->broken -
+ results->skipped - results->warnings);
+ }

        }

        do_cleanup();
@@ -1529,7 +1533,6 @@ unsigned int tst_variant;
 void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 {
        int ret = 0;

- unsigned int test_variants = 1;

        lib_pid = getpid();
        tst_test = self;

--
Regards,
Li Wang

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#7 lore raw

Hi Li,

...

Quoted 32 lines

Yes, I understand. So how about this way:

lib/tst_test.c +2 -2
@@ -785,6 +785,8 @@ static void print_failure_hints(void)
        print_failure_hint("known-fail", "hit by known kernel failures", NULL);
 }
 
+static unsigned int test_variants = 1;
+
 static void do_exit(int ret)
 {
        if (results) {
@@ -811,9 +813,11 @@ static void do_exit(int ret)
                fprintf(stderr, "broken   %d\n", results->broken);
                fprintf(stderr, "skipped  %d\n", results->skipped);
                fprintf(stderr, "warnings %d\n", results->warnings);
-               fprintf(stderr, "not run  %d\n", tst_test->tcnt -
-                               results->passed - results->failed -

results->broken -

lib/tst_test.c +2 -1
-                               results->skipped - results->warnings);
+               if (tst_test->tcnt) {
+                       fprintf(stderr, "not-run  %d\n",

(tst_test->tcnt * test_variants) -
+ results->passed -
results->failed - results->broken -

lib/tst_test.c +2 -0
+                                       results->skipped - results->warnings);
+               }
        }
 
        do_cleanup();

lib/tst_test.c unsigned int tst_variant; · line 1538

Quoted 7 lines
lib/tst_test.c +0 -1
@@ -1529,7 +1533,6 @@ unsigned int tst_variant;
 void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 {
        int ret = 0;
-       unsigned int test_variants = 1;
 
        lib_pid = getpid();
        tst_test = self;

Yep, ack this one, works well. Will you please send a proper patch?

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#8 lore raw
Quoted 41 lines · Petr Vorel

Hi Li,

...

Quoted 36 lines

Yes, I understand. So how about this way:

lib/tst_test.c +2 -2
@@ -785,6 +785,8 @@ static void print_failure_hints(void)
        print_failure_hint("known-fail", "hit by known kernel failures", NULL);
 }
 
+static unsigned int test_variants = 1;
+
 static void do_exit(int ret)
 {
        if (results) {
@@ -811,9 +813,11 @@ static void do_exit(int ret)
                fprintf(stderr, "broken   %d\n", results->broken);
                fprintf(stderr, "skipped  %d\n", results->skipped);
                fprintf(stderr, "warnings %d\n", results->warnings);
-               fprintf(stderr, "not run  %d\n", tst_test->tcnt -
-                               results->passed - results->failed -

results->broken -

lib/tst_test.c +2 -1
-                               results->skipped - results->warnings);
+               if (tst_test->tcnt) {
+                       fprintf(stderr, "not-run  %d\n",

(tst_test->tcnt * test_variants) -
+ results->passed -
results->failed - results->broken -

lib/tst_test.c +2 -1
+                                       results->skipped - results->warnings);
+               }
        }
 
        do_cleanup();
@@ -1529,7 +1533,6 @@ unsigned int tst_variant;
 void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 {
        int ret = 0;
-       unsigned int test_variants = 1;
Quoted 4 lines

lib/tst_test.c unsigned int tst_variant; · line 1537

Quoted 2 lines
lib/tst_test.c +0 -0
        lib_pid = getpid();
        tst_test = self;

Yep, ack this one, works well. Will you please send a proper patch?

Sure, but I'd hold this until Xu Yang review. In case I have something thoughtless.

Ps. He might reply late since now is on China NewYear holidays.

Signature · 6 lines
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#9 lore raw

Hi Li,

...

Quoted 4 lines
Quoted 1 line

Yep, ack this one, works well. Will you please send a proper patch?

Sure, but I'd hold this until Xu Yang review. In case I have something thoughtless.

Sure

Quoted 1 line

Ps. He might reply late since now is on China NewYear holidays.

Enjoy the New Year and your vacation.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#10 lore raw
In reply to Li Wang

Hi Li

Quoted 26 lines · Petr Vorel
Quoted 25 lines

Hi Li,

...

Quoted 21 lines

Yes, I understand. So how about this way:

lib/tst_test.c +2 -2
@@ -785,6 +785,8 @@ static void print_failure_hints(void)
         print_failure_hint("known-fail", "hit by known kernel failures", NULL);
  }
 
+static unsigned int test_variants = 1;
+
  static void do_exit(int ret)
  {
         if (results) {
@@ -811,9 +813,11 @@ static void do_exit(int ret)
                 fprintf(stderr, "broken   %d\n", results->broken);
                 fprintf(stderr, "skipped  %d\n", results->skipped);
                 fprintf(stderr, "warnings %d\n", results->warnings);
-               fprintf(stderr, "not run  %d\n", tst_test->tcnt -
-                               results->passed - results->failed -

results->broken -
- results->skipped - results->warnings);

Quoted 8 lines
Quoted 8 lines

lib/tst_test.c

Quoted 8 lines
lib/tst_test.c +2 -0
+               if (tst_test->tcnt) {
+                       fprintf(stderr, "not-run  %d\n",

(tst_test->tcnt * test_variants) -
+ results->passed -
results->failed - results->broken -

lib/tst_test.c +2 -0
+                                       results->skipped - results->warnings);
+               }
         }

In fact, we don't have mandatory rules that TAPSS or TFAIL only can occur one time. a example ie memcontrol02.c

Best Regards
Yang Xu

Quoted 18 lines
Quoted 11 lines

lib/tst_test.c unsigned int tst_variant; · line 1538

Quoted 9 lines
         do_cleanup();
lib/tst_test.c +0 -1
@@ -1529,7 +1533,6 @@ unsigned int tst_variant;
  void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
  {
         int ret = 0;
-       unsigned int test_variants = 1;
 
         lib_pid = getpid();
         tst_test = self;

Yep, ack this one, works well. Will you please send a proper patch?

Sure, but I'd hold this until Xu Yang review. In case I have something thoughtless.

Ps. He might reply late since now is on China NewYear holidays.

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#11 lore raw
Quoted 3 lines · [email protected] <[email protected]> wrote:

In fact, we don't have mandatory rules that TAPSS or TFAIL only can occur one time. a example ie memcontrol02.c

Right, that is my hesitant part for counting that.
Seems many tests abuse the TPASS|TFAIL for defining test fail bound.

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#12 lore raw

Hi Li, Xu,

Quoted 5 lines
Quoted 2 lines

In fact, we don't have mandatory rules that TAPSS or TFAIL only can occur one time. a example ie memcontrol02.c

Right, that is my hesitant part for counting that.
Seems many tests abuse the TPASS|TFAIL for defining test fail bound.

OK, while it'd be useful for some tests, it'd be confusing due this for other. I guess printing (tst_test->tcnt * test_variants) number can be confusing either.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#13 lore raw
Quoted 5 lines · Petr Vorel

Hi Li, Xu,

Quoted 2 lines
Quoted 2 lines

In fact, we don't have mandatory rules that TAPSS or TFAIL only can occur one time. a example ie memcontrol02.c

Quoted 8 lines
Quoted 2 lines

Right, that is my hesitant part for counting that.
Seems many tests abuse the TPASS|TFAIL for defining test fail bound.

OK, while it'd be useful for some tests, it'd be confusing due this for other.
I guess printing (tst_test->tcnt * test_variants) number can be confusing either.

Agree, so we might need more time coming up with a better solution. Or, we go another way to limit the abuse in TPASS|TFAIL in the test.

But both sound not easy at this moment.

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
author
#14 lore raw
Quoted 16 lines · Petr Vorel
Quoted 6 lines

Hi Li, Xu,

Quoted 2 lines
Quoted 2 lines

In fact, we don't have mandatory rules that TAPSS or TFAIL only can occur one time. a example ie memcontrol02.c

Quoted 7 lines
Quoted 2 lines

Right, that is my hesitant part for counting that.
Seems many tests abuse the TPASS|TFAIL for defining test fail bound.

OK, while it'd be useful for some tests, it'd be confusing due this for other.
I guess printing (tst_test->tcnt * test_variants) number can be confusing either.

Quoted 4 lines

Agree, so we might need more time coming up with a better solution. Or, we go another way to limit the abuse in TPASS|TFAIL in the test.

But both sound not easy at this moment.

Given number of the tests we have (and how many of them we need to rewrite) it'd be hard (and probably not worth of doing it).

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp