Skip to contentSkip to conversation

fzsync: skip test when avaliable CPUs less than 2

PATCH New Li Wang 5 participants 14 messages Nov 25, 2020 – Nov 30, 2020

lore
Share & tools
lore

Li Wang

author
#0 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:

=== 100% reproducible on a 1cpu guest ===

cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s
../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period ended ../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024, delay_bias = 0 ../../../include/tst_fuzzy_sync.h:318: TINFO: start_a - start_b: { avg = -137522ns, avg_dev = 854248ns, dev_ratio = 6.21 }

../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a  : { avg =  1915ns, avg_dev =   535ns, dev_ratio = 0.28 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b  : { avg =  1885ns, avg_dev =    42ns, dev_ratio = 0.02 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b    : { avg = -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 }
../../../include/tst_fuzzy_sync.h:318: TINFO: spins            : { avg = 554786  , avg_dev =  7355  , dev_ratio = 0.01 }

../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time, requesting exit af_alg07.c:96: TFAIL: fchownat() failed to fail, kernel may be vulnerable

CC Richard Palethorpe <[email protected]>
---
 include/tst_fuzzy_sync.h | 3 +++
 1 file changed, 3 insertions(+)
include/tst_fuzzy_sync.h +3 -0
index 4141f5c64..2e864b312 100644
@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)
 static void tst_fzsync_pair_reset(struct tst_fzsync_pair *pair,
 				  void *(*run_b)(void *))
 {
+	if (get_nprocs() < 2)
+		tst_brk(TCONF, "Fuzzy Sync requires@least two CPUs available");
+
 	tst_fzsync_pair_cleanup(pair);
 
 	tst_init_stat(&pair->diff_ss);
Signature · 1 line
2.21.3
#1 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Hello Li,

Quoted 4 lines · Li Wang

It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:

Most of the tests using FuzzySync do not need true parallism. We were able to reproduce a number of race conditions on a single vCPU. Infact it may actually benefit some races because one thread has to pause to allow the other to run, perhaps creating a huge race window.

Quoted 29 lines

=== 100% reproducible on a 1cpu guest ===

cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s
../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period ended ../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024, delay_bias = 0 ../../../include/tst_fuzzy_sync.h:318: TINFO: start_a - start_b: { avg = -137522ns, avg_dev = 854248ns, dev_ratio = 6.21 }

../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a  : { avg =  1915ns, avg_dev =   535ns, dev_ratio = 0.28 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b  : { avg =  1885ns, avg_dev =    42ns, dev_ratio = 0.02 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b    : { avg = -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 }
../../../include/tst_fuzzy_sync.h:318: TINFO: spins            : { avg = 554786  , avg_dev =  7355  , dev_ratio = 0.01 }

../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time, requesting exit af_alg07.c:96: TFAIL: fchownat() failed to fail, kernel may be vulnerable

CC Richard Palethorpe <[email protected]>
---
 include/tst_fuzzy_sync.h | 3 +++
 1 file changed, 3 insertions(+)
include/tst_fuzzy_sync.h +0 -0
index 4141f5c64..2e864b312 100644

include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 289

Quoted 9 lines
include/tst_fuzzy_sync.h +3 -0
@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)
 static void tst_fzsync_pair_reset(struct tst_fzsync_pair *pair,
 				  void *(*run_b)(void *))
 {
+	if (get_nprocs() < 2)
+		tst_brk(TCONF, "Fuzzy Sync requires at least two CPUs available");
+
 	tst_fzsync_pair_cleanup(pair);
 
 	tst_init_stat(&pair->diff_ss);

Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.

Why not print a warning instead of refusing to run?

Signature · 2 lines
Thank you,
Richard.
#2 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Hi!

Quoted 6 lines

Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.

Why not print a warning instead of refusing to run?

That's not a solution either, warning would end up in a test results as well.

I guess that we can add something as .min_cpus to the tst_test structure and set it for this test?

Signature · 2 lines
Cyril Hrubis
[email protected]
#3 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Quoted 6 lines · Cyril Hrubis

Hi!

Quoted 5 lines

Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.

Quoted 7 lines
Quoted 1 line

Why not print a warning instead of refusing to run?

That's not a solution either, warning would end up in a test results as well.

I guess that we can add something as .min_cpus to the tst_test structure and set it for this test?

+1 for .min_cpus from me.

Signature · 7 lines
Martin Doucha   [email protected]
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic
#4 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

In reply to Richard Palethorpe
Quoted 51 lines · Richard Palethorpe

Hello Li,

Quoted 4 lines · Li Wang

It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:

Most of the tests using FuzzySync do not need true parallism. We were able to reproduce a number of race conditions on a single vCPU. Infact it may actually benefit some races because one thread has to pause to allow the other to run, perhaps creating a huge race window.

Quoted 37 lines

=== 100% reproducible on a 1cpu guest ===

cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s
../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period ended ../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024, delay_bias = 0 ../../../include/tst_fuzzy_sync.h:318: TINFO: start_a - start_b: { avg = -137522ns, avg_dev = 854248ns, dev_ratio = 6.21 }

../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a  : { avg =  1915ns, avg_dev =   535ns, dev_ratio = 0.28 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b  : { avg =  1885ns, avg_dev =    42ns, dev_ratio = 0.02 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b    : { avg = -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 }
../../../include/tst_fuzzy_sync.h:318: TINFO: spins            : { avg = 554786  , avg_dev =  7355  , dev_ratio = 0.01 }

../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time, requesting exit af_alg07.c:96: TFAIL: fchownat() failed to fail, kernel may be vulnerable

CC Richard Palethorpe <[email protected]>
---
 include/tst_fuzzy_sync.h | 3 +++
 1 file changed, 3 insertions(+)
include/tst_fuzzy_sync.h +3 -0
index 4141f5c64..2e864b312 100644
@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)
 static void tst_fzsync_pair_reset(struct tst_fzsync_pair *pair,
 				  void *(*run_b)(void *))
 {
+	if (get_nprocs() < 2)
+		tst_brk(TCONF, "Fuzzy Sync requires at least two CPUs available");
+
 	tst_fzsync_pair_cleanup(pair);
 
Quoted 6 lines

include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 289

Quoted 1 line
include/tst_fuzzy_sync.h +0 -0
 	tst_init_stat(&pair->diff_ss);

Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.

No, af_alg07 requires 2 CPUs, otherwise it'll report false positives. The test will pass only if fchownat() hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.

Signature · 7 lines
Martin Doucha   [email protected]
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic
author
#5 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Quoted 70 lines · Martin Doucha
Quoted 9 lines · Richard Palethorpe

Hello Li,

Quoted 4 lines · Li Wang

It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:

Quoted 15 lines

Most of the tests using FuzzySync do not need true parallism. We were able to reproduce a number of race conditions on a single vCPU. Infact it may actually benefit some races because one thread has to pause to allow the other to run, perhaps creating a huge race window.

Quoted 9 lines

=== 100% reproducible on a 1cpu guest ===

cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s ../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period

ended

Quoted 1 line
Quoted 1 line

../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024, delay_bias =

0

Quoted 1 line
Quoted 1 line

../../../include/tst_fuzzy_sync.h:318: TINFO: start_a - start_b: { avg

= -137522ns, avg_dev = 854248ns, dev_ratio = 6.21 }

Quoted 1 line
Quoted 1 line

../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a : { avg

= 1915ns, avg_dev = 535ns, dev_ratio = 0.28 }

Quoted 1 line
Quoted 1 line

../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b : { avg

= 1885ns, avg_dev = 42ns, dev_ratio = 0.02 }

Quoted 1 line
Quoted 1 line

../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b : { avg

= -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 }

Quoted 1 line
Quoted 1 line

../../../include/tst_fuzzy_sync.h:318: TINFO: spins : { avg

= 554786 , avg_dev = 7355 , dev_ratio = 0.01 }

Quoted 1 line
Quoted 1 line

../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time,

requesting exit

Quoted 1 line
Quoted 1 line

af_alg07.c:96: TFAIL: fchownat() failed to fail, kernel may be

vulnerable

Quoted 7 lines
Quoted 7 lines
CC Richard Palethorpe <[email protected]>
---
 include/tst_fuzzy_sync.h | 3 +++
 1 file changed, 3 insertions(+)
Quoted 10 lines

include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 285

Quoted 10 lines
include/tst_fuzzy_sync.h +2 -0
index 4141f5c64..2e864b312 100644
@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)
 static void tst_fzsync_pair_reset(struct tst_fzsync_pair *pair,
                               void *(*run_b)(void *))
 {
+    if (get_nprocs() < 2)
+            tst_brk(TCONF, "Fuzzy Sync requires at least two CPUs

available");

Quoted 9 lines

include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 289

Quoted 4 lines
include/tst_fuzzy_sync.h +1 -0
+
     tst_fzsync_pair_cleanup(pair);
 
     tst_init_stat(&pair->diff_ss);

Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.

Quoted 6 lines

No, af_alg07 requires 2 CPUs, otherwise it'll report false positives. The test will pass only if fchownat() hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.

+1
Correct, I stand by Martin's point.

And we can avoid adding this patch to FuzzySync lib, but for af_alg07 2cpus is required.
(maybe go with Cyril's suggest to add .min_cpus)

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20201125/630b5666/attachment.htm>
#6 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Hi!

Quoted 3 lines

And we can avoid adding this patch to FuzzySync lib, but for af_alg07 2cpus is required.
(maybe go with Cyril's suggest to add .min_cpus)

I would go for that and we can make use of it in getcwd04.c as well.

Signature · 2 lines
Cyril Hrubis
[email protected]
#7 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

In reply to Martin Doucha

Hello,

Quoted 59 lines · Martin Doucha
Quoted 51 lines · Richard Palethorpe

Hello Li,

Quoted 4 lines · Li Wang

It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:

Most of the tests using FuzzySync do not need true parallism. We were able to reproduce a number of race conditions on a single vCPU. Infact it may actually benefit some races because one thread has to pause to allow the other to run, perhaps creating a huge race window.

Quoted 37 lines

=== 100% reproducible on a 1cpu guest ===

cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s
../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period ended ../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024, delay_bias = 0 ../../../include/tst_fuzzy_sync.h:318: TINFO: start_a - start_b: { avg = -137522ns, avg_dev = 854248ns, dev_ratio = 6.21 }

../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a  : { avg =  1915ns, avg_dev =   535ns, dev_ratio = 0.28 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b  : { avg =  1885ns, avg_dev =    42ns, dev_ratio = 0.02 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b    : { avg = -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 }
../../../include/tst_fuzzy_sync.h:318: TINFO: spins            : { avg = 554786  , avg_dev =  7355  , dev_ratio = 0.01 }

../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time, requesting exit af_alg07.c:96: TFAIL: fchownat() failed to fail, kernel may be vulnerable

CC Richard Palethorpe <[email protected]>
---
 include/tst_fuzzy_sync.h | 3 +++
 1 file changed, 3 insertions(+)
include/tst_fuzzy_sync.h +3 -0
index 4141f5c64..2e864b312 100644
@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)
 static void tst_fzsync_pair_reset(struct tst_fzsync_pair *pair,
 				  void *(*run_b)(void *))
 {
+	if (get_nprocs() < 2)
+		tst_brk(TCONF, "Fuzzy Sync requires at least two CPUs available");
+
 	tst_fzsync_pair_cleanup(pair);
 
Quoted 6 lines

include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 289

Quoted 1 line
include/tst_fuzzy_sync.h +0 -0
 	tst_init_stat(&pair->diff_ss);

Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.

Quoted 5 lines

No, af_alg07 requires 2 CPUs, otherwise it'll report false positives. The test will pass only if fchownat() hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.

Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?

Signature · 2 lines
Thank you,
Richard.
#8 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Hi,

Quoted 7 lines
Quoted 5 lines

No, af_alg07 requires 2 CPUs, otherwise it'll report false positives. The test will pass only if fchownat() hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.

Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?

It looks like the test can fail even with more than one cpu. I've seen this sporadic failure on different hardware with more than two cores, at least on intel denverton (x86_64) and renesas r-car (aarch64) systems. Both with kernel 4.19 with the fix included, on the denverton system the rt parches were included and on the r-car not. The test passes most of the time, but sometimes fails with the message Li posted.

It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860

Additionally I tested on qemu-x86 with 4.19 with and without rt patches. The test succeeds even with only one virtualized cpu. So either Martin's assumption is wrong or it holds only for newer kernel versions?

J?rg

author
#9 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Hi Joerg,

Quoted 15 lines · Joerg Vehlow

Hi,

Quoted 7 lines
Quoted 5 lines

No, af_alg07 requires 2 CPUs, otherwise it'll report false positives. The test will pass only if fchownat() hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.

Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?

It looks like the test can fail even with more than one cpu. I've seen this sporadic failure on different hardware with more than two cores, at least on intel denverton (x86_64) and renesas r-car (aarch64) systems. Both with kernel 4.19 with the fix included, on the denverton system the rt parches were included and on the r-car not. The test passes most of the time, but sometimes fails with the message Li posted.

Quoted 7 lines

It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860

Additionally I tested on qemu-x86 with 4.19 with and without rt patches. The test succeeds even with only one virtualized cpu. So either Martin's assumption is wrong or it holds only for newer kernel versions?

No, Mertin is not wrong, and you are also right.

They are totally two different issues of af_alg07, the test on 1CPU should be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.

Can you try this on your aarm64 platform?
-----------------------------
fzsync can't get a random delay range on hpe-moonshot systems, so run with delay=0 during all the tests. This is probably the hardware issue such as cache line design so can't get a stable state during the execution of the critical
section. Provide an experience delay value on hpe-moonshot to make it hit the race window immediately without exceeding samples.

---
 testcases/kernel/crypto/af_alg07.c | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0

b/testcases/kernel/crypto/af_alg07.c
index 6ad86f4f3..24f5b8088 100644

testcases/kernel/crypto/af_alg07.c +1 -0
@@ -47,6 +47,7 @@ static void setup(void)
  fd = SAFE_OPEN("tmpfile", O_RDWR | O_CREAT, 0644);
 
  tst_fzsync_pair_init(&fzsync_pair);
+ fzsync_pair.delay_bias = 700;
 }
 
 static void *thread_run(void *arg)
Signature · 8 lines
2.19.1

-- 
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20201130/c06d8f1b/attachment-0001.htm>
#10 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Hi Li,

Quoted 40 lines · Li Wang

Hi Joerg,

On Mon, Nov 30, 2020 at 3:53 PM Joerg Vehlow <[email protected] <mailto:[email protected]>> wrote:

Hi,
>> No, af_alg07 requires 2 CPUs, otherwise it'll report false positives.
>> The test will pass only if fchownat() hits a half-closed socket and >> returns error. But IIRC the half-closed socket will be destroyed during
>> reschedule which means there's no race window to hit anymore. But it
>> would be better to put the TCONF condition into the test itself. > Interesting, I wonder if this is also true for the real-time kernel with
> the threads set to RT priority?
It looks like the test can fail even with more than one cpu. I've seen
this sporadic failure on different hardware with more than two cores, at
least on intel denverton (x86_64) and renesas r-car (aarch64) systems.
Both with kernel 4.19 with the fix included, on the denverton system the
rt parches were included and on the r-car not. The test passes most of
the time, but sometimes fails with the message Li posted.

It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860
<https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860>

Additionally I tested on qemu-x86 with 4.19 with and without rt patches.
The test succeeds even with only one virtualized cpu. So either Martin's
assumption is wrong or it holds only for newer kernel versions?

Quoted 6 lines

No, Mertin is not wrong, and you are also right.

They are totally two different issues of af_alg07, the test on 1CPU should?be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.

I think you misunderstood something. I see random fails with "TFAIL:
fchownat() failed to fail, kernel may be vulnerable" on both x86_64 and aarch64 with more than one cpu core (4 for x86_64 and 2 or 4 for aarch64).

I see no error ("TPASS: fchownat() failed successfully: ENOENT (2)") on single core qemu-x86. This is why I think Martin's assumption may be wrong. If it was right, it should never succeed on a single core system right?

J?rg

author
#11 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Hi Joerg,

Quoted 46 lines · Joerg Vehlow

Hi Li,

Quoted 9 lines · Li Wang

Hi Joerg,

On Mon, Nov 30, 2020 at 3:53 PM Joerg Vehlow <[email protected] <mailto:[email protected]>> wrote:

Hi,
>> No, af_alg07 requires 2 CPUs, otherwise it'll report false positives.
>> The test will pass only if fchownat() hits a half-closed socket

and

Quoted 33 lines
Quoted 1 line
Quoted 1 line

returns error. But IIRC the half-closed socket will be

destroyed during
>> reschedule which means there's no race window to hit anymore. But it
>> would be better to put the TCONF condition into the test itself. > Interesting, I wonder if this is also true for the real-time kernel with
> the threads set to RT priority?
It looks like the test can fail even with more than one cpu. I've seen
this sporadic failure on different hardware with more than two cores, at
least on intel denverton (x86_64) and renesas r-car (aarch64) systems.
Both with kernel 4.19 with the fix included, on the denverton system the
rt parches were included and on the r-car not. The test passes most of
the time, but sometimes fails with the message Li posted.

It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860
<https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860>

Additionally I tested on qemu-x86 with 4.19 with and without rt patches.
The test succeeds even with only one virtualized cpu. So either Martin's
assumption is wrong or it holds only for newer kernel versions?

No, Mertin is not wrong, and you are also right.

Quoted 8 lines
Quoted 4 lines

They are totally two different issues of af_alg07, the test on 1CPU should be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.

I think you misunderstood something. I see random fails with "TFAIL:
fchownat() failed to fail, kernel may be vulnerable" on both x86_64 and aarch64 with more than one cpu core (4 for x86_64 and 2 or 4 for aarch64).

Well, seems I was somewhat arbitrary on this problem a moment ago.

Probably I was missing the 4cores fails on x86_64 you mentioned, we just observed that FAIL on 1CPU x86_64 and hpe_moonshot(aarch64) so far. The tentative conclusion of our debugging result:

1. FAIL with 1CPU KVM x86_64 is false positives
2. FAIL with hpe_moonshot aarch64 is caused by cache line design

Quoted 6 lines

I see no error ("TPASS: fchownat() failed successfully: ENOENT (2)") on single core qemu-x86. This is why I think Martin's assumption may be wrong. If it was right, it should never succeed on a single core system right?

Hmm, it's hard to say never, it is also possible to create a race window on a single-core system.
Anyway, we need to do more investigation.

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20201130/14a94947/attachment-0001.htm>
#12 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

In reply to Li Wang

Hello,

Quoted 53 lines · Li Wang

Hi Joerg,

Quoted 15 lines · Joerg Vehlow

Hi,

Quoted 7 lines
Quoted 5 lines

No, af_alg07 requires 2 CPUs, otherwise it'll report false positives. The test will pass only if fchownat() hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.

Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?

It looks like the test can fail even with more than one cpu. I've seen this sporadic failure on different hardware with more than two cores, at least on intel denverton (x86_64) and renesas r-car (aarch64) systems. Both with kernel 4.19 with the fix included, on the denverton system the rt parches were included and on the r-car not. The test passes most of the time, but sometimes fails with the message Li posted.

Quoted 7 lines

It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860

Additionally I tested on qemu-x86 with 4.19 with and without rt patches. The test succeeds even with only one virtualized cpu. So either Martin's assumption is wrong or it holds only for newer kernel versions?

No, Mertin is not wrong, and you are also right.

They are totally two different issues of af_alg07, the test on 1CPU should be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.

Can you try this on your aarm64 platform?
-----------------------------
fzsync can't get a random delay range on hpe-moonshot systems, so run with delay=0 during all the tests. This is probably the hardware issue such as cache line design so can't get a stable state during the execution of the critical
section. Provide an experience delay value on hpe-moonshot to make it hit the race window immediately without exceeding samples.

---
 testcases/kernel/crypto/af_alg07.c | 1 +
 1 file changed, 1 insertion(+)
testcases/kernel/crypto/af_alg07.c +0 -0
index 6ad86f4f3..24f5b8088 100644

testcases/kernel/crypto/af_alg07.c static void setup(void) · line 50

Quoted 4 lines
testcases/kernel/crypto/af_alg07.c +1 -0
@@ -47,6 +47,7 @@ static void setup(void)
  fd = SAFE_OPEN("tmpfile", O_RDWR | O_CREAT, 0644);
 
  tst_fzsync_pair_init(&fzsync_pair);
+ fzsync_pair.delay_bias = 700;

I hope there is some way to set this dynamically. Similar to CVE-2016-7117.

If we know that we should get some particular error we could modify the bias until the error happens.

Quoted 5 lines
testcases/kernel/crypto/af_alg07.c +0 -0
 }
 
 static void *thread_run(void *arg)
Signature · 1 line
2.19.1
Signature · 2 lines
Thank you,
Richard.
#13 lore raw

PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2

Quoted 59 lines · Richard Palethorpe

Hello,

Quoted 55 lines · Li Wang

Hi Joerg,

Quoted 15 lines · Joerg Vehlow

Hi,

Quoted 7 lines
Quoted 5 lines

No, af_alg07 requires 2 CPUs, otherwise it'll report false positives. The test will pass only if fchownat() hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.

Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?

It looks like the test can fail even with more than one cpu. I've seen this sporadic failure on different hardware with more than two cores, at least on intel denverton (x86_64) and renesas r-car (aarch64) systems. Both with kernel 4.19 with the fix included, on the denverton system the rt parches were included and on the r-car not. The test passes most of the time, but sometimes fails with the message Li posted.

Quoted 7 lines

It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860

Additionally I tested on qemu-x86 with 4.19 with and without rt patches. The test succeeds even with only one virtualized cpu. So either Martin's assumption is wrong or it holds only for newer kernel versions?

No, Mertin is not wrong, and you are also right.

They are totally two different issues of af_alg07, the test on 1CPU should be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.

Can you try this on your aarm64 platform?
-----------------------------
fzsync can't get a random delay range on hpe-moonshot systems, so run with delay=0 during all the tests. This is probably the hardware issue such as cache line design so can't get a stable state during the execution of the critical
section. Provide an experience delay value on hpe-moonshot to make it hit the race window immediately without exceeding samples.

---
 testcases/kernel/crypto/af_alg07.c | 1 +
 1 file changed, 1 insertion(+)
testcases/kernel/crypto/af_alg07.c +0 -0
index 6ad86f4f3..24f5b8088 100644
@@ -47,6 +47,7 @@ static void setup(void)
  fd = SAFE_OPEN("tmpfile", O_RDWR | O_CREAT, 0644);
 
Quoted 8 lines

testcases/kernel/crypto/af_alg07.c static void setup(void) · line 50

Quoted 2 lines
testcases/kernel/crypto/af_alg07.c +1 -0
  tst_fzsync_pair_init(&fzsync_pair);
+ fzsync_pair.delay_bias = 700;

I hope there is some way to set this dynamically. Similar to CVE-2016-7117.

If we know that we should get some particular error we could modify the bias until the error happens.

There are three possible outcomes of the race:
1) fchownat() returns 0 => fchownat() was called too early or the kernel is vulnerable, you can adjust bias here
2) fchwonat() fails with ENOENT => kernel is fixed, print TPASS and exit
3) fchownat() fails with EBADF => fchownat() was called too late, you can adjust bias here

IIRC I didn't play with bias in this test because on x86_64 it passes almost instantly on a fixed kernel. Feel free to add dynamic bias adjustment for ARM.

Signature · 7 lines
Martin Doucha   [email protected]
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic