fzsync: skip test when avaliable CPUs less than 2
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:
=== 100% reproducible on a 1cpu guest ===
cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period ended ../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024, delay_bias = 0 ../../../include/tst_fuzzy_sync.h:318: TINFO: start_a - start_b: { avg = -137522ns, avg_dev = 854248ns, dev_ratio = 6.21 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a : { avg = 1915ns, avg_dev = 535ns, dev_ratio = 0.28 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b : { avg = 1885ns, avg_dev = 42ns, dev_ratio = 0.02 }
../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b : { avg = -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 }
../../../include/tst_fuzzy_sync.h:318: TINFO: spins : { avg = 554786 , avg_dev = 7355 , dev_ratio = 0.01 }../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time, requesting exit af_alg07.c:96: TFAIL: fchownat() failed to fail, kernel may be vulnerable
--- include/tst_fuzzy_sync.h | 3 +++ 1 file changed, 3 insertions(+)
include/tst_fuzzy_sync.h +3 -0
@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)static void tst_fzsync_pair_reset (struct tst_fzsync_pair *pair,void *(*run_b)(void *)) { +if (get_nprocs () <2 ) +tst_brk (TCONF ,"Fuzzy Sync requires@least two CPUs available" ); +tst_fzsync_pair_cleanup (pair);tst_init_stat (&pair->diff_ss);
Signature · 1 line
2.21.3
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Hello Li,
Quoted 4 lines · Li Wang
It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:
Most of the tests using FuzzySync do not need true parallism. We were able to reproduce a number of race conditions on a single vCPU. Infact it may actually benefit some races because one thread has to pause to allow the other to run, perhaps creating a huge race window.
Quoted 29 lines
=== 100% reproducible on a 1cpu guest ===
cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period ended../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024,delay_bias= 0../../../include/tst_fuzzy_sync.h:318: TINFO:start_a-start_b: { avg = -137522ns,avg_dev= 854248ns,dev_ratio= 6.21 }../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a : { avg = 1915ns, avg_dev = 535ns, dev_ratio = 0.28 } ../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b : { avg = 1885ns, avg_dev = 42ns, dev_ratio = 0.02 } ../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b : { avg = -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 } ../../../include/tst_fuzzy_sync.h:318: TINFO: spins : { avg = 554786 , avg_dev = 7355 , dev_ratio = 0.01 }
../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time, requesting exit af_alg07.c:96: TFAIL:fchownat()failed to fail, kernel may be vulnerableSigned-off-by Li Wang <[email protected]>CC Richard Palethorpe <[email protected]>--- include/tst_fuzzy_sync.h | 3 +++ 1 file changed, 3 insertions(+)include/tst_fuzzy_sync.h +0
-0
include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 289
Quoted 9 lines
include/tst_fuzzy_sync.h +3
-0@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)static void tst_fzsync_pair_reset (struct tst_fzsync_pair *pair,void *(*run_b)(void *)) { +if (get_nprocs () <2 ) +tst_brk (TCONF ,"Fuzzy Sync requires at least two CPUs available" ); +tst_fzsync_pair_cleanup (pair);tst_init_stat (&pair->diff_ss);
Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.
Why not print a warning instead of refusing to run?
Signature · 2 lines
Thank you, Richard.
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Hi!
Quoted 6 lines
Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.
Why not print a warning instead of refusing to run?
That's not a solution either, warning would end up in a test results as well.
I guess that we can add something as .min_cpus to the tst_test structure and set it for this test?
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Quoted 6 lines · Cyril Hrubis
Hi!
Quoted 5 lines
Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.
Quoted 7 lines
Quoted 1 line
Why not print a warning instead of refusing to run?
That's not a solution either, warning would end up in a test results as well.
I guess that we can add something as .min_cpus to the
tst_teststructure and set it for this test?
+1 for .min_cpus from me.
Signature · 7 lines
Martin Doucha [email protected] QA Engineer for Software Maintenance SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
In reply to Richard PalethorpeQuoted 51 lines · Richard Palethorpe
Hello Li,
Quoted 4 lines · Li Wang
It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:
Most of the tests using FuzzySync do not need true parallism. We were able to reproduce a number of race conditions on a single vCPU. Infact it may actually benefit some races because one thread has to pause to allow the other to run, perhaps creating a huge race window.
Quoted 37 lines
=== 100% reproducible on a 1cpu guest ===
cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period ended../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024,delay_bias= 0../../../include/tst_fuzzy_sync.h:318: TINFO:start_a-start_b: { avg = -137522ns,avg_dev= 854248ns,dev_ratio= 6.21 }../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a : { avg = 1915ns, avg_dev = 535ns, dev_ratio = 0.28 } ../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b : { avg = 1885ns, avg_dev = 42ns, dev_ratio = 0.02 } ../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b : { avg = -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 } ../../../include/tst_fuzzy_sync.h:318: TINFO: spins : { avg = 554786 , avg_dev = 7355 , dev_ratio = 0.01 }
../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time, requesting exit af_alg07.c:96: TFAIL:fchownat()failed to fail, kernel may be vulnerableSigned-off-by Li Wang <[email protected]>CC Richard Palethorpe <[email protected]>--- include/tst_fuzzy_sync.h | 3 +++ 1 file changed, 3 insertions(+)include/tst_fuzzy_sync.h +3
-0@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)static void tst_fzsync_pair_reset (struct tst_fzsync_pair *pair,void *(*run_b)(void *)) { +if (get_nprocs () <2 ) +tst_brk (TCONF ,"Fuzzy Sync requires at least two CPUs available" ); +tst_fzsync_pair_cleanup (pair);
Quoted 6 lines
include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 289
Quoted 1 line
include/tst_fuzzy_sync.h +0
-0tst_init_stat (&pair->diff_ss);Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.
No, af_alg07 requires 2 CPUs, otherwise it'll report false positives. The test will pass only if fchownat() hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.
Signature · 7 lines
Martin Doucha [email protected] QA Engineer for Software Maintenance SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Quoted 70 lines · Martin Doucha
Quoted 9 lines · Richard Palethorpe
Hello Li,
Quoted 4 lines · Li Wang
It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:
Quoted 15 lines
Most of the tests using FuzzySync do not need true parallism. We were able to reproduce a number of race conditions on a single vCPU. Infact it may actually benefit some races because one thread has to pause to allow the other to run, perhaps creating a huge race window.
Quoted 9 lines
=== 100% reproducible on a 1cpu guest ===
cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling periodended
Quoted 1 line
Quoted 1 line
../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024,delay_bias=0
Quoted 1 line
Quoted 1 line
../../../include/tst_fuzzy_sync.h:318: TINFO:start_a-start_b: { avg= -137522ns,
avg_dev= 854248ns,dev_ratio= 6.21 }Quoted 1 line
Quoted 1 line
../../../include/tst_fuzzy_sync.h:318: TINFO:end_a-start_a: { avg= 1915ns,
avg_dev= 535ns,dev_ratio= 0.28 }Quoted 1 line
Quoted 1 line
../../../include/tst_fuzzy_sync.h:318: TINFO:end_b-start_b: { avg= 1885ns,
avg_dev= 42ns,dev_ratio= 0.02 }Quoted 1 line
Quoted 1 line
../../../include/tst_fuzzy_sync.h:318: TINFO:end_a-end_b: { avg= -137492ns,
avg_dev= 854818ns,dev_ratio= 6.22 }Quoted 1 line
Quoted 1 line
../../../include/tst_fuzzy_sync.h:318: TINFO: spins : { avg= 554786 ,
avg_dev= 7355 ,dev_ratio= 0.01 }Quoted 1 line
Quoted 1 line
../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time,requesting exit
Quoted 1 line
Quoted 1 line
af_alg07.c:96: TFAIL:
fchownat()failed to fail, kernel may bevulnerable
Quoted 7 lines
Quoted 7 lines
Signed-off-by Li Wang <[email protected]>CC Richard Palethorpe <[email protected]>--- include/tst_fuzzy_sync.h | 3 +++ 1 file changed, 3 insertions(+)Quoted 10 lines
include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 285
Quoted 10 lines
include/tst_fuzzy_sync.h +2
-0@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)static void tst_fzsync_pair_reset (struct tst_fzsync_pair *pair,void *(*run_b)(void *)) { +if (get_nprocs () <2 ) + tst_brk(TCONF, "Fuzzy Sync requires at least two CPUsavailable");
Quoted 9 lines
include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 289
Quoted 4 lines
include/tst_fuzzy_sync.h +1
-0+tst_fzsync_pair_cleanup (pair);tst_init_stat (&pair->diff_ss);Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.
Quoted 6 lines
No,
af_alg07requires 2 CPUs, otherwise it'll report false positives. The test will pass only iffchownat()hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.
+1
Correct, I stand by Martin's point.
And we can avoid adding this patch to FuzzySync lib, but for af_alg07 2cpus is required.
(maybe go with Cyril's suggest to add .min_cpus)
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20201125/630b5666/attachment.htm>
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Hi!
Quoted 3 lines
And we can avoid adding this patch to FuzzySync lib, but for
af_alg072cpus is required.
(maybe go with Cyril's suggest to add .min_cpus)
I would go for that and we can make use of it in getcwd04.c as well.
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
In reply to Martin DouchaHello,
Quoted 59 lines · Martin Doucha
Quoted 51 lines · Richard Palethorpe
Hello Li,
Quoted 4 lines · Li Wang
It makes no sense to run parallel thread to simulate race conditions on system with CPU number less than two, especially for kvm guest, it does not have any chance to get real parallel running and probably encounter failure as below:
Most of the tests using FuzzySync do not need true parallism. We were able to reproduce a number of race conditions on a single vCPU. Infact it may actually benefit some races because one thread has to pause to allow the other to run, perhaps creating a huge race window.
Quoted 37 lines
=== 100% reproducible on a 1cpu guest ===
cmdline="af_alg07"
contacts=""
analysis=exit
<<<test_output>>>
tst_test.c:1248: TINFO: Timeout per run is 0h 05m 00s../../../include/tst_fuzzy_sync.h:507: TINFO: Minimum sampling period ended../../../include/tst_fuzzy_sync.h:330: TINFO: loop = 1024,delay_bias= 0../../../include/tst_fuzzy_sync.h:318: TINFO:start_a-start_b: { avg = -137522ns,avg_dev= 854248ns,dev_ratio= 6.21 }../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - start_a : { avg = 1915ns, avg_dev = 535ns, dev_ratio = 0.28 } ../../../include/tst_fuzzy_sync.h:318: TINFO: end_b - start_b : { avg = 1885ns, avg_dev = 42ns, dev_ratio = 0.02 } ../../../include/tst_fuzzy_sync.h:318: TINFO: end_a - end_b : { avg = -137492ns, avg_dev = 854818ns, dev_ratio = 6.22 } ../../../include/tst_fuzzy_sync.h:318: TINFO: spins : { avg = 554786 , avg_dev = 7355 , dev_ratio = 0.01 }
../../../include/tst_fuzzy_sync.h:636: TINFO: Exceeded execution time, requesting exit af_alg07.c:96: TFAIL:fchownat()failed to fail, kernel may be vulnerableSigned-off-by Li Wang <[email protected]>CC Richard Palethorpe <[email protected]>--- include/tst_fuzzy_sync.h | 3 +++ 1 file changed, 3 insertions(+)include/tst_fuzzy_sync.h +3
-0@@ -281,6 +281,9 @@ static void tst_init_stat(struct tst_fzsync_stat *s)static void tst_fzsync_pair_reset (struct tst_fzsync_pair *pair,void *(*run_b)(void *)) { +if (get_nprocs () <2 ) +tst_brk (TCONF ,"Fuzzy Sync requires at least two CPUs available" ); +tst_fzsync_pair_cleanup (pair);Quoted 6 lines
include/tst_fuzzy_sync.h static void tst_init_stat(struct tst_fzsync_stat *s) · line 289
Quoted 1 line
include/tst_fuzzy_sync.h +0
-0tst_init_stat (&pair->diff_ss);Perhaps this test would pass with more loops and a big enough delay range, but this is also wasting time on a single vCPU. I'm not sure whether we should filter this test at the LTP level; it may trigger the bug on some single CPU configs.
Quoted 5 lines
No,
af_alg07requires 2 CPUs, otherwise it'll report false positives. The test will pass only iffchownat()hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.
Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?
Signature · 2 lines
Thank you, Richard.
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Hi,
Quoted 7 lines
Quoted 5 lines
No,
af_alg07requires 2 CPUs, otherwise it'll report false positives. The test will pass only iffchownat()hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?
It looks like the test can fail even with more than one cpu. I've seen this sporadic failure on different hardware with more than two cores, at least on intel denverton (x86_64) and renesas r-car (aarch64) systems. Both with kernel 4.19 with the fix included, on the denverton system the rt parches were included and on the r-car not. The test passes most of the time, but sometimes fails with the message Li posted.
It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860
Additionally I tested on qemu-x86 with 4.19 with and without rt patches. The test succeeds even with only one virtualized cpu. So either Martin's assumption is wrong or it holds only for newer kernel versions?
J?rg
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Hi Joerg,
Quoted 15 lines · Joerg Vehlow
Hi,
Quoted 7 lines
Quoted 5 lines
No,
af_alg07requires 2 CPUs, otherwise it'll report false positives. The test will pass only iffchownat()hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?
It looks like the test can fail even with more than one cpu. I've seen this sporadic failure on different hardware with more than two cores, at least on intel denverton (
x86_64) and renesas r-car (aarch64) systems. Both with kernel 4.19 with the fix included, on the denverton system the rt parches were included and on the r-car not. The test passes most of the time, but sometimes fails with the message Li posted.
Quoted 7 lines
It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860Additionally I tested on qemu-x86 with 4.19 with and without rt patches. The test succeeds even with only one virtualized cpu. So either Martin's assumption is wrong or it holds only for newer kernel versions?
No, Mertin is not wrong, and you are also right.
They are totally two different issues of af_alg07, the test on 1CPU should be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.
Can you try this on your aarm64 platform?
-----------------------------
fzsync can't get a random delay range on hpe-moonshot systems, so run with delay=0 during all the tests. This is probably the hardware issue such as cache line design so can't get a stable state during the execution of the critical
section. Provide an experience delay value on hpe-moonshot to make it hit the race window immediately without exceeding samples.
--- testcases/kernel/crypto/af_alg07.c | 1 + 1 file changed, 1 insertion(+)
(quoted diff) +0 -0
b/testcases/kernel/crypto/af_alg07.c
index 6ad86f4f3..24f5b8088 100644
testcases/kernel/crypto/af_alg07.c +1 -0
@@ -47,6 +47,7 @@ static void setup(void) fd =SAFE_OPEN ("tmpfile" ,O_RDWR |O_CREAT ,0644 );tst_fzsync_pair_init (&fzsync_pair); + fzsync_pair.delay_bias =700 ; }static void *thread_run (void *arg)
Signature · 8 lines
2.19.1 -- Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20201130/c06d8f1b/attachment-0001.htm>
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Hi Li,
Quoted 40 lines · Li Wang
Hi Joerg,
On Mon, Nov 30, 2020 at 3:53 PM Joerg Vehlow <[email protected] <mailto:[email protected]>> wrote:
Hi,
>> No,af_alg07requires 2 CPUs, otherwise it'll report false positives.
>> The test will pass only iffchownat()hits a half-closed socket and >> returns error. But IIRC the half-closed socket will be destroyed during
>> reschedule which means there's no race window to hit anymore. But it
>> would be better to put the TCONF condition into the test itself. > Interesting, I wonder if this is also true for the real-time kernel with
> the threads set to RT priority?
It looks like the test can fail even with more than one cpu. I've seen
this sporadic failure on different hardware with more than two cores, at
least on intel denverton (x86_64) and renesas r-car (aarch64) systems.
Both with kernel 4.19 with the fix included, on the denverton system the
rt parches were included and on the r-car not. The test passes most of
the time, but sometimes fails with the message Li posted.It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860
<https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860>Additionally I tested on qemu-x86 with 4.19 with and without rt patches.
The test succeeds even with only one virtualized cpu. So either Martin's
assumption is wrong or it holds only for newer kernel versions?
Quoted 6 lines
No, Mertin is not wrong, and you are also right.
They are totally two different issues of
af_alg07, the test on 1CPU should?be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.
I think you misunderstood something. I see random fails with "TFAIL:fchownat() failed to fail, kernel may be vulnerable" on both x86_64 and aarch64 with more than one cpu core (4 for x86_64 and 2 or 4 for aarch64).
I see no error ("TPASS: fchownat() failed successfully: ENOENT (2)") on single core qemu-x86. This is why I think Martin's assumption may be wrong. If it was right, it should never succeed on a single core system right?
J?rg
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Hi Joerg,
Quoted 46 lines · Joerg Vehlow
Hi Li,
Quoted 9 lines · Li Wang
Hi Joerg,
On Mon, Nov 30, 2020 at 3:53 PM Joerg Vehlow <[email protected] <mailto:[email protected]>> wrote:
Hi,
>> No,af_alg07requires 2 CPUs, otherwise it'll report false positives.
>> The test will pass only iffchownat()hits a half-closed socketand
Quoted 33 lines
Quoted 1 line
Quoted 1 line
returns error. But IIRC the half-closed socket will be
destroyed during
>> reschedule which means there's no race window to hit anymore. But it
>> would be better to put the TCONF condition into the test itself. > Interesting, I wonder if this is also true for the real-time kernel with
> the threads set to RT priority?
It looks like the test can fail even with more than one cpu. I've seen
this sporadic failure on different hardware with more than two cores, at
least on intel denverton (x86_64) and renesas r-car (aarch64) systems.
Both with kernel 4.19 with the fix included, on the denverton system the
rt parches were included and on the r-car not. The test passes most of
the time, but sometimes fails with the message Li posted.It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860
<https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860>Additionally I tested on qemu-x86 with 4.19 with and without rt patches.
The test succeeds even with only one virtualized cpu. So either Martin's
assumption is wrong or it holds only for newer kernel versions?No, Mertin is not wrong, and you are also right.
Quoted 8 lines
Quoted 4 lines
They are totally two different issues of
af_alg07, the test on 1CPU should be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.I think you misunderstood something. I see random fails with "TFAIL:
fchownat()failed to fail, kernel may be vulnerable" on bothx86_64and aarch64 with more than one cpu core (4 forx86_64and 2 or 4 for aarch64).
Well, seems I was somewhat arbitrary on this problem a moment ago.
Probably I was missing the 4cores fails on x86_64 you mentioned, we just observed that FAIL on 1CPU x86_64 and hpe_moonshot(aarch64) so far. The tentative conclusion of our debugging result:
1. FAIL with 1CPU KVM x86_64 is false positives
2. FAIL with hpe_moonshot aarch64 is caused by cache line design
Quoted 6 lines
I see no error ("TPASS:
fchownat()failed successfully: ENOENT (2)") on single core qemu-x86. This is why I think Martin's assumption may be wrong. If it was right, it should never succeed on a single core system right?
Hmm, it's hard to say never, it is also possible to create a race window on a single-core system.
Anyway, we need to do more investigation.
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20201130/14a94947/attachment-0001.htm>
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
In reply to Li WangHello,
Quoted 53 lines · Li Wang
Hi Joerg,
Quoted 15 lines · Joerg Vehlow
Hi,
Quoted 7 lines
Quoted 5 lines
No,
af_alg07requires 2 CPUs, otherwise it'll report false positives. The test will pass only iffchownat()hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?
It looks like the test can fail even with more than one cpu. I've seen this sporadic failure on different hardware with more than two cores, at least on intel denverton (
x86_64) and renesas r-car (aarch64) systems. Both with kernel 4.19 with the fix included, on the denverton system the rt parches were included and on the r-car not. The test passes most of the time, but sometimes fails with the message Li posted.Quoted 7 lines
It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860Additionally I tested on qemu-x86 with 4.19 with and without rt patches. The test succeeds even with only one virtualized cpu. So either Martin's assumption is wrong or it holds only for newer kernel versions?
No, Mertin is not wrong, and you are also right.
They are totally two different issues of
af_alg07, the test on 1CPU should be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.Can you try this on your aarm64 platform?
-----------------------------
fzsync can't get a random delay range on hpe-moonshot systems, so run with delay=0 during all the tests. This is probably the hardware issue such as cache line design so can't get a stable state during the execution of the critical
section. Provide an experience delay value on hpe-moonshot to make it hit the race window immediately without exceeding samples.--- testcases/kernel/crypto/af_alg07.c | 1 + 1 file changed, 1 insertion(+)testcases/kernel/crypto/af_alg07.c +0
-0
testcases/kernel/crypto/af_alg07.c static void setup(void) · line 50
Quoted 4 lines
testcases/kernel/crypto/af_alg07.c +1
-0@@ -47,6 +47,7 @@ static void setup(void) fd =SAFE_OPEN ("tmpfile" ,O_RDWR |O_CREAT ,0644 );tst_fzsync_pair_init (&fzsync_pair); + fzsync_pair.delay_bias =700 ;
I hope there is some way to set this dynamically. Similar to CVE-2016-7117.
If we know that we should get some particular error we could modify the bias until the error happens.
Quoted 5 lines
testcases/kernel/crypto/af_alg07.c +0
-0}static void *thread_run (void *arg)Signature · 1 line
2.19.1
Signature · 2 lines
Thank you, Richard.
PATCH 1/1 fzsync: skip test when avaliable CPUs less than 2
Quoted 59 lines · Richard Palethorpe
Hello,
Quoted 55 lines · Li Wang
Hi Joerg,
Quoted 15 lines · Joerg Vehlow
Hi,
Quoted 7 lines
Quoted 5 lines
No,
af_alg07requires 2 CPUs, otherwise it'll report false positives. The test will pass only iffchownat()hits a half-closed socket and returns error. But IIRC the half-closed socket will be destroyed during reschedule which means there's no race window to hit anymore. But it would be better to put the TCONF condition into the test itself.Interesting, I wonder if this is also true for the real-time kernel with the threads set to RT priority?
It looks like the test can fail even with more than one cpu. I've seen this sporadic failure on different hardware with more than two cores, at least on intel denverton (
x86_64) and renesas r-car (aarch64) systems. Both with kernel 4.19 with the fix included, on the denverton system the rt parches were included and on the r-car not. The test passes most of the time, but sometimes fails with the message Li posted.Quoted 7 lines
It also seems to fail sporadically on other systems as well:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1892860Additionally I tested on qemu-x86 with 4.19 with and without rt patches. The test succeeds even with only one virtualized cpu. So either Martin's assumption is wrong or it holds only for newer kernel versions?
No, Mertin is not wrong, and you are also right.
They are totally two different issues of
af_alg07, the test on 1CPU should be fixed with TCONF. But the fail with aarch64 is more like a hardware issue, Chunyu has a drafted patch to add init delay value for such a system.Can you try this on your aarm64 platform?
-----------------------------
fzsync can't get a random delay range on hpe-moonshot systems, so run with delay=0 during all the tests. This is probably the hardware issue such as cache line design so can't get a stable state during the execution of the critical
section. Provide an experience delay value on hpe-moonshot to make it hit the race window immediately without exceeding samples.--- testcases/kernel/crypto/af_alg07.c | 1 + 1 file changed, 1 insertion(+)testcases/kernel/crypto/af_alg07.c +0
-0@@ -47,6 +47,7 @@ static void setup(void) fd =SAFE_OPEN ("tmpfile" ,O_RDWR |O_CREAT ,0644 );
Quoted 8 lines
testcases/kernel/crypto/af_alg07.c static void setup(void) · line 50
Quoted 2 lines
testcases/kernel/crypto/af_alg07.c +1
-0tst_fzsync_pair_init (&fzsync_pair); + fzsync_pair.delay_bias =700 ;I hope there is some way to set this dynamically. Similar to CVE-2016-7117.
If we know that we should get some particular error we could modify the bias until the error happens.
There are three possible outcomes of the race:
1) fchownat() returns 0 => fchownat() was called too early or the kernel is vulnerable, you can adjust bias here
2) fchwonat() fails with ENOENT => kernel is fixed, print TPASS and exit
3) fchownat() fails with EBADF => fchownat() was called too late, you can adjust bias here
IIRC I didn't play with bias in this test because on x86_64 it passes almost instantly on a fixed kernel. Feel free to add dynamic bias adjustment for ARM.
Signature · 7 lines
Martin Doucha [email protected] QA Engineer for Software Maintenance SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic