red_autotest random failures
https://bugs.dpdk.org/show_bug.cgi?id=826
Bug ID: 826
Summary: red_autotest random failures
Product: DPDK
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: Normal
Component: other
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected], [email protected]
Target Milestone: ---
A recent failure can be found at:
https://lab.dpdk.org/results/dashboard/patchsets/19223/
50/94 DPDK:fast-tests / red_autotest FAIL 0.86s
exit status 1
23:08:32 MALLOC_PERTURB_=116 DPDK_TEST=red_autotest /home-local/jenkins-local/jenkins-agent/workspace/Generic-Unit-Test-DPDK/dpdk/build/app/test/dpdk-test '-l 0-15' --file-prefix=red_autotest
----------------------------------- output ----------------------------------- stdout:
RTE>>red_autotest
--------------------------------------------------------------------------------
functional test 1 : use one rte_red configuration,
increase average queue size to various levels, compare drop rate to drop probability
avg queue size enqueued dropped drop prob %
drop rate % diff % tolerance %
6 10000 0 0.0000
0.0000 0.0000 50.0000
12 10000 0 0.0000
0.0000 0.0000 50.0000
18 10000 0 0.0000
0.0000 0.0000 50.0000
24 10000 0 0.0000
0.0000 0.0000 50.0000
30 10000 0 0.0000
0.0000 0.0000 50.0000
36 9954 46 0.4167
0.4600 0.0000 50.0000
42 9901 99 1.0417
0.9900 0.0000 50.0000
48 9831 169 1.6667
1.6900 0.0000 50.0000
54 9761 239 2.2917
2.3900 0.0000 50.0000
60 9701 299 2.9167
2.9900 0.0000 50.0000
66 9618 382 3.5417
3.8200 0.0000 50.0000
72 9581 419 4.1667
4.1900 0.0000 50.0000
78 9490 510 4.7917
5.1000 0.0000 50.0000
84 9460 540 5.4167
5.4000 0.0000 50.0000
90 9398 602 6.0417
6.0200 0.0000 50.0000
96 9371 629 6.6667
6.2900 0.0000 50.0000
102 9259 741 7.2917
7.4100 0.0000 50.0000
108 9211 789 7.9167
7.8900 0.0000 50.0000
114 9133 867 8.5417
8.6700 0.0000 50.0000
120 9058 942 9.1667
9.4200 0.0000 50.0000
126 8987 1013 9.7917
10.1300 0.0000 50.0000
132 0 10000 100.0000
100.0000 0.0000 50.0000
138 0 10000 100.0000
100.0000 0.0000 50.0000
144 0 10000 100.0000 100.0000 0.0000 50.0000
-------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
functional test 2 : use several RED configurations,
increase average queue size to just below maximum
threshold,
compare drop rate to drop probability
RED config avg queue size min threshold max threshold drop prob % drop
rate % diff % tolerance %
0 127 32 128 9.8958 9.9400 0.0000 50.0000 1 127 32 128 4.9479 4.8100 0.0000 50.0000 2 127 32 128 3.2986 2.6800 0.0000 50.0000 3 127 32 128 2.4740 1.6300 0.0000 50.0000 4 127 32 128 1.9792 1.2700 0.0000 50.0000 5 127 32 128 1.6493 1.0300 0.0000 50.0000 6 127 32 128 1.4137 0.8300 0.0000 50.0000 7 127 32 128 1.2370 0.7300 0.0000 50.0000 8 127 32 128 1.0995 0.6200 0.0000 50.0000 9 127 32 128 0.9896 0.5500 0.0000 50.0000
-------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
functional test 3 : use one RED configuration,
increase average queue size to target level, dequeue all packets until queue is empty, confirm that average queue size is computed correctly while
queue is empty
q avg before q avg after expected difference % tolerance %
result
1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass
-------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
functional test 5 : use several queues (each with its own run-time data),
use several RED configurations (such that each
configuration is shared by multiple queues),
increase average queue size to just below maximum
threshold,
compare drop rate to drop probability, (this is a larger scale version of functional test 2)
queue config avg queue size min threshold max threshold drop
prob % drop rate % diff % tolerance %
0 0 127 32 128 9.8958 9.9700 0.0000 50.0000 1 0 127 32 128 9.8958 9.7200 0.0000 50.0000 2 1 127 32 128 4.9479 4.8900 0.0000 50.0000 3 1 127 32 128 4.9479 4.8200 0.0000 50.0000
-------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
functional test 6 : use several queues (each with its own run-time data),
use several RED configurations (such that each
configuration is sharte_red by multiple queues),
increase average queue size to target level, dequeue all packets until queue is empty, confirm that average queue size is computed correctly while
queue is empty
(this is a larger scale version of functional test 3)
queue config q avg before q avg after expected difference % tolerance % result 0 0 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1 0 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 2 1 1022.0000 1022.0000 1010.1483 1.1733 5.0000 pass 3 1 1022.0000 937.1660 1010.1483 7.2249 5.0000 fail
-------------------------------------<fail>-------------------------------------
--------------------------------------------------------------------------------
overflow test 1 : use one RED configuration,
increase average queue size to target level, check maximum number of bits requirte_red to represent avg_s
avg queue size wq_log2 fraction bits max queue avg num bits enqueued
dropped drop prob % drop rate %
1023 12 10 0xffc00000 32 0
941366 100.00 100.00
-------------------------------------<pass>------------------------------------- [total: 6, pass: 5, fail: 1]
Test Failed
Signature · 2 lines
You are receiving this mail because: You are on the CC list for the bug.
Quoted 18 lines · On Fri, Oct 8, 2021 at 9:24 AM <[email protected]> wrote:
https://bugs.dpdk.org/show_bug.cgi?id=826
Bug ID: 826
Summary:red_autotestrandom failures
Product: DPDK
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: Normal
Component: other
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected], [email protected]
Target Milestone: ---
Quoted 5 lines
A recent failure can be found at:
https://lab.dpdk.org/results/dashboard/patchsets/19223/50/94 DPDK:fast-tests /
red_autotestFAIL 0.86s
exit status 1
functional test 6 : use several queues (each with its own run-time data),
use several RED configurations (such that each
configuration is sharte_red by multiple queues),
increase average queue size to target level, dequeue all packets until queue is empty,
confirm that average queue size is computed correctly
while queue is empty
(this is a larger scale version of functional test 3)
queue config q avg before q avg after expected
difference % tolerance % result
0 0 1022.0000 1022.0000 1016.0627
0.5843 5.0000 pass
1 0 1022.0000 1022.0000 1016.0627
0.5843 5.0000 pass
2 1 1022.0000 1022.0000 1010.1483
1.1733 5.0000 pass
3 1 1022.0000 937.1660 1010.1483
7.2249 5.0000 fail-------------------------------------<fail>-------------------------------------
This failure keeps on popping in the CI.
The bug report is one month old, with no reply.
I sent a proposal of removing red_autotest from the list executed by the CI.
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
It might be the best solution waiting for an analysis.
Signature · 1 line
David Marchand
Quoted 58 lines · David Marchand
Quoted 18 lines · On Fri, Oct 8, 2021 at 9:24 AM <[email protected]> wrote:
https://bugs.dpdk.org/show_bug.cgi?id=826
Bug ID: 826
Summary:red_autotestrandom failures
Product: DPDK
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: Normal
Component: other
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected], [email protected]
Target Milestone: ---Quoted 4 lines
A recent failure can be found at:
https://lab.dpdk.org/results/dashboard/patchsets/19223/50/94 DPDK:fast-tests /
red_autotestFAIL0.86s
Quoted 1 line
exit status 1
functional test 6 : use several queues (each with its own run-time data),
use several RED configurations (such that each
configuration issharte_redby multiple queues),
increase average queue size to target level, dequeue all packets until queue is empty,
confirm that average queue size is computed correctly
while queue is empty
(this is a larger scale version of functional test 3)queue config q avg before q avg after expected
difference % tolerance % result
0 0 1022.0000 1022.0000 1016.06270.5843 5.0000 pass 1 0 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 2 1 1022.0000 1022.0000 1010.1483 1.1733 5.0000 pass 3 1 1022.0000 937.1660 1010.1483 7.2249 5.0000 fail-------------------------------------<fail>-------------------------------------
This failure keeps on popping in the CI.
The bug report is one month old, with no reply.I sent a proposal of removing
red_autotestfrom the list executed by the CI.
Quoted 9 lines
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
It might be the best solution waiting for an analysis.
--
David Marchand
Hi David,
My understanding is, removing the test would require removing it from the DPDK unit tests, we are just running the fast-tests suite for the unit tests. DPDK's unit test structure / framework does not allow removing or customizing the suite of tests beyond the suites.
In the lab, Brandon has been looking into and trying different configurations for running the tests within the containers along the lines of the CPU pinning requirements that might be assumed by the unit tests. So far, everything he has tried has still had the similar failures / issues. We are still looking into it, so the bug is not sitting without action, just no final resolution.
Cheers,
Lincoln
Signature · 7 lines
*Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 [email protected] https://www.iol.unh.edu +1-603-674-2755 (m) <https://www.iol.unh.edu>
Quoted 10 lines · Lincoln Lavoie
Quoted 10 lines
This failure keeps on popping in the CI.
The bug report is one month old, with no reply.I sent a proposal of removing
red_autotestfrom the list executed by the CI.
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/It might be the best solution waiting for an analysis.
Quoted 7 lines
Quoted 3 lines
--
David MarchandHi David,
My understanding is, removing the test would require removing it from the DPDK unit tests, we are just running the fast-tests suite for the unit tests. DPDK's unit test structure / framework does not allow removing or customizing the suite of tests beyond the suites.
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Quoted 2 lines
In the lab, Brandon has been looking into and trying different configurations for running the tests within the containers along the lines of the CPU pinning requirements that might be assumed by the unit tests. So far, everything he has tried has still had the similar failures / issues. We are still looking into it, so the bug is not sitting without action, just no final resolution.
The mail I sent was not a comment for the investigation on UNH side. The ask is for Cristian to have a look too.
Signature · 1 line
David Marchand
Quoted 48 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, November 12, 2021 2:16 PM
To: Lincoln Lavoie <[email protected]>
Cc: Dumitrescu, Cristian <[email protected]>; dev <[email protected]>; Aaron Conole <[email protected]>; Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [Bug 826]red_autotestrandom failuresQuoted 5 lines · Lincoln Lavoie
Quoted 5 lines
This failure keeps on popping in the CI.
The bug report is one month old, with no reply.I sent a proposal of removing
red_autotestfrom the list executed by theCI.
Quoted 1 line
Quoted 11 lines
Quoted 7 lines
It might be the best solution waiting for an analysis.
--
David MarchandHi David,
My understanding is, removing the test would require removing it from the
DPDK unit tests, we are just running the fast-tests suite for the unit tests. DPDK's unit test structure / framework does not allow removing or customizing the suite of tests beyond the suites.
https://patchwork.dpdk.org/project/dpdk/patch/20211027140458.2502-2-
[email protected]/Quoted 2 lines
In the lab, Brandon has been looking into and trying different
configurations for running the tests within the containers along the lines of the CPU pinning requirements that might be assumed by the unit tests. So far, everything he has tried has still had the similar failures / issues. We are still looking into it, so the bug is not sitting without action, just no final resolution.
Quoted 6 lines
The mail I sent was not a comment for the investigation on UNH side. The ask is for Cristian to have a look too.
--
David Marchand
Wojciech, Megha,
Are you able to take a look at why is the RED autotest failing, please?
Thanks,
Cristian
Hi,
Sure, I will have a look.
Best Regards,
Wojciech
-----Original Message-----
From: Dumitrescu, Cristian <[email protected]>
Sent: Monday, November 15, 2021 12:51 PM
To: David Marchand <[email protected]>; Lincoln Lavoie <[email protected]>; Liguzinski, WojciechX <[email protected]>; Ajmera, Megha <[email protected]>; Singh, Jasvinder <[email protected]>
Cc: dev <[email protected]>; Aaron Conole <[email protected]>; Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [Bug 826] red_autotest random failures
Quoted 48 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, November 12, 2021 2:16 PM
To: Lincoln Lavoie <[email protected]>
Cc: Dumitrescu, Cristian <[email protected]>; dev <[email protected]>; Aaron Conole <[email protected]>; Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [Bug 826]red_autotestrandom failuresQuoted 6 lines · Lincoln Lavoie
Quoted 6 lines
This failure keeps on popping in the CI.
The bug report is one month old, with no reply.I sent a proposal of removing
red_autotestfrom the list executed by theCI.
Quoted 1 line
Quoted 4 lines
Quoted 4 lines
It might be the best solution waiting for an analysis.
Quoted 8 lines
Quoted 3 lines
--
David MarchandHi David,
My understanding is, removing the test would require removing it from the
DPDK unit tests, we are just running the fast-tests suite for the unit tests. DPDK's unit test structure / framework does not allow removing or customizing the suite of tests beyond the suites.
https://patchwork.dpdk.org/project/dpdk/patch/20211027140458.2502-2-
[email protected]/Quoted 2 lines
In the lab, Brandon has been looking into and trying different
configurations for running the tests within the containers along the lines of the CPU pinning requirements that might be assumed by the unit tests. So far, everything he has tried has still had the similar failures / issues. We are still looking into it, so the bug is not sitting without action, just no final resolution.
Quoted 6 lines
The mail I sent was not a comment for the investigation on UNH side. The ask is for Cristian to have a look too.
--
David Marchand
Wojciech, Megha,
Are you able to take a look at why is the RED autotest failing, please?
Thanks,
Cristian
Hi,
I was trying to reproduce this test failure, but for me RED tests are passing.
I was running the exact test command like the one described in Bug 826 - 'red_autotest' on the current main branch.
Here is an example when DPDK is build without RTE_SCHED_CMAN enabled, but with this flag set to true tests are also not failing.
root@silpixa00400629:~/wojtek/dpdk/build/app/test# ./dpdk-test '-l 0-15' --file-prefix=red_autotest EAL: Detected CPU lcores: 96 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/red_autotest/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized TELEMETRY: No legacy callbacks, legacy socket not created APP: HPET is not enabled, using TSC as default timer RTE>>red_autotest
--------------------------------------------------------------------------------
functional test 1 : use one rte_red configuration,
increase average queue size to various levels,
compare drop rate to drop probability
avg queue size enqueued dropped drop prob % drop rate % diff % tolerance %
6 10000 0 0.0000 0.0000 0.0000 50.0000
12 10000 0 0.0000 0.0000 0.0000 50.0000
18 10000 0 0.0000 0.0000 0.0000 50.0000
24 10000 0 0.0000 0.0000 0.0000 50.0000
30 10000 0 0.0000 0.0000 0.0000 50.0000
36 9961 39 0.4167 0.3900 0.0000 50.0000
42 9898 102 1.0417 1.0200 0.0000 50.0000
48 9835 165 1.6667 1.6500 0.0000 50.0000
54 9785 215 2.2917 2.1500 0.0000 50.0000
60 9703 297 2.9167 2.9700 0.0000 50.0000
66 9627 373 3.5417 3.7300 0.0000 50.0000
72 9580 420 4.1667 4.2000 0.0000 50.0000
78 9511 489 4.7917 4.8900 0.0000 50.0000
84 9462 538 5.4167 5.3800 0.0000 50.0000
90 9398 602 6.0417 6.0200 0.0000 50.0000
96 9366 634 6.6667 6.3400 0.0000 50.0000
102 9267 733 7.2917 7.3300 0.0000 50.0000
108 9212 788 7.9167 7.8800 0.0000 50.0000
114 9146 854 8.5417 8.5400 0.0000 50.0000
120 9102 898 9.1667 8.9800 0.0000 50.0000
126 8984 1016 9.7917 10.1600 0.0000 50.0000
132 0 10000 100.0000 100.0000 0.0000 50.0000
138 0 10000 100.0000 100.0000 0.0000 50.0000
144 0 10000 100.0000 100.0000 0.0000 50.0000 -------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
functional test 2 : use several RED configurations,
increase average queue size to just below maximum threshold, compare drop rate to drop probability
RED config avg queue size min threshold max threshold drop prob % drop rate % diff % tolerance % 0 127 32 128 9.8958 10.0100 0.0000 50.0000 1 127 32 128 4.9479 4.9700 0.0000 50.0000 2 127 32 128 3.2986 2.6800 0.0000 50.0000 3 127 32 128 2.4740 1.7000 0.0000 50.0000 4 127 32 128 1.9792 1.2700 0.0000 50.0000 5 127 32 128 1.6493 1.0500 0.0000 50.0000 6 127 32 128 1.4137 0.8100 0.0000 50.0000 7 127 32 128 1.2370 0.7100 0.0000 50.0000 8 127 32 128 1.0995 0.6200 0.0000 50.0000 9 127 32 128 0.9896 0.5600 0.0000 50.0000
-------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
functional test 3 : use one RED configuration,
increase average queue size to target level,
dequeue all packets until queue is empty,
confirm that average queue size is computed correctly while queue is empty
q avg before q avg after expected difference % tolerance % result 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass
-------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
functional test 5 : use several queues (each with its own run-time data),
use several RED configurations (such that each configuration is shared by multiple queues), increase average queue size to just below maximum threshold,
compare drop rate to drop probability,
(this is a larger scale version of functional test 2)
queue config avg queue size min threshold max threshold drop prob % drop rate % diff % tolerance % 0 0 127 32 128 9.8958 9.9200 0.0000 50.0000 1 0 127 32 128 9.8958 9.9700 0.0000 50.0000 2 1 127 32 128 4.9479 4.8600 0.0000 50.0000 3 1 127 32 128 4.9479 4.9400 0.0000 50.0000
-------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
functional test 6 : use several queues (each with its own run-time data),
use several RED configurations (such that each configuration is shared by multiple queues), increase average queue size to target level,
dequeue all packets until queue is empty,
confirm that average queue size is computed correctly while queue is empty (this is a larger scale version of functional test 3)
queue config q avg before q avg after expected difference % tolerance % result 0 0 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 1 0 1022.0000 1022.0000 1016.0627 0.5843 5.0000 pass 2 1 1022.0000 1022.0000 1010.1483 1.1733 5.0000 pass 3 1 1022.0000 1022.0000 1010.1483 1.1733 5.0000 pass
-------------------------------------<pass>-------------------------------------
--------------------------------------------------------------------------------
overflow test 1 : use one RED configuration,
increase average queue size to target level, check maximum number of bits requirte_red to represent avg_s
avg queue size wq_log2 fraction bits max queue avg num bits enqueued dropped drop prob % drop rate % 1023 12 10 0xffc00000 32 0 941366 100.00 100.00
-------------------------------------<pass>------------------------------------- [total: 6, pass: 6]
Test OK
RTE>>quit
Kind Regards,
Wojtek
-----Original Message-----
From: Liguzinski, WojciechX
Sent: Monday, November 15, 2021 6:27 PM
To: Dumitrescu, Cristian <[email protected]>; David Marchand <[email protected]>; Lincoln Lavoie <[email protected]>; Ajmera, Megha <[email protected]>; Singh, Jasvinder <[email protected]>
Cc: dev <[email protected]>; Aaron Conole <[email protected]>; Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [Bug 826] red_autotest random failures
Hi,
Sure, I will have a look.
Best Regards,
Wojciech
-----Original Message-----
From: Dumitrescu, Cristian <[email protected]>
Sent: Monday, November 15, 2021 12:51 PM
To: David Marchand <[email protected]>; Lincoln Lavoie <[email protected]>; Liguzinski, WojciechX <[email protected]>; Ajmera, Megha <[email protected]>; Singh, Jasvinder <[email protected]>
Cc: dev <[email protected]>; Aaron Conole <[email protected]>; Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [Bug 826] red_autotest random failures
Quoted 48 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, November 12, 2021 2:16 PM
To: Lincoln Lavoie <[email protected]>
Cc: Dumitrescu, Cristian <[email protected]>; dev <[email protected]>; Aaron Conole <[email protected]>; Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [Bug 826]red_autotestrandom failuresQuoted 6 lines · Lincoln Lavoie
Quoted 6 lines
This failure keeps on popping in the CI.
The bug report is one month old, with no reply.I sent a proposal of removing
red_autotestfrom the list executed by theCI.
Quoted 1 line
Quoted 4 lines
Quoted 4 lines
It might be the best solution waiting for an analysis.
Quoted 8 lines
Quoted 3 lines
--
David MarchandHi David,
My understanding is, removing the test would require removing it from the
DPDK unit tests, we are just running the fast-tests suite for the unit tests. DPDK's unit test structure / framework does not allow removing or customizing the suite of tests beyond the suites.
https://patchwork.dpdk.org/project/dpdk/patch/20211027140458.2502-2-
[email protected]/Quoted 2 lines
In the lab, Brandon has been looking into and trying different
configurations for running the tests within the containers along the lines of the CPU pinning requirements that might be assumed by the unit tests. So far, everything he has tried has still had the similar failures / issues. We are still looking into it, so the bug is not sitting without action, just no final resolution.
Quoted 6 lines
The mail I sent was not a comment for the investigation on UNH side. The ask is for Cristian to have a look too.
--
David Marchand
Wojciech, Megha,
Are you able to take a look at why is the RED autotest failing, please?
Thanks,
Cristian
18/11/2021 23:10, Liguzinski, WojciechX:
Quoted 4 lines
Hi,
I was trying to reproduce this test failure, but for me RED tests are passing.
I was running the exact test command like the one described in Bug 826 - 'red_autotest' on the current main branch.
The test is not always failing.
There are some failing conditions, please find them.
I think you should try in a container with more limited resources.
Quoted 18 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Friday, November 19, 2021 7:26 AM
To: Dumitrescu, Cristian <[email protected]>; David Marchand <[email protected]>; Lincoln Lavoie <[email protected]>; Ajmera, Megha <[email protected]>; Singh, Jasvinder <[email protected]>; Liguzinski, WojciechX <[email protected]>
Cc: dev <[email protected]>; Aaron Conole <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Zegota, AnnaX <[email protected]>
Subject: Re: [dpdk-dev] [Bug 826]red_autotestrandom failures18/11/2021 23:10, Liguzinski, WojciechX:
Quoted 2 lines
Hi,
Quoted 8 lines
Quoted 2 lines
I was trying to reproduce this test failure, but for me RED tests are passing. I was running the exact test command like the one described in Bug 826 -
'
red_autotest' on the current main branch.The test is not always failing.
There are some failing conditions, please find them.
I think you should try in a container with more limited resources.
Hi Thomas,
This is not a fair request IMO. We want to avoid wasting everybody's time, including Wojciech's time. Can the bug originator provide the details on the setup to reproduce the failure, please? Thank you!
On a different point, we should probably tweak our autotests to differentiate between logical failures and those failures related to resources not being available, and flag the test result accordingly in the report. For example, if memory allocation fails, the test should be flagged as "Not enough resources" instead of simply "Failed". In the first case, the next step should be fixing the test setup, while in the second case the next step should be fixing the code. What do people think on this?
Regards,
Cristian
Hi All,
I'm not sure if it will help, but this is an example of a failing case in the CI: https://lab.dpdk.org/results/dashboard/patchsets/20222/
The test is running within a docker container. CI is set up to only allow one active unit test at a time, so the host might be running compile jobs, but not other unit tests. This ensures there isn't "competition" for resources like hugepages between two running unit test jobs. The host is actually a VM running on VMware vCenter, not a bare-metal host, the VM's sole purpose is running the docker jobs.
The command to start the unit test run is pretty generic (script is below).
#!/bin/bash
####################################################
# $1 argument: extra arguments to send to meson test
####################################################
# Exit on first command failure
set -e
# Extract dpdk.tar.gz
tar xzfm dpdk.tar.gz
# Compile DPDK
cd dpdk
meson build --werror
ninja -C build install
# Unit test
cd build
meson test --suite fast-tests -t 60 $1
I think a starting point is to understand if the unit test expects or makes assumptions on the system / environment. If it has sole access to a CPU core, minimum number of hugepages, etc. If it would help, I can also give you the DockerFile to build the container (note the RHEL images have to be built on a licensed Redhat server, based on being able to install the required packages).
Cheers,
Lincoln
Quoted 46 lines · Cristian
Quoted 15 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Friday, November 19, 2021 7:26 AM
To: Dumitrescu, Cristian <[email protected]>; David Marchand <[email protected]>; Lincoln Lavoie <[email protected]>; Ajmera, Megha <[email protected]>; Singh, Jasvinder <[email protected]>; Liguzinski, WojciechX <[email protected]>
Cc: dev <[email protected]>; Aaron Conole <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Zegota, AnnaX <[email protected]>
Subject: Re: [dpdk-dev] [Bug 826]red_autotestrandom failuresQuoted 4 lines
18/11/2021 23:10, Liguzinski, WojciechX:
Quoted 3 lines
Hi,
I was trying to reproduce this test failure, but for me RED tests are
passing.
Quoted 1 line
Quoted 1 line
I was running the exact test command like the one described in Bug 826
-
Quoted 6 lines
'
red_autotest' on the current main branch.The test is not always failing.
There are some failing conditions, please find them.
I think you should try in a container with more limited resources.Hi Thomas,
This is not a fair request IMO. We want to avoid wasting everybody's time, including Wojciech's time. Can the bug originator provide the details on the setup to reproduce the failure, please? Thank you!
On a different point, we should probably tweak our autotests to differentiate between logical failures and those failures related to resources not being available, and flag the test result accordingly in the report. For example, if memory allocation fails, the test should be flagged as "Not enough resources" instead of simply "Failed". In the first case, the next step should be fixing the test setup, while in the second case the next step should be fixing the code. What do people think on this?
Regards,
Cristian
Signature · 7 lines
*Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 [email protected] https://www.iol.unh.edu +1-603-674-2755 (m) <https://www.iol.unh.edu>
Quoted 1 line · Cristian
On a different point, we should probably tweak our autotests to differentiate between logical failures and those failures related to resources not being available, and flag the test result accordingly in the report. For example, if memory allocation fails, the test should be flagged as "Not enough resources" instead of simply "Failed". In the first case, the next step should be fixing the test setup, while in the second case the next step should be fixing the code. What do people think on this?
In such case, the test must return TEST_SKIPPED.
I did a pass for cores count / specific hw requirements, some time ago. See https://git.dpdk.org/dpdk/commit/?id=e0f4a0ed4237
Signature · 1 line
David Marchand
Quoted 10 lines · David Marchand
Quoted 1 line · Cristian
On a different point, we should probably tweak our autotests to
differentiate between logical failures and those failures related to resources not being available, and flag the test result accordingly in the report. For example, if memory allocation fails, the test should be flagged as "Not enough resources" instead of simply "Failed". In the first case, the next step should be fixing the test setup, while in the second case the next step should be fixing the code. What do people think on this?
Quoted 3 lines
In such case, the test must return
TEST_SKIPPED.If the purpose of the component / function being tested is to get / create
/ reserve the resource(s), the failure might be valid. So it can't be applied across the board. But places where the test is checking other functionality, this might at least prevent some failures that are transient (i.e. based on what the test could "get" from the system at that moment in time).
Quoted 8 lines
I did a pass for cores count / specific hw requirements, some time ago. See https://git.dpdk.org/dpdk/commit/?id=e0f4a0ed4237
--
David Marchand
Signature · 7 lines
*Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 [email protected] https://www.iol.unh.edu +1-603-674-2755 (m) <https://www.iol.unh.edu>
Hi,
Thanks Lincoln, I will also have a try with such script.
Cheers,
Wojciech
From: Lincoln Lavoie <[email protected]>
Sent: Friday, November 19, 2021 6:26 PM
To: Dumitrescu, Cristian <[email protected]>
Cc: Thomas Monjalon <[email protected]>; David Marchand <[email protected]>; Ajmera, Megha <[email protected]>; Singh, Jasvinder <[email protected]>; Liguzinski, WojciechX <[email protected]>; dev <[email protected]>; Aaron Conole <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Zegota, AnnaX <[email protected]>
Subject: Re: [dpdk-dev] [Bug 826] red_autotest random failures
Hi All,
I'm not sure if it will help, but this is an example of a failing case in the CI: https://lab.dpdk.org/results/dashboard/patchsets/20222/
The test is running within a docker container. CI is set up to only allow one active unit test at a time, so the host might be running compile jobs, but not other unit tests. This ensures there isn't "competition" for resources like hugepages between two running unit test jobs. The host is actually a VM running on VMware vCenter, not a bare-metal host, the VM's sole purpose is running the docker jobs.
The command to start the unit test run is pretty generic (script is below).
#!/bin/bash
####################################################
# $1 argument: extra arguments to send to meson test
####################################################
# Exit on first command failure
set -e
# Extract dpdk.tar.gz
tar xzfm dpdk.tar.gz
# Compile DPDK
cd dpdk
meson build --werror
ninja -C build install
# Unit test
cd build
meson test --suite fast-tests -t 60 $1
I think a starting point is to understand if the unit test expects or makes assumptions on the system / environment. If it has sole access to a CPU core, minimum number of hugepages, etc. If it would help, I can also give you the DockerFile to build the container (note the RHEL images have to be built on a licensed Redhat server, based on being able to install the required packages).
Cheers,
Lincoln
Quoted 16 lines · On Fri, Nov 19, 2021 at 11:54 AM Dumitrescu, Cristian…
-----Original Message-----
From: Thomas Monjalon <[email protected]<mailto:[email protected]>>
Sent: Friday, November 19, 2021 7:26 AM
To: Dumitrescu, Cristian <[email protected]<mailto:[email protected]>>; David Marchand <[email protected]<mailto:[email protected]>>; Lincoln Lavoie <[email protected]<mailto:[email protected]>>; Ajmera, Megha <[email protected]<mailto:[email protected]>>; Singh, Jasvinder <[email protected]<mailto:[email protected]>>; Liguzinski, WojciechX <[email protected]<mailto:[email protected]>>
Cc: dev <[email protected]<mailto:[email protected]>>; Aaron Conole <[email protected]<mailto:[email protected]>>; Yigit, Ferruh <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; Zegota, AnnaX <[email protected]<mailto:[email protected]>>
Subject: Re: [dpdk-dev] [Bug 826]red_autotestrandom failures18/11/2021 23:10, Liguzinski, WojciechX:
Quoted 2 lines
Hi,
Quoted 8 lines
Quoted 2 lines
I was trying to reproduce this test failure, but for me RED tests are passing. I was running the exact test command like the one described in Bug 826 -
'
red_autotest' on the current main branch.The test is not always failing.
There are some failing conditions, please find them.
I think you should try in a container with more limited resources.
Hi Thomas,
This is not a fair request IMO. We want to avoid wasting everybody's time, including Wojciech's time. Can the bug originator provide the details on the setup to reproduce the failure, please? Thank you!
On a different point, we should probably tweak our autotests to differentiate between logical failures and those failures related to resources not being available, and flag the test result accordingly in the report. For example, if memory allocation fails, the test should be flagged as "Not enough resources" instead of simply "Failed". In the first case, the next step should be fixing the test setup, while in the second case the next step should be fixing the code. What do people think on this?
Regards,
Cristian
--
Lincoln Lavoie
Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 [email protected]<mailto:[email protected]>
https://www.iol.unh.edu
+1-603-674-2755 (m)
[Image removed by sender.]<https://www.iol.unh.edu/>
Quoted 4 lines · WojciechX
Hi,
Quoted 9 lines
Thanks Lincoln, I will also have a try with such script.
Cheers,
Wojciech
Hello Wojciech,
I also recommend trying to run the test with around 4GB of RAM and 2GB of hugepages to see if it fails. That is roughly the number of resources we have per machine that is completely dedicated to unit tests. The amount of RAM available can sometimes increase depending on how many jobs are running per machine, but 4GB is the lowest it can go for the unit test job.
Thanks,
Brandon
Signature · 5 lines
Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 [email protected] www.iol.unh.edu
Ok, thanks Brandon for the tip :)
Let’s see if I can setup the machine with such configuration.
Cheers,
Wojciech
From: Brandon Lo <[email protected]>
Sent: Monday, November 29, 2021 6:58 PM
To: Liguzinski, WojciechX <[email protected]>
Cc: Lincoln Lavoie <[email protected]>; Dumitrescu, Cristian <[email protected]>; Thomas Monjalon <[email protected]>; David Marchand <[email protected]>; Ajmera, Megha <[email protected]>; Singh, Jasvinder <[email protected]>; dev <[email protected]>; Aaron Conole <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Zegota, AnnaX <[email protected]>
Subject: Re: [dpdk-dev] [Bug 826] red_autotest random failures
On Wed, Nov 24, 2021 at 2:48 AM Liguzinski, WojciechX <[email protected]<mailto:[email protected]>> wrote:
Hi,
Thanks Lincoln, I will also have a try with such script.
Cheers,
Wojciech
Hello Wojciech,
I also recommend trying to run the test with around 4GB of RAM and 2GB of hugepages to see if it fails. That is roughly the number of resources we have per machine that is completely dedicated to unit tests. The amount of RAM available can sometimes increase depending on how many jobs are running per machine, but 4GB is the lowest it can go for the unit test job.
Thanks,
Brandon
--
Brandon Lo
UNH InterOperability Laboratory
21 Madbury Rd, Suite 100, Durham, NH 03824
[email protected]<mailto:[email protected]>
www.iol.unh.edu<http://www.iol.unh.edu>
Hi,
Unfortunately, I haven’t been able to move the investigation much further.
I have been running those tests on machines with higher amount of RAM than 4GB, but with hugepages set there to 1GB and using the script provided by Lincoln. For several runs red_autotest tests didn’t fail even once, not giving any clue what might be the cause of what’s happening on CI.
+Adding Marcin Danilewicz
To let you know, Marcin Danilewicz will be taking over my tasks, so for any further aspects please include or direct messages to him.
Best Regards,
Wojciech
From: Liguzinski, WojciechX <[email protected]>
Sent: Tuesday, November 30, 2021 8:51 AM
To: Brandon Lo <[email protected]>
Cc: Lincoln Lavoie <[email protected]>; Dumitrescu, Cristian <[email protected]>; Thomas Monjalon <[email protected]>; David Marchand <[email protected]>; Ajmera, Megha <[email protected]>; Singh, Jasvinder <[email protected]>; dev <[email protected]>; Aaron Conole <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Zegota, AnnaX <[email protected]>
Subject: RE: [dpdk-dev] [Bug 826] red_autotest random failures
Ok, thanks Brandon for the tip :)
Let’s see if I can setup the machine with such configuration.
Cheers,
Wojciech
From: Brandon Lo <[email protected]<mailto:[email protected]>>
Sent: Monday, November 29, 2021 6:58 PM
To: Liguzinski, WojciechX <[email protected]<mailto:[email protected]>>
Cc: Lincoln Lavoie <[email protected]<mailto:[email protected]>>; Dumitrescu, Cristian <[email protected]<mailto:[email protected]>>; Thomas Monjalon <[email protected]<mailto:[email protected]>>; David Marchand <[email protected]<mailto:[email protected]>>; Ajmera, Megha <[email protected]<mailto:[email protected]>>; Singh, Jasvinder <[email protected]<mailto:[email protected]>>; dev <[email protected]<mailto:[email protected]>>; Aaron Conole <[email protected]<mailto:[email protected]>>; Yigit, Ferruh <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; Zegota, AnnaX <[email protected]<mailto:[email protected]>>
Subject: Re: [dpdk-dev] [Bug 826] red_autotest random failures
On Wed, Nov 24, 2021 at 2:48 AM Liguzinski, WojciechX <[email protected]<mailto:[email protected]>> wrote:
Hi,
Thanks Lincoln, I will also have a try with such script.
Cheers,
Wojciech
Hello Wojciech,
I also recommend trying to run the test with around 4GB of RAM and 2GB of hugepages to see if it fails. That is roughly the number of resources we have per machine that is completely dedicated to unit tests. The amount of RAM available can sometimes increase depending on how many jobs are running per machine, but 4GB is the lowest it can go for the unit test job.
Thanks,
Brandon
--
Brandon Lo
UNH InterOperability Laboratory
21 Madbury Rd, Suite 100, Durham, NH 03824
[email protected]<mailto:[email protected]>
www.iol.unh.edu<http://www.iol.unh.edu>