Flaky bpf cg_storage_* tests
Martin KaFai Lau
reviewerHi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to time. A recent example is https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924
Could you help to take a look? may be run it under netns and also have better filtering by ip/port when counting packets?
Thanks!
Quoted 7 lines · Martin KaFai Lau
Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to time. A recent example is
https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924Could you help to take a look? may be run it under netns and also have better filtering by ip/port when counting packets?
Error: #43/2 cg_storage_multi/isolated test_isolated:PASS:skel-load 0 nsec test_isolated:PASS:parent-egress1-cg-attach 0 nsec test_isolated:PASS:parent-egress2-cg-attach 0 nsec test_isolated:PASS:parent-ingress-cg-attach 0 nsec test_isolated:PASS:first-connect-send 0 nsec test_isolated:FAIL:first-invoke invocations=2
Error: #43/3 cg_storage_multi/shared
test_shared:PASS:skel-load 0 nsec
test_shared:PASS:parent-egress1-cg-attach 0 nsec
test_shared:PASS:parent-egress2-cg-attach 0 nsec
test_shared:PASS:parent-ingress-cg-attach 0 nsec
test_shared:PASS:first-connect-send 0 nsec
test_shared:FAIL:first-invoke invocations=2
Probably because we're using tcp? And race with syn vs syn+ack (invocatoins=1 vs invocations=2)?
YiFei, maybe we should count only pure syns?
Quoted 1 line
Thanks!
Quoted 26 lines · Stanislav Fomichev
Quoted 7 lines · Martin KaFai Lau
Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to time. A recent example is
https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924Could you help to take a look? may be run it under netns and also have better filtering by ip/port when counting packets?
Error: #43/2 cg_storage_multi/isolated test_isolated:PASS:skel-load 0 nsec test_isolated:PASS:parent-egress1-cg-attach 0 nsec test_isolated:PASS:parent-egress2-cg-attach 0 nsec test_isolated:PASS:parent-ingress-cg-attach 0 nsec test_isolated:PASS:first-connect-send 0 nsec test_isolated:FAIL:first-invoke invocations=2
Error: #43/3 cg_storage_multi/shared
test_shared:PASS:skel-load 0 nsec
test_shared:PASS:parent-egress1-cg-attach 0 nsec
test_shared:PASS:parent-egress2-cg-attach 0 nsec
test_shared:PASS:parent-ingress-cg-attach 0 nsec
test_shared:PASS:first-connect-send 0 nsec
test_shared:FAIL:first-invoke invocations=2
Quoted 2 lines
Probably because we're using tcp? And race with syn vs syn+ack (invocatoins=1 vs invocations=2)?
I don't remember what I wrote in the test :)
Nope it's not TCP. I see line 65:
server_fd = start_server(AF_INET, SOCK_DGRAM, NULL, 0, 0);
However I see line 169:
* Assert that there is three runs, two with parent cgroup egress and
* one with parent cgroup ingress, stored in separate parent storages.
Expected 3 got 2, is it possible we are racing against ingress?
Quoted 3 lines
YiFei, maybe we should count only pure syns?
Quoted 1 line
Thanks!
Quoted 33 lines · YiFei Zhu
Quoted 26 lines · Stanislav Fomichev
Quoted 7 lines · Martin KaFai Lau
Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to time. A recent example is
https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924Could you help to take a look? may be run it under netns and also have better filtering by ip/port when counting packets?
Error: #43/2 cg_storage_multi/isolated test_isolated:PASS:skel-load 0 nsec test_isolated:PASS:parent-egress1-cg-attach 0 nsec test_isolated:PASS:parent-egress2-cg-attach 0 nsec test_isolated:PASS:parent-ingress-cg-attach 0 nsec test_isolated:PASS:first-connect-send 0 nsec test_isolated:FAIL:first-invoke invocations=2
Error: #43/3 cg_storage_multi/shared
test_shared:PASS:skel-load 0 nsec
test_shared:PASS:parent-egress1-cg-attach 0 nsec
test_shared:PASS:parent-egress2-cg-attach 0 nsec
test_shared:PASS:parent-ingress-cg-attach 0 nsec
test_shared:PASS:first-connect-send 0 nsec
test_shared:FAIL:first-invoke invocations=2Quoted 2 lines
Probably because we're using tcp? And race with syn vs syn+ack (invocatoins=1 vs invocations=2)?
I don't remember what I wrote in the test :)
Quoted 8 lines
Nope it's not TCP. I see line 65:
server_fd=start_server(AF_INET, SOCK_DGRAM, NULL, 0, 0);However I see line 169:
* Assert that there is three runs, two with parent cgroup egress and
* one with parent cgroup ingress, stored in separate parent storages.Expected 3 got 2, is it possible we are racing against ingress?
Not sure. Can you try to see if you can reproduce locally?
Quoted 3 lines
Quoted 3 lines
YiFei, maybe we should count only pure syns?
Quoted 1 line
Thanks!