Skip to contentSkip to conversation

Flaky bpf cg_storage_* tests

DISCUSSION Martin KaFai Lau 3 participants 4 messages Mar 28, 2023 BPF

lore
Share & tools
lore
#1 lore raw
Quoted 7 lines · Martin KaFai Lau

Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to time. A recent example is
https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924

Could you help to take a look? may be run it under netns and also have better filtering by ip/port when counting packets?

Error: #43/2 cg_storage_multi/isolated test_isolated:PASS:skel-load 0 nsec test_isolated:PASS:parent-egress1-cg-attach 0 nsec test_isolated:PASS:parent-egress2-cg-attach 0 nsec test_isolated:PASS:parent-ingress-cg-attach 0 nsec test_isolated:PASS:first-connect-send 0 nsec test_isolated:FAIL:first-invoke invocations=2

Error: #43/3 cg_storage_multi/shared
test_shared:PASS:skel-load 0 nsec
test_shared:PASS:parent-egress1-cg-attach 0 nsec
test_shared:PASS:parent-egress2-cg-attach 0 nsec
test_shared:PASS:parent-ingress-cg-attach 0 nsec
test_shared:PASS:first-connect-send 0 nsec
test_shared:FAIL:first-invoke invocations=2

Probably because we're using tcp? And race with syn vs syn+ack (invocatoins=1 vs invocations=2)?
YiFei, maybe we should count only pure syns?

Quoted 1 line

Thanks!

#2 lore raw
Quoted 26 lines · Stanislav Fomichev
Quoted 7 lines · Martin KaFai Lau

Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to time. A recent example is
https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924

Could you help to take a look? may be run it under netns and also have better filtering by ip/port when counting packets?

Error: #43/2 cg_storage_multi/isolated test_isolated:PASS:skel-load 0 nsec test_isolated:PASS:parent-egress1-cg-attach 0 nsec test_isolated:PASS:parent-egress2-cg-attach 0 nsec test_isolated:PASS:parent-ingress-cg-attach 0 nsec test_isolated:PASS:first-connect-send 0 nsec test_isolated:FAIL:first-invoke invocations=2

Error: #43/3 cg_storage_multi/shared
test_shared:PASS:skel-load 0 nsec
test_shared:PASS:parent-egress1-cg-attach 0 nsec
test_shared:PASS:parent-egress2-cg-attach 0 nsec
test_shared:PASS:parent-ingress-cg-attach 0 nsec
test_shared:PASS:first-connect-send 0 nsec
test_shared:FAIL:first-invoke invocations=2

Quoted 2 lines

Probably because we're using tcp? And race with syn vs syn+ack (invocatoins=1 vs invocations=2)?

I don't remember what I wrote in the test :)

Nope it's not TCP. I see line 65:
server_fd = start_server(AF_INET, SOCK_DGRAM, NULL, 0, 0);

However I see line 169:
* Assert that there is three runs, two with parent cgroup egress and
* one with parent cgroup ingress, stored in separate parent storages.

Expected 3 got 2, is it possible we are racing against ingress?

Quoted 3 lines

YiFei, maybe we should count only pure syns?

Quoted 1 line

Thanks!

#3 lore raw
Quoted 33 lines · YiFei Zhu
Quoted 26 lines · Stanislav Fomichev
Quoted 7 lines · Martin KaFai Lau

Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to time. A recent example is
https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924

Could you help to take a look? may be run it under netns and also have better filtering by ip/port when counting packets?

Error: #43/2 cg_storage_multi/isolated test_isolated:PASS:skel-load 0 nsec test_isolated:PASS:parent-egress1-cg-attach 0 nsec test_isolated:PASS:parent-egress2-cg-attach 0 nsec test_isolated:PASS:parent-ingress-cg-attach 0 nsec test_isolated:PASS:first-connect-send 0 nsec test_isolated:FAIL:first-invoke invocations=2

Error: #43/3 cg_storage_multi/shared
test_shared:PASS:skel-load 0 nsec
test_shared:PASS:parent-egress1-cg-attach 0 nsec
test_shared:PASS:parent-egress2-cg-attach 0 nsec
test_shared:PASS:parent-ingress-cg-attach 0 nsec
test_shared:PASS:first-connect-send 0 nsec
test_shared:FAIL:first-invoke invocations=2

Quoted 2 lines

Probably because we're using tcp? And race with syn vs syn+ack (invocatoins=1 vs invocations=2)?

I don't remember what I wrote in the test :)

Quoted 8 lines

Nope it's not TCP. I see line 65:
server_fd = start_server(AF_INET, SOCK_DGRAM, NULL, 0, 0);

However I see line 169:
* Assert that there is three runs, two with parent cgroup egress and
* one with parent cgroup ingress, stored in separate parent storages.

Expected 3 got 2, is it possible we are racing against ingress?

Not sure. Can you try to see if you can reproduce locally?

Quoted 3 lines
Quoted 3 lines

YiFei, maybe we should count only pure syns?

Quoted 1 line

Thanks!