Skip to contentSkip to conversation

selftests/bpf: Add exponential backoff to map_update_retriable in test_maps

PATCH Applied Yucong Sun 4 participants 6 messages Aug 16, 2021 – Aug 17, 2021

Subsystems (3)
lore
Share & tools
lore

Yucong Sun

author
#0 lore raw

PATCH 1/1 selftests/bpf: Add exponential backoff to map_update_retriable in test_maps

Using a fixed delay of 1ms is proven flaky in slow CPU environment, eg. github action CI system. This patch adds exponential backoff with a cap of 50ms, to reduce the flakyness of the test.

---
 tools/testing/selftests/bpf/test_maps.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/test_maps.c +6 -1
index 14cea869235b..ed92d56c19cf 100644
@@ -1400,11 +1400,16 @@ static void test_map_stress(void)
 static int map_update_retriable(int map_fd, const void *key, const void *value,
 				int flags, int attempts)
 {
+	int delay = 1;
+
 	while (bpf_map_update_elem(map_fd, key, value, flags)) {
 		if (!attempts || (errno != EAGAIN && errno != EBUSY))
 			return -errno;
 
-		usleep(1);
+		if (delay < 50)
+			delay *= 2;
+
+		usleep(delay);
 		attempts--;
 	}
 
Signature · 1 line
2.30.2
reviewer
#1 lore raw
Quoted 4 lines · Yucong Sun

Using a fixed delay of 1ms is proven flaky in slow CPU environment, eg. github action CI system. This patch adds exponential backoff with a cap of 50ms, to reduce the flakyness of the test.

Do we have data showing how flaky the test is before and after this change?

Quoted 15 lines
---
 tools/testing/selftests/bpf/test_maps.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/test_maps.c +1 -0
index 14cea869235b..ed92d56c19cf 100644
@@ -1400,11 +1400,16 @@ static void test_map_stress(void)
 static int map_update_retriable(int map_fd, const void *key, const void *value,
                                int flags, int attempts)
 {
+       int delay = 1;

tools/testing/selftests/bpf/test_maps.c static void test_map_stress(void) · line 1412

Quoted 10 lines
tools/testing/selftests/bpf/test_maps.c +5 -1
+
        while (bpf_map_update_elem(map_fd, key, value, flags)) {
                if (!attempts || (errno != EAGAIN && errno != EBUSY))
                        return -errno;
 
-               usleep(1);
+               if (delay < 50)
+                       delay *= 2;
+
+               usleep(delay);

It is a little weird that the delay times in microseconds are 2, 4, 8, 16, 32, 64, 64, ...
Maybe just use rand()?

Thanks,
Song

Quoted 6 lines
tools/testing/selftests/bpf/test_maps.c +0 -1
                attempts--;
        }
 
--

2.30.2

#2 lore raw
Quoted 8 lines · Song Liu
Quoted 4 lines · Yucong Sun

Using a fixed delay of 1ms is proven flaky in slow CPU environment, eg. github action CI system. This patch adds exponential backoff with a cap of 50ms, to reduce the flakiness of the test.

Do we have data showing how flaky the test is before and after this change?

Before the change, on 2 CPU KVM on my laptop the test is perfectly fine, on Github action (2 emulated CPU) , it appeared to fail roughly 1 in 10 runs or even more frequently.
After the change, it appears pretty robust both on my laptop and on github action, I ran the github action a couple times and it succeeded every time.

Quoted 21 lines
Quoted 21 lines
---
 tools/testing/selftests/bpf/test_maps.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/test_maps.c +2 -0
index 14cea869235b..ed92d56c19cf 100644
@@ -1400,11 +1400,16 @@ static void test_map_stress(void)
 static int map_update_retriable(int map_fd, const void *key, const void *value,
                                int flags, int attempts)
 {
+       int delay = 1;
+
        while (bpf_map_update_elem(map_fd, key, value, flags)) {
                if (!attempts || (errno != EAGAIN && errno != EBUSY))
                        return -errno;
 
Quoted 9 lines

tools/testing/selftests/bpf/test_maps.c static void test_map_stress(void) · line 1412

Quoted 5 lines
tools/testing/selftests/bpf/test_maps.c +4 -1
-               usleep(1);
+               if (delay < 50)
+                       delay *= 2;
+
+               usleep(delay);

It is a little weird that the delay times in microseconds are 2, 4, 8, 16, 32, 64, 64, ...
Maybe just use rand()?

map_update_retriable is called by test_map_update() , which is being parallel executed in 1024 threads, so the lock contention is intentional, I think if we introduce randomness in the delay it kind of defeats the purpose of the test.
My original proposal is to just increase the attempts to 10X , Andrii proposed to use an exponential back-off, which is what I ended up implementing.

Quoted 10 lines

Thanks,
Song

Quoted 6 lines
tools/testing/selftests/bpf/test_maps.c +0 -1
                attempts--;
        }
 
--

2.30.2

reviewer Acked-by
#3 lore raw
Quoted 9 lines · On Mon, Aug 16, 2021 at 4:45 PM [email protected] <[email protected]> wrote:
Quoted 7 lines · Song Liu
Quoted 4 lines · Yucong Sun

Using a fixed delay of 1ms is proven flaky in slow CPU environment, eg. github action CI system. This patch adds exponential backoff with a cap of 50ms, to reduce the flakiness of the test.

Quoted 8 lines
Quoted 1 line

Do we have data showing how flaky the test is before and after this change?

Before the change, on 2 CPU KVM on my laptop the test is perfectly fine, on Github action (2 emulated CPU) , it appeared to fail roughly 1 in 10 runs or even more frequently.
After the change, it appears pretty robust both on my laptop and on github action, I ran the github action a couple times and it succeeded every time.

Thanks for the data!

We should include this in the commit log. Maybe the maintainer could just amend it when applying the patch.

Quoted 32 lines
Quoted 22 lines
Quoted 22 lines
---
 tools/testing/selftests/bpf/test_maps.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/test_maps.c +2 -0
index 14cea869235b..ed92d56c19cf 100644
@@ -1400,11 +1400,16 @@ static void test_map_stress(void)
 static int map_update_retriable(int map_fd, const void *key, const void *value,
                                int flags, int attempts)
 {
+       int delay = 1;
+
        while (bpf_map_update_elem(map_fd, key, value, flags)) {
                if (!attempts || (errno != EAGAIN && errno != EBUSY))
                        return -errno;
 
Quoted 9 lines

tools/testing/selftests/bpf/test_maps.c static void test_map_stress(void) · line 1412

Quoted 5 lines
tools/testing/selftests/bpf/test_maps.c +4 -1
-               usleep(1);
+               if (delay < 50)
+                       delay *= 2;
+
+               usleep(delay);

It is a little weird that the delay times in microseconds are 2, 4, 8, 16, 32, 64, 64, ...
Maybe just use rand()?

Quoted 7 lines

map_update_retriable is called by test_map_update() , which is being parallel executed in 1024 threads, so the lock contention is intentional, I think if we introduce randomness in the delay it kind of defeats the purpose of the test.
My original proposal is to just increase the attempts to 10X , Andrii proposed to use an exponential back-off, which is what I ended up implementing.

If that is what we agreed on, it works.

#4 lore raw
In reply to [email protected]
Quoted 49 lines · On Mon, Aug 16, 2021 at 4:45 PM [email protected] <[email protected]> wrote:
Quoted 8 lines · Song Liu
Quoted 4 lines · Yucong Sun

Using a fixed delay of 1ms is proven flaky in slow CPU environment, eg. github action CI system. This patch adds exponential backoff with a cap of 50ms, to reduce the flakiness of the test.

Do we have data showing how flaky the test is before and after this change?

Before the change, on 2 CPU KVM on my laptop the test is perfectly fine, on Github action (2 emulated CPU) , it appeared to fail roughly 1 in 10 runs or even more frequently.
After the change, it appears pretty robust both on my laptop and on github action, I ran the github action a couple times and it succeeded every time.

Quoted 21 lines
Quoted 21 lines
---
 tools/testing/selftests/bpf/test_maps.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/test_maps.c +2 -0
index 14cea869235b..ed92d56c19cf 100644
@@ -1400,11 +1400,16 @@ static void test_map_stress(void)
 static int map_update_retriable(int map_fd, const void *key, const void *value,
                                int flags, int attempts)
 {
+       int delay = 1;
+
        while (bpf_map_update_elem(map_fd, key, value, flags)) {
                if (!attempts || (errno != EAGAIN && errno != EBUSY))
                        return -errno;
 
Quoted 9 lines

tools/testing/selftests/bpf/test_maps.c static void test_map_stress(void) · line 1412

Quoted 5 lines
tools/testing/selftests/bpf/test_maps.c +4 -1
-               usleep(1);
+               if (delay < 50)
+                       delay *= 2;
+
+               usleep(delay);

It is a little weird that the delay times in microseconds are 2, 4, 8, 16, 32, 64, 64, ...
Maybe just use rand()?

Quoted 4 lines

map_update_retriable is called by test_map_update() , which is being parallel executed in 1024 threads, so the lock contention is intentional, I think if we introduce randomness in the delay it kind of defeats the purpose of the test.

Not really, the purpose of the test is to test a lot of concurrent updates with some collisions. It doesn't matter if there is one collision or 20. We will still get an initial collision (we only usleep() after the initial attempt fails with EAGAIN or EBUSY) even with randomized initial sleep *after* the first collision, but after that we should make sure that test eventually completes, so for that random initial delay is a good thing.

I reworked the code a bit, added initial random delay between [0ms, 5ms), and then actually capped delay under 50ms (it can't go to

Quoted 1 line

50ms). Note also that your code is actually working with microseconds

(usleep() takes microseconds), while commit was actually talking about milliseconds.

Applied to bpf-next, thanks.

Quoted 14 lines

My original proposal is to just increase the attempts to 10X , Andrii proposed to use an exponential back-off, which is what I ended up implementing.

Quoted 10 lines

Thanks,
Song

Quoted 6 lines
tools/testing/selftests/bpf/test_maps.c +0 -1
                attempts--;
        }
 
--

2.30.2

#5 lore raw
Quoted 64 lines · Andrii Nakryiko
Quoted 49 lines · On Mon, Aug 16, 2021 at 4:45 PM [email protected] <[email protected]> wrote:
Quoted 8 lines · Song Liu
Quoted 4 lines · Yucong Sun

Using a fixed delay of 1ms is proven flaky in slow CPU environment, eg. github action CI system. This patch adds exponential backoff with a cap of 50ms, to reduce the flakiness of the test.

Do we have data showing how flaky the test is before and after this change?

Before the change, on 2 CPU KVM on my laptop the test is perfectly fine, on Github action (2 emulated CPU) , it appeared to fail roughly 1 in 10 runs or even more frequently.
After the change, it appears pretty robust both on my laptop and on github action, I ran the github action a couple times and it succeeded every time.

Quoted 21 lines
Quoted 21 lines
---
 tools/testing/selftests/bpf/test_maps.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/test_maps.c +2 -0
index 14cea869235b..ed92d56c19cf 100644
@@ -1400,11 +1400,16 @@ static void test_map_stress(void)
 static int map_update_retriable(int map_fd, const void *key, const void *value,
                                int flags, int attempts)
 {
+       int delay = 1;
+
        while (bpf_map_update_elem(map_fd, key, value, flags)) {
                if (!attempts || (errno != EAGAIN && errno != EBUSY))
                        return -errno;
 
Quoted 9 lines

tools/testing/selftests/bpf/test_maps.c static void test_map_stress(void) · line 1412

Quoted 5 lines
tools/testing/selftests/bpf/test_maps.c +4 -1
-               usleep(1);
+               if (delay < 50)
+                       delay *= 2;
+
+               usleep(delay);

It is a little weird that the delay times in microseconds are 2, 4, 8, 16, 32, 64, 64, ...
Maybe just use rand()?

Quoted 4 lines

map_update_retriable is called by test_map_update() , which is being parallel executed in 1024 threads, so the lock contention is intentional, I think if we introduce randomness in the delay it kind of defeats the purpose of the test.

Not really, the purpose of the test is to test a lot of concurrent updates with some collisions. It doesn't matter if there is one collision or 20. We will still get an initial collision (we only usleep() after the initial attempt fails with EAGAIN or EBUSY) even with randomized initial sleep *after* the first collision, but after that we should make sure that test eventually completes, so for that random initial delay is a good thing.

Quoted 7 lines

I reworked the code a bit, added initial random delay between [0ms, 5ms), and then actually capped delay under 50ms (it can't go to

Quoted 1 line

50ms). Note also that your code is actually working with microseconds

(usleep() takes microseconds), while commit was actually talking about milliseconds.

Applied to bpf-next, thanks.

Fun, two out of three test runs now fail with (see [0], for an example):

error -16 16
test_maps: test_maps.c:1374: test_update_delete: Assertion `err == 0' failed.
test_maps: test_maps.c:1312: __run_parallel: Assertion `status == 0' failed. ./libbpf/travis-ci/vmtest/run_selftests.sh: line 19: 1226 Aborted

              ./test_maps

Seems like we need the same trick for bpf_map_delete_elem() calls?..

[0] https://github.com/kernel-patches/bpf/pull/1632/checks?check_run_id=3345916773

Quoted 15 lines
Quoted 15 lines

My original proposal is to just increase the attempts to 10X , Andrii proposed to use an exponential back-off, which is what I ended up implementing.

Quoted 10 lines

Thanks,
Song

Quoted 6 lines
tools/testing/selftests/bpf/test_maps.c +0 -1
                attempts--;
        }
 
--

2.30.2