once: add DO_ONCE_SLOW() for sleepable contexts
Subsystems (4)
PATCH 1/1 once: add DO_ONCE_SLOW() for sleepable contexts
From: Eric Dumazet <[email protected]>
Christophe Leroy reported a ~80ms latency spike
happening at first TCP connect() time.
This is because __inet_hash_connect() uses get_random_once()
to populate a perturbation table which became quite big
after commit 4c2c8f03a5ab ("tcp: increase source port perturb table to 2^16")
get_random_once() uses DO_ONCE(), which block hard irqs for the duration of the operation.
This patch adds DO_ONCE_SLOW() which uses a mutex instead of a spinlock for operations where we prefer to stay in process context.
Then __inet_hash_connect() can use get_random_slow_once() to populate its perturbation table.
--- include/linux/once.h | 28 ++++++++++++++++++++++++++++ lib/once.c | 30 ++++++++++++++++++++++++++++++ net/ipv4/inet_hashtables.c | 4 ++-- 3 files changed, 60 insertions(+), 2 deletions(-)
include/linux/once.h +28 -0
@@ -5,10 +5,18 @@#include <linux/types.h> #include <linux/jump_label.h> +/* Helpers used from arbitrary contexts. +* Hard irqs are blocked, be cautious. +*/ bool __do_once_start (bool *done,unsigned long *flags);void __do_once_done (bool *done,struct static_key_true *once_key,unsigned long *flags,struct module *mod); +/* Variant for process contexts only. */ +bool __do_once_slow_start (bool *done); +void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod); +/* Call a function exactly once. The idea of DO_ONCE() is to perform * a function call such as initialization of random seeds, etc, only * once, where DO_ONCE() can live in the fast-path. After @func has
@@ -52,7 +60,27 @@ void __do_once_done(bool *done, struct static_key_true *once_key, ___ret; \ }) +/* Variant of DO_ONCE() for process/sleepable contexts. */ +#define DO_ONCE_SLOW (func, ...) \ + ({ \ + bool ___ret = false; \ + static bool __section(".data.once") ___done = false; \ + static DEFINE_STATIC_KEY_TRUE(___once_key); \ + if (static_branch_unlikely(&___once_key)) { \ + ___ret = __do_once_slow_start(&___done); \ + if (unlikely(___ret)) { \ + func(__VA_ARGS__); \ + __do_once_slow_done(&___done, &___once_key, \ + THIS_MODULE); \ + } \ + } \ + ___ret; \ + }) +#define get_random_once (buf, nbytes) \ DO_ONCE(get_random_bytes, (buf), (nbytes)) +#define get_random_slow_once (buf, nbytes) \ + DO_ONCE_SLOW(get_random_bytes, (buf), (nbytes)) +#endif /* _LINUX_ONCE_H */
lib/once.c +30 -0
@@ -66,3 +66,33 @@ void __do_once_done(bool *done, struct static_key_true *once_key,once_disable_jump (once_key, mod); }EXPORT_SYMBOL (__do_once_done); + +static DEFINE_MUTEX (once_mutex); + +bool __do_once_slow_start (bool *done) +__acquires (once_mutex) +{ +mutex_lock (&once_mutex); +if (*done) { +mutex_unlock (&once_mutex); +/* Keep sparse happy by restoring an even lock count on +* this mutex. In case we return here, we don't call into +* __do_once_done but return early in the DO_ONCE_SLOW() macro. +*/ +__acquire (once_mutex); +return false; + } + +return true; +} +EXPORT_SYMBOL (__do_once_slow_start); + +void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod) +__releases (once_mutex) +{ + *done = true; +mutex_unlock (&once_mutex); +once_disable_jump (once_key, mod); +} +EXPORT_SYMBOL (__do_once_slow_done);
net/ipv4/inet_hashtables.c +2 -2
@@ -958,8 +958,8 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,if (likely (remaining >1 )) remaining &= ~1U ; -net_get_random_once (table_perturb, -INET_TABLE_PERTURB_SIZE *sizeof (*table_perturb)); +get_random_slow_once (table_perturb, +INET_TABLE_PERTURB_SIZE *sizeof (*table_perturb)); index = port_offset & (INET_TABLE_PERTURB_SIZE -1 ); offset =READ_ONCE (table_perturb[index]) + (port_offset >>32 );
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Hi Eric,
Quoted 4 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Christophe Leroy reported a ~80ms latency spike
happening at first TCPconnect()time.
Seeing Christophe's message also made me wonder if we didn't break something back then :-/
Quoted 9 lines
This is because
__inet_hash_connect()usesget_random_once()
to populate a perturbation table which became quite big
after commit4c2c8f03a5ab("tcp: increase source port perturb table to 2^16")
get_random_once()usesDO_ONCE(), which block hard irqs for the duration of the operation.This patch adds
DO_ONCE_SLOW()which uses a mutex instead of a spinlock for operations where we prefer to stay in process context.
That's a nice improvement I think. I was wondering if, for this special case, we *really* need an exclusive DO_ONCE(). I mean, we're getting random bytes, we really do not care if two CPUs change them in parallel provided that none uses them before the table is entirely filled. Thus that could probably end up as something like:
if (!atomic_read(&done)) {
get_random_bytes(array);
atomic_set(&done, 1);
}In any case, your solution remains cleaner and more robust, though.
Thanks,
Willy
Quoted 27 lines · Willy Tarreau
Hi Eric,
Quoted 4 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Christophe Leroy reported a ~80ms latency spike
happening at first TCPconnect()time.Seeing Christophe's message also made me wonder if we didn't break something back then :-/
Quoted 9 lines
This is because
__inet_hash_connect()usesget_random_once()
to populate a perturbation table which became quite big
after commit4c2c8f03a5ab("tcp: increase source port perturb table to 2^16")
get_random_once()usesDO_ONCE(), which block hard irqs for the duration of the operation.This patch adds
DO_ONCE_SLOW()which uses a mutex instead of a spinlock for operations where we prefer to stay in process context.That's a nice improvement I think. I was wondering if, for this special case, we *really* need an exclusive
DO_ONCE(). I mean, we're getting random bytes, we really do not care if two CPUs change them in parallel provided that none uses them before the table is entirely filled. Thus that could probably end up as something like:
Quoted 4 lines
if (!atomic_read(&done)) {
get_random_bytes(array);
atomic_set(&done, 1);
}If you don't care about the tables being consistent between CPUs, then yea, sure, that seems like a reasonable approach, and I like not polluting once.{c,h} with some _SLOW() special cases. If you don't want the atomic read in there you could also do the same pattern with a static branch, like what DO_ONCE() does:
if (static_branch_unlikely(&need_bytes)) {
get_random_bytes(array);
static_branch_disable(&need_bytes);
}Anyway, same thing as your suggestion more or less.
Jason
Quoted 9 lines · Jason A. Donenfeld
Quoted 9 lines
Quoted 2 lines
This patch adds
DO_ONCE_SLOW()which uses a mutex instead of a spinlock for operations where we prefer to stay in process context.That's a nice improvement I think. I was wondering if, for this special case, we *really* need an exclusive
DO_ONCE(). I mean, we're getting random bytes, we really do not care if two CPUs change them in parallel provided that none uses them before the table is entirely filled. Thus that could probably end up as something like:
Quoted 8 lines
Quoted 4 lines
if (!atomic_read(&done)) { get_random_bytes(array); atomic_set(&done, 1); }If you don't care about the tables being consistent between CPUs, then yea, sure, that seems like a reasonable approach, and I like not polluting once.{c,h} with some
_SLOW()special cases.
I don't see this as pollution, it possibly is a nice addition for certain use cases or early fast paths where the risk of contention is high.
Quoted 10 lines
If you don't want
the atomic read in there you could also do the same pattern with a static branch, like whatDO_ONCE()does:if (static_branch_unlikely(&need_bytes)) { get_random_bytes(array); static_branch_disable(&need_bytes); }Anyway, same thing as your suggestion more or less.
What I don't know in fact is if the code patching itself can be responsible for a measurable part of the extra time Christophe noticed. Anyway at least Christophe now has a few approaches to try, let's first see if any of them fixes the regression.
Willy
Quoted 64 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Christophe Leroy reported a ~80ms latency spike
happening at first TCPconnect()time.This is because
__inet_hash_connect()usesget_random_once()
to populate a perturbation table which became quite big
after commit4c2c8f03a5ab("tcp: increase source port perturb table to 2^16")
get_random_once()usesDO_ONCE(), which block hard irqs for the duration of the operation.This patch adds
DO_ONCE_SLOW()which uses a mutex instead of a spinlock for operations where we prefer to stay in process context.Then
__inet_hash_connect()can useget_random_slow_once()to populate its perturbation table.Fixes 4c2c8f03a5ab ("tcp: increase source port perturb table to 2^16")Fixes 190cc82489f4 ("tcp: change source port randomizarion at connect() time")Reported-by Christophe Leroy <[email protected]>Link https://lore.kernel.org/netdev/CANn89iLAEYBaoYajy0Y9UmGFff5GPxDUoG-ErVB2jDdRNQ5Tug@mail.gmail.com/T/#t open hereSigned-off-by Eric Dumazet <[email protected]>Cc Willy Tarreau <[email protected]>--- include/linux/once.h | 28 ++++++++++++++++++++++++++++ lib/once.c | 30 ++++++++++++++++++++++++++++++ net/ipv4/inet_hashtables.c | 4 ++-- 3 files changed, 60 insertions(+), 2 deletions(-)include/linux/once.h +15
-0@@ -5,10 +5,18 @@#include <linux/types.h> #include <linux/jump_label.h> +/* Helpers used from arbitrary contexts. +* Hard irqs are blocked, be cautious. +*/ bool __do_once_start (bool *done,unsigned long *flags);void __do_once_done (bool *done,struct static_key_true *once_key,unsigned long *flags,struct module *mod); +/* Variant for process contexts only. */ +bool __do_once_slow_start (bool *done); +void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod); +/* Call a function exactly once. The idea of DO_ONCE() is to perform * a function call such as initialization of random seeds, etc, only * once, where DO_ONCE() can live in the fast-path. After @func has @@ -52,7 +60,27 @@ void __do_once_done(bool *done, struct static_key_true *once_key, ___ret; \ }) +/* Variant of DO_ONCE() for process/sleepable contexts. */ +#define DO_ONCE_SLOW (func, ...) \ + ({ \ + bool ___ret = false; \ + static bool __section(".data.once") ___done = false; \ + static DEFINE_STATIC_KEY_TRUE(___once_key); \ + if (static_branch_unlikely(&___once_key)) { \
include/linux/once.h void __do_once_done(bool *done, struct static_key_true *once_key, · line 79
Quoted 10 lines
include/linux/once.h +10
-0+ ___ret =__do_once_slow_start (&___done); \ +if (unlikely (___ret)) { \ +func (__VA_ARGS__); \ +__do_once_slow_done (&___done, &___once_key, \ +THIS_MODULE ); \ + } \ + } \ + ___ret; \ + }) +
Hmm, I dunno about this macro-choice explosion here. The whole thing with DO_ONCE() is that the static branch makes it zero cost most of the time while being somewhat expensive the rest of the time, but who cares, because "the rest" is just once.
So instead, why not just branch on whether or not we can sleep here, if that can be worked out dynamically? If not, and if you really do need two sets of macros and functions, at least you can call the new one something other than "slow"? Maybe something about being _SLEEPABLE() instead?
Also, the __do_once_slow_done() function misses a really nice optimization, which is that the static branch can be changed synchronously instead of having to allocate and fire off that workqueue, since by definition we're in sleepable context here.
Jason
Quoted 82 lines · Jason A. Donenfeld
Quoted 64 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Christophe Leroy reported a ~80ms latency spike
happening at first TCPconnect()time.This is because
__inet_hash_connect()usesget_random_once()
to populate a perturbation table which became quite big
after commit4c2c8f03a5ab("tcp: increase source port perturb table to 2^16")
get_random_once()usesDO_ONCE(), which block hard irqs for the duration of the operation.This patch adds
DO_ONCE_SLOW()which uses a mutex instead of a spinlock for operations where we prefer to stay in process context.Then
__inet_hash_connect()can useget_random_slow_once()to populate its perturbation table.Fixes 4c2c8f03a5ab ("tcp: increase source port perturb table to 2^16")Fixes 190cc82489f4 ("tcp: change source port randomizarion at connect() time")Reported-by Christophe Leroy <[email protected]>Link https://lore.kernel.org/netdev/CANn89iLAEYBaoYajy0Y9UmGFff5GPxDUoG-ErVB2jDdRNQ5Tug@mail.gmail.com/T/#t open hereSigned-off-by Eric Dumazet <[email protected]>Cc Willy Tarreau <[email protected]>--- include/linux/once.h | 28 ++++++++++++++++++++++++++++ lib/once.c | 30 ++++++++++++++++++++++++++++++ net/ipv4/inet_hashtables.c | 4 ++-- 3 files changed, 60 insertions(+), 2 deletions(-)include/linux/once.h +15
-0@@ -5,10 +5,18 @@#include <linux/types.h> #include <linux/jump_label.h> +/* Helpers used from arbitrary contexts. +* Hard irqs are blocked, be cautious. +*/ bool __do_once_start (bool *done,unsigned long *flags);void __do_once_done (bool *done,struct static_key_true *once_key,unsigned long *flags,struct module *mod); +/* Variant for process contexts only. */ +bool __do_once_slow_start (bool *done); +void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod); +/* Call a function exactly once. The idea of DO_ONCE() is to perform * a function call such as initialization of random seeds, etc, only * once, where DO_ONCE() can live in the fast-path. After @func has @@ -52,7 +60,27 @@ void __do_once_done(bool *done, struct static_key_true *once_key, ___ret; \ }) +/* Variant of DO_ONCE() for process/sleepable contexts. */ +#define DO_ONCE_SLOW (func, ...) \ + ({ \ + bool ___ret = false; \ + static bool __section(".data.once") ___done = false; \ + static DEFINE_STATIC_KEY_TRUE(___once_key); \ + if (static_branch_unlikely(&___once_key)) { \include/linux/once.h void __do_once_done(bool *done, struct static_key_true *once_key, · line 79
Quoted 10 lines
include/linux/once.h +10
-0+ ___ret =__do_once_slow_start (&___done); \ +if (unlikely (___ret)) { \ +func (__VA_ARGS__); \ +__do_once_slow_done (&___done, &___once_key, \ +THIS_MODULE ); \ + } \ + } \ + ___ret; \ + }) +Hmm, I dunno about this macro-choice explosion here. The whole thing with
DO_ONCE()is that the static branch makes it zero cost most of the time while being somewhat expensive the rest of the time, but who cares, because "the rest" is just once.
Quoted 5 lines
So instead, why not just branch on whether or not we can sleep here, if that can be worked out dynamically? If not, and if you really do need two sets of macros and functions, at least you can call the new one something other than "slow"? Maybe something about being
_SLEEPABLE()instead?
No idea what you mean. I do not want to over engineer code that yet have to be adopted by other callers. If you think you want to spend week end time on this, feel free to take over at this point.
Quoted 6 lines
Also, the
__do_once_slow_done()function misses a really nice optimization, which is that the static branch can be changed synchronously instead of having to allocate and fire off that workqueue, since by definition we're in sleepable context here.
This was deliberate. We already spent a lot of time in the called function, better just return to the caller as fast as possible.
This really does not matter, the work queue is fired once by definition.
Jakub Kicinski
maintainerQuoted 2 lines · On Sun, 2 Oct 2022 00:44:28 +0200 Jason A. Donenfeld wrote:
So instead, why not just branch on whether or not we can sleep here, if that can be worked out dynamically?
IDK if we can dynamically work out if _all_ _possible_ callers are in a specific context, can we?
Quoted 3 lines
If not, and if you really do need two sets of macros and functions, at least you can call the new one something other than "slow"? Maybe something about being
_SLEEPABLE()instead?
+1 for s/SLOW/SLEEPABLE/. I was about to suggest s/SLOW/TASK/. But I guess it's already applied..
Quoted 8 lines · Jakub Kicinski
Quoted 2 lines · On Sun, 2 Oct 2022 00:44:28 +0200 Jason A. Donenfeld wrote:
So instead, why not just branch on whether or not we can sleep here, if that can be worked out dynamically?
IDK if we can dynamically work out if _all_ _possible_ callers are in a specific context, can we?
Quoted 6 lines
Quoted 3 lines
If not, and if you really do need two sets of macros and functions, at least you can call the new one something other than "slow"? Maybe something about being
_SLEEPABLE()instead?+1 for s/SLOW/SLEEPABLE/. I was about to suggest s/SLOW/TASK/. But I guess it's already applied..
I'll send a patch to change it in a minute.
PATCH 1/1 once: rename _SLOW to _SLEEPABLE
The _SLOW designation wasn't really descriptive of anything. This is meant to be called from process context when it's possible to sleep. So name this more aptly _SLEEPABLE, which better fits its intended use.
--- include/linux/once.h | 38 +++++++++++++++++++------------------- lib/once.c | 10 +++++----- net/ipv4/inet_hashtables.c | 4 ++-- 3 files changed, 26 insertions(+), 26 deletions(-)
include/linux/once.h +19 -19
@@ -13,9 +13,9 @@ void __do_once_done(bool *done, struct static_key_true *once_key, unsigned long *flags, struct module *mod);/* Variant for process contexts only. */ -bool __do_once_slow_start (bool *done); -void __do_once_slow_done (bool *done,struct static_key_true *once_key, -struct module *mod); +bool __do_once_sleepable_start (bool *done); +void __do_once_sleepable_done (bool *done,struct static_key_true *once_key, +struct module *mod);/* Call a function exactly once. The idea of DO_ONCE() is to perform * a function call such as initialization of random seeds, etc, only
@@ -61,26 +61,26 @@ void __do_once_slow_done(bool *done, struct static_key_true *once_key, })/* Variant of DO_ONCE() for process/sleepable contexts. */ -#define DO_ONCE_SLOW (func, ...) \ - ({ \ - bool ___ret = false; \ - static bool __section(".data.once") ___done = false; \ - static DEFINE_STATIC_KEY_TRUE(___once_key); \ - if (static_branch_unlikely(&___once_key)) { \ - ___ret = __do_once_slow_start(&___done); \ - if (unlikely(___ret)) { \ - func(__VA_ARGS__); \ - __do_once_slow_done(&___done, &___once_key, \ - THIS_MODULE); \ - } \ - } \ - ___ret; \ +#define DO_ONCE_SLEEPABLE (func, ...) \ + ({ \ + bool ___ret = false; \ + static bool __section(".data.once") ___done = false; \ + static DEFINE_STATIC_KEY_TRUE(___once_key); \ + if (static_branch_unlikely(&___once_key)) { \ + ___ret = __do_once_sleepable_start(&___done); \ + if (unlikely(___ret)) { \ + func(__VA_ARGS__); \ + __do_once_sleepable_done(&___done, &___once_key,\ + THIS_MODULE); \ + } \ + } \ + ___ret; \ })#define get_random_once (buf, nbytes) \ DO_ONCE(get_random_bytes, (buf), (nbytes)) -#define get_random_slow_once (buf, nbytes) \ - DO_ONCE_SLOW(get_random_bytes, (buf), (nbytes)) +#define get_random_sleepable_once (buf, nbytes) \ + DO_ONCE_SLEEPABLE(get_random_bytes, (buf), (nbytes))#endif /* _LINUX_ONCE_H */
lib/once.c +5 -5
@@ -69,7 +69,7 @@ EXPORT_SYMBOL(__do_once_done);static DEFINE_MUTEX (once_mutex); -bool __do_once_slow_start (bool *done) +bool __do_once_sleepable_start (bool *done)__acquires (once_mutex) {mutex_lock (&once_mutex);
@@ -77,7 +77,7 @@ bool __do_once_slow_start(bool *done)mutex_unlock (&once_mutex);/* Keep sparse happy by restoring an even lock count on * this mutex. In case we return here, we don't call into -* __do_once_done but return early in the DO_ONCE_SLOW() macro. +* __do_once_done but return early in the DO_ONCE_SLEEPABLE() macro. */ __acquire (once_mutex);return false;
@@ -85,9 +85,9 @@ bool __do_once_slow_start(bool *done)return true; } -EXPORT_SYMBOL (__do_once_slow_start); +EXPORT_SYMBOL (__do_once_sleepable_start); -void __do_once_slow_done (bool *done,struct static_key_true *once_key, +void __do_once_sleepable_done (bool *done,struct static_key_true *once_key,struct module *mod)__releases (once_mutex) {
@@ -95,4 +95,4 @@ void __do_once_slow_done(bool *done, struct static_key_true *once_key,mutex_unlock (&once_mutex);once_disable_jump (once_key, mod); } -EXPORT_SYMBOL (__do_once_slow_done); +EXPORT_SYMBOL (__do_once_sleepable_done);
net/ipv4/inet_hashtables.c +2 -2
@@ -958,8 +958,8 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,if (likely (remaining >1 )) remaining &= ~1U ; -get_random_slow_once (table_perturb, -INET_TABLE_PERTURB_SIZE *sizeof (*table_perturb)); +get_random_sleepable_once (table_perturb, +INET_TABLE_PERTURB_SIZE *sizeof (*table_perturb)); index = port_offset & (INET_TABLE_PERTURB_SIZE -1 ); offset =READ_ONCE (table_perturb[index]) + (port_offset >>32 );
Signature · 1 line
2.37.3
Re: [PATCH] once: rename _SLOW to _SLEEPABLE
Quoted 6 lines · Jason A. Donenfeld
The
_SLOWdesignation wasn't really descriptive of anything. This is meant to be called from process context when it's possible to sleep. So name this more aptly_SLEEPABLE, which better fits its intended use.Fixes 62c07983bef9 ("once: add DO_ONCE_SLOW() for sleepable contexts")
Yes, this works for me, thank you.
Le 01/10/2022 à 22:51, Eric Dumazet a écrit :
Quoted 9 lines
From: Eric Dumazet <[email protected]>
Christophe Leroy reported a ~80ms latency spike
happening at first TCPconnect()time.This is because
__inet_hash_connect()usesget_random_once()
to populate a perturbation table which became quite big
after commit4c2c8f03a5ab("tcp: increase source port perturb table to 2^16")
Quoted 8 lines
get_random_once()usesDO_ONCE(), which block hard irqs for the duration of the operation.This patch adds
DO_ONCE_SLOW()which uses a mutex instead of a spinlock for operations where we prefer to stay in process context.Then
__inet_hash_connect()can useget_random_slow_once()to populate its perturbation table.
Many thanks for your quick answer and your patch.
It works great, now the irqsoff tracer reports a 2ms latency in a spi transfert. So the issue with tcp connect is gone.
Quoted 7 lines
Fixes 4c2c8f03a5ab ("tcp: increase source port perturb table to 2^16")Fixes 190cc82489f4 ("tcp: change source port randomizarion at connect() time")Reported-by Christophe Leroy <[email protected]>Link https://lore.kernel.org/netdev/CANn89iLAEYBaoYajy0Y9UmGFff5GPxDUoG-ErVB2jDdRNQ5Tug@mail.gmail.com/T/#t open hereSigned-off-by Eric Dumazet <[email protected]>Cc Willy Tarreau <[email protected]>
Quoted 113 lines
--- include/linux/once.h | 28 ++++++++++++++++++++++++++++ lib/once.c | 30 ++++++++++++++++++++++++++++++ net/ipv4/inet_hashtables.c | 4 ++-- 3 files changed, 60 insertions(+), 2 deletions(-)include/linux/once.h +28
-0@@ -5,10 +5,18 @@#include <linux/types.h> #include <linux/jump_label.h> +/* Helpers used from arbitrary contexts. +* Hard irqs are blocked, be cautious. +*/ bool __do_once_start (bool *done,unsigned long *flags);void __do_once_done (bool *done,struct static_key_true *once_key,unsigned long *flags,struct module *mod); +/* Variant for process contexts only. */ +bool __do_once_slow_start (bool *done); +void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod); +/* Call a function exactly once. The idea of DO_ONCE() is to perform * a function call such as initialization of random seeds, etc, only * once, where DO_ONCE() can live in the fast-path. After @func has @@ -52,7 +60,27 @@ void __do_once_done(bool *done, struct static_key_true *once_key, ___ret; \ }) +/* Variant of DO_ONCE() for process/sleepable contexts. */ +#define DO_ONCE_SLOW (func, ...) \ + ({ \ + bool ___ret = false; \ + static bool __section(".data.once") ___done = false; \ + static DEFINE_STATIC_KEY_TRUE(___once_key); \ + if (static_branch_unlikely(&___once_key)) { \ + ___ret = __do_once_slow_start(&___done); \ + if (unlikely(___ret)) { \ + func(__VA_ARGS__); \ + __do_once_slow_done(&___done, &___once_key, \ + THIS_MODULE); \ + } \ + } \ + ___ret; \ + }) +#define get_random_once (buf, nbytes) \ DO_ONCE(get_random_bytes, (buf), (nbytes)) +#define get_random_slow_once (buf, nbytes) \ + DO_ONCE_SLOW(get_random_bytes, (buf), (nbytes)) +#endif /* _LINUX_ONCE_H */ lib/once.c +30
-0@@ -66,3 +66,33 @@ void __do_once_done(bool *done, struct static_key_true *once_key,once_disable_jump (once_key, mod); }EXPORT_SYMBOL (__do_once_done); + +static DEFINE_MUTEX (once_mutex); + +bool __do_once_slow_start (bool *done) +__acquires (once_mutex) +{ +mutex_lock (&once_mutex); +if (*done) { +mutex_unlock (&once_mutex); +/* Keep sparse happy by restoring an even lock count on +* this mutex. In case we return here, we don't call into +* __do_once_done but return early in the DO_ONCE_SLOW() macro. +*/ +__acquire (once_mutex); +return false; + } + +return true; +} +EXPORT_SYMBOL (__do_once_slow_start); + +void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod) +__releases (once_mutex) +{ + *done = true; +mutex_unlock (&once_mutex); +once_disable_jump (once_key, mod); +} +EXPORT_SYMBOL (__do_once_slow_done);net/ipv4/inet_hashtables.c +2
-3@@ -958,8 +958,8 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,if (likely (remaining >1 )) remaining &= ~1U ; -net_get_random_once (table_perturb, -INET_TABLE_PERTURB_SIZE *sizeof (*table_perturb)); +get_random_slow_once (table_perturb, +INET_TABLE_PERTURB_SIZE *sizeof (*table_perturb)); index = port_offset & (INET_TABLE_PERTURB_SIZE -1 ); offset =READ_ONCE (table_perturb[index]) + (port_offset >>32 ); --2.38.0.rc1.362.ged0d419d3c-goog
This patch was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Quoted 10 lines · On Sat, 1 Oct 2022 13:51:02 -0700 you wrote:
From: Eric Dumazet <[email protected]>
Christophe Leroy reported a ~80ms latency spike
happening at first TCPconnect()time.This is because
__inet_hash_connect()usesget_random_once()
to populate a perturbation table which became quite big
after commit4c2c8f03a5ab("tcp: increase source port perturb table to 2^16")[...]
Here is the summary with links:
- [net-next] once: add DO_ONCE_SLOW() for sleepable contexts
https://git.kernel.org/netdev/net-next/c/62c07983bef9
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
lib/once.c void __do_once_done(bool *done, struct static_key_true *once_key, · line 96
Quoted 7 lines · Eric Dumazet
lib/once.c +7
-0+void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod) +__releases (once_mutex) +{ + *done = true; +mutex_unlock (&once_mutex); +once_disable_jump (once_key, mod);
This seems to have been cut & pasted from __do_once_done(). But is there a reason for the "sleepable" version to defer resetting the static key in a work queue? Can't we just inline do:
BUG_ON(!static_key_enabled(once_key)); static_branch_disable(once_key);
Quoted 1 line
+}
-Tony
Credit to Reinette for raising this question. Blame me if I didn't spot why a work queue is needed.
Quoted 17 lines · Tony
lib/once.c void __do_once_done(bool *done, struct static_key_true *once_key, · line 96
Quoted 7 lines · Eric Dumazet
lib/once.c +7
-0+void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod) +__releases (once_mutex) +{ + *done = true; +mutex_unlock (&once_mutex); +once_disable_jump (once_key, mod);This seems to have been cut & pasted from
__do_once_done(). But is there a reason for the "sleepable" version to defer resetting the static key in a work queue? Can't we just inline do:BUG_ON(!static_key_enabled(once_key)); static_branch_disable(once_key);
Quoted 6 lines
Quoted 1 line
+}
-Tony
Credit to Reinette for raising this question. Blame me if I didn't spot why a work queue is needed.
Note this is used from one single place, one time...
Why would you care ?
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Tony
lib/once.c void __do_once_done(bool *done, struct static_key_true *once_key, · line 96
Quoted 7 lines · Eric Dumazet
lib/once.c +7
-0+void __do_once_slow_done (bool *done,struct static_key_true *once_key, +struct module *mod) +__releases (once_mutex) +{ + *done = true; +mutex_unlock (&once_mutex); +once_disable_jump (once_key, mod);This seems to have been cut & pasted from
__do_once_done(). But is there a reason for the "sleepable" version to defer resetting the static key in a work queue? Can't we just inline do:BUG_ON(!static_key_enabled(once_key)); static_branch_disable(once_key);
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
+}
-Tony
Credit to Reinette for raising this question. Blame me if I didn't spot why a work queue is needed.
Note this is used from one single place, one time...
Boris pointed me to <linux/once.h> when reviewing a patch with open-coded "do this once" section. So there may be about to be a second user. Perhaps more to follow as the DO_ONCE*() macros look to be a very neat, concise, and obvious way to call a function just once.
Quoted 1 line
Why would you care ?
It looks like unnecessary overhead, but I wondered if there was something I was missing. Some reason why the SLEEPABLE version needed to delay clearing the static key to a work queue.
-Tony