net: tcp: un-pin tw timer
Subsystems (3)
v7 cover net: tcp: un-pin tw timer
This is v7 of the series where the tw_timer is un-pinned to get rid of interferences in isolated CPUs setups.
First patch makes necessary preparations, existing code relies on TIMER_PINNED to avoid races.
Second patch un-pins the TW timer. Could be folded into the first one, but it might help wrt. bisection.
Third patch is a minor cleanup to move a helper from .h to the only remaining compilation unit.
Tested with iperf3 and stress-ng socket mode.
Changes since previous iteration:
- use timer_shutdown_sync, ehash lock can be released before
- fix 'ddcp' typo in patch subject
Florian Westphal (2):
net: tcp: un-pin the tw_timer
tcp: move inet_twsk_schedule helper out of header
Valentin Schneider (1):
net: tcp/dccp: prepare for tw_timer un-pinning
include/net/inet_timewait_sock.h | 11 +++--- net/dccp/minisocks.c | 3 +- net/ipv4/inet_timewait_sock.c | 59 +++++++++++++++++++++++++++----- net/ipv4/tcp_minisocks.c | 3 +- 4 files changed, 57 insertions(+), 19 deletions(-)
Signature · 1 line
2.44.2
PATCH v7 1/3 net: tcp/dccp: prepare for tw_timer un-pinning
From: Valentin Schneider <[email protected]>
The TCP timewait timer is proving to be problematic for setups where scheduler CPU isolation is achieved at runtime via cpusets (as opposed to statically via isolcpus=domains).
What happens there is a CPU goes through tcp_time_wait(), arming the time_wait timer, then gets isolated. TCP_TIMEWAIT_LEN later, the timer fires, causing interference for the now-isolated CPU. This is conceptually similar to the issue described in commit e02b93124855 ("workqueue: Unbind kworkers before sending them to exit()")
Move inet_twsk_schedule() to within inet_twsk_hashdance(), with the ehash lock held. Expand the lock's critical section from inet_twsk_kill() to inet_twsk_deschedule_put(), serializing the scheduling vs descheduling of the timer. IOW, this prevents the following race:
tcp_time_wait() inet_twsk_hashdance()
inet_twsk_deschedule_put()
del_timer_sync()
inet_twsk_schedule()
Thanks to Paolo Abeni for suggesting to leverage the ehash lock.
This also restores a comment from commit ec94c2696f0b ("tcp/dccp: avoid one atomic operation for timewait hashdance") as inet_twsk_hashdance() had a "Step 1" and "Step 3" comment, but the "Step 2" had gone missing.
inet_twsk_deschedule_put() now acquires the ehash spinlock to synchronize with inet_twsk_hashdance_schedule().
To ease possible regression search, actual un-pin is done in next patch.
---
changes:
- use timer_shutdown_sync, ehash lock can be released beforehand
- fix 'ddcp' typo in patch subject
include/net/inet_timewait_sock.h | 6 ++-- net/dccp/minisocks.c | 3 +- net/ipv4/inet_timewait_sock.c | 52 +++++++++++++++++++++++++++----- net/ipv4/tcp_minisocks.c | 3 +- 4 files changed, 51 insertions(+), 13 deletions(-)
include/net/inet_timewait_sock.h +4 -2
@@ -93,8 +93,10 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk,struct inet_timewait_death_row *dr, const int state); -void inet_twsk_hashdance (struct inet_timewait_sock *tw,struct sock *sk, -struct inet_hashinfo *hashinfo); +void inet_twsk_hashdance_schedule (struct inet_timewait_sock *tw, +struct sock *sk, +struct inet_hashinfo *hashinfo, +int timeo);void __inet_twsk_schedule (struct inet_timewait_sock *tw,int timeo,bool rearm);
net/dccp/minisocks.c +1 -2
@@ -59,11 +59,10 @@ void dccp_time_wait(struct sock *sk, int state, int timeo)* we complete the initialization. */ local_bh_disable (); -inet_twsk_schedule (tw, timeo);/* Linkage updates. * Note that access to tw after this point is illegal. */ -inet_twsk_hashdance (tw, sk, &dccp_hashinfo); +inet_twsk_hashdance_schedule (tw, sk, &dccp_hashinfo, timeo);local_bh_enable (); }else {/* Sorry, if we're out of memory, just CLOSE this
net/ipv4/inet_timewait_sock.c +45 -7
87 lines not on this page · raw message
net/ipv4/tcp_minisocks.c +1 -2
@@ -344,11 +344,10 @@ void tcp_time_wait(struct sock *sk, int state, int timeo)* we complete the initialization. */ local_bh_disable (); -inet_twsk_schedule (tw, timeo);/* Linkage updates. * Note that access to tw after this point is illegal. */ -inet_twsk_hashdance (tw, sk, net->ipv4.tcp_death_row.hashinfo); +inet_twsk_hashdance_schedule (tw, sk, net->ipv4.tcp_death_row.hashinfo, timeo);local_bh_enable (); }else {/* Sorry, if we're out of memory, just CLOSE this
Signature · 1 line
2.44.2
Re: [PATCH net-next v7 1/3] net: tcp/dccp: prepare for tw_timer un-pinning
Quoted 30 lines · Florian Westphal
From: Valentin Schneider <[email protected]>
The TCP timewait timer is proving to be problematic for setups where scheduler CPU isolation is achieved at runtime via cpusets (as opposed to statically via isolcpus=domains).
What happens there is a CPU goes through
tcp_time_wait(), arming thetime_waittimer, then gets isolated.TCP_TIMEWAIT_LENlater, the timer fires, causing interference for the now-isolated CPU. This is conceptually similar to the issue described in commite02b93124855("workqueue: Unbind kworkers before sending them toexit()")Move
inet_twsk_schedule()to withininet_twsk_hashdance(), with the ehash lock held. Expand the lock's critical section frominet_twsk_kill()toinet_twsk_deschedule_put(), serializing the scheduling vs descheduling of the timer. IOW, this prevents the following race:tcp_time_wait() inet_twsk_hashdance()
inet_twsk_deschedule_put()
del_timer_sync()
inet_twsk_schedule()Thanks to Paolo Abeni for suggesting to leverage the ehash lock.
This also restores a comment from commit
ec94c2696f0b("tcp/dccp: avoid one atomic operation for timewait hashdance") asinet_twsk_hashdance()had a "Step 1" and "Step 3" comment, but the "Step 2" had gone missing.
Quoted 8 lines
inet_twsk_deschedule_put()now acquires the ehash spinlock to synchronize withinet_twsk_hashdance_schedule().To ease possible regression search, actual un-pin is done in next patch.
Signed-off-by Valentin Schneider <[email protected]>Signed-off-by Florian Westphal <[email protected]>
Thanks.
Re: [PATCH net-next v7 1/3] net: tcp/dccp: prepare for tw_timer un-pinning
In reply to Florian Westphalinclude/net/inet_timewait_sock.h struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, · line 99
Quoted 12 lines · Florian Westphal
include/net/inet_timewait_sock.h +4
-2@@ -93,8 +93,10 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk,struct inet_timewait_death_row *dr, const int state); -void inet_twsk_hashdance (struct inet_timewait_sock *tw,struct sock *sk, -struct inet_hashinfo *hashinfo); +void inet_twsk_hashdance_schedule (struct inet_timewait_sock *tw, +struct sock *sk, +struct inet_hashinfo *hashinfo, +int timeo);
Since this is gone now, could please also update the reference to in tcp_twsk_unique().
Sebastian
Re: [PATCH net-next v7 1/3] net: tcp/dccp: prepare for tw_timer un-pinning
In reply to Florian WestphalQuoted 21 lines · Florian Westphal
net/ipv4/inet_timewait_sock.c +16
-1@@ -217,7 +228,34 @@ EXPORT_SYMBOL_GPL(inet_twsk_alloc);*/ void inet_twsk_deschedule_put (struct inet_timewait_sock *tw) { -if (del_timer_sync (&tw->tw_timer)) +struct inet_hashinfo *hashinfo = tw->tw_dr->hashinfo; +spinlock_t *lock =inet_ehash_lockp (hashinfo, tw->tw_hash); + +/* inet_twsk_purge() walks over all sockets, including tw ones, +* and removes them via inet_twsk_deschedule_put() after a +* refcount_inc_not_zero(). +* +* inet_twsk_hashdance_schedule() must (re)init the refcount before +* arming the timer, i.e. inet_twsk_purge can obtain a reference to +* a twsk that did not yet schedule the timer. +* +* The ehash lock synchronizes these two: +* After acquiring the lock, the timer is always scheduled (else +* timer_shutdown returns false), because hashdance_schedule releases +* the ehash lock only after completing the timer initialization. +*
net/ipv4/inet_timewait_sock.c EXPORT_SYMBOL_GPL(inet_twsk_alloc); · line 256
Quoted 10 lines
net/ipv4/inet_timewait_sock.c +10
-0+* Without grabbing the ehash lock, we get: +* 1) cpu x sets twsk refcount to 3 +* 2) cpu y bumps refcount to 4 +* 3) cpu y calls inet_twsk_deschedule_put() and shuts timer down +* 4) cpu x tries to start timer, but mod_timer is a noop post-shutdown +* -> timer refcount is never decremented. +*/ +spin_lock (lock); +/* Makes sure hashdance_schedule() has completed */ +spin_unlock (lock);
Ah, this is clever! Thanks a ton for finding a way to get this to work!
PATCH v7 2/3 net: tcp: un-pin the tw_timer
In reply to Florian WestphalAfter previous patch, even if timer fires immediately on another CPU, context that schedules the timer now holds the ehash spinlock, so timer cannot reap tw socket until ehash lock is released.
---
no changes.
net/ipv4/inet_timewait_sock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/ipv4/inet_timewait_sock.c +1 -1
@@ -203,7 +203,7 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk, tw->tw_prot = sk->sk_prot_creator;atomic64_set (&tw->tw_cookie,atomic64_read (&sk->sk_cookie));twsk_net_set (tw,sock_net (sk)); -timer_setup (&tw->tw_timer, tw_timer_handler,TIMER_PINNED ); +timer_setup (&tw->tw_timer, tw_timer_handler,0 );/* * Because we use RCU lookups, we should not set tw_refcnt * to a non null value before everything is setup for this
Signature · 1 line
2.44.2
Re: [PATCH net-next v7 2/3] net: tcp: un-pin the tw_timer
Quoted 6 lines · Florian Westphal
After previous patch, even if timer fires immediately on another CPU, context that schedules the timer now holds the ehash spinlock, so timer cannot reap tw socket until ehash lock is released.
Signed-off-by Florian Westphal <[email protected]>
Re: [PATCH net-next v7 2/3] net: tcp: un-pin the tw_timer
In reply to Florian WestphalQuoted 3 lines · Florian Westphal
After previous patch, even if timer fires immediately on another CPU, context that schedules the timer now holds the ehash spinlock, so timer cannot reap tw socket until ehash lock is released.
The two callers of inet_twsk_hashdance_schedule() have the comment:
| /* tw_timer is pinned, so we need to make sure BH are disabled | * in following section, otherwise timer handler could run before | * we complete the initialization. | */
This isn't accurate anymore. I would suggest to simply remove the comment. The only reason for local_bh_disable() is (are) the lock(s).
Quoted 1 line
Signed-off-by Florian Westphal <[email protected]>
Sebastian
PATCH v7 3/3 tcp: move inet_twsk_schedule helper out of header
In reply to Florian WestphalIts no longer used outside inet_timewait_sock.c, so move it there.
---
no changes.
include/net/inet_timewait_sock.h | 5 ----- net/ipv4/inet_timewait_sock.c | 5 +++++ 2 files changed, 5 insertions(+), 5 deletions(-)
include/net/inet_timewait_sock.h +0 -5
@@ -101,11 +101,6 @@ void inet_twsk_hashdance_schedule(struct inet_timewait_sock *tw,void __inet_twsk_schedule (struct inet_timewait_sock *tw,int timeo,bool rearm); -static inline void inet_twsk_schedule (struct inet_timewait_sock *tw,int timeo) -{ -__inet_twsk_schedule (tw, timeo, false); -} -static inline void inet_twsk_reschedule (struct inet_timewait_sock *tw,int timeo) {__inet_twsk_schedule (tw, timeo, true);
net/ipv4/inet_timewait_sock.c +5 -0
@@ -92,6 +92,11 @@ static void inet_twsk_add_node_rcu(struct inet_timewait_sock *tw,hlist_nulls_add_head_rcu (&tw->tw_node, list); } +static void inet_twsk_schedule (struct inet_timewait_sock *tw,int timeo) +{ +__inet_twsk_schedule (tw, timeo, false); +} +/* * Enter the time wait state. This is called with locally disabled BH. * Essentially we whip up a timewait bucket, copy the relevant info into it
Signature · 1 line
2.44.2
Re: [PATCH net-next v7 3/3] tcp: move inet_twsk_schedule helper out of header
Quoted 4 lines · Florian Westphal
Its no longer used outside
inet_timewait_sock.c, so move it there.Signed-off-by Florian Westphal <[email protected]>
Quoted 5 lines · Florian Westphal
This is v7 of the series where the
tw_timeris un-pinned to get rid of interferences in isolated CPUs setups.First patch makes necessary preparations, existing code relies on
TIMER_PINNEDto avoid races.
And BH-disabled.
Thank you.
Sebastian
Quoted 9 lines · Florian Westphal
This is v7 of the series where the
tw_timeris un-pinned to get rid of interferences in isolated CPUs setups.First patch makes necessary preparations, existing code relies on
TIMER_PINNEDto avoid races.Second patch un-pins the TW timer. Could be folded into the first one, but it might help wrt. bisection.
Quoted 9 lines
Third patch is a minor cleanup to move a helper from .h to the only remaining compilation unit.
Tested with iperf3 and stress-ng socket mode.
Changes since previous iteration:
- usetimer_shutdown_sync, ehash lock can be released before
- fix 'ddcp' typo in patch subject
With Sebastian's suggested changes: