net_sched: sch_fq: properly set sk->sk_pacing_status
Subsystems (3)
PATCH 1/1 net_sched: sch_fq: properly set sk->sk_pacing_status
If fq_classify() recycles a struct fq_flow because a socket structure has been reallocated, we do not set sk->sk_pacing_status immediately, but later if the flow becomes detached.
This means that any flow requiring pacing (BBR, or SO_MAX_PACING_RATE) might fallback to TCP internal pacing, which requires a per-socket high resolution timer, and therefore more cpu cycles.
--- net/sched/sch_fq.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
net/sched/sch_fq.c +8 -9
@@ -301,6 +301,9 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q) f->socket_hash != sk->sk_hash)) { f->credit = q->initial_quantum; f->socket_hash = sk->sk_hash; +if (q->rate_enable) +smp_store_release (&sk->sk_pacing_status, +SK_PACING_FQ );if (fq_flow_is_throttled (f))fq_flow_unset_throttled (q, f); f->time_next_packet =0ULL ;
@@ -322,8 +325,12 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q)fq_flow_set_detached (f); f->sk = sk; -if (skb->sk == sk) +if (skb->sk == sk) { f->socket_hash = sk->sk_hash; +if (q->rate_enable) +smp_store_release (&sk->sk_pacing_status, +SK_PACING_FQ ); + } f->credit = q->initial_quantum;rb_link_node (&f->fq_node, parent, p);
@@ -428,17 +435,9 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch, f->qlen++;qdisc_qstats_backlog_inc (sch, skb);if (fq_flow_is_detached (f)) { -struct sock *sk = skb->sk; -fq_flow_add_tail (&q->new_flows, f);if (time_after (jiffies, f->age + q->flow_refill_delay)) f->credit =max_t (u32, f->credit, q->quantum); -if (sk && q->rate_enable) { -if (unlikely (smp_load_acquire (&sk->sk_pacing_status) != -SK_PACING_FQ )) -smp_store_release (&sk->sk_pacing_status, -SK_PACING_FQ ); - } q->inactive_flows--; }
Signature · 1 line
2.24.1.735.g03f4e72817-goog
Quoted 6 lines · Eric Dumazet
If
fq_classify()recycles a structfq_flowbecause a socket structure has been reallocated, we do not setsk->sk_pacing_statusimmediately, but later if the flow becomes detached.
Quoted 8 lines
This means that any flow requiring pacing (BBR, or
SO_MAX_PACING_RATE) might fallback to TCP internal pacing, which requires a per-socket high resolution timer, and therefore more cpu cycles.Fixes 218af599fa63 ("tcp: internal implementation for pacing")Signed-off-by Eric Dumazet <[email protected]>Cc Soheil Hassas Yeganeh <[email protected]>Cc Neal Cardwell <[email protected]>
Nice catch! Thanks for the fix!
Quoted 53 lines
--- net/sched/sch_fq.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)net/sched/sch_fq.c +8
-10@@ -301,6 +301,9 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q) f->socket_hash != sk->sk_hash)) { f->credit = q->initial_quantum; f->socket_hash = sk->sk_hash; +if (q->rate_enable) +smp_store_release (&sk->sk_pacing_status, +SK_PACING_FQ );if (fq_flow_is_throttled (f))fq_flow_unset_throttled (q, f); f->time_next_packet =0ULL ;@@ -322,8 +325,12 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q)fq_flow_set_detached (f); f->sk = sk; -if (skb->sk == sk) +if (skb->sk == sk) { f->socket_hash = sk->sk_hash; +if (q->rate_enable) +smp_store_release (&sk->sk_pacing_status, +SK_PACING_FQ ); + } f->credit = q->initial_quantum;rb_link_node (&f->fq_node, parent, p);@@ -428,17 +435,9 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch, f->qlen++;qdisc_qstats_backlog_inc (sch, skb);if (fq_flow_is_detached (f)) { -struct sock *sk = skb->sk; -fq_flow_add_tail (&q->new_flows, f);if (time_after (jiffies, f->age + q->flow_refill_delay)) f->credit =max_t (u32, f->credit, q->quantum); -if (sk && q->rate_enable) { -if (unlikely (smp_load_acquire (&sk->sk_pacing_status) != -SK_PACING_FQ )) -smp_store_release (&sk->sk_pacing_status, -SK_PACING_FQ ); - } q->inactive_flows--; } --2.24.1.735.g03f4e72817-goog
David Miller
maintainerFrom: Eric Dumazet <[email protected]>
Date: Mon, 23 Dec 2019 11:13:24 -0800
Quoted 11 lines
If
fq_classify()recycles a structfq_flowbecause a socket structure has been reallocated, we do not setsk->sk_pacing_statusimmediately, but later if the flow becomes detached.This means that any flow requiring pacing (BBR, or
SO_MAX_PACING_RATE) might fallback to TCP internal pacing, which requires a per-socket high resolution timer, and therefore more cpu cycles.Fixes 218af599fa63 ("tcp: internal implementation for pacing")Signed-off-by Eric Dumazet <[email protected]>
Applied and queued up for -stable, thanks Eric.