Skip to contentSkip to conversation

net_sched: sch_fq: properly set sk->sk_pacing_status

PATCH In review Eric Dumazet 3 participants 3 messages Dec 23, 2019 – Dec 26, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net_sched: sch_fq: properly set sk->sk_pacing_status

If fq_classify() recycles a struct fq_flow because a socket structure has been reallocated, we do not set sk->sk_pacing_status immediately, but later if the flow becomes detached.

This means that any flow requiring pacing (BBR, or SO_MAX_PACING_RATE) might fallback to TCP internal pacing, which requires a per-socket high resolution timer, and therefore more cpu cycles.

Fixes 218af599fa63 ("tcp: internal implementation for pacing")
Cc Soheil Hassas Yeganeh <[email protected]>
Cc Neal Cardwell <[email protected]>
---
 net/sched/sch_fq.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
net/sched/sch_fq.c +8 -9
index b1c7e726ce5d1ae139f765c5b92dfdaea9bee258..ff4c5e9d0d7778d86f20f4bd67cc627eed0713d9 100644
@@ -301,6 +301,9 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q)
 				     f->socket_hash != sk->sk_hash)) {
 				f->credit = q->initial_quantum;
 				f->socket_hash = sk->sk_hash;
+				if (q->rate_enable)
+					smp_store_release(&sk->sk_pacing_status,
+							  SK_PACING_FQ);
 				if (fq_flow_is_throttled(f))
 					fq_flow_unset_throttled(q, f);
 				f->time_next_packet = 0ULL;
@@ -322,8 +325,12 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q)
 
 	fq_flow_set_detached(f);
 	f->sk = sk;
-	if (skb->sk == sk)
+	if (skb->sk == sk) {
 		f->socket_hash = sk->sk_hash;
+		if (q->rate_enable)
+			smp_store_release(&sk->sk_pacing_status,
+					  SK_PACING_FQ);
+	}
 	f->credit = q->initial_quantum;
 
 	rb_link_node(&f->fq_node, parent, p);
@@ -428,17 +435,9 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
 	f->qlen++;
 	qdisc_qstats_backlog_inc(sch, skb);
 	if (fq_flow_is_detached(f)) {
-		struct sock *sk = skb->sk;
-
 		fq_flow_add_tail(&q->new_flows, f);
 		if (time_after(jiffies, f->age + q->flow_refill_delay))
 			f->credit = max_t(u32, f->credit, q->quantum);
-		if (sk && q->rate_enable) {
-			if (unlikely(smp_load_acquire(&sk->sk_pacing_status) !=
-				     SK_PACING_FQ))
-				smp_store_release(&sk->sk_pacing_status,
-						  SK_PACING_FQ);
-		}
 		q->inactive_flows--;
 	}
 
Signature · 1 line
2.24.1.735.g03f4e72817-goog
Acked-by on v1 1/1
#1 lore raw
Quoted 6 lines · Eric Dumazet

If fq_classify() recycles a struct fq_flow because a socket structure has been reallocated, we do not set sk->sk_pacing_status immediately, but later if the flow becomes detached.

Quoted 8 lines

This means that any flow requiring pacing (BBR, or SO_MAX_PACING_RATE) might fallback to TCP internal pacing, which requires a per-socket high resolution timer, and therefore more cpu cycles.

Fixes 218af599fa63 ("tcp: internal implementation for pacing")
Cc Soheil Hassas Yeganeh <[email protected]>
Cc Neal Cardwell <[email protected]>

Nice catch! Thanks for the fix!

Quoted 53 lines
---
 net/sched/sch_fq.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
net/sched/sch_fq.c +8 -10
index b1c7e726ce5d1ae139f765c5b92dfdaea9bee258..ff4c5e9d0d7778d86f20f4bd67cc627eed0713d9 100644
@@ -301,6 +301,9 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q)
                                     f->socket_hash != sk->sk_hash)) {
                                f->credit = q->initial_quantum;
                                f->socket_hash = sk->sk_hash;
+                               if (q->rate_enable)
+                                       smp_store_release(&sk->sk_pacing_status,
+                                                         SK_PACING_FQ);
                                if (fq_flow_is_throttled(f))
                                        fq_flow_unset_throttled(q, f);
                                f->time_next_packet = 0ULL;
@@ -322,8 +325,12 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q)
 
        fq_flow_set_detached(f);
        f->sk = sk;
-       if (skb->sk == sk)
+       if (skb->sk == sk) {
                f->socket_hash = sk->sk_hash;
+               if (q->rate_enable)
+                       smp_store_release(&sk->sk_pacing_status,
+                                         SK_PACING_FQ);
+       }
        f->credit = q->initial_quantum;
 
        rb_link_node(&f->fq_node, parent, p);
@@ -428,17 +435,9 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch,
        f->qlen++;
        qdisc_qstats_backlog_inc(sch, skb);
        if (fq_flow_is_detached(f)) {
-               struct sock *sk = skb->sk;
-
                fq_flow_add_tail(&q->new_flows, f);
                if (time_after(jiffies, f->age + q->flow_refill_delay))
                        f->credit = max_t(u32, f->credit, q->quantum);
-               if (sk && q->rate_enable) {
-                       if (unlikely(smp_load_acquire(&sk->sk_pacing_status) !=
-                                    SK_PACING_FQ))
-                               smp_store_release(&sk->sk_pacing_status,
-                                                 SK_PACING_FQ);
-               }
                q->inactive_flows--;
        }
 
--

2.24.1.735.g03f4e72817-goog

maintainer
#2 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Mon, 23 Dec 2019 11:13:24 -0800

Quoted 11 lines

If fq_classify() recycles a struct fq_flow because a socket structure has been reallocated, we do not set sk->sk_pacing_status immediately, but later if the flow becomes detached.

This means that any flow requiring pacing (BBR, or SO_MAX_PACING_RATE) might fallback to TCP internal pacing, which requires a per-socket high resolution timer, and therefore more cpu cycles.

Fixes 218af599fa63 ("tcp: internal implementation for pacing")

Applied and queued up for -stable, thanks Eric.