Skip to contentSkip to conversation

qdisc bulk dequeuing and utilizing delayed tailptr updates

PATCH Applied Jesper Dangaard Brouer 7 participants 24 messages Sep 2, 2014 – Sep 5, 2014

Subsystems (3)

v1 of 6 Applied Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover qdisc bulk dequeuing and utilizing delayed tailptr updates

This patchset uses DaveM's recent API changes to dev_hard_start_xmit(), from the qdisc layer, to implement dequeue bulking.

Open questions:

- For now set bulk limit to 8 packets, don't want to stress the driver avail ring_buffer space.

- We are not doing proper accounting for weight_p/quota in __qdisc_run(). Do we care?

- Is the (!skb->next) check in dequeue necessary?

- Do we need some checks in dev_requeue_skb() as we could be requeuing a SKB list?

- Should we rename dequeue_skb() to dequeue_skb_list() ?

Based on top of:
commit 364a9e93243d ("sock: deduplicate errqueue dequeue")

---

Jesper Dangaard Brouer (3):
qdisc: sysctl to adjust bulk dequeue limit
qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE
qdisc: adjustments for API allowing skb list xmits

 include/net/sch_generic.h  |    2 ++
 net/core/sysctl_net_core.c |    9 +++++++++
 net/sched/sch_generic.c    |   33 ++++++++++++++++++++++++++++-----
 3 files changed, 39 insertions(+), 5 deletions(-)
#1 lore raw

PATCH 1/3 qdisc: adjustments for API allowing skb list xmits

Minor adjustments for merge commit 53fda7f7f9e (Merge branch 'xmit_list') that allows us to work with a list of SKBs.

Update code doc to function sch_direct_xmit().

In handle_dev_cpu_collision() use kfree_skb_list() in error handling.

---

 net/sched/sch_generic.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
net/sched/sch_generic.c +4 -4
index a8bf9f9..5b261e9 100644
@@ -93,7 +93,7 @@ static inline int handle_dev_cpu_collision(struct sk_buff *skb,
 		 * detect it by checking xmit owner and drop the packet when
 		 * deadloop is detected. Return OK to try the next skb.
 		 */
-		kfree_skb(skb);
+		kfree_skb_list(skb);
 		net_warn_ratelimited("Dead loop on netdevice %s, fix it urgently!\n",
 				     dev_queue->dev->name);
 		ret = qdisc_qlen(q);
@@ -110,9 +110,9 @@ static inline int handle_dev_cpu_collision(struct sk_buff *skb,
 }
 
 /*
- * Transmit one skb, and handle the return status as required. Holding the
- * __QDISC___STATE_RUNNING bit guarantees that only one CPU can execute this
- * function.
+ * Transmit possibly several skbs, and handle the return status as
+ * required. Holding the __QDISC___STATE_RUNNING bit guarantees that
+ * only one CPU can execute this function.
  *
  * Returns to the caller:
  *				0  - queue is empty or throttled.
#10 lore raw

PATCH 2/3 qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE

In reply to Jesper Dangaard Brouer

Based on DaveM's recent API work on dev_hard_start_xmit(), that allows sending/processing an entire skb list.

This patch implements qdisc bulk dequeue, by allowing multiple packets to be dequeued in dequeue_skb().

One restriction of the new API is that every SKB must belong to the same TXQ. This patch takes the easy way out, by restricting bulk dequeue to qdisc's with the TCQ_F_ONETXQUEUE flag, that specifies the qdisc only have attached a single TXQ.

Testing if this have the desired effect is the challenging part. Generating enough packets for a backlog queue to form at the qdisc is a challenge (because overhead else-where is a limiting factor e.g. I've measured the pure skb_alloc/free cycle to cost 80ns).

After trying many qdisc setups, I figured out that, the easiest way to make a backlog form is to fully load the system, all CPUs. And I can even demonstrate this with the default MQ disc.

This is a 12 core CPU (without HT) running trafgen on all 12 cores,
via qdisc-path using sendto():
* trafgen --cpp --dev $DEV --conf udp_example02_const.trafgen --qdisc-path -t0 --cpus 12

Measuring TX pps:
* Baseline : 12,815,925 pps
* This patch: 14,892,001 pps

This is crazy fast. This measurement is actually "too-high" as 10Gbit/s wirespeed is 14,880,952 (11049 pps too fast).

---

 net/sched/sch_generic.c |   23 ++++++++++++++++++++++-
 1 files changed, 22 insertions(+), 1 deletions(-)
net/sched/sch_generic.c +22 -1
index 5b261e9..30814ef 100644
@@ -56,6 +56,9 @@ static inline int dev_requeue_skb(struct sk_buff *skb, struct Qdisc *q)
 	return 0;
 }
 
+/* Note that dequeue_skb can possibly return a SKB list (via skb->next).
+ * A requeued skb (via q->gso_skb) can also be a SKB list.
+ */
 static inline struct sk_buff *dequeue_skb(struct Qdisc *q)
 {
 	struct sk_buff *skb = q->gso_skb;
@@ -70,10 +73,28 @@ static inline struct sk_buff *dequeue_skb(struct Qdisc *q)
 		} else
 			skb = NULL;
 	} else {
-		if (!(q->flags & TCQ_F_ONETXQUEUE) || !netif_xmit_frozen_or_stopped(txq)) {
+		if (!(q->flags & TCQ_F_ONETXQUEUE)
+		    || !netif_xmit_frozen_or_stopped(txq)) {
 			skb = q->dequeue(q);
 			if (skb)
 				skb = validate_xmit_skb(skb, qdisc_dev(q));
+			/* bulk dequeue */
+			if (skb && !skb->next && (q->flags & TCQ_F_ONETXQUEUE)) {
+				struct sk_buff *new, *head = skb;
+				int limit = 7;
+
+				do {
+					new = q->dequeue(q);
+					if (new)
+						new = validate_xmit_skb(
+							new, qdisc_dev(q));
+					if (new) {
+						skb->next = new;
+						skb = new;
+					}
+				} while (new && --limit);
+				skb = head;
+			}
 		}
 	}
 
#16 lore raw

PATCH 3/3 qdisc: sysctl to adjust bulk dequeue limit

In reply to Jesper Dangaard Brouer

Allow userspace to adjust how many packet the qdisc is allowed to bulk dequeue.

---
Question should we allow this to be adjusted?

 include/net/sch_generic.h  |    2 ++
 net/core/sysctl_net_core.c |    9 +++++++++
 net/sched/sch_generic.c    |    4 +++-
 3 files changed, 14 insertions(+), 1 deletions(-)
include/net/sch_generic.h +2 -0
index a3cfb8e..b0ac7b5 100644
@@ -14,6 +14,8 @@ struct qdisc_walker;
 struct tcf_walker;
 struct module;
 
+extern int qdisc_bulk_dequeue_limit;
+
 struct qdisc_rate_table {
 	struct tc_ratespec rate;
 	u32		data[256];
net/core/sysctl_net_core.c +9 -0
index cf9cd13..5505841 100644
@@ -361,6 +361,15 @@ static struct ctl_table net_core_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_dointvec
 	},
+	{
+		.procname	= "qdisc_bulk_dequeue_limit",
+		.data		= &qdisc_bulk_dequeue_limit,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.extra1		= &zero,
+		.extra2		= &ushort_max,
+		.proc_handler	= proc_dointvec_minmax
+	},
 	{ }
 };
 
net/sched/sch_generic.c +3 -1
index 30814ef..9cb08c0 100644
@@ -34,6 +34,8 @@
 const struct Qdisc_ops *default_qdisc_ops = &pfifo_fast_ops;
 EXPORT_SYMBOL(default_qdisc_ops);
 
+int qdisc_bulk_dequeue_limit __read_mostly = 7;
+
 /* Main transmission queue. */
 
 /* Modifications to data participating in scheduling must be protected with
@@ -81,7 +83,7 @@ static inline struct sk_buff *dequeue_skb(struct Qdisc *q)
 			/* bulk dequeue */
 			if (skb && !skb->next && (q->flags & TCQ_F_ONETXQUEUE)) {
 				struct sk_buff *new, *head = skb;
-				int limit = 7;
+				int limit = qdisc_bulk_dequeue_limit;
 
 				do {
 					new = q->dequeue(q);
#11 lore raw

Re: [net-next PATCH 2/3] qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE

In reply to Jesper Dangaard Brouer
Quoted 5 lines · Jesper Dangaard Brouer

This is crazy fast. This measurement is actually "too-high" as 10Gbit/s wirespeed is 14,880,952 (11049 pps too fast).

---

This looks buggy, you forgot about GSO.

(You did the test only for first dequeued packet, not the followings)

Make sure you test your patch with something else than pktgen.

Also, our idea was to use a byte count limit (aka BQL)

If we dequeue 8 64KB packets, this patch adds head of line blocking, which we fought hard.

#17 lore raw

Re: [net-next PATCH 3/3] qdisc: sysctl to adjust bulk dequeue limit

In reply to Jesper Dangaard Brouer
Quoted 60 lines · On Tue, 02 Sep 2014 16:36:09 +0200 Jesper Dangaard Brouer <[email protected]>…

Allow userspace to adjust how many packet the qdisc is allowed to bulk dequeue.

---
Question should we allow this to be adjusted?

 include/net/sch_generic.h  |    2 ++
 net/core/sysctl_net_core.c |    9 +++++++++
 net/sched/sch_generic.c    |    4 +++-
 3 files changed, 14 insertions(+), 1 deletions(-)
include/net/sch_generic.h +2 -0
index a3cfb8e..b0ac7b5 100644
@@ -14,6 +14,8 @@ struct qdisc_walker;
 struct tcf_walker;
 struct module;
 
+extern int qdisc_bulk_dequeue_limit;
+
 struct qdisc_rate_table {
 	struct tc_ratespec rate;
 	u32		data[256];
net/core/sysctl_net_core.c +9 -0
index cf9cd13..5505841 100644
@@ -361,6 +361,15 @@ static struct ctl_table net_core_table[] = {
 		.mode		= 0644,
 		.proc_handler	= proc_dointvec
 	},
+	{
+		.procname	= "qdisc_bulk_dequeue_limit",
+		.data		= &qdisc_bulk_dequeue_limit,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.extra1		= &zero,
+		.extra2		= &ushort_max,
+		.proc_handler	= proc_dointvec_minmax
+	},
 	{ }
 };
 
net/sched/sch_generic.c +2 -0
index 30814ef..9cb08c0 100644
@@ -34,6 +34,8 @@
 const struct Qdisc_ops *default_qdisc_ops = &pfifo_fast_ops;
 EXPORT_SYMBOL(default_qdisc_ops);
 
+int qdisc_bulk_dequeue_limit __read_mostly = 7;
+
 /* Main transmission queue. */
 
 /* Modifications to data participating in scheduling must be protected with

net/sched/sch_generic.c static inline struct sk_buff *dequeue_skb(struct Qdisc *q) · line 84

Quoted 2 lines
net/sched/sch_generic.c +0 -0
@@ -81,7 +83,7 @@ static inline struct sk_buff *dequeue_skb(struct Qdisc *q)
 			/* bulk dequeue */
 			if (skb && !skb->next && (q->flags & TCQ_F_ONETXQUEUE)) {

Will update the patch with:
- if (skb && !skb->next && (q->flags & TCQ_F_ONETXQUEUE)) {
+ if (qdisc_bulk_dequeue_limit && skb &&
+ !skb->next && (q->flags & TCQ_F_ONETXQUEUE)) {

To allow qdisc_bulk_dequeue_limit==0 to actually disable bulking.

Quoted 7 lines
net/sched/sch_generic.c +1 -1
 				struct sk_buff *new, *head = skb;
-				int limit = 7;
+				int limit = qdisc_bulk_dequeue_limit;
 
 				do {
 					new = q->dequeue(q);
Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
#2 lore raw

Re: [net-next PATCH 1/3] qdisc: adjustments for API allowing skb list xmits

In reply to Jesper Dangaard Brouer
Quoted 9 lines · Jesper Dangaard Brouer

Minor adjustments for merge commit 53fda7f7f9e (Merge branch 'xmit_list') that allows us to work with a list of SKBs.

Update code doc to function sch_direct_xmit().

In handle_dev_cpu_collision() use kfree_skb_list() in error handling.

---

Please fix all other similar issues in a single patch.

qdisc_reset() and qdisc_destroy() are buggy as well.

#18 lore raw

Re: [net-next PATCH 3/3] qdisc: sysctl to adjust bulk dequeue limit

In reply to Jesper Dangaard Brouer
Quoted 31 lines · Jesper Dangaard Brouer

Allow userspace to adjust how many packet the qdisc is allowed to bulk dequeue.

---
Question should we allow this to be adjusted?

  include/net/sch_generic.h  |    2 ++
  net/core/sysctl_net_core.c |    9 +++++++++
  net/sched/sch_generic.c    |    4 +++-
  3 files changed, 14 insertions(+), 1 deletions(-)
include/net/sch_generic.h +2 -0
index a3cfb8e..b0ac7b5 100644
@@ -14,6 +14,8 @@ struct qdisc_walker;
  struct tcf_walker;
  struct module;
 
+extern int qdisc_bulk_dequeue_limit;
+
  struct qdisc_rate_table {
  	struct tc_ratespec rate;
  	u32		data[256];
net/core/sysctl_net_core.c +0 -0
index cf9cd13..5505841 100644

net/core/sysctl_net_core.c static struct ctl_table net_core_table[] = { · line 369

Quoted 9 lines
net/core/sysctl_net_core.c +6 -0
@@ -361,6 +361,15 @@ static struct ctl_table net_core_table[] = {
  		.mode		= 0644,
  		.proc_handler	= proc_dointvec
  	},
+	{
+		.procname	= "qdisc_bulk_dequeue_limit",
+		.data		= &qdisc_bulk_dequeue_limit,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.extra1		= &zero,

If zero, then this would bulk INT_MAX due to a wrap around in patch 2.

Quoted 11 lines
net/core/sysctl_net_core.c +3 -0
+		.extra2		= &ushort_max,
+		.proc_handler	= proc_dointvec_minmax
+	},
  	{ }
  };
 
net/sched/sch_generic.c +0 -0
index 30814ef..9cb08c0 100644

net/sched/sch_generic.c line 38

Quoted 5 lines
net/sched/sch_generic.c +2 -0
@@ -34,6 +34,8 @@
  const struct Qdisc_ops *default_qdisc_ops = &pfifo_fast_ops;
  EXPORT_SYMBOL(default_qdisc_ops);
 
+int qdisc_bulk_dequeue_limit __read_mostly = 7;
+

Just nit, but this should be adjusted plus the exit condition in dequeue_skb() to work with 8 when you mention the max limit of 8 as bulk dequeue.

It would be great, if we could go with 0 knobs first, though.

Quoted 13 lines
net/sched/sch_generic.c +1 -1
  /* Main transmission queue. */
 
  /* Modifications to data participating in scheduling must be protected with
@@ -81,7 +83,7 @@ static inline struct sk_buff *dequeue_skb(struct Qdisc *q)
  			/* bulk dequeue */
  			if (skb && !skb->next && (q->flags & TCQ_F_ONETXQUEUE)) {
  				struct sk_buff *new, *head = skb;
-				int limit = 7;
+				int limit = qdisc_bulk_dequeue_limit;
 
  				do {
  					new = q->dequeue(q);
#14 lore raw

Re: [net-next PATCH 2/3] qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE

In reply to Jesper Dangaard Brouer
Quoted 20 lines · Jesper Dangaard Brouer

Based on DaveM's recent API work on dev_hard_start_xmit(), that allows sending/processing an entire skb list.

This patch implements qdisc bulk dequeue, by allowing multiple packets to be dequeued in dequeue_skb().

One restriction of the new API is that every SKB must belong to the same TXQ. This patch takes the easy way out, by restricting bulk dequeue to qdisc's with the TCQ_F_ONETXQUEUE flag, that specifies the qdisc only have attached a single TXQ.

Testing if this have the desired effect is the challenging part. Generating enough packets for a backlog queue to form at the qdisc is a challenge (because overhead else-where is a limiting factor e.g. I've measured the pure skb_alloc/free cycle to cost 80ns).

After trying many qdisc setups, I figured out that, the easiest way to make a backlog form is to fully load the system, all CPUs. And I can even demonstrate this with the default MQ disc.

Quoted 7 lines

This is a 12 core CPU (without HT) running trafgen on all 12 cores,
via qdisc-path using sendto():
* trafgen --cpp --dev $DEV --conf udp_example02_const.trafgen --qdisc-path -t0 --cpus 12

Measuring TX pps:
* Baseline : 12,815,925 pps
* This patch: 14,892,001 pps

I'm curious about *_RR tests and *_STREAM results e.g. from super_netperf.

One thing we might want to be careful about when comparing before and after numbers though is that now we still have the old quota limit, but don't adjust it here. It probably depends on how you interpret quota, but we now do more work within our quota than before.

Quoted 4 lines

This is crazy fast. This measurement is actually "too-high" as 10Gbit/s wirespeed is 14,880,952 (11049 pps too fast).

...

Quoted 19 lines
  net/sched/sch_generic.c |   23 ++++++++++++++++++++++-
  1 files changed, 22 insertions(+), 1 deletions(-)
net/sched/sch_generic.c +3 -0
index 5b261e9..30814ef 100644
@@ -56,6 +56,9 @@ static inline int dev_requeue_skb(struct sk_buff *skb, struct Qdisc *q)
  	return 0;
  }
 
+/* Note that dequeue_skb can possibly return a SKB list (via skb->next).
+ * A requeued skb (via q->gso_skb) can also be a SKB list.
+ */
  static inline struct sk_buff *dequeue_skb(struct Qdisc *q)
  {
  	struct sk_buff *skb = q->gso_skb;
@@ -70,10 +73,28 @@ static inline struct sk_buff *dequeue_skb(struct Qdisc *q)
  		} else

net/sched/sch_generic.c static inline struct sk_buff *dequeue_skb(struct Qdisc *q) · line 82

Quoted 10 lines
net/sched/sch_generic.c +4 -1
  			skb = NULL;
  	} else {
-		if (!(q->flags & TCQ_F_ONETXQUEUE) || !netif_xmit_frozen_or_stopped(txq)) {
+		if (!(q->flags & TCQ_F_ONETXQUEUE)
+		    || !netif_xmit_frozen_or_stopped(txq)) {
  			skb = q->dequeue(q);
  			if (skb)
  				skb = validate_xmit_skb(skb, qdisc_dev(q));
+			/* bulk dequeue */
+			if (skb && !skb->next && (q->flags & TCQ_F_ONETXQUEUE)) {

This check should better be an inline for sch_generic.h, e.g. ...

static inline bool qdisc_may_bulk(const struct Qdisc *qdisc,
const struct sk_buff *skb)

{
	return (qdisc->flags & TCQ_F_ONETXQUEUE) && !skb->next;
}

net/sched/sch_generic.c static inline struct sk_buff *dequeue_skb(struct Qdisc *q) · line 90

Quoted 8 lines
net/sched/sch_generic.c +8 -0
+				struct sk_buff *new, *head = skb;
+				int limit = 7;
+
+				do {
+					new = q->dequeue(q);
+					if (new)
+						new = validate_xmit_skb(
+							new, qdisc_dev(q));

This and above dequeue() + validate_xmit_skb() code should probably also go into a helper if you're at it, e.g. ...

static inline struct sk_buff *qdisc_dequeue_validate(struct Qdisc *qdisc)

{
	struct sk_buff *skb = qdisc->dequeue(qdisc);

	if (skb != NULL)
		skb = validate_xmit_skb(skb, qdisc_dev(qdisc));

	return skb;
}
Quoted 16 lines
net/sched/sch_generic.c +7 -1
+					if (new) {
+						skb->next = new;
+						skb = new;
+					}
+				} while (new && --limit);
+				skb = head;
+			}
  		}
  	}
 
 
--

To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw
In reply to Jesper Dangaard Brouer
Quoted 8 lines · Jesper Dangaard Brouer

This patchset uses DaveM's recent API changes to dev_hard_start_xmit(), from the qdisc layer, to implement dequeue bulking.

Open questions:

- For now set bulk limit to 8 packets, don't want to stress the driver avail ring_buffer space.

Please get limit from BQL also, see:
[PATCH net-next] net: Functions to report space available in device TX queues

Quoted 30 lines

- We are not doing proper accounting for weight_p/quota in __qdisc_run(). Do we care?

- Is the (!skb->next) check in dequeue necessary?

- Do we need some checks in dev_requeue_skb() as we could be requeuing a SKB list?

- Should we rename dequeue_skb() to dequeue_skb_list() ?

Based on top of:
commit 364a9e93243d ("sock: deduplicate errqueue dequeue")

---

Jesper Dangaard Brouer (3):
qdisc: sysctl to adjust bulk dequeue limit
qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE
qdisc: adjustments for API allowing skb list xmits

 include/net/sch_generic.h  |    2 ++
 net/core/sysctl_net_core.c |    9 +++++++++
 net/sched/sch_generic.c    |   33 ++++++++++++++++++++++++++++-----
 3 files changed, 39 insertions(+), 5 deletions(-)

--
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

maintainer
#23 lore raw
In reply to Jesper Dangaard Brouer

From: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 02 Sep 2014 16:35:19 +0200

Quoted 4 lines

Open questions:

- For now set bulk limit to 8 packets, don't want to stress the driver avail ring_buffer space.

Maybe we can start out with something even lower, like 5.

Quoted 3 lines

- Is the (!skb->next) check in dequeue necessary?

- Do we need some checks in dev_requeue_skb() as we could be requeuing a SKB list?

The skb->next stuff is to handle GSO software segmented frames.

dev_requeue_skb() naturally handles any list of SKBs already, GSO software segmented or multi-dequeue, it shouldn't care.

maintainer Applied
#9 lore raw

Re: [net-next PATCH 1/3] qdisc: adjustments for API allowing skb list xmits

In reply to Jesper Dangaard Brouer

From: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 02 Sep 2014 16:35:33 +0200

Quoted 8 lines

Minor adjustments for merge commit 53fda7f7f9e (Merge branch 'xmit_list') that allows us to work with a list of SKBs.

Update code doc to function sch_direct_xmit().

In handle_dev_cpu_collision() use kfree_skb_list() in error handling.

Applied, thanks for catching the kfree_skb_list() issue.

#19 lore raw

Re: [net-next PATCH 3/3] qdisc: sysctl to adjust bulk dequeue limit

In reply to Jesper Dangaard Brouer
Quoted 8 lines · Jesper Dangaard Brouer

Allow userspace to adjust how many packet the qdisc is allowed to bulk dequeue.

---
Question should we allow this to be adjusted?

A sysctl is ugly and seems not fit well with Qdisc which always uses netlink, so I think a netlink flag might be better if we can find a generic one.

Also, you forgot to document it.

#20 lore raw

Re: [net-next PATCH 3/3] qdisc: sysctl to adjust bulk dequeue limit

Quoted 6 lines · Cong Wang
Quoted 3 lines · Jesper Dangaard Brouer

Allow userspace to adjust how many packet the qdisc is allowed to bulk dequeue.

Quoted 9 lines
Quoted 5 lines

---
Question should we allow this to be adjusted?

A sysctl is ugly and seems not fit well with Qdisc which always uses netlink, so I think a netlink flag might be better if we can find a generic one.

You could also make this a device specific attribute easily enough.

Quoted 5 lines

Also, you forgot to document it.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw

Re: [net-next PATCH 2/3] qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE

In reply to Eric Dumazet
Quoted 9 lines · On Tue, 02 Sep 2014 08:22:42 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 5 lines · Jesper Dangaard Brouer

This is crazy fast. This measurement is actually "too-high" as 10Gbit/s wirespeed is 14,880,952 (11049 pps too fast).

---

This looks buggy, you forgot about GSO.

The GSO check is (!skb->next), right(?)

Quoted 1 line

(You did the test only for first dequeued packet, not the followings)

Ah! yes, for the following packets!

Quoted 1 line

Make sure you test your patch with something else than pktgen.

I were not using pktgen, but trafgen (raw/af_packet) ;-) And I've also been using netperf and super_netperf for testing.

pktgen is difficult to use in this scenerio, because it bypass'es the qdisc layer. But I have actually (with help from John Fastabend) constructed a setup, where I add a VLAN interface, which pktgen can xmit on, which will make packets travel the qdisc layer of the underlying real device. (pktgen's clone_skb must be 0 in this setup). So, I'm not currently using pktgen...

Quoted 1 line

Also, our idea was to use a byte count limit (aka BQL)

It makes sense. I will look into using the BQL limits.

Quoted 2 lines

If we dequeue 8 64KB packets, this patch adds head of line blocking, which we fought hard.

I should not be able to add GSO packets to the skb list in dequeue_skb(), or did I miss something.

Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
#3 lore raw

PATCH 1/1 qdisc: exit case fixes for skb list handling in qdisc layer

In reply to Eric Dumazet

More minor fixes to merge commit 53fda7f7f9e (Merge branch 'xmit_list') that allows us to work with a list of SKBs.

Fixing exit cases in qdisc_reset() and qdisc_destroy(), where a leftover requeued SKB (qdisc->gso_skb) can have the potential of being a skb list, thus use kfree_skb_list().

This is a followup to commit 10770bc2d1 ("qdisc: adjustments for API allowing skb list xmits").

---

 net/sched/sch_generic.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
net/sched/sch_generic.c +2 -2
index 5b261e9..19696eb 100644
@@ -621,7 +621,7 @@ void qdisc_reset(struct Qdisc *qdisc)
 		ops->reset(qdisc);
 
 	if (qdisc->gso_skb) {
-		kfree_skb(qdisc->gso_skb);
+		kfree_skb_list(qdisc->gso_skb);
 		qdisc->gso_skb = NULL;
 		qdisc->q.qlen = 0;
 	}
@@ -657,7 +657,7 @@ void qdisc_destroy(struct Qdisc *qdisc)
 	module_put(ops->owner);
 	dev_put(qdisc_dev(qdisc));
 
-	kfree_skb(qdisc->gso_skb);
+	kfree_skb_list(qdisc->gso_skb);
 	/*
 	 * gen_estimator est_timer() might access qdisc->q.lock,
 	 * wait a RCU grace period before freeing qdisc.
#13 lore raw

Re: [net-next PATCH 2/3] qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE

In reply to Jesper Dangaard Brouer
Quoted 8 lines · Jesper Dangaard Brouer
Quoted 5 lines · On Tue, 02 Sep 2014 08:22:42 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 3 lines · Jesper Dangaard Brouer

This is crazy fast. This measurement is actually "too-high" as 10Gbit/s wirespeed is 14,880,952 (11049 pps too fast).

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

---

This looks buggy, you forgot about GSO.

The GSO check is (!skb->next), right(?)

No, you'd want to use skb_is_gso() helper.

Quoted 3 lines
Quoted 1 line

Also, our idea was to use a byte count limit (aka BQL)

It makes sense. I will look into using the BQL limits.

Thanks Jesper.

Quoted 2 lines
Quoted 2 lines

If we dequeue 8 64KB packets, this patch adds head of line blocking, which we fought hard.

Just to make sure: Using bql limits would resolve your HOL blocking concerns, right?

#15 lore raw

Re: [net-next PATCH 2/3] qdisc: bulk dequeue support for qdiscs with TCQ_F_ONETXQUEUE

In reply to Daniel Borkmann
Quoted 5 lines · On Tue, 02 Sep 2014 18:14:36 +0200 Daniel Borkmann <[email protected]> wrote:
Quoted 3 lines · Jesper Dangaard Brouer

This patch implements qdisc bulk dequeue, by allowing multiple packets to be dequeued in dequeue_skb().

[...]

Quoted 7 lines

This check should better be an inline for sch_generic.h, e.g. ...

static inline bool qdisc_may_bulk(const struct Qdisc *qdisc,
const struct sk_buff *skb)

{
	return (qdisc->flags & TCQ_F_ONETXQUEUE) && !skb->next;
}

Good ideas of inlining these functions, it should help readability of the code, thanks.

Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
#22 lore raw
In reply to Tom Herbert
Quoted 5 lines · On Tue, 2 Sep 2014 11:04:06 -0700 Tom Herbert <[email protected]> wrote:
Quoted 3 lines · Jesper Dangaard Brouer

This patchset uses DaveM's recent API changes to dev_hard_start_xmit(), from the qdisc layer, to implement dequeue bulking.

Quoted 7 lines
Quoted 5 lines

Open questions:

- For now set bulk limit to 8 packets, don't want to stress the driver avail ring_buffer space.

Please get limit from BQL also, see:
[PATCH net-next] net: Functions to report space available in device TX queues

Thank you Tom. I will include that patch in my stack (keeping you as author), and use the functions you are proposing.

Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
maintainer Applied
#4 lore raw

Re: [net-next PATCH] qdisc: exit case fixes for skb list handling in qdisc layer

In reply to Jesper Dangaard Brouer

From: Jesper Dangaard Brouer <[email protected]>
Date: Wed, 03 Sep 2014 12:12:50 +0200

Quoted 11 lines

More minor fixes to merge commit 53fda7f7f9e (Merge branch 'xmit_list') that allows us to work with a list of SKBs.

Fixing exit cases in qdisc_reset() and qdisc_destroy(), where a leftover requeued SKB (qdisc->gso_skb) can have the potential of being a skb list, thus use kfree_skb_list().

This is a followup to commit 10770bc2d1 ("qdisc: adjustments for API allowing skb list xmits").

Applied.

#5 lore raw

Re: [net-next PATCH] qdisc: exit case fixes for skb list handling in qdisc layer

Quoted 10 lines · David Miller

From: Jesper Dangaard Brouer <[email protected]>
Date: Wed, 03 Sep 2014 12:12:50 +0200

Quoted 7 lines

More minor fixes to merge commit 53fda7f7f9e (Merge branch 'xmit_list') that allows us to work with a list of SKBs.

Fixing exit cases in qdisc_reset() and qdisc_destroy(), where a leftover requeued SKB (qdisc->gso_skb) can have the potential of being a skb list, thus use kfree_skb_list().

Quoted 6 lines
Quoted 4 lines

This is a followup to commit 10770bc2d1 ("qdisc: adjustments for API allowing skb list xmits").

Applied.

Strange, I sent same patch a little earlier ;)

#6 lore raw

Re: [net-next PATCH] qdisc: exit case fixes for skb list handling in qdisc layer

Quoted 11 lines · On Wed, 03 Sep 2014 22:24:19 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 10 lines · David Miller

From: Jesper Dangaard Brouer <[email protected]>
Date: Wed, 03 Sep 2014 12:12:50 +0200

Quoted 7 lines

More minor fixes to merge commit 53fda7f7f9e (Merge branch 'xmit_list') that allows us to work with a list of SKBs.

Fixing exit cases in qdisc_reset() and qdisc_destroy(), where a leftover requeued SKB (qdisc->gso_skb) can have the potential of being a skb list, thus use kfree_skb_list().

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

This is a followup to commit 10770bc2d1 ("qdisc: adjustments for API allowing skb list xmits").

Applied.

Strange, I sent same patch a little earlier ;)

Yes, very strange, especially as Dave also said he had applied your patch too (I did comment on your post, apologizing for not noticing your patch, before sending this one).

Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
maintainer
#7 lore raw

Re: [net-next PATCH] qdisc: exit case fixes for skb list handling in qdisc layer

From: Jesper Dangaard Brouer <[email protected]>
Date: Thu, 4 Sep 2014 07:39:12 +0200

Quoted 19 lines
Quoted 16 lines · On Wed, 03 Sep 2014 22:24:19 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 15 lines · David Miller

From: Jesper Dangaard Brouer <[email protected]>
Date: Wed, 03 Sep 2014 12:12:50 +0200

Quoted 11 lines

More minor fixes to merge commit 53fda7f7f9e (Merge branch 'xmit_list') that allows us to work with a list of SKBs.

Fixing exit cases in qdisc_reset() and qdisc_destroy(), where a leftover requeued SKB (qdisc->gso_skb) can have the potential of being a skb list, thus use kfree_skb_list().

This is a followup to commit 10770bc2d1 ("qdisc: adjustments for API allowing skb list xmits").

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Applied.

Strange, I sent same patch a little earlier ;)

Yes, very strange, especially as Dave also said he had applied your patch too (I did comment on your post, apologizing for not noticing your patch, before sending this one).

This is my fault.

I put Eric's copy into my GIT tree on my workstation at home, build tested it, but did not push it out.

Then I did GIT work from my laptop for the following two days. :-/

Sorry about that.

#8 lore raw

Re: [net-next PATCH] qdisc: exit case fixes for skb list handling in qdisc layer

Quoted 6 lines · David Miller

I put Eric's copy into my GIT tree on my workstation at home, build tested it, but did not push it out.

Then I did GIT work from my laptop for the following two days. :-/

Sorry about that.

No worries, I knew it was something like that ;)

Thanks !