Skip to contentSkip to conversation

Flow sorted receive skb lists

RFC PATCH New Steffen Klassert 1 participant 3 messages Sep 12, 2018

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Flow sorted receive skb lists

This patchset consists of two patches. Patch 1 adds support for flow sorted rx skb lists for IPv4. This means that it sorts the skb list so that packets from the same flow can to travel together through the stack.

The second patch of this pachset is just a hack that disables GRO and does skb list receive instead. I don't have a NIC whose driver supports to build skb lists to be received by netif_receive_skb_list(), so I used this hack to test patch 1.

This is early stage work, so it might be not complete and still buggy. I send this now because I want to hear some comments on the approach itself before I spend more time to work on this.

Forwarding performance measurements:

I used used my IPsec forwarding test setup for this:

           ------------         ------------
        -->| router 1 |-------->| router 2 |--
        |  ------------         ------------  |
        |                                     |
        |       --------------------          |
        --------|Spirent Testcenter|<----------
                --------------------

net-next (September 6th):

Single stream UDP frame size 1460 Bytes: 1.258.446 fps.

Single stream UDP frame size 64 Bytes: 1.250.000 fps.

----------------------------------------------------------------------

net-next (September 6th) + patch 2 (list receive):

Single stream UDP frame size 1460 Bytes: 1.469.595 fps (+16%).

Single stream UDP frame size 64 Bytes: 1.502.976 fps (+20%).

----------------------------------------------------------------------

net-next (September 6th) + patch 1 + patch 2 (flow sorted list receive):

Single stream UDP frame size 1460 Bytes: 2.525.338 fps (+100%).

Single stream UDP frame size 64 Bytes: 2.541.881 fps (+103%).

-----------------------------------------------------------------------

#1 lore raw

RFC PATCH 1/2 net: Support flow sorted RX skb lists for IPv4.

This patch sorts RX skb lists into separate flows, using a flow dissector, at the IP input layer. Packets of the same flow are chained at the frag_list pointer of the first skb of this flow.

After ip_list_rcv_finish() the skb list has this layout:

|---------|        |---------|        |---------|
|flow 1   |        |flow 1   |        |flow 1   |
|---------|        |---------|        |---------|
|frag_list|<-\     |frag_list|        |frag_list|
|---------|   \    |---------|        |---------|
|next     |<-\ \---|next     |<-------|next     |
|---------|   \    |---------|        |---------|
              |
              |
              |    |---------|        |---------|        |---------|
              |    |flow 2   |        |flow 2   |        |flow 2   |
              |    |---------|        |---------|        |---------|
              |    |frag_list|<-\     |frag_list|        |frag_list|
              |    |---------|   \    |---------|        |---------|
              |----|next     |<-\ \---|next     |<-------|next     |
                   |---------|   \    |---------|        |---------|
                                 |
                                 |
                                 |    |---------|        |---------|       |---------|
                                 |    |flow 3   |        |flow 3   |       |flow 3   |
                                 |    |---------|        |---------|       |---------|
                                 |    |frag_list|<-\     |frag_list|       |frag_list|
                                 |    |---------|   \    |---------|       |---------|
                                 |----|next     |    \---|next     |<------|next     |
                                      |---------|        |---------|       |---------|

With this approach route lookups etc. are done just for one representative packet of a given flow instead for each packet.

ip_sublist_rcv_finish() splits these lists then into:

|---------|        |---------|        |---------|
|flow 1   |        |flow 1   |        |flow 1   |
|---------|        |---------|        |---------|
|frag_list|<-\     |frag_list|        |frag_list|
|---------|   \    |---------|        |---------|
|next     |    \---|next     |<-------|next     |
|---------|        |---------|        |---------|

Packets of the same flow can still travel together after that point.

On input, this is plumbed through to ip_local_deliver_finish(), here the skb chain is split back into single packets.

My hope is that this can be plumbed through to the sockets receive queue. I have a patch for UDP, but it has still problems with UDP encapsulaion, so it is not included here.

On forward, the skb chain can travel together to the TX path. __skb_gso_segment() will build a standard skb list from this.

For now, this is only enabled if the receiving device allows forwarding, as the forwarding path has currently the most gain from this.

Known issues:

- I don't have a NIC whose driver supports to build skb lists to be received by netif_receive_skb_list(). To test this codepath I used a hack that builds skb lists at the napi layer.

- Performance measurements were done with this hack, so I don't know if these measurements are really meaningful.

- This is early stage work, so the functional tests are only done on a basic level, it might be still buggy.

- This still uses the skb->next, skb->prev pointers to build skb lists. So needs to be converted to standard list handling at some point.

---
 include/linux/skbuff.h    |   5 ++
 net/core/dev.c            |  45 +++++++++++-
 net/core/flow_dissector.c |  40 +++++++++++
 net/core/skbuff.c         |  52 ++++++++++++++
 net/ipv4/ip_input.c       | 139 ++++++++++++++++++++++++++++++++++----
 net/ipv4/ip_output.c      |   3 +-
 6 files changed, 270 insertions(+), 14 deletions(-)
include/linux/skbuff.h +5 -0
index 17a13e4785fc..d070d073a1dc 100644
@@ -575,6 +575,8 @@ enum {
 	SKB_GSO_UDP = 1 << 16,
 
 	SKB_GSO_UDP_L4 = 1 << 17,
+
+	SKB_GSO_FRAGLIST = 1 << 18,
 };
 
 #if BITS_PER_LONG > 32
@@ -1226,6 +1228,8 @@ skb_flow_dissect_flow_keys_basic(const struct sk_buff *skb,
 				  data, proto, nhoff, hlen, flags);
 }
 
+u32 skb_flow_keys_rx_digest(struct sk_buff *skb, struct flow_keys_digest *digest);
+
 void
 skb_flow_dissect_tunnel_info(const struct sk_buff *skb,
 			     struct flow_dissector *flow_dissector,
@@ -3302,6 +3306,7 @@ int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen);
 void skb_scrub_packet(struct sk_buff *skb, bool xnet);
 bool skb_gso_validate_network_len(const struct sk_buff *skb, unsigned int mtu);
 bool skb_gso_validate_mac_len(const struct sk_buff *skb, unsigned int len);
+void skb_segment_list(struct sk_buff *skb);
 struct sk_buff *skb_segment(struct sk_buff *skb, netdev_features_t features);
 struct sk_buff *skb_vlan_untag(struct sk_buff *skb);
 int skb_ensure_writable(struct sk_buff *skb, int write_len);
net/core/dev.c +44 -1

63 lines not on this page · raw message

net/core/flow_dissector.c +40 -0
index ce9eeeb7c024..8ca7e09dca5e 100644
@@ -1211,6 +1211,46 @@ static inline u32 ___skb_get_hash(const struct sk_buff *skb,
 	return __flow_hash_from_keys(keys, keyval);
 }
 
+struct _flow_keys_rx_digest_data {
+	__be16	n_proto;
+	u8	ip_proto;
+	u8	poff;
+	__be32	ports;
+	__be32	src;
+	__be32	dst;
+};
+
+u32 skb_flow_keys_rx_digest(struct sk_buff *skb, struct flow_keys_digest *digest)
+{
+	struct flow_keys keys;
+	struct _flow_keys_rx_digest_data *data =
+	    (struct _flow_keys_rx_digest_data *)digest;
+	struct flow_keys_basic *bkeys;
+	u32 poff;
+
+	__flow_hash_secret_init();
+
+	skb_flow_dissect_flow_keys(skb, &keys,
+				   FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL);
+
+	bkeys = (struct flow_keys_basic *)&keys;
+	poff = __skb_get_poff(skb, skb->data, bkeys, skb_headlen(skb));
+	if (poff > 255)
+		poff = 0;
+
+	BUILD_BUG_ON(sizeof(*data) > sizeof(*digest));
+
+	data->n_proto = keys.basic.n_proto;
+	data->ip_proto = keys.basic.ip_proto;
+	data->ports = keys.ports.ports;
+	data->poff = poff;
+	data->src = keys.addrs.v4addrs.src;
+	data->dst = keys.addrs.v4addrs.dst;
+
+	return poff;
+}
+EXPORT_SYMBOL(skb_flow_keys_rx_digest);
+
 struct _flow_keys_digest_data {
 	__be16	n_proto;
 	u8	ip_proto;
net/core/skbuff.c +52 -0

58 lines not on this page · raw message

net/ipv4/ip_input.c +127 -12

207 lines not on this page · raw message

net/ipv4/ip_output.c +2 -1
index 9c4e72e9c60a..00d8a2576266 100644
@@ -272,7 +272,8 @@ static int ip_finish_output_gso(struct net *net, struct sock *sk,
 		return -ENOMEM;
 	}
 
-	consume_skb(skb);
+	if (segs != skb)
+		consume_skb(skb);
 
 	do {
 		struct sk_buff *nskb = segs->next;
Signature · 1 line
2.17.1
#2 lore raw

RFC PATCH 2/2 net: Hack to enable skb list receive in the napi layer.

In reply to Steffen Klassert

This patch was used to test patch ("net: Support flow sorted skb lists for IPv4.") It is just a hack that disables GRO and does skb list receive instead. Not for merging!

---
 include/linux/netdevice.h |  5 ++++-
 net/core/dev.c            | 23 ++++++++++++++---------
 net/ipv4/ip_input.c       |  1 +
 3 files changed, 19 insertions(+), 10 deletions(-)
include/linux/netdevice.h +4 -1
index e2b3bd750c98..6518ceeda05d 100644
@@ -333,7 +333,10 @@ struct napi_struct {
 	int			poll_owner;
 #endif
 	struct net_device	*dev;
-	struct gro_list		gro_hash[GRO_HASH_BUCKETS];
+	union {
+		struct list_head	rx_list;
+		struct gro_list		gro_hash[GRO_HASH_BUCKETS];
+	};
 	struct sk_buff		*skb;
 	struct hrtimer		timer;
 	struct list_head	dev_list;
net/core/dev.c +14 -9
index 147da35d7380..68e00727c657 100644
@@ -5470,6 +5470,10 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 	int same_flow;
 	int grow;
 
+	list_add_tail(&skb->list, &napi->rx_list);
+	ret = GRO_CONSUMED;
+	goto out;
+
 	if (netif_elide_gro(skb->dev))
 		goto normal;
 
@@ -5559,7 +5563,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 	} else if (test_bit(hash, &napi->gro_bitmask)) {
 		__clear_bit(hash, &napi->gro_bitmask);
 	}
-
+out:
 	return ret;
 
 normal:
@@ -5958,7 +5962,7 @@ bool napi_complete_done(struct napi_struct *n, int work_done)
 				 NAPIF_STATE_IN_BUSY_POLL)))
 		return false;
 
-	if (n->gro_bitmask) {
+	if (!list_empty(&n->rx_list)) {
 		unsigned long timeout = 0;
 
 		if (work_done)
@@ -5967,8 +5971,10 @@ bool napi_complete_done(struct napi_struct *n, int work_done)
 		if (timeout)
 			hrtimer_start(&n->timer, ns_to_ktime(timeout),
 				      HRTIMER_MODE_REL_PINNED);
-		else
-			napi_gro_flush(n, false);
+		else {
+			netif_receive_skb_list(&n->rx_list);
+			INIT_LIST_HEAD(&n->rx_list);
+		}
 	}
 	if (unlikely(!list_empty(&n->poll_list))) {
 		/* If n->poll_list is not empty, we need to mask irqs */
@@ -6189,6 +6195,7 @@ void netif_napi_add(struct net_device *dev, struct napi_struct *napi,
 		    int (*poll)(struct napi_struct *, int), int weight)
 {
 	INIT_LIST_HEAD(&napi->poll_list);
+	INIT_LIST_HEAD(&napi->rx_list);
 	hrtimer_init(&napi->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_PINNED);
 	napi->timer.function = napi_watchdog;
 	init_gro_hash(napi);
@@ -6289,11 +6296,9 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll)
 		goto out_unlock;
 	}
 
-	if (n->gro_bitmask) {
-		/* flush too old packets
-		 * If HZ < 1000, flush all packets.
-		 */
-		napi_gro_flush(n, HZ >= 1000);
+	if (!list_empty(&n->rx_list)) {
+		netif_receive_skb_list(&n->rx_list);
+		INIT_LIST_HEAD(&n->rx_list);
 	}
 
 	/* Some drivers may have called napi_schedule
net/ipv4/ip_input.c +1 -0
index bf710bf95fea..847965f83a2f 100644
@@ -729,6 +729,7 @@ void ip_list_rcv(struct list_head *head, struct packet_type *pt,
 		}
 		list_add_tail(&skb->list, &sublist);
 	}
+
 	/* dispatch final sublist */
 	ip_sublist_rcv(&sublist, curr_dev, curr_net);
 }
Signature · 1 line
2.17.1