Flow sorted receive skb lists
Subsystems (3)
cover Flow sorted receive skb lists
This patchset consists of two patches. Patch 1 adds support for flow sorted rx skb lists for IPv4. This means that it sorts the skb list so that packets from the same flow can to travel together through the stack.
The second patch of this pachset is just a hack that disables GRO and does skb list receive instead. I don't have a NIC whose driver supports to build skb lists to be received by netif_receive_skb_list(), so I used this hack to test patch 1.
This is early stage work, so it might be not complete and still buggy. I send this now because I want to hear some comments on the approach itself before I spend more time to work on this.
Forwarding performance measurements:
I used used my IPsec forwarding test setup for this:
------------ ------------
-->| router 1 |-------->| router 2 |--
| ------------ ------------ |
| |
| -------------------- |
--------|Spirent Testcenter|<----------
--------------------net-next (September 6th):
Single stream UDP frame size 1460 Bytes: 1.258.446 fps.
Single stream UDP frame size 64 Bytes: 1.250.000 fps.
----------------------------------------------------------------------
net-next (September 6th) + patch 2 (list receive):
Single stream UDP frame size 1460 Bytes: 1.469.595 fps (+16%).
Single stream UDP frame size 64 Bytes: 1.502.976 fps (+20%).
----------------------------------------------------------------------
net-next (September 6th) + patch 1 + patch 2 (flow sorted list receive):
Single stream UDP frame size 1460 Bytes: 2.525.338 fps (+100%).
Single stream UDP frame size 64 Bytes: 2.541.881 fps (+103%).
-----------------------------------------------------------------------
RFC PATCH 1/2 net: Support flow sorted RX skb lists for IPv4.
This patch sorts RX skb lists into separate flows, using a flow dissector, at the IP input layer. Packets of the same flow are chained at the frag_list pointer of the first skb of this flow.
After ip_list_rcv_finish() the skb list has this layout:
|---------| |---------| |---------|
|flow 1 | |flow 1 | |flow 1 |
|---------| |---------| |---------|
|frag_list|<-\ |frag_list| |frag_list|
|---------| \ |---------| |---------|
|next |<-\ \---|next |<-------|next |
|---------| \ |---------| |---------|
|
|
| |---------| |---------| |---------|
| |flow 2 | |flow 2 | |flow 2 |
| |---------| |---------| |---------|
| |frag_list|<-\ |frag_list| |frag_list|
| |---------| \ |---------| |---------|
|----|next |<-\ \---|next |<-------|next |
|---------| \ |---------| |---------|
|
|
| |---------| |---------| |---------|
| |flow 3 | |flow 3 | |flow 3 |
| |---------| |---------| |---------|
| |frag_list|<-\ |frag_list| |frag_list|
| |---------| \ |---------| |---------|
|----|next | \---|next |<------|next |
|---------| |---------| |---------|With this approach route lookups etc. are done just for one representative packet of a given flow instead for each packet.
ip_sublist_rcv_finish() splits these lists then into:
|---------| |---------| |---------| |flow 1 | |flow 1 | |flow 1 | |---------| |---------| |---------| |frag_list|<-\ |frag_list| |frag_list| |---------| \ |---------| |---------| |next | \---|next |<-------|next | |---------| |---------| |---------|
Packets of the same flow can still travel together after that point.
On input, this is plumbed through to ip_local_deliver_finish(), here the skb chain is split back into single packets.
My hope is that this can be plumbed through to the sockets receive queue. I have a patch for UDP, but it has still problems with UDP encapsulaion, so it is not included here.
On forward, the skb chain can travel together to the TX path. __skb_gso_segment() will build a standard skb list from this.
For now, this is only enabled if the receiving device allows forwarding, as the forwarding path has currently the most gain from this.
Known issues:
- I don't have a NIC whose driver supports to build skb lists to be received by netif_receive_skb_list(). To test this codepath I used a hack that builds skb lists at the napi layer.
- Performance measurements were done with this hack, so I don't know if these measurements are really meaningful.
- This is early stage work, so the functional tests are only done on a basic level, it might be still buggy.
- This still uses the skb->next, skb->prev pointers to build skb lists. So needs to be converted to standard list handling at some point.
--- include/linux/skbuff.h | 5 ++ net/core/dev.c | 45 +++++++++++- net/core/flow_dissector.c | 40 +++++++++++ net/core/skbuff.c | 52 ++++++++++++++ net/ipv4/ip_input.c | 139 ++++++++++++++++++++++++++++++++++---- net/ipv4/ip_output.c | 3 +- 6 files changed, 270 insertions(+), 14 deletions(-)
include/linux/skbuff.h +5 -0
@@ -575,6 +575,8 @@ enum {SKB_GSO_UDP =1 <<16 ,SKB_GSO_UDP_L4 =1 <<17 , + + SKB_GSO_FRAGLIST = 1 << 18, };#if BITS_PER_LONG >32
@@ -1226,6 +1228,8 @@ skb_flow_dissect_flow_keys_basic(const struct sk_buff *skb, data, proto, nhoff, hlen, flags); } +u32 skb_flow_keys_rx_digest (struct sk_buff *skb,struct flow_keys_digest *digest); + void skb_flow_dissect_tunnel_info(const struct sk_buff *skb, struct flow_dissector *flow_dissector,
@@ -3302,6 +3306,7 @@ int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen);void skb_scrub_packet (struct sk_buff *skb,bool xnet);bool skb_gso_validate_network_len (const struct sk_buff *skb,unsigned int mtu);bool skb_gso_validate_mac_len (const struct sk_buff *skb,unsigned int len); +void skb_segment_list (struct sk_buff *skb);struct sk_buff *skb_segment (struct sk_buff *skb,netdev_features_t features);struct sk_buff *skb_vlan_untag (struct sk_buff *skb);int skb_ensure_writable (struct sk_buff *skb,int write_len);
net/core/dev.c +44 -1
63 lines not on this page · raw message
net/core/flow_dissector.c +40 -0
@@ -1211,6 +1211,46 @@ static inline u32 ___skb_get_hash(const struct sk_buff *skb,return __flow_hash_from_keys (keys, keyval); } +struct _flow_keys_rx_digest_data { +__be16 n_proto; +u8 ip_proto; +u8 poff; +__be32 ports; +__be32 src; +__be32 dst; +}; + +u32 skb_flow_keys_rx_digest (struct sk_buff *skb,struct flow_keys_digest *digest) +{ +struct flow_keys keys; +struct _flow_keys_rx_digest_data *data = + (struct _flow_keys_rx_digest_data *)digest; +struct flow_keys_basic *bkeys; +u32 poff; + +__flow_hash_secret_init (); + +skb_flow_dissect_flow_keys (skb, &keys, +FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL ); + + bkeys = (struct flow_keys_basic *)&keys; + poff =__skb_get_poff (skb, skb->data, bkeys,skb_headlen (skb)); +if (poff >255 ) + poff =0 ; + +BUILD_BUG_ON (sizeof (*data) >sizeof (*digest)); + + data->n_proto = keys.basic.n_proto; + data->ip_proto = keys.basic.ip_proto; + data->ports = keys.ports.ports; + data->poff = poff; + data->src = keys.addrs.v4addrs.src; + data->dst = keys.addrs.v4addrs.dst; + +return poff; +} +EXPORT_SYMBOL (skb_flow_keys_rx_digest); +struct _flow_keys_digest_data {__be16 n_proto;u8 ip_proto;
net/core/skbuff.c +52 -0
58 lines not on this page · raw message
net/ipv4/ip_input.c +127 -12
207 lines not on this page · raw message
net/ipv4/ip_output.c +2 -1
@@ -272,7 +272,8 @@ static int ip_finish_output_gso(struct net *net, struct sock *sk,return -ENOMEM ; } -consume_skb (skb); +if (segs != skb) +consume_skb (skb); do { struct sk_buff *nskb = segs->next;
Signature · 1 line
2.17.1
RFC PATCH 2/2 net: Hack to enable skb list receive in the napi layer.
In reply to Steffen KlassertThis patch was used to test patch ("net: Support flow sorted skb lists for IPv4.") It is just a hack that disables GRO and does skb list receive instead. Not for merging!
--- include/linux/netdevice.h | 5 ++++- net/core/dev.c | 23 ++++++++++++++--------- net/ipv4/ip_input.c | 1 + 3 files changed, 19 insertions(+), 10 deletions(-)
include/linux/netdevice.h +4 -1
@@ -333,7 +333,10 @@ struct napi_struct {int poll_owner;#endif struct net_device *dev; -struct gro_list gro_hash[GRO_HASH_BUCKETS ]; +union { +struct list_head rx_list; +struct gro_list gro_hash[GRO_HASH_BUCKETS ]; + };struct sk_buff *skb;struct hrtimer timer;struct list_head dev_list;
net/core/dev.c +14 -9
@@ -5470,6 +5470,10 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buffint same_flow;int grow; +list_add_tail (&skb->list, &napi->rx_list); + ret =GRO_CONSUMED ; + goto out; +if (netif_elide_gro (skb->dev)) goto normal;
@@ -5559,7 +5563,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff }else if (test_bit (hash, &napi->gro_bitmask)) {__clear_bit (hash, &napi->gro_bitmask); } - +out:return ret; normal:
@@ -5958,7 +5962,7 @@ bool napi_complete_done(struct napi_struct *n, int work_done) NAPIF_STATE_IN_BUSY_POLL)))return false; -if (n->gro_bitmask) { +if (!list_empty (&n->rx_list)) {unsigned long timeout =0 ;if (work_done)
@@ -5967,8 +5971,10 @@ bool napi_complete_done(struct napi_struct *n, int work_done)if (timeout)hrtimer_start (&n->timer,ns_to_ktime (timeout),HRTIMER_MODE_REL_PINNED ); -else -napi_gro_flush (n, false); +else { +netif_receive_skb_list (&n->rx_list); +INIT_LIST_HEAD (&n->rx_list); + } }if (unlikely (!list_empty (&n->poll_list))) {/* If n->poll_list is not empty, we need to mask irqs */
@@ -6189,6 +6195,7 @@ void netif_napi_add(struct net_device *dev, struct napi_struct *napi, int (*poll)(struct napi_struct *, int), int weight) {INIT_LIST_HEAD (&napi->poll_list); +INIT_LIST_HEAD (&napi->rx_list);hrtimer_init (&napi->timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL_PINNED ); napi->timer.function = napi_watchdog;init_gro_hash (napi);
@@ -6289,11 +6296,9 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll) goto out_unlock; } -if (n->gro_bitmask) { -/* flush too old packets -* If HZ < 1000, flush all packets. -*/ -napi_gro_flush (n,HZ >=1000 ); +if (!list_empty (&n->rx_list)) { +netif_receive_skb_list (&n->rx_list); +INIT_LIST_HEAD (&n->rx_list); }/* Some drivers may have called napi_schedule
net/ipv4/ip_input.c +1 -0
@@ -729,6 +729,7 @@ void ip_list_rcv(struct list_head *head, struct packet_type *pt, }list_add_tail (&skb->list, &sublist); } +/* dispatch final sublist */ ip_sublist_rcv (&sublist, curr_dev, curr_net); }
Signature · 1 line
2.17.1