net: foo-over-udp (fou)
Subsystems (3)
v2 cover net: foo-over-udp (fou)
This patch series implements foo-over-udp. The idea is that we can encapsulate different IP protocols in UDP packets. The rationale for this is that networking devices such as NICs and switches are usually implemented with UDP (and TCP) specific mechanims for processing. For instance, many switches and routers will implement a 5-tuple hash for UDP packets to perform Equal Cost Multipath Routing (ECMP) or RSS (on NICs). Many NICs also only provide rudimentary checksum offload (basic TCP and UDP packet), with foo-over-udp we may be able to leverage these NICs to offload checksums of tunneled packets (using checksum unnecessary conversion and eventually remote checksum offload)
An example encapsulation of IPIP over FOU is diagrammed below. As illustrated, the packet overhead for FOU is the 8 byte UDP header.
+------------------+ | IPv4 hdr | +------------------+ | UDP hdr | +------------------+ | IPv4 hdr | +------------------+ | TCP hdr | +------------------+ | TCP payload | +------------------+
Conceptually, FOU should be able to encapsulate any IP protocol. The FOU header (UDP hdr.) is essentially an inserted header between the IP header and transport, so in the case of TCP or UDP encapsulation the pseudo header would be based on the outer IP header and its length field must not include the UDP header.
* Receive
In this patch set the RX path for FOU is implemented in a new fou module. To enable FOU for a particular protocol, a UDP-FOU socket is opened to the port to receive FOU packets. The socket is mapped to the IP protocol for the packets. The XFRM mechanism used to receive encapsulated packets (udp_encap_rcv) for the port. Upon reception, the UDP is removed and packet is reinjected in the stack for the corresponding protocol associated with the socket (return -protocol from udp_encap_rcv function).
GRO is provided with the appropriate fou_gro_receive and fou_gro_complete. These routines need to know the encapsulation protocol so we save that in udp_offloads structure with the port and pass it in the napi_gro_cb structure.
* TX
This patch series implements FOU transmit encapsulation for IPIP, GRE, and SIT. This done by some common infrastructure in ip_tunnel including an ip_tunnel_encap to perform FOU encapsulation and common configuration to enable FOU on IP tunnels. FOU is configured on existing tunnels and does not create any new interfaces. The transmit and receive paths are independent, so use of FOU may be assymetric between tunnel endpoints.
* Configuration
The fou module using netlink to configure FOU receive ports. The ip command can be augmented with a fou subcommand to support this. e.g. to configure FOU for IPIP on port 5555:
ip fou add port 5555 ipproto 4
GRE, IPIP, and SIT have been modified with netlink commands to configure use of FOU on transmit. The "ip link" command will be augmented with an encap subcommand (for supporting various forms of secondary encapsulation). For instance, to configure an ipip tunnel with FOU on port 5555:
ip link add name tun1 type ipip \
remote 192.168.1.1 local 192.168.1.2 ttl 225 \
encap fou encap-sport auto encap-dport 5555* Notes
- This patch set does not implement GSO for FOU. The UDP encapsulation code assumes TEB, so that will need to be reimplemented.
- When a packet is received through FOU, the UDP header is not actually removed for the skbuf, pointers to transport header and length in the IP header are updated (like in ESP/UDP RX). A side effect is the IP header will now appear to have an incorrect checksum by an external observer (e.g. tcpdump), it will be off by sizeof UDP header. If necessary we could adjust the checksum to compensate.
- Performance results are below. My expectation is that FOU should entail little overhead (clearly there is some work to do :-) ). Optimizing UDP socket lookup for encapsulation ports should help significantly.
- I really don't expect/want devices to have special support for any of this. Generic checksum offload mechanisms (NETIF_HW_CSUM and use of CHECKSUM_COMPLETE) should be sufficient. RSS and flow steering is provided by commonly implemented UDP hashing. GRO/GSO seem fairly comparable with LRO/TSO already.
* Performance
Ran netperf TCP_RR and TCP_STREAM tests across various configurations. This was performed on bnx2x and I disabled TSO/GSO on sender to get fair comparison for FOU versus non-FOU. CPU utilization is reported for receive in TCP_STREAM.
GRE
IPv4, FOU, UDP checksum enabled
TCP_STREAM
24.85% CPU utilization
9310.6 Mbps
TCP_RR
94.2% CPU utilization
155/249/460 90/95/99% latencies
1.17018e+06 tps
IPv4, FOU, UDP checksum disabled
TCP_STREAM
31.04% CPU utilization
9302.22 Mbps
TCP_RR
94.13% CPU utilization
154/239/419 90/95/99% latencies
1.17555e+06 tps
IPv4, no FOU
TCP_STREAM
23.13% CPU utilization
9354.58 Mbps
TCP_RR
90.24% CPU utilization
156/228/360 90/95/99% latencies
1.18169e+06 tpsIPIP
FOU, UDP checksum enabled
TCP_STREAM
24.13% CPU utilization
9328 Mbps
TCP_RR
94.23
149/237/429 90/95/99% latencies
1.19553e+06 tps
FOU, UDP checksum disabled
TCP_STREAM
29.13% CPU utilization
9370.25 Mbps
TCP_RR
94.13% CPU utilization
149/232/398 90/95/99% latencies
1.19225e+06 tps
No FOU
TCP_STREAM
10.43% CPU utilization
5302.03 Mbps
TCP_RR
51.53% CPU utilization
215/324/475 90/95/99% latencies
864998 tpsSIT
FOU, UDP checksum enabled
TCP_STREAM
30.38% CPU utilization
9176.76 Mbps
TCP_RR
96.9% CPU utilization
170/281/581 90/95/99% latencies
1.03372e+06 tps
FOU, UDP checksum disabled
TCP_STREAM
39.6% CPU utilization
9176.57 Mbps
TCP_RR
97.14% CPU utilization
167/272/548 90/95/99% latencies
1.03203e+06 tps
No FOU
TCP_STREAM
11.2% CPU utilization
4636.05 Mbps
TCP_RR
59.51% CPU utilization
232/346/489 90/95/99% latencies
813199 tpsv2:
- Removed encap IP tunnel ioctls, configuration is done by netlink only.
- Don't export fou_create and fou_destroy, they are currently intended to be called within fou module only.
- Filled on tunnel netlink structures and functions for new values.
Tom Herbert (7):
net: Export inet_offloads and inet6_offloads
fou: Support for foo-over-udp RX path
fou: Add GRO support
net: Changes to ip_tunnel to support foo-over-udp encapsulation
sit: TX path for sit/UDP foo-over-udp encapsulation
ipip: TX path for IPIP/UDP foo-over-udp encapsulation
gre: TX path for GRE/UDP foo-over-udp encapsulation
include/linux/netdevice.h | 3 +- include/net/ip_tunnels.h | 19 ++- include/uapi/linux/fou.h | 32 ++++ include/uapi/linux/if_tunnel.h | 16 ++ net/ipv4/Kconfig | 10 ++ net/ipv4/Makefile | 1 + net/ipv4/fou.c | 369 +++++++++++++++++++++++++++++++++++++++++ net/ipv4/ip_gre.c | 98 ++++++++++- net/ipv4/ip_tunnel.c | 91 +++++++++- net/ipv4/ipip.c | 86 +++++++++- net/ipv4/protocol.c | 1 + net/ipv4/udp_offload.c | 5 +- net/ipv6/protocol.c | 1 + net/ipv6/sit.c | 107 ++++++++++-- 14 files changed, 815 insertions(+), 24 deletions(-) create mode 100644 include/uapi/linux/fou.h create mode 100644 net/ipv4/fou.c
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH v2 1/7 net: Export inet_offloads and inet6_offloads
Want to be able to call this in foo-over-udp offloads, etc.
--- net/ipv4/protocol.c | 1 + net/ipv6/protocol.c | 1 + 2 files changed, 2 insertions(+)
net/ipv4/protocol.c +1 -0
@@ -30,6 +30,7 @@const struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS ]__read_mostly ;const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS ]__read_mostly ; +EXPORT_SYMBOL (inet_offloads);int inet_add_protocol (const struct net_protocol *prot,unsigned char protocol) {
net/ipv6/protocol.c +1 -0
@@ -51,6 +51,7 @@ EXPORT_SYMBOL(inet6_del_protocol);#endif const struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS ]__read_mostly ; +EXPORT_SYMBOL (inet6_offloads);int inet6_add_offload (const struct net_offload *prot,unsigned char protocol) {
Signature · 1 line
2.1.0.rc2.206.gedb03e5
Re: [PATCH v2 net-next 1/7] net: Export inet_offloads and inet6_offloads
Quoted 1 line · Tom Herbert
Want to be able to call this in foo-over-udp offloads, etc.
In the L2 gro case, we did dedicated helpers gro_find_receive/complete_by_type, not sure what was the exact rational there but worth checking, jerry?
Quoted 38 lines
Signed-off-by Tom Herbert <[email protected]>--- net/ipv4/protocol.c | 1 + net/ipv6/protocol.c | 1 + 2 files changed, 2 insertions(+)net/ipv4/protocol.c +1
-0@@ -30,6 +30,7 @@const struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS ]__read_mostly ;const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS ]__read_mostly ; +EXPORT_SYMBOL (inet_offloads);int inet_add_protocol (const struct net_protocol *prot,unsigned char protocol) {net/ipv6/protocol.c +1
-1@@ -51,6 +51,7 @@ EXPORT_SYMBOL(inet6_del_protocol);#endif const struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS ]__read_mostly ; +EXPORT_SYMBOL (inet6_offloads);int inet6_add_offload (const struct net_offload *prot,unsigned char protocol) { --2.1.0.rc2.206.gedb03e5
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 net-next 1/7] net: Export inet_offloads and inet6_offloads
Quoted 7 lines · Or Gerlitz
Quoted 1 line · Tom Herbert
Want to be able to call this in foo-over-udp offloads, etc.
In the L2 gro case, we did dedicated helpers gro_find_receive/complete_by_type, not sure what was the exact rational there but worth checking, jerry?
It allows offload_base to be kept a static, but then gro_find_receive_by_type can't be inlined.
Quoted 39 lines
Quoted 39 lines
Signed-off-by Tom Herbert <[email protected]>--- net/ipv4/protocol.c | 1 + net/ipv6/protocol.c | 1 + 2 files changed, 2 insertions(+)net/ipv4/protocol.c +1
-0@@ -30,6 +30,7 @@const struct net_protocol __rcu *inet_protos[MAX_INET_PROTOS ]__read_mostly ;const struct net_offload __rcu *inet_offloads[MAX_INET_PROTOS ]__read_mostly ; +EXPORT_SYMBOL (inet_offloads);int inet_add_protocol (const struct net_protocol *prot,unsigned char protocol) {net/ipv6/protocol.c +1
-1@@ -51,6 +51,7 @@ EXPORT_SYMBOL(inet6_del_protocol);#endif const struct net_offload __rcu *inet6_offloads[MAX_INET_PROTOS ]__read_mostly ; +EXPORT_SYMBOL (inet6_offloads);int inet6_add_offload (const struct net_offload *prot,unsigned char protocol) { --2.1.0.rc2.206.gedb03e5
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 net-next 1/7] net: Export inet_offloads and inet6_offloads
Quoted 10 lines · Tom Herbert
Quoted 7 lines · Or Gerlitz
Quoted 1 line · Tom Herbert
Want to be able to call this in foo-over-udp offloads, etc.
In the L2 gro case, we did dedicated helpers gro_find_receive/complete_by_type, not sure what was the exact rational there but worth checking, jerry?
It allows
offload_baseto be kept a static, but thengro_find_receive_by_typecan't be inlined.
so we have two similar locations in the networking stack acting differently on the same/similar simple
matter... a bit problematic maintainance wise, I would say.
Re: [PATCH v2 net-next 1/7] net: Export inet_offloads and inet6_offloads
Quoted 9 lines · Or Gerlitz
Quoted 8 lines · Tom Herbert
Quoted 7 lines · Or Gerlitz
Quoted 1 line · Tom Herbert
Want to be able to call this in foo-over-udp offloads, etc.
In the L2 gro case, we did dedicated helpers gro_find_receive/complete_by_type, not sure what was the exact rational there but worth checking, jerry?
Quoted 7 lines
Quoted 2 lines
It allows
offload_baseto be kept a static, but thengro_find_receive_by_typecan't be inlined.so we have two similar locations in the networking stack acting differently on the same/similar simple
matter... a bit problematic maintainance wise, I would say.
Yes, these should be similar. I think we'd need to cleanup gro_find_receive_by_type first: export udp_offload base and inline these functions. Have skb_mac_gso_segment call this also.
David Miller
maintainerRe: [PATCH v2 net-next 1/7] net: Export inet_offloads and inet6_offloads
In reply to Tom HerbertFrom: Tom Herbert <[email protected]>
Date: Sun, 14 Sep 2014 20:07:57 -0700
Quoted 3 lines
Want to be able to call this in foo-over-udp offloads, etc.
Signed-off-by Tom Herbert <[email protected]>
I don't think inet{,6}_offloads are symbols you can "call" :-)
PATCH v2 2/7 fou: Support for foo-over-udp RX path
In reply to Tom HerbertThis patch provides a receive path for foo-over-udp. This allows direct encapsulation of IP protocols over UDP. The bound destination port is used to map to an IP protocol, and the XFRM framework (udp_encap_rcv) is used to receive encapsulated packets. Upon reception, the encapsulation header is logically removed (pointer to transport header is advanced) and the packet is reinjected into the receive path with the IP protocol indicated by the mapping.
Netlink is used to configure FOU ports. The configuration information includes the port number to bind to and the IP protocol corresponding to that port.
This should support GRE/UDP
(http://tools.ietf.org/html/draft-yong-tsvwg-gre-in-udp-encap-02),
as will as the other IP tunneling protocols (IPIP, SIT).
--- include/uapi/linux/fou.h | 32 ++++++ net/ipv4/Kconfig | 10 ++ net/ipv4/Makefile | 1 + net/ipv4/fou.c | 279 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 322 insertions(+) create mode 100644 include/uapi/linux/fou.h create mode 100644 net/ipv4/fou.c
include/uapi/linux/fou.h new file +32 -0
@@ -0,0 +1,32 @@ +/* fou.h - FOU Interface */ + +#ifndef _UAPI_LINUX_FOU_H +#define _UAPI_LINUX_FOU_H + +/* NETLINK_GENERIC related info +*/ +#define FOU_GENL_NAME "fou" +#define FOU_GENL_VERSION 0x1 + +enum { +FOU_ATTR_UNSPEC , +FOU_ATTR_PORT ,/* u16 */ +FOU_ATTR_AF ,/* u8 */ +FOU_ATTR_IPPROTO ,/* u8 */ + + __FOU_ATTR_MAX, +}; + +#define FOU_ATTR_MAX (__FOU_ATTR_MAX - 1) + +enum { +FOU_CMD_UNSPEC , +FOU_CMD_ADD , +FOU_CMD_DEL , + + __FOU_CMD_MAX, +}; + +#define FOU_CMD_MAX (__FOU_CMD_MAX - 1) + +#endif /* _UAPI_LINUX_FOU_H */
net/ipv4/Kconfig +10 -0
@@ -311,6 +311,16 @@ config NET_UDP_TUNNEL tristate default n +config NET_FOU + tristate "IP: Foo (IP protocols) over UDP" + select XFRM + select NET_UDP_TUNNEL + ---help--- + Foo over UDP allows any IP protocol to be directly encapsulated + over UDP include tunnels (IPIP, GRE, SIT). By encapsulating in UDP + network mechanisms and optimizations for UDP (such as ECMP + and RSS) can be leveraged to provide better service. + config INET_AH tristate "IP: AH transformation" select XFRM_ALGO
net/ipv4/Makefile +1 -0
@@ -20,6 +20,7 @@ obj-$(CONFIG_IP_MULTIPLE_TABLES) += fib_rules.o obj-$(CONFIG_IP_MROUTE) += ipmr.o obj-$(CONFIG_NET_IPIP) += ipip.o gre-y := gre_demux.o +obj-$(CONFIG_NET_FOU) += fou.o obj-$(CONFIG_NET_IPGRE_DEMUX) += gre.o obj-$(CONFIG_NET_IPGRE) += ip_gre.o obj-$(CONFIG_NET_UDP_TUNNEL) += udp_tunnel.o
net/ipv4/fou.c new file +279 -0
279 lines not on this page · raw message
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH v2 3/7 fou: Add GRO support
In reply to Tom HerbertImplement fou_gro_receive and fou_gro_complete, and populate these in the correponsing udp_offloads for the socket. Added ipproto to udp_offloads and pass this from UDP to the fou GRO routine in proto field of napi_gro_cb structure.
--- include/linux/netdevice.h | 3 +- net/ipv4/fou.c | 90 +++++++++++++++++++++++++++++++++++++++++++++++ net/ipv4/udp_offload.c | 5 ++- 3 files changed, 96 insertions(+), 2 deletions(-)
include/linux/netdevice.h +2 -1
@@ -1874,7 +1874,7 @@ struct napi_gro_cb {/* jiffies when first packet was created/queued */ unsigned long age; -/* Used in ipv6_gro_receive() */ +/* Used in ipv6_gro_receive() and foo-over-udp */ u16 proto;/* Used in udp_gro_receive */
@@ -1925,6 +1925,7 @@ struct packet_offload {struct udp_offload {__be16 port; +u8 ipproto;struct offload_callbacks callbacks; };
net/ipv4/fou.c +90 -0
120 lines not on this page · raw message
net/ipv4/udp_offload.c +4 -1
@@ -276,6 +276,7 @@ unflush:skb_gro_pull (skb,sizeof (struct udphdr ));/* pull encapsulating udp header */ skb_gro_postpull_rcsum (skb, uh,sizeof (struct udphdr )); +NAPI_GRO_CB (skb)->proto = uo_priv->offload->ipproto; pp = uo_priv->offload->callbacks.gro_receive (head, skb); out_unlock:
@@ -329,8 +330,10 @@ int udp_gro_complete(struct sk_buff *skb, int nhoff)break ; } -if (uo_priv !=NULL ) +if (uo_priv !=NULL ) { +NAPI_GRO_CB (skb)->proto = uo_priv->offload->ipproto; err = uo_priv->offload->callbacks.gro_complete (skb, nhoff +sizeof (struct udphdr )); + }rcu_read_unlock ();return err;
Signature · 1 line
2.1.0.rc2.206.gedb03e5
Re: [PATCH v2 net-next 3/7] fou: Add GRO support
Quoted 4 lines · Tom Herbert
Implement
fou_gro_receiveandfou_gro_complete, and populate these in the correponsingudp_offloadsfor the socket. Added ipproto toudp_offloadsand pass this from UDP to the fou GRO routine in proto field ofnapi_gro_cbstructure.
Do we really need that extra hop of fou4_gro_receive/complete? can't we somehow plant the gro receive/complete (say) GRE handlers in the udp offload
struct with the UDP port that related to (say) GRE over UDP tunneling?
Or.
Re: [PATCH v2 net-next 3/7] fou: Add GRO support
Quoted 7 lines · Or Gerlitz
Quoted 4 lines · Tom Herbert
Implement
fou_gro_receiveandfou_gro_complete, and populate these in the correponsingudp_offloadsfor the socket. Added ipproto toudp_offloadsand pass this from UDP to the fou GRO routine in proto field ofnapi_gro_cbstructure.
Quoted 5 lines
Do we really need that extra hop of fou4_gro_receive/complete? can't we somehow plant the gro receive/complete (say) GRE handlers in the udp offload
struct with the UDP port that related to (say) GRE over UDP tunneling?
That would be nice, but it isn't obvious to me how to manage the references. The offload functions are accessed with RCU pretty consistently.
Tom
Quoted 1 line
Or.
Re: [PATCH v2 net-next 3/7] fou: Add GRO support
Quoted 8 lines · Tom Herbert
Quoted 7 lines · Or Gerlitz
Quoted 4 lines · Tom Herbert
Implement
fou_gro_receiveandfou_gro_complete, and populate these in the correponsingudp_offloadsfor the socket. Added ipproto toudp_offloadsand pass this from UDP to the fou GRO routine in proto field ofnapi_gro_cbstructure.
Quoted 7 lines
Quoted 5 lines
Do we really need that extra hop of fou4_gro_receive/complete? can't we somehow plant the gro receive/complete (say) GRE handlers in the udp offload
struct with the UDP port that related to (say) GRE over UDP tunneling?That would be nice, but it isn't obvious to me how to manage the references. The offload functions are accessed with RCU pretty consistently.
Currently udp_gro_receive calls rcu_read_lock() before it invokes fou4/6_gro_receive
and rcu_read_unlock after the fou calls returns. The Fou call repeats the same practice
w.r.t the (say) GRE gro receive callback, so... what happens if we eliminate the fou part all together?
Re: [PATCH v2 net-next 3/7] fou: Add GRO support
Quoted 17 lines · Or Gerlitz
Quoted 8 lines · Tom Herbert
Quoted 7 lines · Or Gerlitz
Quoted 4 lines · Tom Herbert
Implement
fou_gro_receiveandfou_gro_complete, and populate these in the correponsingudp_offloadsfor the socket. Added ipproto toudp_offloadsand pass this from UDP to the fou GRO routine in proto field ofnapi_gro_cbstructure.Quoted 7 lines
Quoted 5 lines
Do we really need that extra hop of fou4_gro_receive/complete? can't we somehow plant the gro receive/complete (say) GRE handlers in the udp offload
struct with the UDP port that related to (say) GRE over UDP tunneling?That would be nice, but it isn't obvious to me how to manage the references. The offload functions are accessed with RCU pretty consistently.
Quoted 6 lines
Currently
udp_gro_receivecallsrcu_read_lock()before it invokes fou4/6_gro_receive
andrcu_read_unlockafter the fou calls returns. The Fou call repeats the same practice
w.r.t the (say) GRE gro receive callback, so... what happens if we eliminate the fou part all together?
Yes, we could conceivably index into inet_offloads directly from udp_gro_receive in lieu of calling the offload functions in the structure. It is more special case code in udp though and I'm not sure that makes it a win.
David Miller
maintainerRe: [PATCH v2 net-next 3/7] fou: Add GRO support
In reply to Tom HerbertFrom: Tom Herbert <[email protected]>
Date: Sun, 14 Sep 2014 20:07:59 -0700
net/ipv4/fou.c static int fou_udp_recv(struct sock *sk, struct sk_buff *skb) · line 70
Quoted 8 lines
net/ipv4/fou.c +4
-0@@ -62,6 +64,70 @@ static int fou_udp_recv(struct sock *sk, struct sk_buff *skb) sizeof(struct udphdr)); } +static inline struct sk_buff **fou_gro_receive (struct sk_buff **head, +struct sk_buff *skb, +const struct net_offload + **offloads)
Please drop the "inline" in foo.c files and let the compiler decide. And then you won't need to split up the "offloads" variable declaration like that.
PATCH v2 4/7 net: Changes to ip_tunnel to support foo-over-udp encapsulation
In reply to Tom HerbertThis patch changes IP tunnel to support (secondary) encapsulation, Foo-over-UDP. Changes include:
1) Adding tun_hlen as the tunnel header length, encap_hlen as the encapsulation header length, and hlen becomes the grand total of these.
2) Added common netlink define to support FOU encapsulation.
3) Routines to perform FOU encapsulation.
--- include/net/ip_tunnels.h | 19 ++++++++- include/uapi/linux/if_tunnel.h | 12 ++++++ net/ipv4/ip_tunnel.c | 91 +++++++++++++++++++++++++++++++++++++++++- 3 files changed, 120 insertions(+), 2 deletions(-)
include/net/ip_tunnels.h +18 -1
@@ -10,6 +10,7 @@#include <net/gro_cells.h> #include <net/inet_ecn.h> #include <net/ip.h> +#include <net/netns/generic.h> #include <net/rtnetlink.h> #if IS_ENABLED (CONFIG_IPV6 )
@@ -31,6 +32,13 @@ struct ip_tunnel_6rd_parm { };#endif +struct ip_tunnel_encap { +__u16 type; +__u16 flags; +__be16 sport; +__be16 dport; +}; +struct ip_tunnel_prl_entry {struct ip_tunnel_prl_entry __rcu *next;__be32 addr;
@@ -56,13 +64,18 @@ struct ip_tunnel {/* These four fields used only by GRE */ __u32 i_seqno;/* The last seen seqno */ __u32 o_seqno;/* The last output seqno */ -int hlen;/* Precalculated header length */ +int tun_hlen;/* Precalculated header length */ int mlink;struct ip_tunnel_dst __percpu *dst_cache;struct ip_tunnel_parm parms; +int encap_hlen;/* Encap header length (FOU,GUE) */ +struct ip_tunnel_encap encap; + +int hlen;/* tun_hlen + encap_hlen */ +/* for SIT */ #ifdef CONFIG_IPV6_SIT_6RD struct ip_tunnel_6rd_parm ip6rd;
@@ -114,6 +127,8 @@ void ip_tunnel_delete_net(struct ip_tunnel_net *itn, struct rtnl_link_ops *ops);void ip_tunnel_xmit (struct sk_buff *skb,struct net_device *dev,const struct iphdr *tnl_params,const u8 protocol);int ip_tunnel_ioctl (struct net_device *dev,struct ip_tunnel_parm *p,int cmd); +int ip_tunnel_encap (struct sk_buff *skb,struct ip_tunnel *t, +u8 *protocol,struct flowi4 *fl4);int ip_tunnel_change_mtu (struct net_device *dev,int new_mtu); struct rtnl_link_stats64 *ip_tunnel_get_stats64(struct net_device *dev,
@@ -131,6 +146,8 @@ int ip_tunnel_newlink(struct net_device *dev, struct nlattr *tb[], struct ip_tunnel_parm *p);void ip_tunnel_setup (struct net_device *dev,int net_id);void ip_tunnel_dst_reset_all (struct ip_tunnel *t); +int ip_tunnel_encap_setup (struct ip_tunnel *t, +struct ip_tunnel_encap *ipencap);/* Extract dsfield from inner protocol */ static inline u8 ip_tunnel_get_dsfield(const struct iphdr *iph,
include/uapi/linux/if_tunnel.h +12 -0
@@ -53,10 +53,22 @@ enum {IFLA_IPTUN_6RD_RELAY_PREFIX ,IFLA_IPTUN_6RD_PREFIXLEN ,IFLA_IPTUN_6RD_RELAY_PREFIXLEN , +IFLA_IPTUN_ENCAP_TYPE , +IFLA_IPTUN_ENCAP_FLAGS , +IFLA_IPTUN_ENCAP_SPORT , +IFLA_IPTUN_ENCAP_DPORT , __IFLA_IPTUN_MAX, };#define IFLA_IPTUN_MAX (__IFLA_IPTUN_MAX - 1) +enum tunnel_encap_types { +TUNNEL_ENCAP_NONE , +TUNNEL_ENCAP_FOU , +}; + +#define TUNNEL_ENCAP_FLAG_CSUM (1<<0) +#define TUNNEL_ENCAP_FLAG_CSUM6 (1<<1) +/* SIT-mode i_flags */ #define SIT_ISATAP 0x0001
net/ipv4/ip_tunnel.c +90 -1
115 lines not on this page · raw message
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH v2 5/7 sit: TX path for sit/UDP foo-over-udp encapsulation
In reply to Tom Herbert--- net/ipv6/sit.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 97 insertions(+), 10 deletions(-)
net/ipv6/sit.c +97 -10
@@ -822,6 +822,8 @@ static netdev_tx_t ipip6_tunnel_xmit(struct sk_buff *skb,int addr_type;u8 ttl;int err; +u8 protocol =IPPROTO_IPV6 ; +int t_hlen = tunnel->hlen +sizeof (struct iphdr );if (skb->protocol !=htons (ETH_P_IPV6 )) goto tx_error;
@@ -911,8 +913,14 @@ static netdev_tx_t ipip6_tunnel_xmit(struct sk_buff *skb, goto tx_error; } + skb =iptunnel_handle_offloads (skb, false,SKB_GSO_SIT ); +if (IS_ERR (skb)) { +ip_rt_put (rt); + goto out; + } +if (df) { - mtu =dst_mtu (&rt->dst) -sizeof (struct iphdr ); + mtu =dst_mtu (&rt->dst) - t_hlen;if (mtu <68 ) { dev->stats.collisions++;
@@ -947,7 +955,7 @@ static netdev_tx_t ipip6_tunnel_xmit(struct sk_buff *skb,/* * Okay, now see if we can stuff it in the buffer as-is. */ - max_headroom =LL_RESERVED_SPACE (tdev)+sizeof (struct iphdr ); + max_headroom =LL_RESERVED_SPACE (tdev) + t_hlen;if (skb_headroom (skb) < max_headroom ||skb_shared (skb) || (skb_cloned (skb) && !skb_clone_writable (skb,0 ))) {
@@ -969,14 +977,13 @@ static netdev_tx_t ipip6_tunnel_xmit(struct sk_buff *skb, ttl = iph6->hop_limit; tos =INET_ECN_encapsulate (tos,ipv6_get_dsfield (iph6)); - skb =iptunnel_handle_offloads (skb, false,SKB_GSO_SIT ); -if (IS_ERR (skb)) { +if (ip_tunnel_encap (skb, tunnel, &protocol, &fl4) <0 ) {ip_rt_put (rt); - goto out; + goto tx_error; } err =iptunnel_xmit (skb->sk, rt, skb, fl4.saddr, fl4.daddr, -IPPROTO_IPV6 , tos, ttl, df, + protocol, tos, ttl, df, !net_eq (tunnel->net,dev_net (dev)));iptunnel_xmit_stats (err, &dev->stats, dev->tstats);return NETDEV_TX_OK ;
@@ -1059,8 +1066,10 @@ static void ipip6_tunnel_bind_dev(struct net_device *dev) tdev =__dev_get_by_index (tunnel->net, tunnel->parms.link);if (tdev) { +int t_hlen = tunnel->hlen +sizeof (struct iphdr ); + dev->hard_header_len = tdev->hard_header_len +sizeof (struct iphdr ); - dev->mtu = tdev->mtu -sizeof (struct iphdr ); + dev->mtu = tdev->mtu - t_hlen;if (dev->mtu <IPV6_MIN_MTU ) dev->mtu =IPV6_MIN_MTU ; }
@@ -1307,7 +1316,10 @@ done:static int ipip6_tunnel_change_mtu (struct net_device *dev,int new_mtu) { -if (new_mtu <IPV6_MIN_MTU || new_mtu >0xFFF8 -sizeof (struct iphdr )) +struct ip_tunnel *tunnel =netdev_priv (dev); +int t_hlen = tunnel->hlen +sizeof (struct iphdr ); + +if (new_mtu <IPV6_MIN_MTU || new_mtu >0xFFF8 - t_hlen)return -EINVAL ; dev->mtu = new_mtu;return 0 ;
@@ -1338,12 +1350,15 @@ static void ipip6_dev_free(struct net_device *dev)static void ipip6_tunnel_setup (struct net_device *dev) { +struct ip_tunnel *tunnel =netdev_priv (dev); +int t_hlen = tunnel->hlen +sizeof (struct iphdr ); + dev->netdev_ops = &ipip6_netdev_ops; dev->destructor = ipip6_dev_free; dev->type =ARPHRD_SIT ; - dev->hard_header_len =LL_MAX_HEADER +sizeof (struct iphdr ); - dev->mtu =ETH_DATA_LEN -sizeof (struct iphdr ); + dev->hard_header_len =LL_MAX_HEADER + t_hlen; + dev->mtu =ETH_DATA_LEN - t_hlen; dev->flags =IFF_NOARP ; dev->priv_flags &= ~IFF_XMIT_DST_RELEASE ; dev->iflink =0 ;
@@ -1466,6 +1481,40 @@ static void ipip6_netlink_parms(struct nlattr *data[], } +/* This function returns true when ENCAP attributes are present in the nl msg */ +static bool ipip6_netlink_encap_parms (struct nlattr *data[], +struct ip_tunnel_encap *ipencap) +{ +bool ret = false; + +memset (ipencap,0 ,sizeof (*ipencap)); + +if (!data) +return ret; + +if (data[IFLA_IPTUN_ENCAP_TYPE ]) { + ret = true; + ipencap->type =nla_get_u16 (data[IFLA_IPTUN_ENCAP_TYPE ]); + } + +if (data[IFLA_IPTUN_ENCAP_FLAGS ]) { + ret = true; + ipencap->flags =nla_get_u16 (data[IFLA_IPTUN_ENCAP_FLAGS ]); + } + +if (data[IFLA_IPTUN_ENCAP_SPORT ]) { + ret = true; + ipencap->sport =nla_get_u16 (data[IFLA_IPTUN_ENCAP_SPORT ]); + } + +if (data[IFLA_IPTUN_ENCAP_DPORT ]) { + ret = true; + ipencap->dport =nla_get_u16 (data[IFLA_IPTUN_ENCAP_DPORT ]); + } + +return ret; +} +#ifdef CONFIG_IPV6_SIT_6RD /* This function returns true when 6RD attributes are present in the nl msg */ static bool ipip6_netlink_6rd_parms(struct nlattr *data[],
@@ -1509,12 +1558,20 @@ static int ipip6_newlink(struct net *src_net, struct net_device *dev, {struct net *net =dev_net (dev);struct ip_tunnel *nt; +struct ip_tunnel_encap ipencap;#ifdef CONFIG_IPV6_SIT_6RD struct ip_tunnel_6rd ip6rd;#endif int err; nt =netdev_priv (dev); + +if (ipip6_netlink_encap_parms (data, &ipencap)) { + err =ip_tunnel_encap_setup (nt, &ipencap); +if (err <0 ) +return err; + } +ipip6_netlink_parms (data, &nt->parms);if (ipip6_tunnel_locate (net, &nt->parms,0 ))
@@ -1537,15 +1594,23 @@ static int ipip6_changelink(struct net_device *dev, struct nlattr *tb[], {struct ip_tunnel *t =netdev_priv (dev);struct ip_tunnel_parm p; +struct ip_tunnel_encap ipencap;struct net *net = t->net;struct sit_net *sitn =net_generic (net, sit_net_id);#ifdef CONFIG_IPV6_SIT_6RD struct ip_tunnel_6rd ip6rd;#endif +int err;if (dev == sitn->fb_tunnel_dev)return -EINVAL ; +if (ipip6_netlink_encap_parms (data, &ipencap)) { + err =ip_tunnel_encap_setup (t, &ipencap); +if (err <0 ) +return err; + } +ipip6_netlink_parms (data, &p); if (((dev->flags & IFF_POINTOPOINT) && !p.iph.daddr) ||
@@ -1599,6 +1664,14 @@ static size_t ipip6_get_size(const struct net_device *dev)/* IFLA_IPTUN_6RD_RELAY_PREFIXLEN */ nla_total_size (2 ) + #endif +/* IFLA_IPTUN_ENCAP_TYPE */ +nla_total_size (2 ) + +/* IFLA_IPTUN_ENCAP_FLAGS */ +nla_total_size (2 ) + +/* IFLA_IPTUN_ENCAP_SPORT */ +nla_total_size (2 ) + +/* IFLA_IPTUN_ENCAP_DPORT */ +nla_total_size (2 ) + 0; }
@@ -1630,6 +1703,16 @@ static int ipip6_fill_info(struct sk_buff *skb, const struct net_device *dev) goto nla_put_failure;#endif +if (nla_put_u16 (skb,IFLA_IPTUN_ENCAP_TYPE , + tunnel->encap.type) || +nla_put_u16 (skb,IFLA_IPTUN_ENCAP_SPORT , + tunnel->encap.sport) || +nla_put_u16 (skb,IFLA_IPTUN_ENCAP_DPORT , + tunnel->encap.dport) || +nla_put_u16 (skb,IFLA_IPTUN_ENCAP_FLAGS , + tunnel->encap.dport)) + goto nla_put_failure; +return 0 ; nla_put_failure:
@@ -1651,6 +1734,10 @@ static const struct nla_policy ipip6_policy[IFLA_IPTUN_MAX + 1] = { [IFLA_IPTUN_6RD_PREFIXLEN] = { .type = NLA_U16 }, [IFLA_IPTUN_6RD_RELAY_PREFIXLEN] = { .type = NLA_U16 }, #endif + [IFLA_IPTUN_ENCAP_TYPE] = { .type = NLA_U16 }, + [IFLA_IPTUN_ENCAP_FLAGS] = { .type = NLA_U16 }, + [IFLA_IPTUN_ENCAP_SPORT] = { .type = NLA_U16 }, + [IFLA_IPTUN_ENCAP_DPORT] = { .type = NLA_U16 }, };static void ipip6_dellink (struct net_device *dev,struct list_head *head)
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH v2 6/7 ipip: TX path for IPIP/UDP foo-over-udp encapsulation
In reply to Tom Herbert--- net/ipv4/ipip.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)
net/ipv4/ipip.c +83 -3
@@ -301,7 +301,8 @@ static int ipip_tunnel_init(struct net_device *dev)memcpy (dev->dev_addr, &tunnel->parms.iph.saddr,4 );memcpy (dev->broadcast, &tunnel->parms.iph.daddr,4 ); - tunnel->hlen =0 ; + tunnel->tun_hlen =0 ; + tunnel->hlen = tunnel->tun_hlen + tunnel->encap_hlen; tunnel->parms.iph.protocol =IPPROTO_IPIP ;return ip_tunnel_init (dev); }
@@ -340,19 +341,73 @@ static void ipip_netlink_parms(struct nlattr *data[], parms->iph.frag_off =htons (IP_DF ); } +/* This function returns true when ENCAP attributes are present in the nl msg */ +static bool ipip_netlink_encap_parms (struct nlattr *data[], +struct ip_tunnel_encap *ipencap) +{ +bool ret = false; + +memset (ipencap,0 ,sizeof (*ipencap)); + +if (!data) +return ret; + +if (data[IFLA_IPTUN_ENCAP_TYPE ]) { + ret = true; + ipencap->type =nla_get_u16 (data[IFLA_IPTUN_ENCAP_TYPE ]); + } + +if (data[IFLA_IPTUN_ENCAP_FLAGS ]) { + ret = true; + ipencap->flags =nla_get_u16 (data[IFLA_IPTUN_ENCAP_FLAGS ]); + } + +if (data[IFLA_IPTUN_ENCAP_SPORT ]) { + ret = true; + ipencap->sport =nla_get_u16 (data[IFLA_IPTUN_ENCAP_SPORT ]); + } + +if (data[IFLA_IPTUN_ENCAP_DPORT ]) { + ret = true; + ipencap->dport =nla_get_u16 (data[IFLA_IPTUN_ENCAP_DPORT ]); + } + +return ret; +} +static int ipip_newlink (struct net *src_net,struct net_device *dev,struct nlattr *tb[],struct nlattr *data[]) {struct ip_tunnel_parm p; +struct ip_tunnel *t =netdev_priv (dev); +struct ip_tunnel_encap ipencap; +int err; + +if (ipip_netlink_encap_parms (data, &ipencap)) { + err =ip_tunnel_encap_setup (t, &ipencap); +if (err <0 ) +return err; + }ipip_netlink_parms (data, &p); -return ip_tunnel_newlink (dev, tb, &p); + err =ip_tunnel_newlink (dev, tb, &p); + +return err; }static int ipip_changelink (struct net_device *dev,struct nlattr *tb[],struct nlattr *data[]) { +struct ip_tunnel *t =netdev_priv (dev);struct ip_tunnel_parm p; +struct ip_tunnel_encap ipencap; +int err; + +if (ipip_netlink_encap_parms (data, &ipencap)) { + err =ip_tunnel_encap_setup (t, &ipencap); +if (err <0 ) +return err; + }ipip_netlink_parms (data, &p);
@@ -360,7 +415,9 @@ static int ipip_changelink(struct net_device *dev, struct nlattr *tb[], (!(dev->flags & IFF_POINTOPOINT) && p.iph.daddr)) return -EINVAL; -return ip_tunnel_changelink (dev, tb, &p); + err =ip_tunnel_changelink (dev, tb, &p); + +return err; }static size_t ipip_get_size (const struct net_device *dev)
@@ -378,6 +435,14 @@ static size_t ipip_get_size(const struct net_device *dev)nla_total_size (1 ) +/* IFLA_IPTUN_PMTUDISC */ nla_total_size (1 ) + +/* IFLA_IPTUN_ENCAP_TYPE */ +nla_total_size (2 ) + +/* IFLA_IPTUN_ENCAP_FLAGS */ +nla_total_size (2 ) + +/* IFLA_IPTUN_ENCAP_SPORT */ +nla_total_size (2 ) + +/* IFLA_IPTUN_ENCAP_DPORT */ +nla_total_size (2 ) +0 ; }
@@ -394,6 +459,17 @@ static int ipip_fill_info(struct sk_buff *skb, const struct net_device *dev) nla_put_u8(skb, IFLA_IPTUN_PMTUDISC, !!(parm->iph.frag_off & htons(IP_DF)))) goto nla_put_failure; + +if (nla_put_u16 (skb,IFLA_IPTUN_ENCAP_TYPE , + tunnel->encap.type) || +nla_put_u16 (skb,IFLA_IPTUN_ENCAP_SPORT , + tunnel->encap.sport) || +nla_put_u16 (skb,IFLA_IPTUN_ENCAP_DPORT , + tunnel->encap.dport) || +nla_put_u16 (skb,IFLA_IPTUN_ENCAP_FLAGS , + tunnel->encap.dport)) + goto nla_put_failure; +return 0 ; nla_put_failure:
@@ -407,6 +483,10 @@ static const struct nla_policy ipip_policy[IFLA_IPTUN_MAX + 1] = { [IFLA_IPTUN_TTL] = { .type = NLA_U8 }, [IFLA_IPTUN_TOS] = { .type = NLA_U8 }, [IFLA_IPTUN_PMTUDISC] = { .type = NLA_U8 }, + [IFLA_IPTUN_ENCAP_TYPE] = { .type = NLA_U16 }, + [IFLA_IPTUN_ENCAP_FLAGS] = { .type = NLA_U16 }, + [IFLA_IPTUN_ENCAP_SPORT] = { .type = NLA_U16 }, + [IFLA_IPTUN_ENCAP_DPORT] = { .type = NLA_U16 }, };static struct rtnl_link_ops ipip_link_ops__read_mostly = {
Signature · 1 line
2.1.0.rc2.206.gedb03e5
PATCH v2 7/7 gre: TX path for GRE/UDP foo-over-udp encapsulation
In reply to Tom Herbert--- include/uapi/linux/if_tunnel.h | 4 ++ net/ipv4/ip_gre.c | 98 +++++++++++++++++++++++++++++++++++++++--- 2 files changed, 95 insertions(+), 7 deletions(-)
include/uapi/linux/if_tunnel.h +4 -0
@@ -106,6 +106,10 @@ enum {IFLA_GRE_ENCAP_LIMIT ,IFLA_GRE_FLOWINFO ,IFLA_GRE_FLAGS , +IFLA_GRE_ENCAP_TYPE , +IFLA_GRE_ENCAP_FLAGS , +IFLA_GRE_ENCAP_SPORT , +IFLA_GRE_ENCAP_DPORT , __IFLA_GRE_MAX, };
net/ipv4/ip_gre.c +91 -7
160 lines not on this page · raw message
Signature · 1 line
2.1.0.rc2.206.gedb03e5
On Mon, Sep 15, 2014 at 6:07 AM, Tom Herbert <[email protected]> wrote:
[...]
Quoted 3 lines
* Notes
- This patch set does not implement GSO for FOU. The UDP encapsulation code assumes TEB, so that will need to be reimplemented.
Can you please clarify this point little further? Specifically, today few NICs are
advertizing NETIF_F_GSO_UDP_TUNNEL when they are practically GSO capable only w.r.t to VXLAN. What happens when such NIC expose this cap and a large guest frame goes through GRE over UDP or alike tunneling?
Quoted 5 lines
- I really don't expect/want devices to have special support for any of this. Generic checksum offload mechanisms (
NETIF_HW_CSUMand use ofCHECKSUM_COMPLETE) should be sufficient. RSS and flow steering is provided by commonly implemented UDP hashing. GRO/GSO seem fairly comparable with LRO/TSO already.
Again, today NICs are advertizing checksum offloads capability in enc_hw_features but aren't capable to compute (say) the TCP checksum of the inner
packet regardless of which actual tunneling is used (e.g VXLAN vs GRE), a bit inconsistent?
Quoted 6 lines · Or Gerlitz
On Mon, Sep 15, 2014 at 6:07 AM, Tom Herbert <[email protected]> wrote:
[...]Quoted 3 lines
* Notes
- This patch set does not implement GSO for FOU. The UDP encapsulation code assumes TEB, so that will need to be reimplemented.
Quoted 6 lines
Can you please clarify this point little further? Specifically, today few NICs are
advertizingNETIF_F_GSO_UDP_TUNNELwhen they are practically GSO capable only w.r.t to VXLAN. What happens when such NIC expose this cap and a large guest frame goes through GRE over UDP or alike tunneling?
My interpretation is that NETIF_F_GSO_UDP_TUNNEL means L3/L4 encapsulation over UDP, not VXLAN. If the NIC implements things properly following the generic interface then I believe it should work with various flavors of UDP encapsulation (FOU, GUE, VXLAN, VXLAN-gpe, geneve, LISP, L2TP, nvgre, or whatever else people might dream up). This presumes that any encapsulation headers doesn't require any per segment update (so no GRE csum for instance). The stack will set up inner headers as needed, which should enough to provide to devices the offsets inner IP and TCP header which are needed for the the TSO operation (outer IP and UDP can be deduced also).
Quoted 11 lines
Quoted 5 lines
- I really don't expect/want devices to have special support for any of this. Generic checksum offload mechanisms (
NETIF_HW_CSUMand use ofCHECKSUM_COMPLETE) should be sufficient. RSS and flow steering is provided by commonly implemented UDP hashing. GRO/GSO seem fairly comparable with LRO/TSO already.Again, today NICs are advertizing checksum offloads capability in
enc_hw_featuresbut aren't capable to compute (say) the TCP checksum of the inner
packet regardless of which actual tunneling is used (e.g VXLAN vs GRE), a bit inconsistent?
I doubt this is true of all NICs! For instance, a NIC that implements NETIF_F_HW_CSUM should have no problem computing an encapsulated checksums in just about any scenario. Both, checksum offload and TSO can be supported for arbitrary flavors of UDP encapsulation if NICs use protocol agnostic means as opposed to protocol specific means that require a lot of parsing the packets themselves. Look at the long standing comments in sk_buff about why protocol specific methods like CHECKSUM_UNNECESSARY and NETIF_F_IP_CSUM are bad ideas. With the emergence of encapsulation these are now *really* bad ideas!
If devices are interpreting NETIF_F_GSO_UDP_TUNNEL as VXLAN would break when presented with any other flavor of UDP encapsulation, then we should probably define NETIF_F_GSO_VXLAN_TUNNEL just for that case to maintain backwards compatibility.
Quoted 24 lines · Tom Herbert
Quoted 6 lines · Or Gerlitz
On Mon, Sep 15, 2014 at 6:07 AM, Tom Herbert <[email protected]> wrote:
[...]Quoted 3 lines
* Notes
- This patch set does not implement GSO for FOU. The UDP encapsulation code assumes TEB, so that will need to be reimplemented.Quoted 6 lines
Can you please clarify this point little further? Specifically, today few NICs are
advertizingNETIF_F_GSO_UDP_TUNNELwhen they are practically GSO capable only w.r.t to VXLAN. What happens when such NIC expose this cap and a large guest frame goes through GRE over UDP or alike tunneling?My interpretation is that
NETIF_F_GSO_UDP_TUNNELmeans L3/L4 encapsulation over UDP, not VXLAN. If the NIC implements things properly following the generic interface then I believe it should work with various flavors of UDP encapsulation (FOU, GUE, VXLAN, VXLAN-gpe, geneve, LISP, L2TP, nvgre, or whatever else people might dream up). This presumes that any encapsulation headers doesn't require any per segment update (so no GRE csum for instance). The stack will set up inner headers as needed, which should enough to provide to devices the offsets inner IP and TCP header which are needed for the the TSO operation (outer IP and UDP can be deduced also).
Quoted 1 line
From the NICs that I am familiar with this is mostly true. The main
part that is missing from the current implementation is a length
limit: just because the hardware can skip over headers to an offset doesn't mean that it can do so to an arbitrary depth. For example, in the NICs that are exposing VXLAN as NETIF_F_GSO_UDP_TUNNEL we can probably assume that this is limited to 8 bytes. With the Intel NICs that were just announced with Geneve support, this limit has been increased to 64. If we add a parameter to the driver interface to expose this then it should be generic across tunnels.
Quoted 25 lines · Jesse Gross
Quoted 13 lines · Tom Herbert
Quoted 6 lines · Or Gerlitz
On Mon, Sep 15, 2014 at 6:07 AM, Tom Herbert <[email protected]> wrote:
[...]Quoted 3 lines
* Notes
- This patch set does not implement GSO for FOU. The UDP encapsulation code assumes TEB, so that will need to be reimplemented.Quoted 6 lines
Can you please clarify this point little further? Specifically, today few NICs are
advertizingNETIF_F_GSO_UDP_TUNNELwhen they are practically GSO capable only w.r.t to VXLAN. What happens when such NIC expose this cap and a large guest frame goes through GRE over UDP or alike tunneling?Quoted 10 lines
My interpretation is that
NETIF_F_GSO_UDP_TUNNELmeans L3/L4 encapsulation over UDP, not VXLAN. If the NIC implements things properly following the generic interface then I believe it should work with various flavors of UDP encapsulation (FOU, GUE, VXLAN, VXLAN-gpe, geneve, LISP, L2TP, nvgre, or whatever else people might dream up). This presumes that any encapsulation headers doesn't require any per segment update (so no GRE csum for instance). The stack will set up inner headers as needed, which should enough to provide to devices the offsets inner IP and TCP header which are needed for the the TSO operation (outer IP and UDP can be deduced also).
Quoted 9 lines
From the NICs that I am familiar with this is mostly true. The main part that is missing from the current implementation is a length
limit: just because the hardware can skip over headers to an offset doesn't mean that it can do so to an arbitrary depth. For example, in the NICs that are exposing VXLAN asNETIF_F_GSO_UDP_TUNNELwe can probably assume that this is limited to 8 bytes. With the Intel NICs that were just announced with Geneve support, this limit has been increased to 64. If we add a parameter to the driver interface to expose this then it should be generic across tunnels.
Sounds reasonable, although I think you'll need to define precisely what length refers to.
Tom
Quoted 36 lines · Tom Herbert
Quoted 25 lines · Jesse Gross
Quoted 13 lines · Tom Herbert
Quoted 6 lines · Or Gerlitz
On Mon, Sep 15, 2014 at 6:07 AM, Tom Herbert <[email protected]> wrote:
[...]Quoted 3 lines
* Notes
- This patch set does not implement GSO for FOU. The UDP encapsulation code assumes TEB, so that will need to be reimplemented.Quoted 6 lines
Can you please clarify this point little further? Specifically, today few NICs are
advertizingNETIF_F_GSO_UDP_TUNNELwhen they are practically GSO capable only w.r.t to VXLAN. What happens when such NIC expose this cap and a large guest frame goes through GRE over UDP or alike tunneling?Quoted 10 lines
My interpretation is that
NETIF_F_GSO_UDP_TUNNELmeans L3/L4 encapsulation over UDP, not VXLAN. If the NIC implements things properly following the generic interface then I believe it should work with various flavors of UDP encapsulation (FOU, GUE, VXLAN, VXLAN-gpe, geneve, LISP, L2TP, nvgre, or whatever else people might dream up). This presumes that any encapsulation headers doesn't require any per segment update (so no GRE csum for instance). The stack will set up inner headers as needed, which should enough to provide to devices the offsets inner IP and TCP header which are needed for the the TSO operation (outer IP and UDP can be deduced also).Quoted 9 lines
From the NICs that I am familiar with this is mostly true. The main part that is missing from the current implementation is a length
limit: just because the hardware can skip over headers to an offset doesn't mean that it can do so to an arbitrary depth. For example, in the NICs that are exposing VXLAN asNETIF_F_GSO_UDP_TUNNELwe can probably assume that this is limited to 8 bytes. With the Intel NICs that were just announced with Geneve support, this limit has been increased to 64. If we add a parameter to the driver interface to expose this then it should be generic across tunnels.
Quoted 2 lines
Sounds reasonable, although I think you'll need to define precisely what length refers to.
I agree, the definition is important.
Quoted 15 lines · Jesse Gross
Quoted 10 lines · Tom Herbert
My interpretation is that
NETIF_F_GSO_UDP_TUNNELmeans L3/L4
encapsulation over UDP, not VXLAN.
If the NIC implements things properly following the generic interface then I believe it should work with various flavors of UDP encapsulation (FOU, GUE, VXLAN, VXLAN-gpe,
geneve, LISP, L2TP, nvgre, or whatever else people might dream up).
This presumes that any encapsulation headers doesn't require any per
segment update (so no GRE csum for instance). The stack will set up
inner headers as needed, which should enough to provide to devices the
offsets inner IP and TCP header which are needed for the the TSO
operation (outer IP and UDP can be deduced also).
Quoted 9 lines
From the NICs that I am familiar with this is mostly true. The main part that is missing from the current implementation is a length
limit: just because the hardware can skip over headers to an offset doesn't mean that it can do so to an arbitrary depth. For example, in the NICs that are exposing VXLAN asNETIF_F_GSO_UDP_TUNNELwe can probably assume that this is limited to 8 bytes. With the Intel NICs that were just announced with Geneve support, this limit has been increased to 64. If we add a parameter to the driver interface to expose this then it should be generic across tunnels.
I'm not sure to see why the length limit became our primary concern here...
Fact is that we have nice set of NICs drivers in the kernel that do advertize the GSO_UDP_TUNNEL feature but their HW isn't capable to segment all of:
FOU, GUE, VXLAN, VXLAN-gpe, geneve, LISP, L2TP, nvgre, or whatever else people might dream up, right?
So we need to fix that and let each NIC properly advertize up to the stack what they
can segment in HW and what not which means that networking code would have to do that in SW for (say) 64KB guest TCP segment that just went through this encapsulation.
As long as Linux didn't support any UDP encapsulation other then VXLAN it worked,
but soon will too easily broken, and I vote for the fix to be part of the FOU series, so we have
the kernel functional also once it applied...
Even if the encapsulated headers need no update per segment (and they always do, e.g the IP ID field of the outer IP header) still a certain HW may not be able
to do TCP segmentation under any encapsulation scheme.
And in that respect, I am not sure to follow on the " If the NIC implements things properly
following the generic interface" comment.
Or.
Quoted 26 lines · Or Gerlitz
Quoted 15 lines · Jesse Gross
Quoted 10 lines · Tom Herbert
My interpretation is that
NETIF_F_GSO_UDP_TUNNELmeans L3/L4
encapsulation over UDP, not VXLAN.
If the NIC implements things properly following the generic interface then I believe it should work with various flavors of UDP encapsulation (FOU, GUE, VXLAN, VXLAN-gpe,
geneve, LISP, L2TP, nvgre, or whatever else people might dream up).
This presumes that any encapsulation headers doesn't require any per
segment update (so no GRE csum for instance). The stack will set up
inner headers as needed, which should enough to provide to devices the
offsets inner IP and TCP header which are needed for the the TSO
operation (outer IP and UDP can be deduced also).Quoted 9 lines
From the NICs that I am familiar with this is mostly true. The main part that is missing from the current implementation is a length
limit: just because the hardware can skip over headers to an offset doesn't mean that it can do so to an arbitrary depth. For example, in the NICs that are exposing VXLAN asNETIF_F_GSO_UDP_TUNNELwe can probably assume that this is limited to 8 bytes. With the Intel NICs that were just announced with Geneve support, this limit has been increased to 64. If we add a parameter to the driver interface to expose this then it should be generic across tunnels.
Quoted 2 lines
I'm not sure to see why the length limit became our primary concern here...
Like Jesse mentioned above, looks like some NICs may have assumed all encapsulation headers are eight bytes (which allows HW to implement everything in fixed offsets). But this length is not a universal
constant: FOU is zero length encapsulation headers, GUE or geneve is variable. The driver should really be checking if NIC can handle the length and if it can't perform GSO in software-- I don't think we'll need to expose this in the features.
Quoted 27 lines
Fact is that we have nice set of NICs drivers in the kernel that do advertize the
GSO_UDP_TUNNELfeature but their HW isn't capable to segment all of:
FOU, GUE, VXLAN, VXLAN-gpe, geneve, LISP, L2TP, nvgre, or whatever else people might dream up, right?So we need to fix that and let each NIC properly advertize up to the stack what they
can segment in HW and what not which means that networking code would have to do that in SW for (say) 64KB guest TCP segment that just went through this encapsulation.As long as Linux didn't support any UDP encapsulation other then VXLAN it worked,
but soon will too easily broken, and I vote for the fix to be part of the FOU series, so we have
the kernel functional also once it applied...Even if the encapsulated headers need no update per segment (and they always do, e.g the IP ID field of the outer IP header) still a certain HW may not be able
to do TCP segmentation under any encapsulation scheme.And in that respect, I am not sure to follow on the " If the NIC implements things properly
following the generic interface" comment.Or.
Quoted 29 lines · Tom Herbert
Quoted 26 lines · Or Gerlitz
Quoted 15 lines · Jesse Gross
Quoted 10 lines · Tom Herbert
My interpretation is that
NETIF_F_GSO_UDP_TUNNELmeans L3/L4
encapsulation over UDP, not VXLAN.
If the NIC implements things properly following the generic interface then I believe it should work with various flavors of UDP encapsulation (FOU, GUE, VXLAN, VXLAN-gpe,
geneve, LISP, L2TP, nvgre, or whatever else people might dream up).
This presumes that any encapsulation headers doesn't require any per
segment update (so no GRE csum for instance). The stack will set up
inner headers as needed, which should enough to provide to devices the
offsets inner IP and TCP header which are needed for the the TSO
operation (outer IP and UDP can be deduced also).Quoted 9 lines
From the NICs that I am familiar with this is mostly true. The main part that is missing from the current implementation is a length
limit: just because the hardware can skip over headers to an offset doesn't mean that it can do so to an arbitrary depth. For example, in the NICs that are exposing VXLAN asNETIF_F_GSO_UDP_TUNNELwe can probably assume that this is limited to 8 bytes. With the Intel NICs that were just announced with Geneve support, this limit has been increased to 64. If we add a parameter to the driver interface to expose this then it should be generic across tunnels.Quoted 1 line
I'm not sure to see why the length limit became our primary concern here...
Quoted 7 lines
Like Jesse mentioned above, looks like some NICs may have assumed all encapsulation headers are eight bytes (which allows HW to implement everything in fixed offsets). But this length is not a universal
constant: FOU is zero length encapsulation headers, GUE or geneve is variable. The driver should really be checking if NIC can handle the length and if it can't perform GSO in software-- I don't think we'll need to expose this in the features.
I understand that for some NICs there's a claim saying the essence of the limitation lies in an assumption on fixed length of the encapsulation headers -- and BTW for VXLAN it's 50 (= 14 + 20 + 8 +
8) bytes, not eight. So newer NICs or new brands of existing NICs should be more flexible.
If I correctly read your comment "The driver should really be checking if NIC can handle the length and if it can't perform GSO in software" as saying that a SW GSO call should be made from within the driver when they can't serve GSO under some encap scheme -- I don't think this is the correct track, the driver should advertize up what they can do in HW so the stack does in SW what's not supported.
Another clarification - so FOU doesn't supersedes GUE? what's their difference...?
Or.
Quoted 44 lines · Or Gerlitz
Quoted 29 lines · Tom Herbert
Quoted 26 lines · Or Gerlitz
Quoted 15 lines · Jesse Gross
Quoted 10 lines · Tom Herbert
My interpretation is that
NETIF_F_GSO_UDP_TUNNELmeans L3/L4
encapsulation over UDP, not VXLAN.
If the NIC implements things properly following the generic interface then I believe it should work with various flavors of UDP encapsulation (FOU, GUE, VXLAN, VXLAN-gpe,
geneve, LISP, L2TP, nvgre, or whatever else people might dream up).
This presumes that any encapsulation headers doesn't require any per
segment update (so no GRE csum for instance). The stack will set up
inner headers as needed, which should enough to provide to devices the
offsets inner IP and TCP header which are needed for the the TSO
operation (outer IP and UDP can be deduced also).Quoted 9 lines
From the NICs that I am familiar with this is mostly true. The main part that is missing from the current implementation is a length
limit: just because the hardware can skip over headers to an offset doesn't mean that it can do so to an arbitrary depth. For example, in the NICs that are exposing VXLAN asNETIF_F_GSO_UDP_TUNNELwe can probably assume that this is limited to 8 bytes. With the Intel NICs that were just announced with Geneve support, this limit has been increased to 64. If we add a parameter to the driver interface to expose this then it should be generic across tunnels.Quoted 1 line
I'm not sure to see why the length limit became our primary concern here...
Quoted 7 lines
Like Jesse mentioned above, looks like some NICs may have assumed all encapsulation headers are eight bytes (which allows HW to implement everything in fixed offsets). But this length is not a universal
constant: FOU is zero length encapsulation headers, GUE or geneve is variable. The driver should really be checking if NIC can handle the length and if it can't perform GSO in software-- I don't think we'll need to expose this in the features.I understand that for some NICs there's a claim saying the essence of the limitation lies in an assumption on fixed length of the encapsulation headers -- and BTW for VXLAN it's 50 (= 14 + 20 + 8 +
8) bytes, not eight. So newer NICs or new brands of existing NICs should be more flexible.
Quoted 7 lines
If I correctly read your comment "The driver should really be checking if NIC can handle the length and if it can't perform GSO in software" as saying that a SW GSO call should be made from within the driver when they can't serve GSO under some encap scheme -- I don't think this is the correct track, the driver should advertize up what they can do in HW so the stack does in SW what's not supported.
The problem is that it is likely impractical for drivers to advertise all possible constraints of their HW. Right now we have the features flags, but that is very limited and we really can't afford to add a new value for every permutation. There are just to many dimensions. Some devices might expect fixed length headers, some might not but might have other length constraints. Some may perfectly happy with v4/v4 but maybe choke on combinations with IPv6. Others may not mind IP options or extension headers, some might be okay. Some devices might not like certain packet layouts, etc.
There is precedent for the driver punting to software mechanisms when it can't handle something. For instance, cxgb veth call skb_checksum_help for resolving UDP checksums, myri10ge and marvell controller call it for headers that are too large. gianfar calls it per some errata condition...
If we can't do GSO from within the drivers, then another alternative would be to add an ndo_gso_check function to call when stack is deciding whether to do SW GSO.
Quoted 3 lines
Another clarification - so FOU doesn't supersedes GUE? what's their difference...?
FOU is direct encapsulation of IP protocol packets in UDP payload. GUE is an encapsulation header between UDP and the encapsulated IP protocol packet. http://tools.ietf.org/html/draft-herbert-gue-01
Tom
Quoted 1 line
Or.
Quoted 9 lines · Tom Herbert
Quoted 7 lines · Or Gerlitz
Quoted 5 lines · Tom Herbert
- I really don't expect/want devices to have special support for any of this. Generic checksum offload mechanisms (
NETIF_HW_CSUMand use ofCHECKSUM_COMPLETE) should be sufficient. RSS and flow steering is provided by commonly implemented UDP hashing. GRO/GSO seem fairly comparable with LRO/TSO already.
Quoted 8 lines
Quoted 4 lines
Again, today NICs are advertizing checksum offloads capability in
enc_hw_featuresbut aren't capable to compute (say) the TCP checksum of the inner packet regardless of which actual tunneling is used (e.g VXLAN vs GRE), a bit inconsistent?I doubt this is true of all NICs! For instance, a NIC that implements
NETIF_F_HW_CSUMshould have no problem computing an encapsulated checksums in just about any scenario.
The comment for NETIF_F_HW_CSUM says "Can checksum all the packets" -- so your interpretation is that NICs supporting that will always report CHECKSUM_COMPLETE, OK. But there are bunch of 10/40Gbs NIC drivers that don't report the HW_CSUM bit in neither of the ->features and ->hw_enc_features, the system should act in a manner that supports them.
Quoted 7 lines
Both, checksum offload and TSO
can be supported for arbitrary flavors of UDP encapsulation if NICs use protocol agnostic means as opposed to protocol specific means that require a lot of parsing the packets themselves. Look at the long standing comments insk_buffabout why protocol specific methods likeCHECKSUM_UNNECESSARYandNETIF_F_IP_CSUMare bad ideas. With the emergence of encapsulation these are now *really* bad ideas!
So we go and throw away the HW?
Or.
Quoted 33 lines · Or Gerlitz
Quoted 9 lines · Tom Herbert
Quoted 7 lines · Or Gerlitz
Quoted 5 lines · Tom Herbert
- I really don't expect/want devices to have special support for any of this. Generic checksum offload mechanisms (
NETIF_HW_CSUMand use ofCHECKSUM_COMPLETE) should be sufficient. RSS and flow steering is provided by commonly implemented UDP hashing. GRO/GSO seem fairly comparable with LRO/TSO already.Quoted 8 lines
Quoted 4 lines
Again, today NICs are advertizing checksum offloads capability in
enc_hw_featuresbut aren't capable to compute (say) the TCP checksum of the inner packet regardless of which actual tunneling is used (e.g VXLAN vs GRE), a bit inconsistent?I doubt this is true of all NICs! For instance, a NIC that implements
NETIF_F_HW_CSUMshould have no problem computing an encapsulated checksums in just about any scenario.The comment for
NETIF_F_HW_CSUMsays "Can checksum all the packets" -- so your interpretation is that NICs supporting that will always reportCHECKSUM_COMPLETE, OK. But there are bunch of 10/40Gbs NIC drivers that don't report theHW_CSUMbit in neither of the ->features and ->hw_enc_features, the system should act in a manner that supports them.Quoted 7 lines
Both, checksum offload and TSO
can be supported for arbitrary flavors of UDP encapsulation if NICs use protocol agnostic means as opposed to protocol specific means that require a lot of parsing the packets themselves. Look at the long standing comments insk_buffabout why protocol specific methods likeCHECKSUM_UNNECESSARYandNETIF_F_IP_CSUMare bad ideas. With the emergence of encapsulation these are now *really* bad ideas!
Quoted 2 lines
So we go and throw away the HW?
No, and this is exactly the point! I shouldn't have to throw away all my deployed hardware to just to get support for the latest encapsulation protocol du jour. Fortunately, we'll be able to code around mosts of the limitations of deployed NICs that don't implement generic mechanisms (with UDP RSS, checksum conversions, remote checksum offload). But if you're contemplating a new NIC, *please* consider implementing generic, protocol agnostic mechanisms.
Quoted 1 line
Or.
Quoted 4 lines · Tom Herbert
[...] Fortunately, we'll be able to code
around mosts of the limitations of deployed NICs that don't implement generic mechanisms (with UDP RSS, checksum conversions, remote checksum offload) [...]
So UDP RSS is a clear requirement and NICs have it by now.
Re checksum conversions, I assume you mean the ability on the RX path to report CHECKSUM_COMPLETE on any sort of IP packet potentially having multiple encapsulations, right? how would you phrase (and model) the other way around, e.g what is the generic requirement on the TX path w.r.t checksum offload?
And I wasn't sure what do you mean by remote checksum offload, can you clarify?
Or.
David Miller
maintainerFrom: Or Gerlitz <[email protected]>
Date: Tue, 16 Sep 2014 23:04:30 +0300
Quoted 2 lines
And I wasn't sure what do you mean by remote checksum offload, can you clarify?
http://vger.kernel.org/encapsulation_offloads.pdf
Page 14.
Quoted 7 lines · David Miller
From: Or Gerlitz <[email protected]>
Date: Tue, 16 Sep 2014 23:04:30 +0300Quoted 1 line
And I wasn't sure what do you mean by remote checksum offload, can you clarify?
Oh, thanks for the pointer, will take a look.
Or.
Quoted 8 lines · Or Gerlitz
Quoted 4 lines · Tom Herbert
[...] Fortunately, we'll be able to code
around mosts of the limitations of deployed NICs that don't implement generic mechanisms (with UDP RSS, checksum conversions, remote checksum offload) [...]So UDP RSS is a clear requirement and NICs have it by now.
Quoted 8 lines
Re checksum conversions, I assume you mean the ability on the RX path to report
CHECKSUM_COMPLETEon any sort of IP packet potentially having multiple encapsulations, right? how would you phrase (and model) the other way around, e.g what is the generic requirement on the TX path w.r.t checksum offload?And I wasn't sure what do you mean by remote checksum offload, can you clarify?
This is the part intend to solve TX checksum offload for encapsulation for "dumb" devices. Unlike, checksum conversion, this one requires a bit of protocol support so we need an extensible encapsulation header.
http://tools.ietf.org/html/draft-herbert-remotecsumoffload-00
Quoted 1 line
Or.