Skip to contentSkip to conversation

Improve UDP multicast receive latency

PATCH In review Shawn Bohrer 3 participants 7 messages Oct 7, 2013 – Oct 9, 2013

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v3 cover Improve UDP multicast receive latency

From: Shawn Bohrer <[email protected]>

The removal of the routing cache in 3.6 had impacted the latency of our UDP multicast workload. This patch series brings down the latency to what we were seeing with 3.4.

Patch 1 "udp: Only allow busy read/poll on connected sockets" is mostly done for correctness and because it allows unifying the unicast and multicast paths when a socket is found in early demux. It can also improve latency for a connected multicast socket if busy read/poll is used.

Patches 2&3 remove the fib lookups and restore latency for our workload to the pre 3.6 levels.

Benchmark results from a netperf UDP_RR test:
v3.12-rc3-447-g40dc9ab kernel 87961.22 transactions/s
v3.12-rc3-447-g40dc9ab + series 90587.62 transactions/s

Benchmark results from a fio 1 byte UDP multicast pingpong test (Multicast one way unicast response):
v3.12-rc3-447-g40dc9ab kernel 12.97us RTT
v3.12-rc3-447-g40dc9ab + series 12.48us RTT

v2 Changes:
- Unicast UDP early demux now requires an exact socket match and only tests first socket in UDP hash chain.
- ipv4_pktinfo_prepare() now takes a const struct sock*

v3 Changes:
- Use secondary hash for UDP unicast early demux lookup
- Double check socket match after increasing refcount in both unicast and multicast early demux lookups

Shawn Bohrer (3):
udp: Only allow busy read/poll on connected sockets
udp: ipv4: Add udp early demux
net: ipv4 only populate IP_PKTINFO when needed

 include/net/ip.h       |    2 +-
 include/net/sock.h     |    2 +-
 include/net/udp.h      |    1 +
 net/ipv4/af_inet.c     |    1 +
 net/ipv4/ip_sockglue.c |    5 +-
 net/ipv4/raw.c         |    2 +-
 net/ipv4/udp.c         |  209 +++++++++++++++++++++++++++++++++++++++++++-----
 net/ipv6/udp.c         |    5 +-
 8 files changed, 199 insertions(+), 28 deletions(-)
Signature · 1 line
1.7.7.6
#1 lore raw

PATCH v3 1/3 udp: Only allow busy read/poll on connected sockets

From: Shawn Bohrer <[email protected]>

UDP sockets can receive packets from multiple endpoints and thus may be received on multiple receive queues. Since packets packets can arrive on multiple receive queues we should not mark the napi_id for all packets. This makes busy read/poll only work for connected UDP sockets.

This additionally enables busy read/poll for UDP multicast packets as long as the socket is connected by moving the check into __udp_queue_rcv_skb().

---
 net/ipv4/udp.c |    5 +++--
 net/ipv6/udp.c |    5 +++--
 2 files changed, 6 insertions(+), 4 deletions(-)
net/ipv4/udp.c +3 -2
index c41833e..5950e12 100644
@@ -1405,8 +1405,10 @@ static int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
 {
 	int rc;
 
-	if (inet_sk(sk)->inet_daddr)
+	if (inet_sk(sk)->inet_daddr) {
 		sock_rps_save_rxhash(sk, skb);
+		sk_mark_napi_id(sk, skb);
+	}
 
 	rc = sock_queue_rcv_skb(sk, skb);
 	if (rc < 0) {
@@ -1716,7 +1718,6 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
 	if (sk != NULL) {
 		int ret;
 
-		sk_mark_napi_id(sk, skb);
 		ret = udp_queue_rcv_skb(sk, skb);
 		sock_put(sk);
 
net/ipv6/udp.c +3 -2
index 8119791..3753247 100644
@@ -549,8 +549,10 @@ static int __udpv6_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
 {
 	int rc;
 
-	if (!ipv6_addr_any(&inet6_sk(sk)->daddr))
+	if (!ipv6_addr_any(&inet6_sk(sk)->daddr)) {
 		sock_rps_save_rxhash(sk, skb);
+		sk_mark_napi_id(sk, skb);
+	}
 
 	rc = sock_queue_rcv_skb(sk, skb);
 	if (rc < 0) {
@@ -844,7 +846,6 @@ int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
 	if (sk != NULL) {
 		int ret;
 
-		sk_mark_napi_id(sk, skb);
 		ret = udpv6_queue_rcv_skb(sk, skb);
 		sock_put(sk);
 
Signature · 1 line
1.7.7.6
#2 lore raw

PATCH v3 2/3 udp: ipv4: Add udp early demux

In reply to Shawn Bohrer

From: Shawn Bohrer <[email protected]>

The removal of the routing cache introduced a performance regression for some UDP workloads since a dst lookup must be done for each packet. This change caches the dst per socket in a similar manner to what we do for TCP by implementing early_demux.

For UDP multicast we can only cache the dst if there is only one receiving socket on the host. Since caching only works when there is one receiving socket we do the multicast socket lookup using RCU.

For UDP unicast we only demux sockets with an exact match in order to not break forwarding setups. Additionally since the hash chains may be long we only check the first socket to see if it is a match and not waste extra time searching the whole chain when we might not find an exact match.

Benchmark results from a netperf UDP_RR test:
Before 87961.22 transactions/s
After 89789.68 transactions/s

Benchmark results from a fio 1 byte UDP multicast pingpong test (Multicast one way unicast response):
Before 12.97us RTT
After 12.63us RTT

---
v3 Changes:
- Use secondary hash for UDP unicast early demux lookup
- Double check socket match after increasing refcount in both unicast and multicast early demux lookups

 include/net/sock.h |    2 +-
 include/net/udp.h  |    1 +
 net/ipv4/af_inet.c |    1 +
 net/ipv4/udp.c     |  202 +++++++++++++++++++++++++++++++++++++++++++++++-----
 4 files changed, 187 insertions(+), 19 deletions(-)
include/net/sock.h +1 -1
index e3bf213..7953254 100644
@@ -218,7 +218,7 @@ struct cg_proto;
   *	@sk_lock:	synchronizer
   *	@sk_rcvbuf: size of receive buffer in bytes
   *	@sk_wq: sock wait queue and async head
-  *	@sk_rx_dst: receive input route used by early tcp demux
+  *	@sk_rx_dst: receive input route used by early demux
   *	@sk_dst_cache: destination cache
   *	@sk_dst_lock: destination cache lock
   *	@sk_policy: flow policy
include/net/udp.h +1 -0
index 510b8cb..fe4ba9f 100644
@@ -175,6 +175,7 @@ int udp_lib_get_port(struct sock *sk, unsigned short snum,
 		     unsigned int hash2_nulladdr);
 
 /* net/ipv4/udp.c */
+void udp_v4_early_demux(struct sk_buff *skb);
 int udp_get_port(struct sock *sk, unsigned short snum,
 		 int (*saddr_cmp)(const struct sock *,
 				  const struct sock *));
net/ipv4/af_inet.c +1 -0
index cfeb85c..35913fb 100644
@@ -1546,6 +1546,7 @@ static const struct net_protocol tcp_protocol = {
 };
 
 static const struct net_protocol udp_protocol = {
+	.early_demux =	udp_v4_early_demux,
 	.handler =	udp_rcv,
 	.err_handler =	udp_err,
 	.no_policy =	1,
net/ipv4/udp.c +184 -18

239 lines not on this page · raw message

Signature · 1 line
1.7.7.6
#3 lore raw

PATCH v3 3/3 net: ipv4 only populate IP_PKTINFO when needed

In reply to Shawn Bohrer

From: Shawn Bohrer <[email protected]>

The since the removal of the routing cache computing fib_compute_spec_dst() does a fib_table lookup for each UDP multicast packet received. This has introduced a performance regression for some UDP workloads.

This change skips populating the packet info for sockets that do not have IP_PKTINFO set.

Benchmark results from a netperf UDP_RR test:
Before 89789.68 transactions/s
After 90587.62 transactions/s

Benchmark results from a fio 1 byte UDP multicast pingpong test (Multicast one way unicast response):
Before 12.63us RTT
After 12.48us RTT

---
 include/net/ip.h       |    2 +-
 net/ipv4/ip_sockglue.c |    5 +++--
 net/ipv4/raw.c         |    2 +-
 net/ipv4/udp.c         |    2 +-
 4 files changed, 6 insertions(+), 5 deletions(-)
include/net/ip.h +1 -1
index 16078f4..b39ebe5 100644
@@ -459,7 +459,7 @@ int ip_options_rcv_srr(struct sk_buff *skb);
  *	Functions provided by ip_sockglue.c
  */
 
-void ipv4_pktinfo_prepare(struct sk_buff *skb);
+void ipv4_pktinfo_prepare(const struct sock *sk, struct sk_buff *skb);
 void ip_cmsg_recv(struct msghdr *msg, struct sk_buff *skb);
 int ip_cmsg_send(struct net *net, struct msghdr *msg, struct ipcm_cookie *ipc);
 int ip_setsockopt(struct sock *sk, int level, int optname, char __user *optval,
net/ipv4/ip_sockglue.c +3 -2
index 56e3445..0626f2c 100644
@@ -1052,11 +1052,12 @@ e_inval:
  * destination in skb->cb[] before dst drop.
  * This way, receiver doesnt make cache line misses to read rtable.
  */
-void ipv4_pktinfo_prepare(struct sk_buff *skb)
+void ipv4_pktinfo_prepare(const struct sock *sk, struct sk_buff *skb)
 {
 	struct in_pktinfo *pktinfo = PKTINFO_SKB_CB(skb);
 
-	if (skb_rtable(skb)) {
+	if ((inet_sk(sk)->cmsg_flags & IP_CMSG_PKTINFO) &&
+	    skb_rtable(skb)) {
 		pktinfo->ipi_ifindex = inet_iif(skb);
 		pktinfo->ipi_spec_dst.s_addr = fib_compute_spec_dst(skb);
 	} else {
net/ipv4/raw.c +1 -1
index b2fa14c..41e1d28 100644
@@ -299,7 +299,7 @@ static int raw_rcv_skb(struct sock *sk, struct sk_buff *skb)
 {
 	/* Charge it to the socket. */
 
-	ipv4_pktinfo_prepare(skb);
+	ipv4_pktinfo_prepare(sk, skb);
 	if (sock_queue_rcv_skb(sk, skb) < 0) {
 		kfree_skb(skb);
 		return NET_RX_DROP;
net/ipv4/udp.c +1 -1
index 262ea39..4226c53 100644
@@ -1544,7 +1544,7 @@ int udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
 
 	rc = 0;
 
-	ipv4_pktinfo_prepare(skb);
+	ipv4_pktinfo_prepare(sk, skb);
 	bh_lock_sock(sk);
 	if (!sock_owned_by_user(sk))
 		rc = __udp_queue_rcv_skb(sk, skb);
Signature · 1 line
1.7.7.6
maintainer
#4 lore raw
In reply to Shawn Bohrer

From: Shawn Bohrer <[email protected]>
Date: Mon, 7 Oct 2013 11:01:37 -0500

Quoted 15 lines

From: Shawn Bohrer <[email protected]>

The removal of the routing cache in 3.6 had impacted the latency of our UDP multicast workload. This patch series brings down the latency to what we were seeing with 3.4.

Patch 1 "udp: Only allow busy read/poll on connected sockets" is mostly done for correctness and because it allows unifying the unicast and multicast paths when a socket is found in early demux. It can also improve latency for a connected multicast socket if busy read/poll is used.

Patches 2&3 remove the fib lookups and restore latency for our workload to the pre 3.6 levels.

Quoted 8 lines

Benchmark results from a netperf UDP_RR test:
v3.12-rc3-447-g40dc9ab kernel 87961.22 transactions/s
v3.12-rc3-447-g40dc9ab + series 90587.62 transactions/s

Benchmark results from a fio 1 byte UDP multicast pingpong test (Multicast one way unicast response):
v3.12-rc3-447-g40dc9ab kernel 12.97us RTT
v3.12-rc3-447-g40dc9ab + series 12.48us RTT

Great work, all applied to net-next, thanks!

#5 lore raw

PATCH 1/1 udp: fix a typo in __udp4_lib_mcast_demux_lookup

From: Eric Dumazet <[email protected]>

At this point sk might contain garbage.

---

net/ipv4/udp.c +1 -1
index 4226c53..9f27bb8 100644
@@ -1847,7 +1847,7 @@ begin:
 		if (count != 1 ||
 		    unlikely(!atomic_inc_not_zero_hint(&result->sk_refcnt, 2)))
 			result = NULL;
-		else if (unlikely(!__udp_is_mcast_sock(net, sk,
+		else if (unlikely(!__udp_is_mcast_sock(net, result,
 						       loc_port, loc_addr,
 						       rmt_port, rmt_addr,
 						       dif, hnum))) {