udp: Record RPS flow in socket operations

When receiving a packet on an unconnected UDP socket clear the flow table for the corresponding hash. This is needed so flows over unconnected UDP sockets will use RPS instead of using what is present in the flow table. In particular, this avoids having flows over unconnected sockets be perpetually steered by unrelated entries in the flow table (idle TCP connections for instance).

Tested:

First filled up the RPS flow tables by creating a bunch of TCP connections and letting them turn idle. Next, run netperf UDP_RR with 200 flows.

Before fix:
Client (connected UDP)

    81.15% CPU uilization
  Server (unneconnedted UDP)
    83.63% CPU uilization

118/167/249 90/95/99% latencies
1.59215e+06 tps

After fix:
Client (connected UDP)

    81.13% CPU uilization
  Server (unneconnedted UDP)
    80.68% CPU uilization

116/167/248 90/95/99% latencies
1.61048e+06 tps

Signed-off-by Tom Herbert <[email protected]>

---
 net/ipv4/udp.c | 5 +++++
 1 file changed, 5 insertions(+)

net/ipv4/udp.c +0 -0

index 9a0d346..e58d841 100644

Quoted 39 lines · Tom Herbert

When receiving a packet on an unconnected UDP socket clear the flow table for the corresponding hash. This is needed so flows over unconnected UDP sockets will use RPS instead of using what is present in the flow table. In particular, this avoids having flows over unconnected sockets be perpetually steered by unrelated entries in the flow table (idle TCP connections for instance).

Tested:

First filled up the RPS flow tables by creating a bunch of TCP connections and letting them turn idle. Next, run netperf UDP_RR with 200 flows.

Before fix:
Client (connected UDP)

    81.15% CPU uilization
  Server (unneconnedted UDP)
    83.63% CPU uilization

118/167/249 90/95/99% latencies
1.59215e+06 tps

After fix:
Client (connected UDP)

    81.13% CPU uilization
  Server (unneconnedted UDP)
    80.68% CPU uilization

116/167/248 90/95/99% latencies
1.61048e+06 tps

Signed-off-by Tom Herbert <[email protected]>

---
 net/ipv4/udp.c | 5 +++++
 1 file changed, 5 insertions(+)

net/ipv4/udp.c +0 -0

index 9a0d346..e58d841 100644

net/ipv4/udp.c static int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb) · line 1459

Quoted 9 lines

net/ipv4/udp.c +5 -0

@@ -1451,6 +1451,11 @@ static int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
      if (inet_sk(sk)->inet_daddr) {
              sock_rps_save_rxhash(sk, skb);
              sk_mark_napi_id(sk, skb);
+     } else {
+             /* For an unconnected socket reset flow hash so that related
+              * flow will use RPS.
+              */
+             sock_rps_reset_flow_hash(skb->hash);
      }

I believe I already said this patch was wrong Tom.

We need something else for UDP packets.

Its not because RFS is wrong for UDP packets that we want to make it worse for TCP traffic.

We do now want UDP packets to gradually make flow table empty.

Please try this patch and provide real data to support your points.

If a TCP connection is hot it will continually refresh the table for that connection, if connection becomes idle it only takes one received packet to restore the CPU. The only time there could be a persistent problem is if collision rate is high (which probably means table is too small).

Quoted 2 lines · Tom Herbert

Please try this patch and provide real data to support your points.

Yep. This is not good, I confirm my fear.

Google servers are shifting to serve both TCP & UDP traffic (QUIC protocol), with an increasing UDP load.

Millions of packets per second per host, from millions of different sources...

And your patch voids the RFS table, adds another cache miss in fast path for UDP rx path which is already too expensive.

Quoted 5 lines

If a TCP connection is hot it will continually refresh the table for that connection, if connection becomes idle it only takes one received packet to restore the CPU. The only time there could be a persistent problem is if collision rate is high (which probably means table is too small).

RFS already has a low hit/miss rate, this patch does not help neither UDP or TCP.

Ideally, RFS should be enabled on a protocol base, not an agnostic u32 flow hash.

Whatever strategy you implement, as long as different protocols share a common hash table, it wont be perfect for mixed workloads.

Fundamental problem is that when an UDP packet comes, its not possible to know if its a 'flow' or 'not', unless we perform an expensive lookup, and then RPS/RFS cost becomes prohibitive.

While for TCP, the current RFS cache miss is good enough, because about all packets are for connected flows. We eventually have bad steering for <not yet established> flows where the stack performs poorly anyway.

This indicates nothing about the merits of this patch. Nevertheless, in order to avoid further rat-holing and since this patch does change a long standing behavior I'll will respin to make it enabled only by sysctl.

From: Tom Herbert <[email protected]>
Date: Mon, 27 Oct 2014 18:09:25 -0700

Quoted 4 lines

This indicates nothing about the merits of this patch. Nevertheless, in order to avoid further rat-holing and since this patch does change a long standing behavior I'll will respin to make it enabled only by sysctl.

Kind of disappointed on my end that you haven't addressed Eric's main point, which is that:

1) A hash table shared between protocols will perform poorly for mixed workloads which are becomming increasingly common.

2) UDP is fundamentally different from TCP in that the issue of 'flow' vs. 'non-flow' packets

I personally do not see you avoiding this conversation by simply hiding the new behavior behind a sysctl, I still want you to address it before I apply anything.

UDP tunnels are becoming increasingly common. VXLAN, FOU, GUE, geneve, l2tp, esp/UDP, GRE/UDP, nvgre, etc. all rely on steering based on the outer header without deep inspection. When the source port is set to inner hash RFS works as is and steering is effectively done based inner TCP connections. If aRFS supports UDP, then this should just work also for UDP tunnels (another instance where we don't need protocol specific support in devices for tunneling).

Quoted 7 lines · Tom Herbert

UDP tunnels are becoming increasingly common. VXLAN, FOU, GUE, geneve, l2tp, esp/UDP, GRE/UDP, nvgre, etc. all rely on steering based on the outer header without deep inspection. When the source port is set to inner hash RFS works as is and steering is effectively done based inner TCP connections. If aRFS supports UDP, then this should just work also for UDP tunnels (another instance where we don't need protocol specific support in devices for tunneling).

If you really wanted to solve this, you would need to change RFS to be aware of the tunnel and find L4 information, instead of current implementation stopping at first UDP layer.

But as get_rps_cpu() / __skb_flow_dissect() have no way to find this, you instead chose to invalidate RFS and maybe rely on RPS, because it might help your workload.

Just to be clear : I tested the patch and saw a regression in my tests, sending as little as one million UDP packets per second on the target.

Not only UDP rx processing was slower, but TCP flows were impacted.

With a table of 65536 slots, each slot was written 16 times per second in average.

Google kernels have RFS_Hit/FRS_Miss snmp counters to catch this kind of problems. Maybe I should upstream this part.

As I said, for some applications (like DNS which I suspect you're basically emulating) it is infeasible to size the table. Try disabling RFS for your test.

Quoted 7 lines · Tom Herbert

UDP tunnels are becoming increasingly common. VXLAN, FOU, GUE, geneve, l2tp, esp/UDP, GRE/UDP, nvgre, etc. all rely on steering based on the outer header without deep inspection. When the source port is set to inner hash RFS works as is and steering is effectively done based inner TCP connections. If aRFS supports UDP, then this should just work also for UDP tunnels (another instance where we don't need protocol specific support in devices for tunneling).

If you really wanted to solve this, you would need to change RFS to be aware of the tunnel and find L4 information, instead of current implementation stopping at first UDP layer.

But as get_rps_cpu() / __skb_flow_dissect() have no way to find this, you instead chose to invalidate RFS and maybe rely on RPS, because it might help your workload.

Just to be clear : I tested the patch and saw a regression in my tests, sending as little as one million UDP packets per second on the target.

Not only UDP rx processing was slower, but TCP flows were impacted.

With a table of 65536 slots, each slot was written 16 times per second in average.

Google kernels have RFS_Hit/FRS_Miss snmp counters to catch this kind of problems. Maybe I should upstream this part.

Add calls to sock_rps_record_flow for udp_sendmsg, udp_sendpage and udp_recvmsg. This enables RFS for connected UDP sockets.

Tested:

Signed-off-by Tom Herbert <[email protected]>

---
 net/ipv4/udp.c | 6 ++++++
 1 file changed, 6 insertions(+)

net/ipv4/udp.c +4 -0

index cd0db54..9a0d346 100644
@@ -881,6 +881,8 @@ int udp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 	struct sk_buff *skb;
 	struct ip_options_data opt_copy;
 
+	sock_rps_record_flow(sk);
+
 	if (len > 0xFFFF)
 		return -EMSGSIZE;
 

@@ -1113,6 +1115,8 @@ int udp_sendpage(struct sock *sk, struct page *page, int offset,
 	struct udp_sock *up = udp_sk(sk);
 	int ret;
 
+	sock_rps_record_flow(sk);
+
 	if (flags & MSG_SENDPAGE_NOTLAST)
 		flags |= MSG_MORE;