Skip to contentSkip to conversation

packet: packet fanout rollover during socket overload

PATCH In review Willem de Bruijn 3 participants 8 messages Mar 12, 2013 – Mar 14, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 packet: packet fanout rollover during socket overload

Minimize packet drop in a fanout group. If one socket is full, roll over packets to another from the group. The intended use is to maintain flow affinity during normal load using an rxhash or cpu fanout policy, while dispersing unexpected traffic storms that hit a single cpu, such as spoofed-source DoS flows. This mechanism breaks affinity for flows arriving at saturated sockets during those conditions.

The patch adds a fanout policy ROLLOVER that rotates between sockets, filling each socket before moving to the next. It also adds a fanout flag ROLLOVER. If passed along with any other fanout policy, the primary policy is applied until the chosen socket is full. Then, rollover selects another socket, to delay packet drop until the entire system is saturated.

Probing sockets is not free. Selecting the last used socket, as rollover does, is a greedy approach that maximizes chance of success, at the cost of extreme load imbalance. In practice, with sufficiently long queues to handle rate, sockets are drained in parallel and load balance looks uniform in top.

To avoid contention, scales counters with number of sockets and accesses them lockfree. Values are bounds checked to ensure correctness. An alternative would be to use atomic rr_cur.

Tested using an application with 9 threads pinned to CPUs, one socket per thread and sufficient busywork per packet operation to limits each thread to handling 32 Kpps. When sent 500 Kpps single UDP stream packets, a FANOUT_CPU setup processes 32 Kpps in total without this patch, 270 Kpps with the patch. Tested with read() and with a packet ring (V1).

---
 include/linux/if_packet.h |   2 +
 net/packet/af_packet.c    | 112 ++++++++++++++++++++++++++++++++++++----------
 2 files changed, 90 insertions(+), 24 deletions(-)
include/linux/if_packet.h +2 -0
index f379929..20a77cc 100644
@@ -54,6 +54,8 @@ struct sockaddr_ll {
 #define PACKET_FANOUT_HASH		0
 #define PACKET_FANOUT_LB		1
 #define PACKET_FANOUT_CPU		2
+#define PACKET_FANOUT_ROLLOVER		3
+#define PACKET_FANOUT_FLAG_ROLLOVER	0x1000
 #define PACKET_FANOUT_FLAG_DEFRAG	0x8000
 
 struct tpacket_stats {
net/packet/af_packet.c +88 -24

212 lines not on this page · raw message

Signature · 1 line
1.8.1.3
#1 lore raw

PATCH 1/1 packet: packet fanout rollover during socket overload

Minimize packet drop in a fanout group. If one socket is full, roll over packets to another from the group. The intended use is to maintain flow affinity during normal load using an rxhash or cpu fanout policy, while dispersing unexpected traffic storms that hit a single cpu, such as spoofed-source DoS flows. This mechanism breaks affinity for flows arriving at saturated sockets during those conditions.

The patch adds a fanout policy ROLLOVER that rotates between sockets, filling each socket before moving to the next. It also adds a fanout flag ROLLOVER. If passed along with any other fanout policy, the primary policy is applied until the chosen socket is full. Then, rollover selects another socket, to delay packet drop until the entire system is saturated.

Probing sockets is not free. Selecting the last used socket, as rollover does, is a greedy approach that maximizes chance of success, at the cost of extreme load imbalance. In practice, with sufficiently long queues to handle rate, sockets are drained in parallel and load balance looks uniform in top.

To avoid contention, scales counters with number of sockets and accesses them lockfree. Values are bounds checked to ensure correctness. An alternative would be to use atomic rr_cur.

Tested using an application with 9 threads pinned to CPUs, one socket per thread and sufficient busywork per packet operation to limits each thread to handling 32 Kpps. When sent 500 Kpps single UDP stream packets, a FANOUT_CPU setup processes 32 Kpps in total without this patch, 270 Kpps with the patch. Tested with read() and with a packet ring (V1).

---
 include/linux/if_packet.h |   2 +
 net/packet/af_packet.c    | 112 ++++++++++++++++++++++++++++++++++++----------
 2 files changed, 90 insertions(+), 24 deletions(-)
include/linux/if_packet.h +2 -0
index f379929..20a77cc 100644
@@ -54,6 +54,8 @@ struct sockaddr_ll {
 #define PACKET_FANOUT_HASH		0
 #define PACKET_FANOUT_LB		1
 #define PACKET_FANOUT_CPU		2
+#define PACKET_FANOUT_ROLLOVER		3
+#define PACKET_FANOUT_FLAG_ROLLOVER	0x1000
 #define PACKET_FANOUT_FLAG_DEFRAG	0x8000
 
 struct tpacket_stats {
net/packet/af_packet.c +88 -24

212 lines not on this page · raw message

Signature · 1 line
1.8.1.3
#2 lore raw
Quoted 40 lines · Willem de Bruijn

Minimize packet drop in a fanout group. If one socket is full, roll over packets to another from the group. The intended use is to maintain flow affinity during normal load using an rxhash or cpu fanout policy, while dispersing unexpected traffic storms that hit a single cpu, such as spoofed-source DoS flows. This mechanism breaks affinity for flows arriving at saturated sockets during those conditions.

The patch adds a fanout policy ROLLOVER that rotates between sockets, filling each socket before moving to the next. It also adds a fanout flag ROLLOVER. If passed along with any other fanout policy, the primary policy is applied until the chosen socket is full. Then, rollover selects another socket, to delay packet drop until the entire system is saturated.

Probing sockets is not free. Selecting the last used socket, as rollover does, is a greedy approach that maximizes chance of success, at the cost of extreme load imbalance. In practice, with sufficiently long queues to handle rate, sockets are drained in parallel and load balance looks uniform in top.

To avoid contention, scales counters with number of sockets and accesses them lockfree. Values are bounds checked to ensure correctness. An alternative would be to use atomic rr_cur.

Tested using an application with 9 threads pinned to CPUs, one socket per thread and sufficient busywork per packet operation to limits each thread to handling 32 Kpps. When sent 500 Kpps single UDP stream packets, a FANOUT_CPU setup processes 32 Kpps in total without this patch, 270 Kpps with the patch. Tested with read() and with a packet ring (V1).

---
 include/linux/if_packet.h |   2 +
 net/packet/af_packet.c    | 112 ++++++++++++++++++++++++++++++++++++----------
 2 files changed, 90 insertions(+), 24 deletions(-)
net/packet/af_packet.c +0 -1
-static struct sock *fanout_demux_cpu(struct packet_fanout *f, struct sk_buff *skb, unsigned int num)

net/packet/af_packet.c static struct sock *fanout_demux_lb(struct packet_fanout *f, struct sk_buff *skb · line 1249

Quoted 10 lines
net/packet/af_packet.c +6 -2
+static unsigned int fanout_demux_rollover(struct packet_fanout *f,
+					  struct sk_buff *skb,
+					  unsigned int idx, unsigned int skip,
+					  unsigned int num)
 {
-	unsigned int cpu = smp_processor_id();
+	unsigned int i, j;
 
-	return f->arr[cpu % num];
+	i = j = min(f->next[idx], (int) f->num_members - 1);

min_t(int, f->next[idx], f->num_members - 1);

BTW, num_members can be 0

You really should do

int members = ACCESS_ONCE(f->num_members) - 1;

if (members < 0)

	return idx;

and only use members in your loop.

Quoted 13 lines
net/packet/af_packet.c +13 -0
+	do {
+		if (i != skip && packet_rcv_has_room(pkt_sk(f->arr[i]), skb)) {
+			if (i != j)
+				f->next[idx] = i;
+			return i;
+		}
+		if (++i >= f->num_members)
+			i = 0;
+	} while (i != j && idx < f->num_members);
+
+	return idx;
+}
+
#3 lore raw
Quoted 57 lines · Eric Dumazet
Quoted 40 lines · Willem de Bruijn

Minimize packet drop in a fanout group. If one socket is full, roll over packets to another from the group. The intended use is to maintain flow affinity during normal load using an rxhash or cpu fanout policy, while dispersing unexpected traffic storms that hit a single cpu, such as spoofed-source DoS flows. This mechanism breaks affinity for flows arriving at saturated sockets during those conditions.

The patch adds a fanout policy ROLLOVER that rotates between sockets, filling each socket before moving to the next. It also adds a fanout flag ROLLOVER. If passed along with any other fanout policy, the primary policy is applied until the chosen socket is full. Then, rollover selects another socket, to delay packet drop until the entire system is saturated.

Probing sockets is not free. Selecting the last used socket, as rollover does, is a greedy approach that maximizes chance of success, at the cost of extreme load imbalance. In practice, with sufficiently long queues to handle rate, sockets are drained in parallel and load balance looks uniform in top.

To avoid contention, scales counters with number of sockets and accesses them lockfree. Values are bounds checked to ensure correctness. An alternative would be to use atomic rr_cur.

Tested using an application with 9 threads pinned to CPUs, one socket per thread and sufficient busywork per packet operation to limits each thread to handling 32 Kpps. When sent 500 Kpps single UDP stream packets, a FANOUT_CPU setup processes 32 Kpps in total without this patch, 270 Kpps with the patch. Tested with read() and with a packet ring (V1).

---
 include/linux/if_packet.h |   2 +
 net/packet/af_packet.c    | 112 ++++++++++++++++++++++++++++++++++++----------
 2 files changed, 90 insertions(+), 24 deletions(-)
net/packet/af_packet.c +0 -1
-static struct sock *fanout_demux_cpu(struct packet_fanout *f, struct sk_buff *skb, unsigned int num)

net/packet/af_packet.c static struct sock *fanout_demux_lb(struct packet_fanout *f, struct sk_buff *skb · line 1249

Quoted 10 lines
net/packet/af_packet.c +6 -2
+static unsigned int fanout_demux_rollover(struct packet_fanout *f,
+                                       struct sk_buff *skb,
+                                       unsigned int idx, unsigned int skip,
+                                       unsigned int num)
 {
-     unsigned int cpu = smp_processor_id();
+     unsigned int i, j;
 
-     return f->arr[cpu % num];
+     i = j = min(f->next[idx], (int) f->num_members - 1);

min_t(int, f->next[idx], f->num_members - 1);

BTW, num_members can be 0

Quoted 8 lines

You really should do

int members = ACCESS_ONCE(f->num_members) - 1;

if (members < 0)

        return idx;

and only use members in your loop.

Thanks for catching that. I'll revise as mentioned.

Quoted 14 lines
Quoted 4 lines
net/packet/af_packet.c +2 -0
+     do {
+             if (i != skip && packet_rcv_has_room(pkt_sk(f->arr[i]), skb)) {

net/packet/af_packet.c static struct sock *fanout_demux_lb(struct packet_fanout *f, struct sk_buff *skb · line 1261

Quoted 10 lines
net/packet/af_packet.c +10 -0
+                     if (i != j)
+                             f->next[idx] = i;
+                     return i;
+             }
+             if (++i >= f->num_members)
+                     i = 0;
+     } while (i != j && idx < f->num_members);
+
+     return idx;
+}

+

It is probably also better if the rollover flag always leaves some room in the socket chosen with f->next. This to ensure that that socket can still enqueue its own packets, avoiding sending it into rollover mode, too.

maintainer
#5 lore raw

From: David Miller <[email protected]>
Date: Wed, 13 Mar 2013 04:44:05 -0400 (EDT)

Quoted 2 lines

My kingdom for a proper signoff.

Nevermind I see you did this in the second copy you posted.

Please mention that such things have changed below the "---" seperator so that people can know instead of just guess.

#6 lore raw
Quoted 10 lines · David Miller

From: David Miller <[email protected]>
Date: Wed, 13 Mar 2013 04:44:05 -0400 (EDT)

Quoted 2 lines

My kingdom for a proper signoff.

Nevermind I see you did this in the second copy you posted.

Please mention that such things have changed below the "---" seperator so that people can know instead of just guess.

Will do. Apologies for the botched emails, including the html reply that the list thankfully rejected. I'll write some fanout documentation for the packet man page as penance.

#7 lore raw
Quoted 3 lines · Willem de Bruijn

Will do. Apologies for the botched emails, including the html reply that the list thankfully rejected. I'll write some fanout documentation for the packet man page as penance.

Sounds good ! Thanks Willem !