Skip to contentSkip to conversation

virtio-net: per-queue RPS config

PATCH Applied Willem de Bruijn 4 participants 6 messages Jan 18, 2019 – Jan 19, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 virtio-net: per-queue RPS config

From: Willem de Bruijn <[email protected]>

On multiqueue network devices, RPS maps are configured independently for each receive queue through /sys/class/net/$DEV/queues/rx-*.

On virtio-net currently all packets use the map from rx-0, because the real rx queue is not known at time of map lookup by get_rps_cpu.

Call skb_record_rx_queue in the driver rx path to make lookup work.

Recording the receive queue has ramifications beyond RPS, such as in sticky load balancing decisions for sockets (skb_tx_hash) and XPS.

---
 drivers/net/virtio_net.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/virtio_net.c +1 -0
index 0237250860467..236ba5d5fb4bb 100644
@@ -1035,6 +1035,7 @@ static void receive_buf(struct virtnet_info *vi, struct receive_queue *rq,
 		goto frame_err;
 	}
 
+	skb_record_rx_queue(skb, vq2rxq(rq->vq));
 	skb->protocol = eth_type_trans(skb, dev);
 	pr_debug("Receiving skb proto 0x%04x len %i type %i\n",
 		 ntohs(skb->protocol), skb->len, skb->pkt_type);
Signature · 1 line
2.20.1.321.g9e740568ce-goog
#1 lore raw
Quoted 8 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

On multiqueue network devices, RPS maps are configured independently for each receive queue through /sys/class/net/$DEV/queues/rx-*.

On virtio-net currently all packets use the map from rx-0, because the real rx queue is not known at time of map lookup by get_rps_cpu.

Quoted 7 lines

Call skb_record_rx_queue in the driver rx path to make lookup work.

Recording the receive queue has ramifications beyond RPS, such as in sticky load balancing decisions for sockets (skb_tx_hash) and XPS.

And any examples how to see the benefit of this?

Quoted 18 lines
---
 drivers/net/virtio_net.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/virtio_net.c +1 -0
index 0237250860467..236ba5d5fb4bb 100644
@@ -1035,6 +1035,7 @@ static void receive_buf(struct virtnet_info *vi, struct receive_queue *rq,
 		goto frame_err;
 	}
 
+	skb_record_rx_queue(skb, vq2rxq(rq->vq));
 	skb->protocol = eth_type_trans(skb, dev);
 	pr_debug("Receiving skb proto 0x%04x len %i type %i\n",
 		 ntohs(skb->protocol), skb->len, skb->pkt_type);
Signature · 1 line
2.20.1.321.g9e740568ce-goog
#2 lore raw
Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

On multiqueue network devices, RPS maps are configured independently for each receive queue through /sys/class/net/$DEV/queues/rx-*.

On virtio-net currently all packets use the map from rx-0, because the real rx queue is not known at time of map lookup by get_rps_cpu.

Call skb_record_rx_queue in the driver rx path to make lookup work.

Quoted 7 lines
Quoted 5 lines

Recording the receive queue has ramifications beyond RPS, such as in sticky load balancing decisions for sockets (skb_tx_hash) and XPS.

And any examples how to see the benefit of this?

When there are fewer queues than cpus and rps is used to spread load across all cpus, it can be preferable to setup disjoint sets, such that each cpu handling an rxq interrupt spreads to an exclusive set of neighbors instead of having all interrupt handling cores contend on all other cores' softnet_data.

More subtly, even if the policy is to spread uniformly, it can be preferable to set the RPS map to all cores except the core that handled the interrupt, as it already had to do some work in the initial receive path.

It is also simply expected behavior for network devices to be able to configure rxq rps maps individually, so the current silent fallback to rx0 is confusing, especially since rx-1/rps_cpus, .. rx-n/rps_cpus files do exist and can be configured.

maintainer Acked-by
#3 lore raw
Quoted 32 lines · Willem de Bruijn
Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

On multiqueue network devices, RPS maps are configured independently for each receive queue through /sys/class/net/$DEV/queues/rx-*.

On virtio-net currently all packets use the map from rx-0, because the real rx queue is not known at time of map lookup by get_rps_cpu.

Call skb_record_rx_queue in the driver rx path to make lookup work.

Quoted 7 lines
Quoted 5 lines

Recording the receive queue has ramifications beyond RPS, such as in sticky load balancing decisions for sockets (skb_tx_hash) and XPS.

And any examples how to see the benefit of this?

When there are fewer queues than cpus and rps is used to spread load across all cpus, it can be preferable to setup disjoint sets, such that each cpu handling an rxq interrupt spreads to an exclusive set of neighbors instead of having all interrupt handling cores contend on all other cores' softnet_data.

More subtly, even if the policy is to spread uniformly, it can be preferable to set the RPS map to all cores except the core that handled the interrupt, as it already had to do some work in the initial receive path.

Quoted 4 lines

It is also simply expected behavior for network devices to be able to configure rxq rps maps individually, so the current silent fallback to rx0 is confusing, especially since rx-1/rps_cpus, .. rx-n/rps_cpus files do exist and can be configured.

OK I think I got it.

Acked-by on v1 1/1
#4 lore raw
In reply to Willem de Bruijn
Quoted 24 lines · Willem de Bruijn

From: Willem de Bruijn <[email protected]>

On multiqueue network devices, RPS maps are configured independently for each receive queue through /sys/class/net/$DEV/queues/rx-*.

On virtio-net currently all packets use the map from rx-0, because the real rx queue is not known at time of map lookup by get_rps_cpu.

Call skb_record_rx_queue in the driver rx path to make lookup work.

Recording the receive queue has ramifications beyond RPS, such as in sticky load balancing decisions for sockets (skb_tx_hash) and XPS.

---
  drivers/net/virtio_net.c | 1 +
  1 file changed, 1 insertion(+)
drivers/net/virtio_net.c +0 -0
index 0237250860467..236ba5d5fb4bb 100644

drivers/net/virtio_net.c static void receive_buf(struct virtnet_info *vi, struct receive_queue *rq, · line 1041

Quoted 7 lines
drivers/net/virtio_net.c +1 -0
@@ -1035,6 +1035,7 @@ static void receive_buf(struct virtnet_info *vi, struct receive_queue *rq,
  		goto frame_err;
  	}
 
+	skb_record_rx_queue(skb, vq2rxq(rq->vq));
  	skb->protocol = eth_type_trans(skb, dev);
  	pr_debug("Receiving skb proto 0x%04x len %i type %i\n",
  		 ntohs(skb->protocol), skb->len, skb->pkt_type);
maintainer Applied
#5 lore raw
In reply to Willem de Bruijn

From: Willem de Bruijn <[email protected]>
Date: Thu, 17 Jan 2019 20:08:53 -0500

Quoted 8 lines

From: Willem de Bruijn <[email protected]>

On multiqueue network devices, RPS maps are configured independently for each receive queue through /sys/class/net/$DEV/queues/rx-*.

On virtio-net currently all packets use the map from rx-0, because the real rx queue is not known at time of map lookup by get_rps_cpu.

Quoted 7 lines

Call skb_record_rx_queue in the driver rx path to make lookup work.

Recording the receive queue has ramifications beyond RPS, such as in sticky load balancing decisions for sockets (skb_tx_hash) and XPS.

Applied, thanks Willem.