Skip to contentSkip to conversation

RDMA/rxe: Fetch skb packets from ethernet layer

PATCH In review Zhu Yanjun 3 participants 6 messages Nov 5, 2020 – Nov 11, 2020 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore

Zhu Yanjun

author
#0 lore raw

PATCH 1/1 RDMA/rxe: Fetch skb packets from ethernet layer

In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.

Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.

This increases bandwidth and decreases latency.

---
 drivers/infiniband/sw/rxe/rxe_net.c |   45 ++++++++++++++++++++++++++++++++++-
 1 files changed, 44 insertions(+), 1 deletions(-)
drivers/infiniband/sw/rxe/rxe_net.c +44 -1
index 2e490e5..8ea68b6 100644
@@ -18,6 +18,7 @@
 #include "rxe_loc.h"
 
 static struct rxe_recv_sockets recv_sockets;
+static struct net_device *g_ndev;
 
 struct device *rxe_dma_device(struct rxe_dev *rxe)
 {
@@ -113,7 +114,7 @@ static int rxe_udp_encap_recv(struct sock *sk, struct sk_buff *skb)
 	}
 
 	tnl_cfg.encap_type = 1;
-	tnl_cfg.encap_rcv = rxe_udp_encap_recv;
+	tnl_cfg.encap_rcv = NULL;
 
 	/* Setup UDP tunnel */
 	setup_udp_tunnel_sock(net, sock, &tnl_cfg);
@@ -357,6 +358,38 @@ struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av,
 	return rxe->ndev->name;
 }
 
+static rx_handler_result_t rxe_handle_frame(struct sk_buff **pskb)
+{
+	struct sk_buff *skb = *pskb;
+	struct iphdr *iph;
+	struct udphdr *udph;
+
+	if (unlikely(skb->pkt_type == PACKET_LOOPBACK))
+		return RX_HANDLER_PASS;
+
+	if (!is_valid_ether_addr(eth_hdr(skb)->h_source)) {
+		kfree(skb);
+		return RX_HANDLER_CONSUMED;
+	}
+
+	if (eth_hdr(skb)->h_proto != cpu_to_be16(ETH_P_IP))
+		return RX_HANDLER_PASS;
+
+	iph = ip_hdr(skb);
+
+	if (iph->protocol != IPPROTO_UDP)
+		return RX_HANDLER_PASS;
+
+	udph = udp_hdr(skb);
+
+	if (udph->dest != cpu_to_be16(ROCE_V2_UDP_DPORT))
+		return RX_HANDLER_PASS;
+
+	rxe_udp_encap_recv(NULL, skb);
+
+	return RX_HANDLER_CONSUMED;
+}
+
 int rxe_net_add(const char *ibdev_name, struct net_device *ndev)
 {
 	int err;
@@ -367,6 +400,7 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)
 		return -ENOMEM;
 
 	rxe->ndev = ndev;
+	g_ndev = ndev;
 
 	err = rxe_add(rxe, ndev->mtu, ibdev_name);
 	if (err) {
@@ -374,6 +408,12 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)
 		return err;
 	}
 
+	rtnl_lock();
+	err = netdev_rx_handler_register(ndev, rxe_handle_frame, rxe);
+	rtnl_unlock();
+	if (err)
+		return err;
+
 	return 0;
 }
 
@@ -498,6 +538,9 @@ static int rxe_net_ipv6_init(void)
 
 void rxe_net_exit(void)
 {
+	rtnl_lock();
+	netdev_rx_handler_unregister(g_ndev);
+	rtnl_unlock();
 	rxe_release_udp_tunnel(recv_sockets.sk6);
 	rxe_release_udp_tunnel(recv_sockets.sk4);
 	unregister_netdevice_notifier(&rxe_net_notifier);
Signature · 1 line
1.7.1
maintainer
#1 lore raw
Quoted 10 lines · On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:

In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.

Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.

This increases bandwidth and decreases latency.

Nope, no stealing UDP packets with some random rx handlers.

The tunnel socket is a correct approach.

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#3 lore raw
Quoted 21 lines · Zhu Yanjun

-------- Forwarded Message --------
Subject: Re: [PATCH 1/1] RDMA/rxe: Fetch skb packets from ethernet layer
Date: Sat, 7 Nov 2020 12:26:17 -0800
From: Jakub Kicinski <[email protected]>
To: Zhu Yanjun <[email protected]>
CC: [email protected], [email protected], [email protected], [email protected]

On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:

In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.

Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.

Quoted 6 lines

This increases bandwidth and decreases latency.

Nope, no stealing UDP packets with some random rx handlers.

Why? Is there any risks?

Zhu Yanjun

Quoted 2 lines

The tunnel socket is a correct approach.

maintainer
#4 lore raw
Quoted 10 lines · On Sun, 8 Nov 2020 13:27:32 +0800 Zhu Yanjun wrote:
Quoted 9 lines · Zhu Yanjun

On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:

In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.

Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.

Quoted 8 lines
Quoted 6 lines

This increases bandwidth and decreases latency.

Nope, no stealing UDP packets with some random rx handlers.

Why? Is there any risks?

Are there risks in layering violations? Yes.

For example - you do absolutely no protocol parsing, checksum validation, only support IPv4, etc.

Besides it also makes the code far less maintainable, rx_handler is a singleton, etc. etc.

Quoted 1 line
Quoted 1 line

The tunnel socket is a correct approach.

maintainer
#5 lore raw
Quoted 17 lines · Jakub Kicinski
Quoted 15 lines · On Sun, 8 Nov 2020 13:27:32 +0800 Zhu Yanjun wrote:
Quoted 14 lines · Zhu Yanjun

On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:

In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.

Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.

This increases bandwidth and decreases latency.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Nope, no stealing UDP packets with some random rx handlers.

Why? Is there any risks?

Are there risks in layering violations? Yes.

For example - you do absolutely no protocol parsing,

Protocol parsing is in rxe driver.

Quoted 1 line

checksum validation, only support IPv4, etc.

Since only ipv4 is supported in rxe, if ipv6 is supported in rxe, I will add ipv6.

Quoted 2 lines

Besides it also makes the code far less maintainable, rx_handler is a

This rx_handler is also used in openvswitch and bridge.

Zhu Yanjun

Quoted 3 lines

singleton, etc. etc.

Quoted 1 line
Quoted 1 line

The tunnel socket is a correct approach.

maintainer
#6 lore raw
Quoted 31 lines · Zhu Yanjun
Quoted 17 lines · Jakub Kicinski
Quoted 15 lines · On Sun, 8 Nov 2020 13:27:32 +0800 Zhu Yanjun wrote:
Quoted 14 lines · Zhu Yanjun

On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:

In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.

Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.

This increases bandwidth and decreases latency.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Nope, no stealing UDP packets with some random rx handlers.

Why? Is there any risks?

Are there risks in layering violations? Yes.

For example - you do absolutely no protocol parsing,

Protocol parsing is in rxe driver.

Quoted 1 line

checksum validation, only support IPv4, etc.

Quoted 7 lines

Since only ipv4 is supported in rxe, if ipv6 is supported in rxe, I will add ipv6.

Quoted 2 lines

Besides it also makes the code far less maintainable, rx_handler is a

This rx_handler is also used in openvswitch and bridge.

in Vacation. I will reply as soon as I come back.

Zhu Yanjun

Quoted 6 lines

Zhu Yanjun

Quoted 3 lines

singleton, etc. etc.

Quoted 1 line
Quoted 1 line

The tunnel socket is a correct approach.