RDMA/rxe: Fetch skb packets from ethernet layer
Subsystems (4)
PATCH 1/1 RDMA/rxe: Fetch skb packets from ethernet layer
In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.
Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.
This increases bandwidth and decreases latency.
--- drivers/infiniband/sw/rxe/rxe_net.c | 45 ++++++++++++++++++++++++++++++++++- 1 files changed, 44 insertions(+), 1 deletions(-)
drivers/infiniband/sw/rxe/rxe_net.c +44 -1
@@ -18,6 +18,7 @@#include "rxe_loc.h" static struct rxe_recv_sockets recv_sockets; +static struct net_device *g_ndev;struct device *rxe_dma_device (struct rxe_dev *rxe) {
@@ -113,7 +114,7 @@ static int rxe_udp_encap_recv(struct sock *sk, struct sk_buff *skb) } tnl_cfg.encap_type =1 ; - tnl_cfg.encap_rcv = rxe_udp_encap_recv; + tnl_cfg.encap_rcv =NULL ;/* Setup UDP tunnel */ setup_udp_tunnel_sock (net, sock, &tnl_cfg);
@@ -357,6 +358,38 @@ struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av,return rxe->ndev->name; } +static rx_handler_result_t rxe_handle_frame (struct sk_buff **pskb) +{ +struct sk_buff *skb = *pskb; +struct iphdr *iph; +struct udphdr *udph; + +if (unlikely (skb->pkt_type ==PACKET_LOOPBACK )) +return RX_HANDLER_PASS ; + +if (!is_valid_ether_addr (eth_hdr (skb)->h_source)) { +kfree (skb); +return RX_HANDLER_CONSUMED ; + } + +if (eth_hdr (skb)->h_proto !=cpu_to_be16 (ETH_P_IP )) +return RX_HANDLER_PASS ; + + iph =ip_hdr (skb); + +if (iph->protocol !=IPPROTO_UDP ) +return RX_HANDLER_PASS ; + + udph =udp_hdr (skb); + +if (udph->dest !=cpu_to_be16 (ROCE_V2_UDP_DPORT )) +return RX_HANDLER_PASS ; + +rxe_udp_encap_recv (NULL , skb); + +return RX_HANDLER_CONSUMED ; +} +int rxe_net_add (const char *ibdev_name,struct net_device *ndev) {int err;
@@ -367,6 +400,7 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)return -ENOMEM ; rxe->ndev = ndev; + g_ndev = ndev; err =rxe_add (rxe, ndev->mtu, ibdev_name);if (err) {
@@ -374,6 +408,12 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)return err; } +rtnl_lock (); + err =netdev_rx_handler_register (ndev, rxe_handle_frame, rxe); +rtnl_unlock (); +if (err) +return err; +return 0 ; }
@@ -498,6 +538,9 @@ static int rxe_net_ipv6_init(void)void rxe_net_exit (void ) { +rtnl_lock (); +netdev_rx_handler_unregister (g_ndev); +rtnl_unlock ();rxe_release_udp_tunnel (recv_sockets.sk6);rxe_release_udp_tunnel (recv_sockets.sk4);unregister_netdevice_notifier (&rxe_net_notifier);
Signature · 1 line
1.7.1
Jakub Kicinski
maintainerQuoted 10 lines · On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:
In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.
Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.
This increases bandwidth and decreases latency.
Signed-off-by Zhu Yanjun <[email protected]>
Nope, no stealing UDP packets with some random rx handlers.
The tunnel socket is a correct approach.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Zhu Yanjun
maintainerQuoted 21 lines · Zhu Yanjun
-------- Forwarded Message --------
Subject: Re: [PATCH 1/1] RDMA/rxe: Fetch skb packets from ethernet layer
Date: Sat, 7 Nov 2020 12:26:17 -0800
From: Jakub Kicinski <[email protected]>
To: Zhu Yanjun <[email protected]>
CC: [email protected], [email protected], [email protected], [email protected]On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:
In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.
Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.
Quoted 6 lines
This increases bandwidth and decreases latency.
Signed-off-by Zhu Yanjun <[email protected]>Nope, no stealing UDP packets with some random rx handlers.
Why? Is there any risks?
Zhu Yanjun
Quoted 2 lines
The tunnel socket is a correct approach.
Jakub Kicinski
maintainerQuoted 10 lines · On Sun, 8 Nov 2020 13:27:32 +0800 Zhu Yanjun wrote:
Quoted 9 lines · Zhu Yanjun
On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:
In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.
Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.
Quoted 8 lines
Quoted 6 lines
This increases bandwidth and decreases latency.
Signed-off-by Zhu Yanjun <[email protected]>Nope, no stealing UDP packets with some random rx handlers.
Why? Is there any risks?
Are there risks in layering violations? Yes.
For example - you do absolutely no protocol parsing, checksum validation, only support IPv4, etc.
Besides it also makes the code far less maintainable, rx_handler is a singleton, etc. etc.
Quoted 1 line
Quoted 1 line
The tunnel socket is a correct approach.
Zhu Yanjun
maintainerQuoted 17 lines · Jakub Kicinski
Quoted 15 lines · On Sun, 8 Nov 2020 13:27:32 +0800 Zhu Yanjun wrote:
Quoted 14 lines · Zhu Yanjun
On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:
In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.
Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.
This increases bandwidth and decreases latency.
Signed-off-by Zhu Yanjun <[email protected]>
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Nope, no stealing UDP packets with some random rx handlers.
Why? Is there any risks?
Are there risks in layering violations? Yes.
For example - you do absolutely no protocol parsing,
Protocol parsing is in rxe driver.
Quoted 1 line
checksum validation, only support IPv4, etc.
Since only ipv4 is supported in rxe, if ipv6 is supported in rxe, I will add ipv6.
Quoted 2 lines
Besides it also makes the code far less maintainable,
rx_handleris a
This rx_handler is also used in openvswitch and bridge.
Zhu Yanjun
Quoted 3 lines
singleton, etc. etc.
Quoted 1 line
Quoted 1 line
The tunnel socket is a correct approach.
Zhu Yanjun
maintainerQuoted 31 lines · Zhu Yanjun
Quoted 17 lines · Jakub Kicinski
Quoted 15 lines · On Sun, 8 Nov 2020 13:27:32 +0800 Zhu Yanjun wrote:
Quoted 14 lines · Zhu Yanjun
On Thu, 5 Nov 2020 19:12:01 +0800 Zhu Yanjun wrote:
In the original design, in rx, skb packet would pass ethernet layer and IP layer, eventually reach udp tunnel.
Now rxe fetches the skb packets from the ethernet layer directly. So this bypasses the IP and UDP layer. As such, the skb packets are sent to the upper protocals directly from the ethernet layer.
This increases bandwidth and decreases latency.
Signed-off-by Zhu Yanjun <[email protected]>Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Nope, no stealing UDP packets with some random rx handlers.
Why? Is there any risks?
Are there risks in layering violations? Yes.
For example - you do absolutely no protocol parsing,
Protocol parsing is in rxe driver.
Quoted 1 line
checksum validation, only support IPv4, etc.
Quoted 7 lines
Since only ipv4 is supported in rxe, if ipv6 is supported in rxe, I will add ipv6.
Quoted 2 lines
Besides it also makes the code far less maintainable,
rx_handleris aThis
rx_handleris also used in openvswitch and bridge.
in Vacation. I will reply as soon as I come back.
Zhu Yanjun
Quoted 6 lines
Zhu Yanjun
Quoted 3 lines
singleton, etc. etc.
Quoted 1 line
Quoted 1 line
The tunnel socket is a correct approach.