Skip to contentSkip to conversation

RDMA/rxe: Cleanup rxe_init_packet()

PATCH In review Bob Pearson 2 participants 3 messages Jul 27, 2022 – Jul 28, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 RDMA/rxe: Cleanup rxe_init_packet()

Currently rxe_init_packet() recomputes ndev from the address vector but struct rxe_dev already holds a reference to ndev.

Cleanup rxe_init_packet to use the value of ndev in rxe and drop an unneeded parameter paylen since it is already carried in the packet info struct.

---
v2
Per Haris Iqbal remove redundant 'ack->paylen = paylen'

 drivers/infiniband/sw/rxe/rxe_loc.h  |  2 +-
 drivers/infiniband/sw/rxe/rxe_net.c  | 49 +++++++++++-----------------
 drivers/infiniband/sw/rxe/rxe_req.c  |  2 +-
 drivers/infiniband/sw/rxe/rxe_resp.c |  4 +--
 4 files changed, 23 insertions(+), 34 deletions(-)
drivers/infiniband/sw/rxe/rxe_loc.h +1 -1
index f9af30f582c6..7f98d296bc0d 100644
@@ -93,7 +93,7 @@ void rxe_mw_cleanup(struct rxe_pool_elem *elem);
 
 /* rxe_net.c */
 struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av,
-				int paylen, struct rxe_pkt_info *pkt);
+				struct rxe_pkt_info *pkt);
 int rxe_prepare(struct rxe_av *av, struct rxe_pkt_info *pkt,
 		struct sk_buff *skb);
 int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
drivers/infiniband/sw/rxe/rxe_net.c +19 -30

74 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_req.c +1 -1
index 49e8f54db6f5..e72db960d7ea 100644
@@ -397,7 +397,7 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp,
 	pkt->paylen = paylen;
 
 	/* init skb */
-	skb = rxe_init_packet(rxe, av, paylen, pkt);
+	skb = rxe_init_packet(rxe, av, pkt);
 	if (unlikely(!skb))
 		return NULL;
 
drivers/infiniband/sw/rxe/rxe_resp.c +2 -2
index b36ec5c4d5e0..19d0537278f6 100644
@@ -670,15 +670,15 @@ static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp,
 	 */
 	pad = (-payload) & 0x3;
 	paylen = rxe_opcode[opcode].length + payload + pad + RXE_ICRC_SIZE;
+	ack->paylen = paylen;
 
-	skb = rxe_init_packet(rxe, &qp->pri_av, paylen, ack);
+	skb = rxe_init_packet(rxe, &qp->pri_av, ack);
 	if (!skb)
 		return NULL;
 
 	ack->qp = qp;
 	ack->opcode = opcode;
 	ack->mask = rxe_opcode[opcode].mask;
-	ack->paylen = paylen;
 	ack->psn = psn;
 
 	bth_init(ack, opcode, 0, 0, pad, IB_DEFAULT_PKEY_FULL,
Signature · 1 line
2.34.1
maintainer
#1 lore raw

drivers/infiniband/sw/rxe/rxe_net.c int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, · line 457

Quoted 10 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_net.c +1 -6
 	unsigned int hdr_len;
 	struct sk_buff *skb = NULL;
-	struct net_device *ndev;
-	const struct ib_gid_attr *attr;
+	struct net_device *ndev = rxe->ndev;
 	const int port_num = 1;
-
-	attr = rdma_get_gid_attr(&rxe->ib_dev, port_num, av->grh.sgid_index);
-	if (IS_ERR(attr))
-		return NULL;

An ib_device can have many netdevs associated with the gid indexes, eg from VLANs or LAG. The core code creates these things

I think it is nonsense for rxe to work like this, and perhaps it doesn't work at all, but until rxe blocks creation of these other gid indexes I'm not sure it makes sense to delete this code..

Jason

#2 lore raw
Quoted 15 lines · Jason Gunthorpe

drivers/infiniband/sw/rxe/rxe_net.c int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, · line 457

Quoted 10 lines · Bob Pearson
drivers/infiniband/sw/rxe/rxe_net.c +1 -6
 	unsigned int hdr_len;
 	struct sk_buff *skb = NULL;
-	struct net_device *ndev;
-	const struct ib_gid_attr *attr;
+	struct net_device *ndev = rxe->ndev;
 	const int port_num = 1;
-
-	attr = rdma_get_gid_attr(&rxe->ib_dev, port_num, av->grh.sgid_index);
-	if (IS_ERR(attr))
-		return NULL;

An ib_device can have many netdevs associated with the gid indexes, eg from VLANs or LAG. The core code creates these things

Quoted 5 lines

I think it is nonsense for rxe to work like this, and perhaps it doesn't work at all, but until rxe blocks creation of these other gid indexes I'm not sure it makes sense to delete this code..

Jason

Somehow I had the vague impression that rxe didn't support vlans but I just looked at the following commit

commit fd49ddaf7e266b5892d659eb99d9f77841e5b4c0

Author: Mohammad Heib <[email protected]>

Date: Tue Aug 11 18:04:15 2020 +0300

RDMA/rxe: prevent rxe creation on top of vlan interface

Creating rxe device on top of vlan interface will create a non-functional

device that has an empty gids table and can't be used for rdma cm

communication.

This is caused by the logic in

enum_all_gids_of_dev_cb()/is_eth_port_of_netdev(), which only considers

networks connected to "upper devices" of the configured network device,

resulting in an empty set of gids for a vlan interface, and attempts to

connect via this rdma device fail in cm_init_av_for_response because no

gids can be resolved.

Apparently, this behavior was implemented to fit the HW-RoCE devices that

create RoCE device per port, therefore RXE must behave the same like

HW-RoCE devices and create rxe device per real device only.

In order to communicate via a vlan interface, the user must use the gid

index of the vlan address instead of creating rxe over vlan.

which jibes with what you are saying. The immediate impact of this is that rxe->ndev should probably not be used unless you know you want the physical device.

Bob