Skip to contentSkip to conversation

RDMA/siw: [re-]introduce module parameters and add MPA V1

PATCH In review Stefan Metzmacher 5 participants 13 messages Jul 3, 2025 – Jul 6, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover RDMA/siw: [re-]introduce module parameters and add MPA V1

While working on smbdirect cleanup I'm using siw.ko for testing against Windows using Chelsio T404-BT (only supporting MPA V1) and T520-BT (supporting MPA V2) cards.

Up to now I used old patches to add MPA V1 support to siw.ko and compiled against the running kernel each time. I don't want to do that forever, so I re-introduced module parameters in order to avoid patching and compiling.

For my testing I'm using something like this:

echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer

echo 1 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v1_eth0 type siw netdev eth0

echo 2 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v2_eth1 type siw netdev eth1

The global parameters are copied at 'rdma link add' time and will stay as is for the lifetime of the specific device.

So I can testing against the T520-BT card using siw_v2_eth1 and against the T404-BT card using siw_v1_eth0.

It would be great if this can go upstream.

Stefan Metzmacher (8):
RDMA/siw: make mpa_version = MPA_REVISION_2 const RDMA/siw: remove unused loopback_enabled = true RDMA/siw: add and remember siw_device_options per device. RDMA/siw: make use of sdev->options.* and avoid globals RDMA/siw: combine global options into siw_default_options RDMA/siw: move rtr_type to siw_device_options
RDMA/siw: [re-]introduce module parameters to alter the behavior at
runtime
RDMA/siw: add support for MPA V1 and IRD/ORD negotiation based on

    [MS-SMBD]
 drivers/infiniband/sw/siw/iwarp.h     |   8 +
 drivers/infiniband/sw/siw/siw.h       |  21 +-
 drivers/infiniband/sw/siw/siw_cm.c    | 268 +++++++++++++++++++++-----
 drivers/infiniband/sw/siw/siw_cm.h    |   2 +
 drivers/infiniband/sw/siw/siw_main.c  | 173 ++++++++++++++---
 drivers/infiniband/sw/siw/siw_qp_tx.c |   3 +-
 drivers/infiniband/sw/siw/siw_verbs.c |   2 +-
 7 files changed, 395 insertions(+), 82 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/8 RDMA/siw: make mpa_version = MPA_REVISION_2 const

There's currently no way to change that value at runtime.

---
 drivers/infiniband/sw/siw/siw.h      | 2 +-
 drivers/infiniband/sw/siw/siw_cm.c   | 6 ------
 drivers/infiniband/sw/siw/siw_main.c | 2 +-
 3 files changed, 2 insertions(+), 8 deletions(-)
drivers/infiniband/sw/siw/siw.h +1 -1
index f5fd71717b80..3e04357ab197 100644
@@ -494,7 +494,7 @@ extern const bool loopback_enabled;
 extern const bool mpa_crc_required;
 extern const bool mpa_crc_strict;
 extern const bool siw_tcp_nagle;
-extern u_char mpa_version;
+extern const u_char mpa_version;
 extern const bool peer_to_peer;
 extern struct task_struct *siw_tx_thread[];
 
drivers/infiniband/sw/siw/siw_cm.c +0 -6
index 708b13993fdf..aba97d674402 100644
@@ -1454,12 +1454,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 	 * MPA Rev. according to module parameter 'mpa_version', Key 'Request'.
 	 */
 	cep->mpa.hdr.params.bits = 0;
-	if (version > MPA_REVISION_2) {
-		pr_warn("Setting MPA version to %u\n", MPA_REVISION_2);
-		version = MPA_REVISION_2;
-		/* Adjust also module parameter */
-		mpa_version = MPA_REVISION_2;
-	}
 	__mpa_rr_set_revision(&cep->mpa.hdr.params.bits, version);
 
 	if (try_gso)
drivers/infiniband/sw/siw/siw_main.c +1 -1
index 5168307229a9..4e1d29832ac8 100644
@@ -51,7 +51,7 @@ const bool mpa_crc_strict;
 const bool siw_tcp_nagle;
 
 /* Select MPA version to be used during connection setup */
-u_char mpa_version = MPA_REVISION_2;
+const u_char mpa_version = MPA_REVISION_2;
 
 /* Selects MPA P2P mode (additional handshake during connection
  * setup, if true.
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/8 RDMA/siw: remove unused loopback_enabled = true

In reply to Stefan Metzmacher

Devices are created explicitly by the administrator using 'rdma link add siw_lo type siw netdev lo'.

---
 drivers/infiniband/sw/siw/siw.h      | 1 -
 drivers/infiniband/sw/siw/siw_main.c | 5 +----
 2 files changed, 1 insertion(+), 5 deletions(-)
drivers/infiniband/sw/siw/siw.h +0 -1
index 3e04357ab197..3bdc17eedbe7 100644
@@ -490,7 +490,6 @@ struct siw_user_mmap_entry {
 /* Global siw parameters. Currently set in siw_main.c */
 extern const bool zcopy_tx;
 extern const bool try_gso;
-extern const bool loopback_enabled;
 extern const bool mpa_crc_required;
 extern const bool mpa_crc_strict;
 extern const bool siw_tcp_nagle;
drivers/infiniband/sw/siw/siw_main.c +1 -4
index 4e1d29832ac8..ba238b0b43a3 100644
@@ -38,8 +38,6 @@ const bool zcopy_tx = true;
  */
 const bool try_gso;
 
-/* Attach siw also with loopback devices */
-const bool loopback_enabled = true;
 
 /* We try to negotiate CRC on, if true */
 const bool mpa_crc_required;
@@ -94,8 +92,7 @@ static int siw_dev_qualified(struct net_device *netdev)
 	 * <linux/if_arp.h> for type identifiers.
 	 */
 	if (netdev->type == ARPHRD_ETHER || netdev->type == ARPHRD_IEEE802 ||
-	    netdev->type == ARPHRD_NONE ||
-	    (netdev->type == ARPHRD_LOOPBACK && loopback_enabled))
+	    netdev->type == ARPHRD_NONE || netdev->type == ARPHRD_LOOPBACK)
 		return 1;
 
 	return 0;
Signature · 1 line
2.34.1
#4 lore raw

PATCH 3/8 RDMA/siw: add and remember siw_device_options per device.

In reply to Stefan Metzmacher

In future there should be a way to control these values on a per device basis.

---
 drivers/infiniband/sw/siw/siw.h      | 11 +++++++++++
 drivers/infiniband/sw/siw/siw_main.c |  8 ++++++++
 2 files changed, 19 insertions(+)
drivers/infiniband/sw/siw/siw.h +11 -0
index 3bdc17eedbe7..b38e78909c06 100644
@@ -70,8 +70,19 @@ struct siw_pd {
 	struct ib_pd base_pd;
 };
 
+struct siw_device_options {
+	bool zcopy_tx;
+	bool try_gso;
+	bool crc_required;
+	bool crc_strict;
+	bool tcp_nagle;
+	bool peer_to_peer;
+	u8 mpa_version;
+};
+
 struct siw_device {
 	struct ib_device base_dev;
+	struct siw_device_options options;
 	struct siw_dev_cap attrs;
 
 	u32 vendor_part_id;
drivers/infiniband/sw/siw/siw_main.c +8 -0
index ba238b0b43a3..bbec1442dfa1 100644
@@ -324,6 +324,14 @@ static struct siw_device *siw_device_create(struct net_device *netdev)
 	/* Disable TCP port mapping */
 	base_dev->iw_driver_flags = IW_F_NO_PORT_MAP;
 
+	sdev->options.zcopy_tx = zcopy_tx;
+	sdev->options.try_gso = try_gso;
+	sdev->options.crc_required = mpa_crc_required;
+	sdev->options.crc_strict = mpa_crc_strict;
+	sdev->options.tcp_nagle = siw_tcp_nagle;
+	sdev->options.peer_to_peer = peer_to_peer;
+	sdev->options.mpa_version = mpa_version;
+
 	sdev->attrs.max_qp = SIW_MAX_QP;
 	sdev->attrs.max_qp_wr = SIW_MAX_QP_WR;
 	sdev->attrs.max_ord = SIW_MAX_ORD_QP;
Signature · 1 line
2.34.1
#5 lore raw

PATCH 4/8 RDMA/siw: make use of sdev->options.* and avoid globals

In reply to Stefan Metzmacher
---
 drivers/infiniband/sw/siw/siw_cm.c    | 42 +++++++++++++++++----------
 drivers/infiniband/sw/siw/siw_qp_tx.c |  3 +-
 drivers/infiniband/sw/siw/siw_verbs.c |  2 +-
 3 files changed, 29 insertions(+), 18 deletions(-)
drivers/infiniband/sw/siw/siw_cm.c +26 -16
index aba97d674402..9640450e1f87 100644
@@ -601,6 +601,9 @@ static int siw_recv_mpa_rr(struct siw_cep *cep)
  */
 static int siw_proc_mpareq(struct siw_cep *cep)
 {
+	struct siw_device *sdev = cep->sdev;
+	const bool local_crc_required = sdev->options.crc_required;
+	const bool local_crc_strict = sdev->options.crc_strict;
 	struct mpa_rr *req;
 	int version, rv;
 	u16 pd_len;
@@ -648,11 +651,11 @@ static int siw_proc_mpareq(struct siw_cep *cep)
 		 * connection with CRC if local CRC off enforced by
 		 * 'mpa_crc_strict' module parameter.
 		 */
-		if (!mpa_crc_required && mpa_crc_strict)
+		if (!local_crc_required && local_crc_strict)
 			goto reject_conn;
 
 		/* Enable CRC if requested by module parameter */
-		if (mpa_crc_required)
+		if (local_crc_required)
 			req->params.bits |= MPA_RR_FLAG_CRC;
 	}
 	if (cep->enhanced_rdma_conn_est) {
@@ -705,13 +708,13 @@ static int siw_proc_mpareq(struct siw_cep *cep)
 reject_conn:
 	siw_dbg_cep(cep, "reject: crc %d:%d:%d, m %d:%d\n",
 		    req->params.bits & MPA_RR_FLAG_CRC ? 1 : 0,
-		    mpa_crc_required, mpa_crc_strict,
+		    local_crc_required, local_crc_strict,
 		    req->params.bits & MPA_RR_FLAG_MARKERS ? 1 : 0, 0);
 
 	req->params.bits &= ~MPA_RR_FLAG_MARKERS;
 	req->params.bits |= MPA_RR_FLAG_REJECT;
 
-	if (!mpa_crc_required && mpa_crc_strict)
+	if (!local_crc_required && local_crc_strict)
 		req->params.bits &= ~MPA_RR_FLAG_CRC;
 
 	if (pd_len)
@@ -729,6 +732,9 @@ static int siw_proc_mpareply(struct siw_cep *cep)
 	struct siw_qp_attrs qp_attrs;
 	enum siw_qp_attr_mask qp_attr_mask;
 	struct siw_qp *qp = cep->qp;
+	struct siw_device *sdev = cep->sdev;
+	const bool local_crc_required = sdev->options.crc_required;
+	const bool local_crc_strict = sdev->options.crc_strict;
 	struct mpa_rr *rep;
 	int rv;
 	u16 rep_ord;
@@ -762,17 +768,17 @@ static int siw_proc_mpareply(struct siw_cep *cep)
 
 		return -ECONNRESET;
 	}
-	if (try_gso && rep->params.bits & MPA_RR_FLAG_GSO_EXP) {
+	if (sdev->options.try_gso && rep->params.bits & MPA_RR_FLAG_GSO_EXP) {
 		siw_dbg_cep(cep, "peer allows GSO on TX\n");
 		qp->tx_ctx.gso_seg_limit = 0;
 	}
 	if ((rep->params.bits & MPA_RR_FLAG_MARKERS) ||
-	    (mpa_crc_required && !(rep->params.bits & MPA_RR_FLAG_CRC)) ||
-	    (mpa_crc_strict && !mpa_crc_required &&
+	    (local_crc_required && !(rep->params.bits & MPA_RR_FLAG_CRC)) ||
+	    (local_crc_strict && !local_crc_required &&
 	     (rep->params.bits & MPA_RR_FLAG_CRC))) {
 		siw_dbg_cep(cep, "reply unsupp: crc %d:%d:%d, m %d:%d\n",
 			    rep->params.bits & MPA_RR_FLAG_CRC ? 1 : 0,
-			    mpa_crc_required, mpa_crc_strict,
+			    local_crc_required, local_crc_strict,
 			    rep->params.bits & MPA_RR_FLAG_MARKERS ? 1 : 0, 0);
 
 		siw_cm_upcall(cep, IW_CM_EVENT_CONNECT_REPLY, -ECONNREFUSED);
@@ -920,6 +926,7 @@ static int siw_proc_mpareply(struct siw_cep *cep)
 static void siw_accept_newconn(struct siw_cep *cep)
 {
 	struct socket *s = cep->sock;
+	struct siw_device *sdev = cep->sdev;
 	struct socket *new_s = NULL;
 	struct siw_cep *new_cep = NULL;
 	int rv = 0; /* debug only. should disappear */
@@ -927,7 +934,7 @@ static void siw_accept_newconn(struct siw_cep *cep)
 	if (cep->state != SIW_EPSTATE_LISTENING)
 		goto error;
 
-	new_cep = siw_cep_alloc(cep->sdev);
+	new_cep = siw_cep_alloc(sdev);
 	if (!new_cep)
 		goto error;
 
@@ -960,7 +967,7 @@ static void siw_accept_newconn(struct siw_cep *cep)
 	siw_cep_get(new_cep);
 	new_s->sk->sk_user_data = new_cep;
 
-	if (siw_tcp_nagle == false)
+	if (sdev->options.tcp_nagle == false)
 		tcp_sock_set_nodelay(new_s->sk);
 	new_cep->state = SIW_EPSTATE_AWAIT_MPAREQ;
 
@@ -1357,9 +1364,11 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 	struct socket *s = NULL;
 	struct sockaddr *laddr = (struct sockaddr *)&id->local_addr,
 			*raddr = (struct sockaddr *)&id->remote_addr;
-	bool p2p_mode = peer_to_peer, v4 = true;
+	bool p2p_mode = sdev->options.peer_to_peer;
+	bool v4 = true;
 	u16 pd_len = params->private_data_len;
-	int version = mpa_version, rv;
+	int version = sdev->options.mpa_version;
+	int rv;
 
 	if (pd_len > MPA_MAX_PRIVDATA)
 		return -EINVAL;
@@ -1405,7 +1414,7 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 		siw_dbg_qp(qp, "kernel_bindconnect: error %d\n", rv);
 		goto error;
 	}
-	if (siw_tcp_nagle == false)
+	if (sdev->options.tcp_nagle == false)
 		tcp_sock_set_nodelay(s->sk);
 	cep = siw_cep_alloc(sdev);
 	if (!cep) {
@@ -1456,10 +1465,10 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 	cep->mpa.hdr.params.bits = 0;
 	__mpa_rr_set_revision(&cep->mpa.hdr.params.bits, version);
 
-	if (try_gso)
+	if (sdev->options.try_gso)
 		cep->mpa.hdr.params.bits |= MPA_RR_FLAG_GSO_EXP;
 
-	if (mpa_crc_required)
+	if (sdev->options.crc_required)
 		cep->mpa.hdr.params.bits |= MPA_RR_FLAG_CRC;
 
 	/*
@@ -1577,7 +1586,8 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 		goto error_unlock;
 	siw_dbg_cep(cep, "[QP %d]\n", params->qpn);
 
-	if (try_gso && cep->mpa.hdr.params.bits & MPA_RR_FLAG_GSO_EXP) {
+	if (sdev->options.try_gso &&
+	    cep->mpa.hdr.params.bits & MPA_RR_FLAG_GSO_EXP) {
 		siw_dbg_cep(cep, "peer allows GSO on TX\n");
 		qp->tx_ctx.gso_seg_limit = 0;
 	}
drivers/infiniband/sw/siw/siw_qp_tx.c +2 -1
index 3a08f57d2211..b12fc5ec656e 100644
@@ -794,6 +794,7 @@ static int siw_check_sgl_tx(struct ib_pd *pd, struct siw_wqe *wqe,
 static int siw_qp_sq_proc_tx(struct siw_qp *qp, struct siw_wqe *wqe)
 {
 	struct siw_iwarp_tx *c_tx = &qp->tx_ctx;
+	struct siw_device *sdev = qp->sdev;
 	struct socket *s = qp->attrs.sk;
 	int rv = 0, burst_len = qp->tx_ctx.burst;
 	enum rdmap_ecode ecode = RDMAP_ECODE_CATASTROPHIC_STREAM;
@@ -868,7 +869,7 @@ static int siw_qp_sq_proc_tx(struct siw_qp *qp, struct siw_wqe *wqe)
 		enum siw_opcode tx_type = tx_type(wqe);
 		unsigned int msg_flags;
 
-		if (siw_sq_empty(qp) || !siw_tcp_nagle || burst_len == 1)
+		if (siw_sq_empty(qp) || !sdev->options.tcp_nagle || burst_len == 1)
 			/*
 			 * End current TCP segment, if SQ runs empty,
 			 * or siw_tcp_nagle is not set, or we bail out
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
index 2b2a7b8e93b0..1ca85b10e807 100644
@@ -434,7 +434,7 @@ int siw_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *attrs,
 
 	/* Make those two tunables fixed for now. */
 	qp->tx_ctx.gso_seg_limit = 1;
-	qp->tx_ctx.zcopy_tx = zcopy_tx;
+	qp->tx_ctx.zcopy_tx = sdev->options.zcopy_tx;
 
 	qp->attrs.state = SIW_QP_STATE_IDLE;
 
Signature · 1 line
2.34.1
#6 lore raw

PATCH 5/8 RDMA/siw: combine global options into siw_default_options

In reply to Stefan Metzmacher
---
 drivers/infiniband/sw/siw/siw.h      |  8 ----
 drivers/infiniband/sw/siw/siw_main.c | 61 ++++++++++++++++------------
 2 files changed, 34 insertions(+), 35 deletions(-)
drivers/infiniband/sw/siw/siw.h +0 -8
index b38e78909c06..4d8e1f857099 100644
@@ -498,14 +498,6 @@ struct siw_user_mmap_entry {
 	void *address;
 };
 
-/* Global siw parameters. Currently set in siw_main.c */
-extern const bool zcopy_tx;
-extern const bool try_gso;
-extern const bool mpa_crc_required;
-extern const bool mpa_crc_strict;
-extern const bool siw_tcp_nagle;
-extern const u_char mpa_version;
-extern const bool peer_to_peer;
 extern struct task_struct *siw_tx_thread[];
 
 extern struct iwarp_msg_info iwarp_pktinfo[RDMAP_TERMINATE + 1];
drivers/infiniband/sw/siw/siw_main.c +34 -27

79 lines not on this page · raw message

Signature · 1 line
2.34.1
#7 lore raw

PATCH 6/8 RDMA/siw: move rtr_type to siw_device_options

In reply to Stefan Metzmacher
---
 drivers/infiniband/sw/siw/siw.h      | 1 +
 drivers/infiniband/sw/siw/siw_cm.c   | 6 +-----
 drivers/infiniband/sw/siw/siw_main.c | 6 ++++++
 3 files changed, 8 insertions(+), 5 deletions(-)
drivers/infiniband/sw/siw/siw.h +1 -0
index 4d8e1f857099..f42418f44c0d 100644
@@ -77,6 +77,7 @@ struct siw_device_options {
 	bool crc_strict;
 	bool tcp_nagle;
 	bool peer_to_peer;
+	__be16 rtr_type;
 	u8 mpa_version;
 };
 
drivers/infiniband/sw/siw/siw_cm.c +1 -5
index 9640450e1f87..c1c66ae1fa97 100644
@@ -25,11 +25,6 @@
 #include "siw.h"
 #include "siw_cm.h"
 
-/*
- * Set to any combination of
- * MPA_V2_RDMA_NO_RTR, MPA_V2_RDMA_READ_RTR, MPA_V2_RDMA_WRITE_RTR
- */
-static __be16 rtr_type = MPA_V2_RDMA_READ_RTR | MPA_V2_RDMA_WRITE_RTR;
 static const bool relaxed_ird_negotiation = true;
 
 static void siw_cm_llp_state_change(struct sock *s);
@@ -1365,6 +1360,7 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 	struct sockaddr *laddr = (struct sockaddr *)&id->local_addr,
 			*raddr = (struct sockaddr *)&id->remote_addr;
 	bool p2p_mode = sdev->options.peer_to_peer;
+	__be16 rtr_type = sdev->options.rtr_type;
 	bool v4 = true;
 	u16 pd_len = params->private_data_len;
 	int version = sdev->options.mpa_version;
drivers/infiniband/sw/siw/siw_main.c +6 -0
index 954990278256..ff121f14d5b7 100644
@@ -63,6 +63,12 @@ static const struct siw_device_options siw_default_options = {
 	 */
 	.peer_to_peer = false,
 
+	/*
+	 * Set to any combination of
+	 * MPA_V2_RDMA_NO_RTR, MPA_V2_RDMA_READ_RTR, MPA_V2_RDMA_WRITE_RTR
+	 */
+	.rtr_type = MPA_V2_RDMA_READ_RTR | MPA_V2_RDMA_WRITE_RTR,
+
 	/*
 	 * Select MPA version to be used during connection setup
 	 */
Signature · 1 line
2.34.1
#8 lore raw

PATCH 7/8 RDMA/siw: [re-]introduce module parameters to alter the behavior at runtime

In reply to Stefan Metzmacher

During the out of tree development of the siw.ko driver it had module parameters to change the runtime behavior. This got lost in the process of bringing it upstream in order to reduce the complexity.

In order to interop with Chelsio iwarp cards (tested with T404-BT, T520-BT and T520-CR) running from Windows in order to support smbdirect it is needed to
change some parameters:

echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer

The T404-BT card also requires:
echo 1 > /sys/module/siw/parameters/mpa_version
(which will be supported in the next commit.

These only have an effect on following invocations of 'rdma link add siw_eth0 type siw netdev eth0', which will make per device copy of the global parameters.

That means that each invocation of an
'rdma link add SIWDEV type siw netdev NETDEV' can have its own set of values, which makes it easy to test different settings on different interfaces.

---
 drivers/infiniband/sw/siw/siw_cm.c   |   2 +-
 drivers/infiniband/sw/siw/siw_main.c | 111 ++++++++++++++++++++++++++-
 2 files changed, 111 insertions(+), 2 deletions(-)
drivers/infiniband/sw/siw/siw_cm.c +1 -1
index c1c66ae1fa97..7feed4a02d58 100644
@@ -1471,7 +1471,7 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 	 * If MPA version == 2:
 	 * o Include ORD and IRD.
 	 * o Indicate peer-to-peer mode, if required by module
-	 *   parameter 'peer_to_peer'.
+	 *   parameter 'mpa_peer_to_peer'.
 	 */
 	if (version == MPA_REVISION_2) {
 		cep->enhanced_rdma_conn_est = true;
drivers/infiniband/sw/siw/siw_main.c +110 -1

123 lines not on this page · raw message

Signature · 1 line
2.34.1
#9 lore raw

PATCH 8/8 RDMA/siw: add support for MPA V1 and IRD/ORD negotiation based on [MS-SMBD]

In reply to Stefan Metzmacher

This is needed in order to interop with the Chelsio T404-BT card it needs this:

echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer echo 1 > /sys/module/siw/parameters/mpa_version

before calling 'rdma link add siw_eth0 type siw netdev eth0'.

---
 drivers/infiniband/sw/siw/iwarp.h    |   8 +
 drivers/infiniband/sw/siw/siw_cm.c   | 214 ++++++++++++++++++++++++---
 drivers/infiniband/sw/siw/siw_cm.h   |   2 +
 drivers/infiniband/sw/siw/siw_main.c |   2 +-
 4 files changed, 204 insertions(+), 22 deletions(-)
drivers/infiniband/sw/siw/iwarp.h +8 -0
index 8cf69309827d..b30d9289950d 100644
@@ -71,6 +71,14 @@ struct mpa_v2_data {
 	__be16 ord;
 };
 
+/*
+ * [MS-SMBD] 6 Appendix A: RDMA Provider IRD/ORD Negotiation
+ */
+struct mpa_v1_smbd_data {
+	__be32 ird;
+	__be32 ord;
+};
+
 struct mpa_marker {
 	__be16 rsvd;
 	__be16 fpdu_hmd; /* FPDU header-marker distance (= MPA's FPDUPTR) */
drivers/infiniband/sw/siw/siw_cm.c +193 -21

360 lines not on this page · raw message

drivers/infiniband/sw/siw/siw_cm.h +2 -0
index 7011c8a8ee7b..506de01a701b 100644
@@ -28,6 +28,8 @@ struct siw_mpa_info {
 	struct mpa_rr hdr; /* peer mpa hdr in host byte order */
 	struct mpa_v2_data v2_ctrl;
 	struct mpa_v2_data v2_ctrl_req;
+	struct mpa_v1_smbd_data v1_ctrl;
+	struct mpa_v1_smbd_data v1_ctrl_req;
 	char *pdata;
 	int bytes_rcvd;
 };
drivers/infiniband/sw/siw/siw_main.c +1 -1
index c7d1e44ec8fa..208a6366e9e9 100644
@@ -169,7 +169,7 @@ static int siw_mpa_version_set(const char *val, const struct kernel_param *kp)
 
 	switch (uval) {
 	case MPA_REVISION_2:
-	/* TODO case MPA_REVISION_1: */
+	case MPA_REVISION_1:
 		siw_default_options.mpa_version = uval;
 		return 0;
 	}
Signature · 1 line
2.34.1
#10 lore raw
In reply to Stefan Metzmacher
Quoted 43 lines · Stefan Metzmacher

While working on smbdirect cleanup I'm using siw.ko for testing against Windows using Chelsio T404-BT (only supporting MPA V1) and T520-BT (supporting MPA V2) cards.

Up to now I used old patches to add MPA V1 support to siw.ko and compiled against the running kernel each time. I don't want to do that forever, so I re-introduced module parameters in order to avoid patching and compiling.

For my testing I'm using something like this:

echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer

echo 1 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v1_eth0 type siw netdev eth0

echo 2 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v2_eth1 type siw netdev eth1

The global parameters are copied at 'rdma link add' time and will stay as is for the lifetime of the specific device.

So I can testing against the T520-BT card using siw_v2_eth1 and against the T404-BT card using siw_v1_eth0.

It would be great if this can go upstream.

Stefan Metzmacher (8):
RDMA/siw: make mpa_version = MPA_REVISION_2 const RDMA/siw: remove unused loopback_enabled = true RDMA/siw: add and remember siw_device_options per device. RDMA/siw: make use of sdev->options.* and avoid globals RDMA/siw: combine global options into siw_default_options RDMA/siw: move rtr_type to siw_device_options
RDMA/siw: [re-]introduce module parameters to alter the behavior at
runtime
RDMA/siw: add support for MPA V1 and IRD/ORD negotiation based on

    [MS-SMBD]
Quoted 9 lines
 drivers/infiniband/sw/siw/iwarp.h     |   8 +
 drivers/infiniband/sw/siw/siw.h       |  21 +-
 drivers/infiniband/sw/siw/siw_cm.c    | 268 +++++++++++++++++++++-----
 drivers/infiniband/sw/siw/siw_cm.h    |   2 +
 drivers/infiniband/sw/siw/siw_main.c  | 173 ++++++++++++++---
 drivers/infiniband/sw/siw/siw_qp_tx.c |   3 +-
 drivers/infiniband/sw/siw/siw_verbs.c |   2 +-
 7 files changed, 395 insertions(+), 82 deletions(-)

Without addressing the question of whether siw should implement MPAv1, I would like to request a user interface that is more friendly to containers and to automation (CI). I know there is plenty of precedent for "pull in the current setting of some random global module parameter", but I don't think that is up to today's most common deployment scenarios.

Surely rxe will also need to have a similar switch between RoCEv1 and RoCEv2? Could something be added in the "rdma" tool to make this a per- device protocol version setting?

Signature · 1 line
Chuck Lever
#3 lore raw

Re: [PATCH 2/8] RDMA/siw: remove unused loopback_enabled = true

In reply to Stefan Metzmacher
Quoted 2 lines · Stefan Metzmacher

Devices are created explicitly by the administrator using 'rdma link add siw_lo type siw netdev lo'.

In the file drivers/infiniband/core/addr.c:

496 static int rdma_set_src_addr_rcu(struct rdma_dev_addr *dev_addr,

497                                  unsigned int *ndev_flags,
498                                  const struct sockaddr *dst_in,
499                                  const struct dst_entry *dst)
500 {
501         struct net_device *ndev = READ_ONCE(dst->dev);
502
503         *ndev_flags = ndev->flags;
504         /* A physical device must be the RDMA device to use */
505         if (ndev->flags & IFF_LOOPBACK) {
506                 /*
507                  * RDMA (IB/RoCE, iWarp) doesn't run on lo interface or
508                  * loopback IP address. So if route is resolved to loopback
509                  * interface, translate that to a real ndev based on non
510                  * loopback IP address.
511                  */

512 ndev = rdma_find_ndev_for_src_ip_rcu(dev_net(ndev), dst_in);
513 if (IS_ERR(ndev))
514 return -ENODEV;
515 }
516
517 return copy_src_l2_addr(dev_addr, dst_in, dst, ndev);
518 }

I am not sure whether the above comments are correct or not because you are creating a siw device on lo netdev.

Yanjun.Zhu

Quoted 44 lines
---
  drivers/infiniband/sw/siw/siw.h      | 1 -
  drivers/infiniband/sw/siw/siw_main.c | 5 +----
  2 files changed, 1 insertion(+), 5 deletions(-)
drivers/infiniband/sw/siw/siw.h +0 -1
index 3e04357ab197..3bdc17eedbe7 100644
@@ -490,7 +490,6 @@ struct siw_user_mmap_entry {
  /* Global siw parameters. Currently set in siw_main.c */
  extern const bool zcopy_tx;
  extern const bool try_gso;
-extern const bool loopback_enabled;
  extern const bool mpa_crc_required;
  extern const bool mpa_crc_strict;
  extern const bool siw_tcp_nagle;
drivers/infiniband/sw/siw/siw_main.c +1 -4
index 4e1d29832ac8..ba238b0b43a3 100644
@@ -38,8 +38,6 @@ const bool zcopy_tx = true;
   */
  const bool try_gso;
 
-/* Attach siw also with loopback devices */
-const bool loopback_enabled = true;
 
  /* We try to negotiate CRC on, if true */
  const bool mpa_crc_required;
@@ -94,8 +92,7 @@ static int siw_dev_qualified(struct net_device *netdev)
  	 * <linux/if_arp.h> for type identifiers.
  	 */
  	if (netdev->type == ARPHRD_ETHER || netdev->type == ARPHRD_IEEE802 ||
-	    netdev->type == ARPHRD_NONE ||
-	    (netdev->type == ARPHRD_LOOPBACK && loopback_enabled))
+	    netdev->type == ARPHRD_NONE || netdev->type == ARPHRD_LOOPBACK)
  		return 1;
 
  	return 0;
#11 lore raw
In reply to Chuck Lever
Quoted 11 lines

-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Thursday, July 3, 2025 9:17 PM
To: [email protected]
Cc: Bernard Metzler <[email protected]>; Stefan Metzmacher <[email protected]>
Subject: [EXTERNAL] Re: [PATCH 0/8] RDMA/siw: [re-]introduce module parameters and add MPA V1

Quoted 11 lines
Quoted 10 lines · Stefan Metzmacher

While working on smbdirect cleanup I'm using siw.ko for testing against Windows using Chelsio T404-BT (only supporting MPA V1) and T520-BT (supporting MPA V2) cards.

Up to now I used old patches to add MPA V1 support to siw.ko and compiled against the running kernel each time. I don't want to do that forever, so I re-introduced module parameters in order to avoid patching and compiling.

Using siw for testing upper layer RDMA protocols/applications and involving 'real' RDMA hardware at peer side is one of the major purposes siw was originally done for. So I appreciate it is being used that way and can follow the arguments.

However, we shall take care not to incorporate the simplification of a single upper layer test case like smbdirect into a kernel driver. Especially, I am really not convinced smbdirect packet header syntax to appear within siw. This is an upper layer protocol and its semantics are void for iWarp.

I agree with Chuck that exporting module parameters is an outdated solution. The proposed alternative - the addition of control mechanics to the 'rdma tool' sounds promising to me. We might invent wire protocol specific options for RoCE, iWarp, ... devices. Looking closer, however, for siw (and I think for rxe as well), options could be even set per endpoint. This would further improve the benefits of using those software RDMA drivers for testing.
I am not sure if it is a good idea to add QPs as configurable resources to the rdma tool, or if we might consider using struct ib_qp_attr{}, which has tons of parameters useless for iWarp, or extending rdma_cm (rdma_set_option()), or if we better forget about per-connection configuration.

What do others think?

Further comment after quickly looking through the patch:
Please no upper case characters in variable or function names (siw_cep_mpa_vX_ctrl_len or vX_ctrl, etc.) We have reserved capital letters for macros and constants; we live a spartan life. 😊

Many thanks,
Bernard.

Quoted 57 lines
Quoted 33 lines

For my testing I'm using something like this:

echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer

echo 1 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v1_eth0 type siw netdev eth0

echo 2 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v2_eth1 type siw netdev eth1

The global parameters are copied at 'rdma link add' time and will stay as is for the lifetime of the specific device.

So I can testing against the T520-BT card using siw_v2_eth1 and against the T404-BT card using siw_v1_eth0.

It would be great if this can go upstream.

Stefan Metzmacher (8):
RDMA/siw: make mpa_version = MPA_REVISION_2 const RDMA/siw: remove unused loopback_enabled = true RDMA/siw: add and remember siw_device_options per device. RDMA/siw: make use of sdev->options.* and avoid globals RDMA/siw: combine global options into siw_default_options RDMA/siw: move rtr_type to siw_device_options
RDMA/siw: [re-]introduce module parameters to alter the behavior at
runtime
RDMA/siw: add support for MPA V1 and IRD/ORD negotiation based on

    [MS-SMBD]
Quoted 9 lines
 drivers/infiniband/sw/siw/iwarp.h     |   8 +
 drivers/infiniband/sw/siw/siw.h       |  21 +-
 drivers/infiniband/sw/siw/siw_cm.c    | 268 +++++++++++++++++++++-----
 drivers/infiniband/sw/siw/siw_cm.h    |   2 +
 drivers/infiniband/sw/siw/siw_main.c  | 173 ++++++++++++++---
 drivers/infiniband/sw/siw/siw_qp_tx.c |   3 +-
 drivers/infiniband/sw/siw/siw_verbs.c |   2 +-
 7 files changed, 395 insertions(+), 82 deletions(-)

Without addressing the question of whether siw should implement MPAv1, I would like to request a user interface that is more friendly to containers and to automation (CI). I know there is plenty of precedent for "pull in the current setting of some random global module parameter", but I don't think that is up to today's most common deployment scenarios.

Surely rxe will also need to have a similar switch between RoCEv1 and RoCEv2? Could something be added in the "rdma" tool to make this a per- device protocol version setting?

--
Chuck Lever

#12 lore raw
In reply to Stefan Metzmacher
Quoted 23 lines · Stefan Metzmacher

While working on smbdirect cleanup I'm using siw.ko for testing against Windows using Chelsio T404-BT (only supporting MPA V1) and T520-BT (supporting MPA V2) cards.

Up to now I used old patches to add MPA V1 support to siw.ko and compiled against the running kernel each time. I don't want to do that forever, so I re-introduced module parameters in order to avoid patching and compiling.

For my testing I'm using something like this:

echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer

echo 1 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v1_eth0 type siw netdev eth0

echo 2 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v2_eth1 type siw netdev eth1

Quoted 7 lines

The global parameters are copied at 'rdma link add' time and will stay as is for the lifetime of the specific device.

So I can testing against the T520-BT card using siw_v2_eth1 and against the T404-BT card using siw_v1_eth0.

It would be great if this can go upstream.

Sorry no, there is no benefit in adding complexity for such an old device, like Chelsio T404-BT.

Thanks