RDMA/siw: [re-]introduce module parameters and add MPA V1
Subsystems (2)
cover RDMA/siw: [re-]introduce module parameters and add MPA V1
While working on smbdirect cleanup I'm using siw.ko for testing against Windows using Chelsio T404-BT (only supporting MPA V1) and T520-BT (supporting MPA V2) cards.
Up to now I used old patches to add MPA V1 support to siw.ko and compiled against the running kernel each time. I don't want to do that forever, so I re-introduced module parameters in order to avoid patching and compiling.
For my testing I'm using something like this:
echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer
echo 1 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v1_eth0 type siw netdev eth0
echo 2 > /sys/module/siw/parameters/mpa_version
rdma link add siw_v2_eth1 type siw netdev eth1
The global parameters are copied at 'rdma link add' time and will stay as is for the lifetime of the specific device.
So I can testing against the T520-BT card using siw_v2_eth1 and against the T404-BT card using siw_v1_eth0.
It would be great if this can go upstream.
Stefan Metzmacher (8):
RDMA/siw: make mpa_version = MPA_REVISION_2 const RDMA/siw: remove unused loopback_enabled = true RDMA/siw: add and remember siw_device_options per device. RDMA/siw: make use of sdev->options.* and avoid globals RDMA/siw: combine global options into siw_default_options RDMA/siw: move rtr_type to siw_device_options
RDMA/siw: [re-]introduce module parameters to alter the behavior at
runtime
RDMA/siw: add support for MPA V1 and IRD/ORD negotiation based on
[MS-SMBD]
drivers/infiniband/sw/siw/iwarp.h | 8 + drivers/infiniband/sw/siw/siw.h | 21 +- drivers/infiniband/sw/siw/siw_cm.c | 268 +++++++++++++++++++++----- drivers/infiniband/sw/siw/siw_cm.h | 2 + drivers/infiniband/sw/siw/siw_main.c | 173 ++++++++++++++--- drivers/infiniband/sw/siw/siw_qp_tx.c | 3 +- drivers/infiniband/sw/siw/siw_verbs.c | 2 +- 7 files changed, 395 insertions(+), 82 deletions(-)
Signature · 1 line
2.34.1
PATCH 1/8 RDMA/siw: make mpa_version = MPA_REVISION_2 const
There's currently no way to change that value at runtime.
--- drivers/infiniband/sw/siw/siw.h | 2 +- drivers/infiniband/sw/siw/siw_cm.c | 6 ------ drivers/infiniband/sw/siw/siw_main.c | 2 +- 3 files changed, 2 insertions(+), 8 deletions(-)
drivers/infiniband/sw/siw/siw.h +1 -1
@@ -494,7 +494,7 @@ extern const bool loopback_enabled;extern const bool mpa_crc_required;extern const bool mpa_crc_strict;extern const bool siw_tcp_nagle; -extern u_char mpa_version; +extern const u_char mpa_version;extern const bool peer_to_peer;extern struct task_struct *siw_tx_thread[];
drivers/infiniband/sw/siw/siw_cm.c +0 -6
@@ -1454,12 +1454,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)* MPA Rev. according to module parameter 'mpa_version', Key 'Request'. */ cep->mpa.hdr.params.bits =0 ; -if (version >MPA_REVISION_2 ) { -pr_warn ("Setting MPA version to %u\n" ,MPA_REVISION_2 ); - version =MPA_REVISION_2 ; -/* Adjust also module parameter */ - mpa_version =MPA_REVISION_2 ; - }__mpa_rr_set_revision (&cep->mpa.hdr.params.bits, version);if (try_gso)
drivers/infiniband/sw/siw/siw_main.c +1 -1
@@ -51,7 +51,7 @@ const bool mpa_crc_strict;const bool siw_tcp_nagle;/* Select MPA version to be used during connection setup */ -u_char mpa_version =MPA_REVISION_2 ; +const u_char mpa_version =MPA_REVISION_2 ;/* Selects MPA P2P mode (additional handshake during connection * setup, if true.
Signature · 1 line
2.34.1
PATCH 2/8 RDMA/siw: remove unused loopback_enabled = true
In reply to Stefan MetzmacherDevices are created explicitly by the administrator using 'rdma link add siw_lo type siw netdev lo'.
--- drivers/infiniband/sw/siw/siw.h | 1 - drivers/infiniband/sw/siw/siw_main.c | 5 +---- 2 files changed, 1 insertion(+), 5 deletions(-)
drivers/infiniband/sw/siw/siw.h +0 -1
@@ -490,7 +490,6 @@ struct siw_user_mmap_entry {/* Global siw parameters. Currently set in siw_main.c */ extern const bool zcopy_tx;extern const bool try_gso; -extern const bool loopback_enabled;extern const bool mpa_crc_required;extern const bool mpa_crc_strict;extern const bool siw_tcp_nagle;
drivers/infiniband/sw/siw/siw_main.c +1 -4
@@ -38,8 +38,6 @@ const bool zcopy_tx = true;*/ const bool try_gso; -/* Attach siw also with loopback devices */ -const bool loopback_enabled = true;/* We try to negotiate CRC on, if true */ const bool mpa_crc_required;
@@ -94,8 +92,7 @@ static int siw_dev_qualified(struct net_device *netdev)* <linux/if_arp.h> for type identifiers. */ if (netdev->type ==ARPHRD_ETHER || netdev->type ==ARPHRD_IEEE802 || - netdev->type ==ARPHRD_NONE || - (netdev->type ==ARPHRD_LOOPBACK && loopback_enabled)) + netdev->type ==ARPHRD_NONE || netdev->type ==ARPHRD_LOOPBACK )return 1 ;return 0 ;
Signature · 1 line
2.34.1
PATCH 3/8 RDMA/siw: add and remember siw_device_options per device.
In reply to Stefan MetzmacherIn future there should be a way to control these values on a per device basis.
--- drivers/infiniband/sw/siw/siw.h | 11 +++++++++++ drivers/infiniband/sw/siw/siw_main.c | 8 ++++++++ 2 files changed, 19 insertions(+)
drivers/infiniband/sw/siw/siw.h +11 -0
@@ -70,8 +70,19 @@ struct siw_pd {struct ib_pd base_pd; }; +struct siw_device_options { +bool zcopy_tx; +bool try_gso; +bool crc_required; +bool crc_strict; +bool tcp_nagle; +bool peer_to_peer; +u8 mpa_version; +}; +struct siw_device {struct ib_device base_dev; +struct siw_device_options options;struct siw_dev_cap attrs;u32 vendor_part_id;
drivers/infiniband/sw/siw/siw_main.c +8 -0
@@ -324,6 +324,14 @@ static struct siw_device *siw_device_create(struct net_device *netdev)/* Disable TCP port mapping */ base_dev->iw_driver_flags =IW_F_NO_PORT_MAP ; + sdev->options.zcopy_tx = zcopy_tx; + sdev->options.try_gso = try_gso; + sdev->options.crc_required = mpa_crc_required; + sdev->options.crc_strict = mpa_crc_strict; + sdev->options.tcp_nagle = siw_tcp_nagle; + sdev->options.peer_to_peer = peer_to_peer; + sdev->options.mpa_version = mpa_version; + sdev->attrs.max_qp =SIW_MAX_QP ; sdev->attrs.max_qp_wr =SIW_MAX_QP_WR ; sdev->attrs.max_ord =SIW_MAX_ORD_QP ;
Signature · 1 line
2.34.1
PATCH 4/8 RDMA/siw: make use of sdev->options.* and avoid globals
In reply to Stefan Metzmacher--- drivers/infiniband/sw/siw/siw_cm.c | 42 +++++++++++++++++---------- drivers/infiniband/sw/siw/siw_qp_tx.c | 3 +- drivers/infiniband/sw/siw/siw_verbs.c | 2 +- 3 files changed, 29 insertions(+), 18 deletions(-)
drivers/infiniband/sw/siw/siw_cm.c +26 -16
@@ -601,6 +601,9 @@ static int siw_recv_mpa_rr(struct siw_cep *cep)*/ static int siw_proc_mpareq (struct siw_cep *cep) { +struct siw_device *sdev = cep->sdev; +const bool local_crc_required = sdev->options.crc_required; +const bool local_crc_strict = sdev->options.crc_strict;struct mpa_rr *req;int version, rv;u16 pd_len;
@@ -648,11 +651,11 @@ static int siw_proc_mpareq(struct siw_cep *cep)* connection with CRC if local CRC off enforced by * 'mpa_crc_strict' module parameter. */ -if (!mpa_crc_required && mpa_crc_strict) +if (!local_crc_required && local_crc_strict) goto reject_conn;/* Enable CRC if requested by module parameter */ -if (mpa_crc_required) +if (local_crc_required) req->params.bits |=MPA_RR_FLAG_CRC ; }if (cep->enhanced_rdma_conn_est) {
@@ -705,13 +708,13 @@ static int siw_proc_mpareq(struct siw_cep *cep) reject_conn:siw_dbg_cep (cep,"reject: crc %d:%d:%d, m %d:%d\n" , req->params.bits &MPA_RR_FLAG_CRC ?1 :0 , - mpa_crc_required, mpa_crc_strict, + local_crc_required, local_crc_strict, req->params.bits &MPA_RR_FLAG_MARKERS ?1 :0 ,0 ); req->params.bits &= ~MPA_RR_FLAG_MARKERS ; req->params.bits |=MPA_RR_FLAG_REJECT ; -if (!mpa_crc_required && mpa_crc_strict) +if (!local_crc_required && local_crc_strict) req->params.bits &= ~MPA_RR_FLAG_CRC ;if (pd_len)
@@ -729,6 +732,9 @@ static int siw_proc_mpareply(struct siw_cep *cep)struct siw_qp_attrs qp_attrs;enum siw_qp_attr_mask qp_attr_mask;struct siw_qp *qp = cep->qp; +struct siw_device *sdev = cep->sdev; +const bool local_crc_required = sdev->options.crc_required; +const bool local_crc_strict = sdev->options.crc_strict;struct mpa_rr *rep;int rv;u16 rep_ord;
@@ -762,17 +768,17 @@ static int siw_proc_mpareply(struct siw_cep *cep)return -ECONNRESET ; } -if (try_gso && rep->params.bits &MPA_RR_FLAG_GSO_EXP ) { +if (sdev->options.try_gso && rep->params.bits &MPA_RR_FLAG_GSO_EXP ) {siw_dbg_cep (cep,"peer allows GSO on TX\n" ); qp->tx_ctx.gso_seg_limit =0 ; }if ((rep->params.bits &MPA_RR_FLAG_MARKERS ) || - (mpa_crc_required && !(rep->params.bits &MPA_RR_FLAG_CRC )) || - (mpa_crc_strict && !mpa_crc_required && + (local_crc_required && !(rep->params.bits &MPA_RR_FLAG_CRC )) || + (local_crc_strict && !local_crc_required && (rep->params.bits &MPA_RR_FLAG_CRC ))) {siw_dbg_cep (cep,"reply unsupp: crc %d:%d:%d, m %d:%d\n" , rep->params.bits &MPA_RR_FLAG_CRC ?1 :0 , - mpa_crc_required, mpa_crc_strict, + local_crc_required, local_crc_strict, rep->params.bits &MPA_RR_FLAG_MARKERS ?1 :0 ,0 );siw_cm_upcall (cep,IW_CM_EVENT_CONNECT_REPLY , -ECONNREFUSED );
@@ -920,6 +926,7 @@ static int siw_proc_mpareply(struct siw_cep *cep)static void siw_accept_newconn (struct siw_cep *cep) {struct socket *s = cep->sock; +struct siw_device *sdev = cep->sdev;struct socket *new_s =NULL ;struct siw_cep *new_cep =NULL ;int rv =0 ;/* debug only. should disappear */
@@ -927,7 +934,7 @@ static void siw_accept_newconn(struct siw_cep *cep)if (cep->state !=SIW_EPSTATE_LISTENING ) goto error; - new_cep =siw_cep_alloc (cep->sdev); + new_cep =siw_cep_alloc (sdev);if (!new_cep) goto error;
@@ -960,7 +967,7 @@ static void siw_accept_newconn(struct siw_cep *cep)siw_cep_get (new_cep); new_s->sk->sk_user_data = new_cep; -if (siw_tcp_nagle == false) +if (sdev->options.tcp_nagle == false)tcp_sock_set_nodelay (new_s->sk); new_cep->state =SIW_EPSTATE_AWAIT_MPAREQ ;
@@ -1357,9 +1364,11 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)struct socket *s =NULL ;struct sockaddr *laddr = (struct sockaddr *)&id->local_addr, *raddr = (struct sockaddr *)&id->remote_addr; -bool p2p_mode = peer_to_peer, v4 = true; +bool p2p_mode = sdev->options.peer_to_peer; +bool v4 = true;u16 pd_len = params->private_data_len; -int version = mpa_version, rv; +int version = sdev->options.mpa_version; +int rv;if (pd_len >MPA_MAX_PRIVDATA )return -EINVAL ;
@@ -1405,7 +1414,7 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)siw_dbg_qp (qp,"kernel_bindconnect: error %d\n" , rv); goto error; } -if (siw_tcp_nagle == false) +if (sdev->options.tcp_nagle == false)tcp_sock_set_nodelay (s->sk); cep =siw_cep_alloc (sdev);if (!cep) {
@@ -1456,10 +1465,10 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) cep->mpa.hdr.params.bits =0 ;__mpa_rr_set_revision (&cep->mpa.hdr.params.bits, version); -if (try_gso) +if (sdev->options.try_gso) cep->mpa.hdr.params.bits |=MPA_RR_FLAG_GSO_EXP ; -if (mpa_crc_required) +if (sdev->options.crc_required) cep->mpa.hdr.params.bits |=MPA_RR_FLAG_CRC ;/*
@@ -1577,7 +1586,8 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params) goto error_unlock;siw_dbg_cep (cep,"[QP %d]\n" , params->qpn); -if (try_gso && cep->mpa.hdr.params.bits &MPA_RR_FLAG_GSO_EXP ) { +if (sdev->options.try_gso && + cep->mpa.hdr.params.bits &MPA_RR_FLAG_GSO_EXP ) {siw_dbg_cep (cep,"peer allows GSO on TX\n" ); qp->tx_ctx.gso_seg_limit =0 ; }
drivers/infiniband/sw/siw/siw_qp_tx.c +2 -1
@@ -794,6 +794,7 @@ static int siw_check_sgl_tx(struct ib_pd *pd, struct siw_wqe *wqe,static int siw_qp_sq_proc_tx (struct siw_qp *qp,struct siw_wqe *wqe) {struct siw_iwarp_tx *c_tx = &qp->tx_ctx; +struct siw_device *sdev = qp->sdev;struct socket *s = qp->attrs.sk;int rv =0 , burst_len = qp->tx_ctx.burst;enum rdmap_ecode ecode =RDMAP_ECODE_CATASTROPHIC_STREAM ;
@@ -868,7 +869,7 @@ static int siw_qp_sq_proc_tx(struct siw_qp *qp, struct siw_wqe *wqe)enum siw_opcode tx_type =tx_type (wqe);unsigned int msg_flags; -if (siw_sq_empty (qp) || !siw_tcp_nagle || burst_len ==1 ) +if (siw_sq_empty (qp) || !sdev->options.tcp_nagle || burst_len ==1 )/* * End current TCP segment, if SQ runs empty, * or siw_tcp_nagle is not set, or we bail out
drivers/infiniband/sw/siw/siw_verbs.c +1 -1
@@ -434,7 +434,7 @@ int siw_create_qp(struct ib_qp *ibqp, struct ib_qp_init_attr *attrs,/* Make those two tunables fixed for now. */ qp->tx_ctx.gso_seg_limit =1 ; - qp->tx_ctx.zcopy_tx = zcopy_tx; + qp->tx_ctx.zcopy_tx = sdev->options.zcopy_tx; qp->attrs.state =SIW_QP_STATE_IDLE ;
Signature · 1 line
2.34.1
PATCH 5/8 RDMA/siw: combine global options into siw_default_options
In reply to Stefan Metzmacher--- drivers/infiniband/sw/siw/siw.h | 8 ---- drivers/infiniband/sw/siw/siw_main.c | 61 ++++++++++++++++------------ 2 files changed, 34 insertions(+), 35 deletions(-)
drivers/infiniband/sw/siw/siw.h +0 -8
@@ -498,14 +498,6 @@ struct siw_user_mmap_entry {void *address; }; -/* Global siw parameters. Currently set in siw_main.c */ -extern const bool zcopy_tx; -extern const bool try_gso; -extern const bool mpa_crc_required; -extern const bool mpa_crc_strict; -extern const bool siw_tcp_nagle; -extern const u_char mpa_version; -extern const bool peer_to_peer;extern struct task_struct *siw_tx_thread[];extern struct iwarp_msg_info iwarp_pktinfo[RDMAP_TERMINATE +1 ];
drivers/infiniband/sw/siw/siw_main.c +34 -27
79 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 6/8 RDMA/siw: move rtr_type to siw_device_options
In reply to Stefan Metzmacher--- drivers/infiniband/sw/siw/siw.h | 1 + drivers/infiniband/sw/siw/siw_cm.c | 6 +----- drivers/infiniband/sw/siw/siw_main.c | 6 ++++++ 3 files changed, 8 insertions(+), 5 deletions(-)
drivers/infiniband/sw/siw/siw.h +1 -0
@@ -77,6 +77,7 @@ struct siw_device_options {bool crc_strict;bool tcp_nagle;bool peer_to_peer; +__be16 rtr_type;u8 mpa_version; };
drivers/infiniband/sw/siw/siw_cm.c +1 -5
@@ -25,11 +25,6 @@#include "siw.h" #include "siw_cm.h" -/* -* Set to any combination of -* MPA_V2_RDMA_NO_RTR, MPA_V2_RDMA_READ_RTR, MPA_V2_RDMA_WRITE_RTR -*/ -static __be16 rtr_type =MPA_V2_RDMA_READ_RTR |MPA_V2_RDMA_WRITE_RTR ;static const bool relaxed_ird_negotiation = true;static void siw_cm_llp_state_change (struct sock *s);
@@ -1365,6 +1360,7 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)struct sockaddr *laddr = (struct sockaddr *)&id->local_addr, *raddr = (struct sockaddr *)&id->remote_addr;bool p2p_mode = sdev->options.peer_to_peer; +__be16 rtr_type = sdev->options.rtr_type;bool v4 = true;u16 pd_len = params->private_data_len;int version = sdev->options.mpa_version;
drivers/infiniband/sw/siw/siw_main.c +6 -0
@@ -63,6 +63,12 @@ static const struct siw_device_options siw_default_options = {*/ .peer_to_peer = false, +/* +* Set to any combination of +* MPA_V2_RDMA_NO_RTR, MPA_V2_RDMA_READ_RTR, MPA_V2_RDMA_WRITE_RTR +*/ + .rtr_type = MPA_V2_RDMA_READ_RTR | MPA_V2_RDMA_WRITE_RTR, +/* * Select MPA version to be used during connection setup */
Signature · 1 line
2.34.1
PATCH 7/8 RDMA/siw: [re-]introduce module parameters to alter the behavior at runtime
In reply to Stefan MetzmacherDuring the out of tree development of the siw.ko driver it had module parameters to change the runtime behavior. This got lost in the process of bringing it upstream in order to reduce the complexity.
In order to interop with Chelsio iwarp cards (tested with T404-BT, T520-BT and T520-CR) running from Windows in order to support smbdirect it is needed to
change some parameters:
echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer
The T404-BT card also requires:
echo 1 > /sys/module/siw/parameters/mpa_version
(which will be supported in the next commit.
These only have an effect on following invocations of 'rdma link add siw_eth0 type siw netdev eth0', which will make per device copy of the global parameters.
That means that each invocation of an
'rdma link add SIWDEV type siw netdev NETDEV' can have its own set of values, which makes it easy to test different settings on different interfaces.
--- drivers/infiniband/sw/siw/siw_cm.c | 2 +- drivers/infiniband/sw/siw/siw_main.c | 111 ++++++++++++++++++++++++++- 2 files changed, 111 insertions(+), 2 deletions(-)
drivers/infiniband/sw/siw/siw_cm.c +1 -1
@@ -1471,7 +1471,7 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)* If MPA version == 2: * o Include ORD and IRD. * o Indicate peer-to-peer mode, if required by module -* parameter 'peer_to_peer'. +* parameter 'mpa_peer_to_peer'. */ if (version ==MPA_REVISION_2 ) { cep->enhanced_rdma_conn_est = true;
drivers/infiniband/sw/siw/siw_main.c +110 -1
123 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 8/8 RDMA/siw: add support for MPA V1 and IRD/ORD negotiation based on [MS-SMBD]
In reply to Stefan MetzmacherThis is needed in order to interop with the Chelsio T404-BT card it needs this:
echo 1 > /sys/module/siw/parameters/mpa_crc_required echo 1 > /sys/module/siw/parameters/mpa_crc_strict echo 0 > /sys/module/siw/parameters/mpa_rdma_write_rtr echo 1 > /sys/module/siw/parameters/mpa_peer_to_peer echo 1 > /sys/module/siw/parameters/mpa_version
before calling 'rdma link add siw_eth0 type siw netdev eth0'.
--- drivers/infiniband/sw/siw/iwarp.h | 8 + drivers/infiniband/sw/siw/siw_cm.c | 214 ++++++++++++++++++++++++--- drivers/infiniband/sw/siw/siw_cm.h | 2 + drivers/infiniband/sw/siw/siw_main.c | 2 +- 4 files changed, 204 insertions(+), 22 deletions(-)
drivers/infiniband/sw/siw/iwarp.h +8 -0
@@ -71,6 +71,14 @@ struct mpa_v2_data {__be16 ord; }; +/* +* [MS-SMBD] 6 Appendix A: RDMA Provider IRD/ORD Negotiation +*/ +struct mpa_v1_smbd_data { +__be32 ird; +__be32 ord; +}; +struct mpa_marker {__be16 rsvd;__be16 fpdu_hmd;/* FPDU header-marker distance (= MPA's FPDUPTR) */
drivers/infiniband/sw/siw/siw_cm.c +193 -21
360 lines not on this page · raw message
drivers/infiniband/sw/siw/siw_cm.h +2 -0
@@ -28,6 +28,8 @@ struct siw_mpa_info {struct mpa_rr hdr;/* peer mpa hdr in host byte order */ struct mpa_v2_data v2_ctrl;struct mpa_v2_data v2_ctrl_req; +struct mpa_v1_smbd_data v1_ctrl; +struct mpa_v1_smbd_data v1_ctrl_req;char *pdata;int bytes_rcvd; };
drivers/infiniband/sw/siw/siw_main.c +1 -1
@@ -169,7 +169,7 @@ static int siw_mpa_version_set(const char *val, const struct kernel_param *kp)switch (uval) {case MPA_REVISION_2 : -/* TODO case MPA_REVISION_1: */ +case MPA_REVISION_1 : siw_default_options.mpa_version = uval;return 0 ; }
Signature · 1 line
2.34.1
Quoted 43 lines · Stefan Metzmacher
While working on smbdirect cleanup I'm using siw.ko for testing against Windows using Chelsio T404-BT (only supporting MPA V1) and T520-BT (supporting MPA V2) cards.
Up to now I used old patches to add MPA V1 support to siw.ko and compiled against the running kernel each time. I don't want to do that forever, so I re-introduced module parameters in order to avoid patching and compiling.
For my testing I'm using something like this:
echo 1 >
/sys/module/siw/parameters/mpa_crc_requiredecho 1 >/sys/module/siw/parameters/mpa_crc_strictecho 0 >/sys/module/siw/parameters/mpa_rdma_write_rtrecho 1 >/sys/module/siw/parameters/mpa_peer_to_peerecho 1 >
/sys/module/siw/parameters/mpa_version
rdma link addsiw_v1_eth0type siw netdev eth0echo 2 >
/sys/module/siw/parameters/mpa_version
rdma link addsiw_v2_eth1type siw netdev eth1The global parameters are copied at 'rdma link add' time and will stay as is for the lifetime of the specific device.
So I can testing against the T520-BT card using
siw_v2_eth1and against the T404-BT card usingsiw_v1_eth0.It would be great if this can go upstream.
Stefan Metzmacher (8):
RDMA/siw: makempa_version=MPA_REVISION_2const RDMA/siw: remove unusedloopback_enabled= true RDMA/siw: add and remembersiw_device_optionsper device. RDMA/siw: make use of sdev->options.* and avoid globals RDMA/siw: combine global options intosiw_default_optionsRDMA/siw: movertr_typetosiw_device_options
RDMA/siw: [re-]introduce module parameters to alter the behavior at
runtime
RDMA/siw: add support for MPA V1 and IRD/ORD negotiation based on[MS-SMBD]
Quoted 9 lines
drivers/infiniband/sw/siw/iwarp.h | 8 + drivers/infiniband/sw/siw/siw.h | 21 +- drivers/infiniband/sw/siw/siw_cm.c | 268 +++++++++++++++++++++----- drivers/infiniband/sw/siw/siw_cm.h | 2 + drivers/infiniband/sw/siw/siw_main.c | 173 ++++++++++++++--- drivers/infiniband/sw/siw/siw_qp_tx.c | 3 +- drivers/infiniband/sw/siw/siw_verbs.c | 2 +- 7 files changed, 395 insertions(+), 82 deletions(-)
Without addressing the question of whether siw should implement MPAv1, I would like to request a user interface that is more friendly to containers and to automation (CI). I know there is plenty of precedent for "pull in the current setting of some random global module parameter", but I don't think that is up to today's most common deployment scenarios.
Surely rxe will also need to have a similar switch between RoCEv1 and RoCEv2? Could something be added in the "rdma" tool to make this a per- device protocol version setting?
Signature · 1 line
Chuck Lever
Re: [PATCH 2/8] RDMA/siw: remove unused loopback_enabled = true
In reply to Stefan MetzmacherQuoted 2 lines · Stefan Metzmacher
Devices are created explicitly by the administrator using 'rdma link add
siw_lotype siw netdev lo'.
In the file drivers/infiniband/core/addr.c:
496 static int rdma_set_src_addr_rcu(struct rdma_dev_addr *dev_addr,
497 unsigned int *ndev_flags,
498 const struct sockaddr *dst_in,
499 const struct dst_entry *dst)
500 {
501 struct net_device *ndev = READ_ONCE(dst->dev);
502
503 *ndev_flags = ndev->flags;
504 /* A physical device must be the RDMA device to use */
505 if (ndev->flags & IFF_LOOPBACK) {
506 /*
507 * RDMA (IB/RoCE, iWarp) doesn't run on lo interface or
508 * loopback IP address. So if route is resolved to loopback
509 * interface, translate that to a real ndev based on non
510 * loopback IP address.
511 */512 ndev = rdma_find_ndev_for_src_ip_rcu(dev_net(ndev), dst_in);
513 if (IS_ERR(ndev))
514 return -ENODEV;
515 }
516
517 return copy_src_l2_addr(dev_addr, dst_in, dst, ndev);
518 }
I am not sure whether the above comments are correct or not because you are creating a siw device on lo netdev.
Yanjun.Zhu
Quoted 44 lines
Cc Bernard Metzler <[email protected]>Signed-off-by Stefan Metzmacher <[email protected]>--- drivers/infiniband/sw/siw/siw.h | 1 - drivers/infiniband/sw/siw/siw_main.c | 5 +---- 2 files changed, 1 insertion(+), 5 deletions(-)drivers/infiniband/sw/siw/siw.h +0
-1@@ -490,7 +490,6 @@ struct siw_user_mmap_entry {/* Global siw parameters. Currently set in siw_main.c */ extern const bool zcopy_tx;extern const bool try_gso; -extern const bool loopback_enabled;extern const bool mpa_crc_required;extern const bool mpa_crc_strict;extern const bool siw_tcp_nagle;drivers/infiniband/sw/siw/siw_main.c +1
-4@@ -38,8 +38,6 @@ const bool zcopy_tx = true;*/ const bool try_gso; -/* Attach siw also with loopback devices */ -const bool loopback_enabled = true;/* We try to negotiate CRC on, if true */ const bool mpa_crc_required;@@ -94,8 +92,7 @@ static int siw_dev_qualified(struct net_device *netdev)* <linux/if_arp.h> for type identifiers. */ if (netdev->type ==ARPHRD_ETHER || netdev->type ==ARPHRD_IEEE802 || - netdev->type ==ARPHRD_NONE || - (netdev->type ==ARPHRD_LOOPBACK && loopback_enabled)) + netdev->type ==ARPHRD_NONE || netdev->type ==ARPHRD_LOOPBACK )return 1 ;return 0 ;
Quoted 11 lines
-----Original Message-----
From: Chuck Lever <[email protected]>
Sent: Thursday, July 3, 2025 9:17 PM
To: [email protected]
Cc: Bernard Metzler <[email protected]>; Stefan Metzmacher <[email protected]>
Subject: [EXTERNAL] Re: [PATCH 0/8] RDMA/siw: [re-]introduce module parameters and add MPA V1
Quoted 11 lines
Quoted 10 lines · Stefan Metzmacher
While working on smbdirect cleanup I'm using siw.ko for testing against Windows using Chelsio T404-BT (only supporting MPA V1) and T520-BT (supporting MPA V2) cards.
Up to now I used old patches to add MPA V1 support to siw.ko and compiled against the running kernel each time. I don't want to do that forever, so I re-introduced module parameters in order to avoid patching and compiling.
Using siw for testing upper layer RDMA protocols/applications and involving 'real' RDMA hardware at peer side is one of the major purposes siw was originally done for. So I appreciate it is being used that way and can follow the arguments.
However, we shall take care not to incorporate the simplification of a single upper layer test case like smbdirect into a kernel driver. Especially, I am really not convinced smbdirect packet header syntax to appear within siw. This is an upper layer protocol and its semantics are void for iWarp.
I agree with Chuck that exporting module parameters is an outdated solution. The proposed alternative - the addition of control mechanics to the 'rdma tool' sounds promising to me. We might invent wire protocol specific options for RoCE, iWarp, ... devices. Looking closer, however, for siw (and I think for rxe as well), options could be even set per endpoint. This would further improve the benefits of using those software RDMA drivers for testing.
I am not sure if it is a good idea to add QPs as configurable resources to the rdma tool, or if we might consider using struct ib_qp_attr{}, which has tons of parameters useless for iWarp, or extending rdma_cm (rdma_set_option()), or if we better forget about per-connection configuration.
What do others think?
Further comment after quickly looking through the patch:
Please no upper case characters in variable or function names (siw_cep_mpa_vX_ctrl_len or vX_ctrl, etc.) We have reserved capital letters for macros and constants; we live a spartan life. 😊
Many thanks,
Bernard.
Quoted 57 lines
Quoted 33 lines
For my testing I'm using something like this:
echo 1 >
/sys/module/siw/parameters/mpa_crc_requiredecho 1 >/sys/module/siw/parameters/mpa_crc_strictecho 0 >/sys/module/siw/parameters/mpa_rdma_write_rtrecho 1 >/sys/module/siw/parameters/mpa_peer_to_peerecho 1 >
/sys/module/siw/parameters/mpa_version
rdma link addsiw_v1_eth0type siw netdev eth0echo 2 >
/sys/module/siw/parameters/mpa_version
rdma link addsiw_v2_eth1type siw netdev eth1The global parameters are copied at 'rdma link add' time and will stay as is for the lifetime of the specific device.
So I can testing against the T520-BT card using
siw_v2_eth1and against the T404-BT card usingsiw_v1_eth0.It would be great if this can go upstream.
Stefan Metzmacher (8):
RDMA/siw: makempa_version=MPA_REVISION_2const RDMA/siw: remove unusedloopback_enabled= true RDMA/siw: add and remembersiw_device_optionsper device. RDMA/siw: make use of sdev->options.* and avoid globals RDMA/siw: combine global options intosiw_default_optionsRDMA/siw: movertr_typetosiw_device_options
RDMA/siw: [re-]introduce module parameters to alter the behavior at
runtime
RDMA/siw: add support for MPA V1 and IRD/ORD negotiation based on[MS-SMBD]Quoted 9 lines
drivers/infiniband/sw/siw/iwarp.h | 8 + drivers/infiniband/sw/siw/siw.h | 21 +- drivers/infiniband/sw/siw/siw_cm.c | 268 +++++++++++++++++++++----- drivers/infiniband/sw/siw/siw_cm.h | 2 + drivers/infiniband/sw/siw/siw_main.c | 173 ++++++++++++++--- drivers/infiniband/sw/siw/siw_qp_tx.c | 3 +- drivers/infiniband/sw/siw/siw_verbs.c | 2 +- 7 files changed, 395 insertions(+), 82 deletions(-)Without addressing the question of whether siw should implement MPAv1, I would like to request a user interface that is more friendly to containers and to automation (CI). I know there is plenty of precedent for "pull in the current setting of some random global module parameter", but I don't think that is up to today's most common deployment scenarios.
Surely rxe will also need to have a similar switch between RoCEv1 and RoCEv2? Could something be added in the "rdma" tool to make this a per- device protocol version setting?
--
Chuck Lever
Quoted 23 lines · Stefan Metzmacher
While working on smbdirect cleanup I'm using siw.ko for testing against Windows using Chelsio T404-BT (only supporting MPA V1) and T520-BT (supporting MPA V2) cards.
Up to now I used old patches to add MPA V1 support to siw.ko and compiled against the running kernel each time. I don't want to do that forever, so I re-introduced module parameters in order to avoid patching and compiling.
For my testing I'm using something like this:
echo 1 >
/sys/module/siw/parameters/mpa_crc_requiredecho 1 >/sys/module/siw/parameters/mpa_crc_strictecho 0 >/sys/module/siw/parameters/mpa_rdma_write_rtrecho 1 >/sys/module/siw/parameters/mpa_peer_to_peerecho 1 >
/sys/module/siw/parameters/mpa_version
rdma link addsiw_v1_eth0type siw netdev eth0echo 2 >
/sys/module/siw/parameters/mpa_version
rdma link addsiw_v2_eth1type siw netdev eth1
Quoted 7 lines
The global parameters are copied at 'rdma link add' time and will stay as is for the lifetime of the specific device.
So I can testing against the T520-BT card using
siw_v2_eth1and against the T404-BT card usingsiw_v1_eth0.It would be great if this can go upstream.
Sorry no, there is no benefit in adding complexity for such an old device, like Chelsio T404-BT.
Thanks