Skip to contentSkip to conversation

RDMA/smbdirect: introduce and use rdma_restrict_node_type()

RFC PATCH In review Stefan Metzmacher 4 participants 11 messages Jan 21 – Feb 3

Subsystems (4)
lore
Share & tools
lore

Stefan Metzmacher

author reviewer
#0 lore raw

cover RDMA/smbdirect: introduce and use rdma_restrict_node_type()

Hi,

for smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is marked as RFC as I want to get feedback if the rdma_restrict_node_type() function is acceptable for the RDMA layer. And because the current form of the smb patches are not tested, I only tested the rdma part with my branch the prepares IPPROTO_SMBDIRECT sockets.

I'm not sure if this would be acceptable for 6.19 in order to avoid the smb layer problems, if the RDMA layer change is only acceptable for 7.0 that's also fine.

This is based on the following fix applied:
smb: server: reset smb_direct_port = SMB_DIRECT_PORT_INFINIBAND on init
https://lore.kernel.org/linux-cifs/[email protected]/
It's not yet in Linus' tree, so if this gets ready
before it's merged we can squash it.

Stefan Metzmacher (3):
RDMA/core: introduce rdma_restrict_node_type()
smb: client: make use of rdma_restrict_node_type()
smb: server: make use of rdma_restrict_node_type()

 drivers/infiniband/core/cma.c      |  30 ++++++++
 drivers/infiniband/core/cma_priv.h |   1 +
 fs/smb/client/smbdirect.c          |  26 +++++++
 fs/smb/server/transport_rdma.c     | 108 +++++++++++++++++++++--------
 include/rdma/rdma_cm.h             |  17 +++++
 5 files changed, 154 insertions(+), 28 deletions(-)
Signature · 1 line
2.43.0
author reviewer
#1 lore raw

RFC PATCH 1/3 RDMA/core: introduce rdma_restrict_node_type()

For smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is designed to be called before calling any of rdma_bind_addr(), rdma_resolve_addr() or rdma_listen().

---
 drivers/infiniband/core/cma.c      | 30 ++++++++++++++++++++++++++++++
 drivers/infiniband/core/cma_priv.h |  1 +
 include/rdma/rdma_cm.h             | 17 +++++++++++++++++
 3 files changed, 48 insertions(+)
drivers/infiniband/core/cma.c +30 -0
index f00f1d3fbd9c..0ee855a71ed4 100644
@@ -793,6 +793,9 @@ static int cma_acquire_dev_by_src_ip(struct rdma_id_private *id_priv)
 
 	mutex_lock(&lock);
 	list_for_each_entry(cma_dev, &dev_list, list) {
+		if (id_priv->restricted_node_type != RDMA_NODE_UNSPECIFIED &&
+		    id_priv->restricted_node_type != cma_dev->device->node_type)
+			continue;
 		rdma_for_each_port (cma_dev->device, port) {
 			gidp = rdma_protocol_roce(cma_dev->device, port) ?
 			       &iboe_gid : &gid;
@@ -1015,6 +1018,7 @@ __rdma_create_id(struct net *net, rdma_cm_event_handler event_handler,
 		return ERR_PTR(-ENOMEM);
 
 	id_priv->state = RDMA_CM_IDLE;
+	id_priv->restricted_node_type = RDMA_NODE_UNSPECIFIED;
 	id_priv->id.context = context;
 	id_priv->id.event_handler = event_handler;
 	id_priv->id.ps = ps;
@@ -4177,6 +4181,32 @@ int rdma_resolve_addr(struct rdma_cm_id *id, struct sockaddr *src_addr,
 }
 EXPORT_SYMBOL(rdma_resolve_addr);
 
+int rdma_restrict_node_type(struct rdma_cm_id *id, u8 node_type)
+{
+	struct rdma_id_private *id_priv =
+		container_of(id, struct rdma_id_private, id);
+	int ret = 0;
+
+	switch (node_type) {
+	case RDMA_NODE_UNSPECIFIED:
+	case RDMA_NODE_IB_CA:
+	case RDMA_NODE_RNIC:
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	mutex_lock(&lock);
+	if (id_priv->cma_dev)
+		ret = -EALREADY;
+	else
+		id_priv->restricted_node_type = node_type;
+	mutex_unlock(&lock);
+
+	return ret;
+}
+EXPORT_SYMBOL(rdma_restrict_node_type);
+
 int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr)
 {
 	struct rdma_id_private *id_priv =
drivers/infiniband/core/cma_priv.h +1 -0
index c604b601f4d9..04332eb82668 100644
@@ -72,6 +72,7 @@ struct rdma_id_private {
 
 	int			internal_id;
 	enum rdma_cm_state	state;
+	u8			restricted_node_type;
 	spinlock_t		lock;
 	struct mutex		qp_mutex;
 
include/rdma/rdma_cm.h +17 -0
index 9bd930a83e6e..6de6fd8bd15e 100644
@@ -168,6 +168,23 @@ struct rdma_cm_id *rdma_create_user_id(rdma_cm_event_handler event_handler,
   */
 void rdma_destroy_id(struct rdma_cm_id *id);
 
+/**
+ * rdma_restrict_node_type - Restrict an RDMA identifier to specific
+ *   RDMA device node type.
+ *
+ * @id: RDMA identifier.
+ * @node_type: The device node type. Only RDMA_NODE_UNSPECIFIED (default),
+ *   RDMA_NODE_RNIC and RDMA_NODE_IB_CA are allowed
+ *
+ * This allows the caller to restrict the possible devices
+ * used to iWarp (RDMA_NODE_RNIC) or InfiniBand/RoCEv1/RoCEv2 (RDMA_NODE_IB_CA).
+ *
+ * It needs to be called before the RDMA identifier is bound
+ * to an device, which mean it should be called before
+ * rdma_bind_addr(), rdma_bind_addr() and rdma_listen().
+ */
+int rdma_restrict_node_type(struct rdma_cm_id *id, u8 node_type);
+
 /**
  * rdma_bind_addr - Bind an RDMA identifier to a source address and
  *   associated RDMA device, if needed.
Signature · 1 line
2.43.0
Acked-by on v1 1/3
#2 lore raw

Re: [RFC PATCH 1/3] RDMA/core: introduce rdma_restrict_node_type()

Quoted 38 lines · Stefan Metzmacher

For smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is designed to be called before calling any of rdma_bind_addr(), rdma_resolve_addr() or rdma_listen().

Quoted 5 lines
---
 drivers/infiniband/core/cma.c      | 30 ++++++++++++++++++++++++++++++
 drivers/infiniband/core/cma_priv.h |  1 +
 include/rdma/rdma_cm.h             | 17 +++++++++++++++++
 3 files changed, 48 insertions(+)

Thanks,

author reviewer
#3 lore raw

RFC PATCH 2/3 smb: client: make use of rdma_restrict_node_type()

In reply to Stefan Metzmacher

For smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

---
 fs/smb/client/smbdirect.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
fs/smb/client/smbdirect.c +26 -0
index 2900b02f90cd..e69bce88cc9c 100644
@@ -903,6 +903,7 @@ static struct rdma_cm_id *smbd_create_id(
 {
 	struct smbdirect_socket_parameters *sp = &sc->parameters;
 	struct rdma_cm_id *id;
+	u8 node_type = RDMA_NODE_UNSPECIFIED;
 	int rc;
 	__be16 *sport;
 
@@ -914,6 +915,31 @@ static struct rdma_cm_id *smbd_create_id(
 		return id;
 	}
 
+	switch (port) {
+	case SMBD_PORT:
+		/*
+		 * only allow iWarp devices
+		 * for port 5445.
+		 */
+		node_type = RDMA_NODE_RNIC;
+		break;
+	case SMB_PORT:
+		/*
+		 * only allow InfiniBand, RoCEv1 or RoCEv2
+		 * devices for port 445.
+		 *
+		 * (Basically don't allow iWarp devices)
+		 */
+		node_type = RDMA_NODE_IB_CA;
+		break;
+	}
+	rc = rdma_restrict_node_type(id, node_type);
+	if (rc) {
+		log_rdma_event(ERR, "rdma_restrict_node_type(%u) failed %i\n",
+			       node_type, rc);
+		goto out;
+	}
+
 	if (dstaddr->sa_family == AF_INET6)
 		sport = &((struct sockaddr_in6 *)dstaddr)->sin6_port;
 	else
Signature · 1 line
2.43.0
author reviewer
#4 lore raw

RFC PATCH 3/3 smb: server: make use of rdma_restrict_node_type()

In reply to Stefan Metzmacher

For smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

Now we do a wildcard listen on port 5445 only for iWarp devices and another wildcard listen on port 445 of any InfiniBand, RoCEv1 and/or RoCEv2 devices.

The wildcard listeners also work if there is
no device of the requested node_type, this
is the same logic as we had before, but before
we had to decide between port 5445 or 445
and now both are possible at the same time.

---
 fs/smb/server/transport_rdma.c | 108 ++++++++++++++++++++++++---------
 1 file changed, 80 insertions(+), 28 deletions(-)
fs/smb/server/transport_rdma.c +80 -28
index 541e51a7c0ce..675194a24e36 100644
@@ -61,9 +61,6 @@
  * Those may change after a SMB_DIRECT negotiation
  */
 
-/* Set 445 port to SMB Direct port by default */
-static int smb_direct_port = SMB_DIRECT_PORT_INFINIBAND;
-
 /* The local peer's maximum number of credits to grant to the peer */
 static int smb_direct_receive_credit_max = 255;
 
@@ -90,8 +87,9 @@ struct smb_direct_device {
 };
 
 static struct smb_direct_listener {
+	int			port;
 	struct rdma_cm_id	*cm_id;
-} smb_direct_listener;
+} smb_direct_ib_listener, smb_direct_iw_listener;
 
 static struct workqueue_struct *smb_direct_wq;
 
@@ -2621,6 +2619,7 @@ static bool rdma_frwr_is_supported(struct ib_device_attr *attrs)
 static int smb_direct_handle_connect_request(struct rdma_cm_id *new_cm_id,
 					     struct rdma_cm_event *event)
 {
+	struct smb_direct_listener *listener = new_cm_id->context;
 	struct smb_direct_transport *t;
 	struct smbdirect_socket *sc;
 	struct smbdirect_socket_parameters *sp;
@@ -2709,7 +2708,7 @@ static int smb_direct_handle_connect_request(struct rdma_cm_id *new_cm_id,
 
 	handler = kthread_run(ksmbd_conn_handler_loop,
 			      KSMBD_TRANS(t)->conn, "ksmbd:r%u",
-			      smb_direct_port);
+			      listener->port);
 	if (IS_ERR(handler)) {
 		ret = PTR_ERR(handler);
 		pr_err("Can't start thread\n");
@@ -2746,39 +2745,73 @@ static int smb_direct_listen_handler(struct rdma_cm_id *cm_id,
 	return 0;
 }
 
-static int smb_direct_listen(int port)
+static int smb_direct_listen(struct smb_direct_listener *listener,
+			     int port)
 {
 	int ret;
 	struct rdma_cm_id *cm_id;
+	u8 node_type = RDMA_NODE_UNSPECIFIED;
 	struct sockaddr_in sin = {
 		.sin_family		= AF_INET,
 		.sin_addr.s_addr	= htonl(INADDR_ANY),
 		.sin_port		= htons(port),
 	};
 
+	switch (port) {
+	case SMB_DIRECT_PORT_IWARP:
+		/*
+		 * only allow iWarp devices
+		 * for port 5445.
+		 */
+		node_type = RDMA_NODE_RNIC;
+		break;
+	case SMB_DIRECT_PORT_INFINIBAND:
+		/*
+		 * only allow InfiniBand, RoCEv1 or RoCEv2
+		 * devices for port 445.
+		 *
+		 * (Basically don't allow iWarp devices)
+		 */
+		node_type = RDMA_NODE_IB_CA;
+		break;
+	default:
+		pr_err("unsupported smbdirect port=%d!\n", port);
+		return -ENODEV;
+	}
+
 	cm_id = rdma_create_id(&init_net, smb_direct_listen_handler,
-			       &smb_direct_listener, RDMA_PS_TCP, IB_QPT_RC);
+			       listener, RDMA_PS_TCP, IB_QPT_RC);
 	if (IS_ERR(cm_id)) {
 		pr_err("Can't create cm id: %ld\n", PTR_ERR(cm_id));
 		return PTR_ERR(cm_id);
 	}
 
+	ret = rdma_restrict_node_type(cm_id, node_type);
+	if (ret) {
+		pr_err("rdma_restrict_node_type(%u) failed %d\n",
+		       node_type, ret);
+		goto err;
+	}
+
 	ret = rdma_bind_addr(cm_id, (struct sockaddr *)&sin);
 	if (ret) {
 		pr_err("Can't bind: %d\n", ret);
 		goto err;
 	}
 
-	smb_direct_listener.cm_id = cm_id;
-
 	ret = rdma_listen(cm_id, 10);
 	if (ret) {
 		pr_err("Can't listen: %d\n", ret);
 		goto err;
 	}
+
+	listener->port = port;
+	listener->cm_id = cm_id;
+
 	return 0;
 err:
-	smb_direct_listener.cm_id = NULL;
+	listener->port = 0;
+	listener->cm_id = NULL;
 	rdma_destroy_id(cm_id);
 	return ret;
 }
@@ -2787,10 +2820,6 @@ static int smb_direct_ib_client_add(struct ib_device *ib_dev)
 {
 	struct smb_direct_device *smb_dev;
 
-	/* Set 5445 port if device type is iWARP(No IB) */
-	if (ib_dev->node_type != RDMA_NODE_IB_CA)
-		smb_direct_port = SMB_DIRECT_PORT_IWARP;
-
 	if (!rdma_frwr_is_supported(&ib_dev->attrs))
 		return 0;
 
@@ -2833,8 +2862,9 @@ int ksmbd_rdma_init(void)
 {
 	int ret;
 
-	smb_direct_port = SMB_DIRECT_PORT_INFINIBAND;
-	smb_direct_listener.cm_id = NULL;
+	smb_direct_ib_listener = smb_direct_iw_listener = (struct smb_direct_listener) {
+		.cm_id = NULL,
+	};
 
 	ret = ib_register_client(&smb_direct_ib_client);
 	if (ret) {
@@ -2850,31 +2880,53 @@ int ksmbd_rdma_init(void)
 	smb_direct_wq = alloc_workqueue("ksmbd-smb_direct-wq",
 					WQ_HIGHPRI | WQ_MEM_RECLAIM | WQ_PERCPU,
 					0);
-	if (!smb_direct_wq)
-		return -ENOMEM;
+	if (!smb_direct_wq) {
+		ret = -ENOMEM;
+		goto err;
+	}
 
-	ret = smb_direct_listen(smb_direct_port);
+	ret = smb_direct_listen(&smb_direct_ib_listener,
+				SMB_DIRECT_PORT_INFINIBAND);
 	if (ret) {
-		destroy_workqueue(smb_direct_wq);
-		smb_direct_wq = NULL;
-		pr_err("Can't listen: %d\n", ret);
-		return ret;
+		pr_err("Can't listen on InfiniBand/RoCEv1/RoCEv2: %d\n", ret);
+		goto err;
 	}
 
-	ksmbd_debug(RDMA, "init RDMA listener. cm_id=%p\n",
-		    smb_direct_listener.cm_id);
+	ksmbd_debug(RDMA, "InfiniBand/RoCEv1/RoCEv2 RDMA listener. cm_id=%p\n",
+		    smb_direct_ib_listener.cm_id);
+
+	ret = smb_direct_listen(&smb_direct_iw_listener,
+				SMB_DIRECT_PORT_IWARP);
+	if (ret) {
+		pr_err("Can't listen on iWarp: %d\n", ret);
+		goto err;
+	}
+
+	ksmbd_debug(RDMA, "iWarp RDMA listener. cm_id=%p\n",
+		    smb_direct_iw_listener.cm_id);
+
 	return 0;
+err:
+	ksmbd_rdma_stop_listening();
+	ksmbd_rdma_destroy();
+	return ret;
 }
 
 void ksmbd_rdma_stop_listening(void)
 {
-	if (!smb_direct_listener.cm_id)
+	if (!smb_direct_ib_listener.cm_id && !smb_direct_iw_listener.cm_id)
 		return;
 
 	ib_unregister_client(&smb_direct_ib_client);
-	rdma_destroy_id(smb_direct_listener.cm_id);
 
-	smb_direct_listener.cm_id = NULL;
+	if (smb_direct_ib_listener.cm_id)
+		rdma_destroy_id(smb_direct_ib_listener.cm_id);
+	if (smb_direct_iw_listener.cm_id)
+		rdma_destroy_id(smb_direct_iw_listener.cm_id);
+
+	smb_direct_ib_listener = smb_direct_iw_listener = (struct smb_direct_listener) {
+		.cm_id = NULL,
+	};
 }
 
 void ksmbd_rdma_destroy(void)
Signature · 1 line
2.43.0
#5 lore raw
In reply to Stefan Metzmacher
Quoted 45 lines · Stefan Metzmacher

Hi,

for smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is marked as RFC as I want to get feedback if the rdma_restrict_node_type() function is acceptable for the RDMA layer. And because the current form of the smb patches are not tested, I only tested the rdma part with my branch the prepares IPPROTO_SMBDIRECT sockets.

I'm not sure if this would be acceptable for 6.19 in order to avoid the smb layer problems, if the RDMA layer change is only acceptable for 7.0 that's also fine.

This is based on the following fix applied:
smb: server: reset smb_direct_port = SMB_DIRECT_PORT_INFINIBAND on init
https://lore.kernel.org/linux-cifs/[email protected]/
It's not yet in Linus' tree, so if this gets ready
before it's merged we can squash it.

Quoted 4 lines

Stefan Metzmacher (3):
RDMA/core: introduce rdma_restrict_node_type()
smb: client: make use of rdma_restrict_node_type()
smb: server: make use of rdma_restrict_node_type()

The approach looks reasonable. Do you want me to take it through RDMA tree?

Thanks

Quoted 11 lines
 drivers/infiniband/core/cma.c      |  30 ++++++++
 drivers/infiniband/core/cma_priv.h |   1 +
 fs/smb/client/smbdirect.c          |  26 +++++++
 fs/smb/server/transport_rdma.c     | 108 +++++++++++++++++++++--------
 include/rdma/rdma_cm.h             |  17 +++++
 5 files changed, 154 insertions(+), 28 deletions(-)
Signature · 1 line
2.43.0
author reviewer
#6 lore raw

Am 28.01.26 um 15:11 schrieb Leon Romanovsky:

Quoted 46 lines · Stefan Metzmacher
Quoted 45 lines

Hi,

for smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is marked as RFC as I want to get feedback if the rdma_restrict_node_type() function is acceptable for the RDMA layer. And because the current form of the smb patches are not tested, I only tested the rdma part with my branch the prepares IPPROTO_SMBDIRECT sockets.

I'm not sure if this would be acceptable for 6.19 in order to avoid the smb layer problems, if the RDMA layer change is only acceptable for 7.0 that's also fine.

This is based on the following fix applied:
smb: server: reset smb_direct_port = SMB_DIRECT_PORT_INFINIBAND on init
https://lore.kernel.org/linux-cifs/[email protected]/
It's not yet in Linus' tree, so if this gets ready
before it's merged we can squash it.

Quoted 6 lines
Quoted 4 lines

Stefan Metzmacher (3):
RDMA/core: introduce rdma_restrict_node_type()
smb: client: make use of rdma_restrict_node_type()
smb: server: make use of rdma_restrict_node_type()

The approach looks reasonable.

Thanks!

Quoted 2 lines

Do you want me to take it through RDMA
tree?

As I also have other smb patches on top changing/using it I guess it would be easier if Steve would take them.

Steve, Leon what do you think?

Thanks!
metze

#7 lore raw
Quoted 56 lines · Stefan Metzmacher

Am 28.01.26 um 15:11 schrieb Leon Romanovsky:

Quoted 46 lines · Stefan Metzmacher
Quoted 45 lines

Hi,

for smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is marked as RFC as I want to get feedback if the rdma_restrict_node_type() function is acceptable for the RDMA layer. And because the current form of the smb patches are not tested, I only tested the rdma part with my branch the prepares IPPROTO_SMBDIRECT sockets.

I'm not sure if this would be acceptable for 6.19 in order to avoid the smb layer problems, if the RDMA layer change is only acceptable for 7.0 that's also fine.

This is based on the following fix applied:
smb: server: reset smb_direct_port = SMB_DIRECT_PORT_INFINIBAND on init
https://lore.kernel.org/linux-cifs/[email protected]/
It's not yet in Linus' tree, so if this gets ready
before it's merged we can squash it.

Quoted 6 lines
Quoted 4 lines

Stefan Metzmacher (3):
RDMA/core: introduce rdma_restrict_node_type()
smb: client: make use of rdma_restrict_node_type()
smb: server: make use of rdma_restrict_node_type()

The approach looks reasonable.

Thanks!

Quoted 7 lines
Quoted 2 lines

Do you want me to take it through RDMA
tree?

As I also have other smb patches on top changing/using it I guess it would be easier if Steve would take them.

Steve, Leon what do you think?

I'm ok with that, let's me add my Acked-by on first patch.

Thanks

Quoted 3 lines

Thanks!
metze

#8 lore raw
In reply to Stefan Metzmacher
Quoted 57 lines · Stefan Metzmacher

Am 28.01.26 um 15:11 schrieb Leon Romanovsky:

Quoted 46 lines · Stefan Metzmacher
Quoted 45 lines

Hi,

for smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is marked as RFC as I want to get feedback if the rdma_restrict_node_type() function is acceptable for the RDMA layer. And because the current form of the smb patches are not tested, I only tested the rdma part with my branch the prepares IPPROTO_SMBDIRECT sockets.

I'm not sure if this would be acceptable for 6.19 in order to avoid the smb layer problems, if the RDMA layer change is only acceptable for 7.0 that's also fine.

This is based on the following fix applied:
smb: server: reset smb_direct_port = SMB_DIRECT_PORT_INFINIBAND on init
https://lore.kernel.org/linux-cifs/[email protected]/
It's not yet in Linus' tree, so if this gets ready
before it's merged we can squash it.

Quoted 6 lines
Quoted 4 lines

Stefan Metzmacher (3):
RDMA/core: introduce rdma_restrict_node_type()
smb: client: make use of rdma_restrict_node_type()
smb: server: make use of rdma_restrict_node_type()

The approach looks reasonable.

Thanks!

Quoted 7 lines
Quoted 2 lines

Do you want me to take it through RDMA
tree?

As I also have other smb patches on top changing/using it I guess it would be easier if Steve would take them.

Steve, Leon what do you think?

I am ok with taking it via the ksmbd tree (smb3-kernel ksmbd-for-next branch), unless it is practical to merge the RDMA changes through the RDMA tree in the first few days of the merge window and merge the ksmbd-for-next branch a few days later (which sounds potentially trickier)

Signature · 3 lines
Thanks,

Steve
#9 lore raw
Quoted 64 lines · Steve French
Quoted 63 lines · Stefan Metzmacher

Am 28.01.26 um 15:11 schrieb Leon Romanovsky:

Quoted 46 lines · Stefan Metzmacher
Quoted 45 lines

Hi,

for smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is marked as RFC as I want to get feedback if the rdma_restrict_node_type() function is acceptable for the RDMA layer. And because the current form of the smb patches are not tested, I only tested the rdma part with my branch the prepares IPPROTO_SMBDIRECT sockets.

I'm not sure if this would be acceptable for 6.19 in order to avoid the smb layer problems, if the RDMA layer change is only acceptable for 7.0 that's also fine.

This is based on the following fix applied:
smb: server: reset smb_direct_port = SMB_DIRECT_PORT_INFINIBAND on init
https://lore.kernel.org/linux-cifs/[email protected]/
It's not yet in Linus' tree, so if this gets ready
before it's merged we can squash it.

Quoted 6 lines
Quoted 4 lines

Stefan Metzmacher (3):
RDMA/core: introduce rdma_restrict_node_type()
smb: client: make use of rdma_restrict_node_type()
smb: server: make use of rdma_restrict_node_type()

The approach looks reasonable.

Thanks!

Quoted 2 lines

Do you want me to take it through RDMA
tree?

As I also have other smb patches on top changing/using it I guess it would be easier if Steve would take them.

Quoted 7 lines
Quoted 1 line

Steve, Leon what do you think?

I am ok with taking it via the ksmbd tree (smb3-kernel ksmbd-for-next branch), unless it is practical to merge the RDMA changes through the RDMA tree in the first few days of the merge window and merge the ksmbd-for-next branch a few days later (which sounds potentially trickier)

The latter is a common merge workflow used to avoid merge conflicts, but it is not needed here. There are no changes in the rdma-cm area for this cycle, so I do not expect any merge conflicts during the final days of the cycle.

Thanks

Quoted 7 lines
Signature · 4 lines
-- 
Thanks,

Steve
maintainer Acked-by on the v1 series
#10 lore raw
In reply to Stefan Metzmacher
Quoted 40 lines · Stefan Metzmacher

Hi,

for smbdirect it required to use different ports depending on the RDMA protocol. E.g. for iWarp 5445 is needed (as tcp port 445 already used by the raw tcp transport for SMB), while InfiniBand, RoCEv1 and RoCEv2 use port 445, as they use an independent port range (even for RoCEv2, which uses udp port 4791 itself).

Currently ksmbd is not able to function correctly at all if the system has iWarp (RDMA_NODE_RNIC) interface(s) and any InfiniBand, RoCEv1 and/or RoCEv2 interface(s) at the same time.

And cifs.ko uses 5445 with a fallback to 445, which means depending on the available interfaces, it tries 5445 in the RoCE range or may tries iWarp with 445 as a fallback. This leads to strange error messages and strange network captures.

To avoid these problems they will be able to
use rdma_restrict_node_type(RDMA_NODE_RNIC) before trying port 5445 and rdma_restrict_node_type(RDMA_NODE_IB_CA) before trying port 445. It means we'll get early -ENODEV early from rdma_resolve_addr() without any network traffic and timeouts.

This is marked as RFC as I want to get feedback if the rdma_restrict_node_type() function is acceptable for the RDMA layer. And because the current form of the smb patches are not tested, I only tested the rdma part with my branch the prepares IPPROTO_SMBDIRECT sockets.

I'm not sure if this would be acceptable for 6.19 in order to avoid the smb layer problems, if the RDMA layer change is only acceptable for 7.0 that's also fine.

Quoted 5 lines

This is based on the following fix applied:
smb: server: reset smb_direct_port = SMB_DIRECT_PORT_INFINIBAND on init
https://lore.kernel.org/linux-cifs/[email protected]/
It's not yet in Linus' tree, so if this gets ready
before it's merged we can squash it.

Thanks!

Quoted 16 lines

Stefan Metzmacher (3):
RDMA/core: introduce rdma_restrict_node_type()
smb: client: make use of rdma_restrict_node_type()
smb: server: make use of rdma_restrict_node_type()

 drivers/infiniband/core/cma.c      |  30 ++++++++
 drivers/infiniband/core/cma_priv.h |   1 +
 fs/smb/client/smbdirect.c          |  26 +++++++
 fs/smb/server/transport_rdma.c     | 108 +++++++++++++++++++++--------
 include/rdma/rdma_cm.h             |  17 +++++
 5 files changed, 154 insertions(+), 28 deletions(-)

--
2.43.0