Skip to contentSkip to conversation

smb: client/smbdirect: connect bug fixes/cleanups and smbdirect_socket.status_wait

PATCH Superseded Stefan Metzmacher 3 participants 13 messages Aug 7, 2025 – Aug 8, 2025

Subsystems (3)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Stefan Metzmacher

author reviewer
#0 lore raw

cover smb: client/smbdirect: connect bug fixes/cleanups and smbdirect_socket.status_wait

Hi,

this is the next step towards a common smbdirect layer between cifs.ko and ksmbd.ko, with the aim to provide a socket layer for userspace usage at the end of the road.

This patchset focuses on the client side today.

The first one is a fix for very long timeouts against unreachable servers.

The others prepare the use of a single wait_queue for state changes. This removes a lot of special handling during the connect and negotiate phases.

The last two move the state_wait queue into the common smbdirect_socket.status_wait.

For the server I have only a single patch that also uses smbdirect_socket.status_wait, but I'm skipping the server patches today.

I plan a lot more progress on the server side tomorrow and hopefully finish the moving everything from struct smb_direct_transport into struct smbdirect_socket.

I used the following xfstests as regression tests:
cifs/001 generic/001 generic/002 generic/005 generic/006 generic/007 generic/010 generic/011

Between cifs.ko against ksmbd.ko via siw.ko.

This is on top of the patches for the client I posted yesterday...

Stefan Metzmacher (9):
smb: client: return an error if rdma_connect does not return within 5
seconds
smb: client: improve logging in smbd_conn_upcall()
smb: client: don't call init_waitqueue_head(&info->conn_wait) twice in
_smbd_get_connection
smb: client: only use a single wait_queue to monitor smbdirect
connection status
smb: client/smbdirect: replace SMBDIRECT_SOCKET_CONNECTING with more
detailed states
smb: client: use status_wait and SMBDIRECT_SOCKET_NEGOTIATE_RUNNING
for completion
smb: client: use status_wait and
SMBDIRECT_SOCKET_RESOLVE_{ADDR,ROUTE}_RUNNING for completion
smb: smbdirect: introduce smbdirect_socket.status_wait
smb: client: make use of smbdirect_socket.status_wait

 fs/smb/client/smbdirect.c                  | 137 ++++++++++++++-------
 fs/smb/client/smbdirect.h                  |   8 --
 fs/smb/common/smbdirect/smbdirect_socket.h |  15 ++-
 3 files changed, 105 insertions(+), 55 deletions(-)
Signature · 1 line
2.43.0
author reviewer
#1 lore raw

PATCH 1/9 smb: client: return an error if rdma_connect does not return within 5 seconds

This matches the timeout for tcp connections.

Cc Steve French <[email protected]>
Cc Tom Talpey <[email protected]>
Cc Long Li <[email protected]>
Fixes f198186aa9bb ("CIFS: SMBD: Establish SMB Direct connection")
---
 fs/smb/client/smbdirect.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
fs/smb/client/smbdirect.c +4 -2
index 6c2af00be44c..181349eda7a3 100644
@@ -1653,8 +1653,10 @@ static struct smbd_connection *_smbd_get_connection(
 		goto rdma_connect_failed;
 	}
 
-	wait_event_interruptible(
-		info->conn_wait, sc->status != SMBDIRECT_SOCKET_CONNECTING);
+	wait_event_interruptible_timeout(
+		info->conn_wait,
+		sc->status != SMBDIRECT_SOCKET_CONNECTING,
+		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
 	if (sc->status != SMBDIRECT_SOCKET_CONNECTED) {
 		log_rdma_event(ERR, "rdma_connect failed port=%d\n", port);
Signature · 1 line
2.43.0
author reviewer
#2 lore raw

PATCH 2/9 smb: client: improve logging in smbd_conn_upcall()

In reply to Stefan Metzmacher
---
 fs/smb/client/smbdirect.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
fs/smb/client/smbdirect.c +10 -4
index 181349eda7a3..8ed4ab6f1d3a 100644
@@ -178,9 +178,10 @@ static int smbd_conn_upcall(
 {
 	struct smbd_connection *info = id->context;
 	struct smbdirect_socket *sc = &info->socket;
+	const char *event_name = rdma_event_msg(event->event);
 
-	log_rdma_event(INFO, "event=%d status=%d\n",
-		event->event, event->status);
+	log_rdma_event(INFO, "event=%s status=%d\n",
+		event_name, event->status);
 
 	switch (event->event) {
 	case RDMA_CM_EVENT_ADDR_RESOLVED:
@@ -190,17 +191,19 @@ static int smbd_conn_upcall(
 		break;
 
 	case RDMA_CM_EVENT_ADDR_ERROR:
+		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		info->ri_rc = -EHOSTUNREACH;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_ERROR:
+		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		info->ri_rc = -ENETUNREACH;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
-		log_rdma_event(INFO, "connected event=%d\n", event->event);
+		log_rdma_event(INFO, "connected event=%s\n", event_name);
 		sc->status = SMBDIRECT_SOCKET_CONNECTED;
 		wake_up_interruptible(&info->conn_wait);
 		break;
@@ -208,7 +211,7 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_CONNECT_ERROR:
 	case RDMA_CM_EVENT_UNREACHABLE:
 	case RDMA_CM_EVENT_REJECTED:
-		log_rdma_event(INFO, "connecting failed event=%d\n", event->event);
+		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
 		wake_up_interruptible(&info->conn_wait);
 		break;
@@ -217,6 +220,7 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_DISCONNECTED:
 		/* This happens when we fail the negotiation */
 		if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) {
+			log_rdma_event(ERR, "event=%s during negotiation\n", event_name);
 			sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
 			wake_up(&info->conn_wait);
 			break;
@@ -229,6 +233,8 @@ static int smbd_conn_upcall(
 		break;
 
 	default:
+		log_rdma_event(ERR, "unexpected event=%s status=%d\n",
+			       event_name, event->status);
 		break;
 	}
 
Signature · 1 line
2.43.0
author reviewer
#3 lore raw

PATCH 3/9 smb: client: don't call init_waitqueue_head(&info->conn_wait) twice in _smbd_get_connection

In reply to Stefan Metzmacher

It is already called long before we may hit this cleanup code path.

---
 fs/smb/client/smbdirect.c | 1 -
 1 file changed, 1 deletion(-)
fs/smb/client/smbdirect.c +0 -1
index 8ed4ab6f1d3a..c819cc6dcc4f 100644
@@ -1716,7 +1716,6 @@ static struct smbd_connection *_smbd_get_connection(
 	cancel_delayed_work_sync(&info->idle_timer_work);
 	destroy_caches_and_workqueue(info);
 	sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
-	init_waitqueue_head(&info->conn_wait);
 	rdma_disconnect(sc->rdma.cm_id);
 	wait_event(info->conn_wait,
 		sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
Signature · 1 line
2.43.0
author reviewer
#4 lore raw

PATCH 4/9 smb: client: only use a single wait_queue to monitor smbdirect connection status

In reply to Stefan Metzmacher

There's no need for separate conn_wait and disconn_wait queues.

This will simplify the move to common code, the server code already a single wait_queue for this.

---
 fs/smb/client/smbdirect.c | 17 ++++++++---------
 fs/smb/client/smbdirect.h |  3 +--
 2 files changed, 9 insertions(+), 11 deletions(-)
fs/smb/client/smbdirect.c +8 -9
index c819cc6dcc4f..c628e91c328b 100644
@@ -205,7 +205,7 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_ESTABLISHED:
 		log_rdma_event(INFO, "connected event=%s\n", event_name);
 		sc->status = SMBDIRECT_SOCKET_CONNECTED;
-		wake_up_interruptible(&info->conn_wait);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_CONNECT_ERROR:
@@ -213,7 +213,7 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_REJECTED:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-		wake_up_interruptible(&info->conn_wait);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_DEVICE_REMOVAL:
@@ -222,12 +222,12 @@ static int smbd_conn_upcall(
 		if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) {
 			log_rdma_event(ERR, "event=%s during negotiation\n", event_name);
 			sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-			wake_up(&info->conn_wait);
+			wake_up(&info->status_wait);
 			break;
 		}
 
 		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-		wake_up_interruptible(&info->disconn_wait);
+		wake_up_interruptible(&info->status_wait);
 		wake_up_interruptible(&sc->recv_io.reassembly.wait_queue);
 		wake_up_interruptible_all(&info->wait_send_queue);
 		break;
@@ -1325,7 +1325,7 @@ void smbd_destroy(struct TCP_Server_Info *server)
 		rdma_disconnect(sc->rdma.cm_id);
 		log_rdma_event(INFO, "wait for transport being disconnected\n");
 		wait_event_interruptible(
-			info->disconn_wait,
+			info->status_wait,
 			sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 	}
 
@@ -1650,8 +1650,7 @@ static struct smbd_connection *_smbd_get_connection(
 	log_rdma_event(INFO, "connecting to IP %pI4 port %d\n",
 		&addr_in->sin_addr, port);
 
-	init_waitqueue_head(&info->conn_wait);
-	init_waitqueue_head(&info->disconn_wait);
+	init_waitqueue_head(&info->status_wait);
 	init_waitqueue_head(&sc->recv_io.reassembly.wait_queue);
 	rc = rdma_connect(sc->rdma.cm_id, &conn_param);
 	if (rc) {
@@ -1660,7 +1659,7 @@ static struct smbd_connection *_smbd_get_connection(
 	}
 
 	wait_event_interruptible_timeout(
-		info->conn_wait,
+		info->status_wait,
 		sc->status != SMBDIRECT_SOCKET_CONNECTING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
@@ -1717,7 +1716,7 @@ static struct smbd_connection *_smbd_get_connection(
 	destroy_caches_and_workqueue(info);
 	sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
 	rdma_disconnect(sc->rdma.cm_id);
-	wait_event(info->conn_wait,
+	wait_event(info->status_wait,
 		sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 
 allocate_cache_failed:
fs/smb/client/smbdirect.h +1 -2
index 0d4d45428c85..e45aa9ddd71d 100644
@@ -47,8 +47,7 @@ struct smbd_connection {
 
 	int ri_rc;
 	struct completion ri_done;
-	wait_queue_head_t conn_wait;
-	wait_queue_head_t disconn_wait;
+	wait_queue_head_t status_wait;
 
 	struct completion negotiate_completion;
 	bool negotiate_done;
Signature · 1 line
2.43.0
author reviewer
#5 lore raw

PATCH 5/9 smb: client/smbdirect: replace SMBDIRECT_SOCKET_CONNECTING with more detailed states

In reply to Stefan Metzmacher

The process of reaching a functional connection regresented by SMBDIRECT_SOCKET_CONNECTED, is more complex than using a single SMBDIRECT_SOCKET_CONNECTING state.

This will allow us to remove a lot of special variables and completions in the following commits.

---
 fs/smb/client/smbdirect.c                  | 44 +++++++++++++++++++---
 fs/smb/common/smbdirect/smbdirect_socket.h | 14 ++++++-
 2 files changed, 51 insertions(+), 7 deletions(-)
fs/smb/client/smbdirect.c +39 -5
index c628e91c328b..f95ecc153249 100644
@@ -185,26 +185,39 @@ static int smbd_conn_upcall(
 
 	switch (event->event) {
 	case RDMA_CM_EVENT_ADDR_RESOLVED:
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED;
+		info->ri_rc = 0;
+		complete(&info->ri_done);
+		break;
+
 	case RDMA_CM_EVENT_ROUTE_RESOLVED:
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED;
 		info->ri_rc = 0;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ADDR_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED;
 		info->ri_rc = -EHOSTUNREACH;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED;
 		info->ri_rc = -ENETUNREACH;
 		complete(&info->ri_done);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
 		log_rdma_event(INFO, "connected event=%s\n", event_name);
-		sc->status = SMBDIRECT_SOCKET_CONNECTED;
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_NEGOTIATE_NEEDED;
 		wake_up_interruptible(&info->status_wait);
 		break;
 
@@ -212,7 +225,8 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_UNREACHABLE:
 	case RDMA_CM_EVENT_REJECTED:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
-		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
+		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_FAILED;
 		wake_up_interruptible(&info->status_wait);
 		break;
 
@@ -481,6 +495,12 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
 		info->negotiate_done =
 			process_negotiation_response(response, wc->byte_len);
 		put_receive_buffer(info, response);
+		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING);
+		if (!info->negotiate_done) {
+			sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
+		} else {
+			sc->status = SMBDIRECT_SOCKET_CONNECTED;
+		}
 		complete(&info->negotiate_completion);
 		return;
 
@@ -556,6 +576,7 @@ static struct rdma_cm_id *smbd_create_id(
 		struct smbd_connection *info,
 		struct sockaddr *dstaddr, int port)
 {
+	struct smbdirect_socket *sc = &info->socket;
 	struct rdma_cm_id *id;
 	int rc;
 	__be16 *sport;
@@ -578,6 +599,8 @@ static struct rdma_cm_id *smbd_create_id(
 	init_completion(&info->ri_done);
 	info->ri_rc = -ETIMEDOUT;
 
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_NEEDED);
+	sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING;
 	rc = rdma_resolve_addr(id, NULL, (struct sockaddr *)dstaddr,
 		RDMA_RESOLVE_TIMEOUT);
 	if (rc) {
@@ -598,6 +621,8 @@ static struct rdma_cm_id *smbd_create_id(
 	}
 
 	info->ri_rc = -ETIMEDOUT;
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED);
+	sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING;
 	rc = rdma_resolve_route(id, RDMA_RESOLVE_TIMEOUT);
 	if (rc) {
 		log_rdma_event(ERR, "rdma_resolve_route() failed %i\n", rc);
@@ -644,6 +669,9 @@ static int smbd_ia_open(
 	struct smbdirect_socket *sc = &info->socket;
 	int rc;
 
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_CREATED);
+	sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_NEEDED;
+
 	sc->rdma.cm_id = smbd_create_id(info, dstaddr, port);
 	if (IS_ERR(sc->rdma.cm_id)) {
 		rc = PTR_ERR(sc->rdma.cm_id);
@@ -1085,6 +1113,9 @@ static int smbd_negotiate(struct smbd_connection *info)
 	int rc;
 	struct smbdirect_recv_io *response = get_receive_buffer(info);
 
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_NEGOTIATE_NEEDED);
+	sc->status = SMBDIRECT_SOCKET_NEGOTIATE_RUNNING;
+
 	sc->recv_io.expected = SMBDIRECT_EXPECT_NEGOTIATE_REP;
 	rc = smbd_post_recv(info, response);
 	log_rdma_event(INFO, "smbd_post_recv rc=%d iov.addr=0x%llx iov.length=%u iov.lkey=0x%x\n",
@@ -1540,7 +1571,7 @@ static struct smbd_connection *_smbd_get_connection(
 	sc = &info->socket;
 	sp = &sc->parameters;
 
-	sc->status = SMBDIRECT_SOCKET_CONNECTING;
+	sc->status = SMBDIRECT_SOCKET_CREATED;
 	rc = smbd_ia_open(info, dstaddr, port);
 	if (rc) {
 		log_rdma_event(INFO, "smbd_ia_open rc=%d\n", rc);
@@ -1652,6 +1683,9 @@ static struct smbd_connection *_smbd_get_connection(
 
 	init_waitqueue_head(&info->status_wait);
 	init_waitqueue_head(&sc->recv_io.reassembly.wait_queue);
+
+	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED);
+	sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING;
 	rc = rdma_connect(sc->rdma.cm_id, &conn_param);
 	if (rc) {
 		log_rdma_event(ERR, "rdma_connect() failed with %i\n", rc);
@@ -1660,10 +1694,10 @@ static struct smbd_connection *_smbd_get_connection(
 
 	wait_event_interruptible_timeout(
 		info->status_wait,
-		sc->status != SMBDIRECT_SOCKET_CONNECTING,
+		sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
-	if (sc->status != SMBDIRECT_SOCKET_CONNECTED) {
+	if (sc->status != SMBDIRECT_SOCKET_NEGOTIATE_NEEDED) {
 		log_rdma_event(ERR, "rdma_connect failed port=%d\n", port);
 		goto rdma_connect_failed;
 	}
fs/smb/common/smbdirect/smbdirect_socket.h +12 -2
index 3c4a8d627aa3..f43eabdd413d 100644
@@ -8,9 +8,19 @@
 
 enum smbdirect_socket_status {
 	SMBDIRECT_SOCKET_CREATED,
-	SMBDIRECT_SOCKET_CONNECTING,
-	SMBDIRECT_SOCKET_CONNECTED,
+	SMBDIRECT_SOCKET_RESOLVE_ADDR_NEEDED,
+	SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING,
+	SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED,
+	SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED,
+	SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING,
+	SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED,
+	SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED,
+	SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING,
+	SMBDIRECT_SOCKET_RDMA_CONNECT_FAILED,
+	SMBDIRECT_SOCKET_NEGOTIATE_NEEDED,
+	SMBDIRECT_SOCKET_NEGOTIATE_RUNNING,
 	SMBDIRECT_SOCKET_NEGOTIATE_FAILED,
+	SMBDIRECT_SOCKET_CONNECTED,
 	SMBDIRECT_SOCKET_DISCONNECTING,
 	SMBDIRECT_SOCKET_DISCONNECTED,
 	SMBDIRECT_SOCKET_DESTROYED
Signature · 1 line
2.43.0
author reviewer
#6 lore raw

PATCH 6/9 smb: client: use status_wait and SMBDIRECT_SOCKET_NEGOTIATE_RUNNING for completion

In reply to Stefan Metzmacher

We can use the state change from SMBDIRECT_SOCKET_NEGOTIATE_RUNNING to SMBDIRECT_SOCKET_CONNECTED or SMBDIRECT_SOCKET_NEGOTIATE_FAILED in order to notify the caller if the negotiation is over.

---
 fs/smb/client/smbdirect.c | 19 ++++++++++---------
 fs/smb/client/smbdirect.h |  3 ---
 2 files changed, 10 insertions(+), 12 deletions(-)
fs/smb/client/smbdirect.c +10 -9
index f95ecc153249..bdc4600d03e3 100644
@@ -470,6 +470,7 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
 	struct smbd_connection *info =
 		container_of(sc, struct smbd_connection, socket);
 	int data_length = 0;
+	bool negotiate_done = false;
 
 	log_rdma_recv(INFO, "response=0x%p type=%d wc status=%d wc opcode %d byte_len=%d pkey_index=%u\n",
 		      response, sc->recv_io.expected, wc->status, wc->opcode,
@@ -492,16 +493,16 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
 	case SMBDIRECT_EXPECT_NEGOTIATE_REP:
 		dump_smbdirect_negotiate_resp(smbdirect_recv_io_payload(response));
 		sc->recv_io.reassembly.full_packet_received = true;
-		info->negotiate_done =
+		negotiate_done =
 			process_negotiation_response(response, wc->byte_len);
 		put_receive_buffer(info, response);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING);
-		if (!info->negotiate_done) {
+		if (!negotiate_done) {
 			sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
 		} else {
 			sc->status = SMBDIRECT_SOCKET_CONNECTED;
 		}
-		complete(&info->negotiate_completion);
+		wake_up_interruptible(&info->status_wait);
 		return;
 
 	/* SMBD data transfer packet */
@@ -1124,17 +1125,17 @@ static int smbd_negotiate(struct smbd_connection *info)
 	if (rc)
 		return rc;
 
-	init_completion(&info->negotiate_completion);
-	info->negotiate_done = false;
 	rc = smbd_post_send_negotiate_req(info);
 	if (rc)
 		return rc;
 
-	rc = wait_for_completion_interruptible_timeout(
-		&info->negotiate_completion, SMBD_NEGOTIATE_TIMEOUT * HZ);
-	log_rdma_event(INFO, "wait_for_completion_timeout rc=%d\n", rc);
+	rc = wait_event_interruptible_timeout(
+		info->status_wait,
+		sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING,
+		secs_to_jiffies(SMBD_NEGOTIATE_TIMEOUT));
+	log_rdma_event(INFO, "wait_event_interruptible_timeout rc=%d\n", rc);
 
-	if (info->negotiate_done)
+	if (sc->status == SMBDIRECT_SOCKET_CONNECTED)
 		return 0;
 
 	if (rc == 0)
fs/smb/client/smbdirect.h +0 -3
index e45aa9ddd71d..82505df4a751 100644
@@ -49,9 +49,6 @@ struct smbd_connection {
 	struct completion ri_done;
 	wait_queue_head_t status_wait;
 
-	struct completion negotiate_completion;
-	bool negotiate_done;
-
 	struct work_struct disconnect_work;
 	struct work_struct post_send_credits_work;
 
Signature · 1 line
2.43.0
author reviewer
#7 lore raw

PATCH 7/9 smb: client: use status_wait and SMBDIRECT_SOCKET_RESOLVE_{ADDR,ROUTE}_RUNNING for completion

In reply to Stefan Metzmacher

We can use the state change from SMBDIRECT_SOCKET_RESOLVE_{ADDR,ROUTE}_RUNNING to the next state in order to wake the caller to do the next step.

---
 fs/smb/client/smbdirect.c | 46 ++++++++++++++++++++++-----------------
 fs/smb/client/smbdirect.h |  2 --
 2 files changed, 26 insertions(+), 22 deletions(-)
fs/smb/client/smbdirect.c +26 -20
index bdc4600d03e3..ab5b7ae04032 100644
@@ -187,31 +187,27 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_ADDR_RESOLVED:
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED;
-		info->ri_rc = 0;
-		complete(&info->ri_done);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_RESOLVED:
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED;
-		info->ri_rc = 0;
-		complete(&info->ri_done);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ADDR_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED;
-		info->ri_rc = -EHOSTUNREACH;
-		complete(&info->ri_done);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED;
-		info->ri_rc = -ENETUNREACH;
-		complete(&info->ri_done);
+		wake_up_interruptible(&info->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
@@ -597,9 +593,6 @@ static struct rdma_cm_id *smbd_create_id(
 
 	*sport = htons(port);
 
-	init_completion(&info->ri_done);
-	info->ri_rc = -ETIMEDOUT;
-
 	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_NEEDED);
 	sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING;
 	rc = rdma_resolve_addr(id, NULL, (struct sockaddr *)dstaddr,
@@ -608,20 +601,26 @@ static struct rdma_cm_id *smbd_create_id(
 		log_rdma_event(ERR, "rdma_resolve_addr() failed %i\n", rc);
 		goto out;
 	}
-	rc = wait_for_completion_interruptible_timeout(
-		&info->ri_done, msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
+	rc = wait_event_interruptible_timeout(
+		info->status_wait,
+		sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING,
+		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 	/* e.g. if interrupted returns -ERESTARTSYS */
 	if (rc < 0) {
 		log_rdma_event(ERR, "rdma_resolve_addr timeout rc: %i\n", rc);
 		goto out;
 	}
-	rc = info->ri_rc;
-	if (rc) {
+	if (sc->status == SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING) {
+		rc = -ETIMEDOUT;
+		log_rdma_event(ERR, "rdma_resolve_addr() completed %i\n", rc);
+		goto out;
+	}
+	if (sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED) {
+		rc = -EHOSTUNREACH;
 		log_rdma_event(ERR, "rdma_resolve_addr() completed %i\n", rc);
 		goto out;
 	}
 
-	info->ri_rc = -ETIMEDOUT;
 	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED);
 	sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING;
 	rc = rdma_resolve_route(id, RDMA_RESOLVE_TIMEOUT);
@@ -629,15 +628,22 @@ static struct rdma_cm_id *smbd_create_id(
 		log_rdma_event(ERR, "rdma_resolve_route() failed %i\n", rc);
 		goto out;
 	}
-	rc = wait_for_completion_interruptible_timeout(
-		&info->ri_done, msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
+	rc = wait_event_interruptible_timeout(
+		info->status_wait,
+		sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING,
+		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 	/* e.g. if interrupted returns -ERESTARTSYS */
 	if (rc < 0)  {
 		log_rdma_event(ERR, "rdma_resolve_addr timeout rc: %i\n", rc);
 		goto out;
 	}
-	rc = info->ri_rc;
-	if (rc) {
+	if (sc->status == SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING) {
+		rc = -ETIMEDOUT;
+		log_rdma_event(ERR, "rdma_resolve_route() completed %i\n", rc);
+		goto out;
+	}
+	if (sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED) {
+		rc = -ENETUNREACH;
 		log_rdma_event(ERR, "rdma_resolve_route() completed %i\n", rc);
 		goto out;
 	}
fs/smb/client/smbdirect.h +0 -2
index 82505df4a751..62458a8fd109 100644
@@ -45,8 +45,6 @@ enum keep_alive_status {
 struct smbd_connection {
 	struct smbdirect_socket socket;
 
-	int ri_rc;
-	struct completion ri_done;
 	wait_queue_head_t status_wait;
 
 	struct work_struct disconnect_work;
Signature · 1 line
2.43.0
author reviewer
#8 lore raw

PATCH 8/9 smb: smbdirect: introduce smbdirect_socket.status_wait

In reply to Stefan Metzmacher

This will be used by server and client soon.

---
 fs/smb/common/smbdirect/smbdirect_socket.h | 1 +
 1 file changed, 1 insertion(+)
fs/smb/common/smbdirect/smbdirect_socket.h +1 -0
index f43eabdd413d..aac4b040606b 100644
@@ -28,6 +28,7 @@ enum smbdirect_socket_status {
 
 struct smbdirect_socket {
 	enum smbdirect_socket_status status;
+	wait_queue_head_t status_wait;
 
 	/* RDMA related */
 	struct {
Signature · 1 line
2.43.0
author reviewer
#9 lore raw

PATCH 9/9 smb: client: make use of smbdirect_socket.status_wait

In reply to Stefan Metzmacher

This will allow us to have common helper functions soon.

---
 fs/smb/client/smbdirect.c | 32 ++++++++++++++++----------------
 fs/smb/client/smbdirect.h |  2 --
 2 files changed, 16 insertions(+), 18 deletions(-)
fs/smb/client/smbdirect.c +16 -16
index ab5b7ae04032..f36226e0331b 100644
@@ -187,34 +187,34 @@ static int smbd_conn_upcall(
 	case RDMA_CM_EVENT_ADDR_RESOLVED:
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_RESOLVED:
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ADDR_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ROUTE_ERROR:
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_ESTABLISHED:
 		log_rdma_event(INFO, "connected event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_NEGOTIATE_NEEDED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_CONNECT_ERROR:
@@ -223,7 +223,7 @@ static int smbd_conn_upcall(
 		log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
 		WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
 		sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_FAILED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		break;
 
 	case RDMA_CM_EVENT_DEVICE_REMOVAL:
@@ -232,12 +232,12 @@ static int smbd_conn_upcall(
 		if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) {
 			log_rdma_event(ERR, "event=%s during negotiation\n", event_name);
 			sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-			wake_up(&info->status_wait);
+			wake_up(&sc->status_wait);
 			break;
 		}
 
 		sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		wake_up_interruptible(&sc->recv_io.reassembly.wait_queue);
 		wake_up_interruptible_all(&info->wait_send_queue);
 		break;
@@ -498,7 +498,7 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
 		} else {
 			sc->status = SMBDIRECT_SOCKET_CONNECTED;
 		}
-		wake_up_interruptible(&info->status_wait);
+		wake_up_interruptible(&sc->status_wait);
 		return;
 
 	/* SMBD data transfer packet */
@@ -602,7 +602,7 @@ static struct rdma_cm_id *smbd_create_id(
 		goto out;
 	}
 	rc = wait_event_interruptible_timeout(
-		info->status_wait,
+		sc->status_wait,
 		sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 	/* e.g. if interrupted returns -ERESTARTSYS */
@@ -629,7 +629,7 @@ static struct rdma_cm_id *smbd_create_id(
 		goto out;
 	}
 	rc = wait_event_interruptible_timeout(
-		info->status_wait,
+		sc->status_wait,
 		sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 	/* e.g. if interrupted returns -ERESTARTSYS */
@@ -1136,7 +1136,7 @@ static int smbd_negotiate(struct smbd_connection *info)
 		return rc;
 
 	rc = wait_event_interruptible_timeout(
-		info->status_wait,
+		sc->status_wait,
 		sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING,
 		secs_to_jiffies(SMBD_NEGOTIATE_TIMEOUT));
 	log_rdma_event(INFO, "wait_event_interruptible_timeout rc=%d\n", rc);
@@ -1363,7 +1363,7 @@ void smbd_destroy(struct TCP_Server_Info *server)
 		rdma_disconnect(sc->rdma.cm_id);
 		log_rdma_event(INFO, "wait for transport being disconnected\n");
 		wait_event_interruptible(
-			info->status_wait,
+			sc->status_wait,
 			sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 	}
 
@@ -1688,7 +1688,7 @@ static struct smbd_connection *_smbd_get_connection(
 	log_rdma_event(INFO, "connecting to IP %pI4 port %d\n",
 		&addr_in->sin_addr, port);
 
-	init_waitqueue_head(&info->status_wait);
+	init_waitqueue_head(&sc->status_wait);
 	init_waitqueue_head(&sc->recv_io.reassembly.wait_queue);
 
 	WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED);
@@ -1700,7 +1700,7 @@ static struct smbd_connection *_smbd_get_connection(
 	}
 
 	wait_event_interruptible_timeout(
-		info->status_wait,
+		sc->status_wait,
 		sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING,
 		msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
@@ -1757,7 +1757,7 @@ static struct smbd_connection *_smbd_get_connection(
 	destroy_caches_and_workqueue(info);
 	sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
 	rdma_disconnect(sc->rdma.cm_id);
-	wait_event(info->status_wait,
+	wait_event(sc->status_wait,
 		sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 
 allocate_cache_failed:
fs/smb/client/smbdirect.h +0 -2
index 62458a8fd109..79ab43b7ac19 100644
@@ -45,8 +45,6 @@ enum keep_alive_status {
 struct smbd_connection {
 	struct smbdirect_socket socket;
 
-	wait_queue_head_t status_wait;
-
 	struct work_struct disconnect_work;
 	struct work_struct post_send_credits_work;
 
Signature · 1 line
2.43.0
#12 lore raw
In reply to Stefan Metzmacher

Tentatively merged into cifs-2.6.git for-next pending additional review/testing

I did fix up minor checkpatch warnings in patches 5 and 6 (unneeded { and } in if ... else)

Quoted 59 lines · Stefan Metzmacher

Hi,

this is the next step towards a common smbdirect layer between cifs.ko and ksmbd.ko, with the aim to provide a socket layer for userspace usage at the end of the road.

This patchset focuses on the client side today.

The first one is a fix for very long timeouts against unreachable servers.

The others prepare the use of a single wait_queue for state changes. This removes a lot of special handling during the connect and negotiate phases.

The last two move the state_wait queue into the common smbdirect_socket.status_wait.

For the server I have only a single patch that also uses smbdirect_socket.status_wait, but I'm skipping the server patches today.

I plan a lot more progress on the server side tomorrow and hopefully finish the moving everything from struct smb_direct_transport into struct smbdirect_socket.

I used the following xfstests as regression tests:
cifs/001 generic/001 generic/002 generic/005 generic/006 generic/007 generic/010 generic/011

Between cifs.ko against ksmbd.ko via siw.ko.

This is on top of the patches for the client I posted yesterday...

Stefan Metzmacher (9):
smb: client: return an error if rdma_connect does not return within 5
seconds
smb: client: improve logging in smbd_conn_upcall()
smb: client: don't call init_waitqueue_head(&info->conn_wait) twice in
_smbd_get_connection
smb: client: only use a single wait_queue to monitor smbdirect
connection status
smb: client/smbdirect: replace SMBDIRECT_SOCKET_CONNECTING with more
detailed states
smb: client: use status_wait and SMBDIRECT_SOCKET_NEGOTIATE_RUNNING
for completion
smb: client: use status_wait and
SMBDIRECT_SOCKET_RESOLVE_{ADDR,ROUTE}_RUNNING for completion
smb: smbdirect: introduce smbdirect_socket.status_wait
smb: client: make use of smbdirect_socket.status_wait

 fs/smb/client/smbdirect.c                  | 137 ++++++++++++++-------
 fs/smb/client/smbdirect.h                  |   8 --
 fs/smb/common/smbdirect/smbdirect_socket.h |  15 ++-
 3 files changed, 105 insertions(+), 55 deletions(-)

--
2.43.0

Signature · 3 lines
Thanks,

Steve
#10 lore raw

Re: [PATCH 9/9] smb: client: make use of smbdirect_socket.status_wait

In reply to Stefan Metzmacher

Hi Metze,

&sc -> status_wait is initialized in _smbd_get_connection() in line 1691 but it is being used by smbd_ia_open() -> smbd_create_id() before that. This is giving an OOPS on RDMA mount (attached). Could you please check?

Thanks
Meetakshi

Quoted 168 lines · Stefan Metzmacher

This will allow us to have common helper functions soon.

---
 fs/smb/client/smbdirect.c | 32 ++++++++++++++++----------------
 fs/smb/client/smbdirect.h |  2 --
 2 files changed, 16 insertions(+), 18 deletions(-)
fs/smb/client/smbdirect.c +16 -16
index ab5b7ae04032..f36226e0331b 100644
@@ -187,34 +187,34 @@ static int smbd_conn_upcall(
        case RDMA_CM_EVENT_ADDR_RESOLVED:
                WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
                sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_NEEDED;
-               wake_up_interruptible(&info->status_wait);
+               wake_up_interruptible(&sc->status_wait);
                break;
 
        case RDMA_CM_EVENT_ROUTE_RESOLVED:
                WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
                sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED;
-               wake_up_interruptible(&info->status_wait);
+               wake_up_interruptible(&sc->status_wait);
                break;
 
        case RDMA_CM_EVENT_ADDR_ERROR:
                log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
                WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING);
                sc->status = SMBDIRECT_SOCKET_RESOLVE_ADDR_FAILED;
-               wake_up_interruptible(&info->status_wait);
+               wake_up_interruptible(&sc->status_wait);
                break;
 
        case RDMA_CM_EVENT_ROUTE_ERROR:
                log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
                WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING);
                sc->status = SMBDIRECT_SOCKET_RESOLVE_ROUTE_FAILED;
-               wake_up_interruptible(&info->status_wait);
+               wake_up_interruptible(&sc->status_wait);
                break;
 
        case RDMA_CM_EVENT_ESTABLISHED:
                log_rdma_event(INFO, "connected event=%s\n", event_name);
                WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
                sc->status = SMBDIRECT_SOCKET_NEGOTIATE_NEEDED;
-               wake_up_interruptible(&info->status_wait);
+               wake_up_interruptible(&sc->status_wait);
                break;
 
        case RDMA_CM_EVENT_CONNECT_ERROR:
@@ -223,7 +223,7 @@ static int smbd_conn_upcall(
                log_rdma_event(ERR, "connecting failed event=%s\n", event_name);
                WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING);
                sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_FAILED;
-               wake_up_interruptible(&info->status_wait);
+               wake_up_interruptible(&sc->status_wait);
                break;
 
        case RDMA_CM_EVENT_DEVICE_REMOVAL:
@@ -232,12 +232,12 @@ static int smbd_conn_upcall(
                if (sc->status == SMBDIRECT_SOCKET_NEGOTIATE_FAILED) {
                        log_rdma_event(ERR, "event=%s during negotiation\n", event_name);
                        sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-                       wake_up(&info->status_wait);
+                       wake_up(&sc->status_wait);
                        break;
                }
 
                sc->status = SMBDIRECT_SOCKET_DISCONNECTED;
-               wake_up_interruptible(&info->status_wait);
+               wake_up_interruptible(&sc->status_wait);
                wake_up_interruptible(&sc->recv_io.reassembly.wait_queue);
                wake_up_interruptible_all(&info->wait_send_queue);
                break;
@@ -498,7 +498,7 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
                } else {
                        sc->status = SMBDIRECT_SOCKET_CONNECTED;
                }
-               wake_up_interruptible(&info->status_wait);
+               wake_up_interruptible(&sc->status_wait);
                return;
 
        /* SMBD data transfer packet */
@@ -602,7 +602,7 @@ static struct rdma_cm_id *smbd_create_id(
                goto out;
        }
        rc = wait_event_interruptible_timeout(
-               info->status_wait,
+               sc->status_wait,
                sc->status != SMBDIRECT_SOCKET_RESOLVE_ADDR_RUNNING,
                msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
        /* e.g. if interrupted returns -ERESTARTSYS */
@@ -629,7 +629,7 @@ static struct rdma_cm_id *smbd_create_id(
                goto out;
        }
        rc = wait_event_interruptible_timeout(
-               info->status_wait,
+               sc->status_wait,
                sc->status != SMBDIRECT_SOCKET_RESOLVE_ROUTE_RUNNING,
                msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
        /* e.g. if interrupted returns -ERESTARTSYS */
@@ -1136,7 +1136,7 @@ static int smbd_negotiate(struct smbd_connection *info)
                return rc;
 
        rc = wait_event_interruptible_timeout(
-               info->status_wait,
+               sc->status_wait,
                sc->status != SMBDIRECT_SOCKET_NEGOTIATE_RUNNING,
                secs_to_jiffies(SMBD_NEGOTIATE_TIMEOUT));
        log_rdma_event(INFO, "wait_event_interruptible_timeout rc=%d\n", rc);
@@ -1363,7 +1363,7 @@ void smbd_destroy(struct TCP_Server_Info *server)
                rdma_disconnect(sc->rdma.cm_id);
                log_rdma_event(INFO, "wait for transport being disconnected\n");
                wait_event_interruptible(
-                       info->status_wait,
+                       sc->status_wait,
                        sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
        }
 
@@ -1688,7 +1688,7 @@ static struct smbd_connection *_smbd_get_connection(
        log_rdma_event(INFO, "connecting to IP %pI4 port %d\n",
                &addr_in->sin_addr, port);
 
-       init_waitqueue_head(&info->status_wait);
+       init_waitqueue_head(&sc->status_wait);
        init_waitqueue_head(&sc->recv_io.reassembly.wait_queue);
 
        WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED);
@@ -1700,7 +1700,7 @@ static struct smbd_connection *_smbd_get_connection(
        }
 
        wait_event_interruptible_timeout(
-               info->status_wait,
+               sc->status_wait,
                sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING,
                msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT));
 
@@ -1757,7 +1757,7 @@ static struct smbd_connection *_smbd_get_connection(
        destroy_caches_and_workqueue(info);
        sc->status = SMBDIRECT_SOCKET_NEGOTIATE_FAILED;
        rdma_disconnect(sc->rdma.cm_id);
-       wait_event(info->status_wait,
+       wait_event(sc->status_wait,
                sc->status == SMBDIRECT_SOCKET_DISCONNECTED);
 
 allocate_cache_failed:
fs/smb/client/smbdirect.h +0 -3
index 62458a8fd109..79ab43b7ac19 100644
@@ -45,8 +45,6 @@ enum keep_alive_status {
 struct smbd_connection {
        struct smbdirect_socket socket;
 
-       wait_queue_head_t status_wait;
-
        struct work_struct disconnect_work;
        struct work_struct post_send_credits_work;
 
--
Quoted 3 lines

2.43.0

[ 2972.050661] CIFS: Attempting to mount //10.5.0.13/MyShare
[ 2972.050704] BUG: unable to handle page fault for address: ffffffffffffffe8
[ 2972.054229] #PF: supervisor read access in kernel mode
[ 2972.056242] #PF: error_code(0x0000) - not-present page
[ 2972.058357] PGD a3163f067 P4D a3163f067 PUD a31641067 PMD 0
[ 2972.060597] Oops: Oops: 0000 [#1] SMP NOPTI
[ 2972.062256] CPU: 12 UID: 0 PID: 208040 Comm: mount.cifs Tainted: G W OE 6.16.06.17+ #1 VOLUNTARY
[ 2972.066177] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 2972.068729] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 03/08/2024
[ 2972.073001] RIP: 0010:prepare_to_wait_event+0x10e/0x130
[ 2972.075024] Code: 08 4d 8d 44 24 08 48 8d 57 e8 48 89 fe 4c 39 c7 74 1d 4c 89 c7 eb 13 48 8b 42 18 48 89 f7 48 8d 50 e8 49 39 c0 74 08 48 89 c6 <f6> 02 10 75 e8 48 8b 07 48 89 48 08 49 89 47 18 49 89 7f 20 48 89
[ 2972.082356] RSP: 0018:ffffd1da0e71b850 EFLAGS: 00010007
[ 2972.084435] RAX: ffffd1da0e71b8f0 RBX: 0000000000000001 RCX: ffffd1da0e71b8f0
[ 2972.087239] RDX: ffffffffffffffe8 RSI: 0000000000000000 RDI: ffff8f1ec3a92810
[ 2972.090116] RBP: ffffd1da0e71b878 R08: ffff8f1ec3a92810 R09: 0000000000000000
[ 2972.092970] R10: ffffd1da0e71b888 R11: 0000000000000002 R12: ffff8f1ec3a92808
[ 2972.095815] R13: ffff8f1ed864a8c0 R14: 0000000000000246 R15: ffffd1da0e71b8d8
[ 2972.098768] FS: 000070cd1d31b780(0000) GS:ffff8f271bce3000(0000) knlGS:0000000000000000
[ 2972.101955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2972.104323] CR2: ffffffffffffffe8 CR3: 000000014197d000 CR4: 0000000000350ef0
[ 2972.107135] Call Trace:
[ 2972.108265] <TASK>
[ 2972.109145] _smbd_get_connection+0xa43/0x1970 [cifs]
[ 2972.111310] ? srso_return_thunk+0x5/0x5f
[ 2972.112940] ? __pfx_autoremove_wake_function+0x10/0x10
[ 2972.115013] smbd_get_connection+0x26/0x50 [cifs]
[ 2972.116957] cifs_get_tcp_session+0x6a8/0x970 [cifs]
[ 2972.119152] cifs_mount_get_session+0x48/0x1c0 [cifs]
[ 2972.121358] dfs_mount_share+0x7b/0xb10 [cifs]
[ 2972.123161] ? srso_return_thunk+0x5/0x5f
[ 2972.124767] ? __wake_up_klogd+0x56/0x70
[ 2972.126407] ? srso_return_thunk+0x5/0x5f
[ 2972.128179] ? vprintk_emit+0x15d/0x370
[ 2972.129846] ? srso_return_thunk+0x5/0x5f
[ 2972.131461] ? __kmalloc_node_track_caller_noprof+0x172/0x520
[ 2972.133813] cifs_mount+0x5b/0x2e0 [cifs]
[ 2972.135549] cifs_smb3_do_mount+0x180/0x7e0 [cifs]
[ 2972.137626] ? vprintk+0x18/0x40
[ 2972.138997] ? srso_return_thunk+0x5/0x5f
[ 2972.140692] ? _printk+0x5f/0x80
[ 2972.142163] smb3_get_tree+0x16d/0x360 [cifs]
[ 2972.144015] vfs_get_tree+0x2a/0xf0
[ 2972.145411] path_mount+0x500/0xb00
[ 2972.146839] ? srso_return_thunk+0x5/0x5f
[ 2972.148479] __x64_sys_mount+0x116/0x150
[ 2972.150493] x64_sys_call+0x2082/0x20d0
[ 2972.152667] do_syscall_64+0x7b/0x4f0
[ 2972.154649] ? srso_return_thunk+0x5/0x5f
[ 2972.156815] ? __handle_mm_fault+0xa9c/0x1090
[ 2972.159232] ? __put_user_8+0xd/0x20
[ 2972.161060] ? srso_return_thunk+0x5/0x5f
[ 2972.163109] ? count_memcg_events+0xba/0x1a0
[ 2972.165247] ? srso_return_thunk+0x5/0x5f
[ 2972.167192] ? handle_mm_fault+0xbc/0x300
[ 2972.169118] ? srso_return_thunk+0x5/0x5f
[ 2972.171012] ? do_user_addr_fault+0x1b9/0x860
[ 2972.173152] ? srso_return_thunk+0x5/0x5f
[ 2972.175090] ? irqentry_exit_to_user_mode+0x2e/0x240
[ 2972.177519] ? srso_return_thunk+0x5/0x5f
[ 2972.179467] ? irqentry_exit+0x1d/0x30
[ 2972.181473] ? srso_return_thunk+0x5/0x5f
[ 2972.183554] ? exc_page_fault+0x84/0x150
[ 2972.185402] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 2972.187708] RIP: 0033:0x70cd1d12af0e
[ 2972.189706] Code: 48 8b 0d 0d 7f 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d da 7e 0d 00 f7 d8 64 89 01 48
[ 2972.197683] RSP: 002b:00007ffecb09f028 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
[ 2972.201254] RAX: ffffffffffffffda RBX: 000000000000000a RCX: 000070cd1d12af0e
[ 2972.204410] RDX: 000060acc558a47e RSI: 000060acc558a4e4 RDI: 00007ffecb0a0608
[ 2972.207687] RBP: 00007ffecb09f0e0 R08: 000060acea83eeb0 R09: 0000000000000000
[ 2972.210865] R10: 0000000000000000 R11: 0000000000000246 R12: 000060acc558a047
[ 2972.214110] R13: 000060acea83ff20 R14: 00007ffecb0a0608 R15: 000070cd1d32a000
[ 2972.217378] </TASK>
[ 2972.218656] Modules linked in: rpcrdma sunrpc rdma_ucm ib_iser libiscsi scsi_transport_iscsi siw ib_uverbs cifs(OE) nls_utf8 cifs_md4 netfs tls qrtr cfg80211 8021q garp mrp stp llc xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_owner xt_tcpudp nft_compat nf_tables x_tables binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common joydev mac_hid hyperv_drm polyval_clmulni serio_raw hid_generic ghash_clmulni_intel hid_hyperv aesni_intel hv_netvsc hyperv_fb hid hyperv_keyboard vmgenid ksmbd sch_fq_codel cifs_arc4 nls_ucs2_utils rdma_cm iw_cm ib_cm ib_core dm_multipath msr nvme_fabrics efi_pstore nfnetlink autofs4 [last unloaded: cifs]
[ 2972.244156] CR2: ffffffffffffffe8
[ 2972.246119] ---[ end trace 0000000000000000 ]---
[ 2972.345267] RIP: 0010:prepare_to_wait_event+0x10e/0x130
[ 2972.347983] Code: 08 4d 8d 44 24 08 48 8d 57 e8 48 89 fe 4c 39 c7 74 1d 4c 89 c7 eb 13 48 8b 42 18 48 89 f7 48 8d 50 e8 49 39 c0 74 08 48 89 c6 <f6> 02 10 75 e8 48 8b 07 48 89 48 08 49 89 47 18 49 89 7f 20 48 89
[ 2972.356513] RSP: 0018:ffffd1da0e71b850 EFLAGS: 00010007
[ 2972.359150] RAX: ffffd1da0e71b8f0 RBX: 0000000000000001 RCX: ffffd1da0e71b8f0
[ 2972.362453] RDX: ffffffffffffffe8 RSI: 0000000000000000 RDI: ffff8f1ec3a92810
[ 2972.365782] RBP: ffffd1da0e71b878 R08: ffff8f1ec3a92810 R09: 0000000000000000
[ 2972.369295] R10: ffffd1da0e71b888 R11: 0000000000000002 R12: ffff8f1ec3a92808
[ 2972.372907] R13: ffff8f1ed864a8c0 R14: 0000000000000246 R15: ffffd1da0e71b8d8
[ 2972.376455] FS: 000070cd1d31b780(0000) GS:ffff8f271bce3000(0000) knlGS:0000000000000000
[ 2972.380271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2972.383161] CR2: ffffffffffffffe8 CR3: 000000014197d000 CR4: 0000000000350ef0
[ 2972.386551] note: mount.cifs[208040] exited with irqs disabled
author reviewer
#11 lore raw

Re: [PATCH 9/9] smb: client: make use of smbdirect_socket.status_wait

Hi Meetakshi,

Quoted 4 lines

&sc -> status_wait is initialized in _smbd_get_connection() in line 1691 but it is being used by smbd_ia_open() -> smbd_create_id() before that. This is giving an OOPS on RDMA mount (attached). Could you please check?

The problem comes with this patch:
[PATCH 7/9] smb: client: use status_wait and SMBDIRECT_SOCKET_RESOLVE_{ADDR, ROUTE}_RUNNING for completion

And yesterday I was thinking the whole day about in which patch
I move init_waitqueue_head(&info->status_wait); to the top and then forgot it.

The strange thing is that all tests I made yesterday evening worked just fine something between 3-5 times.

And today I got the crash immediately and was about to debug it.

Thanks that you found it! And sorry for introducing the stupid problem.

I'll post v2 soon.

Thanks!
metze