Skip to contentSkip to conversation

ceph: stop forwarding the request when exceeding 256 times

PATCH Superseded [email protected] 3 participants 5 messages Mar 29, 2022 Ceph client

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: stop forwarding the request when exceeding 256 times

From: Xiubo Li <[email protected]>

The type of 'num_fwd' in ceph 'MClientRequestForward' is 'int32_t', while in 'ceph_mds_request_head' the type is '__u8'. So in case the request bounces between MDSes exceeding 256 times, the client will get stuck.

In this case it's ususally a bug in MDS and continue bouncing the request makes no sense.

---
 fs/ceph/mds_client.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
fs/ceph/mds_client.c +28 -3
index a89ee866ebbb..0bb6e7bc499c 100644
@@ -3293,6 +3293,7 @@ static void handle_forward(struct ceph_mds_client *mdsc,
 	int err = -EINVAL;
 	void *p = msg->front.iov_base;
 	void *end = p + msg->front.iov_len;
+	bool aborted = false;
 
 	ceph_decode_need(&p, end, 2*sizeof(u32), bad);
 	next_mds = ceph_decode_32(&p);
@@ -3309,8 +3310,28 @@ static void handle_forward(struct ceph_mds_client *mdsc,
 		dout("forward tid %llu aborted, unregistering\n", tid);
 		__unregister_request(mdsc, req);
 	} else if (fwd_seq <= req->r_num_fwd) {
-		dout("forward tid %llu to mds%d - old seq %d <= %d\n",
-		     tid, next_mds, req->r_num_fwd, fwd_seq);
+		/*
+		 * The type of 'num_fwd' in ceph 'MClientRequestForward'
+		 * is 'int32_t', while in 'ceph_mds_request_head' the
+		 * type is '__u8'. So in case the request bounces between
+		 * MDSes exceeding 256 times, the client will get stuck.
+		 *
+		 * In this case it's ususally a bug in MDS and continue
+		 * bouncing the request makes no sense.
+		 */
+		if (req->r_num_fwd == 256) {
+			mutex_lock(&req->r_fill_mutex);
+			req->r_err = -EIO;
+			set_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags);
+			mutex_unlock(&req->r_fill_mutex);
+			aborted = true;
+			dout("forward tid %llu to mds%d - seq overflowed %d <= %d\n",
+			     tid, next_mds, req->r_num_fwd, fwd_seq);
+			goto out;
+		} else {
+			dout("forward tid %llu to mds%d - old seq %d <= %d\n",
+			     tid, next_mds, req->r_num_fwd, fwd_seq);
+		}
 	} else {
 		/* resend. forward race not possible; mds would drop */
 		dout("forward tid %llu to mds%d (we resend)\n", tid, next_mds);
@@ -3322,9 +3343,13 @@ static void handle_forward(struct ceph_mds_client *mdsc,
 		put_request_session(req);
 		__do_request(mdsc, req);
 	}
-	ceph_mdsc_put_request(req);
 out:
 	mutex_unlock(&mdsc->mutex);
+
+	/* kick calling process */
+	if (aborted)
+		complete_request(mdsc, req);
+	ceph_mdsc_put_request(req);
 	return;
 
 bad:
Signature · 1 line
2.27.0
#1 lore raw
Quoted 9 lines · [email protected] writes:

From: Xiubo Li <[email protected]>

The type of 'num_fwd' in ceph 'MClientRequestForward' is 'int32_t', while in 'ceph_mds_request_head' the type is '__u8'. So in case the request bounces between MDSes exceeding 256 times, the client will get stuck.

In this case it's ususally a bug in MDS and continue bouncing the request makes no sense.

Ouch. Nice catch. This patch looks OK to me, just 2 minor comments bellow.

Quoted 25 lines
---
 fs/ceph/mds_client.c | 31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
fs/ceph/mds_client.c +3 -2
index a89ee866ebbb..0bb6e7bc499c 100644
@@ -3293,6 +3293,7 @@ static void handle_forward(struct ceph_mds_client *mdsc,
 	int err = -EINVAL;
 	void *p = msg->front.iov_base;
 	void *end = p + msg->front.iov_len;
+	bool aborted = false;
 
 	ceph_decode_need(&p, end, 2*sizeof(u32), bad);
 	next_mds = ceph_decode_32(&p);
@@ -3309,8 +3310,28 @@ static void handle_forward(struct ceph_mds_client *mdsc,
 		dout("forward tid %llu aborted, unregistering\n", tid);
 		__unregister_request(mdsc, req);
 	} else if (fwd_seq <= req->r_num_fwd) {
-		dout("forward tid %llu to mds%d - old seq %d <= %d\n",
-		     tid, next_mds, req->r_num_fwd, fwd_seq);
+		/*
+		 * The type of 'num_fwd' in ceph 'MClientRequestForward'

fs/ceph/mds_client.c static void handle_forward(struct ceph_mds_client *mdsc, · line 3324

Quoted 10 lines
fs/ceph/mds_client.c +10 -0
+		 * is 'int32_t', while in 'ceph_mds_request_head' the
+		 * type is '__u8'. So in case the request bounces between
+		 * MDSes exceeding 256 times, the client will get stuck.
+		 *
+		 * In this case it's ususally a bug in MDS and continue
+		 * bouncing the request makes no sense.
+		 */
+		if (req->r_num_fwd == 256) {
+			mutex_lock(&req->r_fill_mutex);
+			req->r_err = -EIO;

Not sure -EIO is the most appropriate. Maybe -E2BIG... although not quite it either.

fs/ceph/mds_client.c static void handle_forward(struct ceph_mds_client *mdsc, · line 3330

Quoted 6 lines
fs/ceph/mds_client.c +6 -0
+			set_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags);
+			mutex_unlock(&req->r_fill_mutex);
+			aborted = true;
+			dout("forward tid %llu to mds%d - seq overflowed %d <= %d\n",
+			     tid, next_mds, req->r_num_fwd, fwd_seq);
+			goto out;

This 'goto' statement can be dropped, but one before (when the lookup_get_request() fails) needs to be adjusted, otherwise ceph_mdsc_put_request() may be called with a NULL pointer.

Cheers,

Signature · 1 line
Luís
Quoted 26 lines
fs/ceph/mds_client.c +9 -1
+		} else {
+			dout("forward tid %llu to mds%d - old seq %d <= %d\n",
+			     tid, next_mds, req->r_num_fwd, fwd_seq);
+		}
 	} else {
 		/* resend. forward race not possible; mds would drop */
 		dout("forward tid %llu to mds%d (we resend)\n", tid, next_mds);
@@ -3322,9 +3343,13 @@ static void handle_forward(struct ceph_mds_client *mdsc,
 		put_request_session(req);
 		__do_request(mdsc, req);
 	}
-	ceph_mdsc_put_request(req);
 out:
 	mutex_unlock(&mdsc->mutex);
+
+	/* kick calling process */
+	if (aborted)
+		complete_request(mdsc, req);
+	ceph_mdsc_put_request(req);
 	return;
 
 bad:
Signature · 1 line
2.27.0
author
#2 lore raw
Quoted 43 lines · Luís Henriques
Quoted 9 lines · [email protected] writes:

From: Xiubo Li <[email protected]>

The type of 'num_fwd' in ceph 'MClientRequestForward' is 'int32_t', while in 'ceph_mds_request_head' the type is '__u8'. So in case the request bounces between MDSes exceeding 256 times, the client will get stuck.

In this case it's ususally a bug in MDS and continue bouncing the request makes no sense.

Ouch. Nice catch. This patch looks OK to me, just 2 minor comments bellow.

Quoted 29 lines
---
  fs/ceph/mds_client.c | 31 ++++++++++++++++++++++++++++---
  1 file changed, 28 insertions(+), 3 deletions(-)
fs/ceph/mds_client.c +7 -2
index a89ee866ebbb..0bb6e7bc499c 100644
@@ -3293,6 +3293,7 @@ static void handle_forward(struct ceph_mds_client *mdsc,
  	int err = -EINVAL;
  	void *p = msg->front.iov_base;
  	void *end = p + msg->front.iov_len;
+	bool aborted = false;
 
  	ceph_decode_need(&p, end, 2*sizeof(u32), bad);
  	next_mds = ceph_decode_32(&p);
@@ -3309,8 +3310,28 @@ static void handle_forward(struct ceph_mds_client *mdsc,
  		dout("forward tid %llu aborted, unregistering\n", tid);
  		__unregister_request(mdsc, req);
  	} else if (fwd_seq <= req->r_num_fwd) {
-		dout("forward tid %llu to mds%d - old seq %d <= %d\n",
-		     tid, next_mds, req->r_num_fwd, fwd_seq);
+		/*
+		 * The type of 'num_fwd' in ceph 'MClientRequestForward'
+		 * is 'int32_t', while in 'ceph_mds_request_head' the
+		 * type is '__u8'. So in case the request bounces between
+		 * MDSes exceeding 256 times, the client will get stuck.
+		 *
Quoted 9 lines

fs/ceph/mds_client.c static void handle_forward(struct ceph_mds_client *mdsc, · line 3324

Quoted 6 lines
fs/ceph/mds_client.c +6 -0
+		 * In this case it's ususally a bug in MDS and continue
+		 * bouncing the request makes no sense.
+		 */
+		if (req->r_num_fwd == 256) {
+			mutex_lock(&req->r_fill_mutex);
+			req->r_err = -EIO;

Not sure -EIO is the most appropriate. Maybe -E2BIG... although not quite it either.

Yeah, I also not very sure here.

Jeff ?

Quoted 9 lines

fs/ceph/mds_client.c static void handle_forward(struct ceph_mds_client *mdsc, · line 3330

Quoted 6 lines
fs/ceph/mds_client.c +6 -0
+			set_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags);
+			mutex_unlock(&req->r_fill_mutex);
+			aborted = true;
+			dout("forward tid %llu to mds%d - seq overflowed %d <= %d\n",
+			     tid, next_mds, req->r_num_fwd, fwd_seq);
+			goto out;

This 'goto' statement can be dropped, but one before (when the lookup_get_request() fails) needs to be adjusted, otherwise ceph_mdsc_put_request() may be called with a NULL pointer.

Yeah, will fix it.

Thanks.

-- Xiubo

Quoted 1 line

Cheers,

#3 lore raw
Quoted 53 lines · Xiubo Li
Quoted 43 lines · Luís Henriques
Quoted 9 lines · [email protected] writes:

From: Xiubo Li <[email protected]>

The type of 'num_fwd' in ceph 'MClientRequestForward' is 'int32_t', while in 'ceph_mds_request_head' the type is '__u8'. So in case the request bounces between MDSes exceeding 256 times, the client will get stuck.

In this case it's ususally a bug in MDS and continue bouncing the request makes no sense.

Ouch. Nice catch. This patch looks OK to me, just 2 minor comments bellow.

Quoted 29 lines
---
  fs/ceph/mds_client.c | 31 ++++++++++++++++++++++++++++---
  1 file changed, 28 insertions(+), 3 deletions(-)
fs/ceph/mds_client.c +7 -2
index a89ee866ebbb..0bb6e7bc499c 100644
@@ -3293,6 +3293,7 @@ static void handle_forward(struct ceph_mds_client *mdsc,
  	int err = -EINVAL;
  	void *p = msg->front.iov_base;
  	void *end = p + msg->front.iov_len;
+	bool aborted = false;
 
  	ceph_decode_need(&p, end, 2*sizeof(u32), bad);
  	next_mds = ceph_decode_32(&p);
@@ -3309,8 +3310,28 @@ static void handle_forward(struct ceph_mds_client *mdsc,
  		dout("forward tid %llu aborted, unregistering\n", tid);
  		__unregister_request(mdsc, req);
  	} else if (fwd_seq <= req->r_num_fwd) {
-		dout("forward tid %llu to mds%d - old seq %d <= %d\n",
-		     tid, next_mds, req->r_num_fwd, fwd_seq);
+		/*
+		 * The type of 'num_fwd' in ceph 'MClientRequestForward'
+		 * is 'int32_t', while in 'ceph_mds_request_head' the
+		 * type is '__u8'. So in case the request bounces between
+		 * MDSes exceeding 256 times, the client will get stuck.
+		 *
Quoted 9 lines

fs/ceph/mds_client.c static void handle_forward(struct ceph_mds_client *mdsc, · line 3324

Quoted 6 lines
fs/ceph/mds_client.c +6 -0
+		 * In this case it's ususally a bug in MDS and continue
+		 * bouncing the request makes no sense.
+		 */
+		if (req->r_num_fwd == 256) {
+			mutex_lock(&req->r_fill_mutex);
+			req->r_err = -EIO;

Not sure -EIO is the most appropriate. Maybe -E2BIG... although not quite it either.

Quoted 4 lines

Yeah, I also not very sure here.

Jeff ?

Matching errors like this really comes down to a judgement call. E2BIG usually means that some buffer was sized too small, so you'll have users trying to figure out what they passed in wrong if you return that here.

-EIO is the usual "default" when you don't know what else to use. There's also -EREMOTEIO which may be closer here since this is indicative of MDS problems. Given that, it may also be a good idea to log a pr_warn or pr_notice message at the same time explaining what happened.

Quoted 20 lines
Quoted 10 lines

fs/ceph/mds_client.c static void handle_forward(struct ceph_mds_client *mdsc, · line 3330

Quoted 7 lines
fs/ceph/mds_client.c +6 -0
+			set_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags);
+			mutex_unlock(&req->r_fill_mutex);
+			aborted = true;
+			dout("forward tid %llu to mds%d - seq overflowed %d <= %d\n",
+			     tid, next_mds, req->r_num_fwd, fwd_seq);
+			goto out;

This 'goto' statement can be dropped, but one before (when the lookup_get_request() fails) needs to be adjusted, otherwise ceph_mdsc_put_request() may be called with a NULL pointer.

Yeah, will fix it.

Thanks.

-- Xiubo

Quoted 1 line

Cheers,

Signature · 1 line
Jeff Layton <[email protected]>
author
#4 lore raw
Quoted 62 lines · Jeff Layton
Quoted 53 lines · Xiubo Li
Quoted 43 lines · Luís Henriques
Quoted 9 lines · [email protected] writes:

From: Xiubo Li <[email protected]>

The type of 'num_fwd' in ceph 'MClientRequestForward' is 'int32_t', while in 'ceph_mds_request_head' the type is '__u8'. So in case the request bounces between MDSes exceeding 256 times, the client will get stuck.

In this case it's ususally a bug in MDS and continue bouncing the request makes no sense.

Ouch. Nice catch. This patch looks OK to me, just 2 minor comments bellow.

Quoted 29 lines
---
   fs/ceph/mds_client.c | 31 ++++++++++++++++++++++++++++---
   1 file changed, 28 insertions(+), 3 deletions(-)
fs/ceph/mds_client.c +7 -2
index a89ee866ebbb..0bb6e7bc499c 100644
@@ -3293,6 +3293,7 @@ static void handle_forward(struct ceph_mds_client *mdsc,
   	int err = -EINVAL;
   	void *p = msg->front.iov_base;
   	void *end = p + msg->front.iov_len;
+	bool aborted = false;
 
   	ceph_decode_need(&p, end, 2*sizeof(u32), bad);
   	next_mds = ceph_decode_32(&p);
@@ -3309,8 +3310,28 @@ static void handle_forward(struct ceph_mds_client *mdsc,
   		dout("forward tid %llu aborted, unregistering\n", tid);
   		__unregister_request(mdsc, req);
   	} else if (fwd_seq <= req->r_num_fwd) {
-		dout("forward tid %llu to mds%d - old seq %d <= %d\n",
-		     tid, next_mds, req->r_num_fwd, fwd_seq);
+		/*
+		 * The type of 'num_fwd' in ceph 'MClientRequestForward'
+		 * is 'int32_t', while in 'ceph_mds_request_head' the
+		 * type is '__u8'. So in case the request bounces between
+		 * MDSes exceeding 256 times, the client will get stuck.
+		 *
Quoted 9 lines

fs/ceph/mds_client.c static void handle_forward(struct ceph_mds_client *mdsc, · line 3324

Quoted 6 lines
fs/ceph/mds_client.c +6 -0
+		 * In this case it's ususally a bug in MDS and continue
+		 * bouncing the request makes no sense.
+		 */
+		if (req->r_num_fwd == 256) {
+			mutex_lock(&req->r_fill_mutex);
+			req->r_err = -EIO;

Not sure -EIO is the most appropriate. Maybe -E2BIG... although not quite it either.

Quoted 4 lines

Yeah, I also not very sure here.

Jeff ?

Matching errors like this really comes down to a judgement call. E2BIG usually means that some buffer was sized too small, so you'll have users trying to figure out what they passed in wrong if you return that here.

Quoted 5 lines

-EIO is the usual "default" when you don't know what else to use. There's also -EREMOTEIO which may be closer here since this is indicative of MDS problems. Given that, it may also be a good idea to log a pr_warn or pr_notice message at the same time explaining what happened.

As discussed in IRC, have switched to EMULTIHOP instead.

Thanks Jeff.

-- Xiubo

Quoted 18 lines
Quoted 18 lines
Quoted 10 lines

fs/ceph/mds_client.c static void handle_forward(struct ceph_mds_client *mdsc, · line 3330

Quoted 7 lines
fs/ceph/mds_client.c +6 -0
+			set_bit(CEPH_MDS_R_ABORTED, &req->r_req_flags);
+			mutex_unlock(&req->r_fill_mutex);
+			aborted = true;
+			dout("forward tid %llu to mds%d - seq overflowed %d <= %d\n",
+			     tid, next_mds, req->r_num_fwd, fwd_seq);
+			goto out;

This 'goto' statement can be dropped, but one before (when the lookup_get_request() fails) needs to be adjusted, otherwise ceph_mdsc_put_request() may be called with a NULL pointer.

Yeah, will fix it.

Thanks.

-- Xiubo

Quoted 1 line

Cheers,