Skip to contentSkip to conversation

lustre: some old patches from whamcloud tree

PATCH New James Simmons 3 participants 19 messages Jul 25, 2019 – Aug 14, 2019

lore
Share & tools
lore
#0 lore raw

cover lustre: some old patches from whamcloud tree

Some old patches missing from the linux lustre client. I suspect these weren't port since they most are likely server side only but I want to make sure. Also we are missing patch:

http://review.whamcloud.com/20939 from LU-8319 but I don't know if that is needed anymore with the earlier changes done. Feedback welcomed.

Andreas Dilger (1):
lustre: seq: make seq_proc_write_common() safer

Gregoire Pichon (1):
lustre: tests: testcases for multiple modify RPCs feature

Jeremy Filizetti (1):
lustre: ldlm: Don't check opcode with NULL rq_reqmsg

Li Wei (1):
lustre: ptlrpc: Fix an rq_no_reply assertion failure

Wang Di (1):
lustre: fld: retry fld rpc until the import is closed

wang di (3):
lustre: fld: resend seq lookup RPC if it is on LWP
lustre: fld: retry fld rpc even for ESHUTDOWN
lustre: fld: fld client lookup should retry

 fs/lustre/fid/lproc_fid.c       | 12 +++++++-----
 fs/lustre/fld/fld_request.c     | 42 ++++++++++++++++++++++++++++++++++++++---
 fs/lustre/include/obd_support.h |  2 ++
 fs/lustre/ldlm/ldlm_lib.c       |  7 +++++++
 fs/lustre/ptlrpc/service.c      | 10 ++++++++++
 5 files changed, 65 insertions(+), 8 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH 1/8 lustre: seq: make seq_proc_write_common() safer

From: Andreas Dilger <[email protected]>

Don't allow seq_proc_write_common() to specify arbitrary ranges, since this can permanently corrupt the sequence controller and/or sequnece server. That would allow duplicate FID allocation, or possibly prevent any new files to be created or servers to be added to the filesystem.

Instead, limit the sequence range that can be written via /proc to a subset of the sequence range currently allocated to that node. Add the "clear" keyword to allow dropping the entire local sequence and force a new one to be fetched from the sequence server.

WC-bug-id: https://jira.whamcloud.com/browse/LU-3642
Lustre-commit: 05f69f5ee20eeffcc26f643333cedcfb53ba6669

Reviewed-on: http://review.whamcloud.com/7123

---
 fs/lustre/fid/lproc_fid.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
fs/lustre/fid/lproc_fid.c +7 -5
index 94869d4..e2e47df 100644
@@ -52,14 +52,18 @@
 /* Format: [0x64BIT_INT - 0x64BIT_INT] + 32 bytes just in case */
 #define MAX_FID_RANGE_STRLEN (32 + 2 * 2 * sizeof(u64))
 /*
- * Note: this function is only used for testing, it is no safe for production
- * use.
+ * Reduce the SEQ range allocated to a node to a strict subset of the range
+ * currently-allocated SEQ range.  If the specified range is "clear", then
+ * drop all allocated sequences and request a new one from the master.
+ *
+ * Note: this function should only be used for testing, it is not necessarily
+ * safe for production use.
  */
 static int
 ldebugfs_fid_write_common(const char __user *buffer, size_t count,
 			  struct lu_seq_range *range)
 {
-	struct lu_seq_range tmp;
+	struct lu_seq_range tmp = { 0, };
 	int rc;
 	char kernbuf[MAX_FID_RANGE_STRLEN];
 
@@ -82,8 +86,6 @@
 	rc = sscanf(kernbuf, "[%llx - %llx]\n",
 		    (unsigned long long *)&tmp.lsr_start,
 		    (unsigned long long *)&tmp.lsr_end);
-	if (rc != 2)
-		return -EINVAL;
 	if (!lu_seq_range_is_sane(&tmp) || lu_seq_range_is_zero(&tmp) ||
 	    tmp.lsr_start < range->lsr_start || tmp.lsr_end > range->lsr_end)
 		return -EINVAL;
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH 1/8 lustre: seq: make seq_proc_write_common() safer

Quoted 16 lines · James Simmons

From: Andreas Dilger <[email protected]>

Don't allow seq_proc_write_common() to specify arbitrary ranges, since this can permanently corrupt the sequence controller and/or sequnece server. That would allow duplicate FID allocation, or possibly prevent any new files to be created or servers to be added to the filesystem.

Instead, limit the sequence range that can be written via /proc to a subset of the sequence range currently allocated to that node. Add the "clear" keyword to allow dropping the entire local sequence and force a new one to be fetched from the sequence server.

WC-bug-id: https://jira.whamcloud.com/browse/LU-3642
Lustre-commit: 05f69f5ee20eeffcc26f643333cedcfb53ba6669

Quoted 8 lines

Reviewed-on: http://review.whamcloud.com/7123

---
 fs/lustre/fid/lproc_fid.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

I already have this is my backport branch.
a8ab6db57383

NeilBrown

Quoted 38 lines
fs/lustre/fid/lproc_fid.c +7 -5
index 94869d4..e2e47df 100644
@@ -52,14 +52,18 @@
 /* Format: [0x64BIT_INT - 0x64BIT_INT] + 32 bytes just in case */
 #define MAX_FID_RANGE_STRLEN (32 + 2 * 2 * sizeof(u64))
 /*
- * Note: this function is only used for testing, it is no safe for production
- * use.
+ * Reduce the SEQ range allocated to a node to a strict subset of the range
+ * currently-allocated SEQ range.  If the specified range is "clear", then
+ * drop all allocated sequences and request a new one from the master.
+ *
+ * Note: this function should only be used for testing, it is not necessarily
+ * safe for production use.
  */
 static int
 ldebugfs_fid_write_common(const char __user *buffer, size_t count,
 			  struct lu_seq_range *range)
 {
-	struct lu_seq_range tmp;
+	struct lu_seq_range tmp = { 0, };
 	int rc;
 	char kernbuf[MAX_FID_RANGE_STRLEN];
 
@@ -82,8 +86,6 @@
 	rc = sscanf(kernbuf, "[%llx - %llx]\n",
 		    (unsigned long long *)&tmp.lsr_start,
 		    (unsigned long long *)&tmp.lsr_end);
-	if (rc != 2)
-		return -EINVAL;
 	if (!lu_seq_range_is_sane(&tmp) || lu_seq_range_is_zero(&tmp) ||
 	    tmp.lsr_start < range->lsr_start || tmp.lsr_end > range->lsr_end)
 		return -EINVAL;
Signature · 1 line
1.8.3.1

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20190726/bc2590d3/attachment.sig>

#3 lore raw

PATCH 1/8 lustre: seq: make seq_proc_write_common() safer

Quoted 26 lines
Quoted 17 lines

From: Andreas Dilger <[email protected]>

Don't allow seq_proc_write_common() to specify arbitrary ranges, since this can permanently corrupt the sequence controller and/or sequnece server. That would allow duplicate FID allocation, or possibly prevent any new files to be created or servers to be added to the filesystem.

Instead, limit the sequence range that can be written via /proc to a subset of the sequence range currently allocated to that node. Add the "clear" keyword to allow dropping the entire local sequence and force a new one to be fetched from the sequence server.

WC-bug-id: https://jira.whamcloud.com/browse/LU-3642
Lustre-commit: 05f69f5ee20eeffcc26f643333cedcfb53ba6669

Quoted 8 lines

Reviewed-on: http://review.whamcloud.com/7123

---
 fs/lustre/fid/lproc_fid.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
Quoted 2 lines

I already have this is my backport branch.
a8ab6db57383

I pushed the first batch from your backport branch. I remember remember pushing some patches before for fid / fld that ended up being server only patches. Its not really clear what is server or client in that code. I was hoping for some feedback from Andreas and / or Oleg about this patch collect being server only.

Quoted 38 lines
Quoted 37 lines
fs/lustre/fid/lproc_fid.c +7 -5
index 94869d4..e2e47df 100644
@@ -52,14 +52,18 @@
 /* Format: [0x64BIT_INT - 0x64BIT_INT] + 32 bytes just in case */
 #define MAX_FID_RANGE_STRLEN (32 + 2 * 2 * sizeof(u64))
 /*
- * Note: this function is only used for testing, it is no safe for production
- * use.
+ * Reduce the SEQ range allocated to a node to a strict subset of the range
+ * currently-allocated SEQ range.  If the specified range is "clear", then
+ * drop all allocated sequences and request a new one from the master.
+ *
+ * Note: this function should only be used for testing, it is not necessarily
+ * safe for production use.
  */
 static int
 ldebugfs_fid_write_common(const char __user *buffer, size_t count,
 			  struct lu_seq_range *range)
 {
-	struct lu_seq_range tmp;
+	struct lu_seq_range tmp = { 0, };
 	int rc;
 	char kernbuf[MAX_FID_RANGE_STRLEN];
 
@@ -82,8 +86,6 @@
 	rc = sscanf(kernbuf, "[%llx - %llx]\n",
 		    (unsigned long long *)&tmp.lsr_start,
 		    (unsigned long long *)&tmp.lsr_end);
-	if (rc != 2)
-		return -EINVAL;
 	if (!lu_seq_range_is_sane(&tmp) || lu_seq_range_is_zero(&tmp) ||
 	    tmp.lsr_start < range->lsr_start || tmp.lsr_end > range->lsr_end)
 		return -EINVAL;
Signature · 1 line
1.8.3.1
#4 lore raw

PATCH 2/8 lustre: ptlrpc: Fix an rq_no_reply assertion failure

In reply to James Simmons

From: Li Wei <[email protected]>

An OSS had an assertion failure:

LustreError: 5366:0:(ldlm_lib.c:2689:target_bulk_io()) @@@ timeout on bulk GET after 0+0s req at ffff88083a61b400 x1476486691018500/t0(4300509964) o4->8dda3382-83f8-6445-5eea-828fd59e4a06 at 192.168.1.116@o2ib1:0/0 lens 504/448 e 391470 to 0 dl 1408494729 ref 2 fl Complete:/4/0 rc 0/0
LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) ASSERTION( req->rq_no_reply == 0 ) failed:
Lustre: soaked-OST0000: Bulk IO write error with 8dda3382-83f8-6445-5eea-828fd59e4a06 (at 192.168.1.116 at o2ib1), client will retry: rc -110
LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) LBUG
Pid: 5432, comm: ll_ost_io03_003

  Call Trace:
  [<ffffffffa0641895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
  [<ffffffffa0641e97>] lbug_with_loc+0x47/0xb0 [libcfs]
  [<ffffffffa09cda4c>] ptlrpc_send_reply+0x4ec/0x7f0 [ptlrpc]
  [<ffffffffa09d4aae>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
  [<ffffffffa09e4d75>] ptlrpc_at_check_timed+0xcd5/0x1370 [ptlrpc]
  [<ffffffffa09dc1e9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
  [<ffffffffa09e66f8>] ptlrpc_main+0x12e8/0x1990 [ptlrpc]
  [<ffffffff81069290>] ? pick_next_task_fair+0xd0/0x130
  [<ffffffff81529246>] ? schedule+0x176/0x3b0
  [<ffffffffa09e5410>] ? ptlrpc_main+0x0/0x1990 [ptlrpc]
  [<ffffffff8109abf6>] kthread+0x96/0xa0
  [<ffffffff8100c20a>] child_rip+0xa/0x20
  [<ffffffff8109ab60>] ? kthread+0x0/0xa0
  [<ffffffff8100c200>] ? child_rip+0x0/0x20

The thread in tgt_brw_write() had decided not to reply by setting rq_no_reply, right before another thread tried to send an early reply for the request.

WC-bug-id: https://jira.whamcloud.com/browse/LU-5537
Lustre-commit: a8d448e4cd5978c546911f98067232bcdd30b651

Reviewed-on: http://review.whamcloud.com/11740

---
 fs/lustre/ptlrpc/service.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
fs/lustre/ptlrpc/service.c +10 -0
index a40e964..c9ab9c3 100644
@@ -1098,6 +1098,16 @@ static int ptlrpc_at_send_early_reply(struct ptlrpc_request *req)
 	reqcopy->rq_reqmsg = reqmsg;
 	memcpy(reqmsg, req->rq_reqmsg, req->rq_reqlen);
 
+	/*
+	 * tgt_brw_read() and tgt_brw_write() may have decided not to reply.
+	 * Without this check, we would fail the rq_no_reply assertion in
+	 * ptlrpc_send_reply().
+	 */
+	if (reqcopy->rq_no_reply) {
+		rc = -ETIMEDOUT;
+		goto out;
+	}
+
 	LASSERT(atomic_read(&req->rq_refcount));
 	/** if it is last refcount then early reply isn't needed */
 	if (atomic_read(&req->rq_refcount) == 1) {
Signature · 1 line
1.8.3.1
#5 lore raw

PATCH 2/8 lustre: ptlrpc: Fix an rq_no_reply assertion failure

This is definitely server code.

Cheers, Andreas

Quoted 74 lines · James Simmons

From: Li Wei <[email protected]>

An OSS had an assertion failure:

LustreError: 5366:0:(ldlm_lib.c:2689:target_bulk_io()) @@@ timeout on bulk GET after 0+0s req at ffff88083a61b400 x1476486691018500/t0(4300509964) o4->8dda3382-83f8-6445-5eea-828fd59e4a06 at 192.168.1.116@o2ib1:0/0 lens 504/448 e 391470 to 0 dl 1408494729 ref 2 fl Complete:/4/0 rc 0/0
LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) ASSERTION( req->rq_no_reply == 0 ) failed:
Lustre: soaked-OST0000: Bulk IO write error with 8dda3382-83f8-6445-5eea-828fd59e4a06 (at 192.168.1.116 at o2ib1), client will retry: rc -110
LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) LBUG
Pid: 5432, comm: ll_ost_io03_003

 Call Trace:
 [<ffffffffa0641895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
 [<ffffffffa0641e97>] lbug_with_loc+0x47/0xb0 [libcfs]
 [<ffffffffa09cda4c>] ptlrpc_send_reply+0x4ec/0x7f0 [ptlrpc]
 [<ffffffffa09d4aae>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
 [<ffffffffa09e4d75>] ptlrpc_at_check_timed+0xcd5/0x1370 [ptlrpc]
 [<ffffffffa09dc1e9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
 [<ffffffffa09e66f8>] ptlrpc_main+0x12e8/0x1990 [ptlrpc]
 [<ffffffff81069290>] ? pick_next_task_fair+0xd0/0x130
 [<ffffffff81529246>] ? schedule+0x176/0x3b0
 [<ffffffffa09e5410>] ? ptlrpc_main+0x0/0x1990 [ptlrpc]
 [<ffffffff8109abf6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ab60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20

The thread in tgt_brw_write() had decided not to reply by setting rq_no_reply, right before another thread tried to send an early reply for the request.

WC-bug-id: https://jira.whamcloud.com/browse/LU-5537
Lustre-commit: a8d448e4cd5978c546911f98067232bcdd30b651

Reviewed-on: http://review.whamcloud.com/11740

---
fs/lustre/ptlrpc/service.c | 10 ++++++++++

1 file changed, 10 insertions(+)
fs/lustre/ptlrpc/service.c +10 -0
index a40e964..c9ab9c3 100644
@@ -1098,6 +1098,16 @@ static int ptlrpc_at_send_early_reply(struct ptlrpc_request *req)
   reqcopy->rq_reqmsg = reqmsg;
   memcpy(reqmsg, req->rq_reqmsg, req->rq_reqlen);
 
+    /*
+     * tgt_brw_read() and tgt_brw_write() may have decided not to reply.
+     * Without this check, we would fail the rq_no_reply assertion in
+     * ptlrpc_send_reply().
+     */
+    if (reqcopy->rq_no_reply) {
+        rc = -ETIMEDOUT;
+        goto out;
+    }
+
   LASSERT(atomic_read(&req->rq_refcount));
   /** if it is last refcount then early reply isn't needed */
   if (atomic_read(&req->rq_refcount) == 1) {
Signature · 1 line
1.8.3.1
#6 lore raw

PATCH 3/8 lustre: fld: resend seq lookup RPC if it is on LWP

In reply to James Simmons

From: wang di <[email protected]>

Because Light Weight connection might be evicted after restart, then cause inflight RPC fails, to avoid this, we need resend seq lookup RPC.

remove "-f" from "stop mdt" in sanity 17m, so umount can keep the the connection, and otherwise the OSP might be evicted.

WC-bug-id: https://jira.whamcloud.com/browse/LU-4571
Lustre-commit: cf7f66d87e52293535cde6e8cc7386e6c1bdfa46

Reviewed-on: http://review.whamcloud.com/9106

---
 fs/lustre/fld/fld_request.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
fs/lustre/fld/fld_request.c +21 -2
index 248fffa..ec45ea6 100644
@@ -314,6 +314,7 @@ int fld_client_rpc(struct obd_export *exp,
 
 	LASSERT(exp);
 
+again:
 	imp = class_exp2cliimp(exp);
 	switch (fld_op) {
 	case FLD_QUERY:
@@ -329,8 +330,15 @@ int fld_client_rpc(struct obd_export *exp,
 		op = req_capsule_client_get(&req->rq_pill, &RMF_FLD_OPC);
 		*op = FLD_LOOKUP;
 
-		if (imp->imp_connect_flags_orig & OBD_CONNECT_MDS_MDS)
+		/* For MDS_MDS seq lookup, it will always use LWP connection,
+		 * but LWP will be evicted after restart, so cause the error.
+		 * so we will set no_delay for seq lookup request, once the
+		 * request fails because of the eviction. always retry here
+		 */
+		if (imp->imp_connect_flags_orig & OBD_CONNECT_MDS_MDS) {
 			req->rq_allow_replay = 1;
+			req->rq_no_delay = 1;
+		}
 		break;
 	case FLD_READ:
 		req = ptlrpc_request_alloc_pack(imp, &RQF_FLD_READ,
@@ -358,8 +366,19 @@ int fld_client_rpc(struct obd_export *exp,
 	obd_get_request_slot(&exp->exp_obd->u.cli);
 	rc = ptlrpc_queue_wait(req);
 	obd_put_request_slot(&exp->exp_obd->u.cli);
-	if (rc)
+	if (rc != 0) {
+		if (rc == -EWOULDBLOCK) {
+			/* For no_delay req(see above), EWOULDBLOCK means the
+			 * connection is being evicted, but this seq lookup
+			 * should not return error, since it would cause
+			 * unecessary failure of the application, instead
+			 * it should retry here
+			 */
+			ptlrpc_req_finished(req);
+			goto again;
+		}
 		goto out_req;
+	}
 
 	if (fld_op == FLD_QUERY) {
 		prange = req_capsule_server_get(&req->rq_pill, &RMF_FLD_MDFLD);
Signature · 1 line
1.8.3.1
#7 lore raw

PATCH 3/8 lustre: fld: resend seq lookup RPC if it is on LWP

This is functionality used only by the server (LWP connection and also MDS-MDS connection flag). But as I wrote previously, it will be tough to track this patch to only apply it when the server code is landed. Instead, it would likely just be a hard-to-find bug that needs to be tracked down again and fixed again.

My preference would be to land this and other similar patches in shared code that is not easily separated into client- and server-only sections.

Cheers, Andreas

Quoted 76 lines · James Simmons

From: wang di <[email protected]>

Because Light Weight connection might be evicted after restart, then cause inflight RPC fails, to avoid this, we need resend seq lookup RPC.

remove "-f" from "stop mdt" in sanity 17m, so umount can keep the the connection, and otherwise the OSP might be evicted.

WC-bug-id: https://jira.whamcloud.com/browse/LU-4571
Lustre-commit: cf7f66d87e52293535cde6e8cc7386e6c1bdfa46

Reviewed-on: http://review.whamcloud.com/9106

---
fs/lustre/fld/fld_request.c | 23 +++++++++++++++++++++--

1 file changed, 21 insertions(+), 2 deletions(-)
fs/lustre/fld/fld_request.c +21 -2
index 248fffa..ec45ea6 100644
@@ -314,6 +314,7 @@ int fld_client_rpc(struct obd_export *exp,
 
   LASSERT(exp);
 
+again:
   imp = class_exp2cliimp(exp);
   switch (fld_op) {
   case FLD_QUERY:
@@ -329,8 +330,15 @@ int fld_client_rpc(struct obd_export *exp,
       op = req_capsule_client_get(&req->rq_pill, &RMF_FLD_OPC);
       *op = FLD_LOOKUP;
 
-        if (imp->imp_connect_flags_orig & OBD_CONNECT_MDS_MDS)
+        /* For MDS_MDS seq lookup, it will always use LWP connection,
+         * but LWP will be evicted after restart, so cause the error.
+         * so we will set no_delay for seq lookup request, once the
+         * request fails because of the eviction. always retry here
+         */
+        if (imp->imp_connect_flags_orig & OBD_CONNECT_MDS_MDS) {
           req->rq_allow_replay = 1;
+            req->rq_no_delay = 1;
+        }
       break;
   case FLD_READ:
       req = ptlrpc_request_alloc_pack(imp, &RQF_FLD_READ,
@@ -358,8 +366,19 @@ int fld_client_rpc(struct obd_export *exp,
   obd_get_request_slot(&exp->exp_obd->u.cli);
   rc = ptlrpc_queue_wait(req);
   obd_put_request_slot(&exp->exp_obd->u.cli);
-    if (rc)
+    if (rc != 0) {
+        if (rc == -EWOULDBLOCK) {
+            /* For no_delay req(see above), EWOULDBLOCK means the
+             * connection is being evicted, but this seq lookup
+             * should not return error, since it would cause
+             * unecessary failure of the application, instead
+             * it should retry here
+             */
+            ptlrpc_req_finished(req);
+            goto again;
+        }
       goto out_req;
+    }
 
   if (fld_op == FLD_QUERY) {
       prange = req_capsule_server_get(&req->rq_pill, &RMF_FLD_MDFLD);
Signature · 1 line
1.8.3.1
#8 lore raw

PATCH 4/8 lustre: fld: retry fld rpc even for ESHUTDOWN

In reply to James Simmons

From: wang di <[email protected]>

when LWP is being evicted, because it is not replayable, the request might return ESHUTDOWN or EWOULDBLOCK, instead of failed, which might cause application failure, fld client will retry RPC, until the connection is being setup again or the LWP is being closed.

WC-bug-id: https://jira.whamcloud.com/browse/LU-4420
Lustre-commit: d335e310d4bf490509998ddbb1824e38cff20998

Reviewed-on: http://review.whamcloud.com/10285

---
 fs/lustre/fld/fld_request.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
fs/lustre/fld/fld_request.c +6 -6
index ec45ea6..ba0ef82 100644
@@ -367,12 +367,12 @@ int fld_client_rpc(struct obd_export *exp,
 	rc = ptlrpc_queue_wait(req);
 	obd_put_request_slot(&exp->exp_obd->u.cli);
 	if (rc != 0) {
-		if (rc == -EWOULDBLOCK) {
-			/* For no_delay req(see above), EWOULDBLOCK means the
-			 * connection is being evicted, but this seq lookup
-			 * should not return error, since it would cause
-			 * unecessary failure of the application, instead
-			 * it should retry here
+		if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN) {
+			/* For no_delay req(see above), EWOULDBLOCK and
+			 * ESHUTDOWN means the connection is being evicted,
+			 * but this seq lookup should not return error,
+			 * since it would cause unecessary failure of the
+			 * application, instead it should retry here
 			 */
 			ptlrpc_req_finished(req);
 			goto again;
Signature · 1 line
1.8.3.1
#9 lore raw

PATCH 4/8 lustre: fld: retry fld rpc even for ESHUTDOWN

This is mostly useful only on the servers, but clients also allocate "sequences" of 1, so it doesn't hurt to apply it to clients as well.

Cheers, Andreas

Quoted 46 lines · James Simmons

From: wang di <[email protected]>

when LWP is being evicted, because it is not replayable, the request might return ESHUTDOWN or EWOULDBLOCK, instead of failed, which might cause application failure, fld client will retry RPC, until the connection is being setup again or the LWP is being closed.

WC-bug-id: https://jira.whamcloud.com/browse/LU-4420
Lustre-commit: d335e310d4bf490509998ddbb1824e38cff20998

Reviewed-on: http://review.whamcloud.com/10285

---
fs/lustre/fld/fld_request.c | 12 ++++++------

1 file changed, 6 insertions(+), 6 deletions(-)
fs/lustre/fld/fld_request.c +6 -6
index ec45ea6..ba0ef82 100644
@@ -367,12 +367,12 @@ int fld_client_rpc(struct obd_export *exp,
   rc = ptlrpc_queue_wait(req);
   obd_put_request_slot(&exp->exp_obd->u.cli);
   if (rc != 0) {
-        if (rc == -EWOULDBLOCK) {
-            /* For no_delay req(see above), EWOULDBLOCK means the
-             * connection is being evicted, but this seq lookup
-             * should not return error, since it would cause
-             * unecessary failure of the application, instead
-             * it should retry here
+        if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN) {
+            /* For no_delay req(see above), EWOULDBLOCK and
+             * ESHUTDOWN means the connection is being evicted,
+             * but this seq lookup should not return error,
+             * since it would cause unecessary failure of the
+             * application, instead it should retry here
            */
           ptlrpc_req_finished(req);
           goto again;
Signature · 1 line
1.8.3.1
#10 lore raw

PATCH 4/8 lustre: fld: retry fld rpc even for ESHUTDOWN

In reply to James Simmons

While FLD is used on both client and server, the LWP connections are only used on clients. However, I don't see any harm to keep this code consistent with the master branch.

Otherwise, you would have to remember to apply this patch at some point in the future when the server code is landed, which is more likely to leave the bug in place

Cheers, Andreas

Quoted 46 lines · James Simmons

From: wang di <[email protected]>

when LWP is being evicted, because it is not replayable, the request might return ESHUTDOWN or EWOULDBLOCK, instead of failed, which might cause application failure, fld client will retry RPC, until the connection is being setup again or the LWP is being closed.

WC-bug-id: https://jira.whamcloud.com/browse/LU-4420
Lustre-commit: d335e310d4bf490509998ddbb1824e38cff20998

Reviewed-on: http://review.whamcloud.com/10285

---
fs/lustre/fld/fld_request.c | 12 ++++++------

1 file changed, 6 insertions(+), 6 deletions(-)
fs/lustre/fld/fld_request.c +6 -6
index ec45ea6..ba0ef82 100644
@@ -367,12 +367,12 @@ int fld_client_rpc(struct obd_export *exp,
   rc = ptlrpc_queue_wait(req);
   obd_put_request_slot(&exp->exp_obd->u.cli);
   if (rc != 0) {
-        if (rc == -EWOULDBLOCK) {
-            /* For no_delay req(see above), EWOULDBLOCK means the
-             * connection is being evicted, but this seq lookup
-             * should not return error, since it would cause
-             * unecessary failure of the application, instead
-             * it should retry here
+        if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN) {
+            /* For no_delay req(see above), EWOULDBLOCK and
+             * ESHUTDOWN means the connection is being evicted,
+             * but this seq lookup should not return error,
+             * since it would cause unecessary failure of the
+             * application, instead it should retry here
            */
           ptlrpc_req_finished(req);
           goto again;
Signature · 1 line
1.8.3.1
#11 lore raw

PATCH 5/8 lustre: fld: retry fld rpc until the import is closed

In reply to James Simmons

From: Wang Di <[email protected]>

Since LWP is not replayable, it should retry RPC until the import is closed, otherwise it would cause unnecessary failure of the application.

WC-bug-id: https://jira.whamcloud.com/browse/LU-5312
Lustre-commit: 07d481957c16832c782bb7d0c1fc436bcb148ea6

Reviewed-on: http://review.whamcloud.com/11039

---
 fs/lustre/fld/fld_request.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
fs/lustre/fld/fld_request.c +4 -6
index ba0ef82..60e7105 100644
@@ -367,12 +367,10 @@ int fld_client_rpc(struct obd_export *exp,
 	rc = ptlrpc_queue_wait(req);
 	obd_put_request_slot(&exp->exp_obd->u.cli);
 	if (rc != 0) {
-		if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN) {
-			/* For no_delay req(see above), EWOULDBLOCK and
-			 * ESHUTDOWN means the connection is being evicted,
-			 * but this seq lookup should not return error,
-			 * since it would cause unecessary failure of the
-			 * application, instead it should retry here
+		if (imp->imp_state != LUSTRE_IMP_CLOSED) {
+			/* Since LWP is not replayable, so it will keep
+			 * trying unless umount happens, otherwise it would
+			 * cause unecessary failure of the application.
 			 */
 			ptlrpc_req_finished(req);
 			goto again;
Signature · 1 line
1.8.3.1
#12 lore raw

PATCH 5/8 lustre: fld: retry fld rpc until the import is closed

You may as well merge this with the previous patch.

Cheers, Andreas

Quoted 43 lines · James Simmons

From: Wang Di <[email protected]>

Since LWP is not replayable, it should retry RPC until the import is closed, otherwise it would cause unnecessary failure of the application.

WC-bug-id: https://jira.whamcloud.com/browse/LU-5312
Lustre-commit: 07d481957c16832c782bb7d0c1fc436bcb148ea6

Reviewed-on: http://review.whamcloud.com/11039

---
fs/lustre/fld/fld_request.c | 10 ++++------

1 file changed, 4 insertions(+), 6 deletions(-)
fs/lustre/fld/fld_request.c +4 -6
index ba0ef82..60e7105 100644
@@ -367,12 +367,10 @@ int fld_client_rpc(struct obd_export *exp,
   rc = ptlrpc_queue_wait(req);
   obd_put_request_slot(&exp->exp_obd->u.cli);
   if (rc != 0) {
-        if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN) {
-            /* For no_delay req(see above), EWOULDBLOCK and
-             * ESHUTDOWN means the connection is being evicted,
-             * but this seq lookup should not return error,
-             * since it would cause unecessary failure of the
-             * application, instead it should retry here
+        if (imp->imp_state != LUSTRE_IMP_CLOSED) {
+            /* Since LWP is not replayable, so it will keep
+             * trying unless umount happens, otherwise it would
+             * cause unecessary failure of the application.
            */
           ptlrpc_req_finished(req);
           goto again;
Signature · 1 line
1.8.3.1
#13 lore raw

PATCH 6/8 lustre: fld: fld client lookup should retry

In reply to James Simmons

From: wang di <[email protected]>

If FLD client lookup fails because of the remote target is shutdown (or deactive), it should retry another target, otherwise it will cause the application failure.

And FLD client should stop retry if the import has been deactive.

WC-bug-id: https://jira.whamcloud.com/browse/LU-6419
Lustre-commit: 3ededde903c92f8485cae0dc9f958f194ff0b140

Reviewed-on: http://review.whamcloud.com/14313

---
 fs/lustre/fld/fld_request.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
fs/lustre/fld/fld_request.c +21 -2
index 60e7105..dfd4ae9 100644
@@ -367,7 +367,7 @@ int fld_client_rpc(struct obd_export *exp,
 	rc = ptlrpc_queue_wait(req);
 	obd_put_request_slot(&exp->exp_obd->u.cli);
 	if (rc != 0) {
-		if (imp->imp_state != LUSTRE_IMP_CLOSED) {
+		if (imp->imp_state != LUSTRE_IMP_CLOSED && !imp->imp_deactive) {
 			/* Since LWP is not replayable, so it will keep
 			 * trying unless umount happens, otherwise it would
 			 * cause unecessary failure of the application.
@@ -404,6 +404,7 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds,
 {
 	struct lu_seq_range res = { 0 };
 	struct lu_fld_target *target;
+	struct lu_fld_target *origin;
 	int rc;
 
 	rc = fld_cache_lookup(fld->lcf_cache, seq, &res);
@@ -415,7 +416,8 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds,
 	/* Can not find it in the cache */
 	target = fld_client_get_target(fld, seq);
 	LASSERT(target);
-
+	origin = target;
+again:
 	CDEBUG(D_INFO,
 	       "%s: Lookup fld entry (seq: %#llx) on target %s (idx %llu)\n",
 	       fld->lcf_name, seq, fld_target_name(target), target->ft_idx);
@@ -424,6 +426,23 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds,
 	fld_range_set_type(&res, flags);
 	rc = fld_client_rpc(target->ft_exp, &res, FLD_QUERY, NULL);
 
+	if (rc == -ESHUTDOWN) {
+		/* If fld lookup failed because the target has been shutdown,
+		 * then try next target in the list, until trying all targets
+		 * or fld lookup succeeds
+		 */
+		spin_lock(&fld->lcf_lock);
+		if (target->ft_chain.next == fld->lcf_targets.prev)
+			target = list_entry(fld->lcf_targets.next,
+					    struct lu_fld_target, ft_chain);
+		else
+			target = list_entry(target->ft_chain.next,
+						 struct lu_fld_target,
+						 ft_chain);
+		spin_unlock(&fld->lcf_lock);
+		if (target != origin)
+			goto again;
+	}
 	if (rc == 0) {
 		*mds = res.lsr_index;
 
Signature · 1 line
1.8.3.1
#14 lore raw

PATCH 6/8 lustre: fld: fld client lookup should retry

This is definitely client code.

Cheers, Andreas

Quoted 81 lines · James Simmons

From: wang di <[email protected]>

If FLD client lookup fails because of the remote target is shutdown (or deactive), it should retry another target, otherwise it will cause the application failure.

And FLD client should stop retry if the import has been deactive.

WC-bug-id: https://jira.whamcloud.com/browse/LU-6419
Lustre-commit: 3ededde903c92f8485cae0dc9f958f194ff0b140

Reviewed-on: http://review.whamcloud.com/14313

---
fs/lustre/fld/fld_request.c | 23 +++++++++++++++++++++--

1 file changed, 21 insertions(+), 2 deletions(-)
fs/lustre/fld/fld_request.c +21 -2
index 60e7105..dfd4ae9 100644
@@ -367,7 +367,7 @@ int fld_client_rpc(struct obd_export *exp,
   rc = ptlrpc_queue_wait(req);
   obd_put_request_slot(&exp->exp_obd->u.cli);
   if (rc != 0) {
-        if (imp->imp_state != LUSTRE_IMP_CLOSED) {
+        if (imp->imp_state != LUSTRE_IMP_CLOSED && !imp->imp_deactive) {
           /* Since LWP is not replayable, so it will keep
            * trying unless umount happens, otherwise it would
            * cause unecessary failure of the application.
@@ -404,6 +404,7 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds,
 {
   struct lu_seq_range res = { 0 };
   struct lu_fld_target *target;
+    struct lu_fld_target *origin;
   int rc;
 
   rc = fld_cache_lookup(fld->lcf_cache, seq, &res);
@@ -415,7 +416,8 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds,
   /* Can not find it in the cache */
   target = fld_client_get_target(fld, seq);
   LASSERT(target);
-
+    origin = target;
+again:
   CDEBUG(D_INFO,
          "%s: Lookup fld entry (seq: %#llx) on target %s (idx %llu)\n",
          fld->lcf_name, seq, fld_target_name(target), target->ft_idx);
@@ -424,6 +426,23 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds,
   fld_range_set_type(&res, flags);
   rc = fld_client_rpc(target->ft_exp, &res, FLD_QUERY, NULL);
 
+    if (rc == -ESHUTDOWN) {
+        /* If fld lookup failed because the target has been shutdown,
+         * then try next target in the list, until trying all targets
+         * or fld lookup succeeds
+         */
+        spin_lock(&fld->lcf_lock);
+        if (target->ft_chain.next == fld->lcf_targets.prev)
+            target = list_entry(fld->lcf_targets.next,
+                        struct lu_fld_target, ft_chain);
+        else
+            target = list_entry(target->ft_chain.next,
+                         struct lu_fld_target,
+                         ft_chain);
+        spin_unlock(&fld->lcf_lock);
+        if (target != origin)
+            goto again;
+    }
   if (rc == 0) {
       *mds = res.lsr_index;
 
Signature · 1 line
1.8.3.1
#15 lore raw

PATCH 7/8 lustre: tests: testcases for multiple modify RPCs feature

In reply to James Simmons

From: Gregoire Pichon <[email protected]>

This patch creates new testcases in the Auster test suite to test the support of multiple modify RPCs in flight feature.

Two new OBD_FAIL codes are added to allow several failure occurences of the reint requests or replies. This is needed because the current fail checks impose the OBD_FAIL_ONCE flag.

Added testcases are :

- sanity         245
- conf-sanity    90a 90b 90c 90d
- replay-single  102a 102b 102c 102d

This patch also removes replay-single tests 53a and 53b from the ALWAYS_EXCEPT list.

WC-bug-id: https://jira.whamcloud.com/browse/LU-5319
Lustre-commit: c2d27a0f12688c0d029880919f8b002e557b540c

Reviewed-on: http://review.whamcloud.com/14861

---
 fs/lustre/include/obd_support.h | 2 ++
 fs/lustre/ldlm/ldlm_lib.c       | 3 +++
 2 files changed, 5 insertions(+)
fs/lustre/include/obd_support.h +2 -0
index 3e15cac..7999ac6 100644
@@ -189,6 +189,8 @@
 #define OBD_FAIL_MDS_SWAP_LAYOUTS_NET			0x14f
 #define OBD_FAIL_MDS_HSM_ACTION_NET			0x150
 #define OBD_FAIL_MDS_CHANGELOG_INIT			0x151
+#define OBD_FAIL_MDS_REINT_MULTI_NET			0x159
+#define OBD_FAIL_MDS_REINT_MULTI_NET_REP		0x15a
 
 /* layout lock */
 #define OBD_FAIL_MDS_NO_LL_GETATTR			0x170
fs/lustre/ldlm/ldlm_lib.c +3 -0
index 887507d..a3b8df4 100644
@@ -689,6 +689,9 @@ int target_pack_pool_reply(struct ptlrpc_request *req)
 		DEBUG_REQ(D_ERROR, req, "dropping reply");
 		return -ECOMM;
 	}
+	if (unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT &&
+		     OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP)))
+		return -ECOMM;
 
 	if (unlikely(rc)) {
 		DEBUG_REQ(D_NET, req, "processing error (%d)", rc);
Signature · 1 line
1.8.3.1
#16 lore raw

PATCH 7/8 lustre: tests: testcases for multiple modify RPCs feature

This is needed on the client for the test cases to work.

Cheers, Andreas

Quoted 60 lines · James Simmons

From: Gregoire Pichon <[email protected]>

This patch creates new testcases in the Auster test suite to test the support of multiple modify RPCs in flight feature.

Two new OBD_FAIL codes are added to allow several failure occurences of the reint requests or replies. This is needed because the current fail checks impose the OBD_FAIL_ONCE flag.

Added testcases are :

- sanity         245
- conf-sanity    90a 90b 90c 90d
- replay-single  102a 102b 102c 102d

This patch also removes replay-single tests 53a and 53b from the ALWAYS_EXCEPT list.

WC-bug-id: https://jira.whamcloud.com/browse/LU-5319
Lustre-commit: c2d27a0f12688c0d029880919f8b002e557b540c

Reviewed-on: http://review.whamcloud.com/14861

---
fs/lustre/include/obd_support.h | 2 ++
fs/lustre/ldlm/ldlm_lib.c | 3 +++

2 files changed, 5 insertions(+)
fs/lustre/include/obd_support.h +2 -0
index 3e15cac..7999ac6 100644
@@ -189,6 +189,8 @@
 #define OBD_FAIL_MDS_SWAP_LAYOUTS_NET            0x14f
 #define OBD_FAIL_MDS_HSM_ACTION_NET            0x150
 #define OBD_FAIL_MDS_CHANGELOG_INIT            0x151
+#define OBD_FAIL_MDS_REINT_MULTI_NET            0x159
+#define OBD_FAIL_MDS_REINT_MULTI_NET_REP        0x15a
 
 /* layout lock */
 #define OBD_FAIL_MDS_NO_LL_GETATTR            0x170
fs/lustre/ldlm/ldlm_lib.c +3 -0
index 887507d..a3b8df4 100644
@@ -689,6 +689,9 @@ int target_pack_pool_reply(struct ptlrpc_request *req)
       DEBUG_REQ(D_ERROR, req, "dropping reply");
       return -ECOMM;
   }
+    if (unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT &&
+             OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP)))
+        return -ECOMM;
 
   if (unlikely(rc)) {
       DEBUG_REQ(D_NET, req, "processing error (%d)", rc);
Signature · 1 line
1.8.3.1
#17 lore raw

PATCH 8/8 lustre: ldlm: Don't check opcode with NULL rq_reqmsg

In reply to James Simmons

From: Jeremy Filizetti <[email protected]>

When GSS is enabled it's possible to have a NULL rq_reqmsg if a bad signature or no context is returned during the unwrap of the request. Don't check the opcode in this case.

WC-bug-id: https://jira.whamcloud.com/browse/LU-7508
Lustre-commit: 3f4572caef5f25f4a9b5347b2ccf933fdad9db9c

Reviewed-on: http://review.whamcloud.com/17414

---
 fs/lustre/ldlm/ldlm_lib.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
fs/lustre/ldlm/ldlm_lib.c +6 -2
index a3b8df4..04982b8 100644
@@ -689,8 +689,12 @@ int target_pack_pool_reply(struct ptlrpc_request *req)
 		DEBUG_REQ(D_ERROR, req, "dropping reply");
 		return -ECOMM;
 	}
-	if (unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT &&
-		     OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP)))
+	/* We can have a null rq_reqmsg in the event of bad signature or
+	 * no context when unwrapping
+	 */
+	if (req->rq_reqmsg &&
+	    unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT &&
+	    OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP)))
 		return -ECOMM;
 
 	if (unlikely(rc)) {
Signature · 1 line
1.8.3.1
#18 lore raw

PATCH 8/8 lustre: ldlm: Don't check opcode with NULL rq_reqmsg

The ldlm code on the client acts as a "server" for some RPCs from the OSS and MDS, namely lock cancellation and granting callbacks (ASTs).

So even if some of this looks like server code it is also used on the client.

Cheers, Andreas

Quoted 41 lines · James Simmons

From: Jeremy Filizetti <[email protected]>

When GSS is enabled it's possible to have a NULL rq_reqmsg if a bad signature or no context is returned during the unwrap of the request. Don't check the opcode in this case.

WC-bug-id: https://jira.whamcloud.com/browse/LU-7508
Lustre-commit: 3f4572caef5f25f4a9b5347b2ccf933fdad9db9c

Reviewed-on: http://review.whamcloud.com/17414

---
fs/lustre/ldlm/ldlm_lib.c | 8 ++++++--

1 file changed, 6 insertions(+), 2 deletions(-)
fs/lustre/ldlm/ldlm_lib.c +6 -2
index a3b8df4..04982b8 100644
@@ -689,8 +689,12 @@ int target_pack_pool_reply(struct ptlrpc_request *req)
       DEBUG_REQ(D_ERROR, req, "dropping reply");
       return -ECOMM;
   }
-    if (unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT &&
-             OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP)))
+    /* We can have a null rq_reqmsg in the event of bad signature or
+     * no context when unwrapping
+     */
+    if (req->rq_reqmsg &&
+        unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT &&
+        OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP)))
       return -ECOMM;
 
   if (unlikely(rc)) {
Signature · 1 line
1.8.3.1