Skip to contentSkip to conversation

NFSv4 state recovery observability

RFC PATCH New Chuck Lever 1 participant 3 messages Nov 5, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover NFSv4 state recovery observability

Add some trace points to the state manager to help track down common in-the-field failures. We see "Lock reclaim failed!" a lot during testing, for example, and want to enable our Q/A staff to capture the failure completely and quickly.

---

Chuck Lever (2):
NFS4: Trace state recovery operation
NFS4: Trace lock reclaims

 fs/nfs/nfs4_fs.h   |    2 -
 fs/nfs/nfs4state.c |    4 +
 fs/nfs/nfs4trace.h |  171 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 175 insertions(+), 2 deletions(-)

--
Chuck Lever

#1 lore raw

RFC PATCH 1/2 NFS4: Trace state recovery operation

Add a trace point in the main state manager loop to observe state recovery operation. Help track down state recovery bugs.

---
 fs/nfs/nfs4state.c |    3 ++
 fs/nfs/nfs4trace.h |   93 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 96 insertions(+)
fs/nfs/nfs4state.c +3 -0
index 0c6d53dc3672..b69c33c3600c 100644
@@ -60,6 +60,7 @@
 #include "nfs4session.h"
 #include "pnfs.h"
 #include "netns.h"
+#include "nfs4trace.h"
 
 #define NFSDBG_FACILITY		NFSDBG_STATE
 
@@ -2508,6 +2509,7 @@ static void nfs4_state_manager(struct nfs_client *clp)
 
 	/* Ensure exclusive access to NFSv4 state */
 	do {
+		trace_nfs4_state_mgr(clp);
 		clear_bit(NFS4CLNT_RUN_MANAGER, &clp->cl_state);
 		if (test_bit(NFS4CLNT_PURGE_STATE, &clp->cl_state)) {
 			section = "purge state";
@@ -2621,6 +2623,7 @@ static void nfs4_state_manager(struct nfs_client *clp)
 out_error:
 	if (strlen(section))
 		section_sep = ": ";
+	trace_nfs4_state_mgr_failed(clp, section, status);
 	pr_warn_ratelimited("NFS: state manager%s%s failed on NFSv4 server %s"
 			" with error %d\n", section_sep, section,
 			clp->cl_hostname, -status);
fs/nfs/nfs4trace.h +93 -0

99 lines not on this page · raw message

#2 lore raw

RFC PATCH 2/2 NFS4: Trace lock reclaims

In reply to Chuck Lever

One of the most frustrating messages our sustaining team sees is the "Lock reclaim failed!" message. Add some observability in the client's lock reclaim logic so we can capture better data the first time a problem occurs.

---
 fs/nfs/nfs4_fs.h   |    2 -
 fs/nfs/nfs4state.c |    1 +
 fs/nfs/nfs4trace.h |   78 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 79 insertions(+), 2 deletions(-)
fs/nfs/nfs4_fs.h +0 -2
index 16b2e5cc3e94..935bf335a3c2 100644
@@ -166,9 +166,7 @@ enum {
 	NFS_STATE_RECOVERY_FAILED,	/* OPEN stateid state recovery failed */
 	NFS_STATE_MAY_NOTIFY_LOCK,	/* server may CB_NOTIFY_LOCK */
 	NFS_STATE_CHANGE_WAIT,		/* A state changing operation is outstanding */
-#ifdef CONFIG_NFS_V4_2
 	NFS_CLNT_DST_SSC_COPY_STATE,    /* dst server open state on client*/
-#endif /* CONFIG_NFS_V4_2 */
 };
 
 struct nfs4_state {
fs/nfs/nfs4state.c +1 -0
index b69c33c3600c..e72fbc842025 100644
@@ -1594,6 +1594,7 @@ static int __nfs4_reclaim_open_state(struct nfs4_state_owner *sp, struct nfs4_st
 	if (!test_bit(NFS_DELEGATED_STATE, &state->flags)) {
 		spin_lock(&state->state_lock);
 		list_for_each_entry(lock, &state->lock_states, ls_locks) {
+			trace_nfs4_state_lock_reclaim(state, lock);
 			if (!test_bit(NFS_LOCK_INITIALIZED, &lock->ls_flags))
 				pr_warn_ratelimited("NFS: %s: Lock reclaim failed!\n", __func__);
 		}
fs/nfs/nfs4trace.h +78 -0

84 lines not on this page · raw message