Skip to contentSkip to conversation

NFSD: Improve fault injection

PATCH Superseded [email protected] 2 participants 14 messages Sep 17, 2012 – Oct 3, 2012

Subsystems (2)
NFS client NFS server

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

[email protected]

author
#0 lore raw

cover NFSD: Improve fault injection

From: Bryan Schumaker <[email protected]>

While working on p2p-nfs, I discovered that I sometimes need to clear state for a specific client to test all possible error recovery conditions. The current fault injection code deletes state as it find it, so it can be difficult to guess which state will be forgotten. In addition, I currently print out the amount of state forgotten but I don't give details like "Forgot 3 locks from client w.x.y.z". These patches set out to improve that.

The first 7 patches clean up the current code and prepare it for specific client state removal. Patch 8 adds prints information to the server's logs when a fault injection file is read (such as "Client w.x.y.z has 3 open files"). Patch 9 adds in a custom file operations structure so users can write strings to fault injection files in addition to u64s. Finally, patch 10 allows users to remove state by writing a client's IP address to one of the files.

Questions, comments and suggestions are appreciated!

- Bryan

Bryan Schumaker (10):
NFSD: Fold fault_inject.h into state.h
NFSD: Lock state before calling fault injection function
NFSD: Clean up forgetting clients
NFSD: Clean up forgetting locks
NFSD: Clean up forgetting openowners
NFSD: Clean up forgetting and recalling delegations
NFSD: Fault injection operations take a per-client forget function
NFSD: Reading a fault injection file prints a state count
NFSD: Add a custom file operations structure for fault injection
NFSD: Forget state for a specific client

 fs/nfsd/fault_inject.c | 108 ++++++++++++++++++++++++-----
 fs/nfsd/fault_inject.h |  28 --------
 fs/nfsd/nfs4state.c    | 182 +++++++++++++++++++++++++++++++------------------
 fs/nfsd/nfsctl.c       |   2 +-
 fs/nfsd/state.h        |  23 +++++++
 5 files changed, 230 insertions(+), 113 deletions(-)
 delete mode 100644 fs/nfsd/fault_inject.h
Signature · 1 line
1.7.12
author
#1 lore raw

PATCH 1/10 NFSD: Fold fault_inject.h into state.h

From: Bryan Schumaker <[email protected]>

There were only a small number of functions in this file and since they all affect stored state I think it makes sense to put them in state.h instead. I also dropped most static inline declarations since there are no callers when fault injection is not enabled.

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/fault_inject.c |  2 --
 fs/nfsd/fault_inject.h | 28 ----------------------------
 fs/nfsd/nfs4state.c    |  1 -
 fs/nfsd/nfsctl.c       |  2 +-
 fs/nfsd/state.h        | 15 +++++++++++++++
 5 files changed, 16 insertions(+), 32 deletions(-)
 delete mode 100644 fs/nfsd/fault_inject.h
fs/nfsd/fault_inject.c +0 -2
index e6c3815..ae94da1 100644
@@ -8,9 +8,7 @@
 #include <linux/fs.h>
 #include <linux/debugfs.h>
 #include <linux/module.h>
-
 #include "state.h"
-#include "fault_inject.h"
 
 struct nfsd_fault_inject_op {
 	char *file;
fs/nfsd/fault_inject.h deleted +0 -28
deleted file mode 100644
index 90bd057..0000000
@@ -1,28 +0,0 @@
-/*
- * Copyright (c) 2011 Bryan Schumaker <[email protected]>
- *
- * Function definitions for fault injection
- */
-
-#ifndef LINUX_NFSD_FAULT_INJECT_H
-#define LINUX_NFSD_FAULT_INJECT_H
-
-#ifdef CONFIG_NFSD_FAULT_INJECTION
-int nfsd_fault_inject_init(void);
-void nfsd_fault_inject_cleanup(void);
-void nfsd_forget_clients(u64);
-void nfsd_forget_locks(u64);
-void nfsd_forget_openowners(u64);
-void nfsd_forget_delegations(u64);
-void nfsd_recall_delegations(u64);
-#else /* CONFIG_NFSD_FAULT_INJECTION */
-static inline int nfsd_fault_inject_init(void) { return 0; }
-static inline void nfsd_fault_inject_cleanup(void) {}
-static inline void nfsd_forget_clients(u64 num) {}
-static inline void nfsd_forget_locks(u64 num) {}
-static inline void nfsd_forget_openowners(u64 num) {}
-static inline void nfsd_forget_delegations(u64 num) {}
-static inline void nfsd_recall_delegations(u64 num) {}
-#endif /* CONFIG_NFSD_FAULT_INJECTION */
-
-#endif /* LINUX_NFSD_FAULT_INJECT_H */
fs/nfsd/nfs4state.c +0 -1
index 5122e17..d024ffd 100644
@@ -44,7 +44,6 @@
 #include "xdr4.h"
 #include "vfs.h"
 #include "current_stateid.h"
-#include "fault_inject.h"
 
 #include "netns.h"
fs/nfsd/nfsctl.c +1 -1
index dab350d..7e0d6db 100644
@@ -19,7 +19,7 @@
 #include "idmap.h"
 #include "nfsd.h"
 #include "cache.h"
-#include "fault_inject.h"
+#include "state.h"
 #include "netns.h"
 
 /*
fs/nfsd/state.h +15 -0
index ec513fd..f25fee2 100644
@@ -481,4 +481,19 @@ extern void nfsd4_client_record_create(struct nfs4_client *clp);
 extern void nfsd4_client_record_remove(struct nfs4_client *clp);
 extern int nfsd4_client_record_check(struct nfs4_client *clp);
 extern void nfsd4_record_grace_done(struct net *net, time_t boot_time);
+
+/* nfs fault injection functions */
+#ifdef CONFIG_NFSD_FAULT_INJECTION
+int nfsd_fault_inject_init(void);
+void nfsd_fault_inject_cleanup(void);
+void nfsd_forget_clients(u64);
+void nfsd_forget_locks(u64);
+void nfsd_forget_openowners(u64);
+void nfsd_forget_delegations(u64);
+void nfsd_recall_delegations(u64);
+#else /* CONFIG_NFSD_FAULT_INJECTION */
+static inline int nfsd_fault_inject_init(void) { return 0; }
+static inline void nfsd_fault_inject_cleanup(void) {}
+#endif /* CONFIG_NFSD_FAULT_INJECTION */
+
 #endif   /* NFSD4_STATE_H */
Signature · 1 line
1.7.12
author
#2 lore raw

PATCH 2/10 NFSD: Lock state before calling fault injection function

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

Each function touches state in some way, so getting the lock earlier can help simplify code.

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/fault_inject.c |  2 ++
 fs/nfsd/nfs4state.c    | 18 ++----------------
 2 files changed, 4 insertions(+), 16 deletions(-)
fs/nfsd/fault_inject.c +2 -0
index ae94da1..3dac486 100644
@@ -50,7 +50,9 @@ static int nfsd_inject_set(void *op_ptr, u64 val)
 	else
 		printk(KERN_INFO "NFSD Fault Injection: %s (n = %llu)", op->file, val);
 
+	nfs4_lock_state();
 	op->func(val);
+	nfs4_unlock_state();
 	return 0;
 }
fs/nfsd/nfs4state.c +2 -16
index d024ffd..f7278d7 100644
@@ -4544,14 +4544,12 @@ void nfsd_forget_clients(u64 num)
 	struct nfs4_client *clp, *next;
 	int count = 0;
 
-	nfs4_lock_state();
 	list_for_each_entry_safe(clp, next, &client_lru, cl_lru) {
 		nfsd4_client_record_remove(clp);
 		expire_client(clp);
 		if (++count == num)
 			break;
 	}
-	nfs4_unlock_state();
 
 	printk(KERN_INFO "NFSD: Forgot %d clients", count);
 }

@@ -4586,23 +4584,13 @@ static int nfsd_release_n_owners(u64 num, bool is_open_owner,
 
 void nfsd_forget_locks(u64 num)
 {
-	int count;
-
-	nfs4_lock_state();
-	count = nfsd_release_n_owners(num, false, release_lockowner_sop);
-	nfs4_unlock_state();
-
+	int count = nfsd_release_n_owners(num, false, release_lockowner_sop);
 	printk(KERN_INFO "NFSD: Forgot %d locks", count);
 }
 
 void nfsd_forget_openowners(u64 num)
 {
-	int count;
-
-	nfs4_lock_state();
-	count = nfsd_release_n_owners(num, true, release_openowner_sop);
-	nfs4_unlock_state();
-
+	int count = nfsd_release_n_owners(num, true, release_openowner_sop);
 	printk(KERN_INFO "NFSD: Forgot %d open owners", count);
 }
 

@@ -4635,10 +4623,8 @@ void nfsd_forget_delegations(u64 num)
 	count = nfsd_process_n_delegations(num, &victims);
 	spin_unlock(&recall_lock);
 
-	nfs4_lock_state();
 	list_for_each_entry_safe(dp, dnext, &victims, dl_recall_lru)
 		unhash_delegation(dp);
-	nfs4_unlock_state();
 
 	printk(KERN_INFO "NFSD: Forgot %d delegations", count);
 }
Signature · 1 line
1.7.12
author
#3 lore raw

PATCH 3/10 NFSD: Clean up forgetting clients

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

I added in a generic for-each loop that takes a pass over the client_lru list and calls some function. The next few patches will update other operations to use this function as well. A value of 0 still means "forget everything that is found".

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/nfs4state.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
fs/nfsd/nfs4state.c +18 -6
index f7278d7..6321dc3 100644
@@ -4539,19 +4539,31 @@ nfs4_check_open_reclaim(clientid_t *clid)
 
 #ifdef CONFIG_NFSD_FAULT_INJECTION
 
-void nfsd_forget_clients(u64 num)
+static u64 nfsd_forget_client(struct nfs4_client *clp, u64 max)
+{
+	nfsd4_client_record_remove(clp);
+	expire_client(clp);
+	return 1;
+}
+
+static u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
 {
 	struct nfs4_client *clp, *next;
-	int count = 0;
+	u64 count = 0;
 
 	list_for_each_entry_safe(clp, next, &client_lru, cl_lru) {
-		nfsd4_client_record_remove(clp);
-		expire_client(clp);
-		if (++count == num)
+		count += func(clp, max - count);
+		if ((max != 0) && (count >= max))
 			break;
 	}
 
-	printk(KERN_INFO "NFSD: Forgot %d clients", count);
+	return count;
+}
+
+void nfsd_forget_clients(u64 num)
+{
+	u64 count = nfsd_for_n_state(num, nfsd_forget_client);
+	printk(KERN_INFO "NFSD: Forgot %llu clients", count);
 }
 
 static void release_lockowner_sop(struct nfs4_stateowner *sop)
Signature · 1 line
1.7.12
author
#4 lore raw

PATCH 4/10 NFSD: Clean up forgetting locks

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

I use the new "forget_n_state()" function to iterate through each client first when searching for locks. This may slow down forgetting locks a little bit, but it implements most of the code needed to forget a specified client's locks.

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/nfs4state.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)
fs/nfsd/nfs4state.c +20 -7
index 6321dc3..631c16b 100644
@@ -4546,6 +4546,24 @@ static u64 nfsd_forget_client(struct nfs4_client *clp, u64 max)
 	return 1;
 }
 
+static u64 nfsd_forget_client_locks(struct nfs4_client *clp, u64 max)
+{
+	struct nfs4_stateowner *sop, *next;
+	u64 count = 0;
+	int i;
+
+	for (i = 0; i < OWNER_HASH_SIZE; i++) {
+		list_for_each_entry_safe(sop, next, &ownerstr_hashtbl[i], so_strhash) {
+			if (!sop->so_is_open_owner && (sop->so_client == clp)) {
+				release_lockowner(lockowner(sop));
+				count++;
+			}
+		}
+	}
+
+	return count;
+}
+
 static u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
 {
 	struct nfs4_client *clp, *next;

@@ -4566,11 +4584,6 @@ void nfsd_forget_clients(u64 num)
 	printk(KERN_INFO "NFSD: Forgot %llu clients", count);
 }
 
-static void release_lockowner_sop(struct nfs4_stateowner *sop)
-{
-	release_lockowner(lockowner(sop));
-}
-
 static void release_openowner_sop(struct nfs4_stateowner *sop)
 {
 	release_openowner(openowner(sop));

@@ -4596,8 +4609,8 @@ static int nfsd_release_n_owners(u64 num, bool is_open_owner,
 
 void nfsd_forget_locks(u64 num)
 {
-	int count = nfsd_release_n_owners(num, false, release_lockowner_sop);
-	printk(KERN_INFO "NFSD: Forgot %d locks", count);
+	u64 count = nfsd_for_n_state(num, nfsd_forget_client_locks);
+	printk(KERN_INFO "NFSD: Forgot %llu locks", count);
 }
 
 void nfsd_forget_openowners(u64 num)
Signature · 1 line
1.7.12
author
#5 lore raw

PATCH 5/10 NFSD: Clean up forgetting openowners

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

Using "forget_n_state()" forces me to implement the code needed to forget a specific client's openowners.

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/nfs4state.c | 41 ++++++++++++++++-------------------------
 1 file changed, 16 insertions(+), 25 deletions(-)
fs/nfsd/nfs4state.c +16 -25
index 631c16b..611a827 100644
@@ -4564,6 +4564,20 @@ static u64 nfsd_forget_client_locks(struct nfs4_client *clp, u64 max)
 	return count;
 }
 
+static u64 nfsd_forget_client_openowners(struct nfs4_client *clp, u64 max)
+{
+	struct nfs4_openowner *oop, *next;
+	u64 count = 0;
+
+	list_for_each_entry_safe(oop, next, &clp->cl_openowners, oo_perclient) {
+		release_openowner(oop);
+		if (++count == max)
+			break;
+	}
+
+	return count;
+}
+
 static u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
 {
 	struct nfs4_client *clp, *next;

@@ -4584,29 +4598,6 @@ void nfsd_forget_clients(u64 num)
 	printk(KERN_INFO "NFSD: Forgot %llu clients", count);
 }
 
-static void release_openowner_sop(struct nfs4_stateowner *sop)
-{
-	release_openowner(openowner(sop));
-}
-
-static int nfsd_release_n_owners(u64 num, bool is_open_owner,
-				void (*release_sop)(struct nfs4_stateowner *))
-{
-	int i, count = 0;
-	struct nfs4_stateowner *sop, *next;
-
-	for (i = 0; i < OWNER_HASH_SIZE; i++) {
-		list_for_each_entry_safe(sop, next, &ownerstr_hashtbl[i], so_strhash) {
-			if (sop->so_is_open_owner != is_open_owner)
-				continue;
-			release_sop(sop);
-			if (++count == num)
-				return count;
-		}
-	}
-	return count;
-}
-
 void nfsd_forget_locks(u64 num)
 {
 	u64 count = nfsd_for_n_state(num, nfsd_forget_client_locks);

@@ -4615,8 +4606,8 @@ void nfsd_forget_locks(u64 num)
 
 void nfsd_forget_openowners(u64 num)
 {
-	int count = nfsd_release_n_owners(num, true, release_openowner_sop);
-	printk(KERN_INFO "NFSD: Forgot %d open owners", count);
+	u64 count = nfsd_for_n_state(num, nfsd_forget_client_openowners);
+	printk(KERN_INFO "NFSD: Forgot %llu open owners", count);
 }
 
 static int nfsd_process_n_delegations(u64 num, struct list_head *list)
Signature · 1 line
1.7.12
author
#6 lore raw

PATCH 6/10 NFSD: Clean up forgetting and recalling delegations

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

Once I have a client, I can easily use its delegation list rather than searching the file hash table for delegations to remove.

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/nfs4state.c | 94 ++++++++++++++++++++++++++++-------------------------
 1 file changed, 50 insertions(+), 44 deletions(-)
fs/nfsd/nfs4state.c +50 -44
index 611a827..c91aae8 100644
@@ -4578,6 +4578,52 @@ static u64 nfsd_forget_client_openowners(struct nfs4_client *clp, u64 max)
 	return count;
 }
 
+static u64 nfsd_find_n_delegations(struct nfs4_client *clp, u64 max,
+				   struct list_head *victims)
+{
+	struct nfs4_delegation *dp, *next;
+	u64 count = 0;
+
+	list_for_each_entry_safe(dp, next, &clp->cl_delegations, dl_perclnt) {
+		list_move(&dp->dl_recall_lru, victims);
+		if (++count == max)
+			break;
+	}
+	return count;
+}
+
+static u64 nfsd_forget_client_delegations(struct nfs4_client *clp, u64 max)
+{
+	struct nfs4_delegation *dp, *next;
+	LIST_HEAD(victims);
+	u64 count;
+
+	spin_lock(&recall_lock);
+	count = nfsd_find_n_delegations(clp, max, &victims);
+	spin_unlock(&recall_lock);
+
+	list_for_each_entry_safe(dp, next, &victims, dl_recall_lru)
+		unhash_delegation(dp);
+
+	return count;
+}
+
+static u64 nfsd_recall_client_delegations(struct nfs4_client *clp, u64 max)
+{
+	struct nfs4_delegation *dp, *next;
+	LIST_HEAD(victims);
+	u64 count;
+
+	spin_lock(&recall_lock);
+
+	count = nfsd_find_n_delegations(clp, max, &victims);
+	list_for_each_entry_safe(dp, next, &victims, dl_recall_lru)
+		nfsd_break_one_deleg(dp);
+
+	spin_unlock(&recall_lock);
+	return count;
+}
+
 static u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
 {
 	struct nfs4_client *clp, *next;

@@ -4610,56 +4656,16 @@ void nfsd_forget_openowners(u64 num)
 	printk(KERN_INFO "NFSD: Forgot %llu open owners", count);
 }
 
-static int nfsd_process_n_delegations(u64 num, struct list_head *list)
-{
-	int i, count = 0;
-	struct nfs4_file *fp, *fnext;
-	struct nfs4_delegation *dp, *dnext;
-
-	for (i = 0; i < FILE_HASH_SIZE; i++) {
-		list_for_each_entry_safe(fp, fnext, &file_hashtbl[i], fi_hash) {
-			list_for_each_entry_safe(dp, dnext, &fp->fi_delegations, dl_perfile) {
-				list_move(&dp->dl_recall_lru, list);
-				if (++count == num)
-					return count;
-			}
-		}
-	}
-
-	return count;
-}
-
 void nfsd_forget_delegations(u64 num)
 {
-	unsigned int count;
-	LIST_HEAD(victims);
-	struct nfs4_delegation *dp, *dnext;
-
-	spin_lock(&recall_lock);
-	count = nfsd_process_n_delegations(num, &victims);
-	spin_unlock(&recall_lock);
-
-	list_for_each_entry_safe(dp, dnext, &victims, dl_recall_lru)
-		unhash_delegation(dp);
-
-	printk(KERN_INFO "NFSD: Forgot %d delegations", count);
+	u64 count = nfsd_for_n_state(num, nfsd_forget_client_delegations);
+	printk(KERN_INFO "NFSD: Forgot %llu delegations", count);
 }
 
 void nfsd_recall_delegations(u64 num)
 {
-	unsigned int count;
-	LIST_HEAD(victims);
-	struct nfs4_delegation *dp, *dnext;
-
-	spin_lock(&recall_lock);
-	count = nfsd_process_n_delegations(num, &victims);
-	list_for_each_entry_safe(dp, dnext, &victims, dl_recall_lru) {
-		list_del(&dp->dl_recall_lru);
-		nfsd_break_one_deleg(dp);
-	}
-	spin_unlock(&recall_lock);
-
-	printk(KERN_INFO "NFSD: Recalled %d delegations", count);
+	u64 count = nfsd_for_n_state(num, nfsd_recall_client_delegations);
+	printk(KERN_INFO "NFSD: Recalled %llu delegations", count);
 }
 
 #endif /* CONFIG_NFSD_FAULT_INJECTION */
Signature · 1 line
1.7.12
author
#7 lore raw

PATCH 7/10 NFSD: Fault injection operations take a per-client forget function

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

The eventual goal is to forget state based on ip address, so it makes sense to call this function in a for-each-client loop until the correct amount of state is forgotten. I also use this patch as an opportunity to rename the forget function from "func()" to "forget()".

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/fault_inject.c | 16 +++++++++-------
 fs/nfsd/nfs4state.c    | 42 ++++++------------------------------------
 fs/nfsd/state.h        | 12 +++++++-----
 3 files changed, 22 insertions(+), 48 deletions(-)
fs/nfsd/fault_inject.c +9 -7
index 3dac486..09feb80 100644
@@ -12,29 +12,29 @@
 
 struct nfsd_fault_inject_op {
 	char *file;
-	void (*func)(u64);
+	u64 (*forget)(struct nfs4_client *, u64);
 };
 
 static struct nfsd_fault_inject_op inject_ops[] = {
 	{
 		.file   = "forget_clients",
-		.func   = nfsd_forget_clients,
+		.forget = nfsd_forget_client,
 	},
 	{
 		.file   = "forget_locks",
-		.func   = nfsd_forget_locks,
+		.forget = nfsd_forget_client_locks,
 	},
 	{
 		.file   = "forget_openowners",
-		.func   = nfsd_forget_openowners,
+		.forget = nfsd_forget_client_openowners,
 	},
 	{
 		.file   = "forget_delegations",
-		.func   = nfsd_forget_delegations,
+		.forget = nfsd_forget_client_delegations,
 	},
 	{
 		.file   = "recall_delegations",
-		.func   = nfsd_recall_delegations,
+		.forget = nfsd_recall_client_delegations,
 	},
 };
 

@@ -43,6 +43,7 @@ static struct dentry *debug_dir;
 
 static int nfsd_inject_set(void *op_ptr, u64 val)
 {
+	u64 count = 0;
 	struct nfsd_fault_inject_op *op = op_ptr;
 
 	if (val == 0)

@@ -51,8 +52,9 @@ static int nfsd_inject_set(void *op_ptr, u64 val)
 		printk(KERN_INFO "NFSD Fault Injection: %s (n = %llu)", op->file, val);
 
 	nfs4_lock_state();
-	op->func(val);
+	count = nfsd_for_n_state(val, op->forget);
 	nfs4_unlock_state();
+	printk(KERN_INFO "NFSD: %s: found %llu", op->file, count);
 	return 0;
 }
fs/nfsd/nfs4state.c +6 -36

84 lines not on this page · raw message

fs/nfsd/state.h +7 -5
index f25fee2..5628a33 100644
@@ -486,11 +486,13 @@ extern void nfsd4_record_grace_done(struct net *net, time_t boot_time);
 #ifdef CONFIG_NFSD_FAULT_INJECTION
 int nfsd_fault_inject_init(void);
 void nfsd_fault_inject_cleanup(void);
-void nfsd_forget_clients(u64);
-void nfsd_forget_locks(u64);
-void nfsd_forget_openowners(u64);
-void nfsd_forget_delegations(u64);
-void nfsd_recall_delegations(u64);
+u64 nfsd_for_n_state(u64, u64 (*)(struct nfs4_client *, u64));
+
+u64 nfsd_forget_client(struct nfs4_client *, u64);
+u64 nfsd_forget_client_locks(struct nfs4_client*, u64);
+u64 nfsd_forget_client_openowners(struct nfs4_client *, u64);
+u64 nfsd_forget_client_delegations(struct nfs4_client *, u64);
+u64 nfsd_recall_client_delegations(struct nfs4_client *, u64);
 #else /* CONFIG_NFSD_FAULT_INJECTION */
 static inline int nfsd_fault_inject_init(void) { return 0; }
 static inline void nfsd_fault_inject_cleanup(void) {}
Signature · 1 line
1.7.12
author
#8 lore raw

PATCH 8/10 NFSD: Reading a fault injection file prints a state count

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

I also log basic information that I can figure out about the type of state (such as number of locks for each client IP address). This can be useful for checking that state was actually dropped and later for checking if the client was able to recover.

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/fault_inject.c | 13 +++++++++--
 fs/nfsd/nfs4state.c    | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/state.h        |  5 +++++
 3 files changed, 74 insertions(+), 2 deletions(-)
fs/nfsd/fault_inject.c +11 -2
index 09feb80..1fdf535 100644
@@ -13,28 +13,34 @@
 struct nfsd_fault_inject_op {
 	char *file;
 	u64 (*forget)(struct nfs4_client *, u64);
+	u64 (*print)(struct nfs4_client *, u64);
 };
 
 static struct nfsd_fault_inject_op inject_ops[] = {
 	{
 		.file   = "forget_clients",
 		.forget = nfsd_forget_client,
+		.print  = nfsd_print_client,
 	},
 	{
 		.file   = "forget_locks",
 		.forget = nfsd_forget_client_locks,
+		.print  = nfsd_print_client_locks,
 	},
 	{
 		.file   = "forget_openowners",
 		.forget = nfsd_forget_client_openowners,
+		.print  = nfsd_print_client_openowners,
 	},
 	{
 		.file   = "forget_delegations",
 		.forget = nfsd_forget_client_delegations,
+		.print  = nfsd_print_client_delegations,
 	},
 	{
 		.file   = "recall_delegations",
 		.forget = nfsd_recall_client_delegations,
+		.print  = nfsd_print_client_delegations,
 	},
 };
 

@@ -58,9 +64,12 @@ static int nfsd_inject_set(void *op_ptr, u64 val)
 	return 0;
 }
 
-static int nfsd_inject_get(void *data, u64 *val)
+static int nfsd_inject_get(void *op_ptr, u64 *val)
 {
-	*val = 0;
+	struct nfsd_fault_inject_op *op = op_ptr;
+	nfs4_lock_state();
+	*val = nfsd_for_n_state(0, op->print);
+	nfs4_unlock_state();
 	return 0;
 }
fs/nfsd/nfs4state.c +58 -0

82 lines not on this page · raw message

fs/nfsd/state.h +5 -0
index 5628a33..881781a 100644
@@ -493,6 +493,11 @@ u64 nfsd_forget_client_locks(struct nfs4_client*, u64);
 u64 nfsd_forget_client_openowners(struct nfs4_client *, u64);
 u64 nfsd_forget_client_delegations(struct nfs4_client *, u64);
 u64 nfsd_recall_client_delegations(struct nfs4_client *, u64);
+
+u64 nfsd_print_client(struct nfs4_client *, u64);
+u64 nfsd_print_client_locks(struct nfs4_client *, u64);
+u64 nfsd_print_client_openowners(struct nfs4_client *, u64);
+u64 nfsd_print_client_delegations(struct nfs4_client *, u64);
 #else /* CONFIG_NFSD_FAULT_INJECTION */
 static inline int nfsd_fault_inject_init(void) { return 0; }
 static inline void nfsd_fault_inject_cleanup(void) {}
Signature · 1 line
1.7.12
author
#9 lore raw

PATCH 9/10 NFSD: Add a custom file operations structure for fault injection

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

Controlling the read and write functions allows me to add in "forget client w.x.y.z", since we won't be limited to reading and writing only u64 values.

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/fault_inject.c | 56 +++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 49 insertions(+), 7 deletions(-)
fs/nfsd/fault_inject.c +49 -7
index 1fdf535..19b3348 100644
@@ -8,6 +8,7 @@
 #include <linux/fs.h>
 #include <linux/debugfs.h>
 #include <linux/module.h>
+#include <asm/uaccess.h>
 #include "state.h"
 
 struct nfsd_fault_inject_op {

@@ -47,10 +48,9 @@ static struct nfsd_fault_inject_op inject_ops[] = {
 static long int NUM_INJECT_OPS = sizeof(inject_ops) / sizeof(struct nfsd_fault_inject_op);
 static struct dentry *debug_dir;
 
-static int nfsd_inject_set(void *op_ptr, u64 val)
+static void nfsd_inject_set(struct nfsd_fault_inject_op *op, u64 val)
 {
 	u64 count = 0;
-	struct nfsd_fault_inject_op *op = op_ptr;
 
 	if (val == 0)
 		printk(KERN_INFO "NFSD Fault Injection: %s (all)", op->file);

@@ -61,19 +61,61 @@ static int nfsd_inject_set(void *op_ptr, u64 val)
 	count = nfsd_for_n_state(val, op->forget);
 	nfs4_unlock_state();
 	printk(KERN_INFO "NFSD: %s: found %llu", op->file, count);
-	return 0;
 }
 
-static int nfsd_inject_get(void *op_ptr, u64 *val)
+static void nfsd_inject_get(struct nfsd_fault_inject_op *op, u64 *val)
 {
-	struct nfsd_fault_inject_op *op = op_ptr;
 	nfs4_lock_state();
 	*val = nfsd_for_n_state(0, op->print);
 	nfs4_unlock_state();
-	return 0;
 }
 
-DEFINE_SIMPLE_ATTRIBUTE(fops_nfsd, nfsd_inject_get, nfsd_inject_set, "%llu\n");
+static ssize_t fault_inject_read(struct file *file, char __user *buf,
+				 size_t len, loff_t *ppos)
+{
+	static u64 val;
+	char read_buf[25];
+	size_t size, ret;
+	loff_t pos = *ppos;
+
+	if (!pos)
+		nfsd_inject_get(file->f_dentry->d_inode->i_private, &val);
+	size = scnprintf(read_buf, sizeof(read_buf), "%llu\n", val);
+
+	if (pos < 0)
+		return -EINVAL;
+	if (pos >= size || !len)
+		return 0;
+	if (len > size - pos)
+		len = size - pos;
+	ret = copy_to_user(buf, read_buf + pos, len);
+	if (ret == len)
+		return -EFAULT;
+	len -= ret;
+	*ppos = pos + len;
+	return len;
+}
+
+static ssize_t fault_inject_write(struct file *file, const char __user *buf,
+				  size_t len, loff_t *ppos)
+{
+	char write_buf[24];
+	size_t size = min(sizeof(write_buf), len) - 1;
+	u64 val;
+
+	if (copy_from_user(write_buf, buf, size))
+		return -EFAULT;
+
+	val = simple_strtoll(write_buf, NULL, 0);
+	nfsd_inject_set(file->f_dentry->d_inode->i_private, val);
+	return len; /* on success, claim we got the whole input */
+}
+
+static const struct file_operations fops_nfsd = {
+	.owner   = THIS_MODULE,
+	.read    = fault_inject_read,
+	.write   = fault_inject_write,
+};
 
 void nfsd_fault_inject_cleanup(void)
 {
Signature · 1 line
1.7.12
author
#10 lore raw

PATCH 10/10 NFSD: Forget state for a specific client

In reply to [email protected]

From: Bryan Schumaker <[email protected]>

Write the client's ip address to any state file and all appropriate state for that client will be forgotten.

Signed-off-by Bryan Schumaker <[email protected]>
---
 fs/nfsd/fault_inject.c | 31 +++++++++++++++++++++++++++----
 fs/nfsd/nfs4state.c    | 11 +++++++++++
 fs/nfsd/state.h        |  1 +
 3 files changed, 39 insertions(+), 4 deletions(-)
fs/nfsd/fault_inject.c +27 -4
index 19b3348..bb6b487 100644
@@ -8,6 +8,8 @@
 #include <linux/fs.h>
 #include <linux/debugfs.h>
 #include <linux/module.h>
+#include <linux/nsproxy.h>
+#include <linux/sunrpc/clnt.h>
 #include <asm/uaccess.h>
 #include "state.h"
 

@@ -63,6 +65,19 @@ static void nfsd_inject_set(struct nfsd_fault_inject_op *op, u64 val)
 	printk(KERN_INFO "NFSD: %s: found %llu", op->file, count);
 }
 
+static void nfsd_inject_set_client(struct nfsd_fault_inject_op *op,
+				   struct sockaddr_storage *addr,
+				   size_t addr_size)
+{
+	struct nfs4_client *clp;
+
+	nfs4_lock_state();
+	clp = nfsd_find_client(addr, addr_size);
+	if (clp)
+		op->forget(clp, 0);
+	nfs4_unlock_state();
+}
+
 static void nfsd_inject_get(struct nfsd_fault_inject_op *op, u64 *val)
 {
 	nfs4_lock_state();

@@ -99,15 +114,23 @@ static ssize_t fault_inject_read(struct file *file, char __user *buf,
 static ssize_t fault_inject_write(struct file *file, const char __user *buf,
 				  size_t len, loff_t *ppos)
 {
-	char write_buf[24];
+	char write_buf[INET6_ADDRSTRLEN];
 	size_t size = min(sizeof(write_buf), len) - 1;
+	struct net *net = current->nsproxy->net_ns;
+	struct sockaddr_storage sa;
 	u64 val;
 
 	if (copy_from_user(write_buf, buf, size))
 		return -EFAULT;
-
-	val = simple_strtoll(write_buf, NULL, 0);
-	nfsd_inject_set(file->f_dentry->d_inode->i_private, val);
+	write_buf[size] = '\0';
+
+	size = rpc_pton(net, write_buf, size, (struct sockaddr *)&sa, sizeof(sa));
+	if (size > 0)
+		nfsd_inject_set_client(file->f_dentry->d_inode->i_private, &sa, size);
+	else {
+		val = simple_strtoll(write_buf, NULL, 0);
+		nfsd_inject_set(file->f_dentry->d_inode->i_private, val);
+	}
 	return len; /* on success, claim we got the whole input */
 }
fs/nfsd/nfs4state.c +11 -0
index a5a3b46d..9dd145e 100644
@@ -4696,6 +4696,17 @@ u64 nfsd_for_n_state(u64 max, u64 (*func)(struct nfs4_client *, u64))
 	return count;
 }
 
+struct nfs4_client *nfsd_find_client(struct sockaddr_storage *addr, size_t addr_size)
+{
+	struct nfs4_client *clp;
+
+	list_for_each_entry(clp, &client_lru, cl_lru) {
+		if (memcmp(&clp->cl_addr, addr, addr_size) == 0)
+			return clp;
+	}
+	return NULL;
+}
+
 #endif /* CONFIG_NFSD_FAULT_INJECTION */
 
 /* initialization to perform at module load time: */
fs/nfsd/state.h +1 -0
index 881781a..17435d9 100644
@@ -487,6 +487,7 @@ extern void nfsd4_record_grace_done(struct net *net, time_t boot_time);
 int nfsd_fault_inject_init(void);
 void nfsd_fault_inject_cleanup(void);
 u64 nfsd_for_n_state(u64, u64 (*)(struct nfs4_client *, u64));
+struct nfs4_client *nfsd_find_client(struct sockaddr_storage *, size_t);
 
 u64 nfsd_forget_client(struct nfs4_client *, u64);
 u64 nfsd_forget_client_locks(struct nfs4_client*, u64);
Signature · 1 line
1.7.12
author
#11 lore raw
In reply to [email protected]

Bruce, have you had a chance to look at these yet? I don't see them in your git tree...

- Bryan

Quoted 42 lines · On 09/17/2012 04:16 PM, [email protected] wrote:

From: Bryan Schumaker <[email protected]>

While working on p2p-nfs, I discovered that I sometimes need to clear state for a specific client to test all possible error recovery conditions. The current fault injection code deletes state as it find it, so it can be difficult to guess which state will be forgotten. In addition, I currently print out the amount of state forgotten but I don't give details like "Forgot 3 locks from client w.x.y.z". These patches set out to improve that.

The first 7 patches clean up the current code and prepare it for specific client state removal. Patch 8 adds prints information to the server's logs when a fault injection file is read (such as "Client w.x.y.z has 3 open files"). Patch 9 adds in a custom file operations structure so users can write strings to fault injection files in addition to u64s. Finally, patch 10 allows users to remove state by writing a client's IP address to one of the files.

Questions, comments and suggestions are appreciated!

- Bryan

Bryan Schumaker (10):
NFSD: Fold fault_inject.h into state.h
NFSD: Lock state before calling fault injection function
NFSD: Clean up forgetting clients
NFSD: Clean up forgetting locks
NFSD: Clean up forgetting openowners
NFSD: Clean up forgetting and recalling delegations
NFSD: Fault injection operations take a per-client forget function
NFSD: Reading a fault injection file prints a state count
NFSD: Add a custom file operations structure for fault injection
NFSD: Forget state for a specific client

 fs/nfsd/fault_inject.c | 108 ++++++++++++++++++++++++-----
 fs/nfsd/fault_inject.h |  28 --------
 fs/nfsd/nfs4state.c    | 182 +++++++++++++++++++++++++++++++------------------
 fs/nfsd/nfsctl.c       |   2 +-
 fs/nfsd/state.h        |  23 +++++++
 5 files changed, 230 insertions(+), 113 deletions(-)
 delete mode 100644 fs/nfsd/fault_inject.h
#12 lore raw
Quoted 1 line · Bryan Schumaker

Bruce, have you had a chance to look at these yet? I don't see them in your git tree...

Sorry, no, it hasn't been a high priority.

I'll try to take a look soon. They may end up being queued up for 3.8.

--b.

Quoted 47 lines

- Bryan

Quoted 42 lines · On 09/17/2012 04:16 PM, [email protected] wrote:

From: Bryan Schumaker <[email protected]>

While working on p2p-nfs, I discovered that I sometimes need to clear state for a specific client to test all possible error recovery conditions. The current fault injection code deletes state as it find it, so it can be difficult to guess which state will be forgotten. In addition, I currently print out the amount of state forgotten but I don't give details like "Forgot 3 locks from client w.x.y.z". These patches set out to improve that.

The first 7 patches clean up the current code and prepare it for specific client state removal. Patch 8 adds prints information to the server's logs when a fault injection file is read (such as "Client w.x.y.z has 3 open files"). Patch 9 adds in a custom file operations structure so users can write strings to fault injection files in addition to u64s. Finally, patch 10 allows users to remove state by writing a client's IP address to one of the files.

Questions, comments and suggestions are appreciated!

- Bryan

Bryan Schumaker (10):
NFSD: Fold fault_inject.h into state.h
NFSD: Lock state before calling fault injection function
NFSD: Clean up forgetting clients
NFSD: Clean up forgetting locks
NFSD: Clean up forgetting openowners
NFSD: Clean up forgetting and recalling delegations
NFSD: Fault injection operations take a per-client forget function
NFSD: Reading a fault injection file prints a state count
NFSD: Add a custom file operations structure for fault injection
NFSD: Forget state for a specific client

 fs/nfsd/fault_inject.c | 108 ++++++++++++++++++++++++-----
 fs/nfsd/fault_inject.h |  28 --------
 fs/nfsd/nfs4state.c    | 182 +++++++++++++++++++++++++++++++------------------
 fs/nfsd/nfsctl.c       |   2 +-
 fs/nfsd/state.h        |  23 +++++++
 5 files changed, 230 insertions(+), 113 deletions(-)
 delete mode 100644 fs/nfsd/fault_inject.h
author
#13 lore raw
Quoted 6 lines · J. Bruce Fields
Quoted 1 line · Bryan Schumaker

Bruce, have you had a chance to look at these yet? I don't see them in your git tree...

Sorry, no, it hasn't been a high priority.

I'll try to take a look soon. They may end up being queued up for 3.8.

Sure, I was just making sure they don't get lost :).

- Bryan

Quoted 50 lines

--b.

Quoted 47 lines

- Bryan

Quoted 42 lines · On 09/17/2012 04:16 PM, [email protected] wrote:

From: Bryan Schumaker <[email protected]>

While working on p2p-nfs, I discovered that I sometimes need to clear state for a specific client to test all possible error recovery conditions. The current fault injection code deletes state as it find it, so it can be difficult to guess which state will be forgotten. In addition, I currently print out the amount of state forgotten but I don't give details like "Forgot 3 locks from client w.x.y.z". These patches set out to improve that.

The first 7 patches clean up the current code and prepare it for specific client state removal. Patch 8 adds prints information to the server's logs when a fault injection file is read (such as "Client w.x.y.z has 3 open files"). Patch 9 adds in a custom file operations structure so users can write strings to fault injection files in addition to u64s. Finally, patch 10 allows users to remove state by writing a client's IP address to one of the files.

Questions, comments and suggestions are appreciated!

- Bryan

Bryan Schumaker (10):
NFSD: Fold fault_inject.h into state.h
NFSD: Lock state before calling fault injection function
NFSD: Clean up forgetting clients
NFSD: Clean up forgetting locks
NFSD: Clean up forgetting openowners
NFSD: Clean up forgetting and recalling delegations
NFSD: Fault injection operations take a per-client forget function
NFSD: Reading a fault injection file prints a state count
NFSD: Add a custom file operations structure for fault injection
NFSD: Forget state for a specific client

 fs/nfsd/fault_inject.c | 108 ++++++++++++++++++++++++-----
 fs/nfsd/fault_inject.h |  28 --------
 fs/nfsd/nfs4state.c    | 182 +++++++++++++++++++++++++++++++------------------
 fs/nfsd/nfsctl.c       |   2 +-
 fs/nfsd/state.h        |  23 +++++++
 5 files changed, 230 insertions(+), 113 deletions(-)
 delete mode 100644 fs/nfsd/fault_inject.h