Skip to contentSkip to conversation

NFSD duplicate reply cache optimizations

PATCH Superseded Chuck Lever 3 participants 6 messages Aug 20, 2025 – Aug 21, 2025

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Chuck Lever

author maintainer
#0 lore raw

cover NFSD duplicate reply cache optimizations

From: Chuck Lever <[email protected]>

Two minor optimizations for the NFSD duplicate reply cache. These showed slight improvement in WRITE throughput on exported fast file systems.

Chuck Lever (2):
NFSD: Delay adding new entries to LRU
NFSD: Reduce DRC bucket size

 fs/nfsd/nfscache.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
Signature · 1 line
2.50.0
author maintainer
#1 lore raw

PATCH 1/2 NFSD: Delay adding new entries to LRU

From: Chuck Lever <[email protected]>

Neil Brown observes:

Quoted 3 lines

I would not include RC_INPROG entries in the lru at all - they are always ignored, and will be added when they are switched to RCU_DONE.

I also removed a stale comment.

---
 fs/nfsd/nfscache.c | 6 ------
 1 file changed, 6 deletions(-)
fs/nfsd/nfscache.c +0 -6
index ba9d326b3de6..6c06cf24b5c7 100644
@@ -237,10 +237,6 @@ void nfsd_reply_cache_shutdown(struct nfsd_net *nn)
 
 }
 
-/*
- * Move cache entry to end of LRU list, and queue the cleaner to run if it's
- * not already scheduled.
- */
 static void
 lru_put_end(struct nfsd_drc_bucket *b, struct nfsd_cacherep *rp)
 {
@@ -453,8 +449,6 @@ nfsd_cache_insert(struct nfsd_drc_bucket *b, struct nfsd_cacherep *key,
 				nn->longest_chain_cachesize,
 				atomic_read(&nn->num_drc_entries));
 	}
-
-	lru_put_end(b, ret);
 	return ret;
 }
 
Signature · 1 line
2.50.0
reviewer Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH v1 1/2] NFSD: Delay adding new entries to LRU

Quoted 31 lines · Chuck Lever

From: Chuck Lever <[email protected]>

Neil Brown observes:

Quoted 3 lines

I would not include RC_INPROG entries in the lru at all - they are always ignored, and will be added when they are switched to RCU_DONE.

I also removed a stale comment.

---
 fs/nfsd/nfscache.c | 6 ------
 1 file changed, 6 deletions(-)
fs/nfsd/nfscache.c +0 -4
index ba9d326b3de6..6c06cf24b5c7 100644
@@ -237,10 +237,6 @@ void nfsd_reply_cache_shutdown(struct nfsd_net *nn)
 
 }
 
-/*
- * Move cache entry to end of LRU list, and queue the cleaner to run if it's
- * not already scheduled.
- */
 static void
 lru_put_end(struct nfsd_drc_bucket *b, struct nfsd_cacherep *rp)
 {

fs/nfsd/nfscache.c nfsd_cache_insert(struct nfsd_drc_bucket *b, struct nfsd_cacherep *key, · line 452

Quoted 6 lines
fs/nfsd/nfscache.c +0 -2
@@ -453,8 +449,6 @@ nfsd_cache_insert(struct nfsd_drc_bucket *b, struct nfsd_cacherep *key,
 				nn->longest_chain_cachesize,
 				atomic_read(&nn->num_drc_entries));
 	}
-
-	lru_put_end(b, ret);
 	return ret;

A result of this change is that entries in the lru never have

    ->c_state == RC_INPROG

They are always RC_DONE.

So this can be added to the patch:

fs/nfsd/nfscache.c +0 -6
@@ -272,12 +272,6 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
 
 	/* The bucket LRU is ordered oldest-first. */
 	list_for_each_entry_safe(rp, tmp, &b->lru_head, c_lru) {
-		/*
-		 * Don't free entries attached to calls that are still
-		 * in-progress, but do keep scanning the list.
-		 */
-		if (rp->c_state == RC_INPROG)
-			continue;
 
 		if (atomic_read(&nn->num_drc_entries) <= nn->max_drc_entries &&
 		    time_before(expiry, rp->c_timestamp))

With that added:

Thanks,
NeilBrown

Quoted 6 lines
fs/nfsd/nfscache.c +0 -0
 }
 
Signature · 1 line
2.50.0
author maintainer
#3 lore raw

Re: [PATCH v1 1/2] NFSD: Delay adding new entries to LRU

Quoted 50 lines · NeilBrown
Quoted 31 lines · Chuck Lever

From: Chuck Lever <[email protected]>

Neil Brown observes:

Quoted 3 lines

I would not include RC_INPROG entries in the lru at all - they are always ignored, and will be added when they are switched to RCU_DONE.

I also removed a stale comment.

---
 fs/nfsd/nfscache.c | 6 ------
 1 file changed, 6 deletions(-)
fs/nfsd/nfscache.c +0 -4
index ba9d326b3de6..6c06cf24b5c7 100644
@@ -237,10 +237,6 @@ void nfsd_reply_cache_shutdown(struct nfsd_net *nn)
 
 }
 
-/*
- * Move cache entry to end of LRU list, and queue the cleaner to run if it's
- * not already scheduled.
- */
 static void
 lru_put_end(struct nfsd_drc_bucket *b, struct nfsd_cacherep *rp)
 {

fs/nfsd/nfscache.c nfsd_cache_insert(struct nfsd_drc_bucket *b, struct nfsd_cacherep *key, · line 452

Quoted 6 lines
fs/nfsd/nfscache.c +0 -2
@@ -453,8 +449,6 @@ nfsd_cache_insert(struct nfsd_drc_bucket *b, struct nfsd_cacherep *key,
 				nn->longest_chain_cachesize,
 				atomic_read(&nn->num_drc_entries));
 	}
-
-	lru_put_end(b, ret);
 	return ret;

A result of this change is that entries in the lru never have

    ->c_state == RC_INPROG

They are always RC_DONE.

So this can be added to the patch:

fs/nfsd/nfscache.c +0 -0
@@ -272,12 +272,6 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
 
 	/* The bucket LRU is ordered oldest-first. */

fs/nfsd/nfscache.c nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b, · line 277

Quoted 10 lines
fs/nfsd/nfscache.c +0 -6
 	list_for_each_entry_safe(rp, tmp, &b->lru_head, c_lru) {
-		/*
-		 * Don't free entries attached to calls that are still
-		 * in-progress, but do keep scanning the list.
-		 */
-		if (rp->c_state == RC_INPROG)
-			continue;
 
 		if (atomic_read(&nn->num_drc_entries) <= nn->max_drc_entries &&
 		    time_before(expiry, rp->c_timestamp))

I didn't add this bit because I couldn't convince myself that it was impossible for an entry in RC_INPROG state to get into the LRU somehow. But if you think it is not possible, I'll buy that and add this snippet.

Quoted 15 lines

With that added:

Thanks,
NeilBrown

Quoted 6 lines
fs/nfsd/nfscache.c +0 -0
 }
 
Signature · 1 line
2.50.0
Signature · 1 line
Chuck Lever
author maintainer
#4 lore raw

PATCH 2/2 NFSD: Reduce DRC bucket size

In reply to Chuck Lever

From: Chuck Lever <[email protected]>

The common case is that a DRC lookup will not find the XID in the bucket. Reduce the amount of pointer chasing during the lookup by keeping fewer entries in each hash bucket.

Changing the bucket size constant forces the size of the DRC hash table to increase, and the height of each bucket r-b tree to be reduced.

---
 fs/nfsd/nfscache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfsd/nfscache.c +1 -1
index 6c06cf24b5c7..053223e2a4fa 100644
@@ -27,7 +27,7 @@
  * cache size, the idea being that when the cache is at its maximum number
  * of entries, then this should be the average number of entries per bucket.
  */
-#define TARGET_BUCKET_SIZE	64
+#define TARGET_BUCKET_SIZE	8
 
 struct nfsd_drc_bucket {
 	struct rb_root rb_head;
Signature · 1 line
2.50.0
maintainer Reviewed-by on the v1 series
#5 lore raw
In reply to Chuck Lever
Quoted 6 lines · Chuck Lever

From: Chuck Lever <[email protected]>

Two minor optimizations for the NFSD duplicate reply cache. These showed slight improvement in WRITE throughput on exported fast file systems.

Quoted 6 lines

Chuck Lever (2):
NFSD: Delay adding new entries to LRU
NFSD: Reduce DRC bucket size

 fs/nfsd/nfscache.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)