Skip to contentSkip to conversation

sunrpc: raise kernel RPC channel buffer size

PATCH In review Roberto Bergantinos Corpas 3 participants 6 messages Oct 19, 2020 – Oct 24, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sunrpc: raise kernel RPC channel buffer size

Its possible that using AUTH_SYS and mountd manage-gids option a user may hit the 8k RPC channel buffer limit. This have been observed on field, causing unanswered RPCs on clients after mountd fails to write on channel :

rpc.mountd[11231]: auth_unix_gid: error writing reply

Userland nfs-utils uses a buffer size of 32k (RPC_CHAN_BUF_SIZE), so lets match those two.

---
 net/sunrpc/cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/cache.c +1 -1
index baef5ee43dbb..08df4c599ab3 100644
@@ -908,7 +908,7 @@ static ssize_t cache_do_downcall(char *kaddr, const char __user *buf,
 static ssize_t cache_slow_downcall(const char __user *buf,
 				   size_t count, struct cache_detail *cd)
 {
-	static char write_buf[8192]; /* protected by queue_io_mutex */
+	static char write_buf[32768]; /* protected by queue_io_mutex */
 	ssize_t ret = -EINVAL;
 
 	if (count >= sizeof(write_buf))
Signature · 1 line
2.21.0
#1 lore raw
Quoted 9 lines · Roberto Bergantinos Corpas

Its possible that using AUTH_SYS and mountd manage-gids option a user may hit the 8k RPC channel buffer limit. This have been observed on field, causing unanswered RPCs on clients after mountd fails to write on channel :

rpc.mountd[11231]: auth_unix_gid: error writing reply

Userland nfs-utils uses a buffer size of 32k (RPC_CHAN_BUF_SIZE), so lets match those two.

Thanks, applying.

That should allow about 4000 group memberships. If that doesn't do it then maybe it's time to rethink....

--b.

Quoted 21 lines
---
 net/sunrpc/cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/cache.c +1 -1
index baef5ee43dbb..08df4c599ab3 100644
@@ -908,7 +908,7 @@ static ssize_t cache_do_downcall(char *kaddr, const char __user *buf,
 static ssize_t cache_slow_downcall(const char __user *buf,
 				   size_t count, struct cache_detail *cd)
 {
-	static char write_buf[8192]; /* protected by queue_io_mutex */
+	static char write_buf[32768]; /* protected by queue_io_mutex */
 	ssize_t ret = -EINVAL;
 
 	if (count >= sizeof(write_buf))
Signature · 1 line
2.21.0
#2 lore raw

Hi Bruce, Roberto,

Quoted 36 lines · J. Bruce Fields
Quoted 9 lines · Roberto Bergantinos Corpas

Its possible that using AUTH_SYS and mountd manage-gids option a user may hit the 8k RPC channel buffer limit. This have been observed on field, causing unanswered RPCs on clients after mountd fails to write on channel :

rpc.mountd[11231]: auth_unix_gid: error writing reply

Userland nfs-utils uses a buffer size of 32k (RPC_CHAN_BUF_SIZE), so lets match those two.

Thanks, applying.

That should allow about 4000 group memberships. If that doesn't do it then maybe it's time to rethink....

--b.

Quoted 18 lines
---
 net/sunrpc/cache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/cache.c +1 -1
index baef5ee43dbb..08df4c599ab3 100644
@@ -908,7 +908,7 @@ static ssize_t cache_do_downcall(char *kaddr, const char __user *buf,
 static ssize_t cache_slow_downcall(const char __user *buf,
 				   size_t count, struct cache_detail *cd)
 {
-	static char write_buf[8192]; /* protected by queue_io_mutex */
+	static char write_buf[32768]; /* protected by queue_io_mutex */
 	ssize_t ret = -EINVAL;
 
Quoted 1 line

net/sunrpc/cache.c static ssize_t cache_do_downcall(char *kaddr, const char __user *buf, · line 914

Quoted 1 line
net/sunrpc/cache.c +0 -0
 	if (count >= sizeof(write_buf))

This is now commit 27a1e8a0f79e643d ("sunrpc: raise kernel RPC channel buffer size") upstream, and increases kernel size by 24 KiB, even if RPC is not used.

Can this buffer allocated dynamically instead? This code path seems to be a slow path anyway. If it's critical, perhaps this buffer can be allocated on first use?

Thanks!

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

#3 lore raw
Quoted 45 lines · Geert Uytterhoeven

Hi Bruce, Roberto,

Quoted 36 lines · J. Bruce Fields
Quoted 9 lines · Roberto Bergantinos Corpas

Its possible that using AUTH_SYS and mountd manage-gids option a user may hit the 8k RPC channel buffer limit. This have been observed on field, causing unanswered RPCs on clients after mountd fails to write on channel :

rpc.mountd[11231]: auth_unix_gid: error writing reply

Userland nfs-utils uses a buffer size of 32k (RPC_CHAN_BUF_SIZE), so lets match those two.

Thanks, applying.

That should allow about 4000 group memberships. If that doesn't do it then maybe it's time to rethink....

--b.

Quoted 18 lines

---
net/sunrpc/cache.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/cache.c +1 -1
index baef5ee43dbb..08df4c599ab3 100644
@@ -908,7 +908,7 @@ static ssize_t cache_do_downcall(char *kaddr, const char __user *buf,
 static ssize_t cache_slow_downcall(const char __user *buf,
 				   size_t count, struct cache_detail *cd)
 {
-	static char write_buf[8192]; /* protected by queue_io_mutex */
+	static char write_buf[32768]; /* protected by queue_io_mutex */
 	ssize_t ret = -EINVAL;
 
Quoted 1 line

net/sunrpc/cache.c static ssize_t cache_do_downcall(char *kaddr, const char __user *buf, · line 914

Quoted 1 line
net/sunrpc/cache.c +0 -0
 	if (count >= sizeof(write_buf))

This is now commit 27a1e8a0f79e643d ("sunrpc: raise kernel RPC channel buffer size") upstream, and increases kernel size by 24 KiB, even if RPC is not used.

Quoted 3 lines

Can this buffer allocated dynamically instead? This code path seems to be a slow path anyway. If it's critical, perhaps this buffer can be allocated on first use?

Sure. Actually it is using an allocated buffer typically, see cache_downcall().

Looking back at the history.... That was added by Trond in 2009 (da77005f0d64 "SUNRPC: Remove the global temporary write buffer in net/sunrpc/cache.c").

Before that there's a pre-git change from 2004 which replaced a simple kmalloc(PAGE_SIZE).

So I guess the point was to be careful about higher-order allocations, but probably it was overkill.

How about making it a kvmalloc?

--b.

#4 lore raw

Good point Geert !

Quoted 1 line

How about making it a kvmalloc?

I can post a new patch using kvmalloc, Bruce looks we can also prescind of queue_io_mutex, what do you think ?

Quoted 3 lines

--b.

#5 lore raw
Quoted 6 lines · Roberto Bergantinos Corpas

Good point Geert !

Quoted 1 line

How about making it a kvmalloc?

I can post a new patch using kvmalloc, Bruce looks we can also prescind of queue_io_mutex, what do you think ?

And revert da77005f0d64, I think.

Maybe there's something I'm missing, but I don't think we need all that complexity.

--b.