Skip to contentSkip to conversation

_rpc_dtablesize: decrease to fix excessive memory usage

PATCH In review Zhuohao Bai 2 participants 4 messages Oct 31, 2023 – Jan 5, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover _rpc_dtablesize: decrease to fix excessive memory usage

In the client code, the function _rpc_dtablesize() is used to determine the memory allocation for the __svc_xports array.

However, some operating systems (including the recent Manjaro OS) can have _SC_OPEN_MAX values as high as 1073741816, which can cause the __svc_xports array to become too large. This results in the process being killed.

There is a limit to the maximum number of files. To avoid this problem, a possible solution is to set the size to the lesser of 1024 and this value to ensure that the array space for open files is not too large, thus preventing the process from terminating.

Also discovered that some users have taken action on the issue. It is necessary to address this for all users. Ultimately, we determined that adjusting the size value of _rpc_dtablesize() and streamlining the existing user code would be the most effective solution.

---
Changes in v1:
Clean up the existing code in user

---
Links:
RFC:https://lore.kernel.org/linux-nfs/[email protected]/T/#u

Zhuohao Bai (2):
_rpc_dtablesize: Decrease the value of size.
_rpc_dtablesize: Cleaning up the existing code

 src/rpc_dtablesize.c | 2 ++
 src/svc.c            | 2 --
 2 files changed, 2 insertions(+), 2 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/2 _rpc_dtablesize: Decrease the value of size.

To fix the bug caused by a Size value that is too large and leads to an array that requires excessive memory, which subsequently results in the failure of rpcbind to start properly.

---
 src/rpc_dtablesize.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
src/rpc_dtablesize.c +1 -1
index bce97e8..12f80c1 100644
@@ -41,7 +41,7 @@ _rpc_dtablesize(void)
 	static int size;
 
 	if (size == 0) {
-		size = sysconf(_SC_OPEN_MAX);
+		size = min(1024, sysconf(_SC_OPEN_MAX));
 	}
 	return (size);
 }
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/2 _rpc_dtablesize: Cleaning up the existing code

In reply to Zhuohao Bai

Some users have already taken steps to address this issue. To prevent duplication, we need to remove the modified code and modify the rpc_dtablesize function.

---
 src/rpc_dtablesize.c | 4 +++-
 src/svc.c            | 2 --
 2 files changed, 3 insertions(+), 3 deletions(-)
src/rpc_dtablesize.c +3 -1
index 12f80c1..e88698f 100644
@@ -41,7 +41,9 @@ _rpc_dtablesize(void)
 	static int size;
 
 	if (size == 0) {
-		size = min(1024, sysconf(_SC_OPEN_MAX));
+		size = sysconf(_SC_OPEN_MAX);
+		if (size > FD_SETSIZE)
+			size = FD_SETSIZE;
 	}
 	return (size);
 }
src/svc.c +0 -2
index 3a8709f..9b932a5 100644
@@ -657,8 +657,6 @@ svc_getreqset (readfds)
   assert (readfds != NULL);
 
   setsize = _rpc_dtablesize ();
-  if (setsize > FD_SETSIZE)
-    setsize = FD_SETSIZE;
   maskp = readfds->fds_bits;
   for (sock = 0; sock < setsize; sock += NFDBITS)
     {
Signature · 1 line
2.25.1
#3 lore raw
In reply to Zhuohao Bai
Quoted 27 lines · Zhuohao Bai

In the client code, the function _rpc_dtablesize() is used to determine the memory allocation for the __svc_xports array.

However, some operating systems (including the recent Manjaro OS) can have _SC_OPEN_MAX values as high as 1073741816, which can cause the __svc_xports array to become too large. This results in the process being killed.

There is a limit to the maximum number of files. To avoid this problem, a possible solution is to set the size to the lesser of 1024 and this value to ensure that the array space for open files is not too large, thus preventing the process from terminating.

Also discovered that some users have taken action on the issue. It is necessary to address this for all users. Ultimately, we determined that adjusting the size value of _rpc_dtablesize() and streamlining the existing user code would be the most effective solution.

---
Changes in v1:
Clean up the existing code in user

---
Links:
RFC:https://lore.kernel.org/linux-nfs/[email protected]/T/#u

Quoted 8 lines

Zhuohao Bai (2):
_rpc_dtablesize: Decrease the value of size.
_rpc_dtablesize: Cleaning up the existing code

  src/rpc_dtablesize.c | 2 ++
  src/svc.c            | 2 --
  2 files changed, 2 insertions(+), 2 deletions(-)

Both Committed... (tag: libtirpc-1-3-5-rc2)

steved.