selftests/core: fix unshare_test with large fs.nr_open
PATCH 1/1 selftests/core: fix unshare_test with large fs.nr_open
The test assumes fs.nr_open is close to the default 1048576, but some systems set it much higher (e.g. 1073741816). This is systemd's doing:
since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to their largest possible values on boot, as file descriptors are already accounted for by memcg [1].
In that case, dup2() to nr_open + 64 requires the kernel to allocate a file descriptor table with ~1 billion entries, which fails with ENOMEM.
Cap the nr_open value used for the test's own arithmetic to a known reasonable base value (1048576) and restore the true original value once the test has completed.
[1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
("main: bump fs.nr_open + fs.max-file to their largest possible values")
--- tools/testing/selftests/core/unshare_test.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
tools/testing/selftests/core/unshare_test.c +17 -2
@@ -40,6 +40,14 @@ TEST(unshare_EMFILE)ASSERT_EQ (sscanf (buf,"%d" , &nr_open),1 ); +/* +* Cap nr_open for the duration of the test to avoid ENOMEM from a +* huge fd table allocation; buf/n keep the real original value so +* fs.nr_open can be restored to it once the test is done. +*/ +if (nr_open >1024 *1024 ) + nr_open =1024 *1024 ; +ASSERT_EQ (0 ,getrlimit (RLIMIT_NOFILE , &rlimit));/* bump fs.nr_open */
@@ -73,10 +81,13 @@ TEST(unshare_EMFILE)if (pid ==0 ) {int err; +char buf3[32 ]; +ssize_t n3; -/* restore fs.nr_open */ +/* restore fs.nr_open to the (possibly capped) test baseline */ + n3 =sprintf (buf3,"%d\n" , nr_open);lseek (fd,0 ,SEEK_SET ); -write (fd, buf, n); +write (fd, buf3, n3);/* ... and now unshare(CLONE_FILES) must fail with EMFILE */ err =unshare (CLONE_FILES );EXPECT_EQ (err,-1 )
@@ -89,6 +100,10 @@ TEST(unshare_EMFILE)EXPECT_EQ (waitpid (pid, &status,0 ), pid);EXPECT_EQ (true,WIFEXITED (status));EXPECT_EQ (0 ,WEXITSTATUS (status)); + +/* restore the real fs.nr_open value */ +lseek (fd,0 ,SEEK_SET ); +write (fd, buf, n); }TEST_HARNESS_MAIN
Signature · 1 line
2.47.1
Quoted 8 lines · On Fri, 14 Aug 2026 18:57:09 +0200 Konstantin Khorenko…
The test assumes fs.nr_open is close to the default 1048576, but some systems set it much higher (e.g. 1073741816). This is systemd's doing:
since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to their largest possible values on boot, as file descriptors are already accounted for by memcg [1].In that case,
dup2()tonr_open+ 64 requires the kernel to allocate a file descriptor table with ~1 billion entries, which fails with ENOMEM.
2018. Can you suggest why we (or I) haven't heard about this?
Quoted 6 lines
Cap the
nr_openvalue used for the test's own arithmetic to a known reasonable base value (1048576) and restore the true original value once the test has completed.[1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
("main: bump fs.nr_open + fs.max-file to their largest possible values")
Well, we do want selftests to run well on the kernel with which they are shipped, so I'm thinking we should backport this into -stable kernels. We can probably skip the Fixes:, but a cc:stable should be added to capture this.
Thoughts?
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Fri, 14 Aug 2026 18:57:09 +0200 Konstantin Khorenko…
The test assumes fs.nr_open is close to the default 1048576, but some systems set it much higher (e.g. 1073741816). This is systemd's doing:
since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to their largest possible values on boot, as file descriptors are already accounted for by memcg [1].
Quoted 4 lines
Quoted 2 lines
In that case,
dup2()tonr_open+ 64 requires the kernel to allocate a file descriptor table with ~1 billion entries, which fails with ENOMEM.2018. Can you suggest why we (or I) haven't heard about this?
The 2018 date only says when systemd started doing this, not when it began to matter. Laid out as a timeline:
Oct 2018 systemd a8b627aaed409 ("main: bump fs.nr_open + fs.max-file
to their largest possible values"), released in v240. From here on PID 1 raises fs.nr_open to 1073741816 on boot. Harmless in itself: nothing was asking for an fd table anywhere near that size.
Aug 2024 611fbeb44a777 ("selftests:core: test coverage for dup_fd()
failure handling in unshare_fd()"), v6.12-rc1. The test is added, and it is the first thing in tree to deliberately dup2() to a descriptor just past fs.nr_open. This is the point where the two meet, so the exposure window is about two years, not eight.
Jun 2025 04a2c4b4511d1 ("fs: Prevent file descriptor table
allocations exceeding INT_MAX"), v6.17-rc1, cc:stable. Somebody did hit it and did fix it - but on the kernel side. Its changelog explicitly names tools/testing/selftests/core/unshare_test.c as a way to reach the WARNING.
So the report exists and the kernel half is fixed; what nobody did was go back and fix the test that provoked it, which is what this patch is for.
As for why it is not seen more widely, distributions differ on whether the bump happens at all. The systemd option is 'bump-proc-sys-fs-nr-open', declared as a boolean with no explicit value in meson_options.txt, so it defaults to true.
i know for sure Ubuntu passes -Dbump-proc-sys-fs-nr-open=false in rules, so on Ubuntu nr_open stays at the kernel default of 1048576 and the test passes.
On the other hand i tried Almalinux 10 (fs.nr_open = 1073741816) + latest mainstream kernel and the testcase still fails there.
One correction to my own changelog while we are here: on a kernel that already carries 04a2c4b4511d1, dup2() no longer fails with ENOMEM. The allocation is now rejected up front and the caller gets EMFILE instead, without the WARNING, but the test still fails.
Quoted 13 lines
Quoted 6 lines
Cap the
nr_openvalue used for the test's own arithmetic to a known reasonable base value (1048576) and restore the true original value once the test has completed.[1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
("main: bump fs.nr_open + fs.max-file to their largest possible values")Well, we do want selftests to run well on the kernel with which they are shipped, so I'm thinking we should backport this into -stable kernels. We can probably skip the Fixes:, but a cc:stable should be added to capture this.
Thoughts?