Skip to contentSkip to conversation

selftests/prctl: Fix non-anonymous VMA mapping in set-anon-vma-name test

PATCH Applied Hongfu Li 2 participants 3 messages Aug 3 – Aug 4 Kselftest

lore
Share & tools
lore

Hongfu Li

author
#0 lore raw

PATCH 1/1 selftests/prctl: Fix non-anonymous VMA mapping in set-anon-vma-name test

From: Hongfu Li <[email protected]>

The test creates a non-anonymous VMA (ptr_not_anon) via mmap() with MAP_PRIVATE but without MAP_ANONYMOUS, using fd=0 (stdin) as the file descriptor. This always fails because fd=0 is not a regular file, and the failure was hidden because ASSERT_NE() incorrectly checked for NULL instead of MAP_FAILED.

Fix by using mkstemp() + ftruncate() to create a real temporary file, then mapping it with MAP_PRIVATE to obtain a genuine file-backed VMA. Also fix the mmap() error checks to use MAP_FAILED instead of NULL, and pass fd=-1 for the anonymous mapping for clarity. The temp file is unlinked immediately so it does not persist on disk.

---
 .../selftests/prctl/set-anon-vma-name-test.c  | 21 ++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
tools/testing/selftests/prctl/set-anon-vma-name-test.c +16 -5
index ac6721b184a6..5f9589534da8 100644
@@ -4,11 +4,13 @@
  */
 
 #include <errno.h>
+#include <fcntl.h>
 #include <sys/prctl.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <sys/mman.h>
 #include <string.h>
+#include <unistd.h>
 
 #include "kselftest_harness.h"
 
@@ -73,15 +75,24 @@ int was_renaming_successful(char *target_name, unsigned long ptr)
 
 FIXTURE(vma) {
 	void *ptr_anon, *ptr_not_anon;
+	int fd_not_anon;
 };
 
 FIXTURE_SETUP(vma) {
+	char template[] = "./set-anon-vma-test-XXXXXX";
+
 	self->ptr_anon = mmap(NULL, AREA_SIZE, PROT_READ | PROT_WRITE,
-					MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
-	ASSERT_NE(self->ptr_anon, NULL);
+					MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	ASSERT_NE(self->ptr_anon, MAP_FAILED);
+
+	self->fd_not_anon = mkstemp(template);
+	ASSERT_NE(self->fd_not_anon, -1);
+	unlink(template);
+	ASSERT_EQ(ftruncate(self->fd_not_anon, AREA_SIZE), 0);
 	self->ptr_not_anon = mmap(NULL, AREA_SIZE, PROT_READ | PROT_WRITE,
-					MAP_PRIVATE, 0, 0);
-	ASSERT_NE(self->ptr_not_anon, NULL);
+					MAP_PRIVATE, self->fd_not_anon, 0);
+	ASSERT_NE(self->ptr_not_anon, MAP_FAILED);
+	close(self->fd_not_anon);
 }
 
 FIXTURE_TEARDOWN(vma) {
@@ -98,7 +109,7 @@ TEST_F(vma, renaming) {
 	EXPECT_EQ(rename_vma((unsigned long)self->ptr_anon, AREA_SIZE, BAD_NAME), -EINVAL);
 
 	TH_LOG("Try to rename non-anonymous VMA");
-	EXPECT_EQ(rename_vma((unsigned long) self->ptr_not_anon, AREA_SIZE, GOOD_NAME), -EINVAL);
+	EXPECT_EQ(rename_vma((unsigned long) self->ptr_not_anon, AREA_SIZE, GOOD_NAME), -EBADF);
 }
 
 TEST_HARNESS_MAIN
Signature · 1 line
2.54.0
#1 lore raw
Quoted 5 lines · On Mon, 3 Aug 2026 18:30:46 +0800 Hongfu Li <[email protected]> wrote:

The test creates a non-anonymous VMA (ptr_not_anon) via mmap() with MAP_PRIVATE but without MAP_ANONYMOUS, using fd=0 (stdin) as the file descriptor. This always fails because fd=0 is not a regular file, and the failure was hidden because ASSERT_NE() incorrectly checked for NULL instead of MAP_FAILED.

It's remarkable how often code assumes that mmap(...)==NULL is the error return.

Quoted 5 lines

Fix by using mkstemp() + ftruncate() to create a real temporary file, then mapping it with MAP_PRIVATE to obtain a genuine file-backed VMA. Also fix the mmap() error checks to use MAP_FAILED instead of NULL, and pass fd=-1 for the anonymous mapping for clarity. The temp file is unlinked immediately so it does not persist on disk.

Thanks.

AI review found one glitch which I don't think we need to care about:

	https://sashiko.dev/#/patchset/[email protected]

and one possible pre-existing issue which is kinda related and which you may choose to address. Please take a look and let us know?

author
#2 lore raw
Quoted 5 lines

AI review found one glitch which I don't think we need to care about:

	https://sashiko.dev/#/patchset/[email protected]

and one possible pre-existing issue which is kinda related and which you may choose to address. Please take a look and let us know?

Thanks for the review. I've carefully considered the AI review's findings.

The /dev/zero case raised by the AI reviewer may be addressed by commit cc3b846182db ("mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous"), which makes these mappings truly anonymous (no vm_file, no vm_ops). So the vma->vm_file checks in madvise.c should not wrongly reject them.

That said, I agree that vma_is_anonymous(vma) better expresses the intent than vma->vm_file in these sites. I'm happy to send a separate patch for that if needed.

Best regards,
Hongfu