mm/huge_memory: copy PMDs at fork like the PTE path does

copy_huge_pmd() decides what to copy at fork by its own rules, apart from copy_present_ptes(): a lockless check for special PMDs, and a skip of every file THP. Following David's review of v3, this series makes it decide the way the PTE path does, under the page table lock, with vm_normal_page_pmd(). That fixes the two bugs v1-v3 were about. The patches:

1. folio_dup_file_rmap_pmd() duplicated a PMD mapping as HPAGE_PMD_NR PTE mappings. It had no caller until patch 2.
2. File THPs are copied at fork instead of being refaulted by the child, as their PTEs would be.
3. The special-PMD read moves under the lock. That closes the race with a driver zap (vfio-pci, panthor) and the missing page table deposit on ppc64 hash that Sashiko found in v1.
4. A selftest for 1 and 2.

Stable
======

The race (since v6.12) and the ppc64 hash NULL deref (since v7.1) are fixed in patch 3, which only works on top of the copy rework in patch 2, and patch 2 is the first caller of the helper patch 1 fixes. So patches 1-3 carry Cc: stable, as David suggested on v4. They will not apply to 6.12 as they are, I will do the backports.

A related refcount bug in copy_huge_pud() for device DAX is fixed in a separate series, as it does not depend on this one:
https://patch.msgid.link/20261021-vstor-150143-copy_huge_pud-folio-refcount-v1-0-8bd88551f9be@virtuozzo.com

Testing
=======

v5 on v7.4-rc1, x86_64 with 4 NUMA nodes, the series with CONFIG_DEBUG_VM, KASAN and lockdep:

- fork_file_thp (patch 4) skips on v7.4-rc1, where fork leaves file THPs to a refault, and fails in both cases without patch 1, where ShmemPmdMapped drops when the child exits and the kernel reports "Bad page cache";
- with the series it passes, also under migrate_pages() of the parent and under reclaim in a small memcg, where some runs skip; with other processes churning PMD-mapped shmem it passes or skips, never fails;
- cow, guard-regions, uffd-unit-tests, uffd-wp-mremap, split_huge_page_test, mremap_test, soft-dirty, pagemap_ioctl, migration, madv_populate, transhuge-stress and khugepaged pass as on v7.4-rc1.

The runs below are from v4, on v7.3-rc7 in a KVM guest with 4 vCPUs, x86_64. Patches 1-3 changed since only by the device private CoW check.

Race with a driver zap. The guest's QXL device is bound to vfio-pci in no-IOMMU mode, BAR0 of 64 MB mapped with PMDs. The reproducer below forks while a second thread clears PCI_COMMAND_MEMORY through the vfio config region after a random delay of up to 1 ms; the child waits for the write to finish and reads one byte of each 2 MB of the BAR. A child that reads anything without SIGBUS kept a stale mapping. I have not hit it without help, so a test-only debugfs knob adds an mdelay(2) between copy_pmd_range()'s lockless pmd_is_huge() and the PMD locks:

  +-----------------------+-------+-------+----------------+
  | kernel                | delay | forks | stale children |
  +-----------------------+-------+-------+----------------+
  | 7.3-rc7               | none  | 20900 | 0              |
  | 7.3-rc7               | 2 ms  | 150   | 128            |
  | 7.3-rc7 + this series | none  | 3000  | 0              |
  | 7.3-rc7 + this series | 2 ms  | 400   | 0              |
  +-----------------------+-------+-------+----------------+

ppc64 hash is not tested on hardware. On x86 a test-only hack makes arch_needs_pgtable_deposit() true and mremap move the table, as on ppc64 hash. Without the series the first child of the reproducer oopses at exit in pgtable_trans_huge_withdraw(); with it the reproducer and the mm selftests pass.

Cost
====

fork() now copies file THPs instead of leaving them to a refault, which costs fork() a reference, an rmap update and a counter per huge PMD and saves the child a fault per huge PMD it touches. fork() of a process with 4 GB of shmem THPs mapped MAP_PRIVATE and an anon_vma on the VMA, medians of 2000 forks:

  +-----------------------+------------+-------------------------+
  | kernel                | fork()     | fork(), child reads all |
  +-----------------------+------------+-------------------------+
  | 7.3-rc7               | 86-129 us  | 2380-2620 us            |
  | 7.3-rc7 + this series | 230-355 us | 820-930 us              |
  +-----------------------+------------+-------------------------+

Notes
=====

- drm_gem_shmem and mshv_vtl map special PMDs with vmf_insert_pfn_pmd() as well.
- copy_huge_pud() still differs from the PMD tail in that it does not mark a shared PUD clean in the child or handle uffd bits.

Reproducer. It writes the BAR as a control and reads it with the memory disabled, which is harmless for QXL VRAM in a guest; do not point it at a BAR that holds device registers. Build with
gcc -O2 -pthread -o fork_revoke fork_revoke.c

run as root with the device bound to vfio-pci; arguments: BDF, iterations, BAR index, max revoke delay in microseconds, e.g.
./fork_revoke 0000:00:02.0 100 0 3000

    // SPDX-License-Identifier: GPL-2.0
    /*
     * Fork while vfio-pci revokes a BAR mapped with huge PMDs.
     *
     * The parent maps a BAR of a vfio-pci device and faults it in. A second
     * thread disables the device memory through the command register, which
     * makes vfio-pci zap all BAR mappings, while the main thread forks. Once
     * the memory is disabled, every access to the BAR must raise SIGBUS. A
     * child that can still read a BAR chunk kept a mapping the zap removed.
     *
     * usage: fork_revoke <domain:bus:dev.fn> [iterations [bar [max_delay_us]]]
     *
     * The revoke starts after a random delay of up to max_delay_us, so that it
     * lands at different points of fork.
     */
    #define _GNU_SOURCE
    #include <errno.h>
    #include <fcntl.h>
    #include <limits.h>
    #include <pthread.h>
    #include <sched.h>
    #include <setjmp.h>
    #include <signal.h>
    #include <stdint.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <unistd.h>
    #include <sys/ioctl.h>
    #include <sys/mman.h>
    #include <sys/wait.h>
    #include <linux/pci_regs.h>
    #include <linux/vfio.h>

    #define SZ_2M	(2UL << 20)

    static int dev;
    static off_t cfg_off;
    static int go_pipe[2], done_pipe[2];
    static volatile uint32_t *shared;	/* [0]: iteration whose revoke finished */
    static unsigned long iter;
    static unsigned int max_delay_us;
    static sigjmp_buf jb;

    static void die(const char *what)
    {
    	fprintf(stderr, "%s: %s\n", what, strerror(errno));
    	exit(2);
    }

    static int device_fd(const char *bdf)
    {
    	char path[PATH_MAX], link[PATH_MAX], *group_id;
    	int container, group, noiommu;
    	ssize_t len;

    	snprintf(path, sizeof(path), "/sys/bus/pci/devices/%s/iommu_group", bdf);
    	len = readlink(path, link, sizeof(link) - 1);
    	if (len < 0)
    		die("iommu_group");
    	link[len] = 0;
    	group_id = strrchr(link, '/') + 1;
    	snprintf(path, sizeof(path), "/dev/vfio/%s", group_id);
    	noiommu = access(path, F_OK);
    	if (noiommu)
    		snprintf(path, sizeof(path), "/dev/vfio/noiommu-%s", group_id);
    	container = open("/dev/vfio/vfio", O_RDWR);
    	group = open(path, O_RDWR);
    	if (container < 0 || group < 0)
    		die(path);
    	if (ioctl(group, VFIO_GROUP_SET_CONTAINER, &container) ||
    	    ioctl(container, VFIO_SET_IOMMU,
    		  noiommu ? VFIO_NOIOMMU_IOMMU : VFIO_TYPE1v2_IOMMU))
    		die("vfio container");
    	return ioctl(group, VFIO_GROUP_GET_DEVICE_FD, bdf);
    }

    static void set_memory(int on)
    {
    	uint16_t cmd;

    	if (pread(dev, &cmd, 2, cfg_off + PCI_COMMAND) != 2)
    		die("read PCI_COMMAND");
    	cmd = on ? cmd | PCI_COMMAND_MEMORY : cmd & ~PCI_COMMAND_MEMORY;
    	if (pwrite(dev, &cmd, 2, cfg_off + PCI_COMMAND) != 2)
    		die("write PCI_COMMAND");
    }

    static void *revoker(void *arg)
    {
    	unsigned long i;

    	while (read(go_pipe[0], &i, sizeof(i)) == sizeof(i)) {
    		if (max_delay_us)
    			usleep(random() % max_delay_us);
    		set_memory(0);
    		shared[0] = i;
    		if (write(done_pipe[1], &i, sizeof(i)) != sizeof(i))
    			die("done pipe");
    	}
    	return NULL;
    }

    static void sigbus(int sig)
    {
    	siglongjmp(jb, 1);
    }

    /* Number of 2MB chunks of the BAR that can be read without SIGBUS. */
    static unsigned long readable(volatile uint8_t *bar, size_t size)
    {
    	volatile unsigned long n = 0;
    	volatile size_t off;

    	for (off = 0; off < size; off += SZ_2M) {
    		if (sigsetjmp(jb, 1))
    			continue;
    		(void)bar[off];
    		n++;
    	}
    	return n;
    }

    int main(int argc, char **argv)
    {
    	unsigned long iterations = argc > 2 ? strtoul(argv[2], NULL, 0) : 1000;
    	unsigned int index = argc > 3 ? atoi(argv[3]) : 0;
    	struct vfio_region_info cfg = { .argsz = sizeof(cfg),
    					.index = VFIO_PCI_CONFIG_REGION_INDEX };
    	struct vfio_region_info bar = { .argsz = sizeof(bar), .index = index };
    	unsigned long stale_children = 0, stale_chunks = 0, n;
    	volatile uint8_t *map;
    	pthread_t thread;
    	size_t size, off;
    	uint8_t *resv;
    	int status;
    	pid_t pid;

    	if (argc < 2) {
    		fprintf(stderr, "usage: %s <bdf> [iterations [bar [max_delay_us]]]\n",
    			argv[0]);
    		return 2;
    	}
    	max_delay_us = argc > 4 ? atoi(argv[4]) : 0;
    	signal(SIGBUS, sigbus);
    	dev = device_fd(argv[1]);
    	if (dev < 0)
    		die("device fd");
    	if (ioctl(dev, VFIO_DEVICE_GET_REGION_INFO, &cfg) ||
    	    ioctl(dev, VFIO_DEVICE_GET_REGION_INFO, &bar))
    		die("region info");
    	if (!(bar.flags & VFIO_REGION_INFO_FLAG_MMAP) || bar.size < SZ_2M) {
    		fprintf(stderr, "BAR %u: no mmap or smaller than 2MB\n", index);
    		return 2;
    	}
    	cfg_off = cfg.offset;
    	size = bar.size & ~(SZ_2M - 1);

    	/* A 2MB aligned address lets vfio-pci map the BAR with huge PMDs. */
    	resv = mmap(NULL, size + SZ_2M, PROT_NONE,
    		    MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
    	if (resv == MAP_FAILED)
    		die("reserve");
    	resv = (uint8_t *)(((uintptr_t)resv + SZ_2M - 1) & ~(SZ_2M - 1));
    	map = mmap(resv, size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED,
    		   dev, bar.offset);
    	if (map == MAP_FAILED)
    		die("mmap BAR");
    	shared = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
    		      MAP_SHARED | MAP_ANONYMOUS, -1, 0);
    	if (shared == MAP_FAILED)
    		die("shared page");

    	/* Control: without a revoke the child reads what the parent wrote. */
    	set_memory(1);
    	for (off = 0; off < size; off += SZ_2M)
    		map[off] = (uint8_t)(off / SZ_2M) ^ 0x5a;
    	pid = fork();
    	if (!pid) {
    		for (off = 0; off < size; off += SZ_2M)
    			if (map[off] != ((uint8_t)(off / SZ_2M) ^ 0x5a))
    				_exit(1);
    		_exit(0);
    	}
    	if (pid < 0 || waitpid(pid, &status, 0) != pid ||
    	    !WIFEXITED(status) || WEXITSTATUS(status)) {
    		fprintf(stderr, "control: child does not see the parent's BAR\n");
    		return 2;
    	}
    	printf("BAR %u: %zu MB, %lu iterations, revoke delay < %u us\n",
    	       index, size >> 20, iterations, max_delay_us);

    	/* Premise: with the memory disabled the BAR cannot be read. */
    	set_memory(0);
    	n = readable(map, size);
    	set_memory(1);
    	if (n) {
    		fprintf(stderr, "memory disabled, parent still reads %lu chunks\n", n);
    		return 2;
    	}

    	if (pipe(go_pipe) || pipe(done_pipe))
    		die("pipe");
    	if (pthread_create(&thread, NULL, revoker, NULL))
    		die("thread");

    	for (iter = 1; iter <= iterations; iter++) {
    		set_memory(1);
    		for (off = 0; off < size; off += SZ_2M)
    			(void)map[off];
    		if (write(go_pipe[1], &iter, sizeof(iter)) != sizeof(iter))
    			die("go pipe");
    		pid = fork();
    		if (pid < 0)
    			die("fork");
    		if (!pid) {
    			while (shared[0] != iter)
    				sched_yield();
    			n = readable(map, size);
    			_exit(n > 255 ? 255 : n);
    		}
    		if (read(done_pipe[0], &n, sizeof(n)) != sizeof(n))
    			die("done pipe");
    		if (waitpid(pid, &status, 0) != pid || !WIFEXITED(status))
    			die("child");
    		if (WEXITSTATUS(status)) {
    			stale_children++;
    			stale_chunks += WEXITSTATUS(status);
    		}
    	}
    	set_memory(1);
    	printf("children with a stale BAR mapping: %lu of %lu (%lu 2MB chunks)\n",
    	       stale_children, iterations, stale_chunks);
    	return stale_children ? 1 : 0;
    }

---
Changes in v5:
- Rebase onto v7.4-rc1.
- Patches 1-3: Cc: stable (David, Lorenzo).
- Patch 1: drop the list of consequences, the helper had no caller
(Zi Yan).
- Patch 2: the device private write path gets the CoW check of
copy_nonpresent_pte() (David). Drop fuse DAX as the example of
VM_MIXEDMAP, it no longer sets the flag. Say on the Cc: stable line
that the next patch needs it.
- Patch 4: up to three attempts, each on a fresh file. Fail only if
every attempt sees ShmemPmdMapped drop by exactly the parent's share,
pass on the first clean one, skip if none is clean: other processes
change the counter and reclaim or migration take PMDs from the parent
(Kiryl). Keep every mapping until exit: after the bug, an unmap
drops the counter once more, which hid the bug from the second case.
Use check_huge_file(), check_huge_shmem() is gone.
- Cover: v5 test results, box tables.
- Link to v4: https://patch.msgid.link/20261021-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v4-0-66f00b7a564c@virtuozzo.com

Changes in v4:
- Rework after David's review: copy_huge_pmd() decides under the PTL
with vm_normal_page_pmd(), like copy_present_ptes(), and copies file
THPs (patches 2, 3). The pmd_none() recheck and the separate deposit
patch are gone.
- New patch 1: fix folio_dup_file_rmap_pmd(), whose first caller is
patch 2.
- New selftest, patch 4.
- Move Assisted-by from the patches to the cover letter, after commit
1e035297ffdad ("docs: remove Assisted-by references").
- Link to v3: https://patch.msgid.link/20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v3-0-85eaec8417f6@virtuozzo.com

Changes in v3:
- Patch 1: recheck the PMD with pmd_none(), as only a zap can change
it and a zap leaves it none (Lance Yang).
- Link to v2: https://patch.msgid.link/20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v2-0-e1548488fa68@virtuozzo.com

Changes in v2:
- Add patch 2, the page table deposit for ppc64 hash, reported by
Sashiko.
- Patch 1: name panthor instead of drm_gem_shmem as the driver whose
stale child mapping reaches freed pages: drm_gem_shmem keeps the
pages of a purged object until the last mapping goes away.
- Link to v1: https://patch.msgid.link/20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-v1-1-d225c60bc316@virtuozzo.com

---
Konstantin Khorenko (4):
mm/rmap: duplicate a PMD mapping as one in folio_dup_file_rmap_pmd() mm/huge_memory: copy file THPs at fork like copy_present_ptes() does mm/huge_memory: read a special PMD under the lock in copy_huge_pmd() selftests/mm: test fork of PMD-mapped shmem THPs

 MAINTAINERS                                |   1 +
 include/linux/rmap.h                       |   2 +-
 mm/huge_memory.c                           |  85 +++++-----
 tools/testing/selftests/mm/Makefile        |   1 +
 tools/testing/selftests/mm/fork_file_thp.c | 256 +++++++++++++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh  |   1 +
 6 files changed, 301 insertions(+), 45 deletions(-)

---
base-commit: 05441d697f7f9a927264f4058cbbc6a03dfcac9d
change-id: 20261014-b4-prep-vstor-149847-copy_huge_pmd-recheck-special-pmd-bdf7efb612c2

Best regards,

Konstantin Khorenko,
Virtuozzo Linux Kernel Team

folio_dup_file_rmap_pmd() passes PGTABLE_LEVEL_PTE to __folio_dup_file_rmap() for HPAGE_PMD_NR pages, so it adds HPAGE_PMD_NR PTE mappings to the folio instead of one PMD mapping.

The helper has no caller yet, the next patch adds the first one. Pass PGTABLE_LEVEL_PMD, as folio_try_dup_anon_rmap_pmd() does.

---
 include/linux/rmap.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 74cca0e3c7264..87c88a9811933 100644
@@ -558,7 +558,7 @@ static inline void folio_dup_file_rmap_pmd(struct folio *folio,
 		struct page *page, struct vm_area_struct *dst_vma)
 {
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	__folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, dst_vma, PGTABLE_LEVEL_PTE);
+	__folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, dst_vma, PGTABLE_LEVEL_PMD);
 #else
 	WARN_ON_ONCE(true);
 #endif

2.43.0

LGTM:

cheers,
jose a. p-a

This fixes no bug: it makes fork copy a PMD the way copy_present_ptes() copies a PTE. Until now copy_huge_pmd() skipped every folio PMD of a non-anonymous VMA and left the child to fault them in again, while the PTEs of the same VMA are copied.

Treat file folios as the PTE path does: take a reference, duplicate the rmap and account them. fork then pays a reference and an rmap update per PMD, and saves the child a fault per PMD it touches. The next patch sends PFN mappings through the same tail.

That covers the VMAs vma_needs_copy() sends here:

- private file mappings with an anon_vma;
- VM_COPY_ON_FORK ones: uffd-wp and uffd-rwp, VM_MIXEDMAP, and VM_MAYBE_GUARD, which shared shmem, file and DAX mappings get from a guard region.

The DAX huge zero PMD is copied too, with a reference taken by mm_get_huge_zero_folio() as for anonymous VMAs. A migration entry of a file folio is copied and accounted like copy_nonpresent_pte() does, and made read-only in both processes only in a CoW mapping. A device private entry gets the same CoW check, as in copy_nonpresent_pte().

The copy then gets the same tail as __copy_present_ptes(): write protect both sides in a CoW mapping, mark the child's copy clean in a shared one, and drop the uffd-wp bit where the child is not armed. An anonymous THP is now write protected only when its PMD is writable, as a PTE is.

A page table is deposited under the copy where the fault paths deposit
one: in anonymous VMAs, and wherever the architecture needs one.

---
 mm/huge_memory.c | 63 +++++++++++++++++++++++++++++++++++---------------------
 1 file changed, 39 insertions(+), 24 deletions(-)

index bd44d58e86845..e00632edb3856 100644
@@ -1897,12 +1897,13 @@ static void copy_huge_non_present_pmd(
 		pmd_t pmd, pgtable_t pgtable)
 {
 	softleaf_t entry = softleaf_from_pmd(pmd);
-	struct folio *src_folio;
+	struct folio *src_folio = softleaf_to_folio(entry);
 
 	VM_WARN_ON_ONCE(!pmd_is_valid_softleaf(pmd));
 
-	if (softleaf_is_migration_write(entry) ||
-	    softleaf_is_migration_read_exclusive(entry)) {
+	if ((softleaf_is_migration_write(entry) ||
+	     softleaf_is_migration_read_exclusive(entry)) &&
+	    vma_is_cow_mapping(src_vma)) {
 		entry = make_readable_migration_entry(swp_offset(entry));
 		pmd = softleaf_to_pmd(entry);
 		if (pmd_swp_soft_dirty(*src_pmd))

@@ -1915,7 +1916,8 @@ static void copy_huge_non_present_pmd(
 		 * For device private entries, since there are no
 		 * read exclusive entries, writable = !readable
 		 */
-		if (softleaf_is_device_private_write(entry)) {
+		if (softleaf_is_device_private_write(entry) &&
+		    vma_is_cow_mapping(src_vma)) {
 			entry = make_readable_device_private_entry(swp_offset(entry));
 			pmd = softleaf_to_pmd(entry);
 

@@ -1926,7 +1928,6 @@ static void copy_huge_non_present_pmd(
 			set_pmd_at(src_mm, addr, src_pmd, pmd);
 		}
 
-		src_folio = softleaf_to_folio(entry);
 		VM_WARN_ON(!folio_test_large(src_folio));
 
 		folio_get(src_folio);

@@ -1938,9 +1939,11 @@ static void copy_huge_non_present_pmd(
 					    dst_vma, src_vma);
 	}
 
-	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
-	mm_inc_nr_ptes(dst_mm);
-	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
+	add_mm_counter(dst_mm, mm_counter(src_folio), HPAGE_PMD_NR);
+	if (pgtable) {
+		mm_inc_nr_ptes(dst_mm);
+		pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
+	}
 	if (!userfaultfd_protected(dst_vma))
 		pmd = pmd_swp_clear_uffd(pmd);
 	set_pmd_at(dst_mm, addr, dst_pmd, pmd);

@@ -1956,6 +1959,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	pmd_t pmd;
 	pgtable_t pgtable = NULL;
 	int ret = -ENOMEM;
+	bool writable;
 
 	pmd = pmdp_get_lockless(src_pmd);
 	if (unlikely(pmd_present(pmd) && pmd_special(pmd) &&

@@ -1976,13 +1980,11 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		goto set_pmd;
 	}
 
-	/* Skip if can be re-fill on fault */
-	if (!vma_is_anonymous(dst_vma))
-		return 0;
-
-	pgtable = pte_alloc_one(dst_mm);
-	if (unlikely(!pgtable))
-		goto out;
+	if (vma_is_anonymous(dst_vma) || arch_needs_pgtable_deposit()) {
+		pgtable = pte_alloc_one(dst_mm);
+		if (unlikely(!pgtable))
+			goto out;
+	}
 
 	dst_ptl = pmd_lock(dst_mm, dst_pmd);
 	src_ptl = pmd_lockptr(src_mm, src_pmd);

@@ -2000,7 +2002,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	}
 
 	if (unlikely(!pmd_trans_huge(pmd))) {
-		pte_free(dst_mm, pgtable);
+		if (pgtable)
+			pte_free(dst_mm, pgtable);
 		goto out_unlock;
 	}
 	/*

@@ -2015,7 +2018,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		 * copy. It just takes a reference.
 		 */
 		mm_get_huge_zero_folio(dst_mm);
-		goto out_zero_page;
+		goto copy_pmd;
 	}
 
 	src_page = pmd_page(pmd);

@@ -2023,6 +2026,12 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	src_folio = page_folio(src_page);
 
 	folio_get(src_folio);
+	if (!folio_test_anon(src_folio)) {
+		folio_dup_file_rmap_pmd(src_folio, src_page, dst_vma);
+		add_mm_counter(dst_mm, mm_counter_file(src_folio),
+			       HPAGE_PMD_NR);
+		goto copy_pmd;
+	}
 	if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, dst_vma, src_vma))) {
 		/* Page maybe pinned: split and retry the fault on PTEs. */
 		folio_put(src_folio);

@@ -2033,19 +2042,25 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		return -EAGAIN;
 	}
 	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
-out_zero_page:
-	mm_inc_nr_ptes(dst_mm);
-	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
+copy_pmd:
+	if (pgtable) {
+		mm_inc_nr_ptes(dst_mm);
+		pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
+	}
 
-	/* See __copy_present_ptes(): restore accessible protection. */
+	/* See __copy_present_ptes(). */
+	writable = pmd_write(pmd);
 	if (!userfaultfd_protected(dst_vma)) {
 		if (userfaultfd_rwp(src_vma) && pmd_uffd(pmd))
 			pmd = pmd_modify(pmd, dst_vma->vm_page_prot);
 		pmd = pmd_clear_uffd(pmd);
 	}
-
-	pmdp_set_wrprotect(src_mm, addr, src_pmd);
-	pmd = pmd_wrprotect(pmd);
+	if (vma_is_cow_mapping(src_vma) && writable) {
+		pmdp_set_wrprotect(src_mm, addr, src_pmd);
+		pmd = pmd_wrprotect(pmd);
+	}
+	if (src_vma->vm_flags & VM_SHARED)
+		pmd = pmd_mkclean(pmd);
 set_pmd:
 	pmd = pmd_mkold(pmd);
 	set_pmd_at(dst_mm, addr, dst_pmd, pmd);

2.43.0

LGTM:

cheers,
jose a. p-a

LGTM.

Best Regards,
Yan, Zi

copy_huge_pmd() reads a special PMD without the page table lock and copies that value to the child. The mmap write lock held by fork does not stop unmap_mapping_range(): it zaps under i_mmap_rwsem and the PMD lock.

The zap visits the parent VMA before the child's, as dup_mmap() links the child right after the parent in i_mmap. If it clears the parent PMD after fork has read it, and passes the child before fork copies it, the child keeps a mapping the driver has revoked, for example:

- vfio-pci: the child reaches the BAR after the device memory was disabled or the device was reset, instead of getting SIGBUS;
- panthor: the child maps the pages of an evicted buffer object after the driver has released them.

The same branch deposits no page table under the copy. ppc64 in hash mode keeps the hash slot array of every huge PMD in that table, so the child dereferences a NULL table on the first hash fault, in __hash_page_thp(), and on the zap, in hash__pmdp_huge_get_and_clear().

Take the locks first and let vm_normal_page_pmd() tell a PFN mapping from a folio under them, as copy_present_ptes() does with vm_normal_page(). The zap clears the parent PMD under the same lock, so either fork finds it none and skips it, or the copy is in place before the zap reaches the child.

The copy of a PFN mapping then goes through the same tail as a folio:

- it gets a page table where the architecture needs one;
- in a shared mapping it is marked clean in the child, as a PTE is;
- in a CoW mapping, which only VM_MIXEDMAP allows, both sides are write protected instead of a warning being printed.

---
 mm/huge_memory.c | 24 +++---------------------
 1 file changed, 3 insertions(+), 21 deletions(-)

index e00632edb3856..9cb5f99aba87f 100644
@@ -1961,25 +1961,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	int ret = -ENOMEM;
 	bool writable;
 
-	pmd = pmdp_get_lockless(src_pmd);
-	if (unlikely(pmd_present(pmd) && pmd_special(pmd) &&
-		     !is_huge_zero_pmd(pmd))) {
-		dst_ptl = pmd_lock(dst_mm, dst_pmd);
-		src_ptl = pmd_lockptr(src_mm, src_pmd);
-		spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
-		/*
-		 * No need to recheck the pmd, it can't change with write
-		 * mmap lock held here.
-		 *
-		 * Meanwhile, making sure it's not a CoW VMA with writable
-		 * mapping, otherwise it means either the anon page wrongly
-		 * applied special bit, or we made the PRIVATE mapping be
-		 * able to wrongly write to the backend MMIO.
-		 */
-		VM_WARN_ON_ONCE(vma_is_cow_mapping(src_vma) && pmd_write(pmd));
-		goto set_pmd;
-	}
-
 	if (vma_is_anonymous(dst_vma) || arch_needs_pgtable_deposit()) {
 		pgtable = pte_alloc_one(dst_mm);
 		if (unlikely(!pgtable))

@@ -2021,7 +2002,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 		goto copy_pmd;
 	}
 
-	src_page = pmd_page(pmd);
+	src_page = vm_normal_page_pmd(src_vma, addr, pmd);
+	if (!src_page)
+		goto copy_pmd;
 	VM_BUG_ON_PAGE(!PageHead(src_page), src_page);
 	src_folio = page_folio(src_page);
 

@@ -2061,7 +2044,6 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	}
 	if (src_vma->vm_flags & VM_SHARED)
 		pmd = pmd_mkclean(pmd);
-set_pmd:
 	pmd = pmd_mkold(pmd);
 	set_pmd_at(dst_mm, addr, dst_pmd, pmd);
 

2.43.0

LGTM:

cheers,
jose a. p-a

Makes sense.

Best Regards,
Yan, Zi

Thanks Konstantin, for fixing this. Without this patch I can reproduce the problem on Hash on Power (on my test dev setup), after fork(), when child touches those pages.

[   62.543247][  T242] Kernel attempted to read user page (0) - exploit attempt? (uid: 0)
[   62.544992][  T242] BUG: Kernel NULL pointer dereference on read at 0x00000000
[   62.545335][  T242] Faulting instruction address: 0xc0000000000a0ee8
cpu 0x0: Vector: 300 (Data Access) at [c00000002035f9d0]
    pc: c0000000000a0ee8: __hash_page_thp+0x218/0x560
    lr: c0000000000a0e14: __hash_page_thp+0x144/0x560
    sp: c00000002035fc70

msr: 8000000000009033
dar: 0
dsisr: 40000000
current = 0xc00000001fb08480
paca = 0xc000000005010000 irqmask: 0x03 irq_happened: 0x09
pid = 242, comm = vfio-fork
enter ? for help
[c00000002035fc70] 0000000000000000 (unreliable) [c00000002035fd30] c000000000098a40 hash_page_mm+0x688/0x8d0 [c00000002035fe10] c000000000098dc8 do_hash_fault+0x140/0x368 [c00000002035fe50] c000000000008be0 data_access_common_virt+0x210/0x220 ---- Exception: 300 (Data Access) at 0000000010001b60
SP (7fffde6dd7b0) is in userspace

With this series, I see that the issue is fixed. Thanks for helping in fixing it. Please feel free to add:

Fork a process that maps shmem THPs with PMDs, in a VMA whose page tables fork copies: a private mapping with an anon_vma, and a shared one with a guard region. The child must find the PMDs before it touches the mapping, and its exit must leave the parent's mapping and the system-wide ShmemPmdMapped counter as they were.

With folio_dup_file_rmap_pmd() adding PTE mappings, the child's exit takes the parent's share off ShmemPmdMapped while the parent still maps the folios. Unmapping them drops it once more, so every mapping stays until exit, where the kernel reports "Bad page cache", or hits a VM_BUG_ON with CONFIG_DEBUG_VM. A kernel that leaves file THPs to a refault in the child skips the test.

ShmemPmdMapped is system-wide, and reclaim or NUMA balancing can take a PMD from the parent while the test runs. So the test makes up to three attempts, each on a fresh file, and fails only if all of them see the counter drop by exactly the parent's share: the bug does that on every fork, other processes only by chance. It passes on the first clean attempt and skips when none is clean.

The test mounts a tmpfs with huge=always over /tmp in a private mount namespace, so it leaves the THP settings and the host's mounts and files alone.

---
 MAINTAINERS                                |   1 +
 tools/testing/selftests/mm/Makefile        |   1 +
 tools/testing/selftests/mm/fork_file_thp.c | 256 +++++++++++++++++++++++++++++
 tools/testing/selftests/mm/run_vmtests.sh  |   1 +
 4 files changed, 259 insertions(+)

index 93abbb315065d..1e6c35a30a369 100644
@@ -17859,6 +17859,7 @@ F:	mm/collapse.h
 F:	mm/huge_memory.c
 F:	mm/khugepaged.c
 F:	mm/mm_slot.h
+F:	tools/testing/selftests/mm/fork_file_thp.c
 F:	tools/testing/selftests/mm/khugepaged.c
 F:	tools/testing/selftests/mm/split_huge_page_test.c
 F:	tools/testing/selftests/mm/transhuge-stress.c

index 0ea72e9979347..2489027b2a36b 100644
@@ -108,6 +108,7 @@ TEST_GEN_FILES += folio_split_race_test
 TEST_GEN_FILES += folio_order_check
 TEST_GEN_FILES += khugepaged_sync_check
 TEST_GEN_FILES += khugepaged_race
+TEST_GEN_FILES += fork_file_thp
 TEST_GEN_FILES += soft-dirty
 
 ifeq ($(ARCH),x86_64)

index a1b45a3dedaee..790a6b6746465 100755
@@ -433,6 +433,7 @@ fi
 CATEGORY="thp" run_test ./transhuge-stress -d 20
 
 CATEGORY="thp" run_test ./folio_split_race_test
+CATEGORY="thp" run_test ./fork_file_thp
 
 CATEGORY="migration" run_test ./migration
 

2.43.0

Right, my fault: I based all the versions on Linus' tree, which still has read_pmd_pagesize(). On mm-new this patch needs a one-line s/read_pmd_pagesize/pmd_psize/, patches 1-3 apply as they are.

If a new version is needed, I'll base it on mm-new.

Thanks for the review!

Cheers,
 Konstantin.

The second column assumes that the child touches all the file THPs, yes? If the child touches only a small proportion of those mappings, or just does fork+exec, that changes the "Cost" comparison?

And on ppc64 hash-mode setups this also allocates deposited page tables for mappings which the child might never touch? So a modest increase in memory use is to be expected? I can't say I care about this (sorry!).

But I worry that this change is not an overall win for thp-file mappings, which are the common case.

I'm not understanding why we need to mess with thp-file behavior at all in order to address a special-PMD bug?

Changelog is soooo long. I replaced this Reproducer section with simply

- A reproducer is at the Link;

So interested people can still find it.

Anyway, bugfixes come first so I'll fix the selftest pmd_psize thing and I'll toss this onto the pile, thanks.

I'm attempting to runtime test this on local LLM-driven live kernel test setup, but it keeps on kicking me out and calling me a cybercriminal. Sigh. Giving up.

Success.

tl;dr: all seems good. The expected performance gains and losses were observed. Can't performance-test powerpc because the unpatched kernels crashed due to the bug the series fixes.

: I tested the four v5 patches and the selftest fix on matched x86-64 : kernels. Both shmem fork tests passed with the series and skipped on : the pre-series kernel, as expected. They checked inherited PMDs, : private and shared writes, parent mappings and ShmemPmdMapped after : child exit.
:
: The special-PMD race was reproduced with a virtual QXL device mapped : through VFIO. Revocation before fork produced SIGBUS on both kernels. : When revocation completed during a controlled pause in fork, the child : accessed the revoked mapping before the series and received SIGBUS with : the series. Both pauses were reached without timing out. No BUG, : WARNING, KASAN report or oops appeared in these x86-64 runs. :
: I also built matched PowerPC kernels and tested a POWER9/pseries : hash-MMU guest under software emulation: two vCPUs, 2 GiB RAM, 64 KiB : base pages and 16 MiB PMDs. A trace confirmed order-8 VFIO PMD : insertion from a 64 MiB virtual PCI shared-memory BAR. On the : pre-series kernel, the child's first access after fork caused a NULL : pointer dereference in __hash_page_thp() and a soft lockup; that guest : required a bounded timeout. With the series, the child read and exited : normally. Revocation before fork gave the child SIGBUS with the series. : This is one comparison, not a PowerPC stress test.
:
: I measured fork costs with a 512 MiB private shmem mapping, including : 510 MiB actually mapped with PMDs. Median times from 100 forks per : condition on each x86-64 kernel, following five warmups:
:
: Before With series
: fork only 49 us 63 us
: fork through child exit,
: no child reads 132 us 202 us
: fork through child exit,
: one read per PMD 637 us 326 us
:
: This workload pays more at fork and takes about half as long end-to-end : when the child reads every PMD. These are single-boot measurements, not : a broader performance characterization. The timing kernels had KASAN, : lockdep and DEBUG_VM disabled; the correctness kernels had them enabled. : The x86-64 KVM guests had 8 GiB, four vCPUs (one virtual socket, four : single-threaded virtual cores) and one virtual NUMA node. The host has : 64 physical cores and 128 SMT threads on one physical NUMA node; : individual vCPUs were not pinned.
:
: I attempted a PowerPC VFIO fork-only timing comparison. The pre-series : kernel faulted on the first child exit, even without a child read, and : the guest was stopped after a 90-second timeout. The patched kernel : completed 100 fork/exit operations, averaging 2.77 ms under software : emulation with tracing enabled. There is no matched PowerPC performance : comparison.
:
: Tested-by: Andrew Morton <[email protected]>

Yes. I measured it again on bare metal, with a fork()+exec column:
x86_64 with 4 NUMA nodes, the benchmark bound to node 0, both kernels v7.4-rc1 with the same config, 4 GB of shmem THPs mapped MAP_PRIVATE with an anon_vma, medians of 2000 forks, range over the rounds. The exec is of /bin/true.

  +-----------------+------------+-----------------+--------------+
  | kernel          | fork()     | child reads all | fork()+exec  |
  +-----------------+------------+-----------------+--------------+
  | v7.4-rc1        | 135-141 us | 3820-4036 us    | 559-567 us   |
  | v7.4-rc1+series | 429-440 us | 1007-1036 us    | 1190-1226 us |
  +-----------------+------------+-----------------+--------------+

So a child that execs right away, or touches little, pays about 0.3 us per huge PMD: the copy at fork and the teardown at exec or exit. A child that faults a PMD in itself pays about 1.8 us for it, so the copy wins once the child touches more than about a sixth of them.

Yes, but a small one, judging from the code (I have no ppc64 to test):
one 4 KB page table fragment per copied 16 MB PMD, 1/4096 of the mapping. Radix and other arches deposit nothing.

It is freed when the child unmaps the range, execs or exits, and a PMD the child faults in itself deposits the same 4 KB anyway.

Only a narrow case pays the extra cost, a file THP that is:

- mapped MAP_PRIVATE;
- in a VMA written to at least once, so it has an anon_vma;
- a PMD-sized folio in the page cache, mapped by one PMD;
- itself never written to.

For example:

- a big data file mapped MAP_PRIVATE that is mostly read and patched in a few places;
- a private mapping of a memfd or tmpfs file.

The common case does not get here. fork() skips a VMA without an anon_vma and without VM_COPY_ON_FORK (vma_needs_copy()), so shared file mappings and private ones that were never written are still left to faults.

In these VMAs fork() already copies the PTEs of the same file folios; only their PMDs were skipped.

It came from the review of v3: David asked to make copy_huge_pmd() decide the way copy_present_ptes() does, instead of keeping rules of its own for PMDs:

https://lore.kernel.org/all/[email protected]/

Sure, thanks. And thanks for the pmd_psize fixup.

--
Cheers,
Konstantin

I will take another look and queue it once ready.

Andrew, please drop it if you already applied it. This will conflict with the refcount overflow handling that will have to go in first.

Cheers,

David