Skip to content

mm: fixes for device-exclusive entries (hmm)

David Hildenbrand dri-devel linux-doc linux-mm lkml nouveau stable 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest12 patches 43 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12mm/gup: reject FOLL_SPLIT_PMD with hugetlb VMAsmm/gup.c+3 -03 replies
2/12mm/rmap: reject hugetlb folios in folio_make_device_exclusive()mm/rmap.c+1 -11 reply
3/12mm/rmap: convert make_device_exclusive_range() to make_device_exclusive()Documentation/mm/hmm.rst Documentation/translations/zh_CN/mm/hmm.rst drivers/gpu/drm/nouveau/nouveau_svm.c +4 more+75 -766 replies
4/12mm/rmap: implement make_device_exclusive() using folio_walk instead of rmap walkmm/rmap.c+55 -13311 replies
5/12mm/memory: detect writability in restore_exclusive_pte() through can_change_pte_writable()mm/memory.c+7 -47 replies
6/12mm: use single SWP_DEVICE_EXCLUSIVE entry typeinclude/linux/swap.h include/linux/swapops.h mm/mprotect.c +2 more+10 -392 replies
7/12mm/page_vma_mapped: device-private entries are not migration entriesmm/page_vma_mapped.c+1 -22 replies
8/12mm/rmap: handle device-exclusive entries correctly in try_to_unmap_one()mm/rmap.c+40 -136 replies
9/12mm/rmap: handle device-exclusive entries correctly in try_to_migrate_one()mm/rmap.c+51 -740 replies
10/12mm/rmap: handle device-exclusive entries correctly in folio_referenced_one()mm/rmap.c+8 -20 replies
11/12mm/rmap: handle device-exclusive entries correctly in page_vma_mkclean_one()mm/rmap.c+8 -00 replies
12/12mm/rmap: keep mapcount untouched for device-exclusive entriesmm/memory.c mm/rmap.c+1 -235 replies

Reviews trailers per patch

5 of 12 patches have a Reviewed-by or Acked-by.

PatchJohn HubbardAlistair PoppleSimona Vetter
1/12R-b R-b
2/12R-b
3/12R-b A-b
4/12
5/12
6/12R-b A-b
7/12R-b
8/12
9/12
10/12
11/12
12/12

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v1 thread

Cover letter [PATCH v1 00/12] mm: fixes for device-exclusive entries (hmm)

Discussing the PageTail() call in make_device_exclusive_range() with Willy, I recently discovered [1] that device-exclusive handling does not properly work with THP, making the hmm-tests selftests fail if THPs are enabled on the system.

Looking into more details, I found that hugetlb is not properly fenced, and I realized that something that was bugging me for longer -- how device-exclusive entries interact with mapcounts -- completely breaks migration and swapout of these folios while they have device-exclusive PTEs.

The program below can be used to allocate 1 GiB worth of pages and making them device-exclusive on a kernel with CONFIG_TEST_HMM.

Once they are device-exclusive, these folios cannot get swapped out (/proc/$pid/smaps_rollup will always indicate 1 GiB RSS no matter how much one forces memory reclaim), and when having a memory block onlined to ZONE_MOVABLE, trying to offline it will loop forever and complain about failed migration of a page that should be movable.

# echo offline > /sys/devices/system/memory/memory136/state
# echo online_movable > /sys/devices/system/memory/memory136/state
# ./hmm-swap &
... wait until everything is device-exclusive
# echo offline > /sys/devices/system/memory/memory136/state
[  285.193431][T14882] page: refcount:2 mapcount:0 mapping:0000000000000000
  index:0x7f20671f7 pfn:0x442b6a
[  285.196618][T14882] memcg:ffff888179298000
[  285.198085][T14882] anon flags: 0x5fff0000002091c(referenced|uptodate|
  dirty|active|owner_2|swapbacked|node=1|zone=3|lastcpupid=0x7ff)
[  285.201734][T14882] raw: ...
[  285.204464][T14882] raw: ...
[  285.207196][T14882] page dumped because: migration failure
[  285.209072][T14882] page_owner tracks the page as allocated
[  285.210915][T14882] page last allocated via order 0, migratetype
  Movable, gfp_mask 0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO),
  id 14926, tgid 14926 (hmm-swap), ts 254506295376, free_ts 227402023774
[  285.216765][T14882]  post_alloc_hook+0x197/0x1b0
[  285.218874][T14882]  get_page_from_freelist+0x76e/0x3280
[  285.220864][T14882]  __alloc_frozen_pages_noprof+0x38e/0x2740
[  285.223302][T14882]  alloc_pages_mpol+0x1fc/0x540
[  285.225130][T14882]  folio_alloc_mpol_noprof+0x36/0x340
[  285.227222][T14882]  vma_alloc_folio_noprof+0xee/0x1a0
[  285.229074][T14882]  __handle_mm_fault+0x2b38/0x56a0
[  285.230822][T14882]  handle_mm_fault+0x368/0x9f0
...

This series fixes all issues I found so far. There is no easy way to fix without a bigger rework/cleanup. I'll send out some additional cleanups that are not strictly required separately on top.

I wish we could just use some special present PROT_NONE PTEs instead of these (non-present, non-none) fake-swap entries; but that just results in the same problem we keep having (lack of spare PTE bits), and staring at other similar fake-swap entries, that ship has sailed.

With this series, make_device_exclusive() doesn't actually belong into mm/rmap.c anymore, but I'll leave moving that for another day.

I only tested this series with the hmm-tests selftests due to lack of HW, so I'd appreciate some testing, especially if the interaction between two GPUs wanting a device-exclusive entry works as expected.

I was able to trigger the -EBUSY in the hmm-tests a couple of times; but it doesn't retry yet. We should look into converting the folio_try_lock() into a folio_lock(), and just retry GUP immediately if the folio_walk fails in loop.

<program>
#include <stdio.h>
#include <fcntl.h>
#include <stdint.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <linux/types.h>
#include <linux/ioctl.h>

#define HMM_DMIRROR_EXCLUSIVE _IOWR('H', 0x05, struct hmm_dmirror_cmd)

struct hmm_dmirror_cmd {
	__u64 addr;
	__u64 ptr;
	__u64 npages;
	__u64 cpages;
	__u64 faults;
};

const size_t size = 1 * 1024 * 1024 * 1024ul;
const size_t chunk_size = 2 * 1024 * 1024ul;

int main(void)

{
	struct hmm_dmirror_cmd cmd;
	size_t cur_size;
	int fd, ret;
	char *addr, *mirror;

	fd = open("/dev/hmm_dmirror1", O_RDWR, 0);
	if (fd < 0) {
		perror("open failed\n");
		exit(1);
	}

	addr = mmap(NULL, size, PROT_READ | PROT_WRITE,
		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
	if (addr == MAP_FAILED) {
		perror("mmap failed\n");
		exit(1);
	}
	madvise(addr, size, MADV_NOHUGEPAGE);
	memset(addr, 1, size);

	mirror = malloc(chunk_size);

	for (cur_size = 0; cur_size < size; cur_size += chunk_size) {
		cmd.addr = (uintptr_t)addr + cur_size;
		cmd.ptr = (uintptr_t)mirror;
		cmd.npages = chunk_size / getpagesize();
		ret = ioctl(fd, HMM_DMIRROR_EXCLUSIVE, &cmd);
		if (ret) {
			perror("ioctl failed\n");
			exit(1);
		}
	}
	pause();
	return 0;
}

</program>

[1] https://lkml.kernel.org/r/[email protected]

Cc Andrew Morton <[email protected]>
Cc "Jérôme Glisse" <[email protected]>
Cc Jonathan Corbet <[email protected]>
Cc Alex Shi <[email protected]>
Cc Yanteng Si <[email protected]>
Cc Karol Herbst <[email protected]>
Cc Lyude Paul <[email protected]>
Cc Danilo Krummrich <[email protected]>
Cc David Airlie <[email protected]>
Cc Simona Vetter <[email protected]>
Cc "Liam R. Howlett" <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Jann Horn <[email protected]>
Cc Pasha Tatashin <[email protected]>
Cc Peter Xu <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Jason Gunthorpe <[email protected]>

David Hildenbrand (12):
mm/gup: reject FOLL_SPLIT_PMD with hugetlb VMAs
mm/rmap: reject hugetlb folios in folio_make_device_exclusive() mm/rmap: convert make_device_exclusive_range() to
make_device_exclusive()
mm/rmap: implement make_device_exclusive() using folio_walk instead of
rmap walk
mm/memory: detect writability in restore_exclusive_pte() through
can_change_pte_writable()
mm: use single SWP_DEVICE_EXCLUSIVE entry type mm/page_vma_mapped: device-private entries are not migration entries mm/rmap: handle device-exclusive entries correctly in

    try_to_unmap_one()

mm/rmap: handle device-exclusive entries correctly in
try_to_migrate_one()
mm/rmap: handle device-exclusive entries correctly in
folio_referenced_one()
mm/rmap: handle device-exclusive entries correctly in
page_vma_mkclean_one()
mm/rmap: keep mapcount untouched for device-exclusive entries

 Documentation/mm/hmm.rst                    |   2 +-
 Documentation/translations/zh_CN/mm/hmm.rst |   2 +-
 drivers/gpu/drm/nouveau/nouveau_svm.c       |   5 +-
 include/linux/mmu_notifier.h                |   2 +-
 include/linux/rmap.h                        |   5 +-
 include/linux/swap.h                        |   7 +-
 include/linux/swapops.h                     |  27 +-
 lib/test_hmm.c                              |  45 +-
 mm/gup.c                                    |   3 +
 mm/memory.c                                 |  28 +-
 mm/mprotect.c                               |   8 -
 mm/page_table_check.c                       |   5 +-
 mm/page_vma_mapped.c                        |   3 +-
 mm/rmap.c                                   | 457 +++++++++-----------
 14 files changed, 246 insertions(+), 353 deletions(-)

base-commit: 4845035bec80db8d716a94b80bb5593bf1d69270

Signature · 1 line
2.48.1