kselftest: mm: fix some failure of split_huge_page_test

split_huge_page_test can fail for the following reasons:

1. During the test, khugepaged may collapse previously split pages again, causing intermittent failures.

2. Since glibc commit 321e1fc73f (“malloc: Enable 2MB THP by default on AArch64”), glibc may call madvise(MADV_HUGEPAGE) for sufficiently large allocations made by memalign(). The underlying VMA may start at a different address from the aligned address returned by memalign(). Moreover, a subsequent madvise(MADV_HUGEPAGE) call does not split the VMA because it already has the same advice.

This causes the test to fail because the check_huge_xxx() helpers incorrectly require the address returned by memalign() to match the VMA start address reported in /proc/self/smaps.

Address these issues by applying MADV_NOHUGEPAGE after faulting in the huge page, preventing khugepaged from collapsing it again, and instead of relying on /proc/self/smaps, use /proc/self/pagemap and /proc/kpageflags to detect huge-page mappings and large folios:

1. If hpage_size == pmd_pagesize, check PAGE_IS_HUGE instead of using check_large_folios(), since only the mapping type matters. This identifies PMD-mapped huge pages.
2. Otherwise, use check_large_folios() to detect large folios. This covers mTHP cases.
3. Check the folio flags according to the type of huge page.

Since check_huge_shmem() was required to distinguish shmem huge pages because /proc/self/smaps reports them using a dedicated “ShmemPmdMapped” entry, as opposed to “FilePmdMapped” for file-backed huge pages.

Now that /proc/self/smaps is no longer used to detect huge pages and /proc/kpageflags is used instead, it is sufficient to distinguish
between file-backed and anonymous pages since the ShmemPmdMapped is also
kind of file-backed.

Therefore, remove check_huge_shmem() and use check_huge_file() instead.

This patch based on mm-unstable

---
Changes in v9:
- fix some typos in comment and Fixes tag.
- separating huge and type checking.
- add r-b tags.
- Link to v8: https://lore.kernel.org/r/[email protected]

Changes in v8:
- add intermediate patch for __check_pmd_huge().
- add r-b, a-b tags.
- Link to v7: https://lore.kernel.org/r/[email protected]

Changes in v7:
- change khugepaged's check_huge() callback.
- Link to v6: https://lore.kernel.org/r/[email protected]

Changes in v6:
- drop alloc_isolated_mem() patch.
- remove check_huge_shmem().
- Link to v5: https://lore.kernel.org/r/[email protected]

Changes in v5:
- rename madv_nohuge() to disable_khugepaged().
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- reparse commit message.
- introduce alloc_isolated_mem().
- make madv_nohuge() helper according to suggestion.
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- change message in case of failure of madvise() with MADV_NOHUGEPAGE.
- fix collapse_single_mthp (mthp_khugepaged:anon) test case failure.
- Link to v2: https://lore.kernel.org/all/[email protected]/

Changes in v2:
- rebase to mm-unstable.
- add message in case of failure of madvise() with MADV_NOHUGEPAGE.
- fix wrong setup expected_huge in check_huge_shmem().
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Yeoreum Yun (4):
kselftest: mm: prevent random failure of huge page split for khugepaged
kselftest: mm: replace usage of /proc/self/smaps for __check_pmd_huge()
kselftest: mm: integrate huge page checks
kselftest: mm: remove check_huge_shmem()

 tools/testing/selftests/mm/folio_split_race_test.c |   2 +-
 tools/testing/selftests/mm/khugepaged.c            |  38 ++-----
 tools/testing/selftests/mm/split_huge_page_test.c  |  16 +++
 tools/testing/selftests/mm/uffd-common.c           |   4 +-
 tools/testing/selftests/mm/vm_util.c               | 113 ++++++++++++++-------
 tools/testing/selftests/mm/vm_util.h               |   1 -
 6 files changed, 100 insertions(+), 74 deletions(-)

---
base-commit: e8d0f6a1b2a447d02845984fa6288787543cb03c
change-id: 20260820-fix_split-f44939ec44b8

Best regards,

Sincerely,
Yeoreum Yun