Skip to contentSkip to conversation

x86/hugetlb: AMD F15h VA alignment offset breaks MAP_HUGETLB alignment

REGRESSION Karsten Desler 4 participants 10 messages May 27 – May 28 Memory management

lore
Share & tools
lore
#0 lore raw

Hi,

I found a reproducible hugetlb regression on an AMD Family 15h system.

On some boots, mmap(MAP_HUGETLB) returns a virtual address that is not aligned to the hugepage size. The mapping is nevertheless installed as a hugetlb VMA. When the process exits, the kernel later BUGs in __unmap_hugepage_range().

6.18.33 x86_64, AMD opteron 6238, 2M hugepages

Example bad mapping captured from /proc/$pid/maps:

7fc67f604000-7fc67f804000 rw-p 00000000 00:0f 12340 /anon_hugepage (deleted)

The address has offset 0x4000 within a 2 MiB hugepage.

smaps confirms it is really hugetlb:

  KernelPageSize:     2048 kB
  MMUPageSize:        2048 kB
  Private_Hugetlb:    2048 kB

VmFlags: rd wr mr mw me de ht

Minimal reproducer:

echo 1000 > /proc/sys/vm/nr_hugepages

mmap(NULL, 1229824, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE|MAP_HUGETLB, -1, 0)

On bad boots this returns e.g.:

mmap returned 0x7fc67f604000 aligned=no offset=16384

and exiting the process triggers:

  Kernel BUG at __unmap_hugepage_range+0x5ef/0x640
  RIP: __unmap_hugepage_range+0x5ef/0x640
  Fixing recursive fault but reboot is needed!

The following is AI work, sorry if that's total BS but at the very least, I can reproduce the kernelBUG and booting with
align_va_addr=off
works around the issue.

This is boot-dependent. Some boots work, some fail. The reason appears to be the per-boot AMD F15h VA alignment offset.

The old x86 hugetlb path in arch/x86/mm/hugetlbpage.c only set:

  info.align_mask = PAGE_MASK & ~huge_page_mask(h);

It did not add the AMD F15h align offset.

After the v6.13-rc1 hugetlb mmap rework, hugetlb mappings go through arch_get_unmapped_area*(), and x86 currently does:

  if (filp) {
          info.align_mask = get_align_mask(filp);
          info.align_offset += get_align_bits();
  }

For hugetlb, get_align_mask(filp) correctly returns the hugepage alignment mask, but get_align_bits() can still return the AMD F15h per-boot offset, e.g. 0x4000. That produces a non-hugepage-aligned hugetlb VMA.

Likely introduced by the v6.13-rc1 series:

1317a5e7f7b1 arch/x86: teach arch_get_unmapped_area_vmflags to handle hugetlb mappings 7bd3f1e1a9ae mm: make hugetlb mappings go through mm_get_unmapped_area_vmflags cc92882ee218 mm: drop hugetlb_get_unmapped_area{_*} functions

AI suggests passing filp to get_align_bits and doing if (filp && is_file_hugepages(filp))

	return 0;

Best regards,
Karsten

#1 lore raw
Quoted 9 lines · Karsten Desler

Hi,

I found a reproducible hugetlb regression on an AMD Family 15h system.

On some boots, mmap(MAP_HUGETLB) returns a virtual address that is not aligned to the hugepage size. The mapping is nevertheless installed as a hugetlb VMA. When the process exits, the kernel later BUGs in __unmap_hugepage_range().

6.18.33 x86_64, AMD opteron 6238, 2M hugepages

Thanks Kartsten for reporting this.

Ooops, that would be me.

Quoted 30 lines

Example bad mapping captured from /proc/$pid/maps:

7fc67f604000-7fc67f804000 rw-p 00000000 00:0f 12340 /anon_hugepage (deleted)

The address has offset 0x4000 within a 2 MiB hugepage.

smaps confirms it is really hugetlb:

  KernelPageSize:     2048 kB
  MMUPageSize:        2048 kB
  Private_Hugetlb:    2048 kB

VmFlags: rd wr mr mw me de ht

Minimal reproducer:

echo 1000 > /proc/sys/vm/nr_hugepages

mmap(NULL, 1229824, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE|MAP_HUGETLB, -1, 0)

On bad boots this returns e.g.:

mmap returned 0x7fc67f604000 aligned=no offset=16384

and exiting the process triggers:

  Kernel BUG at __unmap_hugepage_range+0x5ef/0x640
  RIP: __unmap_hugepage_range+0x5ef/0x640
  Fixing recursive fault but reboot is needed!
Quoted 7 lines

The following is AI work, sorry if that's total BS but at the very least, I can reproduce the kernelBUG and booting with
align_va_addr=off
works around the issue.

This is boot-dependent. Some boots work, some fail. The reason appears to be the per-boot AMD F15h VA alignment offset.

I have to confess that I completely overlooked that scenario, so let me apologyze.

Quoted 6 lines

The old x86 hugetlb path in arch/x86/mm/hugetlbpage.c only set:

  info.align_mask = PAGE_MASK & ~huge_page_mask(h);

It did not add the AMD F15h align offset.

Quoted 7 lines

After the v6.13-rc1 hugetlb mmap rework, hugetlb mappings go through arch_get_unmapped_area*(), and x86 currently does:

  if (filp) {
          info.align_mask = get_align_mask(filp);
          info.align_offset += get_align_bits();
  }

Ok, I see.

Quoted 10 lines

For hugetlb, get_align_mask(filp) correctly returns the hugepage alignment mask, but get_align_bits() can still return the AMD F15h per-boot offset, e.g. 0x4000. That produces a non-hugepage-aligned hugetlb VMA.

Likely introduced by the v6.13-rc1 series:

1317a5e7f7b1 arch/x86: teach arch_get_unmapped_area_vmflags to handle hugetlb mappings 7bd3f1e1a9ae mm: make hugetlb mappings go through mm_get_unmapped_area_vmflags cc92882ee218 mm: drop hugetlb_get_unmapped_area{_*} functions

Yes, that was part of a refactoring I did some time ago.

I will fix it up later today/early tomorrow.

Would you be available for a quick test once I have the patch?

Signature · 2 lines
Oscar Salvador
SUSE Labs
#2 lore raw
Quoted 12 lines · Oscar Salvador (SUSE)
Quoted 10 lines · Karsten Desler

For hugetlb, get_align_mask(filp) correctly returns the hugepage alignment mask, but get_align_bits() can still return the AMD F15h per-boot offset, e.g. 0x4000. That produces a non-hugepage-aligned hugetlb VMA.

Likely introduced by the v6.13-rc1 series:

1317a5e7f7b1 arch/x86: teach arch_get_unmapped_area_vmflags to handle hugetlb mappings 7bd3f1e1a9ae mm: make hugetlb mappings go through mm_get_unmapped_area_vmflags cc92882ee218 mm: drop hugetlb_get_unmapped_area{_*} functions

Quoted 5 lines

Yes, that was part of a refactoring I did some time ago.

I will fix it up later today/early tomorrow.

Would you be available for a quick test once I have the patch?

Maybe this? Can you give it a shot?

diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c
index 776ae6fa7f2d..60f876dce8e5 100644
--- a/arch/x86/kernel/sys_x86_64.c
+++ b/arch/x86/kernel/sys_x86_64.c
@@ -157,7 +157,12 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len,
}
if (filp) {
info.align_mask = get_align_mask(filp);
- info.align_offset += get_align_bits();
+ /*
+ * Hugepages must remain hugepage-aligned, so skip adding an offset
+ * in case we enabled 'align_va_addr'.
+ */
+ if (!is_file_hugepages(filp))
+ info.align_offset += get_align_bits();
}

return vm_unmapped_area(&info);
@@ -222,7 +227,12 @@ arch_get_unmapped_area_topdown(struct file *filp, unsigned long addr0,

if (filp) {
info.align_mask = get_align_mask(filp);
- info.align_offset += get_align_bits();
+ /*
+ * Hugepages must remain hugepage-aligned, so skip adding an offset
+ * in case we enabled 'align_va_addr'.
+ */
+ if (!is_file_hugepages(filp))
+ info.align_offset += get_align_bits();
}
addr = vm_unmapped_area(&info);
if (!(addr & ~PAGE_MASK))

Signature · 2 lines
Oscar Salvador
SUSE Labs
#3 lore raw
Quoted 1 line · Oscar Salvador (SUSE)

Maybe this? Can you give it a shot?

Thanks, applied and tested across 5 reboots.
All mmaps return properly aligned hugepages, don't cause kernelBUGs on exit and show the expected munmap behavior.

mmap(NULL, 1229824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE|MAP_HUGETLB, -1, 0) = 0x7f7e49a00000
munmap(0x7f7e49a00000, 1229824) = -1 EINVAL (Invalid argument)
munmap(0x7f7e49a00000, 2097152) = 0

Best regards,
Karsten

#4 lore raw
In reply to Oscar Salvador (SUSE)
Quoted 10 lines · Oscar Salvador (SUSE)

if (filp) {
info.align_mask = get_align_mask(filp);
- info.align_offset += get_align_bits();
+ /*
+ * Hugepages must remain hugepage-aligned, so skip adding an offset
+ * in case we enabled 'align_va_addr'.
+ */
+ if (!is_file_hugepages(filp))
+ info.align_offset += get_align_bits();
}

That's a good hack to show the scope of the problem.

But I'd really rather this be dealt with in the arch-independent code, not by adding hugetlb hacks to arch code. It isn't even clear to me what exactly goes wrong when you set a tiny ->align_offset and have a larger ->align_mask. Shouldn't the tiny offset just get masked off?

        gap += (info->align_offset - gap) & info->align_mask;

I spent a whole five seconds looking at it, but something seems to be missing from the problem description.

#5 lore raw
Quoted 12 lines · Dave Hansen
Quoted 10 lines · Oscar Salvador (SUSE)

if (filp) {
info.align_mask = get_align_mask(filp);
- info.align_offset += get_align_bits();
+ /*
+ * Hugepages must remain hugepage-aligned, so skip adding an offset
+ * in case we enabled 'align_va_addr'.
+ */
+ if (!is_file_hugepages(filp))
+ info.align_offset += get_align_bits();
}

Quoted 1 line

That's a good hack to show the scope of the problem.

Haha, do not worry, I myself have 0 interestin spreading hugetlb-specific code around (on the contrary), but I wanted to proof the point.

Quoted 7 lines

But I'd really rather this be dealt with in the arch-independent code, not by adding hugetlb hacks to arch code. It isn't even clear to me what exactly goes wrong when you set a tiny ->align_offset and have a larger ->align_mask. Shouldn't the tiny offset just get masked off?

        gap += (info->align_offset - gap) & info->align_mask;

I would assume so, but something is definitely going in with the calculation, as I could reproduce this if I artificially set align_offset to 0x1000 for hugetlb mappings.

I will find some time today to have a deep look.

Signature · 2 lines
Oscar Salvador
SUSE Labs
#6 lore raw
Quoted 13 lines · Oscar Salvador (SUSE)
Quoted 12 lines · Dave Hansen
Quoted 10 lines · Oscar Salvador (SUSE)

if (filp) {
info.align_mask = get_align_mask(filp);
- info.align_offset += get_align_bits();
+ /*
+ * Hugepages must remain hugepage-aligned, so skip adding an offset
+ * in case we enabled 'align_va_addr'.
+ */
+ if (!is_file_hugepages(filp))
+ info.align_offset += get_align_bits();
}

Quoted 12 lines
Quoted 1 line

That's a good hack to show the scope of the problem.

Haha, do not worry, I myself have 0 interestin spreading hugetlb-specific code around (on the contrary), but I wanted to proof the point.

Quoted 7 lines

But I'd really rather this be dealt with in the arch-independent code, not by adding hugetlb hacks to arch code. It isn't even clear to me what exactly goes wrong when you set a tiny ->align_offset and have a larger ->align_mask. Shouldn't the tiny offset just get masked off?

        gap += (info->align_offset - gap) & info->align_mask;

Ok, I finally got to it.

So, let us assume we ask for a 2MB hugetlb page.

~huge_page_mask = 0x1fffff

huge_page_mask_align = PAGE_MASK & ~huge_page_mask(hstate_file(file)) = 0x1ff000

unmapped_area_topdown()
info->length = 0x200000 (2MB)
info->align_mask = 0x1ff000

/* Adjust search length to account for worst case alignment overhead */ total_gap_lenght_requested = info->length + info->align_mask = 0x3ff000

We find a gap: 0x7f28cfb10000 - 0x7f28cfd10000 (2MB) and assuming align_offset is = 0:

gap -= (gap - info->align_offset) & info->align_mask 0x7f28cfb10000 -= 0x7f28cfb10000 & 0x1ff000 = 7f28cfa00000 (2MB aligned)

IIUC, we mask what we got with align_mask to know how much we need to substract in order to be properly aligned (and since we already accounted for extra length before, we are sure we do not overstep anything below).

Now, I have to acknowledge that I had to look at the code several times, because it was not clear to me why we were not just masking off 2MB, but I guess if we do we lose whatever align_offset gives us (if smaller than align_mask).

I mean, code was already like that before my refactoring, just that back then we did "align_offset = 0" unilaterally for hugetlb mappings.

The only thing bugging is, should not the same happen for THP-file-backed mappings?

Signature · 2 lines
Oscar Salvador
SUSE Labs
#7 lore raw
Quoted 1 line · Oscar Salvador (SUSE)

The only thing bugging is, should not the same happen for THP-file-backed mappings?

Aha, no, for THP we do not set align_mask (at least on x86), and the masking off is being done in __thp_get_unmapped_area().

Signature · 2 lines
Oscar Salvador
SUSE Labs
#8 lore raw
Quoted 5 lines · Oscar Salvador (SUSE)
Quoted 1 line · Oscar Salvador (SUSE)

The only thing bugging is, should not the same happen for THP-file-backed mappings?

Aha, no, for THP we do not set align_mask (at least on x86), and the masking off is being done in __thp_get_unmapped_area().

I hope you found this in the proces:

dfb09f9b7ab0 ("x86, amd: Avoid cache aliasing penalties on AMD family 15h")

It is from a different century tho and perhaps not too relevant but I thought I should mention it...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#9 lore raw
Quoted 10 lines · Borislav Petkov
Quoted 5 lines · Oscar Salvador (SUSE)
Quoted 1 line · Oscar Salvador (SUSE)

The only thing bugging is, should not the same happen for THP-file-backed mappings?

Aha, no, for THP we do not set align_mask (at least on x86), and the masking off is being done in __thp_get_unmapped_area().

I hope you found this in the proces:

dfb09f9b7ab0 ("x86, amd: Avoid cache aliasing penalties on AMD family 15h")

Ei Boris, thanks for pointing it out.

Actually, checking the other arches that have their own get_align_mask() for setting the mask (s390 and sparc), they both skip info.align_offset if we are dealing with hugetlb, e.g: s390:

info.align_mask = get_align_mask(filp, flags);
if (!(filp && is_file_hugepages(filp)))

         info.align_offset = pgoff << PAGE_SHIFT;

So, maybe for the time being we can do the same in x86 in order to fix the regression (although the refactoring is 2 years old and first time we heard about it was yesterday) and then we can think of a nicer way to handle this in non-arch code so s390 and sparc would get cleaned up as well. Thoughts?

Signature · 2 lines
Oscar Salvador
SUSE Labs