Skip to contentSkip to conversation

x86/mm changes for v5.9

GIT PULL Merged Ingo Molnar 6 participants 18 messages Aug 3, 2020 – Aug 14, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Linus,

Please pull the latest x86/mm git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-mm-2020-08-03

# HEAD: 2b32ab031e82a109e2c5b0d30ce563db0fe286b4 x86/mm/64: Make sync_global_pgds() static

The biggest change is to not sync the vmalloc and ioremap ranges for x86-64 anymore.

Thanks,

Ingo

------------------>
Joerg Roedel (3):
x86/mm: Pre-allocate P4D/PUD pages for vmalloc area x86/mm/64: Do not sync vmalloc/ioremap mappings x86/mm/64: Make sync_global_pgds() static

 arch/x86/include/asm/pgtable_64.h       |  2 --
 arch/x86/include/asm/pgtable_64_types.h |  2 --
 arch/x86/mm/init_64.c                   | 59 +++++++++++++++++++++++++++++----
 3 files changed, 53 insertions(+), 10 deletions(-)
#2 lore raw
In reply to Ingo Molnar

Hi Ingo, Joerg,

Quoted 12 lines · Ingo Molnar

Linus,

Please pull the latest x86/mm git tree from:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-mm-2020-08-03

   # HEAD: 2b32ab031e82a109e2c5b0d30ce563db0fe286b4 x86/mm/64: Make sync_global_pgds() static

The biggest change is to not sync the vmalloc and ioremap ranges for x86-64 anymore.

Thanks,

Quoted 7 lines

Ingo

------------------>
Joerg Roedel (3):
x86/mm: Pre-allocate P4D/PUD pages for vmalloc area x86/mm/64: Do not sync vmalloc/ioremap mappings x86/mm/64: Make sync_global_pgds() static

The commit 8bb9bf242d1f ("x86/mm/64: Do not sync vmalloc/ioremap mappings") causes the OOPS below, in Linus' tree and in linux-next, unearthed by my CI on <https://www.wireguard.com/build-status/>.
Bisecting reveals 8bb9bf242d1f, and reverting this makes the OOPS go away.

Jason

BUG: unable to handle page fault for address: ffffe8ffffd00608
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP
CPU: 2 PID: 22 Comm: kworker/2:0 Not tainted 5.8.0+ #154
RIP: 0010:process_one_work+0x2c/0x2d0
Code: 41 56 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 08 48 8b 06 4c 8b 67 40 49 89 c6 45 30 f6 a8 04 b8 00 00 00 00 4c 0f 44 f0 <49> 8b 46 08 44 8b a8 00 01 05
RSP: 0018:ffff88800016be98 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff888000010c00 RCX: 0000000000000000
RDX: 00000000fffedb85 RSI: ffff888000391900 RDI: ffff888000010c00
RBP: ffff888000391900 R08: ffff888000391900 R09: ffff888000350090
R10: ffffffffff770df6 R11: ffff88800f91f840 R12: ffff88800f91f140
R13: ffff88800f91f160 R14: ffffe8ffffd00600 R15: ffff88800f91f140
FS:  0000000000000000(0000) GS:ffff88800f900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffe8ffffd00608 CR3: 00000000001be005 CR4: 00000000001706a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 worker_thread+0x4b/0x3b0
 ? rescuer_thread+0x360/0x360
 kthread+0x116/0x140
 ? __kthread_create_worker+0x110/0x110
 ret_from_fork+0x1f/0x30
CR2: ffffe8ffffd00608
---[ end trace ca0cfc5a79414fb2 ]---
RIP: 0010:process_one_work+0x2c/0x2d0
Code: 41 56 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 08 48 8b 06 4c 8b 67 40 49 89 c6 45 30 f6 a8 04 b8 00 00 00 00 4c 0f 44 f0 <49> 8b 46 08 44 8b a8 00 01 05
RSP: 0018:ffff88800016be98 EFLAGS: 00010002
RAX: 0000000000000000 RBX: ffff888000010c00 RCX: 0000000000000000
RDX: 00000000fffedb85 RSI: ffff888000391900 RDI: ffff888000010c00
RBP: ffff888000391900 R08: ffff888000391900 R09: ffff888000350090
R10: ffffffffff770df6 R11: ffff88800f91f840 R12: ffff88800f91f140
R13: ffff88800f91f160 R14: ffffe8ffffd00600 R15: ffff88800f91f140
FS:  0000000000000000(0000) GS:ffff88800f900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffe8ffffd00608 CR3: 00000000001be005 CR4: 00000000001706a0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Kernel panic - not syncing: Fatal exception
#3 lore raw
Quoted 6 lines · Jason A. Donenfeld

The commit 8bb9bf242d1f ("x86/mm/64: Do not sync vmalloc/ioremap mappings") causes the OOPS below, in Linus' tree and in linux-next, unearthed by my CI on <https://www.wireguard.com/build-status/>.
Bisecting reveals 8bb9bf242d1f, and reverting this makes the OOPS go away.

The oops happens early in the function, and the "Code:" line actually gets almost the whole function prologue in it (missing first two bytes are probably "push %rbp"):

   0: 41 56                push   %r14
   2: 41 55                push   %r13
   4: 41 54                push   %r12
   6: 55                    push   %rbp
   7: 48 89 f5              mov    %rsi,%rbp
   a: 53                    push   %rbx
   b: 48 89 fb              mov    %rdi,%rbx
   e: 48 83 ec 08          sub    $0x8,%rsp
  12: 48 8b 06              mov    (%rsi),%rax
  15: 4c 8b 67 40          mov    0x40(%rdi),%r12
  19: 49 89 c6              mov    %rax,%r14
  1c: 45 30 f6              xor    %r14b,%r14b

1f: a8 04 test $0x4,%al
21: b8 00 00 00 00 mov $0x0,%eax
26: 4c 0f 44 f0 cmove %rax,%r14
2a:* 49 8b 46 08 mov 0x8(%r14),%rax <-- trapping instruction

Quoted 4 lines
BUG: unable to handle page fault for address: ffffe8ffffd00608

#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0

Yeah, missing page table because it wasn't copied.

Presumably because that kthread is using the active_mm of some random user space process that didn't get sync'ed.

And the sync_global_pgds() may have ended up being sufficient synchronization with whoever allocated thigns, even if it wasn't about the TLB contents themselves.

So apparently the "the page-table pages are all pre-allocated now" is simply not true. Joerg?

Unless somebody can figure this out fairly quickly, I think it should just be reverted.

Linus

#4 lore raw
Quoted 44 lines · Linus Torvalds
Quoted 6 lines · Jason A. Donenfeld

The commit 8bb9bf242d1f ("x86/mm/64: Do not sync vmalloc/ioremap mappings") causes the OOPS below, in Linus' tree and in linux-next, unearthed by my CI on <https://www.wireguard.com/build-status/>.
Bisecting reveals 8bb9bf242d1f, and reverting this makes the OOPS go away.

The oops happens early in the function, and the "Code:" line actually gets almost the whole function prologue in it (missing first two bytes are probably "push %rbp"):

   0: 41 56                push   %r14
   2: 41 55                push   %r13
   4: 41 54                push   %r12
   6: 55                    push   %rbp
   7: 48 89 f5              mov    %rsi,%rbp
   a: 53                    push   %rbx
   b: 48 89 fb              mov    %rdi,%rbx
   e: 48 83 ec 08          sub    $0x8,%rsp
  12: 48 8b 06              mov    (%rsi),%rax
  15: 4c 8b 67 40          mov    0x40(%rdi),%r12
  19: 49 89 c6              mov    %rax,%r14
  1c: 45 30 f6              xor    %r14b,%r14b

1f: a8 04 test $0x4,%al
21: b8 00 00 00 00 mov $0x0,%eax
26: 4c 0f 44 f0 cmove %rax,%r14
2a:* 49 8b 46 08 mov 0x8(%r14),%rax <-- trapping instruction

Quoted 4 lines
BUG: unable to handle page fault for address: ffffe8ffffd00608

#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0

Yeah, missing page table because it wasn't copied.

Presumably because that kthread is using the active_mm of some random user space process that didn't get sync'ed.

And the sync_global_pgds() may have ended up being sufficient synchronization with whoever allocated thigns, even if it wasn't about the TLB contents themselves.

Quoted 5 lines

So apparently the "the page-table pages are all pre-allocated now" is simply not true. Joerg?

Unless somebody can figure this out fairly quickly, I think it should just be reverted.

Agreed. Joerg?

Thanks,

Ingo

#5 lore raw
Quoted 4 lines · Ingo Molnar
Quoted 2 lines · Linus Torvalds

So apparently the "the page-table pages are all pre-allocated now" is simply not true. Joerg?

It pre-allocates the whole vmalloc/ioremap PUD/P4D pages, but I actually only tested it with 4-level paging, as I don't have access to 5-level paging hardware.

I'll try to make sense of that oops.

Quoted 4 lines
Quoted 2 lines

Unless somebody can figure this out fairly quickly, I think it should just be reverted.

Agreed. Joerg?

Yeah, please just revert that commit for now until I have an idea what went wrong.

Joerg

#6 lore raw
Quoted 10 lines · Joerg Roedel
Quoted 4 lines · Ingo Molnar
Quoted 2 lines · Linus Torvalds

So apparently the "the page-table pages are all pre-allocated now" is simply not true. Joerg?

It pre-allocates the whole vmalloc/ioremap PUD/P4D pages, but I actually only tested it with 4-level paging, as I don't have access to 5-level paging hardware.

I don't think Jason has either.

The

        PGD 0 P4D 0

line tells us that "pgd_present()" is true, even though PGD is 0 (otherwise it wouldn't print the P4D part). That means that he doesn't have l5 enabled.

But you may obviously have different settings for CONFIG_X86_5LEVEL, and maybe that ends up changing something?

But since apparently it's not immediately obvious what the problem is, I'll revert it for now.

Linus

#7 lore raw
Quoted 5 lines · Linus Torvalds

But you may obviously have different settings for CONFIG_X86_5LEVEL, and maybe that ends up changing something?

But since apparently it's not immediately obvious what the problem is, I'll revert it for now.

Yes, that's the best for now. My gut feeling is that the fault Jason is seeing didn't happen on a vmalloc address, but I can't prove that yet.

And if this is true it means that more work is needed before the syncing on x86-64 can be removed, so reverting is the best for now.

Jason, can you share more details about the test setup which triggers this? Like the .config and the machine setup, ideally a qemu command-line, and how to reproduce it on that setup.

Thanks,

Joerg

#8 lore raw
Quoted 3 lines · Joerg Roedel

Yes, that's the best for now. My gut feeling is that the fault Jason is seeing didn't happen on a vmalloc address, but I can't prove that yet.

No, it's definitely fairly high in the vmalloc space. Look at the faulting address:

   BUG: unable to handle page fault for address: ffffe8ffffd00608

and the code sequence is this:

Quoted 7 lines

12: 48 8b 06 mov (%rsi),%rax
15: 4c 8b 67 40 mov 0x40(%rdi),%r12

  19: 49 89 c6              mov    %rax,%r14
  1c: 45 30 f6              xor    %r14b,%r14b

1f: a8 04 test $0x4,%al
21: b8 00 00 00 00 mov $0x0,%eax
26: 4c 0f 44 f0 cmove %rax,%r14

that admittedly odd sequence is get_work_pwq(work)

And then the faulting instruction is:

Quoted 1 line

2a:* 49 8b 46 08 mov 0x8(%r14),%rax <-- trapping instruction

and this is the "->wq" dereference.

So it's the pwq->wq that traps, with 'pwq' being the trapping base pointer, and clearly being in the vmalloc space.

I think pwq may a percpu allocation, so not _directly_ vmalloc(). Adding Tejun to the cc in case he can clarify ("No, silly Linus, it's allocated here..").

Linus

#9 lore raw
Quoted 6 lines · Linus Torvalds

that admittedly odd sequence is get_work_pwq(work)

And then the faulting instruction is:

Quoted 1 line

2a:* 49 8b 46 08 mov 0x8(%r14),%rax <-- trapping instruction

Quoted 8 lines

and this is the "->wq" dereference.

So it's the pwq->wq that traps, with 'pwq' being the trapping base pointer, and clearly being in the vmalloc space.

I think pwq may a percpu allocation, so not _directly_ vmalloc(). Adding Tejun to the cc in case he can clarify ("No, silly Linus, it's allocated here..").

Hey, silly Linus, yeap, they're per-cpu allocations and will be in vmalloc address space for per-cpu workqueues. For unbound workqueues, they're regular kmallocs. The per-cpu allocation happens in alloc_and_link_pwqs():

  static int alloc_and_link_pwqs(struct workqueue_struct *wq)
  {
          bool highpri = wq->flags & WQ_HIGHPRI;
          int cpu, ret;
          if (!(wq->flags & WQ_UNBOUND)) {
                  wq->cpu_pwqs = alloc_percpu(struct pool_workqueue);
                  if (!wq->cpu_pwqs)
                          return -ENOMEM;

Thanks.

Signature · 1 line
tejun
#10 lore raw
In reply to Joerg Roedel

Hey Joerg,

Quoted 3 lines · Joerg Roedel

Jason, can you share more details about the test setup which triggers this? Like the .config and the machine setup, ideally a qemu command-line, and how to reproduce it on that setup.

make -C tools/testing/selftests/wireguard/qemu -j$(nproc)

I can also confirm that the patch you sent to the list fixes the issue.

Jason

#11 lore raw
In reply to Linus Torvalds
Quoted 22 lines · Linus Torvalds
Quoted 10 lines · Joerg Roedel
Quoted 4 lines · Ingo Molnar
Quoted 2 lines · Linus Torvalds

So apparently the "the page-table pages are all pre-allocated now" is simply not true. Joerg?

It pre-allocates the whole vmalloc/ioremap PUD/P4D pages, but I actually only tested it with 4-level paging, as I don't have access to 5-level paging hardware.

I don't think Jason has either.

The

        PGD 0 P4D 0

line tells us that "pgd_present()" is true, even though PGD is 0 (otherwise it wouldn't print the P4D part). That means that he doesn't have l5 enabled.

Quoted 5 lines

But you may obviously have different settings for CONFIG_X86_5LEVEL, and maybe that ends up changing something?

But since apparently it's not immediately obvious what the problem is, I'll revert it for now.

I've reverted it in x86/urgent as well earlier today, can send you that tree right now if you prefer that route.

Thanks,

Ingo

#12 lore raw
Quoted 2 lines · Ingo Molnar

I've reverted it in x86/urgent as well earlier today, can send you that tree right now if you prefer that route.

I sent a fix for preallocate_vmalloc_pages() to correctly pre-allocate the vmalloc PGD entries. I verified that it works and that swapper_pg_dir contains the correct entries now. This should also fix the issue Jason is seeing.

Sorry for screwing this up :-(

Regards,

Joerg

#13 lore raw
Quoted 8 lines · Joerg Roedel
Quoted 2 lines · Ingo Molnar

I've reverted it in x86/urgent as well earlier today, can send you that tree right now if you prefer that route.

I sent a fix for preallocate_vmalloc_pages() to correctly pre-allocate the vmalloc PGD entries. I verified that it works and that swapper_pg_dir contains the correct entries now. This should also fix the issue Jason is seeing.

Thanks!

There's one thing left to do. Linus has reverted the patch which exposed this bug:

7b4ea9456dd3: ("Revert "x86/mm/64: Do not sync vmalloc/ioremap mappings"")

and has applied your fix:

995909a4e22b: ("x86/mm/64: Do not dereference non-present PGD entries")

I think now we can re-apply the original commit:

8bb9bf242d1f: ("x86/mm/64: Do not sync vmalloc/ioremap mappings")

Mind re-sending it, with an updated changelog that explains why it's now truly safe?

Would be tentatively scheduled for v5.10 though, we've had enough excitement in this area for v5.9 I think. :-/

Quoted 1 line

Sorry for screwing this up :-(

No problem, and it was my fault too: I sent 8bb9bf242d1f to Linus too quickly, just 7 days after applying it - x86/mm patches usually need a few weeks of testing.

Thanks,

Ingo

#14 lore raw
Quoted 3 lines · Ingo Molnar

Would be tentatively scheduled for v5.10 though, we've had enough excitement in this area for v5.9 I think. :-/

I think I can take it for 5.9 again if you send a pull request my way.

If it causes any other problems, we'll obviously revert it again and at that point it's final for 5.9.

But the one report it got was fixed early and quickly enough that I don't feel like things were _horribly_ broken, just a mistake that wasn't something horribly fundamental that needs another release to think about.

Linus

#15 lore raw
In reply to Ingo Molnar

Hi Ingo,

Quoted 2 lines · Ingo Molnar

Mind re-sending it, with an updated changelog that explains why it's now truly safe?

I have updated the commit and will send it out later today, maybe together with a comment update in the preallocate_vmalloc_pages() function.

Thanks,

Joerg

#16 lore raw
In reply to Jason A. Donenfeld
Quoted 1 line · Jason A. Donenfeld
BUG: unable to handle page fault for address: ffffe8ffffd00608

Okay, looks like my usage of the page-table macros is bogus, seems I don't understand their usage as good as I thought. The p?d_none checks in the allocation path are wrong and led to the bug. In fact it caused only the first PUD entry to be allocated and in the later iterations of the loop it always ended up on the same PUD entry.

I still don't fully understand why, but its late here and my head spins. So I take another look tomorrow in the hope to understand it better. Please remind me to not take vacation again during merge windows :)

Anyway...

Jason, does the attached diff fix the issue in your testing? For me it makes all PUD pages correctly allocated.

arch/x86/mm/init.c +1 -0
index c7a47603537f..e4abf73167d0 100644
@@ -696,6 +696,7 @@ static void __init memory_map_bottom_up(unsigned long map_start,
 static void __init init_trampoline(void)
 {
 #ifdef CONFIG_X86_64
+
 	if (!kaslr_memory_enabled())
 		trampoline_pgd_entry = init_top_pgt[pgd_index(__PAGE_OFFSET)];
 	else
arch/x86/mm/init_64.c +12 -16
index e65b96f381a7..351fac590b02 100644
@@ -1248,27 +1248,23 @@ static void __init preallocate_vmalloc_pages(void)
 		p4d_t *p4d;
 		pud_t *pud;
 
-		p4d = p4d_offset(pgd, addr);
-		if (p4d_none(*p4d)) {
-			/* Can only happen with 5-level paging */
-			p4d = p4d_alloc(&init_mm, pgd, addr);
-			if (!p4d) {
-				lvl = "p4d";
-				goto failed;
-			}
+		p4d = p4d_alloc(&init_mm, pgd, addr);
+		if (!p4d) {
+			lvl = "p4d";
+			goto failed;
 		}
 
 		if (pgtable_l5_enabled())
 			continue;
 
-		pud = pud_offset(p4d, addr);
-		if (pud_none(*pud)) {
-			/* Ends up here only with 4-level paging */
-			pud = pud_alloc(&init_mm, p4d, addr);
-			if (!pud) {
-				lvl = "pud";
-				goto failed;
-			}
+		/*
+		 * With 4-level paging the P4D is folded, so allocate a
+		 * PUD to have one level below PGD present.
+		 */
+		pud = pud_alloc(&init_mm, p4d, addr);
+		if (!pud) {
+			lvl = "pud";
+			goto failed;
 		}
 	}
 
#17 lore raw
Quoted 2 lines · Joerg Roedel

I still don't fully understand why, but its late here and my head spins. So I take another look tomorrow in the hope to understand it better.

I think I know what the bug is, I am calling p4d_offset(pgd, addr) without checking for pgd_none() first. That is still a bug in the code wich needs fixing, even with the revert.

Joerg