Skip to contentSkip to conversation

Xen PV domain regression with KASLR enabled (kernel 3.16)

PATCH In review Stefan Bader 7 participants 28 messages Aug 8, 2014 – Sep 3, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

-Stefan

19:35:02 [ 2.547049] ------------[ cut here ]------------
19:35:02 [ 2.547065] WARNING: CPU: 0 PID: 97 at /build/buildd/linux-3.16.0/mm/vmalloc.c:128 vmap_page_range_noflush+0x2d1/0x370() 19:35:02 [ 2.547069] Modules linked in:
19:35:02 [ 2.547073] CPU: 0 PID: 97 Comm: systemd-udevd Not tainted 3.16.0-6-generic #11-Ubuntu
19:35:02 [ 2.547077] 0000000000000009 ffff880002defb98 ffffffff81755538 0000000000000000
19:35:02 [ 2.547082] ffff880002defbd0 ffffffff8106bb0d ffff88000400ec88 0000000000000001
19:35:02 [ 2.547086] ffff880002fcfb00 ffffffffc0391000 0000000000000000 ffff880002defbe0
19:35:02 [ 2.547090] Call Trace:
19:35:02 [ 2.547096] [<ffffffff81755538>] dump_stack+0x45/0x56 19:35:02 [ 2.547101] [<ffffffff8106bb0d>] warn_slowpath_common+0x7d/0xa0 19:35:02 [ 2.547104] [<ffffffff8106bbea>] warn_slowpath_null+0x1a/0x20 19:35:02 [ 2.547108] [<ffffffff81197c31>] vmap_page_range_noflush+0x2d1/0x370 19:35:02 [ 2.547112] [<ffffffff81197cfe>] map_vm_area+0x2e/0x40 19:35:02 [ 2.547115] [<ffffffff8119a058>] __vmalloc_node_range+0x188/0x280 19:35:02 [ 2.547120] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547124] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547129] [<ffffffff8104f294>] module_alloc+0x74/0xd0 19:35:02 [ 2.547132] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547135] [<ffffffff810e92b4>] module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547146] [<ffffffff810e9a6c>] layout_and_allocate+0x74c/0xc70 19:35:02 [ 2.547149] [<ffffffff810ea063>] load_module+0xd3/0x1b70 19:35:02 [ 2.547154] [<ffffffff811cfeb1>] ? vfs_read+0xf1/0x170 19:35:02 [ 2.547157] [<ffffffff810e7aa1>] ? copy_module_from_fd.isra.46+0x121/0x180 19:35:02 [ 2.547161] [<ffffffff810ebc76>] SyS_finit_module+0x86/0xb0 19:35:02 [ 2.547167] [<ffffffff8175de7f>] tracesys+0xe1/0xe6
19:35:02 [ 2.547169] ---[ end trace 8a5de7fc66e75fe4 ]---
19:35:02 [ 2.547172] vmalloc: allocation failure, allocated 20480 of 24576 bytes 19:35:02 [ 2.547175] systemd-udevd: page allocation failure: order:0, mode:0xd2

[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350522

#1 lore raw
Quoted 5 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Quoted 8 lines

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

#2 lore raw
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

-Stefan

#3 lore raw
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

-Kees

Signature · 2 lines
Kees Cook
Chrome OS Security
#4 lore raw
Quoted 30 lines · Kees Cook
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Quoted 6 lines

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

-Kees

Making my little steps towards more understanding I figured out that it isn't the code that does the relocation. Even with that completely disabled there were the vmalloc issues. What causes it seems to be the default of the upper limit and that this changes the split between kernel and modules to 1G+1G instead of 512M+1.5G. That is the reason why nokaslr has no effect.

-Stefan

#5 lore raw
Quoted 35 lines · Stefan Bader
Quoted 34 lines · Kees Cook
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

Quoted 7 lines
Quoted 2 lines

-Kees

Making my little steps towards more understanding I figured out that it isn't the code that does the relocation. Even with that completely disabled there were the vmalloc issues. What causes it seems to be the default of the upper limit and that this changes the split between kernel and modules to 1G+1G instead of 512M+1.5G. That is the reason why nokaslr has no effect.

Oh! That's very interesting. There must be some assumption in Xen about the kernel VM layout then?

-Kees

Signature · 2 lines
Kees Cook
Chrome OS Security
#6 lore raw
Quoted 39 lines · Kees Cook
Quoted 37 lines · Stefan Bader
Quoted 36 lines · Kees Cook
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

-Kees

Quoted 8 lines
Quoted 5 lines

Making my little steps towards more understanding I figured out that it isn't the code that does the relocation. Even with that completely disabled there were the vmalloc issues. What causes it seems to be the default of the upper limit and that this changes the split between kernel and modules to 1G+1G instead of 512M+1.5G. That is the reason why nokaslr has no effect.

Oh! That's very interesting. There must be some assumption in Xen about the kernel VM layout then?

No. I think most of the changes that look at PTE and PMDs are are all in arch/x86/xen/mmu.c. I wonder if this is xen_cleanhighmap being too aggressive

Quoted 11 lines

-Kees

Signature · 7 lines
Kees Cook
Chrome OS Security

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
#7 lore raw
Quoted 46 lines · Konrad Rzeszutek Wilk
Quoted 45 lines · Kees Cook
Quoted 42 lines · Stefan Bader
Quoted 30 lines · Kees Cook
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Quoted 6 lines

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

-Kees

Making my little steps towards more understanding I figured out that it isn't the code that does the relocation. Even with that completely disabled there were the vmalloc issues. What causes it seems to be the default of the upper limit and that this changes the split between kernel and modules to 1G+1G instead of 512M+1.5G. That is the reason why nokaslr has no effect.

Quoted 6 lines
Quoted 2 lines

Oh! That's very interesting. There must be some assumption in Xen about the kernel VM layout then?

No. I think most of the changes that look at PTE and PMDs are are all in arch/x86/xen/mmu.c. I wonder if this is xen_cleanhighmap being too aggressive

(Sorry I had to cut our chat short at Kernel Summit!)

I sounded like there was another region of memory that Xen was setting aside for page tables? But Stefan's investigation seems to show this isn't about layout at boot (since the kaslr=0 case means no relocation is done). Sounds more like the split between kernel and modules area, so I'm not sure how the memory area after the initrd would be part of this. What should next steps be, do you think?

-Kees

Quoted 11 lines
Quoted 11 lines

-Kees

--
Kees Cook
Chrome OS Security

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Signature · 2 lines
Kees Cook
Chrome OS Security
#8 lore raw
Quoted 55 lines · Kees Cook
Quoted 46 lines · Konrad Rzeszutek Wilk
Quoted 45 lines · Kees Cook
Quoted 42 lines · Stefan Bader
Quoted 30 lines · Kees Cook
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Quoted 6 lines

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

-Kees

Making my little steps towards more understanding I figured out that it isn't the code that does the relocation. Even with that completely disabled there were the vmalloc issues. What causes it seems to be the default of the upper limit and that this changes the split between kernel and modules to 1G+1G instead of 512M+1.5G. That is the reason why nokaslr has no effect.

Quoted 6 lines
Quoted 2 lines

Oh! That's very interesting. There must be some assumption in Xen about the kernel VM layout then?

No. I think most of the changes that look at PTE and PMDs are are all in arch/x86/xen/mmu.c. I wonder if this is xen_cleanhighmap being too aggressive

Quoted 8 lines

(Sorry I had to cut our chat short at Kernel Summit!)

I sounded like there was another region of memory that Xen was setting aside for page tables? But Stefan's investigation seems to show this isn't about layout at boot (since the kaslr=0 case means no relocation is done). Sounds more like the split between kernel and modules area, so I'm not sure how the memory area after the initrd would be part of this. What should next steps be, do you think?

Maybe layout, but not about placement of the kernel. Basically leaving KASLR enabled but shrink the possible range back to the original kernel/module split is fine as well.

I am bouncing between feeling close to understand to being confused. Konrad suggested xen_cleanhighmap being overly aggressive. But maybe its the other way round. The warning that occurs first indicates that PTE that was obtained for some vmalloc mapping is not unused (0) as it is expected. So it feels rather like some cleanup has *not* been done.

Let me think aloud a bit... What seems to cause this, is the change of the kernel/module split from 512M:1.5G to 1G:1G (not exactly since there is 8M vsyscalls and 2M hole at the end). Which in vaddr terms means:

Before:
ffffffff80000000 - ffffffff9fffffff (=512 MB) kernel text mapping, from phys 0 ffffffffa0000000 - ffffffffff5fffff (=1526 MB) module mapping space

After:
ffffffff80000000 - ffffffffbfffffff (=1024 MB) kernel text mapping, from phys 0 ffffffffc0000000 - ffffffffff5fffff (=1014 MB) module mapping space

Now, *if* I got this right, this means the kernel starts on a vaddr that is pointed at by:

PGD[510]->PUD[510]->PMD[0]->PTE[0]

In the old layout the module vaddr area would start in the same PUD area, but with the change the kernel would cover PUD[510] and the module vaddr + vsyscalls and the hole would cover PUD[511].

xen_cleanhighmap operates only on the kernel_level2_pgt which (speculating a bit since I am not sure I understand enough details) I believe is the one PMD pointed at by PGD[510]->PUD[510]. That could mean that before the change xen_cleanhighmap may touch some (the initial 512M) of the module vaddr space but not after the change. Maybe that also means it always should have covered more but this would not be observed as long as modules would not claim more than 512M? I still need to check the vaddr ranges for which xen_cleanhighmap is actually called. The modules vaddr space would normally not be touched (only with DEBUG set). I moved that to be unconditionally done but then this might be of no use when it needs to cover a different PMD...

Really not sure here. But maybe a starter for others...

-Stefan

Quoted 18 lines

-Kees

Quoted 11 lines
Quoted 11 lines

-Kees

--
Kees Cook
Chrome OS Security

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#9 lore raw
Quoted 87 lines · Stefan Bader
Quoted 55 lines · Kees Cook
Quoted 46 lines · Konrad Rzeszutek Wilk
Quoted 45 lines · Kees Cook
Quoted 42 lines · Stefan Bader
Quoted 30 lines · Kees Cook
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Quoted 6 lines

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

-Kees

Making my little steps towards more understanding I figured out that it isn't the code that does the relocation. Even with that completely disabled there were the vmalloc issues. What causes it seems to be the default of the upper limit and that this changes the split between kernel and modules to 1G+1G instead of 512M+1.5G. That is the reason why nokaslr has no effect.

Quoted 6 lines
Quoted 2 lines

Oh! That's very interesting. There must be some assumption in Xen about the kernel VM layout then?

No. I think most of the changes that look at PTE and PMDs are are all in arch/x86/xen/mmu.c. I wonder if this is xen_cleanhighmap being too aggressive

Quoted 8 lines

(Sorry I had to cut our chat short at Kernel Summit!)

I sounded like there was another region of memory that Xen was setting aside for page tables? But Stefan's investigation seems to show this isn't about layout at boot (since the kaslr=0 case means no relocation is done). Sounds more like the split between kernel and modules area, so I'm not sure how the memory area after the initrd would be part of this. What should next steps be, do you think?

Maybe layout, but not about placement of the kernel. Basically leaving KASLR enabled but shrink the possible range back to the original kernel/module split is fine as well.

I am bouncing between feeling close to understand to being confused. Konrad suggested xen_cleanhighmap being overly aggressive. But maybe its the other way round. The warning that occurs first indicates that PTE that was obtained for some vmalloc mapping is not unused (0) as it is expected. So it feels rather like some cleanup has *not* been done.

Let me think aloud a bit... What seems to cause this, is the change of the kernel/module split from 512M:1.5G to 1G:1G (not exactly since there is 8M vsyscalls and 2M hole at the end). Which in vaddr terms means:

Before:
ffffffff80000000 - ffffffff9fffffff (=512 MB) kernel text mapping, from phys 0 ffffffffa0000000 - ffffffffff5fffff (=1526 MB) module mapping space

After:
ffffffff80000000 - ffffffffbfffffff (=1024 MB) kernel text mapping, from phys 0 ffffffffc0000000 - ffffffffff5fffff (=1014 MB) module mapping space

Quoted 8 lines

Now, *if* I got this right, this means the kernel starts on a vaddr that is pointed at by:

PGD[510]->PUD[510]->PMD[0]->PTE[0]

In the old layout the module vaddr area would start in the same PUD area, but with the change the kernel would cover PUD[510] and the module vaddr + vsyscalls and the hole would cover PUD[511].

I think there is a fixmap there too?

Quoted 4 lines

xen_cleanhighmap operates only on the kernel_level2_pgt which (speculating a bit since I am not sure I understand enough details) I believe is the one PMD pointed at by PGD[510]->PUD[510]. That could mean that before the change

That sounds right.

I don't know if you saw:

1248 #ifdef DEBUG
1249 /* This is superflous and is not neccessary, but you know what
1250 * lets do it. The MODULES_VADDR -> MODULES_END should be clear of
1251 * anything at this stage. */
1252 xen_cleanhighmap(MODULES_VADDR, roundup(MODULES_VADDR, PUD_SIZE) - 1);
1253 #endif
1254 }

Which was me being a bit paranoid and figured it might help in troubleshooting. If you disable that does it work?

Quoted 7 lines

xen_cleanhighmap may touch some (the initial 512M) of the module vaddr space but not after the change. Maybe that also means it always should have covered more but this would not be observed as long as modules would not claim more than 512M? I still need to check the vaddr ranges for which xen_cleanhighmap is actually called. The modules vaddr space would normally not be touched (only with DEBUG set). I moved that to be unconditionally done but then this might be of no use when it needs to cover a different PMD...

What does the toolstack say in regards to allocating the memory? It is pretty verbose (domainloginfo..something) in printing out the vaddr of where it stashes the kernel, ramdisk, P2M, and the pagetables (which of course need to fit all within the 512MB, now 1GB area).

Quoted 25 lines

Really not sure here. But maybe a starter for others...

-Stefan

Quoted 18 lines

-Kees

Quoted 11 lines
Quoted 11 lines

-Kees

--
Kees Cook
Chrome OS Security

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#10 lore raw
Quoted 91 lines · Konrad Rzeszutek Wilk
Quoted 90 lines · Stefan Bader
Quoted 55 lines · Kees Cook
Quoted 46 lines · Konrad Rzeszutek Wilk
Quoted 45 lines · Kees Cook
Quoted 42 lines · Stefan Bader
Quoted 30 lines · Kees Cook
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Quoted 6 lines

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

-Kees

Making my little steps towards more understanding I figured out that it isn't the code that does the relocation. Even with that completely disabled there were the vmalloc issues. What causes it seems to be the default of the upper limit and that this changes the split between kernel and modules to 1G+1G instead of 512M+1.5G. That is the reason why nokaslr has no effect.

Quoted 6 lines
Quoted 2 lines

Oh! That's very interesting. There must be some assumption in Xen about the kernel VM layout then?

No. I think most of the changes that look at PTE and PMDs are are all in arch/x86/xen/mmu.c. I wonder if this is xen_cleanhighmap being too aggressive

Quoted 8 lines

(Sorry I had to cut our chat short at Kernel Summit!)

I sounded like there was another region of memory that Xen was setting aside for page tables? But Stefan's investigation seems to show this isn't about layout at boot (since the kaslr=0 case means no relocation is done). Sounds more like the split between kernel and modules area, so I'm not sure how the memory area after the initrd would be part of this. What should next steps be, do you think?

Maybe layout, but not about placement of the kernel. Basically leaving KASLR enabled but shrink the possible range back to the original kernel/module split is fine as well.

I am bouncing between feeling close to understand to being confused. Konrad suggested xen_cleanhighmap being overly aggressive. But maybe its the other way round. The warning that occurs first indicates that PTE that was obtained for some vmalloc mapping is not unused (0) as it is expected. So it feels rather like some cleanup has *not* been done.

Let me think aloud a bit... What seems to cause this, is the change of the kernel/module split from 512M:1.5G to 1G:1G (not exactly since there is 8M vsyscalls and 2M hole at the end). Which in vaddr terms means:

Before:
ffffffff80000000 - ffffffff9fffffff (=512 MB) kernel text mapping, from phys 0 ffffffffa0000000 - ffffffffff5fffff (=1526 MB) module mapping space

After:
ffffffff80000000 - ffffffffbfffffff (=1024 MB) kernel text mapping, from phys 0 ffffffffc0000000 - ffffffffff5fffff (=1014 MB) module mapping space

Now, *if* I got this right, this means the kernel starts on a vaddr that is pointed at by:

Quoted 7 lines
Quoted 5 lines

PGD[510]->PUD[510]->PMD[0]->PTE[0]

In the old layout the module vaddr area would start in the same PUD area, but with the change the kernel would cover PUD[510] and the module vaddr + vsyscalls and the hole would cover PUD[511].

I think there is a fixmap there too?

Right, they forgot that in Documentation/x86/x86_64/mm... but head_64.S has it. So fixmap seems to be in the 2M space before the vsyscalls.
Btw, apparently I got the PGD index wrong. It is of course 511, not 510.

init_level4_pgt[511]->level3_kernel_pgt[510]->level2_kernel_pgt[0..255]->kernel

                                                               [256..511]->mod
                                       [511]->level2_fixmap_pgt[0..505]->mod
                                                               [506]->fixmap
                                                               [507..510]->vsysc
                                                               [511]->hole

With the change being level2_kernel_pgt completely covering kernel only.

Quoted 9 lines
Quoted 4 lines

xen_cleanhighmap operates only on the kernel_level2_pgt which (speculating a bit since I am not sure I understand enough details) I believe is the one PMD pointed at by PGD[510]->PUD[510]. That could mean that before the change

That sounds right.

I don't know if you saw:

Quoted 7 lines

1248 #ifdef DEBUG
1249 /* This is superflous and is not neccessary, but you know what
1250 * lets do it. The MODULES_VADDR -> MODULES_END should be clear of
1251 * anything at this stage. */
1252 xen_cleanhighmap(MODULES_VADDR, roundup(MODULES_VADDR, PUD_SIZE) - 1);
1253 #endif
1254 }

I saw that but it would have no effect, even with running it. Because xen_cleanhighmap clamps the pmds it walks over to the kernel_level2_pgt page. Now MODULES_VADDR is mapped only from level2_fixmap_pgt.
Even with the old layout it might do less that anticipated as it would only cover 512M and stop then. But I think it really does not matter.

Quoted 12 lines

Which was me being a bit paranoid and figured it might help in troubleshooting. If you disable that does it work?

Quoted 7 lines

xen_cleanhighmap may touch some (the initial 512M) of the module vaddr space but not after the change. Maybe that also means it always should have covered more but this would not be observed as long as modules would not claim more than 512M? I still need to check the vaddr ranges for which xen_cleanhighmap is actually called. The modules vaddr space would normally not be touched (only with DEBUG set). I moved that to be unconditionally done but then this might be of no use when it needs to cover a different PMD...

Quoted 4 lines

What does the toolstack say in regards to allocating the memory? It is pretty verbose (domainloginfo..something) in printing out the vaddr of where it stashes the kernel, ramdisk, P2M, and the pagetables (which of course need to fit all within the 512MB, now 1GB area).

That is taken from starting a 2G PV domU with pvgrub (not pygrub):

Xen Minimal OS!
start_info: 0xd90000(VA)

    nr_pages: 0x80000
  shared_inf: 0xdfe92000(MA)
     pt_base: 0xd93000(VA)
nr_pt_frames: 0xb
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line:
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x6000d(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80b00(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: da1
    max_pfn: 80000

Mapping memory range 0x1000000 - 0x80000000
setting 0x0-0x78000 readonly

For a moment I was puzzled by the use of max_pfn_mapped in the generic cleanup_highmap function of 64bit x86. It limits the cleanup to the start of the mfn_list. And the max_pfn_mapped value changes soon after to reflect the total amount of memory of the guest.
Making a copy showed it to be around 51M at the time of cleanup. That initially looks suspect but Xen already replaced the page tables. The compile-time variants would have 2M large pages on the whole level2_kernel_pgt range. But as far as I can see, the Xen provided ones don't put in mappings for anything beyond the provided boot stack which is clean in the xen_cleanhighmap.

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

-Stefan

Quoted 28 lines
Quoted 26 lines

Really not sure here. But maybe a starter for others...

-Stefan

Quoted 18 lines

-Kees

Quoted 11 lines
Quoted 11 lines

-Kees

--
Kees Cook
Chrome OS Security

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#11 lore raw
Quoted 189 lines · Stefan Bader
Quoted 91 lines · Konrad Rzeszutek Wilk
Quoted 90 lines · Stefan Bader
Quoted 55 lines · Kees Cook
Quoted 46 lines · Konrad Rzeszutek Wilk
Quoted 45 lines · Kees Cook
Quoted 42 lines · Stefan Bader
Quoted 30 lines · Kees Cook
Quoted 21 lines · Stefan Bader
Quoted 12 lines · David Vrabel
Quoted 11 lines · Stefan Bader

Unfortunately I have not yet figured out why this happens, but can confirm by compiling with or without CONFIG_RANDOMIZE_BASE being set that without KASLR all is ok, but with it enabled there are issues (actually a dom0 does not even boot as a follow up error).

Details can be seen in [1] but basically this is always some portion of a vmalloc allocation failing after hitting a freshly allocated PTE space not being PTE_NONE (usually from a module load triggered by systemd-udevd). In the non-dom0 case this repeats many times but ends in a guest that allows login. In the dom0 case there is a more fatal error at some point causing a crash.

Quoted 8 lines
Quoted 2 lines

I have not tried this for a normal PV guest but for dom0 it also does not help to add "nokaslr" to the kernel command-line.

Maybe it's overlapping with regions of the virtual address space reserved for Xen? What the the VA that fails?

David

Quoted 7 lines

Yeah, there is some code to avoid some regions of memory (like initrd). Maybe missing p2m tables? I probably need to add debugging to find the failing VA (iow not sure whether it might be somewhere in the stacktraces in the report).

The kernel-command line does not seem to be looked at. It should put something into dmesg and that never shows up. Also today's random feature is other PV guests crashing after a bit somewhere in the check_for_corruption area...

Quoted 6 lines

Right now, the kaslr code just deals with initrd, cmdline, etc. If there are other reserved regions that aren't listed in the e820, it'll need to locate and skip them.

-Kees

Making my little steps towards more understanding I figured out that it isn't the code that does the relocation. Even with that completely disabled there were the vmalloc issues. What causes it seems to be the default of the upper limit and that this changes the split between kernel and modules to 1G+1G instead of 512M+1.5G. That is the reason why nokaslr has no effect.

Quoted 6 lines
Quoted 2 lines

Oh! That's very interesting. There must be some assumption in Xen about the kernel VM layout then?

No. I think most of the changes that look at PTE and PMDs are are all in arch/x86/xen/mmu.c. I wonder if this is xen_cleanhighmap being too aggressive

Quoted 8 lines

(Sorry I had to cut our chat short at Kernel Summit!)

I sounded like there was another region of memory that Xen was setting aside for page tables? But Stefan's investigation seems to show this isn't about layout at boot (since the kaslr=0 case means no relocation is done). Sounds more like the split between kernel and modules area, so I'm not sure how the memory area after the initrd would be part of this. What should next steps be, do you think?

Maybe layout, but not about placement of the kernel. Basically leaving KASLR enabled but shrink the possible range back to the original kernel/module split is fine as well.

I am bouncing between feeling close to understand to being confused. Konrad suggested xen_cleanhighmap being overly aggressive. But maybe its the other way round. The warning that occurs first indicates that PTE that was obtained for some vmalloc mapping is not unused (0) as it is expected. So it feels rather like some cleanup has *not* been done.

Let me think aloud a bit... What seems to cause this, is the change of the kernel/module split from 512M:1.5G to 1G:1G (not exactly since there is 8M vsyscalls and 2M hole at the end). Which in vaddr terms means:

Before:
ffffffff80000000 - ffffffff9fffffff (=512 MB) kernel text mapping, from phys 0 ffffffffa0000000 - ffffffffff5fffff (=1526 MB) module mapping space

After:
ffffffff80000000 - ffffffffbfffffff (=1024 MB) kernel text mapping, from phys 0 ffffffffc0000000 - ffffffffff5fffff (=1014 MB) module mapping space

Now, *if* I got this right, this means the kernel starts on a vaddr that is pointed at by:

Quoted 7 lines
Quoted 5 lines

PGD[510]->PUD[510]->PMD[0]->PTE[0]

In the old layout the module vaddr area would start in the same PUD area, but with the change the kernel would cover PUD[510] and the module vaddr + vsyscalls and the hole would cover PUD[511].

I think there is a fixmap there too?

Right, they forgot that in Documentation/x86/x86_64/mm... but head_64.S has it. So fixmap seems to be in the 2M space before the vsyscalls.
Btw, apparently I got the PGD index wrong. It is of course 511, not 510.

init_level4_pgt[511]->level3_kernel_pgt[510]->level2_kernel_pgt[0..255]->kernel

                                                               [256..511]->mod
                                       [511]->level2_fixmap_pgt[0..505]->mod
                                                               [506]->fixmap
                                                               [507..510]->vsysc
                                                               [511]->hole

With the change being level2_kernel_pgt completely covering kernel only.

Quoted 9 lines
Quoted 4 lines

xen_cleanhighmap operates only on the kernel_level2_pgt which (speculating a bit since I am not sure I understand enough details) I believe is the one PMD pointed at by PGD[510]->PUD[510]. That could mean that before the change

That sounds right.

I don't know if you saw:

Quoted 7 lines

1248 #ifdef DEBUG
1249 /* This is superflous and is not neccessary, but you know what
1250 * lets do it. The MODULES_VADDR -> MODULES_END should be clear of
1251 * anything at this stage. */
1252 xen_cleanhighmap(MODULES_VADDR, roundup(MODULES_VADDR, PUD_SIZE) - 1);
1253 #endif
1254 }

I saw that but it would have no effect, even with running it. Because xen_cleanhighmap clamps the pmds it walks over to the kernel_level2_pgt page. Now MODULES_VADDR is mapped only from level2_fixmap_pgt.
Even with the old layout it might do less that anticipated as it would only cover 512M and stop then. But I think it really does not matter.

Quoted 12 lines

Which was me being a bit paranoid and figured it might help in troubleshooting. If you disable that does it work?

Quoted 7 lines

xen_cleanhighmap may touch some (the initial 512M) of the module vaddr space but not after the change. Maybe that also means it always should have covered more but this would not be observed as long as modules would not claim more than 512M? I still need to check the vaddr ranges for which xen_cleanhighmap is actually called. The modules vaddr space would normally not be touched (only with DEBUG set). I moved that to be unconditionally done but then this might be of no use when it needs to cover a different PMD...

Quoted 4 lines

What does the toolstack say in regards to allocating the memory? It is pretty verbose (domainloginfo..something) in printing out the vaddr of where it stashes the kernel, ramdisk, P2M, and the pagetables (which of course need to fit all within the 512MB, now 1GB area).

That is taken from starting a 2G PV domU with pvgrub (not pygrub):

Xen Minimal OS!
start_info: 0xd90000(VA)

    nr_pages: 0x80000
  shared_inf: 0xdfe92000(MA)
     pt_base: 0xd93000(VA)
nr_pt_frames: 0xb
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line:
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x6000d(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80b00(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: da1
    max_pfn: 80000

Mapping memory range 0x1000000 - 0x80000000
setting 0x0-0x78000 readonly

For a moment I was puzzled by the use of max_pfn_mapped in the generic cleanup_highmap function of 64bit x86. It limits the cleanup to the start of the mfn_list. And the max_pfn_mapped value changes soon after to reflect the total amount of memory of the guest.
Making a copy showed it to be around 51M at the time of cleanup. That initially looks suspect but Xen already replaced the page tables. The compile-time variants would have 2M large pages on the whole level2_kernel_pgt range. But as far as I can see, the Xen provided ones don't put in mappings for anything beyond the provided boot stack which is clean in the xen_cleanhighmap.

Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

#12 lore raw

PATCH 1/1 Solved the Xen PV/KASLR riddle

Quoted 8 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

So at some point I realized that level2_fixmap_pgt seemed to contain an oddly high number of entries (given that the virtual address that failed would be mapped by entry 0). And suddenly I realized that apart from entries 506 and 507 (actual fixmap and vsyscalls) the whole list actually was the same as level2_kernel_gpt (without the first 16M cleared).

And then I realized that xen_setup_kernel_pagetable is wrong to a degree which makes one wonder how this ever worked. Adding PMD_SIZE as an offset (2M) isn't changing l2 at all. This just copies Xen's kernel mapping, AGAIN!.

I guess we all just were lucky that in most cases modules would not use more than 512M (which is the correctly cleaned up remainder of kernel_level2_pgt)...

I still need to compile a kernel with the patch and the old layout but I kind of got exited by the find. At least this is tested with the 1G/~1G layout and it comes up without vmalloc errors.

-Stefan

Quoted 1 line

From 4b9a9a45177284e29d345eb54c545bd1da718e1b Mon Sep 17 00:00:00 2001

From: Stefan Bader <[email protected]>
Date: Thu, 28 Aug 2014 19:17:00 +0200
Subject: [PATCH] x86/xen: Fix setup of 64bit kernel pagetables

This seemed to be one of those what-the-heck moments. When trying to figure out why changing the kernel/module split (which enabling KASLR does) causes vmalloc to run wild on boot of 64bit PV guests, after much scratching my head, found that the current Xen code copies the same L2 table not only to the level2_ident_pgt and level2_kernel_pgt, but also (due to miscalculating the offset) to level2_fixmap_pgt.

This only worked because the normal kernel image size only covers the first half of level2_kernel_pgt and module space starts after that. With the split changing, the kernel image uses the full PUD range of 1G and module space starts in the level2_fixmap_pgt. So basically:

L4[511]->level3_kernel_pgt[510]->level2_kernel_pgt

                          [511]->level2_fixmap_pgt

And now the incorrect copy of the kernel mapping in that range bites (hard).

This change might not be the fully correct approach as it basically removes the pre-set page table entry for the fixmap that is compile time set (level2_fixmap_pgt[506]->level1_fixmap_pgt). For one the level1 page table is not yet declared in C headers (that might be fixed). But also with the current bug, it was removed, too. Since the Xen mappings for level2_kernel_pgt only covered kernel + initrd and some Xen data this did never reach that far. And still, something does create entries at level2_fixmap_pgt[506..507]. So it should be ok. At least I was able to successfully boot a kernel with 1G kernel image size without any vmalloc whinings.

---
 arch/x86/xen/mmu.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
arch/x86/xen/mmu.c +17 -9
index e8a1201..803034c 100644
@@ -1902,8 +1902,22 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
 		/* L3_i[0] -> level2_ident_pgt */
 		convert_pfn_mfn(level3_ident_pgt);
 		/* L3_k[510] -> level2_kernel_pgt
-		 * L3_i[511] -> level2_fixmap_pgt */
+		 * L3_k[511] -> level2_fixmap_pgt */
 		convert_pfn_mfn(level3_kernel_pgt);
+
+		/* level2_fixmap_pgt contains a single entry for the
+		 * fixmap area at offset 506. The correct way would
+		 * be to convert level2_fixmap_pgt to mfn and set the
+		 * level1_fixmap_pgt (which is completely empty) to RO,
+		 * too. But currently this page table is not delcared,
+		 * so it would be a bit of voodoo to get its address.
+		 * And also the fixmap entry was never set anyway due
+		 * to using the wrong l2 when getting Xen's tables.
+		 * So let's just nuke it.
+		 * This orphans level1_fixmap_pgt, but that should be
+		 * as it always has been.
+		 */
+		memset(level2_fixmap_pgt, 0, 512*sizeof(long));
 	}
 	/* We get [511][511] and have Xen's version of level2_kernel_pgt */
 	l3 = m2v(pgd[pgd_index(__START_KERNEL_map)].pgd);
@@ -1913,21 +1927,15 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
 	addr[1] = (unsigned long)l3;
 	addr[2] = (unsigned long)l2;
 	/* Graft it onto L4[272][0]. Note that we creating an aliasing problem:
-	 * Both L4[272][0] and L4[511][511] have entries that point to the same
+	 * Both L4[272][0] and L4[511][510] have entries that point to the same
 	 * L2 (PMD) tables. Meaning that if you modify it in __va space
 	 * it will be also modified in the __ka space! (But if you just
 	 * modify the PMD table to point to other PTE's or none, then you
 	 * are OK - which is what cleanup_highmap does) */
 	copy_page(level2_ident_pgt, l2);
-	/* Graft it onto L4[511][511] */
+	/* Graft it onto L4[511][510] */
 	copy_page(level2_kernel_pgt, l2);
 
-	/* Get [511][510] and graft that in level2_fixmap_pgt */
-	l3 = m2v(pgd[pgd_index(__START_KERNEL_map + PMD_SIZE)].pgd);
-	l2 = m2v(l3[pud_index(__START_KERNEL_map + PMD_SIZE)].pud);
-	copy_page(level2_fixmap_pgt, l2);
-	/* Note that we don't do anything with level1_fixmap_pgt which
-	 * we don't need. */
 	if (!xen_feature(XENFEAT_auto_translated_physmap)) {
 		/* Make pagetable pieces RO */
 		set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
Signature · 1 line
1.9.1
#13 lore raw

Re: [PATCH] Solved the Xen PV/KASLR riddle

Quoted 20 lines · Stefan Bader
Quoted 8 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

So at some point I realized that level2_fixmap_pgt seemed to contain an oddly high number of entries (given that the virtual address that failed would be mapped by entry 0). And suddenly I realized that apart from entries 506 and 507 (actual fixmap and vsyscalls) the whole list actually was the same as level2_kernel_gpt (without the first 16M cleared).

Quoted 4 lines

And then I realized that xen_setup_kernel_pagetable is wrong to a degree which makes one wonder how this ever worked. Adding PMD_SIZE as an offset (2M) isn't changing l2 at all. This just copies Xen's kernel mapping, AGAIN!.

Woo! Nice find!

-Kees

Quoted 105 lines

I guess we all just were lucky that in most cases modules would not use more than 512M (which is the correctly cleaned up remainder of kernel_level2_pgt)...

I still need to compile a kernel with the patch and the old layout but I kind of got exited by the find. At least this is tested with the 1G/~1G layout and it comes up without vmalloc errors.

-Stefan

From 4b9a9a45177284e29d345eb54c545bd1da718e1b Mon Sep 17 00:00:00 2001
From: Stefan Bader <[email protected]>
Date: Thu, 28 Aug 2014 19:17:00 +0200
Subject: [PATCH] x86/xen: Fix setup of 64bit kernel pagetables

This seemed to be one of those what-the-heck moments. When trying to figure out why changing the kernel/module split (which enabling KASLR does) causes vmalloc to run wild on boot of 64bit PV guests, after much scratching my head, found that the current Xen code copies the same L2 table not only to the level2_ident_pgt and level2_kernel_pgt, but also (due to miscalculating the offset) to level2_fixmap_pgt.

This only worked because the normal kernel image size only covers the first half of level2_kernel_pgt and module space starts after that. With the split changing, the kernel image uses the full PUD range of 1G and module space starts in the level2_fixmap_pgt. So basically:

L4[511]->level3_kernel_pgt[510]->level2_kernel_pgt

                          [511]->level2_fixmap_pgt

And now the incorrect copy of the kernel mapping in that range bites (hard).

This change might not be the fully correct approach as it basically removes the pre-set page table entry for the fixmap that is compile time set (level2_fixmap_pgt[506]->level1_fixmap_pgt). For one the level1 page table is not yet declared in C headers (that might be fixed). But also with the current bug, it was removed, too. Since the Xen mappings for level2_kernel_pgt only covered kernel + initrd and some Xen data this did never reach that far. And still, something does create entries at level2_fixmap_pgt[506..507]. So it should be ok. At least I was able to successfully boot a kernel with 1G kernel image size without any vmalloc whinings.

---
 arch/x86/xen/mmu.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
arch/x86/xen/mmu.c +17 -10
index e8a1201..803034c 100644
@@ -1902,8 +1902,22 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
                /* L3_i[0] -> level2_ident_pgt */
                convert_pfn_mfn(level3_ident_pgt);
                /* L3_k[510] -> level2_kernel_pgt
-                * L3_i[511] -> level2_fixmap_pgt */
+                * L3_k[511] -> level2_fixmap_pgt */
                convert_pfn_mfn(level3_kernel_pgt);
+
+               /* level2_fixmap_pgt contains a single entry for the
+                * fixmap area at offset 506. The correct way would
+                * be to convert level2_fixmap_pgt to mfn and set the
+                * level1_fixmap_pgt (which is completely empty) to RO,
+                * too. But currently this page table is not delcared,
+                * so it would be a bit of voodoo to get its address.
+                * And also the fixmap entry was never set anyway due
+                * to using the wrong l2 when getting Xen's tables.
+                * So let's just nuke it.
+                * This orphans level1_fixmap_pgt, but that should be
+                * as it always has been.
+                */
+               memset(level2_fixmap_pgt, 0, 512*sizeof(long));
        }
        /* We get [511][511] and have Xen's version of level2_kernel_pgt */
        l3 = m2v(pgd[pgd_index(__START_KERNEL_map)].pgd);
@@ -1913,21 +1927,15 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
        addr[1] = (unsigned long)l3;
        addr[2] = (unsigned long)l2;
        /* Graft it onto L4[272][0]. Note that we creating an aliasing problem:
-        * Both L4[272][0] and L4[511][511] have entries that point to the same
+        * Both L4[272][0] and L4[511][510] have entries that point to the same
         * L2 (PMD) tables. Meaning that if you modify it in __va space
         * it will be also modified in the __ka space! (But if you just
         * modify the PMD table to point to other PTE's or none, then you
         * are OK - which is what cleanup_highmap does) */
        copy_page(level2_ident_pgt, l2);
-       /* Graft it onto L4[511][511] */
+       /* Graft it onto L4[511][510] */
        copy_page(level2_kernel_pgt, l2);
 
-       /* Get [511][510] and graft that in level2_fixmap_pgt */
-       l3 = m2v(pgd[pgd_index(__START_KERNEL_map + PMD_SIZE)].pgd);
-       l2 = m2v(l3[pud_index(__START_KERNEL_map + PMD_SIZE)].pud);
-       copy_page(level2_fixmap_pgt, l2);
-       /* Note that we don't do anything with level1_fixmap_pgt which
-        * we don't need. */
        if (!xen_feature(XENFEAT_auto_translated_physmap)) {
                /* Make pagetable pieces RO */
                set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
--

1.9.1

Signature · 2 lines
Kees Cook
Chrome OS Security
#14 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

In reply to Stefan Bader
Quoted 10 lines · Stefan Bader
Quoted 7 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

The M2P map lives in the Xen reserved virtual address space in each PV guest, and forms part of the PV ABI. It is mapped read-only, in the native width of the guest.

32bit PV (PAE) at 0xF5800000
64bit PV at 0xFFFF800000000000

This is represented by the MACH2PHYS_VIRT_START symbol from the Xen public header files. You should be able to blindly construct a pointer to it (if you have nothing better to hand), as it will be hooked into the guests pagetables before execution starts. Therefore, "MACH2PHYS_VIRT_START[(unsigned long)pfn]" ought to do in a pinch.

~Andrew

#15 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

Quoted 16 lines · Andrew Cooper
Quoted 10 lines · Stefan Bader
Quoted 7 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

The M2P map lives in the Xen reserved virtual address space in each PV guest, and forms part of the PV ABI. It is mapped read-only, in the native width of the guest.

Quoted 8 lines

32bit PV (PAE) at 0xF5800000
64bit PV at 0xFFFF800000000000

This is represented by the MACH2PHYS_VIRT_START symbol from the Xen public header files. You should be able to blindly construct a pointer to it (if you have nothing better to hand), as it will be hooked into the guests pagetables before execution starts. Therefore, "MACH2PHYS_VIRT_START[(unsigned long)pfn]" ought to do in a pinch.

machine_to_phys_mapping is set to that address but its not mapped inside the crash dump. Somehow vtop in crash handles translations. I need to have a look at their code, I guess.

Thanks,
Stefan

Quoted 3 lines

~Andrew

#19 lore raw

Re: [PATCH] Solved the Xen PV/KASLR riddle

In reply to Stefan Bader
Quoted 25 lines · Stefan Bader
Quoted 8 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

So at some point I realized that level2_fixmap_pgt seemed to contain an oddly high number of entries (given that the virtual address that failed would be mapped by entry 0). And suddenly I realized that apart from entries 506 and 507 (actual fixmap and vsyscalls) the whole list actually was the same as level2_kernel_gpt (without the first 16M cleared).

And then I realized that xen_setup_kernel_pagetable is wrong to a degree which makes one wonder how this ever worked. Adding PMD_SIZE as an offset (2M) isn't changing l2 at all. This just copies Xen's kernel mapping, AGAIN!.

Quoted 7 lines

I guess we all just were lucky that in most cases modules would not use more than 512M (which is the correctly cleaned up remainder of kernel_level2_pgt)..

I still need to compile a kernel with the patch and the old layout but I kind of got exited by the find. At least this is tested with the 1G/~1G layout and it comes up without vmalloc errors.

Woot!

Quoted 4 lines

-Stefan

Quoted 1 line

From 4b9a9a45177284e29d345eb54c545bd1da718e1b Mon Sep 17 00:00:00 2001

Quoted 8 lines

From: Stefan Bader <[email protected]>
Date: Thu, 28 Aug 2014 19:17:00 +0200
Subject: [PATCH] x86/xen: Fix setup of 64bit kernel pagetables

This seemed to be one of those what-the-heck moments. When trying to figure out why changing the kernel/module split (which enabling KASLR does) causes vmalloc to run wild on boot of 64bit PV guests, after much scratching my head, found that the current Xen code copies the

s/current Xen/xen_setup_kernel_pagetable/

Quoted 11 lines

same L2 table not only to the level2_ident_pgt and level2_kernel_pgt, but also (due to miscalculating the offset) to level2_fixmap_pgt.

This only worked because the normal kernel image size only covers the first half of level2_kernel_pgt and module space starts after that. With the split changing, the kernel image uses the full PUD range of 1G and module space starts in the level2_fixmap_pgt. So basically:

L4[511]->level3_kernel_pgt[510]->level2_kernel_pgt

                          [511]->level2_fixmap_pgt

Perhaps you could add a similar drawing of what it looked like without the kaslr enabled? AS in the 'normal kernel image' scenario?

Quoted 2 lines

And now the incorrect copy of the kernel mapping in that range bites (hard).

Want to include the vmalloc warning you got?

Quoted 24 lines

This change might not be the fully correct approach as it basically removes the pre-set page table entry for the fixmap that is compile time set (level2_fixmap_pgt[506]->level1_fixmap_pgt). For one the level1 page table is not yet declared in C headers (that might be fixed). But also with the current bug, it was removed, too. Since the Xen mappings for level2_kernel_pgt only covered kernel + initrd and some Xen data this did never reach that far. And still, something does create entries at level2_fixmap_pgt[506..507]. So it should be ok. At least I was able to successfully boot a kernel with 1G kernel image size without any vmalloc whinings.

---
 arch/x86/xen/mmu.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
arch/x86/xen/mmu.c +0 -0
index e8a1201..803034c 100644
@@ -1902,8 +1902,22 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
 		/* L3_i[0] -> level2_ident_pgt */
 		convert_pfn_mfn(level3_ident_pgt);

arch/x86/xen/mmu.c void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn) · line 1912

Quoted 10 lines
arch/x86/xen/mmu.c +7 -1
 		/* L3_k[510] -> level2_kernel_pgt
-		 * L3_i[511] -> level2_fixmap_pgt */
+		 * L3_k[511] -> level2_fixmap_pgt */
 		convert_pfn_mfn(level3_kernel_pgt);
+
+		/* level2_fixmap_pgt contains a single entry for the
+		 * fixmap area at offset 506. The correct way would
+		 * be to convert level2_fixmap_pgt to mfn and set the
+		 * level1_fixmap_pgt (which is completely empty) to RO,
+		 * too. But currently this page table is not delcared,

declared.

arch/x86/xen/mmu.c void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn) · line 1914

Quoted 2 lines
arch/x86/xen/mmu.c +2 -0
+		 * so it would be a bit of voodoo to get its address.
+		 * And also the fixmap entry was never set anyway due

s/anyway//

arch/x86/xen/mmu.c void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn) · line 1918

Quoted 4 lines
arch/x86/xen/mmu.c +4 -0
+		 * to using the wrong l2 when getting Xen's tables.
+		 * So let's just nuke it.
+		 * This orphans level1_fixmap_pgt, but that should be
+		 * as it always has been.

'as it always has been.' ? Not sure I follow that sentence?

Quoted 16 lines
arch/x86/xen/mmu.c +3 -1
+		 */
+		memset(level2_fixmap_pgt, 0, 512*sizeof(long));
 	}
 	/* We get [511][511] and have Xen's version of level2_kernel_pgt */
 	l3 = m2v(pgd[pgd_index(__START_KERNEL_map)].pgd);
@@ -1913,21 +1927,15 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
 	addr[1] = (unsigned long)l3;
 	addr[2] = (unsigned long)l2;
 	/* Graft it onto L4[272][0]. Note that we creating an aliasing problem:
-	 * Both L4[272][0] and L4[511][511] have entries that point to the same
+	 * Both L4[272][0] and L4[511][510] have entries that point to the same
 	 * L2 (PMD) tables. Meaning that if you modify it in __va space
 	 * it will be also modified in the __ka space! (But if you just
 	 * modify the PMD table to point to other PTE's or none, then you
 	 * are OK - which is what cleanup_highmap does) */
 	copy_page(level2_ident_pgt, l2);

arch/x86/xen/mmu.c void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn) · line 1930

Quoted 10 lines
arch/x86/xen/mmu.c +1 -7
-	/* Graft it onto L4[511][511] */
+	/* Graft it onto L4[511][510] */
 	copy_page(level2_kernel_pgt, l2);
 
-	/* Get [511][510] and graft that in level2_fixmap_pgt */
-	l3 = m2v(pgd[pgd_index(__START_KERNEL_map + PMD_SIZE)].pgd);
-	l2 = m2v(l3[pud_index(__START_KERNEL_map + PMD_SIZE)].pud);
-	copy_page(level2_fixmap_pgt, l2);
-	/* Note that we don't do anything with level1_fixmap_pgt which
-	 * we don't need. */

Later during bootup we do set the fixmap with entries. I recall (vaguely) that on the SLES kernels (Classic) the fixmap was needed during early bootup. The reason was that it used the right away for bootparams (maybe?).

I think your patch is correct in ripping out level1_fixmap_pgt and level2_fixmap_pgt.

Quoted 6 lines
arch/x86/xen/mmu.c +0 -0
 	if (!xen_feature(XENFEAT_auto_translated_physmap)) {
 		/* Make pagetable pieces RO */
 		set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
Signature · 1 line
1.9.1
#16 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

In reply to Stefan Bader
Quoted 24 lines · Stefan Bader
Quoted 15 lines · Andrew Cooper
Quoted 10 lines · Stefan Bader
Quoted 7 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

The M2P map lives in the Xen reserved virtual address space in each PV guest, and forms part of the PV ABI. It is mapped read-only, in the native width of the guest.

Quoted 8 lines

32bit PV (PAE) at 0xF5800000
64bit PV at 0xFFFF800000000000

This is represented by the MACH2PHYS_VIRT_START symbol from the Xen public header files. You should be able to blindly construct a pointer to it (if you have nothing better to hand), as it will be hooked into the guests pagetables before execution starts. Therefore, "MACH2PHYS_VIRT_START[(unsigned long)pfn]" ought to do in a pinch.

Quoted 6 lines

machine_to_phys_mapping is set to that address but its not mapped inside the crash dump. Somehow vtop in crash handles translations. I need to have a look at their code, I guess.

Thanks,
Stefan

What context is the crash dump? If it is a Xen+dom0 kexec()d to native linux, then the m2p should still be accessible given dom0's cr3. If it is some state copied off-host then you will need to adjust the copy to include that virtual range.

~Andrew

#20 lore raw

Re: [PATCH] Solved the Xen PV/KASLR riddle

In reply to Konrad Rzeszutek Wilk
Quoted 30 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Stefan Bader
Quoted 8 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

So at some point I realized that level2_fixmap_pgt seemed to contain an oddly high number of entries (given that the virtual address that failed would be mapped by entry 0). And suddenly I realized that apart from entries 506 and 507 (actual fixmap and vsyscalls) the whole list actually was the same as level2_kernel_gpt (without the first 16M cleared).

And then I realized that xen_setup_kernel_pagetable is wrong to a degree which makes one wonder how this ever worked. Adding PMD_SIZE as an offset (2M) isn't changing l2 at all. This just copies Xen's kernel mapping, AGAIN!.

I guess we all just were lucky that in most cases modules would not use more than 512M (which is the correctly cleaned up remainder of kernel_level2_pgt)..

Quoted 5 lines
Quoted 3 lines

I still need to compile a kernel with the patch and the old layout but I kind of got exited by the find. At least this is tested with the 1G/~1G layout and it comes up without vmalloc errors.

Woot!

Oh yeah! :)

Quoted 8 lines
Quoted 8 lines

-Stefan

Quoted 1 line

From 4b9a9a45177284e29d345eb54c545bd1da718e1b Mon Sep 17 00:00:00 2001

From: Stefan Bader <[email protected]>
Date: Thu, 28 Aug 2014 19:17:00 +0200
Subject: [PATCH] x86/xen: Fix setup of 64bit kernel pagetables

Quoted 6 lines
Quoted 4 lines

This seemed to be one of those what-the-heck moments. When trying to figure out why changing the kernel/module split (which enabling KASLR does) causes vmalloc to run wild on boot of 64bit PV guests, after much scratching my head, found that the current Xen code copies the

s/current Xen/xen_setup_kernel_pagetable/

ok

Quoted 9 lines
Quoted 9 lines

same L2 table not only to the level2_ident_pgt and level2_kernel_pgt, but also (due to miscalculating the offset) to level2_fixmap_pgt.

This only worked because the normal kernel image size only covers the first half of level2_kernel_pgt and module space starts after that. With the split changing, the kernel image uses the full PUD range of 1G and module space starts in the level2_fixmap_pgt. So basically:

Quoted 6 lines
Quoted 3 lines

L4[511]->level3_kernel_pgt[510]->level2_kernel_pgt

                          [511]->level2_fixmap_pgt

Perhaps you could add a similar drawing of what it looked like without the kaslr enabled? AS in the 'normal kernel image' scenario?

Sure. Btw, someone also contacted me saying they have the same problem without changing the layout but having really big initrd (500M). While that feels like it should be impossible (if the kernel+initrd+xen stuff has to fix the 512M kernel image size area then). But if it can happen, then surely it does cause mappings to be where the module space starts then.

Quoted 5 lines
Quoted 3 lines

And now the incorrect copy of the kernel mapping in that range bites (hard).

Want to include the vmalloc warning you got?

Yeah, that is a good idea.

Quoted 37 lines
Quoted 25 lines

This change might not be the fully correct approach as it basically removes the pre-set page table entry for the fixmap that is compile time set (level2_fixmap_pgt[506]->level1_fixmap_pgt). For one the level1 page table is not yet declared in C headers (that might be fixed). But also with the current bug, it was removed, too. Since the Xen mappings for level2_kernel_pgt only covered kernel + initrd and some Xen data this did never reach that far. And still, something does create entries at level2_fixmap_pgt[506..507]. So it should be ok. At least I was able to successfully boot a kernel with 1G kernel image size without any vmalloc whinings.

---
 arch/x86/xen/mmu.c | 26 +++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)
arch/x86/xen/mmu.c +0 -0
index e8a1201..803034c 100644
@@ -1902,8 +1902,22 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
 		/* L3_i[0] -> level2_ident_pgt */
 		convert_pfn_mfn(level3_ident_pgt);

arch/x86/xen/mmu.c void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn) · line 1912

Quoted 10 lines
arch/x86/xen/mmu.c +7 -1
 		/* L3_k[510] -> level2_kernel_pgt
-		 * L3_i[511] -> level2_fixmap_pgt */
+		 * L3_k[511] -> level2_fixmap_pgt */
 		convert_pfn_mfn(level3_kernel_pgt);
+
+		/* level2_fixmap_pgt contains a single entry for the
+		 * fixmap area at offset 506. The correct way would
+		 * be to convert level2_fixmap_pgt to mfn and set the
+		 * level1_fixmap_pgt (which is completely empty) to RO,
+		 * too. But currently this page table is not delcared,

declared.

Quoted 4 lines

arch/x86/xen/mmu.c void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn) · line 1914

Quoted 2 lines
arch/x86/xen/mmu.c +2 -0
+		 * so it would be a bit of voodoo to get its address.
+		 * And also the fixmap entry was never set anyway due

s/anyway//

Too much slang I guess. Ok :)

Quoted 6 lines

arch/x86/xen/mmu.c void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn) · line 1918

Quoted 4 lines
arch/x86/xen/mmu.c +4 -0
+		 * to using the wrong l2 when getting Xen's tables.
+		 * So let's just nuke it.
+		 * This orphans level1_fixmap_pgt, but that should be
+		 * as it always has been.

'as it always has been.' ? Not sure I follow that sentence?

Probably need to be more verbose and say "the same basically happens with the current code, too".

Quoted 21 lines
Quoted 21 lines
arch/x86/xen/mmu.c +4 -2
+		 */
+		memset(level2_fixmap_pgt, 0, 512*sizeof(long));
 	}
 	/* We get [511][511] and have Xen's version of level2_kernel_pgt */
 	l3 = m2v(pgd[pgd_index(__START_KERNEL_map)].pgd);
@@ -1913,21 +1927,15 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
 	addr[1] = (unsigned long)l3;
 	addr[2] = (unsigned long)l2;
 	/* Graft it onto L4[272][0]. Note that we creating an aliasing problem:
-	 * Both L4[272][0] and L4[511][511] have entries that point to the same
+	 * Both L4[272][0] and L4[511][510] have entries that point to the same
 	 * L2 (PMD) tables. Meaning that if you modify it in __va space
 	 * it will be also modified in the __ka space! (But if you just
 	 * modify the PMD table to point to other PTE's or none, then you
 	 * are OK - which is what cleanup_highmap does) */
 	copy_page(level2_ident_pgt, l2);
-	/* Graft it onto L4[511][511] */
+	/* Graft it onto L4[511][510] */
 	copy_page(level2_kernel_pgt, l2);
 
Quoted 10 lines

arch/x86/xen/mmu.c void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn) · line 1930

Quoted 6 lines
arch/x86/xen/mmu.c +0 -6
-	/* Get [511][510] and graft that in level2_fixmap_pgt */
-	l3 = m2v(pgd[pgd_index(__START_KERNEL_map + PMD_SIZE)].pgd);
-	l2 = m2v(l3[pud_index(__START_KERNEL_map + PMD_SIZE)].pud);
-	copy_page(level2_fixmap_pgt, l2);
-	/* Note that we don't do anything with level1_fixmap_pgt which
-	 * we don't need. */

Later during bootup we do set the fixmap with entries. I recall (vaguely) that on the SLES kernels (Classic) the fixmap was needed during early bootup. The reason was that it used the right away for bootparams (maybe?).

Ah ok. Yeah I was confident that it got set somewhere else after xen_setup_kernel_pagetable ran. Because in the dump fixmap and vsyscall entries where in place while I was pretty sure they are not set in the l2 table that is copied in from the pagetables provided by the xen toolstack.

Quoted 3 lines

I think your patch is correct in ripping out level1_fixmap_pgt and level2_fixmap_pgt.

l1 effectively was already ripped out (not used). The only reason I could think of for preserving the compile-time l2 table would have been if fixmap would be not there. But obviously that is (and was) created along with some dynamic l1 table.

Ok, I rework the patch and re-send it (freshly, iow not part of this thread). And while I am at it, I would add the stable tag.

-Stefan

Quoted 7 lines
Quoted 7 lines
arch/x86/xen/mmu.c +0 -0
 	if (!xen_feature(XENFEAT_auto_translated_physmap)) {
 		/* Make pagetable pieces RO */
 		set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
Signature · 1 line
1.9.1
#21 lore raw

Re: [PATCH] Solved the Xen PV/KASLR riddle

Quoted 3 lines · Stefan Bader

Ok, I rework the patch and re-send it (freshly, iow not part of this thread). And while I am at it, I would add the stable tag.

Can you use a different title? Perhaps:

x86/xen: fix 64-bit PV guest kernel page tables for KASLR

David

#17 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

In reply to Andrew Cooper
Quoted 29 lines · Andrew Cooper
Quoted 28 lines · Stefan Bader
Quoted 15 lines · Andrew Cooper
Quoted 10 lines · Stefan Bader
Quoted 7 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

The M2P map lives in the Xen reserved virtual address space in each PV guest, and forms part of the PV ABI. It is mapped read-only, in the native width of the guest.

Quoted 8 lines

32bit PV (PAE) at 0xF5800000
64bit PV at 0xFFFF800000000000

This is represented by the MACH2PHYS_VIRT_START symbol from the Xen public header files. You should be able to blindly construct a pointer to it (if you have nothing better to hand), as it will be hooked into the guests pagetables before execution starts. Therefore, "MACH2PHYS_VIRT_START[(unsigned long)pfn]" ought to do in a pinch.

machine_to_phys_mapping is set to that address but its not mapped inside the crash dump. Somehow vtop in crash handles translations. I need to have a look at their code, I guess.

Quoted 7 lines
Quoted 2 lines

Thanks,
Stefan

What context is the crash dump? If it is a Xen+dom0 kexec()d to native linux, then the m2p should still be accessible given dom0's cr3. If it is some state copied off-host then you will need to adjust the copy to include that virtual range.

No its a domU dump of a PV guest taken with "xl dump-core" (or actually the result of on-crash trigger).

Quoted 3 lines

~Andrew

#22 lore raw

Re: [PATCH] Solved the Xen PV/KASLR riddle

In reply to David Vrabel
Quoted 11 lines · David Vrabel
Quoted 3 lines · Stefan Bader

Ok, I rework the patch and re-send it (freshly, iow not part of this thread). And while I am at it, I would add the stable tag.

Can you use a different title? Perhaps:

x86/xen: fix 64-bit PV guest kernel page tables for KASLR

David

I can change the title but would not want to include KASLR. Because that is just indirectly responsible. This is a fix for a problem that always was there but just escaped notice until KASLR caused the layout to change. This could have done independently before.

Or, like I mentioned in the last email, be caused by a large initrd.

-Stefan

#18 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

In reply to Stefan Bader
Quoted 30 lines · Stefan Bader
Quoted 29 lines · Andrew Cooper
Quoted 28 lines · Stefan Bader
Quoted 15 lines · Andrew Cooper
Quoted 10 lines · Stefan Bader
Quoted 7 lines
Quoted 4 lines

So not much further... but then I think I know what I do next. Probably should have done before. I'll replace the WARN_ON in vmalloc that triggers by a panic and at least get a crash dump of that situation when it occurs. Then I can dig in there with crash (really should have thought of that before)...

<nods> I dug a bit in the code (arch/x86/xen/mmu.c) but there is nothing there that screams at me, so I fear I will have to wait until you get the crash and get some clues from that.

Ok, what a journey. So after long hours of painful staring at the code... (and btw, if someone could tell me how the heck one can do a mfn_to_pfn in crash, I really would appreaciate :-P)

The M2P map lives in the Xen reserved virtual address space in each PV guest, and forms part of the PV ABI. It is mapped read-only, in the native width of the guest.

Quoted 8 lines

32bit PV (PAE) at 0xF5800000
64bit PV at 0xFFFF800000000000

This is represented by the MACH2PHYS_VIRT_START symbol from the Xen public header files. You should be able to blindly construct a pointer to it (if you have nothing better to hand), as it will be hooked into the guests pagetables before execution starts. Therefore, "MACH2PHYS_VIRT_START[(unsigned long)pfn]" ought to do in a pinch.

machine_to_phys_mapping is set to that address but its not mapped inside the crash dump. Somehow vtop in crash handles translations. I need to have a look at their code, I guess.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Thanks,
Stefan

What context is the crash dump? If it is a Xen+dom0 kexec()d to native linux, then the m2p should still be accessible given dom0's cr3. If it is some state copied off-host then you will need to adjust the copy to include that virtual range.

No its a domU dump of a PV guest taken with "xl dump-core" (or actually the result of on-crash trigger).

Ah - I believe the m2p lives in one of the Xen elf notes for a domain coredump. See what readelf -n shows.

~Andrew

#23 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

In reply to Stefan Bader
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 29.08.14 at 16:27, <[email protected]> wrote:

Sure. Btw, someone also contacted me saying they have the same problem without
changing the layout but having really big initrd (500M). While that feels like
it should be impossible (if the kernel+initrd+xen stuff has to fix the 512M kernel image size area then). But if it can happen, then surely it does cause
mappings to be where the module space starts then.

Since the initrd doesn't really need to be mapped into the (limited) virtual address space a pv guest starts with, we specifically got

/*
* Whether or not the guest can deal with being passed an initrd not
* mapped through its initial page tables.
*/
#define XEN_ELFNOTE_MOD_START_PFN 16

to deal with that situation. The hypervisor side for Dom0 is in place, and the kernel side works in our (classic) kernels. Whether it got implemented for DomU meanwhile I don't know; I'm pretty certain pv-ops kernels don't support it so far.

Jan

#24 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

Quoted 19 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 29.08.14 at 16:27, <[email protected]> wrote:

Sure. Btw, someone also contacted me saying they have the same problem without
changing the layout but having really big initrd (500M). While that feels like
it should be impossible (if the kernel+initrd+xen stuff has to fix the 512M kernel image size area then). But if it can happen, then surely it does cause
mappings to be where the module space starts then.

Since the initrd doesn't really need to be mapped into the (limited) virtual address space a pv guest starts with, we specifically got

/*
* Whether or not the guest can deal with being passed an initrd not
* mapped through its initial page tables.
*/
#define XEN_ELFNOTE_MOD_START_PFN 16

Quoted 4 lines

to deal with that situation. The hypervisor side for Dom0 is in place, and the kernel side works in our (classic) kernels. Whether it got implemented for DomU meanwhile I don't know; I'm pretty certain pv-ops kernels don't support it so far.

Correct - Not implemented. Here is what I had mentioned in the past:
(see http://lists.xen.org/archives/html/xen-devel/2014-03/msg00580.html)

XEN_ELFNOTE_INIT_P2M, XEN_ELFNOTE_MOD_START_PFN - I had been looking
at that but I can't figure out a nice way of implementing this without the usage of SPARSEMAP_VMAP virtual addresses - which is how the classic Xen does it. But then - I don't know who is using huge PV guests - as the PVHVM does a fine job? But then with PVH, now you can boot with large amount of memory (1TB?) - so some of these issues would go away? Except the 'large ramdisk' as that would eat in the MODULES_VADDR I think? Needs more thinking.

.. and then I left it and to my suprise saw on Luis's slides that Jurgen is going to take a look at that (500GB support).

Quoted 3 lines

Jan

maintainer
#25 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

Quoted 38 lines · Konrad Rzeszutek Wilk
Quoted 19 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 29.08.14 at 16:27, <[email protected]> wrote:

Sure. Btw, someone also contacted me saying they have the same problem without
changing the layout but having really big initrd (500M). While that feels like
it should be impossible (if the kernel+initrd+xen stuff has to fix the 512M kernel image size area then). But if it can happen, then surely it does cause
mappings to be where the module space starts then.

Since the initrd doesn't really need to be mapped into the (limited) virtual address space a pv guest starts with, we specifically got

/*
* Whether or not the guest can deal with being passed an initrd not
* mapped through its initial page tables.
*/
#define XEN_ELFNOTE_MOD_START_PFN 16

Quoted 4 lines

to deal with that situation. The hypervisor side for Dom0 is in place, and the kernel side works in our (classic) kernels. Whether it got implemented for DomU meanwhile I don't know; I'm pretty certain pv-ops kernels don't support it so far.

Correct - Not implemented. Here is what I had mentioned in the past:
(see http://lists.xen.org/archives/html/xen-devel/2014-03/msg00580.html)

XEN_ELFNOTE_INIT_P2M, XEN_ELFNOTE_MOD_START_PFN - I had been looking
at that but I can't figure out a nice way of implementing this without the usage of SPARSEMAP_VMAP virtual addresses - which is how the classic Xen does it. But then - I don't know who is using huge PV guests - as the PVHVM does a fine job? But then with PVH, now you can boot with large amount of memory (1TB?) - so some of these issues would go away? Except the 'large ramdisk' as that would eat in the MODULES_VADDR I think? Needs more thinking.

Quoted 2 lines

.. and then I left it and to my suprise saw on Luis's slides that Jurgen is going to take a look at that (500GB support).

I have a patch which should do the job. It is based on the classic kernel patch Jan mentioned above. The system is coming up with it, I haven't tested it with a huge initrd up to now. My plan was to post the patch together with the rest of the >500GB support, but I can send it on it's own if required.

Juergen

#26 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

Quoted 39 lines · Juergen Gross
Quoted 38 lines · Konrad Rzeszutek Wilk
Quoted 19 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 29.08.14 at 16:27, <[email protected]> wrote:

Sure. Btw, someone also contacted me saying they have the same problem without
changing the layout but having really big initrd (500M). While that feels like
it should be impossible (if the kernel+initrd+xen stuff has to fix the 512M kernel image size area then). But if it can happen, then surely it does cause
mappings to be where the module space starts then.

Since the initrd doesn't really need to be mapped into the (limited) virtual address space a pv guest starts with, we specifically got

/*
* Whether or not the guest can deal with being passed an initrd not
* mapped through its initial page tables.
*/
#define XEN_ELFNOTE_MOD_START_PFN 16

Quoted 4 lines

to deal with that situation. The hypervisor side for Dom0 is in place, and the kernel side works in our (classic) kernels. Whether it got implemented for DomU meanwhile I don't know; I'm pretty certain pv-ops kernels don't support it so far.

Correct - Not implemented. Here is what I had mentioned in the past:
(see http://lists.xen.org/archives/html/xen-devel/2014-03/msg00580.html)

XEN_ELFNOTE_INIT_P2M, XEN_ELFNOTE_MOD_START_PFN - I had been looking
at that but I can't figure out a nice way of implementing this without the usage of SPARSEMAP_VMAP virtual addresses - which is how the classic Xen does it. But then - I don't know who is using huge PV guests - as the PVHVM does a fine job? But then with PVH, now you can boot with large amount of memory (1TB?) - so some of these issues would go away? Except the 'large ramdisk' as that would eat in the MODULES_VADDR I think? Needs more thinking.

Quoted 8 lines
Quoted 2 lines

.. and then I left it and to my suprise saw on Luis's slides that Jurgen is going to take a look at that (500GB support).

I have a patch which should do the job. It is based on the classic kernel patch Jan mentioned above. The system is coming up with it, I haven't tested it with a huge initrd up to now. My plan was to post the patch together with the rest of the >500GB support, but I can send it on it's own if required.

Oooh goodies! I think it makes sense to post it whenever you think it is in the right state to be posted.

Now that your pvSCSI drivers are in, you have tons of free time I suspect :-)

Quoted 3 lines

Juergen

maintainer
#27 lore raw

Re: [Xen-devel] [PATCH] Solved the Xen PV/KASLR riddle

Quoted 49 lines · Konrad Rzeszutek Wilk
Quoted 39 lines · Juergen Gross
Quoted 38 lines · Konrad Rzeszutek Wilk
Quoted 19 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 29.08.14 at 16:27, <[email protected]> wrote:

Sure. Btw, someone also contacted me saying they have the same problem without
changing the layout but having really big initrd (500M). While that feels like
it should be impossible (if the kernel+initrd+xen stuff has to fix the 512M kernel image size area then). But if it can happen, then surely it does cause
mappings to be where the module space starts then.

Since the initrd doesn't really need to be mapped into the (limited) virtual address space a pv guest starts with, we specifically got

/*
* Whether or not the guest can deal with being passed an initrd not
* mapped through its initial page tables.
*/
#define XEN_ELFNOTE_MOD_START_PFN 16

Quoted 4 lines

to deal with that situation. The hypervisor side for Dom0 is in place, and the kernel side works in our (classic) kernels. Whether it got implemented for DomU meanwhile I don't know; I'm pretty certain pv-ops kernels don't support it so far.

Correct - Not implemented. Here is what I had mentioned in the past:
(see http://lists.xen.org/archives/html/xen-devel/2014-03/msg00580.html)

XEN_ELFNOTE_INIT_P2M, XEN_ELFNOTE_MOD_START_PFN - I had been looking
at that but I can't figure out a nice way of implementing this without the usage of SPARSEMAP_VMAP virtual addresses - which is how the classic Xen does it. But then - I don't know who is using huge PV guests - as the PVHVM does a fine job? But then with PVH, now you can boot with large amount of memory (1TB?) - so some of these issues would go away? Except the 'large ramdisk' as that would eat in the MODULES_VADDR I think? Needs more thinking.

Quoted 8 lines
Quoted 2 lines

.. and then I left it and to my suprise saw on Luis's slides that Jurgen is going to take a look at that (500GB support).

I have a patch which should do the job. It is based on the classic kernel patch Jan mentioned above. The system is coming up with it, I haven't tested it with a huge initrd up to now. My plan was to post the patch together with the rest of the >500GB support, but I can send it on it's own if required.

Quoted 5 lines

Oooh goodies! I think it makes sense to post it whenever you think it is in the right state to be posted.

Now that your pvSCSI drivers are in, you have tons of free time I suspect :-)

Oh yeah. Only one or two lines missing in xl to support it. :-)

I hope to have the >500GB patch ready for testing soon. I'd prefer to combine this and the large initrd patch in one series, as both need the same headers to be synced with Xen. In case I'm meeting some serious issues I'll post the large initrd patch on Friday.

Juergen