NUMA_BALANCING and Xen PV guest regression in 3.20-rc0
Mel,
The NUMA_BALANCING series beginning with 5d833062139d (mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically 8a0516ed8b90 (mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets _PAGE_GLOBAL (== _PAGE_PROTNONE) on all present userspace page table entries.
Note that the comment in asm/pgtable_types.h that says that _PAGE_BIT_PROTNONE is only valid on non-present entries.
/* If _PAGE_BIT_PRESENT is clear, we use these: */
/* - if the user mapped it with PROT_NONE; pte_present gives true */
#define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL
Adjusting pte_protnone() and pmd_protnone() to check for the absence of _PAGE_PRESENT allows 64-bit Xen PV guests to work correctly again (see following patch), but I'm not sure if NUMA_BALANCING would correctly work with this change.
David
8<---------------------------
x86: pte_protnone() and pmd_protnone() must check entry is
not present
Since _PAGE_PROTNONE aliases _PAGE_GLOBAL it is only valid if _PAGE_PRESENT is clear. Make pte_protnone() and pmd_protnone() check for this.
This fixes a 64-bit Xen PV guest regression introduced by 8a0516ed8b90c95ffa1363b420caa37418149f21 (mm: convert p[te|md]_numa users to p[te|md]_protnone_numa). Any userspace process would endlessly fault.
In a 64-bit PV guest, userspace page table entries have _PAGE_GLOBAL set by the hypervisor. This meant that any fault on a present userspace entry (e.g., a write to a read-only mapping) would be misinterpreted as a NUMA hinting fault and the fault would not be correctly handled, resulting in the access endlessly faulting.
--- arch/x86/include/asm/pgtable.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/include/asm/pgtable.h +4 -2
@@ -476,12 +476,14 @@ static inline int pmd_present(pmd_t pmd)*/ static inline int pte_protnone (pte_t pte) { -return pte_flags (pte) & _PAGE_PROTNONE; +return (pte_flags (pte) & (_PAGE_PROTNONE | _PAGE_PRESENT)) + == _PAGE_PROTNONE; }static inline int pmd_protnone (pmd_t pmd) { -return pmd_flags (pmd) & _PAGE_PROTNONE; +return (pmd_flags (pmd) & (_PAGE_PROTNONE | _PAGE_PRESENT)) + == _PAGE_PROTNONE; }#endif /* CONFIG_NUMA_BALANCING */
Signature · 1 line
1.7.10.4
Mel Gorman
Acked-byQuoted 7 lines · David Vrabel
Mel,
The
NUMA_BALANCINGseries beginning with5d833062139d(mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically8a0516ed8b90(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Quoted 7 lines
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets
_PAGE_GLOBAL(==_PAGE_PROTNONE) on all present userspace page table entries.
I see, this is a variation of the problem where the NUMA hinted PTE was treated as special due to the paravirt interfaces not being used.
Quoted 12 lines
Note that the comment in
asm/pgtable_types.hthat says that_PAGE_BIT_PROTNONEis only valid on non-present entries./* If
_PAGE_BIT_PRESENTis clear, we use these: */
/* - if the user mapped it withPROT_NONE;pte_presentgives true */
#define_PAGE_BIT_PROTNONE_PAGE_BIT_GLOBALAdjusting
pte_protnone()andpmd_protnone()to check for the absence of_PAGE_PRESENTallows 64-bit Xen PV guests to work correctly again (see following patch), but I'm not sure ifNUMA_BALANCINGwould correctly work with this change.
Thanks for the analysis and the reminder of some of the details from the previous discussion.
Quoted 14 lines
8<---------------------------
x86:pte_protnone()andpmd_protnone()must check entry is
not presentSince
_PAGE_PROTNONEaliases_PAGE_GLOBALit is only valid if_PAGE_PRESENTis clear. Makepte_protnone()andpmd_protnone()check for this.This fixes a 64-bit Xen PV guest regression introduced by
8a0516ed8b90c95ffa1363b420caa37418149f21(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa). Any userspace process would endlessly fault.
Quoted 8 lines
In a 64-bit PV guest, userspace page table entries have
_PAGE_GLOBALset by the hypervisor. This meant that any fault on a present userspace entry (e.g., a write to a read-only mapping) would be misinterpreted as a NUMA hinting fault and the fault would not be correctly handled, resulting in the access endlessly faulting.Signed-off-by David Vrabel <[email protected]>Cc Mel Gorman <[email protected]>
I cannot think of a reason why this would fail for NUMA balancing on bare metal. The PAGE_NONE protection clears the present bit on p[te|md]_modify so the expectations are matched before or after the patch is applied. So, for bare metal at least
I *think* this will work ok with Xen but I cannot 100% convince myself. I'm adding Wei Liu to the cc who may have a Xen PV setup handy that supports NUMA and may be able to test the patch to confirm.
Signature · 2 lines
Mel Gorman SUSE Labs
Mel Gorman
Acked-byQuoted 7 lines · David Vrabel
Mel,
The
NUMA_BALANCINGseries beginning with5d833062139d(mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically8a0516ed8b90(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Quoted 7 lines
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets
_PAGE_GLOBAL(==_PAGE_PROTNONE) on all present userspace page table entries.
I see, this is a variation of the problem where the NUMA hinted PTE was treated as special due to the paravirt interfaces not being used.
Quoted 12 lines
Note that the comment in
asm/pgtable_types.hthat says that_PAGE_BIT_PROTNONEis only valid on non-present entries./* If
_PAGE_BIT_PRESENTis clear, we use these: */
/* - if the user mapped it withPROT_NONE;pte_presentgives true */
#define_PAGE_BIT_PROTNONE_PAGE_BIT_GLOBALAdjusting
pte_protnone()andpmd_protnone()to check for the absence of_PAGE_PRESENTallows 64-bit Xen PV guests to work correctly again (see following patch), but I'm not sure ifNUMA_BALANCINGwould correctly work with this change.
Thanks for the analysis and the reminder of some of the details from the previous discussion.
Quoted 14 lines
8<---------------------------
x86:pte_protnone()andpmd_protnone()must check entry is
not presentSince
_PAGE_PROTNONEaliases_PAGE_GLOBALit is only valid if_PAGE_PRESENTis clear. Makepte_protnone()andpmd_protnone()check for this.This fixes a 64-bit Xen PV guest regression introduced by
8a0516ed8b90c95ffa1363b420caa37418149f21(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa). Any userspace process would endlessly fault.
Quoted 8 lines
In a 64-bit PV guest, userspace page table entries have
_PAGE_GLOBALset by the hypervisor. This meant that any fault on a present userspace entry (e.g., a write to a read-only mapping) would be misinterpreted as a NUMA hinting fault and the fault would not be correctly handled, resulting in the access endlessly faulting.Signed-off-by David Vrabel <[email protected]>Cc Mel Gorman <[email protected]>
I cannot think of a reason why this would fail for NUMA balancing on bare metal. The PAGE_NONE protection clears the present bit on p[te|md]_modify so the expectations are matched before or after the patch is applied. So, for bare metal at least
I *think* this will work ok with Xen but I cannot 100% convince myself. I'm adding Wei Liu to the cc who may have a Xen PV setup handy that supports NUMA and may be able to test the patch to confirm.
Signature · 2 lines
Mel Gorman SUSE Labs
Hi everyone,
Quoted 7 lines · Mel Gorman
On Thu, Feb 19, 2015 at 01:06:53PM +0000, David Vrabel wrote:
I cannot think of a reason why this would fail for NUMA balancing on bare metal. The
PAGE_NONEprotection clears the present bit on p[te|md]_modify so the expectations are matched before or after the patch is applied. So, for bare metal at least
Quoted 6 lines
Acked-by Mel Gorman <[email protected]>I *think* this will work ok with Xen but I cannot 100% convince myself. I'm adding Wei Liu to the cc who may have a Xen PV setup handy that supports NUMA and may be able to test the patch to confirm.
I'm not Wei, but I've been able to test a kernel with David's patch in the following conditions:
1. as Dom0 kernel, when Xen does not have any virtual NUMA support
2. as DomU PV kernel, when Xen does not have any virtual NUMA support
3. as DomU PV kernel, when Xen _does_ _have_ virtual NUMA support
(i.e., Wei's code)
Cases 1. and 2. have been, I believe, tested by David already, but anyways... :-)
Case 3. worked well for me, as the following commands show. In fact, with this in guest config file:
vnuma = [ [ "pnode=0","size=1000","vcpus=0-3","vdistances=10,20" ],
[ "pnode=1","size=1000","vcpus=4-7","vdistances=20,10" ],
]This is what I get from inside the guest:
root@test-pv:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3
node 0 size: 951 MB
node 0 free: 868 MB
node 1 cpus: 4 5 6 7
node 1 size: 968 MB
node 1 free: 924 MB
node distances:
node 0 1
0: 10 20
1: 20 10And this is it from the host:
root@Zhaman:~# xl debug-keys u ; xl dmesg |tail -12
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 1047417):
(XEN) Node 0: 1031009
(XEN) Node 1: 16408
(XEN) Domain 1 (total: 512000):
(XEN) Node 0: 256000
(XEN) Node 1: 256000
(XEN) 2 vnodes, 8 vcpus, guest physical layout:
(XEN) 0: pnode 0, vcpus 0-3
(XEN) 0000000000000000 - 000000003e800000
(XEN) 1: pnode 1, vcpus 4-7
(XEN) 000000003e800000 - 000000007d000000Still inside the guest, I see this:
root@test-pv:~# cat /proc/sys/kernel/numa_balancing
1And this:
root@test-pv:~# grep numa /proc/vmstat
numa_hit 65987
numa_miss 0
numa_foreign 0
numa_interleave 14473
numa_local 58642
numa_other 7345
numa_pte_updates 596
numa_huge_pte_updates 0
numa_hint_faults 479
numa_hint_faults_local 420
numa_pages_migrated 51So, yes, I would say this wok with Xen, is that correct, Mel?
I'll give it a try at running more complex stuff like 'perf bench numa' inside the guest and see what happens...
Regards,
Dario
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
Quoted 10 lines · Dario Faggioli
Hi everyone,
Quoted 7 lines · Mel Gorman
On Thu, Feb 19, 2015 at 01:06:53PM +0000, David Vrabel wrote:
I cannot think of a reason why this would fail for NUMA balancing on bare metal. The
PAGE_NONEprotection clears the present bit on p[te|md]_modify so the expectations are matched before or after the patch is applied. So, for bare metal at least
Quoted 9 lines
Quoted 6 lines
Acked-by Mel Gorman <[email protected]>I *think* this will work ok with Xen but I cannot 100% convince myself. I'm adding Wei Liu to the cc who may have a Xen PV setup handy that supports NUMA and may be able to test the patch to confirm.
I'm not Wei, but I've been able to test a kernel with David's patch in the following conditions:
Thanks very much, it's appreciated.
Quoted 28 lines
1. as Dom0 kernel, when Xen does not have any virtual NUMA support
2. as DomU PV kernel, when Xen does not have any virtual NUMA support
3. as DomU PV kernel, when Xen _does_ _have_ virtual NUMA support(i.e., Wei's code)Cases 1. and 2. have been, I believe, tested by David already, but anyways... :-)
Case 3. worked well for me, as the following commands show. In fact, with this in guest config file:
<SNIP>
And this:
root@test-pv:~# grep numa /proc/vmstat numa_hit 65987 numa_miss 0 numa_foreign 0 numa_interleave 14473 numa_local 58642 numa_other 7345 numa_pte_updates 596 numa_huge_pte_updates 0 numa_hint_faults 479 numa_hint_faults_local 420 numa_pages_migrated 51
Quoted 2 lines
So, yes, I would say this wok with Xen, is that correct, Mel?
Yes, these stats indicate that NUMA balancing is active. There is no much activity but it's there.
Quoted 3 lines
I'll give it a try at running more complex stuff like 'perf bench numa' inside the guest and see what happens...
Thanks.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 10 lines · Dario Faggioli
Hi everyone,
Quoted 7 lines · Mel Gorman
On Thu, Feb 19, 2015 at 01:06:53PM +0000, David Vrabel wrote:
I cannot think of a reason why this would fail for NUMA balancing on bare metal. The
PAGE_NONEprotection clears the present bit on p[te|md]_modify so the expectations are matched before or after the patch is applied. So, for bare metal at least
Quoted 9 lines
Quoted 6 lines
Acked-by Mel Gorman <[email protected]>I *think* this will work ok with Xen but I cannot 100% convince myself. I'm adding Wei Liu to the cc who may have a Xen PV setup handy that supports NUMA and may be able to test the patch to confirm.
I'm not Wei, but I've been able to test a kernel with David's patch in the following conditions:
Thanks very much, it's appreciated.
Quoted 28 lines
1. as Dom0 kernel, when Xen does not have any virtual NUMA support
2. as DomU PV kernel, when Xen does not have any virtual NUMA support
3. as DomU PV kernel, when Xen _does_ _have_ virtual NUMA support(i.e., Wei's code)Cases 1. and 2. have been, I believe, tested by David already, but anyways... :-)
Case 3. worked well for me, as the following commands show. In fact, with this in guest config file:
<SNIP>
And this:
root@test-pv:~# grep numa /proc/vmstat numa_hit 65987 numa_miss 0 numa_foreign 0 numa_interleave 14473 numa_local 58642 numa_other 7345 numa_pte_updates 596 numa_huge_pte_updates 0 numa_hint_faults 479 numa_hint_faults_local 420 numa_pages_migrated 51
Quoted 2 lines
So, yes, I would say this wok with Xen, is that correct, Mel?
Yes, these stats indicate that NUMA balancing is active. There is no much activity but it's there.
Quoted 3 lines
I'll give it a try at running more complex stuff like 'perf bench numa' inside the guest and see what happens...
Thanks.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 6 lines · David Vrabel
The
NUMA_BALANCINGseries beginning with5d833062139d(mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically8a0516ed8b90(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Quoted 6 lines
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets
_PAGE_GLOBAL(==_PAGE_PROTNONE) on all present userspace page table entries.
That's some crazy stuff, but whatever. The patch is clearly good. Applied,
Linus
Quoted 6 lines · David Vrabel
The
NUMA_BALANCINGseries beginning with5d833062139d(mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically8a0516ed8b90(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Quoted 6 lines
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets
_PAGE_GLOBAL(==_PAGE_PROTNONE) on all present userspace page table entries.
That's some crazy stuff, but whatever. The patch is clearly good. Applied,
Linus
Quoted 7 lines · Linus Torvalds
Quoted 6 lines · David Vrabel
The
NUMA_BALANCINGseries beginning with5d833062139d(mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically8a0516ed8b90(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Quoted 8 lines
Quoted 6 lines
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets
_PAGE_GLOBAL(==_PAGE_PROTNONE) on all present userspace page table entries.That's some crazy stuff, but whatever. The patch is clearly good. Applied,
Thanks.
Xen PV guests do not use any hardware virtualization features. In particular they do not use nested paging.
A 64-bit PV guest runs in user mode for both kernel and userspace. On kernel to user mode transitions, the hypervisor flips between two sets of page tables (the user mode tables do not contain any kernel mappings, but the kernel mode tables contain both). By setting _PAGE_GLOBAL on the userspace entries, a kernel to user transition can avoid flushing the userspace mappings from the TLB.
David
Quoted 7 lines · Linus Torvalds
Quoted 6 lines · David Vrabel
The
NUMA_BALANCINGseries beginning with5d833062139d(mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically8a0516ed8b90(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Quoted 8 lines
Quoted 6 lines
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets
_PAGE_GLOBAL(==_PAGE_PROTNONE) on all present userspace page table entries.That's some crazy stuff, but whatever. The patch is clearly good. Applied,
Thanks.
Xen PV guests do not use any hardware virtualization features. In particular they do not use nested paging.
A 64-bit PV guest runs in user mode for both kernel and userspace. On kernel to user mode transitions, the hypervisor flips between two sets of page tables (the user mode tables do not contain any kernel mappings, but the kernel mode tables contain both). By setting _PAGE_GLOBAL on the userspace entries, a kernel to user transition can avoid flushing the userspace mappings from the TLB.
David
Quoted 7 lines · David Vrabel
Mel,
The
NUMA_BALANCINGseries beginning with5d833062139d(mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically8a0516ed8b90(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Quoted 6 lines
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets
_PAGE_GLOBAL(==_PAGE_PROTNONE) on all present userspace page table entries.
I'm feeling I miss very basic background on how Xen works, but why does it set _PAGE_GLOBAL on userspace entries? It sounds strange to me.
Quoted 12 lines
Note that the comment in
asm/pgtable_types.hthat says that_PAGE_BIT_PROTNONEis only valid on non-present entries./* If
_PAGE_BIT_PRESENTis clear, we use these: */
/* - if the user mapped it withPROT_NONE;pte_presentgives true */
#define_PAGE_BIT_PROTNONE_PAGE_BIT_GLOBALAdjusting
pte_protnone()andpmd_protnone()to check for the absence of_PAGE_PRESENTallows 64-bit Xen PV guests to work correctly again (see following patch), but I'm not sure ifNUMA_BALANCINGwould correctly work with this change.
Signature · 1 line
Kirill A. Shutemov
Quoted 7 lines · David Vrabel
Mel,
The
NUMA_BALANCINGseries beginning with5d833062139d(mm: numa: do not dereference pmd outside of the lock during NUMA hinting fault) and specifically8a0516ed8b90(mm: convert p[te|md]_numa users to p[te|md]_protnone_numa) breaks Xen 64-bit PV guests.
Quoted 6 lines
Any fault on a present userspace mapping (e.g., a write to a read-only mapping) is being misinterpreted as a NUMA hinting fault and not handled correctly. All userspace programs end up continuously faulting.
This is because the hypervisor sets
_PAGE_GLOBAL(==_PAGE_PROTNONE) on all present userspace page table entries.
I'm feeling I miss very basic background on how Xen works, but why does it set _PAGE_GLOBAL on userspace entries? It sounds strange to me.
Quoted 12 lines
Note that the comment in
asm/pgtable_types.hthat says that_PAGE_BIT_PROTNONEis only valid on non-present entries./* If
_PAGE_BIT_PRESENTis clear, we use these: */
/* - if the user mapped it withPROT_NONE;pte_presentgives true */
#define_PAGE_BIT_PROTNONE_PAGE_BIT_GLOBALAdjusting
pte_protnone()andpmd_protnone()to check for the absence of_PAGE_PRESENTallows 64-bit Xen PV guests to work correctly again (see following patch), but I'm not sure ifNUMA_BALANCINGwould correctly work with this change.
Signature · 1 line
Kirill A. Shutemov
Quoted 3 lines · Kirill A. Shutemov
I'm feeling I miss very basic background on how Xen works, but why does it set
_PAGE_GLOBALon userspace entries? It sounds strange to me.
It is definitely strange. I'm guessing that it's some ancient Xen hack for the early Intel virtualization that used to have absolutely horrendous vmenter/exit costs, including very much the TLB overhead. \
These days, Intel has address space identifiers, and doesn't flush the whole TLB on VM entry/exit, so it's probably pointless to play games with the global bit.
I get the feeling that a lot of Xen stuff is that kind of "legacy hacks" that should just be cleaned up, but nobody has the energy or the interest. There was the whole odd crazy SHARED_KERNEL_PMD hackery too.
Linus
Quoted 3 lines · Kirill A. Shutemov
I'm feeling I miss very basic background on how Xen works, but why does it set
_PAGE_GLOBALon userspace entries? It sounds strange to me.
It is definitely strange. I'm guessing that it's some ancient Xen hack for the early Intel virtualization that used to have absolutely horrendous vmenter/exit costs, including very much the TLB overhead. \
These days, Intel has address space identifiers, and doesn't flush the whole TLB on VM entry/exit, so it's probably pointless to play games with the global bit.
I get the feeling that a lot of Xen stuff is that kind of "legacy hacks" that should just be cleaned up, but nobody has the energy or the interest. There was the whole odd crazy SHARED_KERNEL_PMD hackery too.
Linus
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Kirill A. Shutemov
I'm feeling I miss very basic background on how Xen works, but why does it set
_PAGE_GLOBALon userspace entries? It sounds strange to me.It is definitely strange. I'm guessing that it's some ancient Xen hack for the early Intel virtualization that used to have absolutely horrendous vmenter/exit costs, including very much the TLB overhead. \
These days, Intel has address space identifiers, and doesn't flush the whole TLB on VM entry/exit, so it's probably pointless to play games with the global bit.
It was introduced in 2006, but has nothing to do with VT-x
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=6f562e72cdc4b7e1519e23be75f812aebbf41db3
As long mode drops segment limit checking, the only way to protect a 64bit PV kernel from its userspace (both of which run in ring3 on user pages) is to maintain two sets of pagetables and switch between them on guest kernel/user context switches. The user set lack kernel mappings.
I can't comment about the performance impact of the patch (way before my time), but the justification was to try and reduce the overhead of guest context switches.
Quoted 4 lines
I get the feeling that a lot of Xen stuff is that kind of "legacy hacks" that should just be cleaned up, but nobody has the energy or the interest.
Time, mainly.
There certainly are areas which should be up for re-evaluation, given 9 years of change in hardware.
Quoted 1 line
There was the whole odd crazy
SHARED_KERNEL_PMDhackery too.
SHARED_KERNEL_PMD is an artefact of Xen living in the same virtual address space as a PV guest, and needing to maintain linear mappings.
~Andrew
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Kirill A. Shutemov
I'm feeling I miss very basic background on how Xen works, but why does it set
_PAGE_GLOBALon userspace entries? It sounds strange to me.It is definitely strange. I'm guessing that it's some ancient Xen hack for the early Intel virtualization that used to have absolutely horrendous vmenter/exit costs, including very much the TLB overhead. \
These days, Intel has address space identifiers, and doesn't flush the whole TLB on VM entry/exit, so it's probably pointless to play games with the global bit.
It was introduced in 2006, but has nothing to do with VT-x
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=6f562e72cdc4b7e1519e23be75f812aebbf41db3
As long mode drops segment limit checking, the only way to protect a 64bit PV kernel from its userspace (both of which run in ring3 on user pages) is to maintain two sets of pagetables and switch between them on guest kernel/user context switches. The user set lack kernel mappings.
I can't comment about the performance impact of the patch (way before my time), but the justification was to try and reduce the overhead of guest context switches.
Quoted 4 lines
I get the feeling that a lot of Xen stuff is that kind of "legacy hacks" that should just be cleaned up, but nobody has the energy or the interest.
Time, mainly.
There certainly are areas which should be up for re-evaluation, given 9 years of change in hardware.
Quoted 1 line
There was the whole odd crazy
SHARED_KERNEL_PMDhackery too.
SHARED_KERNEL_PMD is an artefact of Xen living in the same virtual address space as a PV guest, and needing to maintain linear mappings.
~Andrew
Quoted 17 lines · Andrew Cooper
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Kirill A. Shutemov
I'm feeling I miss very basic background on how Xen works, but why does it set
_PAGE_GLOBALon userspace entries? It sounds strange to me.It is definitely strange. I'm guessing that it's some ancient Xen hack for the early Intel virtualization that used to have absolutely horrendous vmenter/exit costs, including very much the TLB overhead. \
These days, Intel has address space identifiers, and doesn't flush the whole TLB on VM entry/exit, so it's probably pointless to play games with the global bit.
It was introduced in 2006, but has nothing to do with VT-x
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=6f562e72cdc4b7e1519e23be75f812aebbf41db3
Quoted 8 lines
As long mode drops segment limit checking, the only way to protect a 64bit PV kernel from its userspace (both of which run in ring3 on user pages) is to maintain two sets of pagetables and switch between them on guest kernel/user context switches. The user set lack kernel mappings.
I can't comment about the performance impact of the patch (way before my time), but the justification was to try and reduce the overhead of guest context switches.
IIUC, it tries to reduce userspace->kernel switch in guest. It's still
hopeless: kernel mappings are always TLB-cold, right?
Quoted 8 lines
Quoted 3 lines
I get the feeling that a lot of Xen stuff is that kind of "legacy hacks" that should just be cleaned up, but nobody has the energy or the interest.
Time, mainly.
There certainly are areas which should be up for re-evaluation, given 9 years of change in hardware.
Is Xen PV still widely used? I'm surprised that users can tolerate this kind of overhead.
Signature · 1 line
Kirill A. Shutemov
Quoted 17 lines · Andrew Cooper
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Kirill A. Shutemov
I'm feeling I miss very basic background on how Xen works, but why does it set
_PAGE_GLOBALon userspace entries? It sounds strange to me.It is definitely strange. I'm guessing that it's some ancient Xen hack for the early Intel virtualization that used to have absolutely horrendous vmenter/exit costs, including very much the TLB overhead. \
These days, Intel has address space identifiers, and doesn't flush the whole TLB on VM entry/exit, so it's probably pointless to play games with the global bit.
It was introduced in 2006, but has nothing to do with VT-x
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=6f562e72cdc4b7e1519e23be75f812aebbf41db3
Quoted 8 lines
As long mode drops segment limit checking, the only way to protect a 64bit PV kernel from its userspace (both of which run in ring3 on user pages) is to maintain two sets of pagetables and switch between them on guest kernel/user context switches. The user set lack kernel mappings.
I can't comment about the performance impact of the patch (way before my time), but the justification was to try and reduce the overhead of guest context switches.
IIUC, it tries to reduce userspace->kernel switch in guest. It's still
hopeless: kernel mappings are always TLB-cold, right?
Quoted 8 lines
Quoted 3 lines
I get the feeling that a lot of Xen stuff is that kind of "legacy hacks" that should just be cleaned up, but nobody has the energy or the interest.
Time, mainly.
There certainly are areas which should be up for re-evaluation, given 9 years of change in hardware.
Is Xen PV still widely used? I'm surprised that users can tolerate this kind of overhead.
Signature · 1 line
Kirill A. Shutemov
Quoted 22 lines · Kirill A. Shutemov
Quoted 21 lines · Andrew Cooper
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Kirill A. Shutemov
I'm feeling I miss very basic background on how Xen works, but why does it set
_PAGE_GLOBALon userspace entries? It sounds strange to me.It is definitely strange. I'm guessing that it's some ancient Xen hack for the early Intel virtualization that used to have absolutely horrendous vmenter/exit costs, including very much the TLB overhead. \
These days, Intel has address space identifiers, and doesn't flush the whole TLB on VM entry/exit, so it's probably pointless to play games with the global bit.
It was introduced in 2006, but has nothing to do with VT-x
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=6f562e72cdc4b7e1519e23be75f812aebbf41db3
As long mode drops segment limit checking, the only way to protect a 64bit PV kernel from its userspace (both of which run in ring3 on user pages) is to maintain two sets of pagetables and switch between them on guest kernel/user context switches. The user set lack kernel mappings.
Quoted 5 lines
Quoted 3 lines
I can't comment about the performance impact of the patch (way before my time), but the justification was to try and reduce the overhead of guest context switches.
IIUC, it tries to reduce
userspace->kernelswitch in guest. It's still
hopeless: kernel mappings are always TLB-cold, right?
There is no way to avoid the kernel mappings being TLB-cold on a guest user -> kernel context switch.
A lot of the "legacy 32bit stuff" which was dropped in AMD64 were exactly the bits Xen was using to efficiently make PV guests safe.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
I get the feeling that a lot of Xen stuff is that kind of "legacy hacks" that should just be cleaned up, but nobody has the energy or the interest.
Time, mainly.
There certainly are areas which should be up for re-evaluation, given 9 years of change in hardware.
Is Xen PV still widely used?
Dom0 realistically still needs to be PV.
PVH (hardware extensions but no qemu emulating a motherboard) is on the horizon but still very much experimental with open issues needing to be solved.
Quoted 1 line
I'm surprised that users can tolerate this kind of overhead.
For modern hardware, most workloads are now "better" in HVM guests, but even only 5 years ago, the vmentry/vmexit overhead tended to outweigh the PV overheads.
On the other hand, unikernel virtual machines can always be more efficient as PV guests, so PV is not going to die any time soon.
~Andrew
Quoted 22 lines · Kirill A. Shutemov
Quoted 21 lines · Andrew Cooper
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Kirill A. Shutemov
I'm feeling I miss very basic background on how Xen works, but why does it set
_PAGE_GLOBALon userspace entries? It sounds strange to me.It is definitely strange. I'm guessing that it's some ancient Xen hack for the early Intel virtualization that used to have absolutely horrendous vmenter/exit costs, including very much the TLB overhead. \
These days, Intel has address space identifiers, and doesn't flush the whole TLB on VM entry/exit, so it's probably pointless to play games with the global bit.
It was introduced in 2006, but has nothing to do with VT-x
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=6f562e72cdc4b7e1519e23be75f812aebbf41db3
As long mode drops segment limit checking, the only way to protect a 64bit PV kernel from its userspace (both of which run in ring3 on user pages) is to maintain two sets of pagetables and switch between them on guest kernel/user context switches. The user set lack kernel mappings.
Quoted 5 lines
Quoted 3 lines
I can't comment about the performance impact of the patch (way before my time), but the justification was to try and reduce the overhead of guest context switches.
IIUC, it tries to reduce
userspace->kernelswitch in guest. It's still
hopeless: kernel mappings are always TLB-cold, right?
There is no way to avoid the kernel mappings being TLB-cold on a guest user -> kernel context switch.
A lot of the "legacy 32bit stuff" which was dropped in AMD64 were exactly the bits Xen was using to efficiently make PV guests safe.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
I get the feeling that a lot of Xen stuff is that kind of "legacy hacks" that should just be cleaned up, but nobody has the energy or the interest.
Time, mainly.
There certainly are areas which should be up for re-evaluation, given 9 years of change in hardware.
Is Xen PV still widely used?
Dom0 realistically still needs to be PV.
PVH (hardware extensions but no qemu emulating a motherboard) is on the horizon but still very much experimental with open issues needing to be solved.
Quoted 1 line
I'm surprised that users can tolerate this kind of overhead.
For modern hardware, most workloads are now "better" in HVM guests, but even only 5 years ago, the vmentry/vmexit overhead tended to outweigh the PV overheads.
On the other hand, unikernel virtual machines can always be more efficient as PV guests, so PV is not going to die any time soon.
~Andrew