i386: PAE entries must have their low word cleared first
PATCH 1/1 i386: PAE entries must have their low word cleared first
During Xen development we noticed that writes to 64-bit variables may get coded as high-word-before-low-word writes by gcc. Consequently, when invalidating a PAE page table entry, the low word must be explicitly written first, as otherwise, as pointed out by Keir, speculative execution may result in a partially modified (and still valid) translation to be used, which has, according to the specification, the potential of dead-locking the machine.
diff -Npru /home/jbeulich/tmp/linux-2.6.17-rc2/include/asm-i386/pgtable.h 2.6.17-rc2-i386-pae-ptep_get_and_clear_full/include/asm-i386/pgtable.h
include/asm-i386/pgtable.h renamed from home/jbeulich/tmp/linux-2.6.17-rc2/include/asm-i386/pgtable.h +8 -0
@@ -268,7 +268,15 @@ static inline pte_t ptep_get_and_clear_fpte_t pte;if (full) { pte = *ptep; +#ifdef CONFIG_X86_PAE +/* Cannot do this in a single step, as the compiler +may issue the two stores in either order. */ + ptep->pte_low =0 ; +barrier (); + ptep->pte_high =0 ; +#else *ptep =__pte (0 ); +#endif }else { pte =ptep_get_and_clear (mm, addr, ptep); }
Quoted 6 lines · Jan Beulich
During Xen development we noticed that writes to 64-bit variables may get coded as high-word-before-low-word writes by gcc. Consequently, when invalidating a PAE page table entry, the low word must be explicitly written first, as otherwise, as pointed out by Keir, speculative execution may result in a partially modified (and still valid) translation to be used, which has, according to the specification, the potential of dead-locking the machine.
If that's so (I don't trust my judgement on matters of speculative execution), then I think you'd do better to replace the *ptep = __pte(0) by pte_clear(mm, addr, ptep), and so avoid your ugly #ifdef'ing: please check, but I think you'll find that reduces to just the barrier you want. CC'ed Zach since it's his optimization, and he'll judge that spexecution.
Hugh
Quoted 31 lines
Signed-off-by Jan Beulich <[email protected]>Cc Keir Fraser <[email protected]>diff -Npru
/home/jbeulich/tmp/linux-2.6.17-rc2/include/asm-i386/pgtable.h2.6.17-rc2-i386-pae-ptep_get_and_clear_full/include/asm-i386/pgtable.hinclude/asm-i386/pgtable.h renamed from home/jbeulich/tmp/linux-2.6.17-rc2/include/asm-i386/pgtable.h +8
-1@@ -268,7 +268,15 @@ static inline pte_t ptep_get_and_clear_fpte_t pte;if (full) { pte = *ptep; +#ifdef CONFIG_X86_PAE +/* Cannot do this in a single step, as the compiler +may issue the two stores in either order. */ + ptep->pte_low =0 ; +barrier (); + ptep->pte_high =0 ; +#else *ptep =__pte (0 ); +#endif }else { pte =ptep_get_and_clear (mm, addr, ptep); } -To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Hugh Dickins
If that's so (I don't trust my judgement on matters of speculative execution), then I think you'd do better to replace the *ptep =
__pte(0)bypte_clear(mm, addr, ptep), and so avoid your ugly #ifdef'ing: please check, but I think you'll find that reduces to just the barrier you want. CC'ed Zach since it's his optimization, and he'll judge that spexecution.
In more detail the problem is that, since we're still running on the page tables while clearing them, the CPU may choose to prefetch a half-cleared pte into its TLB, and then execute speculative memory accesses based on that mapping (including ones that may write-allocate cachelines, leading to problems like the AMD AGP GART deadlock Linux had a year or so back).
The barrier is needed to ensure that the bottom half is cleared before the top half. In fact it probably ought to be wmb() -- that's clearer in intent and actually reduces to barrier() on all PAE-capable platforms.
We cannot use pte_clear() unless we redefine it for PAE. Currently it reduces to set_pte() which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).
Also, I think wmb() should be used in PAE's set_pte(), rather than the current smp_wmb(). They reduce to the same thing, but wmb() makes it clear this is is not an issue specific to SMP systems.
-- Keir
Quoted 2 lines · Keir Fraser
We cannot use
pte_clear()unless we redefine it for PAE. Currently it reduces toset_pte()which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).
Yes, that means pte_clear() has the same bug. Why don't we redefine pte_clear for PAE? Then the ifdef can go away.
Quoted 1 line
Also, I think
wmb()should be used in PAE'sset_pte(), rather than the currentsmp_wmb(). They reduce to the same thing, butwmb()makes it clear this is is not an issue specific to SMP systems.
I agree with that. The problem is not related to SMP at all. I would propose this approach.
Proposed fix for ptep_get_and_clear_full PAE bug. Pte_clear had the same bug, so use the same fix for both.
include/asm-i386/pgtable.h +1 -2
@@ -204,7 +204,6 @@ extern unsigned long long __PAGE_KERNEL,extern unsigned long pg0[];#define pte_present (x) ((x).pte_low & (_PAGE_PRESENT | _PAGE_PROTNONE)) -#define pte_clear (mm,addr,xp) do { set_pte_at(mm, addr, xp, __pte(0)); } while (0)/* To avoid harmful races, pmd_none(x) should check only the lower when PAE */ #define pmd_none (x) (!(unsigned long)pmd_val(x))
@@ -268,7 +267,7 @@ static inline pte_t ptep_get_and_clear_fpte_t pte;if (full) { pte = *ptep; - *ptep =__pte (0 ); +pte_clear (mm, addr, ptep); }else { pte =ptep_get_and_clear (mm, addr, ptep); }
include/asm-i386/pgtable-3level.h +7 -0
@@ -85,6 +85,13 @@ static inline void pud_clear (pud_t * pu#define pmd_offset (pud, address) ((pmd_t *) pud_page(*(pud)) + \ pmd_index(address)) +static inline void pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) +{ + ptep->pte_low =0 ; +wmb (); + ptep->pte_high =0 ; +} +static inline pte_t ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {pte_t res;
include/asm-i386/pgtable-2level.h +2 -0
@@ -18,6 +18,8 @@#define set_pte_atomic (pteptr, pteval) set_pte(pteptr,pteval)#define set_pmd (pmdptr, pmdval) (*(pmdptr) = (pmdval)) +#define pte_clear (mm,addr,xp) do { set_pte_at(mm, addr, xp, __pte(0)); } while (0) +#define ptep_get_and_clear (mm,addr,xp) __pte(xchg(&(xp)->pte_low, 0))#define pte_same (a, b) ((a).pte_low == (b).pte_low)#define pte_page (x) pfn_to_page(pte_pfn(x))
Quoted 8 lines · Zachary Amsden
Quoted 4 lines · Keir Fraser
We cannot use
pte_clear()unless we redefine it for PAE. Currently it reduces toset_pte()which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).Yes, that means
pte_clear()has the same bug. Why don't we redefinepte_clearfor PAE? Then the ifdef can go away.
Good, we agree.
Quoted 6 lines
Quoted 3 lines
Also, I think
wmb()should be used in PAE'sset_pte(), rather than the currentsmp_wmb(). They reduce to the same thing, butwmb()makes it clear this is is not an issue specific to SMP systems.I agree with that. The problem is not related to SMP at all. I would propose this approach.
Attachment included...
Quoted 2 lines
Proposed fix for
ptep_get_and_clear_fullPAE bug.Pte_clearhad the same bug, so use the same fix for both.
You need to expand that comment with text from Jan & Keir's patch:
Andrew will want to know just what this bug was. The patch looks good to me, except for the unnecessary do { } while (0) in the definition of pte_clear: ah, you're only copying what was already there, can't blame you for that, let's not worry about it.
Hugh
Quoted 54 lines
Signed-off-by Zachary Amsden <[email protected]>Index:
linux-2.6.17-rc/include/asm-i386/pgtable.h
===================================================================include/asm-i386/pgtable.h +1
-2@@ -204,7 +204,6 @@ extern unsigned long long __PAGE_KERNEL,extern unsigned long pg0[];#define pte_present (x) ((x).pte_low & (_PAGE_PRESENT | _PAGE_PROTNONE)) -#define pte_clear (mm,addr,xp) do { set_pte_at(mm, addr, xp, __pte(0)); } while (0)/* To avoid harmful races, pmd_none(x) should check only the lower when PAE */ #define pmd_none (x) (!(unsigned long)pmd_val(x))@@ -268,7 +267,7 @@ static inline pte_t ptep_get_and_clear_fpte_t pte;if (full) { pte = *ptep; - *ptep =__pte (0 ); +pte_clear (mm, addr, ptep); }else { pte =ptep_get_and_clear (mm, addr, ptep); }Index:
linux-2.6.17-rc/include/asm-i386/pgtable-3level.h
===================================================================include/asm-i386/pgtable-3level.h +7
-0@@ -85,6 +85,13 @@ static inline void pud_clear (pud_t * pu#define pmd_offset (pud, address) ((pmd_t *) pud_page(*(pud)) + \ pmd_index(address)) +static inline void pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) +{ + ptep->pte_low =0 ; +wmb (); + ptep->pte_high =0 ; +} +static inline pte_t ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {pte_t res;Index:
linux-2.6.17-rc/include/asm-i386/pgtable-2level.h
===================================================================include/asm-i386/pgtable-2level.h +2
-0@@ -18,6 +18,8 @@#define set_pte_atomic (pteptr, pteval) set_pte(pteptr,pteval)#define set_pmd (pmdptr, pmdval) (*(pmdptr) = (pmdval)) +#define pte_clear (mm,addr,xp) do { set_pte_at(mm, addr, xp, __pte(0)); } while (0) +#define ptep_get_and_clear (mm,addr,xp) __pte(xchg(&(xp)->pte_low, 0))#define pte_same (a, b) ((a).pte_low == (b).pte_low)#define pte_page (x) pfn_to_page(pte_pfn(x))
Quoted 8 lines · Hugh Dickins
Quoted 2 lines
Proposed fix for
ptep_get_and_clear_fullPAE bug.Pte_clearhad the same bug, so use the same fix for both.You need to expand that comment with text from Jan & Keir's patch:
Andrew will want to know just what this bug was. The patch looks good to me, except for the unnecessary do { } while (0) in the definition ofpte_clear: ah, you're only copying what was already there, can't blame you for that, let's not worry about it.
I agree: the patch comment is rather brief, but the patch itself looks good. I suppose switching PAE's set_pte() from smp_wmb() to wmb() would logically belong in a separate patch.
-- Keir
Quoted 4 lines · Keir Fraser
We cannot use
pte_clear()unless we redefine it for PAE. Currently it reduces toset_pte()which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).
I overlooked that reversal completely. What a very good point. I think that actually pte_clear() _does_ need to be redefined for PAE, to reverse that ordering as you point out. Take a look at its use in mm/highmem.c (where a comment states it's safe against speculative execution, but a comment can't guarantee that!): what do you think?
Hugh
Quoted 7 lines · Hugh Dickins
Quoted 3 lines · Keir Fraser
We cannot use
pte_clear()unless we redefine it for PAE. Currently it reduces toset_pte()which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).
Quoted 5 lines
I overlooked that reversal completely. What a very good point. I think that actually
pte_clear()_does_ need to be redefined for PAE, to reverse that ordering as you point out. Take a look at its use inmm/highmem.c(where a comment states it's safe against speculative execution, but a comment can't guarantee that!): what do you think?
Speculative execution is safe I think (and so is ptep_get_and_clear_full, because in neither case will the virtual address be visible).
Speculative prefetching + tlb instantiation, apparently no.
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Quoted 11 lines · Keir Fraser
Quoted 5 lines · Hugh Dickins
If that's so (I don't trust my judgement on matters of speculative execution), then I think you'd do better to replace the *ptep =
__pte(0)bypte_clear(mm, addr, ptep), and so avoid your ugly #ifdef'ing: please check, but I think you'll find that reduces to just the barrier you want. CC'ed Zach since it's his optimization, and he'll judge that spexecution.In more detail the problem is that, since we're still running on the page tables while clearing them, the CPU may choose to prefetch a half-cleared pte into its TLB, and then execute speculative memory accesses based on that mapping (including ones that may write-allocate cachelines, leading to problems like the AMD AGP GART deadlock Linux had a year or so back).
What do you mean, you're still running on the page tables? The CPU can still walk the pagetables?
Because if ptep_get_and_clear_full is passed non zero in the full argument, then that specific translation should never see another access. I didn't know CPUs now actually resolve TLB misses as part of speculative prefetching... does this really happen?
Do you have a pointer to where the AMD AGP GART deadlock was discussed? Google is having some trouble finding it.
Quoted 2 lines
The barrier is needed to ensure that the bottom half is cleared before the top half. In fact it probably ought to be
wmb()-- that's clearer in intent and actually reduces tobarrier()on all PAE-capable platforms.
I think barrier is correct there.
The full argument means that the mm is being torn down and we are the last thread in the mm. So these pagetables can't be visible to other CPUs, can they?
Quoted 2 lines
We cannot use
pte_clear()unless we redefine it for PAE. Currently it reduces toset_pte()which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).
Presumably you would redefine it for PAE also (in this case you may actually need an smp_wmb rather than just barrier).
Unless you can explain why this is a problem and not the other pte_clear callers.
Quoted 2 lines
Also, I think
wmb()should be used in PAE'sset_pte(), rather than the currentsmp_wmb(). They reduce to the same thing, butwmb()makes it clear this is is not an issue specific to SMP systems.
That is what smp_wmb() means. wmb() means it should also order io memory. Clear as mud?
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com
Quoted 18 lines · Nick Piggin
Quoted 10 lines · Keir Fraser
Quoted 5 lines · Hugh Dickins
If that's so (I don't trust my judgement on matters of speculative execution), then I think you'd do better to replace the *ptep =
__pte(0)bypte_clear(mm, addr, ptep), and so avoid your ugly #ifdef'ing: please check, but I think you'll find that reduces to just the barrier you want. CC'ed Zach since it's his optimization, and he'll judge that spexecution.Quoted 6 lines
In more detail the problem is that, since we're still running on the page tables while clearing them, the CPU may choose to prefetch a half-cleared pte into its TLB, and then execute speculative memory accesses based on that mapping (including ones that may write-allocate cachelines, leading to problems like the AMD AGP GART deadlock Linux had a year or so back).
Quoted 7 lines
What do you mean, you're still running on the page tables? The CPU can still walk the pagetables?
Because if
ptep_get_and_clear_fullis passed non zero in the full argument, then that specific translation should never see another access. I didn't know CPUs now actually resolve TLB misses as part of speculative prefetching... does this really happen?
For instance, during speculative execution on POWER, we can take a TLB miss for a speculative load and start a table-walk.
I'm not sure what "speculative prefetching" means in this case... just regular hardware-initiated prefetching (where I suppose one could use the modifier "speculative") on POWER will only prefetch to a page-boundary.
So, slightly OT, as this is not about x86 CPUs... but thought people might be interested.
(Added Anton to CC to call any BS on my part :-)
Sonny
Quoted 15 lines · Sonny Rao
Quoted 11 lines · Nick Piggin
Quoted 1 line · Keir Fraser
In more detail the problem is that, since we're still running on the page tables while clearing them, the CPU may choose to prefetch a half-cleared pte into its TLB, and then execute speculative memory accesses based on that mapping (including ones that may write-allocate cachelines, leading to problems like the AMD AGP GART deadlock Linux had a year or so back).
What do you mean, you're still running on the page tables? The CPU can still walk the pagetables?
Because if
ptep_get_and_clear_fullis passed non zero in the full argument, then that specific translation should never see another access. I didn't know CPUs now actually resolve TLB misses as part of speculative prefetching... does this really happen?
Quoted 6 lines
For instance, during speculative execution on POWER, we can take a TLB miss for a speculative load and start a table-walk.
I'm not sure what "speculative prefetching" means in this case... just regular hardware-initiated prefetching (where I suppose one could use the modifier "speculative") on POWER will only prefetch to a page-boundary.
speculative prefetching / hardware prefetching...
I believe this bug wouldn't happen as a result of speculative execution, because a load from this virtual address should never be in the instruction stream. So it would require some hardware initiated prefetch to establish the tlb entry.
Quoted 3 lines
So, slightly OT, as this is not about x86 CPUs... but thought people might be interested.
Thanks!
Signature · 2 lines
SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com