Skip to contentSkip to conversation

i386: PAE entries must have their low word cleared first

PATCH In review Jan Beulich 6 participants 11 messages Apr 26, 2006 – Apr 27, 2006

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 i386: PAE entries must have their low word cleared first

During Xen development we noticed that writes to 64-bit variables may get coded as high-word-before-low-word writes by gcc. Consequently, when invalidating a PAE page table entry, the low word must be explicitly written first, as otherwise, as pointed out by Keir, speculative execution may result in a partially modified (and still valid) translation to be used, which has, according to the specification, the potential of dead-locking the machine.

diff -Npru /home/jbeulich/tmp/linux-2.6.17-rc2/include/asm-i386/pgtable.h 2.6.17-rc2-i386-pae-ptep_get_and_clear_full/include/asm-i386/pgtable.h

include/asm-i386/pgtable.h renamed from home/jbeulich/tmp/linux-2.6.17-rc2/include/asm-i386/pgtable.h +8 -0
@@ -268,7 +268,15 @@ static inline pte_t ptep_get_and_clear_f
 	pte_t pte;
 	if (full) {
 		pte = *ptep;
+#ifdef CONFIG_X86_PAE
+		/* Cannot do this in a single step, as the compiler
+		   may issue the two stores in either order. */
+		ptep->pte_low = 0;
+		barrier();
+		ptep->pte_high = 0;
+#else
 		*ptep = __pte(0);
+#endif
 	} else {
 		pte = ptep_get_and_clear(mm, addr, ptep);
 	}
#1 lore raw
Quoted 6 lines · Jan Beulich

During Xen development we noticed that writes to 64-bit variables may get coded as high-word-before-low-word writes by gcc. Consequently, when invalidating a PAE page table entry, the low word must be explicitly written first, as otherwise, as pointed out by Keir, speculative execution may result in a partially modified (and still valid) translation to be used, which has, according to the specification, the potential of dead-locking the machine.

If that's so (I don't trust my judgement on matters of speculative execution), then I think you'd do better to replace the *ptep = __pte(0) by pte_clear(mm, addr, ptep), and so avoid your ugly #ifdef'ing: please check, but I think you'll find that reduces to just the barrier you want. CC'ed Zach since it's his optimization, and he'll judge that spexecution.

Hugh

Quoted 31 lines

diff -Npru /home/jbeulich/tmp/linux-2.6.17-rc2/include/asm-i386/pgtable.h 2.6.17-rc2-i386-pae-ptep_get_and_clear_full/include/asm-i386/pgtable.h

include/asm-i386/pgtable.h renamed from home/jbeulich/tmp/linux-2.6.17-rc2/include/asm-i386/pgtable.h +8 -1
@@ -268,7 +268,15 @@ static inline pte_t ptep_get_and_clear_f
 	pte_t pte;
 	if (full) {
 		pte = *ptep;
+#ifdef CONFIG_X86_PAE
+		/* Cannot do this in a single step, as the compiler
+		   may issue the two stores in either order. */
+		ptep->pte_low = 0;
+		barrier();
+		ptep->pte_high = 0;
+#else
 		*ptep = __pte(0);
+#endif
 	} else {
 		pte = ptep_get_and_clear(mm, addr, ptep);
 	}
 
 
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 5 lines · Hugh Dickins

If that's so (I don't trust my judgement on matters of speculative execution), then I think you'd do better to replace the *ptep = __pte(0) by pte_clear(mm, addr, ptep), and so avoid your ugly #ifdef'ing: please check, but I think you'll find that reduces to just the barrier you want. CC'ed Zach since it's his optimization, and he'll judge that spexecution.

In more detail the problem is that, since we're still running on the page tables while clearing them, the CPU may choose to prefetch a half-cleared pte into its TLB, and then execute speculative memory accesses based on that mapping (including ones that may write-allocate cachelines, leading to problems like the AMD AGP GART deadlock Linux had a year or so back).

The barrier is needed to ensure that the bottom half is cleared before the top half. In fact it probably ought to be wmb() -- that's clearer in intent and actually reduces to barrier() on all PAE-capable platforms.

We cannot use pte_clear() unless we redefine it for PAE. Currently it reduces to set_pte() which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).

Also, I think wmb() should be used in PAE's set_pte(), rather than the current smp_wmb(). They reduce to the same thing, but wmb() makes it clear this is is not an issue specific to SMP systems.

-- Keir

#3 lore raw
Quoted 2 lines · Keir Fraser

We cannot use pte_clear() unless we redefine it for PAE. Currently it reduces to set_pte() which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).

Yes, that means pte_clear() has the same bug. Why don't we redefine pte_clear for PAE? Then the ifdef can go away.

Quoted 1 line

Also, I think wmb() should be used in PAE's set_pte(), rather than the current smp_wmb(). They reduce to the same thing, but wmb() makes it clear this is is not an issue specific to SMP systems.

I agree with that. The problem is not related to SMP at all. I would propose this approach.

Proposed fix for ptep_get_and_clear_full PAE bug. Pte_clear had the same bug, so use the same fix for both.

include/asm-i386/pgtable.h +1 -2
Index: linux-2.6.17-rc/include/asm-i386/pgtable.h
===================================================================
@@ -204,7 +204,6 @@ extern unsigned long long __PAGE_KERNEL,
 extern unsigned long pg0[];
 
 #define pte_present(x)	((x).pte_low & (_PAGE_PRESENT | _PAGE_PROTNONE))
-#define pte_clear(mm,addr,xp)	do { set_pte_at(mm, addr, xp, __pte(0)); } while (0)
 
 /* To avoid harmful races, pmd_none(x) should check only the lower when PAE */
 #define pmd_none(x)	(!(unsigned long)pmd_val(x))
@@ -268,7 +267,7 @@ static inline pte_t ptep_get_and_clear_f
 	pte_t pte;
 	if (full) {
 		pte = *ptep;
-		*ptep = __pte(0);
+		pte_clear(mm, addr, ptep);
 	} else {
 		pte = ptep_get_and_clear(mm, addr, ptep);
 	}
include/asm-i386/pgtable-3level.h +7 -0
Index: linux-2.6.17-rc/include/asm-i386/pgtable-3level.h
===================================================================
@@ -85,6 +85,13 @@ static inline void pud_clear (pud_t * pu
 #define pmd_offset(pud, address) ((pmd_t *) pud_page(*(pud)) + \
 			pmd_index(address))
 
+static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
+{
+	ptep->pte_low = 0;
+	wmb();
+	ptep->pte_high = 0;
+}
+
 static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
 {
 	pte_t res;
include/asm-i386/pgtable-2level.h +2 -0
Index: linux-2.6.17-rc/include/asm-i386/pgtable-2level.h
===================================================================
@@ -18,6 +18,8 @@
 #define set_pte_atomic(pteptr, pteval) set_pte(pteptr,pteval)
 #define set_pmd(pmdptr, pmdval) (*(pmdptr) = (pmdval))
 
+#define pte_clear(mm,addr,xp)	do { set_pte_at(mm, addr, xp, __pte(0)); } while (0)
+
 #define ptep_get_and_clear(mm,addr,xp)	__pte(xchg(&(xp)->pte_low, 0))
 #define pte_same(a, b)		((a).pte_low == (b).pte_low)
 #define pte_page(x)		pfn_to_page(pte_pfn(x))
#4 lore raw
Quoted 8 lines · Zachary Amsden
Quoted 4 lines · Keir Fraser

We cannot use pte_clear() unless we redefine it for PAE. Currently it reduces to set_pte() which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).

Yes, that means pte_clear() has the same bug. Why don't we redefine pte_clear for PAE? Then the ifdef can go away.

Good, we agree.

Quoted 6 lines
Quoted 3 lines

Also, I think wmb() should be used in PAE's set_pte(), rather than the current smp_wmb(). They reduce to the same thing, but wmb() makes it clear this is is not an issue specific to SMP systems.

I agree with that. The problem is not related to SMP at all. I would propose this approach.

Attachment included...

Quoted 2 lines

Proposed fix for ptep_get_and_clear_full PAE bug. Pte_clear had the same bug, so use the same fix for both.

You need to expand that comment with text from Jan & Keir's patch:
Andrew will want to know just what this bug was. The patch looks good to me, except for the unnecessary do { } while (0) in the definition of pte_clear: ah, you're only copying what was already there, can't blame you for that, let's not worry about it.

Hugh

Quoted 54 lines

Index: linux-2.6.17-rc/include/asm-i386/pgtable.h
===================================================================

include/asm-i386/pgtable.h +1 -2
@@ -204,7 +204,6 @@ extern unsigned long long __PAGE_KERNEL,
 extern unsigned long pg0[];
 
 #define pte_present(x)	((x).pte_low & (_PAGE_PRESENT | _PAGE_PROTNONE))
-#define pte_clear(mm,addr,xp)	do { set_pte_at(mm, addr, xp, __pte(0)); } while (0)
 
 /* To avoid harmful races, pmd_none(x) should check only the lower when PAE */
 #define pmd_none(x)	(!(unsigned long)pmd_val(x))
@@ -268,7 +267,7 @@ static inline pte_t ptep_get_and_clear_f
 	pte_t pte;
 	if (full) {
 		pte = *ptep;
-		*ptep = __pte(0);
+		pte_clear(mm, addr, ptep);
 	} else {
 		pte = ptep_get_and_clear(mm, addr, ptep);
 	}

Index: linux-2.6.17-rc/include/asm-i386/pgtable-3level.h
===================================================================

include/asm-i386/pgtable-3level.h +7 -0
@@ -85,6 +85,13 @@ static inline void pud_clear (pud_t * pu
 #define pmd_offset(pud, address) ((pmd_t *) pud_page(*(pud)) + \
 			pmd_index(address))
 
+static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
+{
+	ptep->pte_low = 0;
+	wmb();
+	ptep->pte_high = 0;
+}
+
 static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
 {
 	pte_t res;

Index: linux-2.6.17-rc/include/asm-i386/pgtable-2level.h
===================================================================

include/asm-i386/pgtable-2level.h +2 -0
@@ -18,6 +18,8 @@
 #define set_pte_atomic(pteptr, pteval) set_pte(pteptr,pteval)
 #define set_pmd(pmdptr, pmdval) (*(pmdptr) = (pmdval))
 
+#define pte_clear(mm,addr,xp)	do { set_pte_at(mm, addr, xp, __pte(0)); } while (0)
+
 #define ptep_get_and_clear(mm,addr,xp)	__pte(xchg(&(xp)->pte_low, 0))
 #define pte_same(a, b)		((a).pte_low == (b).pte_low)
 #define pte_page(x)		pfn_to_page(pte_pfn(x))
#5 lore raw
Quoted 8 lines · Hugh Dickins
Quoted 2 lines

Proposed fix for ptep_get_and_clear_full PAE bug. Pte_clear had the same bug, so use the same fix for both.

You need to expand that comment with text from Jan & Keir's patch:
Andrew will want to know just what this bug was. The patch looks good to me, except for the unnecessary do { } while (0) in the definition of pte_clear: ah, you're only copying what was already there, can't blame you for that, let's not worry about it.

I agree: the patch comment is rather brief, but the patch itself looks good. I suppose switching PAE's set_pte() from smp_wmb() to wmb() would logically belong in a separate patch.

-- Keir

#6 lore raw
In reply to Keir Fraser
Quoted 4 lines · Keir Fraser

We cannot use pte_clear() unless we redefine it for PAE. Currently it reduces to set_pte() which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).

I overlooked that reversal completely. What a very good point. I think that actually pte_clear() _does_ need to be redefined for PAE, to reverse that ordering as you point out. Take a look at its use in mm/highmem.c (where a comment states it's safe against speculative execution, but a comment can't guarantee that!): what do you think?

Hugh

#7 lore raw
Quoted 7 lines · Hugh Dickins
Quoted 3 lines · Keir Fraser

We cannot use pte_clear() unless we redefine it for PAE. Currently it reduces to set_pte() which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).

Quoted 5 lines

I overlooked that reversal completely. What a very good point. I think that actually pte_clear() _does_ need to be redefined for PAE, to reverse that ordering as you point out. Take a look at its use in mm/highmem.c (where a comment states it's safe against speculative execution, but a comment can't guarantee that!): what do you think?

Speculative execution is safe I think (and so is ptep_get_and_clear_full, because in neither case will the virtual address be visible).

Speculative prefetching + tlb instantiation, apparently no.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#8 lore raw
In reply to Keir Fraser
Quoted 11 lines · Keir Fraser
Quoted 5 lines · Hugh Dickins

If that's so (I don't trust my judgement on matters of speculative execution), then I think you'd do better to replace the *ptep = __pte(0) by pte_clear(mm, addr, ptep), and so avoid your ugly #ifdef'ing: please check, but I think you'll find that reduces to just the barrier you want. CC'ed Zach since it's his optimization, and he'll judge that spexecution.

In more detail the problem is that, since we're still running on the page tables while clearing them, the CPU may choose to prefetch a half-cleared pte into its TLB, and then execute speculative memory accesses based on that mapping (including ones that may write-allocate cachelines, leading to problems like the AMD AGP GART deadlock Linux had a year or so back).

What do you mean, you're still running on the page tables? The CPU can still walk the pagetables?

Because if ptep_get_and_clear_full is passed non zero in the full argument, then that specific translation should never see another access. I didn't know CPUs now actually resolve TLB misses as part of speculative prefetching... does this really happen?

Do you have a pointer to where the AMD AGP GART deadlock was discussed? Google is having some trouble finding it.

Quoted 2 lines

The barrier is needed to ensure that the bottom half is cleared before the top half. In fact it probably ought to be wmb() -- that's clearer in intent and actually reduces to barrier() on all PAE-capable platforms.

I think barrier is correct there.

The full argument means that the mm is being torn down and we are the last thread in the mm. So these pagetables can't be visible to other CPUs, can they?

Quoted 2 lines

We cannot use pte_clear() unless we redefine it for PAE. Currently it reduces to set_pte() which explicitly uses the wrong ordering (sets high *then* low, because it's normally used to introduce a mapping).

Presumably you would redefine it for PAE also (in this case you may actually need an smp_wmb rather than just barrier).

Unless you can explain why this is a problem and not the other pte_clear callers.

Quoted 2 lines

Also, I think wmb() should be used in PAE's set_pte(), rather than the current smp_wmb(). They reduce to the same thing, but wmb() makes it clear this is is not an issue specific to SMP systems.

That is what smp_wmb() means. wmb() means it should also order io memory. Clear as mud?

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#9 lore raw
Quoted 18 lines · Nick Piggin
Quoted 10 lines · Keir Fraser
Quoted 5 lines · Hugh Dickins

If that's so (I don't trust my judgement on matters of speculative execution), then I think you'd do better to replace the *ptep = __pte(0) by pte_clear(mm, addr, ptep), and so avoid your ugly #ifdef'ing: please check, but I think you'll find that reduces to just the barrier you want. CC'ed Zach since it's his optimization, and he'll judge that spexecution.

Quoted 6 lines

In more detail the problem is that, since we're still running on the page tables while clearing them, the CPU may choose to prefetch a half-cleared pte into its TLB, and then execute speculative memory accesses based on that mapping (including ones that may write-allocate cachelines, leading to problems like the AMD AGP GART deadlock Linux had a year or so back).

Quoted 7 lines

What do you mean, you're still running on the page tables? The CPU can still walk the pagetables?

Because if ptep_get_and_clear_full is passed non zero in the full argument, then that specific translation should never see another access. I didn't know CPUs now actually resolve TLB misses as part of speculative prefetching... does this really happen?

For instance, during speculative execution on POWER, we can take a TLB miss for a speculative load and start a table-walk.

I'm not sure what "speculative prefetching" means in this case... just regular hardware-initiated prefetching (where I suppose one could use the modifier "speculative") on POWER will only prefetch to a page-boundary.

So, slightly OT, as this is not about x86 CPUs... but thought people might be interested.

(Added Anton to CC to call any BS on my part :-)

Sonny

#10 lore raw
Quoted 15 lines · Sonny Rao
Quoted 11 lines · Nick Piggin
Quoted 1 line · Keir Fraser

In more detail the problem is that, since we're still running on the page tables while clearing them, the CPU may choose to prefetch a half-cleared pte into its TLB, and then execute speculative memory accesses based on that mapping (including ones that may write-allocate cachelines, leading to problems like the AMD AGP GART deadlock Linux had a year or so back).

What do you mean, you're still running on the page tables? The CPU can still walk the pagetables?

Because if ptep_get_and_clear_full is passed non zero in the full argument, then that specific translation should never see another access. I didn't know CPUs now actually resolve TLB misses as part of speculative prefetching... does this really happen?

Quoted 6 lines

For instance, during speculative execution on POWER, we can take a TLB miss for a speculative load and start a table-walk.

I'm not sure what "speculative prefetching" means in this case... just regular hardware-initiated prefetching (where I suppose one could use the modifier "speculative") on POWER will only prefetch to a page-boundary.

speculative prefetching / hardware prefetching...

I believe this bug wouldn't happen as a result of speculative execution, because a load from this virtual address should never be in the instruction stream. So it would require some hardware initiated prefetch to establish the tlb entry.

Quoted 3 lines

So, slightly OT, as this is not about x86 CPUs... but thought people might be interested.

Thanks!

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com