Fix x86_64 page fault scheduler race

Ok, please tell me where I am wrong then.. by looking into arch/x86/mm/fault.c, I see that vmalloc_sync_all() touches pgd_list entries while the pgd_lock spinlock is taken, with interrupts disabled. So it's protected against concurrent pgd_list modification from

a - vmalloc_sync_all() on other CPUs
b - local interrupts

However, a completely normal interrupt can come on a remote CPU, run vmalloc_fault() and issue a set_pgd concurrently. Therefore I conclude this interrupt disable is not there to insure any kind of protection against concurrent updates.

Also, we see that vmalloc_fault has comments such as :

(for x86_32)
* Do _not_ use "current" here. We might be inside
* an interrupt in the middle of a task switch..

So it takes the pgd_addr from cr3, not from current. Using only the stack/registers makes this NMI-safe even if "current" is invalid when the NMI comes. This is caused by the fact that __switch_to will update the registers before updating current_task without disabling interrupts.

You are right in that x86_64 does not seems to play as safely as x86_32 on this matter; it uses current->mm. Probably it shouldn't assume "current" is valid. Actually, I don't see where x86_64 disables interrupts around __switch_to, so this would seem to be a race condition. Or have I missed something ?

---
 arch/x86/mm/fault.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Index: linux-2.6-lttng/arch/x86/mm/fault.c
===================================================================
@@ -513,6 +513,7 @@ static int vmalloc_fault(unsigned long a
 		return -1;
 	return 0;
 #else
+	unsigned long pgd_paddr;
 	pgd_t *pgd, *pgd_ref;
 	pud_t *pud, *pud_ref;
 	pmd_t *pmd, *pmd_ref;

@@ -526,7 +527,8 @@ static int vmalloc_fault(unsigned long a
 	   happen within a race in page table update. In the later
 	   case just flush. */
 
-	pgd = pgd_offset(current->mm ?: &init_mm, address);
+	pgd_paddr = read_cr3();
+	pgd = __va(pgd_paddr) + pgd_index(address);
 	pgd_ref = pgd_offset_k(address);
 	if (pgd_none(*pgd_ref))
 		return -1;

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

the scheduler disables interrupts around __switch_to(). (x86 does not set __ARCH_WANT_INTERRUPTS_ON_CTXSW)

Ingo

Ok, so I guess it's only useful to NMIs then. However, it makes me wonder why this comment was there in the first place on x86_32 vmalloc_fault() and why it uses read_cr3() :

* Do _not_ use "current" here. We might be inside
* an interrupt in the middle of a task switch..

Mathieu

Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

hm, i guess it's still useful to keep the __ARCH_WANT_INTERRUPTS_ON_CTXSW case working too. On -rt we used to enable it to squeeze a tiny bit more latency out of the system.

Ingo