Skip to contentSkip to conversation

update kmap_atomic on !HIGHMEM

PATCH In review Daniel Walker 2 participants 10 messages Dec 27, 2006 – Dec 28, 2006

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 update kmap_atomic on !HIGHMEM

I got some scheduling while atomic on x86-64 , and since x86-64 doesn't seem to have HIGHMEM there's no workaround for kmap_atomic() .

This patch adds the same as i386 HIGHMEM for !HIGHMEM.

---
 include/linux/highmem.h |   14 +++++++++++++-
 1 files changed, 13 insertions(+), 1 deletion(-)
include/linux/highmem.h +13 -1
Index: linux-2.6.19/include/linux/highmem.h
===================================================================
@@ -42,13 +42,25 @@ static inline void *kmap(struct page *pa
 
 #define kunmap(page) do { (void) (page); } while (0)
 
+#ifdef CONFIG_PREEMPT_RT
+/*
+ * kmap and kmunmap are above, and they don't really do anything
+ * interesting.
+ */
+# define kmap_atomic(page, idx)		kmap(page)
+# define kmap_atomic_pfn(pfn, idx)	kmap(pfn_to_page(pfn))
+# define kunmap_atomic(kvaddr, idx)	kunmap(kvaddr)
+#else
 #define kmap_atomic(page, idx) \
 	({ pagefault_disable(); page_address(page); })
 #define kunmap_atomic(addr, idx)	do { pagefault_enable(); } while (0)
 #define kmap_atomic_pfn(pfn, idx)	kmap_atomic(pfn_to_page(pfn), (idx))
-#define kmap_atomic_to_page(ptr)	virt_to_page(ptr)
 #endif
 
+#define kmap_atomic_to_page(ptr)	virt_to_page(ptr)
+
+#endif /* ARCH_HAS_KMAP */
+
 #endif /* CONFIG_HIGHMEM */
 
 /* when CONFIG_HIGHMEM is not set these will be plain clear/copy_page */

--

#1 lore raw
Quoted 1 line · Daniel Walker

I got some scheduling while atomic on x86-64 , [...]

please post those messages too.

Ingo

#2 lore raw
Quoted 7 lines · Ingo Molnar
Quoted 1 line · Daniel Walker

I got some scheduling while atomic on x86-64 , [...]

please post those messages too.

Ingo

Ok .. This is from 2.6.20-rc2-rt0 tho .

BUG: scheduling while atomic: kswapd0/0x00000001/244, CPU#1

Call Trace:
[<ffffffff80268f80>] __sched_text_start+0xb0/0xb9f
[<ffffffff8028fa8c>] task_rq_lock+0x4c/0x90
[<ffffffff8028fdd3>] rt_mutex_setprio+0xb3/0xd0
[<ffffffff802bb2c4>] add_preempt_count+0x14/0xf0
[<ffffffff802b1353>] task_blocks_on_rt_mutex+0x163/0x1d0
[<ffffffff80269da5>] schedule+0xe5/0x110
[<ffffffff8026afa2>] rt_spin_lock_slowlock+0x102/0x1b0
[<ffffffff802c9b71>] swap_duplicate+0x51/0x100
[<ffffffff802c94b2>] move_to_swap_cache+0x32/0x70
[<ffffffff802cf4ba>] shmem_writepage+0xaa/0x1a0
[<ffffffff802c35e0>] shrink_inactive_list+0x430/0x950
[<ffffffff802134f4>] shrink_zone+0xe4/0x110
[<ffffffff8025e629>] kswapd+0x379/0x510
[<ffffffff802a8270>] autoremove_wake_function+0x0/0x30
[<ffffffff8025e2b0>] kswapd+0x0/0x510
[<ffffffff80234c89>] kthread+0xd9/0x120
[<ffffffff8022981c>] schedule_tail+0xdc/0x130
[<ffffffff802a80a0>] keventd_create_kthread+0x0/0x90
[<ffffffff80265ef8>] child_rip+0xa/0x12
[<ffffffff802a80a0>] keventd_create_kthread+0x0/0x90
[<ffffffff80234bb0>] kthread+0x0/0x120
[<ffffffff80265eee>] child_rip+0x0/0x12

---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<ffffffff802cc111>] .... shmem_swp_entry+0x31/0x1a0
.....[<00000000>] .. ( <= 0x0)

#3 lore raw
In reply to Daniel Walker
Quoted 4 lines · Daniel Walker

I got some scheduling while atomic on x86-64 , and since x86-64 doesn't seem to have HIGHMEM there's no workaround for kmap_atomic() .

This patch adds the same as i386 HIGHMEM for !HIGHMEM.

the problem is that this does not disable pagefaulting while kmap-atomic. Could you try the patch below, does it solve the assert?

Ingo

------------------------->
Subject: [patch] clean up the page fault disabling logic
From: Ingo Molnar <[email protected]>

decouple the pagefault-disabled logic from the preempt count.

---
 arch/arm/mm/fault.c     |    2 +-
 arch/i386/mm/fault.c    |    2 +-
 arch/mips/mm/fault.c    |    2 +-
 arch/powerpc/mm/fault.c |    2 +-
 arch/x86_64/mm/fault.c  |    2 +-
 include/linux/sched.h   |    1 +
 include/linux/uaccess.h |   33 +++------------------------------
 kernel/fork.c           |    1 +
 mm/memory.c             |   20 ++++++++++++++++++++
 9 files changed, 30 insertions(+), 35 deletions(-)
arch/arm/mm/fault.c +1 -1
Index: linux/arch/arm/mm/fault.c
===================================================================
@@ -230,7 +230,7 @@ do_page_fault(unsigned long addr, unsign
 	 * If we're in an interrupt or have no user
 	 * context, we must not take the fault..
 	 */
-	if (in_atomic() || !mm)
+	if (in_atomic() || !mm || current->pagefault_disabled)
 		goto no_context;
 
 	/*
arch/i386/mm/fault.c +1 -1
Index: linux/arch/i386/mm/fault.c
===================================================================
@@ -382,7 +382,7 @@ fastcall notrace void __kprobes do_page_
 	 * If we're in an interrupt, have no user context or are running in an
 	 * atomic region then we must not take the fault..
 	 */
-	if (in_atomic() || !mm)
+	if (in_atomic() || !mm || current->pagefault_disabled)
 		goto bad_area_nosemaphore;
 
 	/* When running in the kernel we expect faults to occur only to
arch/mips/mm/fault.c +1 -1
Index: linux/arch/mips/mm/fault.c
===================================================================
@@ -69,7 +69,7 @@ asmlinkage void do_page_fault(struct pt_
 	 * If we're in an interrupt or have no user
 	 * context, we must not take the fault..
 	 */
-	if (in_atomic() || !mm)
+	if (in_atomic() || !mm || current->pagefault_disabled)
 		goto bad_area_nosemaphore;
 
 	down_read(&mm->mmap_sem);
arch/powerpc/mm/fault.c +1 -1
Index: linux/arch/powerpc/mm/fault.c
===================================================================
@@ -196,7 +196,7 @@ int __kprobes notrace do_page_fault(stru
 	}
 #endif /* !(CONFIG_4xx || CONFIG_BOOKE)*/
 
-	if (in_atomic() || mm == NULL) {
+	if (in_atomic() || mm == NULL || current->pagefault_disabled) {
 		if (!user_mode(regs))
 			return SIGSEGV;
 		/* in_atomic() in user mode is really bad,
arch/x86_64/mm/fault.c +1 -1
Index: linux/arch/x86_64/mm/fault.c
===================================================================
@@ -405,7 +405,7 @@ asmlinkage void __kprobes do_page_fault(
 	 * If we're in an interrupt or have no user
 	 * context, we must not take the fault..
 	 */
-	if (unlikely(in_atomic() || !mm))
+	if (unlikely(in_atomic() || !mm || current->pagefault_disabled))
 		goto bad_area_nosemaphore;
 
  again:
include/linux/sched.h +1 -0
Index: linux/include/linux/sched.h
===================================================================
@@ -1147,6 +1147,7 @@ struct task_struct {
 	/* mutex deadlock detection */
 	struct mutex_waiter *blocked_on;
 #endif
+	int pagefault_disabled;
 #ifdef CONFIG_TRACE_IRQFLAGS
 	unsigned int irq_events;
 	int hardirqs_enabled;
include/linux/uaccess.h +3 -30
Index: linux/include/linux/uaccess.h
===================================================================
@@ -6,37 +6,10 @@
 
 /*
  * These routines enable/disable the pagefault handler in that
- * it will not take any locks and go straight to the fixup table.
- *
- * They have great resemblance to the preempt_disable/enable calls
- * and in fact they are identical; this is because currently there is
- * no other way to make the pagefault handlers do this. So we do
- * disable preemption but we don't necessarily care about that.
+ * it will not take any MM locks and go straight to the fixup table.
  */
-static inline void pagefault_disable(void)
-{
-	inc_preempt_count();
-	/*
-	 * make sure to have issued the store before a pagefault
-	 * can hit.
-	 */
-	barrier();
-}
-
-static inline void pagefault_enable(void)
-{
-	/*
-	 * make sure to issue those last loads/stores before enabling
-	 * the pagefault handler again.
-	 */
-	barrier();
-	dec_preempt_count();
-	/*
-	 * make sure we do..
-	 */
-	barrier();
-	preempt_check_resched();
-}
+extern void pagefault_disable(void);
+extern void pagefault_enable(void);
 
 #ifndef ARCH_HAS_NOCACHE_UACCESS
 
kernel/fork.c +1 -0
Index: linux/kernel/fork.c
===================================================================
@@ -1128,6 +1128,7 @@ static struct task_struct *copy_process(
 	p->hardirq_context = 0;
 	p->softirq_context = 0;
 #endif
+	p->pagefault_disabled = 0;
 #ifdef CONFIG_LOCKDEP
 	p->lockdep_depth = 0; /* no locks held yet */
 	p->curr_chain_key = 0;
mm/memory.c +20 -0
Index: linux/mm/memory.c
===================================================================
@@ -2481,6 +2481,26 @@ unlock:
 	return VM_FAULT_MINOR;
 }
 
+void pagefault_disable(void)
+{
+	current->pagefault_disabled++;
+	/*
+	 * make sure to have issued the store before a pagefault
+	 * can hit.
+	 */
+	barrier();
+}
+
+void pagefault_enable(void)
+{
+	/*
+	 * make sure to issue those last loads/stores before enabling
+	 * the pagefault handler again.
+	 */
+	barrier();
+	current->pagefault_disabled--;
+}
+
 /*
  * By the time we get here, we already hold the mm semaphore
  */
#4 lore raw

plus on i386 the fix below is needed as well.

Ingo

---
 include/asm-i386/highmem.h |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
include/asm-i386/highmem.h +3 -3
Index: linux/include/asm-i386/highmem.h
===================================================================
@@ -81,9 +81,9 @@ struct page *__kmap_atomic_to_page(void 
  * on PREEMPT_RT kmap_atomic() is a wrapper that uses kmap():
  */
 #ifdef CONFIG_PREEMPT_RT
-#  define kmap_atomic(page, type)	kmap(page)
-#  define kmap_atomic_pfn(pfn, type)	kmap(pfn_to_page(pfn))
-#  define kunmap_atomic(kvaddr, type)	kunmap_virt(kvaddr)
+#  define kmap_atomic(page, type)	({ pagefault_disable(); kmap(page); })
+#  define kmap_atomic_pfn(pfn, type)	kmap_atomic(pfn_to_page(pfn), type)
+#  define kunmap_atomic(kvaddr, type)	do { pagefault_enable(); kunmap_virt(kvaddr); } while (0)
 #  define kmap_atomic_to_page(kvaddr)	kmap_to_page(kvaddr)
 #else
 # define kmap_atomic(page, type)	__kmap_atomic(page, type)
#5 lore raw
Quoted 2 lines · Ingo Molnar

plus on i386 the fix below is needed as well.

We do it on most other arches .. PowerPC , and I think ARM too.

Daniel

#6 lore raw
Quoted 5 lines · Daniel Walker
Quoted 2 lines · Ingo Molnar

plus on i386 the fix below is needed as well.

We do it on most other arches .. PowerPC , and I think ARM too.

I dont think they currently work with PREEMPT_RT and HIGHMEM enabled, but if/once they do they'll need a similar fix.

Ingo

#7 lore raw
In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar
Quoted 4 lines · Daniel Walker

I got some scheduling while atomic on x86-64 , and since x86-64 doesn't seem to have HIGHMEM there's no workaround for kmap_atomic() .

This patch adds the same as i386 HIGHMEM for !HIGHMEM.

the problem is that this does not disable pagefaulting while kmap-atomic. Could you try the patch below, does it solve the assert?

That goes for the other CONFIG_HIGHMEM changes also? I mean if we can't do it here we shouldn't do it else where..

(note, I just used testscripts/ltpstress.sh from LTP to reproduce this.)

Daniel

#8 lore raw
In reply to Ingo Molnar
Quoted 11 lines · Ingo Molnar
mm/memory.c +10 -0
+void pagefault_disable(void)
+{
+       current->pagefault_disabled++;
+       /*
+        * make sure to have issued the store before a pagefault
+        * can hit.
+        */
+       barrier();
+}
+

mm/memory.c unlock: · line 2503

Quoted 10 lines
mm/memory.c +10 -0
+void pagefault_enable(void)
+{
+       /*
+        * make sure to issue those last loads/stores before enabling
+        * the pagefault handler again.
+        */
+       barrier();
+       current->pagefault_disabled--;
+}
+

I'll test it, but clearly I'm not going to get the same scheduling while atomic using this change.. So may as well just put it in -rt and see if something crashes ..

Daniel

#9 lore raw
In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar
Quoted 4 lines · Daniel Walker

I got some scheduling while atomic on x86-64 , and since x86-64 doesn't seem to have HIGHMEM there's no workaround for kmap_atomic() .

This patch adds the same as i386 HIGHMEM for !HIGHMEM.

the problem is that this does not disable pagefaulting while kmap-atomic. Could you try the patch below, does it solve the assert?

It hasn't crashed after about an hour of stress.. One comment on the patch tho , the pagefault_enable/pagefault_disable need module exports, I saw the build complaining about them.

Daniel