sched, x86: Prevent resched interrupts if task in kernel mode and !CONFIG_PREEMPT

RFC In review Kirill Tkhai 3 participants 7 messages Jan 23, 2015 – Feb 3, 2015 x86

Share & tools

Kirill Tkhai

author

2015-01-23 15:53 UTC #0 lore raw

RFC 1/1 sched, x86: Prevent resched interrupts if task in kernel mode and !CONFIG_PREEMPT

It's useless to send reschedule interrupts in such situations. The earliest point, where schedule() call is possible, is sysret_careful(). But in that function we directly test TIF_NEED_RESCHED.

So it's possible to get rid of that type of interrupts.

How about this idea? Is set_bit() cheap on x86 machines?

---
 arch/x86/kernel/entry_64.S |   10 ++++++++++
 1 file changed, 10 insertions(+)

arch/x86/kernel/entry_64.S +10 -0

index c653dc4..a046ba8 100644
@@ -409,6 +409,13 @@ GLOBAL(system_call_after_swapgs)
 	movq_cfi rax,(ORIG_RAX-ARGOFFSET)
 	movq  %rcx,RIP-ARGOFFSET(%rsp)
 	CFI_REL_OFFSET rip,RIP-ARGOFFSET
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+	/*
+	 * Tell resched_curr() do not send useless interrupts to us.
+	 * Kernel isn't preemptible till sysret_careful() anyway.
+	 */
+	LOCK ; bts $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
 	testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
 	jnz tracesys
 system_call_fastpath:

@@ -427,6 +434,9 @@ GLOBAL(system_call_after_swapgs)
  * Has incomplete stack frame and undefined top of stack.
  */
 ret_from_sys_call:
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+	LOCK ; btr $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
 	movl $_TIF_ALLWORK_MASK,%edi
 	/* edi:	flagmask */
 sysret_check:

Peter Zijlstra

2015-01-23 16:07 UTC #1 lore raw

Quoted 7 lines · Kirill Tkhai

It's useless to send reschedule interrupts in such situations. The earliest point, where schedule() call is possible, is sysret_careful(). But in that function we directly test TIF_NEED_RESCHED.
So it's possible to get rid of that type of interrupts.
How about this idea? Is set_bit() cheap on x86 machines?

So you set TIF_POLLING_NRFLAG on syscall entry and clear it again on exit? Thereby we avoid the IPI, because the exit path already checks for TIF_NEED_RESCHED.

Should work I suppose, but I'm not too familiar with all that entry.S muck. Andy might know and appreciate this.

Quoted 35 lines

---
 arch/x86/kernel/entry_64.S |   10 ++++++++++
 1 file changed, 10 insertions(+)

arch/x86/kernel/entry_64.S +10 -0

index c653dc4..a046ba8 100644
@@ -409,6 +409,13 @@ GLOBAL(system_call_after_swapgs)
 	movq_cfi rax,(ORIG_RAX-ARGOFFSET)
 	movq  %rcx,RIP-ARGOFFSET(%rsp)
 	CFI_REL_OFFSET rip,RIP-ARGOFFSET
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+	/*
+	 * Tell resched_curr() do not send useless interrupts to us.
+	 * Kernel isn't preemptible till sysret_careful() anyway.
+	 */
+	LOCK ; bts $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
 	testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
 	jnz tracesys
 system_call_fastpath:

@@ -427,6 +434,9 @@ GLOBAL(system_call_after_swapgs)
  * Has incomplete stack frame and undefined top of stack.
  */
 ret_from_sys_call:
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+	LOCK ; btr $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
 	movl $_TIF_ALLWORK_MASK,%edi
 	/* edi:	flagmask */
 sysret_check:

Andy Lutomirski

2015-01-23 16:25 UTC #2 lore raw

Quoted 5 lines · Peter Zijlstra

Quoted 4 lines · Kirill Tkhai
It's useless to send reschedule interrupts in such situations. The earliest point, where schedule() call is possible, is sysret_careful(). But in that function we directly test TIF_NEED_RESCHED.

Quoted 7 lines

Quoted 3 lines
So it's possible to get rid of that type of interrupts.
How about this idea? Is set_bit() cheap on x86 machines?
So you set TIF_POLLING_NRFLAG on syscall entry and clear it again on exit? Thereby we avoid the IPI, because the exit path already checks for TIF_NEED_RESCHED.

The idle code says:

/*
* If the arch has a polling bit, we maintain an invariant:
*
* Our polling bit is clear if we're not scheduled (i.e. if
* rq->curr != rq->idle). This means that, if rq->idle has
* the polling bit set, then setting need_resched is
* guaranteed to cause the cpu to reschedule.
*/

Setting polling on non-idle tasks like this will either involve weakening this a bit (it'll still be true for rq->idle) or changing the polling state on context switch.

Quoted 8 lines

Should work I suppose, but I'm not too familiar with all that entry.S muck. Andy might know and appreciate this.
Quoted 4 lines
---
 arch/x86/kernel/entry_64.S |   10 ++++++++++
 1 file changed, 10 insertions(+)

Quoted 15 lines

Quoted 5 lines

arch/x86/kernel/entry_64.S +0 -0
index c653dc4..a046ba8 100644

arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 418

Quoted 10 lines

arch/x86/kernel/entry_64.S +7 -0

@@ -409,6 +409,13 @@ GLOBAL(system_call_after_swapgs)
      movq_cfi rax,(ORIG_RAX-ARGOFFSET)
      movq  %rcx,RIP-ARGOFFSET(%rsp)
      CFI_REL_OFFSET rip,RIP-ARGOFFSET
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+     /*
+      * Tell resched_curr() do not send useless interrupts to us.
+      * Kernel isn't preemptible till sysret_careful() anyway.
+      */
+     LOCK ; bts $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif

That's kind of expensive. What's the !SMP part for?

Quoted 10 lines

arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 439

Quoted 10 lines

arch/x86/kernel/entry_64.S +3 -0

      testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
      jnz tracesys
 system_call_fastpath:

@@ -427,6 +434,9 @@ GLOBAL(system_call_after_swapgs)
  * Has incomplete stack frame and undefined top of stack.
  */
 ret_from_sys_call:
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+     LOCK ; btr $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif

If only it were this simple. There are lots of ways out of syscalls, and this is only one of them :( If we did this, I'd rather do it through the do_notify_resume mechanism or something.

I don't see any way to do this without at least one atomic op or smp_mb per syscall, and that's kind of expensive.

Would it make sense to try to use context tracking instead? On systems that use context tracking, syscalls are already expensive, and we're already keeping track of which CPUs are in user mode.

--Andy

Kirill Tkhai

author

2015-01-23 17:10 UTC #3 lore raw

В Пт, 23/01/2015 в 08:24 -0800, Andy Lutomirski пишет:

Quoted 46 lines · Peter Zijlstra

Quoted 5 lines · Kirill Tkhai
Quoted 4 lines
It's useless to send reschedule interrupts in such situations. The earliest point, where schedule() call is possible, is sysret_careful(). But in that function we directly test TIF_NEED_RESCHED.
Quoted 7 lines
Quoted 3 lines
So it's possible to get rid of that type of interrupts.
How about this idea? Is set_bit() cheap on x86 machines?
So you set TIF_POLLING_NRFLAG on syscall entry and clear it again on exit? Thereby we avoid the IPI, because the exit path already checks for TIF_NEED_RESCHED.
The idle code says:
/*
* If the arch has a polling bit, we maintain an invariant:
*
* Our polling bit is clear if we're not scheduled (i.e. if
* rq->curr != rq->idle). This means that, if rq->idle has
* the polling bit set, then setting need_resched is
* guaranteed to cause the cpu to reschedule.
*/
Setting polling on non-idle tasks like this will either involve weakening this a bit (it'll still be true for rq->idle) or changing the polling state on context switch.
Quoted 17 lines
Should work I suppose, but I'm not too familiar with all that entry.S muck. Andy might know and appreciate this.
Quoted 13 lines
---
 arch/x86/kernel/entry_64.S |   10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/kernel/entry_64.S +1 -0
index c653dc4..a046ba8 100644
@@ -409,6 +409,13 @@ GLOBAL(system_call_after_swapgs)
      movq_cfi rax,(ORIG_RAX-ARGOFFSET)
      movq  %rcx,RIP-ARGOFFSET(%rsp)
      CFI_REL_OFFSET rip,RIP-ARGOFFSET
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)

Quoted 8 lines

Quoted 6 lines
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 418
Quoted 6 lines
arch/x86/kernel/entry_64.S +6 -0
+     /*
+      * Tell resched_curr() do not send useless interrupts to us.
+      * Kernel isn't preemptible till sysret_careful() anyway.
+      */
+     LOCK ; bts $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
That's kind of expensive. What's the !SMP part for?

smp_send_reschedule() is NOP on UP. There is no problem.

Quoted 15 lines

Quoted 11 lines
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 439
Quoted 11 lines
arch/x86/kernel/entry_64.S +3 -0
      testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
      jnz tracesys
 system_call_fastpath:
@@ -427,6 +434,9 @@ GLOBAL(system_call_after_swapgs)
  * Has incomplete stack frame and undefined top of stack.
  */
 ret_from_sys_call:
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+     LOCK ; btr $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
If only it were this simple. There are lots of ways out of syscalls, and this is only one of them :( If we did this, I'd rather do it through the do_notify_resume mechanism or something.

Yes, syscall is the only thing I did as an example.

Quoted 2 lines

I don't see any way to do this without at least one atomic op or smp_mb per syscall, and that's kind of expensive.

JFI, doesn't x86 set_bit() lock a small area of memory? I thought it's not very expensive on this arch (some bus optimizations or something like this).

Quoted 3 lines

Would it make sense to try to use context tracking instead? On systems that use context tracking, syscalls are already expensive, and we're already keeping track of which CPUs are in user mode.

I'll look at context_tracking, but I'm not sure some smp synchronization there.

Thanks,
Kirill

Andy Lutomirski

2015-01-24 02:37 UTC #4 lore raw

Quoted 25 lines · Kirill Tkhai

В Пт, 23/01/2015 в 08:24 -0800, Andy Lutomirski пишет:
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Kirill Tkhai
Quoted 13 lines
---
 arch/x86/kernel/entry_64.S |   10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/kernel/entry_64.S +1 -0
index c653dc4..a046ba8 100644
@@ -409,6 +409,13 @@ GLOBAL(system_call_after_swapgs)
      movq_cfi rax,(ORIG_RAX-ARGOFFSET)
      movq  %rcx,RIP-ARGOFFSET(%rsp)
      CFI_REL_OFFSET rip,RIP-ARGOFFSET
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
Quoted 8 lines
Quoted 6 lines
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 418
Quoted 6 lines
arch/x86/kernel/entry_64.S +6 -0
+     /*
+      * Tell resched_curr() do not send useless interrupts to us.
+      * Kernel isn't preemptible till sysret_careful() anyway.
+      */
+     LOCK ; bts $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
That's kind of expensive. What's the !SMP part for?

Quoted 1 line

smp_send_reschedule() is NOP on UP. There is no problem.

Shouldn't it be #if !defined(CONFIG_PREEMPT) && defined(CONFIG_SMP) then?

Quoted 17 lines

Quoted 16 lines
Quoted 12 lines
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 439
Quoted 12 lines
arch/x86/kernel/entry_64.S +3 -0
      testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
      jnz tracesys
 system_call_fastpath:
@@ -427,6 +434,9 @@ GLOBAL(system_call_after_swapgs)
  * Has incomplete stack frame and undefined top of stack.
  */
 ret_from_sys_call:
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+     LOCK ; btr $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
If only it were this simple. There are lots of ways out of syscalls, and this is only one of them :( If we did this, I'd rather do it through the do_notify_resume mechanism or something.

Quoted 8 lines

Yes, syscall is the only thing I did as an example.
Quoted 2 lines
I don't see any way to do this without at least one atomic op or smp_mb per syscall, and that's kind of expensive.
JFI, doesn't x86 set_bit() lock a small area of memory? I thought it's not very expensive on this arch (some bus optimizations or something like this).

An entire syscall on x86 is well under 200 cycles. lock addl is >20 cycles for me, and I don't see why the atomic bitops would be faster. (Oddly, mfence is slower than lock addl, which is really odd, since lock addl implies mfence.) So this overhead may actually matter.

Quoted 7 lines

Quoted 4 lines
Would it make sense to try to use context tracking instead? On systems that use context tracking, syscalls are already expensive, and we're already keeping track of which CPUs are in user mode.
I'll look at context_tracking, but I'm not sure some smp synchronization there.

It could be combinable with existing synchronization there.

--Andy

Kirill Tkhai

author

2015-01-26 11:59 UTC #5 lore raw

В Пт, 23/01/2015 в 18:36 -0800, Andy Lutomirski пишет:

Quoted 26 lines · Kirill Tkhai

Quoted 25 lines
В Пт, 23/01/2015 в 08:24 -0800, Andy Lutomirski пишет:
Quoted 23 lines · Peter Zijlstra
Quoted 5 lines · Kirill Tkhai
Quoted 4 lines
---
 arch/x86/kernel/entry_64.S |   10 ++++++++++
 1 file changed, 10 insertions(+)
Quoted 15 lines
Quoted 5 lines
arch/x86/kernel/entry_64.S +0 -0
index c653dc4..a046ba8 100644
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 418
Quoted 10 lines
arch/x86/kernel/entry_64.S +7 -0
@@ -409,6 +409,13 @@ GLOBAL(system_call_after_swapgs)
      movq_cfi rax,(ORIG_RAX-ARGOFFSET)
      movq  %rcx,RIP-ARGOFFSET(%rsp)
      CFI_REL_OFFSET rip,RIP-ARGOFFSET
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+     /*
+      * Tell resched_curr() do not send useless interrupts to us.
+      * Kernel isn't preemptible till sysret_careful() anyway.
+      */
+     LOCK ; bts $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
That's kind of expensive. What's the !SMP part for?

Quoted 3 lines

Quoted 1 line
smp_send_reschedule() is NOP on UP. There is no problem.
Shouldn't it be #if !defined(CONFIG_PREEMPT) && defined(CONFIG_SMP) then?

Definitely, thanks.

Quoted 23 lines

Quoted 23 lines
Quoted 17 lines
Quoted 13 lines
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 439
Quoted 13 lines
arch/x86/kernel/entry_64.S +3 -0
      testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
      jnz tracesys
 system_call_fastpath:
@@ -427,6 +434,9 @@ GLOBAL(system_call_after_swapgs)
  * Has incomplete stack frame and undefined top of stack.
  */
 ret_from_sys_call:
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+     LOCK ; btr $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
If only it were this simple. There are lots of ways out of syscalls, and this is only one of them :( If we did this, I'd rather do it through the do_notify_resume mechanism or something.
Yes, syscall is the only thing I did as an example.
Quoted 2 lines
I don't see any way to do this without at least one atomic op or smp_mb per syscall, and that's kind of expensive.

Quoted 8 lines

Quoted 3 lines
JFI, doesn't x86 set_bit() lock a small area of memory? I thought it's not very expensive on this arch (some bus optimizations or something like this).
An entire syscall on x86 is well under 200 cycles. lock addl is >20 cycles for me, and I don't see why the atomic bitops would be faster. (Oddly, mfence is slower than lock addl, which is really odd, since lock addl implies mfence.) So this overhead may actually matter.

Yeah, it's really big overhead.

Quoted 9 lines

Quoted 7 lines
Quoted 4 lines
Would it make sense to try to use context tracking instead? On systems that use context tracking, syscalls are already expensive, and we're already keeping track of which CPUs are in user mode.
I'll look at context_tracking, but I'm not sure some smp synchronization there.
It could be combinable with existing synchronization there.

I'll look at this. Thanks!

Kirill

Kirill Tkhai

author

2015-02-03 17:15 UTC #6 lore raw

В Пн, 26/01/2015 в 14:58 +0300, Kirill Tkhai пишет:

Quoted 72 lines

В Пт, 23/01/2015 в 18:36 -0800, Andy Lutomirski пишет:
Quoted 26 lines · Kirill Tkhai
Quoted 25 lines
В Пт, 23/01/2015 в 08:24 -0800, Andy Lutomirski пишет:
Quoted 23 lines · Peter Zijlstra
Quoted 5 lines · Kirill Tkhai
Quoted 4 lines
---
 arch/x86/kernel/entry_64.S |   10 ++++++++++
 1 file changed, 10 insertions(+)
Quoted 15 lines
Quoted 5 lines
arch/x86/kernel/entry_64.S +0 -0
index c653dc4..a046ba8 100644
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 418
Quoted 10 lines
arch/x86/kernel/entry_64.S +7 -0
@@ -409,6 +409,13 @@ GLOBAL(system_call_after_swapgs)
      movq_cfi rax,(ORIG_RAX-ARGOFFSET)
      movq  %rcx,RIP-ARGOFFSET(%rsp)
      CFI_REL_OFFSET rip,RIP-ARGOFFSET
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+     /*
+      * Tell resched_curr() do not send useless interrupts to us.
+      * Kernel isn't preemptible till sysret_careful() anyway.
+      */
+     LOCK ; bts $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
That's kind of expensive. What's the !SMP part for?
Quoted 3 lines
Quoted 1 line
smp_send_reschedule() is NOP on UP. There is no problem.
Shouldn't it be #if !defined(CONFIG_PREEMPT) && defined(CONFIG_SMP) then?
Definitely, thanks.
Quoted 23 lines
Quoted 23 lines
Quoted 17 lines
Quoted 13 lines
arch/x86/kernel/entry_64.S GLOBAL(system_call_after_swapgs) · line 439
Quoted 13 lines
arch/x86/kernel/entry_64.S +3 -0
      testl $_TIF_WORK_SYSCALL_ENTRY,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
      jnz tracesys
 system_call_fastpath:
@@ -427,6 +434,9 @@ GLOBAL(system_call_after_swapgs)
  * Has incomplete stack frame and undefined top of stack.
  */
 ret_from_sys_call:
+#if !defined(CONFIG_PREEMPT) || !defined(SMP)
+     LOCK ; btr $TIF_POLLING_NRFLAG,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
+#endif
If only it were this simple. There are lots of ways out of syscalls, and this is only one of them :( If we did this, I'd rather do it through the do_notify_resume mechanism or something.
Yes, syscall is the only thing I did as an example.
Quoted 2 lines
I don't see any way to do this without at least one atomic op or smp_mb per syscall, and that's kind of expensive.
Quoted 8 lines
Quoted 3 lines
JFI, doesn't x86 set_bit() lock a small area of memory? I thought it's not very expensive on this arch (some bus optimizations or something like this).
An entire syscall on x86 is well under 200 cycles. lock addl is >20 cycles for me, and I don't see why the atomic bitops would be faster. (Oddly, mfence is slower than lock addl, which is really odd, since lock addl implies mfence.) So this overhead may actually matter.
Yeah, it's really big overhead.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Would it make sense to try to use context tracking instead? On systems that use context tracking, syscalls are already expensive, and we're already keeping track of which CPUs are in user mode.

Quoted 6 lines

Quoted 4 lines
Quoted 2 lines
I'll look at context_tracking, but I'm not sure some smp synchronization there.
It could be combinable with existing synchronization there.
I'll look at this. Thanks!

Сontinuing the theme. I've tried the idea with RCU. Fast & dirty patch which prevents unnecessary interrupts.

It prevents 2% of reschedule IPIs. The cost is atomic_read() in resched_curr(). It looks like the profit is not too much...

Kirill