2.5: don't miss a preemption

Current kernel preemption code opens a small window between the check for need_resched in schedule and the setting of preempt_count to zero in preempt_schedule. While this window is generally short (a few cycles) the resulting period of non-preemptibility could be as long as the next timer tick - much longer, in fact, if a lock is obtained in the interim.

This patch checks for need_resched in preempt_schedule after setting preempt_count back to zero, before returning. The overhead is negligible and it is crucial to never miss a preemption opportunity.

Also fixes/clarifies some comments. Patch is against 2.5.8 ... enjoy,

Robert Love

diff -urN linux-2.5.8/kernel/sched.c linux/kernel/sched.c
@@ -765,8 +765,8 @@
 	spin_lock_irq(&rq->lock);
 
 	/*
-	 * if entering from preempt_schedule, off a kernel preemption,
-	 * go straight to picking the next task.
+	 * if entering off of a kernel preemption go
+	 * straight to picking the next task.
 	 */
 	if (unlikely(preempt_get_count() & PREEMPT_ACTIVE))
 		goto pick_next_task;

@@ -842,7 +842,9 @@
 
 #ifdef CONFIG_PREEMPT
 /*
- * this is is the entry point to schedule() from in-kernel preemption.
+ * this is is the entry point to schedule() from in-kernel preemption
+ * off of preempt_enable.  Preemptions off return-from-interrupt are
+ * handled directly in that codepath.
  */
 asmlinkage void preempt_schedule(void)
 {

@@ -851,10 +853,15 @@
 	if (unlikely(ti->preempt_count))
 		return;
 
+need_resched:
 	ti->preempt_count = PREEMPT_ACTIVE;
 	schedule();
 	ti->preempt_count = 0;
 	barrier();
+
+	/* we could miss a preemption between schedule() and now */
+	if (unlikely(test_thread_flag(TIF_NEED_RESCHED)))
+		goto need_resched;
 }
 #endif /* CONFIG_PREEMPT */
 

I'm curious: why is it crucial to never miss a preemption opportunity?

Hugh

Two main reasons:

(1) In 2.5, we have a kernel preemption model that makes the fully preemptible, subject to SMP locking constraints and a few other rules. Without this patch, we break this model and do not allow preemption when it is in fact legal.

(2) Like I said, it may be awhile before we can preempt again. If we take a lock after return from schedule but before the next interrupt, it can be many tens (or hundreds) of milliseconds before we release the lock and subsequently preempt. If need_resched was set in response to an important real-time application, the wait can be detrimental. Servicing apps as soon as they become runnable is the point of preempt-kernel, anyhow.

It is not crucial in the sense we break anything; merely that we are working toward providing very efficient response and dispatch to interactive and real-time applications and we _must_ respond to them as soon as possible.

Robert Love