Linux 3.10.47-rt50-rc1

Dear RT Folks,

This is the RT stable review cycle of patch 3.10.47-rt50-rc1.

Please scream at me if I messed something up. Please test the patches too.

The -rc release will be uploaded to kernel.org and will be deleted when the final release is out. This is just a review release (or release candidate).

The pre-releases will not be pushed to the git repository, only the final release is.

If all goes well, this patch will be converted to the next main release on 7/17/2014.

Enjoy,

-- Steve

To build 3.10.47-rt50-rc1 directly, the following patches should be applied:

http://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz

http://www.kernel.org/pub/linux/kernel/v3.x/patch-3.10.47.xz

http://www.kernel.org/pub/linux/kernel/projects/rt/3.10/patch-3.10.47-rt50-rc1.patch.xz

You can also build from 3.10.47-rt49 by applying the incremental patch:

http://www.kernel.org/pub/linux/kernel/projects/rt/3.10/incr/patch-3.10.47-rt49-rt50-rc1.patch.xz

Changes from 3.10.47-rt49:

---

Steven Rostedt (1):
sched: Do not clear PF_NO_SETAFFINITY flag in select_fallback_rq()

Steven Rostedt (Red Hat) (1):

      Linux 3.10.47-rt50-rc1

Thomas Gleixner (1):
workqueue: Prevent deadlock/stall on RT

----

 kernel/sched/core.c | 13 +++++-------
 kernel/workqueue.c  | 61 +++++++++++++++++++++++++++++++++++++++++------------
 localversion-rt     |  2 +-
 3 files changed, 54 insertions(+), 22 deletions(-)

3.10.47-rt50-rc1 stable review patch.
If anyone has any objections, please let me know.

------------------

From: Steven Rostedt <[email protected]>

I talked with Peter Zijlstra about this, and he told me that the clearing of the PF_NO_SETAFFINITY flag was to deal with the optimization of migrate_disable/enable() that ignores tasks that have that flag set. But that optimization was removed when I did a rework of the cpu hotplug code.

I found that ignoring tasks that had that flag set would cause those tasks to not sync with the hotplug code and cause the kernel to crash. Thus it needed to not treat them special and those tasks had to go though the same work as tasks without that flag set.

Now that those tasks are not treated special, there's no reason to clear the flag.

May still need to be tested as the migrate_me() code does not ignore those flags.

---
 kernel/sched/core.c | 6 ------
 1 file changed, 6 deletions(-)

index a060a092e92b..b8acecc0600f 100644
@@ -1289,12 +1289,6 @@ out:
 		}
 	}
 
-	/*
-	 * Clear PF_NO_SETAFFINITY, otherwise we wreckage
-	 * migrate_disable/enable. See optimization for
-	 * PF_NO_SETAFFINITY tasks there.
-	 */
-	p->flags &= ~PF_NO_SETAFFINITY;
 	return dest_cpu;
 }
 

2.0.0

3.10.47-rt50-rc1 stable review patch.
If anyone has any objections, please let me know.

------------------

From: Thomas Gleixner <[email protected]>

Austin reported a XFS deadlock/stall on RT where scheduled work gets never exececuted and tasks are waiting for each other for ever.

The underlying problem is the modification of the RT code to the handling of workers which are about to go to sleep. In mainline a worker thread which goes to sleep wakes an idle worker if there is more work to do. This happens from the guts of the schedule() function. On RT this must be outside and the accessed data structures are not protected against scheduling due to the spinlock to rtmutex conversion. So the naive solution to this was to move the code outside of the scheduler and protect the data structures by the pool lock. That approach turned out to be a little naive as we cannot call into that code when the thread blocks on a lock, as it is not allowed to block on two locks in parallel. So we dont call into the worker wakeup magic when the worker is blocked on a lock, which causes the deadlock/stall observed by Austin and Mike.

Looking deeper into that worker code it turns out that the only relevant data structure which needs to be protected is the list of idle workers which can be woken up.

So the solution is to protect the list manipulation operations with preempt_enable/disable pairs on RT and call unconditionally into the worker code even when the worker is blocked on a lock. The preemption protection is safe as there is nothing which can fiddle with the list outside of thread context.

---
 kernel/sched/core.c |  7 ++++--
 kernel/workqueue.c  | 61 +++++++++++++++++++++++++++++++++++++++++------------
 2 files changed, 53 insertions(+), 15 deletions(-)

index b8acecc0600f..f7aa4ca0cedb 100644
@@ -3217,9 +3217,8 @@ need_resched:
 
 static inline void sched_submit_work(struct task_struct *tsk)
 {
-	if (!tsk->state || tsk_is_pi_blocked(tsk))
+	if (!tsk->state)
 		return;
-
 	/*
 	 * If a worker went to sleep, notify and ask workqueue whether
 	 * it wants to wake up a task to maintain concurrency.

@@ -3227,6 +3226,10 @@ static inline void sched_submit_work(struct task_struct *tsk)
 	if (tsk->flags & PF_WQ_WORKER)
 		wq_worker_sleeping(tsk);
 
+
+	if (tsk_is_pi_blocked(tsk))
+		return;
+
 	/*
 	 * If we are going to sleep and we have plugged IO queued,
 	 * make sure to submit it to avoid deadlocks.

2.0.0

3.10.47-rt50-rc1 stable review patch.
If anyone has any objections, please let me know.

------------------

From: "Steven Rostedt (Red Hat)" <[email protected]>

---
 localversion-rt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 4b7dca68a5b4..e8a9a36bb066 100644
@@ -1 +1 @@
--rt49
+-rt50-rc1

2.0.0

Can we get:

tracing-use-migrate_disable-to-prevent-beeing-pushed.patch

into 3.10, also?

Thanks,

-corey

Sure except that patch is buggy:

- preempt_disable();

+               migrate_disable();
                /* The update must run on the CPU that is being updated. */
                if (cpu_id == smp_processor_id() || !cpu_online(cpu_id))
                        rb_update_pages(cpu_buffer);
                else {
-                       /*
-                        * Can not disable preemption for schedule_work_on()
-                        * on PREEMPT_RT.
-                        */
-                       preempt_enable();
                        schedule_work_on(cpu_id,
                                         &cpu_buffer->update_pages_work);
                        wait_for_completion(&cpu_buffer->update_done);
-                       preempt_disable();
                }

- preempt_enable();
+ migrate_enable();

migrate_disable() on non-PREEMPT_RT is preempt_disable(). You can't call wait_or_completion with preemption disabled.

When that gets fixed in mainline -rt, I'll add it to the stable branches too.

-- Steve

I originally did a patch that just always did the else clause (the schedule_work_on() and wait_for_completion()) on all CPUs. That seemed to work just fine and simplifies the code a bit and gets rid of all the preempt/migrate calls. You could try that approach, or I could submit something if you liked.

-corey

Yeah, perhaps that's the way to go. But it needs to go to mainline before it goes to -rt.

Can you resend it against my for-next branch.

Thanks,

-- Steve