Scheduler questions

I'd like to understand which task structure elements are protected by which locks (as far as scheduling is concerned). Is there somewhere a paper summarizing the mutual exclusion rules ?

Let's take some code e.g. from the 2.6.5 kernel:

set_cpus_allowed(task_t *p, cpumask_t new_mask):

	rq = task_rq_lock(p, &flags)
	__set_cpus_allowed(p, new_mask, &req):

		p->cpus_allowed = new_mask
		/*
		 * If the task is not on a runqueue (and not running), then
		 * it is sufficient to simply update the task's cpu field.
		 */
		if (!p->array && !task_running(rq, p))
			set_task_cpu(p, any_online_cpu(p->cpus_allowed))
		/* ... */

	task_rq_unlock(rq, &flags)

Apparently, the "p->cpus_allowed" and the "p->thread_info->cpu" fields are protected by the "(&per_cpu(runqueues, (p->thread_info->cpu)))->lock".

Which are the other task structure elements protected by the same lock ?

Let's take an example:

- I've got a sleeping task that ran on the CPU #3 previously
- I want to set its CPU mask equal to {1, 2}
- I take the lock of the run queue #3
- I do set the CPU mask of the task
- It's not running (BTW when can happen that "p->array" is NULL and the task is still running ?)
- I do set the task's CPU e.g. equal to 2. As a consequence, the task falls out of the protection provided by the lock of the run queue #3.
- Someone else deciding to play with the same task, s/he takes the lock of the run queue #2 !!!
- Me, I have not arrived yet to the unlock. There are stuffs to do before.
- Both of us think to have the exclusive access right to the task...

Can someone explain me, please, why I have to take a run queue lock to protect a not running task, and why we do not use "proc_lock" instead ?

Thanks,

Zoltán Menyhárt

I do "some" modification to the virtual memory subsystem. How can I make sure that the compatibility is kept ? Have you got some VM validation packages, stress tests, etc. ? Could you please, indicate how I can pick up to these tests ?

Thanks,

Zoltán Menyhárt

There are few, if any ABI compatibility test suites. By and large, important breakages are almost immediately observable, and the less important ones can be fixed up afterward, so I'd encourage you to proceed with testing by means of targeted testcases when a specific area is in doubt, and general benchmarks and stress tests for the rest.

-- wli

Okay.

Yes. ->cpus_allowed is protected by rq->lock. Do not look at the midget behind the curtain for what protects ->cpus_allowed of a sleeping task.

You looked at the midget behind the curtain. Anyway, rq->lock is just reused based on the sleeping task's cpu affinity.

->array == NULL while a task is running should not happen apart from a dequeueing in progress. ->array should be protected by per_cpu(runqueues, task_cpu(task)).lock.

Scheduler-private fields of tasks are considered to be under the protection of per_cpu(runqueues, task_cpu(task)).lock until someone succeeds in a double runqueue lock acquisition and moves the task between two cpus, or as an optimization, to reset task->thread_info->cpu under its current rq->lock without holding the target's lock if it's not queued anywhere.

There is some retry logic in task_rq_lock() to make sure the cpu and the runqueue are consistent that resolves the remainder of the cases.

-- wli