Skip to contentSkip to conversation

timers: Timer migration

RFC PATCH In review Arun R Bharadwaj 3 participants 5 messages Oct 16, 2008 – Oct 17, 2008

lore
Share & tools
lore
#0 lore raw

cover timers: Timer migration

Hi,

(This is merely for discussion purposes and not for inclusion).

This is a re-work of the earlier patch which i had sent. Link to the earlier patch - http://lkml.org/lkml/2008/9/16/40
I have made the following changes from my previous patch:

1) Created a new framework for identifying cpu-pinned hrtimers,so that such hrtimers are ignored during migration of timers.

2)A better sysfs interface which allows you to echo a target cpu number to the per-cpu sysfs entry and all timers are migrated to that cpu, instead of choosing cpu0 by default.

This patch set is based on the kernel version 2.6.27.

Here's a brief introduction as to why we need timer migration.

An idle cpu on which device drivers have initialized timers, or any timer that is (re)queued from a softirq context has to be frequently woken up to service the timers. So, consolidation of timers onto a fewer number of cpus is important. Migration of timers from idle cpus onto lesser idle cpus is necessary. Currently, timers are migrated during the cpu offline operation. However cpu-hotplug for the sake of idle system power management is too heavy. So, this patch implements a lightweight timer migration framework.

Also, in machines with large number of CPUs, when utilization is not high enough, but is not 0% either, we would want to consolidate all the system activity to as fewer number of packages as possible.

a) Interrupts are usually re-routed using the power-aware irqbalance daemon.

b) For tasks, we have hooks in the scheduler which can consolidate tasks to a fewer number of CPUs.

c) The remaining part of the system activity is the timers, which can be queued from a task or a softirq context.

c-1)If they're queued from the task context, then they migrate whenever the task is migrated.

c-2) However, if they're requeued from a softirq context, then it's not possible to currently migrate them unless the CPU is offlined.

Hence, the need for a minimalistic framework which allows to migrate the last remaining system activity from the last few idle-but-serving-timers CPUs of an otherwise idle package, to a package which is already having some amount of system activity on them.

Also, this kind of a framework will be helpful for a certain class of applications like the High Performance (HPC) applications, where we would want to restrict the system housekeeping activities to as fewer number of CPUs as possible, in order to minimize the jitter caused by these housekeeping activities.

Lastly, the algorithm which decides to which cpu the timer should be migrated to should be conservative in the sense that, it should migrate only if the target cpu is sufficiently busy so that it is not woken up from an idle state. In that case where even target cpu is idle, the penalty of wake up would be same on either of the cpus.

Tests carried out:

a) I have tested this patch by stressing the system using a script which continuously hotplug-add and removes the cpus.

b) Also ran kernbench. The kernbench results with and without my patches on were fairly similar.

#1 lore raw

On Thu, 16 Oct 2008 17:12:39 +0530

Quoted 4 lines

Here's a brief introduction as to why we need timer migration.

as I've said before, for reducing power consumption I don't buy into needing this concept....

If you have high frequency timers, you've lost no matter what (and we fixed almost all of them in the ecosystem); if you don't have those then this patch doesn't really add anything.
(and the range-hrtimers fixes the intermediate state)

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#2 lore raw
In reply to Arun R Bharadwaj

On Thu, 16 Oct 2008 17:12:39 +0530

Quoted 4 lines

Here's a brief introduction as to why we need timer migration.

as I've said before, for reducing power consumption I don't buy into needing this concept....

If you have high frequency timers, you've lost no matter what (and we fixed almost all of them in the ecosystem); if you don't have those then this patch doesn't really add anything.
(and the range-hrtimers fixes the intermediate state)

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#3 lore raw
Quoted 8 lines

From: Arjan van de Ven
Sent: Thursday, October 16, 2008 10:38 PM
On Thu, 16 Oct 2008 17:12:39 +0530

Quoted 4 lines

Here's a brief introduction as to why we need timer migration.

Quoted 8 lines

as I've said before, for reducing power consumption I don't buy into needing this concept....

If you have high frequency timers, you've lost no matter what (and we fixed almost all of them in the ecosystem); if you don't have those then this patch doesn't really add anything.
(and the range-hrtimers fixes the intermediate state)

Does range-hrtimers work in per-cpu style or globally? If the former I guess this timer migration may help range-hrtimers feature to further align rest timers. If globally, then small room remains.

Arun, did you have any actual power improvement data with it?

Thanks,
Kevin

#4 lore raw
In reply to Arjan van de Ven
Quoted 8 lines

From: Arjan van de Ven
Sent: Thursday, October 16, 2008 10:38 PM
On Thu, 16 Oct 2008 17:12:39 +0530

Quoted 4 lines

Here's a brief introduction as to why we need timer migration.

Quoted 8 lines

as I've said before, for reducing power consumption I don't buy into needing this concept....

If you have high frequency timers, you've lost no matter what (and we fixed almost all of them in the ecosystem); if you don't have those then this patch doesn't really add anything.
(and the range-hrtimers fixes the intermediate state)

Does range-hrtimers work in per-cpu style or globally? If the former I guess this timer migration may help range-hrtimers feature to further align rest timers. If globally, then small room remains.

Arun, did you have any actual power improvement data with it?

Thanks,
Kevin