Skip to contentSkip to conversation

Sending some form of notifaction when sched_fifo throttling kicks in...

DISCUSSION torbenh 5 participants 7 messages Dec 23, 2010 – Dec 28, 2010

lore
Share & tools
lore

torbenh

author
#0 lore raw

hi...

when the rt_runtime budget is exceeded, the kernel silently stops scheduling RT tasks. there is no way to distinguish this from a task taking very long to complete.

it would be very nice, if the kernel would send some form of notifaction when it starts throttling things.

recording the timestamp of the last occurence of throttling in a /proc file would be sufficient, if there were no cgroups.

would it be possible to add a readonly property to the cpu subsystem ? there is already a printk_once, but thats pretty useless :)

from -rt kernel kernel/sched_rt.c:

----------------------------------------------------------------------------------
static int sched_rt_runtime_exceeded(struct rt_rq *rt_rq)

{
	u64 runtime = sched_rt_runtime(rt_rq);

	if (rt_rq->rt_throttled)
		return rt_rq_throttled(rt_rq);

	if (sched_rt_runtime(rt_rq) >= sched_rt_period(rt_rq))
		return 0;

	balance_runtime(rt_rq);
	runtime = sched_rt_runtime(rt_rq);
	if (runtime == RUNTIME_INF)
		return 0;

	if (rt_rq->rt_time > runtime) {
		rt_rq->rt_throttled = 1;
		printk_once(KERN_WARNING "sched: RT throttling activated\n");
+ // send some form of notification. 
		if (rt_rq_throttled(rt_rq)) {
			sched_rt_rq_dequeue(rt_rq);
			return 1;
		}
	}

	return 0;
}

-----------------------------------------------------------------------------------

Signature · 1 line
torben Hohn
#1 lore raw
Quoted 2 lines · torbenh

hi...

Hi,

Quoted 4 lines

when the rt_runtime budget is exceeded, the kernel silently stops scheduling RT tasks. there is no way to distinguish this from a task taking very long to complete.

Well, this depends on how you do the accounting in your (user-level) code.

Quoted 3 lines

it would be very nice, if the kernel would send some form of notifaction when it starts throttling things.

This might be tricky, if the meaning is signals or something to the throttled tasks, since you can have a (or more!) runqueue full of them... Are we signalling them all? Moreover, they'll get the notification only after resuming, and it's not guaranteed that this helps in finding out who is "responsible for" the throttling and... By the way...

Quoted 5 lines

recording the timestamp of the last occurence of throttling in a /proc file would be sufficient, if there were no cgroups.

would it be possible to add a readonly property to the cpu subsystem ?

... If you think you're fine with some /proc (and perhaps cpuacct, if cgroups are being used) readable, I can try to come up with something.

I think that the number of times that throttling fired and the last throttling instant could be exported this way without much issues.

Do others have some idea and/or comments about that? This is ABI/interface, and that really scares me! :-P

Thanks and Regards,
Dario

Signature · 5 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
----------------------------------------------------------------------
Dario Faggioli, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa  (Italy)

http://retis.sssup.it/people/faggioli -- [email protected]
#2 lore raw

?

Quoted 4 lines
Quoted 1 line

... If you think you're fine with some /proc (and perhaps cpuacct, if cgroups are being used) readable, I can try to come up with something.

There is no point in putting it in cpuacct since cpuacct can be used separately from cpu.

Quoted 5 lines

I think that the number of times that throttling fired and the last throttling instant could be exported this way without much issues.

Do others have some idea and/or comments about that? This is ABI/interface, and that really scares me! :-P

IIRC, your patchset had something like this for getting the statistics? Starting fromt hre, would schedstats be the right place?

Dhaval

#3 lore raw
Quoted 7 lines · Dhaval Giani
Quoted 3 lines

... If you think you're fine with some /proc (and perhaps cpuacct, if cgroups are being used) readable, I can try to come up with something.

There is no point in putting it in cpuacct since cpuacct can be used separately from cpu.

Which would mean that you'd need both for having such stat. Anyway, I'm fine with putting this in 'cpu' as well, just trying to find a consensus on what the right place is.

Quoted 6 lines
Quoted 2 lines

Do others have some idea and/or comments about that? This is ABI/interface, and that really scares me! :-P

IIRC, your patchset had something like this for getting the statistics? Starting fromt hre, would schedstats be the right place?

SCHED_DEADLINE patchset has both signaling capabilities and some statistic reporting, bat it's a different thing.

I think schedstat could be the right place for _this_ thing here, but since each cgroup could be throttled, we also need something which is per-cgroup... Don't you agree?

Regards,
Dario

Signature · 5 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
----------------------------------------------------------------------
Dario Faggioli, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa  (Italy)

http://retis.sssup.it/people/faggioli -- [email protected]
#4 lore raw
Quoted 18 lines · Dario Faggioli
Quoted 7 lines · Dhaval Giani
Quoted 3 lines

... If you think you're fine with some /proc (and perhaps cpuacct, if cgroups are being used) readable, I can try to come up with something.

There is no point in putting it in cpuacct since cpuacct can be used separately from cpu.

Which would mean that you'd need both for having such stat. Anyway, I'm fine with putting this in 'cpu' as well, just trying to find a consensus on what the right place is.

Quoted 6 lines
Quoted 2 lines

Do others have some idea and/or comments about that? This is ABI/interface, and that really scares me! :-P

IIRC, your patchset had something like this for getting the statistics? Starting fromt hre, would schedstats be the right place?

Quoted 7 lines

SCHED_DEADLINE patchset has both signaling capabilities and some statistic reporting, bat it's a different thing.

I think schedstat could be the right place for _this_ thing here, but since each cgroup could be throttled, we also need something which is per-cgroup... Don't you agree?

You definitely need something per cgroup, have you looked at the events framework in cgroups and its implementation in memcgroup?

Signature · 2 lines
	Three Cheers,
	Balbir
#5 lore raw
In reply to Dario Faggioli
Quoted 27 lines · Dario Faggioli
Quoted 2 lines · torbenh

hi...

Hi,

Quoted 4 lines

when the rt_runtime budget is exceeded, the kernel silently stops scheduling RT tasks. there is no way to distinguish this from a task taking very long to complete.

Well, this depends on how you do the accounting in your (user-level) code.

Quoted 3 lines

it would be very nice, if the kernel would send some form of notifaction when it starts throttling things.

This might be tricky, if the meaning is signals or something to the throttled tasks, since you can have a (or more!) runqueue full of them... Are we signalling them all? Moreover, they'll get the notification only after resuming, and it's not guaranteed that this helps in finding out who is "responsible for" the throttling and... By the way...

Quoted 5 lines

recording the timestamp of the last occurence of throttling in a /proc file would be sufficient, if there were no cgroups.

would it be possible to add a readonly property to the cpu subsystem ?

Quoted 8 lines

... If you think you're fine with some /proc (and perhaps cpuacct, if cgroups are being used) readable, I can try to come up with something.

I think that the number of times that throttling fired and the last throttling instant could be exported this way without much issues.

Do others have some idea and/or comments about that? This is ABI/interface, and that really scares me! :-P

You might want to note that CFS bandwidth control patches export some such stats via a new per-cgroup file cpu.stat. It exports 3 statistics:

nr_periods: Number of enforcement intervals that have elapsed.
nr_throttled: Number of times the group has been throttled/limited.
throttled_time: The total time duration (in nanoseconds) for which the group remained throttled.

Regards,
Bharata.

#6 lore raw
Quoted 6 lines · Bharata B Rao
Quoted 2 lines

Do others have some idea and/or comments about that? This is ABI/interface, and that really scares me! :-P

You might want to note that CFS bandwidth control patches export some such stats via a new per-cgroup file cpu.stat. It exports 3 statistics:

Yeah, I saw that. What can be also done is to extend schedstats and then apply it to cgroups as well... I think Dhaval told me he has some plans about that, am I wrong?

Quoted 5 lines

nr_periods: Number of enforcement intervals that have elapsed.
nr_throttled: Number of times the group has been throttled/limited.
throttled_time: The total time duration (in nanoseconds) for which the group remained throttled.

More than reasonable reporting, to me at least. :-)

Regards,
Dario

Signature · 5 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
----------------------------------------------------------------------
Dario Faggioli, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa  (Italy)

http://retis.sssup.it/people/faggioli -- [email protected]