RFC: documentation of the autogroup feature

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Regarding the previous paragraph...  My tests  indi‐ │
       │cate  that writing *any* value to the autogroup file │
       │causes the task group to get a lower priority.  This │

Quoted 6 lines · Michael Kerrisk (man-pages)

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Regarding the previous paragraph...  My tests  indi‐ │
       │cate  that writing *any* value to the autogroup file │
       │causes the task group to get a lower priority.  This │

Because autogroup didn't call the then meaningless scale_load()...

Autogroup nice level adjustment has been broken ever since load resolution was increased for 64bit kernels. Use scale_load() to scale group weight.

Signed-off-by Mike Galbraith <[email protected]>

Reported-by Michael Kerrisk <[email protected]>

Cc [email protected]

---
 kernel/sched/auto_group.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

kernel/sched/auto_group.c +3 -1

@@ -192,6 +192,7 @@ int proc_sched_autogroup_set_nice(struct
 {
 	static unsigned long next = INITIAL_JIFFIES;
 	struct autogroup *ag;
+	unsigned long shares;
 	int err;
 
 	if (nice < MIN_NICE || nice > MAX_NICE)

@@ -210,9 +211,10 @@ int proc_sched_autogroup_set_nice(struct
 
 	next = HZ / 10 + jiffies;
 	ag = autogroup_task_get(p);
+	shares = scale_load(sched_prio_to_weight[nice + 20]);
 
 	down_write(&ag->lock);
-	err = sched_group_set_shares(ag->tg, sched_prio_to_weight[nice + 20]);
+	err = sched_group_set_shares(ag->tg, shares);
 	if (!err)
 		ag->nice = nice;
 	up_write(&ag->lock);

Hello Mike,

Quoted 10 lines · Mike Galbraith

Quoted 6 lines · Michael Kerrisk (man-pages)

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Regarding the previous paragraph...  My tests  indi‐ │
       │cate  that writing *any* value to the autogroup file │
       │causes the task group to get a lower priority.  This │

Quoted 1 line

Because autogroup didn't call the then meaningless scale_load()...

So, does that mean that this buglet kicked in starting (only) in Linux 4.7 with commit 2159197d66770ec01f75c93fb11dc66df81fd45b?

Quoted 13 lines · Michael Kerrisk (man-pages)

Hello Mike,

Quoted 10 lines · Mike Galbraith

Quoted 6 lines · Michael Kerrisk (man-pages)

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Regarding the previous paragraph...  My tests  indi‐ │
       │cate  that writing *any* value to the autogroup file │
       │causes the task group to get a lower priority.  This │

Quoted 4 lines

Quoted 1 line

Because autogroup didn't call the then meaningless scale_load()...

So, does that mean that this buglet kicked in starting (only) in Linux 4.7 with commit 2159197d66770ec01f75c93fb11dc66df81fd45b?

Yeah, that gave it teeth.

Thanks for the confirmation. Are you aiming to see the fix merged for 4.9, or will this wait for 4.10?

Regarding the previous paragraph... My tests indicate that writing *any* value to the autogroup [nice priority level] file causes the task group to get a lower priority.

Applied and tested against 4.9-rc6 on an Intel u7 (4 cores). Test setup:

Terminal window 1: running 40 CPU burner jobs
Terminal window 2: running 40 CPU burner jobs
Terminal window 1: running 1 CPU burner job

Demonstrated that:
* Writing "0" to the autogroup file for TW1 now causes no change to the rate at which the process on the terminal consume CPU.
* Writing -20 to the autogroup file for TW1 caused those processes to get the lion's share of CPU while TW2 TW3 get a tiny amount.
* Writing -20 to the autogroup files for TW1 and TW3 allowed the process on TW3 to get as much CPU as it was getting as when the autogroup nice values for both terminals were 0.

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │The following is a little vague. Does it need to  be │
       │made more precise?                                   │
       └─────────────────────────────────────────────────────┘

The CFS scheduler employs an algorithm that distributes the CPU across task groups. As a result of this algorithm, the pro‐ cesses in task groups that contain multiple CPU-intensive pro‐ cesses are in effect disfavored by the scheduler.

A process's autogroup (task group) membership can be viewed via the file /proc/[pid]/autogroup:

$ cat /proc/1/autogroup
/autogroup-1 nice 0

This file can also be used to modify the CPU bandwidth allo‐ cated to a task group. This is done by writing a number in the "nice" range to the file to set the task group's nice value. The allowed range is from +19 (low priority) to -20 (high pri‐
ority). Note that all values in this range cause a task group
to be further disfavored by the scheduler, with -20 resulting in the scheduler mildy disfavoring the task group and +19 greatly disfavoring it.

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Regarding the previous paragraph...  My tests  indi‐ │
       │cate  that writing *any* value to the autogroup file │
       │causes the task group to get a lower priority.

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Is the following correct? Does the statement need to │
       │be  more  precise? (E.g., in precisely which circum‐ │
       │stances does the use of cgroups override autogroup?) │
       └─────────────────────────────────────────────────────┘
       The use of the cgroups(7) CPU controller overrides  the  effect

of autogrouping.

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │What  needs to be said about autogroup and real-time │
       │tasks?                                               │
       └─────────────────────────────────────────────────────┘

Quoted 10 lines · Michael Kerrisk (man-pages)

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │The following is a little vague. Does it need to  be │
       │made more precise?                                   │
       └─────────────────────────────────────────────────────┘

The CFS scheduler employs an algorithm that distributes the CPU across task groups. As a result of this algorithm, the pro‐ cesses in task groups that contain multiple CPU-intensive pro‐ cesses are in effect disfavored by the scheduler.

Mmmm, they're actually equalized (modulo smp fairness goop), but I see what you mean.

Quoted 7 lines

A process's autogroup (task group) membership can be viewed via the file /proc/[pid]/autogroup:

$ cat /proc/1/autogroup
/autogroup-1 nice 0

Quoted 8 lines

This file can also be used to modify the CPU bandwidth allo‐ cated to a task group. This is done by writing a number in the "nice" range to the file to set the task group's nice value. The allowed range is from +19 (low priority) to -20 (high pri‐
ority). Note that all values in this range cause a task group
to be further disfavored by the scheduler, with -20 resulting in the scheduler mildy disfavoring the task group and +19 greatly disfavoring it.

Group nice levels exactly work the same as task nice levels, ie negative nice increases share, positive nice decreases it relative to the default nice 0.

Quoted 6 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Regarding the previous paragraph...  My tests  indi‐ │
       │cate  that writing *any* value to the autogroup file │
       │causes the task group to get a lower priority.

(patchlet..

I'd prefer to whack the knob, but like the on/off switch, it may be in use, so I guess we're stuck with it)

Quoted 9 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Is the following correct? Does the statement need to │
       │be  more  precise? (E.g., in precisely which circum‐ │
       │stances does the use of cgroups override autogroup?) │
       └─────────────────────────────────────────────────────┘
       The use of the cgroups(7) CPU controller overrides  the  effect

of autogrouping.

Correct, autogroup defers to cgroups. Perhaps mention that moving a task back to the root task group will result in the autogroup again taking effect.

Quoted 6 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │What  needs to be said about autogroup and real-time │
       │tasks?                                               │
       └─────────────────────────────────────────────────────┘

That it does not group realtime tasks, they are auto-deflected to the root task group.

Hi Mike,

First off, I better say that I'm not at all intimate with the details of the scheduler, so bear with me...

Quoted 13 lines · Mike Galbraith

Quoted 10 lines · Michael Kerrisk (man-pages)

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │The following is a little vague. Does it need to  be │
       │made more precise?                                   │
       └─────────────────────────────────────────────────────┘

The CFS scheduler employs an algorithm that distributes the CPU across task groups. As a result of this algorithm, the pro‐ cesses in task groups that contain multiple CPU-intensive pro‐ cesses are in effect disfavored by the scheduler.

Quoted 2 lines

Mmmm, they're actually equalized (modulo smp fairness goop), but I see what you mean.

I couldn't quite grok that sentence. My problem is resolving "they". Do you mean: "the CPU scheduler equalizes the distribution of CPU cycles across task groups"?

Quoted 9 lines

Quoted 8 lines

       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Is the following correct? Does the statement need to │
       │be  more  precise? (E.g., in precisely which circum‐ │
       │stances does the use of cgroups override autogroup?) │
       └─────────────────────────────────────────────────────┘
       The use of the cgroups(7) CPU controller overrides  the  effect

of autogrouping.

Quoted 3 lines

Correct, autogroup defers to cgroups. Perhaps mention that moving a task back to the root task group will result in the autogroup again taking effect.

In what circumstances does a process get moved back to the root task group?

Actually, can you define for me what the root task group is, and why it exists? That may be worth some words in this man page.

Quoted 22 lines · Michael Kerrisk (man-pages)

Hi Mike,

First off, I better say that I'm not at all intimate with the details of the scheduler, so bear with me...

Quoted 15 lines · Mike Galbraith

Quoted 10 lines · Michael Kerrisk (man-pages)

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │The following is a little vague. Does it need to  be │
       │made more precise?                                   │
       └─────────────────────────────────────────────────────┘

The CFS scheduler employs an algorithm that distributes the CPU across task groups. As a result of this algorithm, the pro‐ cesses in task groups that contain multiple CPU-intensive pro‐ cesses are in effect disfavored by the scheduler.

Mmmm, they're actually equalized (modulo smp fairness goop), but I see what you mean.

Quoted 3 lines

I couldn't quite grok that sentence. My problem is resolving "they". Do you mean: "the CPU scheduler equalizes the distribution of CPU cycles across task groups"?

Sort of. "They" are scheduler entities, runqueue (group) or task. The scheduler equalizes entity vruntimes.

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

 
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │Is the following correct? Does the statement need to │
       │be  more  precise? (E.g., in precisely which circum‐ │
       │stances does the use of cgroups override autogroup?) │
       └─────────────────────────────────────────────────────┘
       The use of the cgroups(7) CPU controller overrides  the  effect

of autogrouping.

Quoted 6 lines

Quoted 3 lines

Correct, autogroup defers to cgroups. Perhaps mention that moving a task back to the root task group will result in the autogroup again taking effect.

In what circumstances does a process get moved back to the root task group?

Userspace actions, tool or human fingers.

Quoted 2 lines

Actually, can you define for me what the root task group is, and why it exists? That may be worth some words in this man page.

I don't think we need group scheduling details, there's plenty of documentation elsewhere for those who want theory.

Autogroup is for
those who don't want to have to care (which is also why it should have never grown nice knob).

Quoted 4 lines

Quoted 2 lines

In what circumstances does a process get moved back to the root task group?

Userspace actions, tool or human fingers.

Could you say a little more please. What Kernel-user-space APIs/system calls/etc. cause this to happen?

Quoted 5 lines

Quoted 2 lines

Actually, can you define for me what the root task group is, and why it exists? That may be worth some words in this man page.

I don't think we need group scheduling details, there's plenty of documentation elsewhere for those who want theory.

Well, you suggested above

Perhaps mention that moving a task back to the root task group will result in the autogroup again taking effect.

Suppose that there are two autogroups competing for the same CPU. The first group contains ten CPU-bound processes from a kernel build started with make -j10. The other contains a sin‐
gle CPU-bound process: a video player. The effect of auto‐
grouping is that the two groups will each receive half of the CPU cycles. That is, the video player will receive 50% of the CPU cycles, rather just 9% of the cycles, which would likely

lead to degraded video playback. Or to put things another way:
an autogroup that contains a large number of CPU-bound pro‐ cesses does not end up overwhelming the CPU at the expense of the other jobs on the system.

Hi,

Quoted 7 lines · Michael Kerrisk (man-pages)

Suppose that there are two autogroups competing for the same CPU. The first group contains ten CPU-bound processes from a kernel build started with make -j10. The other contains a sin‐
gle CPU-bound process: a video player. The effect of auto‐
grouping is that the two groups will each receive half of the CPU cycles. That is, the video player will receive 50% of the CPU cycles, rather just 9% of the cycles, which would likely

                            ^^^^
                            than ?

Regards
afzal

Quoted 5 lines

lead to degraded video playback. Or to put things another way:
an autogroup that contains a large number of CPU-bound pro‐ cesses does not end up overwhelming the CPU at the expense of the other jobs on the system.

Suppose that there are two autogroups competing for the same CPU. The first group contains ten CPU-bound processes from a kernel build started with make -j10. The other contains a sin‐
gle CPU-bound process: a video player. The effect of auto‐
grouping is that the two groups will each receive half of the CPU cycles. That is, the video player will receive 50% of the CPU cycles, rather just 9% of the cycles, which would likely lead to degraded video playback. Or to put things another way:
an autogroup that contains a large number of CPU-bound pro‐ cesses does not end up overwhelming the CPU at the expense of the other jobs on the system.

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │How do the nice value of  a  process  and  the  nice │
       │value of an autogroup interact? Which has priority?  │
       │                                                     │
       │It  *appears*  that the autogroup nice value is used │
       │for CPU distribution between task groups,  and  that │
       │the  process nice value has no effect there.  (I.e., │

       │suppose two  autogroups  each  contain  a  CPU-bound │
       │process,  with  one  process  having nice==0 and the │
       │other having nice==19.  It appears  that  they  each │
       │get  50%  of  the CPU.)  It appears that the process │
       │nice value has effect only with respect to  schedul‐ │
       │ing  relative to other processes in the *same* auto‐ │
       │group.  Is this correct?                             │
       └─────────────────────────────────────────────────────┘

Quoted 11 lines · Michael Kerrisk (man-pages)

Suppose that there are two autogroups competing for the same CPU. The first group contains ten CPU-bound processes from a kernel build started with make -j10. The other contains a sin‐
gle CPU-bound process: a video player. The effect of auto‐
grouping is that the two groups will each receive half of the CPU cycles. That is, the video player will receive 50% of the CPU cycles, rather just 9% of the cycles, which would likely lead to degraded video playback. Or to put things another way:
an autogroup that contains a large number of CPU-bound pro‐ cesses does not end up overwhelming the CPU at the expense of the other jobs on the system.

I'd say something more wishy-washy here, like cycles are distributed fairly across groups and leave it at that,

as your detailed example is
incorrect due to SMP fairness

(which I don't like much because [very
unlikely] worst case scenario renders a box sized group incapable of utilizing more that a single CPU total). For example, if a group of NR_CPUS size competes with a singleton, load balancing will try to give the singleton a full CPU of its very own. If groups intersect for whatever reason on say my quad lappy, distribution is 80/20 in favor of the singleton.

Quoted 9 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │How do the nice value of  a  process  and  the  nice │
       │value of an autogroup interact? Which has priority?  │
       │                                                     │
       │It  *appears*  that the autogroup nice value is used │
       │for CPU distribution between task groups,  and  that │
       │the  process nice value has no effect there.  (I.e., │

Quoted 8 lines

       │suppose two  autogroups  each  contain  a  CPU-bound │
       │process,  with  one  process  having nice==0 and the │
       │other having nice==19.  It appears  that  they  each │
       │get  50%  of  the CPU.)  It appears that the process │
       │nice value has effect only with respect to  schedul‐ │
       │ing  relative to other processes in the *same* auto‐ │
       │group.  Is this correct?                             │
       └─────────────────────────────────────────────────────┘

Yup, entity nice level affects distribution among peer entities.

Hi Mike,

Quoted 18 lines · Mike Galbraith

Quoted 9 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │How do the nice value of  a  process  and  the  nice │
       │value of an autogroup interact? Which has priority?  │
       │                                                     │
       │It  *appears*  that the autogroup nice value is used │
       │for CPU distribution between task groups,  and  that │
       │the  process nice value has no effect there.  (I.e., │

Quoted 8 lines

       │suppose two  autogroups  each  contain  a  CPU-bound │
       │process,  with  one  process  having nice==0 and the │
       │other having nice==19.  It appears  that  they  each │
       │get  50%  of  the CPU.)  It appears that the process │
       │nice value has effect only with respect to  schedul‐ │
       │ing  relative to other processes in the *same* auto‐ │
       │group.  Is this correct?                             │
       └─────────────────────────────────────────────────────┘

Quoted 1 line

Yup, entity nice level affects distribution among peer entities.

Huh! I only just learned about this via my experiments while investigating autogroups.

How long have things been like this? Always? (I don't think so.) Since the arrival of CFS? Since the arrival of autogrouping? (I'm guessing not.) Since some other point? (When?)

It seems to me that this renders the traditional process nice pretty much useless. (I bet I'm not the only one who'd be surprised by the current behavior.)

Quoted 18 lines

Quoted 9 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │How do the nice value of  a  process  and  the  nice │
       │value of an autogroup interact? Which has priority?  │
       │                                                     │
       │It  *appears*  that the autogroup nice value is used │
       │for CPU distribution between task groups,  and  that │
       │the  process nice value has no effect there.  (I.e., │

Quoted 8 lines

       │suppose two  autogroups  each  contain  a  CPU-bound │
       │process,  with  one  process  having nice==0 and the │
       │other having nice==19.  It appears  that  they  each │
       │get  50%  of  the CPU.)  It appears that the process │
       │nice value has effect only with respect to  schedul‐ │
       │ing  relative to other processes in the *same* auto‐ │
       │group.  Is this correct?                             │
       └─────────────────────────────────────────────────────┘

Quoted 1 line

Yup, entity nice level affects distribution among peer entities.

Huh! I only just learned about this via my experiments while investigating autogroups.

How long have things been like this? Always? (I don't think so.) Since the arrival of CFS? Since the arrival of autogrouping? (I'm guessing not.) Since some other point? (When?)

It seems to me that this renders the traditional process nice pretty much useless. (I bet I'm not the only one who'd be surprised by the current behavior.)

Quoted 23 lines · Michael Kerrisk (man-pages)

Quoted 18 lines

Quoted 9 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │How do the nice value of  a  process  and  the  nice │
       │value of an autogroup interact? Which has priority?  │
       │                                                     │
       │It  *appears*  that the autogroup nice value is used │
       │for CPU distribution between task groups,  and  that │
       │the  process nice value has no effect there.  (I.e., │

Quoted 8 lines

       │suppose two  autogroups  each  contain  a  CPU-bound │
       │process,  with  one  process  having nice==0 and the │
       │other having nice==19.  It appears  that  they  each │
       │get  50%  of  the CPU.)  It appears that the process │
       │nice value has effect only with respect to  schedul‐ │
       │ing  relative to other processes in the *same* auto‐ │
       │group.  Is this correct?                             │
       └─────────────────────────────────────────────────────┘

Quoted 1 line

Yup, entity nice level affects distribution among peer entities.

Huh! I only just learned about this via my experiments while investigating autogroups.

Quoted 4 lines

How long have things been like this? Always? (I don't think so.) Since the arrival of CFS? Since the arrival of autogrouping? (I'm guessing not.) Since some other point? (When?)

Always. Before CFS there just were no non-peers :)

Quoted 3 lines

It seems to me that this renders the traditional process nice pretty much useless. (I bet I'm not the only one who'd be surprised by the current behavior.)

Yup, group scheduling is not a single edged sword, those don't exist.
Box wide nice loss is not the only thing that can bite you, fairness,
whether group or task oriented cuts both ways.

On 11/25/2016 04:51 PM, Mike Galbraith wrote:
Well that's one way of looking at it. So, the change that I'm talking about came in 2.6.32 with CFS then?

Quoted 3 lines · Michael Kerrisk (man-pages)

On 11/25/2016 04:51 PM, Mike Galbraith wrote:
Well that's one way of looking at it. So, the change that I'm talking about came in 2.6.32 with CFS then?

cfs-cgroup landed later I think, and it was fairly wobbly in the first few release (as per usual I'd say for major features).

Quoted 5 lines · Peter Zijlstra

Quoted 3 lines · Michael Kerrisk (man-pages)

On 11/25/2016 04:51 PM, Mike Galbraith wrote:
Well that's one way of looking at it. So, the change that I'm talking about came in 2.6.32 with CFS then?

Quoted 2 lines

cfs-cgroup landed later I think, and it was fairly wobbly in the first few release (as per usual I'd say for major features).

So I've been searching git logs and elsewhere, but didn't yet find a likely commit(s). Any clues what I should be looking for. I'd like this info, because while documenting the changes, I'd also like to document when they occurred.

So, part of what I was struggling with was what you meant by cfs-cgroup. Do you mean the CFS bandwidth control features added in Linux 3.2?

Quoted 2 lines · Michael Kerrisk (man-pages)

So, part of what I was struggling with was what you meant by cfs-cgroup. Do you mean the CFS bandwidth control features added in Linux 3.2?

Nope, /me digs around for a bit... around here I suppose:

68318b8e0b61 ("Hook up group scheduler with control groups")

68318b8e0b61 v2.6.24-rc1~151

But I really have no idea what that looked like.

In any case, for the case of autogroup, the behaviour has always been, autogroups came quite late.

Quoted 3 lines

In any case, for the case of autogroup, the behaviour has always been, autogroups came quite late.

This ("the behavior has always been") isn't quite true. Yes, group scheduling has been around since Linux 2.6.24, but in terms of the semantics of the thread nice value, there was no visible change then, *unless* explicit action was taken to create cgroups.

The arrival of autogroups in Linux 2.6.38 was different. With this feature enabled (which is the default), task

groups were implicitly created *without the user needing to do anything*. Thus, [two terminal windows] == [two task groups] and in those two terminal windows, nice(1) on a CPU-bound command in one terminal did nothing in terms of improving CPU access for a CPU-bound tasks running on the other terminal window.

Put more succinctly: in Linux 2.6.38, autogrouping broke nice(1) for many use cases.

Once I came to that simple summary it was easy to find multiple reports of problems from users:

http://serverfault.com/questions/405092/nice-level-not-working-on-linux
http://superuser.com/questions/805599/nice-has-no-effect-in-linux-unless-the-same-shell-is-used
https://www.reddit.com/r/linux/comments/1c4jew/nice_has_no_effect/
http://stackoverflow.com/questions/10342470/process-niceness-priority-setting-has-no-effect-on-linux

Someone else quickly pointed out to me another such report:

https://bbs.archlinux.org/viewtopic.php?id=149553

And when I quickly surveyed a few more or less savvy Linux users in one room, most understood what nice does, but none of them knew about the behavior change wrought by autogroup.

I haven't looked at all of the mails in the old threads that discussed the implementation of this feature, but so far none of those that I saw mentioned this behavior change. It's unfortunate that it never even got documented.

Quoted 9 lines · Michael Kerrisk (man-pages)

Quoted 3 lines

In any case, for the case of autogroup, the behaviour has always been, autogroups came quite late.

This ("the behavior has always been") isn't quite true. Yes, group scheduling has been around since Linux 2.6.24, but in terms of the semantics of the thread nice value, there was no visible change then, *unless* explicit action was taken to create cgroups.

Quoted 2 lines

The arrival of autogroups in Linux 2.6.38 was different. With this feature enabled (which is the default), task

I don't think the SCHED_AUTOGROUP symbol is default y, most distros might have default enabled it, but that's not something I can help.

Quoted 18 lines

groups were implicitly created *without the user needing to do anything*. Thus, [two terminal windows] == [two task groups] and in those two terminal windows, nice(1) on a CPU-bound command in one terminal did nothing in terms of improving CPU access for a CPU-bound tasks running on the other terminal window.

Put more succinctly: in Linux 2.6.38, autogrouping broke nice(1) for many use cases.

Once I came to that simple summary it was easy to find multiple reports of problems from users:

http://serverfault.com/questions/405092/nice-level-not-working-on-linux
http://superuser.com/questions/805599/nice-has-no-effect-in-linux-unless-the-same-shell-is-used
https://www.reddit.com/r/linux/comments/1c4jew/nice_has_no_effect/
http://stackoverflow.com/questions/10342470/process-niceness-priority-setting-has-no-effect-on-linux

Quoted 3 lines

Someone else quickly pointed out to me another such report:

https://bbs.archlinux.org/viewtopic.php?id=149553

Well, none of that ever got back to me, so again, nothing I could do about that.

Quoted 8 lines

And when I quickly surveyed a few more or less savvy Linux users in one room, most understood what nice does, but none of them knew about the behavior change wrought by autogroup.

I haven't looked at all of the mails in the old threads that discussed the implementation of this feature, but so far none of those that I saw mentioned this behavior change. It's unfortunate that it never even got documented.

Well, when we added the feature people (most notable Linus) understood what cgroups did. So no surprises for any of us.

Quoted 18 lines

Quoted 9 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │How do the nice value of  a  process  and  the  nice │
       │value of an autogroup interact? Which has priority?  │
       │                                                     │
       │It  *appears*  that the autogroup nice value is used │
       │for CPU distribution between task groups,  and  that │
       │the  process nice value has no effect there.  (I.e., │

Quoted 8 lines

       │suppose two  autogroups  each  contain  a  CPU-bound │
       │process,  with  one  process  having nice==0 and the │
       │other having nice==19.  It appears  that  they  each │
       │get  50%  of  the CPU.)  It appears that the process │
       │nice value has effect only with respect to  schedul‐ │
       │ing  relative to other processes in the *same* auto‐ │
       │group.  Is this correct?                             │
       └─────────────────────────────────────────────────────┘

Quoted 1 line

Yup, entity nice level affects distribution among peer entities.

Huh! I only just learned about this via my experiments while investigating autogroups.

How long have things been like this? Always? (I don't think so.) Since the arrival of CFS? Since the arrival of autogrouping? (I'm guessing not.) Since some other point? (When?)

It seems to me that this renders the traditional process nice pretty much useless. (I bet I'm not the only one who'd be surprised by the current behavior.)

That is; the whole thing
becomes, where l denotes the level in the hierarchy and i an entity on that level:

l w_g,i
dt_l,i = dt \Prod ----------

                g=0  \Sum w_g,j

Or more concretely, for E:

                      w_E
  dt_1,E = dt_0,C -----------
                  (w_E + w_F)

                        w_C               w_E
         = dt ----------------------- -----------
              (w_A + w_B + w_C + w_D) (w_E + w_F)

Quoted 23 lines · Michael Kerrisk (man-pages)

Quoted 18 lines

Quoted 9 lines

       ┌─────────────────────────────────────────────────────┐
       │FIXME                                                │
       ├─────────────────────────────────────────────────────┤
       │How do the nice value of  a  process  and  the  nice │
       │value of an autogroup interact? Which has priority?  │
       │                                                     │
       │It  *appears*  that the autogroup nice value is used │
       │for CPU distribution between task groups,  and  that │
       │the  process nice value has no effect there.  (I.e., │

Quoted 8 lines

       │suppose two  autogroups  each  contain  a  CPU-bound │
       │process,  with  one  process  having nice==0 and the │
       │other having nice==19.  It appears  that  they  each │
       │get  50%  of  the CPU.)  It appears that the process │
       │nice value has effect only with respect to  schedul‐ │
       │ing  relative to other processes in the *same* auto‐ │
       │group.  Is this correct?                             │
       └─────────────────────────────────────────────────────┘

Quoted 1 line

Yup, entity nice level affects distribution among peer entities.

Huh! I only just learned about this via my experiments while investigating autogroups.

Quoted 4 lines

How long have things been like this? Always? (I don't think so.) Since the arrival of CFS? Since the arrival of autogrouping? (I'm guessing not.) Since some other point? (When?)

Ever since cfs-cgroup,

this is a fundamental design point of cgroups,
and has therefore always been the case for autogroups (as that is nothing more than an application of the cgroup code).

Quoted 3 lines

It seems to me that this renders the traditional process nice pretty much useless. (I bet I'm not the only one who'd be surprised by the current behavior.)

Its really rather fundamental to how the whole hierarchical things works.

CFS is a weighted fair queueing scheduler; this means each entity receives:

               w_i
  dt_i = dt --------
	    \Sum w_j


		CPU
	  ______/ \______
	 /    |     |	 \
      A     B     C     D

So if each entity {A,B,C,D} has equal weight, then they will receive equal time. Explicitly, for C you get:

                      w_C
  dt_C = dt -----------------------
            (w_A + w_B + w_C + w_D)

Extending this to a hierarchy, we get:

		CPU
	  ______/ \______
	 /    |     |	 \
      A     B     C     D
	           / \
		  E   F

Where C becomes a 'server' for entities {E,F}. The weight of C does not depend on its child entities. This way the time of {E,F} becomes a straight product of their ratio with C. That is; the whole thing becomes, where l denotes the level in the hierarchy and i an entity on that level:

l w_g,i
dt_l,i = dt \Prod ----------

                g=0  \Sum w_g,j

Or more concretely, for E:

                      w_E
  dt_1,E = dt_0,C -----------
                  (w_E + w_F)

                        w_C               w_E
         = dt ----------------------- -----------
              (w_A + w_B + w_C + w_D) (w_E + w_F)

And this 'trivially' extends to SMP, with the tricky bit being that the sums over all entities end up being machine wide, instead of per CPU, which is a real and royal pain for performance.

Note that this property, where the weight of the server entity is independent from its child entities is a desired feature. Without that it would be impossible to control the relative weights of groups, and that is the sole parameter of the WFQ model.

It is also why Linus so likes autogroups, each session competes equally amongst one another.

Okay -- you're really quite the ASCII artist. And somehow, I think you needed to compose the mail in LaTeX. But thanks for the detail. It's helpful, for me at least.

Quoted 7 lines

Note that this property, where the weight of the server entity is independent from its child entities is a desired feature. Without that it would be impossible to control the relative weights of groups, and that is the sole parameter of the WFQ model.

It is also why Linus so likes autogroups, each session competes equally amongst one another.

I get it. But, the behavior changes for the process nice value are undocumented, and they should be documented. I understand what the behavior change was. But not yet when.

I don't think we need group scheduling details, there's plenty of documentation elsewhere for those who want theory.

Quoted 2 lines

I don't think we need group scheduling details, there's plenty of documentation elsewhere for those who want theory.

Actually, which documentation were you referring to here?

Quoted 4 lines · Michael Kerrisk (man-pages)

Quoted 2 lines

I don't think we need group scheduling details, there's plenty of documentation elsewhere for those who want theory.

Actually, which documentation were you referring to here?

Documentation/scheduler/*

Quoted 1 line · Michael Kerrisk (man-pages)

Hi Mike,

Quoted 2 lines

Actually, can you define for me what the root task group is, and why it exists? That may be worth some words in this man page.

I don't think we need group scheduling details, there's plenty of documentation elsewhere for those who want theory. Autogroup is for those who don't want to have to care (which is also why it should have never grown nice knob).

Here's my attempt to define the root task group:

* If autogrouping is disabled, then all processes in the root CPU cgroup form a scheduling group (sometimes called the "root task group").

Can you improve on this?

Quoted 7 lines · Michael Kerrisk (man-pages)

Here's my attempt to define the root task group:

* If autogrouping is disabled, then all processes in the root CPU cgroup form a scheduling group (sometimes called the "root task group").

Can you improve on this?

A task group is a set of percpu runqueues.

The root task group is the
top level set in a hierarchy of such sets when group scheduling is enabled, or the only set when group scheduling is not enabled. The autogroup hierarchy has a depth of one, ie all autogroups are peers who's common parent is the root task group.

Let's try and go further. How's this:

When scheduling non-real-time processes (i.e., those scheduled under the SCHED_OTHER, SCHED_BATCH, and SCHED_IDLE policies), the CFS scheduler employs a technique known as "group scheduling", if the kernel was configured with the CONFIG_FAIR_GROUP_SCHED option (which is typical).

Under group scheduling, threads are scheduled in "task groups". Task groups have a hierarchical relationship, rooted under the initial task group on the system, known as the "root task group". Task groups are formed in the following circumstances:

* All of the threads in a CPU cgroup form a task group. The par‐ ent of this task group is the task group of the corresponding parent cgroup.

* If autogrouping is enabled, then all of the threads that are (implicitly) placed in an autogroup (i.e., the same session, as created by setsid(2)) form a task group. Each new autogroup is thus a separate task group. The root task group is the parent of all such autogroups.

* If autogrouping is enabled, then the root task group consists of all processes in the root CPU cgroup that were not otherwise implicitly placed into a new autogroup.

* If autogrouping is disabled, then the root task group consists of all processes in the root CPU cgroup.

* If group scheduling was disabled (i.e., the kernel was config‐ ured without CONFIG_FAIR_GROUP_SCHED), then all of the pro‐ cesses on the system are notionally placed in a single task group.