Skip to contentSkip to conversation

fix load average accounting

PATCH In review Thomas Gleixner 10 participants 37 messages Jul 17, 2009 – Jul 18, 2009 3 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover fix load average accounting

The following series fixes two problems in the load average accounting:

1) a thinko in the new load average code vs. CPU hotplug

2) the complete accounting wreckage caused by a braindamaged attempt to fix the freezer accounting of nr_uninterruptible.

Thanks,

tglx

#1 lore raw

PATCH 1/2 sched: fix load average accounting vs. cpu hotplug

The new load average code clears rq->calc_load_active on CPU_ONLINE. That's wrong as the new onlined CPU might have got a scheduler tick already and accounted the delta to the stale value of the time we offlined the CPU.

Clear the value when we cleanup the dead CPU instead.

Also move the update of the calc_load_update time for the newly online CPU to CPU_UP_PREPARE to avoid that the CPU plays catch up with the stale update time value.

---
 kernel/sched.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched.c +2 -2
Index: linux-2.6/kernel/sched.c
===================================================================
@@ -7289,6 +7289,7 @@ static void migrate_dead_tasks(unsigned 
 static void calc_global_load_remove(struct rq *rq)
 {
 	atomic_long_sub(rq->calc_load_active, &calc_load_tasks);
+	rq->calc_load_active = 0;
 }
 #endif /* CONFIG_HOTPLUG_CPU */
 
@@ -7515,6 +7516,7 @@ migration_call(struct notifier_block *nf
 		task_rq_unlock(rq, &flags);
 		get_task_struct(p);
 		cpu_rq(cpu)->migration_thread = p;
+		rq->calc_load_update = calc_load_update;
 		break;
 
 	case CPU_ONLINE:
@@ -7525,8 +7527,6 @@ migration_call(struct notifier_block *nf
 		/* Update our root-domain */
 		rq = cpu_rq(cpu);
 		spin_lock_irqsave(&rq->lock, flags);
-		rq->calc_load_update = calc_load_update;
-		rq->calc_load_active = 0;
 		if (rq->rd) {
 			BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
 
#2 lore raw

PATCH 2/2 sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner

commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Cc Nathan Lynch <[email protected]>
Cc Andrew Morton <akpm-de/[email protected]>
Cc Nigel Cunningham <nigel-jrkICu/[email protected]>
Cc containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
Cc linux-pm-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
Cc Matt Helsley <matthltc-r/[email protected]>
---
 include/linux/sched.h |    3 ++-
 kernel/freezer.c      |    7 +++++++
 2 files changed, 9 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -1
Index: linux-2.6/include/linux/sched.h
===================================================================
@@ -209,7 +209,7 @@ extern unsigned long long time_sync_thre
 			((task->state & (__TASK_STOPPED | __TASK_TRACED)) != 0)
 #define task_contributes_to_load(task)	\
 				((task->state & TASK_UNINTERRUPTIBLE) != 0 && \
-				 (task->flags & PF_FROZEN) == 0)
+				 (task->flags & PF_FREEZING) == 0)
 
 #define __set_task_state(tsk, state_value)		\
 	do { (tsk)->state = (state_value); } while (0)
@@ -1680,6 +1680,7 @@ extern cputime_t task_gtime(struct task_
 #define PF_MEMALLOC	0x00000800	/* Allocating memory */
 #define PF_FLUSHER	0x00001000	/* responsible for disk writeback */
 #define PF_USED_MATH	0x00002000	/* if unset the fpu must be initialized before use */
+#define PF_FREEZING	0x00004000	/* freeze in progress. do not account to load */
 #define PF_NOFREEZE	0x00008000	/* this thread should not be frozen */
 #define PF_FROZEN	0x00010000	/* frozen for system suspend */
 #define PF_FSTRANS	0x00020000	/* inside a filesystem transaction */
kernel/freezer.c +7 -0
Index: linux-2.6/kernel/freezer.c
===================================================================
@@ -44,12 +44,19 @@ void refrigerator(void)
 	recalc_sigpending(); /* We sent fake signal, clean it up */
 	spin_unlock_irq(&current->sighand->siglock);
 
+	/* prevent accounting of that task to load */
+	current->flags |= PF_FREEZING;
+
 	for (;;) {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		if (!frozen(current))
 			break;
 		schedule();
 	}
+
+	/* Remove the accounting blocker */
+	current->flags &= ~PF_FREEZING;
+
 	pr_debug("%s left refrigerator\n", current->comm);
 	__set_current_state(save);
 }
#3 lore raw

PATCH 2/2 sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner

commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

---
 include/linux/sched.h |    3 ++-
 kernel/freezer.c      |    7 +++++++
 2 files changed, 9 insertions(+), 1 deletion(-)
include/linux/sched.h +2 -1
Index: linux-2.6/include/linux/sched.h
===================================================================
@@ -209,7 +209,7 @@ extern unsigned long long time_sync_thre
 			((task->state & (__TASK_STOPPED | __TASK_TRACED)) != 0)
 #define task_contributes_to_load(task)	\
 				((task->state & TASK_UNINTERRUPTIBLE) != 0 && \
-				 (task->flags & PF_FROZEN) == 0)
+				 (task->flags & PF_FREEZING) == 0)
 
 #define __set_task_state(tsk, state_value)		\
 	do { (tsk)->state = (state_value); } while (0)
@@ -1680,6 +1680,7 @@ extern cputime_t task_gtime(struct task_
 #define PF_MEMALLOC	0x00000800	/* Allocating memory */
 #define PF_FLUSHER	0x00001000	/* responsible for disk writeback */
 #define PF_USED_MATH	0x00002000	/* if unset the fpu must be initialized before use */
+#define PF_FREEZING	0x00004000	/* freeze in progress. do not account to load */
 #define PF_NOFREEZE	0x00008000	/* this thread should not be frozen */
 #define PF_FROZEN	0x00010000	/* frozen for system suspend */
 #define PF_FSTRANS	0x00020000	/* inside a filesystem transaction */
kernel/freezer.c +7 -0
Index: linux-2.6/kernel/freezer.c
===================================================================
@@ -44,12 +44,19 @@ void refrigerator(void)
 	recalc_sigpending(); /* We sent fake signal, clean it up */
 	spin_unlock_irq(&current->sighand->siglock);
 
+	/* prevent accounting of that task to load */
+	current->flags |= PF_FREEZING;
+
 	for (;;) {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		if (!frozen(current))
 			break;
 		schedule();
 	}
+
+	/* Remove the accounting blocker */
+	current->flags &= ~PF_FREEZING;
+
 	pr_debug("%s left refrigerator\n", current->comm);
 	__set_current_state(save);
 }

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

The changelog reads:

---
commit e3c8ca8336707062f3f7cb1cd7e6b3c753baccdd

Author: Nathan Lynch <[email protected]>
Date:   Wed Apr 8 19:45:12 2009 -0500

sched: do not count frozen tasks toward load

Freezing tasks via the cgroup freezer causes the load average to climb because the freezer's current implementation puts frozen tasks in uninterruptible sleep (D state).

Some applications which perform job-scheduling functions consult the load average when making decisions. If a cgroup is frozen, the load average does not provide a useful measure of the system's utilization to such applications. This is especially inconvenient if the job scheduler employs the cgroup freezer as a mechanism for preempting low priority jobs. Contrast this with using SIGSTOP for the same purpose:
the stopped tasks do not count toward system load.

Change task_contributes_to_load() to return false if the task is frozen. This results in /proc/loadavg behavior that better meets users' expectations.
---

It appears to me that a frozen cgroup is a transient state. Either you would typically do something like:

freeze -> {snapshot, migrate} -> {thaw, destroy}

Therefore a short increase in load doesn't seem like too big a problem, its going to be gone soon anyway.

Hmm?

#6 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to a message not in the archive
Quoted 11 lines · Thomas Gleixner

commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

I'm sorry, it's not clear to me based on the description what problem is being fixed. As far as I can see PF_FROZEN is almost exactly the same as PF_FREEZING. When a task is being frozen TIF_FREEZE is set. Then the task enters the refrigerator, sets PF_FROZEN, and schedule()s until PF_FROZEN is no longer set. The original code with extra comments:

static inline void frozen_process(void)

{
        if (!unlikely(current->flags & PF_NOFREEZE)) {
                current->flags |= PF_FROZEN;
                wmb();
        }
        clear_freeze_flag(current);
}

/* Refrigerator is place where frozen processes are stored :-). */ void refrigerator(void)

{
        /* Hmm, should we be allowed to suspend when there are realtime
           processes around? */
        long save;

        task_lock(current);
        if (freezing(current)) {
		/* prevent accounting of that task to load */
                frozen_process();  /* <-- sets PF_FROZEN */
                task_unlock(current);
        } else {
                task_unlock(current);
                return;
        }
        save = current->state;
        pr_debug("%s entered refrigerator\n", current->comm);

        spin_lock_irq(&current->sighand->siglock);
        recalc_sigpending(); /* We sent fake signal, clean it up */
        spin_unlock_irq(&current->sighand->siglock);

	/* you set PF_FREEZING here */
        for (;;) {
                set_current_state(TASK_UNINTERRUPTIBLE);
                if (!frozen(current))  /* <-- checks PF_FROZEN */
                        break;
                schedule();
        }
	/* you clear PF_FREEZING here */
        pr_debug("%s left refrigerator\n", current->comm);
        __set_current_state(save);
}
Quoted 47 lines
Cc Nathan Lynch <[email protected]>
Cc Andrew Morton <akpm-de/[email protected]>
Cc Nigel Cunningham <nigel-jrkICu/[email protected]>
Cc containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
Cc linux-pm-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
Cc Matt Helsley <matthltc-r/[email protected]>
---
 include/linux/sched.h |    3 ++-
 kernel/freezer.c      |    7 +++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

Index: linux-2.6/include/linux/sched.h
===================================================================

include/linux/sched.h +2 -1
@@ -209,7 +209,7 @@ extern unsigned long long time_sync_thre
 			((task->state & (__TASK_STOPPED | __TASK_TRACED)) != 0)
 #define task_contributes_to_load(task)	\
 				((task->state & TASK_UNINTERRUPTIBLE) != 0 && \
-				 (task->flags & PF_FROZEN) == 0)
+				 (task->flags & PF_FREEZING) == 0)
 
 #define __set_task_state(tsk, state_value)		\
 	do { (tsk)->state = (state_value); } while (0)
@@ -1680,6 +1680,7 @@ extern cputime_t task_gtime(struct task_
 #define PF_MEMALLOC	0x00000800	/* Allocating memory */
 #define PF_FLUSHER	0x00001000	/* responsible for disk writeback */
 #define PF_USED_MATH	0x00002000	/* if unset the fpu must be initialized before use */
+#define PF_FREEZING	0x00004000	/* freeze in progress. do not account to load */
 #define PF_NOFREEZE	0x00008000	/* this thread should not be frozen */
 #define PF_FROZEN	0x00010000	/* frozen for system suspend */
 #define PF_FSTRANS	0x00020000	/* inside a filesystem transaction */

Index: linux-2.6/kernel/freezer.c
===================================================================

kernel/freezer.c +3 -0
@@ -44,12 +44,19 @@ void refrigerator(void)
 	recalc_sigpending(); /* We sent fake signal, clean it up */
 	spin_unlock_irq(&current->sighand->siglock);
 
+	/* prevent accounting of that task to load */
+	current->flags |= PF_FREEZING;
+

kernel/freezer.c void refrigerator(void) · line 59

Quoted 10 lines
kernel/freezer.c +4 -0
 	for (;;) {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		if (!frozen(current))
 			break;
 		schedule();
 	}
+
+	/* Remove the accounting blocker */
+	current->flags &= ~PF_FREEZING;
+

Hence PF_FREEZING covers slightly less time than PF_FROZEN but otherwise does not change the way nr_uninterruptible is incremented or decremented (in (de)activate_task()).

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Cheers,
-Matt Helsley

#7 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to a message not in the archive
Quoted 11 lines · Thomas Gleixner

commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Thanks for the fix, and sorry for the regression. I'll test this early next week but I don't anticipate any problems for the use case the original patch was addressing.

maintainer
#8 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner
Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

The changelog reads:

---
commit e3c8ca8336707062f3f7cb1cd7e6b3c753baccdd

Author: Nathan Lynch <[email protected]>
Date:   Wed Apr 8 19:45:12 2009 -0500

sched: do not count frozen tasks toward load

Freezing tasks via the cgroup freezer causes the load average to climb because the freezer's current implementation puts frozen tasks in uninterruptible sleep (D state).

Some applications which perform job-scheduling functions consult the load average when making decisions. If a cgroup is frozen, the load average does not provide a useful measure of the system's utilization to such applications. This is especially inconvenient if the job scheduler employs the cgroup freezer as a mechanism for preempting low priority jobs. Contrast this with using SIGSTOP for the same purpose:
the stopped tasks do not count toward system load.

Change task_contributes_to_load() to return false if the task is frozen. This results in /proc/loadavg behavior that better meets users' expectations.
---

It appears to me that a frozen cgroup is a transient state. Either you would typically do something like:

freeze -> {snapshot, migrate} -> {thaw, destroy}

Therefore a short increase in load doesn't seem like too big a problem, its going to be gone soon anyway.

Hmm?

maintainer
#9 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner
Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

The changelog reads:

---
commit e3c8ca8336707062f3f7cb1cd7e6b3c753baccdd

Author: Nathan Lynch <[email protected]>
Date:   Wed Apr 8 19:45:12 2009 -0500

sched: do not count frozen tasks toward load

Freezing tasks via the cgroup freezer causes the load average to climb because the freezer's current implementation puts frozen tasks in uninterruptible sleep (D state).

Some applications which perform job-scheduling functions consult the load average when making decisions. If a cgroup is frozen, the load average does not provide a useful measure of the system's utilization to such applications. This is especially inconvenient if the job scheduler employs the cgroup freezer as a mechanism for preempting low priority jobs. Contrast this with using SIGSTOP for the same purpose:
the stopped tasks do not count toward system load.

Change task_contributes_to_load() to return false if the task is frozen. This results in /proc/loadavg behavior that better meets users' expectations.
---

It appears to me that a frozen cgroup is a transient state. Either you would typically do something like:

freeze -> {snapshot, migrate} -> {thaw, destroy}

Therefore a short increase in load doesn't seem like too big a problem, its going to be gone soon anyway.

Hmm?

#10 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 6 lines
Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

Fine with me, but it seems that the cgroup folks have some luser space stuff looking at proc/loadavg which goes berserk when loadavg increases rapidly due to freezing. OTOH that stuff seems to be oblivious to the fact that the commit in question brings loadavg irreversibly to 0 when you do enough freeze/thaw cycles.

So either we revert or apply the fix, which keeps the accounting straight and solves the freezer loadavg thing. No strong opinion on that.

Thanks,

tglx

#11 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 6 lines
Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

Fine with me, but it seems that the cgroup folks have some luser space stuff looking at proc/loadavg which goes berserk when loadavg increases rapidly due to freezing. OTOH that stuff seems to be oblivious to the fact that the commit in question brings loadavg irreversibly to 0 when you do enough freeze/thaw cycles.

So either we revert or apply the fix, which keeps the accounting straight and solves the freezer loadavg thing. No strong opinion on that.

Thanks,

tglx

#12 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 6 lines
Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

Fine with me, but it seems that the cgroup folks have some luser space stuff looking at proc/loadavg which goes berserk when loadavg increases rapidly due to freezing. OTOH that stuff seems to be oblivious to the fact that the commit in question brings loadavg irreversibly to 0 when you do enough freeze/thaw cycles.

So either we revert or apply the fix, which keeps the accounting straight and solves the freezer loadavg thing. No strong opinion on that.

Thanks,

tglx

#13 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Hi.

Quoted 15 lines · Thomas Gleixner
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Quoted 8 lines
Quoted 2 lines

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

Fine with me, but it seems that the cgroup folks have some luser space stuff looking at proc/loadavg which goes berserk when loadavg increases rapidly due to freezing. OTOH that stuff seems to be oblivious to the fact that the commit in question brings loadavg irreversibly to 0 when you do enough freeze/thaw cycles.

That's right. I don't remember the exact details (it's been ages since this was first reported to me), but sendmail (IIRC) was looking at the load average and stopping delivery until it dropped sufficiently.

Historically, my solution has been to save and restore the load average values (on the theory that hibernation should be transparent to userspace), but I dropped that in favour of this approach.

Regards,

Nigel

A message that is not in the archive (sent off-list or not yet received): alpine.LFD.2.00.0907171441020.11571-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]

#15 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Hi.

Quoted 15 lines · Thomas Gleixner
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Quoted 8 lines
Quoted 2 lines

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

Fine with me, but it seems that the cgroup folks have some luser space stuff looking at proc/loadavg which goes berserk when loadavg increases rapidly due to freezing. OTOH that stuff seems to be oblivious to the fact that the commit in question brings loadavg irreversibly to 0 when you do enough freeze/thaw cycles.

That's right. I don't remember the exact details (it's been ages since this was first reported to me), but sendmail (IIRC) was looking at the load average and stopping delivery until it dropped sufficiently.

Historically, my solution has been to save and restore the load average values (on the theory that hibernation should be transparent to userspace), but I dropped that in favour of this approach.

Regards,

Nigel

#16 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner

Hi.

Quoted 15 lines · Thomas Gleixner
Quoted 13 lines · Peter Zijlstra
Quoted 12 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Quoted 8 lines
Quoted 2 lines

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

Fine with me, but it seems that the cgroup folks have some luser space stuff looking at proc/loadavg which goes berserk when loadavg increases rapidly due to freezing. OTOH that stuff seems to be oblivious to the fact that the commit in question brings loadavg irreversibly to 0 when you do enough freeze/thaw cycles.

That's right. I don't remember the exact details (it's been ages since this was first reported to me), but sendmail (IIRC) was looking at the load average and stopping delivery until it dropped sufficiently.

Historically, my solution has been to save and restore the load average values (on the theory that hibernation should be transparent to userspace), but I dropped that in favour of this approach.

Regards,

Nigel

#17 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 46 lines · Peter Zijlstra
Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

The changelog reads:

---
commit e3c8ca8336707062f3f7cb1cd7e6b3c753baccdd

Author: Nathan Lynch <[email protected]>
Date:   Wed Apr 8 19:45:12 2009 -0500

sched: do not count frozen tasks toward load

Freezing tasks via the cgroup freezer causes the load average to climb because the freezer's current implementation puts frozen tasks in uninterruptible sleep (D state).

Some applications which perform job-scheduling functions consult the load average when making decisions. If a cgroup is frozen, the load average does not provide a useful measure of the system's utilization to such applications. This is especially inconvenient if the job scheduler employs the cgroup freezer as a mechanism for preempting low priority jobs. Contrast this with using SIGSTOP for the same purpose:
the stopped tasks do not count toward system load.

Change task_contributes_to_load() to return false if the task is frozen. This results in /proc/loadavg behavior that better meets users' expectations.
---

It appears to me that a frozen cgroup is a transient state. Either you would typically do something like:

Quoted 6 lines

freeze -> {snapshot, migrate} -> {thaw, destroy}

Therefore a short increase in load doesn't seem like too big a problem, its going to be gone soon anyway.

Hmm?

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress. Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

Cheers,
-Matt Helsley

#18 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 46 lines · Peter Zijlstra
Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

The changelog reads:

---
commit e3c8ca8336707062f3f7cb1cd7e6b3c753baccdd

Author: Nathan Lynch <[email protected]>
Date:   Wed Apr 8 19:45:12 2009 -0500

sched: do not count frozen tasks toward load

Freezing tasks via the cgroup freezer causes the load average to climb because the freezer's current implementation puts frozen tasks in uninterruptible sleep (D state).

Some applications which perform job-scheduling functions consult the load average when making decisions. If a cgroup is frozen, the load average does not provide a useful measure of the system's utilization to such applications. This is especially inconvenient if the job scheduler employs the cgroup freezer as a mechanism for preempting low priority jobs. Contrast this with using SIGSTOP for the same purpose:
the stopped tasks do not count toward system load.

Change task_contributes_to_load() to return false if the task is frozen. This results in /proc/loadavg behavior that better meets users' expectations.
---

It appears to me that a frozen cgroup is a transient state. Either you would typically do something like:

Quoted 6 lines

freeze -> {snapshot, migrate} -> {thaw, destroy}

Therefore a short increase in load doesn't seem like too big a problem, its going to be gone soon anyway.

Hmm?

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress. Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

Cheers,
-Matt Helsley

#19 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 46 lines · Peter Zijlstra
Quoted 9 lines · Thomas Gleixner

plain text document attachment (freezer-fix-accounting-for-real.patch) commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Quoted 3 lines

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Right, so I'm wondering why we don't fully revert e3c8ca8336 to begin with.

The changelog reads:

---
commit e3c8ca8336707062f3f7cb1cd7e6b3c753baccdd

Author: Nathan Lynch <[email protected]>
Date:   Wed Apr 8 19:45:12 2009 -0500

sched: do not count frozen tasks toward load

Freezing tasks via the cgroup freezer causes the load average to climb because the freezer's current implementation puts frozen tasks in uninterruptible sleep (D state).

Some applications which perform job-scheduling functions consult the load average when making decisions. If a cgroup is frozen, the load average does not provide a useful measure of the system's utilization to such applications. This is especially inconvenient if the job scheduler employs the cgroup freezer as a mechanism for preempting low priority jobs. Contrast this with using SIGSTOP for the same purpose:
the stopped tasks do not count toward system load.

Change task_contributes_to_load() to return false if the task is frozen. This results in /proc/loadavg behavior that better meets users' expectations.
---

It appears to me that a frozen cgroup is a transient state. Either you would typically do something like:

Quoted 6 lines

freeze -> {snapshot, migrate} -> {thaw, destroy}

Therefore a short increase in load doesn't seem like too big a problem, its going to be gone soon anyway.

Hmm?

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress. Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

Cheers,
-Matt Helsley

maintainer
#20 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Quoted 4 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress. Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

#21 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress.

Right, the job scheduler uses the cgroup freezer as a mechanism to preempt a low priority job for a higher priority job. (It had used SIGSTOP in the past.) So in this scenario a frozen cgroup may remain in that state for a while. Load average is consulted as a measure of system utilization.

Quoted 4 lines
Quoted 2 lines

Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

I think Matt was referring to system-wide suspend/resume/hibernate, not a behavior of the job scheduler, if that's your concern.

#22 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress.

Right, the job scheduler uses the cgroup freezer as a mechanism to preempt a low priority job for a higher priority job. (It had used SIGSTOP in the past.) So in this scenario a frozen cgroup may remain in that state for a while. Load average is consulted as a measure of system utilization.

Quoted 4 lines
Quoted 2 lines

Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

I think Matt was referring to system-wide suspend/resume/hibernate, not a behavior of the job scheduler, if that's your concern.

#23 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress.

Right, the job scheduler uses the cgroup freezer as a mechanism to preempt a low priority job for a higher priority job. (It had used SIGSTOP in the past.) So in this scenario a frozen cgroup may remain in that state for a while. Load average is consulted as a measure of system utilization.

Quoted 4 lines
Quoted 2 lines

Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

I think Matt was referring to system-wide suspend/resume/hibernate, not a behavior of the job scheduler, if that's your concern.

maintainer
#24 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Quoted 11 lines · Nathan Lynch
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress.

Right, the job scheduler uses the cgroup freezer as a mechanism to preempt a low priority job for a higher priority job. (It had used SIGSTOP in the past.) So in this scenario a frozen cgroup may remain in that state for a while. Load average is consulted as a measure of system utilization.

I think that this is an utterly broken use for it, if you want something like that make a signal cgroup or something and deliver SIGSTOP to all of them.

In other words, why is the freezer any better than the SIGSTOP approach?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

I think Matt was referring to system-wide suspend/resume/hibernate, not a behavior of the job scheduler, if that's your concern.

I understood he referred to the crazy use-case you mentioned above, IMHO frozen should be a temporary state used for things like snapshot/migrate.

I'm still very tempted to plain simply revert that original patch.

#25 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Nathan Lynch
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress.

Right, the job scheduler uses the cgroup freezer as a mechanism to preempt a low priority job for a higher priority job. (It had used SIGSTOP in the past.) So in this scenario a frozen cgroup may remain in that state for a while. Load average is consulted as a measure of system utilization.

Quoted 5 lines

I think that this is an utterly broken use for it, if you want something like that make a signal cgroup or something and deliver SIGSTOP to all of them.

In other words, why is the freezer any better than the SIGSTOP approach?

Documentation/cgroups/freezer-subsystem.txt happens to document this use case and the disadvantages of SIGSTOP/SIGCONT. Does that change your opinion at all?

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

I think Matt was referring to system-wide suspend/resume/hibernate, not a behavior of the job scheduler, if that's your concern.

I understood he referred to the crazy use-case you mentioned above, IMHO frozen should be a temporary state used for things like snapshot/migrate.

But snapshot (or checkpoint) and migration aren't possible with mainline at this time. As far as I know, the use case to which you object is the primary use of the cgroup freezer on production systems.

#26 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Nathan Lynch
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress.

Right, the job scheduler uses the cgroup freezer as a mechanism to preempt a low priority job for a higher priority job. (It had used SIGSTOP in the past.) So in this scenario a frozen cgroup may remain in that state for a while. Load average is consulted as a measure of system utilization.

Quoted 5 lines

I think that this is an utterly broken use for it, if you want something like that make a signal cgroup or something and deliver SIGSTOP to all of them.

In other words, why is the freezer any better than the SIGSTOP approach?

Documentation/cgroups/freezer-subsystem.txt happens to document this use case and the disadvantages of SIGSTOP/SIGCONT. Does that change your opinion at all?

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

I think Matt was referring to system-wide suspend/resume/hibernate, not a behavior of the job scheduler, if that's your concern.

I understood he referred to the crazy use-case you mentioned above, IMHO frozen should be a temporary state used for things like snapshot/migrate.

But snapshot (or checkpoint) and migration aren't possible with mainline at this time. As far as I know, the use case to which you object is the primary use of the cgroup freezer on production systems.

maintainer
#27 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Nathan Lynch
Quoted 11 lines · Nathan Lynch
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress.

Right, the job scheduler uses the cgroup freezer as a mechanism to preempt a low priority job for a higher priority job. (It had used SIGSTOP in the past.) So in this scenario a frozen cgroup may remain in that state for a while. Load average is consulted as a measure of system utilization.

I think that this is an utterly broken use for it, if you want something like that make a signal cgroup or something and deliver SIGSTOP to all of them.

In other words, why is the freezer any better than the SIGSTOP approach?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

I think Matt was referring to system-wide suspend/resume/hibernate, not a behavior of the job scheduler, if that's your concern.

I understood he referred to the crazy use-case you mentioned above, IMHO frozen should be a temporary state used for things like snapshot/migrate.

I'm still very tempted to plain simply revert that original patch.

maintainer
#28 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Matt Helsley
Quoted 4 lines · Matt Helsley

The job scheduler in question does not use FROZEN as a transient state and does not use checkpoint/restart at all since c/r is still a work in progress. Even when used for power management it seems wrong to count frozen tasks towards the loadavg since they aren't using CPU time or waiting for IO.

You're abusing it for _WHAT_?

#29 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner
Quoted 11 lines · Thomas Gleixner

commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

I'm sorry, it's not clear to me based on the description what problem is being fixed. As far as I can see PF_FROZEN is almost exactly the same as PF_FREEZING. When a task is being frozen TIF_FREEZE is set. Then the task enters the refrigerator, sets PF_FROZEN, and schedule()s until PF_FROZEN is no longer set. The original code with extra comments:

static inline void frozen_process(void)

{
        if (!unlikely(current->flags & PF_NOFREEZE)) {
                current->flags |= PF_FROZEN;
                wmb();
        }
        clear_freeze_flag(current);
}

/* Refrigerator is place where frozen processes are stored :-). */ void refrigerator(void)

{
        /* Hmm, should we be allowed to suspend when there are realtime
           processes around? */
        long save;

        task_lock(current);
        if (freezing(current)) {
		/* prevent accounting of that task to load */
                frozen_process();  /* <-- sets PF_FROZEN */
                task_unlock(current);
        } else {
                task_unlock(current);
                return;
        }
        save = current->state;
        pr_debug("%s entered refrigerator\n", current->comm);

        spin_lock_irq(&current->sighand->siglock);
        recalc_sigpending(); /* We sent fake signal, clean it up */
        spin_unlock_irq(&current->sighand->siglock);

	/* you set PF_FREEZING here */
        for (;;) {
                set_current_state(TASK_UNINTERRUPTIBLE);
                if (!frozen(current))  /* <-- checks PF_FROZEN */
                        break;
                schedule();
        }
	/* you clear PF_FREEZING here */
        pr_debug("%s left refrigerator\n", current->comm);
        __set_current_state(save);
}
Quoted 47 lines
---
 include/linux/sched.h |    3 ++-
 kernel/freezer.c      |    7 +++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

Index: linux-2.6/include/linux/sched.h
===================================================================

include/linux/sched.h +2 -1
@@ -209,7 +209,7 @@ extern unsigned long long time_sync_thre
 			((task->state & (__TASK_STOPPED | __TASK_TRACED)) != 0)
 #define task_contributes_to_load(task)	\
 				((task->state & TASK_UNINTERRUPTIBLE) != 0 && \
-				 (task->flags & PF_FROZEN) == 0)
+				 (task->flags & PF_FREEZING) == 0)
 
 #define __set_task_state(tsk, state_value)		\
 	do { (tsk)->state = (state_value); } while (0)
@@ -1680,6 +1680,7 @@ extern cputime_t task_gtime(struct task_
 #define PF_MEMALLOC	0x00000800	/* Allocating memory */
 #define PF_FLUSHER	0x00001000	/* responsible for disk writeback */
 #define PF_USED_MATH	0x00002000	/* if unset the fpu must be initialized before use */
+#define PF_FREEZING	0x00004000	/* freeze in progress. do not account to load */
 #define PF_NOFREEZE	0x00008000	/* this thread should not be frozen */
 #define PF_FROZEN	0x00010000	/* frozen for system suspend */
 #define PF_FSTRANS	0x00020000	/* inside a filesystem transaction */

Index: linux-2.6/kernel/freezer.c
===================================================================

kernel/freezer.c +3 -0
@@ -44,12 +44,19 @@ void refrigerator(void)
 	recalc_sigpending(); /* We sent fake signal, clean it up */
 	spin_unlock_irq(&current->sighand->siglock);
 
+	/* prevent accounting of that task to load */
+	current->flags |= PF_FREEZING;
+

kernel/freezer.c void refrigerator(void) · line 59

Quoted 10 lines
kernel/freezer.c +4 -0
 	for (;;) {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		if (!frozen(current))
 			break;
 		schedule();
 	}
+
+	/* Remove the accounting blocker */
+	current->flags &= ~PF_FREEZING;
+

Hence PF_FREEZING covers slightly less time than PF_FROZEN but otherwise does not change the way nr_uninterruptible is incremented or decremented (in (de)activate_task()).

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Cheers,
-Matt Helsley

#30 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner
Quoted 11 lines · Thomas Gleixner

commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

I'm sorry, it's not clear to me based on the description what problem is being fixed. As far as I can see PF_FROZEN is almost exactly the same as PF_FREEZING. When a task is being frozen TIF_FREEZE is set. Then the task enters the refrigerator, sets PF_FROZEN, and schedule()s until PF_FROZEN is no longer set. The original code with extra comments:

static inline void frozen_process(void)

{
        if (!unlikely(current->flags & PF_NOFREEZE)) {
                current->flags |= PF_FROZEN;
                wmb();
        }
        clear_freeze_flag(current);
}

/* Refrigerator is place where frozen processes are stored :-). */ void refrigerator(void)

{
        /* Hmm, should we be allowed to suspend when there are realtime
           processes around? */
        long save;

        task_lock(current);
        if (freezing(current)) {
		/* prevent accounting of that task to load */
                frozen_process();  /* <-- sets PF_FROZEN */
                task_unlock(current);
        } else {
                task_unlock(current);
                return;
        }
        save = current->state;
        pr_debug("%s entered refrigerator\n", current->comm);

        spin_lock_irq(&current->sighand->siglock);
        recalc_sigpending(); /* We sent fake signal, clean it up */
        spin_unlock_irq(&current->sighand->siglock);

	/* you set PF_FREEZING here */
        for (;;) {
                set_current_state(TASK_UNINTERRUPTIBLE);
                if (!frozen(current))  /* <-- checks PF_FROZEN */
                        break;
                schedule();
        }
	/* you clear PF_FREEZING here */
        pr_debug("%s left refrigerator\n", current->comm);
        __set_current_state(save);
}
Quoted 47 lines
---
 include/linux/sched.h |    3 ++-
 kernel/freezer.c      |    7 +++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

Index: linux-2.6/include/linux/sched.h
===================================================================

include/linux/sched.h +2 -1
@@ -209,7 +209,7 @@ extern unsigned long long time_sync_thre
 			((task->state & (__TASK_STOPPED | __TASK_TRACED)) != 0)
 #define task_contributes_to_load(task)	\
 				((task->state & TASK_UNINTERRUPTIBLE) != 0 && \
-				 (task->flags & PF_FROZEN) == 0)
+				 (task->flags & PF_FREEZING) == 0)
 
 #define __set_task_state(tsk, state_value)		\
 	do { (tsk)->state = (state_value); } while (0)
@@ -1680,6 +1680,7 @@ extern cputime_t task_gtime(struct task_
 #define PF_MEMALLOC	0x00000800	/* Allocating memory */
 #define PF_FLUSHER	0x00001000	/* responsible for disk writeback */
 #define PF_USED_MATH	0x00002000	/* if unset the fpu must be initialized before use */
+#define PF_FREEZING	0x00004000	/* freeze in progress. do not account to load */
 #define PF_NOFREEZE	0x00008000	/* this thread should not be frozen */
 #define PF_FROZEN	0x00010000	/* frozen for system suspend */
 #define PF_FSTRANS	0x00020000	/* inside a filesystem transaction */

Index: linux-2.6/kernel/freezer.c
===================================================================

kernel/freezer.c +3 -0
@@ -44,12 +44,19 @@ void refrigerator(void)
 	recalc_sigpending(); /* We sent fake signal, clean it up */
 	spin_unlock_irq(&current->sighand->siglock);
 
+	/* prevent accounting of that task to load */
+	current->flags |= PF_FREEZING;
+

kernel/freezer.c void refrigerator(void) · line 59

Quoted 10 lines
kernel/freezer.c +4 -0
 	for (;;) {
 		set_current_state(TASK_UNINTERRUPTIBLE);
 		if (!frozen(current))
 			break;
 		schedule();
 	}
+
+	/* Remove the accounting blocker */
+	current->flags &= ~PF_FREEZING;
+

Hence PF_FREEZING covers slightly less time than PF_FROZEN but otherwise does not change the way nr_uninterruptible is incremented or decremented (in (de)activate_task()).

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Cheers,
-Matt Helsley

A message that is not in the archive (sent off-list or not yet received): 20090717155116.GB5878-52DBMbEzqgQ/wnmkkaCWp/[email protected]

#32 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Quoted 2 lines · Matt Helsley

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Yep, PF_FROZEN is only observed going into schedule(), on the way back out thaw_process() will already have cleared the flag.

So we don't increment in deactivate_task(), but we do decrement in activate_task(), yielding for some very unbalanced accounting.

maintainer
#33 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Matt Helsley
Quoted 2 lines · Matt Helsley

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Yep, PF_FROZEN is only observed going into schedule(), on the way back out thaw_process() will already have cleared the flag.

So we don't increment in deactivate_task(), but we do decrement in activate_task(), yielding for some very unbalanced accounting.

maintainer
#34 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Matt Helsley
Quoted 2 lines · Matt Helsley

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Yep, PF_FROZEN is only observed going into schedule(), on the way back out thaw_process() will already have cleared the flag.

So we don't increment in deactivate_task(), but we do decrement in activate_task(), yielding for some very unbalanced accounting.

#35 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Yep, PF_FROZEN is only observed going into schedule(), on the way back out thaw_process() will already have cleared the flag.

So we don't increment in deactivate_task(), but we do decrement in activate_task(), yielding for some very unbalanced accounting.

Ahhh, right!

OK, so the fix looks good then.

Cheers,
-Matt Helsley

#36 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Yep, PF_FROZEN is only observed going into schedule(), on the way back out thaw_process() will already have cleared the flag.

So we don't increment in deactivate_task(), but we do decrement in activate_task(), yielding for some very unbalanced accounting.

Ahhh, right!

OK, so the fix looks good then.

Cheers,
-Matt Helsley

#37 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 2 lines · Matt Helsley

So it's not clear to me how adding PF_FREEZING fixes anything. Am I missing something?

Yep, PF_FROZEN is only observed going into schedule(), on the way back out thaw_process() will already have cleared the flag.

So we don't increment in deactivate_task(), but we do decrement in activate_task(), yielding for some very unbalanced accounting.

Ahhh, right!

OK, so the fix looks good then.

Cheers,
-Matt Helsley

#38 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner
Quoted 11 lines · Thomas Gleixner

commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Thanks for the fix, and sorry for the regression. I'll test this early next week but I don't anticipate any problems for the use case the original patch was addressing.

#39 lore raw

Re: [patch 2/2] sched: fix nr_uninterruptible accounting of frozen tasks really

In reply to Thomas Gleixner
Quoted 11 lines · Thomas Gleixner

commit e3c8ca8336 (sched: do not count frozen tasks toward load) broke the nr_uninterruptible accounting on freeze/thaw. On freeze the task is excluded from accounting with a check for (task->flags & PF_FROZEN), but that flag is cleared before the task is thawed. So while we prevent that the freezing task with state TASK_UNINTERRUPTIBLE is accounted to nr_uninterruptible we decrement nr_uninterruptible on thaw.

Use a separate flag which is handled by the freezing task itself. Set it before calling the scheduler with TASK_UNINTERRUPTIBLE state and clear it after we return from frozen state.

Thanks for the fix, and sorry for the regression. I'll test this early next week but I don't anticipate any problems for the use case the original patch was addressing.