Skip to contentSkip to conversation

cputime patch for 2.6.30-rc6

GIT PULL Martin Schwidefsky 4 participants 16 messages May 18, 2009 – May 25, 2009 Proc filesystem

lore
Share & tools
lore
#0 lore raw

Hi Linus,

please pull from 'cputime' branch of

	git://git390.marist.edu/pub/scm/linux-2.6.git cputime

to receive the following updates:

Michael Abbott (1):
Fix idle time field in /proc/uptime

 fs/proc/uptime.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)
fs/proc/uptime.c +7 -1
index 0c10a0b..c0ac0d7 100644
@@ -4,13 +4,19 @@
 #include <linux/sched.h>
 #include <linux/seq_file.h>
 #include <linux/time.h>
+#include <linux/kernel_stat.h>
 #include <asm/cputime.h>
 
 static int uptime_proc_show(struct seq_file *m, void *v)
 {
 	struct timespec uptime;
 	struct timespec idle;
-	cputime_t idletime = cputime_add(init_task.utime, init_task.stime);
+	int len, i;
+	cputime_t idletime = 0;
+
+	for_each_possible_cpu(i)
+		idletime = cputime64_add(idletime, kstat_cpu(i).cpustat.idle);
+	idletime = cputime64_to_clock_t(idletime);
 
 	do_posix_clock_monotonic_gettime(&uptime);
 	monotonic_to_bootbased(&uptime);
#1 lore raw
Quoted 22 lines · Martin Schwidefsky

Hi Linus,

please pull from 'cputime' branch of

	git://git390.marist.edu/pub/scm/linux-2.6.git cputime

to receive the following updates:

Michael Abbott (1):
Fix idle time field in /proc/uptime

 fs/proc/uptime.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)
fs/proc/uptime.c +0 -0
index 0c10a0b..c0ac0d7 100644
@@ -4,13 +4,19 @@
 #include <linux/sched.h>
 #include <linux/seq_file.h>
 #include <linux/time.h>

fs/proc/uptime.c line 15

Quoted 10 lines
fs/proc/uptime.c +3 -1
+#include <linux/kernel_stat.h>
 #include <asm/cputime.h>
 
 static int uptime_proc_show(struct seq_file *m, void *v)
 {
 	struct timespec uptime;
 	struct timespec idle;
-	cputime_t idletime = cputime_add(init_task.utime, init_task.stime);
+	int len, i;
+	cputime_t idletime = 0;

cputime_zero, I guess..

fs/proc/uptime.c line 22

Quoted 6 lines
fs/proc/uptime.c +3 -0
+	for_each_possible_cpu(i)
+		idletime = cputime64_add(idletime, kstat_cpu(i).cpustat.idle);
+	idletime = cputime64_to_clock_t(idletime);
 
 	do_posix_clock_monotonic_gettime(&uptime);
 	monotonic_to_bootbased(&uptime);

This is a world readable proc file, adding a for_each_possible_cpu() in there scares me a little (this wouldn't be the first and only such case though).

Suppose you have lots of cpus, and all those cpus are dirtying those cachelines (who's updating idle time when they're idle?), then this loop can cause a massive cacheline bounce fest.

Then think about userspace doing:
while :; do cat /proc/uptime > /dev/null; done

#2 lore raw
Quoted 47 lines · Peter Zijlstra
Quoted 22 lines · Martin Schwidefsky

Hi Linus,

please pull from 'cputime' branch of

	git://git390.marist.edu/pub/scm/linux-2.6.git cputime

to receive the following updates:

Michael Abbott (1):
Fix idle time field in /proc/uptime

 fs/proc/uptime.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)
fs/proc/uptime.c +0 -0
index 0c10a0b..c0ac0d7 100644
@@ -4,13 +4,19 @@
 #include <linux/sched.h>
 #include <linux/seq_file.h>
 #include <linux/time.h>

fs/proc/uptime.c line 15

Quoted 10 lines
fs/proc/uptime.c +3 -1
+#include <linux/kernel_stat.h>
 #include <asm/cputime.h>
 
 static int uptime_proc_show(struct seq_file *m, void *v)
 {
 	struct timespec uptime;
 	struct timespec idle;
-	cputime_t idletime = cputime_add(init_task.utime, init_task.stime);
+	int len, i;
+	cputime_t idletime = 0;

cputime_zero, I guess..

fs/proc/uptime.c line 22

Quoted 6 lines
fs/proc/uptime.c +3 -0
+	for_each_possible_cpu(i)
+		idletime = cputime64_add(idletime, kstat_cpu(i).cpustat.idle);
+	idletime = cputime64_to_clock_t(idletime);
 
 	do_posix_clock_monotonic_gettime(&uptime);
 	monotonic_to_bootbased(&uptime);

This is a world readable proc file, adding a for_each_possible_cpu() in there scares me a little (this wouldn't be the first and only such case though).

Quoted 6 lines

Suppose you have lots of cpus, and all those cpus are dirtying those cachelines (who's updating idle time when they're idle?), then this loop can cause a massive cacheline bounce fest.

Then think about userspace doing:
while :; do cat /proc/uptime > /dev/null; done

Well, the offending code derives pretty well directly from /proc/stat, which is used, for example, by top. So if there is an issue then I guess it already exists.

There is a pending problem in this code: for a multiple cpu system we'll end up with more idle time than elapsed time, which is not really very nice. Unfortunately *something* has to be done here, as it looks as if .utime and .stime (at least for init_task) have lost any meaning. I sort of though of dividing by number of cpus, but that's not going to work very well..

I came to this problem from a uni-processor instrument which uses /proc/uptime to determine whether the system is overloaded (and discovers on the current kernel that it is, permanently!). This fix is definitely imperfect, but I think a better fix will require rather deeper knowledge of kernel time accounting than I can offer.

#3 lore raw
Quoted 22 lines · On Mon, 18 May 2009 17:28:53 +0100 (BST) Michael Abbott…
Quoted 11 lines

fs/proc/uptime.c line 22

Quoted 6 lines
fs/proc/uptime.c +3 -0
+	for_each_possible_cpu(i)
+		idletime = cputime64_add(idletime, kstat_cpu(i).cpustat.idle);
+	idletime = cputime64_to_clock_t(idletime);
 
 	do_posix_clock_monotonic_gettime(&uptime);
 	monotonic_to_bootbased(&uptime);

This is a world readable proc file, adding a for_each_possible_cpu() in there scares me a little (this wouldn't be the first and only such case though).

Quoted 6 lines

Suppose you have lots of cpus, and all those cpus are dirtying those cachelines (who's updating idle time when they're idle?), then this loop can cause a massive cacheline bounce fest.

Then think about userspace doing:
while :; do cat /proc/uptime > /dev/null; done

Well, the offending code derives pretty well directly from /proc/stat, which is used, for example, by top. So if there is an issue then I guess it already exists.

Quoted 6 lines

There is a pending problem in this code: for a multiple cpu system we'll end up with more idle time than elapsed time, which is not really very nice. Unfortunately *something* has to be done here, as it looks as if .utime and .stime (at least for init_task) have lost any meaning. I sort of though of dividing by number of cpus, but that's not going to work very well..

I don't see a problem here. In an idle multiple cpu system there IS more idle time than elapsed time. What would makes sense is to compare elapsed time * #cpus with the idle time. But then there is cpu hotplug which forces you to look at the delta of two measuring points where the number of cpus did not change.

Quoted 5 lines

I came to this problem from a uni-processor instrument which uses /proc/uptime to determine whether the system is overloaded (and discovers on the current kernel that it is, permanently!). This fix is definitely imperfect, but I think a better fix will require rather deeper knowledge of kernel time accounting than I can offer.

Hmm, I would use the idle time field from /proc/stat for that.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#4 lore raw
Quoted 32 lines · Martin Schwidefsky
Quoted 22 lines · On Mon, 18 May 2009 17:28:53 +0100 (BST) Michael Abbott…
Quoted 11 lines

fs/proc/uptime.c line 22

Quoted 6 lines
fs/proc/uptime.c +3 -0
+	for_each_possible_cpu(i)
+		idletime = cputime64_add(idletime, kstat_cpu(i).cpustat.idle);
+	idletime = cputime64_to_clock_t(idletime);
 
 	do_posix_clock_monotonic_gettime(&uptime);
 	monotonic_to_bootbased(&uptime);

This is a world readable proc file, adding a for_each_possible_cpu() in there scares me a little (this wouldn't be the first and only such case though).

Quoted 6 lines

Suppose you have lots of cpus, and all those cpus are dirtying those cachelines (who's updating idle time when they're idle?), then this loop can cause a massive cacheline bounce fest.

Then think about userspace doing:
while :; do cat /proc/uptime > /dev/null; done

Well, the offending code derives pretty well directly from /proc/stat, which is used, for example, by top. So if there is an issue then I guess it already exists.

Quoted 6 lines

There is a pending problem in this code: for a multiple cpu system we'll end up with more idle time than elapsed time, which is not really very nice. Unfortunately *something* has to be done here, as it looks as if .utime and .stime (at least for init_task) have lost any meaning. I sort of though of dividing by number of cpus, but that's not going to work very well..

Quoted 5 lines

I don't see a problem here. In an idle multiple cpu system there IS more idle time than elapsed time. What would makes sense is to compare elapsed time * #cpus with the idle time. But then there is cpu hotplug which forces you to look at the delta of two measuring points where the number of cpus did not change.

Sure, this one case isn't that bad, esp. as you note its about idle time. However, see for example /proc/stat and fs/proc/stat.c:

        for_each_possible_cpu(i) {
                user = cputime64_add(user, kstat_cpu(i).cpustat.user);
                nice = cputime64_add(nice, kstat_cpu(i).cpustat.nice);
                system = cputime64_add(system, kstat_cpu(i).cpustat.system);
                idle = cputime64_add(idle, kstat_cpu(i).cpustat.idle);
                idle = cputime64_add(idle, arch_idle_time(i));
                iowait = cputime64_add(iowait, kstat_cpu(i).cpustat.iowait);
                irq = cputime64_add(irq, kstat_cpu(i).cpustat.irq);
                softirq = cputime64_add(softirq, kstat_cpu(i).cpustat.softirq);
                steal = cputime64_add(steal, kstat_cpu(i).cpustat.steal);
                guest = cputime64_add(guest, kstat_cpu(i).cpustat.guest);
                for_each_irq_nr(j) {
                        sum += kstat_irqs_cpu(j, i);
                }
                sum += arch_irq_stat_cpu(i);
        }

If that isn't a problem on a large machine, then I don't know what is.

#5 lore raw
Quoted 27 lines · On Tue, 19 May 2009 11:31:28 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 5 lines · Martin Schwidefsky

I don't see a problem here. In an idle multiple cpu system there IS more idle time than elapsed time. What would makes sense is to compare elapsed time * #cpus with the idle time. But then there is cpu hotplug which forces you to look at the delta of two measuring points where the number of cpus did not change.

Sure, this one case isn't that bad, esp. as you note its about idle time. However, see for example /proc/stat and fs/proc/stat.c:

        for_each_possible_cpu(i) {
                user = cputime64_add(user, kstat_cpu(i).cpustat.user);
                nice = cputime64_add(nice, kstat_cpu(i).cpustat.nice);
                system = cputime64_add(system, kstat_cpu(i).cpustat.system);
                idle = cputime64_add(idle, kstat_cpu(i).cpustat.idle);
                idle = cputime64_add(idle, arch_idle_time(i));
                iowait = cputime64_add(iowait, kstat_cpu(i).cpustat.iowait);
                irq = cputime64_add(irq, kstat_cpu(i).cpustat.irq);
                softirq = cputime64_add(softirq, kstat_cpu(i).cpustat.softirq);
                steal = cputime64_add(steal, kstat_cpu(i).cpustat.steal);
                guest = cputime64_add(guest, kstat_cpu(i).cpustat.guest);
                for_each_irq_nr(j) {
                        sum += kstat_irqs_cpu(j, i);
                }
                sum += arch_irq_stat_cpu(i);
        }
Quoted 1 line

If that isn't a problem on a large machine, then I don't know what is.

Well, we better distinguish between the semantical problem and the performance consideration, no? One thing is what the proc field is supposed to contain, the other is how fast you can do it.

I have been refering to the semantical problem, but your point with the performance is very valid as well. So
1) are we agreed that the second field of /proc/uptime should contain the aggregate idle time of all cpus?
2) I agree that an endless loop of 'cat /proc/uptime' or 'cat /proc/stat' can have negative performance impact on a system with many cpus. One way to deal with it would be to restrict access to the interface. I do not like that idea to much. Another way would be to limit the number of for_each_possible_cpu loops per second. Create global variables that contain the aggregate values for the different fields. If the last update has been too recent (e.g. less than 0.1 seconds ago), just print the old values again.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#6 lore raw
Quoted 32 lines · Martin Schwidefsky
Quoted 27 lines · On Tue, 19 May 2009 11:31:28 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 5 lines · Martin Schwidefsky

I don't see a problem here. In an idle multiple cpu system there IS more idle time than elapsed time. What would makes sense is to compare elapsed time * #cpus with the idle time. But then there is cpu hotplug which forces you to look at the delta of two measuring points where the number of cpus did not change.

Sure, this one case isn't that bad, esp. as you note its about idle time. However, see for example /proc/stat and fs/proc/stat.c:

        for_each_possible_cpu(i) {
                user = cputime64_add(user, kstat_cpu(i).cpustat.user);
                nice = cputime64_add(nice, kstat_cpu(i).cpustat.nice);
                system = cputime64_add(system, kstat_cpu(i).cpustat.system);
                idle = cputime64_add(idle, kstat_cpu(i).cpustat.idle);
                idle = cputime64_add(idle, arch_idle_time(i));
                iowait = cputime64_add(iowait, kstat_cpu(i).cpustat.iowait);
                irq = cputime64_add(irq, kstat_cpu(i).cpustat.irq);
                softirq = cputime64_add(softirq, kstat_cpu(i).cpustat.softirq);
                steal = cputime64_add(steal, kstat_cpu(i).cpustat.steal);
                guest = cputime64_add(guest, kstat_cpu(i).cpustat.guest);
                for_each_irq_nr(j) {
                        sum += kstat_irqs_cpu(j, i);
                }
                sum += arch_irq_stat_cpu(i);
        }
Quoted 1 line

If that isn't a problem on a large machine, then I don't know what is.

Quoted 8 lines

Well, we better distinguish between the semantical problem and the performance consideration, no? One thing is what the proc field is supposed to contain, the other is how fast you can do it.

I have been refering to the semantical problem, but your point with the performance is very valid as well. So
1) are we agreed that the second field of /proc/uptime should contain the aggregate idle time of all cpus?

More or less, yeah ;-)

Quoted 8 lines

2) I agree that an endless loop of 'cat /proc/uptime' or 'cat /proc/stat' can have negative performance impact on a system with many cpus. One way to deal with it would be to restrict access to the interface. I do not like that idea to much. Another way would be to limit the number of for_each_possible_cpu loops per second. Create global variables that contain the aggregate values for the different fields. If the last update has been too recent (e.g. less than 0.1 seconds ago), just print the old values again.

Right, I was thinking about using things like percpu_counter and vmstat which get more inaccurate the larger your machine is, that is inaccurate in the temporal sense, they do converge to the actual result in that nothing is lost, it might just take a while to show up.

#7 lore raw
In reply to Martin Schwidefsky
Quoted 12 lines · Martin Schwidefsky
Quoted 6 lines · On Tue, 19 May 2009 11:31:28 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 5 lines · Martin Schwidefsky

I don't see a problem here. In an idle multiple cpu system there IS more idle time than elapsed time. What would makes sense is to compare elapsed time * #cpus with the idle time. But then there is cpu hotplug which forces you to look at the delta of two measuring points where the number of cpus did not change.

Well, we better distinguish between the semantical problem and the performance consideration, no? One thing is what the proc field is supposed to contain, the other is how fast you can do it.

Quoted 5 lines

I have been refering to the semantical problem, but your point with the performance is very valid as well. So

1) are we agreed that the second field of /proc/uptime should contain the aggregate idle time of all cpus?

I think that this very simple node (only two fields, let me call them uptime and idle) should be amenable to simple interpretation. In particular, I'd like to be able to compute

busy = uptime - idle

In other words, idle should be in the same units of time as uptime, which is to say it should be in units of elapsed time, not aggregate CPU time.

As far as I can see, that is how it used to be (at least, that's how it behaves on a dual core processor running a hugely patched 2.6.9 kernel that I'm looking at here).

Thus, the patch to /proc/uptime that we're discussing has, in the view I'm trying to argue, replaced a clearly broken idle field with a more subtly broken version.

The strongest point in my case is this: the first field is in units of elapsed wall clock time, and therefore the second should also be. Of course, established practice here is also important, but there is some ambiguity in the man pages I'm looking at -- this field is described in proc(5) as:

the amount of time spent in idle process (seconds)

Not aggregate CPU time, but elapsed time, I think is meant here, but the wording is ambiguous. Grepping for uptime in Documentation doesn't come up with anything clear either, but I can quote old (2.6.9) precedent (don't have a more recent multi-core system to hand).

#8 lore raw
Quoted 16 lines · On Wed, 20 May 2009 09:44:33 +0100 (BST) Michael Abbott…
Quoted 15 lines · Martin Schwidefsky
Quoted 6 lines · On Tue, 19 May 2009 11:31:28 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 5 lines · Martin Schwidefsky

I don't see a problem here. In an idle multiple cpu system there IS more idle time than elapsed time. What would makes sense is to compare elapsed time * #cpus with the idle time. But then there is cpu hotplug which forces you to look at the delta of two measuring points where the number of cpus did not change.

Well, we better distinguish between the semantical problem and the performance consideration, no? One thing is what the proc field is supposed to contain, the other is how fast you can do it.

I have been refering to the semantical problem, but your point with the performance is very valid as well. So

Quoted 8 lines
Quoted 2 lines

1) are we agreed that the second field of /proc/uptime should contain the aggregate idle time of all cpus?

I think that this very simple node (only two fields, let me call them uptime and idle) should be amenable to simple interpretation. In particular, I'd like to be able to compute

busy = uptime - idle

On a single cpu system this works. On an SMP with cpu hotplug it breaks.

Quoted 2 lines

In other words, idle should be in the same units of time as uptime, which is to say it should be in units of elapsed time, not aggregate CPU time.

They are, no? The idle time is in units of cputime but what is measured is wall-clock time. In fact the idle time is the result of the calculation
idle = wall_clock - user - nice - system -
iowait - irq - softirq - steal - guest
Trouble is that this is fundamentally per cpu. Just showing a single cpu is kaputt, showing the sum of all cpus divided by the number of cpus has problem with cpu hotplug. That leaves the simple approach to just do the sum over all cpus.

Quoted 3 lines

Thus, the patch to /proc/uptime that we're discussing has, in the view I'm trying to argue, replaced a clearly broken idle field with a more subtly broken version.

What is the least broken version?

Quoted 6 lines

The strongest point in my case is this: the first field is in units of elapsed wall clock time, and therefore the second should also be. Of course, established practice here is also important, but there is some ambiguity in the man pages I'm looking at -- this field is described in proc(5) as:

Quoted 6 lines

the amount of time spent in idle process (seconds)

Not aggregate CPU time, but elapsed time, I think is meant here, but the wording is ambiguous. Grepping for uptime in Documentation doesn't come up with anything clear either, but I can quote old (2.6.9) precedent (don't have a more recent multi-core system to hand).

In my view of the world the idle field IS wall clock. That is how we calculate it on s390.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#9 lore raw
In reply to Peter Zijlstra
Quoted 26 lines · On Mon, 18 May 2009 17:24:03 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 25 lines · Martin Schwidefsky

Hi Linus,

please pull from 'cputime' branch of

	git://git390.marist.edu/pub/scm/linux-2.6.git cputime

to receive the following updates:

Michael Abbott (1):
Fix idle time field in /proc/uptime

 fs/proc/uptime.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)
fs/proc/uptime.c +1 -0
index 0c10a0b..c0ac0d7 100644
@@ -4,13 +4,19 @@
 #include <linux/sched.h>
 #include <linux/seq_file.h>
 #include <linux/time.h>
+#include <linux/kernel_stat.h>
 #include <asm/cputime.h>
 
Quoted 9 lines

fs/proc/uptime.c line 15

Quoted 7 lines
fs/proc/uptime.c +2 -1
 static int uptime_proc_show(struct seq_file *m, void *v)
 {
 	struct timespec uptime;
 	struct timespec idle;
-	cputime_t idletime = cputime_add(init_task.utime, init_task.stime);
+	int len, i;
+	cputime_t idletime = 0;

cputime_zero, I guess..

Yes, this needs to be cputime_zero.

Quoted 11 lines

fs/proc/uptime.c line 22

Quoted 6 lines
fs/proc/uptime.c +3 -0
+	for_each_possible_cpu(i)
+		idletime = cputime64_add(idletime, kstat_cpu(i).cpustat.idle);
+	idletime = cputime64_to_clock_t(idletime);
 
 	do_posix_clock_monotonic_gettime(&uptime);
 	monotonic_to_bootbased(&uptime);

This is a world readable proc file, adding a for_each_possible_cpu() in there scares me a little (this wouldn't be the first and only such case though).

Quoted 6 lines

Suppose you have lots of cpus, and all those cpus are dirtying those cachelines (who's updating idle time when they're idle?), then this loop can cause a massive cacheline bounce fest.

Then think about userspace doing:
while :; do cat /proc/uptime > /dev/null; done

Well, don't do stupid things like that. That falls into the same category as programs calling gettimeofday all the time.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#10 lore raw
Quoted 2 lines · Martin Schwidefsky

Well, don't do stupid things like that. That falls into the same category as programs calling gettimeofday all the time.

Is gtod a global DoS? The worry I have about these cputime for_each_cpu() loops in the proc code is that any odd unpriv user can take the whole machine down (provided the machine is large enough).

Granted, this would only be a real problem for SGI and their insane boxes atm, but it might very well be a problem for us in the near future, given the rate Intel seems to increase cores.

So, I'm really not objecting too much to the patch at hand, but I'd love to find a solution to this problem.

My personal favourite is to lower the resolution/accuracy of the answer the bigger the box gets.

#11 lore raw
Quoted 2 lines · On Tue, 19 May 2009 11:00:35 +0200 Peter Zijlstra <[email protected]> wrote:

So, I'm really not objecting too much to the patch at hand, but I'd love to find a solution to this problem.

It is not hard so solve the problem for /proc/uptime, e.g. like this:

static u64 uptime_jiffies = INITIAL_JIFFIES;
static struct timespec ts_uptime;
static struct timespec ts_idle;

static int uptime_proc_show(struct seq_file *m, void *v)

{
        cputime_t idletime;
        u64 now;
        int i;

        now = get_jiffies_64();
        if (uptime_jiffies != now) {
                uptime_jiffies = now;
                idletime = cputime_zero;
                for_each_possible_cpu(i)
                        idletime = cputime64_add(idletime,
                                                 kstat_cpu(i).cpustat.idle);
                do_posix_clock_monotonic_gettime(&ts_uptime);
                monotonic_to_bootbased(&ts_uptime);
                cputime_to_timespec(idletime, &ts_idle);
        }

        seq_printf(m, "%lu.%02lu %lu.%02lu\n",
                        (unsigned long) ts_uptime.tv_sec,
                        (ts_uptime.tv_nsec / (NSEC_PER_SEC / 100)),
                        (unsigned long) ts_idle.tv_sec,
                        (ts_idle.tv_nsec / (NSEC_PER_SEC / 100)));
        return 0;
}

For /proc/stat it is less clear. Just storing the values in static variables is not such a good idea as there are lots of values. 10*NR_CPUS + NR_IRQS values to be exact. With NR_CPUS in the thousands this will waste quite a bit of memory.

I fixed another problem with Michael original patch and added the new one to reduce the frequency of accesses to kstat_cpu for /proc/uptime. You'll find both at:

	git://git390.marist.edu/pub/scm/linux-2.6.git cputime
Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#12 lore raw
Quoted 38 lines · Martin Schwidefsky
Quoted 2 lines · On Tue, 19 May 2009 11:00:35 +0200 Peter Zijlstra <[email protected]> wrote:

So, I'm really not objecting too much to the patch at hand, but I'd love to find a solution to this problem.

It is not hard so solve the problem for /proc/uptime, e.g. like this:

static u64 uptime_jiffies = INITIAL_JIFFIES;
static struct timespec ts_uptime;
static struct timespec ts_idle;

static int uptime_proc_show(struct seq_file *m, void *v)

{
        cputime_t idletime;
        u64 now;
        int i;

        now = get_jiffies_64();
        if (uptime_jiffies != now) {
                uptime_jiffies = now;
                idletime = cputime_zero;
                for_each_possible_cpu(i)
                        idletime = cputime64_add(idletime,
                                                 kstat_cpu(i).cpustat.idle);
                do_posix_clock_monotonic_gettime(&ts_uptime);
                monotonic_to_bootbased(&ts_uptime);
                cputime_to_timespec(idletime, &ts_idle);
        }

        seq_printf(m, "%lu.%02lu %lu.%02lu\n",
                        (unsigned long) ts_uptime.tv_sec,
                        (ts_uptime.tv_nsec / (NSEC_PER_SEC / 100)),
                        (unsigned long) ts_idle.tv_sec,
                        (ts_idle.tv_nsec / (NSEC_PER_SEC / 100)));
        return 0;
}
Quoted 4 lines

For /proc/stat it is less clear. Just storing the values in static variables is not such a good idea as there are lots of values. 10*NR_CPUS + NR_IRQS values to be exact. With NR_CPUS in the thousands this will waste quite a bit of memory.

Right, I know of for_each_possible_cpu() loops that took longer than a jiffy and caused general melt-down -- not saying the loop for idle time will be one such a loop, but then it seems silly anyway, who's incrementing the idle time when we're idle?

I really prefer using things like percpu_counter/vmstat that have error bounds that scale with the number of cpus in the system.

We simply have to start educating people that numbers on the global state of the machine are inaccurate (they were anyway, because by the time the userspace bits that read the /proc file get scheduled again the numbers will have changed again).

There's a variant of Heisenberg's uncertainty principle applicable to (parallel) computers in that one either gets concurrency or accuracy on global state, you cannot have both.

#13 lore raw
Quoted 4 lines · Peter Zijlstra

I really prefer using things like percpu_counter/vmstat that have error bounds that scale with the number of cpus in the system.

Quoted 9 lines

We simply have to start educating people that numbers on the global state of the machine are inaccurate (they were anyway, because by the time the userspace bits that read the /proc file get scheduled again the numbers will have changed again).

There's a variant of Heisenberg's uncertainty principle applicable to (parallel) computers in that one either gets concurrency or accuracy on global state, you cannot have both.

Inaccuracies are fine, esp. for something like monotic increasing idle uptime.

#14 lore raw
In reply to Peter Zijlstra
Quoted 44 lines · On Mon, 25 May 2009 13:09:26 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 38 lines · Martin Schwidefsky
Quoted 2 lines · On Tue, 19 May 2009 11:00:35 +0200 Peter Zijlstra <[email protected]> wrote:

So, I'm really not objecting too much to the patch at hand, but I'd love to find a solution to this problem.

It is not hard so solve the problem for /proc/uptime, e.g. like this:

static u64 uptime_jiffies = INITIAL_JIFFIES;
static struct timespec ts_uptime;
static struct timespec ts_idle;

static int uptime_proc_show(struct seq_file *m, void *v)

{
        cputime_t idletime;
        u64 now;
        int i;

        now = get_jiffies_64();
        if (uptime_jiffies != now) {
                uptime_jiffies = now;
                idletime = cputime_zero;
                for_each_possible_cpu(i)
                        idletime = cputime64_add(idletime,
                                                 kstat_cpu(i).cpustat.idle);
                do_posix_clock_monotonic_gettime(&ts_uptime);
                monotonic_to_bootbased(&ts_uptime);
                cputime_to_timespec(idletime, &ts_idle);
        }

        seq_printf(m, "%lu.%02lu %lu.%02lu\n",
                        (unsigned long) ts_uptime.tv_sec,
                        (ts_uptime.tv_nsec / (NSEC_PER_SEC / 100)),
                        (unsigned long) ts_idle.tv_sec,
                        (ts_idle.tv_nsec / (NSEC_PER_SEC / 100)));
        return 0;
}
Quoted 4 lines

For /proc/stat it is less clear. Just storing the values in static variables is not such a good idea as there are lots of values. 10*NR_CPUS + NR_IRQS values to be exact. With NR_CPUS in the thousands this will waste quite a bit of memory.

Quoted 4 lines

Right, I know of for_each_possible_cpu() loops that took longer than a jiffy and caused general melt-down -- not saying the loop for idle time will be one such a loop, but then it seems silly anyway, who's incrementing the idle time when we're idle?

Psst, I do ;-) Look at the arch_idle_time macro in fs/proc/stat.c..

Quoted 7 lines

I really prefer using things like percpu_counter/vmstat that have error bounds that scale with the number of cpus in the system.

We simply have to start educating people that numbers on the global state of the machine are inaccurate (they were anyway, because by the time the userspace bits that read the /proc file get scheduled again the numbers will have changed again).

That is one problem, the other is that the values you'll get are not atomic in any way. Not even the totals in /proc/stat match the sum over the cpus.

Quoted 3 lines

There's a variant of Heisenberg's uncertainty principle applicable to (parallel) computers in that one either gets concurrency or accuracy on global state, you cannot have both.

If the time you need to generate a value is longer than the maximum error you do have a problem.

Signature · 4 lines
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.
#15 lore raw
In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra

fs/proc/uptime.c line 22

Quoted 6 lines
fs/proc/uptime.c +3 -0
+	for_each_possible_cpu(i)
+		idletime = cputime64_add(idletime, kstat_cpu(i).cpustat.idle);
+	idletime = cputime64_to_clock_t(idletime);
 
 	do_posix_clock_monotonic_gettime(&uptime);
 	monotonic_to_bootbased(&uptime);

This is a world readable proc file, adding a for_each_possible_cpu() in there scares me a little (this wouldn't be the first and only such case though).

Well maybe procfs could gain chmod support (with the same lifetime guarantee that one would have with tmpfs), so that init scripts of big boxes can chmod it 400 on boot if there is that potential user threat:

Quoted 2 lines

Then think about userspace doing:
while :; do cat /proc/uptime > /dev/null; done