Skip to contentSkip to conversation

Nasty suprise with uptime

PATCH Tim Schmielau 7 participants 9 messages Oct 31, 2001 – Nov 1, 2001 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 3 lines
Quoted 2 lines

I would say that the race is so rare that it should not be handled, especially since it adds extra code in the timer interrupt.

The expensive code in the timer interupt will be executed every 497.1 days, so that's bearable.

Quoted 18 lines

Race will be handled by readers of 64bit jiffy. There won't be many of them.

Quoted 11 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+	/* We need to make sure jiffies_high does not change while
+	 * reading jiffies and jiffies_high */
+	do {
+		jiffies_high_tmp = jiffies_high_shadow;
+		barrier();
+		jiffies_tmp = jiffies;
+		barrier();
+	} while (jiffies_high != jiffies_high_tmp);

Maybe this could be condensed into a macro/inline, so that people don't screw it up (and it looks cleaner). Like get_jiffies64() or so, for

Inline! Inline! Don't use macro unless you must!

// extern or static? which is correct?
// I see both types in kernel .h :-(

Quoted 10 lines
extern inline u64 get_jiffies64() {
	unsigned long hi,lo;
	do {
		hi = jiffies_hi;
		barrier();
		lo = jiffies;
		barrier();
	} while (hi != jiffies_hi);
	return lo + (((u64)hi) << 32);
}

should be

	return lo + (((u64)hi) << BITS_PER_LONG);

to not break 64 bit platforms.

Admittedly, I broke 64 bit myself by submitting a wrong version of my patch. The condition

	if (jiffies != 0xffffffffUL) {
		jiffies++;
in do_timer needs to be
	if (jiffies != (unsigned long)-1) {
		jiffies++;
Quoted 6 lines

And guys, why you don't comment on my hand-crafted asm version of union { u32 jiffies; u64 jiffies64; }; ? Is it flawed? --
vda

I believe not, but it is platform-specific.

Tim

#2 lore raw
Quoted 6 lines · Tim Schmielau
Quoted 3 lines
Quoted 2 lines

I would say that the race is so rare that it should not be handled, especially since it adds extra code in the timer interrupt.

The expensive code in the timer interupt will be executed every 497.1 days, so that's bearable.

The expensive code in the timer interrupt will be executed every timer interrupt! There is no magic way to connect two 32-bit long-words to make a 64-bit object, no matter how well it's hidden by the 'C' compiler. An increment of the low long-word won't magically bump the high long-word. This takes code, and the simplist code to do it was shown. This:
adcl $1,(jiffies) # INCL won't set CY 4 clocks
adcl $0,(jiffies_hi) # 4 clocks
... takes 8 clocks.
This:
incl (jiffies)
... takes 2 clocks

Also gcc isn't as "kind" as this. It generates volumes of strange code to bump a 'long long'.

That's 6 extra clocks every Hz or 600 clocks per second. By the time you've reached the 497.1 days, you have wasted.... 0xffffffff/6 = 715,827,882 CPU clocks just so 'uptime' is correct? I don't think so. I'd reboot.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

I was going to compile a list of innovations that could be attributed to Microsoft. Once I realized that Ctrl-Alt-Del was handled in the BIOS, I found that there aren't any.

#3 lore raw
Quoted 4 lines · Richard B. Johnson

That's 6 extra clocks every Hz or 600 clocks per second. By the time you've reached the 497.1 days, you have wasted.... 0xffffffff/6 = 715,827,882 CPU clocks just so 'uptime' is correct? I don't think so. I'd reboot.

Actually, you don't need to reboot.

The mail/dns servers I mentioned are running fine, processing smtp and pop3 mail without a hitch, serving up dns info for 230 dns zones, and providing ntpd and big brother services for quite a few other linux and linux-like systems here.

You only need to reboot if you demand that the uptime counter be correct -

I just add 497 days for now....

cu

jjs

#4 lore raw
In reply to Tim Schmielau
Quoted 5 lines · Richard B. Johnson

That's 6 extra clocks every Hz or 600 clocks per second. By the time you've reached the 497.1 days, you have wasted.... 0xffffffff/6 = 715,827,882 CPU clocks just so 'uptime' is correct? I don't think so. I'd reboot.

For proportion: 716 million CPU clocks, on an "average" PC today is less than one second of CPU time. Over the course of 497 days, that's not much overhead at all.

Charles

Signature · 4 lines
-----------------------------------------------------------------------
Charles Cazabon                            <[email protected]>
GPL'ed software available at:  http://www.qcc.sk.ca/~charlesc/software/
-----------------------------------------------------------------------
#5 lore raw
In reply to Tim Schmielau
Quoted 10 lines · Charles Cazabon
Quoted 5 lines · Richard B. Johnson

That's 6 extra clocks every Hz or 600 clocks per second. By the time you've reached the 497.1 days, you have wasted.... 0xffffffff/6 = 715,827,882 CPU clocks just so 'uptime' is correct? I don't think so. I'd reboot.

For proportion: 716 million CPU clocks, on an "average" PC today is less than one second of CPU time. Over the course of 497 days, that's not much overhead at all.

Not to mention that it takes alot more clocks to setup and return from the interupt.

Kurt

#6 lore raw
In reply to a message not in the archive
Quoted 33 lines · vda
Quoted 6 lines · Andreas Dilger
Quoted 2 lines · Tim Schmielau

The idea was that all drivers that use the 32 bit jiffies counter have to be aware of the wraparound anyways, and won't see a difference.

Agreed. I also like the change that you initialize jiffies to a pre-wrap value, so the jiffies wrap bugs can more easily be found/fixed.

Yes indeed. We should chase bugs, not run away from them :-) I seriously suggest setting juffy to close to wrap value if kernel hacking is enabled.

Quoted 2 lines

I would say that the race is so rare that it should not be handled, especially since it adds extra code in the timer interrupt.

Race will be handled by readers of 64bit jiffy. There won't be many of them.

Quoted 11 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+	/* We need to make sure jiffies_high does not change while
+	 * reading jiffies and jiffies_high */
+	do {
+		jiffies_high_tmp = jiffies_high_shadow;
+		barrier();
+		jiffies_tmp = jiffies;
+		barrier();
+	} while (jiffies_high != jiffies_high_tmp);

Maybe this could be condensed into a macro/inline, so that people don't screw it up (and it looks cleaner). Like get_jiffies64() or so, for

Inline! Inline! Don't use macro unless you must!

// extern or static? which is correct?
// I see both types in kernel .h :-(

Quoted 11 lines
extern inline u64 get_jiffies64() {
	unsigned long hi,lo;
	do {
		hi = jiffies_hi;
		barrier();
		lo = jiffies;
		barrier();
	} while (hi != jiffies_hi);
	return lo + (((u64)hi) << 32);
}

OK, I introduced get_jiffies64, corrected my 64 bit mistake and subtract INITIAL_JIFFIES to obtain uptime, while leaving it at at pre-wrap value for error-chasing.
Still this patch introduces jiffies_high on 64 bit platforms which will be useless until the year 571234830.

************************************************************
* DISCLAIMER: This patch WILL make your linux box unstable *
* unless you set INITIAL_JIFFIES to zero!!! *
************************************************************

btw.: can someone please explain to me why do_timer uses

	(*(unsigned long *)&jiffies)++;

instead of just doing jiffies++ ?

Tim

proc/proc_misc.c renamed from proc/proc_misc.c.orig +13 -8
@@ -39,6 +39,7 @@
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>
 #include <asm/io.h>
+#include <asm/div64.h>
 
 
 #define LOAD_INT(x) ((x) >> FSHIFT)
@@ -103,15 +104,19 @@
 static int uptime_read_proc(char *page, char **start, off_t off,
 				 int count, int *eof, void *data)
 {
-	unsigned long uptime;
+	u64 uptime;
+	unsigned long remainder;
 	unsigned long idle;
 	int len;
 
-	uptime = jiffies;
+	uptime = get_jiffies64() - INITIAL_JIFFIES;
+	remainder = (unsigned long) do_div(uptime, HZ);
+
 	idle = init_tasks[0]->times.tms_utime + init_tasks[0]->times.tms_stime;
 
-	/* The formula for the fraction parts really is ((t * 100) / HZ) % 100, but
-	   that would overflow about every five days at HZ == 100.
+	/* The formula for the fraction part of the idle time really is
+	   ((t * 100) / HZ) % 100, but that would overflow about
+	    every five days at HZ == 100.
 	   Therefore the identity a = (a / b) * b + a % b is used so that it is
 	   calculated as (((t / HZ) * 100) + ((t % HZ) * 100) / HZ) % 100.
 	   The part in front of the '+' always evaluates as 0 (mod 100). All divisions
@@ -121,14 +126,14 @@
 	 */
 #if HZ!=100
 	len = sprintf(page,"%lu.%02lu %lu.%02lu\n",
-		uptime / HZ,
-		(((uptime % HZ) * 100) / HZ) % 100,
+		(unsigned long) uptime,
+		((remainder * 100) / HZ) % 100,
 		idle / HZ,
 		(((idle % HZ) * 100) / HZ) % 100);
 #else
 	len = sprintf(page,"%lu.%02lu %lu.%02lu\n",
-		uptime / HZ,
-		uptime % HZ,
+		(unsigned long) uptime,
+		remainder,
 		idle / HZ,
 		idle % HZ);
 #endif
timer.c renamed from timer.c.orig +20 -4
@@ -65,7 +65,8 @@
 
 extern int do_setitimer(int, struct itimerval *, struct itimerval *);
 
-unsigned long volatile jiffies;
+unsigned long volatile jiffies = INITIAL_JIFFIES;
+unsigned long volatile jiffies_hi, jiffies_hi_shadow;
 
 unsigned int * prof_buffer;
 unsigned long prof_len;
@@ -117,7 +118,7 @@
 		INIT_LIST_HEAD(tv1.vec + i);
 }
 
-static unsigned long timer_jiffies;
+static unsigned long timer_jiffies = INITIAL_JIFFIES;
 
 static inline void internal_add_timer(struct timer_list *timer)
 {
@@ -638,7 +639,7 @@
 }
 
 /* jiffies at the most recent update of wall time */
-unsigned long wall_jiffies;
+unsigned long wall_jiffies = INITIAL_JIFFIES;
 
 /*
  * This spinlock protect us from races in SMP while playing with xtime. -arca
@@ -673,7 +674,22 @@
 
 void do_timer(struct pt_regs *regs)
 {
-	(*(unsigned long *)&jiffies)++;
+	/* we assume that two calls to do_timer can never overlap
+	 * since they are one jiffie apart in time */
+	if (jiffies != (unsigned long)(-1)) {
+		jiffies++;
+	} else {
+		/* We still need to care about the race with readers of
+		 * jiffies_hi. Readers have to discard the values if
+		 * jiffies_hi != jiffies_hi_shadow when read with
+		 * proper barriers in between. */
+		jiffies_hi++;
+		barrier();
+		jiffies++;
+		barrier();
+		jiffies_hi_shadow = jiffies_hi;
+		barrier();
+	}
 #ifndef CONFIG_SMP
 	/* SMP process accounting uses the local APIC timer */
 
info.c renamed from info.c.orig +5 -1
@@ -12,15 +12,19 @@
 #include <linux/smp_lock.h>
 
 #include <asm/uaccess.h>
+#include <asm/div64.h>
 
 asmlinkage long sys_sysinfo(struct sysinfo *info)
 {
 	struct sysinfo val;
+	u64 uptime;
 
 	memset((char *)&val, 0, sizeof(struct sysinfo));
 
 	cli();
-	val.uptime = jiffies / HZ;
+	uptime = get_jiffies64() - INITIAL_JIFFIES;
+	do_div(uptime, HZ);
+	val.uptime = uptime;
 
 	val.loads[0] = avenrun[0] << (SI_LOAD_SHIFT - FSHIFT);
 	val.loads[1] = avenrun[1] << (SI_LOAD_SHIFT - FSHIFT);
linux/sched.h renamed from linux/sched.h.orig +15 -1
@@ -543,7 +543,21 @@
 
 #include <asm/current.h>
 
-extern unsigned long volatile jiffies, jiffies_high, jiffies_high_shadow;
+#define INITIAL_JIFFIES 0xFFFFD000ul
+extern unsigned long volatile jiffies, jiffies_hi, jiffies_hi_shadow;
+static inline u64 get_jiffies64() {
+	unsigned long hi,lo;
+	/* We need to make sure jiffies_hi does not change while
+	 * reading jiffies and jiffies_hi */
+	do {
+	        hi = jiffies_hi;
+	        barrier();
+	        lo = jiffies;
+	        barrier();
+	} while (hi != jiffies_hi);
+	return lo + (((u64)hi) << BITS_PER_LONG);
+}
+
 extern unsigned long itimer_ticks;
 extern unsigned long itimer_next;
 extern struct timeval xtime;
#9 lore raw
In reply to a message not in the archive
Quoted 5 lines · Tim Schmielau

OK, I introduced get_jiffies64, corrected my 64 bit mistake and subtract INITIAL_JIFFIES to obtain uptime, while leaving it at at pre-wrap value for error-chasing.
Still this patch introduces jiffies_high on 64 bit platforms which will be useless until the year 571234830.

It probably won't be accepted until the whole thing disappears for 64-bit systems.

Quoted 3 lines

btw.: can someone please explain to me why do_timer uses

	(*(unsigned long *)&jiffies)++;

instead of just doing jiffies++ ?

Can't say for sure, but it may be a compiler issue, maybe historical.

Review

Quoted 9 lines
(quoted diff) +2 -2
 	 */
 #if HZ!=100
 	len = sprintf(page,"%lu.%02lu %lu.%02lu\n",
-		uptime / HZ,
-		(((uptime % HZ) * 100) / HZ) % 100,
+		(unsigned long) uptime,
+		((remainder * 100) / HZ) % 100,
 		idle / HZ,
 		(((idle % HZ) * 100) / HZ) % 100);

Given that uptime is now 64-bits, do we need this mathematical hoop jumping? Also, won't the remainder already be modulus HZ in this case?

Quoted 9 lines

void do_timer(struct pt_regs *regs)
{

(quoted diff) +6 -1
-	(*(unsigned long *)&jiffies)++;
+	/* we assume that two calls to do_timer can never overlap
+	 * since they are one jiffie apart in time */
+	if (jiffies != (unsigned long)(-1)) {
+		jiffies++;
+	} else {
+		/* We still need to care about the race with readers of

Review

Quoted 10 lines
(quoted diff) +10 -0
+		 * jiffies_hi. Readers have to discard the values if
+		 * jiffies_hi != jiffies_hi_shadow when read with
+		 * proper barriers in between. */
+		jiffies_hi++;
+		barrier();
+		jiffies++;
+		barrier();
+		jiffies_hi_shadow = jiffies_hi;
+		barrier();
+	}

I think this is the part of the patch that people object to. See my other posting of how to handle 64-bit jiffies with only an impact to users of the 64-bit value.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#7 lore raw
In reply to Tim Schmielau
Quoted 2 lines · Tim Schmielau

On Wed, 31 Oct 2001, vda wrote:

[SNIPPED...]

Quoted 6 lines

btw.: can someone please explain to me why do_timer uses

	(*(unsigned long *)&jiffies)++;

instead of just doing jiffies++ ?

Tim

It's an attempt to prevent the 'C' compiler from doing this:

	movl	(jiffies), %eax   # Read
	incl	%eax              # Modify
	movl	%eax, (jiffies)   # Write

Instead, we want it to do this:

movl jiffies, %ebx # init a pointer
incl (%ebx) # Modify directly

With some gcc versions, it works. Others, it doesn't hurt.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

I was going to compile a list of innovations that could be attributed to Microsoft. Once I realized that Ctrl-Alt-Del was handled in the BIOS, I found that there aren't any.

#8 lore raw
In reply to Tim Schmielau
Quoted 3 lines · Tim Schmielau

btw.: can someone please explain to me why do_timer uses

	(*(unsigned long *)&jiffies)++;

instead of just doing jiffies++ ?

This casts away volatile -> gcc generates potentially faster code

timer.c line 69

Quoted 3 lines
timer.c renamed from timer.c.orig +2 -1
-unsigned long volatile jiffies;
+unsigned long volatile jiffies = INITIAL_JIFFIES;
+unsigned long volatile jiffies_hi, jiffies_hi_shadow;

Grepped your patch for uses of jiffies_hi_shadow, found none except for stores. Seems to be unused?

Quoted 9 lines

void do_timer(struct pt_regs *regs)
{

timer.c renamed from timer.c.orig +6 -1
-	(*(unsigned long *)&jiffies)++;
+	/* we assume that two calls to do_timer can never overlap
+	 * since they are one jiffie apart in time */
+	if (jiffies != (unsigned long)(-1)) {
+		jiffies++;
+	} else {
+		/* We still need to care about the race with readers of

timer.c line 692

Quoted 10 lines
timer.c renamed from timer.c.orig +10 -0
+		 * jiffies_hi. Readers have to discard the values if
+		 * jiffies_hi != jiffies_hi_shadow when read with
+		 * proper barriers in between. */
+		jiffies_hi++;
+		barrier();
+		jiffies++;
+		barrier();
+		jiffies_hi_shadow = jiffies_hi;
+		barrier();
+	}

Looks like gross overkill for 64bit i++. I see no flaw in much simpler:
+ if(++jiffies == 0) {
+ /* barrier(); --vda: needed? I have some doubts... */
+ jiffies_hi++;
+ }

To avoid races 64bit readers must read in reverse order: hi,lo,check hi; your patch already does this just right:

Quoted 7 lines
linux/sched.h renamed from linux/sched.h.orig +7 -0
+static inline u64 get_jiffies64() {
+	unsigned long hi,lo;
+	/* We need to make sure jiffies_hi does not change while
+	 * reading jiffies and jiffies_hi */
+	do {
+	        hi = jiffies_hi;
+	        barrier();

linux/sched.h

Quoted 10 lines
linux/sched.h renamed from linux/sched.h.orig +8 -0
+	        lo = jiffies;
+	        barrier();
+	} while (hi != jiffies_hi);
+	return lo + (((u64)hi) << BITS_PER_LONG);
+}
+
 
 
+#define INITIAL_JIFFIES 0xFFFFD000ul
+extern unsigned long volatile jiffies, jiffies_hi, jiffies_hi_shadow;

This belongs to some .h file, not .c (most likely include/linux/sched.h)

Also consider

+#if defined(CONFIG_DEBUG_KERNEL)
+/* Rollover in 1000 secs */
+#define INITIAL_JIFFIES ((unsigned long) -1000*HZ) +#else
+#define INITIAL_JIFFIES 0
+#endif

Hope your patch will go into mainline soon!
--
vda