Skip to contentSkip to conversation

More breakage in native_rdtsc out of line in git-x86

PATCH Andi Kleen 8 participants 18 messages Jan 9, 2008 – Jan 11, 2008 x86

lore
Share & tools
lore

Andi Kleen

author
#0 lore raw

I had some boot failures here with git-x86 with init and hotplug all segfaulting early on userland with new glibc. Bisecting found

commit 6aea5bc37fa790eaf3a942f0785985914568e214

Author: Ingo Molnar <[email protected]>
Date:   Sat Jan 5 13:27:08 2008 +0100

x86: move native_read_tsc() offline

move native_read_tsc() offline.

I think the problem is that the vsyscall/vdso code calls it through vread and for that it has to be exported. There seems to be also another bug with the old style vsyscalls not using the TSC vread that masks it on older glibc

Stepping with gdb through old style vgettimeofday() confirms that RDTSC is not used.

A long time ago we had a similar problem once and it was because of a problem exporting the vsyscall variables in vmlinux.lds.S -- looks like that has reappeared.

I think the new glibc shows it because it uses the vDSO not the older vsyscall and the new vDSO probably still works. Anyways haven't investigated why that is in detail yet, but that's a separate regression.

Back to the boot failure:

Unfortunately simply adding __vsyscall_fn to native_read_tsc doesn't work -- causes early kernel faults like

PANIC: early exception rip ffffffffff600105 error 10 cr2 ffffffffff600105
Pid: 0, comm: swapper Not tainted 2.6.24-rc6 #58

Call Trace:
 [<ffffffff80211dec>] native_sched_clock+0x9/0x3f
 [<ffffffff8022b758>] init_idle+0x33/0xd1
 [<ffffffff80825449>] sched_init+0x26d/0x283
 [<ffffffff80815899>] start_kernel+0x10b/0x2bd
 [<ffffffff80815114>] _sinittext+0x114/0x11b

Not sure why that is -- in theory the vsyscall functions should be callable from the main kernel. Might be a binutils problem or another code regression.

Anyways it looks like the only good fix is to either revert that or fork into two functions one for vread() and another for normal tsc ->read()

This is all in addition to the problem of it having incorrect barriers. I note that my original patch didn't have any of these problems.

I'm using the appended revert patch here as a workaround for now.

-Andi

Revert rdtsc out of line change

Reverts

commit 6aea5bc37fa790eaf3a942f0785985914568e214

Author: Ingo Molnar <[email protected]>
Date:   Sat Jan 5 13:27:08 2008 +0100

x86: move native_read_tsc() offline

move native_read_tsc() offline.

The function is called by vsyscalls in ring 3, so it can't be out of line this way.

arch/x86/kernel/rtc.c +0 -11
Index: linux/arch/x86/kernel/rtc.c
===================================================================
@@ -194,14 +194,3 @@ int update_persistent_clock(struct times
 {
 	return set_rtc_mmss(now.tv_sec);
 }
-
-unsigned long long native_read_tsc(void)
-{
-	DECLARE_ARGS(val, low, high);
-
-	asm volatile("rdtsc" : EAX_EDX_RET(val, low, high));
-	rdtsc_barrier();
-
-	return EAX_EDX_VAL(val, low, high);
-}
-
include/asm-x86/msr.h +7 -1
Index: linux/include/asm-x86/msr.h
===================================================================
@@ -91,7 +91,13 @@ static inline int native_write_msr_safe(
 	return err;
 }
 
-extern unsigned long long native_read_tsc(void);
+static inline unsigned long long native_read_tsc(void)
+{
+	DECLARE_ARGS(val, low, high);
+
+	asm volatile("rdtsc" : EAX_EDX_RET(val, low, high));
+	return EAX_EDX_VAL(val, low, high);
+}
 
 static inline unsigned long long native_read_pmc(int counter)
 {
author
#1 lore raw

Re: More breakage in native_rdtsc out of line in git-x86 II

Quoted 8 lines

I think the problem is that the vsyscall/vdso code calls it through vread and for that it has to be exported. There seems to be also another bug with the old style vsyscalls not using the TSC vread that masks it on older glibc

Stepping with gdb through old style vgettimeofday() confirms that RDTSC is not used.

Quoted 8 lines

A long time ago we had a similar problem once and it was because of a problem exporting the vsyscall variables in vmlinux.lds.S -- looks like that has reappeared.

I think the new glibc shows it because it uses the vDSO not the older vsyscall and the new vDSO probably still works. Anyways haven't investigated why that is in detail yet, but that's a separate regression.

Actually that seems to be because the test system using the older glibc didn't use the TSC because it was marked unstable due a unsynchronized TSC. It should not have been -- this is a Core2 dual core single socket. Will investigate later what happened there.

-Andi

#2 lore raw
In reply to Andi Kleen
Quoted 13 lines · Andi Kleen

Unfortunately simply adding __vsyscall_fn to native_read_tsc doesn't work -- causes early kernel faults like

PANIC: early exception rip ffffffffff600105 error 10 cr2 ffffffffff600105
Pid: 0, comm: swapper Not tainted 2.6.24-rc6 #58

Call Trace:
 [<ffffffff80211dec>] native_sched_clock+0x9/0x3f
 [<ffffffff8022b758>] init_idle+0x33/0xd1
 [<ffffffff80825449>] sched_init+0x26d/0x283
 [<ffffffff80815899>] start_kernel+0x10b/0x2bd
 [<ffffffff80815114>] _sinittext+0x114/0x11b
Quoted 3 lines

Not sure why that is -- in theory the vsyscall functions should be callable from the main kernel. Might be a binutils problem or another code regression.

nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an __initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?

Ingo

arch/x86/kernel/rtc.c +1 -1
Index: linux-x86.q/arch/x86/kernel/rtc.c
===================================================================
@@ -195,7 +195,7 @@ int update_persistent_clock(struct times
 	return set_rtc_mmss(now.tv_sec);
 }
 
-unsigned long long native_read_tsc(void)
+unsigned long long __vsyscall_fn native_read_tsc(void)
 {
 	DECLARE_ARGS(val, low, high);
 
arch/x86/kernel/setup_64.c +2 -0
Index: linux-x86.q/arch/x86/kernel/setup_64.c
===================================================================
@@ -46,6 +46,7 @@
 #include <asm/mtrr.h>
 #include <asm/uaccess.h>
 #include <asm/system.h>
+#include <asm/vsyscall.h>
 #include <asm/io.h>
 #include <asm/smp.h>
 #include <asm/msr.h>
@@ -464,6 +465,7 @@ void __init setup_arch(char **cmdline_p)
 #endif
 	reserve_crashkernel();
 	paging_init();
+	map_vsyscall();
 
 	early_quirks();
 
arch/x86/kernel/vsyscall_64.c +1 -2
Index: linux-x86.q/arch/x86/kernel/vsyscall_64.c
===================================================================
@@ -319,7 +319,7 @@ cpu_vsyscall_notifier(struct notifier_bl
 	return NOTIFY_DONE;
 }
 
-static void __init map_vsyscall(void)
+void __init map_vsyscall(void)
 {
 	extern char __vsyscall_0;
 	unsigned long physaddr_page0 = __pa_symbol(&__vsyscall_0);
@@ -335,7 +335,6 @@ static int __init vsyscall_init(void)
 	BUG_ON((unsigned long) &vtime != VSYSCALL_ADDR(__NR_vtime));
 	BUG_ON((VSYSCALL_ADDR(0) != __fix_to_virt(VSYSCALL_FIRST_PAGE)));
 	BUG_ON((unsigned long) &vgetcpu != VSYSCALL_ADDR(__NR_vgetcpu));
-	map_vsyscall();
 #ifdef CONFIG_SYSCTL
 	register_sysctl_table(kernel_root_table2);
 #endif
author
#3 lore raw
Quoted 3 lines

nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an __initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?

That gives

/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'

But after I add a prototype to vsyscall.h it seems to work.

Just the barriers are still broken -- you have not replied to any of my emails on that topic. What's up with that?

-Andi

#4 lore raw
Quoted 9 lines · Andi Kleen
Quoted 3 lines

nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an __initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?

That gives

/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'

i guess you have an old repository.

Quoted 3 lines

But after I add a prototype to vsyscall.h it seems to work.

Just the barriers are still broken [...]

since yesterday there's a full barrier around rdtsc.

Ingo

author
#5 lore raw
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Andi Kleen
Quoted 3 lines

nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an __initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?

Quoted 7 lines
Quoted 5 lines

That gives

/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'

i guess you have an old repository.

I just applied the patch you sent out against yesterday's git-x86.

Quoted 6 lines
Quoted 4 lines

But after I add a prototype to vsyscall.h it seems to work.

Just the barriers are still broken [...]

since yesterday there's a full barrier around rdtsc.

Great.

-Andi

#6 lore raw
Quoted 10 lines · Andi Kleen
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Andi Kleen
Quoted 3 lines

nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an __initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?

That gives

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'

i guess you have an old repository.

I just applied the patch you sent out against yesterday's git-x86.

then you have a truly ancient x86.git repository ;-)

think of x86.git#mm as an open development tree. It's high-flux, based against Linux-bleeding-edge, it's frequently updated (daily, sometimes hourly), breakages are possible (and likely) and fixes and other feedback is more than welcome. And please feel free to complain about patches that are included. (like you did in the past) Also please try to post your patches as early as possible instead of in big chunks - last week's 75 patches patchbomb from you was (and still is) ... challenging ;-)

Quoted 3 lines
Quoted 1 line

since yesterday there's a full barrier around rdtsc.

Great.

i have measured the impact of the barriers and it was in the noise level. Barriers are notoriously easy to get wrong (because almost nothing tells the programmer that they are wrong), that's why i did this barrier-safe rdtsc() [& friends]. We had so much trouble with RDTSC during the past 10 years of its existence that being a bit more conservative with it is the only really maintainable option.

Ingo

author
#7 lore raw
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Andi Kleen
Quoted 13 lines · Ingo Molnar
Quoted 9 lines · Andi Kleen
Quoted 3 lines

nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an __initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?

That gives

/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

i guess you have an old repository.

I just applied the patch you sent out against yesterday's git-x86.

then you have a truly ancient x86.git repository ;-)

I update only infrequently because frankly git's remote branch tracking is a mess. At least it doesn't really work for x86#mm here.

I usually have to blow away the repository and reclone to get back to a sane state.

Quoted 5 lines

think of x86.git#mm as an open development tree. It's high-flux, based against Linux-bleeding-edge, it's frequently updated (daily, sometimes hourly), breakages are possible (and likely) and fixes and other feedback is more than welcome. And please feel free to complain about

Right now Jan's cpa change you merged makes the cpa selftest to not pass anymore on 32bit. While that was likely a latent bug it just triggered it's a little inconvenient.

There's also the problem on NX handling still not being right.

Quoted 3 lines

patches that are included. (like you did in the past) Also please try to post your patches as early as possible instead of in big chunks - last week's 75 patches patchbomb from you was (and still is) ... challenging

There are still quite a lot outstanding.

% wc -l patches/series
90 patches/series

Some of that is debug or tbd or obsolete, but most isn't.

Quoted 10 lines
Quoted 3 lines
Quoted 1 line

since yesterday there's a full barrier around rdtsc.

Great.

i have measured the impact of the barriers and it was in the noise level. Barriers are notoriously easy to get wrong (because almost nothing tells the programmer that they are wrong), that's why i did this barrier-safe rdtsc() [& friends]. We had so much trouble with RDTSC during the past 10 years of its existence that being a bit more conservative with it is the only really maintainable option.

I would still think that the explicit barriers would make this all clearer, with long term giving cleaner semantics.

Admittedly I should have written some more comments in the code.

I'm also a little unhappy on how many function calls the vgtod() fast path contains now. Long ago it was a really lean function, but it gets messed up more and more. Back when ->vread was introduced the latency already suffered significantly (iirc multi digit percent range), i suspect it now got worse again.

And after all that's still by far the most common system call (not only for databases; i profiled this using systemtap in some loads some time ago and it usually came up with >50%) and quite important for many workloads.

We have a lot far uglier code for less gain in far less critical calls.

-Andi

#8 lore raw

On Wed, 9 Jan 2008 18:48:00 +0100

Quoted 5 lines

And after all that's still by far the most common system call (not only for databases; i profiled this using systemtap in some loads some time ago and it usually came up with >50%) and quite important for many workloads.

btw be careful with this; the X server uses gettimeofday in it's equivalent of udelay()... (and we all know how useful it is to make the delay loops faster ;-)

author
#9 lore raw
Quoted 9 lines · Arjan van de Ven

On Wed, 9 Jan 2008 18:48:00 +0100

Quoted 5 lines

And after all that's still by far the most common system call (not only for databases; i profiled this using systemtap in some loads some time ago and it usually came up with >50%) and quite important for many workloads.

btw be careful with this; the X server uses gettimeofday in it's equivalent of udelay()... (and we all know how useful it is to make the delay loops faster ;-)

People tend to make jokes about optimizing the idle loop too, but they're actually wrong. Exit latency for the idle loop is important -- it decides how quickly you can react to load changes on idle CPUs.

For short udelays I suspect shorter exit latency is also moderately useful.

But anyways there are plenty of gtod users outside the X server.

e.g. common not user space case is packet timestamps.

-Andi

#10 lore raw
Quoted 10 lines · Andi Kleen

People tend to make jokes about optimizing the idle loop too, but they're
actually wrong. Exit latency for the idle loop is important -- it decides how quickly you can react to load changes on idle CPUs.

For short udelays I suspect shorter exit latency is also moderately useful.

But anyways there are plenty of gtod users outside the X server.

e.g. common not user space case is packet timestamps.

It's also very common to put gtod calls around select and poll, because the standard interface doesn't tell you how long you have been away (Linux at least used to have it, but glibc "fixed" that...)

-hpa

#11 lore raw
In reply to Andi Kleen
Quoted 8 lines · Andi Kleen
Quoted 1 line · Ingo Molnar

then you have a truly ancient x86.git repository ;-)

I update only infrequently because frankly git's remote branch tracking is a mess. At least it doesn't really work for x86#mm here.

I usually have to blow away the repository and reclone to get back to a sane state.

Someone in #git had a similar problem today. Conclusion was that x86/mm is not "stable" in the sense that commits are only added, instead history gets rewritten. That breaks pull/merge/"basic rebase".

Basically, you'll want to "rebase --onto", taking your local commits from the old branch history to the new branch history. One way to make that bearable is to have two branches (or a branch and a tag). One branch is used to keep your work, the other branch (or tag) is used to "mark" where the old upstream ended and your work started.

Assuming that your remote is called "x86", this could look like this:

git branch myStuff x86/mm
git branch myStuff_start x86/mm

work work work commit commit commit

git fetch x86 mm

git rebase --onto x86/mm myStuff_start myStuff
git branch -f myStuff_start x86/mm

The rebase will take all of the commits that you added to myStuff, on top of the old x86/mm (referred to by myStuff_start) and try to apply them on top of the new x86/mm, updating myStuff to point to the rebased commits afterwards.

Then myStuff_start is updated to point to the now current start of your stuff, so that you can do the rebase again later.

That's not actually a problem of git's tracking branches, because they're not made for that style of work at all. For such stuff, rebase is the only useful option AFAIK.

HTH,
Björn

author
#12 lore raw
Quoted 7 lines · Björn Steinbrink
Quoted 6 lines · Andi Kleen
Quoted 1 line · Ingo Molnar

then you have a truly ancient x86.git repository ;-)

I update only infrequently because frankly git's remote branch tracking is a mess. At least it doesn't really work for x86#mm here.

Quoted 8 lines
Quoted 2 lines

I usually have to blow away the repository and reclone to get back to a sane state.

Someone in #git had a similar problem today. Conclusion was that x86/mm is not "stable" in the sense that commits are only added, instead history gets rewritten. That breaks pull/merge/"basic rebase".

Basically, you'll want to "rebase --onto", taking your local commits

I don't really have any local commits. I just want to use read-only git to generate a patch that I can then import using quilt and only look at log files and use git show. The fanciest thing I use is git bisect occasionally.

Quoted 9 lines

that bearable is to have two branches (or a branch and a tag). One branch is used to keep your work, the other branch (or tag) is used to "mark" where the old upstream ended and your work started.

Assuming that your remote is called "x86", this could look like this:

git branch myStuff x86/mm
git branch myStuff_start x86/mm

Quoted 6 lines

work work work commit commit commit

git fetch x86 mm

git rebase --onto x86/mm myStuff_start myStuff
git branch -f myStuff_start x86/mm

Do you have a simple recipe to just update from the the remote branch, assuming there are no local changes or local branches?

-Andi

#13 lore raw
Quoted 4 lines · Andi Kleen

Do you have a simple recipe to just update from the the remote branch, assuming there are no local changes or local branches?

-Andi

For staying up to date I use the following:

# Add Linus's tree as a remote
git remote --add linus
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

# Add Ingo's tree as a remote
git remote --add x86
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

# With that setup, just run the following to get any changes you # don't have. It will also notice any new branches Ingo/Linus

# add to their repo.  Look in .git/config afterwards, the format
# to add new remotes is easy to figure out.
git remote update

Cheers,

Harvey

author
#14 lore raw
Quoted 17 lines · Harvey Harrison
Quoted 4 lines · Andi Kleen

Do you have a simple recipe to just update from the the remote branch, assuming there are no local changes or local branches?

-Andi

For staying up to date I use the following:

# Add Linus's tree as a remote
git remote --add linus
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

# Add Ingo's tree as a remote
git remote --add x86
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

Quoted 5 lines

# With that setup, just run the following to get any changes you # don't have. It will also notice any new branches Ingo/Linus

# add to their repo.  Look in .git/config afterwards, the format
# to add new remotes is easy to figure out.
git remote update

I'm already cloning the branches; the problem is not getting conflicts etc. when updating.

-Andi

#15 lore raw
Quoted 18 lines · Andi Kleen
Quoted 17 lines · Harvey Harrison
Quoted 4 lines · Andi Kleen

Do you have a simple recipe to just update from the the remote branch, assuming there are no local changes or local branches?

-Andi

For staying up to date I use the following:

# Add Linus's tree as a remote
git remote --add linus
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

# Add Ingo's tree as a remote
git remote --add x86
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git

Quoted 8 lines
Quoted 5 lines

# With that setup, just run the following to get any changes you # don't have. It will also notice any new branches Ingo/Linus

# add to their repo.  Look in .git/config afterwards, the format
# to add new remotes is easy to figure out.
git remote update

I'm already cloning the branches; the problem is not getting conflicts etc. when updating.

I guess you're using "git pull" to update a local branch? That will try to merge the new state of x86/mm into your local branch, and that breaks. If you just want to have a local branch that gives you a "fixed" state of x86/mm regardless of whether or not you already fetched newer ones, you can do:

git branch myThing x86/mm # create the branch

work/test/whatever

To fetch a new "state" from the remote:
git fetch x86/mm # or git remote update, or whatever

To update your branch to point to the new state:
git branch - f myThing x86/mm

That basically replaces it with a new branch of the same name, but pointing to the new x86/mm.

Or if you want to get your working tree to that state at the same time, you can also do:
git checkout myThing
git reset --hard x86/mm

Björn

#16 lore raw
In reply to Andi Kleen
Quoted 2 lines · Andi Kleen

I'm already cloning the branches; the problem is not getting conflicts etc. when updating.

Isn't git pull --force what you are looking for?

Ciao,

Signature · 2 lines
Paolo
http://paolo.ciarrocchi.googlepages.com/
#17 lore raw
Quoted 5 lines · Paolo Ciarrocchi
Quoted 2 lines · Andi Kleen

I'm already cloning the branches; the problem is not getting conflicts etc. when updating.

Quoted 7 lines

Isn't git pull --force what you are looking for?

Ciao,
--
Paolo
http://paolo.ciarrocchi.googlepages.com/

Actually...no, it wouldn't help. Conflicts will coming up and only the solution for now is to use remote tracking as Harvey wrote in prev mail. (btw, git pull that would not conflict is when it called with '-s ours' but that is not good)

- Cyrill -