More breakage in native_rdtsc out of line in git-x86
I had some boot failures here with git-x86 with init and hotplug all segfaulting early on userland with new glibc. Bisecting found
commit 6aea5bc37fa790eaf3a942f0785985914568e214
Author: Ingo Molnar <[email protected]> Date: Sat Jan 5 13:27:08 2008 +0100
x86: move native_read_tsc() offline
move native_read_tsc() offline.
I think the problem is that the vsyscall/vdso code calls it through vread and for that it has to be exported. There seems to be also another bug with the old style vsyscalls not using the TSC vread that masks it on older glibc
Stepping with gdb through old style vgettimeofday() confirms that RDTSC is not used.
A long time ago we had a similar problem once and it was because of a problem exporting the vsyscall variables in vmlinux.lds.S -- looks like that has reappeared.
I think the new glibc shows it because it uses the vDSO not the older vsyscall and the new vDSO probably still works. Anyways haven't investigated why that is in detail yet, but that's a separate regression.
Back to the boot failure:
Unfortunately simply adding __vsyscall_fn to native_read_tsc doesn't work -- causes early kernel faults like
PANIC: early exception rip ffffffffff600105 error 10 cr2 ffffffffff600105
Pid: 0, comm: swapper Not tainted 2.6.24-rc6 #58
Call Trace: [<ffffffff80211dec>] native_sched_clock+0x9/0x3f [<ffffffff8022b758>] init_idle+0x33/0xd1 [<ffffffff80825449>] sched_init+0x26d/0x283 [<ffffffff80815899>] start_kernel+0x10b/0x2bd [<ffffffff80815114>] _sinittext+0x114/0x11b
Not sure why that is -- in theory the vsyscall functions should be callable from the main kernel. Might be a binutils problem or another code regression.
Anyways it looks like the only good fix is to either revert that or fork into two functions one for vread() and another for normal tsc ->read()
This is all in addition to the problem of it having incorrect barriers. I note that my original patch didn't have any of these problems.
I'm using the appended revert patch here as a workaround for now.
-Andi
Revert rdtsc out of line change
Reverts
commit 6aea5bc37fa790eaf3a942f0785985914568e214
Author: Ingo Molnar <[email protected]> Date: Sat Jan 5 13:27:08 2008 +0100
x86: move native_read_tsc() offline
move native_read_tsc() offline.
The function is called by vsyscalls in ring 3, so it can't be out of line this way.
arch/x86/kernel/rtc.c +0 -11
@@ -194,14 +194,3 @@ int update_persistent_clock(struct times {return set_rtc_mmss (now.tv_sec); } - -unsigned long long native_read_tsc (void ) -{ -DECLARE_ARGS (val, low, high); - - asm volatile("rdtsc" : EAX_EDX_RET(val, low, high)); -rdtsc_barrier (); - -return EAX_EDX_VAL (val, low, high); -} -
include/asm-x86/msr.h +7 -1
@@ -91,7 +91,13 @@ static inline int native_write_msr_safe(return err; } -extern unsigned long long native_read_tsc (void ); +static inline unsigned long long native_read_tsc (void ) +{ +DECLARE_ARGS (val, low, high); + + asm volatile("rdtsc" : EAX_EDX_RET(val, low, high)); +return EAX_EDX_VAL (val, low, high); +}static inline unsigned long long native_read_pmc (int counter) {
Re: More breakage in native_rdtsc out of line in git-x86 II
Quoted 8 lines
I think the problem is that the vsyscall/vdso code calls it through vread and for that it has to be exported. There seems to be also another bug with the old style vsyscalls not using the TSC vread that masks it on older glibc
Stepping with gdb through old style
vgettimeofday()confirms that RDTSC is not used.
Quoted 8 lines
A long time ago we had a similar problem once and it was because of a problem exporting the vsyscall variables in
vmlinux.lds.S-- looks like that has reappeared.I think the new glibc shows it because it uses the vDSO not the older vsyscall and the new vDSO probably still works. Anyways haven't investigated why that is in detail yet, but that's a separate regression.
Actually that seems to be because the test system using the older glibc didn't use the TSC because it was marked unstable due a unsynchronized TSC. It should not have been -- this is a Core2 dual core single socket. Will investigate later what happened there.
-Andi
Quoted 13 lines · Andi Kleen
Unfortunately simply adding
__vsyscall_fntonative_read_tscdoesn't work -- causes early kernel faults likePANIC: early exception rip
ffffffffff600105error 10 cr2ffffffffff600105
Pid: 0, comm: swapper Not tainted 2.6.24-rc6 #58Call Trace: [<ffffffff80211dec>] native_sched_clock+0x9/0x3f [<ffffffff8022b758>] init_idle+0x33/0xd1 [<ffffffff80825449>] sched_init+0x26d/0x283 [<ffffffff80815899>] start_kernel+0x10b/0x2bd [<ffffffff80815114>] _sinittext+0x114/0x11b
Quoted 3 lines
Not sure why that is -- in theory the vsyscall functions should be callable from the main kernel. Might be a binutils problem or another code regression.
nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an __initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?
Ingo
arch/x86/kernel/rtc.c +1 -1
@@ -195,7 +195,7 @@ int update_persistent_clock(struct timesreturn set_rtc_mmss (now.tv_sec); } -unsigned long long native_read_tsc (void ) +unsigned long long __vsyscall_fn native_read_tsc (void ) {DECLARE_ARGS (val, low, high);
arch/x86/kernel/setup_64.c +2 -0
@@ -46,6 +46,7 @@#include <asm/mtrr.h> #include <asm/uaccess.h> #include <asm/system.h> +#include <asm/vsyscall.h> #include <asm/io.h> #include <asm/smp.h> #include <asm/msr.h>
@@ -464,6 +465,7 @@ void __init setup_arch(char **cmdline_p)#endif reserve_crashkernel ();paging_init (); +map_vsyscall ();early_quirks ();
arch/x86/kernel/vsyscall_64.c +1 -2
@@ -319,7 +319,7 @@ cpu_vsyscall_notifier(struct notifier_blreturn NOTIFY_DONE ; } -static void __init map_vsyscall (void ) +void __init map_vsyscall (void ) {extern char __vsyscall_0;unsigned long physaddr_page0 =__pa_symbol (&__vsyscall_0);
@@ -335,7 +335,6 @@ static int __init vsyscall_init(void)BUG_ON ((unsigned long ) &vtime !=VSYSCALL_ADDR (__NR_vtime));BUG_ON ((VSYSCALL_ADDR (0 ) !=__fix_to_virt (VSYSCALL_FIRST_PAGE )));BUG_ON ((unsigned long ) &vgetcpu !=VSYSCALL_ADDR (__NR_vgetcpu)); -map_vsyscall ();#ifdef CONFIG_SYSCTL register_sysctl_table (kernel_root_table2);#endif
Quoted 3 lines
nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an
__initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?
That gives
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'
But after I add a prototype to vsyscall.h it seems to work.
Just the barriers are still broken -- you have not replied to any of my emails on that topic. What's up with that?
-Andi
Quoted 9 lines · Andi Kleen
Quoted 3 lines
nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an
__initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?That gives
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'
i guess you have an old repository.
Quoted 3 lines
But after I add a prototype to
vsyscall.hit seems to work.Just the barriers are still broken [...]
since yesterday there's a full barrier around rdtsc.
Ingo
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Andi Kleen
Quoted 3 lines
nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an
__initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?
Quoted 7 lines
Quoted 5 lines
That gives
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'i guess you have an old repository.
I just applied the patch you sent out against yesterday's git-x86.
Quoted 6 lines
Quoted 4 lines
But after I add a prototype to
vsyscall.hit seems to work.Just the barriers are still broken [...]
since yesterday there's a full barrier around rdtsc.
Great.
-Andi
Quoted 10 lines · Andi Kleen
Quoted 9 lines · Ingo Molnar
Quoted 6 lines · Andi Kleen
Quoted 3 lines
nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an
__initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?That gives
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'i guess you have an old repository.
I just applied the patch you sent out against yesterday's git-x86.
then you have a truly ancient x86.git repository ;-)
think of x86.git#mm as an open development tree. It's high-flux, based against Linux-bleeding-edge, it's frequently updated (daily, sometimes hourly), breakages are possible (and likely) and fixes and other feedback is more than welcome. And please feel free to complain about patches that are included. (like you did in the past) Also please try to post your patches as early as possible instead of in big chunks - last week's 75 patches patchbomb from you was (and still is) ... challenging ;-)
Quoted 3 lines
Quoted 1 line
since yesterday there's a full barrier around rdtsc.
Great.
i have measured the impact of the barriers and it was in the noise level. Barriers are notoriously easy to get wrong (because almost nothing tells the programmer that they are wrong), that's why i did this barrier-safe rdtsc() [& friends]. We had so much trouble with RDTSC during the past 10 years of its existence that being a bit more conservative with it is the only really maintainable option.
Ingo
Quoted 17 lines · Ingo Molnar
Quoted 14 lines · Andi Kleen
Quoted 13 lines · Ingo Molnar
Quoted 9 lines · Andi Kleen
Quoted 3 lines
nope, it's a 64-bit setup/dependency bug/problem: the vsyscall mappings are installed via an
__initcall, and that's too late for early use. The combo patch below fixed the crash for me, does it work on your box too?That gives
/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c: In function 'setup_arch':/home/lsrc/quilt/linux/arch/x86/kernel/setup_64.c:468: error: implicit declarati on of function 'map_vsyscall'
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
i guess you have an old repository.
I just applied the patch you sent out against yesterday's git-x86.
then you have a truly ancient x86.git repository ;-)
I update only infrequently because frankly git's remote branch tracking is a mess. At least it doesn't really work for x86#mm here.
I usually have to blow away the repository and reclone to get back to a sane state.
Quoted 5 lines
think of x86.git#mm as an open development tree. It's high-flux, based against Linux-bleeding-edge, it's frequently updated (daily, sometimes hourly), breakages are possible (and likely) and fixes and other feedback is more than welcome. And please feel free to complain about
Right now Jan's cpa change you merged makes the cpa selftest to not pass anymore on 32bit. While that was likely a latent bug it just triggered it's a little inconvenient.
There's also the problem on NX handling still not being right.
Quoted 3 lines
patches that are included. (like you did in the past) Also please try to post your patches as early as possible instead of in big chunks - last week's 75 patches patchbomb from you was (and still is) ... challenging
There are still quite a lot outstanding.
% wc -l patches/series
90 patches/series
Some of that is debug or tbd or obsolete, but most isn't.
Quoted 10 lines
Quoted 3 lines
Quoted 1 line
since yesterday there's a full barrier around rdtsc.
Great.
i have measured the impact of the barriers and it was in the noise level. Barriers are notoriously easy to get wrong (because almost nothing tells the programmer that they are wrong), that's why i did this barrier-safe
rdtsc()[& friends]. We had so much trouble with RDTSC during the past 10 years of its existence that being a bit more conservative with it is the only really maintainable option.
I would still think that the explicit barriers would make this all clearer, with long term giving cleaner semantics.
Admittedly I should have written some more comments in the code.
I'm also a little unhappy on how many function calls the vgtod() fast path contains now. Long ago it was a really lean function, but it gets messed up more and more. Back when ->vread was introduced the latency already suffered significantly (iirc multi digit percent range), i suspect it now got worse again.
And after all that's still by far the most common system call (not only for databases; i profiled this using systemtap in some loads some time ago and it usually came up with >50%) and quite important for many workloads.
We have a lot far uglier code for less gain in far less critical calls.
-Andi
On Wed, 9 Jan 2008 18:48:00 +0100
Quoted 5 lines
And after all that's still by far the most common system call (not only for databases; i profiled this using systemtap in some loads some time ago and it usually came up with >50%) and quite important for many workloads.
btw be careful with this; the X server uses gettimeofday in it's equivalent of udelay()... (and we all know how useful it is to make the delay loops faster ;-)
Quoted 9 lines · Arjan van de Ven
On Wed, 9 Jan 2008 18:48:00 +0100
Quoted 5 lines
And after all that's still by far the most common system call (not only for databases; i profiled this using systemtap in some loads some time ago and it usually came up with >50%) and quite important for many workloads.
btw be careful with this; the X server uses gettimeofday in it's equivalent of
udelay()... (and we all know how useful it is to make the delay loops faster ;-)
People tend to make jokes about optimizing the idle loop too, but they're actually wrong. Exit latency for the idle loop is important -- it decides how quickly you can react to load changes on idle CPUs.
For short udelays I suspect shorter exit latency is also moderately useful.
But anyways there are plenty of gtod users outside the X server.
e.g. common not user space case is packet timestamps.
-Andi
H. Peter Anvin
reviewerQuoted 10 lines · Andi Kleen
People tend to make jokes about optimizing the idle loop too, but they're
actually wrong. Exit latency for the idle loop is important -- it decides how quickly you can react to load changes on idle CPUs.For short udelays I suspect shorter exit latency is also moderately useful.
But anyways there are plenty of gtod users outside the X server.
e.g. common not user space case is packet timestamps.
It's also very common to put gtod calls around select and poll, because the standard interface doesn't tell you how long you have been away (Linux at least used to have it, but glibc "fixed" that...)
-hpa
Quoted 8 lines · Andi Kleen
Quoted 1 line · Ingo Molnar
then you have a truly ancient x86.git repository ;-)
I update only infrequently because frankly git's remote branch tracking is a mess. At least it doesn't really work for x86#mm here.
I usually have to blow away the repository and reclone to get back to a sane state.
Someone in #git had a similar problem today. Conclusion was that x86/mm is not "stable" in the sense that commits are only added, instead history gets rewritten. That breaks pull/merge/"basic rebase".
Basically, you'll want to "rebase --onto", taking your local commits from the old branch history to the new branch history. One way to make that bearable is to have two branches (or a branch and a tag). One branch is used to keep your work, the other branch (or tag) is used to "mark" where the old upstream ended and your work started.
Assuming that your remote is called "x86", this could look like this:
git branch myStuff x86/mm
git branch myStuff_start x86/mm
work work work commit commit commit
git fetch x86 mm
git rebase --onto x86/mm myStuff_start myStuff
git branch -f myStuff_start x86/mm
The rebase will take all of the commits that you added to myStuff, on top of the old x86/mm (referred to by myStuff_start) and try to apply them on top of the new x86/mm, updating myStuff to point to the rebased commits afterwards.
Then myStuff_start is updated to point to the now current start of your stuff, so that you can do the rebase again later.
That's not actually a problem of git's tracking branches, because they're not made for that style of work at all. For such stuff, rebase is the only useful option AFAIK.
HTH,
Björn
Quoted 7 lines · Björn Steinbrink
Quoted 6 lines · Andi Kleen
Quoted 1 line · Ingo Molnar
then you have a truly ancient x86.git repository ;-)
I update only infrequently because frankly git's remote branch tracking is a mess. At least it doesn't really work for x86#mm here.
Quoted 8 lines
Quoted 2 lines
I usually have to blow away the repository and reclone to get back to a sane state.
Someone in #git had a similar problem today. Conclusion was that x86/mm is not "stable" in the sense that commits are only added, instead history gets rewritten. That breaks pull/merge/"basic rebase".
Basically, you'll want to "rebase --onto", taking your local commits
I don't really have any local commits. I just want to use read-only git to generate a patch that I can then import using quilt and only look at log files and use git show. The fanciest thing I use is git bisect occasionally.
Quoted 9 lines
that bearable is to have two branches (or a branch and a tag). One branch is used to keep your work, the other branch (or tag) is used to "mark" where the old upstream ended and your work started.
Assuming that your remote is called "x86", this could look like this:
git branch myStuff x86/mm
git branchmyStuff_startx86/mm
Quoted 6 lines
work work work commit commit commit
git fetch x86 mm
git rebase --onto x86/mm
myStuff_startmyStuff
git branch -fmyStuff_startx86/mm
Do you have a simple recipe to just update from the the remote branch, assuming there are no local changes or local branches?
-Andi
Quoted 4 lines · Andi Kleen
Do you have a simple recipe to just update from the the remote branch, assuming there are no local changes or local branches?
-Andi
For staying up to date I use the following:
# Add Linus's tree as a remote
git remote --add linusgit://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
# Add Ingo's tree as a remote
git remote --add x86git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
# With that setup, just run the following to get any changes you # don't have. It will also notice any new branches Ingo/Linus
# add to their repo. Look in .git/config afterwards, the format # to add new remotes is easy to figure out. git remote update
Cheers,
Harvey
Quoted 17 lines · Harvey Harrison
Quoted 4 lines · Andi Kleen
Do you have a simple recipe to just update from the the remote branch, assuming there are no local changes or local branches?
-Andi
For staying up to date I use the following:
# Add Linus's tree as a remote
git remote --add linusgit://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git# Add Ingo's tree as a remote
git remote --add x86git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
Quoted 5 lines
# With that setup, just run the following to get any changes you # don't have. It will also notice any new branches Ingo/Linus
# add to their repo. Look in .git/config afterwards, the format # to add new remotes is easy to figure out. git remote update
I'm already cloning the branches; the problem is not getting conflicts etc. when updating.
-Andi
Quoted 18 lines · Andi Kleen
Quoted 17 lines · Harvey Harrison
Quoted 4 lines · Andi Kleen
Do you have a simple recipe to just update from the the remote branch, assuming there are no local changes or local branches?
-Andi
For staying up to date I use the following:
# Add Linus's tree as a remote
git remote --add linusgit://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git# Add Ingo's tree as a remote
git remote --add x86git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-x86.git
Quoted 8 lines
Quoted 5 lines
# With that setup, just run the following to get any changes you # don't have. It will also notice any new branches Ingo/Linus
# add to their repo. Look in .git/config afterwards, the format # to add new remotes is easy to figure out. git remote updateI'm already cloning the branches; the problem is not getting conflicts etc. when updating.
I guess you're using "git pull" to update a local branch? That will try to merge the new state of x86/mm into your local branch, and that breaks. If you just want to have a local branch that gives you a "fixed" state of x86/mm regardless of whether or not you already fetched newer ones, you can do:
git branch myThing x86/mm # create the branch
work/test/whatever
To fetch a new "state" from the remote:
git fetch x86/mm # or git remote update, or whatever
To update your branch to point to the new state:
git branch - f myThing x86/mm
That basically replaces it with a new branch of the same name, but pointing to the new x86/mm.
Or if you want to get your working tree to that state at the same time, you can also do:
git checkout myThing
git reset --hard x86/mm
Björn
Quoted 2 lines · Andi Kleen
I'm already cloning the branches; the problem is not getting conflicts etc. when updating.
Isn't git pull --force what you are looking for?
Ciao,
Signature · 2 lines
Paolo http://paolo.ciarrocchi.googlepages.com/
Quoted 5 lines · Paolo Ciarrocchi
Quoted 2 lines · Andi Kleen
I'm already cloning the branches; the problem is not getting conflicts etc. when updating.
Quoted 7 lines
Isn't git pull --force what you are looking for?
Ciao,
--
Paolo
http://paolo.ciarrocchi.googlepages.com/
Actually...no, it wouldn't help. Conflicts will coming up and only the solution for now is to use remote tracking as Harvey wrote in prev mail. (btw, git pull that would not conflict is when it called with '-s ours' but that is not good)
- Cyrill -