Important per-cpu fix.
Subsystems (2)
PATCH 1/1 Important per-cpu fix.
Frankly, I'm amazed the kernel worked for long without this.
Every linker script thinks the section is called .data.percpu. Without this patch, every CPU ends up sharing the same "per-cpu" variable.
This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
include/asm-generic/percpu.h renamed from include/asm-generic/percpu.h.~1~ +1 -1
@@ -10,7 +10,7 @@/* Separate out the type, so (int[3], foo) works. */ #ifndef MODULE #define DEFINE_PER_CPU (type, name) \ - __attribute__((__section__(".percpu"))) __typeof__(type) name##__per_cpu + __attribute__((__section__(".data.percpu"))) __typeof__(type) name##__per_cpu#endif /* var is in discarded region: offset to particular copy we want */
Quoted 7 lines · Rusty Russell
Frankly, I'm amazed the kernel worked for long without this.
Every linker script thinks the section is called .data.percpu. Without this patch, every CPU ends up sharing the same "per-cpu" variable.
wow. Either this was working by fluke, or Bill's softirq problem just got solved.
Quoted 4 lines · Rusty Russell
Quoted 4 lines
Frankly, I'm amazed the kernel worked for long without this. Every linker script thinks the section is called .data.percpu. Without this patch, every CPU ends up sharing the same "per-cpu" variable.
Quoted 2 lines · Andrew Morton
wow. Either this was working by fluke, or Bill's softirq problem just got solved.
It's pretty easy to find out, my machines don't get past execve("/sbin/init", argv_init, envp_init); without some kind of softirq fix. The boot cycle is not swift though... in progress.
Cheers,
Bill
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000
This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...
Verifying... no without the explicit initializers the per-cpu stuff still ends up in the BSS with egcs-2.9X, even with your fix applied.
In message <[email protected]> you write:
Quoted 8 lines
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...
Verifying... no without the explicit initializers the per-cpu stuff still ends up in the BSS with egcs-2.9X, even with your fix applied.
OK. I really hate working around wierd toolchain bugs (I use 2.95.4 here and it's fine), and adding an initializer to the macro is ugly.
If you're not going to upgrade your compiler, will you accept a gcc patch to fix this? If so, where can I get the source to your exact version?
Thanks,
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Followup to: <[email protected]> By author: Rusty Russell <[email protected]> In newsgroup: linux.dev.kernel
Quoted 11 lines
In message <[email protected]> you write:
Quoted 8 lines
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...
Verifying... no without the explicit initializers the per-cpu stuff still ends up in the BSS with egcs-2.9X, even with your fix applied.
Quoted 7 lines
OK. I really hate working around wierd toolchain bugs (I use 2.95.4 here and it's fine), and adding an initializer to the macro is ugly.
If you're not going to upgrade your compiler, will you accept a gcc patch to fix this? If so, where can I get the source to your exact version?
gcc puts all uninitialized variables in .bss, and it apparently can't be overridden. This seems to be a side effect of the way gcc handles common variables.
-hpa
Signature · 3 lines
<[email protected]> at work, <[email protected]> in private! "Unix gives you enough rope to shoot yourself in the foot." http://www.zytor.com/~hpa/puzzle.txt <[email protected]>
Quoted 14 lines · On 3 Sep 2002 21:52:45 -0700 "H. Peter Anvin" <[email protected]> wrote:
Followup to: <[email protected]> By author: Rusty Russell <[email protected]> In newsgroup: linux.dev.kernelQuoted 11 lines
In message <[email protected]> you write:
Quoted 8 lines
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...
Verifying... no without the explicit initializers the per-cpu stuff still ends up in the BSS with egcs-2.9X, even with your fix applied.
Quoted 6 lines
Quoted 2 lines
OK. I really hate working around wierd toolchain bugs (I use 2.95.4 here and it's fine), and adding an initializer to the macro is ugly.
gcc puts all uninitialized variables in .bss, and it apparently can't be overridden. This seems to be a side effect of the way gcc handles common variables.
Err... no, as I said, it doesn't happen with 2.95.4 or 3.0.4.
Rusty.
Signature · 2 lines
there are those who do and those who hang on and you don't see too many doers quoting their contemporaries. -- Larry McVoy
From: Rusty Russell <[email protected]>
Date: Mon, 9 Sep 2002 18:06:19 +1000
Quoted 3 lines · On 3 Sep 2002 21:52:45 -0700 "H. Peter Anvin" <[email protected]> wrote:
gcc puts all uninitialized variables in .bss, and it apparently can't be overridden. This seems to be a side effect of the way gcc handles common variables.
Err... no, as I said, it doesn't happen with 2.95.4 or 3.0.4.
But note that older GCCs do have the problem and it isn't platform specific at all.
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 14:04:22 +1000
OK. I really hate working around wierd toolchain bugs (I use 2.95.4 here and it's fine), and adding an initializer to the macro is ugly.
Mine is 2.92.11
If you're not going to upgrade your compiler, will you accept a gcc patch to fix this? If so, where can I get the source to your exact version?
Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the problem. If you impose 3.1 or whatever, I get less people testing on sparc64 as a result.
Quoted 3 lines · David S. Miller
Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the problem. If you impose 3.1 or whatever, I get less people testing on sparc64 as a result.
Well, I don't think you have to go to 3.1.x.
gcc-2.96 at least seems to do all right. Apparently 2.95 does too. It's only the truly ancient compilers that don't work.
Linus
From: Linus Torvalds <[email protected]>
Date: Tue, 3 Sep 2002 22:17:39 -0700 (PDT)
Well, I don't think you have to go to 3.1.x.
Yes I do, anything after our standard sparc64 (ie. egcs-2.92.11) compiler up until 3.1.x is seriously unusable for kernel builds.
In message <[email protected]> you write:
Quoted 7 lines
If you're not going to upgrade your compiler, will you accept a gcc patch to fix this? If so, where can I get the source to your exact version?
Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the problem. If you impose 3.1 or whatever, I get less people testing on sparc64 as a result.
I understand. It's OK to piss off one Sparc64 Linux user, but if you piss off both of them, you're in trouble 8)
I was actually proposing to provide a patch to an egcs version of your choice which you could provide as kgcc to your users. But RH seem to have taken down the SRPM for 2.92.11 (egcs64-19980921-4.src.rpm).
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 16:00:35 +1000
I was actually proposing to provide a patch to an egcs version of your choice which you could provide as kgcc to your users. But RH seem to have taken down the SRPM for 2.92.11 (egcs64-19980921-4.src.rpm).
The last sparc64 Red Hat distribution was 6.2 And it still works unchanged with current kernels. :-)
Actually Rusty what's the big deal, add an "initializer" arg to the macro. It doesn't hurt anyone, it doesn't lose any space in the kernel image, and the macro arg reminds people to do it.
I think it's a small price to pay to keep a longer range of compilers supported :-)
In message <[email protected]> you write:
Quoted 8 lines
Actually Rusty what's the big deal, add an "initializer" arg to the macro. It doesn't hurt anyone, it doesn't lose any space in the kernel image, and the macro arg reminds people to do it.
I think it's a small price to pay to keep a longer range of compilers supported :-)
I disagree. They might not have a convenient (static) initializer, in which case it's simply cruel and unusual, to work around an obscure compiler bug.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Hi!
Quoted 11 lines
Quoted 7 lines
Actually Rusty what's the big deal, add an "initializer" arg to the macro. It doesn't hurt anyone, it doesn't lose any space in the kernel image, and the macro arg reminds people to do it.
I think it's a small price to pay to keep a longer range of compilers supported :-)
I disagree. They might not have a convenient (static) initializer, in which case it's simply cruel and unusual, to work around an obscure compiler bug.
Ugh? of course it will always have convient initializer, namely zero.
Pavel
Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt, details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.
In message <[email protected]> you write:
Quoted 7 lines
Hi!
Quoted 5 lines
Quoted 5 lines
Actually Rusty what's the big deal, add an "initializer" arg to the macro. It doesn't hurt anyone, it doesn't lose any space in the kernel image, and the macro arg reminds people to do it.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
I think it's a small price to pay to keep a longer range of compilers supported :-)
I disagree. They might not have a convenient (static) initializer, in which case it's simply cruel and unusual, to work around an obscure compiler bug.
Ugh? of course it will always have convient initializer, namely zero.
What if you really need to initialize them at runtime? You're putting a static initializer there simply to mask an obscure toolchain bug. I'd really prefer dotting:
/* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */ DEFINE_PER_CPU(int, xxx) = 0;
everywhere, which can be deleted later, to enforcing it for everyone in the infrastructure when it doesn't always make sense.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
From: Rusty Russell <[email protected]>
Date: Mon, 09 Sep 2002 13:45:02 +1000
/* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */ DEFINE_PER_CPU(int, xxx) = 0;
everywhere, which can be deleted later
Why everywhere? If you do it in the macro, then when you want to delete the initializer remove the macro arg. Then you cover both in-kernel and cases in external sources because the build will break for them so they'll know to fixup their macro invocation.
I hate when things change silently, and if you don't put it in the macro, someone will get it wrong and we'll find out only when someone with an older compiler uses that feature on an SMP machine.
Read as: This is bad programming practice Rusty, regardless of
your personal views about this particular GCC bug or what you think is a legitimate initializer.
See spinlock_t's declaration for older GCC versions, that is how we traditionally deal with compiler bugs like this, explicitly so that it is impossible to "miss" something and get it wrong.
Everytime I asked Linus to rearrange some piece of code that used spinlock_t initializers to workaround the "empty initializer at end of struct" GCC bug he always told me "someone will get it wrong if we do it that way" And I think he's right.
In message <[email protected]> you write:
Quoted 8 lines
From: Rusty Russell <[email protected]> Date: Mon, 09 Sep 2002 13:45:02 +1000 /* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */ DEFINE_PER_CPU(int, xxx) = 0; everywhere, which can be deleted later
Quoted 4 lines
Why everywhere? If you do it in the macro, then when you want to delete the initializer remove the macro arg. Then you cover both in-kernel and cases in external sources because the build will break for them so they'll know to fixup their macro invocation.
So, you break everyone's compile later *and* you have to figure out which initializers were required, and which ones were bogus. *Planning* to break an interface is just stupid.
Quoted 3 lines
See spinlock_t's declaration for older GCC versions, that is how we traditionally deal with compiler bugs like this, explicitly so that it is impossible to "miss" something and get it wrong.
Yeah, but you can still leave a spinlock uninitialized, and it'll work.
I'm not making an interface harder to use because of one of 17 architectures refuses to fix a bug in their toolchain. Just not happening, sorry.
I want *you* to feel the pain, not spread it around by leaving turds throughout the code long after the bug is forgotten:
Makefile +3 -0
@@ -282,6 +282,8 @@ cmd_link_vmlinux = $(LD) $(LDFLAGS) $(LD -o vmlinux # set -e makes the rule exit immediately on error +# Final awk script makes sure per-cpu vars are in per-cpu section, as +# old gcc (eg egcs 2.92.11) ignores section attribute if uninitialized. define rule_link_vmlinux set -e
@@ -292,6 +294,7 @@ define rule_link_vmlinux $(call cmd,link_vmlinux) echo 'cmd_$@ := $(cmd_link_vmlinux)' > $(@D)/.$(@F).cmd $(NM) vmlinux | grep -v '\(compiled\)\|\(\.o$$\)\|\( [aUw] \)\|\(\.\.ng$$\)\|\(LASH[RL]DI\)' | sort > System.map + awk '/ __per_cpu_start$/ { IN_PER_CPU=1 } / __per_cpu_end$/ { IN_PER_CPU=0 } /__per_cpu$/ { if (!IN_PER_CPU) { print $3 " not in per-cpu section" > "/dev/stderr"; FOUND=1; } } END { exit FOUND; }' < System.map endef vmlinux: $(vmlinux-objs) FORCE
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
From: Rusty Russell <[email protected]>
Date: Mon, 09 Sep 2002 18:17:22 +1000
In message <[email protected]> you write:
I'm not making an interface harder to use because of one of 17 architectures refuses to fix a bug in their toolchain.
It's not a sparc64 specific problem, as has been noted. We are extended the life of an older toolchain on more than one platform.
Note that Andrew Morton found the problem on one of his older x86 EGCS's about the same time I found it on sparc64.
I want *you* to feel the pain, not spread it around by leaving turds throughout the code long after the bug is forgotten:
Aha, but it is you putting the turd comments all over. I'm suggesting to put the turd in one place, the header file.
And how difficult is it to discern which initializers were needed? Hmmm let me see, if it was all zero --> removing it is harmless.
Wow, that was hard. :-)
Both of us are advocating adding shit to the tree, the only argument is which stinks less from a maintainence perspective.
Quoted 2 lines · David S. Miller
Note that Andrew Morton found the problem on one of his older x86 EGCS's about the same time I found it on sparc64.
egcs gets so many long long things wrong on x86 that its only valid use IMHO for 2.5 is as a syntax checker. Is it really worth an ugly hack for a compiler one major developer has a personal affliction for and a port that has a tiny user base and now has a working compiler.
Maybe if we put less gunk in the kernel they'd fix gcc more often 8)
Quoted 11 lines · Alan Cox
Quoted 2 lines · David S. Miller
Note that Andrew Morton found the problem on one of his older x86 EGCS's about the same time I found it on sparc64.
egcs gets so many long long things wrong on x86 that its only valid use IMHO for 2.5 is as a syntax checker. Is it really worth an ugly hack for a compiler one major developer has a personal affliction for and a port that has a tiny user base and now has a working compiler.
Maybe if we put less gunk in the kernel they'd fix gcc more often 8)
Please. I'm not a reason for hanging onto egcs-1.1.2; and I'll downgrade to 2.95.2 when egcs-1.1.2 is retired (as I did a while back).
But as long as we need to support 1.1.2, I use it. To detect breakage, and because it compiles kernels 30% faster.
In message <[email protected]> you write:
Quoted 5 lines
I want *you* to feel the pain, not spread it around by leaving turds throughout the code long after the bug is forgotten:
Aha, but it is you putting the turd comments all over. I'm suggesting to put the turd in one place, the header file.
.... and every user of it...
Quoted 3 lines
And how difficult is it to discern which initializers were needed? Hmmm let me see, if it was all zero --> removing it is harmless.
Let's not get onto initializer wars: I initialize all my variables exactly once, so there's serious semantic difference between "static int x;" and "static int x = 0;" in my code.
Quoted 2 lines
Both of us are advocating adding shit to the tree, the only argument is which stinks less from a maintainence perspective.
Hey, I'm not stopping you sending a patch to Linus, but given how we deprecated compilers on x86, I don't think he'll have sympathy for you in 2.6.
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Hi,
Quoted 3 lines · David S. Miller
Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the problem. If you impose 3.1 or whatever, I get less people testing on sparc64 as a result.
Oh, well. I couldn't use gcc 3.* on sparc64 for some weird reason, gcc 2.95* is unable to handle all that crap, and the egcs hackup can't handle the current kernel code. So what?
Thunder
Signature · 4 lines
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .- --/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..- .- -/---/--/---/.-./.-./---/.--/.-.-.- --./.-/-.../.-./.././.-../.-.-.-
From: Thunder from the hill <[email protected]>
Date: Wed, 4 Sep 2002 09:44:51 -0600 (MDT)
I couldn't use gcc 3.* on sparc64 for some weird reason
3.0 should be broken yes, but 3.1 definitely works as I use it here for occaisionaly test builds all the time.
Quoted 6 lines · Rusty Russell
Frankly, I'm amazed the kernel worked for long without this. Every linker script thinks the section is called .data.percpu. Without this patch, every CPU ends up sharing the same "per-cpu" variable.
This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...
Hmm, 2.5.33 is doing some *really* weird crap. OTOH it doesn't appear to be tripping the BUG() in softirq.c, and disks seem to be doing okay. It survived 4 parallel mkfs's. I'll follow up with some kind of bugreport on the PCI and/or starfire.c front in a separate post.
Cheers,
Bill