Skip to contentSkip to conversation

Important per-cpu fix.

PATCH In review Rusty Russell 10 participants 26 messages Sep 4, 2002 – Sep 10, 2002

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Important per-cpu fix.

Frankly, I'm amazed the kernel worked for long without this.

Every linker script thinks the section is called .data.percpu. Without this patch, every CPU ends up sharing the same "per-cpu" variable.

This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

include/asm-generic/percpu.h renamed from include/asm-generic/percpu.h.~1~ +1 -1
@@ -10,7 +10,7 @@
 /* Separate out the type, so (int[3], foo) works. */
 #ifndef MODULE
 #define DEFINE_PER_CPU(type, name) \
-    __attribute__((__section__(".percpu"))) __typeof__(type) name##__per_cpu
+    __attribute__((__section__(".data.percpu"))) __typeof__(type) name##__per_cpu
 #endif
 
 /* var is in discarded region: offset to particular copy we want */
#1 lore raw
Quoted 7 lines · Rusty Russell

Frankly, I'm amazed the kernel worked for long without this.

Every linker script thinks the section is called .data.percpu. Without this patch, every CPU ends up sharing the same "per-cpu" variable.

wow. Either this was working by fluke, or Bill's softirq problem just got solved.

#2 lore raw
Quoted 4 lines · Rusty Russell
Quoted 4 lines

Frankly, I'm amazed the kernel worked for long without this. Every linker script thinks the section is called .data.percpu. Without this patch, every CPU ends up sharing the same "per-cpu" variable.

Quoted 2 lines · Andrew Morton

wow. Either this was working by fluke, or Bill's softirq problem just got solved.

It's pretty easy to find out, my machines don't get past execve("/sbin/init", argv_init, envp_init); without some kind of softirq fix. The boot cycle is not swift though... in progress.

Cheers,
Bill

#3 lore raw
In reply to Rusty Russell

From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000

This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...

Verifying... no without the explicit initializers the per-cpu stuff still ends up in the BSS with egcs-2.9X, even with your fix applied.

#4 lore raw

In message <[email protected]> you write:

Quoted 8 lines

From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000

This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...

Verifying... no without the explicit initializers the per-cpu stuff still ends up in the BSS with egcs-2.9X, even with your fix applied.

OK. I really hate working around wierd toolchain bugs (I use 2.95.4 here and it's fine), and adding an initializer to the macro is ugly.

If you're not going to upgrade your compiler, will you accept a gcc patch to fix this? If so, where can I get the source to your exact version?

Thanks,
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

#5 lore raw
Followup to:  <[email protected]>
By author:    Rusty Russell <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 11 lines

In message <[email protected]> you write:

Quoted 8 lines

From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000

This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...

Verifying... no without the explicit initializers the per-cpu stuff still ends up in the BSS with egcs-2.9X, even with your fix applied.

Quoted 7 lines

OK. I really hate working around wierd toolchain bugs (I use 2.95.4 here and it's fine), and adding an initializer to the macro is ugly.

If you're not going to upgrade your compiler, will you accept a gcc patch to fix this? If so, where can I get the source to your exact version?

gcc puts all uninitialized variables in .bss, and it apparently can't be overridden. This seems to be a side effect of the way gcc handles common variables.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#6 lore raw
Quoted 14 lines · On 3 Sep 2002 21:52:45 -0700 "H. Peter Anvin" <[email protected]> wrote:
Followup to:  <[email protected]>
By author:    Rusty Russell <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 11 lines

In message <[email protected]> you write:

Quoted 8 lines

From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 12:35:41 +1000

This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...

Verifying... no without the explicit initializers the per-cpu stuff still ends up in the BSS with egcs-2.9X, even with your fix applied.

Quoted 6 lines
Quoted 2 lines

OK. I really hate working around wierd toolchain bugs (I use 2.95.4 here and it's fine), and adding an initializer to the macro is ugly.

gcc puts all uninitialized variables in .bss, and it apparently can't be overridden. This seems to be a side effect of the way gcc handles common variables.

Err... no, as I said, it doesn't happen with 2.95.4 or 3.0.4.

Rusty.

Signature · 2 lines
   there are those who do and those who hang on and you don't see too
   many doers quoting their contemporaries.  -- Larry McVoy
#7 lore raw

From: Rusty Russell <[email protected]>
Date: Mon, 9 Sep 2002 18:06:19 +1000

Quoted 3 lines · On 3 Sep 2002 21:52:45 -0700 "H. Peter Anvin" <[email protected]> wrote:

gcc puts all uninitialized variables in .bss, and it apparently can't be overridden. This seems to be a side effect of the way gcc handles common variables.

Err... no, as I said, it doesn't happen with 2.95.4 or 3.0.4.

But note that older GCCs do have the problem and it isn't platform specific at all.

#8 lore raw
In reply to Rusty Russell

From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 14:04:22 +1000

OK. I really hate working around wierd toolchain bugs (I use 2.95.4 here and it's fine), and adding an initializer to the macro is ugly.

Mine is 2.92.11

If you're not going to upgrade your compiler, will you accept a gcc patch to fix this? If so, where can I get the source to your exact version?

Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the problem. If you impose 3.1 or whatever, I get less people testing on sparc64 as a result.

#9 lore raw
Quoted 3 lines · David S. Miller

Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the problem. If you impose 3.1 or whatever, I get less people testing on sparc64 as a result.

Well, I don't think you have to go to 3.1.x.

gcc-2.96 at least seems to do all right. Apparently 2.95 does too. It's only the truly ancient compilers that don't work.

Linus

#10 lore raw

From: Linus Torvalds <[email protected]>
Date: Tue, 3 Sep 2002 22:17:39 -0700 (PDT)

Well, I don't think you have to go to 3.1.x.

Yes I do, anything after our standard sparc64 (ie. egcs-2.92.11) compiler up until 3.1.x is seriously unusable for kernel builds.

#11 lore raw
In reply to David S. Miller

In message <[email protected]> you write:

Quoted 7 lines

If you're not going to upgrade your compiler, will you accept a gcc patch to fix this? If so, where can I get the source to your exact version?

Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the problem. If you impose 3.1 or whatever, I get less people testing on sparc64 as a result.

I understand. It's OK to piss off one Sparc64 Linux user, but if you piss off both of them, you're in trouble 8)

I was actually proposing to provide a patch to an egcs version of your choice which you could provide as kgcc to your users. But RH seem to have taken down the SRPM for 2.92.11 (egcs64-19980921-4.src.rpm).

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

#12 lore raw

From: Rusty Russell <[email protected]>
Date: Wed, 04 Sep 2002 16:00:35 +1000

I was actually proposing to provide a patch to an egcs version of your choice which you could provide as kgcc to your users. But RH seem to have taken down the SRPM for 2.92.11 (egcs64-19980921-4.src.rpm).

The last sparc64 Red Hat distribution was 6.2 And it still works unchanged with current kernels. :-)

#13 lore raw

Actually Rusty what's the big deal, add an "initializer" arg to the macro. It doesn't hurt anyone, it doesn't lose any space in the kernel image, and the macro arg reminds people to do it.

I think it's a small price to pay to keep a longer range of compilers supported :-)

#14 lore raw

In message <[email protected]> you write:

Quoted 8 lines

Actually Rusty what's the big deal, add an "initializer" arg to the macro. It doesn't hurt anyone, it doesn't lose any space in the kernel image, and the macro arg reminds people to do it.

I think it's a small price to pay to keep a longer range of compilers supported :-)

I disagree. They might not have a convenient (static) initializer, in which case it's simply cruel and unusual, to work around an obscure compiler bug.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

#15 lore raw

Hi!

Quoted 11 lines
Quoted 7 lines

Actually Rusty what's the big deal, add an "initializer" arg to the macro. It doesn't hurt anyone, it doesn't lose any space in the kernel image, and the macro arg reminds people to do it.

I think it's a small price to pay to keep a longer range of compilers supported :-)

I disagree. They might not have a convenient (static) initializer, in which case it's simply cruel and unusual, to work around an obscure compiler bug.

Ugh? of course it will always have convient initializer, namely zero.

Pavel

Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.
#16 lore raw

In message <[email protected]> you write:

Quoted 7 lines

Hi!

Quoted 5 lines
Quoted 5 lines

Actually Rusty what's the big deal, add an "initializer" arg to the macro. It doesn't hurt anyone, it doesn't lose any space in the kernel image, and the macro arg reminds people to do it.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I think it's a small price to pay to keep a longer range of compilers supported :-)

I disagree. They might not have a convenient (static) initializer, in which case it's simply cruel and unusual, to work around an obscure compiler bug.

Ugh? of course it will always have convient initializer, namely zero.

What if you really need to initialize them at runtime? You're putting a static initializer there simply to mask an obscure toolchain bug. I'd really prefer dotting:

	/* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */
	DEFINE_PER_CPU(int, xxx) = 0;

everywhere, which can be deleted later, to enforcing it for everyone in the infrastructure when it doesn't always make sense.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

#17 lore raw

From: Rusty Russell <[email protected]>
Date: Mon, 09 Sep 2002 13:45:02 +1000

/* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */ DEFINE_PER_CPU(int, xxx) = 0;

everywhere, which can be deleted later

Why everywhere? If you do it in the macro, then when you want to delete the initializer remove the macro arg. Then you cover both in-kernel and cases in external sources because the build will break for them so they'll know to fixup their macro invocation.

I hate when things change silently, and if you don't put it in the macro, someone will get it wrong and we'll find out only when someone with an older compiler uses that feature on an SMP machine.

Read as: This is bad programming practice Rusty, regardless of
your personal views about this particular GCC bug or what you think is a legitimate initializer.

See spinlock_t's declaration for older GCC versions, that is how we traditionally deal with compiler bugs like this, explicitly so that it is impossible to "miss" something and get it wrong.

Everytime I asked Linus to rearrange some piece of code that used spinlock_t initializers to workaround the "empty initializer at end of struct" GCC bug he always told me "someone will get it wrong if we do it that way" And I think he's right.

#18 lore raw

In message <[email protected]> you write:

Quoted 8 lines
   From: Rusty Russell <[email protected]>
   Date: Mon, 09 Sep 2002 13:45:02 +1000

   	/* FIXME: Initializer required so gcc 2.96 doesn't put in BSS */
   	DEFINE_PER_CPU(int, xxx) = 0;
   
   everywhere, which can be deleted later
Quoted 4 lines

Why everywhere? If you do it in the macro, then when you want to delete the initializer remove the macro arg. Then you cover both in-kernel and cases in external sources because the build will break for them so they'll know to fixup their macro invocation.

So, you break everyone's compile later *and* you have to figure out which initializers were required, and which ones were bogus. *Planning* to break an interface is just stupid.

Quoted 3 lines

See spinlock_t's declaration for older GCC versions, that is how we traditionally deal with compiler bugs like this, explicitly so that it is impossible to "miss" something and get it wrong.

Yeah, but you can still leave a spinlock uninitialized, and it'll work.

I'm not making an interface harder to use because of one of 17 architectures refuses to fix a bug in their toolchain. Just not happening, sorry.

I want *you* to feel the pain, not spread it around by leaving turds throughout the code long after the bug is forgotten:

Makefile +3 -0
diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal working-2.5.33-cset-1.621/Makefile working-2.5.33-cset-1.621-catch-percpu/Makefile
@@ -282,6 +282,8 @@ cmd_link_vmlinux = $(LD) $(LDFLAGS) $(LD
 		-o vmlinux
 
 #	set -e makes the rule exit immediately on error
+#	Final awk script makes sure per-cpu vars are in per-cpu section, as
+#	old gcc (eg egcs 2.92.11) ignores section attribute if uninitialized.
 
 define rule_link_vmlinux
 	set -e
@@ -292,6 +294,7 @@ define rule_link_vmlinux
 	$(call cmd,link_vmlinux)
 	echo 'cmd_$@ := $(cmd_link_vmlinux)' > $(@D)/.$(@F).cmd
 	$(NM) vmlinux | grep -v '\(compiled\)\|\(\.o$$\)\|\( [aUw] \)\|\(\.\.ng$$\)\|\(LASH[RL]DI\)' | sort > System.map
+	awk '/ __per_cpu_start$/ { IN_PER_CPU=1 } / __per_cpu_end$/ { IN_PER_CPU=0 } /__per_cpu$/ { if (!IN_PER_CPU) { print $3 " not in per-cpu section" > "/dev/stderr"; FOUND=1; } } END { exit FOUND; }' < System.map
 endef
 
 vmlinux: $(vmlinux-objs) FORCE

--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

#19 lore raw

From: Rusty Russell <[email protected]>
Date: Mon, 09 Sep 2002 18:17:22 +1000

In message <[email protected]> you write:
I'm not making an interface harder to use because of one of 17 architectures refuses to fix a bug in their toolchain.

It's not a sparc64 specific problem, as has been noted. We are extended the life of an older toolchain on more than one platform.

Note that Andrew Morton found the problem on one of his older x86 EGCS's about the same time I found it on sparc64.

I want *you* to feel the pain, not spread it around by leaving turds throughout the code long after the bug is forgotten:

Aha, but it is you putting the turd comments all over. I'm suggesting to put the turd in one place, the header file.

And how difficult is it to discern which initializers were needed? Hmmm let me see, if it was all zero --> removing it is harmless.

Wow, that was hard. :-)

Both of us are advocating adding shit to the tree, the only argument is which stinks less from a maintainence perspective.

#20 lore raw
Quoted 2 lines · David S. Miller

Note that Andrew Morton found the problem on one of his older x86 EGCS's about the same time I found it on sparc64.

egcs gets so many long long things wrong on x86 that its only valid use IMHO for 2.5 is as a syntax checker. Is it really worth an ugly hack for a compiler one major developer has a personal affliction for and a port that has a tiny user base and now has a working compiler.

Maybe if we put less gunk in the kernel they'd fix gcc more often 8)

#21 lore raw
Quoted 11 lines · Alan Cox
Quoted 2 lines · David S. Miller

Note that Andrew Morton found the problem on one of his older x86 EGCS's about the same time I found it on sparc64.

egcs gets so many long long things wrong on x86 that its only valid use IMHO for 2.5 is as a syntax checker. Is it really worth an ugly hack for a compiler one major developer has a personal affliction for and a port that has a tiny user base and now has a working compiler.

Maybe if we put less gunk in the kernel they'd fix gcc more often 8)

Please. I'm not a reason for hanging onto egcs-1.1.2; and I'll downgrade to 2.95.2 when egcs-1.1.2 is retired (as I did a while back).

But as long as we need to support 1.1.2, I use it. To detect breakage, and because it compiles kernels 30% faster.

#22 lore raw
In reply to David S. Miller

In message <[email protected]> you write:

Quoted 5 lines

I want *you* to feel the pain, not spread it around by leaving turds throughout the code long after the bug is forgotten:

Aha, but it is you putting the turd comments all over. I'm suggesting to put the turd in one place, the header file.

.... and every user of it...

Quoted 3 lines

And how difficult is it to discern which initializers were needed? Hmmm let me see, if it was all zero --> removing it is harmless.

Let's not get onto initializer wars: I initialize all my variables exactly once, so there's serious semantic difference between "static int x;" and "static int x = 0;" in my code.

Quoted 2 lines

Both of us are advocating adding shit to the tree, the only argument is which stinks less from a maintainence perspective.

Hey, I'm not stopping you sending a patch to Linus, but given how we deprecated compilers on x86, I don't think he'll have sympathy for you in 2.6.

Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

#23 lore raw
In reply to David S. Miller

Hi,

Quoted 3 lines · David S. Miller

Oh, "I'm" willing to upgrade "my" compiler, it's my users that are the problem. If you impose 3.1 or whatever, I get less people testing on sparc64 as a result.

Oh, well. I couldn't use gcc 3.* on sparc64 for some weird reason, gcc 2.95* is unable to handle all that crap, and the egcs hackup can't handle the current kernel code. So what?

Thunder

Signature · 4 lines
--./../...-/. -.--/---/..-/.-./..././.-../..-. .---/..-/.../- .-
--/../-./..-/-/./--..-- ../.----./.-../.-.. --./../...-/. -.--/---/..-
.- -/---/--/---/.-./.-./---/.--/.-.-.-
--./.-/-.../.-./.././.-../.-.-.-
#24 lore raw

From: Thunder from the hill <[email protected]>
Date: Wed, 4 Sep 2002 09:44:51 -0600 (MDT)

I couldn't use gcc 3.* on sparc64 for some weird reason

3.0 should be broken yes, but 3.1 definitely works as I use it here for occaisionaly test builds all the time.

#25 lore raw
In reply to Rusty Russell
Quoted 6 lines · Rusty Russell

Frankly, I'm amazed the kernel worked for long without this. Every linker script thinks the section is called .data.percpu. Without this patch, every CPU ends up sharing the same "per-cpu" variable.
This might explain the wierd per-cpu problem reports from Andrew and Dave, and also that nagging feeling that I'm an idiot...

Hmm, 2.5.33 is doing some *really* weird crap. OTOH it doesn't appear to be tripping the BUG() in softirq.c, and disks seem to be doing okay. It survived 4 parallel mkfs's. I'll follow up with some kind of bugreport on the PCI and/or starfire.c front in a separate post.

Cheers,
Bill