Skip to contentSkip to conversation

2.6.39-rc6-mmotm0506 and -next - __kfree_rcu breaks third-party kernel code

DISCUSSION [email protected] 4 participants 7 messages May 9, 2011 – May 11, 2011 RCU

lore
Share & tools
lore

[email protected]

author
#0 lore raw

So I have a third-party kernel module which wants to check what kernel version it's building against, and it fails to build under -rc6-mmotm0506 (it works just fine under -rc5-mmotm0429).

I finally isolated it down to a two-liner section of code, which seems to die because somebody didn't include something that rcupdate.h wanted to see:

cat > /tmp/kernconf.c <<EOF
#include <linux/version.h>
#include <linux/utsname.h>
EOF
+ cat

# This builds fine
kern="2.6.39-rc5-mmotm0429"
+ kern=2.6.39-rc5-mmotm0429
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/${kern}/build/include -include /lib/modules/${kern}/build/include/generated/autoconf.h -I/lib/modules/${kern}/build/arch/x86/include -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc5-mmotm0429/build/include -include /lib/modules/2.6.39-rc5-mmotm0429/build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc5-mmotm0429/build/arch/x86/include -c /tmp/kernconf.c

# this one dies
kern="2.6.39-rc6-mmotm0506"
+ kern=2.6.39-rc6-mmotm0506
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/${kern}/build/include -include /lib/modules/${kern}/build/include/generated/autoconf.h -I/lib/modules/${kern}/build/arch/x86/include -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506/build/include -include /lib/modules/2.6.39-rc6-mmotm0506/build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506/build/arch/x86/include -c /tmp/kernconf.c
In file included from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/sem.h:81:0,
                 from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/sched.h:72,
                 from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/utsname.h:35,
                 from /tmp/kernconf.c:2:
/lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

rm /tmp/kernconf.c
+ rm /tmp/kernconf.c

'git blame include/linux/rcudate.h' points at this commit:

commit 896d4eb94e0b7cd896ee92c380fadc1e9d867cfc

Author: Lai Jiangshan <[email protected]>
Date:   Fri Mar 18 11:15:47 2011 +0800

rcu: introduce kfree_rcu()

#1 lore raw
Quoted 34 lines · On Mon, 09 May 2011 11:05:00 -0400 [email protected] wrote:

So I have a third-party kernel module which wants to check what kernel version it's building against, and it fails to build under -rc6-mmotm0506 (it works just fine under -rc5-mmotm0429).

I finally isolated it down to a two-liner section of code, which seems to die because somebody didn't include something that rcupdate.h wanted to see:

cat > /tmp/kernconf.c <<EOF
#include <linux/version.h>
#include <linux/utsname.h>
EOF
+ cat

# This builds fine
kern="2.6.39-rc5-mmotm0429"
+ kern=2.6.39-rc5-mmotm0429
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/${kern}/build/include -include /lib/modules/${kern}/build/include/generated/autoconf.h -I/lib/modules/${kern}/build/arch/x86/include -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc5-mmotm0429/build/include -include /lib/modules/2.6.39-rc5-mmotm0429/build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc5-mmotm0429/build/arch/x86/include -c /tmp/kernconf.c

# this one dies
kern="2.6.39-rc6-mmotm0506"
+ kern=2.6.39-rc6-mmotm0506
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/${kern}/build/include -include /lib/modules/${kern}/build/include/generated/autoconf.h -I/lib/modules/${kern}/build/arch/x86/include -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506/build/include -include /lib/modules/2.6.39-rc6-mmotm0506/build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506/build/arch/x86/include -c /tmp/kernconf.c
In file included from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/sem.h:81:0,
                 from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/sched.h:72,
                 from /lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/utsname.h:35,
                 from /tmp/kernconf.c:2:
/lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506/build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

rm /tmp/kernconf.c
+ rm /tmp/kernconf.c

Quoted 7 lines

'git blame include/linux/rcudate.h' points at this commit:

commit 896d4eb94e0b7cd896ee92c380fadc1e9d867cfc

Author: Lai Jiangshan <[email protected]>
Date:   Fri Mar 18 11:15:47 2011 +0800

rcu: introduce kfree_rcu()

I'd be happy to see a simple fix, but we don't generally support building external modules without using the kernel kbuild infrastructure AFAIK -- and when I add a Makefile along with your kernconf.c file, it builds successfully on mmotm-2011-0506-1639.

Makefile is:
# usage:

# cd /path/to/kernel/source/ && make SUBDIRS=/path/to/source/procfs_ex/ modules
# aka:
# make -C /path/to/kernel/source M=/path/to/module/source [O=/path/to/kernel/build] modules

obj-m := kernconf.o

clean-files := *.o *.ko *.mod.c
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

then:

Quoted 1 line

make -C /lnx/src/MM/mmotm-2011-0506-1639 M=$PWD O=/lnx/src/MM/mmotm-2011-0506-1639/xx64 modules 2>&1 | tee bld1.out

make: Entering directory `/lnx/src/MM/mmotm-2011-0506-1639'
CC [M] /home/rddunlap/src/kernmodule/kernconf.o Building modules, stage 2.
MODPOST 1 modules
CC /home/rddunlap/src/kernmodule/kernconf.mod.o
LD [M] /home/rddunlap/src/kernmodule/kernconf.ko
make: Leaving directory `/lnx/src/MM/mmotm-2011-0506-1639'

done.
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

author
#2 lore raw
Quoted 4 lines · Randy Dunlap

I'd be happy to see a simple fix, but we don't generally support building external modules without using the kernel kbuild infrastructure AFAIK -- and when I add a Makefile along with your kernconf.c file, it builds successfully on mmotm-2011-0506-1639.

And in fact the module *does* use the Makefile stuff for the actual build - it was just the installer's "am I looking at a sane 2.6 tree" sanity check that bombed. So I go looking for what special sauce the Makefile adds to make it work, and....

OK, *now* I'm confused. The magic sauce appears to be "-Os". Adding that makes the compile work. And for no obvious reason - there's something in there that causes indigestion, but only if the optimizer isn't invoked.

*WTF*?

LANG=C ./kern.sh
cat > /tmp/kernconf.c <<EOF
#include <linux/version.h>
#include <linux/utsname.h>
EOF
+ cat

kern="/lib/modules/2.6.39-rc6-mmotm0506/"
+ kern=/lib/modules/2.6.39-rc6-mmotm0506/
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -Os -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -Os -c /tmp/kernconf.c cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -c /tmp/kernconf.c In file included from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sem.h:81:0,

                 from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sched.h:72,
                 from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/utsname.h:35,
                 from /tmp/kernconf.c:2:

/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

rm /tmp/kernconf.c
+ rm /tmp/kernconf.c

author
#3 lore raw
Quoted 3 lines · On Mon, 09 May 2011 12:56:32 EDT, [email protected] said:

OK, *now* I'm confused. The magic sauce appears to be "-Os". Adding that makes the compile work. And for no obvious reason - there's something in there that causes indigestion, but only if the optimizer isn't invoked.

/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

Apparently, it's in fact bad code - but the entire __kfree_rcu() function is being reaped by the optimizer as dead code before the code generator sees the size issue with the array. Without optimization, the code hangs around long enough for the problem to crop up.

#4 lore raw
In reply to [email protected]
Quoted 7 lines · On 05/09/11 09:56, [email protected] wrote:
Quoted 4 lines · Randy Dunlap

I'd be happy to see a simple fix, but we don't generally support building external modules without using the kernel kbuild infrastructure AFAIK -- and when I add a Makefile along with your kernconf.c file, it builds successfully on mmotm-2011-0506-1639.

Quoted 8 lines

And in fact the module *does* use the Makefile stuff for the actual build - it was just the installer's "am I looking at a sane 2.6 tree" sanity check that bombed. So I go looking for what special sauce the Makefile adds to make it work, and....

OK, *now* I'm confused. The magic sauce appears to be "-Os". Adding that makes the compile work. And for no obvious reason - there's something in there that causes indigestion, but only if the optimizer isn't invoked.

You can disable that by disabling:
CONFIG_CC_OPTIMIZE_FOR_SIZE=y

Quoted 1 line

*WTF*?

I dunno either.

Quoted 14 lines

LANG=C ./kern.sh
cat > /tmp/kernconf.c <<EOF
#include <linux/version.h>
#include <linux/utsname.h>
EOF
+ cat

kern="/lib/modules/2.6.39-rc6-mmotm0506/"
+ kern=/lib/modules/2.6.39-rc6-mmotm0506/
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -Os -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -Os -c /tmp/kernconf.c cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -c /tmp/kernconf.c In file included from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sem.h:81:0,

Quoted 8 lines
                 from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sched.h:72,
                 from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/utsname.h:35,
                 from /tmp/kernconf.c:2:

/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

rm /tmp/kernconf.c
+ rm /tmp/kernconf.c

rcupdate.h should #include <linux/kernel.h>, but I don't think that will fix this problem.

Signature · 2 lines
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***
#5 lore raw
Quoted 17 lines · Randy Dunlap
Quoted 7 lines · On 05/09/11 09:56, [email protected] wrote:
Quoted 4 lines · Randy Dunlap

I'd be happy to see a simple fix, but we don't generally support building external modules without using the kernel kbuild infrastructure AFAIK -- and when I add a Makefile along with your kernconf.c file, it builds successfully on mmotm-2011-0506-1639.

Quoted 8 lines

And in fact the module *does* use the Makefile stuff for the actual build - it was just the installer's "am I looking at a sane 2.6 tree" sanity check that bombed. So I go looking for what special sauce the Makefile adds to make it work, and....

OK, *now* I'm confused. The magic sauce appears to be "-Os". Adding that makes the compile work. And for no obvious reason - there's something in there that causes indigestion, but only if the optimizer isn't invoked.

Quoted 6 lines

You can disable that by disabling:
CONFIG_CC_OPTIMIZE_FOR_SIZE=y

Quoted 1 line

*WTF*?

I dunno either.

It appears that some compilers at some optimization levels aren't willing to trace build-time constants through local parameters... Could you please try out the following patch?

Thanx, Paul

------------------------------------------------------------------------

rcu: avoid build error for third-party modules

The initial definition of __kfree_rcu() checked a static inline function argument to see if it was a compile-time constant. Apparently not all compilers are willing to put up with this at all optimization levels. Add a nasty comment and remove the warning, relying on the fact that __kfree_rcu() is called only from kfree_rcu(), which always passes in a compile-time constant.

Signed-off-by Paul E. McKenney <[email protected]>
include/linux/rcupdate.h +3 -2
index 99f9aa7..58b13f1 100644
@@ -814,13 +814,14 @@ static __always_inline bool __is_kfree_rcu_offset(unsigned long offset)
 	return offset < 4096;
 }
 
+/*
+ * Intended to be called only from the kfree_rcu() macro.
+ */
 static __always_inline
 void __kfree_rcu(struct rcu_head *head, unsigned long offset)
 {
 	typedef void (*rcu_callback)(struct rcu_head *);
 
-	BUILD_BUG_ON(!__builtin_constant_p(offset));
-
 	/* See the kfree_rcu() header comment. */
 	BUILD_BUG_ON(!__is_kfree_rcu_offset(offset));
#6 lore raw
In reply to [email protected]
Quoted 32 lines · On 05/10/2011 12:56 AM, [email protected] wrote:
Quoted 4 lines · Randy Dunlap

I'd be happy to see a simple fix, but we don't generally support building external modules without using the kernel kbuild infrastructure AFAIK -- and when I add a Makefile along with your kernconf.c file, it builds successfully on mmotm-2011-0506-1639.

And in fact the module *does* use the Makefile stuff for the actual build - it was just the installer's "am I looking at a sane 2.6 tree" sanity check that bombed. So I go looking for what special sauce the Makefile adds to make it work, and....

OK, *now* I'm confused. The magic sauce appears to be "-Os". Adding that makes the compile work. And for no obvious reason - there's something in there that causes indigestion, but only if the optimizer isn't invoked.

*WTF*?

LANG=C ./kern.sh
cat > /tmp/kernconf.c <<EOF
#include <linux/version.h>
#include <linux/utsname.h>
EOF
+ cat

kern="/lib/modules/2.6.39-rc6-mmotm0506/"
+ kern=/lib/modules/2.6.39-rc6-mmotm0506/
cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -Os -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -Os -c /tmp/kernconf.c cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I${kern}/build/include -I$kern/arch/x86/include/generated -include ${kern}/build/include/generated/autoconf.h -I${kern}/build/arch/x86/include -I${kern}/include/generated -c /tmp/kernconf.c
+ cc -D__KERNEL__ -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.6.0/include -I/lib/modules/2.6.39-rc6-mmotm0506//build/include -I/lib/modules/2.6.39-rc6-mmotm0506//arch/x86/include/generated -include /lib/modules/2.6.39-rc6-mmotm0506//build/include/generated/autoconf.h -I/lib/modules/2.6.39-rc6-mmotm0506//build/arch/x86/include -I/lib/modules/2.6.39-rc6-mmotm0506//include/generated -c /tmp/kernconf.c In file included from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sem.h:81:0,

Quoted 10 lines
                 from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/sched.h:72,
                 from /lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/utsname.h:35,
                 from /tmp/kernconf.c:2:

/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h: In function '__kfree_rcu':
/lib/modules/2.6.39-rc6-mmotm0506//build/include/linux/rcupdate.h:822:2: error: size of unnamed array is negative

rm /tmp/kernconf.c
+ rm /tmp/kernconf.c

We use __always_inline for __Kfree_rcu(), we hope compiler does not compile it stand-alone, but it seems that __always_inline can't work without optimization, so you can add "-O2"(kernel build option) or "-Os" option for you third-party kernel modules' build.