i386: enable REGPARM by default

This patch should _not_ go into Linus' tree.

At some time in the future, we want to unconditionally enable REGPARM on i386.

Let's give it a bit broader testing coverage among -mm users.

This patch:
- removes the dependency of REGPARM on EXPERIMENTAL
- let REGPARM default to y

This patch assumes that people who use -mm are willing to test some more experimental features.

After this patch, REGPARM is still a config option users can disable.

---

This patch was already sent on:
- 30 May 2005
- 15 May 2005

@@ -911,9 +911,8 @@
 	default y
 
 config REGPARM
-	bool "Use register arguments (EXPERIMENTAL)"
-	depends on EXPERIMENTAL
-	default n
+	bool "Use register arguments"
+	default y
 	help
 	Compile the kernel with -mregparm=3. This uses a different ABI
 	and passes the first three arguments of a function call in registers.

hm, a compromise.

One other concern I have with this is that I expect -mregparm will make kgdb (and now crashdump) less useful. When incoming args are on the stack you have a good chance of being able to see what their value is by walking the stack slots.

When the incoming args are in registers I'd expect that it would be a lot harder (or impossible) to work out their value.

Have the kdump guys thought about (or encountered) this?

fre 2005-06-24 klockan 13:28 -0700 skrev Andrew Morton:

Hmmm. I played a bit with this. Without regparm there is some argument output although not correct. For example the argument 'ptr' to b_first should have been 0xbeef. both x=0 and y=299264 are incorrect. Not sure why, have to look into that.

#0 b_second (pid=1, ptr=0xbeef, x=0, y=299264) at arch/i386/kernel/process.c:180
#1 0xc0100ce8 in b_first (pid=1, ptr=0x1) at arch/i386/kernel/process.c:188 -------------- (only the above is interesting)

#2  0xc0100d66 in cpu_idle () at arch/i386/kernel/process.c:221
#3  0xc010027b in rest_init () at init/main.c:393
#4  0xc033e838 in start_kernel () at init/main.c:534
#5  0xc0100199 in is386 () at arch/i386/kernel/head.S:327

Now with regparm. Suprisingly enough only y=0 is now errenous and the rest are correct (double-checked, have to look into this aswell).

#0 b_second (pid=1, ptr=0xbeef, x=65297, y=0) at
arch/i386/kernel/process.c:180
#1 0xc0100c87 in b_first (pid=1, ptr=0xbeef) at
arch/i386/kernel/process.c:188
-------------- (only the above is interesting)

#2  0xc0100d01 in cpu_idle () at arch/i386/kernel/process.c:221
#3  0xc010026d in rest_init () at init/main.c:393
#4  0xc03247d1 in start_kernel () at init/main.c:534
#5  0xc0100199 in is386 () at arch/i386/kernel/head.S:327

This was at first glance but is interesting.

Adrian, why do we want REGPARM on by default? Performance? I haven't seen any figures

[copying to [email protected]]

I too played a little bit with this (gdb stack trace outputs without REGPARM and with REGPARM). Following are stack traces of both the cases.

REGPARM Disabled
----------------
(gdb) bt
#0 0xc0113d22 in crash_get_current_regs (regs=0xedf33e6c)

    at arch/i386/kernel/crash.c:102
#1  0xc0113d9d in crash_save_self (saved_regs=0xedf33e6c)
    at arch/i386/kernel/crash.c:134
#2  0xc013d19a in crash_kexec (regs=0x2) at kernel/kexec.c:1059
#3  0xc011c999 in panic (fmt=0x2 <Address 0x2 out of bounds>)
    at kernel/panic.c:86
#4  0xc014086a in func2 (a=2, b=0x2, c=100 'd') at kernel/ksysfs.c:36
#5  0xc01408a5 in func1 (a=20, b=0xc044cae8, c=100 'd') at kernel/ksysfs.c:42
#6  0xc01408f8 in stack_debug_show (subsys=0xc044cb00,
    page=0x2 <Address 0x2 out of bounds>) at kernel/ksysfs.c:59
#7  0xc0198e3f in subsys_attr_show (kobj=0xc044cb18, attr=0x2,
    page=0x2 <Address 0x2 out of bounds>) at fs/sysfs/file.c:30
#8  0xc0198ed8 in fill_read_buffer (dentry=0x1, buffer=0xee33db20)
    at fs/sysfs/file.c:86
#9  0xc0198fe7 in sysfs_read_file (file=0x1,
    buf=0x2 <Address 0x2 out of bounds>, count=2, ppos=0x2)
    at fs/sysfs/file.c:153
#10 0xc0160599 in vfs_read (file=0xef3d8880,
    buf=0x804d858 <Address 0x804d858 out of bounds>, count=4096,
    pos=0xedf33fa4) at fs/read_write.c:238
#11 0xc0160871 in sys_read (fd=2, buf=0x2 <Address 0x2 out of bounds>,
count=2)
    at fs/read_write.c:321

Here all the function arguments does not seem to be right. I introduced func1 and func2 and value of parameters is partially correct. (Right values are a=20, b=0xecab5f18,, c='d').

Time to talk to gdb folks.

With REGPARM Enabled
--------------------
(gdb) bt
#0 0xc0112a20 in crash_get_current_regs (regs=0xee661e94)

    at arch/i386/kernel/crash.c:102
#1  0xc0112a89 in crash_save_self (saved_regs=0x0)
    at arch/i386/kernel/crash.c:134

#2 0xc0138be2 in crash_kexec (regs=0x5400) at kernel/kexec.c:1059 #3 0xc011adeb in panic (fmt=0xc03b7499 "Vivek: Invoked panic\n")
at kernel/panic.c:86
#4 0xc013beab in func2 (a=-295297388, b=0x5400, c=-108 '\224')
at kernel/ksysfs.c:36
#5 0xc013bedb in func1 (a=-1069483288, b=0x14, c=-108 '\224')
at kernel/ksysfs.c:42
#6 0xc013bf36 in stack_debug_show (subsys=0xee661e94, page=0xede0f000 "")
at kernel/ksysfs.c:60
#7 0xc018d31c in subsys_attr_show (kobj=0xee661e94, attr=0x5400,

    page=0xede0f000 "") at fs/sysfs/file.c:30
#8  0xc018d38e in fill_read_buffer (dentry=0xee661e94, buffer=0xefd32360)
    at fs/sysfs/file.c:86
#9  0xc018d481 in sysfs_read_file (file=0xee0909c0,
    buf=0x804d858 <Address 0x804d858 out of bounds>, count=4096,
    ppos=0xee661e94) at fs/sysfs/file.c:153
#10 0xc0159260 in vfs_read (file=0xee0909c0,
    buf=0x804d858 <Address 0x804d858 out of bounds>, count=4096,
    pos=0xee661fa4) at fs/read_write.c:238
#11 0xc01594e9 in sys_read (fd=3999669908,
    buf=0x804d858 <Address 0x804d858 out of bounds>, count=4096)
    at fs/read_write.c:321

Here all the arguments of func1 and func2 are bad. Looks like REGPARM might worsen the output.

Will look more into this.

Thanks
Vivek

GDB is more than capable of handling this - if your compiler is saving arguments to the stack and dumping out useful information for the debugger about where it put them. Recent GCC versions are generally pretty good about either saving the argument or clearly telling GDB that it was not saved.

Daniel Jacobowitz
CodeSourcery, LLC

Thanks. Any idea what might be amiss with my case where I am not seeing proper function parameter values while analyzing kdump generated crash dump with gdb. I am using following gdb and gcc versions.

GNU gdb Red Hat Linux (6.1post-1.20040607.62rh)
gcc (GCC) 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)

Inlined with the mail is a test patch. This patch just invokes func1() and func2() upon reading a sysfs file "debug_stack" and finally calls panic() and boots into a new kernel.

Associated stack traces retrieved from core dump file are available at following link.

http://marc.theaimsgroup.com/?l=linux-kernel&m=111988996408170&w=2

Thanks
Vivek

---

 linux-2.6.12-rc6-mm1-1M-root/kernel/ksysfs.c |   24 ++++++++++++++++++++++++
 1 files changed, 24 insertions(+)

diff -puN kernel/ksysfs.c~kdump-gdb-stack-debug kernel/ksysfs.c
@@ -30,6 +30,19 @@ static ssize_t hotplug_seqnum_show(struc
 KERNEL_ATTR_RO(hotplug_seqnum);
 #endif
 
+int func2(int a, int *b, char c)
+{
+        printk("a=%d, b=%p, c=%c \n", a, b, c);
+	panic("Vivek: Invoked panic\n");
+	return 0;
+}
+int func1(int a, int *b, char c)
+{
+        printk("a=%d, b=%p, c=%c\n", a, b, c);
+        func2(a, b, c);
+	return 0;
+}
+
 #ifdef CONFIG_KEXEC
 #include <asm/kexec.h>
 

@@ -38,6 +51,16 @@ static ssize_t crash_notes_show(struct s
 	return sprintf(page, "%p\n", (void *)crash_notes);
 }
 KERNEL_ATTR_RO(crash_notes);
+static ssize_t stack_debug_show(struct subsystem *subsys, char *page)
+{
+	int a=20;
+	int *b=&a;
+	char c='d';
+	printk("Vivek: value of b is %p\n", b);
+	func1(a, b, c);
+	return sprintf(page, "%s\n", "Vivek copied");
+}
+KERNEL_ATTR_RO(stack_debug);
 #endif
 
 decl_subsys(kernel, NULL, NULL);

@@ -49,6 +72,7 @@ static struct attribute * kernel_attrs[]
 #endif
 #ifdef CONFIG_KEXEC
 	&crash_notes_attr.attr,
+	&stack_debug_attr.attr,
 #endif
 	NULL
 };

_

Some more info. I dumped the stack contents and it seems that stack is fine and parameters are intact on stack. So now it seems to be a matter of how gdb is interpreting the stack contents. Any guess, what the problem is? Why func2() and func1() are not showing right parameter values.

Unfortunately I lost the previous dump image and vmlinux. I have taken a fresh dump and reproduced the problem again. So I am reporting all the details again.

- Test Patch
- gdb back trace
- Raw stack dump
- Relevant vmlinux objdump output.

Thanks
Vivek

Following is the test patch I applied on 2.6.12-rc6-mm1.

---

 linux-2.6.12-rc6-mm1-1M-root/kernel/ksysfs.c |   23 +++++++++++++++++++++++
 1 files changed, 23 insertions(+)

diff -puN kernel/ksysfs.c~kdump-gdb-stack-debug kernel/ksysfs.c
@@ -30,6 +30,19 @@ static ssize_t hotplug_seqnum_show(struc
 KERNEL_ATTR_RO(hotplug_seqnum);
 #endif
 
+int func2(int a, int *b, char c)
+{
+	printk("a=%d, b=%p, c=%c \n", a, b, c);
+	panic("Vivek: Invoked panic\n");
+	return 0;
+}
+int func1(int a, int *b, char c)
+{
+	printk("a=%d, b=%p, c=%c\n", a, b, c);
+	func2(a, b, c);
+	return 0;
+}
+
 #ifdef CONFIG_KEXEC
 #include <asm/kexec.h>
 

@@ -38,6 +51,15 @@ static ssize_t crash_notes_show(struct s
 	return sprintf(page, "%p\n", (void *)crash_notes);
 }
 KERNEL_ATTR_RO(crash_notes);
+static ssize_t stack_debug_show(struct subsystem *subsys, char *page)
+{
+	int a=20;
+	int *b=&a;
+	char c='d';
+	func1(a, b, c);
+	return sprintf(page, "%s\n", "copied");
+}
+KERNEL_ATTR_RO(stack_debug);
 #endif
 
 decl_subsys(kernel, NULL, NULL);

@@ -49,6 +71,7 @@ static struct attribute * kernel_attrs[]
 #endif
 #ifdef CONFIG_KEXEC
 	&crash_notes_attr.attr,
+	&stack_debug_attr.attr,
 #endif
 	NULL
 };

Following is the stack trace I received from gdb when it opened kdump generated kernel dump image.

(gdb) bt
#0 0xc0113d22 in crash_get_current_regs (regs=0xec9cfe6c)

    at arch/i386/kernel/crash.c:102
#1  0xc0113d9d in crash_save_self (saved_regs=0xec9cfe6c)
    at arch/i386/kernel/crash.c:134
#2  0xc013d19a in crash_kexec (regs=0x3) at kernel/kexec.c:1059
#3  0xc011c999 in panic (fmt=0x3 <Address 0x3 out of bounds>)
    at kernel/panic.c:86
#4  0xc014086a in func2 (a=3, b=0x3, c=100 'd') at kernel/ksysfs.c:36
#5  0xc01408a5 in func1 (a=20, b=0xc044cae8, c=100 'd') at kernel/ksysfs.c:42
#6  0xc01408f8 in stack_debug_show (subsys=0xc044cb00,
    page=0x3 <Address 0x3 out of bounds>) at kernel/ksysfs.c:59
#7  0xc0198e3f in subsys_attr_show (kobj=0xc044cb18, attr=0x3,
    page=0x3 <Address 0x3 out of bounds>) at fs/sysfs/file.c:30
#8  0xc0198ed8 in fill_read_buffer (dentry=0x1, buffer=0xefd9bda0)
    at fs/sysfs/file.c:86
#9  0xc0198fe7 in sysfs_read_file (file=0x1,
    buf=0x3 <Address 0x3 out of bounds>, count=3, ppos=0x3)
    at fs/sysfs/file.c:153
#10 0xc0160599 in vfs_read (file=0xedafd560,
    buf=0x804d858 <Address 0x804d858 out of bounds>, count=4096,
    pos=0xec9cffa4) at fs/read_write.c:238
#11 0xc0160871 in sys_read (fd=3, buf=0x3 <Address 0x3 out of bounds>, count=3)
    at fs/read_write.c:321

Following is the raw stack dump. I have tried segregating/mapping calls to func1 and func2.

(gdb) info registers

eax            0x3      3
ecx            0xec9cfe6c       -325255572
edx            0x1      1
ebx            0xec9cfe6c       -325255572
esp            0xec9cfe60       0xec9cfe60
ebp            0xc044cae8       0xc044cae8
esi            0x3      3
edi            0x14     20
eip            0xc0113d22       0xc0113d22
eflags         0x46     70
cs             0x60     96
ss             0x68     104
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x33     51

(gdb) x/100x 0xec9cfe60

0xec9cfe60:     0xc0113d9d      0xec9cfe6c      0x00000003      0xec9cfe6c
0xec9cfe70:     0xec9cfe6c      0x00000001      0x00000003      0x00000000
0xec9cfe80:     0xc044c000      0x00000003      0x00000000      0xc0113a75
0xec9cfe90:     0x00000004      0x013ef000      0x01455498      0x00000001
0xec9cfea0:     0x013ef000      0xefc948a0      0xec9cff1c      0x00000014
0xec9cfeb0:     0xc044cae8      0xc013d19a      0xefc948a0      0x00000064
0xec9cfec0:     0xc011c999      0x00000000      0xc054e140      0xc03f3d4e
0xec9cfed0:     0xec9cfee0      0x00000064      0xc014086a      0xc03f3d39
0xec9cfee0:     0x00000014
                0xec9cff1c
                0x00000064
                0xc01408a5      (Call to func2)
0xec9cfef0:     0x00000014      (parameter a)
                0xec9cff1c      (parameter b)
                0x00000064      (parameter c)
                0x00000064
0xec9cff00:     0xc044cb00   
                0xc044cb18  
                0xc04537a0 
                0xc01408f8      (Call to func1)
0xec9cff10:     0x00000014      (parameter a)
                0xec9cff1c      (parameter b)
                0x00000064      (parameter c)
                0x00000014

0xec9cff20: 0xc0198e3f 0xc044cb00 0xeccfc000 0xefd9bda0

Following is objdump output of vmlinux. (Snippet of relevant portion)

c0140836 <func2>:

c0140836:       83 ec 10                sub    $0x10,%esp
c0140839:       0f be 44 24 1c          movsbl 0x1c(%esp),%eax
c014083e:       c7 04 24 26 3d 3f c0    movl   $0xc03f3d26,(%esp)
c0140845:       89 44 24 0c             mov    %eax,0xc(%esp)
c0140849:       8b 44 24 18             mov    0x18(%esp),%eax
c014084d:       89 44 24 08             mov    %eax,0x8(%esp)
c0140851:       8b 44 24 14             mov    0x14(%esp),%eax
c0140855:       89 44 24 04             mov    %eax,0x4(%esp)
c0140859:       e8 b6 c9 fd ff          call   c011d214 <printk>
c014085e:       c7 04 24 39 3d 3f c0    movl   $0xc03f3d39,(%esp)
c0140865:       e8 c5 c0 fd ff          call   c011c92f <panic>

c014086a <func1>:

c014086a:       57                      push   %edi
c014086b:       56                      push   %esi
c014086c:       53                      push   %ebx
c014086d:       83 ec 10                sub    $0x10,%esp
c0140870:       8b 7c 24 20             mov    0x20(%esp),%edi
c0140874:       8b 74 24 24             mov    0x24(%esp),%esi
c0140878:       0f be 5c 24 28          movsbl 0x28(%esp),%ebx
c014087d:       89 5c 24 0c             mov    %ebx,0xc(%esp)
c0140881:       89 74 24 08             mov    %esi,0x8(%esp)
c0140885:       89 7c 24 04             mov    %edi,0x4(%esp)
c0140889:       c7 04 24 4f 3d 3f c0    movl   $0xc03f3d4f,(%esp)
c0140890:       e8 7f c9 fd ff          call   c011d214 <printk>
c0140895:       89 5c 24 08             mov    %ebx,0x8(%esp)
c0140899:       89 74 24 04             mov    %esi,0x4(%esp)
c014089d:       89 3c 24                mov    %edi,(%esp)
c01408a0:       e8 91 ff ff ff          call   c0140836 <func2>
c01408a5:       31 c0                   xor    %eax,%eax
c01408a7:       83 c4 10                add    $0x10,%esp
c01408aa:       5b                      pop    %ebx
c01408ab:       5e                      pop    %esi
c01408ac:       5f                      pop    %edi
c01408ad:       c3                      ret

c01408d1 <stack_debug_show>:

c01408d1:       83 ec 10                sub    $0x10,%esp
c01408d4:       8d 44 24 0c             lea    0xc(%esp),%eax
c01408d8:       c7 44 24 0c 14 00 00    movl   $0x14,0xc(%esp)
c01408df:       00
c01408e0:       c7 44 24 08 64 00 00    movl   $0x64,0x8(%esp)
c01408e7:       00
c01408e8:       89 44 24 04             mov    %eax,0x4(%esp)
c01408ec:       c7 04 24 14 00 00 00    movl   $0x14,(%esp)
c01408f3:       e8 72 ff ff ff          call   c014086a <func1>
c01408f8:       8b 44 24 18             mov    0x18(%esp),%eax
c01408fc:       c7 44 24 08 6d 3d 3f    movl   $0xc03f3d6d,0x8(%esp)
c0140903:       c0
c0140904:       c7 44 24 04 36 67 40    movl   $0xc0406736,0x4(%esp)
c014090b:       c0
c014090c:       89 04 24                mov    %eax,(%esp)
c014090f:       e8 64 94 0c 00          call   c0209d78 <sprintf>
c0140914:       83 c4 10                add    $0x10,%esp
c0140917:       c3                      ret

Date: Tue, 28 Jun 2005 16:54:12 +0530
From: Vivek Goyal <[email protected]>

Some more info. I dumped the stack contents and it seems that stack is fine and parameters are intact on stack. So now it seems to be a matter of how gdb is interpreting the stack contents. Any guess, what the problem is?

I'd say the problem is with a user building stuff with non-standard "optimizations", probably even stripping his executable, and expecting to be able to debug the result.

Why func2() and func1() are not showing right parameter values.

Repeating what Daniel said before, by using "regparm", function arguments are now passed in registers instead of on the stack. It's extremely unlikely that these function arguments will stay in those registers for ever, especially since you've only got a handfull of them on the i386. So eventually they will be moved to some other register or, more likely, to memory. If the compiler doesn't tell gdb about it, gdb will still think the value is in the register, and display whatever what's there now, which is likely to be the wrong value. There are two ways the compiler can tell gdb where things are:

1. By explicitly specifying the new location. Both DWARF 2 and stabs debugging formats can do this, but AFAIK, GCC won't do this if a register is spilled to the stack.

2. By specifying where registers are saved. Only DWARF 2 can do this.

We've seen cases where the information generated by GCC for 1 or 2 is either incomplete or wrong. There also have been cases where GDB didn't interpret that information correctly. And then some people tend to remove some of the debug information by stripping their code or using broken linker scripts. You'll need to find out where the problem is, but my bet is that its's a problem with GCC since you make it generate non-standard code.

Oh, by the way, don't expect gdb to be able to call "regparm" functions either.

Mark

In this case I am building linux kernel with debug info (-g) and -mregparm is not specified. So parameters should be passed on stack. Following is the effective command line to build kernel/sysfs.c. I am not sure if any of the below mentioned options are going to affect the gdb results.

gcc -m32 -Wp,-MD,kernel/.ksysfs.o.d -nostdinc -isystem /usr/lib/gcc/i386-redhat-linux/3.4.3/include -D__KERNEL__ -Iinclude -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -O2 -fomit-frame-pointer -g -pipe -msoft-float -mpreferred-stack-boundary=2 -fno-unit-at-a-time -march=i686 -mtune=pentium4 -Iinclude/asm-i386/mach-default -Wdeclaration-after-statement -DKBUILD_BASENAME=ksysfs -DKBUILD_MODNAME=ksysfs -c -o kernel/ksysfs.o kernel/ksysfs.c

Sorry for the confusion. In the last mail all the results were reported with REGPARM disabled. I wanted to make sure that first normal case works fine and then discuss the REGPARM case later.

Ok. I will try to figure out where the problem is. Time to jump into gcc and gdb code. :-)

Thanks
Vivek

If you're prepared to do some more tests, you might want to check out what happens if you leave out -O2 and -fomit-frame-pointer, and then add back only -O2

Mark

I built another kernel with -fno-omit-frame-pointer and output seems to have worsen a lot now. I am not able to build a kernel without -02. There seems to be some dependencies which I am sorting out.

With frame pointer support, following is the command line.

gcc -m32 -Wp,-MD,kernel/.ksysfs.o.d -nostdinc -isystem /usr/lib/gcc/i386-redhat-linux/3.4.3/include -D__KERNEL__ -Iinclude -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -O2 -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -pipe -msoft-float -mpreferred-stack-boundary=2 -fno-unit-at-a-time -march=i686 -mtune=pentium4 -Iinclude/asm-i386/mach-default -Wdeclaration-after-statement -DKBUILD_BASENAME=ksysfs -DKBUILD_MODNAME=ksysfs -c -o kernel/ksysfs.o kernel/ksysfs.c

And the gdb trace has worsened. Trace is not even showing all the calls as it was showing when kernel was built with -fomit-frame-pointer.

#0 crash_get_current_regs (regs=0xec3b5e34) at arch/i386/kernel/crash.c:103 #1 0xc0114077 in crash_save_self (saved_regs=0xec3b5e34)

    at arch/i386/kernel/crash.c:134
#2  0xec3b5f04 in ?? ()
#3  0x00000014 in ?? ()
#4  0xec3b5e98 in ?? ()
#5  0xc013d7e6 in crash_kexec (regs=0x2) at kernel/kexec.c:1059
#6  0xecb29c80 in ?? ()
#7  0xecb29c80 in ?? ()
#8  0x00000000 in ?? ()

Thanks
Vivek

Jens Axboe had hit an obscure bug with regparm just yesterday. It happened for him with gcc 3.3.5.

I have a preprocessed .c file which allows to reporduce this. For me, gcc 3.3.6 is okay. need to build 3.3.5 and test.

Meanwhile, maybe we shall prohibit regparm if gcc <=3.3.6 or 3.4? --
vda

It triggered without regparm as well, so I don't think that is to blame here.

Jens Axboe