problems with ifup-a on etch compiled kernels

I don't know whether to blame a recent update to debian testing from stable --- likely it would be a glibc issue, but... I did run into the _syscall0() problem after that, and I found that I could get around it by appropriately #define _KERNEL prior to #include <linux/unistd.h>. What I don't get about the remark of Paolo is that _syscall0() is a userland thing, not a kernel thing.

I.e. any user program should be able to use _syscall() to get an open coded call to the kernel.

Anyway, my problem: 2.6.15, 2.6.17.13 (both patched as above), and 2.6.19-rc3 all fail during ifup -a:

+ echo -n 'Configuring network interfaces: '
Configuring network interfaces: + ifup -a

BUG: soft lockup detected on CPU#0!
083bf9f0:  [<080632a2>] dump_stack+0x22/0x30
083bfa08:  [<080a25c4>] softlockup_tick+0x84/0xa0
083bfa20:  [<0808ca32>] run_local_timers+0x12/0x20
083bfa28:  [<0808c796>] update_process_times+0x36/0x90
083bfa48:  [<080636fc>] timer_handler+0x3c/0x70
083bfa64:  [<080798f9>] sig_handler_common_skas+0xa9/0x100
083bfa88:  [<08075413>] real_alarm_handler+0x23/0x60
083bfaa0:  [<080754a2>] alarm_handler+0x52/0x70
083bfabc:  [<08077eda>] hard_handler+0x1a/0x20
083bfacc:  [<ffffe420>] _etext+0xf7df5404/0x0
083bfe14:  [<081a3431>] inet_ioctl+0x61/0xa0
083bfe2c:  [<081525a4>] sock_ioctl+0x144/0x2b0
083bfe60:  [<080d022f>] do_ioctl+0x5f/0x70
083bfe80:  [<080d04e2>] vfs_ioctl+0x62/0x2d0
083bfebc:  [<080d0791>] sys_ioctl+0x41/0x70
083bfee4:  [<08065a83>] handle_syscall+0xd3/0xe0
083bff54:  [<0807897a>] handle_trap+0x2a/0x130
083bff7c:  [<08079074>] userspace+0x1c4/0x220
083bffd4:  [<0806569e>] fork_handler+0x9e/0xb0
083bfffc:  [<00000000>] _sinittext+0xf7fb7000/0x20

If I boot with "linux emergency" login to the single user shell and run "ifup -a", everything is fine.

This is on my laptop running 2.6.17.4 as the host kernel, not under XenU as I previously had been posting about.

(I'm in an airport lounge at Chicago)

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

I've tracked this down to a number of places. It looks like the ifa->ifa_dev is not valid when the notification chain is called:

08707a08:  [<08074329>] sig_handler_common_skas+0xa9/0x120
08707a30:  [<0806ff25>] sig_handler+0x35/0x70
08707a4c:  [<080728ea>] hard_handler+0x1a/0x20
08707a5c:  [<ffffe420>] _etext+0xf7defac0/0x0
08707d64:  [<0808bcdc>] notifier_call_chain+0x6c/0x90
08707d94:  [<0808beb0>] blocking_notifier_call_chain+0x30/0x50
08707db0:  [<081a2d14>] __inet_insert_ifa+0xd4/0x160
08707dd4:  [<081a2dbd>] inet_insert_ifa+0x1d/0x20

The stack item <ffffe420>] _etext+0xf7defac0/0x0, seems to be bogus. I instrumented kernel/sys.c, to print the functions which notifier_call_chain was calling, and learnt that it was crashing in arch/um/drivers/net_kern.c because of:

	struct net_device *dev = ifa->ifa_dev->dev;

in uml_inetaddr_event.

sure enough, ifa->ifa_dev was NULL. Naturally, if I run things manually, or under GDB, it fails. Furthermore, this happens with 2.6.15 and 2.6.17.13 (patched to compile on etch), and with 2.6.19-rc3 (which I'm using as my debug base). I tried with gcc-3.3 and with gcc-4.1.2.

So, whatever is going on is related somehow to the glibc, (my guess), but represents some real bug that has been hidden for awhile.

I patched around the problem in uml_inetaddr_event (return immediately if ifa_dev==NULL), and found the next instance of it net/ipv4/fib_frontend.c, in fib_netdev_event.
Clearly, either we aren't initializing something right, or it's getting blown away at some point. Perhaps a different malloc policy in this glibc?

I will spend the rest of today on this (I'm at IETF in San Diego), but afterwards, I'll begin to start reverting to sarge (if I can), so that I can continue working on my real problem. I should be on IRC and in jabber. ([email protected], [email protected]).

(Hmm. I'm trying Mozilla news for gmane.org reading. I don't like the composer much, I have no idea what column I'm on...)

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

On a hunch, I replaced /sbin/ifup in my UML tree with the one from etch. ifup-a finishes now, but:

Configuring network interfaces: + ifup -a
ifup: failed to open statefile /etc/network/run/ifstate: No such file or directory
+ echo done.
done.
+ exit 0
Cleaning: /tmp /var/lock Trying to vfree() bad address (08978f40)

BUG: warning at mm/vmalloc.c:322/__vunmap()
0839bd04:  [<0805def2>] dump_stack+0x22/0x30
0839bd1c:  [<080b5d29>] __vunmap+0xb9/0x150
0839bd3c:  [<080b5dee>] vfree+0x2e/0x70
0839bd58:  [<080b6000>] __vmalloc_area_node+0x110/0x170
0839bd84:  [<080b60f6>] __vmalloc_node+0x66/0x70
0839bda0:  [<080b6128>] __vmalloc+0x28/0x30
0839bdb8:  [<080b6153>] vmalloc+0x23/0x30
0839bdcc:  [<0805ca61>] um_vmalloc+0x11/0x20
0839bdd8:  [<0806e278>] __wrap_malloc+0x38/0x60
0839bde8:  [<081cbaac>] __opendir+0xbc/0x120
0839be70:  [<081117a2>] open_dir+0x12/0x30
0839be80:  [<0811007b>] hostfs_readdir+0x3b/0xf0
0839becc:  [<080ce584>] vfs_readdir+0x74/0x90
0839bef0:  [<080cebf4>] sys_getdents64+0x84/0x110
0839bf28:  [<08060542>] handle_syscall+0x92/0xc0
0839bf7c:  [<08073a92>] userspace+0x202/0x370
0839bfe4:  [<080601b4>] fork_handler+0x74/0x90
0839bffc:  [<00352025>] 0x352025

I am not adding that. Maybe gmane.org or sf.net is doing that? sf.net, I guess.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

A UML built on debian sarge (machine name cassidy), (gcc-3.3.5, libc-2.3.2) works.
The same code built on debian etch (machine name marajade) (gcc-3.3.6 or gcc-4.1.2, glibc-2.3.6) fails.

A kernel built on the cassidy machine runs more successfully on the marajade machine. It does fail when shutting down, which it didn't on cassidy.

I conclude that it may well be that my host kernel on marade is at fault.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Can you tell me how to reproduce this, or send me a filesystem on which this happens?

Jeff

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeff> On Mon, Nov 06, 2006 at 01:44:56PM -0800, Michael Richardson Jeff> wrote:
>> I've tracked this down to a number of places. It looks like the >> ifa->ifa_dev is not valid when the notification chain is called:

Jeff> Can you tell me how to reproduce this, or send me a filesystem Jeff> on which this happens?

Unfortunately, I'm unable to determine what the key element is. It fails on my laptop, running identical user-mode-linux code and file
system that works on two devel machine I have. My laptop is is etch+2.6.17.4, (but I went back a 2.6.12.3 kernel I had, and it still failed). It works on a debian sarge+2.4.30 kernel, and a debian sarge+2.6.18.16 system.

I copied my / (not /usr) to the 2.6.18.16, setup a chroot, etc. and
ran the binaries compiled on my laptop against the file system there. No fault.

My laptop while at IETF has a public IP, so if you want I could make
you an account...

- --

]            Bear: "Me, I'm just the shape of a bear."          |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] [email protected]      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[

] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Finger me for keys

iQEVAwUBRVJiXYCLcPvd0N1lAQI2pAgAlKjw0034ueE0cP2h4BfXZuJoMDrIplh5 OPUxX/KR7rFSTkSwxgHLYdxJrb/7WanRf72kfU+8FZVZRxs/GDMwYClHnCPW1ufs IlrarkhKjTZVotE4Xt3CJGGqcjN5eyTDsWzikKDg0Gpkdn8icvZ/qVBFXsbygsMC cEuPhkytPuPzhhxd4nfs0gW7MYtgkapPajJ/Vi0ADo3umMn4q1mpK3+zhnAm5ilB frEfZVsojSsL9rmxllf3R05YfOCuDdN4CSAPBsijOZ2wtiR03dGY6aQ+dX1MGMQL z1SniQSh39cKae/YVHw/pK9amt4iUe1JjrxCB+1wt1XupHzrxODEtA== =iKlS
-----END PGP SIGNATURE-----

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

I think that I finally can.
I have replaced my file system on laptop with a fresh one, and my problem has gone away. It was the last thing to try. Clearly, if a userland change can cause a kernel oops, there is a problem.

I will try my broken file system on a working host, and if it reproduces the problem, I will tell you were to get the file system, and/or I can give you a login.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel