segfault on shutdown if containers running

lxc-devel Daniel Lezcano 4 participants 4 messages Jun 29, 2009 – Jul 2, 2009 11 messages not in the archive

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Daniel Lezcano

author

2009-06-29 08:44 UTC #4 lore raw

Quoted 29 lines · Adam Majer

Quoted 3 lines · Daniel Lezcano
Quoted 1 line
Is this a known problem? If not, I'll provide the backtrace.
Yes, please, maybe I am missing something but I was not able to reproduce it.
I see to be failing at being able to safe this segfault. The computer has no serial port so I've been trying to get netconsole to work, and it does, but the entire OOPS is only visible on local console, not transmitted. Only part of the oops is transmitted :(
[  635.526580] BUG: unable to handle kernel NULL pointer dereference at
000000000000009c
[  635.526692] IP: [<ffffffff803b26f1>] tty_open+0x1e6/0x411
[  635.526775] PGD 13dd44067 PUD 1379d5067 PMD 0
[  635.526905] Oops: 0000 [#1] SMP
[  635.526999] last sysfs file: /sys/devices/virtual/net/lo/operstate
[  635.527045] CPU 1
[  635.527110] Modules linked in: netconsole configfs veth nvidia(P)
binfmt_misc powernow_k8 cpufreq_powersave cpufreq_conservative cpufreq_stats cpufreq_userspace tun bridge stp ext2 mbcache it87 hwmon_vid loop firewire_sbp2 snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device usblp k8temp i2c_nforce2 pcspkr snd evdev asus_atk0110 i2c_core soundcore snd_page_alloc button processor xfs exportfs raid1 md_mod

Quoted 1 line

So, I've copied the stacktrace on paper and typing it back,

Thanks for taking the time to do that !

Quoted 11 lines

Process getty (pid:3940) <<<-- looks like a process in the container?
chrdev_open + 0x148/0x167
chrdev_open + 0x0/0x167
__dentry_open + 0x148/0x260
do_flip_open + 0x468/0x85a
alloc_fd + ....
do_sys_gen + ...
system_call_fastpath + ....
RIP tty_open

This stack trace reminds me something :)
Let me check if I can reproduce it with a small program.

Suka ? Are you aware of this bug ?

Thanks
-- Daniel

A message that is not in the archive (sent off-list or not yet received): [email protected]

Daniel Lezcano

2009-07-01 08:14 UTC #10 lore raw

Quoted 27 lines · Adam Majer

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
chrdev_open + 0x148/0x167
chrdev_open + 0x0/0x167
__dentry_open + 0x148/0x260
do_flip_open + 0x468/0x85a
alloc_fd + ....
do_sys_gen + ...
system_call_fastpath + ....
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
RIP tty_open
which kernel are you using ? could you run :
    $ addr2line -e <vmlinux> ffffffff803b26f1
Would setting this to yes produce a backtrace with line numbers?
Yes and maybe CONFIG_FRAME_POINTER too.
Well, adding symbols didn't add line numbers to the backtrace. But now I can use addr2line with the vmlinux (all 78mb of it) to get you guys a backtrace with line numbers :)
So,
OOPS (NULL pointer deference) at
drivers/char/tty_io.c:1321 (tty_open)

Quoted 8 lines

Backtrace,
fs/char_dev.c:397 (chrdev_open) fs/char_dev.c:357 (chrdev_open)
fs/open.c:841 (__dentry_open)
arch/x86/include/asm/atomic_64.h:117 (do_flip_open)
fs/file.c:459 (alloc_fd)
...

Excellent !

Suka ? Isn't this oops related to the newpts instance ?

Thanks
-- Daniel

A message that is not in the archive (sent off-list or not yet received): 4A4B1AF7.8090009-NmTC/[email protected]

Sukadev Bhattiprolu

2009-07-02 01:04 UTC #12 lore raw

Quoted 29 lines · Daniel Lezcano [dlezcano-NmTC/[email protected]] wrote:

Quoted 14 lines · Adam Majer
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
chrdev_open + 0x148/0x167
chrdev_open + 0x0/0x167
__dentry_open + 0x148/0x260
do_flip_open + 0x468/0x85a
alloc_fd + ....
do_sys_gen + ...
system_call_fastpath + ....
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
RIP tty_open
which kernel are you using ? could you run :
    $ addr2line -e <vmlinux> ffffffff803b26f1
Would setting this to yes produce a backtrace with line numbers?
Yes and maybe CONFIG_FRAME_POINTER too.
Quoted 13 lines
Well, adding symbols didn't add line numbers to the backtrace. But now I can use addr2line with the vmlinux (all 78mb of it) to get you guys a backtrace with line numbers :)
So,
OOPS (NULL pointer deference) at
drivers/char/tty_io.c:1321 (tty_open)
Backtrace,
fs/char_dev.c:397 (chrdev_open)
fs/char_dev.c:357 (chrdev_open)
fs/open.c:841 (__dentry_open)
arch/x86/include/asm/atomic_64.h:117 (do_flip_open)
fs/file.c:459 (alloc_fd)
...

Quoted 4 lines

Excellent !
Suka ? Isn't this oops related to the newpts instance ?

What version of the kernel are you running ?

Could it be this bug: http://lkml.org/lkml/2009/1/26/274
It was fixed by following commit and should be in 2.6.29.

	commit 808ffa3d302257b9dc37b1412c1fcdf976fcddac
	Author: Eric Paris <[email protected]>
	Date:   Tue Jan 27 11:50:37 2009 +0000

Sukadev

A message that is not in the archive (sent off-list or not yet received): 20090702010424.GA28523-r/[email protected]

Adam Majer

2009-07-02 04:29 UTC #14 lore raw

Quoted 37 lines · Sukadev Bhattiprolu

Quoted 28 lines · Daniel Lezcano [dlezcano-NmTC/[email protected]] wrote:
Quoted 14 lines · Adam Majer
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
chrdev_open + 0x148/0x167
chrdev_open + 0x0/0x167
__dentry_open + 0x148/0x260
do_flip_open + 0x468/0x85a
alloc_fd + ....
do_sys_gen + ...
system_call_fastpath + ....
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
RIP tty_open
which kernel are you using ? could you run :
    $ addr2line -e <vmlinux> ffffffff803b26f1
Would setting this to yes produce a backtrace with line numbers?
Yes and maybe CONFIG_FRAME_POINTER too.
Quoted 13 lines
Well, adding symbols didn't add line numbers to the backtrace. But now I can use addr2line with the vmlinux (all 78mb of it) to get you guys a backtrace with line numbers :)
So,
OOPS (NULL pointer deference) at
drivers/char/tty_io.c:1321 (tty_open)
Backtrace,
fs/char_dev.c:397 (chrdev_open)
fs/char_dev.c:357 (chrdev_open)
fs/open.c:841 (__dentry_open)
arch/x86/include/asm/atomic_64.h:117 (do_flip_open)
fs/file.c:459 (alloc_fd)
...
Quoted 4 lines
Excellent !
Suka ? Isn't this oops related to the newpts instance ?
What version of the kernel are you running ?

Quoted 8 lines

Could it be this bug: http://lkml.org/lkml/2009/1/26/274
It was fixed by following commit and should be in 2.6.29.
	commit 808ffa3d302257b9dc37b1412c1fcdf976fcddac
	Author: Eric Paris <[email protected]>
	Date:   Tue Jan 27 11:50:37 2009 +0000
Sukadev

It's a Debian stock 2.6.30-1 kernel, so the 2.6.30. The original OOPS was on the AMD64, but I've also just had a very similar OOPS in a 686 (32-bit) version of the same kernel.

The original oops (above) occurred when I did,

lxc-start -n container &
poweroff (in root system)

segfault occurred each time after the prompt "attempting to kill all processes...." then oops then a "failed" from the cleanup process. If the container is shut down prior to poweroff, no oops.

The following is all a 686 2.6.30 stock debian kernel,

Another OOPS (same function, slightly different backtrace),

1. ssh to root machine
2. lxc-start -n container &
3. lxc-console -n container -t 1 &
4. close ssh on originating machine
5. re-ssh into the machine

at this point I had a console into the container and to the root machine (where I ssh in) that were intermixed. Then the segfault occurred with the following backtrace. Debugging info not compiled into it, but it is similar symbols,

IP: tty_open + 0x1a5

      chrdev_open
      dput
      chrdev_open
      __dentry_open
      ....

Again, I had a prompt from within the container and from the parent machine when I re-ssh into the parent machine. So some ttys got intermixed.

I've also managed to generate a hard lockup of the kernel from within the container, but that seems to be related to networking. The process for that was,

1. bridge + veth interface
2. start container and networking up from within container
3. ssh into container
4. ifdown in cantainer
5. at this point the ssh session remained in spite of container
networking down!!
6. ifup container (new IP)
7. hard lockup occurred with the backtrace scrolled way off screen

- Adam

PS. Running lxc-checkconfig on the Debian kernel yields,

Found kernel config file /boot/config-2.6.30-1-686 --- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup namespace: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: disabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled

I can provide you with the complete kernel config if you want it.