Hanging using 'iptables -L' in UML vm booted with con0=pts

There is an outstanding kernel + mod_utils bug I am tracking via RedHat that exhibits similar behaviour - lockup with 100% CPU in modprobe.

https://bugzilla.redhat.com/show_bug.cgi?id=212922

Quoted 4 lines · Richard Andrews

There is an outstanding kernel + mod_utils bug I am tracking via RedHat that exhibits similar behaviour - lockup with 100% CPU in modprobe.

https://bugzilla.redhat.com/show_bug.cgi?id=212922

This one seems not to have anything to do with UML, while the bug that we're talking about here seems to, as modprobe sometimes works and sometimes doesn't.

This is the trace (as you can see I're repeated the experiment twice, the result is basically the same), hoping it could be useful to diagnose the problem and fix eventual bugs in the UML kernel. See below regarding the test procedure.

(gdb) bt

#0  0xb7f9d7f2 in ?? () from /lib/ld-linux.so.2
#1  0xb7f19ab0 in tcsetattr () from /lib/i686/cmov/libc.so.6
#2  0x080610d5 in generic_console_write (fd=7,
    buf=0x828fc13 "ip_tables: (C) 2000-2006 Netfilter Core Team\n", n=45)
    at arch/um/drivers/chan_user.c:37
#3  0x0806096e in console_write_chan (chans=0x82756f0,
    buf=0x828fc13 "ip_tables: (C) 2000-2006 Netfilter Core Team\n", len=45)
    at arch/um/drivers/chan_kern.c:352

Quoted 6 lines · Fermín Galán Márquez

This is the trace (as you can see I're repeated the experiment twice, the result is basically the same), hoping it could be useful to diagnose the problem and fix eventual bugs in the UML kernel. See below regarding the test procedure.

(gdb) bt

Quoted 8 lines

#0  0xb7f9d7f2 in ?? () from /lib/ld-linux.so.2
#1  0xb7f19ab0 in tcsetattr () from /lib/i686/cmov/libc.so.6
#2  0x080610d5 in generic_console_write (fd=7,
    buf=0x828fc13 "ip_tables: (C) 2000-2006 Netfilter Core Team\n", n=45)
    at arch/um/drivers/chan_user.c:37
#3  0x0806096e in console_write_chan (chans=0x82756f0,
    buf=0x828fc13 "ip_tables: (C) 2000-2006 Netfilter Core Team\n", len=45)
    at arch/um/drivers/chan_kern.c:352

Are you capturing UML output with a script or something? This is the signature of the script not reading from the pty, so everything just backs up, waiting for the pty to clear up.

Dear Jeff,

Quoted 16 lines · Jeff Dike

Quoted 6 lines · Fermín Galán Márquez

This is the trace (as you can see I're repeated the experiment twice, the result is basically the same), hoping it could be useful to diagnose the problem and fix eventual bugs in the UML kernel. See below regarding the test procedure.

(gdb) bt

Quoted 8 lines

#0  0xb7f9d7f2 in ?? () from /lib/ld-linux.so.2
#1  0xb7f19ab0 in tcsetattr () from /lib/i686/cmov/libc.so.6
#2  0x080610d5 in generic_console_write (fd=7,
    buf=0x828fc13 "ip_tables: (C) 2000-2006 Netfilter Core Team\n",
n=45) at arch/um/drivers/chan_user.c:37
#3  0x0806096e in console_write_chan (chans=0x82756f0,
    buf=0x828fc13 "ip_tables: (C) 2000-2006 Netfilter Core Team\n",

len=45) at arch/um/drivers/chan_kern.c:352

Quoted 3 lines

Are you capturing UML output with a script or something? This is the signature of the script not reading from the pty, so everything just backs up, waiting for the pty to clear up.

AFAIK, no script is capturing (I understand you refers to the output of the UML linux executable) and, as a matter of fact, I'm getting some output when execute ./linux. To be precise, this is the exact command (and the output) I get:

cibeles:/tmp# ./linux ubd0=/tmp/root_fs_debug con=null con0=pts
uml_dir=/tmp umid=run
Core dump limits :
         soft - 0
         hard - NONE
Checking that ptrace can change system call numbers...OK
Checking syscall emulation patch for ptrace...OK
Checking advanced syscall emulation patch for ptrace...OK
Checking for tmpfs mount on /dev/shm...OK
Checking PROT_EXEC mmap in /dev/shm/...OK
Checking for the skas3 patch in the host:
   - /proc/mm...not found: No such file or directory
   - PTRACE_FAULTINFO...not found
   - PTRACE_LDT...not found
UML running in SKAS0 mode
[no more output, no new prompt]

Then, in a second terminal in the host, I get the pts device and open a screen in the vm:

cibeles:/tmp# uml_mconsole run/mconsole config con0
Warning: couldn't stat file: /root/.uml/run/mconsole/mconsole - No such
file or directory
Warning: couldn't stat file: /tmp/uml/run/mconsole/mconsole - No such
file or directory
OK pts:/dev/pts/1
cibeles:/tmp# screen /dev/pts/1
[the terminal "goes into" the vm. I use it to cause the fail, as seen
below] Debian GNU/Linux lenny/sid vm tty0

vm login: root
Password:
Last login: Wed Sep 12 23:32:36 2007 on tty0
Linux vm 2.6.22.4-xt-2m #1 Wed Sep 12 20:24:20 CEST 2007 i686

The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law.

vm:~# iptables -L
[no output, linux process reach 100% CPU usage in the host]

In a third host terminal I run gdb:

cibeles:/tmp# cat run/pid
6627
cibeles:/tmp# gdb ./linux 6627

GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i486-linux-gnu"...Using host libthread_db library "/lib/i686/cmov/libthread_db.so.1".

Attaching to program: /tmp/linux, process 6627 Reading symbols from /lib/i686/cmov/libutil.so.1...done. Loaded symbols for /lib/i686/cmov/libutil.so.1 Reading symbols from /lib/i686/cmov/libc.so.6...done. Loaded symbols for /lib/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 0xb7f947f2 in ?? () from /lib/ld-linux.so.2 (gdb) bt
[showing the trace I've already sent in the other mail]

Maybe there are "other outputs" apart of the output that the ./linux produces which could be causing the problem?

Please try logging in via SSH and reproducing the problem and the stacktrace, and also removing con=null - also have you double checked con=null is ok (maybe it was con=none, I'm not sure). I'm not sure screen is perfectly safe to use (it should be).

Hi,

After some more debugging I think I've solved the problem (at least :) An explanation follows.

First, although apparently it seemed that the modules causing the problem were the ones related with networking (as David Fernández said in his mail) that assumption was wrong. In fact, the modules causing the problem are the ones that print some message in the "kernel message buffer" (I don't know what is the right name, but I'm referring to the message log that appears when the dmesg command is used) when modprobe load them. For example, ip_tables prints something like "ip_tables: (C) 2000-2006 Netfilter Core Team".

The solution to the problem is using con1 instead of con0, I mean:

./linux ubd0=/tmp/root_fs_debug con=null con1=pts uml_dir=/tmp umid=run

instead of

./linux ubd0=/tmp/root_fs_debug con=null con1=pts uml_dir=/tmp umid=run

In that case you can 'iptables -L' works without problems.

Why? When the module is loaded it prints its message not only in the internal kernel message buffer but also in con0 (this can be checked running simply "./linux ubd0=/tmp/root_fs_debug", without con= redirectors). When con0 is redirected to null (con0=null) there is no problem, but if con0 is redirected to a pts (con0=pts) I guess that, in the moment of printing the message, some problem occurs with the output (as Jeff suggest in its mail) thus causing the vm hang.

This explains also why putting the module in /etc/modules works. The loading of the list of modules in /etc/modules is performed before UML assign virtual console to pts devices (the sequence can be checked observing the booting log).

I think that my solution is more a workaround that a definitive solution. Why when con0 is assigned to a pts modules can not (because of it hangs the vm) print its message but when it is assigned to null it works? Is there a bug in the UML kernel that need to be fixed? Or maybe the bug is in modprobe? I leave the question open for the experts in the UML internals... :)

Regarding the tests suggested by Paolo:

Quoted 4 lines

Please try logging in via SSH and reproducing the problem and the stacktrace, and also removing con=null - also have you double checked con=null is ok (maybe it was con=none, I'm not sure). I'm not sure screen is perfectly safe to use (it should be).

Do you really need that I perform these test or considers the report above is enough? If it's really needed I can do them, but it would take me some time (and maybe now it isn't a good idea because of they won't provide additional useful information :)

Quoted 7 lines

The solution to the problem is using con1 instead of con0, I mean:

./linux ubd0=/tmp/root_fs_debug con=null con1=pts uml_dir=/tmp umid=run

instead of

./linux ubd0=/tmp/root_fs_debug con=null con1=pts uml_dir=/tmp umid=run

This was con0=pts, I checked your other mail. Right?

-----Mensaje original-----
De: Fermín Galán Márquez [mailto:[email protected]] Enviado el: miércoles, 19 de septiembre de 2007 21:42
Para: Paolo Giarrusso
CC: [email protected]; David Fernández
Asunto: Re: [uml-devel] (Workarounded) Hanging using 'iptables -L' in UML vm booted with con0=pts

Dear Paolo,

Quoted 1 line

Quoted 1 line

The solution to the problem is using con1 instead of

con0, I mean:

Quoted 10 lines

Quoted 8 lines

./linux ubd0=/tmp/root_fs_debug con=null con1=pts uml_dir=/tmp umid=run

instead of

./linux ubd0=/tmp/root_fs_debug con=null con1=pts uml_dir=/tmp umid=run

This was con0=pts, I checked your other mail. Right?

Absolutely :)

Best regards,

--------------------
Fermín Galán Márquez
CTTC - Centre Tecnològic de Telecomunicacions de Catalunya Parc Mediterrani de la Tecnologia, Av. del Canal Olímpic s/n, 08860 Castelldefels, Spain Room 1.02 Tel : +34 93 645 29 12 Fax : +34 93 645 29 01 Email address: fermin dot galan at cttc dot es