Oops in CDC ACM after an Openmoko phone (neo1973) changes its usb interface from ACM to Ethernet
Starting since about 2.6.31-rc2, kernel oopses when I boot a kernel from Openmoko's bootloader. The bootloader is connected through CDC ACM, and when the kernel boots, the usb device changes to usb ethernet. Shortly after that kernel oopses:
[ 29.346931] usb 5-1: USB disconnect, address 2 [ 29.347343] usb0: unregister 'cdc_ether' usb-0000:00:1d.0-1, CDC Ethernet Device [ 30.573530] usb 5-1: new full speed USB device using uhci_hcd and address 4 [ 30.750689] usb 5-1: New USB device found, idVendor=1457, idProduct=5119 [ 30.750795] usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 30.750923] usb 5-1: Product: Neo1973 Bootloader U-Boot 1.3.2-rc2 [ 30.751005] usb 5-1: Manufacturer: OpenMoko, Inc [ 30.751071] usb 5-1: SerialNumber: 0000000 [ 30.752032] usb 5-1: configuration #1 chosen from 2 choices [ 30.793924] cdc_acm 5-1:1.0: This device cannot do calls on its own. It is not a modem. [ 30.794103] cdc_acm 5-1:1.0: ttyACM0: USB ACM device [ 30.799509] usbcore: registered new interface driver cdc_acm [ 30.799610] cdc_acm: v0.26:USB Abstract Control Model driver for USB modems and ISDN adapters [ 47.346852] usb 5-1: USB disconnect, address 4 [ 47.352989] tty_port_close_start: count = -1 [ 48.358107] tty_port_close_start: count = -1 [ 49.362088] tty_port_close_start: count = -1 [ 50.365975] tty_port_close_start: count = -1 [ 51.369831] tty_port_close_start: count = -1 [ 52.373640] tty_port_close_start: count = -1 [ 53.377348] tty_port_close_start: count = -1 [ 54.381038] tty_port_close_start: count = -1 [ 55.384717] tty_port_close_start: count = -1 [ 56.388336] tty_port_close_start: count = -1 [ 57.391932] tty_port_close_start: count = -1 [ 58.395485] tty_port_close_start: count = -1 [ 59.399000] tty_port_close_start: count = -1 [ 60.400401] tty_port_close_start: count = -1 [ 61.401795] tty_port_close_start: count = -1 [ 62.405239] tty_port_close_start: count = -1 [ 63.408676] tty_port_close_start: count = -1 [ 64.412031] tty_port_close_start: count = -1 [ 65.415241] tty_port_close_start: count = -1 [ 66.418339] tty_port_close_start: count = -1 [ 67.421433] tty_port_close_start: count = -1 [ 68.424520] tty_port_close_start: count = -1 [ 69.427630] tty_port_close_start: count = -1 [ 70.430787] tty_port_close_start: count = -1 [ 71.433903] tty_port_close_start: count = -1 [ 72.437040] tty_port_close_start: count = -1 [ 73.440171] tty_port_close_start: count = -1 [ 74.441520] tty_port_close_start: count = -1 [ 75.073519] usb 5-1: new full speed USB device using uhci_hcd and address 5 [ 75.271699] usb 5-1: New USB device found, idVendor=1457, idProduct=5122 [ 75.271709] usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 75.271716] usb 5-1: Product: RNDIS/Ethernet Gadget [ 75.271722] usb 5-1: Manufacturer: Linux 2.6.24/s3c2410_udc [ 75.272788] usb 5-1: configuration #1 chosen from 2 choices [ 75.288865] usb0: register 'cdc_ether' at usb-0000:00:1d.0-1, CDC Ethernet Device, 1a:e0:cd:b1:57:c0 [ 75.442858] tty_port_close_start: count = -1 [ 75.442881] general protection fault: 0000 [#1] SMP [ 75.444775] last sysfs file: /sys/devices/pci0000:00/0000:00:1d.0/usb5/5-1/uevent [ 75.446041] CPU 0 [ 75.446041] Modules linked in: cdc_acm uvcvideo cdc_ether usbnet ohci1394 wmi videodev v4l1_compat v4l2_compat_ioctl32 ieee1394 mii [ 75.446041] Pid: 2127, comm: minicom Not tainted 2.6.31-rc2-t #64 XPS M1330 [ 75.446041] RIP: 0010:[<ffffffff812a76e1>] [<ffffffff812a76e1>] usb_autopm_do_interface+0x35/0xeb [ 75.453444] RSP: 0018:ffff88007d05faf8 EFLAGS: 00010292 [ 75.453444] RAX: 656369766564646a RBX: 00000000ffffffed RCX: 0000000000000000 [ 75.453444] RDX: ffffffffa00755b8 RSI: 0000000000000001 RDI: 656369766564646a [ 75.453444] RBP: ffff88007d05fb28 R08: 0000000000000001 R09: 0000000000000000 [ 75.453444] R10: ffffffff810594a7 R11: 0000000000000082 R12: ffff88007c527800 [ 75.453444] R13: 6563697665645ed2 R14: 6563697665645f62 R15: 0000000000000001 [ 75.453444] FS: 00007fe65d7916f0(0000) GS:ffff880001e56000(0000) knlGS:0000000000000000 [ 75.453444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 75.453444] CR2: 00007f83ba6c3ce0 CR3: 000000007d0e1000 CR4: 00000000000006f0 [ 75.453444] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 75.453444] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 75.453444] Process minicom (pid: 2127, threadinfo ffff88007d05e000, task ffff88007dd33da0) [ 75.453444] Stack: [ 75.453444] 656369766564646a 0000000000000000 ffff88007dead000 ffff88007dead000 [ 75.453444] <0> 0000000000000000 0000000000000010 ffff88007d05fb38 ffffffff812a77b2 [ 75.453444] <0> ffff88007d05fbc8 ffffffffa0072e19 0000000000000282 ffff88007d2f7000 [ 75.453444] Call Trace: [ 75.453444] [<ffffffff812a77b2>] usb_autopm_get_interface+0xe/0x10 [ 75.453444] [<ffffffffa0072e19>] acm_port_down+0x3f/0x1bb [cdc_acm] [ 75.453444] [<ffffffff811f0af1>] ? tty_port_close_start+0xc1/0x153 [ 75.453444] [<ffffffffa0073258>] acm_tty_close+0x3d/0x83 [cdc_acm] [ 75.453444] [<ffffffff811eaca7>] tty_release_dev+0x1bf/0x4e8 [ 75.453444] [<ffffffffa00732c3>] ? acm_tty_open+0x25/0x2ae [cdc_acm] [ 75.453444] [<ffffffff8141e0f2>] ? mutex_unlock+0x9/0xb [ 75.453444] [<ffffffffa00734fc>] ? acm_tty_open+0x25e/0x2ae [cdc_acm] [ 75.453444] [<ffffffff811e8346>] ? check_tty_count+0x21/0xac [ 75.453444] [<ffffffff811eb55d>] tty_open+0x33c/0x427 [ 75.453444] [<ffffffff810c3a81>] chrdev_open+0x150/0x16f [ 75.453444] [<ffffffff8141fba7>] ? _spin_unlock+0x26/0x2a [ 75.453444] [<ffffffff810c1a44>] ? file_move+0x1e/0x50 [ 75.453444] [<ffffffff810c3931>] ? chrdev_open+0x0/0x16f [ 75.453444] [<ffffffff810bf594>] __dentry_open+0x150/0x26f [ 75.453444] [<ffffffff810bf780>] nameidata_to_filp+0x41/0x52 [ 75.453444] [<ffffffff810cc55e>] do_filp_open+0x4ee/0x9ce [ 75.453444] [<ffffffff810d5700>] ? alloc_fd+0x11d/0x12e [ 75.453444] [<ffffffff810bf359>] do_sys_open+0x5d/0x10b [ 75.453444] [<ffffffff810bf430>] sys_open+0x1b/0x1d [ 75.453444] [<ffffffff8100baab>] system_call_fastpath+0x16/0x1b [ 75.453444] Code: 56 41 55 41 54 49 89 fc 53 bb ed ff ff ff 48 83 ec 08 4c 8b 77 30 49 8d 86 08 05 00 00 4d 8d ae 70 ff ff ff 48 89 45 d0 48 89 c7 <41> 0f b6 b5 6b 03 00 00 e8 00 6e 17 00 41 83 7c 24 24 00 0f 84 [ 75.453444] RIP [<ffffffff812a76e1>] usb_autopm_do_interface+0x35/0xeb [ 75.453444] RSP <ffff88007d05faf8> [ 75.455529] ---[ end trace 03be7af14b545501 ]---
Full dmesg attached.
The later kernels have multiple oopses in a row and lock hard, so I could not get
the oopses for something more recent than "something past 2.6.31-rc2". I'll try to bisect, by it'll take some time.
Am Sonntag, 2. August 2009 18:40:56 schrieb Alex Riesen:
Quoted 7 lines
[ 75.453444] Call Trace:
[ 75.453444] [<ffffffff812a77b2>]usb_autopm_get_interface+0xe/0x10[ 75.453444] [<ffffffffa0072e19>]acm_port_down+0x3f/0x1bb[cdc_acm] [ 75.453444] [<ffffffff811f0af1>] ?tty_port_close_start+0xc1/0x153[ 75.453444] [<ffffffffa0073258>]acm_tty_close+0x3d/0x83[cdc_acm] [ 75.453444] [<ffffffff811eaca7>]tty_release_dev+0x1bf/0x4e8[ 75.453444] [<ffffffffa00732c3>] ?acm_tty_open+0x25/0x2ae[cdc_acm]
This can happen only if acm_port_down() fails to detect that the device has been disconnected, as it calls usb_autopm_get_interface()
only if acm->dev != NULL.
This however should not happen if acm_disconnect runs.
Can you put a printk into acm_disconnect and recompile with CONFIG_USB_DEBUG?
Regards
Oliver
Quoted 9 lines · Oliver Neukum
Am Sonntag, 2. August 2009 18:40:56 schrieb Alex Riesen:
Quoted 7 lines
[ 75.453444] Call Trace:
[ 75.453444] [<ffffffff812a77b2>]usb_autopm_get_interface+0xe/0x10[ 75.453444] [<ffffffffa0072e19>]acm_port_down+0x3f/0x1bb[cdc_acm] [ 75.453444] [<ffffffff811f0af1>] ?tty_port_close_start+0xc1/0x153[ 75.453444] [<ffffffffa0073258>]acm_tty_close+0x3d/0x83[cdc_acm] [ 75.453444] [<ffffffff811eaca7>]tty_release_dev+0x1bf/0x4e8[ 75.453444] [<ffffffffa00732c3>] ?acm_tty_open+0x25/0x2ae[cdc_acm]
Quoted 6 lines
This can happen only if
acm_port_down()fails to detect that the device has been disconnected, as it callsusb_autopm_get_interface()
only ifacm->dev!= NULL.
This however should not happen ifacm_disconnectruns.Can you put a printk into
acm_disconnectand recompile withCONFIG_USB_DEBUG?
I put the printks at some random lines in acm_disconnect (as of v2.6.31-rc2) and turned CONFIG_USB_DEBUG on. It is definitely not the same kernel I tried before (and sadly, it does not go oops), but it still hastty_port_close_start: count = -1
Dmesg attached
I'm retrying with recent kernels (hopefully I'll get enough data left on the screen),
I shall try to find the -rc2 which oopses (I suspect it is the one just before -rc3).
Quoted 3 lines · Alex Riesen
I shall try to find the -rc2 which oopses (I suspect it is the one just before -rc3).
And this is after I put "for(;;);" at the end of dump_stack in v2.6.31-rc3:
http://home.arcor.de/fork0/bug/neocrash.jpg (~110k)
And this one is with v2.6.31-rc5 (it has usb_disconnect in it):
http://home.arcor.de/fork0/bug/neocrash-rc5.jpg (~86k) (config: http://home.arcor.de/fork0/bug/neocrash-config-rc5.txt)
I couldn't find the -rc2 which didn't lock up. They all do now. I don't know anymore how I made that -rc2 where I got the previous oops from.
Any idea how to disable stack dumps so that there is more place for other output on a laptop screen? (no chance for serial console) Or maybe a high-speed video camera can stand a chance?
Quoted 2 lines · Alex Riesen
Any idea how to disable stack dumps so that there is more place for other output on a laptop screen? (no chance for serial console)
Put a "return" statement at the start of dump_stack().
Alan Stern
Quoted 7 lines · Alan Stern
Quoted 2 lines · Alex Riesen
Any idea how to disable stack dumps so that there is more place for other output on a laptop screen? (no chance for serial console)
Put a "return" statement at the start of
dump_stack().
That wasn't enough, but a "return" in die helped.
So, I connect the device, run its bootloader, which causes a CDC ACM interface to appear. The /dev/ttyACM0 node appears, and I open it (with minicom). Then I just switch the device off and kernel oopses with:
uhci_hcd 0000:00:1d.0: release dev 4 ep81-INT, period 128, phase 0, 23 us usb 5-1: uhci_result_common: failed with status 440000
usb 5-1: uhci_result_common: failed with status 440000 uhci_hcd 0000:00:1d.0: shutdown urb ffff88007de6bf00 ep3in-bulk
BUG: unable to handle kernel NULL pointer dereference at 0000000000000003 IP: [<ffffffffa005ce38>] acm_rx_tasklet+0x24e/0x3a3 [cdc_acm] ... something will be missing here ---[ end trace ...]--- Kernel panic - not syncing: Fatal exception in interrupt
Am Sonntag, 2. August 2009 23:09:37 schrieb Alex Riesen:
Quoted 4 lines
So, I connect the device, run its bootloader, which causes a CDC ACM interface to appear. The
/dev/ttyACM0node appears, and I open it (with minicom). Then I just switch the device off and kernel oopses with:
It oopses _before_ you terminate minicom?
Regards
Oliver
Quoted 8 lines · Oliver Neukum
Am Sonntag, 2. August 2009 23:09:37 schrieb Alex Riesen:
Quoted 4 lines
So, I connect the device, run its bootloader, which causes a CDC ACM interface to appear. The
/dev/ttyACM0node appears, and I open it (with minicom). Then I just switch the device off and kernel oopses with:It oopses _before_ you terminate minicom?
Yes. minicom was running at the time, and had the ttyACM0 open.
Quoted 2 lines · Alex Riesen
BUG: unable to handle kernel NULL pointer dereference at 0000000000000003 IP: [<ffffffffa005ce38>] acm_rx_tasklet+0x24e/0x3a3 [cdc_acm]
I tried to look a little around acm_rx_tasklet (cdc-acm.c, -rc5). I think it is this code:
0000000000001bea <acm_rx_tasklet>:
...
1e27: 4d 8b 44 24 18 mov 0x18(%r12),%r8
1e2c: 89 f8 mov %edi,%eax
1e2e: c1 f8 0f sar $0xf,%eax
1e31: 83 e0 0f and $0xf,%eax
1e34: 48 8b 14 c2 mov (%rdx,%rax,8),%rdx
1e38: 0f b6 42 03 movzbl 0x3(%rdx),%eax ; I think it crashed herewhich is this:
movq 24(%r12), %r8 # <variable>.base, temp.1149 movl %edi, %eax # D.22806, tmp117 sarl $15, %eax #, tmp117 andl $15, %eax #, tmp119 movq (%rdx,%rax,8), %rdx #, ep movzbl 3(%rdx), %eax # <variable>.desc.bmAttributes,
<variable>.desc.bmAttributes
and correspondingly:
466 ep = (usb_pipein(acm->rx_endpoint) ? acm->dev->ep_in : acm->dev->ep_out)
467 [usb_pipeendpoint(acm->rx_endpoint)];
468 if (usb_endpoint_xfer_int(&ep->desc))
So it looks like ep is NULL (usb_endpoint_xfer_int accesses desc.bmAttributes, which is 3 bytes off the beginning of struct usb_host_endpoint (desc being the first field in usb_host_endpoint).
Any help with further debugging is very much appreciated (because I don't know where to continue yet).
Am Montag, 3. August 2009 11:17:05 schrieb Alex Riesen:> 466 ep = (usb_pipein(acm->rx_endpoint) ? acm->dev->ep_in :> acm->dev->ep_out)> 467 [usb_pipeendpoint(acm->rx_endpoint)];> 468 if (usb_endpoint_xfer_int(&ep->desc))>> So it looks like ep is NULL (usb_endpoint_xfer_int accesses> desc.bmAttributes, which is 3 bytes off the beginning of struct> usb_host_endpoint (desc being the first field in usb_host_endpoint). This tells you that the device has been disconnected. The questionis not why this function fails, but why it is called even though thedevice has been disconnected.
Regards Oliver
���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�
Quoted 6 lines · Oliver Neukum
Am Montag, 3. August 2009 11:17:05 schrieb Alex Riesen:
Quoted 5 lines
466 ep = (
usb_pipein(acm->rx_endpoint)?acm->dev->ep_in:acm->dev->ep_out)
467 [usb_pipeendpoint(acm->rx_endpoint)]; 468 if (usb_endpoint_xfer_int(&ep->desc))
Quoted 7 lines
Quoted 3 lines
So it looks like ep is NULL (
usb_endpoint_xfer_intaccesses desc.bmAttributes, which is 3 bytes off the beginning of structusb_host_endpoint(desc being the first field inusb_host_endpoint).This tells you that the device has been disconnected. The question is not why this function fails, but why it is called even though the device has been disconnected.
Cool. How can I help?