Skip to contentSkip to conversation

Oops in CDC ACM after an Openmoko phone (neo1973) changes its usb interface from ACM to Ethernet

DISCUSSION Alex Riesen 3 participants 11 messages Aug 2, 2009 – Aug 3, 2009

lore
Share & tools
lore
#0 lore raw

Starting since about 2.6.31-rc2, kernel oopses when I boot a kernel from Openmoko's bootloader. The bootloader is connected through CDC ACM, and when the kernel boots, the usb device changes to usb ethernet. Shortly after that kernel oopses:

[   29.346931] usb 5-1: USB disconnect, address 2
[   29.347343] usb0: unregister 'cdc_ether' usb-0000:00:1d.0-1, CDC
Ethernet Device
[   30.573530] usb 5-1: new full speed USB device using uhci_hcd and address 4
[   30.750689] usb 5-1: New USB device found, idVendor=1457, idProduct=5119
[   30.750795] usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   30.750923] usb 5-1: Product: Neo1973 Bootloader U-Boot 1.3.2-rc2
[   30.751005] usb 5-1: Manufacturer: OpenMoko, Inc
[   30.751071] usb 5-1: SerialNumber: 0000000
[   30.752032] usb 5-1: configuration #1 chosen from 2 choices
[   30.793924] cdc_acm 5-1:1.0: This device cannot do calls on its
own. It is not a modem.
[   30.794103] cdc_acm 5-1:1.0: ttyACM0: USB ACM device
[   30.799509] usbcore: registered new interface driver cdc_acm
[   30.799610] cdc_acm: v0.26:USB Abstract Control Model driver for
USB modems and ISDN adapters
[   47.346852] usb 5-1: USB disconnect, address 4
[   47.352989] tty_port_close_start: count = -1
[   48.358107] tty_port_close_start: count = -1
[   49.362088] tty_port_close_start: count = -1
[   50.365975] tty_port_close_start: count = -1
[   51.369831] tty_port_close_start: count = -1
[   52.373640] tty_port_close_start: count = -1
[   53.377348] tty_port_close_start: count = -1
[   54.381038] tty_port_close_start: count = -1
[   55.384717] tty_port_close_start: count = -1
[   56.388336] tty_port_close_start: count = -1
[   57.391932] tty_port_close_start: count = -1
[   58.395485] tty_port_close_start: count = -1
[   59.399000] tty_port_close_start: count = -1
[   60.400401] tty_port_close_start: count = -1
[   61.401795] tty_port_close_start: count = -1
[   62.405239] tty_port_close_start: count = -1
[   63.408676] tty_port_close_start: count = -1
[   64.412031] tty_port_close_start: count = -1
[   65.415241] tty_port_close_start: count = -1
[   66.418339] tty_port_close_start: count = -1
[   67.421433] tty_port_close_start: count = -1
[   68.424520] tty_port_close_start: count = -1
[   69.427630] tty_port_close_start: count = -1
[   70.430787] tty_port_close_start: count = -1
[   71.433903] tty_port_close_start: count = -1
[   72.437040] tty_port_close_start: count = -1
[   73.440171] tty_port_close_start: count = -1
[   74.441520] tty_port_close_start: count = -1
[   75.073519] usb 5-1: new full speed USB device using uhci_hcd and address 5
[   75.271699] usb 5-1: New USB device found, idVendor=1457, idProduct=5122
[   75.271709] usb 5-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   75.271716] usb 5-1: Product: RNDIS/Ethernet Gadget
[   75.271722] usb 5-1: Manufacturer: Linux 2.6.24/s3c2410_udc
[   75.272788] usb 5-1: configuration #1 chosen from 2 choices
[   75.288865] usb0: register 'cdc_ether' at usb-0000:00:1d.0-1, CDC
Ethernet Device, 1a:e0:cd:b1:57:c0
[   75.442858] tty_port_close_start: count = -1
[   75.442881] general protection fault: 0000 [#1] SMP
[   75.444775] last sysfs file:
/sys/devices/pci0000:00/0000:00:1d.0/usb5/5-1/uevent
[   75.446041] CPU 0
[   75.446041] Modules linked in: cdc_acm uvcvideo cdc_ether usbnet
ohci1394 wmi videodev v4l1_compat v4l2_compat_ioctl32 ieee1394 mii
[   75.446041] Pid: 2127, comm: minicom Not tainted 2.6.31-rc2-t #64
XPS M1330
[   75.446041] RIP: 0010:[<ffffffff812a76e1>]  [<ffffffff812a76e1>]
usb_autopm_do_interface+0x35/0xeb
[   75.453444] RSP: 0018:ffff88007d05faf8  EFLAGS: 00010292
[   75.453444] RAX: 656369766564646a RBX: 00000000ffffffed RCX: 0000000000000000
[   75.453444] RDX: ffffffffa00755b8 RSI: 0000000000000001 RDI: 656369766564646a
[   75.453444] RBP: ffff88007d05fb28 R08: 0000000000000001 R09: 0000000000000000
[   75.453444] R10: ffffffff810594a7 R11: 0000000000000082 R12: ffff88007c527800
[   75.453444] R13: 6563697665645ed2 R14: 6563697665645f62 R15: 0000000000000001
[   75.453444] FS:  00007fe65d7916f0(0000) GS:ffff880001e56000(0000)
knlGS:0000000000000000
[   75.453444] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   75.453444] CR2: 00007f83ba6c3ce0 CR3: 000000007d0e1000 CR4: 00000000000006f0
[   75.453444] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   75.453444] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   75.453444] Process minicom (pid: 2127, threadinfo
ffff88007d05e000, task ffff88007dd33da0)
[   75.453444] Stack:
[   75.453444]  656369766564646a 0000000000000000 ffff88007dead000
ffff88007dead000
[   75.453444] <0> 0000000000000000 0000000000000010 ffff88007d05fb38
ffffffff812a77b2
[   75.453444] <0> ffff88007d05fbc8 ffffffffa0072e19 0000000000000282
ffff88007d2f7000
[   75.453444] Call Trace:
[   75.453444]  [<ffffffff812a77b2>] usb_autopm_get_interface+0xe/0x10
[   75.453444]  [<ffffffffa0072e19>] acm_port_down+0x3f/0x1bb [cdc_acm]
[   75.453444]  [<ffffffff811f0af1>] ? tty_port_close_start+0xc1/0x153
[   75.453444]  [<ffffffffa0073258>] acm_tty_close+0x3d/0x83 [cdc_acm]
[   75.453444]  [<ffffffff811eaca7>] tty_release_dev+0x1bf/0x4e8
[   75.453444]  [<ffffffffa00732c3>] ? acm_tty_open+0x25/0x2ae [cdc_acm]
[   75.453444]  [<ffffffff8141e0f2>] ? mutex_unlock+0x9/0xb
[   75.453444]  [<ffffffffa00734fc>] ? acm_tty_open+0x25e/0x2ae [cdc_acm]
[   75.453444]  [<ffffffff811e8346>] ? check_tty_count+0x21/0xac
[   75.453444]  [<ffffffff811eb55d>] tty_open+0x33c/0x427
[   75.453444]  [<ffffffff810c3a81>] chrdev_open+0x150/0x16f
[   75.453444]  [<ffffffff8141fba7>] ? _spin_unlock+0x26/0x2a
[   75.453444]  [<ffffffff810c1a44>] ? file_move+0x1e/0x50
[   75.453444]  [<ffffffff810c3931>] ? chrdev_open+0x0/0x16f
[   75.453444]  [<ffffffff810bf594>] __dentry_open+0x150/0x26f
[   75.453444]  [<ffffffff810bf780>] nameidata_to_filp+0x41/0x52
[   75.453444]  [<ffffffff810cc55e>] do_filp_open+0x4ee/0x9ce
[   75.453444]  [<ffffffff810d5700>] ? alloc_fd+0x11d/0x12e
[   75.453444]  [<ffffffff810bf359>] do_sys_open+0x5d/0x10b
[   75.453444]  [<ffffffff810bf430>] sys_open+0x1b/0x1d
[   75.453444]  [<ffffffff8100baab>] system_call_fastpath+0x16/0x1b
[   75.453444] Code: 56 41 55 41 54 49 89 fc 53 bb ed ff ff ff 48 83
ec 08 4c 8b 77 30 49 8d 86 08 05 00 00 4d 8d ae 70 ff ff ff 48 89 45
d0 48 89 c7 <41> 0f b6 b5 6b 03 00 00 e8 00 6e 17 00 41 83 7c 24 24 00
0f 84
[   75.453444] RIP  [<ffffffff812a76e1>] usb_autopm_do_interface+0x35/0xeb
[   75.453444]  RSP <ffff88007d05faf8>
[   75.455529] ---[ end trace 03be7af14b545501 ]---

Full dmesg attached.
The later kernels have multiple oopses in a row and lock hard, so I could not get
the oopses for something more recent than "something past 2.6.31-rc2". I'll try to bisect, by it'll take some time.

#1 lore raw

Am Sonntag, 2. August 2009 18:40:56 schrieb Alex Riesen:

Quoted 7 lines

[   75.453444] Call Trace:
[   75.453444]  [<ffffffff812a77b2>] usb_autopm_get_interface+0xe/0x10 [   75.453444]  [<ffffffffa0072e19>] acm_port_down+0x3f/0x1bb [cdc_acm] [   75.453444]  [<ffffffff811f0af1>] ? tty_port_close_start+0xc1/0x153 [   75.453444]  [<ffffffffa0073258>] acm_tty_close+0x3d/0x83 [cdc_acm] [   75.453444]  [<ffffffff811eaca7>] tty_release_dev+0x1bf/0x4e8 [   75.453444]  [<ffffffffa00732c3>] ? acm_tty_open+0x25/0x2ae [cdc_acm]

This can happen only if acm_port_down() fails to detect that the device has been disconnected, as it calls usb_autopm_get_interface()
only if acm->dev != NULL.
This however should not happen if acm_disconnect runs.

Can you put a printk into acm_disconnect and recompile with CONFIG_USB_DEBUG?

Regards
Oliver

#2 lore raw
Quoted 9 lines · Oliver Neukum

Am Sonntag, 2. August 2009 18:40:56 schrieb Alex Riesen:

Quoted 7 lines

[   75.453444] Call Trace:
[   75.453444]  [<ffffffff812a77b2>] usb_autopm_get_interface+0xe/0x10 [   75.453444]  [<ffffffffa0072e19>] acm_port_down+0x3f/0x1bb [cdc_acm] [   75.453444]  [<ffffffff811f0af1>] ? tty_port_close_start+0xc1/0x153 [   75.453444]  [<ffffffffa0073258>] acm_tty_close+0x3d/0x83 [cdc_acm] [   75.453444]  [<ffffffff811eaca7>] tty_release_dev+0x1bf/0x4e8 [   75.453444]  [<ffffffffa00732c3>] ? acm_tty_open+0x25/0x2ae [cdc_acm]

Quoted 6 lines

This can happen only if acm_port_down() fails to detect that the device has been disconnected, as it calls usb_autopm_get_interface()
only if acm->dev != NULL.
This however should not happen if acm_disconnect runs.

Can you put a printk into acm_disconnect and recompile with CONFIG_USB_DEBUG?

I put the printks at some random lines in acm_disconnect (as of v2.6.31-rc2) and turned CONFIG_USB_DEBUG on. It is definitely not the same kernel I tried before (and sadly, it does not go oops), but it still has
tty_port_close_start: count = -1
Dmesg attached

I'm retrying with recent kernels (hopefully I'll get enough data left on the screen),
I shall try to find the -rc2 which oopses (I suspect it is the one just before -rc3).

#3 lore raw
Quoted 3 lines · Alex Riesen

I shall try to find the -rc2 which oopses (I suspect it is the one just before -rc3).

And this is after I put "for(;;);" at the end of dump_stack in v2.6.31-rc3:
http://home.arcor.de/fork0/bug/neocrash.jpg (~110k)

And this one is with v2.6.31-rc5 (it has usb_disconnect in it):
http://home.arcor.de/fork0/bug/neocrash-rc5.jpg (~86k) (config: http://home.arcor.de/fork0/bug/neocrash-config-rc5.txt)

I couldn't find the -rc2 which didn't lock up. They all do now. I don't know anymore how I made that -rc2 where I got the previous oops from.

Any idea how to disable stack dumps so that there is more place for other output on a laptop screen? (no chance for serial console) Or maybe a high-speed video camera can stand a chance?

#4 lore raw
Quoted 2 lines · Alex Riesen

Any idea how to disable stack dumps so that there is more place for other output on a laptop screen? (no chance for serial console)

Put a "return" statement at the start of dump_stack().

Alan Stern

#5 lore raw
Quoted 7 lines · Alan Stern
Quoted 2 lines · Alex Riesen

Any idea how to disable stack dumps so that there is more place for other output on a laptop screen? (no chance for serial console)

Put a "return" statement at the start of dump_stack().

That wasn't enough, but a "return" in die helped.

So, I connect the device, run its bootloader, which causes a CDC ACM interface to appear. The /dev/ttyACM0 node appears, and I open it (with minicom). Then I just switch the device off and kernel oopses with:

uhci_hcd 0000:00:1d.0: release dev 4 ep81-INT, period 128, phase 0, 23 us usb 5-1: uhci_result_common: failed with status 440000
usb 5-1: uhci_result_common: failed with status 440000 uhci_hcd 0000:00:1d.0: shutdown urb ffff88007de6bf00 ep3in-bulk

BUG: unable to handle kernel NULL pointer dereference at 0000000000000003
IP: [<ffffffffa005ce38>] acm_rx_tasklet+0x24e/0x3a3 [cdc_acm]
... something will be missing here
---[ end trace ...]---
Kernel panic - not syncing: Fatal exception in interrupt
#6 lore raw

Am Sonntag, 2. August 2009 23:09:37 schrieb Alex Riesen:

Quoted 4 lines

So, I connect the device, run its bootloader, which causes a CDC ACM interface to appear. The /dev/ttyACM0 node appears, and I open it (with minicom). Then I just switch the device off and kernel oopses with:

It oopses _before_ you terminate minicom?

Regards
Oliver

#7 lore raw
Quoted 8 lines · Oliver Neukum

Am Sonntag, 2. August 2009 23:09:37 schrieb Alex Riesen:

Quoted 4 lines

So, I connect the device, run its bootloader, which causes a CDC ACM interface to appear. The /dev/ttyACM0 node appears, and I open it (with minicom). Then I just switch the device off and kernel oopses with:

It oopses _before_ you terminate minicom?

Yes. minicom was running at the time, and had the ttyACM0 open.

#8 lore raw
In reply to Alex Riesen
Quoted 2 lines · Alex Riesen
BUG: unable to handle kernel NULL pointer dereference at 0000000000000003
IP: [<ffffffffa005ce38>] acm_rx_tasklet+0x24e/0x3a3 [cdc_acm]

I tried to look a little around acm_rx_tasklet (cdc-acm.c, -rc5). I think it is this code:

0000000000001bea <acm_rx_tasklet>:
...

    1e27:	4d 8b 44 24 18       	mov    0x18(%r12),%r8
    1e2c:	89 f8                	mov    %edi,%eax
    1e2e:	c1 f8 0f             	sar    $0xf,%eax
    1e31:	83 e0 0f             	and    $0xf,%eax
    1e34:	48 8b 14 c2          	mov    (%rdx,%rax,8),%rdx
    1e38:	0f b6 42 03          	movzbl 0x3(%rdx),%eax ; I think it crashed here

which is this:

	movq	24(%r12), %r8	# <variable>.base, temp.1149
	movl	%edi, %eax	# D.22806, tmp117
	sarl	$15, %eax	#, tmp117
	andl	$15, %eax	#, tmp119
	movq	(%rdx,%rax,8), %rdx	#, ep
	movzbl	3(%rdx), %eax	# <variable>.desc.bmAttributes,

<variable>.desc.bmAttributes

and correspondingly:

466 ep = (usb_pipein(acm->rx_endpoint) ? acm->dev->ep_in : acm->dev->ep_out)
467 [usb_pipeendpoint(acm->rx_endpoint)];
468 if (usb_endpoint_xfer_int(&ep->desc))

So it looks like ep is NULL (usb_endpoint_xfer_int accesses desc.bmAttributes, which is 3 bytes off the beginning of struct usb_host_endpoint (desc being the first field in usb_host_endpoint).

Any help with further debugging is very much appreciated (because I don't know where to continue yet).

#9 lore raw

Am Montag, 3. August 2009 11:17:05 schrieb Alex Riesen:> 466             ep = (usb_pipein(acm->rx_endpoint) ? acm->dev->ep_in :> acm->dev->ep_out)> 467                             [usb_pipeendpoint(acm->rx_endpoint)];> 468             if (usb_endpoint_xfer_int(&ep->desc))>> So it looks like ep is NULL (usb_endpoint_xfer_int accesses> desc.bmAttributes, which is 3 bytes off the beginning of struct> usb_host_endpoint (desc being the first field in usb_host_endpoint). This tells you that the device has been disconnected. The questionis not why this function fails, but why it is called even though thedevice has been disconnected.
Regards Oliver
���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#10 lore raw
Quoted 6 lines · Oliver Neukum

Am Montag, 3. August 2009 11:17:05 schrieb Alex Riesen:

Quoted 5 lines

466             ep = (usb_pipein(acm->rx_endpoint) ? acm->dev->ep_in :
acm->dev->ep_out)
467                             [usb_pipeendpoint(acm->rx_endpoint)]; 468             if (usb_endpoint_xfer_int(&ep->desc))

Quoted 7 lines
Quoted 3 lines

So it looks like ep is NULL (usb_endpoint_xfer_int accesses desc.bmAttributes, which is 3 bytes off the beginning of struct usb_host_endpoint (desc being the first field in usb_host_endpoint).

This tells you that the device has been disconnected. The question is not why this function fails, but why it is called even though the device has been disconnected.

Cool. How can I help?