USB SMP race in 2.4.11
Hello!
I have caught kernel oops that is related to SMP race on usb modules deregistering.
2.4.10 was fine with the same setup.
USB core is compiled-in, hub driver is uhci (as module). Here is the decoded oops:
Unable to handle kernel paging request at virtual address d890c138 c018f709 *pde = 01662067 Oops: 0000 CPU: 1 EIP: 0010:[<c018f709>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286
eax: d890c12c ebx: c1664800 ecx: c1661ef4 edx: d71cff60
esi: 00000064 edi: c1661ef0 ebp: c1660000 esp: c1661ee0
ds: 0018 es: 0018 ss: 0018
Process khubd (pid: 10, stackpage=c1661000)
Stack: d71cff60 c018f7fb d71cff60 c1661f00 00000001 c1661f08 c1661f08 00000000
00000000 c1660000 c1661ef4 c1661ef4 c1664800 80000180 c160e720 c1661fbc
c018f96f d71cff60 00000064 c1661f30 c1660000 c1661fbc c160e720 00000064
Call Trace: [<c018f7fb>] [<c018f96f>] [<c018fa04>] [<c0191dbc>] [<c0192b7b>]
[<c0192d95>] [<c0105000>] [<c0105656>] [<c0192d50>]
Code: ff 50 0c 5a c3 89 f6 b8 ed ff ff ff c3 8d 76 00 8d bc 27 00 Quoted 1 line
Quoted 1 line
EIP;
c018f709<usb_submit_urb+19/30> <=====
Trace; c018f7fb <usb_start_wait_urb+8b/1a0> Trace; c018f96f <usb_internal_control_msg+5f/70> Trace; c018fa04 <usb_control_msg+84/a0>
Trace; c0191dbc <usb_get_port_status+3c/40> Trace; c0192b7b <usb_hub_events+eb/2c0>
Trace; c0192d95 <usb_hub_thread+45/a0>
Trace; c0105000 <_stext+0/0>
Trace; c0105656 <kernel_thread+26/30>
Trace; c0192d50 <usb_hub_thread+0/a0>
Code; c018f709 <usb_submit_urb+19/30>
00000000 <_EIP>:
Code; c018f709 <usb_submit_urb+19/30> <=====
0: ff 50 0c call *0xc(%eax) <=====
Code; c018f70c <usb_submit_urb+1c/30>
3: 5a pop %edx
Code; c018f70d <usb_submit_urb+1d/30>
4: c3 ret
Code; c018f70e <usb_submit_urb+1e/30>
5: 89 f6 mov %esi,%esi
Code; c018f710 <usb_submit_urb+20/30>
7: b8 ed ff ff ff mov $0xffffffed,%eax
Code; c018f715 <usb_submit_urb+25/30>
c: c3 ret
Code; c018f716 <usb_submit_urb+26/30>
d: 8d 76 00 lea 0x0(%esi),%esi
Code; c018f719 <usb_submit_urb+29/30>
10: 8d bc 27 00 00 00 00 lea 0x0(%edi,1),%edi
It seems to die while dereferencing urb->dev->bus->op->submit_urb(urb), and urb->dev->bus->op pointer is bogus.
Looking at the kernel messages, I have found that it dies at usb bus deregistering (when RedHat initscripts unload usb host controller driver for some reason).
Here are the messages:
uhci.c: USB Universal Host Controller Interface driver v1.1uhci.c: USB UHCI at I/O 0xc400, IRQ 19usb.c: new USB bus registered, assigned bus number 1
usb: raced timeout, pipe 0x80000000 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0hub.c: USB hub found
usb: raced timeout, pipe 0x80000180 status 0 time left 0hub.c: 2 ports detected
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0uhci.c: USB UHCI at I/O 0xc800, IRQ 19usb.c: new USB bus registered, assigned bus number 2
usb: raced timeout, pipe 0x80000100 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000000 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0hub.c: USB new device connect on bus1/1, assigned device number 2 usb.c: USB device 2 (vend/prod 0x49f/0x505a) is not claimed by any active driver. usb.c: registered new driver usbnet
usb0: register usbnet 001/002, Linux Device
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0hub.c: USB hub found
usb: raced timeout, pipe 0x80000180 status 0 time left 0hub.c: 2 ports detected
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0
usb: raced timeout, pipe 0x80000180 status 0 time left 0
usb: raced timeout, pipe 0x80000100 status 0 time left 0usb.c: USB disconnect on device 1usb.c: USB disconnect on device 2
usb0: unregister usbnet 001/002, Linux Deviceusb.c: USB bus 1 deregisteredusb.c: USB disconnect on device 1usb.c: USB bus 2 deregistered
usb: raced timeout, pipe 0x80000100 status 0 time left 0
Unable to handle kernel paging request at virtual address d890c138
...
Bye,
Oleg
Quoted 117 lines · Oleg Drokin
I have caught kernel oops that is related to SMP race on usb modules deregistering.
2.4.10 was fine with the same setup.
USB core is compiled-in, hub driver is uhci (as module). Here is the decoded oops:Unable to handle kernel paging request at virtual address d890c138 c018f709 *pde = 01662067 Oops: 0000 CPU: 1 EIP: 0010:[<c018f709>] Not taintedUsing defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010286
eax:d890c12cebx:c1664800ecx:c1661ef4edx:d71cff60
esi: 00000064 edi:c1661ef0ebp:c1660000esp:c1661ee0
ds: 0018 es: 0018 ss: 0018
Process khubd (pid: 10, stackpage=c1661000)Stack: d71cff60 c018f7fb d71cff60 c1661f00 00000001 c1661f08 c1661f08 00000000 00000000 c1660000 c1661ef4 c1661ef4 c1664800 80000180 c160e720 c1661fbc c018f96f d71cff60 00000064 c1661f30 c1660000 c1661fbc c160e720 00000064 Call Trace: [<c018f7fb>] [<c018f96f>] [<c018fa04>] [<c0191dbc>] [<c0192b7b>] [<c0192d95>] [<c0105000>] [<c0105656>] [<c0192d50>] Code: ff 50 0c 5a c3 89 f6 b8 ed ff ff ff c3 8d 76 00 8d bc 27 00Quoted 1 line
Quoted 1 line
EIP;
c018f709<usb_submit_urb+19/30> <=====Trace;
c018f7fb<usb_start_wait_urb+8b/1a0> Trace;c018f96f<usb_internal_control_msg+5f/70> Trace;c018fa04<usb_control_msg+84/a0>
Trace;c0191dbc<usb_get_port_status+3c/40> Trace;c0192b7b<usb_hub_events+eb/2c0>
Trace;c0192d95<usb_hub_thread+45/a0>
Trace;c0105000<_stext+0/0>
Trace;c0105656<kernel_thread+26/30>
Trace;c0192d50<usb_hub_thread+0/a0>
Code;c018f709<usb_submit_urb+19/30>
00000000 <_EIP>:
Code;c018f709<usb_submit_urb+19/30> <=====
0: ff 50 0c call *0xc(%eax) <=====
Code;c018f70c<usb_submit_urb+1c/30>
3: 5a pop %edx
Code;c018f70d<usb_submit_urb+1d/30>
4: c3 ret
Code;c018f70e<usb_submit_urb+1e/30>
5: 89 f6 mov %esi,%esi
Code;c018f710<usb_submit_urb+20/30>
7: b8 ed ff ff ff mov $0xffffffed,%eax
Code;c018f715<usb_submit_urb+25/30>
c: c3 ret
Code;c018f716<usb_submit_urb+26/30>
d: 8d 76 00 lea 0x0(%esi),%esi
Code;c018f719<usb_submit_urb+29/30>
10: 8d bc 27 00 00 00 00 lea 0x0(%edi,1),%ediIt seems to die while dereferencing
urb->dev->bus->op->submit_urb(urb), andurb->dev->bus->oppointer is bogus.
Looking at the kernel messages, I have found that it dies at usb bus deregistering (when RedHat initscripts unload usb host controller driver for some reason).
Here are the messages:
uhci.c: USB Universal Host Controller Interface driver v1.1uhci.c: USB UHCI at I/O0xc400, IRQ 19usb.c: new USB bus registered, assigned bus number 1
usb: raced timeout, pipe0x80000000status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0hub.c: USB hub found
usb: raced timeout, pipe0x80000180status 0 time left 0hub.c: 2 ports detected
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0uhci.c: USB UHCI at I/O0xc800, IRQ 19usb.c: new USB bus registered, assigned bus number 2
usb: raced timeout, pipe0x80000100status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000000status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0hub.c: USB new device connect on bus1/1, assigned device number 2usb.c: USB device 2 (vend/prod 0x49f/0x505a) is not claimed by any active driver.usb.c: registered new driver usbnet
usb0: register usbnet 001/002, Linux Device
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0hub.c: USB hub found
usb: raced timeout, pipe0x80000180status 0 time left 0hub.c: 2 ports detected
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0
usb: raced timeout, pipe0x80000180status 0 time left 0
usb: raced timeout, pipe0x80000100status 0 time left 0usb.c: USB disconnect on device 1usb.c: USB disconnect on device 2
usb0: unregister usbnet 001/002, Linux Deviceusb.c: USB bus 1 deregisteredusb.c: USB disconnect on device 1usb.c: USB bus 2 deregistered
usb: raced timeout, pipe0x80000100status 0 time left 0
Quoted 2 lines
Unable to handle kernel paging request at virtual address d890c138...
Did you remove the uhci module?
This patch will fix the raced timeout messages, but you may have found a reference counting bug as well.
JE
drivers/usb/uhci.c +1 -3
@@ -1594,9 +1594,7 @@ }uhci_unlink_generic (uhci, urb); -uhci_destroy_urb_priv (urb); - -usb_dec_dev_use (urb->dev); +uhci_call_completion (urb);return ret; }
Quoted 7 lines · Oleg Drokin
Hello!
I have caught kernel oops that is related to SMP race on usb modules deregistering.
2.4.10 was fine with the same setup.
USB core is compiled-in, hub driver is uhci (as module). Here is the decoded oops:
It's not SMP related coz i've got the same stuff on UP
- Davide
Quoted 7 lines · Oleg Drokin
Hello!
I have caught kernel oops that is related to SMP race on usb modules deregistering.
2.4.10 was fine with the same setup.
USB core is compiled-in, hub driver is uhci (as module). Here is the decoded oops:
Can you try the attached patch and let us know if the oops still happens?
thanks,
greg k-h
drivers/usb/uhci.c +1 -3
@@ -1594,9 +1594,7 @@ }uhci_unlink_generic (uhci, urb); -uhci_destroy_urb_priv (urb); - -usb_dec_dev_use (urb->dev); +uhci_call_completion (urb);return ret; }