kmemleak report on isp1763 and sierra MC8705
Hi Guys,
I am debugging a reported kmemleak involving a sierra wireless MC8705 connected through isp1763 on powerpc linux-3.0.22
We are still isolating the exact trigger, but this is a pretty good one so far
send "at!reset" to the modem control tty, wait until it finishes rebooting then try to bring up a PPP link that will fail (non existent ISP).
After some time, we got the report (included at the end) from kmemleak.
There seems to be two variants of trace that is prevalent:
something like this:
unreferenced object 0xd58e58c8 (size 8):
comm "khubd", pid 1034, jiffies 74467293 (age 2380.122s) hex dump (first 8 bytes):
4d 43 38 37 30 35 00 00 MC8705..
backtrace:
[<e30efd74>] usb_cache_string+0x74/0xac [usbcore]
[<e30e77bc>] usb_enumerate_device+0x44/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68and something like this:
unreferenced object 0xd5893e00 (size 512):
comm "khubd", pid 1034, jiffies 74467270 (age 2378.786s) hex dump (first 32 bytes):
09 02 a8 00 06 01 01 e0 00 00 00 00 d5 87 d6 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<e30f1740>] usb_get_configuration+0x5c/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68Some questions:
1. Have you guys seen anything like this before?
2. The report does not point to sierra or isp1763, so our current understanding is that the memory is allocated outside these drivers and it is supposed to mark it done for someone to free it. We think this way because if we rigged a driver to leak a memory it allocates, kmemleak will trace right into it. Is this understanding correct?
3. Any ideas on how to deepen the probe to get more understanding of what happens?
4. Michael, is this similar to the problem you reported here?
http://marc.info/?l=linux-usb&m=133432571801643&w=4
From reading your report (serial device hanging), It doesn't look like it...
5. Our current hypothesis is this:
we open the /dev/ttyUSB to send "at!reset", then a race begins between closing the file handle and freeing the driver resources and the modem hardware actually resetting, which then caused the leak. Can this be it? and if so, any ideas on how to solve it?
To test this we are power cycling the modem using a gpio (without opening /dev/ttyUSB) to see if this is the culprit.
6. There is a worrisome line in our (old version) of isp1763 inherited from isp1760:
isp1760_endpoint_disable()
...
qh_destroy(qh);
ep->hcpriv = NULL;
/* remove requests and leak them.
* ATL are pretty fast done, INT could take a while...
* The latter shoule be removed
*/
What is leaking here? qh_destroy release the memory already.Thanks for everyone's time!
-- Richard Retanubun
--------------------------------------------------------------------------------
unreferenced object 0xd5922c00 (size 1024):
comm "khubd", pid 1034, jiffies 74467113 (age 2378.943s) hex dump (first 32 bytes):
ff ff ff ff 31 2e 32 00 00 00 00 00 00 00 00 00 ....1.2.........
00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 03 ................
backtrace:
[<e30e4718>] usb_alloc_dev+0x48/0x290 [usbcore]
[<e30e91ec>] hub_thread+0x654/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd58e52b0 (size 8):
comm "khubd", pid 1034, jiffies 74467113 (age 2378.943s)
hex dump (first 8 bytes):
32 2d 31 2e 32 00 04 00 2-1.2...
backtrace:
[<c018a9ec>] kvasprintf+0x58/0x88
[<c0180910>] kobject_set_name_vargs+0x34/0x84
[<c01b3d20>] dev_set_name+0x50/0x60
[<e30e4860>] usb_alloc_dev+0x190/0x290 [usbcore]
[<e30e91ec>] hub_thread+0x654/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5893e00 (size 512):
comm "khubd", pid 1034, jiffies 74467270 (age 2378.786s)
hex dump (first 32 bytes):
09 02 a8 00 06 01 01 e0 00 00 00 00 d5 87 d6 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<e30f1740>] usb_get_configuration+0x5c/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd58e5930 (size 8):
comm "khubd", pid 1034, jiffies 74467270 (age 2378.786s)
hex dump (first 8 bytes):
d5 8a dc c0 00 00 00 00 ........
backtrace:
[<e30f1760>] usb_get_configuration+0x7c/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd58adcc0 (size 192):
comm "khubd", pid 1034, jiffies 74467271 (age 2378.786s)
hex dump (first 32 bytes):
09 02 a8 00 06 01 01 e0 00 09 04 00 00 02 ff ff ................
ff 00 07 05 81 02 00 02 20 07 05 01 02 00 02 20 ........ ......
backtrace:
[<e30f1804>] usb_get_configuration+0x120/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd59555c0 (size 64):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.766s)
hex dump (first 32 bytes):
00 00 00 01 00 00 00 01 09 04 00 00 02 ff ff ff ................
00 00 00 00 d5 92 7a e0 00 00 00 00 d5 8a dc d2 ......z.........
backtrace:
[<e30f1cb0>] usb_get_configuration+0x5cc/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5955400 (size 64):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.766s)
hex dump (first 32 bytes):
00 00 00 01 00 00 00 01 09 04 01 00 02 ff ff ff ................
00 00 00 00 d5 92 7a 20 00 00 00 00 d5 8a dc e9 ......z ........
backtrace:
[<e30f1cb0>] usb_get_configuration+0x5cc/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5955280 (size 64):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.779s)
hex dump (first 32 bytes):
00 00 00 01 00 00 00 01 09 04 02 00 02 ff ff ff ................
00 00 00 00 d5 92 70 00 00 00 00 00 d5 8a dd 00 ......p.........
backtrace:
[<e30f1cb0>] usb_get_configuration+0x5cc/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd59554c0 (size 64):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.779s)
hex dump (first 32 bytes):
00 00 00 01 00 00 00 01 09 04 03 00 03 ff ff ff ................
00 00 00 00 d5 8a d5 40 00 00 00 00 d5 8a dd 17 .......@........
backtrace:
[<e30f1cb0>] usb_get_configuration+0x5cc/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5955580 (size 64):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.779s)
hex dump (first 32 bytes):
00 00 00 01 00 00 00 01 09 04 04 00 03 ff ff ff ................
00 00 00 00 d5 8a d2 40 00 00 00 00 d5 8a dd 35 [email protected]
backtrace:
[<e30f1cb0>] usb_get_configuration+0x5cc/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5955300 (size 64):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.779s)
hex dump (first 32 bytes):
00 00 00 01 00 00 00 01 09 04 07 00 03 ff ff ff ................
00 00 00 00 d5 8a d3 00 00 00 00 00 d5 8a dd 53 ...............S
backtrace:
[<e30f1cb0>] usb_get_configuration+0x5cc/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5927ae0 (size 96):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.779s)
hex dump (first 32 bytes):
07 05 81 02 00 02 20 00 00 00 00 00 00 00 00 00 ...... .........
d5 92 7a f0 d5 92 7a f0 00 00 00 00 00 00 00 00 ..z...z.........
backtrace:
[<e30f2514>] usb_get_configuration+0xe30/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5927a20 (size 96):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.779s)
hex dump (first 32 bytes):
07 05 82 02 00 02 20 00 00 00 00 00 00 00 00 00 ...... .........
d5 92 7a 30 d5 92 7a 30 00 00 00 00 00 00 00 00 ..z0..z0........
backtrace:
[<e30f2514>] usb_get_configuration+0xe30/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5927000 (size 96):
comm "khubd", pid 1034, jiffies 74467291 (age 2378.780s)
hex dump (first 32 bytes):
07 05 83 02 00 02 20 00 00 00 00 00 00 00 00 00 ...... .........
d5 92 70 10 d5 92 70 10 00 00 00 00 00 00 00 00 ..p...p.........
backtrace:
[<e30f2514>] usb_get_configuration+0xe30/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd58ad540 (size 192):
comm "khubd", pid 1034, jiffies 74467291 (age 2380.124s)
hex dump (first 32 bytes):
07 05 84 03 40 00 05 00 00 00 00 00 00 00 00 00 ....@...........
d5 8a d5 50 d5 8a d5 50 00 00 00 00 00 00 00 00 ...P...P........
backtrace:
[<e30f2514>] usb_get_configuration+0xe30/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd58ad240 (size 192):
comm "khubd", pid 1034, jiffies 74467291 (age 2380.124s)
hex dump (first 32 bytes):
07 05 86 03 40 00 05 00 00 00 00 00 00 00 00 00 ....@...........
d5 8a d2 50 d5 8a d2 50 00 00 00 00 00 00 00 00 ...P...P........
backtrace:
[<e30f2514>] usb_get_configuration+0xe30/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd58ad300 (size 192):
comm "khubd", pid 1034, jiffies 74467291 (age 2380.124s)
hex dump (first 32 bytes):
07 05 88 03 40 00 05 00 00 00 00 00 00 00 00 00 ....@...........
d5 8a d3 10 d5 8a d3 10 00 00 00 00 00 00 00 00 ................
backtrace:
[<e30f2514>] usb_get_configuration+0xe30/0x13a8 [usbcore]
[<e30e7850>] usb_enumerate_device+0xd8/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd58e58c8 (size 8):
comm "khubd", pid 1034, jiffies 74467293 (age 2380.122s)
hex dump (first 8 bytes):
4d 43 38 37 30 35 00 00 MC8705..
backtrace:
[<e30efd74>] usb_cache_string+0x74/0xac [usbcore]
[<e30e77bc>] usb_enumerate_device+0x44/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd587d3c0 (size 32):
comm "khubd", pid 1034, jiffies 74467293 (age 2380.122s)
hex dump (first 32 bytes):
53 69 65 72 72 61 20 57 69 72 65 6c 65 73 73 2c Sierra Wireless,
20 49 6e 63 6f 72 70 6f 72 61 74 65 64 00 64 00 Incorporated.d.
backtrace:
[<e30efd74>] usb_cache_string+0x74/0xac [usbcore]
[<e30e77cc>] usb_enumerate_device+0x54/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd670a1a0 (size 16):
comm "khubd", pid 1034, jiffies 74467294 (age 2380.122s)
hex dump (first 16 bytes):
33 35 33 35 36 37 30 34 30 31 31 31 37 39 32 00 353567040111792.
backtrace:
[<e30efd74>] usb_cache_string+0x74/0xac [usbcore]
[<e30e77dc>] usb_enumerate_device+0x64/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd5927480 (size 96):
comm "khubd", pid 1034, jiffies 74467294 (age 2380.122s)
hex dump (first 32 bytes):
d5 92 74 80 d5 92 74 80 c0 1b 35 4c c0 1b 36 a8 ..t...t...5L..6.
00 00 00 00 00 10 01 00 00 20 02 00 00 00 00 00 ......... ......
backtrace:
[<c01b46c4>] device_private_init+0x34/0x8c
[<c01b4f28>] device_add+0x27c/0x6a8
[<e30e7b00>] usb_new_device+0x9c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68
unreferenced object 0xd587d600 (size 32):
comm "khubd", pid 1034, jiffies 74467300 (age 2380.792s)
hex dump (first 32 bytes):
53 69 65 72 72 61 20 43 6f 6e 66 69 67 75 72 61 Sierra Configura
74 69 6f 6e 00 2f 52 00 df 82 09 60 df 40 64 00 tion./R....`.@d.
backtrace:
[<e30efd74>] usb_cache_string+0x74/0xac [usbcore]
[<e30f0264>] usb_set_configuration+0x4b8/0x60c [usbcore]
[<e30f8850>] generic_probe+0x48/0xb8 [usbcore]
[<e30f0b00>] usb_probe_device+0x38/0x70 [usbcore]
[<c01b79e8>] driver_probe_device+0xc0/0x2a8
[<c01b6be4>] bus_for_each_drv+0x70/0xac
[<c01b7df4>] device_attach+0xb4/0xd8
[<c01b6340>] bus_probe_device+0x2c/0x44
[<c01b51b8>] device_add+0x50c/0x6a8
[<e30e7b00>] usb_new_device+0x9c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68Quoted 4 lines · Richard Retanubun
Hi Guys,
I am debugging a reported kmemleak involving a sierra wireless MC8705 connected through isp1763 on powerpc linux-3.0.22
Does this also happen on 3.6.3?
thanks,
greg k-h
Quoted 11 lines · Greg KH
Quoted 4 lines · Richard Retanubun
Hi Guys,
I am debugging a reported kmemleak involving a sierra wireless MC8705 connected through isp1763 on powerpc linux-3.0.22
Does this also happen on 3.6.3?
thanks,
greg k-h
Hi Greg,
Unfortunately, it is not trivial for us to update the kernel on this platform, Is there a specific experiment/patch I should look at for 3.0.22?
I will be attempting to use 3.6.3 on another platform, but this may take some time.
I am thinking there may be an action that can be done at /sysfs or /procfs to do the disconnect without actually removing the power to the device.
I tried "echo 1 > /sys/bus/usb/drivers/usb/2-1.2/remove" and then take down the power but this produced the same leak signature even before I take down the power.
Update on trigger to problem
============================
This will happen as the modem is powered down and /dev/ttyUSB from sierra is teared down either by powering it off/removing it, or sending at!reset.
It does not happen when the same thing is done using a simple usb to serial converter (pl2303)
Focusing down on one of the dumps:
unreferenced object 0xd3849740 (size 8):
comm "khubd", pid 1026, jiffies 232553037 (age 506.597s) hex dump (first 8 bytes):
4d 43 38 37 30 35 00 00 MC8705..
backtrace:
[<e30efd74>] usb_cache_string+0x74/0xac [usbcore]
[<e30e77bc>] usb_enumerate_device+0x44/0xf8 [usbcore]
[<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore]
[<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore]
[<c0043aa8>] kthread+0x7c/0x80
[<c000ed48>] kernel_thread+0x4c/0x68I have a small question. How does the memory kmalloc-ed() in usb_cache_string is supposed to be released? (during usb_serial_disconnect()?) Is the sierra driver is supposed to participate
in the tear down process (in sierra_release() maybe) and not doing something that is expected? I am still missing the link between the actions done by the hub_thread() for the caching the stings and the sierra driver code.
Thanks a lot for your time.
Quoted 9 lines · Richard Retanubun
Quoted 8 lines · Greg KH
Quoted 4 lines · Richard Retanubun
Hi Guys,
I am debugging a reported kmemleak involving a sierra wireless MC8705 connected through isp1763 on powerpc linux-3.0.22
Does this also happen on 3.6.3?
Quoted 8 lines
Quoted 3 lines
thanks,
greg k-h
Hi Greg,
Unfortunately, it is not trivial for us to update the kernel on this platform, Is there a specific experiment/patch I should look at for 3.0.22?
I have no idea, given that 3.0.22 is based on 18 month old kernel, tens of thousands of patches ago.
You are on your own here, sorry.
Best of luck,
greg k-h
Quoted 25 lines · Richard Retanubun
do the disconnect without actually removing the power to the device.
I tried "echo 1 >/sys/bus/usb/drivers/usb/2-1.2/remove" and then take down the power but this produced the same leak signature even before I take down the power.Update on trigger to problem
============================
This will happen as the modem is powered down and/dev/ttyUSBfrom sierra is teared down either by powering it off/removing it, or sending at!reset.It does not happen when the same thing is done using a simple usb to serial converter (pl2303)
Focusing down on one of the dumps:
unreferenced object
0xd3849740(size 8):comm "khubd", pid 1026, jiffies 232553037 (age 506.597s) hex dump (first 8 bytes): 4d 43 38 37 30 35 00 00 MC8705.. backtrace: [<e30efd74>] usb_cache_string+0x74/0xac [usbcore] [<e30e77bc>] usb_enumerate_device+0x44/0xf8 [usbcore] [<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore] [<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore] [<c0043aa8>] kthread+0x7c/0x80 [<c000ed48>] kernel_thread+0x4c/0x68
Quoted 2 lines
I have a small question. How does the memory kmalloc-ed() in
usb_cache_stringis supposed to be released? (duringusb_serial_disconnect()?)
It doesn't get released during usb_serial_disconnect(). It gets released during usb_release_dev() in drivers/usb/core/usb.c.
Quoted 2 lines
Is the sierra driver is supposed to participate
in the tear down process (insierra_release()maybe) and not doing something that is expected?
Probably not.
Quoted 2 lines
I am still missing the link between the actions done by the
hub_thread()for the caching the stings and the sierra driver code.
They aren't all that closely related.
usb_release_dev() won't be called until all references to the USB device have been dropped. Maybe there's an extra reference hanging around.
Alan Stern
Quoted 29 lines · Alan Stern
Quoted 14 lines · Richard Retanubun
Focusing down on one of the dumps:
unreferenced object
0xd3849740(size 8):comm "khubd", pid 1026, jiffies 232553037 (age 506.597s) hex dump (first 8 bytes): 4d 43 38 37 30 35 00 00 MC8705.. backtrace: [<e30efd74>] usb_cache_string+0x74/0xac [usbcore] [<e30e77bc>] usb_enumerate_device+0x44/0xf8 [usbcore] [<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore] [<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore] [<c0043aa8>] kthread+0x7c/0x80 [<c000ed48>] kernel_thread+0x4c/0x68Quoted 2 lines
I have a small question. How does the memory kmalloc-ed() in
usb_cache_stringis supposed to be released? (duringusb_serial_disconnect()?)It doesn't get released during
usb_serial_disconnect(). It gets released duringusb_release_dev()indrivers/usb/core/usb.c.Quoted 2 lines
Is the sierra driver is supposed to participate
in the tear down process (insierra_release()maybe) and not doing something that is expected?Probably not.
Quoted 2 lines
I am still missing the link between the actions done by the
hub_thread()for the caching the stings and the sierra driver code.
Quoted 8 lines
They aren't all that closely related.
usb_release_dev()won't be called until all references to the USB device have been dropped. Maybe there's an extra reference hanging around.Alan Stern
Thanks a lot for the hint Alan.
I added a dev_dbg print in usb_release_dev() and saw that in the builds where there is a leak, this was simply never called! the last line printed in a trace with all dev_dbg on is this "usb_disable_device nuking all URBs"
When the sierra modem is unplugged, the cleanup sequence never calls usb_release_dev() (on PL2303 it always calls usb_release_dev()
This is the current state of versions from linux-stable
3.0.y (3.0.51) - Have the issue
3.2.y (3.2.33) - Have the issue
3.4.y (3.4.18) - Have the issue
3.5.y (3.5.7) - Does not have the issue (but leaks because the portdata patches is not backported yet) 3.6.y (3.6.6) - Does not have the issue
So a diff between 3.4.y and 3.5.y ought to narrow it down further.
I am posting just in case someone recalls a particular patch I should be trying out first...
-- RR --
Quoted 64 lines · Richard Retanubun
Quoted 37 lines · Alan Stern
Quoted 15 lines · Richard Retanubun
Focusing down on one of the dumps:
unreferenced object
0xd3849740(size 8):comm "khubd", pid 1026, jiffies 232553037 (age 506.597s) hex dump (first 8 bytes): 4d 43 38 37 30 35 00 00 MC8705.. backtrace: [<e30efd74>] usb_cache_string+0x74/0xac [usbcore] [<e30e77bc>] usb_enumerate_device+0x44/0xf8 [usbcore] [<e30e7aa0>] usb_new_device+0x3c/0x13c [usbcore] [<e30e9824>] hub_thread+0xc8c/0x1544 [usbcore] [<c0043aa8>] kthread+0x7c/0x80 [<c000ed48>] kernel_thread+0x4c/0x68Quoted 3 lines
I have a small question. How does the memory kmalloc-ed() in
usb_cache_stringis supposed to be released? (duringusb_serial_disconnect()?)It doesn't get released during
usb_serial_disconnect(). It gets released duringusb_release_dev()indrivers/usb/core/usb.c.Quoted 3 lines
Is the sierra driver is supposed to participate
in the tear down process (insierra_release()maybe) and not doing something that is expected?Probably not.
Quoted 3 lines
I am still missing the link between the actions done by the
hub_thread()for the caching the stings
and the sierra driver code.Quoted 8 lines
They aren't all that closely related.
usb_release_dev()won't be called until all references to the USB device have been dropped. Maybe there's an extra reference hanging around.Alan Stern
Thanks a lot for the hint Alan.
I added a
dev_dbgprint inusb_release_dev()and saw that in the builds where there is a leak, this was simply never called!
the last line printed in a trace with alldev_dbgon is this "usb_disable_devicenuking all URBs"
When the sierra modem is unplugged, the cleanup sequence never callsusb_release_dev()(on PL2303 it always callsusb_release_dev()This is the current state of versions from linux-stable
3.0.y (3.0.51) - Have the issue
3.2.y (3.2.33) - Have the issue
3.4.y (3.4.18) - Have the issue
3.5.y (3.5.7) - Does not have the issue (but leaks because the portdata patches is not backported yet)
3.6.y (3.6.6) - Does not have the issue
Quoted 4 lines
So a diff between 3.4.y and 3.5.y ought to narrow it down further.
I am posting just in case someone recalls a particular patch I should be trying out first...
There was a reference-count fix for the probe error path that went in to v3.5. Haven't read all the details on how you trigger your leak, but at the face of it, it could be related.
Have a look at 0658a3366db7e27fa ("usb: use usb_serial_put in usb_serial_probe errors). If related, you should be seeing "Ignoring blacklisted interface #n" messages when you enable debug (e.g. #define DEBUG) in the sierra driver.
Greg, it seems to me that the fix referred to above should be backported to the earlier stable trees either way.
Thanks,
Johan
On 10/11/12 09:30 AM, Johan Hovold wrote:
Hi Johan,
Quoted 8 lines
There was a reference-count fix for the probe error path that went in to v3.5. Haven't read all the details on how you trigger your leak, but at the face of it, it could be related.
Have a look at
0658a3366db7e27fa("usb: useusb_serial_putinusb_serial_probeerrors). If related, you should be seeing "Ignoring blacklisted interface #n" messages when you enable debug (e.g. #define DEBUG) in the sierra driver.
That was it! Thanks so much for the research.
I can apply it cleanly to 3.0.22 and see usb_release_dev() being called and thus no more kmemleak.
Quoted 3 lines
Greg, it seems to me that the fix referred to above should be backported to the earlier stable trees either way.
I would vote "yes" for this also.
While my setup circumstances may be a corner case, (modem kept resetting to re-establish PPP connection) it was leaking 1192 bytes per occurrence.
Thanks for everyone's time.
-- Richard Retanubun.
Quoted 17 lines · Richard Retanubun
On 10/11/12 09:30 AM, Johan Hovold wrote:
Hi Johan,Quoted 8 lines
There was a reference-count fix for the probe error path that went in to v3.5. Haven't read all the details on how you trigger your leak, but at the face of it, it could be related.
Have a look at
0658a3366db7e27fa("usb: useusb_serial_putinusb_serial_probeerrors). If related, you should be seeing "Ignoring blacklisted interface #n" messages when you enable debug (e.g. #define DEBUG) in the sierra driver.That was it! Thanks so much for the research.
I can apply it cleanly to 3.0.22 and seeusb_release_dev()being called and thus no more kmemleak.Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Greg, it seems to me that the fix referred to above should be backported to the earlier stable trees either way.
I would vote "yes" for this also.
While my setup circumstances may be a corner case, (modem kept resetting to re-establish PPP connection) it was leaking 1192 bytes per occurrence.
The leak affects every failed probe, for example due to blacklisted interfaces which is quite common, so commit 0658a3366db7 ("usb: use usb_serial_put in usb_serial_probe errors) should be backported to the <= 3.4 stable trees.
Thanks for reporting,
Johan
Greg Kroah-Hartman
maintainerQuoted 22 lines · Johan Hovold
Quoted 21 lines · Richard Retanubun
On 10/11/12 09:30 AM, Johan Hovold wrote:
Hi Johan,Quoted 8 lines
There was a reference-count fix for the probe error path that went in to v3.5. Haven't read all the details on how you trigger your leak, but at the face of it, it could be related.
Have a look at
0658a3366db7e27fa("usb: useusb_serial_putinusb_serial_probeerrors). If related, you should be seeing "Ignoring blacklisted interface #n" messages when you enable debug (e.g. #define DEBUG) in the sierra driver.That was it! Thanks so much for the research.
I can apply it cleanly to 3.0.22 and seeusb_release_dev()being called and thus no more kmemleak.Quoted 3 lines
Greg, it seems to me that the fix referred to above should be backported to the earlier stable trees either way.
I would vote "yes" for this also.
Quoted 8 lines
Quoted 3 lines
While my setup circumstances may be a corner case, (modem kept resetting to re-establish PPP connection) it was leaking 1192 bytes per occurrence.
The leak affects every failed probe, for example due to blacklisted interfaces which is quite common, so commit
0658a3366db7("usb: useusb_serial_putinusb_serial_probeerrors) should be backported to the <= 3.4 stable trees.
Thanks, now applied.
greg k-h
Quoted 27 lines · Greg Kroah-Hartman
Quoted 26 lines · Johan Hovold
Quoted 24 lines · Richard Retanubun
On 10/11/12 09:30 AM, Johan Hovold wrote:
Hi Johan,Quoted 8 lines
There was a reference-count fix for the probe error path that went in to v3.5. Haven't read all the details on how you trigger your leak, but at the face of it, it could be related.
Have a look at
0658a3366db7e27fa("usb: useusb_serial_putinusb_serial_probeerrors). If related, you should be seeing "Ignoring blacklisted interface #n" messages when you enable debug (e.g. #define DEBUG) in the sierra driver.That was it! Thanks so much for the research.
I can apply it cleanly to 3.0.22 and seeusb_release_dev()being called and thus no more kmemleak.Quoted 3 lines
Greg, it seems to me that the fix referred to above should be backported to the earlier stable trees either way.
I would vote "yes" for this also.
While my setup circumstances may be a corner case, (modem kept resetting to re-establish PPP connection) it was leaking 1192 bytes per occurrence.
Quoted 6 lines
Quoted 4 lines
The leak affects every failed probe, for example due to blacklisted interfaces which is quite common, so commit
0658a3366db7("usb: useusb_serial_putinusb_serial_probeerrors) should be backported to the <= 3.4 stable trees.Thanks, now applied.
Also queued up for 3.2.
Ben.
Signature · 2 lines
Ben Hutchings Never attribute to conspiracy what can adequately be explained by stupidity.