Race with ip=dhcp bootparameter in ip_rcv_finish on am335x
Subsystems (4)
Hi,
I am using am335x (beaglebone white) with "ip=dhcp" inside my bootparameters. It seems there exists some race, rarely it boots fine and often it crashed with following messages:
First (Without successful dhcp ip assign and is very rarely):
[ 6.249083] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.398943] . [ 7.400822] Unable to handle kernel paging request at virtual address 7402a8c8 [ 7.408419] pgd = c0004000 [ 7.411351] [7402a8c8] *pgd=00000000 [ 7.415121] Internal error: Oops: 5 [#1] SMP ARM [ 7.419968] Modules linked in: [ 7.423197] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 4.2.0-11195-ge81be90 #124 [ 7.432118] Hardware name: Generic AM33XX (Flattened Device Tree) [ 7.438509] task: c0a996c8 ti: c0a92000 task.ti: c0a92000 [ 7.444189] PC is at ip_route_input_noref+0x620/0xeb4 [ 7.449491] LR is at trace_hardirqs_on_caller+0xb0/0x1e8 [ 7.455055] pc : [<c054b49c>] lr : [<c008e198>] psr: 20000113 [ 7.455055] sp : c0a93c70 ip : 0000000f fp : ffffff8f [ 7.467072] r10: c0b35c80 r9 : 00000000 r8 : ce5f9400 [ 7.472543] r7 : ce5ef000 r6 : 00000000 r5 : ce615640 r4 : ce5ffd40 [ 7.479378] r3 : 7402a8c0 r2 : 00000001 r1 : c0549f74 r0 : ce5ffd40 [ 7.486216] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 7.493686] Control: 10c5387d Table: 80004019 DAC: 00000051 [ 7.499699] Process swapper/0 (pid: 0, stack limit = 0xc0a92218) [ 7.505989] Stack: (0xc0a93c70 to 0xc0a94000) [ 7.510561] 3c60: 00000000 00000000 c054afa4 00000000 [ 7.519131] 3c80: 00000000 c0b56080 00000000 00000000 00000000 00070044 00000004 00000000 [ 7.527705] 3ca0: 7402a8c0 0000012d 00000000 00000004 00000000 00000000 c0586298 00000000 [ 7.536278] 3cc0: 00000000 00000000 0102a8c0 7402a8c0 cfba6c34 00000000 c0a996c8 ce615640 [ 7.544852] 3ce0: c0551310 ce61b850 c07c6b10 c0b560b8 c0a95744 00000000 00000000 c0550668 [ 7.553422] 3d00: ce5ef000 c0a5ed00 60000113 c0090d44 00000001 ce615640 c04f85d0 00000050 [ 7.561996] 3d20: ce61b800 ce5ef000 c0a95744 00000000 00000000 c0551310 00000001 00000000 [ 7.570563] 3d40: c04f85d0 ce000140 ce638100 00000050 ce61b800 cfba6b10 ce61b99a ce61b99a [ 7.579135] 3d60: c0a82730 c0a92000 ce61b850 c0a5ed94 00000100 0102a8c0 c0a996c8 c0a9791c [ 7.587709] 3d80: ce5ef000 ce5ef048 00000008 ce615640 ce5ef05c ce6156ec ce615640 c04f85d0 [ 7.596289] 3da0: c0ad3f0c 00000000 c0a92000 ce5ef054 00000000 c04fb4e8 60000113 ce615640 [ 7.604869] 3dc0: ce5ef05c 00000000 00000000 c04fb4e8 00000000 00000000 c0b5697c ce6381c0 [ 7.613445] 3de0: 00000020 ce615640 c0b55f7e ce615640 ce5ef5c0 00000158 ce5f4550 ffff8db6 [ 7.622027] 3e00: 00000100 c04fb534 00000000 00000000 c04fb4e8 ce5ef000 00000000 40000113 [ 7.630603] 3e20: ce64f900 c04ed184 ce64f942 ce5ef000 ce5ef000 ce5ef000 ce6381c0 c041fec4 [ 7.639172] 3e40: 00010000 d0978000 ce5f2c50 00010000 d0978000 ce5f2c70 00000158 c041d35c [ 7.647750] 3e60: 00000001 00000000 ce5f2c50 00000040 cfb9f0c0 c0a93eb0 00000040 c041d3d8 [ 7.656322] 3e80: ce5ef5e8 ce5ef5e8 00000040 c0421f44 ce5ef5e8 00000001 0000012c c04fcd3c [ 7.664892] 3ea0: 60000193 0f10e000 cfb9f0c0 c0a910c0 c0a93eb0 c0a93eb0 c0a93eb8 c0a93eb8 [ 7.673466] 3ec0: c0b5697c c0a9408c 00000000 00000008 00000003 c0b565c4 00000004 0000000c [ 7.682039] 3ee0: 00000100 c0042614 c0a996c8 ce199900 0000000a ffff8db5 00200000 00000001 [ 7.690602] 3f00: 000000aa c0a8e828 00000000 c0a95744 00000000 00000001 ce017000 00000000 [ 7.699170] 3f20: c077471c c0042b3c c0a8e828 c009bb5c c0a93f50 c0010218 20000113 ffffffff [ 7.707733] 3f40: c0a93f84 c0b56e14 00000000 c076cef8 00000001 00000001 00000000 c0a996c8 [ 7.716309] 3f60: c0a92000 c0a94b2c 00000000 c0a94acc c0b56e14 00000000 00000000 c077471c [ 7.724881] 3f80: 00000000 c0a93fa0 c008e21c c0010218 20000113 ffffffff 00000051 c0010214 [ 7.733453] 3fa0: 00000000 c0083cb4 c0a8c4e4 c0a8f8c8 c0b64000 c0a94a00 00000000 c0a19c1c [ 7.742025] 3fc0: ffffffff ffffffff 00000000 c0a1967c 00000000 c0a76a48 c0b64214 c0a94ab0 [ 7.750588] 3fe0: c0a76a44 c0a9b258 80004059 413fc082 00000000 8000807c 00000000 00000000 [ 7.759169] [<c054b49c>] (ip_route_input_noref) from [<c0550668>] (ip_rcv_finish+0xd4/0x67c) [ 7.768010] [<c0550668>] (ip_rcv_finish) from [<c0551310>] (ip_rcv+0x490/0x72c) [ 7.775672] [<c0551310>] (ip_rcv) from [<c04f85d0>] (__netif_receive_skb_core+0x240/0x97c) [ 7.784339] [<c04f85d0>] (__netif_receive_skb_core) from [<c04fb534>] (netif_receive_skb_internal+0x5c/0x1d8) [ 7.794742] [<c04fb534>] (netif_receive_skb_internal) from [<c041fec4>] (cpsw_rx_handler+0xac/0x1e8) [ 7.804318] [<c041fec4>] (cpsw_rx_handler) from [<c041d35c>] (__cpdma_chan_process+0xe8/0x128) [ 7.813346] [<c041d35c>] (__cpdma_chan_process) from [<c041d3d8>] (cpdma_chan_process+0x3c/0x5c) [ 7.822544] [<c041d3d8>] (cpdma_chan_process) from [<c0421f44>] (cpsw_rx_poll+0x14/0x94) [ 7.831024] [<c0421f44>] (cpsw_rx_poll) from [<c04fcd3c>] (net_rx_action+0x1dc/0x33c) [ 7.839232] [<c04fcd3c>] (net_rx_action) from [<c0042614>] (__do_softirq+0x138/0x340) [ 7.847432] [<c0042614>] (__do_softirq) from [<c0042b3c>] (irq_exit+0xbc/0x130) [ 7.855103] [<c0042b3c>] (irq_exit) from [<c009bb5c>] (__handle_domain_irq+0x6c/0xe0) [ 7.863324] [<c009bb5c>] (__handle_domain_irq) from [<c076cef8>] (__irq_svc+0x58/0x78) [ 7.871639] [<c076cef8>] (__irq_svc) from [<c0010218>] (arch_cpu_idle+0x20/0x3c) [ 7.879392] [<c0010218>] (arch_cpu_idle) from [<c0083cb4>] (cpu_startup_entry+0x274/0x2fc) [ 7.888063] [<c0083cb4>] (cpu_startup_entry) from [<c0a19c1c>] (start_kernel+0x35c/0x3d0) [ 7.896632] [<c0a19c1c>] (start_kernel) from [<8000807c>] (0x8000807c) [ 7.903470] Code: e3530000 e3a02001 e5c4205e e584102c (15933008) [ 7.909931] ---[ end trace 4db905a3f2d0c634 ]--- [ 7.914774] Kernel panic - not syncing: Fatal exception in interrupt [ 7.921437] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
Second (With successful ip assign from dhcp, happens often):
[ 8.448158] task: ce58c540 ti: ce672000 task.ti: ce672000 [ 8.453822] PC is at iput+0x1a8/0x1f0 [ 8.457660] LR is at xprt_autoclose+0x2c/0x54 [ 8.462227] pc : [<c016c3d0>] lr : [<c066cc78>] psr: 20000113 [ 8.462227] sp : ce673e80 ip : 00000000 fp : c0b56688 [ 8.474251] r10: 00000001 r9 : ce673ec8 r8 : 00000000 [ 8.479720] r7 : feff3100 r6 : ce624800 r5 : ce624bc0 r4 : ce624b54 [ 8.486548] r3 : 00000060 r2 : 0000000f r1 : 0f10e000 r0 : cdbed720 [ 8.493383] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 8.500854] Control: 10c5387d Table: 80004019 DAC: 00000051 [ 8.506872] Process kworker/0:1H (pid: 768, stack limit = 0xce672218) [ 8.513619] Stack: (0xce673e80 to 0xce674000) [ 8.518195] 3e80: c066fc1c ce624b54 ce624bc0 ce624800 feff3100 00000000 ce673ec8 c066cc78 [ 8.526766] 3ea0: ce624b54 ce60f1c0 cfb9e340 c0057928 00000001 00000000 c00578b4 cfb9e340 [ 8.535336] 3ec0: c0057cc8 00000000 c137972c c0cc1960 00000000 c0997af8 cfb9e340 cfb9e340 [ 8.543904] 3ee0: ce60f1d8 cfb9e370 ce672000 00000008 c0b55ba0 ce60f1c0 cfb9e340 c0057c54 [ 8.552476] 3f00: ce654880 ce60f1c0 c0057c18 00000000 ce654880 ce60f1c0 c0057c18 00000000 [ 8.561042] 3f20: 00000000 00000000 00000000 c005d918 c0b5697c 00000000 00000000 ce60f1c0 [ 8.569613] 3f40: 00000000 00000000 dead4ead ffffffff ffffffff c0b65d60 00000000 00000000 [ 8.578184] 3f60: c09220a8 ce673f64 ce673f64 00000000 00000000 dead4ead ffffffff ffffffff [ 8.586762] 3f80: c0b65d60 00000000 00000000 c09220a8 ce673f90 ce673f90 ce673fac ce654880 [ 8.595340] 3fa0: c005d844 00000000 00000000 c000f770 00000000 00000000 00000000 00000000 [ 8.603913] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 8.612489] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 [ 8.621059] [<c016c3d0>] (iput) from [<c066cc78>] (xprt_autoclose+0x2c/0x54) [ 8.628460] [<c066cc78>] (xprt_autoclose) from [<c0057928>] (process_one_work+0x19c/0x48c) [ 8.637118] [<c0057928>] (process_one_work) from [<c0057c54>] (worker_thread+0x3c/0x4a0) [ 8.645603] [<c0057c54>] (worker_thread) from [<c005d918>] (kthread+0xd4/0xf0) [ 8.653186] [<c005d918>] (kthread) from [<c000f770>] (ret_from_fork+0x14/0x24) [ 8.660755] Code: e59f0044 e59f1044 ebfb467a eaffffc1 (e7f001f2) [ 8.667150] ---[ end trace ea119a84b2295a26 ]--- [ 8.672727] Unable to handle kernel paging request at virtual address ffffffd0 [ 8.680317] pgd = c0004000 [ 8.683156] [ffffffd0] *pgd=8fdf6861, *pte=00000000, *ppte=00000000 [ 8.689763] Internal error: Oops: 37 [#2] SMP ARM [ 8.694694] Modules linked in: [ 8.697915] CPU: 0 PID: 768 Comm: kworker/0:1H Tainted: G D W 4.2.0-11404-g227b9e8 #137 [ 8.707289] Hardware name: Generic AM33XX (Flattened Device Tree) [ 8.713689] task: ce58c540 ti: ce672000 task.ti: ce672000 [ 8.719351] PC is at kthread_data+0x4/0xc [ 8.723560] LR is at wq_worker_sleeping+0xc/0xd4 [ 8.728397] pc : [<c005dff0>] lr : [<c00589ec>] psr: 20000193 [ 8.728397] sp : ce673c18 ip : ce09ed9c fp : ce673c7c [ 8.740412] r10: 00000000 r9 : cfb9e640 r8 : ce58c90c [ 8.745880] r7 : c0a90640 r6 : cfb9e650 r5 : ce58c540 r4 : 00000000 [ 8.752712] r3 : 00000000 r2 : 00000000 r1 : 00000000 r0 : ce58c540 [ 8.759543] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none [ 8.767105] Control: 10c5387d Table: 80004019 DAC: 00000051 [ 8.773123] Process kworker/0:1H (pid: 768, stack limit = 0xce672218) [ 8.779862] Stack: (0xce673c18 to 0xce674000) [ 8.784429] 3c00: 00000020 cfb9e640 [ 8.793009] 3c20: ce58c540 c07679c0 ce58c540 c003fd14 00000001 00000000 c004046c c0ad6460 [ 8.801578] 3c40: c0b5697c c004046c ce297184 c0767d74 ce288dc8 00000001 c0a8c400 ce672000 [ 8.810146] 3c60: ce673954 ce673c98 ce58c868 00000000 00000001 c016c3d2 ce673c8c c0767d74 [ 8.818716] 3c80: 0420816c ce58c540 00000051 c0040554 c009b3f4 ce673cb4 ce673c98 ce673c98 [ 8.827288] 3ca0: c0a9b228 c0b64244 c0a9b228 60000193 0000000b 00000001 c0a9b228 c016c3d2 [ 8.835857] 3cc0: 00000051 c0013ca4 ce672218 0000000b c016c3d4 00000000 00000008 00000055 [ 8.844423] 3ce0: 00000051 00000055 65000000 30663935 20343430 66393565 34343031 66626520 [ 8.852995] 3d00: 37363462 61652061 66666666 28203163 30663765 32663130 c0002029 c016c3d0 [ 8.861565] 3d20: ce673df0 00030001 e7f001f2 c016c3d0 ce673df0 00000000 c076d4a4 ce672000 [ 8.870128] 3d40: c0b56688 c0009174 00000006 c13261e4 00000004 00000000 00030001 c016c3d0 [ 8.878700] 3d60: c0cc0970 c00136c0 c10b33c8 c000f770 00000000 00000000 00000000 ce673fb0 [ 8.887272] 3d80: c000f770 c000f770 c10b33c8 c0eb2788 00000020 c0cc1960 c0cc0970 ce58cac0 [ 8.895839] 3da0: c1351adc c008fd70 945722ab 0000000a ce58ca98 ce58ca70 00000001 ce58cab8 [ 8.904403] 3dc0: 00000000 ce58caa0 ce58c580 00000000 00000000 c016c3d4 00000000 c076d958 [ 8.912978] 3de0: 00000000 e7100000 c076d4a4 c076d4a4 cdbed720 0f10e000 0000000f 00000060 [ 8.921541] 3e00: ce624b54 ce624bc0 ce624800 feff3100 00000000 ce673ec8 00000001 c0b56688 [ 8.930105] 3e20: 00000000 ce673e80 c066cc78 c016c3d0 20000113 ffffffff 00000051 00000004 [ 8.938675] 3e40: 00000006 ce673ec8 00000001 c008e0b8 c0042974 ce58c540 00000001 ce6129b4 [ 8.947250] 3e60: 00000000 c008e21c 00000200 c066fb58 cdbed700 c0042974 ce58c540 ce624800 [ 8.955822] 3e80: c066fc1c ce624b54 ce624bc0 ce624800 feff3100 00000000 ce673ec8 c066cc78 [ 8.964397] 3ea0: ce624b54 ce60f1c0 cfb9e340 c0057928 00000001 00000000 c00578b4 cfb9e340 [ 8.972965] 3ec0: c0057cc8 00000000 c137972c c0cc1960 00000000 c0997af8 cfb9e340 cfb9e340 [ 8.981536] 3ee0: ce60f1d8 cfb9e370 ce672000 00000008 c0b55ba0 ce60f1c0 cfb9e340 c0057c54 [ 8.990104] 3f00: ce654880 ce60f1c0 c0057c18 00000000 ce654880 ce60f1c0 c0057c18 00000000 [ 8.998677] 3f20: 00000000 00000000 00000000 c005d918 c0b5697c 00000000 00000000 ce60f1c0 [ 9.007243] 3f40: 00000000 00000000 dead4ead ffffffff ffffffff c0b65d60 00000000 00000000 [ 9.015816] 3f60: c09220a8 ce673f64 ce673f64 00000001 00010001 dead4ead ffffffff ffffffff [ 9.024387] 3f80: c0b65d60 00000000 00000000 c09220a8 ce673f90 ce673f90 ce673fac ce654880 [ 9.032958] 3fa0: c005d844 00000000 00000000 c000f770 00000000 00000000 00000000 00000000 [ 9.041522] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 9.050089] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 [ 9.058673] [<c005dff0>] (kthread_data) from [<c00589ec>] (wq_worker_sleeping+0xc/0xd4) [ 9.067071] [<c00589ec>] (wq_worker_sleeping) from [<c07679c0>] (__schedule+0x684/0x9f8) [ 9.075548] [<c07679c0>] (__schedule) from [<c0767d74>] (schedule+0x40/0x98) [ 9.082940] [<c0767d74>] (schedule) from [<c0040554>] (do_exit+0x660/0x9bc) [ 9.090235] [<c0040554>] (do_exit) from [<c0013ca4>] (die+0x210/0x474) [ 9.097075] [<c0013ca4>] (die) from [<c0009174>] (do_undefinstr+0x15c/0x21c) [ 9.104475] [<c0009174>] (do_undefinstr) from [<c076d4a4>] (__und_svc_finish+0x0/0x3c) [ 9.112770] Exception stack(0xce673df0 to 0xce673e38) [ 9.118061] 3de0: cdbed720 0f10e000 0000000f 00000060 [ 9.126633] 3e00: ce624b54 ce624bc0 ce624800 feff3100 00000000 ce673ec8 00000001 c0b56688 [ 9.135199] 3e20: 00000000 ce673e80 c066cc78 c016c3d0 20000113 ffffffff [ 9.142136] [<c076d4a4>] (__und_svc_finish) from [<c016c3d0>] (iput+0x1a8/0x1f0) [ 9.149906] [<c016c3d0>] (iput) from [<c066cc78>] (xprt_autoclose+0x2c/0x54) [ 9.157301] [<c066cc78>] (xprt_autoclose) from [<c0057928>] (process_one_work+0x19c/0x48c) [ 9.165967] [<c0057928>] (process_one_work) from [<c0057c54>] (worker_thread+0x3c/0x4a0) [ 9.174450] [<c0057c54>] (worker_thread) from [<c005d918>] (kthread+0xd4/0xf0) [ 9.182025] [<c005d918>] (kthread) from [<c000f770>] (ret_from_fork+0x14/0x24) [ 9.189593] Code: e3a02002 eafffec7 c0774688 e5903380 (e5130030) [ 9.195979] ---[ end trace ea119a84b2295a27 ]--- [ 9.200814] Fixing recursive fault but reboot is needed!
--------
I tried to bisect this issue without success, sometimes it's booting fine and sometimes I got the above messages. Could anybody help me here to fixing this issue?
- Alex
Quoted 8 lines · Alexander Aring
Hi,
I am using am335x (beaglebone white) with "ip=dhcp" inside my bootparameters. It seems there exists some race, rarely it boots fine and often it crashed with following messages:
First (Without successful dhcp ip assign and is very rarely):
Quoted 12 lines
[ 6.249083] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.398943] . [ 7.400822] Unable to handle kernel paging request at virtual address 7402a8c8 [ 7.408419] pgd = c0004000 [ 7.411351] [7402a8c8] *pgd=00000000 [ 7.415121] Internal error: Oops: 5 [#1] SMP ARM [ 7.419968] Modules linked in: [ 7.423197] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 4.2.0-11195-ge81be90 #124 [ 7.432118] Hardware name: Generic AM33XX (Flattened Device Tree) [ 7.438509] task: c0a996c8 ti: c0a92000 task.ti: c0a92000 [ 7.444189] PC is at ip_route_input_noref+0x620/0xeb4 [ 7.449491] LR is at trace_hardirqs_on_caller+0xb0/0x1e8
<<<snip>>>
Quoted 1 line
[ 7.759169] [<c054b49c>] (ip_route_input_noref) from [<c0550668>] (ip_rcv_finish+0xd4/0x67c)
<<<snip>>>
Quoted 8 lines
--------
I tried to bisect this issue without success, sometimes it's booting fine and sometimes I got the above messages. Could anybody help me here to fixing this issue?
- Alex
Hi Alex,
This should be fixed by commit bde6f9ded1bd ("net: Initialize table in fib result") in net-next.
Cheers,
Nik
Hi,
Quoted 31 lines · Nikolay Aleksandrov
Quoted 8 lines · Alexander Aring
Hi,
I am using am335x (beaglebone white) with "ip=dhcp" inside my bootparameters. It seems there exists some race, rarely it boots fine and often it crashed with following messages:
First (Without successful dhcp ip assign and is very rarely):
Quoted 12 lines
[ 6.249083] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.398943] . [ 7.400822] Unable to handle kernel paging request at virtual address 7402a8c8 [ 7.408419] pgd = c0004000 [ 7.411351] [7402a8c8] *pgd=00000000 [ 7.415121] Internal error: Oops: 5 [#1] SMP ARM [ 7.419968] Modules linked in: [ 7.423197] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 4.2.0-11195-ge81be90 #124 [ 7.432118] Hardware name: Generic AM33XX (Flattened Device Tree) [ 7.438509] task: c0a996c8 ti: c0a92000 task.ti: c0a92000 [ 7.444189] PC is at ip_route_input_noref+0x620/0xeb4 [ 7.449491] LR is at trace_hardirqs_on_caller+0xb0/0x1e8<<<snip>>>
Quoted 1 line
[ 7.759169] [<c054b49c>] (ip_route_input_noref) from [<c0550668>] (ip_rcv_finish+0xd4/0x67c)<<<snip>>>
Quoted 7 lines
--------
I tried to bisect this issue without success, sometimes it's booting fine and sometimes I got the above messages. Could anybody help me here to fixing this issue?
Quoted 6 lines
Quoted 1 line
- Alex
Hi Alex,
This should be fixed by commitbde6f9ded1bd("net: Initialize table in fib result") in net-next.
I tried to cherry-picked this commit and saw this is already in net-next. I also tried to boot current net, the race is still there.
- Alex
Quoted 31 lines · Nikolay Aleksandrov
Quoted 8 lines · Alexander Aring
Hi,
I am using am335x (beaglebone white) with "ip=dhcp" inside my bootparameters. It seems there exists some race, rarely it boots fine and often it crashed with following messages:
First (Without successful dhcp ip assign and is very rarely):
Quoted 12 lines
[ 6.249083] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 6.398943] . [ 7.400822] Unable to handle kernel paging request at virtual address 7402a8c8 [ 7.408419] pgd = c0004000 [ 7.411351] [7402a8c8] *pgd=00000000 [ 7.415121] Internal error: Oops: 5 [#1] SMP ARM [ 7.419968] Modules linked in: [ 7.423197] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 4.2.0-11195-ge81be90 #124 [ 7.432118] Hardware name: Generic AM33XX (Flattened Device Tree) [ 7.438509] task: c0a996c8 ti: c0a92000 task.ti: c0a92000 [ 7.444189] PC is at ip_route_input_noref+0x620/0xeb4 [ 7.449491] LR is at trace_hardirqs_on_caller+0xb0/0x1e8<<<snip>>>
Quoted 1 line
[ 7.759169] [<c054b49c>] (ip_route_input_noref) from [<c0550668>] (ip_rcv_finish+0xd4/0x67c)<<<snip>>>
Quoted 7 lines
--------
I tried to bisect this issue without success, sometimes it's booting fine and sometimes I got the above messages. Could anybody help me here to fixing this issue?
Quoted 6 lines
Quoted 1 line
- Alex
Hi Alex,
This should be fixed by commitbde6f9ded1bd("net: Initialize table in fib result") in net-next.
Ok, I think I have two issues with two different races the first one was fixed by bde6f9ded1bd ("net: Initialize table in fib result"), but the second one is still there:
[ 8.615806] ------------[ cut here ]------------ [ 8.620678] Kernel BUG at c016c3d0 [verbose debug info unavailable] [ 8.627229] Internal error: Oops - BUG: 0 [#1] SMP ARM [ 8.632611] Modules linked in: [ 8.635836] CPU: 0 PID: 766 Comm: kworker/0:1H Tainted: G W 4.2.0-11248-gfbd0351 #140 [ 8.645208] Hardware name: Generic AM33XX (Flattened Device Tree) [ 8.651616] Workqueue: rpciod xprt_autoclose [ 8.656091] task: ce3c52c0 ti: ce642000 task.ti: ce642000 [ 8.661744] PC is at iput+0x1a8/0x1f0 [ 8.665579] LR is at xprt_autoclose+0x2c/0x54 [ 8.670136] pc : [<c016c3d0>] lr : [<c066c884>] psr: 20000113 [ 8.670136] sp : ce643e80 ip : 00000000 fp : c0b56688 [ 8.682133] r10: 00000001 r9 : ce643ec8 r8 : 00000000 [ 8.687599] r7 : feff3000 r6 : ce615800 r5 : ce615bc0 r4 : ce615b54 [ 8.694421] r3 : 00000060 r2 : 0000000f r1 : 0f10e000 r0 : cdbed720 [ 8.701254] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 8.708718] Control: 10c5387d Table: 80004019 DAC: 00000051 [ 8.714732] Process kworker/0:1H (pid: 766, stack limit = 0xce642218) [ 8.721464] Stack: (0xce643e80 to 0xce644000) [ 8.726033] 3e80: c066f828 ce615b54 ce615bc0 ce615800 feff3000 00000000 ce643ec8 c066c884 [ 8.734596] 3ea0: ce615b54 ce5ff440 cfb9e340 c0057928 00000001 00000000 c00578b4 cfb9e340 [ 8.743152] 3ec0: c0057cc8 00000000 c137972c c0cc1960 00000000 c09979f4 cfb9e340 cfb9e340 [ 8.751714] 3ee0: ce5ff458 cfb9e370 ce642000 00000008 c0b55ba0 ce5ff440 cfb9e340 c0057c54 [ 8.760274] 3f00: ce659940 ce5ff440 c0057c18 00000000 ce659940 ce5ff440 c0057c18 00000000 [ 8.768834] 3f20: 00000000 00000000 00000000 c005d918 c0b5697c 00000000 00000000 ce5ff440 [ 8.777390] 3f40: 00000000 00000000 dead4ead ffffffff ffffffff c0b65d60 00000000 00000000 [ 8.785951] 3f60: c0922088 ce643f64 ce643f64 00000000 00000000 dead4ead ffffffff ffffffff [ 8.794513] 3f80: c0b65d60 00000000 00000000 c0922088 ce643f90 ce643f90 ce643fac ce659940 [ 8.803069] 3fa0: c005d844 00000000 00000000 c000f770 00000000 00000000 00000000 00000000 [ 8.811628] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 8.820185] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 8fdf6861 8fdf6c61 [ 8.828741] [<c016c3d0>] (iput) from [<c066c884>] (xprt_autoclose+0x2c/0x54) [ 8.836133] [<c066c884>] (xprt_autoclose) from [<c0057928>] (process_one_work+0x19c/0x48c) [ 8.844784] [<c0057928>] (process_one_work) from [<c0057c54>] (worker_thread+0x3c/0x4a0) [ 8.853256] [<c0057c54>] (worker_thread) from [<c005d918>] (kthread+0xd4/0xf0) [ 8.860827] [<c005d918>] (kthread) from [<c000f770>] (ret_from_fork+0x14/0x24) [ 8.868387] Code: e59f0044 e59f1044 ebfb467a eaffffc1 (e7f001f2) [ 8.874771] ---[ end trace 6c5e5808fb5c8423 ]--- [ 8.880045] Unable to handle kernel paging request at virtual address ffffffd0 [ 8.887603] pgd = c0004000 [ 8.890438] [ffffffd0] *pgd=8fdf6861, *pte=00000000, *ppte=00000000 [ 8.897034] Internal error: Oops: 37 [#2] SMP ARM [ 8.901950] Modules linked in: [ 8.905165] CPU: 0 PID: 766 Comm: kworker/0:1H Tainted: G D W 4.2.0-11248-gfbd0351 #140 [ 8.914530] Hardware name: Generic AM33XX (Flattened Device Tree) [ 8.920923] task: ce3c52c0 ti: ce642000 task.ti: ce642000 [ 8.926576] PC is at kthread_data+0x4/0xc [ 8.930776] LR is at wq_worker_sleeping+0xc/0xd4 [ 8.935609] pc : [<c005dff0>] lr : [<c00589ec>] psr: 20000193 [ 8.935609] sp : ce643c18 ip : ce0b6e1c fp : ce643c7c [ 8.947599] r10: 00000000 r9 : cfb9e640 r8 : ce3c568c [ 8.953059] r7 : c0a90640 r6 : cfb9e650 r5 : ce3c52c0 r4 : 00000000 [ 8.959883] r3 : 00000000 r2 : 00000000 r1 : 00000000 r0 : ce3c52c0 [ 8.966708] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none [ 8.974263] Control: 10c5387d Table: 80004019 DAC: 00000051 [ 8.980271] Process kworker/0:1H (pid: 766, stack limit = 0xce642218) [ 8.987000] Stack: (0xce643c18 to 0xce644000) [ 8.991563] 3c00: 00000020 cfb9e640 [ 9.000123] 3c20: ce3c52c0 c0767578 ce3c52c0 c003fd14 00000001 00000000 c004046c c0ad6460 [ 9.008679] 3c40: c0b5697c c004046c ce2c31c4 c076792c ce390808 00000001 c0a8c400 ce642000 [ 9.017237] 3c60: ce643954 ce643c98 ce3c55e8 00000000 00000001 c016c3d2 ce643c8c c076792c [ 9.025794] 3c80: 0420816c ce3c52c0 00000051 c0040554 c009b3f4 ce643cb4 ce643c98 ce643c98 [ 9.034347] 3ca0: c0a9b228 c0b64244 c0a9b228 60000193 0000000b 00000001 c0a9b228 c016c3d2 [ 9.042911] 3cc0: 00000051 c0013ca4 ce642218 0000000b c016c3d4 00000000 00000008 00000055 [ 9.051470] 3ce0: 00000051 00000055 65000000 30663935 20343430 66393565 34343031 66626520 [ 9.060030] 3d00: 37363462 61652061 66666666 28203163 30663765 32663130 c0002029 c016c3d0 [ 9.068591] 3d20: ce643df0 00030001 e7f001f2 c016c3d0 ce643df0 00000000 c076d064 ce642000 [ 9.077155] 3d40: c0b56688 c0009174 00000006 c13261e4 00000004 00000000 00030001 c016c3d0 [ 9.085710] 3d60: c0cc0a80 c00136c0 c10b33c8 c000f770 00000000 00000000 00000000 ce643fb0 [ 9.094274] 3d80: c000f770 c000f770 c10b33c8 c0eb2788 00000020 c0cc1960 c0cc0a80 ce3c5840 [ 9.102834] 3da0: c1351adc c008fd70 985722ac 0000000a ce3c5818 ce3c57f0 00000001 ce3c5838 [ 9.111390] 3dc0: 00000000 ce3c5820 ce3c5300 00000000 00000000 c016c3d4 00000000 c076d518 [ 9.119949] 3de0: 00000000 e7100000 c076d064 c076d064 cdbed720 0f10e000 0000000f 00000060 [ 9.128508] 3e00: ce615b54 ce615bc0 ce615800 feff3000 00000000 ce643ec8 00000001 c0b56688 [ 9.137066] 3e20: 00000000 ce643e80 c066c884 c016c3d0 20000113 ffffffff 00000051 00000004 [ 9.145618] 3e40: 00000006 ce643ec8 00000001 c008e0b8 c0042974 ce3c52c0 00000001 ce6619b4 [ 9.154183] 3e60: 00000000 c008e21c 00000200 c066f764 cdbed700 c0042974 ce3c52c0 ce615800 [ 9.162742] 3e80: c066f828 ce615b54 ce615bc0 ce615800 feff3000 00000000 ce643ec8 c066c884 [ 9.171301] 3ea0: ce615b54 ce5ff440 cfb9e340 c0057928 00000001 00000000 c00578b4 cfb9e340 [ 9.179856] 3ec0: c0057cc8 00000000 c137972c c0cc1960 00000000 c09979f4 cfb9e340 cfb9e340 [ 9.188414] 3ee0: ce5ff458 cfb9e370 ce642000 00000008 c0b55ba0 ce5ff440 cfb9e340 c0057c54 [ 9.196969] 3f00: ce659940 ce5ff440 c0057c18 00000000 ce659940 ce5ff440 c0057c18 00000000 [ 9.205528] 3f20: 00000000 00000000 00000000 c005d918 c0b5697c 00000000 00000000 ce5ff440 [ 9.214086] 3f40: 00000000 00000000 dead4ead ffffffff ffffffff c0b65d60 00000000 00000000 [ 9.222642] 3f60: c0922088 ce643f64 ce643f64 00000001 00010001 dead4ead ffffffff ffffffff [ 9.231196] 3f80: c0b65d60 00000000 00000000 c0922088 ce643f90 ce643f90 ce643fac ce659940 [ 9.239755] 3fa0: c005d844 00000000 00000000 c000f770 00000000 00000000 00000000 00000000 [ 9.248316] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 9.256869] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 8fdf6861 8fdf6c61 [ 9.265435] [<c005dff0>] (kthread_data) from [<c00589ec>] (wq_worker_sleeping+0xc/0xd4) [ 9.273822] [<c00589ec>] (wq_worker_sleeping) from [<c0767578>] (__schedule+0x684/0x9f8) [ 9.282288] [<c0767578>] (__schedule) from [<c076792c>] (schedule+0x40/0x98) [ 9.289673] [<c076792c>] (schedule) from [<c0040554>] (do_exit+0x660/0x9bc) [ 9.296958] [<c0040554>] (do_exit) from [<c0013ca4>] (die+0x210/0x474) [ 9.303790] [<c0013ca4>] (die) from [<c0009174>] (do_undefinstr+0x15c/0x21c) [ 9.311177] [<c0009174>] (do_undefinstr) from [<c076d064>] (__und_svc_finish+0x0/0x3c) [ 9.319453] Exception stack(0xce643df0 to 0xce643e38) [ 9.324738] 3de0: cdbed720 0f10e000 0000000f 00000060 [ 9.333293] 3e00: ce615b54 ce615bc0 ce615800 feff3000 00000000 ce643ec8 00000001 c0b56688 [ 9.341848] 3e20: 00000000 ce643e80 c066c884 c016c3d0 20000113 ffffffff [ 9.348771] [<c076d064>] (__und_svc_finish) from [<c016c3d0>] (iput+0x1a8/0x1f0) [ 9.356513] [<c016c3d0>] (iput) from [<c066c884>] (xprt_autoclose+0x2c/0x54) [ 9.363897] [<c066c884>] (xprt_autoclose) from [<c0057928>] (process_one_work+0x19c/0x48c) [ 9.372543] [<c0057928>] (process_one_work) from [<c0057c54>] (worker_thread+0x3c/0x4a0) [ 9.381005] [<c0057c54>] (worker_thread) from [<c005d918>] (kthread+0xd4/0xf0) [ 9.388565] [<c005d918>] (kthread) from [<c000f770>] (ret_from_fork+0x14/0x24) [ 9.396129] Code: e3a02002 eafffec7 c0774688 e5903380 (e5130030) [ 9.402509] ---[ end trace 6c5e5808fb5c8424 ]--- [ 9.407333] Fixing recursive fault but reboot is needed!
- Alex
Hi,
On Wed, Sep 23, 2015 at 01:57:57PM +0200, Alexander Aring wrote:
...
Quoted 35 lines
Quoted 1 line
Ok, I think I have two issues with two different races the first one was fixed by
bde6f9ded1bd("net: Initialize table in fib result"), but the second one is still there:[ 8.615806] ------------[ cut here ]------------ [ 8.620678] Kernel BUG at c016c3d0 [verbose debug info unavailable] [ 8.627229] Internal error: Oops - BUG: 0 [#1] SMP ARM [ 8.632611] Modules linked in: [ 8.635836] CPU: 0 PID: 766 Comm: kworker/0:1H Tainted: G W 4.2.0-11248-gfbd0351 #140 [ 8.645208] Hardware name: Generic AM33XX (Flattened Device Tree) [ 8.651616] Workqueue: rpciod xprt_autoclose [ 8.656091] task: ce3c52c0 ti: ce642000 task.ti: ce642000 [ 8.661744] PC is at iput+0x1a8/0x1f0 [ 8.665579] LR is at xprt_autoclose+0x2c/0x54 [ 8.670136] pc : [<c016c3d0>] lr : [<c066c884>] psr: 20000113 [ 8.670136] sp : ce643e80 ip : 00000000 fp : c0b56688 [ 8.682133] r10: 00000001 r9 : ce643ec8 r8 : 00000000 [ 8.687599] r7 : feff3000 r6 : ce615800 r5 : ce615bc0 r4 : ce615b54 [ 8.694421] r3 : 00000060 r2 : 0000000f r1 : 0f10e000 r0 : cdbed720 [ 8.701254] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 8.708718] Control: 10c5387d Table: 80004019 DAC: 00000051 [ 8.714732] Process kworker/0:1H (pid: 766, stack limit = 0xce642218) [ 8.721464] Stack: (0xce643e80 to 0xce644000) [ 8.726033] 3e80: c066f828 ce615b54 ce615bc0 ce615800 feff3000 00000000 ce643ec8 c066c884 [ 8.734596] 3ea0: ce615b54 ce5ff440 cfb9e340 c0057928 00000001 00000000 c00578b4 cfb9e340 [ 8.743152] 3ec0: c0057cc8 00000000 c137972c c0cc1960 00000000 c09979f4 cfb9e340 cfb9e340 [ 8.751714] 3ee0: ce5ff458 cfb9e370 ce642000 00000008 c0b55ba0 ce5ff440 cfb9e340 c0057c54 [ 8.760274] 3f00: ce659940 ce5ff440 c0057c18 00000000 ce659940 ce5ff440 c0057c18 00000000 [ 8.768834] 3f20: 00000000 00000000 00000000 c005d918 c0b5697c 00000000 00000000 ce5ff440 [ 8.777390] 3f40: 00000000 00000000 dead4ead ffffffff ffffffff c0b65d60 00000000 00000000 [ 8.785951] 3f60: c0922088 ce643f64 ce643f64 00000000 00000000 dead4ead ffffffff ffffffff [ 8.794513] 3f80: c0b65d60 00000000 00000000 c0922088 ce643f90 ce643f90 ce643fac ce659940 [ 8.803069] 3fa0: c005d844 00000000 00000000 c000f770 00000000 00000000 00000000 00000000
Quoted 8 lines
[ 8.811628] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 8.820185] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 8fdf6861 8fdf6c61 [ 8.828741] [<c016c3d0>] (iput) from [<c066c884>] (xprt_autoclose+0x2c/0x54) [ 8.836133] [<c066c884>] (xprt_autoclose) from [<c0057928>] (process_one_work+0x19c/0x48c) [ 8.844784] [<c0057928>] (process_one_work) from [<c0057c54>] (worker_thread+0x3c/0x4a0) [ 8.853256] [<c0057c54>] (worker_thread) from [<c005d918>] (kthread+0xd4/0xf0) [ 8.860827] [<c005d918>] (kthread) from [<c000f770>] (ret_from_fork+0x14/0x24) [ 8.868387] Code: e59f0044 e59f1044 ebfb467a eaffffc1 (e7f001f2)
Additional missing information is that I am booting via nfsroot and xprt_autoclose is something from sunrpc.
Finally I figured out that commit 4876cc779ff525b9c2376d8076edf47815e71f2c ("SUNRPC: Ensure we release the TCP socket once it has been closed") occur this races. After reverting this commit everything works fine.
I added now:
Steven Rostedt <[email protected]>
Trond Myklebust <[email protected]>
to cc to report about this issue.
- Alex
On Wed, 23 Sep 2015 14:27:15 +0200
Alexander Aring <[email protected]> wrote:
\
Quoted 4 lines
Finally I figured out that commit
4876cc779ff525b9c2376d8076edf47815e71f2c("SUNRPC: Ensure we release the TCP socket once it has been closed") occur this races. After reverting this commit everything works fine.
Yay! Everybody sing! "Fix one bug and replace it by another..."
-- Steve
+linux-nfs mailing list
Quoted 56 lines · Alexander Aring
Hi,
On Wed, Sep 23, 2015 at 01:57:57PM +0200, Alexander Aring wrote:
...Quoted 35 lines
Quoted 1 line
Ok, I think I have two issues with two different races the first one was fixed by
bde6f9ded1bd("net: Initialize table in fib result"), but the second one is still there:[ 8.615806] ------------[ cut here ]------------ [ 8.620678] Kernel BUG at c016c3d0 [verbose debug info unavailable] [ 8.627229] Internal error: Oops - BUG: 0 [#1] SMP ARM [ 8.632611] Modules linked in: [ 8.635836] CPU: 0 PID: 766 Comm: kworker/0:1H Tainted: G W 4.2.0-11248-gfbd0351 #140 [ 8.645208] Hardware name: Generic AM33XX (Flattened Device Tree) [ 8.651616] Workqueue: rpciod xprt_autoclose [ 8.656091] task: ce3c52c0 ti: ce642000 task.ti: ce642000 [ 8.661744] PC is at iput+0x1a8/0x1f0 [ 8.665579] LR is at xprt_autoclose+0x2c/0x54 [ 8.670136] pc : [<c016c3d0>] lr : [<c066c884>] psr: 20000113 [ 8.670136] sp : ce643e80 ip : 00000000 fp : c0b56688 [ 8.682133] r10: 00000001 r9 : ce643ec8 r8 : 00000000 [ 8.687599] r7 : feff3000 r6 : ce615800 r5 : ce615bc0 r4 : ce615b54 [ 8.694421] r3 : 00000060 r2 : 0000000f r1 : 0f10e000 r0 : cdbed720 [ 8.701254] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 8.708718] Control: 10c5387d Table: 80004019 DAC: 00000051 [ 8.714732] Process kworker/0:1H (pid: 766, stack limit = 0xce642218) [ 8.721464] Stack: (0xce643e80 to 0xce644000) [ 8.726033] 3e80: c066f828 ce615b54 ce615bc0 ce615800 feff3000 00000000 ce643ec8 c066c884 [ 8.734596] 3ea0: ce615b54 ce5ff440 cfb9e340 c0057928 00000001 00000000 c00578b4 cfb9e340 [ 8.743152] 3ec0: c0057cc8 00000000 c137972c c0cc1960 00000000 c09979f4 cfb9e340 cfb9e340 [ 8.751714] 3ee0: ce5ff458 cfb9e370 ce642000 00000008 c0b55ba0 ce5ff440 cfb9e340 c0057c54 [ 8.760274] 3f00: ce659940 ce5ff440 c0057c18 00000000 ce659940 ce5ff440 c0057c18 00000000 [ 8.768834] 3f20: 00000000 00000000 00000000 c005d918 c0b5697c 00000000 00000000 ce5ff440 [ 8.777390] 3f40: 00000000 00000000 dead4ead ffffffff ffffffff c0b65d60 00000000 00000000 [ 8.785951] 3f60: c0922088 ce643f64 ce643f64 00000000 00000000 dead4ead ffffffff ffffffff [ 8.794513] 3f80: c0b65d60 00000000 00000000 c0922088 ce643f90 ce643f90 ce643fac ce659940 [ 8.803069] 3fa0: c005d844 00000000 00000000 c000f770 00000000 00000000 00000000 00000000Quoted 8 lines
[ 8.811628] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 8.820185] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 8fdf6861 8fdf6c61 [ 8.828741] [<c016c3d0>] (iput) from [<c066c884>] (xprt_autoclose+0x2c/0x54) [ 8.836133] [<c066c884>] (xprt_autoclose) from [<c0057928>] (process_one_work+0x19c/0x48c) [ 8.844784] [<c0057928>] (process_one_work) from [<c0057c54>] (worker_thread+0x3c/0x4a0) [ 8.853256] [<c0057c54>] (worker_thread) from [<c005d918>] (kthread+0xd4/0xf0) [ 8.860827] [<c005d918>] (kthread) from [<c000f770>] (ret_from_fork+0x14/0x24) [ 8.868387] Code: e59f0044 e59f1044 ebfb467a eaffffc1 (e7f001f2)Additional missing information is that I am booting via nfsroot and
xprt_autocloseis something from sunrpc.Finally I figured out that commit
4876cc779ff525b9c2376d8076edf47815e71f2c("SUNRPC: Ensure we release the TCP socket once it has been closed") occur this races. After reverting this commit everything works fine.
Quoted 7 lines
I added now:
Steven Rostedt <[email protected]>
Trond Myklebust <[email protected]>to cc to report about this issue.
Is that happening when the transport is being torn down? If so, is it
fixed by http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=commitdiff;h=79234c3db6842a3de03817211d891e0c2878f756
?
Hi,
On Wed, Sep 23, 2015 at 09:16:00AM -0400, Trond Myklebust wrote:
...
Quoted 4 lines
Is that happening when the transport is being torn down? If so, is it
fixed by http://git.linux-nfs.org/?p=trondmy/linux-nfs.git;a=commitdiff;h=79234c3db6842a3de03817211d891e0c2878f756
?
thanks. This patch fixed my issue.
- Alex