Skip to contentSkip to conversation

bat_events: page allocation failure (batman-adv maint)

B.A.T.M.A.N. Applied Linus Lüssing 5 participants 37 messages Jan 23, 2010 – Mar 4, 2010

lore
Share & tools
lore
#0 lore raw

Hey guys,

I've been installing a 9 node setup here in our cellar. They are all running B.A.T.M.A.N. adv 0.2.1-beta r1545 (so the current batman-adv maintance version in OpenWRT).

The result over night:
- 1x: bat_events: page allocation failure
- 3x: WARNING: at net/sched/sch_generic.c:226 0x801c3554() (those 3 had TQ values of about 10-15 in batctl)

For the last error, I'm not sure if this is related to batman-adv or to the madwifi driver. I was using OpenWRT trunk 18405 on all routers.

I'm attaching some backtraces. Sorry again I can't get more out of them than that with ksymoops. The first two ones are the bat_events problem, the others the scheduling thingy.

Cheers, Linus

root@OpenWrt:~# dmesg
01a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<8022329c>] 0x8022329c
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<8022329c>] 0x8022329c
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
wifi0: ath_uapsd_processtriggers: Dropping; skb is NULL in received ath_buf.
wifi0: ath_uapsd_processtriggers: Dropping; skb is NULL in received ath_buf.
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<8007d3e4>] 0x8007d3e4
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<80041444>] 0x80041444
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<8018c538>] 0x8018c538
[<800bcf78>] 0x800bcf78
[<800bd450>] 0x800bd450
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<80074174>] 0x80074174
[<80b42070>] 0x80b42070
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:149 active_file:564 inactive_anon:159
inactive_file:828 dirty:0 writeback:0 unstable:0
free:48 slab:1126 mapped:279 pagetables:55 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:596kB inactive_anon:636kB active_file:2256kB inactive_file:3312kB present:16256kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
1417 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
699 pages reserved
1395 pages shared
2630 pages non-shared
device VPN entered promiscuous mode
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 2(VPN) entering forwarding state
root@OpenWrt:~#

ksymoops 2.4.11 on x86_64 2.6.31-1-amd64. Options used

     -v /home/linus/lff-test/kamikaze/build_dir/linux-atheros/linux-2.6.30.9/vmlinux (specified)
     -K (specified)
     -L (specified)
     -o /home/linus/lff-test/kamikaze/build_dir/linux-atheros -o /home/linus/lff-test/kamikaze/build_dir/target-mips_uClibc-0.9.30.1 (specified)
     -m /home/linus/lff-test/kamikaze/build_dir/linux-atheros/linux-2.6.30.9/System.map (specified)

No modules in ksyms, skipping objects
01a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<8022329c>] 0x8022329c
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 8009ac88 <__alloc_pages_internal+3ec/434>
Trace; 800bcf78 <cache_alloc_refill+2d8/6fc>
Trace; 800bd538 <kmem_cache_alloc+78/e8>
Trace; 80b41a58 <END_OF_CODE+88c328/????>
Trace; 80074b5c <queue_delayed_work_on+f8/108>
Trace; 80b42674 <END_OF_CODE+88cf44/????>
Trace; 80b479f8 <END_OF_CODE+8922c8/????>
Trace; 8052129c <END_OF_CODE+26bb6c/????>
Trace; 8051c178 <END_OF_CODE+266a48/????>
Trace; 80525a20 <END_OF_CODE+2702f0/????>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 805269e0 <END_OF_CODE+2712b0/????>
Trace; 80525ed8 <END_OF_CODE+2707a8/????>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 80503f50 <END_OF_CODE+24e820/????>
Trace; 8051910c <END_OF_CODE+2639dc/????>
Trace; 801b0fb8 <net_rx_action+a8/24c>
Trace; 801ae580 <net_tx_action+1c8/1f0>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 8009ac90 <__alloc_pages_internal+3f4/434>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 801784a0 <mem_serial_out+0/20>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 800bd470 <__kmalloc+d4/124>
Trace; 801a8664 <__alloc_skb+70/148>
Trace; 801a9120 <skb_copy+40/c8>
Trace; 8018c4b8 <ar231x_start_xmit+0/d0>
Trace; 80b42088 <END_OF_CODE+88c958/????>
Trace; 80b42070 <END_OF_CODE+88c940/????>
Trace; 80b419d4 <END_OF_CODE+88c2a4/????>
Trace; 8007425c <queue_work_on+90/a0>
Trace; 800745c8 <worker_thread+1ac/260>
Trace; 800789b4 <kthread+0/a0>
Trace; 800790a8 <autoremove_wake_function+0/4c>
Trace; 8007441c <worker_thread+0/260>
Trace; 80078a10 <kthread+5c/a0>
Trace; 8004e988 <kernel_thread_helper+10/18>

1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<8022329c>] 0x8022329c
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 8009ac88 <__alloc_pages_internal+3ec/434>
Trace; 800bcf78 <cache_alloc_refill+2d8/6fc>
Trace; 800bd538 <kmem_cache_alloc+78/e8>
Trace; 80b41a58 <END_OF_CODE+88c328/????>
Trace; 80074b5c <queue_delayed_work_on+f8/108>
Trace; 80b42674 <END_OF_CODE+88cf44/????>
Trace; 80b479f8 <END_OF_CODE+8922c8/????>
Trace; 8052129c <END_OF_CODE+26bb6c/????>
Trace; 8051c178 <END_OF_CODE+266a48/????>
Trace; 80525a20 <END_OF_CODE+2702f0/????>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 805269e0 <END_OF_CODE+2712b0/????>
Trace; 80525ed8 <END_OF_CODE+2707a8/????>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 80503f50 <END_OF_CODE+24e820/????>
Trace; 8051910c <END_OF_CODE+2639dc/????>
Trace; 801b0fb8 <net_rx_action+a8/24c>
Trace; 801ae580 <net_tx_action+1c8/1f0>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 8009ac90 <__alloc_pages_internal+3f4/434>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 801784a0 <mem_serial_out+0/20>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 800bd470 <__kmalloc+d4/124>
Trace; 801a8664 <__alloc_skb+70/148>
Trace; 801a9120 <skb_copy+40/c8>
Trace; 8018c4b8 <ar231x_start_xmit+0/d0>
Trace; 80b42088 <END_OF_CODE+88c958/????>
Trace; 80b42070 <END_OF_CODE+88c940/????>
Trace; 80b419d4 <END_OF_CODE+88c2a4/????>
Trace; 8007425c <queue_work_on+90/a0>
Trace; 800745c8 <worker_thread+1ac/260>
Trace; 800789b4 <kthread+0/a0>
Trace; 800790a8 <autoremove_wake_function+0/4c>
Trace; 8007441c <worker_thread+0/260>
Trace; 80078a10 <kthread+5c/a0>
Trace; 8004e988 <kernel_thread_helper+10/18>

1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 8009ac88 <__alloc_pages_internal+3ec/434>
Trace; 800bcf78 <cache_alloc_refill+2d8/6fc>
Trace; 800bd538 <kmem_cache_alloc+78/e8>
Trace; 80b41a58 <END_OF_CODE+88c328/????>
Trace; 80074b5c <queue_delayed_work_on+f8/108>
Trace; 80b42674 <END_OF_CODE+88cf44/????>
Trace; 80b479f8 <END_OF_CODE+8922c8/????>
Trace; 8052129c <END_OF_CODE+26bb6c/????>
Trace; 8051c178 <END_OF_CODE+266a48/????>
Trace; 80525a20 <END_OF_CODE+2702f0/????>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 805269e0 <END_OF_CODE+2712b0/????>
Trace; 80525ed8 <END_OF_CODE+2707a8/????>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 80503f50 <END_OF_CODE+24e820/????>
Trace; 8051910c <END_OF_CODE+2639dc/????>
Trace; 801b0fb8 <net_rx_action+a8/24c>
Trace; 801ae580 <net_tx_action+1c8/1f0>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 8009ac90 <__alloc_pages_internal+3f4/434>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 801784a0 <mem_serial_out+0/20>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 800bd470 <__kmalloc+d4/124>
Trace; 801a8664 <__alloc_skb+70/148>
Trace; 801a9120 <skb_copy+40/c8>
Trace; 8018c4b8 <ar231x_start_xmit+0/d0>
Trace; 80b42088 <END_OF_CODE+88c958/????>
Trace; 80b42070 <END_OF_CODE+88c940/????>
Trace; 80b419d4 <END_OF_CODE+88c2a4/????>
Trace; 8007425c <queue_work_on+90/a0>
Trace; 800745c8 <worker_thread+1ac/260>
Trace; 800789b4 <kthread+0/a0>
Trace; 800790a8 <autoremove_wake_function+0/4c>
Trace; 8007441c <worker_thread+0/260>
Trace; 80078a10 <kthread+5c/a0>
Trace; 8004e988 <kernel_thread_helper+10/18>

1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 8009ac88 <__alloc_pages_internal+3ec/434>
Trace; 800bcf78 <cache_alloc_refill+2d8/6fc>
Trace; 800bd538 <kmem_cache_alloc+78/e8>
Trace; 80b41a58 <END_OF_CODE+88c328/????>
Trace; 80074b5c <queue_delayed_work_on+f8/108>
Trace; 80b42674 <END_OF_CODE+88cf44/????>
Trace; 80b479f8 <END_OF_CODE+8922c8/????>
Trace; 8052129c <END_OF_CODE+26bb6c/????>
Trace; 8051c178 <END_OF_CODE+266a48/????>
Trace; 80525a20 <END_OF_CODE+2702f0/????>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 805269e0 <END_OF_CODE+2712b0/????>
Trace; 80525ed8 <END_OF_CODE+2707a8/????>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 80503f50 <END_OF_CODE+24e820/????>
Trace; 8051910c <END_OF_CODE+2639dc/????>
Trace; 801b0fb8 <net_rx_action+a8/24c>
Trace; 801ae580 <net_tx_action+1c8/1f0>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 8009ac90 <__alloc_pages_internal+3f4/434>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 801784a0 <mem_serial_out+0/20>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 800bd470 <__kmalloc+d4/124>
Trace; 801a8664 <__alloc_skb+70/148>
Trace; 801a9120 <skb_copy+40/c8>
Trace; 8018c4b8 <ar231x_start_xmit+0/d0>
Trace; 80b42088 <END_OF_CODE+88c958/????>
Trace; 80b42070 <END_OF_CODE+88c940/????>
Trace; 80b419d4 <END_OF_CODE+88c2a4/????>
Trace; 8007425c <queue_work_on+90/a0>
Trace; 800745c8 <worker_thread+1ac/260>
Trace; 800789b4 <kthread+0/a0>
Trace; 800790a8 <autoremove_wake_function+0/4c>
Trace; 8007441c <worker_thread+0/260>
Trace; 80078a10 <kthread+5c/a0>
Trace; 8004e988 <kernel_thread_helper+10/18>

1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 8009ac88 <__alloc_pages_internal+3ec/434>
Trace; 800bcf78 <cache_alloc_refill+2d8/6fc>
Trace; 800bd538 <kmem_cache_alloc+78/e8>
Trace; 80b41a58 <END_OF_CODE+88c328/????>
Trace; 80074b5c <queue_delayed_work_on+f8/108>
Trace; 80b42674 <END_OF_CODE+88cf44/????>
Trace; 80b479f8 <END_OF_CODE+8922c8/????>
Trace; 8052129c <END_OF_CODE+26bb6c/????>
Trace; 8051c178 <END_OF_CODE+266a48/????>
Trace; 80525a20 <END_OF_CODE+2702f0/????>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 805269e0 <END_OF_CODE+2712b0/????>
Trace; 80525ed8 <END_OF_CODE+2707a8/????>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 80503f50 <END_OF_CODE+24e820/????>
Trace; 8051910c <END_OF_CODE+2639dc/????>
Trace; 801b0fb8 <net_rx_action+a8/24c>
Trace; 801ae580 <net_tx_action+1c8/1f0>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 8009ac90 <__alloc_pages_internal+3f4/434>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 801784a0 <mem_serial_out+0/20>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 800bd470 <__kmalloc+d4/124>
Trace; 801a8664 <__alloc_skb+70/148>
Trace; 801a9120 <skb_copy+40/c8>
Trace; 8018c4b8 <ar231x_start_xmit+0/d0>
Trace; 80b42088 <END_OF_CODE+88c958/????>
Trace; 80b42070 <END_OF_CODE+88c940/????>
Trace; 80b419d4 <END_OF_CODE+88c2a4/????>
Trace; 8007425c <queue_work_on+90/a0>
Trace; 800745c8 <worker_thread+1ac/260>
Trace; 800789b4 <kthread+0/a0>
Trace; 800790a8 <autoremove_wake_function+0/4c>
Trace; 8007441c <worker_thread+0/260>
Trace; 80078a10 <kthread+5c/a0>
Trace; 8004e988 <kernel_thread_helper+10/18>

1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80074b5c>] 0x80074b5c
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<8051c178>] 0x8051c178
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<8022329c>] 0x8022329c
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<8022329c>] 0x8022329c
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 8009ac88 <__alloc_pages_internal+3ec/434>
Trace; 800bcf78 <cache_alloc_refill+2d8/6fc>
Trace; 800bd538 <kmem_cache_alloc+78/e8>
Trace; 80b41a58 <END_OF_CODE+88c328/????>
Trace; 80074b5c <queue_delayed_work_on+f8/108>
Trace; 80b42674 <END_OF_CODE+88cf44/????>
Trace; 80b479f8 <END_OF_CODE+8922c8/????>
Trace; 8052129c <END_OF_CODE+26bb6c/????>
Trace; 8051c178 <END_OF_CODE+266a48/????>
Trace; 80525a20 <END_OF_CODE+2702f0/????>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 805269e0 <END_OF_CODE+2712b0/????>
Trace; 80525ed8 <END_OF_CODE+2707a8/????>
Trace; 8022329c <br_handle_frame_finish+0/1a4>
Trace; 80503f50 <END_OF_CODE+24e820/????>
Trace; 8051910c <END_OF_CODE+2639dc/????>
Trace; 801b0fb8 <net_rx_action+a8/24c>
Trace; 801ae580 <net_tx_action+1c8/1f0>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 8009ac90 <__alloc_pages_internal+3f4/434>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 801784a0 <mem_serial_out+0/20>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 800bd470 <__kmalloc+d4/124>
Trace; 801a8664 <__alloc_skb+70/148>
Trace; 801a9120 <skb_copy+40/c8>
Trace; 8018c4b8 <ar231x_start_xmit+0/d0>
Trace; 80b42088 <END_OF_CODE+88c958/????>
Trace; 80b42070 <END_OF_CODE+88c940/????>
Trace; 80b419d4 <END_OF_CODE+88c2a4/????>
Trace; 8007425c <queue_work_on+90/a0>
Trace; 800745c8 <worker_thread+1ac/260>
Trace; 800789b4 <kthread+0/a0>
Trace; 800790a8 <autoremove_wake_function+0/4c>
Trace; 8007441c <worker_thread+0/260>
Trace; 80078a10 <kthread+5c/a0>
Trace; 8004e988 <kernel_thread_helper+10/18>

1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<8007d3e4>] 0x8007d3e4
[<800bcf78>] 0x800bcf78
[<800bd538>] 0x800bd538
[<80b41a58>] 0x80b41a58
[<80b42674>] 0x80b42674
[<80b479f8>] 0x80b479f8
[<8052129c>] 0x8052129c
[<80525a20>] 0x80525a20
[<800bd450>] 0x800bd450
[<80041444>] 0x80041444
[<805269e0>] 0x805269e0
[<80525ed8>] 0x80525ed8
[<80503f50>] 0x80503f50
[<8051910c>] 0x8051910c
[<801b0fb8>] 0x801b0fb8
[<801ae580>] 0x801ae580
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<8009ac90>] 0x8009ac90
[<80041444>] 0x80041444
[<801784a0>] 0x801784a0
[<800bd450>] 0x800bd450
[<800bd470>] 0x800bd470
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<8018c4b8>] 0x8018c4b8
[<80b42088>] 0x80b42088
[<80b42070>] 0x80b42070
[<80b419d4>] 0x80b419d4
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 8009ac88 <__alloc_pages_internal+3ec/434>
Trace; 8007d3e4 <ktime_get+18/3c>
Trace; 800bcf78 <cache_alloc_refill+2d8/6fc>
Trace; 800bd538 <kmem_cache_alloc+78/e8>
Trace; 80b41a58 <END_OF_CODE+88c328/????>
Trace; 80b42674 <END_OF_CODE+88cf44/????>
Trace; 80b479f8 <END_OF_CODE+8922c8/????>
Trace; 8052129c <END_OF_CODE+26bb6c/????>
Trace; 80525a20 <END_OF_CODE+2702f0/????>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 805269e0 <END_OF_CODE+2712b0/????>
Trace; 80525ed8 <END_OF_CODE+2707a8/????>
Trace; 80503f50 <END_OF_CODE+24e820/????>
Trace; 8051910c <END_OF_CODE+2639dc/????>
Trace; 801b0fb8 <net_rx_action+a8/24c>
Trace; 801ae580 <net_tx_action+1c8/1f0>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 8009ac90 <__alloc_pages_internal+3f4/434>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 801784a0 <mem_serial_out+0/20>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 800bd470 <__kmalloc+d4/124>
Trace; 801a8664 <__alloc_skb+70/148>
Trace; 801a9120 <skb_copy+40/c8>
Trace; 8018c4b8 <ar231x_start_xmit+0/d0>
Trace; 80b42088 <END_OF_CODE+88c958/????>
Trace; 80b42070 <END_OF_CODE+88c940/????>
Trace; 80b419d4 <END_OF_CODE+88c2a4/????>
Trace; 8007425c <queue_work_on+90/a0>
Trace; 800745c8 <worker_thread+1ac/260>
Trace; 800789b4 <kthread+0/a0>
Trace; 800790a8 <autoremove_wake_function+0/4c>
Trace; 8007441c <worker_thread+0/260>
Trace; 80078a10 <kthread+5c/a0>
Trace; 8004e988 <kernel_thread_helper+10/18>

1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<8009ac88>] 0x8009ac88
[<8018c538>] 0x8018c538
[<800bcf78>] 0x800bcf78
[<800bd450>] 0x800bd450
[<801a8664>] 0x801a8664
[<801a9120>] 0x801a9120
[<80074174>] 0x80074174
[<80b42070>] 0x80b42070
[<8007425c>] 0x8007425c
[<800745c8>] 0x800745c8
[<800789b4>] 0x800789b4
[<800790a8>] 0x800790a8
[<8007441c>] 0x8007441c
[<80078a10>] 0x80078a10
[<8004e988>] 0x8004e988
CPU 0: hi: 0, btch: 1 usd: 0
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 8009ac88 <__alloc_pages_internal+3ec/434>
Trace; 8018c538 <ar231x_start_xmit+80/d0>
Trace; 800bcf78 <cache_alloc_refill+2d8/6fc>
Trace; 800bd450 <__kmalloc+b4/124>
Trace; 801a8664 <__alloc_skb+70/148>
Trace; 801a9120 <skb_copy+40/c8>
Trace; 80074174 <__queue_work+7c/b8>
Trace; 80b42070 <END_OF_CODE+88c940/????>
Trace; 8007425c <queue_work_on+90/a0>
Trace; 800745c8 <worker_thread+1ac/260>
Trace; 800789b4 <kthread+0/a0>
Trace; 800790a8 <autoremove_wake_function+0/4c>
Trace; 8007441c <worker_thread+0/260>
Trace; 80078a10 <kthread+5c/a0>
Trace; 8004e988 <kernel_thread_helper+10/18>

1417 total pagecache pages
4096 pages RAM
1395 pages shared
2630 pages non-shared

8 warnings issued. Results may not be reliable.

BusyBox v1.14.4 (2010-01-22 23:22:23 CET) built-in shell (ash) Enter 'help' for a list of built-in commands.

_______ ________ __

 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (bleeding edge, r18405) ------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
root@OpenWrt:~# batctl o -b
  Originator     (#/255)           Nexthop [outgoingIF]:   Potential nexthops ... [B.A.T.M.A.N. adv 0.2.1-beta r1545, MainIF/MAC: ath2/0a:24:01:b7:6a:d1]
06:22:b0:98:87:f9  (  3) 0a:24:01:b7:61:1b [      ath2]: 06:22:b0:98:87:f9 (  0) 0a:24:01:b7:61:1b (  3) 06:24:01:b7:69:c1 (  1)
06:24:01:b7:69:c1  ( 11) 06:24:01:b7:69:c1 [      ath2]: 06:24:01:b7:69:c1 ( 11)
0a:24:01:b7:61:19  (  3) 0a:24:01:b7:6a:fd [      ath2]: 0a:24:01:b7:61:19 (  0) 0a:24:01:b7:6a:fd (  3)
0a:24:01:b7:61:1b  ( 19) 0a:24:01:b7:61:1b [      ath2]: 0a:24:01:b7:61:1b ( 19) 06:24:01:b7:69:c1 (  0) 06:22:b0:98:87:f9 (  0)
0a:24:01:b7:6a:fd  ( 10) 0a:24:01:b7:6a:fd [      ath2]: 0a:24:01:b7:6a:fd ( 10) 0a:24:01:b7:61:49 (  0)
0a:24:01:b7:61:49  ( 10) 0a:24:01:b7:61:49 [      ath2]: 0a:24:01:b7:61:49 ( 10) 0a:24:01:b7:61:1b (  1)
root@OpenWrt:~# dmesg
Linux version 2.6.30.9 (linus@Linus-Debian) (gcc version 4.3.3 (GCC) ) #56 Fri Jan 22 23:24:25 CET 2010
CPU revision is: 00019064 (MIPS 4KEc)
Determined physical RAM map:
 memory: 01000000 @ 00000000 (usable)
Initrd not found or empty - disabling initrd
Zone PFN ranges:
  Normal   0x00000000 -> 0x00001000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00001000
On node 0 totalpages: 4096
free_area_init_node: node 0, pgdat 8027ef40, node_mem_map 802b7000
  Normal zone: 32 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 4064 pages, LIFO batch:0
Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 4064
Kernel command line: console=ttyS0,9600 rootfstype=squashfs,jffs2
Primary instruction cache 16kB, VIPT, 4-way, linesize 16 bytes.
Primary data cache 16kB, 4-way, VIPT, no aliases, linesize 16 bytes
NR_IRQS:128
PID hash table entries: 64 (order: 6, 256 bytes)
console [ttyS0] enabled
Dentry cache hash table entries: 2048 (order: 1, 8192 bytes)
Inode-cache hash table entries: 1024 (order: 0, 4096 bytes)
Memory: 13460k/16384k available (1976k kernel code, 2924k reserved, 324k data, 128k init, 0k highmem)
Calibrating delay loop... 183.50 BogoMIPS (lpj=917504)
Mount-cache hash table entries: 512
net_namespace: 732 bytes
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
Switched to high resolution mode on CPU 0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 512 (order: 0, 4096 bytes)
TCP bind hash table entries: 512 (order: -1, 2048 bytes)
TCP: Hash tables configured (established 512 bind 512)
TCP reno registered
NET: Registered protocol family 1
Radio config found at offset 0xf8(0x1f8)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering mini_fo version $Id$
JFFS2 version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
msgmni has been set to 26
io scheduler noop registered
io scheduler deadline registered (default)
gpiodev: gpio device registered with major 254
gpiodev: gpio platform device registered with access mask FFFFFFFF
Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
serial8250: ttyS0 at MMIO 0xb1100003 (irq = 37) is a 16550A
eth0: Atheros AR231x: 00:24:01:b7:6a:d2, irq 4
ar231x_eth_mii: probed
eth0: attached PHY driver [IC+ IP175C] (mii_bus:phy_addr=0:00)
cmdlinepart partition parsing not available
Searching for RedBoot partition table in spiflash at offset 0x3d0000
Searching for RedBoot partition table in spiflash at offset 0x3e0000
6 RedBoot partitions found on MTD device spiflash
Creating 6 MTD partitions on "spiflash":
0x000000000000-0x000000030000 : "RedBoot"
0x000000030000-0x0000002f0000 : "rootfs"
mtd: partition "rootfs" set to be root filesystem
mtd: partition "rootfs_data" created automatically, ofs=250000, len=A0000
0x000000250000-0x0000002f0000 : "rootfs_data"
0x0000002f0000-0x0000003d0000 : "vmlinux.bin.l7"
0x0000003e0000-0x0000003ef000 : "FIS directory"
0x0000003ef000-0x0000003f0000 : "RedBoot config"
0x0000003f0000-0x000000400000 : "boardconfig"
TCP westwood registered
NET: Registered protocol family 17
Bridge firewalling registered
802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
All bugs added by David S. Miller <[email protected]>
VFS: Mounted root (squashfs filesystem) readonly on device 31:1.
Freeing unused kernel memory: 128k freed
Please be patient, while OpenWrt loads ...
mini_fo: using base directory: /
mini_fo: using storage directory: /tmp/root
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
device eth0.1 entered promiscuous mode
device eth0 entered promiscuous mode
br-mesh: port 1(eth0.1) entering forwarding state
ip_tables: (C) 2000-2006 Netfilter Core Team
Ebtables v2.0 registered
ip6_tables: (C) 2000-2006 Netfilter Core Team
batman-adv:B.A.T.M.A.N. advanced 0.2.1-beta r1545 (compatibility version 8) loaded
device eth0.4 entered promiscuous mode
br-wan_vpn: port 1(eth0.4) entering forwarding state
ath_hal: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint
ath_hal: 2009-05-08 (AR5212, AR5312, RF5111, RF5112, RF2316, RF2317, REGOPS_FUNC, TX_DESC_SWAP, XR)
br-wan_vpn: starting userspace STP failed, starting kernel STP
eth0.1: no IPv6 routers present
ath_ahb: trunk
wlan: trunk
wlan: mac acl policy registered
ath_rate_minstrel: Minstrel automatic rate control algorithm 1.2 (trunk)
br-mesh: no IPv6 routers present
ath_rate_minstrel: look around rate set to 10%
ath_rate_minstrel: EWMA rolloff level set to 75%
ath_rate_minstrel: max segment size in the mrr set to 6000 us
Atheros HAL provided by OpenWrt, DD-WRT and MakSat Technologies
wifi0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
wifi0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: turboG rates: 6Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: H/W encryption support: WEP AES AES_CCM TKIP
ath_ahb: wifi0: Atheros 2317 WiSoC REV1: mem=0xb0000000, irq=3
IRQ 3/wifi0: IRQF_DISABLED is not guaranteed on shared IRQs
eth0.2: no IPv6 routers present
device bat0 entered promiscuous mode
br-mesh: port 2(bat0) entering forwarding state
eth0.3: no IPv6 routers present
eth0.4: no IPv6 routers present
device ath0 entered promiscuous mode
br-mesh: port 3(ath0) entering forwarding state
device ath0 left promiscuous mode
br-mesh: port 3(ath0) entering disabled state
device ath0 entered promiscuous mode
br-mesh: port 3(ath0) entering forwarding state
bat0: no IPv6 routers present
device ath1 entered promiscuous mode
br-wan_vpn: topology change detected, sending tcn bpdu
br-wan_vpn: port 2(ath1) entering forwarding state
device ath1 left promiscuous mode
br-wan_vpn: port 2(ath1) entering disabled state
device ath1 entered promiscuous mode
br-wan_vpn: topology change detected, sending tcn bpdu
br-wan_vpn: port 2(ath1) entering forwarding state
ath0: no IPv6 routers present
ath1: no IPv6 routers present
br-wan_vpn: port 2(ath1) entering disabled state
br-wan_vpn: port 1(eth0.4) entering disabled state
br-wan_vpn: topology change detected, propagating
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 2(ath1) entering forwarding state
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-mesh: port 3(ath0) entering disabled state
br-mesh: port 2(bat0) entering disabled state
br-mesh: port 1(eth0.1) entering disabled state
br-mesh: port 3(ath0) entering forwarding state
br-mesh: port 2(bat0) entering forwarding state
br-mesh: port 1(eth0.1) entering forwarding state
ath2: no IPv6 routers present
br-mesh: no IPv6 routers present
------------[ cut here ]------------
WARNING: at net/sched/sch_generic.c:226 0x801c3554()
NETDEV WATCHDOG: wifi0 (): transmit timed out
Modules linked in: ath_ahb ath_hal(P) batman_adv ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables ebt_redirect ebt_mark ebt_vlan ebt_stp ebt_pkttype ebt_mark_m ebt_limit ebt_among ebt_802_3 ebtable_nat ebtable_filter ebtable_broute ebtables xt_quota xt_pkttype xt_physdev ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables tun ipv6
Call Trace:[<800493bc>] 0x800493bc
[<800493bc>] 0x800493bc
[<80060914>] 0x80060914
[<801c3554>] 0x801c3554
[<80060994>] 0x80060994
[<800790bc>] 0x800790bc
[<801c3554>] 0x801c3554
[<8005aaf8>] 0x8005aaf8
[<800741b0>] 0x800741b0
[<80222220>] 0x80222220
[<8007d674>] 0x8007d674
[<80222138>] 0x80222138
[<801c33b8>] 0x801c33b8
[<8006c01c>] 0x8006c01c
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<80041444>] 0x80041444
---[ end trace 31db24b68cfbba59 ]---

batman-adv:Adding interface: ath2
batman-adv:Interface activated: ath2
batman-adv:Adding interface: eth0.2
batman-adv:Interface activated: eth0.2
batman-adv:Adding interface: eth0.3
batman-adv:The newly added mac address (00:24:01:b7:6a:d2) already exists on: eth0.2 batman-adv:It is strongly recommended to keep mac addresses unique to avoid problems! batman-adv:Interface activated: eth0.3
jffs2_scan_eraseblock(): End of filesystem marker found at 0x0 jffs2_build_filesystem(): unlocking the mtd device... done. jffs2_build_filesystem(): erasing all blocks after the end marker... done.
mini_fo: using base directory: /
mini_fo: using storage directory: /jffs
device VPN entered promiscuous mode
br-wan_vpn: topology change detected, sending tcn bpdu
br-wan_vpn: port 3(VPN) entering forwarding state
br-wan_vpn: neighbor 8000.00:22:b0:98:8c:34 lost on port 1(eth0.4)
br-wan_vpn: topology change detected, propagating
root@OpenWrt:~#

ksymoops 2.4.11 on x86_64 2.6.31-1-amd64. Options used

     -v /home/linus/lff-test/kamikaze/build_dir/linux-atheros/linux-2.6.30.9/vmlinux (specified)
     -K (specified)
     -L (specified)
     -o /home/linus/lff-test/kamikaze/build_dir/linux-atheros -o /home/linus/lff-test/kamikaze/build_dir/target-mips_uClibc-0.9.30.1 (specified)
     -m /home/linus/lff-test/kamikaze/build_dir/linux-atheros/linux-2.6.30.9/System.map (specified)

No modules in ksyms, skipping objects
lo: Disabled Privacy Extensions

WARNING: at net/sched/sch_generic.c:226 0x801c3554()
Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<80060914>] 0x80060914
[<801c3554>] 0x801c3554
[<80060994>] 0x80060994
[<800790bc>] 0x800790bc
[<801c3554>] 0x801c3554
[<8005aaf8>] 0x8005aaf8
[<800741b0>] 0x800741b0
[<80222220>] 0x80222220
[<8007d674>] 0x8007d674
[<80222138>] 0x80222138
[<801c33b8>] 0x801c33b8
[<8006c01c>] 0x8006c01c
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<80041444>] 0x80041444
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 80060914 <warn_slowpath_common+70/b0>
Trace; 801c3554 <dev_watchdog+19c/338>
Trace; 80060994 <warn_slowpath_fmt+24/30>
Trace; 800790bc <autoremove_wake_function+14/4c>
Trace; 801c3554 <dev_watchdog+19c/338>
Trace; 8005aaf8 <__wake_up_common+58/c0>
Trace; 800741b0 <delayed_work_timer_fn+0/1c>
Trace; 80222220 <br_fdb_cleanup+e8/124>
Trace; 8007d674 <hrtimer_interrupt+26c/2a4>
Trace; 80222138 <br_fdb_cleanup+0/124>
Trace; 801c33b8 <dev_watchdog+0/338>
Trace; 8006c01c <run_timer_softirq+17c/234>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 80041444 <ret_from_irq+0/4>

1 warning issued. Results may not be reliable.

root@OpenWrt:~# batctl o -b
  Originator     (#/255)           Nexthop [outgoingIF]:   Potential nexthops ... [B.A.T.M.A.N. adv 0.2.1-beta r1545, MainIF/MAC: ath2/0a:24:01:b7:61:1b]
06:22:b0:98:87:f9  ( 23) 06:22:b0:98:87:f9 [      ath2]: 06:22:b0:98:87:f9 ( 23)
06:24:01:b7:69:c1  ( 21) 06:24:01:b7:69:c1 [      ath2]: 06:24:01:b7:69:c1 ( 21) 0a:24:01:b7:6a:d1 (  1)
0a:24:01:b7:61:19  (  4) 0a:24:01:b7:61:49 [      ath2]: 0a:24:01:b7:61:19 (  0) 0a:24:01:b7:6a:d1 (  1) 0a:24:01:b7:61:49 (  4)
0a:24:01:b7:6a:d1  ( 35) 0a:24:01:b7:6a:d1 [      ath2]: 0a:24:01:b7:6a:d1 ( 35) 06:24:01:b7:69:c1 (  0)
0a:24:01:b7:6a:fd  ( 10) 0a:24:01:b7:6a:fd [      ath2]: 0a:24:01:b7:6a:fd ( 10) 0a:24:01:b7:6a:d1 (  0) 0a:24:01:b7:61:49 (  0)
0a:24:01:b7:61:49  ( 10) 0a:24:01:b7:61:49 [      ath2]: 0a:24:01:b7:61:49 ( 10) 0a:24:01:b7:6a:fd (  6)
root@OpenWrt:~# dmesg
Linux version 2.6.30.9 (linus@Linus-Debian) (gcc version 4.3.3 (GCC) ) #49 Fri Jan 22 18:48:49 CET 2010
CPU revision is: 00019064 (MIPS 4KEc)
Determined physical RAM map:
 memory: 01000000 @ 00000000 (usable)
Initrd not found or empty - disabling initrd
Zone PFN ranges:
  Normal   0x00000000 -> 0x00001000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00001000
On node 0 totalpages: 4096
free_area_init_node: node 0, pgdat 8027ef40, node_mem_map 802b7000
  Normal zone: 32 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 4064 pages, LIFO batch:0
Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 4064
Kernel command line: console=ttyS0,9600 rootfstype=squashfs,jffs2
Primary instruction cache 16kB, VIPT, 4-way, linesize 16 bytes.
Primary data cache 16kB, 4-way, VIPT, no aliases, linesize 16 bytes
NR_IRQS:128
PID hash table entries: 64 (order: 6, 256 bytes)
console [ttyS0] enabled
Dentry cache hash table entries: 2048 (order: 1, 8192 bytes)
Inode-cache hash table entries: 1024 (order: 0, 4096 bytes)
Memory: 13460k/16384k available (1976k kernel code, 2924k reserved, 324k data, 128k init, 0k highmem)
Calibrating delay loop... 183.50 BogoMIPS (lpj=917504)
Mount-cache hash table entries: 512
net_namespace: 732 bytes
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
Switched to high resolution mode on CPU 0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 512 (order: 0, 4096 bytes)
TCP bind hash table entries: 512 (order: -1, 2048 bytes)
TCP: Hash tables configured (established 512 bind 512)
TCP reno registered
NET: Registered protocol family 1
Radio config found at offset 0xf8(0x1f8)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering mini_fo version $Id$
JFFS2 version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
msgmni has been set to 26
io scheduler noop registered
io scheduler deadline registered (default)
gpiodev: gpio device registered with major 254
gpiodev: gpio platform device registered with access mask FFFFFFFF
Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
serial8250: ttyS0 at MMIO 0xb1100003 (irq = 37) is a 16550A
eth0: Atheros AR231x: 00:24:01:b7:61:1c, irq 4
ar231x_eth_mii: probed
eth0: attached PHY driver [IC+ IP175C] (mii_bus:phy_addr=0:00)
cmdlinepart partition parsing not available
Searching for RedBoot partition table in spiflash at offset 0x3d0000
Searching for RedBoot partition table in spiflash at offset 0x3e0000
6 RedBoot partitions found on MTD device spiflash
Creating 6 MTD partitions on "spiflash":
0x000000000000-0x000000030000 : "RedBoot"
0x000000030000-0x0000002f0000 : "rootfs"
mtd: partition "rootfs" set to be root filesystem
mtd: partition "rootfs_data" created automatically, ofs=250000, len=A0000
0x000000250000-0x0000002f0000 : "rootfs_data"
0x0000002f0000-0x0000003d0000 : "vmlinux.bin.l7"
0x0000003e0000-0x0000003ef000 : "FIS directory"
0x0000003ef000-0x0000003f0000 : "RedBoot config"
0x0000003f0000-0x000000400000 : "boardconfig"
TCP westwood registered
NET: Registered protocol family 17
Bridge firewalling registered
802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
All bugs added by David S. Miller <[email protected]>
VFS: Mounted root (squashfs filesystem) readonly on device 31:1.
Freeing unused kernel memory: 128k freed
Please be patient, while OpenWrt loads ...
mini_fo: using base directory: /
mini_fo: using storage directory: /jffs
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
device eth0.1 entered promiscuous mode
device eth0 entered promiscuous mode
br-mesh: port 1(eth0.1) entering forwarding state
ip_tables: (C) 2000-2006 Netfilter Core Team
Ebtables v2.0 registered
ip6_tables: (C) 2000-2006 Netfilter Core Team
batman-adv:B.A.T.M.A.N. advanced 0.2.1-beta r1545 (compatibility version 8) loaded
device eth0.4 entered promiscuous mode
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-wan_vpn: starting userspace STP failed, starting kernel STP
ath_hal: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint
ath_hal: 2009-05-08 (AR5212, AR5312, RF5111, RF5112, RF2316, RF2317, REGOPS_FUNC, TX_DESC_SWAP, XR)
eth0.1: no IPv6 routers present
br-mesh: no IPv6 routers present
ath_ahb: trunk
wlan: trunk
wlan: mac acl policy registered
ath_rate_minstrel: Minstrel automatic rate control algorithm 1.2 (trunk)
ath_rate_minstrel: look around rate set to 10%
ath_rate_minstrel: EWMA rolloff level set to 75%
ath_rate_minstrel: max segment size in the mrr set to 6000 us
Atheros HAL provided by OpenWrt, DD-WRT and MakSat Technologies
wifi0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
wifi0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: turboG rates: 6Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: H/W encryption support: WEP AES AES_CCM TKIP
ath_ahb: wifi0: Atheros 2317 WiSoC REV1: mem=0xb0000000, irq=3
IRQ 3/wifi0: IRQF_DISABLED is not guaranteed on shared IRQs
eth0.2: no IPv6 routers present
device bat0 entered promiscuous mode
br-mesh: port 2(bat0) entering forwarding state
eth0.3: no IPv6 routers present
br-wan_vpn: no IPv6 routers present
eth0.4: no IPv6 routers present
device ath0 entered promiscuous mode
br-mesh: port 3(ath0) entering forwarding state
bat0: no IPv6 routers present
device ath0 left promiscuous mode
br-mesh: port 3(ath0) entering disabled state
device ath0 entered promiscuous mode
br-mesh: port 3(ath0) entering forwarding state
device ath1 entered promiscuous mode
br-wan_vpn: topology change detected, sending tcn bpdu
br-wan_vpn: port 2(ath1) entering forwarding state
device ath1 left promiscuous mode
br-wan_vpn: port 2(ath1) entering disabled state
device ath1 entered promiscuous mode
br-wan_vpn: topology change detected, sending tcn bpdu
br-wan_vpn: port 2(ath1) entering forwarding state
ath0: no IPv6 routers present
br-wan_vpn: port 2(ath1) entering disabled state
br-wan_vpn: port 1(eth0.4) entering disabled state
br-wan_vpn: topology change detected, propagating
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 2(ath1) entering forwarding state
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-mesh: port 3(ath0) entering disabled state
br-mesh: port 2(bat0) entering disabled state
br-mesh: port 1(eth0.1) entering disabled state
br-mesh: port 3(ath0) entering forwarding state
br-mesh: port 2(bat0) entering forwarding state
br-mesh: port 1(eth0.1) entering forwarding state
ath1: no IPv6 routers present
ath2: no IPv6 routers present
br-mesh: no IPv6 routers present
batman-adv:Adding interface: ath2
batman-adv:Interface activated: ath2
batman-adv:Adding interface: eth0.2
batman-adv:Interface activated: eth0.2
batman-adv:Adding interface: eth0.3
batman-adv:The newly added mac address (00:24:01:b7:61:1c) already exists on: eth0.2
batman-adv:It is strongly recommended to keep mac addresses unique to avoid problems!
batman-adv:Interface activated: eth0.3
br-wan_vpn: port 2(ath1) entering disabled state
device VPN entered promiscuous mode
br-wan_vpn: topology change detected, sending tcn bpdu
br-wan_vpn: port 3(VPN) entering forwarding state
br-wan_vpn: port 3(VPN) entering disabled state
br-wan_vpn: topology change detected, sending tcn bpdu
br-wan_vpn: port 3(VPN) entering forwarding state
br-wan_vpn: neighbor 8000.00:22:b0:98:8c:34 lost on port 1(eth0.4)
br-wan_vpn: topology change detected, propagating
------------[ cut here ]------------
WARNING: at net/sched/sch_generic.c:226 0x801c3554()
NETDEV WATCHDOG: wifi0 (): transmit timed out
Modules linked in: ath_ahb ath_hal(P) batman_adv ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables ebt_redirect ebt_mark ebt_vlan ebt_stp ebt_pkttype ebt_mark_m ebt_limit ebt_among ebt_802_3 ebtable_nat ebtable_filter ebtable_broute ebtables xt_quota xt_pkttype xt_physdev ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables tun ipv6
Call Trace:[<800493bc>] 0x800493bc
[<800493bc>] 0x800493bc
[<80060914>] 0x80060914
[<801c3554>] 0x801c3554
[<80060994>] 0x80060994
[<801c3554>] 0x801c3554
[<80086bf4>] 0x80086bf4
[<8007d674>] 0x8007d674
[<801b0838>] 0x801b0838
[<801c33b8>] 0x801c33b8
[<8006c01c>] 0x8006c01c
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<80041444>] 0x80041444
[<80041660>] 0x80041660
[<8018c4b8>] 0x8018c4b8
[<8004e85c>] 0x8004e85c
[<80041680>] 0x80041680
[<80281a4c>] 0x80281a4c
[<80281370>] 0x80281370
---[ end trace bf8b652a122b3db4 ]---

root@OpenWrt:~#

ksymoops 2.4.11 on x86_64 2.6.31-1-amd64. Options used

     -v /home/linus/lff-test/kamikaze/build_dir/linux-atheros/linux-2.6.30.9/vmlinux (specified)
     -K (specified)
     -L (specified)
     -o /home/linus/lff-test/kamikaze/build_dir/linux-atheros -o /home/linus/lff-test/kamikaze/build_dir/target-mips_uClibc-0.9.30.1 (specified)
     -m /home/linus/lff-test/kamikaze/build_dir/linux-atheros/linux-2.6.30.9/System.map (specified)

No modules in ksyms, skipping objects
lo: Disabled Privacy Extensions

WARNING: at net/sched/sch_generic.c:226 0x801c3554()
Call Trace:[<800493bc>] 0x800493bc

[<800493bc>] 0x800493bc
[<80060914>] 0x80060914
[<801c3554>] 0x801c3554
[<80060994>] 0x80060994
[<801c3554>] 0x801c3554
[<80086bf4>] 0x80086bf4
[<8007d674>] 0x8007d674
[<801b0838>] 0x801b0838
[<801c33b8>] 0x801c33b8
[<8006c01c>] 0x8006c01c
[<80066d84>] 0x80066d84
[<80066e90>] 0x80066e90
[<80041444>] 0x80041444
[<80041660>] 0x80041660
[<8018c4b8>] 0x8018c4b8
[<8004e85c>] 0x8004e85c
[<80041680>] 0x80041680
[<80281a4c>] 0x80281a4c
[<80281370>] 0x80281370
Warning (Oops_read): Code line not seen, dumping what data is available

Trace; 800493bc <dump_stack+20/50>
Trace; 800493bc <dump_stack+20/50>
Trace; 80060914 <warn_slowpath_common+70/b0>
Trace; 801c3554 <dev_watchdog+19c/338>
Trace; 80060994 <warn_slowpath_fmt+24/30>
Trace; 801c3554 <dev_watchdog+19c/338>
Trace; 80086bf4 <tick_program_event+28/34>
Trace; 8007d674 <hrtimer_interrupt+26c/2a4>
Trace; 801b0838 <process_backlog+d4/14c>
Trace; 801c33b8 <dev_watchdog+0/338>
Trace; 8006c01c <run_timer_softirq+17c/234>
Trace; 80066d84 <__do_softirq+a4/154>
Trace; 80066e90 <do_softirq+5c/98>
Trace; 80041444 <ret_from_irq+0/4>
Trace; 80041660 <r4k_wait+0/40>
Trace; 8018c4b8 <ar231x_start_xmit+0/d0>
Trace; 8004e85c <cpu_idle+24/4c>
Trace; 80041680 <r4k_wait+20/40>
Trace; 80281a4c <start_kernel+36c/384>
Trace; 80281370 <unknown_bootoption+0/2fc>

1 warning issued. Results may not be reliable.

#1 lore raw
Quoted 8 lines · Linus L??ssing

Hey guys,

I've been installing a 9 node setup here in our cellar. They are all running B.A.T.M.A.N. adv 0.2.1-beta r1545 (so the current batman-adv maintance version in OpenWRT).

The result over night:
- 1x: bat_events: page allocation failure

This looks like a memory leak somewhere. It could be anywhere, batman or some other part of the kernel.

Can you build your kernel with the kernel memory leak detector enabled? It is under the Kernel Hacking options. You probably also need to build the kernel with symbols, which will help with ksymopps anyway. Documentation for kmemleak is in Documentation/kmemleak.txt. I've never used it myself, so i've no idea how good it actually is...

Interestingly, did you notice the warnings:

batman-adv:The newly added mac address (00:24:01:b7:6a:d2) already exists on: eth0.2 batman-adv:It is strongly recommended to keep mac addresses unique to avoid +problems!

I guess this is because you are using VLANs. Have you seen these problems without using VLANs?

Andrew

#2 lore raw

Ok, this is with symbols enabled and I changed the mac addresses, so they are defintely different now. Here is the new backtrace for the second issue. It occured after about 1-2 hours.

Cheers, Linus

Quoted 21 lines · Andrew Lunn
Quoted 8 lines · Linus L??ssing

Hey guys,

I've been installing a 9 node setup here in our cellar. They are all running B.A.T.M.A.N. adv 0.2.1-beta r1545 (so the current batman-adv maintance version in OpenWRT).

The result over night:
- 1x: bat_events: page allocation failure

This looks like a memory leak somewhere. It could be anywhere, batman or some other part of the kernel.

Can you build your kernel with the kernel memory leak detector enabled? It is under the Kernel Hacking options. You probably also need to build the kernel with symbols, which will help with ksymopps anyway. Documentation for kmemleak is in Documentation/kmemleak.txt. I've never used it myself, so i've no idea how good it actually is...

Interestingly, did you notice the warnings:

Quoted 9 lines

batman-adv:The newly added mac address (00:24:01:b7:6a:d2) already exists on: eth0.2 batman-adv:It is strongly recommended to keep mac addresses unique to avoid +problems!

I guess this is because you are using VLANs. Have you seen these problems without using VLANs?

Andrew

root@OpenWrt:/# ------------[ cut here ]------------
WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x19c/0x338()
NETDEV WATCHDOG: wifi0 (): transmit timed out
Modules linked in: ath_ahb ath_hal(P) batman_adv ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables ebt_redirect ebt_mark ebt_vlan ebt_stp ebt_pkttype ebt_mark_m ebt_limit ebt_among ebt_802_3 ebtable_nat ebtable_filter ebtable_broute ebtables xt_quota xt_pkttype xt_physdev ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables tun ipv6
Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<80062a84>] warn_slowpath_common+0x70/0xb0
[<80062b04>] warn_slowpath_fmt+0x24/0x30
[<801c9c98>] dev_watchdog+0x19c/0x338
[<8006e1b8>] run_timer_softirq+0x17c/0x234
[<80068f20>] __do_softirq+0xa4/0x154
[<8006902c>] do_softirq+0x5c/0x98
[<80041444>] ret_from_irq+0x0/0x4
[<80041680>] r4k_wait+0x20/0x40
[<8004e910>] cpu_idle+0x24/0x4c
[<802a9a4c>] start_kernel+0x36c/0x384
---[ end trace 57451ac66d72b45d ]--
#3 lore raw
In reply to Andrew Lunn

Hi Andrew,

Here is the output with the symbols inserted. I can reproduce this memory leak very reliable now, by just activating the vis server (didn't do that with the logs I've posted before) in a couple of minutes. free is showing a nice countdown then :).

For kmemleak... I'm using 2.6.30.10 here and I think Torvalds merged it into 2.6.33 a month ago. Anyone having a 2.6.33 kernel running at the moment, patches for kmemleak in 2.6.30 or another idea of how to debug this in 2.6.30?

"Debug slab memory allocations" (CONFIG_DEBUG_SLAB) and "Memory leak debugging" (DEBUG_SLAB_LEAK) are activated on the kernel I'm using here, but they are not of any use, are they?

Cheers, Linus

Quoted 21 lines · Andrew Lunn
Quoted 8 lines · Linus L??ssing

Hey guys,

I've been installing a 9 node setup here in our cellar. They are all running B.A.T.M.A.N. adv 0.2.1-beta r1545 (so the current batman-adv maintance version in OpenWRT).

The result over night:
- 1x: bat_events: page allocation failure

This looks like a memory leak somewhere. It could be anywhere, batman or some other part of the kernel.

Can you build your kernel with the kernel memory leak detector enabled? It is under the Kernel Hacking options. You probably also need to build the kernel with symbols, which will help with ksymopps anyway. Documentation for kmemleak is in Documentation/kmemleak.txt. I've never used it myself, so i've no idea how good it actually is...

Interestingly, did you notice the warnings:

Quoted 9 lines

batman-adv:The newly added mac address (00:24:01:b7:6a:d2) already exists on: eth0.2 batman-adv:It is strongly recommended to keep mac addresses unique to avoid +problems!

I guess this is because you are using VLANs. Have you seen these problems without using VLANs?

Andrew

+Ethernet eth0: MAC address 00:22:b0:98:87:de
IP: 192.168.1.1/255.255.255.0, Gateway: 0.0.0.0
Default server: 192.168.1.2

RedBoot(tm) bootstrap and debug environment [ROMRAM] production release, version "2.1.3" - built 18:43:19, Sep 20 2007

Platform: ap61 (Atheros WiSOC)
Copyright (C) 2000, 2001, 2002, 2003, 2004 Red Hat, Inc. Copyright (C) 2007, NewMedia-NET GmbH.

Board: DLINK DIR-300
RAM: 0x80000000-0x81000000, [0x80040580-0x80fe1000] available
FLASH: 0xbfc00000 - 0xbfff0000, 64 blocks of 0x00010000 bytes each.
== Executing boot script in 5.000 seconds - enter ^C to abort
DD-WRT> fis load -l vmlinux.bin.l7
Image loaded from 0x80041000-0x802c8200
DD-WRT> exec
Now booting linux kernel:
Base address 0x80030000 Entry 0x80041000
Cmdline :
Linux version 2.6.30.10 (linus@Linus-Debian) (gcc version 4.3.3 (GCC) ) #2 Sun Jan 24 04:12:11 CET 2010 CPU revision is: 00019064 (MIPS 4KEc)
Determined physical RAM map:
memory: 01000000 @ 00000000 (usable)
Initrd not found or empty - disabling initrd
Zone PFN ranges:
Normal 0x00000000 -> 0x00001000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges

    0: 0x00000000 -> 0x00001000

Built 1 zonelists in Zone order, mobility grouping off. Total pages: 4064
Kernel command line: console=ttyS0,9600 rootfstype=squashfs,jffs2
Primary instruction cache 16kB, VIPT, 4-way, linesize 16 bytes.
Primary data cache 16kB, 4-way, VIPT, no aliases, linesize 16 bytes
NR_IRQS:128
PID hash table entries: 64 (order: 6, 256 bytes)
console [ttyS0] enabled
Dentry cache hash table entries: 2048 (order: 1, 8192 bytes)
Inode-cache hash table entries: 1024 (order: 0, 4096 bytes)
Memory: 13300k/16384k available (2002k kernel code, 3084k reserved, 458k data, 128k init, 0k highmem) Calibrating delay loop... 183.50 BogoMIPS (lpj=917504)
Mount-cache hash table entries: 512
net_namespace: 732 bytes
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 512 (order: 0, 4096 bytes)
TCP bind hash table entries: 512 (order: -1, 2048 bytes)
TCP: Hash tables configured (established 512 bind 512)
TCP reno registered
NET: Registered protocol family 1
Radio config found at offset 0xf8(0x1f8)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering mini_fo version $Id$
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
msgmni has been set to 25
io scheduler noop registered
io scheduler deadline registered (default)
gpiodev: gpio device registered with major 254
gpiodev: gpio platform device registered with access mask FFFFFFFF
Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
serial8250: ttyS0 at MMIO 0xb1100003 (irq = 37) is a 16550A
eth0: Atheros AR231x: 00:22:b0:98:87:de, irq 4
ar231x_eth_mii: probed
eth0: attached PHY driver [IC+ IP175C] (mii_bus:phy_addr=0:00)
cmdlinepart partition parsing not available
Searching for RedBoot partition table in spiflash at offset 0x3d0000
Searching for RedBoot partition table in spiflash at offset 0x3e0000
6 RedBoot partitions found on MTD device spiflash
Creating 6 MTD partitions on "spiflash":
0x000000000000-0x000000030000 : "RedBoot"
0x000000030000-0x0000002f0000 : "rootfs"
mtd: partition "rootfs" set to be root filesystem
mtd: partition "rootfs_data" created automatically, ofs=230000, len=C0000 0x000000230000-0x0000002f0000 : "rootfs_data"
0x0000002f0000-0x0000003d0000 : "vmlinux.bin.l7"
0x0000003e0000-0x0000003ef000 : "FIS directory"
0x0000003ef000-0x0000003f0000 : "RedBoot config"
0x0000003f0000-0x000000400000 : "boardconfig"
TCP westwood registered
NET: Registered protocol family 17
Bridge firewalling registered
802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
All bugs added by David S. Miller <[email protected]>
VFS: Mounted root (squashfs filesystem) readonly on device 31:1.
Freeing unused kernel memory: 128k freed
Please be patient, while OpenWrt loads ...
- preinit -
Press CTRL-C for failsafe
switching to jffs2
mini_fo: using base directory: /
mini_fo: using storage directory: /jffs
- init -

Please press Enter to activate this console. NET: Registered protocol family 10
lo: Disabled Privacy Extensions
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
device eth0.1 entered promiscuous mode
device eth0 entered promiscuous mode
br-mesh: port 1(eth0.1) entering forwarding state
ip_tables: (C) 2000-2006 Netfilter Core Team
Ebtables v2.0 registered
ip6_tables: (C) 2000-2006 Netfilter Core Team
device eth0.4 entered promiscuous mode
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-wan_vpn: starting userspace STP failed, starting kernel STP batman-adv:B.A.T.M.A.N. advanced 0.2.1-beta r1545 (compatibility version 8) loaded
ath_hal: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint
ath_hal: 2009-05-08 (AR5212, AR5312, RF5111, RF5112, RF2316, RF2317, REGOPS_FUNC, TX_DESC_SWAP, XR)
ath_ahb: trunk
wlan: trunk
wlan: mac acl policy registered
ath_rate_minstrel: Minstrel automatic rate control algorithm 1.2 (trunk)
ath_rate_minstrel: look around rate set to 10%
ath_rate_minstrel: EWMA rolloff level set to 75%
ath_rate_minstrel: max segment size in the mrr set to 6000 us
Atheros HAL provided by OpenWrt, DD-WRT and MakSat Technologies
wifi0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
wifi0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: turboG rates: 6Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: H/W encryption support: WEP AES AES_CCM TKIP
ath_ahb: wifi0: Atheros 2317 WiSoC REV1: mem=0xb0000000, irq=3
IRQ 3/wifi0: IRQF_DISABLED is not guaranteed on shared IRQs
device bat0 entered promiscuous mode
br-mesh: port 2(bat0) entering forwarding state
br-wan_vpn: port 1(eth0.4) entering disabled state
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-mesh: port 2(bat0) entering disabled state
br-mesh: port 1(eth0.1) entering disabled state
br-mesh: port 2(bat0) entering forwarding state
br-mesh: port 1(eth0.1) entering forwarding state
batman-adv:Adding interface: ath0
batman-adv:Interface activated: ath0
batman-adv:Adding interface: eth0.2
batman-adv:Interface activated: eth0.2
batman-adv:Adding interface: eth0.3
batman-adv:Interface activated: eth0.3
device VPN entered promiscuous mode
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 2(VPN) entering forwarding state

BusyBox v1.15.3 (2010-01-24 04:59:20 CET) built-in shell (ash) Enter 'help' for a list of built-in commands.

_______ ________ __

 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 KAMIKAZE (bleeding edge, r19304) ------------------
  * 10 oz Vodka       Shake well with ice and strain
  * 10 oz Triple sec  mixture into 10 shot glasses.
  * 10 oz lime juice  Salute!
 ---------------------------------------------------
root@OpenWrt:/# batctl
Usage: batctl [options] commands
commands:
        interface|if  [none|interface]          display or modify the interface settings
        originators|o                           display the originator table
        interval|it   [orig_interval]           display or modify the originator interval in ms
        loglevel|ll   [level]                   display or modify the log level
        log|l                                   read the log produced by the kernel module
        translocal|tl                           display the local translation table
        transglobal|tg                          display the global translation table
        vis_server|vs [enable|disable]          display or modify the status of the VIS server
        vis_data|vd [dot|JSON]                  display the VIS data in dot or JSON format
        aggregation|ag   [0|1]                  display or modify the packet aggregation setting
        ping|p        <destination>             ping another batman adv host via layer 2
        traceroute|tr <destination>             traceroute another batman adv host via layer 2
        tcpdump|td    <interface>               tcpdump layer 2 traffic on the given interface
        bisect        <file1> .. <fileN>        analyze given log files for routing stability
options:
        -h print this help (or 'batctl <command> -h' for the command specific help)
        -v print version
root@OpenWrt:/# batctl vs enabled
batman-adv:Setting VIS mode to server (enabling vis server)
root@OpenWrt:/# watchdog invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<8009b644>] oom_kill_process+0x6c/0x1fc
[<8009bd08>] __out_of_memory+0x184/0x1bc
[<8009bdb0>] out_of_memory+0x70/0xa0
[<8009f03c>] __alloc_pages_internal+0x338/0x434
[<800a1e64>] __do_page_cache_readahead+0xe0/0x278
[<800a206c>] do_page_cache_readahead+0x70/0x8c
[<80098880>] filemap_fault+0x19c/0x464
[<800ae778>] __do_fault+0x74/0x4e8
[<800b0e18>] handle_mm_fault+0x314/0x6fc
[<80055d3c>] do_page_fault+0x110/0x30c
[<80041420>] ret_from_exception+0x0/0x24

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:160 active_file:0 inactive_anon:164
inactive_file:10 dirty:0 writeback:0 unstable:0
free:126 slab:2222 mapped:0 pagetables:49 bounce:0
Normal free:500kB min:508kB low:632kB high:760kB active_anon:640kB inactive_anon:656kB active_file:0kB inactive_file:40kB present:16256kB pages_scanned:1533 all_unreclaimable? no lowmem_reserve[]: 0 0
Normal: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 500kB
27 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
739 pages reserved
10 pages shared
3105 pages non-shared
Out of memory: kill process 967 (dnsmasq) score 225 or a child
Killed process 967 (dnsmasq)
swapper: page allocation failure. order:0, mode:0x20

Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<8009f0f0>] __alloc_pages_internal+0x3ec/0x434
[<800c2990>] cache_alloc_refill+0x39c/0x950
[<800c30c8>] __kmalloc_track_caller+0x184/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<8069c2c8>] ieee80211_dev_alloc_skb+0x18/0x60 [ath_ahb]
[<80683f98>] ath_sysctl_unregister+0x2680/0x6ecc [ath_ahb]

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:160 active_file:3 inactive_anon:164
inactive_file:3 dirty:0 writeback:0 unstable:0
free:48 slab:2304 mapped:0 pagetables:49 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:640kB inactive_anon:656kB active_file:12kB inactive_file:12kB present:16256kB pages_scanned:2290 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
23 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
739 pages reserved
6 pages shared
3185 pages non-shared
swapper: page allocation failure. order:0, mode:0x20

Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<8009f0f0>] __alloc_pages_internal+0x3ec/0x434
[<800c2990>] cache_alloc_refill+0x39c/0x950
[<800c30c8>] __kmalloc_track_caller+0x184/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<8069c2c8>] ieee80211_dev_alloc_skb+0x18/0x60 [ath_ahb]
[<80683f98>] ath_sysctl_unregister+0x2680/0x6ecc [ath_ahb]

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:160 active_file:3 inactive_anon:164
inactive_file:3 dirty:0 writeback:0 unstable:0
free:48 slab:2304 mapped:0 pagetables:49 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:640kB inactive_anon:656kB active_file:12kB inactive_file:12kB present:16256kB pages_scanned:2290 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
23 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
739 pages reserved
6 pages shared
3185 pages non-shared
wifi0: ath_reset: Unable to start receive logic.
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<8009f0f0>] __alloc_pages_internal+0x3ec/0x434
[<800c2990>] cache_alloc_refill+0x39c/0x950
[<800c30c8>] __kmalloc_track_caller+0x184/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<80bc2144>] send_raw_packet+0x3c/0xc0 [batman_adv]
[<80bc23bc>] send_outstanding_bat_packet+0x1f4/0x250 [batman_adv]
[<800767b8>] worker_thread+0x1b0/0x278
[<8007ac14>] kthread+0x5c/0xa0
[<8004ec88>] kernel_thread_helper+0x10/0x18

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:160 active_file:3 inactive_anon:164
inactive_file:3 dirty:0 writeback:0 unstable:0
free:48 slab:2304 mapped:0 pagetables:49 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:640kB inactive_anon:656kB active_file:12kB inactive_file:12kB present:16256kB pages_scanned:2290 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
23 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
739 pages reserved
6 pages shared
3185 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<8009f0f0>] __alloc_pages_internal+0x3ec/0x434
[<800c2990>] cache_alloc_refill+0x39c/0x950
[<800c30c8>] __kmalloc_track_caller+0x184/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<8069c2c8>] ieee80211_dev_alloc_skb+0x18/0x60 [ath_ahb]
[<80683f98>] ath_sysctl_unregister+0x2680/0x6ecc [ath_ahb]

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:160 active_file:3 inactive_anon:164
inactive_file:3 dirty:0 writeback:0 unstable:0
free:48 slab:2304 mapped:0 pagetables:49 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:640kB inactive_anon:656kB active_file:12kB inactive_file:12kB present:16256kB pages_scanned:2290 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
23 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
739 pages reserved
6 pages shared
3185 pages non-shared
wifi0: ath_reset: Unable to start receive logic.
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<8009f0f0>] __alloc_pages_internal+0x3ec/0x434
[<800c2990>] cache_alloc_refill+0x39c/0x950
[<800c30c8>] __kmalloc_track_caller+0x184/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<8022bb7c>] br_send_bpdu+0x2c/0x150
[<8022beb8>] br_send_config_bpdu+0x1d0/0x1ec
[<8022b398>] br_transmit_config+0x148/0x180
[<8022b4b4>] br_config_bpdu_generation+0x60/0x8c
[<8022cc58>] br_hello_timer_expired+0x28/0x64
[<8006e204>] run_timer_softirq+0x17c/0x234
[<80068f6c>] __do_softirq+0xa4/0x154
[<80069078>] do_softirq+0x5c/0x98
[<80041444>] ret_from_irq+0x0/0x4
[<800c30e8>] __kmalloc_track_caller+0x1a4/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<80bc2144>] send_raw_packet+0x3c/0xc0 [batman_adv]
[<80bc23bc>] send_outstanding_bat_packet+0x1f4/0x250 [batman_adv]
[<800767b8>] worker_thread+0x1b0/0x278
[<8007ac14>] kthread+0x5c/0xa0
[<8004ec88>] kernel_thread_helper+0x10/0x18

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:160 active_file:3 inactive_anon:164
inactive_file:3 dirty:0 writeback:0 unstable:0
free:48 slab:2304 mapped:0 pagetables:49 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:640kB inactive_anon:656kB active_file:12kB inactive_file:12kB present:16256kB pages_scanned:2290 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
23 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
739 pages reserved
6 pages shared
3185 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<8009f0f0>] __alloc_pages_internal+0x3ec/0x434
[<800c2990>] cache_alloc_refill+0x39c/0x950
[<800c30c8>] __kmalloc_track_caller+0x184/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<8022bb7c>] br_send_bpdu+0x2c/0x150
[<8022beb8>] br_send_config_bpdu+0x1d0/0x1ec
[<8022b398>] br_transmit_config+0x148/0x180
[<8022b4b4>] br_config_bpdu_generation+0x60/0x8c
[<8022cc58>] br_hello_timer_expired+0x28/0x64
[<8006e204>] run_timer_softirq+0x17c/0x234
[<80068f6c>] __do_softirq+0xa4/0x154
[<80069078>] do_softirq+0x5c/0x98
[<80041444>] ret_from_irq+0x0/0x4
[<800c30e8>] __kmalloc_track_caller+0x1a4/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<80bc2144>] send_raw_packet+0x3c/0xc0 [batman_adv]
[<80bc23bc>] send_outstanding_bat_packet+0x1f4/0x250 [batman_adv]
[<800767b8>] worker_thread+0x1b0/0x278
[<8007ac14>] kthread+0x5c/0xa0
[<8004ec88>] kernel_thread_helper+0x10/0x18

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:160 active_file:3 inactive_anon:164
inactive_file:3 dirty:0 writeback:0 unstable:0
free:48 slab:2304 mapped:0 pagetables:49 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:640kB inactive_anon:656kB active_file:12kB inactive_file:12kB present:16256kB pages_scanned:2290 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
23 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
739 pages reserved
6 pages shared
3185 pages non-shared
bat_events: page allocation failure. order:0, mode:0x20

Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<8009f0f0>] __alloc_pages_internal+0x3ec/0x434
[<800c2990>] cache_alloc_refill+0x39c/0x950
[<800c30c8>] __kmalloc_track_caller+0x184/0x208
[<801aee18>] __alloc_skb+0x7c/0x158
[<801afd2c>] dev_alloc_skb+0x1c/0x4c
[<8069c2c8>] ieee80211_dev_alloc_skb+0x18/0x60 [ath_ahb]
[<80683f98>] ath_sysctl_unregister+0x2680/0x6ecc [ath_ahb]

Mem-Info:
Normal per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
Active_anon:160 active_file:3 inactive_anon:164
inactive_file:3 dirty:0 writeback:0 unstable:0
free:48 slab:2304 mapped:0 pagetables:49 bounce:0
Normal free:192kB min:508kB low:632kB high:760kB active_anon:640kB inactive_anon:656kB active_file:12kB inactive_file:12kB present:16256kB pages_scanned:2290 all_unreclaimable? yes lowmem_reserve[]: 0 0
Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 192kB
23 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
4096 pages RAM
739 pages reserved
6 pages shared
3185 pages non-shared
wifi0: ath_reset: Unable to start receive logic.
watchdog expired, rebooting system
+

#4 lore raw
Quoted 6 lines · Linus L??ssing

Hi Andrew,

Here is the output with the symbols inserted. I can reproduce this memory leak very reliable now, by just activating the vis server (didn't do that with the logs I've posted before) in a couple of minutes. free is showing a nice countdown then :).

So it is vis server functionality which is leaking? If vis server is not enabled, there is no leak?

If so, we don't need kmemcheck. I expect just looking at the vis code is probably enough to find the missing free.

Andrew

#5 lore raw
Quoted 8 lines · Andrew Lunn
Quoted 6 lines · Linus L??ssing

Hi Andrew,

Here is the output with the symbols inserted. I can reproduce this memory leak very reliable now, by just activating the vis server (didn't do that with the logs I've posted before) in a couple of minutes. free is showing a nice countdown then :).

Quoted 5 lines

So it is vis server functionality which is leaking? If vis server is not enabled, there is no leak?

If so, we don't need kmemcheck. I expect just looking at the vis code is probably enough to find the missing free.

It looks pretty obvious:

Working our way via the call stack:

In batman_skb_recv() we have:

        case BAT_VIS:
                ret = recv_vis_packet(skb);
                break;
        default:
                ret = NET_RX_DROP;
        }
        if (ret == NET_RX_DROP)
                kfree_skb(skb);

i.e. the packet is only freed if recv_vis_packet() returns NET_RX_DROP

In recv_vis_packet() we have:

       switch (vis_packet->vis_type) {
        case VIS_TYPE_SERVER_SYNC:
                /* TODO: handle fragmented skbs properly */
                receive_server_sync_packet(vis_packet, skb_headlen(skb));
                ret = NET_RX_SUCCESS;
                break;

        case VIS_TYPE_CLIENT_UPDATE:
                /* TODO: handle fragmented skbs properly */
                receive_client_update_packet(vis_packet, skb_headlen(skb));
                ret = NET_RX_SUCCESS;
                break;

        default:        /* ignore unknown packet */
                ret = NET_RX_DROP;
                break;
        }
        return ret;

i.e. receive_client_update_packet() and receive_server_sync_packet() need to sometime result in the packet being freed. batman_skb_recv() will not free the packet.

void receive_server_sync_packet(struct vis_packet *vis_packet, int vis_info_len)

{
        struct vis_info *info;
        int is_new;
        unsigned long flags;
        int vis_server = atomic_read(&vis_mode);

        spin_lock_irqsave(&vis_hash_lock, flags);
        info = add_packet(vis_packet, vis_info_len, &is_new);
        if (info == NULL)
                goto end;

        /* only if we are server ourselves and packet is newer than the one in
         * hash.*/
        if (vis_server == VIS_TYPE_SERVER_SYNC && is_new) {
                memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
                if (list_empty(&info->send_list))
                        list_add_tail(&info->send_list, &send_list);
        }
end:
        spin_unlock_irqrestore(&vis_hash_lock, flags);
}

No free here. vis_packet is a pointer into the skbuf, but we never add vis_packet to any queue here, just info is queued.

add_packet():

        info = kmalloc(sizeof(struct vis_info) + vis_info_len, GFP_ATOMIC);
        if (info == NULL)
                return NULL;

        INIT_LIST_HEAD(&info->send_list);
        INIT_LIST_HEAD(&info->recv_list);
        info->first_seen = jiffies;
        memcpy(&info->packet, vis_packet,
               sizeof(struct vis_packet) + vis_info_len);

We allocate memory can copy the contents of the packet into the allocated memory. So again, the skbuf is not queued.

As far as i can see, the skbuf is never put on a queue. We just copy the needed data out of it. This means the skbuf does need freeing.

I had a very quick look at receive_client_update_packet(). It seems to have the same structure.

I will submit a fix for this soon. Is there any other similar code which might have the same bug?

Andrew

#6 lore raw
In reply to Linus Lüssing

Staging: batman-adv: Fix skbuff leak in VIS code.

The vis code takes a copy of the data inside the skbuf if it is interesting for us, so we always need to release the skbuf.

routing.c +4 -3
Index: routing.c
===================================================================
@@ -1135,21 +1135,22 @@
 	if (is_my_mac(vis_packet->sender_orig))
 		return NET_RX_DROP;
 
+	/* We take a copy of the data in the packet, so we should
+	   always free the skbuf. */
+	ret = NET_RX_DROP;
+	
 	switch (vis_packet->vis_type) {
 	case VIS_TYPE_SERVER_SYNC:
 		/* TODO: handle fragmented skbs properly */
 		receive_server_sync_packet(vis_packet, skb_headlen(skb));
-		ret = NET_RX_SUCCESS;
 		break;
 
 	case VIS_TYPE_CLIENT_UPDATE:
 		/* TODO: handle fragmented skbs properly */
 		receive_client_update_packet(vis_packet, skb_headlen(skb));
-		ret = NET_RX_SUCCESS;
 		break;
 
 	default:	/* ignore unknown packet */
-		ret = NET_RX_DROP;
 		break;
 	}
 	return ret;
#7 lore raw
Quoted 4 lines · On Monday 25 January 2010 14:47:47 Andrew Lunn wrote:

Staging: batman-adv: Fix skbuff leak in VIS code.

The vis code takes a copy of the data inside the skbuf if it is
interesting for us, so we always need to release the skbuf.

Applied in rev 1564.

Thanks,
Marek

#9 lore raw
In reply to Linus Lüssing

And another (different) one. Which occured right after flashing and booting causing a kernel panic.
Sorry couldn't find a method for reproducing them yet.

Cheers, Linus

root@OpenWrt:/# n+Warning: IP address 192.168.1.1 in use
Ethernet eth0: MAC address 00:22:b0:98:87:de
IP: 192.168.1.1/255.255.255.0, Gateway: 0.0.0.0
Default server: 192.168.1.2

RedBoot(tm) bootstrap and debug environment [ROMRAM] production release, version "2.1.3" - built 18:43:19, Sep 20 2007

Platform: ap61 (Atheros WiSOC)
Copyright (C) 2000, 2001, 2002, 2003, 2004 Red Hat, Inc. Copyright (C) 2007, NewMedia-NET GmbH.

Board: DLINK DIR-300
RAM: 0x80000000-0x81000000, [0x80040580-0x80fe1000] available
FLASH: 0xbfc00000 - 0xbfff0000, 64 blocks of 0x00010000 bytes each. == Executing boot script in 5.000 seconds - enter ^C to abort +Warning: IP address 192.168.1.1 in use
Ethernet eth0: MAC address 00:22:b0:98:87:de
IP: 192.168.1.1/255.255.255.0, Gateway: 0.0.0.0
Default server: 192.168.1.2

RedBoot(tm) bootstrap and debug environment [ROMRAM] production release, version "2.1.3" - built 18:43:19, Sep 20 2007

Platform: ap61 (Atheros WiSOC)
Copyright (C) 2000, 2001, 2002, 2003, 2004 Red Hat, Inc. Copyright (C) 2007, NewMedia-NET GmbH.

Board: DLINK DIR-300
RAM: 0x80000000-0x81000000, [0x80040580-0x80fe1000] available
FLASH: 0xbfc00000 - 0xbfff0000, 64 blocks of 0x00010000 bytes each.
== Executing boot script in 5.000 seconds - enter ^C to abort
DD-WRT> fis load -l vmlinux.bin.l7
Image loaded from 0x80041000-0x802c8200
DD-WRT> exec
Now booting linux kernel:
Base address 0x80030000 Entry 0x80041000
Cmdline :
Linux version 2.6.30.10 (linus@Linus-Debian) (gcc version 4.3.3 (GCC) ) #2 Sun Jan 24 04:12:11 CET 2010 CPU revision is: 00019064 (MIPS 4KEc)
Determined physical RAM map:
memory: 01000000 @ 00000000 (usable)
Initrd not found or empty - disabling initrd
Zone PFN ranges:
Normal 0x00000000 -> 0x00001000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges

    0: 0x00000000 -> 0x00001000

Built 1 zonelists in Zone order, mobility grouping off. Total pages: 4064
Kernel command line: console=ttyS0,9600 rootfstype=squashfs,jffs2
Primary instruction cache 16kB, VIPT, 4-way, linesize 16 bytes.
Primary data cache 16kB, 4-way, VIPT, no aliases, linesize 16 bytes
NR_IRQS:128
PID hash table entries: 64 (order: 6, 256 bytes)
console [ttyS0] enabled
Dentry cache hash table entries: 2048 (order: 1, 8192 bytes)
Inode-cache hash table entries: 1024 (order: 0, 4096 bytes)
Memory: 13300k/16384k available (2002k kernel code, 3084k reserved, 458k data, 128k init, 0k highmem) Calibrating delay loop... 183.50 BogoMIPS (lpj=917504)
Mount-cache hash table entries: 512
net_namespace: 732 bytes
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 512 (order: 0, 4096 bytes)
TCP bind hash table entries: 512 (order: -1, 2048 bytes)
TCP: Hash tables configured (established 512 bind 512)
TCP reno registered
NET: Registered protocol family 1
Radio config found at offset 0xf8(0x1f8)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering mini_fo version $Id$
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
msgmni has been set to 25
io scheduler noop registered
io scheduler deadline registered (default)
gpiodev: gpio device registered with major 254
gpiodev: gpio platform device registered with access mask FFFFFFFF
Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
serial8250: ttyS0 at MMIO 0xb1100003 (irq = 37) is a 16550A
eth0: Atheros AR231x: 00:22:b0:98:87:de, irq 4
ar231x_eth_mii: probed
eth0: attached PHY driver [IC+ IP175C] (mii_bus:phy_addr=0:00)
cmdlinepart partition parsing not available
Searching for RedBoot partition table in spiflash at offset 0x3d0000
Searching for RedBoot partition table in spiflash at offset 0x3e0000
6 RedBoot partitions found on MTD device spiflash
Creating 6 MTD partitions on "spiflash":
0x000000000000-0x000000030000 : "RedBoot"
0x000000030000-0x0000002f0000 : "rootfs"
mtd: partition "rootfs" set to be root filesystem
mtd: partition "rootfs_data" created automatically, ofs=230000, len=C0000 0x000000230000-0x0000002f0000 : "rootfs_data"
0x0000002f0000-0x0000003d0000 : "vmlinux.bin.l7"
0x0000003e0000-0x0000003ef000 : "FIS directory"
0x0000003ef000-0x0000003f0000 : "RedBoot config"
0x0000003f0000-0x000000400000 : "boardconfig"
TCP westwood registered
NET: Registered protocol family 17
Bridge firewalling registered
802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
All bugs added by David S. Miller <[email protected]>
VFS: Mounted root (squashfs filesystem) readonly on device 31:1.
Freeing unused kernel memory: 128k freed
Please be patient, while OpenWrt loads ...
- preinit -
Press CTRL-C for failsafe
jffs2 not ready yet; using ramdisk
mini_fo: using base directory: /
mini_fo: using storage directory: /tmp/root
- init -

Please press Enter to activate this console. NET: Registered protocol family 10
lo: Disabled Privacy Extensions
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
device eth0.1 entered promiscuous mode
device eth0 entered promiscuous mode
br-mesh: port 1(eth0.1) entering forwarding state
ip_tables: (C) 2000-2006 Netfilter Core Team
Ebtables v2.0 registered
ip6_tables: (C) 2000-2006 Netfilter Core Team
device eth0.4 entered promiscuous mode
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-wan_vpn: starting userspace STP failed, starting kernel STP
batman-adv:B.A.T.M.A.N. advanced 0.2.1-beta r1545 (compatibility version 8) loaded
ath_hal: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint
ath_hal: 2009-05-08 (AR5212, AR5312, RF5111, RF5112, RF2316, RF2317, REGOPS_FUNC, TX_DESC_SWAP, XR)
ath_ahb: trunk
wlan: trunk
wlan: mac acl policy registered
ath_rate_minstrel: Minstrel automatic rate control algorithm 1.2 (trunk)
ath_rate_minstrel: look around rate set to 10%
ath_rate_minstrel: EWMA rolloff level set to 75%
ath_rate_minstrel: max segment size in the mrr set to 6000 us
Atheros HAL provided by OpenWrt, DD-WRT and MakSat Technologies
wifi0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
wifi0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: turboG rates: 6Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: H/W encryption support: WEP AES AES_CCM TKIP
ath_ahb: wifi0: Atheros 2317 WiSoC REV1: mem=0xb0000000, irq=3
IRQ 3/wifi0: IRQF_DISABLED is not guaranteed on shared IRQs
device bat0 entered promiscuous mode
br-mesh: port 2(bat0) entering forwarding state
br-wan_vpn: port 1(eth0.4) entering disabled state
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-mesh: port 2(bat0) entering disabled state
br-mesh: port 1(eth0.1) entering disabled state
br-mesh: port 2(bat0) entering forwarding state
br-mesh: port 1(eth0.1) entering forwarding state
batman-adv:Adding interface: ath0
batman-adv:Interface activated: ath0
CPU 0 Unable to handle kernel paging request at virtual address 00000010, epc == 80be3584, ra == 80be34e4 Oops[#1]:
Cpu 0

$ 0   : 00000000 10009c00 00000010 80afd800
$ 4   : 00000000 00000000 00000000 80be85f8
$ 8   : 0000611b 80afd68e 00000010 00000000
$12   : 00000000 feced300 ffffffff 61636520
$16   : 8064ca40 8064ca00 80afd680 8064ca40
$20   : 8064ca00 00000000 00000001 80640060
$24   : 00000010 8069c74c
$28   : 8046c000 8046d9a8 802a0000 80be34e4
Hi    : 00000000
Lo    : 0000000b
epc   : 80be3584 receive_bat_packet+0x3d8/0x6ec [batman_adv]
    Tainted: P
ra    : 80be34e4 receive_bat_packet+0x338/0x6ec [batman_adv]
Status: 10009c02    KERNEL EXL
Cause : 10800008
BadVA : 00000010
PrId  : 00019064 (MIPS 4KEc)
Modules linked in: ath_ahb ath_hal(P) batman_adv ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables ebt_redirect ebt_mark ebt_vlan ebt_stp ebt_pkttype ebt_mark_m ebt_limit ebt_among ebt_802_3 ebtable_nat ebtable_filter ebtable_broute ebtables xt_quota xt_pkttype xt_physdev ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables tun ipv6
Process batctl (pid: 1140, threadinfo=8046c000, task=8052bad0, tls=00000000)
Stack : 00000000 806a2aac 10009c01 80303620 802a0000 80f38a90 8064006c 00000001
        80640058 00000000 00000002 80640052 80640074 0000000c 80afd800 00000000
        80640060 00000000 80640060 00000040 80640052 80afd800 00004305 00000000
        80640040 80be8094 80f38a90 8005cd6c 00000001 80c27798 80afd800 80c27798
        80303620 80c176b8 80640052 80afd800 10009c01 80547000 80547000 80be2ee4
        ...
Call Trace:
[<80be3584>] receive_bat_packet+0x3d8/0x6ec [batman_adv]
[<80be8094>] receive_aggr_bat_packet+0x7c/0xbc [batman_adv]
[<80be2ee4>] recv_bat_packet+0x94/0x24c [batman_adv]
[<80be79c8>] batman_skb_recv+0x128/0x1dc [batman_adv]
[<806a1494>] ieee80211_saveath+0xb24/0xb80 [ath_ahb]
Code: 9245000e  84640008  00441021 <90440000> 00a4102b  00a2200b  10800003  00000000  14a00003
Kernel panic - not syncing: Fatal exception in interrupt
#10 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

In reply to Linus Lüssing

I'm now able to reproduce the second problem as well:
Activating the vis-server on one node causes the vis-clients to throw this call trace with its slow path warning imediately. Nevertheless, vis output works anyway, clients are sending vis-packets and the server is drawing nice graphs. See the attachment for an example call trace.

Cheers, Linus

Linux version 2.6.30.10 (linus@Linus-Debian) (gcc version 4.3.3 (GCC) ) #3 Tue Jan 26 02:18:56 CET 2010 CPU revision is: 00019064 (MIPS 4KEc)
Determined physical RAM map:
memory: 01000000 @ 00000000 (usable)
Initrd not found or empty - disabling initrd
Zone PFN ranges:
Normal 0x00000000 -> 0x00001000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges

    0: 0x00000000 -> 0x00001000

On node 0 totalpages: 4096
free_area_init_node: node 0, pgdat 802a0f40, node_mem_map 802d9000
Normal zone: 32 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 4064 pages, LIFO batch:0
Built 1 zonelists in Zone order, mobility grouping off. Total pages: 4064
Kernel command line: console=ttyS0,9600 rootfstype=squashfs,jffs2
Primary instruction cache 16kB, VIPT, 4-way, linesize 16 bytes.
Primary data cache 16kB, 4-way, VIPT, no aliases, linesize 16 bytes
NR_IRQS:128
PID hash table entries: 64 (order: 6, 256 bytes)
console [ttyS0] enabled
Dentry cache hash table entries: 2048 (order: 1, 8192 bytes)
Inode-cache hash table entries: 1024 (order: 0, 4096 bytes)
Memory: 13324k/16384k available (1985k kernel code, 3060k reserved, 452k data, 128k init, 0k highmem) Calibrating delay loop... 183.50 BogoMIPS (lpj=917504)
Mount-cache hash table entries: 512
net_namespace: 732 bytes
NET: Registered protocol family 16
bio: create slab <bio-0> at 0
Switched to high resolution mode on CPU 0
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 512 (order: 0, 4096 bytes)
TCP bind hash table entries: 512 (order: -1, 2048 bytes)
TCP: Hash tables configured (established 512 bind 512)
TCP reno registered
NET: Registered protocol family 1
Radio config found at offset 0xf8(0x1f8)
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering mini_fo version $Id$
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
msgmni has been set to 26
io scheduler noop registered
io scheduler deadline registered (default)
gpiodev: gpio device registered with major 254
gpiodev: gpio platform device registered with access mask FFFFFFFF
Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
serial8250: ttyS0 at MMIO 0xb1100003 (irq = 37) is a 16550A
eth0: Atheros AR231x: 00:22:b0:98:87:fa, irq 4
ar231x_eth_mii: probed
eth0: attached PHY driver [IC+ IP175C] (mii_bus:phy_addr=0:00)
cmdlinepart partition parsing not available
Searching for RedBoot partition table in spiflash at offset 0x3d0000
Searching for RedBoot partition table in spiflash at offset 0x3e0000
6 RedBoot partitions found on MTD device spiflash
Creating 6 MTD partitions on "spiflash":
0x000000000000-0x000000030000 : "RedBoot"
0x000000030000-0x0000002f0000 : "rootfs"
mtd: partition "rootfs" set to be root filesystem
mtd: partition "rootfs_data" created automatically, ofs=230000, len=C0000 0x000000230000-0x0000002f0000 : "rootfs_data"
0x0000002f0000-0x0000003d0000 : "vmlinux.bin.l7"
0x0000003e0000-0x0000003ef000 : "FIS directory"
0x0000003ef000-0x0000003f0000 : "RedBoot config"
0x0000003f0000-0x000000400000 : "boardconfig"
TCP westwood registered
NET: Registered protocol family 17
Bridge firewalling registered
802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
All bugs added by David S. Miller <[email protected]>
VFS: Mounted root (squashfs filesystem) readonly on device 31:1.
Freeing unused kernel memory: 128k freed
Please be patient, while OpenWrt loads ...
mini_fo: using base directory: /
mini_fo: using storage directory: /tmp/root
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
ip_tables: (C) 2000-2006 Netfilter Core Team
device eth0.1 entered promiscuous mode
device eth0 entered promiscuous mode
br-mesh: port 1(eth0.1) entering forwarding state
Ebtables v2.0 registered
ip6_tables: (C) 2000-2006 Netfilter Core Team
batman-adv:B.A.T.M.A.N. advanced 0.2.1-beta r1564 (compatibility version 8) loaded
ath_hal: module license 'Proprietary' taints kernel.
Disabling lock debugging due to kernel taint
ath_hal: 2009-05-08 (AR5212, AR5312, RF5111, RF5112, RF2316, RF2317, REGOPS_FUNC, TX_DESC_SWAP, XR) device eth0.4 entered promiscuous mode
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-wan_vpn: starting userspace STP failed, starting kernel STP
eth0.1: no IPv6 routers present
br-mesh: no IPv6 routers present
ath_ahb: trunk
wlan: trunk
wlan: mac acl policy registered
ath_rate_minstrel: Minstrel automatic rate control algorithm 1.2 (trunk)
ath_rate_minstrel: look around rate set to 10%
ath_rate_minstrel: EWMA rolloff level set to 75%
ath_rate_minstrel: max segment size in the mrr set to 6000 us
Atheros HAL provided by OpenWrt, DD-WRT and MakSat Technologies
wifi0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
wifi0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: turboG rates: 6Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
wifi0: H/W encryption support: WEP AES AES_CCM TKIP
ath_ahb: wifi0: Atheros 2317 WiSoC REV1: mem=0xb0000000, irq=3
IRQ 3/wifi0: IRQF_DISABLED is not guaranteed on shared IRQs
eth0.2: no IPv6 routers present
device bat0 entered promiscuous mode
br-mesh: port 2(bat0) entering forwarding state
eth0.3: no IPv6 routers present
eth0.4: no IPv6 routers present
bat0: no IPv6 routers present
br-wan_vpn: port 1(eth0.4) entering disabled state
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 1(eth0.4) entering forwarding state
br-mesh: port 2(bat0) entering disabled state
br-mesh: port 1(eth0.1) entering disabled state
br-mesh: port 2(bat0) entering forwarding state
br-mesh: port 1(eth0.1) entering forwarding state
br-wan_vpn: received tcn bpdu on port 1(eth0.4)
br-wan_vpn: topology change detected, propagating
ath0: no IPv6 routers present
br-mesh: no IPv6 routers present
batman-adv:Adding interface: ath0
batman-adv:Interface activated: ath0
batman-adv:Adding interface: eth0.2
batman-adv:Interface activated: eth0.2
batman-adv:Adding interface: eth0.3
batman-adv:Interface activated: eth0.3
jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
jffs2_build_filesystem(): unlocking the mtd device... done.
jffs2_build_filesystem(): erasing all blocks after the end marker... <6>device VPN entered promiscuous mode
br-wan_vpn: topology change detected, propagating
br-wan_vpn: port 2(VPN) entering forwarding state
done.
mini_fo: using base directory: /
mini_fo: using storage directory: /jffs
br-wan_vpn: received tcn bpdu on port 1(eth0.4)
br-wan_vpn: topology change detected, propagating
br-wan_vpn: received tcn bpdu on port 1(eth0.4)
br-wan_vpn: topology change detected, propagating
br-wan_vpn: received tcn bpdu on port 1(eth0.4)
br-wan_vpn: topology change detected, propagating

------------[ cut here ]------------
WARNING: at kernel/softirq.c:141 local_bh_enable+0x48/0xa0()
Modules linked in: ath_ahb ath_hal(P) batman_adv ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables ebt_redirect ebt_mark ebt_vlan ebt_stp ebt_pkttype ebt_mark_m ebt_limit ebt_among ebt_802_3 ebtable_nat ebtable_filter ebtable_broute ebtables xt_quota xt_pkttype xt_physdev ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables tun ipv6
Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<80060cd8>] warn_slowpath_common+0x70/0xb0
[<800674ac>] local_bh_enable+0x48/0xa0
[<801b3ef8>] dev_queue_xmit+0x388/0x3dc
[<8062c454>] ieee80211_parent_queue_xmit+0x8c/0xb4 [ath_ahb]
[<8062c778>] ieee80211_hardstart+0x2fc/0x34c [ath_ahb]
[<801b3e5c>] dev_queue_xmit+0x2ec/0x3dc
[<80bb68d8>] vis_quit+0x7c4/0x950 [batman_adv]

---[ end trace e96b16c908cf7c40 ]---

br-wan_vpn: received tcn bpdu on port 1(eth0.4)
br-wan_vpn: topology change detected, propagating
root@OpenWrt:~#

#11 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 6 lines · On Tuesday 26 January 2010 14:13:11 Linus Lüssing wrote:

I'm now able to reproduce the second problem as well:
Activating the vis-server on one node causes the vis-clients to throw this call trace with its slow path warning imediately. Nevertheless, vis output works anyway, clients are sending vis-packets and the server is drawing nice graphs. See the attachment for an example call trace.

Please try the attached patch and see if it helps.

Regards,
Marek

batman-adv-kernelland/vis.c +47 -24
index b118d1e..a6c235f 100644
@@ -405,6 +405,9 @@ static void purge_vis_packets(void)
 {
 	HASHIT(hashit);
 	struct vis_info *info;
+	unsigned long flags;
+
+	spin_lock_irqsave(&vis_hash_lock, flags);
 
 	while (hash_iterate(vis_hash, &hashit)) {
 		info = hashit.bucket->data;
@@ -416,13 +419,17 @@ static void purge_vis_packets(void)
 			free_info(info);
 		}
 	}
+
+	spin_unlock_irqrestore(&vis_hash_lock, flags);
 }
 
 static void broadcast_vis_packet(struct vis_info *info, int packet_length)
 {
 	HASHIT(hashit);
 	struct orig_node *orig_node;
+	struct batman_if *batman_if;
 	unsigned long flags;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 
@@ -431,45 +438,60 @@ static void broadcast_vis_packet(struct vis_info *info, int packet_length)
 		orig_node = hashit.bucket->data;
 
 		/* if it's a vis server and reachable, send it. */
-		if (orig_node &&
-		    (orig_node->flags & VIS_SERVER) &&
-		    orig_node->batman_if &&
-		    orig_node->router) {
+		if ((!orig_node) || (!orig_node->batman_if) ||
+		    (!orig_node->router))
+			continue;
 
-			/* don't send it if we already received the packet from
-			 * this node. */
-			if (recv_list_is_in(&info->recv_list, orig_node->orig))
-				continue;
+		if (!(orig_node->flags & VIS_SERVER))
+			continue;
 
-			memcpy(info->packet.target_orig,
-			       orig_node->orig, ETH_ALEN);
+		/* don't send it if we already received the packet from
+		 * this node. */
+		if (recv_list_is_in(&info->recv_list, orig_node->orig))
+			continue;
 
-			send_raw_packet((unsigned char *) &info->packet,
-					packet_length,
-					orig_node->batman_if,
-					orig_node->router->addr);
-		}
+		spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+		memcpy(info->packet.target_orig, orig_node->orig, ETH_ALEN);
+		batman_if = orig_node->batman_if;
+		memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+
+		send_raw_packet((unsigned char *)&info->packet,
+				packet_length, batman_if, dstaddr);
+
+		spin_lock_irqsave(&orig_hash_lock, flags);
 	}
-	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
+
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
+	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
 }
 
 static void unicast_vis_packet(struct vis_info *info, int packet_length)
 {
 	struct orig_node *orig_node;
+	struct batman_if *batman_if;
 	unsigned long flags;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 	orig_node = ((struct orig_node *)
 		     hash_find(orig_hash, info->packet.target_orig));
 
-	if ((orig_node != NULL) &&
-	    (orig_node->batman_if != NULL) &&
-	    (orig_node->router != NULL)) {
-		send_raw_packet((unsigned char *) &info->packet, packet_length,
-				orig_node->batman_if,
-				orig_node->router->addr);
-	}
+	if ((!orig_node) || (!orig_node->batman_if) || (!orig_node->router))
+		goto out;
+
+	/* don't lock while sending the packets ... we therefore
+	 * copy the required data before sending */
+	batman_if = orig_node->batman_if;
+	memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+	send_raw_packet((unsigned char *)&info->packet,
+			packet_length, batman_if, dstaddr);
+
+	return;
+
+out:
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
@@ -502,17 +524,18 @@ static void send_vis_packets(struct work_struct *work)
 	struct vis_info *info, *temp;
 	unsigned long flags;
 
-	spin_lock_irqsave(&vis_hash_lock, flags);
 	purge_vis_packets();
 
 	if (generate_vis_packet() == 0)
 		/* schedule if generation was successful */
 		list_add_tail(&my_vis_info->send_list, &send_list);
 
+	spin_lock_irqsave(&vis_hash_lock, flags);
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
 		list_del_init(&info->send_list);
 		send_vis_packet(info);
 	}
+
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();
 }
#12 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hi Marek,

nope, does not seem to work, still the same issue I also tried it on my laptop here with the patch installed and it is a very similar call trace, although here it is send_vis_packets instead of vis_quit in the call trace.

Cheers, Linus

Quoted 139 lines · Marek Lindner
Quoted 6 lines · On Tuesday 26 January 2010 14:13:11 Linus Lüssing wrote:

I'm now able to reproduce the second problem as well:
Activating the vis-server on one node causes the vis-clients to throw this call trace with its slow path warning imediately. Nevertheless, vis output works anyway, clients are sending vis-packets and the server is drawing nice graphs. See the attachment for an example call trace.

Please try the attached patch and see if it helps.

Regards,
Marek

batman-adv-kernelland/vis.c +45 -24
index b118d1e..a6c235f 100644
@@ -405,6 +405,9 @@ static void purge_vis_packets(void)
 {
 	HASHIT(hashit);
 	struct vis_info *info;
+	unsigned long flags;
+
+	spin_lock_irqsave(&vis_hash_lock, flags);
 
 	while (hash_iterate(vis_hash, &hashit)) {
 		info = hashit.bucket->data;
@@ -416,13 +419,17 @@ static void purge_vis_packets(void)
 			free_info(info);
 		}
 	}
+
+	spin_unlock_irqrestore(&vis_hash_lock, flags);
 }
 
 static void broadcast_vis_packet(struct vis_info *info, int packet_length)
 {
 	HASHIT(hashit);
 	struct orig_node *orig_node;
+	struct batman_if *batman_if;
 	unsigned long flags;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 
@@ -431,45 +438,60 @@ static void broadcast_vis_packet(struct vis_info *info, int packet_length)
 		orig_node = hashit.bucket->data;
 
 		/* if it's a vis server and reachable, send it. */
-		if (orig_node &&
-		    (orig_node->flags & VIS_SERVER) &&
-		    orig_node->batman_if &&
-		    orig_node->router) {
+		if ((!orig_node) || (!orig_node->batman_if) ||
+		    (!orig_node->router))
+			continue;
 
-			/* don't send it if we already received the packet from
-			 * this node. */
-			if (recv_list_is_in(&info->recv_list, orig_node->orig))
-				continue;
+		if (!(orig_node->flags & VIS_SERVER))
+			continue;
 
-			memcpy(info->packet.target_orig,
-			       orig_node->orig, ETH_ALEN);
+		/* don't send it if we already received the packet from
+		 * this node. */
+		if (recv_list_is_in(&info->recv_list, orig_node->orig))
+			continue;
 
-			send_raw_packet((unsigned char *) &info->packet,
-					packet_length,
-					orig_node->batman_if,
-					orig_node->router->addr);
-		}
+		spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+		memcpy(info->packet.target_orig, orig_node->orig, ETH_ALEN);
+		batman_if = orig_node->batman_if;
+		memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+
+		send_raw_packet((unsigned char *)&info->packet,
+				packet_length, batman_if, dstaddr);
+
+		spin_lock_irqsave(&orig_hash_lock, flags);
 	}
-	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
+
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
+	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
 }
 
 static void unicast_vis_packet(struct vis_info *info, int packet_length)
 {
 	struct orig_node *orig_node;
+	struct batman_if *batman_if;
 	unsigned long flags;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 	orig_node = ((struct orig_node *)
 		     hash_find(orig_hash, info->packet.target_orig));
 
-	if ((orig_node != NULL) &&
-	    (orig_node->batman_if != NULL) &&
-	    (orig_node->router != NULL)) {
-		send_raw_packet((unsigned char *) &info->packet, packet_length,
-				orig_node->batman_if,
-				orig_node->router->addr);
-	}
+	if ((!orig_node) || (!orig_node->batman_if) || (!orig_node->router))
+		goto out;
+
+	/* don't lock while sending the packets ... we therefore
+	 * copy the required data before sending */
+	batman_if = orig_node->batman_if;
+	memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+	send_raw_packet((unsigned char *)&info->packet,
+			packet_length, batman_if, dstaddr);
+
+	return;
+
+out:
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
@@ -502,17 +524,18 @@ static void send_vis_packets(struct work_struct *work)
 	struct vis_info *info, *temp;
 	unsigned long flags;
 
-	spin_lock_irqsave(&vis_hash_lock, flags);
 	purge_vis_packets();
 
 	if (generate_vis_packet() == 0)
 		/* schedule if generation was successful */
 		list_add_tail(&my_vis_info->send_list, &send_list);

batman-adv-kernelland/vis.c static void send_vis_packets(struct work_struct *work) · line 541

Quoted 10 lines
batman-adv-kernelland/vis.c +2 -0
 
+	spin_lock_irqsave(&vis_hash_lock, flags);
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
 		list_del_init(&info->send_list);
 		send_vis_packet(info);
 	}
+
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();
 }
[22933.613178] ------------[ cut here ]------------
[22933.613199] WARNING: at /tmp/buildd/linux-2.6-2.6.31/debian/build/source_i386_none/kernel/softirq.c:143 _local_bh_enable_ip+0x41/0x97()
[22933.613206] Hardware name: ���������������
[22933.613211] Modules linked in: batman_adv isofs udf nvidia(P) ppdev lp parport sco bridge stp bnep rfcomm kvm_intel kvm acpi_cpufreq cpufreq_powersave cpufreq_conservative cpufreq_userspace cpufreq_stats l2cap nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc fuse dm_snapshot dm_mirror dm_region_hash dm_log firewire_sbp2 loop snd_hda_codec_realtek snd_hda_intel snd_hda_codec arc4 ecb acer_wmi iwlagn snd_hwdep iwlcore snd_pcm_oss snd_mixer_oss snd_pcm mac80211 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device uvcvideo videodev snd soundcore i2c_i801 cp210x v4l1_compat btusb snd_page_alloc cfg80211 v4l2_compat_ioctl32 psmouse bluetooth wmi serio_raw rfkill i2c_core usbserial evdev pcspkr ac processor button battery ext3 jbd mbcache sha256_generic cryptd aes_x86_64 aes_generic cbc dm_crypt dm_mod sd_mod crc_t10dif ide_cd_mod cdrom ata_generic ide_pci_generic ahci sdhci_pci sdhci uhci_hcd mmc_core libata tg3 led_class libphy firewire_ohci firewire_core crc_itu_t ricoh_mmc video output piix ide_core scsi_mod ehci_hcd intel_agp thermal fan thermal_sys [last unloaded: batman_adv]
[22933.613436] Pid: 14882, comm: bat_events Tainted: P           2.6.31-1-amd64 #1
[22933.613442] Call Trace:
[22933.613453]  [<ffffffff810503fa>] ? _local_bh_enable_ip+0x41/0x97
[22933.613462]  [<ffffffff810503fa>] ? _local_bh_enable_ip+0x41/0x97
[22933.613471]  [<ffffffff8104af08>] ? warn_slowpath_common+0x77/0xa3
[22933.613480]  [<ffffffff810503fa>] ? _local_bh_enable_ip+0x41/0x97
[22933.613490]  [<ffffffff812c821f>] ? run_filter+0x33/0x3a
[22933.613498]  [<ffffffff812c8c56>] ? packet_rcv+0xa9/0x2fb
[22933.613511]  [<ffffffff812389e5>] ? dev_hard_start_xmit+0x143/0x2ba
[22933.613521]  [<ffffffff8124ac7e>] ? __qdisc_run+0xe2/0x1ec
[22933.613529]  [<ffffffff81238e8f>] ? dev_queue_xmit+0x232/0x32c
[22933.613551]  [<ffffffffa0fe8945>] ? send_vis_packets+0x4f5/0x5d5 [batman_adv]
[22933.613571]  [<ffffffffa0fe8450>] ? send_vis_packets+0x0/0x5d5 [batman_adv]
[22933.613581]  [<ffffffff8105b43b>] ? worker_thread+0x174/0x211
[22933.613590]  [<ffffffff8105f39e>] ? autoremove_wake_function+0x0/0x2e
[22933.613599]  [<ffffffff8105b2c7>] ? worker_thread+0x0/0x211
[22933.613606]  [<ffffffff8105f042>] ? kthread+0x8b/0x93
[22933.613616]  [<ffffffff81011baa>] ? child_rip+0xa/0x20
[22933.613623]  [<ffffffff8105efb7>] ? kthread+0x0/0x93
[22933.613631]  [<ffffffff81011ba0>] ? child_rip+0x0/0x20
[22933.613636] ---[ end trace 1758b3e01bf5c445 ]---
#13 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hi,

Quoted 4 lines

nope, does not seem to work, still the same issue I also tried it on my laptop here with the patch installed and it is a very similar call trace, although here it is send_vis_packets instead of vis_quit in the call trace.

you should take the stack trace with a grain of salt. Stack tracing such an issue is a quite tricky thing, hence can not be fully trusted. Since the skb changes the code runs in interrupt context which probably introduced this bug. All previous warnings of this kind were related to holding a lock while sending packets. My patch unlocked the problematic orig_hash - maybe that was not enough ? By adding retrun statements at the beginning of the vis send function you might be able to get down to the problem.

Regards,
Marek

#14 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 8 lines · Marek Lindner

Hi,

Quoted 4 lines

nope, does not seem to work, still the same issue I also tried it on my laptop here with the patch installed and it is a very similar call trace, although here it is send_vis_packets instead of vis_quit in the call trace.

Quoted 7 lines

you should take the stack trace with a grain of salt. Stack tracing such an issue is a quite tricky thing, hence can not be fully trusted. Since the skb changes the code runs in interrupt context which probably introduced this bug. All previous warnings of this kind were related to holding a lock while sending packets. My patch unlocked the problematic orig_hash - maybe that was not enough ? By adding retrun statements at the beginning of the vis send function you might be able to get down to the problem.

It might also be worth running lockdep on the code. Normally you don^t need the actually lockup, you just need to execute the code path that would lockup under whatever conditions are required for it to lockup.

Andrew

#15 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

In reply to Linus Lüssing
Quoted 13 lines
WARNING: at kernel/softirq.c:141 local_bh_enable+0x48/0xa0()
Modules linked in: ath_ahb ath_hal(P) batman_adv ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables ebt_redirect ebt_mark ebt_vlan ebt_stp ebt_pkttype ebt_mark_m ebt_limit ebt_among ebt_802_3 ebtable_nat ebtable_filter ebtable_broute ebtables xt_quota xt_pkttype xt_physdev ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables tun ipv6
Call Trace:
[<800493a4>] dump_stack+0x8/0x34
[<80060cd8>] warn_slowpath_common+0x70/0xb0
[<800674ac>] local_bh_enable+0x48/0xa0
[<801b3ef8>] dev_queue_xmit+0x388/0x3dc
[<8062c454>] ieee80211_parent_queue_xmit+0x8c/0xb4 [ath_ahb]
[<8062c778>] ieee80211_hardstart+0x2fc/0x34c [ath_ahb]
[<801b3e5c>] dev_queue_xmit+0x2ec/0x3dc
[<80bb68d8>] vis_quit+0x7c4/0x950 [batman_adv]

---[ end trace e96b16c908cf7c40 ]---
Quoted 3 lines

br-wan_vpn: received tcn bpdu on port 1(eth0.4)
br-wan_vpn: topology change detected, propagating
root@OpenWrt:~#

I dug a little deeper into what his means. It means we are in interrupt context and interrupts are disabled. This seems to be not allowed when calling local_bh_enable(), which probably means it is not allowed when calling dev_queue_xmit.

What i don't understand yet is the call path. The Call Trace above does not make much sense. vis_quit() does not send any packets.

Looking at Marek's patch it is going in the right direction. send_skb_packet() cannot be called with interrupts disabled.
which means send_raw_packet() cannot be called with interrupts disabled. which means broadcast_vis_packet() needs to spin_unlock_irqrestore() before sending. which means unicast_vis_packet() needs to spin_unlock_irqrestore() before sending.

This is what Mareks patch does. However, this part of Marek's patch looks wrong:

        struct vis_info *info, *temp;
        unsigned long flags;
-       spin_lock_irqsave(&vis_hash_lock, flags);
        purge_vis_packets();

        if (generate_vis_packet() == 0)
                /* schedule if generation was successful */
                list_add_tail(&my_vis_info->send_list, &send_list);
+       spin_lock_irqsave(&vis_hash_lock, flags);
        list_for_each_entry_safe(info, temp, &send_list, send_list) {
                list_del_init(&info->send_list);
                send_vis_packet(info);
        }
+
        spin_unlock_irqrestore(&vis_hash_lock, flags);
        start_vis_timer();

}

You are disable interrupts and then call send_vis_packet(). This will not work since all calls under send_vis_packet will have interrupts disabled.

Andrew

#16 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 2 lines · On Friday 29 January 2010 16:25:45 Andrew Lunn wrote:

This is what Mareks patch does. However, this part of Marek's patch looks
wrong:

I'd like to point out that this lock wasn't introduced by my previously posted patch. I managed to reduce the effect of this lock but you can't remove it entirely without changing the whole sending mechanism.

Quoted 3 lines

You are disable interrupts and then call send_vis_packet(). This will not work since all calls under send_vis_packet will have interrupts disabled.

Yes, that is correct. The reason for this is simple: The interrupt flags are stored in send_vis_packets() and without passing them on we can't re-enable the interrupts. That did not appear to be a good solution. ;)
As far as I understand the vis_lock's only purpose is to avoid that the "info" pointer which is passed to send_vis_packet() can't be deleted while we use it. That might be better solved using reference counting.

Cheers,
Marek

#17 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hi Linus

Please could you try this patch and see if it fixes the vis problem. It compiles cleanly with 2.6.32. It is checkpatch clean, it is sparse clean. However, since i've not actually tried running the code i would not be too surprised if it deadlocked, leaked memory, oopsed, ...

Andrew

Staging: batman-adv: Don't have interrupts disabled while sending.

send_vis_packets() would disable interrupts before calling dev_queue_xmit() which resulting in a backtrace in local_bh_enable(). Fix this by using kref on the vis_info object so that we can call send_vis_packets() without holding vis_hash_lock. vis_hash_lock also used to protect recv_list, so we now need a new lock to protect that instead of vis_hash_lock.

Also a few checkpatch cleanups.

vis.c +102 -48
Index: vis.c
===================================================================
@@ -30,22 +30,26 @@
 
 struct hashtable_t *vis_hash;
 DEFINE_SPINLOCK(vis_hash_lock);
+static DEFINE_SPINLOCK(recv_list_lock);
 static struct vis_info *my_vis_info;
 static struct list_head send_list;	/* always locked with vis_hash_lock */
 
 static void start_vis_timer(void);
 
 /* free the info */
-static void free_info(void *data)
+static void free_info(struct kref *ref)
 {
-	struct vis_info *info = data;
+	struct vis_info *info = container_of(ref, struct vis_info, refcount);
 	struct recvlist_node *entry, *tmp;
+	unsigned long flags;
 
 	list_del_init(&info->send_list);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry_safe(entry, tmp, &info->recv_list, list) {
 		list_del(&entry->list);
 		kfree(entry);
 	}
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	kfree(info);
 }
 
@@ -147,32 +151,41 @@
 static void recv_list_add(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
+
 	entry = kmalloc(sizeof(struct recvlist_node), GFP_ATOMIC);
 	if (!entry)
 		return;
 
 	memcpy(entry->mac, mac, ETH_ALEN);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_add_tail(&entry->list, recv_list);
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 }
 
 /* returns 1 if this mac is in the recv_list */
 static int recv_list_is_in(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
 
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry(entry, recv_list, list) {
-		if (memcmp(entry->mac, mac, ETH_ALEN) == 0)
+		if (memcmp(entry->mac, mac, ETH_ALEN) == 0) {
+			spin_unlock_irqrestore(&recv_list_lock, flags);
 			return 1;
+		}
 	}
-
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	return 0;
 }
 
 /* try to add the packet to the vis_hash. return NULL if invalid (e.g. too old,
- * broken.. ).  vis hash must be locked outside.  is_new is set when the packet
+ * broken.. ).	vis hash must be locked outside.  is_new is set when the packet
  * is newer than old entries in the hash. */
 static struct vis_info *add_packet(struct vis_packet *vis_packet,
-				   int vis_info_len, int *is_new)
+				   int vis_info_len, int *is_new,
+				   int make_broadcast)
 {
 	struct vis_info *info, *old_info;
 	struct vis_info search_elem;
@@ -199,7 +212,7 @@
 		}
 		/* remove old entry */
 		hash_remove(vis_hash, old_info);
-		free_info(old_info);
+		kref_put(&old_info->refcount, free_info);
 	}
 
 	info = kmalloc(sizeof(struct vis_info) + vis_info_len, GFP_ATOMIC);
@@ -208,6 +221,7 @@
 
 	INIT_LIST_HEAD(&info->send_list);
 	INIT_LIST_HEAD(&info->recv_list);
+	kref_init(&info->refcount);
 	info->first_seen = jiffies;
 	memcpy(&info->packet, vis_packet,
 	       sizeof(struct vis_packet) + vis_info_len);
@@ -215,16 +229,21 @@
 	/* initialize and add new packet. */
 	*is_new = 1;
 
+	/* Make it a broadcast packet, if required */
+	if (make_broadcast)
+		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
+
 	/* repair if entries is longer than packet. */
 	if (info->packet.entries * sizeof(struct vis_info_entry) > vis_info_len)
-		info->packet.entries = vis_info_len / sizeof(struct vis_info_entry);
+		info->packet.entries = vis_info_len /
+			sizeof(struct vis_info_entry);
 
 	recv_list_add(&info->recv_list, info->packet.sender_orig);
 
 	/* try to add it */
 	if (hash_add(vis_hash, info) < 0) {
 		/* did not work (for some reason) */
-		free_info(info);
+		kref_put(&old_info->refcount, free_info);
 		info = NULL;
 	}
 
@@ -235,19 +254,20 @@
 void receive_server_sync_packet(struct vis_packet *vis_packet, int vis_info_len)
 {
 	struct vis_info *info;
-	int is_new;
+	int is_new, make_broadcast;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
 
+	make_broadcast = (vis_server == VIS_TYPE_SERVER_SYNC);
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, make_broadcast);
 	if (info == NULL)
 		goto end;
 
 	/* only if we are server ourselves and packet is newer than the one in
 	 * hash.*/
 	if (vis_server == VIS_TYPE_SERVER_SYNC && is_new) {
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
 		if (list_empty(&info->send_list))
 			list_add_tail(&info->send_list, &send_list);
 	}
@@ -263,24 +283,27 @@
 	int is_new;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
+	int are_target = 0;
 
 	/* clients shall not broadcast. */
 	if (is_bcast(vis_packet->target_orig))
 		return;
 
+	/* Are we the target for this VIS packet? */
+	if (vis_server == VIS_TYPE_SERVER_SYNC	&&
+	    is_my_mac(info->packet.target_orig))
+		are_target = 1;
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, are_target);
 	if (info == NULL)
 		goto end;
 	/* note that outdated packets will be dropped at this point. */
 
 
 	/* send only if we're the target server or ... */
-	if (vis_server == VIS_TYPE_SERVER_SYNC  &&
-	    is_my_mac(info->packet.target_orig) &&
-	    is_new) {
+	if (are_target && is_new) {
 		info->packet.vis_type = VIS_TYPE_SERVER_SYNC;	/* upgrade! */
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
 		if (list_empty(&info->send_list))
 			list_add_tail(&info->send_list, &send_list);
 
@@ -362,14 +385,17 @@
 	while (hash_iterate(orig_hash, &hashit_global)) {
 		orig_node = hashit_global.bucket->data;
 		if (orig_node->router != NULL
-			&& compare_orig(orig_node->router->addr, orig_node->orig)
+			&& compare_orig(orig_node->router->addr,
+					orig_node->orig)
 			&& orig_node->batman_if
 			&& (orig_node->batman_if->if_active == IF_ACTIVE)
 		    && orig_node->router->tq_avg > 0) {
 
 			/* fill one entry into buffer. */
 			entry = &entry_array[info->packet.entries];
-			memcpy(entry->src, orig_node->batman_if->net_dev->dev_addr, ETH_ALEN);
+			memcpy(entry->src,
+			       orig_node->batman_if->net_dev->dev_addr,
+			       ETH_ALEN);
 			memcpy(entry->dest, orig_node->orig, ETH_ALEN);
 			entry->quality = orig_node->router->tq_avg;
 			info->packet.entries++;
@@ -401,6 +427,8 @@
 	return 0;
 }
 
+/* free old vis packets. Must be called with this vis_hash_lock
+ * held */
 static void purge_vis_packets(void)
 {
 	HASHIT(hashit);
@@ -413,7 +441,7 @@
 		if (time_after(jiffies,
 			       info->first_seen + (VIS_TIMEOUT*HZ)/1000)) {
 			hash_remove_bucket(vis_hash, &hashit);
-			free_info(info);
+			kref_put(&info->refcount, free_info);
 		}
 	}
 }
@@ -423,6 +451,8 @@
 	HASHIT(hashit);
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 
@@ -431,46 +461,57 @@
 		orig_node = hashit.bucket->data;
 
 		/* if it's a vis server and reachable, send it. */
-		if (orig_node &&
-		    (orig_node->flags & VIS_SERVER) &&
-		    orig_node->batman_if &&
-		    orig_node->router) {
+		if ((!orig_node) || (!orig_node->batman_if) ||
+		    (!orig_node->router))
+			continue;
+		if (!(orig_node->flags & VIS_SERVER))
+			continue;
+		/* don't send it if we already received the packet from
+		 * this node. */
+		if (recv_list_is_in(&info->recv_list, orig_node->orig))
+			continue;
 
-			/* don't send it if we already received the packet from
-			 * this node. */
-			if (recv_list_is_in(&info->recv_list, orig_node->orig))
-				continue;
+		memcpy(info->packet.target_orig, orig_node->orig, ETH_ALEN);
+		batman_if = orig_node->batman_if;
+		memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+		spin_unlock_irqrestore(&orig_hash_lock, flags);
 
-			memcpy(info->packet.target_orig,
-			       orig_node->orig, ETH_ALEN);
+		send_raw_packet((unsigned char *)&info->packet,
+				packet_length, batman_if, dstaddr);
 
-			send_raw_packet((unsigned char *) &info->packet,
-					packet_length,
-					orig_node->batman_if,
-					orig_node->router->addr);
-		}
+		spin_lock_irqsave(&orig_hash_lock, flags);
+
 	}
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 static void unicast_vis_packet(struct vis_info *info, int packet_length)
 {
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 	orig_node = ((struct orig_node *)
 		     hash_find(orig_hash, info->packet.target_orig));
 
-	if ((orig_node != NULL) &&
-	    (orig_node->batman_if != NULL) &&
-	    (orig_node->router != NULL)) {
-		send_raw_packet((unsigned char *) &info->packet, packet_length,
-				orig_node->batman_if,
-				orig_node->router->addr);
-	}
+	if ((!orig_node) || (!orig_node->batman_if) || (!orig_node->router))
+		goto out;
+
+	/* don't lock while sending the packets ... we therefore
+	 * copy the required data before sending */
+	batman_if = orig_node->batman_if;
+	memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+	send_raw_packet((unsigned char *)&info->packet,
+			packet_length, batman_if, dstaddr);
+	return;
+
+out:
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 /* only send one vis packet. called from send_vis_packets() */
@@ -503,6 +544,7 @@
 	unsigned long flags;
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
+
 	purge_vis_packets();
 
 	if (generate_vis_packet() == 0)
@@ -511,7 +553,11 @@
 
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
 		list_del_init(&info->send_list);
+		kref_get(&info->refcount);
+		spin_unlock_irqrestore(&vis_hash_lock, flags);
 		send_vis_packet(info);
+		spin_lock_irqsave(&vis_hash_lock, flags);
+		kref_put(&info->refcount, free_info);
 	}
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();
@@ -544,6 +590,7 @@
 	my_vis_info->first_seen = jiffies - atomic_read(&vis_interval);
 	INIT_LIST_HEAD(&my_vis_info->recv_list);
 	INIT_LIST_HEAD(&my_vis_info->send_list);
+	kref_init(&my_vis_info->refcount);
 	my_vis_info->packet.version = COMPAT_VERSION;
 	my_vis_info->packet.packet_type = BAT_VIS;
 	my_vis_info->packet.ttl = TTL;
@@ -557,9 +604,9 @@
 
 	if (hash_add(vis_hash, my_vis_info) < 0) {
 		printk(KERN_ERR
-			  "batman-adv:Can't add own vis packet into hash\n");
-		free_info(my_vis_info);	/* not in hash, need to remove it
-					 * manually. */
+		       "batman-adv:Can't add own vis packet into hash\n");
+		/* not in hash, need to remove it manually. */
+		kref_put(&my_vis_info->refcount, free_info);
 		goto err;
 	}
 
@@ -573,6 +620,13 @@
 	return 0;
 }
 
+/* Decrease the reference count on a hash item info */
+static void free_info_ref(void *data)
+{
+	struct vis_info *info = data;
+	kref_put(&info->refcount, free_info);
+}
+
 /* shutdown vis-server */
 void vis_quit(void)
 {
@@ -584,7 +638,7 @@
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
 	/* properly remove, kill timers ... */
-	hash_delete(vis_hash, free_info);
+	hash_delete(vis_hash, free_info_ref);
 	vis_hash = NULL;
 	my_vis_info = NULL;
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
@@ -594,5 +648,5 @@
 static void start_vis_timer(void)
 {
 	queue_delayed_work(bat_event_workqueue, &vis_timer_wq,
-			   (atomic_read(&vis_interval) * HZ ) / 1000);
+			   (atomic_read(&vis_interval) * HZ) / 1000);
 }
vis.h +1 -0
Index: vis.h
===================================================================
@@ -29,6 +29,7 @@
 			    /* list of server-neighbors we received a vis-packet
 			     * from.  we should not reply to them. */
 	struct list_head send_list;
+	struct kref refcount;
 	/* this packet might be part of the vis send queue. */
 	struct vis_packet packet;
 	/* vis_info may follow here*/
#18 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hi Andrew,

Yes, ehm it does indeed cause a kernel panic on the server side immediately :). See the attachment for the full call trace.

Cheers, Linus

Quoted 405 lines · Andrew Lunn

Hi Linus

Please could you try this patch and see if it fixes the vis problem. It compiles cleanly with 2.6.32. It is checkpatch clean, it is sparse clean. However, since i've not actually tried running the code i would not be too surprised if it deadlocked, leaked memory, oopsed, ...

Andrew

Staging: batman-adv: Don't have interrupts disabled while sending.

send_vis_packets() would disable interrupts before calling dev_queue_xmit() which resulting in a backtrace in local_bh_enable(). Fix this by using kref on the vis_info object so that we can call send_vis_packets() without holding vis_hash_lock. vis_hash_lock also used to protect recv_list, so we now need a new lock to protect that instead of vis_hash_lock.

Also a few checkpatch cleanups.

Index: vis.c
===================================================================

vis.c +102 -48
@@ -30,22 +30,26 @@
 
 struct hashtable_t *vis_hash;
 DEFINE_SPINLOCK(vis_hash_lock);
+static DEFINE_SPINLOCK(recv_list_lock);
 static struct vis_info *my_vis_info;
 static struct list_head send_list;	/* always locked with vis_hash_lock */
 
 static void start_vis_timer(void);
 
 /* free the info */
-static void free_info(void *data)
+static void free_info(struct kref *ref)
 {
-	struct vis_info *info = data;
+	struct vis_info *info = container_of(ref, struct vis_info, refcount);
 	struct recvlist_node *entry, *tmp;
+	unsigned long flags;
 
 	list_del_init(&info->send_list);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry_safe(entry, tmp, &info->recv_list, list) {
 		list_del(&entry->list);
 		kfree(entry);
 	}
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	kfree(info);
 }
 
@@ -147,32 +151,41 @@
 static void recv_list_add(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
+
 	entry = kmalloc(sizeof(struct recvlist_node), GFP_ATOMIC);
 	if (!entry)
 		return;
 
 	memcpy(entry->mac, mac, ETH_ALEN);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_add_tail(&entry->list, recv_list);
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 }
 
 /* returns 1 if this mac is in the recv_list */
 static int recv_list_is_in(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
 
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry(entry, recv_list, list) {
-		if (memcmp(entry->mac, mac, ETH_ALEN) == 0)
+		if (memcmp(entry->mac, mac, ETH_ALEN) == 0) {
+			spin_unlock_irqrestore(&recv_list_lock, flags);
 			return 1;
+		}
 	}
-
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	return 0;
 }
 
 /* try to add the packet to the vis_hash. return NULL if invalid (e.g. too old,
- * broken.. ).  vis hash must be locked outside.  is_new is set when the packet
+ * broken.. ).	vis hash must be locked outside.  is_new is set when the packet
  * is newer than old entries in the hash. */
 static struct vis_info *add_packet(struct vis_packet *vis_packet,
-				   int vis_info_len, int *is_new)
+				   int vis_info_len, int *is_new,
+				   int make_broadcast)
 {
 	struct vis_info *info, *old_info;
 	struct vis_info search_elem;
@@ -199,7 +212,7 @@
 		}
 		/* remove old entry */
 		hash_remove(vis_hash, old_info);
-		free_info(old_info);
+		kref_put(&old_info->refcount, free_info);
 	}
 
 	info = kmalloc(sizeof(struct vis_info) + vis_info_len, GFP_ATOMIC);
@@ -208,6 +221,7 @@
 
 	INIT_LIST_HEAD(&info->send_list);
 	INIT_LIST_HEAD(&info->recv_list);
+	kref_init(&info->refcount);
 	info->first_seen = jiffies;
 	memcpy(&info->packet, vis_packet,
 	       sizeof(struct vis_packet) + vis_info_len);
@@ -215,16 +229,21 @@
 	/* initialize and add new packet. */
 	*is_new = 1;
 
+	/* Make it a broadcast packet, if required */
+	if (make_broadcast)
+		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
+
 	/* repair if entries is longer than packet. */
 	if (info->packet.entries * sizeof(struct vis_info_entry) > vis_info_len)
-		info->packet.entries = vis_info_len / sizeof(struct vis_info_entry);
+		info->packet.entries = vis_info_len /
+			sizeof(struct vis_info_entry);
 
 	recv_list_add(&info->recv_list, info->packet.sender_orig);
 
 	/* try to add it */
 	if (hash_add(vis_hash, info) < 0) {
 		/* did not work (for some reason) */
-		free_info(info);
+		kref_put(&old_info->refcount, free_info);
 		info = NULL;
 	}
 
@@ -235,19 +254,20 @@
 void receive_server_sync_packet(struct vis_packet *vis_packet, int vis_info_len)
 {
 	struct vis_info *info;
-	int is_new;
+	int is_new, make_broadcast;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
 
+	make_broadcast = (vis_server == VIS_TYPE_SERVER_SYNC);
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, make_broadcast);
 	if (info == NULL)
 		goto end;
 
 	/* only if we are server ourselves and packet is newer than the one in
 	 * hash.*/
 	if (vis_server == VIS_TYPE_SERVER_SYNC && is_new) {
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
 		if (list_empty(&info->send_list))
 			list_add_tail(&info->send_list, &send_list);
 	}
@@ -263,24 +283,27 @@
 	int is_new;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
+	int are_target = 0;
 
 	/* clients shall not broadcast. */
 	if (is_bcast(vis_packet->target_orig))
 		return;
 
+	/* Are we the target for this VIS packet? */
+	if (vis_server == VIS_TYPE_SERVER_SYNC	&&
+	    is_my_mac(info->packet.target_orig))
+		are_target = 1;
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, are_target);
 	if (info == NULL)
 		goto end;
 	/* note that outdated packets will be dropped at this point. */
 
 
 	/* send only if we're the target server or ... */
-	if (vis_server == VIS_TYPE_SERVER_SYNC  &&
-	    is_my_mac(info->packet.target_orig) &&
-	    is_new) {
+	if (are_target && is_new) {
 		info->packet.vis_type = VIS_TYPE_SERVER_SYNC;	/* upgrade! */
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
 		if (list_empty(&info->send_list))
 			list_add_tail(&info->send_list, &send_list);
 
@@ -362,14 +385,17 @@
 	while (hash_iterate(orig_hash, &hashit_global)) {
 		orig_node = hashit_global.bucket->data;
 		if (orig_node->router != NULL
-			&& compare_orig(orig_node->router->addr, orig_node->orig)
+			&& compare_orig(orig_node->router->addr,
+					orig_node->orig)
 			&& orig_node->batman_if
 			&& (orig_node->batman_if->if_active == IF_ACTIVE)
 		    && orig_node->router->tq_avg > 0) {
 
 			/* fill one entry into buffer. */
 			entry = &entry_array[info->packet.entries];
-			memcpy(entry->src, orig_node->batman_if->net_dev->dev_addr, ETH_ALEN);
+			memcpy(entry->src,
+			       orig_node->batman_if->net_dev->dev_addr,
+			       ETH_ALEN);
 			memcpy(entry->dest, orig_node->orig, ETH_ALEN);
 			entry->quality = orig_node->router->tq_avg;
 			info->packet.entries++;
@@ -401,6 +427,8 @@
 	return 0;
 }
 
+/* free old vis packets. Must be called with this vis_hash_lock
+ * held */
 static void purge_vis_packets(void)
 {
 	HASHIT(hashit);
@@ -413,7 +441,7 @@
 		if (time_after(jiffies,
 			       info->first_seen + (VIS_TIMEOUT*HZ)/1000)) {
 			hash_remove_bucket(vis_hash, &hashit);
-			free_info(info);
+			kref_put(&info->refcount, free_info);
 		}
 	}
 }
@@ -423,6 +451,8 @@
 	HASHIT(hashit);
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 
@@ -431,46 +461,57 @@
 		orig_node = hashit.bucket->data;
 
 		/* if it's a vis server and reachable, send it. */
-		if (orig_node &&
-		    (orig_node->flags & VIS_SERVER) &&
-		    orig_node->batman_if &&
-		    orig_node->router) {
+		if ((!orig_node) || (!orig_node->batman_if) ||
+		    (!orig_node->router))
+			continue;
+		if (!(orig_node->flags & VIS_SERVER))
+			continue;
+		/* don't send it if we already received the packet from
+		 * this node. */
+		if (recv_list_is_in(&info->recv_list, orig_node->orig))
+			continue;
 
-			/* don't send it if we already received the packet from
-			 * this node. */
-			if (recv_list_is_in(&info->recv_list, orig_node->orig))
-				continue;
+		memcpy(info->packet.target_orig, orig_node->orig, ETH_ALEN);
+		batman_if = orig_node->batman_if;
+		memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+		spin_unlock_irqrestore(&orig_hash_lock, flags);
 
-			memcpy(info->packet.target_orig,
-			       orig_node->orig, ETH_ALEN);
+		send_raw_packet((unsigned char *)&info->packet,
+				packet_length, batman_if, dstaddr);
 
-			send_raw_packet((unsigned char *) &info->packet,
-					packet_length,
-					orig_node->batman_if,
-					orig_node->router->addr);
-		}
+		spin_lock_irqsave(&orig_hash_lock, flags);
+
 	}
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 static void unicast_vis_packet(struct vis_info *info, int packet_length)
 {
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 	orig_node = ((struct orig_node *)
 		     hash_find(orig_hash, info->packet.target_orig));
 
-	if ((orig_node != NULL) &&
-	    (orig_node->batman_if != NULL) &&
-	    (orig_node->router != NULL)) {
-		send_raw_packet((unsigned char *) &info->packet, packet_length,
-				orig_node->batman_if,
-				orig_node->router->addr);
-	}
+	if ((!orig_node) || (!orig_node->batman_if) || (!orig_node->router))
+		goto out;
+
+	/* don't lock while sending the packets ... we therefore
+	 * copy the required data before sending */
+	batman_if = orig_node->batman_if;
+	memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+	send_raw_packet((unsigned char *)&info->packet,
+			packet_length, batman_if, dstaddr);
+	return;
+
+out:
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 /* only send one vis packet. called from send_vis_packets() */
@@ -503,6 +544,7 @@
 	unsigned long flags;
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
+
 	purge_vis_packets();
 
 	if (generate_vis_packet() == 0)
@@ -511,7 +553,11 @@
 
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
 		list_del_init(&info->send_list);
+		kref_get(&info->refcount);
+		spin_unlock_irqrestore(&vis_hash_lock, flags);
 		send_vis_packet(info);
+		spin_lock_irqsave(&vis_hash_lock, flags);
+		kref_put(&info->refcount, free_info);
 	}
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();
@@ -544,6 +590,7 @@
 	my_vis_info->first_seen = jiffies - atomic_read(&vis_interval);
 	INIT_LIST_HEAD(&my_vis_info->recv_list);
 	INIT_LIST_HEAD(&my_vis_info->send_list);
+	kref_init(&my_vis_info->refcount);
 	my_vis_info->packet.version = COMPAT_VERSION;
 	my_vis_info->packet.packet_type = BAT_VIS;
 	my_vis_info->packet.ttl = TTL;
@@ -557,9 +604,9 @@
 
 	if (hash_add(vis_hash, my_vis_info) < 0) {
 		printk(KERN_ERR
-			  "batman-adv:Can't add own vis packet into hash\n");
-		free_info(my_vis_info);	/* not in hash, need to remove it
-					 * manually. */
+		       "batman-adv:Can't add own vis packet into hash\n");
+		/* not in hash, need to remove it manually. */
+		kref_put(&my_vis_info->refcount, free_info);
 		goto err;
 	}
 
@@ -573,6 +620,13 @@
 	return 0;
 }
 
+/* Decrease the reference count on a hash item info */
+static void free_info_ref(void *data)
+{
+	struct vis_info *info = data;
+	kref_put(&info->refcount, free_info);
+}
+
 /* shutdown vis-server */
 void vis_quit(void)
 {
@@ -584,7 +638,7 @@
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
 	/* properly remove, kill timers ... */
-	hash_delete(vis_hash, free_info);
+	hash_delete(vis_hash, free_info_ref);
 	vis_hash = NULL;
 	my_vis_info = NULL;
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
@@ -594,5 +648,5 @@
 static void start_vis_timer(void)
 {
 	queue_delayed_work(bat_event_workqueue, &vis_timer_wq,
-			   (atomic_read(&vis_interval) * HZ ) / 1000);
+			   (atomic_read(&vis_interval) * HZ) / 1000);
 }

Index: vis.h
===================================================================

vis.h line 35

Quoted 7 lines
vis.h +1 -0
@@ -29,6 +29,7 @@
 			    /* list of server-neighbors we received a vis-packet
 			     * from.  we should not reply to them. */
 	struct list_head send_list;
+	struct kref refcount;
 	/* this packet might be part of the vis send queue. */
 	struct vis_packet packet;
 	/* vis_info may follow here*/
root@OpenWrt:/# batctl vs
[ ] client mode (server disabled)
[x] server mode (server enabled)
root@OpenWrt:/# CPU 0 Unable to handle kernel paging request at virtual address 00000024, epc == 8014d58c, ra == 80410154
Oops[#1]:
Cpu 0
$ 0   : 00000000 10009c00 8070e140 00000006
$ 4   : 8070e140 00000024 00000006 00000001
$ 8   : 00000024 00000000 00000000 00000000
$12   : 00000000 7fd89ed8 00000413 00430000
$16   : 80fef500 8041cdc0 00000024 8070e000
$20   : 0000004c 80d63000 00004305 80c05040
$24   : 00000002 8061c874
$28   : 80290000 80291ca8 00000000 80410154
Hi    : 000000f5
Lo    : 15a32180
epc   : 8014d58c memcmp+0xc/0x38
    Tainted: P
ra    : 80410154 is_my_mac+0x3c/0x78 [batman_adv]
Status: 10009c03    KERNEL EXL IE
Cause : 10800008
BadVA : 00000024
PrId  : 00019064 (MIPS 4KEc)
Modules linked in: ath_ahb ath_hal(P) batman_adv ip6t_REJECT ip6t_LOG ip6t_rt ip6t_hbh ip6t_mh ip6t_ipv6header ip6t_frag ip6t_eui64 ip6t_ah ip6table_raw ip6_queue ip6table_mangle ip6table_filter ip6_tables ebt_redirect ebt_mark ebt_vlan ebt_stp ebt_pkttype ebt_mark_m ebt_limit ebt_among ebt_802_3 ebtable_nat ebtable_filter ebtable_broute ebtables xt_quota xt_pkttype xt_physdev ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables tun ipv6
Process swapper (pid: 0, threadinfo=80290000, task=80292170, tls=00000000)
Stack : 80225540 00000000 80a11c00 8022b330 00000000 00000000 80c05060 80417264
        00000000 8022a728 80000000 806c0052 c02e4000 c02e7000 80f49a40 80c05060
        80c05052 8070e000 80d63000 804124e4 80225540 00000000 80a11c00 00000000
        80c14e40 80fef500 80f49a40 80417bf4 8070e2c0 80c05052 8070e000 80d63000
        80d63000 80f49a40 8070e2c0 806215c4 00000000 80292170 80291db0 000332b6
        ...
Call Trace:
[<8014d58c>] memcmp+0xc/0x38
[<80410154>] is_my_mac+0x3c/0x78 [batman_adv]
[<80417264>] receive_client_update_packet+0x4c/0x154 [batman_adv]
[<804124e4>] recv_vis_packet+0xa4/0xbc [batman_adv]
[<80417bf4>] batman_skb_recv+0x168/0x1dc [batman_adv]
[<806215c4>] ieee80211_saveath+0xb24/0xb80 [ath_ahb]
Code: 08053567  00003821  90430000 <91020000> 00621023  14400006  24c6ffff  00871021  00a74021
Kernel panic - not syncing: Fatal exception in interrupt

Rebooting in 3 seconds..+

#19 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 4 lines · Linus L??ssing

Hi Andrew,

Yes, ehm it does indeed cause a kernel panic on the server side immediately :). See the attachment for the full call trace.

Oh yes, dumb error. I will give you a new patch tomorrow. I might even try it first this time!

Thanks for testing,

Andrew

#20 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

In reply to Andrew Lunn

Hi Linus

Here is a new version of the patch. I've tested it this time using five UML machines. It should not immediately opps now.

Please could you test it with your test setup.

Thanks
Andrew

Staging: batman-adv: Don't have interrupts disabled while sending.

send_vis_packets() would disable interrupts before calling dev_queue_xmit() which resulting in a backtrace in local_bh_enable(). Fix this by using kref on the vis_info object so that we can call send_vis_packets() without holding vis_hash_lock. vis_hash_lock also used to protect recv_list, so we now need a new lock to protect that instead of vis_hash_lock.

Also a few checkpatch cleanups.

Index: vis.c

vis.c +102 -48
Index: vis.c
===================================================================
@@ -30,22 +30,26 @@
 
 struct hashtable_t *vis_hash;
 DEFINE_SPINLOCK(vis_hash_lock);
+static DEFINE_SPINLOCK(recv_list_lock);
 static struct vis_info *my_vis_info;
 static struct list_head send_list;	/* always locked with vis_hash_lock */
 
 static void start_vis_timer(void);
 
 /* free the info */
-static void free_info(void *data)
+static void free_info(struct kref *ref)
 {
-	struct vis_info *info = data;
+	struct vis_info *info = container_of(ref, struct vis_info, refcount);
 	struct recvlist_node *entry, *tmp;
+	unsigned long flags;
 
 	list_del_init(&info->send_list);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry_safe(entry, tmp, &info->recv_list, list) {
 		list_del(&entry->list);
 		kfree(entry);
 	}
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	kfree(info);
 }
 
@@ -147,32 +151,41 @@
 static void recv_list_add(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
+
 	entry = kmalloc(sizeof(struct recvlist_node), GFP_ATOMIC);
 	if (!entry)
 		return;
 
 	memcpy(entry->mac, mac, ETH_ALEN);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_add_tail(&entry->list, recv_list);
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 }
 
 /* returns 1 if this mac is in the recv_list */
 static int recv_list_is_in(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
 
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry(entry, recv_list, list) {
-		if (memcmp(entry->mac, mac, ETH_ALEN) == 0)
+		if (memcmp(entry->mac, mac, ETH_ALEN) == 0) {
+			spin_unlock_irqrestore(&recv_list_lock, flags);
 			return 1;
+		}
 	}
-
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	return 0;
 }
 
 /* try to add the packet to the vis_hash. return NULL if invalid (e.g. too old,
- * broken.. ).  vis hash must be locked outside.  is_new is set when the packet
+ * broken.. ).	vis hash must be locked outside.  is_new is set when the packet
  * is newer than old entries in the hash. */
 static struct vis_info *add_packet(struct vis_packet *vis_packet,
-				   int vis_info_len, int *is_new)
+				   int vis_info_len, int *is_new,
+				   int make_broadcast)
 {
 	struct vis_info *info, *old_info;
 	struct vis_info search_elem;
@@ -199,7 +212,7 @@
 		}
 		/* remove old entry */
 		hash_remove(vis_hash, old_info);
-		free_info(old_info);
+		kref_put(&old_info->refcount, free_info);
 	}
 
 	info = kmalloc(sizeof(struct vis_info) + vis_info_len, GFP_ATOMIC);
@@ -208,6 +221,7 @@
 
 	INIT_LIST_HEAD(&info->send_list);
 	INIT_LIST_HEAD(&info->recv_list);
+	kref_init(&info->refcount);
 	info->first_seen = jiffies;
 	memcpy(&info->packet, vis_packet,
 	       sizeof(struct vis_packet) + vis_info_len);
@@ -215,16 +229,21 @@
 	/* initialize and add new packet. */
 	*is_new = 1;
 
+	/* Make it a broadcast packet, if required */
+	if (make_broadcast)
+		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
+
 	/* repair if entries is longer than packet. */
 	if (info->packet.entries * sizeof(struct vis_info_entry) > vis_info_len)
-		info->packet.entries = vis_info_len / sizeof(struct vis_info_entry);
+		info->packet.entries = vis_info_len /
+			sizeof(struct vis_info_entry);
 
 	recv_list_add(&info->recv_list, info->packet.sender_orig);
 
 	/* try to add it */
 	if (hash_add(vis_hash, info) < 0) {
 		/* did not work (for some reason) */
-		free_info(info);
+		kref_put(&old_info->refcount, free_info);
 		info = NULL;
 	}
 
@@ -235,19 +254,20 @@
 void receive_server_sync_packet(struct vis_packet *vis_packet, int vis_info_len)
 {
 	struct vis_info *info;
-	int is_new;
+	int is_new, make_broadcast;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
 
+	make_broadcast = (vis_server == VIS_TYPE_SERVER_SYNC);
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, make_broadcast);
 	if (info == NULL)
 		goto end;
 
 	/* only if we are server ourselves and packet is newer than the one in
 	 * hash.*/
 	if (vis_server == VIS_TYPE_SERVER_SYNC && is_new) {
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
 		if (list_empty(&info->send_list))
 			list_add_tail(&info->send_list, &send_list);
 	}
@@ -263,24 +283,27 @@
 	int is_new;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
+	int are_target = 0;
 
 	/* clients shall not broadcast. */
 	if (is_bcast(vis_packet->target_orig))
 		return;
 
+	/* Are we the target for this VIS packet? */
+	if (vis_server == VIS_TYPE_SERVER_SYNC	&&
+	    is_my_mac(vis_packet->target_orig))
+		are_target = 1;
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, are_target);
 	if (info == NULL)
 		goto end;
 	/* note that outdated packets will be dropped at this point. */
 
 
 	/* send only if we're the target server or ... */
-	if (vis_server == VIS_TYPE_SERVER_SYNC  &&
-	    is_my_mac(info->packet.target_orig) &&
-	    is_new) {
+	if (are_target && is_new) {
 		info->packet.vis_type = VIS_TYPE_SERVER_SYNC;	/* upgrade! */
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
 		if (list_empty(&info->send_list))
 			list_add_tail(&info->send_list, &send_list);
 
@@ -362,14 +385,17 @@
 	while (hash_iterate(orig_hash, &hashit_global)) {
 		orig_node = hashit_global.bucket->data;
 		if (orig_node->router != NULL
-			&& compare_orig(orig_node->router->addr, orig_node->orig)
+			&& compare_orig(orig_node->router->addr,
+					orig_node->orig)
 			&& orig_node->batman_if
 			&& (orig_node->batman_if->if_active == IF_ACTIVE)
 		    && orig_node->router->tq_avg > 0) {
 
 			/* fill one entry into buffer. */
 			entry = &entry_array[info->packet.entries];
-			memcpy(entry->src, orig_node->batman_if->net_dev->dev_addr, ETH_ALEN);
+			memcpy(entry->src,
+			       orig_node->batman_if->net_dev->dev_addr,
+			       ETH_ALEN);
 			memcpy(entry->dest, orig_node->orig, ETH_ALEN);
 			entry->quality = orig_node->router->tq_avg;
 			info->packet.entries++;
@@ -401,6 +427,8 @@
 	return 0;
 }
 
+/* free old vis packets. Must be called with this vis_hash_lock
+ * held */
 static void purge_vis_packets(void)
 {
 	HASHIT(hashit);
@@ -413,7 +441,7 @@
 		if (time_after(jiffies,
 			       info->first_seen + (VIS_TIMEOUT*HZ)/1000)) {
 			hash_remove_bucket(vis_hash, &hashit);
-			free_info(info);
+			kref_put(&info->refcount, free_info);
 		}
 	}
 }
@@ -423,6 +451,8 @@
 	HASHIT(hashit);
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 
@@ -431,46 +461,57 @@
 		orig_node = hashit.bucket->data;
 
 		/* if it's a vis server and reachable, send it. */
-		if (orig_node &&
-		    (orig_node->flags & VIS_SERVER) &&
-		    orig_node->batman_if &&
-		    orig_node->router) {
+		if ((!orig_node) || (!orig_node->batman_if) ||
+		    (!orig_node->router))
+			continue;
+		if (!(orig_node->flags & VIS_SERVER))
+			continue;
+		/* don't send it if we already received the packet from
+		 * this node. */
+		if (recv_list_is_in(&info->recv_list, orig_node->orig))
+			continue;
 
-			/* don't send it if we already received the packet from
-			 * this node. */
-			if (recv_list_is_in(&info->recv_list, orig_node->orig))
-				continue;
+		memcpy(info->packet.target_orig, orig_node->orig, ETH_ALEN);
+		batman_if = orig_node->batman_if;
+		memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+		spin_unlock_irqrestore(&orig_hash_lock, flags);
 
-			memcpy(info->packet.target_orig,
-			       orig_node->orig, ETH_ALEN);
+		send_raw_packet((unsigned char *)&info->packet,
+				packet_length, batman_if, dstaddr);
 
-			send_raw_packet((unsigned char *) &info->packet,
-					packet_length,
-					orig_node->batman_if,
-					orig_node->router->addr);
-		}
+		spin_lock_irqsave(&orig_hash_lock, flags);
+
 	}
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 static void unicast_vis_packet(struct vis_info *info, int packet_length)
 {
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 	orig_node = ((struct orig_node *)
 		     hash_find(orig_hash, info->packet.target_orig));
 
-	if ((orig_node != NULL) &&
-	    (orig_node->batman_if != NULL) &&
-	    (orig_node->router != NULL)) {
-		send_raw_packet((unsigned char *) &info->packet, packet_length,
-				orig_node->batman_if,
-				orig_node->router->addr);
-	}
+	if ((!orig_node) || (!orig_node->batman_if) || (!orig_node->router))
+		goto out;
+
+	/* don't lock while sending the packets ... we therefore
+	 * copy the required data before sending */
+	batman_if = orig_node->batman_if;
+	memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+	send_raw_packet((unsigned char *)&info->packet,
+			packet_length, batman_if, dstaddr);
+	return;
+
+out:
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 /* only send one vis packet. called from send_vis_packets() */
@@ -503,6 +544,7 @@
 	unsigned long flags;
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
+
 	purge_vis_packets();
 
 	if (generate_vis_packet() == 0)
@@ -511,7 +553,11 @@
 
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
 		list_del_init(&info->send_list);
+		kref_get(&info->refcount);
+		spin_unlock_irqrestore(&vis_hash_lock, flags);
 		send_vis_packet(info);
+		spin_lock_irqsave(&vis_hash_lock, flags);
+		kref_put(&info->refcount, free_info);
 	}
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();
@@ -544,6 +590,7 @@
 	my_vis_info->first_seen = jiffies - atomic_read(&vis_interval);
 	INIT_LIST_HEAD(&my_vis_info->recv_list);
 	INIT_LIST_HEAD(&my_vis_info->send_list);
+	kref_init(&my_vis_info->refcount);
 	my_vis_info->packet.version = COMPAT_VERSION;
 	my_vis_info->packet.packet_type = BAT_VIS;
 	my_vis_info->packet.ttl = TTL;
@@ -557,9 +604,9 @@
 
 	if (hash_add(vis_hash, my_vis_info) < 0) {
 		printk(KERN_ERR
-			  "batman-adv:Can't add own vis packet into hash\n");
-		free_info(my_vis_info);	/* not in hash, need to remove it
-					 * manually. */
+		       "batman-adv:Can't add own vis packet into hash\n");
+		/* not in hash, need to remove it manually. */
+		kref_put(&my_vis_info->refcount, free_info);
 		goto err;
 	}
 
@@ -573,6 +620,13 @@
 	return 0;
 }
 
+/* Decrease the reference count on a hash item info */
+static void free_info_ref(void *data)
+{
+	struct vis_info *info = data;
+	kref_put(&info->refcount, free_info);
+}
+
 /* shutdown vis-server */
 void vis_quit(void)
 {
@@ -584,7 +638,7 @@
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
 	/* properly remove, kill timers ... */
-	hash_delete(vis_hash, free_info);
+	hash_delete(vis_hash, free_info_ref);
 	vis_hash = NULL;
 	my_vis_info = NULL;
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
@@ -594,5 +648,5 @@
 static void start_vis_timer(void)
 {
 	queue_delayed_work(bat_event_workqueue, &vis_timer_wq,
-			   (atomic_read(&vis_interval) * HZ ) / 1000);
+			   (atomic_read(&vis_interval) * HZ) / 1000);
 }
vis.h +1 -0
Index: vis.h
===================================================================
@@ -29,6 +29,7 @@
 			    /* list of server-neighbors we received a vis-packet
 			     * from.  we should not reply to them. */
 	struct list_head send_list;
+	struct kref refcount;
 	/* this packet might be part of the vis send queue. */
 	struct vis_packet packet;
 	/* vis_info may follow here*/
#21 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 4 lines · Andrew Lunn

Hi Linus

Here is a new version of the patch. I've tested it this time using five UML machines. It should not immediately opps now.

Instead is will leak memory and crash after a while...

I will try to find the memory leak.

Andrew

#22 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hi Andrew,

Sorry, didn't have the time to try your patch any earlier, I'm right in the middle of my exams :).
Your patch already looks quite good, I couldn't reproduce any memory leaks or crashes here (tried that with three routers and 1 or 2 vis servers activated, also activating/deactivating them a lot, no problems with that). And yes, the slow-path warning has gone with your patch.
However, I'm having some weird output when connecting two routers over wifi _and_ over ethernet cable. The setup:

Before plugging in the cable:
r1-ath1 <-- wifi --> r2-ath1
------------

root@OpenWrt:~# batctl vd dot
digraph {
        "r1-ath1" -> "r2-ath1" [label="1.32"]
        "r1-ath1" -> "r1-hna" [label="HNA"]
        "r1-ath1" -> "5a:2e:1e:1f:64:6b" [label="HNA"]
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
        }
        "r2-ath1" -> "r1-ath1" [label="1.11"]
        "r2-ath1" -> "r2-hna" [label="HNA"]
        "r2-ath1" -> "82:31:95:f9:14:6f" [label="HNA"]
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
        }
}
------------
After plugging in the cable:
r1-ath1 <-- wifi --> r2-ath1 +
r1-eth0.3 <-- cable --> r2-eth0.3
------------
root@OpenWrt:~# batctl vd dot
digraph {
        "r1-ath1" -> "r2-ath1" [label="1.0"]
        "r1-ath1" -> "r2-eth0.3" [label="1.66"]
        "r1-ath1" -> "r1-hna" [label="HNA"]
        "r1-ath1" -> "5a:2e:1e:1f:64:6b" [label="HNA"]
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
                "r1-eth0.3"
        }
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
        }
        "r2-ath1" -> "r1-ath1" [label="1.0"]
        "r2-ath1" -> "r1-eth0.3" [label="1.15"]
        "r2-ath1" -> "r2-hna" [label="HNA"]
        "r2-ath1" -> "82:31:95:f9:14:6f" [label="HNA"]
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
                "r2-eth0.3"
        }
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
        }
}
root@OpenWrt:~# cat /proc/net/batman-adv/vis_data
06:22:b0:98:87:dd,TQ 04:22:b0:98:87:fa 251, HNA 00:22:b0:98:87:dd, HNA 5a:2e:1e:1f:64:6b, PRIMARY, SEC 04:22:b0:98:87:de,
06:22:b0:98:87:f9,TQ 06:22:b0:98:87:dd 255, TQ 04:22:b0:98:87:de 251, HNA 00:22:b0:98:87:f9, HNA 82:31:95:f9:14:6f, SEC 04:22:b0:98:87:fa, PRIMARY,
----------
So the second 'subgraph "cluster_r1-ath1"' is obviously
unnecessary. Also "r1-ath1" -> "r2-eth0.3" looks wrong, should be
"r1-eth0.3" -> "r2-eth0.3" instead (and the same with r2 a few
lines later).

Cheers, Linus

Quoted 12 lines · Andrew Lunn
Quoted 4 lines · Andrew Lunn

Hi Linus

Here is a new version of the patch. I've tested it this time using five UML machines. It should not immediately opps now.

Instead is will leak memory and crash after a while...

I will try to find the memory leak.

Andrew

#23 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 4 lines · Linus L??ssing

Hi Andrew,

Sorry, didn't have the time to try your patch any earlier, I'm right in the middle of my exams :).

Hi Linus

Marek told me. No problems. I remember what its like studying for exams. However, it is nice to sometimes take a break and do something totally different.

Quoted 5 lines

Your patch already looks quite good, I couldn't reproduce any memory leaks or crashes here (tried that with three routers and 1 or 2 vis servers activated, also activating/deactivating them a lot, no problems with that). And yes, the slow-path warning has gone with your patch.

Great. So we are on the right tracks.

Quoted 46 lines

However, I'm having some weird output when connecting two routers over wifi _and_ over ethernet cable. The setup:

Before plugging in the cable:
r1-ath1 <-- wifi --> r2-ath1
------------

root@OpenWrt:~# batctl vd dot
digraph {
        "r1-ath1" -> "r2-ath1" [label="1.32"]
        "r1-ath1" -> "r1-hna" [label="HNA"]
        "r1-ath1" -> "5a:2e:1e:1f:64:6b" [label="HNA"]
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
        }
        "r2-ath1" -> "r1-ath1" [label="1.11"]
        "r2-ath1" -> "r2-hna" [label="HNA"]
        "r2-ath1" -> "82:31:95:f9:14:6f" [label="HNA"]
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
        }
}
------------
After plugging in the cable:
r1-ath1 <-- wifi --> r2-ath1 +
r1-eth0.3 <-- cable --> r2-eth0.3
------------
root@OpenWrt:~# batctl vd dot
digraph {
        "r1-ath1" -> "r2-ath1" [label="1.0"]
        "r1-ath1" -> "r2-eth0.3" [label="1.66"]
        "r1-ath1" -> "r1-hna" [label="HNA"]
        "r1-ath1" -> "5a:2e:1e:1f:64:6b" [label="HNA"]
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
                "r1-eth0.3"
        }
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
        }
        "r2-ath1" -> "r1-ath1" [label="1.0"]
        "r2-ath1" -> "r1-eth0.3" [label="1.15"]
        "r2-ath1" -> "r2-hna" [label="HNA"]
        "r2-ath1" -> "82:31:95:f9:14:6f" [label="HNA"]
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
                "r2-eth0.3"
Quoted 8 lines
        }
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
        }
}
root@OpenWrt:~# cat /proc/net/batman-adv/vis_data
06:22:b0:98:87:dd,TQ 04:22:b0:98:87:fa 251, HNA 00:22:b0:98:87:dd, HNA 5a:2e:1e:1f:64:6b, PRIMARY, SEC 04:22:b0:98:87:de,
06:22:b0:98:87:f9,TQ 06:22:b0:98:87:dd 255, TQ 04:22:b0:98:87:de 251, HNA 00:22:b0:98:87:f9, HNA 82:31:95:f9:14:6f, SEC 04:22:b0:98:87:fa, PRIMARY,

Actually, this vis_data to does not map to the dot above! There are the wrong number of HNA, wrong order etc.

Here is what i think your bat-host file contains:
06:22:b0:98:87:dd r1-ath1
06:22:b0:98:87:f9 r2-ath1
00:22:b0:98:87:dd r1-hna
04:22:b0:98:87:de r1-eth0.3
00:22:b0:98:87:f9 r2-hna
04:22:b0:98:87:fa r2-eth0.3

and this is what i get, assuming i got the MAC->name mapping correct:

digraph {
	"r1-ath1" -> "r2-eth0.3" [label="1.15"]
	"r1-ath1" -> "r1-hna" [label="HNA"]
	"r1-ath1" -> "5a:2e:1e:1f:64:6b" [label="HNA"]
	subgraph "cluster_r1-ath1" {
		"r1-ath1" [peripheries=2]
	}
	subgraph "cluster_r1-ath1" {
		"r1-ath1" [peripheries=2]
		"r1-eth0.3"
	}
	"r2-ath1" -> "r1-ath1" [label="1.0"]
	"r2-ath1" -> "r1-eth0.3" [label="1.15"]
	"r2-ath1" -> "r2-hna" [label="HNA"]
	"r2-ath1" -> "82:31:95:f9:14:6f" [label="HNA"]
	subgraph "cluster_r2-ath1" {
		"r2-ath1" [peripheries=2]
		"r2-eth0.3"
	}
	subgraph "cluster_r2-ath1" {
		"r2-ath1" [peripheries=2]
	}
}

batctl parses top-to-bottom, left-to-right. It does not consolidate the PRIMARY and the SECONDARY into one cluster. It leaves DOT to do that. Hence there are two cluster statements for each cluster actually drawn.

Quoted 2 lines

So the second 'subgraph "cluster_r1-ath1"' is obviously unnecessary.

Yes, unnecessary, but makes the batctl code easier.

Also "r1-ath1" -> "r2-eth0.3" looks wrong, should be

Quoted 2 lines

"r1-eth0.3" -> "r2-eth0.3" instead (and the same with r2 a few lines later).

These comments i agree with. A wireless and a wired device should not be neighbours. We don't have any records which originate from the secondary MAC address. That is guess is the major problem here.

So, did my/Mareks patch break it, or was it broken before?

First i suggest you go back to just before Simon's patch which introduced receiving using skbufs:

http://open-mesh.org/changeset/1517

That will tell us if we need to go back further, or our patch broke it.

If you need to go back further, i would suggest just before:

http://open-mesh.org/changeset/1510

However, if it is our patch then we can chop the patch into two:

Use Mareks patch:

https://lists.open-mesh.org/pipermail/b.a.t.m.a.n/2010-January/002261.html

and

vis.c +5 -0
Index: vis.c
===================================================================
@@ -444,10 +444,15 @@
 			memcpy(info->packet.target_orig,
 			       orig_node->orig, ETH_ALEN);
 
+spin_unlock_irqrestore(&orig_hash_lock, flags);
+
 			send_raw_packet((unsigned char *) &info->packet,
 					packet_length,
 					orig_node->batman_if,
 					orig_node->router->addr);
+
+spin_lock_irqsave(&orig_hash_lock, flags);
+
 		}
 	}
 	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);

This adds a race condition, which i hope if O.K. for debugging purposes, but i hope allows the send to happen without the slowpath errors. If so, we can test Marek's part of the patch.

I'm on vacation for a week now. I will have Internet access some time, but not much.

Have fun debugging.

Andrew

#24 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 75 lines · Andrew Lunn
Quoted 4 lines · Linus L??ssing

Hi Andrew,

Sorry, didn't have the time to try your patch any earlier, I'm right in the middle of my exams :).

Hi Linus

Marek told me. No problems. I remember what its like studying for exams. However, it is nice to sometimes take a break and do something totally different.

Quoted 5 lines

Your patch already looks quite good, I couldn't reproduce any memory leaks or crashes here (tried that with three routers and 1 or 2 vis servers activated, also activating/deactivating them a lot, no problems with that). And yes, the slow-path warning has gone with your patch.

Great. So we are on the right tracks.

Quoted 46 lines

However, I'm having some weird output when connecting two routers over wifi _and_ over ethernet cable. The setup:

Before plugging in the cable:
r1-ath1 <-- wifi --> r2-ath1
------------

root@OpenWrt:~# batctl vd dot
digraph {
        "r1-ath1" -> "r2-ath1" [label="1.32"]
        "r1-ath1" -> "r1-hna" [label="HNA"]
        "r1-ath1" -> "5a:2e:1e:1f:64:6b" [label="HNA"]
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
        }
        "r2-ath1" -> "r1-ath1" [label="1.11"]
        "r2-ath1" -> "r2-hna" [label="HNA"]
        "r2-ath1" -> "82:31:95:f9:14:6f" [label="HNA"]
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
        }
}
------------
After plugging in the cable:
r1-ath1 <-- wifi --> r2-ath1 +
r1-eth0.3 <-- cable --> r2-eth0.3
------------
root@OpenWrt:~# batctl vd dot
digraph {
        "r1-ath1" -> "r2-ath1" [label="1.0"]
        "r1-ath1" -> "r2-eth0.3" [label="1.66"]
        "r1-ath1" -> "r1-hna" [label="HNA"]
        "r1-ath1" -> "5a:2e:1e:1f:64:6b" [label="HNA"]
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
                "r1-eth0.3"
        }
        subgraph "cluster_r1-ath1" {
                "r1-ath1" [peripheries=2]
        }
        "r2-ath1" -> "r1-ath1" [label="1.0"]
        "r2-ath1" -> "r1-eth0.3" [label="1.15"]
        "r2-ath1" -> "r2-hna" [label="HNA"]
        "r2-ath1" -> "82:31:95:f9:14:6f" [label="HNA"]
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
                "r2-eth0.3"
Quoted 8 lines
        }
        subgraph "cluster_r2-ath1" {
                "r2-ath1" [peripheries=2]
        }
}
root@OpenWrt:~# cat /proc/net/batman-adv/vis_data
06:22:b0:98:87:dd,TQ 04:22:b0:98:87:fa 251, HNA 00:22:b0:98:87:dd, HNA 5a:2e:1e:1f:64:6b, PRIMARY, SEC 04:22:b0:98:87:de,
06:22:b0:98:87:f9,TQ 06:22:b0:98:87:dd 255, TQ 04:22:b0:98:87:de 251, HNA 00:22:b0:98:87:f9, HNA 82:31:95:f9:14:6f, SEC 04:22:b0:98:87:fa, PRIMARY,
Quoted 2 lines

Actually, this vis_data to does not map to the dot above! There are the wrong number of HNA, wrong order etc.

Hmm, just noticed, the output also seems to be flapping between those two from time to time:
------------------

root@OpenWrt:~# cat /proc/net/batman-adv/vis
06:22:b0:98:87:dd,TQ 04:22:b0:98:87:fa 251, HNA 00:22:b0:98:87:dd, HNA f6:ae:97:b3:9a:5c, PRIMARY, SEC 04:22:b0:98:87:de,
06:22:b0:98:87:f9,TQ 04:22:b0:98:87:de 251, HNA da:3e:79:2c:d3:3e, HNA 00:22:b0:98:87:f9, PRIMARY, SEC 04:22:b0:98:87:fa,
root@OpenWrt:~# cat /proc/net/batman-adv/vis
06:22:b0:98:87:dd,TQ 04:22:b0:98:87:fa 251, HNA 00:22:b0:98:87:dd, HNA f6:ae:97:b3:9a:5c, PRIMARY, SEC 04:22:b0:98:87:de,
06:22:b0:98:87:f9,TQ 06:22:b0:98:87:dd 255, TQ 04:22:b0:98:87:de 251, HNA da:3e:79:2c:d3:3e, HNA 00:22:b0:98:87:f9, SEC 04:22:b0:98:87:fa, PRIMARY,
------------------
Quoted 10 lines

Here is what i think your bat-host file contains:
06:22:b0:98:87:dd r1-ath1
06:22:b0:98:87:f9 r2-ath1
00:22:b0:98:87:dd r1-hna
04:22:b0:98:87:de r1-eth0.3
00:22:b0:98:87:f9 r2-hna
04:22:b0:98:87:fa r2-eth0.3

and this is what i get, assuming i got the MAC->name mapping correct:

Yes, correct mapping :).

Quoted 48 lines
digraph {
	"r1-ath1" -> "r2-eth0.3" [label="1.15"]
	"r1-ath1" -> "r1-hna" [label="HNA"]
	"r1-ath1" -> "5a:2e:1e:1f:64:6b" [label="HNA"]
	subgraph "cluster_r1-ath1" {
		"r1-ath1" [peripheries=2]
	}
	subgraph "cluster_r1-ath1" {
		"r1-ath1" [peripheries=2]
		"r1-eth0.3"
	}
	"r2-ath1" -> "r1-ath1" [label="1.0"]
	"r2-ath1" -> "r1-eth0.3" [label="1.15"]
	"r2-ath1" -> "r2-hna" [label="HNA"]
	"r2-ath1" -> "82:31:95:f9:14:6f" [label="HNA"]
	subgraph "cluster_r2-ath1" {
		"r2-ath1" [peripheries=2]
		"r2-eth0.3"
	}
	subgraph "cluster_r2-ath1" {
		"r2-ath1" [peripheries=2]
	}
}

batctl parses top-to-bottom, left-to-right. It does not consolidate the PRIMARY and the SECONDARY into one cluster. It leaves DOT to do that. Hence there are two cluster statements for each cluster actually drawn.

Quoted 2 lines

So the second 'subgraph "cluster_r1-ath1"' is obviously unnecessary.

Yes, unnecessary, but makes the batctl code easier.

Also "r1-ath1" -> "r2-eth0.3" looks wrong, should be

Quoted 2 lines

"r1-eth0.3" -> "r2-eth0.3" instead (and the same with r2 a few lines later).

These comments i agree with. A wireless and a wired device should not be neighbours. We don't have any records which originate from the secondary MAC address. That is guess is the major problem here.

So, did my/Mareks patch break it, or was it broken before?

First i suggest you go back to just before Simon's patch which introduced receiving using skbufs:

Quoted 8 lines

http://open-mesh.org/changeset/1517

That will tell us if we need to go back further, or our patch broke it.

If you need to go back further, i would suggest just before:

http://open-mesh.org/changeset/1510

Okay, just checked, this got introduced with 1510 already, yes. I might have a closer look at this next week.

Cheers, Linus

#25 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

In reply to Andrew Lunn

Hello,

i can reproduce the slowpath warning in the trunk within my qemu setup. When i start a server, some of the clients show this slow path warning.

I've tried Andrews patch, and i can not see any memory leaks - it varies in a 100k window. However some of the servers crash after some time (30 minutes) without any special trigger i am aware of. Stack trace is this:

root@OpenWrt:/# ------------[ cut here ]------------
WARNING: at lib/kref.c:43 kref_get+0x18/0x30()
Hardware name:
Modules linked in: via_velocity via_rhine tg3 sis900 r8169 pcnet32 ne2k_pci 8390 e1000 e100 batman_adv 8139too 3c59x nf_nat_tftp nf_conntrack_tftp nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp ipt_MASQUERADE iptable_nat nf_nat xt_NOTRACK iptable_raw xt_state nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack pppoe pppox libphy ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables ppp_async ppp_generic slhc natsemi crc_ccitt ipv6
Pid: 570, comm: bat_events Not tainted 2.6.31.1 #32
Call Trace:
 [<c10dd608>] ? kref_get+0x18/0x30
 [<c101f55f>] ? warn_slowpath_common+0x7f/0xb0
 [<c10dd608>] ? kref_get+0x18/0x30
 [<c101f5a3>] ? warn_slowpath_null+0x13/0x20
 [<c10dd608>] ? kref_get+0x18/0x30
 [<c2ac51a2>] ? proc_vis_read_prim_sec+0x352/0x5a0 [batman_adv]
 [<c2ac4ee0>] ? proc_vis_read_prim_sec+0x90/0x5a0 [batman_adv]
 [<c102d23a>] ? worker_thread+0xca/0x150
 [<c102fd80>] ? autoremove_wake_function+0x0/0x50
 [<c102d170>] ? worker_thread+0x0/0x150
 [<c102fbc3>] ? kthread+0x73/0x90
 [<c102fb50>] ? kthread+0x0/0x90
 [<c1003813>] ? kernel_thread_helper+0x7/0x14
---[ end trace 4f770856ce7e0712 ]---
------------[ cut here ]------------
kernel BUG at mm/slub.c:2929!
invalid opcode: 0000 [#1]
last sysfs file: /sys/kernel/uevent_seqnum
Modules linked in: via_velocity via_rhine tg3 sis900 r8169 pcnet32 ne2k_pci 8390 e1000 e100 batman_adv 8139too 3c59x nf_nat_tftp nf_conntrack_tftp nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp ipt_MASQUERADE iptable_nat nf_nat xt_NOTRACK iptable_raw xt_state nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack pppoe pppox libphy ipt_REJECT xt_TCPMSS ipt_LOG xt_multiport xt_mac xt_limit iptable_mangle iptable_filter ip_tables xt_tcpudp x_tables ppp_async ppp_generic slhc natsemi crc_ccitt ipv6

Pid: 570, comm: bat_events Tainted: G W (2.6.31.1 #32)
EIP: 0060:[<c1064669>] EFLAGS: 00010046 CPU: 0
EIP is at kfree+0x49/0xc0
EAX: 00000000 EBX: c1f5dd94 ECX: 40080000 EDX: c133cba0
ESI: c2ac5540 EDI: c1f5dd80 EBP: 00000216 ESP: c0d06eec
DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process bat_events (pid: 570, ti=c0d06000 task=c18d0000 task.ti=c0d06000)

Stack:
 00000002 c1f5dd94 c1f5dd94 c2ac5540 c1f5dda4 c10dd5cf c1f5dda4 00000000
<0> c2ac538b c0d06f68 c0e0504a c1f5dd84 c1f5dd94 c1f5dd80 c1f5dd80 000022a0
<0> 00000046 c0e05024 c0e05030 00000286 00000086 c1f5dda4 0000003f 00000080
Call Trace:
 [<c2ac5540>] ? vis_init+0x150/0x1b0 [batman_adv]
 [<c10dd5cf>] ? kref_put+0x4f/0x70
 [<c2ac538b>] ? proc_vis_read_prim_sec+0x53b/0x5a0 [batman_adv]
 [<c2ac4ee0>] ? proc_vis_read_prim_sec+0x90/0x5a0 [batman_adv]
 [<c102d23a>] ? worker_thread+0xca/0x150
 [<c102fd80>] ? autoremove_wake_function+0x0/0x50
 [<c102d170>] ? worker_thread+0x0/0x150
 [<c102fbc3>] ? kthread+0x73/0x90
 [<c102fb50>] ? kthread+0x0/0x90
 [<c1003813>] ? kernel_thread_helper+0x7/0x14
Code: 00 40 a1 80 cb 2e c1 c1 ea 0c c1 e2 05 01 c2 8b 0a 89 c8 25 00 80 00 00 66 85 c0 74 05 8b 52 0c 8b 0a 84 c9 78 24 f6 c5 c0 75 09 <0f> 0b 90 8d 74 26 00 eb fe 8b 5c 24 08 89 d0 8b 74 24 0c 8b 7c
EIP: [<c1064669>] kfree+0x49/0xc0 SS:ESP 0068:c0d06eec
---[ end trace 4f770856ce7e0713 ]---

Please note that in this moment, i have not queried the vis_data file, so the proc_vis_read_prim_sec call is probably not correctly reported. Furthermore there are some more stack traces in vfs/mount and then the boxes reboot - we probably write at some bad memory position.

I've changed Andrews patch a little bit, and could not crash my 9 kvm instances when running it some hours last night. I've added kref_get() calls when the info packets get referenced from the send_list, this reference is removed with kref_put() in the send_vis_packets() after sending the packet. The disadvantage of this solution is that in a race condition, we might (unnecessarily) send out an old packet. No memory leaks as far as i could see after running some hours.

Please give it a try (patch is attached to this e-mail)!

best regards,
Simon

Quoted 12 lines · Andrew Lunn
Quoted 4 lines · Andrew Lunn

Hi Linus

Here is a new version of the patch. I've tested it this time using five UML machines. It should not immediately opps now.

Instead is will leak memory and crash after a while...

I will try to find the memory leak.

Andrew

taging: batman-adv: Don't have interrupts disabled while sending.

send_vis_packets() would disable interrupts before calling dev_queue_xmit() which resulting in a backtrace in local_bh_enable(). Fix this by using kref on the vis_info object so that we can call send_vis_packets() without holding vis_hash_lock. vis_hash_lock also used to protect recv_list, so we now need a new lock to protect that instead of vis_hash_lock.

Also a few checkpatch cleanups.

===================================================================

vis.c +116 -53
@@ -30,22 +30,26 @@
 
 struct hashtable_t *vis_hash;
 DEFINE_SPINLOCK(vis_hash_lock);
+static DEFINE_SPINLOCK(recv_list_lock);
 static struct vis_info *my_vis_info;
 static struct list_head send_list;	/* always locked with vis_hash_lock */
 
 static void start_vis_timer(void);
 
 /* free the info */
-static void free_info(void *data)
+static void free_info(struct kref *ref)
 {
-	struct vis_info *info = data;
+	struct vis_info *info = container_of(ref, struct vis_info, refcount);
 	struct recvlist_node *entry, *tmp;
+	unsigned long flags;
 
 	list_del_init(&info->send_list);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry_safe(entry, tmp, &info->recv_list, list) {
 		list_del(&entry->list);
 		kfree(entry);
 	}
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	kfree(info);
 }
 
@@ -147,32 +151,41 @@
 static void recv_list_add(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
+
 	entry = kmalloc(sizeof(struct recvlist_node), GFP_ATOMIC);
 	if (!entry)
 		return;
 
 	memcpy(entry->mac, mac, ETH_ALEN);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_add_tail(&entry->list, recv_list);
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 }
 
 /* returns 1 if this mac is in the recv_list */
 static int recv_list_is_in(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
 
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry(entry, recv_list, list) {
-		if (memcmp(entry->mac, mac, ETH_ALEN) == 0)
+		if (memcmp(entry->mac, mac, ETH_ALEN) == 0) {
+			spin_unlock_irqrestore(&recv_list_lock, flags);
 			return 1;
+		}
 	}
-
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	return 0;
 }
 
 /* try to add the packet to the vis_hash. return NULL if invalid (e.g. too old,
- * broken.. ).  vis hash must be locked outside.  is_new is set when the packet
+ * broken.. ).	vis hash must be locked outside.  is_new is set when the packet
  * is newer than old entries in the hash. */
 static struct vis_info *add_packet(struct vis_packet *vis_packet,
-				   int vis_info_len, int *is_new)
+				   int vis_info_len, int *is_new,
+				   int make_broadcast)
 {
 	struct vis_info *info, *old_info;
 	struct vis_info search_elem;
@@ -199,13 +212,14 @@
 		}
 		/* remove old entry */
 		hash_remove(vis_hash, old_info);
-		free_info(old_info);
+		kref_put(&old_info->refcount, free_info);
 	}
 
 	info = kmalloc(sizeof(struct vis_info) + vis_info_len, GFP_ATOMIC);
 	if (info == NULL)
 		return NULL;
 
+	kref_init(&info->refcount);
 	INIT_LIST_HEAD(&info->send_list);
 	INIT_LIST_HEAD(&info->recv_list);
 	info->first_seen = jiffies;
@@ -215,16 +229,21 @@
 	/* initialize and add new packet. */
 	*is_new = 1;
 
+	/* Make it a broadcast packet, if required */
+	if (make_broadcast)
+		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
+
 	/* repair if entries is longer than packet. */
 	if (info->packet.entries * sizeof(struct vis_info_entry) > vis_info_len)
-		info->packet.entries = vis_info_len / sizeof(struct vis_info_entry);
+		info->packet.entries = vis_info_len /
+			sizeof(struct vis_info_entry);
 
 	recv_list_add(&info->recv_list, info->packet.sender_orig);
 
 	/* try to add it */
 	if (hash_add(vis_hash, info) < 0) {
 		/* did not work (for some reason) */
-		free_info(info);
+		kref_put(&old_info->refcount, free_info);
 		info = NULL;
 	}
 
@@ -235,21 +254,24 @@
 void receive_server_sync_packet(struct vis_packet *vis_packet, int vis_info_len)
 {
 	struct vis_info *info;
-	int is_new;
+	int is_new, make_broadcast;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
 
+	make_broadcast = (vis_server == VIS_TYPE_SERVER_SYNC);
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, make_broadcast);
 	if (info == NULL)
 		goto end;
 
 	/* only if we are server ourselves and packet is newer than the one in
 	 * hash.*/
 	if (vis_server == VIS_TYPE_SERVER_SYNC && is_new) {
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-		if (list_empty(&info->send_list))
+		if (list_empty(&info->send_list)) {
+			kref_get(&info->refcount);
 			list_add_tail(&info->send_list, &send_list);
+		}
 	}
 end:
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
@@ -263,31 +285,38 @@
 	int is_new;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
+	int are_target = 0;
 
 	/* clients shall not broadcast. */
 	if (is_bcast(vis_packet->target_orig))
 		return;
 
+	/* Are we the target for this VIS packet? */
+	if (vis_server == VIS_TYPE_SERVER_SYNC	&&
+	    is_my_mac(vis_packet->target_orig))
+		are_target = 1;
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, are_target);
 	if (info == NULL)
 		goto end;
 	/* note that outdated packets will be dropped at this point. */
 
 
 	/* send only if we're the target server or ... */
-	if (vis_server == VIS_TYPE_SERVER_SYNC  &&
-	    is_my_mac(info->packet.target_orig) &&
-	    is_new) {
+	if (are_target && is_new) {
 		info->packet.vis_type = VIS_TYPE_SERVER_SYNC;	/* upgrade! */
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-		if (list_empty(&info->send_list))
+		if (list_empty(&info->send_list)) {
+			kref_get(&info->refcount);
 			list_add_tail(&info->send_list, &send_list);
+		}
 
 		/* ... we're not the recipient (and thus need to forward). */
 	} else if (!is_my_mac(info->packet.target_orig)) {
-		if (list_empty(&info->send_list))
+		if (list_empty(&info->send_list)) {
+			kref_get(&info->refcount);
 			list_add_tail(&info->send_list, &send_list);
+		}
 	}
 end:
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
@@ -362,14 +391,17 @@
 	while (hash_iterate(orig_hash, &hashit_global)) {
 		orig_node = hashit_global.bucket->data;
 		if (orig_node->router != NULL
-			&& compare_orig(orig_node->router->addr, orig_node->orig)
+			&& compare_orig(orig_node->router->addr,
+					orig_node->orig)
 			&& orig_node->batman_if
 			&& (orig_node->batman_if->if_active == IF_ACTIVE)
 		    && orig_node->router->tq_avg > 0) {
 
 			/* fill one entry into buffer. */
 			entry = &entry_array[info->packet.entries];
-			memcpy(entry->src, orig_node->batman_if->net_dev->dev_addr, ETH_ALEN);
+			memcpy(entry->src,
+			       orig_node->batman_if->net_dev->dev_addr,
+			       ETH_ALEN);
 			memcpy(entry->dest, orig_node->orig, ETH_ALEN);
 			entry->quality = orig_node->router->tq_avg;
 			info->packet.entries++;
@@ -401,6 +433,8 @@
 	return 0;
 }
 
+/* free old vis packets. Must be called with this vis_hash_lock
+ * held */
 static void purge_vis_packets(void)
 {
 	HASHIT(hashit);
@@ -413,7 +447,7 @@
 		if (time_after(jiffies,
 			       info->first_seen + (VIS_TIMEOUT*HZ)/1000)) {
 			hash_remove_bucket(vis_hash, &hashit);
-			free_info(info);
+			kref_put(&info->refcount, free_info);
 		}
 	}
 }
@@ -423,6 +457,8 @@
 	HASHIT(hashit);
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 
@@ -431,46 +467,57 @@
 		orig_node = hashit.bucket->data;
 
 		/* if it's a vis server and reachable, send it. */
-		if (orig_node &&
-		    (orig_node->flags & VIS_SERVER) &&
-		    orig_node->batman_if &&
-		    orig_node->router) {
+		if ((!orig_node) || (!orig_node->batman_if) ||
+		    (!orig_node->router))
+			continue;
+		if (!(orig_node->flags & VIS_SERVER))
+			continue;
+		/* don't send it if we already received the packet from
+		 * this node. */
+		if (recv_list_is_in(&info->recv_list, orig_node->orig))
+			continue;
 
-			/* don't send it if we already received the packet from
-			 * this node. */
-			if (recv_list_is_in(&info->recv_list, orig_node->orig))
-				continue;
+		memcpy(info->packet.target_orig, orig_node->orig, ETH_ALEN);
+		batman_if = orig_node->batman_if;
+		memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+		spin_unlock_irqrestore(&orig_hash_lock, flags);
 
-			memcpy(info->packet.target_orig,
-			       orig_node->orig, ETH_ALEN);
+		send_raw_packet((unsigned char *)&info->packet,
+				packet_length, batman_if, dstaddr);
 
-			send_raw_packet((unsigned char *) &info->packet,
-					packet_length,
-					orig_node->batman_if,
-					orig_node->router->addr);
-		}
+		spin_lock_irqsave(&orig_hash_lock, flags);
+
 	}
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 static void unicast_vis_packet(struct vis_info *info, int packet_length)
 {
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 	orig_node = ((struct orig_node *)
 		     hash_find(orig_hash, info->packet.target_orig));
 
-	if ((orig_node != NULL) &&
-	    (orig_node->batman_if != NULL) &&
-	    (orig_node->router != NULL)) {
-		send_raw_packet((unsigned char *) &info->packet, packet_length,
-				orig_node->batman_if,
-				orig_node->router->addr);
-	}
+	if ((!orig_node) || (!orig_node->batman_if) || (!orig_node->router))
+		goto out;
+
+	/* don't lock while sending the packets ... we therefore
+	 * copy the required data before sending */
+	batman_if = orig_node->batman_if;
+	memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+	send_raw_packet((unsigned char *)&info->packet,
+			packet_length, batman_if, dstaddr);
+	return;
+
+out:
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 /* only send one vis packet. called from send_vis_packets() */
@@ -503,15 +550,23 @@
 	unsigned long flags;
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
+
 	purge_vis_packets();
 
-	if (generate_vis_packet() == 0)
+	if (generate_vis_packet() == 0) {
 		/* schedule if generation was successful */
+		kref_get(&my_vis_info->refcount);
 		list_add_tail(&my_vis_info->send_list, &send_list);
+	}
 
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
+		spin_unlock_irqrestore(&vis_hash_lock, flags);
+
+		send_vis_packet(info);
+
+		spin_lock_irqsave(&vis_hash_lock, flags);
 		list_del_init(&info->send_list);
-		send_vis_packet(info);
+		kref_put(&info->refcount, free_info);
 	}
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();
@@ -544,6 +599,7 @@
 	my_vis_info->first_seen = jiffies - atomic_read(&vis_interval);
 	INIT_LIST_HEAD(&my_vis_info->recv_list);
 	INIT_LIST_HEAD(&my_vis_info->send_list);
+	kref_init(&my_vis_info->refcount);
 	my_vis_info->packet.version = COMPAT_VERSION;
 	my_vis_info->packet.packet_type = BAT_VIS;
 	my_vis_info->packet.ttl = TTL;
@@ -557,9 +613,9 @@
 
 	if (hash_add(vis_hash, my_vis_info) < 0) {
 		printk(KERN_ERR
-			  "batman-adv:Can't add own vis packet into hash\n");
-		free_info(my_vis_info);	/* not in hash, need to remove it
-					 * manually. */
+		       "batman-adv:Can't add own vis packet into hash\n");
+		/* not in hash, need to remove it manually. */
+		kref_put(&my_vis_info->refcount, free_info);
 		goto err;
 	}
 
@@ -573,6 +629,13 @@
 	return 0;
 }
 
+/* Decrease the reference count on a hash item info */
+static void free_info_ref(void *data)
+{
+	struct vis_info *info = data;
+	kref_put(&info->refcount, free_info);
+}
+
 /* shutdown vis-server */
 void vis_quit(void)
 {
@@ -584,7 +647,7 @@
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
 	/* properly remove, kill timers ... */
-	hash_delete(vis_hash, free_info);
+	hash_delete(vis_hash, free_info_ref);
 	vis_hash = NULL;
 	my_vis_info = NULL;
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
@@ -594,5 +657,5 @@
 static void start_vis_timer(void)
 {
 	queue_delayed_work(bat_event_workqueue, &vis_timer_wq,
-			   (atomic_read(&vis_interval) * HZ ) / 1000);
+			   (atomic_read(&vis_interval) * HZ) / 1000);
 }
vis.h +1 -0
Index: batman-adv-kernelland/vis.h
===================================================================
@@ -29,6 +29,7 @@
 			    /* list of server-neighbors we received a vis-packet
 			     * from.  we should not reply to them. */
 	struct list_head send_list;
+	struct kref refcount;
 	/* this packet might be part of the vis send queue. */
 	struct vis_packet packet;
 	/* vis_info may follow here*/
#26 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hi Simon

Quoted 16 lines
vis.c +5 -1
@@ -503,15 +550,23 @@
 	unsigned long flags;
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
+
 	purge_vis_packets();
 
-	if (generate_vis_packet() == 0)
+	if (generate_vis_packet() == 0) {
 		/* schedule if generation was successful */
+		kref_get(&my_vis_info->refcount);
 		list_add_tail(&my_vis_info->send_list, &send_list);
+	}
 
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
+		spin_unlock_irqrestore(&vis_hash_lock, flags);

vis.c line 572

Quoted 10 lines
vis.c +5 -1
+
+		send_vis_packet(info);
+
+		spin_lock_irqsave(&vis_hash_lock, flags);
 		list_del_init(&info->send_list);
-		send_vis_packet(info);
+		kref_put(&info->refcount, free_info);
 	}
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();

Although you say this works, i just looks wrong. It looks unbalanced. Where is the kref_put for my_vis_info->refcount? There is a kref_put inside what looks like a loop, but no kref_get inside the loop.

Can you explain this is more detail? Can we make it look correctly balanced?

Thanks
Andrew

#27 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hey Andrew,

all list-adds and list-removes do a kref_get or kref_put respectively, but that probably was not very clear. As soon as a refence is created (by linking into the hash or a list), we should also kref_get() it.

Please find attached a new version of this patch with separate send_list_add/remove functions which include the put/get functions. I've also added the kref_put/get call to the send_list loop again. It should look more balanced now. Tested in my 9 qemu setup again, no memory leaks or crashes but i only did a very quick test (15 minutes)

best regards,
Simon

Quoted 34 lines · Andrew Lunn

Hi Simon

Quoted 16 lines
(quoted diff) +5 -1
@@ -503,15 +550,23 @@
 	unsigned long flags;
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
+
 	purge_vis_packets();
 
-	if (generate_vis_packet() == 0)
+	if (generate_vis_packet() == 0) {
 		/* schedule if generation was successful */
+		kref_get(&my_vis_info->refcount);
 		list_add_tail(&my_vis_info->send_list, &send_list);
+	}
 
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
+		spin_unlock_irqrestore(&vis_hash_lock, flags);

Review line 572

Quoted 10 lines
(quoted diff) +5 -1
+
+		send_vis_packet(info);
+
+		spin_lock_irqsave(&vis_hash_lock, flags);
 		list_del_init(&info->send_list);
-		send_vis_packet(info);
+		kref_put(&info->refcount, free_info);
 	}
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();

Although you say this works, i just looks wrong. It looks unbalanced. Where is the kref_put for my_vis_info->refcount? There is a kref_put inside what looks like a loop, but no kref_get inside the loop.

Quoted 6 lines

Can you explain this is more detail? Can we make it look correctly balanced?

Thanks
Andrew

taging: batman-adv: Don't have interrupts disabled while sending.

send_vis_packets() would disable interrupts before calling dev_queue_xmit() which resulting in a backtrace in local_bh_enable(). Fix this by using kref on the vis_info object so that we can call send_vis_packets() without holding vis_hash_lock. vis_hash_lock also used to protect recv_list, so we now need a new lock to protect that instead of vis_hash_lock.

Also a few checkpatch cleanups.

batman-adv-kernelland/vis.c +137 -59
Index: a/batman-adv-kernelland/vis.c
===================================================================
@@ -30,22 +30,26 @@
 
 struct hashtable_t *vis_hash;
 DEFINE_SPINLOCK(vis_hash_lock);
+static DEFINE_SPINLOCK(recv_list_lock);
 static struct vis_info *my_vis_info;
 static struct list_head send_list;	/* always locked with vis_hash_lock */
 
 static void start_vis_timer(void);
 
 /* free the info */
-static void free_info(void *data)
+static void free_info(struct kref *ref)
 {
-	struct vis_info *info = data;
+	struct vis_info *info = container_of(ref, struct vis_info, refcount);
 	struct recvlist_node *entry, *tmp;
+	unsigned long flags;
 
 	list_del_init(&info->send_list);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry_safe(entry, tmp, &info->recv_list, list) {
 		list_del(&entry->list);
 		kfree(entry);
 	}
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	kfree(info);
 }
 
@@ -143,36 +147,65 @@
 	}
 }
 
+/* add the info packet to the send list, if it was not
+ * already linked in. */
+static void send_list_add(struct vis_info *info)
+{
+	if (list_empty(&info->send_list)) {
+		kref_get(&info->refcount);
+		list_add_tail(&info->send_list, &send_list);
+	}
+}
+
+/* delete the info packet from the send list, if it was
+ * linked in. */
+static void send_list_del(struct vis_info *info)
+{
+	if (!list_empty(&info->send_list)) {
+		list_del_init(&info->send_list);
+		kref_put(&info->refcount, free_info);
+	}
+}
+
 /* tries to add one entry to the receive list. */
 static void recv_list_add(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
+
 	entry = kmalloc(sizeof(struct recvlist_node), GFP_ATOMIC);
 	if (!entry)
 		return;
 
 	memcpy(entry->mac, mac, ETH_ALEN);
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_add_tail(&entry->list, recv_list);
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 }
 
 /* returns 1 if this mac is in the recv_list */
 static int recv_list_is_in(struct list_head *recv_list, char *mac)
 {
 	struct recvlist_node *entry;
+	unsigned long flags;
 
+	spin_lock_irqsave(&recv_list_lock, flags);
 	list_for_each_entry(entry, recv_list, list) {
-		if (memcmp(entry->mac, mac, ETH_ALEN) == 0)
+		if (memcmp(entry->mac, mac, ETH_ALEN) == 0) {
+			spin_unlock_irqrestore(&recv_list_lock, flags);
 			return 1;
+		}
 	}
-
+	spin_unlock_irqrestore(&recv_list_lock, flags);
 	return 0;
 }
 
 /* try to add the packet to the vis_hash. return NULL if invalid (e.g. too old,
- * broken.. ).  vis hash must be locked outside.  is_new is set when the packet
+ * broken.. ).	vis hash must be locked outside.  is_new is set when the packet
  * is newer than old entries in the hash. */
 static struct vis_info *add_packet(struct vis_packet *vis_packet,
-				   int vis_info_len, int *is_new)
+				   int vis_info_len, int *is_new,
+				   int make_broadcast)
 {
 	struct vis_info *info, *old_info;
 	struct vis_info search_elem;
@@ -199,13 +232,15 @@
 		}
 		/* remove old entry */
 		hash_remove(vis_hash, old_info);
-		free_info(old_info);
+		send_list_del(old_info);
+		kref_put(&old_info->refcount, free_info);
 	}
 
 	info = kmalloc(sizeof(struct vis_info) + vis_info_len, GFP_ATOMIC);
 	if (info == NULL)
 		return NULL;
 
+	kref_init(&info->refcount);
 	INIT_LIST_HEAD(&info->send_list);
 	INIT_LIST_HEAD(&info->recv_list);
 	info->first_seen = jiffies;
@@ -215,16 +250,21 @@
 	/* initialize and add new packet. */
 	*is_new = 1;
 
+	/* Make it a broadcast packet, if required */
+	if (make_broadcast)
+		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
+
 	/* repair if entries is longer than packet. */
 	if (info->packet.entries * sizeof(struct vis_info_entry) > vis_info_len)
-		info->packet.entries = vis_info_len / sizeof(struct vis_info_entry);
+		info->packet.entries = vis_info_len /
+			sizeof(struct vis_info_entry);
 
 	recv_list_add(&info->recv_list, info->packet.sender_orig);
 
 	/* try to add it */
 	if (hash_add(vis_hash, info) < 0) {
 		/* did not work (for some reason) */
-		free_info(info);
+		kref_put(&old_info->refcount, free_info);
 		info = NULL;
 	}
 
@@ -235,22 +275,21 @@
 void receive_server_sync_packet(struct vis_packet *vis_packet, int vis_info_len)
 {
 	struct vis_info *info;
-	int is_new;
+	int is_new, make_broadcast;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
 
+	make_broadcast = (vis_server == VIS_TYPE_SERVER_SYNC);
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, make_broadcast);
 	if (info == NULL)
 		goto end;
 
 	/* only if we are server ourselves and packet is newer than the one in
 	 * hash.*/
-	if (vis_server == VIS_TYPE_SERVER_SYNC && is_new) {
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-		if (list_empty(&info->send_list))
-			list_add_tail(&info->send_list, &send_list);
-	}
+	if (vis_server == VIS_TYPE_SERVER_SYNC && is_new)
+		send_list_add(info);
 end:
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 }
@@ -263,31 +302,32 @@
 	int is_new;
 	unsigned long flags;
 	int vis_server = atomic_read(&vis_mode);
+	int are_target = 0;
 
 	/* clients shall not broadcast. */
 	if (is_bcast(vis_packet->target_orig))
 		return;
 
+	/* Are we the target for this VIS packet? */
+	if (vis_server == VIS_TYPE_SERVER_SYNC	&&
+	    is_my_mac(vis_packet->target_orig))
+		are_target = 1;
+
 	spin_lock_irqsave(&vis_hash_lock, flags);
-	info = add_packet(vis_packet, vis_info_len, &is_new);
+	info = add_packet(vis_packet, vis_info_len, &is_new, are_target);
 	if (info == NULL)
 		goto end;
 	/* note that outdated packets will be dropped at this point. */
 
 
 	/* send only if we're the target server or ... */
-	if (vis_server == VIS_TYPE_SERVER_SYNC  &&
-	    is_my_mac(info->packet.target_orig) &&
-	    is_new) {
+	if (are_target && is_new) {
 		info->packet.vis_type = VIS_TYPE_SERVER_SYNC;	/* upgrade! */
-		memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-		if (list_empty(&info->send_list))
-			list_add_tail(&info->send_list, &send_list);
+		send_list_add(info);
 
 		/* ... we're not the recipient (and thus need to forward). */
 	} else if (!is_my_mac(info->packet.target_orig)) {
-		if (list_empty(&info->send_list))
-			list_add_tail(&info->send_list, &send_list);
+		send_list_add(info);
 	}
 end:
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
@@ -362,14 +402,17 @@
 	while (hash_iterate(orig_hash, &hashit_global)) {
 		orig_node = hashit_global.bucket->data;
 		if (orig_node->router != NULL
-			&& compare_orig(orig_node->router->addr, orig_node->orig)
+			&& compare_orig(orig_node->router->addr,
+					orig_node->orig)
 			&& orig_node->batman_if
 			&& (orig_node->batman_if->if_active == IF_ACTIVE)
 		    && orig_node->router->tq_avg > 0) {
 
 			/* fill one entry into buffer. */
 			entry = &entry_array[info->packet.entries];
-			memcpy(entry->src, orig_node->batman_if->net_dev->dev_addr, ETH_ALEN);
+			memcpy(entry->src,
+			       orig_node->batman_if->net_dev->dev_addr,
+			       ETH_ALEN);
 			memcpy(entry->dest, orig_node->orig, ETH_ALEN);
 			entry->quality = orig_node->router->tq_avg;
 			info->packet.entries++;
@@ -401,6 +444,8 @@
 	return 0;
 }
 
+/* free old vis packets. Must be called with this vis_hash_lock
+ * held */
 static void purge_vis_packets(void)
 {
 	HASHIT(hashit);
@@ -413,7 +458,8 @@
 		if (time_after(jiffies,
 			       info->first_seen + (VIS_TIMEOUT*HZ)/1000)) {
 			hash_remove_bucket(vis_hash, &hashit);
-			free_info(info);
+			send_list_del(info);
+			kref_put(&info->refcount, free_info);
 		}
 	}
 }
@@ -423,6 +469,8 @@
 	HASHIT(hashit);
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 
@@ -431,46 +479,57 @@
 		orig_node = hashit.bucket->data;
 
 		/* if it's a vis server and reachable, send it. */
-		if (orig_node &&
-		    (orig_node->flags & VIS_SERVER) &&
-		    orig_node->batman_if &&
-		    orig_node->router) {
+		if ((!orig_node) || (!orig_node->batman_if) ||
+		    (!orig_node->router))
+			continue;
+		if (!(orig_node->flags & VIS_SERVER))
+			continue;
+		/* don't send it if we already received the packet from
+		 * this node. */
+		if (recv_list_is_in(&info->recv_list, orig_node->orig))
+			continue;
 
-			/* don't send it if we already received the packet from
-			 * this node. */
-			if (recv_list_is_in(&info->recv_list, orig_node->orig))
-				continue;
+		memcpy(info->packet.target_orig, orig_node->orig, ETH_ALEN);
+		batman_if = orig_node->batman_if;
+		memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
+		spin_unlock_irqrestore(&orig_hash_lock, flags);
 
-			memcpy(info->packet.target_orig,
-			       orig_node->orig, ETH_ALEN);
+		send_raw_packet((unsigned char *)&info->packet,
+				packet_length, batman_if, dstaddr);
 
-			send_raw_packet((unsigned char *) &info->packet,
-					packet_length,
-					orig_node->batman_if,
-					orig_node->router->addr);
-		}
+		spin_lock_irqsave(&orig_hash_lock, flags);
+
 	}
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 	memcpy(info->packet.target_orig, broadcastAddr, ETH_ALEN);
-	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 static void unicast_vis_packet(struct vis_info *info, int packet_length)
 {
 	struct orig_node *orig_node;
 	unsigned long flags;
+	struct batman_if *batman_if;
+	uint8_t dstaddr[ETH_ALEN];
 
 	spin_lock_irqsave(&orig_hash_lock, flags);
 	orig_node = ((struct orig_node *)
 		     hash_find(orig_hash, info->packet.target_orig));
 
-	if ((orig_node != NULL) &&
-	    (orig_node->batman_if != NULL) &&
-	    (orig_node->router != NULL)) {
-		send_raw_packet((unsigned char *) &info->packet, packet_length,
-				orig_node->batman_if,
-				orig_node->router->addr);
-	}
+	if ((!orig_node) || (!orig_node->batman_if) || (!orig_node->router))
+		goto out;
+
+	/* don't lock while sending the packets ... we therefore
+	 * copy the required data before sending */
+	batman_if = orig_node->batman_if;
+	memcpy(dstaddr, orig_node->router->addr, ETH_ALEN);
 	spin_unlock_irqrestore(&orig_hash_lock, flags);
+
+	send_raw_packet((unsigned char *)&info->packet,
+			packet_length, batman_if, dstaddr);
+	return;
+
+out:
+	spin_unlock_irqrestore(&orig_hash_lock, flags);
 }
 
 /* only send one vis packet. called from send_vis_packets() */
@@ -503,15 +562,24 @@
 	unsigned long flags;
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
+
 	purge_vis_packets();
 
-	if (generate_vis_packet() == 0)
+	if (generate_vis_packet() == 0) {
 		/* schedule if generation was successful */
-		list_add_tail(&my_vis_info->send_list, &send_list);
+		send_list_add(my_vis_info);
+	}
 
 	list_for_each_entry_safe(info, temp, &send_list, send_list) {
-		list_del_init(&info->send_list);
+
+		kref_get(&info->refcount);
+		spin_unlock_irqrestore(&vis_hash_lock, flags);
+
 		send_vis_packet(info);
+
+		spin_lock_irqsave(&vis_hash_lock, flags);
+		send_list_del(info);
+		kref_put(&info->refcount, free_info);
 	}
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
 	start_vis_timer();
@@ -544,6 +612,7 @@
 	my_vis_info->first_seen = jiffies - atomic_read(&vis_interval);
 	INIT_LIST_HEAD(&my_vis_info->recv_list);
 	INIT_LIST_HEAD(&my_vis_info->send_list);
+	kref_init(&my_vis_info->refcount);
 	my_vis_info->packet.version = COMPAT_VERSION;
 	my_vis_info->packet.packet_type = BAT_VIS;
 	my_vis_info->packet.ttl = TTL;
@@ -557,9 +626,9 @@
 
 	if (hash_add(vis_hash, my_vis_info) < 0) {
 		printk(KERN_ERR
-			  "batman-adv:Can't add own vis packet into hash\n");
-		free_info(my_vis_info);	/* not in hash, need to remove it
-					 * manually. */
+		       "batman-adv:Can't add own vis packet into hash\n");
+		/* not in hash, need to remove it manually. */
+		kref_put(&my_vis_info->refcount, free_info);
 		goto err;
 	}
 
@@ -573,6 +642,15 @@
 	return 0;
 }
 
+/* Decrease the reference count on a hash item info */
+static void free_info_ref(void *data)
+{
+	struct vis_info *info = data;
+
+	send_list_del(info);
+	kref_put(&info->refcount, free_info);
+}
+
 /* shutdown vis-server */
 void vis_quit(void)
 {
@@ -584,7 +662,7 @@
 
 	spin_lock_irqsave(&vis_hash_lock, flags);
 	/* properly remove, kill timers ... */
-	hash_delete(vis_hash, free_info);
+	hash_delete(vis_hash, free_info_ref);
 	vis_hash = NULL;
 	my_vis_info = NULL;
 	spin_unlock_irqrestore(&vis_hash_lock, flags);
@@ -594,5 +672,5 @@
 static void start_vis_timer(void)
 {
 	queue_delayed_work(bat_event_workqueue, &vis_timer_wq,
-			   (atomic_read(&vis_interval) * HZ ) / 1000);
+			   (atomic_read(&vis_interval) * HZ) / 1000);
 }
batman-adv-kernelland/vis.h +1 -0
Index: a/batman-adv-kernelland/vis.h
===================================================================
@@ -29,6 +29,7 @@
 			    /* list of server-neighbors we received a vis-packet
 			     * from.  we should not reply to them. */
 	struct list_head send_list;
+	struct kref refcount;
 	/* this packet might be part of the vis send queue. */
 	struct vis_packet packet;
 	/* vis_info may follow here*/
#28 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 6 lines · Simon Wunderlich

Hey Andrew,

all list-adds and list-removes do a kref_get or kref_put respectively, but that probably was not very clear. As soon as a refence is created (by linking into the hash or a list), we should also kref_get() it.

Quoted 6 lines

Please find attached a new version of this patch with separate send_list_add/remove functions which include the put/get functions. I've also added the kref_put/get call to the send_list loop again. It should look more balanced now. Tested in my 9 qemu setup again, no memory leaks or crashes but i only did a very quick test (15 minutes)

Hi Simon

This looks better. Did it survive longer testing?

Andrew

#29 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hello Andrew,

no, sorry, and i won't be able to do it this week as i am at the cebit this whole week (if anyone else is there, please tell me. ;).

I would therefore like to ask you and Linus to test it as well, i will perform more tests on sunday.

Thanks,
Simon

Quoted 20 lines · Andrew Lunn
Quoted 6 lines · Simon Wunderlich

Hey Andrew,

all list-adds and list-removes do a kref_get or kref_put respectively, but that probably was not very clear. As soon as a refence is created (by linking into the hash or a list), we should also kref_get() it.

Quoted 6 lines

Please find attached a new version of this patch with separate send_list_add/remove functions which include the put/get functions. I've also added the kref_put/get call to the send_list loop again. It should look more balanced now. Tested in my 9 qemu setup again, no memory leaks or crashes but i only did a very quick test (15 minutes)

Hi Simon

This looks better. Did it survive longer testing?

Andrew

#30 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Hello Simon -

I'll be in hall 13 at the booth of Atcom E42 to present the Mesh-Potato router. (With B.A.T.M.A.N. inside, of course) Will arrive tomorrow afternoon and be there until Saturday 18:00.

Cheers,
Elektra

Quoted 2 lines

no, sorry, and i won't be able to do it this week as i am at the cebit this whole week (if anyone else is there, please tell me. ;).

#31 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

In reply to Simon Wunderlich

Hi Simon and Andrew,

I already tried both Andrew's intial and Simon's latest patch yesterday and had both of them running for about 1h+, also with alternate vis-intervals (1s, 50ms, 30s) but so far I wasn't able to reproduce any crashes or leaks on two Debian stable VMs connected with each other. I'll run Andrew's patch version again this night for some more hours to make sure that I'm able to reproduce the crash at all. Any other hints for reproducing it more quickly welcome :).

Cheers, Linus

PS: With how many nodes and what kind of systems could you reproduce the crash? Were all those nodes connected directly with each other? How many vis-servers were running?

Quoted 32 lines · Simon Wunderlich

Hello Andrew,

no, sorry, and i won't be able to do it this week as i am at the cebit this whole week (if anyone else is there, please tell me. ;).

I would therefore like to ask you and Linus to test it as well, i will perform more tests on sunday.

Thanks,
Simon

Quoted 20 lines · Andrew Lunn
Quoted 6 lines · Simon Wunderlich

Hey Andrew,

all list-adds and list-removes do a kref_get or kref_put respectively, but that probably was not very clear. As soon as a refence is created (by linking into the hash or a list), we should also kref_get() it.

Quoted 6 lines

Please find attached a new version of this patch with separate send_list_add/remove functions which include the put/get functions. I've also added the kref_put/get call to the send_list loop again. It should look more balanced now. Tested in my 9 qemu setup again, no memory leaks or crashes but i only did a very quick test (15 minutes)

Hi Simon

This looks better. Did it survive longer testing?

Andrew

#32 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

I've been running 3 qemu x86 instances with Debian stable here (2.6.26 kernel) as well as 3 Atheros ap61 / mips based routers with OpenWRT (rev. 19468, 2.6.30.10 kernel) for 22 hours. None of them crashed with Andrew's patch... so, sorry, still can't reproduce the bug you're talking about.

Cheers, Linus

#33 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 5 lines · Linus L??ssing

I've been running 3 qemu x86 instances with Debian stable here (2.6.26 kernel) as well as 3 Atheros ap61 / mips based routers with OpenWRT (rev. 19468, 2.6.30.10 kernel) for 22 hours. None of them crashed with Andrew's patch... so, sorry, still can't reproduce the bug you're talking about.

If you cannot reproduce it, its probably fixed :-)

Marek, could you commit the patch.

Thanks
Andrew

#34 lore raw

Re: [B.A.T.M.A.N.] slowpath warning

Quoted 10 lines · On Thursday 04 March 2010 16:57:24 Andrew Lunn wrote:
Quoted 5 lines · Linus L??ssing

I've been running 3 qemu x86 instances with Debian stable here (2.6.26 kernel) as well as 3 Atheros ap61 / mips based routers with OpenWRT (rev. 19468, 2.6.30.10 kernel) for 22 hours. None of them crashed with Andrew's patch... so, sorry, still can't reproduce the bug you're talking about.

If you cannot reproduce it, its probably fixed :-)

Marek, could you commit the patch.

Sure but which one ? I have to admit I lost the overview (there were too many patches flying around) .. :-)

Cheers,
Marek