Skip to contentSkip to conversation

mad mad batman ...

B.A.T.M.A.N. Nicolás Echániz 4 participants 8 messages Oct 13, 2013 – Nov 26, 2013

lore
Share & tools
lore
#0 lore raw

While I'm still in Europe I've observed that the network in Quintana has started performing very poorly today. It was working perfectly fine until yesterday.

The logs on every router have started showing entries like these:

Oct 13 18:09:43 frigorifico kern.warn kernel: [12018.150000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.040000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.040000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.550000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.550000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.570000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.580000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:46 frigorifico kern.warn kernel: [12021.040000] br-lan:

received packet on bat0 with own address as source address

As you can see there are many per second.

I've pasted a bit of batctl ll batman; batctl log here:

http://pastebin.com/HX1LBPK4

...it's only showing the "originator packet from myself" lines and one line before. (the sample is less than 5 secs of logs)

Every node I checked is showing the same.

Last time this happened it was due to a router that had been affected by a nearby lightning bolt. The switch went crazy.
It took a while to detect it and the network was 15 nodes big. Now it's 40 and we are quite far away :)

If anyone has an idea of how to better test where the problem is originated, I'll be glad to hear it. Also if any batman devel wishes to log in to the net to check first hand, just let me know.

Cheers!
Nico

PS: batman version is 2012.4

#1 lore raw

back in Quintana... this problem is still showing in every node. The network is unstable and so it's difficult to debug. If anyone has a clue as to where to look for the origin I'll be glad to read your thoughts.

cheers,
Nico

Quoted 54 lines · Nicolás Echániz

While I'm still in Europe I've observed that the network in Quintana has started performing very poorly today. It was working perfectly fine until yesterday.

The logs on every router have started showing entries like these:

Oct 13 18:09:43 frigorifico kern.warn kernel: [12018.150000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.040000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.040000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.550000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.550000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.570000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.580000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:46 frigorifico kern.warn kernel: [12021.040000] br-lan:

received packet on bat0 with own address as source address

As you can see there are many per second.

I've pasted a bit of batctl ll batman; batctl log here:

http://pastebin.com/HX1LBPK4

...it's only showing the "originator packet from myself" lines and one line before. (the sample is less than 5 secs of logs)

Every node I checked is showing the same.

Last time this happened it was due to a router that had been affected by a nearby lightning bolt. The switch went crazy.
It took a while to detect it and the network was 15 nodes big. Now it's 40 and we are quite far away :)

If anyone has an idea of how to better test where the problem is originated, I'll be glad to hear it. Also if any batman devel wishes to log in to the net to check first hand, just let me know.

Cheers!
Nico

PS: batman version is 2012.4

#2 lore raw

Hi Nico,

I have no real clue, but is it possible that there is a loop somewhere? I imagine you have checked already..but I can't come with something more useful at the moment..

Cheers,

Quoted 63 lines · Nicolás Echániz

back in Quintana... this problem is still showing in every node. The network is unstable and so it's difficult to debug. If anyone has a clue as to where to look for the origin I'll be glad to read your thoughts.

cheers,
Nico

Quoted 54 lines · Nicolás Echániz

While I'm still in Europe I've observed that the network in Quintana has started performing very poorly today. It was working perfectly fine until yesterday.

The logs on every router have started showing entries like these:

Oct 13 18:09:43 frigorifico kern.warn kernel: [12018.150000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.040000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.040000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.550000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.550000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.570000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:45 frigorifico kern.warn kernel: [12020.580000] br-lan:
received packet on bat0 with own address as source address
Oct 13 18:09:46 frigorifico kern.warn kernel: [12021.040000] br-lan:

received packet on bat0 with own address as source address

As you can see there are many per second.

I've pasted a bit of batctl ll batman; batctl log here:

http://pastebin.com/HX1LBPK4

...it's only showing the "originator packet from myself" lines and one line before. (the sample is less than 5 secs of logs)

Every node I checked is showing the same.

Last time this happened it was due to a router that had been affected by a nearby lightning bolt. The switch went crazy.
It took a while to detect it and the network was 15 nodes big. Now it's 40 and we are quite far away :)

If anyone has an idea of how to better test where the problem is originated, I'll be glad to hear it. Also if any batman devel wishes to log in to the net to check first hand, just let me know.

Cheers!
Nico

PS: batman version is 2012.4

Signature · 1 line
Antonio Quartulli
#3 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 5 lines · Antonio Quartulli

Hi Nico,

I have no real clue, but is it possible that there is a loop somewhere? I imagine you have checked already..but I can't come with something more useful at the moment..

Ok, I've spent the afternoon turning off semi-inaccessible nodes one by one until I found the one causing the problem.

It's installed on a public lighting post, so it may take a while to take it down for inspection.

I don't know if you guys remember I had brought to the battlemesh a crazy node (nicknamed Jocker), that started misbehaving after a lightning bolt hit nearby. The symptom was the same I observed now:
every node in the net would start repeatedly showing the message:
"received packet on bat0 with own address as source address".

I was in Europe during the time this second node started behaving like this so I still don't know much about the moment it started.

Do you think this matter could be addressed at batman level somehow? In a 50 node network this is already quite difficult to diagnose. I can't imagine how a bigger network where no single person has remote access to every node would coordinate to isolate the problematic router...

If you are interested in looking at this first hand we can try to set up an isolated test-bed with IPv6 connectivity for you to log in and play around.

Am I the only one who has bumped into this (twice)?

cheers.

Quoted 62 lines · Nicolás Echániz
Quoted 61 lines

back in Quintana... this problem is still showing in every node. The network is unstable and so it's difficult to debug. If anyone has a clue as to where to look for the origin I'll be glad to read your thoughts.

cheers, Nico

Quoted 51 lines · Nicolás Echániz

While I'm still in Europe I've observed that the network in Quintana has started performing very poorly today. It was working perfectly fine until yesterday.

The logs on every router have started showing entries like these:

Oct 13 18:09:43 frigorifico kern.warn kernel: [12018.150000]
br-lan: received packet on bat0 with own address as source
address Oct 13 18:09:45 frigorifico kern.warn kernel:
[12020.040000] br-lan: received packet on bat0 with own address
as source address Oct 13 18:09:45 frigorifico kern.warn kernel:
[12020.040000] br-lan: received packet on bat0 with own address
as source address Oct 13 18:09:45 frigorifico kern.warn kernel:
[12020.550000] br-lan: received packet on bat0 with own address
as source address Oct 13 18:09:45 frigorifico kern.warn kernel:
[12020.550000] br-lan: received packet on bat0 with own address
as source address Oct 13 18:09:45 frigorifico kern.warn kernel:
[12020.570000] br-lan: received packet on bat0 with own address
as source address Oct 13 18:09:45 frigorifico kern.warn kernel:
[12020.580000] br-lan: received packet on bat0 with own address
as source address Oct 13 18:09:46 frigorifico kern.warn kernel:
[12021.040000] br-lan: received packet on bat0 with own address

as source address

As you can see there are many per second.

I've pasted a bit of batctl ll batman; batctl log here:

http://pastebin.com/HX1LBPK4

...it's only showing the "originator packet from myself" lines and one line before. (the sample is less than 5 secs of logs)

Every node I checked is showing the same.

Last time this happened it was due to a router that had been affected by a nearby lightning bolt. The switch went crazy. It took a while to detect it and the network was 15 nodes big. Now it's 40 and we are quite far away :)

If anyone has an idea of how to better test where the problem is originated, I'll be glad to hear it. Also if any batman devel wishes to log in to the net to check first hand, just let me know.

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines

Cheers! Nico

PS: batman version is 2012.4

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iQEcBAEBAgAGBQJSgs47AAoJEPHORz17N0TF/BsH/0R+d/k10hrPDOb6zX5rwkZJ oSbqjrBttmBRAJFGJ2hywFXm0nnL8Teo696Ulrf8wuh9wrl6aR9MD/fsrKdyfPM+ /+MTKGb6uak20Kb5pbUY+8/xl+ZmtmWkXlolEdT/PFsFFh+Ap4EiMgQIDr64EXZq IF6AkFFuNv/Oovc3COnlY+gWhxton+DRNKoHpZw37h46jTH6zSy0MATIIGKueS44 2zVpuEgbkyUVvyJYKhBt9GjX5SGQ4TVBMX0EnZuN/bEk2al64EiEOySjUYvf3xTD iqRWbvyloPZi5mhgEI876gt5qTDXmcs6YjEsvpOU+YiZoawtIAI1J8vo3kUzdBs= =APtL
-----END PGP SIGNATURE-----

#4 lore raw
Quoted 1 line · Nicolás Echániz

Am I the only one who has bumped into this (twice)?

I have also seen a lot of these messages with an indoor mesh, so no lightning involved 8-) but with v2013.04 this is gone. (same network).

bye, bastian

#5 lore raw
Quoted 6 lines · Bastian Bittorf
Quoted 1 line · Nicolás Echániz

Am I the only one who has bumped into this (twice)?

I have also seen a lot of these messages with an indoor mesh, so no lightning involved 8-) but with v2013.04 this is gone. (same network).

this message is the symptom of a loop. The causes can be gazillions.

Cheers,

Signature · 1 line
Antonio Quartulli
#6 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 11 lines · Antonio Quartulli
Quoted 7 lines · Bastian Bittorf

* Nicolás Echániz <[email protected]> [13.11.2013 08:59]:

Quoted 1 line

Am I the only one who has bumped into this (twice)?

I have also seen a lot of these messages with an indoor mesh, so no lightning involved 8-) but with v2013.04 this is gone. (same network).

this message is the symptom of a loop. The causes can be gazillions.

Well... it took about a week to finally find the node creating this problem. As before, it's failing hardware that caused the issue.

When this happens every node in the net is repeatedly showing that message. It is not the same with any "loop symptom" I believe... At least I've never seen this happen on every node being caused by something else.

I really would like to find out more about how this condition comes to happen and how to diagnose and prevent it. The whole batman-adv cloud dies when this happens and it's a pain in the ass to "debug".

All the failing routers are WR842ND. There are many more of the same model working just fine.

I now have three routers which produce this symptom, so if anyone who can understand the problem better is willing to test, I can set up a dedicated mini-test-bed.

Cheers,
NicoEchániz

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Icedove - http://www.enigmail.net/

iQEcBAEBAgAGBQJSlBvtAAoJEPHORz17N0TFf+sH/jC2qGbx3SpHZSNGgUoQrD/f J9n+lUMugbsM7uOodBnU+iiswpOrNVn4NsddWNYXFmydpbIbnO6wuUTlDZu3wNig gOMNDZqaze5fcdopwgcEuxvp1S5QRiAe2DN+ci0jQ3OGdOjuZgZJjpRxARDbSExT guF4TgB4izZUy5/5OTg4pJdNZ8zUsMap45Gy4xzslP58EZUBKkeqol+kaWuiq4eX FbUpezTpeTXlUi401pOzvrVMg7/JXhN7JULMgjPPR6y8359tfZFFQq4JwuYNSwz1 Fg6L3/kJSKm4DF7MyYa2Id/3DJx7NLZk1B7RQR05ISmuvGLDBqobqv6EY5a+Z0M= =0BD+
-----END PGP SIGNATURE-----

#7 lore raw

Hi Nico,

Quoted 10 lines · Nicolás Echániz
Quoted 9 lines · Antonio Quartulli
Quoted 7 lines · Bastian Bittorf

* Nicolás Echániz <[email protected]> [13.11.2013 08:59]:

Quoted 1 line

Am I the only one who has bumped into this (twice)?

I have also seen a lot of these messages with an indoor mesh, so no lightning involved 8-) but with v2013.04 this is gone. (same network).

Quoted 5 lines
Quoted 2 lines

this message is the symptom of a loop. The causes can be gazillions.

Well... it took about a week to finally find the node creating this problem. As before, it's failing hardware that caused the issue.

Interesting. Could you be more specific in which way the hardware fails? Does it reboot frequently? Does it send broken OGM packets?

Could you make a checksum of the flashed squashfs, does it differ from the one you've built?

Quoted 6 lines

When this happens every node in the net is repeatedly showing that message. It is not the same with any "loop symptom" I believe... At least I've never seen this happen on every node being caused by something else.

Quoted 6 lines

I really would like to find out more about how this condition comes to happen and how to diagnose and prevent it. The whole batman-adv cloud dies when this happens and it's a pain in the ass to "debug".

All the failing routers are WR842ND. There are many more of the same model working just fine.

We are also using quite a lot of 842NDs, 841NDs and 3600NDs, as well as some 741ND, 1043ND and 4300NDs. We've never had the issue of one broken node taking down the whole network yet, not in Hamburg, Kiel or Lübeck.

Would be interesting to figure out the differences between our setups. Maybe I missed it so far, did you say you were using bridge loop avoidance (we don't)? We are using batman-adv 2013.1.0 mostly with a few still on 2012.4.0 and some on 2013.4.0.

Quoted 12 lines

I now have three routers which produce this symptom, so if anyone who can understand the problem better is willing to test, I can set up a dedicated mini-test-bed.

Cheers,
NicoEchániz

Cheers, Linus