Skip to contentSkip to conversation

how well does batman-adv scale

B.A.T.M.A.N. fuumind 5 participants 10 messages Apr 26, 2017 – May 2, 2017

lore
Share & tools
lore

fuumind

author
#0 lore raw

Hi list!

Been lurking for almost a year on the battlemesh list and recently joined here as well.

I'm curious about how well batman-adv scales. Would a network of 10 000 nodes work well? What about 100 000 nodes or 1 000 000?

Thanks!
fuumind

#1 lore raw

i think this could scale , but you will need sufficient power / ethernet capacities at the nodes.

and the biggest problem may be that these endpoints make much noise on the layer2 level.
(disovery protocolls and stuff like this)

you could easily imagine what would happen if you have a layer2 switch with 100000 ports.

i think that most of the time you will not do this, and implement some sort of routing between some clouds of batman-adv networks.

Quoted 11 lines · fuumind

Hi list!

Been lurking for almost a year on the battlemesh list and recently joined here as well.

I'm curious about how well batman-adv scales. Would a network of 10 000 nodes work well? What about 100 000 nodes or 1 000 000?

Thanks!
fuumind

Signature · 1 line
make the world nicer, please use PGP encryption
#2 lore raw

I can't see how it could scale. Each node needs a table of all other nodes to calculate the cost for each path. This is a LOT of data and each node rebroadcasts it's neighbor's information so the more neighbors, the more rebroadcasting.

Maybe if you had a long string of nodes with each node only seeing 1-2 other nodes, it might scale up higher, but that's not the point of mesh networking.

It really means that under idea circumstances, overhead is going to scale up with node count by percent. So a few nodes is low overhead, but 50 nodes might be as much as 50% and so on.

This is a critical flaw in batman-adv to scale up. Requiring each node to know all other nodes. If this rule could be changed to have batman-adv only keep track of gateway nodes and nodes in the path to a gateway, then it could scale vastly further.

Quoted 31 lines · jens

i think this could scale , but you will need sufficient power / ethernet capacities at the nodes.

and the biggest problem may be that these endpoints make much noise on the layer2 level.
(disovery protocolls and stuff like this)

you could easily imagine what would happen if you have a layer2 switch with 100000 ports.

i think that most of the time you will not do this, and implement some sort of routing between some clouds of batman-adv networks.

Quoted 11 lines · fuumind

Hi list!

Been lurking for almost a year on the battlemesh list and recently joined here as well.

I'm curious about how well batman-adv scales. Would a network of 10 000 nodes work well? What about 100 000 nodes or 1 000 000?

Thanks!
fuumind

--
make the world nicer, please use PGP encryption

author
#3 lore raw
In reply to jens

Hi Jens!

By 'sufficient power', do you mean processing power to handle overhead traffic? I'm imagining a network which primarily connects WLANs.

Clouds of batman-adv networks sounds like a good idea. Any idea about how to implement the interfaces between these clouds? How big should they be allowed to grow before forming a new cloud?

fuumind

ons 2017-04-26 klockan 17:10 +0200 skrev jens:

Quoted 33 lines

i think this could scale , but you will need sufficient power / ethernet
capacities at the nodes.

and the biggest problem may be that these endpoints make much noise on
the layer2 level.
(disovery protocolls and stuff like this)

you could easily imagine what would happen if you have a layer2 switch
with 100000 ports.

i think that most of the time you will not do this, and implement some
sort of routing between some clouds of batman-adv networks.

Quoted 13 lines · fuumind

Hi list!

Been lurking for almost a year on the battlemesh list and recently joined here as well.

I'm curious about how well batman-adv scales. Would a network of 10 000
nodes work well? What about 100 000 nodes or 1 000 000?

Thanks!
fuumind

#4 lore raw

I don't think it's CPU that's the issue, it's WLAN overhead. Too many nodes is too much overhead traffic.

The web is telling me there is an upper limit of a 'basic' cluster of about 300, and if you add some multicast filtering you can hit maybe 1500. OGM's are eventually going to saturate the network though.

That makes me think you should be looking at small clouds and some other 'routing' protocol to connect those clusters together. Maybe do an OSPF ring linking up the batman-adv clouds.

Quoted 46 lines · fuumind

Hi Jens!

By 'sufficient power', do you mean processing power to handle overhead traffic? I'm imagining a network which primarily connects WLANs.

Clouds of batman-adv networks sounds like a good idea. Any idea about how to implement the interfaces between these clouds? How big should they be allowed to grow before forming a new cloud?

fuumind

ons 2017-04-26 klockan 17:10 +0200 skrev jens:

Quoted 33 lines

i think this could scale , but you will need sufficient power / ethernet
capacities at the nodes.

and the biggest problem may be that these endpoints make much noise on
the layer2 level.
(disovery protocolls and stuff like this)

you could easily imagine what would happen if you have a layer2 switch
with 100000 ports.

i think that most of the time you will not do this, and implement some
sort of routing between some clouds of batman-adv networks.

Quoted 13 lines · fuumind

Hi list!

Been lurking for almost a year on the battlemesh list and recently joined here as well.

I'm curious about how well batman-adv scales. Would a network of 10 000
nodes work well? What about 100 000 nodes or 1 000 000?

Thanks!
fuumind

#5 lore raw
In reply to fuumind

Hi,

"works well" depends much on your scenario.

I've seen many well-working networks with 100-300 nodes. There are Freifunk community networks with over 1000 nodes running batman-adv (i.e. "standard" users like laptops and smartphones), but they employ a lot of filtering to avoid too much broadcast.

Unless you have special restrictions like minimal broadcast traffic or high capacity on the wireless links, memory and CPU, I would not recommend to plan for 10 000 nodes and beyond.

Cheers,
Simon

Quoted 10 lines · CEST fuumind

Hi list!

Been lurking for almost a year on the battlemesh list and recently joined here as well.

I'm curious about how well batman-adv scales. Would a network of 10 000 nodes work well? What about 100 000 nodes or 1 000 000?

Thanks!
fuumind

author
#6 lore raw

Ok, so clouds of a few hundred nodes would work reasonably well and at the same time keep customization to a minimum.

What makes OSPF a better candidate for interfacing between the clouds? Does it keep a more limited routing table and thus minimize overhead traffic? I haven't been able to figure it out good enough yet.

fuumind

ons 2017-04-26 klockan 18:05 +0200 skrev Simon Wunderlich:

Quoted 34 lines

Hi,

"works well" depends much on  your scenario.

I've seen many well-working networks with 100-300 nodes. There are Freifunk
community networks with over 1000 nodes running batman-adv (i.e. "standard"
users like laptops and smartphones), but they employ a lot of filtering to
avoid too much broadcast.

Unless you have special restrictions like minimal broadcast traffic or high
capacity on the wireless links, memory and CPU, I would not recommend to plan
for 10 000 nodes and beyond.

Cheers,
     Simon

Quoted 12 lines · CEST fuumind

Hi list!

Been lurking for almost a year on the battlemesh list and recently joined here as well.

I'm curious about how well batman-adv scales. Would a network of 10 000
nodes work well? What about 100 000 nodes or 1 000 000?

Thanks!
fuumind

#7 lore raw

Hi,

Quoted 2 lines · CEST fuumind

Ok, so clouds of a few hundred nodes would work reasonably well and at the same time keep customization to a minimum.

Yes

Quoted 4 lines

What makes OSPF a better candidate for interfacing between the clouds? Does it keep a more limited routing table and thus minimize overhead traffic? I haven't been able to figure it out good enough yet.

The idea would be that you do a layer 3 protocol only on a few selected "gateway" nodes of the clouds. Those gateways would forward traffic from one cloud to the other, without having the details (and thus not having the overhead) of the state of the mesh.

Which technology is the best is beyond my knowledge.

Cheers,
Simon

author
#8 lore raw

Now I get it. :) That makes sense.

Thanks!

fre 2017-04-28 klockan 14:23 +0200 skrev Simon Wunderlich:

Quoted 29 lines

Hi,

Quoted 4 lines · CEST fuumind

Ok, so clouds of a few hundred nodes would work reasonably well and at
the same time keep customization to a minimum.

Yes

Quoted 7 lines

What makes OSPF a better candidate for interfacing between the clouds?
Does it keep a more limited routing table and thus minimize overhead
traffic? I haven't been able to figure it out good enough yet.

The idea would be that you do a layer 3 protocol only on a few selected
"gateway" nodes of the clouds. Those gateways would forward traffic from one
cloud to the other, without having the details (and thus not having the
overhead) of the state of the mesh.

Which technology is the best is beyond my knowledge.

Cheers,
      Simon

#9 lore raw
In reply to Simon Wunderlich
Quoted 8 lines · Simon Wunderlich
Quoted 2 lines · CEST fuumind

Ok, so clouds of a few hundred nodes would work reasonably well and at the same time keep customization to a minimum.

Yes

Quoted 3 lines

What makes OSPF a better candidate for interfacing between the clouds? Does it keep a more limited routing table and thus minimize overhead traffic? I haven't been able to figure it out good enough yet.

Which technology is the best is beyond my knowledge.

We (Freifunk Stuttgart) use a setup similar to this.
A split in different Layer3 Routingdomains had been necessary, as the batman-management traffic reached 1mbit/s at about 300 batman nodes. Geographically we depend on vpn-links to servers for our network, so we rarely have longer lines that could aggregate some information. The local uplinks are often consumer internet lines als slow as 16mbit/s down- and 1mbit/s upstream.
Our different batman-segments are distributed between several core gateways. These gateway are linked together via a backbone vpn, no batman there, just routing within that vpn.
Our first idea had been tinc vpn (meshing vpn) + OSPF, however, tinc mesh and ospf don't play nice, we had a really unstable network. In my eyes, ospf needs a stable network, some vpn between some internet servers as a virtual switch is not good enough.
Atm we use tinc + tinc internal routing. This works well but is somewhat unflexible when it comes to new networks.
In future we will try and convert to p2p vpn links (probably openvpn) between the gateways and use bgp for the routing stuff.

Regards,
Adrian

Signature · 4 lines
LiHAS - Adrian Reyer - Hessenwiesenstraße 10 - D-70565 Stuttgart
Fon: +49 (7 11) 78 28 50 90 - Fax:  +49 (7 11) 78 28 50 91
Mail: [email protected] - Web: http://lihas.de
Linux, Netzwerke, Consulting & Support - USt-ID: DE 227 816 626 Stuttgart