0.3.1 rv1152 - some test results

Hello,

thought I'd just let you know what I experienced when testing rv1152 on an outdoor network with atheroses and broadcoms, olsr in parallel. Interfaces are started with
ifconfig eth1:1 10.4.2.29 netmask 255.255.0.0 broadcast 10.4.255.255

The batman test area looks something like this: http://preveli.gr/mesh/bat-228-2b.gif

-1- gateway:
a gw is started with batmand -s 10.4.2.50 -g 2mbit/256kbit eth1:1 the client 2.29 with batmand -s 10.4.2.50 -a 10.2.29.0/24 -r 2 ath0:1

When pinging the gateway through the client node or when accessing the internet through the node it logs

Nov 26 22:52:04 (none) kern.err batmand[1287]: Error - got packet from unknown client: 10.4.2.29 (virtual ip 10.2.29.136)

The tunnel itself is flickering, this happens with -r1,2,3 and -p

client log:
Gateway client - gateway (10.4.2.2) says: IP (169.254.0.1) is expired Deleting default route via gate0 (table 68)
Adding default route to 10.4.2.2 (gw_flags: 40, tq: 178, gw_product: 0) Error - couldn't create tunnel: old tunnel is still active Adding default route to 10.4.2.2 (gw_flags: 40, tq: 179, gw_product: 0) Gateway client - got IP (169.254.0.1) from gateway: 10.4.2.2 Adding default route via gate0 (table 68)
Gateway client - gateway (10.4.2.2) says: IP (169.254.0.1) is expired Deleting default route via gate0 (table 68)
Adding default route to 10.4.2.2 (gw_flags: 40, tq: 181, gw_product: 0) Gateway client - got IP (169.254.0.1) from gateway: 10.4.2.2 Adding default route via gate0 (table 68)

gateway log:
Gateway - assigned 169.254.0.1 to client: 10.4.2.29 Gateway - assigned 169.254.0.1 to client: 10.4.2.29 Gateway - assigned 169.254.0.1 to client: 10.4.2.29

-?- What is batgat for, does it disannounce a gateway which in fact is down e.g due to dsl-failure ?

-2- together with olsrd.
All olsr nodes are on the subnet 192.168.x.x, all batman interfaces are aliases except on fonera 2.29 which is batman only.

From a gateway node running olsrd in parallel the route to an announced network drops out towards the internet, the node itself is reachable:

[email protected]:~# traceroute -n 10.4.2.29
traceroute to 10.4.2.29 (10.4.2.29), 30 hops max, 40 byte packets
 1  10.4.2.95  13.918 ms  8.674 ms  4.966 ms
 2  10.4.2.72  20.074 ms  6.971 ms  32.104 ms
 3  10.4.2.29  41.365 ms  12.472 ms  33.477 ms
[email protected]:~# traceroute -n 10.2.29.1
traceroute to 10.2.29.1 (10.2.29.1), 30 hops max, 40 byte packets
 1  * * *
 2  62.103.x.x  46.254 ms  35.242 ms * (internet)

From another node (8.107) the host is pingable, but traceroute drops out to the olsr-network

PING 10.4.2.29 (10.4.2.29): 56 data bytes
64 bytes from 10.4.2.29: seq=0 ttl=59 time=19.229 ms 64 bytes from 10.4.2.29: seq=1 ttl=59 time=14.573 ms

traceroute to 10.4.2.29 (10.4.2.29), 30 hops max, 38 byte packets

 1  192.168.8.105  11.337 ms  2.999 ms  3.366 ms
 2  192.168.8.106  9.791 ms  5.150 ms  5.598 ms
 3  192.168.106.2  10.965 ms  5.981 ms  7.978 ms
 4  192.168.2.95  36.130 ms  6.777 ms  8.472 ms
 5  192.168.2.72  18.389 ms  20.188 ms  10.949 ms
 6  * * *
 7  * *

traceroute to 10.2.29.1 (10.2.29.1), 30 hops max, 38 byte packets

 1  192.168.8.102  6.833 ms  5.335 ms  2.720 ms
 2  192.168.8.106  17.269 ms  5.819 ms  8.960 ms
 3  192.168.106.2  16.785 ms  6.966 ms  5.525 ms
 4  *  (internet)

As I tested this amoung three nanostations (8.105,106,107) all went fine including traceroute to an announced subnet. Of these three only 8.106 has errors of the following kind - may this strange behaviour be an endian issue ? 8.106 atheros is lan-connected to 2.2 wrt54g broadcom:

-3- previously announced networks are not deleted (8.106), the routing table collects multiple entries for the same destination

Nov 26 23:18:42 Nano5-106 daemon.err batmand[574]: Error - can't add throw route to 10.2.29.0/24 via 10.8.106.100 (table 65): File exists
Nov 26 23:18:42 Nano5-106 daemon.err batmand[574]: Error - can't add route to 10.2.29.0/24 via 10.8.106.100 (table 65): File exists
Nov 26 23:18:47 Nano5-106 daemon.err batmand[574]: Error - can't add throw route to 10.8.106.100/32 via 10.8.106.100 (table 65): File exists
Nov 26 23:18:47 Nano5-106 daemon.err batmand[574]: Error - can't add route to 10.8.106.100/32 via 10.8.106.100 (table 65): File exists

root@Nano5-106:~# ip route show table 65
throw 10.5.30.100  proto static
10.5.30.100 via 10.4.8.105 dev ath0  proto static  src 10.4.8.106
10.5.30.100 via 10.4.8.109 dev ath0  proto static  src 10.4.8.106
10.5.30.100 via 10.4.8.102 dev ath0  proto static  src 10.4.8.106
throw 10.2.50.0/24  proto static
10.2.50.0/24 via 10.8.106.100 dev eth0  proto static  src 10.8.106.1
throw 10.2.29.0/24  proto static
10.2.29.0/24 via 10.4.8.102 dev ath0  proto static  src 10.4.8.106
10.2.29.0/24 via 10.8.106.100 dev eth0  proto static  src 10.8.106.1
10.2.29.0/24 via 10.4.8.105 dev ath0  proto static  src 10.4.8.106

root@Nano5-106:~# ip rule
0:      from all lookup local
6600:   from all to 10.4.0.0/16 lookup 66
6601:   from all to 10.8.106.0/24 lookup 66
6699:   from all lookup 65
6700:   from all to 10.4.0.0/16 lookup 67
6701:   from all to 10.8.106.0/24 lookup 67
32766:  from all lookup main
32767:  from all lookup default

This happens on 2.29 as well but not on every node, 8.107,105,104 and maybe others stay clear. It occurs on 8.109 which is lan-connected to broadcom 5.30 (see the map).

Well, I'll keep on testing ;-)

Chris

Hey,

wow - looks pretty good.

You probably experience the same issue as Derek. You could verify whether it is related to endianess or not by looking in the dmesg output on the gateway. Such seldom errors are reported in the kernel log (on OpenWRT you may use logread to access it). We are looking for messages like: "Error - got packet from unknown client ..."

The batman daemon maintains a tunnel connection to every "batman internet client". Every packet that goes to the internet or comes back has to go through this tunnel. As it is a user space tunnel a lot of copying between user space and kernel land is necessary. Depending on the number of clients and the CPU power available this might be a bottleneck.
The batgat kernel module tries to overcome this limitation. Once loaded the batman daemon will detect its presence automatically on startup. The daemon will activate the kernel module to let it handle the tunneling, hence avoiding the expensive copy operations. There is no difference between the daemon tunneling and the kernel tunneling other than that.

I will put this into our FAQ.

Sorry, I don't understand this. Can you explain a bit more ?

Thanks for the hint. Fixed in revision 1159.

Cool ! Many thanks for your hints - we are waiting for feedback. :-)

Regards,
Marek

Hello,

It is, alltogether it's about 50 nodes. This vis is geopraphically more
accurate: http://preveli.gr/mesh/bat-228-3.gif

Now rv1161 is on, the reduced test area is
http://preveli.gr/mesh/bat-228-3b.gif

Yes, looks similar to Derek's.
Not to mess with endian issues, I tested on broadcom only with no atheros connected. The tunnel builds up but isn't stable.

Gateway client - gateway (10.4.2.2) says: IP (169.254.0.1) is invalid (maybe expired)
Deleting default route via gate0 (table 68)
Adding default route to 10.4.2.71 (gw_flags: 40, tq: 188, gw_product:
2747168)
Gateway client - got IP (169.254.0.1) from gateway: 10.4.2.71 Adding default route via gate0 (table 68)

The node itself can connect through the tunnel. Everything else which doesn't come from the nodes IP can't pass, the gateway logs:

Nov 28 16:51:59 (none) kern.err batmand[18644]: Error - got packet from

unknown client: 10.4.2.50 (virtual ip 10.2.50.99)

Same error with olsr traffic which has dropped into the tunnel.

Thank you for the explanation. Is this module available for OpenWrt 0.9 broadcom somewhere ?

Another point is that
even if a gateway's internet connection is down, but announced, it is chosen as tunnel endpoint. Something similar to the olsrd-dyn-gw would be great.

The described effects may be due to routing problems and things like that on batman, olsr's routing is last so it takes all the rest. I'd better not describe this in detail as it may occur due to the following point.

Yes, no multiple entries anymore in 1161.

The announced network of 2.72 gets lost after some minutes on all nodes. Logs show

Nov 28 17:13:43 (none) kern.err batmand[18641]: Error - can't add route
to 10.2.72.0/24 via 10.4.2.95 (table 65): File exists
Nov 28 17:13:43 (none) kern.err batmand[18641]: Error - can't add route

to 10.2.72.1/32 via 10.4.2.95 (table 65): File exists

[email protected]:~# ip ro sh ta 65
10.2.50.0/24 via 10.4.2.50 dev eth1  proto static  src 10.4.2.72

[email protected]:~# ip ro sh ta 65
throw 10.2.50.0/24  proto static

Node 2.2 knows the route but 2.72 doesn't route on to its subnet.

[email protected]:~# ip ro sh ta 65
10.2.72.1 via 10.4.2.95 dev eth1  proto static  src 10.4.2.2
10.2.50.0/24 via 10.4.2.95 dev eth1  proto static  src 10.4.2.2
10.2.72.0/24 via 10.4.2.95 dev eth1  proto static  src 10.4.2.2

[email protected]:~# traceroute 10.2.72.1
traceroute to 10.2.72.1 (10.2.72.1), 30 hops max, 40 byte packets
 1  10.4.2.95 (10.4.2.95)  74.843 ms  7.151 ms  28.312 ms
 2  10.4.2.71 (10.4.2.71)  5.363 ms  7.632 ms  3.652 ms
 3  * * *
 4

The announced network of 2.50 is reachable.

[email protected]:~# traceroute 10.2.50.1
traceroute to 10.2.50.1 (10.2.50.1), 30 hops max, 40 byte packets
 1  10.4.2.95 (10.4.2.95)  52.299 ms  7.063 ms  26.876 ms
 2  10.4.2.72 (10.4.2.72)  24.764 ms  4.44 ms  43.759 ms
 3  10.2.50.1 (10.2.50.1)  15.334 ms  6.379 ms  22.462 ms

The laptop (10.2.50.99) connected to 2.50 cannot be reached:

[email protected]:~# ip ro sh ta 66
10.4.2.95 via 10.4.2.72 dev ath0  proto static  src 10.4.2.50
10.4.2.72 dev ath0  proto static  scope link  src 10.4.2.50
10.4.2.71 via 10.4.2.72 dev ath0  proto static  src 10.4.2.50
10.4.2.2 via 10.4.2.72 dev ath0  proto static  src 10.4.2.50
throw 10.2.50.0/24  proto static

[email protected]:~# ip ro sh ta 67
throw 10.2.50.0/24  proto static
unreachable default  proto static

thank you too for the protocol ...

bye,
Christian

Hello Marek,

with rv 1162 routing seems to be much better,
all nodes and subnets are reachable.

After some minutes I saw this error on 10.4.2.50:

Jan  1 05:55:40 wgt634u_50 daemon.err batmand[7357]: Error - can't add throw route to 10.2.50.0/24 via 0.0.0.0 (table 68): File exists

greetings,
Chris

Hello Chris,

just a short note: rev 1160 - 1162 should [tm] only be cosmetic changes. We added some defines to avoid confusion in the routing code. Therefore rev 1159 and rev 1162 should behave equally (assuming that no new bugs were introduced ... ;] ).

best regards,
Simon

Hey,

The "invalid" messages probably is caused by the bad IPs (see below). The gateway sends an "IP invalid"-message back if a not advertised IP was used. The client then disconnects from the gateway.

We should probably change this policy. ;)

Only the advertised IPs are allowed. However you can build a NAT for your gateway, this would ensure that only the advertised IP is used. Something like

iptables -t nat -A POSTROUTING -o gate0 -j MASQUERADE

should do the job. (maybe add something like "--source 192.168.100.0/24" to further constrain the traffic).

I don't know of precompiled packages, maybe someone else does? :)

Actually we have a mechanism to check for black hole gateways (on the client).

Mhm, this is probably a regression in the new hna_buff_delete() function ... but i'm not sure. The "File exists" Error should not be visible in any case.

Could you please (if possible) create a complete dump until the "File exists" problems happen with debug level 4, and send this log to me and Marek? We can have a more detailed view what's happening then.

(call batman with -cd4 for example)

best regards,
Simon

I'd second this.

Uh, this would mean double NAT in the network, sip phones don't like it.

well, I'll find a place ;-)

Checked it with -r 2 -p 10.4.5.30 which is down at this time, reachable via Atheros nodes.

=> 10.4.5.30       ( 86)       10.4.2.72 [    ath0:1], gw_class  40 - 2048KBit/256KBit, gateway failures: 0
   10.4.2.2        (140)       10.4.2.72 [    ath0:1], gw_class  40 - 

2048KBit/256KBit, gateway failures: 0
10.4.2.71 (150) 10.4.2.72 [ ath0:1], gw_class 40 -
2048KBit/256KBit, gateway failures: 0

dumped

greetings,
Christian

Offering kernel module packages is rather cumbersome as the kernel modules has to match the given kernel _exactly_. I'm not sure how far this goes (some tests would be needed) but I'm sure you would need the same sources (including all patches, same compiler, may be even binary compability?).

Batman has an internal blackhole detection. The olsr dyn_gw is a hack (its just pinging the outside world which works in many cases) - batman has this functionality inside. As every traffic flows through the batman tunnel (forth and back) batman can "observe" the traffic and detect blackholes (nodes announcing internet without a connection) much better. As soon as you generate traffic batman will switch the gateway.

Regards,
Marek

Hello Marek,

You're right, modules are an individual thing. I rather hoped for a hint to a backported openwrt-0.9 makefile ;-).

It is working as you described, here it takes about one minute to switch to an active gateway.

greetings,
Christian

I'm not sure what an "openwrt-0.9" is but may be this helps you:
https://dev.open-mesh.net/batman/wiki/Patches

Yeah, it always takes one minute - its the internal timeout for inactive gateways. :-)

By the way, you still can ping around and test the connection yourself. Either by using a simple cron script which runs every x minutes or by triggering your test script via the routing script. Depending on your setup/firewall/ISP you might experience problems pinging around. Also, if your users expect the internet to work for 2 hours per day you waste time and bandwidth in the other 22 hours of the day.

Regards,
Marek

That's it, didn't search there, thanks.

greetings,
Chris

I overlooked that comment thats why my answer comes a bit late:

Are you sure that the batman setup is a problem for SIP ? I'm definitely not the SIP expert but I don't think it is a problem (correct me if I'm wrong).

The normal double NAT setup looks like this:

NET_A -> NET_B -> Internet

SIP & STUN get in trouble here.

In the batman network it looks a bit different:

NET_A -> NET_B -> NET_A -> Internet

The second NAT is used for internal transport only. I guess STUN will not notice that there is the second network in between. Could you verify that ?

Regards,
Marek

Hello there,

meanwhile with rv 1172 the previously mentioned issues concerning wrong or double routing entries are solved: nodes and announced networks are reachable on broadcom and on atheros devices and the gateway tunnel works as described when NAT'ed. Great, thanx !

The sip test with NAT is still to come.

best regards,
Chris