From: Kristof Provost Date: Mon, 22 Aug 2022 14:55:09 +0200 Message-ID: <21B552DE-38FE-41F0-8FC7-606EA4C06093@...1376...> In-Reply-To: References: <20220812134154.16729-3-kprovost@...2661...> <202208201401.27KE12Cm014083@...2054...> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_MailMate_C427DE8D-62AB-4CCF-BAE5-EAC24188FEEF_=" Content-Transfer-Encoding: 8bit Subject: Re: [Openvpn-devel] route/iroute handling on FreeBSD List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Gert Doering Cc: openvpn-devel@lists.sourceforge.net --=_MailMate_C427DE8D-62AB-4CCF-BAE5-EAC24188FEEF_= Content-Type: text/plain; charset="utf-8"; format="flowed" Content-Transfer-Encoding: 8bit On 20 Aug 2022, at 16:33, Gert Doering wrote: >> - iroute installation works for the easy cases (--route in >> server.conf, >> --iroute with a more-specific of that in ccd/). It does not work >> for the nasty cases (--route and --iroute with same netbits). >> >> I will send a followup e-mail with more details on this soon. > > *This* is the bit I want to cover separately, because this e-mail is > going to be long :-) > > So, here's my test cases, to see if I could get DCO to misbehave. > > Server config has: > > # grep ^route server.conf > route 10.114.200.0 255.255.255.0 > route-ipv6 fd00:abcd:114:200::/64 > route 10.114.201.0 255.255.255.0 > route-ipv6 fd00:abcd:114:201::/64 > > ("pull two /24 and two /64 towards the tun interface") > > There is one particular client which has many addresses configured on > its loopback > > lo0: flags=8049 metric 0 mtu 16384 > inet 10.114.200.74 netmask 0xffffffff > inet 10.114.201.74 netmask 0xffffffff > inet 10.114.202.74 netmask 0xffffffff > inet6 fd00:abcd:114:200::74 prefixlen 128 > inet6 fd00:abcd:114:201::74 prefixlen 128 > inet6 fd00:abcd:114:202::74 prefixlen 128 > > ... and when that client connects, the expectation is that *other* > clients > can ping these 6 addresses (note it's 200, 201, 202 for v4 and v6). > > Here's the ccd file for this client: > > # cat /root/t_server/tun-udp-p2mp/ccd/freebsd-74-amd64 > # Host iroute > iroute 10.114.200.74 255.255.255.255 > iroute-ipv6 fd00:abcd:114:200::74/128 > # Subnet-iroute > iroute 10.114.201.0 255.255.255.0 > iroute-ipv6 fd00:abcd:114:201::/64 > # iroute-without-route > # 10.114.202, fd00:abcd:114:202::/64 > iroute 10.114.202.74 255.255.255.255 > iroute-ipv6 fd00:abcd:114:202::74/128 > > > So the "200" routes are "install a host route for IPv4 or IPv6 that is > part of an aggregate". This works fine. > > The "202" routes are "install a host route for IPv4 or IPv6 where > there > is no covering --route" - this is basically something OpenVPN could > not > do before DCO (update host routes on-demand on client connect), but is > something I've long waited for :-) - this also works fine. > > Now, the 201 routes are "there whole network specified in --route is > routed to a single client". This is not such an unusual setup in > cases like "there is a central office OpenVPN server, and one of the > branch offices wants it 192.168.77.0/24 reachable from the other > clients". This *fails*. > > For IPv4, we end up with "netstat -rn" looking like this: > > 10.114.201.0/24 10.114.2.2 UGS tun1 > 10.114.201.0/24 10.114.2.8 UGS tun1 > > and "route get" telling us the the first one is preferred > > $ route get 10.114.201.74 > route to: 10.114.201.74 > destination: 10.114.201.0 > mask: 255.255.255.0 > gateway: 10.114.2.2 > fib: 0 > interface: tun1 > > so packets loop... (10.114.2.2 is not an address that belongs to an > actual client, but "the tun interface peer") the client is 10.114.2.8. > > $ traceroute 10.114.201.74 > traceroute to 10.114.201.74 (10.114.201.74), 64 hops max, 40 byte > packets > 1 10.114.2.2 (10.114.2.2) 1.706 ms 1.240 ms 0.921 ms > 2 localhost (127.0.0.1) 0.865 ms 0.809 ms 0.807 ms > 3 10.114.2.2 (10.114.2.2) 1.603 ms 1.605 ms 1.595 ms > 4 localhost (127.0.0.1) 1.533 ms 1.383 ms 1.768 ms > > > For IPv6, things look different - route installation fails > > 2022-08-20 15:47:24 freebsd-74-amd64/2001:608:0:814::f000:3 > /sbin/route -6 add -net fd00:abcd:114:201::/64 fd00:abcd:114:2::1006 > -fib 0 > add net fd00:abcd:114:201::/64: gateway fd00:abcd:114:2::1006 fib 0: > route already in table > > so, the iroute never makes it to kernel, "netstat -rn" remains at > > fd00:abcd:114:201::/64 link#14 US > tun1 > > and packets loop as well > > $ traceroute6 fd00:abcd:114:201::74 > traceroute6 to fd00:abcd:114:201::74 (fd00:abcd:114:201::74) from > fd00:abcd:114:2::1, 64 hops max, 28 byte packets > 1 fd00:abcd:114:2::1000 1.749 ms 1.246 ms 0.930 ms > 2 fd00:abcd:114:2::1 0.836 ms 1.225 ms 1.067 ms > 3 fd00:abcd:114:2::1000 1.555 ms 1.631 ms 1.575 ms > 4 fd00:abcd:114:2::1 1.472 ms 1.531 ms 1.677 ms > > > > So, what we do on *Linux* to handle this, is to work with route > metrics > (10.220.* = "same instances, different host, so different address > range") > > linux$ ip route |grep tun1 > 10.220.200.0/24 via 10.220.2.2 dev tun1 metric 200 > 10.220.200.74 via 10.220.2.8 dev tun1 metric 100 > 10.220.201.0/24 via 10.220.2.8 dev tun1 metric 100 > 10.220.201.0/24 via 10.220.2.2 dev tun1 metric 200 > 10.220.202.74 via 10.220.2.8 dev tun1 metric 100 > linux$ ip -6 route |grep tun1 > fd00:abcd:220:200::74 via fd00:abcd:220:2::1006 dev tun1 metric 100 > pref medium > fd00:abcd:220:200::/64 dev tun1 metric 200 pref medium > fd00:abcd:220:201::/64 via fd00:abcd:220:2::1006 dev tun1 metric 100 > pref medium > fd00:abcd:220:201::/64 dev tun1 metric 200 pref medium > fd00:abcd:220:202::74 via fd00:abcd:220:2::1006 dev tun1 metric 100 > pref medium > > so there's always 5 routes in the active table - 2 "--route" routes > (according to server.conf) with *metric 200*, and then 3 "--iroute" > routes > (from ccd/) with metric DCO_IROUTE_METRIC = 100. > > Linux handles this as would a "Cisco router" do - if there are two > identical routes with different metric, the lower-metric route is > used, and the other one is ignored. So this works very nicely. > > > Now, back to FreeBSD. > > - our code does not try to set metrics on FreeBSD > - my reading of route(8) does not show me anything in that direction > (metric, preference, administrative distance, ...) > - I do not understand FreeBSD internals well enough to know whether > it can be done at all, or not. > It’s rather poorly (i.e. not) documented, but it is possible to set a route metric. It’s called a ‘weight’, and not actually mentioned in the man page. Or shown in netstat output. But: $ sudo route add 172.16.2.0/24 10.0.2.1 add net 172.16.2.0: gateway 10.0.2.1 $ sudo route add 172.16.2.0/24 -weight 2 10.0.2.254 add net 172.16.2.0: gateway 10.0.2.254 So we can add multiple routes for the same network, as long as they have a different gateway. The usual `netstat -rn` doesn’t show the weight. You have to use the json/xml output to get it: $ netstat -rn --libxo json | jq … { "destination": "172.16.2.0/24", "gateway": "10.0.2.1", "flags": "UGS", "flags_pretty": [ "up", "gateway", "static" ], "weight": 1, "interface-name": "bnxt0" }, { "destination": "172.16.2.0/24", "gateway": "10.0.2.254", "flags": "UGS", "flags_pretty": [ "up", "gateway", "static" ], "weight": 2, "interface-name": "bnxt0" }, A higher weight gives a higher priority, so that’s the opposite of what Linux does with metric, I believe. > If it can not be done, we need to document this as "one of the issues > with FreeBSD DCO", and it can of course be easily workarounded in most > cases > > --route 10.114.201.0/24 > --iroute 10.114.201.0/25 > --iroute 10.114.201.128/25 > > will get the same thing done "when client is online route the whole > /25 > down there", but it's harder to get into people's heads :-) (adding > code to auto-split such a prefix is technically possible but would be > quite a complication, so I'd rather solve this "with documentation"). > > So, what shall we do? > If I’m understanding everything correctly it should be relatively simple to extend networking_freebsd.c to also set a weight for each route it installs. We’d have to convert the metric into a weight, which I think we could do as `weight = RT_MAX_WEIGHT - metric` (RT_MAX_WEIGHT is 16777215 /* 3 bytes */). Can you try this? diff --git a/src/openvpn/networking_freebsd.c b/src/openvpn/networking_freebsd.c index 0633dce7..e79af2bf 100644 --- a/src/openvpn/networking_freebsd.c +++ b/src/openvpn/networking_freebsd.c @@ -10,6 +10,8 @@ #if defined(TARGET_FREEBSD) +#include + static int net_route_v4(const char *op, const in_addr_t *dst, int prefixlen, const in_addr_t *gw, const char *iface, uint32_t table, @@ -23,12 +25,13 @@ net_route_v4(const char *op, const in_addr_t *dst, int prefixlen, _dst = ntohl(*dst); _gw = ntohl(*gw); - argv_printf(&argv, "%s %s -net %s/%d %s -fib %d", + argv_printf(&argv, "%s %s -net %s/%d %s -fib %d -weight %d", ROUTE_PATH, op, inet_ntop(AF_INET, &_dst, buf1, sizeof(buf1)), prefixlen, inet_ntop(AF_INET, &_gw, buf2, sizeof(buf2)), - table); + table, + RT_MAX_WEIGHT - metric); argv_msg(M_INFO, &argv); status = openvpn_execve_check(&argv, NULL, 0, @@ -48,12 +51,13 @@ net_route_v6(const char *op, const struct in6_addr *dst, struct argv argv = argv_new(); bool status; - argv_printf(&argv, "%s -6 %s -net %s/%d %s -fib %d", + argv_printf(&argv, "%s -6 %s -net %s/%d %s -fib %d -weight %d", ROUTE_PATH, op, inet_ntop(AF_INET6, dst, buf1, sizeof(buf1)), prefixlen, inet_ntop(AF_INET6, gw, buf2, sizeof(buf2)), - table); + table, + RT_MAX_WEIGHT - metric); argv_msg(M_INFO, &argv); status = openvpn_execve_check(&argv, NULL, 0, Kristof --=_MailMate_C427DE8D-62AB-4CCF-BAE5-EAC24188FEEF_= Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On 20 Aug 2022, at 16:33, Gert Doering wrote:

- iroute installation works for the easy cases (--route = in server.conf,
--iroute with a more-specific of that in ccd/). It does not work
for the nasty cases (--route and --iroute with same netbits).

I will send a followup e-mail with more details on this = soon.

*This* is the bit I want to cover separately, = because this e-mail is
going to be long :-)

So, here's my test cases, to see if I could get DCO to misb= ehave.

Server config has:

# grep ^route server.conf
route 10.114.200.0 255.255.255.0
route-ipv6 fd00:abcd:114:200::/64
route 10.114.201.0 255.255.255.0
route-ipv6 fd00:abcd:114:201::/64

("pull two /24 and two /64 towards the tun interface")

There is one particular client which has many addresses con= figured on
its loopback

lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metr= ic 0 mtu 16384
inet 10.114.200.74 netmask 0xffffffff
inet 10.114.201.74 netmask 0xffffffff
inet 10.114.202.74 netmask 0xffffffff
inet6 fd00:abcd:114:200::74 prefixlen 128
inet6 fd00:abcd:114:201::74 prefixlen 128
inet6 fd00:abcd:114:202::74 prefixlen 128

... and when that client connects, the expectation is that = *other* clients
can ping these 6 addresses (note it's 200, 201, 202 for v4 and v6).

Here's the ccd file for this client:

# cat /root/t_server/tun-udp-p2mp/ccd/freebsd-74-amd64
# Host iroute
iroute 10.114.200.74 255.255.255.255
iroute-ipv6 fd00:abcd:114:200::74/128
# Subnet-iroute
iroute 10.114.201.0 255.255.255.0
iroute-ipv6 fd00:abcd:114:201::/64
# iroute-without-route
# 10.114.202, fd00:abcd:114:202::/64
iroute 10.114.202.74 255.255.255.255
iroute-ipv6 fd00:abcd:114:202::74/128

So the "200" routes are "install a host route for IPv4 or I= Pv6 that is
part of an aggregate". This works fine.

The "202" routes are "install a host route for IPv4 or IPv6= where there
is no covering --route" - this is basically something OpenVPN could not
do before DCO (update host routes on-demand on client connect), but is
something I've long waited for :-) - this also works fine.

Now, the 201 routes are "there whole network specified in -= -route is
routed to a single client". This is not such an unusual setup in
cases like "there is a central office OpenVPN server, and one of the
branch offices wants it 192.168.77.0/24 reachable from the other
clients". This *fails*.

For IPv4, we end up with "netstat -rn" looking like this:

10.114.201.0/24 10.114.2.2 UGS tun1
10.114.201.0/24 10.114.2.8 UGS tun1

and "route get" telling us the the first one is preferred

$ route get 10.114.201.74
route to: 10.114.201.74
destination: 10.114.201.0
mask: 255.255.255.0
gateway: 10.114.2.2
fib: 0
interface: tun1

so packets loop... (10.114.2.2 is not an address that belon= gs to an
actual client, but "the tun interface peer") the client is 10.114.2.8.

$ traceroute 10.114.201.74
traceroute to 10.114.201.74 (10.114.201.74), 64 hops max, 40 byte packets
1 10.114.2.2 (10.114.2.2) 1.706 ms 1.240 ms 0.921 ms
2 localhost (127.0.0.1) 0.865 ms 0.809 ms 0.807 ms
3 10.114.2.2 (10.114.2.2) 1.603 ms 1.605 ms 1.595 ms
4 localhost (127.0.0.1) 1.533 ms 1.383 ms 1.768 ms

For IPv6, things look different - route installation fails<= /p>

2022-08-20 15:47:24 freebsd-74-amd64/2001:608:0:814::f000:3= /sbin/route -6 add -net fd00:abcd:114:201::/64 fd00:abcd:114:2::1006 -fib 0
add net fd00:abcd:114:201::/64: gateway fd00:abcd:114:2::1006 fib 0: route = already in table

so, the iroute never makes it to kernel, "netstat -rn" rema= ins at

fd00:abcd:114:201::/64 link#14 = US tun1

and packets loop as well

$ traceroute6 fd00:abcd:114:201::74
traceroute6 to fd00:abcd:114:201::74 (fd00:abcd:114:201::74) from fd00:abcd= :114:2::1, 64 hops max, 28 byte packets
1 fd00:abcd:114:2::1000 1.749 ms 1.246 ms 0.930 ms
2 fd00:abcd:114:2::1 0.836 ms 1.225 ms 1.067 ms
3 fd00:abcd:114:2::1000 1.555 ms 1.631 ms 1.575 ms
4 fd00:abcd:114:2::1 1.472 ms 1.531 ms 1.677 ms

So, what we do on *Linux* to handle this, is to work with r= oute metrics
(10.220.* =3D "same instances, different host, so different address range")=

linux$ ip route |grep tun1
10.220.200.0/24 via 10.220.2.2 dev tun1 metric 200
10.220.200.74 via 10.220.2.8 dev tun1 metric 100
10.220.201.0/24 via 10.220.2.8 dev tun1 metric 100
10.220.201.0/24 via 10.220.2.2 dev tun1 metric 200
10.220.202.74 via 10.220.2.8 dev tun1 metric 100
linux$ ip -6 route |grep tun1
fd00:abcd:220:200::74 via fd00:abcd:220:2::1006 dev tun1 metric 100 pref me= dium
fd00:abcd:220:200::/64 dev tun1 metric 200 pref medium
fd00:abcd:220:201::/64 via fd00:abcd:220:2::1006 dev tun1 metric 100 pref m= edium
fd00:abcd:220:201::/64 dev tun1 metric 200 pref medium
fd00:abcd:220:202::74 via fd00:abcd:220:2::1006 dev tun1 metric 100 pref me= dium

so there's always 5 routes in the active table - 2 "--route= " routes
(according to server.conf) with *metric 200*, and then 3 "--iroute" routes
(from ccd/) with metric DCO_IROUTE_METRIC =3D 100.

Linux handles this as would a "Cisco router" do - if there = are two
identical routes with different metric, the lower-metric route is
used, and the other one is ignored. So this works very nicely.

Now, back to FreeBSD.

- our code does not try to set metrics on FreeBSD
- my reading of route(8) does not show me anything in that direction
(metric, preference, administrative distance, ...)
- I do not understand FreeBSD internals well enough to know whether
it can be done at all, or not.


It=E2=80=99s rather poorly (i.e. not) documented, but it is= possible to set a route metric. It=E2=80=99s called a =E2=80=98weight=E2= =80=99, and not actually mentioned in the man page. Or shown in netstat out= put.

But:

$ sudo route add 172.16.2.0/24 10.0.2.1
add net 172.16.2.0: gateway 10.0.2.1
$ sudo route add 172.16.2.0/24 -weight 2 10.0.2.254
add net 172.16.2.0: gateway 10.0.2.254

So we can add multiple routes for the same network, as long= as they have a different gateway.

The usual netstat -rn doesn=E2=80=99t show the= weight. You have to use the json/xml output to get it:

$ netstat -rn --libxo json | jq
=E2=80=A6
          {
            "destination": "172.16.2.0/24",
            "gateway": "10.0.2.1",
            "flags": "UGS",
            "flags_pretty": [
              "up",
              "gateway",
              "static"
            ],
            "weight": 1,
            "interface-name": "bnxt0"
          },
          {
            "destination": "172.16.2.0/24",
            "gateway": "10.0.2.254",
            "flags": "UGS",
            "flags_pretty": [
              "up",
              "gateway",
              "static"
            ],
            "weight": 2,
            "interface-name": "bnxt0"
          },

A higher weight gives a higher priority, so that=E2=80=99s = the opposite of what Linux does with metric, I believe.

If it can not be done, we need to documen= t this as "one of the issues
with FreeBSD DCO", and it can of course be easily workarounded in most
cases

--route 10.114.201.0/24
--iroute 10.114.201.0/25
--iroute 10.114.201.128/25

will get the same thing done "when client is online route t= he whole /25
down there", but it's harder to get into people's heads :-) (adding
code to auto-split such a prefix is technically possible but would be
quite a complication, so I'd rather solve this "with documentation").

So, what shall we do?


If I=E2=80=99m understanding everything correctly it should= be relatively simple to extend networking_freebsd.c to also set a weight f= or each route it installs.
We=E2=80=99d have to convert the metric into a weight, which I think we cou= ld do as weight =3D RT_MAX_WEIGHT - metric (RT_MAX_WEIGHT is 1= 6777215 /* 3 bytes */).

Can you try this?

diff --git a/src/openvpn/networking_freebsd.c b/src/openvpn/netw=
orking_freebsd.c
index 0633dce7..e79af2bf 100644
--- a/src/openvpn/networking_freebsd.c
+++ b/src/openvpn/networking_freebsd.c
@@ -10,6 +10,8 @@

 #if defined(TARGET_FREEBSD)

+#include <net/route.h>
+
 static int
 net_route_v4(const char *op, const in_addr_t *dst, int prefixlen,
              const in_addr_t *gw, const char *iface, uint32_t table,
@@ -23,12 +25,13 @@ net_route_v4(const char *op, const in_addr_t *dst, int =
prefixlen,
     _dst =3D ntohl(*dst);
     _gw =3D ntohl(*gw);

-    argv_printf(&argv, "%s %s -net %s/%d %s -fib %d",
+    argv_printf(&argv, "%s %s -net %s/%d %s -fib %d -weight %d&qu=
ot;,
                 ROUTE_PATH, op,
                 inet_ntop(AF_INET, &_dst, buf1, sizeof(buf1)),
                 prefixlen,
                 inet_ntop(AF_INET, &_gw, buf2, sizeof(buf2)),
-                table);
+                table,
+                RT_MAX_WEIGHT - metric);

     argv_msg(M_INFO, &argv);
     status =3D openvpn_execve_check(&argv, NULL, 0,
@@ -48,12 +51,13 @@ net_route_v6(const char *op, const struct in6_addr *dst,
     struct argv argv =3D argv_new();
     bool status;

-    argv_printf(&argv, "%s -6 %s -net %s/%d %s -fib %d",
+    argv_printf(&argv, "%s -6 %s -net %s/%d %s -fib %d -weight %d=
",
                 ROUTE_PATH, op,
                 inet_ntop(AF_INET6, dst, buf1, sizeof(buf1)),
                 prefixlen,
                 inet_ntop(AF_INET6, gw, buf2, sizeof(buf2)),
-                table);
+                table,
+                RT_MAX_WEIGHT - metric);

     argv_msg(M_INFO, &argv);
     status =3D openvpn_execve_check(&argv, NULL, 0,

Kristof

--=_MailMate_C427DE8D-62AB-4CCF-BAE5-EAC24188FEEF_=--