How to override the default source ipv4 address on packets originating from a router
The mailing list has been dormant for 2 years, so I wonder whether anyone is still listening for new questions?
My broadband router runs PPPoA and is dynamically assigned a single ipv4 internet address by my ISP. I have a static subnet which I host on a linux router/firewall (called chenin). The linux firewall and the adsl router communicate via a non-internet-addressable private subnet. Here is the topology:
Internet -- adsl-router-ppp0-ipv4-dynamic
-- adsl-router-eth0-172.16.101.1
--
-- firewall-router-eth0-172.16.101.2
-- firewall-router-217.154.193.209
--
-- static-subnet-hosts-217.154.193.154.208/28Each of the hosts on the static subnet use 217.154.193.209 as their own default route. The adsl router forwards all incoming packets to the firewall/router's eth0. The firewall/router forwards all incoming packets to the static subnet via its own eth1. The firewall/router does not need to perform NAT, but it implements a simple set of iptables rules for blacklisting, etc. /All this works perfectly./
My problem is that I need to download software updates (debian apt-get http) for the firewall/router from a repository out on the internet.
The firewall/router can successfully ping the repo-server when I force the source address like this:
ping -I 217.154.193.209 163.1.221.67
... but a simple "ping 163.1.221.67" (i.e. using the default source address selection algorithm) fails. Wireshark confirms these unanswered packets go out on eth0 with a source address of 172.16.101.2.
I believe I should be able to resolve this problem with iproute2 policy routing, but so far I have not been successful. I've tried several variations, but they all give me the same "wrong" source address.
Here is my simplest effort:
brian@chenin:~$ ip rule list 0: from all lookup local 32765: from 172.16.101.2 lookup CHENIN_ONLY 32766: from all lookup main 32767: from all lookup default
brian@chenin:~$ ip route list table CHENIN_ONLY 163.1.221.67 via 172.16.101.1 dev eth0 src 217.154.193.209
brian@chenin:~$ sudo ip route flush cache
brian@chenin:~$ ip route get 163.1.221.67
163.1.221.67 via 172.16.101.1 dev eth0 src 172.16.101.2
cacheThis shows me the source address in my policy rule has NOT been used, or the routing table entry does not work the way I think.
The only rule table with higher priority than CHENIN_ONLY is local, which contains routes for addresses local to the firewall/router - nothing about remote addresses, i.e.
brian@chenin:~$ ip route list table local broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast 172.16.101.0 dev eth0 proto kernel scope link src 172.16.101.2 local 172.16.101.2 dev eth0 proto kernel scope host src 172.16.101.2 broadcast 172.16.101.255 dev eth0 proto kernel scope link src 172.16.101.2 broadcast 217.154.193.208 dev eth1 proto kernel scope link src 217.154.193.209 local 217.154.193.209 dev eth1 proto kernel scope host src 217.154.193.209 broadcast 217.154.193.223 dev eth1 proto kernel scope link src 217.154.193.209
and the main table looks like this:
brian@chenin:~$ ip route list table main default via 172.16.101.1 dev eth0 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209
I wonder whether my unsuccessful "naked" pings to 163.1.221.67 are passing through the tables three times:
1. My explicit policy should be applied to select a source address
of 217.154.193.209.
2. The main table is used to select the default route via 172.16.101.1:
3. The main table then provides the explicit route to the default router, which seems to rewrite the source address to 172.16.101.2.
I have read the latest lartc HOWTO sections relevant to policy routing, but didn't see anything similar to my situation. I also didn't see how to get a verbose (i.e. blow by blow) output from the "ip route get" command to show how it is selecting its route.
Am I trying to do the impossible here, or am I just making a mistake in the way I am doing it?
I hope this strikes someone as an interesting question. If I can achieve a solution, I would be happy to add the scenario to the HOWTO.
Brian
Your problem is one I've come up against a number of times in the past. You are definitely on the right route using "policy routing" (otherwise known as utilizing IP Rules), but the snag you're hitting is that the source address of a locally generated packet isn't chosen until a destination route has already been selected. Using the from statement in an ip rules table is only useful when you're routing packets from other subnets.
Now, just so you know, the local table should normally be left alone. It's used for locally connected subnets only. It is given the highest priority on purpose... don't mess with it! :)
There are a few methods you could try here, but I'm only going to recommend one of them. Have your main routing table hold the routes that your local machine will use, and then have a separate table for all routed hosts. Then, use ip rule statements to force any traffic coming in from a routed host into that routing table.
IE: on firewall-router (assuming eth1 is facing your static subnet hosts):
# ip rule list 0: from all lookup local 32765: iif eth1 lookup static-hosts 32766: from all lookup main 32767: from all lookup default
Have table main include all of the routes your local firewall-router should use, and have static-hosts contain all of the routes that your static-subnet-hosts should use.
Joel Gerber
Network Specialist
Network Operations
Eastlink
E: [email protected] T: 519.786.1241
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Brian Burch
Sent: December-20-13 7:05 AM
To: Linux Advanced Routing
Subject: How to override the default source ipv4 address on packets originating from a router
The mailing list has been dormant for 2 years, so I wonder whether anyone is still listening for new questions?
My broadband router runs PPPoA and is dynamically assigned a single ipv4 internet address by my ISP. I have a static subnet which I host on a linux router/firewall (called chenin). The linux firewall and the adsl router communicate via a non-internet-addressable private subnet. Here is the topology:
Internet -- adsl-router-ppp0-ipv4-dynamic
-- adsl-router-eth0-172.16.101.1
--
-- firewall-router-eth0-172.16.101.2
-- firewall-router-217.154.193.209
--
-- static-subnet-hosts-217.154.193.154.208/28Each of the hosts on the static subnet use 217.154.193.209 as their own default route. The adsl router forwards all incoming packets to the firewall/router's eth0. The firewall/router forwards all incoming packets to the static subnet via its own eth1. The firewall/router does not need to perform NAT, but it implements a simple set of iptables rules for blacklisting, etc. /All this works perfectly./
My problem is that I need to download software updates (debian apt-get http) for the firewall/router from a repository out on the internet.
The firewall/router can successfully ping the repo-server when I force the source address like this:
ping -I 217.154.193.209 163.1.221.67
... but a simple "ping 163.1.221.67" (i.e. using the default source address selection algorithm) fails. Wireshark confirms these unanswered packets go out on eth0 with a source address of 172.16.101.2.
I believe I should be able to resolve this problem with iproute2 policy routing, but so far I have not been successful. I've tried several variations, but they all give me the same "wrong" source address.
Here is my simplest effort:
brian@chenin:~$ ip rule list 0: from all lookup local 32765: from 172.16.101.2 lookup CHENIN_ONLY 32766: from all lookup main 32767: from all lookup default
brian@chenin:~$ ip route list table CHENIN_ONLY 163.1.221.67 via 172.16.101.1 dev eth0 src 217.154.193.209
brian@chenin:~$ sudo ip route flush cache brian@chenin:~$ ip route get 163.1.221.67
163.1.221.67 via 172.16.101.1 dev eth0 src 172.16.101.2
cacheThis shows me the source address in my policy rule has NOT been used, or the routing table entry does not work the way I think.
The only rule table with higher priority than CHENIN_ONLY is local, which contains routes for addresses local to the firewall/router - nothing about remote addresses, i.e.
brian@chenin:~$ ip route list table local broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast 172.16.101.0 dev eth0 proto kernel scope link src 172.16.101.2 local 172.16.101.2 dev eth0 proto kernel scope host src 172.16.101.2 broadcast 172.16.101.255 dev eth0 proto kernel scope link src 172.16.101.2 broadcast 217.154.193.208 dev eth1 proto kernel scope link src 217.154.193.209 local 217.154.193.209 dev eth1 proto kernel scope host src 217.154.193.209 broadcast 217.154.193.223 dev eth1 proto kernel scope link src 217.154.193.209
and the main table looks like this:
brian@chenin:~$ ip route list table main default via 172.16.101.1 dev eth0 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209
I wonder whether my unsuccessful "naked" pings to 163.1.221.67 are passing through the tables three times:
1. My explicit policy should be applied to select a source address
of 217.154.193.209.
2. The main table is used to select the default route via 172.16.101.1:
3. The main table then provides the explicit route to the default router, which seems to rewrite the source address to 172.16.101.2.
I have read the latest lartc HOWTO sections relevant to policy routing, but didn't see anything similar to my situation. I also didn't see how to get a verbose (i.e. blow by blow) output from the "ip route get" command to show how it is selecting its route.
Am I trying to do the impossible here, or am I just making a mistake in the way I am doing it?
I hope this strikes someone as an interesting question. If I can achieve a solution, I would be happy to add the scenario to the HOWTO.
Brian
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Brian,
Quoted 2 lines · Brian Burch
The mailing list has been dormant for 2 years, so I wonder whether anyone is still listening for new questions?
You never know. ;-)
Quoted 29 lines
My broadband router runs PPPoA and is dynamically assigned a single ipv4 internet address by my ISP. I have a static subnet which I host on a linux router/firewall (called chenin). The linux firewall and the adsl router communicate via a non-internet-addressable private subnet. Here is the topology:
Internet -- adsl-router-ppp0-ipv4-dynamic
-- adsl-router-eth0-172.16.101.1 -- -- firewall-router-eth0-172.16.101.2 -- firewall-router-217.154.193.209 -- -- static-subnet-hosts-217.154.193.154.208/28Each of the hosts on the static subnet use 217.154.193.209 as their own default route. The adsl router forwards all incoming packets to the firewall/router's eth0. The firewall/router forwards all incoming packets to the static subnet via its own eth1. The firewall/router does not need to perform NAT, but it implements a simple set of iptables rules for blacklisting, etc. /All this works perfectly./
My problem is that I need to download software updates (debian apt-get http) for the firewall/router from a repository out on the internet.
The firewall/router can successfully ping the repo-server when I force the source address like this:
Quoted 6 lines
ping -I 217.154.193.209 163.1.221.67... but a simple "ping 163.1.221.67" (i.e. using the default source address selection algorithm) fails. Wireshark confirms these unanswered packets go out on eth0 with a source address of 172.16.101.2.
I see several generic ways to solve this. Use one of:
1) Use NAT for the transfer network addresses (i.e. "masquerading" for the 172.16.101.2 address) on the ADSL router.
2) Use SNAT from 172.16.101.2 to 217.154.193.209 on the firewall for packets going to the internet.
3) Use a subnet from your routable IP addresses for the transit network.
4) Terminate PPPoE on the firewall, using the ADSL router as an ADSL modem.
You may be able to define the source IP to use for the update process. If so, you do not need any of the above.
Quoted 4 lines
I believe I should be able to resolve this problem with iproute2 policy routing, but so far I have not been successful. I've tried several variations, but they all give me the same "wrong" source address.
Policy routing is used _after_ deciding on the source IP address to use.
Quoted 2 lines
Am I trying to do the impossible here, or am I just making a mistake in the way I am doing it?
You need to change the source IP address seen on the internet, that cannot be done with policy routing. It can be done using NAT, network renumbering or software configuration.
HTH,
Erik
Signature · 2 lines
Always use the right tool for the job.
-- Rob PikeQuoted 2 lines · Anton 'EvilMan' Danilov
Hello.
What's about 'ip route get <dst> from <src>' command?
<snip/>
Thanks very much for responding Anton. By now you have probably read the responses from Joel and Erik, who both state categorically that the source address for packets originating on the host is not selected until /after/ the outbound route has been determined.
This fact explains why "ip route get to 163.1.221.67" returns the identical result no matter whether I omit the "from", or use "from 217.154.193.209", or use "from 172.16.101.2".
In every case the route returned is "via 172.16.101.1" as expected, but in every case the source address is that of the eth0 interface 172.16.101.2.
I have more to say, but will do it by replying directly to Joel and Eric's suggestions.
Thanks again,
Brian
Quoted 1 line · Joel Gerber
Your problem is one I've come up against a number of times in the past. You are definitely on the right route using "policy routing" (otherwise known as utilizing IP Rules), but the snag you're hitting is that the source address of a locally generated packet isn't chosen until a destination route has already been selected. Using the from statement in an ip rules table is only useful when you're routing packets from other subnets.
What a quick and helpful reply! Not only did my problem motivate you to respond, but I am very relieved to find the answer isn't just RTFM.
Your explanation about the timing for kernel selection of the source address is crucial and now my failed experiments all make sense. Your suggested solution also makes sense, although it is counter-intuitive... at least to me!
Quoted 1 line
Now, just so you know, the local table should normally be left alone. It's used for locally connected subnets only. It is given the highest priority on purpose... don't mess with it! :)
I read somewhere (possibly the howto guide or the ip-cref) a statement that the kernel treats this table as read-only. My system is running ubuntu 13.04 with a custom non-pae 3.8.0 kernel, so it is reasonably modern. When I thought more about the purpose of this table, I decided there would be no point in trying to modify it, even if that was possible.
Quoted 11 lines
There are a few methods you could try here, but I'm only going to recommend one of them. Have your main routing table hold the routes that your local machine will use, and then have a separate table for all routed hosts. Then, use ip rule statements to force any traffic coming in from a routed host into that routing table.
IE: on firewall-router (assuming eth1 is facing your static subnet hosts):
# ip rule list 0: from all lookup local 32765: iif eth1 lookup static-hosts 32766: from all lookup main 32767: from all lookup defaultHave table main include all of the routes your local firewall-router should use, and have static-hosts contain all of the routes that your static-subnet-hosts should use.
Great! I understand the principle you propose and will try to implement it over the weekend. I will update this thread when I have something to report.
Once again, thanks for helping,
Brian
Quoted 110 lines
Joel Gerber
Network Specialist
Network Operations
Eastlink
E: [email protected] T: 519.786.1241
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Brian Burch
Sent: December-20-13 7:05 AM
To: Linux Advanced Routing
Subject: How to override the default source ipv4 address on packets originating from a routerThe mailing list has been dormant for 2 years, so I wonder whether anyone is still listening for new questions?
My broadband router runs PPPoA and is dynamically assigned a single ipv4 internet address by my ISP. I have a static subnet which I host on a linux router/firewall (called chenin). The linux firewall and the adsl router communicate via a non-internet-addressable private subnet. Here is the topology:
Internet -- adsl-router-ppp0-ipv4-dynamic
-- adsl-router-eth0-172.16.101.1 -- -- firewall-router-eth0-172.16.101.2 -- firewall-router-217.154.193.209 -- -- static-subnet-hosts-217.154.193.154.208/28Each of the hosts on the static subnet use 217.154.193.209 as their own default route. The adsl router forwards all incoming packets to the firewall/router's eth0. The firewall/router forwards all incoming packets to the static subnet via its own eth1. The firewall/router does not need to perform NAT, but it implements a simple set of iptables rules for blacklisting, etc. /All this works perfectly./
My problem is that I need to download software updates (debian apt-get http) for the firewall/router from a repository out on the internet.
The firewall/router can successfully ping the repo-server when I force the source address like this:
ping -I 217.154.193.209 163.1.221.67... but a simple "ping 163.1.221.67" (i.e. using the default source address selection algorithm) fails. Wireshark confirms these unanswered packets go out on eth0 with a source address of 172.16.101.2.
I believe I should be able to resolve this problem with iproute2 policy routing, but so far I have not been successful. I've tried several variations, but they all give me the same "wrong" source address.
Here is my simplest effort:
brian@chenin:~$ ip rule list 0: from all lookup local 32765: from 172.16.101.2 lookup CHENIN_ONLY 32766: from all lookup main 32767: from all lookup defaultbrian@chenin:~$ ip route list table CHENIN_ONLY 163.1.221.67 via 172.16.101.1 dev eth0 src 217.154.193.209brian@chenin:~$ sudo ip route flush cache brian@chenin:~$ ip route get 163.1.221.67 163.1.221.67 via 172.16.101.1 dev eth0 src 172.16.101.2 cacheThis shows me the source address in my policy rule has NOT been used, or the routing table entry does not work the way I think.
The only rule table with higher priority than
CHENIN_ONLYis local, which contains routes for addresses local to the firewall/router - nothing about remote addresses, i.e.brian@chenin:~$ ip route list table local broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast 172.16.101.0 dev eth0 proto kernel scope link src 172.16.101.2 local 172.16.101.2 dev eth0 proto kernel scope host src 172.16.101.2 broadcast 172.16.101.255 dev eth0 proto kernel scope link src 172.16.101.2 broadcast 217.154.193.208 dev eth1 proto kernel scope link src 217.154.193.209 local 217.154.193.209 dev eth1 proto kernel scope host src 217.154.193.209 broadcast 217.154.193.223 dev eth1 proto kernel scope link src 217.154.193.209and the main table looks like this:
brian@chenin:~$ ip route list table main default via 172.16.101.1 dev eth0 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209I wonder whether my unsuccessful "naked" pings to 163.1.221.67 are passing through the tables three times:
1. My explicit policy should be applied to select a source address
of 217.154.193.209.2. The main table is used to select the default route via 172.16.101.1:
3. The main table then provides the explicit route to the default router, which seems to rewrite the source address to 172.16.101.2.I have read the latest lartc HOWTO sections relevant to policy routing, but didn't see anything similar to my situation. I also didn't see how to get a verbose (i.e. blow by blow) output from the "ip route get" command to show how it is selecting its route.
Am I trying to do the impossible here, or am I just making a mistake in the way I am doing it?
I hope this strikes someone as an interesting question. If I can achieve a solution, I would be happy to add the scenario to the HOWTO.
Brian
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Erik Auerswald
Hi Brian,
Quoted 2 lines · Brian Burch
The mailing list has been dormant for 2 years, so I wonder whether anyone is still listening for new questions?
You never know. ;-)
Well, it is great to discover three people were listening and cared enough to help me with my puzzle. Thanks very much.
Quoted 36 lines
Quoted 29 lines
My broadband router runs PPPoA and is dynamically assigned a single ipv4 internet address by my ISP. I have a static subnet which I host on a linux router/firewall (called chenin). The linux firewall and the adsl router communicate via a non-internet-addressable private subnet. Here is the topology:
Internet -- adsl-router-ppp0-ipv4-dynamic
-- adsl-router-eth0-172.16.101.1 -- -- firewall-router-eth0-172.16.101.2 -- firewall-router-217.154.193.209 -- -- static-subnet-hosts-217.154.193.154.208/28Each of the hosts on the static subnet use 217.154.193.209 as their own default route. The adsl router forwards all incoming packets to the firewall/router's eth0. The firewall/router forwards all incoming packets to the static subnet via its own eth1. The firewall/router does not need to perform NAT, but it implements a simple set of iptables rules for blacklisting, etc. /All this works perfectly./
My problem is that I need to download software updates (debian apt-get http) for the firewall/router from a repository out on the internet.
The firewall/router can successfully ping the repo-server when I force the source address like this:
Quoted 6 lines
ping -I 217.154.193.209 163.1.221.67... but a simple "ping 163.1.221.67" (i.e. using the default source address selection algorithm) fails. Wireshark confirms these unanswered packets go out on eth0 with a source address of 172.16.101.2.
Quoted 4 lines
I see several generic ways to solve this. Use one of:
1) Use NAT for the transfer network addresses (i.e. "masquerading" for the
172.16.101.2 address) on the ADSL router.
Yes, NAT would certainly work. However, I always felt it was overkill for what I felt looked like a routing problem. NAT is essential when a host ONLY has interfaces that are non-internet-addressable (aka private ipv4 addresses). This particular host has an interface assigned an internet-routeable address, so I felt the most elegant solution would be to somehow coerce the source address to that of the "wrong" interface in similar manner to the way ping's "-I" argument does.
Quoted 2 lines
2) Use SNAT from 172.16.101.2 to 217.154.193.209 on the firewall for packets going to the internet.
Yes, I can see that would work too, but it is still NAT.
Quoted 1 line
3) Use a subnet from your routable IP addresses for the transit network.
Good point. I have a couple of spare addresses that I use occasionally when working on the dirty side of my real firewalls. I realised I could permanently assign a second subnet address to the router/firewall, but there were three things that made me hesitate:
a) I don't have many spare addresses, and I'm stingy.
b) I couldn't sub-net my subnet without losing too many addresses. I don't want to redesign my internet-facing services to use fewer addresses.
c) I couldn't see a simple way to "hang" an extra address on the router/firewall so that it would have highest preference during source address selection.
Quoted 1 line
4) Terminate PPPoE on the firewall, using the ADSL router as an ADSL modem.
Been there and got the T-shirt! My ISP is very good to me and has some excellent tech documents on why many UK ADSL users can't use PPPoE, even though it initially seems to work. I learnt the hard way that my own line is currently assigned to buggy exchange equipment that doesn't and will never work properly with PPPoE. That is why I reverted to using PPPoA and had to use a private subnet between the ADSL router and the internet-subnet router.
Quoted 2 lines
You may be able to define the source IP to use for the update process. If so, you do not need any of the above.
Hmmm. I don't know how to do that with apt-get.
Quoted 6 lines
Quoted 4 lines
I believe I should be able to resolve this problem with iproute2 policy routing, but so far I have not been successful. I've tried several variations, but they all give me the same "wrong" source address.
Policy routing is used _after_ deciding on the source IP address to use.
Yes... Joel also pointed out the same restriction. (I think you mis-typed and meant to say "policy routing happens _before_ selection of the source IP address").
Quoted 6 lines
Quoted 2 lines
Am I trying to do the impossible here, or am I just making a mistake in the way I am doing it?
You need to change the source IP address seen on the internet, that cannot be done with policy routing. It can be done using NAT, network renumbering or software configuration.
Joel suggested a way to turn the problem upside down and solve it with policy routing, so I am going to try his approach first.
Quoted 1 line
HTH,
OH YES IT DID! Thanks very much for your suggestions. I hope you are not offended that I am currently shelving them if/until I become more desperate.
Regards,
Brian
Quoted 2 lines
Erik
Hi Brian,
Quoted 52 lines · Brian Burch
Quoted 37 lines · Erik Auerswald
Quoted 29 lines · Brian Burch
My broadband router runs PPPoA and is dynamically assigned a single ipv4 internet address by my ISP. I have a static subnet which I host on a linux router/firewall (called chenin). The linux firewall and the adsl router communicate via a non-internet-addressable private subnet. Here is the topology:
Internet -- adsl-router-ppp0-ipv4-dynamic
-- adsl-router-eth0-172.16.101.1 -- -- firewall-router-eth0-172.16.101.2 -- firewall-router-217.154.193.209 -- -- static-subnet-hosts-217.154.193.154.208/28Each of the hosts on the static subnet use 217.154.193.209 as their own default route. The adsl router forwards all incoming packets to the firewall/router's eth0. The firewall/router forwards all incoming packets to the static subnet via its own eth1. The firewall/router does not need to perform NAT, but it implements a simple set of iptables rules for blacklisting, etc. /All this works perfectly./
My problem is that I need to download software updates (debian apt-get http) for the firewall/router from a repository out on the internet.
The firewall/router can successfully ping the repo-server when I force the source address like this:
Quoted 6 lines
ping -I 217.154.193.209 163.1.221.67... but a simple "ping 163.1.221.67" (i.e. using the default source address selection algorithm) fails. Wireshark confirms these unanswered packets go out on eth0 with a source address of 172.16.101.2.
Quoted 5 lines
I see several generic ways to solve this. Use one of:
1) Use NAT for the transfer network addresses (i.e. "masquerading" for the
172.16.101.2 address) on the ADSL router.Yes, NAT would certainly work. However, I always felt it was overkill for what I felt looked like a routing problem. NAT is essential when a host ONLY has interfaces that are non-internet-addressable (aka private ipv4 addresses). This particular host has an interface assigned an internet-routeable address, so I felt the most elegant solution would be to somehow coerce the source address to that of the "wrong" interface in similar manner to the way ping's "-I" argument does.
Quoted 4 lines
Quoted 2 lines
2) Use SNAT from 172.16.101.2 to 217.154.193.209 on the firewall for packets going to the internet.
Yes, I can see that would work too, but it is still NAT.
Indeed, and IMHO that is the best solution. It is implemented on the machine that needs to choose the correct IP address.
Quoted 12 lines
Quoted 1 line
3) Use a subnet from your routable IP addresses for the transit network.
Good point. I have a couple of spare addresses that I use occasionally when working on the dirty side of my real firewalls. I realised I could permanently assign a second subnet address to the router/firewall, but there were three things that made me hesitate:
a) I don't have many spare addresses, and I'm stingy.
b) I couldn't sub-net my subnet without losing too many addresses. I don't want to redesign my internet-facing services to use fewer addresses.
c) I couldn't see a simple way to "hang" an extra address on the router/firewall so that it would have highest preference during source address selection.
No surprise. ;-)
Quoted 10 lines
Quoted 2 lines
4) Terminate PPPoE on the firewall, using the ADSL router as an ADSL modem.
Been there and got the T-shirt! My ISP is very good to me and has some excellent tech documents on why many UK ADSL users can't use PPPoE, even though it initially seems to work. I learnt the hard way that my own line is currently assigned to buggy exchange equipment that doesn't and will never work properly with PPPoE. That is why I reverted to using PPPoA and had to use a private subnet between the ADSL router and the internet-subnet router.
My fault, I did not read carefully and did not notice the A(TM)...
Quoted 4 lines
Quoted 2 lines
You may be able to define the source IP to use for the update process. If so, you do not need any of the above.
Hmmm. I don't know how to do that with apt-get.
I did a quick check and did not see a configuration option for this. :-(
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
I believe I should be able to resolve this problem with iproute2 policy routing, but so far I have not been successful. I've tried several variations, but they all give me the same "wrong" source address.
Policy routing is used _after_ deciding on the source IP address to use.
Yes... Joel also pointed out the same restriction. (I think you mis-typed and meant to say "policy routing happens _before_ selection of the source IP address").
Well, the interface to use and thus the source IP address is decided by a route table lookup. But a source specific route can be found only _after_ the IP has been chosen.
Quoted 11 lines
Quoted 8 lines
Quoted 2 lines
Am I trying to do the impossible here, or am I just making a mistake in the way I am doing it?
You need to change the source IP address seen on the internet, that cannot
be done with policy routing. It can be done using NAT, network renumbering
or software configuration.Joel suggested a way to turn the problem upside down and solve it with policy routing, so I am going to try his approach first.
Please report to the list if this works (and how exactly).
You want the IP packet to egress the interface to the ADSL router, but use the IP address of the internal interface as source address. Linux determines the source address by using the egress interface's address.
[If a source address is specified by the application, that is used instead (the 'ping -I' case).]
To replace a source address I'd use SNAT. I don't know the exact command off the top of my head, though.
I am really curious if the problem can be solved with policy routing as well. :-)
Regards,
Erik
Quoted 42 lines · Brian Burch
Quoted 7 lines · Joel Gerber
Your problem is one I've come up against a number of times in the past. You are definitely on the right route using "policy routing" (otherwise known as utilizing IP Rules), but the snag you're hitting is that the source address of a locally generated packet isn't chosen until a destination route has already been selected. Using the from statement in an ip rules table is only useful when you're routing packets from other subnets.
What a quick and helpful reply! Not only did my problem motivate you to respond, but I am very relieved to find the answer isn't just RTFM.
Your explanation about the timing for kernel selection of the source address is crucial and now my failed experiments all make sense. Your suggested solution also makes sense, although it is counter-intuitive... at least to me!
Quoted 3 lines
Now, just so you know, the local table should normally be left alone. It's used for locally connected subnets only. It is given the highest priority on purpose... don't mess with it! :)
I read somewhere (possibly the howto guide or the ip-cref) a statement that the kernel treats this table as read-only. My system is running ubuntu 13.04 with a custom non-pae 3.8.0 kernel, so it is reasonably modern. When I thought more about the purpose of this table, I decided there would be no point in trying to modify it, even if that was possible.
Quoted 15 lines
There are a few methods you could try here, but I'm only going to recommend one of them. Have your main routing table hold the routes that your local machine will use, and then have a separate table for all routed hosts. Then, use ip rule statements to force any traffic coming in from a routed host into that routing table.
IE: on firewall-router (assuming eth1 is facing your static subnet hosts):
# ip rule list 0: from all lookup local 32765: iif eth1 lookup static-hosts 32766: from all lookup main 32767: from all lookup default
Quoted 7 lines
Quoted 3 lines
Have table main include all of the routes your local firewall-router should use, and have static-hosts contain all of the routes that your static-subnet-hosts should use.
Great! I understand the principle you propose and will try to implement it over the weekend. I will update this thread when I have something to report.
Excellent news!! My router/firewall is currently collecting 200+ updated packages from the repository server!
The solution was achieved using ONLY policy routing, based on Joel's suggested approach.
However, I hit some fairly nasty snags along the way and had to do a lot of trial and error fiddling to make it work. I am certain my working solution can be made more elegant, but that will take a couple of hours pen-and-pencil time to map out the current routing table entries in selection order and then perform some experiments.
I don't have time to do it at the moment, but will get started over the next couple of days. If I get stuck, I will post the "current best" configuration and ask for help to improve it.
Regards,
Brian
Quoted 140 lines
Once again, thanks for helping,
Brian
Quoted 122 lines
Joel Gerber
Network Specialist
Network Operations
Eastlink
E: [email protected] T: 519.786.1241 -----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Brian Burch
Sent: December-20-13 7:05 AM
To: Linux Advanced Routing
Subject: How to override the default source ipv4 address on packets originating from a routerThe mailing list has been dormant for 2 years, so I wonder whether anyone is still listening for new questions?
My broadband router runs PPPoA and is dynamically assigned a single ipv4 internet address by my ISP. I have a static subnet which I host on a linux router/firewall (called chenin). The linux firewall and the adsl router communicate via a non-internet-addressable private subnet. Here is the topology:
Internet -- adsl-router-ppp0-ipv4-dynamic
-- adsl-router-eth0-172.16.101.1 -- -- firewall-router-eth0-172.16.101.2 -- firewall-router-217.154.193.209 -- -- static-subnet-hosts-217.154.193.154.208/28Each of the hosts on the static subnet use 217.154.193.209 as their own default route. The adsl router forwards all incoming packets to the firewall/router's eth0. The firewall/router forwards all incoming packets to the static subnet via its own eth1. The firewall/router does not need to perform NAT, but it implements a simple set of iptables rules for blacklisting, etc. /All this works perfectly./
My problem is that I need to download software updates (debian apt-get http) for the firewall/router from a repository out on the internet.
The firewall/router can successfully ping the repo-server when I force the source address like this:
ping -I 217.154.193.209 163.1.221.67... but a simple "ping 163.1.221.67" (i.e. using the default source address selection algorithm) fails. Wireshark confirms these unanswered packets go out on eth0 with a source address of 172.16.101.2.
I believe I should be able to resolve this problem with iproute2 policy routing, but so far I have not been successful. I've tried several variations, but they all give me the same "wrong" source address.
Here is my simplest effort:
brian@chenin:~$ ip rule list 0: from all lookup local 32765: from 172.16.101.2 lookup CHENIN_ONLY 32766: from all lookup main 32767: from all lookup defaultbrian@chenin:~$ ip route list table CHENIN_ONLY 163.1.221.67 via 172.16.101.1 dev eth0 src 217.154.193.209brian@chenin:~$ sudo ip route flush cache brian@chenin:~$ ip route get 163.1.221.67 163.1.221.67 via 172.16.101.1 dev eth0 src 172.16.101.2 cacheThis shows me the source address in my policy rule has NOT been used, or the routing table entry does not work the way I think.
The only rule table with higher priority than
CHENIN_ONLYis local, which contains routes for addresses local to the firewall/router - nothing about remote addresses, i.e.brian@chenin:~$ ip route list table local broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast 172.16.101.0 dev eth0 proto kernel scope link src 172.16.101.2 local 172.16.101.2 dev eth0 proto kernel scope host src 172.16.101.2 broadcast 172.16.101.255 dev eth0 proto kernel scope link src 172.16.101.2 broadcast 217.154.193.208 dev eth1 proto kernel scope link src 217.154.193.209 local 217.154.193.209 dev eth1 proto kernel scope host src 217.154.193.209 broadcast 217.154.193.223 dev eth1 proto kernel scope link src 217.154.193.209and the main table looks like this:
brian@chenin:~$ ip route list table main default via 172.16.101.1 dev eth0 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209I wonder whether my unsuccessful "naked" pings to 163.1.221.67 are passing through the tables three times:
1. My explicit policy should be applied to select a source address
of 217.154.193.209.2. The main table is used to select the default route via 172.16.101.1:
3. The main table then provides the explicit route to the default router, which seems to rewrite the source address to 172.16.101.2.I have read the latest lartc HOWTO sections relevant to policy routing, but didn't see anything similar to my situation. I also didn't see how to get a verbose (i.e. blow by blow) output from the "ip route get" command to show how it is selecting its route.
Am I trying to do the impossible here, or am I just making a mistake in the way I am doing it?
Quoted 9 lines
I hope this strikes someone as an interesting question. If I can achieve a solution, I would be happy to add the scenario to the HOWTO.
Brian
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 64 lines · Brian Burch
Quoted 43 lines · Brian Burch
Quoted 7 lines · Joel Gerber
Your problem is one I've come up against a number of times in the past. You are definitely on the right route using "policy routing" (otherwise known as utilizing IP Rules), but the snag you're hitting is that the source address of a locally generated packet isn't chosen until a destination route has already been selected. Using the from statement in an ip rules table is only useful when you're routing packets from other subnets.
What a quick and helpful reply! Not only did my problem motivate you to respond, but I am very relieved to find the answer isn't just RTFM.
Your explanation about the timing for kernel selection of the source address is crucial and now my failed experiments all make sense. Your suggested solution also makes sense, although it is counter-intuitive... at least to me!
Quoted 3 lines
Now, just so you know, the local table should normally be left alone. It's used for locally connected subnets only. It is given the highest priority on purpose... don't mess with it! :)
I read somewhere (possibly the howto guide or the ip-cref) a statement that the kernel treats this table as read-only. My system is running ubuntu 13.04 with a custom non-pae 3.8.0 kernel, so it is reasonably modern. When I thought more about the purpose of this table, I decided there would be no point in trying to modify it, even if that was possible.
Quoted 15 lines
There are a few methods you could try here, but I'm only going to recommend one of them. Have your main routing table hold the routes that your local machine will use, and then have a separate table for all routed hosts. Then, use ip rule statements to force any traffic coming in from a routed host into that routing table.
IE: on firewall-router (assuming eth1 is facing your static subnet hosts):
# ip rule list 0: from all lookup local 32765: iif eth1 lookup static-hosts 32766: from all lookup main 32767: from all lookup defaultQuoted 7 lines
Quoted 3 lines
Have table main include all of the routes your local firewall-router should use, and have static-hosts contain all of the routes that your static-subnet-hosts should use.
Great! I understand the principle you propose and will try to implement it over the weekend. I will update this thread when I have something to report.
Excellent news!! My router/firewall is currently collecting 200+ updated packages from the repository server!
The solution was achieved using ONLY policy routing, based on Joel's suggested approach.
However, I hit some fairly nasty snags along the way and had to do a lot of trial and error fiddling to make it work. I am certain my working solution can be made more elegant, but that will take a couple of hours pen-and-pencil time to map out the current routing table entries in selection order and then perform some experiments.
Quoted 3 lines
I don't have time to do it at the moment, but will get started over the next couple of days. If I get stuck, I will post the "current best" configuration and ask for help to improve it.
<snip>
I tried to improve my "current best" definitions, but my efforts to remove apparent redundancy have all failed. Here is the setup:
brian@chenin:~$ ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UNKNOWN qlen 1000
link/ether 00:15:8a:01:91:e3 brd ff:ff:ff:ff:ff:ff
inet 217.154.193.209/28 brd 217.154.193.223 scope global eth1
inet6 fe80::215:8aff:fe01:91e3/64 scope link
valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UNKNOWN qlen 1000
link/ether 00:40:63:f5:a5:72 brd ff:ff:ff:ff:ff:ff
inet 172.16.101.2/24 brd 172.16.101.255 scope global eth0
inet6 fe80::240:63ff:fef5:a572/64 scope link
valid_lft forever preferred_lft foreverbrian@chenin:~$ ip rule list 0: from all lookup local 32764: from 217.154.193.209 lookup CHENIN_TO_INTERNET 32765: from all iif eth1 lookup SUBNET_217 32766: from all lookup main 32767: from all lookup default
brian@chenin:~$ ip route list table CHENIN_TO_INTERNET 163.1.221.67 via 172.16.101.1 dev eth0
brian@chenin:~$ ip route list table SUBNET_217 default via 172.16.101.1 dev eth0
brian@chenin:~$ ip route list table main 163.1.221.67 dev eth0 scope link src 217.154.193.209 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209
I know Joel's suggestion did not include my CHENIN_TO_INTERNET rule, but without it chenin simply times out on its http gets.
I also tried keeping CHENIN_TO_INTERNET, and removing the explicit route in the main table, but that fails too, although I wasn't very surprised.
I have tried trimming these rules and routes to discover the minimum requirement, but this seems to be the smallest working subset.
Here is my theory to account for what is happening:
1. The SUBNET_217 default route MUST be able to handle ANY traffic arriving on eth1. When I changed the rule to match on just 217.154.193.208/28, the forwarded traffic became very glitchy, with poor response times and some dropped tcp sessions.
2. The route to 163.1.221.67 in the main table is required to assign the eth1 source address 217.154.193.209 to its outbound packets. This works because the kernel cannot find any other matching routes from chenin to 163.1.221.67 (which might encourage it to assign eth0's 172.16.101.2 as the source address).
4. Having assigned the 217.154.193.209 source address, the packet must be routed to the next hop. The CHENIN_TO_INTERNET rule matches the explicit source host address at a higher priority than the SUBNET_217 rule, so 172.16.101 is selected.
5. The main table has a route to 172.16.101.1 via eth0, so the packet is sent on its way.
That seems to be three passes through the policy routing tables. Am I correct?
I still feel it should be possible to improve the solution, but I don't understand what is going on well enough to see how. I would be very grateful if anyone can enlighten me.
Thanks,
Brian
You're going to have to re-acquaint me with your network layout. I'm having a hard time remembering what all of your interfaces are being used for and how they fit into your network.
Joel Gerber
Network Specialist
Network Operations
Eastlink
E: [email protected] T: 519.786.1241
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Brian Burch
Sent: January-08-14 11:18 AM
To: Linux Advanced Routing List
Subject: Re: How to override the default source ipv4 address on packets originating from a router
Quoted 64 lines · Brian Burch
Quoted 43 lines · Brian Burch
Quoted 7 lines · Joel Gerber
Your problem is one I've come up against a number of times in the past. You are definitely on the right route using "policy routing" (otherwise known as utilizing IP Rules), but the snag you're hitting is that the source address of a locally generated packet isn't chosen until a destination route has already been selected. Using the from statement in an ip rules table is only useful when you're routing packets from other subnets.
What a quick and helpful reply! Not only did my problem motivate you to respond, but I am very relieved to find the answer isn't just RTFM.
Your explanation about the timing for kernel selection of the source address is crucial and now my failed experiments all make sense. Your suggested solution also makes sense, although it is counter-intuitive... at least to me!
Quoted 3 lines
Now, just so you know, the local table should normally be left alone. It's used for locally connected subnets only. It is given the highest priority on purpose... don't mess with it! :)
I read somewhere (possibly the howto guide or the ip-cref) a statement that the kernel treats this table as read-only. My system is running ubuntu 13.04 with a custom non-pae 3.8.0 kernel, so it is reasonably modern. When I thought more about the purpose of this table, I decided there would be no point in trying to modify it, even if that was possible.
Quoted 15 lines
There are a few methods you could try here, but I'm only going to recommend one of them. Have your main routing table hold the routes that your local machine will use, and then have a separate table for all routed hosts. Then, use ip rule statements to force any traffic coming in from a routed host into that routing table.
IE: on firewall-router (assuming eth1 is facing your static subnet hosts):
# ip rule list 0: from all lookup local 32765: iif eth1 lookup static-hosts 32766: from all lookup main 32767: from all lookup defaultQuoted 7 lines
Quoted 3 lines
Have table main include all of the routes your local firewall-router should use, and have static-hosts contain all of the routes that your static-subnet-hosts should use.
Great! I understand the principle you propose and will try to implement it over the weekend. I will update this thread when I have something to report.
Excellent news!! My router/firewall is currently collecting 200+ updated packages from the repository server!
The solution was achieved using ONLY policy routing, based on Joel's suggested approach.
However, I hit some fairly nasty snags along the way and had to do a lot of trial and error fiddling to make it work. I am certain my working solution can be made more elegant, but that will take a couple of hours pen-and-pencil time to map out the current routing table entries in selection order and then perform some experiments.
Quoted 3 lines
I don't have time to do it at the moment, but will get started over the next couple of days. If I get stuck, I will post the "current best" configuration and ask for help to improve it.
<snip>
I tried to improve my "current best" definitions, but my efforts to remove apparent redundancy have all failed. Here is the setup:
brian@chenin:~$ ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:15:8a:01:91:e3 brd ff:ff:ff:ff:ff:ff
inet 217.154.193.209/28 brd 217.154.193.223 scope global eth1
inet6 fe80::215:8aff:fe01:91e3/64 scope link
valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:40:63:f5:a5:72 brd ff:ff:ff:ff:ff:ff
inet 172.16.101.2/24 brd 172.16.101.255 scope global eth0
inet6 fe80::240:63ff:fef5:a572/64 scope link
valid_lft forever preferred_lft foreverbrian@chenin:~$ ip rule list 0: from all lookup local 32764: from 217.154.193.209 lookup CHENIN_TO_INTERNET 32765: from all iif eth1 lookup SUBNET_217 32766: from all lookup main 32767: from all lookup default
brian@chenin:~$ ip route list table CHENIN_TO_INTERNET 163.1.221.67 via 172.16.101.1 dev eth0
brian@chenin:~$ ip route list table SUBNET_217 default via 172.16.101.1 dev eth0
brian@chenin:~$ ip route list table main 163.1.221.67 dev eth0 scope link src 217.154.193.209 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209
I know Joel's suggestion did not include my CHENIN_TO_INTERNET rule, but without it chenin simply times out on its http gets.
I also tried keeping CHENIN_TO_INTERNET, and removing the explicit route in the main table, but that fails too, although I wasn't very surprised.
I have tried trimming these rules and routes to discover the minimum requirement, but this seems to be the smallest working subset.
Here is my theory to account for what is happening:
1. The SUBNET_217 default route MUST be able to handle ANY traffic arriving on eth1. When I changed the rule to match on just 217.154.193.208/28, the forwarded traffic became very glitchy, with poor response times and some dropped tcp sessions.
2. The route to 163.1.221.67 in the main table is required to assign the eth1 source address 217.154.193.209 to its outbound packets. This works because the kernel cannot find any other matching routes from chenin to 163.1.221.67 (which might encourage it to assign eth0's 172.16.101.2 as the source address).
4. Having assigned the 217.154.193.209 source address, the packet must be routed to the next hop. The CHENIN_TO_INTERNET rule matches the explicit source host address at a higher priority than the SUBNET_217 rule, so 172.16.101 is selected.
5. The main table has a route to 172.16.101.1 via eth0, so the packet is sent on its way.
That seems to be three passes through the policy routing tables. Am I correct?
I still feel it should be possible to improve the solution, but I don't understand what is going on well enough to see how. I would be very grateful if anyone can enlighten me.
Thanks,
Brian
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Joel Gerber
You're going to have to re-acquaint me with your network layout. I'm having a hard time remembering what all of your interfaces are being used for and how they fit into your network.
Oh! Sorry, Joel. I forgot this is a simple mailing list, without a cute threaded web interface. I "helpfully" snipped out our initial discussion to lighten the payload...
I will resend my last post within the entire reply, then you will have everything in one place. Discard this one and the "full" version will follow soon.
Thanks,
Brian
Quoted 173 lines
Joel Gerber
Network Specialist
Network Operations
Eastlink
E: [email protected] T: 519.786.1241-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Brian Burch
Sent: January-08-14 11:18 AM
To: Linux Advanced Routing List
Subject: Re: How to override the default source ipv4 address on packets originating from a routerQuoted 64 lines · Brian Burch
Quoted 43 lines · Brian Burch
Quoted 7 lines · Joel Gerber
Your problem is one I've come up against a number of times in the past. You are definitely on the right route using "policy routing" (otherwise known as utilizing IP Rules), but the snag you're hitting is that the source address of a locally generated packet isn't chosen until a destination route has already been selected. Using the from statement in an ip rules table is only useful when you're routing packets from other subnets.
What a quick and helpful reply! Not only did my problem motivate you to respond, but I am very relieved to find the answer isn't just RTFM.
Your explanation about the timing for kernel selection of the source address is crucial and now my failed experiments all make sense. Your suggested solution also makes sense, although it is counter-intuitive... at least to me!
Quoted 3 lines
Now, just so you know, the local table should normally be left alone. It's used for locally connected subnets only. It is given the highest priority on purpose... don't mess with it! :)
I read somewhere (possibly the howto guide or the ip-cref) a statement that the kernel treats this table as read-only. My system is running ubuntu 13.04 with a custom non-pae 3.8.0 kernel, so it is reasonably modern. When I thought more about the purpose of this table, I decided there would be no point in trying to modify it, even if that was possible.
Quoted 15 lines
There are a few methods you could try here, but I'm only going to recommend one of them. Have your main routing table hold the routes that your local machine will use, and then have a separate table for all routed hosts. Then, use ip rule statements to force any traffic coming in from a routed host into that routing table.
IE: on firewall-router (assuming eth1 is facing your static subnet hosts):
# ip rule list 0: from all lookup local 32765: iif eth1 lookup static-hosts 32766: from all lookup main 32767: from all lookup defaultQuoted 7 lines
Quoted 3 lines
Have table main include all of the routes your local firewall-router should use, and have static-hosts contain all of the routes that your static-subnet-hosts should use.
Great! I understand the principle you propose and will try to implement it over the weekend. I will update this thread when I have something to report.
Excellent news!! My router/firewall is currently collecting 200+ updated packages from the repository server!
The solution was achieved using ONLY policy routing, based on Joel's suggested approach.
However, I hit some fairly nasty snags along the way and had to do a lot of trial and error fiddling to make it work. I am certain my working solution can be made more elegant, but that will take a couple of hours pen-and-pencil time to map out the current routing table entries in selection order and then perform some experiments.
Quoted 3 lines
I don't have time to do it at the moment, but will get started over the next couple of days. If I get stuck, I will post the "current best" configuration and ask for help to improve it.
<snip>
I tried to improve my "current best" definitions, but my efforts to remove apparent redundancy have all failed. Here is the setup:
brian@chenin:~$ ip address show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000 link/ether 00:15:8a:01:91:e3 brd ff:ff:ff:ff:ff:ff inet 217.154.193.209/28 brd 217.154.193.223 scope global eth1 inet6 fe80::215:8aff:fe01:91e3/64 scope link valid_lft forever preferred_lft forever 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000 link/ether 00:40:63:f5:a5:72 brd ff:ff:ff:ff:ff:ff inet 172.16.101.2/24 brd 172.16.101.255 scope global eth0 inet6 fe80::240:63ff:fef5:a572/64 scope link valid_lft forever preferred_lft foreverbrian@chenin:~$ ip rule list 0: from all lookup local 32764: from 217.154.193.209 lookup CHENIN_TO_INTERNET 32765: from all iif eth1 lookup SUBNET_217 32766: from all lookup main 32767: from all lookup defaultbrian@chenin:~$ ip route list table CHENIN_TO_INTERNET 163.1.221.67 via 172.16.101.1 dev eth0brian@chenin:~$ ip route list table SUBNET_217 default via 172.16.101.1 dev eth0brian@chenin:~$ ip route list table main 163.1.221.67 dev eth0 scope link src 217.154.193.209 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209I know Joel's suggestion did not include my
CHENIN_TO_INTERNETrule, but without it chenin simply times out on its http gets.I also tried keeping
CHENIN_TO_INTERNET, and removing the explicit route in the main table, but that fails too, although I wasn't very surprised.I have tried trimming these rules and routes to discover the minimum requirement, but this seems to be the smallest working subset.
Here is my theory to account for what is happening:
1. The
SUBNET_217default route MUST be able to handle ANY traffic arriving on eth1. When I changed the rule to match on just 217.154.193.208/28, the forwarded traffic became very glitchy, with poor response times and some dropped tcp sessions.2. The route to 163.1.221.67 in the main table is required to assign the eth1 source address 217.154.193.209 to its outbound packets. This works because the kernel cannot find any other matching routes from chenin to 163.1.221.67 (which might encourage it to assign eth0's 172.16.101.2 as the source address).
4. Having assigned the 217.154.193.209 source address, the packet must be routed to the next hop. The
CHENIN_TO_INTERNETrule matches the explicit source host address at a higher priority than theSUBNET_217rule, so 172.16.101 is selected.5. The main table has a route to 172.16.101.1 via eth0, so the packet is sent on its way.
That seems to be three passes through the policy routing tables. Am I correct?
I still feel it should be possible to improve the solution, but I don't understand what is going on well enough to see how. I would be very grateful if anyone can enlighten me.
Thanks,
Brian
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
[Unsnipped version of previous post follows]
Quoted 64 lines · Brian Burch
Quoted 43 lines · Brian Burch
Quoted 7 lines · Joel Gerber
Your problem is one I've come up against a number of times in the past. You are definitely on the right route using "policy routing" (otherwise known as utilizing IP Rules), but the snag you're hitting is that the source address of a locally generated packet isn't chosen until a destination route has already been selected. Using the from statement in an ip rules table is only useful when you're routing packets from other subnets.
What a quick and helpful reply! Not only did my problem motivate you to respond, but I am very relieved to find the answer isn't just RTFM.
Your explanation about the timing for kernel selection of the source address is crucial and now my failed experiments all make sense. Your suggested solution also makes sense, although it is counter-intuitive... at least to me!
Quoted 3 lines
Now, just so you know, the local table should normally be left alone. It's used for locally connected subnets only. It is given the highest priority on purpose... don't mess with it! :)
I read somewhere (possibly the howto guide or the ip-cref) a statement that the kernel treats this table as read-only. My system is running ubuntu 13.04 with a custom non-pae 3.8.0 kernel, so it is reasonably modern. When I thought more about the purpose of this table, I decided there would be no point in trying to modify it, even if that was possible.
Quoted 15 lines
There are a few methods you could try here, but I'm only going to recommend one of them. Have your main routing table hold the routes that your local machine will use, and then have a separate table for all routed hosts. Then, use ip rule statements to force any traffic coming in from a routed host into that routing table.
IE: on firewall-router (assuming eth1 is facing your static subnet hosts):
# ip rule list 0: from all lookup local 32765: iif eth1 lookup static-hosts 32766: from all lookup main 32767: from all lookup defaultQuoted 7 lines
Quoted 3 lines
Have table main include all of the routes your local firewall-router should use, and have static-hosts contain all of the routes that your static-subnet-hosts should use.
Great! I understand the principle you propose and will try to implement it over the weekend. I will update this thread when I have something to report.
Excellent news!! My router/firewall is currently collecting 200+ updated packages from the repository server!
The solution was achieved using ONLY policy routing, based on Joel's suggested approach.
However, I hit some fairly nasty snags along the way and had to do a lot of trial and error fiddling to make it work. I am certain my working solution can be made more elegant, but that will take a couple of hours pen-and-pencil time to map out the current routing table entries in selection order and then perform some experiments.
Quoted 3 lines
I don't have time to do it at the moment, but will get started over the next couple of days. If I get stuck, I will post the "current best" configuration and ask for help to improve it.
I tried to improve my "current best" definitions, but my efforts to remove apparent redundancy have all failed. Here is the setup:
brian@chenin:~$ ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UNKNOWN qlen 1000
link/ether 00:15:8a:01:91:e3 brd ff:ff:ff:ff:ff:ff
inet 217.154.193.209/28 brd 217.154.193.223 scope global eth1
inet6 fe80::215:8aff:fe01:91e3/64 scope link
valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UNKNOWN qlen 1000
link/ether 00:40:63:f5:a5:72 brd ff:ff:ff:ff:ff:ff
inet 172.16.101.2/24 brd 172.16.101.255 scope global eth0
inet6 fe80::240:63ff:fef5:a572/64 scope link
valid_lft forever preferred_lft foreverbrian@chenin:~$ ip rule list 0: from all lookup local 32764: from 217.154.193.209 lookup CHENIN_TO_INTERNET 32765: from all iif eth1 lookup SUBNET_217 32766: from all lookup main 32767: from all lookup default
brian@chenin:~$ ip route list table CHENIN_TO_INTERNET 163.1.221.67 via 172.16.101.1 dev eth0
brian@chenin:~$ ip route list table SUBNET_217 default via 172.16.101.1 dev eth0
brian@chenin:~$ ip route list table main 163.1.221.67 dev eth0 scope link src 217.154.193.209 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209
I know Joel's suggestion did not include my CHENIN_TO_INTERNET rule, but without it chenin simply times out on its http gets.
I also tried keeping CHENIN_TO_INTERNET, and removing the explicit route in the main table, but that fails too, although I wasn't very surprised.
I have tried trimming these rules and routes to discover the minimum requirement, but this seems to be the smallest working subset.
Here is my theory to account for what is happening:
1. The SUBNET_217 default route MUST be able to handle ANY traffic arriving on eth1. When I changed the rule to match on just 217.154.193.208/28, the forwarded traffic became very glitchy, with poor response times and some dropped tcp sessions.
2. The route to 163.1.221.67 in the main table is required to assign the eth1 source address 217.154.193.209 to its outbound packets. This works because the kernel cannot find any other matching routes from chenin to 163.1.221.67 (which might encourage it to assign eth0's 172.16.101.2 as the source address).
4. Having assigned the 217.154.193.209 source address, the packet must be routed to the next hop. The CHENIN_TO_INTERNET rule matches the explicit source host address at a higher priority than the SUBNET_217 rule, so 172.16.101 is selected.
5. The main table has a route to 172.16.101.1 via eth0, so the packet is sent on its way.
That seems to be three passes through the policy routing tables. Am I correct?
I still feel it should be possible to improve the solution, but I don't understand what is going on well enough to see how. I would be very grateful if anyone can enlighten me.
Thanks,
Brian
Quoted 151 lines
Regards,
Brian
Quoted 138 lines
Once again, thanks for helping,
Brian
Quoted 124 lines
Joel Gerber
Network Specialist
Network Operations
Eastlink
E: [email protected] T: 519.786.1241 -----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Brian Burch
Sent: December-20-13 7:05 AM
To: Linux Advanced Routing
Subject: How to override the default source ipv4 address on packets originating from a routerThe mailing list has been dormant for 2 years, so I wonder whether anyone is still listening for new questions?
My broadband router runs PPPoA and is dynamically assigned a single ipv4 internet address by my ISP. I have a static subnet which I host on a linux router/firewall (called chenin). The linux firewall and the adsl router communicate via a non-internet-addressable private subnet. Here is the topology:
Internet -- adsl-router-ppp0-ipv4-dynamic
-- adsl-router-eth0-172.16.101.1 -- -- firewall-router-eth0-172.16.101.2 -- firewall-router-217.154.193.209 -- -- static-subnet-hosts-217.154.193.154.208/28Each of the hosts on the static subnet use 217.154.193.209 as their own default route. The adsl router forwards all incoming packets to the firewall/router's eth0. The firewall/router forwards all incoming packets to the static subnet via its own eth1. The firewall/router does not need to perform NAT, but it implements a simple set of iptables rules for blacklisting, etc. /All this works perfectly./
My problem is that I need to download software updates (debian apt-get http) for the firewall/router from a repository out on the internet.
The firewall/router can successfully ping the repo-server when I force the source address like this:
ping -I 217.154.193.209 163.1.221.67... but a simple "ping 163.1.221.67" (i.e. using the default source address selection algorithm) fails. Wireshark confirms these unanswered packets go out on eth0 with a source address of 172.16.101.2.
I believe I should be able to resolve this problem with iproute2 policy routing, but so far I have not been successful. I've tried several variations, but they all give me the same "wrong" source address.
Here is my simplest effort:
brian@chenin:~$ ip rule list 0: from all lookup local 32765: from 172.16.101.2 lookup CHENIN_ONLY 32766: from all lookup main 32767: from all lookup defaultbrian@chenin:~$ ip route list table CHENIN_ONLY 163.1.221.67 via 172.16.101.1 dev eth0 src 217.154.193.209brian@chenin:~$ sudo ip route flush cache brian@chenin:~$ ip route get 163.1.221.67 163.1.221.67 via 172.16.101.1 dev eth0 src 172.16.101.2 cacheThis shows me the source address in my policy rule has NOT been used, or the routing table entry does not work the way I think.
The only rule table with higher priority than
CHENIN_ONLYis local, which contains routes for addresses local to the firewall/router - nothing about remote addresses, i.e.brian@chenin:~$ ip route list table local broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast 172.16.101.0 dev eth0 proto kernel scope link src 172.16.101.2 local 172.16.101.2 dev eth0 proto kernel scope host src 172.16.101.2 broadcast 172.16.101.255 dev eth0 proto kernel scope link src 172.16.101.2 broadcast 217.154.193.208 dev eth1 proto kernel scope link src 217.154.193.209 local 217.154.193.209 dev eth1 proto kernel scope host src 217.154.193.209 broadcast 217.154.193.223 dev eth1 proto kernel scope link src 217.154.193.209and the main table looks like this:
brian@chenin:~$ ip route list table main default via 172.16.101.1 dev eth0 169.254.0.0/16 dev eth0 scope link metric 1000 172.16.101.0/24 dev eth0 proto kernel scope link src 172.16.101.2 217.154.193.208/28 dev eth1 proto kernel scope link src 217.154.193.209I wonder whether my unsuccessful "naked" pings to 163.1.221.67 are passing through the tables three times:
1. My explicit policy should be applied to select a source address
of 217.154.193.209.2. The main table is used to select the default route via 172.16.101.1:
3. The main table then provides the explicit route to the default router, which seems to rewrite the source address to 172.16.101.2.I have read the latest lartc HOWTO sections relevant to policy routing, but didn't see anything similar to my situation. I also didn't see how to get a verbose (i.e. blow by blow) output from the "ip route get" command to show how it is selecting its route.
Am I trying to do the impossible here, or am I just making a mistake in the way I am doing it?
Quoted 9 lines
I hope this strikes someone as an interesting question. If I can achieve a solution, I would be happy to add the scenario to the HOWTO.
Brian
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe lartc" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html