Skip to contentSkip to conversation

Fwmark problem - policy routing does not work.

LARTC Thilo Schulz 5 participants 12 messages Apr 23, 2002 – Apr 27, 2002

lore
Share & tools
lore
#0 lore raw

Well, I'm stating my problem from the beginning:
I have a computer in my LAN with 2 ips (192.168.1.10 and 192.168.1.11), the latter one created by "ip addr add" and users on that computer, and depending on the uid of the created packets i want those to take another source address. Of course, it should start with marking the packets with a simple iptables rule:

iptables -t mangle -A OUTPUT -m owner --uid-owner 1001 -j MARK --set-mark 1

so, next step should be, to create a new table, so i entered this into /etc/iproute2/rt_tables:
1 thilo

so that table 1 should have the identifier "thilo".

Next step is, to enter a rule for this table:
ip rule add fwmark 1 table thilo

and indeed:

router:/etc/init.d# ip rule ls
0:      from all lookup local
32765:  from all fwmark        1 lookup thilo
32766:  from all lookup main
32767:  from all lookup default
router:/etc/init.d#

now, i should only have to enter the route into the table:
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.11 proto kernel

a final "ip route flush cash", and this should be it.
As soon, as i log in as user "thilo" (uid 1001), and i for example connect per ssh to 192.168.1.4 (another computer in my lan), it _should_ show me logged in from 192.168.1.11, but instead it still shows:
thilo pts/8 Apr 23 18:37 (192.168.1.10).

The table as well as the route entry are correct, i am sure of this, for i have tested removing the fwmark rule and instead added:
ip rule add to 192.168.1.4 table thilo
everything worked then as i have intended, and it showed me logged in from 192.168.1.11 then.

So it might be, that the Marks are not being set, i checked that too, i replaced the
iptables -t mangle -A OUTPUT -m owner --uid-owner 1001 -j MARK --set-mark 1 with
iptables -t mangle -A OUTPUT -p tcp --dport 22 -j MARK --set-mark 1
- still no success. So I tried placing this rule into the PREROUTING chain (for testing though, INPUT should be absolutely correct), and - oh wonder - no success either ;)

Another problems checked too: I have enabled MARK based routing in the kernel (policy routing, everything switched on ..)

So I only can ask you on this mailing list - anyone got an idea of what is/what i did wrong?

- Thilo Schulz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw
Quoted 10 lines

\> So it might be, that the Marks are not being set, i checked that too, i

Quoted 7 lines

replaced the
iptables -t mangle -A OUTPUT -m owner --uid-owner 1001 -j MARK --set-mark 1 with
iptables -t mangle -A OUTPUT -p tcp --dport 22 -j MARK --set-mark 1
- still no success. So I tried placing this rule into the PREROUTING chain (for testing though, INPUT should be absolutely correct), and - oh wonder - no success either ;)

Quoted 3 lines

If you do "iptables -L -v -n -t mangle", do the counters increment ?

Stef

Yes, they do.
I am also sure now, that marking is working:
I have tried the same on another host that uses a gateway to access the internet, but has several ips on the net. If i do not enter the route (via option) the packets are not reaching their destination, if i am creating those as a given user, as all other user it does work. If i give the via option in the "ip route add table thilo src ....." command, the packets are reaching their destination again - yet even though i have given the source ip that should have been used, it is still using the primary one.

Could this be a bug in the routing code?

- Thilo Schulz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Thilo Schulz
Quoted 11 lines

Yes, they do.
I am also sure now, that marking is working:
I have tried the same on another host that uses a gateway to access the internet, but has several ips on the net. If i do not enter the route (via option) the packets are not reaching their destination, if i am creating those as a given user, as all other user it does work. If i give the via option in the "ip route add table thilo src ....." command, the packets are reaching their destination again - yet even though i have given the source ip that should have been used, it is still using the primary one.

Could this be a bug in the routing code?

I can only help you with the marking stuff :(

Stef

Signature · 7 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.openprojects.net
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#4 lore raw
In reply to Thilo Schulz

On Tue, Apr 23, 2002 at 07:23:58PM +0200, Thilo Schulz wrote:
[...]

Quoted 7 lines

I have a computer in my LAN with 2 ips (192.168.1.10 and 192.168.1.11), the latter one created by "ip addr add" and users on that computer, and depending on the uid of the created packets i want those to take another source address. Of course, it should start with marking the packets with a simple iptables rule:

iptables -t mangle -A OUTPUT -m owner --uid-owner 1001 -j MARK --set-mark 1

[...]

Quoted 5 lines

The table as well as the route entry are correct, i am sure of this, for i have tested removing the fwmark rule and instead added:
ip rule add to 192.168.1.4 table thilo
everything worked then as i have intended, and it showed me logged in from 192.168.1.11 then.

Are you sure it showed you as logged in from 192.168.1.11? I just tried the same setup here, and with either setup you describe, it always shows me as having logged in from the primary IP.

In my case, I setup 192.168.1.1 and 192.168.1.2 and tried to connect to 192.168.1.5. In both cases above, it showed me as having logged in from 192.168.1.1.

The counters are incrementing though, so marking is working.

Quoted 5 lines

Another problems checked too: I have enabled MARK based routing in the kernel (policy routing, everything switched on ..)

So I only can ask you on this mailing list - anyone got an idea of what is/what i did wrong?

Depending on what the results of your second setup above were (ip rule add to 192.168.1.4 table thilo), I have a theory...

When you add a route that sets a src like:

ip route add table <table> 192.168.1.0/24 src 192.168.1.11 dev eth0

The "src" doesn't specify the source IP to put in the packet (it's not network address translation, like SNAT in iptables), it just specifies which local source IP the routing mechanisms should use to determine where to route the packet.

For example, I've got policy routing setup with FreeS/WAN on a gateway with an internal and external interface, where I do:

# ip rule add table 1 prio 100
# ip route add table 1 <remoteLAN> dev ipsec0 src <localIP>

This forces the box to route all packets to the remote LAN via the internal interface, rather than the external interface.

However, the packets that show up at the other end don't contain a source IP of <localIP> from the table 1 route, rather they contain the source IP of the client machine that sent them.

This led me to believe that the "src" option only adjusts the way the routing machinery in the kernel decides where and how to route the packet, but doesn't change/rewrite the source address in the packets themselves.

This makes sense to me as well, because if setup routes for 192.168.1.4 through 192.168.1.11, and send packets from 192.168.1.10, they'll have a source address of 192.168.1.10, but get routed through the 192.168.1.11 interface (which is transparent at 192.168.1.4's end).

If you really want the packets to look like they're coming from 192.168.1.11, why don't you just use iptables and SNAT them?

--
Adrian Chung (adrian at enfusion-group dot com)
http://www.enfusion-group.com/~adrian
GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [toad.enfusion-group.com] up 34 days, 1:10, 17 users

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Thilo Schulz
Quoted 3 lines

Are you sure it showed you as logged in from 192.168.1.11? I just tried the same setup here, and with either setup you describe, it always shows me as having logged in from the primary IP.

Absolutely sure, it only worked with the "to ipaddress" rule.

Quoted 4 lines

The "src" doesn't specify the source IP to put in the packet (it's not network address translation, like SNAT in iptables), it just specifies which local source IP the routing mechanisms should use to determine where to route the packet.

However, somewhere inthe iproute tools this HAS to be possible ...

Quoted 2 lines

If you really want the packets to look like they're coming from 192.168.1.11, why don't you just use iptables and SNAT them?

I have thought of that as well, but the socket that has been created should only listen to the primary IP, so that the source and destination address do not match.

I was going to try it though, and have already compiled a fully NAT able kernel (it was not able to previously, because i thought NAT is not necessary)

Also i'm going to contact one of the Policy Routing developers.

Thanks for your help.
- Thilo Schulz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Thilo Schulz

On Fri, Apr 26, 2002 at 10:01:48PM +0200, Thilo Schulz wrote:
[...]

Quoted 6 lines
Quoted 4 lines

The "src" doesn't specify the source IP to put in the packet (it's not network address translation, like SNAT in iptables), it just specifies which local source IP the routing mechanisms should use to determine where to route the packet.

However, somewhere inthe iproute tools this HAS to be possible ...

I read in the documentation for iproute2 somewhere, sometime that there was a 'nat' directive, but I thought I'd also read that it wasn't very well supported, probably didn't work, and probably wouldn't be rewritten to work because netfilter/SNAT already do a much better job of NAT in general.

[..]

Quoted 1 line

Also i'm going to contact one of the Policy Routing developers.

Hope you get it working!

--
Adrian Chung (adrian at enfusion-group dot com)
http://www.enfusion-group.com/~adrian
GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [toad.enfusion-group.com] up 34 days, 1:40, 17 users

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to Thilo Schulz
Quoted 8 lines · Adrian Chung

When you add a route that sets a src like:

ip route add table <table> 192.168.1.0/24 src 192.168.1.11 dev eth0

The "src" doesn't specify the source IP to put in the packet (it's not network address translation, like SNAT in iptables), it just specifies which local source IP the routing mechanisms should use to determine where to route the packet.

Actually, it is more subtle than that. The 'src' *does* specify the source IP to put in the packet *if* the packet doesn't have a source IP yet. This only holds true for packets generated locally.

Quoted 9 lines

For example, I've got policy routing setup with FreeS/WAN on a gateway with an internal and external interface, where I do:

# ip rule add table 1 prio 100
# ip route add table 1 <remoteLAN> dev ipsec0 src <localIP>

This forces the box to route all packets to the remote LAN via the internal interface, rather than the external interface.

Quoted 8 lines

However, the packets that show up at the other end don't contain a source IP of <localIP> from the table 1 route, rather they contain the source IP of the client machine that sent them.

This led me to believe that the "src" option only adjusts the way the routing machinery in the kernel decides where and how to route the packet, but doesn't change/rewrite the source address in the packets themselves.

It does not. The ip rule does that. Routing does not mangle packets, unless the packet is locally generated and incomplete.

Doei, Arthur.

Signature · 7 lines
  /\    / |      [email protected]      | Work like you don't need the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there's nobody watching

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#8 lore raw
In reply to Thilo Schulz
Quoted 3 lines

Actually, it is more subtle than that. The 'src' *does* specify the source IP to put in the packet *if* the packet doesn't have a source IP yet. This only holds true for packets generated locally.

Then why does it not work together with the fwmark policer?

Quoted 2 lines

It does not. The ip rule does that. Routing does not mangle packets, unless the packet is locally generated and incomplete.

it is generated locally in my case.

- Thilo Schulz
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
In reply to Thilo Schulz

Hello,

Quoted 4 lines · Thilo Schulz
Quoted 2 lines

It does not. The ip rule does that. Routing does not mangle packets, unless the packet is locally generated and incomplete.

it is generated locally in my case.

May be your problem is that LOCALOUT is after routing,
the socket already created connected route and is bound to specific src IP. It is too late to mark packets just to select different src IP via routing. May be you have to use SNAT to change the src IP to the desired one. I didn't tried it and I don't know whether it works. And I see in recent sources some funny things, for example, ipt_local_hook(MANGLE) calls ip_route_me_harder from LOCAL_IN but I'm not sure it will hurt your tests. May be you can ask for new LOCAL_ROUTING hook where MANGLE can select different src IP for different users before route connection :)))

Quoted 1 line

- Thilo Schulz

Regards

--
Julian Anastasov <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#10 lore raw
In reply to Thilo Schulz
Quoted 6 lines · Arthur van Leeuwen
Quoted 4 lines · Adrian Chung

When you add a route that sets a src like:

ip route add table <table> 192.168.1.0/24 src 192.168.1.11 dev eth0

Quoted 8 lines
Quoted 4 lines

The "src" doesn't specify the source IP to put in the packet (it's not network address translation, like SNAT in iptables), it just specifies which local source IP the routing mechanisms should use to determine where to route the packet.

Actually, it is more subtle than that. The 'src' *does* specify the source IP to put in the packet *if* the packet doesn't have a source IP yet. This only holds true for packets generated locally.

Ah okay, that makes sense... But I think in both our cases the packets were generated locally, so the 'src' flag should have set the source IP.

Is it possible for the application (telnet in my case) to explicitly bind to a socket and set it's source IP? That could explain why the rule has no effect since by the time the packet reaches the routing system, it already has a source IP set.

--
Adrian Chung (adrian at enfusion-group dot com)
http://www.enfusion-group.com/~adrian
GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [toad.enfusion-group.com] up 34 days, 17:14, 17 users

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#11 lore raw
In reply to Thilo Schulz
Quoted 16 lines · Adrian Chung
Quoted 6 lines · Arthur van Leeuwen
Quoted 4 lines · Adrian Chung

When you add a route that sets a src like:

ip route add table <table> 192.168.1.0/24 src 192.168.1.11 dev eth0

Quoted 8 lines
Quoted 4 lines

The "src" doesn't specify the source IP to put in the packet (it's not network address translation, like SNAT in iptables), it just specifies which local source IP the routing mechanisms should use to determine where to route the packet.

Actually, it is more subtle than that. The 'src' *does* specify the source IP to put in the packet *if* the packet doesn't have a source IP yet. This only holds true for packets generated locally.

Quoted 8 lines

Ah okay, that makes sense... But I think in both our cases the packets were generated locally, so the 'src' flag should have set the source IP.

Is it possible for the application (telnet in my case) to explicitly bind to a socket and set it's source IP? That could explain why the rule has no effect since by the time the packet reaches the routing system, it already has a source IP set.

Yes, that is possible.

Doei, Arthur.

Signature · 7 lines
  /\    / |      [email protected]      | Work like you don't need the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there's nobody watching

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/