Add sk_mark route lookup support for IPv4 listening sockets, and for IPv4 multicast forwarding
Subsystems (4)
PATCH 1/1 Add sk_mark route lookup support for IPv4 listening sockets, and for IPv4 multicast forwarding
This is followup to my previous route lookup patch, it adds sk_mark based routing lookup support in even more places. Now it is possible to have e.g. TCP server listening to connection requests in a specific routing table specified by SO_MARK socket option.
Also, correct me if I'm wrong, but syncookie route lookup still seems to be broken in more than one way. The sk_bound_dev_if interface from socket is not used; and route lookup is done in init_net, instead of using sock_net(sk). cookie_v6_check() in net/ipv6/syncookies.c probably needs a patch as well.
Add support for route lookup using sk_mark on IPv4 listening sockets, and for IPv4 multicast forwarding
--- net/ipv4/inet_connection_sock.c | 1 + net/ipv4/ipmr.c | 2 ++ net/ipv4/syncookies.c | 3 ++- 3 files changed, 5 insertions(+), 1 deletion(-)
net/ipv4/inet_connection_sock.c +1 -0
@@ -358,6 +358,7 @@ struct dst_entry *inet_csk_route_req(struct sock *sk,const struct inet_request_sock *ireq =inet_rsk (req);struct ip_options *opt =inet_rsk (req)->opt;struct flowi fl = { .oif = sk->sk_bound_dev_if, + .mark = sk->sk_mark, .nl_u = { .ip4_u = { .daddr = ((opt && opt->srr) ? opt->faddr :
net/ipv4/ipmr.c +2 -0
@@ -1238,6 +1238,7 @@ static void ipmr_queue_xmit(struct sk_buff *skb, struct mfc_cache *c, int vifi)if (vif->flags&VIFF_TUNNEL ) {struct flowi fl = { .oif = vif->link, + .mark = skb->mark, .nl_u = { .ip4_u = { .daddr = vif->remote, .saddr = vif->local,
@@ -1248,6 +1249,7 @@ static void ipmr_queue_xmit(struct sk_buff *skb, struct mfc_cache *c, int vifi) encap =sizeof (struct iphdr ); }else {struct flowi fl = { .oif = vif->link, + .mark = skb->mark, .nl_u = { .ip4_u = { .daddr = iph->daddr, .tos = RT_TOS(iph->tos) } },
net/ipv4/syncookies.c +2 -1
@@ -333,7 +333,8 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb,* no easy way to do this. */ { -struct flowi fl = { .nl_u = { .ip4_u = +struct flowi fl = { .mark = sk->sk_mark, + .nl_u = { .ip4_u = { .daddr = ((opt && opt->srr) ? opt->faddr : ireq->rmt_addr),
David Miller
maintainerFrom: Atis Elsts <[email protected]>
Date: Mon, 5 Oct 2009 16:46:34 +0300
net/ipv4/ipmr.c static void ipmr_queue_xmit(struct sk_buff *skb, struct mfc_cache *c, int vifi) · line 1244
Quoted 8 lines
net/ipv4/ipmr.c +1
-0@@ -1238,6 +1238,7 @@ static void ipmr_queue_xmit(struct sk_buff *skb, struct mfc_cache *c, int vifi)if (vif->flags&VIFF_TUNNEL ) {struct flowi fl = { .oif = vif->link, + .mark = skb->mark, .nl_u = { .ip4_u = { .daddr = vif->remote, .saddr = vif->local,
I'm not so sure if this is right.
I understand what you're trying to do, inherit the socket's mark when it goes over a multicast tunnel.
But I'm not so sure that's what we want to do, semantically.
Could you split out these skb->mark cases into a seperate patch? The parts that only use sk->mark are fine and I would like to apply a patch from you which just does that while we discuss the skb->mark case.
Thanks.
Quoted 17 lines · On Wednesday 07 October 2009 13:19:57 David Miller wrote:
From: Atis Elsts <[email protected]>
Date: Mon, 5 Oct 2009 16:46:34 +0300Review static void ipmr_queue_xmit(struct sk_buff *skb, struct mfc_cache *c, int vifi) · line 1244
Quoted 8 lines
(quoted diff) +1
-0@@ -1238,6 +1238,7 @@ static void ipmr_queue_xmit(struct sk_buff *skb, struct mfc_cache *c, int vifi) if (vif->flags&VIFF_TUNNEL) { struct flowi fl = { .oif = vif->link, + .mark = skb->mark, .nl_u = { .ip4_u = { .daddr = vif->remote, .saddr = vif->local,I'm not so sure if this is right.
I understand what you're trying to do, inherit the socket's mark when it goes over a multicast tunnel.
Quoted 7 lines
But I'm not so sure that's what we want to do, semantically.
Could you split out these
skb->markcases into a seperate patch? The parts that only usesk->markare fine and I would like to apply a patch from you which just does that while we discuss theskb->markcase.
Here is the sk_mark part.
As for the ipmr.c code, I agree with your comment. Using mark from skb probably is wrong in case of tunnel interface (i.e. in the "if (vif->flags&VIFF_TUNNEL)" part of the patch), my mistake. I still think that the "else" part is correct, though, because using mark from skb there mirrors behaviour for unicast forwarding routing lookup in ip_route_input_slow(). The same applies to IPv6 code in ip6mr_forward2().
Add support for route lookup using sk_mark on IPv4 listening sockets.
--- net/ipv4/inet_connection_sock.c | 1 + net/ipv4/syncookies.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-)
net/ipv4/inet_connection_sock.c +1 -0
@@ -358,6 +358,7 @@ struct dst_entry *inet_csk_route_req(struct sock *sk,const struct inet_request_sock *ireq =inet_rsk (req);struct ip_options *opt =inet_rsk (req)->opt;struct flowi fl = { .oif = sk->sk_bound_dev_if, + .mark = sk->sk_mark, .nl_u = { .ip4_u = { .daddr = ((opt && opt->srr) ? opt->faddr :
net/ipv4/syncookies.c +2 -1
@@ -333,7 +333,8 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb,* no easy way to do this. */ { -struct flowi fl = { .nl_u = { .ip4_u = +struct flowi fl = { .mark = sk->sk_mark, + .nl_u = { .ip4_u = { .daddr = ((opt && opt->srr) ? opt->faddr : ireq->rmt_addr),
From: Atis Elsts <[email protected]>
Date: Wed, 7 Oct 2009 15:59:56 +0300
Quoted 1 line
Here is the
sk_markpart.
Applied, thanks.
Quoted 7 lines
As for the
ipmr.ccode, I agree with your comment. Using mark from skb probably is wrong in case of tunnel interface (i.e. in the "if (vif->flags&VIFF_TUNNEL)" part of the patch), my mistake. I still think that the "else" part is correct, though, because using mark from skb there mirrors behaviour for unicast forwarding routing lookup inip_route_input_slow(). The same applies to IPv6 code inip6mr_forward2().
Ok submit just the else part and we'll have a look at it.
Thanks.
Should wrapping a packet into a tunnel clear the mark? Should perhaps the mark be a parameter of the tunnel (like ttl or qos, can be set or can be inherited)?
Quoted 19 lines · David Miller
From: Atis Elsts <[email protected]>
Date: Wed, 7 Oct 2009 15:59:56 +0300Quoted 1 line
Here is the
sk_markpart.Applied, thanks.
Quoted 7 lines
As for the
ipmr.ccode, I agree with your comment. Using mark from skb probably is wrong in case of tunnel interface (i.e. in the "if (vif->flags&VIFF_TUNNEL)" part of the patch), my mistake. I still think that the "else" part is correct, though, because using mark from skb there mirrors behaviour for unicast forwarding routing lookup inip_route_input_slow(). The same applies to IPv6 code inip6mr_forward2().Ok submit just the else part and we'll have a look at it.
Thanks.
David Miller
maintainerFrom: Maciej Żenczykowski <[email protected]>
Date: Wed, 7 Oct 2009 17:03:30 -0700
Quoted 3 lines
Should wrapping a packet into a tunnel clear the mark? Should perhaps the mark be a parameter of the tunnel (like ttl or qos, can be set or can be inherited)?
That's the big question.
It may be that the only logical thing we can do is only apply the socket mark to the top-level path and that's what happens now.
Because we really don't have any reasonable way to let the user control where it applies. We have no way for it to say "don't apply the mark to the top-level path, but do apply it to the multicast tunnel' or something like that.
So, to be honest, I'd say we should skip these skb->mark cases for now.
I'm thinking that the mark should be a tunnel parameter with values of inherit or a constant.
The reasoning being that this allows you to mark all packets leaving on a tunnel with a specific value and you can then use that mark to use a different routing table just for that tunnel.
The default routing table could point at the tunnel, and if the tunnel is marked than the mark on those packets could make them not use the default routing table, and instead use a routing table which is closer to physical network layout.
I believe nowadays you usually solve this by having an explicit route for the other endpoint of the tunnel in the main routing table... which means you can't actually tunnel the normal traffic directed at the other endpoint of the tunnel.
This would fix that.
Hi,
Quoted 3 lines · Maciej Żenczykowski
I'm thinking that the mark should be a tunnel parameter with values of inherit or a constant.
Why not do this on a per-route basis (i.e. lets suppose we add a "setmark" parameter to each route) and this would allow changing a mark when a packet matches the route. This not only solves the tunnel case, but would be generically useful as well.
Since we have to look up routes anyway, it shouldn't add any real overhead to the routing process and we can benefit from all the existing infrastructure (route cache, etc).
Steve.
David Miller
maintainerFrom: [email protected]
Date: Wed, 14 Oct 2009 08:23:19 +0100
Quoted 12 lines · Maciej Żenczykowski
Quoted 3 lines
I'm thinking that the mark should be a tunnel parameter with values of inherit or a constant.
Why not do this on a per-route basis (i.e. lets suppose we add a "setmark" parameter to each route) and this would allow changing a mark when a packet matches the route. This not only solves the tunnel case, but would be generically useful as well.
Since we have to look up routes anyway, it shouldn't add any real overhead to the routing process and we can benefit from all the existing infrastructure (route cache, etc).
This idea, I like :-)
Problem is the primary purpose of the mark is to enable matching on the mark in the routing tables.
See 'ip rule ... fwmark X ...'
ie. that fails due to circular dependency.
Quoted 18 lines · David Miller
From: [email protected]
Date: Wed, 14 Oct 2009 08:23:19 +0100Quoted 12 lines · Maciej Żenczykowski
Quoted 3 lines
I'm thinking that the mark should be a tunnel parameter with values of inherit or a constant.
Why not do this on a per-route basis (i.e. lets suppose we add a "setmark" parameter to each route) and this would allow changing a mark when a packet matches the route. This not only solves the tunnel case, but would be generically useful as well.
Since we have to look up routes anyway, it shouldn't add any real overhead to the routing process and we can benefit from all the existing infrastructure (route cache, etc).
This idea, I like :-)
Hi,
Quoted 8 lines · Maciej Żenczykowski
Problem is the primary purpose of the mark is to enable matching on the mark in the routing tables.
See 'ip rule ... fwmark X ...'
ie. that fails due to circular dependency.
I don't agree. There are two route lookups with a tunnel, the internal one and the tunnel one. Here is an example of what I'm thinking:
1. Look up a route which points at a remote ip addres via a tunnel device. The "setmark" on this route sets the skb mark
2. Look up a route on the tunnel itself (i.e. the tunnel endpoint not the socket endpoint) using the mark from the initial lookup. This route can depend on the previous lookup (if there are multiple routes for multiple marks) and also set the mark to use.
The default would be to inherit the mark over a route lookup, in case that no "setmark" had been specified for that route. In other words, it would be the same as it is now.
The mark is supposed to be a generic thing, not just for routing lookups, it can be used for classification, etc as well. I would expect to see such a thing used for maybe specifying a VLAN or a reference to an MPLS label stack, or something similar too,
Steve.
Quoted 3 lines · On Wednesday 14 October 2009 12:27:43 [email protected] wrote:
The mark is supposed to be a generic thing, not just for routing lookups, it can be used for classification, etc as well.
In general, sounds like a good idea, but IMHO exactly this could be a problem. skb->mark is already used for a lot of things. What if I am setting the mark by a firewall rule in prerouting chain, and matching it by a postrouting rule? If routing lookup was changing the mark, then my setup would break.
Perhaps one more field could be added dst_entry? The field would be filled from route's table (if "setmark" for that route was specified). The use of that field would be similar to tclassid (e.g matching in firewall), except that it would also be used in routing lookups, if set.
Quoted 5 lines
I would
expect to see such a thing used for maybe specifying a VLAN or a reference to an MPLS label stack, or something similar too,Steve.
Hi,
Quoted 10 lines · Atis Elsts
Quoted 3 lines · On Wednesday 14 October 2009 12:27:43 [email protected] wrote:
The mark is supposed to be a generic thing, not just for routing lookups, it can be used for classification, etc as well.
In general, sounds like a good idea, but IMHO exactly this could be a problem.
skb->markis already used for a lot of things. What if I am setting the mark by a firewall rule in prerouting chain, and matching it by a postrouting rule? If routing lookup was changing the mark, then my setup would break.
Yes, thats exactly why I said that it should default to the current behaviour.
Quoted 5 lines
Perhaps one more field could be added
dst_entry? The field would be filled from route's table (if "setmark" for that route was specified). The use of that field would be similar to tclassid (e.g matching in firewall), except that it would also be used in routing lookups, if set.
Yes, I think that we are both thinking along the same lines. There must obviously be a default "don't touch" setting,
Steve.
Quoted 6 lines
I don't agree. There are two route lookups with a tunnel, the internal one and the tunnel one. Here is an example of what I'm thinking:
1. Look up a route which points at a remote ip addres via a tunnel device. The "setmark" on this route sets the skb mark
imho, this is much better done by having the mark setting performed explicitly by the tunnel device itself.
That's also were we set ttl and qos (or inherit) on the outgoing packet).
Quoted 4 lines
2. Look up a route on the tunnel itself (i.e. the tunnel endpoint not the socket endpoint) using the mark from the initial lookup. This route can depend on the previous lookup (if there are multiple routes for multiple marks) and also set the mark to use.
we would get the mark set by the tunneling module here.
Quoted 3 lines
The default would be to inherit the mark over a route lookup, in case that no "setmark" had been specified for that route. In other words, it would be the same as it is now.
I'm not saying your solution wouldn't work, but I think it's less clean. I don't think marking should be inherited (in the general case) in case of packet wrapping (whether via gre, ipip, sit, or other methods).
Quoted 4 lines
The mark is supposed to be a generic thing, not just for routing lookups, it can be used for classification, etc as well. I would expect to see such a thing used for maybe specifying a VLAN or a reference to an MPLS label stack, or something similar too,
Right, the mark can currently (as far as I know) be set in one of two ways - either from the mangle table (and it can also be matched on in netfilter) or by using setsockopt(SO_MARK).
Imagine a situation where you have a machine with routing already configured (pretty complex setup, tunnels, firewalls, etc) and you want to run a user space application that verifies (health-checks) some remote host (or something). As part of the health check you want to verify a particular route to the destination. This requires per-socket routing, which can (almost) be achieved by having proper routing (on fwmark) setup and using setsockopt(SO_MARK) on the health check socket in order to force specific routing. These health checks may then of course be feedback into the routing system (ie. if they fail the routing rules get modified). Note, that in particular we may want to be healthchecking routes that aren't even available in the default routing table (because they've currently been removed from the default table, because previous health checks failed).
Maciej
Hi,
Quoted 11 lines · Maciej Żenczykowski
Quoted 6 lines
I don't agree. There are two route lookups with a tunnel, the internal one and the tunnel one. Here is an example of what I'm thinking:
1. Look up a route which points at a remote ip addres via a tunnel device. The "setmark" on this route sets the skb mark
imho, this is much better done by having the mark setting performed explicitly by the tunnel device itself.
That's also were we set ttl and qos (or inherit) on the outgoing packet).
Yes, I think (having looked at the code a bit more in the mean time) that there is an argument for doing that. Although I still think that setting the mark via the routing table would be a useful feature too.
Quoted 7 lines
Quoted 4 lines
2. Look up a route on the tunnel itself (i.e. the tunnel endpoint not the socket endpoint) using the mark from the initial lookup. This route can depend on the previous lookup (if there are multiple routes for multiple marks) and also set the mark to use.
we would get the mark set by the tunneling module here.
Quoted 9 lines
Quoted 3 lines
The default would be to inherit the mark over a route lookup, in case that no "setmark" had been specified for that route. In other words, it would be the same as it is now.
I'm not saying your solution wouldn't work, but I think it's less clean. I don't think marking should be inherited (in the general case) in case of packet wrapping (whether via gre, ipip, sit, or other methods).
I guess we could say that inheriting the mark would not be the default if the packet has gone through a device (whether virtual or physical) then. That still seems ok to me since its basically what happens currently I think.
Quoted 23 lines
Quoted 4 lines
The mark is supposed to be a generic thing, not just for routing lookups, it can be used for classification, etc as well. I would expect to see such a thing used for maybe specifying a VLAN or a reference to an MPLS label stack, or something similar too,
Right, the mark can currently (as far as I know) be set in one of two ways - either from the mangle table (and it can also be matched on in netfilter) or by using
setsockopt(SO_MARK).Imagine a situation where you have a machine with routing already configured (pretty complex setup, tunnels, firewalls, etc) and you want to run a user space application that verifies (health-checks) some remote host (or something). As part of the health check you want to verify a particular route to the destination. This requires per-socket routing, which can (almost) be achieved by having proper routing (on fwmark) setup and using
setsockopt(SO_MARK)on the health check socket in order to force specific routing. These health checks may then of course be feedback into the routing system (ie. if they fail the routing rules get modified). Note, that in particular we may want to be healthchecking routes that aren't even available in the default routing table (because they've currently been removed from the default table, because previous health checks failed).
Quoted 1 line
Maciej
Yes, thats a good use case. I think there are a lot of other potential use cases too though. A while back when I was looking into MPLS I wondered about using the mark to index into a set of outgoing label stacks. That was the original reason that I thought setting the mark via the routing table would be useful. I've not really had the time to continue my MPLS investigations recently though :(
Another potential use case would be to segregate traffic into different routing domains (and thus being able to change the mark when moving from one routing domain to another might be useful).
Steve.
Quoted 4 lines · On Monday 19 October 2009 11:20:33 [email protected] wrote:
Another potential use case would be to segregate traffic into different routing domains (and thus being able to change the mark when moving from one routing domain to another might be useful).
I agree. Actually, one of our users recenlty requested adding matcher in firewall that would match outgoing the packets by the routing table that was used to route them. (For now we found a workaround using tclassid, but that requires manual configuration.) So yes, it's an useful feature even excluding the tunnel cases.
I don't like the idea of using skb->mark for storing that information though, because I think these multiple uses of the same field would be too confusing for users, even if the default behavior remained the same as now. Also, consider the case when someone watch to match packets in post routing chain *both* by the mark that was set in prerouting chain, and routing table used to route the packet?
There already is free space (padding fieds) in struct dst_entry, so why not use this space to store the routing table? Speed is also not an issue, because the field only needs to be filled in slowpath routing lookup, and will be used only
1) if iptables are explicitly configured to match by it;
2) (?) in tunnel routing lookups. (no idea which is the best option here)
I see that struct rt6_info already has field
struct fib6_table *rt6i_table
so this matcher already can be made for IPv6 firewall. But IPv4 still is more imporant at the moment :)
Atis
PATCH 1/1 net: Use routing mark from skb in multicast forwarding routing lookups
In reply to David MillerQuoted 3 lines · On Wednesday 07 October 2009 23:56:27 David Miller wrote:
Ok submit just the else part and we'll have a look at it.
Here is a try.
Use routing mark from skb in routing lookup in IPv4 and IPv6 multicast forwarding code.
--- net/ipv4/ipmr.c | 1 + net/ipv6/ip6mr.c | 1 + 2 files changed, 2 insertions(+)
net/ipv4/ipmr.c +1 -0
@@ -1248,6 +1248,7 @@ static void ipmr_queue_xmit(struct sk_buff *skb, struct mfc_cache *c, int vifi) encap =sizeof (struct iphdr ); }else {struct flowi fl = { .oif = vif->link, + .mark = skb->mark, .nl_u = { .ip4_u = { .daddr = iph->daddr, .tos = RT_TOS(iph->tos) } },
net/ipv6/ip6mr.c +1 -0
@@ -1523,6 +1523,7 @@ static int ip6mr_forward2(struct sk_buff *skb, struct mfc6_cache *c, int vifi) fl = (struct flowi ) { .oif = vif->link, + .mark = skb->mark, .nl_u = { .ip6_u = { .daddr = ipv6h->daddr, } }
David Miller
maintainerRe: [PATCH] net: Use routing mark from skb in multicast forwarding routing lookups
From: Atis Elsts <[email protected]>
Date: Thu, 8 Oct 2009 16:19:44 +0300
Quoted 8 lines · On Wednesday 07 October 2009 23:56:27 David Miller wrote:
Quoted 3 lines
Ok submit just the else part and we'll have a look at it.
Here is a try.
Use routing mark from skb in routing lookup in IPv4 and IPv6 multicast forwarding code.
Signed-off-by Atis Elsts <[email protected]>
The more I think about the less this makes sense.
There is no way to setup these semantics of lower level routes and satisfy every reasonable use case.
We'll need to provide another facility or suggest other methods to control marking of encapsulated routes from the socket level, sorry.