NAT before IPsec with 2.6

Hi there,

with the native IPsec stack in 2.6.1 kernel it's impossible to do SNAT on outgoing packets that are leaving through an IPsec link. The problem is that the forwarded packets are encapsulated (with IPSec ESP/AH) yet before the POSTROUTING chain is reached.

So I hacked up this simple solution - I understand that it won't be accepted as-is, because now the POSTROUTING chain is reached twice (once from ip_forward and once from ip_output). But anyways - is this a correct approach and should I go this way with some cleanups?

Or is there another approach? I was thinking about extending the FORWARD chain and let it do SNAT/MASQUERADE. That would be cleaner, but more intrusive than my patch and would probably also require some changes in the userspace.

Comments, opinions, ...?

(Please Cc me on replies, thanks!)

Michal Ludvig

SUSE Labs                    [email protected] | Cray is the only computer
(+420) 296.545.373        http://www.suse.cz | that runs an endless loop
Personal homepage http://www.logix.cz/michal | in just four hours.

diff -rup linux-2.6.1.orig/net/ipv4/ip_forward.c linux-2.6.1.naga/net/ipv4/ip_forward.c
@@ -46,12 +46,30 @@ static inline int ip_forward_finish(stru
 {
 	struct ip_options * opt	= &(IPCB(skb)->opt);
 
+	if (!xfrm4_route_forward(skb))
+		goto drop;
+
+#ifdef CONFIG_NETFILTER_DEBUG
+	skb->nf_debug &= ~(1 << NF_IP_POST_ROUTING);
+#endif
+
 	IP_INC_STATS_BH(IpForwDatagrams);
 
 	if (unlikely(opt->optlen))
 		ip_forward_options(skb);
 
 	return dst_output(skb);
+
+drop:
+	kfree_skb(skb);
+	return NET_RX_DROP;
+}
+
+static inline int ip_forward_postroute(struct sk_buff *skb)
+{
+	struct rtable *rt = (struct rtable*)skb->dst;
+
+	return NF_HOOK(PF_INET, NF_IP_POST_ROUTING, skb, skb->dev, rt->u.dst.dev, ip_forward_finish);
 }
 
 int ip_forward(struct sk_buff *skb)

@@ -109,7 +131,7 @@ int ip_forward(struct sk_buff *skb)
 	skb->priority = rt_tos2priority(iph->tos);
 
 	return NF_HOOK(PF_INET, NF_IP_FORWARD, skb, skb->dev, rt->u.dst.dev,
-		       ip_forward_finish);
+		       ip_forward_postroute);
 
 sr_failed:
         /*

Hi,

I don't know if it's a correct approach, but I really think that all tables and chains should apply to a consistent level of encapsulation, whatever it is. Before, it was easy to process the packet at different levels because it was sent to one (or even several) interemediate virtual interfaces. To keep control of what goes in and out, now we should :

1) re-evaluate all previously evaluated chains when a packet has been

     decapsulated ;

2) re-inject encapsulated packets into the output & postrouting chains (it's a locally generated packet after all, it has no reason to bypass output filtering).
3) keep assigned fwmark between passes so that we can track what each packet does.

I believe that 1) & 3) are already done, but I don't know if there's a general consensus towards or against 2).

Cheers,
Willy

The ip_forward call should be moved to just before where the packets gets encapsulated when using IPSec. When using IPSec this is the conceptual POSTROUTING point in the stack for the plain packets.

Other than that I see no direct problems.

Regards
Henrik

Henrik Nordstrom told me that:

Without my patch the path is:

	ip_forward()
	     |
	check xfrm policy (here it's decided whether or not to encrypt)
	     |
	NF_IP_FORWARD (unfortunately no NAT available here...)
	     |
	ip_forward_finish()
	     |
	dst_output() => esp_output()

With my patch the path is:

	ip_forward()
	     |
	check xfrm policy (could probably be omitted, we'll do it after
	                   NAT once again)
	     |
	NF_IP_FORWARD (the packet may be e.g. dropped here)
	     |
	ip_forward_postroute()
	     |
	NF_IP_POST_ROUTING (do NAT here)
	     |
	ip_forward_finish()
	     |
	check xfrm policy (here it's decided whether or not to encrypt)
	     |
	dst_output() => esp_output()

I.e. the postrouting on the unencrypted packet is really called right before it gets encrypted. And the encrypted packet then hits the POSTROUTING again, but it's already a different packet, actually.

Was this your point or did I misunderstand you?

Michal Ludvig

SUSE Labs                    [email protected] | Cray is the only computer
(+420) 296.545.373        http://www.suse.cz | that runs an endless loop
Personal homepage http://www.logix.cz/michal | in just four hours.

My issue is with packets not destinated for an IPSec tunnel.. from what I read your patch these will hit POSTROUTING twice. But maybe I misread your patch?

Regards
Henrik

Henrik Nordstrom told me that:

Yes, that's true. But how to solve it? The best would be to skip the first POSTROUTING if the packet won't be encapsulated. But we can't know whether or not it will go to IPsec until we perform NAT on it. So it looks like we must skip the second POSTROUTING in the case where the packet was encapsulated. But how can we recognise these two situations? Wild idea - comparing some hashes (src/dst ip/port) before and after the first POSTROUTING call?

Michal Ludvig

SUSE Labs                    [email protected] | Cray is the only computer
(+420) 296.545.373        http://www.suse.cz | that runs an endless loop
Personal homepage http://www.logix.cz/michal | in just four hours.

As I said the hook needs to be just before the packet is about to enter IPSec.

Any decisions on what to IPSec or not needs to be based on other information than NAT, i.e. via nfmark routing or similar.

Regards
Henrik

I'm concerned that passing the unencrypted packet through POSTROUTING apparently makes it impossible for firewall rules to enforce encryption to a remote host or network.

It's my opinion that when the ipsec<n> devices where eliminated, the baby got thrown out with the bath water.

My $.02...

-Tom

Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ [email protected]

Hi,

That was my point too. Even if the packet is only encapsulated, from an IP point of view, this is a completely new locally generated packet, and we need to be able to apply filtering before and after encapsulation.

Willy

Yes, and to do that the required netfilter hooks needs to be added to the new IPSec code when packets leave the IP stack and enters IPSec or the reverse, similar to how there was hooks for the ipsecX: interfaces.

These hooks have not yet been done, mostly because nobody has contributed the code adding these hooks to the new IPSec implementation in a reasonable manner I guess.

Regards
Henrik

OK, that's good news. At first, I thought that people were against this mode of operation. This is now just a matter of time for anyone of us to understand how to add the hooks and send a patch ! I wish I add time...

Regards,
Willy

I have to agree with Henrik. Once there is an implementation that the netfilter core team is happy with, we're certainly going to push very hard to include this in the mainstream kernel.

I have to admit that I haven't yet found a 'perfect' solution of that problem - but I'm following the ongoing discussion.

JFYI: I really don't like the 'postrouting called twice' solution. This violates one of the core basics of netfilter/iptables so far.

I also disagree that an ESP packet should be trated as locally
generated (and thus iterate over OUTPUT). From my perspective, locally
generated means more something like: sent via a local userspace process using PF_INET sockets. If we consider a packet after any kind of change as locally-generated, we could argue doing this with NAT'ed packets, too. Please try to convince me, if I'm missing some point.

One of the fundamental principles behind netfilter/iptables, especially NAT, is it's symmetric behaviour (between incoming and outgoing packets,
between the hooks, ...). So with any possible solution, this should be
kept in mind.

Also, I could expect way more headaches with regard to NAT of IPsec encapsulated packets. NAT is based on connection tracking and thus connection tracking has to be fully compatible with the 2.6.x IPsec implementation.

Right now, conntrack doesn't even notice that there are ESP/AH packets. PREROUTING creates a new conntrack for the unencapsulated packet. Later the packet is encapsulated, and confirmed at POSTROUTING time [where it is still the same skb, but one should argue whether that ESP packet really belongs to the original connection].

My proposal:
From conntracks view (and thus from NAT view), there (shuld be?) two seperate connections for any given to-be-encapsulated packet:

- the payload connection (tcp/udp/whatever)
- conntrack entry created at PREROUTING
- conntrack entry confirmed (put in hash) at XXX
- the AH/ESP connection
- conntrack entry created at YYY
- conntrack entry confirmed at POSTROUTING

To do this, somewhen between esp_output() is called and the beginning of the modification of the packet payload, we need to call nf_hook(POST_ROUTING). This way, conntrack would be able to put the connection in the hash, and people can do SNAT-like operations in nat->POSTROUTING. We could even pass a dummy output device structure with an interface name "esp" so people can SNAT everything heading for esp encapsulation.

As for the AH/ESP connection, this has to be created just after the packet was encapsulated. We need ot clear skb->nfct (and drop refcount to old conntrack) after encapsulation, and call nf_hook(PRE_ROUTING).

The then-encapsulated packet will then finally go via the traditional ipv4 stack and come to the POST_ROUTING hook, where the ESP connection can be confirmed.

This should solve all our NAT problems for free, shouldn't it?

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

I partly, but at the same time it becomes somewhat complex when making firewalling policies when not.

For me it is easier to grasp what is happending with the packet if it is viewed like the original packet is forwarded to the ipsec engine and the ipsec engine then generates a new packet.

This is very different from NAT as ESP/AH is a complete protocol transformation where the resulting packet does not resemble the original packet in any manner. This is also very much visible to conntrack where the original packets needs to be tracked as one connection while the ipsec gateway<->gateway traffic needs to be a separate connection. IPSec is essentially a tunneling protocol, tunneling IP packets over an encrypted connection. The tunnel endpoints are the IPSec gateways.

Even in the Host<->Host IPSec case the tunnel view is most appropriate. As soon as the packets enter the IPSec encapsulation they loose all the relevant propoerties of the original connection and becomes part of the single IPSec Host <-> IPSec Host connection all packets between these two IPSec host is transmitted over.

What may be interesting in netfilter is to preserve some kind of inheritance, allowing "OUTPUT" rules to know important details about the original packet when seeing an ESP packet.

My view of where hooks should be for IPSec:

(Client) -> Gateway -> (Gateway)

1. PREROUTING, original client packet
2. FORWARD, original client packet. Destination the IPSec SPI
3. POSTROUTING, original client packet. Destination the IPSec SPI
4. OUTPUT, ESP packet. Preferably with some preserved information about
the original packet attached.
5. POSTROUTING, ESP packet.

If the packet is rerouted in POSTROUTING then this should include IPSec routing. If the new route is to an IPSec SPI then it is. If not then it is not.

(Gateway) -> Gateway -> (client)

1. PREROUTING, ESP packet
2. FORWARD, ESP packet. Destination the IPSec SPI
3. POSTROUTING, ESP packet. Destination the IPSec SPI.
4. PREROUTING, decapsulated packet. With reference to the IPSec SPI.
5. FORWARD, decapsulated packet. With reference to the IPSec SPI.
6. POSTROUTING, decapsulated packet. With reference to the IPSec SPI.

Host -> (Gateway)

1. OUTPUT, original packet
2. POSTROUTING, original packet. Destination the IPSec SPI
3. OUTPUT, ESP packet. Preferably with some preserved information about
the original packet attached.
4. POSTROUTING, ESP packet. Preferably with some preserved information
about the original packet attached.

(Gateway) -> Host

1. PREROUTING, ESP packet.
2. INPUT, ESP packet.
3. PREROUTING, Decapsulated packet, with reference to the IPSec SPI
4. INPUT, Decapsulated packet, with reference to the IPSec SPI

Non-IPSec traffic:

As is today. There must be no change.

These hooks fits naturally into the concepts of the netfilter hook principles if IPSec is viewed as a tunneling method, which it in my opinion is best viewed as. It also fits the requirements for conntrack to be able to track the different types of connections involved, including NAT operations on both original and ESP traffic (assuming local NAT enabled if DNAT is required on ESP).

A NAT:ed connection is still the same connection at the packet level. ESP moves the packet into a new connection carrying all kinds of packets between the two IPSec endpoints where this host is one of the endpoints.

Granted.

Ah, we agree! ;-)

The above hook orders should match your expectations I think. Maybe we don't agree on the use of INPUT/OUTPUT in the case of forwarded client traffic, but I see no other clean way. These packets are deliberately addressed to/from this host, not the client, and part of the IPSec connection they are forwarded over between the two IPSec endpoints, not the original connection.

I save myself from diging into exacly where the hooks need to be in the kernel as I have not looked in detail at the 2.6 IPSec implementation, but I take for granted that several need to be at the corresponding locations in the IPSec protocol kernel. I also see it likely there needs to be some skb extensions to add the requires SPI tracking information to allow rulesets a better view of what is going on (optional).

Without knowing in detail how the 2.6 IPSec policies are maintained I see a slight risk that the 2.6 IPSec policy implementation maybe can not easily match the routing expectations, but it should be possible to find a way to at least give reasonable hook order I think. But there is a risk it is not possible to reroute packets to/from an IPSec route and the DNAT capabilities may be somewhat limited in such case until the IPSec kernel is extended with what may be required to do this cleanly. (i.e. slightly more than just inserting the appropriate hooks may be required)

Regards
Henrik

agreed.

yes, but that's again another chapter (like how to preserve that kind of information, ..

I agree to all but 4. My proposal was PREROUTING, yours OUTPUT. In
the end, none of them really fits. But I would consider this a minor issue. We could also create a new hook 'POSTENCAP' or whatever... Maybe we could agree at that point, that there is a new hook NF_IP_POST_XFRM. It's a matter of iptable_nat/... which chain they want to attach to that hook.

You are talking about the first POSTROUTING, Number 3? Then I would definitely agree.

This is not symmetric, despite being the exact opposite of the previous case. I don't like that. If we make it similar to the previous case, it would be:

1. PREROUTING, ESP packet
2. INPUT, ESP packet
3. PREROUTING, decapsulated packet
4. FORWARD, decapsulated packet
5. POSTROUTING, decapsulated packet.

So here we have the analgy of a locally received (INPUT) packet, as the opposite of the locally received (OUTPUT) packet in your previous case.

this would make it similar to the case #1.

agreed.

yes, at least in the largest part.

I guess I cannot save myself from doing so. Unfortunately I have very limited time at this moment, so don't expect it to be done today.

yes, but let's that make step 2. First we have to agree on the first step, do a decent implementation and then submit it to netdev.

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

I am.

Right. My error. Thinks one thing, writes another. Brain obviously not fully functional today.

Agreed.

Regards
Henrik

It is very hard to distinguis the different IPSec cases here. Save of adding yet another hook for encapsulated and to be decapsulated packets OUTPUT(/INPUT) is the closest match in my opinion. The packet it sees is the packet from this IPSec endpoint to the other IPSec endpoint on the IPSec SPI connection between the two.

The packet inside may be a forwarded packet or locally generated, it's all the same. The ESP is an exchange between the two endpoints OUTPUT -> POSTROUTING <-net-> PREROUTING -> INPUT

Use of PREROUTING introduces a completely new semantic unless you want to also call FORWARD on these packets which I do not think is appropriate.

If introducing new hooks then we would have

XXXX -> POSTROUTING <-net-> PREROUTING -> YYYY

if XXXX and YYYY can be the same hook name in this case or not I'll leave to the core team to ponder upon?

What exacly is the semantical problem of using OUTPUT/INPUT? In my view the ESP/AH packet is locally generated, and belongs to a connection with a local endpoint.

Regards
Henrik

Please don't confuse hooks with chains. I suggested to add new netfilter hooks (NF_IP_{PRE,POST}_XFRM), but attach the old INPUT/OUTPUT (or even PREROUTING/POSTROUTING) chains to those hooks.

Hm. You have a point, but I'm still not really sure. I'll do it the way you proposed, but I might change my mind again in the future ;)

well, you convinced me (for now). new hooks, old chains.

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

Sorry for the naming mismatch. With the current close relation between hooks and chains it is easy to confuse the two when one is not installing hooks every day. All thoughts was in netfilter hooks not iptables chains.

Using the existing existing hooks with NF_IP_LOCAL_IN/OUT for the IPSec wrapped packets is a exact match on what is required for conntrack, but to answer my own question on what is wrong with INPUT/OUTPUT there is nothing wrong but NAT and mangle may benefit from using a couple of new hooks for the "original" packets just at the border to IPSec due to the different rerouting requirements. NF_IP_LOCAL_IN/OUT is however an exact match for all existing netfilter module requirements on the encapsulated packets from what I can tell.

If there is a desire to use new hooks for the encapsulated packets then this should be done for the whole path of those packets, which obviously becomes somewhat tricky as it is not known yet at the NF_IP_PRE_ROUTING time of incoming packets that the encapsulated packet will go to IPSec and a bastard on final output of transmitted packets.

My slightly revised proposal is now (this time using hook names to avoid confusion)

(Client) -> Gateway -> (Gateway)

1. NF_IP_PRE_ROUTING
2. NF_IP_FORWARD
3. NF_IP_XXXXXXXX, transparently mapped to the POSTROUTING chain in iptables.

[IPSec encapsulation]

4. NF_IP_LOCAL_OUT
5. NF_IP_POST_ROUTING

(Gateway) -> Gateway -> (client)

1. NF_IP_PRE_ROUTING
2. NF_IP_LOCAL_IN

[IPSec decapsulation]

3. NF_IP_YYYYYYYY, transparently mapped to the PREROUTING chain in iptables
4. NF_IP_FORWARD
5. NF_IP_POST_ROUTING

And similarily for the Host -> Gateway/Host case (locally generated packets).

But I'd say the use of new hooks should be avoided unless seen needed when implementing rerouting. Similar "rerouting" needs to be done in the normal hook to allow netfilter packet modifications to change the routing/policy decision to cause a previously clear text packet to be sent over IPSec as is needed on the same changes to packets decided to go to an IPSec SPI.

My problem with adding new hooks is that I don't see a value in adding new types of hooks if all uses of this hook will be mapped to the exact same code paths.

What is pretty clear is that the "rerouting" requirements is far from obvious and needs to be analyzed carefully. If one thought rerouting of packets in the plain IP with ip_route_me_harder was a mess the situation is now some orders of magnitude more complex it seems, at least from a quick evaluation without digging into the actual 2.6 IPSec code.

Regards
Henrik

Well, I now had to find out that libiptc depends on a 'one hook per chain' assumption, too bad. My idea was to attach the same chain to multiple hooks. That is not possible, as of now.

As you can see in your patch, I gave up on all of that and just use the existing hooks.

yes, indeed. Unfortunately, analyzing this whole issue has alrady cost me too much of my time the last couple of days :( I'll have to defer it for now.

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

Hi guys,

On Tue, Jan 27, 2004 at 09:51:23PM +0100, Harald Welte wrote:
<...>

Cool, I'm very glad that we're at least several ones to think it the same way. I didn't respond because my proposal would have been identical to Henrik's. So please count my vote :-)

Thanks,
Willy

Hi!

I've now hacked a preliminary patch. Be warned, I didn't even test if it compiles. Maybe someone who actually has a running 2.6.x ipsec setup can give it a try.

What it should be doing
- traverse the additional chains as described in the last email
- have connection tracking recognize two seperate connections, one for the decapsulated traffic, one for the encapsulated
- thus, even SNAT/DNAT should be working.
- locally-encapsulated traffic shows up with input device "ah4" or "esp4" in POSTROUTING.
- locally-encapsulated traffic shows up with output device "ah4" or "esp4" in OUTPUT.

What is missing (TODO):
- no dummy device names in INPUT/PREROUTING for locally-decapsulated packets. This is somewhat harder
- no real output device shown in OUTPUT for locally-encapsulated packets. I'm not sure if it is legal to typecast the just-popped dst_entry to 'struct rtable' and derive the output interface from there.

Please give feedback.

P.S.: Yes, the #ifdef's are ugly. I know.

diff -Nru linuxppc25bh-031218-orinoco_monitor-gpl/net/ipv4/ah4.c linuxppc25bh-031218-orinoco_monitor-gpl-netfipsec/net/ipv4/ah4.c
@@ -9,6 +9,12 @@
 #include <net/icmp.h>
 #include <asm/scatterlist.h>
 
+static struct net_device *ah_dummydev;
+
+static int ah_xmit_bypass(struct sk_buff *skb)
+{
+	return NET_XMIT_BYPASS;
+}
 
 /* Clear mutable options and find final destination to substitute
  * into IP header for icv calculation. Options are already checked

@@ -54,7 +60,7 @@
 	return 0;
 }
 
-static int ah_output(struct sk_buff *skb)
+static int ah_output_finish(struct sk_buff *skb)
 {
 	int err;
 	struct dst_entry *dst = skb->dst;

@@ -145,7 +151,17 @@
 		err = -EHOSTUNREACH;
 		goto error_nolock;
 	}
-	return NET_XMIT_BYPASS;
+
+#ifdef CONFIG_NETFILTER
+	nf_conntrack_put(skb->conntrack);
+	skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+	skb->nf_debug = 0;
+#endif
+#endif
+
+	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, ah_dummydev, NULL,
+		       ah_xmit_bypass);
 
 error:
 	spin_unlock_bh(&x->lock);

@@ -154,6 +170,12 @@
 	return err;
 }
 
+static ah_output(struct sk_buff *skb)
+{
+	return NF_HOOK(PF_INET, IP_NF_POST_ROUTING, skb, skb->dev, ah_dummydev,
+		       ah_output_finish);
+}
+
 int ah_input(struct xfrm_state *x, struct xfrm_decap_state *decap, struct sk_buff *skb)
 {
 	int ah_hlen;

@@ -354,11 +376,18 @@
 		xfrm_unregister_type(&ah_type, AF_INET);
 		return -EAGAIN;
 	}
+	ah_dummydev = alloc_netdev(0, "ah4", NULL);
+	if (!ah_dummydev) {
+		printk(KERN_INFO "ip ah init: can't allocate dummydev\n");
+		inet_del_protocol(&ah4_protocol, IPPROTO_AH);
+		xfrm_unregister_type(&ah_type, AF_INET);
+	}
 	return 0;
 }
 
 static void __exit ah4_fini(void)
 {
+	free_netdev(ah_dummydev);
 	if (inet_del_protocol(&ah4_protocol, IPPROTO_AH) < 0)
 		printk(KERN_INFO "ip ah close: can't remove protocol\n");
 	if (xfrm_unregister_type(&ah_type, AF_INET) < 0)

diff -Nru linuxppc25bh-031218-orinoco_monitor-gpl/net/ipv4/esp4.c linuxppc25bh-031218-orinoco_monitor-gpl-netfipsec/net/ipv4/esp4.c
@@ -20,7 +20,16 @@
 	__u8		proto;
 };
 
-int esp_output(struct sk_buff *skb)
+static struct net_device *esp_dummydev;
+
+/* dummy function for netfilter.  esp_output_finish() has to return
+ * that particular value upon success */
+static int esp_xmit_bypass(struct sk_buff *skb)
+{
+	return NET_XMIT_BYPASS;
+}
+
+static int esp_output_finish(struct sk_buff *skb)
 {
 	int err;
 	struct dst_entry *dst = skb->dst;

@@ -199,7 +208,17 @@
 		err = -EHOSTUNREACH;
 		goto error_nolock;
 	}
-	return NET_XMIT_BYPASS;
+
+#ifdef CONFIG_NETFILTER
+	nf_conntrack_put(skb->conntrack);
+	skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+	skb->nf_debug = 0;
+#endif
+#endif
+	/* FIXME: return real destination device */
+	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, esp_dummydev, NULL,
+		       esp_xmit_bypass);
 
 error:
 	spin_unlock_bh(&x->lock);

@@ -208,6 +227,12 @@
 	return err;
 }
 
+int esp_output(struct sk_buff *skb)
+{
+	return NF_HOOK(PF_INET, NF_IP_POST_ROUTING, skb, skb->dev, esp_dummydev,
+		       esp_output_finish);
+}
+
 /*
  * Note: detecting truncated vs. non-truncated authentication data is very
  * expensive, so we only support truncated data, which is the recommended

@@ -596,11 +621,19 @@
 		xfrm_unregister_type(&esp_type, AF_INET);
 		return -EAGAIN;
 	}
+	esp_dummydev = alloc_netdev(0, "esp4", NULL);
+	if (!esp_dummydev) {
+		printk(KERN_INFO "ip esp init: can't allocate dummydev\n");
+		inet_del_protocol(&esp4_protocol, IPPROTO_ESP);
+		xfrm_unregister_type(&esp_type, AF_INET);
+		return -EAGAIN;
+	}
 	return 0;
 }
 
 static void __exit esp4_fini(void)
 {
+	free_netdev(esp_dummydev);
 	if (inet_del_protocol(&esp4_protocol, IPPROTO_ESP) < 0)
 		printk(KERN_INFO "ip esp close: can't remove protocol\n");
 	if (xfrm_unregister_type(&esp_type, AF_INET) < 0)

diff -Nru linuxppc25bh-031218-orinoco_monitor-gpl/net/ipv4/xfrm4_input.c linuxppc25bh-031218-orinoco_monitor-gpl-netfipsec/net/ipv4/xfrm4_input.c
@@ -6,6 +6,8 @@
  *		Split up af-specific portion
  *	Derek Atkins <[email protected]>
  *		Add Encapsulation support
+ *	Harald Welte <[email protected]>
+ *		Drop conntrack reference before calling netif_rx()
  * 	
  */
 

@@ -130,6 +132,13 @@
 			dst_release(skb->dst);
 			skb->dst = NULL;
 		}
+#ifdef CONFIG_NETFILTER
+		nf_conntrack_put(skb->conntrack);
+		skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+		skb->nf_debug = 0;
+#endif
+#endif
 		netif_rx(skb);
 		return 0;
 	} else {

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

I have somewhere a working 2.6 ipsec backport to 2.4 thanks to Davem and Herbert Xu. I'll see how it applies on it and if it works too, because this is nearly the only thing refraining me from definitely switching over this new stack. But don't expect any feedback before this week-end.

Thanks,
willy

Hi!

[please ignore the last message, I mistakenly sent an outdated, incomplete
patch]

I've now hacked a preliminary patch. Be warned, I didn't even test if it compiles. Maybe someone who actually has a running 2.6.x ipsec setup can give it a try.

What it should be doing
- traverse the additional chains as described in the last email
- have connection tracking recognize two seperate connections, one for the decapsulated traffic, one for the encapsulated
- thus, even SNAT/DNAT should be working.
- locally-encapsulated traffic shows up with input device "ah4" or "esp4" in POSTROUTING.
- locally-encapsulated traffic shows up with output device "ah4" or "esp4" in OUTPUT.

What is missing (TODO):
- no dummy device names in INPUT/PREROUTING for locally-decapsulated packets. This is somewhat harder
- no real output device shown in OUTPUT for locally-encapsulated packets. I'm not sure if it is legal to typecast the just-popped dst_entry to 'struct rtable' and derive the output interface from there.

Please give feedback.

P.S.: Yes, the #ifdef's are ugly. I know.

diff -Nru linuxppc25bh-031218-orinoco_monitor-gpl/net/ipv4/ah4.c linuxppc25bh-031218-orinoco_monitor-gpl-netfipsec/net/ipv4/ah4.c
@@ -7,8 +7,15 @@
 #include <linux/crypto.h>
 #include <linux/pfkeyv2.h>
 #include <net/icmp.h>
+#include <linux/netfilter_ipv4.h>
 #include <asm/scatterlist.h>
 
+static struct net_device *ah_dummydev;
+
+static int ah_xmit_bypass(struct sk_buff *skb)
+{
+	return NET_XMIT_BYPASS;
+}
 
 /* Clear mutable options and find final destination to substitute
  * into IP header for icv calculation. Options are already checked

@@ -54,7 +61,7 @@
 	return 0;
 }
 
-static int ah_output(struct sk_buff *skb)
+static int ah_output_finish(struct sk_buff *skb)
 {
 	int err;
 	struct dst_entry *dst = skb->dst;

@@ -145,7 +152,17 @@
 		err = -EHOSTUNREACH;
 		goto error_nolock;
 	}
-	return NET_XMIT_BYPASS;
+
+#ifdef CONFIG_NETFILTER
+	nf_conntrack_put(skb->nfct);
+	skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+	skb->nf_debug = 0;
+#endif
+#endif
+
+	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, ah_dummydev, NULL,
+		       ah_xmit_bypass);
 
 error:
 	spin_unlock_bh(&x->lock);

@@ -154,6 +171,12 @@
 	return err;
 }
 
+static int ah_output(struct sk_buff *skb)
+{
+	return NF_HOOK(PF_INET, NF_IP_POST_ROUTING, skb, skb->dev, ah_dummydev,
+		       ah_output_finish);
+}
+
 int ah_input(struct xfrm_state *x, struct xfrm_decap_state *decap, struct sk_buff *skb)
 {
 	int ah_hlen;

@@ -354,11 +377,18 @@
 		xfrm_unregister_type(&ah_type, AF_INET);
 		return -EAGAIN;
 	}
+	ah_dummydev = alloc_netdev(0, "ah4", NULL);
+	if (!ah_dummydev) {
+		printk(KERN_INFO "ip ah init: can't allocate dummydev\n");
+		inet_del_protocol(&ah4_protocol, IPPROTO_AH);
+		xfrm_unregister_type(&ah_type, AF_INET);
+	}
 	return 0;
 }
 
 static void __exit ah4_fini(void)
 {
+	free_netdev(ah_dummydev);
 	if (inet_del_protocol(&ah4_protocol, IPPROTO_AH) < 0)
 		printk(KERN_INFO "ip ah close: can't remove protocol\n");
 	if (xfrm_unregister_type(&ah_type, AF_INET) < 0)

diff -Nru linuxppc25bh-031218-orinoco_monitor-gpl/net/ipv4/esp4.c linuxppc25bh-031218-orinoco_monitor-gpl-netfipsec/net/ipv4/esp4.c
@@ -8,6 +8,7 @@
 #include <linux/crypto.h>
 #include <linux/pfkeyv2.h>
 #include <linux/random.h>
+#include <linux/netfilter_ipv4.h>
 #include <net/icmp.h>
 #include <net/udp.h>
 

@@ -20,7 +21,16 @@
 	__u8		proto;
 };
 
-int esp_output(struct sk_buff *skb)
+static struct net_device *esp_dummydev;
+
+/* dummy function for netfilter.  esp_output_finish() has to return
+ * that particular value upon success */
+static int esp_xmit_bypass(struct sk_buff *skb)
+{
+	return NET_XMIT_BYPASS;
+}
+
+static int esp_output_finish(struct sk_buff *skb)
 {
 	int err;
 	struct dst_entry *dst = skb->dst;

@@ -199,7 +209,17 @@
 		err = -EHOSTUNREACH;
 		goto error_nolock;
 	}
-	return NET_XMIT_BYPASS;
+
+#ifdef CONFIG_NETFILTER
+	nf_conntrack_put(skb->nfct);
+	skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+	skb->nf_debug = 0;
+#endif
+#endif
+	/* FIXME: return real destination device */
+	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, esp_dummydev, NULL,
+		       esp_xmit_bypass);
 
 error:
 	spin_unlock_bh(&x->lock);

@@ -208,6 +228,12 @@
 	return err;
 }
 
+int esp_output(struct sk_buff *skb)
+{
+	return NF_HOOK(PF_INET, NF_IP_POST_ROUTING, skb, skb->dev, esp_dummydev,
+		       esp_output_finish);
+}
+
 /*
  * Note: detecting truncated vs. non-truncated authentication data is very
  * expensive, so we only support truncated data, which is the recommended

@@ -596,11 +622,19 @@
 		xfrm_unregister_type(&esp_type, AF_INET);
 		return -EAGAIN;
 	}
+	esp_dummydev = alloc_netdev(0, "esp4", NULL);
+	if (!esp_dummydev) {
+		printk(KERN_INFO "ip esp init: can't allocate dummydev\n");
+		inet_del_protocol(&esp4_protocol, IPPROTO_ESP);
+		xfrm_unregister_type(&esp_type, AF_INET);
+		return -EAGAIN;
+	}
 	return 0;
 }
 
 static void __exit esp4_fini(void)
 {
+	free_netdev(esp_dummydev);
 	if (inet_del_protocol(&esp4_protocol, IPPROTO_ESP) < 0)
 		printk(KERN_INFO "ip esp close: can't remove protocol\n");
 	if (xfrm_unregister_type(&esp_type, AF_INET) < 0)

diff -Nru linuxppc25bh-031218-orinoco_monitor-gpl/net/ipv4/xfrm4_input.c linuxppc25bh-031218-orinoco_monitor-gpl-netfipsec/net/ipv4/xfrm4_input.c
@@ -6,6 +6,8 @@
  *		Split up af-specific portion
  *	Derek Atkins <[email protected]>
  *		Add Encapsulation support
+ *	Harald Welte <[email protected]>
+ *		Drop conntrack reference before calling netif_rx()
  * 	
  */
 

@@ -130,6 +132,13 @@
 			dst_release(skb->dst);
 			skb->dst = NULL;
 		}
+#ifdef CONFIG_NETFILTER
+		nf_conntrack_put(skb->nfct);
+		skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+		skb->nf_debug = 0;
+#endif
+#endif
 		netif_rx(skb);
 		return 0;
 	} else {

--
- Harald Welte <[email protected]> http://www.netfilter.org/
============================================================================
"Fragmentation is like classful addressing -- an interesting early
architectural error that shows how much experimentation was going
on while IP was being designed." -- Paul Vixie

I see two problems with this approach. The dummy devices don't have any ip config, so f.e. REDIRECT will fail. The bigger problem is hooking in output routines that return NET_XMIT_BYPASS. dst_output loops until the return code of skb->dst->output != NET_XMIT_BYPASS. These output routines replace skb->dst when finished by calling dst_pop. If we pass the packet through netfilter in between, the dst_entry might get replaced in ip_route_me_harder or elsewhere and not all transformations will be applied. If NAT is used, ip_route_{input,output} might even return a different policy bundle.

Anyways, I'm testing it (slightly hacked) as soon as the compile finishes ;)

Regards,
Patrick

Bad news, I've tested the hook part (using skb->dev instead of ah4/esp4 devices), this is how the traces look:

IN= OUT=eth0 SRC=172.20.0.3 DST=192.168.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=28443 SEQ=1 IN= OUT= SRC=172.20.0.3 DST=192.168.0.1 LEN=152 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ESP SPI=0xb9f7900 IN= OUT= SRC=172.20.0.3 DST=192.168.0.1 LEN=176 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=AH SPI=0xc46e27

After applying a MARK rule to the first packet (causing rerouting):

IN= OUT=eth0 SRC=172.20.0.3 DST=192.168.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=51484 SEQ=1

Sniffing in between shows the unencrypted packet.

Ok, let's ignore that for now. dunno if the dummydev idea was so good at all.

I see. So we would have to modify all code that changes skb->dst to
check if there is a dst stack. If yes, it would have to iterate over all of them and just change the last one. Shouldn't be too hard to intergrate that change.

However, it is hard to tell what is the correct behaviour in that case.

In POST_ROUTING of the original, unencapsulated packet, I would say it is correct to not apply those transformations, in case rerouting of the packet occurs. If somebody reroutes here, he wants to affect the original packet, not the encapsulated one.

However, once we did do the first encapsulation, it doesn't make sense to encapsulate further layers until we've reached the real dst. At this point, LOCAL_OUT would be called with the heading-for-the-wire packet and rerouting will and should affect the final device.

However, exposing all those intermediate steps to the netfilter-hook-attached code is not so bad. It enables people to do whatever they want... they just need to be careful with their rules. If there's too much interference with normal OUTPUT/POSTROUTING rules, we could still go for new hooks+new chains.

The question is, again: What ist the desired behaviour? Should the policy be determined on the un-NAT'ed packet or on the NAT'ed one?

good luck :)

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

Harald, I believe it's important to remember that NAT is not the only usage of this extension. For many people seeking security (=those who install VPN gateways for customers), it is very important to :
- be able to filter what enters and leaves a tunnel
- be able to filter where the encapsulated packets go to/come from

NAT is complementary IMHO.

Regards,
Willy

Yes, I totally agree. My major point was conntrack being broken, and thus stateful packet filtering not being possible. The respective iptables chains are missing.

Both of them are needed for the additional making-NAT-work.

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

I agree. And the 2.4 implementation of IPSEC allowed this control to be exercised in a natural and uniform way (the same model applied for all tunnel types including IPSEC).

- Unencrypted traffic is exchanged between the client and a special device.
- encrypted traffic is exchanged between the gateways.

These are separately tracked connections that obey the normal netfilter rules.

For configured tunnels, I still believe that this is the proper model.

Opportunistic keying OTOH probably shouldn't follow that model.

But there are instances where it is necessary.

-Tom

Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ [email protected]

I believe we pass the real device and provide a new match. The real device might be interesting for filtering, too. But as you say, let's ignore it for now.

Agreed. The problem is outfn is already set to {ah,esp}_output when POST_ROUTING of the original packet is called, we need to handle this somehow.

We also should handle the opposite case, after a (normal) packet is SNATed ip_route_me_harder will replace the dst_entry, the new one might include ipsec transformations if a policy for the now-different packet exists, but the packet is always passed to ip_finish_output2. If we would call dst_output instead strange hook orders can occur:
PRE_ROUTING FORWARD POST_ROUTING <SNAT, encapsulate>
POST_ROUTING LOCAL_OUT <SNAT again ;)> ..
I can't imagine a good solution right now.

I'm not sure if you understand you correctly. Do you mean to say that it
doesn't make sense to pass packets to netfilter hooks until all transformations have been applied ? If so, I agree, I don't think there is much use in doing stuff to half-done packets. We can easily detect
the final dst_entry, it should have dst->xfrm = NULL (at least I would
think so). But I don't know how to skip intermediate ones, I guess
they look similar to the first one.

This makes me think I misunderstood your statement above. I recall there was some confusion between hook functions and hooks before, it seems I got me, too ;) Do you propose to make these steps visible to iptables chains or just to the registered hook functions, which would hide them from the user by immediately returning in the iptables case ?

The NATed one. When the policy says encrypt 10.0.0.1<->10.0.0.2, NAT should bypass that. IIRC Herbert Xu mentioned some things about when policy checks need to be done in the "2.6 IPSEC + SNAT" thread. I'm going to read it again later.

IN my opinion it must be on the NAT'ed one, or else there is a very large risk the packet does not match the selected policy and the other endpoint will reject the resulting IPSec packet.

Regards
Henrik

I've hacked together a working version, it's a ugly hack but it shows what parts are required for getting NAT working. Comments are welcome.

The patch is split into three parts: 01-hooks.diff, 02-nat-original.diff
and 03-nat-reply.diff. I'm going to discuss each part seperately.

01-hooks.diff:
This patch adds the required hooks. Locally generated and forwarded packets pass through POST_ROUTING before beeing encypted and pass through LOCAL_OUT afterwards. This is currently not consistent with input packets which pass through PRE_ROUTING and LOCAL_IN encrypted and then again after beeing decapsulated from a tunnel-mode encapsulation. This means packets from nested tunnels will pass multiple times, packets from transport-mode connections won't pass the hook after decryption at all.

02-nat-original.diff:
This patch adds policy lookups to ip_route_me_harder and changes ip_nat_standalone to reroute in both LOCAL_OUT and POST_ROUTING if the routing key or the key used for policy lookups changed. This allows to NAT packets and have the correct policy applied to them afterwards.

Since the POST_ROUTING hook is called with ip_finish_output2 as outfn the packets which have a transformer-dst_entry after ip_route_me_harder are manually confirmed (conntrack) and passed to dst_output. These packets will be passed through LOCAL_OUT after encryption has been done. There is a danger of loops, the encrypted packets could be natted in LOCAL_OUT and POST_ROUTING again and again match a policy.

The packets matching a policy after rerouting in LOCAL_OUT don't need this kind of special handling, LOCAL_OUT is called with dst_output as outfn. They are currently not passwd through POST_ROUTING before encryption, which I just noticed while writing this mail.

03-nat-reply.diff:
This patch changes xfrm_policy_check to lookup the correct policy for input packets after NAT has been applied. Unfortunately some policy checks are performed after skb->nfct has been released so I had to introduce a leak.

To summarize the problems:
- Not completely consistens picture wrt. hooks passed and ordering. Should be easily fixable.
- loop danger. suggestions ?
- nat information required at socket receive time for policy checks, currently achieved by introducing a leak. possible solution is to store the required data in new skb fields, it's only 6 bytes ..

What is currently working is the following:
- Filter traffic before/after encryption (more or less)
- SNAT/DNAT packets and have correct policies applied, including none
- Have policy checks find correct policy for SNAT/DNAT/SNAT+DNATed packets

Regards,
Patrick

===== net/ipv4/ah4.c 1.29 vs edited =====

@@ -6,6 +6,8 @@
 #include <net/ah.h>
 #include <linux/crypto.h>
 #include <linux/pfkeyv2.h>
+#include <linux/netfilter.h>
+#include <linux/netfilter_ipv4.h>
 #include <net/icmp.h>
 #include <asm/scatterlist.h>
 

@@ -54,6 +56,11 @@
 	return 0;
 }
 
+static inline int ah_finish_output(struct sk_buff *skb)
+{
+	return NET_XMIT_BYPASS;
+}
+
 static int ah_output(struct sk_buff *skb)
 {
 	int err;

@@ -144,6 +151,18 @@
 	if ((skb->dst = dst_pop(dst)) == NULL) {
 		err = -EHOSTUNREACH;
 		goto error_nolock;
+	}
+	/* final packet goes through LOCAL_OUT hook */
+	if (skb->dst->xfrm == NULL) {
+#ifdef CONFIG_NETFILTER
+		nf_conntrack_put(skb->nfct);
+		skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+		skb->nf_debug = 0;
+#endif
+#endif
+		return NF_HOOK(AF_INET, NF_IP_LOCAL_OUT, skb, NULL,
+		               skb->dst->dev, ah_finish_output);
 	}
 	return NET_XMIT_BYPASS;
 

===== net/ipv4/esp4.c 1.35 vs edited =====

@@ -8,6 +8,8 @@
 #include <linux/crypto.h>
 #include <linux/pfkeyv2.h>
 #include <linux/random.h>
+#include <linux/netfilter.h>
+#include <linux/netfilter_ipv4.h>
 #include <net/icmp.h>
 #include <net/udp.h>
 

@@ -20,6 +22,11 @@
 	__u8		proto;
 };
 
+static inline int esp_finish_output(struct sk_buff *skb)
+{
+	return NET_XMIT_BYPASS;
+}
+
 int esp_output(struct sk_buff *skb)
 {
 	int err;

@@ -198,6 +205,18 @@
 	if ((skb->dst = dst_pop(dst)) == NULL) {
 		err = -EHOSTUNREACH;
 		goto error_nolock;
+	}
+	/* final packet goes through LOCAL_OUT hook */
+	if (skb->dst->xfrm == NULL) {
+#ifdef CONFIG_NETFILTER
+		nf_conntrack_put(skb->nfct);
+		skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+		skb->nf_debug = 0;
+#endif
+#endif
+		return NF_HOOK(AF_INET, NF_IP_LOCAL_OUT, skb, NULL,
+		               skb->dst->dev, esp_finish_output);
 	}
 	return NET_XMIT_BYPASS;
 

===== net/ipv4/ip_forward.c 1.9 vs edited =====

@@ -51,6 +51,10 @@
 	if (unlikely(opt->optlen))
 		ip_forward_options(skb);
 
+	if (skb->dst->xfrm != NULL)
+		return NF_HOOK(PF_INET, NF_IP_POST_ROUTING, skb, NULL, skb->dst->dev,
+		               dst_output);
+
 	return dst_output(skb);
 }
 

===== net/ipv4/ip_output.c 1.48 vs edited =====

@@ -122,6 +122,14 @@
 	return ttl;
 }
 
+static inline int ip_dst_output(struct sk_buff *skb)
+{
+	if (skb->dst->xfrm != NULL)
+		return NF_HOOK(PF_INET, NF_IP_POST_ROUTING, skb, NULL,
+		               skb->dst->dev, dst_output);
+	return dst_output(skb);
+}
+
 /* 
  *		Add an ip header to a skbuff and send it out.
  *

@@ -164,7 +172,7 @@
 
 	/* Send it out. */
 	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
-		       dst_output);
+		       ip_dst_output);
 }
 
 static inline int ip_finish_output2(struct sk_buff *skb)

@@ -386,7 +394,7 @@
 	skb->priority = sk->sk_priority;
 
 	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
-		       dst_output);
+		       ip_dst_output);
 
 no_route:
 	IP_INC_STATS(IpOutNoRoutes);

@@ -1164,7 +1172,7 @@
 
 	/* Netfilter gets whole the not fragmented skb. */
 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, 
-		      skb->dst->dev, dst_output);
+		      skb->dst->dev, ip_dst_output);
 	if (err) {
 		if (err > 0)
 			err = inet->recverr ? net_xmit_errno(err) : 0;

===== net/ipv4/xfrm4_input.c 1.9 vs edited =====

@@ -130,6 +130,13 @@
 			dst_release(skb->dst);
 			skb->dst = NULL;
 		}
+#ifdef CONFIG_NETFILTER
+		nf_conntrack_put(skb->nfct);
+		skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+		skb->nf_debug = 0;
+#endif
+#endif
 		netif_rx(skb);
 		return 0;
 	} else {

===== net/core/netfilter.c 1.26 vs edited =====

@@ -25,6 +25,7 @@
 #include <linux/icmp.h>
 #include <net/sock.h>
 #include <net/route.h>
+#include <net/xfrm.h>
 #include <linux/ip.h>
 
 #define __KERNEL_SYSCALLS__

@@ -627,6 +628,9 @@
 	struct flowi fl = {};
 	struct dst_entry *odst;
 	unsigned int hh_len;
+#ifdef CONFIG_XFRM
+	struct xfrm_policy_afinfo *afinfo;
+#endif
 
 	/* some non-standard hacks like ipt_REJECT.c:send_reset() can cause
 	 * packets with foreign saddr to appear on the NF_IP_LOCAL_OUT hook.

@@ -665,6 +669,16 @@
 	if ((*pskb)->dst->error)
 		return -1;
 
+#ifdef CONFIG_XFRM
+	afinfo = xfrm_policy_get_afinfo(AF_INET);
+	if (afinfo != NULL) {
+		afinfo->decode_session(*pskb, &fl);
+		xfrm_policy_put_afinfo(afinfo);
+		if (xfrm_lookup(&(*pskb)->dst, &fl, (*pskb)->sk, 1) != 0)
+			return -1;
+	}
+#endif
+
 	/* Change in oif may mean change in hh_len. */
 	hh_len = (*pskb)->dst->dev->hard_header_len;
 	if (skb_headroom(*pskb) < hh_len) {

===== net/ipv4/netfilter/ip_conntrack_standalone.c 1.23 vs edited =====

@@ -481,6 +481,7 @@
 EXPORT_SYMBOL(ip_conntrack_alter_reply);
 EXPORT_SYMBOL(ip_conntrack_destroyed);
 EXPORT_SYMBOL(ip_conntrack_get);
+EXPORT_SYMBOL(__ip_conntrack_confirm);
 EXPORT_SYMBOL(need_ip_conntrack);
 EXPORT_SYMBOL(ip_conntrack_helper_register);
 EXPORT_SYMBOL(ip_conntrack_helper_unregister);

===== net/ipv4/netfilter/ip_nat_standalone.c 1.28 vs edited =====

===== include/linux/netfilter.h 1.6 vs edited =====

@@ -166,5 +166,21 @@
 #define NF_HOOK(pf, hook, skb, indev, outdev, okfn) (okfn)(skb)
 #endif /*CONFIG_NETFILTER*/
 
+#ifdef CONFIG_XFRM
+#ifdef CONFIG_IP_NF_NAT_NEEDED
+struct flowi;
+extern void nf_nat_decode_session4(struct sk_buff *skb, struct flowi *fl);
+
+static inline void
+nf_nat_decode_session(struct sk_buff *skb, struct flowi *fl, int family)
+{
+	if (family == AF_INET)
+		nf_nat_decode_session4(skb, fl);
+}
+#else /* CONFIG_IP_NF_NAT_NEEDED */
+#define nf_nat_decode_session(skb, fl, family)
+#endif /* CONFIG_IP_NF_NAT_NEEDED */
+#endif /* CONFIG_XFRM */
+
 #endif /*__KERNEL__*/
 #endif /*__LINUX_NETFILTER_H*/

===== net/core/netfilter.c 1.26 vs edited =====

===== net/ipv4/ip_input.c 1.20 vs edited =====

@@ -207,12 +207,14 @@
 
 	__skb_pull(skb, ihl);
 
+#if 0
 #ifdef CONFIG_NETFILTER
 	/* Free reference early: we don't need it any more, and it may
            hold ip_conntrack module loaded indefinitely. */
 	nf_conntrack_put(skb->nfct);
 	skb->nfct = NULL;
 #endif /*CONFIG_NETFILTER*/
+#endif
 
         /* Point into the IP datagram, just past the header. */
         skb->h.raw = skb->data;

===== net/xfrm/xfrm_policy.c 1.47 vs edited =====

@@ -21,6 +21,7 @@
 #include <linux/workqueue.h>
 #include <linux/notifier.h>
 #include <linux/netdevice.h>
+#include <linux/netfilter.h>
 #include <net/xfrm.h>
 #include <net/ip.h>
 

@@ -908,6 +909,7 @@
 
 	if (_decode_session(skb, &fl, family) < 0)
 		return 0;
+	nf_nat_decode_session(skb, &fl, family);
 
 	/* First, check used SA against their selectors. */
 	if (skb->sp) {

I've continued to work on the IPSEC+NAT issue, this is the current status. The hooks for pre/post-ipsec are now entirely contained in ip_output.c. The transformers mark transformed packets with a new flag in the control buffer, this is used for directing ipsec packets to LOCAL_OUT in ip_output(). It also solves the loop danger when redirecting packets in ip_nat_out after rerouting, ip_route_me_harder only looks up a policy for untransformed packets. Redirected packets are manually confirmed, should this be changed to just mark them for redirection and do the actual redirecting in the conntrack hook ? The policy check part is fine now, the conntrack reference is kept as long as required for the ip protocol handlers that do policy checks, but it should be dropped on all paths before packets are queued to the socket. The nf_nat_decode_packet function now handles all cases of NAT which affect policy lookups, the last patches didn't handle DNAT rules in PREROUTING. The last big problem is the input side, I don't know how to make hook ordering symetric to output. Suggestions are very welcome.

There are 4 patches:

01-nf_reset.diff
Move common nf_conntrack_put/nfct=NULL/nf_debug=0 code to new inline function nf_reset.

02-hooks.diff
Make packets to be encrypted visible on POST_ROUTING hook and encrypted packets on LOCAL_OUT. Reset nfct etc. before reposting the packet into the stack on reception.

03-nat-policy-lookup.diff
Add policy lookups to ip_route_me_harder and change NAT to reroute for any change that affects routing or policy lookups

04-nat-policy-checks.diff
Make xfrm_policy_check find correct policy for NATed packets

Best regards,
Patrick

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/02/16 00:43:03+01:00 [email protected]
# Add new function 'nf_reset' to reset netfilter related skb-fields
#

# net/ipv6/sit.c
#   2004/02/16 00:41:03+01:00 [email protected] +2 -14
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# net/ipv6/ip6_tunnel.c
#   2004/02/16 00:41:03+01:00 [email protected] +1 -7
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# net/ipv4/ipip.c
#   2004/02/16 00:41:03+01:00 [email protected] +2 -14
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# net/ipv4/ip_input.c
#   2004/02/16 00:41:03+01:00 [email protected] +1 -5
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# net/ipv4/ip_gre.c
#   2004/02/16 00:41:03+01:00 [email protected] +2 -14
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# include/linux/skbuff.h
#   2004/02/16 00:41:03+01:00 [email protected] +12 -3
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
diff -Nru a/include/linux/skbuff.h b/include/linux/skbuff.h
--- a/include/linux/skbuff.h	Mon Feb 16 02:12:55 2004
+++ b/include/linux/skbuff.h	Mon Feb 16 02:12:55 2004
@@ -1201,6 +1201,14 @@
 	if (nfct)
 		atomic_inc(&nfct->master->use);
 }
+static inline void nf_reset(struct sk_buff *skb)
+{
+	nf_conntrack_put(skb->nfct);
+	skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+	skb->nf_debug = 0;
+#endif
+}

#ifdef CONFIG_BRIDGE_NETFILTER
static inline void nf_bridge_put(struct nf_bridge_info *nf_bridge)
@@ -1213,9 +1221,10 @@
if (nf_bridge)
atomic_inc(&nf_bridge->use);
}
-#endif
-
-#endif
+#endif /* CONFIG_BRIDGE_NETFILTER */
+#else /* CONFIG_NETFILTER */
+static inline void nf_reset(struct sk_buff *skb) {} +#endif /* CONFIG_NETFILTER */

#endif /* __KERNEL__ */
#endif /* _LINUX_SKBUFF_H */

diff -Nru a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
@@ -643,13 +643,7 @@
 		skb->dev = tunnel->dev;
 		dst_release(skb->dst);
 		skb->dst = NULL;
-#ifdef CONFIG_NETFILTER
-		nf_conntrack_put(skb->nfct);
-		skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-		skb->nf_debug = 0;
-#endif
-#endif
+		nf_reset(skb);
 		ipgre_ecn_decapsulate(iph, skb);
 		netif_rx(skb);
 		read_unlock(&ipgre_lock);

@@ -877,13 +871,7 @@
 		}
 	}
 
-#ifdef CONFIG_NETFILTER
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-	skb->nf_debug = 0;
-#endif
-#endif
+	nf_reset(skb);
 
 	IPTUNNEL_XMIT();
 	tunnel->recursion--;

diff -Nru a/net/ipv4/ip_input.c b/net/ipv4/ip_input.c
@@ -202,17 +202,13 @@
 
 #ifdef CONFIG_NETFILTER_DEBUG
 	nf_debug_ip_local_deliver(skb);
-	skb->nf_debug = 0;
 #endif /*CONFIG_NETFILTER_DEBUG*/
 
 	__skb_pull(skb, ihl);
 
-#ifdef CONFIG_NETFILTER
 	/* Free reference early: we don't need it any more, and it may
            hold ip_conntrack module loaded indefinitely. */
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#endif /*CONFIG_NETFILTER*/
+	nf_reset(skb);
 
         /* Point into the IP datagram, just past the header. */
         skb->h.raw = skb->data;

diff -Nru a/net/ipv4/ipip.c b/net/ipv4/ipip.c
@@ -496,13 +496,7 @@
 		skb->dev = tunnel->dev;
 		dst_release(skb->dst);
 		skb->dst = NULL;
-#ifdef CONFIG_NETFILTER
-		nf_conntrack_put(skb->nfct);
-		skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-		skb->nf_debug = 0;
-#endif
-#endif
+		nf_reset(skb);
 		ipip_ecn_decapsulate(iph, skb);
 		netif_rx(skb);
 		read_unlock(&ipip_lock);

@@ -647,13 +641,7 @@
 	if ((iph->ttl = tiph->ttl) == 0)
 		iph->ttl	=	old_iph->ttl;
 
-#ifdef CONFIG_NETFILTER
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-	skb->nf_debug = 0;
-#endif
-#endif
+	nf_reset(skb);
 
 	IPTUNNEL_XMIT();
 	tunnel->recursion--;

diff -Nru a/net/ipv6/ip6_tunnel.c b/net/ipv6/ip6_tunnel.c
@@ -715,13 +715,7 @@
 	ipv6h->nexthdr = proto;
 	ipv6_addr_copy(&ipv6h->saddr, &fl.fl6_src);
 	ipv6_addr_copy(&ipv6h->daddr, &fl.fl6_dst);
-#ifdef CONFIG_NETFILTER
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-	skb->nf_debug = 0;
-#endif
-#endif
+	nf_reset(skb);
 	pkt_len = skb->len;
 	err = NF_HOOK(PF_INET6, NF_IP6_LOCAL_OUT, skb, NULL, 
 		      skb->dst->dev, dst_output);

diff -Nru a/net/ipv6/sit.c b/net/ipv6/sit.c
@@ -388,13 +388,7 @@
 		skb->dev = tunnel->dev;
 		dst_release(skb->dst);
 		skb->dst = NULL;
-#ifdef CONFIG_NETFILTER
-		nf_conntrack_put(skb->nfct);
-		skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-		skb->nf_debug = 0;
-#endif
-#endif
+		nf_reset(skb);
 		ipip6_ecn_decapsulate(iph, skb);
 		netif_rx(skb);
 		read_unlock(&ipip6_lock);

@@ -580,13 +574,7 @@
 	if ((iph->ttl = tiph->ttl) == 0)
 		iph->ttl	=	iph6->hop_limit;
 
-#ifdef CONFIG_NETFILTER
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-	skb->nf_debug = 0;
-#endif
-#endif
+	nf_reset(skb);
 
 	IPTUNNEL_XMIT();
 	tunnel->recursion--;

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet

#   2004/02/16 00:50:24+01:00 [email protected] 
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
# net/ipv4/xfrm4_tunnel.c
#   2004/02/16 00:50:18+01:00 [email protected] +1 -0
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
# net/ipv4/xfrm4_input.c
#   2004/02/16 00:50:18+01:00 [email protected] +1 -0
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
# net/ipv4/ipcomp.c
#   2004/02/16 00:50:18+01:00 [email protected] +1 -0
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
# net/ipv4/ip_output.c
#   2004/02/16 00:50:18+01:00 [email protected] +22 -4
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
# net/ipv4/ip_forward.c
#   2004/02/16 00:50:18+01:00 [email protected] +4 -0
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
# net/ipv4/esp4.c
#   2004/02/16 00:50:18+01:00 [email protected] +1 -0
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
# net/ipv4/ah4.c
#   2004/02/16 00:50:18+01:00 [email protected] +1 -0
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
# include/net/ip.h
#   2004/02/16 00:50:18+01:00 [email protected] +1 -0
#   Make packets visible to POST_ROUTING before encryption and LOCAL_OUT
#   afterwards, reset netfilter fields before re-posting packet into the
#   stack on reception.
# 
diff -Nru a/include/net/ip.h b/include/net/ip.h
--- a/include/net/ip.h	Mon Feb 16 02:13:07 2004
+++ b/include/net/ip.h	Mon Feb 16 02:13:07 2004
@@ -48,6 +48,7 @@
 #define IPSKB_TRANSLATED	2
 #define IPSKB_FORWARDED		4
 #define IPSKB_XFRM_TUNNEL_SIZE	8
+#define IPSKB_XFRM_TRANSFORMED	16
 };

struct ipcm_cookie

diff -Nru a/net/ipv4/ah4.c b/net/ipv4/ah4.c
@@ -137,6 +137,7 @@
 	ip_send_check(top_iph);
 
 	skb->nh.raw = skb->data;
+	IPCB(skb)->flags |= IPSKB_XFRM_TRANSFORMED;
 
 	x->curlft.bytes += skb->len;
 	x->curlft.packets++;

diff -Nru a/net/ipv4/esp4.c b/net/ipv4/esp4.c
@@ -191,6 +191,7 @@
 	ip_send_check(top_iph);
 
 	skb->nh.raw = skb->data;
+	IPCB(skb)->flags |= IPSKB_XFRM_TRANSFORMED;
 
 	x->curlft.bytes += skb->len;
 	x->curlft.packets++;

diff -Nru a/net/ipv4/ip_forward.c b/net/ipv4/ip_forward.c
@@ -51,6 +51,10 @@
 	if (unlikely(opt->optlen))
 		ip_forward_options(skb);
 
+	if (skb->dst->xfrm != NULL)
+		return NF_HOOK(PF_INET, NF_IP_POST_ROUTING, skb, NULL,
+		               skb->dst->dev, dst_output);
+
 	return dst_output(skb);
 }
 

diff -Nru a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
@@ -122,6 +122,14 @@
 	return ttl;
 }
 
+static inline int ip_dst_output(struct sk_buff *skb)
+{
+	if (skb->dst->xfrm != NULL)
+		return NF_HOOK(PF_INET, NF_IP_POST_ROUTING, skb, NULL,
+		               skb->dst->dev, dst_output);
+	return dst_output(skb);
+}
+
 /* 
  *		Add an ip header to a skbuff and send it out.
  *

@@ -164,7 +172,7 @@
 
 	/* Send it out. */
 	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
-		       dst_output);
+		       ip_dst_output);
 }
 
 static inline int ip_finish_output2(struct sk_buff *skb)

@@ -282,7 +290,7 @@
 		return ip_finish_output(skb);
 }
 
-int ip_output(struct sk_buff *skb)
+static inline int ip_output2(struct sk_buff *skb)
 {
 	IP_INC_STATS(IpOutRequests);
 

@@ -293,6 +301,16 @@
 		return ip_finish_output(skb);
 }
 
+int ip_output(struct sk_buff *skb)
+{
+	if (IPCB(skb)->flags & IPSKB_XFRM_TRANSFORMED) {
+		nf_reset(skb);
+		return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
+		               skb->dst->dev, ip_output2);
+	}
+	return ip_output2(skb);
+}
+
 int ip_queue_xmit(struct sk_buff *skb, int ipfragok)
 {
 	struct sock *sk = skb->sk;

@@ -386,7 +404,7 @@
 	skb->priority = sk->sk_priority;
 
 	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
-		       dst_output);
+		       ip_dst_output);
 
 no_route:
 	IP_INC_STATS(IpOutNoRoutes);

@@ -1165,7 +1183,7 @@
 
 	/* Netfilter gets whole the not fragmented skb. */
 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, 
-		      skb->dst->dev, dst_output);
+		      skb->dst->dev, ip_dst_output);
 	if (err) {
 		if (err > 0)
 			err = inet->recverr ? net_xmit_errno(err) : 0;

diff -Nru a/net/ipv4/ipcomp.c b/net/ipv4/ipcomp.c
@@ -223,6 +223,7 @@
 	skb->nh.raw = skb->data;
 
 out_ok:
+	IPCB(skb)->flags |= IPSKB_XFRM_TRANSFORMED;
 	x->curlft.bytes += skb->len;
 	x->curlft.packets++;
 	spin_unlock_bh(&x->lock);

diff -Nru a/net/ipv4/xfrm4_input.c b/net/ipv4/xfrm4_input.c
@@ -130,6 +130,7 @@
 			dst_release(skb->dst);
 			skb->dst = NULL;
 		}
+		nf_reset(skb);
 		netif_rx(skb);
 		return 0;
 	} else {

diff -Nru a/net/ipv4/xfrm4_tunnel.c b/net/ipv4/xfrm4_tunnel.c
@@ -66,6 +66,7 @@
 	ip_send_check(top_iph);
 
 	skb->nh.raw = skb->data;
+	IPCB(skb)->flags |= IPSKB_XFRM_TRANSFORMED;
 	x->curlft.bytes += skb->len;
 	x->curlft.packets++;
 

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet

#   2004/02/16 00:51:09+01:00 [email protected] 
#   Add policy lookups to ip_route_me_harder, make NAT reroute for any
#   change in route/policy key
# 
# net/ipv4/netfilter/ip_nat_standalone.c
#   2004/02/16 00:51:03+01:00 [email protected] +72 -8
#   Add policy lookups to ip_route_me_harder, make NAT reroute for any
#   change in route/policy key
# 
# net/ipv4/netfilter/ip_conntrack_standalone.c
#   2004/02/16 00:51:03+01:00 [email protected] +1 -0
#   Add policy lookups to ip_route_me_harder, make NAT reroute for any
#   change in route/policy key
# 
# net/core/netfilter.c
#   2004/02/16 00:51:03+01:00 [email protected] +16 -0
#   Add policy lookups to ip_route_me_harder, make NAT reroute for any
#   change in route/policy key
# 
diff -Nru a/net/core/netfilter.c b/net/core/netfilter.c
--- a/net/core/netfilter.c	Mon Feb 16 02:13:24 2004
+++ b/net/core/netfilter.c	Mon Feb 16 02:13:24 2004
@@ -25,6 +25,8 @@
 #include <linux/icmp.h>
 #include <net/sock.h>
 #include <net/route.h>
+#include <net/xfrm.h>
+#include <net/ip.h>
 #include <linux/ip.h>

#define __KERNEL_SYSCALLS__
@@ -664,6 +666,20 @@

if ((*pskb)->dst->error)
return -1;
+
+#ifdef CONFIG_XFRM
+ if (!(IPCB(*pskb)->flags & IPSKB_XFRM_TRANSFORMED)) {
+ struct xfrm_policy_afinfo *afinfo;
+
+ afinfo = xfrm_policy_get_afinfo(AF_INET);
+ if (afinfo != NULL) {
+ afinfo->decode_session(*pskb, &fl);
+ xfrm_policy_put_afinfo(afinfo);
+ if (xfrm_lookup(&(*pskb)->dst, &fl, (*pskb)->sk, 0) != 0)
+ return -1;
+ }
+ }
+#endif

/* Change in oif may mean change in hh_len. */ hh_len = (*pskb)->dst->dev->hard_header_len;

diff -Nru a/net/ipv4/netfilter/ip_conntrack_standalone.c b/net/ipv4/netfilter/ip_conntrack_standalone.c
@@ -489,6 +489,7 @@
 EXPORT_SYMBOL(ip_conntrack_alter_reply);
 EXPORT_SYMBOL(ip_conntrack_destroyed);
 EXPORT_SYMBOL(ip_conntrack_get);
+EXPORT_SYMBOL(__ip_conntrack_confirm);
 EXPORT_SYMBOL(need_ip_conntrack);
 EXPORT_SYMBOL(ip_conntrack_helper_register);
 EXPORT_SYMBOL(ip_conntrack_helper_unregister);

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/02/16 00:54:30+01:00 [email protected]
# Make policy checks work with NAT
#

# net/xfrm/xfrm_policy.c
#   2004/02/16 00:54:24+01:00 [email protected] +2 -0
#   Make policy checks work with NAT
# 
# net/ipv4/udp.c
#   2004/02/16 00:54:24+01:00 [email protected] +2 -0
#   Make policy checks work with NAT
# 
# net/ipv4/tcp_ipv4.c
#   2004/02/16 00:54:24+01:00 [email protected] +1 -0
#   Make policy checks work with NAT
# 
# net/ipv4/raw.c
#   2004/02/16 00:54:24+01:00 [email protected] +1 -0
#   Make policy checks work with NAT
# 
# net/ipv4/ip_input.c
#   2004/02/16 00:54:23+01:00 [email protected] +6 -8
#   Make policy checks work with NAT
# 
# net/core/netfilter.c
#   2004/02/16 00:54:23+01:00 [email protected] +43 -0
#   Make policy checks work with NAT
# 
# include/linux/netfilter.h
#   2004/02/16 00:54:23+01:00 [email protected] +16 -0
#   Make policy checks work with NAT
# 
diff -Nru a/include/linux/netfilter.h b/include/linux/netfilter.h
--- a/include/linux/netfilter.h	Mon Feb 16 02:13:39 2004
+++ b/include/linux/netfilter.h	Mon Feb 16 02:13:39 2004
@@ -166,5 +166,21 @@
 #define NF_HOOK(pf, hook, skb, indev, outdev, okfn) (okfn)(skb)
 #endif /*CONFIG_NETFILTER*/

+#ifdef CONFIG_XFRM
+#ifdef CONFIG_IP_NF_NAT_NEEDED
+struct flowi;
+extern void nf_nat_decode_session4(struct sk_buff *skb, struct flowi *fl); +
+static inline void
+nf_nat_decode_session(struct sk_buff *skb, struct flowi *fl, int family) +{
+ if (family == AF_INET)
+ nf_nat_decode_session4(skb, fl);
+}
+#else /* CONFIG_IP_NF_NAT_NEEDED */
+#define nf_nat_decode_session(skb,fl,family)
+#endif /* CONFIG_IP_NF_NAT_NEEDED */
+#endif /* CONFIG_XFRM */
+
#endif /*__KERNEL__*/
#endif /*__LINUX_NETFILTER_H*/

diff -Nru a/net/ipv4/ip_input.c b/net/ipv4/ip_input.c
@@ -206,10 +206,6 @@
 
 	__skb_pull(skb, ihl);
 
-	/* Free reference early: we don't need it any more, and it may
-           hold ip_conntrack module loaded indefinitely. */
-	nf_reset(skb);
-
         /* Point into the IP datagram, just past the header. */
         skb->h.raw = skb->data;
 

@@ -235,10 +231,12 @@
 			int ret;
 
 			smp_read_barrier_depends();
-			if (!ipprot->no_policy &&
-			    !xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) {
-				kfree_skb(skb);
-				goto out;
+			if (!ipprot->no_policy) {
+				if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) {
+					kfree_skb(skb);
+					goto out;
+				}
+				nf_reset(skb);
 			}
 			ret = ipprot->handler(skb);
 			if (ret < 0) {

diff -Nru a/net/ipv4/raw.c b/net/ipv4/raw.c
@@ -249,6 +249,7 @@
 		kfree_skb(skb);
 		return NET_RX_DROP;
 	}
+	nf_reset(skb);
 
 	skb_push(skb, skb->data - skb->nh.raw);
 

diff -Nru a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
@@ -1785,6 +1785,7 @@
 
 	if (!xfrm4_policy_check(sk, XFRM_POLICY_IN, skb))
 		goto discard_and_relse;
+	nf_reset(skb);
 
 	if (sk_filter(sk, skb, 0))
 		goto discard_and_relse;

diff -Nru a/net/ipv4/udp.c b/net/ipv4/udp.c
@@ -1027,6 +1027,7 @@
 		kfree_skb(skb);
 		return -1;
 	}
+	nf_reset(skb);
 
 	if (up->encap_type) {
 		/*

@@ -1192,6 +1193,7 @@
 
 	if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb))
 		goto drop;
+	nf_reset(skb);
 
 	/* No socket. Drop packet silently, if checksum is wrong */
 	if (udp_checksum_complete(skb))

diff -Nru a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
@@ -21,6 +21,7 @@
 #include <linux/workqueue.h>
 #include <linux/notifier.h>
 #include <linux/netdevice.h>
+#include <linux/netfilter.h>
 #include <net/xfrm.h>
 #include <net/ip.h>
 

@@ -908,6 +909,7 @@
 
 	if (_decode_session(skb, &fl, family) < 0)
 		return 0;
+	nf_nat_decode_session(skb, &fl, family);
 
 	/* First, check used SA against their selectors. */
 	if (skb->sp) {

Here is another patch on top of the last ones, it makes incoming packets from transport mode SAs visible to netfilter hooks and emulates normal behaviour. With this patch you can share a transport-mode SA between an entire network using MASQUERADE, but also do weird stuff like DNAT packets coming from a transport mode SA to remote hosts ;)

The patch is not exactly nice, but with it, everything should be working fine except for nested tunnels. I would really appreciate if I can get some comments for this part and the other ones so I can prepare some final patches to discuss with davem.

Regards
Patrick

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/02/18 15:42:49+01:00 [email protected]
# Emulate netfilter hooks for transport mode packets
#

# net/ipv4/xfrm4_input.c
#   2004/02/18 15:42:42+01:00 [email protected] +46 -1
#   Emulate netfilter hooks for transport mode packets
# 
diff -Nru a/net/ipv4/xfrm4_input.c b/net/ipv4/xfrm4_input.c
--- a/net/ipv4/xfrm4_input.c	Wed Feb 18 15:45:17 2004
+++ b/net/ipv4/xfrm4_input.c	Wed Feb 18 15:45:17 2004
@@ -9,10 +9,53 @@
  * 	
  */

+#include <linux/config.h>
+#include <linux/skbuff.h>
+#include <linux/netfilter.h>
+#include <linux/netfilter_ipv4.h>
#include <net/inet_ecn.h>
#include <net/ip.h>
#include <net/xfrm.h>

+#ifdef CONFIG_NETFILTER
+static inline int emulate_nf_done(struct sk_buff *skb)
+{
+ return 0;
+}
+
+static inline int emulate_nf_hooks2(struct sk_buff *skb)
+{
+ if (inet_addr_type(skb->nh.iph->daddr) != RTN_LOCAL) {
+ if (ip_route_me_harder(&skb) == 0)
+ dst_input(skb);
+ else
+ kfree_skb(skb);
+ return -1;
+ }
+
+ return NF_HOOK(PF_INET, NF_IP_LOCAL_IN, skb, skb->dev, NULL,
+ emulate_nf_done);
+}
+
+static inline int emulate_nf_hooks(struct sk_buff *skb)
+{
+ int off = skb->data - skb->nh.raw;
+
+ skb_push(skb, off);
+ skb->nh.iph->tot_len = htons(skb->len);
+ ip_send_check(skb->nh.iph);
+
+ if (NF_HOOK(PF_INET, NF_IP_PRE_ROUTING, skb, skb->dev, NULL,
+ emulate_nf_hooks2) != 0)
+ return -1;
+
+ skb_pull(skb, off);
+ return 0;
+}
+#else /* CONFIG_NETFILTER */
+static inline int emulate_nf_hooks(struct sk_buff *skb) {}
+#endif /* CONFIG_NETFILTER */
+
int xfrm4_rcv(struct sk_buff *skb)
{
return xfrm4_rcv_encap(skb, 0);
@@ -125,15 +168,17 @@
memcpy(skb->sp->x+skb->sp->len, xfrm_vec, xfrm_nr*sizeof(struct sec_decap_state)); skb->sp->len += xfrm_nr;

+ nf_reset(skb);
if (decaps) {
if (!(skb->dev->flags&IFF_LOOPBACK)) {
dst_release(skb->dst); skb->dst = NULL;
}
- nf_reset(skb);
netif_rx(skb);
return 0;
} else {
+ if (emulate_nf_hooks(skb) != 0)
+ return 0;
return -skb->nh.iph->protocol;
}

Hi Willy,

Thanks a lot! Decapsulated packets go the usual way, in fact the patch doesn't change anything for tunnel mode except that it drops the conntrack reference before packets are posted into the stack again. For transport mode packets it's a bit different, and I too am not entirely sure if the packet is always in valid state with the emulate_nf_hooks stuff, especially when NAT-Traversal is used. I'm investigating this after I fix a bug Michal and a second tester reported. Regarding hooks passed, packets SNATed in POST_ROUTING which have a matching policy afterwards won't pass the SELINUX and CONNTRACK hooks. The mangle table may also cause problems when something causes rerouting, I haven't thought about the possible effects yet. Other than that I can currently not think of more problems ..

Best regards,
Patrick

(long CC list, if anyone doesn't want to be on it anymore please say so)

This is the latest version of the netfilter+ipsec patches, I will check them in in a couple of days after some minor changes.

(Noteworthy) changes since last version:

- hook-traversal on input is symetrical to output now. This means packets will traverse PRE_ROUTING/LOCAL_IN before decryption and PRE_REROUTING/LOCAL_IN or PRE_ROUTING/FORWARD afterwards.

The encrypted packets go through the stack, including the hooks, the usual way. Packets reposted into the stack (tunnel-mode SAs) skip the hooks until we know decryption is finished. If after input-routing one of these packets has a non-local destination, we know decyption is finished. The function nf_postxfrm_nonlocal() which passes them to PRE_ROUTING is called for these packets. Packets with local-destination are handled in ip_local_deliver_finish(), if the ip-protocol handler is not marked as xfrm_prot (all ipsec-transformers), decryption is finished and packets go to nf_postxfrm_input() where they will traverse the PRE_ROUTING hook, potentially be rerouted or traverse the LOCAL_IN hook, before beeing delivered to the ip-protocol handler. Packets from transport-mode SAs are handled exactly like local-packets from tunnel-mode SAs.

The only problem with this approach I'm currently aware of is that one of the ip-protocol handlers marked with xfrm_prot (xfrm4_tunnel.c) is also responsible for dispatching packets to ipip.c; the encapsulated packets heading for a ipip-tunnel won't traverse the netfilter hooks on input.

- new match "policy" for matching the policy that was used during decapsulation (well, the used SAs, policy checks come later), and the policy that will be used for encapsulation.

I had some examples but I accidentally killed X before sending, for now I have just attached the test-script I used, if there are questions just ask.

Regards
Patrick

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/02/22 14:27:46+01:00 [email protected]
# Add new function 'nf_reset' to reset netfilter related skb-fields
#

# net/ipv6/sit.c
#   2004/02/22 14:27:39+01:00 [email protected] +2 -14
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# net/ipv6/ip6_tunnel.c
#   2004/02/22 14:27:39+01:00 [email protected] +1 -7
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# net/ipv4/ipip.c
#   2004/02/22 14:27:39+01:00 [email protected] +2 -14
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# net/ipv4/ip_input.c
#   2004/02/22 14:27:39+01:00 [email protected] +1 -5
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# net/ipv4/ip_gre.c
#   2004/02/22 14:27:39+01:00 [email protected] +2 -14
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
# include/linux/skbuff.h
#   2004/02/22 14:27:39+01:00 [email protected] +12 -3
#   Add new function 'nf_reset' to reset netfilter related skb-fields
# 
diff -Nru a/include/linux/skbuff.h b/include/linux/skbuff.h
--- a/include/linux/skbuff.h	Sun Feb 22 14:35:04 2004
+++ b/include/linux/skbuff.h	Sun Feb 22 14:35:04 2004
@@ -1204,6 +1204,14 @@
 	if (nfct)
 		atomic_inc(&nfct->master->use);
 }
+static inline void nf_reset(struct sk_buff *skb)
+{
+	nf_conntrack_put(skb->nfct);
+	skb->nfct = NULL;
+#ifdef CONFIG_NETFILTER_DEBUG
+	skb->nf_debug = 0;
+#endif
+}

#ifdef CONFIG_BRIDGE_NETFILTER
static inline void nf_bridge_put(struct nf_bridge_info *nf_bridge)
@@ -1216,9 +1224,10 @@
if (nf_bridge)
atomic_inc(&nf_bridge->use);
}
-#endif
-
-#endif
+#endif /* CONFIG_BRIDGE_NETFILTER */
+#else /* CONFIG_NETFILTER */
+static inline void nf_reset(struct sk_buff *skb) {} +#endif /* CONFIG_NETFILTER */

#endif /* __KERNEL__ */
#endif /* _LINUX_SKBUFF_H */

diff -Nru a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
@@ -643,13 +643,7 @@
 		skb->dev = tunnel->dev;
 		dst_release(skb->dst);
 		skb->dst = NULL;
-#ifdef CONFIG_NETFILTER
-		nf_conntrack_put(skb->nfct);
-		skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-		skb->nf_debug = 0;
-#endif
-#endif
+		nf_reset(skb);
 		ipgre_ecn_decapsulate(iph, skb);
 		netif_rx(skb);
 		read_unlock(&ipgre_lock);

@@ -877,13 +871,7 @@
 		}
 	}
 
-#ifdef CONFIG_NETFILTER
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-	skb->nf_debug = 0;
-#endif
-#endif
+	nf_reset(skb);
 
 	IPTUNNEL_XMIT();
 	tunnel->recursion--;

diff -Nru a/net/ipv4/ip_input.c b/net/ipv4/ip_input.c
@@ -202,17 +202,13 @@
 
 #ifdef CONFIG_NETFILTER_DEBUG
 	nf_debug_ip_local_deliver(skb);
-	skb->nf_debug = 0;
 #endif /*CONFIG_NETFILTER_DEBUG*/
 
 	__skb_pull(skb, ihl);
 
-#ifdef CONFIG_NETFILTER
 	/* Free reference early: we don't need it any more, and it may
            hold ip_conntrack module loaded indefinitely. */
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#endif /*CONFIG_NETFILTER*/
+	nf_reset(skb);
 
         /* Point into the IP datagram, just past the header. */
         skb->h.raw = skb->data;

diff -Nru a/net/ipv4/ipip.c b/net/ipv4/ipip.c
@@ -496,13 +496,7 @@
 		skb->dev = tunnel->dev;
 		dst_release(skb->dst);
 		skb->dst = NULL;
-#ifdef CONFIG_NETFILTER
-		nf_conntrack_put(skb->nfct);
-		skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-		skb->nf_debug = 0;
-#endif
-#endif
+		nf_reset(skb);
 		ipip_ecn_decapsulate(iph, skb);
 		netif_rx(skb);
 		read_unlock(&ipip_lock);

@@ -647,13 +641,7 @@
 	if ((iph->ttl = tiph->ttl) == 0)
 		iph->ttl	=	old_iph->ttl;
 
-#ifdef CONFIG_NETFILTER
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-	skb->nf_debug = 0;
-#endif
-#endif
+	nf_reset(skb);
 
 	IPTUNNEL_XMIT();
 	tunnel->recursion--;

diff -Nru a/net/ipv6/ip6_tunnel.c b/net/ipv6/ip6_tunnel.c
@@ -715,13 +715,7 @@
 	ipv6h->nexthdr = proto;
 	ipv6_addr_copy(&ipv6h->saddr, &fl.fl6_src);
 	ipv6_addr_copy(&ipv6h->daddr, &fl.fl6_dst);
-#ifdef CONFIG_NETFILTER
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-	skb->nf_debug = 0;
-#endif
-#endif
+	nf_reset(skb);
 	pkt_len = skb->len;
 	err = NF_HOOK(PF_INET6, NF_IP6_LOCAL_OUT, skb, NULL, 
 		      skb->dst->dev, dst_output);

diff -Nru a/net/ipv6/sit.c b/net/ipv6/sit.c
@@ -388,13 +388,7 @@
 		skb->dev = tunnel->dev;
 		dst_release(skb->dst);
 		skb->dst = NULL;
-#ifdef CONFIG_NETFILTER
-		nf_conntrack_put(skb->nfct);
-		skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-		skb->nf_debug = 0;
-#endif
-#endif
+		nf_reset(skb);
 		ipip6_ecn_decapsulate(iph, skb);
 		netif_rx(skb);
 		read_unlock(&ipip6_lock);

@@ -580,13 +574,7 @@
 	if ((iph->ttl = tiph->ttl) == 0)
 		iph->ttl	=	iph6->hop_limit;
 
-#ifdef CONFIG_NETFILTER
-	nf_conntrack_put(skb->nfct);
-	skb->nfct = NULL;
-#ifdef CONFIG_NETFILTER_DEBUG
-	skb->nf_debug = 0;
-#endif
-#endif
+	nf_reset(skb);
 
 	IPTUNNEL_XMIT();
 	tunnel->recursion--;

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/02/22 14:29:05+01:00 [email protected]
# Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
#

# net/ipv4/xfrm4_tunnel.c
#   2004/02/22 14:28:58+01:00 [email protected] +1 -0
#   Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
# 
# net/ipv4/ipcomp.c
#   2004/02/22 14:28:58+01:00 [email protected] +1 -0
#   Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
# 
# net/ipv4/ip_output.c
#   2004/02/22 14:28:58+01:00 [email protected] +20 -4
#   Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
# 
# net/ipv4/ip_forward.c
#   2004/02/22 14:28:58+01:00 [email protected] +2 -1
#   Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
# 
# net/ipv4/esp4.c
#   2004/02/22 14:28:58+01:00 [email protected] +1 -0
#   Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
# 
# net/ipv4/ah4.c
#   2004/02/22 14:28:58+01:00 [email protected] +1 -0
#   Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
# 
# include/net/ip.h
#   2004/02/22 14:28:58+01:00 [email protected] +1 -0
#   Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
# 
# include/linux/netfilter.h
#   2004/02/22 14:28:58+01:00 [email protected] +9 -4
#   Pass packets to POST_ROUTING hook before encryption and LOCAL_OUT afterwards
# 
diff -Nru a/include/linux/netfilter.h b/include/linux/netfilter.h
--- a/include/linux/netfilter.h	Sun Feb 22 14:35:13 2004
+++ b/include/linux/netfilter.h	Sun Feb 22 14:35:13 2004
@@ -119,12 +119,14 @@
 /* This is gross, but inline doesn't cut it for avoiding the function
    call in fast path: gcc doesn't inline (needs value tracking?). --RR */
 #ifdef CONFIG_NETFILTER_DEBUG
-#define NF_HOOK(pf, hook, skb, indev, outdev, okfn)			\
- nf_hook_slow((pf), (hook), (skb), (indev), (outdev), (okfn), INT_MIN)
+#define NF_HOOK_COND(pf, hook, skb, indev, outdev, okfn, cond)		\
+(!(cond)									\
+ ? (okfn)(skb) 								\
+ : nf_hook_slow((pf), (hook), (skb), (indev), (outdev), (okfn), INT_MIN))
 #define NF_HOOK_THRESH nf_hook_slow
 #else
-#define NF_HOOK(pf, hook, skb, indev, outdev, okfn)			\
-(list_empty(&nf_hooks[(pf)][(hook)])					\
+#define NF_HOOK_COND(pf, hook, skb, indev, outdev, okfn, cond)		\
+(!(cond) || list_empty(&nf_hooks[(pf)][(hook)])				\
  ? (okfn)(skb)								\
  : nf_hook_slow((pf), (hook), (skb), (indev), (outdev), (okfn), INT_MIN))
 #define NF_HOOK_THRESH(pf, hook, skb, indev, outdev, okfn, thresh)	\
@@ -132,6 +134,8 @@
  ? (okfn)(skb)								\
  : nf_hook_slow((pf), (hook), (skb), (indev), (outdev), (okfn), (thresh)))
 #endif
+#define NF_HOOK(pf, hook, skb, indev, outdev, okfn)			\
+ NF_HOOK_COND((pf), (hook), (skb), (indev), (outdev), (okfn), 1)

int nf_hook_slow(int pf, unsigned int hook, struct sk_buff *skb,
struct net_device *indev, struct net_device *outdev,
@@ -164,6 +168,7 @@

#else /* !CONFIG_NETFILTER */
#define NF_HOOK(pf, hook, skb, indev, outdev, okfn) (okfn)(skb)
+#define NF_HOOK_COND NF_HOOK
#endif /*CONFIG_NETFILTER*/

#endif /*__KERNEL__*/

diff -Nru a/include/net/ip.h b/include/net/ip.h
@@ -48,6 +48,7 @@
 #define IPSKB_TRANSLATED	2
 #define IPSKB_FORWARDED		4
 #define IPSKB_XFRM_TUNNEL_SIZE	8
+#define IPSKB_XFRM_TRANSFORMED	16
 };
 
 struct ipcm_cookie

diff -Nru a/net/ipv4/ah4.c b/net/ipv4/ah4.c
@@ -145,6 +145,7 @@
 		err = -EHOSTUNREACH;
 		goto error_nolock;
 	}
+	IPCB(skb)->flags |= IPSKB_XFRM_TRANSFORMED;
 	return NET_XMIT_BYPASS;
 
 error:

diff -Nru a/net/ipv4/esp4.c b/net/ipv4/esp4.c
@@ -199,6 +199,7 @@
 		err = -EHOSTUNREACH;
 		goto error_nolock;
 	}
+	IPCB(skb)->flags |= IPSKB_XFRM_TRANSFORMED;
 	return NET_XMIT_BYPASS;
 
 error:

diff -Nru a/net/ipv4/ip_forward.c b/net/ipv4/ip_forward.c
@@ -51,7 +51,8 @@
 	if (unlikely(opt->optlen))
 		ip_forward_options(skb);
 
-	return dst_output(skb);
+	return NF_HOOK_COND(PF_INET, NF_IP_POST_ROUTING, skb, NULL,
+	                    skb->dst->dev, dst_output, skb->dst->xfrm != NULL);
 }
 
 int ip_forward(struct sk_buff *skb)

diff -Nru a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
@@ -122,6 +122,12 @@
 	return ttl;
 }
 
+static inline int ip_dst_output(struct sk_buff *skb)
+{
+	return NF_HOOK_COND(PF_INET, NF_IP_POST_ROUTING, skb, NULL,
+	                    skb->dst->dev, dst_output, skb->dst->xfrm != NULL);
+}
+
 /* 
  *		Add an ip header to a skbuff and send it out.
  *

@@ -164,7 +170,7 @@
 
 	/* Send it out. */
 	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
-		       dst_output);
+		       ip_dst_output);
 }
 
 static inline int ip_finish_output2(struct sk_buff *skb)

@@ -282,7 +288,7 @@
 		return ip_finish_output(skb);
 }
 
-int ip_output(struct sk_buff *skb)
+static inline int ip_output2(struct sk_buff *skb)
 {
 	IP_INC_STATS(IpOutRequests);
 

@@ -293,6 +299,16 @@
 		return ip_finish_output(skb);
 }
 
+int ip_output(struct sk_buff *skb)
+{
+	int transformed = IPCB(skb)->flags & IPSKB_XFRM_TRANSFORMED;
+
+	if (transformed)
+		nf_reset(skb);
+	return NF_HOOK_COND(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
+	                    skb->dst->dev, ip_output2, transformed);
+}
+
 int ip_queue_xmit(struct sk_buff *skb, int ipfragok)
 {
 	struct sock *sk = skb->sk;

@@ -386,7 +402,7 @@
 	skb->priority = sk->sk_priority;
 
 	return NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
-		       dst_output);
+		       ip_dst_output);
 
 no_route:
 	IP_INC_STATS(IpOutNoRoutes);

@@ -1165,7 +1181,7 @@
 
 	/* Netfilter gets whole the not fragmented skb. */
 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, 
-		      skb->dst->dev, dst_output);
+		      skb->dst->dev, ip_dst_output);
 	if (err) {
 		if (err > 0)
 			err = inet->recverr ? net_xmit_errno(err) : 0;

diff -Nru a/net/ipv4/ipcomp.c b/net/ipv4/ipcomp.c
@@ -231,6 +231,7 @@
 		err = -EHOSTUNREACH;
 		goto error_nolock;
 	}
+	IPCB(skb)->flags |= IPSKB_XFRM_TRANSFORMED;
 	err = NET_XMIT_BYPASS;
 
 out_exit:

diff -Nru a/net/ipv4/xfrm4_tunnel.c b/net/ipv4/xfrm4_tunnel.c
@@ -76,6 +76,7 @@
 		err = -EHOSTUNREACH;
 		goto error_nolock;
 	}
+	IPCB(skb)->flags |= IPSKB_XFRM_TRANSFORMED;
 	return NET_XMIT_BYPASS;
 
 error_nolock:

===== include/linux/netfilter_ipv4.h 1.6 vs edited =====

@@ -83,6 +83,21 @@
    Returns true or false. */
 extern int skb_ip_make_writable(struct sk_buff **pskb,
 				unsigned int writable_len);
+
+#ifdef CONFIG_XFRM
+extern int nf_postxfrm_input(struct sk_buff *skb);
+extern int nf_postxfrm_nonlocal(struct sk_buff *skb);
+#else /* CONFIG_XFRM */
+static inline int nf_postxfrm_input(struct sk_buff *skb)
+{
+	return 0;
+}
+
+static inline int nf_postxfrm_nonlocal(struct sk_buff *skb)
+{
+	return 0;
+}
+#endif /* CONFIG_XFRM */
 #endif /*__KERNEL__*/
 
 #endif /*__LINUX_IP_NETFILTER_H*/

===== include/net/protocol.h 1.10 vs edited =====

@@ -39,6 +39,7 @@
 	int			(*handler)(struct sk_buff *skb);
 	void			(*err_handler)(struct sk_buff *skb, u32 info);
 	int			no_policy;
+	int			xfrm_prot;
 };
 
 #if defined(CONFIG_IPV6) || defined (CONFIG_IPV6_MODULE)

===== net/core/netfilter.c 1.26 vs edited =====

===== net/ipv4/ah4.c 1.30 vs edited =====

@@ -344,6 +344,7 @@
 	.handler	=	xfrm4_rcv,
 	.err_handler	=	ah4_err,
 	.no_policy	=	1,
+	.xfrm_prot	=	1,
 };
 
 static int __init ah4_init(void)

===== net/ipv4/esp4.c 1.36 vs edited =====

@@ -575,6 +575,7 @@
 	.handler	=	xfrm4_rcv,
 	.err_handler	=	esp4_err,
 	.no_policy	=	1,
+	.xfrm_prot	=	1,
 };
 
 static int __init esp4_init(void)

===== net/ipv4/ip_input.c 1.21 vs edited =====

@@ -224,6 +224,12 @@
 	resubmit:
 		hash = protocol & (MAX_INET_PROTOS - 1);
 		raw_sk = sk_head(&raw_v4_htable[hash]);
+		ipprot = inet_protos[hash];
+		smp_read_barrier_depends();
+
+		if (skb->sp && !ipprot->xfrm_prot)
+			if (nf_postxfrm_input(skb))
+				goto out;
 
 		/* If there maybe a raw socket we must check - if not we
 		 * don't care less

@@ -231,10 +237,9 @@
 		if (raw_sk)
 			raw_v4_input(skb, skb->nh.iph, hash);
 
-		if ((ipprot = inet_protos[hash]) != NULL) {
+		if (ipprot != NULL) {
 			int ret;
 
-			smp_read_barrier_depends();
 			if (!ipprot->no_policy &&
 			    !xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) {
 				kfree_skb(skb);

@@ -279,8 +284,8 @@
 			return 0;
 	}
 
-	return NF_HOOK(PF_INET, NF_IP_LOCAL_IN, skb, skb->dev, NULL,
-		       ip_local_deliver_finish);
+	return NF_HOOK_COND(PF_INET, NF_IP_LOCAL_IN, skb, skb->dev, NULL,
+	                    ip_local_deliver_finish, !skb->sp);
 }
 
 static inline int ip_rcv_finish(struct sk_buff *skb)

@@ -346,6 +351,10 @@
 		}
 	}
 
+	if (skb->sp && !(((struct rtable *)skb->dst)->rt_flags&RTCF_LOCAL))
+		if (nf_postxfrm_nonlocal(skb))
+			goto drop;
+
 	return dst_input(skb);
 
 inhdr_error:

@@ -418,8 +427,8 @@
 		}
 	}
 
-	return NF_HOOK(PF_INET, NF_IP_PRE_ROUTING, skb, dev, NULL,
-		       ip_rcv_finish);
+	return NF_HOOK_COND(PF_INET, NF_IP_PRE_ROUTING, skb, dev, NULL,
+	                    ip_rcv_finish, !skb->sp);
 
 inhdr_error:
 	IP_INC_STATS_BH(IpInHdrErrors);

===== net/ipv4/ipcomp.c 1.19 vs edited =====

@@ -408,6 +408,7 @@
 	.handler	=	xfrm4_rcv,
 	.err_handler	=	ipcomp4_err,
 	.no_policy	=	1,
+	.xfrm_prot	=	1,
 };
 
 static int __init ipcomp4_init(void)

===== net/ipv4/xfrm4_tunnel.c 1.10 vs edited =====

@@ -171,6 +171,7 @@
 	.handler	=	ipip_rcv,
 	.err_handler	=	ipip_err,
 	.no_policy	=	1,
+	.xfrm_prot	=	1,
 };
 
 static int __init ipip_init(void)

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet

#   2004/02/22 14:31:33+01:00 [email protected] 
#   Add policy lookups to ip_route_me_harder, make NAT reroute for any change
#   in route/policy key
# 
# net/ipv4/netfilter/ip_nat_standalone.c
#   2004/02/22 14:31:26+01:00 [email protected] +72 -8
#   Add policy lookups to ip_route_me_harder, make NAT reroute for any change
#   in route/policy key
# 
# net/ipv4/netfilter/ip_conntrack_standalone.c
#   2004/02/22 14:31:26+01:00 [email protected] +1 -0
#   Add policy lookups to ip_route_me_harder, make NAT reroute for any change
#   in route/policy key
# 
# net/core/netfilter.c
#   2004/02/22 14:31:26+01:00 [email protected] +15 -0
#   Add policy lookups to ip_route_me_harder, make NAT reroute for any change
#   in route/policy key
# 
diff -Nru a/net/core/netfilter.c b/net/core/netfilter.c
--- a/net/core/netfilter.c	Sun Feb 22 14:35:30 2004
+++ b/net/core/netfilter.c	Sun Feb 22 14:35:30 2004
@@ -27,6 +27,7 @@
 #include <net/sock.h>
 #include <net/route.h>
 #include <net/xfrm.h>
+#include <net/ip.h>
 #include <linux/ip.h>

#define __KERNEL_SYSCALLS__
@@ -663,6 +664,20 @@

if ((*pskb)->dst->error)
return -1;
+
+#ifdef CONFIG_XFRM
+ if (!(IPCB(*pskb)->flags & IPSKB_XFRM_TRANSFORMED)) {
+ struct xfrm_policy_afinfo *afinfo;
+
+ afinfo = xfrm_policy_get_afinfo(AF_INET);
+ if (afinfo != NULL) {
+ afinfo->decode_session(*pskb, &fl);
+ xfrm_policy_put_afinfo(afinfo);
+ if (xfrm_lookup(&(*pskb)->dst, &fl, (*pskb)->sk, 0) != 0)
+ return -1;
+ }
+ }
+#endif

/* Change in oif may mean change in hh_len. */ hh_len = (*pskb)->dst->dev->hard_header_len;

diff -Nru a/net/ipv4/netfilter/ip_conntrack_standalone.c b/net/ipv4/netfilter/ip_conntrack_standalone.c
@@ -489,6 +489,7 @@
 EXPORT_SYMBOL(ip_conntrack_alter_reply);
 EXPORT_SYMBOL(ip_conntrack_destroyed);
 EXPORT_SYMBOL(ip_conntrack_get);
+EXPORT_SYMBOL(__ip_conntrack_confirm);
 EXPORT_SYMBOL(need_ip_conntrack);
 EXPORT_SYMBOL(ip_conntrack_helper_register);
 EXPORT_SYMBOL(ip_conntrack_helper_unregister);

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/02/22 14:33:14+01:00 [email protected]
# Make policy checks find correct policy after NAT
#

# net/xfrm/xfrm_policy.c
#   2004/02/22 14:33:07+01:00 [email protected] +2 -0
#   Make policy checks find correct policy after NAT
# 
# net/ipv4/udp.c
#   2004/02/22 14:33:07+01:00 [email protected] +2 -0
#   Make policy checks find correct policy after NAT
# 
# net/ipv4/tcp_ipv4.c
#   2004/02/22 14:33:07+01:00 [email protected] +1 -0
#   Make policy checks find correct policy after NAT
# 
# net/ipv4/raw.c
#   2004/02/22 14:33:07+01:00 [email protected] +1 -0
#   Make policy checks find correct policy after NAT
# 
# net/ipv4/ip_input.c
#   2004/02/22 14:33:07+01:00 [email protected] +6 -8
#   Make policy checks find correct policy after NAT
# 
# net/core/netfilter.c
#   2004/02/22 14:33:07+01:00 [email protected] +43 -0
#   Make policy checks find correct policy after NAT
# 
# include/linux/netfilter.h
#   2004/02/22 14:33:07+01:00 [email protected] +16 -0
#   Make policy checks find correct policy after NAT
# 
diff -Nru a/include/linux/netfilter.h b/include/linux/netfilter.h
--- a/include/linux/netfilter.h	Sun Feb 22 14:35:36 2004
+++ b/include/linux/netfilter.h	Sun Feb 22 14:35:36 2004
@@ -171,5 +171,21 @@
 #define NF_HOOK_COND NF_HOOK
 #endif /*CONFIG_NETFILTER*/

+#ifdef CONFIG_XFRM
+#ifdef CONFIG_IP_NF_NAT_NEEDED
+struct flowi;
+extern void nf_nat_decode_session4(struct sk_buff *skb, struct flowi *fl); +
+static inline void
+nf_nat_decode_session(struct sk_buff *skb, struct flowi *fl, int family) +{
+ if (family == AF_INET)
+ nf_nat_decode_session4(skb, fl);
+}
+#else /* CONFIG_IP_NF_NAT_NEEDED */
+#define nf_nat_decode_session(skb,fl,family)
+#endif /* CONFIG_IP_NF_NAT_NEEDED */
+#endif /* CONFIG_XFRM */
+
#endif /*__KERNEL__*/
#endif /*__LINUX_NETFILTER_H*/

diff -Nru a/net/ipv4/ip_input.c b/net/ipv4/ip_input.c
@@ -206,10 +206,6 @@
 
 	__skb_pull(skb, ihl);
 
-	/* Free reference early: we don't need it any more, and it may
-           hold ip_conntrack module loaded indefinitely. */
-	nf_reset(skb);
-
         /* Point into the IP datagram, just past the header. */
         skb->h.raw = skb->data;
 

@@ -240,10 +236,12 @@
 		if (ipprot != NULL) {
 			int ret;
 
-			if (!ipprot->no_policy &&
-			    !xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) {
-				kfree_skb(skb);
-				goto out;
+			if (!ipprot->no_policy) {
+				if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) {
+					kfree_skb(skb);
+					goto out;
+				}
+				nf_reset(skb);
 			}
 			ret = ipprot->handler(skb);
 			if (ret < 0) {

diff -Nru a/net/ipv4/raw.c b/net/ipv4/raw.c
@@ -249,6 +249,7 @@
 		kfree_skb(skb);
 		return NET_RX_DROP;
 	}
+	nf_reset(skb);
 
 	skb_push(skb, skb->data - skb->nh.raw);
 

diff -Nru a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
@@ -1785,6 +1785,7 @@
 
 	if (!xfrm4_policy_check(sk, XFRM_POLICY_IN, skb))
 		goto discard_and_relse;
+	nf_reset(skb);
 
 	if (sk_filter(sk, skb, 0))
 		goto discard_and_relse;

diff -Nru a/net/ipv4/udp.c b/net/ipv4/udp.c
@@ -1027,6 +1027,7 @@
 		kfree_skb(skb);
 		return -1;
 	}
+	nf_reset(skb);
 
 	if (up->encap_type) {
 		/*

@@ -1192,6 +1193,7 @@
 
 	if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb))
 		goto drop;
+	nf_reset(skb);
 
 	/* No socket. Drop packet silently, if checksum is wrong */
 	if (udp_checksum_complete(skb))

diff -Nru a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
@@ -21,6 +21,7 @@
 #include <linux/workqueue.h>
 #include <linux/notifier.h>
 #include <linux/netdevice.h>
+#include <linux/netfilter.h>
 #include <net/xfrm.h>
 #include <net/ip.h>
 

@@ -908,6 +909,7 @@
 
 	if (_decode_session(skb, &fl, family) < 0)
 		return 0;
+	nf_nat_decode_session(skb, &fl, family);
 
 	/* First, check used SA against their selectors. */
 	if (skb->sp) {

# This is a BitKeeper generated diff -Nru style patch. #
# ChangeSet
# 2004/03/02 06:08:55+01:00 [email protected]
# Add policy match
#

# net/ipv4/netfilter/ipt_policy.c
#   2004/03/02 06:08:46+01:00 [email protected] +176 -0
# 
# include/linux/netfilter_ipv4/ipt_policy.h
#   2004/03/02 06:08:46+01:00 [email protected] +52 -0
# 
# net/ipv4/netfilter/ipt_policy.c
#   2004/03/02 06:08:46+01:00 [email protected] +0 -0
#   BitKeeper file /home/kaber/src/nf/ipsec/linux-2.6/net/ipv4/netfilter/ipt_policy.c
# 
# net/ipv4/netfilter/Makefile
#   2004/03/02 06:08:46+01:00 [email protected] +1 -0
#   Add policy match
# 
# net/ipv4/netfilter/Kconfig
#   2004/03/02 06:08:46+01:00 [email protected] +10 -0
#   Add policy match
# 
# include/linux/netfilter_ipv4/ipt_policy.h
#   2004/03/02 06:08:46+01:00 [email protected] +0 -0
#   BitKeeper file /home/kaber/src/nf/ipsec/linux-2.6/include/linux/netfilter_ipv4/ipt_policy.h
# 
diff -Nru a/include/linux/netfilter_ipv4/ipt_policy.h b/include/linux/netfilter_ipv4/ipt_policy.h
--- /dev/null	Wed Dec 31 16:00:00 1969
+++ b/include/linux/netfilter_ipv4/ipt_policy.h	Tue Mar  2 06:53:14 2004
@@ -0,0 +1,52 @@
+#ifndef _IPT_POLICY_H
+#define _IPT_POLICY_H
+
+#define POLICY_MAX_ELEM	4
+
+enum ipt_policy_flags
+{
+	POLICY_MATCH_IN		= 0x1,
+	POLICY_MATCH_OUT	= 0x2,
+	POLICY_MATCH_NONE	= 0x4,
+	POLICY_MATCH_STRICT	= 0x8,
+};
+
+enum ipt_policy_modes
+{
+	POLICY_MODE_TRANSPORT,
+	POLICY_MODE_TUNNEL
+};
+
+struct ipt_policy_spec
+{
+	u_int8_t	saddr:1,
+			daddr:1,
+			proto:1,
+			mode:1,
+			spi:1,
+			reqid:1;
+};
+
+struct ipt_policy_elem
+{
+	u_int32_t	saddr;
+	u_int32_t	smask;
+	u_int32_t	daddr;
+	u_int32_t	dmask;
+	u_int32_t	spi;
+	u_int32_t	reqid;
+	u_int8_t	proto;
+	u_int8_t	mode;
+
+	struct ipt_policy_spec	match;
+	struct ipt_policy_spec	invert;
+};
+
+struct ipt_policy_info
+{
+	struct ipt_policy_elem pol[POLICY_MAX_ELEM];
+	u_int16_t flags;
+	u_int16_t len;
+};
+
+#endif /* _IPT_POLICY_H */
diff -Nru a/net/ipv4/netfilter/Kconfig b/net/ipv4/netfilter/Kconfig
--- a/net/ipv4/netfilter/Kconfig	Tue Mar  2 06:53:14 2004
+++ b/net/ipv4/netfilter/Kconfig	Tue Mar  2 06:53:14 2004
@@ -127,6 +127,16 @@

To compile it as a module, choose M here. If unsure, say N.

+config IP_NF_MATCH_POLICY
+ tristate "IPsec policy match support"
+ depends on IP_NF_IPTABLES && XFRM
+ help

+	  Policy matching allows you to match packets based on the
+	  IPsec policy that was used during decapsulation/will
+	  be used during encapsulation.

+
+ To compile it as a module, choose M here. If unsure, say N.
+
config IP_NF_MATCH_MARK
tristate "netfilter MARK match support"
depends on IP_NF_IPTABLES

diff -Nru a/net/ipv4/netfilter/Makefile b/net/ipv4/netfilter/Makefile
@@ -65,6 +65,7 @@
 obj-$(CONFIG_IP_NF_MATCH_TCPMSS) += ipt_tcpmss.o
 
 obj-$(CONFIG_IP_NF_MATCH_PHYSDEV) += ipt_physdev.o
+obj-$(CONFIG_IP_NF_MATCH_POLICY) += ipt_policy.o
 
 # targets
 obj-$(CONFIG_IP_NF_TARGET_REJECT) += ipt_REJECT.o

diff -urN a/extensions/.policy-test b/extensions/.policy-test
@@ -1,2 +1,3 @@
 #!/bin/sh
-[ -f $KERNEL_DIR/net/ipv4/netfilter/ipt_policy.c -a -f $KERNEL_DIR/include/linux/netfilter_ipv4/ipt_policy.h ] && echo policy
+#
+[ -f $KERNEL_DIR/include/linux/netfilter_ipv4/ipt_policy.h ] && echo policy

diff -urN a/include/iptables.h b/include/iptables.h
@@ -122,6 +122,7 @@
 extern void register_match(struct iptables_match *me);
 extern void register_target(struct iptables_target *me);
 
+extern char *proto_to_name(u_int8_t proto, int nolookup);
 extern struct in_addr *dotted_to_addr(const char *dotted);
 extern char *addr_to_dotted(const struct in_addr *addrp);
 extern char *addr_to_anyname(const struct in_addr *addr);

diff -urN a/iptables.c b/iptables.c
@@ -235,11 +235,12 @@
 	{ "icmp", IPPROTO_ICMP },
 	{ "esp", IPPROTO_ESP },
 	{ "ah", IPPROTO_AH },
+	{ "ipcomp", IPPROTO_COMP },
 	{ "sctp", IPPROTO_SCTP },
 	{ "all", 0 },
 };
 
-static char *
+char *
 proto_to_name(u_int8_t proto, int nolookup)
 {
 	unsigned int i;

#! /bin/bash

PATH="/usr/local/sbin"

iptables -F

iptables -A INPUT -m policy --dir in --pol none
iptables -A INPUT -m policy --dir in --pol ipsec
iptables -A INPUT -m policy --dir in --pol ipsec --mode tunnel
iptables -A INPUT -m policy --dir in --pol ipsec --mode transport
iptables -A INPUT -m policy --dir in --pol ipsec --proto ah
iptables -A INPUT -m policy --dir in --pol ipsec --proto esp
iptables -A INPUT -m policy --dir in --pol ipsec --strict --mode tunnel
iptables -A INPUT -m policy --dir in --pol ipsec --strict --mode transport
iptables -A INPUT -m policy --dir in --pol ipsec --strict --proto ah
iptables -A INPUT -m policy --dir in --pol ipsec --strict --proto esp
iptables -A INPUT -m policy --dir in --pol ipsec --strict --proto esp --next --proto ah
iptables -A INPUT -m policy --dir in --pol ipsec --strict --proto esp --mode tunnel --tunnel-src 192.168.0.1 --next --proto ah --mode transport iptables -A INPUT -m policy --dir in --pol ipsec --strict --proto esp --mode tunnel --tunnel-src 192.168.0.1 --tunnel-dst 192.168.0.23 --next --proto ah --mode transport

iptables -A OUTPUT -m policy --dir out --pol none
iptables -A OUTPUT -m policy --dir out --pol ipsec
iptables -A OUTPUT -m policy --dir out --pol ipsec --mode tunnel
iptables -A OUTPUT -m policy --dir out --pol ipsec --mode transport
iptables -A OUTPUT -m policy --dir out --pol ipsec --proto ah
iptables -A OUTPUT -m policy --dir out --pol ipsec --proto esp
iptables -A OUTPUT -m policy --dir out --pol ipsec --strict --mode tunnel
iptables -A OUTPUT -m policy --dir out --pol ipsec --strict --mode transport
iptables -A OUTPUT -m policy --dir out --pol ipsec --strict --proto ah
iptables -A OUTPUT -m policy --dir out --pol ipsec --strict --proto esp
iptables -A OUTPUT -m policy --dir out --pol ipsec --strict --proto esp --next --proto ah
iptables -A OUTPUT -m policy --dir out --pol ipsec --strict --proto esp --mode tunnel --tunnel-dst 192.168.0.0/24 --next --proto ah iptables -A OUTPUT -m policy --dir out --pol ipsec --strict --proto esp --mode tunnel --tunnel-dst ! 192.168.0.0/24 --next --proto ah iptables -A OUTPUT -m policy --dir out --pol ipsec --strict --proto esp --mode tunnel --tunnel-dst 192.168.0.1 --next --proto ah --mode transport

Hi Patrick,

no pb, please keep me in :-)

This is excellent. So if I understand it right, a packet now passes this way (please excuse the ascii-art, it seems I'll never take the time to start xfig)

            [incoming packet]
                    |<-----------------. 
                    v                  |
               PREROUTING              |
    dest!=local?  /  \  dest==local?   |
                /      \               |
         FORWARD        LOCAL_IN       |
             |            |            |
             |      (ipsec?decrypt:nop)|
             |            |   \        / dest!=local ? nf_postxfrm_nonlocal()
             |            |    `------'
             v            v  dest==local ? ip_local_deliver_finish()

So if it behaves like this (which is what seems the best solution to me), is there a risk that a carefully crafted packet loops many times on the right side (multiple encapsulation with local DIP) ? And if so, is it risky or not, wrt resource leak, multiple locks, etc... ?

I don't know well the internals of IPSEC (bought the book but didn't read it yet). Are IPIP tunnels often used ? or are they simply used according to a negociation, in which case the ipsec implementation on the netfilter side could refuse them ?

Even more interesting... Now we can for sure check that packets coming from a tunnel have not been spoofed by another tunnel.

Thanks a lot Patrick.
My main goal is to use this on 2.4 with davem/herbert xu's ipsec backport. I will soon check if your patches apply on top of their implementation.

Cheers,
Willy

Willy thanks for the diagram, too lazy to do one myself.

Q do I understand right that encrypted packets can or can't be acted upon by the hooks in LOCAL_IN.

Or another way of putting it does a packet travel the tables twice once as an encrypted packet and once as a de crypted packet ?

Alex

Exactly, input looks like this:

(encrypted) PRE_ROUTING -> LOCAL_IN ->
(plain) PRE_ROUTING -> LOCAL_IN/FORWARD

output looks like this:

(plain) FORWARD/LOCAL_OUT -> POST_ROUTING ->
(encrypted) LOCAL_OUT -> POST_ROUTING

This is the same as with freeswan, only without the ipsec devices, the policy match can be used as a easy replacement for them (-m policy --pol ipsec).

Regards,
Patrick

Great, i also presume this means that NAT + IPSEC is now working, will give it a try tonight.

Thanks

Patrick

I seem to have found a bug in your patches, but only when used in conjuction with Herbert's mangle patch.

It seems like there is a loop caused when the packet traverses the tablesi, in particular ip_route_me_harder.

I tested this on my laptop with debian 2.6.3-2 source with these patches that you provided on this thread, as well as the Herbert mangle patch.

It seem like the packet on the way out gets encapsulated and then the encrypted packets try to get re encrypted.

example ipsec.conf

conn wireless

	left=10.0.4.129
	leftsubnet=0/0
	authby=secret
	pfs=no
	auto=add
	right=%defaultroute

By the dump it looks like a loop, I added a printk("%d\n", iph->protocol); in ip_route_me_harder just before Herberts fix to test that.

When I changed the config to look like this

conn wireless

	left=10.0.4.129
	leftsubnet=10.6.0/24
	authby=secret
	pfs=no
	auto=add
	right=%defaultroute

It worked fine

Any other question ask, I have deb's of the image and headers too if you want.

Alex

Thanks for the report, for now the easiest solution is to back out Herbert's patch.

Regards
Patrick

I have done that in my local build, but I beleive it is in 2.6.4 (the changelog)

Hi Willy,

Not exactly. the ipsec transformers are ip protocol handlers, their input handler (xfrm4_input) is called from ip_local_deliver_finish. After decrypting the payload the packets are either reposted into the network stack (in case of a tunnel-mode SA) or again delivered to the ip-protocol contained in the next header. packets reposted into the stack may be in some intermediate state in case of a transport mode encapsulation following a tunnel mode encapsulation. The difficulty is to know when the packet is "done". So nf_postxfrm_nonlocal is called directly after input routing if the destination is non-local, but only for packets which were reposted into the stack. The fact that they have a non-local destination means ipsec must be done with them. For packet with local destination, we look at what the ipprotocol handler does in ip_local_deliver_finish (xfrm_prot or not) and if the packet was handled by ipsec before and the protocol handler is no xfrm_prot, ipsec is done, so the packets are passed through nf_postxfrm_input.

I attached a xfig file with a little picture of how it works, I hope it's understandable. The blue lines represent the flow as it happens without my patches, the red lines represent whats new. While drawing it, I noticed a bug, packets with their destination changed in nf_postxfrm_nonlocal may need to be delivered locally afterwards.

Regarding loop-danger, I'm not totally sure yet, but I don't think that it would be triggerable by crafted packets but only by adventurous configurations. Multiple encapsulations are no error and they are limited to 4.

It's not IPIP with IPSEC that misbehaves but plaintext-IPIP tunnels. xfrm4_tunnel.c is the registered handler for IPIP, it used to be ipip.c. ipip.c now registers itself with xfrm4_tunnel, but xfrm4_tunnel is marked as a xfrm_prot which is not true for ipip.c. Right now I'm not even sure that it is a problem and under what exact circumstances, I need to think about this some more (tomorrow).

That should also be handled by the policy checks, although at a later time (at socket-delivery/forwarding). I'm unsure if the policy checks should happen before packets are looked at by netfilter, after all it manipulates it's state from information in these untrusted packets.

Thanks for your comments, I'm looking forward to your testing.

Best regards
Patrick

Hi Patrick,

Thanks for the picture, indeed it's far better now ;-)

OK. Thanks for your clarifications, I understand better now.

Cheers,
Willy

This patch (libipt_policy.diff) was broken in the last mail, fixed version attached.

diff -urN a/extensions/.policy-test b/extensions/.policy-test
@@ -0,0 +1,3 @@
+#!/bin/sh
+#
+[ -f $KERNEL_DIR/include/linux/netfilter_ipv4/ipt_policy.h ] && echo policy

diff -urN a/include/iptables.h b/include/iptables.h
@@ -122,6 +122,7 @@
 extern void register_match(struct iptables_match *me);
 extern void register_target(struct iptables_target *me);
 
+extern char *proto_to_name(u_int8_t proto, int nolookup);
 extern struct in_addr *dotted_to_addr(const char *dotted);
 extern char *addr_to_dotted(const struct in_addr *addrp);
 extern char *addr_to_anyname(const struct in_addr *addr);

diff -urN a/iptables.c b/iptables.c
@@ -235,11 +235,12 @@
 	{ "icmp", IPPROTO_ICMP },
 	{ "esp", IPPROTO_ESP },
 	{ "ah", IPPROTO_AH },
+	{ "ipcomp", IPPROTO_COMP },
 	{ "sctp", IPPROTO_SCTP },
 	{ "all", 0 },
 };
 
-static char *
+char *
 proto_to_name(u_int8_t proto, int nolookup)
 {
 	unsigned int i;

good work, patrick. I'm looking forward to seeing those changes in the mainline kernel.

one minor note:
please #ifdef CONFIG_NETFILTER the nf_reset function (or #define it to nothing otherwise)

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

ah_output to ah_output_finish and a new ah_output
with:

yes, please wait until we have implemented in one case and proven that it does what it is supposed to do.. after that we will sync up.

yes.

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

-----BEGIN PGP SIGNED MESSAGE-----

Henrik> For me it is easier to grasp what is happending with the packet Henrik> if it is viewed like the original packet is forwarded to the Henrik> ipsec engine and the ipsec engine then generates a new packet.

Yes, I agree strongly.

Henrik> Even in the Host<->Host IPSec case the tunnel view is most Henrik> appropriate. As soon as the packets enter the IPSec encapsulation Henrik> they loose all the relevant propoerties of the original Henrik> connection and becomes part of the single IPSec Host <-> IPSec Henrik> Host connection all packets between these two IPSec host is Henrik> transmitted over.

Actually, there are a number of properities that many will wish to retain.
This includes:
1) VLAN
2) QoS - DSCP, IPv6 flow label, other application set parameters
3) possibly some pieces of nfmark (it will be scenario specific)
4) ECN gets copied

It is my understanding that the stackable dst permits most of this to
be retained - but the IPsec actually needs to decide what to do.

Finally, there are some packets which will enter the IPsec system, but
emerge unchanged - they may excepted. Those packets will not be new.

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] [email protected]      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[

] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys

iQCVAwUBQBbCDYqHRg3pndX9AQGjCgP9EWNAvrsEwopZXv1aJhBAyrFbawM5bPI4 CoENcH4FFXURohmE7GDsXl3Ei8Hf7Kuru8EG1WKZCNJXYpgpsMFRXH6YVOzl497o aVdE2y279vv1JOyY7Jo/Lwj6nm5NMqkv1EMMOMK8IZm3+AgQ6Fzw/IosMpQUEAkR CrWi68BCtE0=
=usFv
-----END PGP SIGNATURE-----

Hello,

This is what happens with incoming packets: they hit INPUT twice.

You are missing the point that the netfilter code changes the packets here, thus if you would want netfilter to do something else, you could code that in netfilter rules.

Now let's review an IPsec tunnel. On the input side, everything works as
expected: an ESP packet hits PREROUTING, then INPUT, gets unencrypted, then the new packet (that has different from and to, and may be a different protocol altogether) hits PREROUTING again, then INPUT (or FORWARD).

On output, however, things are not so transparant. A packet hits OUTPUT, then gets encrypted. A totally different packet hits POSTROUTING. This makes no sense.

There's no symmetric behaviour now. And from an "input" perspective, I think a decapsulated packet that has a locally ending payload should hit INPUT again. Same with output: a locally generated packet hits output, then should hit OUTPUT again if it was really destined for the outside world.

V.

http://www.openoffice.nl/   Open Office - Linux Office Solutions
Valentijn Sessink  [email protected]

And there the required hooks needs to be inserted in the IPSec implementation to call

a) POSTROUTING before the final decision that this packet is to go via IPSec. Note that this may need to allow reroute of the packet including IPSec policy. Because of this some changes may be needed in the IPSec implementation for this to be complete.

b) After encrypting OUTPUT needs to be called on the encrypted packet before the packet leaves IPSec and continues the normal path leading to POSTROUTING.

We all know that, and is why this discussion exists in order to find out how it should be.

To this we all agree from what I can tell.

Regards
Henrik

yes, but once before and once after decryption. use either fwmark or an ipip tunnel interface to destinguish between already decrypted and was-never-encrypted packets.

use an interface, and you will see the packet twice, plus the interface does the route lookup on the encrypted packet. so no ugly hacks in the routing table are necessary.

(ugly hacks scenario: consider a road warrior.
he is 192.168.3.2 and his vpn gateway is 192.168.3.1. he wants to put the company net on that GW address. But without a GW he doesn't know on which interface to put it. it could be either wlan (if the tunnel is established with his wireless lan card), or the eth0 interface (if he is using ethernet cable to connect to the net). sure, the ipsec software could manipulate the routing table after altering local and remote tunnel endpoints, but that is an ugly hack and a bad design.)

[symetry]

that does not work:
netfilter wants NAT to be the first thing for incoming packets and the last thing for outgoing packets. thats symetry.

but you want ipsec encryption and decryption as soon as possible:
both before looking at the routing table.

mixing those two will never result in symetry. if you try it (routing before encapsulation), the result has problems.

Andreas

Cc'ing netdev, since this is now a very clear proposal and not some internal netfilter rambling.

yes, and this is desired. However, the packets also hit PREROUTING
twice, which is exactly what we want (and get it for free, since xfrm4_input.c calls netif_rx() again).

However, it doesn't reset the skb->nfct pointer, so connection tracking sees both encapsulated and decapsulated packet as belonging to the same connection. This is wrong, and dangerous (application helpers might be called for the encapsulated traffic, which now has a completely different layer 4 protocol.

So with stock 2.6.x you have

- no working connection tracking in any kind of ipsec scenario
- conntrack and NAT helpers trying to find readable application data within already-encrypted packets
- no working DNAT/SNAT, mainly because of not-working connection tracking

Did anybody propose ugly hacks in the routing table?

yes, it is symmetric. As is the symmetry between prerouting/postrouting and input/output hooks. As well as the symmetry between SNAT and DNAt,
...

Yes, that's why what IPsec is doing for incoming ESP/AH packets the right thing: go through PRE_ROUTING, route to LOCAL_IN, decapsulate, and start over again.

However, for outgoing ESP/AH packets (locally-encapsulated), it doesn't show the symmetric behaviour of what it does on the input side: go through FORWARD, encapsulate, route, POST_ROUTING. No starting over.

So we absolutely _need_ a symmetric-to-incoming behaviour like:

a) for locally-originated packets
LOCAL_OUT, POST_ROUTING, encapsulate, LOCAL_OUT, POST_ROUTING.

b) for forwarded, locally-encapsulated packets
PRE_ROUTING, FORWARD, POST_ROUTING, encapsulate, LOCAL_OUT, POST_ROUTING.

No, we don't achieve this by manipulating the routing code, but by placing the respective hooks in ah{4,6}.c and esp{4,6}.c {ah,esp}_output() function respectively. We also need to (again) reset the skb->nfct and drop the conntrack reference again.

This way we
1) make connection tracking work again
2) do not change any routing of the current implementation
3) enable users to
a) filter unencapsulated, encapsulated and decapsulated packeets at their 'expected' place
b) do DNAT on the incoming decapsulated packets (which doesn't work right now, but should)
c) do SNAT on the outgoing/forwarded to-be-encapsualated packets (which doesn't work right now, but should)

I'm not proposing to route before encapsulating. I just propose of calling the same netfilter functions that we usually call before/after routing at different places :)

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

the tunnels do.

In summary it looks to me, like you want to have everything that a real tunnel with an interface offers, but for some reason you don't want the tunnel with the interface. right?

so, why?

that is not true. ipip_rcv:

                nf_conntrack_put(skb->nfct);
                skb->nfct = NULL;
ipip_xmit:
        nf_conntrack_put(skb->nfct);
        skb->nfct = NULL;

the only problem is stability and scheduling in atomic with xfrm+ipip (i guess a bug, I hope someone who understands the code could take a look at it).

if you don't use an interface, you need them in real world setups. with an interface both gw and client put the route to each others static vpn ip on ipsec0 and then configure the tunnel ips, and those ip will be used to lookup the interface where to send the encrypted packet.

with the current code without using interfaces, the ip of the vpn tunnel needs to be moved to the interface that will be used to reach the vpn gateway. even worse, you need to play with source address selection to make sure an outgoing packet has your static vpn ip, and not the dynamic ip of whatever net you are currently using.

example:

      up ip addr add 192.168.3.1 dev wlan0
      up ip route del default dev wlan0
      up ip route add default via 192.168.1.1 src 192.168.3.1 dev wlan0
      down ip addr del 192.168.3.1 dev wlan0

and swithcing from wlan0 to eth0 does not only require the tunnel to be reconfigured (local and remote ip), but also changes in the routing table. that is horrible.

and yes, of course you want static ip addresses for "road warrior" clients! for example smtp servers need to these to allow relaying or not.

use an tunnel interface and you have that.
(adding pre_/post_ routing is very simple, simply cut&paste to the _rcv and _xmit functions.

sure, you can archive the same thing without using tunnel interfaces. But i wonder if that creates a simple solution, easy to understand.

my basic question is: how will these changes affect setups with tunnel interfaces?

[other issue]
if we are to look at the calls to xfrm anyway: would it be possible to add a debugging facility for dropped packets? I know, that is not a netfilter issue, but I guess many people will find it problematic, if they cannot see what packets are dropped by the xfrm logic. I'm only rasing the issue, because that code is most likely put to the same places we are discussiong right now (ah, esp, ipip_rcv, ipip_xmit, ...).

that is currently done. I'm glad if that "feature" is removed.

fine. I don't understand what the benefit of doing everything like with an interface, but avoiding the interface. but please keep tunnel with interface and tunnel without interface in sync.

Regards, Andreas

yes, I'm well aware of that.

Because it's needed for interoperability with other IPsec implementations, in other words if you are not running Linux on both ends. Yes, I know... in an ideal world everybody would run linux on both ends... but if you do: Why use IPsec at all, if not for interoperability.

s/ipsec scen/plain non-tunnel ipsec scen/

oh well, what you are describing are entries in the routing table, that's fine. I thought you were talking about hacking the routing code.

well, this is just like the 2.6.x ipsec implementation is like. It's not the netfilter project's will or task to change that implementation. We just want to get the same netfilter/iptables functionality, independent of somebody using ipsec or not.

yes, but as stated above, other implementations might not be able to work with that implementation.

Unfortunately there is no simple or easy solution. We just want to get it working at all.

I think you would see the packet even more often:

1) LOCAL_OUT of original packet
2) POST_ROUTING of original packet
3) LOCAL_OUT of tunnelled packet
4) POST_ROUTING of tunneled packet
5) LOCAL_OUT of ESP/AH packet
6) POST_ROUTING of ESP/AH packet

yes, but I'd rather don't put both of them into one patch.

Again, I misunderstood you. I do not intend to change the IPsec implementation in any way, I just want netfilter to acommodate it.

What do you mean by that? From my point of view, it should be like
(1-6) above. This way anybody can filter on any kind of header bit at
any layer of the encapsulation.

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

Hi Harald,

The userland could be improved to be more flexible, then it could configure the same issue with and without interface, depending on the needs. From protocol perspective that wouldn't be visible, so I realy don't see any interoperability issues here.

ip, esp, inner ip, tcp/udp, payload.
the packet can look the same, whether you use some interface to assemble it or not. ike and setting spd plus tunnel is the issue, but thats userspace.

ok, but thats not a netfilter topic.

one change you want to add is POST_ROUTING before encapsilating. add that hook at the start of ipip_xmit, too (renameing ipip_xmit to ipip_xmit_finish and a new ipip_xmit with only the hook), and things should be in sync. same for the other tunnels.

the other change you want to make is set the interface to some other value after encapsulation. what value? will it be possible to specify the value?
I'm still thinking about that and in what scenario you would need to differ say ipsec0 and ipsec1. but such scenarios could exist.

maybe one vpn/gateway/router shared by two small companies, both want to keep their network private ?

[setkey debugging]

sure. are there netfilter routines that can be reused, or are they somehow tied to the NF core? netfilter formats the packets nicely for debugging.

if you add a hook in ah/esp, add the same hooks to ipip, ipgre, ip6_tunnel etc. Currently netfilter/tunneling is more or less consistent - without interface there are some issues, but the tunnelling fits fine into the big netfilter picture (the one with 5 hooks explained).

if ah/esp used different hooks than the real tunnels, that would make the situation less easy.

also, what is the status of the new hooks?
dropped or do you still plan to add a different hook, even if it calls the same table?
POST_XFRM is a bad name, better something
like POST_ENCAP or POST_DECAP, because IMO
tunnels should call that hook, too, so it
should be a generic name, not tied to xfrm.

ah, wait. it will be only 4 steps,
3+5 and 4+6 are merged, because the tunnel calls xfrm itself, so it does both at once. that could be changed of course, if you want that flexibility.

but i wonder:
if you want to change the untunneld packet, you can do that in 1/2. if you want to change the encrypted packet, you can do that i 5/6. 3/4 has the same (as payload) you see in 1/2, and the same header that has 5/6. so what is the benefit of two additional steps?

Regards, Andreas

Hi I try compile nf-log module from CVS snap 20040128 on 2.6.2-rc2 kernel and I found undefinef BR_NETPROTO_LOCK.

Can anyone check and solve this problem ?

Thanx.
Sas.

{Wojciech 'Sas' Cieciwa}  {Member of PLD Team                               }
{e-mail: [email protected], http://www2.zarz.agh.edu.pl/~cieciwa}

Why not just do this right when we pop into the dst_output() call in ip_output.c This way we don't have to add all of this stuff for every new encapsulator we ever implement.

Maybe not like this precisely, but something like it.

I like this proposal. I assume that on input, payload packets would also have this dummy device as their input device? And on output that payload packets would have "esp" as their output device in the FORWARD and OUTPUT hooks as well?

I would also like to register my vote for having the AH/ESP packets go through INPUT and OUTPUT. This would allow Shorewall to treat IPSEC in the same way as it does all other tunnel types:

a) the payload connection uses a dedicated device ("esp" in this case)
b) the encapsulated traffic originates at the gateway on output and is directed to the gateway on input.

-Tom

Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ [email protected]

Great :)

sure, makes sense.

no, that's way more difficult. I'm not sure whether it can be done at all (without adding rediculous kludges to the code).

Mh. Let's say we stick with the INPUT/OUTPUT chains for now.

However, I would like to introduce new netfilter hooks. At least for the beginning (due to lack of better ideas), I'm going to register the INPUT/OUTPUT chains with those new hooks.

the idea of the new hooks is, that in reality they are at different locations in the stack. And at some point we might have some module that is only interested in xfrm'ed packets.

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

Bummer -- I'm trying to avoid equally ridiculous kludges in my own code :-)

I understand.

-Tom

Tom Eastep    \ Nothing is foolproof to a sufficiently talented fool
Shoreline,     \ http://shorewall.net
Washington USA  \ [email protected]

-----BEGIN PGP SIGNED MESSAGE-----

Harald> To do this, somewhen between esp_output() is called and the Harald> beginning of the modification of the packet payload, we need to Harald> call nf_hook(POST_ROUTING). This way, conntrack would be able to Harald> put the connection in the hash, and people can do SNAT-like Harald> operations in nat-> POSTROUTING. We could even pass a dummy Harald> output device structure with an interface name "esp" so people Harald> can SNAT everything heading for esp encapsulation.

I would suggest that many administrators will want to have possibly a
pseudo-interface per SA. These aren't necessarily real interfaces like ipsecX, but they do need to be named in some way.

(There are a number of scenarios where having a proper virtual interface
is really the best way to interact properly with routing daemons)

The original IPsec implementation that was done at NRL created virtual
tunnels (using the "route" command) with virtual tunnel devices - one per tunnel. They were criticised that this didn't scale - that's because at the time (1993/1994) people weren't building kernels that assumed that there could be thousands of devices.
Linux people *are* doing this, and doing it correct. The examples include:
MPLS, PPP, PPPoE, VLAN

]       ON HUMILITY: to err is human. To moo, bovine.           |  firewalls  [
]   Michael Richardson,    Xelerance Corporation, Ottawa, ON    |net architect[
] [email protected]      http://www.sandelman.ottawa.on.ca/mcr/ |device driver[

] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [ -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys

iQCVAwUBQBbBJ4qHRg3pndX9AQEaHAQA3+VNcmq/9RzhqIW/d5ecLN1bnh+/BQYf hUELO1Drczud1ne1FxUHSDlvDtwRE5EzZ3tJwSA2NH+5s2yEYnk9Ydb+JNYPiWAL F/KohhfcQxsasloCSpQ6KErazBU240pJfRYjlSwqNPsZLVC64CqyenEwAm4vxXJJ X7uzK/idyc4=
=eeVc
-----END PGP SIGNATURE-----