Skip to contentSkip to conversation

New: kernel panic when receiving an ipsec packet

Bug 42809 In review Stephen Hemminger 7 participants 21 messages Feb 22, 2012 – Feb 23, 2012 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 18 lines · On Wed, 22 Feb 2012 10:24:19 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=42809

           Summary: kernel panic when receiving an ipsec packet
           Product: Networking
           Version: 2.5
    Kernel Version: 2.6.32.54
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: IPV4
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: No
Quoted 5 lines

As soon as I receive an ipsec packet (NETKEY of course) I get a kernel panic. Even magicsysrq keys do not work anymore.

O.S. Debian Squeeze amd64. I tried both Strongswan and Openswan.

That is a really old kernel now. Could you try with later kernel?

#2 lore raw

Thanks for CCing netdev, I was thinking about doing the same.

Il 22/02/2012 18:20, Stephen Hemminger ha scritto:

Quoted 1 line

That is a really old kernel now. Could you try with later kernel?

I can (a few minutes during night) but I really need a fix for 2.6.32. This is a production server with dozens of Xen virtual machines and a heavily patched kernel (xen dom0, layer 7, imq, esfq, advanced routing etc). I did test a "vanilla" kernel last night (not really vanilla, but a clean official debian kernel) and I had the same problem.

Thanks,
Niccolò

#3 lore raw

Hi,
The bug is still present in latest 3.2.7 vanilla kernel. I wasted the whole day debugging that damn thing and I finally discovered the root cause.
The problem is with my Traverse Solos multi-port ADSL2+ PCI card[1] (which has open source drivers included in the kernel) when using RFC 2684 routed.
I have two adsl lines, the first one connected using RFC 2684 routed, the second one using PPPoA.
If I create a vpn toward the PPPoA line it works flawlessly, while if I create a vpn toward the RFC 2684 routed line the whole system hangs in a kernel panic (with both 2.6.32.54 and 3.2.7). I really don't know how to fix it and I need to setup that damn ipsec vpn :(

This is the bug on bugzilla.kernel.org:
https://bugzilla.kernel.org/show_bug.cgi?id=42809

Niccolò

[1]http://www.traverse.com.au/productview.php?product_id=116

#4 lore raw
Quoted 13 lines · Niccolò Belli

Hi,
The bug is still present in latest 3.2.7 vanilla kernel. I wasted the whole day debugging that damn thing and I finally discovered the root cause. The problem is with my Traverse Solos multi-port ADSL2+ PCI card[1] (which has open source drivers included in the kernel) when using RFC 2684 routed.
I have two adsl lines, the first one connected using RFC 2684 routed, the second one using PPPoA.
If I create a vpn toward the PPPoA line it works flawlessly, while if I create a vpn toward the RFC 2684 routed line the whole system hangs in a kernel panic (with both 2.6.32.54 and 3.2.7).
I really don't know how to fix it and I need to setup that damn ipsec vpn :(

Quoted 7 lines

This is the bug on bugzilla.kernel.org:
https://bugzilla.kernel.org/show_bug.cgi?id=42809

Niccolò

[1]http://www.traverse.com.au/productview.php?product_id=116

Which driver handles this Traverse Solos card ?

#5 lore raw
Quoted 1 line · Eric Dumazet

Which driver handles this Traverse Solos card ?

solos_pci, which also requires the atm module.

(I'm just trying to help here; I'm not affected by the bug.)

#6 lore raw
In reply to Eric Dumazet
Quoted 1 line · Eric Dumazet

Which driver handles this Traverse Solos card ?

If br2684_push() is used, it seems it lacks proper call to skb_reset_mac_header(skb) in paths where eth_type_trans() is not called.

Later in xfrm4_mode_tunnel_input() we crash because we assume skb_mac_header() is valid.

#8 lore raw
In reply to Eric Dumazet

Il 23/02/2012 02:38, Eric Dumazet ha scritto:

Quoted 1 line

Which driver handles this Traverse Solos card ?

drivers/atm/solos-pci.c

~# lsmod | grep solos

solos_pci              20009  2
atm                    32378  7 pppoatm,br2684,solos_pci

Niccolò

#21 lore raw
In reply to Eric Dumazet

Il 23/02/2012 02:38, Eric Dumazet ha scritto:

Quoted 1 line

Which driver handles this Traverse Solos card ?

drivers/atm/solos-pci.c

~# lsmod | grep solos

solos_pci              20009  2
atm                    32378  7 pppoatm,br2684,solos_pci

Niccolò

#9 lore raw
In reply to Niccolò Belli
Quoted 9 lines · Niccolò Belli

Il 23/02/2012 02:38, Eric Dumazet ha scritto:

Quoted 1 line

Which driver handles this Traverse Solos card ?

drivers/atm/solos-pci.c

~# lsmod | grep solos

solos_pci              20009  2
atm                    32378  7 pppoatm,br2684,solos_pci

Thanks !

Please try following patch.

net/ipv4/xfrm4_mode_tunnel.c +5 -3
index 534972e..f170933 100644
@@ -84,9 +84,11 @@ static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 	if (!(x->props.flags & XFRM_STATE_NOECN))
 		ipip_ecn_decapsulate(skb);
 
-	old_mac = skb_mac_header(skb);
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	if (skb_mac_header_was_set(skb)) {
+		old_mac = skb_mac_header(skb);
+		skb_set_mac_header(skb, -skb->mac_len);
+		memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	}
 	skb_reset_network_header(skb);
 	err = 0;
 
#10 lore raw

Il 23/02/2012 03:06, Eric Dumazet ha scritto:

Quoted 10 lines

Please try following patch.

net/ipv4/xfrm4_mode_tunnel.c +0 -0
index 534972e..f170933 100644
@@ -84,9 +84,11 @@ static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
  	if (!(x->props.flags&  XFRM_STATE_NOECN))
  		ipip_ecn_decapsulate(skb);
 

net/ipv4/xfrm4_mode_tunnel.c static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb) · line 93

Quoted 10 lines
net/ipv4/xfrm4_mode_tunnel.c +5 -3
-	old_mac = skb_mac_header(skb);
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	if (skb_mac_header_was_set(skb)) {
+		old_mac = skb_mac_header(skb);
+		skb_set_mac_header(skb, -skb->mac_len);
+		memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	}
  	skb_reset_network_header(skb);
  	err = 0;

Wow, thanks! It will try it as soon as possible tomorrow, going sleeping now :)

Niccolò

#11 lore raw
In reply to Eric Dumazet

Il 23/02/2012 03:06, Eric Dumazet ha scritto:

Quoted 13 lines

Thanks !

Please try following patch.

net/ipv4/xfrm4_mode_tunnel.c +0 -1
index 534972e..f170933 100644
@@ -84,9 +84,11 @@ static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
  	if (!(x->props.flags&  XFRM_STATE_NOECN))
  		ipip_ecn_decapsulate(skb);
 
-	old_mac = skb_mac_header(skb);

net/ipv4/xfrm4_mode_tunnel.c static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb) · line 94

Quoted 10 lines
net/ipv4/xfrm4_mode_tunnel.c +5 -2
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	if (skb_mac_header_was_set(skb)) {
+		old_mac = skb_mac_header(skb);
+		skb_set_mac_header(skb, -skb->mac_len);
+		memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	}
  	skb_reset_network_header(skb);
  	err = 0;
 

Your patch does solve the problem, thanks!

Niccolò

#7 lore raw

Bug#660804: [Bug 42809] New: kernel panic when receiving an ipsec packet

In reply to Eric Dumazet
Quoted 9 lines · On Thu, 23 Feb 2012 02:54:39 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 1 line · Eric Dumazet

Which driver handles this Traverse Solos card ?

If br2684_push() is used, it seems it lacks proper call to skb_reset_mac_header(skb) in paths where eth_type_trans() is not called.

Later in xfrm4_mode_tunnel_input() we crash because we assume skb_mac_header() is valid.

when br2684_push() doesnt call eth_type_trans() the underlying packet doesnt have a mac address header -- just an llc header that says 'ip packet is next'.

Tested-by Niccolò Belli
#12 lore raw
In reply to Niccolò Belli
Quoted 3 lines

Your patch does solve the problem, thanks!

Thanks for testing.

Here is the official patch I submit for review then.

[PATCH] ipsec: be careful of non existing mac headers

Nicollo Belli reported ipsec crashes in case we handle a frame without mac header (atm in his case)

Before copying mac header, better make sure it is present.

Bugzilla reference: https://bugzilla.kernel.org/show_bug.cgi?id=42809

---
 net/ipv4/xfrm4_mode_beet.c   |    9 +++++----
 net/ipv4/xfrm4_mode_tunnel.c |   10 ++++++----
 net/ipv6/xfrm6_mode_beet.c   |    9 +++++----
 net/ipv6/xfrm6_mode_tunnel.c |   10 ++++++----
 4 files changed, 22 insertions(+), 16 deletions(-)
net/ipv4/xfrm4_mode_beet.c +5 -4
index 6341818..d3451f6 100644
@@ -111,10 +111,11 @@ static int xfrm4_beet_input(struct xfrm_state *x, struct sk_buff *skb)
 	skb_push(skb, sizeof(*iph));
 	skb_reset_network_header(skb);
 
-	memmove(skb->data - skb->mac_len, skb_mac_header(skb),
-		skb->mac_len);
-	skb_set_mac_header(skb, -skb->mac_len);
-
+	if (skb_mac_header_was_set(skb)) {
+		memmove(skb->data - skb->mac_len, skb_mac_header(skb),
+			skb->mac_len);
+		skb_set_mac_header(skb, -skb->mac_len);
+	}
 	xfrm4_beet_make_header(skb);
 
 	iph = ip_hdr(skb);
net/ipv4/xfrm4_mode_tunnel.c +6 -4
index 534972e..a646f30 100644
@@ -66,7 +66,6 @@ static int xfrm4_mode_tunnel_output(struct xfrm_state *x, struct sk_buff *skb)
 
 static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 {
-	const unsigned char *old_mac;
 	int err = -EINVAL;
 
 	if (XFRM_MODE_SKB_CB(skb)->protocol != IPPROTO_IPIP)
@@ -84,9 +83,12 @@ static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 	if (!(x->props.flags & XFRM_STATE_NOECN))
 		ipip_ecn_decapsulate(skb);
 
-	old_mac = skb_mac_header(skb);
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	if (skb_mac_header_was_set(skb)) {
+		const unsigned char *old_mac = skb_mac_header(skb);
+
+		skb_set_mac_header(skb, -skb->mac_len);
+		memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	}
 	skb_reset_network_header(skb);
 	err = 0;
 
net/ipv6/xfrm6_mode_beet.c +5 -4
index a81ce94..74c4b92 100644
@@ -80,7 +80,6 @@ static int xfrm6_beet_output(struct xfrm_state *x, struct sk_buff *skb)
 static int xfrm6_beet_input(struct xfrm_state *x, struct sk_buff *skb)
 {
 	struct ipv6hdr *ip6h;
-	const unsigned char *old_mac;
 	int size = sizeof(struct ipv6hdr);
 	int err;
 
@@ -91,10 +90,12 @@ static int xfrm6_beet_input(struct xfrm_state *x, struct sk_buff *skb)
 	__skb_push(skb, size);
 	skb_reset_network_header(skb);
 
-	old_mac = skb_mac_header(skb);
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	if (skb_mac_header_was_set(skb)) {
+		const unsigned char *old_mac = skb_mac_header(skb);
 
+		skb_set_mac_header(skb, -skb->mac_len);
+		memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	}
 	xfrm6_beet_make_header(skb);
 
 	ip6h = ipv6_hdr(skb);
net/ipv6/xfrm6_mode_tunnel.c +6 -4
index 261e6e6..edb7091 100644
@@ -63,7 +63,6 @@ static int xfrm6_mode_tunnel_output(struct xfrm_state *x, struct sk_buff *skb)
 static int xfrm6_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 {
 	int err = -EINVAL;
-	const unsigned char *old_mac;
 
 	if (XFRM_MODE_SKB_CB(skb)->protocol != IPPROTO_IPV6)
 		goto out;
@@ -80,9 +79,12 @@ static int xfrm6_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 	if (!(x->props.flags & XFRM_STATE_NOECN))
 		ipip6_ecn_decapsulate(skb);
 
-	old_mac = skb_mac_header(skb);
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	if (skb_mac_header_was_set(skb)) {
+		const unsigned char *old_mac = skb_mac_header(skb);
+
+		skb_set_mac_header(skb, -skb->mac_len);
+		memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	}
 	skb_reset_network_header(skb);
 	err = 0;
 
#13 lore raw
Quoted 4 lines · Eric Dumazet

[PATCH] ipsec: be careful of non existing mac headers

Nicollo Belli reported ipsec crashes in case we handle a frame without mac header (atm in his case)

Oops sorry for your name being mangled in changelog, its Niccolò as correctly spelled in the "Reported-by" and "Tested-by" tags

#14 lore raw

Il 23/02/2012 15:39, Eric Dumazet ha scritto:

Quoted 2 lines

Oops sorry for your name being mangled in changelog, its Niccolò as correctly spelled in the "Reported-by" and "Tested-by" tags

Don't worry and thanks for your help. I'm currently running the official patch you submitted for review.

Niccolò

maintainer
#15 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Thu, 23 Feb 2012 15:36:26 +0100

Quoted 5 lines

[PATCH] ipsec: be careful of non existing mac headers

Nicollo Belli reported ipsec crashes in case we handle a frame without mac header (atm in his case)

Quoted 7 lines

Before copying mac header, better make sure it is present.

Bugzilla reference: https://bugzilla.kernel.org/show_bug.cgi?id=42809

Three instances of the same piece of code, maybe a helper function is appropriate at that point? :-) You might even get ambitious and add a big comment to that helper function explaining the situation.

#16 lore raw
Quoted 3 lines · David Miller

Three instances of the same piece of code, maybe a helper function is appropriate at that point? :-) You might even get ambitious and add a big comment to that helper function explaining the situation.

I did have this idea but refrained because this kind of things is harder to backport to stable kernels.

I'll make a cleanup in net-next if you agree ?

maintainer
#17 lore raw

From: Eric Dumazet <[email protected]>
Date: Thu, 23 Feb 2012 21:17:23 +0100

Quoted 9 lines · David Miller
Quoted 3 lines

Three instances of the same piece of code, maybe a helper function is appropriate at that point? :-) You might even get ambitious and add a big comment to that helper function explaining the situation.

I did have this idea but refrained because this kind of things is harder to backport to stable kernels.

I'll make a cleanup in net-next if you agree ?

I think the work to backport is equal whether you use a helper function or not. Heck, use an inline function so you don't have to worry about module exports or any details like that.

Besides, I'm the one who likely has to backport this thing, so... :-)

#18 lore raw
Quoted 5 lines · David Miller

I think the work to backport is equal whether you use a helper function or not. Heck, use an inline function so you don't have to worry about module exports or any details like that.

Besides, I'm the one who likely has to backport this thing, so... :-)

Great, I'll send an updated version in a couple of minutes ;)

#19 lore raw

PATCH v2 1/1 ipsec: be careful of non existing mac headers

In reply to David Miller

Niccolo Belli reported ipsec crashes in case we handle a frame without mac header (atm in his case)

Before copying mac header, better make sure it is present.

Bugzilla reference: https://bugzilla.kernel.org/show_bug.cgi?id=42809

---
V2: added skb_mac_header_rebuild() helper as David suggested.

 include/linux/skbuff.h       |   10 ++++++++++
 net/ipv4/xfrm4_mode_beet.c   |    5 +----
 net/ipv4/xfrm4_mode_tunnel.c |    6 ++----
 net/ipv6/xfrm6_mode_beet.c   |    6 +-----
 net/ipv6/xfrm6_mode_tunnel.c |    6 ++----
 5 files changed, 16 insertions(+), 17 deletions(-)
include/linux/skbuff.h +10 -0
index 50db9b0..ae86ade 100644
@@ -1465,6 +1465,16 @@ static inline void skb_set_mac_header(struct sk_buff *skb, const int offset)
 }
 #endif /* NET_SKBUFF_DATA_USES_OFFSET */
 
+static inline void skb_mac_header_rebuild(struct sk_buff *skb)
+{
+	if (skb_mac_header_was_set(skb)) {
+		const unsigned char *old_mac = skb_mac_header(skb);
+
+		skb_set_mac_header(skb, -skb->mac_len);
+		memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	}
+}
+
 static inline int skb_checksum_start_offset(const struct sk_buff *skb)
 {
 	return skb->csum_start - skb_headroom(skb);
net/ipv4/xfrm4_mode_beet.c +1 -4
index 6341818..e3db3f9 100644
@@ -110,10 +110,7 @@ static int xfrm4_beet_input(struct xfrm_state *x, struct sk_buff *skb)
 
 	skb_push(skb, sizeof(*iph));
 	skb_reset_network_header(skb);
-
-	memmove(skb->data - skb->mac_len, skb_mac_header(skb),
-		skb->mac_len);
-	skb_set_mac_header(skb, -skb->mac_len);
+	skb_mac_header_rebuild(skb);
 
 	xfrm4_beet_make_header(skb);
 
net/ipv4/xfrm4_mode_tunnel.c +2 -4
index 534972e..ed4bf11 100644
@@ -66,7 +66,6 @@ static int xfrm4_mode_tunnel_output(struct xfrm_state *x, struct sk_buff *skb)
 
 static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 {
-	const unsigned char *old_mac;
 	int err = -EINVAL;
 
 	if (XFRM_MODE_SKB_CB(skb)->protocol != IPPROTO_IPIP)
@@ -84,10 +83,9 @@ static int xfrm4_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 	if (!(x->props.flags & XFRM_STATE_NOECN))
 		ipip_ecn_decapsulate(skb);
 
-	old_mac = skb_mac_header(skb);
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
 	skb_reset_network_header(skb);
+	skb_mac_header_rebuild(skb);
+
 	err = 0;
 
 out:
net/ipv6/xfrm6_mode_beet.c +1 -5
index a81ce94..9949a35 100644
@@ -80,7 +80,6 @@ static int xfrm6_beet_output(struct xfrm_state *x, struct sk_buff *skb)
 static int xfrm6_beet_input(struct xfrm_state *x, struct sk_buff *skb)
 {
 	struct ipv6hdr *ip6h;
-	const unsigned char *old_mac;
 	int size = sizeof(struct ipv6hdr);
 	int err;
 
@@ -90,10 +89,7 @@ static int xfrm6_beet_input(struct xfrm_state *x, struct sk_buff *skb)
 
 	__skb_push(skb, size);
 	skb_reset_network_header(skb);
-
-	old_mac = skb_mac_header(skb);
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
+	skb_mac_header_rebuild(skb);
 
 	xfrm6_beet_make_header(skb);
 
net/ipv6/xfrm6_mode_tunnel.c +2 -4
index 261e6e6..9f2095b 100644
@@ -63,7 +63,6 @@ static int xfrm6_mode_tunnel_output(struct xfrm_state *x, struct sk_buff *skb)
 static int xfrm6_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 {
 	int err = -EINVAL;
-	const unsigned char *old_mac;
 
 	if (XFRM_MODE_SKB_CB(skb)->protocol != IPPROTO_IPV6)
 		goto out;
@@ -80,10 +79,9 @@ static int xfrm6_mode_tunnel_input(struct xfrm_state *x, struct sk_buff *skb)
 	if (!(x->props.flags & XFRM_STATE_NOECN))
 		ipip6_ecn_decapsulate(skb);
 
-	old_mac = skb_mac_header(skb);
-	skb_set_mac_header(skb, -skb->mac_len);
-	memmove(skb_mac_header(skb), old_mac, skb->mac_len);
 	skb_reset_network_header(skb);
+	skb_mac_header_rebuild(skb);
+
 	err = 0;
 
 out:
maintainer
#20 lore raw

Re: [PATCH V2] ipsec: be careful of non existing mac headers

From: Eric Dumazet <[email protected]>
Date: Thu, 23 Feb 2012 21:55:02 +0100

Quoted 10 lines

Niccolo Belli reported ipsec crashes in case we handle a frame without mac header (atm in his case)

Before copying mac header, better make sure it is present.

Bugzilla reference: https://bugzilla.kernel.org/show_bug.cgi?id=42809

Applied and queued up for -stable, thanks.

BTW, if anyone wants to see what's queued up for -stable I have a public patchwork bundle for it:

http://patchwork.ozlabs.org/bundle/davem/stable/

Just FYI.