Skip to contentSkip to conversation

Help needed - Kernel lockup while running ipsec

PATCH In review Vakul Garg 3 participants 15 messages Aug 19, 2019 – Aug 22, 2019

Subsystems (4)
lore
Share & tools
lore

Vakul Garg

author
#0 lore raw

Hi

With kernel 4.14.122, I am getting a kernel softlockup while running single static ipsec tunnel. The problem reproduces mostly after running 8-10 hours of ipsec encap test (on my dual core arm board).

I found that in function xfrm_policy_lookup_bytype(), the policy in variable 'ret' shows refcnt=0 under problem situation. This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the lockup.

Can some body please provide me pointers about 'refcnt'?
Is it legitimate for 'refcnt' to become '0'? Under what condition can it become '0'?

Regards
Vakul

#1 lore raw
Quoted 10 lines · Vakul Garg

Hi

With kernel 4.14.122, I am getting a kernel softlockup while running single static ipsec tunnel. The problem reproduces mostly after running 8-10 hours of ipsec encap test (on my dual core arm board).

I found that in function xfrm_policy_lookup_bytype(), the policy in variable 'ret' shows refcnt=0 under problem situation. This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the lockup.

Can some body please provide me pointers about 'refcnt'?
Is it legitimate for 'refcnt' to become '0'? Under what condition can it become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

author
#2 lore raw

Thanks for your response.

Quoted 18 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Monday, August 19, 2019 11:08 PM
To: Vakul Garg <[email protected]>
Cc: [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 3 lines · Vakul Garg

Hi

With kernel 4.14.122, I am getting a kernel softlockup while running single

static ipsec tunnel.

Quoted 1 line

The problem reproduces mostly after running 8-10 hours of ipsec encap

test (on my dual core arm board).

Quoted 2 lines

I found that in function xfrm_policy_lookup_bytype(), the policy in variable

'ret' shows refcnt=0 under problem situation.

Quoted 1 line

This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the

Quoted 8 lines

lockup.

Quoted 3 lines

Can some body please provide me pointers about 'refcnt'?
Is it legitimate for 'refcnt' to become '0'? Under what condition can it

become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

It seems that policy reference count never gets decremented during packet ipsec encap. It is getting incremented for every frame that hits the policy.
In setkey -DP output, I see refcnt to be wrapping around after '0'.

Is this designed to be like this or is it weird?

#3 lore raw
Quoted 12 lines · Vakul Garg
Quoted 12 lines
Quoted 1 line

With kernel 4.14.122, I am getting a kernel softlockup while running single

static ipsec tunnel.

Quoted 1 line

The problem reproduces mostly after running 8-10 hours of ipsec encap

test (on my dual core arm board).

Quoted 2 lines

I found that in function xfrm_policy_lookup_bytype(), the policy in variable

'ret' shows refcnt=0 under problem situation.

Quoted 1 line

This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the

lockup.

Quoted 3 lines

Can some body please provide me pointers about 'refcnt'?
Is it legitimate for 'refcnt' to become '0'? Under what condition can it

Quoted 8 lines
Quoted 4 lines

become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

It seems that policy reference count never gets decremented during packet ipsec encap. It is getting incremented for every frame that hits the policy.
In setkey -DP output, I see refcnt to be wrapping around after '0'.

Thats a bug. Does this affect 4.14 only or does this happen on current tree as well?

author
#4 lore raw
Quoted 32 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 2:53 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 22 lines · Vakul Garg
Quoted 21 lines
Quoted 2 lines

With kernel 4.14.122, I am getting a kernel softlockup while running single

static ipsec tunnel.

Quoted 2 lines

The problem reproduces mostly after running 8-10 hours of ipsec encap

test (on my dual core arm board).

Quoted 3 lines

I found that in function xfrm_policy_lookup_bytype(), the policy in variable

'ret' shows refcnt=0 under problem situation.

Quoted 2 lines

This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the

lockup.

Quoted 4 lines

Can some body please provide me pointers about 'refcnt'? Is it legitimate for 'refcnt' to become '0'? Under what condition can it

become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

Quoted 7 lines
Quoted 1 line

It seems that policy reference count never gets decremented during packet

ipsec encap.

Quoted 2 lines

It is getting incremented for every frame that hits the policy. In setkey -DP output, I see refcnt to be wrapping around after '0'.

Thats a bug. Does this affect 4.14 only or does this happen on current tree as well?

I am yet to try it on 4.19.
Can you help me with the right fix? Which part of code should it get decremented? I am not conversant with xfrm code.

author
#5 lore raw
Quoted 34 lines
Quoted 33 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 2:53 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 23 lines · Vakul Garg
Quoted 14 lines
Quoted 2 lines

With kernel 4.14.122, I am getting a kernel softlockup while running single

static ipsec tunnel.

Quoted 2 lines

The problem reproduces mostly after running 8-10 hours of ipsec encap

test (on my dual core arm board).

Quoted 3 lines

I found that in function xfrm_policy_lookup_bytype(), the policy in variable

'ret' shows refcnt=0 under problem situation.

Quoted 2 lines

This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the

lockup.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

Can some body please provide me pointers about 'refcnt'? Is it legitimate for 'refcnt' to become '0'? Under what condition can it

become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

It seems that policy reference count never gets decremented during

packet

Quoted 8 lines
Quoted 6 lines

ipsec encap.

Quoted 2 lines

It is getting incremented for every frame that hits the policy. In setkey -DP output, I see refcnt to be wrapping around after '0'.

Thats a bug. Does this affect 4.14 only or does this happen on current tree as well?

I am yet to try it on 4.19.

Correction: I am yet to try it on current tree.

Quoted 3 lines

Can you help me with the right fix? Which part of code should it get decremented?
I am not conversant with xfrm code.

#6 lore raw
In reply to Vakul Garg
Quoted 37 lines · Vakul Garg
Quoted 37 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 2:53 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 17 lines · Vakul Garg
Quoted 17 lines
Quoted 2 lines

With kernel 4.14.122, I am getting a kernel softlockup while running single

static ipsec tunnel.

Quoted 2 lines

The problem reproduces mostly after running 8-10 hours of ipsec encap

test (on my dual core arm board).

Quoted 3 lines

I found that in function xfrm_policy_lookup_bytype(), the policy in variable

'ret' shows refcnt=0 under problem situation.

Quoted 2 lines

This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the

lockup.

Quoted 4 lines

Can some body please provide me pointers about 'refcnt'? Is it legitimate for 'refcnt' to become '0'? Under what condition can it

Quoted 6 lines
Quoted 4 lines

become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

It seems that policy reference count never gets decremented during packet

ipsec encap.

Quoted 2 lines

It is getting incremented for every frame that hits the policy. In setkey -DP output, I see refcnt to be wrapping around after '0'.

Quoted 6 lines
Quoted 2 lines

Thats a bug. Does this affect 4.14 only or does this happen on current tree as well?

I am yet to try it on 4.19.
Can you help me with the right fix? Which part of code should it get decremented? I am not conversant with xfrm code.

Normally policy reference counts get decremented when the skb is free'd, via dst destruction (xfrm_dst_destroy()).

Do you see a dst leak as well?

author
#7 lore raw
Quoted 53 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 3:08 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 35 lines · Vakul Garg
Quoted 35 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 2:53 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 17 lines · Vakul Garg
Quoted 17 lines
Quoted 2 lines

With kernel 4.14.122, I am getting a kernel softlockup while running single

static ipsec tunnel.

Quoted 2 lines

The problem reproduces mostly after running 8-10 hours of ipsec encap

test (on my dual core arm board).

Quoted 3 lines

I found that in function xfrm_policy_lookup_bytype(), the policy in variable

'ret' shows refcnt=0 under problem situation.

Quoted 2 lines

This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the

lockup.

Quoted 4 lines

Can some body please provide me pointers about 'refcnt'? Is it legitimate for 'refcnt' to become '0'? Under what condition can it

Quoted 7 lines
Quoted 4 lines

become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

It seems that policy reference count never gets decremented during packet

ipsec encap.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

It is getting incremented for every frame that hits the policy. In setkey -DP output, I see refcnt to be wrapping around after '0'.

Thats a bug. Does this affect 4.14 only or does this happen on current tree as well?

I am yet to try it on 4.19.
Can you help me with the right fix? Which part of code should it get

Quoted 7 lines

decremented?

Quoted 1 line

I am not conversant with xfrm code.

Normally policy reference counts get decremented when the skb is free'd, via dst destruction (xfrm_dst_destroy()).

Do you see a dst leak as well?

Can you please guide me how to detect it?

(I am checking refcount on recent kernel and will let you know.)

author
#8 lore raw
Quoted 58 lines
Quoted 52 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 3:08 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 41 lines · Vakul Garg
Quoted 40 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 2:53 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 17 lines · Vakul Garg
Quoted 17 lines
Quoted 2 lines

With kernel 4.14.122, I am getting a kernel softlockup while running single

static ipsec tunnel.

Quoted 2 lines

The problem reproduces mostly after running 8-10 hours of ipsec encap

test (on my dual core arm board).

Quoted 3 lines

I found that in function xfrm_policy_lookup_bytype(), the policy in variable

'ret' shows refcnt=0 under problem situation.

Quoted 2 lines

This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the

lockup.

Quoted 4 lines

Can some body please provide me pointers about 'refcnt'? Is it legitimate for 'refcnt' to become '0'? Under what condition can it

Quoted 7 lines
Quoted 4 lines

become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

It seems that policy reference count never gets decremented during packet

ipsec encap.

Quoted 2 lines

It is getting incremented for every frame that hits the policy. In setkey -DP output, I see refcnt to be wrapping around after '0'.

Thats a bug. Does this affect 4.14 only or does this happen on current tree as well?

Quoted 6 lines
Quoted 2 lines

I am yet to try it on 4.19.
Can you help me with the right fix? Which part of code should it get

decremented?

Quoted 1 line

I am not conversant with xfrm code.

Normally policy reference counts get decremented when the skb is free'd,

Quoted 8 lines

via

Quoted 3 lines

dst destruction (xfrm_dst_destroy()).

Do you see a dst leak as well?

Can you please guide me how to detect it?

(I am checking refcount on recent kernel and will let you know.)

Policy refcount is decreasing properly on 4.19.
Same should be on the latest kernel too.

author
#9 lore raw
Quoted 71 lines

-----Original Message-----
From: Vakul Garg
Sent: Tuesday, August 20, 2019 4:08 PM
To: Florian Westphal <[email protected]>
Cc: [email protected]
Subject: RE: Help needed - Kernel lockup while running ipsec

Quoted 50 lines
Quoted 50 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 3:08 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 41 lines · Vakul Garg
Quoted 40 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Tuesday, August 20, 2019 2:53 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 17 lines · Vakul Garg
Quoted 17 lines
Quoted 2 lines

With kernel 4.14.122, I am getting a kernel softlockup while running single

static ipsec tunnel.

Quoted 2 lines

The problem reproduces mostly after running 8-10 hours of ipsec encap

test (on my dual core arm board).

Quoted 3 lines

I found that in function xfrm_policy_lookup_bytype(), the policy in variable

'ret' shows refcnt=0 under problem situation.

Quoted 2 lines

This creates an infinite loop in xfrm_policy_lookup_bytype() and hence the

lockup.

Quoted 4 lines

Can some body please provide me pointers about 'refcnt'? Is it legitimate for 'refcnt' to become '0'? Under what condition can it

Quoted 7 lines
Quoted 4 lines

become '0'?

Yes, when policy is destroyed and the last user calls xfrm_pol_put() which will invoke call_rcu to free the structure.

It seems that policy reference count never gets decremented during packet

ipsec encap.

Quoted 2 lines

It is getting incremented for every frame that hits the policy. In setkey -DP output, I see refcnt to be wrapping around after '0'.

Thats a bug. Does this affect 4.14 only or does this happen on current tree as well?

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

I am yet to try it on 4.19.
Can you help me with the right fix? Which part of code should it get

decremented?

Quoted 1 line

I am not conversant with xfrm code.

Normally policy reference counts get decremented when the skb is

free'd,

Quoted 3 lines

via

Quoted 2 lines

dst destruction (xfrm_dst_destroy()).

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Do you see a dst leak as well?

Can you please guide me how to detect it?

(I am checking refcount on recent kernel and will let you know.)

Policy refcount is decreasing properly on 4.19.
Same should be on the latest kernel too.

On kernel-4.14, I find dst_release() is getting called through xfrm_output_one(). However since dst->__refcnt gets decremented to '1',
the call_rcu(&dst->rcu_head, dst_destroy_rcu) is not invoked.

On kernel-4.19, dst->__refcnt gets decremented to '0', hence things fall in place and dst_destroy_rcu() eventually executes.

Any further help/pointers for kernel-4.14 would be deeply appreciated.

#10 lore raw
Quoted 11 lines · Vakul Garg
Quoted 2 lines

Policy refcount is decreasing properly on 4.19.
Same should be on the latest kernel too.

On kernel-4.14, I find dst_release() is getting called through xfrm_output_one(). However since dst->__refcnt gets decremented to '1',
the call_rcu(&dst->rcu_head, dst_destroy_rcu) is not invoked.

On kernel-4.19, dst->__refcnt gets decremented to '0', hence things fall in place and dst_destroy_rcu() eventually executes.

Any further help/pointers for kernel-4.14 would be deeply appreciated.

Can you try getting rid of the pcpu dst cache?

I had a look at 4.14-stable and it at least lacks 2950278d2d04ff531.

I've attached an (untested) revert of the pcpu cache (its gone in 4.19 and onwards).

From 058cb6719223d10dc57743dbf5c20424f118e7e7 Mon Sep 17 00:00:00 2001
From: Florian Westphal <[email protected]>
Date: Mon, 25 Jun 2018 17:26:02 +0200
Subject: [PATCH 4.4.14.y] xfrm: policy: remove pcpu policy cache

commit e4db5b61c572475bbbcf63e3c8a2606bfccf2c9d upstream.

Kristian Evensen says:
In a project I am involved in, we are running ipsec (Strongswan) on different mt7621-based routers. Each router is configured as an initiator and has around ~30 tunnels to different responders (running on misc. devices). Before the flow cache was removed (kernel 4.9), we got a combined throughput of around 70Mbit/s for all tunnels on one router. However, we recently switched to kernel 4.14 (4.14.48), and the total throughput is somewhere around 57Mbit/s (best-case). I.e., a drop of around 20%. Reverting the flow cache removal restores, as expected, performance levels to that of kernel 4.9.

When pcpu xdst exists, it has to be validated first before it can be used.

A negative hit thus increases cost vs. no-cache.

As number of tunnels increases, hit rate decreases so this pcpu caching isn't a viable strategy.

Furthermore, the xdst cache also needs to run with BH off, so when removing this the bh disable/enable pairs can be removed too.

Kristian tested a 4.14.y backport of this change and reported increased performance:

In our tests, the throughput reduction has been reduced from around -20% to -5%. We also see that the overall throughput is independent of the number of tunnels, while before the throughput was reduced as the number of tunnels increased.

---
 include/net/xfrm.h     |   1 -
 net/xfrm/xfrm_device.c |  10 ---
 net/xfrm/xfrm_policy.c | 138 +----------------------------------------
 net/xfrm/xfrm_state.c  |   5 +-
 4 files changed, 3 insertions(+), 151 deletions(-)
include/net/xfrm.h +0 -1
index db99efb2d1d0..bdf185ae93db 100644
@@ -323,7 +323,6 @@ int xfrm_policy_register_afinfo(const struct xfrm_policy_afinfo *afinfo, int fam
 void xfrm_policy_unregister_afinfo(const struct xfrm_policy_afinfo *afinfo);
 void km_policy_notify(struct xfrm_policy *xp, int dir,
 		      const struct km_event *c);
-void xfrm_policy_cache_flush(void);
 void km_state_notify(struct xfrm_state *x, const struct km_event *c);
 
 struct xfrm_tmpl;
net/xfrm/xfrm_device.c +0 -10
index 30e5746085b8..4e458fd9236a 100644
@@ -153,12 +153,6 @@ static int xfrm_dev_register(struct net_device *dev)
 	return NOTIFY_DONE;
 }
 
-static int xfrm_dev_unregister(struct net_device *dev)
-{
-	xfrm_policy_cache_flush();
-	return NOTIFY_DONE;
-}
-
 static int xfrm_dev_feat_change(struct net_device *dev)
 {
 	if ((dev->features & NETIF_F_HW_ESP) && !dev->xfrmdev_ops)
@@ -178,7 +172,6 @@ static int xfrm_dev_down(struct net_device *dev)
 	if (dev->features & NETIF_F_HW_ESP)
 		xfrm_dev_state_flush(dev_net(dev), dev, true);
 
-	xfrm_policy_cache_flush();
 	return NOTIFY_DONE;
 }
 
@@ -190,9 +183,6 @@ static int xfrm_dev_event(struct notifier_block *this, unsigned long event, void
 	case NETDEV_REGISTER:
 		return xfrm_dev_register(dev);
 
-	case NETDEV_UNREGISTER:
-		return xfrm_dev_unregister(dev);
-
 	case NETDEV_FEAT_CHANGE:
 		return xfrm_dev_feat_change(dev);
 
net/xfrm/xfrm_policy.c +1 -137

191 lines not on this page · raw message

net/xfrm/xfrm_state.c +2 -3
index 0cd2bdf3b217..7c093de68780 100644
@@ -735,10 +735,9 @@ int xfrm_state_flush(struct net *net, u8 proto, bool task_valid)
 	}
 out:
 	spin_unlock_bh(&net->xfrm.xfrm_state_lock);
-	if (cnt) {
+	if (cnt)
 		err = 0;
-		xfrm_policy_cache_flush();
-	}
+
 	return err;
 }
 EXPORT_SYMBOL(xfrm_state_flush);
Signature · 1 line
2.21.0
author
#11 lore raw
Quoted 22 lines

-----Original Message-----
From: Florian Westphal <[email protected]>
Sent: Wednesday, August 21, 2019 9:42 PM
To: Vakul Garg <[email protected]>
Cc: Florian Westphal <[email protected]>; [email protected]
Subject: Re: Help needed - Kernel lockup while running ipsec

Quoted 4 lines · Vakul Garg
Quoted 2 lines

Policy refcount is decreasing properly on 4.19.
Same should be on the latest kernel too.

On kernel-4.14, I find dst_release() is getting called through

xfrm_output_one().

Quoted 7 lines

However since dst->__refcnt gets decremented to '1', the call_rcu(&dst->rcu_head, dst_destroy_rcu) is not invoked.

On kernel-4.19, dst->__refcnt gets decremented to '0', hence things fall in place and
dst_destroy_rcu() eventually executes.

Quoted 9 lines
Quoted 1 line

Any further help/pointers for kernel-4.14 would be deeply appreciated.

Can you try getting rid of the pcpu dst cache?

I had a look at 4.14-stable and it at least lacks 2950278d2d04ff531.

I've attached an (untested) revert of the pcpu cache (its gone in 4.19 and onwards).

This patch fixed the refcnt issue. Many thanks for your help. Would you send this patch for inclusion into 4.14-stable?

#12 lore raw

PATCH 1/1 xfrm: policy: remove pcpu policy cache

commit e4db5b61c572475bbbcf63e3c8a2606bfccf2c9d upstream.

Kristian Evensen says:
In a project I am involved in, we are running ipsec (Strongswan) on different mt7621-based routers. Each router is configured as an initiator and has around ~30 tunnels to different responders (running on misc. devices). Before the flow cache was removed (kernel 4.9), we got a combined throughput of around 70Mbit/s for all tunnels on one router. However, we recently switched to kernel 4.14 (4.14.48), and the total throughput is somewhere around 57Mbit/s (best-case). I.e., a drop of around 20%. Reverting the flow cache removal restores, as expected, performance levels to that of kernel 4.9.

When pcpu xdst exists, it has to be validated first before it can be used.

A negative hit thus increases cost vs. no-cache.

As number of tunnels increases, hit rate decreases so this pcpu caching isn't a viable strategy.

Furthermore, the xdst cache also needs to run with BH off, so when removing this the bh disable/enable pairs can be removed too.

Kristian tested a 4.14.y backport of this change and reported increased performance:

In our tests, the throughput reduction has been reduced from around -20% to -5%. We also see that the overall throughput is independent of the number of tunnels, while before the throughput was reduced as the number of tunnels increased.

---
Vakul Garg reports traffic going via ipsec tunnels will cause the kernel to spin in an infinite loop due to xfrm policy reference count overflowing and becoming 0.
The refcount leak is in the pcpu cache. Instead of fixing this, just remove the pcpu cache -- its not present in any other stable release. Vakul reported that this patch fixes the problem.

There are no major deviations from the upstream revert; conflicts were only due to context.

 include/net/xfrm.h     |   1 -
 net/xfrm/xfrm_device.c |  10 ---
 net/xfrm/xfrm_policy.c | 138 +----------------------------------------
 net/xfrm/xfrm_state.c  |   5 +-
 4 files changed, 3 insertions(+), 151 deletions(-)
include/net/xfrm.h +0 -1
index db99efb2d1d0..bdf185ae93db 100644
@@ -323,7 +323,6 @@ int xfrm_policy_register_afinfo(const struct xfrm_policy_afinfo *afinfo, int fam
 void xfrm_policy_unregister_afinfo(const struct xfrm_policy_afinfo *afinfo);
 void km_policy_notify(struct xfrm_policy *xp, int dir,
 		      const struct km_event *c);
-void xfrm_policy_cache_flush(void);
 void km_state_notify(struct xfrm_state *x, const struct km_event *c);
 
 struct xfrm_tmpl;
net/xfrm/xfrm_device.c +0 -10
index 30e5746085b8..4e458fd9236a 100644
@@ -153,12 +153,6 @@ static int xfrm_dev_register(struct net_device *dev)
 	return NOTIFY_DONE;
 }
 
-static int xfrm_dev_unregister(struct net_device *dev)
-{
-	xfrm_policy_cache_flush();
-	return NOTIFY_DONE;
-}
-
 static int xfrm_dev_feat_change(struct net_device *dev)
 {
 	if ((dev->features & NETIF_F_HW_ESP) && !dev->xfrmdev_ops)
@@ -178,7 +172,6 @@ static int xfrm_dev_down(struct net_device *dev)
 	if (dev->features & NETIF_F_HW_ESP)
 		xfrm_dev_state_flush(dev_net(dev), dev, true);
 
-	xfrm_policy_cache_flush();
 	return NOTIFY_DONE;
 }
 
@@ -190,9 +183,6 @@ static int xfrm_dev_event(struct notifier_block *this, unsigned long event, void
 	case NETDEV_REGISTER:
 		return xfrm_dev_register(dev);
 
-	case NETDEV_UNREGISTER:
-		return xfrm_dev_unregister(dev);
-
 	case NETDEV_FEAT_CHANGE:
 		return xfrm_dev_feat_change(dev);
 
net/xfrm/xfrm_policy.c +1 -137

191 lines not on this page · raw message

net/xfrm/xfrm_state.c +2 -3
index 0cd2bdf3b217..7c093de68780 100644
@@ -735,10 +735,9 @@ int xfrm_state_flush(struct net *net, u8 proto, bool task_valid)
 	}
 out:
 	spin_unlock_bh(&net->xfrm.xfrm_state_lock);
-	if (cnt) {
+	if (cnt)
 		err = 0;
-		xfrm_policy_cache_flush();
-	}
+
 	return err;
 }
 EXPORT_SYMBOL(xfrm_state_flush);
Signature · 1 line
2.21.0
#13 lore raw

Re: [PATCH 4.14.y stable] xfrm: policy: remove pcpu policy cache

Quoted 43 lines · Florian Westphal

commit e4db5b61c572475bbbcf63e3c8a2606bfccf2c9d upstream.

Kristian Evensen says:
In a project I am involved in, we are running ipsec (Strongswan) on different mt7621-based routers. Each router is configured as an initiator and has around ~30 tunnels to different responders (running on misc. devices). Before the flow cache was removed (kernel 4.9), we got a combined throughput of around 70Mbit/s for all tunnels on one router. However, we recently switched to kernel 4.14 (4.14.48), and the total throughput is somewhere around 57Mbit/s (best-case). I.e., a drop of around 20%. Reverting the flow cache removal restores, as expected, performance levels to that of kernel 4.9.

When pcpu xdst exists, it has to be validated first before it can be used.

A negative hit thus increases cost vs. no-cache.

As number of tunnels increases, hit rate decreases so this pcpu caching isn't a viable strategy.

Furthermore, the xdst cache also needs to run with BH off, so when removing this the bh disable/enable pairs can be removed too.

Kristian tested a 4.14.y backport of this change and reported increased performance:

In our tests, the throughput reduction has been reduced from around -20% to -5%. We also see that the overall throughput is independent of the number of tunnels, while before the throughput was reduced as the number of tunnels increased.

---
Vakul Garg reports traffic going via ipsec tunnels will cause the kernel to spin in an infinite loop due to xfrm policy reference count overflowing and becoming 0.
The refcount leak is in the pcpu cache. Instead of fixing this, just remove the pcpu cache -- its not present in any other stable release. Vakul reported that this patch fixes the problem.

Quoted 2 lines

There are no major deviations from the upstream revert; conflicts were only due to context.

Now queued up, does 4.9.y also need this?

thanks,

greg k-h

#14 lore raw

Re: [PATCH 4.14.y stable] xfrm: policy: remove pcpu policy cache

Quoted 44 lines · Greg KH
Quoted 43 lines · Florian Westphal

commit e4db5b61c572475bbbcf63e3c8a2606bfccf2c9d upstream.

Kristian Evensen says:
In a project I am involved in, we are running ipsec (Strongswan) on different mt7621-based routers. Each router is configured as an initiator and has around ~30 tunnels to different responders (running on misc. devices). Before the flow cache was removed (kernel 4.9), we got a combined throughput of around 70Mbit/s for all tunnels on one router. However, we recently switched to kernel 4.14 (4.14.48), and the total throughput is somewhere around 57Mbit/s (best-case). I.e., a drop of around 20%. Reverting the flow cache removal restores, as expected, performance levels to that of kernel 4.9.

When pcpu xdst exists, it has to be validated first before it can be used.

A negative hit thus increases cost vs. no-cache.

As number of tunnels increases, hit rate decreases so this pcpu caching isn't a viable strategy.

Furthermore, the xdst cache also needs to run with BH off, so when removing this the bh disable/enable pairs can be removed too.

Kristian tested a 4.14.y backport of this change and reported increased performance:

In our tests, the throughput reduction has been reduced from around -20% to -5%. We also see that the overall throughput is independent of the number of tunnels, while before the throughput was reduced as the number of tunnels increased.

---
Vakul Garg reports traffic going via ipsec tunnels will cause the kernel to spin in an infinite loop due to xfrm policy reference count overflowing and becoming 0.
The refcount leak is in the pcpu cache. Instead of fixing this, just remove the pcpu cache -- its not present in any other stable release. Vakul reported that this patch fixes the problem.

Quoted 4 lines
Quoted 2 lines

There are no major deviations from the upstream revert; conflicts were only due to context.

Now queued up, does 4.9.y also need this?

No, 4.14 was the first kernel with this thing and its already gone in 4.19.