pull request (net): ipsec 2013-11-01
Subsystems (3)
Steffen Klassert
maintainer1) Fix a possible race on ipcomp scratch buffers because of too early enabled siftirqs. From Michal Kubecek.
2) The current xfrm garbage collector threshold is too small for some workloads, resulting in bad performance on these workloads. Increase the threshold from 1024 to 32768.
3) Some codepaths might not have a dst_entry attached to the skb when calling xfrm_decode_session(). So add a check to prevent a null pointer dereference in this case.
Please pull or let me know if there are problems.
Thanks!
The following changes since commit e9e4ea74f06635f2ffc1dffe5ef40c854faa0a90:
net: smc91x: dont't use SMC_outw for fixing up halfword-aligned data (2013-10-11 17:50:59 -0400)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/klassert/ipsec.git master
for you to fetch changes up to 84502b5ef9849a9694673b15c31bd3ac693010ae:
xfrm: Fix null pointer dereference when decoding sessions (2013-11-01 07:08:46 +0100)
----------------------------------------------------------------
Michal Kubecek (1):
xfrm: prevent ipcomp scratch buffer race condition
Steffen Klassert (2):
xfrm: Increase the garbage collector threshold
xfrm: Fix null pointer dereference when decoding sessions
net/ipv4/xfrm4_policy.c | 8 ++++++-- net/ipv6/xfrm6_policy.c | 8 ++++++-- net/xfrm/xfrm_ipcomp.c | 12 ++++++------ 3 files changed, 18 insertions(+), 10 deletions(-)
Steffen Klassert
maintainerPATCH 1/3 xfrm: prevent ipcomp scratch buffer race condition
From: Michal Kubecek <[email protected]>
In ipcomp_compress(), sortirq is enabled too early, allowing the per-cpu scratch buffer to be rewritten by ipcomp_decompress() (called on the same CPU in softirq context) between populating the buffer and copying the compressed data to the skb.
v2: as pointed out by Steffen Klassert, if we also move the local_bh_disable() before reading the per-cpu pointers, we can get rid of get_cpu()/put_cpu().
v3: removed ipcomp_decompress part (as explained by Herbert Xu, it cannot be called from process context), get rid of cpu variable (thanks to Eric Dumazet)
--- net/xfrm/xfrm_ipcomp.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
net/xfrm/xfrm_ipcomp.c +6 -6
@@ -141,14 +141,14 @@ static int ipcomp_compress(struct xfrm_state *x, struct sk_buff *skb)const int plen = skb->len;int dlen =IPCOMP_SCRATCH_SIZE ;u8 *start = skb->data; -const int cpu =get_cpu (); -u8 *scratch = *per_cpu_ptr (ipcomp_scratches, cpu); -struct crypto_comp *tfm = *per_cpu_ptr (ipcd->tfms, cpu); +struct crypto_comp *tfm; +u8 *scratch;int err;local_bh_disable (); + scratch = *this_cpu_ptr (ipcomp_scratches); + tfm = *this_cpu_ptr (ipcd->tfms); err =crypto_comp_compress (tfm, start, plen, scratch, &dlen); -local_bh_enable ();if (err) goto out;
@@ -158,13 +158,13 @@ static int ipcomp_compress(struct xfrm_state *x, struct sk_buff *skb) }memcpy (start +sizeof (struct ip_comp_hdr ), scratch, dlen); -put_cpu (); +local_bh_enable ();pskb_trim (skb, dlen +sizeof (struct ip_comp_hdr ));return 0 ; out: -put_cpu (); +local_bh_enable ();return err; }
Signature · 1 line
1.7.9.5
Steffen Klassert
maintainerPATCH 2/3 xfrm: Increase the garbage collector threshold
In reply to Steffen KlassertWith the removal of the routing cache, we lost the option to tweak the garbage collector threshold along with the maximum routing cache size. So git commit 703fb94ec ("xfrm: Fix the gc threshold value for ipv4") moved back to a static threshold.
It turned out that the current threshold before we start garbage collecting is much to small for some workloads, so increase it from 1024 to 32768. This means that we start the garbage collector if we have more than 32768 dst entries in the system and refuse new allocations if we are above 65536.
--- net/ipv4/xfrm4_policy.c | 2 +- net/ipv6/xfrm6_policy.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
net/ipv4/xfrm4_policy.c +1 -1
@@ -236,7 +236,7 @@ static struct dst_ops xfrm4_dst_ops = { .destroy = xfrm4_dst_destroy, .ifdown = xfrm4_dst_ifdown, .local_out = __ip_local_out, - .gc_thresh = 1024, + .gc_thresh = 32768, };static struct xfrm_policy_afinfo xfrm4_policy_afinfo = {
net/ipv6/xfrm6_policy.c +1 -1
@@ -285,7 +285,7 @@ static struct dst_ops xfrm6_dst_ops = { .destroy = xfrm6_dst_destroy, .ifdown = xfrm6_dst_ifdown, .local_out = __ip6_local_out, - .gc_thresh = 1024, + .gc_thresh = 32768, };static struct xfrm_policy_afinfo xfrm6_policy_afinfo = {
Signature · 1 line
1.7.9.5
Re: [PATCH 2/3] xfrm: Increase the garbage collector threshold
On Fri, 2013-11-01 at 09:21 +0100, Steffen Klassert wrote:
Hello Steffen,
Quoted 6 lines
It turned out that the current threshold before we start garbage collecting is much to small for some workloads, so increase it from 1024 to 32768. This means that we start the garbage collector if we have more than 32768 dst entries in the system and refuse new allocations if we are above 65536.
sorry to dig an old thread, but could you please explain the lifetime of those dst entries ?
After basic tests it seems the ipv4 dst entry needs to be released first before the xfrm dst can be released by gc.
Signature · 1 line
Maxime
Steffen Klassert
maintainerRe: [PATCH 2/3] xfrm: Increase the garbage collector threshold
Quoted 12 lines · Maxime Bizon
On Fri, 2013-11-01 at 09:21 +0100, Steffen Klassert wrote:
Hello Steffen,
Quoted 6 lines
It turned out that the current threshold before we start garbage collecting is much to small for some workloads, so increase it from 1024 to 32768. This means that we start the garbage collector if we have more than 32768 dst entries in the system and refuse new allocations if we are above 65536.
Quoted 6 lines
sorry to dig an old thread, but could you please explain the lifetime of those dst entries ?
After basic tests it seems the ipv4 dst entry needs to be released first before the xfrm dst can be released by gc.
Can you please be a bit more precise with your problem description?
This patch changes only the number of cache entries before we start garbage collecting. It does not change anything on the garbage collector itself.
Re: [PATCH 2/3] xfrm: Increase the garbage collector threshold
Quoted 1 line · Steffen Klassert
Can you please be a bit more precise with your problem description?
yes sorry I wasn't clear.
I have a problem with an even simpler workload that the one using apache bench in the original bug report.
I am using ipsec transport mode between two hosts and just run this on one side:
while :; do wget -O /dev/null http://remote_host/; done
I was surprised to see it fails after only 1024 requests (ENOBUF on connect), and how long I had to wait to be able to do new requests.
After debugging I saw that the xfrm gc was called but was not able to release anything.
after running "ip route flush cache", which forces all ipv4 dst entries to be released, suddenly the xfrm gc had something to free, and xfrm entry count went to zero.
So if it is correct that once a ipv4 dst entry exists, the xfrm entry cannot be gc-ed, then we need to make sure we allow more xfrm entries to be allocated than ipv4 dst.
Quoted 3 lines
This patch changes only the number of cache entries before we start garbage collecting. It does not change anything on the garbage collector itself.
Yes my point was that it just hides another underlying problem.
Signature · 1 line
Maxime
Steffen Klassert
maintainerRe: [PATCH 2/3] xfrm: Increase the garbage collector threshold
Quoted 18 lines · Maxime Bizon
Quoted 1 line · Steffen Klassert
Can you please be a bit more precise with your problem description?
yes sorry I wasn't clear.
I have a problem with an even simpler workload that the one using apache bench in the original bug report.
I am using ipsec transport mode between two hosts and just run this on one side:
while :; do wget -O
/dev/nullhttp://remote_host/; doneI was surprised to see it fails after only 1024 requests (ENOBUF on connect), and how long I had to wait to be able to do new requests.
Quoted 6 lines
After debugging I saw that the xfrm gc was called but was not able to release anything.
after running "ip route flush cache", which forces all ipv4 dst entries to be released, suddenly the xfrm gc had something to free, and xfrm entry count went to zero.
Well, "ip route flush cache" bumps the rt_genid what marks all routes as invalid. The xfrm garbage collector will notice this on the next run and deletes all invalid cached routes.
Garbage collecting is different from flushing, it only removes stale routes and keeps everything that was used recently. That's the whole point of this caching, we cache recently used IPsec routes to avoid a slow path lookup.
Quoted 4 lines
So if it is correct that once a ipv4 dst entry exists, the xfrm entry cannot be gc-ed, then we need to make sure we allow more xfrm entries to be allocated than ipv4 dst.
No, we don't cache plain ipv4 routes, we cache only IPsec routes. The original ipv4 dst_entry is cached together with the xfrm dst_entry as a xfrm bundle at the IPsec flow cache, so it does not make sense to ensure something like that.
Re: [PATCH 2/3] xfrm: Increase the garbage collector threshold
Quoted 12 lines · Steffen Klassert
Quoted 11 lines
while :; do wget -O
/dev/nullhttp://remote_host/; doneI was surprised to see it fails after only 1024 requests (ENOBUF on connect), and how long I had to wait to be able to do new requests.
After debugging I saw that the xfrm gc was called but was not able to release anything.
after running "ip route flush cache", which forces all ipv4 dst entries to be released, suddenly the xfrm gc had something to free, and xfrm entry count went to zero.
Quoted 8 lines
Well, "ip route flush cache" bumps the
rt_genidwhat marks all routes as invalid. The xfrm garbage collector will notice this on the next run and deletes all invalid cached routes.Garbage collecting is different from flushing, it only removes stale routes and keeps everything that was used recently. That's the whole point of this caching, we cache recently used IPsec routes to avoid a slow path lookup.
ok I don't know this part of the code at all.
but if it is only a *cache*, then it should not limit any new allocations for the reason that the cache is full, it should take slow path instead.
Signature · 1 line
Maxime
Steffen Klassert
maintainerPATCH 3/3 xfrm: Fix null pointer dereference when decoding sessions
In reply to Steffen KlassertOn some codepaths the skb does not have a dst entry when xfrm_decode_session() is called. So check for a valid skb_dst() before dereferencing the device interface index. We use 0 as the device index if there is no valid skb_dst(), or at reverse decoding we use skb_iif as device interface index.
Bug was introduced with git commit bafd4bd4dc ("xfrm: Decode sessions with output interface.").
--- net/ipv4/xfrm4_policy.c | 6 +++++- net/ipv6/xfrm6_policy.c | 6 +++++- 2 files changed, 10 insertions(+), 2 deletions(-)
net/ipv4/xfrm4_policy.c +5 -1
@@ -104,10 +104,14 @@ _decode_session4(struct sk_buff *skb, struct flowi *fl, int reverse)const struct iphdr *iph =ip_hdr (skb);u8 *xprth =skb_network_header (skb) + iph->ihl *4 ;struct flowi4 *fl4 = &fl->u.ip4; +int oif =0 ; + +if (skb_dst (skb)) + oif =skb_dst (skb)->dev->ifindex;memset (fl4,0 ,sizeof (struct flowi4 )); fl4->flowi4_mark = skb->mark; - fl4->flowi4_oif =skb_dst (skb)->dev->ifindex; + fl4->flowi4_oif = reverse ? skb->skb_iif : oif;if (!ip_is_fragment (iph)) {switch (iph->protocol) {
net/ipv6/xfrm6_policy.c +5 -1
@@ -135,10 +135,14 @@ _decode_session6(struct sk_buff *skb, struct flowi *fl, int reverse)struct ipv6_opt_hdr *exthdr;const unsigned char *nh =skb_network_header (skb);u8 nexthdr = nh[IP6CB (skb)->nhoff]; +int oif =0 ; + +if (skb_dst (skb)) + oif =skb_dst (skb)->dev->ifindex;memset (fl6,0 ,sizeof (struct flowi6 )); fl6->flowi6_mark = skb->mark; - fl6->flowi6_oif =skb_dst (skb)->dev->ifindex; + fl6->flowi6_oif = reverse ? skb->skb_iif : oif; fl6->daddr = reverse ? hdr->saddr : hdr->daddr; fl6->saddr = reverse ? hdr->daddr : hdr->saddr;
Signature · 1 line
1.7.9.5
Re: [PATCH 3/3] xfrm: Fix null pointer dereference when decoding sessions
Quoted 7 lines · Steffen Klassert
On some codepaths the skb does not have a dst entry when
xfrm_decode_session()is called. So check for a validskb_dst()before dereferencing the device interface index. We use 0 as the device index if there is no validskb_dst(), or at reverse decoding we useskb_iifas device interface index.
Quoted 6 lines
Bug was introduced with git commit
bafd4bd4dc("xfrm: Decode sessions with output interface.").Reported-by Meelis Roos <[email protected]>Tested-by Meelis Roos <[email protected]>Signed-off-by Steffen Klassert <[email protected]>
Please queue this for 3.12-stable, I've seen this bug trigger a panic when heavily using Bittorrent (screenshot:
http://i.imgur.com/OIKVccM.jpg )
Thanks,
Matthias
Re: [PATCH 3/3] xfrm: Fix null pointer dereference when decoding sessions
Quoted 15 lines · Matthias Schiffer
Quoted 7 lines · Steffen Klassert
On some codepaths the skb does not have a dst entry when
xfrm_decode_session()is called. So check for a validskb_dst()before dereferencing the device interface index. We use 0 as the device index if there is no validskb_dst(), or at reverse decoding we useskb_iifas device interface index.Quoted 6 lines
Bug was introduced with git commit
bafd4bd4dc("xfrm: Decode sessions with output interface.").Reported-by Meelis Roos <[email protected]>Tested-by Meelis Roos <[email protected]>Signed-off-by Steffen Klassert <[email protected]>
Quoted 7 lines
Please queue this for 3.12-stable, I've seen this bug trigger a panic when heavily using Bittorrent (screenshot:
http://i.imgur.com/OIKVccM.jpg )Thanks,
Matthias
Ping? Several people I know have hit this panic with 3.12.x at least once or twice by now during heavy IPv6 usage.
David Miller
maintainerRe: [PATCH 3/3] xfrm: Fix null pointer dereference when decoding sessions
From: Matthias Schiffer <[email protected]>
Date: Sun, 01 Dec 2013 05:19:53 +0100
Quoted 20 lines · Matthias Schiffer
Quoted 19 lines · Steffen Klassert
Quoted 7 lines
On some codepaths the skb does not have a dst entry when
xfrm_decode_session()is called. So check for a validskb_dst()before dereferencing the device interface index. We use 0 as the device index if there is no validskb_dst(), or at reverse decoding we useskb_iifas device interface index.Quoted 6 lines
Bug was introduced with git commit
bafd4bd4dc("xfrm: Decode sessions with output interface.").Reported-by Meelis Roos <[email protected]>Tested-by Meelis Roos <[email protected]>Signed-off-by Steffen Klassert <[email protected]>Please queue this for 3.12-stable, I've seen this bug trigger a panic when heavily using Bittorrent (screenshot:
http://i.imgur.com/OIKVccM.jpg )
Quoted 6 lines
Quoted 3 lines
Thanks,
MatthiasPing? Several people I know have hit this panic with 3.12.x at least once or twice by now during heavy IPv6 usage.
Steffen submitted it to [email protected] for inclusion on Monday November 25th.
Steffen Klassert
maintainerRe: [PATCH 3/3] xfrm: Fix null pointer dereference when decoding sessions
Ccing stable.
Quoted 27 lines · David Miller
From: Matthias Schiffer <[email protected]>
Date: Sun, 01 Dec 2013 05:19:53 +0100Quoted 24 lines · Matthias Schiffer
Quoted 22 lines · Steffen Klassert
Quoted 7 lines
On some codepaths the skb does not have a dst entry when
xfrm_decode_session()is called. So check for a validskb_dst()before dereferencing the device interface index. We use 0 as the device index if there is no validskb_dst(), or at reverse decoding we useskb_iifas device interface index.Quoted 6 lines
Bug was introduced with git commit
bafd4bd4dc("xfrm: Decode sessions with output interface.").Reported-by Meelis Roos <[email protected]>Tested-by Meelis Roos <[email protected]>Signed-off-by Steffen Klassert <[email protected]>Please queue this for 3.12-stable, I've seen this bug trigger a panic when heavily using Bittorrent (screenshot:
http://i.imgur.com/OIKVccM.jpg )Thanks,
Matthias
Quoted 5 lines
Quoted 2 lines
Ping? Several people I know have hit this panic with 3.12.x at least once or twice by now during heavy IPv6 usage.
Steffen submitted it to [email protected] for inclusion on Monday November 25th.
It apparently did not make it into v3.12.2 and it is not in the v3.12.3 review included. So I wonder if I need to do anything additional to get it included. It is the first time that I did a stable submission myself.
I tried to follow the instructions at Documentation/stable_kernel_rules.txt but maybe I've missed something.
Re: [PATCH 3/3] xfrm: Fix null pointer dereference when decoding sessions
Am Dienstag, 3. Dezember 2013, 10:56:45 schrieb Steffen Klassert:
Quoted 35 lines
Ccing stable.
Quoted 23 lines · David Miller
From: Matthias Schiffer <[email protected]>
Date: Sun, 01 Dec 2013 05:19:53 +0100Quoted 20 lines · Matthias Schiffer
Quoted 19 lines · Steffen Klassert
Quoted 7 lines
On some codepaths the skb does not have a dst entry when
xfrm_decode_session()is called. So check for a validskb_dst()before dereferencing the device interface index. We use 0 as the device index if there is no validskb_dst(), or at reverse decoding we useskb_iifas device interface index.Quoted 6 lines
Bug was introduced with git commit
bafd4bd4dc("xfrm: Decode sessions with output interface.").Reported-by Meelis Roos <[email protected]>Tested-by Meelis Roos <[email protected]>Signed-off-by Steffen Klassert <[email protected]>Please queue this for 3.12-stable, I've seen this bug trigger a panic when heavily using Bittorrent (screenshot:
http://i.imgur.com/OIKVccM.jpg )Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Thanks,
MatthiasPing? Several people I know have hit this panic with 3.12.x at least once or twice by now during heavy IPv6 usage.
Steffen submitted it to [email protected] for inclusion on Monday November 25th.
Quoted 7 lines
It apparently did not make it into v3.12.2 and it is not in the v3.12.3 review included. So I wonder if I need to do anything additional to get it included. It is the first time that I did a stable submission myself.
I tried to follow the instructions at
Documentation/stable_kernel_rules.txtbut maybe I've missed something.
There is another weakness with ipv6 in 3.12.x which - under certain circumstances - can be exploited remotely:
ipv6 fragmentation-related panic in netfilter
I would be very glad if this would make it into 3.12 stable soon.
Regards,
Signature · 3 lines
Wolfgang Walter Studentenwerk München Anstalt des öffentlichen Rechts
Re: [PATCH 3/3] xfrm: Fix null pointer dereference when decoding sessions
Quoted 44 lines · Wolfgang Walter
Am Dienstag, 3. Dezember 2013, 10:56:45 schrieb Steffen Klassert:
Quoted 35 lines
Ccing stable.
Quoted 23 lines · David Miller
From: Matthias Schiffer <[email protected]>
Date: Sun, 01 Dec 2013 05:19:53 +0100Quoted 20 lines · Matthias Schiffer
Quoted 19 lines · Steffen Klassert
Quoted 7 lines
On some codepaths the skb does not have a dst entry when
xfrm_decode_session()is called. So check for a validskb_dst()before dereferencing the device interface index. We use 0 as the device index if there is no validskb_dst(), or at reverse decoding we useskb_iifas device interface index.Quoted 6 lines
Bug was introduced with git commit
bafd4bd4dc("xfrm: Decode sessions with output interface.").Reported-by Meelis Roos <[email protected]>Tested-by Meelis Roos <[email protected]>Signed-off-by Steffen Klassert <[email protected]>Please queue this for 3.12-stable, I've seen this bug trigger a panic when heavily using Bittorrent (screenshot:
http://i.imgur.com/OIKVccM.jpg )Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Thanks,
MatthiasPing? Several people I know have hit this panic with 3.12.x at least once or twice by now during heavy IPv6 usage.
Steffen submitted it to [email protected] for inclusion on Monday November 25th.
Quoted 7 lines
It apparently did not make it into v3.12.2 and it is not in the v3.12.3 review included. So I wonder if I need to do anything additional to get it included. It is the first time that I did a stable submission myself.
I tried to follow the instructions at
Documentation/stable_kernel_rules.txtbut maybe I've missed something.
Quoted 6 lines
There is another weakness with ipv6 in 3.12.x which - under certain circumstances - can be exploited remotely:
ipv6 fragmentation-related panic in netfilter
I would be very glad if this would make it into 3.12 stable soon.
These patches are already queued up for stable by David:
http://patchwork.ozlabs.org/bundle/davem/stable/?state=*
Greetings,
Hannes
David Miller
maintainerRe: [PATCH 3/3] xfrm: Fix null pointer dereference when decoding sessions
In reply to Steffen KlassertFrom: Steffen Klassert <[email protected]>
Date: Tue, 3 Dec 2013 10:56:45 +0100
Quoted 4 lines
It apparently did not make it into v3.12.2 and it is not in the v3.12.3 review included. So I wonder if I need to do anything additional to get it included. It is the first time that I did a stable submission myself.
I'm taking care of this myself now, don't worry any more about it.
Thanks.
David Miller
maintainerFrom: Steffen Klassert <[email protected]>
Date: Fri, 1 Nov 2013 09:21:44 +0100
Quoted 7 lines
1) Fix a possible race on ipcomp scratch buffers because of too early enabled siftirqs. From Michal Kubecek.
2) The current xfrm garbage collector threshold is too small for some workloads, resulting in bad performance on these workloads. Increase the threshold from 1024 to 32768.
Quoted 5 lines
3) Some codepaths might not have a
dst_entryattached to the skb when callingxfrm_decode_session(). So add a check to prevent a null pointer dereference in this case.Please pull or let me know if there are problems.
Pulled, thanks a lot Steffen!