net-next: NULL pointer dereference on adding a net namespace and a system freeze

Hi!

Running Fedora 20 with net-next I get the following warning when libvirt or rtkit comes up:

[  272.143488] kmem_cache_sanity_check (flow_cache): Cache name already exists.
[  272.143586] CPU: 0 PID: 975 Comm: libvirtd Not tainted 3.14.0-rc5+ #1
[  272.143589] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  272.143591]  0000000000000000 ffff88003ceadba0 ffffffff8167baf0 ffff88003db3d300
[  272.143595]  ffff88003ceadc18 ffffffff8117795b ffff88003ceadbc8 ffff88003b235158
[  272.143599]  0000000000000000 0000000000040000 0000000000000068 0000000000000000
[  272.143602] Call Trace:
[  272.143610]  [<ffffffff8167baf0>] dump_stack+0x4d/0x66
[  272.143615]  [<ffffffff8117795b>] kmem_cache_create_memcg+0x12b/0x420
[  272.143618]  [<ffffffff81177c7b>] kmem_cache_create+0x2b/0x30
[  272.143622]  [<ffffffff815c4a0e>] flow_cache_init+0x2e/0x2b0
[  272.143626]  [<ffffffff8164b017>] xfrm_net_init+0x227/0x360
[  272.143629]  [<ffffffff8164af41>] ? xfrm_net_init+0x151/0x360
[  272.143632]  [<ffffffff815a5921>] ops_init+0x41/0x150
[  272.143635]  [<ffffffff815a5aa3>] setup_net+0x73/0x110
[  272.143638]  [<ffffffff815a5fe2>] copy_net_ns+0x72/0x100
[  272.143642]  [<ffffffff810943f9>] create_new_namespaces+0xf9/0x190
[  272.143645]  [<ffffffff81094560>] copy_namespaces+0xd0/0xf0
[  272.143648]  [<ffffffff81094495>] ? copy_namespaces+0x5/0xf0
[  272.143651]  [<ffffffff81069be0>] copy_process.part.31+0x950/0x1b30
[  272.143655]  [<ffffffff8106af95>] do_fork+0xd5/0x370
[  272.143658]  [<ffffffff811c1b2d>] ? __fput+0x17d/0x240
[  272.143662]  [<ffffffff8110440c>] ? __audit_syscall_entry+0x9c/0xf0
[  272.143665]  [<ffffffff8106b2b6>] SyS_clone+0x16/0x20
[  272.143669]  [<ffffffff8168cf19>] stub_clone+0x69/0x90
[  272.143673]  [<ffffffff8168cb69>] ? system_call_fastpath+0x16/0x1b

When I try to add a netns with

# ip netns add abcd
I it dies with:

Quoted 33 lines · Jakub Kicinski

Hi!

Running Fedora 20 with net-next I get the following warning when libvirt or rtkit comes up:

[  272.143488] kmem_cache_sanity_check (flow_cache): Cache name already exists.
[  272.143586] CPU: 0 PID: 975 Comm: libvirtd Not tainted 3.14.0-rc5+ #1
[  272.143589] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  272.143591]  0000000000000000 ffff88003ceadba0 ffffffff8167baf0 ffff88003db3d300
[  272.143595]  ffff88003ceadc18 ffffffff8117795b ffff88003ceadbc8 ffff88003b235158
[  272.143599]  0000000000000000 0000000000040000 0000000000000068 0000000000000000
[  272.143602] Call Trace:
[  272.143610]  [<ffffffff8167baf0>] dump_stack+0x4d/0x66
[  272.143615]  [<ffffffff8117795b>] kmem_cache_create_memcg+0x12b/0x420
[  272.143618]  [<ffffffff81177c7b>] kmem_cache_create+0x2b/0x30
[  272.143622]  [<ffffffff815c4a0e>] flow_cache_init+0x2e/0x2b0
[  272.143626]  [<ffffffff8164b017>] xfrm_net_init+0x227/0x360
[  272.143629]  [<ffffffff8164af41>] ? xfrm_net_init+0x151/0x360
[  272.143632]  [<ffffffff815a5921>] ops_init+0x41/0x150
[  272.143635]  [<ffffffff815a5aa3>] setup_net+0x73/0x110
[  272.143638]  [<ffffffff815a5fe2>] copy_net_ns+0x72/0x100
[  272.143642]  [<ffffffff810943f9>] create_new_namespaces+0xf9/0x190
[  272.143645]  [<ffffffff81094560>] copy_namespaces+0xd0/0xf0
[  272.143648]  [<ffffffff81094495>] ? copy_namespaces+0x5/0xf0
[  272.143651]  [<ffffffff81069be0>] copy_process.part.31+0x950/0x1b30
[  272.143655]  [<ffffffff8106af95>] do_fork+0xd5/0x370
[  272.143658]  [<ffffffff811c1b2d>] ? __fput+0x17d/0x240
[  272.143662]  [<ffffffff8110440c>] ? __audit_syscall_entry+0x9c/0xf0
[  272.143665]  [<ffffffff8106b2b6>] SyS_clone+0x16/0x20
[  272.143669]  [<ffffffff8168cf19>] stub_clone+0x69/0x90
[  272.143673]  [<ffffffff8168cb69>] ? system_call_fastpath+0x16/0x1b

Quoted 3 lines

When I try to add a netns with

# ip netns add abcd
I it dies with:

Yep, commit ca925cf1534ebcec332c08719a7dee6ee1782ce4 is buggy.

flowcache: Make flow cache name space aware

Inserting a entry into flowcache, or flushing flowcache should be based on per net scope. The reason to do so is flushing operation from fat netns crammed with flow entries will also making the slim netns with only a few flow cache entries go away in original implementation.

Since flowcache is tightly coupled with IPsec, so it would be easier to put flow cache global parameters into xfrm namespace part. And one last thing needs to do is bumping flow cache genid, and flush flow cache should also be made in per net style.

Signed-off-by Fan Du <[email protected]>

Signed-off-by Steffen Klassert <[email protected]>

I fail to understand why the kmem_cache must be private to a netns.

Quoted 53 lines · Eric Dumazet

Quoted 33 lines · Jakub Kicinski

Hi!

Running Fedora 20 with net-next I get the following warning when libvirt or rtkit comes up:

[  272.143488] kmem_cache_sanity_check (flow_cache): Cache name already exists.
[  272.143586] CPU: 0 PID: 975 Comm: libvirtd Not tainted 3.14.0-rc5+ #1
[  272.143589] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  272.143591]  0000000000000000 ffff88003ceadba0 ffffffff8167baf0 ffff88003db3d300
[  272.143595]  ffff88003ceadc18 ffffffff8117795b ffff88003ceadbc8 ffff88003b235158
[  272.143599]  0000000000000000 0000000000040000 0000000000000068 0000000000000000
[  272.143602] Call Trace:
[  272.143610]  [<ffffffff8167baf0>] dump_stack+0x4d/0x66
[  272.143615]  [<ffffffff8117795b>] kmem_cache_create_memcg+0x12b/0x420
[  272.143618]  [<ffffffff81177c7b>] kmem_cache_create+0x2b/0x30
[  272.143622]  [<ffffffff815c4a0e>] flow_cache_init+0x2e/0x2b0
[  272.143626]  [<ffffffff8164b017>] xfrm_net_init+0x227/0x360
[  272.143629]  [<ffffffff8164af41>] ? xfrm_net_init+0x151/0x360
[  272.143632]  [<ffffffff815a5921>] ops_init+0x41/0x150
[  272.143635]  [<ffffffff815a5aa3>] setup_net+0x73/0x110
[  272.143638]  [<ffffffff815a5fe2>] copy_net_ns+0x72/0x100
[  272.143642]  [<ffffffff810943f9>] create_new_namespaces+0xf9/0x190
[  272.143645]  [<ffffffff81094560>] copy_namespaces+0xd0/0xf0
[  272.143648]  [<ffffffff81094495>] ? copy_namespaces+0x5/0xf0
[  272.143651]  [<ffffffff81069be0>] copy_process.part.31+0x950/0x1b30
[  272.143655]  [<ffffffff8106af95>] do_fork+0xd5/0x370
[  272.143658]  [<ffffffff811c1b2d>] ? __fput+0x17d/0x240
[  272.143662]  [<ffffffff8110440c>] ? __audit_syscall_entry+0x9c/0xf0
[  272.143665]  [<ffffffff8106b2b6>] SyS_clone+0x16/0x20
[  272.143669]  [<ffffffff8168cf19>] stub_clone+0x69/0x90
[  272.143673]  [<ffffffff8168cb69>] ? system_call_fastpath+0x16/0x1b

Quoted 3 lines

When I try to add a netns with

# ip netns add abcd
I it dies with:

Yep, commit ca925cf1534ebcec332c08719a7dee6ee1782ce4 is buggy.

flowcache: Make flow cache name space aware

Inserting a entry into flowcache, or flushing flowcache should be based on per net scope. The reason to do so is flushing operation from fat netns crammed with flow entries will also making the slim netns with only a few flow cache entries go away in original implementation.

Since flowcache is tightly coupled with IPsec, so it would be easier to put flow cache global parameters into xfrm namespace part. And one last thing needs to do is bumping flow cache genid, and flush flow cache should also be made in per net style.

Quoted 4 lines

Signed-off-by Fan Du<[email protected]>

Signed-off-by Steffen Klassert<[email protected]>

I fail to understand why the kmem_cache must be private to a netns.

Could you please try the following patch ?

include/net/netns/xfrm.h +0 -1

index 51f0dce7b643..3492434baf88 100644
@@ -64,7 +64,6 @@ struct netns_xfrm {
 
  	/* flow cache part */
  	struct flow_cache	flow_cache_global;
-	struct kmem_cache	*flow_cachep;
  	atomic_t		flow_cache_genid;
  	struct list_head	flow_cache_gc_list;
  	spinlock_t		flow_cache_gc_lock;

net/core/flow.c +8 -6

index 344a184011fd..102f8ea2eb6e 100644
@@ -45,6 +45,8 @@ struct flow_flush_info {
  	struct completion		completion;
  };
 
+static struct kmem_cache *flow_cachep __read_mostly;
+
  #define flow_cache_hash_size(cache)	(1<<  (cache)->hash_shift)
  #define FLOW_HASH_RND_PERIOD		(10 * 60 * HZ)
 

@@ -75,7 +77,7 @@ static void flow_entry_kill(struct flow_cache_entry *fle,
  {
  	if (fle->object)
  		fle->object->ops->delete(fle->object);
-	kmem_cache_free(xfrm->flow_cachep, fle);
+	kmem_cache_free(flow_cachep, fle);
  }
 
  static void flow_cache_gc_task(struct work_struct *work)

@@ -230,7 +232,7 @@ flow_cache_lookup(struct net *net, const struct flowi *key, u16 family, u8 dir,
  		if (fcp->hash_count>  fc->high_watermark)
  			flow_cache_shrink(fc, fcp);
 
-		fle = kmem_cache_alloc(net->xfrm.flow_cachep, GFP_ATOMIC);
+		fle = kmem_cache_alloc(flow_cachep, GFP_ATOMIC);
  		if (fle) {
  			fle->net = net;
  			fle->family = family;

@@ -435,10 +437,10 @@ int flow_cache_init(struct net *net)
  	int i;
  	struct flow_cache *fc =&net->xfrm.flow_cache_global;
 
-	/* Initialize per-net flow cache global variables here */
-	net->xfrm.flow_cachep = kmem_cache_create("flow_cache",
-					sizeof(struct flow_cache_entry),
-					0, SLAB_PANIC, NULL);
+	if (!flow_cachep)
+		flow_cachep = kmem_cache_create("flow_cache",
+						sizeof(struct flow_cache_entry),
+						0, SLAB_PANIC, NULL);
  	spin_lock_init(&net->xfrm.flow_cache_gc_lock);
  	INIT_LIST_HEAD(&net->xfrm.flow_cache_gc_list);
  	INIT_WORK(&net->xfrm.flow_cache_gc_work, flow_cache_gc_task);

Sometimes network activity only on netns could trigger bugs like memory leakage, using per-netns kmem_cache could help to identify which netns to be blamed.

Anyway if this is inappropriate, let's make it global as you did below.

From: Eric Dumazet <[email protected]>

It is not legal to create multiple kmem_cache having the same name.

flowcache can use a single kmem_cache, no need for a per netns one.

Fixes ca925cf1534e ("flowcache: Make flow cache name space aware")

Reported-by Jakub Kicinski <[email protected]>

Tested-by Jakub Kicinski <[email protected]>

Tested-by Fan Du <[email protected]>

Signed-off-by Eric Dumazet <[email protected]>

Quoted 53 lines · Eric Dumazet

Quoted 33 lines · Jakub Kicinski

Hi!

Running Fedora 20 with net-next I get the following warning when libvirt or rtkit comes up:

[  272.143488] kmem_cache_sanity_check (flow_cache): Cache name already exists.
[  272.143586] CPU: 0 PID: 975 Comm: libvirtd Not tainted 3.14.0-rc5+ #1
[  272.143589] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  272.143591]  0000000000000000 ffff88003ceadba0 ffffffff8167baf0 ffff88003db3d300
[  272.143595]  ffff88003ceadc18 ffffffff8117795b ffff88003ceadbc8 ffff88003b235158
[  272.143599]  0000000000000000 0000000000040000 0000000000000068 0000000000000000
[  272.143602] Call Trace:
[  272.143610]  [<ffffffff8167baf0>] dump_stack+0x4d/0x66
[  272.143615]  [<ffffffff8117795b>] kmem_cache_create_memcg+0x12b/0x420
[  272.143618]  [<ffffffff81177c7b>] kmem_cache_create+0x2b/0x30
[  272.143622]  [<ffffffff815c4a0e>] flow_cache_init+0x2e/0x2b0
[  272.143626]  [<ffffffff8164b017>] xfrm_net_init+0x227/0x360
[  272.143629]  [<ffffffff8164af41>] ? xfrm_net_init+0x151/0x360
[  272.143632]  [<ffffffff815a5921>] ops_init+0x41/0x150
[  272.143635]  [<ffffffff815a5aa3>] setup_net+0x73/0x110
[  272.143638]  [<ffffffff815a5fe2>] copy_net_ns+0x72/0x100
[  272.143642]  [<ffffffff810943f9>] create_new_namespaces+0xf9/0x190
[  272.143645]  [<ffffffff81094560>] copy_namespaces+0xd0/0xf0
[  272.143648]  [<ffffffff81094495>] ? copy_namespaces+0x5/0xf0
[  272.143651]  [<ffffffff81069be0>] copy_process.part.31+0x950/0x1b30
[  272.143655]  [<ffffffff8106af95>] do_fork+0xd5/0x370
[  272.143658]  [<ffffffff811c1b2d>] ? __fput+0x17d/0x240
[  272.143662]  [<ffffffff8110440c>] ? __audit_syscall_entry+0x9c/0xf0
[  272.143665]  [<ffffffff8106b2b6>] SyS_clone+0x16/0x20
[  272.143669]  [<ffffffff8168cf19>] stub_clone+0x69/0x90
[  272.143673]  [<ffffffff8168cb69>] ? system_call_fastpath+0x16/0x1b

Quoted 3 lines

When I try to add a netns with

# ip netns add abcd
I it dies with:

Yep, commit ca925cf1534ebcec332c08719a7dee6ee1782ce4 is buggy.

flowcache: Make flow cache name space aware

Inserting a entry into flowcache, or flushing flowcache should be based on per net scope. The reason to do so is flushing operation from fat netns crammed with flow entries will also making the slim netns with only a few flow cache entries go away in original implementation.

Since flowcache is tightly coupled with IPsec, so it would be easier to put flow cache global parameters into xfrm namespace part. And one last thing needs to do is bumping flow cache genid, and flush flow cache should also be made in per net style.

Quoted 4 lines

Signed-off-by Fan Du <[email protected]>

Signed-off-by Steffen Klassert <[email protected]>

I fail to understand why the kmem_cache must be private to a netns.

Could you please try the following patch ?

include/net/netns/xfrm.h +0 -1

index 51f0dce7b643..3492434baf88 100644
@@ -64,7 +64,6 @@ struct netns_xfrm {
 
 	/* flow cache part */
 	struct flow_cache	flow_cache_global;
-	struct kmem_cache	*flow_cachep;
 	atomic_t		flow_cache_genid;
 	struct list_head	flow_cache_gc_list;
 	spinlock_t		flow_cache_gc_lock;

net/core/flow.c +8 -6

index 344a184011fd..102f8ea2eb6e 100644
@@ -45,6 +45,8 @@ struct flow_flush_info {
 	struct completion		completion;
 };
 
+static struct kmem_cache *flow_cachep __read_mostly;
+
 #define flow_cache_hash_size(cache)	(1 << (cache)->hash_shift)
 #define FLOW_HASH_RND_PERIOD		(10 * 60 * HZ)
 

@@ -75,7 +77,7 @@ static void flow_entry_kill(struct flow_cache_entry *fle,
 {
 	if (fle->object)
 		fle->object->ops->delete(fle->object);
-	kmem_cache_free(xfrm->flow_cachep, fle);
+	kmem_cache_free(flow_cachep, fle);
 }
 
 static void flow_cache_gc_task(struct work_struct *work)

@@ -230,7 +232,7 @@ flow_cache_lookup(struct net *net, const struct flowi *key, u16 family, u8 dir,
 		if (fcp->hash_count > fc->high_watermark)
 			flow_cache_shrink(fc, fcp);
 
-		fle = kmem_cache_alloc(net->xfrm.flow_cachep, GFP_ATOMIC);
+		fle = kmem_cache_alloc(flow_cachep, GFP_ATOMIC);
 		if (fle) {
 			fle->net = net;
 			fle->family = family;

@@ -435,10 +437,10 @@ int flow_cache_init(struct net *net)
 	int i;
 	struct flow_cache *fc = &net->xfrm.flow_cache_global;
 
-	/* Initialize per-net flow cache global variables here */
-	net->xfrm.flow_cachep = kmem_cache_create("flow_cache",
-					sizeof(struct flow_cache_entry),
-					0, SLAB_PANIC, NULL);
+	if (!flow_cachep)
+		flow_cachep = kmem_cache_create("flow_cache",
+						sizeof(struct flow_cache_entry),
+						0, SLAB_PANIC, NULL);
 	spin_lock_init(&net->xfrm.flow_cache_gc_lock);
 	INIT_LIST_HEAD(&net->xfrm.flow_cache_gc_list);
 	INIT_WORK(&net->xfrm.flow_cache_gc_work, flow_cache_gc_task);

It helps with the flow_cache warning and BUG ("ip netns add" works) but machine still freezes after 5-10 minutes. I will try to revert ca925cf1534ebcec332c08719a7dee6ee1782ce4 and see if that solves it.

Quoted 3 lines · Jakub Kiciński

It helps with the flow_cache warning and BUG ("ip netns add" works) but machine still freezes after 5-10 minutes. I will try to revert ca925cf1534ebcec332c08719a7dee6ee1782ce4 and see if that solves it.

I think the other issue is separate. It might be triggered by conntracking change.

Quoted 8 lines · Eric Dumazet

Quoted 3 lines · Jakub Kiciński

It helps with the flow_cache warning and BUG ("ip netns add" works) but machine still freezes after 5-10 minutes. I will try to revert ca925cf1534ebcec332c08719a7dee6ee1782ce4 and see if that solves it.

I think the other issue is separate. It might be triggered by conntracking change.

I bisected the other issue to be caused/uncovered by:

commit 1a1ccc96abb2ed9b8fbb71018e64b97324caef53

Author: Steffen Klassert <[email protected]>
Date:   Wed Feb 19 10:07:34 2014 +0100

xfrm: Remove caching of xfrm_policy_sk_bundles

We currently cache socket policy bundles at xfrm_policy_sk_bundles. These cached bundles are never used. Instead we create and cache a new one whenever xfrm_lookup() is called on a socket policy.

Most protocols cache the used routes to the socket, so let's remove the unused caching of socket policy bundles in xfrm.

Signed-off-by Steffen Klassert <[email protected]>

Machine freezes after FLOW_HASH_RND_PERIOD (default 10 minutes).

I bisected the other issue to be caused/uncovered by:

commit 1a1ccc96abb2ed9b8fbb71018e64b97324caef53

Author: Steffen Klassert <[email protected]>
Date:   Wed Feb 19 10:07:34 2014 +0100

xfrm: Remove caching of xfrm_policy_sk_bundles

We currently cache socket policy bundles at xfrm_policy_sk_bundles. These cached bundles are never used. Instead we create and cache a new one whenever xfrm_lookup() is called on a socket policy.

Most protocols cache the used routes to the socket, so let's remove the unused caching of socket policy bundles in xfrm.

Signed-off-by Steffen Klassert <[email protected]>

Machine freezes after FLOW_HASH_RND_PERIOD (default 10 minutes). Now get this warning during boot:

[   31.664820] ------------[ cut here ]------------
[   31.664824] WARNING: CPU: 2 PID: 3560 at /home/kuba/Development/Linux/net-next/lib/list_debug.c:33 __list_add+0xac/0xc0()
[   31.664826] list_add corruption. prev->next should be next (ffff880224579598), but was           (null). (prev=ffff8802106140e8).
[   31.664827] Modules linked in: xt_CHECKSUM tun bridge stp llc ccm xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ftdi_sio arc4 rt2800pci rt2800mmio rt2800lib crc_ccitt eeprom_93cx6 rt2x00pci kvm_amd rt2x00mmio rt2x00lib mac80211 kvm snd_ca0106 cfg80211 e1000e snd_ac97_codec ac97_bus microcode serio_raw ptp i2c_piix4 k10temp acpi_cpufreq pps_core wmi r8169 mii rfkill nfsd auth_rpcgss nfs_acl lockd binfmt_misc sunrpc usb_storage radeon drm_kms_helper ttm
[   31.664855] CPU: 2 PID: 3560 Comm: (t-daemon) Not tainted 3.14.0-rc2-1a1ccc96abb2ed9b8fbb71018e64b97324caef53+ #11
[   31.664856] Hardware name: Gigabyte Technology Co., Ltd. GA-MA790XT-UD4P/GA-MA790XT-UD4P, BIOS F9b 08/17/2012
[   31.664857]  0000000000000009 ffff8802242e7c70 ffffffff81627878 ffff8802242e7cb8
[   31.664859]  ffff8802242e7ca8 ffffffff8104a28d ffff880210610ea8 ffff880224579598
[   31.664861]  ffff8802106140e8 ffff880224578000 0000000000000000 ffff8802242e7d08
[   31.664863] Call Trace:
[   31.664865]  [<ffffffff81627878>] dump_stack+0x4d/0x66
[   31.664867]  [<ffffffff8104a28d>] warn_slowpath_common+0x7d/0xa0
[   31.664869]  [<ffffffff8104a2fc>] warn_slowpath_fmt+0x4c/0x50
[   31.664871]  [<ffffffff812fdd8c>] __list_add+0xac/0xc0
[   31.664873]  [<ffffffff81055d33>] __internal_add_timer+0x113/0x130
[   31.664875]  [<ffffffff81055f47>] internal_add_timer+0x17/0x40
[   31.664876]  [<ffffffff810587b2>] mod_timer+0x102/0x230
[   31.664878]  [<ffffffff810588f8>] add_timer+0x18/0x20
[   31.664880]  [<ffffffff81572204>] flow_cache_init+0x224/0x2b0
[   31.664882]  [<ffffffff815f7247>] xfrm_net_init+0x227/0x360

[   31.664884]  [<ffffffff815f7171>] ? xfrm_net_init+0x151/0x360
[   31.664886]  [<ffffffff81553131>] ops_init+0x41/0x150
[   31.664888]  [<ffffffff815532b3>] setup_net+0x73/0x110
[   31.664890]  [<ffffffff815537f2>] copy_net_ns+0x72/0x100
[   31.664892]  [<ffffffff81072619>] create_new_namespaces+0xf9/0x190
[   31.664894]  [<ffffffff81072891>] unshare_nsproxy_namespaces+0x61/0xa0
[   31.664895]  [<ffffffff81049949>] SyS_unshare+0x159/0x270
[   31.664897]  [<ffffffff81638092>] system_call_fastpath+0x16/0x1b

I was unable to reproduce this here, but it looks like the flowcache namespace changes are still not complete. We leak an active timer and all the allocated resources when we exit a namespace.

Could you please try the patch below?

Also, please send your config if the patch does not fix your problem.

Thanks!

---
 include/net/flow.h     |    1 +
 net/core/flow.c        |   18 ++++++++++++++++++
 net/xfrm/xfrm_policy.c |    7 ++++++-
 3 files changed, 25 insertions(+), 1 deletion(-)

include/net/flow.h +1 -0

index bee3741..64fd248 100644
@@ -219,6 +219,7 @@ struct flow_cache_object *flow_cache_lookup(struct net *net,
 					    u8 dir, flow_resolve_t resolver,
 					    void *ctx);
 int flow_cache_init(struct net *net);
+void flow_cache_fini(struct net *net);
 
 void flow_cache_flush(struct net *net);
 void flow_cache_flush_deferred(struct net *net);

net/core/flow.c +0 -0

index 102f8ea..d31c3c4 100644

net/core/flow.c +11 -0

+
+	for_each_possible_cpu(i) {
+		struct flow_cache_percpu *fcp = per_cpu_ptr(fc->percpu, i);
+		kfree(fcp->hash_table);
+		fcp->hash_table = NULL;
+	}
+
+	free_percpu(fc->percpu);
+	fc->percpu = NULL;
+}
+EXPORT_SYMBOL(flow_cache_fini);

net/xfrm/xfrm_policy.c +6 -1

index a75fae4..f02f511 100644
@@ -2913,15 +2913,19 @@ static int __net_init xfrm_net_init(struct net *net)
 	rv = xfrm_sysctl_init(net);
 	if (rv < 0)
 		goto out_sysctl;
+	rv = flow_cache_init(net);
+	if (rv < 0)
+		goto out;
 
 	/* Initialize the per-net locks here */
 	spin_lock_init(&net->xfrm.xfrm_state_lock);
 	rwlock_init(&net->xfrm.xfrm_policy_lock);
 	mutex_init(&net->xfrm.xfrm_cfg_mutex);
 
-	flow_cache_init(net);
 	return 0;
 
+out:
+	xfrm_sysctl_fini(net);
 out_sysctl:
 	xfrm_policy_fini(net);
 out_policy:

@@ -2934,6 +2938,7 @@ out_statistics:
 
 static void __net_exit xfrm_net_exit(struct net *net)
 {
+	flow_cache_fini(net);
 	xfrm_sysctl_fini(net);
 	xfrm_policy_fini(net);
 	xfrm_state_fini(net);

Quoted 38 lines · Steffen Klassert

I was unable to reproduce this here, but it looks like the flowcache namespace changes are still not complete. We leak an active timer and all the allocated resources when we exit a namespace.

Could you please try the patch below?

Also, please send your config if the patch does not fix your problem.

Thanks!

---
 include/net/flow.h     |    1 +
 net/core/flow.c        |   18 ++++++++++++++++++
 net/xfrm/xfrm_policy.c |    7 ++++++-
 3 files changed, 25 insertions(+), 1 deletion(-)

include/net/flow.h +1 -0

index bee3741..64fd248 100644
@@ -219,6 +219,7 @@ struct flow_cache_object *flow_cache_lookup(struct net *net,
 					    u8 dir, flow_resolve_t resolver,
 					    void *ctx);
 int flow_cache_init(struct net *net);
+void flow_cache_fini(struct net *net);
 
 void flow_cache_flush(struct net *net);
 void flow_cache_flush_deferred(struct net *net);

net/core/flow.c +2 -0

index 102f8ea..d31c3c4 100644
@@ -484,3 +484,21 @@ err:
 	return -ENOMEM;
 }
 EXPORT_SYMBOL(flow_cache_init);
+
+void flow_cache_fini(struct net *net)

net/core/flow.c err: · line 493

Quoted 5 lines

net/core/flow.c +5 -0

+{
+	int i;
+	struct flow_cache *fc = &net->xfrm.flow_cache_global;
+
+	del_timer(&fc->rnd_timer);

del_timer_sync() I guess is better.

Quoted 40 lines · Eric Dumazet

Quoted 38 lines · Steffen Klassert

I was unable to reproduce this here, but it looks like the flowcache namespace changes are still not complete. We leak an active timer and all the allocated resources when we exit a namespace.

Could you please try the patch below?

Also, please send your config if the patch does not fix your problem.

Thanks!

---
 include/net/flow.h     |    1 +
 net/core/flow.c        |   18 ++++++++++++++++++
 net/xfrm/xfrm_policy.c |    7 ++++++-
 3 files changed, 25 insertions(+), 1 deletion(-)

include/net/flow.h +1 -0

index bee3741..64fd248 100644
@@ -219,6 +219,7 @@ struct flow_cache_object *flow_cache_lookup(struct net *net,
 					    u8 dir, flow_resolve_t resolver,
 					    void *ctx);
 int flow_cache_init(struct net *net);
+void flow_cache_fini(struct net *net);
 
 void flow_cache_flush(struct net *net);
 void flow_cache_flush_deferred(struct net *net);

net/core/flow.c +2 -0

index 102f8ea..d31c3c4 100644
@@ -484,3 +484,21 @@ err:
 	return -ENOMEM;
 }
 EXPORT_SYMBOL(flow_cache_init);
+
+void flow_cache_fini(struct net *net)

Quoted 9 lines

net/core/flow.c err: · line 493

Quoted 5 lines

net/core/flow.c +5 -0

+{
+	int i;
+	struct flow_cache *fc = &net->xfrm.flow_cache_global;
+
+	del_timer(&fc->rnd_timer);

del_timer_sync() I guess is better.

Right.

I've just noticed that I also forgot to do a
unregister_hotcpu_notifier(&fc->hotcpu_notifier).

Will update the next version according to that.

Tested with del_timer_sync and unregister_hotcpu_notifier:

void flow_cache_fini(struct net *net)

{
	int i;
	struct flow_cache *fc = &net->xfrm.flow_cache_global;

	unregister_hotcpu_notifier(&fc->hotcpu_notifier);
	del_timer_sync(&fc->rnd_timer);

	for_each_possible_cpu(i) {
		struct flow_cache_percpu *fcp = per_cpu_ptr(fc->percpu, i);
		kfree(fcp->hash_table);
		fcp->hash_table = NULL;
	}

	free_percpu(fc->percpu);
	fc->percpu = NULL;
}

EXPORT_SYMBOL(flow_cache_fini);

Seems to solve both the freeze and the bug on restarting libvirtd/rtkit!

We leak an active timer, the hotcpu notifier and all allocated resources when we exit a namespace. Fix this by introducing a flow_cache_fini() function where we release the resources before we exit.

Fixes ca925cf1534e ("flowcache: Make flow cache name space aware")

Reported-by Jakub Kicinski <[email protected]>

Tested-by Jakub Kicinski <[email protected]>

Cc Eric Dumazet <[email protected]>

Cc Fan Du <[email protected]>

Signed-off-by Steffen Klassert <[email protected]>

---

We leak an active timer, the hotcpu notifier and all allocated resources when we exit a namespace. Fix this by introducing a flow_cache_fini() function where we release the resources before we exit.

Fixes ca925cf1534e ("flowcache: Make flow cache name space aware")

Reported-by Jakub Kicinski <[email protected]>

Tested-by Jakub Kicinski <[email protected]>

Cc Eric Dumazet <[email protected]>

Cc Fan Du <[email protected]>

Signed-off-by Steffen Klassert <[email protected]>

Quoted 13 lines · Jakub Kiciński

I bisected the other issue to be caused/uncovered by:

commit 1a1ccc96abb2ed9b8fbb71018e64b97324caef53

Author: Steffen Klassert <[email protected]>
Date:   Wed Feb 19 10:07:34 2014 +0100

xfrm: Remove caching of xfrm_policy_sk_bundles

We currently cache socket policy bundles at xfrm_policy_sk_bundles. These cached bundles are never used. Instead we create and cache a new one whenever xfrm_lookup() is called on a socket policy.

Quoted 5 lines

Most protocols cache the used routes to the socket, so let's remove the unused caching of socket policy bundles in xfrm.

Signed-off-by Steffen Klassert <[email protected]>

This patch should affect only on the usage of IPsec socket policies. Do you use socket policies, or do you use IPsec at all?

Quoted 24 lines

Machine freezes after FLOW_HASH_RND_PERIOD (default 10 minutes). Now get this warning during boot:

[   31.664820] ------------[ cut here ]------------
[   31.664824] WARNING: CPU: 2 PID: 3560 at /home/kuba/Development/Linux/net-next/lib/list_debug.c:33 __list_add+0xac/0xc0()
[   31.664826] list_add corruption. prev->next should be next (ffff880224579598), but was           (null). (prev=ffff8802106140e8).
[   31.664827] Modules linked in: xt_CHECKSUM tun bridge stp llc ccm xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ftdi_sio arc4 rt2800pci rt2800mmio rt2800lib crc_ccitt eeprom_93cx6 rt2x00pci kvm_amd rt2x00mmio rt2x00lib mac80211 kvm snd_ca0106 cfg80211 e1000e snd_ac97_codec ac97_bus microcode serio_raw ptp i2c_piix4 k10temp acpi_cpufreq pps_core wmi r8169 mii rfkill nfsd auth_rpcgss nfs_acl lockd binfmt_misc sunrpc usb_storage radeon drm_kms_helper ttm
[   31.664855] CPU: 2 PID: 3560 Comm: (t-daemon) Not tainted 3.14.0-rc2-1a1ccc96abb2ed9b8fbb71018e64b97324caef53+ #11
[   31.664856] Hardware name: Gigabyte Technology Co., Ltd. GA-MA790XT-UD4P/GA-MA790XT-UD4P, BIOS F9b 08/17/2012
[   31.664857]  0000000000000009 ffff8802242e7c70 ffffffff81627878 ffff8802242e7cb8
[   31.664859]  ffff8802242e7ca8 ffffffff8104a28d ffff880210610ea8 ffff880224579598
[   31.664861]  ffff8802106140e8 ffff880224578000 0000000000000000 ffff8802242e7d08
[   31.664863] Call Trace:
[   31.664865]  [<ffffffff81627878>] dump_stack+0x4d/0x66
[   31.664867]  [<ffffffff8104a28d>] warn_slowpath_common+0x7d/0xa0
[   31.664869]  [<ffffffff8104a2fc>] warn_slowpath_fmt+0x4c/0x50
[   31.664871]  [<ffffffff812fdd8c>] __list_add+0xac/0xc0
[   31.664873]  [<ffffffff81055d33>] __internal_add_timer+0x113/0x130
[   31.664875]  [<ffffffff81055f47>] internal_add_timer+0x17/0x40
[   31.664876]  [<ffffffff810587b2>] mod_timer+0x102/0x230
[   31.664878]  [<ffffffff810588f8>] add_timer+0x18/0x20
[   31.664880]  [<ffffffff81572204>] flow_cache_init+0x224/0x2b0
[   31.664882]  [<ffffffff815f7247>] xfrm_net_init+0x227/0x360

Quoted 9 lines

[   31.664884]  [<ffffffff815f7171>] ? xfrm_net_init+0x151/0x360
[   31.664886]  [<ffffffff81553131>] ops_init+0x41/0x150
[   31.664888]  [<ffffffff815532b3>] setup_net+0x73/0x110
[   31.664890]  [<ffffffff815537f2>] copy_net_ns+0x72/0x100
[   31.664892]  [<ffffffff81072619>] create_new_namespaces+0xf9/0x190
[   31.664894]  [<ffffffff81072891>] unshare_nsproxy_namespaces+0x61/0xa0
[   31.664895]  [<ffffffff81049949>] SyS_unshare+0x159/0x270
[   31.664897]  [<ffffffff81638092>] system_call_fastpath+0x16/0x1b

I was unable to reproduce this here, but it looks like the flowcache namespace changes are still not complete. We leak an active timer and all the allocated resources when we exit a namespace.

Could you please try the patch below?

Also, please send your config if the patch does not fix your problem.

Quoted 13 lines · Jakub Kiciński

Quoted 13 lines

I bisected the other issue to be caused/uncovered by:

commit 1a1ccc96abb2ed9b8fbb71018e64b97324caef53
Author: Steffen Klassert<[email protected]>
Date: Wed Feb 19 10:07:34 2014 +0100

xfrm: Remove caching of xfrm_policy_sk_bundles

We currently cache socket policy bundles at xfrm_policy_sk_bundles. These cached bundles are never used. Instead we create and cache a new one whenever xfrm_lookup() is called on a socket policy.

Quoted 5 lines

Quoted 5 lines

Most protocols cache the used routes to the socket, so let's remove the unused caching of socket policy bundles in xfrm.

Signed-off-by Steffen Klassert<[email protected]>

This patch should affect only on the usage of IPsec socket policies. Do you use socket policies, or do you use IPsec at all?

Quoted 25 lines

Quoted 25 lines

Machine freezes after FLOW_HASH_RND_PERIOD (default 10 minutes). Now get this warning during boot:

 [   31.664820] ------------[ cut here ]------------
 [   31.664824] WARNING: CPU: 2 PID: 3560 at /home/kuba/Development/Linux/net-next/lib/list_debug.c:33 __list_add+0xac/0xc0()
 [   31.664826] list_add corruption. prev->next should be next (ffff880224579598), but was           (null). (prev=ffff8802106140e8).
 [   31.664827] Modules linked in: xt_CHECKSUM tun bridge stp llc ccm xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ftdi_sio arc4 rt2800pci rt2800mmio rt2800lib crc_ccitt eeprom_93cx6 rt2x00pci kvm_amd rt2x00mmio rt2x00lib mac80211 kvm snd_ca0106 cfg80211 e1000e snd_ac97_codec ac97_bus microcode serio_raw ptp i2c_piix4 k10temp acpi_cpufreq pps_core wmi r8169 mii rfkill nfsd auth_rpcgss nfs_acl lockd binfmt_misc sunrpc usb_storage radeon drm_kms_helper ttm
 [   31.664855] CPU: 2 PID: 3560 Comm: (t-daemon) Not tainted 3.14.0-rc2-1a1ccc96abb2ed9b8fbb71018e64b97324caef53+ #11
 [   31.664856] Hardware name: Gigabyte Technology Co., Ltd. GA-MA790XT-UD4P/GA-MA790XT-UD4P, BIOS F9b 08/17/2012
 [   31.664857]  0000000000000009 ffff8802242e7c70 ffffffff81627878 ffff8802242e7cb8
 [   31.664859]  ffff8802242e7ca8 ffffffff8104a28d ffff880210610ea8 ffff880224579598
 [   31.664861]  ffff8802106140e8 ffff880224578000 0000000000000000 ffff8802242e7d08
 [   31.664863] Call Trace:
 [   31.664865]  [<ffffffff81627878>] dump_stack+0x4d/0x66
 [   31.664867]  [<ffffffff8104a28d>] warn_slowpath_common+0x7d/0xa0
 [   31.664869]  [<ffffffff8104a2fc>] warn_slowpath_fmt+0x4c/0x50
 [   31.664871]  [<ffffffff812fdd8c>] __list_add+0xac/0xc0
 [   31.664873]  [<ffffffff81055d33>] __internal_add_timer+0x113/0x130
 [   31.664875]  [<ffffffff81055f47>] internal_add_timer+0x17/0x40
 [   31.664876]  [<ffffffff810587b2>] mod_timer+0x102/0x230
 [   31.664878]  [<ffffffff810588f8>] add_timer+0x18/0x20
 [   31.664880]  [<ffffffff81572204>] flow_cache_init+0x224/0x2b0
 [   31.664882]  [<ffffffff815f7247>] xfrm_net_init+0x227/0x360
 [   31.664884]  [<ffffffff815f7171>] ? xfrm_net_init+0x151/0x360

Quoted 8 lines

Quoted 8 lines

 [   31.664886]  [<ffffffff81553131>] ops_init+0x41/0x150
 [   31.664888]  [<ffffffff815532b3>] setup_net+0x73/0x110
 [   31.664890]  [<ffffffff815537f2>] copy_net_ns+0x72/0x100
 [   31.664892]  [<ffffffff81072619>] create_new_namespaces+0xf9/0x190
 [   31.664894]  [<ffffffff81072891>] unshare_nsproxy_namespaces+0x61/0xa0
 [   31.664895]  [<ffffffff81049949>] SyS_unshare+0x159/0x270
 [   31.664897]  [<ffffffff81638092>] system_call_fastpath+0x16/0x1b

I was unable to reproduce this here, but it looks like the flowcache namespace changes are still not complete. We leak an active timer and all the allocated resources when we exit a namespace.

Could you please try the patch below?