in 3.18-rc1: ppp crashes kernel
Subsystems (3)
Hi all,
since 3.18-rc1, setting up a PPP interface kills my kernel with
[ 163.433251] PPP generic driver version 2.4.2 [ 164.452474] ------------[ cut here ]------------ [ 164.453327] kernel BUG at ../mm/vmalloc.c:1316! [ 164.453327] invalid opcode: 0000 [#1] PREEMPT SMP [ 164.453327] Modules linked in: ppp_async crc_ccitt ppp_generic slhc af_packet xfs libcrc32c coretemp kvm_intel snd_hda_codec_conexant iTCO_wdt snd_hda_codec_generic iTCO_vendor_support uvcvideo snd_hda_intel snd_hda_controller mac80211 videobuf2_vmalloc snd_hda_codec kvm e1000e videobuf2_memops cfg80211 videobuf2_core v4l2_common snd_hwdep i2c_i801 videodev snd_pcm pcspkr thinkpad_acpi serio_raw wmi lpc_ich snd_timer thermal snd rfkill mfd_core tpm_tis shpchp mei_me soundcore ptp mei pps_core acpi_cpufreq tpm battery processor ac dm_mod btrfs xor raid6_pq i915 i2c_algo_bit drm_kms_helper drm video button sg [ 164.453327] CPU: 0 PID: 6927 Comm: pppd Not tainted 3.18.0-rc3-3.ge706e91-desktop #1 [ 164.453327] Hardware name: LENOVO 7470E36/7470E36, BIOS 6DET61WW (3.11 ) 11/10/2009
This is easy to reproduce with:
linux:~ # cat bin/crashme.sh
----
#!/bin/bash -x
pppd local pty "netcat -l 1234" &
sleep 1
pppd local pty "netcat localhost 1234" &
sleep 1
----
3.17 works fine.
I bisected the issue multiple times and always arrived at
# first bad commit: [d6dd50e07c5bec00db2005969b1a01f8ca3d25ef] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
which is a merge commit unfortunately.
The BUG encountered above is in:
1309 static struct vm_struct *__get_vm_area_node(unsigned long size,
1310 unsigned long align, unsigned long flags, unsigned long start,
1311 unsigned long end, int node, gfp_t gfp_mask, const void *caller)
1312 {
1313 struct vmap_area *va;
1314 struct vm_struct *area;
1315
1316 BUG_ON(in_interrupt());
1317 if (flags & VM_IOREMAP)
1318 align = 1ul << clamp(fls(size), PAGE_SHIFT, IOREMAP_MAX_ORDER);
1319
the call trace is:
[ 164.453327] Call Trace: [ 164.453327] [<ffffffff811974bd>] __vmalloc_node_range+0x6d/0x290 [ 164.453327] [<ffffffff8119771e>] __vmalloc+0x3e/0x50 [ 164.453327] [<ffffffff81146950>] bpf_prog_alloc+0x30/0xa0 [ 164.453327] [<ffffffff8157b716>] bpf_prog_create+0x46/0xb0 [ 164.453327] [<ffffffffa07ecb90>] ppp_ioctl+0x420/0xe9a [ppp_generic] [ 164.453327] [<ffffffff811df1c7>] do_vfs_ioctl+0x2e7/0x4c0 [ 164.453327] [<ffffffff811df421>] SyS_ioctl+0x81/0xa0 [ 164.453327] [<ffffffff8165ee2d>] system_call_fastpath+0x16/0x1b [ 164.453327] [<00007f4502d87397>] 0x7f4502d87397
I have a crashdump of the kernel, but given this is easily reproducible, I doubt that I need to send this to anyone :-)
Best regards,
Stefan
Signature · 6 lines
Stefan Seyfried Linux Consultant & Developer -- GPG Key: 0x731B665B B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Quoted 5 lines · Stefan Seyfried
I bisected the issue multiple times and always arrived at
# first bad commit: [d6dd50e07c5bec00db2005969b1a01f8ca3d25ef] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tipwhich is a merge commit unfortunately.
That merge commit actually does add some code:
git show d6dd50e07c5bec00db2005969b1a01f8ca3d25ef
[...]
init/main.c +0 -0
@@@ -583,6 -585,6 +583,7 @@@ asmlinkage __visible void __init start_early_irq_init ();init_IRQ ();tick_init (); ++rcu_init_nohz ();init_timers ();hrtimers_init ();softirq_init ();
Wild guess: is CONFIG_RCU_NOCB_CPU perhaps set in your v3.18-rc3 .config?
Paul Bolle
Hi Paul,
Am 07.11.2014 um 12:53 schrieb Paul Bolle:
Quoted 24 lines · Stefan Seyfried
Quoted 5 lines
I bisected the issue multiple times and always arrived at
# first bad commit: [d6dd50e07c5bec00db2005969b1a01f8ca3d25ef] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tipwhich is a merge commit unfortunately.
That merge commit actually does add some code:
git show
d6dd50e07c5bec00db2005969b1a01f8ca3d25ef
[...]init/main.c +1
-0@@@ -583,6 -585,6 +583,7 @@@ asmlinkage __visible void __init start_early_irq_init ();init_IRQ ();tick_init (); ++rcu_init_nohz ();init_timers ();hrtimers_init ();softirq_init ();
Quoted 2 lines
Wild guess: is
CONFIG_RCU_NOCB_CPUperhaps set in your v3.18-rc3 .config?
Yes it is:
tux@linux:~> zgrep CONFIG_RCU_NOCB_CPU /proc/config.gz CONFIG_RCU_NOCB_CPU=y
# CONFIG_RCU_NOCB_CPU_NONE is not set
# CONFIG_RCU_NOCB_CPU_ZERO is not set CONFIG_RCU_NOCB_CPU_ALL=y
And I'll try without it, but looking at the backtrace and the actual BUG_ON() in the code, I cannot really believe it is the real problems.
But I'll try with the config changed and with the above line removed.
Thanks,
Stefan
Signature · 6 lines
Stefan Seyfried Linux Consultant & Developer -- GPG Key: 0x731B665B B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Am 07.11.2014 um 12:56 schrieb Stefan Seyfried:
Quoted 13 lines
Hi Paul,
Am 07.11.2014 um 12:53 schrieb Paul Bolle:
Quoted 2 lines
Wild guess: is
CONFIG_RCU_NOCB_CPUperhaps set in your v3.18-rc3 .config?Yes it is:
tux@linux:~> zgrepCONFIG_RCU_NOCB_CPU/proc/config.gzCONFIG_RCU_NOCB_CPU=y
#CONFIG_RCU_NOCB_CPU_NONEis not set
#CONFIG_RCU_NOCB_CPU_ZEROis not setCONFIG_RCU_NOCB_CPU_ALL=y
Quoted 4 lines
And I'll try without it, but looking at the backtrace and the actual
BUG_ON()in the code, I cannot really believe it is the real problems.But I'll try with the config changed and with the above line removed.
JFTR, this did not help:
tux@linux:~/linux> zgrep CONFIG_RCU_NOCB_CPU /proc/config.gz # CONFIG_RCU_NOCB_CPU is not set
neither did:
init/main.c +1 -1
@@ -583,7 +583,7 @@ asmlinkage __visible void __init start_kernel(void)early_irq_init ();init_IRQ ();tick_init (); -rcu_init_nohz (); +// rcu_init_nohz(); init_timers ();hrtimers_init ();softirq_init ();
Signature · 6 lines
Stefan Seyfried Linux Consultant & Developer -- GPG Key: 0x731B665B B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Quoted 9 lines · Stefan Seyfried
JFTR, this did not help:
tux@linux:~/linux> zgrepCONFIG_RCU_NOCB_CPU/proc/config.gz#CONFIG_RCU_NOCB_CPUis not setneither did:
init/main.c asmlinkage __visible void __init start_kernel(void) · line 589
Quoted 8 lines
init/main.c +1
-1@@ -583,7 +583,7 @@ asmlinkage __visible void __init start_kernel(void)early_irq_init ();init_IRQ ();tick_init (); -rcu_init_nohz (); +// rcu_init_nohz(); init_timers ();hrtimers_init ();softirq_init ();
Thanks. I guess this is not my lucky day. I won't be visiting a casino tonight...
Paul Bolle
At Fri, 07 Nov 2014 12:10:46 +0100,
Quoted 47 lines · Stefan Seyfried
Hi all,
since 3.18-rc1, setting up a PPP interface kills my kernel with
[ 163.433251] PPP generic driver version 2.4.2 [ 164.452474] ------------[ cut here ]------------ [ 164.453327] kernel BUG at ../mm/vmalloc.c:1316! [ 164.453327] invalid opcode: 0000 [#1] PREEMPT SMP [ 164.453327] Modules linked in: ppp_async crc_ccitt ppp_generic slhc af_packet xfs libcrc32c coretemp kvm_intel snd_hda_codec_conexant iTCO_wdt snd_hda_codec_generic iTCO_vendor_support uvcvideo snd_hda_intel snd_hda_controller mac80211 videobuf2_vmalloc snd_hda_codec kvm e1000e videobuf2_memops cfg80211 videobuf2_core v4l2_common snd_hwdep i2c_i801 videodev snd_pcm pcspkr thinkpad_acpi serio_raw wmi lpc_ich snd_timer thermal snd rfkill mfd_core tpm_tis shpchp mei_me soundcore ptp mei pps_core acpi_cpufreq tpm battery processor ac dm_mod btrfs xor raid6_pq i915 i2c_algo_bit drm_kms_helper drm video button sg [ 164.453327] CPU: 0 PID: 6927 Comm: pppd Not tainted 3.18.0-rc3-3.ge706e91-desktop #1 [ 164.453327] Hardware name: LENOVO 7470E36/7470E36, BIOS 6DET61WW (3.11 ) 11/10/2009This is easy to reproduce with:
linux:~ # cat
bin/crashme.sh
----
#!/bin/bash -x
pppd local pty "netcat -l 1234" &
sleep 1
pppd local pty "netcat localhost 1234" &
sleep 1
----3.17 works fine.
I bisected the issue multiple times and always arrived at# first bad commit: [d6dd50e07c5bec00db2005969b1a01f8ca3d25ef] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tipwhich is a merge commit unfortunately.
The BUG encountered above is in:
1309 static struct
vm_struct*__get_vm_area_node(unsigned long size,
1310 unsigned long align, unsigned long flags, unsigned long start,
1311 unsigned long end, int node,gfp_tgfp_mask, const void *caller)
1312 {
1313 structvmap_area*va;
1314 structvm_struct*area;
1315
1316 BUG_ON(in_interrupt());
1317 if (flags &VM_IOREMAP)
1318 align = 1ul <<clamp(fls(size), PAGE_SHIFT, IOREMAP_MAX_ORDER);
1319the call trace is:
Quoted 10 lines
[ 164.453327] Call Trace: [ 164.453327] [<ffffffff811974bd>] __vmalloc_node_range+0x6d/0x290 [ 164.453327] [<ffffffff8119771e>] __vmalloc+0x3e/0x50 [ 164.453327] [<ffffffff81146950>] bpf_prog_alloc+0x30/0xa0 [ 164.453327] [<ffffffff8157b716>] bpf_prog_create+0x46/0xb0 [ 164.453327] [<ffffffffa07ecb90>] ppp_ioctl+0x420/0xe9a [ppp_generic] [ 164.453327] [<ffffffff811df1c7>] do_vfs_ioctl+0x2e7/0x4c0 [ 164.453327] [<ffffffff811df421>] SyS_ioctl+0x81/0xa0 [ 164.453327] [<ffffffff8165ee2d>] system_call_fastpath+0x16/0x1b [ 164.453327] [<00007f4502d87397>] 0x7f4502d87397
bpf_prog_create() is called inside spin_lock_bh(), and the BUG_ON() hits. Below is a quick fix.
Takashi
-- 8< --
From: Takashi Iwai <[email protected]>
Subject: [PATCH] net: ppp: Don't call bpf_prog_create() in ppp_lock
In ppp_ioctl(), bpf_prog_create() is called inside ppp_lock, which eventually calls vmalloc() and hits BUG_ON() in vmalloc.c. This patch works around the problem by moving the allocation outside the lock.
--- drivers/net/ppp/ppp_generic.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-)
drivers/net/ppp/ppp_generic.c +20 -20
@@ -755,23 +755,23 @@ static long ppp_ioctl(struct file *file, unsigned int cmd, unsigned long arg) err =get_filter (argp, &code);if (err >=0 ) { +struct bpf_prog *pass_filter =NULL ;struct sock_fprog_kern fprog = { .len = err, .filter = code, }; -ppp_lock (ppp); -if (ppp->pass_filter) { -bpf_prog_destroy (ppp->pass_filter); - ppp->pass_filter =NULL ; + err =0 ; +if (fprog.filter) + err =bpf_prog_create (&pass_filter, &fprog); +if (!err) { +ppp_lock (ppp); +if (ppp->pass_filter) +bpf_prog_destroy (ppp->pass_filter); + ppp->pass_filter = pass_filter; +ppp_unlock (ppp); } -if (fprog.filter !=NULL ) - err =bpf_prog_create (&ppp->pass_filter, - &fprog); -else - err =0 ;kfree (code); -ppp_unlock (ppp); }break ; }
@@ -781,23 +781,23 @@ static long ppp_ioctl(struct file *file, unsigned int cmd, unsigned long arg) err =get_filter (argp, &code);if (err >=0 ) { +struct bpf_prog *active_filter =NULL ;struct sock_fprog_kern fprog = { .len = err, .filter = code, }; -ppp_lock (ppp); -if (ppp->active_filter) { -bpf_prog_destroy (ppp->active_filter); - ppp->active_filter =NULL ; + err =0 ; +if (fprog.filter) + err =bpf_prog_create (&active_filter, &fprog); +if (!err) { +ppp_lock (ppp); +if (ppp->active_filter) +bpf_prog_destroy (ppp->active_filter); + ppp->active_filter = active_filter; +ppp_unlock (ppp); } -if (fprog.filter !=NULL ) - err =bpf_prog_create (&ppp->active_filter, - &fprog); -else - err =0 ;kfree (code); -ppp_unlock (ppp); }break ; }
Signature · 1 line
2.1.3
Stefan Seyfried
Tested-byHi Takashi,
yes, this no longer crashes. No real-world test yet, but the obvious crash is gone. Thanks!
Am 07.11.2014 um 14:22 schrieb Takashi Iwai:
Quoted 70 lines
At Fri, 07 Nov 2014 12:10:46 +0100,
Quoted 47 lines · Stefan Seyfried
Hi all,
since 3.18-rc1, setting up a PPP interface kills my kernel with
[ 163.433251] PPP generic driver version 2.4.2 [ 164.452474] ------------[ cut here ]------------ [ 164.453327] kernel BUG at ../mm/vmalloc.c:1316! [ 164.453327] invalid opcode: 0000 [#1] PREEMPT SMP [ 164.453327] Modules linked in: ppp_async crc_ccitt ppp_generic slhc af_packet xfs libcrc32c coretemp kvm_intel snd_hda_codec_conexant iTCO_wdt snd_hda_codec_generic iTCO_vendor_support uvcvideo snd_hda_intel snd_hda_controller mac80211 videobuf2_vmalloc snd_hda_codec kvm e1000e videobuf2_memops cfg80211 videobuf2_core v4l2_common snd_hwdep i2c_i801 videodev snd_pcm pcspkr thinkpad_acpi serio_raw wmi lpc_ich snd_timer thermal snd rfkill mfd_core tpm_tis shpchp mei_me soundcore ptp mei pps_core acpi_cpufreq tpm battery processor ac dm_mod btrfs xor raid6_pq i915 i2c_algo_bit drm_kms_helper drm video button sg [ 164.453327] CPU: 0 PID: 6927 Comm: pppd Not tainted 3.18.0-rc3-3.ge706e91-desktop #1 [ 164.453327] Hardware name: LENOVO 7470E36/7470E36, BIOS 6DET61WW (3.11 ) 11/10/2009This is easy to reproduce with:
linux:~ # cat
bin/crashme.sh
----
#!/bin/bash -x
pppd local pty "netcat -l 1234" &
sleep 1
pppd local pty "netcat localhost 1234" &
sleep 1
----3.17 works fine.
I bisected the issue multiple times and always arrived at# first bad commit: [d6dd50e07c5bec00db2005969b1a01f8ca3d25ef] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tipwhich is a merge commit unfortunately.
The BUG encountered above is in:
1309 static struct
vm_struct*__get_vm_area_node(unsigned long size,
1310 unsigned long align, unsigned long flags, unsigned long start,
1311 unsigned long end, int node,gfp_tgfp_mask, const void *caller)
1312 {
1313 structvmap_area*va;
1314 structvm_struct*area;
1315
1316 BUG_ON(in_interrupt());
1317 if (flags &VM_IOREMAP)
1318 align = 1ul <<clamp(fls(size), PAGE_SHIFT, IOREMAP_MAX_ORDER);
1319the call trace is:
Quoted 10 lines
[ 164.453327] Call Trace: [ 164.453327] [<ffffffff811974bd>] __vmalloc_node_range+0x6d/0x290 [ 164.453327] [<ffffffff8119771e>] __vmalloc+0x3e/0x50 [ 164.453327] [<ffffffff81146950>] bpf_prog_alloc+0x30/0xa0 [ 164.453327] [<ffffffff8157b716>] bpf_prog_create+0x46/0xb0 [ 164.453327] [<ffffffffa07ecb90>] ppp_ioctl+0x420/0xe9a [ppp_generic] [ 164.453327] [<ffffffff811df1c7>] do_vfs_ioctl+0x2e7/0x4c0 [ 164.453327] [<ffffffff811df421>] SyS_ioctl+0x81/0xa0 [ 164.453327] [<ffffffff8165ee2d>] system_call_fastpath+0x16/0x1b [ 164.453327] [<00007f4502d87397>] 0x7f4502d87397
bpf_prog_create()is called insidespin_lock_bh(), and theBUG_ON()hits. Below is a quick fix.Takashi
-- 8< --
From: Takashi Iwai <[email protected]>
Subject: [PATCH] net: ppp: Don't callbpf_prog_create()inppp_lock
Quoted 6 lines
In
ppp_ioctl(),bpf_prog_create()is called insideppp_lock, which eventually callsvmalloc()and hitsBUG_ON()invmalloc.c. This patch works around the problem by moving the allocation outside the lock.Reported-by Stefan Seyfried <[email protected]>Signed-off-by Takashi Iwai <[email protected]>
FWIW :-)
Quoted 77 lines
--- drivers/net/ppp/ppp_generic.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-)drivers/net/ppp/ppp_generic.c +20
-20@@ -755,23 +755,23 @@ static long ppp_ioctl(struct file *file, unsigned int cmd, unsigned long arg) err =get_filter (argp, &code);if (err >=0 ) { +struct bpf_prog *pass_filter =NULL ;struct sock_fprog_kern fprog = { .len = err, .filter = code, }; -ppp_lock (ppp); -if (ppp->pass_filter) { -bpf_prog_destroy (ppp->pass_filter); - ppp->pass_filter =NULL ; + err =0 ; +if (fprog.filter) + err =bpf_prog_create (&pass_filter, &fprog); +if (!err) { +ppp_lock (ppp); +if (ppp->pass_filter) +bpf_prog_destroy (ppp->pass_filter); + ppp->pass_filter = pass_filter; +ppp_unlock (ppp); } -if (fprog.filter !=NULL ) - err =bpf_prog_create (&ppp->pass_filter, - &fprog); -else - err =0 ;kfree (code); -ppp_unlock (ppp); }break ; }@@ -781,23 +781,23 @@ static long ppp_ioctl(struct file *file, unsigned int cmd, unsigned long arg) err =get_filter (argp, &code);if (err >=0 ) { +struct bpf_prog *active_filter =NULL ;struct sock_fprog_kern fprog = { .len = err, .filter = code, }; -ppp_lock (ppp); -if (ppp->active_filter) { -bpf_prog_destroy (ppp->active_filter); - ppp->active_filter =NULL ; + err =0 ; +if (fprog.filter) + err =bpf_prog_create (&active_filter, &fprog); +if (!err) { +ppp_lock (ppp); +if (ppp->active_filter) +bpf_prog_destroy (ppp->active_filter); + ppp->active_filter = active_filter; +ppp_unlock (ppp); } -if (fprog.filter !=NULL ) - err =bpf_prog_create (&ppp->active_filter, - &fprog); -else - err =0 ;kfree (code); -ppp_unlock (ppp); }break ; }
Signature · 6 lines
Stefan Seyfried Linux Consultant & Developer -- GPG Key: 0x731B665B B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537