Linux 2.6.25.4 task_struct leak
[1.] One line summary of the problem:
Linux 2.6.25.4 x86_64 task_struct leak on host when running UML 2.6.23.16 compiled for i386
[2.] Full description of the problem/report:
I'm seeing a massive task_struct leak on vanilla Linux 2.6.25.4 x86_64 (64-bit) running User Mode Linux 2.6.23.16 kernels compiled for i386 (32-bit). It seems a task_struct leaks on the HOST for every process that has been created and destroyed by an UML guest. The task_structs remain allocated on the host even after the UML guests have been shut down completely.
Other 32-bit applications do NOT leak task_structs.
Also there is no task_struct leak when running the same 32-bit UML guests on a Linux 2.6.25.4 i386 (32-bit) host kernel.
[3.] Keywords (i.e., modules, networking, kernel):task_struct leak, UML
[4.] Kernel information
[4.1.] Kernel version (from /proc/version):
Linux version 2.6.25.4 (tek@tek01) (gcc version 3.3.6 (Debian 1:3.3.6-15)) #2 SMP PREEMPT Sat May 24 21:46:37 CEST 2008
[4.2.] Kernel .config file:
Config attached!
[5.] Most recent kernel version which did not have the bug:
2.6.23.17 does not leak task_structs.
2.6.24 - 2.6.25.3 has not been tested
Running 2.6.23.16 i386 UML on a 2.6.25.4 i386 host does NOT leak task_structs.
Running 64-bit UML guests has not been tested.
[6.] Output of Oops.. message (if applicable) with symbolic information
resolved (see Documentation/oops-tracing.txt)
No Ooops!
[7.] A small shell script or example program which triggers the
problem (if possible)
Start 32-bit UML on 64-bit host, then shut it down again. Compare /proc/slabinfo with number of running tasks/threads.
Before running UML guests, number of active task_structs in /proc/slabinfo matches number of running tasks/threads on host. After running and shutting down UML guest differ. I had >500000 active task_structs in /proc/slabinfo with less than 500 tasks/threads running on the host.
[8.] Environment
[8.1.] Software (add the output of the ver_linux script here) Linux tek01 2.6.25.4 #2 SMP PREEMPT Sat May 24 21:46:37 CEST 2008 x86_64 GNU/Linux
Gnu C 4.1.2 Gnu make 3.81 binutils 2.17 util-linux 2.12r mount 2.12r module-init-tools 3.3-pre2 e2fsprogs 1.40-WIP reiserfsprogs 3.6.19 xfsprogs 2.8.11 Linux C Library 2.3.6 Dynamic linker (ldd) 2.3.6 Procps 3.2.7 Net-tools 1.60 Console-tools 0.2.3 oprofile 0.9.2 Sh-utils 5.97 udev 105 Modules Loaded nvidia rfcomm l2cap bluetooth nfs lockd nfs_acl
sunrpc af_packet ppdev lp nf_conntrack_ipv6 ip6t_REJECT ip6t_LOG ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_state ipt_REJECT ipt_LOG iptable_raw iptable_mangle iptable_nat iptable_filter dm_crypt crypto_blkcipher dm_snapshot dm_mirror dm_mod deadline_iosched fuse cryptoloop loop tun zaphfc zaptel crc_ccitt powernow_k8 freq_table nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack ip_tables x_tables w83627hf hwmon_vid eeprom i2c_viapro sg sr_mod sbp2 cx88_blackbird cx2341x usbhid snd_via82xx wm8775 gameport snd_mpu401_uart tuner tea5767 tda8290 tda18271 tda827x tuner_xc2028 xc5000 snd_via82xx_modem snd_ac97_codec ac97_bus firmware_class snd_seq_oss tda9887 tuner_simple mt20xx tea5761 snd_seq_midi cx88_alsa snd_pcm_oss snd_mixer_oss snd_rawmidi snd_seq_midi_event snd_pcm snd_seq firewire_ohci firewire_core cx8802 cx8800 cx88xx ir_common compat_ioctl32 videodev v4l1_compat hisax snd_timer k8temp snd_seq_device i2c_algo_bit tveeprom crc_itu_t isdn snd_page_alloc v4l2_common i2c_core psmouse rtc snd hwmon ehci_hcd parport_pc parport serio_raw pcspkr uhci_hcd soundcore btcx_risc videobuf_dma_sg videobuf_core usbcore slhc skge ohci1394 ieee1394 thermal button processor evdev
[8.2.] Processor information (from /proc/cpuinfo):
processor : 0vendor_id : AuthenticAMD
cpu family : 15
model : 35
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+
stepping : 2
cpu MHz : 2200.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fpu : yesfpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni lahf_lm cmp_legacy
bogomips : 4409.04
TLB size : 1024 4K pages
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp
processor : 1vendor_id : AuthenticAMD
cpu family : 15
model : 35
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+
stepping : 2
cpu MHz : 2200.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fpu : yesfpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni lahf_lm cmp_legacy
bogomips : 4405.71
TLB size : 1024 4K pages
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp
[8.3.] Module information (from /proc/modules):
Not relevant!
[8.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem) Not relevant!
[8.5.] PCI information ('lspci -vvv' as root)
Not relevant!
[8.6.] SCSI information (from /proc/scsi/scsi)
Not relevant!
[8.7.] Other information that might be relevant to the problem
(please look in /proc and include all information that you think to be relevant):
[X.] Other notes, patches, fixes, workarounds:
Probably some kind of 64-bit kernel running 32-bit user-space applications, that is only triggered by running UML.
Regards,
Thorsten
Signature · 3 lines
___ | | / E-Mail: [email protected] |horsten |/\nabe WWW: http://linux.thorsten-knabe.de
(Cc: Jeff)
Quoted 3 lines · * Thorsten Knabe ([email protected]) wrote:
[5.] Most recent kernel version which did not have the bug:
2.6.23.17 does not leaktask_structs.
2.6.24 - 2.6.25.3 has not been tested
Would you be able bisect this (or even just narrow it down a bit)?
Quoted 3 lines
Running 2.6.23.16 i386 UML on a 2.6.25.4 i386 host does NOT leak
task_structs.
Running 64-bit UML guests has not been tested.
Quoted 4 lines · Chris Wright
Quoted 3 lines · * Thorsten Knabe ([email protected]) wrote:
[5.] Most recent kernel version which did not have the bug:
2.6.23.17 does not leaktask_structs.
2.6.24 - 2.6.25.3 has not been tested
Has this been seen on non-UML?
Jeff
Signature · 1 line
Work email - jdike at linux dot intel dot com
Quoted 7 lines · Jeff Dike
Quoted 4 lines · Chris Wright
Quoted 3 lines · * Thorsten Knabe ([email protected]) wrote:
[5.] Most recent kernel version which did not have the bug:
2.6.23.17 does not leaktask_structs.
2.6.24 - 2.6.25.3 has not been testedHas this been seen on non-UML?
Hello Jeff.
I'm not sure, if I understood your question correctly. The task_struct leak is on the host running a vanilla Linux 2.6.25.4 x86_64 on real hardware and not within the UML instances. I'll try to describe the situation more precisely:
The host system is a vanilla Linux 2.6.25.4 x86_64 running on real hardware with a Debian Etch x86_64 user space. However I've created a Debian Etch i386 (32bit) userspace chroot environment on the system, which is used, among other things, to compile the Linux 2.6.23.17 UML kernels and start various UML instances.
I can start other 32-bit applications, for example compile an UML kernel, within the chroot without leaking task_structs, but as soon as I start an UML instance, I see leaked task_structs. Starting and immediately shutting down an UML instacne leaks approximately 2000 task_structs. The number of leaked task_structs on the host seems to be equal to the number of processes that have been created (and destroyed) within the UML instances.
The host system was upgraded from 2.6.23.17 x86_64 which did not leak task_structs to 2.6.25.4 x86_64. Also running the same UML kernels and filesystems on my laptop with a vanilla Linux 2.6.25.4 i386 kernel does not leak task_structs. The version of the UML kernel (tested 2.6.23.17 and 2.6.25.4) does not seem to matter.
As far as I understand the UML code in the kernel, an UML kernel uses some unusual clone() flags when creating new processes, which are seldom used by other applications and could be related to the bug.
If there is no obvious bug in the code, I'll try to bisect that beast next weekend as suggested by Chris.
Regards
Thorsten
Signature · 3 lines
___ | | / E-Mail: [email protected] |horsten |/\nabe WWW: http://linux.thorsten-knabe.de
Quoted 7 lines · Thorsten Knabe
I can start other 32-bit applications, for example compile an UML kernel, within the chroot without leaking
task_structs, but as soon as I start an UML instance, I see leakedtask_structs. Starting and immediately shutting down an UML instacne leaks approximately 2000task_structs. The number of leakedtask_structson the host seems to be equal to the number of processes that have been created (and destroyed) within the UML instances.
I misunderstood - I thought you were seeing a task_struct leak within UML rather than a leak on the host elicited by UML.
Quoted 3 lines
As far as I understand the UML code in the kernel, an UML kernel uses some unusual
clone()flags when creating new processes, which are seldom used by other applications and could be related to the bug.
Yes, it does. I don't see the flags causing a leak, though. What might be more likely (although I really have no idea) is ptrace. Possibly a reference is held when it should have been dropped. This might also show up with strace or gdb.
Jeff
Signature · 1 line
Work email - jdike at linux dot intel dot com
Quoted 2 lines · * Jeff Dike ([email protected]) wrote:
Yes, it does. I don't see the flags causing a leak, though. What might be more likely (although I really have no idea) is ptrace.
Yeah, ptrace was my guess as well, and I was hoping you'd have an idea ;-)
Quoted 2 lines
Possibly a reference is held when it should have been dropped. This might also show up with strace or gdb.
Quoted 11 lines · Jeff Dike
I misunderstood - I thought you were seeing a
task_structleak within UML rather than a leak on the host elicited by UML.Quoted 3 lines
As far as I understand the UML code in the kernel, an UML kernel uses some unusual
clone()flags when creating new processes, which are seldom used by other applications and could be related to the bug.Yes, it does. I don't see the flags causing a leak, though. What might be more likely (although I really have no idea) is ptrace. Possibly a reference is held when it should have been dropped. This might also show up with strace or gdb.
Hello Jeff.
Your assumption about ptrace causing the task_struct leak seems to be right. I bisected the problem down to a few commits using the repository at git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git.
Commit b7b71725fb9584454bfe5f231223bd63421798fb is the last known commit that does not leak task_structs, whereas commit a97f52e67890fda6b373c1c1895ff1c1c69b36c8 is leaking task_structs. Revisions in between do not even compile.
Also I had to apply the changes from commit f9cb02b0be4de3c51edfdd701754e13d9a2d20d6 to most of the kernels I have tested, otherwise the UML process would crash on startup.
HTH
Thorsten
Signature · 3 lines
___ | | / E-Mail: [email protected] |horsten |/\nabe WWW: http://linux.thorsten-knabe.de
Quoted 7 lines · Thorsten Knabe
Hello Jeff.
Your assumption about ptrace causing the
task_structleak seems to be right. I bisected the problem down to a few commits using the repository atgit://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git.
Quoted 7 lines
Commit
b7b71725fb9584454bfe5f231223bd63421798fbis the last known commit that does not leaktask_structs, whereas commita97f52e67890fda6b373c1c1895ff1c1c69b36c8is leakingtask_structs. Revisions in between do not even compile.
Also I had to apply the changes from commitf9cb02b0be4de3c51edfdd701754e13d9a2d20d6to most of the kernels I have tested, otherwise the UML process would crash on startup.
Add Cc to Roland McGrath <[email protected]>.
Signature · 2 lines
Hi, I'm a .signature virus, please copy/paste me to help me spread all over the world.
Oops, mea culpa. I added the bug in commit 5a4646a4efed8c835f76c3b88f3155f6ab5b8d9b.
It's already gone for 2.6.26 with the compat_arch_ptrace cleanup.
-stable patch to follow.
Thanks,
Roland
PATCH 1/1 x86_64 ptrace: fix sys32_ptrace task_struct leak
In reply to WANG CongCommit 5a4646a4efed8c835f76c3b88f3155f6ab5b8d9b introduced a leak of task_struct refs into sys32_ptrace. This bug has already gone away in for 2.6.26 in commit 562b80bafffaf42a6d916b0a2ee3d684220a1c10.
--- arch/x86/kernel/ptrace.c | 45 ++++++++++++++++++++++++++------------------- 1 files changed, 26 insertions(+), 19 deletions(-)
arch/x86/kernel/ptrace.c +26 -19
@@ -1309,42 +1309,49 @@ asmlinkage long sys32_ptrace(long request, u32 pid, u32 addr, u32 data)break ;case PTRACE_GETREGS :/* Get all gp regs from the child. */ -return copy_regset_to_user (child, &user_x86_32_view, -REGSET_GENERAL , -0 ,sizeof (struct user_regs_struct32 ), - datap); + ret =copy_regset_to_user (child, &user_x86_32_view, +REGSET_GENERAL , +0 ,sizeof (struct user_regs_struct32 ), + datap); +break ;case PTRACE_SETREGS :/* Set all gp regs in the child. */ -return copy_regset_from_user (child, &user_x86_32_view, -REGSET_GENERAL ,0 , -sizeof (struct user_regs_struct32 ), - datap); + ret =copy_regset_from_user (child, &user_x86_32_view, +REGSET_GENERAL ,0 , +sizeof (struct user_regs_struct32 ), + datap); +break ;case PTRACE_GETFPREGS :/* Get the child FPU state. */ -return copy_regset_to_user (child, &user_x86_32_view, -REGSET_FP ,0 , -sizeof (struct user_i387_ia32_struct ), - datap); + ret =copy_regset_to_user (child, &user_x86_32_view, +REGSET_FP ,0 , +sizeof (struct user_i387_ia32_struct ), + datap); +break ;case PTRACE_SETFPREGS :/* Set the child FPU state. */ -return copy_regset_from_user ( + ret =copy_regset_from_user ( child, &user_x86_32_view,REGSET_FP ,0 ,sizeof (struct user_i387_ia32_struct ), datap); +break ;case PTRACE_GETFPXREGS :/* Get the child extended FPU state. */ -return copy_regset_to_user (child, &user_x86_32_view, -REGSET_XFP ,0 , -sizeof (struct user32_fxsr_struct ), - datap); + ret =copy_regset_to_user (child, &user_x86_32_view, +REGSET_XFP ,0 , +sizeof (struct user32_fxsr_struct ), + datap); +break ;case PTRACE_SETFPXREGS :/* Set the child extended FPU state. */ -return copy_regset_from_user (child, &user_x86_32_view, + ret =copy_regset_from_user (child, &user_x86_32_view,REGSET_XFP ,0 ,sizeof (struct user32_fxsr_struct ), datap); +break ;default : -return compat_ptrace_request (child, request, addr, data); + ret =compat_ptrace_request (child, request, addr, data); +break ; } out:
Re: [PATCH stable-2.6.25] x86_64 ptrace: fix sys32_ptrace task_struct leak
Quoted 5 lines · Roland McGrath
Commit
5a4646a4efed8c835f76c3b88f3155f6ab5b8d9bintroduced a leak oftask_structrefs intosys32_ptrace. This bug has already gone away in for 2.6.26 in commit562b80bafffaf42a6d916b0a2ee3d684220a1c10.Signed-off-by Roland McGrath <[email protected]>
Ingo