Skip to contentSkip to conversation

mips qemu test failures in -next due to "kthread: Fix use-after-free if kthread fork fails"

DISCUSSION Guenter Roeck 2 participants 6 messages May 27, 2017 – May 28, 2017 MIPS

lore
Share & tools
lore
#0 lore raw

Hi,

my qemu testis of mips images are failing in -next. Symptom is a hang during boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.

I bisected the problem in next-20170526. It points to commit 4d6501dce079c ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch fixes the problem.

Bisect log is attached.

Guenter

---
# bad: [494a19b0aa3c905b6c5221f49058526896b06a23] Add linux-next specific files for 20170526 # good: [08332893e37af6ae779367e78e444f8f9571511d] Linux 4.12-rc2
git bisect start 'HEAD' 'v4.12-rc2'
# good: [61f91dc3a7c15bdc06e746b0fa30a2b125bd41b6] Merge remote-tracking branch 'drm/drm-next' git bisect good 61f91dc3a7c15bdc06e746b0fa30a2b125bd41b6
# bad: [854cfdd6fc9298018f6625d5e495405b2687b13c] Merge remote-tracking branch 'tip/auto-latest' git bisect bad 854cfdd6fc9298018f6625d5e495405b2687b13c
# good: [bb4b4365c7a19d593d190ec6a380a7835b20e3fc] Merge remote-tracking branch 'sound/for-next' git bisect good bb4b4365c7a19d593d190ec6a380a7835b20e3fc
# good: [830aade0f6121bf1bfa6504aa588b23469694803] Merge remote-tracking branch 'battery/for-next' git bisect good 830aade0f6121bf1bfa6504aa588b23469694803
# bad: [0f90e8b13c713120ae0d2485533e5acf956f8321] Merge branch 'x86/boot'
git bisect bad 0f90e8b13c713120ae0d2485533e5acf956f8321
# good: [d04e31a23c3c828456cb5613f391ce4ac4e5765f] cpufreq/pasemi: Adjust system_state check git bisect good d04e31a23c3c828456cb5613f391ce4ac4e5765f
# bad: [a451941cac2792181be56857e70233cbb9957cd2] Merge branch 'perf/urgent'
git bisect bad a451941cac2792181be56857e70233cbb9957cd2
# bad: [2c510368365cd0ce0c48c3f97a7048dc07ccd94a] Merge branch 'locking/urgent'
git bisect bad 2c510368365cd0ce0c48c3f97a7048dc07ccd94a
# good: [b9de5068b2d7c22de1bb89d62a7119a6b8dff88f] x86/ioapic: Remove unused IO_APIC_irq_trigger() function git bisect good b9de5068b2d7c22de1bb89d62a7119a6b8dff88f
# bad: [79824ccf1691d0b8d4ef83d5d212ba0b31d8081e] Merge branch 'core/urgent'
git bisect bad 79824ccf1691d0b8d4ef83d5d212ba0b31d8081e
# bad: [4d6501dce079c1eb6bf0b1d8f528a5e81770109e] kthread: Fix use-after-free if kthread fork fails git bisect bad 4d6501dce079c1eb6bf0b1d8f528a5e81770109e

# first bad commit: [4d6501dce079c1eb6bf0b1d8f528a5e81770109e] kthread: Fix use-after-free if kthread fork fails
#1 lore raw
Quoted 10 lines · Guenter Roeck

Hi,

my qemu testis of mips images are failing in -next. Symptom is a hang during boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.

I bisected the problem in next-20170526. It points to commit 4d6501dce079c ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch fixes the problem.

Bisect log is attached.

Hi,

Thanks for the report and sorry for the breakage :-/

I can't immediately spot what's going wrong, but I am able to reproduce it on mips so I will try to debug.

Are you sure it's this commit, though? I checked out linus/master and I get a boot hang even after reverting it.

Vegard

#2 lore raw

+cc Linus, as mainline is now affected.

Quoted 14 lines · Vegard Nossum
Quoted 10 lines · Guenter Roeck

Hi,

my qemu testis of mips images are failing in -next. Symptom is a hang during boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.

I bisected the problem in next-20170526. It points to commit 4d6501dce079c ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch fixes the problem.

Bisect log is attached.

Hi,

Quoted 8 lines

Thanks for the report and sorry for the breakage :-/

I can't immediately spot what's going wrong, but I am able to reproduce it on mips so I will try to debug.

Are you sure it's this commit, though? I checked out linus/master and I get a boot hang even after reverting it.

Reverting the patch fixed the problem for me in -next. The last mainline build on my test system was with v4.12-rc2-228-gc86daad, which was fine except for the known build breakages (frv, hexagon) and the known tracebacks in crisv32

	WARNING: CPU: 0 PID: 1 at drivers/mtd/mtdcore.c:734 mtd_device_parse_register+0x17a/0x1aa
and ppc
	refcount_t: underflow; use-after-free.
	WARNING: CPU: 0 PID: 1 at lib/refcount.c:184 refcount_sub_and_test+0x84/0xe0

It may well be that something else is broken in mainline. The build of v4.12-rc2-284-g249f1ef on my builders isn't complete yet. We'll see in a couple of hours if there is other damage. I'll bisect later today if necessary.

Guenter

#3 lore raw
In reply to Vegard Nossum
Quoted 14 lines · Vegard Nossum
Quoted 10 lines · Guenter Roeck

Hi,

my qemu testis of mips images are failing in -next. Symptom is a hang during boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.

I bisected the problem in next-20170526. It points to commit 4d6501dce079c ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch fixes the problem.

Bisect log is attached.

Hi,

Quoted 7 lines

Thanks for the report and sorry for the breakage :-/

I can't immediately spot what's going wrong, but I am able to reproduce it on mips so I will try to debug.

Are you sure it's this commit, though? I checked out linus/master and I get a boot hang even after reverting it.

My mistake; I ran into a different bug which made me think it was hanging when it wasn't.

However, I think I found the problem; does this patch fix it for you too?

arch/mips/kernel/process.c +0 -1
index 918d4c73e951..5351e1f3950d 100644
@@ -120,7 +120,6 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long usp,
        struct thread_info *ti = task_thread_info(p);
        struct pt_regs *childregs, *regs = current_pt_regs();
        unsigned long childksp;
-       p->set_child_tid = p->clear_child_tid = NULL;
 
        childksp = (unsigned long)task_stack_page(p) + THREAD_SIZE - 32;
 

The problem is that when we moved the p->{set,clear}_child_tid assignments inside copy_process(), the above assignments would clear them out. The assignments only exist on mips and openrisc (which would need the same patch), which explains why I didn't see it in my x86 testing. I think the patch above should be safe given that we're now always setting these fields in copy_process() at an appropriate moment.

Looks like those assignments came from commit 3c37026d43c47 ("NPTL, round one."); Ralf?

Oleg?

Vegard

#4 lore raw
Quoted 20 lines · Vegard Nossum
Quoted 19 lines · Vegard Nossum
Quoted 10 lines · Guenter Roeck

Hi,

my qemu testis of mips images are failing in -next. Symptom is a hang during boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.

I bisected the problem in next-20170526. It points to commit 4d6501dce079c ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch fixes the problem.

Bisect log is attached.

Hi,

Thanks for the report and sorry for the breakage :-/

I can't immediately spot what's going wrong, but I am able to reproduce it on mips so I will try to debug.

Quoted 8 lines
Quoted 2 lines

Are you sure it's this commit, though? I checked out linus/master and I get a boot hang even after reverting it.

My mistake; I ran into a different bug which made me think it was hanging when it wasn't.

However, I think I found the problem; does this patch fix it for you too?

I'll give it a try.

I tried my qemu emulation on mainline after reverting your patch; it passed. What kernel configuration and qemu command line did you use in your test ?

Quoted 12 lines
arch/mips/kernel/process.c +0 -1
index 918d4c73e951..5351e1f3950d 100644
@@ -120,7 +120,6 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long usp,
         struct thread_info *ti = task_thread_info(p);
         struct pt_regs *childregs, *regs = current_pt_regs();
         unsigned long childksp;
-       p->set_child_tid = p->clear_child_tid = NULL;
 
         childksp = (unsigned long)task_stack_page(p) + THREAD_SIZE - 32;
 
Quoted 4 lines

The problem is that when we moved the p->{set,clear}_child_tid assignments inside copy_process(), the above assignments would clear them out. The assignments only exist on mips and openrisc (which would need the same patch), which explains why I didn't see it in my x86

Interestingly, my openrisc test passed. Of course, that is just a boot test, so it may not hit the problem.

Thanks,
Guenter

Quoted 11 lines

testing. I think the patch above should be safe given that we're now always setting these fields in copy_process() at an appropriate moment.

Looks like those assignments came from commit 3c37026d43c47 ("NPTL, round one."); Ralf?

Oleg?

Vegard

#5 lore raw
In reply to Vegard Nossum
Quoted 33 lines · Vegard Nossum
Quoted 14 lines · Vegard Nossum
Quoted 10 lines · Guenter Roeck

Hi,

my qemu testis of mips images are failing in -next. Symptom is a hang during boot; see http://kerneltests.org/builders/qemu-mips-next for some examples.

I bisected the problem in next-20170526. It points to commit 4d6501dce079c ("kthread: Fix use-after-free if kthread fork fails"). Reverting that patch fixes the problem.

Bisect log is attached.

Hi,

Quoted 7 lines

Thanks for the report and sorry for the breakage :-/

I can't immediately spot what's going wrong, but I am able to reproduce it on mips so I will try to debug.

Are you sure it's this commit, though? I checked out linus/master and I get a boot hang even after reverting it.

My mistake; I ran into a different bug which made me think it was hanging when it wasn't.

However, I think I found the problem; does this patch fix it for you too?

arch/mips/kernel/process.c +0 -0
index 918d4c73e951..5351e1f3950d 100644

arch/mips/kernel/process.c int copy_thread_tls(unsigned long clone_flags, unsigned long usp, · line 125

Quoted 7 lines
arch/mips/kernel/process.c +0 -1
@@ -120,7 +120,6 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long usp,
         struct thread_info *ti = task_thread_info(p);
         struct pt_regs *childregs, *regs = current_pt_regs();
         unsigned long childksp;
-       p->set_child_tid = p->clear_child_tid = NULL;
 
         childksp = (unsigned long)task_stack_page(p) + THREAD_SIZE - 32;
 

Confirmed; that patch fixes the problem for me.

Guenter

Quoted 15 lines

The problem is that when we moved the p->{set,clear}_child_tid assignments inside copy_process(), the above assignments would clear them out. The assignments only exist on mips and openrisc (which would need the same patch), which explains why I didn't see it in my x86 testing. I think the patch above should be safe given that we're now always setting these fields in copy_process() at an appropriate moment.

Looks like those assignments came from commit 3c37026d43c47 ("NPTL, round one."); Ralf?

Oleg?

Vegard