Skip to contentSkip to conversation

CPU hotplug totally broken on HPC nx6325 (x86_64)

2.6.23-rc1 REGRESSION Rafael J. Wysocki 3 participants 7 messages Jul 26, 2007 – Jul 31, 2007

lore
Share & tools
lore
#0 lore raw

Hi,

On my Turion64-based HPC nx6325 with the 2.6.23-rc1 x86_64 kernel doing

# echo 0 > /sys/devices/system/cpu/cpu1/online

causes the system to crash in a spectacular fashion (call traces going continuously on the console, no reaction to anything except for the power button). For this reason, suspend and hibernation don't work as well.

Bisection has shown that your commit 19d36ccdc34f5ed444f8a6af0cbfdb6790eb1177 "x86: Fix alternatives and kprobes to remap write-protected kernel text" causes that to happen and indeed after applying the appended patch the problem doesn't show up any more.

[In the first version of the appended patch I forgot to add the chunk in mark_rodata_ro() and thus I got an oops from the line added to nop_out(). I wonder if that might be related to the $subject problem.]

Strangely enough, everything works fine on my second test box based on a dual-core Athlon 64.

Unfortunately, I can't get any debugging information from the box after the failure. I can try to set up netconsole, but I don't really expect it to work.

Greetings,
Rafael

---
 arch/i386/kernel/alternative.c |    8 ++++++--
 arch/i386/kernel/kprobes.c     |    8 ++++++--
 arch/x86_64/kernel/kprobes.c   |    9 +++++++--
 arch/x86_64/mm/init.c          |   10 ++++++++++
 4 files changed, 29 insertions(+), 6 deletions(-)
arch/i386/kernel/alternative.c +6 -2
Index: linux-2.6.23-rc1/arch/i386/kernel/alternative.c
===================================================================
@@ -156,7 +156,7 @@ static void nop_out(void *insns, unsigne
 		unsigned int noplen = len;
 		if (noplen > ASM_NOP_MAX)
 			noplen = ASM_NOP_MAX;
-		text_poke(insns, noptable[noplen], noplen);
+		memcpy(insns, noptable[noplen], noplen);
 		insns += noplen;
 		len -= noplen;
 	}
@@ -208,7 +208,7 @@ static void alternatives_smp_lock(u8 **s
 			continue;
 		if (*ptr > text_end)
 			continue;
-		text_poke(*ptr, ((unsigned char []){0xf0}), 1); /* add lock prefix */
+		**ptr = 0xf0; /* lock prefix */
 	};
 }
 
@@ -366,6 +366,10 @@ void apply_paravirt(struct paravirt_patc
 		/* Pad the rest with nops */
 		nop_out(p->instr + used, p->len - used);
 	}
+
+	/* Sync to be conservative, in case we patched following
+	 * instructions */
+	sync_core();
 }
 extern struct paravirt_patch_site __start_parainstructions[],
 	__stop_parainstructions[];
arch/i386/kernel/kprobes.c +6 -2
Index: linux-2.6.23-rc1/arch/i386/kernel/kprobes.c
===================================================================
@@ -170,12 +170,16 @@ int __kprobes arch_prepare_kprobe(struct
 
 void __kprobes arch_arm_kprobe(struct kprobe *p)
 {
-	text_poke(p->addr, ((unsigned char []){BREAKPOINT_INSTRUCTION}), 1);
+	*p->addr = BREAKPOINT_INSTRUCTION;
+	flush_icache_range((unsigned long) p->addr,
+			   (unsigned long) p->addr + sizeof(kprobe_opcode_t));
 }
 
 void __kprobes arch_disarm_kprobe(struct kprobe *p)
 {
-	text_poke(p->addr, &p->opcode, 1);
+	*p->addr = p->opcode;
+	flush_icache_range((unsigned long) p->addr,
+			   (unsigned long) p->addr + sizeof(kprobe_opcode_t));
 }
 
 void __kprobes arch_remove_kprobe(struct kprobe *p)
arch/x86_64/kernel/kprobes.c +7 -2
Index: linux-2.6.23-rc1/arch/x86_64/kernel/kprobes.c
===================================================================
@@ -39,6 +39,7 @@
 #include <linux/module.h>
 #include <linux/kdebug.h>
 
+#include <asm/cacheflush.h>
 #include <asm/pgtable.h>
 #include <asm/uaccess.h>
 #include <asm/alternative.h>
@@ -209,12 +210,16 @@ static void __kprobes arch_copy_kprobe(s
 
 void __kprobes arch_arm_kprobe(struct kprobe *p)
 {
-	text_poke(p->addr, ((unsigned char []){BREAKPOINT_INSTRUCTION}), 1);
+	*p->addr = BREAKPOINT_INSTRUCTION;
+	flush_icache_range((unsigned long) p->addr,
+			   (unsigned long) p->addr + sizeof(kprobe_opcode_t));
 }
 
 void __kprobes arch_disarm_kprobe(struct kprobe *p)
 {
-	text_poke(p->addr, &p->opcode, 1);
+	*p->addr = p->opcode;
+	flush_icache_range((unsigned long) p->addr,
+			   (unsigned long) p->addr + sizeof(kprobe_opcode_t));
 }
 
 void __kprobes arch_remove_kprobe(struct kprobe *p)
arch/x86_64/mm/init.c +10 -0
Index: linux-2.6.23-rc1/arch/x86_64/mm/init.c
===================================================================
@@ -600,6 +600,16 @@ void mark_rodata_ro(void)
 {
 	unsigned long start = (unsigned long)_stext, end;
 
+#ifdef CONFIG_HOTPLUG_CPU
+	/* It must still be possible to apply SMP alternatives. */
+	if (num_possible_cpus() > 1)
+		start = (unsigned long)_etext;
+#endif
+
+#ifdef CONFIG_KPROBES
+	start = (unsigned long)__start_rodata;
+#endif
+	
 	end = (unsigned long)__end_rodata;
 	start = (start + PAGE_SIZE - 1) & PAGE_MASK;
 	end &= PAGE_MASK;
#1 lore raw
Quoted 8 lines · Rafael J. Wysocki

On my Turion64-based HPC nx6325 with the 2.6.23-rc1 x86_64 kernel doing

# echo 0 > /sys/devices/system/cpu/cpu1/online

causes the system to crash in a spectacular fashion (call traces going continuously on the console, no reaction to anything except for the power button). For this reason, suspend and hibernation don't work as well.

Yeah, I really shouldn't have applied that patch. I didn't notice that it not only cleaned up the direct memcpy's, it also re-introduced the damn broken code that we fixed once already.

Dammit, that read-only debug support IS NOT WORTH THIS CRAP.

I absolutely *detest* it when "debugging features" end up being the thing that causes crashes. I think it shows a total lack of taste and understanding, and I'm totally tired of it. This has happened too many times.

Andi: please don't send this patch *ever* again. If a patch that is supposed to help debugging just causes problems, that patch should be thrown away FOREVER.

Andrew - on that same note: please throw away the dwarf traceback crap from your tree that Andi is still apparently pushing. Exact same issue.

Debugging "helper" code that has historically only caused problems. We could equally well just enable frame pointers for debugging and add the trivial code to follow that instead, and it would work (the way it has worked on x86 basically forever).

Rafael, does reverting just this part (and leaving the "text_poke()" cleanups) work for you?

Linus

---
 arch/i386/kernel/alternative.c |   10 ----------
 arch/i386/mm/init.c            |   14 +++++++++++---
 arch/x86_64/mm/init.c          |   10 ++++++++++
 3 files changed, 21 insertions(+), 13 deletions(-)
arch/i386/kernel/alternative.c +0 -10
index c3750c2..c0d0a89 100644
@@ -432,20 +432,10 @@ void __init alternative_instructions(void)
  */
 void __kprobes text_poke(void *oaddr, unsigned char *opcode, int len)
 {
-        u8 *addr = oaddr;
-	if (!pte_write(*lookup_address((unsigned long)addr))) {
-		struct page *p[2] = { virt_to_page(addr), virt_to_page(addr+PAGE_SIZE) };
-		addr = vmap(p, 2, VM_MAP, PAGE_KERNEL);
-		if (!addr)
-			return;
-		addr += ((unsigned long)oaddr) % PAGE_SIZE;
-	}
 	memcpy(addr, opcode, len);
 	sync_core();
 	/* Not strictly needed, but can speed CPU recovery up. Ignore cross cacheline
 	   case. */
 	if (cpu_has_clflush)
 		asm("clflush (%0) " :: "r" (oaddr) : "memory");
-	if (addr != oaddr)
-		vunmap(addr);
 }
arch/i386/mm/init.c +11 -3
index 1b1a1e6..4c4809f 100644
@@ -800,9 +800,17 @@ void mark_rodata_ro(void)
 	unsigned long start = PFN_ALIGN(_text);
 	unsigned long size = PFN_ALIGN(_etext) - start;
 
-	change_page_attr(virt_to_page(start),
-	                 size >> PAGE_SHIFT, PAGE_KERNEL_RX);
-	printk("Write protecting the kernel text: %luk\n", size >> 10);
+#ifndef CONFIG_KPROBES
+#ifdef CONFIG_HOTPLUG_CPU
+	/* It must still be possible to apply SMP alternatives. */
+	if (num_possible_cpus() <= 1)
+#endif
+	{
+		change_page_attr(virt_to_page(start),
+		                 size >> PAGE_SHIFT, PAGE_KERNEL_RX);
+		printk("Write protecting the kernel text: %luk\n", size >> 10);
+	}
+#endif
 	start += size;
 	size = (unsigned long)__end_rodata - start;
 	change_page_attr(virt_to_page(start),
arch/x86_64/mm/init.c +10 -0
index 38f5d63..458893b 100644
@@ -600,6 +600,16 @@ void mark_rodata_ro(void)
 {
 	unsigned long start = (unsigned long)_stext, end;
 
+#ifdef CONFIG_HOTPLUG_CPU
+	/* It must still be possible to apply SMP alternatives. */
+	if (num_possible_cpus() > 1)
+		start = (unsigned long)_etext;
+#endif
+
+#ifdef CONFIG_KPROBES
+	start = (unsigned long)__start_rodata;
+#endif
+	
 	end = (unsigned long)__end_rodata;
 	start = (start + PAGE_SIZE - 1) & PAGE_MASK;
 	end &= PAGE_MASK;
#2 lore raw
Quoted 29 lines · Linus Torvalds
Quoted 8 lines · Rafael J. Wysocki

On my Turion64-based HPC nx6325 with the 2.6.23-rc1 x86_64 kernel doing

# echo 0 > /sys/devices/system/cpu/cpu1/online

causes the system to crash in a spectacular fashion (call traces going continuously on the console, no reaction to anything except for the power button). For this reason, suspend and hibernation don't work as well.

Yeah, I really shouldn't have applied that patch. I didn't notice that it not only cleaned up the direct memcpy's, it also re-introduced the damn broken code that we fixed once already.

Dammit, that read-only debug support IS NOT WORTH THIS CRAP.

I absolutely *detest* it when "debugging features" end up being the thing that causes crashes. I think it shows a total lack of taste and understanding, and I'm totally tired of it. This has happened too many times.

Andi: please don't send this patch *ever* again. If a patch that is supposed to help debugging just causes problems, that patch should be thrown away FOREVER.

Andrew - on that same note: please throw away the dwarf traceback crap from your tree that Andi is still apparently pushing. Exact same issue.

Quoted 7 lines

Debugging "helper" code that has historically only caused problems. We could equally well just enable frame pointers for debugging and add the trivial code to follow that instead, and it would work (the way it has worked on x86 basically forever).

Rafael, does reverting just this part (and leaving the "text_poke()" cleanups) work for you?

Yes, it does, with the appended fix on top. :-)

Greetings,
Rafael

---
 arch/i386/kernel/alternative.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/i386/kernel/alternative.c +1 -1
Index: linux-2.6.23-rc1/arch/i386/kernel/alternative.c
===================================================================
@@ -432,7 +432,7 @@ void __init alternative_instructions(voi
  */
 void __kprobes text_poke(void *oaddr, unsigned char *opcode, int len)
 {
-	memcpy(addr, opcode, len);
+	memcpy(oaddr, opcode, len);
 	sync_core();
 	/* Not strictly needed, but can speed CPU recovery up. Ignore cross cacheline
 	   case. */
#3 lore raw
Quoted 5 lines · Rafael J. Wysocki
Quoted 3 lines

Rafael, does reverting just this part (and leaving the "text_poke()" cleanups) work for you?

Yes, it does, with the appended fix on top. :-)

Heh, I noticed that myself, but assumed you'd figure it out.

I ended up renaming "oaddr" as "addr", so the thing I committed is slightly different from the patch (+ your fix), but it should be otherwise 100% equivalent.

Linus

#4 lore raw
In reply to Linus Torvalds
Quoted 5 lines · * Linus Torvalds ([email protected]) wrote:
Quoted 4 lines · Rafael J. Wysocki

On my Turion64-based HPC nx6325 with the 2.6.23-rc1 x86_64 kernel doing

# echo 0 > /sys/devices/system/cpu/cpu1/online
Quoted 7 lines
Quoted 3 lines

causes the system to crash in a spectacular fashion (call traces going continuously on the console, no reaction to anything except for the power button). For this reason, suspend and hibernation don't work as well.

Yeah, I really shouldn't have applied that patch. I didn't notice that it not only cleaned up the direct memcpy's, it also re-introduced the damn broken code that we fixed once already.

This also fixes paravirt patching which was broken when text_poke() tried to patch the various pv ops in lookup_address.

thanks,
-chris

#5 lore raw
Quoted 3 lines · Chris Wright

This also fixes paravirt patching which was broken when text_poke() tried to patch the various pv ops in lookup_address.

Hmm. What is "this"? The revert?

That said, I do wonder whether virtualization still has problems with CONFIG_RODATA, though. We limit the RODATA memory ranges based on KPROBES and HOTPLUG_CPU, but not based on VIRTUALIZATION.

I'd expect any virtualization fixups to hit the same problems that the SMP alternatives hit. No?

Linus

#6 lore raw
Quoted 5 lines · * Linus Torvalds ([email protected]) wrote:
Quoted 2 lines · Chris Wright

This also fixes paravirt patching which was broken when text_poke() tried to patch the various pv ops in lookup_address.

Hmm. What is "this"? The revert?

Yes, sorry, your revert also fixes paravirt patching.

Quoted 6 lines

That said, I do wonder whether virtualization still has problems with CONFIG_RODATA, though. We limit the RODATA memory ranges based on KPROBES and HOTPLUG_CPU, but not based on VIRTUALIZATION.

I'd expect any virtualization fixups to hit the same problems that the SMP alternatives hit. No?

Hmm, patching should've already happened, and aside of module loading, isn't typically done again. I think it's OK as it is.

thanks,
-chris