Skip to contentSkip to conversation

2.6.34-rc2 - crash on shutdown

DISCUSSION David R 4 participants 7 messages Mar 23, 2010

Subsystems (2)
lore
Share & tools
lore

David R

author
#0 lore raw

I recently upgraded my kernel to the .34 rc, and I'm getting the following crash on shutdown (see attached console image)

(I never ran rc1 so the problem may well be present there too).

I can follow up with bootup dmesg and config later this evening if required.

Cheers
David

#1 lore raw
Quoted 2 lines · David R

I recently upgraded my kernel to the .34 rc, and I'm getting the following crash on shutdown (see attached console image)

Mee too, also in amd_pmu_cpu_offline().

The only pointer access in this function is cpuhw->amd_nb, but I don't see any obvious bugs.

Regards,
Clemens

#2 lore raw
Quoted 10 lines · Clemens Ladisch
Quoted 2 lines · David R

I recently upgraded my kernel to the .34 rc, and I'm getting the following crash on shutdown (see attached console image)

Mee too, also in amd_pmu_cpu_offline().

The only pointer access in this function is cpuhw->amd_nb, but I don't see any obvious bugs.

I reported a problem with the AMD initialization just last week. There is an issue with amd_pmu_cpu_online() which gets called too early, and thus fails. That leaves some bogus state and causes a crash in amd_pmu_cpu_offline().

I proposed a fix which was rejected. The alternative involves moving some the of CPU initialization code (on AMD) to an earlier position,i.e., which would be executed before the CPU_STARTED notifier. Nobody has proposed anything else so far.

Quoted 3 lines

Regards,
Clemens

Signature · 8 lines
Stephane Eranian  | EMEA Software Engineering
Google France | 38 avenue de l'Opéra | 75002 Paris
Tel : +33 (0) 1 42 68 53 00
This email may be confidential or privileged. If you received this
communication by mistake, please
don't forward it to anyone else, please erase all copies and
attachments, and please let me know that
it went to the wrong person. Thanks
#3 lore raw
Quoted 9 lines · Stephane Eranian
Quoted 2 lines · Clemens Ladisch

The only pointer access in this function is cpuhw->amd_nb, but I don't see any obvious bugs.

I reported a problem with the AMD initialization just last week. There is an issue with amd_pmu_cpu_online() which gets called too early, and thus fails. That leaves some bogus state and causes a crash in amd_pmu_cpu_offline().

Quoted 4 lines

I proposed a fix which was rejected. The alternative involves moving some the of CPU initialization code (on AMD) to an earlier position,i.e., which would be executed before the CPU_STARTED notifier. Nobody has proposed anything else so far.

I don't know about the early bootmem stuff, but regardless of this issue, if amd_pmu_cpu_online() can fail, then amd_pmu_cpu_offline() must be able to handle this without blowing up. Something like this (untested):

arch/x86/kernel/cpu/perf_event_amd.c +4 -4
@@ -324,17 +324,17 @@ static void amd_pmu_cpu_online(int cpu)
 	if (boot_cpu_data.x86_max_cores < 2)
 		return;
 
+	cpu1 = &per_cpu(cpu_hw_events, cpu);
+	cpu1->amd_nb = NULL;
+
 	/*
 	 * function may be called too early in the
 	 * boot process, in which case nb_id is bogus
 	 */
 	nb_id = amd_get_nb_id(cpu);
 	if (nb_id == BAD_APICID)
 		return;
 
-	cpu1 = &per_cpu(cpu_hw_events, cpu);
-	cpu1->amd_nb = NULL;
-
 	raw_spin_lock(&amd_nb_lock);
 
 	for_each_online_cpu(i) {
@@ -370,7 +370,7 @@ static void amd_pmu_cpu_offline(int cpu)
 
 	raw_spin_lock(&amd_nb_lock);
 
-	if (--cpuhw->amd_nb->refcnt == 0)
+	if (cpuhw->amd_nb && --cpuhw->amd_nb->refcnt == 0)
 		kfree(cpuhw->amd_nb);
 
 	cpuhw->amd_nb = NULL;
#4 lore raw
Quoted 10 lines · Clemens Ladisch
Quoted 9 lines · Stephane Eranian
Quoted 2 lines · Clemens Ladisch

The only pointer access in this function is cpuhw->amd_nb, but I don't see any obvious bugs.

I reported a problem with the AMD initialization just last week. There is an issue with amd_pmu_cpu_online() which gets called too early, and thus fails. That leaves some bogus state and causes a crash in amd_pmu_cpu_offline().

Quoted 8 lines
Quoted 4 lines

I proposed a fix which was rejected. The alternative involves moving some the of CPU initialization code (on AMD) to an earlier position,i.e., which would be executed before the CPU_STARTED notifier. Nobody has proposed anything else so far.

I don't know about the early bootmem stuff, but regardless of this issue, if amd_pmu_cpu_online() can fail, then amd_pmu_cpu_offline() must be able to handle this without blowing up. Something like this (untested):

I guess we handle that already:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a90110c61073eab95d1986322693c2b9a8a6a5f6

Rafael

#5 lore raw
Quoted 16 lines · Rafael J. Wysocki
Quoted 15 lines · Clemens Ladisch
Quoted 13 lines · Stephane Eranian
Quoted 2 lines · Clemens Ladisch

The only pointer access in this function is cpuhw->amd_nb, but I don't see any obvious bugs.

I reported a problem with the AMD initialization just last week. There is an issue with amd_pmu_cpu_online() which gets called too early, and thus fails. That leaves some bogus state and causes a crash in amd_pmu_cpu_offline().

I proposed a fix which was rejected. The alternative involves moving some the of CPU initialization code (on AMD) to an earlier position,i.e., which would be executed before the CPU_STARTED notifier. Nobody has proposed anything else so far.

Quoted 8 lines
Quoted 3 lines

I don't know about the early bootmem stuff, but regardless of this issue, if amd_pmu_cpu_online() can fail, then amd_pmu_cpu_offline() must be able to handle this without blowing up.  Something like this (untested):

I guess we handle that already:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a90110c61073eab95d1986322693c2b9a8a6a5f6

Ok, the fix avoids the crash but perf_events support for AMD is still broken.

The root of the problem is elsewhere as I pointed out last week. Peter proposed a patch today and I think this would be enough to avoid the crash and have perf_events working again on AMD.

#6 lore raw
Quoted 23 lines · Stephane Eranian
Quoted 22 lines · Rafael J. Wysocki
Quoted 10 lines · Clemens Ladisch
Quoted 9 lines · Stephane Eranian
Quoted 2 lines · Clemens Ladisch

The only pointer access in this function is cpuhw->amd_nb, but I don't see any obvious bugs.

I reported a problem with the AMD initialization just last week. There is an issue with amd_pmu_cpu_online() which gets called too early, and thus fails. That leaves some bogus state and causes a crash in amd_pmu_cpu_offline().

Quoted 8 lines
Quoted 4 lines

I proposed a fix which was rejected. The alternative involves moving some the of CPU initialization code (on AMD) to an earlier position,i.e., which would be executed before the CPU_STARTED notifier. Nobody has proposed anything else so far.

I don't know about the early bootmem stuff, but regardless of this issue, if amd_pmu_cpu_online() can fail, then amd_pmu_cpu_offline() must be able to handle this without blowing up. Something like this (untested):

I guess we handle that already:

Quoted 7 lines
Quoted 2 lines

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a90110c61073eab95d1986322693c2b9a8a6a5f6

Ok, the fix avoids the crash but perf_events support for AMD is still broken.

The root of the problem is elsewhere as I pointed out last week. Peter proposed a patch today and I think this would be enough to avoid the crash and have perf_events working again on AMD.

Yes, I saw the Peter's patch.

Thanks,
Rafael