Skip to contentSkip to conversation

x86/MCE/AMD: fix warning about sleep-in-atomic at early boot

PATCH Applied Konstantin Khlebnikov 4 participants 8 messages Oct 31, 2019 – Dec 17, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/MCE/AMD: fix warning about sleep-in-atomic at early boot

Function smca_configure() is called only for current cpu thus rdmsr_safe_on_cpu() could be replaced with atomic rdmsr_safe().

 BUG: sleeping function called from invalid context at kernel/sched/completion.c:99
 in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/1
 CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.79-16 #1
 Hardware name: GIGABYTE R181-Z90-00/MZ91-FS0-00, BIOS R11 10/25/2019
 Call Trace:
  dump_stack+0x5c/0x7b
  ___might_sleep+0xec/0x110
  wait_for_completion+0x39/0x160
  ? __rdmsr_safe_on_cpu+0x45/0x60
  rdmsr_safe_on_cpu+0xae/0xf0
  ? wrmsr_on_cpus+0x20/0x20
  ? machine_check_poll+0xfd/0x1f0
  ? mce_amd_feature_init+0x190/0x2d0
  mce_amd_feature_init+0x190/0x2d0
  mcheck_cpu_init+0x11a/0x460
  identify_cpu+0x3e2/0x560
  identify_secondary_cpu+0x13/0x80
  smp_store_cpu_info+0x45/0x50
  start_secondary+0xaa/0x200
  secondary_startup_64+0xa4/0xb0

Except warning in kernel log everything works fine.

Fixes 5896820e0aa3 ("x86/mce/AMD, EDAC/mce_amd: Define and use tables for known SMCA IP types")
---
 arch/x86/kernel/cpu/mce/amd.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/amd.c +1 -1
index 6ea7fdc82f3c..c7ab0d38af79 100644
@@ -269,7 +269,7 @@ static void smca_configure(unsigned int bank, unsigned int cpu)
 	if (smca_banks[bank].hwid)
 		return;
 
-	if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
+	if (rdmsr_safe(MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
 		pr_warn("Failed to read MCA_IPID for bank %d\n", bank);
 		return;
 	}
maintainer
#1 lore raw
Quoted 6 lines · Konstantin Khlebnikov

Function smca_configure() is called only for current cpu thus rdmsr_safe_on_cpu() could be replaced with atomic rdmsr_safe().

 BUG: sleeping function called from invalid context at kernel/sched/completion.c:99
 in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/1
 CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.79-16 #1
					     ^^^^^^^^^^

I'm assuming you hit this on latest upstream too?

Quoted 31 lines
 Hardware name: GIGABYTE R181-Z90-00/MZ91-FS0-00, BIOS R11 10/25/2019
 Call Trace:
  dump_stack+0x5c/0x7b
  ___might_sleep+0xec/0x110
  wait_for_completion+0x39/0x160
  ? __rdmsr_safe_on_cpu+0x45/0x60
  rdmsr_safe_on_cpu+0xae/0xf0
  ? wrmsr_on_cpus+0x20/0x20
  ? machine_check_poll+0xfd/0x1f0
  ? mce_amd_feature_init+0x190/0x2d0
  mce_amd_feature_init+0x190/0x2d0
  mcheck_cpu_init+0x11a/0x460
  identify_cpu+0x3e2/0x560
  identify_secondary_cpu+0x13/0x80
  smp_store_cpu_info+0x45/0x50
  start_secondary+0xaa/0x200
  secondary_startup_64+0xa4/0xb0

Except warning in kernel log everything works fine.

Fixes 5896820e0aa3 ("x86/mce/AMD, EDAC/mce_amd: Define and use tables for known SMCA IP types")
---
 arch/x86/kernel/cpu/mce/amd.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/amd.c +0 -0
index 6ea7fdc82f3c..c7ab0d38af79 100644

arch/x86/kernel/cpu/mce/amd.c static void smca_configure(unsigned int bank, unsigned int cpu) · line 272

Quoted 5 lines
arch/x86/kernel/cpu/mce/amd.c +1 -1
@@ -269,7 +269,7 @@ static void smca_configure(unsigned int bank, unsigned int cpu)
 	if (smca_banks[bank].hwid)
 		return;
 
-	if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
+	if (rdmsr_safe(MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {

Yazen, any objections?

Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
Reviewed-by
#2 lore raw
Quoted 52 lines

-----Original Message-----
From: Borislav Petkov <[email protected]>
Sent: Thursday, October 31, 2019 10:30 AM
To: Konstantin Khlebnikov <[email protected]>; Ghannam, Yazen <[email protected]>
Cc: Tony Luck <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH] x86/MCE/AMD: fix warning about sleep-in-atomic at early boot

Quoted 6 lines · Konstantin Khlebnikov

Function smca_configure() is called only for current cpu thus rdmsr_safe_on_cpu() could be replaced with atomic rdmsr_safe().

 BUG: sleeping function called from invalid context at kernel/sched/completion.c:99
 in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/1
 CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.79-16 #1
					     ^^^^^^^^^^

I'm assuming you hit this on latest upstream too?

Quoted 34 lines
 Hardware name: GIGABYTE R181-Z90-00/MZ91-FS0-00, BIOS R11 10/25/2019
 Call Trace:
  dump_stack+0x5c/0x7b
  ___might_sleep+0xec/0x110
  wait_for_completion+0x39/0x160
  ? __rdmsr_safe_on_cpu+0x45/0x60
  rdmsr_safe_on_cpu+0xae/0xf0
  ? wrmsr_on_cpus+0x20/0x20
  ? machine_check_poll+0xfd/0x1f0
  ? mce_amd_feature_init+0x190/0x2d0
  mce_amd_feature_init+0x190/0x2d0
  mcheck_cpu_init+0x11a/0x460
  identify_cpu+0x3e2/0x560
  identify_secondary_cpu+0x13/0x80
  smp_store_cpu_info+0x45/0x50
  start_secondary+0xaa/0x200
  secondary_startup_64+0xa4/0xb0

Except warning in kernel log everything works fine.

Fixes 5896820e0aa3 ("x86/mce/AMD, EDAC/mce_amd: Define and use tables for known SMCA IP types")
---
 arch/x86/kernel/cpu/mce/amd.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/amd.c +0 -0
index 6ea7fdc82f3c..c7ab0d38af79 100644
@@ -269,7 +269,7 @@ static void smca_configure(unsigned int bank, unsigned int cpu)
 	if (smca_banks[bank].hwid)
 		return;
 
Quoted 5 lines

arch/x86/kernel/cpu/mce/amd.c static void smca_configure(unsigned int bank, unsigned int cpu) · line 272

Quoted 2 lines
arch/x86/kernel/cpu/mce/amd.c +1 -1
-	if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
+	if (rdmsr_safe(MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {

Yazen, any objections?

This looks good to me.

We can go further and remove the "cpu" parameter from this entire function. But that can be another patch.

Thanks,
Yazen

#3 lore raw
In reply to Borislav Petkov
Quoted 10 lines · Borislav Petkov
Quoted 6 lines · Konstantin Khlebnikov

Function smca_configure() is called only for current cpu thus rdmsr_safe_on_cpu() could be replaced with atomic rdmsr_safe().

  BUG: sleeping function called from invalid context at kernel/sched/completion.c:99
  in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/1
  CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.79-16 #1
					     ^^^^^^^^^^

I'm assuming you hit this on latest upstream too?

I tried 5.4 once but there was no warning.
Code in 4.19 and in mainline almost the same.

Probably hardware needs full power cycle to reset state or something else.

Quoted 40 lines
Quoted 32 lines
  Hardware name: GIGABYTE R181-Z90-00/MZ91-FS0-00, BIOS R11 10/25/2019
  Call Trace:
   dump_stack+0x5c/0x7b
   ___might_sleep+0xec/0x110
   wait_for_completion+0x39/0x160
   ? __rdmsr_safe_on_cpu+0x45/0x60
   rdmsr_safe_on_cpu+0xae/0xf0
   ? wrmsr_on_cpus+0x20/0x20
   ? machine_check_poll+0xfd/0x1f0
   ? mce_amd_feature_init+0x190/0x2d0
   mce_amd_feature_init+0x190/0x2d0
   mcheck_cpu_init+0x11a/0x460
   identify_cpu+0x3e2/0x560
   identify_secondary_cpu+0x13/0x80
   smp_store_cpu_info+0x45/0x50
   start_secondary+0xaa/0x200
   secondary_startup_64+0xa4/0xb0

Except warning in kernel log everything works fine.

Fixes 5896820e0aa3 ("x86/mce/AMD, EDAC/mce_amd: Define and use tables for known SMCA IP types")
---
  arch/x86/kernel/cpu/mce/amd.c |    2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/amd.c +0 -0
index 6ea7fdc82f3c..c7ab0d38af79 100644

arch/x86/kernel/cpu/mce/amd.c static void smca_configure(unsigned int bank, unsigned int cpu) · line 272

Quoted 5 lines
arch/x86/kernel/cpu/mce/amd.c +1 -1
@@ -269,7 +269,7 @@ static void smca_configure(unsigned int bank, unsigned int cpu)
  	if (smca_banks[bank].hwid)
  		return;
 
-	if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
+	if (rdmsr_safe(MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {

Yazen, any objections?

maintainer
#4 lore raw
Quoted 2 lines · Konstantin Khlebnikov

I tried 5.4 once but there was no warning.
Code in 4.19 and in mainline almost the same.

Yes, but early boot code has changed a lot since 4.19. If you can't trigger it on 5.4, then I'll drop the BUG splat from your commit message and change it to talk about replacing the IPI-sending function, which is a good cleanup in itself.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#5 lore raw
Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Konstantin Khlebnikov

I tried 5.4 once but there was no warning.
Code in 4.19 and in mainline almost the same.

Yes, but early boot code has changed a lot since 4.19. If you can't trigger it on 5.4, then I'll drop the BUG splat from your commit message and change it to talk about replacing the IPI-sending function, which is a good cleanup in itself.

Ok, I was able to trigger it myself:

[    0.822602] BUG: sleeping function called from invalid context at kernel/sched/completion.c:99
[    0.822602] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/1
[    0.822602] no locks held by swapper/1/0.
[    0.822602] irq event stamp: 0
[    0.822602] hardirqs last  enabled at (0): [<0000000000000000>] 0x0
[    0.822602] hardirqs last disabled at (0): [<ffffffff8106dda9>] copy_process+0x8b9/0x1ca0
[    0.822602] softirqs last  enabled at (0): [<ffffffff8106dda9>] copy_process+0x8b9/0x1ca0
[    0.822602] softirqs last disabled at (0): [<0000000000000000>] 0x0
[    0.822602] Preemption disabled at:
[    0.822602] [<ffffffff8104703b>] start_secondary+0x3b/0x190
[    0.822602] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.5.0-rc2+ #1
[    0.822602] Hardware name: GIGABYTE MZ01-CE1-00/MZ01-CE1-00, BIOS F02 08/29/2018
[    0.822602] Call Trace:
[    0.822602]  dump_stack+0x71/0xa0
[    0.822602]  ___might_sleep.cold.92+0xf7/0x11f
[    0.822602]  wait_for_completion+0x3c/0x180
[    0.822602]  ? generic_exec_single+0xca/0x100
[    0.822602]  rdmsr_safe_on_cpu+0xe8/0x100
[    0.822602]  ? wrmsr_on_cpus+0x20/0x20
[    0.822602]  mce_amd_feature_init+0x2ab/0x590
[    0.822602]  mcheck_cpu_init+0x17a/0x4d0
[    0.822602]  identify_cpu+0x3f0/0x750
[    0.822602]  identify_secondary_cpu+0x13/0x80
[    0.822602]  smp_store_cpu_info+0x45/0x50
[    0.822602]  start_secondary+0x50/0x190
[    0.822602]  secondary_startup_64+0xa4/0xb0

Rerouting patch...

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#6 lore raw

[tip: ras/core] x86/MCE/AMD: Do not use rdmsr_safe_on_cpu() in smca_configure()

In reply to Konstantin Khlebnikov
The following commit has been merged into the ras/core branch of tip:

The following commit has been merged into the ras/core branch of tip:

Commit-ID:     17ba1939e3676417590ec6f7555608e137e1719a
Gitweb:        https://git.kernel.org/tip/17ba1939e3676417590ec6f7555608e137e1719a
Author:        Konstantin Khlebnikov <[email protected]>
AuthorDate:    Thu, 31 Oct 2019 16:04:48 +03:00
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Mon, 09 Dec 2019 14:54:59 +01:00

x86/MCE/AMD: Do not use rdmsr_safe_on_cpu() in smca_configure()

The function smca_configure() is called only on the current CPU therefore replace rdmsr_safe_on_cpu() with atomic rdmsr_safe() and avoid the IPI.

[ bp: Cleanup commit message. ]

---
 arch/x86/kernel/cpu/mce/amd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/amd.c +1 -1
index 5167bd2..e41e3b4 100644
@@ -269,7 +269,7 @@ static void smca_configure(unsigned int bank, unsigned int cpu)
 	if (smca_banks[bank].hwid)
 		return;
 
-	if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
+	if (rdmsr_safe(MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
 		pr_warn("Failed to read MCA_IPID for bank %d\n", bank);
 		return;
 	}
#7 lore raw

[tip: ras/urgent] x86/MCE/AMD: Do not use rdmsr_safe_on_cpu() in smca_configure()

In reply to Konstantin Khlebnikov
The following commit has been merged into the ras/urgent branch of tip:

The following commit has been merged into the ras/urgent branch of tip:

Commit-ID:     246ff09f89e54fdf740a8d496176c86743db3ec7
Gitweb:        https://git.kernel.org/tip/246ff09f89e54fdf740a8d496176c86743db3ec7
Author:        Konstantin Khlebnikov <[email protected]>
AuthorDate:    Thu, 31 Oct 2019 16:04:48 +03:00
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Tue, 17 Dec 2019 09:39:33 +01:00

x86/MCE/AMD: Do not use rdmsr_safe_on_cpu() in smca_configure()

... because interrupts are disabled that early and sending IPIs can deadlock:

  BUG: sleeping function called from invalid context at kernel/sched/completion.c:99
  in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/1
  no locks held by swapper/1/0.
  irq event stamp: 0
  hardirqs last  enabled at (0): [<0000000000000000>] 0x0
  hardirqs last disabled at (0): [<ffffffff8106dda9>] copy_process+0x8b9/0x1ca0
  softirqs last  enabled at (0): [<ffffffff8106dda9>] copy_process+0x8b9/0x1ca0
  softirqs last disabled at (0): [<0000000000000000>] 0x0
  Preemption disabled at:
  [<ffffffff8104703b>] start_secondary+0x3b/0x190
  CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.5.0-rc2+ #1
  Hardware name: GIGABYTE MZ01-CE1-00/MZ01-CE1-00, BIOS F02 08/29/2018
  Call Trace:
   dump_stack
   ___might_sleep.cold.92
   wait_for_completion
   ? generic_exec_single
   rdmsr_safe_on_cpu
   ? wrmsr_on_cpus
   mce_amd_feature_init
   mcheck_cpu_init
   identify_cpu
   identify_secondary_cpu
   smp_store_cpu_info
   start_secondary
   secondary_startup_64

The function smca_configure() is called only on the current CPU anyway, therefore replace rdmsr_safe_on_cpu() with atomic rdmsr_safe() and avoid the IPI.

[ bp: Update commit message. ]

---
 arch/x86/kernel/cpu/mce/amd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/mce/amd.c +1 -1
index 5167bd2..e41e3b4 100644
@@ -269,7 +269,7 @@ static void smca_configure(unsigned int bank, unsigned int cpu)
 	if (smca_banks[bank].hwid)
 		return;
 
-	if (rdmsr_safe_on_cpu(cpu, MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
+	if (rdmsr_safe(MSR_AMD64_SMCA_MCx_IPID(bank), &low, &high)) {
 		pr_warn("Failed to read MCA_IPID for bank %d\n", bank);
 		return;
 	}