Skip to contentSkip to conversation

powerpc/64s: fix and improve machine check handling

PATCH In review Nicholas Piggin 4 participants 14 messages Nov 28, 2020 – Dec 10, 2020

Subsystems (2)
lore
Share & tools
lore

Nicholas Piggin

author reviewer
#0 lore raw

cover powerpc/64s: fix and improve machine check handling

First patch is a nasty memory scribble introduced by me :( That should go into fixes.

The next ones could wait for next merge window. They get things to the point where misbehaving or buggy guest isn't so painful for the host, and also get the guest SLB dumping code working (because the host no longer clears them before delivering the MCE to the guest).

I have a crasher guest vmlinux with a few SLB handling bugs introduced which now bumbles along okay without bothering the host so much.

I don't know what the picture or high level strategy really is for UE memory errors in the guest, particularly with PowerVM, so some review there would be good (I haven't changed anything really in that space AFAIKS, but as an overall "is this the right way to go" kind of thing).

Thanks,
Nick

Nicholas Piggin (8):
powerpc/64s/powernv: Fix memory corruption when saving SLB entries on
MCE
powerpc/64s/powernv: Allow KVM to handle guest machine check details
KVM: PPC: Book3S HV: Don't attempt to recover machine checks for FWNMI
enabled guests
KVM: PPC: Book3S HV: Ratelimit machine check messages coming from
guests
powerpc/64s/powernv: ratelimit harmless HMI error printing powerpc/64s/pseries: Add ERAT specific machine check handler powerpc/64s: Remove "Host" from MCE logging
powerpc/64s: tidy machine check SLB logging

 arch/powerpc/include/asm/mce.h            |  1 +
 arch/powerpc/kernel/mce.c                 |  4 +-
 arch/powerpc/kernel/mce_power.c           | 98 +++++++++++++----------
 arch/powerpc/kvm/book3s_hv.c              | 11 ++-
 arch/powerpc/kvm/book3s_hv_ras.c          | 23 ++++--
 arch/powerpc/mm/book3s64/slb.c            | 39 ++++-----
 arch/powerpc/platforms/powernv/opal-hmi.c | 27 ++++---
 arch/powerpc/platforms/powernv/setup.c    |  9 ++-
 arch/powerpc/platforms/pseries/ras.c      |  5 +-
 9 files changed, 129 insertions(+), 88 deletions(-)
Signature · 1 line
2.23.0
author reviewer
#1 lore raw

PATCH 1/8 powerpc/64s/powernv: Fix memory corruption when saving SLB entries on MCE

This can be hit by an HPT guest running on an HPT host and bring down the host, so it's quite important to fix.

Fixes 7290f3b3d3e66 ("powerpc/64s/powernv: machine check dump SLB contents")
---
 arch/powerpc/platforms/powernv/setup.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
arch/powerpc/platforms/powernv/setup.c +6 -2
index 46115231a3b2..4426a109ec2f 100644
@@ -211,11 +211,16 @@ static void __init pnv_init(void)
 		add_preferred_console("hvc", 0, NULL);
 
 	if (!radix_enabled()) {
+		size_t size = sizeof(struct slb_entry) * mmu_slb_size;
 		int i;
 
 		/* Allocate per cpu area to save old slb contents during MCE */
-		for_each_possible_cpu(i)
-			paca_ptrs[i]->mce_faulty_slbs = memblock_alloc_node(mmu_slb_size, __alignof__(*paca_ptrs[i]->mce_faulty_slbs), cpu_to_node(i));
+		for_each_possible_cpu(i) {
+			paca_ptrs[i]->mce_faulty_slbs +					memblock_alloc_node(size,
+						__alignof__(struct slb_entry),
+						cpu_to_node(i));
+		}
 	}
 }
 
Signature · 1 line
2.23.0
author reviewer
#3 lore raw

PATCH 2/8 powerpc/64s/powernv: Allow KVM to handle guest machine check details

In reply to Nicholas Piggin

KVM has strategies to perform machine check recovery. If a MCE hits in a guest, have the low level handler just decode and save the MCE but not try to recover anything, so KVM can deal with it.

The host does not own SLBs and does not need to report the SLB state in case of a multi-hit for example, or know about the virtual memory map of the guest.

UE and memory poisoning of guest pages in the host is one thing that is possibly not completely robust at the moment, but this too needs to go via KVM (possibly via the guest and back out to host via hcall) rather than being handled at a low level in the host handler.

---
 arch/powerpc/kernel/mce.c       |  2 +-
 arch/powerpc/kernel/mce_power.c | 96 ++++++++++++++++++---------------
 2 files changed, 55 insertions(+), 43 deletions(-)
arch/powerpc/kernel/mce.c +1 -1
index 63702c0badb9..8afe8d37b983 100644
@@ -577,7 +577,7 @@ void machine_check_print_event_info(struct machine_check_event *evt,
 
 #ifdef CONFIG_PPC_BOOK3S_64
 	/* Display faulty slb contents for SLB errors. */
-	if (evt->error_type = MCE_ERROR_TYPE_SLB)
+	if (evt->error_type = MCE_ERROR_TYPE_SLB && !in_guest)
 		slb_dump_contents(local_paca->mce_faulty_slbs);
 #endif
 }
arch/powerpc/kernel/mce_power.c +54 -42

145 lines not on this page · raw message

Signature · 1 line
2.23.0
author reviewer
#4 lore raw

PATCH 3/8 KVM: PPC: Book3S HV: Don't attempt to recover machine checks for FWNMI enabled guests

In reply to Nicholas Piggin

Guests that can deal with machine checks would actually prefer the hypervisor not to try recover for them. For example if SLB multi-hits are recovered by the hypervisor by clearing the SLB then the guest will not be able to log the contents and debug its programming error.

If guests don't register for FWNMI, they may not be so capable and so the hypervisor will continue to recover for those.

---
 arch/powerpc/kvm/book3s_hv_ras.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
arch/powerpc/kvm/book3s_hv_ras.c +16 -7
index 6028628ea3ac..d4bca93b79f6 100644
@@ -65,10 +65,9 @@ static void reload_slb(struct kvm_vcpu *vcpu)
  * On POWER7, see if we can handle a machine check that occurred inside
  * the guest in real mode, without switching to the host partition.
  */
-static void kvmppc_realmode_mc_power7(struct kvm_vcpu *vcpu)
+static long kvmppc_realmode_mc_power7(struct kvm_vcpu *vcpu)
 {
 	unsigned long srr1 = vcpu->arch.shregs.msr;
-	struct machine_check_event mce_evt;
 	long handled = 1;
 
 	if (srr1 & SRR1_MC_LDSTERR) {
@@ -106,6 +105,21 @@ static void kvmppc_realmode_mc_power7(struct kvm_vcpu *vcpu)
 		handled = 0;
 	}
 
+	return handled;
+}
+
+void kvmppc_realmode_machine_check(struct kvm_vcpu *vcpu)
+{
+	struct machine_check_event mce_evt;
+	long handled;
+
+	if (vcpu->kvm->arch.fwnmi_enabled) {
+		/* FWNMI guests handle their own recovery */
+		handled = 0;
+	} else {
+		handled = kvmppc_realmode_mc_power7(vcpu);
+	}
+
 	/*
 	 * Now get the event and stash it in the vcpu struct so it can
 	 * be handled by the primary thread in virtual mode.  We can't
@@ -122,11 +136,6 @@ static void kvmppc_realmode_mc_power7(struct kvm_vcpu *vcpu)
 	vcpu->arch.mce_evt = mce_evt;
 }
 
-void kvmppc_realmode_machine_check(struct kvm_vcpu *vcpu)
-{
-	kvmppc_realmode_mc_power7(vcpu);
-}
-
 /* Check if dynamic split is in force and return subcore size accordingly. */
 static inline int kvmppc_cur_subcore_size(void)
 {
Signature · 1 line
2.23.0
author reviewer
#5 lore raw

PATCH 4/8 KVM: PPC: Book3S HV: Ratelimit machine check messages coming from guests

In reply to Nicholas Piggin

A number of machine check exceptions are triggerable by the guest. Ratelimit these to avoid a guest flooding the host console and logs.

---
 arch/powerpc/kvm/book3s_hv.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
arch/powerpc/kvm/book3s_hv.c +8 -3
index e3b1839fc251..c94f9595133d 100644
@@ -1328,8 +1328,12 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,
 		r = RESUME_GUEST;
 		break;
 	case BOOK3S_INTERRUPT_MACHINE_CHECK:
-		/* Print the MCE event to host console. */
-		machine_check_print_event_info(&vcpu->arch.mce_evt, false, true);
+		/*
+		 * Print the MCE event to host console. Ratelimit so the guest
+		 * can't flood the host log.
+		 */
+		if (printk_ratelimit())
+			machine_check_print_event_info(&vcpu->arch.mce_evt,false, true);
 
 		/*
 		 * If the guest can do FWNMI, exit to userspace so it can
@@ -1519,7 +1523,8 @@ static int kvmppc_handle_nested_exit(struct kvm_vcpu *vcpu)
 		/* Pass the machine check to the L1 guest */
 		r = RESUME_HOST;
 		/* Print the MCE event to host console. */
-		machine_check_print_event_info(&vcpu->arch.mce_evt, false, true);
+		if (printk_ratelimit())
+			machine_check_print_event_info(&vcpu->arch.mce_evt, false, true);
 		break;
 	/*
 	 * We get these next two if the guest accesses a page which it thinks
Signature · 1 line
2.23.0
author reviewer
#7 lore raw

PATCH 5/8 powerpc/64s/powernv: ratelimit harmless HMI error printing

In reply to Nicholas Piggin

Harmless HMI errors can be triggered by guests in some cases, and don't contain much useful information anyway. Ratelimit these to avoid flooding the console/logs.

---
 arch/powerpc/platforms/powernv/opal-hmi.c | 27 +++++++++++++----------
 1 file changed, 15 insertions(+), 12 deletions(-)
arch/powerpc/platforms/powernv/opal-hmi.c +15 -12
index 3e1f064a18db..959da6df0227 100644
@@ -240,19 +240,22 @@ static void print_hmi_event_info(struct OpalHMIEvent *hmi_evt)
 		break;
 	}
 
-	printk("%s%s Hypervisor Maintenance interrupt [%s]\n",
-		level, sevstr,
-		hmi_evt->disposition = OpalHMI_DISPOSITION_RECOVERED ?
-		"Recovered" : "Not recovered");
-	error_info = hmi_evt->type < ARRAY_SIZE(hmi_error_types) ?
-			hmi_error_types[hmi_evt->type]
-			: "Unknown";
-	printk("%s Error detail: %s\n", level, error_info);
-	printk("%s	HMER: %016llx\n", level, be64_to_cpu(hmi_evt->hmer));
-	if ((hmi_evt->type = OpalHMI_ERROR_TFAC) ||
-		(hmi_evt->type = OpalHMI_ERROR_TFMR_PARITY))
-		printk("%s	TFMR: %016llx\n", level,
+	if (hmi_evt->severity != OpalHMI_SEV_NO_ERROR || printk_ratelimit()) {
+		printk("%s%s Hypervisor Maintenance interrupt [%s]\n",
+			level, sevstr,
+			hmi_evt->disposition = OpalHMI_DISPOSITION_RECOVERED ?
+			"Recovered" : "Not recovered");
+		error_info = hmi_evt->type < ARRAY_SIZE(hmi_error_types) ?
+				hmi_error_types[hmi_evt->type]
+				: "Unknown";
+		printk("%s Error detail: %s\n", level, error_info);
+		printk("%s	HMER: %016llx\n", level,
+					be64_to_cpu(hmi_evt->hmer));
+		if ((hmi_evt->type = OpalHMI_ERROR_TFAC) ||
+			(hmi_evt->type = OpalHMI_ERROR_TFMR_PARITY))
+			printk("%s	TFMR: %016llx\n", level,
 						be64_to_cpu(hmi_evt->tfmr));
+	}
 
 	if (hmi_evt->version < OpalHMIEvt_V2)
 		return;
Signature · 1 line
2.23.0
author reviewer
#9 lore raw

PATCH 6/8 powerpc/64s/pseries: Add ERAT specific machine check handler

In reply to Nicholas Piggin

Don't treat ERAT MCEs as SLB, don't save the SLB and use a specific ERAT flush to recover it.

---
 arch/powerpc/include/asm/mce.h       | 1 +
 arch/powerpc/kernel/mce_power.c      | 2 +-
 arch/powerpc/platforms/pseries/ras.c | 5 ++++-
 3 files changed, 6 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/mce.h +1 -0
index 89aa8248a57d..e6c27ae843dc 100644
@@ -228,6 +228,7 @@ int mce_register_notifier(struct notifier_block *nb);
 int mce_unregister_notifier(struct notifier_block *nb);
 #ifdef CONFIG_PPC_BOOK3S_64
 void flush_and_reload_slb(void);
+void flush_erat(void);
 long __machine_check_early_realmode_p7(struct pt_regs *regs);
 long __machine_check_early_realmode_p8(struct pt_regs *regs);
 long __machine_check_early_realmode_p9(struct pt_regs *regs);
arch/powerpc/kernel/mce_power.c +1 -1
index 1372ce3f7bdd..667104d4c455 100644
@@ -97,7 +97,7 @@ void flush_and_reload_slb(void)
 }
 #endif
 
-static void flush_erat(void)
+void flush_erat(void)
 {
 #ifdef CONFIG_PPC_BOOK3S_64
 	if (!early_cpu_has_feature(CPU_FTR_ARCH_300)) {
arch/powerpc/platforms/pseries/ras.c +4 -1
index b2b245b25edb..149cec2212e6 100644
@@ -526,8 +526,11 @@ static int mce_handle_err_realmode(int disposition, u8 error_type)
 #ifdef CONFIG_PPC_BOOK3S_64
 	if (disposition = RTAS_DISP_NOT_RECOVERED) {
 		switch (error_type) {
-		case	MC_ERROR_TYPE_SLB:
 		case	MC_ERROR_TYPE_ERAT:
+			flush_erat();
+			disposition = RTAS_DISP_FULLY_RECOVERED;
+			break;
+		case	MC_ERROR_TYPE_SLB:
 			/*
 			 * Store the old slb content in paca before flushing.
 			 * Print this when we go to virtual mode.
Signature · 1 line
2.23.0
author reviewer
#10 lore raw

PATCH 7/8 powerpc/64s: Remove "Host" from MCE logging

In reply to Nicholas Piggin

"Host" caused machine check is printed when the kernel sees a MCE hit in this kernel or userspace, and "Guest" if it hit one of its guests. This is confusing when a guest kernel handles a hypervisor- delivered MCE, it also prints "Host".

Just remove "Host". "Guest" is adequate to make the distinction.

---
 arch/powerpc/kernel/mce.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/kernel/mce.c +1 -1
index 8afe8d37b983..9f3e133b57b7 100644
@@ -555,7 +555,7 @@ void machine_check_print_event_info(struct machine_check_event *evt,
 	}
 
 	printk("%sMCE: CPU%d: machine check (%s) %s %s %s %s[%s]\n",
-		level, evt->cpu, sevstr, in_guest ? "Guest" : "Host",
+		level, evt->cpu, sevstr, in_guest ? "Guest" : "",
 		err_type, subtype, dar_str,
 		evt->disposition = MCE_DISPOSITION_RECOVERED ?
 		"Recovered" : "Not recovered");
Signature · 1 line
2.23.0
author reviewer
#11 lore raw

PATCH 8/8 powerpc/64s: tidy machine check SLB logging

In reply to Nicholas Piggin

Since ISA v3.0, SLB no longer uses the slb_cache, and stab_rr is no longer correlated with SLB allocation. Move those to pre-3.0.

While here, improve some alignments and reduce whitespace.

---
 arch/powerpc/mm/book3s64/slb.c | 39 ++++++++++++++++++----------------
 1 file changed, 21 insertions(+), 18 deletions(-)
arch/powerpc/mm/book3s64/slb.c +21 -18
index c30fcbfa0e32..6d720c1c08a4 100644
@@ -255,7 +255,6 @@ void slb_dump_contents(struct slb_entry *slb_ptr)
 		return;
 
 	pr_err("SLB contents of cpu 0x%x\n", smp_processor_id());
-	pr_err("Last SLB entry inserted at slot %d\n", get_paca()->stab_rr);
 
 	for (i = 0; i < mmu_slb_size; i++) {
 		e = slb_ptr->esid;
@@ -265,34 +264,38 @@ void slb_dump_contents(struct slb_entry *slb_ptr)
 		if (!e && !v)
 			continue;
 
-		pr_err("%02d %016lx %016lx\n", i, e, v);
+		pr_err("%02d %016lx %016lx %s\n", i, e, v,
+				(e & SLB_ESID_V) ? "VALID" : "NOT VALID");
 
-		if (!(e & SLB_ESID_V)) {
-			pr_err("\n");
+		if (!(e & SLB_ESID_V))
 			continue;
-		}
+
 		llp = v & SLB_VSID_LLP;
 		if (v & SLB_VSID_B_1T) {
-			pr_err("  1T  ESID=%9lx  VSID=%13lx LLP:%3lx\n",
+			pr_err("     1T ESID=%9lx VSID=%13lx LLP:%3lx\n",
 			       GET_ESID_1T(e),
 			       (v & ~SLB_VSID_B) >> SLB_VSID_SHIFT_1T, llp);
 		} else {
-			pr_err(" 256M ESID=%9lx  VSID=%13lx LLP:%3lx\n",
+			pr_err("   256M ESID=%9lx VSID=%13lx LLP:%3lx\n",
 			       GET_ESID(e),
 			       (v & ~SLB_VSID_B) >> SLB_VSID_SHIFT, llp);
 		}
 	}
-	pr_err("----------------------------------\n");
-
-	/* Dump slb cache entires as well. */
-	pr_err("SLB cache ptr value = %d\n", get_paca()->slb_save_cache_ptr);
-	pr_err("Valid SLB cache entries:\n");
-	n = min_t(int, get_paca()->slb_save_cache_ptr, SLB_CACHE_ENTRIES);
-	for (i = 0; i < n; i++)
-		pr_err("%02d EA[0-35]=%9x\n", i, get_paca()->slb_cache[i]);
-	pr_err("Rest of SLB cache entries:\n");
-	for (i = n; i < SLB_CACHE_ENTRIES; i++)
-		pr_err("%02d EA[0-35]=%9x\n", i, get_paca()->slb_cache[i]);
+
+	if (!early_cpu_has_feature(CPU_FTR_ARCH_300)) {
+		/* RR is not so useful as it's often not used for allocation */
+		pr_err("SLB RR allocator index %d\n", get_paca()->stab_rr);
+
+		/* Dump slb cache entires as well. */
+		pr_err("SLB cache ptr value = %d\n", get_paca()->slb_save_cache_ptr);
+		pr_err("Valid SLB cache entries:\n");
+		n = min_t(int, get_paca()->slb_save_cache_ptr, SLB_CACHE_ENTRIES);
+		for (i = 0; i < n; i++)
+			pr_err("%02d EA[0-35]=%9x\n", i, get_paca()->slb_cache[i]);
+		pr_err("Rest of SLB cache entries:\n");
+		for (i = n; i < SLB_CACHE_ENTRIES; i++)
+			pr_err("%02d EA[0-35]=%9x\n", i, get_paca()->slb_cache[i]);
+	}
 }
 
 void slb_vmalloc_update(void)
Signature · 1 line
2.23.0
Acked-by on v1 1/8
#2 lore raw

Re: [PATCH 1/8] powerpc/64s/powernv: Fix memory corruption when saving SLB entries on MCE

In reply to Nicholas Piggin
Quoted 14 lines · Nicholas Piggin

This can be hit by an HPT guest running on an HPT host and bring down the host, so it's quite important to fix.

Fixes 7290f3b3d3e66 ("powerpc/64s/powernv: machine check dump SLB contents")
---
 arch/powerpc/platforms/powernv/setup.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
arch/powerpc/platforms/powernv/setup.c +0 -0
index 46115231a3b2..4426a109ec2f 100644

arch/powerpc/platforms/powernv/setup.c static void __init pnv_init(void) · line 214

Quoted 4 lines
arch/powerpc/platforms/powernv/setup.c +1 -0
@@ -211,11 +211,16 @@ static void __init pnv_init(void)
 		add_preferred_console("hvc", 0, NULL);
 
 	if (!radix_enabled()) {
+		size_t size = sizeof(struct slb_entry) * mmu_slb_size;

Thanks,
-Mahesh.

Quoted 17 lines
arch/powerpc/platforms/powernv/setup.c +6 -2
 		int i;
 
 		/* Allocate per cpu area to save old slb contents during MCE */
-		for_each_possible_cpu(i)
-			paca_ptrs[i]->mce_faulty_slbs = memblock_alloc_node(mmu_slb_size, __alignof__(*paca_ptrs[i]->mce_faulty_slbs), cpu_to_node(i));
+		for_each_possible_cpu(i) {
+			paca_ptrs[i]->mce_faulty_slbs =
+					memblock_alloc_node(size,
+						__alignof__(struct slb_entry),
+						cpu_to_node(i));
+		}
 	}
 }
 
Signature · 1 line
2.23.0
Signature · 1 line
Mahesh J Salgaonkar
#6 lore raw

Re: [PATCH 4/8] KVM: PPC: Book3S HV: Ratelimit machine check messages coming from guests

In reply to Nicholas Piggin
Quoted 14 lines · Nicholas Piggin

A number of machine check exceptions are triggerable by the guest. Ratelimit these to avoid a guest flooding the host console and logs.

---
 arch/powerpc/kvm/book3s_hv.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
arch/powerpc/kvm/book3s_hv.c +0 -0
index e3b1839fc251..c94f9595133d 100644
@@ -1328,8 +1328,12 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,
 		r = RESUME_GUEST;

arch/powerpc/kvm/book3s_hv.c static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu, · line 1336

Quoted 10 lines
arch/powerpc/kvm/book3s_hv.c +6 -2
 		break;
 	case BOOK3S_INTERRUPT_MACHINE_CHECK:
-		/* Print the MCE event to host console. */
-		machine_check_print_event_info(&vcpu->arch.mce_evt, false, true);
+		/*
+		 * Print the MCE event to host console. Ratelimit so the guest
+		 * can't flood the host log.
+		 */
+		if (printk_ratelimit())
+			machine_check_print_event_info(&vcpu->arch.mce_evt,false, true);

You're not supposed to use printk_ratelimit(), because there's a single rate limit state for all printks. ie. some other noisty printk() can cause this one to never be printed.

I folded this in:

arch/powerpc/kvm/book3s_hv.c +10 -3
index cbbc4f0a26fe..cfaa91b27112 100644
@@ -1327,12 +1327,14 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,
 	case BOOK3S_INTERRUPT_SYSTEM_RESET:
 		r = RESUME_GUEST;
 		break;
-	case BOOK3S_INTERRUPT_MACHINE_CHECK:
+	case BOOK3S_INTERRUPT_MACHINE_CHECK: {
+		static DEFINE_RATELIMIT_STATE(rs, DEFAULT_RATELIMIT_INTERVAL,
+					      DEFAULT_RATELIMIT_BURST);
 		/*
 		 * Print the MCE event to host console. Ratelimit so the guest
 		 * can't flood the host log.
 		 */
-		if (printk_ratelimit())
+		if (__ratelimit(&rs))
 			machine_check_print_event_info(&vcpu->arch.mce_evt,false, true);
 
 		/*
@@ -1361,6 +1363,7 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,
 
 		r = RESUME_HOST;
 		break;
+	}
 	case BOOK3S_INTERRUPT_PROGRAM:
 	{
 		ulong flags;
@@ -1520,12 +1523,16 @@ static int kvmppc_handle_nested_exit(struct kvm_vcpu *vcpu)
 		r = RESUME_GUEST;
 		break;
 	case BOOK3S_INTERRUPT_MACHINE_CHECK:
+	{
+		static DEFINE_RATELIMIT_STATE(rs, DEFAULT_RATELIMIT_INTERVAL,
+					      DEFAULT_RATELIMIT_BURST);
 		/* Pass the machine check to the L1 guest */
 		r = RESUME_HOST;
 		/* Print the MCE event to host console. */
-		if (printk_ratelimit())
+		if (__ratelimit(&rs))
 			machine_check_print_event_info(&vcpu->arch.mce_evt, false, true);
 		break;
+	}
 	/*
 	 * We get these next two if the guest accesses a page which it thinks
 	 * it has mapped but which is not actually present, either because

cheers

#8 lore raw

Re: [PATCH 5/8] powerpc/64s/powernv: ratelimit harmless HMI error printing

In reply to Nicholas Piggin
Quoted 35 lines · Nicholas Piggin

Harmless HMI errors can be triggered by guests in some cases, and don't contain much useful information anyway. Ratelimit these to avoid flooding the console/logs.

---
 arch/powerpc/platforms/powernv/opal-hmi.c | 27 +++++++++++++----------
 1 file changed, 15 insertions(+), 12 deletions(-)
arch/powerpc/platforms/powernv/opal-hmi.c +6 -12
index 3e1f064a18db..959da6df0227 100644
@@ -240,19 +240,22 @@ static void print_hmi_event_info(struct OpalHMIEvent *hmi_evt)
 		break;
 	}
 
-	printk("%s%s Hypervisor Maintenance interrupt [%s]\n",
-		level, sevstr,
-		hmi_evt->disposition = OpalHMI_DISPOSITION_RECOVERED ?
-		"Recovered" : "Not recovered");
-	error_info = hmi_evt->type < ARRAY_SIZE(hmi_error_types) ?
-			hmi_error_types[hmi_evt->type]
-			: "Unknown";
-	printk("%s Error detail: %s\n", level, error_info);
-	printk("%s	HMER: %016llx\n", level, be64_to_cpu(hmi_evt->hmer));
-	if ((hmi_evt->type = OpalHMI_ERROR_TFAC) ||
-		(hmi_evt->type = OpalHMI_ERROR_TFMR_PARITY))
-		printk("%s	TFMR: %016llx\n", level,
+	if (hmi_evt->severity != OpalHMI_SEV_NO_ERROR || printk_ratelimit()) {
+		printk("%s%s Hypervisor Maintenance interrupt [%s]\n",
+			level, sevstr,
+			hmi_evt->disposition = OpalHMI_DISPOSITION_RECOVERED ?
+			"Recovered" : "Not recovered");
+		error_info = hmi_evt->type < ARRAY_SIZE(hmi_error_types) ?

arch/powerpc/platforms/powernv/opal-hmi.c static void print_hmi_event_info(struct OpalHMIEvent *hmi_evt) · line 258

Quoted 10 lines
arch/powerpc/platforms/powernv/opal-hmi.c +9 -0
+				hmi_error_types[hmi_evt->type]
+				: "Unknown";
+		printk("%s Error detail: %s\n", level, error_info);
+		printk("%s	HMER: %016llx\n", level,
+					be64_to_cpu(hmi_evt->hmer));
+		if ((hmi_evt->type = OpalHMI_ERROR_TFAC) ||
+			(hmi_evt->type = OpalHMI_ERROR_TFMR_PARITY))
+			printk("%s	TFMR: %016llx\n", level,
 						be64_to_cpu(hmi_evt->tfmr));
+	}

Same comment RE printk_ratelimit(), I folded this in:

arch/powerpc/platforms/powernv/opal-hmi.c +3 -1
index 959da6df0227..f0c1830deb51 100644
@@ -213,6 +213,8 @@ static void print_hmi_event_info(struct OpalHMIEvent *hmi_evt)
 		"A hypervisor resource error occurred",
 		"CAPP recovery process is in progress",
 	};
+	static DEFINE_RATELIMIT_STATE(rs, DEFAULT_RATELIMIT_INTERVAL,
+				      DEFAULT_RATELIMIT_BURST);
 
 	/* Print things out */
 	if (hmi_evt->version < OpalHMIEvt_V1) {
@@ -240,7 +242,7 @@ static void print_hmi_event_info(struct OpalHMIEvent *hmi_evt)
 		break;
 	}
 
-	if (hmi_evt->severity != OpalHMI_SEV_NO_ERROR || printk_ratelimit()) {
+	if (hmi_evt->severity != OpalHMI_SEV_NO_ERROR || __ratelimit(&rs)) {
 		printk("%s%s Hypervisor Maintenance interrupt [%s]\n",
 			level, sevstr,
 			hmi_evt->disposition = OpalHMI_DISPOSITION_RECOVERED ?

cheers

#12 lore raw
In reply to Nicholas Piggin
Quoted 9 lines · Nicholas Piggin

First patch is a nasty memory scribble introduced by me :( That should go into fixes.

The next ones could wait for next merge window. They get things to the point where misbehaving or buggy guest isn't so painful for the host, and also get the guest SLB dumping code working (because the host no longer clears them before delivering the MCE to the guest).

[...]

Patch 1 applied to powerpc/fixes.

[1/8] powerpc/64s/powernv: Fix memory corruption when saving SLB entries on MCE

      https://git.kernel.org/powerpc/c/a1ee28117077c3bf24e5ab6324c835eaab629c45

cheers

#13 lore raw
In reply to Nicholas Piggin
Quoted 9 lines · Nicholas Piggin

First patch is a nasty memory scribble introduced by me :( That should go into fixes.

The next ones could wait for next merge window. They get things to the point where misbehaving or buggy guest isn't so painful for the host, and also get the guest SLB dumping code working (because the host no longer clears them before delivering the MCE to the guest).

[...]

Patches 2-8 applied to powerpc/next.

[2/8] powerpc/64s/powernv: Allow KVM to handle guest machine check details

      https://git.kernel.org/powerpc/c/0ce2382657f39ced2adbb927355360c3aaeb05f8
[3/8] KVM: PPC: Book3S HV: Don't attempt to recover machine checks for FWNMI enabled guests
      https://git.kernel.org/powerpc/c/067c9f9c98c8804b07751994c51d8557e440821e
[4/8] KVM: PPC: Book3S HV: Ratelimit machine check messages coming from guests
      https://git.kernel.org/powerpc/c/1d15ffdfc94127d75e04a88344ee1ce8c79f05fd
[5/8] powerpc/64s/powernv: Ratelimit harmless HMI error printing
      https://git.kernel.org/powerpc/c/f4b239e4c6bddf63d00cd460eabb933232dbc326
[6/8] powerpc/64s/pseries: Add ERAT specific machine check handler
      https://git.kernel.org/powerpc/c/82f70a05108c98aea4f140067c44a606262d2af7
[7/8] powerpc/64s: Remove "Host" from MCE logging
      https://git.kernel.org/powerpc/c/4a869531ddbf5939c45eab6ff389e4e58c8ed19c
[8/8] powerpc/64s: Tidy machine check SLB logging
      https://git.kernel.org/powerpc/c/865ae6f27789dcc3f92341d935f4439e8730a9fe

cheers