Skip to contentSkip to conversation

arm: pmu: Fixes for bare metal

PATCH Superseded Ricardo Koller 5 participants 11 messages Aug 3, 2022 – Aug 8, 2022

Subsystems (2)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover arm: pmu: Fixes for bare metal

There are some tests that fail when running on bare metal (including a passthrough prototype). There are three issues with the tests. The first one is that there are some missing isb()'s between enabling event counting and the actual counting. This wasn't an issue on KVM as trapping on registers served as context synchronization events. The second issue is that some tests assume that registers reset to 0. And finally, the third issue is that overflowing the low counter of a chained event sets the overflow flag in PMVOS and some tests fail by checking for it not being set.

Addressed all comments from the previous version:
- added some isb()'s as suggested by Alexandru.
- fixed a couple of confusing comments (Alexandru).
- check for overflow in the low counter in the interrupt_overflow test (Marc).

Thanks!
Ricardo

Ricardo Koller (3):
arm: pmu: Add missing isb()'s after sys register writing
arm: pmu: Reset the pmu registers before starting some tests
arm: pmu: Check for overflow in the low counter in chained counters
tests

 arm/pmu.c | 55 ++++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 38 insertions(+), 17 deletions(-)
Signature · 6 lines
2.37.1.455.g008518b4e5-goog

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#1 lore raw

PATCH v2 1/3 arm: pmu: Add missing isb()'s after sys register writing

There are various pmu tests that require an isb() between enabling counting and the actual counting. This can lead to count registers reporting less events than expected; the actual enabling happens after some events have happened. For example, some missing isb()'s in the pmu-sw-incr test lead to the following errors on bare-metal:

INFO: pmu: pmu-sw-incr: SW_INCR counter #0 has value 4294967280
PASS: pmu: pmu-sw-incr: PWSYNC does not increment if PMCR.E is unset
FAIL: pmu: pmu-sw-incr: counter #1 after + 100 SW_INCR
FAIL: pmu: pmu-sw-incr: counter #0 after + 100 SW_INCR
INFO: pmu: pmu-sw-incr: counter values after 100 SW_INCR #0=82 #1=98
PASS: pmu: pmu-sw-incr: overflow on counter #0 after 100 SW_INCR
SUMMARY: 4 tests, 2 unexpected failures

Add the missing isb()'s on all failing tests, plus some others that seem required:
- after clearing the overflow signal in the IRQ handler to avoid spurious interrupts.
- after direct writes to PMSWINC_EL0 for software to read the correct value for PMEVNCTR0_EL0 (from ARM DDI 0487H.a, page D13-5237).

---
 arm/pmu.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arm/pmu.c +16 -0
index 15c542a2..76156f78 100644
@@ -307,6 +307,7 @@ static void irq_handler(struct pt_regs *regs)
 			}
 		}
 		write_sysreg(ALL_SET, pmovsclr_el0);
+		isb();
 	} else {
 		pmu_stats.unexpected = true;
 	}
@@ -534,10 +535,12 @@ static void test_sw_incr(void)
 	write_sysreg_s(0x3, PMCNTENSET_EL0);
 
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
+	isb();
 
 	for (i = 0; i < 100; i++)
 		write_sysreg(0x1, pmswinc_el0);
 
+	isb();
 	report_info("SW_INCR counter #0 has value %ld", read_regn_el0(pmevcntr, 0));
 	report(read_regn_el0(pmevcntr, 0) == PRE_OVERFLOW,
 		"PWSYNC does not increment if PMCR.E is unset");
@@ -547,10 +550,12 @@ static void test_sw_incr(void)
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
 	write_sysreg_s(0x3, PMCNTENSET_EL0);
 	set_pmcr(pmu.pmcr_ro | PMU_PMCR_E);
+	isb();
 
 	for (i = 0; i < 100; i++)
 		write_sysreg(0x3, pmswinc_el0);
 
+	isb();
 	report(read_regn_el0(pmevcntr, 0)  == 84, "counter #1 after + 100 SW_INCR");
 	report(read_regn_el0(pmevcntr, 1)  == 100,
 		"counter #0 after + 100 SW_INCR");
@@ -618,9 +623,12 @@ static void test_chained_sw_incr(void)
 
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
 	set_pmcr(pmu.pmcr_ro | PMU_PMCR_E);
+	isb();
+
 	for (i = 0; i < 100; i++)
 		write_sysreg(0x1, pmswinc_el0);
 
+	isb();
 	report(!read_sysreg(pmovsclr_el0) && (read_regn_el0(pmevcntr, 1) == 1),
 		"no overflow and chain counter incremented after 100 SW_INCR/CHAIN");
 	report_info("overflow=0x%lx, #0=%ld #1=%ld", read_sysreg(pmovsclr_el0),
@@ -634,9 +642,12 @@ static void test_chained_sw_incr(void)
 	write_regn_el0(pmevcntr, 1, ALL_SET);
 	write_sysreg_s(0x3, PMCNTENSET_EL0);
 	set_pmcr(pmu.pmcr_ro | PMU_PMCR_E);
+	isb();
+
 	for (i = 0; i < 100; i++)
 		write_sysreg(0x1, pmswinc_el0);
 
+	isb();
 	report((read_sysreg(pmovsclr_el0) == 0x2) &&
 		(read_regn_el0(pmevcntr, 1) == 0) &&
 		(read_regn_el0(pmevcntr, 0) == 84),
@@ -821,10 +832,13 @@ static void test_overflow_interrupt(void)
 	report(expect_interrupts(0), "no overflow interrupt after preset");
 
 	set_pmcr(pmu.pmcr_ro | PMU_PMCR_E);
+	isb();
+
 	for (i = 0; i < 100; i++)
 		write_sysreg(0x2, pmswinc_el0);
 
 	set_pmcr(pmu.pmcr_ro);
+	isb();
 	report(expect_interrupts(0), "no overflow interrupt after counting");
 
 	/* enable interrupts */
@@ -879,6 +893,7 @@ static bool check_cycles_increase(void)
 	set_pmccfiltr(0); /* count cycles in EL0, EL1, but not EL2 */
 
 	set_pmcr(get_pmcr() | PMU_PMCR_LC | PMU_PMCR_C | PMU_PMCR_E);
+	isb();
 
 	for (int i = 0; i < NR_SAMPLES; i++) {
 		uint64_t a, b;
@@ -894,6 +909,7 @@ static bool check_cycles_increase(void)
 	}
 
 	set_pmcr(get_pmcr() & ~PMU_PMCR_E);
+	isb();
 
 	return success;
 }
Signature · 1 line
2.37.1.455.g008518b4e5-goog
#2 lore raw

Re: [kvm-unit-tests PATCH v2 1/3] arm: pmu: Add missing isb()'s after sys register writing

Hi,

Quoted 14 lines · Ricardo Koller

There are various pmu tests that require an isb() between enabling counting and the actual counting. This can lead to count registers reporting less events than expected; the actual enabling happens after some events have happened. For example, some missing isb()'s in the pmu-sw-incr test lead to the following errors on bare-metal:

INFO: pmu: pmu-sw-incr: SW_INCR counter #0 has value 4294967280
PASS: pmu: pmu-sw-incr: PWSYNC does not increment if PMCR.E is unset
FAIL: pmu: pmu-sw-incr: counter #1 after + 100 SW_INCR
FAIL: pmu: pmu-sw-incr: counter #0 after + 100 SW_INCR
INFO: pmu: pmu-sw-incr: counter values after 100 SW_INCR #0=82 #1=98
PASS: pmu: pmu-sw-incr: overflow on counter #0 after 100 SW_INCR
SUMMARY: 4 tests, 2 unexpected failures

Quoted 4 lines

Add the missing isb()'s on all failing tests, plus some others that seem required:
- after clearing the overflow signal in the IRQ handler to avoid spurious interrupts.

Nitpick, but it doesn't avoid (eliminates) spurious interrupts, it makes them less likely.

Quoted 14 lines

- after direct writes to PMSWINC_EL0 for software to read the correct value for PMEVNCTR0_EL0 (from ARM DDI 0487H.a, page D13-5237).

---
 arm/pmu.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arm/pmu.c +0 -0
index 15c542a2..76156f78 100644
[..]

arm/pmu.c static void test_overflow_interrupt(void) · line 841

Quoted 10 lines
arm/pmu.c +3 -0
@@ -821,10 +832,13 @@ static void test_overflow_interrupt(void)
 	report(expect_interrupts(0), "no overflow interrupt after preset");
 
 	set_pmcr(pmu.pmcr_ro | PMU_PMCR_E);
+	isb();
+
 	for (i = 0; i < 100; i++)
 		write_sysreg(0x2, pmswinc_el0);
 
 	set_pmcr(pmu.pmcr_ro);
+	isb();

A context synchronization event affects system register writes that come before the context synchronization event in program order, but if there are multiple system register writes, it doesn't perform them in program order (if that makes sense).

So it might happen that the CPU decides to perform the write to PMCR_EL1 which disables the PMU *before* the writes to PMSWINC_EL0. Which means that even if PMINTENSET_EL1 allows the PMU to assert interrupts when it shouldn't (thus causing the test to fail), those interrupt won't be asserted by the PMU because the PMU is disabled and the test would pass.

You need an ISB after the PMSWINC_EL0 writes and before disabling the PMU.

Thanks,
Alex

arm/pmu.c static void test_overflow_interrupt(void) · line 842

Quoted 1 line
arm/pmu.c +0 -0
 	report(expect_interrupts(0), "no overflow interrupt after counting");

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#3 lore raw

Re: [kvm-unit-tests PATCH v2 1/3] arm: pmu: Add missing isb()'s after sys register writing

Quoted 47 lines · Alexandru Elisei

Hi,

Quoted 14 lines · Ricardo Koller

There are various pmu tests that require an isb() between enabling counting and the actual counting. This can lead to count registers reporting less events than expected; the actual enabling happens after some events have happened. For example, some missing isb()'s in the pmu-sw-incr test lead to the following errors on bare-metal:

INFO: pmu: pmu-sw-incr: SW_INCR counter #0 has value 4294967280
PASS: pmu: pmu-sw-incr: PWSYNC does not increment if PMCR.E is unset
FAIL: pmu: pmu-sw-incr: counter #1 after + 100 SW_INCR
FAIL: pmu: pmu-sw-incr: counter #0 after + 100 SW_INCR
INFO: pmu: pmu-sw-incr: counter values after 100 SW_INCR #0=82 #1=98
PASS: pmu: pmu-sw-incr: overflow on counter #0 after 100 SW_INCR
SUMMARY: 4 tests, 2 unexpected failures

Quoted 4 lines

Add the missing isb()'s on all failing tests, plus some others that seem required:
- after clearing the overflow signal in the IRQ handler to avoid spurious interrupts.

Nitpick, but it doesn't avoid (eliminates) spurious interrupts, it makes them less likely.

Quoted 22 lines

- after direct writes to PMSWINC_EL0 for software to read the correct value for PMEVNCTR0_EL0 (from ARM DDI 0487H.a, page D13-5237).

---
 arm/pmu.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arm/pmu.c +2 -0
index 15c542a2..76156f78 100644
[..]
@@ -821,10 +832,13 @@ static void test_overflow_interrupt(void)
 	report(expect_interrupts(0), "no overflow interrupt after preset");
 
 	set_pmcr(pmu.pmcr_ro | PMU_PMCR_E);
+	isb();
+
 	for (i = 0; i < 100; i++)
 		write_sysreg(0x2, pmswinc_el0);
 
Quoted 7 lines

arm/pmu.c static void test_overflow_interrupt(void) · line 841

Quoted 2 lines
arm/pmu.c +1 -0
 	set_pmcr(pmu.pmcr_ro);
+	isb();

A context synchronization event affects system register writes that come before the context synchronization event in program order, but if there are multiple system register writes, it doesn't perform them in program order (if that makes sense).

Good point, didn't think of that case. Added the missing isb() in v3.

Thanks,
Ricardo

Quoted 13 lines

So it might happen that the CPU decides to perform the write to PMCR_EL1 which disables the PMU *before* the writes to PMSWINC_EL0. Which means that even if PMINTENSET_EL1 allows the PMU to assert interrupts when it shouldn't (thus causing the test to fail), those interrupt won't be asserted by the PMU because the PMU is disabled and the test would pass.

You need an ISB after the PMSWINC_EL0 writes and before disabling the PMU.

Thanks,
Alex

Quoted 1 line
arm/pmu.c +0 -0
 	report(expect_interrupts(0), "no overflow interrupt after counting");
#4 lore raw

PATCH v2 2/3 arm: pmu: Reset the pmu registers before starting some tests

In reply to Ricardo Koller

Some registers like the PMOVS reset to an architecturally UNKNOWN value. Most tests expect them to be reset (mostly zeroed) using pmu_reset(). Add a pmu_reset() on all the tests that need one.

As a bonus, fix a couple of comments related to the register state before a sub-test.

---
 arm/pmu.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arm/pmu.c +4 -2
index 76156f78..7c5bc259 100644
@@ -826,7 +826,7 @@ static void test_overflow_interrupt(void)
 	write_regn_el0(pmevcntr, 1, PRE_OVERFLOW);
 	isb();
 
-	/* interrupts are disabled */
+	/* interrupts are disabled (PMINTENSET_EL1 == 0) */
 
 	mem_access_loop(addr, 200, pmu.pmcr_ro | PMU_PMCR_E);
 	report(expect_interrupts(0), "no overflow interrupt after preset");
@@ -841,7 +841,7 @@ static void test_overflow_interrupt(void)
 	isb();
 	report(expect_interrupts(0), "no overflow interrupt after counting");
 
-	/* enable interrupts */
+	/* enable interrupts (PMINTENSET_EL1 <= ALL_SET) */
 
 	pmu_reset_stats();
 
@@ -889,6 +889,7 @@ static bool check_cycles_increase(void)
 	bool success = true;
 
 	/* init before event access, this test only cares about cycle count */
+	pmu_reset();
 	set_pmcntenset(1 << PMU_CYCLE_IDX);
 	set_pmccfiltr(0); /* count cycles in EL0, EL1, but not EL2 */
 
@@ -943,6 +944,7 @@ static bool check_cpi(int cpi)
 	uint32_t pmcr = get_pmcr() | PMU_PMCR_LC | PMU_PMCR_C | PMU_PMCR_E;
 
 	/* init before event access, this test only cares about cycle count */
+	pmu_reset();
 	set_pmcntenset(1 << PMU_CYCLE_IDX);
 	set_pmccfiltr(0); /* count cycles in EL0, EL1, but not EL2 */
 
Signature · 1 line
2.37.1.455.g008518b4e5-goog
Reviewed-by on v2 2/3
#5 lore raw

Re: [kvm-unit-tests PATCH v2 2/3] arm: pmu: Reset the pmu registers before starting some tests

Hi Ricardo,

Quoted 8 lines · Ricardo Koller

Some registers like the PMOVS reset to an architecturally UNKNOWN value. Most tests expect them to be reset (mostly zeroed) using pmu_reset(). Add a pmu_reset() on all the tests that need one.

As a bonus, fix a couple of comments related to the register state before a sub-test.

Eric

Quoted 35 lines
---
 arm/pmu.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arm/pmu.c +3 -2
index 76156f78..7c5bc259 100644
@@ -826,7 +826,7 @@ static void test_overflow_interrupt(void)
 	write_regn_el0(pmevcntr, 1, PRE_OVERFLOW);
 	isb();
 
-	/* interrupts are disabled */
+	/* interrupts are disabled (PMINTENSET_EL1 == 0) */
 
 	mem_access_loop(addr, 200, pmu.pmcr_ro | PMU_PMCR_E);
 	report(expect_interrupts(0), "no overflow interrupt after preset");
@@ -841,7 +841,7 @@ static void test_overflow_interrupt(void)
 	isb();
 	report(expect_interrupts(0), "no overflow interrupt after counting");
 
-	/* enable interrupts */
+	/* enable interrupts (PMINTENSET_EL1 <= ALL_SET) */
 
 	pmu_reset_stats();
 
@@ -889,6 +889,7 @@ static bool check_cycles_increase(void)
 	bool success = true;
 
 	/* init before event access, this test only cares about cycle count */
+	pmu_reset();
 	set_pmcntenset(1 << PMU_CYCLE_IDX);
 	set_pmccfiltr(0); /* count cycles in EL0, EL1, but not EL2 */
 

arm/pmu.c static bool check_cpi(int cpi) · line 950

Quoted 7 lines
arm/pmu.c +1 -0
@@ -943,6 +944,7 @@ static bool check_cpi(int cpi)
 	uint32_t pmcr = get_pmcr() | PMU_PMCR_LC | PMU_PMCR_C | PMU_PMCR_E;
 
 	/* init before event access, this test only cares about cycle count */
+	pmu_reset();
 	set_pmcntenset(1 << PMU_CYCLE_IDX);
 	set_pmccfiltr(0); /* count cycles in EL0, EL1, but not EL2 */
 

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#6 lore raw

PATCH v2 3/3 arm: pmu: Check for overflow in the low counter in chained counters tests

In reply to Ricardo Koller

A chained event overflowing on the low counter can set the overflow flag in PMOVS. KVM does not set it, but real HW and the fast-model seem to. Moreover, the AArch64.IncrementEventCounter() pseudocode in the ARM ARM (DDI 0487H.a, J1.1.1 "aarch64/debug") also sets the PMOVS bit on overflow.

The pmu chain tests fail on bare metal when checking the overflow flag of the low counter _not_ being set on overflow. Fix by checking for overflow. Note that this test fails in KVM without the respective fix.

---
 arm/pmu.c | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)
arm/pmu.c +18 -15
index 7c5bc259..258780f4 100644
@@ -583,7 +583,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 
 	report(read_regn_el0(pmevcntr, 1) == 1, "CHAIN counter #1 incremented");
-	report(!read_sysreg(pmovsclr_el0), "no overflow recorded for chained incr #1");
+	report(read_sysreg(pmovsclr_el0) == 0x1, "overflow recorded for chained incr #1");
 
 	/* test 64b overflow */
 
@@ -595,7 +595,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 	report_info("overflow reg = 0x%lx", read_sysreg(pmovsclr_el0));
 	report(read_regn_el0(pmevcntr, 1) == 2, "CHAIN counter #1 set to 2");
-	report(!read_sysreg(pmovsclr_el0), "no overflow recorded for chained incr #2");
+	report(read_sysreg(pmovsclr_el0) == 0x1, "overflow recorded for chained incr #2");
 
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
 	write_regn_el0(pmevcntr, 1, ALL_SET);
@@ -603,7 +603,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 	report_info("overflow reg = 0x%lx", read_sysreg(pmovsclr_el0));
 	report(!read_regn_el0(pmevcntr, 1), "CHAIN counter #1 wrapped");
-	report(read_sysreg(pmovsclr_el0) == 0x2, "overflow on chain counter");
+	report(read_sysreg(pmovsclr_el0) == 0x3, "overflow on even and odd counters");
 }
 
 static void test_chained_sw_incr(void)
@@ -629,8 +629,9 @@ static void test_chained_sw_incr(void)
 		write_sysreg(0x1, pmswinc_el0);
 
 	isb();
-	report(!read_sysreg(pmovsclr_el0) && (read_regn_el0(pmevcntr, 1) == 1),
-		"no overflow and chain counter incremented after 100 SW_INCR/CHAIN");
+	report((read_sysreg(pmovsclr_el0) == 0x1) &&
+		(read_regn_el0(pmevcntr, 1) == 1),
+		"overflow and chain counter incremented after 100 SW_INCR/CHAIN");
 	report_info("overflow=0x%lx, #0=%ld #1=%ld", read_sysreg(pmovsclr_el0),
 		    read_regn_el0(pmevcntr, 0), read_regn_el0(pmevcntr, 1));
 
@@ -648,10 +649,10 @@ static void test_chained_sw_incr(void)
 		write_sysreg(0x1, pmswinc_el0);
 
 	isb();
-	report((read_sysreg(pmovsclr_el0) == 0x2) &&
+	report((read_sysreg(pmovsclr_el0) == 0x3) &&
 		(read_regn_el0(pmevcntr, 1) == 0) &&
 		(read_regn_el0(pmevcntr, 0) == 84),
-		"overflow on chain counter and expected values after 100 SW_INCR/CHAIN");
+		"overflow on even and odd counters,  and expected values after 100 SW_INCR/CHAIN");
 	report_info("overflow=0x%lx, #0=%ld #1=%ld", read_sysreg(pmovsclr_el0),
 		    read_regn_el0(pmevcntr, 0), read_regn_el0(pmevcntr, 1));
 }
@@ -731,8 +732,9 @@ static void test_chain_promotion(void)
 	report_info("MEM_ACCESS counter #0 has value 0x%lx",
 		    read_regn_el0(pmevcntr, 0));
 
-	report((read_regn_el0(pmevcntr, 1) == 1) && !read_sysreg(pmovsclr_el0),
-		"CHAIN counter enabled: CHAIN counter was incremented and no overflow");
+	report((read_regn_el0(pmevcntr, 1) == 1) &&
+		(read_sysreg(pmovsclr_el0) == 0x1),
+		"CHAIN counter enabled: CHAIN counter was incremented and overflow");
 
 	report_info("CHAIN counter #1 = 0x%lx, overflow=0x%lx",
 		read_regn_el0(pmevcntr, 1), read_sysreg(pmovsclr_el0));
@@ -759,8 +761,9 @@ static void test_chain_promotion(void)
 	report_info("MEM_ACCESS counter #0 has value 0x%lx",
 		    read_regn_el0(pmevcntr, 0));
 
-	report((read_regn_el0(pmevcntr, 1) == 1) && !read_sysreg(pmovsclr_el0),
-		"32b->64b: CHAIN counter incremented and no overflow");
+	report((read_regn_el0(pmevcntr, 1) == 1) &&
+		(read_sysreg(pmovsclr_el0) == 0x1),
+		"32b->64b: CHAIN counter incremented and overflow");
 
 	report_info("CHAIN counter #1 = 0x%lx, overflow=0x%lx",
 		read_regn_el0(pmevcntr, 1), read_sysreg(pmovsclr_el0));
@@ -867,8 +870,8 @@ static void test_overflow_interrupt(void)
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
 	isb();
 	mem_access_loop(addr, 200, pmu.pmcr_ro | PMU_PMCR_E);
-	report(expect_interrupts(0),
-		"no overflow interrupt expected on 32b boundary");
+	report(expect_interrupts(0x1),
+		"expect overflow interrupt on 32b boundary");
 
 	/* overflow on odd counter */
 	pmu_reset_stats();
@@ -876,8 +879,8 @@ static void test_overflow_interrupt(void)
 	write_regn_el0(pmevcntr, 1, ALL_SET);
 	isb();
 	mem_access_loop(addr, 400, pmu.pmcr_ro | PMU_PMCR_E);
-	report(expect_interrupts(0x2),
-		"expect overflow interrupt on odd counter");
+	report(expect_interrupts(0x3),
+		"expect overflow interrupt on even and odd counter");
 }
 #endif
 
Signature · 1 line
2.37.1.455.g008518b4e5-goog
#7 lore raw

Re: [kvm-unit-tests PATCH v2 3/3] arm: pmu: Check for overflow in the low counter in chained counters tests

Quoted 59 lines · Ricardo Koller

A chained event overflowing on the low counter can set the overflow flag in PMOVS. KVM does not set it, but real HW and the fast-model seem to. Moreover, the AArch64.IncrementEventCounter() pseudocode in the ARM ARM (DDI 0487H.a, J1.1.1 "aarch64/debug") also sets the PMOVS bit on overflow.

The pmu chain tests fail on bare metal when checking the overflow flag of the low counter _not_ being set on overflow. Fix by checking for overflow. Note that this test fails in KVM without the respective fix.

---
 arm/pmu.c | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)
arm/pmu.c +6 -5
index 7c5bc259..258780f4 100644
@@ -583,7 +583,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 
 	report(read_regn_el0(pmevcntr, 1) == 1, "CHAIN counter #1 incremented");
-	report(!read_sysreg(pmovsclr_el0), "no overflow recorded for chained incr #1");
+	report(read_sysreg(pmovsclr_el0) == 0x1, "overflow recorded for chained incr #1");
 
 	/* test 64b overflow */
 
@@ -595,7 +595,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 	report_info("overflow reg = 0x%lx", read_sysreg(pmovsclr_el0));
 	report(read_regn_el0(pmevcntr, 1) == 2, "CHAIN counter #1 set to 2");
-	report(!read_sysreg(pmovsclr_el0), "no overflow recorded for chained incr #2");
+	report(read_sysreg(pmovsclr_el0) == 0x1, "overflow recorded for chained incr #2");
 
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
 	write_regn_el0(pmevcntr, 1, ALL_SET);
@@ -603,7 +603,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 	report_info("overflow reg = 0x%lx", read_sysreg(pmovsclr_el0));
 	report(!read_regn_el0(pmevcntr, 1), "CHAIN counter #1 wrapped");
-	report(read_sysreg(pmovsclr_el0) == 0x2, "overflow on chain counter");
+	report(read_sysreg(pmovsclr_el0) == 0x3, "overflow on even and odd counters");
 }
 
 static void test_chained_sw_incr(void)
@@ -629,8 +629,9 @@ static void test_chained_sw_incr(void)
 		write_sysreg(0x1, pmswinc_el0);
 
 	isb();
-	report(!read_sysreg(pmovsclr_el0) && (read_regn_el0(pmevcntr, 1) == 1),
-		"no overflow and chain counter incremented after 100 SW_INCR/CHAIN");
+	report((read_sysreg(pmovsclr_el0) == 0x1) &&
+		(read_regn_el0(pmevcntr, 1) == 1),
+		"overflow and chain counter incremented after 100 SW_INCR/CHAIN");
 	report_info("overflow=0x%lx, #0=%ld #1=%ld", read_sysreg(pmovsclr_el0),
 		    read_regn_el0(pmevcntr, 0), read_regn_el0(pmevcntr, 1));
 

arm/pmu.c static void test_chained_sw_incr(void) · line 655

Quoted 9 lines
arm/pmu.c +2 -2
@@ -648,10 +649,10 @@ static void test_chained_sw_incr(void)
 		write_sysreg(0x1, pmswinc_el0);
 
 	isb();
-	report((read_sysreg(pmovsclr_el0) == 0x2) &&
+	report((read_sysreg(pmovsclr_el0) == 0x3) &&
 		(read_regn_el0(pmevcntr, 1) == 0) &&
 		(read_regn_el0(pmevcntr, 0) == 84),
-		"overflow on chain counter and expected values after 100 SW_INCR/CHAIN");
+		"overflow on even and odd counters,  and expected values after 100 SW_INCR/CHAIN");

Besides the extra space, this doesn't read well (to me).

Thanks,
drew

#8 lore raw

Re: [kvm-unit-tests PATCH v2 3/3] arm: pmu: Check for overflow in the low counter in chained counters tests

Quoted 62 lines · Andrew Jones
Quoted 61 lines · Ricardo Koller

A chained event overflowing on the low counter can set the overflow flag in PMOVS. KVM does not set it, but real HW and the fast-model seem to. Moreover, the AArch64.IncrementEventCounter() pseudocode in the ARM ARM (DDI 0487H.a, J1.1.1 "aarch64/debug") also sets the PMOVS bit on overflow.

The pmu chain tests fail on bare metal when checking the overflow flag of the low counter _not_ being set on overflow. Fix by checking for overflow. Note that this test fails in KVM without the respective fix.

---
 arm/pmu.c | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)
arm/pmu.c +6 -5
index 7c5bc259..258780f4 100644
@@ -583,7 +583,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 
 	report(read_regn_el0(pmevcntr, 1) == 1, "CHAIN counter #1 incremented");
-	report(!read_sysreg(pmovsclr_el0), "no overflow recorded for chained incr #1");
+	report(read_sysreg(pmovsclr_el0) == 0x1, "overflow recorded for chained incr #1");
 
 	/* test 64b overflow */
 
@@ -595,7 +595,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 	report_info("overflow reg = 0x%lx", read_sysreg(pmovsclr_el0));
 	report(read_regn_el0(pmevcntr, 1) == 2, "CHAIN counter #1 set to 2");
-	report(!read_sysreg(pmovsclr_el0), "no overflow recorded for chained incr #2");
+	report(read_sysreg(pmovsclr_el0) == 0x1, "overflow recorded for chained incr #2");
 
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
 	write_regn_el0(pmevcntr, 1, ALL_SET);
@@ -603,7 +603,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 	report_info("overflow reg = 0x%lx", read_sysreg(pmovsclr_el0));
 	report(!read_regn_el0(pmevcntr, 1), "CHAIN counter #1 wrapped");
-	report(read_sysreg(pmovsclr_el0) == 0x2, "overflow on chain counter");
+	report(read_sysreg(pmovsclr_el0) == 0x3, "overflow on even and odd counters");
 }
 
 static void test_chained_sw_incr(void)
@@ -629,8 +629,9 @@ static void test_chained_sw_incr(void)
 		write_sysreg(0x1, pmswinc_el0);
 
 	isb();
-	report(!read_sysreg(pmovsclr_el0) && (read_regn_el0(pmevcntr, 1) == 1),
-		"no overflow and chain counter incremented after 100 SW_INCR/CHAIN");
+	report((read_sysreg(pmovsclr_el0) == 0x1) &&
+		(read_regn_el0(pmevcntr, 1) == 1),
+		"overflow and chain counter incremented after 100 SW_INCR/CHAIN");
 	report_info("overflow=0x%lx, #0=%ld #1=%ld", read_sysreg(pmovsclr_el0),
 		    read_regn_el0(pmevcntr, 0), read_regn_el0(pmevcntr, 1));
 
@@ -648,10 +649,10 @@ static void test_chained_sw_incr(void)
 		write_sysreg(0x1, pmswinc_el0);
 
Quoted 9 lines

arm/pmu.c static void test_chained_sw_incr(void) · line 655

Quoted 7 lines
arm/pmu.c +2 -2
 	isb();
-	report((read_sysreg(pmovsclr_el0) == 0x2) &&
+	report((read_sysreg(pmovsclr_el0) == 0x3) &&
 		(read_regn_el0(pmevcntr, 1) == 0) &&
 		(read_regn_el0(pmevcntr, 0) == 84),
-		"overflow on chain counter and expected values after 100 SW_INCR/CHAIN");
+		"overflow on even and odd counters,  and expected values after 100 SW_INCR/CHAIN");

Besides the extra space, this doesn't read well (to me).

Replaced the sentence with something simpler and hopefully nicer (in v3).

Thanks,
Ricardo

Quoted 3 lines

Thanks,
drew

Reviewed-by on v2 3/3
#9 lore raw

Re: [kvm-unit-tests PATCH v2 3/3] arm: pmu: Check for overflow in the low counter in chained counters tests

In reply to Ricardo Koller

Hi Ricardo,

Quoted 107 lines · Ricardo Koller

A chained event overflowing on the low counter can set the overflow flag in PMOVS. KVM does not set it, but real HW and the fast-model seem to. Moreover, the AArch64.IncrementEventCounter() pseudocode in the ARM ARM (DDI 0487H.a, J1.1.1 "aarch64/debug") also sets the PMOVS bit on overflow.

The pmu chain tests fail on bare metal when checking the overflow flag of the low counter _not_ being set on overflow. Fix by checking for overflow. Note that this test fails in KVM without the respective fix.

---
 arm/pmu.c | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)
arm/pmu.c +16 -13
index 7c5bc259..258780f4 100644
@@ -583,7 +583,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 
 	report(read_regn_el0(pmevcntr, 1) == 1, "CHAIN counter #1 incremented");
-	report(!read_sysreg(pmovsclr_el0), "no overflow recorded for chained incr #1");
+	report(read_sysreg(pmovsclr_el0) == 0x1, "overflow recorded for chained incr #1");
 
 	/* test 64b overflow */
 
@@ -595,7 +595,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 	report_info("overflow reg = 0x%lx", read_sysreg(pmovsclr_el0));
 	report(read_regn_el0(pmevcntr, 1) == 2, "CHAIN counter #1 set to 2");
-	report(!read_sysreg(pmovsclr_el0), "no overflow recorded for chained incr #2");
+	report(read_sysreg(pmovsclr_el0) == 0x1, "overflow recorded for chained incr #2");
 
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
 	write_regn_el0(pmevcntr, 1, ALL_SET);
@@ -603,7 +603,7 @@ static void test_chained_counters(void)
 	precise_instrs_loop(22, pmu.pmcr_ro | PMU_PMCR_E);
 	report_info("overflow reg = 0x%lx", read_sysreg(pmovsclr_el0));
 	report(!read_regn_el0(pmevcntr, 1), "CHAIN counter #1 wrapped");
-	report(read_sysreg(pmovsclr_el0) == 0x2, "overflow on chain counter");
+	report(read_sysreg(pmovsclr_el0) == 0x3, "overflow on even and odd counters");
 }
 
 static void test_chained_sw_incr(void)
@@ -629,8 +629,9 @@ static void test_chained_sw_incr(void)
 		write_sysreg(0x1, pmswinc_el0);
 
 	isb();
-	report(!read_sysreg(pmovsclr_el0) && (read_regn_el0(pmevcntr, 1) == 1),
-		"no overflow and chain counter incremented after 100 SW_INCR/CHAIN");
+	report((read_sysreg(pmovsclr_el0) == 0x1) &&
+		(read_regn_el0(pmevcntr, 1) == 1),
+		"overflow and chain counter incremented after 100 SW_INCR/CHAIN");
 	report_info("overflow=0x%lx, #0=%ld #1=%ld", read_sysreg(pmovsclr_el0),
 		    read_regn_el0(pmevcntr, 0), read_regn_el0(pmevcntr, 1));
 
@@ -648,10 +649,10 @@ static void test_chained_sw_incr(void)
 		write_sysreg(0x1, pmswinc_el0);
 
 	isb();
-	report((read_sysreg(pmovsclr_el0) == 0x2) &&
+	report((read_sysreg(pmovsclr_el0) == 0x3) &&
 		(read_regn_el0(pmevcntr, 1) == 0) &&
 		(read_regn_el0(pmevcntr, 0) == 84),
-		"overflow on chain counter and expected values after 100 SW_INCR/CHAIN");
+		"overflow on even and odd counters,  and expected values after 100 SW_INCR/CHAIN");
 	report_info("overflow=0x%lx, #0=%ld #1=%ld", read_sysreg(pmovsclr_el0),
 		    read_regn_el0(pmevcntr, 0), read_regn_el0(pmevcntr, 1));
 }
@@ -731,8 +732,9 @@ static void test_chain_promotion(void)
 	report_info("MEM_ACCESS counter #0 has value 0x%lx",
 		    read_regn_el0(pmevcntr, 0));
 
-	report((read_regn_el0(pmevcntr, 1) == 1) && !read_sysreg(pmovsclr_el0),
-		"CHAIN counter enabled: CHAIN counter was incremented and no overflow");
+	report((read_regn_el0(pmevcntr, 1) == 1) &&
+		(read_sysreg(pmovsclr_el0) == 0x1),
+		"CHAIN counter enabled: CHAIN counter was incremented and overflow");
 
 	report_info("CHAIN counter #1 = 0x%lx, overflow=0x%lx",
 		read_regn_el0(pmevcntr, 1), read_sysreg(pmovsclr_el0));
@@ -759,8 +761,9 @@ static void test_chain_promotion(void)
 	report_info("MEM_ACCESS counter #0 has value 0x%lx",
 		    read_regn_el0(pmevcntr, 0));
 
-	report((read_regn_el0(pmevcntr, 1) == 1) && !read_sysreg(pmovsclr_el0),
-		"32b->64b: CHAIN counter incremented and no overflow");
+	report((read_regn_el0(pmevcntr, 1) == 1) &&
+		(read_sysreg(pmovsclr_el0) == 0x1),
+		"32b->64b: CHAIN counter incremented and overflow");
 
 	report_info("CHAIN counter #1 = 0x%lx, overflow=0x%lx",
 		read_regn_el0(pmevcntr, 1), read_sysreg(pmovsclr_el0));
@@ -867,8 +870,8 @@ static void test_overflow_interrupt(void)
 	write_regn_el0(pmevcntr, 0, PRE_OVERFLOW);
 	isb();
 	mem_access_loop(addr, 200, pmu.pmcr_ro | PMU_PMCR_E);
-	report(expect_interrupts(0),
-		"no overflow interrupt expected on 32b boundary");
+	report(expect_interrupts(0x1),
+		"expect overflow interrupt on 32b boundary");
 
 	/* overflow on odd counter */
 	pmu_reset_stats();

arm/pmu.c static void test_overflow_interrupt(void) · line 886

Quoted 10 lines
arm/pmu.c +2 -2
@@ -876,8 +879,8 @@ static void test_overflow_interrupt(void)
 	write_regn_el0(pmevcntr, 1, ALL_SET);
 	isb();
 	mem_access_loop(addr, 400, pmu.pmcr_ro | PMU_PMCR_E);
-	report(expect_interrupts(0x2),
-		"expect overflow interrupt on odd counter");
+	report(expect_interrupts(0x3),
+		"expect overflow interrupt on even and odd counter");
 }
 #endif
 

Besides Drew's comment and assuming Marc's fix in sync,

I definitively missed that spec detail when originally writing the test, thank you for fixing.

Eric

maintainer
#10 lore raw

Re: [kvm-unit-tests PATCH v2 3/3] arm: pmu: Check for overflow in the low counter in chained counters tests

Quoted 3 lines · Eric Auger

I definitively missed that spec detail when originally writing the test, thank you for fixing.

To be fair, it is only in a very recent version of the ARM ARM that this detail came to light, and we collectively misunderstood the spec for a few years, resulting in a over-complicated KVM infrastructure (the new emulation code is so much simpler).

BTW, I think I forgot to Cc you and Andrew on the KVM rework at [1], sorry about that.

Thanks,

	M.

[1] https://lore.kernel.org/kvm/[email protected]

Signature · 1 line
Without deviation from the norm, progress is not possible.