Skip to contentSkip to conversation

arm64: Restore trapless ptimer access

PATCH Applied Colton Lewis 7 participants 11 messages Aug 31, 2023 – Sep 8, 2023 KVM

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 arm64: Restore trapless ptimer access

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9
for additional context.

---
 target/arm/kvm64.c | 1 +
 1 file changed, 1 insertion(+)
target/arm/kvm64.c +1 -0
index 4d904a1d11..2dd46e0a99 100644
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
  */
 static const CPRegStateLevel non_runtime_cpregs[] = {
     { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
 };
 
 int kvm_arm_cpreg_level(uint64_t regidx)
Signature · 1 line
2.42.0.283.g2d96d420d3-goog
Reviewed-by on v1 1/1
#1 lore raw
Quoted 16 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

Quoted 5 lines

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9
for additional context.

r~

Quoted 16 lines
---
  target/arm/kvm64.c | 1 +
  1 file changed, 1 insertion(+)
target/arm/kvm64.c +1 -0
index 4d904a1d11..2dd46e0a99 100644
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
   */
  static const CPRegStateLevel non_runtime_cpregs[] = {
      { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
  };
 
  int kvm_arm_cpreg_level(uint64_t regidx)
#2 lore raw
In reply to Colton Lewis
Quoted 16 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

Quoted 2 lines

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9

The link can be shorter with

https://lore.kernel.org/all/[email protected]/

Quoted 3 lines

for additional context.

Thanks for the testing and posting, Colton. Please add your s-o-b and a Tested-by tag as well.

Thanks,
drew

Quoted 19 lines
---
 target/arm/kvm64.c | 1 +
 1 file changed, 1 insertion(+)
target/arm/kvm64.c +1 -0
index 4d904a1d11..2dd46e0a99 100644
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
  */
 static const CPRegStateLevel non_runtime_cpregs[] = {
     { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
 };
 
 int kvm_arm_cpreg_level(uint64_t regidx)
Signature · 1 line
2.42.0.283.g2d96d420d3-goog
author Tested-by
#3 lore raw
Quoted 17 lines · Andrew Jones
Quoted 16 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

Quoted 6 lines
Quoted 2 lines

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9

The link can be shorter with

https://lore.kernel.org/all/[email protected]/

I will keep that in mind next time.

Quoted 6 lines
Quoted 3 lines

for additional context.

Thanks for the testing and posting, Colton. Please add your s-o-b and a Tested-by tag as well.

Assuming it is sufficient to add here instead of reposting the whole patch:

Quoted 19 lines
Quoted 19 lines
---
 target/arm/kvm64.c | 1 +
 1 file changed, 1 insertion(+)
target/arm/kvm64.c +1 -1
index 4d904a1d11..2dd46e0a99 100644
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
  */
 static const CPRegStateLevel non_runtime_cpregs[] = {
     { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
 };
 
 int kvm_arm_cpreg_level(uint64_t regidx)
--

2.42.0.283.g2d96d420d3-goog

#4 lore raw

Hi,

I think this discussion from ~2015 could potentially be be historically relevant for context,
at the time we had the problem with CNTVOFF IIRC so KVM_REG_ARM_TIMER_CNT being read and rewritten causing time warps in the guest:

https://patchwork.kernel.org/project/linux-arm-kernel/patch/[email protected]/

I could not remember or find if/where the problem was fixed in the end in QEMU,

Ciao,

Claudio

Quoted 59 lines · Colton Lewis
Quoted 17 lines · Andrew Jones
Quoted 16 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

Quoted 6 lines
Quoted 2 lines

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9

The link can be shorter with

https://lore.kernel.org/all/[email protected]/

I will keep that in mind next time.

Quoted 6 lines
Quoted 3 lines

for additional context.

Thanks for the testing and posting, Colton. Please add your s-o-b and a Tested-by tag as well.

Assuming it is sufficient to add here instead of reposting the whole patch:

Quoted 19 lines
Quoted 19 lines
---
 target/arm/kvm64.c | 1 +
 1 file changed, 1 insertion(+)
target/arm/kvm64.c +1 -1
index 4d904a1d11..2dd46e0a99 100644
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
  */
 static const CPRegStateLevel non_runtime_cpregs[] = {
     { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
 };
 
 int kvm_arm_cpreg_level(uint64_t regidx)
--

2.42.0.283.g2d96d420d3-goog

#5 lore raw
Quoted 8 lines · Claudio Fontana

Hi,

I think this discussion from ~2015 could potentially be be historically relevant for context,
at the time we had the problem with CNTVOFF IIRC so KVM_REG_ARM_TIMER_CNT being read and rewritten causing time warps in the guest:

https://patchwork.kernel.org/project/linux-arm-kernel/patch/[email protected]/

I could not remember or find if/where the problem was fixed in the end in QEMU,

It's most likely commit 4b7a6bf402bd ("target-arm: kvm: Differentiate registers based on write-back levels")

Thanks,
drew

Quoted 66 lines

Ciao,

Claudio

Quoted 59 lines · Colton Lewis
Quoted 17 lines · Andrew Jones
Quoted 16 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

Quoted 6 lines
Quoted 2 lines

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9

The link can be shorter with

https://lore.kernel.org/all/[email protected]/

I will keep that in mind next time.

Quoted 6 lines
Quoted 3 lines

for additional context.

Thanks for the testing and posting, Colton. Please add your s-o-b and a Tested-by tag as well.

Assuming it is sufficient to add here instead of reposting the whole patch:

Quoted 19 lines
Quoted 19 lines
---
 target/arm/kvm64.c | 1 +
 1 file changed, 1 insertion(+)
target/arm/kvm64.c +1 -1
index 4d904a1d11..2dd46e0a99 100644
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
  */
 static const CPRegStateLevel non_runtime_cpregs[] = {
     { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
 };
 
 int kvm_arm_cpreg_level(uint64_t regidx)
--

2.42.0.283.g2d96d420d3-goog

#6 lore raw
Quoted 6 lines · Andrew Jones
Quoted 5 lines · Claudio Fontana

Hi,

I think this discussion from ~2015 could potentially be be historically relevant for context,
at the time we had the problem with CNTVOFF IIRC so KVM_REG_ARM_TIMER_CNT being read and rewritten causing time warps in the guest:

Quoted 6 lines
Quoted 3 lines

https://patchwork.kernel.org/project/linux-arm-kernel/patch/[email protected]/

I could not remember or find if/where the problem was fixed in the end in QEMU,

It's most likely commit 4b7a6bf402bd ("target-arm: kvm: Differentiate registers based on write-back levels")

Indeed, thanks!

C

Quoted 69 lines

Thanks,
drew

Quoted 66 lines

Ciao,

Claudio

Quoted 59 lines · Colton Lewis
Quoted 17 lines · Andrew Jones
Quoted 16 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

Quoted 6 lines
Quoted 2 lines

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9

The link can be shorter with

https://lore.kernel.org/all/[email protected]/

I will keep that in mind next time.

Quoted 6 lines
Quoted 3 lines

for additional context.

Thanks for the testing and posting, Colton. Please add your s-o-b and a Tested-by tag as well.

Assuming it is sufficient to add here instead of reposting the whole patch:

Quoted 19 lines
Quoted 19 lines
---
 target/arm/kvm64.c | 1 +
 1 file changed, 1 insertion(+)
target/arm/kvm64.c +1 -1
index 4d904a1d11..2dd46e0a99 100644
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
  */
 static const CPRegStateLevel non_runtime_cpregs[] = {
     { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
 };
 
 int kvm_arm_cpreg_level(uint64_t regidx)
--

2.42.0.283.g2d96d420d3-goog

#7 lore raw
In reply to Colton Lewis
Quoted 30 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9
for additional context.

---
  target/arm/kvm64.c | 1 +
  1 file changed, 1 insertion(+)
target/arm/kvm64.c +0 -0
index 4d904a1d11..2dd46e0a99 100644

target/arm/kvm64.c typedef struct CPRegStateLevel { · line 678

Quoted 7 lines
target/arm/kvm64.c +1 -0
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
   */
  static const CPRegStateLevel non_runtime_cpregs[] = {
      { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
  };
 
  int kvm_arm_cpreg_level(uint64_t regidx)

While this patch itself is one-liner and trivial and all, I'd rather not apply this to the trivial-patches tree, - it requires a little bit more than trivial expertise in this area.

So basically, ping for qemu-arm@ ? :)

Thanks,

/mjt

#8 lore raw
Quoted 42 lines · Michael Tokarev
Quoted 30 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9
for additional context.

---
  target/arm/kvm64.c | 1 +
  1 file changed, 1 insertion(+)
target/arm/kvm64.c +0 -0
index 4d904a1d11..2dd46e0a99 100644

target/arm/kvm64.c typedef struct CPRegStateLevel { · line 677

Quoted 6 lines
target/arm/kvm64.c +1 -0
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
   */
  static const CPRegStateLevel non_runtime_cpregs[] = {
      { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
  };
  int kvm_arm_cpreg_level(uint64_t regidx)

While this patch itself is one-liner and trivial and all, I'd rather not apply this to the trivial-patches tree, - it requires a little bit more than trivial expertise in this area.

Quoted 2 lines

So basically, ping for qemu-arm@ ? :)

I agree that qemu-trivial should not have been CC'ed for this patch.

Thanks,
drew

#9 lore raw
In reply to Michael Tokarev
Quoted 38 lines · Michael Tokarev
Quoted 36 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9
for additional context.

---
  target/arm/kvm64.c | 1 +
  1 file changed, 1 insertion(+)
target/arm/kvm64.c +1 -0
index 4d904a1d11..2dd46e0a99 100644
@@ -672,6 +672,7 @@ typedef struct CPRegStateLevel {
   */
  static const CPRegStateLevel non_runtime_cpregs[] = {
      { KVM_REG_ARM_TIMER_CNT, KVM_PUT_FULL_STATE },
+    { KVM_REG_ARM_PTIMER_CNT, KVM_PUT_FULL_STATE },
  };
 
Quoted 7 lines

target/arm/kvm64.c typedef struct CPRegStateLevel { · line 678

Quoted 1 line
target/arm/kvm64.c +0 -0
  int kvm_arm_cpreg_level(uint64_t regidx)

While this patch itself is one-liner and trivial and all, I'd rather not apply this to the trivial-patches tree, - it requires a little bit more than trivial expertise in this area.

So basically, ping for qemu-arm@ ? :)

It is on my to-review/apply queue, yes.

thanks
-- PMM

#10 lore raw
In reply to Colton Lewis
Quoted 17 lines · Colton Lewis

Due to recent KVM changes, QEMU is setting a ptimer offset resulting in unintended trap and emulate access and a consequent performance hit. Filter out the PTIMER_CNT register to restore trapless ptimer access.

Quoting Andrew Jones:

Simply reading the CNT register and writing back the same value is enough to set an offset, since the timer will have certainly moved past whatever value was read by the time it's written. QEMU frequently saves and restores all registers in the get-reg-list array, unless they've been explicitly filtered out (with Linux commit 680232a94c12, KVM_REG_ARM_PTIMER_CNT is now in the array). So, to restore trapless ptimer accesses, we need a QEMU patch to filter out the register.

Quoted 6 lines

See
https://lore.kernel.org/kvmarm/[email protected]/T/#m0770023762a821db2a3f0dd0a7dc6aa54e0d0da9
for additional context.

---

Applied to target-arm.next, thanks.

-- PMM