Skip to contentSkip to conversation

Fix diagnostics and capability probes for protected VMs

PATCH In review Fuad Tabba 3 participants 17 messages Aug 31 – Oct 9

Subsystems (2)
lore
Share & tools
lore

Fuad Tabba

author reviewer
#0 lore raw

cover Fix diagnostics and capability probes for protected VMs

Hi folks,

These are the kvmtool changes that go with the pKVM series confining a protected VM's vCPU state to EL2 [1].

That series stops the host reading a protected vCPU's registers once it has run, and stops it setting the PVTIME attribute. kvmtool dies on both. Its arm64 register-dump helpers call die() when KVM_GET_ONE_REG fails, which turns a diagnostic into a VMM abort. kvm_cpu__setup_pvtime() runs into the new -EPERM and fails vCPU init.

The PVTIME failure is a probe on the wrong fd. kvmtool asks the global /dev/kvm fd whether steal-time is available, and that fd advertises it for a protected VM, which cannot use it.
The last patch fixes the same wrong-fd probe for the counter offset, where "lkvm run --protected --counter-offset" already fails on Linux 7.3-rc1, without the kernel series.
The same conversion went in for the vCPU feature probes in July [2].

Patches 2 and 3 are not specific to protected VMs. They fix the exit-reason table on the same panic path as patch 1. The lookup ran off the end of it, and the table had stopped naming anything KVM added after KVM_EXIT_INTERNAL_ERROR, which on arm64 includes KVM_EXIT_SYSTEM_EVENT and KVM_EXIT_ARM_NISV.

Patch 1's path needs a protected VM that is already running: "lkvm debug -d" against one kills the VMM without this series, and prints the errno with it.

Based on kvmtool master (f67bc0b).

Cheers,
/fuad

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Fuad Tabba (5):
arm64: Do not abort on register-dump failures
kvm: Bound-check the exit-reason string lookup
kvm: Name every exit reason the UAPI header defines
arm64: Query steal-time support on the VM fd
arm64: Query counter-offset support on the VM fd

 arm64/kvm-cpu.c   | 36 ++++++++++++++++++++++++------------
 arm64/kvm.c       |  2 +-
 arm64/pvtime.c    |  4 ++--
 builtin-run.c     |  2 +-
 include/kvm/kvm.h |  2 +-
 kvm.c             | 37 ++++++++++++++++++++++++++++++++++---
 6 files changed, 63 insertions(+), 20 deletions(-)

base-commit: f67bc0bdae9433a9cfd05e65ea2c1bb6102566d9

Signature · 1 line
2.39.5
author reviewer
#1 lore raw

PATCH 1/5 arm64: Do not abort on register-dump failures

kvm_cpu__show_registers() and kvm_cpu__show_code() read the vCPU's core registers with KVM_GET_ONE_REG, and die() if the ioctl fails. Both are diagnostics. They run from the KVM_EXIT_DEBUG case in kvm_cpu__start(), from the "lkvm debug -d" dump path in handle_sigusr1(), and from the panic dump in kvm_cpu_thread().

Once a protected vCPU has run, its registers belong to the guest and KVM_GET_ONE_REG returns -EPERM. Dying on that turns a diagnostic into a VMM abort.

Report that the register state is unavailable, with the errno, and return instead of dying.

---
 arm64/kvm-cpu.c | 36 ++++++++++++++++++++++++------------
 1 file changed, 24 insertions(+), 12 deletions(-)
arm64/kvm-cpu.c +24 -12
index 3aa7684..c58d61f 100644
@@ -476,15 +476,19 @@ void kvm_cpu__show_code(struct kvm_cpu *vcpu)
 
 	dprintf(debug_fd, "\n*pc:\n");
 	reg.id = ARM64_CORE_REG(regs.pc);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (show_code @ PC)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pc): %s", strerror(errno));
+		return;
+	}
 
 	kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
 
 	dprintf(debug_fd, "\n*lr:\n");
 	reg.id = ARM64_CORE_REG(regs.regs[30]);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (show_code @ LR)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (lr): %s", strerror(errno));
+		return;
+	}
 
 	kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
 }
@@ -499,23 +503,31 @@ void kvm_cpu__show_registers(struct kvm_cpu *vcpu)
 	dprintf(debug_fd, "\n Registers:\n");
 
 	reg.id		= ARM64_CORE_REG(regs.pc);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (pc)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pc): %s", strerror(errno));
+		return;
+	}
 	dprintf(debug_fd, " PC:    0x%lx\n", data);
 
 	reg.id		= ARM64_CORE_REG(regs.pstate);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (pstate)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pstate): %s", strerror(errno));
+		return;
+	}
 	dprintf(debug_fd, " PSTATE:    0x%lx\n", data);
 
 	reg.id		= ARM64_CORE_REG(sp_el1);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (sp_el1)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (sp_el1): %s", strerror(errno));
+		return;
+	}
 	dprintf(debug_fd, " SP_EL1:    0x%lx\n", data);
 
 	reg.id		= ARM64_CORE_REG(regs.regs[30]);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (lr)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (lr): %s", strerror(errno));
+		return;
+	}
 	dprintf(debug_fd, " LR:    0x%lx\n", data);
 }
 
Signature · 1 line
2.39.5
reviewer Reviewed-by on v1 1/5
#2 lore raw

Re: [PATCH kvmtool 1/5] arm64: Do not abort on register-dump failures

Quoted 9 lines · Fuad Tabba

kvm_cpu__show_registers() and kvm_cpu__show_code() read the vCPU's core registers with KVM_GET_ONE_REG, and die() if the ioctl fails. Both are diagnostics. They run from the KVM_EXIT_DEBUG case in kvm_cpu__start(), from the "lkvm debug -d" dump path in handle_sigusr1(), and from the panic dump in kvm_cpu_thread().

Once a protected vCPU has run, its registers belong to the guest and KVM_GET_ONE_REG returns -EPERM. Dying on that turns a diagnostic into a VMM abort.

Marc once suggested that these could always succeed with junk values for protected/Realm VMs. Do you think that is an option for pKVM ?

If we go with this approach :

Quoted 78 lines

Report that the register state is unavailable, with the errno, and return instead of dying.

---
  arm64/kvm-cpu.c | 36 ++++++++++++++++++++++++------------
  1 file changed, 24 insertions(+), 12 deletions(-)
arm64/kvm-cpu.c +8 -4
index 3aa7684..c58d61f 100644
@@ -476,15 +476,19 @@ void kvm_cpu__show_code(struct kvm_cpu *vcpu)
 
  	dprintf(debug_fd, "\n*pc:\n");
  	reg.id = ARM64_CORE_REG(regs.pc);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (show_code @ PC)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pc): %s", strerror(errno));
+		return;
+	}
 
 
  	kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
 
  	dprintf(debug_fd, "\n*lr:\n");
  	reg.id = ARM64_CORE_REG(regs.regs[30]);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (show_code @ LR)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (lr): %s", strerror(errno));
+		return;
+	}
 
  	kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
  }
arm64/kvm-cpu.c +16 -8
@@ -499,23 +503,31 @@ void kvm_cpu__show_registers(struct kvm_cpu *vcpu)
  	dprintf(debug_fd, "\n Registers:\n");
 
  	reg.id		= ARM64_CORE_REG(regs.pc);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (pc)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pc): %s", strerror(errno));
+		return;
+	}
  	dprintf(debug_fd, " PC:    0x%lx\n", data);
 
  	reg.id		= ARM64_CORE_REG(regs.pstate);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (pstate)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (pstate): %s", strerror(errno));
+		return;
+	}
  	dprintf(debug_fd, " PSTATE:    0x%lx\n", data);
 
  	reg.id		= ARM64_CORE_REG(sp_el1);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (sp_el1)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (sp_el1): %s", strerror(errno));
+		return;
+	}
  	dprintf(debug_fd, " SP_EL1:    0x%lx\n", data);
 
  	reg.id		= ARM64_CORE_REG(regs.regs[30]);
-	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-		die("KVM_GET_ONE_REG failed (lr)");
+	if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+		pr_err("register state unavailable (lr): %s", strerror(errno));
+		return;
+	}
  	dprintf(debug_fd, " LR:    0x%lx\n", data);
  }
 
author reviewer
#3 lore raw

Re: [PATCH kvmtool 1/5] arm64: Do not abort on register-dump failures

Hi Suzuki,

Quoted 8 lines · Suzuki K Poulose
Quoted 6 lines · Fuad Tabba

kvm_cpu__show_registers() and kvm_cpu__show_code() read the vCPU's core registers with KVM_GET_ONE_REG, and die() if the ioctl fails. Both are diagnostics. They run from the KVM_EXIT_DEBUG case in kvm_cpu__start(), from the "lkvm debug -d" dump path in handle_sigusr1(), and from the panic dump in kvm_cpu_thread().

Quoted 6 lines
Quoted 3 lines

Once a protected vCPU has run, its registers belong to the guest and KVM_GET_ONE_REG returns -EPERM. Dying on that turns a diagnostic into a VMM abort.

Marc once suggested that these could always succeed with junk values for protected/Realm VMs. Do you think that is an option for pKVM ?

It's possible, the host copy is still there, but I'd rather not. Once the vCPU has run that copy is a mix rather than junk: the boot state the VMM wrote, plus what the exit handlers copy out (the PSTATE mode, x0 for an MMIO write, x0 to x2 for a forwarded PSCI call). A dump would show a live PSTATE beside the PC from boot, and the VMM has no way to distinguish them. With the error it prints that the state is unavailable, which is what this patch does.

It's also what the tree does for protected state elsewhere: s390 returns -EINVAL from KVM_GET/SET_ONE_REG on a protected VM, and x86 does the same from the register ioctls for the SEV-ES VM type, SNP and TDX. Legacy SEV-ES is the one place that succeeds silently, kept for backwards compatibility, and the commit that added the errors (517987e3fb19) describes that as a problem. Sean's rationale for TDX:
KVM can't provide sane data, so it's userspace's job not to ask for it [1].

Quoted 4 lines

If we go with this approach :

Thanks, I'll add the tag. :)

Cheers,
/fuad

[1] https://lore.kernel.org/all/482264f17fa0652faad9bd5364d652d11cb2ecb8.1625186503.git.isaku.yamahata@intel.com/

Quoted 81 lines
Quoted 80 lines

Report that the register state is unavailable, with the errno, and return instead of dying.

---
  arm64/kvm-cpu.c | 36 ++++++++++++++++++++++++------------
  1 file changed, 24 insertions(+), 12 deletions(-)
arm64/kvm-cpu.c +8 -4
index 3aa7684..c58d61f 100644
@@ -476,15 +476,19 @@ void kvm_cpu__show_code(struct kvm_cpu *vcpu)
 
      dprintf(debug_fd, "\n*pc:\n");
      reg.id = ARM64_CORE_REG(regs.pc);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (show_code @ PC)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (pc): %s", strerror(errno));
+             return;
+     }
 
 
      kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
 
      dprintf(debug_fd, "\n*lr:\n");
      reg.id = ARM64_CORE_REG(regs.regs[30]);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (show_code @ LR)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (lr): %s", strerror(errno));
+             return;
+     }
 
      kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
  }
arm64/kvm-cpu.c +16 -8
@@ -499,23 +503,31 @@ void kvm_cpu__show_registers(struct kvm_cpu *vcpu)
      dprintf(debug_fd, "\n Registers:\n");
 
      reg.id          = ARM64_CORE_REG(regs.pc);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (pc)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (pc): %s", strerror(errno));
+             return;
+     }
      dprintf(debug_fd, " PC:    0x%lx\n", data);
 
      reg.id          = ARM64_CORE_REG(regs.pstate);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (pstate)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (pstate): %s", strerror(errno));
+             return;
+     }
      dprintf(debug_fd, " PSTATE:    0x%lx\n", data);
 
      reg.id          = ARM64_CORE_REG(sp_el1);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (sp_el1)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (sp_el1): %s", strerror(errno));
+             return;
+     }
      dprintf(debug_fd, " SP_EL1:    0x%lx\n", data);
 
      reg.id          = ARM64_CORE_REG(regs.regs[30]);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (lr)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (lr): %s", strerror(errno));
+             return;
+     }
      dprintf(debug_fd, " LR:    0x%lx\n", data);
  }
 
author reviewer
#4 lore raw

Re: [PATCH kvmtool 1/5] arm64: Do not abort on register-dump failures

Quoted 37 lines · Fuad Tabba

Hi Suzuki,

Quoted 8 lines · Suzuki K Poulose
Quoted 6 lines · Fuad Tabba

kvm_cpu__show_registers() and kvm_cpu__show_code() read the vCPU's core registers with KVM_GET_ONE_REG, and die() if the ioctl fails. Both are diagnostics. They run from the KVM_EXIT_DEBUG case in kvm_cpu__start(), from the "lkvm debug -d" dump path in handle_sigusr1(), and from the panic dump in kvm_cpu_thread().

Quoted 6 lines
Quoted 3 lines

Once a protected vCPU has run, its registers belong to the guest and KVM_GET_ONE_REG returns -EPERM. Dying on that turns a diagnostic into a VMM abort.

Marc once suggested that these could always succeed with junk values for protected/Realm VMs. Do you think that is an option for pKVM ?

It's possible, the host copy is still there, but I'd rather not. Once the vCPU has run that copy is a mix rather than junk: the boot state the VMM wrote, plus what the exit handlers copy out (the PSTATE mode, x0 for an MMIO write, x0 to x2 for a forwarded PSCI call). A dump would show a live PSTATE beside the PC from boot, and the VMM has no way to distinguish them. With the error it prints that the state is unavailable, which is what this patch does.

It's also what the tree does for protected state elsewhere: s390 returns -EINVAL from KVM_GET/SET_ONE_REG on a protected VM, and x86 does the same from the register ioctls for the SEV-ES VM type, SNP and TDX. Legacy SEV-ES is the one place that succeeds silently, kept for backwards compatibility, and the commit that added the errors (517987e3fb19) describes that as a problem. Sean's rationale for TDX:
KVM can't provide sane data, so it's userspace's job not to ask for it [1].

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

If we go with this approach :

Thanks, I'll add the tag. :)

Unless I misunderstood what "this" refers to exactly, please shout if that's the case, or continue the discussion :)

/fuad

Quoted 87 lines

Cheers,
/fuad

[1] https://lore.kernel.org/all/482264f17fa0652faad9bd5364d652d11cb2ecb8.1625186503.git.isaku.yamahata@intel.com/

Quoted 81 lines
Quoted 80 lines

Report that the register state is unavailable, with the errno, and return instead of dying.

---
  arm64/kvm-cpu.c | 36 ++++++++++++++++++++++++------------
  1 file changed, 24 insertions(+), 12 deletions(-)
arm64/kvm-cpu.c +8 -4
index 3aa7684..c58d61f 100644
@@ -476,15 +476,19 @@ void kvm_cpu__show_code(struct kvm_cpu *vcpu)
 
      dprintf(debug_fd, "\n*pc:\n");
      reg.id = ARM64_CORE_REG(regs.pc);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (show_code @ PC)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (pc): %s", strerror(errno));
+             return;
+     }
 
 
      kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
 
      dprintf(debug_fd, "\n*lr:\n");
      reg.id = ARM64_CORE_REG(regs.regs[30]);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (show_code @ LR)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (lr): %s", strerror(errno));
+             return;
+     }
 
      kvm__dump_mem(vcpu->kvm, data, 32, debug_fd);
  }
arm64/kvm-cpu.c +16 -8
@@ -499,23 +503,31 @@ void kvm_cpu__show_registers(struct kvm_cpu *vcpu)
      dprintf(debug_fd, "\n Registers:\n");
 
      reg.id          = ARM64_CORE_REG(regs.pc);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (pc)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (pc): %s", strerror(errno));
+             return;
+     }
      dprintf(debug_fd, " PC:    0x%lx\n", data);
 
      reg.id          = ARM64_CORE_REG(regs.pstate);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (pstate)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (pstate): %s", strerror(errno));
+             return;
+     }
      dprintf(debug_fd, " PSTATE:    0x%lx\n", data);
 
      reg.id          = ARM64_CORE_REG(sp_el1);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (sp_el1)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (sp_el1): %s", strerror(errno));
+             return;
+     }
      dprintf(debug_fd, " SP_EL1:    0x%lx\n", data);
 
      reg.id          = ARM64_CORE_REG(regs.regs[30]);
-     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0)
-             die("KVM_GET_ONE_REG failed (lr)");
+     if (ioctl(vcpu->vcpu_fd, KVM_GET_ONE_REG, &reg) < 0) {
+             pr_err("register state unavailable (lr): %s", strerror(errno));
+             return;
+     }
      dprintf(debug_fd, " LR:    0x%lx\n", data);
  }
 
#5 lore raw

Re: [PATCH kvmtool 1/5] arm64: Do not abort on register-dump failures

Quoted 28 lines · Fuad Tabba
Quoted 27 lines · Fuad Tabba

Hi Suzuki,

Quoted 8 lines · Suzuki K Poulose
Quoted 6 lines · Fuad Tabba

kvm_cpu__show_registers() and kvm_cpu__show_code() read the vCPU's core registers with KVM_GET_ONE_REG, and die() if the ioctl fails. Both are diagnostics. They run from the KVM_EXIT_DEBUG case in kvm_cpu__start(), from the "lkvm debug -d" dump path in handle_sigusr1(), and from the panic dump in kvm_cpu_thread().

Quoted 6 lines
Quoted 3 lines

Once a protected vCPU has run, its registers belong to the guest and KVM_GET_ONE_REG returns -EPERM. Dying on that turns a diagnostic into a VMM abort.

Marc once suggested that these could always succeed with junk values for protected/Realm VMs. Do you think that is an option for pKVM ?

It's possible, the host copy is still there, but I'd rather not. Once the vCPU has run that copy is a mix rather than junk: the boot state the VMM wrote, plus what the exit handlers copy out (the PSTATE mode, x0 for an MMIO write, x0 to x2 for a forwarded PSCI call). A dump would show a live PSTATE beside the PC from boot, and the VMM has no way to distinguish them. With the error it prints that the state is unavailable, which is what this patch does.

Quoted 9 lines
Quoted 9 lines

It's also what the tree does for protected state elsewhere: s390 returns -EINVAL from KVM_GET/SET_ONE_REG on a protected VM, and x86 does the same from the register ioctls for the SEV-ES VM type, SNP and TDX. Legacy SEV-ES is the one place that succeeds silently, kept for backwards compatibility, and the commit that added the errors (517987e3fb19) describes that as a problem. Sean's rationale for TDX:
KVM can't provide sane data, so it's userspace's job not to ask for it [1].

Thanks, I am aware of this. I will leave the decision to Marc. For the record, for CCA we use GET_ONE_REG/SET_ONE_REG to configure the SVE vector length, PMU Counter and the Debug HW BPRs and watch points.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

If we go with this approach :

Thanks, I'll add the tag. :)

Unless I misunderstood what "this" refers to exactly, please shout if that's the case, or continue the discussion :)

You got it right. I meant if we are happy to go with approach in the patch.

author reviewer
#6 lore raw

PATCH 2/5 kvm: Bound-check the exit-reason string lookup

In reply to Fuad Tabba

kvm_cpu_thread()'s panic path uses the kernel's exit_reason to index kvm_exit_reasons[] without checking it against the array's size. The table does not cover every exit reason KVM can return, so a reason past its last entry reads out of bounds. KVM_EXIT_SYSTEM_EVENT and KVM_EXIT_ARM_NISV are both past it.

Move the table behind kvm__exit_reason_str(), which checks the index first. The array now has a single caller inside kvm.c, so make it static const.

---
 builtin-run.c     |  2 +-
 include/kvm/kvm.h |  2 +-
 kvm.c             | 10 +++++++++-
 3 files changed, 11 insertions(+), 3 deletions(-)
builtin-run.c +1 -1
index 81f255f..f36d3e2 100644
@@ -298,7 +298,7 @@ static void *kvm_cpu_thread(void *arg)
 panic_kvm:
 	pr_err("KVM exit reason: %u (\"%s\")",
 		current_kvm_cpu->kvm_run->exit_reason,
-		kvm_exit_reasons[current_kvm_cpu->kvm_run->exit_reason]);
+		kvm__exit_reason_str(current_kvm_cpu->kvm_run->exit_reason));
 
 	if (current_kvm_cpu->kvm_run->exit_reason == KVM_EXIT_UNKNOWN) {
 		pr_err("KVM exit code: %llu",
include/kvm/kvm.h +1 -1
index a9376b6..8619070 100644
@@ -255,7 +255,7 @@ int kvm__for_each_mem_bank(struct kvm *kvm, enum kvm_mem_type type,
  */
 void kvm__dump_mem(struct kvm *kvm, unsigned long addr, unsigned long size, int debug_fd);
 
-extern const char *kvm_exit_reasons[];
+const char *kvm__exit_reason_str(__u32 exit_reason);
 
 static inline bool host_ptr_in_ram(struct kvm *kvm, void *p)
 {
kvm.c +9 -1
index 96583f9..b416f6f 100644
@@ -33,7 +33,7 @@
 
 #define DEFINE_KVM_EXIT_REASON(reason) [reason] = #reason
 
-const char *kvm_exit_reasons[] = {
+static const char * const kvm_exit_reasons[] = {
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_UNKNOWN),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_EXCEPTION),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_IO),
@@ -57,6 +57,14 @@ const char *kvm_exit_reasons[] = {
 #endif
 };
 
+const char *kvm__exit_reason_str(__u32 exit_reason)
+{
+	if (exit_reason >= ARRAY_SIZE(kvm_exit_reasons))
+		return "UNKNOWN";
+
+	return kvm_exit_reasons[exit_reason];
+}
+
 static int pause_event;
 static DEFINE_MUTEX(pause_lock);
 static struct kvm_cpu *pause_req_cpu;
Signature · 1 line
2.39.5
reviewer Reviewed-by on v1 2/5
#7 lore raw

Re: [PATCH kvmtool 2/5] kvm: Bound-check the exit-reason string lookup

Quoted 11 lines · Fuad Tabba

kvm_cpu_thread()'s panic path uses the kernel's exit_reason to index kvm_exit_reasons[] without checking it against the array's size. The table does not cover every exit reason KVM can return, so a reason past its last entry reads out of bounds. KVM_EXIT_SYSTEM_EVENT and KVM_EXIT_ARM_NISV are both past it.

Move the table behind kvm__exit_reason_str(), which checks the index first. The array now has a single caller inside kvm.c, so make it static const.

Quoted 60 lines
---
  builtin-run.c     |  2 +-
  include/kvm/kvm.h |  2 +-
  kvm.c             | 10 +++++++++-
  3 files changed, 11 insertions(+), 3 deletions(-)
builtin-run.c +1 -1
index 81f255f..f36d3e2 100644
@@ -298,7 +298,7 @@ static void *kvm_cpu_thread(void *arg)
  panic_kvm:
  	pr_err("KVM exit reason: %u (\"%s\")",
  		current_kvm_cpu->kvm_run->exit_reason,
-		kvm_exit_reasons[current_kvm_cpu->kvm_run->exit_reason]);
+		kvm__exit_reason_str(current_kvm_cpu->kvm_run->exit_reason));
 
  	if (current_kvm_cpu->kvm_run->exit_reason == KVM_EXIT_UNKNOWN) {
  		pr_err("KVM exit code: %llu",
include/kvm/kvm.h +1 -1
index a9376b6..8619070 100644
@@ -255,7 +255,7 @@ int kvm__for_each_mem_bank(struct kvm *kvm, enum kvm_mem_type type,
   */
  void kvm__dump_mem(struct kvm *kvm, unsigned long addr, unsigned long size, int debug_fd);
 
-extern const char *kvm_exit_reasons[];
+const char *kvm__exit_reason_str(__u32 exit_reason);
 
  static inline bool host_ptr_in_ram(struct kvm *kvm, void *p)
  {
kvm.c +9 -1
index 96583f9..b416f6f 100644
@@ -33,7 +33,7 @@
 
  #define DEFINE_KVM_EXIT_REASON(reason) [reason] = #reason
 
-const char *kvm_exit_reasons[] = {
+static const char * const kvm_exit_reasons[] = {
  	DEFINE_KVM_EXIT_REASON(KVM_EXIT_UNKNOWN),
  	DEFINE_KVM_EXIT_REASON(KVM_EXIT_EXCEPTION),
  	DEFINE_KVM_EXIT_REASON(KVM_EXIT_IO),
@@ -57,6 +57,14 @@ const char *kvm_exit_reasons[] = {
  #endif
  };
 
+const char *kvm__exit_reason_str(__u32 exit_reason)
+{
+	if (exit_reason >= ARRAY_SIZE(kvm_exit_reasons))
+		return "UNKNOWN";
+
+	return kvm_exit_reasons[exit_reason];
+}
+
  static int pause_event;
  static DEFINE_MUTEX(pause_lock);
  static struct kvm_cpu *pause_req_cpu;
author reviewer
#8 lore raw

PATCH 3/5 kvm: Name every exit reason the UAPI header defines

In reply to Fuad Tabba

kvm_exit_reasons[] stops at KVM_EXIT_INTERNAL_ERROR, so the panic path prints "UNKNOWN" for every reason KVM has gained since. On arm64 that covers KVM_EXIT_SYSTEM_EVENT and KVM_EXIT_ARM_NISV, and the newer KVM_EXIT_MEMORY_FAULT, KVM_EXIT_ARM_SEA and KVM_EXIT_ARM_LDST64B.

Name the rest of the reasons the synced uapi header defines, up to KVM_EXIT_SNP_REQ_CERTS. KVM_EXIT_PAPR_HCALL also loses its CONFIG_PPC64 guard, which the build never defines, so that entry was missing from every configuration. The neighbouring PPC reasons are unconditional already.

---
 kvm.c | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)
kvm.c +25 -2
index b416f6f..828da64 100644
@@ -52,9 +52,32 @@ static const char * const kvm_exit_reasons[] = {
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_DCR),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_NMI),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_INTERNAL_ERROR),
-#ifdef CONFIG_PPC64
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_OSI),
 	DEFINE_KVM_EXIT_REASON(KVM_EXIT_PAPR_HCALL),
-#endif
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_S390_UCONTROL),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_WATCHDOG),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_S390_TSCH),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_EPR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_SYSTEM_EVENT),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_S390_STSI),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_IOAPIC_EOI),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_HYPERV),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_ARM_NISV),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_X86_RDMSR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_X86_WRMSR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_DIRTY_RING_FULL),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_AP_RESET_HOLD),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_X86_BUS_LOCK),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_XEN),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_RISCV_SBI),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_RISCV_CSR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_NOTIFY),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_LOONGARCH_IOCSR),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_MEMORY_FAULT),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_TDX),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_ARM_SEA),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_ARM_LDST64B),
+	DEFINE_KVM_EXIT_REASON(KVM_EXIT_SNP_REQ_CERTS),
 };
 
 const char *kvm__exit_reason_str(__u32 exit_reason)
Signature · 1 line
2.39.5
reviewer Reviewed-by on v1 3/5
#9 lore raw

Re: [PATCH kvmtool 3/5] kvm: Name every exit reason the UAPI header defines

Quoted 5 lines · Fuad Tabba

kvm_exit_reasons[] stops at KVM_EXIT_INTERNAL_ERROR, so the panic path prints "UNKNOWN" for every reason KVM has gained since. On arm64 that covers KVM_EXIT_SYSTEM_EVENT and KVM_EXIT_ARM_NISV, and the newer KVM_EXIT_MEMORY_FAULT, KVM_EXIT_ARM_SEA and KVM_EXIT_ARM_LDST64B.

Quoted 7 lines

Name the rest of the reasons the synced uapi header defines, up to KVM_EXIT_SNP_REQ_CERTS. KVM_EXIT_PAPR_HCALL also loses its CONFIG_PPC64 guard, which the build never defines, so that entry was missing from every configuration. The neighbouring PPC reasons are unconditional already.

author reviewer
#10 lore raw

PATCH 4/5 arm64: Query steal-time support on the VM fd

In reply to Fuad Tabba

kvm_cpu__setup_pvtime() probes KVM_CAP_STEAL_TIME with kvm__supports_extension(), which issues KVM_CHECK_EXTENSION on the global /dev/kvm fd. That reports the host's raw capabilities, unaware of any per-VM restrictions.

pKVM does not offer steal-time to a protected VM, and reflects that on the VM fd alone. kvmtool goes on to KVM_HAS_DEVICE_ATTR, the kernel refuses the PVTIME attribute with -EPERM, and kvm_cpu__arch_init() dies:

Fatal: Unable to initialise vcpu

Query it on the VM fd via kvm__supports_vm_extension(), as commit 84464ba0246b ("arm64: Query per-VM capabilities when selecting vCPU features") did for the vCPU feature probes. kvmtool then takes the no_pvtime path it already has when steal-time is unsupported.

---
 arm64/pvtime.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arm64/pvtime.c +2 -2
index 2933ac7..839aa8a 100644
@@ -58,8 +58,8 @@ int kvm_cpu__setup_pvtime(struct kvm_cpu *vcpu)
 	if (kvm_cfg->no_pvtime)
 		return 0;
 
-	has_stolen_time = kvm__supports_extension(vcpu->kvm,
-						  KVM_CAP_STEAL_TIME);
+	has_stolen_time = kvm__supports_vm_extension(vcpu->kvm,
+						     KVM_CAP_STEAL_TIME);
 	if (!has_stolen_time) {
 		kvm_cfg->no_pvtime = true;
 		return 0;
Signature · 1 line
2.39.5
reviewer Reviewed-by on v1 4/5
#11 lore raw

Re: [PATCH kvmtool 4/5] arm64: Query steal-time support on the VM fd

Quoted 9 lines · Fuad Tabba

kvm_cpu__setup_pvtime() probes KVM_CAP_STEAL_TIME with kvm__supports_extension(), which issues KVM_CHECK_EXTENSION on the global /dev/kvm fd. That reports the host's raw capabilities, unaware of any per-VM restrictions.

pKVM does not offer steal-time to a protected VM, and reflects that on the VM fd alone. kvmtool goes on to KVM_HAS_DEVICE_ATTR, the kernel refuses the PVTIME attribute with -EPERM, and kvm_cpu__arch_init() dies:

Quoted 8 lines

Fatal: Unable to initialise vcpu

Query it on the VM fd via kvm__supports_vm_extension(), as commit 84464ba0246b ("arm64: Query per-VM capabilities when selecting vCPU features") did for the vCPU feature probes. kvmtool then takes the no_pvtime path it already has when steal-time is unsupported.

Quoted 19 lines
---
  arm64/pvtime.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
arm64/pvtime.c +2 -2
index 2933ac7..839aa8a 100644
@@ -58,8 +58,8 @@ int kvm_cpu__setup_pvtime(struct kvm_cpu *vcpu)
  	if (kvm_cfg->no_pvtime)
  		return 0;
 
-	has_stolen_time = kvm__supports_extension(vcpu->kvm,
-						  KVM_CAP_STEAL_TIME);
+	has_stolen_time = kvm__supports_vm_extension(vcpu->kvm,
+						     KVM_CAP_STEAL_TIME);
  	if (!has_stolen_time) {
  		kvm_cfg->no_pvtime = true;
  		return 0;
author reviewer
#12 lore raw

PATCH 5/5 arm64: Query counter-offset support on the VM fd

In reply to Fuad Tabba

kvm__arch_set_counter_offset() probes KVM_CAP_COUNTER_OFFSET with kvm__supports_extension(), which issues KVM_CHECK_EXTENSION on the global /dev/kvm fd. That reports the host's raw capabilities, unaware of any per-VM restrictions.

pKVM does not offer the counter offset to a protected VM, and reflects that on the VM fd alone. kvmtool's own check passes, and the KVM_ARM_SET_COUNTER_OFFSET that follows is refused by commit b12b3b04f6ba ("KVM: arm64: Check whether a VM IOCTL is allowed in pKVM"):

KVM_ARM_SET_COUNTER_OFFSET: Invalid argument

Query it on the VM fd via kvm__supports_vm_extension(), so --counter-offset on a protected VM fails with kvmtool's own message:

Fatal: No support for global counter offset

---
 arm64/kvm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arm64/kvm.c +1 -1
index 36b3284..0bb3535 100644
@@ -155,7 +155,7 @@ static void kvm__arch_set_counter_offset(struct kvm *kvm)
 	if (!kvm->cfg.arch.counter_offset)
 		return;
 
-	if (!kvm__supports_extension(kvm, KVM_CAP_COUNTER_OFFSET))
+	if (!kvm__supports_vm_extension(kvm, KVM_CAP_COUNTER_OFFSET))
 		die("No support for global counter offset");
 
 	if (ioctl(kvm->vm_fd, KVM_ARM_SET_COUNTER_OFFSET, &offset))
Signature · 1 line
2.39.5
reviewer Reviewed-by on v1 5/5
#13 lore raw

Re: [PATCH kvmtool 5/5] arm64: Query counter-offset support on the VM fd

Quoted 11 lines · Fuad Tabba

kvm__arch_set_counter_offset() probes KVM_CAP_COUNTER_OFFSET with kvm__supports_extension(), which issues KVM_CHECK_EXTENSION on the global /dev/kvm fd. That reports the host's raw capabilities, unaware of any per-VM restrictions.

pKVM does not offer the counter offset to a protected VM, and reflects that on the VM fd alone. kvmtool's own check passes, and the KVM_ARM_SET_COUNTER_OFFSET that follows is refused by commit b12b3b04f6ba ("KVM: arm64: Check whether a VM IOCTL is allowed in pKVM"):

Quoted 8 lines

KVM_ARM_SET_COUNTER_OFFSET: Invalid argument

Query it on the VM fd via kvm__supports_vm_extension(), so --counter-offset on a protected VM fails with kvmtool's own message:

Fatal: No support for global counter offset

Quoted 17 lines
---
  arm64/kvm.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arm64/kvm.c +1 -1
index 36b3284..0bb3535 100644
@@ -155,7 +155,7 @@ static void kvm__arch_set_counter_offset(struct kvm *kvm)
  	if (!kvm->cfg.arch.counter_offset)
  		return;
 
-	if (!kvm__supports_extension(kvm, KVM_CAP_COUNTER_OFFSET))
+	if (!kvm__supports_vm_extension(kvm, KVM_CAP_COUNTER_OFFSET))
  		die("No support for global counter offset");
 
  	if (ioctl(kvm->vm_fd, KVM_ARM_SET_COUNTER_OFFSET, &offset))
author reviewer
#14 lore raw
In reply to Fuad Tabba

Hi Will,

Gentle ping on this series. Marc has merged the pKVM core series into kvmarm/next, including the KVM_GET_ONE_REG -EPERM that patch 1 handles, and all five patches have Suzuki's Reviewed-by.

Cheers,
/fuad

#15 lore raw
Quoted 3 lines · Fuad Tabba

Gentle ping on this series. Marc has merged the pKVM core series into kvmarm/next, including the KVM_GET_ONE_REG -EPERM that patch 1 handles, and all five patches have Suzuki's Reviewed-by.

I usually wait for the patches to land in Linus' tree before I update kvmtool (although I may be fishing when that happens).

Will

author reviewer
#16 lore raw
Quoted 10 lines · Will Deacon
Quoted 3 lines · Fuad Tabba

Gentle ping on this series. Marc has merged the pKVM core series into kvmarm/next, including the KVM_GET_ONE_REG -EPERM that patch 1 handles, and all five patches have Suzuki's Reviewed-by.

I usually wait for the patches to land in Linus' tree before I update kvmtool (although I may be fishing when that happens).

(although I may be fishing when that happens).

Took me a while to understand that part :D

No rush, and enjoy the fishing.

/fuad

Quoted 3 lines

Will