Skip to contentSkip to conversation

aarch64: Add stolen time support

PATCH Superseded Sebastian Ene 2 participants 9 messages Mar 7, 2022 – Mar 8, 2022

Subsystems (3)

v8 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v8 cover aarch64: Add stolen time support

These patches add support for stolen time functionality.

Patch #1 moves the vCPU structure initialisation before the target->init() call to allow early access to the kvm structure from the vCPU during target->init().

Patch #2 modifies the memory layout in arm-common/kvm-arch.h and adds a new MMIO device PVTIME after the RTC region. A new flag is added in kvm-config.h that will be used to control [enable/disable] the pvtime functionality. Stolen time is enabled by default when the host supports KVM_CAP_STEAL_TIME.

Patch #3 adds a new command line argument to disable the stolen time functionality(by default is enabled).

Changelog since v7:
- patch #2:
- move teardown to kvm_cpu__delete()
- don’t call kvm__supports_extension for the remainig cpu’s if it failed once and force 'no_pvtime' to 1
- replace variable name 'is_supported' with 'is_failed_cfg'

Sebastian Ene (3):
aarch64: Populate the vCPU struct before target->init()
aarch64: Add stolen time support
Add --no-pvtime command line argument

 Makefile                               |   1 +
 arm/aarch64/arm-cpu.c                  |   2 +-
 arm/aarch64/include/kvm/kvm-cpu-arch.h |   2 +
 arm/aarch64/pvtime.c                   | 108 +++++++++++++++++++++++++
 arm/include/arm-common/kvm-arch.h      |   6 +-
 arm/kvm-cpu.c                          |  15 ++--
 builtin-run.c                          |   2 +
 include/kvm/kvm-config.h               |   1 +
 8 files changed, 128 insertions(+), 9 deletions(-)
 create mode 100644 arm/aarch64/pvtime.c
Signature · 6 lines
2.35.1.616.g0bdcbb4464-goog

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#1 lore raw

PATCH v8 1/3 aarch64: Populate the vCPU struct before target->init()

Move the vCPU structure initialisation before the target->init() call to
keep a reference to the kvm structure during init().
This is required by the pvtime peripheral to reserve a memory region while the vCPU is beeing initialised.

---
 arm/kvm-cpu.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
arm/kvm-cpu.c +7 -7
index 6a2408c..84ac1e9 100644
@@ -116,6 +116,13 @@ struct kvm_cpu *kvm_cpu__arch_init(struct kvm *kvm, unsigned long cpu_id)
 			die("Unable to find matching target");
 	}
 
+	/* Populate the vcpu structure. */
+	vcpu->kvm		= kvm;
+	vcpu->cpu_id		= cpu_id;
+	vcpu->cpu_type		= vcpu_init.target;
+	vcpu->cpu_compatible	= target->compatible;
+	vcpu->is_running	= true;
+
 	if (err || target->init(vcpu))
 		die("Unable to initialise vcpu");
 
@@ -125,13 +132,6 @@ struct kvm_cpu *kvm_cpu__arch_init(struct kvm *kvm, unsigned long cpu_id)
 		vcpu->ring = (void *)vcpu->kvm_run +
 			     (coalesced_offset * PAGE_SIZE);
 
-	/* Populate the vcpu structure. */
-	vcpu->kvm		= kvm;
-	vcpu->cpu_id		= cpu_id;
-	vcpu->cpu_type		= vcpu_init.target;
-	vcpu->cpu_compatible	= target->compatible;
-	vcpu->is_running	= true;
-
 	if (kvm_cpu__configure_features(vcpu))
 		die("Unable to configure requested vcpu features");
 
Signature · 6 lines
2.35.1.616.g0bdcbb4464-goog

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#2 lore raw

Re: [PATCH kvmtool v8 1/3] aarch64: Populate the vCPU struct before target->init()

Hi,

Quoted 6 lines · Sebastian Ene

Move the vCPU structure initialisation before the target->init() call to
keep a reference to the kvm structure during init().
This is required by the pvtime peripheral to reserve a memory region while the vCPU is beeing initialised.

I think you're missing Marc's Reviewed-by tag.

Thanks,
Alex

Quoted 43 lines
---
 arm/kvm-cpu.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
arm/kvm-cpu.c +7 -7
index 6a2408c..84ac1e9 100644
@@ -116,6 +116,13 @@ struct kvm_cpu *kvm_cpu__arch_init(struct kvm *kvm, unsigned long cpu_id)
 			die("Unable to find matching target");
 	}
 
+	/* Populate the vcpu structure. */
+	vcpu->kvm		= kvm;
+	vcpu->cpu_id		= cpu_id;
+	vcpu->cpu_type		= vcpu_init.target;
+	vcpu->cpu_compatible	= target->compatible;
+	vcpu->is_running	= true;
+
 	if (err || target->init(vcpu))
 		die("Unable to initialise vcpu");
 
@@ -125,13 +132,6 @@ struct kvm_cpu *kvm_cpu__arch_init(struct kvm *kvm, unsigned long cpu_id)
 		vcpu->ring = (void *)vcpu->kvm_run +
 			     (coalesced_offset * PAGE_SIZE);
 
-	/* Populate the vcpu structure. */
-	vcpu->kvm		= kvm;
-	vcpu->cpu_id		= cpu_id;
-	vcpu->cpu_type		= vcpu_init.target;
-	vcpu->cpu_compatible	= target->compatible;
-	vcpu->is_running	= true;
-
 	if (kvm_cpu__configure_features(vcpu))
 		die("Unable to configure requested vcpu features");
 
Signature · 6 lines
2.35.1.616.g0bdcbb4464-goog

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#3 lore raw

Re: [PATCH kvmtool v8 1/3] aarch64: Populate the vCPU struct before target->init()

Quoted 2 lines · Alexandru Elisei

Hi,

Hi,

Quoted 10 lines · Sebastian Ene
Quoted 6 lines

Move the vCPU structure initialisation before the target->init() call to
keep a reference to the kvm structure during init().
This is required by the pvtime peripheral to reserve a memory region while the vCPU is beeing initialised.

I think you're missing Marc's Reviewed-by tag.

I will add the Reviewed-by tag to the next iteration.

Quoted 3 lines

Thanks,
Alex

Thanks,
Sebastian

Quoted 43 lines
Quoted 43 lines
---
 arm/kvm-cpu.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
arm/kvm-cpu.c +7 -7
index 6a2408c..84ac1e9 100644
@@ -116,6 +116,13 @@ struct kvm_cpu *kvm_cpu__arch_init(struct kvm *kvm, unsigned long cpu_id)
 			die("Unable to find matching target");
 	}
 
+	/* Populate the vcpu structure. */
+	vcpu->kvm		= kvm;
+	vcpu->cpu_id		= cpu_id;
+	vcpu->cpu_type		= vcpu_init.target;
+	vcpu->cpu_compatible	= target->compatible;
+	vcpu->is_running	= true;
+
 	if (err || target->init(vcpu))
 		die("Unable to initialise vcpu");
 
@@ -125,13 +132,6 @@ struct kvm_cpu *kvm_cpu__arch_init(struct kvm *kvm, unsigned long cpu_id)
 		vcpu->ring = (void *)vcpu->kvm_run +
 			     (coalesced_offset * PAGE_SIZE);
 
-	/* Populate the vcpu structure. */
-	vcpu->kvm		= kvm;
-	vcpu->cpu_id		= cpu_id;
-	vcpu->cpu_type		= vcpu_init.target;
-	vcpu->cpu_compatible	= target->compatible;
-	vcpu->is_running	= true;
-
 	if (kvm_cpu__configure_features(vcpu))
 		die("Unable to configure requested vcpu features");
 
Signature · 6 lines
2.35.1.616.g0bdcbb4464-goog

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#4 lore raw

PATCH v8 2/3 aarch64: Add stolen time support

In reply to Sebastian Ene

This patch adds support for stolen time by sharing a memory region with the guest which will be used by the hypervisor to store the stolen time information. Reserve a 64kb MMIO memory region after the RTC peripheral to be used by pvtime. The exact format of the structure stored by the hypervisor is described in the ARM DEN0057A document.

---
 Makefile                               |   1 +
 arm/aarch64/arm-cpu.c                  |   2 +-
 arm/aarch64/include/kvm/kvm-cpu-arch.h |   2 +
 arm/aarch64/pvtime.c                   | 108 +++++++++++++++++++++++++
 arm/include/arm-common/kvm-arch.h      |   6 +-
 arm/kvm-cpu.c                          |   1 +
 include/kvm/kvm-config.h               |   1 +
 7 files changed, 119 insertions(+), 2 deletions(-)
 create mode 100644 arm/aarch64/pvtime.c
Makefile +1 -0
index f251147..e9121dc 100644
@@ -182,6 +182,7 @@ ifeq ($(ARCH), arm64)
 	OBJS		+= arm/aarch64/arm-cpu.o
 	OBJS		+= arm/aarch64/kvm-cpu.o
 	OBJS		+= arm/aarch64/kvm.o
+	OBJS		+= arm/aarch64/pvtime.o
 	ARCH_INCLUDE	:= $(HDRS_ARM_COMMON)
 	ARCH_INCLUDE	+= -Iarm/aarch64/include
 
arm/aarch64/arm-cpu.c +1 -1
index d7572b7..7e4a3c1 100644
@@ -22,7 +22,7 @@ static void generate_fdt_nodes(void *fdt, struct kvm *kvm)
 static int arm_cpu__vcpu_init(struct kvm_cpu *vcpu)
 {
 	vcpu->generate_fdt_nodes = generate_fdt_nodes;
-	return 0;
+	return kvm_cpu__setup_pvtime(vcpu);
 }
 
 static struct kvm_arm_target target_generic_v8 = {
arm/aarch64/include/kvm/kvm-cpu-arch.h +2 -0
index 8dfb82e..35996dc 100644
@@ -19,5 +19,7 @@
 
 void kvm_cpu__select_features(struct kvm *kvm, struct kvm_vcpu_init *init);
 int kvm_cpu__configure_features(struct kvm_cpu *vcpu);
+int kvm_cpu__setup_pvtime(struct kvm_cpu *vcpu);
+int kvm_cpu__teardown_pvtime(struct kvm *kvm);
 
 #endif /* KVM__KVM_CPU_ARCH_H */
arm/aarch64/pvtime.c new file +108 -0

108 lines not on this page · raw message

arm/include/arm-common/kvm-arch.h +5 -1
index c645ac0..43b1f77 100644
@@ -15,7 +15,8 @@
  * |  PCI  |////| plat  |       |        |     |         |
  * |  I/O  |////| MMIO: | Flash | virtio | GIC |   PCI   |  DRAM
  * | space |////| UART, |       |  MMIO  |     |  (AXI)  |
- * |       |////| RTC   |       |        |     |         |
+ * |       |////| RTC,  |       |        |     |         |
+ * |       |////| PVTIME|       |        |     |         |
  * +-------+----+-------+-------+--------+-----+---------+---......
  */
 
@@ -34,6 +35,9 @@
 #define ARM_RTC_MMIO_BASE	(ARM_UART_MMIO_BASE + ARM_UART_MMIO_SIZE)
 #define ARM_RTC_MMIO_SIZE	0x10000
 
+#define ARM_PVTIME_BASE		(ARM_RTC_MMIO_BASE + ARM_RTC_MMIO_SIZE)
+#define ARM_PVTIME_SIZE		SZ_64K
+
 #define KVM_FLASH_MMIO_BASE	(ARM_MMIO_AREA + 0x1000000)
 #define KVM_FLASH_MAX_SIZE	0x1000000
 
arm/kvm-cpu.c +1 -0
index 84ac1e9..00660d6 100644
@@ -144,6 +144,7 @@ void kvm_cpu__arch_nmi(struct kvm_cpu *cpu)
 
 void kvm_cpu__delete(struct kvm_cpu *vcpu)
 {
+	kvm_cpu__teardown_pvtime(vcpu->kvm);
 	free(vcpu);
 }
 
include/kvm/kvm-config.h +1 -0
index 6a5720c..48adf27 100644
@@ -62,6 +62,7 @@ struct kvm_config {
 	bool no_dhcp;
 	bool ioport_debug;
 	bool mmio_debug;
+	bool no_pvtime;
 };
 
 #endif
Signature · 6 lines
2.35.1.616.g0bdcbb4464-goog

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#5 lore raw

Hi,

Quoted 152 lines · Sebastian Ene

This patch adds support for stolen time by sharing a memory region with the guest which will be used by the hypervisor to store the stolen time information. Reserve a 64kb MMIO memory region after the RTC peripheral to be used by pvtime. The exact format of the structure stored by the hypervisor is described in the ARM DEN0057A document.

---
 Makefile                               |   1 +
 arm/aarch64/arm-cpu.c                  |   2 +-
 arm/aarch64/include/kvm/kvm-cpu-arch.h |   2 +
 arm/aarch64/pvtime.c                   | 108 +++++++++++++++++++++++++
 arm/include/arm-common/kvm-arch.h      |   6 +-
 arm/kvm-cpu.c                          |   1 +
 include/kvm/kvm-config.h               |   1 +
 7 files changed, 119 insertions(+), 2 deletions(-)
 create mode 100644 arm/aarch64/pvtime.c
Makefile +1 -0
index f251147..e9121dc 100644
@@ -182,6 +182,7 @@ ifeq ($(ARCH), arm64)
 	OBJS		+= arm/aarch64/arm-cpu.o
 	OBJS		+= arm/aarch64/kvm-cpu.o
 	OBJS		+= arm/aarch64/kvm.o
+	OBJS		+= arm/aarch64/pvtime.o
 	ARCH_INCLUDE	:= $(HDRS_ARM_COMMON)
 	ARCH_INCLUDE	+= -Iarm/aarch64/include
 
arm/aarch64/arm-cpu.c +1 -1
index d7572b7..7e4a3c1 100644
@@ -22,7 +22,7 @@ static void generate_fdt_nodes(void *fdt, struct kvm *kvm)
 static int arm_cpu__vcpu_init(struct kvm_cpu *vcpu)
 {
 	vcpu->generate_fdt_nodes = generate_fdt_nodes;
-	return 0;
+	return kvm_cpu__setup_pvtime(vcpu);
 }
 
 static struct kvm_arm_target target_generic_v8 = {
arm/aarch64/include/kvm/kvm-cpu-arch.h +2 -0
index 8dfb82e..35996dc 100644
@@ -19,5 +19,7 @@
 
 void kvm_cpu__select_features(struct kvm *kvm, struct kvm_vcpu_init *init);
 int kvm_cpu__configure_features(struct kvm_cpu *vcpu);
+int kvm_cpu__setup_pvtime(struct kvm_cpu *vcpu);
+int kvm_cpu__teardown_pvtime(struct kvm *kvm);
 
 #endif /* KVM__KVM_CPU_ARCH_H */
arm/aarch64/pvtime.c new file +91 -0
new file mode 100644
index 0000000..4db2e9f
@@ -0,0 +1,108 @@
+#include "kvm/kvm.h"
+#include "kvm/kvm-cpu.h"
+#include "kvm/util.h"
+
+#include <linux/byteorder.h>
+#include <linux/types.h>
+
+#define ARM_PVTIME_STRUCT_SIZE		(64)
+
+struct pvtime_data_priv {
+	bool	is_failed_cfg;
+	void	*usr_mem;
+};
+
+static struct pvtime_data_priv pvtime_data = {
+	.is_failed_cfg	= true,
+	.usr_mem	= NULL
+};
+
+static int pvtime__alloc_region(struct kvm *kvm)
+{
+	char *mem;
+	int ret = 0;
+
+	mem = mmap(NULL, ARM_PVTIME_BASE, PROT_RW,
+		   MAP_ANON_NORESERVE, -1, 0);
+	if (mem == MAP_FAILED)
+		return -errno;
+
+	ret = kvm__register_ram(kvm, ARM_PVTIME_BASE,
+				ARM_PVTIME_BASE, mem);
+	if (ret) {
+		munmap(mem, ARM_PVTIME_BASE);
+		return ret;
+	}
+
+	pvtime_data.usr_mem = mem;
+	return ret;
+}
+
+static int pvtime__teardown_region(struct kvm *kvm)
+{
+	if (pvtime_data.usr_mem == NULL)
+		return 0;
+
+	kvm__destroy_mem(kvm, ARM_PVTIME_BASE,
+			 ARM_PVTIME_BASE, pvtime_data.usr_mem);
+	munmap(pvtime_data.usr_mem, ARM_PVTIME_BASE);
+	pvtime_data.usr_mem = NULL;
+	return 0;
+}
+
+int kvm_cpu__setup_pvtime(struct kvm_cpu *vcpu)
+{
+	int ret;
+	bool has_stolen_time;
+	u64 pvtime_guest_addr = ARM_PVTIME_BASE + vcpu->cpu_id *
+		ARM_PVTIME_STRUCT_SIZE;
+	struct kvm_config *kvm_cfg = NULL;
+	struct kvm_device_attr pvtime_attr = (struct kvm_device_attr) {
+		.group	= KVM_ARM_VCPU_PVTIME_CTRL,
+		.addr	= KVM_ARM_VCPU_PVTIME_IPA
+	};
+
+	kvm_cfg = &vcpu->kvm->cfg;
+	if (kvm_cfg->no_pvtime)
+		return 0;
+
+	if (!pvtime_data.is_failed_cfg)
+		return -ENOTSUP;
+
+	has_stolen_time = kvm__supports_extension(vcpu->kvm,
+						  KVM_CAP_STEAL_TIME);
+	if (!has_stolen_time) {
+		kvm_cfg->no_pvtime = true;
+		return 0;
+	}
+
+	ret = ioctl(vcpu->vcpu_fd, KVM_HAS_DEVICE_ATTR, &pvtime_attr);
+	if (ret) {
+		perror("KVM_HAS_DEVICE_ATTR failed\n");
+		goto out_err;
+	}
+
+	if (!pvtime_data.usr_mem) {
+		ret = pvtime__alloc_region(vcpu->kvm);
+		if (ret) {
+			perror("Failed allocating pvtime region\n");
+			goto out_err;
+		}
+	}

arm/aarch64/pvtime.c line 101

Quoted 10 lines
arm/aarch64/pvtime.c new file +10 -0
+
+	pvtime_attr.addr = (u64)&pvtime_guest_addr;
+	ret = ioctl(vcpu->vcpu_fd, KVM_SET_DEVICE_ATTR, &pvtime_attr);
+	if (!ret)
+		return 0;
+
+	perror("KVM_SET_DEVICE_ATTR failed\n");
+	pvtime__teardown_region(vcpu->kvm);
+out_err:
+	pvtime_data.is_failed_cfg = false;

kvm_cpu__init() calls kvm_cpu__arch_init()->kvm_cpu__setup_pvtime() for each VCPU from the main thread (so sequentually), not from the VCPU threads. If this function returns an error, kvm_cpu__arch_init() calls die(), which means that kvmtool will terminate without calling kvm_cpu__setup_pvtime() for the other VCPUs.

What I'm trying to say is that the field is_failed_cfg is not useful, because if one VCPU fails initialization, then kvmtool will not attempt to initialize the rest of the VCPUs.

If you drop is_failed_cfg you can also drop the pvtime_data_priv struct and use a static user_mem variable (up to you).

Quoted 36 lines
arm/aarch64/pvtime.c new file +7 -0
+	return ret;
+}
+
+int kvm_cpu__teardown_pvtime(struct kvm *kvm)
+{
+	return pvtime__teardown_region(kvm);
+}
arm/include/arm-common/kvm-arch.h +5 -1
index c645ac0..43b1f77 100644
@@ -15,7 +15,8 @@
  * |  PCI  |////| plat  |       |        |     |         |
  * |  I/O  |////| MMIO: | Flash | virtio | GIC |   PCI   |  DRAM
  * | space |////| UART, |       |  MMIO  |     |  (AXI)  |
- * |       |////| RTC   |       |        |     |         |
+ * |       |////| RTC,  |       |        |     |         |
+ * |       |////| PVTIME|       |        |     |         |
  * +-------+----+-------+-------+--------+-----+---------+---......
  */
 
@@ -34,6 +35,9 @@
 #define ARM_RTC_MMIO_BASE	(ARM_UART_MMIO_BASE + ARM_UART_MMIO_SIZE)
 #define ARM_RTC_MMIO_SIZE	0x10000
 
+#define ARM_PVTIME_BASE		(ARM_RTC_MMIO_BASE + ARM_RTC_MMIO_SIZE)
+#define ARM_PVTIME_SIZE		SZ_64K
+
 #define KVM_FLASH_MMIO_BASE	(ARM_MMIO_AREA + 0x1000000)
 #define KVM_FLASH_MAX_SIZE	0x1000000
 
arm/kvm-cpu.c +0 -0
index 84ac1e9..00660d6 100644

arm/kvm-cpu.c void kvm_cpu__arch_nmi(struct kvm_cpu *cpu) · line 147

Quoted 4 lines
arm/kvm-cpu.c +1 -0
@@ -144,6 +144,7 @@ void kvm_cpu__arch_nmi(struct kvm_cpu *cpu)
 
 void kvm_cpu__delete(struct kvm_cpu *vcpu)
 {
+	kvm_cpu__teardown_pvtime(vcpu->kvm);

This causes compilation for aarch32 to fail:

arm/kvm-cpu.c: In function ‘kvm_cpu__delete’:
arm/kvm-cpu.c:147:2: error: implicit declaration of function ‘kvm_cpu__teardown_pvtime’ [-Werror=implicit-function-declaration]

  147 |  kvm_cpu__teardown_pvtime(vcpu->kvm);
      |  ^~~~~~~~~~~~~~~~~~~~~~~~

arm/kvm-cpu.c:147:2: error: nested extern declaration of ‘kvm_cpu__teardown_pvtime’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make: *** [Makefile:482: arm/kvm-cpu.o] Error 1

The reason for that is that there is no stub for kvm_cpu__teardown_pvtime() for aarch32. This fixes the compilation error for me:

arm/aarch32/include/kvm/kvm-cpu-arch.h +4 -0
index 780e0e2f0934..ae77a136d0ce 100644
@@ -19,5 +19,9 @@ static inline int kvm_cpu__configure_features(struct kvm_cpu *vcpu)
 {
        return 0;
 }
+static inline int kvm_cpu__teardown_pvtime(struct kvm *kvm)
+{
+       return 0;
+}
 
 #endif /* KVM__KVM_CPU_ARCH_H */

Thanks,
Alex

Quoted 18 lines
arm/kvm-cpu.c +0 -0
 	free(vcpu);
 }
 
include/kvm/kvm-config.h +1 -0
index 6a5720c..48adf27 100644
@@ -62,6 +62,7 @@ struct kvm_config {
 	bool no_dhcp;
 	bool ioport_debug;
 	bool mmio_debug;
+	bool no_pvtime;
 };
 
 #endif
Signature · 1 line
2.35.1.616.g0bdcbb4464-goog
#6 lore raw
Quoted 1 line · Alexandru Elisei

Hi,

Hi,

Quoted 171 lines · Sebastian Ene
Quoted 152 lines

This patch adds support for stolen time by sharing a memory region with the guest which will be used by the hypervisor to store the stolen time information. Reserve a 64kb MMIO memory region after the RTC peripheral to be used by pvtime. The exact format of the structure stored by the hypervisor is described in the ARM DEN0057A document.

---
 Makefile                               |   1 +
 arm/aarch64/arm-cpu.c                  |   2 +-
 arm/aarch64/include/kvm/kvm-cpu-arch.h |   2 +
 arm/aarch64/pvtime.c                   | 108 +++++++++++++++++++++++++
 arm/include/arm-common/kvm-arch.h      |   6 +-
 arm/kvm-cpu.c                          |   1 +
 include/kvm/kvm-config.h               |   1 +
 7 files changed, 119 insertions(+), 2 deletions(-)
 create mode 100644 arm/aarch64/pvtime.c
Makefile +1 -0
index f251147..e9121dc 100644
@@ -182,6 +182,7 @@ ifeq ($(ARCH), arm64)
 	OBJS		+= arm/aarch64/arm-cpu.o
 	OBJS		+= arm/aarch64/kvm-cpu.o
 	OBJS		+= arm/aarch64/kvm.o
+	OBJS		+= arm/aarch64/pvtime.o
 	ARCH_INCLUDE	:= $(HDRS_ARM_COMMON)
 	ARCH_INCLUDE	+= -Iarm/aarch64/include
 
arm/aarch64/arm-cpu.c +1 -1
index d7572b7..7e4a3c1 100644
@@ -22,7 +22,7 @@ static void generate_fdt_nodes(void *fdt, struct kvm *kvm)
 static int arm_cpu__vcpu_init(struct kvm_cpu *vcpu)
 {
 	vcpu->generate_fdt_nodes = generate_fdt_nodes;
-	return 0;
+	return kvm_cpu__setup_pvtime(vcpu);
 }
 
 static struct kvm_arm_target target_generic_v8 = {
arm/aarch64/include/kvm/kvm-cpu-arch.h +2 -0
index 8dfb82e..35996dc 100644
@@ -19,5 +19,7 @@
 
 void kvm_cpu__select_features(struct kvm *kvm, struct kvm_vcpu_init *init);
 int kvm_cpu__configure_features(struct kvm_cpu *vcpu);
+int kvm_cpu__setup_pvtime(struct kvm_cpu *vcpu);
+int kvm_cpu__teardown_pvtime(struct kvm *kvm);
 
 #endif /* KVM__KVM_CPU_ARCH_H */
arm/aarch64/pvtime.c new file +91 -0
new file mode 100644
index 0000000..4db2e9f
@@ -0,0 +1,108 @@
+#include "kvm/kvm.h"
+#include "kvm/kvm-cpu.h"
+#include "kvm/util.h"
+
+#include <linux/byteorder.h>
+#include <linux/types.h>
+
+#define ARM_PVTIME_STRUCT_SIZE		(64)
+
+struct pvtime_data_priv {
+	bool	is_failed_cfg;
+	void	*usr_mem;
+};
+
+static struct pvtime_data_priv pvtime_data = {
+	.is_failed_cfg	= true,
+	.usr_mem	= NULL
+};
+
+static int pvtime__alloc_region(struct kvm *kvm)
+{
+	char *mem;
+	int ret = 0;
+
+	mem = mmap(NULL, ARM_PVTIME_BASE, PROT_RW,
+		   MAP_ANON_NORESERVE, -1, 0);
+	if (mem == MAP_FAILED)
+		return -errno;
+
+	ret = kvm__register_ram(kvm, ARM_PVTIME_BASE,
+				ARM_PVTIME_BASE, mem);
+	if (ret) {
+		munmap(mem, ARM_PVTIME_BASE);
+		return ret;
+	}
+
+	pvtime_data.usr_mem = mem;
+	return ret;
+}
+
+static int pvtime__teardown_region(struct kvm *kvm)
+{
+	if (pvtime_data.usr_mem == NULL)
+		return 0;
+
+	kvm__destroy_mem(kvm, ARM_PVTIME_BASE,
+			 ARM_PVTIME_BASE, pvtime_data.usr_mem);
+	munmap(pvtime_data.usr_mem, ARM_PVTIME_BASE);
+	pvtime_data.usr_mem = NULL;
+	return 0;
+}
+
+int kvm_cpu__setup_pvtime(struct kvm_cpu *vcpu)
+{
+	int ret;
+	bool has_stolen_time;
+	u64 pvtime_guest_addr = ARM_PVTIME_BASE + vcpu->cpu_id *
+		ARM_PVTIME_STRUCT_SIZE;
+	struct kvm_config *kvm_cfg = NULL;
+	struct kvm_device_attr pvtime_attr = (struct kvm_device_attr) {
+		.group	= KVM_ARM_VCPU_PVTIME_CTRL,
+		.addr	= KVM_ARM_VCPU_PVTIME_IPA
+	};
+
+	kvm_cfg = &vcpu->kvm->cfg;
+	if (kvm_cfg->no_pvtime)
+		return 0;
+
+	if (!pvtime_data.is_failed_cfg)
+		return -ENOTSUP;
+
+	has_stolen_time = kvm__supports_extension(vcpu->kvm,
+						  KVM_CAP_STEAL_TIME);
+	if (!has_stolen_time) {
+		kvm_cfg->no_pvtime = true;
+		return 0;
+	}
+
+	ret = ioctl(vcpu->vcpu_fd, KVM_HAS_DEVICE_ATTR, &pvtime_attr);
+	if (ret) {
+		perror("KVM_HAS_DEVICE_ATTR failed\n");
+		goto out_err;
+	}
+
+	if (!pvtime_data.usr_mem) {
+		ret = pvtime__alloc_region(vcpu->kvm);
+		if (ret) {
+			perror("Failed allocating pvtime region\n");
+			goto out_err;
+		}
+	}

arm/aarch64/pvtime.c line 101

Quoted 10 lines
arm/aarch64/pvtime.c new file +10 -0
+
+	pvtime_attr.addr = (u64)&pvtime_guest_addr;
+	ret = ioctl(vcpu->vcpu_fd, KVM_SET_DEVICE_ATTR, &pvtime_attr);
+	if (!ret)
+		return 0;
+
+	perror("KVM_SET_DEVICE_ATTR failed\n");
+	pvtime__teardown_region(vcpu->kvm);
+out_err:
+	pvtime_data.is_failed_cfg = false;

kvm_cpu__init() calls kvm_cpu__arch_init()->kvm_cpu__setup_pvtime() for each VCPU from the main thread (so sequentually), not from the VCPU threads. If this function returns an error, kvm_cpu__arch_init() calls die(), which means that kvmtool will terminate without calling kvm_cpu__setup_pvtime() for the other VCPUs.

Quoted 7 lines

What I'm trying to say is that the field is_failed_cfg is not useful, because if one VCPU fails initialization, then kvmtool will not attempt to initialize the rest of the VCPUs.

If you drop is_failed_cfg you can also drop the pvtime_data_priv struct and use a static user_mem variable (up to you).

Got it, thanks I will do the cleanup by dropping those pieces.

Quoted 45 lines
Quoted 36 lines
arm/aarch64/pvtime.c new file +7 -0
+	return ret;
+}
+
+int kvm_cpu__teardown_pvtime(struct kvm *kvm)
+{
+	return pvtime__teardown_region(kvm);
+}
arm/include/arm-common/kvm-arch.h +5 -1
index c645ac0..43b1f77 100644
@@ -15,7 +15,8 @@
  * |  PCI  |////| plat  |       |        |     |         |
  * |  I/O  |////| MMIO: | Flash | virtio | GIC |   PCI   |  DRAM
  * | space |////| UART, |       |  MMIO  |     |  (AXI)  |
- * |       |////| RTC   |       |        |     |         |
+ * |       |////| RTC,  |       |        |     |         |
+ * |       |////| PVTIME|       |        |     |         |
  * +-------+----+-------+-------+--------+-----+---------+---......
  */
 
@@ -34,6 +35,9 @@
 #define ARM_RTC_MMIO_BASE	(ARM_UART_MMIO_BASE + ARM_UART_MMIO_SIZE)
 #define ARM_RTC_MMIO_SIZE	0x10000
 
+#define ARM_PVTIME_BASE		(ARM_RTC_MMIO_BASE + ARM_RTC_MMIO_SIZE)
+#define ARM_PVTIME_SIZE		SZ_64K
+
 #define KVM_FLASH_MMIO_BASE	(ARM_MMIO_AREA + 0x1000000)
 #define KVM_FLASH_MAX_SIZE	0x1000000
 
arm/kvm-cpu.c +0 -0
index 84ac1e9..00660d6 100644

arm/kvm-cpu.c void kvm_cpu__arch_nmi(struct kvm_cpu *cpu) · line 147

Quoted 4 lines
arm/kvm-cpu.c +1 -0
@@ -144,6 +144,7 @@ void kvm_cpu__arch_nmi(struct kvm_cpu *cpu)
 
 void kvm_cpu__delete(struct kvm_cpu *vcpu)
 {
+	kvm_cpu__teardown_pvtime(vcpu->kvm);

This causes compilation for aarch32 to fail:

arm/kvm-cpu.c: In function ‘kvm_cpu__delete’:
arm/kvm-cpu.c:147:2: error: implicit declaration of function ‘kvm_cpu__teardown_pvtime’ [-Werror=implicit-function-declaration]

Quoted 9 lines
  147 |  kvm_cpu__teardown_pvtime(vcpu->kvm);
      |  ^~~~~~~~~~~~~~~~~~~~~~~~

arm/kvm-cpu.c:147:2: error: nested extern declaration of ‘kvm_cpu__teardown_pvtime’ [-Werror=nested-externs]
cc1: all warnings being treated as errors
make: *** [Makefile:482: arm/kvm-cpu.o] Error 1

The reason for that is that there is no stub for kvm_cpu__teardown_pvtime() for aarch32. This fixes the compilation error for me:

I am afraid I didn't check on aarch32. Thanks for raising this issue, I will fix the problem.

Quoted 15 lines
arm/aarch32/include/kvm/kvm-cpu-arch.h +4 -0
index 780e0e2f0934..ae77a136d0ce 100644
@@ -19,5 +19,9 @@ static inline int kvm_cpu__configure_features(struct kvm_cpu *vcpu)
 {
        return 0;
 }
+static inline int kvm_cpu__teardown_pvtime(struct kvm *kvm)
+{
+       return 0;
+}
 
 #endif /* KVM__KVM_CPU_ARCH_H */
Quoted 5 lines

Thanks,
Alex

Quoted 2 lines
arm/kvm-cpu.c +0 -0
 	free(vcpu);
 }

Thanks for the review,
Sebastian

Quoted 16 lines
Quoted 16 lines
include/kvm/kvm-config.h +1 -0
index 6a5720c..48adf27 100644
@@ -62,6 +62,7 @@ struct kvm_config {
 	bool no_dhcp;
 	bool ioport_debug;
 	bool mmio_debug;
+	bool no_pvtime;
 };
 
 #endif
Signature · 1 line
2.35.1.616.g0bdcbb4464-goog
#7 lore raw

PATCH v8 3/3 Add --no-pvtime command line argument

In reply to Sebastian Ene

The command line argument disables the stolen time functionality when is specified.

---
 builtin-run.c | 2 ++
 1 file changed, 2 insertions(+)
builtin-run.c +2 -0
index 9a1a0c1..7c8be9d 100644
@@ -128,6 +128,8 @@ void kvm_run_set_wrapper_sandbox(void)
 			" rootfs"),					\
 	OPT_STRING('\0', "hugetlbfs", &(cfg)->hugetlbfs_path, "path",	\
 			"Hugetlbfs path"),				\
+	OPT_BOOLEAN('\0', "no-pvtime", &(cfg)->no_pvtime, "Disable"	\
+			" stolen time"),				\
 									\
 	OPT_GROUP("Kernel options:"),					\
 	OPT_STRING('k', "kernel", &(cfg)->kernel_filename, "kernel",	\
Signature · 1 line
2.35.1.616.g0bdcbb4464-goog
Reviewed-by on v8 3/3
#8 lore raw

Re: [PATCH kvmtool v8 3/3] Add --no-pvtime command line argument

Hi,

Looks good to me:

Thanks,
Alex

Quoted 24 lines · Sebastian Ene

The command line argument disables the stolen time functionality when is specified.

---
 builtin-run.c | 2 ++
 1 file changed, 2 insertions(+)
builtin-run.c +2 -0
index 9a1a0c1..7c8be9d 100644
@@ -128,6 +128,8 @@ void kvm_run_set_wrapper_sandbox(void)
 			" rootfs"),					\
 	OPT_STRING('\0', "hugetlbfs", &(cfg)->hugetlbfs_path, "path",	\
 			"Hugetlbfs path"),				\
+	OPT_BOOLEAN('\0', "no-pvtime", &(cfg)->no_pvtime, "Disable"	\
+			" stolen time"),				\
 									\
 	OPT_GROUP("Kernel options:"),					\
 	OPT_STRING('k', "kernel", &(cfg)->kernel_filename, "kernel",	\
Signature · 1 line
2.35.1.616.g0bdcbb4464-goog