Skip to contentSkip to conversation

target/ppc: vcpu hotplug failure handling fixes

PATCH Superseded Harsh Prateek Bora 4 participants 14 messages May 16, 2024 – May 17, 2024 1 message not in the archive

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover target/ppc: vcpu hotplug failure handling fixes

On ppc64, the PowerVM hypervisor runs with limited memory and a VCPU creation during hotplug may fail during kvm_ioctl for KVM_CREATE_VCPU, leading to termination of guest since errp is set to &error_fatal while calling kvm_init_vcpu. This unexpected behaviour can be avoided by pre-creating and parking vcpu on success or return error otherwise. This enables graceful error delivery for any vcpu hotplug failures while the guest can keep running.

This series adds another helper to create and park vcpu (based on below patch by Salil), exports cpu_get_free_index to be reused later and adds ppc arch specfic handling for vcpu hotplug failure.

Based on api refactoring to create/park vcpus introduced in 1/8 of patch series:
https://lore.kernel.org/qemu-devel/[email protected]/

PS: I have just included patch 1 of above series after fixing a rebase failure along with this series for better review purpose only.

Changelog:

v2: Addressed review comments from Nick
v1: Initial patch

Harsh Prateek Bora (3):
accel/kvm: Introduce kvm_create_and_park_vcpu() helper cpu-common.c: export cpu_get_free_index to be reused later target/ppc: handle vcpu hotplug failure gracefully

Salil Mehta (1):
accel/kvm: Extract common KVM vCPU {creation, parking} code

 include/exec/cpu-common.h |  2 ++
 include/sysemu/kvm.h      | 23 ++++++++++++
 accel/kvm/kvm-all.c       | 76 +++++++++++++++++++++++++++++++--------
 cpu-common.c              |  7 ++--
 target/ppc/kvm.c          | 24 +++++++++++++
 accel/kvm/trace-events    |  5 ++-
 6 files changed, 118 insertions(+), 19 deletions(-)
Signature · 1 line
2.39.3
#1 lore raw

PATCH v2 1/4 accel/kvm: Extract common KVM vCPU {creation, parking} code

From: Salil Mehta <[email protected]>

KVM vCPU creation is done once during the vCPU realization when Qemu vCPU thread is spawned. This is common to all the architectures as of now.

Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the corresponding KVM vCPU object in the Host KVM is not destroyed as KVM doesn't support vCPU removal. Therefore, its representative KVM vCPU object/context in Qemu is parked.

Refactor architecture common logic so that some APIs could be reused by vCPU Hotplug code of some architectures likes ARM, Loongson etc. Update new/old APIs with trace events instead of DPRINTF. No functional change is intended here.

---
 include/sysemu/kvm.h   | 15 ++++++++++
 accel/kvm/kvm-all.c    | 64 ++++++++++++++++++++++++++++++++----------
 accel/kvm/trace-events |  5 +++-
 3 files changed, 68 insertions(+), 16 deletions(-)
include/sysemu/kvm.h +15 -0
index eaf801bc93..fa3ec74442 100644
@@ -434,6 +434,21 @@ void kvm_set_sigmask_len(KVMState *s, unsigned int sigmask_len);
 
 int kvm_physical_memory_addr_from_host(KVMState *s, void *ram_addr,
                                        hwaddr *phys_addr);
+/**
+ * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM vCPU
+ * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_vcpu(CPUState *cpu);
+
+/**
+ * kvm_park_vcpu - Park QEMU KVM vCPU context
+ * @cpu: QOM CPUState object for which QEMU KVM vCPU context has to be parked.
+ *
+ * @returns: none
+ */
+void kvm_park_vcpu(CPUState *cpu);
 
 #endif /* COMPILING_PER_TARGET */
 
accel/kvm/kvm-all.c +49 -15

111 lines not on this page · raw message

accel/kvm/trace-events +4 -1
index 681ccb667d..75c1724e78 100644
@@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
 kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to retrieve ONEREG %" PRIu64 " from KVM: %s"
 kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set ONEREG %" PRIu64 " to KVM: %s"
 kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
+kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
+kvm_get_vcpu(unsigned long arch_cpu_id) "id: %lu"
+kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
+kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu"
 kvm_irqchip_commit_routes(void) ""
 kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s vector %d virq %d"
 kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
@@ -25,7 +29,6 @@ kvm_dirty_ring_reaper(const char *s) "%s"
 kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64" pages (took %"PRIi64" us)"
 kvm_dirty_ring_reaper_kick(const char *reason) "%s"
 kvm_dirty_ring_flush(int finished) "%d"
-kvm_destroy_vcpu(void) ""
 kvm_failed_get_vcpu_mmap_size(void) ""
 kvm_cpu_exec(void) ""
 kvm_interrupt_exit_request(void) ""
Signature · 1 line
2.39.3
#2 lore raw

RE: [PATCH v2 1/4] accel/kvm: Extract common KVM vCPU {creation, parking} code

Hi Harsh,

Thanks for your interest in the patch-set but taking away patches like this from other series without any discussion can disrupt others work and its acceptance on time. This is because we will have to put lot of effort in rebasing bigger series and then testing overhead comes along with it.

The patch-set (from where this patch has been taken) is part of even bigger series and there have been many people and companies toiling to fix the bugs collectively in that series and for years.

I'm about float the V9 version of the Arch agnostic series which this patch is part of and you can rebase your patch-set from there. I'm hopeful that it will get accepted in this cycle.

Many thanks
Salil.

Quoted 220 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 6:32 AM

From: Salil Mehta <[email protected]>

KVM vCPU creation is done once during the vCPU realization when Qemu vCPU thread is spawned. This is common to all the architectures as of now.

Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the corresponding KVM vCPU object in the Host KVM is not destroyed as KVM doesn't support vCPU removal. Therefore, its representative KVM vCPU object/context in Qemu is parked.

Refactor architecture common logic so that some APIs could be reused by vCPU Hotplug code of some architectures likes ARM, Loongson etc. Update new/old APIs with trace events instead of DPRINTF. No functional change is intended here.

---

  include/sysemu/kvm.h   | 15 ++++++++++
  accel/kvm/kvm-all.c    | 64 ++++++++++++++++++++++++++++++++--------
 --
  accel/kvm/trace-events |  5 +++-
  3 files changed, 68 insertions(+), 16 deletions(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index eaf801bc93..fa3ec74442 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -434,6 +434,21 @@ void kvm_set_sigmask_len(KVMState *s, unsigned int sigmask_len);

int kvm_physical_memory_addr_from_host(KVMState *s, void
*ram_addr,

                                         hwaddr *phys_addr);

+/**
+ * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM vCPU
+ * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_vcpu(CPUState *cpu);
+
+/**
+ * kvm_park_vcpu - Park QEMU KVM vCPU context
+ * @cpu: QOM CPUState object for which QEMU KVM vCPU context has to be parked.
+ *
+ * @returns: none
+ */
+void kvm_park_vcpu(CPUState *cpu);

  #endif /* COMPILING_PER_TARGET */

diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index d7281b93f3..30d42847de 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -128,6 +128,7 @@ static QemuMutex kml_slots_lock; #define kvm_slots_unlock() qemu_mutex_unlock(&kml_slots_lock)

static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
+static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id);

static inline void kvm_resample_fd_remove(int gsi) { @@ -340,14 +341,53
@@ err:

      return ret;

}

+void kvm_park_vcpu(CPUState *cpu)
+{
+ struct KVMParkedVcpu *vcpu;
+
+ trace_kvm_park_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
+

 +    vcpu = g_malloc0(sizeof(*vcpu));
 +    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
 +    vcpu->kvm_fd = cpu->kvm_fd;
 +    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node); }
 +
 +int kvm_create_vcpu(CPUState *cpu)
 +{
 +    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
 +    KVMState *s = kvm_state;
 +    int kvm_fd;
 +
 +    trace_kvm_create_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
 +
 +    /* check if the KVM vCPU already exist but is parked */
 +    kvm_fd = kvm_get_vcpu(s, vcpu_id);
 +    if (kvm_fd < 0) {
 +        /* vCPU not parked: create a new KVM vCPU */
 +        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
 +        if (kvm_fd < 0) {
 +            error_report("KVM_CREATE_VCPU IOCTL failed for vCPU %lu",
 vcpu_id);
 +            return kvm_fd;
 +        }
 +    }
 +
 +    cpu->kvm_fd = kvm_fd;
 +    cpu->kvm_state = s;
 +    cpu->vcpu_dirty = true;
 +    cpu->dirty_pages = 0;
 +    cpu->throttle_us_per_full = 0;
 +
 +    return 0;
 +}
 +
  static int do_kvm_destroy_vcpu(CPUState *cpu)  {
      KVMState *s = kvm_state;
      long mmap_size;
 -    struct KVMParkedVcpu *vcpu = NULL;
      int ret = 0;

- trace_kvm_destroy_vcpu();
+ trace_kvm_destroy_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));

      ret = kvm_arch_destroy_vcpu(cpu);
      if (ret < 0) {
 @@ -373,10 +413,7 @@ static int do_kvm_destroy_vcpu(CPUState *cpu)
          }
      }
 -    vcpu = g_malloc0(sizeof(*vcpu));
 -    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
 -    vcpu->kvm_fd = cpu->kvm_fd;
 -    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
 +    kvm_park_vcpu(cpu);
  err:
      return ret;
  }
 @@ -397,6 +434,8 @@ static int kvm_get_vcpu(KVMState *s, unsigned long
 vcpu_id)
          if (cpu->vcpu_id == vcpu_id) {
              int kvm_fd;

+ trace_kvm_get_vcpu(vcpu_id);
+

              QLIST_REMOVE(cpu, node);
              kvm_fd = cpu->kvm_fd;
              g_free(cpu);
 @@ -404,7 +443,7 @@ static int kvm_get_vcpu(KVMState *s, unsigned long
 vcpu_id)
          }
      }

- return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
+ return -ENOENT;
}

int kvm_init_vcpu(CPUState *cpu, Error **errp) @@ -415,19 +454,14 @@
int kvm_init_vcpu(CPUState *cpu, Error **errp)

      trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));

- ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));

 +    ret = kvm_create_vcpu(cpu);
      if (ret < 0) {
 -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed
 (%lu)",
 +        error_setg_errno(errp, -ret,
 +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
                           kvm_arch_vcpu_id(cpu));
          goto err;
      }
 -    cpu->kvm_fd = ret;
 -    cpu->kvm_state = s;
 -    cpu->vcpu_dirty = true;
 -    cpu->dirty_pages = 0;
 -    cpu->throttle_us_per_full = 0;
 -
      mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
      if (mmap_size < 0) {
          ret = mmap_size;

diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events index 681ccb667d..75c1724e78 100644
--- a/accel/kvm/trace-events
+++ b/accel/kvm/trace-events
@@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to
retrieve ONEREG %" PRIu64 " from KVM: %s"
kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set
ONEREG %" PRIu64 " to KVM: %s"
kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id:
%lu"
+kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d
id: %lu"
+kvm_get_vcpu(unsigned long arch_cpu_id) "id: %lu" +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d
id: %lu"
+kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id:
%lu"
kvm_irqchip_commit_routes(void) "" kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s
vector %d virq %d"
kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
@@ -25,7 +29,6 @@ kvm_dirty_ring_reaper(const char *s) "%s"
kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64" pages
(took %"PRIi64" us)"
kvm_dirty_ring_reaper_kick(const char *reason) "%s" kvm_dirty_ring_flush(int finished) "%d"
-kvm_destroy_vcpu(void) ""
kvm_failed_get_vcpu_mmap_size(void) ""
kvm_cpu_exec(void) ""
kvm_interrupt_exit_request(void) ""
--
2.39.3

#3 lore raw

Re: [PATCH v2 1/4] accel/kvm: Extract common KVM vCPU {creation, parking} code

Hi Salil,

Thanks for your email.
Your patch 1/8 is included here based on review comments on my previous patch from one of the maintainers in the community and therefore I had kept you in CC to be aware of the desire of having this independent patch to get merged earlier even if your other patches in the series may go through further reviews.

I am hoping to see your v9 soon and thereafter maintainer(s) may choose to pick the latest independent patch if needs to be merged earlier.

Thanks for your work and let's be hopeful it gets merged soon.

regards,
Harsh

Quoted 241 lines · Salil Mehta

Hi Harsh,

Thanks for your interest in the patch-set but taking away patches like this from other series without any discussion can disrupt others work and its acceptance on time. This is because we will have to put lot of effort in rebasing bigger series and then testing overhead comes along with it.

The patch-set (from where this patch has been taken) is part of even bigger series and there have been many people and companies toiling to fix the bugs collectively in that series and for years.

I'm about float the V9 version of the Arch agnostic series which this patch is part of and you can rebase your patch-set from there. I'm hopeful that it will get accepted in this cycle.

Many thanks
Salil.

Quoted 220 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 6:32 AM

From: Salil Mehta <[email protected]>

KVM vCPU creation is done once during the vCPU realization when Qemu vCPU thread is spawned. This is common to all the architectures as of now.

Hot-unplug of vCPU results in destruction of the vCPU object in QOM but the corresponding KVM vCPU object in the Host KVM is not destroyed as KVM doesn't support vCPU removal. Therefore, its representative KVM vCPU object/context in Qemu is parked.

Refactor architecture common logic so that some APIs could be reused by vCPU Hotplug code of some architectures likes ARM, Loongson etc. Update new/old APIs with trace events instead of DPRINTF. No functional change is intended here.

---

   include/sysemu/kvm.h   | 15 ++++++++++
   accel/kvm/kvm-all.c    | 64 ++++++++++++++++++++++++++++++++--------
  --
   accel/kvm/trace-events |  5 +++-
   3 files changed, 68 insertions(+), 16 deletions(-)
  diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index
  eaf801bc93..fa3ec74442 100644
  --- a/include/sysemu/kvm.h
  +++ b/include/sysemu/kvm.h
  @@ -434,6 +434,21 @@ void kvm_set_sigmask_len(KVMState *s, unsigned
  int sigmask_len);

int kvm_physical_memory_addr_from_host(KVMState *s, void
*ram_addr,

                                          hwaddr *phys_addr);

+/**
+ * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM vCPU
+ * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_vcpu(CPUState *cpu);
+
+/**
+ * kvm_park_vcpu - Park QEMU KVM vCPU context
+ * @cpu: QOM CPUState object for which QEMU KVM vCPU context has to be parked.
+ *
+ * @returns: none
+ */
+void kvm_park_vcpu(CPUState *cpu);

   #endif /* COMPILING_PER_TARGET */
  diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index
  d7281b93f3..30d42847de 100644
  --- a/accel/kvm/kvm-all.c
  +++ b/accel/kvm/kvm-all.c
  @@ -128,6 +128,7 @@ static QemuMutex kml_slots_lock;  #define
  kvm_slots_unlock()  qemu_mutex_unlock(&kml_slots_lock)
   static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
  +static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id);
  
   static inline void kvm_resample_fd_remove(int gsi)  { @@ -340,14 +341,53
  @@ err:
       return ret;
   }

+void kvm_park_vcpu(CPUState *cpu)
+{
+ struct KVMParkedVcpu *vcpu;
+
+ trace_kvm_park_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
+

  +    vcpu = g_malloc0(sizeof(*vcpu));
  +    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  +    vcpu->kvm_fd = cpu->kvm_fd;
  +    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node); }
  +
  +int kvm_create_vcpu(CPUState *cpu)
  +{
  +    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
  +    KVMState *s = kvm_state;
  +    int kvm_fd;
  +
  +    trace_kvm_create_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
  +
  +    /* check if the KVM vCPU already exist but is parked */
  +    kvm_fd = kvm_get_vcpu(s, vcpu_id);
  +    if (kvm_fd < 0) {
  +        /* vCPU not parked: create a new KVM vCPU */
  +        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
  +        if (kvm_fd < 0) {
  +            error_report("KVM_CREATE_VCPU IOCTL failed for vCPU %lu",
  vcpu_id);
  +            return kvm_fd;
  +        }
  +    }
  +
  +    cpu->kvm_fd = kvm_fd;
  +    cpu->kvm_state = s;
  +    cpu->vcpu_dirty = true;
  +    cpu->dirty_pages = 0;
  +    cpu->throttle_us_per_full = 0;
  +
  +    return 0;
  +}
  +
   static int do_kvm_destroy_vcpu(CPUState *cpu)  {
       KVMState *s = kvm_state;
       long mmap_size;
  -    struct KVMParkedVcpu *vcpu = NULL;
       int ret = 0;

- trace_kvm_destroy_vcpu();
+ trace_kvm_destroy_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));

       ret = kvm_arch_destroy_vcpu(cpu);
       if (ret < 0) {
  @@ -373,10 +413,7 @@ static int do_kvm_destroy_vcpu(CPUState *cpu)
           }
       }
  -    vcpu = g_malloc0(sizeof(*vcpu));
  -    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  -    vcpu->kvm_fd = cpu->kvm_fd;
  -    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node);
  +    kvm_park_vcpu(cpu);
   err:
       return ret;
   }
  @@ -397,6 +434,8 @@ static int kvm_get_vcpu(KVMState *s, unsigned long
  vcpu_id)
           if (cpu->vcpu_id == vcpu_id) {
               int kvm_fd;
  +            trace_kvm_get_vcpu(vcpu_id);
  +
               QLIST_REMOVE(cpu, node);
               kvm_fd = cpu->kvm_fd;
               g_free(cpu);
  @@ -404,7 +443,7 @@ static int kvm_get_vcpu(KVMState *s, unsigned long
  vcpu_id)
           }
       }

- return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);

  +    return -ENOENT;
   }
  
   int kvm_init_vcpu(CPUState *cpu, Error **errp) @@ -415,19 +454,14 @@

int kvm_init_vcpu(CPUState *cpu, Error **errp)

       trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));

- ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));

  +    ret = kvm_create_vcpu(cpu);
       if (ret < 0) {
  -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed
  (%lu)",
  +        error_setg_errno(errp, -ret,
  +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
                            kvm_arch_vcpu_id(cpu));
           goto err;
       }
  -    cpu->kvm_fd = ret;
  -    cpu->kvm_state = s;
  -    cpu->vcpu_dirty = true;
  -    cpu->dirty_pages = 0;
  -    cpu->throttle_us_per_full = 0;
  -
       mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
       if (mmap_size < 0) {
           ret = mmap_size;

diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events index 681ccb667d..75c1724e78 100644
--- a/accel/kvm/trace-events
+++ b/accel/kvm/trace-events
@@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd %d, type 0x%x, arg %p"
kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to
retrieve ONEREG %" PRIu64 " from KVM: %s"
kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to set
ONEREG %" PRIu64 " to KVM: %s"
kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id:
%lu"
+kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d
id: %lu"
+kvm_get_vcpu(unsigned long arch_cpu_id) "id: %lu" +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d
id: %lu"
+kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id:
%lu"
kvm_irqchip_commit_routes(void) "" kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s
vector %d virq %d"
kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d"
@@ -25,7 +29,6 @@ kvm_dirty_ring_reaper(const char *s) "%s"
kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64" pages
(took %"PRIi64" us)"
kvm_dirty_ring_reaper_kick(const char *reason) "%s" kvm_dirty_ring_flush(int finished) "%d"
-kvm_destroy_vcpu(void) ""
kvm_failed_get_vcpu_mmap_size(void) ""
kvm_cpu_exec(void) ""
kvm_interrupt_exit_request(void) ""
--
2.39.3

#4 lore raw

RE: [PATCH v2 1/4] accel/kvm: Extract common KVM vCPU {creation, parking} code

Hi Harsh,

Quoted 11 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 11:15 AM

Hi Salil,

Thanks for your email.
Your patch 1/8 is included here based on review comments on my previous patch from one of the maintainers in the community and therefore I had kept you in CC to be aware of the desire of having this independent patch to get merged earlier even if your other patches in the series may go through further reviews.

I really don’t know which discussion are you pointing at? Please understand you are fixing a bug and we are pushing a feature which has got large series. It will break the patch-set which is about t be merged.

There will be significant overhead of testing on us for the work we have been carrying forward for large time. This will be disruptive. Please dont!

Quoted 3 lines

I am hoping to see your v9 soon and thereafter maintainer(s) may choose to pick the latest independent patch if needs to be merged earlier.

I don’t think you are understanding what problem it is causing. For your small bug fix you are causing significant delays at our end.

Thanks
Salil.

Quoted 275 lines

Thanks for your work and let's be hopeful it gets merged soon.

regards,
Harsh

Quoted 12 lines · Salil Mehta

Hi Harsh,

Thanks for your interest in the patch-set but taking away patches like this from other series without any discussion can disrupt others work and its acceptance on time. This is because we will have to put lot of effort in rebasing bigger series and then testing overhead comes along with it.

The patch-set (from where this patch has been taken) is part of even bigger series and there have been many people and companies toiling to fix the bugs collectively in that series and for years.

Quoted 14 lines

I'm about float the V9 version of the Arch agnostic series which this patch is part of and you can rebase your patch-set from there. I'm hopeful that it will get accepted in this cycle.

Many thanks
Salil.

Quoted 6 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 6:32 AM

From: Salil Mehta <[email protected]>

KVM vCPU creation is done once during the vCPU realization when

Qemu

Quoted 1 line
Quoted 1 line

vCPU thread is spawned. This is common to all the architectures as of

now.

Quoted 2 lines
Quoted 2 lines

Hot-unplug of vCPU results in destruction of the vCPU object in QOM

but

Quoted 1 line
Quoted 1 line

the corresponding KVM vCPU object in the Host KVM is not destroyed

as

Quoted 4 lines
Quoted 4 lines

KVM doesn't support vCPU removal. Therefore, its representative KVM vCPU object/context in Qemu is parked.

Refactor architecture common logic so that some APIs could be reused

by

Quoted 1 line
Quoted 1 line

vCPU Hotplug code of some architectures likes ARM, Loongson etc.

Update

Quoted 1 line
Quoted 1 line

new/old APIs with trace events instead of DPRINTF. No functional

change is

Quoted 14 lines
Quoted 11 lines

intended here.

Quoted 3 lines

---

   include/sysemu/kvm.h   | 15 ++++++++++
   accel/kvm/kvm-all.c    | 64 ++++++++++++++++++++++++++++++++---

-----

Quoted 9 lines
Quoted 9 lines

--

   accel/kvm/trace-events |  5 +++-
   3 files changed, 68 insertions(+), 16 deletions(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index eaf801bc93..fa3ec74442 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -434,6 +434,21 @@ void kvm_set_sigmask_len(KVMState *s,

unsigned

Quoted 7 lines
Quoted 7 lines

int sigmask_len);

int kvm_physical_memory_addr_from_host(KVMState *s, void
*ram_addr,

                                          hwaddr *phys_addr);

+/**
+ * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM

vCPU

Quoted 10 lines
Quoted 10 lines

+ * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_vcpu(CPUState *cpu);
+
+/**
+ * kvm_park_vcpu - Park QEMU KVM vCPU context
+ * @cpu: QOM CPUState object for which QEMU KVM vCPU context

has to

Quoted 15 lines
Quoted 15 lines

be parked.
+ *
+ * @returns: none
+ */
+void kvm_park_vcpu(CPUState *cpu);

   #endif /* COMPILING_PER_TARGET */
  diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index
  d7281b93f3..30d42847de 100644
  --- a/accel/kvm/kvm-all.c
  +++ b/accel/kvm/kvm-all.c
  @@ -128,6 +128,7 @@ static QemuMutex kml_slots_lock;  #define
  kvm_slots_unlock()  qemu_mutex_unlock(&kml_slots_lock)
Quoted 4 lines
Quoted 4 lines
   static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
  +static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id);

static inline void kvm_resample_fd_remove(int gsi) { @@ -340,14

+341,53

Quoted 4 lines
Quoted 4 lines
  @@ err:
       return ret;
   }
Quoted 10 lines
Quoted 10 lines

+void kvm_park_vcpu(CPUState *cpu)
+{
+ struct KVMParkedVcpu *vcpu;
+
+ trace_kvm_park_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
+

  +    vcpu = g_malloc0(sizeof(*vcpu));
  +    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  +    vcpu->kvm_fd = cpu->kvm_fd;
  +    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,

node); }

Quoted 36 lines
Quoted 36 lines

+
+int kvm_create_vcpu(CPUState *cpu)
+{

  +    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
  +    KVMState *s = kvm_state;
  +    int kvm_fd;
  +
  +    trace_kvm_create_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
  +
  +    /* check if the KVM vCPU already exist but is parked */
  +    kvm_fd = kvm_get_vcpu(s, vcpu_id);
  +    if (kvm_fd < 0) {
  +        /* vCPU not parked: create a new KVM vCPU */
  +        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
  +        if (kvm_fd < 0) {
  +            error_report("KVM_CREATE_VCPU IOCTL failed for vCPU %lu",
  vcpu_id);
  +            return kvm_fd;
  +        }
  +    }
  +
  +    cpu->kvm_fd = kvm_fd;
  +    cpu->kvm_state = s;
  +    cpu->vcpu_dirty = true;
  +    cpu->dirty_pages = 0;
  +    cpu->throttle_us_per_full = 0;
  +
  +    return 0;
  +}
  +
   static int do_kvm_destroy_vcpu(CPUState *cpu)  {
       KVMState *s = kvm_state;
       long mmap_size;
  -    struct KVMParkedVcpu *vcpu = NULL;
       int ret = 0;
Quoted 2 lines
Quoted 2 lines

- trace_kvm_destroy_vcpu();
+ trace_kvm_destroy_vcpu(cpu->cpu_index,

kvm_arch_vcpu_id(cpu));

Quoted 4 lines
Quoted 4 lines
       ret = kvm_arch_destroy_vcpu(cpu);
       if (ret < 0) {

@@ -373,10 +413,7 @@ static int do_kvm_destroy_vcpu(CPUState

*cpu)

Quoted 7 lines
Quoted 7 lines
           }
       }
  -    vcpu = g_malloc0(sizeof(*vcpu));
  -    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  -    vcpu->kvm_fd = cpu->kvm_fd;
  -    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,

node);

Quoted 5 lines
Quoted 5 lines
  +    kvm_park_vcpu(cpu);
   err:
       return ret;
   }

@@ -397,6 +434,8 @@ static int kvm_get_vcpu(KVMState *s, unsigned

long

Quoted 10 lines
Quoted 10 lines
  vcpu_id)
           if (cpu->vcpu_id == vcpu_id) {
               int kvm_fd;

  +            trace_kvm_get_vcpu(vcpu_id);
  +
               QLIST_REMOVE(cpu, node);
               kvm_fd = cpu->kvm_fd;
               g_free(cpu);
  @@ -404,7 +443,7 @@ static int kvm_get_vcpu(KVMState *s, unsigned

long

Quoted 9 lines
Quoted 9 lines
  vcpu_id)
           }
       }

- return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);

  +    return -ENOENT;
   }

   int kvm_init_vcpu(CPUState *cpu, Error **errp) @@ -415,19 +454,14

@@

Quoted 15 lines
Quoted 15 lines

int kvm_init_vcpu(CPUState *cpu, Error **errp)

       trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));

- ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));

  +    ret = kvm_create_vcpu(cpu);
       if (ret < 0) {
  -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed
  (%lu)",
  +        error_setg_errno(errp, -ret,
  +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
                            kvm_arch_vcpu_id(cpu));
           goto err;
       }
Quoted 14 lines
Quoted 9 lines
  -    cpu->kvm_fd = ret;
  -    cpu->kvm_state = s;
  -    cpu->vcpu_dirty = true;
  -    cpu->dirty_pages = 0;
  -    cpu->throttle_us_per_full = 0;
  -
       mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
       if (mmap_size < 0) {
           ret = mmap_size;
Quoted 5 lines

diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events index 681ccb667d..75c1724e78 100644
--- a/accel/kvm/trace-events
+++ b/accel/kvm/trace-events
@@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd

%d,

Quoted 4 lines
Quoted 4 lines

type 0x%x, arg %p"
kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to
retrieve ONEREG %" PRIu64 " from KVM: %s"
kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to

set

Quoted 2 lines
Quoted 2 lines

ONEREG %" PRIu64 " to KVM: %s"
kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d

id:

Quoted 2 lines
Quoted 2 lines

%lu"
+kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "index:

%d

Quoted 3 lines
Quoted 3 lines

id: %lu"
+kvm_get_vcpu(unsigned long arch_cpu_id) "id: %lu" +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "index:

%d

Quoted 2 lines
Quoted 2 lines

id: %lu"
+kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d

id:

Quoted 5 lines
Quoted 5 lines

%lu"
kvm_irqchip_commit_routes(void) "" kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s
vector %d virq %d"
kvm_irqchip_update_msi_route(int virq) "Updating MSI route

virq=%d"

Quoted 2 lines
Quoted 2 lines

@@ -25,7 +29,6 @@ kvm_dirty_ring_reaper(const char *s) "%s"
kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64"

pages

Quoted 10 lines
Quoted 9 lines

(took %"PRIi64" us)"
kvm_dirty_ring_reaper_kick(const char *reason) "%s" kvm_dirty_ring_flush(int finished) "%d"
-kvm_destroy_vcpu(void) ""
kvm_failed_get_vcpu_mmap_size(void) "" kvm_cpu_exec(void) "" kvm_interrupt_exit_request(void) ""
--
2.39.3

#5 lore raw

Re: [PATCH v2 1/4] accel/kvm: Extract common KVM vCPU {creation, parking} code

Hi Salil,

Quoted 14 lines · Salil Mehta

Hi Harsh,

Quoted 11 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 11:15 AM

Hi Salil,

Thanks for your email.
Your patch 1/8 is included here based on review comments on my previous patch from one of the maintainers in the community and therefore I had kept you in CC to be aware of the desire of having this independent patch to get merged earlier even if your other patches in the series may go through further reviews.

Quoted 7 lines

I really don’t know which discussion are you pointing at? Please understand you are fixing a bug and we are pushing a feature which has got large series. It will break the patch-set which is about t be merged.

There will be significant overhead of testing on us for the work we have been carrying forward for large time. This will be disruptive. Please dont!

I was referring to the review discussion on my prev patch here:
https://lore.kernel.org/qemu-devel/[email protected]/

Although your patch was included with this series only to facilitate review of the additional patches depending on just one of your patch.

I am not sure what is appearing disruptive here. It is a common practive in the community that maintainer(s) can pick individual patches from the series if it has been vetted by siginificant number of reviewers.

However, in this case, since you have mentioned to post next version soon, you need not worry about it as that would be the preferred version for both of the series.

Quoted 9 lines
Quoted 4 lines

I am hoping to see your v9 soon and thereafter maintainer(s) may choose to pick the latest independent patch if needs to be merged earlier.

I don’t think you are understanding what problem it is causing. For your small bug fix you are causing significant delays at our end.

I hope I clarfied above that including your patch here doesnt delay anything. Hoping to see your v9 soon!

Thanks
Harsh

Quoted 278 lines

Thanks
Salil.

Quoted 275 lines

Thanks for your work and let's be hopeful it gets merged soon.

regards,
Harsh

  On 5/16/24 14:00, Salil Mehta wrote:
  > Hi Harsh,
  >
  > Thanks for your interest in the patch-set but taking away patches like
  > this from other series without any discussion can disrupt others work
  > and its acceptance on time. This is because we will have to put lot of
  > effort in rebasing bigger series and then testing overhead comes along
  > with it.
  >
  > The patch-set (from where this  patch has been taken) is part of even
  > bigger series and there have been many people and companies toiling to
  > fix the bugs collectively in that series and for years.
  >
  > I'm about float the V9 version of the Arch agnostic series which this
  > patch is part of and you can rebase your patch-set from there. I'm
  > hopeful that it will get accepted in this cycle.
  >
  >
  > Many thanks
  > Salil.
  >
  >>   From: Harsh Prateek Bora <[email protected]>
  >>   Sent: Thursday, May 16, 2024 6:32 AM
  >>
  >>   From: Salil Mehta <[email protected]>
  >>
  >>   KVM vCPU creation is done once during the vCPU realization when
  Qemu
  >>   vCPU thread is spawned. This is common to all the architectures as of
  now.
  >>
  >>   Hot-unplug of vCPU results in destruction of the vCPU object in QOM
  but
  >>   the corresponding KVM vCPU object in the Host KVM is not destroyed
  as
  >>   KVM doesn't support vCPU removal. Therefore, its representative KVM
  >>   vCPU object/context in Qemu is parked.
  >>
  >>   Refactor architecture common logic so that some APIs could be reused
  by
  >>   vCPU Hotplug code of some architectures likes ARM, Loongson etc.
  Update
  >>   new/old APIs with trace events instead of DPRINTF. No functional
  change is
  >>   intended here.
  >>
  >>   Signed-off-by: Salil Mehta <[email protected]>
  >>   Reviewed-by: Gavin Shan <[email protected]>
  >>   Tested-by: Vishnu Pajjuri <[email protected]>
  >>   Reviewed-by: Jonathan Cameron <[email protected]>
  >>   Tested-by: Xianglai Li <[email protected]>
  >>   Tested-by: Miguel Luis <[email protected]>
  >>   Reviewed-by: Shaoqin Huang <[email protected]>
  >>   [harshpb: fixed rebase failures in include/sysemu/kvm.h]
  >>   Signed-off-by: Harsh Prateek Bora <[email protected]>
  >>   ---
  >>    include/sysemu/kvm.h   | 15 ++++++++++
  >>    accel/kvm/kvm-all.c    | 64 ++++++++++++++++++++++++++++++++---
  -----
  >>   --
  >>    accel/kvm/trace-events |  5 +++-
  >>    3 files changed, 68 insertions(+), 16 deletions(-)
  >>
  >>   diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index
  >>   eaf801bc93..fa3ec74442 100644
  >>   --- a/include/sysemu/kvm.h
  >>   +++ b/include/sysemu/kvm.h
  >>   @@ -434,6 +434,21 @@ void kvm_set_sigmask_len(KVMState *s,
  unsigned
  >>   int sigmask_len);
  >>
  >>    int kvm_physical_memory_addr_from_host(KVMState *s, void
  >>   *ram_addr,
  >>                                           hwaddr *phys_addr);
  >>   +/**
  >>   + * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM
  vCPU
  >>   + * @cpu: QOM CPUState object for which KVM vCPU has to be
  >>   fetched/created.
  >>   + *
  >>   + * @returns: 0 when success, errno (<0) when failed.
  >>   + */
  >>   +int kvm_create_vcpu(CPUState *cpu);
  >>   +
  >>   +/**
  >>   + * kvm_park_vcpu - Park QEMU KVM vCPU context
  >>   + * @cpu: QOM CPUState object for which QEMU KVM vCPU context
  has to
  >>   be parked.
  >>   + *
  >>   + * @returns: none
  >>   + */
  >>   +void kvm_park_vcpu(CPUState *cpu);
  >>
  >>    #endif /* COMPILING_PER_TARGET */
  >>
  >>   diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index
  >>   d7281b93f3..30d42847de 100644
  >>   --- a/accel/kvm/kvm-all.c
  >>   +++ b/accel/kvm/kvm-all.c
  >>   @@ -128,6 +128,7 @@ static QemuMutex kml_slots_lock;  #define
  >>   kvm_slots_unlock()  qemu_mutex_unlock(&kml_slots_lock)
  >>
  >>    static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
  >>   +static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id);
  >>
  >>    static inline void kvm_resample_fd_remove(int gsi)  { @@ -340,14
  +341,53
  >>   @@ err:
  >>        return ret;
  >>    }
  >>
  >>   +void kvm_park_vcpu(CPUState *cpu)
  >>   +{
  >>   +    struct KVMParkedVcpu *vcpu;
  >>   +
  >>   +    trace_kvm_park_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
  >>   +
  >>   +    vcpu = g_malloc0(sizeof(*vcpu));
  >>   +    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  >>   +    vcpu->kvm_fd = cpu->kvm_fd;
  >>   +    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,
  node); }
  >>   +
  >>   +int kvm_create_vcpu(CPUState *cpu)
  >>   +{
  >>   +    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
  >>   +    KVMState *s = kvm_state;
  >>   +    int kvm_fd;
  >>   +
  >>   +    trace_kvm_create_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
  >>   +
  >>   +    /* check if the KVM vCPU already exist but is parked */
  >>   +    kvm_fd = kvm_get_vcpu(s, vcpu_id);
  >>   +    if (kvm_fd < 0) {
  >>   +        /* vCPU not parked: create a new KVM vCPU */
  >>   +        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
  >>   +        if (kvm_fd < 0) {
  >>   +            error_report("KVM_CREATE_VCPU IOCTL failed for vCPU %lu",
  >>   vcpu_id);
  >>   +            return kvm_fd;
  >>   +        }
  >>   +    }
  >>   +
  >>   +    cpu->kvm_fd = kvm_fd;
  >>   +    cpu->kvm_state = s;
  >>   +    cpu->vcpu_dirty = true;
  >>   +    cpu->dirty_pages = 0;
  >>   +    cpu->throttle_us_per_full = 0;
  >>   +
  >>   +    return 0;
  >>   +}
  >>   +
  >>    static int do_kvm_destroy_vcpu(CPUState *cpu)  {
  >>        KVMState *s = kvm_state;
  >>        long mmap_size;
  >>   -    struct KVMParkedVcpu *vcpu = NULL;
  >>        int ret = 0;
  >>
  >>   -    trace_kvm_destroy_vcpu();
  >>   +    trace_kvm_destroy_vcpu(cpu->cpu_index,
  kvm_arch_vcpu_id(cpu));
  >>
  >>        ret = kvm_arch_destroy_vcpu(cpu);
  >>        if (ret < 0) {
  >>   @@ -373,10 +413,7 @@ static int do_kvm_destroy_vcpu(CPUState
  *cpu)
  >>            }
  >>        }
  >>
  >>   -    vcpu = g_malloc0(sizeof(*vcpu));
  >>   -    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  >>   -    vcpu->kvm_fd = cpu->kvm_fd;
  >>   -    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,
  node);
  >>   +    kvm_park_vcpu(cpu);
  >>    err:
  >>        return ret;
  >>    }
  >>   @@ -397,6 +434,8 @@ static int kvm_get_vcpu(KVMState *s, unsigned
  long
  >>   vcpu_id)
  >>            if (cpu->vcpu_id == vcpu_id) {
  >>                int kvm_fd;
  >>
  >>   +            trace_kvm_get_vcpu(vcpu_id);
  >>   +
  >>                QLIST_REMOVE(cpu, node);
  >>                kvm_fd = cpu->kvm_fd;
  >>                g_free(cpu);
  >>   @@ -404,7 +443,7 @@ static int kvm_get_vcpu(KVMState *s, unsigned
  long
  >>   vcpu_id)
  >>            }
  >>        }
  >>
  >>   -    return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
  >>   +    return -ENOENT;
  >>    }
  >>
  >>    int kvm_init_vcpu(CPUState *cpu, Error **errp) @@ -415,19 +454,14
  @@
  >>   int kvm_init_vcpu(CPUState *cpu, Error **errp)
  >>
  >>        trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
  >>
  >>   -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
  >>   +    ret = kvm_create_vcpu(cpu);
  >>        if (ret < 0) {
  >>   -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu failed
  >>   (%lu)",
  >>   +        error_setg_errno(errp, -ret,
  >>   +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
  >>                             kvm_arch_vcpu_id(cpu));
  >>            goto err;
  >>        }
  >>
  >>   -    cpu->kvm_fd = ret;
  >>   -    cpu->kvm_state = s;
  >>   -    cpu->vcpu_dirty = true;
  >>   -    cpu->dirty_pages = 0;
  >>   -    cpu->throttle_us_per_full = 0;
  >>   -
  >>        mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
  >>        if (mmap_size < 0) {
  >>            ret = mmap_size;
  >>   diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events index
  >>   681ccb667d..75c1724e78 100644
  >>   --- a/accel/kvm/trace-events
  >>   +++ b/accel/kvm/trace-events
  >>   @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg) "dev fd
  %d,
  >>   type 0x%x, arg %p"
  >>    kvm_failed_reg_get(uint64_t id, const char *msg) "Warning: Unable to
  >>   retrieve ONEREG %" PRIu64 " from KVM: %s"
  >>    kvm_failed_reg_set(uint64_t id, const char *msg) "Warning: Unable to
  set
  >>   ONEREG %" PRIu64 " to KVM: %s"
  >>    kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d
  id:
  >>   %lu"
  >>   +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id) "index:
  %d
  >>   id: %lu"
  >>   +kvm_get_vcpu(unsigned long arch_cpu_id) "id: %lu"
  >>   +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "index:
  %d
  >>   id: %lu"
  >>   +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d
  id:
  >>   %lu"
  >>    kvm_irqchip_commit_routes(void) ""
  >>    kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s
  >>   vector %d virq %d"
  >>    kvm_irqchip_update_msi_route(int virq) "Updating MSI route
  virq=%d"
  >>   @@ -25,7 +29,6 @@ kvm_dirty_ring_reaper(const char *s) "%s"
  >>    kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64"
  pages
  >>   (took %"PRIi64" us)"
  >>    kvm_dirty_ring_reaper_kick(const char *reason) "%s"
  >>    kvm_dirty_ring_flush(int finished) "%d"
  >>   -kvm_destroy_vcpu(void) ""
  >>    kvm_failed_get_vcpu_mmap_size(void) ""
  >>    kvm_cpu_exec(void) ""
  >>    kvm_interrupt_exit_request(void) ""
  >>   --
  >>   2.39.3
  >
#6 lore raw

RE: [PATCH v2 1/4] accel/kvm: Extract common KVM vCPU {creation, parking} code

Quoted 25 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 2:07 PM

Hi Salil,

Quoted 18 lines · Salil Mehta

Hi Harsh,

Quoted 11 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 11:15 AM

Hi Salil,

Thanks for your email.
Your patch 1/8 is included here based on review comments on my previous patch from one of the maintainers in the community and therefore I had kept you in CC to be aware of the desire of having this independent patch to get merged earlier even if your other patches in the series may go through further reviews.

I really don’t know which discussion are you pointing at? Please understand you are fixing a bug and we are pushing a feature which has got large series. It will break the patch-set which is about t be merged.

Quoted 6 lines
Quoted 3 lines

There will be significant overhead of testing on us for the work we have been carrying forward for large time. This will be disruptive. Please dont!

I was referring to the review discussion on my prev patch here:
https://lore.kernel.org/qemu-devel/[email protected]/

Sure, I'm, not sure what this means.

Quoted 2 lines

Although your patch was included with this series only to facilitate review of the additional patches depending on just one of your patch.

Generally you rebase your patch-set over the other and clearly state on the cover letter that this patch-set is dependent upon such and such patch-set. Just imagine if everyone starts to unilaterally pick up patches from each other's patch-set it will create a chaos not only for the feature owners but also for the maintainers.

Quoted 4 lines

I am not sure what is appearing disruptive here. It is a common practive in the community that maintainer(s) can pick individual patches from the series if it has been vetted by siginificant number of reviewers.

Don’t you think this patch-set is asking for acceptance for a patch already part of another patch-set which is about to be accepted and is a bigger feature? Will it cause maintenance overhead at the last moment? Yes, of course!

Quoted 3 lines

However, in this case, since you have mentioned to post next version soon, you need not worry about it as that would be the preferred version for both of the series.

Yes, but please understand we are working for the benefit of overall community. Please cooperate here.

Quoted 320 lines
Quoted 4 lines
Quoted 4 lines

I am hoping to see your v9 soon and thereafter maintainer(s) may

choose to

Quoted 6 lines
Quoted 1 line

pick the latest independent patch if needs to be merged earlier.

I don’t think you are understanding what problem it is causing. For your small bug fix you are causing significant delays at our end.

I hope I clarfied above that including your patch here doesnt delay anything. Hoping to see your v9 soon!

Thanks
Harsh

Quoted 16 lines

Thanks
Salil.

Quoted 13 lines

Thanks for your work and let's be hopeful it gets merged soon.

regards,
Harsh

Quoted 6 lines · Salil Mehta

Hi Harsh,

Thanks for your interest in the patch-set but taking away patches like this from other series without any discussion can disrupt others work and its acceptance on time. This is because we will have to put lot of effort in rebasing bigger series and then testing overhead comes

along

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

with it.

The patch-set (from where this patch has been taken) is part of even bigger series and there have been many people and companies toiling

to

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

fix the bugs collectively in that series and for years.

I'm about float the V9 version of the Arch agnostic series which this patch is part of and you can rebase your patch-set from there. I'm hopeful that it will get accepted in this cycle.

Many thanks
Salil.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 6:32 AM

From: Salil Mehta <[email protected]>

KVM vCPU creation is done once during the vCPU realization when

Qemu

Quoted 1 line
Quoted 1 line

vCPU thread is spawned. This is common to all the architectures as

of

Quoted 3 lines
Quoted 3 lines

now.

Quoted 2 lines
Quoted 2 lines

Hot-unplug of vCPU results in destruction of the vCPU object in

QOM

Quoted 2 lines
Quoted 2 lines

but

Quoted 1 line
Quoted 1 line

the corresponding KVM vCPU object in the Host KVM is not

destroyed

Quoted 2 lines
Quoted 2 lines

as

Quoted 1 line
Quoted 1 line

KVM doesn't support vCPU removal. Therefore, its representative

KVM

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

vCPU object/context in Qemu is parked.

Refactor architecture common logic so that some APIs could be

reused

Quoted 11 lines
Quoted 11 lines

by

Quoted 1 line
Quoted 1 line

vCPU Hotplug code of some architectures likes ARM, Loongson etc.

Update

Quoted 1 line
Quoted 1 line

new/old APIs with trace events instead of DPRINTF. No functional

change is

Quoted 6 lines
Quoted 6 lines

intended here.

<[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
[harshpb: fixed rebase failures in include/sysemu/kvm.h]

---

   include/sysemu/kvm.h   | 15 ++++++++++
   accel/kvm/kvm-all.c    | 64

++++++++++++++++++++++++++++++++---

Quoted 13 lines
Quoted 13 lines

-----

Quoted 9 lines
Quoted 9 lines

--

   accel/kvm/trace-events |  5 +++-
   3 files changed, 68 insertions(+), 16 deletions(-)

diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index eaf801bc93..fa3ec74442 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -434,6 +434,21 @@ void kvm_set_sigmask_len(KVMState *s,

unsigned

Quoted 2 lines
Quoted 2 lines

int sigmask_len);

Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

int kvm_physical_memory_addr_from_host(KVMState *s, void
*ram_addr,

                                          hwaddr *phys_addr);

+/**
+ * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM

Quoted 11 lines

vCPU

Quoted 10 lines
Quoted 10 lines

+ * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_vcpu(CPUState *cpu);
+
+/**
+ * kvm_park_vcpu - Park QEMU KVM vCPU context
+ * @cpu: QOM CPUState object for which QEMU KVM vCPU

context

Quoted 25 lines
Quoted 25 lines
  has to
  >>   be parked.
  >>   + *
  >>   + * @returns: none
  >>   + */
  >>   +void kvm_park_vcpu(CPUState *cpu);
  >>
  >>    #endif /* COMPILING_PER_TARGET */
  >>
  >>   diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index
  >>   d7281b93f3..30d42847de 100644
  >>   --- a/accel/kvm/kvm-all.c
  >>   +++ b/accel/kvm/kvm-all.c
  >>   @@ -128,6 +128,7 @@ static QemuMutex kml_slots_lock;  #define
  >>   kvm_slots_unlock()  qemu_mutex_unlock(&kml_slots_lock)
  >>
  >>    static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
  >>   +static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id);
  >>
  >>    static inline void kvm_resample_fd_remove(int gsi)  { @@ -340,14
  +341,53
  >>   @@ err:
  >>        return ret;
  >>    }
  >>
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

+void kvm_park_vcpu(CPUState *cpu)
+{
+ struct KVMParkedVcpu *vcpu;
+
+ trace_kvm_park_vcpu(cpu->cpu_index,

kvm_arch_vcpu_id(cpu));

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

+

  +    vcpu = g_malloc0(sizeof(*vcpu));
  +    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  +    vcpu->kvm_fd = cpu->kvm_fd;
  +    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,

node); }

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+
+int kvm_create_vcpu(CPUState *cpu)
+{

  +    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
  +    KVMState *s = kvm_state;
  +    int kvm_fd;

+
+ trace_kvm_create_vcpu(cpu->cpu_index,

kvm_arch_vcpu_id(cpu));

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+

  +    /* check if the KVM vCPU already exist but is parked */
  +    kvm_fd = kvm_get_vcpu(s, vcpu_id);
  +    if (kvm_fd < 0) {
  +        /* vCPU not parked: create a new KVM vCPU */
  +        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
  +        if (kvm_fd < 0) {

+ error_report("KVM_CREATE_VCPU IOCTL failed for vCPU

%lu",

Quoted 31 lines
Quoted 31 lines
Quoted 20 lines
Quoted 20 lines

vcpu_id);
+ return kvm_fd;
+ }
+ }
+

  +    cpu->kvm_fd = kvm_fd;
  +    cpu->kvm_state = s;
  +    cpu->vcpu_dirty = true;
  +    cpu->dirty_pages = 0;
  +    cpu->throttle_us_per_full = 0;
  +
  +    return 0;
  +}
  +
   static int do_kvm_destroy_vcpu(CPUState *cpu)  {
       KVMState *s = kvm_state;
       long mmap_size;
  -    struct KVMParkedVcpu *vcpu = NULL;
       int ret = 0;
Quoted 2 lines
Quoted 2 lines

- trace_kvm_destroy_vcpu();
+ trace_kvm_destroy_vcpu(cpu->cpu_index,

kvm_arch_vcpu_id(cpu));

Quoted 4 lines
Quoted 4 lines
       ret = kvm_arch_destroy_vcpu(cpu);
       if (ret < 0) {

@@ -373,10 +413,7 @@ static int do_kvm_destroy_vcpu(CPUState

*cpu)

Quoted 3 lines
Quoted 3 lines
           }
       }
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
  -    vcpu = g_malloc0(sizeof(*vcpu));
  -    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  -    vcpu->kvm_fd = cpu->kvm_fd;
  -    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,

node);

Quoted 5 lines
Quoted 5 lines
  +    kvm_park_vcpu(cpu);
   err:
       return ret;
   }

@@ -397,6 +434,8 @@ static int kvm_get_vcpu(KVMState *s,

unsigned

Quoted 11 lines
Quoted 11 lines
  long
  >>   vcpu_id)
  >>            if (cpu->vcpu_id == vcpu_id) {
  >>                int kvm_fd;
  >>
  >>   +            trace_kvm_get_vcpu(vcpu_id);
  >>   +
  >>                QLIST_REMOVE(cpu, node);
  >>                kvm_fd = cpu->kvm_fd;
  >>                g_free(cpu);
  >>   @@ -404,7 +443,7 @@ static int kvm_get_vcpu(KVMState *s,

unsigned

Quoted 10 lines
Quoted 10 lines
  long
  >>   vcpu_id)
  >>            }
  >>        }
  >>
  >>   -    return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
  >>   +    return -ENOENT;
  >>    }
  >>
  >>    int kvm_init_vcpu(CPUState *cpu, Error **errp) @@ -415,19

+454,14

Quoted 9 lines
Quoted 9 lines
  @@
  >>   int kvm_init_vcpu(CPUState *cpu, Error **errp)
  >>
  >>        trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
  >>
  >>   -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
  >>   +    ret = kvm_create_vcpu(cpu);
  >>        if (ret < 0) {
  >>   -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu

failed

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

(%lu)",
+ error_setg_errno(errp, -ret,
+ "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",

                            kvm_arch_vcpu_id(cpu));
           goto err;
       }
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
  -    cpu->kvm_fd = ret;
  -    cpu->kvm_state = s;
  -    cpu->vcpu_dirty = true;
  -    cpu->dirty_pages = 0;
  -    cpu->throttle_us_per_full = 0;

-

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
       mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
       if (mmap_size < 0) {
           ret = mmap_size;

diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events index 681ccb667d..75c1724e78 100644
--- a/accel/kvm/trace-events
+++ b/accel/kvm/trace-events
@@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg)

"dev fd

Quoted 3 lines
Quoted 3 lines

%d,

Quoted 2 lines
Quoted 2 lines

type 0x%x, arg %p"
kvm_failed_reg_get(uint64_t id, const char *msg) "Warning:

Unable to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

retrieve ONEREG %" PRIu64 " from KVM: %s"
kvm_failed_reg_set(uint64_t id, const char *msg) "Warning:

Unable to

Quoted 3 lines
Quoted 3 lines

set

Quoted 2 lines
Quoted 2 lines

ONEREG %" PRIu64 " to KVM: %s"
kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index:

%d

Quoted 3 lines
Quoted 3 lines

id:

Quoted 2 lines
Quoted 2 lines

%lu"
+kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id)

"index:

Quoted 4 lines
Quoted 4 lines

%d

Quoted 3 lines
Quoted 3 lines

id: %lu"
+kvm_get_vcpu(unsigned long arch_cpu_id) "id: %lu" +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id)

"index:

Quoted 3 lines
Quoted 3 lines

%d

Quoted 2 lines
Quoted 2 lines

id: %lu"
+kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id)

"index: %d

Quoted 4 lines
Quoted 4 lines

id:

Quoted 3 lines
Quoted 3 lines

%lu"
kvm_irqchip_commit_routes(void) "" kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev

%s

Quoted 16 lines
Quoted 16 lines
Quoted 2 lines
Quoted 2 lines

vector %d virq %d"
kvm_irqchip_update_msi_route(int virq) "Updating MSI route

virq=%d"

Quoted 2 lines
Quoted 2 lines

@@ -25,7 +29,6 @@ kvm_dirty_ring_reaper(const char *s) "%s"
kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64"

pages

Quoted 10 lines
Quoted 9 lines

(took %"PRIi64" us)"
kvm_dirty_ring_reaper_kick(const char *reason) "%s" kvm_dirty_ring_flush(int finished) "%d"
-kvm_destroy_vcpu(void) ""
kvm_failed_get_vcpu_mmap_size(void) "" kvm_cpu_exec(void) "" kvm_interrupt_exit_request(void) ""
--
2.39.3

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw

PATCH v2 1/4 accel/kvm: Extract common KVM vCPU {creation, parking} code

[+] Adding this email address to the conversation.

(sorry for the noise)

Quoted 18 lines

From: Salil Mehta
Sent: Thursday, May 16, 2024 2:36 PM

Quoted 15 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 2:07 PM

Hi Salil,

Quoted 9 lines · Salil Mehta

Hi Harsh,

Quoted 7 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 11:15 AM

Hi Salil,

Thanks for your email.
Your patch 1/8 is included here based on review comments on my

previous

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

patch from one of the maintainers in the community and therefore

I had

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

kept you in CC to be aware of the desire of having this

independent patch to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

get merged earlier even if your other patches in the series may

go through

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

further reviews.

I really don’t know which discussion you are pointing at? Please understand you are fixing a bug and we are pushing a feature which

has got large series.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

It will break the patch-set which is about to be merged.

There will be significant overhead of testing on us for the work we have been carrying forward for large time. This will be disruptive.

Please dont!

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

I was referring to the review discussion on my prev patch here:

https://lore.kernel.org/qemu-devel/[email protected]/

Quoted 10 lines

Ok, I'm, not sure what this means.

Quoted 2 lines

Although your patch was included with this series only to facilitate
review of the additional patches depending on just one of your patch.

Generally you rebase your patch-set over the other and clearly state on

the

Quoted 1 line

cover letter that this patch-set is dependent upon such and such

patch-set.

Quoted 2 lines

Just imagine if everyone starts to unilaterally pick up patches from each other's patch-set it will create a chaos not only for the feature owners

but

Quoted 7 lines

also for the maintainers.

Quoted 4 lines

I am not sure what is appearing disruptive here. It is a common
practive in the community that maintainer(s) can pick individual patches from the series if it has been vetted by siginificant number

of reviewers.

Quoted 3 lines

Don’t you think this patch-set is asking for acceptance for a patch

already

Quoted 2 lines

part of another patch-set which is about to be accepted and is a bigger feature? Will it cause maintenance overhead at the last moment? Yes, of

course!

Quoted 3 lines
Quoted 3 lines

However, in this case, since you have mentioned to post next version

soon,

Quoted 1 line
Quoted 1 line

you need not worry about it as that would be the preferred version

for both

Quoted 10 lines
Quoted 1 line

of the series.

Yes, but please understand we are working for the benefit of overall community. Please cooperate here.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I am hoping to see your v9 soon and thereafter maintainer(s) may

choose to

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

pick the latest independent patch if needs to be merged earlier.

I don’t think you are understanding what problem it is causing. For your small bug fix you are causing significant delays at our end.

I hope I clarfied above that including your patch here doesnt delay

anything.

Quoted 10 lines
Quoted 10 lines

Hoping to see your v9 soon!

Thanks
Harsh

Quoted 6 lines

Thanks
Salil.

Quoted 3 lines

Thanks for your work and let's be hopeful it gets merged soon.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

regards,
Harsh

Quoted 3 lines · Salil Mehta

Hi Harsh,

Thanks for your interest in the patch-set but taking away

patches like

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

this from other series without any discussion can disrupt

others work

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

and its acceptance on time. This is because we will have to

put lot of

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

effort in rebasing bigger series and then testing overhead

comes along

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

with it.

The patch-set (from where this patch has been taken) is part

of even

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

bigger series and there have been many people and companies

toiling to

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

fix the bugs collectively in that series and for years.

I'm about float the V9 version of the Arch agnostic series

which this

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

patch is part of and you can rebase your patch-set from there.

I'm

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

hopeful that it will get accepted in this cycle.

Many thanks
Salil.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 6:32 AM

From: Salil Mehta <[email protected]>

KVM vCPU creation is done once during the vCPU realization

when Qemu

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

vCPU thread is spawned. This is common to all the

architectures as of now.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Hot-unplug of vCPU results in destruction of the vCPU

object in QOM but

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the corresponding KVM vCPU object in the Host KVM is not

destroyed as

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

KVM doesn't support vCPU removal. Therefore, its

representative KVM

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

vCPU object/context in Qemu is parked.

Refactor architecture common logic so that some APIs could

be reused by

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

vCPU Hotplug code of some architectures likes ARM, Loongson

etc. Update

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

new/old APIs with trace events instead of DPRINTF. No

functional change is

Quoted 20 lines
Quoted 20 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines

intended here.

Quoted 3 lines

---

   include/sysemu/kvm.h   | 15 ++++++++++
   accel/kvm/kvm-all.c    | 64

++++++++++++++++++++++++++++++++---

Quoted 5 lines
Quoted 5 lines

-----

Quoted 4 lines
Quoted 4 lines

--

   accel/kvm/trace-events |  5 +++-
   3 files changed, 68 insertions(+), 16 deletions(-)
Quoted 14 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
  diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h

index

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines

eaf801bc93..fa3ec74442 100644
--- a/include/sysemu/kvm.h
+++ b/include/sysemu/kvm.h
@@ -434,6 +434,21 @@ void kvm_set_sigmask_len(KVMState *s,

unsigned

Quoted 7 lines
Quoted 7 lines

int sigmask_len);

int kvm_physical_memory_addr_from_host(KVMState *s, void
*ram_addr,

                                          hwaddr *phys_addr);

+/**
+ * kvm_create_vcpu - Gets a parked KVM vCPU or creates a

KVM

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

vCPU

Quoted 10 lines
Quoted 10 lines

+ * @cpu: QOM CPUState object for which KVM vCPU has to be fetched/created.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_vcpu(CPUState *cpu);
+
+/**
+ * kvm_park_vcpu - Park QEMU KVM vCPU context
+ * @cpu: QOM CPUState object for which QEMU KVM vCPU

context has to

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

be parked.
+ *
+ * @returns: none
+ */
+void kvm_park_vcpu(CPUState *cpu);

Quoted 12 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
   #endif /* COMPILING_PER_TARGET */
  diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index
  d7281b93f3..30d42847de 100644
  --- a/accel/kvm/kvm-all.c
  +++ b/accel/kvm/kvm-all.c
  @@ -128,6 +128,7 @@ static QemuMutex kml_slots_lock;

#define

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

kvm_slots_unlock() qemu_mutex_unlock(&kml_slots_lock)

   static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
  +static int kvm_get_vcpu(KVMState *s, unsigned long

vcpu_id);

Quoted 13 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

static inline void kvm_resample_fd_remove(int gsi) { @@ -

340,14

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
  +341,53
  >>   @@ err:
  >>        return ret;
  >>    }
  >>
  >>   +void kvm_park_vcpu(CPUState *cpu)
  >>   +{
  >>   +    struct KVMParkedVcpu *vcpu;
  >>   +
  >>   +    trace_kvm_park_vcpu(cpu->cpu_index,

kvm_arch_vcpu_id(cpu));

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

+

  +    vcpu = g_malloc0(sizeof(*vcpu));
  +    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  +    vcpu->kvm_fd = cpu->kvm_fd;
  +    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,

node); }

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+
+int kvm_create_vcpu(CPUState *cpu)
+{

  +    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
  +    KVMState *s = kvm_state;
  +    int kvm_fd;

+
+ trace_kvm_create_vcpu(cpu->cpu_index,

kvm_arch_vcpu_id(cpu));

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+

  +    /* check if the KVM vCPU already exist but is parked */
  +    kvm_fd = kvm_get_vcpu(s, vcpu_id);
  +    if (kvm_fd < 0) {
  +        /* vCPU not parked: create a new KVM vCPU */
  +        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
  +        if (kvm_fd < 0) {

+ error_report("KVM_CREATE_VCPU IOCTL failed for

vCPU %lu", vcpu_id);

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

+ return kvm_fd;
+ }
+ }
+

  +    cpu->kvm_fd = kvm_fd;
  +    cpu->kvm_state = s;
  +    cpu->vcpu_dirty = true;
  +    cpu->dirty_pages = 0;
  +    cpu->throttle_us_per_full = 0;
  +
  +    return 0;
  +}
  +
   static int do_kvm_destroy_vcpu(CPUState *cpu)  {
       KVMState *s = kvm_state;
       long mmap_size;
  -    struct KVMParkedVcpu *vcpu = NULL;
       int ret = 0;
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

- trace_kvm_destroy_vcpu();
+ trace_kvm_destroy_vcpu(cpu->cpu_index,

kvm_arch_vcpu_id(cpu));

Quoted 4 lines
Quoted 4 lines
       ret = kvm_arch_destroy_vcpu(cpu);
       if (ret < 0) {

@@ -373,10 +413,7 @@ static int

do_kvm_destroy_vcpu(CPUState *cpu)

Quoted 36 lines
Quoted 19 lines
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Quoted 7 lines
           }
       }
  -    vcpu = g_malloc0(sizeof(*vcpu));
  -    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  -    vcpu->kvm_fd = cpu->kvm_fd;
  -    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,

node);

Quoted 5 lines
Quoted 5 lines
  +    kvm_park_vcpu(cpu);
   err:
       return ret;
   }

@@ -397,6 +434,8 @@ static int kvm_get_vcpu(KVMState *s,

unsigned

Quoted 5 lines
Quoted 5 lines
  long
  >>   vcpu_id)
  >>            if (cpu->vcpu_id == vcpu_id) {
  >>                int kvm_fd;
  >>
Quoted 13 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
  +            trace_kvm_get_vcpu(vcpu_id);
  +
               QLIST_REMOVE(cpu, node);
               kvm_fd = cpu->kvm_fd;
               g_free(cpu);
  @@ -404,7 +443,7 @@ static int kvm_get_vcpu(KVMState *s,

unsigned

Quoted 6 lines
Quoted 6 lines

long

Quoted 5 lines
Quoted 5 lines
  vcpu_id)
           }
       }

- return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void

*)vcpu_id);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
  +    return -ENOENT;
   }
Quoted 12 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
   int kvm_init_vcpu(CPUState *cpu, Error **errp) @@ -415,19

+454,14

Quoted 4 lines
Quoted 4 lines

@@

Quoted 3 lines
Quoted 3 lines

int kvm_init_vcpu(CPUState *cpu, Error **errp)

       trace_kvm_init_vcpu(cpu->cpu_index,

kvm_arch_vcpu_id(cpu));

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

- ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));

  +    ret = kvm_create_vcpu(cpu);
       if (ret < 0) {

- error_setg_errno(errp, -ret, "kvm_init_vcpu:

kvm_get_vcpu

Quoted 4 lines
Quoted 4 lines

failed

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

(%lu)",
+ error_setg_errno(errp, -ret,
+ "kvm_init_vcpu: kvm_create_vcpu

failed (%lu)",

Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
                            kvm_arch_vcpu_id(cpu));
           goto err;
       }
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
  -    cpu->kvm_fd = ret;
  -    cpu->kvm_state = s;
  -    cpu->vcpu_dirty = true;
  -    cpu->dirty_pages = 0;
  -    cpu->throttle_us_per_full = 0;
  -
       mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
       if (mmap_size < 0) {
           ret = mmap_size;

diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events

Quoted 5 lines

index

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

681ccb667d..75c1724e78 100644
--- a/accel/kvm/trace-events
+++ b/accel/kvm/trace-events
@@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void

*arg)

Quoted 11 lines
Quoted 11 lines

"dev fd

Quoted 3 lines
Quoted 3 lines

%d,

Quoted 2 lines
Quoted 2 lines

type 0x%x, arg %p"
kvm_failed_reg_get(uint64_t id, const char *msg) "Warning:

Unable to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

retrieve ONEREG %" PRIu64 " from KVM: %s"
kvm_failed_reg_set(uint64_t id, const char *msg) "Warning:

Unable to

Quoted 3 lines
Quoted 3 lines

set

Quoted 2 lines
Quoted 2 lines

ONEREG %" PRIu64 " to KVM: %s"
kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id)

Quoted 19 lines

"index:

Quoted 18 lines

%d

Quoted 3 lines
Quoted 3 lines

id:

Quoted 2 lines
Quoted 2 lines

%lu"
+kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id)

"index:

Quoted 4 lines
Quoted 4 lines

%d

Quoted 3 lines
Quoted 3 lines

id: %lu"
+kvm_get_vcpu(unsigned long arch_cpu_id) "id: %lu" +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id)

"index:

Quoted 3 lines
Quoted 3 lines

%d

Quoted 2 lines
Quoted 2 lines

id: %lu"
+kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id)

"index: %d

Quoted 4 lines
Quoted 4 lines

id:

Quoted 3 lines
Quoted 3 lines

%lu"
kvm_irqchip_commit_routes(void) "" kvm_irqchip_add_msi_route(char *name, int vector, int

virq) "dev

Quoted 6 lines
Quoted 6 lines

%s

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

vector %d virq %d"
kvm_irqchip_update_msi_route(int virq) "Updating MSI route

virq=%d"

Quoted 2 lines
Quoted 2 lines

@@ -25,7 +29,6 @@ kvm_dirty_ring_reaper(const char *s) "%s"
kvm_dirty_ring_reap(uint64_t count, int64_t t)

"reaped %"PRIu64"

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

pages

Quoted 10 lines
Quoted 9 lines

(took %"PRIi64" us)"
kvm_dirty_ring_reaper_kick(const char *reason) "%s" kvm_dirty_ring_flush(int finished) "%d"
-kvm_destroy_vcpu(void) ""
kvm_failed_get_vcpu_mmap_size(void) "" kvm_cpu_exec(void) "" kvm_interrupt_exit_request(void) ""
--
2.39.3

#9 lore raw

Re: [PATCH v2 1/4] accel/kvm: Extract common KVM vCPU {creation, parking} code

In reply to Salil Mehta via

Hi Salil,

Quoted 29 lines · Salil Mehta
Quoted 29 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 2:07 PM

Hi Salil,

Quoted 21 lines · Salil Mehta

Hi Harsh,

Quoted 11 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 11:15 AM

Hi Salil,

Thanks for your email.
Your patch 1/8 is included here based on review comments on my previous patch from one of the maintainers in the community and therefore I had kept you in CC to be aware of the desire of having this independent patch to get merged earlier even if your other patches in the series may go through further reviews.

I really don’t know which discussion are you pointing at? Please understand you are fixing a bug and we are pushing a feature which has got large series. It will break the patch-set which is about t be merged.

There will be significant overhead of testing on us for the work we have been carrying forward for large time. This will be disruptive. Please dont!

Quoted 6 lines
Quoted 2 lines

I was referring to the review discussion on my prev patch here:
https://lore.kernel.org/qemu-devel/[email protected]/

Sure, I'm, not sure what this means.

No worries. If you had followed the conversation on the review link I shared, I had made it clear that we are expecting a patch update from you and it is included here just to facilitate review of additional patches on the top.

Quoted 10 lines
Quoted 3 lines

Although your patch was included with this series only to facilitate review of the additional patches depending on just one of your patch.

Generally you rebase your patch-set over the other and clearly state on the cover letter that this patch-set is dependent upon such and such patch-set. Just imagine if everyone starts to unilaterally pick up patches from each other's patch-set it will create a chaos not only for the feature owners but also for the maintainers.

Please go through the review discussion on the link I shared above. It was included on the suggestion of one of the maintainers. However, if you are going to send v9 soon, everyone would be happy to wait.

Quoted 10 lines
Quoted 5 lines

I am not sure what is appearing disruptive here. It is a common practive in the community that maintainer(s) can pick individual patches from the series if it has been vetted by siginificant number of reviewers.

Don’t you think this patch-set is asking for acceptance for a patch already part of another patch-set which is about to be accepted and is a bigger feature? Will it cause maintenance overhead at the last moment? Yes, of course!

No, I dont think so.

Quoted 10 lines
Quoted 5 lines

However, in this case, since you have mentioned to post next version soon, you need not worry about it as that would be the preferred version for both of the series.

Yes, but please understand we are working for the benefit of overall community. Please cooperate here.

Hope I cleared your confusion. We are waiting to see your v9 soon.

Quoted 320 lines
Quoted 320 lines
Quoted 4 lines
Quoted 4 lines

I am hoping to see your v9 soon and thereafter maintainer(s) may

choose to

Quoted 6 lines
Quoted 1 line

pick the latest independent patch if needs to be merged earlier.

I don’t think you are understanding what problem it is causing. For your small bug fix you are causing significant delays at our end.

I hope I clarfied above that including your patch here doesnt delay anything. Hoping to see your v9 soon!

  Thanks
  Harsh
  >
  > Thanks
  > Salil.
  >>
  >>   Thanks for your work and let's be hopeful it gets merged soon.
  >>
  >>   regards,
  >>   Harsh
  >>
  >>   On 5/16/24 14:00, Salil Mehta wrote:
  >>   > Hi Harsh,
  >>   >
  >>   > Thanks for your interest in the patch-set but taking away patches like
  >>   > this from other series without any discussion can disrupt others work
  >>   > and its acceptance on time. This is because we will have to put lot of
  >>   > effort in rebasing bigger series and then testing overhead comes
  along
  >>   > with it.
  >>   >
  >>   > The patch-set (from where this  patch has been taken) is part of even
  >>   > bigger series and there have been many people and companies toiling
  to
  >>   > fix the bugs collectively in that series and for years.
  >>   >
  >>   > I'm about float the V9 version of the Arch agnostic series which this
  >>   > patch is part of and you can rebase your patch-set from there. I'm
  >>   > hopeful that it will get accepted in this cycle.
  >>   >
  >>   >
  >>   > Many thanks
  >>   > Salil.
  >>   >
  >>   >>   From: Harsh Prateek Bora <[email protected]>
  >>   >>   Sent: Thursday, May 16, 2024 6:32 AM
  >>   >>
  >>   >>   From: Salil Mehta <[email protected]>
  >>   >>
  >>   >>   KVM vCPU creation is done once during the vCPU realization when
  >>   Qemu
  >>   >>   vCPU thread is spawned. This is common to all the architectures as
  of
  >>   now.
  >>   >>
  >>   >>   Hot-unplug of vCPU results in destruction of the vCPU object in
  QOM
  >>   but
  >>   >>   the corresponding KVM vCPU object in the Host KVM is not
  destroyed
  >>   as
  >>   >>   KVM doesn't support vCPU removal. Therefore, its representative
  KVM
  >>   >>   vCPU object/context in Qemu is parked.
  >>   >>
  >>   >>   Refactor architecture common logic so that some APIs could be
  reused
  >>   by
  >>   >>   vCPU Hotplug code of some architectures likes ARM, Loongson etc.
  >>   Update
  >>   >>   new/old APIs with trace events instead of DPRINTF. No functional
  >>   change is
  >>   >>   intended here.
  >>   >>
  >>   >>   Signed-off-by: Salil Mehta <[email protected]>
  >>   >>   Reviewed-by: Gavin Shan <[email protected]>
  >>   >>   Tested-by: Vishnu Pajjuri <[email protected]>
  >>   >>   Reviewed-by: Jonathan Cameron
  <[email protected]>
  >>   >>   Tested-by: Xianglai Li <[email protected]>
  >>   >>   Tested-by: Miguel Luis <[email protected]>
  >>   >>   Reviewed-by: Shaoqin Huang <[email protected]>
  >>   >>   [harshpb: fixed rebase failures in include/sysemu/kvm.h]
  >>   >>   Signed-off-by: Harsh Prateek Bora <[email protected]>
  >>   >>   ---
  >>   >>    include/sysemu/kvm.h   | 15 ++++++++++
  >>   >>    accel/kvm/kvm-all.c    | 64
  ++++++++++++++++++++++++++++++++---
  >>   -----
  >>   >>   --
  >>   >>    accel/kvm/trace-events |  5 +++-
  >>   >>    3 files changed, 68 insertions(+), 16 deletions(-)
  >>   >>
  >>   >>   diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index
  >>   >>   eaf801bc93..fa3ec74442 100644
  >>   >>   --- a/include/sysemu/kvm.h
  >>   >>   +++ b/include/sysemu/kvm.h
  >>   >>   @@ -434,6 +434,21 @@ void kvm_set_sigmask_len(KVMState *s,
  >>   unsigned
  >>   >>   int sigmask_len);
  >>   >>
  >>   >>    int kvm_physical_memory_addr_from_host(KVMState *s, void
  >>   >>   *ram_addr,
  >>   >>                                           hwaddr *phys_addr);
  >>   >>   +/**
  >>   >>   + * kvm_create_vcpu - Gets a parked KVM vCPU or creates a KVM
  >>   vCPU
  >>   >>   + * @cpu: QOM CPUState object for which KVM vCPU has to be
  >>   >>   fetched/created.
  >>   >>   + *
  >>   >>   + * @returns: 0 when success, errno (<0) when failed.
  >>   >>   + */
  >>   >>   +int kvm_create_vcpu(CPUState *cpu);
  >>   >>   +
  >>   >>   +/**
  >>   >>   + * kvm_park_vcpu - Park QEMU KVM vCPU context
  >>   >>   + * @cpu: QOM CPUState object for which QEMU KVM vCPU
  context
  >>   has to
  >>   >>   be parked.
  >>   >>   + *
  >>   >>   + * @returns: none
  >>   >>   + */
  >>   >>   +void kvm_park_vcpu(CPUState *cpu);
  >>   >>
  >>   >>    #endif /* COMPILING_PER_TARGET */
  >>   >>
  >>   >>   diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index
  >>   >>   d7281b93f3..30d42847de 100644
  >>   >>   --- a/accel/kvm/kvm-all.c
  >>   >>   +++ b/accel/kvm/kvm-all.c
  >>   >>   @@ -128,6 +128,7 @@ static QemuMutex kml_slots_lock;  #define
  >>   >>   kvm_slots_unlock()  qemu_mutex_unlock(&kml_slots_lock)
  >>   >>
  >>   >>    static void kvm_slot_init_dirty_bitmap(KVMSlot *mem);
  >>   >>   +static int kvm_get_vcpu(KVMState *s, unsigned long vcpu_id);
  >>   >>
  >>   >>    static inline void kvm_resample_fd_remove(int gsi)  { @@ -340,14
  >>   +341,53
  >>   >>   @@ err:
  >>   >>        return ret;
  >>   >>    }
  >>   >>
  >>   >>   +void kvm_park_vcpu(CPUState *cpu)
  >>   >>   +{
  >>   >>   +    struct KVMParkedVcpu *vcpu;
  >>   >>   +
  >>   >>   +    trace_kvm_park_vcpu(cpu->cpu_index,
  kvm_arch_vcpu_id(cpu));
  >>   >>   +
  >>   >>   +    vcpu = g_malloc0(sizeof(*vcpu));
  >>   >>   +    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  >>   >>   +    vcpu->kvm_fd = cpu->kvm_fd;
  >>   >>   +    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,
  >>   node); }
  >>   >>   +
  >>   >>   +int kvm_create_vcpu(CPUState *cpu)
  >>   >>   +{
  >>   >>   +    unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);
  >>   >>   +    KVMState *s = kvm_state;
  >>   >>   +    int kvm_fd;
  >>   >>   +
  >>   >>   +    trace_kvm_create_vcpu(cpu->cpu_index,
  kvm_arch_vcpu_id(cpu));
  >>   >>   +
  >>   >>   +    /* check if the KVM vCPU already exist but is parked */
  >>   >>   +    kvm_fd = kvm_get_vcpu(s, vcpu_id);
  >>   >>   +    if (kvm_fd < 0) {
  >>   >>   +        /* vCPU not parked: create a new KVM vCPU */
  >>   >>   +        kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
  >>   >>   +        if (kvm_fd < 0) {
  >>   >>   +            error_report("KVM_CREATE_VCPU IOCTL failed for vCPU
  %lu",
  >>   >>   vcpu_id);
  >>   >>   +            return kvm_fd;
  >>   >>   +        }
  >>   >>   +    }
  >>   >>   +
  >>   >>   +    cpu->kvm_fd = kvm_fd;
  >>   >>   +    cpu->kvm_state = s;
  >>   >>   +    cpu->vcpu_dirty = true;
  >>   >>   +    cpu->dirty_pages = 0;
  >>   >>   +    cpu->throttle_us_per_full = 0;
  >>   >>   +
  >>   >>   +    return 0;
  >>   >>   +}
  >>   >>   +
  >>   >>    static int do_kvm_destroy_vcpu(CPUState *cpu)  {
  >>   >>        KVMState *s = kvm_state;
  >>   >>        long mmap_size;
  >>   >>   -    struct KVMParkedVcpu *vcpu = NULL;
  >>   >>        int ret = 0;
  >>   >>
  >>   >>   -    trace_kvm_destroy_vcpu();
  >>   >>   +    trace_kvm_destroy_vcpu(cpu->cpu_index,
  >>   kvm_arch_vcpu_id(cpu));
  >>   >>
  >>   >>        ret = kvm_arch_destroy_vcpu(cpu);
  >>   >>        if (ret < 0) {
  >>   >>   @@ -373,10 +413,7 @@ static int do_kvm_destroy_vcpu(CPUState
  >>   *cpu)
  >>   >>            }
  >>   >>        }
  >>   >>
  >>   >>   -    vcpu = g_malloc0(sizeof(*vcpu));
  >>   >>   -    vcpu->vcpu_id = kvm_arch_vcpu_id(cpu);
  >>   >>   -    vcpu->kvm_fd = cpu->kvm_fd;
  >>   >>   -    QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu,
  >>   node);
  >>   >>   +    kvm_park_vcpu(cpu);
  >>   >>    err:
  >>   >>        return ret;
  >>   >>    }
  >>   >>   @@ -397,6 +434,8 @@ static int kvm_get_vcpu(KVMState *s,
  unsigned
  >>   long
  >>   >>   vcpu_id)
  >>   >>            if (cpu->vcpu_id == vcpu_id) {
  >>   >>                int kvm_fd;
  >>   >>
  >>   >>   +            trace_kvm_get_vcpu(vcpu_id);
  >>   >>   +
  >>   >>                QLIST_REMOVE(cpu, node);
  >>   >>                kvm_fd = cpu->kvm_fd;
  >>   >>                g_free(cpu);
  >>   >>   @@ -404,7 +443,7 @@ static int kvm_get_vcpu(KVMState *s,
  unsigned
  >>   long
  >>   >>   vcpu_id)
  >>   >>            }
  >>   >>        }
  >>   >>
  >>   >>   -    return kvm_vm_ioctl(s, KVM_CREATE_VCPU, (void *)vcpu_id);
  >>   >>   +    return -ENOENT;
  >>   >>    }
  >>   >>
  >>   >>    int kvm_init_vcpu(CPUState *cpu, Error **errp) @@ -415,19
  +454,14
  >>   @@
  >>   >>   int kvm_init_vcpu(CPUState *cpu, Error **errp)
  >>   >>
  >>   >>        trace_kvm_init_vcpu(cpu->cpu_index, kvm_arch_vcpu_id(cpu));
  >>   >>
  >>   >>   -    ret = kvm_get_vcpu(s, kvm_arch_vcpu_id(cpu));
  >>   >>   +    ret = kvm_create_vcpu(cpu);
  >>   >>        if (ret < 0) {
  >>   >>   -        error_setg_errno(errp, -ret, "kvm_init_vcpu: kvm_get_vcpu
  failed
  >>   >>   (%lu)",
  >>   >>   +        error_setg_errno(errp, -ret,
  >>   >>   +                         "kvm_init_vcpu: kvm_create_vcpu failed (%lu)",
  >>   >>                             kvm_arch_vcpu_id(cpu));
  >>   >>            goto err;
  >>   >>        }
  >>   >>
  >>   >>   -    cpu->kvm_fd = ret;
  >>   >>   -    cpu->kvm_state = s;
  >>   >>   -    cpu->vcpu_dirty = true;
  >>   >>   -    cpu->dirty_pages = 0;
  >>   >>   -    cpu->throttle_us_per_full = 0;
  >>   >>   -
  >>   >>        mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);
  >>   >>        if (mmap_size < 0) {
  >>   >>            ret = mmap_size;
  >>   >>   diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events index
  >>   >>   681ccb667d..75c1724e78 100644
  >>   >>   --- a/accel/kvm/trace-events
  >>   >>   +++ b/accel/kvm/trace-events
  >>   >>   @@ -9,6 +9,10 @@ kvm_device_ioctl(int fd, int type, void *arg)
  "dev fd
  >>   %d,
  >>   >>   type 0x%x, arg %p"
  >>   >>    kvm_failed_reg_get(uint64_t id, const char *msg) "Warning:
  Unable to
  >>   >>   retrieve ONEREG %" PRIu64 " from KVM: %s"
  >>   >>    kvm_failed_reg_set(uint64_t id, const char *msg) "Warning:
  Unable to
  >>   set
  >>   >>   ONEREG %" PRIu64 " to KVM: %s"
  >>   >>    kvm_init_vcpu(int cpu_index, unsigned long arch_cpu_id) "index:
  %d
  >>   id:
  >>   >>   %lu"
  >>   >>   +kvm_create_vcpu(int cpu_index, unsigned long arch_cpu_id)
  "index:
  >>   %d
  >>   >>   id: %lu"
  >>   >>   +kvm_get_vcpu(unsigned long arch_cpu_id) "id: %lu"
  >>   >>   +kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id)
  "index:
  >>   %d
  >>   >>   id: %lu"
  >>   >>   +kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id)
  "index: %d
  >>   id:
  >>   >>   %lu"
  >>   >>    kvm_irqchip_commit_routes(void) ""
  >>   >>    kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev
  %s
  >>   >>   vector %d virq %d"
  >>   >>    kvm_irqchip_update_msi_route(int virq) "Updating MSI route
  >>   virq=%d"
  >>   >>   @@ -25,7 +29,6 @@ kvm_dirty_ring_reaper(const char *s) "%s"
  >>   >>    kvm_dirty_ring_reap(uint64_t count, int64_t t) "reaped %"PRIu64"
  >>   pages
  >>   >>   (took %"PRIi64" us)"
  >>   >>    kvm_dirty_ring_reaper_kick(const char *reason) "%s"
  >>   >>    kvm_dirty_ring_flush(int finished) "%d"
  >>   >>   -kvm_destroy_vcpu(void) ""
  >>   >>    kvm_failed_get_vcpu_mmap_size(void) ""
  >>   >>    kvm_cpu_exec(void) ""
  >>   >>    kvm_interrupt_exit_request(void) ""
  >>   >>   --
  >>   >>   2.39.3
  >>   >
#10 lore raw

Re: [PATCH v2 1/4] accel/kvm: Extract common KVM vCPU {creation, parking} code

In reply to Salil Mehta via
Quoted 57 lines · Salil Mehta
Quoted 25 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 2:07 PM

Hi Salil,

Quoted 18 lines · Salil Mehta

Hi Harsh,

Quoted 11 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 11:15 AM

Hi Salil,

Thanks for your email.
Your patch 1/8 is included here based on review comments on my previous patch from one of the maintainers in the community and therefore I had kept you in CC to be aware of the desire of having this independent patch to get merged earlier even if your other patches in the series may go through further reviews.

I really don’t know which discussion are you pointing at? Please understand you are fixing a bug and we are pushing a feature which has got large series. It will break the patch-set which is about t be merged.

Quoted 6 lines
Quoted 3 lines

There will be significant overhead of testing on us for the work we have been carrying forward for large time. This will be disruptive. Please dont!

I was referring to the review discussion on my prev patch here:
https://lore.kernel.org/qemu-devel/[email protected]/

Sure, I'm, not sure what this means.

Quoted 2 lines

Although your patch was included with this series only to facilitate review of the additional patches depending on just one of your patch.

Generally you rebase your patch-set over the other and clearly state on the cover letter that this patch-set is dependent upon such and such patch-set. Just imagine if everyone starts to unilaterally pick up patches from each other's patch-set it will create a chaos not only for the feature owners but also for the maintainers.

Quoted 4 lines

I am not sure what is appearing disruptive here. It is a common practive in the community that maintainer(s) can pick individual patches from the series if it has been vetted by siginificant number of reviewers.

Don’t you think this patch-set is asking for acceptance for a patch already part of another patch-set which is about to be accepted and is a bigger feature? Will it cause maintenance overhead at the last moment? Yes, of course!

Quoted 7 lines
Quoted 3 lines

However, in this case, since you have mentioned to post next version soon, you need not worry about it as that would be the preferred version for both of the series.

Yes, but please understand we are working for the benefit of overall community. Please cooperate here.

There might be a misunderstanding, Harsh just said there had not been much progress on your series for a while and he wasn't sure what the status was. I mentioned that we *could* take your patch 1 (with your blessing) if there was a hold up with the rest of the series. He was going to check in with you to see how it was going.

This patch 1 was not intended to be merged as is without syncing up with you first, but it's understandable you were concerned because that was probably not communicated with you clearly.

I appreciate you bringing up your concerns, we'll try to do better.

Thanks,
Nick

#11 lore raw

RE: [PATCH v2 1/4] accel/kvm: Extract common KVM vCPU {creation, parking} code

Hi Nick,

Quoted 68 lines

From: Nicholas Piggin <[email protected]>
Sent: Friday, May 17, 2024 4:44 AM

Quoted 37 lines · Salil Mehta
Quoted 25 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 2:07 PM

Hi Salil,

Quoted 18 lines · Salil Mehta

Hi Harsh,

Quoted 11 lines

From: Harsh Prateek Bora <[email protected]>
Sent: Thursday, May 16, 2024 11:15 AM

Hi Salil,

Thanks for your email.
Your patch 1/8 is included here based on review comments on my previous patch from one of the maintainers in the community and therefore I had kept you in CC to be aware of the desire of having this independent patch to get merged earlier even if your other patches in the series may go through further reviews.

I really don’t know which discussion are you pointing at? Please understand you are fixing a bug and we are pushing a feature which has got large series. It will break the patch-set which is about t be merged.

Quoted 7 lines
Quoted 1 line

There will be significant overhead of testing on us for the work

we > have been carrying forward for large time. This will be disruptive. Please dont!

Quoted 1 line

I was referring to the review discussion on my prev patch here:

https://lore.kernel.org/qemu-devel/[email protected]/

Sure, I'm, not sure what this means.

Quoted 8 lines
Quoted 2 lines

Although your patch was included with this series only to
facilitate review of the additional patches depending on just one of your patch.

Generally you rebase your patch-set over the other and clearly state on the cover letter that this patch-set is dependent upon such and such patch-set. Just imagine if everyone starts to unilaterally pick up patches from each other's patch-set it will create a chaos not only for

the feature owners but also for the maintainers.

Quoted 18 lines
Quoted 6 lines

I am not sure what is appearing disruptive here. It is a common
practive in the community that maintainer(s) can pick individual patches from the series if it has been vetted by siginificant number of reviewers.

Don’t you think this patch-set is asking for acceptance for a patch already part of another patch-set which is about to be accepted and is a bigger feature? Will it cause maintenance overhead at the last moment? Yes, of course!

Quoted 3 lines

However, in this case, since you have mentioned to post next
version soon, you need not worry about it as that would be the preferred version for both of the series.

Quoted 8 lines
Quoted 2 lines

Yes, but please understand we are working for the benefit of overall community. Please cooperate here.

There might be a misunderstanding, Harsh just said there had not been much progress on your series for a while and he wasn't sure what the status was. I mentioned that we *could* take your patch 1 (with your blessing) if there was a hold up with the rest of the series. He was going to check in with you to see how it was going.

Thanks for the clarification. No issues. I'm planning to float V9 of this series by Monday and perhaps that’s all you want. 😊

As such, new cycle started on 23rd April and we had been busy rebasing and testing. This series works in conjunction with other series. We have to ensure both are compatible.

Quoted 3 lines

This patch 1 was not intended to be merged as is without syncing up with you first, but it's understandable you were concerned because that was probably not communicated with you clearly.

No issues. I think we all are in the same page now. I understand your requirement. We are trying our best to expedite acceptance of this series. Perhaps your reviews on V9 might help.

Quoted 2 lines

I appreciate you bringing up your concerns, we'll try to do better.

No problem. Thanks

Salil.

Quoted 3 lines

Thanks,
Nick

#12 lore raw

PATCH v2 2/4 accel/kvm: Introduce kvm_create_and_park_vcpu() helper

In reply to Harsh Prateek Bora

There are distinct helpers for creating and parking a KVM vCPU. However, there can be cases where a platform needs to create and immediately park the vCPU during early stages of vcpu init which can later be reused when vcpu thread gets initialized. This would help detect failures with kvm_create_vcpu at an early stage.

Based on api refactoring to create/park vcpus introduced in 1/8 of patch series:
https://lore.kernel.org/qemu-devel/[email protected]/

---
 include/sysemu/kvm.h |  8 ++++++++
 accel/kvm/kvm-all.c  | 12 ++++++++++++
 2 files changed, 20 insertions(+)
include/sysemu/kvm.h +8 -0
index fa3ec74442..221e6bd55b 100644
@@ -450,6 +450,14 @@ int kvm_create_vcpu(CPUState *cpu);
  */
 void kvm_park_vcpu(CPUState *cpu);
 
+/**
+ * kvm_create_and_park_vcpu - Create and park a KVM vCPU
+ * @cpu: QOM CPUState object for which KVM vCPU has to be created and parked.
+ *
+ * @returns: 0 when success, errno (<0) when failed.
+ */
+int kvm_create_and_park_vcpu(CPUState *cpu);
+
 #endif /* COMPILING_PER_TARGET */
 
 void kvm_cpu_synchronize_state(CPUState *cpu);
accel/kvm/kvm-all.c +12 -0
index 30d42847de..3d7e5eaf0b 100644
@@ -381,6 +381,18 @@ int kvm_create_vcpu(CPUState *cpu)
     return 0;
 }
 
+int kvm_create_and_park_vcpu(CPUState *cpu)
+{
+    int ret = 0;
+
+    ret = kvm_create_vcpu(cpu);
+    if (!ret) {
+        kvm_park_vcpu(cpu);
+    }
+
+    return ret;
+}
+
 static int do_kvm_destroy_vcpu(CPUState *cpu)
 {
     KVMState *s = kvm_state;
Signature · 1 line
2.39.3
#13 lore raw

PATCH v2 3/4 cpu-common.c: export cpu_get_free_index to be reused later

In reply to Harsh Prateek Bora

This helper provides an easy way to identify the next available free cpu index which can be used for vcpu creation. Until now, this is being called at a very later stage and there is a need to be able to call it earlier (for now, with ppc64) hence the need to export.

---
 include/exec/cpu-common.h | 2 ++
 cpu-common.c              | 7 ++++---
 2 files changed, 6 insertions(+), 3 deletions(-)
include/exec/cpu-common.h +2 -0
index 6d5318895a..0386f1ab29 100644
@@ -29,6 +29,8 @@ void cpu_list_lock(void);
 void cpu_list_unlock(void);
 unsigned int cpu_list_generation_id_get(void);
 
+int cpu_get_free_index(void);
+
 void tcg_iommu_init_notifier_list(CPUState *cpu);
 void tcg_iommu_free_notifier_list(CPUState *cpu);
 
cpu-common.c +4 -3
index ce78273af5..82bd1b432d 100644
@@ -57,14 +57,12 @@ void cpu_list_unlock(void)
     qemu_mutex_unlock(&qemu_cpu_list_lock);
 }
 
-static bool cpu_index_auto_assigned;
 
-static int cpu_get_free_index(void)
+int cpu_get_free_index(void)
 {
     CPUState *some_cpu;
     int max_cpu_index = 0;
 
-    cpu_index_auto_assigned = true;
     CPU_FOREACH(some_cpu) {
         if (some_cpu->cpu_index >= max_cpu_index) {
             max_cpu_index = some_cpu->cpu_index + 1;
@@ -83,8 +81,11 @@ unsigned int cpu_list_generation_id_get(void)
 
 void cpu_list_add(CPUState *cpu)
 {
+    static bool cpu_index_auto_assigned;
+
     QEMU_LOCK_GUARD(&qemu_cpu_list_lock);
     if (cpu->cpu_index == UNASSIGNED_CPU_INDEX) {
+        cpu_index_auto_assigned = true;
         cpu->cpu_index = cpu_get_free_index();
         assert(cpu->cpu_index != UNASSIGNED_CPU_INDEX);
     } else {
Signature · 1 line
2.39.3
#14 lore raw

PATCH v2 4/4 target/ppc: handle vcpu hotplug failure gracefully

In reply to Harsh Prateek Bora

On ppc64, the PowerVM hypervisor runs with limited memory and a VCPU creation during hotplug may fail during kvm_ioctl for KVM_CREATE_VCPU, leading to termination of guest since errp is set to &error_fatal while calling kvm_init_vcpu. This unexpected behaviour can be avoided by pre-creating and parking vcpu on success or return error otherwise. This enables graceful error delivery for any vcpu hotplug failures while the guest can keep running.

Based on api refactoring to create/park vcpus introduced in 1/8 of patch series:
https://lore.kernel.org/qemu-devel/[email protected]/

Tested OK by repeatedly doing a hotplug/unplug of vcpus as below:

#virsh setvcpus hotplug 40
#virsh setvcpus hotplug 70
error: internal error: unable to execute QEMU command 'device_add':
kvmppc_cpu_realize: vcpu hotplug failed with -12

Signed-off by: Harsh Prateek Bora <[email protected]>

---
 target/ppc/kvm.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
target/ppc/kvm.c +24 -0
index 63930d4a77..25f0cf0ba8 100644
@@ -48,6 +48,8 @@
 #include "qemu/mmap-alloc.h"
 #include "elf.h"
 #include "sysemu/kvm_int.h"
+#include "sysemu/kvm.h"
+#include "hw/core/accel-cpu.h"
 
 #define PROC_DEVTREE_CPU      "/proc/device-tree/cpus/"
 
@@ -2339,6 +2341,26 @@ static void alter_insns(uint64_t *word, uint64_t flags, bool on)
     }
 }
 
+static bool kvmppc_cpu_realize(CPUState *cs, Error **errp)
+{
+    int ret;
+
+    cs->cpu_index = cpu_get_free_index();
+
+    POWERPC_CPU(cs)->vcpu_id = cs->cpu_index;
+
+    if (cs->parent_obj.hotplugged) {
+        /* create and park to fail gracefully in case vcpu hotplug fails */
+        ret = kvm_create_and_park_vcpu(cs);
+        if (ret) {
+            error_setg(errp, "%s: vcpu hotplug failed with %d",
+                             __func__, ret);
+            return false;
+        }
+    }
+    return true;
+}
+
 static void kvmppc_host_cpu_class_init(ObjectClass *oc, void *data)
 {
     PowerPCCPUClass *pcc = POWERPC_CPU_CLASS(oc);
@@ -2958,4 +2980,6 @@ void kvmppc_set_reg_tb_offset(PowerPCCPU *cpu, int64_t tb_offset)
 
 void kvm_arch_accel_class_init(ObjectClass *oc)
 {
+    AccelClass *ac = ACCEL_CLASS(oc);
+    ac->cpu_common_realize = kvmppc_cpu_realize;
 }
Signature · 1 line
2.39.3