Skip to contentSkip to conversation

qemu-kvm: x86: Refactor persistent CPU state

RFC PATCH Superseded Jan Kiszka 3 participants 7 messages May 21, 2009 – May 27, 2009 KVM

lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

RFC PATCH 1/1 qemu-kvm: x86: Refactor persistent CPU state

This patch reworks the KVM-related layout and encoding of the CPU state to be saved to disk or migrated. The goal is to define a format, version 9, that is also acceptable for upstream and can later be merged into QEMU. Besides unconditionally writing KVM states, this format compresses interrupt_bitmap into a single number as there can be no more than one pending IRQ at a time.

Signed-off-by Jan Kiszka <[email protected]>

---

 target-i386/cpu.h     |    2 +-
 target-i386/machine.c |   57 +++++++++++++++++++++++++++++++++++--------------
 2 files changed, 42 insertions(+), 17 deletions(-)
target-i386/cpu.h +1 -1
index f054af1..e07b504 100644
@@ -838,7 +838,7 @@ static inline int cpu_get_time_fast(void)
 #define cpu_signal_handler cpu_x86_signal_handler
 #define cpu_list x86_cpu_list
 
-#define CPU_SAVE_VERSION 8
+#define CPU_SAVE_VERSION 9
 
 /* MMU modes definitions */
 #define MMU_MODE0_SUFFIX _kernel
target-i386/machine.c +41 -16

90 lines not on this page · raw message

#1 lore raw
Quoted 11 lines · Jan Kiszka

This patch reworks the KVM-related layout and encoding of the CPU state to be saved to disk or migrated. The goal is to define a format, version 9, that is also acceptable for upstream and can later be merged into QEMU. Besides unconditionally writing KVM states, this format compresses interrupt_bitmap into a single number as there can be no more than one pending IRQ at a time.

target-i386/machine.c +0 -2
-    if (kvm_enabled()) {
-        for (i = 0; i < sizeof(env->interrupt_bitmap)/8 ; i++) {

target-i386/machine.c void cpu_save(QEMUFile *f, void *opaque) · line 164

Quoted 10 lines
target-i386/machine.c +9 -1
-            qemu_put_be64s(f, &env->interrupt_bitmap[i]);
+    /* KVM-related states */
+
+    /* There can only be one pending IRQ set in the bitmap at a time, so try
+       to find it and save its number instead (-1 for none). */
+    pending_irq = -1;
+    for (i = 0; i < sizeof(env->interrupt_bitmap)/2; i++) {
+        bit = ffs(((uint16_t *)env->interrupt_bitmap)[i]);
+        if (bit) {
+            pending_irq = i * 16 + bit;

bit - 1

Quoted 3 lines
+            break;
         }

Why are you using uint64_t? It would be more natural to use ffsll and the natural type of the bitmap vector.

target-i386/machine.c int cpu_load(QEMUFile *f, void *opaque, int version_id) · line 385

Quoted 7 lines
target-i386/machine.c +3 -0
+        } else {
+            kvm_load_registers(env);
+            kvm_load_tsc(env);
             kvm_load_mpstate(env);
         }
     }

Don't you need to load the pending interrupt?

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#2 lore raw

PATCH v2 1/1 qemu-kvm: x86: Refactor persistent CPU state

Quoted 24 lines · Avi Kivity
Quoted 16 lines · Jan Kiszka

This patch reworks the KVM-related layout and encoding of the CPU state to be saved to disk or migrated. The goal is to define a format, version 9, that is also acceptable for upstream and can later be merged into QEMU. Besides unconditionally writing KVM states, this format compresses interrupt_bitmap into a single number as there can be no more than one pending IRQ at a time.

(quoted diff) +3 -3
-    if (kvm_enabled()) {
-        for (i = 0; i < sizeof(env->interrupt_bitmap)/8 ; i++) {
-            qemu_put_be64s(f, &env->interrupt_bitmap[i]);
+    /* KVM-related states */
+
+    /* There can only be one pending IRQ set in the bitmap at a time,

so try

Review

Quoted 6 lines
(quoted diff) +6 -0
+       to find it and save its number instead (-1 for none). */
+    pending_irq = -1;
+    for (i = 0; i < sizeof(env->interrupt_bitmap)/2; i++) {
+        bit = ffs(((uint16_t *)env->interrupt_bitmap)[i]);
+        if (bit) {
+            pending_irq = i * 16 + bit;
Quoted 1 line

bit - 1

Fixed.

Quoted 7 lines
Quoted 4 lines
+            break;
         }

Why are you using uint64_t? It would be more natural to use ffsll and the natural type of the bitmap vector.

Better don't ask how I interpreted the ffs doc.

Quoted 11 lines

Review

Quoted 8 lines
(quoted diff) +3 -0
+        } else {
+            kvm_load_registers(env);
+            kvm_load_tsc(env);
             kvm_load_mpstate(env);
         }
     }

Don't you need to load the pending interrupt?

For >= v9 this is done in the previous hunk.

------------>

This patch reworks the KVM-related layout and encoding of the CPU state to be saved to disk or migrated. The goal is to define a format, version 9, that is also acceptable for upstream and can later be merged into QEMU. Besides unconditionally writing KVM states, this format compresses interrupt_bitmap into a single number as there can be no more than one pending IRQ at a time.

Changes in v2:
- fixed and simplified interrupt_bitmap parsing

Signed-off-by Jan Kiszka <[email protected]>

---

 target-i386/cpu.h     |    2 +-
 target-i386/machine.c |   57 +++++++++++++++++++++++++++++++++++--------------
 2 files changed, 42 insertions(+), 17 deletions(-)
target-i386/cpu.h +1 -1
index f054af1..e07b504 100644
@@ -838,7 +838,7 @@ static inline int cpu_get_time_fast(void)
 #define cpu_signal_handler cpu_x86_signal_handler
 #define cpu_list x86_cpu_list
 
-#define CPU_SAVE_VERSION 8
+#define CPU_SAVE_VERSION 9
 
 /* MMU modes definitions */
 #define MMU_MODE0_SUFFIX _kernel
target-i386/machine.c +41 -16

90 lines not on this page · raw message

#3 lore raw
Quoted 6 lines · Jan Kiszka
Quoted 3 lines

Why are you using uint64_t? It would be more natural to use ffsll and the natural type of the bitmap vector.

Better don't ask how I interpreted the ffs doc.

It's a misdesigned API anyway.

target-i386/machine.c int cpu_load(QEMUFile *f, void *opaque, int version_id) · line 346

Quoted 7 lines
target-i386/machine.c +5 -0
+    if (version_id >= 9) {
+        qemu_get_sbe32s(f, &pending_irq);
+        memset(&env->interrupt_bitmap, 0, sizeof(env->interrupt_bitmap));
+        if (pending_irq >= 0) {
+            env->interrupt_bitmap[pending_irq / 64] |= 1 << (pending_irq % 64);

(uint64_t)1 so we don't lose the upper word.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw

PATCH v3 1/1 qemu-kvm: x86: Refactor persistent CPU state

This patch reworks the KVM-related layout and encoding of the CPU state to be saved to disk or migrated. The goal is to define a format, version 9, that is also acceptable for upstream and can later be merged into QEMU. Besides unconditionally writing KVM states, this format compresses interrupt_bitmap into a single number as there can be no more than one pending IRQ at a time.

Changes in v3:
- fixed bit-shift overflow
- replace upstream v9 format (should be merged upstream quickly)

Changes in v2:
- fixed and simplified interrupt_bitmap parsing

Signed-off-by Jan Kiszka <[email protected]>

---

 target-i386/machine.c |   51 ++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 44 insertions(+), 7 deletions(-)
target-i386/machine.c +44 -7
index 124f8dc..f280d3d 100644
@@ -29,7 +29,8 @@ void cpu_save(QEMUFile *f, void *opaque)
     uint16_t fptag, fpus, fpuc, fpregs_format;
     uint32_t hflags;
     int32_t a20_mask;
-    int i;
+    int32_t pending_irq;
+    int i, bit;
 
     cpu_synchronize_state(env, 0);
 

@@ -141,11 +142,21 @@ void cpu_save(QEMUFile *f, void *opaque)
         qemu_put_be64s(f, &env->mtrr_var[i].mask);
     }
 
-    for (i = 0; i < sizeof(env->interrupt_bitmap)/8; i++) {
-        qemu_put_be64s(f, &env->interrupt_bitmap[i]);
+    /* KVM-related states */
+
+    /* There can only be one pending IRQ set in the bitmap at a time, so try
+       to find it and save its number instead (-1 for none). */
+    pending_irq = -1;
+    for (i = 0; i < ARRAY_SIZE(env->interrupt_bitmap); i++) {
+        bit = ffsll(env->interrupt_bitmap[i]);
+        if (bit) {
+            pending_irq = i * 64 + bit - 1;
+            break;
+        }
     }
-    qemu_put_be64s(f, &env->tsc);
+    qemu_put_sbe32s(f, &pending_irq);
     qemu_put_be32s(f, &env->mp_state);
+    qemu_put_be64s(f, &env->tsc);
 }
 
 #ifdef USE_X86LDOUBLE

@@ -179,6 +190,7 @@ int cpu_load(QEMUFile *f, void *opaque, int version_id)
     uint32_t hflags;
     uint16_t fpus, fpuc, fptag, fpregs_format;
     int32_t a20_mask;
+    int32_t pending_irq;
 
     if (version_id < 3 || version_id > CPU_SAVE_VERSION)
         return -EINVAL;

@@ -324,12 +336,16 @@ int cpu_load(QEMUFile *f, void *opaque, int version_id)
             qemu_get_be64s(f, &env->mtrr_var[i].mask);
         }
     }
+
     if (version_id >= 9) {
-        for (i = 0; i < sizeof(env->interrupt_bitmap)/8; i++) {
-            qemu_get_be64s(f, &env->interrupt_bitmap[i]);
+        qemu_get_sbe32s(f, &pending_irq);
+        memset(&env->interrupt_bitmap, 0, sizeof(env->interrupt_bitmap));
+        if (pending_irq >= 0) {
+            env->interrupt_bitmap[pending_irq / 64] |=
+                (uint64_t)1 << (pending_irq % 64);
         }
-        qemu_get_be64s(f, &env->tsc);
         qemu_get_be32s(f, &env->mp_state);
+        qemu_get_be64s(f, &env->tsc);
     }
 
     /* XXX: ensure compatiblity for halted bit ? */

@@ -337,5 +353,26 @@ int cpu_load(QEMUFile *f, void *opaque, int version_id)
     env->hflags = hflags;
     tlb_flush(env, 1);
     cpu_synchronize_state(env, 1);
+    if (kvm_enabled()) {
+        /* when in-kernel irqchip is used, env->halted causes deadlock
+           because no userspace IRQs will ever clear this flag */
+        env->halted = 0;
+        if (version_id < 9) {
+            for (i = 0; i < sizeof(env->interrupt_bitmap)/8; i++) {
+                qemu_get_be64s(f, &env->interrupt_bitmap[i]);
+            }
+            qemu_get_be64s(f, &env->tsc);
+            kvm_load_registers(env);
+            kvm_load_tsc(env);
+            if (version_id >= 5) {
+                qemu_get_be32s(f, &env->mp_state);
+                kvm_load_mpstate(env);
+            }
+        } else {
+            kvm_load_registers(env);
+            kvm_load_tsc(env);
+            kvm_load_mpstate(env);
+        }
+    }
     return 0;
 }
Applied
#5 lore raw
Quoted 7 lines · Jan Kiszka

This patch reworks the KVM-related layout and encoding of the CPU state to be saved to disk or migrated. The goal is to define a format, version 9, that is also acceptable for upstream and can later be merged into QEMU. Besides unconditionally writing KVM states, this format compresses interrupt_bitmap into a single number as there can be no more than one pending IRQ at a time.

Applied, thanks.

Signature · 1 line
error compiling committee.c: too many arguments to function
#6 lore raw
Quoted 10 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka

This patch reworks the KVM-related layout and encoding of the CPU state to be saved to disk or migrated. The goal is to define a format, version 9, that is also acceptable for upstream and can later be merged into QEMU. Besides unconditionally writing KVM states, this format compresses interrupt_bitmap into a single number as there can be no more than one pending IRQ at a time.

Applied, thanks.

OK, will push its core upstream now.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux