Skip to contentSkip to conversation

apic: Fix migration breakage of >255 vcpus

PATCH Superseded Peter Xu 4 participants 14 messages Oct 15, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Peter Xu

author
#0 lore raw

cover apic: Fix migration breakage of >255 vcpus

I'm not very certain, but... it seems to be broken starting from when x2apic was introduced in QEMU, until now.

Please review, thanks.

Peter Xu (2):
migration: Boost SaveStateEntry.instance_id to 64 bits
apic: Use 32bit APIC ID for migration instance ID

 hw/intc/apic_common.c        |  2 +-
 include/migration/register.h |  2 +-
 include/migration/vmstate.h  |  4 ++--
 migration/savevm.c           | 10 +++++-----
 stubs/vmstate.c              |  2 +-
 5 files changed, 10 insertions(+), 10 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 1/2 migration: Boost SaveStateEntry.instance_id to 64 bits

It was "int" and used as 32bits fields (see save_section_header()). It's unsafe already because sizeof(int) could be 2 on i386, I think. So at least uint32_t would suite more. While it also uses "-1" as a placeholder of "we want to generate the instance ID automatically". Hence a more proper value should be int64_t.

This will start to be useful after next patch in which we can start to convert a real uint32_t value as instance ID.

---
 include/migration/register.h |  2 +-
 include/migration/vmstate.h  |  4 ++--
 migration/savevm.c           | 10 +++++-----
 stubs/vmstate.c              |  2 +-
 4 files changed, 9 insertions(+), 9 deletions(-)
include/migration/register.h +1 -1
index a13359a08d..54f42c7413 100644
@@ -69,7 +69,7 @@ typedef struct SaveVMHandlers {
 } SaveVMHandlers;
 
 int register_savevm_live(const char *idstr,
-                         int instance_id,
+                         int64_t instance_id,
                          int version_id,
                          const SaveVMHandlers *ops,
                          void *opaque);
include/migration/vmstate.h +2 -2
index 1fbfd099dd..6a7498463c 100644
@@ -1114,14 +1114,14 @@ int vmstate_save_state_v(QEMUFile *f, const VMStateDescription *vmsd,
 bool vmstate_save_needed(const VMStateDescription *vmsd, void *opaque);
 
 /* Returns: 0 on success, -1 on failure */
-int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
+int vmstate_register_with_alias_id(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *base, int alias_id,
                                    int required_for_version,
                                    Error **errp);
 
 /* Returns: 0 on success, -1 on failure */
-static inline int vmstate_register(DeviceState *dev, int instance_id,
+static inline int vmstate_register(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *opaque)
 {
migration/savevm.c +5 -5
index bb9462a54d..dc9281c897 100644
@@ -233,7 +233,7 @@ typedef struct CompatEntry {
 typedef struct SaveStateEntry {
     QTAILQ_ENTRY(SaveStateEntry) entry;
     char idstr[256];
-    int instance_id;
+    int64_t instance_id;
     int alias_id;
     int version_id;
     /* version id read from the stream */
@@ -668,7 +668,7 @@ void dump_vmstate_json_to_file(FILE *out_file)
 static int calculate_new_instance_id(const char *idstr)
 {
     SaveStateEntry *se;
-    int instance_id = 0;
+    int64_t instance_id = 0;
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         if (strcmp(idstr, se->idstr) == 0
@@ -730,7 +730,7 @@ static void savevm_state_handler_insert(SaveStateEntry *nse)
    Meanwhile pass -1 as instance_id if you do not already have a clearly
    distinguishing id for all instances of your device class. */
 int register_savevm_live(const char *idstr,
-                         int instance_id,
+                         int64_t instance_id,
                          int version_id,
                          const SaveVMHandlers *ops,
                          void *opaque)
@@ -784,7 +784,7 @@ void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque)
     }
 }
 
-int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
+int vmstate_register_with_alias_id(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *opaque, int alias_id,
                                    int required_for_version,
@@ -1566,7 +1566,7 @@ int qemu_save_device_state(QEMUFile *f)
     return qemu_file_get_error(f);
 }
 
-static SaveStateEntry *find_se(const char *idstr, int instance_id)
+static SaveStateEntry *find_se(const char *idstr, int64_t instance_id)
 {
     SaveStateEntry *se;
 
stubs/vmstate.c +1 -1
index e1e89b87f0..699003f3b0 100644
@@ -4,7 +4,7 @@
 const VMStateDescription vmstate_dummy = {};
 
 int vmstate_register_with_alias_id(DeviceState *dev,
-                                   int instance_id,
+                                   int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *base, int alias_id,
                                    int required_for_version,
Signature · 1 line
2.21.0
#2 lore raw

Re: [PATCH 1/2] migration: Boost SaveStateEntry.instance_id to 64 bits

Quoted 2 lines · Peter Xu

It was "int" and used as 32bits fields (see save_section_header()). It's unsafe already because sizeof(int) could be 2 on i386,

i386 is 32bits, so int is 32bits O:-)
I really hope that we would never, ever, need a 64bits instance id. It would mean that we have more than 2.000.000.000 objects of the same type, no?

I am pretty sure than in 16bits platforms we have other problems than insntance_id (namely that we don't have enough memory).

Quoted 7 lines

I think.
So at least uint32_t would suite more. While it also uses "-1" as a placeholder of "we want to generate the instance ID automatically". Hence a more proper value should be int64_t.

This will start to be useful after next patch in which we can start to convert a real uint32_t value as instance ID.

Later, Juan.

author
#3 lore raw

Re: [PATCH 1/2] migration: Boost SaveStateEntry.instance_id to 64 bits

Quoted 5 lines · Juan Quintela
Quoted 2 lines · Peter Xu

It was "int" and used as 32bits fields (see save_section_header()). It's unsafe already because sizeof(int) could be 2 on i386,

i386 is 32bits, so int is 32bits O:-)

Right it should be 16 bits systems. And yes I don't think we need to consider that! :)

Signature · 1 line
Peter Xu
#4 lore raw

Re: [PATCH 1/2] migration: Boost SaveStateEntry.instance_id to 64 bits

In reply to Peter Xu
Quoted 10 lines · Peter Xu

It was "int" and used as 32bits fields (see save_section_header()). It's unsafe already because sizeof(int) could be 2 on i386, I think. So at least uint32_t would suite more. While it also uses "-1" as a placeholder of "we want to generate the instance ID automatically". Hence a more proper value should be int64_t.

This will start to be useful after next patch in which we can start to convert a real uint32_t value as instance ID.

Hi

Being more helpful, I think that it is better to just:

* change instance_id to be an uint32_t (notice that for all architectures that we support, it is actually int32_t).

* export calculate_new_instance_id() and adjust callers that use -1.

or

* export a new function that just use the calculate_new_instance_id()

A fast search shows:

10 callers of vmstate_register() with -1
1 caller of vmstate_register_with_alias_id with -1 (but it is the one
that sets all qdev devices).
1 caller of vmstate_register_with_alias_id in apic, where it can be -1. 1 caller of register_savevm_live() with -1 (spapr)

And call it a day?

What do you think, Juan.

Quoted 103 lines
---
 include/migration/register.h |  2 +-
 include/migration/vmstate.h  |  4 ++--
 migration/savevm.c           | 10 +++++-----
 stubs/vmstate.c              |  2 +-
 4 files changed, 9 insertions(+), 9 deletions(-)
include/migration/register.h +1 -1
index a13359a08d..54f42c7413 100644
@@ -69,7 +69,7 @@ typedef struct SaveVMHandlers {
 } SaveVMHandlers;
 
 int register_savevm_live(const char *idstr,
-                         int instance_id,
+                         int64_t instance_id,
                          int version_id,
                          const SaveVMHandlers *ops,
                          void *opaque);
include/migration/vmstate.h +2 -2
index 1fbfd099dd..6a7498463c 100644
@@ -1114,14 +1114,14 @@ int vmstate_save_state_v(QEMUFile *f, const VMStateDescription *vmsd,
 bool vmstate_save_needed(const VMStateDescription *vmsd, void *opaque);
 
 /* Returns: 0 on success, -1 on failure */
-int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
+int vmstate_register_with_alias_id(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *base, int alias_id,
                                    int required_for_version,
                                    Error **errp);
 
 /* Returns: 0 on success, -1 on failure */
-static inline int vmstate_register(DeviceState *dev, int instance_id,
+static inline int vmstate_register(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *opaque)
 {
migration/savevm.c +5 -5
index bb9462a54d..dc9281c897 100644
@@ -233,7 +233,7 @@ typedef struct CompatEntry {
 typedef struct SaveStateEntry {
     QTAILQ_ENTRY(SaveStateEntry) entry;
     char idstr[256];
-    int instance_id;
+    int64_t instance_id;
     int alias_id;
     int version_id;
     /* version id read from the stream */
@@ -668,7 +668,7 @@ void dump_vmstate_json_to_file(FILE *out_file)
 static int calculate_new_instance_id(const char *idstr)
 {
     SaveStateEntry *se;
-    int instance_id = 0;
+    int64_t instance_id = 0;
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         if (strcmp(idstr, se->idstr) == 0
@@ -730,7 +730,7 @@ static void savevm_state_handler_insert(SaveStateEntry *nse)
    Meanwhile pass -1 as instance_id if you do not already have a clearly
    distinguishing id for all instances of your device class. */
 int register_savevm_live(const char *idstr,
-                         int instance_id,
+                         int64_t instance_id,
                          int version_id,
                          const SaveVMHandlers *ops,
                          void *opaque)
@@ -784,7 +784,7 @@ void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque)
     }
 }
 
-int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
+int vmstate_register_with_alias_id(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *opaque, int alias_id,
                                    int required_for_version,
@@ -1566,7 +1566,7 @@ int qemu_save_device_state(QEMUFile *f)
     return qemu_file_get_error(f);
 }
 
-static SaveStateEntry *find_se(const char *idstr, int instance_id)
+static SaveStateEntry *find_se(const char *idstr, int64_t instance_id)
 {
     SaveStateEntry *se;
 
stubs/vmstate.c +1 -1
index e1e89b87f0..699003f3b0 100644
@@ -4,7 +4,7 @@
 const VMStateDescription vmstate_dummy = {};
 
 int vmstate_register_with_alias_id(DeviceState *dev,
-                                   int instance_id,
+                                   int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *base, int alias_id,
                                    int required_for_version,
#5 lore raw

Re: [PATCH 1/2] migration: Boost SaveStateEntry.instance_id to 64 bits

Quoted 16 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Peter Xu

It was "int" and used as 32bits fields (see save_section_header()). It's unsafe already because sizeof(int) could be 2 on i386, I think. So at least uint32_t would suite more. While it also uses "-1" as a placeholder of "we want to generate the instance ID automatically". Hence a more proper value should be int64_t.

This will start to be useful after next patch in which we can start to convert a real uint32_t value as instance ID.

Hi

Being more helpful, I think that it is better to just:

Quoted 8 lines

* change instance_id to be an uint32_t (notice that for all architectures that we support, it is actually int32_t).

* export calculate_new_instance_id() and adjust callers that use -1.

or

* export a new function that just use the calculate_new_instance_id()

Do you mean that we end up with two functions, one that does it automatically, and one that takes an ID?

Dave

Quoted 110 lines

A fast search shows:

10 callers of vmstate_register() with -1
1 caller of vmstate_register_with_alias_id with -1 (but it is the one
that sets all qdev devices).
1 caller of vmstate_register_with_alias_id in apic, where it can be -1. 1 caller of register_savevm_live() with -1 (spapr)

And call it a day?

What do you think, Juan.

Quoted 97 lines
---
 include/migration/register.h |  2 +-
 include/migration/vmstate.h  |  4 ++--
 migration/savevm.c           | 10 +++++-----
 stubs/vmstate.c              |  2 +-
 4 files changed, 9 insertions(+), 9 deletions(-)
include/migration/register.h +1 -1
index a13359a08d..54f42c7413 100644
@@ -69,7 +69,7 @@ typedef struct SaveVMHandlers {
 } SaveVMHandlers;
 
 int register_savevm_live(const char *idstr,
-                         int instance_id,
+                         int64_t instance_id,
                          int version_id,
                          const SaveVMHandlers *ops,
                          void *opaque);
include/migration/vmstate.h +2 -2
index 1fbfd099dd..6a7498463c 100644
@@ -1114,14 +1114,14 @@ int vmstate_save_state_v(QEMUFile *f, const VMStateDescription *vmsd,
 bool vmstate_save_needed(const VMStateDescription *vmsd, void *opaque);
 
 /* Returns: 0 on success, -1 on failure */
-int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
+int vmstate_register_with_alias_id(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *base, int alias_id,
                                    int required_for_version,
                                    Error **errp);
 
 /* Returns: 0 on success, -1 on failure */
-static inline int vmstate_register(DeviceState *dev, int instance_id,
+static inline int vmstate_register(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *opaque)
 {
migration/savevm.c +5 -5
index bb9462a54d..dc9281c897 100644
@@ -233,7 +233,7 @@ typedef struct CompatEntry {
 typedef struct SaveStateEntry {
     QTAILQ_ENTRY(SaveStateEntry) entry;
     char idstr[256];
-    int instance_id;
+    int64_t instance_id;
     int alias_id;
     int version_id;
     /* version id read from the stream */
@@ -668,7 +668,7 @@ void dump_vmstate_json_to_file(FILE *out_file)
 static int calculate_new_instance_id(const char *idstr)
 {
     SaveStateEntry *se;
-    int instance_id = 0;
+    int64_t instance_id = 0;
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         if (strcmp(idstr, se->idstr) == 0
@@ -730,7 +730,7 @@ static void savevm_state_handler_insert(SaveStateEntry *nse)
    Meanwhile pass -1 as instance_id if you do not already have a clearly
    distinguishing id for all instances of your device class. */
 int register_savevm_live(const char *idstr,
-                         int instance_id,
+                         int64_t instance_id,
                          int version_id,
                          const SaveVMHandlers *ops,
                          void *opaque)
@@ -784,7 +784,7 @@ void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque)
     }
 }
 
-int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
+int vmstate_register_with_alias_id(DeviceState *dev, int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *opaque, int alias_id,
                                    int required_for_version,
@@ -1566,7 +1566,7 @@ int qemu_save_device_state(QEMUFile *f)
     return qemu_file_get_error(f);
 }
 
-static SaveStateEntry *find_se(const char *idstr, int instance_id)
+static SaveStateEntry *find_se(const char *idstr, int64_t instance_id)
 {
     SaveStateEntry *se;
 
stubs/vmstate.c +0 -0
index e1e89b87f0..699003f3b0 100644
@@ -4,7 +4,7 @@
 const VMStateDescription vmstate_dummy = {};
 
Quoted 6 lines

stubs/vmstate.c line 10

Quoted 6 lines
stubs/vmstate.c +1 -1
 int vmstate_register_with_alias_id(DeviceState *dev,
-                                   int instance_id,
+                                   int64_t instance_id,
                                    const VMStateDescription *vmsd,
                                    void *base, int alias_id,
                                    int required_for_version,

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#6 lore raw

Re: [PATCH 1/2] migration: Boost SaveStateEntry.instance_id to 64 bits

Quoted 20 lines · Dr. David Alan Gilbert
Quoted 19 lines · * Juan Quintela ([email protected]) wrote:
Quoted 10 lines · Peter Xu

It was "int" and used as 32bits fields (see save_section_header()). It's unsafe already because sizeof(int) could be 2 on i386, I think. So at least uint32_t would suite more. While it also uses "-1" as a placeholder of "we want to generate the instance ID automatically". Hence a more proper value should be int64_t.

This will start to be useful after next patch in which we can start to convert a real uint32_t value as instance ID.

Hi

Being more helpful, I think that it is better to just:

* change instance_id to be an uint32_t (notice that for all architectures that we support, it is actually int32_t).

Quoted 8 lines
Quoted 5 lines

* export calculate_new_instance_id() and adjust callers that use -1.

or

* export a new function that just use the calculate_new_instance_id()

Do you mean that we end up with two functions, one that does it automatically, and one that takes an ID?

That is one option.

The other is that we export calculate_new_instance_id(), and we use that instead of -1.

Later, Juan.

author
#7 lore raw

Re: [PATCH 1/2] migration: Boost SaveStateEntry.instance_id to 64 bits

In reply to Juan Quintela
Quoted 33 lines · Juan Quintela
Quoted 10 lines · Peter Xu

It was "int" and used as 32bits fields (see save_section_header()). It's unsafe already because sizeof(int) could be 2 on i386, I think. So at least uint32_t would suite more. While it also uses "-1" as a placeholder of "we want to generate the instance ID automatically". Hence a more proper value should be int64_t.

This will start to be useful after next patch in which we can start to convert a real uint32_t value as instance ID.

Hi

Being more helpful, I think that it is better to just:

* change instance_id to be an uint32_t (notice that for all architectures that we support, it is actually int32_t).

* export calculate_new_instance_id() and adjust callers that use -1.

or

* export a new function that just use the calculate_new_instance_id()

A fast search shows:

10 callers of vmstate_register() with -1
1 caller of vmstate_register_with_alias_id with -1 (but it is the one
that sets all qdev devices).
1 caller of vmstate_register_with_alias_id in apic, where it can be -1. 1 caller of register_savevm_live() with -1 (spapr)

Quoted 3 lines

And call it a day?

What do you think, Juan.

Sure, I can switch instance_id to uint32_t and add a new flag to both functions (register_savevm_live, vmstate_register_with_alias_id)

Regards,

Signature · 1 line
Peter Xu
author
#8 lore raw

PATCH 2/2 apic: Use 32bit APIC ID for migration instance ID

In reply to Peter Xu

Migration is silently broken now with x2apic config like this:

     -smp 200,maxcpus=288,sockets=2,cores=72,threads=2 \
     -device intel-iommu,intremap=on,eim=on

After migration, the guest kernel could hang at anything, due to x2apic bit not migrated correctly in IA32_APIC_BASE on some vcpus, so any operations related to x2apic could be broken then (e.g., RDMSR on x2apic MSRs could fail because KVM would think that the vcpu hasn't enabled x2apic at all).

The issue is that the x2apic bit was never applied correctly for vcpus whose ID > 255 when migrate completes, and that's because when we migrate APIC we use the APICCommonState.id as instance ID of the migration stream, while that's too short for x2apic.

Let's use the newly introduced initial_apic_id for that.

---
 hw/intc/apic_common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/intc/apic_common.c +1 -1
index aafd8e0e33..6024a3e06a 100644
@@ -315,7 +315,7 @@ static void apic_common_realize(DeviceState *dev, Error **errp)
     APICCommonState *s = APIC_COMMON(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
-    int instance_id = s->id;
+    int64_t instance_id = s->initial_apic_id;
 
     info = APIC_COMMON_GET_CLASS(s);
     info->realize(dev, errp);
Signature · 1 line
2.21.0
#9 lore raw

Re: [PATCH 2/2] apic: Use 32bit APIC ID for migration instance ID

Quoted 28 lines · Peter Xu

Migration is silently broken now with x2apic config like this:

     -smp 200,maxcpus=288,sockets=2,cores=72,threads=2 \
     -device intel-iommu,intremap=on,eim=on

After migration, the guest kernel could hang at anything, due to x2apic bit not migrated correctly in IA32_APIC_BASE on some vcpus, so any operations related to x2apic could be broken then (e.g., RDMSR on x2apic MSRs could fail because KVM would think that the vcpu hasn't enabled x2apic at all).

The issue is that the x2apic bit was never applied correctly for vcpus whose ID > 255 when migrate completes, and that's because when we migrate APIC we use the APICCommonState.id as instance ID of the migration stream, while that's too short for x2apic.

Let's use the newly introduced initial_apic_id for that.

---
 hw/intc/apic_common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/intc/apic_common.c +0 -0
index aafd8e0e33..6024a3e06a 100644

hw/intc/apic_common.c static void apic_common_realize(DeviceState *dev, Error **errp) · line 318

Quoted 5 lines
hw/intc/apic_common.c +1 -1
@@ -315,7 +315,7 @@ static void apic_common_realize(DeviceState *dev, Error **errp)
     APICCommonState *s = APIC_COMMON(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
-    int instance_id = s->id;
+    int64_t instance_id = s->initial_apic_id;

int is ok here.

But damn thing, initial_apic_id is uint32_t. Sniff.

Later, Juan.

#10 lore raw

Re: [PATCH 2/2] apic: Use 32bit APIC ID for migration instance ID

In reply to Peter Xu
Quoted 11 lines · * Peter Xu ([email protected]) wrote:

Migration is silently broken now with x2apic config like this:

     -smp 200,maxcpus=288,sockets=2,cores=72,threads=2 \
     -device intel-iommu,intremap=on,eim=on

After migration, the guest kernel could hang at anything, due to x2apic bit not migrated correctly in IA32_APIC_BASE on some vcpus, so any operations related to x2apic could be broken then (e.g., RDMSR on x2apic MSRs could fail because KVM would think that the vcpu hasn't enabled x2apic at all).

Quoted 6 lines

The issue is that the x2apic bit was never applied correctly for vcpus whose ID > 255 when migrate completes, and that's because when we migrate APIC we use the APICCommonState.id as instance ID of the migration stream, while that's too short for x2apic.

Let's use the newly introduced initial_apic_id for that.

I'd like to understand a few things:
a) Does this change the instance ID of existing APICs on the
migration stream?

     a1) Ever for <256 CPUs?
     a2) For >=256 CPUs?

    [Because changing the ID breaks migration]

b) Is the instance ID constant - I can see it's a property on the APIC, but I cna't see who sets it

c) In the case where it fails, did we end up registering two devices with the same name and instance ID? If so, is it worth adding a check that would error if we tried?

Dave

Quoted 22 lines
---
 hw/intc/apic_common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/intc/apic_common.c +1 -1
index aafd8e0e33..6024a3e06a 100644
@@ -315,7 +315,7 @@ static void apic_common_realize(DeviceState *dev, Error **errp)
     APICCommonState *s = APIC_COMMON(dev);
     APICCommonClass *info;
     static DeviceState *vapic;
-    int instance_id = s->id;
+    int64_t instance_id = s->initial_apic_id;
 
     info = APIC_COMMON_GET_CLASS(s);
     info->realize(dev, errp);
Signature · 1 line
2.21.0

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

author
#11 lore raw

Re: [PATCH 2/2] apic: Use 32bit APIC ID for migration instance ID

Quoted 17 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Peter Xu ([email protected]) wrote:

Migration is silently broken now with x2apic config like this:

     -smp 200,maxcpus=288,sockets=2,cores=72,threads=2 \
     -device intel-iommu,intremap=on,eim=on

After migration, the guest kernel could hang at anything, due to x2apic bit not migrated correctly in IA32_APIC_BASE on some vcpus, so any operations related to x2apic could be broken then (e.g., RDMSR on x2apic MSRs could fail because KVM would think that the vcpu hasn't enabled x2apic at all).

The issue is that the x2apic bit was never applied correctly for vcpus whose ID > 255 when migrate completes, and that's because when we migrate APIC we use the APICCommonState.id as instance ID of the migration stream, while that's too short for x2apic.

Quoted 6 lines
Quoted 1 line

Let's use the newly introduced initial_apic_id for that.

I'd like to understand a few things:
a) Does this change the instance ID of existing APICs on the
migration stream?

     a1) Ever for <256 CPUs?

No.

Quoted 1 line
     a2) For >=256 CPUs?

Yes.

Quoted 2 lines

[Because changing the ID breaks migration]

But if we don't change it, the stream is broken too. :)

Then the destination VM will receive e.g. two apic_id==0 instances (I think the apic_id==256 instance will wrongly overwrite the apic_id==0 one), while the vcpu with apic_id==256 will use the initial apic values.

So IMHO we should still fix this, even if it changes the migration stream. At least we start to make it right.

Quoted 3 lines

b) Is the instance ID constant - I can see it's a property on the APIC, but I cna't see who sets it

For each vcpu, I think yes it should be a constant as long as the topology is the same. This is how I understand it to be set:

(1) In pc_cpus_init(), we init these:

    possible_cpus = mc->possible_cpu_arch_ids(ms);
    for (i = 0; i < ms->smp.cpus; i++) {
        pc_new_cpu(pcms, possible_cpus->cpus[i].arch_id, &error_fatal);
    }

(2) In x86_cpu_apic_create(), we apply the apic_id to "id" property:

    qdev_prop_set_uint32(cpu->apic_state, "id", cpu->apic_id);
Quoted 4 lines

c) In the case where it fails, did we end up registering two devices with the same name and instance ID? If so, is it worth adding a check that would error if we tried?

Sounds doable.

Thanks,

Signature · 1 line
Peter Xu
#12 lore raw

Re: [PATCH 2/2] apic: Use 32bit APIC ID for migration instance ID

Quoted 36 lines · * Peter Xu ([email protected]) wrote:
Quoted 17 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Peter Xu ([email protected]) wrote:

Migration is silently broken now with x2apic config like this:

     -smp 200,maxcpus=288,sockets=2,cores=72,threads=2 \
     -device intel-iommu,intremap=on,eim=on

After migration, the guest kernel could hang at anything, due to x2apic bit not migrated correctly in IA32_APIC_BASE on some vcpus, so any operations related to x2apic could be broken then (e.g., RDMSR on x2apic MSRs could fail because KVM would think that the vcpu hasn't enabled x2apic at all).

The issue is that the x2apic bit was never applied correctly for vcpus whose ID > 255 when migrate completes, and that's because when we migrate APIC we use the APICCommonState.id as instance ID of the migration stream, while that's too short for x2apic.

Quoted 6 lines
Quoted 1 line

Let's use the newly introduced initial_apic_id for that.

I'd like to understand a few things:
a) Does this change the instance ID of existing APICs on the
migration stream?

     a1) Ever for <256 CPUs?

No.

Quoted 1 line
     a2) For >=256 CPUs?

Yes.

Quoted 2 lines

[Because changing the ID breaks migration]

But if we don't change it, the stream is broken too. :)

Quoted 7 lines

Then the destination VM will receive e.g. two apic_id==0 instances (I think the apic_id==256 instance will wrongly overwrite the apic_id==0 one), while the vcpu with apic_id==256 will use the initial apic values.

So IMHO we should still fix this, even if it changes the migration stream. At least we start to make it right.

Yes, that makes sense.
It deserves a doc mention somewhere.

Quoted 9 lines
Quoted 3 lines

b) Is the instance ID constant - I can see it's a property on the APIC, but I cna't see who sets it

For each vcpu, I think yes it should be a constant as long as the topology is the same. This is how I understand it to be set:

(1) In pc_cpus_init(), we init these:

Quoted 8 lines
    possible_cpus = mc->possible_cpu_arch_ids(ms);
    for (i = 0; i < ms->smp.cpus; i++) {
        pc_new_cpu(pcms, possible_cpus->cpus[i].arch_id, &error_fatal);
    }

(2) In x86_cpu_apic_create(), we apply the apic_id to "id" property:

    qdev_prop_set_uint32(cpu->apic_state, "id", cpu->apic_id);

OK, that's fine - as long as it's constaatn and not guest influenced.

Quoted 7 lines
Quoted 4 lines

c) In the case where it fails, did we end up registering two devices with the same name and instance ID? If so, is it worth adding a check that would error if we tried?

Sounds doable.

Great,

Dave

Quoted 4 lines

Thanks,

Signature · 1 line
Peter Xu

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#13 lore raw

Re: [PATCH 2/2] apic: Use 32bit APIC ID for migration instance ID

Quoted 62 lines · Dr. David Alan Gilbert
Quoted 36 lines · * Peter Xu ([email protected]) wrote:
Quoted 17 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Peter Xu ([email protected]) wrote:

Migration is silently broken now with x2apic config like this:

     -smp 200,maxcpus=288,sockets=2,cores=72,threads=2 \
     -device intel-iommu,intremap=on,eim=on

After migration, the guest kernel could hang at anything, due to x2apic bit not migrated correctly in IA32_APIC_BASE on some vcpus, so any operations related to x2apic could be broken then (e.g., RDMSR on x2apic MSRs could fail because KVM would think that the vcpu hasn't enabled x2apic at all).

The issue is that the x2apic bit was never applied correctly for vcpus whose ID > 255 when migrate completes, and that's because when we migrate APIC we use the APICCommonState.id as instance ID of the migration stream, while that's too short for x2apic.

Quoted 6 lines
Quoted 1 line

Let's use the newly introduced initial_apic_id for that.

I'd like to understand a few things:
a) Does this change the instance ID of existing APICs on the
migration stream?

     a1) Ever for <256 CPUs?

No.

Quoted 1 line
     a2) For >=256 CPUs?

Yes.

Quoted 2 lines

[Because changing the ID breaks migration]

But if we don't change it, the stream is broken too. :)

Quoted 7 lines

Then the destination VM will receive e.g. two apic_id==0 instances (I think the apic_id==256 instance will wrongly overwrite the apic_id==0 one), while the vcpu with apic_id==256 will use the initial apic values.

So IMHO we should still fix this, even if it changes the migration stream. At least we start to make it right.

Yes, that makes sense.
It deserves a doc mention somewhere.

Quoted 14 lines
Quoted 3 lines

b) Is the instance ID constant - I can see it's a property on the APIC, but I cna't see who sets it

For each vcpu, I think yes it should be a constant as long as the topology is the same. This is how I understand it to be set:

(1) In pc_cpus_init(), we init these:

    possible_cpus = mc->possible_cpu_arch_ids(ms);
    for (i = 0; i < ms->smp.cpus; i++) {
        pc_new_cpu(pcms, possible_cpus->cpus[i].arch_id, &error_fatal);
    }
Quoted 5 lines
Quoted 3 lines

(2) In x86_cpu_apic_create(), we apply the apic_id to "id" property:

    qdev_prop_set_uint32(cpu->apic_state, "id", cpu->apic_id);

OK, that's fine - as long as it's constaatn and not guest influenced.

The guest may change the CPU APIC ID (although they rarely do), but I believe X86CPU::apic_id is always going to be the initial APIC ID. I'll double check (and maybe send a patch to rename it to initial_apic_id).

Signature · 1 line
Eduardo