Skip to contentSkip to conversation

tools: guest kexec fixes

PATCH Superseded Jason Andryuk 2 participants 5 messages Jan 21, 2023 – Jan 23, 2023

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#1 lore raw

PATCH 1/2 libxl: Fix guest kexec - skip cpuid policy

In reply to Jason Andryuk

When a domain performs a kexec (soft reset), libxl__build_pre() is called with the existing domid. Calling libxl__cpuid_legacy() on the existing domain fails since the cpuid policy has already been set, and the guest isn't rebuilt and doesn't kexec.

xc: error: Failed to set d1's policy (err leaf 0xffffffff, subleaf 0xffffffff, msr 0xffffffff) (17 = File exists): Internal error
libxl: error: libxl_cpuid.c:494:libxl__cpuid_legacy: Domain 1:Failed to apply CPUID policy: File exists
libxl: error: libxl_create.c:1641:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
libxl: error: libxl_xshelp.c:201:libxl__xs_read_mandatory: xenstore read failed: `/libxl/1/type': No such file or directory
libxl: warning: libxl_dom.c:49:libxl__domain_type: unable to get domain type for domid=1, assuming HVM

During a soft_reset, skip calling libxl__cpuid_legacy() to avoid the issue. Before the fixes commit, the libxl__cpuid_legacy() failure would have been ignored, so kexec would continue.

Fixes 34990446ca91 "libxl: don't ignore the return value from xc_cpuid_apply_policy"

---
Probably a backport candidate since this has been broken for a while.

---
 tools/libs/light/libxl_create.c   | 4 ++--
 tools/libs/light/libxl_dom.c      | 5 +++--
 tools/libs/light/libxl_internal.h | 2 +-
 3 files changed, 6 insertions(+), 5 deletions(-)
tools/libs/light/libxl_create.c +2 -2
index 5cddc3df79..587a515dff 100644
@@ -510,7 +510,7 @@ int libxl__domain_build(libxl__gc *gc,
     struct timeval start_time;
     int i, ret;
 
-    ret = libxl__build_pre(gc, domid, d_config, state);
+    ret = libxl__build_pre(gc, domid, d_config, state, false);
     if (ret)
         goto out;
 
@@ -1440,7 +1440,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
         goto out;
     }
 
-    rc = libxl__build_pre(gc, domid, d_config, state);
+    rc = libxl__build_pre(gc, domid, d_config, state, dcs->soft_reset);
     if (rc)
         goto out;
 
tools/libs/light/libxl_dom.c +3 -2
index b454f988fb..7cebf5047f 100644
@@ -241,7 +241,8 @@ static int numa_place_domain(libxl__gc *gc, uint32_t domid,
 }
 
 int libxl__build_pre(libxl__gc *gc, uint32_t domid,
-              libxl_domain_config *d_config, libxl__domain_build_state *state)
+              libxl_domain_config *d_config, libxl__domain_build_state *state,
+              bool soft_reset)
 {
     libxl_domain_build_info *const info = &d_config->b_info;
     libxl_ctx *ctx = libxl__gc_owner(gc);
@@ -382,7 +383,7 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid,
     /* Construct a CPUID policy, but only for brand new domains.  Domains
      * being migrated-in/restored have CPUID handled during the
      * static_data_done() callback. */
-    if (!state->restore)
+    if (!state->restore && !soft_reset)
         rc = libxl__cpuid_legacy(ctx, domid, false, info);
 
 out:
tools/libs/light/libxl_internal.h +1 -1
index 0dc8b8f210..f0af44b523 100644
@@ -1418,7 +1418,7 @@ _hidden void libxl__domain_build_state_dispose(libxl__domain_build_state *s);
 
 _hidden int libxl__build_pre(libxl__gc *gc, uint32_t domid,
               libxl_domain_config * const d_config,
-              libxl__domain_build_state *state);
+              libxl__domain_build_state *state, bool soft_reset);
 _hidden int libxl__build_post(libxl__gc *gc, uint32_t domid,
                libxl_domain_build_info *info, libxl__domain_build_state *state,
                char **vms_ents, char **local_ents);
Signature · 1 line
2.34.1
#0 lore raw

cover tools: guest kexec fixes

Two toolstack fixes for guest kexec. This restored kexec enough that I got Debian bullseye to kexec once. Trying to kexec the guest a second time had it spinning at 100% CPU after initiating the kexec. Haven't looked into that part yet.

Regards,
Jason

Jason Andryuk (2):
libxl: Fix guest kexec - skip cpuid policy
Revert "tools/xenstore: simplify loop handling connection I/O"

 tools/libs/light/libxl_create.c   |  4 ++--
 tools/libs/light/libxl_dom.c      |  5 +++--
 tools/libs/light/libxl_internal.h |  2 +-
 tools/xenstore/xenstored_core.c   | 19 +++++++++++++++++--
 4 files changed, 23 insertions(+), 7 deletions(-)
Signature · 1 line
2.34.1
#3 lore raw

PATCH 2/2 Revert "tools/xenstore: simplify loop handling connection I/O"

I'm observing guest kexec trigger xenstored to abort on a double free.

gdb output:
Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140645614258112) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt

    at ./nptl/pthread_kill.c:44
    at ./nptl/pthread_kill.c:78
    at ./nptl/pthread_kill.c:89
    at ../sysdeps/posix/raise.c:26
    at talloc.c:119
    ptr=ptr@entry=0x559fae724290) at talloc.c:232
    at xenstored_core.c:2945
(gdb) frame 5
    at talloc.c:119
119            TALLOC_ABORT("Bad talloc magic value - double free");
(gdb) frame 7
    at xenstored_core.c:2945

2945 talloc_increase_ref_count(conn);
(gdb) p conn
$1 = (struct connection *) 0x559fae724290

Looking at a xenstore trace, we have:
IN 0x559fae71f250 20230120 17:40:53 READ (/local/domain/3/image/device-model-dom id )
wrl: dom 0 1 msec 10000 credit 1000000 reserve 100 disc
ard
wrl: dom 3 1 msec 10000 credit 1000000 reserve 100 disc
ard
wrl: dom 0 0 msec 10000 credit 1000000 reserve 0 disc
ard
wrl: dom 3 0 msec 10000 credit 1000000 reserve 0 disc
ard
OUT 0x559fae71f250 20230120 17:40:53 ERROR (ENOENT )
wrl: dom 0 1 msec 10000 credit 1000000 reserve 100 disc
ard
wrl: dom 3 1 msec 10000 credit 1000000 reserve 100 disc
ard
IN 0x559fae71f250 20230120 17:40:53 RELEASE (3 )
DESTROY watch 0x559fae73f630
DESTROY watch 0x559fae75ddf0
DESTROY watch 0x559fae75ec30
DESTROY watch 0x559fae75ea60
DESTROY watch 0x559fae732c00
DESTROY watch 0x559fae72cea0
DESTROY watch 0x559fae728fc0
DESTROY watch 0x559fae729570
DESTROY connection 0x559fae724290
orphaned node /local/domain/3/device/suspend/event-channel deleted orphaned node /local/domain/3/device/vbd/51712 deleted
orphaned node /local/domain/3/device/vkbd/0 deleted
orphaned node /local/domain/3/device/vif/0 deleted
orphaned node /local/domain/3/control/shutdown deleted
orphaned node /local/domain/3/control/feature-poweroff deleted orphaned node /local/domain/3/control/feature-reboot deleted orphaned node /local/domain/3/control/feature-suspend deleted orphaned node /local/domain/3/control/feature-s3 deleted
orphaned node /local/domain/3/control/feature-s4 deleted
orphaned node /local/domain/3/control/sysrq deleted
orphaned node /local/domain/3/data deleted
orphaned node /local/domain/3/drivers deleted
orphaned node /local/domain/3/feature deleted
orphaned node /local/domain/3/attr deleted
orphaned node /local/domain/3/error deleted
orphaned node /local/domain/3/console/backend-id deleted

and no further output.

The trace shows that DESTROY was called for connection 0x559fae724290, but that is the same pointer (conn) main() was looping through from connections. So it wasn't actually removed from the connections list?

Reverting commit e8e6e42279a5 "tools/xenstore: simplify loop handling connection I/O" fixes the abort/double free. I think the use of list_for_each_entry_safe is incorrect. list_for_each_entry_safe makes traversal safe for deleting the current iterator, but RELEASE/do_release will delete some other entry in the connections list. I think the observed abort is because list_for_each_entry has next pointing to the deleted connection, and it is used in the subsequent iteration.

Add a comment explaining the unsuitability of list_for_each_entry_safe. Also notice that the old code takes a reference on next which would prevents a use-after-free.

This reverts commit e8e6e42279a5723239c5c40ba4c7f579a979465d.

---
I didn't verify the stale pointers, which is why there are a lot of "I think" qualifiers. But reverting the commit has xenstored still running whereas it was aborting consistently beforehand.

---
 tools/xenstore/xenstored_core.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
tools/xenstore/xenstored_core.c +17 -2
index 78a3edaa4e..029e3852fc 100644
@@ -2941,8 +2941,23 @@ int main(int argc, char *argv[])
 			}
 		}
 
-		list_for_each_entry_safe(conn, next, &connections, list) {
-			talloc_increase_ref_count(conn);
+		/*
+		 * list_for_each_entry_safe is not suitable here because
+		 * handle_input may delete entries besides the current one, but
+		 * those may be in the temporary next which would trigger a
+		 * use-after-free.  list_for_each_entry_safe is only safe for
+		 * deleting the current entry.
+		 */
+		next = list_entry(connections.next, typeof(*conn), list);
+		if (&next->list != &connections)
+			talloc_increase_ref_count(next);
+		while (&next->list != &connections) {
+			conn = next;
+
+			next = list_entry(conn->list.next,
+					  typeof(*conn), list);
+			if (&next->list != &connections)
+				talloc_increase_ref_count(next);
 
 			if (conn_can_read(conn))
 				handle_input(conn);
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 1/2] libxl: Fix guest kexec - skip cpuid policy

In reply to Jason Andryuk
Quoted 30 lines · Jason Andryuk

When a domain performs a kexec (soft reset), libxl__build_pre() is called with the existing domid. Calling libxl__cpuid_legacy() on the existing domain fails since the cpuid policy has already been set, and the guest isn't rebuilt and doesn't kexec.

xc: error: Failed to set d1's policy (err leaf 0xffffffff, subleaf 0xffffffff, msr 0xffffffff) (17 = File exists): Internal error
libxl: error: libxl_cpuid.c:494:libxl__cpuid_legacy: Domain 1:Failed to apply CPUID policy: File exists
libxl: error: libxl_create.c:1641:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
libxl: error: libxl_xshelp.c:201:libxl__xs_read_mandatory: xenstore read failed: `/libxl/1/type': No such file or directory
libxl: warning: libxl_dom.c:49:libxl__domain_type: unable to get domain type for domid=1, assuming HVM

During a soft_reset, skip calling libxl__cpuid_legacy() to avoid the issue. Before the fixes commit, the libxl__cpuid_legacy() failure would have been ignored, so kexec would continue.

Fixes 34990446ca91 "libxl: don't ignore the return value from xc_cpuid_apply_policy"

---
Probably a backport candidate since this has been broken for a while.

---
  tools/libs/light/libxl_create.c   | 4 ++--
  tools/libs/light/libxl_dom.c      | 5 +++--
  tools/libs/light/libxl_internal.h | 2 +-
  3 files changed, 6 insertions(+), 5 deletions(-)
tools/libs/light/libxl_create.c +0 -0
index 5cddc3df79..587a515dff 100644

tools/libs/light/libxl_create.c int libxl__domain_build(libxl__gc *gc, · line 513

Quoted 5 lines
tools/libs/light/libxl_create.c +1 -1
@@ -510,7 +510,7 @@ int libxl__domain_build(libxl__gc *gc,
      struct timeval start_time;
      int i, ret;
 
-    ret = libxl__build_pre(gc, domid, d_config, state);
+    ret = libxl__build_pre(gc, domid, d_config, state, false);

Instead of adding a parameter to libxl__build_pre() I'd rather add another bool "soft_reset" to libxl__domain_build_state.

This would be more similar to the libxl__domain_build_state->restore use case.

Juergen

Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] Revert "tools/xenstore: simplify loop handling connection I/O"

In reply to Jason Andryuk
Quoted 81 lines · Jason Andryuk

I'm observing guest kexec trigger xenstored to abort on a double free.

gdb output:
Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140645614258112) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt

     at ./nptl/pthread_kill.c:44
     at ./nptl/pthread_kill.c:78
     at ./nptl/pthread_kill.c:89
     at ../sysdeps/posix/raise.c:26
     at talloc.c:119
     ptr=ptr@entry=0x559fae724290) at talloc.c:232
     at xenstored_core.c:2945
(gdb) frame 5
     at talloc.c:119
119            TALLOC_ABORT("Bad talloc magic value - double free");
(gdb) frame 7
     at xenstored_core.c:2945

2945 talloc_increase_ref_count(conn);
(gdb) p conn
$1 = (struct connection *) 0x559fae724290

Looking at a xenstore trace, we have:
IN 0x559fae71f250 20230120 17:40:53 READ (/local/domain/3/image/device-model-dom id )
wrl: dom 0 1 msec 10000 credit 1000000 reserve 100 disc
ard
wrl: dom 3 1 msec 10000 credit 1000000 reserve 100 disc
ard
wrl: dom 0 0 msec 10000 credit 1000000 reserve 0 disc
ard
wrl: dom 3 0 msec 10000 credit 1000000 reserve 0 disc
ard
OUT 0x559fae71f250 20230120 17:40:53 ERROR (ENOENT )
wrl: dom 0 1 msec 10000 credit 1000000 reserve 100 disc
ard
wrl: dom 3 1 msec 10000 credit 1000000 reserve 100 disc
ard
IN 0x559fae71f250 20230120 17:40:53 RELEASE (3 )
DESTROY watch 0x559fae73f630
DESTROY watch 0x559fae75ddf0
DESTROY watch 0x559fae75ec30
DESTROY watch 0x559fae75ea60
DESTROY watch 0x559fae732c00
DESTROY watch 0x559fae72cea0
DESTROY watch 0x559fae728fc0
DESTROY watch 0x559fae729570
DESTROY connection 0x559fae724290
orphaned node /local/domain/3/device/suspend/event-channel deleted orphaned node /local/domain/3/device/vbd/51712 deleted
orphaned node /local/domain/3/device/vkbd/0 deleted
orphaned node /local/domain/3/device/vif/0 deleted
orphaned node /local/domain/3/control/shutdown deleted
orphaned node /local/domain/3/control/feature-poweroff deleted orphaned node /local/domain/3/control/feature-reboot deleted orphaned node /local/domain/3/control/feature-suspend deleted orphaned node /local/domain/3/control/feature-s3 deleted
orphaned node /local/domain/3/control/feature-s4 deleted
orphaned node /local/domain/3/control/sysrq deleted
orphaned node /local/domain/3/data deleted
orphaned node /local/domain/3/drivers deleted
orphaned node /local/domain/3/feature deleted
orphaned node /local/domain/3/attr deleted
orphaned node /local/domain/3/error deleted
orphaned node /local/domain/3/console/backend-id deleted

and no further output.

The trace shows that DESTROY was called for connection 0x559fae724290, but that is the same pointer (conn) main() was looping through from connections. So it wasn't actually removed from the connections list?

Reverting commit e8e6e42279a5 "tools/xenstore: simplify loop handling connection I/O" fixes the abort/double free. I think the use of list_for_each_entry_safe is incorrect. list_for_each_entry_safe makes traversal safe for deleting the current iterator, but RELEASE/do_release will delete some other entry in the connections list. I think the observed abort is because list_for_each_entry has next pointing to the deleted connection, and it is used in the subsequent iteration.

Quoted 7 lines

Add a comment explaining the unsuitability of list_for_each_entry_safe. Also notice that the old code takes a reference on next which would prevents a use-after-free.

This reverts commit e8e6e42279a5723239c5c40ba4c7f579a979465d.

Good catch!

with one nit: a "Fixes:" tag for commit e8e6e42279a5 should be added.

Quoted 4 lines

---
I didn't verify the stale pointers, which is why there are a lot of "I think" qualifiers. But reverting the commit has xenstored still running whereas it was aborting consistently beforehand.

Your analysis seems to be fine. Soft reset handling includes a "XS_RELEASE" message for the affected guest, which results in the struct domain and the associated connection to be freed. This can happen to be the connection in the "next" pointer, resulting in the crash you've observed.

Juergen