tools/xenstore: don't store domU's mfn of ring page in xensotred
PATCH 1/1 tools/xenstore: don't store domU's mfn of ring page in xensotred
The XS_INTRODUCE command has two parameters: the mfn (or better: gfn) of the domain's xenstore ring page and the event channel of the domain for communicating with Xenstore.
The gfn is not really needed. It is stored in the per-domain struct in xenstored and in case of another XS_INTRODUCE for the domain it is tested to match the original value. If it doesn't match the command is aborted via EINVAL.
Today there shouldn't be multiple XS_INTRODUCE requests for the same domain issued, so the mfn/gfn can just be ignored and multiple XS_INTRODUCE commands can be rejected without testing the mfn/gfn.
--- tools/xenstore/xenstored_domain.c | 47 ++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 28 deletions(-)
tools/xenstore/xenstored_domain.c +19 -28
@@ -369,7 +369,6 @@ int do_introduce(struct connection *conn, struct buffered_data *in)struct domain *domain;char *vec[3 ];unsigned int domid; -unsigned long mfn;evtchn_port_t port;int rc;struct xenstore_domain_interface *interface;
@@ -381,7 +380,7 @@ int do_introduce(struct connection *conn, struct buffered_data *in)return EACCES ; domid =atoi (vec[0 ]); - mfn =atol (vec[1 ]); +/* Ignore the mfn, we don't need it. */ port =atoi (vec[2 ]);/* Sanity check args. */
@@ -390,34 +389,26 @@ int do_introduce(struct connection *conn, struct buffered_data *in) domain =find_domain_by_domid (domid); -if (domain ==NULL ) { - interface =map_interface (domid); -if (!interface) -return errno; -/* Hang domain off "in" until we're finished. */ - domain =new_domain (in, domid, port); -if (!domain) { - rc = errno; -unmap_interface (interface); -return rc; - } - domain->interface = interface; - domain->mfn = mfn; - -/* Now domain belongs to its connection. */ -talloc_steal (domain->conn, domain); - -fire_watches (NULL , in,"@introduceDomain" , false); - }else if ((domain->mfn == mfn) && (domain->conn != conn)) { -/* Use XS_INTRODUCE for recreating the xenbus event-channel. */ -if (domain->port) -xenevtchn_unbind (xce_handle, domain->port); - rc =xenevtchn_bind_interdomain (xce_handle, domid, port); - domain->port = (rc ==-1 ) ?0 : rc; - domain->remote_port = port; - }else +if (domain)return EINVAL ; + interface =map_interface (domid); +if (!interface) +return errno; +/* Hang domain off "in" until we're finished. */ + domain =new_domain (in, domid, port); +if (!domain) { + rc = errno; +unmap_interface (interface); +return rc; + } + domain->interface = interface; + +/* Now domain belongs to its connection. */ +talloc_steal (domain->conn, domain); + +fire_watches (NULL , in,"@introduceDomain" , false); +domain_conn_reset (domain);send_ack (conn,XS_INTRODUCE );
Signature · 1 line
2.16.4
Quoted 9 lines · Juergen Gross
The
XS_INTRODUCEcommand has two parameters: the mfn (or better: gfn) of the domain's xenstore ring page and the event channel of the domain for communicating with Xenstore.The gfn is not really needed. It is stored in the per-domain struct in xenstored and in case of another
XS_INTRODUCEfor the domain it is tested to match the original value. If it doesn't match the command is aborted via EINVAL.
Quoted 5 lines
Today there shouldn't be multiple
XS_INTRODUCErequests for the same domain issued, so the mfn/gfn can just be ignored and multipleXS_INTRODUCEcommands can be rejected without testing the mfn/gfn.Signed-off-by Juergen Gross <[email protected]>
In hindsight, this is cleanup following c/s 38eeb3864d "tools/xenstored:
Drop mapping of the ring via foreign map".
Hi Juergen,
Quoted 5 lines · Juergen Gross
The
XS_INTRODUCEcommand has two parameters: the mfn (or better: gfn) of the domain's xenstore ring page and the event channel of the domain for communicating with Xenstore.
Quoted 8 lines
The gfn is not really needed. It is stored in the per-domain struct in xenstored and in case of another
XS_INTRODUCEfor the domain it is tested to match the original value. If it doesn't match the command is aborted via EINVAL.Today there shouldn't be multiple
XS_INTRODUCErequests for the same domain issued, so the mfn/gfn can just be ignored and multipleXS_INTRODUCEcommands can be rejected without testing the mfn/gfn.
So there is a comment in the else part:
/* Use XS_INTRODUCE for recreating the xenbus event-channel. */
From the commit message this is not entirely clear why we want to prevent recreating the event-channel. Can you expand it?
Quoted 19 lines
Signed-off-by Juergen Gross <[email protected]>--- tools/xenstore/xenstored_domain.c | 47 ++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 28 deletions(-)tools/xenstore/xenstored_domain.c +0
-1@@ -369,7 +369,6 @@ int do_introduce(struct connection *conn, struct buffered_data *in)struct domain *domain;char *vec[3 ];unsigned int domid; -unsigned long mfn;evtchn_port_t port;int rc;struct xenstore_domain_interface *interface;
tools/xenstore/xenstored_domain.c int do_introduce(struct connection *conn, struct buffered_data *in) · line 383
Quoted 5 lines
tools/xenstore/xenstored_domain.c +1
-1@@ -381,7 +380,7 @@ int do_introduce(struct connection *conn, struct buffered_data *in)return EACCES ; domid =atoi (vec[0 ]); - mfn =atol (vec[1 ]); +/* Ignore the mfn, we don't need it. */
s/mfn/GFN/
Quoted 10 lines
tools/xenstore/xenstored_domain.c +0
-3port =atoi (vec[2 ]);/* Sanity check args. */ @@ -390,34 +389,26 @@ int do_introduce(struct connection *conn, struct buffered_data *in) domain =find_domain_by_domid (domid); -if (domain ==NULL ) { - interface =map_interface (domid); -if (!interface)
tools/xenstore/xenstored_domain.c int do_introduce(struct connection *conn, struct buffered_data *in) · line 405
Quoted 10 lines
tools/xenstore/xenstored_domain.c +0
-10-return errno; -/* Hang domain off "in" until we're finished. */ - domain =new_domain (in, domid, port); -if (!domain) { - rc = errno; -unmap_interface (interface); -return rc; - } - domain->interface = interface; - domain->mfn = mfn;
AFAICT domain->mfn is not used anymore, so could we remove the field?
Cheers,
Quoted 17 lines · Julien Grall
Hi Juergen,
Quoted 5 lines · Juergen Gross
The
XS_INTRODUCEcommand has two parameters: the mfn (or better: gfn) of the domain's xenstore ring page and the event channel of the domain for communicating with Xenstore.Quoted 8 lines
The gfn is not really needed. It is stored in the per-domain struct in xenstored and in case of another
XS_INTRODUCEfor the domain it is tested to match the original value. If it doesn't match the command is aborted via EINVAL.Today there shouldn't be multiple
XS_INTRODUCErequests for the same domain issued, so the mfn/gfn can just be ignored and multipleXS_INTRODUCEcommands can be rejected without testing the mfn/gfn.
Quoted 6 lines
So there is a comment in the else part:
/* Use
XS_INTRODUCEfor recreating the xenbus event-channel. */From the commit message this is not entirely clear why we want to
prevent recreating the event-channel. Can you expand it?
Recreating the event channel would be fine, but I don't see why it would ever be needed. And XS_INTRODUCE is called only at domain creation time today, so there is obviously no need for repeating this call.
Maybe the idea was to do this after sending a XS_RESUME command, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.
Quoted 22 lines
Quoted 22 lines
Signed-off-by Juergen Gross <[email protected]>--- tools/xenstore/xenstored_domain.c | 47 ++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 28 deletions(-)tools/xenstore/xenstored_domain.c +0
-1@@ -369,7 +369,6 @@ int do_introduce(struct connection *conn, struct buffered_data *in)struct domain *domain;char *vec[3 ];unsigned int domid; -unsigned long mfn;evtchn_port_t port;int rc;struct xenstore_domain_interface *interface;@@ -381,7 +380,7 @@ int do_introduce(struct connection *conn, struct buffered_data *in)return EACCES ;
Quoted 5 lines
tools/xenstore/xenstored_domain.c int do_introduce(struct connection *conn, struct buffered_data *in) · line 383
Quoted 3 lines
tools/xenstore/xenstored_domain.c +1
-1domid =atoi (vec[0 ]); - mfn =atol (vec[1 ]); +/* Ignore the mfn, we don't need it. */ s/mfn/GFN/
Okay, then I should probably change the parameter description, too.
Quoted 8 lines
Quoted 8 lines
tools/xenstore/xenstored_domain.c +0
-0port =atoi (vec[2 ]);/* Sanity check args. */ @@ -390,34 +389,26 @@ int do_introduce(struct connection *conn, struct buffered_data *in) domain =find_domain_by_domid (domid);
Quoted 15 lines
tools/xenstore/xenstored_domain.c int do_introduce(struct connection *conn, struct buffered_data *in) · line 405
Quoted 13 lines
tools/xenstore/xenstored_domain.c +0
-13-if (domain ==NULL ) { - interface =map_interface (domid); -if (!interface) -return errno; -/* Hang domain off "in" until we're finished. */ - domain =new_domain (in, domid, port); -if (!domain) { - rc = errno; -unmap_interface (interface); -return rc; - } - domain->interface = interface; - domain->mfn = mfn;AFAICT
domain->mfnis not used anymore, so could we remove the field?
Oh, yes, I missed that.
Juergen
Hi Juergen,
Quoted 25 lines · Jürgen Groß
Quoted 17 lines · Julien Grall
Hi Juergen,
Quoted 5 lines · Juergen Gross
The
XS_INTRODUCEcommand has two parameters: the mfn (or better: gfn) of the domain's xenstore ring page and the event channel of the domain for communicating with Xenstore.Quoted 8 lines
The gfn is not really needed. It is stored in the per-domain struct in xenstored and in case of another
XS_INTRODUCEfor the domain it is tested to match the original value. If it doesn't match the command is aborted via EINVAL.Today there shouldn't be multiple
XS_INTRODUCErequests for the same domain issued, so the mfn/gfn can just be ignored and multipleXS_INTRODUCEcommands can be rejected without testing the mfn/gfn.Quoted 6 lines
So there is a comment in the else part:
/* Use
XS_INTRODUCEfor recreating the xenbus event-channel. */From the commit message this is not entirely clear why we want to prevent recreating the event-channel. Can you expand it?
Quoted 7 lines
Recreating the event channel would be fine, but I don't see why it would ever be needed. And
XS_INTRODUCEis called only at domain creation time today, so there is obviously no need for repeating this call.Maybe the idea was to do this after sending a
XS_RESUMEcommand, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.
Commit f6cc37ea8ac71385b60507c034519f304da75f4c "tools/oxenstored: port XS_INTRODUCE evtchn rebind function from cxenstored" added the exact same behavior in the OCaml XenStored last year.
This was introduced 12 years after C XenStored, so surely someone think this is useful. We should check why this was introduced in OCaml XenStored (I have CCed the author of the patch).
If we still think this is not useful, then you should add an explanation in the commit message why the two implementations diverge and possibly update the spec.
Cheers,
Signature · 1 line
Julien Grall
Quoted 12 lines · Julien Grall
Hi Juergen,
Quoted 8 lines · Jürgen Groß
Recreating the event channel would be fine, but I don't see why it would ever be needed. And
XS_INTRODUCEis called only at domain creation time today, so there is obviously no need for repeating this call.Maybe the idea was to do this after sending a
XS_RESUMEcommand, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.
Quoted 5 lines
Commit
f6cc37ea8ac71385b60507c034519f304da75f4c"tools/oxenstored: portXS_INTRODUCEevtchn rebind function from cxenstored" added the exact same behavior in the OCaml XenStored last year.This was introduced 12 years after C XenStored, so surely someone think this is useful. We should check why this was introduced in OCaml XenStored (I have CCed the author of the patch).
If we still think this is not useful, then you should add an explanation in the commit message why the two implementations diverge and possibly update the spec.
Thanks for CC, Julien.
We indeed already use this functionality in our toolstack for guest kdump functions. It's not possible in XAPI to adopt libxl model where almost everything is restarted for a domain entering kdump, so we have to use this message to rebind xenstore evtchn after soft reset without shutting down backends and recreating the whole subtree (frontends reconnect fine after that).
We obviously only require it for now to be present in oxenstored only. Please don't remove this functionality if possible.
Igor
Quoted 21 lines · Igor Druzhinin
Quoted 12 lines · Julien Grall
Hi Juergen,
Quoted 8 lines · Jürgen Groß
Recreating the event channel would be fine, but I don't see why it would ever be needed. And
XS_INTRODUCEis called only at domain creation time today, so there is obviously no need for repeating this call.Maybe the idea was to do this after sending a
XS_RESUMEcommand, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.Quoted 5 lines
Commit
f6cc37ea8ac71385b60507c034519f304da75f4c"tools/oxenstored: portXS_INTRODUCEevtchn rebind function from cxenstored" added the exact same behavior in the OCaml XenStored last year.This was introduced 12 years after C XenStored, so surely someone think this is useful. We should check why this was introduced in OCaml XenStored (I have CCed the author of the patch).
If we still think this is not useful, then you should add an explanation in the commit message why the two implementations diverge and possibly update the spec.
Thanks for CC, Julien.
Quoted 8 lines
We indeed already use this functionality in our toolstack for guest kdump functions. It's not possible in XAPI to adopt libxl model where almost everything is restarted for a domain entering kdump, so we have to use this message to rebind xenstore evtchn after soft reset without shutting down backends and recreating the whole subtree (frontends reconnect fine after that).
We obviously only require it for now to be present in oxenstored only. Please don't remove this functionality if possible.
If I read handling in libxl correctly, in the soft reset case XS_RELEASE is issued before doing another XS_INTRODUCE. XS_RELEASE will result in xenstored throwing away its related struct domain, so XS_INTRODUCE will be possible again.
Juergen
Quoted 28 lines · Jürgen Groß
Quoted 27 lines · Igor Druzhinin
Quoted 12 lines · Julien Grall
Hi Juergen,
Quoted 8 lines · Jürgen Groß
Recreating the event channel would be fine, but I don't see why it would ever be needed. And
XS_INTRODUCEis called only at domain creation time today, so there is obviously no need for repeating this call.Maybe the idea was to do this after sending a
XS_RESUMEcommand, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.Quoted 5 lines
Commit
f6cc37ea8ac71385b60507c034519f304da75f4c"tools/oxenstored: portXS_INTRODUCEevtchn rebind function from cxenstored" added the exact same behavior in the OCaml XenStored last year.This was introduced 12 years after C XenStored, so surely someone think this is useful. We should check why this was introduced in OCaml XenStored (I have CCed the author of the patch).
If we still think this is not useful, then you should add an explanation in the commit message why the two implementations diverge and possibly update the spec.
Thanks for CC, Julien.
We indeed already use this functionality in our toolstack for guest kdump functions. It's not possible in XAPI to adopt libxl model where almost everything is restarted for a domain entering kdump, so we have to use this message to rebind xenstore evtchn after soft reset without shutting down backends and recreating the whole subtree (frontends reconnect fine after that).
Quoted 7 lines
Quoted 2 lines
We obviously only require it for now to be present in oxenstored only. Please don't remove this functionality if possible.
If I read handling in libxl correctly, in the soft reset case
XS_RELEASEis issued before doing anotherXS_INTRODUCE.XS_RELEASEwill result in xenstored throwing away its related struct domain, soXS_INTRODUCEwill be possible again.
From what I remember it was not possible to keep xenstored data for a domain and at the same time perform release-introduce cycle (at least in oxenstored). It also involved firing @releaseDomain which caused havoc in other part of the toolstack.
Igor
Quoted 37 lines · Igor Druzhinin
Quoted 28 lines · Jürgen Groß
Quoted 27 lines · Igor Druzhinin
Quoted 12 lines · Julien Grall
Hi Juergen,
Quoted 8 lines · Jürgen Groß
Recreating the event channel would be fine, but I don't see why it would ever be needed. And
XS_INTRODUCEis called only at domain creation time today, so there is obviously no need for repeating this call.Maybe the idea was to do this after sending a
XS_RESUMEcommand, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.Quoted 5 lines
Commit
f6cc37ea8ac71385b60507c034519f304da75f4c"tools/oxenstored: portXS_INTRODUCEevtchn rebind function from cxenstored" added the exact same behavior in the OCaml XenStored last year.This was introduced 12 years after C XenStored, so surely someone think this is useful. We should check why this was introduced in OCaml XenStored (I have CCed the author of the patch).
If we still think this is not useful, then you should add an explanation in the commit message why the two implementations diverge and possibly update the spec.
Thanks for CC, Julien.
We indeed already use this functionality in our toolstack for guest kdump functions. It's not possible in XAPI to adopt libxl model where almost everything is restarted for a domain entering kdump, so we have to use this message to rebind xenstore evtchn after soft reset without shutting down backends and recreating the whole subtree (frontends reconnect fine after that).
Quoted 7 lines
Quoted 2 lines
We obviously only require it for now to be present in oxenstored only. Please don't remove this functionality if possible.
If I read handling in libxl correctly, in the soft reset case
XS_RELEASEis issued before doing anotherXS_INTRODUCE.XS_RELEASEwill result in xenstored throwing away its related struct domain, soXS_INTRODUCEwill be possible again.
Quoted 4 lines
From what I remember it was not possible to keep xenstored data for a domain
and at the same time perform release-introduce cycle (at least in oxenstored). It also involved firing @releaseDomain which caused havoc in other part of the toolstack.
Wei, Ian, can you please tell me where I'm wrong?
A soft reset should restart the domain in a clean state. AFAIK libxl is handling that by doing kind of in-place save-restore, including calling xs_release_domain() and later xs_introduce_domain(). This should result in xenstored throwing away all state it has regarding the domain and then restarting with a new (internal) domain instance.
Is XAPI doing soft reset differently? Why should there be a need for keeping xenstored data across a soft reset?
Juergen
Quoted 45 lines · Jürgen Groß
Quoted 37 lines · Igor Druzhinin
Quoted 28 lines · Jürgen Groß
Quoted 27 lines · Igor Druzhinin
Quoted 12 lines · Julien Grall
Hi Juergen,
Quoted 8 lines · Jürgen Groß
Recreating the event channel would be fine, but I don't see why it would ever be needed. And
XS_INTRODUCEis called only at domain creation time today, so there is obviously no need for repeating this call.Maybe the idea was to do this after sending a
XS_RESUMEcommand, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.Quoted 5 lines
Commit
f6cc37ea8ac71385b60507c034519f304da75f4c"tools/oxenstored: portXS_INTRODUCEevtchn rebind function from cxenstored" added the exact same behavior in the OCaml XenStored last year.This was introduced 12 years after C XenStored, so surely someone think this is useful. We should check why this was introduced in OCaml XenStored (I have CCed the author of the patch).
If we still think this is not useful, then you should add an explanation in the commit message why the two implementations diverge and possibly update the spec.
Thanks for CC, Julien.
We indeed already use this functionality in our toolstack for guest kdump functions. It's not possible in XAPI to adopt libxl model where almost everything is restarted for a domain entering kdump, so we have to use this message to rebind xenstore evtchn after soft reset without shutting down backends and recreating the whole subtree (frontends reconnect fine after that).
Quoted 7 lines
Quoted 2 lines
We obviously only require it for now to be present in oxenstored only. Please don't remove this functionality if possible.
If I read handling in libxl correctly, in the soft reset case
XS_RELEASEis issued before doing anotherXS_INTRODUCE.XS_RELEASEwill result in xenstored throwing away its related struct domain, soXS_INTRODUCEwill be possible again.Quoted 4 lines
From what I remember it was not possible to keep xenstored data for a domain and at the same time perform release-introduce cycle (at least in oxenstored). It also involved firing @releaseDomain which caused havoc in other part of the toolstack.
Wei, Ian, can you please tell me where I'm wrong?
Quoted 8 lines
A soft reset should restart the domain in a clean state. AFAIK libxl is handling that by doing kind of in-place save-restore, including calling
xs_release_domain()and laterxs_introduce_domain(). This should result in xenstored throwing away all state it has regarding the domain and then restarting with a new (internal) domain instance.Is XAPI doing soft reset differently? Why should there be a need for keeping xenstored data across a soft reset?
Yes, XAPI is doing soft reset differently from libxl. You need to keep xenstore data to at least keep backends working without the need to reinitialize them before entering kdump kernel. We do the same thing while entering crash kernel in Windows after BSOD (CC Paul as he recommended this approach). There are other reasons to not reset xenstore data.
I considered XS_INTRODUCE functionality to be part of xenstored ABI and we built a lot of infrastructure on top of that. So I don't think it could be easily removed now.
Igor
Quoted 64 lines
-----Original Message-----
From: Igor Druzhinin <[email protected]>
Sent: 29 April 2020 13:50
To: Jürgen Groß <[email protected]>; Julien Grall <[email protected]>; Julien Grall <[email protected]>
Cc: xen-devel <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; [email protected]; Paul Durrant <[email protected]>
Subject: Re: [PATCH]tools/xenstore: don't store domU's mfn of ring page in xensotredQuoted 12 lines · Jürgen Groß
Quoted 11 lines · Igor Druzhinin
Quoted 10 lines · Jürgen Groß
Quoted 9 lines · Igor Druzhinin
Quoted 8 lines · Julien Grall
Hi Juergen,
Quoted 5 lines · Jürgen Groß
Recreating the event channel would be fine, but I don't see why it would ever be needed. And
XS_INTRODUCEis called only at domain creation time today, so there is obviously no need for repeating this call.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Maybe the idea was to do this after sending a
XS_RESUMEcommand, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.Commit
f6cc37ea8ac71385b60507c034519f304da75f4c"tools/oxenstored: portXS_INTRODUCEevtchnrebind function from cxenstored" added the exact same behavior in the OCaml XenStored last year.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This was introduced 12 years after C XenStored, so surely someone think this is useful. We
should check why this was introduced in OCaml XenStored (I have CCed the author of the patch).
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If we still think this is not useful, then you should add an explanation in the commit message
why the two implementations diverge and possibly update the spec.
Quoted 30 lines
Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
Thanks for CC, Julien.
We indeed already use this functionality in our toolstack for guest kdump functions. It's not possible in XAPI to adopt libxl model where almost everything is restarted for a domain entering kdump, so we have to use this message to rebind xenstore evtchn after soft reset without shutting down backends and recreating the whole subtree (frontends reconnect fine after that).
Quoted 7 lines
Quoted 2 lines
We obviously only require it for now to be present in oxenstored only. Please don't remove this functionality if possible.
If I read handling in libxl correctly, in the soft reset case
XS_RELEASEis issued before doing anotherXS_INTRODUCE.XS_RELEASEwill result in xenstored throwing away its related struct domain, soXS_INTRODUCEwill be possible again.Quoted 4 lines
From what I remember it was not possible to keep xenstored data for a domain
and at the same time perform release-introduce cycle (at least in oxenstored). It also involved firing @releaseDomain which caused havoc in other part of the toolstack.Wei, Ian, can you please tell me where I'm wrong?
A soft reset should restart the domain in a clean state. AFAIK libxl is handling that by doing kind of in-place save-restore, including calling
xs_release_domain()and laterxs_introduce_domain(). This should result in xenstored throwing away all state it has regarding the domain and then restarting with a new (internal) domain instance.
Quoted 7 lines
Quoted 2 lines
Is XAPI doing soft reset differently? Why should there be a need for keeping xenstored data across a soft reset?
Yes, XAPI is doing soft reset differently from libxl. You need to keep xenstore data to at least keep backends working without the need to reinitialize them before entering kdump kernel. We do the same thing while entering crash kernel in Windows after BSOD (CC Paul as he recommended this approach).
IIRC I recommended not involving Xen or the toolstack in entering the crash kernel... they don't need to know. Windows quite happily enters its crash kernel, rebuilds its Xen interfaces and re-attaches to PV backends without any external intervention.
Paul
Quoted 6 lines
There are other reasons to not reset xenstore data.
I considered
XS_INTRODUCEfunctionality to be part of xenstored ABI and we built a lot of infrastructure on top of that. So I don't think it could be easily removed now.Igor
Quoted 22 lines · Paul Durrant
Quoted 22 lines
-----Original Message-----
From: Igor Druzhinin <[email protected]>
Sent: 29 April 2020 13:50
To: Jürgen Groß <[email protected]>; Julien Grall <[email protected]>; Julien Grall <[email protected]>
Cc: xen-devel <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; [email protected]; Paul Durrant <[email protected]>
Subject: Re: [PATCH]tools/xenstore: don't store domU's mfn of ring page in xensotredQuoted 11 lines · Jürgen Groß
Wei, Ian, can you please tell me where I'm wrong?
A soft reset should restart the domain in a clean state. AFAIK libxl is handling that by doing kind of in-place save-restore, including calling
xs_release_domain()and laterxs_introduce_domain(). This should result in xenstored throwing away all state it has regarding the domain and then restarting with a new (internal) domain instance.Is XAPI doing soft reset differently? Why should there be a need for keeping xenstored data across a soft reset?
Quoted 6 lines
Quoted 4 lines
Yes, XAPI is doing soft reset differently from libxl. You need to keep xenstore data to at least keep backends working without the need to reinitialize them before entering kdump kernel. We do the same thing while entering crash kernel in Windows after BSOD (CC Paul as he recommended this approach).
IIRC I recommended not involving Xen or the toolstack in entering the crash kernel... they don't need to know. Windows quite happily enters its crash kernel, rebuilds its Xen interfaces and re-attaches to PV backends without any external intervention.
In case of kdump toolstack certainly needs to know as it gets shutdown code 5 and needs to restart the domain.
And I'm not completely sure it's possible to enter kdump without calling soft reset at all. Even if it's possible I'd be cautious to do it in production for the future.
Igor
Quoted 37 lines
-----Original Message-----
From: Igor Druzhinin <[email protected]>
Sent: 29 April 2020 14:17
To: [email protected]; 'Jürgen Groß' <[email protected]>; 'Julien Grall' <[email protected]>; 'Julien Grall' <[email protected]>
Cc: 'xen-devel' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Wei Liu' <[email protected]>; [email protected]
Subject: Re: [PATCH]tools/xenstore: don't store domU's mfn of ring page in xensotredQuoted 27 lines · Paul Durrant
Quoted 26 lines
-----Original Message-----
From: Igor Druzhinin <[email protected]>
Sent: 29 April 2020 13:50
To: Jürgen Groß <[email protected]>; Julien Grall <[email protected]>; Julien Grall <[email protected]>
Cc: xen-devel <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; [email protected]; Paul Durrant <[email protected]>
Subject: Re: [PATCH]tools/xenstore: don't store domU's mfn of ring page in xensotredQuoted 11 lines · Jürgen Groß
Wei, Ian, can you please tell me where I'm wrong?
A soft reset should restart the domain in a clean state. AFAIK libxl is handling that by doing kind of in-place save-restore, including calling
xs_release_domain()and laterxs_introduce_domain(). This should result in xenstored throwing away all state it has regarding the domain and then restarting with a new (internal) domain instance.Is XAPI doing soft reset differently? Why should there be a need for keeping xenstored data across a soft reset?
Yes, XAPI is doing soft reset differently from libxl. You need to keep xenstore data to at least keep backends working without the need to reinitialize them before entering kdump kernel. We do the same thing while entering crash kernel in Windows after BSOD (CC Paul as he recommended this approach).
Quoted 7 lines
Quoted 1 line
IIRC I recommended not involving Xen or the toolstack in entering the crash kernel... they don't
need to know. Windows quite happily enters its crash kernel, rebuilds its Xen interfaces and re- attaches to PV backends without any external intervention.
In case of kdump toolstack certainly needs to know as it gets shutdown code 5 and needs to restart the domain.
The toolstack needs to restart the domain once the crash has completed, yes.
Quoted 3 lines
And I'm not completely sure it's possible to enter kdump without calling soft reset at all. Even if it's possible I'd be cautious to do it in production for the future.
If it is not possible at the moment then IMO it should be made so; having soft reset in the toolstack is a layering violation IMO.
Paul
Quoted 38 lines · Paul Durrant
Quoted 38 lines
-----Original Message-----
From: Igor Druzhinin <[email protected]>
Sent: 29 April 2020 14:17
To: [email protected]; 'Jürgen Groß' <[email protected]>; 'Julien Grall' <[email protected]>; 'Julien Grall' <[email protected]>
Cc: 'xen-devel' <[email protected]>; 'Ian Jackson' <[email protected]>; 'Wei Liu' <[email protected]>; [email protected]
Subject: Re: [PATCH]tools/xenstore: don't store domU's mfn of ring page in xensotredQuoted 28 lines · Paul Durrant
Quoted 19 lines
-----Original Message-----
From: Igor Druzhinin <[email protected]>
Sent: 29 April 2020 13:50
To: Jürgen Groß <[email protected]>; Julien Grall <[email protected]>; Julien Grall <[email protected]>
Cc: xen-devel <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; [email protected]; Paul Durrant <[email protected]>
Subject: Re: [PATCH]tools/xenstore: don't store domU's mfn of ring page in xensotredQuoted 9 lines · Jürgen Groß
Wei, Ian, can you please tell me where I'm wrong?
A soft reset should restart the domain in a clean state. AFAIK libxl is handling that by doing kind of in-place save-restore, including calling
xs_release_domain()and laterxs_introduce_domain(). This should result in xenstored throwing away all state it has regarding the domain and then restarting with a new (internal) domain instance.Quoted 7 lines
Quoted 2 lines
Is XAPI doing soft reset differently? Why should there be a need for keeping xenstored data across a soft reset?
Yes, XAPI is doing soft reset differently from libxl. You need to keep xenstore data to at least keep backends working without the need to reinitialize them before entering kdump kernel. We do the same thing while entering crash kernel in Windows after BSOD (CC Paul as he recommended this approach).
IIRC I recommended not involving Xen or the toolstack in entering the crash kernel... they don't
Quoted 8 lines
Quoted 6 lines
need to know. Windows quite happily enters its crash kernel, rebuilds its Xen interfaces and re- attaches to PV backends without any external intervention.
In case of kdump toolstack certainly needs to know as it gets shutdown code 5 and needs to restart the domain.
The toolstack needs to restart the domain once the crash has completed, yes.
To clarify, what I meant is that once crash happened (before kdump kernel is loaded) toolstack gets code 5 and then it's supposed to call soft reset and unpause the domain to actually load into crash kernel.
I didn't mean that after crash kernel is finished the domain has to be restarted - that's obvious.
Igor
Quoted 55 lines · Igor Druzhinin
Quoted 45 lines · Jürgen Groß
Quoted 37 lines · Igor Druzhinin
Quoted 28 lines · Jürgen Groß
Quoted 27 lines · Igor Druzhinin
Quoted 12 lines · Julien Grall
Hi Juergen,
Quoted 8 lines · Jürgen Groß
Recreating the event channel would be fine, but I don't see why it would ever be needed. And
XS_INTRODUCEis called only at domain creation time today, so there is obviously no need for repeating this call.Maybe the idea was to do this after sending a
XS_RESUMEcommand, which isn't used anywhere in Xen and is another part of Xenstore which doesn't make any sense today.Quoted 5 lines
Commit
f6cc37ea8ac71385b60507c034519f304da75f4c"tools/oxenstored: portXS_INTRODUCEevtchn rebind function from cxenstored" added the exact same behavior in the OCaml XenStored last year.This was introduced 12 years after C XenStored, so surely someone think this is useful. We should check why this was introduced in OCaml XenStored (I have CCed the author of the patch).
If we still think this is not useful, then you should add an explanation in the commit message why the two implementations diverge and possibly update the spec.
Thanks for CC, Julien.
We indeed already use this functionality in our toolstack for guest kdump functions. It's not possible in XAPI to adopt libxl model where almost everything is restarted for a domain entering kdump, so we have to use this message to rebind xenstore evtchn after soft reset without shutting down backends and recreating the whole subtree (frontends reconnect fine after that).
Quoted 7 lines
Quoted 2 lines
We obviously only require it for now to be present in oxenstored only. Please don't remove this functionality if possible.
If I read handling in libxl correctly, in the soft reset case
XS_RELEASEis issued before doing anotherXS_INTRODUCE.XS_RELEASEwill result in xenstored throwing away its related struct domain, soXS_INTRODUCEwill be possible again.Quoted 4 lines
From what I remember it was not possible to keep xenstored data for a domain
and at the same time perform release-introduce cycle (at least in oxenstored). It also involved firing @releaseDomain which caused havoc in other part of the toolstack.Wei, Ian, can you please tell me where I'm wrong?
Quoted 8 lines
A soft reset should restart the domain in a clean state. AFAIK libxl is handling that by doing kind of in-place save-restore, including calling
xs_release_domain()and laterxs_introduce_domain(). This should result in xenstored throwing away all state it has regarding the domain and then restarting with a new (internal) domain instance.Is XAPI doing soft reset differently? Why should there be a need for keeping xenstored data across a soft reset?
Quoted 8 lines
Yes, XAPI is doing soft reset differently from libxl. You need to keep xenstore data to at least keep backends working without the need to reinitialize them before entering kdump kernel. We do the same thing while entering crash kernel in Windows after BSOD (CC Paul as he recommended this approach). There are other reasons to not reset xenstore data.
I considered
XS_INTRODUCEfunctionality to be part of xenstored ABI and we built a lot of infrastructure on top of that. So I don't think it could be easily removed now.
Nobody wants to remove XS_INTRODUCE. It was just questioned there is a need to call XS_INTRODUCE multiple times for a domain without a call of XS_RELEASE in between.
In case there is such a need this means we either should just drop the test for the mfn to match and recreate the event-channel (which will do no harm IMO), or we need to keep the mfn in the live-update state record with the knowledge that it is far from being a good indicator for the test whether a domain is known already or not (two HVM domains using a similar configuration with the very same kernel will use the same gfn probably).
As only dom0 is allowed to use XS_INTRODUCE and I'm not aware of any problems with neither XS_INTRODUCE failing due to illegal calls (no mfn match), nor with potentially recreating the event channel for a "wrong" domain, I suggest to just allow XS_INTRODUCE to be called as often as the toolstack wants to.
In the end that was the primary reason to write this patch: to remove the need to have the mfn in the live-update state record.
Thoughts?
Juergen
Quoted 10 lines · Jürgen Groß
Quoted 9 lines · Igor Druzhinin
Quoted 1 line · Jürgen Groß
On 29.04.20 13:04, Igor Druzhinin wrote:
Yes, XAPI is doing soft reset differently from libxl. You need to keep xenstore data to at least keep backends working without the need to reinitialize them before entering kdump kernel. We do the same thing while entering crash kernel in Windows after BSOD (CC Paul as he recommended this approach). There are other reasons to not reset xenstore data.
Quoted 6 lines
Quoted 2 lines
I considered
XS_INTRODUCEfunctionality to be part of xenstored ABI and we built a lot of infrastructure on top of that. So I don't think it could be easily removed now.Nobody wants to remove
XS_INTRODUCE. It was just questioned there is a need to callXS_INTRODUCEmultiple times for a domain without a call ofXS_RELEASEin between.
Sorry, I didn't mean the whole XS_INTRODUCE but that particular bit that allows it to rebind the channels without calling XS_RELEASE first as the latter has a lot of implications from toolstack point of view.
Quoted 8 lines
In case there is such a need this means we either should just drop the test for the mfn to match and recreate the event-channel (which will do no harm IMO), or we need to keep the mfn in the live-update state record with the knowledge that it is far from being a good indicator for the test whether a domain is known already or not (two HVM domains using a similar configuration with the very same kernel will use the same gfn probably).
Quoted 5 lines
As only dom0 is allowed to use
XS_INTRODUCEand I'm not aware of any problems with neitherXS_INTRODUCEfailing due to illegal calls (no mfn match), nor with potentially recreating the event channel for a "wrong" domain, I suggest to just allowXS_INTRODUCEto be called as often as the toolstack wants to.
If that means keeping the current semantics of XS_INTRODUCE - I'd be quite happy with that.
Igor