xenconsole: Add connected flag
PATCH 1/1 xenconsole: Add connected flag
With hyperlaunch, a domU can start before its console ring is connected by xenconsoled. With nothing emptying the ring, it can quickly fill during boot. In domU_write_console(), __write_console() returns 0 when the ring is full. This loops spins until xenconsoled starts emptying the ring:
while (len) {
ssize_t sent = __write_console(cons, data, len);
if (sent < 0)
return sent;
data += sent;
len -= sent;
if (unlikely(len))
HYPERVISOR_sched_op(SCHEDOP_yield, NULL);
}The goal of this patch is to add a way for the frontend to know when a console is connected. This patch adds a new flag to the end of the console ring structure. It is used for the backend to indicate that it has connected and started servicing the page.
The two values areXENCONSOLE_DISCONNECTED 1XENCONSOLE_CONNECTED 0
XENCONSOLE_DISCONNECTED indicates to the guest that ring is disconnected, so it will not be serviced. The guest can avoid writing into it in that case. A domU can use console hypercalls and only transition to the ring when it is connected and won't fill and block.
Once the backend (xenconsoled) maps and starts servicing the console, the flag will be set to XENCONSOLE_CONNECTED (0) to indicate the backend state to the frontend.
The connected value as 0 will be match the default of a zero-ed console page. Hyperlaunch can set the flag to XENCONSOLE_DISCONNECTED and let xenconsoled set to XENCONSOLE_CONNECTED.
Old domU hvc_xen drivers won't check the flag.
New domU hvc_xen running on a new xen/xenconsoled will work properly. New domU hvc_xen on old xen/xenconsoled should only see a 0 for the flag and behave as if connected.
---
v1:
Remove evtchn notify call
Set connected later when there is no error
RFC v3:
Flip flag values so 0 is connected.
The other option would be to add:uint32_t featuresuint32_t connected
New domUs would check features for a magic value and/or flag to know they can rely on connected transitioning.
I think making XENCONSOLE_CONNECTED == 0 side steps the need for an additional features field. As long as assuming zero-ed memory is acceptable. However, this only matters for a hyperlaunched guest - xenconsoled will normally readily connect the console and set the value.
This assumes that existing frontends are not using the flag space for some other use.
Removed idea:
Send an event channel notification to let the domU know that xenconsoled is connected. Xenstored does similar, but for xenstore, the xenstore driver owns the event channel/irq and can rebind it. For hvc_xen, the hvc subsystem owns the irq, so it isn't readily available for rebinding. This is not implemented.
I had the idea for the kernel to use a static key and switch writing from the hypercall to the PV ring once connected. It didn't actually work in my short attempt - I think changing the static key from within an interupt was wrong. I fell back to just checking the flag directly without an optimization. That would work and would make the event channel notification unnecessary.
--- tools/console/daemon/io.c | 4 ++++ xen/include/public/io/console.h | 13 +++++++++++++ 2 files changed, 17 insertions(+)
tools/console/daemon/io.c +4 -0
@@ -781,6 +781,10 @@ static int console_create_ring(struct console *con) con->log_fd =create_console_log (con); out: +/* Mark the console connected. */ +if (!err && con->interface) + con->interface->flag =XENCONSOLE_CONNECTED ; +return err; }
xen/include/public/io/console.h +13 -0
@@ -19,6 +19,19 @@ struct xencons_interface {char out[2048 ];XENCONS_RING_IDX in_cons, in_prod;XENCONS_RING_IDX out_cons, out_prod; +/* +* Flag values signaling from backend to frontend whether the console is +* connected. i.e. Whether it will be serviced and emptied. +* +* The flag starts as disconnected. +*/ +#define XENCONSOLE_DISCONNECTED 1 +/* +* The flag is set to connected when the backend connects and the console +* will be serviced. +*/ +#define XENCONSOLE_CONNECTED 0 +uint32_t flag; };#ifdef XEN_WANT_FLEX_CONSOLE_RING
Signature · 1 line
2.50.1
Quoted 33 lines · Jason Andryuk
With hyperlaunch, a domU can start before its console ring is connected by xenconsoled. With nothing emptying the ring, it can quickly fill during boot. In
domU_write_console(),__write_console()returns 0 when the ring is full. This loops spins until xenconsoled starts emptying the ring:while (len) { ssize_t sent = __write_console(cons, data, len); if (sent < 0) return sent; data += sent; len -= sent; if (unlikely(len)) HYPERVISOR_sched_op(SCHEDOP_yield, NULL); }The goal of this patch is to add a way for the frontend to know when a console is connected. This patch adds a new flag to the end of the console ring structure. It is used for the backend to indicate that it has connected and started servicing the page.
The two values are
XENCONSOLE_DISCONNECTED1XENCONSOLE_CONNECTED0
XENCONSOLE_DISCONNECTEDindicates to the guest that ring is disconnected, so it will not be serviced. The guest can avoid writing into it in that case. A domU can use console hypercalls and only transition to the ring when it is connected and won't fill and block.
Quoted 7 lines
Once the backend (xenconsoled) maps and starts servicing the console, the flag will be set to
XENCONSOLE_CONNECTED(0) to indicate the backend state to the frontend.The connected value as 0 will be match the default of a zero-ed console page. Hyperlaunch can set the flag to
XENCONSOLE_DISCONNECTEDand let xenconsoled set toXENCONSOLE_CONNECTED.
I think libxenguest should set XENCONSOLE_DISCONNECTED as well (see below).
Quoted 7 lines
Old domU
hvc_xendrivers won't check the flag.
New domUhvc_xenrunning on a new xen/xenconsoled will work properly. New domUhvc_xenon old xen/xenconsoled should only see a 0 for the flag and behave as if connected.Signed-off-by Jason Andryuk <[email protected]>
Adapt the title of the patch?
Quoted 12 lines
---
v1:
Remove evtchn notify call
Set connected later when there is no errorRFC v3:
Flip flag values so 0 is connected.The other option would be to add:
uint32_tfeaturesuint32_tconnected
Quoted 7 lines
New domUs would check features for a magic value and/or flag to know they can rely on connected transitioning.
I think making
XENCONSOLE_CONNECTED== 0 side steps the need for an additional features field. As long as assuming zero-ed memory is acceptable. However, this only matters for a hyperlaunched guest - xenconsoled will normally readily connect the console and set the value.
I'd like to consider other cases as well, e.g. a console driver domain. So any instance creating a domain with a console ring page should set the flag initially to "disconnected".
Quoted 10 lines
This assumes that existing frontends are not using the flag space for some other use.
Removed idea:
Send an event channel notification to let the domU know that xenconsoled is connected. Xenstored does similar, but for xenstore, the xenstore driver owns the event channel/irq and can rebind it. Forhvc_xen, the hvc subsystem owns the irq, so it isn't readily available for rebinding. This is not implemented.
Quoted 4 lines
I had the idea for the kernel to use a static key and switch writing from the hypercall to the PV ring once connected. It didn't actually work in my short attempt - I think changing the static key from within an interupt was wrong. I fell back to just checking the flag directly
You'd need to do the static key changing from a worker thread instead.
Juergen
Quoted 45 lines · Jürgen Groß
Quoted 33 lines · Jason Andryuk
With hyperlaunch, a domU can start before its console ring is connected by xenconsoled. With nothing emptying the ring, it can quickly fill during boot. In
domU_write_console(),__write_console()returns 0 when the ring is full. This loops spins until xenconsoled starts emptying the ring:while (len) {
ssize_tsent =__write_console(cons, data, len);if (sent < 0)
return sent;data += sent;
len -= sent;if (unlikely(len))
HYPERVISOR_sched_op(SCHEDOP_yield, NULL); }The goal of this patch is to add a way for the frontend to know when a console is connected. This patch adds a new flag to the end of the console ring structure. It is used for the backend to indicate that it has connected and started servicing the page.
The two values are
XENCONSOLE_DISCONNECTED1
XENCONSOLE_CONNECTED 0
XENCONSOLE_DISCONNECTEDindicates to the guest that ring is disconnected, so it will not be serviced. The guest can avoid writing into it in that case. A domU can use console hypercalls and only transition to the ring when it is connected and won't fill and block.Quoted 7 lines
Once the backend (xenconsoled) maps and starts servicing the console, the flag will be set to
XENCONSOLE_CONNECTED(0) to indicate the backend state to the frontend.The connected value as 0 will be match the default of a zero-ed console page. Hyperlaunch can set the flag to
XENCONSOLE_DISCONNECTEDand let xenconsoled set toXENCONSOLE_CONNECTED.I think libxenguest should set
XENCONSOLE_DISCONNECTEDas well (see below).Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Old domU
hvc_xendrivers won't check the flag.
New domUhvc_xenrunning on a new xen/xenconsoled will work properly. New domUhvc_xenon old xen/xenconsoled should only see a 0 for the flag and behave as if connected.Signed-off-by Jason Andryuk <[email protected]>Adapt the title of the patch?
With Jan's naming suggestion, it'll be "xenconsole: Add connection field"
Quoted 15 lines
Quoted 15 lines
---
v1:
Remove evtchn notify call
Set connected later when there is no errorRFC v3:
Flip flag values so 0 is connected.The other option would be to add:
uint32_tfeaturesuint32_tconnectedNew domUs would check features for a magic value and/or flag to know they can rely on connected transitioning.
Quoted 8 lines
Quoted 4 lines
I think making
XENCONSOLE_CONNECTED== 0 side steps the need for an additional features field. As long as assuming zero-ed memory is acceptable. However, this only matters for a hyperlaunched guest - xenconsoled will normally readily connect the console and set the value.I'd like to consider other cases as well, e.g. a console driver domain. So any instance creating a domain with a console ring page should set the flag initially to "disconnected".
Sure, that makes sense.
I don't have immediate plans for this, but I had the additional thought of placing the event channel inside the page like we do for xenstore. Today xenconsoled uses xenstore for the ring-ref and event channel, thought it tries the reserved console grant first. With the reserved grant, and the event channel inside, that dependency would be removed. Xenconsoled still needs some way to know when domains come and go, and it depends on xenstore's @introduceDomain/@releaseDomain for that. So it couldn't be completely decoupled without some other mechanism.
Quoted 10 lines
Quoted 10 lines
This assumes that existing frontends are not using the flag space for some other use.
Removed idea:
Send an event channel notification to let the domU know that xenconsoled is connected. Xenstored does similar, but for xenstore, the xenstore driver owns the event channel/irq and can rebind it. Forhvc_xen, the hvc subsystem owns the irq, so it isn't readily available for rebinding. This is not implemented.
Quoted 6 lines
Quoted 4 lines
I had the idea for the kernel to use a static key and switch writing from the hypercall to the PV ring once connected. It didn't actually work in my short attempt - I think changing the static key from within an interupt was wrong. I fell back to just checking the flag directly
You'd need to do the static key changing from a worker thread instead.
Thanks. I'll investigate.
Regards,
Jason
Quoted 69 lines · Jürgen Groß
Quoted 33 lines · Jason Andryuk
With hyperlaunch, a domU can start before its console ring is connected by xenconsoled. With nothing emptying the ring, it can quickly fill during boot. In
domU_write_console(),__write_console()returns 0 when the ring is full. This loops spins until xenconsoled starts emptying the ring:while (len) {
ssize_tsent =__write_console(cons, data, len);if (sent < 0)
return sent;data += sent;
len -= sent;if (unlikely(len))
HYPERVISOR_sched_op(SCHEDOP_yield, NULL); }The goal of this patch is to add a way for the frontend to know when a console is connected. This patch adds a new flag to the end of the console ring structure. It is used for the backend to indicate that it has connected and started servicing the page.
The two values are
XENCONSOLE_DISCONNECTED1
XENCONSOLE_CONNECTED 0
XENCONSOLE_DISCONNECTEDindicates to the guest that ring is disconnected, so it will not be serviced. The guest can avoid writing into it in that case. A domU can use console hypercalls and only transition to the ring when it is connected and won't fill and block.Quoted 7 lines
Once the backend (xenconsoled) maps and starts servicing the console, the flag will be set to
XENCONSOLE_CONNECTED(0) to indicate the backend state to the frontend.The connected value as 0 will be match the default of a zero-ed console page. Hyperlaunch can set the flag to
XENCONSOLE_DISCONNECTEDand let xenconsoled set toXENCONSOLE_CONNECTED.I think libxenguest should set
XENCONSOLE_DISCONNECTEDas well (see below).Quoted 7 lines
Old domU
hvc_xendrivers won't check the flag.
New domUhvc_xenrunning on a new xen/xenconsoled will work properly. New domUhvc_xenon old xen/xenconsoled should only see a 0 for the flag and behave as if connected.Signed-off-by Jason Andryuk <[email protected]>Adapt the title of the patch?
Quoted 15 lines
---
v1:
Remove evtchn notify call
Set connected later when there is no errorRFC v3:
Flip flag values so 0 is connected.The other option would be to add:
uint32_tfeaturesuint32_tconnectedNew domUs would check features for a magic value and/or flag to know they can rely on connected transitioning.
Quoted 8 lines
Quoted 4 lines
I think making
XENCONSOLE_CONNECTED== 0 side steps the need for an additional features field. As long as assuming zero-ed memory is acceptable. However, this only matters for a hyperlaunched guest - xenconsoled will normally readily connect the console and set the value.I'd like to consider other cases as well, e.g. a console driver domain. So any instance creating a domain with a console ring page should set the flag initially to "disconnected".
Setting disconnected for domain creation is fine. Looking at libxenguest, there is also domain restore. There the console could be set to disconnected again before domain restore. Again, this should work and xenconsoled would set it connected again. I originally intended for a single one way transition disconnected -> connected.
Alternatively, restore could skip setting disconnected and just assume xenconsoled will promptly attach. Restore implies a toolstack is running, so there isn't the indefinite time period that is involved with hyperlaunch/dom0less. But I guess an actively changing flag accurately shows the state, so that is preferable.
Quoted 10 lines
Quoted 10 lines
This assumes that existing frontends are not using the flag space for some other use.
Removed idea:
Send an event channel notification to let the domU know that xenconsoled is connected. Xenstored does similar, but for xenstore, the xenstore driver owns the event channel/irq and can rebind it. Forhvc_xen, the hvc subsystem owns the irq, so it isn't readily available for rebinding. This is not implemented.
Quoted 6 lines
Quoted 4 lines
I had the idea for the kernel to use a static key and switch writing from the hypercall to the PV ring once connected. It didn't actually work in my short attempt - I think changing the static key from within an interupt was wrong. I fell back to just checking the flag directly
You'd need to do the static key changing from a worker thread instead.
My static key idea has an issue. The flag needs to be per-instance, primary console and any additional PV consoles, but the kernel has only a single function to handle all of them. Either the primary console needs dedicated ops, or the the flag would need to be checked in the function. If the flag will toggle back and forth, then a static key may not be appropriate.
Regards,
Jason
Quoted 75 lines · Jason Andryuk
Quoted 74 lines · Jürgen Groß
Quoted 33 lines · Jason Andryuk
With hyperlaunch, a domU can start before its console ring is connected by xenconsoled. With nothing emptying the ring, it can quickly fill during boot. In
domU_write_console(),__write_console()returns 0 when the ring is full. This loops spins until xenconsoled starts emptying the ring:while (len) {
ssize_tsent =__write_console(cons, data, len);if (sent < 0)
return sent;data += sent;
len -= sent;if (unlikely(len))
HYPERVISOR_sched_op(SCHEDOP_yield, NULL); }The goal of this patch is to add a way for the frontend to know when a console is connected. This patch adds a new flag to the end of the console ring structure. It is used for the backend to indicate that it has connected and started servicing the page.
The two values are
XENCONSOLE_DISCONNECTED1
XENCONSOLE_CONNECTED 0
XENCONSOLE_DISCONNECTEDindicates to the guest that ring is disconnected, so it will not be serviced. The guest can avoid writing into it in that case. A domU can use console hypercalls and only transition to the ring when it is connected and won't fill and block.Quoted 7 lines
Once the backend (xenconsoled) maps and starts servicing the console, the flag will be set to
XENCONSOLE_CONNECTED(0) to indicate the backend state to the frontend.The connected value as 0 will be match the default of a zero-ed console page. Hyperlaunch can set the flag to
XENCONSOLE_DISCONNECTEDand let xenconsoled set toXENCONSOLE_CONNECTED.I think libxenguest should set
XENCONSOLE_DISCONNECTEDas well (see below).Quoted 7 lines
Old domU
hvc_xendrivers won't check the flag.
New domUhvc_xenrunning on a new xen/xenconsoled will work properly. New domUhvc_xenon old xen/xenconsoled should only see a 0 for the flag and behave as if connected.Signed-off-by Jason Andryuk <[email protected]>Adapt the title of the patch?
Quoted 19 lines
---
v1:
Remove evtchn notify call
Set connected later when there is no errorRFC v3:
Flip flag values so 0 is connected.The other option would be to add:
uint32_tfeaturesuint32_tconnectedNew domUs would check features for a magic value and/or flag to know they can rely on connected transitioning.
I think making
XENCONSOLE_CONNECTED== 0 side steps the need for an additional features field. As long as assuming zero-ed memory is acceptable. However, this only matters for a hyperlaunched guest - xenconsoled will normally readily connect the console and set the value.
Quoted 7 lines
Quoted 3 lines
I'd like to consider other cases as well, e.g. a console driver domain. So any instance creating a domain with a console ring page should set the flag initially to "disconnected".
Setting disconnected for domain creation is fine. Looking at libxenguest, there is also domain restore. There the console could be set to disconnected again before domain restore. Again, this should work and xenconsoled would set it connected again. I originally intended for a single one way transition disconnected -> connected.
Alternatively, restore could skip setting disconnected and just assume xenconsoled will promptly attach. Restore implies a toolstack is running, so there isn't the indefinite time period that is involved with hyperlaunch/ dom0less. But I guess an actively changing flag accurately shows the state, so that is preferable.
I agree.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
This assumes that existing frontends are not using the flag space for some other use.
Removed idea:
Send an event channel notification to let the domU know that xenconsoled is connected. Xenstored does similar, but for xenstore, the xenstore driver owns the event channel/irq and can rebind it. Forhvc_xen, the hvc subsystem owns the irq, so it isn't readily available for rebinding. This is not implemented.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
I had the idea for the kernel to use a static key and switch writing from the hypercall to the PV ring once connected. It didn't actually work in my short attempt - I think changing the static key from within an interupt was wrong. I fell back to just checking the flag directly
You'd need to do the static key changing from a worker thread instead.
My static key idea has an issue. The flag needs to be per-instance, primary console and any additional PV consoles, but the kernel has only a single function to handle all of them. Either the primary console needs dedicated ops, or the the flag would need to be checked in the function. If the flag will toggle back and forth, then a static key may not be appropriate.
Its not as if consoles are high throughput devices. So testing the flag instead of using a static key shouldn't be a big issue.
Juergen
Quoted 10 lines · Jason Andryuk
xen/include/public/io/console.h +4
-0@@ -19,6 +19,19 @@ struct xencons_interface {char out[2048 ];XENCONS_RING_IDX in_cons, in_prod;XENCONS_RING_IDX out_cons, out_prod; +/* +* Flag values signaling from backend to frontend whether the console is +* connected. i.e. Whether it will be serviced and emptied. +*
xen/include/public/io/console.h struct xencons_interface { · line 35
Quoted 10 lines
xen/include/public/io/console.h +9
-0+* The flag starts as disconnected. +*/ +#define XENCONSOLE_DISCONNECTED 1 +/* +* The flag is set to connected when the backend connects and the console +* will be serviced. +*/ +#define XENCONSOLE_CONNECTED 0 +uint32_t flag; };
Even if the field name is singular, there's the possible reading of everything together as the connected state merely taking one bit, with other bits available for future use. IOW I think the field wants giving a less generic name, and perhaps also shrinking to unsigned char (or uint8_t, but char is being used in the structure already anyway).
Jan
Quoted 19 lines · Jan Beulich
Quoted 18 lines · Jason Andryuk
xen/include/public/io/console.h +12
-0@@ -19,6 +19,19 @@ struct xencons_interface {char out[2048 ];XENCONS_RING_IDX in_cons, in_prod;XENCONS_RING_IDX out_cons, out_prod; +/* +* Flag values signaling from backend to frontend whether the console is +* connected. i.e. Whether it will be serviced and emptied. +* +* The flag starts as disconnected. +*/ +#define XENCONSOLE_DISCONNECTED 1 +/* +* The flag is set to connected when the backend connects and the console +* will be serviced. +*/ +#define XENCONSOLE_CONNECTED 0
Quoted 8 lines
xen/include/public/io/console.h struct xencons_interface { · line 35
Quoted 2 lines
xen/include/public/io/console.h +1
-0+uint32_t flag; };Even if the field name is singular, there's the possible reading of everything together as the connected state merely taking one bit, with other bits available for future use. IOW I think the field wants giving a less generic name, and perhaps also shrinking to unsigned char (or
uint8_t, but char is being used in the structure already anyway).
It could become a bit in a multi-flag field, dut the structure has a good bit of space to the end of the page, A named field, "connection", sounds good.
Thanks,
Jason