chardev/char-pty: Avoid losing bytes when the other side just (re-)connected
PATCH 1/1 chardev/char-pty: Avoid losing bytes when the other side just (re-)connected
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit f8278c7d74 ("char-pty: remove the check for connection on write") to avoid some locking.
To ease the situation here at least a little bit, let's check with g_poll() whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commit fac6688a18 ("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.
With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
--- chardev/char-pty.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
chardev/char-pty.c +19 -3
@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr)static int char_pty_chr_write (Chardev *chr,const uint8_t *buf,int len) {PtyChardev *s =PTY_CHARDEV (chr); +GPollFD pfd; +int rc; -if (!s->connected) { -return len; +if (s->connected) { +return io_channel_send (s->ioc, buf, len); } -return io_channel_send (s->ioc, buf, len); + +/* +* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write again: +*/ + pfd.fd =QIO_CHANNEL_FILE (s->ioc)->fd; + pfd.events =G_IO_OUT ; + pfd.revents =0 ; + rc =RETRY_ON_EINTR (g_poll (&pfd,1 ,0 )); +g_assert (rc >=0 ); +if (!(pfd.revents &G_IO_HUP ) && (pfd.revents &G_IO_OUT )) { +io_channel_send (s->ioc, buf, len); + } + +return len; }static GSource *pty_chr_add_watch (Chardev *chr,GIOCondition cond)
Signature · 1 line
2.39.3
Quoted 50 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- chardev/char-pty.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)chardev/char-pty.c +5
-3@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr)static int char_pty_chr_write (Chardev *chr,const uint8_t *buf,int len) {PtyChardev *s =PTY_CHARDEV (chr); +GPollFD pfd; +int rc; -if (!s->connected) { -return len; +if (s->connected) { +return io_channel_send (s->ioc, buf, len); } -return io_channel_send (s->ioc, buf, len); +
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 125
Quoted 10 lines
chardev/char-pty.c +10
-0+/* +* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write again: +*/ + pfd.fd =QIO_CHANNEL_FILE (s->ioc)->fd; + pfd.events =G_IO_OUT ; + pfd.revents =0 ; + rc =RETRY_ON_EINTR (g_poll (&pfd,1 ,0 )); +g_assert (rc >=0 ); +if (!(pfd.revents &G_IO_HUP ) && (pfd.revents &G_IO_OUT )) {
Should (can?) we call
pty_chr_state(chr, 1);
here ?
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 126
Quoted 1 line
chardev/char-pty.c +1
-0+io_channel_send (s->ioc, buf, len);
As it feels a little dirty to be sending data before setting the 'connected == 1' and thus issuing the 'CHR_EVENT_OPENED' event
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 132
Quoted 6 lines
chardev/char-pty.c +3
-0+ } + +return len; }static GSource *pty_chr_add_watch (Chardev *chr,GIOCondition cond)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 64 lines · Daniel P. Berrangé
Quoted 50 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- chardev/char-pty.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)chardev/char-pty.c +5
-3@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr)static int char_pty_chr_write (Chardev *chr,const uint8_t *buf,int len) {PtyChardev *s =PTY_CHARDEV (chr); +GPollFD pfd; +int rc; -if (!s->connected) { -return len; +if (s->connected) { +return io_channel_send (s->ioc, buf, len); } -return io_channel_send (s->ioc, buf, len); +chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 125
Quoted 10 lines
chardev/char-pty.c +10
-0+/* +* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write again: +*/ + pfd.fd =QIO_CHANNEL_FILE (s->ioc)->fd; + pfd.events =G_IO_OUT ; + pfd.revents =0 ; + rc =RETRY_ON_EINTR (g_poll (&pfd,1 ,0 )); +g_assert (rc >=0 ); +if (!(pfd.revents &G_IO_HUP ) && (pfd.revents &G_IO_OUT )) {Should (can?) we call
Quoted 3 lines
pty_chr_state(chr, 1);here ?
As far as I understood commit f8278c7d74c6 and f7ea2038bea04628, this is not possible anymore since the lock has been removed.
Quoted 4 lines
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 126
Quoted 1 line
chardev/char-pty.c +1
-0+io_channel_send (s->ioc, buf, len);As it feels a little dirty to be sending data before setting the 'connected == 1' and thus issuing the '
CHR_EVENT_OPENED' event
I didn't find a really better solution so far. We could maybe introduce a buffer in the char-pty code and store the last second of guest output, but IMHO that's way more complex and thus somewhat ugly, too?
Thomas
Quoted 71 lines · Thomas Huth
Quoted 63 lines · Daniel P. Berrangé
Quoted 49 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- chardev/char-pty.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)chardev/char-pty.c +5
-3@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr)static int char_pty_chr_write (Chardev *chr,const uint8_t *buf,int len) {PtyChardev *s =PTY_CHARDEV (chr); +GPollFD pfd; +int rc; -if (!s->connected) { -return len; +if (s->connected) { +return io_channel_send (s->ioc, buf, len); } -return io_channel_send (s->ioc, buf, len); +chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 124
Quoted 10 lines
chardev/char-pty.c +10
-0+/* +* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write again: +*/ + pfd.fd =QIO_CHANNEL_FILE (s->ioc)->fd; + pfd.events =G_IO_OUT ; + pfd.revents =0 ; + rc =RETRY_ON_EINTR (g_poll (&pfd,1 ,0 )); +g_assert (rc >=0 ); +if (!(pfd.revents &G_IO_HUP ) && (pfd.revents &G_IO_OUT )) {Should (can?) we call
Quoted 3 lines
pty_chr_state(chr, 1);here ?
As far as I understood commit
f8278c7d74c6andf7ea2038bea04628, this is not possible anymore since the lock has been removed.
Quoted 8 lines
Quoted 4 lines
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 126
Quoted 1 line
chardev/char-pty.c +1
-0+io_channel_send (s->ioc, buf, len);As it feels a little dirty to be sending data before setting the 'connected == 1' and thus issuing the '
CHR_EVENT_OPENED' eventI didn't find a really better solution so far. We could maybe introduce a buffer in the char-pty code and store the last second of guest output, but IMHO that's way more complex and thus somewhat ugly, too?
The orignal commit f8278c7d74c6 said
[quote]
char-pty: remove the check for connection on write
This doesn't help much compared to the 1 second poll PTY timer. I can't think of a use case where this would help.
[/quote]
We've now identified a use case where it is actually important.
IOW, there's a justification to revert both f7ea2038bea04628 and f8278c7d74c6, re-adding the locking and write update logic.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi
Quoted 87 lines · Daniel P. Berrangé
Quoted 71 lines · Thomas Huth
Quoted 63 lines · Daniel P. Berrangé
Quoted 49 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- chardev/char-pty.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)chardev/char-pty.c +5
-3@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr)static int char_pty_chr_write (Chardev *chr,const uint8_t *buf,int len) {PtyChardev *s =PTY_CHARDEV (chr); +GPollFD pfd; +int rc; -if (!s->connected) { -return len; +if (s->connected) { +return io_channel_send (s->ioc, buf, len); } -return io_channel_send (s->ioc, buf, len); +chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 124
Quoted 10 lines
chardev/char-pty.c +10
-0+/* +* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write again: +*/ + pfd.fd =QIO_CHANNEL_FILE (s->ioc)->fd; + pfd.events =G_IO_OUT ; + pfd.revents =0 ; + rc =RETRY_ON_EINTR (g_poll (&pfd,1 ,0 )); +g_assert (rc >=0 ); +if (!(pfd.revents &G_IO_HUP ) && (pfd.revents &G_IO_OUT )) {Should (can?) we call
Quoted 3 lines
pty_chr_state(chr, 1);here ?
As far as I understood commit
f8278c7d74c6andf7ea2038bea04628, this is not possible anymore since the lock has been removed.Quoted 8 lines
Quoted 4 lines
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 126
Quoted 1 line
chardev/char-pty.c +1
-0+io_channel_send (s->ioc, buf, len);As it feels a little dirty to be sending data before setting the 'connected == 1' and thus issuing the '
CHR_EVENT_OPENED' eventI didn't find a really better solution so far. We could maybe introduce a buffer in the char-pty code and store the last second of guest output, but IMHO that's way more complex and thus somewhat ugly, too?
The orignal commit
f8278c7d74c6said[quote]
char-pty: remove the check for connection on write
Quoted 8 lines
This doesn't help much compared to the 1 second poll PTY timer. I can't think of a use case where this would help.
[/quote]We've now identified a use case where it is actually important.
IOW, there's a justification to revert both
f7ea2038bea04628andf8278c7d74c6, re-adding the locking and write update logic.
Indeed. But isn't it possible to watch for IO_OUT and get rid of the timer?
Other thing I question is whether the serial shouldn't return BUSY if the chardev is disconnected..
Quoted 94 lines · Marc-André Lureau
Hi
Quoted 91 lines · Daniel P. Berrangé
Quoted 71 lines · Thomas Huth
Quoted 63 lines · Daniel P. Berrangé
Quoted 49 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- chardev/char-pty.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)chardev/char-pty.c +5
-3@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr)static int char_pty_chr_write (Chardev *chr,const uint8_t *buf,int len) {PtyChardev *s =PTY_CHARDEV (chr); +GPollFD pfd; +int rc; -if (!s->connected) { -return len; +if (s->connected) { +return io_channel_send (s->ioc, buf, len); } -return io_channel_send (s->ioc, buf, len); +chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 124
Quoted 10 lines
chardev/char-pty.c +10
-0+/* +* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write again: +*/ + pfd.fd =QIO_CHANNEL_FILE (s->ioc)->fd; + pfd.events =G_IO_OUT ; + pfd.revents =0 ; + rc =RETRY_ON_EINTR (g_poll (&pfd,1 ,0 )); +g_assert (rc >=0 ); +if (!(pfd.revents &G_IO_HUP ) && (pfd.revents &G_IO_OUT )) {Should (can?) we call
Quoted 3 lines
pty_chr_state(chr, 1);here ?
As far as I understood commit
f8278c7d74c6andf7ea2038bea04628, this is not possible anymore since the lock has been removed.Quoted 8 lines
Quoted 4 lines
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 126
Quoted 1 line
chardev/char-pty.c +1
-0+io_channel_send (s->ioc, buf, len);As it feels a little dirty to be sending data before setting the 'connected == 1' and thus issuing the '
CHR_EVENT_OPENED' eventI didn't find a really better solution so far. We could maybe introduce a buffer in the char-pty code and store the last second of guest output, but IMHO that's way more complex and thus somewhat ugly, too?
The orignal commit
f8278c7d74c6said[quote]
char-pty: remove the check for connection on writeThis doesn't help much compared to the 1 second poll PTY timer. I can't think of a use case where this would help.
[/quote]
Quoted 6 lines
Quoted 4 lines
We've now identified a use case where it is actually important.
IOW, there's a justification to revert both
f7ea2038bea04628andf8278c7d74c6, re-adding the locking and write update logic.Indeed. But isn't it possible to watch for
IO_OUTand get rid of the timer?
It might be possible - Marc Hartmayer just sent me a draft patch today that uses qio_channel_add_watch() and gets rid of the timer ... I'll do some experiments with that and send it out if it works reliably.
Thomas
Quoted 97 lines · Thomas Huth
Quoted 96 lines · Marc-André Lureau
Hi
Quoted 93 lines · Daniel P. Berrangé
Quoted 73 lines · Thomas Huth
Quoted 65 lines · Daniel P. Berrangé
Quoted 53 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>---
chardev/char-pty.c| 22 +++++++++++++++++++---1 file changed, 19 insertions(+), 3 deletions(-)chardev/char-pty.c +8
-3@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr) static int char_pty_chr_write(Chardev *chr, const uint8_t *buf, int len) { PtyChardev *s = PTY_CHARDEV(chr); + GPollFD pfd; + int rc; - if (!s->connected) { - return len; + if (s->connected) { + return io_channel_send(s->ioc, buf, len); } - return io_channel_send(s->ioc, buf, len); + + /* +* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write Quoted 8 lines
again:
+ */
+ pfd.fd = QIO_CHANNEL_FILE(s->ioc)->fd;
+ pfd.events =G_IO_OUT;
+ pfd.revents = 0;
+ rc =RETRY_ON_EINTR(g_poll(&pfd, 1, 0));
+g_assert(rc >= 0);
+ if (!(pfd.revents &G_IO_HUP) && (pfd.revents &G_IO_OUT)) {Should (can?) we call
Quoted 3 lines
pty_chr_state(chr, 1);here ?
As far as I understood commit
f8278c7d74c6andf7ea2038bea04628, this is not possible anymore since the lock has been removed.Quoted 8 lines
Quoted 4 lines
Quoted 1 line
+
io_channel_send(s->ioc, buf, len);As it feels a little dirty to be sending data before setting the 'connected == 1' and thus issuing the '
CHR_EVENT_OPENED' eventI didn't find a really better solution so far. We could maybe introduce a buffer in the char-pty code and store the last second of guest output, but IMHO that's way more complex and thus somewhat ugly, too?
The orignal commit
f8278c7d74c6said[quote]
char-pty: remove the check for connection on writeThis doesn't help much compared to the 1 second poll PTY timer. I can't think of a use case where this would help. [/quote]
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
We've now identified a use case where it is actually important.
IOW, there's a justification to revert both
f7ea2038bea04628andf8278c7d74c6, re-adding the locking and write update logic.Indeed. But isn't it possible to watch for
IO_OUTand get rid of the timer?It might be possible - Marc Hartmayer just sent me a draft patch today that uses
qio_channel_add_watch()and gets rid of the timer ... I'll do some experiments with that and send it out if it works reliably.
I did quite a bunch of experiments with the code, but as far as I can see, it's not so straight forward: IO_OUT keeps being enabled once IO_HUP occurs since the kernel can still buffer some output before it de-asserts IO_OUT. So there is no easy way to check for a re-connection without some kind of regular polling, as far as I can see...
Well, maybe something like this might be possible: Add a watch for IO_HUP. Once HUP occurs, don't signal CHR_EVENT_CLOSED yet, but set a flag variable and in case the guest still writes something to the chardev and the flag is true, we write the bytes to the pty one by one (to avoid locking in case the buffer gets full) and check for IO_OUT via g_poll() after each write. Once we see IO_OUT being gone, we can signal CHR_EVENT_CLOSED and add a watch for IO_OUT being enabled again. Once that watch triggers, we know that the other side re-connected and we can signal CHR_EVENT_OPENED and continue normally. ... sounds all quite complicated, is it worth the effort? What do you think?
Thomas
Quoted 104 lines · Thomas Huth
Quoted 103 lines · Thomas Huth
Quoted 102 lines · Marc-André Lureau
Hi
Quoted 98 lines · Daniel P. Berrangé
Quoted 74 lines · Thomas Huth
Quoted 66 lines · Daniel P. Berrangé
Quoted 54 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>---
chardev/char-pty.c| 22 +++++++++++++++++++---1 file changed, 19 insertions(+), 3 deletions(-)chardev/char-pty.c +7
-3@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr) static int char_pty_chr_write(Chardev *chr, const uint8_t *buf, int len) { PtyChardev *s = PTY_CHARDEV(chr); + GPollFD pfd; + int rc; - if (!s->connected) { - return len; + if (s->connected) { + return io_channel_send(s->ioc, buf, len); } - return io_channel_send(s->ioc, buf, len); + + /* +* The other side might already be re-connected, but the timer might
+ * not have fired yet. So let's check here whether we can writeQuoted 8 lines
again:
+ */
+ pfd.fd = QIO_CHANNEL_FILE(s->ioc)->fd;
+ pfd.events =G_IO_OUT;
+ pfd.revents = 0;
+ rc =RETRY_ON_EINTR(g_poll(&pfd, 1, 0));
+g_assert(rc >= 0);
+ if (!(pfd.revents &G_IO_HUP) && (pfd.revents &G_IO_OUT)) {Should (can?) we call
Quoted 3 lines
pty_chr_state(chr, 1);here ?
As far as I understood commit
f8278c7d74c6andf7ea2038bea04628, this is not possible anymore since the lock has been removed.Quoted 8 lines
Quoted 4 lines
Quoted 1 line
+
io_channel_send(s->ioc, buf, len);As it feels a little dirty to be sending data before setting the 'connected == 1' and thus issuing the '
CHR_EVENT_OPENED' eventI didn't find a really better solution so far. We could maybe introduce a buffer in the char-pty code and store the last second of guest output, but IMHO that's way more complex and thus somewhat ugly, too?
The orignal commit
f8278c7d74c6said[quote]
char-pty: remove the check for connection on writeThis doesn't help much compared to the 1 second poll PTY timer. I can't think of a use case where this would help. [/quote]
We've now identified a use case where it is actually important.
IOW, there's a justification to revert both
f7ea2038bea04628andf8278c7d74c6, re-adding the locking and write update logic.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Indeed. But isn't it possible to watch for
IO_OUTand get rid of the timer?It might be possible - Marc Hartmayer just sent me a draft patch today that uses
qio_channel_add_watch()and gets rid of the timer ... I'll do some experiments with that and send it out if it works reliably.I did quite a bunch of experiments with the code, but as far as I can see, it's not so straight forward:
IO_OUTkeeps being enabled onceIO_HUPoccurs since the kernel can still buffer some output before it de-assertsIO_OUT. So there is no easy way to check for a re-connection without some kind of regular polling, as far as I can see...Well, maybe something like this might be possible: Add a watch for
IO_HUP. Once HUP occurs, don't signalCHR_EVENT_CLOSEDyet, but set a flag variable and in case the guest still writes something to the chardev and the flag is true, we write the bytes to the pty one by one (to avoid locking in case the buffer gets full) and check forIO_OUTviag_poll()after each write. Once we seeIO_OUTbeing gone, we can signalCHR_EVENT_CLOSEDand add a watch forIO_OUTbeing enabled again. Once that watch triggers, we know that the other side re-connected and we can signalCHR_EVENT_OPENEDand continue normally. ... sounds all quite complicated, is it worth the effort? What do you think?
Ping!
Marc-André, Daniel, any thoughts on this? Would such a rewrite be worth the effort? Or could we just go ahead with my simple patch here? Or would you prefer reverting the locking patches?
Thomas
Quoted 5 lines
Quoted 5 lines
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
Hi, we recently ran into this problem in a different scenario where we manually invoke QEMU and connect to the PTY before starting the machine. With the existing code, we have to add a one second delay before starting just to make sure that we capture the early boot message. We are now running a patched version of QEMU with pretty much the same change as yours and it seems to work well. We are very interested in seeing this change, or a variant thereof merged.
Quoted 26 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Quoted 5 lines
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- chardev/char-pty.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi Thomas,
(patch merged as commit 4f7689f0817)
Quoted 51 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>--- chardev/char-pty.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)chardev/char-pty.c +6
-3@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr)static int char_pty_chr_write (Chardev *chr,const uint8_t *buf,int len) {PtyChardev *s =PTY_CHARDEV (chr); +GPollFD pfd; +int rc; -if (!s->connected) { -return len; +if (s->connected) { +return io_channel_send (s->ioc, buf, len); } -return io_channel_send (s->ioc, buf, len); + +/*
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 126
Quoted 10 lines
chardev/char-pty.c +10
-0+* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write again: +*/ + pfd.fd =QIO_CHANNEL_FILE (s->ioc)->fd; + pfd.events =G_IO_OUT ; + pfd.revents =0 ; + rc =RETRY_ON_EINTR (g_poll (&pfd,1 ,0 )); +g_assert (rc >=0 ); +if (!(pfd.revents &G_IO_HUP ) && (pfd.revents &G_IO_OUT )) { +io_channel_send (s->ioc, buf, len);
Could io_channel_send() return -1 in this case, and if so is it OK to ignore it?
Quoted 6 lines
chardev/char-pty.c +3
-0+ } + +return len; }static GSource *pty_chr_add_watch (Chardev *chr,GIOCondition cond)
Quoted 37 lines · Philippe Mathieu-Daudé
Hi Thomas,
(patch merged as commit
4f7689f0817)Quoted 32 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non-blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>---
chardev/char-pty.c| 22 +++++++++++++++++++---1 file changed, 19 insertions(+), 3 deletions(-)
Quoted 12 lines
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 110
Quoted 12 lines
chardev/char-pty.c +2
-2@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr) static int char_pty_chr_write(Chardev *chr, const uint8_t *buf, int len) { PtyChardev *s = PTY_CHARDEV(chr); + GPollFD pfd; + int rc; - if (!s->connected) { - return len;
Old code discarded the bytes here if it was not able to send them ...
Quoted 17 lines
Quoted 6 lines
chardev/char-pty.c +4
-1+ if (s->connected) { + return io_channel_send(s->ioc, buf, len); } - return io_channel_send(s->ioc, buf, len); + + /*chardev/char-pty.c static void pty_chr_update_read_handler(Chardev *chr) · line 126
Quoted 10 lines
chardev/char-pty.c +10
-0+* The other side might already be re-connected, but the timer might +* not have fired yet. So let's check here whether we can write again: +*/ + pfd.fd = QIO_CHANNEL_FILE(s->ioc)->fd; + pfd.events = G_IO_OUT; + pfd.revents = 0; + rc = RETRY_ON_EINTR(g_poll(&pfd, 1, 0)); + g_assert(rc >= 0); + if (!(pfd.revents & G_IO_HUP) && (pfd.revents & G_IO_OUT)) { + io_channel_send(s->ioc, buf, len);
Quoted 1 line
Could
io_channel_send()return -1 in this case, and if so is it OK to ignore it?
... so now you get the same behavior here in case they could still not be sent.
Why do you ask, did you run into any problems with this new code here?
Thomas
Quoted 73 lines · Thomas Huth
Quoted 37 lines · Philippe Mathieu-Daudé
Hi Thomas,
(patch merged as commit
4f7689f0817)Quoted 32 lines · Thomas Huth
When starting a guest via libvirt with "virsh start --console ...", the first second of the console output is missing. This is especially annoying on s390x that only has a text console by default and no graphical output - if the bios fails to boot here, the information about what went wrong is completely lost.
One part of the problem (there is also some things to be done on the libvirt side) is that QEMU only checks with a 1 second timer whether the other side of the pty is already connected, so the first second of the console output is always lost.
This likely used to work better in the past, since the code once checked for a re-connection during write, but this has been removed in commit
f8278c7d74("char-pty: remove the check for connection on write") to avoid some locking.To ease the situation here at least a little bit, let's check with
g_poll()whether we could send out the data anyway, even if the connection has not been marked as "connected" yet. The file descriptor is marked as non- blocking anyway since commitfac6688a18("Do not hang on full PTY"), so this should not cause any trouble if the other side is not ready for receiving yet.With this patch applied, I can now successfully see the bios output of a s390x guest when running it with "virsh start --console" (with a patched version of virsh that fixes the remaining issues there, too).
Reported-by Marc Hartmayer <[email protected]>Signed-off-by Thomas Huth <[email protected]>---
chardev/char-pty.c| 22 +++++++++++++++++++---1 file changed, 19 insertions(+), 3 deletions(-)Quoted 13 lines
chardev/char-pty.c static void pty_chr_update_read_handler(Chardev · line 111
Quoted 13 lines
chardev/char-pty.c +2
-2@@ -106,11 +106,27 @@ static void pty_chr_update_read_handler(Chardev *chr) static int char_pty_chr_write(Chardev *chr, const uint8_t *buf, int len) { PtyChardev *s = PTY_CHARDEV(chr); + GPollFD pfd; + int rc; - if (!s->connected) { - return len;Old code discarded the bytes here if it was not able to send them ...
Quoted 19 lines
Quoted 18 lines
chardev/char-pty.c +5
-1+ if (s->connected) { + return io_channel_send(s->ioc, buf, len); } - return io_channel_send(s->ioc, buf, len); + + /* +* The other side might already be re-connected, but the timer might
+ * not have fired yet. So let's check here whether we can write again:
+ */
+ pfd.fd = QIO_CHANNEL_FILE(s->ioc)->fd;
+ pfd.events =G_IO_OUT;
+ pfd.revents = 0;
+ rc =RETRY_ON_EINTR(g_poll(&pfd, 1, 0));
+g_assert(rc >= 0);
+ if (!(pfd.revents &G_IO_HUP) && (pfd.revents &G_IO_OUT)) { +io_channel_send(s->ioc, buf, len);
Quoted 5 lines
Quoted 1 line
Could
io_channel_send()return -1 in this case, and if so is it OK to ignore it?... so now you get the same behavior here in case they could still not be sent.
Why do you ask, did you run into any problems with this new code here?
Peter found in chardev while testing a recent pl011 device patchset:
https://lore.kernel.org/qemu-devel/CAFEAcA_kEndvNtw4EHySXWwQPoGs029yAzZGGBcV=zGHaj7KUQ@mail.gmail.com/
For now I'm auditing our uses to understand better this subsystem, and am looking at the returned values, and noticed this case.
Thanks,
Phil.