io/channel-command: Delay the killing of the child after closing the pipe
PATCH 1/1 io/channel-command: Delay the killing of the child after closing the pipe
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration. The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +3 -3
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ; } }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {error_setg (errp,"Process %llu refused to die" , (unsigned long long )ioc->pid);
Signature · 1 line
1.8.3.1
Quoted 4 lines · Thomas Huth
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.
Ok, yes,that makes sense.
Quoted 19 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-1@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ; } }else if (ret ==0 ) { -if (step ==0 ) {
io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 187
Quoted 10 lines
io/channel-command.c +3
-2+if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {error_setg (errp,"Process %llu refused to die" , (unsigned long long )ioc->pid);
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.
Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;
io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {
Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Who is to say that's enough for a scp or gzip or the like?
Dave
Quoted 6 lines
io/channel-command.c +0
-0error_setg (errp,"Process %llu refused to die" , (unsigned long long )ioc->pid);Signature · 1 line
1.8.3.1
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Quoted 1 line
Who is to say that's enough for a scp or gzip or the like?
We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 47 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Quoted 6 lines
Quoted 1 line
Who is to say that's enough for a scp or gzip or the like?
We could conceivably implement the
qio_channel_shutdown()operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.
Yeh that's better; although when would we call shutdown or close on it?
Dave
Quoted 7 lines
Regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 48 lines · Dr. David Alan Gilbert
Quoted 47 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Who is to say that's enough for a scp or gzip or the like?
We could conceivably implement the
qio_channel_shutdown()operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.Yeh that's better; although when would we call shutdown or close on it?
Doesn't QEMU alredy use shutdown() during the right part of migration, or is that only wrt post-copy ?
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 56 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 53 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Quoted 1 line
Who is to say that's enough for a scp or gzip or the like?
We could conceivably implement the
qio_channel_shutdown()operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.
Quoted 4 lines
Quoted 1 line
Yeh that's better; although when would we call shutdown or close on it?
Doesn't QEMU alredy use
shutdown()during the right part of migration, or is that only wrt post-copy ?
We only use it for cancel and errors, not during the normal behaviour.
Dave
Quoted 8 lines
Regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 57 lines · Dr. David Alan Gilbert
Quoted 56 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 53 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Quoted 1 line
Who is to say that's enough for a scp or gzip or the like?
We could conceivably implement the
qio_channel_shutdown()operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Yeh that's better; although when would we call shutdown or close on it?
Doesn't QEMU alredy use
shutdown()during the right part of migration, or is that only wrt post-copy ?We only use it for cancel and errors, not during the normal behaviour.
So we could do with shutdown() for sake of post-copy anyway, but for normal behaviour maybe the right answer is for close() to just wait a real long time for the child app to exit ? If we close the pipes, and then wait 5 seconds or more before giving up ?
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 63 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 62 lines · Dr. David Alan Gilbert
Quoted 60 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 48 lines · Dr. David Alan Gilbert
Quoted 47 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Who is to say that's enough for a scp or gzip or the like?
We could conceivably implement the
qio_channel_shutdown()operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.Yeh that's better; although when would we call shutdown or close on it?
Doesn't QEMU alredy use
shutdown()during the right part of migration, or is that only wrt post-copy ?
Quoted 6 lines
Quoted 1 line
We only use it for cancel and errors, not during the normal behaviour.
So we could do with
shutdown()for sake of post-copy anyway, but for normal behaviour maybe the right answer is forclose()to just wait a real long time for the child app to exit ? If we close the pipes, and then wait 5 seconds or more before giving up ?
Yes, I'm happier with a much longer arbitrary value than a short arbitrary value; but I do wonder if there's any real need to kill it.
Dave
Quoted 6 lines
Regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 66 lines · Dr. David Alan Gilbert
Quoted 65 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 63 lines · Dr. David Alan Gilbert
Quoted 56 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 53 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Quoted 1 line
Who is to say that's enough for a scp or gzip or the like?
We could conceivably implement the
qio_channel_shutdown()operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.Quoted 4 lines
Quoted 1 line
Yeh that's better; although when would we call shutdown or close on it?
Doesn't QEMU alredy use
shutdown()during the right part of migration, or is that only wrt post-copy ?We only use it for cancel and errors, not during the normal behaviour.
Quoted 7 lines
Quoted 4 lines
So we could do with
shutdown()for sake of post-copy anyway, but for normal behaviour maybe the right answer is forclose()to just wait a real long time for the child app to exit ? If we close the pipes, and then wait 5 seconds or more before giving up ?Yes, I'm happier with a much longer arbitrary value than a short arbitrary value; but I do wonder if there's any real need to kill it.
If we don't kill it, then if it gets stuck for some reason it will live forever. If we don't kill it but just close the FD, then we still need to waitpid at some point otherwise we get a zombie - unless we decide to daemonize the child instead ?
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 72 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 71 lines · Dr. David Alan Gilbert
Quoted 69 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 57 lines · Dr. David Alan Gilbert
Quoted 56 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 53 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:
We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.
Quoted 8 lines
The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls
qio_channel_command_close()which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.Thanks for tracking that down!
Quoted 16 lines
To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.
Signed-off-by Thomas Huth <[email protected]>--- io/channel-command.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)io/channel-command.c +0
-0@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,return -1 ;io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184
Quoted 10 lines
io/channel-command.c +3
-3} }else if (ret ==0 ) { -if (step ==0 ) { +if (step ==4 ) {kill (ioc->pid,SIGTERM ); - }else if (step ==1 ) { + }else if (step ==8 ) {kill (ioc->pid,SIGKILL ); - }else { + }else if (step >=9 ) {Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).
Quoted 1 line
Who is to say that's enough for a scp or gzip or the like?
We could conceivably implement the
qio_channel_shutdown()operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Yeh that's better; although when would we call shutdown or close on it?
Doesn't QEMU alredy use
shutdown()during the right part of migration, or is that only wrt post-copy ?We only use it for cancel and errors, not during the normal behaviour.
So we could do with
shutdown()for sake of post-copy anyway, but for normal behaviour maybe the right answer is forclose()to just wait a real long time for the child app to exit ? If we close the pipes, and then wait 5 seconds or more before giving up ?
Quoted 7 lines
Quoted 2 lines
Yes, I'm happier with a much longer arbitrary value than a short arbitrary value; but I do wonder if there's any real need to kill it.
If we don't kill it, then if it gets stuck for some reason it will live forever. If we don't kill it but just close the FD, then we still need to waitpid at some point otherwise we get a zombie - unless we decide to daemonize the child instead ?
Well we used to rely on popen/pclose which I think just waited for it.
You could rely on migration_cancel calling shutdown and causing that to do the kill, but always waiting in the normal course of things; which is probably OK unless it decides to hang right at the end.
Dave
Quoted 6 lines
Regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK