Skip to contentSkip to conversation

io/channel-command: Delay the killing of the child after closing the pipe

PATCH In review Thomas Huth 3 participants 11 messages Feb 13, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 io/channel-command: Delay the killing of the child after closing the pipe

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration. The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +3 -3
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {
             error_setg(errp,
                        "Process %llu refused to die",
                        (unsigned long long)ioc->pid);
Signature · 1 line
1.8.3.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 4 lines · Thomas Huth

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Ok, yes,that makes sense.

Quoted 19 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -1
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;
         }
     } else if (ret == 0) {
-        if (step == 0) {

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 187

Quoted 10 lines
io/channel-command.c +3 -2
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {
             error_setg(errp,
                        "Process %llu refused to die",
                        (unsigned long long)ioc->pid);

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw
In reply to Thomas Huth
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Who is to say that's enough for a scp or gzip or the like?

Dave

Quoted 6 lines
io/channel-command.c +0 -0
             error_setg(errp,
                        "Process %llu refused to die",
                        (unsigned long long)ioc->pid);
Signature · 1 line
1.8.3.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#3 lore raw
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Quoted 1 line

Who is to say that's enough for a scp or gzip or the like?

We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
Quoted 47 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Quoted 6 lines
Quoted 1 line

Who is to say that's enough for a scp or gzip or the like?

We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.

Yeh that's better; although when would we call shutdown or close on it?

Dave

Quoted 7 lines

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#5 lore raw
Quoted 48 lines · Dr. David Alan Gilbert
Quoted 47 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Who is to say that's enough for a scp or gzip or the like?

We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.

Yeh that's better; although when would we call shutdown or close on it?

Doesn't QEMU alredy use shutdown() during the right part of migration, or is that only wrt post-copy ?

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw
Quoted 56 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 53 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Quoted 1 line

Who is to say that's enough for a scp or gzip or the like?

We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.

Quoted 4 lines
Quoted 1 line

Yeh that's better; although when would we call shutdown or close on it?

Doesn't QEMU alredy use shutdown() during the right part of migration, or is that only wrt post-copy ?

We only use it for cancel and errors, not during the normal behaviour.

Dave

Quoted 8 lines

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#7 lore raw
Quoted 57 lines · Dr. David Alan Gilbert
Quoted 56 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 53 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Quoted 1 line

Who is to say that's enough for a scp or gzip or the like?

We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yeh that's better; although when would we call shutdown or close on it?

Doesn't QEMU alredy use shutdown() during the right part of migration, or is that only wrt post-copy ?

We only use it for cancel and errors, not during the normal behaviour.

So we could do with shutdown() for sake of post-copy anyway, but for normal behaviour maybe the right answer is for close() to just wait a real long time for the child app to exit ? If we close the pipes, and then wait 5 seconds or more before giving up ?

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#8 lore raw
Quoted 63 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 62 lines · Dr. David Alan Gilbert
Quoted 60 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 48 lines · Dr. David Alan Gilbert
Quoted 47 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Who is to say that's enough for a scp or gzip or the like?

We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.

Yeh that's better; although when would we call shutdown or close on it?

Doesn't QEMU alredy use shutdown() during the right part of migration, or is that only wrt post-copy ?

Quoted 6 lines
Quoted 1 line

We only use it for cancel and errors, not during the normal behaviour.

So we could do with shutdown() for sake of post-copy anyway, but for normal behaviour maybe the right answer is for close() to just wait a real long time for the child app to exit ? If we close the pipes, and then wait 5 seconds or more before giving up ?

Yes, I'm happier with a much longer arbitrary value than a short arbitrary value; but I do wonder if there's any real need to kill it.

Dave

Quoted 6 lines

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#9 lore raw
Quoted 66 lines · Dr. David Alan Gilbert
Quoted 65 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 63 lines · Dr. David Alan Gilbert
Quoted 56 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 53 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Quoted 1 line

Who is to say that's enough for a scp or gzip or the like?

We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.

Quoted 4 lines
Quoted 1 line

Yeh that's better; although when would we call shutdown or close on it?

Doesn't QEMU alredy use shutdown() during the right part of migration, or is that only wrt post-copy ?

We only use it for cancel and errors, not during the normal behaviour.

Quoted 7 lines
Quoted 4 lines

So we could do with shutdown() for sake of post-copy anyway, but for normal behaviour maybe the right answer is for close() to just wait a real long time for the child app to exit ? If we close the pipes, and then wait 5 seconds or more before giving up ?

Yes, I'm happier with a much longer arbitrary value than a short arbitrary value; but I do wonder if there's any real need to kill it.

If we don't kill it, then if it gets stuck for some reason it will live forever. If we don't kill it but just close the FD, then we still need to waitpid at some point otherwise we get a zombie - unless we decide to daemonize the child instead ?

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#10 lore raw
Quoted 72 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 71 lines · Dr. David Alan Gilbert
Quoted 69 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 57 lines · Dr. David Alan Gilbert
Quoted 56 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 55 lines · Dr. David Alan Gilbert
Quoted 53 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 46 lines · Dr. David Alan Gilbert
Quoted 4 lines · * Thomas Huth ([email protected]) wrote:

We are currently facing some migration failure on s390x when running certain avocado tests, e.g. when running the test type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec. This test is using 'migrate -d "exec:nc localhost 5200"' for the migration.

Quoted 8 lines

The problem is detected at the receiving side, where the migration stream apparently ends too early. However, the cause for the problem is the sending side: After writing the migration stream into the pipe to netcat, the source QEMU calls qio_channel_command_close() which closes the pipe and immediately (!) kills the child process afterwards. So if the sending netcat did not read the final bytes from the pipe yet, or if it did not manage to send out all its buffers yet, it is killed before the whole migration stream is passed to the destination side.

Thanks for tracking that down!

Quoted 16 lines

To ease the situation at least a little bit, we should give the child process at least some few more time slices before we kill it with SIGTERM and then with SIGKILL. With this change, the avocado test now succeeds here in 10 out of 10 runs.

---
 io/channel-command.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
io/channel-command.c +0 -0
index 319c5ed..f64db3e 100644
@@ -177,11 +177,11 @@ static int qio_channel_command_abort(QIOChannelCommand *ioc,
             return -1;

io/channel-command.c static int qio_channel_command_abort(QIOChannelCommand *ioc, · line 184

Quoted 10 lines
io/channel-command.c +3 -3
         }
     } else if (ret == 0) {
-        if (step == 0) {
+        if (step == 4) {
             kill(ioc->pid, SIGTERM);
-        } else if (step == 1) {
+        } else if (step == 8) {
             kill(ioc->pid, SIGKILL);
-        } else {
+        } else if (step >= 9) {

Hmm. This seems pretty arbitrary; if I understand correctly you're saying it'll get a SIGTERM after 4 (arbitrary) * 10ms (arbitrary).

Quoted 1 line

Who is to say that's enough for a scp or gzip or the like?

We could conceivably implement the qio_channel_shutdown() operation for the QIOChannelCommand class. It would merely close the FD to the child process, but leave it running. That would give it time to read any data still in the pipe from QEMU IIUC.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yeh that's better; although when would we call shutdown or close on it?

Doesn't QEMU alredy use shutdown() during the right part of migration, or is that only wrt post-copy ?

We only use it for cancel and errors, not during the normal behaviour.

So we could do with shutdown() for sake of post-copy anyway, but for normal behaviour maybe the right answer is for close() to just wait a real long time for the child app to exit ? If we close the pipes, and then wait 5 seconds or more before giving up ?

Quoted 7 lines
Quoted 2 lines

Yes, I'm happier with a much longer arbitrary value than a short arbitrary value; but I do wonder if there's any real need to kill it.

If we don't kill it, then if it gets stuck for some reason it will live forever. If we don't kill it but just close the FD, then we still need to waitpid at some point otherwise we get a zombie - unless we decide to daemonize the child instead ?

Well we used to rely on popen/pclose which I think just waited for it.

You could rely on migration_cancel calling shutdown and causing that to do the kill, but always waiting in the normal course of things; which is probably OK unless it decides to hang right at the end.

Dave

Quoted 6 lines

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK