Skip to contentSkip to conversation

some migration bugs

PATCH In review Vladimir Sementsov-Ogievskiy 7 participants 32 messages Feb 25, 2017 – Apr 26, 2017 Block layer

lore
Share & tools
lore
#0 lore raw

cover some migration bugs

Hi all!

Here are some migration related bugs, two about INACTIVE bdses and one use-after-free.

I'm absolutely not sure, that these bugs should be fixed like I'm fixing, but problem definitely exists.

Reset in stopped state is strange case, may be such usage should be restricted.
About INACTIVE - looks like it should be a separate run-state, not only bdrv-flag.
Situation with migration state, which is global, but is set/reset/changed in not controlled manner is not very good too..

Vladimir Sementsov-Ogievskiy (4):
iotests: add migration corner cases test
qmp-cont: invalidate on RUN_STATE_PRELAUNCH
savevm: fix savevm after migration
migration: fix use-after-free of to_dst_file

 block/snapshot.c           |  3 +-
 migration/savevm.c         | 16 +++++++++++
 qmp.c                      |  3 +-
 tests/qemu-iotests/175     | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  5 ++++
 tests/qemu-iotests/group   |  1 +
 6 files changed, 97 insertions(+), 2 deletions(-)
 create mode 100644 tests/qemu-iotests/175
 create mode 100644 tests/qemu-iotests/175.out
Signature · 1 line
2.11.1
#1 lore raw

PATCH 1/4 iotests: add migration corner cases test

---
 tests/qemu-iotests/175     | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  5 ++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 77 insertions(+)
 create mode 100644 tests/qemu-iotests/175
 create mode 100644 tests/qemu-iotests/175.out
tests/qemu-iotests/175 new file +71 -0
new file mode 100644
index 0000000000..ef86c70db5
@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+#
+# Test migration corner-cases
+#
+# Copyright (C) Vladimir Sementsov-Ogievskiy 2017
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import iotests
+import time
+from iotests import qemu_img
+
+disk = os.path.join(iotests.test_dir, 'disk')
+
+class TestMigrationCornerCases(iotests.QMPTestCase):
+    def setUp(self):
+        qemu_img('create', '-f', iotests.imgfmt, disk, '10M')
+        self.vm = iotests.VM().add_drive(disk)
+        self.vm.launch()
+
+    def tearDown(self):
+        self.vm.shutdown()
+        os.remove(disk)
+
+    def test_migrate_reset_cont_write(self):
+        result = self.vm.qmp('migrate', uri='exec:cat>/dev/null')
+        self.assert_qmp(result, 'return', {})
+        time.sleep(4)
+
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='system_reset')
+        self.assert_qmp(result, 'return', '')
+
+        result = self.vm.qmp('cont')
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='qemu-io drive0 "write 0 512"')
+        self.assert_qmp(result, 'return', '')
+
+    def test_migrate_savevm(self):
+        result = self.vm.qmp('migrate', uri='exec:cat>/dev/null')
+        self.assert_qmp(result, 'return', {})
+        time.sleep(4)
+
+        result = self.vm.qmp('human-monitor-command', command_line='savevm')
+        self.assert_qmp(result, 'return', '')
+
+    def test_savevm_set_speed_savevm(self):
+        for i in range(10):
+            result = self.vm.qmp('human-monitor-command', command_line='savevm')
+            self.assert_qmp(result, 'return', '')
+
+            result = self.vm.qmp('migrate_set_speed', value=9223372036853727232)
+            self.assert_qmp(result, 'return', {})
+
+if __name__ == '__main__':
+    iotests.main()
tests/qemu-iotests/175.out new file +5 -0
new file mode 100644
index 0000000000..8d7e996700
@@ -0,0 +1,5 @@
+...
+----------------------------------------------------------------------
+Ran 3 tests
+
+OK
tests/qemu-iotests/group +1 -0
index 985b9a6a36..1f4bf03185 100644
@@ -167,3 +167,4 @@
 172 auto
 173 rw auto
 174 auto
+175 auto quick
Signature · 1 line
2.11.1
#2 lore raw

Re: [Qemu-devel] [PATCH 1/4] iotests: add migration corner cases test

Quoted 46 lines · Vladimir Sementsov-Ogievskiy
---
 tests/qemu-iotests/175     | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  5 ++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 77 insertions(+)
 create mode 100644 tests/qemu-iotests/175
 create mode 100644 tests/qemu-iotests/175.out
tests/qemu-iotests/175 new file +31 -0
new file mode 100644
index 0000000000..ef86c70db5
@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+#
+# Test migration corner-cases
+#
+# Copyright (C) Vladimir Sementsov-Ogievskiy 2017
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import iotests
+import time
+from iotests import qemu_img
+
+disk = os.path.join(iotests.test_dir, 'disk')
+
+class TestMigrationCornerCases(iotests.QMPTestCase):
+    def setUp(self):
+        qemu_img('create', '-f', iotests.imgfmt, disk, '10M')
+        self.vm = iotests.VM().add_drive(disk)

tests/qemu-iotests/175 line 41

Quoted 10 lines
tests/qemu-iotests/175 new file +10 -0
+        self.vm.launch()
+
+    def tearDown(self):
+        self.vm.shutdown()
+        os.remove(disk)
+
+    def test_migrate_reset_cont_write(self):
+        result = self.vm.qmp('migrate', uri='exec:cat>/dev/null')
+        self.assert_qmp(result, 'return', {})
+        time.sleep(4)

Should you query the migration status instead of blink sleep?

Quoted 6 lines
tests/qemu-iotests/175 new file +6 -0
+
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='system_reset')
+        self.assert_qmp(result, 'return', '')
+
+        result = self.vm.qmp('cont')

tests/qemu-iotests/175 line 57

Quoted 10 lines
tests/qemu-iotests/175 new file +10 -0
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='qemu-io drive0 "write 0 512"')
+        self.assert_qmp(result, 'return', '')
+
+    def test_migrate_savevm(self):
+        result = self.vm.qmp('migrate', uri='exec:cat>/dev/null')
+        self.assert_qmp(result, 'return', {})
+        time.sleep(4)

Ditto.

Quoted 30 lines
tests/qemu-iotests/175 new file +14 -0
+
+        result = self.vm.qmp('human-monitor-command', command_line='savevm')
+        self.assert_qmp(result, 'return', '')
+
+    def test_savevm_set_speed_savevm(self):
+        for i in range(10):
+            result = self.vm.qmp('human-monitor-command', command_line='savevm')
+            self.assert_qmp(result, 'return', '')
+
+            result = self.vm.qmp('migrate_set_speed', value=9223372036853727232)
+            self.assert_qmp(result, 'return', {})
+
+if __name__ == '__main__':
+    iotests.main()
tests/qemu-iotests/175.out new file +5 -0
new file mode 100644
index 0000000000..8d7e996700
@@ -0,0 +1,5 @@
+...
+----------------------------------------------------------------------
+Ran 3 tests
+
+OK
tests/qemu-iotests/group +0 -0
index 985b9a6a36..1f4bf03185 100644

tests/qemu-iotests/group line 170

Quoted 4 lines
tests/qemu-iotests/group +1 -0
@@ -167,3 +167,4 @@
 172 auto
 173 rw auto
 174 auto
+175 auto quick

Not sure it still qualifies as "quick" given the two "time.sleep(4)", but more importantly I wonder if the sleep should be improved.

Fam

#3 lore raw

Re: [Qemu-devel] [PATCH 1/4] iotests: add migration corner cases test

In reply to Vladimir Sementsov-Ogievskiy
Quoted 1 line · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:

If I understand this correctly, these are tests that currently fail and are fixed by the following patches; in which case it needs reordering to keep stuff bisectable.

Dave

Quoted 108 lines
---
 tests/qemu-iotests/175     | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  5 ++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 77 insertions(+)
 create mode 100644 tests/qemu-iotests/175
 create mode 100644 tests/qemu-iotests/175.out
tests/qemu-iotests/175 new file +71 -0
new file mode 100644
index 0000000000..ef86c70db5
@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+#
+# Test migration corner-cases
+#
+# Copyright (C) Vladimir Sementsov-Ogievskiy 2017
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import iotests
+import time
+from iotests import qemu_img
+
+disk = os.path.join(iotests.test_dir, 'disk')
+
+class TestMigrationCornerCases(iotests.QMPTestCase):
+    def setUp(self):
+        qemu_img('create', '-f', iotests.imgfmt, disk, '10M')
+        self.vm = iotests.VM().add_drive(disk)
+        self.vm.launch()
+
+    def tearDown(self):
+        self.vm.shutdown()
+        os.remove(disk)
+
+    def test_migrate_reset_cont_write(self):
+        result = self.vm.qmp('migrate', uri='exec:cat>/dev/null')
+        self.assert_qmp(result, 'return', {})
+        time.sleep(4)
+
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='system_reset')
+        self.assert_qmp(result, 'return', '')
+
+        result = self.vm.qmp('cont')
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm.qmp('human-monitor-command',
+                             command_line='qemu-io drive0 "write 0 512"')
+        self.assert_qmp(result, 'return', '')
+
+    def test_migrate_savevm(self):
+        result = self.vm.qmp('migrate', uri='exec:cat>/dev/null')
+        self.assert_qmp(result, 'return', {})
+        time.sleep(4)
+
+        result = self.vm.qmp('human-monitor-command', command_line='savevm')
+        self.assert_qmp(result, 'return', '')
+
+    def test_savevm_set_speed_savevm(self):
+        for i in range(10):
+            result = self.vm.qmp('human-monitor-command', command_line='savevm')
+            self.assert_qmp(result, 'return', '')
+
+            result = self.vm.qmp('migrate_set_speed', value=9223372036853727232)
+            self.assert_qmp(result, 'return', {})
+
+if __name__ == '__main__':
+    iotests.main()
tests/qemu-iotests/175.out new file +5 -0
new file mode 100644
index 0000000000..8d7e996700
@@ -0,0 +1,5 @@
+...
+----------------------------------------------------------------------
+Ran 3 tests
+
+OK
tests/qemu-iotests/group +1 -0
index 985b9a6a36..1f4bf03185 100644
@@ -167,3 +167,4 @@
 172 auto
 173 rw auto
 174 auto
+175 auto quick
Signature · 1 line
2.11.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#4 lore raw

PATCH 2/4 qmp-cont: invalidate on RUN_STATE_PRELAUNCH

In reply to Vladimir Sementsov-Ogievskiy

We must invalidate on RUN_STATE_PRELAUNCH too, as it is available through qmp_system_reset from RUN_STATE_POSTMIGRATE. Otherwise, we will come to

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

       Assertion `!(bs->open_flags & 0x0800)' failed.

on the first write after vm start.

---
 qmp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
qmp.c +2 -1
index dfaabac1a6..e61795d033 100644
@@ -198,7 +198,8 @@ void qmp_cont(Error **errp)
     /* Continuing after completed migration. Images have been inactivated to
      * allow the destination to take control. Need to get control back now. */
     if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||
-        runstate_check(RUN_STATE_POSTMIGRATE))
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
     {
         bdrv_invalidate_cache_all(&local_err);
         if (local_err) {
Signature · 1 line
2.11.1
Reviewed-by on v1 2/4
#5 lore raw

Re: [Qemu-devel] [PATCH 2/4] qmp-cont: invalidate on RUN_STATE_PRELAUNCH

Quoted 31 lines · Vladimir Sementsov-Ogievskiy

We must invalidate on RUN_STATE_PRELAUNCH too, as it is available through qmp_system_reset from RUN_STATE_POSTMIGRATE. Otherwise, we will come to

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

       Assertion `!(bs->open_flags & 0x0800)' failed.

on the first write after vm start.

---
 qmp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
qmp.c +2 -1
index dfaabac1a6..e61795d033 100644
@@ -198,7 +198,8 @@ void qmp_cont(Error **errp)
     /* Continuing after completed migration. Images have been inactivated to
      * allow the destination to take control. Need to get control back now. */
     if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||
-        runstate_check(RUN_STATE_POSTMIGRATE))
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
     {
         bdrv_invalidate_cache_all(&local_err);
         if (local_err) {
Signature · 1 line
2.11.1
#6 lore raw

Re: [Qemu-devel] [PATCH 2/4] qmp-cont: invalidate on RUN_STATE_PRELAUNCH

In reply to Vladimir Sementsov-Ogievskiy

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 10 lines

We must invalidate on RUN_STATE_PRELAUNCH too, as it is available through qmp_system_reset from RUN_STATE_POSTMIGRATE. Otherwise, we will come to

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

       Assertion `!(bs->open_flags & 0x0800)' failed.

on the first write after vm start.

Wouldn't it make more sense to invalidate in qmp_system_reset() where the migration states are left?

Or maybe BDRV_O_INACTIVE could even be tied directly to runstates? Not sure how realistic this one is, but if we start adding invalidate_cache calls all over the place, maybe that's a sign that we need to look for a more central place.

Kevin

Quoted 20 lines
 qmp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
qmp.c +2 -1
index dfaabac1a6..e61795d033 100644
@@ -198,7 +198,8 @@ void qmp_cont(Error **errp)
     /* Continuing after completed migration. Images have been inactivated to
      * allow the destination to take control. Need to get control back now. */
     if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||
-        runstate_check(RUN_STATE_POSTMIGRATE))
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
     {
         bdrv_invalidate_cache_all(&local_err);
         if (local_err) {
Signature · 1 line
2.11.1
#7 lore raw

Re: [Qemu-devel] [PATCH 2/4] qmp-cont: invalidate on RUN_STATE_PRELAUNCH

Quoted 10 lines · Kevin Wolf

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 9 lines

We must invalidate on RUN_STATE_PRELAUNCH too, as it is available through qmp_system_reset from RUN_STATE_POSTMIGRATE. Otherwise, we will come to

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

        Assertion `!(bs->open_flags & 0x0800)' failed.

on the first write after vm start.

Quoted 8 lines
Quoted 1 line

Wouldn't it make more sense to invalidate in qmp_system_reset() where the migration states are left?

Or maybe BDRV_O_INACTIVE could even be tied directly to runstates? Not sure how realistic this one is, but if we start adding invalidate_cache calls all over the place, maybe that's a sign that we need to look for a more central place.

I've proposed it in cover letter) These bugs and my fixes are just show that something should be rethought.. I don't claim that these fixes are true way, they are just the simplest.

Quoted 22 lines

Kevin

Quoted 20 lines
  qmp.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
qmp.c +2 -1
index dfaabac1a6..e61795d033 100644
@@ -198,7 +198,8 @@ void qmp_cont(Error **errp)
      /* Continuing after completed migration. Images have been inactivated to
       * allow the destination to take control. Need to get control back now. */
      if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||
-        runstate_check(RUN_STATE_POSTMIGRATE))
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
      {
          bdrv_invalidate_cache_all(&local_err);
          if (local_err) {
Signature · 1 line
2.11.1
Signature · 2 lines
Best regards,
Vladimir
#8 lore raw

Re: [Qemu-devel] [PATCH 2/4] qmp-cont: invalidate on RUN_STATE_PRELAUNCH

Am 07.03.2017 um 11:11 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 15 lines
Quoted 14 lines · Kevin Wolf

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 10 lines

We must invalidate on RUN_STATE_PRELAUNCH too, as it is available through qmp_system_reset from RUN_STATE_POSTMIGRATE. Otherwise, we will come to

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

       Assertion `!(bs->open_flags & 0x0800)' failed.

on the first write after vm start.

Wouldn't it make more sense to invalidate in qmp_system_reset() where the migration states are left?

Quoted 8 lines
Quoted 4 lines

Or maybe BDRV_O_INACTIVE could even be tied directly to runstates? Not sure how realistic this one is, but if we start adding invalidate_cache calls all over the place, maybe that's a sign that we need to look for a more central place.

I've proposed it in cover letter) These bugs and my fixes are just show that something should be rethought.. I don't claim that these fixes are true way, they are just the simplest.

Ah, sorry, I read the cover letter a while ago, but not again before reading the patches now. You're right that it's something that needs to be addressed in some way. Your patches may be simple, but I think there is no hope to ever get it completely correct with that approach because there are so many cases (basically each QMP command is one case).

The cover letter seems to propose that INACTIVE could be a runstate. I don't think that's quite right, because we already have many runstates that would qualify as inactive in this sense, but that still need to be separate. But possibly we can have a mapping from runstates to the inactive flag.

One (implementation) problem with directly coupling inactive with runstates is that runstate_set() can't fail at the moment, but bdrv_inactivate/invalidate_cache() can.

Kevin

#9 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH 2/4] qmp-cont: invalidate on RUN_STATE_PRELAUNCH

In reply to Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Kevin Wolf

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 10 lines

We must invalidate on RUN_STATE_PRELAUNCH too, as it is available through qmp_system_reset from RUN_STATE_POSTMIGRATE. Otherwise, we will come to

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

        Assertion `!(bs->open_flags & 0x0800)' failed.

on the first write after vm start.

Wouldn't it make more sense to invalidate in qmp_system_reset() where the migration states are left?

Quoted 8 lines
Quoted 4 lines

Or maybe BDRV_O_INACTIVE could even be tied directly to runstates? Not sure how realistic this one is, but if we start adding invalidate_cache calls all over the place, maybe that's a sign that we need to look for a more central place.

I've proposed it in cover letter) These bugs and my fixes are just show that something should be rethought.. I don't claim that these fixes are true way, they are just the simplest.

Hi Vladimir,

I wonder if you have a new version of your patch ("qmp-cont: invalidate on RUN_STATE_PRELAUNCH").

Hailiang Zhang tells me on this the below thread that your patch fixes the issue:

http://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg03925.html
-- [QEMU-2.8] Source QEMU crashes with: "bdrv_co_pwritev: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed"

Signature · 1 line
/kashyap
#10 lore raw

Re: [Qemu-devel] [Qemu-block] [PATCH 2/4] qmp-cont: invalidate on RUN_STATE_PRELAUNCH

Quoted 28 lines · Kashyap Chamarthy
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Kevin Wolf

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 10 lines

We must invalidate on RUN_STATE_PRELAUNCH too, as it is available through qmp_system_reset from RUN_STATE_POSTMIGRATE. Otherwise, we will come to

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

         Assertion `!(bs->open_flags & 0x0800)' failed.

on the first write after vm start.

Wouldn't it make more sense to invalidate in qmp_system_reset() where the migration states are left?

Quoted 7 lines
Quoted 4 lines

Or maybe BDRV_O_INACTIVE could even be tied directly to runstates? Not sure how realistic this one is, but if we start adding invalidate_cache calls all over the place, maybe that's a sign that we need to look for a more central place.

I've proposed it in cover letter) These bugs and my fixes are just show that something should be rethought.. I don't claim that these fixes are true way, they are just the simplest.

Hi Vladimir,

I wonder if you have a new version of your patch ("qmp-cont: invalidate on RUN_STATE_PRELAUNCH").

Quoted 7 lines

Hailiang Zhang tells me on this the below thread that your patch fixes the issue:

   http://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg03925.html
   -- [QEMU-2.8] Source QEMU crashes with: "bdrv_co_pwritev: Assertion
   `!(bs->open_flags & BDRV_O_INACTIVE)' failed"

No, I haven't new version, discussion is unfinished about true way of doing it. This version is ok with the case it fixes.

Signature · 2 lines
Best regards,
Vladimir
#11 lore raw

PATCH 3/4 savevm: fix savevm after migration

In reply to Vladimir Sementsov-Ogievskiy

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:
Assertion `!(bs->open_flags & 0x0800)' failed.

---
 block/snapshot.c   |  3 ++-
 migration/savevm.c | 11 +++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)
block/snapshot.c +2 -1
index bf5c2ca5e1..256d06ac9f 100644
@@ -145,7 +145,8 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
 int bdrv_can_snapshot(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
-    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
+    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs) ||
+        (bs->open_flags & BDRV_O_INACTIVE)) {
         return 0;
     }
 
migration/savevm.c +11 -0
index 5ecd264134..75e56d2d07 100644
@@ -2068,6 +2068,17 @@ int save_vmstate(Monitor *mon, const char *name)
     Error *local_err = NULL;
     AioContext *aio_context;
 
+    if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
+    {
+        bdrv_invalidate_cache_all(&local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            return -EINVAL;
+        }
+    }
+
     if (!bdrv_all_can_snapshot(&bs)) {
         monitor_printf(mon, "Device '%s' is writable but does not "
                        "support snapshots.\n", bdrv_get_device_name(bs));
Signature · 1 line
2.11.1
#12 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Quoted 16 lines · Vladimir Sementsov-Ogievskiy

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.
---
 block/snapshot.c   |  3 ++-
 migration/savevm.c | 11 +++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)
block/snapshot.c +0 -0
index bf5c2ca5e1..256d06ac9f 100644

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 152

Quoted 9 lines
block/snapshot.c +2 -1
@@ -145,7 +145,8 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
 int bdrv_can_snapshot(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
-    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
+    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs) ||
+        (bs->open_flags & BDRV_O_INACTIVE)) {
         return 0;
     }
 

at my opinion we do not need this hunk. It will result in a wrong thing.

Quoted 22 lines
migration/savevm.c +11 -0
index 5ecd264134..75e56d2d07 100644
@@ -2068,6 +2068,17 @@ int save_vmstate(Monitor *mon, const char *name)
     Error *local_err = NULL;
     AioContext *aio_context;
 
+    if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
+    {
+        bdrv_invalidate_cache_all(&local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            return -EINVAL;
+        }
+    }
+
     if (!bdrv_all_can_snapshot(&bs)) {
         monitor_printf(mon, "Device '%s' is writable but does not "
                        "support snapshots.\n", bdrv_get_device_name(bs));
#13 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

In reply to Vladimir Sementsov-Ogievskiy

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command. It might make sense to provide a way to get control without resuming the VM, but I doubt that adding automatic resume to every QMP command is the right way to achieve it.

Dave, Juan, what do you think?

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 151

Quoted 13 lines
block/snapshot.c +2 -1
index bf5c2ca5e1..256d06ac9f 100644
@@ -145,7 +145,8 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
 int bdrv_can_snapshot(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
-    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
+    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs) ||
+        (bs->open_flags & BDRV_O_INACTIVE)) {
         return 0;
     }

I wasn't sure if this doesn't disable too much, but it seems it only makes 'info snapshots' turn up empty, which might not be nice, but maybe tolerable.

At least it should definitely fix the assertion.

Quoted 9 lines
migration/savevm.c +1 -0
index 5ecd264134..75e56d2d07 100644
@@ -2068,6 +2068,17 @@ int save_vmstate(Monitor *mon, const char *name)
     Error *local_err = NULL;
     AioContext *aio_context;
 
+    if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||

migration/savevm.c int save_vmstate(Monitor *mon, const char *name) · line 2081

Quoted 10 lines
migration/savevm.c +10 -0
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
+    {
+        bdrv_invalidate_cache_all(&local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            return -EINVAL;
+        }
+    }
+

This hunk can't go in before the more general question of implicitly or explicitly regaining control after a failed migration is answered.

Kevin

#14 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Quoted 6 lines · Kevin Wolf

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 5 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

    Assertion `!(bs->open_flags & 0x0800)' failed.
Quoted 7 lines
Quoted 1 line

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Yes, I mean on the source. It may not be migration for "mirgration", but for dumping state to file. In that case it seems not wrong to make a snapshot on source.

Quoted 52 lines

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command. It might make sense to provide a way to get control without resuming the VM, but I doubt that adding automatic resume to every QMP command is the right way to achieve it.

Dave, Juan, what do you think?

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 151

Quoted 13 lines
block/snapshot.c +2 -1
index bf5c2ca5e1..256d06ac9f 100644
@@ -145,7 +145,8 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
  int bdrv_can_snapshot(BlockDriverState *bs)
  {
      BlockDriver *drv = bs->drv;
-    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
+    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs) ||
+        (bs->open_flags & BDRV_O_INACTIVE)) {
          return 0;
      }

I wasn't sure if this doesn't disable too much, but it seems it only makes 'info snapshots' turn up empty, which might not be nice, but maybe tolerable.

At least it should definitely fix the assertion.

Quoted 9 lines
migration/savevm.c +1 -0
index 5ecd264134..75e56d2d07 100644
@@ -2068,6 +2068,17 @@ int save_vmstate(Monitor *mon, const char *name)
      Error *local_err = NULL;
      AioContext *aio_context;
 
+    if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||

migration/savevm.c int save_vmstate(Monitor *mon, const char *name) · line 2081

Quoted 10 lines
migration/savevm.c +10 -0
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
+    {
+        bdrv_invalidate_cache_all(&local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            return -EINVAL;
+        }
+    }
+

This hunk can't go in before the more general question of implicitly or explicitly regaining control after a failed migration is answered.

Kevin

Signature · 2 lines
Best regards,
Vladimir
#15 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Am 07.03.2017 um 10:59 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 12 lines
Quoted 11 lines · Kevin Wolf

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Quoted 6 lines
Quoted 2 lines

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Yes, I mean on the source. It may not be migration for "mirgration", but for dumping state to file. In that case it seems not wrong to make a snapshot on source.

Yes, resuming on the source is definitely valid. I'm only questioning if 'savevm' (and by extension, any other command that modifies the VM or its images) should automagically regain control of the VM, or whether that should be more explicit.

Kevin

#16 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Quoted 16 lines · * Kevin Wolf ([email protected]) wrote:

Am 07.03.2017 um 10:59 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 15 lines
Quoted 13 lines · Kevin Wolf

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Quoted 8 lines
Quoted 3 lines

Yes, I mean on the source. It may not be migration for "mirgration", but for dumping state to file. In that case it seems not wrong to make a snapshot on source.

Yes, resuming on the source is definitely valid. I'm only questioning if 'savevm' (and by extension, any other command that modifies the VM or its images) should automagically regain control of the VM, or whether that should be more explicit.

How many things other than 'cont' and 'savevm' are there?

We should definitely fix it so a savevm there doesn't assert, but I'm also unsure if it's worth a separate explicit command.

Dave

Quoted 1 line

Kevin

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#17 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

In reply to Kevin Wolf
Quoted 15 lines · * Kevin Wolf ([email protected]) wrote:

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Quoted 8 lines

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command. It might make sense to provide a way to get control without resuming the VM, but I doubt that adding automatic resume to every QMP command is the right way to achieve it.

Dave, Juan, what do you think?

I'd only ever really thought of 'cont' or retrying the migration. However, it does make sense to me that you might want to do a savevm instead; if you can't migrate then perhaps a savevm is the best you can do before your machine dies. Are there any other things that should be allowed?

We would want to be careful not to accidentally reactivate the disks on the source after what was actually a succesful migration.

As for the actual patch contents, I'd leave that to you to say if it's OK from the block side of things.

Dave

Quoted 18 lines

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 151

Quoted 13 lines
block/snapshot.c +2 -1
index bf5c2ca5e1..256d06ac9f 100644
@@ -145,7 +145,8 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
 int bdrv_can_snapshot(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
-    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
+    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs) ||
+        (bs->open_flags & BDRV_O_INACTIVE)) {
         return 0;
     }

I wasn't sure if this doesn't disable too much, but it seems it only makes 'info snapshots' turn up empty, which might not be nice, but maybe tolerable.

Quoted 1 line

At least it should definitely fix the assertion.

Did Denis have some concerns about this chunk?

Quoted 23 lines
Quoted 9 lines
migration/savevm.c +1 -0
index 5ecd264134..75e56d2d07 100644
@@ -2068,6 +2068,17 @@ int save_vmstate(Monitor *mon, const char *name)
     Error *local_err = NULL;
     AioContext *aio_context;
 
+    if (runstate_check(RUN_STATE_FINISH_MIGRATE) ||

migration/savevm.c int save_vmstate(Monitor *mon, const char *name) · line 2081

Quoted 10 lines
migration/savevm.c +10 -0
+        runstate_check(RUN_STATE_POSTMIGRATE) ||
+        runstate_check(RUN_STATE_PRELAUNCH))
+    {
+        bdrv_invalidate_cache_all(&local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            return -EINVAL;
+        }
+    }
+

This hunk can't go in before the more general question of implicitly or explicitly regaining control after a failed migration is answered.

Quoted 1 line

Kevin

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#18 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Am 28.03.2017 um 12:55 hat Dr. David Alan Gilbert geschrieben:

Quoted 23 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 22 lines

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command. It might make sense to provide a way to get control without resuming the VM, but I doubt that adding automatic resume to every QMP command is the right way to achieve it.

Quoted 7 lines
Quoted 1 line

Dave, Juan, what do you think?

I'd only ever really thought of 'cont' or retrying the migration. However, it does make sense to me that you might want to do a savevm instead; if you can't migrate then perhaps a savevm is the best you can do before your machine dies. Are there any other things that should be allowed?

I think we need to ask the other way round: Any reason _not_ to allow certain operations that you can normally perform on a stopped VM?

Quoted 2 lines

We would want to be careful not to accidentally reactivate the disks on the source after what was actually a succesful migration.

Yes, that's exactly my concern, even with savevm. That's why I suggested we could have a 'cont'-like thing that just gets back control of the images and moves into the normal paused state, but doesn't immediately resume the actual VM.

Kevin

#19 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Quoted 35 lines · * Kevin Wolf ([email protected]) wrote:

Am 28.03.2017 um 12:55 hat Dr. David Alan Gilbert geschrieben:

Quoted 23 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 22 lines

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command. It might make sense to provide a way to get control without resuming the VM, but I doubt that adding automatic resume to every QMP command is the right way to achieve it.

Quoted 7 lines
Quoted 1 line

Dave, Juan, what do you think?

I'd only ever really thought of 'cont' or retrying the migration. However, it does make sense to me that you might want to do a savevm instead; if you can't migrate then perhaps a savevm is the best you can do before your machine dies. Are there any other things that should be allowed?

I think we need to ask the other way round: Any reason _not_ to allow certain operations that you can normally perform on a stopped VM?

Quoted 7 lines
Quoted 2 lines

We would want to be careful not to accidentally reactivate the disks on the source after what was actually a succesful migration.

Yes, that's exactly my concern, even with savevm. That's why I suggested we could have a 'cont'-like thing that just gets back control of the images and moves into the normal paused state, but doesn't immediately resume the actual VM.

OK, lets say we had that block-reactivate (for want of a better name), how would we stop everything asserting if the user tried to do it before they'd run block-reactivate?

Dave

Quoted 1 line

Kevin

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#20 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Am 28.03.2017 um 13:13 hat Dr. David Alan Gilbert geschrieben:

Quoted 39 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 38 lines

Am 28.03.2017 um 12:55 hat Dr. David Alan Gilbert geschrieben:

Quoted 23 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 22 lines

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command. It might make sense to provide a way to get control without resuming the VM, but I doubt that adding automatic resume to every QMP command is the right way to achieve it.

Quoted 7 lines
Quoted 1 line

Dave, Juan, what do you think?

I'd only ever really thought of 'cont' or retrying the migration. However, it does make sense to me that you might want to do a savevm instead; if you can't migrate then perhaps a savevm is the best you can do before your machine dies. Are there any other things that should be allowed?

I think we need to ask the other way round: Any reason _not_ to allow certain operations that you can normally perform on a stopped VM?

Quoted 2 lines

We would want to be careful not to accidentally reactivate the disks on the source after what was actually a succesful migration.

Quoted 8 lines
Quoted 4 lines

Yes, that's exactly my concern, even with savevm. That's why I suggested we could have a 'cont'-like thing that just gets back control of the images and moves into the normal paused state, but doesn't immediately resume the actual VM.

OK, lets say we had that block-reactivate (for want of a better name), how would we stop everything asserting if the user tried to do it before they'd run block-reactivate?

We would have to add checks to the monitor commands that assume that the image is activated and error out if it isn't.

Maybe just adding the check to blk_is_available() would be enough, but we'd have to check carefully whether it covers all cases and causes no false positives.

By the way, I wouldn't call this 'block-reactivate' because I don't think this should be a block-specific command. It's a VM lifecycle command that switches from a postmigrate state (that assumes we have no control over the VM's resources any more) to a paused state (where we do have this control). Maybe something like 'migration-abort'.

Kevin

#21 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Quoted 50 lines · Kevin Wolf

Am 28.03.2017 um 13:13 hat Dr. David Alan Gilbert geschrieben:

Quoted 35 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 34 lines

Am 28.03.2017 um 12:55 hat Dr. David Alan Gilbert geschrieben:

Quoted 22 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 21 lines

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

    Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command. It might make sense to provide a way to get control without resuming the VM, but I doubt that adding automatic resume to every QMP command is the right way to achieve it.

Quoted 6 lines
Quoted 1 line

Dave, Juan, what do you think?

I'd only ever really thought of 'cont' or retrying the migration. However, it does make sense to me that you might want to do a savevm instead; if you can't migrate then perhaps a savevm is the best you can do before your machine dies. Are there any other things that should be allowed?

I think we need to ask the other way round: Any reason _not_ to allow certain operations that you can normally perform on a stopped VM?

Quoted 2 lines

We would want to be careful not to accidentally reactivate the disks on the source after what was actually a succesful migration.

Quoted 7 lines
Quoted 4 lines

Yes, that's exactly my concern, even with savevm. That's why I suggested we could have a 'cont'-like thing that just gets back control of the images and moves into the normal paused state, but doesn't immediately resume the actual VM.

OK, lets say we had that block-reactivate (for want of a better name), how would we stop everything asserting if the user tried to do it before they'd run block-reactivate?

We would have to add checks to the monitor commands that assume that the image is activated and error out if it isn't.

Maybe just adding the check to blk_is_available() would be enough, but we'd have to check carefully whether it covers all cases and causes no false positives.

Quoted 5 lines

By the way, I wouldn't call this 'block-reactivate' because I don't think this should be a block-specific command. It's a VM lifecycle command that switches from a postmigrate state (that assumes we have no control over the VM's resources any more) to a paused state (where we do have this control). Maybe something like 'migration-abort'.

'abort' is not very good too I think. migration is completed, nothing to abort.. (may be successful migration to file for suspend, some kind of vm cloning, etc)

Quoted 2 lines

Kevin

Signature · 2 lines
Best regards,
Vladimir
#22 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Quoted 64 lines · Vladimir Sementsov-Ogievskiy
Quoted 57 lines · Kevin Wolf

Am 28.03.2017 um 13:13 hat Dr. David Alan Gilbert geschrieben:

Quoted 41 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 40 lines

Am 28.03.2017 um 12:55 hat Dr. David Alan Gilbert geschrieben:

Quoted 28 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 27 lines

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 7 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

    Assertion `!(bs->open_flags & 0x0800)' failed.

Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for
me.

Hm... Or do you mean on the source? In that case, I think the operation
must fail, but of course more gracefully than now.

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command.
It might make sense to provide a way to get control without resuming the
VM, but I doubt that adding automatic resume to every QMP command is the
right way to achieve it.

Quoted 6 lines
Quoted 1 line

Dave, Juan, what do you think?

I'd only ever really thought of 'cont' or retrying the migration. However, it does make sense to me that you might want to do a savevm instead; if you can't migrate then perhaps a savevm is the best you can do before your machine dies. Are there any other things that should be allowed?

I think we need to ask the other way round: Any reason _not_ to allow certain operations that you can normally perform on a stopped VM?

Quoted 2 lines

We would want to be careful not to accidentally reactivate the disks on the source after what was actually a succesful migration.

Quoted 8 lines
Quoted 5 lines

Yes, that's exactly my concern, even with savevm. That's why I suggested
we could have a 'cont'-like thing that just gets back control of the images and moves into the normal paused state, but doesn't immediately resume the actual VM.

OK, lets say we had that block-reactivate (for want of a better name), how would we stop everything asserting if the user tried to do it before they'd run block-reactivate?

We would have to add checks to the monitor commands that assume that the image is activated and error out if it isn't.

Maybe just adding the check to blk_is_available() would be enough, but we'd have to check carefully whether it covers all cases and causes no false positives.

Quoted 5 lines

By the way, I wouldn't call this 'block-reactivate' because I don't think this should be a block-specific command. It's a VM lifecycle command that switches from a postmigrate state (that assumes we have no control over the VM's resources any more) to a paused state (where we do have this control). Maybe something like 'migration-abort'.

Quoted 3 lines

'abort' is not very good too I think. migration is completed, nothing to abort.. (may be successful migration to file for suspend, some kind of vm cloning, etc)

There is already migrate_cancel. Does it make sense to make it reactivate fds if migration is completed?

Paolo

#23 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Quoted 67 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 64 lines · Vladimir Sementsov-Ogievskiy
Quoted 62 lines · Kevin Wolf

Am 28.03.2017 um 13:13 hat Dr. David Alan Gilbert geschrieben:

Quoted 41 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 40 lines

Am 28.03.2017 um 12:55 hat Dr. David Alan Gilbert geschrieben:

Quoted 28 lines · * Kevin Wolf ([email protected]) wrote:
Quoted 27 lines

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 7 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

    Assertion `!(bs->open_flags & 0x0800)' failed.

Signed-off-by: Vladimir Sementsov-Ogievskiy
<[email protected]>

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for
me.

Hm... Or do you mean on the source? In that case, I think the operation
must fail, but of course more gracefully than now.

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command.
It might make sense to provide a way to get control without resuming the
VM, but I doubt that adding automatic resume to every QMP command is the
right way to achieve it.

Quoted 6 lines
Quoted 1 line

Dave, Juan, what do you think?

I'd only ever really thought of 'cont' or retrying the migration. However, it does make sense to me that you might want to do a savevm instead; if you can't migrate then perhaps a savevm is the best you can do before your machine dies. Are there any other things that should be allowed?

I think we need to ask the other way round: Any reason _not_ to allow certain operations that you can normally perform on a stopped VM?

Quoted 2 lines

We would want to be careful not to accidentally reactivate the disks on the source after what was actually a succesful migration.

Quoted 8 lines
Quoted 5 lines

Yes, that's exactly my concern, even with savevm. That's why I suggested
we could have a 'cont'-like thing that just gets back control of the images and moves into the normal paused state, but doesn't immediately resume the actual VM.

OK, lets say we had that block-reactivate (for want of a better name), how would we stop everything asserting if the user tried to do it before they'd run block-reactivate?

We would have to add checks to the monitor commands that assume that the image is activated and error out if it isn't.

Maybe just adding the check to blk_is_available() would be enough, but we'd have to check carefully whether it covers all cases and causes no false positives.

By the way, I wouldn't call this 'block-reactivate' because I don't think this should be a block-specific command. It's a VM lifecycle command that switches from a postmigrate state (that assumes we have no control over the VM's resources any more) to a paused state (where we do have this control). Maybe something like 'migration-abort'.

Quoted 6 lines
Quoted 3 lines

'abort' is not very good too I think. migration is completed, nothing to abort.. (may be successful migration to file for suspend, some kind of vm cloning, etc)

There is already migrate_cancel. Does it make sense to make it reactivate fds if migration is completed?

It's potentially racy to do that.
Imagine if your migration is almost finished and you issue a migrate_cancel, what happens?
Maybe it cancelled it.
Maybe it just completed in time - and you really better not be accessing the disks on the source unless you're sure the destination isn't running.

Dave

Quoted 1 line

Paolo

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#24 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

Quoted 11 lines · Dr. David Alan Gilbert
Quoted 5 lines
Quoted 3 lines

'abort' is not very good too I think. migration is completed, nothing to abort.. (may be successful migration to file for suspend, some kind of vm cloning, etc)

There is already migrate_cancel. Does it make sense to make it reactivate fds if migration is completed?

It's potentially racy to do that.
Imagine if your migration is almost finished and you issue a migrate_cancel, what happens?
Maybe it cancelled it.
Maybe it just completed in time - and you really better not be accessing the disks on the source unless you're sure the destination isn't running.

If you want to avoid races, you probably have to use -S anyway on the destination and do more handshaking between source and destination. But yes, just to be safe it'd be better to add a new flag (-f for HMP, 'cancel-completed-migration':true for QMP or something like that).

Paolo

#25 lore raw

[Qemu-devel] ping Re: [PATCH 3/4] savevm: fix savevm after migration

Quoted 13 lines · Paolo Bonzini
Quoted 11 lines · Dr. David Alan Gilbert
Quoted 5 lines
Quoted 3 lines

'abort' is not very good too I think. migration is completed, nothing to abort.. (may be successful migration to file for suspend, some kind of vm cloning, etc)

There is already migrate_cancel. Does it make sense to make it reactivate fds if migration is completed?

It's potentially racy to do that.
Imagine if your migration is almost finished and you issue a migrate_cancel, what happens?
Maybe it cancelled it.
Maybe it just completed in time - and you really better not be accessing the disks on the source unless you're sure the destination isn't running.

Quoted 6 lines

If you want to avoid races, you probably have to use -S anyway on the destination and do more handshaking between source and destination. But yes, just to be safe it'd be better to add a new flag (-f for HMP, 'cancel-completed-migration':true for QMP or something like that).

Paolo

Looks like we haven't final conclusion on this (solve problems, noted in patches 2 and 3). Are someone working on this now?

Signature · 2 lines
Best regards,
Vladimir
#26 lore raw

Re: [Qemu-devel] [PATCH 3/4] savevm: fix savevm after migration

In reply to Dr. David Alan Gilbert
Quoted 53 lines · Dr. David Alan Gilbert
Quoted 14 lines · * Kevin Wolf ([email protected]) wrote:

Am 25.02.2017 um 20:31 hat Vladimir Sementsov-Ogievskiy geschrieben:

Quoted 6 lines

After migration all drives are inactive and savevm will fail with

qemu-kvm: block/io.c:1406: bdrv_co_do_pwritev:

   Assertion `!(bs->open_flags & 0x0800)' failed.

What's the exact state you're in? I tried to reproduce this, but just doing a live migration and then savevm on the destination works fine for me.

Hm... Or do you mean on the source? In that case, I think the operation must fail, but of course more gracefully than now.

Quoted 8 lines

Actually, the question that you're asking implicitly here is how the source qemu process should be "reactivated" after a failed migration. Currently, as far as I know, this is only with issuing a "cont" command. It might make sense to provide a way to get control without resuming the VM, but I doubt that adding automatic resume to every QMP command is the right way to achieve it.

Dave, Juan, what do you think?

I'd only ever really thought of 'cont' or retrying the migration. However, it does make sense to me that you might want to do a savevm instead; if you can't migrate then perhaps a savevm is the best you can do before your machine dies. Are there any other things that should be allowed?

We would want to be careful not to accidentally reactivate the disks on the source after what was actually a succesful migration.

As for the actual patch contents, I'd leave that to you to say if it's OK from the block side of things.

Dave

Quoted 17 lines

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 151

Quoted 13 lines
block/snapshot.c +2 -1
index bf5c2ca5e1..256d06ac9f 100644
@@ -145,7 +145,8 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
 int bdrv_can_snapshot(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
-    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs)) {
+    if (!drv || !bdrv_is_inserted(bs) || bdrv_is_read_only(bs) ||
+        (bs->open_flags & BDRV_O_INACTIVE)) {
         return 0;
     }

I wasn't sure if this doesn't disable too much, but it seems it only makes 'info snapshots' turn up empty, which might not be nice, but maybe tolerable.

Quoted 2 lines
Quoted 1 line

At least it should definitely fix the assertion.

Did Denis have some concerns about this chunk?

Yep. I really think that this check is unnecessary and wrong. Actually all disks are in the INACTIVE state and we will face the problem later on the actual write. This exact operation is sane.

Den

#27 lore raw

PATCH 4/4 migration: fix use-after-free of to_dst_file

In reply to Vladimir Sementsov-Ogievskiy

hmp_savevm calls qemu_savevm_state(f), which sets to_dst_file=f in global migration state. Then hmp_savevm closes f (g_free called).

Next access to to_dst_file in migration state (for example, qmp_migrate_set_speed) will use it after it was freed.

---
 migration/savevm.c | 5 +++++
 1 file changed, 5 insertions(+)
migration/savevm.c +5 -0
index 75e56d2d07..fcb8fd8acd 100644
@@ -1276,6 +1276,11 @@ done:
         status = MIGRATION_STATUS_COMPLETED;
     }
     migrate_set_state(&ms->state, MIGRATION_STATUS_SETUP, status);
+
+    /* f is outer parameter, it should not stay in global migration state after
+     * this function finished */
+    ms->to_dst_file = NULL;
+
     return ret;
 }
 
Signature · 1 line
2.11.1
Reviewed-by on v1 4/4
#28 lore raw

Re: [Qemu-devel] [PATCH 4/4] migration: fix use-after-free of to_dst_file

Quoted 7 lines · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:

hmp_savevm calls qemu_savevm_state(f), which sets to_dst_file=f in global migration state. Then hmp_savevm closes f (g_free called).

Next access to to_dst_file in migration state (for example, qmp_migrate_set_speed) will use it after it was freed.

Quoted 23 lines
---
 migration/savevm.c | 5 +++++
 1 file changed, 5 insertions(+)
migration/savevm.c +5 -0
index 75e56d2d07..fcb8fd8acd 100644
@@ -1276,6 +1276,11 @@ done:
         status = MIGRATION_STATUS_COMPLETED;
     }
     migrate_set_state(&ms->state, MIGRATION_STATUS_SETUP, status);
+
+    /* f is outer parameter, it should not stay in global migration state after
+     * this function finished */
+    ms->to_dst_file = NULL;
+
     return ret;
 }
 
Signature · 1 line
2.11.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#29 lore raw

Re: [Qemu-devel] [PATCH 4/4] migration: fix use-after-free of to_dst_file

In reply to Vladimir Sementsov-Ogievskiy
Quoted 7 lines · * Vladimir Sementsov-Ogievskiy ([email protected]) wrote:

hmp_savevm calls qemu_savevm_state(f), which sets to_dst_file=f in global migration state. Then hmp_savevm closes f (g_free called).

Next access to to_dst_file in migration state (for example, qmp_migrate_set_speed) will use it after it was freed.

Queued just this one.

Quoted 24 lines
---
 migration/savevm.c | 5 +++++
 1 file changed, 5 insertions(+)
migration/savevm.c +5 -0
index 75e56d2d07..fcb8fd8acd 100644
@@ -1276,6 +1276,11 @@ done:
         status = MIGRATION_STATUS_COMPLETED;
     }
     migrate_set_state(&ms->state, MIGRATION_STATUS_SETUP, status);
+
+    /* f is outer parameter, it should not stay in global migration state after
+     * this function finished */
+    ms->to_dst_file = NULL;
+
     return ret;
 }
 
Signature · 1 line
2.11.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#30 lore raw
In reply to Vladimir Sementsov-Ogievskiy
Quoted 21 lines · Vladimir Sementsov-Ogievskiy

Hi all!

Here are some migration related bugs, two about INACTIVE bdses and one use-after-free.

I'm absolutely not sure, that these bugs should be fixed like I'm fixing, but problem definitely exists.

Reset in stopped state is strange case, may be such usage should be restricted.
About INACTIVE - looks like it should be a separate run-state, not only bdrv-flag.
Situation with migration state, which is global, but is set/reset/changed in not controlled manner is not very good too..

Vladimir Sementsov-Ogievskiy (4):
iotests: add migration corner cases test
qmp-cont: invalidate on RUN_STATE_PRELAUNCH
savevm: fix savevm after migration
migration: fix use-after-free of to_dst_file

Quoted 10 lines
 block/snapshot.c           |  3 +-
 migration/savevm.c         | 16 +++++++++++
 qmp.c                      |  3 +-
 tests/qemu-iotests/175     | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/175.out |  5 ++++
 tests/qemu-iotests/group   |  1 +
 6 files changed, 97 insertions(+), 2 deletions(-)
 create mode 100644 tests/qemu-iotests/175
 create mode 100644 tests/qemu-iotests/175.out

guys, what about patches 1-3?

Den

#31 lore raw

[Qemu-devel] ping Re: [PATCH 0/4] some migration bugs

Quoted 32 lines · Denis V. Lunev
Quoted 28 lines · Vladimir Sementsov-Ogievskiy

Hi all!

Here are some migration related bugs, two about INACTIVE bdses and one use-after-free.

I'm absolutely not sure, that these bugs should be fixed like I'm fixing, but problem definitely exists.

Reset in stopped state is strange case, may be such usage should be restricted.
About INACTIVE - looks like it should be a separate run-state, not only bdrv-flag.
Situation with migration state, which is global, but is set/reset/changed in not controlled manner is not very good too..

Vladimir Sementsov-Ogievskiy (4):
iotests: add migration corner cases test
qmp-cont: invalidate on RUN_STATE_PRELAUNCH
savevm: fix savevm after migration
migration: fix use-after-free of to_dst_file

  block/snapshot.c           |  3 +-
  migration/savevm.c         | 16 +++++++++++
  qmp.c                      |  3 +-
  tests/qemu-iotests/175     | 71 ++++++++++++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/175.out |  5 ++++
  tests/qemu-iotests/group   |  1 +
  6 files changed, 97 insertions(+), 2 deletions(-)
Quoted 3 lines

create mode 100644 tests/qemu-iotests/175
create mode 100644 tests/qemu-iotests/175.out

Quoted 3 lines

guys, what about patches 1-3?

Den

At least, with 2 and 3, fixing an assert? Do someone going to work with adding a runstate?

Signature · 2 lines
Best regards,
Vladimir