migration/mapped-ram: Add direct-io support
v3 cover migration/mapped-ram: Add direct-io support
Hi,
Not many changes since v2:
- new patch 1: drop IOC reference on offset check failure
- dropped 2 patches adding direct-io to the single threaded migration. We have agreed to use multifd with 1 channel for that use-case.
CI run: https://gitlab.com/farosas/qemu/-/pipelines/1335719164
Thanks
v2:
https://lore.kernel.org/r/[email protected]
v1:
https://lore.kernel.org/r/[email protected]
Hi everyone, here's the rest of the migration "mapped-ram" feature that didn't get merged for 9.0. This series adds support for direct I/O, the missing piece to get the desired performance improvements.
There's 3 parts to this:
1- The plumbing for the new "direct-io" migration parameter. With this
we can already use direct-io with the file transport + multifd + mapped-ram. Patches 1-3.
Due to the alignment requirements of O_DIRECT and the fact that multifd runs the channels in parallel with the migration thread, we must open the migration file two times, one with O_DIRECT set and another with it clear.
If the user is not passing in a file name which QEMU can open at will, we must then require that the user pass the two file descriptors with the flags already properly set. We'll use the already existing fdset + QMP add-fd infrastructure for this.
2- Changes to the fdset infrastructure to support O_DIRECT. We need
those to be able to select from the user-provided fdset the file descriptor that contains the O_DIRECT flag. Patches 4-5.
3- Some fdset validation to make sure the two-fds requirement is being
met. Patches 6-7.
CI run: https://gitlab.com/farosas/qemu/-/pipelines/1269352083
Fabiano Rosas (15):
migration: Drop reference to QIOChannel if file seeking fails
migration: Fix file migration with fdset tests/qtest/migration: Fix file migration offset check tests/qtest/migration: Add a precopy file test with fdset
monitor: Introduce monitor_fdset_*free
monitor: Stop removing non-duplicated fds
monitor: Simplify fdset and fd removal
monitor: Report errors from monitor_fdset_dup_fd_add
io: Stop using qemu_open_old in channel-file
migration: Add direct-io parameter
migration/multifd: Add direct-io support tests/qtest/migration: Add tests for file migration with direct-io
monitor: fdset: Match against O_DIRECT
migration: Add documentation for fdset with multifd + file tests/qtest/migration: Add a test for mapped-ram with passing of fds
Peter Xu (1):
monitor: Drop monitor_fdset_dup_fd_find/_remove()
docs/devel/migration/main.rst | 24 ++- docs/devel/migration/mapped-ram.rst | 6 +- include/monitor/monitor.h | 3 +- include/qemu/osdep.h | 2 + io/channel-file.c | 8 +- migration/file.c | 45 ++++- migration/file.h | 1 - migration/migration-hmp-cmds.c | 11 ++ migration/migration.c | 23 +++ migration/options.c | 35 ++++ migration/options.h | 1 + monitor/fds.c | 96 +++++----- monitor/hmp.c | 2 - monitor/monitor-internal.h | 1 - monitor/monitor.c | 1 - monitor/qmp.c | 2 - qapi/migration.json | 21 ++- stubs/fdset.c | 7 +- tests/qtest/migration-helpers.c | 44 +++++ tests/qtest/migration-helpers.h | 8 + tests/qtest/migration-test.c | 263 +++++++++++++++++++++++++--- util/osdep.c | 34 ++-- 22 files changed, 508 insertions(+), 130 deletions(-)
base-commit: 05ad1440b8428b0ade9b8e5c01469adb8fbf83e3
Signature · 1 line
2.35.3
PATCH v3 1/16 migration: Drop reference to QIOChannel if file seeking fails
We forgot to drop the reference to the QIOChannel in the error path of the offset adjustment. Do it now.
--- migration/file.c | 1 + 1 file changed, 1 insertion(+)
migration/file.c +1 -0
@@ -94,6 +94,7 @@ void file_start_outgoing_migration(MigrationState *s, ioc =QIO_CHANNEL (fioc);if (offset &&qio_channel_io_seek (ioc, offset,SEEK_SET , errp) <0 ) { +object_unref (OBJECT (fioc));return ; }qio_channel_set_name (ioc,"migration-file-outgoing" );
Signature · 1 line
2.35.3
Re: [PATCH v3 01/16] migration: Drop reference to QIOChannel if file seeking fails
Quoted 4 lines · Fabiano Rosas
We forgot to drop the reference to the QIOChannel in the error path of the offset adjustment. Do it now.
Signed-off-by Fabiano Rosas <[email protected]>
Signature · 1 line
Peter Xu
PATCH v3 2/16 migration: Fix file migration with fdset
In reply to Fabiano RosasWhen the "file:" migration support was added we missed the special case in the qemu_open_old implementation that allows for a particular file name format to be used to refer to a set of file descriptors that have been previously provided to QEMU via the add-fd QMP command.
When using this fdset feature, we should not truncate the migration file because being given an fd means that the management layer is in control of the file and will likely already have some data written to it. This is further indicated by the presence of the 'offset' argument, which indicates the start of the region where QEMU is allowed to write.
Fix the issue by replacing the O_TRUNC flag on open by an ftruncate call, which will take the offset into consideration.
--- migration/file.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
migration/file.c +9 -2
@@ -84,12 +84,19 @@ void file_start_outgoing_migration(MigrationState *s,trace_migration_file_outgoing (filename); - fioc =qio_channel_file_new_path (filename,O_CREAT |O_WRONLY |O_TRUNC , -0600 , errp); + fioc =qio_channel_file_new_path (filename,O_CREAT |O_WRONLY ,0600 , errp);if (!fioc) {return ; } +if (ftruncate (fioc->fd, offset)) { +error_setg_errno (errp, errno, +"failed to truncate migration file to offset %" PRIx64, + offset); +object_unref (OBJECT (fioc)); +return ; + } + outgoing_args.fname =g_strdup (filename); ioc =QIO_CHANNEL (fioc);
Signature · 1 line
2.35.3
Re: [PATCH v3 02/16] migration: Fix file migration with fdset
Quoted 21 lines · Fabiano Rosas
When the "file:" migration support was added we missed the special case in the
qemu_open_oldimplementation that allows for a particular file name format to be used to refer to a set of file descriptors that have been previously provided to QEMU via the add-fd QMP command.When using this fdset feature, we should not truncate the migration file because being given an fd means that the management layer is in control of the file and will likely already have some data written to it. This is further indicated by the presence of the 'offset' argument, which indicates the start of the region where QEMU is allowed to write.
Fix the issue by replacing the
O_TRUNCflag on open by an ftruncate call, which will take the offset into consideration.Fixes 385f510df5 ("migration: file URI offset")Suggested-by Daniel P. Berrangé <[email protected]>Reviewed-by Prasad Pandit <[email protected]>Reviewed-by Peter Xu <[email protected]>Reviewed-by Daniel P. Berrangé <[email protected]>Signed-off-by Fabiano Rosas <[email protected]>
Quoted 3 lines
--- migration/file.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
Is it a stable material?
Thanks,
/mjt
Quoted 26 lines
migration/file.c +9
-2@@ -84,12 +84,19 @@ void file_start_outgoing_migration(MigrationState *s,trace_migration_file_outgoing (filename); - fioc =qio_channel_file_new_path (filename,O_CREAT |O_WRONLY |O_TRUNC , -0600 , errp); + fioc =qio_channel_file_new_path (filename,O_CREAT |O_WRONLY ,0600 , errp);if (!fioc) {return ; } +if (ftruncate (fioc->fd, offset)) { +error_setg_errno (errp, errno, +"failed to truncate migration file to offset %" PRIx64, + offset); +object_unref (OBJECT (fioc)); +return ; + } + outgoing_args.fname =g_strdup (filename); ioc =QIO_CHANNEL (fioc);
Signature · 4 lines
GPG Key transition (from rsa2048 to rsa4096) since 2024-04-24. New key: rsa4096/61AD3D98ECDF2C8E 9D8B E14E 3F2A 9DD7 9199 28F1 61AD 3D98 ECDF 2C8E Old key: rsa2048/457CE0A0804465C5 6EE1 95D1 886E 8FFB 810D 4324 457C E0A0 8044 65C5 Transition statement: http://www.corpit.ru/mjt/gpg-transition-2024.txt
Re: [PATCH v3 02/16] migration: Fix file migration with fdset
Quoted 26 lines · Michael Tokarev
Quoted 21 lines · Fabiano Rosas
When the "file:" migration support was added we missed the special case in the
qemu_open_oldimplementation that allows for a particular file name format to be used to refer to a set of file descriptors that have been previously provided to QEMU via the add-fd QMP command.When using this fdset feature, we should not truncate the migration file because being given an fd means that the management layer is in control of the file and will likely already have some data written to it. This is further indicated by the presence of the 'offset' argument, which indicates the start of the region where QEMU is allowed to write.
Fix the issue by replacing the
O_TRUNCflag on open by an ftruncate call, which will take the offset into consideration.Fixes 385f510df5 ("migration: file URI offset")Suggested-by Daniel P. Berrangé <[email protected]>Reviewed-by Prasad Pandit <[email protected]>Reviewed-by Peter Xu <[email protected]>Reviewed-by Daniel P. Berrangé <[email protected]>Signed-off-by Fabiano Rosas <[email protected]>Quoted 3 lines
--- migration/file.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
Quoted 1 line
Is it a stable material?
I suppose yes. Thanks.
Signature · 1 line
Peter Xu
PATCH v3 3/16 tests/qtest/migration: Fix file migration offset check
In reply to Fabiano RosasWhen doing file migration, QEMU accepts an offset that should be skipped when writing the migration stream to the file. The purpose of the offset is to allow the management layer to put its own metadata at the start of the file.
We have tests for this in migration-test, but only testing that the migration stream starts at the correct offset and not that it actually leaves the data intact. Unsurprisingly, there's been a bug in that area that the tests didn't catch.
Fix the tests to write some data to the offset region and check that it's actually there after the migration.
While here, switch to using g_get_file_contents() which is more portable than mmap().
--- tests/qtest/migration-test.c | 79 ++++++++++++++++++++++-------------- 1 file changed, 48 insertions(+), 31 deletions(-)
tests/qtest/migration-test.c +48 -31
@@ -68,6 +68,7 @@ static QTestMigrationState dst_state;#define QEMU_VM_FILE_MAGIC 0x5145564d#define FILE_TEST_FILENAME "migfile"#define FILE_TEST_OFFSET 0x1000 +#define FILE_TEST_MARKER 'X'#define QEMU_ENV_SRC "QTEST_QEMU_BINARY_SRC"#define QEMU_ENV_DST "QTEST_QEMU_BINARY_DST"
@@ -1693,10 +1694,43 @@ finish:test_migrate_end (from, to, args->result ==MIG_TEST_SUCCEED ); } +static void file_dirty_offset_region (void ) +{ + g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +size_t size =FILE_TEST_OFFSET ; + g_autofree char *data = g_new0(char, size); + +memset (data,FILE_TEST_MARKER , size); +g_assert (g_file_set_contents (path, data, size,NULL )); +} + +static void file_check_offset_region (void ) +{ + g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +size_t size =FILE_TEST_OFFSET ; + g_autofree char *expected = g_new0(char, size); + g_autofree char *actual = NULL; +uint64_t *stream_start; + +/* +* Ensure the skipped offset region's data has not been touched +* and the migration stream starts at the right place. +*/ + +memset (expected,FILE_TEST_MARKER , size); + +g_assert (g_file_get_contents (path, &actual,NULL ,NULL )); +g_assert (!memcmp (actual, expected, size)); + + stream_start = (uint64_t *)(actual + size); + g_assert_cmpint(cpu_to_be64(*stream_start) >> 32, ==, QEMU_VM_FILE_MAGIC); +} +static void test_file_common (MigrateCommon *args,bool stop_src) {QTestState *from, *to;void *data_hook =NULL ; +bool check_offset = false;if (test_migrate_start (&from, &to, args->listen_uri, &args->start)) {return ;
@@ -1709,6 +1743,16 @@ static void test_file_common(MigrateCommon *args, bool stop_src)*/ g_assert_false (args->live); +if (g_strrstr (args->connect_uri,"offset=" )) { + check_offset = true; +/* +* This comes before the start_hook because it's equivalent to +* a management application creating the file and writing to +* it so hooks should expect the file to be already present. +*/ +file_dirty_offset_region (); + } +if (args->start_hook) { data_hook = args->start_hook (from, to); }
@@ -1743,6 +1787,10 @@ static void test_file_common(MigrateCommon *args, bool stop_src)wait_for_serial ("dest_serial" ); +if (check_offset) { +file_check_offset_region (); + } + finish:if (args->finish_hook) { args->finish_hook (from, to, data_hook);
@@ -1942,36 +1990,6 @@ static void test_precopy_file(void)test_file_common (&args, true); } -static void file_offset_finish_hook (QTestState *from,QTestState *to, -void *opaque) -{ -#if defined(__linux__) - g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); -size_t size =FILE_TEST_OFFSET +sizeof (QEMU_VM_FILE_MAGIC ); -uintptr_t *addr, *p; -int fd; - - fd =open (path,O_RDONLY ); -g_assert (fd !=-1 ); - addr =mmap (NULL , size,PROT_READ ,MAP_SHARED , fd,0 ); -g_assert (addr !=MAP_FAILED ); - -/* -* Ensure the skipped offset contains zeros and the migration -* stream starts at the right place. -*/ - p = addr; -while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { -g_assert (*p ==0 ); - p++; - } - g_assert_cmpint(cpu_to_be64(*p) >> 32, ==, QEMU_VM_FILE_MAGIC); - -munmap (addr, size); -close (fd); -#endif -} -static void test_precopy_file_offset (void ) { g_autofree char *uri = g_strdup_printf("file:%s/%s,offset=%d", tmpfs,
@@ -1980,7 +1998,6 @@ static void test_precopy_file_offset(void)MigrateCommon args = { .connect_uri = uri, .listen_uri ="defer" , - .finish_hook = file_offset_finish_hook, };test_file_common (&args, false);
Signature · 1 line
2.35.3
PATCH v3 4/16 tests/qtest/migration: Add a precopy file test with fdset
In reply to Fabiano RosasAdd a test for file migration using fdset. The passing of fds is more complex than using a file path. This is also the scenario where it's most important we ensure that the initial migration stream offset is respected because the fdset interface is the one used by the management layer when providing a non empty migration file.
Note that fd passing is not available on Windows, so anything that uses add-fd needs to exclude that platform.
--- tests/qtest/migration-test.c | 44 ++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)
tests/qtest/migration-test.c +44 -0
@@ -1990,6 +1990,46 @@ static void test_precopy_file(void)test_file_common (&args, true); } +#ifndef _WIN32 +static void fdset_add_fds (QTestState *qts,const char *file,int flags, +int num_fds) +{ +for (int i =0 ; i < num_fds; i++) { +int fd; + + fd =open (file, flags,0660 ); +assert (fd !=-1 ); + +qtest_qmp_fds_assert_success (qts, &fd,1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); +close (fd); + } +} + +static void *file_offset_fdset_start_hook (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); + +fdset_add_fds (from, file,O_WRONLY ,1 ); +fdset_add_fds (to, file,O_RDONLY ,1 ); + +return NULL ; +} + +static void test_precopy_file_offset_fdset (void ) +{ + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d", +FILE_TEST_OFFSET ); +MigrateCommon args = { + .connect_uri = uri, + .listen_uri ="defer" , + .start_hook = file_offset_fdset_start_hook, + }; + +test_file_common (&args, false); +} +#endif +static void test_precopy_file_offset (void ) { g_autofree char *uri = g_strdup_printf("file:%s/%s,offset=%d", tmpfs,
@@ -3527,6 +3567,10 @@ int main(int argc, char **argv) test_precopy_file);migration_test_add ("/migration/precopy/file/offset" , test_precopy_file_offset); +#ifndef _WIN32 +migration_test_add ("/migration/precopy/file/offset/fdset" , + test_precopy_file_offset_fdset); +#endif migration_test_add ("/migration/precopy/file/offset/bad" , test_precopy_file_offset_bad);
Signature · 1 line
2.35.3
PATCH v3 5/16 monitor: Drop monitor_fdset_dup_fd_find/_remove()
In reply to Fabiano RosasFrom: Peter Xu <[email protected]>
Those functions are not needed, one remove function should already work. Clean it up.
Here the code doesn't really care about whether we need to keep that dupfd around if close() failed: when that happens something got very wrong, keeping the dup_fd around the fdsets may not help that situation so far.
--- include/monitor/monitor.h | 1 - monitor/fds.c | 28 ++++++---------------------- stubs/fdset.c | 5 ----- util/osdep.c | 15 +-------------- 4 files changed, 7 insertions(+), 42 deletions(-)
include/monitor/monitor.h +0 -1
@@ -53,7 +53,6 @@ AddfdInfo *monitor_fdset_add_fd(int fd, bool has_fdset_id, int64_t fdset_id, const char *opaque, Error **errp);int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags);void monitor_fdset_dup_fd_remove (int dup_fd); -int64_t monitor_fdset_dup_fd_find (int dup_fd);void monitor_register_hmp (const char *name,bool info,void (*cmd)(Monitor *mon,const QDict *qdict));
monitor/fds.c +6 -22
@@ -458,7 +458,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)#endif } -static int64_t monitor_fdset_dup_fd_find_remove (int dup_fd,bool remove) +void monitor_fdset_dup_fd_remove (int dup_fd) {MonFdset *mon_fdset;MonFdsetFd *mon_fdset_fd_dup;
@@ -467,31 +467,15 @@ static int64_t monitor_fdset_dup_fd_find_remove(int dup_fd, bool remove) QLIST_FOREACH(mon_fdset, &mon_fdsets, next) { QLIST_FOREACH(mon_fdset_fd_dup, &mon_fdset->dup_fds, next) {if (mon_fdset_fd_dup->fd == dup_fd) { -if (remove) { -QLIST_REMOVE (mon_fdset_fd_dup, next); -g_free (mon_fdset_fd_dup); -if (QLIST_EMPTY (&mon_fdset->dup_fds)) { -monitor_fdset_cleanup (mon_fdset); - } -return -1 ; - }else { -return mon_fdset->id; +QLIST_REMOVE (mon_fdset_fd_dup, next); +g_free (mon_fdset_fd_dup); +if (QLIST_EMPTY (&mon_fdset->dup_fds)) { +monitor_fdset_cleanup (mon_fdset); } +return ; } } } - -return -1 ; -} - -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return monitor_fdset_dup_fd_find_remove (dup_fd, false); -} - -void monitor_fdset_dup_fd_remove (int dup_fd) -{ -monitor_fdset_dup_fd_find_remove (dup_fd, true); }int monitor_fd_param (Monitor *mon,const char *fdname,Error **errp)
stubs/fdset.c +0 -5
@@ -9,11 +9,6 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return -1 ; -} -void monitor_fdset_dup_fd_remove (int dupfd) { }
util/osdep.c +1 -14
@@ -398,21 +398,8 @@ int qemu_open_old(const char *name, int flags, ...)int qemu_close (int fd) { -int64_t fdset_id; -/* Close fd that was dup'd from an fdset */ - fdset_id =monitor_fdset_dup_fd_find (fd); -if (fdset_id !=-1 ) { -int ret; - - ret =close (fd); -if (ret ==0 ) { -monitor_fdset_dup_fd_remove (fd); - } - -return ret; - } - +monitor_fdset_dup_fd_remove (fd);return close (fd); }
Signature · 1 line
2.35.3
PATCH v3 6/16 monitor: Introduce monitor_fdset_*free
In reply to Fabiano RosasIntroduce new functions to remove and free no longer used fds and fdsets.
We need those to decouple the remove/free routines from monitor_fdset_cleanup() which will go away in the next patches.
The new functions:
- monitor_fdset_free/_if_empty() will be used when a monitor connection closes and when an fd is removed to cleanup any fdset that is now empty.
- monitor_fdset_fd_free() will be used to remove one or more fds that have been explicitly targeted by qmp_remove_fd().
--- monitor/fds.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)
monitor/fds.c +23 -8
@@ -167,6 +167,27 @@ int monitor_get_fd(Monitor *mon, const char *fdname, Error **errp)return -1 ; } +static void monitor_fdset_free (MonFdset *mon_fdset) +{ +QLIST_REMOVE (mon_fdset, next); +g_free (mon_fdset); +} + +static void monitor_fdset_free_if_empty (MonFdset *mon_fdset) +{ +if (QLIST_EMPTY (&mon_fdset->fds) &&QLIST_EMPTY (&mon_fdset->dup_fds)) { +monitor_fdset_free (mon_fdset); + } +} + +static void monitor_fdset_fd_free (MonFdsetFd *mon_fdset_fd) +{ +close (mon_fdset_fd->fd); +g_free (mon_fdset_fd->opaque); +QLIST_REMOVE (mon_fdset_fd, next); +g_free (mon_fdset_fd); +} +static void monitor_fdset_cleanup (MonFdset *mon_fdset) {MonFdsetFd *mon_fdset_fd;
@@ -176,17 +197,11 @@ static void monitor_fdset_cleanup(MonFdset *mon_fdset)if ((mon_fdset_fd->removed || (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 )) &&runstate_is_running ()) { -close (mon_fdset_fd->fd); -g_free (mon_fdset_fd->opaque); -QLIST_REMOVE (mon_fdset_fd, next); -g_free (mon_fdset_fd); +monitor_fdset_fd_free (mon_fdset_fd); } } -if (QLIST_EMPTY (&mon_fdset->fds) &&QLIST_EMPTY (&mon_fdset->dup_fds)) { -QLIST_REMOVE (mon_fdset, next); -g_free (mon_fdset); - } +monitor_fdset_free_if_empty (mon_fdset); }void monitor_fdsets_cleanup (void )
Signature · 1 line
2.35.3
PATCH v3 7/16 monitor: Stop removing non-duplicated fds
In reply to Fabiano Rosasmonitor_fdsets_cleanup() currently has three responsibilities:
1- Remove the fds that have been marked for removal(->removed=true) by
qmp_remove_fd(). This is overly complicated, but ok.
2- Remove any file descriptors that have been passed into QEMU and
never duplicated[1,2]. A file descriptor without duplicates indicates that no part of QEMU has made use of it. This is problematic because the current implementation does it only if the guest is not running and the monitor is closed.
3- Remove/free fdsets that have become empty due to the above
removals. This is ok.
The scenario described in (2) is starting to show some cracks now that we're trying to consume fds from the migration code:
- Doing cleanup every time the last monitor connection closes works to reap unused fds, but also has the side effect of forcing the management layer to pass the file descriptors again in case of a disconnect/re-connect, if that happened to be the only monitor connection.
Another side effect is that removing an fd with qmp_remove_fd() is effectively delayed until the last monitor connection closes.
The usage of mon_refcount is also problematic because it's racy.
- Checking runstate_is_running() skips the cleanup unless the VM is running and avoids premature cleanup of the fds, but also has the side effect of blocking the legitimate removal of an fd via qmp_remove_fd() if the VM happens to be in another state.
This affects qmp_remove_fd() and qmp_query_fdsets() in particular because requesting a removal at a bad time (guest stopped) might cause an fd to never be removed, or to be removed at a much later point in time, causing the query command to continue showing the supposedly removed fd/fdset.
Note that file descriptors that *have* been duplicated are owned by the code that uses them and will be removed after qemu_close() is called. Therefore we've decided that the best course of action to avoid the undesired side-effects is to stop managing non-duplicated file descriptors.
1- efb87c1697 ("monitor: Clean up fd sets on monitor disconnect") 2- ebe52b592d ("monitor: Prevent removing fd from set during init")
--- monitor/fds.c | 15 ++++++++------- monitor/hmp.c | 2 -- monitor/monitor-internal.h | 1 - monitor/monitor.c | 1 - monitor/qmp.c | 2 -- 5 files changed, 8 insertions(+), 13 deletions(-)
monitor/fds.c +8 -7
@@ -175,6 +175,11 @@ static void monitor_fdset_free(MonFdset *mon_fdset)static void monitor_fdset_free_if_empty (MonFdset *mon_fdset) { +/* +* Only remove an empty fdset. The fds are owned by the user and +* should have been removed with qmp_remove_fd(). The dup_fds are +* owned by QEMU and should have been removed with qemu_close(). +*/ if (QLIST_EMPTY (&mon_fdset->fds) &&QLIST_EMPTY (&mon_fdset->dup_fds)) {monitor_fdset_free (mon_fdset); }
@@ -194,9 +199,7 @@ static void monitor_fdset_cleanup(MonFdset *mon_fdset)MonFdsetFd *mon_fdset_fd_next; QLIST_FOREACH_SAFE(mon_fdset_fd, &mon_fdset->fds, next, mon_fdset_fd_next) { -if ((mon_fdset_fd->removed || - (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 )) && -runstate_is_running ()) { +if (mon_fdset_fd->removed) {monitor_fdset_fd_free (mon_fdset_fd); } }
@@ -211,7 +214,7 @@ void monitor_fdsets_cleanup(void)QEMU_LOCK_GUARD (&mon_fdsets_lock); QLIST_FOREACH_SAFE(mon_fdset, &mon_fdsets, next, mon_fdset_next) { -monitor_fdset_cleanup (mon_fdset); +monitor_fdset_free_if_empty (mon_fdset); } }
@@ -484,9 +487,7 @@ void monitor_fdset_dup_fd_remove(int dup_fd)if (mon_fdset_fd_dup->fd == dup_fd) {QLIST_REMOVE (mon_fdset_fd_dup, next);g_free (mon_fdset_fd_dup); -if (QLIST_EMPTY (&mon_fdset->dup_fds)) { -monitor_fdset_cleanup (mon_fdset); - } +monitor_fdset_free (mon_fdset);return ; } }
monitor/hmp.c +0 -2
@@ -1437,11 +1437,9 @@ static void monitor_event(void *opaque, QEMUChrEvent event)monitor_resume (mon); }qemu_mutex_unlock (&mon->mon_lock); - mon_refcount++;break ;case CHR_EVENT_CLOSED : - mon_refcount--;monitor_fdsets_cleanup ();break ;
monitor/monitor-internal.h +0 -1
@@ -168,7 +168,6 @@ extern bool qmp_dispatcher_co_shutdown;extern QmpCommandList qmp_commands, qmp_cap_negotiation_commands;extern QemuMutex monitor_lock;extern MonitorList mon_list; -extern int mon_refcount;extern HMPCommand hmp_cmds[];
monitor/monitor.c +0 -1
@@ -71,7 +71,6 @@ static GHashTable *monitor_qapi_event_state;static GHashTable *coroutine_mon;/* Maps Coroutine* to Monitor* */ MonitorList mon_list; -int mon_refcount;static bool monitor_destroyed;Monitor *monitor_cur (void )
monitor/qmp.c +0 -2
@@ -466,7 +466,6 @@ static void monitor_qmp_event(void *opaque, QEMUChrEvent event) data =qmp_greeting (mon);qmp_send_response (mon, data);qobject_unref (data); - mon_refcount++;break ;case CHR_EVENT_CLOSED :/*
@@ -479,7 +478,6 @@ static void monitor_qmp_event(void *opaque, QEMUChrEvent event)json_message_parser_destroy (&mon->parser);json_message_parser_init (&mon->parser, handle_qmp_command, mon,NULL ); - mon_refcount--;monitor_fdsets_cleanup ();break ;case CHR_EVENT_BREAK :
Signature · 1 line
2.35.3
PATCH v3 8/16 monitor: Simplify fdset and fd removal
In reply to Fabiano RosasRemove fds right away instead of setting the ->removed flag. We don't need the extra complexity of having a cleanup function reap the removed entries at a later time.
--- monitor/fds.c | 27 ++++++--------------------- 1 file changed, 6 insertions(+), 21 deletions(-)
monitor/fds.c +6 -21
@@ -43,7 +43,6 @@ struct mon_fd_t {typedef struct MonFdsetFd MonFdsetFd ;struct MonFdsetFd {int fd; -bool removed;char *opaque;QLIST_ENTRY (MonFdsetFd) next; };
@@ -193,20 +192,6 @@ static void monitor_fdset_fd_free(MonFdsetFd *mon_fdset_fd)g_free (mon_fdset_fd); } -static void monitor_fdset_cleanup (MonFdset *mon_fdset) -{ -MonFdsetFd *mon_fdset_fd; -MonFdsetFd *mon_fdset_fd_next; - - QLIST_FOREACH_SAFE(mon_fdset_fd, &mon_fdset->fds, next, mon_fdset_fd_next) { -if (mon_fdset_fd->removed) { -monitor_fdset_fd_free (mon_fdset_fd); - } - } - -monitor_fdset_free_if_empty (mon_fdset); -} -void monitor_fdsets_cleanup (void ) {MonFdset *mon_fdset;
@@ -281,7 +266,7 @@ void qmp_get_win32_socket(const char *infos, const char *fdname, Error **errp)void qmp_remove_fd (int64_t fdset_id,bool has_fd,int64_t fd,Error **errp) {MonFdset *mon_fdset; -MonFdsetFd *mon_fdset_fd; +MonFdsetFd *mon_fdset_fd, *mon_fdset_fd_next;char fd_str[60 ];QEMU_LOCK_GUARD (&mon_fdsets_lock);
@@ -289,21 +274,22 @@ void qmp_remove_fd(int64_t fdset_id, bool has_fd, int64_t fd, Error **errp)if (mon_fdset->id != fdset_id) {continue ; } - QLIST_FOREACH(mon_fdset_fd, &mon_fdset->fds, next) { +QLIST_FOREACH_SAFE (mon_fdset_fd, &mon_fdset->fds, next, + mon_fdset_fd_next) {if (has_fd) {if (mon_fdset_fd->fd != fd) {continue ; } - mon_fdset_fd->removed = true; +monitor_fdset_fd_free (mon_fdset_fd);break ; }else { - mon_fdset_fd->removed = true; +monitor_fdset_fd_free (mon_fdset_fd); } }if (has_fd && !mon_fdset_fd) { goto error; } -monitor_fdset_cleanup (mon_fdset); +monitor_fdset_free_if_empty (mon_fdset);return ; }
@@ -413,7 +399,6 @@ AddfdInfo *monitor_fdset_add_fd(int fd, bool has_fdset_id, int64_t fdset_id, mon_fdset_fd =g_malloc0 (sizeof (*mon_fdset_fd)); mon_fdset_fd->fd = fd; - mon_fdset_fd->removed = false; mon_fdset_fd->opaque =g_strdup (opaque);QLIST_INSERT_HEAD (&mon_fdset->fds, mon_fdset_fd, next);
Signature · 1 line
2.35.3
PATCH v3 9/16 monitor: Report errors from monitor_fdset_dup_fd_add
In reply to Fabiano RosasI'm keeping the EACCES because callers expect to be able to look at errno.
--- include/monitor/monitor.h | 2 +- monitor/fds.c | 10 +++++++++- stubs/fdset.c | 2 +- util/osdep.c | 10 +--------- 4 files changed, 12 insertions(+), 12 deletions(-)
include/monitor/monitor.h +1 -1
@@ -51,7 +51,7 @@ int monitor_read_password(MonitorHMP *mon, ReadLineFunc *readline_func,AddfdInfo *monitor_fdset_add_fd (int fd,bool has_fdset_id,int64_t fdset_id,const char *opaque,Error **errp); -int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags); +int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags,Error **errp);void monitor_fdset_dup_fd_remove (int dup_fd); void monitor_register_hmp(const char *name, bool info,
monitor/fds.c +9 -1
@@ -409,9 +409,10 @@ AddfdInfo *monitor_fdset_add_fd(int fd, bool has_fdset_id, int64_t fdset_id,return fdinfo; } -int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags) +int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags,Error **errp) {#ifdef _WIN32 +error_setg (errp,"Platform does not support fd passing (fdset)" );return -ENOENT ;#else MonFdset *mon_fdset;
@@ -431,6 +432,8 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags) QLIST_FOREACH(mon_fdset_fd, &mon_fdset->fds, next) { mon_fd_flags =fcntl (mon_fdset_fd->fd,F_GETFL );if (mon_fd_flags ==-1 ) { +error_setg (errp,"Failed to read file status flags for fd=%d" , + mon_fdset_fd->fd);return -1 ; }
@@ -442,11 +445,15 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)if (fd ==-1 ) { errno =EACCES ; +error_setg (errp, +"Failed to find file descriptor with matching flags=0x%x" , + flags);return -1 ; } dup_fd =qemu_dup_flags (fd, flags);if (dup_fd ==-1 ) { +error_setg (errp,"Failed to dup() given file descriptor fd=%d" , fd);return -1 ; }
@@ -456,6 +463,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags) return dup_fd; } +error_setg (errp,"Failed to find fdset /dev/fdset/%" PRId64, fdset_id); errno =ENOENT ;return -1 ;#endif
stubs/fdset.c +1 -1
@@ -3,7 +3,7 @@#include "monitor/monitor.h" #include "../monitor/monitor-internal.h" -int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags) +int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags,Error **errp) { errno =ENOSYS ;return -1 ;
util/osdep.c +1 -9
@@ -310,7 +310,6 @@ qemu_open_internal(const char *name, int flags, mode_t mode, Error **errp)/* Attempt dup of fd from fd set */ if (strstart (name,"/dev/fdset/" , &fdset_id_str)) {int64_t fdset_id; -int dupfd; fdset_id =qemu_parse_fdset (fdset_id_str);if (fdset_id ==-1 ) {
@@ -319,14 +318,7 @@ qemu_open_internal(const char *name, int flags, mode_t mode, Error **errp) return -1; } - dupfd =monitor_fdset_dup_fd_add (fdset_id, flags); -if (dupfd ==-1 ) { -error_setg_errno (errp, errno,"Could not dup FD for %s flags %x" , - name, flags); -return -1 ; - } - -return dupfd; +return monitor_fdset_dup_fd_add (fdset_id, flags, errp); }#endif
Signature · 1 line
2.35.3
PATCH v3 10/16 io: Stop using qemu_open_old in channel-file
In reply to Fabiano RosasWe want to make use of the Error object to report fdset errors from qemu_open_internal() and passing the error pointer to qemu_open_old() would require changing all callers. Move the file channel to the new API instead.
--- io/channel-file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
io/channel-file.c +5 -3
@@ -68,11 +68,13 @@ qio_channel_file_new_path(const char *path, ioc =QIO_CHANNEL_FILE (object_new (TYPE_QIO_CHANNEL_FILE )); - ioc->fd =qemu_open_old (path, flags, mode); +if (flags &O_CREAT ) { + ioc->fd =qemu_create (path, flags & ~O_CREAT , mode, errp); + }else { + ioc->fd =qemu_open (path, flags, errp); + }if (ioc->fd <0 ) {object_unref (OBJECT (ioc)); -error_setg_errno (errp, errno, -"Unable to open %s" , path);return NULL ; }
Signature · 1 line
2.35.3
PATCH v3 11/16 migration: Add direct-io parameter
In reply to Fabiano RosasAdd the direct-io migration parameter that tells the migration code to use O_DIRECT when opening the migration stream file whenever possible.
This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 35 ++++++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 21 +++++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 76 insertions(+), 3 deletions(-)
include/qemu/osdep.h +2 -0
@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )
migration/migration-hmp-cmds.c +11 -0
@@ -351,6 +351,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) { +monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + } }qapi_free_MigrationParameters (params);
@@ -624,6 +631,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_mode = true;visit_type_MigMode (v, param, &p->mode, &err);break ; +case MIGRATION_PARAMETER_DIRECT_IO : + p->has_direct_io = true; +visit_type_bool (v, param, &p->direct_io, &err); +break ;default :assert (0 ); }
migration/options.c +35 -0
@@ -702,6 +702,25 @@ bool migrate_cpu_throttle_tailslow(void)return s->parameters.cpu_throttle_tailslow; } +bool migrate_direct_io (void ) +{ +MigrationState *s =migrate_get_current (); + +/* +* O_DIRECT is only supported with mapped-ram and multifd. +* +* mapped-ram is needed because filesystems impose restrictions on +* O_DIRECT IO alignment (see MAPPED_RAM_FILE_OFFSET_ALIGNMENT). +* +* multifd is needed to keep the unaligned portion of the stream +* isolated to the main migration thread while multifd channels +* process the aligned data with O_DIRECT enabled. +*/ +return s->parameters.direct_io && + s->capabilities[MIGRATION_CAPABILITY_MAPPED_RAM ] && + s->capabilities[MIGRATION_CAPABILITY_MULTIFD ]; +} +uint64_t migrate_downtime_limit (void ) {MigrationState *s =migrate_get_current ();
@@ -905,6 +924,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->mode = s->parameters.mode; params->has_zero_page_detection = true; params->zero_page_detection = s->parameters.zero_page_detection; + params->has_direct_io = true; + params->direct_io = s->parameters.direct_io;return params; }
@@ -937,6 +958,7 @@ void migrate_params_init(MigrationParameters *params) params->has_vcpu_dirty_limit = true; params->has_mode = true; params->has_zero_page_detection = true; + params->has_direct_io = true; }/*
@@ -1110,6 +1132,11 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)return false; } +if (params->has_direct_io && params->direct_io && !qemu_has_direct_io ()) { +error_setg (errp,"No build-time support for direct-io" ); +return false; + } +return true; }
@@ -1216,6 +1243,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_zero_page_detection) { dest->zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + dest->direct_io = params->direct_io; + } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)
@@ -1341,6 +1372,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_zero_page_detection) { s->parameters.zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + s->parameters.direct_io = params->direct_io; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)
migration/options.h +1 -0
@@ -69,6 +69,7 @@ uint32_t migrate_checkpoint_delay(void);uint8_t migrate_cpu_throttle_increment (void );uint8_t migrate_cpu_throttle_initial (void );bool migrate_cpu_throttle_tailslow (void ); +bool migrate_direct_io (void );uint64_t migrate_downtime_limit (void );uint8_t migrate_max_cpu_throttle (void );uint64_t migrate_max_bandwidth (void );
qapi/migration.json +18 -3
@@ -821,6 +821,10 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# only has effect if the @mapped-ram capability is enabled. +# (Since 9.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and
@@ -845,7 +849,8 @@ { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit', 'mode', - 'zero-page-detection'] } + 'zero-page-detection', + 'direct-io'] } ## # @MigrateSetParameters:
@@ -991,6 +996,10 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# only has effect if the @mapped-ram capability is enabled. +# (Since 9.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and
@@ -1030,7 +1039,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @migrate-set-parameters:
@@ -1190,6 +1200,10 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# only has effect if the @mapped-ram capability is enabled. +# (Since 9.1) +# # Features: # # @unstable: Members @x-checkpoint-delay and
@@ -1226,7 +1240,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @query-migrate-parameters:
util/osdep.c +9 -0
@@ -282,6 +282,15 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive) }#endif +bool qemu_has_direct_io (void ) +{ +#ifdef O_DIRECT +return true; +#else +return false; +#endif +} +static int qemu_open_cloexec (const char *name,int flags,mode_t mode) {int ret;
Signature · 1 line
2.35.3
PATCH v3 12/16 migration/multifd: Add direct-io support
In reply to Fabiano RosasWhen multifd is used along with mapped-ram, we can take benefit of a filesystem that supports the O_DIRECT flag and perform direct I/O in the multifd threads. This brings a significant performance improvement because direct-io writes bypass the page cache which would otherwise be thrashed by the multifd data which is unlikely to be needed again in a short period of time.
To be able to use a multifd channel opened with O_DIRECT, we must ensure that a certain aligment is used. Filesystems usually require a block-size alignment for direct I/O. The way to achieve this is by enabling the mapped-ram feature, which already aligns its I/O properly (see MAPPED_RAM_FILE_OFFSET_ALIGNMENT at ram.c).
By setting O_DIRECT on the multifd channels, all writes to the same file descriptor need to be aligned as well, even the ones that come from outside multifd, such as the QEMUFile I/O from the main migration code. This makes it impossible to use the same file descriptor for the QEMUFile and for the multifd channels. The various flags and metadata written by the main migration code will always be unaligned by virtue of their small size. To workaround this issue, we'll require a second file descriptor to be used exclusively for direct I/O.
The second file descriptor can be obtained by QEMU by re-opening the migration file (already possible), or by being provided by the user or management application (support to be added in future patches).
--- migration/file.c | 33 ++++++++++++++++++++++++++++----- migration/file.h | 1 - migration/migration.c | 23 +++++++++++++++++++++++ 3 files changed, 51 insertions(+), 6 deletions(-)
migration/file.c +28 -5
@@ -50,12 +50,31 @@ void file_cleanup_outgoing_migration(void) outgoing_args.fname =NULL ; } +static void file_enable_direct_io (int *flags) +{ +#ifdef O_DIRECT + *flags |=O_DIRECT ; +#else +/* it should have been rejected when setting the parameter */ +g_assert_not_reached (); +#endif +} +bool file_send_channel_create (gpointer opaque,Error **errp) {QIOChannelFile *ioc;int flags =O_WRONLY ;bool ret = true; +if (migrate_direct_io ()) { +/* +* Enable O_DIRECT for the secondary channels. These are used +* for sending ram pages and writes should be guaranteed to be +* aligned to at least page size. +*/ +file_enable_direct_io (&flags); + } + ioc =qio_channel_file_new_path (outgoing_args.fname, flags,0 , errp);if (!ioc) { ret = false;
@@ -117,21 +136,25 @@ static gboolean file_accept_incoming_migration(QIOChannel *ioc,return G_SOURCE_REMOVE ; } -void file_create_incoming_channels (QIOChannel *ioc,Error **errp) +static void file_create_incoming_channels (QIOChannel *ioc,char *filename, +Error **errp) { -int i, fd, channels =1 ; +int i, channels =1 ; g_autofree QIOChannel **iocs = NULL; +int flags =O_RDONLY ;if (migrate_multifd ()) { channels +=migrate_multifd_channels (); +if (migrate_direct_io ()) { +file_enable_direct_io (&flags); + } } iocs = g_new0(QIOChannel *, channels); - fd =QIO_CHANNEL_FILE (ioc)->fd; iocs[0 ] = ioc;for (i =1 ; i < channels; i++) { -QIOChannelFile *fioc =qio_channel_file_new_dupfd (fd, errp); +QIOChannelFile *fioc =qio_channel_file_new_path (filename, flags,0 , errp);if (!fioc) {while (i) {
@@ -171,7 +194,7 @@ void file_start_incoming_migration(FileMigrationArgs *file_args, Error **errp)return ; } -file_create_incoming_channels (QIO_CHANNEL (fioc), errp); +file_create_incoming_channels (QIO_CHANNEL (fioc), filename, errp); } int file_write_ramblock_iov(QIOChannel *ioc, const struct iovec *iov,
migration/file.h +0 -1
@@ -20,7 +20,6 @@ void file_start_outgoing_migration(MigrationState *s,int file_parse_offset (char *filespec,uint64_t *offsetp,Error **errp);void file_cleanup_outgoing_migration (void );bool file_send_channel_create (gpointer opaque,Error **errp); -void file_create_incoming_channels (QIOChannel *ioc,Error **errp);int file_write_ramblock_iov (QIOChannel *ioc,const struct iovec *iov,int niov,RAMBlock *block,Error **errp);int multifd_file_recv_data (MultiFDRecvParams *p,Error **errp);
migration/migration.c +23 -0
@@ -155,6 +155,16 @@ static bool migration_needs_seekable_channel(void)return migrate_mapped_ram (); } +static bool migration_needs_extra_fds (void ) +{ +/* +* When doing direct-io, multifd requires two different, +* non-duplicated file descriptors so we can use one of them for +* unaligned IO. +*/ +return migrate_multifd () &&migrate_direct_io (); +} +static bool transport_supports_seeking (MigrationAddress *addr) {if (addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ) {
@@ -164,6 +174,12 @@ static bool transport_supports_seeking(MigrationAddress *addr)return false; } +static bool transport_supports_extra_fds (MigrationAddress *addr) +{ +/* file: works because QEMU can open it multiple times */ +return addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ; +} +static bool migration_channels_and_transport_compatible (MigrationAddress *addr,Error **errp)
@@ -180,6 +196,13 @@ migration_channels_and_transport_compatible(MigrationAddress *addr,return false; } +if (migration_needs_extra_fds () && + !transport_supports_extra_fds (addr)) { +error_setg (errp, +"Migration requires a transport that allows for extra fds (e.g. file)" ); +return false; + } +return true; }
Signature · 1 line
2.35.3
Re: [PATCH v3 12/16] migration/multifd: Add direct-io support
Quoted 22 lines · Fabiano Rosas
When multifd is used along with mapped-ram, we can take benefit of a filesystem that supports the
O_DIRECTflag and perform direct I/O in the multifd threads. This brings a significant performance improvement because direct-io writes bypass the page cache which would otherwise be thrashed by the multifd data which is unlikely to be needed again in a short period of time.To be able to use a multifd channel opened with
O_DIRECT, we must ensure that a certain aligment is used. Filesystems usually require a block-size alignment for direct I/O. The way to achieve this is by enabling the mapped-ram feature, which already aligns its I/O properly (seeMAPPED_RAM_FILE_OFFSET_ALIGNMENTatram.c).By setting
O_DIRECTon the multifd channels, all writes to the same file descriptor need to be aligned as well, even the ones that come from outside multifd, such as the QEMUFile I/O from the main migration code. This makes it impossible to use the same file descriptor for the QEMUFile and for the multifd channels. The various flags and metadata written by the main migration code will always be unaligned by virtue of their small size. To workaround this issue, we'll require a second file descriptor to be used exclusively for direct I/O.
Quoted 5 lines
The second file descriptor can be obtained by QEMU by re-opening the migration file (already possible), or by being provided by the user or management application (support to be added in future patches).
Signed-off-by Fabiano Rosas <[email protected]>
Signature · 1 line
Peter Xu
PATCH v3 13/16 tests/qtest/migration: Add tests for file migration with direct-io
In reply to Fabiano RosasThe tests are only allowed to run in systems that know about the O_DIRECT flag and in filesystems which support it.
Note: this also brings back migrate_set_parameter_bool() which went away when we removed the compression tests. I copied it verbatim.
--- tests/qtest/migration-helpers.c | 44 +++++++++++++++++++++++ tests/qtest/migration-helpers.h | 8 +++++ tests/qtest/migration-test.c | 62 +++++++++++++++++++++++++++++++++ 3 files changed, 114 insertions(+)
tests/qtest/migration-helpers.c +44 -0
@@ -18,6 +18,7 @@#include "qapi/error.h" #include "qapi/qmp/qlist.h" #include "qemu/cutils.h" +#include "qemu/memalign.h" #include "migration-helpers.h"
@@ -473,3 +474,46 @@ void migration_test_add(const char *path, void (*fn)(void))qtest_add_data_func_full (path, test, migration_test_wrapper, migration_test_destroy); } + +#ifdef O_DIRECT +/* +* Probe for O_DIRECT support on the filesystem. Since this is used +* for tests, be conservative, if anything fails, assume it's +* unsupported. +*/ +bool probe_o_direct_support (const char *tmpfs) +{ + g_autofree char *filename = g_strdup_printf("%s/probe-o-direct", tmpfs); +int fd, flags =O_CREAT |O_RDWR |O_TRUNC |O_DIRECT ; +void *buf; +ssize_t ret, len; +uint64_t offset; + + fd =open (filename, flags,0660 ); +if (fd <0 ) { +unlink (filename); +return false; + } + +/* +* Using 1MB alignment as conservative choice to satisfy any +* plausible architecture default page size, and/or filesystem +* alignment restrictions. +*/ + len =0x100000 ; + offset =0x100000 ; + + buf =qemu_try_memalign (len, len); +g_assert (buf); + + ret =pwrite (fd, buf, len, offset); +unlink (filename); +g_free (buf); + +if (ret <0 ) { +return false; + } + +return true; +} +#endif
tests/qtest/migration-helpers.h +8 -0
@@ -54,5 +54,13 @@ char *find_common_machine_version(const char *mtype, const char *var1, const char *var2);char *resolve_machine_version (const char *alias,const char *var1,const char *var2); +#ifdef O_DIRECT +bool probe_o_direct_support (const char *tmpfs); +#else +static inline bool probe_o_direct_support (const char *tmpfs) +{ +return false; +} +#endif void migration_test_add (const char *path,void (*fn)(void ));#endif /* MIGRATION_HELPERS_H */
tests/qtest/migration-test.c +62 -0
80 lines not on this page · raw message
Signature · 1 line
2.35.3
PATCH v3 14/16 monitor: fdset: Match against O_DIRECT
In reply to Fabiano RosasWe're about to enable the use of O_DIRECT in the migration code and due to the alignment restrictions imposed by filesystems we need to make sure the flag is only used when doing aligned IO.
The migration will do parallel IO to different regions of a file, so we need to use more than one file descriptor. Those cannot be obtained by duplicating (dup()) since duplicated file descriptors share the file status flags, including O_DIRECT. If one migration channel does unaligned IO while another sets O_DIRECT to do aligned IO, the filesystem would fail the unaligned operation.
The add-fd QMP command along with the fdset code are specifically designed to allow the user to pass a set of file descriptors with different access flags into QEMU to be later fetched by code that needs to alternate between those flags when doing IO.
Extend the fdset matching to behave the same with the O_DIRECT flag.
--- monitor/fds.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
monitor/fds.c +6 -1
@@ -424,6 +424,11 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags, Error **errp)int fd =-1 ;int dup_fd;int mon_fd_flags; +int mask =O_ACCMODE ; + +#ifdef O_DIRECT + mask |=O_DIRECT ; +#endif if (mon_fdset->id != fdset_id) {continue ;
@@ -437,7 +442,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags, Error **errp)return -1 ; } -if ((flags &O_ACCMODE ) == (mon_fd_flags &O_ACCMODE )) { +if ((flags & mask) == (mon_fd_flags & mask)) { fd = mon_fdset_fd->fd;break ; }
Signature · 1 line
2.35.3
PATCH v3 15/16 migration: Add documentation for fdset with multifd + file
In reply to Fabiano RosasWith the last few changes to the fdset infrastructure, we now allow multifd to use an fdset when migrating to a file. This is useful for the scenario where the management layer wants to have control over the migration file.
By receiving the file descriptors directly, QEMU can delegate some high level operating system operations to the management layer (such as mandatory access control). The management layer might also want to add its own headers before the migration stream.
Document the "file:/dev/fdset/#" syntax for the multifd migration with mapped-ram. The requirements for the fdset mechanism are:
- the fdset must contain two fds that are not duplicates between themselves;
- if direct-io is to be used, exactly one of the fds must have the O_DIRECT flag set;
- the file must be opened with WRONLY on the migration source side;
- the file must be opened with RDONLY on the migration destination side.
--- docs/devel/migration/main.rst | 24 +++++++++++++++++++----- docs/devel/migration/mapped-ram.rst | 6 +++++- 2 files changed, 24 insertions(+), 6 deletions(-)
docs/devel/migration/main.rst +19 -5
@@ -47,11 +47,25 @@ over any transport. QEMU interference. Note that QEMU does not flush cached file data/metadata at the end of migration. -In addition, support is included for migration using RDMA, which -transports the page data using ``RDMA``, where the hardware takes care of -transporting the pages, and the load on the CPU is much lower. While the -internals of RDMA migration are a bit different, this isn't really visible -outside the RAM migration code. + The file migration also supports using a file that has already been + opened. A set of file descriptors is passed to QEMU via an "fdset" + (see add-fd QMP command documentation). This method allows a + management application to have control over the migration file + opening operation. There are, however, strict requirements to this + interface if the multifd capability is enabled: + + - the fdset must contain two file descriptors that are not + duplicates between themselves; + - if the direct-io capability is to be used, exactly one of the + file descriptors must have the O_DIRECT flag set; + - the file must be opened with WRONLY on the migration source side + and RDONLY on the migration destination side. + +- rdma migration: support is included for migration using RDMA, which + transports the page data using ``RDMA``, where the hardware takes + care of transporting the pages, and the load on the CPU is much + lower. While the internals of RDMA migration are a bit different, + this isn't really visible outside the RAM migration code. All these migration protocols use the same infrastructure to save/restore state devices. This infrastructure is shared with the
docs/devel/migration/mapped-ram.rst +5 -1
@@ -16,7 +16,7 @@ location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment -restrictions (direct-io support not yet implemented). +restrictions. Usage -----
@@ -35,6 +35,10 @@ Use a ``file:`` URL for migration: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating. +For best performance enable the ``direct-io`` parameter as well: + + ``migrate_set_parameter direct-io on`` + Use-cases ---------
Signature · 1 line
2.35.3
PATCH v3 16/16 tests/qtest/migration: Add a test for mapped-ram with passing of fds
In reply to Fabiano RosasAdd a multifd test for mapped-ram with passing of fds into QEMU. This is how libvirt will consume the feature.
There are a couple of details to the fdset mechanism:
- multifd needs two distinct file descriptors (not duplicated with dup()) so it can enable O_DIRECT only on the channels that do aligned IO. The dup() system call creates file descriptors that share status flags, of which O_DIRECT is one.
- the open() access mode flags used for the fds passed into QEMU need to match the flags QEMU uses to open the file. Currently O_WRONLY for src and O_RDONLY for dst.
Note that fdset code goes under _WIN32 because fd passing is not supported on Windows.
---
- dropped Peter's r-b
- stopped removing the fdset at the end of the tests. The migration code should always cleanup after itself.
--- tests/qtest/migration-test.c | 98 ++++++++++++++++++++++++++++++++++-- 1 file changed, 95 insertions(+), 3 deletions(-)
tests/qtest/migration-test.c +95 -3
@@ -2024,11 +2024,18 @@ static void test_precopy_file(void)#ifndef _WIN32static void fdset_add_fds (QTestState *qts,const char *file,int flags, -int num_fds) +int num_fds,bool direct_io) {for (int i =0 ; i < num_fds; i++) {int fd; +#ifdef O_DIRECT +/* only secondary channels can use direct-io */ +if (direct_io && i !=0 ) { + flags |=O_DIRECT ; + } +#endif + fd =open (file, flags,0660 );assert (fd !=-1 );
@@ -2042,8 +2049,8 @@ static void *file_offset_fdset_start_hook(QTestState *from, QTestState *to) { g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); -fdset_add_fds (from, file,O_WRONLY ,1 ); -fdset_add_fds (to, file,O_RDONLY ,1 ); +fdset_add_fds (from, file,O_WRONLY ,1 , false); +fdset_add_fds (to, file,O_RDONLY ,1 , false);return NULL ; }
@@ -2215,6 +2222,84 @@ static void test_multifd_file_mapped_ram_dio(void)test_file_common (&args, true); } +#ifndef _WIN32 +static void multifd_mapped_ram_fdset_end (QTestState *from,QTestState *to, +void *opaque) +{ +QDict *resp; +QList *fdsets; + +/* +* Make sure no fdsets are left after migration, otherwise a +* second migration would fail due fdset reuse. +*/ + resp =qtest_qmp (from,"{'execute': 'query-fdsets', " +"'arguments': {}}" ); +g_assert (qdict_haskey (resp,"return" )); + fdsets =qdict_get_qlist (resp,"return" ); +g_assert (fdsets &&qlist_empty (fdsets)); +} + +static void *multifd_mapped_ram_fdset_dio (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); + +fdset_add_fds (from, file,O_WRONLY ,2 , true); +fdset_add_fds (to, file,O_RDONLY ,2 , true); + +migrate_multifd_mapped_ram_start (from, to); +migrate_set_parameter_bool (from,"direct-io" , true); +migrate_set_parameter_bool (to,"direct-io" , true); + +return NULL ; +} + +static void *multifd_mapped_ram_fdset (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); + +fdset_add_fds (from, file,O_WRONLY ,2 , false); +fdset_add_fds (to, file,O_RDONLY ,2 , false); + +migrate_multifd_mapped_ram_start (from, to); + +return NULL ; +} + +static void test_multifd_file_mapped_ram_fdset (void ) +{ + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d", +FILE_TEST_OFFSET ); +MigrateCommon args = { + .connect_uri = uri, + .listen_uri ="defer" , + .start_hook = multifd_mapped_ram_fdset, + .finish_hook = multifd_mapped_ram_fdset_end, + }; + +test_file_common (&args, true); +} + +static void test_multifd_file_mapped_ram_fdset_dio (void ) +{ + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d", +FILE_TEST_OFFSET ); +MigrateCommon args = { + .connect_uri = uri, + .listen_uri ="defer" , + .start_hook = multifd_mapped_ram_fdset_dio, + .finish_hook = multifd_mapped_ram_fdset_end, + }; + +if (!probe_o_direct_support (tmpfs)) { +g_test_skip ("Filesystem does not support O_DIRECT" ); +return ; + } + +test_file_common (&args, true); +} +#endif /* !_WIN32 */ +static void test_precopy_tcp_plain (void ) {MigrateCommon args = {
@@ -3654,6 +3739,13 @@ int main(int argc, char **argv)migration_test_add ("/migration/multifd/file/mapped-ram/dio" , test_multifd_file_mapped_ram_dio); +#ifndef _WIN32 +migration_test_add ("/migration/multifd/file/mapped-ram/fdset" , + test_multifd_file_mapped_ram_fdset); +migration_test_add ("/migration/multifd/file/mapped-ram/fdset/dio" , + test_multifd_file_mapped_ram_fdset_dio); +#endif +#ifdef CONFIG_GNUTLS migration_test_add ("/migration/precopy/unix/tls/psk" , test_precopy_unix_tls_psk);
Signature · 1 line
2.35.3
Re: [PATCH v3 16/16] tests/qtest/migration: Add a test for mapped-ram with passing of fds
Quoted 17 lines · Fabiano Rosas
Add a multifd test for mapped-ram with passing of fds into QEMU. This is how libvirt will consume the feature.
There are a couple of details to the fdset mechanism:
- multifd needs two distinct file descriptors (not duplicated with
dup()) so it can enableO_DIRECTonly on the channels that do aligned IO. Thedup()system call creates file descriptors that share status flags, of whichO_DIRECTis one.- the
open()access mode flags used for the fds passed into QEMU need to match the flags QEMU uses to open the file. CurrentlyO_WRONLYfor src andO_RDONLYfor dst.Note that fdset code goes under
_WIN32because fd passing is not supported on Windows.
Quoted 6 lines
Signed-off-by Fabiano Rosas <[email protected]>---
- dropped Peter's r-b- stopped removing the fdset at the end of the tests. The migration code should always cleanup after itself.
Ah, that looks also ok.
Signature · 1 line
Peter Xu
Re: [PATCH v3 16/16] tests/qtest/migration: Add a test for mapped-ram with passing of fds
Quoted 18 lines · Peter Xu
Quoted 17 lines · Fabiano Rosas
Add a multifd test for mapped-ram with passing of fds into QEMU. This is how libvirt will consume the feature.
There are a couple of details to the fdset mechanism:
- multifd needs two distinct file descriptors (not duplicated with
dup()) so it can enableO_DIRECTonly on the channels that do aligned IO. Thedup()system call creates file descriptors that share status flags, of whichO_DIRECTis one.- the
open()access mode flags used for the fds passed into QEMU need to match the flags QEMU uses to open the file. CurrentlyO_WRONLYfor src andO_RDONLYfor dst.Note that fdset code goes under
_WIN32because fd passing is not supported on Windows.
Quoted 8 lines
Quoted 6 lines
Signed-off-by Fabiano Rosas <[email protected]>---
- dropped Peter's r-b- stopped removing the fdset at the end of the tests. The migration code should always cleanup after itself.
Ah, that looks also ok.
I made a mistake here. We still need to require that the management layer explicitly removes the fds they added by calling qmp_remove_fd().
The reason I thought it was ok to not remove the fds was that after your suggestion to use monitor_fdset_free_if_empty() in patch 7, I mistakenly put monitor_fdset_free() instead. So the qmp_remove_fd() was not finding any fdsets and I thought that was QEMU removing everything. Which is silly, because the whole purpose of the patch is to not do that.
I think I'll just fix this in the migration tree. It's just a revert to v2 of this patch and the correction to patch 7.
Re: [PATCH v3 16/16] tests/qtest/migration: Add a test for mapped-ram with passing of fds
Quoted 32 lines · Fabiano Rosas
Quoted 18 lines · Peter Xu
Quoted 17 lines · Fabiano Rosas
Add a multifd test for mapped-ram with passing of fds into QEMU. This is how libvirt will consume the feature.
There are a couple of details to the fdset mechanism:
- multifd needs two distinct file descriptors (not duplicated with
dup()) so it can enableO_DIRECTonly on the channels that do aligned IO. Thedup()system call creates file descriptors that share status flags, of whichO_DIRECTis one.- the
open()access mode flags used for the fds passed into QEMU need to match the flags QEMU uses to open the file. CurrentlyO_WRONLYfor src andO_RDONLYfor dst.Note that fdset code goes under
_WIN32because fd passing is not supported on Windows.Quoted 8 lines
Quoted 6 lines
Signed-off-by Fabiano Rosas <[email protected]>---
- dropped Peter's r-b- stopped removing the fdset at the end of the tests. The migration code should always cleanup after itself.
Ah, that looks also ok.
I made a mistake here. We still need to require that the management layer explicitly removes the fds they added by calling
qmp_remove_fd().
Quoted 8 lines
The reason I thought it was ok to not remove the fds was that after your suggestion to use
monitor_fdset_free_if_empty()in patch 7, I mistakenly putmonitor_fdset_free()instead. So theqmp_remove_fd()was not finding any fdsets and I thought that was QEMU removing everything. Which is silly, because the whole purpose of the patch is to not do that.I think I'll just fix this in the migration tree. It's just a revert to v2 of this patch and the correction to patch 7.
Ah OK. I thought you were talking about when QEMU exit()s, which should cleanup everything too.
Personally I guess I kind of ignored that remove-fd api anyway being there or not, as it seems nobody understand why it needs to exist in the first place..
Signature · 1 line
Peter Xu