migration/mapped-ram: Add direct-io support
cover migration/mapped-ram: Add direct-io support
Hi everyone, here's the rest of the migration "mapped-ram" feature that didn't get merged for 9.0. This series adds support for direct I/O, the missing piece to get the desired performance improvements.
There's 3 parts to this:
1- The plumbing for the new "direct-io" migration parameter. With this
we can already use direct-io with the file transport + multifd + mapped-ram. Patches 1-3.
Due to the alignment requirements of O_DIRECT and the fact that multifd runs the channels in parallel with the migration thread, we must open the migration file two times, one with O_DIRECT set and another with it clear.
If the user is not passing in a file name which QEMU can open at will, we must then require that the user pass the two file descriptors with the flags already properly set. We'll use the already existing fdset + QMP add-fd infrastructure for this.
2- Changes to the fdset infrastructure to support O_DIRECT. We need
those to be able to select from the user-provided fdset the file descriptor that contains the O_DIRECT flag. Patches 4-5.
3- Some fdset validation to make sure the two-fds requirement is being
met. Patches 6-7.
CI run: https://gitlab.com/farosas/qemu/-/pipelines/1269352083
Fabiano Rosas (9):
monitor: Honor QMP request for fd removal immediately
migration: Fix file migration with fdset tests/qtest/migration: Fix file migration offset check
migration: Add direct-io parameter
migration/multifd: Add direct-io support tests/qtest/migration: Add tests for file migration with direct-io
monitor: fdset: Match against O_DIRECT
migration: Add support for fdset with multifd + file tests/qtest/migration: Add a test for mapped-ram with passing of fds
docs/devel/migration/main.rst | 18 +++ docs/devel/migration/mapped-ram.rst | 6 +- include/qemu/osdep.h | 2 + migration/file.c | 108 ++++++++++++++- migration/migration-hmp-cmds.c | 11 ++ migration/migration.c | 23 ++++ migration/options.c | 30 +++++ migration/options.h | 1 + monitor/fds.c | 13 +- qapi/migration.json | 18 ++- tests/qtest/migration-helpers.c | 42 ++++++ tests/qtest/migration-helpers.h | 1 + tests/qtest/migration-test.c | 202 +++++++++++++++++++++++++++- util/osdep.c | 9 ++ 14 files changed, 465 insertions(+), 19 deletions(-)
base-commit: a118c4aff4087eafb68f7132b233ad548cf16376
Signature · 1 line
2.35.3
PATCH 1/9 monitor: Honor QMP request for fd removal immediately
We're enabling using the fdset interface to pass file descriptors for use in the migration code. Since migrations can happen more than once during the VMs lifetime, we need a way to remove an fd from the fdset at the end of migration.
The current code only removes an fd from the fdset if the VM is running. This causes a QMP call to "remove-fd" to not actually remove the fd if the VM happens to be stopped.
While the fd would eventually be removed when monitor_fdset_cleanup() is called again, the user request should be honored and the fd actually removed. Calling remove-fd + query-fdset shows a recently removed fd still present.
The runstate_is_running() check was introduced by commit ebe52b592d ("monitor: Prevent removing fd from set during init"), which by the shortlog indicates that they were trying to avoid removing an yet-unduplicated fd too early.
I don't see why an fd explicitly removed with qmp_remove_fd() should be under runstate_is_running(). I'm assuming this was a mistake when adding the parenthesis around the expression.
Move the runstate_is_running() check to apply only to the QLIST_EMPTY(dup_fds) side of the expression and ignore it when mon_fdset_fd->removed has been explicitly set.
--- monitor/fds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
monitor/fds.c +3 -3
@@ -173,9 +173,9 @@ static void monitor_fdset_cleanup(MonFdset *mon_fdset)MonFdsetFd *mon_fdset_fd_next; QLIST_FOREACH_SAFE(mon_fdset_fd, &mon_fdset->fds, next, mon_fdset_fd_next) { -if ((mon_fdset_fd->removed || - (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 )) && -runstate_is_running ()) { +if (mon_fdset_fd->removed || + (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 && +runstate_is_running ())) {close (mon_fdset_fd->fd);g_free (mon_fdset_fd->opaque);QLIST_REMOVE (mon_fdset_fd, next);
Signature · 1 line
2.35.3
Re: [PATCH 1/9] monitor: Honor QMP request for fd removal immediately
Quoted 19 lines · Fabiano Rosas
We're enabling using the fdset interface to pass file descriptors for use in the migration code. Since migrations can happen more than once during the VMs lifetime, we need a way to remove an fd from the fdset at the end of migration.
The current code only removes an fd from the fdset if the VM is running. This causes a QMP call to "remove-fd" to not actually remove the fd if the VM happens to be stopped.
While the fd would eventually be removed when
monitor_fdset_cleanup()is called again, the user request should be honored and the fd actually removed. Calling remove-fd + query-fdset shows a recently removed fd still present.The
runstate_is_running()check was introduced by commitebe52b592d("monitor: Prevent removing fd from set during init"), which by the shortlog indicates that they were trying to avoid removing an yet-unduplicated fd too early.
Quoted 7 lines
I don't see why an fd explicitly removed with
qmp_remove_fd()should be underrunstate_is_running(). I'm assuming this was a mistake when adding the parenthesis around the expression.Move the
runstate_is_running()check to apply only to theQLIST_EMPTY(dup_fds)side of the expression and ignore it whenmon_fdset_fd->removedhas been explicitly set.
I am confused on why the fdset removal is as complicated. I'm also wondering here whether it's dropped because we checked against "mon_refcount == 0", and maybe monitor_fdset_cleanup() is simply called _before_ a monitor is created? Why do we need such check on the first place?
I'm thinking one case where the only QMP monitor got (for some reason) disconnected, and reconnected again during VM running. Won't current code already lead to unwanted removal of mostly all fds due to mon_refcount==0?
I also am confused why ->removed flags is ever needed, and why we can't already remove the fdsets fds if found matching.
Copy Corey, Eric and Kevin.
Quoted 26 lines
Signed-off-by Fabiano Rosas <[email protected]>--- monitor/fds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)monitor/fds.c +3
-3@@ -173,9 +173,9 @@ static void monitor_fdset_cleanup(MonFdset *mon_fdset)MonFdsetFd *mon_fdset_fd_next; QLIST_FOREACH_SAFE(mon_fdset_fd, &mon_fdset->fds, next, mon_fdset_fd_next) { -if ((mon_fdset_fd->removed || - (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 )) && -runstate_is_running ()) { +if (mon_fdset_fd->removed || + (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 && +runstate_is_running ())) {close (mon_fdset_fd->fd);g_free (mon_fdset_fd->opaque);QLIST_REMOVE (mon_fdset_fd, next);Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH 1/9] monitor: Honor QMP request for fd removal immediately
Hi all, sorry to have been away from this thread for a while, I was trying to catch up on my reviews queue.
Quoted 28 lines · Peter Xu
Quoted 19 lines · Fabiano Rosas
We're enabling using the fdset interface to pass file descriptors for use in the migration code. Since migrations can happen more than once during the VMs lifetime, we need a way to remove an fd from the fdset at the end of migration.
The current code only removes an fd from the fdset if the VM is running. This causes a QMP call to "remove-fd" to not actually remove the fd if the VM happens to be stopped.
While the fd would eventually be removed when
monitor_fdset_cleanup()is called again, the user request should be honored and the fd actually removed. Calling remove-fd + query-fdset shows a recently removed fd still present.The
runstate_is_running()check was introduced by commitebe52b592d("monitor: Prevent removing fd from set during init"), which by the shortlog indicates that they were trying to avoid removing an yet-unduplicated fd too early.Quoted 7 lines
I don't see why an fd explicitly removed with
qmp_remove_fd()should be underrunstate_is_running(). I'm assuming this was a mistake when adding the parenthesis around the expression.Move the
runstate_is_running()check to apply only to theQLIST_EMPTY(dup_fds)side of the expression and ignore it whenmon_fdset_fd->removedhas been explicitly set.
Quoted 6 lines
I am confused on why the fdset removal is as complicated. I'm also wondering here whether it's dropped because we checked against "
mon_refcount== 0", and maybemonitor_fdset_cleanup()is simply called _before_ a monitor is created? Why do we need such check on the first place?
It seems the intention was to reuse monitor_fdset_cleanup() to do cleanup when all monitors connections are closed:
efb87c1697 ("monitor: Clean up fd sets on monitor disconnect")
Author: Corey Bryant <[email protected]> Date: Tue Aug 14 16:43:48 2012 -0400
monitor: Clean up fd sets on monitor disconnect
Fd sets are shared by all monitor connections. Fd sets are considered to be in use while at least one monitor is connected. When the last monitor disconnects, all fds that are members of an fd set with no outstanding dup references are closed. This prevents any fd leakage associated with a client disconnect prior to using a passed fd.
This could have been done differently at monitor_qmp_event():
case CHR_EVENT_CLOSED:
...
mon_refcount--;
if (mon_refcount == 0) {
monitor_fdsets_cleanup();
}But maybe there was a concern about making sure the empty fdsets (last block in monitor_fdset_cleanup) were removed at every refcount decrement and not only when mon_refcount == 0 for some reason.
Quoted 3 lines
I'm thinking one case where the only QMP monitor got (for some reason) disconnected, and reconnected again during VM running. Won't current code already lead to unwanted removal of mostly all fds due to mon_refcount==0?
I think that's the case that the patch in question was trying to avoid. If the last monitor connects and disconnects while fds have not been dup'ed yet, the mon_fdset->dup_fds list will be empty and what you say will happen. There seems to be an assumption that after the guest starts running all fds that need to be dup'ed will already have been dup'ed.
So I think we cannot simply revert the patch as Daniel suggested, because that might regress the original block layer use-case if a monitor open->close causes the removal of all the yet undup'ed fds[1].
For the migration use-case, the dup() only happens after the migrate command has been issued, so the runstate_is_running() check doesn't help us. But it also doesn't hurt. However, we're still exposed to a monitor disconnection removing the undup'ed fds. So AFAICS, we either stop calling monitor_fdset_cleanup() at monitor close or declare that this issue is unlikely to occur (because it is) and leave a comment in the code.
===========
1- I ran a quick test:
connect() // monitor opened: refcnt: 1
{"execute": "add-fd", "arguments": {"fdset-id": 1}} {"return": {"fd": 9, "fdset-id": 1}}
{"execute": "add-fd", "arguments": {"fdset-id": 1}} {"return": {"fd": 21, "fdset-id": 1}}
close() // monitor closed: refcnt: 0
connect() // monitor opened: refcnt: 1
{"execute": "migrate", "arguments": {"uri": "file:/dev/fdset/1,offset=4096"}}
{
"error": {
"class": "GenericError",
"desc": "Outgoing migration needs two fds in the fdset, got 0"
}
}Quoted 4 lines
I also am confused why ->removed flags is ever needed, and why we can't already remove the fdsets fds if found matching.
Prior to commit efb87c1697 ("monitor: Clean up fd sets on monitor disconnect") we only called monitor_fdset_cleanup() from qmp_remove_fd(), so we effectively always removed immediately after setting ->removed = true. I don't see a reason to have the flag either.
Re: [PATCH 1/9] monitor: Honor QMP request for fd removal immediately
In reply to Fabiano RosasQuoted 14 lines · Fabiano Rosas
We're enabling using the fdset interface to pass file descriptors for use in the migration code. Since migrations can happen more than once during the VMs lifetime, we need a way to remove an fd from the fdset at the end of migration.
The current code only removes an fd from the fdset if the VM is running. This causes a QMP call to "remove-fd" to not actually remove the fd if the VM happens to be stopped.
While the fd would eventually be removed when
monitor_fdset_cleanup()is called again, the user request should be honored and the fd actually removed. Calling remove-fd + query-fdset shows a recently removed fd still present.
Quoted 4 lines
The
runstate_is_running()check was introduced by commitebe52b592d("monitor: Prevent removing fd from set during init"), which by the shortlog indicates that they were trying to avoid removing an yet-unduplicated fd too early.
IMHO that should be reverted. The justification says
"If an fd is added to an fd set via the command line, and it is not
referenced by another command line option (ie. -drive), then clean it up after QEMU initialization is complete"
which I think is pretty weak. Why should QEMU forceably stop an app from passing in an FD to be used by a QMP command issued just after the VM starts running ? While it could just use QMP to pass in the FD set, the mgmt app might have its own reason for wanting QEMU to own the passed FD from the very start of the process execve().
Implicitly this cleanup is attempting to "fix" a bug where the mgmt app passes in an FD that it never needed. If any such bug were ever found, then the mgmt app should just be fixed to not pass it in. I don't think QEMU needs to be trying to fix mgmt app bugs.
IOW, this commit is imposing an arbitrary & unecessary usage policy on passed in FD sets, and as your commit explains has further unhelpful (& undocumented) side effects on the 'remove-fd' QMP command.
Just revert it IMHO.
Quoted 34 lines
I don't see why an fd explicitly removed with
qmp_remove_fd()should be underrunstate_is_running(). I'm assuming this was a mistake when adding the parenthesis around the expression.Move the
runstate_is_running()check to apply only to theQLIST_EMPTY(dup_fds)side of the expression and ignore it whenmon_fdset_fd->removedhas been explicitly set.Signed-off-by Fabiano Rosas <[email protected]>--- monitor/fds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)monitor/fds.c +3
-3@@ -173,9 +173,9 @@ static void monitor_fdset_cleanup(MonFdset *mon_fdset)MonFdsetFd *mon_fdset_fd_next; QLIST_FOREACH_SAFE(mon_fdset_fd, &mon_fdset->fds, next, mon_fdset_fd_next) { -if ((mon_fdset_fd->removed || - (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 )) && -runstate_is_running ()) { +if (mon_fdset_fd->removed || + (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 && +runstate_is_running ())) {close (mon_fdset_fd->fd);g_free (mon_fdset_fd->opaque);QLIST_REMOVE (mon_fdset_fd, next);Signature · 1 line
2.35.3
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 1/9] monitor: Honor QMP request for fd removal immediately
Quoted 26 lines · Daniel P. Berrangé
Quoted 14 lines · Fabiano Rosas
We're enabling using the fdset interface to pass file descriptors for use in the migration code. Since migrations can happen more than once during the VMs lifetime, we need a way to remove an fd from the fdset at the end of migration.
The current code only removes an fd from the fdset if the VM is running. This causes a QMP call to "remove-fd" to not actually remove the fd if the VM happens to be stopped.
While the fd would eventually be removed when
monitor_fdset_cleanup()is called again, the user request should be honored and the fd actually removed. Calling remove-fd + query-fdset shows a recently removed fd still present.Quoted 4 lines
The
runstate_is_running()check was introduced by commitebe52b592d("monitor: Prevent removing fd from set during init"), which by the shortlog indicates that they were trying to avoid removing an yet-unduplicated fd too early.IMHO that should be reverted. The justification says
"If an fd is added to an fd set via the command line, and it is not
referenced by another command line option (ie. -drive), then clean it up after QEMU initialization is complete"
Quoted 5 lines
which I think is pretty weak. Why should QEMU forceably stop an app from passing in an FD to be used by a QMP command issued just after the VM starts running ? While it could just use QMP to pass in the FD set, the mgmt app might have its own reason for wanting QEMU to own the passed FD from the very start of the process
execve().
I don't think that's what that patch does. That description is misleading. I read it as:
"If an fd is added to an fd set via the command line, and it is not
referenced by another command line option (ie. -drive), then clean it up ONLY after QEMU initialization is complete"
^
By the subject ("monitor: Prevent removing fd from set during init") and the fact that this function is only called when the monitor connection closes, I believe the idea was to *save* the fds until after the VM starts running, i.e. some fd was being lost because monitor_fdset_cleanup() was being called before the dup().
See my reply to Peter in this same patch (PATCH 1/9).
Quoted 49 lines
Implicitly this cleanup is attempting to "fix" a bug where the mgmt app passes in an FD that it never needed. If any such bug were ever found, then the mgmt app should just be fixed to not pass it in. I don't think QEMU needs to be trying to fix mgmt app bugs.
IOW, this commit is imposing an arbitrary & unecessary usage policy on passed in FD sets, and as your commit explains has further unhelpful (& undocumented) side effects on the 'remove-fd' QMP command.
Just revert it IMHO.
Quoted 34 lines
I don't see why an fd explicitly removed with
qmp_remove_fd()should be underrunstate_is_running(). I'm assuming this was a mistake when adding the parenthesis around the expression.Move the
runstate_is_running()check to apply only to theQLIST_EMPTY(dup_fds)side of the expression and ignore it whenmon_fdset_fd->removedhas been explicitly set.Signed-off-by Fabiano Rosas <[email protected]>--- monitor/fds.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)monitor/fds.c +3
-3@@ -173,9 +173,9 @@ static void monitor_fdset_cleanup(MonFdset *mon_fdset)MonFdsetFd *mon_fdset_fd_next; QLIST_FOREACH_SAFE(mon_fdset_fd, &mon_fdset->fds, next, mon_fdset_fd_next) { -if ((mon_fdset_fd->removed || - (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 )) && -runstate_is_running ()) { +if (mon_fdset_fd->removed || + (QLIST_EMPTY (&mon_fdset->dup_fds) && mon_refcount ==0 && +runstate_is_running ())) {close (mon_fdset_fd->fd);g_free (mon_fdset_fd->opaque);QLIST_REMOVE (mon_fdset_fd, next);Signature · 1 line
2.35.3With regards,
Daniel
Re: [PATCH 1/9] monitor: Honor QMP request for fd removal immediately
Quoted 42 lines · Fabiano Rosas
Quoted 26 lines · Daniel P. Berrangé
Quoted 14 lines · Fabiano Rosas
We're enabling using the fdset interface to pass file descriptors for use in the migration code. Since migrations can happen more than once during the VMs lifetime, we need a way to remove an fd from the fdset at the end of migration.
The current code only removes an fd from the fdset if the VM is running. This causes a QMP call to "remove-fd" to not actually remove the fd if the VM happens to be stopped.
While the fd would eventually be removed when
monitor_fdset_cleanup()is called again, the user request should be honored and the fd actually removed. Calling remove-fd + query-fdset shows a recently removed fd still present.Quoted 4 lines
The
runstate_is_running()check was introduced by commitebe52b592d("monitor: Prevent removing fd from set during init"), which by the shortlog indicates that they were trying to avoid removing an yet-unduplicated fd too early.IMHO that should be reverted. The justification says
"If an fd is added to an fd set via the command line, and it is not
referenced by another command line option (ie. -drive), then clean it up after QEMU initialization is complete"Quoted 5 lines
which I think is pretty weak. Why should QEMU forceably stop an app from passing in an FD to be used by a QMP command issued just after the VM starts running ? While it could just use QMP to pass in the FD set, the mgmt app might have its own reason for wanting QEMU to own the passed FD from the very start of the process
execve().I don't think that's what that patch does. That description is misleading. I read it as:
"If an fd is added to an fd set via the command line, and it is not
referenced by another command line option (ie. -drive), then clean it up ONLY after QEMU initialization is complete"
^
Quoted 5 lines
By the subject ("monitor: Prevent removing fd from set during init") and the fact that this function is only called when the monitor connection closes, I believe the idea was to *save* the fds until after the VM starts running, i.e. some fd was being lost because
monitor_fdset_cleanup()was being called before thedup().
I know that, but I'm saying QEMU should not be doing *any* generic cleanup of passed in FDs at any point.
A passed in FD should be taken by whatever part of the QEMU configuration is told to use it when needed, and this takes responsibility for closing it. If nothing is told to use the fdset /yet/, then it should stay in the fdset untouched for later use.
If an application accidentally passes in a FD that it doesn't reference in any configuration, that's simply a application bug to fix. QEMU does not need to secondguess the app's intent and decide to arbitrarily close it.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 2/9 migration: Fix file migration with fdset
In reply to Fabiano RosasWhen the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.
The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it.
We can fix the first issue by detecting when we're using the fdset vs. the plain file name. This requires storing the fdset_id somewhere. We can then use this stored fdset_id to do cleanup at the end and also fix the second issue.
--- migration/file.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)
migration/file.c +46 -2
@@ -10,6 +10,7 @@#include "qemu/cutils.h" #include "qemu/error-report.h" #include "qapi/error.h" +#include "qapi/qapi-commands-misc.h" #include "channel.h" #include "file.h" #include "migration.h"
@@ -23,6 +24,7 @@static struct FileOutgoingArgs {char *fname; +int64_t fdset_id; } outgoing_args;/* Remove the offset option from @filespec and return it in @offsetp. */
@@ -44,10 +46,39 @@ int file_parse_offset(char *filespec, uint64_t *offsetp, Error **errp)return 0 ; } +static void file_remove_fdset (void ) +{ +if (outgoing_args.fdset_id !=-1 ) { +qmp_remove_fd (outgoing_args.fdset_id, false,-1 ,NULL ); + outgoing_args.fdset_id =-1 ; + } +} + +static bool file_parse_fdset (const char *filename,int64_t *fdset_id, +Error **errp) +{ +const char *fdset_id_str; + + *fdset_id =-1 ; + +if (!strstart (filename,"/dev/fdset/" , &fdset_id_str)) { +return true; + } + + *fdset_id =qemu_parse_fd (fdset_id_str); +if (*fdset_id ==-1 ) { +error_setg_errno (errp,EINVAL ,"Could not parse fdset %s" , fdset_id_str); +return false; + } + +return true; +} +void file_cleanup_outgoing_migration (void ) {g_free (outgoing_args.fname); outgoing_args.fname =NULL ; +file_remove_fdset (); }bool file_send_channel_create (gpointer opaque,Error **errp)
@@ -81,11 +112,24 @@ void file_start_outgoing_migration(MigrationState *s, g_autofree char *filename = g_strdup(file_args->filename);uint64_t offset = file_args->offset;QIOChannel *ioc; +int flags =O_CREAT |O_WRONLY ;trace_migration_file_outgoing (filename); - fioc =qio_channel_file_new_path (filename,O_CREAT |O_WRONLY |O_TRUNC , -0600 , errp); +if (!file_parse_fdset (filename, &outgoing_args.fdset_id, errp)) { +return ; + } + +/* +* Only truncate if it's QEMU opening the file. If an fd has been +* passed in the file will already contain data written by the +* management layer. +*/ +if (outgoing_args.fdset_id ==-1 ) { + flags |=O_TRUNC ; + } + + fioc =qio_channel_file_new_path (filename, flags,0600 , errp);if (!fioc) {return ; }
Signature · 1 line
2.35.3
Re: [PATCH 2/9] migration: Fix file migration with fdset
Quoted 10 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
I'm wondering whether we can use fallocate() instead on the ranges so that we always don't open() with O_TRUNC. Before that.. could you remind me why do we need to truncate in the first place? I definitely missed something else here too.
Quoted 5 lines
The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it.
Let me check what we do when with "fd:" and when migration completes or cancels.
IIUC it's qio_channel_file_close() that does the final cleanup work on e.g. to_dst_file, right? Then there's qemu_close(), and it has:
/* Close fd that was dup'd from an fdset */
fdset_id = monitor_fdset_dup_fd_find(fd);
if (fdset_id != -1) {
int ret;
ret = close(fd);
if (ret == 0) {
monitor_fdset_dup_fd_remove(fd);
}
return ret;
}Shouldn't this done the work already?
Off topic: I think this code is over complicated too, maybe I missed something, but afaict we don't need monitor_fdset_dup_fd_find at all.. we simply walk the list and remove stuff.. I attach a patch at the end that I tried to clean that up, just in case there's early comments. But we can ignore that so we don't get side-tracked, and focus on the direct-io issues.
Thanks,
=======
From 2f6b6d1224486d8ee830a7afe34738a07003b863 Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Fri, 3 May 2024 11:27:20 -0400
Subject: [PATCH] monitor: Drop monitor_fdset_dup_fd_add()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This function is not needed, one remove function should already work. Clean it up.
Here the code doesn't really care about whether we need to keep that dupfd around if close() failed: when that happens something got very wrong, keeping the dup_fd around the fdsets may not help that situation so far.
--- include/monitor/monitor.h | 1 - monitor/fds.c | 27 +++++---------------------- stubs/fdset.c | 5 ----- util/osdep.c | 15 +-------------- 4 files changed, 6 insertions(+), 42 deletions(-)
include/monitor/monitor.h +0 -1
@@ -53,7 +53,6 @@ AddfdInfo *monitor_fdset_add_fd(int fd, bool has_fdset_id, int64_t fdset_id, const char *opaque, Error **errp);int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags);void monitor_fdset_dup_fd_remove (int dup_fd); -int64_t monitor_fdset_dup_fd_find (int dup_fd);void monitor_register_hmp (const char *name,bool info,void (*cmd)(Monitor *mon,const QDict *qdict));
monitor/fds.c +5 -22
@@ -458,7 +458,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)#endif } -static int64_t monitor_fdset_dup_fd_find_remove (int dup_fd,bool remove) +void monitor_fdset_dup_fd_remove (int dup_fd) {MonFdset *mon_fdset;MonFdsetFd *mon_fdset_fd_dup;
@@ -467,31 +467,14 @@ static int64_t monitor_fdset_dup_fd_find_remove(int dup_fd, bool remove) QLIST_FOREACH(mon_fdset, &mon_fdsets, next) { QLIST_FOREACH(mon_fdset_fd_dup, &mon_fdset->dup_fds, next) {if (mon_fdset_fd_dup->fd == dup_fd) { -if (remove) { -QLIST_REMOVE (mon_fdset_fd_dup, next); -g_free (mon_fdset_fd_dup); -if (QLIST_EMPTY (&mon_fdset->dup_fds)) { -monitor_fdset_cleanup (mon_fdset); - } -return -1 ; - }else { -return mon_fdset->id; +QLIST_REMOVE (mon_fdset_fd_dup, next); +g_free (mon_fdset_fd_dup); +if (QLIST_EMPTY (&mon_fdset->dup_fds)) { +monitor_fdset_cleanup (mon_fdset); } } } } - -return -1 ; -} - -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return monitor_fdset_dup_fd_find_remove (dup_fd, false); -} - -void monitor_fdset_dup_fd_remove (int dup_fd) -{ -monitor_fdset_dup_fd_find_remove (dup_fd, true); }int monitor_fd_param (Monitor *mon,const char *fdname,Error **errp)
stubs/fdset.c +0 -5
@@ -9,11 +9,6 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return -1 ; -} -void monitor_fdset_dup_fd_remove (int dupfd) { }
util/osdep.c +1 -14
@@ -393,21 +393,8 @@ int qemu_open_old(const char *name, int flags, ...)int qemu_close (int fd) { -int64_t fdset_id; -/* Close fd that was dup'd from an fdset */ - fdset_id =monitor_fdset_dup_fd_find (fd); -if (fdset_id !=-1 ) { -int ret; - - ret =close (fd); -if (ret ==0 ) { -monitor_fdset_dup_fd_remove (fd); - } - -return ret; - } - +monitor_fdset_dup_fd_remove (fd);return close (fd); }
Signature · 5 lines
2.44.0 -- Peter Xu
Re: [PATCH 2/9] migration: Fix file migration with fdset
Quoted 8 lines · Peter Xu
Quoted 7 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.
Quoted 8 lines
Quoted 3 lines
The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
I'm wondering whether we can use
fallocate()instead on the ranges so that we always don'topen()withO_TRUNC. Before that.. could you remind me why do we need to truncate in the first place? I definitely missed something else here too.
AFAIK, just to avoid any issues if the file is pre-existing. I don't see the difference between O_TRUNC and fallocate in this case.
Quoted 26 lines
Quoted 6 lines
The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it.
Let me check what we do when with "fd:" and when migration completes or cancels.
IIUC it's
qio_channel_file_close()that does the final cleanup work on e.g.to_dst_file, right? Then there'sqemu_close(), and it has:/* Close fd that was dup'd from an fdset */ fdset_id = monitor_fdset_dup_fd_find(fd); if (fdset_id != -1) { int ret; ret = close(fd); if (ret == 0) { monitor_fdset_dup_fd_remove(fd); } return ret; }
Quoted 1 line
Shouldn't this done the work already?
That removes the mon_fdset_fd_dup->fd, we want to remove the mon_fdset_fd->fd.
Quoted 7 lines
Off topic: I think this code is over complicated too, maybe I missed something, but afaict we don't need
monitor_fdset_dup_fd_findat all.. we simply walk the list and remove stuff.. I attach a patch at the end that I tried to clean that up, just in case there's early comments. But we can ignore that so we don't get side-tracked, and focus on the direct-io issues.
Well, I'm not confident touching this code. This is more than a decade old, I have no idea what the original motivations were. The possible interactions with the user via command-line (-add-fd), QMP (add-fd) and the monitor lifetime make me confused. Not to mention the fdset part being plumbed into the guts of a widely used qemu_open_internal() that very misleadingly presents itself as just a wrapper for open().
Quoted 139 lines
Thanks,
=======
From
2f6b6d1224486d8ee830a7afe34738a07003b863Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Fri, 3 May 2024 11:27:20 -0400
Subject: [PATCH] monitor: Dropmonitor_fdset_dup_fd_add()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitThis function is not needed, one remove function should already work. Clean it up.
Here the code doesn't really care about whether we need to keep that dupfd around if
close()failed: when that happens something got very wrong, keeping thedup_fdaround the fdsets may not help that situation so far.Cc Dr. David Alan Gilbert <[email protected]>Cc Markus Armbruster <[email protected]>Cc Philippe Mathieu-Daudé <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Daniel P. Berrangé <[email protected]>Signed-off-by Peter Xu <[email protected]>--- include/monitor/monitor.h | 1 - monitor/fds.c | 27 +++++---------------------- stubs/fdset.c | 5 ----- util/osdep.c | 15 +-------------- 4 files changed, 6 insertions(+), 42 deletions(-)include/monitor/monitor.h +0
-1@@ -53,7 +53,6 @@ AddfdInfo *monitor_fdset_add_fd(int fd, bool has_fdset_id, int64_t fdset_id, const char *opaque, Error **errp);int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags);void monitor_fdset_dup_fd_remove (int dup_fd); -int64_t monitor_fdset_dup_fd_find (int dup_fd);void monitor_register_hmp (const char *name,bool info,void (*cmd)(Monitor *mon,const QDict *qdict));monitor/fds.c +5
-22@@ -458,7 +458,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)#endif } -static int64_t monitor_fdset_dup_fd_find_remove (int dup_fd,bool remove) +void monitor_fdset_dup_fd_remove (int dup_fd) {MonFdset *mon_fdset;MonFdsetFd *mon_fdset_fd_dup;@@ -467,31 +467,14 @@ static int64_t monitor_fdset_dup_fd_find_remove(int dup_fd, bool remove) QLIST_FOREACH(mon_fdset, &mon_fdsets, next) { QLIST_FOREACH(mon_fdset_fd_dup, &mon_fdset->dup_fds, next) {if (mon_fdset_fd_dup->fd == dup_fd) { -if (remove) { -QLIST_REMOVE (mon_fdset_fd_dup, next); -g_free (mon_fdset_fd_dup); -if (QLIST_EMPTY (&mon_fdset->dup_fds)) { -monitor_fdset_cleanup (mon_fdset); - } -return -1 ; - }else { -return mon_fdset->id; +QLIST_REMOVE (mon_fdset_fd_dup, next); +g_free (mon_fdset_fd_dup); +if (QLIST_EMPTY (&mon_fdset->dup_fds)) { +monitor_fdset_cleanup (mon_fdset); } } } } - -return -1 ; -} - -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return monitor_fdset_dup_fd_find_remove (dup_fd, false); -} - -void monitor_fdset_dup_fd_remove (int dup_fd) -{ -monitor_fdset_dup_fd_find_remove (dup_fd, true); }int monitor_fd_param (Monitor *mon,const char *fdname,Error **errp)stubs/fdset.c +0
-5@@ -9,11 +9,6 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return -1 ; -} -void monitor_fdset_dup_fd_remove (int dupfd) { }util/osdep.c +1
-14@@ -393,21 +393,8 @@ int qemu_open_old(const char *name, int flags, ...)int qemu_close (int fd) { -int64_t fdset_id; -/* Close fd that was dup'd from an fdset */ - fdset_id =monitor_fdset_dup_fd_find (fd); -if (fdset_id !=-1 ) { -int ret; - - ret =close (fd); -if (ret ==0 ) { -monitor_fdset_dup_fd_remove (fd); - } - -return ret; - } - +monitor_fdset_dup_fd_remove (fd);return close (fd); }Signature · 1 line
2.44.0
Re: [PATCH 2/9] migration: Fix file migration with fdset
Quoted 14 lines · Fabiano Rosas
Quoted 12 lines · Peter Xu
Quoted 10 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
Quoted 7 lines
Quoted 4 lines
I'm wondering whether we can use
fallocate()instead on the ranges so that we always don'topen()withO_TRUNC. Before that.. could you remind me why do we need to truncate in the first place? I definitely missed something else here too.AFAIK, just to avoid any issues if the file is pre-existing. I don't see the difference between
O_TRUNCand fallocate in this case.
Then, shall we avoid truncations at all, leaving all the feasibility to user (also errors prone to make)?
Quoted 24 lines
Quoted 24 lines
Quoted 7 lines
The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it.
Let me check what we do when with "fd:" and when migration completes or cancels.
IIUC it's
qio_channel_file_close()that does the final cleanup work on e.g.to_dst_file, right? Then there'sqemu_close(), and it has:/* Close fd that was dup'd from an fdset */ fdset_id = monitor_fdset_dup_fd_find(fd); if (fdset_id != -1) { int ret; ret = close(fd); if (ret == 0) { monitor_fdset_dup_fd_remove(fd); }
Quoted 7 lines
Quoted 4 lines
return ret; }Shouldn't this done the work already?
That removes the
mon_fdset_fd_dup->fd, we want to remove themon_fdset_fd->fd.
What I read so far is when we are removing the dup-fds, we'll do one more thing:
monitor_fdset_dup_fd_find_remove():
if (QLIST_EMPTY(&mon_fdset->dup_fds)) {
monitor_fdset_cleanup(mon_fdset);
}It means if we removed all the dup-fds correctly, we should also remove the whole fdset, which includes the ->fds, IIUC.
Quoted 9 lines
Quoted 8 lines
Off topic: I think this code is over complicated too, maybe I missed something, but afaict we don't need
monitor_fdset_dup_fd_findat all.. we simply walk the list and remove stuff.. I attach a patch at the end that I tried to clean that up, just in case there's early comments. But we can ignore that so we don't get side-tracked, and focus on the direct-io issues.
Quoted 6 lines
Well, I'm not confident touching this code. This is more than a decade old, I have no idea what the original motivations were. The possible interactions with the user via command-line (-add-fd), QMP (add-fd) and the monitor lifetime make me confused. Not to mention the fdset part being plumbed into the guts of a widely used
qemu_open_internal()that very misleadingly presents itself as just a wrapper foropen().
If to make QEMU long live, we'll probably need to touch it at some point.. or at least discuss about it and figure things out. We pay tech debts like this when there's no good comment / docs to refer in this case, then the earlier, perhaps also the better.. to try taking the stab, imho.
Definitely not a request to clean everything up. :) Let's see whether others can chim in with better knowledge of the history.
Quoted 141 lines
Quoted 140 lines
Thanks,
=======
From
2f6b6d1224486d8ee830a7afe34738a07003b863Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Fri, 3 May 2024 11:27:20 -0400
Subject: [PATCH] monitor: Dropmonitor_fdset_dup_fd_add()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitThis function is not needed, one remove function should already work. Clean it up.
Here the code doesn't really care about whether we need to keep that dupfd around if
close()failed: when that happens something got very wrong, keeping thedup_fdaround the fdsets may not help that situation so far.Cc Dr. David Alan Gilbert <[email protected]>Cc Markus Armbruster <[email protected]>Cc Philippe Mathieu-Daudé <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Daniel P. Berrangé <[email protected]>Signed-off-by Peter Xu <[email protected]>--- include/monitor/monitor.h | 1 - monitor/fds.c | 27 +++++---------------------- stubs/fdset.c | 5 ----- util/osdep.c | 15 +-------------- 4 files changed, 6 insertions(+), 42 deletions(-)include/monitor/monitor.h +0
-1@@ -53,7 +53,6 @@ AddfdInfo *monitor_fdset_add_fd(int fd, bool has_fdset_id, int64_t fdset_id, const char *opaque, Error **errp);int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags);void monitor_fdset_dup_fd_remove (int dup_fd); -int64_t monitor_fdset_dup_fd_find (int dup_fd);void monitor_register_hmp (const char *name,bool info,void (*cmd)(Monitor *mon,const QDict *qdict));monitor/fds.c +5
-22@@ -458,7 +458,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)#endif } -static int64_t monitor_fdset_dup_fd_find_remove (int dup_fd,bool remove) +void monitor_fdset_dup_fd_remove (int dup_fd) {MonFdset *mon_fdset;MonFdsetFd *mon_fdset_fd_dup;@@ -467,31 +467,14 @@ static int64_t monitor_fdset_dup_fd_find_remove(int dup_fd, bool remove) QLIST_FOREACH(mon_fdset, &mon_fdsets, next) { QLIST_FOREACH(mon_fdset_fd_dup, &mon_fdset->dup_fds, next) {if (mon_fdset_fd_dup->fd == dup_fd) { -if (remove) { -QLIST_REMOVE (mon_fdset_fd_dup, next); -g_free (mon_fdset_fd_dup); -if (QLIST_EMPTY (&mon_fdset->dup_fds)) { -monitor_fdset_cleanup (mon_fdset); - } -return -1 ; - }else { -return mon_fdset->id; +QLIST_REMOVE (mon_fdset_fd_dup, next); +g_free (mon_fdset_fd_dup); +if (QLIST_EMPTY (&mon_fdset->dup_fds)) { +monitor_fdset_cleanup (mon_fdset); } } } } - -return -1 ; -} - -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return monitor_fdset_dup_fd_find_remove (dup_fd, false); -} - -void monitor_fdset_dup_fd_remove (int dup_fd) -{ -monitor_fdset_dup_fd_find_remove (dup_fd, true); }int monitor_fd_param (Monitor *mon,const char *fdname,Error **errp)stubs/fdset.c +0
-5@@ -9,11 +9,6 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return -1 ; -} -void monitor_fdset_dup_fd_remove (int dupfd) { }util/osdep.c +1
-14@@ -393,21 +393,8 @@ int qemu_open_old(const char *name, int flags, ...)int qemu_close (int fd) { -int64_t fdset_id; -/* Close fd that was dup'd from an fdset */ - fdset_id =monitor_fdset_dup_fd_find (fd); -if (fdset_id !=-1 ) { -int ret; - - ret =close (fd); -if (ret ==0 ) { -monitor_fdset_dup_fd_remove (fd); - } - -return ret; - } - +monitor_fdset_dup_fd_remove (fd);return close (fd); }Signature · 1 line
2.44.0
Signature · 1 line
Peter Xu
Re: [PATCH 2/9] migration: Fix file migration with fdset
Quoted 20 lines · Peter Xu
Quoted 19 lines · Fabiano Rosas
Quoted 16 lines · Peter Xu
Quoted 10 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
I'm wondering whether we can use
fallocate()instead on the ranges so that we always don'topen()withO_TRUNC. Before that.. could you remind me why do we need to truncate in the first place? I definitely missed something else here too.
Quoted 6 lines
Quoted 2 lines
AFAIK, just to avoid any issues if the file is pre-existing. I don't see the difference between
O_TRUNCand fallocate in this case.Then, shall we avoid truncations at all, leaving all the feasibility to user (also errors prone to make)?
Is this a big deal? I'd rather close that possible gap and avoid the bug reports.
Quoted 35 lines
Quoted 24 lines
Quoted 24 lines
Quoted 7 lines
The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it.
Let me check what we do when with "fd:" and when migration completes or cancels.
IIUC it's
qio_channel_file_close()that does the final cleanup work on e.g.to_dst_file, right? Then there'sqemu_close(), and it has:/* Close fd that was dup'd from an fdset */ fdset_id = monitor_fdset_dup_fd_find(fd); if (fdset_id != -1) { int ret; ret = close(fd); if (ret == 0) { monitor_fdset_dup_fd_remove(fd); }Quoted 7 lines
Quoted 4 lines
return ret; }Shouldn't this done the work already?
That removes the
mon_fdset_fd_dup->fd, we want to remove themon_fdset_fd->fd.What I read so far is when we are removing the dup-fds, we'll do one more thing:
Quoted 8 lines
monitor_fdset_dup_fd_find_remove():if (QLIST_EMPTY(&mon_fdset->dup_fds)) { monitor_fdset_cleanup(mon_fdset); }It means if we removed all the dup-fds correctly, we should also remove the whole fdset, which includes the ->fds, IIUC.
Since mon_fdset_fd->removed == false, we hit the runstate_is_running() problem. I'm not sure, but probably mon_refcount > 0 as well. So the fd would not be removed.
But I'll retest this on Monday just be sure, it's been a while since I wrote some parts of this.
Quoted 165 lines
Quoted 9 lines
Quoted 8 lines
Off topic: I think this code is over complicated too, maybe I missed something, but afaict we don't need
monitor_fdset_dup_fd_findat all.. we simply walk the list and remove stuff.. I attach a patch at the end that I tried to clean that up, just in case there's early comments. But we can ignore that so we don't get side-tracked, and focus on the direct-io issues.Quoted 6 lines
Well, I'm not confident touching this code. This is more than a decade old, I have no idea what the original motivations were. The possible interactions with the user via command-line (-add-fd), QMP (add-fd) and the monitor lifetime make me confused. Not to mention the fdset part being plumbed into the guts of a widely used
qemu_open_internal()that very misleadingly presents itself as just a wrapper foropen().If to make QEMU long live, we'll probably need to touch it at some point.. or at least discuss about it and figure things out. We pay tech debts like this when there's no good comment / docs to refer in this case, then the earlier, perhaps also the better.. to try taking the stab, imho.
Definitely not a request to clean everything up. :) Let's see whether others can chim in with better knowledge of the history.
Quoted 141 lines
Quoted 140 lines
Thanks,
=======
From
2f6b6d1224486d8ee830a7afe34738a07003b863Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Fri, 3 May 2024 11:27:20 -0400
Subject: [PATCH] monitor: Dropmonitor_fdset_dup_fd_add()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bitThis function is not needed, one remove function should already work. Clean it up.
Here the code doesn't really care about whether we need to keep that dupfd around if
close()failed: when that happens something got very wrong, keeping thedup_fdaround the fdsets may not help that situation so far.Cc Dr. David Alan Gilbert <[email protected]>Cc Markus Armbruster <[email protected]>Cc Philippe Mathieu-Daudé <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Daniel P. Berrangé <[email protected]>Signed-off-by Peter Xu <[email protected]>--- include/monitor/monitor.h | 1 - monitor/fds.c | 27 +++++---------------------- stubs/fdset.c | 5 ----- util/osdep.c | 15 +-------------- 4 files changed, 6 insertions(+), 42 deletions(-)include/monitor/monitor.h +0
-1@@ -53,7 +53,6 @@ AddfdInfo *monitor_fdset_add_fd(int fd, bool has_fdset_id, int64_t fdset_id, const char *opaque, Error **errp);int monitor_fdset_dup_fd_add (int64_t fdset_id,int flags);void monitor_fdset_dup_fd_remove (int dup_fd); -int64_t monitor_fdset_dup_fd_find (int dup_fd);void monitor_register_hmp (const char *name,bool info,void (*cmd)(Monitor *mon,const QDict *qdict));monitor/fds.c +5
-22@@ -458,7 +458,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)#endif } -static int64_t monitor_fdset_dup_fd_find_remove (int dup_fd,bool remove) +void monitor_fdset_dup_fd_remove (int dup_fd) {MonFdset *mon_fdset;MonFdsetFd *mon_fdset_fd_dup;@@ -467,31 +467,14 @@ static int64_t monitor_fdset_dup_fd_find_remove(int dup_fd, bool remove) QLIST_FOREACH(mon_fdset, &mon_fdsets, next) { QLIST_FOREACH(mon_fdset_fd_dup, &mon_fdset->dup_fds, next) {if (mon_fdset_fd_dup->fd == dup_fd) { -if (remove) { -QLIST_REMOVE (mon_fdset_fd_dup, next); -g_free (mon_fdset_fd_dup); -if (QLIST_EMPTY (&mon_fdset->dup_fds)) { -monitor_fdset_cleanup (mon_fdset); - } -return -1 ; - }else { -return mon_fdset->id; +QLIST_REMOVE (mon_fdset_fd_dup, next); +g_free (mon_fdset_fd_dup); +if (QLIST_EMPTY (&mon_fdset->dup_fds)) { +monitor_fdset_cleanup (mon_fdset); } } } } - -return -1 ; -} - -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return monitor_fdset_dup_fd_find_remove (dup_fd, false); -} - -void monitor_fdset_dup_fd_remove (int dup_fd) -{ -monitor_fdset_dup_fd_find_remove (dup_fd, true); }int monitor_fd_param (Monitor *mon,const char *fdname,Error **errp)stubs/fdset.c +0
-5@@ -9,11 +9,6 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -int64_t monitor_fdset_dup_fd_find (int dup_fd) -{ -return -1 ; -} -void monitor_fdset_dup_fd_remove (int dupfd) { }util/osdep.c +1
-14@@ -393,21 +393,8 @@ int qemu_open_old(const char *name, int flags, ...)int qemu_close (int fd) { -int64_t fdset_id; -/* Close fd that was dup'd from an fdset */ - fdset_id =monitor_fdset_dup_fd_find (fd); -if (fdset_id !=-1 ) { -int ret; - - ret =close (fd); -if (ret ==0 ) { -monitor_fdset_dup_fd_remove (fd); - } - -return ret; - } - +monitor_fdset_dup_fd_remove (fd);return close (fd); }Signature · 1 line
2.44.0
Re: [PATCH 2/9] migration: Fix file migration with fdset
Quoted 25 lines · Fabiano Rosas
Quoted 23 lines · Peter Xu
Quoted 21 lines · Fabiano Rosas
Quoted 8 lines · Peter Xu
Quoted 7 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.Quoted 8 lines
Quoted 3 lines
The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
I'm wondering whether we can use
fallocate()instead on the ranges so that we always don'topen()withO_TRUNC. Before that.. could you remind me why do we need to truncate in the first place? I definitely missed something else here too.AFAIK, just to avoid any issues if the file is pre-existing. I don't see the difference between
O_TRUNCand fallocate in this case.
Quoted 6 lines
Quoted 3 lines
Then, shall we avoid truncations at all, leaving all the feasibility to user (also errors prone to make)?
Is this a big deal? I'd rather close that possible gap and avoid the bug reports.
No possible of such report if the user uses Libvirt or even more virt stacks, am I right? While this is only for whoever uses QEMU directly, and only if the one forgot to remove a leftover image file?
I'd not worry about those people who use QEMU directly - they aren't the people we need to care too much about, imho (and I'm definitely one of them..). The problem is I feel it an overkill introducing a migration global var just for this purpose.
No strong opinions, if you feel strongly like so I'm ok with it. But if one day if we want to remove FileOutgoingArgs I'll also leave that to you as a trade-off. :-)
Quoted 45 lines
Quoted 36 lines
Quoted 25 lines
Quoted 25 lines
Quoted 8 lines
The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it.
Let me check what we do when with "fd:" and when migration completes or cancels.
IIUC it's
qio_channel_file_close()that does the final cleanup work on e.g.to_dst_file, right? Then there'sqemu_close(), and it has:/* Close fd that was dup'd from an fdset */ fdset_id = monitor_fdset_dup_fd_find(fd); if (fdset_id != -1) { int ret; ret = close(fd); if (ret == 0) { monitor_fdset_dup_fd_remove(fd); }Quoted 7 lines
Quoted 4 lines
return ret; }Shouldn't this done the work already?
That removes the
mon_fdset_fd_dup->fd, we want to remove themon_fdset_fd->fd.What I read so far is when we are removing the dup-fds, we'll do one more thing:
Quoted 8 lines
monitor_fdset_dup_fd_find_remove():if (QLIST_EMPTY(&mon_fdset->dup_fds)) { monitor_fdset_cleanup(mon_fdset); }It means if we removed all the dup-fds correctly, we should also remove the whole fdset, which includes the ->fds, IIUC.
Quoted 6 lines
Since
mon_fdset_fd->removed== false, we hit therunstate_is_running()problem. I'm not sure, but probablymon_refcount> 0 as well. So the fd would not be removed.But I'll retest this on Monday just be sure, it's been a while since I wrote some parts of this.
Thanks. And I hope we can also get some more clues too when you dig out more out of the whole add-fd API; I hope we don't pile up more complicated logics on top of a mistery. I feel like this is the time we figure things out.
Signature · 1 line
Peter Xu
Re: [PATCH 2/9] migration: Fix file migration with fdset
In reply to Peter XuQuoted 8 lines · Peter Xu
Quoted 7 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.
Quoted 8 lines
Quoted 3 lines
The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
I'm wondering whether we can use
fallocate()instead on the ranges so that we always don'topen()withO_TRUNC. Before that.. could you remind me why do we need to truncate in the first place? I definitely missed something else here too.
You're mixing distinct concepts here. fallocate makes a file region non-sparse, while O_TRUNC removes all existing allocation, making it sparse if we write at non-contiguous offsets. I don't think we would want to call fallocate, since we /want/ a sparse file so that we don't needlessly store large regions of all-zeros as RAM maps.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 2/9] migration: Fix file migration with fdset
Quoted 18 lines · Daniel P. Berrangé
Quoted 8 lines · Peter Xu
Quoted 7 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.Quoted 8 lines
Quoted 3 lines
The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
I'm wondering whether we can use
fallocate()instead on the ranges so that we always don'topen()withO_TRUNC. Before that.. could you remind me why do we need to truncate in the first place? I definitely missed something else here too.
Quoted 5 lines
You're mixing distinct concepts here. fallocate makes a file region non-sparse, while
O_TRUNCremoves all existing allocation, making it sparse if we write at non-contiguous offsets. I don't think we would want to call fallocate, since we /want/ a sparse file so that we don't needlessly store large regions of all-zeros as RAM maps.
I meant fallocate() with FALLOC_FL_PUNCH_HOLE. But now I think it'll be good we avoid both.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 2/9] migration: Fix file migration with fdset
In reply to Fabiano RosasQuoted 16 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it.
Quoted 4 lines
We can fix the first issue by detecting when we're using the fdset vs. the plain file name. This requires storing the
fdset_idsomewhere. We can then use this storedfdset_idto do cleanup at the end and also fix the second issue.
The use of /dev/fdset is supposed to be transparent to code in QEMU, so modifying migration to learn about FD sets to do manual cleanup is breaking that API facade.
IMHO the transparency of the design points towards the mgmt app calling 'remove-fd' set after migration has started, in order that a later migraiton can use the same fdset name.
Ideally the truncation issue needs to be transparent too.
Rather than detecting use of fdset, we can not use O_TRUNC at all. Instead we can call ftruncate(fd, offset), which should work in both normal and fdset scenarios.
Quoted 97 lines
Fixes 385f510df5 ("migration: file URI offset")Signed-off-by Fabiano Rosas <[email protected]>--- migration/file.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)migration/file.c +46
-2@@ -10,6 +10,7 @@#include "qemu/cutils.h" #include "qemu/error-report.h" #include "qapi/error.h" +#include "qapi/qapi-commands-misc.h" #include "channel.h" #include "file.h" #include "migration.h" @@ -23,6 +24,7 @@static struct FileOutgoingArgs {char *fname; +int64_t fdset_id; } outgoing_args;/* Remove the offset option from @filespec and return it in @offsetp. */ @@ -44,10 +46,39 @@ int file_parse_offset(char *filespec, uint64_t *offsetp, Error **errp)return 0 ; } +static void file_remove_fdset (void ) +{ +if (outgoing_args.fdset_id !=-1 ) { +qmp_remove_fd (outgoing_args.fdset_id, false,-1 ,NULL ); + outgoing_args.fdset_id =-1 ; + } +} + +static bool file_parse_fdset (const char *filename,int64_t *fdset_id, +Error **errp) +{ +const char *fdset_id_str; + + *fdset_id =-1 ; + +if (!strstart (filename,"/dev/fdset/" , &fdset_id_str)) { +return true; + } + + *fdset_id =qemu_parse_fd (fdset_id_str); +if (*fdset_id ==-1 ) { +error_setg_errno (errp,EINVAL ,"Could not parse fdset %s" , fdset_id_str); +return false; + } + +return true; +} +void file_cleanup_outgoing_migration (void ) {g_free (outgoing_args.fname); outgoing_args.fname =NULL ; +file_remove_fdset (); }bool file_send_channel_create (gpointer opaque,Error **errp)@@ -81,11 +112,24 @@ void file_start_outgoing_migration(MigrationState *s, g_autofree char *filename = g_strdup(file_args->filename);uint64_t offset = file_args->offset;QIOChannel *ioc; +int flags =O_CREAT |O_WRONLY ;trace_migration_file_outgoing (filename); - fioc =qio_channel_file_new_path (filename,O_CREAT |O_WRONLY |O_TRUNC , -0600 , errp); +if (!file_parse_fdset (filename, &outgoing_args.fdset_id, errp)) { +return ; + } + +/* +* Only truncate if it's QEMU opening the file. If an fd has been +* passed in the file will already contain data written by the +* management layer. +*/ +if (outgoing_args.fdset_id ==-1 ) { + flags |=O_TRUNC ; + } + + fioc =qio_channel_file_new_path (filename, flags,0600 , errp);if (!fioc) {return ; }Signature · 1 line
2.35.3
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 2/9] migration: Fix file migration with fdset
Quoted 22 lines · Daniel P. Berrangé
Quoted 16 lines · Fabiano Rosas
When the migration using the "file:" URI was implemented, I don't think any of us noticed that if you pass in a file name with the format "
/dev/fdset/N", this allows a file descriptor to be passed in to QEMU and that behaves just like the "fd:" URI. So the "file:" support has been added without regard for the fdset part and we got some things wrong.The first issue is that we should not truncate the migration file if we're allowing an fd + offset. We need to leave the file contents untouched.
The second issue is that there's an expectation that QEMU removes the fd after the migration has finished. That's what the "fd:" code does. Otherwise a second migration on the same VM could attempt to provide an fdset with the same name and QEMU would reject it.
Quoted 4 lines
We can fix the first issue by detecting when we're using the fdset vs. the plain file name. This requires storing the
fdset_idsomewhere. We can then use this storedfdset_idto do cleanup at the end and also fix the second issue.
Quoted 7 lines
The use of
/dev/fdsetis supposed to be transparent to code in QEMU, so modifying migration to learn about FD sets to do manual cleanup is breaking that API facade.IMHO the transparency of the design points towards the mgmt app calling 'remove-fd' set after migration has started, in order that a later migraiton can use the same fdset name.
I got this slightly wrong, QEMU doesn't reject the creation of the fdset, it just reuses the old one and adds the new fd to it. That is somewhat worse because then we'd choose the wrong fd when migrating. But I guess we could just require the management layer to do proper management of the fds/fdset.
Quoted 7 lines
Ideally the truncation issue needs to be transparent too.
Rather than detecting use of fdset, we can not use
O_TRUNCat all. Instead we can callftruncate(fd, offset), which should work in both normal and fdset scenarios.
Good idea.
Quoted 100 lines
Quoted 97 lines
Fixes 385f510df5 ("migration: file URI offset")Signed-off-by Fabiano Rosas <[email protected]>--- migration/file.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)migration/file.c +46
-2@@ -10,6 +10,7 @@#include "qemu/cutils.h" #include "qemu/error-report.h" #include "qapi/error.h" +#include "qapi/qapi-commands-misc.h" #include "channel.h" #include "file.h" #include "migration.h" @@ -23,6 +24,7 @@static struct FileOutgoingArgs {char *fname; +int64_t fdset_id; } outgoing_args;/* Remove the offset option from @filespec and return it in @offsetp. */ @@ -44,10 +46,39 @@ int file_parse_offset(char *filespec, uint64_t *offsetp, Error **errp)return 0 ; } +static void file_remove_fdset (void ) +{ +if (outgoing_args.fdset_id !=-1 ) { +qmp_remove_fd (outgoing_args.fdset_id, false,-1 ,NULL ); + outgoing_args.fdset_id =-1 ; + } +} + +static bool file_parse_fdset (const char *filename,int64_t *fdset_id, +Error **errp) +{ +const char *fdset_id_str; + + *fdset_id =-1 ; + +if (!strstart (filename,"/dev/fdset/" , &fdset_id_str)) { +return true; + } + + *fdset_id =qemu_parse_fd (fdset_id_str); +if (*fdset_id ==-1 ) { +error_setg_errno (errp,EINVAL ,"Could not parse fdset %s" , fdset_id_str); +return false; + } + +return true; +} +void file_cleanup_outgoing_migration (void ) {g_free (outgoing_args.fname); outgoing_args.fname =NULL ; +file_remove_fdset (); }bool file_send_channel_create (gpointer opaque,Error **errp)@@ -81,11 +112,24 @@ void file_start_outgoing_migration(MigrationState *s, g_autofree char *filename = g_strdup(file_args->filename);uint64_t offset = file_args->offset;QIOChannel *ioc; +int flags =O_CREAT |O_WRONLY ;trace_migration_file_outgoing (filename); - fioc =qio_channel_file_new_path (filename,O_CREAT |O_WRONLY |O_TRUNC , -0600 , errp); +if (!file_parse_fdset (filename, &outgoing_args.fdset_id, errp)) { +return ; + } + +/* +* Only truncate if it's QEMU opening the file. If an fd has been +* passed in the file will already contain data written by the +* management layer. +*/ +if (outgoing_args.fdset_id ==-1 ) { + flags |=O_TRUNC ; + } + + fioc =qio_channel_file_new_path (filename, flags,0600 , errp);if (!fioc) {return ; }Signature · 1 line
2.35.3With regards,
Daniel
PATCH 3/9 tests/qtest/migration: Fix file migration offset check
In reply to Fabiano RosasWhen doing file migration, QEMU accepts an offset that should be skipped when writing the migration stream to the file. The purpose of the offset is to allow the management layer to put its own metadata at the start of the file.
We have tests for this in migration-test, but only testing that the migration stream starts at the correct offset and not that it actually leaves the data intact. Unsurprisingly, there's been a bug in that area that the tests didn't catch.
Fix the tests to write some data to the offset region and check that it's actually there after the migration.
--- tests/qtest/migration-test.c | 70 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 5 deletions(-)
tests/qtest/migration-test.c +65 -5
@@ -2081,6 +2081,63 @@ static void test_precopy_file(void)test_file_common (&args, true); } +#ifndef _WIN32 +static void file_dirty_offset_region (void ) +{ +#if defined(__linux__) + g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +size_t size =FILE_TEST_OFFSET ; +uintptr_t *addr, *p; +int fd; + + fd =open (path,O_CREAT |O_RDWR ,0660 ); +g_assert (fd !=-1 ); + +g_assert (!ftruncate (fd, size)); + + addr =mmap (NULL , size,PROT_WRITE ,MAP_SHARED , fd,0 ); +g_assert (addr !=MAP_FAILED ); + +/* ensure the skipped offset contains some data */ + p = addr; +while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { + *p = (unsigned long )FILE_TEST_FILENAME ; + p++; + } + +munmap (addr, size); +fsync (fd); +close (fd); +#endif +} + +static void *file_offset_start_hook (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +int src_flags =O_WRONLY ; +int dst_flags =O_RDONLY ; +int fds[2 ]; + +file_dirty_offset_region (); + + fds[0 ] =open (file, src_flags,0660 ); +assert (fds[0 ] !=-1 ); + + fds[1 ] =open (file, dst_flags,0660 ); +assert (fds[1 ] !=-1 ); + +qtest_qmp_fds_assert_success (from, &fds[0 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +qtest_qmp_fds_assert_success (to, &fds[1 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +close (fds[0 ]); +close (fds[1 ]); + +return NULL ; +} +static void file_offset_finish_hook (QTestState *from,QTestState *to,void *opaque) {
@@ -2096,12 +2153,12 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to,g_assert (addr !=MAP_FAILED );/* -* Ensure the skipped offset contains zeros and the migration -* stream starts at the right place. +* Ensure the skipped offset region's data has not been touched +* and the migration stream starts at the right place. */ p = addr;while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { -g_assert (*p ==0 ); + g_assert_cmpstr((char *) *p, ==, FILE_TEST_FILENAME); p++; } g_assert_cmpint(cpu_to_be64(*p) >> 32, ==, QEMU_VM_FILE_MAGIC);
@@ -2113,17 +2170,18 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to,static void test_precopy_file_offset (void ) { - g_autofree char *uri = g_strdup_printf("file:%s/%s,offset=%d", tmpfs, -FILE_TEST_FILENAME , + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d",FILE_TEST_OFFSET );MigrateCommon args = { .connect_uri = uri, .listen_uri ="defer" , + .start_hook = file_offset_start_hook, .finish_hook = file_offset_finish_hook, };test_file_common (&args, false); } +#endif static void test_precopy_file_offset_bad (void ) {
@@ -3636,8 +3694,10 @@ int main(int argc, char **argv)migration_test_add ("/migration/precopy/file" , test_precopy_file); +#ifndef _WIN32migration_test_add ("/migration/precopy/file/offset" , test_precopy_file_offset); +#endif migration_test_add ("/migration/precopy/file/offset/bad" , test_precopy_file_offset_bad);
Signature · 1 line
2.35.3
Re: [PATCH 3/9] tests/qtest/migration: Fix file migration offset check
Quoted 24 lines · Fabiano Rosas
When doing file migration, QEMU accepts an offset that should be skipped when writing the migration stream to the file. The purpose of the offset is to allow the management layer to put its own metadata at the start of the file.
We have tests for this in migration-test, but only testing that the migration stream starts at the correct offset and not that it actually leaves the data intact. Unsurprisingly, there's been a bug in that area that the tests didn't catch.
Fix the tests to write some data to the offset region and check that it's actually there after the migration.
Fixes 3dc35470c8 ("tests/qtest: migration-test: Add tests for file-based migration")Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 70 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 5 deletions(-)tests/qtest/migration-test.c +0
-0
tests/qtest/migration-test.c static void test_precopy_file(void) · line 2087
Quoted 7 lines
tests/qtest/migration-test.c +4
-0@@ -2081,6 +2081,63 @@ static void test_precopy_file(void)test_file_common (&args, true); } +#ifndef _WIN32 +static void file_dirty_offset_region (void ) +{ +#if defined(__linux__)
Hmm, what's the case to cover when !_WIN32 && __linux__? Can we remove one layer of ifdef?
I'm also wondering why it can't work on win32? I thought win32 has all these stuff we used here, but I may miss something.
Quoted 7 lines
tests/qtest/migration-test.c +7
-0+ g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +size_t size =FILE_TEST_OFFSET ; +uintptr_t *addr, *p; +int fd; + + fd =open (path,O_CREAT |O_RDWR ,0660 ); +g_assert (fd !=-1 );
tests/qtest/migration-test.c static void test_precopy_file(void) · line 2104
Quoted 10 lines
tests/qtest/migration-test.c +10
-0+ +g_assert (!ftruncate (fd, size)); + + addr =mmap (NULL , size,PROT_WRITE ,MAP_SHARED , fd,0 ); +g_assert (addr !=MAP_FAILED ); + +/* ensure the skipped offset contains some data */ + p = addr; +while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { + *p = (unsigned long )FILE_TEST_FILENAME ;
This is fine, but not as clear what is assigned.. I think here we assigned is the pointer pointing to the binary's RO section (rather than the chars). Maybe using some random numbers would be more straightforward, but no strong opinions.
Quoted 56 lines
tests/qtest/migration-test.c +39
-3+ p++; + } + +munmap (addr, size); +fsync (fd); +close (fd); +#endif +} + +static void *file_offset_start_hook (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +int src_flags =O_WRONLY ; +int dst_flags =O_RDONLY ; +int fds[2 ]; + +file_dirty_offset_region (); + + fds[0 ] =open (file, src_flags,0660 ); +assert (fds[0 ] !=-1 ); + + fds[1 ] =open (file, dst_flags,0660 ); +assert (fds[1 ] !=-1 ); + +qtest_qmp_fds_assert_success (from, &fds[0 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +qtest_qmp_fds_assert_success (to, &fds[1 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +close (fds[0 ]); +close (fds[1 ]); + +return NULL ; +} +static void file_offset_finish_hook (QTestState *from,QTestState *to,void *opaque) {@@ -2096,12 +2153,12 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to,g_assert (addr !=MAP_FAILED );/* -* Ensure the skipped offset contains zeros and the migration -* stream starts at the right place. +* Ensure the skipped offset region's data has not been touched +* and the migration stream starts at the right place. */ p = addr;while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { -g_assert (*p ==0 ); + g_assert_cmpstr((char *) *p, ==, FILE_TEST_FILENAME); p++; } g_assert_cmpint(cpu_to_be64(*p) >> 32, ==, QEMU_VM_FILE_MAGIC);
tests/qtest/migration-test.c static void file_offset_finish_hook(QTestState *from, QTestState *to, · line 2174
Quoted 7 lines
tests/qtest/migration-test.c +1
-2@@ -2113,17 +2170,18 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to,static void test_precopy_file_offset (void ) { - g_autofree char *uri = g_strdup_printf("file:%s/%s,offset=%d", tmpfs, -FILE_TEST_FILENAME , + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d",FILE_TEST_OFFSET );
Do we want to keep both tests to cover both normal file and fdsets?
Quoted 27 lines
tests/qtest/migration-test.c +4
-0MigrateCommon args = { .connect_uri = uri, .listen_uri ="defer" , + .start_hook = file_offset_start_hook, .finish_hook = file_offset_finish_hook, }; test_file_common(&args, false); } +#endif static void test_precopy_file_offset_bad (void ) {@@ -3636,8 +3694,10 @@ int main(int argc, char **argv)migration_test_add ("/migration/precopy/file" , test_precopy_file); +#ifndef _WIN32migration_test_add ("/migration/precopy/file/offset" , test_precopy_file_offset); +#endif migration_test_add ("/migration/precopy/file/offset/bad" , test_precopy_file_offset_bad);Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH 3/9] tests/qtest/migration: Fix file migration offset check
Quoted 36 lines · Peter Xu
Quoted 24 lines · Fabiano Rosas
When doing file migration, QEMU accepts an offset that should be skipped when writing the migration stream to the file. The purpose of the offset is to allow the management layer to put its own metadata at the start of the file.
We have tests for this in migration-test, but only testing that the migration stream starts at the correct offset and not that it actually leaves the data intact. Unsurprisingly, there's been a bug in that area that the tests didn't catch.
Fix the tests to write some data to the offset region and check that it's actually there after the migration.
Fixes 3dc35470c8 ("tests/qtest: migration-test: Add tests for file-based migration")Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 70 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 5 deletions(-)tests/qtest/migration-test.c +0
-0tests/qtest/migration-test.c static void test_precopy_file(void) · line 2087
Quoted 7 lines
tests/qtest/migration-test.c +4
-0@@ -2081,6 +2081,63 @@ static void test_precopy_file(void)test_file_common (&args, true); } +#ifndef _WIN32 +static void file_dirty_offset_region (void ) +{ +#if defined(__linux__)Hmm, what's the case to cover when !_WIN32 &&
__linux__? Can we remove one layer of ifdef?
Quoted 3 lines
I'm also wondering why it can't work on win32? I thought win32 has all these stuff we used here, but I may miss something.
__linux__ is because of mmap, !_WIN32 is because of the passing of fds. We might be able to keep !_WIN32 only, I'll check.
Quoted 12 lines
Quoted 12 lines
tests/qtest/migration-test.c +12
-0+ g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +size_t size =FILE_TEST_OFFSET ; +uintptr_t *addr, *p; +int fd; + + fd =open (path,O_CREAT |O_RDWR ,0660 ); +g_assert (fd !=-1 ); + +g_assert (!ftruncate (fd, size)); + + addr =mmap (NULL , size,PROT_WRITE ,MAP_SHARED , fd,0 ); +g_assert (addr !=MAP_FAILED );
Quoted 8 lines
tests/qtest/migration-test.c static void test_precopy_file(void) · line 2104
Quoted 5 lines
tests/qtest/migration-test.c +5
-0+ +/* ensure the skipped offset contains some data */ + p = addr; +while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { + *p = (unsigned long )FILE_TEST_FILENAME ;This is fine, but not as clear what is assigned.. I think here we assigned is the pointer pointing to the binary's RO section (rather than the chars).
Haha you're right, I was assigning the FILE_TEST_OFFSET previously and just switched to the FILENAME without thinking. I'll fix it up.
Quoted 60 lines
Maybe using some random numbers would be more straightforward, but no strong opinions.
Quoted 57 lines
tests/qtest/migration-test.c +39
-3+ p++; + } + +munmap (addr, size); +fsync (fd); +close (fd); +#endif +} + +static void *file_offset_start_hook (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +int src_flags =O_WRONLY ; +int dst_flags =O_RDONLY ; +int fds[2 ]; + +file_dirty_offset_region (); + + fds[0 ] =open (file, src_flags,0660 ); +assert (fds[0 ] !=-1 ); + + fds[1 ] =open (file, dst_flags,0660 ); +assert (fds[1 ] !=-1 ); + +qtest_qmp_fds_assert_success (from, &fds[0 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +qtest_qmp_fds_assert_success (to, &fds[1 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +close (fds[0 ]); +close (fds[1 ]); + +return NULL ; +} +static void file_offset_finish_hook (QTestState *from,QTestState *to,void *opaque) {@@ -2096,12 +2153,12 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to,g_assert (addr !=MAP_FAILED );/* -* Ensure the skipped offset contains zeros and the migration -* stream starts at the right place. +* Ensure the skipped offset region's data has not been touched +* and the migration stream starts at the right place. */ p = addr;while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { -g_assert (*p ==0 ); + g_assert_cmpstr((char *) *p, ==, FILE_TEST_FILENAME); p++; } g_assert_cmpint(cpu_to_be64(*p) >> 32, ==, QEMU_VM_FILE_MAGIC);@@ -2113,17 +2170,18 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to,
Quoted 9 lines
tests/qtest/migration-test.c static void file_offset_finish_hook(QTestState *from, QTestState *to, · line 2174
Quoted 6 lines
tests/qtest/migration-test.c +1
-2static void test_precopy_file_offset (void ) { - g_autofree char *uri = g_strdup_printf("file:%s/%s,offset=%d", tmpfs, -FILE_TEST_FILENAME , + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d",FILE_TEST_OFFSET );Do we want to keep both tests to cover both normal file and fdsets?
I think the fdset + offset is the most complex in terms of requirements, so I don't think we need to test the other one.
I'm actually already a bit concerned about the amount of tests we have. I was even thinking of starting playing with some code coverage tools and prune some of the tests if possible.
Quoted 27 lines
Quoted 27 lines
tests/qtest/migration-test.c +4
-0MigrateCommon args = { .connect_uri = uri, .listen_uri ="defer" , + .start_hook = file_offset_start_hook, .finish_hook = file_offset_finish_hook, }; test_file_common(&args, false); } +#endif static void test_precopy_file_offset_bad (void ) {@@ -3636,8 +3694,10 @@ int main(int argc, char **argv)migration_test_add ("/migration/precopy/file" , test_precopy_file); +#ifndef _WIN32migration_test_add ("/migration/precopy/file/offset" , test_precopy_file_offset); +#endif migration_test_add ("/migration/precopy/file/offset/bad" , test_precopy_file_offset_bad);Signature · 1 line
2.35.3
Re: [PATCH 3/9] tests/qtest/migration: Fix file migration offset check
Quoted 38 lines · Fabiano Rosas
Quoted 36 lines · Peter Xu
Quoted 24 lines · Fabiano Rosas
When doing file migration, QEMU accepts an offset that should be skipped when writing the migration stream to the file. The purpose of the offset is to allow the management layer to put its own metadata at the start of the file.
We have tests for this in migration-test, but only testing that the migration stream starts at the correct offset and not that it actually leaves the data intact. Unsurprisingly, there's been a bug in that area that the tests didn't catch.
Fix the tests to write some data to the offset region and check that it's actually there after the migration.
Fixes 3dc35470c8 ("tests/qtest: migration-test: Add tests for file-based migration")Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 70 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 5 deletions(-)tests/qtest/migration-test.c +0
-0tests/qtest/migration-test.c static void test_precopy_file(void) · line 2087
Quoted 7 lines
tests/qtest/migration-test.c +4
-0@@ -2081,6 +2081,63 @@ static void test_precopy_file(void)test_file_common (&args, true); } +#ifndef _WIN32 +static void file_dirty_offset_region (void ) +{ +#if defined(__linux__)Hmm, what's the case to cover when !_WIN32 &&
__linux__? Can we remove one layer of ifdef?
Quoted 6 lines
Quoted 3 lines
I'm also wondering why it can't work on win32? I thought win32 has all these stuff we used here, but I may miss something.
__linux__is because of mmap, !_WIN32 is because of the passing of fds. We might be able to keep !_WIN32 only, I'll check.
Thanks, or simply use __linux__; we don't lose that much if test less on very special hosts. Just feel a bit over-engineer to use two ifdefs for one such test.
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
tests/qtest/migration-test.c +12
-0+ g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +size_t size =FILE_TEST_OFFSET ; +uintptr_t *addr, *p; +int fd; + + fd =open (path,O_CREAT |O_RDWR ,0660 ); +g_assert (fd !=-1 ); + +g_assert (!ftruncate (fd, size)); + + addr =mmap (NULL , size,PROT_WRITE ,MAP_SHARED , fd,0 ); +g_assert (addr !=MAP_FAILED );Quoted 8 lines
tests/qtest/migration-test.c static void test_precopy_file(void) · line 2104
Quoted 5 lines
tests/qtest/migration-test.c +5
-0+ +/* ensure the skipped offset contains some data */ + p = addr; +while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { + *p = (unsigned long )FILE_TEST_FILENAME ;This is fine, but not as clear what is assigned.. I think here we assigned is the pointer pointing to the binary's RO section (rather than the chars).
Quoted 2 lines
Haha you're right, I was assigning the
FILE_TEST_OFFSETpreviously and just switched to the FILENAME without thinking. I'll fix it up.
:)
Quoted 71 lines
Quoted 61 lines
Maybe using some random numbers would be more straightforward, but no strong opinions.
Quoted 57 lines
tests/qtest/migration-test.c +39
-3+ p++; + } + +munmap (addr, size); +fsync (fd); +close (fd); +#endif +} + +static void *file_offset_start_hook (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +int src_flags =O_WRONLY ; +int dst_flags =O_RDONLY ; +int fds[2 ]; + +file_dirty_offset_region (); + + fds[0 ] =open (file, src_flags,0660 ); +assert (fds[0 ] !=-1 ); + + fds[1 ] =open (file, dst_flags,0660 ); +assert (fds[1 ] !=-1 ); + +qtest_qmp_fds_assert_success (from, &fds[0 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +qtest_qmp_fds_assert_success (to, &fds[1 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +close (fds[0 ]); +close (fds[1 ]); + +return NULL ; +} +static void file_offset_finish_hook (QTestState *from,QTestState *to,void *opaque) {@@ -2096,12 +2153,12 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to,g_assert (addr !=MAP_FAILED );/* -* Ensure the skipped offset contains zeros and the migration -* stream starts at the right place. +* Ensure the skipped offset region's data has not been touched +* and the migration stream starts at the right place. */ p = addr;while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { -g_assert (*p ==0 ); + g_assert_cmpstr((char *) *p, ==, FILE_TEST_FILENAME); p++; } g_assert_cmpint(cpu_to_be64(*p) >> 32, ==, QEMU_VM_FILE_MAGIC);@@ -2113,17 +2170,18 @@ static void file_offset_finish_hook(QTestState *from, QTestState *to,Quoted 9 lines
tests/qtest/migration-test.c static void file_offset_finish_hook(QTestState *from, QTestState *to, · line 2174
Quoted 6 lines
tests/qtest/migration-test.c +1
-2static void test_precopy_file_offset (void ) { - g_autofree char *uri = g_strdup_printf("file:%s/%s,offset=%d", tmpfs, -FILE_TEST_FILENAME , + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d",FILE_TEST_OFFSET );Do we want to keep both tests to cover both normal file and fdsets?
Quoted 2 lines
I think the fdset + offset is the most complex in terms of requirements, so I don't think we need to test the other one.
They will still cover different qemu code paths, right? Even if only slightly different.
Quoted 4 lines
I'm actually already a bit concerned about the amount of tests we have. I was even thinking of starting playing with some code coverage tools and prune some of the tests if possible.
IMHO we don't need to drop any test, but if / when we find it runs too slow, we either:
- try to speed it up - I never tried, but I _feel_ like I can make it faster in some way, just like when Dan used to do with reducing migration-test runtimes, perhaps from different angles, or
- mark more tests optional to run by default, then we use getenv() to select those.
Said that, what you're exploring sounds interesting irrelevant.
Quoted 29 lines
Quoted 28 lines
Quoted 28 lines
tests/qtest/migration-test.c +4
-0MigrateCommon args = { .connect_uri = uri, .listen_uri ="defer" , + .start_hook = file_offset_start_hook, .finish_hook = file_offset_finish_hook, }; test_file_common(&args, false); } +#endif static void test_precopy_file_offset_bad (void ) {@@ -3636,8 +3694,10 @@ int main(int argc, char **argv)migration_test_add ("/migration/precopy/file" , test_precopy_file); +#ifndef _WIN32migration_test_add ("/migration/precopy/file/offset" , test_precopy_file_offset); +#endif migration_test_add ("/migration/precopy/file/offset/bad" , test_precopy_file_offset_bad);Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH 3/9] tests/qtest/migration: Fix file migration offset check
In reply to Fabiano RosasQuoted 72 lines · Fabiano Rosas
Quoted 36 lines · Peter Xu
Quoted 24 lines · Fabiano Rosas
When doing file migration, QEMU accepts an offset that should be skipped when writing the migration stream to the file. The purpose of the offset is to allow the management layer to put its own metadata at the start of the file.
We have tests for this in migration-test, but only testing that the migration stream starts at the correct offset and not that it actually leaves the data intact. Unsurprisingly, there's been a bug in that area that the tests didn't catch.
Fix the tests to write some data to the offset region and check that it's actually there after the migration.
Fixes 3dc35470c8 ("tests/qtest: migration-test: Add tests for file-based migration")Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 70 +++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 5 deletions(-)tests/qtest/migration-test.c +0
-0tests/qtest/migration-test.c static void test_precopy_file(void) · line 2087
Quoted 7 lines
tests/qtest/migration-test.c +4
-0@@ -2081,6 +2081,63 @@ static void test_precopy_file(void)test_file_common (&args, true); } +#ifndef _WIN32 +static void file_dirty_offset_region (void ) +{ +#if defined(__linux__)Hmm, what's the case to cover when !_WIN32 &&
__linux__? Can we remove one layer of ifdef?Quoted 3 lines
I'm also wondering why it can't work on win32? I thought win32 has all these stuff we used here, but I may miss something.
__linux__is because of mmap, !_WIN32 is because of the passing of fds. We might be able to keep !_WIN32 only, I'll check.Quoted 12 lines
Quoted 12 lines
tests/qtest/migration-test.c +12
-0+ g_autofree char *path = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +size_t size =FILE_TEST_OFFSET ; +uintptr_t *addr, *p; +int fd; + + fd =open (path,O_CREAT |O_RDWR ,0660 ); +g_assert (fd !=-1 ); + +g_assert (!ftruncate (fd, size)); + + addr =mmap (NULL , size,PROT_WRITE ,MAP_SHARED , fd,0 ); +g_assert (addr !=MAP_FAILED );Quoted 8 lines
tests/qtest/migration-test.c static void test_precopy_file(void) · line 2104
Quoted 5 lines
tests/qtest/migration-test.c +5
-0+ +/* ensure the skipped offset contains some data */ + p = addr; +while (p < addr +FILE_TEST_OFFSET /sizeof (uintptr_t )) { + *p = (unsigned long )FILE_TEST_FILENAME ;This is fine, but not as clear what is assigned.. I think here we assigned is the pointer pointing to the binary's RO section (rather than the chars).
Haha you're right, I was assigning the
FILE_TEST_OFFSETpreviously and just switched to the FILENAME without thinking. I'll fix it up.Quoted 3 lines
Maybe using some random numbers would be more straightforward, but no strong opinions.
Quoted 8 lines
Quoted 8 lines
tests/qtest/migration-test.c static void test_precopy_file(void) · line 2112
Quoted 8 lines
tests/qtest/migration-test.c +8
-0+ p++; + } + +munmap (addr, size); +fsync (fd); +close (fd); +#endif +}
Use of mmap and this loop looks like overkill to me, when we can do it in a fully portable manner with:
g_autofree char *data = g_new0(char *, offset); memset(data, 0x44, offset); g_file_set_contents(path, data, offset, NULL);
and I checked that g_file_set_contents' impl also takes care of fsync.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 4/9 migration: Add direct-io parameter
In reply to Fabiano RosasAdd the direct-io migration parameter that tells the migration code to use O_DIRECT when opening the migration stream file whenever possible.
This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)
include/qemu/osdep.h +2 -0
@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )
migration/migration-hmp-cmds.c +11 -0
@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) { +monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + } }qapi_free_MigrationParameters (params);
@@ -690,6 +697,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_mode = true;visit_type_MigMode (v, param, &p->mode, &err);break ; +case MIGRATION_PARAMETER_DIRECT_IO : + p->has_direct_io = true; +visit_type_bool (v, param, &p->direct_io, &err); +break ;default :assert (0 ); }
migration/options.c +30 -0
@@ -826,6 +826,22 @@ int migrate_decompress_threads(void)return s->parameters.decompress_threads; } +bool migrate_direct_io (void ) +{ +MigrationState *s =migrate_get_current (); + +/* For now O_DIRECT is only supported with mapped-ram */ +if (!s->capabilities[MIGRATION_CAPABILITY_MAPPED_RAM ]) { +return false; + } + +if (s->parameters.has_direct_io) { +return s->parameters.direct_io; + } + +return false; +} +uint64_t migrate_downtime_limit (void ) {MigrationState *s =migrate_get_current ();
@@ -1061,6 +1077,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_zero_page_detection = true; params->zero_page_detection = s->parameters.zero_page_detection; +if (s->parameters.has_direct_io) { + params->has_direct_io = true; + params->direct_io = s->parameters.direct_io; + } +return params; }
@@ -1097,6 +1118,7 @@ void migrate_params_init(MigrationParameters *params) params->has_vcpu_dirty_limit = true; params->has_mode = true; params->has_zero_page_detection = true; + params->has_direct_io =qemu_has_direct_io (); }/*
@@ -1416,6 +1438,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_zero_page_detection) { dest->zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + dest->direct_io = params->direct_io; + } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)
@@ -1570,6 +1596,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_zero_page_detection) { s->parameters.zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + s->parameters.direct_io = params->direct_io; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)
migration/options.h +1 -0
@@ -76,6 +76,7 @@ uint8_t migrate_cpu_throttle_increment(void);uint8_t migrate_cpu_throttle_initial (void );bool migrate_cpu_throttle_tailslow (void );int migrate_decompress_threads (void ); +bool migrate_direct_io (void );uint64_t migrate_downtime_limit (void );uint8_t migrate_max_cpu_throttle (void );uint64_t migrate_max_bandwidth (void );
qapi/migration.json +15 -3
@@ -914,6 +914,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use
@@ -948,7 +951,8 @@ { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit', 'mode', - 'zero-page-detection'] } + 'zero-page-detection', + 'direct-io'] } ## # @MigrateSetParameters:
@@ -1122,6 +1126,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use
@@ -1176,7 +1183,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @migrate-set-parameters:
@@ -1354,6 +1362,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use
@@ -1405,7 +1416,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @query-migrate-parameters:
util/osdep.c +9 -0
@@ -277,6 +277,15 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive) }#endif +bool qemu_has_direct_io (void ) +{ +#ifdef O_DIRECT +return true; +#else +return false; +#endif +} +static int qemu_open_cloexec (const char *name,int flags,mode_t mode) {int ret;
Signature · 1 line
2.35.3
Re: [PATCH 4/9] migration: Add direct-io parameter
Quoted 8 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>
[...]
qapi/migration.json line 919
Quoted 11 lines
qapi/migration.json +3
-0@@ -914,6 +914,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +#
Two spaces between sentences for consistency, please.
Quoted 13 lines
qapi/migration.json +2
-1# Features: # # @deprecated: Member @block-incremental is deprecated. Use@@ -948,7 +951,8 @@ { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit', 'mode', - 'zero-page-detection'] } + 'zero-page-detection', + 'direct-io'] } ## # @MigrateSetParameters:
[...]
Re: [PATCH 4/9] migration: Add direct-io parameter
In reply to Fabiano RosasQuoted 35 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)include/qemu/osdep.h +2
-0@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )migration/migration-hmp-cmds.c +0
-0
migration/migration-hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 406
Quoted 10 lines
migration/migration-hmp-cmds.c +7
-0@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) { +monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + }
This will be the first parameter to optionally display here. I think it's a sign of misuse of has_direct_io field..
IMHO has_direct_io should be best to be kept as "whether direct_io field is valid" and that's all of it. It hopefully shouldn't contain more information than that, or otherwise it'll be another small challenge we need to overcome when we can remove all these has_* fields, and can also be easily overlooked.
IMHO what we should do is assert has_direct_io==true here too, meanwhile...
Quoted 62 lines
migration/migration-hmp-cmds.c +4
-0}qapi_free_MigrationParameters (params);@@ -690,6 +697,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_mode = true;visit_type_MigMode (v, param, &p->mode, &err);break ; +case MIGRATION_PARAMETER_DIRECT_IO : + p->has_direct_io = true; +visit_type_bool (v, param, &p->direct_io, &err); +break ;default :assert (0 ); }migration/options.c +22
-0@@ -826,6 +826,22 @@ int migrate_decompress_threads(void)return s->parameters.decompress_threads; } +bool migrate_direct_io (void ) +{ +MigrationState *s =migrate_get_current (); + +/* For now O_DIRECT is only supported with mapped-ram */ +if (!s->capabilities[MIGRATION_CAPABILITY_MAPPED_RAM ]) { +return false; + } + +if (s->parameters.has_direct_io) { +return s->parameters.direct_io; + } + +return false; +} +uint64_t migrate_downtime_limit (void ) {MigrationState *s =migrate_get_current ();@@ -1061,6 +1077,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_zero_page_detection = true; params->zero_page_detection = s->parameters.zero_page_detection; +if (s->parameters.has_direct_io) { + params->has_direct_io = true; + params->direct_io = s->parameters.direct_io; + } +return params; }@@ -1097,6 +1118,7 @@ void migrate_params_init(MigrationParameters *params) params->has_vcpu_dirty_limit = true; params->has_mode = true; params->has_zero_page_detection = true; + params->has_direct_io =qemu_has_direct_io (); }/*
migration/options.c static void migrate_params_test_apply(MigrateSetParameters *params, · line 1443
Quoted 6 lines
migration/options.c +3
-0@@ -1416,6 +1438,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_zero_page_detection) { dest->zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + dest->direct_io = params->direct_io;
.. do proper check here to make sure the current QEMU is built with direct IO support, then fail QMP migrate-set-parameters otherwise when someone tries to enable it on a QEMU that doesn't support it.
Always displaying direct_io parameter also helps when we simply want to check qemu version and whether it supports this feature in general.
Quoted 32 lines
migration/options.c +5
-0+ } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)@@ -1570,6 +1596,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_zero_page_detection) { s->parameters.zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + s->parameters.direct_io = params->direct_io; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)migration/options.h +1
-0@@ -76,6 +76,7 @@ uint8_t migrate_cpu_throttle_increment(void);uint8_t migrate_cpu_throttle_initial (void );bool migrate_cpu_throttle_tailslow (void );int migrate_decompress_threads (void ); +bool migrate_direct_io (void );uint64_t migrate_downtime_limit (void );uint8_t migrate_max_cpu_throttle (void );uint64_t migrate_max_bandwidth (void );qapi/migration.json +0
-0
qapi/migration.json line 918
Quoted 5 lines
qapi/migration.json +2
-0@@ -914,6 +914,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1)
Here it seems to imply setting direct-io=true will fail if mapped-ram not enabled, but in reality it's fine, it'll just be ignored. I think that's the right thing to do to reduce correlation effects between params/caps (otherwise, when unset mapped-ram cap, we'll need to double check again to unset direct-io too; just cumbersome).
I suggest we state the fact, that this field is ignored when mapped-ram capability is not enabled, rather than "requires mapped-ram". Same to all the rest two places in qapi doc.
Quoted 77 lines
qapi/migration.json +13
-3+# # Features: # # @deprecated: Member @block-incremental is deprecated. Use@@ -948,7 +951,8 @@ { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit', 'mode', - 'zero-page-detection'] } + 'zero-page-detection', + 'direct-io'] } ## # @MigrateSetParameters:@@ -1122,6 +1126,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use@@ -1176,7 +1183,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @migrate-set-parameters:@@ -1354,6 +1362,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use@@ -1405,7 +1416,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @query-migrate-parameters:util/osdep.c +9
-0@@ -277,6 +277,15 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive) }#endif +bool qemu_has_direct_io (void ) +{ +#ifdef O_DIRECT +return true; +#else +return false; +#endif +} +static int qemu_open_cloexec (const char *name,int flags,mode_t mode) {int ret;Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH 4/9] migration: Add direct-io parameter
Quoted 50 lines · Peter Xu
Quoted 35 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)include/qemu/osdep.h +2
-0@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )migration/migration-hmp-cmds.c +0
-0migration/migration-hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 406
Quoted 10 lines
migration/migration-hmp-cmds.c +7
-0@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) { +monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + }This will be the first parameter to optionally display here. I think it's a sign of misuse of
has_direct_iofield..
Quoted 5 lines
IMHO
has_direct_ioshould be best to be kept as "whetherdirect_iofield is valid" and that's all of it. It hopefully shouldn't contain more information than that, or otherwise it'll be another small challenge we need to overcome when we can remove all these has_* fields, and can also be easily overlooked.
I don't think I understand why we have those has_* fields. I thought my usage of 'params->has_direct_io = qemu_has_direct_io()' was the correct one, i.e. checking whether QEMU has any support for that parameter. Can you help me out here?
Quoted 63 lines
IMHO what we should do is assert has_direct_io==true here too, meanwhile...
Quoted 60 lines
migration/migration-hmp-cmds.c +4
-0}qapi_free_MigrationParameters (params);@@ -690,6 +697,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_mode = true;visit_type_MigMode (v, param, &p->mode, &err);break ; +case MIGRATION_PARAMETER_DIRECT_IO : + p->has_direct_io = true; +visit_type_bool (v, param, &p->direct_io, &err); +break ;default :assert (0 ); }migration/options.c +22
-0@@ -826,6 +826,22 @@ int migrate_decompress_threads(void)return s->parameters.decompress_threads; } +bool migrate_direct_io (void ) +{ +MigrationState *s =migrate_get_current (); + +/* For now O_DIRECT is only supported with mapped-ram */ +if (!s->capabilities[MIGRATION_CAPABILITY_MAPPED_RAM ]) { +return false; + } + +if (s->parameters.has_direct_io) { +return s->parameters.direct_io; + } + +return false; +} +uint64_t migrate_downtime_limit (void ) {MigrationState *s =migrate_get_current ();@@ -1061,6 +1077,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_zero_page_detection = true; params->zero_page_detection = s->parameters.zero_page_detection; +if (s->parameters.has_direct_io) { + params->has_direct_io = true; + params->direct_io = s->parameters.direct_io; + } +return params; }@@ -1097,6 +1118,7 @@ void migrate_params_init(MigrationParameters *params) params->has_vcpu_dirty_limit = true; params->has_mode = true; params->has_zero_page_detection = true; + params->has_direct_io =qemu_has_direct_io (); }
Quoted 12 lines
migration/options.c static void migrate_params_test_apply(MigrateSetParameters *params, · line 1443
Quoted 8 lines
migration/options.c +3
-0/* @@ -1416,6 +1438,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_zero_page_detection) { dest->zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + dest->direct_io = params->direct_io;.. do proper check here to make sure the current QEMU is built with direct IO support, then fail QMP migrate-set-parameters otherwise when someone tries to enable it on a QEMU that doesn't support it.
I'm already checking at migrate_params_init() with qemu_has_direct_io(). But ok, you want to move it here... Is this function the correct one instead of migrate_params_check()? I see these TODO comments mentioning QAPI_CLONE(), we can't clone the object if this one parameter needs special treatment. I might be getting all this wrong, bear with me.
Quoted 4 lines
Always displaying
direct_ioparameter also helps when we simply want to check qemu version and whether it supports this feature in general.
Makes sense.
Quoted 44 lines
Quoted 32 lines
migration/options.c +5
-0+ } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)@@ -1570,6 +1596,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_zero_page_detection) { s->parameters.zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + s->parameters.direct_io = params->direct_io; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)migration/options.h +1
-0@@ -76,6 +76,7 @@ uint8_t migrate_cpu_throttle_increment(void);uint8_t migrate_cpu_throttle_initial (void );bool migrate_cpu_throttle_tailslow (void );int migrate_decompress_threads (void ); +bool migrate_direct_io (void );uint64_t migrate_downtime_limit (void );uint8_t migrate_max_cpu_throttle (void );uint64_t migrate_max_bandwidth (void );qapi/migration.json +0
-0qapi/migration.json line 918
Quoted 5 lines
qapi/migration.json +2
-0@@ -914,6 +914,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1)Here it seems to imply setting direct-io=true will fail if mapped-ram not enabled, but in reality it's fine, it'll just be ignored. I think that's the right thing to do to reduce correlation effects between params/caps (otherwise, when unset mapped-ram cap, we'll need to double check again to unset direct-io too; just cumbersome).
Quoted 4 lines
I suggest we state the fact, that this field is ignored when mapped-ram capability is not enabled, rather than "requires mapped-ram". Same to all the rest two places in qapi doc.
Ok.
Quoted 77 lines
Quoted 77 lines
qapi/migration.json +13
-3+# # Features: # # @deprecated: Member @block-incremental is deprecated. Use@@ -948,7 +951,8 @@ { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit', 'mode', - 'zero-page-detection'] } + 'zero-page-detection', + 'direct-io'] } ## # @MigrateSetParameters:@@ -1122,6 +1126,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use@@ -1176,7 +1183,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @migrate-set-parameters:@@ -1354,6 +1362,9 @@ # See description in @ZeroPageDetection. Default is 'multifd'. # (since 9.0) # +# @direct-io: Open migration files with O_DIRECT when possible. This +# requires that the @mapped-ram capability is enabled. (since 9.1) +# # Features: # # @deprecated: Member @block-incremental is deprecated. Use@@ -1405,7 +1416,8 @@ 'features': [ 'unstable' ] }, '*vcpu-dirty-limit': 'uint64', '*mode': 'MigMode', - '*zero-page-detection': 'ZeroPageDetection'} } + '*zero-page-detection': 'ZeroPageDetection', + '*direct-io': 'bool' } } ## # @query-migrate-parameters:util/osdep.c +9
-0@@ -277,6 +277,15 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive) }#endif +bool qemu_has_direct_io (void ) +{ +#ifdef O_DIRECT +return true; +#else +return false; +#endif +} +static int qemu_open_cloexec (const char *name,int flags,mode_t mode) {int ret;Signature · 1 line
2.35.3
Re: [PATCH 4/9] migration: Add direct-io parameter
Quoted 58 lines · Fabiano Rosas
Quoted 50 lines · Peter Xu
Quoted 35 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)include/qemu/osdep.h +2
-0@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )migration/migration-hmp-cmds.c +0
-0migration/migration-hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 406
Quoted 10 lines
migration/migration-hmp-cmds.c +7
-0@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) { +monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + }This will be the first parameter to optionally display here. I think it's a sign of misuse of
has_direct_iofield..Quoted 5 lines
IMHO
has_direct_ioshould be best to be kept as "whetherdirect_iofield is valid" and that's all of it. It hopefully shouldn't contain more information than that, or otherwise it'll be another small challenge we need to overcome when we can remove all these has_* fields, and can also be easily overlooked.
Quoted 4 lines
I don't think I understand why we have those has_* fields. I thought my usage of '
params->has_direct_io=qemu_has_direct_io()' was the correct one, i.e. checking whether QEMU has any support for that parameter. Can you help me out here?
Here params is the pointer to "struct MigrationParameters", which is defined in qapi/migration.json. And we have had "has_*" only because we allow optional fields with asterisks:
{ 'struct': 'MigrationParameters',
'data': { '*announce-initial': 'size',
...
} }So that's why it better only means "whether this field existed", because it's how it is defined.
IIRC we (or say, Markus) used to have some attempts deduplicates those *MigrationParameter* things, and if success we have chance to drop has_* fields (in which case we simply always have them; that "has_" makes more sense only if in a QMP session to allow user only specify one or more things if not all).
Quoted 77 lines
Quoted 64 lines
IMHO what we should do is assert has_direct_io==true here too, meanwhile...
Quoted 60 lines
migration/migration-hmp-cmds.c +4
-0}qapi_free_MigrationParameters (params);@@ -690,6 +697,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_mode = true;visit_type_MigMode (v, param, &p->mode, &err);break ; +case MIGRATION_PARAMETER_DIRECT_IO : + p->has_direct_io = true; +visit_type_bool (v, param, &p->direct_io, &err); +break ;default :assert (0 ); }migration/options.c +22
-0@@ -826,6 +826,22 @@ int migrate_decompress_threads(void)return s->parameters.decompress_threads; } +bool migrate_direct_io (void ) +{ +MigrationState *s =migrate_get_current (); + +/* For now O_DIRECT is only supported with mapped-ram */ +if (!s->capabilities[MIGRATION_CAPABILITY_MAPPED_RAM ]) { +return false; + } + +if (s->parameters.has_direct_io) { +return s->parameters.direct_io; + } + +return false; +} +uint64_t migrate_downtime_limit (void ) {MigrationState *s =migrate_get_current ();@@ -1061,6 +1077,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_zero_page_detection = true; params->zero_page_detection = s->parameters.zero_page_detection; +if (s->parameters.has_direct_io) { + params->has_direct_io = true; + params->direct_io = s->parameters.direct_io; + } +return params; }@@ -1097,6 +1118,7 @@ void migrate_params_init(MigrationParameters *params) params->has_vcpu_dirty_limit = true; params->has_mode = true; params->has_zero_page_detection = true; + params->has_direct_io =qemu_has_direct_io (); }Quoted 12 lines
migration/options.c static void migrate_params_test_apply(MigrateSetParameters *params, · line 1443
Quoted 8 lines
migration/options.c +3
-0/* @@ -1416,6 +1438,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_zero_page_detection) { dest->zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + dest->direct_io = params->direct_io;.. do proper check here to make sure the current QEMU is built with direct IO support, then fail QMP migrate-set-parameters otherwise when someone tries to enable it on a QEMU that doesn't support it.
Quoted 3 lines
I'm already checking at
migrate_params_init()withqemu_has_direct_io(). But ok, you want to move it here... Is this function the correct one instead ofmigrate_params_check()? I see these
Oh I perhaps commented on the wrong line. migrate_params_check() is the place where we should throw such error and check for O_DIRECT for sure..
Quoted 3 lines
TODO comments mentioning
QAPI_CLONE(), we can't clone the object if this one parameter needs special treatment. I might be getting all this wrong, bear with me.
Nah, I think I just wanted to comment inside migrate_params_check() but I did it all wrong, sorry.
Signature · 1 line
Peter Xu
Re: [PATCH 4/9] migration: Add direct-io parameter
Quoted 44 lines · Peter Xu
Quoted 43 lines · Fabiano Rosas
Quoted 41 lines · Peter Xu
Quoted 40 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)include/qemu/osdep.h +2
-0@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )migration/migration-hmp-cmds.c +2
-0@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) {
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
migration/migration-hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 406
Quoted 5 lines
migration/migration-hmp-cmds.c +5
-0+monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + }This will be the first parameter to optionally display here. I think it's a sign of misuse of
has_direct_iofield..
Yes. For similar existing parameters, we do
assert(params->has_FOO);
monitor_printf(mon, "%s: '%s'\n",
MigrationParameter_str(MIGRATION_PARAMETER_FOO),
... params->FOO as string ...)Quoted 20 lines
Quoted 10 lines
Quoted 5 lines
IMHO
has_direct_ioshould be best to be kept as "whetherdirect_iofield is valid" and that's all of it. It hopefully shouldn't contain more information than that, or otherwise it'll be another small challenge we need to overcome when we can remove all these has_* fields, and can also be easily overlooked.I don't think I understand why we have those has_* fields. I thought my usage of '
params->has_direct_io=qemu_has_direct_io()' was the correct one, i.e. checking whether QEMU has any support for that parameter. Can you help me out here?Here params is the pointer to "struct MigrationParameters", which is defined in
qapi/migration.json. And we have had "has_*" only because we allow optional fields with asterisks:{ 'struct': 'MigrationParameters',
'data': { '*announce-initial': 'size', ... } }
Quoted 8 lines
So that's why it better only means "whether this field existed", because it's how it is defined.
IIRC we (or say, Markus) used to have some attempts deduplicates those *MigrationParameter* things, and if success we have chance to drop has_* fields (in which case we simply always have them; that "
has_" makes more sense only if in a QMP session to allow user only specify one or more things if not all).
qapi/migration.json:
##
# @MigrationParameters:
#
--> # The optional members aren't actually optional.
#In other words, the has_FOO generated for the members of MigrationParameters must all be true.
These members became optional when we attempted to de-duplicate MigrationParameters and MigrateSetParameters, but failed (see commit de63ab61241 "migrate: Share common MigrationParameters struct" and commit 1bda8b3c695 "migration: Unshare MigrationParameters struct for now").
Re: [PATCH 4/9] migration: Add direct-io parameter
Quoted 99 lines · Markus Armbruster
Quoted 44 lines · Peter Xu
Quoted 43 lines · Fabiano Rosas
Quoted 41 lines · Peter Xu
Quoted 40 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)include/qemu/osdep.h +2
-0@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )migration/migration-hmp-cmds.c +2
-0@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) {Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
migration/migration-hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 406
Quoted 5 lines
migration/migration-hmp-cmds.c +5
-0+monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + }This will be the first parameter to optionally display here. I think it's a sign of misuse of
has_direct_iofield..Yes. For similar existing parameters, we do
assert(params->has_FOO); monitor_printf(mon, "%s: '%s'\n", MigrationParameter_str(MIGRATION_PARAMETER_FOO), ... params->FOO as string ...)Quoted 20 lines
Quoted 10 lines
Quoted 5 lines
IMHO
has_direct_ioshould be best to be kept as "whetherdirect_iofield is valid" and that's all of it. It hopefully shouldn't contain more information than that, or otherwise it'll be another small challenge we need to overcome when we can remove all these has_* fields, and can also be easily overlooked.I don't think I understand why we have those has_* fields. I thought my usage of '
params->has_direct_io=qemu_has_direct_io()' was the correct one, i.e. checking whether QEMU has any support for that parameter. Can you help me out here?Here params is the pointer to "struct MigrationParameters", which is defined in
qapi/migration.json. And we have had "has_*" only because we allow optional fields with asterisks:{ 'struct': 'MigrationParameters',
'data': { '*announce-initial': 'size', ... } }Quoted 8 lines
So that's why it better only means "whether this field existed", because it's how it is defined.
IIRC we (or say, Markus) used to have some attempts deduplicates those *MigrationParameter* things, and if success we have chance to drop has_* fields (in which case we simply always have them; that "
has_" makes more sense only if in a QMP session to allow user only specify one or more things if not all).
qapi/migration.json:## # @MigrationParameters: # --> # The optional members aren't actually optional. #
Quoted 8 lines
In other words, the
has_FOOgenerated for the members of MigrationParameters must all be true.These members became optional when we attempted to de-duplicate MigrationParameters and MigrateSetParameters, but failed (see commit
de63ab61241"migrate: Share common MigrationParameters struct" and commit1bda8b3c695"migration: Unshare MigrationParameters struct for now").
So what's the blocker for merging MigrationSetParameters and MigrationParameters? Just the tls_* options?
Re: [PATCH 4/9] migration: Add direct-io parameter
Quoted 104 lines · Fabiano Rosas
Quoted 102 lines · Markus Armbruster
Quoted 44 lines · Peter Xu
Quoted 43 lines · Fabiano Rosas
Quoted 41 lines · Peter Xu
Quoted 40 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)include/qemu/osdep.h +2
-0@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )migration/migration-hmp-cmds.c +2
-0@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) {Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
migration/migration-hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 406
Quoted 5 lines
migration/migration-hmp-cmds.c +5
-0+monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + }This will be the first parameter to optionally display here. I think it's a sign of misuse of
has_direct_iofield..Yes. For similar existing parameters, we do
assert(params->has_FOO); monitor_printf(mon, "%s: '%s'\n", MigrationParameter_str(MIGRATION_PARAMETER_FOO), ... params->FOO as string ...)Quoted 20 lines
Quoted 10 lines
Quoted 5 lines
IMHO
has_direct_ioshould be best to be kept as "whetherdirect_iofield is valid" and that's all of it. It hopefully shouldn't contain more information than that, or otherwise it'll be another small challenge we need to overcome when we can remove all these has_* fields, and can also be easily overlooked.I don't think I understand why we have those has_* fields. I thought my usage of '
params->has_direct_io=qemu_has_direct_io()' was the correct one, i.e. checking whether QEMU has any support for that parameter. Can you help me out here?Here params is the pointer to "struct MigrationParameters", which is defined in
qapi/migration.json. And we have had "has_*" only because we allow optional fields with asterisks:{ 'struct': 'MigrationParameters',
'data': { '*announce-initial': 'size', ... } }Quoted 8 lines
So that's why it better only means "whether this field existed", because it's how it is defined.
IIRC we (or say, Markus) used to have some attempts deduplicates those *MigrationParameter* things, and if success we have chance to drop has_* fields (in which case we simply always have them; that "
has_" makes more sense only if in a QMP session to allow user only specify one or more things if not all).
qapi/migration.json:## # @MigrationParameters: # --> # The optional members aren't actually optional. #In other words, the
has_FOOgenerated for the members of MigrationParameters must all be true.
Quoted 8 lines
Quoted 5 lines
These members became optional when we attempted to de-duplicate MigrationParameters and MigrateSetParameters, but failed (see commit
de63ab61241"migrate: Share common MigrationParameters struct" and commit1bda8b3c695"migration: Unshare MigrationParameters struct for now").So what's the blocker for merging MigrationSetParameters and MigrationParameters? Just the tls_* options?
I believe the latest attempt was Peter's "[PATCH v3 0/4] qapi/migration:
Dedup migration parameter objects and fix tls-authz crash" last September.
I can't recall offhand what exactly blocked its merge. Suggest you read the review thread[*], and if that leads to questions, ask them either in replies to the old thread, or right here.
One additional issue hasn't been discussed much so far, I think: merging the two copies of the doc comments. They are big and quite similar (that's why we want to deduplicate), but they're not identical. In particular, MigrationSetParameters' doc comment talks more about defaults. That's because talking about defaults makes no sense whatsoever for MigrationParameters (we do it anyway, of course, just less). Merging the two will require some careful word-smithing, and perhaps some compromises on doc quality.
Re: [PATCH 4/9] migration: Add direct-io parameter
Quoted 112 lines · Markus Armbruster
Quoted 110 lines · Fabiano Rosas
Quoted 107 lines · Markus Armbruster
Quoted 44 lines · Peter Xu
Quoted 43 lines · Fabiano Rosas
Quoted 41 lines · Peter Xu
Quoted 40 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)include/qemu/osdep.h +2
-0@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )migration/migration-hmp-cmds.c +2
-0@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) {Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
migration/migration-hmp-cmds.c void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) · line 406
Quoted 5 lines
migration/migration-hmp-cmds.c +5
-0+monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + }This will be the first parameter to optionally display here. I think it's a sign of misuse of
has_direct_iofield..Yes. For similar existing parameters, we do
assert(params->has_FOO); monitor_printf(mon, "%s: '%s'\n", MigrationParameter_str(MIGRATION_PARAMETER_FOO), ... params->FOO as string ...)Quoted 20 lines
Quoted 10 lines
Quoted 5 lines
IMHO
has_direct_ioshould be best to be kept as "whetherdirect_iofield is valid" and that's all of it. It hopefully shouldn't contain more information than that, or otherwise it'll be another small challenge we need to overcome when we can remove all these has_* fields, and can also be easily overlooked.I don't think I understand why we have those has_* fields. I thought my usage of '
params->has_direct_io=qemu_has_direct_io()' was the correct one, i.e. checking whether QEMU has any support for that parameter. Can you help me out here?Here params is the pointer to "struct MigrationParameters", which is defined in
qapi/migration.json. And we have had "has_*" only because we allow optional fields with asterisks:{ 'struct': 'MigrationParameters',
'data': { '*announce-initial': 'size', ... } }Quoted 8 lines
So that's why it better only means "whether this field existed", because it's how it is defined.
IIRC we (or say, Markus) used to have some attempts deduplicates those *MigrationParameter* things, and if success we have chance to drop has_* fields (in which case we simply always have them; that "
has_" makes more sense only if in a QMP session to allow user only specify one or more things if not all).
qapi/migration.json:## # @MigrationParameters: # --> # The optional members aren't actually optional. #In other words, the
has_FOOgenerated for the members of MigrationParameters must all be true.These members became optional when we attempted to de-duplicate MigrationParameters and MigrateSetParameters, but failed (see commit
de63ab61241"migrate: Share common MigrationParameters struct" and commit1bda8b3c695"migration: Unshare MigrationParameters struct for now").
Quoted 6 lines
Quoted 2 lines
So what's the blocker for merging MigrationSetParameters and MigrationParameters? Just the tls_* options?
I believe the latest attempt was Peter's "[PATCH v3 0/4] qapi/migration:
Dedup migration parameter objects and fix tls-authz crash" last September.
I had a vague memory of this, thanks for bringing it up. I'll go over that series and see what can be done.
Quoted 5 lines
I can't recall offhand what exactly blocked its merge. Suggest you read the review thread[*], and if that leads to questions, ask them either in replies to the old thread, or right here.
Quoted 9 lines
One additional issue hasn't been discussed much so far, I think: merging the two copies of the doc comments. They are big and quite similar (that's why we want to deduplicate), but they're not identical. In particular, MigrationSetParameters' doc comment talks more about defaults. That's because talking about defaults makes no sense whatsoever for MigrationParameters (we do it anyway, of course, just less). Merging the two will require some careful word-smithing, and perhaps some compromises on doc quality.
I was playing with this yesterday and it occurred to me as well that the docs are not quite the same. Maybe we can have a general description common for both use-cases and a few extra words for what happens differently when writing vs. reading.
Quoted 3 lines
Re: [PATCH 4/9] migration: Add direct-io parameter
In reply to Fabiano RosasQuoted 118 lines · Fabiano Rosas
Add the direct-io migration parameter that tells the migration code to use
O_DIRECTwhen opening the migration stream file whenever possible.This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes.
Signed-off-by Fabiano Rosas <[email protected]>--- include/qemu/osdep.h | 2 ++ migration/migration-hmp-cmds.c | 11 +++++++++++ migration/options.c | 30 ++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 +++++++++++++++--- util/osdep.c | 9 +++++++++ 6 files changed, 68 insertions(+), 3 deletions(-)include/qemu/osdep.h +2
-0@@ -612,6 +612,8 @@ int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);bool qemu_has_ofd_lock (void );#endif +bool qemu_has_direct_io (void ); +#if defined(__HAIKU__) && defined(__i386__)#define FMT_pid "%ld"#elif defined(WIN64 )migration/migration-hmp-cmds.c +11
-0@@ -397,6 +397,13 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)monitor_printf (mon,"%s: %s\n" ,MigrationParameter_str (MIGRATION_PARAMETER_MODE ),qapi_enum_lookup (&MigMode_lookup, params->mode)); + +if (params->has_direct_io) { +monitor_printf (mon,"%s: %s\n" , +MigrationParameter_str ( +MIGRATION_PARAMETER_DIRECT_IO ), + params->direct_io ?"on" :"off" ); + } }qapi_free_MigrationParameters (params);@@ -690,6 +697,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_mode = true;visit_type_MigMode (v, param, &p->mode, &err);break ; +case MIGRATION_PARAMETER_DIRECT_IO : + p->has_direct_io = true; +visit_type_bool (v, param, &p->direct_io, &err); +break ;default :assert (0 ); }migration/options.c +26
-0@@ -826,6 +826,22 @@ int migrate_decompress_threads(void)return s->parameters.decompress_threads; } +bool migrate_direct_io (void ) +{ +MigrationState *s =migrate_get_current (); + +/* For now O_DIRECT is only supported with mapped-ram */ +if (!s->capabilities[MIGRATION_CAPABILITY_MAPPED_RAM ]) { +return false; + } + +if (s->parameters.has_direct_io) { +return s->parameters.direct_io; + } + +return false; +} +uint64_t migrate_downtime_limit (void ) {MigrationState *s =migrate_get_current ();@@ -1061,6 +1077,11 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->has_zero_page_detection = true; params->zero_page_detection = s->parameters.zero_page_detection; +if (s->parameters.has_direct_io) { + params->has_direct_io = true; + params->direct_io = s->parameters.direct_io; + } +return params; }@@ -1097,6 +1118,7 @@ void migrate_params_init(MigrationParameters *params) params->has_vcpu_dirty_limit = true; params->has_mode = true; params->has_zero_page_detection = true; + params->has_direct_io =qemu_has_direct_io (); }/* @@ -1416,6 +1438,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_zero_page_detection) { dest->zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + dest->direct_io = params->direct_io; + } }static void migrate_params_apply (MigrateSetParameters *params,Error **errp)
migration/options.c static void migrate_params_apply(MigrateSetParameters *params, Error **errp) · line 1603
Quoted 8 lines
migration/options.c +4
-0@@ -1570,6 +1596,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_zero_page_detection) { s->parameters.zero_page_detection = params->zero_page_detection; } + +if (params->has_direct_io) { + s->parameters.direct_io = params->direct_io; + } }
I would expect to see something added to migrat_params_check() that calls qemu_has_direct_io() and reports an error if the platform lacks O_DIRECT, so mgmt apps see when they're trying to use O_DIRECT on a bad platform straightaway, rather than only when migration starts later.
Alternatively, and perhaps better would be for use to have a meson.build check for O_DIRECT, and then make all the QAPI features have a condition on CONFIG_O_DIRECT, so QAPI rejects any use of 'direct-io' feature at input time.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 5/9 migration/multifd: Add direct-io support
In reply to Fabiano RosasWhen multifd is used along with mapped-ram, we can take benefit of a filesystem that supports the O_DIRECT flag and perform direct I/O in the multifd threads. This brings a significant performance improvement because direct-io writes bypass the page cache which would otherwise be thrashed by the multifd data which is unlikely to be needed again in a short period of time.
To be able to use a multifd channel opened with O_DIRECT, we must ensure that a certain aligment is used. Filesystems usually require a block-size alignment for direct I/O. The way to achieve this is by enabling the mapped-ram feature, which already aligns its I/O properly (see MAPPED_RAM_FILE_OFFSET_ALIGNMENT at ram.c).
By setting O_DIRECT on the multifd channels, all writes to the same file descriptor need to be aligned as well, even the ones that come from outside multifd, such as the QEMUFile I/O from the main migration code. This makes it impossible to use the same file descriptor for the QEMUFile and for the multifd channels. The various flags and metadata written by the main migration code will always be unaligned by virtue of their small size. To workaround this issue, we'll require a second file descriptor to be used exclusively for direct I/O.
The second file descriptor can be obtained by QEMU by re-opening the migration file (already possible), or by being provided by the user or management application (support to be added in future patches).
--- migration/file.c | 22 +++++++++++++++++++--- migration/migration.c | 23 +++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-)
migration/file.c +19 -3
@@ -83,17 +83,33 @@ void file_cleanup_outgoing_migration(void)bool file_send_channel_create (gpointer opaque,Error **errp) { -QIOChannelFile *ioc; +QIOChannelFile *ioc =NULL ;int flags =O_WRONLY ; -bool ret = true; +bool ret = false; + +if (migrate_direct_io ()) { +#ifdef O_DIRECT +/* +* Enable O_DIRECT for the secondary channels. These are used +* for sending ram pages and writes should be guaranteed to be +* aligned to at least page size. +*/ + flags |=O_DIRECT ; +#else +error_setg (errp,"System does not support O_DIRECT" ); +error_append_hint (errp, +"Try disabling direct-io migration capability\n" ); + goto out; +#endif + } ioc =qio_channel_file_new_path (outgoing_args.fname, flags,0 , errp);if (!ioc) { - ret = false; goto out; }multifd_channel_connect (opaque,QIO_CHANNEL (ioc)); + ret = true; out:/*
migration/migration.c +23 -0
@@ -155,6 +155,16 @@ static bool migration_needs_seekable_channel(void)return migrate_mapped_ram (); } +static bool migration_needs_multiple_fds (void ) +{ +/* +* When doing direct-io, multifd requires two different, +* non-duplicated file descriptors so we can use one of them for +* unaligned IO. +*/ +return migrate_multifd () &&migrate_direct_io (); +} +static bool transport_supports_seeking (MigrationAddress *addr) {if (addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ) {
@@ -164,6 +174,12 @@ static bool transport_supports_seeking(MigrationAddress *addr)return false; } +static bool transport_supports_multiple_fds (MigrationAddress *addr) +{ +/* file: works because QEMU can open it multiple times */ +return addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ; +} +static bool migration_channels_and_transport_compatible (MigrationAddress *addr,Error **errp)
@@ -180,6 +196,13 @@ migration_channels_and_transport_compatible(MigrationAddress *addr,return false; } +if (migration_needs_multiple_fds () && + !transport_supports_multiple_fds (addr)) { +error_setg (errp, +"Migration requires a transport that allows for multiple fds (e.g. file)" ); +return false; + } +return true; }
Signature · 1 line
2.35.3
Re: [PATCH 5/9] migration/multifd: Add direct-io support
Quoted 50 lines · Fabiano Rosas
When multifd is used along with mapped-ram, we can take benefit of a filesystem that supports the
O_DIRECTflag and perform direct I/O in the multifd threads. This brings a significant performance improvement because direct-io writes bypass the page cache which would otherwise be thrashed by the multifd data which is unlikely to be needed again in a short period of time.To be able to use a multifd channel opened with
O_DIRECT, we must ensure that a certain aligment is used. Filesystems usually require a block-size alignment for direct I/O. The way to achieve this is by enabling the mapped-ram feature, which already aligns its I/O properly (seeMAPPED_RAM_FILE_OFFSET_ALIGNMENTatram.c).By setting
O_DIRECTon the multifd channels, all writes to the same file descriptor need to be aligned as well, even the ones that come from outside multifd, such as the QEMUFile I/O from the main migration code. This makes it impossible to use the same file descriptor for the QEMUFile and for the multifd channels. The various flags and metadata written by the main migration code will always be unaligned by virtue of their small size. To workaround this issue, we'll require a second file descriptor to be used exclusively for direct I/O.The second file descriptor can be obtained by QEMU by re-opening the migration file (already possible), or by being provided by the user or management application (support to be added in future patches).
Signed-off-by Fabiano Rosas <[email protected]>--- migration/file.c | 22 +++++++++++++++++++--- migration/migration.c | 23 +++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-)migration/file.c +7
-2@@ -83,17 +83,33 @@ void file_cleanup_outgoing_migration(void)bool file_send_channel_create (gpointer opaque,Error **errp) { -QIOChannelFile *ioc; +QIOChannelFile *ioc =NULL ;int flags =O_WRONLY ; -bool ret = true; +bool ret = false; + +if (migrate_direct_io ()) { +#ifdef O_DIRECT +/* +* Enable O_DIRECT for the secondary channels. These are used
migration/file.c void file_cleanup_outgoing_migration(void) · line 103
Quoted 10 lines
migration/file.c +10
-0+* for sending ram pages and writes should be guaranteed to be +* aligned to at least page size. +*/ + flags |=O_DIRECT ; +#else +error_setg (errp,"System does not support O_DIRECT" ); +error_append_hint (errp, +"Try disabling direct-io migration capability\n" ); + goto out; +#endif
Hopefully if we can fail migrate-set-parameters correctly always, we will never trigger this error.
I know Linux used some trick like this to even avoid such ifdefs:
if (qemu_has_direct_io() && migrate_direct_io()) {
// reference O_DIRECT
}So as long as qemu_has_direct_io() can return a constant "false" when O_DIRECT not defined, the compiler is smart enough to ignore the O_DIRECT inside the block.
Even if it won't work, we can still avoid that error (and rely on the set-parameter failure):
#ifdef O_DIRECT
if (migrate_direct_io()) {
// reference O_DIRECT
}#endif
Then it should run the same, just to try making ifdefs as light as possible..
Quoted 18 lines
migration/file.c +2
-1+ } ioc =qio_channel_file_new_path (outgoing_args.fname, flags,0 , errp);if (!ioc) { - ret = false; goto out; }multifd_channel_connect (opaque,QIO_CHANNEL (ioc)); + ret = true; out:/* migration/migration.c +0
-0
migration/migration.c static bool migration_needs_seekable_channel(void) · line 158
Quoted 4 lines
migration/migration.c +1
-0@@ -155,6 +155,16 @@ static bool migration_needs_seekable_channel(void)return migrate_mapped_ram (); } +static bool migration_needs_multiple_fds (void )
If I suggest to rename this, would you agree? :)
I'd try with "migrate_needs_extra_fd()" or "migrate_needs_two_fds()", or... just to avoid "multi" + "fd" used altogether, perhaps.
Other than that looks all good.
Thanks,
Quoted 42 lines
migration/migration.c +22
-0+{ +/* +* When doing direct-io, multifd requires two different, +* non-duplicated file descriptors so we can use one of them for +* unaligned IO. +*/ +return migrate_multifd () &&migrate_direct_io (); +} +static bool transport_supports_seeking (MigrationAddress *addr) {if (addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ) {@@ -164,6 +174,12 @@ static bool transport_supports_seeking(MigrationAddress *addr)return false; } +static bool transport_supports_multiple_fds (MigrationAddress *addr) +{ +/* file: works because QEMU can open it multiple times */ +return addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ; +} +static bool migration_channels_and_transport_compatible (MigrationAddress *addr,Error **errp)@@ -180,6 +196,13 @@ migration_channels_and_transport_compatible(MigrationAddress *addr,return false; } +if (migration_needs_multiple_fds () && + !transport_supports_multiple_fds (addr)) { +error_setg (errp, +"Migration requires a transport that allows for multiple fds (e.g. file)" ); +return false; + } +return true; }Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH 5/9] migration/multifd: Add direct-io support
Quoted 78 lines · Peter Xu
Quoted 50 lines · Fabiano Rosas
When multifd is used along with mapped-ram, we can take benefit of a filesystem that supports the
O_DIRECTflag and perform direct I/O in the multifd threads. This brings a significant performance improvement because direct-io writes bypass the page cache which would otherwise be thrashed by the multifd data which is unlikely to be needed again in a short period of time.To be able to use a multifd channel opened with
O_DIRECT, we must ensure that a certain aligment is used. Filesystems usually require a block-size alignment for direct I/O. The way to achieve this is by enabling the mapped-ram feature, which already aligns its I/O properly (seeMAPPED_RAM_FILE_OFFSET_ALIGNMENTatram.c).By setting
O_DIRECTon the multifd channels, all writes to the same file descriptor need to be aligned as well, even the ones that come from outside multifd, such as the QEMUFile I/O from the main migration code. This makes it impossible to use the same file descriptor for the QEMUFile and for the multifd channels. The various flags and metadata written by the main migration code will always be unaligned by virtue of their small size. To workaround this issue, we'll require a second file descriptor to be used exclusively for direct I/O.The second file descriptor can be obtained by QEMU by re-opening the migration file (already possible), or by being provided by the user or management application (support to be added in future patches).
Signed-off-by Fabiano Rosas <[email protected]>--- migration/file.c | 22 +++++++++++++++++++--- migration/migration.c | 23 +++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-)migration/file.c +7
-2@@ -83,17 +83,33 @@ void file_cleanup_outgoing_migration(void)bool file_send_channel_create (gpointer opaque,Error **errp) { -QIOChannelFile *ioc; +QIOChannelFile *ioc =NULL ;int flags =O_WRONLY ; -bool ret = true; +bool ret = false; + +if (migrate_direct_io ()) { +#ifdef O_DIRECT +/* +* Enable O_DIRECT for the secondary channels. These are used migration/file.c void file_cleanup_outgoing_migration(void) · line 103
Quoted 10 lines
migration/file.c +10
-0+* for sending ram pages and writes should be guaranteed to be +* aligned to at least page size. +*/ + flags |=O_DIRECT ; +#else +error_setg (errp,"System does not support O_DIRECT" ); +error_append_hint (errp, +"Try disabling direct-io migration capability\n" ); + goto out; +#endif Hopefully if we can fail migrate-set-parameters correctly always, we will never trigger this error.
I know Linux used some trick like this to even avoid such ifdefs:
if (qemu_has_direct_io() && migrate_direct_io()) { // reference O_DIRECT }So as long as
qemu_has_direct_io()can return a constant "false" whenO_DIRECTnot defined, the compiler is smart enough to ignore theO_DIRECTinside the block.Even if it won't work, we can still avoid that error (and rely on the set-parameter failure):
Quoted 8 lines
#ifdef O_DIRECT if (migrate_direct_io()) { // reference O_DIRECT }#endif
Then it should run the same, just to try making ifdefs as light as possible..
Ok.
Just FYI, in v2 I'm adding direct-io to migration incoming side as well, so I put this logic into a helper:
static bool file_enable_direct_io(int *flags, Error **errp)
{
if (migrate_direct_io()) {
#ifdef O_DIRECT
*flags |= O_DIRECT;
#else
error_setg(errp, "System does not support O_DIRECT");
error_append_hint(errp,
"Try disabling direct-io migration capability\n");
return false;
#endif
}
return true;
}But I'll apply your suggestions nonetheless.
Quoted 22 lines
Quoted 22 lines
migration/file.c +2
-1+ } ioc =qio_channel_file_new_path (outgoing_args.fname, flags,0 , errp);if (!ioc) { - ret = false; goto out; }multifd_channel_connect (opaque,QIO_CHANNEL (ioc)); + ret = true; out:/* migration/migration.c +0
-0@@ -155,6 +155,16 @@ static bool migration_needs_seekable_channel(void)return migrate_mapped_ram (); }
Quoted 4 lines
migration/migration.c static bool migration_needs_seekable_channel(void) · line 158
Quoted 1 line
migration/migration.c +1
-0+static bool migration_needs_multiple_fds (void )If I suggest to rename this, would you agree? :)
Sure, although this is the more accurate usage than "multifd" hehe.
Quoted 49 lines
I'd try with "
migrate_needs_extra_fd()" or "migrate_needs_two_fds()", or... just to avoid "multi" + "fd" used altogether, perhaps.Other than that looks all good.
Thanks,
Quoted 42 lines
migration/migration.c +22
-0+{ +/* +* When doing direct-io, multifd requires two different, +* non-duplicated file descriptors so we can use one of them for +* unaligned IO. +*/ +return migrate_multifd () &&migrate_direct_io (); +} +static bool transport_supports_seeking (MigrationAddress *addr) {if (addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ) {@@ -164,6 +174,12 @@ static bool transport_supports_seeking(MigrationAddress *addr)return false; } +static bool transport_supports_multiple_fds (MigrationAddress *addr) +{ +/* file: works because QEMU can open it multiple times */ +return addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ; +} +static bool migration_channels_and_transport_compatible (MigrationAddress *addr,Error **errp)@@ -180,6 +196,13 @@ migration_channels_and_transport_compatible(MigrationAddress *addr,return false; } +if (migration_needs_multiple_fds () && + !transport_supports_multiple_fds (addr)) { +error_setg (errp, +"Migration requires a transport that allows for multiple fds (e.g. file)" ); +return false; + } +return true; }Signature · 1 line
2.35.3
Re: [PATCH 5/9] migration/multifd: Add direct-io support
Quoted 110 lines · Fabiano Rosas
Quoted 78 lines · Peter Xu
Quoted 50 lines · Fabiano Rosas
When multifd is used along with mapped-ram, we can take benefit of a filesystem that supports the
O_DIRECTflag and perform direct I/O in the multifd threads. This brings a significant performance improvement because direct-io writes bypass the page cache which would otherwise be thrashed by the multifd data which is unlikely to be needed again in a short period of time.To be able to use a multifd channel opened with
O_DIRECT, we must ensure that a certain aligment is used. Filesystems usually require a block-size alignment for direct I/O. The way to achieve this is by enabling the mapped-ram feature, which already aligns its I/O properly (seeMAPPED_RAM_FILE_OFFSET_ALIGNMENTatram.c).By setting
O_DIRECTon the multifd channels, all writes to the same file descriptor need to be aligned as well, even the ones that come from outside multifd, such as the QEMUFile I/O from the main migration code. This makes it impossible to use the same file descriptor for the QEMUFile and for the multifd channels. The various flags and metadata written by the main migration code will always be unaligned by virtue of their small size. To workaround this issue, we'll require a second file descriptor to be used exclusively for direct I/O.The second file descriptor can be obtained by QEMU by re-opening the migration file (already possible), or by being provided by the user or management application (support to be added in future patches).
Signed-off-by Fabiano Rosas <[email protected]>--- migration/file.c | 22 +++++++++++++++++++--- migration/migration.c | 23 +++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-)migration/file.c +7
-2@@ -83,17 +83,33 @@ void file_cleanup_outgoing_migration(void)bool file_send_channel_create (gpointer opaque,Error **errp) { -QIOChannelFile *ioc; +QIOChannelFile *ioc =NULL ;int flags =O_WRONLY ; -bool ret = true; +bool ret = false; + +if (migrate_direct_io ()) { +#ifdef O_DIRECT +/* +* Enable O_DIRECT for the secondary channels. These are used migration/file.c void file_cleanup_outgoing_migration(void) · line 103
Quoted 10 lines
migration/file.c +10
-0+* for sending ram pages and writes should be guaranteed to be +* aligned to at least page size. +*/ + flags |=O_DIRECT ; +#else +error_setg (errp,"System does not support O_DIRECT" ); +error_append_hint (errp, +"Try disabling direct-io migration capability\n" ); + goto out; +#endif Hopefully if we can fail migrate-set-parameters correctly always, we will never trigger this error.
I know Linux used some trick like this to even avoid such ifdefs:
if (qemu_has_direct_io() && migrate_direct_io()) { // reference O_DIRECT }So as long as
qemu_has_direct_io()can return a constant "false" whenO_DIRECTnot defined, the compiler is smart enough to ignore theO_DIRECTinside the block.Even if it won't work, we can still avoid that error (and rely on the set-parameter failure):
Quoted 8 lines
#ifdef O_DIRECT if (migrate_direct_io()) { // reference O_DIRECT }#endif
Then it should run the same, just to try making ifdefs as light as possible..
Ok.
Just FYI, in v2 I'm adding direct-io to migration incoming side as well, so I put this logic into a helper:
static bool
file_enable_direct_io(int *flags, Error **errp){ if (migrate_direct_io()) { #ifdef O_DIRECT *flags |= O_DIRECT; #else error_setg(errp, "System does not support O_DIRECT"); error_append_hint(errp, "Try disabling direct-io migration capability\n"); return false; #endif } return true; }
Quoted 1 line
But I'll apply your suggestions nonetheless.
Thanks, please give it a shot, I hope it will work with either way.
One thing to mention is, if you want to play with the qemu_has_direct_io() approach with no "#ifdefs", you can't keep qemu_has_direct_io() in osdep.c, but you must define it in osdep.h as static inline functions. Otherwise I think osdep.o is forced to include it as a function so that trick won't work.
Just try compile without O_DIRECT should see.
Signature · 1 line
Peter Xu
Re: [PATCH 5/9] migration/multifd: Add direct-io support
In reply to Fabiano RosasQuoted 50 lines · Fabiano Rosas
When multifd is used along with mapped-ram, we can take benefit of a filesystem that supports the
O_DIRECTflag and perform direct I/O in the multifd threads. This brings a significant performance improvement because direct-io writes bypass the page cache which would otherwise be thrashed by the multifd data which is unlikely to be needed again in a short period of time.To be able to use a multifd channel opened with
O_DIRECT, we must ensure that a certain aligment is used. Filesystems usually require a block-size alignment for direct I/O. The way to achieve this is by enabling the mapped-ram feature, which already aligns its I/O properly (seeMAPPED_RAM_FILE_OFFSET_ALIGNMENTatram.c).By setting
O_DIRECTon the multifd channels, all writes to the same file descriptor need to be aligned as well, even the ones that come from outside multifd, such as the QEMUFile I/O from the main migration code. This makes it impossible to use the same file descriptor for the QEMUFile and for the multifd channels. The various flags and metadata written by the main migration code will always be unaligned by virtue of their small size. To workaround this issue, we'll require a second file descriptor to be used exclusively for direct I/O.The second file descriptor can be obtained by QEMU by re-opening the migration file (already possible), or by being provided by the user or management application (support to be added in future patches).
Signed-off-by Fabiano Rosas <[email protected]>--- migration/file.c | 22 +++++++++++++++++++--- migration/migration.c | 23 +++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-)migration/file.c +7
-2@@ -83,17 +83,33 @@ void file_cleanup_outgoing_migration(void)bool file_send_channel_create (gpointer opaque,Error **errp) { -QIOChannelFile *ioc; +QIOChannelFile *ioc =NULL ;int flags =O_WRONLY ; -bool ret = true; +bool ret = false; + +if (migrate_direct_io ()) { +#ifdef O_DIRECT +/* +* Enable O_DIRECT for the secondary channels. These are used
migration/file.c void file_cleanup_outgoing_migration(void) · line 103
Quoted 10 lines
migration/file.c +10
-0+* for sending ram pages and writes should be guaranteed to be +* aligned to at least page size. +*/ + flags |=O_DIRECT ; +#else +error_setg (errp,"System does not support O_DIRECT" ); +error_append_hint (errp, +"Try disabling direct-io migration capability\n" ); + goto out; +#endif
If we conditionalize existance of 'direct-io' feature in the QAPI schema, then the '#else' clause no longer needs to even exist as it will be unreachable - it could be a g_assert_not_reached();
Quoted 64 lines
migration/file.c +2
-1+ } ioc =qio_channel_file_new_path (outgoing_args.fname, flags,0 , errp);if (!ioc) { - ret = false; goto out; }multifd_channel_connect (opaque,QIO_CHANNEL (ioc)); + ret = true; out:/* migration/migration.c +23
-0@@ -155,6 +155,16 @@ static bool migration_needs_seekable_channel(void)return migrate_mapped_ram (); } +static bool migration_needs_multiple_fds (void ) +{ +/* +* When doing direct-io, multifd requires two different, +* non-duplicated file descriptors so we can use one of them for +* unaligned IO. +*/ +return migrate_multifd () &&migrate_direct_io (); +} +static bool transport_supports_seeking (MigrationAddress *addr) {if (addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ) {@@ -164,6 +174,12 @@ static bool transport_supports_seeking(MigrationAddress *addr)return false; } +static bool transport_supports_multiple_fds (MigrationAddress *addr) +{ +/* file: works because QEMU can open it multiple times */ +return addr->transport ==MIGRATION_ADDRESS_TYPE_FILE ; +} +static bool migration_channels_and_transport_compatible (MigrationAddress *addr,Error **errp)@@ -180,6 +196,13 @@ migration_channels_and_transport_compatible(MigrationAddress *addr,return false; } +if (migration_needs_multiple_fds () && + !transport_supports_multiple_fds (addr)) { +error_setg (errp, +"Migration requires a transport that allows for multiple fds (e.g. file)" ); +return false; + } +return true; }Signature · 1 line
2.35.3
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 6/9 tests/qtest/migration: Add tests for file migration with direct-io
In reply to Fabiano RosasThe tests are only allowed to run in systems that know about the O_DIRECT flag and in filesystems which support it.
--- tests/qtest/migration-helpers.c | 42 +++++++++++++++++++++++++++++++++ tests/qtest/migration-helpers.h | 1 + tests/qtest/migration-test.c | 42 +++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+)
tests/qtest/migration-helpers.c +42 -0
@@ -473,3 +473,45 @@ void migration_test_add(const char *path, void (*fn)(void))qtest_add_data_func_full (path, test, migration_test_wrapper, migration_test_destroy); } + +#ifdef O_DIRECT +/* +* Probe for O_DIRECT support on the filesystem. Since this is used +* for tests, be conservative, if anything fails, assume it's +* unsupported. +*/ +bool probe_o_direct_support (const char *tmpfs) +{ + g_autofree char *filename = g_strdup_printf("%s/probe-o-direct", tmpfs); +int fd, flags =O_CREAT |O_RDWR |O_TRUNC |O_DIRECT ; +void *buf; +ssize_t ret, len; +uint64_t offset; + + fd =open (filename, flags,0660 ); +if (fd <0 ) { +unlink (filename); +return false; + } + +/* +* Assuming 4k should be enough to satisfy O_DIRECT alignment +* requirements. The migration code uses 1M to be conservative. +*/ + len =0x100000 ; + offset =0x100000 ; + + buf =aligned_alloc (len, len); +g_assert (buf); + + ret =pwrite (fd, buf, len, offset); +unlink (filename); +g_free (buf); + +if (ret <0 ) { +return false; + } + +return true; +} +#endif
tests/qtest/migration-helpers.h +1 -0
@@ -54,5 +54,6 @@ char *find_common_machine_version(const char *mtype, const char *var1, const char *var2);char *resolve_machine_version (const char *alias,const char *var1,const char *var2); +bool probe_o_direct_support (const char *tmpfs);void migration_test_add (const char *path,void (*fn)(void ));#endif /* MIGRATION_HELPERS_H */
tests/qtest/migration-test.c +42 -0
54 lines not on this page · raw message
Signature · 1 line
2.35.3
Re: [PATCH 6/9] tests/qtest/migration: Add tests for file migration with direct-io
Quoted 4 lines · Fabiano Rosas
The tests are only allowed to run in systems that know about the
O_DIRECTflag and in filesystems which support it.Signed-off-by Fabiano Rosas <[email protected]>
Mostly:
Two trivial comments below.
Quoted 33 lines
--- tests/qtest/migration-helpers.c | 42 +++++++++++++++++++++++++++++++++ tests/qtest/migration-helpers.h | 1 + tests/qtest/migration-test.c | 42 +++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+)tests/qtest/migration-helpers.c +19
-0@@ -473,3 +473,45 @@ void migration_test_add(const char *path, void (*fn)(void))qtest_add_data_func_full (path, test, migration_test_wrapper, migration_test_destroy); } + +#ifdef O_DIRECT +/* +* Probe for O_DIRECT support on the filesystem. Since this is used +* for tests, be conservative, if anything fails, assume it's +* unsupported. +*/ +bool probe_o_direct_support (const char *tmpfs) +{ + g_autofree char *filename = g_strdup_printf("%s/probe-o-direct", tmpfs); +int fd, flags =O_CREAT |O_RDWR |O_TRUNC |O_DIRECT ; +void *buf; +ssize_t ret, len; +uint64_t offset; + + fd =open (filename, flags,0660 ); +if (fd <0 ) { +unlink (filename); +return false;
tests/qtest/migration-helpers.c void migration_test_add(const char *path, void (*fn)(void)) · line 504
Quoted 10 lines
tests/qtest/migration-helpers.c +10
-0+ } + +/* +* Assuming 4k should be enough to satisfy O_DIRECT alignment +* requirements. The migration code uses 1M to be conservative. +*/ + len =0x100000 ; + offset =0x100000 ; + + buf =aligned_alloc (len, len);
This is the first usage of aligned_alloc() in qemu. IIUC it's just a newer posix_memalign(), which QEMU has one use of, and it's protected with:
#if defined(CONFIG_POSIX_MEMALIGN)
int ret;
ret = posix_memalign(&ptr, alignment, size);
...#endif
Didn't check deeper. Just keep this in mind if you see any compilation issues in future CIs, or simply switch to similar pattern.
Quoted 29 lines
tests/qtest/migration-helpers.c +13
-0+g_assert (buf); + + ret =pwrite (fd, buf, len, offset); +unlink (filename); +g_free (buf); + +if (ret <0 ) { +return false; + } + + return true; +} +#endif tests/qtest/migration-helpers.h +1
-0@@ -54,5 +54,6 @@ char *find_common_machine_version(const char *mtype, const char *var1, const char *var2);char *resolve_machine_version (const char *alias,const char *var1,const char *var2); +bool probe_o_direct_support (const char *tmpfs);void migration_test_add (const char *path,void (*fn)(void ));#endif /* MIGRATION_HELPERS_H */ tests/qtest/migration-test.c +0
-0
tests/qtest/migration-test.c static void test_multifd_file_mapped_ram(void) · line 2302
Quoted 8 lines
tests/qtest/migration-test.c +5
-0@@ -2295,6 +2295,43 @@ static void test_multifd_file_mapped_ram(void)test_file_common (&args, true); } +#ifdef O_DIRECT +static void *migrate_mapped_ram_dio_start (QTestState *from, +QTestState *to) +{ +migrate_mapped_ram_start (from, to);
This line seems redundant, migrate_multifd_mapped_ram_start() should cover that.
Quoted 50 lines
tests/qtest/migration-test.c +37
-0+migrate_set_parameter_bool (from,"direct-io" , true); +migrate_set_parameter_bool (to,"direct-io" , true); + +return NULL ; +} + +static void *migrate_multifd_mapped_ram_dio_start (QTestState *from, +QTestState *to) +{ +migrate_multifd_mapped_ram_start (from, to); +return migrate_mapped_ram_dio_start (from, to); +} + +static void test_multifd_file_mapped_ram_dio (void ) +{ + g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs, +FILE_TEST_FILENAME ); +MigrateCommon args = { + .connect_uri = uri, + .listen_uri ="defer" , + .start_hook = migrate_multifd_mapped_ram_dio_start, + }; + +if (!probe_o_direct_support (tmpfs)) { +g_test_skip ("Filesystem does not support O_DIRECT" ); +return ; + } + +test_file_common (&args, true); +} + +#endif /* O_DIRECT */ static void test_precopy_tcp_plain (void ) {@@ -3719,6 +3756,11 @@ int main(int argc, char **argv)migration_test_add ("/migration/multifd/file/mapped-ram/live" , test_multifd_file_mapped_ram_live); +#ifdef O_DIRECT +migration_test_add ("/migration/multifd/file/mapped-ram/dio" , + test_multifd_file_mapped_ram_dio); +#endif +#ifdef CONFIG_GNUTLS migration_test_add ("/migration/precopy/unix/tls/psk" , test_precopy_unix_tls_psk);Signature · 1 line
2.35.3
Signature · 1 line
Peter Xu
Re: [PATCH 6/9] tests/qtest/migration: Add tests for file migration with direct-io
Quoted 100 lines · Peter Xu
Quoted 4 lines · Fabiano Rosas
The tests are only allowed to run in systems that know about the
O_DIRECTflag and in filesystems which support it.Signed-off-by Fabiano Rosas <[email protected]>Mostly:
Reviewed-by Peter Xu <[email protected]>Two trivial comments below.
Quoted 33 lines
--- tests/qtest/migration-helpers.c | 42 +++++++++++++++++++++++++++++++++ tests/qtest/migration-helpers.h | 1 + tests/qtest/migration-test.c | 42 +++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+)tests/qtest/migration-helpers.c +19
-0@@ -473,3 +473,45 @@ void migration_test_add(const char *path, void (*fn)(void))qtest_add_data_func_full (path, test, migration_test_wrapper, migration_test_destroy); } + +#ifdef O_DIRECT +/* +* Probe for O_DIRECT support on the filesystem. Since this is used +* for tests, be conservative, if anything fails, assume it's +* unsupported. +*/ +bool probe_o_direct_support (const char *tmpfs) +{ + g_autofree char *filename = g_strdup_printf("%s/probe-o-direct", tmpfs); +int fd, flags =O_CREAT |O_RDWR |O_TRUNC |O_DIRECT ; +void *buf; +ssize_t ret, len; +uint64_t offset; + + fd =open (filename, flags,0660 ); +if (fd <0 ) { +unlink (filename); +return false;tests/qtest/migration-helpers.c void migration_test_add(const char *path, void (*fn)(void)) · line 504
Quoted 10 lines
tests/qtest/migration-helpers.c +10
-0+ } + +/* +* Assuming 4k should be enough to satisfy O_DIRECT alignment +* requirements. The migration code uses 1M to be conservative. +*/ + len =0x100000 ; + offset =0x100000 ; + + buf =aligned_alloc (len, len);This is the first usage of
aligned_alloc()in qemu. IIUC it's just a newerposix_memalign(), which QEMU has one use of, and it's protected with:#if defined(CONFIG_POSIX_MEMALIGN) int ret; ret = posix_memalign(&ptr, alignment, size); ...#endif
Didn't check deeper. Just keep this in mind if you see any compilation issues in future CIs, or simply switch to similar pattern.
Quoted 32 lines
tests/qtest/migration-helpers.c +13
-0+g_assert (buf); + + ret =pwrite (fd, buf, len, offset); +unlink (filename); +g_free (buf); + +if (ret <0 ) { +return false; + } + + return true; +} +#endif tests/qtest/migration-helpers.h +1
-0@@ -54,5 +54,6 @@ char *find_common_machine_version(const char *mtype, const char *var1, const char *var2);char *resolve_machine_version (const char *alias,const char *var1,const char *var2); +bool probe_o_direct_support (const char *tmpfs);void migration_test_add (const char *path,void (*fn)(void ));#endif /* MIGRATION_HELPERS_H */ tests/qtest/migration-test.c +0
-0@@ -2295,6 +2295,43 @@ static void test_multifd_file_mapped_ram(void)test_file_common (&args, true); }
Quoted 8 lines
tests/qtest/migration-test.c static void test_multifd_file_mapped_ram(void) · line 2302
Quoted 5 lines
tests/qtest/migration-test.c +5
-0+#ifdef O_DIRECT +static void *migrate_mapped_ram_dio_start (QTestState *from, +QTestState *to) +{ +migrate_mapped_ram_start (from, to);This line seems redundant,
migrate_multifd_mapped_ram_start()should cover that.
This is an artifact of another patch that adds direct-io + mapped-ram without multifd. I'm bringing that back on v2. We were having a discussion[1] about it in the libvirt mailing list. Having direct-io even without multifd might still be useful for libvirt.
1- https://lists.libvirt.org/archives/list/[email protected]/thread/K4BDDJDMJ22XMJEFAUE323H5S5E47VQX/
Quoted 50 lines
Quoted 50 lines
tests/qtest/migration-test.c +37
-0+migrate_set_parameter_bool (from,"direct-io" , true); +migrate_set_parameter_bool (to,"direct-io" , true); + +return NULL ; +} + +static void *migrate_multifd_mapped_ram_dio_start (QTestState *from, +QTestState *to) +{ +migrate_multifd_mapped_ram_start (from, to); +return migrate_mapped_ram_dio_start (from, to); +} + +static void test_multifd_file_mapped_ram_dio (void ) +{ + g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs, +FILE_TEST_FILENAME ); +MigrateCommon args = { + .connect_uri = uri, + .listen_uri ="defer" , + .start_hook = migrate_multifd_mapped_ram_dio_start, + }; + +if (!probe_o_direct_support (tmpfs)) { +g_test_skip ("Filesystem does not support O_DIRECT" ); +return ; + } + +test_file_common (&args, true); +} + +#endif /* O_DIRECT */ static void test_precopy_tcp_plain (void ) {@@ -3719,6 +3756,11 @@ int main(int argc, char **argv)migration_test_add ("/migration/multifd/file/mapped-ram/live" , test_multifd_file_mapped_ram_live); +#ifdef O_DIRECT +migration_test_add ("/migration/multifd/file/mapped-ram/dio" , + test_multifd_file_mapped_ram_dio); +#endif +#ifdef CONFIG_GNUTLS migration_test_add ("/migration/precopy/unix/tls/psk" , test_precopy_unix_tls_psk);Signature · 1 line
2.35.3
Re: [PATCH 6/9] tests/qtest/migration: Add tests for file migration with direct-io
Quoted 9 lines · Fabiano Rosas
Quoted 8 lines
tests/qtest/migration-test.c static void test_multifd_file_mapped_ram(void) · line 2302
Quoted 5 lines
tests/qtest/migration-test.c +5
-0+#ifdef O_DIRECT +static void *migrate_mapped_ram_dio_start (QTestState *from, +QTestState *to) +{ +migrate_mapped_ram_start (from, to);This line seems redundant,
migrate_multifd_mapped_ram_start()should cover that.
Quoted 6 lines
This is an artifact of another patch that adds direct-io + mapped-ram without multifd. I'm bringing that back on v2. We were having a discussion[1] about it in the libvirt mailing list. Having direct-io even without multifd might still be useful for libvirt.
1- https://lists.libvirt.org/archives/list/[email protected]/thread/K4BDDJDMJ22XMJEFAUE323H5S5E47VQX/
Ah that's fine then. Maybe add a comment somewhere for future readers? Or a sentence in the commit log would work too.
Signature · 1 line
Peter Xu
Re: [PATCH 6/9] tests/qtest/migration: Add tests for file migration with direct-io
In reply to Fabiano RosasQuoted 35 lines · Fabiano Rosas
The tests are only allowed to run in systems that know about the
O_DIRECTflag and in filesystems which support it.Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-helpers.c | 42 +++++++++++++++++++++++++++++++++ tests/qtest/migration-helpers.h | 1 + tests/qtest/migration-test.c | 42 +++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+)tests/qtest/migration-helpers.c +17
-0@@ -473,3 +473,45 @@ void migration_test_add(const char *path, void (*fn)(void))qtest_add_data_func_full (path, test, migration_test_wrapper, migration_test_destroy); } + +#ifdef O_DIRECT +/* +* Probe for O_DIRECT support on the filesystem. Since this is used +* for tests, be conservative, if anything fails, assume it's +* unsupported. +*/ +bool probe_o_direct_support (const char *tmpfs) +{ + g_autofree char *filename = g_strdup_printf("%s/probe-o-direct", tmpfs); +int fd, flags =O_CREAT |O_RDWR |O_TRUNC |O_DIRECT ; +void *buf; +ssize_t ret, len; +uint64_t offset; + + fd =open (filename, flags,0660 ); +if (fd <0 ) {
tests/qtest/migration-helpers.c void migration_test_add(const char *path, void (*fn)(void)) · line 502
Quoted 10 lines
tests/qtest/migration-helpers.c +10
-0+unlink (filename); +return false; + } + +/* +* Assuming 4k should be enough to satisfy O_DIRECT alignment +* requirements. The migration code uses 1M to be conservative. +*/ + len =0x100000 ; + offset =0x100000 ;
4k is unlikely insufficient for architectures with a 64k small page size, and filesystem constraints also play a part. Suggest rewording to
/*
* Using 1MB alignment as conservative choice to satisfy
* any plausible architecture default page size, and/or
* filesystem alignment restrictions.
*/
Quoted 89 lines
tests/qtest/migration-helpers.c +15
-0+ + buf =aligned_alloc (len, len); +g_assert (buf); + + ret =pwrite (fd, buf, len, offset); +unlink (filename); +g_free (buf); + +if (ret <0 ) { +return false; + } + + return true; +} +#endif tests/qtest/migration-helpers.h +1
-0@@ -54,5 +54,6 @@ char *find_common_machine_version(const char *mtype, const char *var1, const char *var2);char *resolve_machine_version (const char *alias,const char *var1,const char *var2); +bool probe_o_direct_support (const char *tmpfs);void migration_test_add (const char *path,void (*fn)(void ));#endif /* MIGRATION_HELPERS_H */ tests/qtest/migration-test.c +42
-0@@ -2295,6 +2295,43 @@ static void test_multifd_file_mapped_ram(void)test_file_common (&args, true); } +#ifdef O_DIRECT +static void *migrate_mapped_ram_dio_start (QTestState *from, +QTestState *to) +{ +migrate_mapped_ram_start (from, to); +migrate_set_parameter_bool (from,"direct-io" , true); +migrate_set_parameter_bool (to,"direct-io" , true); + +return NULL ; +} + +static void *migrate_multifd_mapped_ram_dio_start (QTestState *from, +QTestState *to) +{ +migrate_multifd_mapped_ram_start (from, to); +return migrate_mapped_ram_dio_start (from, to); +} + +static void test_multifd_file_mapped_ram_dio (void ) +{ + g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs, +FILE_TEST_FILENAME ); +MigrateCommon args = { + .connect_uri = uri, + .listen_uri ="defer" , + .start_hook = migrate_multifd_mapped_ram_dio_start, + }; + +if (!probe_o_direct_support (tmpfs)) { +g_test_skip ("Filesystem does not support O_DIRECT" ); +return ; + } + +test_file_common (&args, true); +} + +#endif /* O_DIRECT */ static void test_precopy_tcp_plain (void ) {@@ -3719,6 +3756,11 @@ int main(int argc, char **argv)migration_test_add ("/migration/multifd/file/mapped-ram/live" , test_multifd_file_mapped_ram_live); +#ifdef O_DIRECT +migration_test_add ("/migration/multifd/file/mapped-ram/dio" , + test_multifd_file_mapped_ram_dio); +#endif +#ifdef CONFIG_GNUTLS migration_test_add ("/migration/precopy/unix/tls/psk" , test_precopy_unix_tls_psk);Signature · 1 line
2.35.3
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 7/9 monitor: fdset: Match against O_DIRECT
In reply to Fabiano RosasWe're about to enable the use of O_DIRECT in the migration code and due to the alignment restrictions imposed by filesystems we need to make sure the flag is only used when doing aligned IO.
The migration will do parallel IO to different regions of a file, so we need to use more than one file descriptor. Those cannot be obtained by duplicating (dup()) since duplicated file descriptors share the file status flags, including O_DIRECT. If one migration channel does unaligned IO while another sets O_DIRECT to do aligned IO, the filesystem would fail the unaligned operation.
The add-fd QMP command along with the fdset code are specifically designed to allow the user to pass a set of file descriptors with different access flags into QEMU to be later fetched by code that needs to alternate between those flags when doing IO.
Extend the fdset matching to behave the same with the O_DIRECT flag.
--- monitor/fds.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
monitor/fds.c +6 -1
@@ -420,6 +420,11 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)int fd =-1 ;int dup_fd;int mon_fd_flags; +int mask =O_ACCMODE ; + +#ifdef O_DIRECT + mask |=O_DIRECT ; +#endif if (mon_fdset->id != fdset_id) {continue ;
@@ -431,7 +436,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -if ((flags &O_ACCMODE ) == (mon_fd_flags &O_ACCMODE )) { +if ((flags & mask) == (mon_fd_flags & mask)) { fd = mon_fdset_fd->fd;break ; }
Signature · 1 line
2.35.3
Re: [PATCH 7/9] monitor: fdset: Match against O_DIRECT
Quoted 40 lines · Fabiano Rosas
We're about to enable the use of
O_DIRECTin the migration code and due to the alignment restrictions imposed by filesystems we need to make sure the flag is only used when doing aligned IO.The migration will do parallel IO to different regions of a file, so we need to use more than one file descriptor. Those cannot be obtained by duplicating (
dup()) since duplicated file descriptors share the file status flags, includingO_DIRECT. If one migration channel does unaligned IO while another setsO_DIRECTto do aligned IO, the filesystem would fail the unaligned operation.The add-fd QMP command along with the fdset code are specifically designed to allow the user to pass a set of file descriptors with different access flags into QEMU to be later fetched by code that needs to alternate between those flags when doing IO.
Extend the fdset matching to behave the same with the
O_DIRECTflag.Signed-off-by Fabiano Rosas <[email protected]>--- monitor/fds.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)monitor/fds.c +5
-0@@ -420,6 +420,11 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)int fd =-1 ;int dup_fd;int mon_fd_flags; +int mask =O_ACCMODE ; + +#ifdef O_DIRECT + mask |=O_DIRECT ; +#endif if (mon_fdset->id != fdset_id) {continue ;
monitor/fds.c int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags) · line 442
Quoted 8 lines
monitor/fds.c +1
-1@@ -431,7 +436,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -if ((flags &O_ACCMODE ) == (mon_fd_flags &O_ACCMODE )) { +if ((flags & mask) == (mon_fd_flags & mask)) { fd = mon_fdset_fd->fd;break ; }
I think I see what you wanted to do, picking out the right fd out of two when qemu_open_old(), which makes sense.
However what happens if the mgmt app only passes in 1 fd to the fdset? The issue is we have a "fallback dup()" plan right after this chunk of code:
dup_fd = qemu_dup_flags(fd, flags);
if (dup_fd == -1) {
return -1;
}
mon_fdset_fd_dup = g_malloc0(sizeof(*mon_fdset_fd_dup));
mon_fdset_fd_dup->fd = dup_fd;
QLIST_INSERT_HEAD(&mon_fdset->dup_fds, mon_fdset_fd_dup, next);I think it means even if the mgmt app only passes in 1 fd (rather than 2, one with O_DIRECT, one without), QEMU can always successfully call qemu_open_old() twice for each case, even though silently the two FDs will actually impact on each other. This doesn't look ideal if it's true.
But I also must confess I don't really understand this code at all: we dup(), then we try F_SETFL on all the possible flags got passed in. However AFAICT due to the fact that dup()ed FDs will share "struct file" it means mostly all flags will be shared, except close-on-exec. I don't ever see anything protecting that F_SETFL to only touch close-on-exec, I think it means it'll silently change file status flags for the other fd which we dup()ed from. Does it mean that we have issue already with such dup() usage?
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 7/9] monitor: fdset: Match against O_DIRECT
Quoted 53 lines · Peter Xu
Quoted 40 lines · Fabiano Rosas
We're about to enable the use of
O_DIRECTin the migration code and due to the alignment restrictions imposed by filesystems we need to make sure the flag is only used when doing aligned IO.The migration will do parallel IO to different regions of a file, so we need to use more than one file descriptor. Those cannot be obtained by duplicating (
dup()) since duplicated file descriptors share the file status flags, includingO_DIRECT. If one migration channel does unaligned IO while another setsO_DIRECTto do aligned IO, the filesystem would fail the unaligned operation.The add-fd QMP command along with the fdset code are specifically designed to allow the user to pass a set of file descriptors with different access flags into QEMU to be later fetched by code that needs to alternate between those flags when doing IO.
Extend the fdset matching to behave the same with the
O_DIRECTflag.Signed-off-by Fabiano Rosas <[email protected]>--- monitor/fds.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)monitor/fds.c +5
-0@@ -420,6 +420,11 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)int fd =-1 ;int dup_fd;int mon_fd_flags; +int mask =O_ACCMODE ; + +#ifdef O_DIRECT + mask |=O_DIRECT ; +#endif if (mon_fdset->id != fdset_id) {continue ;monitor/fds.c int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags) · line 442
Quoted 8 lines
monitor/fds.c +1
-1@@ -431,7 +436,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -if ((flags &O_ACCMODE ) == (mon_fd_flags &O_ACCMODE )) { +if ((flags & mask) == (mon_fd_flags & mask)) { fd = mon_fdset_fd->fd;break ; }I think I see what you wanted to do, picking out the right fd out of two when
qemu_open_old(), which makes sense.
Quoted 3 lines
However what happens if the mgmt app only passes in 1 fd to the fdset? The issue is we have a "fallback
dup()" plan right after this chunk of code:
I'm validating the fdset at file_parse_fdset() beforehand. If there's anything else than 2 fds then we'll error out:
if (nfds != 2) {
error_setg(errp, "Outgoing migration needs two fds in the fdset, "
"got %d", nfds);
qmp_remove_fd(*id, false, -1, NULL);
*id = -1;
return false;
}Quoted 14 lines
dup_fd = qemu_dup_flags(fd, flags); if (dup_fd == -1) { return -1; } mon_fdset_fd_dup = g_malloc0(sizeof(*mon_fdset_fd_dup)); mon_fdset_fd_dup->fd = dup_fd; QLIST_INSERT_HEAD(&mon_fdset->dup_fds, mon_fdset_fd_dup, next);I think it means even if the mgmt app only passes in 1 fd (rather than 2, one with
O_DIRECT, one without), QEMU can always successfully callqemu_open_old()twice for each case, even though silently the two FDs will actually impact on each other. This doesn't look ideal if it's true.
Quoted 7 lines
But I also must confess I don't really understand this code at all: we
dup(), then we tryF_SETFLon all the possible flags got passed in. However AFAICT due to the fact that dup()ed FDs will share "struct file" it means mostly all flags will be shared, except close-on-exec. I don't ever see anything protecting thatF_SETFLto only touch close-on-exec, I think it means it'll silently change file status flags for the other fd which we dup()ed from. Does it mean that we have issue already with suchdup()usage?
I think you're right, but I also think there's a requirement even from this code that the fds in the fdset cannot be dup()ed. I don't see it enforced anywhere, but maybe that's a consequence of the larger use-case for which this feature was introduced.
For our scenario, the open() man page says one can use kcmp() to compare the fds and determine if they are a result of dup(). Maybe we should do that extra check? We're defining a pretty rigid interface between QEMU and the management layer, so not likely to break once it's written. I'm also not sure how bad would it be to call syscall() directly from QEMU (kcmp has no libc wrapper).
Quoted 2 lines
Thanks,
Re: [PATCH 7/9] monitor: fdset: Match against O_DIRECT
Quoted 92 lines · Fabiano Rosas
Quoted 53 lines · Peter Xu
Quoted 40 lines · Fabiano Rosas
We're about to enable the use of
O_DIRECTin the migration code and due to the alignment restrictions imposed by filesystems we need to make sure the flag is only used when doing aligned IO.The migration will do parallel IO to different regions of a file, so we need to use more than one file descriptor. Those cannot be obtained by duplicating (
dup()) since duplicated file descriptors share the file status flags, includingO_DIRECT. If one migration channel does unaligned IO while another setsO_DIRECTto do aligned IO, the filesystem would fail the unaligned operation.The add-fd QMP command along with the fdset code are specifically designed to allow the user to pass a set of file descriptors with different access flags into QEMU to be later fetched by code that needs to alternate between those flags when doing IO.
Extend the fdset matching to behave the same with the
O_DIRECTflag.Signed-off-by Fabiano Rosas <[email protected]>--- monitor/fds.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)monitor/fds.c +5
-0@@ -420,6 +420,11 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)int fd =-1 ;int dup_fd;int mon_fd_flags; +int mask =O_ACCMODE ; + +#ifdef O_DIRECT + mask |=O_DIRECT ; +#endif if (mon_fdset->id != fdset_id) {continue ;monitor/fds.c int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags) · line 442
Quoted 8 lines
monitor/fds.c +1
-1@@ -431,7 +436,7 @@ int monitor_fdset_dup_fd_add(int64_t fdset_id, int flags)return -1 ; } -if ((flags &O_ACCMODE ) == (mon_fd_flags &O_ACCMODE )) { +if ((flags & mask) == (mon_fd_flags & mask)) { fd = mon_fdset_fd->fd;break ; }I think I see what you wanted to do, picking out the right fd out of two when
qemu_open_old(), which makes sense.Quoted 3 lines
However what happens if the mgmt app only passes in 1 fd to the fdset? The issue is we have a "fallback
dup()" plan right after this chunk of code:I'm validating the fdset at
file_parse_fdset()beforehand. If there's anything else than 2 fds then we'll error out:if (nfds != 2) { error_setg(errp, "Outgoing migration needs two fds in the fdset, " "got %d", nfds); qmp_remove_fd(*id, false, -1, NULL); *id = -1; return false; }Quoted 14 lines
dup_fd = qemu_dup_flags(fd, flags); if (dup_fd == -1) { return -1; } mon_fdset_fd_dup = g_malloc0(sizeof(*mon_fdset_fd_dup)); mon_fdset_fd_dup->fd = dup_fd; QLIST_INSERT_HEAD(&mon_fdset->dup_fds, mon_fdset_fd_dup, next);I think it means even if the mgmt app only passes in 1 fd (rather than 2, one with
O_DIRECT, one without), QEMU can always successfully callqemu_open_old()twice for each case, even though silently the two FDs will actually impact on each other. This doesn't look ideal if it's true.Quoted 7 lines
But I also must confess I don't really understand this code at all: we
dup(), then we tryF_SETFLon all the possible flags got passed in. However AFAICT due to the fact that dup()ed FDs will share "struct file" it means mostly all flags will be shared, except close-on-exec. I don't ever see anything protecting thatF_SETFLto only touch close-on-exec, I think it means it'll silently change file status flags for the other fd which we dup()ed from. Does it mean that we have issue already with suchdup()usage?
Quoted 4 lines
I think you're right, but I also think there's a requirement even from this code that the fds in the fdset cannot be dup()ed. I don't see it enforced anywhere, but maybe that's a consequence of the larger use-case for which this feature was introduced.
I think that's the thing we need to figure out for add-fd usages. The bad thing is there're too many qemu_open_internal() users... so we can't easily tell what we're looking for. May need some time reading the code or the history.. pretty sad. I hope someone can chim in.
Quoted 7 lines
For our scenario, the
open()man page says one can usekcmp()to compare the fds and determine if they are a result ofdup(). Maybe we should do that extra check? We're defining a pretty rigid interface between QEMU and the management layer, so not likely to break once it's written. I'm also not sure how bad would it be to callsyscall()directly from QEMU (kcmp has no libc wrapper).
That should be all fine, see:
$ git grep " syscall(" | wc -l
28And if we want we can also do fcntl(F_GETFL) on both fds later, making sure they have proper flags (one must have O_DIRECT, one must not).
Signature · 1 line
Peter Xu
PATCH 8/9 migration: Add support for fdset with multifd + file
In reply to Fabiano RosasAllow multifd to use an fdset when migrating to a file. This is useful for the scenario where the management layer wants to have control over the migration file.
By receiving the file descriptors directly, QEMU can delegate some high level operating system operations to the management layer (such as mandatory access control). The management layer might also want to add its own headers before the migration stream.
Enable the "file:/dev/fdset/#" syntax for the multifd migration with mapped-ram. The requirements for the fdset mechanism are:
On the migration source side:
- the fdset must contain two fds that are not duplicates between themselves;
- if direct-io is to be used, exactly one of the fds must have the O_DIRECT flag set;
- the file must be opened with WRONLY both times.
On the migration destination side:
- the fdset must contain one fd;
- the file must be opened with RDONLY.
--- docs/devel/migration/main.rst | 18 ++++++++++++++ docs/devel/migration/mapped-ram.rst | 6 ++++- migration/file.c | 38 ++++++++++++++++++++++++++++- 3 files changed, 60 insertions(+), 2 deletions(-)
docs/devel/migration/main.rst +18 -0
@@ -47,6 +47,24 @@ over any transport. QEMU interference. Note that QEMU does not flush cached file data/metadata at the end of migration. + The file migration also supports using a file that has already been + opened. A set of file descriptors is passed to QEMU via an "fdset" + (see add-fd QMP command documentation). This method allows a + management application to have control over the migration file + opening operation. There are, however, strict requirements to this + interface: + + On the migration source side: + - if the multifd capability is to be used, the fdset must contain + two file descriptors that are not duplicates between themselves; + - if the direct-io capability is to be used, exactly one of the + file descriptors must have the O_DIRECT flag set; + - the file must be opened with WRONLY. + + On the migration destination side: + - the fdset must contain one file descriptor; + - the file must be opened with RDONLY. + In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of transporting the pages, and the load on the CPU is much lower. While the
docs/devel/migration/mapped-ram.rst +5 -1
@@ -16,7 +16,7 @@ location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment -restrictions (direct-io support not yet implemented). +restrictions. Usage -----
@@ -35,6 +35,10 @@ Use a ``file:`` URL for migration: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating. +For best performance enable the ``direct-io`` capability as well: + + ``migrate_set_capability direct-io on`` + Use-cases ---------
migration/file.c +37 -1
@@ -17,6 +17,7 @@#include "io/channel-file.h" #include "io/channel-socket.h" #include "io/channel-util.h" +#include "monitor/monitor.h" #include "options.h" #include "trace.h"
@@ -54,10 +55,18 @@ static void file_remove_fdset(void) } } +/* +* With multifd, due to the behavior of the dup() system call, we need +* the fdset to have two non-duplicate fds so we can enable direct IO +* in the secondary channels without affecting the main channel. +*/ static bool file_parse_fdset (const char *filename,int64_t *fdset_id,Error **errp) { +FdsetInfoList *fds_info; +FdsetFdInfoList *fd_info;const char *fdset_id_str; +int nfds =0 ; *fdset_id =-1 ;
@@ -71,6 +80,32 @@ static bool file_parse_fdset(const char *filename, int64_t *fdset_id,return false; } +if (!migrate_multifd () || !migrate_direct_io ()) { +return true; + } + +for (fds_info =qmp_query_fdsets (NULL ); fds_info; + fds_info = fds_info->next) { + +if (*fdset_id != fds_info->value->fdset_id) { +continue ; + } + +for (fd_info = fds_info->value->fds; fd_info; fd_info = fd_info->next) { +if (nfds++ >2 ) { +break ; + } + } + } + +if (nfds !=2 ) { +error_setg (errp,"Outgoing migration needs two fds in the fdset, " +"got %d" , nfds); +qmp_remove_fd (*fdset_id, false,-1 ,NULL ); + *fdset_id =-1 ; +return false; + } +return true; }
@@ -209,10 +244,11 @@ void file_start_incoming_migration(FileMigrationArgs *file_args, Error **errp) g_autofree char *filename = g_strdup(file_args->filename);QIOChannelFile *fioc =NULL ;uint64_t offset = file_args->offset; +int flags =O_RDONLY ;trace_migration_file_incoming (filename); - fioc =qio_channel_file_new_path (filename,O_RDONLY ,0 , errp); + fioc =qio_channel_file_new_path (filename, flags,0 , errp);if (!fioc) {return ; }
Signature · 1 line
2.35.3
Re: [PATCH 8/9] migration: Add support for fdset with multifd + file
Quoted 138 lines · Fabiano Rosas
Allow multifd to use an fdset when migrating to a file. This is useful for the scenario where the management layer wants to have control over the migration file.
By receiving the file descriptors directly, QEMU can delegate some high level operating system operations to the management layer (such as mandatory access control). The management layer might also want to add its own headers before the migration stream.
Enable the "file:/dev/fdset/#" syntax for the multifd migration with mapped-ram. The requirements for the fdset mechanism are:
On the migration source side:
- the fdset must contain two fds that are not duplicates between themselves;
- if direct-io is to be used, exactly one of the fds must have theO_DIRECTflag set;
- the file must be opened with WRONLY both times.On the migration destination side:
- the fdset must contain one fd;
- the file must be opened with RDONLY.Signed-off-by Fabiano Rosas <[email protected]>--- docs/devel/migration/main.rst | 18 ++++++++++++++ docs/devel/migration/mapped-ram.rst | 6 ++++- migration/file.c | 38 ++++++++++++++++++++++++++++- 3 files changed, 60 insertions(+), 2 deletions(-)docs/devel/migration/main.rst +18
-0@@ -47,6 +47,24 @@ over any transport. QEMU interference. Note that QEMU does not flush cached file data/metadata at the end of migration. + The file migration also supports using a file that has already been + opened. A set of file descriptors is passed to QEMU via an "fdset" + (see add-fd QMP command documentation). This method allows a + management application to have control over the migration file + opening operation. There are, however, strict requirements to this + interface: + + On the migration source side: + - if the multifd capability is to be used, the fdset must contain + two file descriptors that are not duplicates between themselves; + - if the direct-io capability is to be used, exactly one of the + file descriptors must have the O_DIRECT flag set; + - the file must be opened with WRONLY. + + On the migration destination side: + - the fdset must contain one file descriptor; + - the file must be opened with RDONLY. + In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of transporting the pages, and the load on the CPU is much lower. While thedocs/devel/migration/mapped-ram.rst +5
-1@@ -16,7 +16,7 @@ location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment -restrictions (direct-io support not yet implemented). +restrictions. Usage -----@@ -35,6 +35,10 @@ Use a ``file:`` URL for migration: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating. +For best performance enable the ``direct-io`` capability as well: + + ``migrate_set_capability direct-io on`` + Use-cases ---------migration/file.c +27
-0@@ -17,6 +17,7 @@#include "io/channel-file.h" #include "io/channel-socket.h" #include "io/channel-util.h" +#include "monitor/monitor.h" #include "options.h" #include "trace.h" @@ -54,10 +55,18 @@ static void file_remove_fdset(void) } } +/* +* With multifd, due to the behavior of the dup() system call, we need +* the fdset to have two non-duplicate fds so we can enable direct IO +* in the secondary channels without affecting the main channel. +*/ static bool file_parse_fdset (const char *filename,int64_t *fdset_id,Error **errp) { +FdsetInfoList *fds_info; +FdsetFdInfoList *fd_info;const char *fdset_id_str; +int nfds =0 ; *fdset_id =-1 ;@@ -71,6 +80,32 @@ static bool file_parse_fdset(const char *filename, int64_t *fdset_id,return false; } +if (!migrate_multifd () || !migrate_direct_io ()) { +return true; + } + +for (fds_info =qmp_query_fdsets (NULL ); fds_info; + fds_info = fds_info->next) { + +if (*fdset_id != fds_info->value->fdset_id) { +continue ; + } + +for (fd_info = fds_info->value->fds; fd_info; fd_info = fd_info->next) { +if (nfds++ >2 ) { +break ; + } + } + } +
migration/file.c static bool file_parse_fdset(const char *filename, int64_t *fdset_id, · line 110
Quoted 10 lines
migration/file.c +8
-0+if (nfds !=2 ) { +error_setg (errp,"Outgoing migration needs two fds in the fdset, " +"got %d" , nfds); +qmp_remove_fd (*fdset_id, false,-1 ,NULL ); + *fdset_id =-1 ; +return false; + } +return true; }
Related to my thoughts in an earlier patch, where I say that use of fdsets ought to be transparent to QEMU code, I'm not a fan of having this logic in migration code.
IIUC, the migration code will call qio_channel_file_new_path twice, once with O_DIRECT and once without. This should trigger two calls into monitor_fdset_dup_fd_add with different flags. If we're matching flags in that monitor_fdset_dup_fd_add(), then if only 1 FD was provided, are we not able to report an error there ?
Quoted 17 lines
migration/file.c +2
-1@@ -209,10 +244,11 @@ void file_start_incoming_migration(FileMigrationArgs *file_args, Error **errp) g_autofree char *filename = g_strdup(file_args->filename);QIOChannelFile *fioc =NULL ;uint64_t offset = file_args->offset; +int flags =O_RDONLY ;trace_migration_file_incoming (filename); - fioc =qio_channel_file_new_path (filename,O_RDONLY ,0 , errp); + fioc =qio_channel_file_new_path (filename, flags,0 , errp);if (!fioc) {return ; }Signature · 1 line
2.35.3
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 8/9] migration: Add support for fdset with multifd + file
Quoted 154 lines · Daniel P. Berrangé
Quoted 138 lines · Fabiano Rosas
Allow multifd to use an fdset when migrating to a file. This is useful for the scenario where the management layer wants to have control over the migration file.
By receiving the file descriptors directly, QEMU can delegate some high level operating system operations to the management layer (such as mandatory access control). The management layer might also want to add its own headers before the migration stream.
Enable the "file:/dev/fdset/#" syntax for the multifd migration with mapped-ram. The requirements for the fdset mechanism are:
On the migration source side:
- the fdset must contain two fds that are not duplicates between themselves;
- if direct-io is to be used, exactly one of the fds must have theO_DIRECTflag set;
- the file must be opened with WRONLY both times.On the migration destination side:
- the fdset must contain one fd;
- the file must be opened with RDONLY.Signed-off-by Fabiano Rosas <[email protected]>--- docs/devel/migration/main.rst | 18 ++++++++++++++ docs/devel/migration/mapped-ram.rst | 6 ++++- migration/file.c | 38 ++++++++++++++++++++++++++++- 3 files changed, 60 insertions(+), 2 deletions(-)docs/devel/migration/main.rst +18
-0@@ -47,6 +47,24 @@ over any transport. QEMU interference. Note that QEMU does not flush cached file data/metadata at the end of migration. + The file migration also supports using a file that has already been + opened. A set of file descriptors is passed to QEMU via an "fdset" + (see add-fd QMP command documentation). This method allows a + management application to have control over the migration file + opening operation. There are, however, strict requirements to this + interface: + + On the migration source side: + - if the multifd capability is to be used, the fdset must contain + two file descriptors that are not duplicates between themselves; + - if the direct-io capability is to be used, exactly one of the + file descriptors must have the O_DIRECT flag set; + - the file must be opened with WRONLY. + + On the migration destination side: + - the fdset must contain one file descriptor; + - the file must be opened with RDONLY. + In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of transporting the pages, and the load on the CPU is much lower. While thedocs/devel/migration/mapped-ram.rst +5
-1@@ -16,7 +16,7 @@ location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment -restrictions (direct-io support not yet implemented). +restrictions. Usage -----@@ -35,6 +35,10 @@ Use a ``file:`` URL for migration: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating. +For best performance enable the ``direct-io`` capability as well: + + ``migrate_set_capability direct-io on`` + Use-cases ---------migration/file.c +27
-0@@ -17,6 +17,7 @@#include "io/channel-file.h" #include "io/channel-socket.h" #include "io/channel-util.h" +#include "monitor/monitor.h" #include "options.h" #include "trace.h" @@ -54,10 +55,18 @@ static void file_remove_fdset(void) } } +/* +* With multifd, due to the behavior of the dup() system call, we need +* the fdset to have two non-duplicate fds so we can enable direct IO +* in the secondary channels without affecting the main channel. +*/ static bool file_parse_fdset (const char *filename,int64_t *fdset_id,Error **errp) { +FdsetInfoList *fds_info; +FdsetFdInfoList *fd_info;const char *fdset_id_str; +int nfds =0 ; *fdset_id =-1 ;@@ -71,6 +80,32 @@ static bool file_parse_fdset(const char *filename, int64_t *fdset_id,return false; } +if (!migrate_multifd () || !migrate_direct_io ()) { +return true; + } + +for (fds_info =qmp_query_fdsets (NULL ); fds_info; + fds_info = fds_info->next) { + +if (*fdset_id != fds_info->value->fdset_id) { +continue ; + } + +for (fd_info = fds_info->value->fds; fd_info; fd_info = fd_info->next) { +if (nfds++ >2 ) { +break ; + } + } + } +migration/file.c static bool file_parse_fdset(const char *filename, int64_t *fdset_id, · line 110
Quoted 10 lines
migration/file.c +8
-0+if (nfds !=2 ) { +error_setg (errp,"Outgoing migration needs two fds in the fdset, " +"got %d" , nfds); +qmp_remove_fd (*fdset_id, false,-1 ,NULL ); + *fdset_id =-1 ; +return false; + } +return true; }Related to my thoughts in an earlier patch, where I say that use of fdsets ought to be transparent to QEMU code, I'm not a fan of having this logic in migration code.
Quoted 5 lines
IIUC, the migration code will call
qio_channel_file_new_pathtwice, once withO_DIRECTand once without. This should trigger two calls intomonitor_fdset_dup_fd_addwith different flags. If we're matching flags in thatmonitor_fdset_dup_fd_add(), then if only 1 FD was provided, are we not able to report an error there ?
Right, this sounds working.
For a real sanity check, we may want to somehow check the two fds returned from qio_channel_file_new_path() to point to the same file underneath.
What mentioned in the other thread (kcmp with KCMP_FILE) might not work, as the whole purpose of having two fds is to make sure they have different struct file to back the fd (and only one of them has O_DIRECT). fstat() might work in this case over the st_ino field, etc. maybe fstatfs() too but perhaps that's over cautious. Just a pain to use two fds as a start..
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 8/9] migration: Add support for fdset with multifd + file
Quoted 155 lines · Peter Xu
Quoted 154 lines · Daniel P. Berrangé
Quoted 138 lines · Fabiano Rosas
Allow multifd to use an fdset when migrating to a file. This is useful for the scenario where the management layer wants to have control over the migration file.
By receiving the file descriptors directly, QEMU can delegate some high level operating system operations to the management layer (such as mandatory access control). The management layer might also want to add its own headers before the migration stream.
Enable the "file:/dev/fdset/#" syntax for the multifd migration with mapped-ram. The requirements for the fdset mechanism are:
On the migration source side:
- the fdset must contain two fds that are not duplicates between themselves;
- if direct-io is to be used, exactly one of the fds must have theO_DIRECTflag set;
- the file must be opened with WRONLY both times.On the migration destination side:
- the fdset must contain one fd;
- the file must be opened with RDONLY.Signed-off-by Fabiano Rosas <[email protected]>--- docs/devel/migration/main.rst | 18 ++++++++++++++ docs/devel/migration/mapped-ram.rst | 6 ++++- migration/file.c | 38 ++++++++++++++++++++++++++++- 3 files changed, 60 insertions(+), 2 deletions(-)docs/devel/migration/main.rst +18
-0@@ -47,6 +47,24 @@ over any transport. QEMU interference. Note that QEMU does not flush cached file data/metadata at the end of migration. + The file migration also supports using a file that has already been + opened. A set of file descriptors is passed to QEMU via an "fdset" + (see add-fd QMP command documentation). This method allows a + management application to have control over the migration file + opening operation. There are, however, strict requirements to this + interface: + + On the migration source side: + - if the multifd capability is to be used, the fdset must contain + two file descriptors that are not duplicates between themselves; + - if the direct-io capability is to be used, exactly one of the + file descriptors must have the O_DIRECT flag set; + - the file must be opened with WRONLY. + + On the migration destination side: + - the fdset must contain one file descriptor; + - the file must be opened with RDONLY. + In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of transporting the pages, and the load on the CPU is much lower. While thedocs/devel/migration/mapped-ram.rst +5
-1@@ -16,7 +16,7 @@ location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment -restrictions (direct-io support not yet implemented). +restrictions. Usage -----@@ -35,6 +35,10 @@ Use a ``file:`` URL for migration: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating. +For best performance enable the ``direct-io`` capability as well: + + ``migrate_set_capability direct-io on`` + Use-cases ---------migration/file.c +27
-0@@ -17,6 +17,7 @@#include "io/channel-file.h" #include "io/channel-socket.h" #include "io/channel-util.h" +#include "monitor/monitor.h" #include "options.h" #include "trace.h" @@ -54,10 +55,18 @@ static void file_remove_fdset(void) } } +/* +* With multifd, due to the behavior of the dup() system call, we need +* the fdset to have two non-duplicate fds so we can enable direct IO +* in the secondary channels without affecting the main channel. +*/ static bool file_parse_fdset (const char *filename,int64_t *fdset_id,Error **errp) { +FdsetInfoList *fds_info; +FdsetFdInfoList *fd_info;const char *fdset_id_str; +int nfds =0 ; *fdset_id =-1 ;@@ -71,6 +80,32 @@ static bool file_parse_fdset(const char *filename, int64_t *fdset_id,return false; } +if (!migrate_multifd () || !migrate_direct_io ()) { +return true; + } + +for (fds_info =qmp_query_fdsets (NULL ); fds_info; + fds_info = fds_info->next) { + +if (*fdset_id != fds_info->value->fdset_id) { +continue ; + } + +for (fd_info = fds_info->value->fds; fd_info; fd_info = fd_info->next) { +if (nfds++ >2 ) { +break ; + } + } + } +migration/file.c static bool file_parse_fdset(const char *filename, int64_t *fdset_id, · line 110
Quoted 10 lines
migration/file.c +8
-0+if (nfds !=2 ) { +error_setg (errp,"Outgoing migration needs two fds in the fdset, " +"got %d" , nfds); +qmp_remove_fd (*fdset_id, false,-1 ,NULL ); + *fdset_id =-1 ; +return false; + } +return true; }Related to my thoughts in an earlier patch, where I say that use of fdsets ought to be transparent to QEMU code, I'm not a fan of having this logic in migration code.
Quoted 7 lines
Quoted 5 lines
IIUC, the migration code will call
qio_channel_file_new_pathtwice, once withO_DIRECTand once without. This should trigger two calls intomonitor_fdset_dup_fd_addwith different flags. If we're matching flags in thatmonitor_fdset_dup_fd_add(), then if only 1 FD was provided, are we not able to report an error there ?Right, this sounds working.
It works, but due to how low-level fdset is, it's difficult to match the low level error to anything meaningful we can report to the user. I'll have to add an errp to monitor_fdset_dup_fd_add(). Its returns are not very useful.
-1 with no errno
-1 with EACCES (should actually be EBADF)
-1 with ENOENT
There has been some discusstion around this before actually:
https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg02544.html
Or, you know, let the management layer figure it out. We seem to be heading in this direction already. I imagine once the code is written to interact with QEMU, it would not have any reason to change, so it might be ok to replace some of the code I'm adding in this series with documentation and call it a day. I don't like this approach very much, but it would definitely make this series way simpler.
Quoted 11 lines
For a real sanity check, we may want to somehow check the two fds returned from
qio_channel_file_new_path()to point to the same file underneath.What mentioned in the other thread (kcmp with
KCMP_FILE) might not work, as the whole purpose of having two fds is to make sure they have different struct file to back the fd (and only one of them hasO_DIRECT).fstat()might work in this case over thest_inofield, etc. maybefstatfs()too but perhaps that's over cautious. Just a pain to use two fds as a start..Thanks,
Re: [PATCH 8/9] migration: Add support for fdset with multifd + file
Quoted 170 lines · Fabiano Rosas
Quoted 155 lines · Peter Xu
Quoted 154 lines · Daniel P. Berrangé
Quoted 138 lines · Fabiano Rosas
Allow multifd to use an fdset when migrating to a file. This is useful for the scenario where the management layer wants to have control over the migration file.
By receiving the file descriptors directly, QEMU can delegate some high level operating system operations to the management layer (such as mandatory access control). The management layer might also want to add its own headers before the migration stream.
Enable the "file:/dev/fdset/#" syntax for the multifd migration with mapped-ram. The requirements for the fdset mechanism are:
On the migration source side:
- the fdset must contain two fds that are not duplicates between themselves;
- if direct-io is to be used, exactly one of the fds must have theO_DIRECTflag set;
- the file must be opened with WRONLY both times.On the migration destination side:
- the fdset must contain one fd;
- the file must be opened with RDONLY.Signed-off-by Fabiano Rosas <[email protected]>--- docs/devel/migration/main.rst | 18 ++++++++++++++ docs/devel/migration/mapped-ram.rst | 6 ++++- migration/file.c | 38 ++++++++++++++++++++++++++++- 3 files changed, 60 insertions(+), 2 deletions(-)docs/devel/migration/main.rst +18
-0@@ -47,6 +47,24 @@ over any transport. QEMU interference. Note that QEMU does not flush cached file data/metadata at the end of migration. + The file migration also supports using a file that has already been + opened. A set of file descriptors is passed to QEMU via an "fdset" + (see add-fd QMP command documentation). This method allows a + management application to have control over the migration file + opening operation. There are, however, strict requirements to this + interface: + + On the migration source side: + - if the multifd capability is to be used, the fdset must contain + two file descriptors that are not duplicates between themselves; + - if the direct-io capability is to be used, exactly one of the + file descriptors must have the O_DIRECT flag set; + - the file must be opened with WRONLY. + + On the migration destination side: + - the fdset must contain one file descriptor; + - the file must be opened with RDONLY. + In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of transporting the pages, and the load on the CPU is much lower. While thedocs/devel/migration/mapped-ram.rst +5
-1@@ -16,7 +16,7 @@ location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment -restrictions (direct-io support not yet implemented). +restrictions. Usage -----@@ -35,6 +35,10 @@ Use a ``file:`` URL for migration: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating. +For best performance enable the ``direct-io`` capability as well: + + ``migrate_set_capability direct-io on`` + Use-cases ---------migration/file.c +27
-0@@ -17,6 +17,7 @@#include "io/channel-file.h" #include "io/channel-socket.h" #include "io/channel-util.h" +#include "monitor/monitor.h" #include "options.h" #include "trace.h" @@ -54,10 +55,18 @@ static void file_remove_fdset(void) } } +/* +* With multifd, due to the behavior of the dup() system call, we need +* the fdset to have two non-duplicate fds so we can enable direct IO +* in the secondary channels without affecting the main channel. +*/ static bool file_parse_fdset (const char *filename,int64_t *fdset_id,Error **errp) { +FdsetInfoList *fds_info; +FdsetFdInfoList *fd_info;const char *fdset_id_str; +int nfds =0 ; *fdset_id =-1 ;@@ -71,6 +80,32 @@ static bool file_parse_fdset(const char *filename, int64_t *fdset_id,return false; } +if (!migrate_multifd () || !migrate_direct_io ()) { +return true; + } + +for (fds_info =qmp_query_fdsets (NULL ); fds_info; + fds_info = fds_info->next) { + +if (*fdset_id != fds_info->value->fdset_id) { +continue ; + } + +for (fd_info = fds_info->value->fds; fd_info; fd_info = fd_info->next) { +if (nfds++ >2 ) { +break ; + } + } + } +migration/file.c static bool file_parse_fdset(const char *filename, int64_t *fdset_id, · line 110
Quoted 10 lines
migration/file.c +8
-0+if (nfds !=2 ) { +error_setg (errp,"Outgoing migration needs two fds in the fdset, " +"got %d" , nfds); +qmp_remove_fd (*fdset_id, false,-1 ,NULL ); + *fdset_id =-1 ; +return false; + } +return true; }Related to my thoughts in an earlier patch, where I say that use of fdsets ought to be transparent to QEMU code, I'm not a fan of having this logic in migration code.
Quoted 7 lines
Quoted 5 lines
IIUC, the migration code will call
qio_channel_file_new_pathtwice, once withO_DIRECTand once without. This should trigger two calls intomonitor_fdset_dup_fd_addwith different flags. If we're matching flags in thatmonitor_fdset_dup_fd_add(), then if only 1 FD was provided, are we not able to report an error there ?Right, this sounds working.
It works, but due to how low-level fdset is, it's difficult to match the low level error to anything meaningful we can report to the user. I'll have to add an errp to
monitor_fdset_dup_fd_add(). Its returns are not very useful.
Quoted 7 lines
-1 with no errno
-1 with EACCES (should actually be EBADF)
-1 with ENOENTThere has been some discusstion around this before actually:
https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg02544.html
The only caller of monitor_fdset_dup_fd_add is qemu_open_internal and that has a "Error **errp" parameter. We should rewrite monitor_fdset_dup_fd_add to also have an "Error **errp" at which point we can actually report useful, actionable error messages from it. Errnos be gone !
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 8/9] migration: Add support for fdset with multifd + file
Quoted 177 lines · Daniel P. Berrangé
Quoted 176 lines · Fabiano Rosas
Quoted 155 lines · Peter Xu
Quoted 154 lines · Daniel P. Berrangé
Quoted 138 lines · Fabiano Rosas
Allow multifd to use an fdset when migrating to a file. This is useful for the scenario where the management layer wants to have control over the migration file.
By receiving the file descriptors directly, QEMU can delegate some high level operating system operations to the management layer (such as mandatory access control). The management layer might also want to add its own headers before the migration stream.
Enable the "file:/dev/fdset/#" syntax for the multifd migration with mapped-ram. The requirements for the fdset mechanism are:
On the migration source side:
- the fdset must contain two fds that are not duplicates between themselves;
- if direct-io is to be used, exactly one of the fds must have theO_DIRECTflag set;
- the file must be opened with WRONLY both times.On the migration destination side:
- the fdset must contain one fd;
- the file must be opened with RDONLY.Signed-off-by Fabiano Rosas <[email protected]>--- docs/devel/migration/main.rst | 18 ++++++++++++++ docs/devel/migration/mapped-ram.rst | 6 ++++- migration/file.c | 38 ++++++++++++++++++++++++++++- 3 files changed, 60 insertions(+), 2 deletions(-)docs/devel/migration/main.rst +18
-0@@ -47,6 +47,24 @@ over any transport. QEMU interference. Note that QEMU does not flush cached file data/metadata at the end of migration. + The file migration also supports using a file that has already been + opened. A set of file descriptors is passed to QEMU via an "fdset" + (see add-fd QMP command documentation). This method allows a + management application to have control over the migration file + opening operation. There are, however, strict requirements to this + interface: + + On the migration source side: + - if the multifd capability is to be used, the fdset must contain + two file descriptors that are not duplicates between themselves; + - if the direct-io capability is to be used, exactly one of the + file descriptors must have the O_DIRECT flag set; + - the file must be opened with WRONLY. + + On the migration destination side: + - the fdset must contain one file descriptor; + - the file must be opened with RDONLY. + In addition, support is included for migration using RDMA, which transports the page data using ``RDMA``, where the hardware takes care of transporting the pages, and the load on the CPU is much lower. While thedocs/devel/migration/mapped-ram.rst +5
-1@@ -16,7 +16,7 @@ location in the file, rather than constantly being added to a sequential stream. Having the pages at fixed offsets also allows the usage of O_DIRECT for save/restore of the migration stream as the pages are ensured to be written respecting O_DIRECT alignment -restrictions (direct-io support not yet implemented). +restrictions. Usage -----@@ -35,6 +35,10 @@ Use a ``file:`` URL for migration: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating. +For best performance enable the ``direct-io`` capability as well: + + ``migrate_set_capability direct-io on`` + Use-cases ---------migration/file.c +27
-0@@ -17,6 +17,7 @@#include "io/channel-file.h" #include "io/channel-socket.h" #include "io/channel-util.h" +#include "monitor/monitor.h" #include "options.h" #include "trace.h" @@ -54,10 +55,18 @@ static void file_remove_fdset(void) } } +/* +* With multifd, due to the behavior of the dup() system call, we need +* the fdset to have two non-duplicate fds so we can enable direct IO +* in the secondary channels without affecting the main channel. +*/ static bool file_parse_fdset (const char *filename,int64_t *fdset_id,Error **errp) { +FdsetInfoList *fds_info; +FdsetFdInfoList *fd_info;const char *fdset_id_str; +int nfds =0 ; *fdset_id =-1 ;@@ -71,6 +80,32 @@ static bool file_parse_fdset(const char *filename, int64_t *fdset_id,return false; } +if (!migrate_multifd () || !migrate_direct_io ()) { +return true; + } + +for (fds_info =qmp_query_fdsets (NULL ); fds_info; + fds_info = fds_info->next) { + +if (*fdset_id != fds_info->value->fdset_id) { +continue ; + } + +for (fd_info = fds_info->value->fds; fd_info; fd_info = fd_info->next) { +if (nfds++ >2 ) { +break ; + } + } + } +migration/file.c static bool file_parse_fdset(const char *filename, int64_t *fdset_id, · line 110
Quoted 10 lines
migration/file.c +8
-0+if (nfds !=2 ) { +error_setg (errp,"Outgoing migration needs two fds in the fdset, " +"got %d" , nfds); +qmp_remove_fd (*fdset_id, false,-1 ,NULL ); + *fdset_id =-1 ; +return false; + } +return true; }Related to my thoughts in an earlier patch, where I say that use of fdsets ought to be transparent to QEMU code, I'm not a fan of having this logic in migration code.
Quoted 7 lines
Quoted 5 lines
IIUC, the migration code will call
qio_channel_file_new_pathtwice, once withO_DIRECTand once without. This should trigger two calls intomonitor_fdset_dup_fd_addwith different flags. If we're matching flags in thatmonitor_fdset_dup_fd_add(), then if only 1 FD was provided, are we not able to report an error there ?Right, this sounds working.
It works, but due to how low-level fdset is, it's difficult to match the low level error to anything meaningful we can report to the user. I'll have to add an errp to
monitor_fdset_dup_fd_add(). Its returns are not very useful.-1 with no errno
-1 with EACCES (should actually be EBADF)
-1 with ENOENTThere has been some discusstion around this before actually:
Quoted 7 lines
The only caller of
monitor_fdset_dup_fd_addisqemu_open_internaland that has a "Error **errp" parameter. We should rewritemonitor_fdset_dup_fd_addto also have an "Error **errp" at which point we can actually report useful, actionable error messages from it. Errnos be gone !
I can do that, but qemu_open_old() does not pass the error in. Please see if this works for you:
-->8--
From 16e333cc5aeca1fab3f75f79048c0ab0d62d5b08 Mon Sep 17 00:00:00 2001
From: Fabiano Rosas <[email protected]>
Date: Fri, 17 May 2024 19:30:39 -0300
Subject: [PATCH] io: Stop using qemu_open_old in channel-file
We want to make use of the Error object to report fdset errors from qemu_open_internal() and passing the error pointer to qemu_open_old() would require changing all callers. Move the file channel to the new API instead.
--- io/channel-file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
io/channel-file.c +5 -3
@@ -68,11 +68,13 @@ qio_channel_file_new_path(const char *path, ioc =QIO_CHANNEL_FILE (object_new (TYPE_QIO_CHANNEL_FILE )); - ioc->fd =qemu_open_old (path, flags, mode); +if (flags &O_CREAT ) { + ioc->fd =qemu_create (path, flags & ~O_CREAT , mode, errp); + }else { + ioc->fd =qemu_open (path, flags, errp); + }if (ioc->fd <0 ) {object_unref (OBJECT (ioc)); -error_setg_errno (errp, errno, -"Unable to open %s" , path);return NULL ; }
Signature · 1 line
2.35.3
Daniel P. Berrangé
Reviewed-byRe: [PATCH 8/9] migration: Add support for fdset with multifd + file
Quoted 38 lines · Fabiano Rosas
Quoted 6 lines · Daniel P. Berrangé
The only caller of
monitor_fdset_dup_fd_addisqemu_open_internaland that has a "Error **errp" parameter. We should rewritemonitor_fdset_dup_fd_addto also have an "Error **errp" at which point we can actually report useful, actionable error messages from it. Errnos be gone !I can do that, but
qemu_open_old()does not pass the error in. Please see if this works for you:-->8--
From16e333cc5aeca1fab3f75f79048c0ab0d62d5b08Mon Sep 17 00:00:00 2001
From: Fabiano Rosas <[email protected]>
Date: Fri, 17 May 2024 19:30:39 -0300
Subject: [PATCH] io: Stop usingqemu_open_oldin channel-fileWe want to make use of the Error object to report fdset errors from
qemu_open_internal()and passing the error pointer toqemu_open_old()would require changing all callers. Move the file channel to the new API instead.Signed-off-by Fabiano Rosas <[email protected]>--- io/channel-file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)io/channel-file.c +1
-1@@ -68,11 +68,13 @@ qio_channel_file_new_path(const char *path, ioc =QIO_CHANNEL_FILE (object_new (TYPE_QIO_CHANNEL_FILE )); - ioc->fd =qemu_open_old (path, flags, mode); +if (flags &O_CREAT ) {
io/channel-file.c qio_channel_file_new_path(const char *path, · line 79
Quoted 10 lines
io/channel-file.c +4
-2+ ioc->fd =qemu_create (path, flags & ~O_CREAT , mode, errp); + }else { + ioc->fd =qemu_open (path, flags, errp); + }if (ioc->fd <0 ) {object_unref (OBJECT (ioc)); -error_setg_errno (errp, errno, -"Unable to open %s" , path);return NULL ; }
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH 9/9 tests/qtest/migration: Add a test for mapped-ram with passing of fds
In reply to Fabiano RosasAdd a multifd test for mapped-ram with passing of fds into QEMU. This is how libvirt will consume the feature.
There are a couple of details to the fdset mechanism:
- multifd needs two distinct file descriptors (not duplicated with dup()) on the outgoing side so it can enable O_DIRECT only on the channels that write with alignment. The dup() system call creates file descriptors that share status flags, of which O_DIRECT is one.
the incoming side doesn't set O_DIRECT, so it can dup() fds and therefore can receive only one in the fdset.
- the open() access mode flags used for the fds passed into QEMU need to match the flags QEMU uses to open the file. Currently O_WRONLY for src and O_RDONLY for dst.
O_DIRECT is not supported on all systems/filesystems, so run the fdset test without O_DIRECT if that's the case. The migration code should still work in that scenario.
--- tests/qtest/migration-test.c | 90 ++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+)
tests/qtest/migration-test.c +90 -0
@@ -2331,8 +2331,93 @@ static void test_multifd_file_mapped_ram_dio(void)test_file_common (&args, true); } +static void migrate_multifd_mapped_ram_fdset_dio_end (QTestState *from, +QTestState *to, +void *opaque) +{ +QDict *resp; +QList *fdsets; + +file_offset_finish_hook (from, to, opaque); + +/* +* Check that we removed the fdsets after migration, otherwise a +* second migration would fail due to too many fdsets. +*/ + + resp =qtest_qmp (from,"{'execute': 'query-fdsets', " +"'arguments': {}}" ); +g_assert (qdict_haskey (resp,"return" )); + fdsets =qdict_get_qlist (resp,"return" ); +g_assert (fdsets &&qlist_empty (fdsets)); +}#endif /* O_DIRECT */ +#ifndef _WIN32 +static void *migrate_multifd_mapped_ram_fdset (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +int fds[3 ]; +int src_flags =O_WRONLY ; + +file_dirty_offset_region (); + +/* main outgoing channel: no O_DIRECT */ + fds[0 ] =open (file, src_flags,0660 ); +assert (fds[0 ] !=-1 ); + +qtest_qmp_fds_assert_success (from, &fds[0 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +#ifdef O_DIRECT + src_flags |=O_DIRECT ; + +/* secondary outgoing channels */ + fds[1 ] =open (file, src_flags,0660 ); +assert (fds[1 ] !=-1 ); + +qtest_qmp_fds_assert_success (from, &fds[1 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +/* incoming channel */ + fds[2 ] =open (file,O_CREAT |O_RDONLY ,0660 ); +assert (fds[2 ] !=-1 ); + +qtest_qmp_fds_assert_success (to, &fds[2 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +migrate_multifd_mapped_ram_dio_start (from, to); +#else +migrate_multifd_mapped_ram_start (from, to); +#endif + +return NULL ; +} + +static void test_multifd_file_mapped_ram_fdset (void ) +{ + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d", +FILE_TEST_OFFSET ); +MigrateCommon args = { + .connect_uri = uri, + .listen_uri ="defer" , + .start_hook = migrate_multifd_mapped_ram_fdset, +#ifdef O_DIRECT + .finish_hook = migrate_multifd_mapped_ram_fdset_dio_end, +#endif + }; + +#ifdef O_DIRECT +if (!probe_o_direct_support (tmpfs)) { +g_test_skip ("Filesystem does not support O_DIRECT" ); +return ; + } +#endif + +test_file_common (&args, true); +} +#endif /* _WIN32 */ +static void test_precopy_tcp_plain (void ) {MigrateCommon args = {
@@ -3761,6 +3846,11 @@ int main(int argc, char **argv) test_multifd_file_mapped_ram_dio);#endif +#ifndef _WIN32 +qtest_add_func ("/migration/multifd/file/mapped-ram/fdset" , + test_multifd_file_mapped_ram_fdset); +#endif +#ifdef CONFIG_GNUTLS migration_test_add ("/migration/precopy/unix/tls/psk" , test_precopy_unix_tls_psk);
Signature · 1 line
2.35.3
Re: [PATCH 9/9] tests/qtest/migration: Add a test for mapped-ram with passing of fds
Quoted 13 lines · Fabiano Rosas
Add a multifd test for mapped-ram with passing of fds into QEMU. This is how libvirt will consume the feature.
There are a couple of details to the fdset mechanism:
- multifd needs two distinct file descriptors (not duplicated with
dup()) on the outgoing side so it can enableO_DIRECTonly on the channels that write with alignment. Thedup()system call creates file descriptors that share status flags, of whichO_DIRECTis one.the incoming side doesn't set
O_DIRECT, so it candup()fds and therefore can receive only one in the fdset.
Quoted 7 lines
- the
open()access mode flags used for the fds passed into QEMU need to match the flags QEMU uses to open the file. CurrentlyO_WRONLYfor src andO_RDONLYfor dst.
O_DIRECTis not supported on all systems/filesystems, so run the fdset test withoutO_DIRECTif that's the case. The migration code should still work in that scenario.
If O_DIRECT is not supported, then we're not setting 'direct-io', and thus isn't this test just duplicating coverage of existing tests ?
If this test is specifically to cover O_DIRECT, then I'd just #ifdef the entire thing with O_DIRECT.
Quoted 119 lines
Signed-off-by Fabiano Rosas <[email protected]>--- tests/qtest/migration-test.c | 90 ++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+)tests/qtest/migration-test.c +90
-0@@ -2331,8 +2331,93 @@ static void test_multifd_file_mapped_ram_dio(void)test_file_common (&args, true); } +static void migrate_multifd_mapped_ram_fdset_dio_end (QTestState *from, +QTestState *to, +void *opaque) +{ +QDict *resp; +QList *fdsets; + +file_offset_finish_hook (from, to, opaque); + +/* +* Check that we removed the fdsets after migration, otherwise a +* second migration would fail due to too many fdsets. +*/ + + resp =qtest_qmp (from,"{'execute': 'query-fdsets', " +"'arguments': {}}" ); +g_assert (qdict_haskey (resp,"return" )); + fdsets =qdict_get_qlist (resp,"return" ); +g_assert (fdsets &&qlist_empty (fdsets)); +}#endif /* O_DIRECT */ +#ifndef _WIN32 +static void *migrate_multifd_mapped_ram_fdset (QTestState *from,QTestState *to) +{ + g_autofree char *file = g_strdup_printf("%s/%s", tmpfs, FILE_TEST_FILENAME); +int fds[3 ]; +int src_flags =O_WRONLY ; + +file_dirty_offset_region (); + +/* main outgoing channel: no O_DIRECT */ + fds[0 ] =open (file, src_flags,0660 ); +assert (fds[0 ] !=-1 ); + +qtest_qmp_fds_assert_success (from, &fds[0 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +#ifdef O_DIRECT + src_flags |=O_DIRECT ; + +/* secondary outgoing channels */ + fds[1 ] =open (file, src_flags,0660 ); +assert (fds[1 ] !=-1 ); + +qtest_qmp_fds_assert_success (from, &fds[1 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +/* incoming channel */ + fds[2 ] =open (file,O_CREAT |O_RDONLY ,0660 ); +assert (fds[2 ] !=-1 ); + +qtest_qmp_fds_assert_success (to, &fds[2 ],1 ,"{'execute': 'add-fd', " +"'arguments': {'fdset-id': 1}}" ); + +migrate_multifd_mapped_ram_dio_start (from, to); +#else +migrate_multifd_mapped_ram_start (from, to); +#endif + +return NULL ; +} + +static void test_multifd_file_mapped_ram_fdset (void ) +{ + g_autofree char *uri = g_strdup_printf("file:/dev/fdset/1,offset=%d", +FILE_TEST_OFFSET ); +MigrateCommon args = { + .connect_uri = uri, + .listen_uri ="defer" , + .start_hook = migrate_multifd_mapped_ram_fdset, +#ifdef O_DIRECT + .finish_hook = migrate_multifd_mapped_ram_fdset_dio_end, +#endif + }; + +#ifdef O_DIRECT +if (!probe_o_direct_support (tmpfs)) { +g_test_skip ("Filesystem does not support O_DIRECT" ); +return ; + } +#endif + +test_file_common (&args, true); +} +#endif /* _WIN32 */ +static void test_precopy_tcp_plain (void ) {MigrateCommon args = {@@ -3761,6 +3846,11 @@ int main(int argc, char **argv) test_multifd_file_mapped_ram_dio);#endif +#ifndef _WIN32 +qtest_add_func ("/migration/multifd/file/mapped-ram/fdset" , + test_multifd_file_mapped_ram_fdset); +#endif +#ifdef CONFIG_GNUTLS migration_test_add ("/migration/precopy/unix/tls/psk" , test_precopy_unix_tls_psk);Signature · 1 line
2.35.3
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 4 lines · Fabiano Rosas
If the user is not passing in a file name which QEMU can open at will, we must then require that the user pass the two file descriptors with the flags already properly set. We'll use the already existing fdset + QMP add-fd infrastructure for this.
Yes I remember such requirement that one extra fd is needed for direct-io, however today when I looked closer at the man page it looks like F_SETFL works with O_DIRECT too?
F_SETFL (int)
Set the file status flags to the value specified by arg. File access mode (O_RDONLY, O_WRONLY, O_RDWR) and file creation flags (i.e., O_CREAT, O_EXCL, O_NOCTTY, O_TRUNC) in arg are ignored. On Linux, this command can change only the O_APPEND, O_ASYNC, O_DIRECT, O_NOATIME, and O_NONBLOCK flags. It is not possible to change the O_DSYNC and O_SYNC flags; see BUGS, below.
====8<====
$ cat fcntl.c #define _GNU_SOURCE #include <stdio.h> #include <fcntl.h> #include <assert.h> #include <unistd.h>
int main(void)
{
int fd, newfd, ret, flags;
fd = open("test.txt", O_RDWR | O_CREAT, 0660);
assert(fd != -1);
flags = fcntl(fd, F_GETFL);
printf("old fd flags: 0x%x\n", flags);
newfd = dup(fd);
assert(newfd != -1);
flags = fcntl(newfd, F_GETFL);
printf("new fd flags: 0x%x\n", flags);
flags |= O_DIRECT;
ret = fcntl(newfd, F_SETFL, flags);
flags = fcntl(fd, F_GETFL);
printf("updated new flags: 0x%x\n", flags);
return 0;
}
$ make fcntl
cc fcntl.c -o fcntl
$ ./fcntl
old fd flags: 0x8002
new fd flags: 0x8002
updated new flags: 0xc002
====8<====Perhaps I missed something important?
Signature · 1 line
Peter Xu
Quoted 59 lines · Peter Xu
Quoted 4 lines · Fabiano Rosas
If the user is not passing in a file name which QEMU can open at will, we must then require that the user pass the two file descriptors with the flags already properly set. We'll use the already existing fdset + QMP add-fd infrastructure for this.
Yes I remember such requirement that one extra fd is needed for direct-io, however today when I looked closer at the man page it looks like
F_SETFLworks withO_DIRECTtoo?
F_SETFL(int)
Set the file status flags to the value specified by arg. File access mode (O_RDONLY,O_WRONLY,O_RDWR) and file creation flags (i.e.,O_CREAT,O_EXCL,O_NOCTTY,O_TRUNC) in arg are ignored. On Linux, this command can change only theO_APPEND,O_ASYNC,O_DIRECT,O_NOATIME, andO_NONBLOCKflags. It is not possible to change theO_DSYNCandO_SYNCflags; see BUGS, below.====8<====
$ cat fcntl.c #define _GNU_SOURCE #include <stdio.h> #include <fcntl.h> #include <assert.h> #include <unistd.h>int main(void)
{ int fd, newfd, ret, flags; fd = open("test.txt", O_RDWR | O_CREAT, 0660); assert(fd != -1); flags = fcntl(fd, F_GETFL); printf("old fd flags: 0x%x\n", flags); newfd = dup(fd); assert(newfd != -1); flags = fcntl(newfd, F_GETFL); printf("new fd flags: 0x%x\n", flags); flags |= O_DIRECT; ret = fcntl(newfd, F_SETFL, flags); flags = fcntl(fd, F_GETFL); printf("updated new flags: 0x%x\n", flags); return 0; } $ make fcntl cc fcntl.c -o fcntl $ ./fcntl old fd flags: 0x8002 new fd flags: 0x8002 updated new flags: 0xc002 ====8<====
Quoted 1 line
Perhaps I missed something important?
The dup()'ed file descriptor shares file status flags with the original fd. Your code example proves just that. In the last two blocks you're doing F_SETFL on the 'newfd' and then seeing the change take effect on 'fd'. That's what we don't want to happen.