Skip to contentSkip to conversation

migrate_fd_close: delete associated io-handler before closing the fd

PATCH In review Uri Lublin 2 participants 5 messages Jun 8, 2009

lore
Share & tools
lore

Uri Lublin

author
#0 lore raw

PATCH 1/1 migrate_fd_close: delete associated io-handler before closing the fd

It may happen that the io-handler is still registered. That causes select() to return with EBADF, not calling handlers for other fds.

The io-handler would be registered when (on the source) the whole state was written but not yet flushed. For example when using QEMUFileBuffered, (tcp-migration) there may be data left in a buffer waiting to be transferred. In such a case buffered_close() calls buffered_flush() which calls migrate_fd_put_buffer, which may, upon EAGAIN, register migrate_fd_put_notify as a handler.

---
 migration.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
migration.c +2 -0
index 401383c..57f2a52 100644
@@ -301,5 +301,7 @@ void migrate_fd_wait_for_unfreeze(void *opaque)
 int migrate_fd_close(void *opaque)
 {
     FdMigrationState *s = opaque;
+
+    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
     return s->close(s);
 }
Signature · 1 line
1.6.2.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 20 lines · Uri Lublin

It may happen that the io-handler is still registered. That causes select() to return with EBADF, not calling handlers for other fds.

The io-handler would be registered when (on the source) the whole state was written but not yet flushed. For example when using QEMUFileBuffered, (tcp-migration) there may be data left in a buffer waiting to be transferred. In such a case buffered_close() calls buffered_flush() which calls migrate_fd_put_buffer, which may, upon EAGAIN, register migrate_fd_put_notify as a handler.

---
 migration.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
migration.c +0 -0
index 401383c..57f2a52 100644

migration.c void migrate_fd_wait_for_unfreeze(void *opaque) · line 305

Quoted 5 lines
migration.c +2 -0
@@ -301,5 +301,7 @@ void migrate_fd_wait_for_unfreeze(void *opaque)
 int migrate_fd_close(void *opaque)
 {
     FdMigrationState *s = opaque;
+
+    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);

Looks good, but perhaps a comment explaining how the I/O handler could possibly be registered here would be useful - at first glance, it seemed to me that the I/O handler should always be de-registered in migrate_fd_cleanup() before getting here.

The key to understanding the problem is that qemu_fclose() on a buffered file can cause I/O to be flushed.

Cheers,
Mark.

author
#2 lore raw
Quoted 32 lines · Mark McLoughlin
Quoted 20 lines · Uri Lublin

It may happen that the io-handler is still registered. That causes select() to return with EBADF, not calling handlers for other fds.

The io-handler would be registered when (on the source) the whole state was written but not yet flushed. For example when using QEMUFileBuffered, (tcp-migration) there may be data left in a buffer waiting to be transferred. In such a case buffered_close() calls buffered_flush() which calls migrate_fd_put_buffer, which may, upon EAGAIN, register migrate_fd_put_notify as a handler.

---
  migration.c |    2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)
migration.c +0 -0
index 401383c..57f2a52 100644

migration.c void migrate_fd_wait_for_unfreeze(void *opaque) · line 305

Quoted 5 lines
migration.c +2 -0
@@ -301,5 +301,7 @@ void migrate_fd_wait_for_unfreeze(void *opaque)
  int migrate_fd_close(void *opaque)
  {
      FdMigrationState *s = opaque;
+
+    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);

Looks good, but perhaps a comment explaining how the I/O handler could possibly be registered here would be useful - at first glance, it seemed to me that the I/O handler should always be de-registered in migrate_fd_cleanup() before getting here.

Quoted 2 lines

The key to understanding the problem is that qemu_fclose() on a buffered file can cause I/O to be flushed.

Do you mean in addition to the log-message (copy part of the log message as a comment in the code) ?

Quoted 2 lines

Thanks for the review,
Uri.

#3 lore raw
Quoted 8 lines · Uri Lublin
Quoted 8 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);

Looks good, but perhaps a comment explaining how the I/O handler could possibly be registered here would be useful - at first glance, it seemed to me that the I/O handler should always be de-registered in migrate_fd_cleanup() before getting here.

Quoted 5 lines
Quoted 2 lines

The key to understanding the problem is that qemu_fclose() on a buffered file can cause I/O to be flushed.

Do you mean in addition to the log-message (copy part of the log message as a comment in the code) ?

It's subtle, so yeah - I think it deserves a comment.

Would the below patch working equally well? But then again, we should really remove the I/O handler before closing the fd.

(The close(s->fd) in migration_fd_cleanup() looks like it can never happen - perhaps we should remove it)

Cheers,
Mark.

migration.c +9 -1
index 401383c..078967f 100644
@@ -154,13 +154,21 @@ void migrate_fd_error(FdMigrationState *s)
 
 void migrate_fd_cleanup(FdMigrationState *s)
 {
-    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
+    int fd;
+
+    /* qemu_fclose() can cause I/O to be flushed (see buffered_close())
+     * which, in turn, can cause an I/O handler to be registered. We
+     * need to delay removing the I/O handler until after qemu_fclose().
+     */
+    fd = s->fd;
 
     if (s->file) {
         dprintf("closing file\n");
         qemu_fclose(s->file);
     }
 
+    qemu_set_fd_handler2(fd, NULL, NULL, NULL, NULL);
+
     if (s->fd != -1)
         close(s->fd);
 
author
#4 lore raw
Quoted 3 lines · Mark McLoughlin

Would the below patch working equally well? But then again, we should really remove the I/O handler before closing the fd.

I think it should work too.
My patch does remove the I/O handler before closing the fd.

Quoted 3 lines

(The close(s->fd) in migration_fd_cleanup() looks like it can never happen - perhaps we should remove it)

Perhaps we should. I think it's like a "plan b" in case qemu_fclose did not actually closed the file descriptor (which, as you mentioned, currently, can never happen).

migration.c void migrate_fd_error(FdMigrationState *s) · line 157

Quoted 9 lines
migration.c +0 -1
index 401383c..078967f 100644
@@ -154,13 +154,21 @@ void migrate_fd_error(FdMigrationState *s)
 
  void migrate_fd_cleanup(FdMigrationState *s)
  {
-    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);

Would it be safer to keep the I/O handler deletion before the call to qemu_fclose (in addition to adding the one below), or it does not matter.

Quoted 9 lines
migration.c +7 -0
+    int fd;
+
+    /* qemu_fclose() can cause I/O to be flushed (see buffered_close())
+     * which, in turn, can cause an I/O handler to be registered. We
+     * need to delay removing the I/O handler until after qemu_fclose().
+     */
+    fd = s->fd;
 

migration.c void migrate_fd_error(FdMigrationState *s) · line 174

Quoted 10 lines
migration.c +2 -0
      if (s->file) {
          dprintf("closing file\n");
          qemu_fclose(s->file);
      }
 
+    qemu_set_fd_handler2(fd, NULL, NULL, NULL, NULL);
+
      if (s->fd != -1)
          close(s->fd);
 

Regards,
Uri.