Skip to contentSkip to conversation

file-posix: Use OFD lock only if the filesystem supports the lock

PATCH Superseded Masayoshi Mizuma 3 participants 15 messages Nov 6, 2020 – Feb 10, 2021 Block layer

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 file-posix: Use OFD lock only if the filesystem supports the lock

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

---
 block/file-posix.c   |  56 ++++++++--------
 include/qemu/osdep.h |   2 +-
 util/osdep.c         | 149 ++++++++++++++++++++++++++++---------------
 3 files changed, 125 insertions(+), 82 deletions(-)
block/file-posix.c +28 -28
index c63926d592..a568dbf125 100644
@@ -584,34 +584,6 @@ static int raw_open_common(BlockDriverState *bs, QDict *options,
     s->use_linux_io_uring = (aio == BLOCKDEV_AIO_OPTIONS_IO_URING);
 #endif
 
-    locking = qapi_enum_parse(&OnOffAuto_lookup,
-                              qemu_opt_get(opts, "locking"),
-                              ON_OFF_AUTO_AUTO, &local_err);
-    if (local_err) {
-        error_propagate(errp, local_err);
-        ret = -EINVAL;
-        goto fail;
-    }
-    switch (locking) {
-    case ON_OFF_AUTO_ON:
-        s->use_lock = true;
-        if (!qemu_has_ofd_lock()) {
-            warn_report("File lock requested but OFD locking syscall is "
-                        "unavailable, falling back to POSIX file locks");
-            error_printf("Due to the implementation, locks can be lost "
-                         "unexpectedly.\n");
-        }
-        break;
-    case ON_OFF_AUTO_OFF:
-        s->use_lock = false;
-        break;
-    case ON_OFF_AUTO_AUTO:
-        s->use_lock = qemu_has_ofd_lock();
-        break;
-    default:
-        abort();
-    }
-
     str = qemu_opt_get(opts, "pr-manager");
     if (str) {
         s->pr_mgr = pr_manager_lookup(str, &local_err);
@@ -641,6 +613,34 @@ static int raw_open_common(BlockDriverState *bs, QDict *options,
     }
     s->fd = fd;
 
+    locking = qapi_enum_parse(&OnOffAuto_lookup,
+                              qemu_opt_get(opts, "locking"),
+                              ON_OFF_AUTO_AUTO, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        ret = -EINVAL;
+        goto fail;
+    }
+    switch (locking) {
+    case ON_OFF_AUTO_ON:
+        s->use_lock = true;
+        if (!qemu_has_ofd_lock(s->fd)) {
+            warn_report("File lock requested but OFD locking syscall is "
+                        "unavailable, falling back to POSIX file locks");
+            error_printf("Due to the implementation, locks can be lost "
+                         "unexpectedly.\n");
+        }
+        break;
+    case ON_OFF_AUTO_OFF:
+        s->use_lock = false;
+        break;
+    case ON_OFF_AUTO_AUTO:
+        s->use_lock = qemu_has_ofd_lock(s->fd);
+        break;
+    default:
+        abort();
+    }
+
     /* Check s->open_flags rather than bdrv_flags due to auto-read-only */
     if (s->open_flags & O_RDWR) {
         ret = check_hdev_writable(s->fd);
include/qemu/osdep.h +1 -1
index f9ec8c84e9..222138a81a 100644
@@ -512,7 +512,7 @@ int qemu_dup(int fd);
 int qemu_lock_fd(int fd, int64_t start, int64_t len, bool exclusive);
 int qemu_unlock_fd(int fd, int64_t start, int64_t len);
 int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);
-bool qemu_has_ofd_lock(void);
+bool qemu_has_ofd_lock(int orig_fd);
 #endif
 
 #if defined(__HAIKU__) && defined(__i386__)
util/osdep.c +96 -53

195 lines not on this page · raw message

Signature · 1 line
2.27.0
#1 lore raw

PATCH 2/2 tests/test-image-locking: Pass the fd to the argument of qemu_has_ofd_lock()

From: Masayoshi Mizuma <[email protected]>

Pass the file descriptor of /dev/null to qemu_has_ofd_lock() because former patch is changed the argument.

---
 tests/test-image-locking.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tests/test-image-locking.c +6 -1
index ba057bd66c..2b823e1588 100644
@@ -144,14 +144,19 @@ static void test_set_perm_abort(void)
 
 int main(int argc, char **argv)
 {
+    int fd;
+
     bdrv_init();
     qemu_init_main_loop(&error_abort);
 
     g_test_init(&argc, &argv, NULL);
 
-    if (qemu_has_ofd_lock()) {
+    fd = open("/dev/null", O_RDONLY);
+
+    if ((fd != -1) && (qemu_has_ofd_lock(fd))) {
         g_test_add_func("/image-locking/basic", test_image_locking_basic);
         g_test_add_func("/image-locking/set-perm-abort", test_set_perm_abort);
+        close(fd);
     }
 
     return g_test_run();
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH 2/2] tests/test-image-locking: Pass the fd to the argument of qemu_has_ofd_lock()

Am 06.11.2020 um 05:01 hat Masayoshi Mizuma geschrieben:

Quoted 19 lines

From: Masayoshi Mizuma <[email protected]>

Pass the file descriptor of /dev/null to qemu_has_ofd_lock() because former patch is changed the argument.

---
 tests/test-image-locking.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tests/test-image-locking.c +1 -0
index ba057bd66c..2b823e1588 100644
@@ -144,14 +144,19 @@ static void test_set_perm_abort(void)
 
 int main(int argc, char **argv)
 {
+    int fd;

tests/test-image-locking.c static void test_set_perm_abort(void) · line 156

Quoted 10 lines
tests/test-image-locking.c +4 -1
+
     bdrv_init();
     qemu_init_main_loop(&error_abort);
 
     g_test_init(&argc, &argv, NULL);
 
-    if (qemu_has_ofd_lock()) {
+    fd = open("/dev/null", O_RDONLY);
+
+    if ((fd != -1) && (qemu_has_ofd_lock(fd))) {

The parentheses are redundant.

tests/test-image-locking.c static void test_set_perm_abort(void) · line 160

Quoted 4 lines
tests/test-image-locking.c +1 -0
         g_test_add_func("/image-locking/basic", test_image_locking_basic);
         g_test_add_func("/image-locking/set-perm-abort", test_set_perm_abort);
+        close(fd);
     }

Please merge this with patch 1, otherwise the build will fail with only patch 1 applied (breaks bisectability).

Kevin

#3 lore raw

Re: [PATCH 2/2] tests/test-image-locking: Pass the fd to the argument of qemu_has_ofd_lock()

Quoted 33 lines · Kevin Wolf

Am 06.11.2020 um 05:01 hat Masayoshi Mizuma geschrieben:

Quoted 19 lines

From: Masayoshi Mizuma <[email protected]>

Pass the file descriptor of /dev/null to qemu_has_ofd_lock() because former patch is changed the argument.

---
 tests/test-image-locking.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
tests/test-image-locking.c +1 -0
index ba057bd66c..2b823e1588 100644
@@ -144,14 +144,19 @@ static void test_set_perm_abort(void)
 
 int main(int argc, char **argv)
 {
+    int fd;

tests/test-image-locking.c static void test_set_perm_abort(void) · line 156

Quoted 10 lines
tests/test-image-locking.c +4 -1
+
     bdrv_init();
     qemu_init_main_loop(&error_abort);
 
     g_test_init(&argc, &argv, NULL);
 
-    if (qemu_has_ofd_lock()) {
+    fd = open("/dev/null", O_RDONLY);
+
+    if ((fd != -1) && (qemu_has_ofd_lock(fd))) {

The parentheses are redundant.

Quoted 7 lines

tests/test-image-locking.c static void test_set_perm_abort(void) · line 160

Quoted 4 lines
tests/test-image-locking.c +1 -0
         g_test_add_func("/image-locking/basic", test_image_locking_basic);
         g_test_add_func("/image-locking/set-perm-abort", test_set_perm_abort);
+        close(fd);
     }

Please merge this with patch 1, otherwise the build will fail with only patch 1 applied (breaks bisectability).

Got it. I'll merge this to patch 1.

Thanks!
Masa

#4 lore raw
In reply to Masayoshi Mizuma

Hello,

Would you review this patch when you have a chance?

Thanks!
Masa

Quoted 314 lines · Masayoshi Mizuma

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

---
 block/file-posix.c   |  56 ++++++++--------
 include/qemu/osdep.h |   2 +-
 util/osdep.c         | 149 ++++++++++++++++++++++++++++---------------
 3 files changed, 125 insertions(+), 82 deletions(-)
block/file-posix.c +28 -28
index c63926d592..a568dbf125 100644
@@ -584,34 +584,6 @@ static int raw_open_common(BlockDriverState *bs, QDict *options,
     s->use_linux_io_uring = (aio == BLOCKDEV_AIO_OPTIONS_IO_URING);
 #endif
 
-    locking = qapi_enum_parse(&OnOffAuto_lookup,
-                              qemu_opt_get(opts, "locking"),
-                              ON_OFF_AUTO_AUTO, &local_err);
-    if (local_err) {
-        error_propagate(errp, local_err);
-        ret = -EINVAL;
-        goto fail;
-    }
-    switch (locking) {
-    case ON_OFF_AUTO_ON:
-        s->use_lock = true;
-        if (!qemu_has_ofd_lock()) {
-            warn_report("File lock requested but OFD locking syscall is "
-                        "unavailable, falling back to POSIX file locks");
-            error_printf("Due to the implementation, locks can be lost "
-                         "unexpectedly.\n");
-        }
-        break;
-    case ON_OFF_AUTO_OFF:
-        s->use_lock = false;
-        break;
-    case ON_OFF_AUTO_AUTO:
-        s->use_lock = qemu_has_ofd_lock();
-        break;
-    default:
-        abort();
-    }
-
     str = qemu_opt_get(opts, "pr-manager");
     if (str) {
         s->pr_mgr = pr_manager_lookup(str, &local_err);
@@ -641,6 +613,34 @@ static int raw_open_common(BlockDriverState *bs, QDict *options,
     }
     s->fd = fd;
 
+    locking = qapi_enum_parse(&OnOffAuto_lookup,
+                              qemu_opt_get(opts, "locking"),
+                              ON_OFF_AUTO_AUTO, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        ret = -EINVAL;
+        goto fail;
+    }
+    switch (locking) {
+    case ON_OFF_AUTO_ON:
+        s->use_lock = true;
+        if (!qemu_has_ofd_lock(s->fd)) {
+            warn_report("File lock requested but OFD locking syscall is "
+                        "unavailable, falling back to POSIX file locks");
+            error_printf("Due to the implementation, locks can be lost "
+                         "unexpectedly.\n");
+        }
+        break;
+    case ON_OFF_AUTO_OFF:
+        s->use_lock = false;
+        break;
+    case ON_OFF_AUTO_AUTO:
+        s->use_lock = qemu_has_ofd_lock(s->fd);
+        break;
+    default:
+        abort();
+    }
+
     /* Check s->open_flags rather than bdrv_flags due to auto-read-only */
     if (s->open_flags & O_RDWR) {
         ret = check_hdev_writable(s->fd);
include/qemu/osdep.h +1 -1
index f9ec8c84e9..222138a81a 100644
@@ -512,7 +512,7 @@ int qemu_dup(int fd);
 int qemu_lock_fd(int fd, int64_t start, int64_t len, bool exclusive);
 int qemu_unlock_fd(int fd, int64_t start, int64_t len);
 int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive);
-bool qemu_has_ofd_lock(void);
+bool qemu_has_ofd_lock(int orig_fd);
 #endif
 
 #if defined(__HAIKU__) && defined(__i386__)
util/osdep.c +96 -53
index 66d01b9160..454e8ef9f4 100644
@@ -117,9 +117,6 @@ int qemu_mprotect_none(void *addr, size_t size)
 
 #ifndef _WIN32
 
-static int fcntl_op_setlk = -1;
-static int fcntl_op_getlk = -1;
-
 /*
  * Dups an fd and sets the flags
  */
@@ -187,68 +184,87 @@ static int qemu_parse_fdset(const char *param)
     return qemu_parse_fd(param);
 }
 
-static void qemu_probe_lock_ops(void)
+bool qemu_has_ofd_lock(int orig_fd)
 {
-    if (fcntl_op_setlk == -1) {
 #ifdef F_OFD_SETLK
-        int fd;
-        int ret;
-        struct flock fl = {
-            .l_whence = SEEK_SET,
-            .l_start  = 0,
-            .l_len    = 0,
-            .l_type   = F_WRLCK,
-        };
-
-        fd = open("/dev/null", O_RDWR);
-        if (fd < 0) {
+    int fd;
+    int ret;
+    struct flock fl = {
+        .l_whence = SEEK_SET,
+        .l_start  = 0,
+        .l_len    = 0,
+        .l_type   = F_RDLCK,
+    };
+
+    fd = qemu_dup(orig_fd);
+    if (fd >= 0) {
+        ret = fcntl_setfl(fd, O_RDONLY);
+        if (ret) {
             fprintf(stderr,
-                    "Failed to open /dev/null for OFD lock probing: %s\n",
-                    strerror(errno));
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
-            return;
-        }
-        ret = fcntl(fd, F_OFD_GETLK, &fl);
-        close(fd);
-        if (!ret) {
-            fcntl_op_setlk = F_OFD_SETLK;
-            fcntl_op_getlk = F_OFD_GETLK;
-        } else {
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
+                    "Failed to fcntl for OFD lock probing.\n");
+            qemu_close(fd);
+            return false;
         }
+    }
+
+    ret = fcntl(fd, F_OFD_GETLK, &fl);
+    qemu_close(fd);
+
+    if (ret == 0) {
+        return true;
+    } else {
+        return false;
+    }
 #else
-        fcntl_op_setlk = F_SETLK;
-        fcntl_op_getlk = F_GETLK;
+    return false;
 #endif
-    }
 }
 
-bool qemu_has_ofd_lock(void)
-{
-    qemu_probe_lock_ops();
 #ifdef F_OFD_SETLK
-    return fcntl_op_setlk == F_OFD_SETLK;
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+    bool ofd_lock = true;
+
+    do {
+        if (ofd_lock) {
+            ret = fcntl(fd, F_OFD_SETLK, fl);
+            if ((ret == -1) && (errno == EINVAL)) {
+                ofd_lock = false;
+            }
+        }
+
+        if (!ofd_lock) {
+            /* Fallback to POSIX lock */
+            ret = fcntl(fd, F_SETLK, fl);
+        }
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
+}
 #else
-    return false;
-#endif
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+
+    do {
+        ret = fcntl(fd, F_SETLK, fl);
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
 }
+#endif
 
 static int qemu_lock_fcntl(int fd, int64_t start, int64_t len, int fl_type)
 {
-    int ret;
     struct flock fl = {
         .l_whence = SEEK_SET,
         .l_start  = start,
         .l_len    = len,
         .l_type   = fl_type,
     };
-    qemu_probe_lock_ops();
-    do {
-        ret = fcntl(fd, fcntl_op_setlk, &fl);
-    } while (ret == -1 && errno == EINTR);
-    return ret == -1 ? -errno : 0;
+
+    return _qemu_lock_fcntl(fd, &fl);
 }
 
 int qemu_lock_fd(int fd, int64_t start, int64_t len, bool exclusive)
@@ -261,22 +277,49 @@ int qemu_unlock_fd(int fd, int64_t start, int64_t len)
     return qemu_lock_fcntl(fd, start, len, F_UNLCK);
 }
 
-int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive)
+#ifdef F_OFD_SETLK
+static int _qemu_lock_fd_test(int fd, struct flock *fl)
 {
     int ret;
+
+    ret = fcntl(fd, F_OFD_GETLK, fl);
+    if ((ret == -1) && (errno != EINVAL)) {
+        return -errno;
+
+    } else if ((ret == -1) && (errno == EINVAL)) {
+        /* Fallback to POSIX lock */
+        ret = fcntl(fd, F_GETLK, fl);
+        if (ret == -1) {
+            return -errno;
+        }
+    }
+
+    return fl->l_type == F_UNLCK ? 0 : -EAGAIN;
+}
+#else
+static int _qemu_lock_fd_test(int fd, struct flock *fl)
+{
+    int ret;
+
+    ret = fcntl(fd, F_GETLK, fl);
+    if (ret == -1) {
+        return -errno;
+    } else {
+        return fl->l_type == F_UNLCK ? 0 : -EAGAIN;
+    }
+}
+#endif
+
+int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive)
+{
     struct flock fl = {
         .l_whence = SEEK_SET,
         .l_start  = start,
         .l_len    = len,
         .l_type   = exclusive ? F_WRLCK : F_RDLCK,
     };
-    qemu_probe_lock_ops();
-    ret = fcntl(fd, fcntl_op_getlk, &fl);
-    if (ret == -1) {
-        return -errno;
-    } else {
-        return fl.l_type == F_UNLCK ? 0 : -EAGAIN;
-    }
+
+    return _qemu_lock_fd_test(fd, &fl);
 }
 #endif
 
Signature · 1 line
2.27.0
#5 lore raw
In reply to Masayoshi Mizuma
Quoted 57 lines · Masayoshi Mizuma

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

---
 block/file-posix.c   |  56 ++++++++--------
 include/qemu/osdep.h |   2 +-
 util/osdep.c         | 149 ++++++++++++++++++++++++++++---------------
 3 files changed, 125 insertions(+), 82 deletions(-)
util/osdep.c +1 -16
index 66d01b9160..454e8ef9f4 100644
@@ -117,9 +117,6 @@ int qemu_mprotect_none(void *addr, size_t size)
 
 #ifndef _WIN32
 
-static int fcntl_op_setlk = -1;
-static int fcntl_op_getlk = -1;
-
 /*
  * Dups an fd and sets the flags
  */
@@ -187,68 +184,87 @@ static int qemu_parse_fdset(const char *param)
     return qemu_parse_fd(param);
 }
 
-static void qemu_probe_lock_ops(void)
+bool qemu_has_ofd_lock(int orig_fd)
 {
-    if (fcntl_op_setlk == -1) {
 #ifdef F_OFD_SETLK
-        int fd;
-        int ret;
-        struct flock fl = {
-            .l_whence = SEEK_SET,
-            .l_start  = 0,
-            .l_len    = 0,
-            .l_type   = F_WRLCK,
-        };
-
-        fd = open("/dev/null", O_RDWR);
-        if (fd < 0) {

util/osdep.c static int qemu_parse_fdset(const char *param) · line 199

Quoted 10 lines
util/osdep.c +10 -0
+    int fd;
+    int ret;
+    struct flock fl = {
+        .l_whence = SEEK_SET,
+        .l_start  = 0,
+        .l_len    = 0,
+        .l_type   = F_RDLCK,
+    };
+
+    fd = qemu_dup(orig_fd);

Consider that we're *not* using OFD locks, and QEMU already has 'foo.qcow2' open for an existing disk backend, and it is locked.

Now someone tries to hot-add 'foo.qcow2' for a second disk by mistake. Doing this qemu_dup + qemu_close will cause the existing locks to be removed AFAICT.

Quoted 53 lines
util/osdep.c +25 -21
+    if (fd >= 0) {
+        ret = fcntl_setfl(fd, O_RDONLY);
+        if (ret) {
             fprintf(stderr,
-                    "Failed to open /dev/null for OFD lock probing: %s\n",
-                    strerror(errno));
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
-            return;
-        }
-        ret = fcntl(fd, F_OFD_GETLK, &fl);
-        close(fd);
-        if (!ret) {
-            fcntl_op_setlk = F_OFD_SETLK;
-            fcntl_op_getlk = F_OFD_GETLK;
-        } else {
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
+                    "Failed to fcntl for OFD lock probing.\n");
+            qemu_close(fd);
+            return false;
         }
+    }
+
+    ret = fcntl(fd, F_OFD_GETLK, &fl);
+    qemu_close(fd);
+
+    if (ret == 0) {
+        return true;
+    } else {
+        return false;
+    }
 #else
-        fcntl_op_setlk = F_SETLK;
-        fcntl_op_getlk = F_GETLK;
+    return false;
 #endif
-    }
 }
 
-bool qemu_has_ofd_lock(void)
-{
-    qemu_probe_lock_ops();
 #ifdef F_OFD_SETLK
-    return fcntl_op_setlk == F_OFD_SETLK;
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+    bool ofd_lock = true;
+
+    do {
+        if (ofd_lock) {
+            ret = fcntl(fd, F_OFD_SETLK, fl);

util/osdep.c static int qemu_parse_fdset(const char *param) · line 241

Quoted 10 lines
util/osdep.c +10 -0
+            if ((ret == -1) && (errno == EINVAL)) {
+                ofd_lock = false;
+            }
+        }
+
+        if (!ofd_lock) {
+            /* Fallback to POSIX lock */
+            ret = fcntl(fd, F_SETLK, fl);
+        }
+    } while (ret == -1 && errno == EINTR);

THis loop is confusing to read. I'd suggest creating a wrapper

qemu_fcntl()

that does the while (ret == -1 && errno == EINTR) loop, so that this locking code can be clearer without the loop.

Quoted 7 lines
util/osdep.c +4 -2
+
+    return ret == -1 ? -errno : 0;
+}
 #else
-    return false;
-#endif
+static int _qemu_lock_fcntl(int fd, struct flock *fl)

util/osdep.c static int qemu_parse_fdset(const char *param) · line 256

Quoted 10 lines
util/osdep.c +9 -0
+{
+    int ret;
+
+    do {
+        ret = fcntl(fd, F_SETLK, fl);
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
 }
+#endif

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw
Quoted 73 lines · Daniel P. Berrangé
Quoted 57 lines · Masayoshi Mizuma

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

---
 block/file-posix.c   |  56 ++++++++--------
 include/qemu/osdep.h |   2 +-
 util/osdep.c         | 149 ++++++++++++++++++++++++++++---------------
 3 files changed, 125 insertions(+), 82 deletions(-)
util/osdep.c +1 -16
index 66d01b9160..454e8ef9f4 100644
@@ -117,9 +117,6 @@ int qemu_mprotect_none(void *addr, size_t size)
 
 #ifndef _WIN32
 
-static int fcntl_op_setlk = -1;
-static int fcntl_op_getlk = -1;
-
 /*
  * Dups an fd and sets the flags
  */
@@ -187,68 +184,87 @@ static int qemu_parse_fdset(const char *param)
     return qemu_parse_fd(param);
 }
 
-static void qemu_probe_lock_ops(void)
+bool qemu_has_ofd_lock(int orig_fd)
 {
-    if (fcntl_op_setlk == -1) {
 #ifdef F_OFD_SETLK
-        int fd;
-        int ret;
-        struct flock fl = {
-            .l_whence = SEEK_SET,
-            .l_start  = 0,
-            .l_len    = 0,
-            .l_type   = F_WRLCK,
-        };
-
-        fd = open("/dev/null", O_RDWR);
-        if (fd < 0) {

util/osdep.c static int qemu_parse_fdset(const char *param) · line 199

Quoted 10 lines
util/osdep.c +10 -0
+    int fd;
+    int ret;
+    struct flock fl = {
+        .l_whence = SEEK_SET,
+        .l_start  = 0,
+        .l_len    = 0,
+        .l_type   = F_RDLCK,
+    };
+
+    fd = qemu_dup(orig_fd);

Consider that we're *not* using OFD locks, and QEMU already has 'foo.qcow2' open for an existing disk backend, and it is locked.

Quoted 3 lines

Now someone tries to hot-add 'foo.qcow2' for a second disk by mistake. Doing this qemu_dup + qemu_close will cause the existing locks to be removed AFAICT.

Thank you for pointing it out. I'll remove this qemu_dup() and check orig_fd directly.

Quoted 68 lines
Quoted 54 lines
util/osdep.c +25 -21
+    if (fd >= 0) {
+        ret = fcntl_setfl(fd, O_RDONLY);
+        if (ret) {
             fprintf(stderr,
-                    "Failed to open /dev/null for OFD lock probing: %s\n",
-                    strerror(errno));
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
-            return;
-        }
-        ret = fcntl(fd, F_OFD_GETLK, &fl);
-        close(fd);
-        if (!ret) {
-            fcntl_op_setlk = F_OFD_SETLK;
-            fcntl_op_getlk = F_OFD_GETLK;
-        } else {
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
+                    "Failed to fcntl for OFD lock probing.\n");
+            qemu_close(fd);
+            return false;
         }
+    }
+
+    ret = fcntl(fd, F_OFD_GETLK, &fl);
+    qemu_close(fd);
+
+    if (ret == 0) {
+        return true;
+    } else {
+        return false;
+    }
 #else
-        fcntl_op_setlk = F_SETLK;
-        fcntl_op_getlk = F_GETLK;
+    return false;
 #endif
-    }
 }
 
-bool qemu_has_ofd_lock(void)
-{
-    qemu_probe_lock_ops();
 #ifdef F_OFD_SETLK
-    return fcntl_op_setlk == F_OFD_SETLK;
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+    bool ofd_lock = true;
+
+    do {
+        if (ofd_lock) {
+            ret = fcntl(fd, F_OFD_SETLK, fl);

util/osdep.c static int qemu_parse_fdset(const char *param) · line 241

Quoted 10 lines
util/osdep.c +10 -0
+            if ((ret == -1) && (errno == EINVAL)) {
+                ofd_lock = false;
+            }
+        }
+
+        if (!ofd_lock) {
+            /* Fallback to POSIX lock */
+            ret = fcntl(fd, F_SETLK, fl);
+        }
+    } while (ret == -1 && errno == EINTR);

THis loop is confusing to read. I'd suggest creating a wrapper

Quoted 5 lines

qemu_fcntl()

that does the while (ret == -1 && errno == EINTR) loop, so that this locking code can be clearer without the loop.

Great idea. I'll make qemu_fcntl().

Thanks!
Masa

Quoted 27 lines
Quoted 8 lines
util/osdep.c +4 -2
+
+    return ret == -1 ? -errno : 0;
+}
 #else
-    return false;
-#endif
+static int _qemu_lock_fcntl(int fd, struct flock *fl)

util/osdep.c static int qemu_parse_fdset(const char *param) · line 256

Quoted 10 lines
util/osdep.c +9 -0
+{
+    int ret;
+
+    do {
+        ret = fcntl(fd, F_SETLK, fl);
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
 }
+#endif

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw
In reply to Masayoshi Mizuma

Am 06.11.2020 um 05:01 hat Masayoshi Mizuma geschrieben:

Quoted 12 lines

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Quoted 4 lines

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

CCing qemu-block, which is the relevant mailing list. You can use the scripts/get_maintainer.pl script to find out who should be CCed on your patches.

As qemu-devel itself is a very high traffic list, it's easy for a patch to get lost if it's only sent there.

Quoted 36 lines
util/osdep.c +3 -16
index 66d01b9160..454e8ef9f4 100644
@@ -117,9 +117,6 @@ int qemu_mprotect_none(void *addr, size_t size)
 
 #ifndef _WIN32
 
-static int fcntl_op_setlk = -1;
-static int fcntl_op_getlk = -1;
-
 /*
  * Dups an fd and sets the flags
  */
@@ -187,68 +184,87 @@ static int qemu_parse_fdset(const char *param)
     return qemu_parse_fd(param);
 }
 
-static void qemu_probe_lock_ops(void)
+bool qemu_has_ofd_lock(int orig_fd)
 {
-    if (fcntl_op_setlk == -1) {
 #ifdef F_OFD_SETLK
-        int fd;
-        int ret;
-        struct flock fl = {
-            .l_whence = SEEK_SET,
-            .l_start  = 0,
-            .l_len    = 0,
-            .l_type   = F_WRLCK,
-        };
-
-        fd = open("/dev/null", O_RDWR);
-        if (fd < 0) {
+    int fd;
+    int ret;

util/osdep.c static int qemu_parse_fdset(const char *param) · line 201

Quoted 10 lines
util/osdep.c +10 -0
+    struct flock fl = {
+        .l_whence = SEEK_SET,
+        .l_start  = 0,
+        .l_len    = 0,
+        .l_type   = F_RDLCK,
+    };
+
+    fd = qemu_dup(orig_fd);
+    if (fd >= 0) {
+        ret = fcntl_setfl(fd, O_RDONLY);

I don't understand this part. Why are you trying to reopen the file descriptor read-only? Shouldn't the test work fine with a read-write file descriptor? /dev/null was opened O_RDWR in the old code.

Quoted 14 lines
util/osdep.c +1 -12
+        if (ret) {
             fprintf(stderr,
-                    "Failed to open /dev/null for OFD lock probing: %s\n",
-                    strerror(errno));
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
-            return;
-        }
-        ret = fcntl(fd, F_OFD_GETLK, &fl);
-        close(fd);
-        if (!ret) {
-            fcntl_op_setlk = F_OFD_SETLK;
-            fcntl_op_getlk = F_OFD_GETLK;
-        } else {

util/osdep.c static int qemu_parse_fdset(const char *param) · line 211

Quoted 10 lines
util/osdep.c +7 -2
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
+                    "Failed to fcntl for OFD lock probing.\n");
+            qemu_close(fd);
+            return false;
         }
+    }
+
+    ret = fcntl(fd, F_OFD_GETLK, &fl);
+    qemu_close(fd);

F_OFD_GETLK doesn't modify the state, so it seems to me that even the qemu_dup() is unnecessary and we could just directly try F_OFD_GETLK on the passed file descriptor (orig_fd).

util/osdep.c static int qemu_parse_fdset(const char *param) · line 217

Quoted 6 lines
util/osdep.c +6 -0
+
+    if (ret == 0) {
+        return true;
+    } else {
+        return false;
+    }

This should be written shorter as return ret == 0;

Quoted 38 lines
util/osdep.c +23 -9
 #else
-        fcntl_op_setlk = F_SETLK;
-        fcntl_op_getlk = F_GETLK;
+    return false;
 #endif
-    }
 }
 
-bool qemu_has_ofd_lock(void)
-{
-    qemu_probe_lock_ops();
 #ifdef F_OFD_SETLK
-    return fcntl_op_setlk == F_OFD_SETLK;
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+    bool ofd_lock = true;
+
+    do {
+        if (ofd_lock) {
+            ret = fcntl(fd, F_OFD_SETLK, fl);
+            if ((ret == -1) && (errno == EINVAL)) {
+                ofd_lock = false;
+            }
+        }
+
+        if (!ofd_lock) {
+            /* Fallback to POSIX lock */
+            ret = fcntl(fd, F_SETLK, fl);
+        }
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
+}
 #else
-    return false;
-#endif
+static int _qemu_lock_fcntl(int fd, struct flock *fl)

util/osdep.c static int qemu_parse_fdset(const char *param) · line 256

Quoted 10 lines
util/osdep.c +9 -0
+{
+    int ret;
+
+    do {
+        ret = fcntl(fd, F_SETLK, fl);
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
 }
+#endif

The logic looks fine to me, at least assuming that EINVAL is really what we will consistently get from the kernel if OFD locks are not supported. Is this documented anywhere? The fcntl manpage doesn't seem to mention this case.

Anyway, I think I would try to minimise the duplication by having only a small #ifdef section inside the function, maybe like this:

#ifdef F_OFD_SETLK
            ret = fcntl(fd, F_OFD_SETLK, fl);
            if ((ret == -1) && (errno == EINVAL)) {
                ofd_lock = false;
            }
#else
            ofd_lock = false;

#endif

Quoted 23 lines
util/osdep.c +2 -6
 static int qemu_lock_fcntl(int fd, int64_t start, int64_t len, int fl_type)
 {
-    int ret;
     struct flock fl = {
         .l_whence = SEEK_SET,
         .l_start  = start,
         .l_len    = len,
         .l_type   = fl_type,
     };
-    qemu_probe_lock_ops();
-    do {
-        ret = fcntl(fd, fcntl_op_setlk, &fl);
-    } while (ret == -1 && errno == EINTR);
-    return ret == -1 ? -errno : 0;
+
+    return _qemu_lock_fcntl(fd, &fl);
 }
 
 int qemu_lock_fd(int fd, int64_t start, int64_t len, bool exclusive)
@@ -261,22 +277,49 @@ int qemu_unlock_fd(int fd, int64_t start, int64_t len)
     return qemu_lock_fcntl(fd, start, len, F_UNLCK);
 }
 

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 288

Quoted 10 lines
util/osdep.c +7 -1
-int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive)
+#ifdef F_OFD_SETLK
+static int _qemu_lock_fd_test(int fd, struct flock *fl)
 {
     int ret;
+
+    ret = fcntl(fd, F_OFD_GETLK, fl);
+    if ((ret == -1) && (errno != EINVAL)) {
+        return -errno;
+

Please remove this empty line.

The parentheses in the condition (above and below) are not stricly necessary.

Quoted 13 lines
util/osdep.c +13 -0
+    } else if ((ret == -1) && (errno == EINVAL)) {
+        /* Fallback to POSIX lock */
+        ret = fcntl(fd, F_GETLK, fl);
+        if (ret == -1) {
+            return -errno;
+        }
+    }
+
+    return fl->l_type == F_UNLCK ? 0 : -EAGAIN;
+}
+#else
+static int _qemu_lock_fd_test(int fd, struct flock *fl)
+{

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 311

Quoted 10 lines
util/osdep.c +10 -0
+    int ret;
+
+    ret = fcntl(fd, F_GETLK, fl);
+    if (ret == -1) {
+        return -errno;
+    } else {
+        return fl->l_type == F_UNLCK ? 0 : -EAGAIN;
+    }
+}
+#endif

Same idea as above: #ifdef only around the fcntl(F_OFD_GETLK) call can minimise the code duplication.

Quoted 9 lines
util/osdep.c +2 -1
+int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive)
+{
     struct flock fl = {
         .l_whence = SEEK_SET,
         .l_start  = start,
         .l_len    = len,
         .l_type   = exclusive ? F_WRLCK : F_RDLCK,
     };
-    qemu_probe_lock_ops();

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 324

Quoted 10 lines
util/osdep.c +2 -6
-    ret = fcntl(fd, fcntl_op_getlk, &fl);
-    if (ret == -1) {
-        return -errno;
-    } else {
-        return fl.l_type == F_UNLCK ? 0 : -EAGAIN;
-    }
+
+    return _qemu_lock_fd_test(fd, &fl);
 }
 #endif

After moving the #ifdef into the function, you can inline _qemu_lock_fd_test() and and _qemu_lock_fcntl() again. This is also good because identifiers starting with an underscore are reserved in the C standard.

Kevin

#8 lore raw
Quoted 16 lines · Kevin Wolf

Am 06.11.2020 um 05:01 hat Masayoshi Mizuma geschrieben:

Quoted 15 lines

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

Quoted 8 lines
Quoted 1 line

CCing qemu-block, which is the relevant mailing list. You can use the scripts/get_maintainer.pl script to find out who should be CCed on your patches.

As qemu-devel itself is a very high traffic list, it's easy for a patch to get lost if it's only sent there.

Thank you for letting me know.
I'll do scripts/get_maintainer.pl to get the mailing list before posting patches.

Quoted 77 lines
Quoted 37 lines
util/osdep.c +3 -16
index 66d01b9160..454e8ef9f4 100644
@@ -117,9 +117,6 @@ int qemu_mprotect_none(void *addr, size_t size)
 
 #ifndef _WIN32
 
-static int fcntl_op_setlk = -1;
-static int fcntl_op_getlk = -1;
-
 /*
  * Dups an fd and sets the flags
  */
@@ -187,68 +184,87 @@ static int qemu_parse_fdset(const char *param)
     return qemu_parse_fd(param);
 }
 
-static void qemu_probe_lock_ops(void)
+bool qemu_has_ofd_lock(int orig_fd)
 {
-    if (fcntl_op_setlk == -1) {
 #ifdef F_OFD_SETLK
-        int fd;
-        int ret;
-        struct flock fl = {
-            .l_whence = SEEK_SET,
-            .l_start  = 0,
-            .l_len    = 0,
-            .l_type   = F_WRLCK,
-        };
-
-        fd = open("/dev/null", O_RDWR);
-        if (fd < 0) {
+    int fd;
+    int ret;

util/osdep.c static int qemu_parse_fdset(const char *param) · line 201

Quoted 10 lines
util/osdep.c +10 -0
+    struct flock fl = {
+        .l_whence = SEEK_SET,
+        .l_start  = 0,
+        .l_len    = 0,
+        .l_type   = F_RDLCK,
+    };
+
+    fd = qemu_dup(orig_fd);
+    if (fd >= 0) {
+        ret = fcntl_setfl(fd, O_RDONLY);

I don't understand this part. Why are you trying to reopen the file descriptor read-only? Shouldn't the test work fine with a read-write file descriptor? /dev/null was opened O_RDWR in the old code.

Quoted 14 lines
util/osdep.c +1 -12
+        if (ret) {
             fprintf(stderr,
-                    "Failed to open /dev/null for OFD lock probing: %s\n",
-                    strerror(errno));
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
-            return;
-        }
-        ret = fcntl(fd, F_OFD_GETLK, &fl);
-        close(fd);
-        if (!ret) {
-            fcntl_op_setlk = F_OFD_SETLK;
-            fcntl_op_getlk = F_OFD_GETLK;
-        } else {

util/osdep.c static int qemu_parse_fdset(const char *param) · line 211

Quoted 10 lines
util/osdep.c +7 -2
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
+                    "Failed to fcntl for OFD lock probing.\n");
+            qemu_close(fd);
+            return false;
         }
+    }
+
+    ret = fcntl(fd, F_OFD_GETLK, &fl);
+    qemu_close(fd);
Quoted 3 lines

F_OFD_GETLK doesn't modify the state, so it seems to me that even the qemu_dup() is unnecessary and we could just directly try F_OFD_GETLK on the passed file descriptor (orig_fd).

OK, I'll change to try F_OFD_GETLK of orig_fd directly.

Quoted 67 lines

util/osdep.c static int qemu_parse_fdset(const char *param) · line 217

Quoted 7 lines
util/osdep.c +6 -0
+
+    if (ret == 0) {
+        return true;
+    } else {
+        return false;
+    }

This should be written shorter as return ret == 0;

Quoted 38 lines
util/osdep.c +23 -9
 #else
-        fcntl_op_setlk = F_SETLK;
-        fcntl_op_getlk = F_GETLK;
+    return false;
 #endif
-    }
 }
 
-bool qemu_has_ofd_lock(void)
-{
-    qemu_probe_lock_ops();
 #ifdef F_OFD_SETLK
-    return fcntl_op_setlk == F_OFD_SETLK;
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+    bool ofd_lock = true;
+
+    do {
+        if (ofd_lock) {
+            ret = fcntl(fd, F_OFD_SETLK, fl);
+            if ((ret == -1) && (errno == EINVAL)) {
+                ofd_lock = false;
+            }
+        }
+
+        if (!ofd_lock) {
+            /* Fallback to POSIX lock */
+            ret = fcntl(fd, F_SETLK, fl);
+        }
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
+}
 #else
-    return false;
-#endif
+static int _qemu_lock_fcntl(int fd, struct flock *fl)

util/osdep.c static int qemu_parse_fdset(const char *param) · line 256

Quoted 10 lines
util/osdep.c +9 -0
+{
+    int ret;
+
+    do {
+        ret = fcntl(fd, F_SETLK, fl);
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
 }
+#endif

The logic looks fine to me, at least assuming that EINVAL is really what we will consistently get from the kernel if OFD locks are not supported. Is this documented anywhere? The fcntl manpage doesn't seem to mention this case.

Anyway, I think I would try to minimise the duplication by having only a small #ifdef section inside the function, maybe like this:

Quoted 8 lines
#ifdef F_OFD_SETLK
            ret = fcntl(fd, F_OFD_SETLK, fl);
            if ((ret == -1) && (errno == EINVAL)) {
                ofd_lock = false;
            }
#else
            ofd_lock = false;

#endif

Great! I'll make this.

Quoted 37 lines
Quoted 24 lines
util/osdep.c +2 -6
 static int qemu_lock_fcntl(int fd, int64_t start, int64_t len, int fl_type)
 {
-    int ret;
     struct flock fl = {
         .l_whence = SEEK_SET,
         .l_start  = start,
         .l_len    = len,
         .l_type   = fl_type,
     };
-    qemu_probe_lock_ops();
-    do {
-        ret = fcntl(fd, fcntl_op_setlk, &fl);
-    } while (ret == -1 && errno == EINTR);
-    return ret == -1 ? -errno : 0;
+
+    return _qemu_lock_fcntl(fd, &fl);
 }
 
 int qemu_lock_fd(int fd, int64_t start, int64_t len, bool exclusive)
@@ -261,22 +277,49 @@ int qemu_unlock_fd(int fd, int64_t start, int64_t len)
     return qemu_lock_fcntl(fd, start, len, F_UNLCK);
 }
 

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 288

Quoted 10 lines
util/osdep.c +7 -1
-int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive)
+#ifdef F_OFD_SETLK
+static int _qemu_lock_fd_test(int fd, struct flock *fl)
 {
     int ret;
+
+    ret = fcntl(fd, F_OFD_GETLK, fl);
+    if ((ret == -1) && (errno != EINVAL)) {
+        return -errno;
+

Please remove this empty line.

Quoted 2 lines

The parentheses in the condition (above and below) are not stricly necessary.

Got it.

Quoted 44 lines
Quoted 14 lines
util/osdep.c +13 -0
+    } else if ((ret == -1) && (errno == EINVAL)) {
+        /* Fallback to POSIX lock */
+        ret = fcntl(fd, F_GETLK, fl);
+        if (ret == -1) {
+            return -errno;
+        }
+    }
+
+    return fl->l_type == F_UNLCK ? 0 : -EAGAIN;
+}
+#else
+static int _qemu_lock_fd_test(int fd, struct flock *fl)
+{

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 311

Quoted 10 lines
util/osdep.c +10 -0
+    int ret;
+
+    ret = fcntl(fd, F_GETLK, fl);
+    if (ret == -1) {
+        return -errno;
+    } else {
+        return fl->l_type == F_UNLCK ? 0 : -EAGAIN;
+    }
+}
+#endif

Same idea as above: #ifdef only around the fcntl(F_OFD_GETLK) call can minimise the code duplication.

Quoted 16 lines
util/osdep.c +3 -7
+int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive)
+{
     struct flock fl = {
         .l_whence = SEEK_SET,
         .l_start  = start,
         .l_len    = len,
         .l_type   = exclusive ? F_WRLCK : F_RDLCK,
     };
-    qemu_probe_lock_ops();
-    ret = fcntl(fd, fcntl_op_getlk, &fl);
-    if (ret == -1) {
-        return -errno;
-    } else {
-        return fl.l_type == F_UNLCK ? 0 : -EAGAIN;
-    }
+
Quoted 8 lines

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 324

Quoted 3 lines
util/osdep.c +1 -0
+    return _qemu_lock_fd_test(fd, &fl);
 }
 #endif

After moving the #ifdef into the function, you can inline _qemu_lock_fd_test() and and _qemu_lock_fcntl() again. This is also good because identifiers starting with an underscore are reserved in the C standard.

Got it, thanks! I'll post v2.

- Masa

#9 lore raw
Quoted 106 lines · Masayoshi Mizuma
Quoted 16 lines · Kevin Wolf

Am 06.11.2020 um 05:01 hat Masayoshi Mizuma geschrieben:

Quoted 15 lines

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

Quoted 8 lines
Quoted 1 line

CCing qemu-block, which is the relevant mailing list. You can use the scripts/get_maintainer.pl script to find out who should be CCed on your patches.

As qemu-devel itself is a very high traffic list, it's easy for a patch to get lost if it's only sent there.

Thank you for letting me know.
I'll do scripts/get_maintainer.pl to get the mailing list before posting patches.

Quoted 77 lines
Quoted 37 lines
util/osdep.c +3 -16
index 66d01b9160..454e8ef9f4 100644
@@ -117,9 +117,6 @@ int qemu_mprotect_none(void *addr, size_t size)
 
 #ifndef _WIN32
 
-static int fcntl_op_setlk = -1;
-static int fcntl_op_getlk = -1;
-
 /*
  * Dups an fd and sets the flags
  */
@@ -187,68 +184,87 @@ static int qemu_parse_fdset(const char *param)
     return qemu_parse_fd(param);
 }
 
-static void qemu_probe_lock_ops(void)
+bool qemu_has_ofd_lock(int orig_fd)
 {
-    if (fcntl_op_setlk == -1) {
 #ifdef F_OFD_SETLK
-        int fd;
-        int ret;
-        struct flock fl = {
-            .l_whence = SEEK_SET,
-            .l_start  = 0,
-            .l_len    = 0,
-            .l_type   = F_WRLCK,
-        };
-
-        fd = open("/dev/null", O_RDWR);
-        if (fd < 0) {
+    int fd;
+    int ret;

util/osdep.c static int qemu_parse_fdset(const char *param) · line 201

Quoted 10 lines
util/osdep.c +10 -0
+    struct flock fl = {
+        .l_whence = SEEK_SET,
+        .l_start  = 0,
+        .l_len    = 0,
+        .l_type   = F_RDLCK,
+    };
+
+    fd = qemu_dup(orig_fd);
+    if (fd >= 0) {
+        ret = fcntl_setfl(fd, O_RDONLY);

I don't understand this part. Why are you trying to reopen the file descriptor read-only? Shouldn't the test work fine with a read-write file descriptor? /dev/null was opened O_RDWR in the old code.

Quoted 24 lines
util/osdep.c +8 -14
+        if (ret) {
             fprintf(stderr,
-                    "Failed to open /dev/null for OFD lock probing: %s\n",
-                    strerror(errno));
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
-            return;
-        }
-        ret = fcntl(fd, F_OFD_GETLK, &fl);
-        close(fd);
-        if (!ret) {
-            fcntl_op_setlk = F_OFD_SETLK;
-            fcntl_op_getlk = F_OFD_GETLK;
-        } else {
-            fcntl_op_setlk = F_SETLK;
-            fcntl_op_getlk = F_GETLK;
+                    "Failed to fcntl for OFD lock probing.\n");
+            qemu_close(fd);
+            return false;
         }
+    }
+
+    ret = fcntl(fd, F_OFD_GETLK, &fl);
+    qemu_close(fd);
Quoted 69 lines
Quoted 3 lines

F_OFD_GETLK doesn't modify the state, so it seems to me that even the qemu_dup() is unnecessary and we could just directly try F_OFD_GETLK on the passed file descriptor (orig_fd).

OK, I'll change to try F_OFD_GETLK of orig_fd directly.

Quoted 63 lines

util/osdep.c static int qemu_parse_fdset(const char *param) · line 217

Quoted 7 lines
util/osdep.c +6 -0
+
+    if (ret == 0) {
+        return true;
+    } else {
+        return false;
+    }

This should be written shorter as return ret == 0;

Quoted 38 lines
util/osdep.c +23 -9
 #else
-        fcntl_op_setlk = F_SETLK;
-        fcntl_op_getlk = F_GETLK;
+    return false;
 #endif
-    }
 }
 
-bool qemu_has_ofd_lock(void)
-{
-    qemu_probe_lock_ops();
 #ifdef F_OFD_SETLK
-    return fcntl_op_setlk == F_OFD_SETLK;
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+    bool ofd_lock = true;
+
+    do {
+        if (ofd_lock) {
+            ret = fcntl(fd, F_OFD_SETLK, fl);
+            if ((ret == -1) && (errno == EINVAL)) {
+                ofd_lock = false;
+            }
+        }
+
+        if (!ofd_lock) {
+            /* Fallback to POSIX lock */
+            ret = fcntl(fd, F_SETLK, fl);
+        }
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
+}
 #else
-    return false;
-#endif
+static int _qemu_lock_fcntl(int fd, struct flock *fl)

util/osdep.c static int qemu_parse_fdset(const char *param) · line 256

Quoted 10 lines
util/osdep.c +9 -0
+{
+    int ret;
+
+    do {
+        ret = fcntl(fd, F_SETLK, fl);
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
 }
+#endif

The logic looks fine to me, at least assuming that EINVAL is really what we will consistently get from the kernel if OFD locks are not supported. Is this documented anywhere? The fcntl manpage doesn't seem to mention this case.

The man page of fcntl(2) says:

EINVAL The value specified in cmd is not recognized by this kernel.

So I think EINVAL is good enough to check whether the filesystem supports OFD locks or not...

Thanks,
Masa

Quoted 113 lines
Quoted 4 lines

Anyway, I think I would try to minimise the duplication by having only a small #ifdef section inside the function, maybe like this:

Quoted 8 lines
#ifdef F_OFD_SETLK
            ret = fcntl(fd, F_OFD_SETLK, fl);
            if ((ret == -1) && (errno == EINVAL)) {
                ofd_lock = false;
            }
#else
            ofd_lock = false;

#endif

Great! I'll make this.

Quoted 37 lines
Quoted 24 lines
util/osdep.c +2 -6
 static int qemu_lock_fcntl(int fd, int64_t start, int64_t len, int fl_type)
 {
-    int ret;
     struct flock fl = {
         .l_whence = SEEK_SET,
         .l_start  = start,
         .l_len    = len,
         .l_type   = fl_type,
     };
-    qemu_probe_lock_ops();
-    do {
-        ret = fcntl(fd, fcntl_op_setlk, &fl);
-    } while (ret == -1 && errno == EINTR);
-    return ret == -1 ? -errno : 0;
+
+    return _qemu_lock_fcntl(fd, &fl);
 }
 
 int qemu_lock_fd(int fd, int64_t start, int64_t len, bool exclusive)
@@ -261,22 +277,49 @@ int qemu_unlock_fd(int fd, int64_t start, int64_t len)
     return qemu_lock_fcntl(fd, start, len, F_UNLCK);
 }
 

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 288

Quoted 10 lines
util/osdep.c +7 -1
-int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive)
+#ifdef F_OFD_SETLK
+static int _qemu_lock_fd_test(int fd, struct flock *fl)
 {
     int ret;
+
+    ret = fcntl(fd, F_OFD_GETLK, fl);
+    if ((ret == -1) && (errno != EINVAL)) {
+        return -errno;
+

Please remove this empty line.

Quoted 2 lines

The parentheses in the condition (above and below) are not stricly necessary.

Got it.

Quoted 44 lines
Quoted 14 lines
util/osdep.c +13 -0
+    } else if ((ret == -1) && (errno == EINVAL)) {
+        /* Fallback to POSIX lock */
+        ret = fcntl(fd, F_GETLK, fl);
+        if (ret == -1) {
+            return -errno;
+        }
+    }
+
+    return fl->l_type == F_UNLCK ? 0 : -EAGAIN;
+}
+#else
+static int _qemu_lock_fd_test(int fd, struct flock *fl)
+{

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 311

Quoted 10 lines
util/osdep.c +10 -0
+    int ret;
+
+    ret = fcntl(fd, F_GETLK, fl);
+    if (ret == -1) {
+        return -errno;
+    } else {
+        return fl->l_type == F_UNLCK ? 0 : -EAGAIN;
+    }
+}
+#endif

Same idea as above: #ifdef only around the fcntl(F_OFD_GETLK) call can minimise the code duplication.

Quoted 16 lines
util/osdep.c +3 -7
+int qemu_lock_fd_test(int fd, int64_t start, int64_t len, bool exclusive)
+{
     struct flock fl = {
         .l_whence = SEEK_SET,
         .l_start  = start,
         .l_len    = len,
         .l_type   = exclusive ? F_WRLCK : F_RDLCK,
     };
-    qemu_probe_lock_ops();
-    ret = fcntl(fd, fcntl_op_getlk, &fl);
-    if (ret == -1) {
-        return -errno;
-    } else {
-        return fl.l_type == F_UNLCK ? 0 : -EAGAIN;
-    }
+
Quoted 8 lines

util/osdep.c int qemu_unlock_fd(int fd, int64_t start, int64_t len) · line 324

Quoted 3 lines
util/osdep.c +1 -0
+    return _qemu_lock_fd_test(fd, &fl);
 }
 #endif

After moving the #ifdef into the function, you can inline _qemu_lock_fd_test() and and _qemu_lock_fcntl() again. This is also good because identifiers starting with an underscore are reserved in the C standard.

Got it, thanks! I'll post v2.

- Masa

#10 lore raw

Am 18.11.2020 um 20:48 hat Masayoshi Mizuma geschrieben:

Quoted 66 lines · Masayoshi Mizuma
Quoted 56 lines · Kevin Wolf
Quoted 55 lines

Am 06.11.2020 um 05:01 hat Masayoshi Mizuma geschrieben:

Quoted 54 lines

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

util/osdep.c +29 -6
-bool qemu_has_ofd_lock(void)
-{
-    qemu_probe_lock_ops();
 #ifdef F_OFD_SETLK
-    return fcntl_op_setlk == F_OFD_SETLK;
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+    bool ofd_lock = true;
+
+    do {
+        if (ofd_lock) {
+            ret = fcntl(fd, F_OFD_SETLK, fl);
+            if ((ret == -1) && (errno == EINVAL)) {
+                ofd_lock = false;
+            }
+        }
+
+        if (!ofd_lock) {
+            /* Fallback to POSIX lock */
+            ret = fcntl(fd, F_SETLK, fl);
+        }
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
+}
 #else
-    return false;
-#endif
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+
+    do {
+        ret = fcntl(fd, F_SETLK, fl);
+    } while (ret == -1 && errno == EINTR);
+
Quoted 8 lines
Quoted 8 lines

util/osdep.c static int qemu_parse_fdset(const char *param) · line 256

Quoted 3 lines
util/osdep.c +2 -0
+    return ret == -1 ? -errno : 0;
 }
+#endif

The logic looks fine to me, at least assuming that EINVAL is really what we will consistently get from the kernel if OFD locks are not supported. Is this documented anywhere? The fcntl manpage doesn't seem to mention this case.

Quoted 6 lines

The man page of fcntl(2) says:

EINVAL The value specified in cmd is not recognized by this kernel.

So I think EINVAL is good enough to check whether the filesystem supports OFD locks or not...

A kernel not knowing the cmd at all is a somewhat different case (and certainly a different code path) than a filesystem not supporting it.

I just had a look at the kernel code, and to me it seems that the difference between POSIX locks and OFD locks is handled entirely in filesystem independent code. A filesystem driver would in theory have ways to distinguish both, but I don't see any driver in the kernel tree that actually does this (and there is probably little reason for a driver to do so).

So now I wonder what filesystem you are using? I'm curious what I missed.

Kevin

#11 lore raw
Quoted 84 lines · Kevin Wolf

Am 18.11.2020 um 20:48 hat Masayoshi Mizuma geschrieben:

Quoted 66 lines · Masayoshi Mizuma
Quoted 56 lines · Kevin Wolf
Quoted 55 lines

Am 06.11.2020 um 05:01 hat Masayoshi Mizuma geschrieben:

Quoted 54 lines

From: Masayoshi Mizuma <[email protected]>

locking=auto doesn't work if the filesystem doesn't support OFD lock. In that situation, following error happens:

qemu-system-x86_64: -blockdev driver=qcow2,node-name=disk,file.driver=file,file.filename=/mnt/guest.qcow2,file.locking=auto: Failed to lock byte 100

qemu_probe_lock_ops() judges whether qemu can use OFD lock or not with doing fcntl(F_OFD_GETLK) to /dev/null. So the error happens if /dev/null supports OFD lock, but the filesystem doesn't support the lock.

Lock the actual file, not /dev/null, using F_OFD_SETLK and if that fails, then fallback to F_SETLK.

util/osdep.c +29 -6
-bool qemu_has_ofd_lock(void)
-{
-    qemu_probe_lock_ops();
 #ifdef F_OFD_SETLK
-    return fcntl_op_setlk == F_OFD_SETLK;
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+    bool ofd_lock = true;
+
+    do {
+        if (ofd_lock) {
+            ret = fcntl(fd, F_OFD_SETLK, fl);
+            if ((ret == -1) && (errno == EINVAL)) {
+                ofd_lock = false;
+            }
+        }
+
+        if (!ofd_lock) {
+            /* Fallback to POSIX lock */
+            ret = fcntl(fd, F_SETLK, fl);
+        }
+    } while (ret == -1 && errno == EINTR);
+
+    return ret == -1 ? -errno : 0;
+}
 #else
-    return false;
-#endif
+static int _qemu_lock_fcntl(int fd, struct flock *fl)
+{
+    int ret;
+
+    do {
+        ret = fcntl(fd, F_SETLK, fl);
+    } while (ret == -1 && errno == EINTR);
+
Quoted 8 lines
Quoted 8 lines

util/osdep.c static int qemu_parse_fdset(const char *param) · line 256

Quoted 3 lines
util/osdep.c +2 -0
+    return ret == -1 ? -errno : 0;
 }
+#endif

The logic looks fine to me, at least assuming that EINVAL is really what we will consistently get from the kernel if OFD locks are not supported. Is this documented anywhere? The fcntl manpage doesn't seem to mention this case.

Quoted 6 lines

The man page of fcntl(2) says:

EINVAL The value specified in cmd is not recognized by this kernel.

So I think EINVAL is good enough to check whether the filesystem supports OFD locks or not...

A kernel not knowing the cmd at all is a somewhat different case (and certainly a different code path) than a filesystem not supporting it.

I just had a look at the kernel code, and to me it seems that the difference between POSIX locks and OFD locks is handled entirely in filesystem independent code. A filesystem driver would in theory have ways to distinguish both, but I don't see any driver in the kernel tree that actually does this (and there is probably little reason for a driver to do so).

Quoted 2 lines

So now I wonder what filesystem you are using? I'm curious what I missed.

I'm using a proprietary filesystem, which isn't in the linux kernel. The filesystem supports posix lock only, doesn't support OFD lock...

Thanks,
Masa

#12 lore raw

Am 20.11.2020 um 00:56 hat Masayoshi Mizuma geschrieben:

Quoted 26 lines · Kevin Wolf
Quoted 25 lines

Am 18.11.2020 um 20:48 hat Masayoshi Mizuma geschrieben:

Quoted 7 lines · Masayoshi Mizuma
Quoted 5 lines · Kevin Wolf
Quoted 4 lines

The logic looks fine to me, at least assuming that EINVAL is really what we will consistently get from the kernel if OFD locks are not supported. Is this documented anywhere? The fcntl manpage doesn't seem to mention this case.

Quoted 6 lines

The man page of fcntl(2) says:

EINVAL The value specified in cmd is not recognized by this kernel.

So I think EINVAL is good enough to check whether the filesystem supports OFD locks or not...

A kernel not knowing the cmd at all is a somewhat different case (and certainly a different code path) than a filesystem not supporting it.

I just had a look at the kernel code, and to me it seems that the difference between POSIX locks and OFD locks is handled entirely in filesystem independent code. A filesystem driver would in theory have ways to distinguish both, but I don't see any driver in the kernel tree that actually does this (and there is probably little reason for a driver to do so).

Quoted 5 lines
Quoted 2 lines

So now I wonder what filesystem you are using? I'm curious what I missed.

I'm using a proprietary filesystem, which isn't in the linux kernel. The filesystem supports posix lock only, doesn't support OFD lock...

Do you know why that proprietary filesystem driver makes a difference between POSIX locks and OFD locks? The main difference between both types is when they are released automatically, and this is handled by generic kernel code and not the filesystem driver.

From a filesystem perspective, I don't see any reason to even distuingish. So unless there are good reasons for making the distinction, I'm currently inclined to view this as a filesystem driver bug.

It makes handling this case hard because when the case isn't even supposed to exist, of course there won't be a defined error code.

Kevin

#13 lore raw
Quoted 38 lines · Kevin Wolf

Am 20.11.2020 um 00:56 hat Masayoshi Mizuma geschrieben:

Quoted 26 lines · Kevin Wolf
Quoted 25 lines

Am 18.11.2020 um 20:48 hat Masayoshi Mizuma geschrieben:

Quoted 7 lines · Masayoshi Mizuma
Quoted 5 lines · Kevin Wolf
Quoted 4 lines

The logic looks fine to me, at least assuming that EINVAL is really what we will consistently get from the kernel if OFD locks are not supported. Is this documented anywhere? The fcntl manpage doesn't seem to mention this case.

Quoted 6 lines

The man page of fcntl(2) says:

EINVAL The value specified in cmd is not recognized by this kernel.

So I think EINVAL is good enough to check whether the filesystem supports OFD locks or not...

A kernel not knowing the cmd at all is a somewhat different case (and certainly a different code path) than a filesystem not supporting it.

I just had a look at the kernel code, and to me it seems that the difference between POSIX locks and OFD locks is handled entirely in filesystem independent code. A filesystem driver would in theory have ways to distinguish both, but I don't see any driver in the kernel tree that actually does this (and there is probably little reason for a driver to do so).

Quoted 5 lines
Quoted 2 lines

So now I wonder what filesystem you are using? I'm curious what I missed.

I'm using a proprietary filesystem, which isn't in the linux kernel. The filesystem supports posix lock only, doesn't support OFD lock...

Do you know why that proprietary filesystem driver makes a difference between POSIX locks and OFD locks? The main difference between both types is when they are released automatically, and this is handled by generic kernel code and not the filesystem driver.

Quoted 7 lines

From a filesystem perspective, I don't see any reason to even distuingish. So unless there are good reasons for making the distinction, I'm currently inclined to view this as a filesystem driver bug.

It makes handling this case hard because when the case isn't even supposed to exist, of course there won't be a defined error code.

Hi Kevin,

The filesystem team found a locking issue in the filesystem. Your comments were very helpful! I really appriciate it.

Thanks,
Masa

#14 lore raw

Hi Masa,

Am 10.02.2021 um 17:43 hat Masayoshi Mizuma geschrieben:

Quoted 7 lines

Hi Kevin,

The filesystem team found a locking issue in the filesystem. Your comments were very helpful! I really appriciate it.

Thanks,
Masa

I'm glad that I could help you to find the root cause. Thanks for reporting back!

Kevin