Skip to contentSkip to conversation

ceph: enable async dirops by default

PATCH In review Jeff Layton 3 participants 4 messages Aug 9, 2021 – Sep 7, 2021 Ceph client

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ceph: enable async dirops by default

Async dirops have been supported in mainline kernels for quite some time now, and we've recently (as of June) started doing regular testing in teuthology with '-o nowsync'. So far, that hasn't uncovered any issues, so I think the time is right to flip the default for this option.

Enable async dirops by default, and change /proc/mounts to show "wsync" when they are disabled rather than "nowsync" when they are enabled.

Cc Patrick Donnelly <[email protected]>
---
 fs/ceph/super.c | 4 ++--
 fs/ceph/super.h | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/ceph/super.c +2 -2
index 609ffc8c2d78..f517ad9eeb26 100644
@@ -698,8 +698,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)
 	if (fsopt->flags & CEPH_MOUNT_OPT_CLEANRECOVER)
 		seq_show_option(m, "recover_session", "clean");
 
-	if (fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS)
-		seq_puts(m, ",nowsync");
+	if (!(fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS))
+		seq_puts(m, ",wsync");
 
 	if (fsopt->wsize != CEPH_MAX_WRITE_SIZE)
 		seq_printf(m, ",wsize=%u", fsopt->wsize);
fs/ceph/super.h +2 -1
index 389b45ac291b..0bc36cf4c683 100644
@@ -48,7 +48,8 @@
 
 #define CEPH_MOUNT_OPT_DEFAULT			\
 	(CEPH_MOUNT_OPT_DCACHE |		\
-	 CEPH_MOUNT_OPT_NOCOPYFROM)
+	 CEPH_MOUNT_OPT_NOCOPYFROM |		\
+	 CEPH_MOUNT_OPT_ASYNC_DIROPS)
 
 #define ceph_set_mount_opt(fsc, opt) \
 	(fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt
Signature · 1 line
2.31.1
#1 lore raw
Quoted 35 lines · Jeff Layton

Async dirops have been supported in mainline kernels for quite some time now, and we've recently (as of June) started doing regular testing in teuthology with '-o nowsync'. So far, that hasn't uncovered any issues, so I think the time is right to flip the default for this option.

Enable async dirops by default, and change /proc/mounts to show "wsync" when they are disabled rather than "nowsync" when they are enabled.

Cc Patrick Donnelly <[email protected]>
---
 fs/ceph/super.c | 4 ++--
 fs/ceph/super.h | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/ceph/super.c +2 -2
index 609ffc8c2d78..f517ad9eeb26 100644
@@ -698,8 +698,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)
 	if (fsopt->flags & CEPH_MOUNT_OPT_CLEANRECOVER)
 		seq_show_option(m, "recover_session", "clean");
 
-	if (fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS)
-		seq_puts(m, ",nowsync");
+	if (!(fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS))
+		seq_puts(m, ",wsync");
 
 	if (fsopt->wsize != CEPH_MAX_WRITE_SIZE)
 		seq_printf(m, ",wsize=%u", fsopt->wsize);
fs/ceph/super.h +0 -0
index 389b45ac291b..0bc36cf4c683 100644

fs/ceph/super.h line 55

Quoted 9 lines
fs/ceph/super.h +2 -1
@@ -48,7 +48,8 @@
 
 #define CEPH_MOUNT_OPT_DEFAULT			\
 	(CEPH_MOUNT_OPT_DCACHE |		\
-	 CEPH_MOUNT_OPT_NOCOPYFROM)
+	 CEPH_MOUNT_OPT_NOCOPYFROM |		\
+	 CEPH_MOUNT_OPT_ASYNC_DIROPS)
 
 #define ceph_set_mount_opt(fsc, opt) \
 	(fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt

I think we ought to wait to merge this into mainline just yet, but I'd like to leave it in the "testing" branch for now.

I've been working this bug with Patrick for the last month or two:

    https://tracker.ceph.com/issues/51279

In at least one case, the problem seems to be that the MDS failed an async create with an ENOSPC error. The kclient's error handling around this is pretty non-existent right now, so it caused an unmount to hang.

It's a pity we still don't have revoke()...

I've started working on a series to clean this up, but in the meantime I think we ought to wait until that's in place before we make nowsync the default.

Sound ok?

Signature · 1 line
Jeff Layton <[email protected]>
maintainer
#2 lore raw
Quoted 58 lines · Jeff Layton
Quoted 35 lines · Jeff Layton

Async dirops have been supported in mainline kernels for quite some time now, and we've recently (as of June) started doing regular testing in teuthology with '-o nowsync'. So far, that hasn't uncovered any issues, so I think the time is right to flip the default for this option.

Enable async dirops by default, and change /proc/mounts to show "wsync" when they are disabled rather than "nowsync" when they are enabled.

Cc Patrick Donnelly <[email protected]>
---
 fs/ceph/super.c | 4 ++--
 fs/ceph/super.h | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/ceph/super.c +2 -2
index 609ffc8c2d78..f517ad9eeb26 100644
@@ -698,8 +698,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)
      if (fsopt->flags & CEPH_MOUNT_OPT_CLEANRECOVER)
              seq_show_option(m, "recover_session", "clean");
 
-     if (fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS)
-             seq_puts(m, ",nowsync");
+     if (!(fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS))
+             seq_puts(m, ",wsync");
 
      if (fsopt->wsize != CEPH_MAX_WRITE_SIZE)
              seq_printf(m, ",wsize=%u", fsopt->wsize);
fs/ceph/super.h +0 -0
index 389b45ac291b..0bc36cf4c683 100644

fs/ceph/super.h line 55

Quoted 9 lines
fs/ceph/super.h +2 -1
@@ -48,7 +48,8 @@
 
 #define CEPH_MOUNT_OPT_DEFAULT                       \
      (CEPH_MOUNT_OPT_DCACHE |                \
-      CEPH_MOUNT_OPT_NOCOPYFROM)
+      CEPH_MOUNT_OPT_NOCOPYFROM |            \
+      CEPH_MOUNT_OPT_ASYNC_DIROPS)
 
 #define ceph_set_mount_opt(fsc, opt) \
      (fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt

I think we ought to wait to merge this into mainline just yet, but I'd like to leave it in the "testing" branch for now.

I've been working this bug with Patrick for the last month or two:

    https://tracker.ceph.com/issues/51279

In at least one case, the problem seems to be that the MDS failed an async create with an ENOSPC error. The kclient's error handling around this is pretty non-existent right now, so it caused an unmount to hang.

Quoted 7 lines

It's a pity we still don't have revoke()...

I've started working on a series to clean this up, but in the meantime I think we ought to wait until that's in place before we make nowsync the default.

Sound ok?

Yeah, I don't see any rush to make it the default especially given that it hasn't been on in testing that long.

Thanks,

Ilya

#3 lore raw
In reply to Jeff Layton
Quoted 58 lines · Jeff Layton
Quoted 35 lines · Jeff Layton

Async dirops have been supported in mainline kernels for quite some time now, and we've recently (as of June) started doing regular testing in teuthology with '-o nowsync'. So far, that hasn't uncovered any issues, so I think the time is right to flip the default for this option.

Enable async dirops by default, and change /proc/mounts to show "wsync" when they are disabled rather than "nowsync" when they are enabled.

Cc Patrick Donnelly <[email protected]>
---
 fs/ceph/super.c | 4 ++--
 fs/ceph/super.h | 3 ++-
 2 files changed, 4 insertions(+), 3 deletions(-)
fs/ceph/super.c +2 -2
index 609ffc8c2d78..f517ad9eeb26 100644
@@ -698,8 +698,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)
      if (fsopt->flags & CEPH_MOUNT_OPT_CLEANRECOVER)
              seq_show_option(m, "recover_session", "clean");
 
-     if (fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS)
-             seq_puts(m, ",nowsync");
+     if (!(fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS))
+             seq_puts(m, ",wsync");
 
      if (fsopt->wsize != CEPH_MAX_WRITE_SIZE)
              seq_printf(m, ",wsize=%u", fsopt->wsize);
fs/ceph/super.h +0 -0
index 389b45ac291b..0bc36cf4c683 100644

fs/ceph/super.h line 55

Quoted 9 lines
fs/ceph/super.h +2 -1
@@ -48,7 +48,8 @@
 
 #define CEPH_MOUNT_OPT_DEFAULT                       \
      (CEPH_MOUNT_OPT_DCACHE |                \
-      CEPH_MOUNT_OPT_NOCOPYFROM)
+      CEPH_MOUNT_OPT_NOCOPYFROM |            \
+      CEPH_MOUNT_OPT_ASYNC_DIROPS)
 
 #define ceph_set_mount_opt(fsc, opt) \
      (fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt

I think we ought to wait to merge this into mainline just yet, but I'd like to leave it in the "testing" branch for now.

I've been working this bug with Patrick for the last month or two:

    https://tracker.ceph.com/issues/51279

In at least one case, the problem seems to be that the MDS failed an async create with an ENOSPC error. The kclient's error handling around this is pretty non-existent right now, so it caused an unmount to hang.

Quoted 7 lines

It's a pity we still don't have revoke()...

I've started working on a series to clean this up, but in the meantime I think we ought to wait until that's in place before we make nowsync the default.

Sound ok?

Sounds good to me.

Signature · 5 lines
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D