ceph: enable async dirops by default
PATCH 1/1 ceph: enable async dirops by default
This has been behaving reasonably well in testing, and enabling this offers significant performance benefits. Enable async dirops by default in the kclient going forward, and change show_options to add "wsync" when they are disabled.
--- fs/ceph/super.c | 4 ++-- fs/ceph/super.h | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-)
fs/ceph/super.c +2 -2
@@ -577,8 +577,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)if (fsopt->flags &CEPH_MOUNT_OPT_CLEANRECOVER )seq_show_option (m,"recover_session" ,"clean" ); -if (fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS ) -seq_puts (m,",nowsync" ); +if (!(fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS )) +seq_puts (m,",wsync" );if (fsopt->wsize !=CEPH_MAX_WRITE_SIZE )seq_printf (m,",wsize=%u" , fsopt->wsize);
fs/ceph/super.h +3 -2
@@ -46,8 +46,9 @@#define CEPH_MOUNT_OPT_ASYNC_DIROPS (1<<15)/* allow async directory ops */ #define CEPH_MOUNT_OPT_DEFAULT \ - (CEPH_MOUNT_OPT_DCACHE | \ - CEPH_MOUNT_OPT_NOCOPYFROM) + (CEPH_MOUNT_OPT_DCACHE | \ + CEPH_MOUNT_OPT_NOCOPYFROM | \ + CEPH_MOUNT_OPT_ASYNC_DIROPS)#define ceph_set_mount_opt (fsc, opt) \ (fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt
Signature · 1 line
2.29.2
Ilya Dryomov
maintainerQuoted 44 lines · Jeff Layton
This has been behaving reasonably well in testing, and enabling this offers significant performance benefits. Enable async dirops by default in the kclient going forward, and change
show_optionsto add "wsync" when they are disabled.Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/super.c | 4 ++-- fs/ceph/super.h | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-)fs/ceph/super.c +2
-2@@ -577,8 +577,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)if (fsopt->flags &CEPH_MOUNT_OPT_CLEANRECOVER )seq_show_option (m,"recover_session" ,"clean" ); -if (fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS ) -seq_puts (m,",nowsync" ); +if (!(fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS )) +seq_puts (m,",wsync" );if (fsopt->wsize !=CEPH_MAX_WRITE_SIZE )seq_printf (m,",wsize=%u" , fsopt->wsize);fs/ceph/super.h +3
-3@@ -46,8 +46,9 @@#define CEPH_MOUNT_OPT_ASYNC_DIROPS (1<<15)/* allow async directory ops */ #define CEPH_MOUNT_OPT_DEFAULT \ - (CEPH_MOUNT_OPT_DCACHE | \ - CEPH_MOUNT_OPT_NOCOPYFROM) + (CEPH_MOUNT_OPT_DCACHE | \ + CEPH_MOUNT_OPT_NOCOPYFROM | \ + CEPH_MOUNT_OPT_ASYNC_DIROPS)#define ceph_set_mount_opt (fsc, opt) \ (fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt --
Quoted 2 lines
2.29.2
Hi Jeff,
Is it being tested by teuthology? I see commit 4181742a3ba8 ("qa:
allow arbitrary mount options on kclient mounts"), but nothing beyond that. I think "nowsync" needs to be turned on and get at least some nightly coverage before the default is flipped.
Thanks,
Ilya
Quoted 50 lines · Ilya Dryomov
Quoted 42 lines · Jeff Layton
This has been behaving reasonably well in testing, and enabling this offers significant performance benefits. Enable async dirops by default in the kclient going forward, and change
show_optionsto add "wsync" when they are disabled.Signed-off-by Jeff Layton <[email protected]>---
fs/ceph/super.c | 4 ++--
fs/ceph/super.h | 5 +++--2 files changed, 5 insertions(+), 4 deletions(-)fs/ceph/super.c +2
-2@@ -577,8 +577,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root) if (fsopt->flags & CEPH_MOUNT_OPT_CLEANRECOVER) seq_show_option(m, "recover_session", "clean"); -if (fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS ) -seq_puts (m,",nowsync" ); +if (!(fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS )) +seq_puts (m,",wsync" ); if (fsopt->wsize != CEPH_MAX_WRITE_SIZE) seq_printf(m, ",wsize=%u", fsopt->wsize);fs/ceph/super.h +3
-2@@ -46,8 +46,9 @@ #define CEPH_MOUNT_OPT_ASYNC_DIROPS (1<<15) /* allow async directory ops */ #define CEPH_MOUNT_OPT_DEFAULT \ - (CEPH_MOUNT_OPT_DCACHE | \ - CEPH_MOUNT_OPT_NOCOPYFROM) + (CEPH_MOUNT_OPT_DCACHE | \ + CEPH_MOUNT_OPT_NOCOPYFROM | \ + CEPH_MOUNT_OPT_ASYNC_DIROPS) #define ceph_set_mount_opt(fsc, opt) \Quoted 4 lines
(fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt --
2.29.2Hi Jeff,
Quoted 8 lines
Is it being tested by teuthology? I see commit
4181742a3ba8("qa:
allow arbitrary mount options on kclient mounts"), but nothing beyond that. I think "nowsync" needs to be turned on and get at least some nightly coverage before the default is flipped.Thanks,
Ilya
Good point. I had thought Patrick had added a qa variant that turned that on, but I don't think that ever got merged. We definitely need that enabled in QA before we make this the default.
The catch is that we probably don't _always_ want nowsync enabled, so is there some way to randomize this? Or do we need some sort of yaml file that turns this on by request? What should we be aiming to do for this?
How are the existing, more obscure mount options usually tested in teuthology today?
Signature · 1 line
Jeff Layton <[email protected]>
Ilya Dryomov
maintainerQuoted 61 lines · Jeff Layton
Quoted 50 lines · Ilya Dryomov
Quoted 44 lines · Jeff Layton
This has been behaving reasonably well in testing, and enabling this offers significant performance benefits. Enable async dirops by default in the kclient going forward, and change
show_optionsto add "wsync" when they are disabled.Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/super.c | 4 ++-- fs/ceph/super.h | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-)fs/ceph/super.c +2
-2@@ -577,8 +577,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)if (fsopt->flags &CEPH_MOUNT_OPT_CLEANRECOVER )seq_show_option (m,"recover_session" ,"clean" ); -if (fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS ) -seq_puts (m,",nowsync" ); +if (!(fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS )) +seq_puts (m,",wsync" );if (fsopt->wsize !=CEPH_MAX_WRITE_SIZE )seq_printf (m,",wsize=%u" , fsopt->wsize);fs/ceph/super.h +3
-3@@ -46,8 +46,9 @@#define CEPH_MOUNT_OPT_ASYNC_DIROPS (1<<15)/* allow async directory ops */ #define CEPH_MOUNT_OPT_DEFAULT \ - (CEPH_MOUNT_OPT_DCACHE | \ - CEPH_MOUNT_OPT_NOCOPYFROM) + (CEPH_MOUNT_OPT_DCACHE | \ + CEPH_MOUNT_OPT_NOCOPYFROM | \ + CEPH_MOUNT_OPT_ASYNC_DIROPS)#define ceph_set_mount_opt (fsc, opt) \ (fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt --Quoted 2 lines
2.29.2
Hi Jeff,
Quoted 8 lines
Is it being tested by teuthology? I see commit
4181742a3ba8("qa:
allow arbitrary mount options on kclient mounts"), but nothing beyond that. I think "nowsync" needs to be turned on and get at least some nightly coverage before the default is flipped.Thanks,
Ilya
Quoted 7 lines
Good point. I had thought Patrick had added a qa variant that turned that on, but I don't think that ever got merged. We definitely need that enabled in QA before we make this the default.
The catch is that we probably don't _always_ want nowsync enabled, so is there some way to randomize this? Or do we need some sort of yaml file that turns this on by request? What should we be aiming to do for this?
It can be randomized, you can choose to run a particular set of jobs with both wsync and nowsync and the rest only with wsync or the other way around, etc. Completely up to you, depending on the sort of coverage you want to get (weighted by the number of jobs added to the suite).
Thanks,
Ilya
Quoted 71 lines · Ilya Dryomov
Quoted 61 lines · Jeff Layton
Quoted 50 lines · Ilya Dryomov
Quoted 44 lines · Jeff Layton
This has been behaving reasonably well in testing, and enabling this offers significant performance benefits. Enable async dirops by default in the kclient going forward, and change
show_optionsto add "wsync" when they are disabled.Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/super.c | 4 ++-- fs/ceph/super.h | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-)fs/ceph/super.c +2
-2@@ -577,8 +577,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)if (fsopt->flags &CEPH_MOUNT_OPT_CLEANRECOVER )seq_show_option (m,"recover_session" ,"clean" ); -if (fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS ) -seq_puts (m,",nowsync" ); +if (!(fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS )) +seq_puts (m,",wsync" );if (fsopt->wsize !=CEPH_MAX_WRITE_SIZE )seq_printf (m,",wsize=%u" , fsopt->wsize);fs/ceph/super.h +3
-3@@ -46,8 +46,9 @@#define CEPH_MOUNT_OPT_ASYNC_DIROPS (1<<15)/* allow async directory ops */ #define CEPH_MOUNT_OPT_DEFAULT \ - (CEPH_MOUNT_OPT_DCACHE | \ - CEPH_MOUNT_OPT_NOCOPYFROM) + (CEPH_MOUNT_OPT_DCACHE | \ + CEPH_MOUNT_OPT_NOCOPYFROM | \ + CEPH_MOUNT_OPT_ASYNC_DIROPS)#define ceph_set_mount_opt (fsc, opt) \ (fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt --Quoted 2 lines
2.29.2
Hi Jeff,
Quoted 8 lines
Is it being tested by teuthology? I see commit
4181742a3ba8("qa:
allow arbitrary mount options on kclient mounts"), but nothing beyond that. I think "nowsync" needs to be turned on and get at least some nightly coverage before the default is flipped.Thanks,
Ilya
Quoted 7 lines
Good point. I had thought Patrick had added a qa variant that turned that on, but I don't think that ever got merged. We definitely need that enabled in QA before we make this the default.
The catch is that we probably don't _always_ want nowsync enabled, so is there some way to randomize this? Or do we need some sort of yaml file that turns this on by request? What should we be aiming to do for this?
Quoted 5 lines
It can be randomized, you can choose to run a particular set of jobs with both wsync and nowsync and the rest only with wsync or the other way around, etc. Completely up to you, depending on the sort of coverage you want to get (weighted by the number of jobs added to the suite).
I wrote up a quick PR for this: https://github.com/ceph/ceph/pull/39505
Signature · 5 lines
Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
Quoted 4 lines · Jeff Layton
This has been behaving reasonably well in testing, and enabling this offers significant performance benefits. Enable async dirops by default in the kclient going forward, and change
show_optionsto add "wsync" when they are disabled.
Sounds good! Maybe it's worth adding this option to the mount options listed in Documentation/filesystems/ceph.rst (although it's quite possible that other mount options are already missing there).
Cheers,
Signature · 1 line
Luis
Quoted 42 lines
Signed-off-by Jeff Layton <[email protected]>--- fs/ceph/super.c | 4 ++-- fs/ceph/super.h | 5 +++-- 2 files changed, 5 insertions(+), 4 deletions(-)fs/ceph/super.c +2
-2@@ -577,8 +577,8 @@ static int ceph_show_options(struct seq_file *m, struct dentry *root)if (fsopt->flags &CEPH_MOUNT_OPT_CLEANRECOVER )seq_show_option (m,"recover_session" ,"clean" ); -if (fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS ) -seq_puts (m,",nowsync" ); +if (!(fsopt->flags &CEPH_MOUNT_OPT_ASYNC_DIROPS )) +seq_puts (m,",wsync" );if (fsopt->wsize !=CEPH_MAX_WRITE_SIZE )seq_printf (m,",wsize=%u" , fsopt->wsize);fs/ceph/super.h +3
-2@@ -46,8 +46,9 @@#define CEPH_MOUNT_OPT_ASYNC_DIROPS (1<<15)/* allow async directory ops */ #define CEPH_MOUNT_OPT_DEFAULT \ - (CEPH_MOUNT_OPT_DCACHE | \ - CEPH_MOUNT_OPT_NOCOPYFROM) + (CEPH_MOUNT_OPT_DCACHE | \ + CEPH_MOUNT_OPT_NOCOPYFROM | \ + CEPH_MOUNT_OPT_ASYNC_DIROPS)#define ceph_set_mount_opt (fsc, opt) \ (fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##optSignature · 1 line
2.29.2