libceph: support for replica reads
Ilya Dryomov
maintainercover libceph: support for replica reads
Hello,
This adds support for replica reads (balanced and localized reads) to rbd and ceph. crush_location syntax is slightly different, see patch 3 for details.
Thanks,
Ilya
Ilya Dryomov (5):
libceph: add non-asserting rbtree insertion helper
libceph: decode CRUSH device/bucket types and names
libceph: crush_location infrastructure
libceph: support for balanced and localized reads
libceph: read_policy option
include/linux/ceph/libceph.h | 13 +- include/linux/ceph/osd_client.h | 1 + include/linux/ceph/osdmap.h | 19 +- include/linux/crush/crush.h | 6 + net/ceph/ceph_common.c | 51 +++++ net/ceph/crush/crush.c | 3 + net/ceph/debugfs.c | 6 +- net/ceph/osd_client.c | 92 +++++++- net/ceph/osdmap.c | 361 +++++++++++++++++++++++++++----- 9 files changed, 491 insertions(+), 61 deletions(-)
Signature · 1 line
2.19.2
Ilya Dryomov
maintainerPATCH 1/5 libceph: add non-asserting rbtree insertion helper
Needed for the next commit and useful for ceph_pg_pool_info tree as well. I'm leaving the asserting helper in for now, but we should look at getting rid of it in the future.
--- include/linux/ceph/libceph.h | 10 ++++-- net/ceph/osdmap.c | 60 +++++++----------------------------- 2 files changed, 19 insertions(+), 51 deletions(-)
include/linux/ceph/libceph.h +8 -2
@@ -188,7 +188,7 @@ static inline int calc_pages_for(u64 off, u64 len)#define RB_CMP3WAY (a, b) ((a) < (b) ? -1 : (a) > (b))#define DEFINE_RB_INSDEL_FUNCS2 (name, type, keyfld, cmpexp, keyexp, nodefld) \ -static void insert_##name(struct rb_root *root, type *t) \ +static bool __insert_##name(struct rb_root *root, type *t) \ { \ struct rb_node **n = &root->rb_node; \ struct rb_node *parent = NULL; \
@@ -206,11 +206,17 @@ static void insert_##name(struct rb_root *root, type *t) \else if (cmp >0 ) \ n = &(*n)->rb_right; \else \ -BUG (); \ +return false; \ } \ \rb_link_node (&t->nodefld, parent, n); \rb_insert_color (&t->nodefld, root); \ +return true; \ +} \ +static void __maybe_unused insert_##name(struct rb_root *root, type *t) \ +{ \ + if (!__insert_##name(root, t)) \ + BUG(); \ } \ static void erase_##name(struct rb_root *root, type *t) \ { \
net/ceph/osdmap.c +11 -49
113 lines not on this page · raw message
Signature · 1 line
2.19.2
Ilya Dryomov
maintainerPATCH 2/5 libceph: decode CRUSH device/bucket types and names
In reply to Ilya DryomovThese would be matched with the provided client location to calculate the locality value.
--- include/linux/crush/crush.h | 6 +++ net/ceph/crush/crush.c | 3 ++ net/ceph/osdmap.c | 85 ++++++++++++++++++++++++++++++++++++- 3 files changed, 92 insertions(+), 2 deletions(-)
include/linux/crush/crush.h +6 -0
@@ -301,6 +301,12 @@ struct crush_map {__u32 *choose_tries;#else +/* device/bucket type id -> type name (CrushWrapper::type_map) */ +struct rb_root type_names; + +/* device/bucket id -> name (CrushWrapper::name_map) */ +struct rb_root names; +/* CrushWrapper::choose_args */ struct rb_root choose_args;#endif
net/ceph/crush/crush.c +3 -0
@@ -2,6 +2,7 @@#ifdef __KERNEL__# include <linux/slab.h> # include <linux/crush/crush.h> +void clear_crush_names (struct rb_root *root);void clear_choose_args (struct crush_map *c);#else # include "crush_compat.h"
@@ -130,6 +131,8 @@ void crush_destroy(struct crush_map *map)#ifndef __KERNEL__kfree (map->choose_tries);#else +clear_crush_names (&map->type_names); +clear_crush_names (&map->names);clear_choose_args (map);#endif kfree (map);
net/ceph/osdmap.c +83 -2
103 lines not on this page · raw message
Signature · 1 line
2.19.2
Ilya Dryomov
maintainerPATCH 3/5 libceph: crush_location infrastructure
In reply to Ilya DryomovAllow expressing client's location in terms of CRUSH hierarchy as a set of (bucket type name, bucket name) pairs. The userspace syntax "crush_location = key1=value1 key2=value2" is incompatible with mount options and needed adaptation:
- ':' separator
- one key:value pair per crush_location option
- crush_location options are combined together
So for:
crush_location = host=foo rack=bar
one would write:
crush_location=host:foo,crush_location=rack:bar
As in userspace, "multipath" locations are supported, so indicating locality for parallel hierarchies is possible:
crush_location=rack:foo1,crush_location=rack:foo2,crush_location=datacenter:bar
--- include/linux/ceph/libceph.h | 1 + include/linux/ceph/osdmap.h | 16 ++++- net/ceph/ceph_common.c | 25 ++++++++ net/ceph/osdmap.c | 114 +++++++++++++++++++++++++++++++++++ 4 files changed, 155 insertions(+), 1 deletion(-)
include/linux/ceph/libceph.h +1 -0
@@ -64,6 +64,7 @@ struct ceph_options {int num_mon;char *name;struct ceph_crypto_key *key; +struct rb_root crush_locs; };/*
include/linux/ceph/osdmap.h +15 -1
@@ -302,9 +302,23 @@ bool ceph_pg_to_primary_shard(struct ceph_osdmap *osdmap,int ceph_pg_to_acting_primary (struct ceph_osdmap *osdmap,const struct ceph_pg *raw_pgid); +struct crush_loc { +char *cl_type_name; +char *cl_name; +}; + +struct crush_loc_node { +struct rb_node cl_node; +struct crush_loc cl_loc;/* pointers into cl_data */ +char cl_data[]; +}; + +int ceph_parse_crush_loc (const char *str,struct rb_root *locs); +int ceph_compare_crush_locs (struct rb_root *locs1,struct rb_root *locs2); +void ceph_clear_crush_locs (struct rb_root *locs); +extern struct ceph_pg_pool_info *ceph_pg_pool_by_id (struct ceph_osdmap *map,u64 id); -extern const char *ceph_pg_pool_name_by_id (struct ceph_osdmap *map,u64 id);extern int ceph_pg_poolid_by_name (struct ceph_osdmap *map,const char *name);u64 ceph_pg_pool_flags (struct ceph_osdmap *map,u64 id);
net/ceph/ceph_common.c +25 -0
@@ -176,6 +176,10 @@ int ceph_compare_options(struct ceph_options *new_opt, } } + ret =ceph_compare_crush_locs (&opt1->crush_locs, &opt2->crush_locs); +if (ret) +return ret; +/* any matching mon ip implies a match */ for (i =0 ; i < opt1->num_mon; i++) { if (ceph_monmap_contains(client->monc.monmap,
@@ -260,6 +264,7 @@ enum { Opt_secret, Opt_key, Opt_ip, + Opt_crush_location,/* string args above */ Opt_share, Opt_crc,
@@ -274,6 +279,7 @@ static const struct fs_parameter_spec ceph_parameters[] = {fsparam_flag_no ("cephx_require_signatures" , Opt_cephx_require_signatures),fsparam_flag_no ("cephx_sign_messages" , Opt_cephx_sign_messages),fsparam_flag_no ("crc" , Opt_crc), +fsparam_string ("crush_location" , Opt_crush_location),fsparam_string ("fsid" , Opt_fsid),fsparam_string ("ip" , Opt_ip), fsparam_string ("key", Opt_key),
@@ -298,6 +304,7 @@ struct ceph_options *ceph_alloc_options(void)if (!opt)return NULL ; + opt->crush_locs =RB_ROOT ; opt->mon_addr =kcalloc (CEPH_MAX_MON ,sizeof (*opt->mon_addr),GFP_KERNEL );if (!opt->mon_addr) {
@@ -320,6 +327,7 @@ void ceph_destroy_options(struct ceph_options *opt)if (!opt)return ; +ceph_clear_crush_locs (&opt->crush_locs);kfree (opt->name);if (opt->key) {ceph_crypto_key_destroy (opt->key);
@@ -454,6 +462,14 @@ int ceph_parse_param(struct fs_parameter *param, struct ceph_options *opt,if (!opt->key)return -ENOMEM ;return get_secret (opt->key, param->string, &log); +case Opt_crush_location: + err =ceph_parse_crush_loc (param->string, &opt->crush_locs); +if (err) { +error_plog (&log,"Failed to parse crush location: %d" , + err); +return err; + } +break ;case Opt_osdtimeout:warn_plog (&log,"Ignoring osdtimeout" );
@@ -536,6 +552,7 @@ int ceph_print_client_options(struct seq_file *m, struct ceph_client *client, {struct ceph_options *opt = client->options;size_t pos = m->count; +struct rb_node *n;if (opt->name) {seq_puts (m,"name=" );
@@ -545,6 +562,14 @@ int ceph_print_client_options(struct seq_file *m, struct ceph_client *client,if (opt->key)seq_puts (m,"secret=<hidden>," ); +for (n =rb_first (&opt->crush_locs); n; n =rb_next (n)) { +struct crush_loc_node *loc = + rb_entry(n, struct crush_loc_node, cl_node); + +seq_printf (m,"crush_location=%s:%s," , + loc->cl_loc.cl_type_name, loc->cl_loc.cl_name); + } +if (opt->flags &CEPH_OPT_FSID )seq_printf (m,"fsid=%pU," , &opt->fsid);if (opt->flags &CEPH_OPT_NOSHARE )
net/ceph/osdmap.c +114 -0
117 lines not on this page · raw message
Signature · 1 line
2.19.2
Re: [PATCH 3/5] libceph: crush_location infrastructure
Quoted 13 lines · Ilya Dryomov
Allow expressing client's location in terms of CRUSH hierarchy as a set of (bucket type name, bucket name) pairs. The userspace syntax "
crush_location= key1=value1 key2=value2" is incompatible with mount options and needed adaptation:- ':' separator
- one key:value pair percrush_locationoption
-crush_locationoptions are combined togetherSo for:
crush_location= host=foo rack=bar
Quoted 9 lines
one would write:
crush_location=host:foo,crush_location=rack:bar
As in userspace, "multipath" locations are supported, so indicating locality for parallel hierarchies is possible:
crush_location=rack:foo1,crush_location=rack:foo2,crush_location=datacenter:bar
Blech, that syntax is hideous. It's also problematic in that the options are additive -- you can't override an option that was given earlier (e.g. in fstab), or in a shell script.
Is it not possible to do something with a single crush_location= option? Maybe:
crush_location=rack:foo1/rack:foo2/datacenter:bar
It's still ugly with the embedded '=' signs, but it would at least make it so that the options aren't additive.
Quoted 256 lines
Signed-off-by Ilya Dryomov <[email protected]>--- include/linux/ceph/libceph.h | 1 + include/linux/ceph/osdmap.h | 16 ++++- net/ceph/ceph_common.c | 25 ++++++++ net/ceph/osdmap.c | 114 +++++++++++++++++++++++++++++++++++ 4 files changed, 155 insertions(+), 1 deletion(-)include/linux/ceph/libceph.h +1
-0@@ -64,6 +64,7 @@ struct ceph_options {int num_mon;char *name;struct ceph_crypto_key *key; +struct rb_root crush_locs; };/* include/linux/ceph/osdmap.h +15
-1@@ -302,9 +302,23 @@ bool ceph_pg_to_primary_shard(struct ceph_osdmap *osdmap,int ceph_pg_to_acting_primary (struct ceph_osdmap *osdmap,const struct ceph_pg *raw_pgid); +struct crush_loc { +char *cl_type_name; +char *cl_name; +}; + +struct crush_loc_node { +struct rb_node cl_node; +struct crush_loc cl_loc;/* pointers into cl_data */ +char cl_data[]; +}; + +int ceph_parse_crush_loc (const char *str,struct rb_root *locs); +int ceph_compare_crush_locs (struct rb_root *locs1,struct rb_root *locs2); +void ceph_clear_crush_locs (struct rb_root *locs); +extern struct ceph_pg_pool_info *ceph_pg_pool_by_id (struct ceph_osdmap *map,u64 id); -extern const char *ceph_pg_pool_name_by_id (struct ceph_osdmap *map,u64 id);extern int ceph_pg_poolid_by_name (struct ceph_osdmap *map,const char *name);u64 ceph_pg_pool_flags (struct ceph_osdmap *map,u64 id);net/ceph/ceph_common.c +25
-0@@ -176,6 +176,10 @@ int ceph_compare_options(struct ceph_options *new_opt, } } + ret =ceph_compare_crush_locs (&opt1->crush_locs, &opt2->crush_locs); +if (ret) +return ret; +/* any matching mon ip implies a match */ for (i =0 ; i < opt1->num_mon; i++) { if (ceph_monmap_contains(client->monc.monmap,@@ -260,6 +264,7 @@ enum { Opt_secret, Opt_key, Opt_ip, + Opt_crush_location,/* string args above */ Opt_share, Opt_crc,@@ -274,6 +279,7 @@ static const struct fs_parameter_spec ceph_parameters[] = {fsparam_flag_no ("cephx_require_signatures" , Opt_cephx_require_signatures),fsparam_flag_no ("cephx_sign_messages" , Opt_cephx_sign_messages),fsparam_flag_no ("crc" , Opt_crc), +fsparam_string ("crush_location" , Opt_crush_location),fsparam_string ("fsid" , Opt_fsid),fsparam_string ("ip" , Opt_ip), fsparam_string ("key", Opt_key),@@ -298,6 +304,7 @@ struct ceph_options *ceph_alloc_options(void)if (!opt)return NULL ; + opt->crush_locs =RB_ROOT ; opt->mon_addr =kcalloc (CEPH_MAX_MON ,sizeof (*opt->mon_addr),GFP_KERNEL );if (!opt->mon_addr) {@@ -320,6 +327,7 @@ void ceph_destroy_options(struct ceph_options *opt)if (!opt)return ; +ceph_clear_crush_locs (&opt->crush_locs);kfree (opt->name);if (opt->key) {ceph_crypto_key_destroy (opt->key);@@ -454,6 +462,14 @@ int ceph_parse_param(struct fs_parameter *param, struct ceph_options *opt,if (!opt->key)return -ENOMEM ;return get_secret (opt->key, param->string, &log); +case Opt_crush_location: + err =ceph_parse_crush_loc (param->string, &opt->crush_locs); +if (err) { +error_plog (&log,"Failed to parse crush location: %d" , + err); +return err; + } +break ;case Opt_osdtimeout:warn_plog (&log,"Ignoring osdtimeout" );@@ -536,6 +552,7 @@ int ceph_print_client_options(struct seq_file *m, struct ceph_client *client, {struct ceph_options *opt = client->options;size_t pos = m->count; +struct rb_node *n;if (opt->name) {seq_puts (m,"name=" );@@ -545,6 +562,14 @@ int ceph_print_client_options(struct seq_file *m, struct ceph_client *client,if (opt->key)seq_puts (m,"secret=<hidden>," ); +for (n =rb_first (&opt->crush_locs); n; n =rb_next (n)) { +struct crush_loc_node *loc = + rb_entry(n, struct crush_loc_node, cl_node); + +seq_printf (m,"crush_location=%s:%s," , + loc->cl_loc.cl_type_name, loc->cl_loc.cl_name); + } +if (opt->flags &CEPH_OPT_FSID )seq_printf (m,"fsid=%pU," , &opt->fsid);if (opt->flags &CEPH_OPT_NOSHARE )net/ceph/osdmap.c +114
-0@@ -2715,3 +2715,117 @@ int ceph_pg_to_acting_primary(struct ceph_osdmap *osdmap,return acting.primary; }EXPORT_SYMBOL (ceph_pg_to_acting_primary); + +static struct crush_loc_node *alloc_crush_loc (size_t type_name_len, +size_t name_len) +{ +struct crush_loc_node *loc; + + loc =kmalloc (sizeof (*loc) + type_name_len + name_len +2 ,GFP_NOIO ); +if (!loc) +return NULL ; + +RB_CLEAR_NODE (&loc->cl_node); +return loc; +} + +static void free_crush_loc (struct crush_loc_node *loc) +{ +WARN_ON (!RB_EMPTY_NODE (&loc->cl_node)); + +kfree (loc); +} + +static int crush_loc_compare (const struct crush_loc *loc1, +const struct crush_loc *loc2) +{ +return strcmp (loc1->cl_type_name, loc2->cl_type_name) ?: +strcmp (loc1->cl_name, loc2->cl_name); +} + +DEFINE_RB_FUNCS2(crush_loc, struct crush_loc_node, cl_loc, crush_loc_compare, + RB_BYPTR, const struct crush_loc *, cl_node) + +/* +* A <bucket type name>:<bucket name> pair, e.g. "zone:us-east". + */ +int ceph_parse_crush_loc (const char *str,struct rb_root *locs) +{ +struct crush_loc_node *loc; +const char *type_name, *name; +size_t type_name_len, name_len; + + type_name = str; + str =strchrnul (str,':' ); +if (*str =='\0' ) +return -EINVAL ;/* no ':' */ + + type_name_len = str - type_name; +if (type_name_len ==0 ) +return -EINVAL ; + + name = ++str; + str =strchrnul (str,':' ); +if (*str !='\0' ) +return -EINVAL ;/* another ':' */ + + name_len = str - name; +if (name_len ==0 ) +return -EINVAL ; + + loc =alloc_crush_loc (type_name_len, name_len); +if (!loc) +return -ENOMEM ; + + loc->cl_loc.cl_type_name = loc->cl_data; +memcpy (loc->cl_loc.cl_type_name, type_name, type_name_len); + loc->cl_loc.cl_type_name[type_name_len] ='\0' ; + + loc->cl_loc.cl_name = loc->cl_data + type_name_len +1 ; +memcpy (loc->cl_loc.cl_name, name, name_len); + loc->cl_loc.cl_name[name_len] ='\0' ; + +if (!__insert_crush_loc (locs, loc)) { +free_crush_loc (loc); +return -EEXIST ; + } + +dout ("%s type_name '%s' name '%s'\n" , __func__, + loc->cl_loc.cl_type_name, loc->cl_loc.cl_name); +return 0 ; +} + +int ceph_compare_crush_locs (struct rb_root *locs1,struct rb_root *locs2) +{ +struct rb_node *n1 =rb_first (locs1); +struct rb_node *n2 =rb_first (locs2); +int ret; + +for ( ; n1 && n2; n1 =rb_next (n1), n2 =rb_next (n2)) { +struct crush_loc_node *loc1 = + rb_entry(n1, struct crush_loc_node, cl_node); +struct crush_loc_node *loc2 = + rb_entry(n2, struct crush_loc_node, cl_node); + + ret =crush_loc_compare (&loc1->cl_loc, &loc2->cl_loc); +if (ret) +return ret; + } + +if (!n1 && n2) +return -1 ; +if (n1 && !n2) +return 1 ; +return 0 ; +} + +void ceph_clear_crush_locs (struct rb_root *locs) +{ +while (!RB_EMPTY_ROOT (locs)) { +struct crush_loc_node *loc = + rb_entry(rb_first(locs), struct crush_loc_node, cl_node); + +erase_crush_loc (locs, loc); +free_crush_loc (loc); + } +}
Signature · 1 line
Jeff Layton <[email protected]>
Ilya Dryomov
maintainerRe: [PATCH 3/5] libceph: crush_location infrastructure
Quoted 29 lines · Jeff Layton
Quoted 13 lines · Ilya Dryomov
Allow expressing client's location in terms of CRUSH hierarchy as a set of (bucket type name, bucket name) pairs. The userspace syntax "
crush_location= key1=value1 key2=value2" is incompatible with mount options and needed adaptation:- ':' separator
- one key:value pair percrush_locationoption
-crush_locationoptions are combined togetherSo for:
crush_location= host=foo rack=barQuoted 9 lines
one would write:
crush_location=host:foo,crush_location=rack:bar
As in userspace, "multipath" locations are supported, so indicating locality for parallel hierarchies is possible:
crush_location=rack:foo1,crush_location=rack:foo2,crush_location=datacenter:bar
Blech, that syntax is hideous. It's also problematic in that the options are additive -- you can't override an option that was given earlier (e.g. in fstab), or in a shell script.
Quoted 7 lines
Is it not possible to do something with a single crush_location= option? Maybe:
crush_location=rack:foo1/rack:foo2/datacenter:bar
It's still ugly with the embedded '=' signs, but it would at least make it so that the options aren't additive.
I suppose we could do something like that at the cost of more parsing boilerplate, but I'm not sure additive options are that hideous. I don't think additive options are unprecedented and more importantly I think many simple boolean and integer options are not properly overridable even in major filesystems.
What embedded '=' signs are you referring to? I see ':' and '/' in your suggested syntax.
Thanks,
Ilya
Re: [PATCH 3/5] libceph: crush_location infrastructure
Quoted 34 lines · Ilya Dryomov
Quoted 33 lines · Jeff Layton
Quoted 13 lines · Ilya Dryomov
Allow expressing client's location in terms of CRUSH hierarchy as a set of (bucket type name, bucket name) pairs. The userspace syntax "
crush_location= key1=value1 key2=value2" is incompatible with mount options and needed adaptation:- ':' separator
- one key:value pair percrush_locationoption
-crush_locationoptions are combined togetherSo for:
crush_location= host=foo rack=barQuoted 9 lines
one would write:
crush_location=host:foo,crush_location=rack:bar
As in userspace, "multipath" locations are supported, so indicating locality for parallel hierarchies is possible:
crush_location=rack:foo1,crush_location=rack:foo2,crush_location=datacenter:bar
Blech, that syntax is hideous. It's also problematic in that the options are additive -- you can't override an option that was given earlier (e.g. in fstab), or in a shell script.
Is it not possible to do something with a single crush_location= option? Maybe:
crush_location=rack:foo1/rack:foo2/datacenter:bar
Quoted 9 lines
Quoted 2 lines
It's still ugly with the embedded '=' signs, but it would at least make it so that the options aren't additive.
I suppose we could do something like that at the cost of more parsing boilerplate, but I'm not sure additive options are that hideous. I don't think additive options are unprecedented and more importantly I think many simple boolean and integer options are not properly overridable even in major filesystems.
That is the long-standing convention though. There are reasons to deviate from it, but I don't see it here. Plus, I think the syntax I proposed above is more readable (and compact) as well.
It would mean a bit more parsing code though, granted.
Quoted 3 lines
What embedded '=' signs are you referring to? I see ':' and '/' in your suggested syntax.
Sorry, yeah... I had originally done one that had '=' chars in it, but converted it to the above. Please disregard that paragraph.
Signature · 1 line
Jeff Layton <[email protected]>
Ilya Dryomov
maintainerRe: [PATCH 3/5] libceph: crush_location infrastructure
Quoted 50 lines · Jeff Layton
Quoted 34 lines · Ilya Dryomov
Quoted 33 lines · Jeff Layton
Quoted 13 lines · Ilya Dryomov
Allow expressing client's location in terms of CRUSH hierarchy as a set of (bucket type name, bucket name) pairs. The userspace syntax "
crush_location= key1=value1 key2=value2" is incompatible with mount options and needed adaptation:- ':' separator
- one key:value pair percrush_locationoption
-crush_locationoptions are combined togetherSo for:
crush_location= host=foo rack=barQuoted 9 lines
one would write:
crush_location=host:foo,crush_location=rack:bar
As in userspace, "multipath" locations are supported, so indicating locality for parallel hierarchies is possible:
crush_location=rack:foo1,crush_location=rack:foo2,crush_location=datacenter:bar
Blech, that syntax is hideous. It's also problematic in that the options are additive -- you can't override an option that was given earlier (e.g. in fstab), or in a shell script.
Is it not possible to do something with a single crush_location= option? Maybe:
crush_location=rack:foo1/rack:foo2/datacenter:bar
Quoted 9 lines
Quoted 2 lines
It's still ugly with the embedded '=' signs, but it would at least make it so that the options aren't additive.
I suppose we could do something like that at the cost of more parsing boilerplate, but I'm not sure additive options are that hideous. I don't think additive options are unprecedented and more importantly I think many simple boolean and integer options are not properly overridable even in major filesystems.
That is the long-standing convention though. There are reasons to deviate from it, but I don't see it here. Plus, I think the syntax I proposed above is more readable (and compact) as well.
Quoted 8 lines
It would mean a bit more parsing code though, granted.
Quoted 3 lines
What embedded '=' signs are you referring to? I see ':' and '/' in your suggested syntax.
Sorry, yeah... I had originally done one that had '=' chars in it, but converted it to the above. Please disregard that paragraph.
One of the reasons I did it this way is that crush_location is inherently additive. I don't have a strong opinion on this though so let's adhere to the convention.
I'll implement the suggested syntax and repost.
Thanks,
Ilya
Ilya Dryomov
maintainerRe: [PATCH 3/5] libceph: crush_location infrastructure
Quoted 58 lines · Ilya Dryomov
Quoted 56 lines · Jeff Layton
Quoted 34 lines · Ilya Dryomov
Quoted 33 lines · Jeff Layton
Quoted 13 lines · Ilya Dryomov
Allow expressing client's location in terms of CRUSH hierarchy as a set of (bucket type name, bucket name) pairs. The userspace syntax "
crush_location= key1=value1 key2=value2" is incompatible with mount options and needed adaptation:- ':' separator
- one key:value pair percrush_locationoption
-crush_locationoptions are combined togetherSo for:
crush_location= host=foo rack=barQuoted 9 lines
one would write:
crush_location=host:foo,crush_location=rack:bar
As in userspace, "multipath" locations are supported, so indicating locality for parallel hierarchies is possible:
crush_location=rack:foo1,crush_location=rack:foo2,crush_location=datacenter:bar
Blech, that syntax is hideous. It's also problematic in that the options are additive -- you can't override an option that was given earlier (e.g. in fstab), or in a shell script.
Is it not possible to do something with a single crush_location= option? Maybe:
crush_location=rack:foo1/rack:foo2/datacenter:bar
Quoted 9 lines
Quoted 2 lines
It's still ugly with the embedded '=' signs, but it would at least make it so that the options aren't additive.
I suppose we could do something like that at the cost of more parsing boilerplate, but I'm not sure additive options are that hideous. I don't think additive options are unprecedented and more importantly I think many simple boolean and integer options are not properly overridable even in major filesystems.
That is the long-standing convention though. There are reasons to deviate from it, but I don't see it here. Plus, I think the syntax I proposed above is more readable (and compact) as well.
It would mean a bit more parsing code though, granted.
Quoted 3 lines
What embedded '=' signs are you referring to? I see ':' and '/' in your suggested syntax.
Quoted 8 lines
Quoted 2 lines
Sorry, yeah... I had originally done one that had '=' chars in it, but converted it to the above. Please disregard that paragraph.
One of the reasons I did it this way is that
crush_locationis inherently additive. I don't have a strong opinion on this though so let's adhere to the convention.I'll implement the suggested syntax and repost.
I went with '|' instead of '/' for the separator to try to stress the additivity (in the OR sense). '/' makes it look like a path to the root of the tree which it really isn't.
Thanks,
Ilya
Ilya Dryomov
maintainerPATCH 4/5 libceph: support for balanced and localized reads
In reply to Ilya DryomovOSD-side issues with reads from replica have been resolved in Octopus. Reading from replica should be safe wrt. unstable or uncommitted state now, so add support for balanced and localized reads.
There are two cases when a read from replica can't be served:
- OSD may silently drop the request, expecting the client to notice that the acting set has changed and resend via the usual means (handled with t->used_replica)
- OSD may return EAGAIN, expecting the client to resend to the primary, ignoring replica read flags (see handle_reply())
--- include/linux/ceph/osd_client.h | 1 + include/linux/ceph/osdmap.h | 3 + net/ceph/debugfs.c | 6 +- net/ceph/osd_client.c | 87 +++++++++++++++++++++++++-- net/ceph/osdmap.c | 102 ++++++++++++++++++++++++++++++++ 5 files changed, 193 insertions(+), 6 deletions(-)
include/linux/ceph/osd_client.h +1 -0
@@ -165,6 +165,7 @@ struct ceph_osd_request_target {bool recovery_deletes;unsigned int flags;/* CEPH_OSD_FLAG_* */ +bool used_replica;bool paused;u32 epoch;
include/linux/ceph/osdmap.h +3 -0
@@ -317,6 +317,9 @@ int ceph_parse_crush_loc(const char *str, struct rb_root *locs);int ceph_compare_crush_locs (struct rb_root *locs1,struct rb_root *locs2);void ceph_clear_crush_locs (struct rb_root *locs); +int ceph_get_crush_locality (struct ceph_osdmap *osdmap,int id, +struct rb_root *locs); +extern struct ceph_pg_pool_info *ceph_pg_pool_by_id (struct ceph_osdmap *map,u64 id);extern const char *ceph_pg_pool_name_by_id (struct ceph_osdmap *map,u64 id);
net/ceph/debugfs.c +4 -2
@@ -81,11 +81,13 @@ static int osdmap_show(struct seq_file *s, void *p)u32 state = map->osd_state[i];char sb[64 ]; -seq_printf (s,"osd%d\t%s\t%3d%%\t(%s)\t%3d%%\n" , +seq_printf (s,"osd%d\t%s\t%3d%%\t(%s)\t%3d%%\t%2d\n" , i,ceph_pr_addr (addr), ((map->osd_weight[i]*100 ) >>16 ),ceph_osdmap_state_str (sb,sizeof (sb), state), - ((ceph_get_primary_affinity (map, i)*100 ) >>16 )); + ((ceph_get_primary_affinity (map, i)*100 ) >>16 ), +ceph_get_crush_locality (map, i, + &client->options->crush_locs)); }for (n =rb_first (&map->pg_temp); n; n =rb_next (n)) { struct ceph_pg_mapping *pg =
net/ceph/osd_client.c +83 -4
124 lines not on this page · raw message
net/ceph/osdmap.c +102 -0
105 lines not on this page · raw message
Signature · 1 line
2.19.2
Ilya Dryomov
maintainerPATCH 5/5 libceph: read_policy option
In reply to Ilya DryomovExpose balanced and localized reads through read_policy=balance and read_policy=localize. The default is to read from primary.
--- include/linux/ceph/libceph.h | 2 ++ net/ceph/ceph_common.c | 26 ++++++++++++++++++++++++++ net/ceph/osd_client.c | 5 ++++- 3 files changed, 32 insertions(+), 1 deletion(-)
include/linux/ceph/libceph.h +2 -0
@@ -52,6 +52,8 @@ struct ceph_options {unsigned long osd_idle_ttl;/* jiffies */ unsigned long osd_keepalive_timeout;/* jiffies */ unsigned long osd_request_timeout;/* jiffies */ +unsigned int osd_req_flags;/* CEPH_OSD_FLAG_*, applied to +each OSD request */ /* * any type that can't be simply compared or doesn't need
net/ceph/ceph_common.c +26 -0
@@ -265,6 +265,7 @@ enum { Opt_key, Opt_ip, Opt_crush_location, + Opt_read_policy,/* string args above */ Opt_share, Opt_crc,
@@ -274,6 +275,17 @@ enum { Opt_abort_on_full, }; +enum { + Opt_read_policy_balance, + Opt_read_policy_localize, +}; + +static const struct constant_table ceph_param_read_policy[] = { + {"balance" , Opt_read_policy_balance}, + {"localize" , Opt_read_policy_localize}, + {} +}; +static const struct fs_parameter_spec ceph_parameters[] = {fsparam_flag ("abort_on_full" , Opt_abort_on_full),fsparam_flag_no ("cephx_require_signatures" , Opt_cephx_require_signatures),
@@ -290,6 +302,8 @@ static const struct fs_parameter_spec ceph_parameters[] = {fsparam_u32 ("osdkeepalive" , Opt_osdkeepalivetimeout),__fsparam (fs_param_is_s32,"osdtimeout" , Opt_osdtimeout, fs_param_deprecated,NULL ), +fsparam_enum ("read_policy" , Opt_read_policy, + ceph_param_read_policy),fsparam_string ("secret" , Opt_secret),fsparam_flag_no ("share" , Opt_share), fsparam_flag_no ("tcp_nodelay", Opt_tcp_nodelay),
@@ -470,6 +484,18 @@ int ceph_parse_param(struct fs_parameter *param, struct ceph_options *opt,return err; }break ; +case Opt_read_policy: +switch (result.uint_32) { +case Opt_read_policy_balance: + opt->osd_req_flags |=CEPH_OSD_FLAG_BALANCE_READS ; +break ; +case Opt_read_policy_localize: + opt->osd_req_flags |=CEPH_OSD_FLAG_LOCALIZE_READS ; +break ; +default : +BUG (); + } +break ;case Opt_osdtimeout:warn_plog (&log,"Ignoring osdtimeout" );
net/ceph/osd_client.c +4 -1
@@ -2425,11 +2425,14 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)static void account_request (struct ceph_osd_request *req) { +struct ceph_osd_client *osdc = req->r_osdc; +WARN_ON (req->r_flags & (CEPH_OSD_FLAG_ACK |CEPH_OSD_FLAG_ONDISK ));WARN_ON (!(req->r_flags & (CEPH_OSD_FLAG_READ |CEPH_OSD_FLAG_WRITE ))); req->r_flags |=CEPH_OSD_FLAG_ONDISK ; -atomic_inc (&req->r_osdc->num_requests); + req->r_flags |= osdc->client->options->osd_req_flags; +atomic_inc (&osdc->num_requests); req->r_start_stamp = jiffies; req->r_start_latency =ktime_get ();
Signature · 1 line
2.19.2
Re: [PATCH 5/5] libceph: read_policy option
Quoted 68 lines · Ilya Dryomov
Expose balanced and localized reads through read_policy=balance and read_policy=localize. The default is to read from primary.
Signed-off-by Ilya Dryomov <[email protected]>--- include/linux/ceph/libceph.h | 2 ++ net/ceph/ceph_common.c | 26 ++++++++++++++++++++++++++ net/ceph/osd_client.c | 5 ++++- 3 files changed, 32 insertions(+), 1 deletion(-)include/linux/ceph/libceph.h +2
-0@@ -52,6 +52,8 @@ struct ceph_options {unsigned long osd_idle_ttl;/* jiffies */ unsigned long osd_keepalive_timeout;/* jiffies */ unsigned long osd_request_timeout;/* jiffies */ +unsigned int osd_req_flags;/* CEPH_OSD_FLAG_*, applied to +each OSD request */ /* * any type that can't be simply compared or doesn't need net/ceph/ceph_common.c +16
-0@@ -265,6 +265,7 @@ enum { Opt_key, Opt_ip, Opt_crush_location, + Opt_read_policy,/* string args above */ Opt_share, Opt_crc,@@ -274,6 +275,17 @@ enum { Opt_abort_on_full, }; +enum { + Opt_read_policy_balance, + Opt_read_policy_localize, +}; + +static const struct constant_table ceph_param_read_policy[] = { + {"balance" , Opt_read_policy_balance}, + {"localize" , Opt_read_policy_localize}, + {} +}; +static const struct fs_parameter_spec ceph_parameters[] = {fsparam_flag ("abort_on_full" , Opt_abort_on_full),fsparam_flag_no ("cephx_require_signatures" , Opt_cephx_require_signatures),@@ -290,6 +302,8 @@ static const struct fs_parameter_spec ceph_parameters[] = {fsparam_u32 ("osdkeepalive" , Opt_osdkeepalivetimeout),__fsparam (fs_param_is_s32,"osdtimeout" , Opt_osdtimeout, fs_param_deprecated,NULL ), +fsparam_enum ("read_policy" , Opt_read_policy, + ceph_param_read_policy),fsparam_string ("secret" , Opt_secret),fsparam_flag_no ("share" , Opt_share), fsparam_flag_no ("tcp_nodelay", Opt_tcp_nodelay),@@ -470,6 +484,18 @@ int ceph_parse_param(struct fs_parameter *param, struct ceph_options *opt,return err; }break ; +case Opt_read_policy: +switch (result.uint_32) {
net/ceph/ceph_common.c int ceph_parse_param(struct fs_parameter *param, struct ceph_options *opt, · line 498
Quoted 10 lines
net/ceph/ceph_common.c +10
-0+case Opt_read_policy_balance: + opt->osd_req_flags |=CEPH_OSD_FLAG_BALANCE_READS ; +break ; +case Opt_read_policy_localize: + opt->osd_req_flags |=CEPH_OSD_FLAG_LOCALIZE_READS ; +break ; +default : +BUG (); + } +break ;
Suppose I specify "-o read_policy=balance,read_policy=localize".
Principle of least surprise says "last one wins", but you'll end up with both flags set here, and I think the final result would still be "balance". I think it'd probably be best to rework this so that the last option specified is what you get.
I also think you want a way to explicitly set it back to default behavior (read_policy=primary ?), as it's not uncommon for people to specify mount options in fstab but then append to them on the command line. e.g.:
# mount /mnt/cephfs -o read_policy=primary
...when fstab already has read_policy=balance.
Quoted 22 lines
net/ceph/ceph_common.c +0
-0case Opt_osdtimeout:warn_plog (&log,"Ignoring osdtimeout" );net/ceph/osd_client.c +4
-1@@ -2425,11 +2425,14 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked)static void account_request (struct ceph_osd_request *req) { +struct ceph_osd_client *osdc = req->r_osdc; +WARN_ON (req->r_flags & (CEPH_OSD_FLAG_ACK |CEPH_OSD_FLAG_ONDISK ));WARN_ON (!(req->r_flags & (CEPH_OSD_FLAG_READ |CEPH_OSD_FLAG_WRITE ))); req->r_flags |=CEPH_OSD_FLAG_ONDISK ; -atomic_inc (&req->r_osdc->num_requests); + req->r_flags |= osdc->client->options->osd_req_flags; +atomic_inc (&osdc->num_requests); req->r_start_stamp = jiffies; req->r_start_latency =ktime_get ();
Signature · 1 line
Jeff Layton <[email protected]>
Ilya Dryomov
maintainerRe: [PATCH 5/5] libceph: read_policy option
Quoted 88 lines · Jeff Layton
Quoted 68 lines · Ilya Dryomov
Expose balanced and localized reads through read_policy=balance and read_policy=localize. The default is to read from primary.
Signed-off-by Ilya Dryomov <[email protected]>--- include/linux/ceph/libceph.h | 2 ++ net/ceph/ceph_common.c | 26 ++++++++++++++++++++++++++ net/ceph/osd_client.c | 5 ++++- 3 files changed, 32 insertions(+), 1 deletion(-)include/linux/ceph/libceph.h +2
-0@@ -52,6 +52,8 @@ struct ceph_options {unsigned long osd_idle_ttl;/* jiffies */ unsigned long osd_keepalive_timeout;/* jiffies */ unsigned long osd_request_timeout;/* jiffies */ +unsigned int osd_req_flags;/* CEPH_OSD_FLAG_*, applied to +each OSD request */ /* * any type that can't be simply compared or doesn't need net/ceph/ceph_common.c +16
-0@@ -265,6 +265,7 @@ enum { Opt_key, Opt_ip, Opt_crush_location, + Opt_read_policy,/* string args above */ Opt_share, Opt_crc,@@ -274,6 +275,17 @@ enum { Opt_abort_on_full, }; +enum { + Opt_read_policy_balance, + Opt_read_policy_localize, +}; + +static const struct constant_table ceph_param_read_policy[] = { + {"balance" , Opt_read_policy_balance}, + {"localize" , Opt_read_policy_localize}, + {} +}; +static const struct fs_parameter_spec ceph_parameters[] = {fsparam_flag ("abort_on_full" , Opt_abort_on_full),fsparam_flag_no ("cephx_require_signatures" , Opt_cephx_require_signatures),@@ -290,6 +302,8 @@ static const struct fs_parameter_spec ceph_parameters[] = {fsparam_u32 ("osdkeepalive" , Opt_osdkeepalivetimeout),__fsparam (fs_param_is_s32,"osdtimeout" , Opt_osdtimeout, fs_param_deprecated,NULL ), +fsparam_enum ("read_policy" , Opt_read_policy, + ceph_param_read_policy),fsparam_string ("secret" , Opt_secret),fsparam_flag_no ("share" , Opt_share), fsparam_flag_no ("tcp_nodelay", Opt_tcp_nodelay),@@ -470,6 +484,18 @@ int ceph_parse_param(struct fs_parameter *param, struct ceph_options *opt,return err; }break ; +case Opt_read_policy: +switch (result.uint_32) {net/ceph/ceph_common.c int ceph_parse_param(struct fs_parameter *param, struct ceph_options *opt, · line 498
Quoted 10 lines
net/ceph/ceph_common.c +10
-0+case Opt_read_policy_balance: + opt->osd_req_flags |=CEPH_OSD_FLAG_BALANCE_READS ; +break ; +case Opt_read_policy_localize: + opt->osd_req_flags |=CEPH_OSD_FLAG_LOCALIZE_READS ; +break ; +default : +BUG (); + } +break ;Suppose I specify "-o read_policy=balance,read_policy=localize".
Principle of least surprise says "last one wins", but you'll end up with both flags set here, and I think the final result would still be "balance". I think it'd probably be best to rework this so that the last option specified is what you get.
Quoted 8 lines
I also think you want a way to explicitly set it back to default behavior (read_policy=primary ?), as it's not uncommon for people to specify mount options in fstab but then append to them on the command line. e.g.:
# mount /mnt/cephfs -o read_policy=primary...when fstab already has read_policy=balance.
Yes, currently balance wins.
Adding read_policy=primary and implementing the override behaviour for read_policy sounds reasonable.
Thanks,
Ilya
A message that is not in the archive (sent off-list or not yet received): CA+aFP1D_povJnBfH5pT8E7OVK_WHWa0TtBwRCxB=OmscTjGf8Q@mail.gmail.com
Ilya Dryomov
maintainerQuoted 4 lines · Jason Dillaman
lgtm -- couple questions:
1) the client adding the options will be responsible for determining if it's safe to enable read-from-replica/balanced reads (i.e. OSDs >= octopus)?
Yes, we can't easily check require_osd_release or similar in the kernel. This is opt-in, and I'll add a warning together with the description of the new options to the man page.
Quoted 1 line
2) is there a way to determine if the kernel supports the new options (or will older kernels just ignore the options w/o complaining)? i.e. can ceph-csi safely add the argument regardless?
No, older kernels will error out. I think ceph-csi would handle this the same way ceph quotas are handled (i.e. with the list of known "good" kernel versions) or alternatively just attempt to map with and then without crush_location and read_balance=localize (probably less appealing given that the kernel version list infrastructure is already in place).
Thanks,
Ilya
Ilya Dryomov
maintainerQuoted 11 lines · Ilya Dryomov
Quoted 4 lines · Jason Dillaman
lgtm -- couple questions:
1) the client adding the options will be responsible for determining if it's safe to enable read-from-replica/balanced reads (i.e. OSDs >= octopus)?
Yes, we can't easily check
require_osd_releaseor similar in the kernel. This is opt-in, and I'll add a warning together with the description of the new options to the man page.
Quoted 6 lines
Quoted 1 line
2) is there a way to determine if the kernel supports the new options (or will older kernels just ignore the options w/o complaining)? i.e. can ceph-csi safely add the argument regardless?
No, older kernels will error out. I think ceph-csi would handle this the same way ceph quotas are handled (i.e. with the list of known "good" kernel versions) or alternatively just attempt to map with and then without
crush_locationand read_balance=localize
^^^
Typo, read_policy=localize.
Thanks,
Ilya