Samba RADOS service registration
RFC PATCH 1/1 Samba RADOS service registration
Hi,
I've been working on having Samba's smbd daemon register itself with the Ceph Cluster via rados_service_register(), and would appreciate some input on what to include in the instance and metadata descriptors.
The attached patch implements basic Samba service registration functionality, but only uses a $hostname:smbd string for the instance and doesn't add any extra metadata (see [1] as a sample service dump).
I'd like to (at the very least) pack the CephFS backed share names and paths into the service registration. For this, I'm leaning towards adding the share name as an instance suffix (i.e. $hostname:smbd:$share), with a separate registration for each share. Each share could then include the CephFS path as instance specific metadata. That still leaves me with a few questions:
- does it make sense to only register shares which are backed by CephFS?
+ consider a server with local and CephFS shares, or worse still, shared printers
+ what about a Samba service without any shares?
- I'd be using a single RADOS cluster connection to register all shares. Can these services can be deregistered individually in case of share removal, or are separate cluster connections required?
+ a rados_service_deregister() hook would be useful
Cheers, David
1. Sample Samba smbd service dump with shares ignored
Quoted 1 line
ceph service dump
{
"epoch": 2,
"modified": "2018-02-13 01:53:09.263133",
"services": {
"samba": {
"daemons": {
"summary": "",
"rapido1:smbd": {
"start_epoch": 2,
"start_stamp": "2018-02-13 01:53:08.112102",
"gid": 4131,
"addr": "192.168.124.104:0/1984347853",
"metadata": {
"arch": "x86_64",
"ceph_version": "ceph version 12.2.1-387-g313479ab90 (313479ab90915289fa4905822d1f4825d1bf1e7c) luminous (stable)",
"cpu": "QEMU Virtual CPU version 2.5+",
"distro": "dracut",
"distro_description": "openSUSE Leap 42.3 dracut-044-29.1 (Initramfs)",
"distro_version": "044-29.1",
"hostname": "rapido1",
"kernel_description": "#21 SMP Tue Feb 13 01:05:27 CET 2018",
"kernel_version": "4.4.115+",
"mem_swap_kb": "0",
"mem_total_kb": "1022828",
"os": "Linux"
}
}
}
}
}
}From fcbcb4443ed70a84ad04f60d52dc2926479b2ca6 Mon Sep 17 00:00:00 2001
From: David Disseldorp <[email protected]>
Date: Wed, 7 Feb 2018 14:58:48 +0100
Subject: [PATCH] ceph: add rados service integration
On startup, Samba advertises its presence to the Ceph cluster via rados_service_register().
--- source3/smbd/rados_service.c | 147 +++++++++++++++++++++++++++++++++++++++++++ source3/smbd/rados_service.h | 30 +++++++++ source3/smbd/server.c | 7 +++ source3/wscript | 10 ++- source3/wscript_build | 12 +++- 5 files changed, 204 insertions(+), 2 deletions(-) create mode 100644 source3/smbd/rados_service.c create mode 100644 source3/smbd/rados_service.h
source3/smbd/rados_service.c new file +147 -0
@@ -0,0 +1,147 @@ +/* +Copyright (C) David Disseldorp 2018 + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include "source3/include/includes.h" +#include <rados/librados.h> +#include "rados_service.h" + +struct ceph_service_state { +rados_t cluster; +}; + +static int ceph_service_destroy (struct ceph_service_state *state) +{ +DBG_DEBUG ("ending Ceph cluster service registration\n" ); +rados_shutdown (state->cluster); +return 0 ; +} + +static int ceph_service_instance_gen (char *buf,size_t buflen, +const char *daemon) +{ +int ret; +size_t hlen; +size_t remain; + + ret =gethostname (buf, buflen); +if (ret ==-1 ) { +return -errno; + } + buf[buflen -1 ] ='\0' ; + hlen =strlen (buf); + remain = buflen - hlen; + + ret =snprintf (buf + hlen, remain,":%s" , daemon); +if (ret >= remain) { +DBG_ERR ("Ceph instance name too long, skipping service " +"registration\n" ); +return -ENAMETOOLONG ; + } + +return 0 ; +} + +int ceph_service_register (TALLOC_CTX *parent_ctx,const char *daemon) +{ +int ret; +const char *user_id =NULL ; +const char *conf_file =NULL ; +const char *cluster_name =NULL ; +char instance_buf[HOST_NAME_MAX +256 ]; +const char *md ="" ; +struct ceph_service_state *state =talloc (parent_ctx, + struct ceph_service_state); + +if (state ==NULL ) { +return -ENOMEM ; + } + +/* +* XXX This does not reuse the existing (share based) "ceph:user_id" +* parameter, to allow for: +* - running Samba with Ceph support, but without service registration. +* - future mapping between Samba and Ceph users. +*/ + user_id =lp_parm_const_string (GLOBAL_SECTION_SNUM ,"ceph" , +"service_user_id" ,NULL ); +if (user_id ==NULL ) { +DBG_DEBUG ("built with Ceph support, but missing ceph:" +"service_user_id configuration - skipping service " +"registration\n" ); + ret =0 ; + goto out_mem_free; + } + + ret =ceph_service_instance_gen (instance_buf,sizeof (instance_buf), + daemon); +if (ret <0 ) { + goto out_mem_free; + } + + cluster_name =lp_parm_const_string (GLOBAL_SECTION_SNUM ,"ceph" , +"cluster_name" ,NULL ); +DBG_DEBUG ("registering as %s with %s Ceph cluster\n" , instance_buf, + (cluster_name !=NULL ? cluster_name :"default" )); + + ret =rados_create2 (&state->cluster, cluster_name, user_id,0 ); +if (ret <0 ) { +DBG_ERR ("failed to create ceph cluster context\n" ); + goto out_mem_free; + } + + conf_file =lp_parm_const_string (GLOBAL_SECTION_SNUM ,"ceph" , +"config_file" ,NULL ); +/* a NULL conf_file sees libceph use the default path */ + ret =rados_conf_read_file (state->cluster, conf_file); +if (ret <0 ) { +DBG_ERR ("failed to parse Ceph cluster config: %s\n" , +strerror (-ret)); + goto err_cluster_free; + } + + ret =rados_connect (state->cluster); +if (ret <0 ) { +DBG_ERR ("failed to connect to ceph cluster\n" ); + goto err_cluster_free; + } + +if (strcmp (daemon,"smbd" ) ==0 ) { +/* +* TODO walk list of shares and add as metadata +* and register for share inventory updates +*/ + } + +/* walk ceph specific shares, and add corresponding metadata */ + ret =rados_service_register (state->cluster,"samba" , instance_buf, + md); +if (ret <0 ) { +DBG_ERR ("failed to register service with ceph cluster\n" ); + goto err_cluster_free; + } + +/* close cluster conn and drop service listing on server exit */ +talloc_set_destructor (state, ceph_service_destroy); + +return 0 ; + +err_cluster_free: +rados_shutdown (state->cluster); +out_mem_free: +talloc_free (state); +return ret; +}
source3/smbd/rados_service.h new file +30 -0
@@ -0,0 +1,30 @@ +/* +Copyright (C) David Disseldorp 2018 + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#ifndef _RADOS_SERVICE_H +#define _RADOS_SERVICE_H + +int ceph_service_register (TALLOC_CTX *mem_ctx,const char *daemon); +#if !defined(HAVE_LIBRADOS_SERVICES ) +/* built without librados service support, do nothing */ +int ceph_service_register (TALLOC_CTX *mem_ctx,const char *daemon) +{ +return 0 ; +} +#endif + +#endif /* _RADOS_SERVICE_H */
source3/smbd/server.c +7 -0
@@ -54,6 +54,7 @@#include "lib/util/sys_rw.h" #include "cleanupdb.h" #include "g_lock.h" +#include "rados_service.h" #ifdef CLUSTER_SUPPORT #include "ctdb_protocol.h"
@@ -1593,6 +1594,7 @@ extern void build_options(bool screen);struct smbd_parent_context *parent =NULL ;TALLOC_CTX *frame;NTSTATUS status; +int ret;struct tevent_context *ev_ctx;struct messaging_context *msg_ctx;struct server_id server_id;
@@ -2053,6 +2055,11 @@ extern void build_options(bool screen);* -- bad things will happen if smbd is launched via inetd * and we fork a copy of ourselves here */ if (is_daemon && !interactive) { + ret =ceph_service_register (parent,"smbd" ); +if (ret <0 ) { +exit_daemon ("Samba failed to register with Ceph " +"cluster" , -ret); + }if (rpc_lsasd_daemon () ==RPC_DAEMON_FORK ) {start_lsasd (ev_ctx, msg_ctx);
source3/wscript +9 -1
@@ -1544,14 +1544,18 @@ main() { conf.env['CCFLAGS_CEPHFS'] = "-D_FILE_OFFSET_BITS=64" if Options.options.libcephfs_dir: + conf.env['CPPPATH_RADOS'] = Options.options.libcephfs_dir + '/include' conf.env['CPPPATH_CEPHFS'] = Options.options.libcephfs_dir + '/include' conf.env['LIBPATH_CEPHFS'] = Options.options.libcephfs_dir + '/lib' conf.env['LIBPATH_CEPH-COMMON'] = Options.options.libcephfs_dir + '/lib/ceph' + conf.env['LIBPATH_RADOS'] = Options.options.libcephfs_dir + '/lib' if (Options.options.with_cephfs and conf.CHECK_HEADERS('cephfs/libcephfs.h', False, False, 'cephfs') and + conf.CHECK_HEADERS('rados/librados.h', False, False, 'rados') and conf.CHECK_LIB('cephfs', shlib=True) and - conf.CHECK_LIB('ceph-common', shlib=True)): + conf.CHECK_LIB('ceph-common', shlib=True) and + conf.CHECK_LIB('rados', shlib=True)): if Options.options.with_acl_support: conf.DEFINE('HAVE_CEPH', '1') if conf.CHECK_FUNCS_IN('ceph_statx', 'cephfs ceph-common',
@@ -1561,6 +1565,10 @@ main() { Logs.warn("ceph support disabled due to --without-acl-support") conf.undefine('HAVE_CEPH') + if conf.CHECK_FUNCS_IN('rados_service_register', 'rados ceph-common', + headers='rados/librados.h'): + conf.DEFINE('HAVE_LIBRADOS_SERVICES', '1') + if Options.options.with_glusterfs: conf.CHECK_CFG(package='glusterfs-api', args='"glusterfs-api >= 4" --cflags --libs', msg='Checking for glusterfs-api >= 4', uselib_store="GFAPI")
source3/wscript_build +11 -1
@@ -1052,6 +1052,16 @@ bld.SAMBA3_SUBSYSTEM('SPOOLSSD', RPC_SOCK_HELPER ''') +smbd_rados_deps = '' +if bld.CONFIG_SET('HAVE_LIBRADOS_SERVICES'): + bld.SAMBA_LIBRARY('rados_service', + source='smbd/rados_service.c', + deps='SMBCONF_PARAM samba-debug', + local_include=False, + public_deps='rados ceph-common', + private_library=True) + smbd_rados_deps += ' rados_service' + ########################## BINARIES ################################# bld.SAMBA3_BINARY('smbd/smbd',
@@ -1064,7 +1074,7 @@ bld.SAMBA3_BINARY('smbd/smbd', FSSD MDSSD SPOOLSSD - ''', + ''' + smbd_rados_deps, install_path='${SBINDIR}')
Signature · 1 line
2.13.6
Quoted 9 lines · David Disseldorp
Hi,
I've been working on having Samba's smbd daemon register itself with the Ceph Cluster via
rados_service_register(), and would appreciate some input on what to include in the instance and metadata descriptors.The attached patch implements basic Samba service registration functionality, but only uses a $hostname:smbd string for the instance and doesn't add any extra metadata (see [1] as a sample service dump).
Just as a heads up, after making tcmu-runner and rbd-mirror daemon both register as service daemons, I found that using a hostname wasn't the best technique since you really need to ensure a unique name or else the last registered service daemon will clobber the previously registered daemon. Since the initial implementation, I now include the global instance id from RADOS in the service name to guarantee uniqueness (especially since the hostname is already automatically included in the metadata).
Quoted 5 lines
I'd like to (at the very least) pack the CephFS backed share names and paths into the service registration. For this, I'm leaning towards adding the share name as an instance suffix (i.e. $hostname:smbd:$share), with a separate registration for each share. Each share could then include the CephFS path as instance specific
Quoted 8 lines
metadata. That still leaves me with a few questions:
- does it make sense to only register shares which are backed by CephFS?
+ consider a server with local and CephFS shares, or worse still, shared printers
+ what about a Samba service without any shares?
- I'd be using a single RADOS cluster connection to register all shares. Can these services can be deregistered individually in case of share removal, or are separate cluster connections required?
Currently you can only register a single service per (mgr) connection. When I added the hooks to tcmu-runner and rbd-mirror daemons, I just used the service status [1] to populate any data that would be dynamically changing.
Quoted 30 lines
+ a
rados_service_deregister()hook would be usefulCheers, David
1. Sample Samba smbd service dump with shares ignored
Quoted 1 line
ceph service dump
{ "epoch": 2, "modified": "2018-02-13 01:53:09.263133", "services": { "samba": { "daemons": { "summary": "", "rapido1:smbd": { "start_epoch": 2, "start_stamp": "2018-02-13 01:53:08.112102", "gid": 4131, "addr": "192.168.124.104:0/1984347853", "metadata": { "arch": "x86_64", "ceph_version": "ceph version 12.2.1-387-g313479ab90 (313479ab90915289fa4905822d1f4825d1bf1e7c) luminous (stable)", "cpu": "QEMU Virtual CPU version 2.5+", "distro": "dracut", "distro_description": "openSUSE Leap 42.3 dracut-044-29.1 (Initramfs)", "distro_version": "044-29.1", "hostname": "rapido1", "kernel_description": "#21 SMP Tue Feb 13 01:05:27 CET 2018", "kernel_version": "4.4.115+", "mem_swap_kb": "0",
Quoted 9 lines
"mem_total_kb": "1022828",
"os": "Linux"
}
}
}
}
}
}[1] https://github.com/ceph/ceph/blob/master/src/tools/rbd_mirror/ServiceDaemon.cc#L241
Signature · 1 line
Jason
Quoted 6 lines · Jason Dillaman
Quoted 5 lines · David Disseldorp
Hi,
I've been working on having Samba's smbd daemon register itself with the Ceph Cluster via
rados_service_register(), and would appreciate some input on what to include in the instance and metadata descriptors.
Nice!
Quoted 4 lines
Quoted 3 lines
The attached patch implements basic Samba service registration functionality, but only uses a $hostname:smbd string for the instance and doesn't add any extra metadata (see [1] as a sample service dump).
Quoted 8 lines
Just as a heads up, after making tcmu-runner and rbd-mirror daemon both register as service daemons, I found that using a hostname wasn't the best technique since you really need to ensure a unique name or else the last registered service daemon will clobber the previously registered daemon. Since the initial implementation, I now include the global instance id from RADOS in the service name to guarantee uniqueness (especially since the hostname is already automatically included in the metadata).
FWIW in our looming dystopian container future the pod name is (I think?) globally unique, and also conveniently what a user/admin will need to know in order to find/kill/whatever the service.
Quoted 14 lines
Quoted 5 lines
I'd like to (at the very least) pack the CephFS backed share names and paths into the service registration. For this, I'm leaning towards adding the share name as an instance suffix (i.e. $hostname:smbd:$share), with a separate registration for each share. Each share could then include the CephFS path as instance specific
Quoted 8 lines
metadata. That still leaves me with a few questions:
- does it make sense to only register shares which are backed by CephFS?
+ consider a server with local and CephFS shares, or worse still, shared printers
+ what about a Samba service without any shares?
- I'd be using a single RADOS cluster connection to register all shares. Can these services can be deregistered individually in case of share removal, or are separate cluster connections required?
Quoted 4 lines
Currently you can only register a single service per (mgr) connection. When I added the hooks to tcmu-runner and rbd-mirror daemons, I just used the service status [1] to populate any data that would be dynamically changing.
I would also worry that the shares count might be very large. In all other cases the service maps to a daemon instance so I'd prefer to stick to that.
sage
Quoted 51 lines
Quoted 31 lines
+ a
rados_service_deregister()hook would be usefulCheers, David
1. Sample Samba smbd service dump with shares ignored
Quoted 1 line
ceph service dump
{ "epoch": 2, "modified": "2018-02-13 01:53:09.263133", "services": { "samba": { "daemons": { "summary": "", "rapido1:smbd": { "start_epoch": 2, "start_stamp": "2018-02-13 01:53:08.112102", "gid": 4131, "addr": "192.168.124.104:0/1984347853", "metadata": { "arch": "x86_64", "ceph_version": "ceph version 12.2.1-387-g313479ab90 (313479ab90915289fa4905822d1f4825d1bf1e7c) luminous (stable)", "cpu": "QEMU Virtual CPU version 2.5+", "distro": "dracut", "distro_description": "openSUSE Leap 42.3 dracut-044-29.1 (Initramfs)", "distro_version": "044-29.1", "hostname": "rapido1", "kernel_description": "#21 SMP Tue Feb 13 01:05:27 CET 2018", "kernel_version": "4.4.115+", "mem_swap_kb": "0",Quoted 9 lines
"mem_total_kb": "1022828", "os": "Linux" } } } } } }[1] https://github.com/ceph/ceph/blob/master/src/tools/rbd_mirror/ServiceDaemon.cc#L241
Signature · 5 lines
Jason -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · David Disseldorp
Hi,
I've been working on having Samba's smbd daemon register itself with the Ceph Cluster via
rados_service_register(), and would appreciate some input on what to include in the instance and metadata descriptors.The attached patch implements basic Samba service registration functionality, but only uses a $hostname:smbd string for the instance and doesn't add any extra metadata (see [1] as a sample service dump).
Very cool!
Quoted 2 lines
I'd like to (at the very least) pack the CephFS backed share names and paths into the service registration.
Echoing back the share configuration probably isn't a good use of this interface -- the idea is for service daemons to register just enough information about themselves to report on their existence, location and health.
In the case of nfs-ganesha, we're moving toward having the option of consuming its share configuration from a central place like a RADOS object, rather than from a local text config file -- in that model, a UI can go look at the configuration directly rather than having the daemon report it back. This is kind of a Ceph-ized version of the way a lot of new projects are storing their configuration in etcd.
John
For this, I'm leaning towards
Quoted 50 lines
adding the share name as an instance suffix (i.e. $hostname:smbd:$share), with a separate registration for each share. Each share could then include the CephFS path as instance specific metadata. That still leaves me with a few questions:
- does it make sense to only register shares which are backed by CephFS?
+ consider a server with local and CephFS shares, or worse still, shared printers
+ what about a Samba service without any shares?
- I'd be using a single RADOS cluster connection to register all shares. Can these services can be deregistered individually in case of share removal, or are separate cluster connections required?
+ arados_service_deregister()hook would be usefulCheers, David
1. Sample Samba smbd service dump with shares ignored
Quoted 1 line
ceph service dump
{ "epoch": 2, "modified": "2018-02-13 01:53:09.263133", "services": { "samba": { "daemons": { "summary": "", "rapido1:smbd": { "start_epoch": 2, "start_stamp": "2018-02-13 01:53:08.112102", "gid": 4131, "addr": "192.168.124.104:0/1984347853", "metadata": { "arch": "x86_64", "ceph_version": "ceph version 12.2.1-387-g313479ab90 (313479ab90915289fa4905822d1f4825d1bf1e7c) luminous (stable)", "cpu": "QEMU Virtual CPU version 2.5+", "distro": "dracut", "distro_description": "openSUSE Leap 42.3 dracut-044-29.1 (Initramfs)", "distro_version": "044-29.1", "hostname": "rapido1", "kernel_description": "#21 SMP Tue Feb 13 01:05:27 CET 2018", "kernel_version": "4.4.115+", "mem_swap_kb": "0", "mem_total_kb": "1022828", "os": "Linux" } } } } } }
Thanks for the feedback everyone...
Quoted 1 line · John Spray
On Tue, Feb 13, 2018 at 1:25 AM, David Disseldorp <[email protected]> wrote:
...
Quoted 7 lines
Quoted 2 lines
I'd like to (at the very least) pack the CephFS backed share names and paths into the service registration.
Echoing back the share configuration probably isn't a good use of this interface -- the idea is for service daemons to register just enough information about themselves to report on their existence, location and health.
Okay, I'll keep it simple then, and just use the Ceph global instance id and daemon name (smbd) for now.
Quoted 6 lines
In the case of nfs-ganesha, we're moving toward having the option of consuming its share configuration from a central place like a RADOS object, rather than from a local text config file -- in that model, a UI can go look at the configuration directly rather than having the daemon report it back. This is kind of a Ceph-ized version of the way a lot of new projects are storing their configuration in etcd.
Interesting. The same should be possible for Samba when we eventually have a Ceph key-value store backend for Samba's dbwrap, and then use Registry configuration.
Cheers, David
Quoted 10 lines · David Disseldorp via samba-technical
Quoted 6 lines
In the case of nfs-ganesha, we're moving toward having the option of consuming its share configuration from a central place like a RADOS object, rather than from a local text config file -- in that model, a UI can go look at the configuration directly rather than having the daemon report it back. This is kind of a Ceph-ized version of the way a lot of new projects are storing their configuration in etcd.
Interesting. The same should be possible for Samba when we eventually have a Ceph key-value store backend for Samba's dbwrap, and then use Registry configuration.
Is there API docs around for this key-value store?
Thanks,
Volker
Signature · 9 lines
Besuchen Sie die verinice.XP 2018 in Berlin, Anwenderkonferenz für Informationssicherheit vom 21.-23.03.2018 im Sofitel Kurfürstendamm Info & Anmeldung hier: http://veriniceXP.org SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen phone: +49-551-370000-0, fax: +49-551-370000-9 AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen http://www.sernet.de, mailto:[email protected]
Re: Samba Ceph dbwrap backend
Quoted 5 lines · Volker Lendecke
Quoted 3 lines
Interesting. The same should be possible for Samba when we eventually have a Ceph key-value store backend for Samba's dbwrap, and then use Registry configuration.
Is there API docs around for this key-value store?
There are a few different possible approaches to take, the simplest would probably be to use Ceph omap directly, which is documented at:
http://docs.ceph.com/docs/master/rados/api/librados/
Locking and transactions would need to be layered atop this, possibly via a separate Samba Object Class[1] running on the Ceph OSDs.
All theoretical at this stage unfortunately, but I hope to get some time to work on it in the future.
Cheers, David
[1] Ceph Object Classes (slide 28)
https://www.slideshare.net/sageweil1/20150311-vault15-librados
Please find a new version of the patchset attached, with the following changes:
- use rados instace guid instead of hostname for instance string
- drop per-share metadata TODO
- avoid multiple rados_shutdown() calls from child processes
- add documentation for new ceph:service_user_id parameter
Feedback / push appreciated.
Cheers, David
From 7b5a4217f40e00cf43bb681c8e673ae4451a2d0d Mon Sep 17 00:00:00 2001
From: David Disseldorp <[email protected]>
Date: Wed, 7 Feb 2018 14:58:48 +0100
Subject: [PATCH 1/2] ceph: add rados service integration
On startup, Samba advertises its presence to the Ceph Manager Service via rados_service_register(). librados spawns a number of threads to maintain the cluster connection and maintain periodic service heartbeats.
--- source3/smbd/rados_service.c | 146 +++++++++++++++++++++++++++++++++++++++++++ source3/smbd/rados_service.h | 30 +++++++++ source3/smbd/server.c | 7 +++ source3/wscript | 10 ++- source3/wscript_build | 12 +++- 5 files changed, 203 insertions(+), 2 deletions(-) create mode 100644 source3/smbd/rados_service.c create mode 100644 source3/smbd/rados_service.h
source3/smbd/rados_service.c new file +146 -0
@@ -0,0 +1,146 @@ +/* +Copyright (C) David Disseldorp 2018 + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include "source3/include/includes.h" +#include <rados/librados.h> +#include "rados_service.h" + +struct ceph_service_state { +pid_t svc_reg_pid; +rados_t cluster; +}; + +static int ceph_service_destroy (struct ceph_service_state *state) +{ +/* +* ensure that processes forked after spawning the service registration +* connection don't additionally attempt cleanup. +*/ +if (state->svc_reg_pid !=getpid ()) { +DBG_DEBUG ("ignoring non-spawner destructor\n" ); +return 0 ; + } + +DBG_DEBUG ("ending Ceph cluster service registration\n" ); +rados_shutdown (state->cluster); +DBG_DEBUG ("rados service connection shutdown\n" ); +return 0 ; +} + +static int ceph_service_instance_gen (char *buf,size_t buflen, +const char *daemon, +uint64_t instance_guid) +{ +int ret; + + ret =snprintf (buf, buflen,"%s:0x%016llx" , daemon, + (unsigned long long )instance_guid); +if (ret >= buflen) { +DBG_ERR ("Ceph instance name too long, skipping service " +"registration\n" ); +return -ENAMETOOLONG ; + } + +return 0 ; +} + +int ceph_service_register (struct tevent_context *ev_ctx,const char *daemon) +{ +int ret; +const char *user_id =NULL ; +const char *conf_file =NULL ; +const char *cluster_name =NULL ; +uint64_t instance_guid; +char instance_buf[128 ]; +struct ceph_service_state *state =talloc (ev_ctx, + struct ceph_service_state); + +if (state ==NULL ) { +return -ENOMEM ; + } + + state->svc_reg_pid =getpid (); + +/* +* XXX This does not reuse the existing (share based) "ceph:user_id" +* parameter, to allow for: +* - running Samba with Ceph support, but without service registration. +* - future mapping between Samba and Ceph users. +*/ + user_id =lp_parm_const_string (GLOBAL_SECTION_SNUM ,"ceph" , +"service_user_id" ,NULL ); +if (user_id ==NULL ) { +DBG_DEBUG ("built with Ceph support, but missing ceph:" +"service_user_id configuration - skipping service " +"registration\n" ); + ret =0 ; + goto out_mem_free; + } + +/* a NULL cluster_name or conf_file sees librados use the defaults */ + cluster_name =lp_parm_const_string (GLOBAL_SECTION_SNUM ,"ceph" , +"cluster_name" ,NULL ); + conf_file =lp_parm_const_string (GLOBAL_SECTION_SNUM ,"ceph" , +"config_file" ,NULL ); + + ret =rados_create2 (&state->cluster, cluster_name, user_id,0 ); +if (ret <0 ) { +DBG_ERR ("failed to create ceph cluster context\n" ); + goto out_mem_free; + } + + ret =rados_conf_read_file (state->cluster, conf_file); +if (ret <0 ) { +DBG_ERR ("failed to parse Ceph cluster config: %s\n" , +strerror (-ret)); + goto err_cluster_free; + } + + ret =rados_connect (state->cluster); +if (ret <0 ) { +DBG_ERR ("failed to connect to ceph cluster\n" ); + goto err_cluster_free; + } + + instance_guid =rados_get_instance_id (state->cluster); + + ret =ceph_service_instance_gen (instance_buf,sizeof (instance_buf), + daemon, instance_guid); +if (ret <0 ) { + goto out_mem_free; + } + +DBG_DEBUG ("registering as %s with %s Ceph cluster\n" , instance_buf, + (cluster_name !=NULL ? cluster_name :"default" )); + + ret =rados_service_register (state->cluster,"samba" , instance_buf,"" ); +if (ret <0 ) { +DBG_ERR ("failed to register service with ceph cluster\n" ); + goto err_cluster_free; + } + +/* close cluster conn and drop service listing on server exit */ +talloc_set_destructor (state, ceph_service_destroy); + +return 0 ; + +err_cluster_free: +rados_shutdown (state->cluster); +out_mem_free: +talloc_free (state); +return ret; +}
source3/smbd/rados_service.h new file +30 -0
@@ -0,0 +1,30 @@ +/* +Copyright (C) David Disseldorp 2018 + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#ifndef _RADOS_SERVICE_H +#define _RADOS_SERVICE_H + +int ceph_service_register (struct tevent_context *ev_ctx,const char *daemon); +#if !defined(HAVE_LIBRADOS_SERVICES ) +/* built without librados service support, do nothing */ +int ceph_service_register (struct tevent_context *ev_ctx,const char *daemon) +{ +return 0 ; +} +#endif + +#endif /* _RADOS_SERVICE_H */
source3/smbd/server.c +7 -0
@@ -54,6 +54,7 @@#include "lib/util/sys_rw.h" #include "cleanupdb.h" #include "g_lock.h" +#include "rados_service.h" #ifdef CLUSTER_SUPPORT #include "ctdb_protocol.h"
@@ -1593,6 +1594,7 @@ extern void build_options(bool screen);struct smbd_parent_context *parent =NULL ;TALLOC_CTX *frame;NTSTATUS status; +int ret;struct tevent_context *ev_ctx;struct messaging_context *msg_ctx;struct server_id server_id;
@@ -2053,6 +2055,11 @@ extern void build_options(bool screen);* -- bad things will happen if smbd is launched via inetd * and we fork a copy of ourselves here */ if (is_daemon && !interactive) { + ret =ceph_service_register (ev_ctx,"smbd" ); +if (ret <0 ) { +exit_daemon ("Samba failed to register with Ceph " +"cluster" , -ret); + }if (rpc_lsasd_daemon () ==RPC_DAEMON_FORK ) {start_lsasd (ev_ctx, msg_ctx);
source3/wscript +9 -1
@@ -1544,14 +1544,18 @@ main() { conf.env['CCFLAGS_CEPHFS'] = "-D_FILE_OFFSET_BITS=64" if Options.options.libcephfs_dir: + conf.env['CPPPATH_RADOS'] = Options.options.libcephfs_dir + '/include' conf.env['CPPPATH_CEPHFS'] = Options.options.libcephfs_dir + '/include' conf.env['LIBPATH_CEPHFS'] = Options.options.libcephfs_dir + '/lib' conf.env['LIBPATH_CEPH-COMMON'] = Options.options.libcephfs_dir + '/lib/ceph' + conf.env['LIBPATH_RADOS'] = Options.options.libcephfs_dir + '/lib' if (Options.options.with_cephfs and conf.CHECK_HEADERS('cephfs/libcephfs.h', False, False, 'cephfs') and + conf.CHECK_HEADERS('rados/librados.h', False, False, 'rados') and conf.CHECK_LIB('cephfs', shlib=True) and - conf.CHECK_LIB('ceph-common', shlib=True)): + conf.CHECK_LIB('ceph-common', shlib=True) and + conf.CHECK_LIB('rados', shlib=True)): if Options.options.with_acl_support: conf.DEFINE('HAVE_CEPH', '1') if conf.CHECK_FUNCS_IN('ceph_statx', 'cephfs ceph-common',
@@ -1561,6 +1565,10 @@ main() { Logs.warn("ceph support disabled due to --without-acl-support") conf.undefine('HAVE_CEPH') + if conf.CHECK_FUNCS_IN('rados_service_register', 'rados ceph-common', + headers='rados/librados.h'): + conf.DEFINE('HAVE_LIBRADOS_SERVICES', '1') + if Options.options.with_glusterfs: conf.CHECK_CFG(package='glusterfs-api', args='"glusterfs-api >= 4" --cflags --libs', msg='Checking for glusterfs-api >= 4', uselib_store="GFAPI")
source3/wscript_build +11 -1
@@ -1052,6 +1052,16 @@ bld.SAMBA3_SUBSYSTEM('SPOOLSSD', RPC_SOCK_HELPER ''') +smbd_rados_deps = '' +if bld.CONFIG_SET('HAVE_LIBRADOS_SERVICES'): + bld.SAMBA_LIBRARY('rados_service', + source='smbd/rados_service.c', + deps='SMBCONF_PARAM samba-debug', + local_include=False, + public_deps='rados ceph-common', + private_library=True) + smbd_rados_deps += ' rados_service' + ########################## BINARIES ################################# bld.SAMBA3_BINARY('smbd/smbd',
@@ -1064,7 +1074,7 @@ bld.SAMBA3_BINARY('smbd/smbd', FSSD MDSSD SPOOLSSD - ''', + ''' + smbd_rados_deps, install_path='${SBINDIR}')
Signature · 12 lines
2.13.6 From 463681ff53b5b0f12c5b6b3be5db4d27aceda881 Mon Sep 17 00:00:00 2001 From: David Disseldorp <[email protected]> Date: Tue, 13 Feb 2018 20:16:48 +0100 Subject: [PATCH 2/2] docs/vfs_ceph: describe ceph:service_user_id parameter Signed-off-by: David Disseldorp <[email protected]> --- docs-xml/manpages/vfs_ceph.8.xml | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
docs-xml/manpages/vfs_ceph.8.xml +14 -0
@@ -97,6 +97,20 @@ </listitem> </varlistentry> + <varlistentry> + <term>ceph:service_user_id = id</term> + <listitem> + <para> + Allows one to explicitly set the client ID used for + registration of the Samba smbd daemon with the Ceph + Manager service. + </para> + <para> + Example: ceph:service_user_id = client.service_reg + </para> + </listitem> + </varlistentry> + </variablelist> </refsect1>
Signature · 1 line
2.13.6
source3/smbd/rados_service.c line 130
Quoted 13 lines · David Disseldorp
source3/smbd/rados_service.c new file +12
-0+ instance_guid =rados_get_instance_id (state->cluster); + + ret =ceph_service_instance_gen (instance_buf,sizeof (instance_buf), + daemon, instance_guid); +if (ret <0 ) { + goto out_mem_free; + } + +DBG_DEBUG ("registering as %s with %s Ceph cluster\n" , instance_buf, + (cluster_name !=NULL ? cluster_name :"default" )); + + ret =rados_service_register (state->cluster,"samba" , instance_buf,"" );
IIUC, if the program (samba in this case) dies, and then is restarted (maybe host reboots?), it'll get a new instance_guid, right? Or am I misunderstanding what rados_get_instance_id will return here?
Signature · 1 line
Jeff Layton <[email protected]>
Hi Jeff,
Quoted 3 lines · Jeff Layton
IIUC, if the program (samba in this case) dies, and then is restarted (maybe host reboots?), it'll get a new
instance_guid, right? Or am I misunderstanding whatrados_get_instance_idwill return here?
Yes, that's right. On smbd restart it'll reconnect and get a new instance ID. The hostname is carried in the metadata, which makes it a little easier to identify what's going on from the Ceph side.
AFAICT the service record (unfortunately) remains present after rados_shutdown() until a timeout on the manager is tripped, so it's quite possible to have to registrations for the same host for a short amount of time.
Cheers, David
Quoted 12 lines · David Disseldorp
Hi Jeff,
Quoted 3 lines · Jeff Layton
IIUC, if the program (samba in this case) dies, and then is restarted (maybe host reboots?), it'll get a new
instance_guid, right? Or am I misunderstanding whatrados_get_instance_idwill return here?Yes, that's right. On smbd restart it'll reconnect and get a new instance ID. The hostname is carried in the metadata, which makes it a little easier to identify what's going on from the Ceph side.
Quoted 6 lines
AFAICT the service record (unfortunately) remains present after
rados_shutdown()until a timeout on the manager is tripped, so it's quite possible to have to registrations for the same host for a short amount of time.Cheers, David
Got it, thanks. I guess the fact that it changes it not a problem here? I'm not that well versed in what ceph-mgr does, tbqh...
One thing you may want to be careful about: I found some thread safety problems a few months ago in how CephContext objects are handled in the ceph code. While I patched them up as best I could, a lot of older shipping versions still have those bugs. Even with those fixes, I think there may still be lurking problems when we have multiple ceph/rados clients within the same process image.
The upshot here is that if you have the rados client here and (e.g.) a vfs_ceph client for exporting cephfs, you can hit some races particularly on setup and teardown of those clients that can cause crashes.
libcephfs has a ceph_create_with_context image and librados has something similar. You might consider having a truly global cct object, and then have the vfs_ceph create a client handle with the same object.
Signature · 1 line
Jeff Layton <[email protected]>