nvme-cli: add "nvme monitor" subcommand
Subsystems (3)
v2 cover nvme-cli: add "nvme monitor" subcommand
From: Martin Wilck <[email protected]>
This patch set adds a new subcommand "nvme monitor". In this mode, nvme-cli runs continuously, monitors various events relevant for discovery, and autoconnects to newly discovered subsystems.
This series is based on, and requires, my previously submitted patch series "Some minor fixes/additions for nvme-cli".
The monitor mode is suitable to be run in a systemd service. An appropriate unit file is provided. As such, "nvme monitor" can be used as an alternative to the current auto-connection mechanism based on udev rules and systemd template units.
This method for discovery and autodetection has some advantages over the current udev-rule based approach:
* The monitor creates persistent discovery controllers if possible, and monitors them for AENs.
* The monitor tracks "/etc/nvme/discovery.conf" changes using inotify.
* The monitor keeps record about existing NMVe transport connections and associated discovery controllers (if any). Thus it can avoid recreating discovery controllers if a persistent discovery controller is already present on a given transport address, without having to search sysfs for a matching controller.
* The monitor is aware of ongoing discoveries (as much as it has started them itself) and can queue up additional processes without taking the risk to miss any events. Missing events is possible with the current systemd-based activation of NVMe discovery.
* I expect slightly less resource usage compared to the current udev-rule based discovery, as less fork()/exec() operations are required. The effect will probably to be small though, and I have no numbers.
* The monitor will be able to support network discovery too, and react on mDNS records being published in the network. This functionality will be implemented using libavahi; Enzo Matsumiya is working on it. Once finished, "nvme monitor" will be able to track discovery events for every NVMeoF transport.
I've tested fc_udev_device handling for NVMeoFC with an Ontap target, and AEN handling for RDMA using a Linux nvmet target.
# Changes wrt "RFC: add "nvme monitor" subcommand" patch series
A lot.
* Separated out those changes that are not directly related to the monitor into a separate series, as requested by Sagi (see above). The part that changes some symbols in fabrics.c from static to global is still part of the "monitor" series though, as it doesn't make sense to do this without the monitor.
* Reorganized the patches into less, bigger chunks, as requested by Hannes.
* Changed the behavior of the monitor:
- Autoconnect by default, and allow to use "-n/--no-connect" for opt-out.
- Always create persistent discovery connections (Sagi): it makes no sense to use temporary discovery controllers if the monitor is running.
- Don't try to create discovery controllers on every transport connection found. Sagi had pointed out that this behavior in the RFC was wrong. Instead, run discoveries from /etc/nvme/discovery.conf on startup.
- Don't automatically disable 70-nvmf-autoconnect.rules (Hannes). I have put this in the systemd service file for now, because I think it makes no sense to run the monitor as a systemd service and run the discovery via udev rules at the same time. If this is also unwanted, I can remove it entirely of course.
* Moved the event handling into a separate "library". This was motivated by the additional events monitored in the v2 series, and by the prospect of adding more (and network-related ones, where timeout handling will become important) when the mDNS support is merged. I've actually spent most work on this part, stabilizing the API, creating tests and fixing issues. I have published this separately on https://github.com/mwilck/minivent,
together with the unit tests that I didn't want to add to the nvme-cli patch set at this time.
* Added new features:
- /etc/nvme/discovery.conf: Parse it on startup, and monitor changes with inotify.
- parent/child messaging: allow children running discovery to communicate with the parent monitor process via a Unix socket. Without this, the discovery of newly created discovery controllers by the parent is fragile, because the monitor has no way to figure out whether a given controller was created by its own child or by another process. Also, it wasn't possible to pass existing discovery controller devices to children running discovery from the conf file, or for referrals. This had the effect that children would create a temporary discovery controller even though persistent controller for the same connection existed already.
* Use the "udev" udev monitor socket by default rather than "kernel". When I made the first submission, I was unaware that filtering on "kernel" netlink sockets is much less efficient than on "udev" sockets. Thus "kernel" is only used if udevd is not available.
* Lots of bugs and minor issues fixed.
# Todo
* Implement support for RDMA and TCP protocols. As noted above, Enzo Matsumiya has been working on this, and we are cooperating to merge our efforts.
Reviews and comments welcome.
Thanks,
PS: I've pushed both this series and the "minor fixes" series to
https://github.com/linux-nvme/nvme-cli/pull/877. The CI fails because I don't know how to resolve the dependency of libudev in the Ubuntu / powerpc cross-compilation environment used there. Help would be appreciated.
Martin Wilck (16):
fabrics: export symbols required for monitor functionality
nvme-cli: add code for event and timeout handling
monitor: add basic "nvme monitor" functionality
monitor: implement uevent handling
conn-db: add simple connection registry
monitor: monitor_discovery(): try to reuse existing controllers
monitor: kill running discovery tasks on exit
monitor: add option --cleanup / -C
monitor: handling of add/remove uevents for nvme controllers
monitor: discover from conf file on startup
monitor: watch discovery.conf with inotify
monitor: add parent/child messaging and "notify" message exchange
monitor: add "query device" message exchange
completions: add completions for nvme monitor
nvmf-autoconnect: add unit file for nvme-monitor.service nvme-monitor(1): add man page for nvme-monitor
.github/workflows/c-cpp.yml | 4 + Documentation/cmds-main.txt | 4 + Documentation/nvme-monitor.1 | 180 +++ Documentation/nvme-monitor.html | 1018 ++++++++++++ Documentation/nvme-monitor.txt | 144 ++ Makefile | 21 +- common.h | 17 + completions/bash-nvme-completion.sh | 6 +- conn-db.c | 425 +++++ conn-db.h | 171 ++ event/event.c | 481 ++++++ event/event.h | 460 ++++++ event/timeout.c | 373 +++++ event/timeout.h | 110 ++ event/ts-util.c | 107 ++ event/ts-util.h | 129 ++ fabrics.c | 436 +++--- fabrics.h | 52 + list.h | 349 +++++ monitor.c | 1370 +++++++++++++++++ monitor.h | 14 + nvme-builtin.h | 1 + nvme.c | 13 + nvmf-autoconnect/systemd/nvme-monitor.service | 18 + util/cleanup.c | 2 + util/cleanup.h | 1 + 26 files changed, 5676 insertions(+), 230 deletions(-) create mode 100644 Documentation/nvme-monitor.1 create mode 100644 Documentation/nvme-monitor.html create mode 100644 Documentation/nvme-monitor.txt create mode 100644 conn-db.c create mode 100644 conn-db.h create mode 100644 event/event.c create mode 100644 event/event.h create mode 100644 event/timeout.c create mode 100644 event/timeout.h create mode 100644 event/ts-util.c create mode 100644 event/ts-util.h create mode 100644 list.h create mode 100644 monitor.c create mode 100644 monitor.h create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 1/16 fabrics: export symbols required for monitor functionality
From: Martin Wilck <[email protected]>
These macros, functions and variables will be used by the "nvme monitor" functionality. Convert them to globally visible symbols.
Being able to access struct config and the "cfg" variable from fabrics.c is essential for the monitor to leverage the existing, well tested code as much as possible. Rename "cfg" to "fabrics_cfg" to make the global variable name less generic.
--- fabrics.c | 384 +++++++++++++++++++++++++----------------------------- fabrics.h | 42 ++++++ 2 files changed, 222 insertions(+), 204 deletions(-)
fabrics.c +180 -204
@@ -67,33 +67,8 @@ const char *conarg_traddr = "traddr";const char *conarg_trsvcid ="trsvcid" ;const char *conarg_host_traddr ="host_traddr" ; -static struct config { -const char *nqn; -const char *transport; -const char *traddr; -const char *trsvcid; -const char *host_traddr; -const char *hostnqn; -const char *hostid; -int nr_io_queues; -int nr_write_queues; -int nr_poll_queues; -int queue_size; -int keep_alive_tmo; -int reconnect_delay; -int ctrl_loss_tmo; -int tos; -const char *raw; -char *device; -int duplicate_connect; -int disable_sqflow; -int hdr_digest; -int data_digest; -bool persistent; -bool matching_only; -const char *output_format; -} cfg = { - .ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO , +struct fabrics_config fabrics_cfg = { + .ctrl_loss_tmo =-1 , .output_format ="normal" , };
@@ -109,7 +84,6 @@ struct connect_args {struct connect_args *tracked_ctrls; -#define BUF_SIZE 4096#define PATH_NVME_FABRICS "/dev/nvme-fabrics"#define PATH_NVMF_DISC "/etc/nvme/discovery.conf"#define PATH_NVMF_HOSTNQN "/etc/nvme/hostnqn"
@@ -129,7 +103,7 @@ static const match_table_t opt_tokens = { { OPT_ERR, NULL }, }; -static const char *arg_str (const char *const *strings, +const char *arg_str (const char *const *strings,size_t array_size,size_t idx) {if (idx < array_size && strings[idx])
@@ -137,7 +111,7 @@ static const char *arg_str(const char * const *strings,return "unrecognized" ; } -static const char *const trtypes[] = { +const char *const trtypes[] = { [NVMF_TRTYPE_RDMA ] ="rdma" , [NVMF_TRTYPE_FC ] ="fc" , [NVMF_TRTYPE_TCP ] ="tcp" ,
@@ -228,14 +202,12 @@ static const char *cms_str(__u8 cm)return arg_str (cms,ARRAY_SIZE (cms), cm); } -static int do_discover (char *argstr,bool connect,enum nvme_print_flags flags); -/* * parse strings with connect arguments to find a particular field. * If field found, return string containing field value. If field * not found, return an empty string. */ -static char *parse_conn_arg (const char *conargs,const char delim,const char *field) +char *parse_conn_arg (const char *conargs,const char delim,const char *field) {char *s, *e;size_t cnt;
@@ -279,7 +251,7 @@ empty_field:return strdup ("\0" ); } -static int ctrl_instance (const char *device) +int ctrl_instance (const char *device) {char d[64 ];const char *p;
@@ -542,7 +514,7 @@ out:return ret; } -static int remove_ctrl (int instance) +int remove_ctrl (int instance) {char *sysfs_path;int ret;
@@ -802,10 +774,10 @@ static void save_discovery_log(struct nvmf_disc_rsp_page_hdr *log, int numrec)int fd;int len, ret; - fd =open (cfg.raw,O_CREAT |O_RDWR |O_TRUNC ,S_IRUSR |S_IWUSR ); + fd =open (fabrics_cfg.raw,O_CREAT |O_RDWR |O_TRUNC ,S_IRUSR |S_IWUSR );if (fd <0 ) {msg (LOG_ERR ,"failed to open %s: %s\n" , - cfg.raw,strerror (errno)); + fabrics_cfg.raw,strerror (errno));return ; }
@@ -814,9 +786,9 @@ static void save_discovery_log(struct nvmf_disc_rsp_page_hdr *log, int numrec) ret =write (fd, log, len);if (ret <0 )msg (LOG_ERR ,"failed to write to %s: %s\n" , - cfg.raw,strerror (errno)); + fabrics_cfg.raw,strerror (errno));else -printf ("Discovery log is saved to %s\n" , cfg.raw); +printf ("Discovery log is saved to %s\n" , fabrics_cfg.raw);close (fd); }
@@ -878,9 +850,9 @@ char *hostnqn_read(void)static int nvmf_hostnqn_file (void ) { - cfg.hostnqn =hostnqn_read (); + fabrics_cfg.hostnqn =hostnqn_read (); -return cfg.hostnqn !=NULL ; +return fabrics_cfg.hostnqn !=NULL ; }static int nvmf_hostid_file (void )
@@ -896,8 +868,8 @@ static int nvmf_hostid_file(void)if (fgets (hostid,sizeof (hostid), f) ==NULL ) goto out; - cfg.hostid =strdup (hostid); -if (!cfg.hostid) + fabrics_cfg.hostid =strdup (hostid); +if (!fabrics_cfg.hostid) goto out; ret = true;
@@ -955,68 +927,71 @@ add_argument(char **argstr, int *max_len, char *arg_str, const char *arg)return 0 ; } -static int build_options (char *argstr,int max_len,bool discover) +int build_options (char *argstr,int max_len,bool discover) {int len; -if (!cfg.transport) { +if (!fabrics_cfg.transport) {msg (LOG_ERR ,"need a transport (-t) argument\n" );return -EINVAL ; } -if (strncmp (cfg.transport,"loop" ,4 )) { -if (!cfg.traddr) { +if (strncmp (fabrics_cfg.transport,"loop" ,4 )) { +if (!fabrics_cfg.traddr) {msg (LOG_ERR ,"need a address (-a) argument\n" );return -EINVAL ; } +/* Use the default ctrl loss timeout if unset */ +if (fabrics_cfg.ctrl_loss_tmo ==-1 ) + fabrics_cfg.ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO ; }/* always specify nqn as first arg - this will init the string */ - len =snprintf (argstr, max_len,"nqn=%s" , cfg.nqn); + len =snprintf (argstr, max_len,"nqn=%s" , fabrics_cfg.nqn);if (len <0 )return -EINVAL ; argstr += len; max_len -= len; -if (add_argument (&argstr, &max_len,"transport" , cfg.transport) || -add_argument (&argstr, &max_len,"traddr" , cfg.traddr) || -add_argument (&argstr, &max_len,"host_traddr" , cfg.host_traddr) || -add_argument (&argstr, &max_len,"trsvcid" , cfg.trsvcid) || - ((cfg.hostnqn ||nvmf_hostnqn_file ()) && -add_argument (&argstr, &max_len,"hostnqn" , cfg.hostnqn)) || - ((cfg.hostid ||nvmf_hostid_file ()) && -add_argument (&argstr, &max_len,"hostid" , cfg.hostid)) || +if (add_argument (&argstr, &max_len,"transport" , fabrics_cfg.transport) || +add_argument (&argstr, &max_len,"traddr" , fabrics_cfg.traddr) || +add_argument (&argstr, &max_len,"host_traddr" , fabrics_cfg.host_traddr) || +add_argument (&argstr, &max_len,"trsvcid" , fabrics_cfg.trsvcid) || + ((fabrics_cfg.hostnqn ||nvmf_hostnqn_file ()) && +add_argument (&argstr, &max_len,"hostnqn" , fabrics_cfg.hostnqn)) || + ((fabrics_cfg.hostid ||nvmf_hostid_file ()) && +add_argument (&argstr, &max_len,"hostid" , fabrics_cfg.hostid)) || (!discover &&add_int_argument (&argstr, &max_len,"nr_io_queues" , - cfg.nr_io_queues, false)) || + fabrics_cfg.nr_io_queues, false)) ||add_int_argument (&argstr, &max_len,"nr_write_queues" , - cfg.nr_write_queues, false) || + fabrics_cfg.nr_write_queues, false) ||add_int_argument (&argstr, &max_len,"nr_poll_queues" , - cfg.nr_poll_queues, false) || + fabrics_cfg.nr_poll_queues, false) || (!discover &&add_int_argument (&argstr, &max_len,"queue_size" , - cfg.queue_size, false)) || + fabrics_cfg.queue_size, false)) ||add_int_argument (&argstr, &max_len,"keep_alive_tmo" , - cfg.keep_alive_tmo, false) || + fabrics_cfg.keep_alive_tmo, false) ||add_int_argument (&argstr, &max_len,"reconnect_delay" , - cfg.reconnect_delay, false) || - (strncmp (cfg.transport,"loop" ,4 ) && + fabrics_cfg.reconnect_delay, false) || + (strncmp (fabrics_cfg.transport,"loop" ,4 ) &&add_int_argument (&argstr, &max_len,"ctrl_loss_tmo" , - cfg.ctrl_loss_tmo, true)) || + fabrics_cfg.ctrl_loss_tmo, true)) ||add_int_argument (&argstr, &max_len,"tos" , - cfg.tos, true) || + fabrics_cfg.tos, true) ||add_bool_argument (&argstr, &max_len,"duplicate_connect" , - cfg.duplicate_connect) || + fabrics_cfg.duplicate_connect) ||add_bool_argument (&argstr, &max_len,"disable_sqflow" , - cfg.disable_sqflow) || -add_bool_argument (&argstr, &max_len,"hdr_digest" , cfg.hdr_digest) || -add_bool_argument (&argstr, &max_len,"data_digest" , cfg.data_digest)) + fabrics_cfg.disable_sqflow) || +add_bool_argument (&argstr, &max_len,"hdr_digest" , fabrics_cfg.hdr_digest) || +add_bool_argument (&argstr, &max_len,"data_digest" , fabrics_cfg.data_digest))return -EINVAL ;return 0 ; } -static void set_discovery_kato (struct config *cfg) +static void set_discovery_kato (struct fabrics_config *cfg) {/* Set kato to NVMF_DEF_DISC_TMO for persistent controllers */ if (cfg->persistent && !cfg->keep_alive_tmo)
@@ -1026,41 +1001,41 @@ static void set_discovery_kato(struct config *cfg) cfg->keep_alive_tmo =0 ; } -static void discovery_trsvcid (struct config *cfg) +static void discovery_trsvcid (struct fabrics_config *fabrics_cfg) { -if (!strcmp (cfg->transport,"tcp" )) { +if (!strcmp (fabrics_cfg->transport,"tcp" )) {/* Default port for NVMe/TCP discovery controllers */ - cfg->trsvcid =__stringify (NVME_DISC_IP_PORT ); - }else if (!strcmp (cfg->transport,"rdma" )) { + fabrics_cfg->trsvcid =__stringify (NVME_DISC_IP_PORT ); + }else if (!strcmp (fabrics_cfg->transport,"rdma" )) {/* Default port for NVMe/RDMA controllers */ - cfg->trsvcid =__stringify (NVME_RDMA_IP_PORT ); + fabrics_cfg->trsvcid =__stringify (NVME_RDMA_IP_PORT ); } } -static bool traddr_is_hostname (struct config *cfg) +static bool traddr_is_hostname (struct fabrics_config *fabrics_cfg) {char addrstr[NVMF_TRADDR_SIZE ]; -if (!cfg->traddr || !cfg->transport) +if (!fabrics_cfg->traddr || !fabrics_cfg->transport)return false; -if (strcmp (cfg->transport,"tcp" ) &&strcmp (cfg->transport,"rdma" )) +if (strcmp (fabrics_cfg->transport,"tcp" ) &&strcmp (fabrics_cfg->transport,"rdma" ))return false; -if (inet_pton (AF_INET , cfg->traddr, addrstr) >0 || -inet_pton (AF_INET6 , cfg->traddr, addrstr) >0 ) +if (inet_pton (AF_INET , fabrics_cfg->traddr, addrstr) >0 || +inet_pton (AF_INET6 , fabrics_cfg->traddr, addrstr) >0 )return false;return true; } -static int hostname2traddr (struct config *cfg) +static int hostname2traddr (struct fabrics_config *fabrics_cfg) {struct addrinfo *host_info, hints = {.ai_family =AF_UNSPEC };char addrstr[NVMF_TRADDR_SIZE ];const char *p;int ret; - ret =getaddrinfo (cfg->traddr,NULL , &hints, &host_info); + ret =getaddrinfo (fabrics_cfg->traddr,NULL , &hints, &host_info);if (ret) { -msg (LOG_ERR ,"failed to resolve host %s info\n" , cfg->traddr); +msg (LOG_ERR ,"failed to resolve host %s info\n" , fabrics_cfg->traddr);return ret; }
@@ -1077,17 +1052,17 @@ static int hostname2traddr(struct config *cfg)break ;default :msg (LOG_ERR ,"unrecognized address family (%d) %s\n" , - host_info->ai_family, cfg->traddr); + host_info->ai_family, fabrics_cfg->traddr); ret = -EINVAL ; goto free_addrinfo; }if (!p) { -msg (LOG_ERR ,"failed to get traddr for %s\n" , cfg->traddr); +msg (LOG_ERR ,"failed to get traddr for %s\n" , fabrics_cfg->traddr); ret = -errno; goto free_addrinfo; } - cfg->traddr =strdup (addrstr); + fabrics_cfg->traddr =strdup (addrstr); free_addrinfo:freeaddrinfo (host_info);
@@ -1121,78 +1096,78 @@ retry:return -EINVAL ; p += len; -if (cfg.hostnqn &&strcmp (cfg.hostnqn,"none" )) { - len =sprintf (p,",hostnqn=%s" , cfg.hostnqn); +if (fabrics_cfg.hostnqn &&strcmp (fabrics_cfg.hostnqn,"none" )) { + len =sprintf (p,",hostnqn=%s" , fabrics_cfg.hostnqn);if (len <0 )return -EINVAL ; p += len; } -if (cfg.hostid &&strcmp (cfg.hostid,"none" )) { - len =sprintf (p,",hostid=%s" , cfg.hostid); +if (fabrics_cfg.hostid &&strcmp (fabrics_cfg.hostid,"none" )) { + len =sprintf (p,",hostid=%s" , fabrics_cfg.hostid);if (len <0 )return -EINVAL ; p += len; } -if (cfg.queue_size && !discover) { - len =sprintf (p,",queue_size=%d" , cfg.queue_size); +if (fabrics_cfg.queue_size && !discover) { + len =sprintf (p,",queue_size=%d" , fabrics_cfg.queue_size);if (len <0 )return -EINVAL ; p += len; } -if (cfg.nr_io_queues && !discover) { - len =sprintf (p,",nr_io_queues=%d" , cfg.nr_io_queues); +if (fabrics_cfg.nr_io_queues && !discover) { + len =sprintf (p,",nr_io_queues=%d" , fabrics_cfg.nr_io_queues);if (len <0 )return -EINVAL ; p += len; } -if (cfg.nr_write_queues) { - len =sprintf (p,",nr_write_queues=%d" , cfg.nr_write_queues); +if (fabrics_cfg.nr_write_queues) { + len =sprintf (p,",nr_write_queues=%d" , fabrics_cfg.nr_write_queues);if (len <0 )return -EINVAL ; p += len; } -if (cfg.nr_poll_queues) { - len =sprintf (p,",nr_poll_queues=%d" , cfg.nr_poll_queues); +if (fabrics_cfg.nr_poll_queues) { + len =sprintf (p,",nr_poll_queues=%d" , fabrics_cfg.nr_poll_queues);if (len <0 )return -EINVAL ; p += len; } -if (cfg.host_traddr &&strcmp (cfg.host_traddr,"none" )) { - len =sprintf (p,",host_traddr=%s" , cfg.host_traddr); +if (fabrics_cfg.host_traddr &&strcmp (fabrics_cfg.host_traddr,"none" )) { + len =sprintf (p,",host_traddr=%s" , fabrics_cfg.host_traddr);if (len <0 )return -EINVAL ; p+= len; } -if (cfg.reconnect_delay) { - len =sprintf (p,",reconnect_delay=%d" , cfg.reconnect_delay); +if (fabrics_cfg.reconnect_delay) { + len =sprintf (p,",reconnect_delay=%d" , fabrics_cfg.reconnect_delay);if (len <0 )return -EINVAL ; p += len; } -if ((e->trtype !=NVMF_TRTYPE_LOOP ) && (cfg.ctrl_loss_tmo >=-1 )) { - len =sprintf (p,",ctrl_loss_tmo=%d" , cfg.ctrl_loss_tmo); +if ((e->trtype !=NVMF_TRTYPE_LOOP ) && (fabrics_cfg.ctrl_loss_tmo >=-1 )) { + len =sprintf (p,",ctrl_loss_tmo=%d" , fabrics_cfg.ctrl_loss_tmo);if (len <0 )return -EINVAL ; p += len; } -if (cfg.tos !=-1 ) { - len =sprintf (p,",tos=%d" , cfg.tos); +if (fabrics_cfg.tos !=-1 ) { + len =sprintf (p,",tos=%d" , fabrics_cfg.tos);if (len <0 )return -EINVAL ; p += len; } -if (cfg.keep_alive_tmo) { - len =sprintf (p,",keep_alive_tmo=%d" , cfg.keep_alive_tmo); +if (fabrics_cfg.keep_alive_tmo) { + len =sprintf (p,",keep_alive_tmo=%d" , fabrics_cfg.keep_alive_tmo);if (len <0 )return -EINVAL ; p += len;
@@ -1210,14 +1185,14 @@ retry:return -EINVAL ; p += len; -if (cfg.hdr_digest) { +if (fabrics_cfg.hdr_digest) { len =sprintf (p,",hdr_digest" );if (len <0 )return -EINVAL ; p += len; } -if (cfg.data_digest) { +if (fabrics_cfg.data_digest) { len =sprintf (p,",data_digest" );if (len <0 )return -EINVAL ;
@@ -1277,7 +1252,7 @@ retry:if (discover) {enum nvme_print_flags flags; - flags =validate_output_format (cfg.output_format); + flags =validate_output_format (fabrics_cfg.output_format);if (flags <0 ) flags =NORMAL ; ret =do_discover (argstr, true, flags);
@@ -1300,7 +1275,7 @@ static bool cargs_match_found(struct nvmf_disc_rsp_page_entry *entry) cargs.transport =strdup (trtype_str (entry->trtype)); cargs.subsysnqn =strdup (entry->subnqn); cargs.trsvcid =strdup (entry->trsvcid); - cargs.host_traddr =strdup (cfg.host_traddr ?:"\0" ); + cargs.host_traddr =strdup (fabrics_cfg.host_traddr ?:"\0" );/* check if we have a match in the discovery recursion */ while (c) {
@@ -1324,11 +1299,11 @@ static bool should_connect(struct nvmf_disc_rsp_page_entry *entry)if (cargs_match_found (entry))return false; -if (!cfg.matching_only || !cfg.traddr) +if (!fabrics_cfg.matching_only || !fabrics_cfg.traddr)return true; len =space_strip_len (NVMF_TRADDR_SIZE , entry->traddr); -return !strncmp (cfg.traddr, entry->traddr, len); +return !strncmp (fabrics_cfg.traddr, entry->traddr, len); }static int connect_ctrls (struct nvmf_disc_rsp_page_hdr *log,int numrec)
@@ -1375,13 +1350,13 @@ static void nvmf_get_host_identifiers(int ctrl_instance)if (asprintf (&path,"%s/nvme%d" ,SYS_NVME , ctrl_instance) <0 )return ; - cfg.hostnqn =nvme_get_ctrl_attr (path,"hostnqn" ); - cfg.hostid =nvme_get_ctrl_attr (path,"hostid" ); + fabrics_cfg.hostnqn =nvme_get_ctrl_attr (path,"hostnqn" ); + fabrics_cfg.hostid =nvme_get_ctrl_attr (path,"hostid" ); } static DEFINE_CLEANUP_FUNC(cleanup_log, struct nvmf_disc_rsp_page_hdr *, free); -static int do_discover (char *argstr,bool connect,enum nvme_print_flags flags) +int do_discover (char *argstr,bool connect,enum nvme_print_flags flags) { struct nvmf_disc_rsp_page_hdr *log __cleanup__(cleanup_log) = NULL;char *dev_name;
@@ -1393,18 +1368,19 @@ static int do_discover(char *argstr, bool connect, enum nvme_print_flags flags)if (!cargs)return -ENOMEM ; -if (cfg.device && !ctrl_matches_connectargs (cfg.device, cargs, true)) { -free (cfg.device); - cfg.device =NULL ; +if (fabrics_cfg.device && + !ctrl_matches_connectargs (fabrics_cfg.device, cargs, true)) { +free (fabrics_cfg.device); + fabrics_cfg.device =NULL ; } -if (!cfg.device) - cfg.device =find_ctrl_with_connectargs (cargs); +if (!fabrics_cfg.device) + fabrics_cfg.device =find_ctrl_with_connectargs (cargs);free_connect_args (cargs); -if (!cfg.device) { +if (!fabrics_cfg.device) { instance =add_ctrl (argstr); }else { - instance =ctrl_instance (cfg.device); + instance =ctrl_instance (fabrics_cfg.device);nvmf_get_host_identifiers (instance); }if (instance <0 )
@@ -1414,9 +1390,9 @@ static int do_discover(char *argstr, bool connect, enum nvme_print_flags flags)return -errno; ret =nvmf_get_log_page_discovery (dev_name, &log, &numrec, &status);free (dev_name); -if (cfg.persistent) +if (fabrics_cfg.persistent)msg (LOG_NOTICE ,"Persistent device: nvme%d\n" , instance); -if (!cfg.device && !cfg.persistent) { +if (!fabrics_cfg.device && !fabrics_cfg.persistent) { err =remove_ctrl (instance);if (err)return err;
@@ -1426,7 +1402,7 @@ static int do_discover(char *argstr, bool connect, enum nvme_print_flags flags)case DISC_OK :if (connect) ret =connect_ctrls (log, numrec); -else if (cfg.raw || flags ==BINARY ) +else if (fabrics_cfg.raw || flags ==BINARY )save_discovery_log (log, numrec);else if (flags ==JSON )json_discovery_log (log, numrec);
@@ -1508,22 +1484,22 @@ static int discover_from_conf_file(const char *desc, char *argstr,if (err) goto free_and_continue; -if (!cfg.transport || !cfg.traddr) +if (!fabrics_cfg.transport || !fabrics_cfg.traddr) goto free_and_continue; - err = flags =validate_output_format (cfg.output_format); + err = flags =validate_output_format (fabrics_cfg.output_format);if (err <0 ) goto free_and_continue; -set_discovery_kato (&cfg); +set_discovery_kato (&fabrics_cfg); -if (traddr_is_hostname (&cfg)) { - ret =hostname2traddr (&cfg); +if (traddr_is_hostname (&fabrics_cfg)) { + ret =hostname2traddr (&fabrics_cfg);if (ret) goto out; } -if (!cfg.trsvcid) -discovery_trsvcid (&cfg); +if (!fabrics_cfg.trsvcid) +discovery_trsvcid (&fabrics_cfg); err =build_options (argstr,BUF_SIZE , true);if (err) {
@@ -1538,8 +1514,8 @@ static int discover_from_conf_file(const char *desc, char *argstr, free_and_continue:free (all_args);free (argv); - cfg.transport = cfg.traddr = cfg.trsvcid = - cfg.host_traddr =NULL ; + fabrics_cfg.transport = fabrics_cfg.traddr = + fabrics_cfg.trsvcid = fabrics_cfg.host_traddr =NULL ; } out:
@@ -1555,42 +1531,42 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect)bool quiet = false;OPT_ARGS (opts) = { -OPT_LIST ("transport" ,'t' , &cfg.transport,"transport type" ), -OPT_LIST ("traddr" ,'a' , &cfg.traddr,"transport address" ), -OPT_LIST ("trsvcid" ,'s' , &cfg.trsvcid,"transport service id (e.g. IP port)" ), -OPT_LIST ("host-traddr" ,'w' , &cfg.host_traddr,"host traddr (e.g. FC WWN's)" ), -OPT_LIST ("hostnqn" ,'q' , &cfg.hostnqn,"user-defined hostnqn (if default not used)" ), -OPT_LIST ("hostid" ,'I' , &cfg.hostid,"user-defined hostid (if default not used)" ), -OPT_LIST ("raw" ,'r' , &cfg.raw,"raw output file" ), -OPT_LIST ("device" ,'d' , &cfg.device,"use existing discovery controller device" ), -OPT_INT ("keep-alive-tmo" ,'k' , &cfg.keep_alive_tmo,"keep alive timeout period in seconds" ), -OPT_INT ("reconnect-delay" ,'c' , &cfg.reconnect_delay,"reconnect timeout period in seconds" ), -OPT_INT ("ctrl-loss-tmo" ,'l' , &cfg.ctrl_loss_tmo,"controller loss timeout period in seconds" ), -OPT_INT ("tos" ,'T' , &cfg.tos,"type of service" ), -OPT_FLAG ("hdr_digest" ,'g' , &cfg.hdr_digest,"enable transport protocol header digest (TCP transport)" ), -OPT_FLAG ("data_digest" ,'G' , &cfg.data_digest,"enable transport protocol data digest (TCP transport)" ), -OPT_INT ("nr-io-queues" ,'i' , &cfg.nr_io_queues,"number of io queues to use (default is core count)" ), -OPT_INT ("nr-write-queues" ,'W' , &cfg.nr_write_queues,"number of write queues to use (default 0)" ), -OPT_INT ("nr-poll-queues" ,'P' , &cfg.nr_poll_queues,"number of poll queues to use (default 0)" ), -OPT_INT ("queue-size" ,'Q' , &cfg.queue_size,"number of io queue elements to use (default 128)" ), -OPT_FLAG ("persistent" ,'p' , &cfg.persistent,"persistent discovery connection" ), +OPT_LIST ("transport" ,'t' , &fabrics_cfg.transport,"transport type" ), +OPT_LIST ("traddr" ,'a' , &fabrics_cfg.traddr,"transport address" ), +OPT_LIST ("trsvcid" ,'s' , &fabrics_cfg.trsvcid,"transport service id (e.g. IP port)" ), +OPT_LIST ("host-traddr" ,'w' , &fabrics_cfg.host_traddr,"host traddr (e.g. FC WWN's)" ), +OPT_LIST ("hostnqn" ,'q' , &fabrics_cfg.hostnqn,"user-defined hostnqn (if default not used)" ), +OPT_LIST ("hostid" ,'I' , &fabrics_cfg.hostid,"user-defined hostid (if default not used)" ), +OPT_LIST ("raw" ,'r' , &fabrics_cfg.raw,"raw output file" ), +OPT_LIST ("device" ,'d' , &fabrics_cfg.device,"existing discovery controller device" ), +OPT_INT ("keep-alive-tmo" ,'k' , &fabrics_cfg.keep_alive_tmo,"keep alive timeout period in seconds" ), +OPT_INT ("reconnect-delay" ,'c' , &fabrics_cfg.reconnect_delay,"reconnect timeout period in seconds" ), +OPT_INT ("ctrl-loss-tmo" ,'l' , &fabrics_cfg.ctrl_loss_tmo,"controller loss timeout period in seconds" ), +OPT_INT ("tos" ,'T' , &fabrics_cfg.tos,"type of service" ), +OPT_FLAG ("hdr_digest" ,'g' , &fabrics_cfg.hdr_digest,"enable transport protocol header digest (TCP transport)" ), +OPT_FLAG ("data_digest" ,'G' , &fabrics_cfg.data_digest,"enable transport protocol data digest (TCP transport)" ), +OPT_INT ("nr-io-queues" ,'i' , &fabrics_cfg.nr_io_queues,"number of io queues to use (default is core count)" ), +OPT_INT ("nr-write-queues" ,'W' , &fabrics_cfg.nr_write_queues,"number of write queues to use (default 0)" ), +OPT_INT ("nr-poll-queues" ,'P' , &fabrics_cfg.nr_poll_queues,"number of poll queues to use (default 0)" ), +OPT_INT ("queue-size" ,'Q' , &fabrics_cfg.queue_size,"number of io queue elements to use (default 128)" ), +OPT_FLAG ("persistent" ,'p' , &fabrics_cfg.persistent,"persistent discovery connection" ),OPT_FLAG ("quiet" ,'S' , &quiet,"suppress already connected errors" ), -OPT_FLAG ("matching" ,'m' , &cfg.matching_only,"connect only records matching the traddr" ), -OPT_FMT ("output-format" ,'o' , &cfg.output_format, output_format), +OPT_FLAG ("matching" ,'m' , &fabrics_cfg.matching_only,"connect only records matching the traddr" ), +OPT_FMT ("output-format" ,'o' , &fabrics_cfg.output_format, output_format),OPT_END () }; - cfg.tos =-1 ; + fabrics_cfg.tos =-1 ; ret =argconfig_parse (argc, argv, desc, opts);if (ret) goto out; - ret = flags =validate_output_format (cfg.output_format); + ret = flags =validate_output_format (fabrics_cfg.output_format);if (ret <0 ) goto out; -if (cfg.device &&strcmp (cfg.device,"none" )) { - cfg.device =strdup (cfg.device); -if (!cfg.device) { +if (fabrics_cfg.device &&strcmp (fabrics_cfg.device,"none" )) { + fabrics_cfg.device =strdup (fabrics_cfg.device); +if (!fabrics_cfg.device) { ret = -ENOMEM ; goto out; }
@@ -1599,24 +1575,24 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect)if (quiet) log_level =LOG_WARNING ; -if (cfg.device && !strcmp (cfg.device,"none" )) - cfg.device =NULL ; +if (fabrics_cfg.device && !strcmp (fabrics_cfg.device,"none" )) + fabrics_cfg.device =NULL ; - cfg.nqn =NVME_DISC_SUBSYS_NAME ; + fabrics_cfg.nqn =NVME_DISC_SUBSYS_NAME ; -if (!cfg.transport && !cfg.traddr) { +if (!fabrics_cfg.transport && !fabrics_cfg.traddr) { ret =discover_from_conf_file (desc, argstr, opts, connect); }else { -set_discovery_kato (&cfg); +set_discovery_kato (&fabrics_cfg); -if (traddr_is_hostname (&cfg)) { - ret =hostname2traddr (&cfg); +if (traddr_is_hostname (&fabrics_cfg)) { + ret =hostname2traddr (&fabrics_cfg);if (ret) goto out; } -if (!cfg.trsvcid) -discovery_trsvcid (&cfg); +if (!fabrics_cfg.trsvcid) +discovery_trsvcid (&fabrics_cfg); ret =build_options (argstr,BUF_SIZE , true);if (ret)
@@ -1635,35 +1611,35 @@ int fabrics_connect(const char *desc, int argc, char **argv)int instance, ret;OPT_ARGS (opts) = { -OPT_LIST ("transport" ,'t' , &cfg.transport,"transport type" ), -OPT_LIST ("nqn" ,'n' , &cfg.nqn,"nqn name" ), -OPT_LIST ("traddr" ,'a' , &cfg.traddr,"transport address" ), -OPT_LIST ("trsvcid" ,'s' , &cfg.trsvcid,"transport service id (e.g. IP port)" ), -OPT_LIST ("host-traddr" ,'w' , &cfg.host_traddr,"host traddr (e.g. FC WWN's)" ), -OPT_LIST ("hostnqn" ,'q' , &cfg.hostnqn,"user-defined hostnqn" ), -OPT_LIST ("hostid" ,'I' , &cfg.hostid,"user-defined hostid (if default not used)" ), -OPT_INT ("nr-io-queues" ,'i' , &cfg.nr_io_queues,"number of io queues to use (default is core count)" ), -OPT_INT ("nr-write-queues" ,'W' , &cfg.nr_write_queues,"number of write queues to use (default 0)" ), -OPT_INT ("nr-poll-queues" ,'P' , &cfg.nr_poll_queues,"number of poll queues to use (default 0)" ), -OPT_INT ("queue-size" ,'Q' , &cfg.queue_size,"number of io queue elements to use (default 128)" ), -OPT_INT ("keep-alive-tmo" ,'k' , &cfg.keep_alive_tmo,"keep alive timeout period in seconds" ), -OPT_INT ("reconnect-delay" ,'c' , &cfg.reconnect_delay,"reconnect timeout period in seconds" ), -OPT_INT ("ctrl-loss-tmo" ,'l' , &cfg.ctrl_loss_tmo,"controller loss timeout period in seconds" ), -OPT_INT ("tos" ,'T' , &cfg.tos,"type of service" ), -OPT_FLAG ("duplicate-connect" ,'D' , &cfg.duplicate_connect,"allow duplicate connections between same transport host and subsystem port" ), -OPT_FLAG ("disable-sqflow" ,'d' , &cfg.disable_sqflow,"disable controller sq flow control (default false)" ), -OPT_FLAG ("hdr-digest" ,'g' , &cfg.hdr_digest,"enable transport protocol header digest (TCP transport)" ), -OPT_FLAG ("data-digest" ,'G' , &cfg.data_digest,"enable transport protocol data digest (TCP transport)" ), +OPT_LIST ("transport" ,'t' , &fabrics_cfg.transport,"transport type" ), +OPT_LIST ("nqn" ,'n' , &fabrics_cfg.nqn,"nqn name" ), +OPT_LIST ("traddr" ,'a' , &fabrics_cfg.traddr,"transport address" ), +OPT_LIST ("trsvcid" ,'s' , &fabrics_cfg.trsvcid,"transport service id (e.g. IP port)" ), +OPT_LIST ("host-traddr" ,'w' , &fabrics_cfg.host_traddr,"host traddr (e.g. FC WWN's)" ), +OPT_LIST ("hostnqn" ,'q' , &fabrics_cfg.hostnqn,"user-defined hostnqn" ), +OPT_LIST ("hostid" ,'I' , &fabrics_cfg.hostid,"user-defined hostid (if default not used)" ), +OPT_INT ("nr-io-queues" ,'i' , &fabrics_cfg.nr_io_queues,"number of io queues to use (default is core count)" ), +OPT_INT ("nr-write-queues" ,'W' , &fabrics_cfg.nr_write_queues,"number of write queues to use (default 0)" ), +OPT_INT ("nr-poll-queues" ,'P' , &fabrics_cfg.nr_poll_queues,"number of poll queues to use (default 0)" ), +OPT_INT ("queue-size" ,'Q' , &fabrics_cfg.queue_size,"number of io queue elements to use (default 128)" ), +OPT_INT ("keep-alive-tmo" ,'k' , &fabrics_cfg.keep_alive_tmo,"keep alive timeout period in seconds" ), +OPT_INT ("reconnect-delay" ,'c' , &fabrics_cfg.reconnect_delay,"reconnect timeout period in seconds" ), +OPT_INT ("ctrl-loss-tmo" ,'l' , &fabrics_cfg.ctrl_loss_tmo,"controller loss timeout period in seconds" ), +OPT_INT ("tos" ,'T' , &fabrics_cfg.tos,"type of service" ), +OPT_FLAG ("duplicate-connect" ,'D' , &fabrics_cfg.duplicate_connect,"allow duplicate connections between same transport host and subsystem port" ), +OPT_FLAG ("disable-sqflow" ,'d' , &fabrics_cfg.disable_sqflow,"disable controller sq flow control (default false)" ), +OPT_FLAG ("hdr-digest" ,'g' , &fabrics_cfg.hdr_digest,"enable transport protocol header digest (TCP transport)" ), +OPT_FLAG ("data-digest" ,'G' , &fabrics_cfg.data_digest,"enable transport protocol data digest (TCP transport)" ),OPT_END () }; - cfg.tos =-1 ; + fabrics_cfg.tos =-1 ; ret =argconfig_parse (argc, argv, desc, opts);if (ret) goto out; -if (traddr_is_hostname (&cfg)) { - ret =hostname2traddr (&cfg); +if (traddr_is_hostname (&fabrics_cfg)) { + ret =hostname2traddr (&fabrics_cfg);if (ret) goto out; }
@@ -1672,7 +1648,7 @@ int fabrics_connect(const char *desc, int argc, char **argv)if (ret) goto out; -if (!cfg.nqn) { +if (!fabrics_cfg.nqn) {msg (LOG_ERR ,"need a -n argument\n" ); ret = -EINVAL ; goto out;
@@ -1775,8 +1751,8 @@ int fabrics_disconnect(const char *desc, int argc, char **argv)int ret;OPT_ARGS (opts) = { -OPT_LIST ("nqn" ,'n' , &cfg.nqn, nqn), -OPT_LIST ("device" ,'d' , &cfg.device, device), +OPT_LIST ("nqn" ,'n' , &fabrics_cfg.nqn, nqn), +OPT_LIST ("device" ,'d' , &fabrics_cfg.device, device),OPT_END () };
@@ -1784,29 +1760,29 @@ int fabrics_disconnect(const char *desc, int argc, char **argv)if (ret) goto out; -if (!cfg.nqn && !cfg.device) { +if (!fabrics_cfg.nqn && !fabrics_cfg.device) {msg (LOG_ERR ,"need a -n or -d argument\n" ); ret = -EINVAL ; goto out; } -if (cfg.nqn) { - ret =disconnect_by_nqn (cfg.nqn); +if (fabrics_cfg.nqn) { + ret =disconnect_by_nqn (fabrics_cfg.nqn);if (ret <0 )msg (LOG_ERR ,"Failed to disconnect by NQN: %s\n" , - cfg.nqn); + fabrics_cfg.nqn);else { -printf ("NQN:%s disconnected %d controller(s)\n" , cfg.nqn, ret); +printf ("NQN:%s disconnected %d controller(s)\n" , fabrics_cfg.nqn, ret); ret =0 ; } } -if (cfg.device) { - ret =disconnect_by_device (cfg.device); +if (fabrics_cfg.device) { + ret =disconnect_by_device (fabrics_cfg.device);if (ret)msg (LOG_ERR ,"Failed to disconnect by device name: %s\n" , - cfg.device); + fabrics_cfg.device); } out:
fabrics.h +42 -0
46 lines not on this page · raw message
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 2/16 nvme-cli: add code for event and timeout handling
In reply to [email protected]From: Martin Wilck <[email protected]>
For the nvme monitor functionality, an event handling mechanism will be necessary which deals with event timeouts. While there are standard solutions for this (e.g. libevent), these add unnecessary complexity and dependencies to nvme-cli.
Add a small, straighforward event and timeout handling code based on epoll and timerfd.
This code is identical to what I've pushed recently to
https://github.com/mwilck/minivent, where I added a couple of unit tests to make sure the code is as robust as it needs to be.
--- Makefile | 11 +- common.h | 17 ++ event/event.c | 481 ++++++++++++++++++++++++++++++++++++++++++++++++ event/event.h | 460 +++++++++++++++++++++++++++++++++++++++++++++ event/timeout.c | 373 +++++++++++++++++++++++++++++++++++++ event/timeout.h | 110 +++++++++++ event/ts-util.c | 107 +++++++++++ event/ts-util.h | 129 +++++++++++++ 8 files changed, 1683 insertions(+), 5 deletions(-) create mode 100644 event/event.c create mode 100644 event/event.h create mode 100644 event/timeout.c create mode 100644 event/timeout.h create mode 100644 event/ts-util.c create mode 100644 event/ts-util.h
Makefile +6 -5
@@ -63,6 +63,7 @@ OBJS := nvme-print.o nvme-ioctl.o nvme-rpmb.o \ nvme-status.o nvme-filters.o nvme-topology.o UTIL_OBJS := util/argconfig.o util/suffix.o util/json.o util/parser.o util/cleanup.o util/log.o +EVENT_OBJS := event/event.o event/timeout.o event/ts-util.o PLUGIN_OBJS := \ plugins/intel/intel-nvme.o \
@@ -83,11 +84,11 @@ PLUGIN_OBJS := \ plugins/transcend/transcend-nvme.o \ plugins/zns/zns.o -nvme: nvme.c nvme.h $(OBJS) $(PLUGIN_OBJS) $(UTIL_OBJS) NVME-VERSION-FILE - $(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) $(INC) $< -o $(NVME) $(OBJS) $(PLUGIN_OBJS) $(UTIL_OBJS) $(LDFLAGS) +nvme: nvme.c nvme.h $(OBJS) $(PLUGIN_OBJS) $(UTIL_OBJS) $(EVENT_OBJS) NVME-VERSION-FILE + $(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) $(INC) $< -o $(NVME) $(OBJS) $(PLUGIN_OBJS) $(UTIL_OBJS) $(EVENT_OBJS) $(LDFLAGS) -verify-no-dep: nvme.c nvme.h $(OBJS) $(UTIL_OBJS) NVME-VERSION-FILE - $(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) $(INC) $< -o $@ $(OBJS) $(UTIL_OBJS) $(LDFLAGS) +verify-no-dep: nvme.c nvme.h $(OBJS) $(UTIL_OBJS)$(EVENT_OBJS) NVME-VERSION-FILE + $(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) $(INC) $< -o $@ $(OBJS) $(UTIL_OBJS) $(EVENT_OBJS) $(LDFLAGS) nvme.o: nvme.c nvme.h nvme-print.h nvme-ioctl.h util/argconfig.h util/suffix.h nvme-lightnvm.h fabrics.h $(QUIET_CC)$(CC) $(CPPFLAGS) $(CFLAGS) $(INC) -c $<
@@ -107,7 +108,7 @@ test: all: doc clean: - $(RM) $(NVME) $(OBJS) $(PLUGIN_OBJS) $(UTIL_OBJS) *~ a.out NVME-VERSION-FILE *.tar* nvme.spec version control nvme-*.deb 70-nvmf-autoconnect.conf + $(RM) $(NVME) $(OBJS) $(PLUGIN_OBJS) $(UTIL_OBJS) $(EVENT_OBJS) *~ a.out NVME-VERSION-FILE *.tar* nvme.spec version control nvme-*.deb 70-nvmf-autoconnect.conf $(MAKE) -C Documentation clean $(RM) tests/*.pyc $(RM) verify-no-dep
common.h +17 -0
@@ -12,4 +12,21 @@ #define __stringify_1(x...) #x #define __stringify(x...) __stringify_1(x) +/** +* container_of - cast a member of a structure out to the containing structure +* +* @ptr: the pointer to the member. +* @type: the type of the container struct this is embedded in. +* @member: the name of the member within the struct. +* +*/ +#define container_of_const (ptr, type, member) ({ \ + typeof( ((const type *)0)->member ) *__mptr = (ptr); \ + (const type *)( (const char *)__mptr - offsetof(type,member) );}) +#define container_of (ptr, type, member) ({ \ + typeof( ((type *)0)->member ) *__mptr = (ptr); \ + (type *)( (char *)__mptr - offsetof(type,member) );}) + +#define STEAL_PTR (p) ({ typeof(p) __tmp = (p); (p) = NULL; __tmp; }) +#endif
event/event.c new file +481 -0
481 lines not on this page · raw message
event/event.h new file +460 -0
460 lines not on this page · raw message
event/timeout.c new file +373 -0
373 lines not on this page · raw message
event/timeout.h new file +110 -0
110 lines not on this page · raw message
event/ts-util.c new file +107 -0
107 lines not on this page · raw message
event/ts-util.h new file +129 -0
129 lines not on this page · raw message
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH v2 02/16] nvme-cli: add code for event and timeout handling
Quoted 8 lines · On Sat, 2021-03-06 at 01:36 +0100, [email protected] wrote:
From: Martin Wilck <[email protected]>
For the nvme monitor functionality, an event handling mechanism will be necessary which deals with event timeouts. While there are standard
solutions for this (e.g. libevent), these add unnecessary complexity and dependencies to nvme-cli.
Quoted 6 lines
Add a small, straighforward event and timeout handling code based on epoll and timerfd.
This code is identical to what I've pushed recently to
https://github.com/mwilck/minivent, where I added a couple of unit tests to make sure the code is as robust as it needs to be.
FTR, I found a use-after-free condition in this code. It's will be fixed in the next iteration of this patch set.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH v2 02/16] nvme-cli: add code for event and timeout handling
Quoted 13 lines · Martin Wilck
Quoted 12 lines · On Sat, 2021-03-06 at 01:36 +0100, [email protected] wrote:
From: Martin Wilck <[email protected]>
For the nvme monitor functionality, an event handling mechanism will be necessary which deals with event timeouts. While there are standard
solutions for this (e.g. libevent), these add unnecessary complexity
and dependencies to nvme-cli.Add a small, straighforward event and timeout handling code based on epoll and timerfd.
Quoted 6 lines
Quoted 3 lines
This code is identical to what I've pushed recently to
https://github.com/mwilck/minivent, where I added a couple of unit tests to make sure the code is as robust as it needs to be.FTR, I found a use-after-free condition in this code. It's will be fixed in the next iteration of this patch set.
The fixed code is in the git repo mentioned above. I suppose that, in case that the general approach gets positive review, it would make sense to pull this in via git subtree or git submodule. Tell me your preferences.
Regards
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH v2 02/16] nvme-cli: add code for event and timeout handling
Quoted 18 lines · Martin Wilck
Quoted 17 lines · Martin Wilck
Quoted 15 lines · On Sat, 2021-03-06 at 01:36 +0100, [email protected] wrote:
From: Martin Wilck <[email protected]>
For the nvme monitor functionality, an event handling mechanism will be necessary which deals with event timeouts. While there are standard
solutions for this (e.g. libevent), these add unnecessary complexity
and dependencies to nvme-cli.Add a small, straighforward event and timeout handling code based on epoll and timerfd.
This code is identical to what I've pushed recently to
https://github.com/mwilck/minivent, where I added a couple of unit tests to make sure the code is as robust as it needs to be.
Quoted 7 lines
Quoted 2 lines
FTR, I found a use-after-free condition in this code. It's will be fixed in the next iteration of this patch set.
The fixed code is in the git repo mentioned above. I suppose that, in case that the general approach gets positive review, it would make sense to pull this in via git subtree or git submodule. Tell me your preferences.
I've resubmitted to https://github.com/linux-nvme/nvme-cli/pull/877 now with the event code incorporated via "git subtree"
from https://github.com/mwilck/minivent. I need to flatten the subtree merge in order to submit this here in patch form, but I'll postpone that until I get some feedback on the v2 series.
Best Regards,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 3/16 monitor: add basic "nvme monitor" functionality
In reply to [email protected]From: Martin Wilck <[email protected]>
"nvme monitor" listens for uevents related to NVMe discovery and attemtps to auto-connect newly discovered controllers. Currently NVMeFC events via fc_udev_device and AEN events from NVMe discovery controllers are supported.
This patch adds the main event listening functionality. Actual event handling will be added in the forthcoming patches.
Options:
-N/--no-connect for "dry run" mode
-S/--silent to suppress LOG_NOTICE -v/--verbose to enable LOG_INFO (overrides -S) -D/--debug to enable LOG_DEBUG (overrides -S, -v)
-t/--timestamps to enable time stamps on log messages.
I tried to use short options that don't conflict with options from nvme connect-all, because many of those options will be added to the monitor later, too.
--- .github/workflows/c-cpp.yml | 4 + Makefile | 10 ++ monitor.c | 246 ++++++++++++++++++++++++++++++++++++ monitor.h | 6 + nvme-builtin.h | 1 + nvme.c | 13 ++ 6 files changed, 280 insertions(+) create mode 100644 monitor.c create mode 100644 monitor.h
.github/workflows/c-cpp.yml +4 -0
@@ -13,6 +13,10 @@ jobs: steps: - uses: actions/checkout@v2 + - name: update + run: sudo apt-get update + - name: dependencies + run: sudo apt-get install --yes libudev-dev - name: make run: sudo apt-get install gcc-10-powerpc* && make clean && make && make clean && make LD=powerpc64le-linux-gnu-ld CC=powerpc64le-linux-gnu-gcc-10 CFLAGS='-O2 -g -Wall -Wformat-security -Werror -m64 -mcpu=power8 -mtune=power8 -I -I/usr/powerpc64-linux-gnu/include/'
Makefile +10 -0
@@ -4,6 +4,7 @@ override CPPFLAGS += -D_GNU_SOURCE -D__CHECK_ENDIAN__ LIBUUID = $(shell $(LD) -o /dev/null -luuid >/dev/null 2>&1; echo $$?) LIBHUGETLBFS = $(shell $(LD) -o /dev/null -lhugetlbfs >/dev/null 2>&1; echo $$?) HAVE_SYSTEMD = $(shell pkg-config --exists libsystemd --atleast-version=242; echo $$?) +HAVE_LIBUDEV = $(shell pkg-config --exists libudev; echo $$?) NVME = nvme INSTALL ?= install DESTDIR =
@@ -32,6 +33,11 @@ endif INC=-Iutil +ifeq ($(HAVE_LIBUDEV),0) + override LDFLAGS += -ludev + override CFLAGS += -DHAVE_LIBUDEV +endif + ifeq ($(HAVE_SYSTEMD),0) override LDFLAGS += -lsystemd override CFLAGS += -DHAVE_SYSTEMD
@@ -62,6 +68,10 @@ OBJS := nvme-print.o nvme-ioctl.o nvme-rpmb.o \ nvme-lightnvm.o fabrics.o nvme-models.o plugin.o \ nvme-status.o nvme-filters.o nvme-topology.o +ifeq ($(HAVE_LIBUDEV),0) + OBJS += monitor.o +endif + UTIL_OBJS := util/argconfig.o util/suffix.o util/json.o util/parser.o util/cleanup.o util/log.o EVENT_OBJS := event/event.o event/timeout.o event/ts-util.o
monitor.c new file +246 -0
246 lines not on this page · raw message
monitor.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _MONITOR_H +#define _MONITOR_H + +extern int aen_monitor (const char *desc,int argc,char **argv); + +#endif
nvme-builtin.h +1 -0
@@ -83,6 +83,7 @@ COMMAND_LIST( ENTRY("dir-send", "Submit a Directive Send command, return results", dir_send) ENTRY("virt-mgmt", "Manage Flexible Resources between Primary and Secondary Controller ", virtual_mgmt) ENTRY("rpmb", "Replay Protection Memory Block commands", rpmb_cmd) +ENTRY ("monitor" ,"Monitor NVMeoF AEN events" , monitor_cmd) );#endif
nvme.c +13 -0
@@ -57,6 +57,7 @@#include "argconfig.h" #include "fabrics.h" +#include "monitor.h" #define CREATE_CMD #include "nvme-builtin.h"
@@ -5561,6 +5562,18 @@ static int disconnect_all_cmd(int argc, char **argv, struct command *command, streturn fabrics_disconnect_all (desc, argc, argv); } +static int monitor_cmd (int argc,char **argv,struct command *command,struct plugin *plugin) +{ +#ifdef HAVE_LIBUDEV +const char *desc ="Monitor NVMeoF AEN events" ; + +return aen_monitor (desc, argc, argv); +#else +fprintf (stderr,"nvme-cli built without libudev doesn't support the \"monitor\" subcommand\n" ); +return EOPNOTSUPP ; +#endif +} +void register_extension (struct plugin *plugin) { plugin->parent = &nvme;
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 4/16 monitor: implement uevent handling
In reply to [email protected]From: Martin Wilck <[email protected]>
This patch implements handling of events received via NVMe-FC (fc_udev_device, detection of FC remote ports with NVMe support) and AEN events from persistent discovery controller connections.
For actual discovery, we fork and basically run a "nvme connect-all" process with for the newly detected discovery controller. The reason for forking discovery tasks is twofold: Firstly, we'd otherwise be forced to make all discovery connections sequentially, which would be slow, as connecting controllers can block on the order of seconds even in successful cases.
Secondly, this allows us to use the some global variables like fabrics_cfg and tracked_ctrls in the discovery code path. Without forking, we'd have to re-write much more code in fabrics.c. In general, the alternative to forking would be creating threads, but the that would require a large rewrite of our code base.
A single-threaded server that forks off the actual discovery makes most sense at this point.
All options known from "nvme connect-all" can be passed to the discovery processes as usual, with the exception of the "peristent" option, which is always enabled: The monitor must try to create persistent discovery connections in order to monitor them.
--- monitor.c | 282 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 277 insertions(+), 5 deletions(-)
monitor.c +277 -5
@@ -17,18 +17,24 @@#include <stddef.h> #include <stdio.h> +#include <stdlib.h> #include <unistd.h> #include <errno.h> #include <libudev.h> #include <signal.h> #include <time.h> #include <syslog.h> +#include <sys/stat.h> +#include <sys/types.h> +#include <sys/wait.h> #include <sys/epoll.h> #include "nvme-status.h" +#include "nvme.h" #include "util/argconfig.h" #include "util/cleanup.h" #include "common.h" +#include "fabrics.h" #include "monitor.h" #define LOG_FUNCNAME 1#include "util/log.h"
@@ -87,12 +93,19 @@ static int create_udev_monitor(struct udev *udev, struct udev_monitor **pmon) }static sig_atomic_t must_exit; +static sig_atomic_t got_sigchld; +static sigset_t orig_sigmask;static void monitor_int_handler (int sig) { must_exit =1 ; } +static void monitor_chld_handler (int sig) +{ + got_sigchld =1 ; +} +static int monitor_init_signals (sigset_t *wait_mask) {sigset_t mask;
@@ -103,26 +116,228 @@ static int monitor_init_signals(sigset_t *wait_mask)* for events. */ sigfillset (&mask); -if (sigprocmask (SIG_BLOCK , &mask,NULL ) ==-1 ) +if (sigprocmask (SIG_BLOCK , &mask, &orig_sigmask) ==-1 )return -errno;if (sigaction (SIGTERM , &sa,NULL ) ==-1 )return -errno;if (sigaction (SIGINT , &sa,NULL ) ==-1 )return -errno; + sa.sa_handler = monitor_chld_handler; +if (sigaction (SIGCHLD , &sa,NULL ) ==-1 ) +return -errno; +/* signal mask to be used in epoll_pwait() */ sigfillset (wait_mask);sigdelset (wait_mask,SIGTERM );sigdelset (wait_mask,SIGINT ); +sigdelset (wait_mask,SIGCHLD );return 0 ; } +static int child_reset_signals (void ) +{ +int err =0 ; +struct sigaction sa = { .sa_handler =SIG_DFL , }; + +if (sigaction (SIGTERM , &sa,NULL ) ==-1 ) + err = errno; +if (sigaction (SIGINT , &sa,NULL ) ==-1 && !err) + err = errno; +if (sigaction (SIGCHLD , &sa,NULL ) ==-1 && !err) + err = errno; + +if (sigprocmask (SIG_SETMASK , &orig_sigmask,NULL ) ==-1 && !err) + err = errno; + +if (err) +msg (LOG_ERR ,"error resetting signal handlers and mask\n" ); +return -err; +} + +static int monitor_get_fc_uev_props (struct udev_device *ud, +char *traddr,size_t tra_sz, +char *host_traddr,size_t htra_sz) +{ +const char *sysname =udev_device_get_sysname (ud); +const char *tra =NULL , *host_tra =NULL ; +bool fc_event_seen = false; +struct udev_list_entry *entry; + + entry =udev_device_get_properties_list_entry (ud); +if (!entry) { +msg (LOG_NOTICE ,"%s: emtpy properties list\n" , sysname); +return -ENOENT ; + } + +for (; entry; entry =udev_list_entry_get_next (entry)) { +const char *name =udev_list_entry_get_name (entry); + +if (!strcmp (name,"FC_EVENT" ) && + !strcmp (udev_list_entry_get_value (entry),"nvmediscovery" )) + fc_event_seen = true; +else if (!strcmp (name,"NVMEFC_HOST_TRADDR" )) + host_tra =udev_list_entry_get_value (entry); +else if (!strcmp (name,"NVMEFC_TRADDR" )) + tra =udev_list_entry_get_value (entry); + } +if (!fc_event_seen) { +msg (LOG_DEBUG ,"%s: FC_EVENT property missing or unsupported\n" , + sysname); +return -EINVAL ; + } +if (!tra || !host_tra) { +msg (LOG_WARNING ,"%s: transport properties missing\n" , sysname); +return -EINVAL ; + } + +if (!memccpy (traddr, tra,'\0' , tra_sz) || + !memccpy (host_traddr, host_tra,'\0' , htra_sz)) { +msg (LOG_ERR ,"traddr (%zu) or host_traddr (%zu) overflow\n" , +strlen (traddr),strlen (host_traddr)); +return -ENAMETOOLONG ; + } + +return 0 ; +} + +static int monitor_discovery (char *transport,char *traddr,char *trsvcid, +char *host_traddr) +{ +char argstr[BUF_SIZE ]; +pid_t pid; +int rc; + + pid =fork (); +if (pid ==-1 ) { +msg (LOG_ERR ,"failed to fork discovery task: %m" ); +return -errno; + }else if (pid >0 ) { +msg (LOG_DEBUG ,"started discovery task %ld\n" , (long )pid); +return 0 ; + } + +child_reset_signals (); +free_dispatcher (mon_dsp); + +msg (LOG_NOTICE ,"starting discovery\n" ); + fabrics_cfg.nqn =NVME_DISC_SUBSYS_NAME ; + fabrics_cfg.transport = transport; + fabrics_cfg.traddr = traddr; + fabrics_cfg.trsvcid = trsvcid; + fabrics_cfg.host_traddr = host_traddr; +/* Without the following, the kernel returns EINVAL */ + fabrics_cfg.tos =-1 ; + fabrics_cfg.persistent = true; + + rc =build_options (argstr,sizeof (argstr), true); +msg (LOG_DEBUG ,"%s\n" , argstr); + rc =do_discover (argstr, mon_cfg.autoconnect,NORMAL ); + +exit (-rc); +/* not reached */ +return rc; +} + +static void monitor_handle_fc_uev (struct udev_device *ud) +{ +const char *action =udev_device_get_action (ud); +const char *sysname =udev_device_get_sysname (ud); +char traddr[NVMF_TRADDR_SIZE ], host_traddr[NVMF_TRADDR_SIZE ]; + +if (strcmp (action,"change" ) ||strcmp (sysname,"fc_udev_device" )) +return ; + +if (monitor_get_fc_uev_props (ud, traddr,sizeof (traddr), + host_traddr,sizeof (host_traddr))) +return ; + +monitor_discovery ("fc" , traddr,NULL , host_traddr); +} + +static int monitor_get_nvme_uev_props (struct udev_device *ud, +char *transport,size_t tr_sz, +char *traddr,size_t tra_sz, +char *trsvcid,size_t trs_sz, +char *host_traddr,size_t htra_sz) +{ +const char *sysname =udev_device_get_sysname (ud); +bool aen_disc = false; +struct udev_list_entry *entry; + + entry =udev_device_get_properties_list_entry (ud); +if (!entry) { +msg (LOG_NOTICE ,"%s: emtpy properties list\n" , sysname); +return -ENOENT ; + } + + *transport = *traddr = *trsvcid = *host_traddr ='\0' ; +for (; entry; entry =udev_list_entry_get_next (entry)) { +const char *name =udev_list_entry_get_name (entry); + +if (!strcmp (name,"NVME_AEN" ) && + !strcmp (udev_list_entry_get_value (entry),"0x70f002" )) + aen_disc = true; +else if (!strcmp (name,"NVME_TRTYPE" )) +memccpy (transport,udev_list_entry_get_value (entry), +'\0' , tr_sz); +else if (!strcmp (name,"NVME_TRADDR" )) +memccpy (traddr,udev_list_entry_get_value (entry), +'\0' , htra_sz); +else if (!strcmp (name,"NVME_TRSVCID" )) +memccpy (trsvcid,udev_list_entry_get_value (entry), +'\0' , trs_sz); +else if (!strcmp (name,"NVME_HOST_TRADDR" )) +memccpy (host_traddr,udev_list_entry_get_value (entry), +'\0' , tra_sz); + } +if (!aen_disc) { +msg (LOG_DEBUG ,"%s: not a \"discovery log changed\" AEN, ignoring event\n" , + sysname); +return -EINVAL ; + } + +if (!*traddr || !*transport) { +msg (LOG_WARNING ,"%s: transport properties missing\n" , sysname); +return -EINVAL ; + } + +return 0 ; +} + +static void monitor_handle_nvme_uev (struct udev_device *ud) +{ +char traddr[NVMF_TRADDR_SIZE ], host_traddr[NVMF_TRADDR_SIZE ]; +char trsvcid[NVMF_TRSVCID_SIZE ], transport[5 ]; + +if (strcmp (udev_device_get_action (ud),"change" )) +return ; + +if (monitor_get_nvme_uev_props (ud, transport,sizeof (transport), + traddr,sizeof (traddr), + trsvcid,sizeof (trsvcid), + host_traddr,sizeof (host_traddr))) +return ; + +monitor_discovery (transport, traddr, +strcmp (trsvcid,"none" ) ? trsvcid :NULL , host_traddr); +} +static void monitor_handle_udevice (struct udev_device *ud) { -msg (LOG_INFO ,"uevent: %s %s\n" , -udev_device_get_action (ud), -udev_device_get_sysname (ud)); +const char *subsys =udev_device_get_subsystem (ud); + +if (log_level >=LOG_INFO ) { +const char *action =udev_device_get_action (ud); +const char *syspath =udev_device_get_syspath (ud); + +msg (LOG_INFO ,"%s %s\n" , action, syspath); + } +if (!strcmp (subsys,"fc" )) +monitor_handle_fc_uev (ud); +else if (!strcmp (subsys,"nvme" )) +monitor_handle_nvme_uev (ud); }struct udev_monitor_event {
@@ -147,6 +362,49 @@ static int monitor_handle_uevents(struct event *ev,return EVENTCB_CONTINUE ; } +static int handle_epoll_err (int errcode) +{ +if (errcode != -EINTR ) +return errcode; +else if (must_exit) { +msg (LOG_NOTICE ,"monitor: exit signal received\n" ); +return ELOOP_QUIT ; + }else if (!got_sigchld) { +msg (LOG_WARNING ,"monitor: unexpected interruption, ignoring\n" ); +return ELOOP_CONTINUE ; + } + + got_sigchld =0 ; +while (true) { +int wstatus; +pid_t pid; + + pid =waitpid (-1 , &wstatus,WNOHANG ); +switch (pid) { +case -1 : +if (errno !=ECHILD ) +msg (LOG_ERR ,"error in waitpid: %m\n" ); + goto out; +case 0 : + goto out; +default : +break ; + } +if (!WIFEXITED (wstatus)) +msg (LOG_WARNING ,"child %ld didn't exit normally\n" , + (long )pid); +else if (WEXITSTATUS (wstatus) !=0 ) +msg (LOG_NOTICE ,"child %ld exited with status \"%s\"\n" , + (long )pid,strerror (WEXITSTATUS (wstatus))); +else +msg (LOG_DEBUG ,"child %ld exited normally\n" , (long )pid); + }; + +out: +/* tell event_loop() to continue */ +return ELOOP_CONTINUE ; +} +static int monitor_parse_opts (const char *desc,int argc,char **argv) {bool quiet = false;
@@ -156,6 +414,19 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)int ret;OPT_ARGS (opts) = {OPT_FLAG ("no-connect" ,'N' , &noauto,"dry run, do not autoconnect to discovered controllers" ), +OPT_LIST ("hostnqn" ,'q' , &fabrics_cfg.hostnqn,"user-defined hostnqn (if default not used)" ), +OPT_LIST ("hostid" ,'I' , &fabrics_cfg.hostid,"user-defined hostid (if default not used)" ), +OPT_INT ("keep-alive-tmo" ,'k' , &fabrics_cfg.keep_alive_tmo,"keep alive timeout period in seconds" ), +OPT_INT ("reconnect-delay" ,'c' , &fabrics_cfg.reconnect_delay,"reconnect timeout period in seconds" ), +OPT_INT ("ctrl-loss-tmo" ,'l' , &fabrics_cfg.ctrl_loss_tmo,"controller loss timeout period in seconds" ), +OPT_INT ("tos" ,'T' , &fabrics_cfg.tos,"type of service" ), +OPT_FLAG ("hdr_digest" ,'g' , &fabrics_cfg.hdr_digest,"enable transport protocol header digest (TCP transport)" ), +OPT_FLAG ("data_digest" ,'G' , &fabrics_cfg.data_digest,"enable transport protocol data digest (TCP transport)" ), +OPT_INT ("nr-io-queues" ,'i' , &fabrics_cfg.nr_io_queues,"number of io queues to use (default is core count)" ), +OPT_INT ("nr-write-queues" ,'W' , &fabrics_cfg.nr_write_queues,"number of write queues to use (default 0)" ), +OPT_INT ("nr-poll-queues" ,'P' , &fabrics_cfg.nr_poll_queues,"number of poll queues to use (default 0)" ), +OPT_INT ("queue-size" ,'Q' , &fabrics_cfg.queue_size,"number of io queue elements to use (default 128)" ), +OPT_FLAG ("matching" ,'m' , &fabrics_cfg.matching_only,"connect only records matching the traddr" ),OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ),OPT_FLAG ("verbose" ,'v' , &verbose,"log level: verbose" ),OPT_FLAG ("debug" ,'D' , &debug,"log level: debug" ),
@@ -163,6 +434,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv) OPT_END() }; + log_pid = true; ret =argconfig_parse (argc, argv, desc, opts);if (ret)return ret;
@@ -238,7 +510,7 @@ int aen_monitor(const char *desc, int argc, char **argv) goto out; } - ret =event_loop (mon_dsp, &wait_mask,NULL ); + ret =event_loop (mon_dsp, &wait_mask, handle_epoll_err); out:free_dispatcher (mon_dsp);
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 5/16 conn-db: add simple connection registry
In reply to [email protected]From: Martin Wilck <[email protected]>
The monitor works best if it maintains a discovery controller connection to every transport address that provides a discovery subsystem.
While controllers are easily tracked in sysfs, addresses ("connections"), i.e. (transport, traddr, trsvid, host_traddr) tuples, are not. Create a simple registry that tracks the state of "connections" and their associated discovery controllers.
A detailed description of the API is provided in the header file conn-db.h.
This patch also adds well-known "list.h" macros from the kernel. The file was taken from multipath-tools, which got it from the kernel long ago.
--- Makefile | 2 +- conn-db.c | 424 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ conn-db.h | 170 ++++++++++++++++++++++ list.h | 349 ++++++++++++++++++++++++++++++++++++++++++++ monitor.c | 49 ++++++- 5 files changed, 987 insertions(+), 7 deletions(-) create mode 100644 conn-db.c create mode 100644 conn-db.h create mode 100644 list.h
Makefile +1 -1
@@ -69,7 +69,7 @@ OBJS := nvme-print.o nvme-ioctl.o nvme-rpmb.o \ nvme-status.o nvme-filters.o nvme-topology.o ifeq ($(HAVE_LIBUDEV),0) - OBJS += monitor.o + OBJS += monitor.o conn-db.o endif UTIL_OBJS := util/argconfig.o util/suffix.o util/json.o util/parser.o util/cleanup.o util/log.o
conn-db.c new file +424 -0
424 lines not on this page · raw message
conn-db.h new file +170 -0
170 lines not on this page · raw message
list.h new file +349 -0
349 lines not on this page · raw message
monitor.c +43 -6
92 lines not on this page · raw message
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 6/16 monitor: monitor_discovery(): try to reuse existing controllers
In reply to [email protected]From: Martin Wilck <[email protected]>
We need to pass cfg.device to do_discovery() if we want to re-use the discovery controller device. Remember the instance number when handling an AEN, and try to reuse this instance if the discovery needs to be repeated.
--- monitor.c | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-)
monitor.c +36 -7
@@ -204,12 +204,14 @@ static int monitor_get_fc_uev_props(struct udev_device *ud, }static int monitor_discovery (const char *transport,const char *traddr, -const char *trsvcid,const char *host_traddr) +const char *trsvcid,const char *host_traddr, +const char *devname) {char argstr[BUF_SIZE ];pid_t pid;int rc, db_rc;struct nvme_connection *co =NULL ; +char *device =NULL ; db_rc =conndb_add (transport, traddr, trsvcid, host_traddr, &co);if (db_rc !=0 && db_rc != -EEXIST )
@@ -230,7 +232,15 @@ static int monitor_discovery(const char *transport, const char *traddr, co->discovery_pending =0 ; co->status =CS_DISC_RUNNING ; co->discovery_task = pid; +if (devname) { +int instance =ctrl_instance (devname); +if (instance <0 ) { +msg (LOG_ERR ,"unexpected devname: %s\n" , + devname); + }else + co->discovery_instance = instance; + }return 0 ; }
@@ -238,12 +248,29 @@ static int monitor_discovery(const char *transport, const char *traddr,free_dispatcher (mon_dsp);conndb_free (); -msg (LOG_NOTICE ,"starting discovery\n" ); +conn_msg (LOG_NOTICE , co,"starting discovery in state %s\n" , +conn_status_str (co->status)); + +/* +* Try to re-use existing controller. do_discovery() will check +* if it matches the connection parameters. +* fabrics_cfg.device must be allocated on the heap! +*/ +if (devname) + device =strdup (devname); +else if (co->discovery_instance >=0 && +asprintf (&device,"nvme%d" , co->discovery_instance) ==-1 ) + device =NULL ; + +if (device) +msg (LOG_INFO ,"using discovery controller %s\n" , device); + fabrics_cfg.nqn =NVME_DISC_SUBSYS_NAME ; fabrics_cfg.transport = transport; fabrics_cfg.traddr = traddr; - fabrics_cfg.trsvcid = trsvcid; - fabrics_cfg.host_traddr = host_traddr; + fabrics_cfg.trsvcid = trsvcid && *trsvcid ? trsvcid :NULL ; + fabrics_cfg.host_traddr = host_traddr && *host_traddr ? host_traddr :NULL ; + fabrics_cfg.device = device;/* Without the following, the kernel returns EINVAL */ fabrics_cfg.tos =-1 ; fabrics_cfg.persistent = true;
@@ -252,6 +279,7 @@ static int monitor_discovery(const char *transport, const char *traddr,msg (LOG_DEBUG ,"%s\n" , argstr); rc =do_discover (argstr, mon_cfg.autoconnect,NORMAL ); +free (device);exit (-rc);/* not reached */ return rc;
@@ -270,7 +298,7 @@ static void monitor_handle_fc_uev(struct udev_device *ud) host_traddr, sizeof(host_traddr)))return ; -monitor_discovery ("fc" , traddr,NULL , host_traddr); +monitor_discovery ("fc" , traddr,NULL , host_traddr,NULL ); } static int monitor_get_nvme_uev_props(struct udev_device *ud,
@@ -338,7 +366,8 @@ static void monitor_handle_nvme_uev(struct udev_device *ud)return ;monitor_discovery (transport, traddr, -strcmp (trsvcid,"none" ) ? trsvcid :NULL , host_traddr); +strcmp (trsvcid,"none" ) ? trsvcid :NULL , host_traddr, +udev_device_get_sysname (ud)); }static void monitor_handle_udevice (struct udev_device *ud)
@@ -433,7 +462,7 @@ static int handle_epoll_err(int errcode)if (co->discovery_pending) {msg (LOG_NOTICE ,"new discovery pending - restarting\n" );monitor_discovery (co->transport, co->traddr, - co->trsvcid, co->host_traddr); + co->trsvcid, co->host_traddr,NULL ); } };
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 7/16 monitor: kill running discovery tasks on exit
In reply to [email protected]From: Martin Wilck <[email protected]>
Make sure running discovery tasks terminate when the main process terminates.
--- monitor.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
monitor.c +30 -0
@@ -471,6 +471,34 @@ out:return ELOOP_CONTINUE ; } +static int monitor_kill_discovery_task (struct nvme_connection *co, +void *arg __attribute__((unused))) +{ +int wstatus; +pid_t pid, wpid =-1 ; + +if (co->status !=CS_DISC_RUNNING ) +return CD_CB_OK ; + + pid = co->discovery_task; + co->status =CS_FAILED ; +if (kill (co->discovery_task,SIGTERM ) ==-1 ) { +msg (LOG_ERR ,"failed to send SIGTERM to pid %ld: %m\n" , + (long )pid); + wpid =waitpid (pid, &wstatus,WNOHANG ); + }else { +msg (LOG_DEBUG ,"sent SIGTERM to pid %ld, waiting\n" , (long )pid); + wpid =waitpid (pid, &wstatus,0 ); + } +if (wpid != pid) { +msg (LOG_ERR ,"failed to wait for %ld: %m\n" , (long )pid); +return CD_CB_ERR ; + }else { +msg (LOG_DEBUG ,"child %ld terminated\n" , (long )pid); +return CD_CB_OK ; + } +} +static int monitor_parse_opts (const char *desc,int argc,char **argv) {bool quiet = false;
@@ -578,6 +606,8 @@ int aen_monitor(const char *desc, int argc, char **argv) ret =event_loop (mon_dsp, &wait_mask, handle_epoll_err); +conndb_for_each (monitor_kill_discovery_task,NULL ); +conndb_free (); out:free_dispatcher (mon_dsp);return nvme_status_to_errno (ret, true);
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 8/16 monitor: add option --cleanup / -C
In reply to [email protected]From: Martin Wilck <[email protected]>
With this option, "nvme monitor" will remove created discovery controllers after it exits. To avoid shutting down connections that were created independently, check existing connections on startup and track them with the discovery_ctrl_existed flag.
--- conn-db.c | 1 + conn-db.h | 1 + monitor.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+)
conn-db.c +1 -0
@@ -382,6 +382,7 @@ int conndb_init_from_sysfs(void) instance =ctrl_instance (devices[i]->d_name);if (instance >=0 ) { ci->c.discovery_instance = instance; + ci->c.discovery_ctrl_existed =1 ;msg (LOG_DEBUG ,"found discovery controller %s\n" , devices[i]->d_name); }
conn-db.h +1 -0
@@ -12,6 +12,7 @@ struct nvme_connection { int discovery_pending:1; int did_discovery:1; int successful_discovery:1; + int discovery_ctrl_existed:1;union {pid_t discovery_task;int discovery_result;
monitor.c +42 -0
80 lines not on this page · raw message
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 9/16 monitor: handling of add/remove uevents for nvme controllers
In reply to [email protected]From: Martin Wilck <[email protected]>
On add, we'd like to detect the creation of a discovery controller for a given connection. More often than not, that doesn't work, because the controller is still in "connecting" state when the uevent arrives, and reading subsysnqn yields "(efault)".
This would need to be fixed by waiting for the connection to complete, but doing this properly is currently out of scope. It's not really a big problem, because without --persistent, the discovery controller will be removed anyway, and with --persistent, we'll notice the discovery controller as soon as it actually receives an AEN and generates an uevent.
--- monitor.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
monitor.c +38 -0
@@ -159,6 +159,36 @@ static int child_reset_signals(void)return -err; } +static void monitor_handle_nvme_add (struct udev_device *ud) +{ +const char *syspath =udev_device_get_syspath (ud); + char *subsysnqn __cleanup__(cleanup_charp) = NULL; + char *state __cleanup__(cleanup_charp) = NULL; + +if (!syspath) +return ; + subsysnqn =nvme_get_ctrl_attr (syspath,"subsysnqn" ); + state =nvme_get_ctrl_attr (syspath,"state" ); +msg (LOG_DEBUG ,"add %s => %s [%s]\n" , syspath, subsysnqn, state); +} + +static void monitor_handle_nvme_remove (struct udev_device *ud) +{ +const char *sysname =udev_device_get_sysname (ud); +struct nvme_connection *co; + +if (!sysname) +return ; + + co =conndb_find_by_ctrl (sysname); +if (co) { +msg (LOG_DEBUG ,"%s: connection discovery controller removed\n" , + sysname); + co->discovery_instance =-1 ; + } +return ; +} +static int monitor_get_fc_uev_props (struct udev_device *ud,char *traddr,size_t tra_sz,char *host_traddr,size_t htra_sz)
@@ -358,6 +388,14 @@ static void monitor_handle_nvme_uev(struct udev_device *ud)char traddr[NVMF_TRADDR_SIZE ], host_traddr[NVMF_TRADDR_SIZE ];char trsvcid[NVMF_TRSVCID_SIZE ], transport[5 ]; +if (!strcmp (udev_device_get_action (ud),"remove" )) { +monitor_handle_nvme_remove (ud); +return ; + } +if (!strcmp (udev_device_get_action (ud),"add" )) { +monitor_handle_nvme_add (ud); +return ; + }if (strcmp (udev_device_get_action (ud),"change" ))return ;
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 10/16 monitor: discover from conf file on startup
In reply to [email protected]From: Martin Wilck <[email protected]>
Implement discovery from /etc/nvme/discovery.conf on startup of the monitor.
The monitor needs to call discover_from_conf_file() in order to be able to do discovery at startup. discover_from_conf_file() takes the argconfig_commandline_options argument from fabrics_discover(). By moving these options into a static variable, we can avoid passing them as function argument. This makes it possible to use discover_from_conf_file() from the monitor without making the options for fabric_discover() globally visible.
--- fabrics.c | 67 +++++++++++++++++++++++++++---------------------------- fabrics.h | 5 +++++ monitor.c | 58 +++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 94 insertions(+), 36 deletions(-)
fabrics.c +33 -34
@@ -85,7 +85,6 @@ struct connect_args {struct connect_args *tracked_ctrls;#define PATH_NVME_FABRICS "/dev/nvme-fabrics" -#define PATH_NVMF_DISC "/etc/nvme/discovery.conf"#define PATH_NVMF_HOSTNQN "/etc/nvme/hostnqn"#define PATH_NVMF_HOSTID "/etc/nvme/hostid"#define MAX_DISC_ARGS 10
@@ -1439,8 +1438,9 @@ int do_discover(char *argstr, bool connect, enum nvme_print_flags flags)return ret; } -static int discover_from_conf_file (const char *desc,char *argstr, -const struct argconfig_commandline_options *opts,bool connect) +static OPT_ARGS (discover_opts); + +int discover_from_conf_file (const char *desc,char *argstr,bool connect) {FILE *f;char line[256 ], *ptr, *all_args, *args, **argv;
@@ -1480,7 +1480,7 @@ static int discover_from_conf_file(const char *desc, char *argstr,while ((ptr =strsep (&args," =\n" )) !=NULL ) argv[argc++] = ptr; - err =argconfig_parse (argc, argv, desc, opts); + err =argconfig_parse (argc, argv, desc, discover_opts);if (err) goto free_and_continue;
@@ -1523,41 +1523,40 @@ out:return ret; } +static OPT_ARGS (discover_opts) = { +OPT_LIST ("transport" ,'t' , &fabrics_cfg.transport,"transport type" ), +OPT_LIST ("traddr" ,'a' , &fabrics_cfg.traddr,"transport address" ), +OPT_LIST ("trsvcid" ,'s' , &fabrics_cfg.trsvcid,"transport service id (e.g. IP port)" ), +OPT_LIST ("host-traddr" ,'w' , &fabrics_cfg.host_traddr,"host traddr (e.g. FC WWN's)" ), +OPT_LIST ("hostnqn" ,'q' , &fabrics_cfg.hostnqn,"user-defined hostnqn (if default not used)" ), +OPT_LIST ("hostid" ,'I' , &fabrics_cfg.hostid,"user-defined hostid (if default not used)" ), +OPT_LIST ("raw" ,'r' , &fabrics_cfg.raw,"raw output file" ), +OPT_LIST ("device" ,'d' , &fabrics_cfg.device,"use existing discovery controller device" ), +OPT_INT ("keep-alive-tmo" ,'k' , &fabrics_cfg.keep_alive_tmo,"keep alive timeout period in seconds" ), +OPT_INT ("reconnect-delay" ,'c' , &fabrics_cfg.reconnect_delay,"reconnect timeout period in seconds" ), +OPT_INT ("ctrl-loss-tmo" ,'l' , &fabrics_cfg.ctrl_loss_tmo,"controller loss timeout period in seconds" ), +OPT_INT ("tos" ,'T' , &fabrics_cfg.tos,"type of service" ), +OPT_FLAG ("hdr_digest" ,'g' , &fabrics_cfg.hdr_digest,"enable transport protocol header digest (TCP transport)" ), +OPT_FLAG ("data_digest" ,'G' , &fabrics_cfg.data_digest,"enable transport protocol data digest (TCP transport)" ), +OPT_INT ("nr-io-queues" ,'i' , &fabrics_cfg.nr_io_queues,"number of io queues to use (default is core count)" ), +OPT_INT ("nr-write-queues" ,'W' , &fabrics_cfg.nr_write_queues,"number of write queues to use (default 0)" ), +OPT_INT ("nr-poll-queues" ,'P' , &fabrics_cfg.nr_poll_queues,"number of poll queues to use (default 0)" ), +OPT_INT ("queue-size" ,'Q' , &fabrics_cfg.queue_size,"number of io queue elements to use (default 128)" ), +OPT_FLAG ("persistent" ,'p' , &fabrics_cfg.persistent,"persistent discovery connection" ), +OPT_FLAG ("quiet" ,'S' , &fabrics_cfg.quiet,"suppress already connected errors" ), +OPT_FLAG ("matching" ,'m' , &fabrics_cfg.matching_only,"connect only records matching the traddr" ), +OPT_FMT ("output-format" ,'o' , &fabrics_cfg.output_format,"Output format: normal|json|binary" ), +OPT_END () +}; +int fabrics_discover (const char *desc,int argc,char **argv,bool connect) {char argstr[BUF_SIZE ];int ret;enum nvme_print_flags flags; -bool quiet = false; - -OPT_ARGS (opts) = { -OPT_LIST ("transport" ,'t' , &fabrics_cfg.transport,"transport type" ), -OPT_LIST ("traddr" ,'a' , &fabrics_cfg.traddr,"transport address" ), -OPT_LIST ("trsvcid" ,'s' , &fabrics_cfg.trsvcid,"transport service id (e.g. IP port)" ), -OPT_LIST ("host-traddr" ,'w' , &fabrics_cfg.host_traddr,"host traddr (e.g. FC WWN's)" ), -OPT_LIST ("hostnqn" ,'q' , &fabrics_cfg.hostnqn,"user-defined hostnqn (if default not used)" ), -OPT_LIST ("hostid" ,'I' , &fabrics_cfg.hostid,"user-defined hostid (if default not used)" ), -OPT_LIST ("raw" ,'r' , &fabrics_cfg.raw,"raw output file" ), -OPT_LIST ("device" ,'d' , &fabrics_cfg.device,"existing discovery controller device" ), -OPT_INT ("keep-alive-tmo" ,'k' , &fabrics_cfg.keep_alive_tmo,"keep alive timeout period in seconds" ), -OPT_INT ("reconnect-delay" ,'c' , &fabrics_cfg.reconnect_delay,"reconnect timeout period in seconds" ), -OPT_INT ("ctrl-loss-tmo" ,'l' , &fabrics_cfg.ctrl_loss_tmo,"controller loss timeout period in seconds" ), -OPT_INT ("tos" ,'T' , &fabrics_cfg.tos,"type of service" ), -OPT_FLAG ("hdr_digest" ,'g' , &fabrics_cfg.hdr_digest,"enable transport protocol header digest (TCP transport)" ), -OPT_FLAG ("data_digest" ,'G' , &fabrics_cfg.data_digest,"enable transport protocol data digest (TCP transport)" ), -OPT_INT ("nr-io-queues" ,'i' , &fabrics_cfg.nr_io_queues,"number of io queues to use (default is core count)" ), -OPT_INT ("nr-write-queues" ,'W' , &fabrics_cfg.nr_write_queues,"number of write queues to use (default 0)" ), -OPT_INT ("nr-poll-queues" ,'P' , &fabrics_cfg.nr_poll_queues,"number of poll queues to use (default 0)" ), -OPT_INT ("queue-size" ,'Q' , &fabrics_cfg.queue_size,"number of io queue elements to use (default 128)" ), -OPT_FLAG ("persistent" ,'p' , &fabrics_cfg.persistent,"persistent discovery connection" ), -OPT_FLAG ("quiet" ,'S' , &quiet,"suppress already connected errors" ), -OPT_FLAG ("matching" ,'m' , &fabrics_cfg.matching_only,"connect only records matching the traddr" ), -OPT_FMT ("output-format" ,'o' , &fabrics_cfg.output_format, output_format), -OPT_END () - }; fabrics_cfg.tos =-1 ; - ret =argconfig_parse (argc, argv, desc, opts); + ret =argconfig_parse (argc, argv, desc, discover_opts);if (ret) goto out;
@@ -1572,7 +1571,7 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect) } } -if (quiet) +if (fabrics_cfg.quiet) log_level =LOG_WARNING ;if (fabrics_cfg.device && !strcmp (fabrics_cfg.device,"none" ))
@@ -1581,7 +1580,7 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect) fabrics_cfg.nqn =NVME_DISC_SUBSYS_NAME ;if (!fabrics_cfg.transport && !fabrics_cfg.traddr) { - ret =discover_from_conf_file (desc, argstr, opts, connect); + ret =discover_from_conf_file (desc, argstr, connect); }else {set_discovery_kato (&fabrics_cfg);
fabrics.h +5 -0
@@ -38,6 +38,7 @@ struct fabrics_config {int data_digest;bool persistent;bool matching_only; +bool quiet;const char *output_format; };extern struct fabrics_config fabrics_cfg;
@@ -45,11 +46,15 @@ extern struct fabrics_config fabrics_cfg;extern const char *const trtypes[];#define BUF_SIZE 4096 +#define PATH_NVMF_CFG_DIR "/etc/nvme" +#define FILE_NVMF_DISC "discovery.conf" +#define PATH_NVMF_DISC PATH_NVMF_CFG_DIR "/" FILE_NVMF_DISCint build_options (char *argstr,int max_len,bool discover);int do_discover (char *argstr,bool connect,enum nvme_print_flags flags);int ctrl_instance (const char *device);char *parse_conn_arg (const char *conargs,const char delim,const char *field);int remove_ctrl (int instance); +int discover_from_conf_file (const char *desc,char *argstr,bool connect);#endif
monitor.c +56 -2
86 lines not on this page · raw message
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 11/16 monitor: watch discovery.conf with inotify
In reply to [email protected]From: Martin Wilck <[email protected]>
Watch discovery.conf, and re-start discovery from conf file whenever it's created or changed.
--- monitor.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++ util/cleanup.c | 2 + util/cleanup.h | 1 + 3 files changed, 117 insertions(+)
monitor.c +114 -0
@@ -20,6 +20,7 @@#include <stdlib.h> #include <unistd.h> #include <errno.h> +#include <inttypes.h> #include <libudev.h> #include <signal.h> #include <time.h>
@@ -28,6 +29,7 @@#include <sys/types.h> #include <sys/wait.h> #include <sys/epoll.h> +#include <sys/inotify.h> #include "nvme-status.h" #include "nvme.h"
@@ -448,6 +450,14 @@ static int monitor_handle_uevents(struct event *ev,return EVENTCB_CONTINUE ; } +static struct { +bool running; +bool pending; +pid_t pid; +} discovery_conf_task; + +static int monitor_discover_from_conf_file (void ); +static int handle_epoll_err (int errcode) {if (errcode != -EINTR )
@@ -488,6 +498,15 @@ static int handle_epoll_err(int errcode) (long)pid, strerror(WEXITSTATUS(wstatus))); else msg(LOG_DEBUG, "child %ld exited normally\n", (long)pid); +if (discovery_conf_task.running && + discovery_conf_task.pid == pid) { + discovery_conf_task.running = false; +if (discovery_conf_task.pending) { +msg (LOG_NOTICE , +"discovery from conf file pending - restarting\n" ); +monitor_discover_from_conf_file (); + } + }continue ; }
@@ -586,6 +605,13 @@ static int monitor_discover_from_conf_file(void)pid_t pid;int rc; +if (discovery_conf_task.running) { +msg (LOG_NOTICE ,"discovery from conf file already running (%ld)\n" , + (long )discovery_conf_task.pid); + discovery_conf_task.pending = true; +return 0 ; + } + pid =fork ();if (pid ==-1 ) {msg (LOG_ERR ,"failed to fork discovery task: %m" );
@@ -593,6 +619,9 @@ static int monitor_discover_from_conf_file(void) }else if (pid >0 ) {msg (LOG_DEBUG ,"started discovery task %ld from conf file\n" , (long )pid); + discovery_conf_task.pending = false; + discovery_conf_task.running = true; + discovery_conf_task.pid = pid;return 0 ; }
@@ -619,6 +648,89 @@ static int discovery_from_conf_file_cb(struct event *ev __attribute__((unused)),return EVENTCB_CLEANUP ; } +static void handle_inotify_event (struct inotify_event *iev) +{ +if ((iev->mask & (IN_CLOSE_WRITE |IN_MOVED_TO )) ==0 ) { +msg (LOG_DEBUG ,"ignoring event mask 0x%" PRIx32"\n" , iev->mask); +return ; + } + +if (!iev->name ||strcmp (iev->name,FILE_NVMF_DISC )) { +msg (LOG_DEBUG ,"ignoring event mask 0x%" PRIx32" for %s\n" , + iev->mask, iev->name ? iev->name :"(null)" ); +return ; + } + +msg (LOG_INFO ,"discovery.conf changed, re-reading\n" ); +monitor_discover_from_conf_file (); +} + +static int inotify_cb (struct event *ev,unsigned int ep_events) +{ +char buf[sizeof (struct inotify_event ) +NAME_MAX +1 ]; +int rc; + +if (ev->reason !=REASON_EVENT_OCCURED || (ep_events &EPOLLIN ) ==0 ) +return EVENTCB_CONTINUE ; + +while (true) { +struct inotify_event *iev; + + rc =read (ev->fd, buf,sizeof (buf)); +if (rc ==-1 ) { +if (errno !=EAGAIN ) +msg (LOG_ERR ,"error reading from inotify fd: %m\n" ); +return EVENTCB_CONTINUE ; + } + + iev = (struct inotify_event *)buf; +if (iev->mask & (IN_DELETE_SELF |IN_MOVE_SELF )) { +if (inotify_rm_watch (ev->fd, iev->wd) ==-1 ) +msg (LOG_ERR ,"failed to remove watch %d: %m\n" , + iev->wd); +msg (LOG_WARNING ,"inotify watch %d removed\n" , iev->wd); +return EVENTCB_CLEANUP ; + } +handle_inotify_event (iev); + } +return EVENTCB_CONTINUE ; +} + +static DEFINE_CLEANUP_FUNC(cleanup_event, struct event *, free); + +static void add_inotify_event (struct dispatcher *dsp) +{ + struct event *inotify_event __cleanup__(cleanup_event) = NULL; + int fd __cleanup__(cleanup_fd) = -1; +int rc; + + inotify_event =calloc (1 ,sizeof *inotify_event); +if (!inotify_event) +return ; + + fd =inotify_init1 (IN_NONBLOCK |IN_CLOEXEC ); +if (fd ==-1 ) { +msg (LOG_ERR ,"failed to init inotify: %m\n" ); +return ; + } + + *inotify_event =EVENT_ON_HEAP (inotify_cb, fd,EPOLLIN ); + rc =inotify_add_watch (inotify_event->fd,PATH_NVMF_CFG_DIR , +IN_CLOSE_WRITE |IN_MOVED_TO | +IN_DELETE_SELF |IN_MOVE_SELF ); +if (rc ==-1 ) +msg (LOG_ERR ,"failed to add inotify watch for %s: %m\n" , +PATH_NVMF_CFG_DIR ); + +if ((rc =event_add (dsp, inotify_event)) <0 ) { +msg (LOG_ERR ,"failed to add inotify event: %s\n" , +strerror (-rc)); +return ; + } + fd =-1 ; + inotify_event =NULL ; +} +static int monitor_parse_opts (const char *desc,int argc,char **argv) {bool quiet = false;
@@ -735,7 +847,9 @@ int aen_monitor(const char *desc, int argc, char **argv) goto out; } +add_inotify_event (mon_dsp);conndb_init_from_sysfs (); + ret =event_loop (mon_dsp, &wait_mask, handle_epoll_err);conndb_for_each (monitor_kill_discovery_task,NULL );
util/cleanup.c +2 -0
@@ -1,4 +1,6 @@#include <stdlib.h> +#include <unistd.h> #include "cleanup.h" DEFINE_CLEANUP_FUNC(cleanup_charp, char *, free); +DEFINE_CLEANUP_FUNC (cleanup_fd, int, close);
util/cleanup.h +1 -0
@@ -14,5 +14,6 @@ DECLARE_CLEANUP_FUNC(name, type) \ } DECLARE_CLEANUP_FUNC(cleanup_charp, char *); +DECLARE_CLEANUP_FUNC (cleanup_fd, int);#endif
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 12/16 monitor: add parent/child messaging and "notify" message exchange
In reply to [email protected]From: Martin Wilck <[email protected]>
Persistent discovery controllers are set up in forked children, possibly using recursion via referrals in do_discover(). The simplest way to keep the parent's connection registry up to date is to communicate freshly created controllers directly from child to parent.
To make this work, a callback function is passed to do_discover(), which (if non-null) will be called after setting up a discovery controller to initiate the message exchange. The callback is then passed down to connect_ctrl() for recursive discoveries (referrals).
--- fabrics.c | 29 ++-- fabrics.h | 9 +- monitor.c | 405 +++++++++++++++++++++++++++++++++++++++++++++++++++++- monitor.h | 6 + 4 files changed, 435 insertions(+), 14 deletions(-)
fabrics.c +19 -10
@@ -50,6 +50,7 @@#include "common.h" #include "util/log.h" #include "util/cleanup.h" +#include "monitor.h" #ifdef HAVE_SYSTEMD #include <systemd/sd-id128.h>
@@ -1068,7 +1069,8 @@ free_addrinfo:return ret; } -static int connect_ctrl (struct nvmf_disc_rsp_page_entry *e) +static int connect_ctrl (struct nvmf_disc_rsp_page_entry *e, +const struct monitor_callbacks *monitor) {char argstr[BUF_SIZE ], *p;const char *transport;
@@ -1254,7 +1256,7 @@ retry: flags =validate_output_format (fabrics_cfg.output_format);if (flags <0 ) flags =NORMAL ; - ret =do_discover (argstr, true, flags); + ret =do_discover (argstr, true, flags, monitor); }else ret =add_ctrl (argstr);if (ret == -EINVAL && e->treq &NVMF_TREQ_DISABLE_SQFLOW ) {
@@ -1305,7 +1307,8 @@ static bool should_connect(struct nvmf_disc_rsp_page_entry *entry)return !strncmp (fabrics_cfg.traddr, entry->traddr, len); } -static int connect_ctrls (struct nvmf_disc_rsp_page_hdr *log,int numrec) +static int connect_ctrls (struct nvmf_disc_rsp_page_hdr *log,int numrec, +const struct monitor_callbacks *monitor) {int i;int instance;
@@ -1315,7 +1318,7 @@ static int connect_ctrls(struct nvmf_disc_rsp_page_hdr *log, int numrec)if (!should_connect (&log->entries[i]))continue ; - instance =connect_ctrl (&log->entries[i]); + instance =connect_ctrl (&log->entries[i], monitor);/* clean success */ if (instance >=0 )
@@ -1355,7 +1358,8 @@ static void nvmf_get_host_identifiers(int ctrl_instance) static DEFINE_CLEANUP_FUNC(cleanup_log, struct nvmf_disc_rsp_page_hdr *, free); -int do_discover (char *argstr,bool connect,enum nvme_print_flags flags) +int do_discover (char *argstr,bool connect,enum nvme_print_flags flags, +const struct monitor_callbacks *monitor) { struct nvmf_disc_rsp_page_hdr *log __cleanup__(cleanup_log) = NULL;char *dev_name;
@@ -1384,6 +1388,9 @@ int do_discover(char *argstr, bool connect, enum nvme_print_flags flags) }if (instance <0 )return instance; +else if (monitor && monitor->notify && + (fabrics_cfg.device || fabrics_cfg.persistent)) + monitor->notify (argstr, instance);if (asprintf (&dev_name,"/dev/nvme%d" , instance) <0 )return -errno;
@@ -1391,6 +1398,7 @@ int do_discover(char *argstr, bool connect, enum nvme_print_flags flags)free (dev_name);if (fabrics_cfg.persistent)msg (LOG_NOTICE ,"Persistent device: nvme%d\n" , instance); +if (!fabrics_cfg.device && !fabrics_cfg.persistent) { err =remove_ctrl (instance);if (err)
@@ -1400,7 +1408,7 @@ int do_discover(char *argstr, bool connect, enum nvme_print_flags flags)switch (ret) {case DISC_OK :if (connect) - ret =connect_ctrls (log, numrec); + ret =connect_ctrls (log, numrec, monitor);else if (fabrics_cfg.raw || flags ==BINARY )save_discovery_log (log, numrec);else if (flags ==JSON )
@@ -1440,7 +1448,8 @@ int do_discover(char *argstr, bool connect, enum nvme_print_flags flags)static OPT_ARGS (discover_opts); -int discover_from_conf_file (const char *desc,char *argstr,bool connect) +int discover_from_conf_file (const char *desc,char *argstr,bool connect, +const struct monitor_callbacks *monitor) {FILE *f;char line[256 ], *ptr, *all_args, *args, **argv;
@@ -1507,7 +1516,7 @@ int discover_from_conf_file(const char *desc, char *argstr, bool connect) goto free_and_continue; } - err =do_discover (argstr, connect, flags); + err =do_discover (argstr, connect, flags, monitor);if (err) ret = err;
@@ -1580,7 +1589,7 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect) fabrics_cfg.nqn =NVME_DISC_SUBSYS_NAME ;if (!fabrics_cfg.transport && !fabrics_cfg.traddr) { - ret =discover_from_conf_file (desc, argstr, connect); + ret =discover_from_conf_file (desc, argstr, connect,NULL ); }else {set_discovery_kato (&fabrics_cfg);
@@ -1597,7 +1606,7 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect)if (ret) goto out; - ret =do_discover (argstr, connect, flags); + ret =do_discover (argstr, connect, flags,NULL ); } out:
fabrics.h +7 -2
@@ -50,11 +50,16 @@ extern const char *const trtypes[];#define FILE_NVMF_DISC "discovery.conf"#define PATH_NVMF_DISC PATH_NVMF_CFG_DIR "/" FILE_NVMF_DISC +struct monitor_callbacks ; +int build_options (char *argstr,int max_len,bool discover); -int do_discover (char *argstr,bool connect,enum nvme_print_flags flags); +int do_discover (char *argstr,bool connect,enum nvme_print_flags flags, +const struct monitor_callbacks *);int ctrl_instance (const char *device);char *parse_conn_arg (const char *conargs,const char delim,const char *field);int remove_ctrl (int instance); -int discover_from_conf_file (const char *desc,char *argstr,bool connect); +int discover_from_conf_file (const char *desc,char *argstr,bool connect, +const struct monitor_callbacks *); +#endif
monitor.c +403 -2
447 lines not on this page · raw message
monitor.h +6 -0
@@ -1,6 +1,12 @@#ifndef _MONITOR_H#define _MONITOR_H +typedef void (*disc_notify_cb )(const char *argstr,int instance); + +struct monitor_callbacks { +disc_notify_cb notify; +}; +extern int aen_monitor (const char *desc,int argc,char **argv);#endif
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 13/16 monitor: add "query device" message exchange
In reply to [email protected]From: Martin Wilck <[email protected]>
This message exchange sends a message with the current connection parameters to the parent monitor process and attempts to obtain the instance number of an existing discovery controller for the given connection.
For making this work, another callback is added to monitor_callbacks() do be called before attempting a discovery connection.
--- fabrics.c | 4 ++ monitor.c | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++++- monitor.h | 2 + 3 files changed, 114 insertions(+), 1 deletion(-)
fabrics.c +4 -0
@@ -1376,6 +1376,10 @@ int do_discover(char *argstr, bool connect, enum nvme_print_flags flags,free (fabrics_cfg.device); fabrics_cfg.device =NULL ; } + +if (!fabrics_cfg.device && monitor && monitor->query_dev) + monitor->query_dev (argstr, &fabrics_cfg.device); +if (!fabrics_cfg.device) fabrics_cfg.device =find_ctrl_with_connectargs (cargs);free_connect_args (cargs);
monitor.c +108 -1
156 lines not on this page · raw message
monitor.h +2 -0
@@ -2,9 +2,11 @@#define _MONITOR_Htypedef void (*disc_notify_cb )(const char *argstr,int instance); +typedef void (*disc_query_dev_cb )(const char *argstr,char **device);struct monitor_callbacks {disc_notify_cb notify; +disc_query_dev_cb query_dev; };extern int aen_monitor (const char *desc,int argc,char **argv);
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 14/16 completions: add completions for nvme monitor
In reply to [email protected]From: Martin Wilck <[email protected]>
Add some completions for the monitor command.
--- completions/bash-nvme-completion.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
completions/bash-nvme-completion.sh +5 -1
@@ -12,7 +12,7 @@ _cmds="list id-ctrl id-ns list-ns id-iocs nvm-id-ctrl create-ns delete-ns \ write-uncor copy reset subsystem-reset show-regs discover \ connect-all connect disconnect version help \ intel lnvm memblaze list-subsys endurance-event-agg-log \ - lba-status-log" + lba-status-log monitor" nvme_list_opts () { local opts=""
@@ -242,6 +242,10 @@ nvme_list_opts () { "disconnect") opts+=" --nqn -n --device -d" ;; + "monitor") + opts+=" --no-connect -N --cleanup -C \ + --hostnqn= -q --verbose -v --timestamps -t" + ;; "version") opts+="" ;;
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 15/16 nvmf-autoconnect: add unit file for nvme-monitor.service
In reply to [email protected]From: Martin Wilck <[email protected]>
Running both this service and the udev-based nvmf autoconnect service is discouraged. A "Conflicts" directive against [email protected] wouldn't work, because it would cause the monitor to be stopped as soon as an event arrives. Therefore, mask the udev rules that would start the parallel discovery.
--- nvmf-autoconnect/systemd/nvme-monitor.service | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service
nvmf-autoconnect/systemd/nvme-monitor.service new file +18 -0
@@ -0,0 +1,18 @@ +[Unit] +Description=NVMe Event Monitor for Automatical Subsystem Connection +Documentation=man:nvme-monitor(1) +DefaultDependencies=false +RequiresMountsFor=/sys /run /dev +Conflicts=shutdown.target +After=systemd-udevd-kernel.socket +Before=sysinit.target systemd-udev-trigger.service nvmefc-boot-connections.service + +[Service] +Type=simple +ExecStartPre=-/usr/bin/ln -sf /dev/null /run/udev/rules.d/70-nvmf-autoconnect.rules +ExecStart=/usr/sbin/nvme monitor +ExecStartPost=-/usr/bin/rm -f /run/udev/rules.d/70-nvmf-autoconnect.rules +KillMode=mixed + +[Install] +WantedBy=sysinit.target
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v2 16/16 nvme-monitor(1): add man page for nvme-monitor
In reply to [email protected]From: Martin Wilck <[email protected]>
Add documentation for the nvme-monitor command.
--- Documentation/cmds-main.txt | 4 + Documentation/nvme-monitor.1 | 180 ++++++ Documentation/nvme-monitor.html | 1018 +++++++++++++++++++++++++++++++ Documentation/nvme-monitor.txt | 144 +++++ 4 files changed, 1346 insertions(+) create mode 100644 Documentation/nvme-monitor.1 create mode 100644 Documentation/nvme-monitor.html create mode 100644 Documentation/nvme-monitor.txt
Documentation/cmds-main.txt +4 -0
@@ -168,3 +168,7 @@ linknvme:nvme-disconnect-all[1]:: linknvme:nvme-get-property[1]:: Reads and shows NVMe-over-Fabrics controller property + +linknvme:nvme-monitor[1]:: + Monitor Discovery events and Discover and Connect automatically +
Documentation/nvme-monitor.1 new file +180 -0
180 lines not on this page · raw message
Documentation/nvme-monitor.html new file +1018 -0
1,018 lines not on this page · raw message
Documentation/nvme-monitor.txt new file +144 -0
144 lines not on this page · raw message
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme