RFC: add "nvme monitor" subcommand
Subsystems (3)
cover RFC: add "nvme monitor" subcommand
From: Martin Wilck <[email protected]>
(Cover letter copied from https://github.com/linux-nvme/nvme-cli/pull/877)
This patch set adds a new subcommand **nvme monitor**. In this mode, **nvme-cli** runs continuously, monitors events (currently, uevents) relevant for discovery, and optionally autoconnects to newly discovered subsystems.
The monitor mode is suitable to be run in a systemd service. An appropriate unit file is provided. As such, **nvme monitor** can be used as an alternative to the current auto-connection mechanism based on udev rules and systemd template units. If --autoconnect is active, **nvme monitor** masks the respective udev rules in order to prevent simultaneous connection attempts from udev and itself.
This method for discovery and autodetection has some advantages over the current udev-rule based approach:
* By using the --persistent option, users can easily control whether persistent discovery controllers for discovered transport addresses should be created and monitored for AEN events. **nvme monitor** watches known transport addresses, creates discovery controllers as required, and re-uses existing ones if possible. It keeps track of persistent discovery controllers it created, and tears them down on exit. When run in --persistent --autoconnect mode *in the initial ramfs*, it will keep discovery controllers alive, so that a new instance started after switching root can simply re-use them.
* In certain situations, the systemd-based approach may miss events due to race conditions. This can happen e.g. if an FC remote port is detected, but shortly after it's detection an FC relogin procedure is necessary e.g. due to an RSCN. In this case, an fc_udev_device uevent will be received on the first detection and handled by an nvme connect-all command run from [email protected]. The connection attempt to the rport in question will fail with "no such device" because of the simultaneous FC relogin. [email protected] may not terminate immediately, because it attempts to establish other connections listed in the Discovery Log page it retrieved. When the FC relogin eventually finishes, a new uevent will be received, and nvmf-connect@ will be started again, but *this has no effect* if the previous nvmf-connect@ service hasn't finished yet. This is the general semantics of systemd services, no easy workaround exists. **nvme monitor** doesn't suffer from this problem. If it sees an uevent for a transport address for which a discovery is already running, it will queue the handling of this event up and restart the discovery after it's finished.
* Resource consumption for handling uevents is lower. Instead of running an udev worker, executing the rules, executing systemctl start from the worker, starting a systemd service, and starting a separate **nvme-cli** instance, only a single fork() operation is necessary. Of course, on the back side, the monitor itself consumes resources while it's running and waiting for events. On my system with 8 persistent discovery controllers, its RSS is ~3MB. CPU consumption is zero as long as no events occur.
* **nvme monitor** could be easily extended to handle events for non-FC transports.
I've tested fc_udev_device handling for NVMeoFC with an Ontap target, and AEN handling for RDMA using a Linux **nvmet** target.
### Implementation notes
I've tried to change the exisiting **nvme-cli** code as little as possible while reusing the code from fabrics.c. The majority of changes in the existing code exports formerly static functions and variables, so that they are usable from the monitor code.
The main process just waits for events using epoll(). When an event is received that necessitates a new discovery, a child is forked. This makes it possible to fill in the configuration parameters for do_discover() without interfering with the main process or other discovery tasks running in parallel. The program tracks *transport addresses* (called "connections" in the code) rather than NVMe controllers. In --persistent mode, it tries to maintain exactly one persistent discovery connection per transport address.
Using epoll() may look over-engineered at this stage. I hope the better flexibility over poll() (in particular, the ability to add new event sources while waiting) will simplify future extensions and improvements.
### Todo
* Referrals are not handled perfectly yet. They will be handled by do_discover() just as it would when called from **nvme connect-all**, but it would be better to pass referrals back to the main process to make it aware of the additional discovery controller rather than using recursion. The main process would e.g. know if a discovery is already running for the transport address in the referrral.
* When "add" uevents for nvme controller devices are received, the controller is consistently not in live state yet, and attempting to read the subsysnqn sysfs attribute returns (efault). While this should arguably be fixed in the kernel, it could be worked around in user space by using timers or polling the state sysfs attribute for changes.
* Parse and handle discovery.conf on startup.
* Implement support for RDMA and TCP protocols.
Martin Wilck (35):
nvme-monitor: add new stub
monitor: create udev socket
monitor: initialize signal handling
monitor: add main loop for uevent monitoring
monitor: add uevent filters
monitor: Create a log() macro.
fabrics: use log() macro
monitor: add command line options to control logging nvme_get_ctrl_attr(): constify "path" argument
fabrics: export do_discover(), build_options() and config
monitor: add option -A / --autoconnect
monitor: add helpers for __attribute__((cleanup))
monitor: disable nvmf-autoconnect udev rules in autoconnect mode
monitor: implement handling of fc_udev_device
monitor: implement handling of nvme AEN events
monitor: reset children's signal disposition
monitor: handle SIGCHLD for terminated child processes
monitor: add "--persistent/-p" flag
fabrics: use "const char *" in struct config
fabrics: export arg_str(), parse_conn_arg(), and remove_ctrl()
nvme-cli: add "list.h"
conn-db: add simple connection registry
monitor: handle restart of pending discoveries
monitor: monitor_discovery(): try to reuse existing controllers
monitor: read existing connections on startup
monitor: implement starting discovery controllers on startup
monitor: implement cleanup of created discovery controllers
monitor: basic handling of add/remove uevents for nvme controllers
monitor: kill running discovery tasks on exit
monitor: add connection property options from connect-all
completions: add completions for nvme monitor
nvmf-autoconnect: add unit file for nvme-monitor.service nvme-connect-all(1): fix documentation for --quiet/-S nvme-monitor(1): add man page for nvme-monitor
monitor: add option --keep/-K
Documentation/cmds-main.txt | 4 + Documentation/nvme-connect-all.1 | 8 +- Documentation/nvme-connect-all.html | 10 +- Documentation/nvme-connect-all.txt | 4 +- Documentation/nvme-monitor.1 | 218 ++++ Documentation/nvme-monitor.html | 1067 +++++++++++++++++ Documentation/nvme-monitor.txt | 170 +++ Makefile | 10 + common.h | 12 + completions/bash-nvme-completion.sh | 6 +- conn-db.c | 341 ++++++ conn-db.h | 141 +++ fabrics.c | 145 +-- fabrics.h | 39 + list.h | 365 ++++++ log.h | 44 + monitor.c | 764 ++++++++++++ monitor.h | 6 + nvme-builtin.h | 1 + nvme-topology.c | 2 +- nvme.c | 13 + nvme.h | 2 +- nvmf-autoconnect/systemd/nvme-monitor.service | 17 + 23 files changed, 3296 insertions(+), 93 deletions(-) create mode 100644 Documentation/nvme-monitor.1 create mode 100644 Documentation/nvme-monitor.html create mode 100644 Documentation/nvme-monitor.txt create mode 100644 conn-db.c create mode 100644 conn-db.h create mode 100644 list.h create mode 100644 log.h create mode 100644 monitor.c create mode 100644 monitor.h create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 1/35 nvme-monitor: add new stub
From: Martin Wilck <[email protected]>
This stub will become the nvme AEN monitor in follow-up commits. monitor functionality depends on libudev.
--- Makefile | 10 ++++++++++ monitor.c | 28 ++++++++++++++++++++++++++++ monitor.h | 6 ++++++ nvme-builtin.h | 1 + nvme.c | 13 +++++++++++++ 5 files changed, 58 insertions(+) create mode 100644 monitor.c create mode 100644 monitor.h
Makefile +10 -0
@@ -4,6 +4,7 @@ override CPPFLAGS += -D_GNU_SOURCE -D__CHECK_ENDIAN__ LIBUUID = $(shell $(LD) -o /dev/null -luuid >/dev/null 2>&1; echo $$?) LIBHUGETLBFS = $(shell $(LD) -o /dev/null -lhugetlbfs >/dev/null 2>&1; echo $$?) HAVE_SYSTEMD = $(shell pkg-config --exists libsystemd --atleast-version=242; echo $$?) +HAVE_LIBUDEV = $(shell pkg-config --exists libudev; echo $$?) NVME = nvme INSTALL ?= install DESTDIR =
@@ -32,6 +33,11 @@ endif INC=-Iutil +ifeq ($(HAVE_LIBUDEV),0) + override LDFLAGS += -ludev + override CFLAGS += -DHAVE_LIBUDEV +endif + ifeq ($(HAVE_SYSTEMD),0) override LDFLAGS += -lsystemd override CFLAGS += -DHAVE_SYSTEMD
@@ -62,6 +68,10 @@ OBJS := nvme-print.o nvme-ioctl.o nvme-rpmb.o \ nvme-lightnvm.o fabrics.o nvme-models.o plugin.o \ nvme-status.o nvme-filters.o nvme-topology.o +ifeq ($(HAVE_LIBUDEV),0) + OBJS += monitor.o +endif + UTIL_OBJS := util/argconfig.o util/suffix.o util/json.o util/parser.o PLUGIN_OBJS := \
monitor.c new file +28 -0
@@ -0,0 +1,28 @@ +/* +* Copyright (C) 2021 SUSE LLC +* +* This program is free software; you can redistribute it and/or +* modify it under the terms of the GNU General Public License version +* 2 as published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +* +* This file implements a simple monitor for NVMe-related uevents. +*/ + +#include <errno.h> + +#include "nvme-status.h" +#include "monitor.h" + +int aen_monitor (const char *desc,int argc,char **argv) +{ +/* Handle as unimplemented for now */ +return nvme_status_to_errno (-ENOTTY , true); +} +
monitor.h new file +6 -0
@@ -0,0 +1,6 @@ +#ifndef _MONITOR_H +#define _MONITOR_H + +extern int aen_monitor (const char *desc,int argc,char **argv); + +#endif
nvme-builtin.h +1 -0
@@ -80,6 +80,7 @@ COMMAND_LIST( ENTRY("dir-send", "Submit a Directive Send command, return results", dir_send) ENTRY("virt-mgmt", "Manage Flexible Resources between Primary and Secondary Controller ", virtual_mgmt) ENTRY("rpmb", "Replay Protection Memory Block commands", rpmb_cmd) +ENTRY ("monitor" ,"Monitor NVMeoF AEN events" , monitor_cmd) );#endif
nvme.c +13 -0
@@ -57,6 +57,7 @@#include "argconfig.h" #include "fabrics.h" +#include "monitor.h" #define CREATE_CMD #include "nvme-builtin.h"
@@ -5320,6 +5321,18 @@ static int disconnect_all_cmd(int argc, char **argv, struct command *command, streturn fabrics_disconnect_all (desc, argc, argv); } +static int monitor_cmd (int argc,char **argv,struct command *command,struct plugin *plugin) +{ +#ifdef HAVE_LIBUDEV +const char *desc ="Monitor NVMeoF AEN events" ; + +return aen_monitor (desc, argc, argv); +#else +fprintf (stderr,"nvme-cli built without libudev doesn't support the \"monitor\" subcommand\n" ); +return EOPNOTSUPP ; +#endif +} +void register_extension (struct plugin *plugin) { plugin->parent = &nvme;
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 2/35 monitor: create udev socket
In reply to [email protected]From: Martin Wilck <[email protected]>
--- monitor.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-)
monitor.c +47 -3
@@ -15,14 +15,58 @@* This file implements a simple monitor for NVMe-related uevents. */ +#include <stddef.h> #include <errno.h> +#include <libudev.h> #include "nvme-status.h" #include "monitor.h" -int aen_monitor (const char *desc,int argc,char **argv) +static struct udev *udev; + +static void cleanup_monitor (struct udev_monitor **pmon) { -/* Handle as unimplemented for now */ -return nvme_status_to_errno (-ENOTTY , true); +if (*pmon) { +udev_monitor_unref (*pmon); + *pmon =NULL ; + } } +static int create_udev_monitor (struct udev_monitor **pmon) +{ + struct udev_monitor *mon __attribute((cleanup(cleanup_monitor))) = NULL; +int ret; + +if (!udev) { + udev =udev_new (); +if (!udev) +return -ENOMEM ; + } + mon =udev_monitor_new_from_netlink (udev,"kernel" ); +if (!mon) +return errno ? -errno : -ENOMEM ; + +/* +* This fails in unpriviliged mode. Use the same value as udevd. +* We may able to decrease this buffer size later. +*/ + (void )udev_monitor_set_receive_buffer_size (mon,128 *1024 *1024 ); + ret =udev_monitor_enable_receiving (mon); +if (ret <0 ) +return ret; + *pmon = mon; + mon =NULL ; +return 0 ; +} + +int aen_monitor (const char *desc,int argc,char **argv) +{ +int ret; +struct udev_monitor *monitor; + + ret =create_udev_monitor (&monitor); +if (ret ==0 ) +udev_monitor_unref (monitor); + udev =udev_unref (udev); +return nvme_status_to_errno (ret, true); +}
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 3/35 monitor: initialize signal handling
In reply to [email protected]From: Martin Wilck <[email protected]>
--- monitor.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
monitor.c +34 -0
@@ -16,8 +16,10 @@*/ #include <stddef.h> +#include <stdio.h> #include <errno.h> #include <libudev.h> +#include <signal.h> #include "nvme-status.h" #include "monitor.h"
@@ -59,14 +61,46 @@ static int create_udev_monitor(struct udev_monitor **pmon)return 0 ; } +static sig_atomic_t must_exit; + +static void monitor_int_handler (int sig) +{ + must_exit =1 ; +} + +static int monitor_init_signals (void ) +{ +sigset_t mask; +struct sigaction sa = { .sa_handler = monitor_int_handler, }; + +/* +* Block all signals. They will be unblocked when we wait +* for events. +*/ +sigfillset (&mask); +if (sigprocmask (SIG_BLOCK , &mask,NULL ) ==-1 ) +return -errno; +if (sigaction (SIGTERM , &sa,NULL ) ==-1 ) +return -errno; +if (sigaction (SIGINT , &sa,NULL ) ==-1 ) +return -errno; +return 0 ; +} +int aen_monitor (const char *desc,int argc,char **argv) {int ret;struct udev_monitor *monitor; + ret =monitor_init_signals (); +if (ret !=0 ) { +fprintf (stderr,"monitor: failed to initialize signals: %m\n" ); + goto out; + } ret =create_udev_monitor (&monitor);if (ret ==0 )udev_monitor_unref (monitor); udev =udev_unref (udev); +out:return nvme_status_to_errno (ret, true); }
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 4/35 monitor: add main loop for uevent monitoring
In reply to [email protected]From: Martin Wilck <[email protected]>
--- monitor.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 77 insertions(+), 1 deletion(-)
monitor.c +77 -1
@@ -17,15 +17,25 @@#include <stddef.h> #include <stdio.h> +#include <unistd.h> #include <errno.h> #include <libudev.h> #include <signal.h> +#include <sys/epoll.h> #include "nvme-status.h" #include "monitor.h" static struct udev *udev; +static void close_ptr (int *p) +{ +if (*p !=-1 ) { +close (*p); + *p =-1 ; + } +} +static void cleanup_monitor (struct udev_monitor **pmon) {if (*pmon) {
@@ -87,6 +97,70 @@ static int monitor_init_signals(void)return 0 ; } +static void monitor_handle_udevice (struct udev_device *ud) +{ +fprintf (stderr,"uevent: %s %s\n" , +udev_device_get_action (ud), +udev_device_get_sysname (ud)); +} + +static void monitor_handle_uevents (struct udev_monitor *monitor) +{ +struct udev_device *ud; + +for (ud =udev_monitor_receive_device (monitor); + ud; + ud =udev_monitor_receive_device (monitor)) { +monitor_handle_udevice (ud); +udev_device_unref (ud); + } +} + +#define MAX_EVENTS 1 +static int monitor_main_loop (struct udev_monitor *monitor) +{ + int ep_fd __attribute__((cleanup(close_ptr))) = -1; +int ret; +struct epoll_event ep_ev = { .events =EPOLLIN , }; +struct epoll_event events[MAX_EVENTS ]; +sigset_t ep_mask; + + ep_fd =epoll_create1 (0 ); +if (ep_fd ==-1 ) +return -errno; + ep_ev.data.ptr = monitor; + ret =epoll_ctl (ep_fd,EPOLL_CTL_ADD , +udev_monitor_get_fd (monitor), &ep_ev); +if (ret ==-1 ) +return -errno; + +sigfillset (&ep_mask); +sigdelset (&ep_mask,SIGTERM ); +sigdelset (&ep_mask,SIGINT ); +while (1 ) { +int rc, i; + + rc =epoll_pwait (ep_fd, events,MAX_EVENTS ,-1 , &ep_mask); +if (rc ==-1 && errno ==EINTR ) { +fprintf (stderr,"monitor: exit signal received\n" ); +return 0 ; + }else if (rc ==-1 ) { +fprintf (stderr,"monitor: epoll_wait: %m\n" ); +return -errno; + }else if (rc ==0 || rc >MAX_EVENTS ) { +fprintf (stderr,"monitor: epoll_wait: unexpected rc=%d\n" , rc); +continue ; + } +for (i =0 ; i <MAX_EVENTS ; i++) { +if (events[i].data.ptr == monitor) + (void )monitor_handle_uevents (monitor); +else +fprintf (stderr,"monitor: unexpected event\n" ); + } + } +return ret; +} +int aen_monitor (const char *desc,int argc,char **argv) {int ret;
@@ -98,8 +172,10 @@ int aen_monitor(const char *desc, int argc, char **argv) goto out; } ret =create_udev_monitor (&monitor); -if (ret ==0 ) +if (ret ==0 ) { + ret =monitor_main_loop (monitor);udev_monitor_unref (monitor); + } udev =udev_unref (udev); out:return nvme_status_to_errno (ret, true);
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 5/35 monitor: add uevent filters
In reply to [email protected]From: Martin Wilck <[email protected]>
--- monitor.c | 4 ++++ 1 file changed, 4 insertions(+)
monitor.c +4 -0
@@ -58,6 +58,10 @@ static int create_udev_monitor(struct udev_monitor **pmon)if (!mon)return errno ? -errno : -ENOMEM ; +/* Add match for NVMe controller devices */ + ret =udev_monitor_filter_add_match_subsystem_devtype (mon,"nvme" ,NULL ); +/* Add match for fc_udev_device */ + ret =udev_monitor_filter_add_match_subsystem_devtype (mon,"fc" ,NULL );/* * This fails in unpriviliged mode. Use the same value as udevd. * We may able to decrease this buffer size later.
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 05/35] monitor: add uevent filters
Quoted 13 lines · On 1/26/21 9:32 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
Signed-off-by Martin Wilck <[email protected]>--- monitor.c | 4 ++++ 1 file changed, 4 insertions(+)monitor.c +0
-0
monitor.c static int create_udev_monitor(struct udev_monitor **pmon) · line 67
Quoted 10 lines
monitor.c +4
-0@@ -58,6 +58,10 @@ static int create_udev_monitor(struct udev_monitor **pmon)if (!mon)return errno ? -errno : -ENOMEM ; +/* Add match for NVMe controller devices */ + ret =udev_monitor_filter_add_match_subsystem_devtype (mon,"nvme" ,NULL ); +/* Add match for fc_udev_device */ + ret =udev_monitor_filter_add_match_subsystem_devtype (mon,"fc" ,NULL );/* * This fails in unpriviliged mode. Use the same value as udevd. * We may able to decrease this buffer size later.
I guess you could merge all these five patches together; no need to keep them separate.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 6/35 monitor: Create a log() macro.
In reply to [email protected]From: Martin Wilck <[email protected]>
As this is a long running program, we need to make the log output configurable. First step: replace fprintf() by log(). The log level and printing of time stamps can be configured at run time using global variables. These will live in fabrics.c.
Allow toggling function name printing at build time. Printing the function name is useful for development, but perhaps not desired for production.
Put '#define LOG_FUNCNAME' before '#include "log.h"' to switch it on.
--- log.h | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 log.h
log.h new file +44 -0
@@ -0,0 +1,44 @@ +#ifndef _LOG_H +#define _LOG_H + +#ifndef MAX_LOGLEVEL +# define MAX_LOGLEVEL LOG_DEBUG +#endif +#ifndef DEFAULT_LOGLEVEL +# define DEFAULT_LOGLEVEL LOG_NOTICE +#endif + +#ifdef LOG_FUNCNAME +#define _func_fmt "%s: " +#define _func_arg __func__ +#else +#define _func_fmt "%s" +#define _func_arg "" +#endif + +extern int log_level; +extern bool log_timestamp; +#define _TIME_FMT "[%ld.%06ld] " +#define log (lvl, format, ...) \ + do { \ + int __lvl = (lvl); \ + \ + if (__lvl <= MAX_LOGLEVEL && __lvl <= log_level) { \ + if (log_timestamp) { \ + struct timespec __ts; \ + \ + clock_gettime(CLOCK_MONOTONIC, &__ts); \ + fprintf(stderr, \ + _TIME_FMT _func_fmt format, \ + __ts.tv_sec, __ts.tv_nsec / 1000,\ + _func_arg, \ + ##__VA_ARGS__); \ + } else { \ + fprintf(stderr, _func_fmt format, \ + _func_arg, \ + ##__VA_ARGS__); \ + }; \ + } \ + } while (0) + +#endif /* _LOG_H */
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 06/35] monitor: Create a log() macro.
Quoted 59 lines · On 1/26/21 9:32 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
As this is a long running program, we need to make the log output configurable. First step: replace
fprintf()bylog(). The log level and printing of time stamps can be configured at run time using global variables. These will live infabrics.c.Allow toggling function name printing at build time. Printing the function name is useful for development, but perhaps not desired for production.
Put '#defineLOG_FUNCNAME' before '#include "log.h"' to switch it on.Signed-off-by Martin Wilck <[email protected]>--- log.h | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)create mode 100644
log.hlog.h new file +34
-0@@ -0,0 +1,44 @@ +#ifndef _LOG_H +#define _LOG_H + +#ifndef MAX_LOGLEVEL +# define MAX_LOGLEVEL LOG_DEBUG +#endif +#ifndef DEFAULT_LOGLEVEL +# define DEFAULT_LOGLEVEL LOG_NOTICE +#endif + +#ifdef LOG_FUNCNAME +#define _func_fmt "%s: " +#define _func_arg __func__ +#else +#define _func_fmt "%s" +#define _func_arg "" +#endif + +extern int log_level; +extern bool log_timestamp; +#define _TIME_FMT "[%ld.%06ld] " +#define log (lvl, format, ...) \ + do { \ + int __lvl = (lvl); \ + \ + if (__lvl <= MAX_LOGLEVEL && __lvl <= log_level) { \ + if (log_timestamp) { \ + struct timespec __ts; \ + \ + clock_gettime(CLOCK_MONOTONIC, &__ts); \ + fprintf(stderr, \ + _TIME_FMT _func_fmt format, \ + __ts.tv_sec, __ts.tv_nsec / 1000,\ + _func_arg, \
log.h line 44
Quoted 10 lines
log.h new file +10
-0+ ##__VA_ARGS__); \ + }else { \ + fprintf(stderr, _func_fmt format, \ + _func_arg, \ + ##__VA_ARGS__); \ + }; \ + } \ + }while (0 ) + +#endif /* _LOG_H */
Urgh. Long macros are always horrible.
Can't you convert it into a function?
It might also be an idea to move this as the first function, as it's arguably an extension to existing functionality, and not directly related to the monitor.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 06/35] monitor: Create a log() macro.
Quoted 35 lines · Hannes Reinecke
Quoted 34 lines · On 1/26/21 9:32 PM, [email protected] wrote:
+ int
__lvl=
(lvl); \
+
\
+ if (__lvl<=MAX_LOGLEVEL&&__lvl<=log_level) { \
+ if (log_timestamp)
{ \
+ struct timespec
__ts; \
+
\
+ clock_gettime(CLOCK_MONOTONIC, &__ts); \
+ fprintf(stderr,
\
+ _TIME_FMT_func_fmtformat, \
+ __ts.tv_sec, __ts.tv_nsec / 1000,\
+ _func_arg,
\
+ ##__VA_ARGS__); \
+ } else
{ \
+ fprintf(stderr,_func_fmtformat, \
+ _func_arg,
\
+ ##__VA_ARGS__); \
+ };
\
Quoted 8 lines
Review
Quoted 6 lines
+ }
\
+ } while (0)(quoted diff) +2
-0+ +#endif /* _LOG_H */Urgh. Long macros are always horrible.
Can't you convert it into a function?
Sure. It won't work without macros, but I can write smaller ones.
Quoted 3 lines
It might also be an idea to move this as the first function, as it's arguably an extension to existing functionality, and not directly related to the monitor.
Ok.
Thanks,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 7/35 fabrics: use log() macro
In reply to [email protected]From: Martin Wilck <[email protected]>
Fabrics functionality will be used by the monitor code, we need consistent logging. This patch causes no functional change for the fabrics module.
--- fabrics.c | 83 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 45 insertions(+), 38 deletions(-)
fabrics.c +45 -38
@@ -32,6 +32,8 @@#include <libgen.h> #include <sys/stat.h> #include <stddef.h> +#include <syslog.h> +#include <time.h> #include <sys/types.h> #include <arpa/inet.h>
@@ -46,6 +48,10 @@#include "util/argconfig.h" #include "common.h" +#include "log.h" + +int log_level =LOG_NOTICE ; +bool log_timestamp;#ifdef HAVE_SYSTEMD #include <systemd/sd-id128.h>
@@ -86,7 +92,6 @@ static struct config {int hdr_digest;int data_digest;bool persistent; -bool quiet;bool matching_only; } cfg = { .ctrl_loss_tmo = NVMF_DEF_CTRL_LOSS_TMO };
@@ -345,7 +350,7 @@ static char *find_ctrl_with_connectargs(struct connect_args *args) n =scandir (SYS_NVME , &devices, scan_ctrls_filter, alphasort);if (n <0 ) { -fprintf (stderr,"no NVMe controller(s) detected.\n" ); +log (LOG_ERR ,"no NVMe controller(s) detected.\n" );return NULL ; }
@@ -353,7 +358,7 @@ static char *find_ctrl_with_connectargs(struct connect_args *args)if (ctrl_matches_connectargs (devices[i]->d_name, args)) { devname =strdup (devices[i]->d_name);if (devname ==NULL ) -fprintf (stderr,"no memory for ctrl name %s\n" , +log (LOG_ERR ,"no memory for ctrl name %s\n" , devices[i]->d_name); goto cleanup_devices; }
@@ -415,7 +420,7 @@ static int add_ctrl(const char *argstr) fd =open (PATH_NVME_FABRICS ,O_RDWR );if (fd <0 ) { -fprintf (stderr,"Failed to open %s: %s\n" , +log (LOG_ERR ,"Failed to open %s: %s\n" ,PATH_NVME_FABRICS ,strerror (errno)); ret = -errno; goto out;
@@ -423,8 +428,8 @@ static int add_ctrl(const char *argstr) ret =write (fd, argstr, len);if (ret != len) { -if (errno !=EALREADY || !cfg.quiet) -fprintf (stderr,"Failed to write to %s: %s\n" , +if (errno !=EALREADY ) +log (LOG_NOTICE ,"Failed to write to %s: %s\n" ,PATH_NVME_FABRICS ,strerror (errno)); ret = -errno; goto out_close;
@@ -432,7 +437,7 @@ static int add_ctrl(const char *argstr) len =read (fd, buf,BUF_SIZE );if (len <0 ) { -fprintf (stderr,"Failed to read from %s: %s\n" , +log (LOG_ERR ,"Failed to read from %s: %s\n" ,PATH_NVME_FABRICS ,strerror (errno)); ret = -errno; goto out_close;
@@ -459,7 +464,7 @@ static int add_ctrl(const char *argstr) } out_fail: -fprintf (stderr,"Failed to parse ctrl info for \"%s\"\n" , argstr); +log (LOG_ERR ,"Failed to parse ctrl info for \"%s\"\n" , argstr); ret = -EINVAL ; out_close:close (fd);
@@ -474,7 +479,7 @@ static int remove_ctrl_by_path(char *sysfs_path) fd =open (sysfs_path,O_WRONLY );if (fd <0 ) { ret = -errno; -fprintf (stderr,"Failed to open %s: %s\n" , sysfs_path, +log (LOG_ERR ,"Failed to open %s: %s\n" , sysfs_path,strerror (errno)); goto out; }
@@ -528,7 +533,7 @@ static int nvmf_get_log_page_discovery(const char *dev_path, fd =open (dev_path,O_RDWR );if (fd <0 ) { error = -errno; -fprintf (stderr,"Failed to open %s: %s\n" , +log (LOG_ERR ,"Failed to open %s: %s\n" , dev_path,strerror (errno)); goto out; }
@@ -699,7 +704,7 @@ static void save_discovery_log(struct nvmf_disc_rsp_page_hdr *log, int numrec) fd =open (cfg.raw,O_CREAT |O_RDWR |O_TRUNC ,S_IRUSR |S_IWUSR );if (fd <0 ) { -fprintf (stderr,"failed to open %s: %s\n" , +log (LOG_ERR ,"failed to open %s: %s\n" , cfg.raw,strerror (errno));return ; }
@@ -708,7 +713,7 @@ static void save_discovery_log(struct nvmf_disc_rsp_page_hdr *log, int numrec) numrec * sizeof(struct nvmf_disc_rsp_page_entry); ret =write (fd, log, len);if (ret <0 ) -fprintf (stderr,"failed to write to %s: %s\n" , +log (LOG_ERR ,"failed to write to %s: %s\n" , cfg.raw,strerror (errno));else printf ("Discovery log is saved to %s\n" , cfg.raw);
@@ -855,13 +860,13 @@ static int build_options(char *argstr, int max_len, bool discover)int len;if (!cfg.transport) { -fprintf (stderr,"need a transport (-t) argument\n" ); +log (LOG_ERR ,"need a transport (-t) argument\n" );return -EINVAL ; }if (strncmp (cfg.transport,"loop" ,4 )) {if (!cfg.traddr) { -fprintf (stderr,"need a address (-a) argument\n" ); +log (LOG_ERR ,"need a address (-a) argument\n" );return -EINVAL ; } }
@@ -945,7 +950,7 @@ static int hostname2traddr(struct config *cfg) ret =getaddrinfo (cfg->traddr,NULL , &hints, &host_info);if (ret) { -fprintf (stderr,"failed to resolve host %s info\n" , cfg->traddr); +log (LOG_ERR ,"failed to resolve host %s info\n" , cfg->traddr);return ret; }
@@ -961,14 +966,14 @@ static int hostname2traddr(struct config *cfg) addrstr, NVMF_TRADDR_SIZE);break ;default : -fprintf (stderr,"unrecognized address family (%d) %s\n" , +log (LOG_ERR ,"unrecognized address family (%d) %s\n" , host_info->ai_family, cfg->traddr); ret = -EINVAL ; goto free_addrinfo; }if (!p) { -fprintf (stderr,"failed to get traddr for %s\n" , cfg->traddr); +log (LOG_ERR ,"failed to get traddr for %s\n" , cfg->traddr); ret = -errno; goto free_addrinfo; }
@@ -996,7 +1001,7 @@ retry:case NVME_NQN_NVME :break ;default : -fprintf (stderr,"skipping unsupported subtype %d\n" , +log (LOG_ERR ,"skipping unsupported subtype %d\n" , e->subtype);return -EINVAL ; }
@@ -1085,7 +1090,7 @@ retry: transport =trtype_str (e->trtype);if (!strcmp (transport,"unrecognized" )) { -fprintf (stderr,"skipping unsupported transport %d\n" , +log (LOG_ERR ,"skipping unsupported transport %d\n" , e->trtype);return -EINVAL ; }
@@ -1131,7 +1136,7 @@ retry: p += len;break ;default : -fprintf (stderr,"skipping unsupported adrfam\n" ); +log (LOG_ERR ,"skipping unsupported adrfam\n" );return -EINVAL ; }break ;
@@ -1146,7 +1151,7 @@ retry: p += len;break ;default : -fprintf (stderr,"skipping unsupported adrfam\n" ); +log (LOG_ERR ,"skipping unsupported adrfam\n" );return -EINVAL ; }break ;
@@ -1231,9 +1236,7 @@ static int connect_ctrls(struct nvmf_disc_rsp_page_hdr *log, int numrec)if (instance == -EALREADY ) {const char *traddr = log->entries[i].traddr; -if (!cfg.quiet) -fprintf (stderr, -"traddr=%.*s is already connected\n" , +log (LOG_NOTICE ,"traddr=%.*s is already connected\n" ,space_strip_len (NVMF_TRADDR_SIZE , traddr), traddr);
@@ -1321,12 +1324,12 @@ static int do_discover(char *argstr, bool connect)print_discovery_log (log, numrec);break ;case DISC_GET_NUMRECS : -fprintf (stderr, +log (LOG_ERR ,"Get number of discovery log entries failed.\n" ); ret = status;break ;case DISC_GET_LOG : -fprintf (stderr,"Get discovery log entries failed.\n" ); +log (LOG_ERR ,"Get discovery log entries failed.\n" ); ret = status;break ;case DISC_NO_LOG :
@@ -1338,12 +1341,12 @@ static int do_discover(char *argstr, bool connect) ret = -EAGAIN ;break ;case DISC_NOT_EQUAL : -fprintf (stderr, +log (LOG_ERR ,"Numrec values of last two get discovery log page not equal\n" ); ret = -EBADSLT ;break ;default : -fprintf (stderr,"Get discovery log page failed: %d\n" , ret); +log (LOG_ERR ,"Get discovery log page failed: %d\n" , ret);break ; }
@@ -1359,7 +1362,7 @@ static int discover_from_conf_file(const char *desc, char *argstr, f =fopen (PATH_NVMF_DISC ,"r" );if (f ==NULL ) { -fprintf (stderr,"No discover params given and no %s conf\n" , +log (LOG_ERR ,"No discover params given and no %s conf\n" ,PATH_NVMF_DISC );return -EINVAL ; }
@@ -1370,14 +1373,14 @@ static int discover_from_conf_file(const char *desc, char *argstr, args =strdup (line);if (!args) { -fprintf (stderr,"failed to strdup args\n" ); +log (LOG_ERR ,"failed to strdup args\n" ); ret = -ENOMEM ; goto out; } argv =calloc (MAX_DISC_ARGS ,BUF_SIZE );if (!argv) { -fprintf (stderr,"failed to allocate argv vector\n" ); +log (LOG_ERR ,"failed to allocate argv vector\n" );free (args); ret = -ENOMEM ; goto out;
@@ -1428,6 +1431,7 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect) {char argstr[BUF_SIZE ];int ret; +bool quiet = false;OPT_ARGS (opts) = {OPT_LIST ("transport" ,'t' , &cfg.transport,"transport type" ),
@@ -1449,7 +1453,7 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect)OPT_INT ("nr-poll-queues" ,'P' , &cfg.nr_poll_queues,"number of poll queues to use (default 0)" ),OPT_INT ("queue-size" ,'Q' , &cfg.queue_size,"number of io queue elements to use (default 128)" ),OPT_FLAG ("persistent" ,'p' , &cfg.persistent,"persistent discovery connection" ), -OPT_FLAG ("quiet" ,'S' , &cfg.quiet,"suppress already connected errors" ), +OPT_FLAG ("quiet" ,'S' , &quiet,"suppress already connected errors" ),OPT_FLAG ("matching" ,'m' , &cfg.matching_only,"connect only records matching the traddr" ), OPT_END() };
@@ -1459,6 +1463,9 @@ int fabrics_discover(const char *desc, int argc, char **argv, bool connect)if (ret) goto out; +if (quiet) + log_level =LOG_WARNING ; +if (cfg.device && !strcmp (cfg.device,"none" )) cfg.device =NULL ;
@@ -1534,7 +1541,7 @@ int fabrics_connect(const char *desc, int argc, char **argv) goto out;if (!cfg.nqn) { -fprintf (stderr,"need a -n argument\n" ); +log (LOG_ERR ,"need a -n argument\n" ); ret = -EINVAL ; goto out; }
@@ -1572,7 +1579,7 @@ static int disconnect_subsys(char *nqn, char *ctrl) fd =open (sysfs_nqn_path,O_RDONLY );if (fd <0 ) { -fprintf (stderr,"Failed to open %s: %s\n" , +log (LOG_ERR ,"Failed to open %s: %s\n" , sysfs_nqn_path,strerror (errno)); goto free; }
@@ -1646,7 +1653,7 @@ int fabrics_disconnect(const char *desc, int argc, char **argv) goto out;if (!cfg.nqn && !cfg.device) { -fprintf (stderr,"need a -n or -d argument\n" ); +log (LOG_ERR ,"need a -n or -d argument\n" ); ret = -EINVAL ; goto out; }
@@ -1654,7 +1661,7 @@ int fabrics_disconnect(const char *desc, int argc, char **argv)if (cfg.nqn) { ret =disconnect_by_nqn (cfg.nqn);if (ret <0 ) -fprintf (stderr,"Failed to disconnect by NQN: %s\n" , +log (LOG_ERR ,"Failed to disconnect by NQN: %s\n" , cfg.nqn);else {printf ("NQN:%s disconnected %d controller(s)\n" , cfg.nqn, ret);
@@ -1665,7 +1672,7 @@ int fabrics_disconnect(const char *desc, int argc, char **argv)if (cfg.device) { ret =disconnect_by_device (cfg.device);if (ret) -fprintf (stderr, +log (LOG_ERR ,"Failed to disconnect by device name: %s\n" , cfg.device); }
@@ -1689,7 +1696,7 @@ int fabrics_disconnect_all(const char *desc, int argc, char **argv) err =scan_subsystems (&t,NULL ,0 ,NULL );if (err) { -fprintf (stderr,"Failed to scan namespaces\n" ); +log (LOG_ERR ,"Failed to scan namespaces\n" ); goto out; }
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 07/35] fabrics: use log() macro
Quoted 11 lines · On 1/26/21 9:32 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
Fabrics functionality will be used by the monitor code, we need consistent logging. This patch causes no functional change for the fabrics module.
Signed-off-by Martin Wilck <[email protected]>--- fabrics.c | 83 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 45 insertions(+), 38 deletions(-)
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 8/35 monitor: add command line options to control logging
In reply to [email protected]From: Martin Wilck <[email protected]>
Standard syslog loglevels are used.
Options:
-S/--silent to suppress LOG_NOTICE -v/--verbose to enable LOG_INFO (overrides -S) -D/--debug to enable LOG_DEBUG (overrides -S, -v)
-C/--clockstamps to enable time stamps on log messages.
I tried to use short options that don't conflict with options from nvme connect-all, because many of those options might be useful for the monitor, too.
--- monitor.c | 47 +++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 41 insertions(+), 6 deletions(-)
monitor.c +41 -6
@@ -21,10 +21,15 @@#include <errno.h> #include <libudev.h> #include <signal.h> +#include <time.h> +#include <syslog.h> #include <sys/epoll.h> #include "nvme-status.h" +#include "util/argconfig.h" #include "monitor.h" +#define LOG_FUNCNAME 1 +#include "log.h" static struct udev *udev;
@@ -103,7 +108,7 @@ static int monitor_init_signals(void)static void monitor_handle_udevice (struct udev_device *ud) { -fprintf (stderr,"uevent: %s %s\n" , +log (LOG_INFO ,"uevent: %s %s\n" ,udev_device_get_action (ud),udev_device_get_sysname (ud)); }
@@ -146,33 +151,63 @@ static int monitor_main_loop(struct udev_monitor *monitor) rc =epoll_pwait (ep_fd, events,MAX_EVENTS ,-1 , &ep_mask);if (rc ==-1 && errno ==EINTR ) { -fprintf (stderr,"monitor: exit signal received\n" ); +log (LOG_NOTICE ,"monitor: exit signal received\n" );return 0 ; }else if (rc ==-1 ) { -fprintf (stderr,"monitor: epoll_wait: %m\n" ); +log (LOG_ERR ,"monitor: epoll_wait: %m\n" );return -errno; }else if (rc ==0 || rc >MAX_EVENTS ) { -fprintf (stderr,"monitor: epoll_wait: unexpected rc=%d\n" , rc); +log (LOG_ERR ,"monitor: epoll_wait: unexpected rc=%d\n" , rc);continue ; }for (i =0 ; i <MAX_EVENTS ; i++) {if (events[i].data.ptr == monitor) (void )monitor_handle_uevents (monitor);else -fprintf (stderr,"monitor: unexpected event\n" ); +log (LOG_ERR ,"monitor: unexpected event\n" ); } }return ret; } +static int monitor_parse_opts (const char *desc,int argc,char **argv) +{ +bool quiet = false; +bool verbose = false; +bool debug = false; +int ret; +OPT_ARGS (opts) = { +OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ), +OPT_FLAG ("verbose" ,'v' , &verbose,"log level: verbose" ), +OPT_FLAG ("debug" ,'D' , &debug,"log level: debug" ), +OPT_FLAG ("clockstamps" ,'C' , &log_timestamp,"print log timestamps" ), +OPT_END () + }; + + ret =argconfig_parse (argc, argv, desc, opts); +if (ret) +return ret; +if (quiet) + log_level =LOG_WARNING ; +if (verbose) + log_level =LOG_INFO ; +if (debug) + log_level =LOG_DEBUG ; + +return ret; +} +int aen_monitor (const char *desc,int argc,char **argv) {int ret;struct udev_monitor *monitor; + ret =monitor_parse_opts (desc, argc, argv); +if (ret) + goto out; ret =monitor_init_signals ();if (ret !=0 ) { -fprintf (stderr,"monitor: failed to initialize signals: %m\n" ); +log (LOG_ERR ,"monitor: failed to initialize signals: %m\n" ); goto out; } ret =create_udev_monitor (&monitor);
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 08/35] monitor: add command line options to control logging
Quoted 14 lines · On 1/26/21 9:32 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
Standard syslog loglevels are used.
Options:-S/--silent to suppress LOG_NOTICE -v/--verbose to enable LOG_INFO (overrides -S) -D/--debug to enable LOG_DEBUG (overrides -S, -v)-C/--clockstamps to enable time stamps on log messages.
I tried to use short options that don't conflict with options from nvme connect-all, because many of those options might be useful for the monitor, too.
Quoted 5 lines
Signed-off-by Martin Wilck <[email protected]>--- monitor.c | 47 +++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 41 insertions(+), 6 deletions(-)... and if you had moved the logging functionality prior to the monitor
code, this patch could be merged with the main monitor functionality, too.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 08/35] monitor: add command line options to control logging
Quoted 15 lines · Hannes Reinecke
Quoted 14 lines · On 1/26/21 9:32 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
Standard syslog loglevels are used.
Options:-S/--silent to suppress
LOG_NOTICE-v/--verbose to enableLOG_INFO(overrides -S) -D/--debug to enableLOG_DEBUG(overrides -S, -v) -C/--clockstamps to enable time stamps on log messages.I tried to use short options that don't conflict with options from nvme connect-all, because many of those options might be useful for the monitor, too.
Quoted 8 lines
Quoted 6 lines
Signed-off-by Martin Wilck <[email protected]>---
monitor.c| 47 +++++++++++++++++++++++++++++++++++++++++------1 file changed, 41 insertions(+), 6 deletions(-)... and if you had moved the logging functionality prior to the monitor
code, this patch could be merged with the main monitor functionality, too.
I expected that reviewers preferred smaller chunks. If that's not the case, fine with me, I'll change i
Thanks,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 9/35 nvme_get_ctrl_attr(): constify "path" argument
In reply to [email protected]From: Martin Wilck <[email protected]>
--- nvme-topology.c | 2 +- nvme.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
nvme-topology.c +1 -1
@@ -45,7 +45,7 @@ close_fd:return subsysnqn; } -char *nvme_get_ctrl_attr (char *path,const char *attr) +char *nvme_get_ctrl_attr (const char *path,const char *attr) {char *attrpath, *value;ssize_t ret;
nvme.h +1 -1
@@ -109,7 +109,7 @@ int scan_subsystems(struct nvme_topology *t, const char *subsysnqn, __u32 ns_instance, char *dev_dir);void free_topology (struct nvme_topology *t);char *get_nvme_subsnqn (char *path); -char *nvme_get_ctrl_attr (char *path,const char *attr); +char *nvme_get_ctrl_attr (const char *path,const char *attr);void *nvme_alloc (size_t len,bool *huge);void nvme_free (void *p,bool huge);
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 09/35] nvme_get_ctrl_attr(): constify "path" argument
Quoted 27 lines · On 1/26/21 9:32 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
Signed-off-by Martin Wilck <[email protected]>--- nvme-topology.c | 2 +- nvme.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)nvme-topology.c +1
-1@@ -45,7 +45,7 @@ close_fd:return subsysnqn; } -char *nvme_get_ctrl_attr (char *path,const char *attr) +char *nvme_get_ctrl_attr (const char *path,const char *attr) {char *attrpath, *value;ssize_t ret;nvme.h +0
-0
nvme.h int scan_subsystems(struct nvme_topology *t, const char *subsysnqn, · line 115
Quoted 8 lines
nvme.h +1
-1@@ -109,7 +109,7 @@ int scan_subsystems(struct nvme_topology *t, const char *subsysnqn, __u32 ns_instance, char *dev_dir);void free_topology (struct nvme_topology *t);char *get_nvme_subsnqn (char *path); -char *nvme_get_ctrl_attr (char *path,const char *attr); +char *nvme_get_ctrl_attr (const char *path,const char *attr);void *nvme_alloc (size_t len,bool *huge);void nvme_free (void *p,bool huge);
Could be moved prior to the monitor code as it's unrelated to that.
Other than that:
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 10/35 fabrics: export do_discover(), build_options() and config
In reply to [email protected]From: Martin Wilck <[email protected]>
These functions will be called by the monitor code. Also export BUF_SIZE as the size of argstr.
Being able to access struct config and the "cfg" variable from fabrics.c is essential for the monitor to leverage the existing, well tested code as much as possible.
--- fabrics.c | 33 ++++----------------------------- fabrics.h | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 29 deletions(-)
fabrics.c +4 -29
@@ -69,31 +69,7 @@ const char *conarg_traddr = "traddr";const char *conarg_trsvcid ="trsvcid" ;const char *conarg_host_traddr ="host_traddr" ; -static struct config { -char *nqn; -char *transport; -char *traddr; -char *trsvcid; -char *host_traddr; -char *hostnqn; -char *hostid; -int nr_io_queues; -int nr_write_queues; -int nr_poll_queues; -int queue_size; -int keep_alive_tmo; -int reconnect_delay; -int ctrl_loss_tmo; -int tos; -char *raw; -char *device; -int duplicate_connect; -int disable_sqflow; -int hdr_digest; -int data_digest; -bool persistent; -bool matching_only; -} cfg = { .ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO }; +struct config cfg = { .ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO };struct connect_args {char *subsysnqn;
@@ -107,7 +83,6 @@ struct connect_args {struct connect_args *tracked_ctrls; -#define BUF_SIZE 4096#define PATH_NVME_FABRICS "/dev/nvme-fabrics"#define PATH_NVMF_DISC "/etc/nvme/discovery.conf"#define PATH_NVMF_HOSTNQN "/etc/nvme/hostnqn"
@@ -226,7 +201,7 @@ static const char *cms_str(__u8 cm)return arg_str (cms,ARRAY_SIZE (cms), cm); } -static int do_discover (char *argstr,bool connect); +int do_discover (char *argstr,bool connect);/* * parse strings with connect arguments to find a particular field.
@@ -855,7 +830,7 @@ add_argument(char **argstr, int *max_len, char *arg_str, char *arg)return 0 ; } -static int build_options (char *argstr,int max_len,bool discover) +int build_options (char *argstr,int max_len,bool discover) {int len;
@@ -1264,7 +1239,7 @@ static void nvmf_get_host_identifiers(int ctrl_instance) cfg.hostid =nvme_get_ctrl_attr (path,"hostid" ); } -static int do_discover (char *argstr,bool connect) +int do_discover (char *argstr,bool connect) {struct nvmf_disc_rsp_page_hdr *log =NULL ;char *dev_name;
fabrics.h +34 -0
@@ -10,4 +10,38 @@ extern int fabrics_connect(const char *desc, int argc, char **argv);extern int fabrics_disconnect (const char *desc,int argc,char **argv);extern int fabrics_disconnect_all (const char *desc,int argc,char **argv); +/* Symbols used by monitor.c */ + +struct config { +char *nqn; +char *transport; +char *traddr; +char *trsvcid; +char *host_traddr; +char *hostnqn; +char *hostid; +int nr_io_queues; +int nr_write_queues; +int nr_poll_queues; +int queue_size; +int keep_alive_tmo; +int reconnect_delay; +int ctrl_loss_tmo; +int tos; +char *raw; +char *device; +int duplicate_connect; +int disable_sqflow; +int hdr_digest; +int data_digest; +bool persistent; +bool matching_only; +}; +extern struct config cfg; + +#define BUF_SIZE 4096 + +int build_options (char *argstr,int max_len,bool discover); +int do_discover (char *argstr,bool connect); +#endif
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 10/35] fabrics: export do_discover(), build_options() and config
Quoted 121 lines · On 1/26/21 9:32 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
These functions will be called by the monitor code. Also export
BUF_SIZEas the size of argstr.Being able to access struct config and the "cfg" variable from
fabrics.cis essential for the monitor to leverage the existing, well tested code as much as possible.Signed-off-by Martin Wilck <[email protected]>--- fabrics.c | 33 ++++----------------------------- fabrics.h | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 29 deletions(-)fabrics.c +4
-29@@ -69,31 +69,7 @@ const char *conarg_traddr = "traddr";const char *conarg_trsvcid ="trsvcid" ;const char *conarg_host_traddr ="host_traddr" ; -static struct config { -char *nqn; -char *transport; -char *traddr; -char *trsvcid; -char *host_traddr; -char *hostnqn; -char *hostid; -int nr_io_queues; -int nr_write_queues; -int nr_poll_queues; -int queue_size; -int keep_alive_tmo; -int reconnect_delay; -int ctrl_loss_tmo; -int tos; -char *raw; -char *device; -int duplicate_connect; -int disable_sqflow; -int hdr_digest; -int data_digest; -bool persistent; -bool matching_only; -} cfg = { .ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO }; +struct config cfg = { .ctrl_loss_tmo =NVMF_DEF_CTRL_LOSS_TMO };struct connect_args {char *subsysnqn;@@ -107,7 +83,6 @@ struct connect_args {struct connect_args *tracked_ctrls; -#define BUF_SIZE 4096#define PATH_NVME_FABRICS "/dev/nvme-fabrics"#define PATH_NVMF_DISC "/etc/nvme/discovery.conf"#define PATH_NVMF_HOSTNQN "/etc/nvme/hostnqn"@@ -226,7 +201,7 @@ static const char *cms_str(__u8 cm)return arg_str (cms,ARRAY_SIZE (cms), cm); } -static int do_discover (char *argstr,bool connect); +int do_discover (char *argstr,bool connect);/* * parse strings with connect arguments to find a particular field. @@ -855,7 +830,7 @@ add_argument(char **argstr, int *max_len, char *arg_str, char *arg)return 0 ; } -static int build_options (char *argstr,int max_len,bool discover) +int build_options (char *argstr,int max_len,bool discover) {int len;@@ -1264,7 +1239,7 @@ static void nvmf_get_host_identifiers(int ctrl_instance) cfg.hostid =nvme_get_ctrl_attr (path,"hostid" ); } -static int do_discover (char *argstr,bool connect) +int do_discover (char *argstr,bool connect) {struct nvmf_disc_rsp_page_hdr *log =NULL ;char *dev_name;fabrics.h +25
-0@@ -10,4 +10,38 @@ extern int fabrics_connect(const char *desc, int argc, char **argv);extern int fabrics_disconnect (const char *desc,int argc,char **argv);extern int fabrics_disconnect_all (const char *desc,int argc,char **argv); +/* Symbols used by monitor.c */ + +struct config { +char *nqn; +char *transport; +char *traddr; +char *trsvcid; +char *host_traddr; +char *hostnqn; +char *hostid; +int nr_io_queues; +int nr_write_queues; +int nr_poll_queues; +int queue_size; +int keep_alive_tmo; +int reconnect_delay; +int ctrl_loss_tmo; +int tos; +char *raw; +char *device; +int duplicate_connect; +int disable_sqflow; +int hdr_digest; +int data_digest; +bool persistent;
fabrics.h extern int fabrics_connect(const char *desc, int argc, char **argv); · line 47
Quoted 10 lines
fabrics.h +9
-0+ bool matching_only; +}; +extern struct config cfg; + +#define BUF_SIZE 4096 + +int build_options (char *argstr,int max_len,bool discover); +int do_discover (char *argstr,bool connect); +#endif
Please name is 'struct config fabrics_cfg'. 'cfg' is used as a generic name throughout the code, and using the same name in a header might lead to an accidental name clash.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 10/35] fabrics: export do_discover(), build_options() and config
Quoted 35 lines · Hannes Reinecke
Quoted 34 lines · On 1/26/21 9:32 PM, [email protected] wrote:
fabrics.h +33
-0+/* Symbols used by monitor.c */ + +struct config { + char *nqn; + char *transport; + char *traddr; + char *trsvcid; + char *host_traddr; + char *hostnqn; + char *hostid; + int nr_io_queues; + int nr_write_queues; + int nr_poll_queues; + int queue_size; + int keep_alive_tmo; + int reconnect_delay; + int ctrl_loss_tmo; + int tos; + char *raw; + char *device; + int duplicate_connect; + int disable_sqflow; + int hdr_digest; + int data_digest; + bool persistent; + bool matching_only; +}; +extern struct config cfg; + +#define BUF_SIZE 4096 + +int build_options (char *argstr,int max_len,bool discover); +int do_discover (char *argstr,bool connect);
Quoted 8 lines
Quoted 3 lines
+
#endifPlease name is 'struct config
fabrics_cfg'. 'cfg' is used as a generic
name throughout the code, and using the same name in a header might lead to an accidental name clash.
Will do (the reason I didn't in the first place was that I wanted to minimize changes of existing code, but you're right of course).
Thanks,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 11/35 monitor: add option -A / --autoconnect
In reply to [email protected]From: Martin Wilck <[email protected]>
With this option, the monitor will try to connect to newly discovered controllers.
--- monitor.c | 5 +++++ 1 file changed, 5 insertions(+)
monitor.c +5 -0
@@ -31,6 +31,10 @@#define LOG_FUNCNAME 1#include "log.h" +static struct monitor_config { +bool autoconnect; +} mon_cfg; +static struct udev *udev;static void close_ptr (int *p)
@@ -177,6 +181,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)bool debug = false;int ret;OPT_ARGS (opts) = { +OPT_FLAG ("autoconnect" ,'A' , &mon_cfg.autoconnect,"automatically connect newly discovered controllers" ),OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ),OPT_FLAG ("verbose" ,'v' , &verbose,"log level: verbose" ),OPT_FLAG ("debug" ,'D' , &debug,"log level: debug" ),
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 11/35] monitor: add option -A / --autoconnect
Quoted 3 lines
With this option, the monitor will try to connect to newly discovered controllers.
Maybe I'm missing something here, but what does it do without autoconnect?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 11/35] monitor: add option -A / --autoconnect
Quoted 8 lines · Sagi Grimberg
Quoted 4 lines
With this option, the monitor will try to connect to newly discovered
controllers.Maybe I'm missing something here, but what does it do without autoconnect?
Basically it just runs "nvme discover". When events come in, it runs the discovery and prints the results, like "discover" would.
We can change that of course.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 11/35] monitor: add option -A / --autoconnect
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
With this option, the monitor will try to connect to newly discovered
controllers.
Quoted 8 lines
Quoted 3 lines
Maybe I'm missing something here, but what does it do without autoconnect?
Basically it just runs "nvme discover". When events come in, it runs the discovery and prints the results, like "discover" would.
We can change that of course.
But its a daemon, seems not very useful, but ok..
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 11/35] monitor: add option -A / --autoconnect
In reply to [email protected]Quoted 27 lines · On 1/26/21 9:33 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
With this option, the monitor will try to connect to newly discovered controllers.
Signed-off-by Martin Wilck <[email protected]>--- monitor.c | 5 +++++ 1 file changed, 5 insertions(+)monitor.c +4
-0@@ -31,6 +31,10 @@#define LOG_FUNCNAME 1#include "log.h" +static struct monitor_config { +bool autoconnect; +} mon_cfg; +static struct udev *udev;static void close_ptr (int *p)
monitor.c static int monitor_parse_opts(const char *desc, int argc, char **argv) · line 187
Quoted 7 lines
monitor.c +1
-0@@ -177,6 +181,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)bool debug = false;int ret;OPT_ARGS (opts) = { +OPT_FLAG ("autoconnect" ,'A' , &mon_cfg.autoconnect,"automatically connect newly discovered controllers" ),OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ),OPT_FLAG ("verbose" ,'v' , &verbose,"log level: verbose" ),OPT_FLAG ("debug" ,'D' , &debug,"log level: debug" ),
This patch is slightly confusing.
It just adds an option, but there is not code evaluating this option. So where does it hook in?
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 12/35 monitor: add helpers for __attribute__((cleanup))
In reply to [email protected]From: Martin Wilck <[email protected]>
__attribute__((cleanup)) is very helpful but ugly. Try to avoid defining lots of cleanup functions with these macros.
Usage: to declare an auto-cleanup variable of type (some_type *), write
CLEANUP_FUNC(some_type)
void some_func(void)
{
CLEANUP(some_type, varname) = NULL;
...
}--- common.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)
common.h +12 -0
@@ -12,4 +12,16 @@ #define __stringify_1(x...) #x #define __stringify(x...) __stringify_1(x) +#define CLEANUP_FUNC (type) \ +static void __cleanup_ ## type ##_p(type ** __p) \ +{ \ + if (*__p) { \ + free(*__p); \ + *__p = NULL; \ + } \ +} + +#define CLEANUP (__t, __v) \ + __t *__v __attribute__((cleanup(__cleanup_ ## __t ## _p))) +#endif
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 12/35] monitor: add helpers for __attribute__((cleanup))
Quoted 33 lines · On 1/26/21 9:33 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
__attribute__((cleanup)) is very helpful but ugly. Try to avoid defining lots of cleanup functions with these macros.
Usage: to declare an auto-cleanup variable of type (
some_type*), write
CLEANUP_FUNC(some_type)void
some_func(void){ CLEANUP(some_type, varname) = NULL; ... }Signed-off-by Martin Wilck <[email protected]>--- common.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)common.h +3
-0@@ -12,4 +12,16 @@ #define __stringify_1(x...) #x #define __stringify(x...) __stringify_1(x) +#define CLEANUP_FUNC (type) \ +static void __cleanup_ ## type ##_p(type ** __p) \ +{ \
common.h line 27
Quoted 10 lines
common.h +9
-0+if (*__p) { \ +free (*__p); \ + *__p =NULL ; \ + } \ +} + +#define CLEANUP (__t, __v) \ + __t *__v __attribute__((cleanup(__cleanup_ ## __t ## _p))) +#endif
And this is used where?
Can't you merge it with the patch usig the __cleanup functionality?
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 13/35 monitor: disable nvmf-autoconnect udev rules in autoconnect mode
In reply to [email protected]From: Martin Wilck <[email protected]>
If autoconnect is enabled, disable the respective udev rules by symlinking /run/udev/rules.d to /dev/null, in order to avoid the connections being set up by the monitor and the udev workers at the same time. This is probably the preferred mode of operation for the monitor.
Users can override this by copying 70-nvmf-autoconnect.rules from /usr/lib/udev/rules.d to /etc/udev/rules.d (/etc/udev/rules.d takes precedence over /run/udev/rules.d).
If the symlink can't be created for some reason, autoconnect will be disabled. There is only one exception: If /run/udev/rules.d/70-nvmf-autoconnect.rules already points to /dev/null at startup, autoconnect can be left on, but the symlink isn't removed on exit.
--- monitor.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 73 insertions(+), 2 deletions(-)
monitor.c +73 -2
@@ -17,14 +17,18 @@#include <stddef.h> #include <stdio.h> +#include <stdlib.h> #include <unistd.h> #include <errno.h> #include <libudev.h> #include <signal.h> #include <time.h> +#include <limits.h> #include <syslog.h> +#include <sys/stat.h> #include <sys/epoll.h> +#include "common.h" #include "nvme-status.h" #include "util/argconfig.h" #include "monitor.h"
@@ -33,6 +37,7 @@static struct monitor_config {bool autoconnect; +bool skip_udev_on_exit; } mon_cfg;static struct udev *udev;
@@ -45,6 +50,8 @@ static void close_ptr(int *p) } } +CLEANUP_FUNC (char ) + static void cleanup_monitor(struct udev_monitor **pmon) {if (*pmon) {
@@ -174,12 +181,64 @@ static int monitor_main_loop(struct udev_monitor *monitor)return ret; } +static const char autoconnect_rules[] ="/run/udev/rules.d/70-nvmf-autoconnect.rules" ; + +static int monitor_disable_udev_rules (void ) +{ +CLEANUP (char, path) =strdup (autoconnect_rules); +char *s1, *s2; +int rc; + +if (!path) +return -ENOMEM ; + + s2 =strrchr (path,'/' ); +for (s1 = s2 -1 ; s1 > path && *s1 !='/' ; s1--); + + *s2 = *s1 ='\0' ; + rc =mkdir (path,0755 ); +if (rc ==0 || errno ==EEXIST ) { + *s1 ='/' ; + rc =mkdir (path,0755 ); +if (rc ==0 || errno ==EEXIST ) { + *s2 ='/' ; + rc =symlink ("/dev/null" , path); + } + } +if (rc) { +if (errno ==EEXIST ) { +char target[PATH_MAX ]; + +if (readlink (path, target,sizeof (target)) !=-1 && + !strcmp (target,"/dev/null" )) { +log (LOG_INFO , +"symlink %s -> /dev/null exists already\n" , + autoconnect_rules); +return 1 ; + } + } +log (LOG_ERR ,"error creating %s: %m\n" , autoconnect_rules); + }else +log (LOG_INFO ,"created %s\n" , autoconnect_rules); + +return rc ? (errno ? -errno : -EIO ) :0 ; +} + +static void monitor_enable_udev_rules (void ) +{ +if (unlink (autoconnect_rules) ==-1 && errno !=ENOENT ) +log (LOG_ERR ,"error removing %s: %m\n" , autoconnect_rules); +else +log (LOG_INFO ,"removed %s\n" , autoconnect_rules); +} +static int monitor_parse_opts (const char *desc,int argc,char **argv) {bool quiet = false;bool verbose = false;bool debug = false; -int ret; +int ret =0 ; +OPT_ARGS (opts) = {OPT_FLAG ("autoconnect" ,'A' , &mon_cfg.autoconnect,"automatically connect newly discovered controllers" ),OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ),
@@ -198,7 +257,17 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv) log_level =LOG_INFO ;if (debug) log_level =LOG_DEBUG ; - +if (mon_cfg.autoconnect) { + ret =monitor_disable_udev_rules (); +if (ret <0 ) { + mon_cfg.autoconnect = false; +log (LOG_WARNING ,"autoconnect disabled\n" ); + ret =0 ; + }else if (ret >0 ) { + mon_cfg.skip_udev_on_exit = true; + ret =0 ; + } + }return ret; }
@@ -221,6 +290,8 @@ int aen_monitor(const char *desc, int argc, char **argv)udev_monitor_unref (monitor); } udev =udev_unref (udev); +if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit) +monitor_enable_udev_rules (); out:return nvme_status_to_errno (ret, true); }
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode
Quoted 12 lines · On 1/26/21 12:33 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
If autoconnect is enabled, disable the respective udev rules by symlinking
/run/udev/rules.dto/dev/null, in order to avoid the connections being set up by the monitor and the udev workers at the same time. This is probably the preferred mode of operation for the monitor.Users can override this by copying 70-nvmf-autoconnect.rules from
/usr/lib/udev/rules.dto/etc/udev/rules.d(/etc/udev/rules.dtakes precedence over/run/udev/rules.d).
Quoted 5 lines
If the symlink can't be created for some reason, autoconnect will be disabled. There is only one exception: If
/run/udev/rules.d/70-nvmf-autoconnect.rulesalready points to/dev/nullat startup, autoconnect can be left on, but the symlink isn't removed on exit.
Can't we just document that just one should be enabled and skip the entire thing? Especially when its not critical if both run..
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode
Quoted 15 lines · Sagi Grimberg
Quoted 12 lines · On 1/26/21 12:33 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
If autoconnect is enabled, disable the respective udev rules by symlinking
/run/udev/rules.dto/dev/null, in order to avoid the connections being set up by the monitor and the udev workers at the same time. This is probably the preferred mode of operation for the monitor.Users can override this by copying 70-nvmf-autoconnect.rules from
/usr/lib/udev/rules.dto/etc/udev/rules.d(/etc/udev/rules.dtakes precedence over/run/udev/rules.d).
Quoted 8 lines
Quoted 5 lines
If the symlink can't be created for some reason, autoconnect will be disabled. There is only one exception: If
/run/udev/rules.d/70-nvmf-autoconnect.rulesalready points to/dev/nullat startup, autoconnect can be left on, but the symlink isn't removed on exit.Can't we just document that just one should be enabled and skip the entire thing? Especially when its not critical if both run..
It's not critical, but it makes no sense to have the rules enabled when the monitor runs. And having to mask/unmask them manually is a pain.
It could be done in the systemd service file using ExecStartPre / ExecStopPost, with small restrictions (we couldn't distinguish whether the rules were already masked before we started). That would be more transparent for users.
I'm ok with not doing it automatically if that's generally preferred.
Regards
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode
Quoted 23 lines
Quoted 14 lines · On 1/26/21 12:33 PM, [email protected] wrote:
Quoted 12 lines
From: Martin Wilck <[email protected]>
If autoconnect is enabled, disable the respective udev rules by symlinking
/run/udev/rules.dto/dev/null, in order to avoid the connections being set up by the monitor and the udev workers at the same time. This is probably the preferred mode of operation for the monitor.Users can override this by copying 70-nvmf-autoconnect.rules from
/usr/lib/udev/rules.dto/etc/udev/rules.d(/etc/udev/rules.dtakes precedence over/run/udev/rules.d).Quoted 8 lines
Quoted 5 lines
If the symlink can't be created for some reason, autoconnect will be disabled. There is only one exception: If
/run/udev/rules.d/70-nvmf-autoconnect.rulesalready points to/dev/nullat startup, autoconnect can be left on, but the symlink isn't removed on exit.Can't we just document that just one should be enabled and skip the entire thing? Especially when its not critical if both run..
Quoted 7 lines
It's not critical, but it makes no sense to have the rules enabled when the monitor runs. And having to mask/unmask them manually is a pain.
It could be done in the systemd service file using ExecStartPre / ExecStopPost, with small restrictions (we couldn't distinguish whether the rules were already masked before we started). That would be more transparent for users.
Maybe 'Conflicts' can help here?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode
In reply to [email protected]Quoted 150 lines · On 1/26/21 9:33 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
If autoconnect is enabled, disable the respective udev rules by symlinking
/run/udev/rules.dto/dev/null, in order to avoid the connections being set up by the monitor and the udev workers at the same time. This is probably the preferred mode of operation for the monitor.Users can override this by copying 70-nvmf-autoconnect.rules from
/usr/lib/udev/rules.dto/etc/udev/rules.d(/etc/udev/rules.dtakes precedence over/run/udev/rules.d).If the symlink can't be created for some reason, autoconnect will be disabled. There is only one exception: If
/run/udev/rules.d/70-nvmf-autoconnect.rulesalready points to/dev/nullat startup, autoconnect can be left on, but the symlink isn't removed on exit.Signed-off-by Martin Wilck <[email protected]>--- monitor.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 73 insertions(+), 2 deletions(-)monitor.c +71
-2@@ -17,14 +17,18 @@#include <stddef.h> #include <stdio.h> +#include <stdlib.h> #include <unistd.h> #include <errno.h> #include <libudev.h> #include <signal.h> #include <time.h> +#include <limits.h> #include <syslog.h> +#include <sys/stat.h> #include <sys/epoll.h> +#include "common.h" #include "nvme-status.h" #include "util/argconfig.h" #include "monitor.h" @@ -33,6 +37,7 @@static struct monitor_config {bool autoconnect; +bool skip_udev_on_exit; } mon_cfg;static struct udev *udev;@@ -45,6 +50,8 @@ static void close_ptr(int *p) } } +CLEANUP_FUNC (char ) + static void cleanup_monitor(struct udev_monitor **pmon) {if (*pmon) {@@ -174,12 +181,64 @@ static int monitor_main_loop(struct udev_monitor *monitor)return ret; } +static const char autoconnect_rules[] ="/run/udev/rules.d/70-nvmf-autoconnect.rules" ; + +static int monitor_disable_udev_rules (void ) +{ +CLEANUP (char, path) =strdup (autoconnect_rules); +char *s1, *s2; +int rc; + +if (!path) +return -ENOMEM ; + + s2 =strrchr (path,'/' ); +for (s1 = s2 -1 ; s1 > path && *s1 !='/' ; s1--); + + *s2 = *s1 ='\0' ; + rc =mkdir (path,0755 ); +if (rc ==0 || errno ==EEXIST ) { + *s1 ='/' ; + rc =mkdir (path,0755 ); +if (rc ==0 || errno ==EEXIST ) { + *s2 ='/' ; + rc =symlink ("/dev/null" , path); + } + } +if (rc) { +if (errno ==EEXIST ) { +char target[PATH_MAX ]; + +if (readlink (path, target,sizeof (target)) !=-1 && + !strcmp (target,"/dev/null" )) { +log (LOG_INFO , +"symlink %s -> /dev/null exists already\n" , + autoconnect_rules); +return 1 ; + } + } +log (LOG_ERR ,"error creating %s: %m\n" , autoconnect_rules); + }else +log (LOG_INFO ,"created %s\n" , autoconnect_rules); + +return rc ? (errno ? -errno : -EIO ) :0 ; +} + +static void monitor_enable_udev_rules (void ) +{ +if (unlink (autoconnect_rules) ==-1 && errno !=ENOENT ) +log (LOG_ERR ,"error removing %s: %m\n" , autoconnect_rules); +else +log (LOG_INFO ,"removed %s\n" , autoconnect_rules); +} +static int monitor_parse_opts (const char *desc,int argc,char **argv) {bool quiet = false;bool verbose = false;bool debug = false; -int ret; +int ret =0 ; +OPT_ARGS (opts) = {OPT_FLAG ("autoconnect" ,'A' , &mon_cfg.autoconnect,"automatically connect newly discovered controllers" ),OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ),@@ -198,7 +257,17 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv) log_level =LOG_INFO ;if (debug) log_level =LOG_DEBUG ; - +if (mon_cfg.autoconnect) { + ret =monitor_disable_udev_rules (); +if (ret <0 ) { + mon_cfg.autoconnect = false; +log (LOG_WARNING ,"autoconnect disabled\n" ); + ret =0 ; + }else if (ret >0 ) { + mon_cfg.skip_udev_on_exit = true; + ret =0 ; + } + }return ret; }
monitor.c int aen_monitor(const char *desc, int argc, char **argv) · line 297
Quoted 8 lines
monitor.c +2
-0@@ -221,6 +290,8 @@ int aen_monitor(const char *desc, int argc, char **argv)udev_monitor_unref (monitor); } udev =udev_unref (udev); +if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit) +monitor_enable_udev_rules (); out:return nvme_status_to_errno (ret, true); }
What on earth ...
No way.
We should _not_ reference individual files or mess with udev rules from a program. If a file needs to be modified, modify it. But do not try to modify a file from a program context, unless the program itself has created it.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 13/35] monitor: disable nvmf-autoconnect udev rules in autoconnect mode
Quoted 174 lines · Hannes Reinecke
Quoted 167 lines · On 1/26/21 9:33 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
If autoconnect is enabled, disable the respective udev rules by symlinking
/run/udev/rules.dto/dev/null, in order to avoid the connections being set up by the monitor and the udev workers at the same time. This is probably the preferred mode of operation for the monitor.Users can override this by copying 70-nvmf-autoconnect.rules from
/usr/lib/udev/rules.dto/etc/udev/rules.d(/etc/udev/rules.dtakes precedence over/run/udev/rules.d).If the symlink can't be created for some reason, autoconnect will be disabled. There is only one exception: If
/run/udev/rules.d/70-nvmf-autoconnect.rulesalready points to/dev/nullat startup, autoconnect can be left on, but the symlink isn't removed on exit.Signed-off-by Martin Wilck <[email protected]>---
monitor.c| 75
+++++++++++++++++++++++++++++++++++++++++++++++++++++--1 file changed, 73 insertions(+), 2 deletions(-)monitor.c +0
-0@@ -17,14 +17,18 @@
#include <stddef.h>
#include <stdio.h>
+#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <libudev.h>
#include <signal.h>
#include <time.h>
+#include <limits.h>
#include <syslog.h>
+#include <sys/stat.h>
#include <sys/epoll.h>
+#include "common.h"
#include "nvme-status.h"
#include "util/argconfig.h"
#include "monitor.h"monitor.c +0
-0@@ -33,6 +37,7 @@
static structmonitor_config{
bool autoconnect;
+ boolskip_udev_on_exit;
}mon_cfg;
static struct udev *udev;monitor.c +0
-0@@ -45,6 +50,8 @@ static void close_ptr(int *p) } }monitor.c +2
-0+CLEANUP_FUNC (char) +static void
cleanup_monitor(struct udev_monitor **pmon){
if (*pmon) {monitor.c +0
-0@@ -174,12 +181,64 @@ static int monitor_main_loop(struct udev_monitor *monitor) return ret; }
+static const char autoconnect_rules[] = "/run/udev/rules.d/70-nvmf-autoconnect.rules";monitor.c +28
-0+ +static int monitor_disable_udev_rules (void ) +{ + CLEANUP(char, path) = strdup(autoconnect_rules); + char *s1, *s2; + int rc; + + if (!path) + return -ENOMEM; + + s2 = strrchr(path, '/'); + for (s1 = s2 - 1; s1 > path && *s1 != '/'; s1--); + + *s2 = *s1 = '\0'; + rc = mkdir(path, 0755); + if (rc == 0 || errno == EEXIST) { + *s1 = '/'; + rc = mkdir(path, 0755); + if (rc == 0 || errno == EEXIST) { + *s2 = '/'; + rc = symlink("/dev/null", path); + } + } + if (rc) { + if (errno == EEXIST) { + char target[PATH_MAX]; + + if (readlink(path, target, sizeof(target))!= -1 &&
+ !strcmp(target, "/dev/null")) { + log(LOG_INFO,
+ "symlink %s ->/dev/nullexists already\n",
+autoconnect_rules); + return 1;
+ }
+ }
+ log(LOG_ERR, "error creating %s: %m\n",autoconnect_rules);
+ } else
+ log(LOG_INFO, "created %s\n",autoconnect_rules);monitor.c +8
-0+ + return rc ? (errno ? -errno : -EIO) : 0; +} + +static void monitor_enable_udev_rules (void ) +{ + if (unlink(autoconnect_rules) == -1 && errno != ENOENT) + log(LOG_ERR, "error removing %s: %m\n",
autoconnect_rules);
+ else
+ log(LOG_INFO, "removed %s\n",autoconnect_rules);monitor.c +2
-0+} +static int
monitor_parse_opts(const char *desc, int argc, char **argv)
{
bool quiet = false;
bool verbose = false;
bool debug = false;
- int ret;
+ int ret = 0;
+
OPT_ARGS(opts) = {
OPT_FLAG("autoconnect", 'A', &mon_cfg.autoconnect, "automatically connect newly discovered controllers"),
OPT_FLAG("silent", 'S', &quiet, "log level: silent"),monitor.c +11
-1@@ -198,7 +257,17 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv) log_level = LOG_INFO; if (debug) log_level = LOG_DEBUG; - + if (mon_cfg.autoconnect) { + ret = monitor_disable_udev_rules(); + if (ret < 0) { + mon_cfg.autoconnect = false; + log(LOG_WARNING, "autoconnect disabled\n"); + ret = 0; + } else if (ret > 0) { + mon_cfg.skip_udev_on_exit = true; + ret = 0; + } + } return ret; }monitor.c +2
-0@@ -221,6 +290,8 @@ int aen_monitor(const char *desc, int argc, char **argv) udev_monitor_unref(monitor); } udev = udev_unref(udev); + if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit) + monitor_enable_udev_rules();Quoted 4 lines
out:
returnnvme_status_to_errno(ret, true);
}What on earth ...
Quoted 6 lines
No way.
We should _not_ reference individual files or mess with udev rules from
a program. If a file needs to be modified, modify it. But do not try to modify a file from a program context, unless the program itself has created it.
Sagi said the same, I'll drop this. I thought it was acceptable because I'd be masking a rule from the same package, and preferable because making the user responsible for it would be cumbersome and error-prone on the part of the user.
Thanks,
Martin
Quoted 4 lines
Cheers,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 14/35 monitor: implement handling of fc_udev_device
In reply to [email protected]From: Martin Wilck <[email protected]>
The reason we fork off discovery tasks is two-fold: Firstly, we'd otherwise be forced to make all discovery connections sequentially, which would be slow, as connecting controllers can block on the order of seconds even in successful cases.
Secondly, this allows us to use the some static variables like cfg and tracked_ctrls in the discovery code path. Without forking, we'd have to re-write much more code in fabrics.c. In general, the alternative to forking would be creating threads, but the that would probably require a large rewrite of our code base.
A single-threaded server that forks off the actual discovery makes most sense at this point.
--- monitor.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 107 insertions(+), 3 deletions(-)
monitor.c +107 -3
@@ -30,7 +30,9 @@#include "common.h" #include "nvme-status.h" +#include "nvme.h" #include "util/argconfig.h" +#include "fabrics.h" #include "monitor.h" #define LOG_FUNCNAME 1#include "log.h"
@@ -117,11 +119,113 @@ static int monitor_init_signals(void)return 0 ; } +static int monitor_get_fc_uev_props (struct udev_device *ud, +char *traddr,size_t tra_sz, +char *host_traddr,size_t htra_sz) +{ +const char *sysname =udev_device_get_sysname (ud); +const char *tra =NULL , *host_tra =NULL ; +bool fc_event_seen = false; +struct udev_list_entry *entry; + + entry =udev_device_get_properties_list_entry (ud); +if (!entry) { +log (LOG_NOTICE ,"%s: emtpy properties list\n" , sysname); +return -ENOENT ; + } + +for (; entry; entry =udev_list_entry_get_next (entry)) { +const char *name =udev_list_entry_get_name (entry); + +if (!strcmp (name,"FC_EVENT" ) && + !strcmp (udev_list_entry_get_value (entry),"nvmediscovery" )) + fc_event_seen = true; +else if (!strcmp (name,"NVMEFC_HOST_TRADDR" )) + host_tra =udev_list_entry_get_value (entry); +else if (!strcmp (name,"NVMEFC_TRADDR" )) + tra =udev_list_entry_get_value (entry); + } +if (!fc_event_seen) { +log (LOG_DEBUG ,"%s: FC_EVENT property missing or unsupported\n" , + sysname); +return -EINVAL ; + } +if (!tra || !host_tra) { +log (LOG_WARNING ,"%s: transport properties missing\n" , sysname); +return -EINVAL ; + } + +if (!memccpy (traddr, tra,'\0' , tra_sz) || + !memccpy (host_traddr, host_tra,'\0' , htra_sz)) { +log (LOG_ERR ,"traddr (%zu) or host_traddr (%zu) overflow\n" , +strlen (traddr),strlen (host_traddr)); +return -ENAMETOOLONG ; + } + +return 0 ; +} + +static int monitor_discovery (char *transport,char *traddr,char *trsvcid, +char *host_traddr) +{ +char argstr[BUF_SIZE ]; +pid_t pid; +int rc; + + pid =fork (); +if (pid ==-1 ) { +log (LOG_ERR ,"failed to fork discovery task: %m" ); +return -errno; + }else if (pid >0 ) +return 0 ; + +log (LOG_NOTICE ,"starting %s discovery for %s==>%s(%s)\n" , + transport, host_traddr, traddr, trsvcid ? trsvcid :"none" ); + cfg.nqn =NVME_DISC_SUBSYS_NAME ; + cfg.transport = transport; + cfg.traddr = traddr; + cfg.trsvcid = trsvcid; + cfg.host_traddr = host_traddr; +/* Without the following, the kernel returns EINVAL */ + cfg.tos =-1 ; + + rc =build_options (argstr,sizeof (argstr), true); +log (LOG_DEBUG ,"%s\n" , argstr); + rc =do_discover (argstr, mon_cfg.autoconnect); + +exit (-rc); +/* not reached */ +return rc; +} + +static void monitor_handle_fc_uev (struct udev_device *ud) +{ +const char *action =udev_device_get_action (ud); +const char *sysname =udev_device_get_sysname (ud); +char traddr[NVMF_TRADDR_SIZE ], host_traddr[NVMF_TRADDR_SIZE ]; + +if (strcmp (action,"change" ) ||strcmp (sysname,"fc_udev_device" )) +return ; + +if (monitor_get_fc_uev_props (ud, traddr,sizeof (traddr), + host_traddr,sizeof (host_traddr))) +return ; + +monitor_discovery ("fc" , traddr,NULL , host_traddr); +} +static void monitor_handle_udevice (struct udev_device *ud) { -log (LOG_INFO ,"uevent: %s %s\n" , -udev_device_get_action (ud), -udev_device_get_sysname (ud)); +const char *subsys =udev_device_get_subsystem (ud); + +if (log_level >=LOG_INFO ) { +const char *action =udev_device_get_action (ud); +const char *syspath =udev_device_get_syspath (ud); + +log (LOG_INFO ,"%s %s\n" , action, syspath); + } +if (!strcmp (subsys,"fc" )) +monitor_handle_fc_uev (ud); }static void monitor_handle_uevents (struct udev_monitor *monitor)
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 15/35 monitor: implement handling of nvme AEN events
In reply to [email protected]From: Martin Wilck <[email protected]>
This is very similar to fc_udev_device events, except that we have different udev properties and must use trsvcid.
--- monitor.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
monitor.c +70 -0
@@ -214,6 +214,74 @@ static void monitor_handle_fc_uev(struct udev_device *ud)monitor_discovery ("fc" , traddr,NULL , host_traddr); } +static int monitor_get_nvme_uev_props (struct udev_device *ud, +char *transport,size_t tr_sz, +char *traddr,size_t tra_sz, +char *trsvcid,size_t trs_sz, +char *host_traddr,size_t htra_sz) +{ +const char *sysname =udev_device_get_sysname (ud); +bool aen_disc = false; +struct udev_list_entry *entry; + + entry =udev_device_get_properties_list_entry (ud); +if (!entry) { +log (LOG_NOTICE ,"%s: emtpy properties list\n" , sysname); +return -ENOENT ; + } + + *transport = *traddr = *trsvcid = *host_traddr ='\0' ; +for (; entry; entry =udev_list_entry_get_next (entry)) { +const char *name =udev_list_entry_get_name (entry); + +if (!strcmp (name,"NVME_AEN" ) && + !strcmp (udev_list_entry_get_value (entry),"0x70f002" )) + aen_disc = true; +else if (!strcmp (name,"NVME_TRTYPE" )) +memccpy (transport,udev_list_entry_get_value (entry), +'\0' , tr_sz); +else if (!strcmp (name,"NVME_TRADDR" )) +memccpy (traddr,udev_list_entry_get_value (entry), +'\0' , htra_sz); +else if (!strcmp (name,"NVME_TRSVCID" )) +memccpy (trsvcid,udev_list_entry_get_value (entry), +'\0' , trs_sz); +else if (!strcmp (name,"NVME_HOST_TRADDR" )) +memccpy (host_traddr,udev_list_entry_get_value (entry), +'\0' , tra_sz); + } +if (!aen_disc) { +log (LOG_DEBUG ,"%s: not a \"discovery log changed\" AEN, ignoring event\n" , + sysname); +return -EINVAL ; + } + +if (!*traddr || !*transport) { +log (LOG_WARNING ,"%s: transport properties missing\n" , sysname); +return -EINVAL ; + } + +return 0 ; +} + +static void monitor_handle_nvme_uev (struct udev_device *ud) +{ +char traddr[NVMF_TRADDR_SIZE ], host_traddr[NVMF_TRADDR_SIZE ]; +char trsvcid[NVMF_TRSVCID_SIZE ], transport[5 ]; + +if (strcmp (udev_device_get_action (ud),"change" )) +return ; + +if (monitor_get_nvme_uev_props (ud, transport,sizeof (transport), + traddr,sizeof (traddr), + trsvcid,sizeof (trsvcid), + host_traddr,sizeof (host_traddr))) +return ; + +monitor_discovery (transport, traddr, +strcmp (trsvcid,"none" ) ? trsvcid :NULL , host_traddr); +} +static void monitor_handle_udevice (struct udev_device *ud) {const char *subsys =udev_device_get_subsystem (ud);
@@ -226,6 +294,8 @@ static void monitor_handle_udevice(struct udev_device *ud) }if (!strcmp (subsys,"fc" ))monitor_handle_fc_uev (ud); +else if (!strcmp (subsys,"nvme" )) +monitor_handle_nvme_uev (ud); }static void monitor_handle_uevents (struct udev_monitor *monitor)
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 16/35 monitor: reset children's signal disposition
In reply to [email protected]From: Martin Wilck <[email protected]>
--- monitor.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
monitor.c +28 -2
@@ -100,6 +100,8 @@ static void monitor_int_handler(int sig) must_exit =1 ; } +static sigset_t orig_sigmask; +static int monitor_init_signals (void ) {sigset_t mask;
@@ -110,7 +112,7 @@ static int monitor_init_signals(void)* for events. */ sigfillset (&mask); -if (sigprocmask (SIG_BLOCK , &mask,NULL ) ==-1 ) +if (sigprocmask (SIG_BLOCK , &mask, &orig_sigmask) ==-1 )return -errno;if (sigaction (SIGTERM , &sa,NULL ) ==-1 )return -errno;
@@ -119,6 +121,26 @@ static int monitor_init_signals(void)return 0 ; } +static int child_reset_signals (void ) +{ +int err =0 ; +struct sigaction sa = { .sa_handler =SIG_DFL , }; + +if (sigaction (SIGTERM , &sa,NULL ) ==-1 ) + err = errno; +if (sigaction (SIGINT , &sa,NULL ) ==-1 && !err) + err = errno; +if (sigaction (SIGCHLD , &sa,NULL ) ==-1 && !err) + err = errno; + +if (sigprocmask (SIG_SETMASK , &orig_sigmask,NULL ) ==-1 && !err) + err = errno; + +if (err) +log (LOG_ERR ,"error resetting signal handlers and mask\n" ); +return -err; +} +static int monitor_get_fc_uev_props (struct udev_device *ud,char *traddr,size_t tra_sz,char *host_traddr,size_t htra_sz)
@@ -176,8 +198,12 @@ static int monitor_discovery(char *transport, char *traddr, char *trsvcid,if (pid ==-1 ) {log (LOG_ERR ,"failed to fork discovery task: %m" );return -errno; - }else if (pid >0 ) + }else if (pid >0 ) { +log (LOG_DEBUG ,"started discovery task %ld\n" , (long )pid);return 0 ; + } + +child_reset_signals ();log (LOG_NOTICE ,"starting %s discovery for %s==>%s(%s)\n" , transport, host_traddr, traddr, trsvcid ? trsvcid :"none" );
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 16/35] monitor: reset children's signal disposition
No commit message?
Quoted 60 lines
Signed-off-by Martin Wilck <[email protected]>--- monitor.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)monitor.c +23
-1@@ -100,6 +100,8 @@ static void monitor_int_handler(int sig) must_exit =1 ; } +static sigset_t orig_sigmask; +static int monitor_init_signals (void ) {sigset_t mask;@@ -110,7 +112,7 @@ static int monitor_init_signals(void)* for events. */ sigfillset (&mask); -if (sigprocmask (SIG_BLOCK , &mask,NULL ) ==-1 ) +if (sigprocmask (SIG_BLOCK , &mask, &orig_sigmask) ==-1 )return -errno;if (sigaction (SIGTERM , &sa,NULL ) ==-1 )return -errno;@@ -119,6 +121,26 @@ static int monitor_init_signals(void)return 0 ; } +static int child_reset_signals (void ) +{ +int err =0 ; +struct sigaction sa = { .sa_handler =SIG_DFL , }; + +if (sigaction (SIGTERM , &sa,NULL ) ==-1 ) + err = errno; +if (sigaction (SIGINT , &sa,NULL ) ==-1 && !err) + err = errno; +if (sigaction (SIGCHLD , &sa,NULL ) ==-1 && !err) + err = errno; + +if (sigprocmask (SIG_SETMASK , &orig_sigmask,NULL ) ==-1 && !err) + err = errno; + +if (err) +log (LOG_ERR ,"error resetting signal handlers and mask\n" ); +return -err; +} +static int monitor_get_fc_uev_props (struct udev_device *ud,char *traddr,size_t tra_sz,char *host_traddr,size_t htra_sz)@@ -176,8 +198,12 @@ static int monitor_discovery(char *transport, char *traddr, char *trsvcid,if (pid ==-1 ) {log (LOG_ERR ,"failed to fork discovery task: %m" );return -errno;
monitor.c static int monitor_discovery(char *transport, char *traddr, char *trsvcid, · line 209
Quoted 10 lines
monitor.c +5
-1- }else if (pid >0 ) + }else if (pid >0 ) { +log (LOG_DEBUG ,"started discovery task %ld\n" , (long )pid);return 0 ; + } + +child_reset_signals ();log (LOG_NOTICE ,"starting %s discovery for %s==>%s(%s)\n" , transport, host_traddr, traddr, trsvcid ? trsvcid :"none" );
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 16/35] monitor: reset children's signal disposition
Quoted 1 line · Sagi Grimberg
No commit message?
I'm sorry. I'll add one next time. For #17 likewise.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 17/35 monitor: handle SIGCHLD for terminated child processes
In reply to [email protected]From: Martin Wilck <[email protected]>
--- monitor.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-)
monitor.c +48 -2
@@ -26,6 +26,8 @@#include <limits.h> #include <syslog.h> #include <sys/stat.h> +#include <sys/types.h> +#include <sys/wait.h> #include <sys/epoll.h> #include "common.h"
@@ -100,6 +102,13 @@ static void monitor_int_handler(int sig) must_exit =1 ; } +static sig_atomic_t got_sigchld; + +static void monitor_chld_handler (int sig) +{ + got_sigchld =1 ; +} +static sigset_t orig_sigmask;static int monitor_init_signals (void )
@@ -118,6 +127,9 @@ static int monitor_init_signals(void)return -errno;if (sigaction (SIGINT , &sa,NULL ) ==-1 )return -errno; + sa.sa_handler = monitor_chld_handler; +if (sigaction (SIGCHLD , &sa,NULL ) ==-1 ) +return -errno;return 0 ; }
@@ -336,6 +348,34 @@ static void monitor_handle_uevents(struct udev_monitor *monitor) } } +static void handle_sigchld (void ) +{ +while (true) { +int wstatus; +pid_t pid; + + pid =waitpid (-1 , &wstatus,WNOHANG ); +switch (pid) { +case -1 : +if (errno !=ECHILD ) +log (LOG_ERR ,"error in waitpid: %m\n" ); +return ; +case 0 : +return ; +default : +break ; + } +if (!WIFEXITED (wstatus)) +log (LOG_WARNING ,"child %ld didn't exit normally\n" , + (long )pid); +else if (WEXITSTATUS (wstatus) !=0 ) +log (LOG_NOTICE ,"child %ld exited with status \"%s\"\n" , + (long )pid,strerror (WEXITSTATUS (wstatus))); +else +log (LOG_DEBUG ,"child %ld exited normally\n" , (long )pid); + }; +} +#define MAX_EVENTS 1static int monitor_main_loop (struct udev_monitor *monitor) {
@@ -357,13 +397,19 @@ static int monitor_main_loop(struct udev_monitor *monitor)sigfillset (&ep_mask);sigdelset (&ep_mask,SIGTERM );sigdelset (&ep_mask,SIGINT ); +sigdelset (&ep_mask,SIGCHLD );while (1 ) {int rc, i; rc =epoll_pwait (ep_fd, events,MAX_EVENTS ,-1 , &ep_mask);if (rc ==-1 && errno ==EINTR ) { -log (LOG_NOTICE ,"monitor: exit signal received\n" ); -return 0 ; +if (must_exit) { +log (LOG_NOTICE ,"monitor: exit signal received\n" ); +return 0 ; + }else if (got_sigchld) { + got_sigchld =0 ; +handle_sigchld (); + } }else if (rc ==-1 ) {log (LOG_ERR ,"monitor: epoll_wait: %m\n" );return -errno;
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 17/35] monitor: handle SIGCHLD for terminated child processes
No commit message?
Quoted 90 lines
Signed-off-by Martin Wilck <[email protected]>--- monitor.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-)monitor.c +41
-2@@ -26,6 +26,8 @@#include <limits.h> #include <syslog.h> #include <sys/stat.h> +#include <sys/types.h> +#include <sys/wait.h> #include <sys/epoll.h> #include "common.h" @@ -100,6 +102,13 @@ static void monitor_int_handler(int sig) must_exit =1 ; } +static sig_atomic_t got_sigchld; + +static void monitor_chld_handler (int sig) +{ + got_sigchld =1 ; +} +static sigset_t orig_sigmask;static int monitor_init_signals (void )@@ -118,6 +127,9 @@ static int monitor_init_signals(void)return -errno;if (sigaction (SIGINT , &sa,NULL ) ==-1 )return -errno; + sa.sa_handler = monitor_chld_handler; +if (sigaction (SIGCHLD , &sa,NULL ) ==-1 ) +return -errno;return 0 ; }@@ -336,6 +348,34 @@ static void monitor_handle_uevents(struct udev_monitor *monitor) } } +static void handle_sigchld (void ) +{ +while (true) { +int wstatus; +pid_t pid; + + pid =waitpid (-1 , &wstatus,WNOHANG ); +switch (pid) { +case -1 : +if (errno !=ECHILD ) +log (LOG_ERR ,"error in waitpid: %m\n" ); +return ; +case 0 : +return ; +default : +break ; + } +if (!WIFEXITED (wstatus)) +log (LOG_WARNING ,"child %ld didn't exit normally\n" , + (long )pid); +else if (WEXITSTATUS (wstatus) !=0 ) +log (LOG_NOTICE ,"child %ld exited with status \"%s\"\n" , + (long )pid,strerror (WEXITSTATUS (wstatus))); +else +log (LOG_DEBUG ,"child %ld exited normally\n" , (long )pid); + }; +} +#define MAX_EVENTS 1static int monitor_main_loop (struct udev_monitor *monitor) {@@ -357,13 +397,19 @@ static int monitor_main_loop(struct udev_monitor *monitor)sigfillset (&ep_mask);sigdelset (&ep_mask,SIGTERM );sigdelset (&ep_mask,SIGINT ); +sigdelset (&ep_mask,SIGCHLD );while (1 ) {int rc, i; rc =epoll_pwait (ep_fd, events,MAX_EVENTS ,-1 , &ep_mask);if (rc ==-1 && errno ==EINTR ) { -log (LOG_NOTICE ,"monitor: exit signal received\n" ); -return 0 ;
monitor.c static int monitor_main_loop(struct udev_monitor *monitor) · line 415
Quoted 10 lines
monitor.c +7
-0+if (must_exit) { +log (LOG_NOTICE ,"monitor: exit signal received\n" ); +return 0 ; + }else if (got_sigchld) { + got_sigchld =0 ; +handle_sigchld (); + } }else if (rc ==-1 ) {log (LOG_ERR ,"monitor: epoll_wait: %m\n" );return -errno;
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 18/35 monitor: add "--persistent/-p" flag
In reply to [email protected]From: Martin Wilck <[email protected]>
Without this flag, AENs on created controllers won't be received. While --persistent is probably the behavior most users will want, it has to be explicitly activated, to be consistent with the connect-all command.
--- monitor.c | 5 +++++ 1 file changed, 5 insertions(+)
monitor.c +5 -0
@@ -487,6 +487,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)OPT_ARGS (opts) = {OPT_FLAG ("autoconnect" ,'A' , &mon_cfg.autoconnect,"automatically connect newly discovered controllers" ), +OPT_FLAG ("persistent" ,'p' , &cfg.persistent,"persistent discovery connections" ),OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ),OPT_FLAG ("verbose" ,'v' , &verbose,"log level: verbose" ),OPT_FLAG ("debug" ,'D' , &debug,"log level: debug" ),
@@ -514,6 +515,10 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv) ret =0 ; } } + +if (cfg.persistent && !cfg.keep_alive_tmo) + cfg.keep_alive_tmo =NVMF_DEF_DISC_TMO ; +return ret; }
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 18/35] monitor: add "--persistent/-p" flag
Quoted 5 lines
Without this flag, AENs on created controllers won't be received. While --persistent is probably the behavior most users will want, it has to be explicitly activated, to be consistent with the connect-all command.
What is the usage model without this flag?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 18/35] monitor: add "--persistent/-p" flag
Quoted 8 lines · Sagi Grimberg
Quoted 5 lines
Without this flag, AENs on created controllers won't be received. While --persistent is probably the behavior most users will want, it has to be explicitly activated, to be consistent with the connect-all command.
What is the usage model without this flag?
- listen to other events that would require a discovery, currently
simply fc_udev_device. In the future, network events could be added.
- listen for AENs on persistent discovery controllers that are created
by other means, e.g. manually.
See also our discussion about the cover letter of this series.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 19/35 fabrics: use "const char *" in struct config
In reply to [email protected]From: Martin Wilck <[email protected]>
This is easily done, and allows passing in some const char* pointers that would otherwise need to be strdup()d first.
constifying cfg->device requires changes in the ctrl_instance() function, because basename() can't be used.
--- fabrics.c | 23 ++++++++++++++--------- fabrics.h | 17 +++++++++-------- 2 files changed, 23 insertions(+), 17 deletions(-)
fabrics.c +14 -9
@@ -252,17 +252,22 @@ empty_field:return strdup ("\0" ); } -static int ctrl_instance (char *device) +int ctrl_instance (const char *device) {char d[64 ]; +const char *p;int ret, instance; - device =basename (device); - ret =sscanf (device,"nvme%d" , &instance); + p =strrchr (device,'/' ); +if (p ==NULL ) + p = device; +else + p++; + ret =sscanf (p,"nvme%d" , &instance);if (ret <=0 )return -EINVAL ;if (snprintf (d,sizeof (d),"nvme%d" , instance) <=0 || -strcmp (device, d)) +strcmp (p, d))return -EINVAL ;return instance; }
@@ -273,7 +278,7 @@ static int ctrl_instance(char *device)* given. * Return true/false based on whether it matches */ -static bool ctrl_matches_connectargs (char *name,struct connect_args *args) +static bool ctrl_matches_connectargs (const char *name,struct connect_args *args) {struct connect_args cargs;bool found = false;
@@ -815,7 +820,7 @@ add_int_argument(char **argstr, int *max_len, char *arg_str, int arg, }static int -add_argument (char **argstr,int *max_len,char *arg_str,char *arg) +add_argument (char **argstr,int *max_len,char *arg_str,const char *arg) {int len;
@@ -1541,7 +1546,7 @@ static int scan_sys_nvme_filter(const struct dirent *d)/* * Returns 1 if disconnect occurred, 0 otherwise. */ -static int disconnect_subsys (char *nqn,char *ctrl) +static int disconnect_subsys (const char *nqn,char *ctrl) {char *sysfs_nqn_path =NULL , *sysfs_del_path =NULL ;char subsysnqn[NVMF_NQN_SIZE ] = {};
@@ -1579,7 +1584,7 @@ static int disconnect_subsys(char *nqn, char *ctrl)/* * Returns the number of controllers successfully disconnected. */ -static int disconnect_by_nqn (char *nqn) +static int disconnect_by_nqn (const char *nqn) {struct dirent **devices =NULL ;int i, n, ret =0 ;
@@ -1601,7 +1606,7 @@ static int disconnect_by_nqn(char *nqn)return ret; } -static int disconnect_by_device (char *device) +static int disconnect_by_device (const char *device) {int instance;
fabrics.h +9 -8
@@ -13,13 +13,13 @@ extern int fabrics_disconnect_all(const char *desc, int argc, char **argv);/* Symbols used by monitor.c */ struct config { -char *nqn; -char *transport; -char *traddr; -char *trsvcid; -char *host_traddr; -char *hostnqn; -char *hostid; +const char *nqn; +const char *transport; +const char *traddr; +const char *trsvcid; +const char *host_traddr; +const char *hostnqn; +const char *hostid;int nr_io_queues;int nr_write_queues;int nr_poll_queues;
@@ -29,7 +29,7 @@ struct config {int ctrl_loss_tmo;int tos;char *raw; -char *device; +const char *device;int duplicate_connect;int disable_sqflow;int hdr_digest;
@@ -43,5 +43,6 @@ extern struct config cfg;int build_options (char *argstr,int max_len,bool discover);int do_discover (char *argstr,bool connect); +int ctrl_instance (const char *device);#endif
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 19/35] fabrics: use "const char *" in struct config
Quoted 126 lines · On 1/26/21 9:33 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
This is easily done, and allows passing in some const char* pointers that would otherwise need to be strdup()d first.
constifying
cfg->devicerequires changes in thectrl_instance()function, becausebasename()can't be used.Signed-off-by Martin Wilck <[email protected]>--- fabrics.c | 23 ++++++++++++++--------- fabrics.h | 17 +++++++++-------- 2 files changed, 23 insertions(+), 17 deletions(-)fabrics.c +14
-9@@ -252,17 +252,22 @@ empty_field:return strdup ("\0" ); } -static int ctrl_instance (char *device) +int ctrl_instance (const char *device) {char d[64 ]; +const char *p;int ret, instance; - device =basename (device); - ret =sscanf (device,"nvme%d" , &instance); + p =strrchr (device,'/' ); +if (p ==NULL ) + p = device; +else + p++; + ret =sscanf (p,"nvme%d" , &instance);if (ret <=0 )return -EINVAL ;if (snprintf (d,sizeof (d),"nvme%d" , instance) <=0 || -strcmp (device, d)) +strcmp (p, d))return -EINVAL ;return instance; }@@ -273,7 +278,7 @@ static int ctrl_instance(char *device)* given. * Return true/false based on whether it matches */ -static bool ctrl_matches_connectargs (char *name,struct connect_args *args) +static bool ctrl_matches_connectargs (const char *name,struct connect_args *args) {struct connect_args cargs;bool found = false;@@ -815,7 +820,7 @@ add_int_argument(char **argstr, int *max_len, char *arg_str, int arg, }static int -add_argument (char **argstr,int *max_len,char *arg_str,char *arg) +add_argument (char **argstr,int *max_len,char *arg_str,const char *arg) {int len;@@ -1541,7 +1546,7 @@ static int scan_sys_nvme_filter(const struct dirent *d)/* * Returns 1 if disconnect occurred, 0 otherwise. */ -static int disconnect_subsys (char *nqn,char *ctrl) +static int disconnect_subsys (const char *nqn,char *ctrl) {char *sysfs_nqn_path =NULL , *sysfs_del_path =NULL ;char subsysnqn[NVMF_NQN_SIZE ] = {};@@ -1579,7 +1584,7 @@ static int disconnect_subsys(char *nqn, char *ctrl)/* * Returns the number of controllers successfully disconnected. */ -static int disconnect_by_nqn (char *nqn) +static int disconnect_by_nqn (const char *nqn) {struct dirent **devices =NULL ;int i, n, ret =0 ;@@ -1601,7 +1606,7 @@ static int disconnect_by_nqn(char *nqn)return ret; } -static int disconnect_by_device (char *device) +static int disconnect_by_device (const char *device) {int instance;fabrics.h +8
-8@@ -13,13 +13,13 @@ extern int fabrics_disconnect_all(const char *desc, int argc, char **argv);/* Symbols used by monitor.c */ struct config { -char *nqn; -char *transport; -char *traddr; -char *trsvcid; -char *host_traddr; -char *hostnqn; -char *hostid; +const char *nqn; +const char *transport; +const char *traddr; +const char *trsvcid; +const char *host_traddr; +const char *hostnqn; +const char *hostid;int nr_io_queues;int nr_write_queues;int nr_poll_queues;@@ -29,7 +29,7 @@ struct config {int ctrl_loss_tmo;int tos;char *raw; -char *device; +const char *device;int duplicate_connect;int disable_sqflow;int hdr_digest;
fabrics.h extern struct config cfg; · line 48
Quoted 6 lines
fabrics.h +1
-0@@ -43,5 +43,6 @@ extern struct config cfg;int build_options (char *argstr,int max_len,bool discover);int do_discover (char *argstr,bool connect); +int ctrl_instance (const char *device);#endif
This really is independent on the monitor code, and should be moved to the beginning of the patch series.
Other than that:
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 20/35 fabrics: export arg_str(), parse_conn_arg(), and remove_ctrl()
In reply to [email protected]From: Martin Wilck <[email protected]>
These functions are used by the monitor functionality that will be added in follow-up commits.
--- fabrics.c | 6 +++--- fabrics.h | 4 ++++ 2 files changed, 7 insertions(+), 3 deletions(-)
fabrics.c +3 -3
@@ -102,7 +102,7 @@ static const match_table_t opt_tokens = { { OPT_ERR, NULL }, }; -static const char *arg_str (const char *const *strings, +const char *arg_str (const char *const *strings,size_t array_size,size_t idx) {if (idx < array_size && strings[idx])
@@ -208,7 +208,7 @@ int do_discover(char *argstr, bool connect);* If field found, return string containing field value. If field * not found, return an empty string. */ -static char *parse_conn_arg (char *conargs,const char delim,const char *field) +char *parse_conn_arg (char *conargs,const char delim,const char *field) {char *s, *e;size_t cnt;
@@ -476,7 +476,7 @@ out:return ret; } -static int remove_ctrl (int instance) +int remove_ctrl (int instance) {char *sysfs_path;int ret;
fabrics.h +4 -0
@@ -12,6 +12,8 @@ extern int fabrics_disconnect_all(const char *desc, int argc, char **argv);/* Symbols used by monitor.c */ +const char *arg_str (const char *const *strings,size_t array_size,size_t idx); +struct config {const char *nqn;const char *transport;
@@ -44,5 +46,7 @@ extern struct config cfg;int build_options (char *argstr,int max_len,bool discover);int do_discover (char *argstr,bool connect);int ctrl_instance (const char *device); +char *parse_conn_arg (char *conargs,const char delim,const char *field); +int remove_ctrl (int instance);#endif
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 21/35 nvme-cli: add "list.h"
In reply to [email protected]From: Martin Wilck <[email protected]>
Adding "list.h" from multipath-tools, which got it from the kernel long ago.
--- list.h | 365 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 365 insertions(+) create mode 100644 list.h
list.h new file +365 -0
@@ -0,0 +1,365 @@ +/* +* Copied from the Linux kernel source tree, version 2.6.0-test1. +* +* Licensed under the GPL v2 as per the whole kernel source tree. +* +*/ + +#ifndef _LIST_H +#define _LIST_H + +#include <stddef.h> + +/** +* container_of - cast a member of a structure out to the containing structure +* +* @ptr: the pointer to the member. +* @type: the type of the container struct this is embedded in. +* @member: the name of the member within the struct. +* +*/ +#define container_of_const (ptr, type, member) ({ \ + const typeof( ((type *)0)->member ) *__mptr = (ptr); \ + (const type *)( (const char *)__mptr - offsetof(type,member) );}) + +#define container_of (ptr, type, member) ({ \ + typeof( ((type *)0)->member ) *__mptr = (ptr); \ + (type *)( (char *)__mptr - offsetof(type,member) );}) + +/* +* These are non-NULL pointers that will result in page faults +* under normal circumstances, used to verify that nobody uses +* non-initialized list entries. +*/ +#define LIST_POISON1 ((void *) 0x00100100) +#define LIST_POISON2 ((void *) 0x00200200) + +/* +* Simple doubly linked list implementation. +* +* Some of the internal functions ("__xxx") are useful when +* manipulating whole lists rather than single entries, as +* sometimes we already know the next/prev entries and we can +* generate better code by using them directly rather than +* using the generic single-entry routines. +*/ + +struct list_head { +struct list_head *next, *prev; +}; + +#define LIST_HEAD_INIT (name) { &(name), &(name) } + +#define LIST_HEAD (name) \ + struct list_head name = LIST_HEAD_INIT(name) + +#define INIT_LIST_HEAD (ptr) do { \ + (ptr)->next = (ptr); (ptr)->prev = (ptr); \ +} while (0) + +/* +* Insert a new entry between two known consecutive entries. +* +* This is only for internal list manipulation where we know +* the prev/next entries already! +*/ +static inline void __list_add (struct list_head *new, +struct list_head *prev, +struct list_head *next) +{ + next->prev = new; + new->next = next; + new->prev = prev; + prev->next = new; +} + +/** +* list_add - add a new entry +* @new: new entry to be added +* @head: list head to add it after +* +* Insert a new entry after the specified head. +* This is good for implementing stacks. +*/ +static inline void list_add (struct list_head *new,struct list_head *head) +{ +__list_add (new, head, head->next); +} + +/** +* list_add_tail - add a new entry +* @new: new entry to be added +* @head: list head to add it before +* +* Insert a new entry before the specified head. +* This is useful for implementing queues. +*/ +static inline void list_add_tail (struct list_head *new,struct list_head *head) +{ +__list_add (new, head->prev, head); +} + +/* +* Delete a list entry by making the prev/next entries +* point to each other. +* +* This is only for internal list manipulation where we know +* the prev/next entries already! +*/ +static inline void __list_del (struct list_head * prev,struct list_head * next) +{ + next->prev = prev; + prev->next = next; +} + +/** +* list_del - deletes entry from list. +* @entry: the element to delete from the list. +* Note: list_empty on entry does not return true after this, the entry is +* in an undefined state. +*/ +static inline void list_del (struct list_head *entry) +{ +__list_del (entry->prev, entry->next); + entry->next =LIST_POISON1 ; + entry->prev =LIST_POISON2 ; +} + +/** +* list_del_init - deletes entry from list and reinitialize it. +* @entry: the element to delete from the list. +*/ +static inline void list_del_init (struct list_head *entry) +{ +__list_del (entry->prev, entry->next); +INIT_LIST_HEAD (entry); +} + +/** +* list_move - delete from one list and add as another's head +* @list: the entry to move +* @head: the head that will precede our entry +*/ +static inline void list_move (struct list_head *list,struct list_head *head) +{ +__list_del (list->prev, list->next); +list_add (list, head); +} + +/** +* list_move_tail - delete from one list and add as another's tail +* @list: the entry to move +* @head: the head that will follow our entry +*/ +static inline void list_move_tail (struct list_head *list, +struct list_head *head) +{ +__list_del (list->prev, list->next); +list_add_tail (list, head); +} + +/** +* list_empty - tests whether a list is empty +* @head: the list to test. +*/ +static inline int list_empty (struct list_head *head) +{ +return head->next == head; +} + +static inline void __list_splice (const struct list_head *list, +struct list_head *prev, +struct list_head *next) +{ +struct list_head *first = list->next; +struct list_head *last = list->prev; + + first->prev = prev; + prev->next = first; + + last->next = next; + next->prev = last; +} + +/** +* list_splice - join two lists +* @list: the new list to add. +* @head: the place to add it in the first list. +*/ +static inline void list_splice (struct list_head *list,struct list_head *head) +{ +if (!list_empty (list)) +__list_splice (list, head, head->next); +} + +/** +* list_splice_tail - join two lists, each list being a queue +* @list: the new list to add. +* @head: the place to add it in the first list. +*/ +static inline void list_splice_tail (struct list_head *list, +struct list_head *head) +{ +if (!list_empty (list)) +__list_splice (list, head->prev, head); +} + +/** +* list_splice_init - join two lists and reinitialise the emptied list. +* @list: the new list to add. +* @head: the place to add it in the first list. +* +* The list at @list is reinitialised +*/ +static inline void list_splice_init (struct list_head *list, +struct list_head *head) +{ +if (!list_empty (list)) { +__list_splice (list, head, head->next); +INIT_LIST_HEAD (list); + } +} + +/** +* list_splice_tail_init - join two lists and reinitialise the emptied list +* @list: the new list to add. +* @head: the place to add it in the first list. +* +* Each of the lists is a queue. +* The list at @list is reinitialised +*/ +static inline void list_splice_tail_init (struct list_head *list, +struct list_head *head) +{ +if (!list_empty (list)) { +__list_splice (list, head->prev, head); +INIT_LIST_HEAD (list); + } +} + +/** +* list_entry - get the struct for this entry +* @ptr: the &struct list_head pointer. +* @type: the type of the struct this is embedded in. +* @member: the name of the list_struct within the struct. +*/ +#define list_entry (ptr, type, member) \ + container_of(ptr, type, member) + +/** +* list_for_each - iterate over a list +* @pos: the &struct list_head to use as a loop counter. +* @head: the head for your list. +*/ +#define list_for_each (pos, head) \ + for (pos = (head)->next; pos != (head); \ + pos = pos->next) + +/** +* __list_for_each - iterate over a list +* @pos: the &struct list_head to use as a loop counter. +* @head: the head for your list. +* +* This variant differs from list_for_each() in that it's the +* simplest possible list iteration code. +* Use this for code that knows the list to be very short (empty +* or 1 entry) most of the time. +*/ +#define __list_for_each (pos, head) \ + for (pos = (head)->next; pos != (head); pos = pos->next) + +/** +* list_for_each_prev - iterate over a list backwards +* @pos: the &struct list_head to use as a loop counter. +* @head: the head for your list. +*/ +#define list_for_each_prev (pos, head) \ + for (pos = (head)->prev; pos != (head); pos = pos->prev) + +/** +* list_for_each_safe - iterate over a list safe against removal of list entry +* @pos: the &struct list_head to use as a loop counter. +* @n: another &struct list_head to use as temporary storage +* @head: the head for your list. +*/ +#define list_for_each_safe (pos, n, head) \ + for (pos = (head)->next, n = pos->next; pos != (head); \ + pos = n, n = pos->next) + +/** +* list_for_each_entry - iterate over list of given type +* @pos: the type * to use as a loop counter. +* @head: the head for your list. +* @member: the name of the list_struct within the struct. +*/ +#define list_for_each_entry (pos, head, member) \ + for (pos = list_entry((head)->next, typeof(*pos), member); \ + &pos->member != (head); \ + pos = list_entry(pos->member.next, typeof(*pos), member)) + +/** +* list_for_each_entry_reverse - iterate backwards over list of given type. +* @pos: the type * to use as a loop counter. +* @head: the head for your list. +* @member: the name of the list_struct within the struct. +*/ +#define list_for_each_entry_reverse (pos, head, member) \ + for (pos = list_entry((head)->prev, typeof(*pos), member); \ + &pos->member != (head); \ + pos = list_entry(pos->member.prev, typeof(*pos), member)) + +/** +* list_for_each_entry_safe - iterate over list of given type safe against removal of list entry +* @pos: the type * to use as a loop counter. +* @n: another type * to use as temporary storage +* @head: the head for your list. +* @member: the name of the list_struct within the struct. +*/ +#define list_for_each_entry_safe (pos, n, head, member) \ + for (pos = list_entry((head)->next, typeof(*pos), member), \ + n = list_entry(pos->member.next, typeof(*pos), member); \ + &pos->member != (head); \ + pos = n, n = list_entry(n->member.next, typeof(*n), member)) + +/** +* list_for_each_entry_reverse_safe - iterate backwards over list of given type safe against removal of list entry +* @pos: the type * to use as a loop counter. +* @n: another type * to use as temporary storage +* @head: the head for your list. +* @member: the name of the list_struct within the struct. +*/ +#define list_for_each_entry_reverse_safe (pos, n, head, member) \ + for (pos = list_entry((head)->prev, typeof(*pos), member), \ + n = list_entry(pos->member.prev, typeof(*pos), member);\ + &pos->member != (head); \ + pos = n, n = list_entry(n->member.prev, typeof(*n), member)) + +/** +* list_for_some_entry_safe - iterate list from the given begin node to the given end node safe against removal of list entry +* @pos: the type * to use as a loop counter. +* @n: another type * to use as temporary storage +* @from: the begin node of the iteration. +* @to: the end node of the iteration. +* @member: the name of the list_struct within the struct. +*/ +#define list_for_some_entry_safe (pos, n, from, to, member) \ + for (pos = list_entry((from)->next, typeof(*pos), member), \ + n = list_entry(pos->member.next, typeof(*pos), member); \ + &pos->member != (to); \ + pos = n, n = list_entry(n->member.next, typeof(*n), member)) + +/** +* list_for_some_entry_reverse_safe - iterate backwards list from the given begin node to the given end node safe against removal of list entry +* @pos: the type * to use as a loop counter. +* @n: another type * to use as temporary storage +* @from: the begin node of the iteration. +* @to: the end node of the iteration. +* @member: the name of the list_struct within the struct. +*/ +#define list_for_some_entry_reverse_safe (pos, n, from, to, member) \ + for (pos = list_entry((from)->prev, typeof(*pos), member), \ + n = list_entry(pos->member.prev, typeof(*pos), member); \ + &pos->member != (to); \ + pos = n, n = list_entry(n->member.prev, typeof(*n), member)) + +#endif /* _LIST_H */
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 22/35 conn-db: add simple connection registry
In reply to [email protected]From: Martin Wilck <[email protected]>
The monitor works best if it maintains a discovery controller connection to every transport address that provides a discovery subsystem.
While controllers are easily tracked in sysfs, addresses ("connections"), i.e. (transport, traddr, trsvid, host_traddr) tuples, are not. Create a simple registry that tracks the state of "connections" and their associated discovery controllers.
A detailed description of the API is provided in the header file conn-db.h.
--- Makefile | 2 +- conn-db.c | 340 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ conn-db.h | 140 ++++++++++++++++++++++ 3 files changed, 481 insertions(+), 1 deletion(-) create mode 100644 conn-db.c create mode 100644 conn-db.h
Makefile +1 -1
@@ -69,7 +69,7 @@ OBJS := nvme-print.o nvme-ioctl.o nvme-rpmb.o \ nvme-status.o nvme-filters.o nvme-topology.o ifeq ($(HAVE_LIBUDEV),0) - OBJS += monitor.o + OBJS += monitor.o conn-db.o endif UTIL_OBJS := util/argconfig.o util/suffix.o util/json.o util/parser.o
conn-db.c new file +340 -0
340 lines not on this page · raw message
conn-db.h new file +140 -0
140 lines not on this page · raw message
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 22/35] conn-db: add simple connection registry
Quoted 6 lines
From: Martin Wilck <[email protected]>
The monitor works best if it maintains a discovery controller connection to every transport address that provides a discovery subsystem.
Quoted 6 lines
While controllers are easily tracked in sysfs, addresses ("connections"), i.e. (transport, traddr, trsvid,
host_traddr) tuples, are not. Create a simple registry that tracks the state of "connections" and their associated discovery controllers.A detailed description of the API is provided in the header file
conn-db.h.
If this is more flexible, perhaps also the main fabrics discover/connect/connect-all can be converted to use it as well.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 22/35] conn-db: add simple connection registry
Quoted 14 lines · Sagi Grimberg
Quoted 14 lines
From: Martin Wilck <[email protected]>
The monitor works best if it maintains a discovery controller connection
to every transport address that provides a discovery subsystem.While controllers are easily tracked in sysfs, addresses ("connections"),
i.e. (transport, traddr, trsvid,host_traddr) tuples, are not. Create
a simple registry that tracks the state of "connections" and their associated discovery controllers.
Quoted 6 lines
Quoted 2 lines
A detailed description of the API is provided in the header file
conn-db.h.If this is more flexible, perhaps also the main fabrics discover/connect/connect-all can be converted to use it as well.
Yes. Could we do that in a separate step later.
Regards,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 23/35 monitor: handle restart of pending discoveries
In reply to [email protected]From: Martin Wilck <[email protected]>
If a new discovery event is received while a discovery task is running, no new task should be started in parallel to the previous one. Rather, we should note that another event has happened, and restart the discovery after the current task has finished.
There is no need to remember more than one pending discovery.
--- monitor.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 49 insertions(+), 9 deletions(-)
monitor.c +49 -9
@@ -36,6 +36,7 @@#include "util/argconfig.h" #include "fabrics.h" #include "monitor.h" +#include "conn-db.h" #define LOG_FUNCNAME 1#include "log.h"
@@ -199,12 +200,22 @@ static int monitor_get_fc_uev_props(struct udev_device *ud,return 0 ; } -static int monitor_discovery (char *transport,char *traddr,char *trsvcid, -char *host_traddr) +static int monitor_discovery (const char *transport,const char *traddr, +const char *trsvcid,const char *host_traddr) {char argstr[BUF_SIZE ];pid_t pid; -int rc; +int rc, db_rc; +struct nvme_connection *co =NULL ; + + db_rc =conndb_add (transport, traddr, trsvcid, host_traddr, &co); +if (db_rc !=0 && db_rc != -EEXIST ) +return db_rc; + +if (co->status ==CS_DISC_RUNNING ) { + co->discovery_pending =1 ; +return -EAGAIN ; + } pid =fork ();if (pid ==-1 ) {
@@ -212,17 +223,26 @@ static int monitor_discovery(char *transport, char *traddr, char *trsvcid,return -errno; }else if (pid >0 ) {log (LOG_DEBUG ,"started discovery task %ld\n" , (long )pid); + + co->discovery_pending =0 ; + co->status =CS_DISC_RUNNING ; + co->discovery_task = pid; +return 0 ; }child_reset_signals (); -log (LOG_NOTICE ,"starting %s discovery for %s==>%s(%s)\n" , - transport, host_traddr, traddr, trsvcid ? trsvcid :"none" ); +log (LOG_NOTICE ,"starting discovery for <%s>: %s ==> %s(%s) in state %s\n" , + transport, host_traddr, traddr, + trsvcid && *trsvcid ? trsvcid :"none" , +conn_status_str (co->status)); + + cfg.nqn =NVME_DISC_SUBSYS_NAME ; cfg.transport = transport; cfg.traddr = traddr; - cfg.trsvcid = trsvcid; + cfg.trsvcid = trsvcid && *trsvcid ? trsvcid :NULL ; cfg.host_traddr = host_traddr;/* Without the following, the kernel returns EINVAL */ cfg.tos =-1 ;
@@ -351,6 +371,7 @@ static void monitor_handle_uevents(struct udev_monitor *monitor)static void handle_sigchld (void ) {while (true) { +struct nvme_connection *co;int wstatus;pid_t pid;
@@ -365,14 +386,33 @@ static void handle_sigchld(void)default :break ; } -if (!WIFEXITED (wstatus)) + co =conndb_find_by_pid (pid); +if (!co) { +log (LOG_ERR ,"no connection found for discovery task %ld\n" , + (long )pid); +continue ; + } +if (!WIFEXITED (wstatus)) {log (LOG_WARNING ,"child %ld didn't exit normally\n" , (long )pid); -else if (WEXITSTATUS (wstatus) !=0 ) + co->status =CS_FAILED ; + }else if (WEXITSTATUS (wstatus) !=0 ) {log (LOG_NOTICE ,"child %ld exited with status \"%s\"\n" , (long )pid,strerror (WEXITSTATUS (wstatus))); -else + co->status =CS_FAILED ; + co->did_discovery =1 ; + co->discovery_result =WEXITSTATUS (wstatus); + }else {log (LOG_DEBUG ,"child %ld exited normally\n" , (long )pid); + co->status =CS_ONLINE ; + co->successful_discovery = co->did_discovery =1 ; + co->discovery_result =0 ; + } +if (co->discovery_pending) { +log (LOG_NOTICE ,"new discovery pending - restarting\n" ); +monitor_discovery (co->transport, co->traddr, + co->trsvcid, co->host_traddr); + } }; }
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 24/35 monitor: monitor_discovery(): try to reuse existing controllers
In reply to [email protected]From: Martin Wilck <[email protected]>
We need to pass cfg.device to do_discovery() if we want to re-use the discovery controller device. Remember the instance number when handling an AEN, and try to reuse this instance if the discovery needs to be repeated.
--- monitor.c | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-)
monitor.c +31 -5
@@ -201,12 +201,14 @@ static int monitor_get_fc_uev_props(struct udev_device *ud, }static int monitor_discovery (const char *transport,const char *traddr, -const char *trsvcid,const char *host_traddr) +const char *trsvcid,const char *host_traddr, +const char *devname) {char argstr[BUF_SIZE ];pid_t pid;int rc, db_rc;struct nvme_connection *co =NULL ; +char *device =NULL ; db_rc =conndb_add (transport, traddr, trsvcid, host_traddr, &co);if (db_rc !=0 && db_rc != -EEXIST )
@@ -227,7 +229,15 @@ static int monitor_discovery(const char *transport, const char *traddr, co->discovery_pending =0 ; co->status =CS_DISC_RUNNING ; co->discovery_task = pid; +if (devname) { +int instance =ctrl_instance (devname); +if (instance <0 ) { +log (LOG_ERR ,"unexpected devname: %s\n" , + devname); + }else + co->discovery_instance = instance; + }return 0 ; }
@@ -238,12 +248,26 @@ static int monitor_discovery(const char *transport, const char *traddr, trsvcid && *trsvcid ? trsvcid :"none" , conn_status_str(co->status)); +/* +* Try to re-use existing controller. do_discovery() will check +* if it matches the connection parameters. +*/ +if (!devname && co->discovery_instance >=0 ) { +if (asprintf (&device,"nvme%d" , co->discovery_instance) ==-1 ) + device =NULL ; +else + devname = device; + } + +if (devname) +log (LOG_INFO ,"using discovery controller %s\n" , devname); cfg.nqn =NVME_DISC_SUBSYS_NAME ; cfg.transport = transport; cfg.traddr = traddr; cfg.trsvcid = trsvcid && *trsvcid ? trsvcid :NULL ; - cfg.host_traddr = host_traddr; + cfg.host_traddr = host_traddr && *host_traddr ? host_traddr :NULL ; + cfg.device = devname;/* Without the following, the kernel returns EINVAL */ cfg.tos =-1 ;
@@ -251,6 +275,7 @@ static int monitor_discovery(const char *transport, const char *traddr,log (LOG_DEBUG ,"%s\n" , argstr); rc =do_discover (argstr, mon_cfg.autoconnect); +free (device);exit (-rc);/* not reached */ return rc;
@@ -269,7 +294,7 @@ static void monitor_handle_fc_uev(struct udev_device *ud) host_traddr, sizeof(host_traddr)))return ; -monitor_discovery ("fc" , traddr,NULL , host_traddr); +monitor_discovery ("fc" , traddr,NULL , host_traddr,NULL ); } static int monitor_get_nvme_uev_props(struct udev_device *ud,
@@ -337,7 +362,8 @@ static void monitor_handle_nvme_uev(struct udev_device *ud)return ;monitor_discovery (transport, traddr, -strcmp (trsvcid,"none" ) ? trsvcid :NULL , host_traddr); +strcmp (trsvcid,"none" ) ? trsvcid :NULL , host_traddr, +udev_device_get_sysname (ud)); }static void monitor_handle_udevice (struct udev_device *ud)
@@ -411,7 +437,7 @@ static void handle_sigchld(void)if (co->discovery_pending) {log (LOG_NOTICE ,"new discovery pending - restarting\n" );monitor_discovery (co->transport, co->traddr, - co->trsvcid, co->host_traddr); + co->trsvcid, co->host_traddr,NULL ); } }; }
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 25/35 monitor: read existing connections on startup
In reply to [email protected]From: Martin Wilck <[email protected]>
Add discovery_ctrl_existed flag to the connection, to determine whether a given (discovery) controller should be shut down on exit.
--- conn-db.c | 1 + conn-db.h | 1 + monitor.c | 2 ++ 3 files changed, 4 insertions(+)
conn-db.c +1 -0
@@ -301,6 +301,7 @@ int conndb_init_from_sysfs(void)if (instance >=0 ) { ci->c.discovery_instance = instance; + ci->c.discovery_ctrl_existed =1 ;log (LOG_DEBUG ,"found discovery controller %s\n" , devices[i]->d_name); }
conn-db.h +1 -0
@@ -11,6 +11,7 @@ struct nvme_connection { int discovery_pending:1; int did_discovery:1; int successful_discovery:1; + int discovery_ctrl_existed:1;union {pid_t discovery_task;int discovery_result;
monitor.c +2 -0
@@ -603,9 +603,11 @@ int aen_monitor(const char *desc, int argc, char **argv) } ret =create_udev_monitor (&monitor);if (ret ==0 ) { +conndb_init_from_sysfs (); ret =monitor_main_loop (monitor);udev_monitor_unref (monitor); } +conndb_free (); udev =udev_unref (udev);if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)monitor_enable_udev_rules ();
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 26/35 monitor: implement starting discovery controllers on startup
In reply to [email protected]From: Martin Wilck <[email protected]>
If the -U/--startup option is used, nvme monitor looks for exsiting nvme controllers on startup, and runs a discovery on the associated transport address (connection); i.e. tries to connect to a discovery subsystem on the same address, retrieve the discovery log pages, and (if --autoconnect is given) connect all listed controllers.
It depends on the --persistent option whether the discovery controllers created in this process will be persistent after startup.
--- monitor.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
monitor.c +18 -0
@@ -43,6 +43,7 @@static struct monitor_config {bool autoconnect;bool skip_udev_on_exit; +bool start_ctrls; } mon_cfg;static struct udev *udev;
@@ -544,6 +545,20 @@ static void monitor_enable_udev_rules(void)log (LOG_INFO ,"removed %s\n" , autoconnect_rules); } +static int monitor_setup_discovery_ctrl (struct nvme_connection *co, +void *arg __attribute__((unused))) +{ +int rc; + +if (co->discovery_instance >=0 ) +return CD_CB_OK ; + + rc =monitor_discovery (co->transport, co->traddr, co->trsvcid, + co->host_traddr,NULL ); + +return rc ?CD_CB_ERR :CD_CB_OK ; +} +static int monitor_parse_opts (const char *desc,int argc,char **argv) {bool quiet = false;
@@ -553,6 +568,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)OPT_ARGS (opts) = {OPT_FLAG ("autoconnect" ,'A' , &mon_cfg.autoconnect,"automatically connect newly discovered controllers" ), +OPT_FLAG ("startup" ,'U' , &mon_cfg.start_ctrls,"start discovery controllers on startup" ),OPT_FLAG ("persistent" ,'p' , &cfg.persistent,"persistent discovery connections" ),OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ),OPT_FLAG ("verbose" ,'v' , &verbose,"log level: verbose" ),
@@ -604,6 +620,8 @@ int aen_monitor(const char *desc, int argc, char **argv) ret =create_udev_monitor (&monitor);if (ret ==0 ) {conndb_init_from_sysfs (); +if (mon_cfg.start_ctrls) +conndb_for_each (monitor_setup_discovery_ctrl,NULL ); ret =monitor_main_loop (monitor);udev_monitor_unref (monitor); }
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 26/35] monitor: implement starting discovery controllers on startup
Quoted 6 lines
If the -U/--startup option is used, nvme monitor looks for exsiting nvme controllers on startup, and runs a discovery on the associated transport address (connection); i.e. tries to connect to a discovery subsystem on the same address, retrieve the discovery log pages, and (if --autoconnect is given) connect all listed controllers.
Oh no no no no... This is a completely wrong assumption to make. Nothing suggest that an nvme controller will have a discovery controller on the same port.
There is a discovery controller and an NVM controller, the only connection between the two is that a discovery controller can refer to an nvme controller (or a different discovery controller).
Information about discovery controllers may be obtained in different ways, but it absolutely cannot be obtained by any nvme controller.
Now I understand a bit more on the --keep discussion. The monitor should today get discovery controller endpoints from a conf file (maybe discovery.conf) and in the future by any other automatic way, not even looking into the currently connected nvme controllers.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 26/35] monitor: implement starting discovery controllers on startup
Quoted 18 lines · Sagi Grimberg
Quoted 7 lines
If the -U/--startup option is used, nvme monitor looks for exsiting nvme controllers on startup, and runs a discovery on the associated transport address (connection); i.e. tries to connect to a discovery
subsystem on the same address, retrieve the discovery log pages, and (if --autoconnect is given) connect all listed controllers.Oh no no no no... This is a completely wrong assumption to make. Nothing
suggest that an nvme controller will have a discovery controller on the
same port.There is a discovery controller and an NVM controller, the only connection between the two is that a discovery controller can refer to an nvme controller (or a different discovery controller).
Quoted 8 lines
Information about discovery controllers may be obtained in different ways, but it absolutely cannot be obtained by any nvme controller.
Now I understand a bit more on the --keep discussion. The monitor should today get discovery controller endpoints from a conf file (maybe discovery.conf) and in the future by any other automatic way, not even looking into the currently connected nvme controllers.
I hear you. I'll drop this in the next version, no problem.
I didn't expect that probing for a discovery controller was a problem. IIUC, if we get an fc_udev_device event for a NVMe remote port, we can't be sure that that remote port offers a discovery controller, either. We try to connect, and give up if it fails.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 26/35] monitor: implement starting discovery controllers on startup
Quoted 27 lines
Quoted 18 lines
Quoted 7 lines
If the -U/--startup option is used, nvme monitor looks for exsiting nvme controllers on startup, and runs a discovery on the associated transport address (connection); i.e. tries to connect to a discovery
subsystem on the same address, retrieve the discovery log pages, and (if --autoconnect is given) connect all listed controllers.Oh no no no no... This is a completely wrong assumption to make. Nothing
suggest that an nvme controller will have a discovery controller on the
same port.There is a discovery controller and an NVM controller, the only connection between the two is that a discovery controller can refer to an nvme controller (or a different discovery controller).
Quoted 8 lines
Information about discovery controllers may be obtained in different ways, but it absolutely cannot be obtained by any nvme controller.
Now I understand a bit more on the --keep discussion. The monitor should today get discovery controller endpoints from a conf file (maybe discovery.conf) and in the future by any other automatic way, not even looking into the currently connected nvme controllers.
Quoted 3 lines
I hear you. I'll drop this in the next version, no problem.
I didn't expect that probing for a discovery controller was a problem.
It's not a "problem" in the sense that nothing devastating will happen. it's just wrong. One can also randomize a transport endpoint and attempt to connect to it.
Quoted 3 lines
IIUC, if we get an
fc_udev_deviceevent for a NVMe remote port, we can't be sure that that remote port offers a discovery controller, either. We try to connect, and give up if it fails.
I'm assuming that this is what the spec mandates the host should do, which is not the case here.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 27/35 monitor: implement cleanup of created discovery controllers
In reply to [email protected]From: Martin Wilck <[email protected]>
If -A or -U was in use, try to clean up discovery controllers we set up during runtime. Discovery connections that were already up when the monitor started will not be teared down.
--- monitor.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
monitor.c +35 -0
@@ -559,6 +559,39 @@ static int monitor_setup_discovery_ctrl(struct nvme_connection *co,return rc ?CD_CB_ERR :CD_CB_OK ; } +static int monitor_remove_discovery_ctrl (struct nvme_connection *co, +void *arg __attribute__((unused))) +{ +char syspath[PATH_MAX ]; +int len; +CLEANUP (char, subsysnqn) =NULL ; + +if (co->discovery_instance ==-1 || co->discovery_ctrl_existed) +return CD_CB_OK ; + + len =snprintf (syspath,sizeof (syspath),SYS_NVME "/nvme%d" , + co->discovery_instance); +if (len <0 || len >=sizeof (syspath)) +return CD_CB_ERR ; + + subsysnqn =nvme_get_ctrl_attr (syspath,"subsysnqn" ); +if (subsysnqn && !strcmp (subsysnqn,NVME_DISC_SUBSYS_NAME )) { +if (remove_ctrl (co->discovery_instance)) { +log (LOG_ERR , +"failed to remove discovery controller /dev/nvme%d: %m\n" , + co->discovery_instance); +return CD_CB_ERR ; + }else +log (LOG_INFO , +"removed discovery controller /dev/nvme%d\n" , + co->discovery_instance); + }else +log (LOG_WARNING , +"unexpected NQN %s on /dev/nvme%d, not removing controller\n" , + subsysnqn, co->discovery_instance); +return CD_CB_OK ; +} +static int monitor_parse_opts (const char *desc,int argc,char **argv) {bool quiet = false;
@@ -625,6 +658,8 @@ int aen_monitor(const char *desc, int argc, char **argv) ret =monitor_main_loop (monitor);udev_monitor_unref (monitor); } +if (mon_cfg.autoconnect || mon_cfg.start_ctrls) +conndb_for_each (monitor_remove_discovery_ctrl,NULL );conndb_free (); udev =udev_unref (udev);if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 28/35 monitor: basic handling of add/remove uevents for nvme controllers
In reply to [email protected]From: Martin Wilck <[email protected]>
On add, we'd like to detect the creation of a discovery controller for a given connection. More often than not, that doesn't work, because the controller is still in "connecting" state when the uevent arrives, and reading subsysnqn yields "(efault)".
This would need to be fixed by waiting for the connection to complete, but doing this properly is currently out of scope. It's not really a big problem, because without --persistent, the discovery controller will be removed anyway, and with --persistent, we'll notice the discovery controller as soon as it actually receives an AEN and generates an uevent.
--- monitor.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
monitor.c +38 -0
@@ -155,6 +155,36 @@ static int child_reset_signals(void)return -err; } +static void monitor_handle_nvme_add (struct udev_device *ud) +{ +const char *syspath =udev_device_get_syspath (ud); +CLEANUP (char, subsysnqn) =NULL ; +CLEANUP (char, state) =NULL ; + +if (!syspath) +return ; + subsysnqn =nvme_get_ctrl_attr (syspath,"subsysnqn" ); + state =nvme_get_ctrl_attr (syspath,"state" ); +log (LOG_DEBUG ,"add %s => %s [%s]\n" , syspath, subsysnqn, state); +} + +static void monitor_handle_nvme_remove (struct udev_device *ud) +{ +const char *sysname =udev_device_get_sysname (ud); +struct nvme_connection *co; + +if (!sysname) +return ; + + co =conndb_find_by_ctrl (sysname); +if (co) { +log (LOG_DEBUG ,"%s: connection discovery controller removed\n" , + sysname); + co->discovery_instance =-1 ; + } +return ; +} +static int monitor_get_fc_uev_props (struct udev_device *ud,char *traddr,size_t tra_sz,char *host_traddr,size_t htra_sz)
@@ -353,6 +383,14 @@ static void monitor_handle_nvme_uev(struct udev_device *ud)char traddr[NVMF_TRADDR_SIZE ], host_traddr[NVMF_TRADDR_SIZE ];char trsvcid[NVMF_TRSVCID_SIZE ], transport[5 ]; +if (!strcmp (udev_device_get_action (ud),"remove" )) { +monitor_handle_nvme_remove (ud); +return ; + } +if (!strcmp (udev_device_get_action (ud),"add" )) { +monitor_handle_nvme_add (ud); +return ; + }if (strcmp (udev_device_get_action (ud),"change" ))return ;
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 29/35 monitor: kill running discovery tasks on exit
In reply to [email protected]From: Martin Wilck <[email protected]>
Make sure discovery tasks terminate when the main process terminates.
--- monitor.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
monitor.c +31 -0
@@ -630,6 +630,34 @@ static int monitor_remove_discovery_ctrl(struct nvme_connection *co,return CD_CB_OK ; } +static int monitor_kill_discovery_task (struct nvme_connection *co, +void *arg __attribute__((unused))) +{ +int wstatus; +pid_t pid, wpid =-1 ; + +if (co->status !=CS_DISC_RUNNING ) +return CD_CB_OK ; + + pid = co->discovery_task; + co->status =CS_FAILED ; +if (kill (co->discovery_task,SIGTERM ) ==-1 ) { +log (LOG_ERR ,"failed to send SIGTERM to pid %ld: %m\n" , + (long )pid); + wpid =waitpid (pid, &wstatus,WNOHANG ); + }else { +log (LOG_DEBUG ,"sent SIGTERM to pid %ld, waiting\n" , (long )pid); + wpid =waitpid (pid, &wstatus,0 ); + } +if (wpid != pid) { +log (LOG_ERR ,"failed to wait for %ld: %m\n" , (long )pid); +return CD_CB_ERR ; + }else { +log (LOG_DEBUG ,"child %ld terminated\n" , (long )pid); +return CD_CB_OK ; + } +} +static int monitor_parse_opts (const char *desc,int argc,char **argv) {bool quiet = false;
@@ -696,9 +724,12 @@ int aen_monitor(const char *desc, int argc, char **argv) ret =monitor_main_loop (monitor);udev_monitor_unref (monitor); } + +conndb_for_each (monitor_kill_discovery_task,NULL );if (mon_cfg.autoconnect || mon_cfg.start_ctrls)conndb_for_each (monitor_remove_discovery_ctrl,NULL );conndb_free (); + udev =udev_unref (udev);if (mon_cfg.autoconnect && !mon_cfg.skip_udev_on_exit)monitor_enable_udev_rules ();
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 30/35 monitor: add connection property options from connect-all
In reply to [email protected]From: Martin Wilck <[email protected]>
This code is simply copied from fabrics_discover(). That's possible because we simply access the fabric "cfg" variable directly. These parameters are assumed to be constant over the runtime of the monitor.
--- monitor.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
monitor.c +13 -0
@@ -669,6 +669,19 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)OPT_FLAG ("autoconnect" ,'A' , &mon_cfg.autoconnect,"automatically connect newly discovered controllers" ),OPT_FLAG ("startup" ,'U' , &mon_cfg.start_ctrls,"start discovery controllers on startup" ),OPT_FLAG ("persistent" ,'p' , &cfg.persistent,"persistent discovery connections" ), +OPT_LIST ("hostnqn" ,'q' , &cfg.hostnqn,"user-defined hostnqn (if default not used)" ), +OPT_LIST ("hostid" ,'I' , &cfg.hostid,"user-defined hostid (if default not used)" ), +OPT_INT ("keep-alive-tmo" ,'k' , &cfg.keep_alive_tmo,"keep alive timeout period in seconds" ), +OPT_INT ("reconnect-delay" ,'c' , &cfg.reconnect_delay,"reconnect timeout period in seconds" ), +OPT_INT ("ctrl-loss-tmo" ,'l' , &cfg.ctrl_loss_tmo,"controller loss timeout period in seconds" ), +OPT_INT ("tos" ,'T' , &cfg.tos,"type of service" ), +OPT_FLAG ("hdr_digest" ,'g' , &cfg.hdr_digest,"enable transport protocol header digest (TCP transport)" ), +OPT_FLAG ("data_digest" ,'G' , &cfg.data_digest,"enable transport protocol data digest (TCP transport)" ), +OPT_INT ("nr-io-queues" ,'i' , &cfg.nr_io_queues,"number of io queues to use (default is core count)" ), +OPT_INT ("nr-write-queues" ,'W' , &cfg.nr_write_queues,"number of write queues to use (default 0)" ), +OPT_INT ("nr-poll-queues" ,'P' , &cfg.nr_poll_queues,"number of poll queues to use (default 0)" ), +OPT_INT ("queue-size" ,'Q' , &cfg.queue_size,"number of io queue elements to use (default 128)" ), +OPT_FLAG ("matching" ,'m' , &cfg.matching_only,"connect only records matching the traddr" ),OPT_FLAG ("silent" ,'S' , &quiet,"log level: silent" ),OPT_FLAG ("verbose" ,'v' , &verbose,"log level: verbose" ), OPT_FLAG("debug", 'D', &debug, "log level: debug"),
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 31/35 completions: add completions for nvme monitor
In reply to [email protected]From: Martin Wilck <[email protected]>
--- completions/bash-nvme-completion.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
completions/bash-nvme-completion.sh +5 -1
@@ -11,7 +11,7 @@ _cmds="list id-ctrl id-ns list-ns id-iocs create-ns delete-ns \ resv-report dsm flush compare read write write-zeroes \ write-uncor copy reset subsystem-reset show-regs discover \ connect-all connect disconnect version help \ - intel lnvm memblaze list-subsys" + intel lnvm memblaze list-subsys monitor" nvme_list_opts () { local opts=""
@@ -231,6 +231,10 @@ nvme_list_opts () { "disconnect") opts+=" --nqn -n --device -d" ;; + "monitor") + opts+=" --autoconnect -A --persistent -p --startup -U \ + --hostnqn= -q --verbose -v --clockstamps -C" + ;; "version") opts+="" ;;
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 32/35 nvmf-autoconnect: add unit file for nvme-monitor.service
In reply to [email protected]From: Martin Wilck <[email protected]>
--- nvmf-autoconnect/systemd/nvme-monitor.service | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service
nvmf-autoconnect/systemd/nvme-monitor.service new file +17 -0
@@ -0,0 +1,17 @@ +[Unit] +Description=NVMe Event Monitor for Automatical Subsystem Connection +Documentation=man:nvme-monitor(1) +DefaultDependencies=false +Requires=systemd-udevd-kernel.socket +RequiresMountsFor=/sys +Conflicts=shutdown.target +After=systemd-udevd-kernel.socket +Before=sysinit.target systemd-udev-trigger.service nvmefc-boot-connections.service + +[Service] +Type=simple +ExecStart=/usr/sbin/nvme monitor -A -p +KillMode=mixed + +[Install] +WantedBy=sysinit.target
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 32/35] nvmf-autoconnect: add unit file for nvme-monitor.service
Quoted 16 lines
Signed-off-by Martin Wilck <[email protected]>--- nvmf-autoconnect/systemd/nvme-monitor.service | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service
nvmf-autoconnect/systemd/nvme-monitor.service new file +3
-0@@ -0,0 +1,17 @@ +[Unit] +Description=NVMe Event Monitor for Automatical Subsystem Connection +Documentation=man:nvme-monitor(1)
nvmf-autoconnect/systemd/nvme-monitor.service line 13
Quoted 10 lines
nvmf-autoconnect/systemd/nvme-monitor.service new file +10
-0+DefaultDependencies=false +Requires=systemd-udevd-kernel.socket +RequiresMountsFor=/sys +Conflicts=shutdown.target +After=systemd-udevd-kernel.socket +Before=sysinit.target systemd-udev-trigger.service nvmefc-boot-connections.service + +[Service] +Type=simple +ExecStart=/usr/sbin/nvme monitor -A -p
I've seen issues with assuming nvme sits in /usr/sbin/
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 32/35] nvmf-autoconnect: add unit file for nvme-monitor.service
Quoted 24 lines · Sagi Grimberg
Quoted 24 lines
Signed-off-by Martin Wilck <[email protected]>---
nvmf-autoconnect/systemd/nvme-monitor.service | 17
+++++++++++++++++1 file changed, 17 insertions(+)create mode 100644 nvmf-autoconnect/systemd/nvme-monitor.service
nvmf- renamed from nvmf-autoconnect/systemd/nvme-monitor.service +0
-0autoconnect/systemd/nvme-monitor.service
new file mode 100644
index 0000000..d85b133nvmf-autoconnect/systemd/nvme-monitor.service new file +2
-0@@ -0,0 +1,17 @@ +[Unit] +Description=NVMe Event Monitor for Automatical SubsystemConnection
nvmf-autoconnect/systemd/nvme-monitor.service new file +6
-0+Documentation=man:nvme-monitor(1) +DefaultDependencies=false +Requires=systemd-udevd-kernel.socket +RequiresMountsFor=/sys +Conflicts=shutdown.target +After=systemd-udevd-kernel.socket
Quoted 8 lines
nvmf-autoconnect/systemd/nvme-monitor.service line 13
Quoted 6 lines
+Before=sysinit.target systemd-udev-trigger.service nvmefc-boot- connections.service
nvmf-autoconnect/systemd/nvme-monitor.service new file +4
-0+ +[Service] +Type=simple +ExecStart=/usr/sbin/nvme monitor -A -pI've seen issues with assuming nvme sits in
/usr/sbin/
Sure, this can be fixed. More fine tuning is possible, too. I learned meanwhile that the dependency on systemd-udevd-kernel.socket is unnecessary.
Regards
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 33/35 nvme-connect-all(1): fix documentation for --quiet/-S
In reply to [email protected]From: Martin Wilck <[email protected]>
--- Documentation/nvme-connect-all.1 | 8 ++++---- Documentation/nvme-connect-all.html | 10 +++++----- Documentation/nvme-connect-all.txt | 4 ++-- 3 files changed, 11 insertions(+), 11 deletions(-)
Documentation/nvme-connect-all.1 +4 -4
@@ -2,12 +2,12 @@ .\" Title: nvme-connect-all .\" Author: [FIXME: author] [see http://www.docbook.org/tdg5/en/html/author] .\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/> -.\" Date: 10/20/2020 +.\" Date: 01/20/2021 .\" Manual: NVMe Manual .\" Source: NVMe .\" Language: English .\" -.TH "NVME\-CONNECT\-ALL" "1" "10/20/2020" "NVMe" "NVMe Manual" +.TH "NVME\-CONNECT\-ALL" "1" "01/20/2021" "NVMe" "NVMe Manual" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" -----------------------------------------------------------------
@@ -51,7 +51,7 @@ nvme-connect-all \- Discover and Connect to Fabrics controllers\&. [\-\-queue\-size=<#> | \-Q <#>] [\-\-matching | \-m] [\-\-persistent | \-p] - [\-\-quiet | \-q] + [\-\-quiet | \-S] .fi .SH "DESCRIPTION" .sp
@@ -193,7 +193,7 @@ If a traddr was specified on the command line or in the configuration file, only Don\(cqt remove the discovery controller after retrieving the discovery log page\&. .RE .PP -\-q, \-\-quiet +\-S, \-\-quiet .RS 4 Suppress error messages\&. .RE
Documentation/nvme-connect-all.html +5 -5
@@ -4,7 +4,7 @@ <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"> <head> <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" /> -<meta name="generator" content="AsciiDoc 8.6.10" /> +<meta name="generator" content="AsciiDoc" /> <title>nvme-connect-all(1)</title> <style type="text/css"> /* Shared CSS for AsciiDoc xhtml11 and html5 backends */
@@ -436,7 +436,7 @@ thead, p.table.header { p.table { margin-top: 0; } -/* Because the table frame attribute is overriden by CSS in most browsers. */ +/* Because the table frame attribute is overridden by CSS in most browsers. */ div.tableblock > table[frame="void"] { border-style: none; }
@@ -768,7 +768,7 @@ nvme-connect-all(1) Manual Page [--queue-size=<#> | -Q <#>] [--matching | -m] [--persistent | -p] - [--quiet | -q]</pre> + [--quiet | -S]</pre> <div class="attribution"> </div></div> </div>
@@ -1048,7 +1048,7 @@ cellspacing="0" cellpadding="4"> </p> </dd> <dt class="hdlist1"> --q +-S </dt> <dt class="hdlist1"> --quiet
@@ -1115,7 +1115,7 @@ nvme-connect(1)</p></div> <div id="footer"> <div id="footer-text"> Last updated - 2020-10-20 16:47:21 PDT + 2021-01-20 23:40:57 CET </div> </div> </body>
Documentation/nvme-connect-all.txt +2 -2
@@ -27,7 +27,7 @@ SYNOPSIS [--queue-size=<#> | -Q <#>] [--matching | -m] [--persistent | -p] - [--quiet | -q] + [--quiet | -S] DESCRIPTION -----------
@@ -154,7 +154,7 @@ OPTIONS Don't remove the discovery controller after retrieving the discovery log page. --q:: +-S:: --quiet:: Suppress error messages.
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 33/35] nvme-connect-all(1): fix documentation for --quiet/-S
Cleanup unrelated patch
Quoted 110 lines · On 1/26/21 12:33 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
Signed-off-by Martin Wilck <[email protected]>--- Documentation/nvme-connect-all.1 | 8 ++++---- Documentation/nvme-connect-all.html | 10 +++++----- Documentation/nvme-connect-all.txt | 4 ++-- 3 files changed, 11 insertions(+), 11 deletions(-)Documentation/nvme-connect-all.1 +4
-4@@ -2,12 +2,12 @@ .\" Title: nvme-connect-all .\" Author: [FIXME: author] [see http://www.docbook.org/tdg5/en/html/author] .\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/> -.\" Date: 10/20/2020 +.\" Date: 01/20/2021 .\" Manual: NVMe Manual .\" Source: NVMe .\" Language: English .\" -.TH "NVME\-CONNECT\-ALL" "1" "10/20/2020" "NVMe" "NVMe Manual" +.TH "NVME\-CONNECT\-ALL" "1" "01/20/2021" "NVMe" "NVMe Manual" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" -----------------------------------------------------------------@@ -51,7 +51,7 @@ nvme-connect-all \- Discover and Connect to Fabrics controllers\&. [\-\-queue\-size=<#> | \-Q <#>] [\-\-matching | \-m] [\-\-persistent | \-p] - [\-\-quiet | \-q] + [\-\-quiet | \-S] .fi .SH "DESCRIPTION" .sp@@ -193,7 +193,7 @@ If a traddr was specified on the command line or in the configuration file, only Don\(cqt remove the discovery controller after retrieving the discovery log page\&. .RE .PP -\-q, \-\-quiet +\-S, \-\-quiet .RS 4 Suppress error messages\&. .REDocumentation/nvme-connect-all.html +5
-5@@ -4,7 +4,7 @@ <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"> <head> <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" /> -<meta name="generator" content="AsciiDoc 8.6.10" /> +<meta name="generator" content="AsciiDoc" /> <title>nvme-connect-all(1)</title> <style type="text/css"> /* Shared CSS for AsciiDoc xhtml11 and html5 backends */@@ -436,7 +436,7 @@ thead, p.table.header { p.table { margin-top: 0; } -/* Because the table frame attribute is overriden by CSS in most browsers. */ +/* Because the table frame attribute is overridden by CSS in most browsers. */ div.tableblock > table[frame="void"] { border-style: none; }@@ -768,7 +768,7 @@ nvme-connect-all(1) Manual Page [--queue-size=<#> | -Q <#>] [--matching | -m] [--persistent | -p] - [--quiet | -q]</pre> + [--quiet | -S]</pre> <div class="attribution"> </div></div> </div>@@ -1048,7 +1048,7 @@ cellspacing="0" cellpadding="4"> </p> </dd> <dt class="hdlist1"> --q +-S </dt> <dt class="hdlist1"> --quiet@@ -1115,7 +1115,7 @@ nvme-connect(1)</p></div> <div id="footer"> <div id="footer-text"> Last updated - 2020-10-20 16:47:21 PDT + 2021-01-20 23:40:57 CET </div> </div> </body>Documentation/nvme-connect-all.txt +1
-1@@ -27,7 +27,7 @@ SYNOPSIS [--queue-size=<#> | -Q <#>] [--matching | -m] [--persistent | -p] - [--quiet | -q] + [--quiet | -S] DESCRIPTION -----------
Documentation/nvme-connect-all.txt OPTIONS · line 160
Quoted 8 lines
Documentation/nvme-connect-all.txt +1
-1@@ -154,7 +154,7 @@ OPTIONS Don't remove the discovery controller after retrieving the discovery log page. --q:: +-S:: --quiet:: Suppress error messages.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 34/35 nvme-monitor(1): add man page for nvme-monitor
In reply to [email protected]From: Martin Wilck <[email protected]>
--- Documentation/cmds-main.txt | 4 + Documentation/nvme-monitor.1 | 204 ++++++ Documentation/nvme-monitor.html | 1049 +++++++++++++++++++++++++++++++ Documentation/nvme-monitor.txt | 169 +++++ 4 files changed, 1426 insertions(+) create mode 100644 Documentation/nvme-monitor.1 create mode 100644 Documentation/nvme-monitor.html create mode 100644 Documentation/nvme-monitor.txt
Documentation/cmds-main.txt +4 -0
@@ -168,3 +168,7 @@ linknvme:nvme-disconnect-all[1]:: linknvme:nvme-get-property[1]:: Reads and shows NVMe-over-Fabrics controller property + +linknvme:nvme-monitor[1]:: + Monitor Discovery events and Discover and Connect automatically +
Documentation/nvme-monitor.1 new file +204 -0
204 lines not on this page · raw message
Documentation/nvme-monitor.html new file +1049 -0
1,049 lines not on this page · raw message
Documentation/nvme-monitor.txt new file +169 -0
169 lines not on this page · raw message
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 35/35 monitor: add option --keep/-K
In reply to [email protected]From: Martin Wilck <[email protected]>
With this option, "nvme monitor" will keep created discovery controllers running even after it exits. If running in the initrd, this is will be done automatically. That has the advantage that the new monitor instance that will be started automatically after switching root can simply take over the already created discovery controllers.
--- Documentation/nvme-monitor.1 | 28 +++++++++++++++++++++------- Documentation/nvme-monitor.html | 30 ++++++++++++++++++++++++------ Documentation/nvme-monitor.txt | 1 + monitor.c | 15 ++++++++++++++- 4 files changed, 60 insertions(+), 14 deletions(-)
Documentation/nvme-monitor.1 +21 -7
@@ -2,12 +2,12 @@ .\" Title: nvme-monitor .\" Author: [FIXME: author] [see http://www.docbook.org/tdg5/en/html/author] .\" Generator: DocBook XSL Stylesheets vsnapshot <http://docbook.sf.net/> -.\" Date: 01/20/2021 +.\" Date: 01/25/2021 .\" Manual: NVMe Manual .\" Source: NVMe .\" Language: English .\" -.TH "NVME\-MONITOR" "1" "01/20/2021" "NVMe" "NVMe Manual" +.TH "NVME\-MONITOR" "1" "01/25/2021" "NVMe" "NVMe Manual" .\" ----------------------------------------------------------------- .\" * Define some portability stuff .\" -----------------------------------------------------------------
@@ -33,9 +33,10 @@ nvme-monitor \- Monitor Discovery events and Discover and Connect automatically .sp .nf \fInvme discover\fR - [\-\-autoconnect | \-A - [\-\-startup | \-U - [\-\-persistent | \-p + [\-\-autoconnect | \-A] + [\-\-startup | \-U] + [\-\-persistent | \-p] + [\-\-keep | \-K] [\-\-hostnqn=<hostnqn> | \-q <hostnqn>] [\-\-hostid=<hostid> | \-I <hostid>] [\-\-keep\-alive\-tmo=<#> | \-k <#>]
@@ -118,9 +119,22 @@ option to make the Discovery Controllers persist after retrieving the Discovery This option has the same meaning as for \fInvme discover\fR; Discovery Controllers will remain after the Discovery Log Page has been retrieved\&. This is necessary for receiving AENs from the transport connection that the Discovery controller is associated with\&. Without this option, AENs can only be received and acted upon if the administrator manually creates persistent Discovery controller connections\&. When \fInvme monitor\fR -terminates, it will attempt to take down all Discovery Controllers it has created (i\&.e\&. ones that didn\(cqt already exist when +terminates and the, it will attempt to take down all Discovery Controllers it has created (i\&.e\&. ones that didn\(cqt already exist when \fInvme monitor\fR -was started\&. +was started (but see +\fI\-\-keep\fR)\&. +.RE +.PP +\-K, \-\-keep +.RS 4 +Do not tear down created persistent discovery controllers when exiting\&. This option makes only sense together with +\fI\-\-persistent\fR\&. When +\fInvme monitor\fR +detects that it is running in the initial RAM FS (initrd), +\fI\-\-keep\fR +is implicitly assumed, i\&.e\&. discovery connections are not stopped\&. This is because after switching from the initrd to the real root file system, another +\fInvme monitor\fR +instance will probably be started that can reuse the created controller instances\&. .RE .PP \-q <hostnqn>, \-\-hostnqn=<hostnqn>, \-I <hostid>, \-\-hostid=<hostid>, \-k <#>, \-\-keep\-alive\-tmo=<#>, \-c <#>, \-\-reconnect\-delay=<#>, \-l <#>, \-\-ctrl\-loss\-tmo=<#>, \-g, \-\-hdr\-digest, \-G, \-\-data\-digest, \-i <#>, \-\-nr\-io\-queues=<#>, \-W <#>, \-\-nr\-write\-queues=<#>, \-P <#>, \-\-nr\-poll\-queues=<#>, \-Q <#>, \-\-queue\-size=<#>, \-m, \-\-matching
Documentation/nvme-monitor.html +24 -6
@@ -750,9 +750,10 @@ nvme-monitor(1) Manual Page <div class="sectionbody"> <div class="verseblock"> <pre class="content"><em>nvme discover</em> - [--autoconnect | -A - [--startup | -U - [--persistent | -p + [--autoconnect | -A] + [--startup | -U] + [--persistent | -p] + [--keep | -K] [--hostnqn=<hostnqn> | -q <hostnqn>] [--hostid=<hostid> | -I <hostid>] [--keep-alive-tmo=<#> | -k <#>]
@@ -863,9 +864,26 @@ commands for further background.</p></div> this option, AENs can only be received and acted upon if the administrator manually creates persistent Discovery controller connections. - When <em>nvme monitor</em> terminates, it will attempt to take down all + When <em>nvme monitor</em> terminates and the, it will attempt to take down all Discovery Controllers it has created (i.e. ones that didn’t already - exist when <em>nvme monitor</em> was started. + exist when <em>nvme monitor</em> was started (but see <em>--keep</em>). +</p> +</dd> +<dt class="hdlist1"> +-K +</dt> +<dt class="hdlist1"> +--keep +</dt> +<dd> +<p> + Do not tear down created persistent discovery controllers when + exiting. This option makes only sense together with <em>--persistent</em>. + When <em>nvme monitor</em> detects that it is running in the initial RAM FS + (initrd), <em>--keep</em> is implicitly assumed, i.e. discovery connections + are not stopped. This is because after switching from the initrd to + the real root file system, another <em>nvme monitor</em> instance will + probably be started that can reuse the created controller instances. </p> </dd> <dt class="hdlist1">
@@ -1042,7 +1060,7 @@ nvme-connect-all(1)</p></div> <div id="footer"> <div id="footer-text"> Last updated - 2021-01-20 23:53:52 CET + 2021-01-25 22:35:29 CET </div> </div> </body>
Documentation/nvme-monitor.txt +1 -0
@@ -12,6 +12,7 @@ SYNOPSIS [--autoconnect | -A] [--startup | -U] [--persistent | -p] + [--keep | -K] [--hostnqn=<hostnqn> | -q <hostnqn>] [--hostid=<hostid> | -I <hostid>] [--keep-alive-tmo=<#> | -k <#>]
monitor.c +14 -1
@@ -44,6 +44,7 @@ static struct monitor_config {bool autoconnect;bool skip_udev_on_exit;bool start_ctrls; +bool keep_ctrls; } mon_cfg;static struct udev *udev;
@@ -669,6 +670,7 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)OPT_FLAG ("autoconnect" ,'A' , &mon_cfg.autoconnect,"automatically connect newly discovered controllers" ),OPT_FLAG ("startup" ,'U' , &mon_cfg.start_ctrls,"start discovery controllers on startup" ),OPT_FLAG ("persistent" ,'p' , &cfg.persistent,"persistent discovery connections" ), +OPT_FLAG ("keep" ,'K' , &mon_cfg.keep_ctrls,"keep discovery controllers after exit" ),OPT_LIST ("hostnqn" ,'q' , &cfg.hostnqn,"user-defined hostnqn (if default not used)" ),OPT_LIST ("hostid" ,'I' , &cfg.hostid,"user-defined hostid (if default not used)" ), OPT_INT("keep-alive-tmo", 'k', &cfg.keep_alive_tmo, "keep alive timeout period in seconds"),
@@ -716,6 +718,11 @@ static int monitor_parse_opts(const char *desc, int argc, char **argv)return ret; } +static bool running_in_initrd (void ) +{ +return access ("/etc/initrd-release" ,F_OK ) >=0 ; +} +int aen_monitor (const char *desc,int argc,char **argv) {int ret;
@@ -724,11 +731,16 @@ int aen_monitor(const char *desc, int argc, char **argv) ret =monitor_parse_opts (desc, argc, argv);if (ret) goto out; + ret =monitor_init_signals ();if (ret !=0 ) {log (LOG_ERR ,"monitor: failed to initialize signals: %m\n" ); goto out; } + +if (running_in_initrd ()) + mon_cfg.keep_ctrls = true; + ret =create_udev_monitor (&monitor);if (ret ==0 ) {conndb_init_from_sysfs ();
@@ -739,7 +751,8 @@ int aen_monitor(const char *desc, int argc, char **argv) }conndb_for_each (monitor_kill_discovery_task,NULL ); -if (mon_cfg.autoconnect || mon_cfg.start_ctrls) +if ((mon_cfg.autoconnect || mon_cfg.start_ctrls) && + !mon_cfg.keep_ctrls)conndb_for_each (monitor_remove_discovery_ctrl,NULL );conndb_free ();
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 35/35] monitor: add option --keep/-K
Quoted 8 lines
From: Martin Wilck <[email protected]>
With this option, "nvme monitor" will keep created discovery controllers running even after it exits. If running in the initrd, this is will be done automatically. That has the advantage that the new monitor instance that will be started automatically after switching root can simply take over the already created discovery controllers.
Why not make it the default?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 35/35] monitor: add option --keep/-K
Quoted 13 lines · Sagi Grimberg
Quoted 12 lines
From: Martin Wilck <[email protected]>
With this option, "nvme monitor" will keep created discovery controllers
running even after it exits. If running in the initrd, this is will be
done automatically. That has the advantage that the new monitor instance
that will be started automatically after switching root can simply take
over the already created discovery controllers.
Quoted 1 line
Why not make it the default?
Well, we've had a lot of discussions about persistent discovery connections not being universally loved. So I thought I'd make cleaning up the default.
All this can be discussed of course, that's why I called it an RFC.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 35/35] monitor: add option --keep/-K
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
With this option, "nvme monitor" will keep created discovery controllers
running even after it exits. If running in the initrd, this is will be
done automatically. That has the advantage that the new monitor instance
that will be started automatically after switching root can simply take
over the already created discovery controllers.
Quoted 5 lines
Quoted 1 line
Why not make it the default?
Well, we've had a lot of discussions about persistent discovery connections not being universally loved. So I thought I'd make cleaning up the default.
Heh... well, I didn't see thus far of any other good suggestion on how to propagate discovery change log events reliably (I've seen some bad ones). And we are assuming that hosts need to know about these changes...
The default matters here. The spec tells us how to learn about change log events so I would suggest we follow it.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 35/35] monitor: add option --keep/-K
Quoted 20 lines · Sagi Grimberg
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
With this option, "nvme monitor" will keep created discovery controllers
running even after it exits. If running in the initrd, this is will
be
done automatically. That has the advantage that the new monitor instance
that will be started automatically after switching root can simply
take
over the already created discovery controllers.Quoted 6 lines
Quoted 1 line
Why not make it the default?
Well, we've had a lot of discussions about persistent discovery connections not being universally loved. So I thought I'd make cleaning
up the default.
Quoted 9 lines
Heh... well, I didn't see thus far of any other good suggestion on how
to propagate discovery change log events reliably (I've seen some bad ones). And we are assuming that hosts need to know about these changes...The default matters here. The spec tells us how to learn about change log events so I would suggest we follow it.
No problem, I can change this.
But I'm not quite following you here, because this option is about keeping controllers up after the monitor exits. Given that the monitor would be the instance that receives the AENs and acts upon them, what's the point of keeping them alive after it has terminated?
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 35/35] monitor: add option --keep/-K
Quoted 30 lines
Quoted 20 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
With this option, "nvme monitor" will keep created discovery controllers
running even after it exits. If running in the initrd, this is will
be
done automatically. That has the advantage that the new monitor instance
that will be started automatically after switching root can simply
take
over the already created discovery controllers.Quoted 6 lines
Quoted 1 line
Why not make it the default?
Well, we've had a lot of discussions about persistent discovery connections not being universally loved. So I thought I'd make cleaning
up the default.Quoted 9 lines
Heh... well, I didn't see thus far of any other good suggestion on how
to propagate discovery change log events reliably (I've seen some bad ones). And we are assuming that hosts need to know about these changes...The default matters here. The spec tells us how to learn about change log events so I would suggest we follow it.
Quoted 6 lines
No problem, I can change this.
But I'm not quite following you here, because this option is about keeping controllers up after the monitor exits. Given that the monitor would be the instance that receives the AENs and acts upon them, what's the point of keeping them alive after it has terminated?
It may run again at some point.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 35/35] monitor: add option --keep/-K
Quoted 12 lines · Sagi Grimberg
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But I'm not quite following you here, because this option is about keeping controllers up after the monitor exits. Given that the monitor
would be the instance that receives the AENs and acts upon them, what's
the point of keeping them alive after it has terminated?It may run again at some point.
If it does, it will recreate discovery controllers for every host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are only necessary for addresses on which no regular (non-discovery) connection exists.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 35/35] monitor: add option --keep/-K
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
But I'm not quite following you here, because this option is about keeping controllers up after the monitor exits. Given that the monitor
would be the instance that receives the AENs and acts upon them, what's
the point of keeping them alive after it has terminated?
Quoted 7 lines
Quoted 2 lines
It may run again at some point.
If it does, it will recreate discovery controllers for every host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are only necessary for addresses on which no regular (non-discovery) connection exists.
Wait, Maybe I'm missing something here, but are you saying that for every traddr/trsvcid it finds (both nvm and discovery) it will attempt to connect a discovery controller?
If so, this is absolutely wrong.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 35/35] monitor: add option --keep/-K
Quoted 20 lines · Sagi Grimberg
Quoted 13 lines
Quoted 10 lines
Quoted 9 lines
But I'm not quite following you here, because this option is about
keeping controllers up after the monitor exits. Given that the monitor
would be the instance that receives the AENs and acts upon them,
what's
the point of keeping them alive after it has terminated?Quoted 2 lines
It may run again at some point.
Quoted 6 lines
If it does, it will recreate discovery controllers for every host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are only
necessary for addresses on which no regular (non-discovery) connection
exists.
Quoted 6 lines
Wait, Maybe I'm missing something here, but are you saying that for every traddr/trsvcid it finds (both nvm and discovery) it will attempt
to connect a discovery controller?If so, this is absolutely wrong.
Currently, it tries to do that on startup, if (and only if) the --startup option is given. My expectation was that the connection attempts would simply fail if there was no discovery subsystem to connect to. Anyway, it's not the default behavior, and can be dropped completely if it's so bad that we shouldn't ever attempt to do it.
If the service is started early on during boot, and event-based discovery works (i.e. we also have the mDNS part in place), this won't be necessary of course.
Regards
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 35/35] monitor: add option --keep/-K
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
If it does, it will recreate discovery controllers for every host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are only
necessary for addresses on which no regular (non-discovery) connection
exists.Wait, Maybe I'm missing something here, but are you saying that for every traddr/trsvcid it finds (both nvm and discovery) it will attempt
to connect a discovery controller?
Quoted 7 lines
Quoted 1 line
If so, this is absolutely wrong.
Currently, it tries to do that on startup, if (and only if) the --startup option is given. My expectation was that the connection attempts would simply fail if there was no discovery subsystem to connect to. Anyway, it's not the default behavior, and can be dropped completely if it's so bad that we shouldn't ever attempt to do it.
IMO it needs to be dropped. I didn't understand this at first because it never even occurred to me that such an assumption can be even made.
Quoted 3 lines
If the service is started early on during boot, and event-based discovery works (i.e. we also have the mDNS part in place), this won't be necessary of course.
This isn't necessary regardless. At best the discovery controller endpoints should be obtained from discovery.conf or equivalent.
I don't even see how does this help in early boot anyways, how do the existing controllers get connected?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 35/35] monitor: add option --keep/-K
Quoted 28 lines · Sagi Grimberg
Quoted 15 lines
Quoted 15 lines
Quoted 8 lines
If it does, it will recreate discovery controllers for every host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are
only
necessary for addresses on which no regular (non-discovery) connection
exists.Wait, Maybe I'm missing something here, but are you saying that for
every traddr/trsvcid it finds (both nvm and discovery) it will attempt
to connect a discovery controller?Quoted 8 lines
Quoted 1 line
If so, this is absolutely wrong.
Currently, it tries to do that on startup, if (and only if) the --startup option is given. My expectation was that the connection attempts would simply fail if there was no discovery subsystem to connect to. Anyway, it's not the default behavior, and can be dropped
completely if it's so bad that we shouldn't ever attempt to do it.IMO it needs to be dropped. I didn't understand this at first because it never even occurred to me that such an assumption can be even made.
Quoted 7 lines
Quoted 4 lines
If the service is started early on during boot, and event-based discovery works (i.e. we also have the mDNS part in place), this won't
be necessary of course.This isn't necessary regardless. At best the discovery controller endpoints should be obtained from discovery.conf or equivalent.
Understood.
Quoted 2 lines
I don't even see how does this help in early boot anyways, how do the existing controllers get connected?
For FC, it works today. You can start the monitor in the initrd, it will receive the fc_udev_device events, and autoconnect (just like traditional udev/systemd based discovery). For other transports, it'd be more work, as we'd need to do mDNS discovery from an initrd environment. I wouldn't say it's impossible though.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 35/35] monitor: add option --keep/-K
Quoted 9 lines
Quoted 3 lines
I don't even see how does this help in early boot anyways, how do the existing controllers get connected?
For FC, it works today. You can start the monitor in the initrd, it will receive the
fc_udev_deviceevents, and autoconnect (just like traditional udev/systemd based discovery). For other transports, it'd be more work, as we'd need to do mDNS discovery from an initrd environment. I wouldn't say it's impossible though.
I was referring to the nvme controller walk and connect a discovery controller to that endpoint. I thought you meant that this helps you at early boot time.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 35/35] monitor: add option --keep/-K
In reply to Martin WilckQuoted 39 lines · Martin Wilck
Quoted 28 lines · Sagi Grimberg
Quoted 15 lines
Quoted 15 lines
Quoted 8 lines
If it does, it will recreate discovery controllers for every host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics are
only
necessary for addresses on which no regular (non-discovery) connection
exists.Wait, Maybe I'm missing something here, but are you saying that for
every traddr/trsvcid it finds (both nvm and discovery) it will attempt
to connect a discovery controller?Quoted 8 lines
Quoted 1 line
If so, this is absolutely wrong.
Currently, it tries to do that on startup, if (and only if) the --startup option is given. My expectation was that the connection attempts would simply fail if there was no discovery subsystem to connect to. Anyway, it's not the default behavior, and can be dropped
completely if it's so bad that we shouldn't ever attempt to do it.IMO it needs to be dropped. I didn't understand this at first because it never even occurred to me that such an assumption can be even made.
Quoted 7 lines
Quoted 4 lines
If the service is started early on during boot, and event-based discovery works (i.e. we also have the mDNS part in place), this won't
be necessary of course.This isn't necessary regardless. At best the discovery controller endpoints should be obtained from discovery.conf or equivalent.
Understood.
Quoted 9 lines
Quoted 2 lines
I don't even see how does this help in early boot anyways, how do the existing controllers get connected?
For FC, it works today. You can start the monitor in the initrd, it will receive the
fc_udev_deviceevents, and autoconnect (just like traditional udev/systemd based discovery). For other transports, it'd be more work, as we'd need to do mDNS discovery from an initrd environment. I wouldn't say it's impossible though.
mDNS isn't related to the initrd, mDNS is for discovery discovery :-)
For everything _but_ FC we have the problem that we don't know where to connect to for the initial discovery (ie we don't have a list of potential discovery controllers). Currently we either have to specify the parameters manually, or add them to /etc/nvme/discovery.conf.
Which isn't _that_ dynamic.
mDNS will provide a way of promoting changes in any remote discovery controllers to the potential hosts, allowing the host to pick up information about potential discovery controllers from there.
So the monitor code would need to interact with mDNS to get the information, and should use that to establish discovery connections.
As this information is transient I would not tear down discovery controller connections on exit, but rather keep them alive.
That way a controller restart could pick up these existing connections and would not need to recreate them (which will be tricky as the monitor code doesn't have the information which connections to restart).
The alternative would be a callout to mDNS to get the current topology, but that doesn't exist yet.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 35/35] monitor: add option --keep/-K
Quoted 52 lines · Hannes Reinecke
Quoted 51 lines · Martin Wilck
Quoted 36 lines · Sagi Grimberg
Quoted 21 lines
Quoted 19 lines
Quoted 10 lines
If it does, it will recreate discovery controllers for every
host_traddr/traddr/trsvcid tuple it finds. "--keep" semantics
are
only
necessary for addresses on which no regular (non-discovery) connection
exists.Wait, Maybe I'm missing something here, but are you saying that
for
every traddr/trsvcid it finds (both nvm and discovery) it will
attempt
to connect a discovery controller?Quoted 1 line
If so, this is absolutely wrong.
Quoted 9 lines
Currently, it tries to do that on startup, if (and only if) the --startup option is given. My expectation was that the connection
attempts would simply fail if there was no discovery subsystem to
connect to. Anyway, it's not the default behavior, and can be dropped
completely if it's so bad that we shouldn't ever attempt to do it.IMO it needs to be dropped. I didn't understand this at first because
it never even occurred to me that such an assumption can be even made.Quoted 8 lines
Quoted 5 lines
If the service is started early on during boot, and event-based discovery works (i.e. we also have the mDNS part in place), this
won't
be necessary of course.This isn't necessary regardless. At best the discovery controller endpoints should be obtained from discovery.conf or equivalent.
Understood.
Quoted 2 lines
I don't even see how does this help in early boot anyways, how do the existing controllers get connected?
Quoted 8 lines
Quoted 7 lines
For FC, it works today. You can start the monitor in the initrd, it will receive the
fc_udev_deviceevents, and autoconnect (just like traditional udev/systemd based discovery). For other transports, it'd
be more work, as we'd need to do mDNS discovery from an initrd environment. I wouldn't say it's impossible though.mDNS isn't related to the initrd, mDNS is for discovery discovery :-)
What I had in mind was booting from NVMeoF (non-FC), without prior local configuration. That requires "discovery discovery" in the initrd, and thus making mDNS calls in the initrd.
Regards
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 9 lines
From: Martin Wilck <[email protected]>
(Cover letter copied from https://github.com/linux-nvme/nvme-cli/pull/877)
This patch set adds a new subcommand **nvme monitor**. In this mode, **nvme-cli** runs continuously, monitors events (currently, uevents) relevant for discovery, and optionally autoconnects to newly discovered subsystems.
Quoted 6 lines
The monitor mode is suitable to be run in a systemd service. An appropriate unit file is provided. As such, **nvme monitor** can be used as an alternative to the current auto-connection mechanism based on udev rules and systemd template units. If
--autoconnectis active, **nvme monitor** masks the respective udev rules in order to prevent simultaneous connection attempts from udev and itself.
I think that a monitor daemon is a good path forward.
Quoted 9 lines
This method for discovery and autodetection has some advantages over the current udev-rule based approach:
* By using the
--persistentoption, users can easily control whether persistent discovery controllers for discovered transport addresses should be created and monitored for AEN events. **nvme monitor** watches known transport addresses, creates discovery controllers as required, and re-uses existing ones if possible.
What does that mean?
It keeps track of persistent discovery
Quoted 4 lines
controllers it created, and tears them down on exit. When run in
--persistent --autoconnectmode *in the initial ramfs*, it will keep discovery controllers alive, so that a new instance started after switching root can simply re-use them.
Nice.
Quoted 8 lines
* In certain situations, the systemd-based approach may miss events due to race conditions. This can happen e.g. if an FC remote port is detected, but shortly after it's detection an FC relogin procedure is necessary e.g. due to an RSCN. In this case, an
fc_udev_deviceuevent will be received on the first detection and handled by annvme connect-allcommand run from[email protected]. The connection attempt to the rport in question will fail with "no such device" because of the simultaneous FC relogin.[email protected]may not terminate immediately, because it
Quoted 8 lines
attempts to establish other connections listed in the Discovery Log page it retrieved. When the FC relogin eventually finishes, a new uevent will be received, and
nvmf-connect@will be started again, but *this has no effect* if the previousnvmf-connect@service hasn't finished yet. This is the general semantics of systemd services, no easy workaround exists. **nvme monitor** doesn't suffer from this problem. If it sees an uevent for a transport address for which a discovery is already running, it will queue the handling of this event up and restart the discovery after it's finished.
While I understand the issue, this reason alone is an overkill for doing this.
Quoted 7 lines
* Resource consumption for handling uevents is lower. Instead of running an udev worker, executing the rules, executing
systemctl startfrom the worker, starting a systemd service, and starting a separate **nvme-cli** instance, only a singlefork()operation is necessary. Of course, on the back side, the monitor itself consumes resources while it's running and waiting for events. On my system with 8 persistent discovery controllers, its RSS is ~3MB. CPU consumption is zero as long as no events occur.
What is the baseline with what we have today?
Quoted 2 lines
* **nvme monitor** could be easily extended to handle events for non-FC transports.
Which events?
Quoted 10 lines
I've tested
fc_udev_devicehandling for NVMeoFC with an Ontap target, and AEN handling for RDMA using a Linux **nvmet** target.### Implementation notes
I've tried to change the exisiting **nvme-cli** code as little as possible while reusing the code from
fabrics.c. The majority of changes in the existing code exports formerly static functions and variables, so that they are usable from the monitor code.
General comment, can you please separate out fixes/cleanups that are not related to the goal of this patchset?
Quoted 13 lines
The main process just waits for events using
epoll(). When an event is received that necessitates a new discovery, a child is forked. This makes it possible to fill in the configuration parameters fordo_discover()without interfering with the main process or other discovery tasks running in parallel. The program tracks *transport addresses* (called "connections" in the code) rather than NVMe controllers. In--persistentmode, it tries to maintain exactly one persistent discovery connection per transport address.Using
epoll()may look over-engineered at this stage. I hope the better flexibility overpoll()(in particular, the ability to add new event sources while waiting) will simplify future extensions and improvements.
Quoted 8 lines
### Todo
* Referrals are not handled perfectly yet. They will be handled by
do_discover()just as it would when called from **nvme connect-all**, but it would be better to pass referrals back to the main process to make it aware of the additional discovery controller rather than using recursion. The main process would e.g. know if a discovery is already running for the transport address in the referrral.
Agree, this way you can also keep new referrals persistently, which I don't know if we do today.
Quoted 5 lines
* When "add" uevents for nvme controller devices are received, the controller is consistently not in
livestate yet, and attempting to read thesubsysnqnsysfs attribute returns(efault). While this should arguably be fixed in the kernel, it could be worked around in user space by using timers or polling thestatesysfs attribute for changes.
This is a bug, what in the code causes this? nothing in controller state should prevent from this sysfs read from executing correctly...
Quoted 1 line
* Parse and handle
discovery.confon startup.
This is a must I think, where do you get the known transport addresses on startup today?
Quoted 1 line
* Implement support for RDMA and TCP protocols.
What is needed for supporting them? Not sure I follow (I thought you mentioned that you tested against linux nvmet-rdma?)
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 42 lines · Sagi Grimberg
Quoted 13 lines
From: Martin Wilck <[email protected]>
(Cover letter copied from
https://github.com/linux-nvme/nvme-cli/pull/877)This patch set adds a new subcommand **nvme monitor**. In this mode,
**nvme-cli** runs continuously, monitors events (currently, uevents) relevant
for discovery, and optionally autoconnects to newly discovered subsystems.Quoted 11 lines
The monitor mode is suitable to be run in a systemd service. An appropriate
unit file is provided. As such, **nvme monitor** can be used as an alternative
to the current auto-connection mechanism based on udev rules and systemd
template units. If--autoconnectis active, **nvme monitor** masks the
respective udev rules in order to prevent simultaneous connection attempts
from udev and itself.I think that a monitor daemon is a good path forward.
Quoted 5 lines
This method for discovery and autodetection has some advantages over the
current udev-rule based approach:Quoted 9 lines
* By using the
--persistentoption, users can easily control whether
persistent discovery controllers for discovered transport addresses should
be created and monitored for AEN events. **nvme monitor** watches known
transport addresses, creates discovery controllers as required, and re-uses
existing ones if possible.
Quoted 1 line
What does that mean?
In general, if the monitor detects a new host_traddr/traddr/trsvcid tuple, it runs a discovery on it, and keeps the discovery controller open if --persistent was given. On startup, it scans existing controllers, and if it finds already existing discovery controllers, re-uses them. These will not be shut down when the monitor exits.
This allows users fine-grained control about what discovery controllers to operate persistently. Users who want all discovery controllers to be persistent just use --persistent. Others can set up those that they want to have (manually or with a script), and not use --persistent.
The background is that hosts may not need every detected discovery controller to be persistent. In multipath scenarios, you may see more discovery subsystems than anything else, and not everyone likes that. That's a generic issue and unrelated to the monitor, but running the monitor with --persistent creates discovery controllers that would otherwise not be visible.
Hope this clarifies it.
Quoted 32 lines
Quoted 23 lines
* In certain situations, the systemd-based approach may miss events due to
race conditions. This can happen e.g. if an FC remote port is detected, but
shortly after it's detection an FC relogin procedure is necessary e.g. due to
an RSCN. In this case, anfc_udev_deviceuevent will be received on the
first detection and handled by annvme connect-allcommand run from
[email protected]. The connection attempt to the rport in question will
fail with "no such device" because of the simultaneous FC relogin.[email protected]may not terminate immediately, because it
attempts to establish other connections listed in the Discovery Log page it
retrieved. When the FC relogin eventually finishes, a new uevent will be
received, andnvmf-connect@will be started again, but *this has no effect*
if the previousnvmf-connect@service hasn't finished yet. This is theQuoted 8 lines
general semantics of systemd services, no easy workaround exists. **nvme
monitor** doesn't suffer from this problem. If it sees an uevent for a
transport address for which a discovery is already running, it will queue the
handling of this event up and restart the discovery after it's finished.
Quoted 3 lines
While I understand the issue, this reason alone is an overkill for doing
this.
I agree. But it's not easy to fix the issue otherwise. In the customer problem where we observed it, I worked around it by adding the udev seqnum to the "instance name" of the systemd service, thus allowing several "nvme connect-all" processes to run for the same transport address simultaneously. But I don't think that would scale well; the monitor can handle it more cleanly.
Quoted 15 lines
Quoted 6 lines
* Resource consumption for handling uevents is lower. Instead of running an
udev worker, executing the rules, executingsystemctl startfrom the
worker, starting a systemd service, and starting a separate **nvme-cli**Quoted 8 lines
instance, only a single
fork()operation is necessary. Of course, on the
back side, the monitor itself consumes resources while it's running and
waiting for events. On my system with 8 persistent discovery controllers,
its RSS is ~3MB. CPU consumption is zero as long as no events occur.
Quoted 1 line
What is the baseline with what we have today?
A meaningful comparsion is difficult and should be done when the monitor functionality is finalized. I made this statement only to provide a rough idea of the resource usage, not more.
Quoted 5 lines
Quoted 3 lines
* **nvme monitor** could be easily extended to handle events for non-FC
transports.Which events?
Network discovery, mDNS or the like. I haven't digged into the details yet.
Quoted 14 lines
Quoted 6 lines
I've tested
fc_udev_devicehandling for NVMeoFC with an Ontap target, and
AEN handling for RDMA using a Linux **nvmet** target.### Implementation notes
Quoted 7 lines
I've tried to change the exisiting **nvme-cli** code as little as possible
while reusing the code fromfabrics.c. The majority of changes in the
existing code exports formerly static functions and variables, so that they
are usable from the monitor code.
Quoted 3 lines
General comment, can you please separate out fixes/cleanups that are not
related to the goal of this patchset?
Which ones are you referring to? 09 and 19? While these are minor improvements to the existing code, I wouldn't say they qualify as fixes or cleanups. They aren't necessary without adding the monitor code.
But yes, I can post all changes to existing code separately.
Quoted 11 lines
Quoted 10 lines
* When "add" uevents for nvme controller devices are received, the
controller is consistently not inlivestate yet, and attempting to read
thesubsysnqnsysfs attribute returns(efault). While this should
arguably be fixed in the kernel, it could be worked around in user space
by using timers or polling thestatesysfs attribute for changes.
Quoted 3 lines
This is a bug, what in the code causes this? nothing in controller state
should prevent from this sysfs read from executing correctly...
I think it can be fixed by making nvme_sysfs_show_subsysnqn() fall back to ctrl->opts->subsysnqn if ctrl->subsys is NULL.
I'll send a patch. Anyway, it'll take time until this is fixed everywhere.
Quoted 6 lines
Quoted 2 lines
* Parse and handle
discovery.confon startup.This is a must I think, where do you get the known transport addresses
on startup today?
There's a systemd service that runs "nvme connect-all" once during boot. That exists today. I'm not sure if it should be integrated in the monitor, perhaps it's good to keep these separate. People who don't need the monitor can still run the existing service only, whereas for others, the two would play together just fine.
Quoted 6 lines
Quoted 2 lines
* Implement support for RDMA and TCP protocols.
What is needed for supporting them? Not sure I follow (I thought you mentioned that you tested against linux nvmet-rdma?)
AENs over existing discovery controllers are supported for all transports. But there's no support for discovery of new transports except for NVMeoFC's "fc_udev_device" mechanism.
Regards
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 16 lines
Quoted 16 lines
Quoted 15 lines
This method for discovery and autodetection has some advantages over the
current udev-rule based approach:* By using the
--persistentoption, users can easily control
whether
persistent discovery controllers for discovered transport
addresses should
be created and monitored for AEN events. **nvme monitor**
watches known
transport addresses, creates discovery controllers as required,
and re-uses
existing ones if possible.
Quoted 7 lines
Quoted 1 line
What does that mean?
In general, if the monitor detects a new host_traddr/traddr/trsvcid tuple, it runs a discovery on it, and keeps the discovery controller open if --persistent was given. On startup, it scans existing controllers, and if it finds already existing discovery controllers, re-uses them. These will not be shut down when the monitor exits.
And if it doesn't run with --persistent it deletes them? even if these weren't created by it?
And, if none exist where does it get new discovery controller details? Today we use discovery.conf for that.
Quoted 5 lines
This allows users fine-grained control about what discovery controllers to operate persistently. Users who want all discovery controllers to be persistent just use --persistent. Others can set up those that they want to have (manually or with a script), and not use --persistent.
Quoted 8 lines
The background is that hosts may not need every detected discovery controller to be persistent. In multipath scenarios, you may see more discovery subsystems than anything else, and not everyone likes that. That's a generic issue and unrelated to the monitor, but running the monitor with --persistent creates discovery controllers that would otherwise not be visible.
Hope this clarifies it.
Well, How do people expect to know if stuff change without a persistent discovery controller? Not sure if people may see more discovery controllers this is a real issue given what they are getting from it.
But OK.
Quoted 36 lines
Quoted 32 lines
Quoted 23 lines
* In certain situations, the systemd-based approach may miss
events due to
race conditions. This can happen e.g. if an FC remote port is
detected, but
shortly after it's detection an FC relogin procedure is
necessary e.g. due to
an RSCN. In this case, anfc_udev_deviceuevent will be
received on the
first detection and handled by annvme connect-allcommand
run from
[email protected]. The connection attempt to the rport in
question will
fail with "no such device" because of the simultaneous FC relogin.[email protected]may not terminate immediately,
because it
attempts to establish other connections listed in the Discovery
Log page it
retrieved. When the FC relogin eventually finishes, a new
uevent will be
received, andnvmf-connect@will be started again, but *this
has no effect*
if the previousnvmf-connect@service hasn't finished yet.
This is theQuoted 8 lines
general semantics of systemd services, no easy workaround
exists. **nvme
monitor** doesn't suffer from this problem. If it sees an
uevent for a
transport address for which a discovery is already running, it
will queue the
handling of this event up and restart the discovery after it's
finished.Quoted 3 lines
While I understand the issue, this reason alone is an overkill for doing
this.
Quoted 6 lines
I agree. But it's not easy to fix the issue otherwise. In the customer problem where we observed it, I worked around it by adding the udev seqnum to the "instance name" of the systemd service, thus allowing several "nvme connect-all" processes to run for the same transport address simultaneously. But I don't think that would scale well; the monitor can handle it more cleanly.
Still changing the entire thing for a corner case...
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
* Resource consumption for handling uevents is lower. Instead of
running an
udev worker, executing the rules, executingsystemctl start
from the
worker, starting a systemd service, and starting a separate
**nvme-cli**
instance, only a singlefork()operation is necessary. Of
course, on the
back side, the monitor itself consumes resources while it's
running and
waiting for events. On my system with 8 persistent discovery
controllers,
its RSS is ~3MB. CPU consumption is zero as long as no events
occur.
Quoted 5 lines
Quoted 1 line
What is the baseline with what we have today?
A meaningful comparsion is difficult and should be done when the monitor functionality is finalized. I made this statement only to provide a rough idea of the resource usage, not more.
You mention that the utilization is lower than what we do today, hence my question is by how much?
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
* **nvme monitor** could be easily extended to handle events for
non-FC
transports.Which events?
Network discovery, mDNS or the like. I haven't digged into the details yet.
Yes, that is possible, would probably be easier to do this in a higher level language but libavahi can also do this...
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
I've tested
fc_udev_devicehandling for NVMeoFC with an Ontap target, and
AEN handling for RDMA using a Linux **nvmet** target.### Implementation notes
I've tried to change the exisiting **nvme-cli** code as little as possible
while reusing the code fromfabrics.c. The majority of changes in the
existing code exports formerly static functions and variables, so that they
are usable from the monitor code.
Quoted 7 lines
Quoted 3 lines
General comment, can you please separate out fixes/cleanups that are not
related to the goal of this patchset?Which ones are you referring to? 09 and 19? While these are minor improvements to the existing code, I wouldn't say they qualify as fixes or cleanups. They aren't necessary without adding the monitor code.
Patches that are not directly related to the goal of this should be splitted out to ease the review.
Quoted 17 lines
But yes, I can post all changes to existing code separately.
Quoted 11 lines
Quoted 10 lines
* When "add" uevents for nvme controller devices are received,
the
controller is consistently not inlivestate yet, and
attempting to read
thesubsysnqnsysfs attribute returns(efault). While this
should
arguably be fixed in the kernel, it could be worked around in
user space
by using timers or polling thestatesysfs attribute for
changes.Quoted 3 lines
This is a bug, what in the code causes this? nothing in controller state
should prevent from this sysfs read from executing correctly...
Quoted 5 lines
I think it can be fixed by making
nvme_sysfs_show_subsysnqn()fall back toctrl->opts->subsysnqnifctrl->subsysis NULL.I'll send a patch. Anyway, it'll take time until this is fixed everywhere.
Thanks.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
* Parse and handle
discovery.confon startup.This is a must I think, where do you get the known transport addresses
on startup today?
Quoted 5 lines
There's a systemd service that runs "nvme connect-all" once during boot. That exists today. I'm not sure if it should be integrated in the monitor, perhaps it's good to keep these separate. People who don't need the monitor can still run the existing service only, whereas for others, the two would play together just fine.
Then doesn't this service need to run after it?
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
* Implement support for RDMA and TCP protocols.
What is needed for supporting them? Not sure I follow (I thought you mentioned that you tested against linux nvmet-rdma?)
AENs over existing discovery controllers are supported for all transports. But there's no support for discovery of new transports except for NVMeoFC's "
fc_udev_device" mechanism.
I see. thanks.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 28 lines · Sagi Grimberg
Quoted 20 lines
Quoted 18 lines
Quoted 6 lines
This method for discovery and autodetection has some advantages over the
current udev-rule based approach:Quoted 11 lines
* By using the
--persistentoption, users can easily control
whether
persistent discovery controllers for discovered transport addresses should
be created and monitored for AEN events. **nvme monitor** watches known
transport addresses, creates discovery controllers as required,
and re-uses
existing ones if possible.Quoted 1 line
What does that mean?
Quoted 7 lines
In general, if the monitor detects a new host_traddr/traddr/trsvcid tuple, it runs a discovery on it, and keeps the discovery controller
open if --persistent was given. On startup, it scans existing controllers, and if it finds already existing discovery controllers,
re-uses them. These will not be shut down when the monitor exits.
Quoted 3 lines
And if it doesn't run with --persistent it deletes them? even if these
weren't created by it?
No, not those. That's what I wanted to express with "These will not be shut down when the monitor exits."
It's not trivial to determine reliably which controller the monitor created and which it didn't. The current code assumes that all discovery controllers that didn't exist at startup were created by the monitor. We can improve the intelligence of the tool in that area, of course. Currently it makes this dumb assumption.
Quoted 4 lines
And, if none exist where does it get new discovery controller details?
Today we use discovery.conf for that.
As I said, this can be done with a separate "nvme connect-all" call, which could be run from an ExecStartPre or from a separate service. Or we can integrate the functionality in the monitor. I don't have a strong opinion either way. I can adapt to what you guys prefer.
Quoted 18 lines
Quoted 18 lines
This allows users fine-grained control about what discovery controllers
to operate persistently. Users who want all discovery controllers to be
persistent just use --persistent. Others can set up those that they want to have (manually or with a script), and not use --persistent.The background is that hosts may not need every detected discovery controller to be persistent. In multipath scenarios, you may see more
discovery subsystems than anything else, and not everyone likes that.
That's a generic issue and unrelated to the monitor, but running the
monitor with --persistent creates discovery controllers that would otherwise not be visible.
Quoted 6 lines
Quoted 1 line
Hope this clarifies it.
Well, How do people expect to know if stuff change without a persistent
discovery controller? Not sure if people may see more discovery controllers this is a real issue given what they are getting from it.
You're right. At this very early stage, I wanted to give users the freedom of choice, and not force dozens of discovery controller connections upon everyone. If there's consensus that the discovery connections should always be created, fine with me.
Quoted 9 lines
Quoted 7 lines
I agree. But it's not easy to fix the issue otherwise. In the customer
problem where we observed it, I worked around it by adding the udev seqnum to the "instance name" of the systemd service, thus allowing several "nvme connect-all" processes to run for the same transport address simultaneously. But I don't think that would scale well; the monitor can handle it more cleanly.Still changing the entire thing for a corner case...
It's just one puzzle piece.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 13 lines
* Resource consumption for handling uevents is lower. Instead of
running an
udev worker, executing the rules, executingsystemctl start
from the
worker, starting a systemd service, and starting a separate
**nvme-cli**
instance, only a singlefork()operation is necessary. Of
course, on the
back side, the monitor itself consumes resources whileQuoted 8 lines
it's
running and
waiting for events. On my system with 8 persistent discovery
controllers,
its RSS is ~3MB. CPU consumption is zero as long as no events
occur.What is the baseline with what we have today?
Quoted 7 lines
Quoted 3 lines
A meaningful comparsion is difficult and should be done when the monitor functionality is finalized. I made this statement only to provide a rough idea of the resource usage, not more.
You mention that the utilization is lower than what we do today, hence
my question is by how much?
I'll try to figure it out and come up with something. Sorry for the hand-waving assertion. I admint It was speculative for the most part.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
* **nvme monitor** could be easily extended to handle events for
non-FC
transports.Which events?
Quoted 7 lines
Quoted 3 lines
Network discovery, mDNS or the like. I haven't digged into the details
yet.Yes, that is possible, would probably be easier to do this in a higher
level language but libavahi can also do this...
Enzo Matsumiya has been working on it, and we've been discussing how to integrate his functionality into the monitor.
Quoted 15 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
* Parse and handle
discovery.confon startup.This is a must I think, where do you get the known transport addresses
on startup today?Quoted 7 lines
There's a systemd service that runs "nvme connect-all" once during boot. That exists today. I'm not sure if it should be integrated in the
monitor, perhaps it's good to keep these separate. People who don't need the monitor can still run the existing service only, whereas for
others, the two would play together just fine.
Quoted 1 line
Then doesn't this service need to run after it?
Not necessarily. It can handle controllers that are created by other processes while it's running.
Thanks for your comments,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 77 lines · Martin Wilck
Quoted 28 lines · Sagi Grimberg
Quoted 20 lines
Quoted 18 lines
Quoted 6 lines
This method for discovery and autodetection has some advantages over the
current udev-rule based approach:Quoted 11 lines
* By using the
--persistentoption, users can easily
control
whether
persistent discovery controllers for discovered transport
addresses should
be created and monitored for AEN events. **nvme monitor**
watches known
transport addresses, creates discovery controllers as
required,
and re-uses
existing ones if possible.Quoted 1 line
What does that mean?
Quoted 7 lines
In general, if the monitor detects a new host_traddr/traddr/trsvcid tuple, it runs a discovery on it, and keeps the discovery controller
open if --persistent was given. On startup, it scans existing controllers, and if it finds already existing discovery controllers,
re-uses them. These will not be shut down when the monitor exits.Quoted 3 lines
And if it doesn't run with --persistent it deletes them? even if these
weren't created by it?No, not those. That's what I wanted to express with "These will not be shut down when the monitor exits."
It's not trivial to determine reliably which controller the monitor created and which it didn't. The current code assumes that all discovery controllers that didn't exist at startup were created by the monitor. We can improve the intelligence of the tool in that area, of course. Currently it makes this dumb assumption.
Quoted 4 lines
And, if none exist where does it get new discovery controller details?
Today we use discovery.conf for that.As I said, this can be done with a separate "nvme connect-all" call, which could be run from an ExecStartPre or from a separate service. Or we can integrate the functionality in the monitor. I don't have a strong opinion either way. I can adapt to what you guys prefer.
Quoted 18 lines
Quoted 18 lines
This allows users fine-grained control about what discovery controllers
to operate persistently. Users who want all discovery controllers to be
persistent just use --persistent. Others can set up those that they want to have (manually or with a script), and not use --persistent.The background is that hosts may not need every detected discovery controller to be persistent. In multipath scenarios, you may see more
discovery subsystems than anything else, and not everyone likes that.
That's a generic issue and unrelated to the monitor, but running the
monitor with --persistent creates discovery controllers that would otherwise not be visible.Quoted 6 lines
Quoted 1 line
Hope this clarifies it.
Well, How do people expect to know if stuff change without a persistent
discovery controller? Not sure if people may see more discovery controllers this is a real issue given what they are getting from it.
Quoted 6 lines
You're right. At this very early stage, I wanted to give users the freedom of choice, and not force dozens of discovery controller connections upon everyone. If there's consensus that the discovery connections should always be created, fine with me.
I would make the monitor code to always create persistent discovery connections. (Or, at least, _attempt_ to create persistent discovery connections.)
_if_ the user starts the monitor he will have a reason for it; and that reason typically is that he _wants_ the system to react on discovery changes. And that is precisely why persistent discovery controllers exist.
Plus there is a provision in the spec allowing the controller to reject persistent connections, so if the controller is worried about increased resource consumption he can just disallow persistent controller connections.
Similarly, I would keep the created discovery connections after exit.
Currently the simple reason is that we don't store the information from the created discovery connections, so if we tear them down on exit we won't be able to recreate them once we restart the monitor.
This situation might change with once we have mDNS integration; and, of course, for FC figuring out this information is trivial.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
I agree. But it's not easy to fix the issue otherwise. In the customer
problem where we observed it, I worked around it by adding the udev seqnum to the "instance name" of the systemd service, thus allowing several "nvme connect-all" processes to run for the same transport address simultaneously. But I don't think that would scale well; the monitor can handle it more cleanly.
Quoted 4 lines
Quoted 1 line
Still changing the entire thing for a corner case...
It's just one puzzle piece.
And it's actually not a corner case, but a real issue we're seeing in our (or rather, our customers) deployments. AENs are being received at the same time, causing 'nvme discover' to run in parallel with the same information. A very nice exercise in finding all race conditions in the code.
And completely pointless to boot.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Hi,
Quoted 7 lines · On 01/26, [email protected] wrote:
From: Martin Wilck <[email protected]>
(Cover letter copied from https://github.com/linux-nvme/nvme-cli/pull/877)
This patch set adds a new subcommand **nvme monitor**. In this mode, **nvme-cli** runs continuously, monitors events (currently, uevents) relevant for discovery, and optionally autoconnects to newly discovered subsystems.
Is there anything missing from this patch set?
Quoted 1 line
* Implement support for RDMA and TCP protocols.
I'm working on the RDMA and TCP implementations with mDNS/Avahi support, and I would like to submit my work based on this monitor solution.
Cheers,
Enzo
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 15 lines · Enzo Matsumiya
Hi,
Quoted 11 lines · On 01/26, [email protected] wrote:
From: Martin Wilck <[email protected]>
(Cover letter copied from
https://github.com/linux-nvme/nvme-cli/pull/877)This patch set adds a new subcommand **nvme monitor**. In this mode,
**nvme-cli** runs continuously, monitors events (currently, uevents) relevant
for discovery, and optionally autoconnects to newly discovered subsystems.
Quoted 7 lines
Is there anything missing from this patch set?
Quoted 1 line
* Implement support for RDMA and TCP protocols.
I'm working on the RDMA and TCP implementations with mDNS/Avahi support, and
I would like to submit my work based on this monitor solution.
I'm still working on a v2. Please have some more patience.
Regards,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme