SUSE SLES resync
Subsystems (2)
cover SUSE SLES resync
Hi Christophe,
here are the patches which I've queued up during SLES12 development. Most of them are (more or less) obvious bugfixes, but there are some things which would warrant a deeper review:
- device-mapper cookie handling: This _really_ is an arcane topic, and I would love to have someone from the device-mapper team to review it. I'm still facing some issues where udev emits a warning 'conflicting device node /dev/mapper/XXX ..', so there might be some issue with it.
- Asynchronous configuration: On large systems configure() as being called during multipathd startup might exceed the systemd job timeout, resulting in multipathd being killed by systemd. For that I've been implementing an udev retrigger method, which just issues an 'add' event for all available block devices.
- Related to the above: I've implemented an asynchronous 'reconfigure' CLI call, as for the same reason systemd might kill the multipathd during a 'restart' operation
- systemd/udev integration: This is _a pain_. It took me months to come up with a working solution. _Especially_ as apparently RH is shipping an '11-dm-multipath.rules' udev ruleset for multipathd, which supposedly should be handling this.
It didn't do it for me, though, so I've attached my own version here. The udev integration I've been settling for works like this:
- 11-dm-mpath.rules checks if the device is eligible for multipathing For that I've implemented a new option '-u' which looks at the uevent variables _only_.
- if a device should be handled by multipathing the 'by-id' and 'by-uuid' persistent symlinks are _not_ generated, as this would trigger systemd to start scanning/mounting/lvm activation/whatnot; the 'by-id' and 'by-uuid' symlinks are only generated for the multipath devices themselves.
- I've added a new multipath.rules file which should be inserted between 55-scsi-sg3_id.rules and 58-scsi_sg3_symlink.rules. This will intercept the SCSI IDENT variables and blank them out if the (block) device is handled by multipathing.
- sysfs VPD page handling. I've planned to implement this as a direct replacement of the original scsi_id program, but then it turned out that our WWID handling is actually too generic for that: Both getuid and uid_attribute can basically take _any_ value, so it's impossible to tell if the system uses the original (scsi_id compatible) IDs or not. And as the VPD parsing code can only generate scsi_id compatible IDs we might run into compability issues.
So it's disabled per default. I'd really would like to clean it up, as I suspect that about 95% of the installations are using scsi_id compatible IDs. But for this to happen we need to clarify/restrict the ID generation mechanism.
As usual, comments and reviews are welcome.
The patchset is based upon the 'find_multipaths' patchset from Ben Marzinski; it can be retrieved from
github.com:/hreinecke/multipath-tools.git branch suse-resync
Cheers,
Hannes
Hannes Reinecke (78):
libmultipath: remove compilation warning in devmapper.c
mpath_persist: Do not call exit() from a shared library
libmultipath: filter for missing property in get_refwwid() Double uevent stacksize yet again
discovery: do not fail discovery on individual devices
libmultipath: Prefer deprecated 'getuid' callout
libmultipath: Skip paths with empty wwid
Make systemd installation path configurable
Add multipath rules for systemd support
Fixup multipathd.socket to resolve ordering dependeny Fixup dependencies in multipathd.service
multipathd: set correct PID when running in debug mode Do not print empty device strings during discovery kpartx.rules: do not call blkid
Use 'SCSI_IDENT_.*' as the default property whitelist Fixup wwid blacklist printing
Allow for empty path argument when printing information Disable reassign maps per default
multipathd: implement 'list path <path>' cli command Make checker_put() and prio_put() idempotent Remove trailing linefeed from sysfs attributes
multipath: implement option '-u' for uevents Install multipath rule under '56-multipath.rules' multipath.rules: Whitelist devices multipath.rules: fixup race condition with systemd 11-dm-mpath.rules: Import blkid values if all paths are down kpartx.rules: Skip kpartx for multipath events
multipathd: handle DOMAP_RETRY
multipathd: cleanup foreground operation
Update hwtable for EMC XtremIO
multipath: check for running daemon when called with '-u' Revert 'return PATH_DOWN for quiesced paths'
Do not treat 'transport-offline' paths as 'offline' Check for valid DM_DEVICE_INFO before proceeding Separate out uevent parsing functions
Use poll() when receiving uevents
mpath_persist: cleanup
kpartx: use standard 'major' and 'minor' macros
multipath: Use standard 'major' macro
Remove sysfs_get_dev
Add paths with a size of '0' as 'ghost' paths Remove last argument from verify_paths()
Fixup device-mapper 'cookie' handling
multipath: do not print state 'orphan' for option '-l' Return error when receiving CLI packet
Implement 'uxsock_timeout' keyword
Do not print empty multipaths section
multipathd: reload map if reinstate failed
multipathd: do not remove paths without uevents Allow zero-sized devices during configuration Rework uev_add_path()
multipathd: Issue warning on CLI command timeout Use strlen() when checking for valid wwid
multipathd: Use standard lists for CLI handling
uxlsnr: use typedef for trigger function
multipathd: lock cli client list
multipath: enable sync support
Remove dm_udev_XXX wrapper functions
Revert to ACT_RELOAD in domap if the map exists
multipathd: use local variable for watchdog configuration Ignore devices when sysfs_get_tgt_nodename fails Skip USB devices during discovery
Read wwid from sysfs vpg_pg83 attribute
Assign local priority for NAA VPD descriptor
Use sysfs attribute vpd_pg80 to read serial number Update multipath.conf.5 to clarify wwid generation
libmultipath: Fall back to SG_IO if no UID could be assigned
libmultipath: unset 'uid_attribute' on failure Separate out vpd parsing functions
multipathd: use SG_IO as fallback to generate uid Do not automatically fall back to vpd uid generation
libmultipath: make vpd page 0x80 optional
multipathd: push down lock in checkerloop()
Allow specific CLI commands to run unlocked
Push down vector lock during uevent processing
multipathd: timeout CLI commands when waiting for lock
multipathd: asynchronous configuration
multipathd: trigger all devices on startup
Makefile.inc | 8 +- kpartx/devmapper.c | 53 +++- kpartx/devmapper.h | 11 +- kpartx/kpartx.c | 35 +-- kpartx/kpartx.rules | 10 +- libmpathpersist/mpath_persist.c | 172 ++++++----- libmpathpersist/mpath_updatepr.c | 20 +- libmpathpersist/mpathpr.h | 3 +- libmultipath/blacklist.c | 8 +- libmultipath/blacklist.h | 2 +- libmultipath/checkers.c | 2 +- libmultipath/config.c | 2 + libmultipath/config.h | 8 +- libmultipath/configure.c | 82 ++++-- libmultipath/defaults.h | 4 +- libmultipath/devmapper.c | 77 +++-- libmultipath/devmapper.h | 4 +- libmultipath/dict.c | 33 +++ libmultipath/discovery.c | 613 ++++++++++++++++++++++++++++++++++----- libmultipath/discovery.h | 8 +- libmultipath/hwtable.c | 13 + libmultipath/list.h | 49 +++- libmultipath/lock.h | 4 +- libmultipath/log_pthread.c | 2 +- libmultipath/print.c | 20 +- libmultipath/prio.c | 4 +- libmultipath/propsel.c | 6 +- libmultipath/structs_vec.c | 29 +- libmultipath/structs_vec.h | 2 +- libmultipath/sysfs.c | 53 ++++ libmultipath/sysfs.h | 2 + libmultipath/uevent.c | 316 ++++++++++++-------- libmultipath/uxsock.c | 32 +- libmultipath/uxsock.h | 4 +- multipath.conf.defaults | 2 +- multipath/11-dm-mpath.rules | 8 +- multipath/Makefile | 2 + multipath/dev_t.h | 3 - multipath/main.c | 53 +++- multipath/multipath.8 | 6 +- multipath/multipath.conf.5 | 42 ++- multipath/multipath.rules | 14 + multipathd/cli.c | 47 ++- multipathd/cli.h | 4 +- multipathd/cli_handlers.c | 64 +++- multipathd/cli_handlers.h | 1 + multipathd/main.c | 400 ++++++++++++++++--------- multipathd/main.h | 2 + multipathd/multipathd.service | 4 +- multipathd/multipathd.socket | 8 +- multipathd/uxclnt.c | 26 +- multipathd/uxclnt.h | 2 +- multipathd/uxlsnr.c | 102 +++++-- multipathd/uxlsnr.h | 8 +- 54 files changed, 1834 insertions(+), 655 deletions(-) delete mode 100644 multipath/dev_t.h create mode 100644 multipath/multipath.rules
Signature · 1 line
1.8.4.5
PATCH 1/78 libmultipath: remove compilation warning in devmapper.c
When LIBDM_API_DEFERRED is not defined the compiler would warn about 'dm_cancel_remove_partmaps' being defined but never used.
--- libmultipath/devmapper.c | 2 ++ 1 file changed, 2 insertions(+)
libmultipath/devmapper.c +2 -0
@@ -32,7 +32,9 @@#define UUID_PREFIX "mpath-"#define UUID_PREFIX_LEN 6 +#ifdef LIBDM_API_DEFERRED static int dm_cancel_remove_partmaps (const char * mapname); +#endif #ifndef LIBDM_API_COOKIE static inline int dm_task_set_cookie (struct dm_task *dmt,uint32_t *c,int a)
Signature · 1 line
1.8.4.5
PATCH 2/78 mpath_persist: Do not call exit() from a shared library
In reply to Hannes Reineckempath_persist is a shared library, and shouldn't call exit(). Just return an error code here.
And cleanup formatting while we're at it.
--- libmpathpersist/mpath_persist.c | 52 ++++++++++++++++++++++++----------------- 1 file changed, 31 insertions(+), 21 deletions(-)
libmpathpersist/mpath_persist.c +31 -21
@@ -386,18 +386,20 @@ void * mpath_prin_pthread_fn (void *p)int ret;struct prin_param * pparam = (struct prin_param *)p; - ret =prin_do_scsi_ioctl (pparam->dev, pparam->rq_servact, pparam->resp, pparam->noisy); - pparam->status = ret; -pthread_exit (NULL ); + ret =prin_do_scsi_ioctl (pparam->dev, pparam->rq_servact, + pparam->resp, pparam->noisy); + pparam->status = ret; +pthread_exit (NULL ); } -int mpath_send_prin_activepath (char * dev,int rq_servact,struct prin_resp * resp,int noisy) +int mpath_send_prin_activepath (char * dev,int rq_servact, +struct prin_resp * resp,int noisy) {int rc; rc =prin_do_scsi_ioctl (dev, rq_servact, resp, noisy); - +return (rc); }
@@ -409,14 +411,14 @@ int mpath_prout_reg(struct multipath *mpp,int rq_servact, int rq_scope,struct pathgroup *pgp =NULL ;struct path *pp =NULL ;int rollback =0 ; -int active_pathcount=0 ; +int active_pathcount=0 ;int rc;int count=0 ;int status =MPATH_PR_SUCCESS ;uint64_t sa_key =0 ;if (!mpp) -return MPATH_PR_DMMP_ERROR ; +return MPATH_PR_DMMP_ERROR ; active_pathcount =pathcount (mpp,PATH_UP ) +pathcount (mpp,PATH_GHOST );
@@ -444,12 +446,13 @@ int mpath_prout_reg(struct multipath *mpp,int rq_servact, int rq_scope,condlog (3 ,"THRED ID [%d] INFO]" , i);condlog (3 ,"rq_servact=%d " , thread[i].param.rq_servact); -condlog (3 ,"rq_scope=%d " , thread[i].param.rq_scope); -condlog (3 ,"rq_type=%d " , thread[i].param.rq_type); -condlog (3 ,"rkey=" ); -condlog (3 ,"paramp->sa_flags =%02x " , thread[i].param.paramp->sa_flags); -condlog (3 ,"noisy=%d " , thread[i].param.noisy); -condlog (3 ,"status=%d " , thread[i].param.status); +condlog (3 ,"rq_scope=%d " , thread[i].param.rq_scope); +condlog (3 ,"rq_type=%d " , thread[i].param.rq_type); +condlog (3 ,"rkey=" ); +condlog (3 ,"paramp->sa_flags =%02x " , + thread[i].param.paramp->sa_flags); +condlog (3 ,"noisy=%d " , thread[i].param.noisy); +condlog (3 ,"status=%d " , thread[i].param.status); }pthread_attr_t attr;
@@ -548,13 +551,15 @@ int mpath_prout_common(struct multipath *mpp,int rq_servact, int rq_scope, vector_foreach_slot (mpp->pg, pgp, j){ vector_foreach_slot (pgp->paths, pp, i){if (!((pp->state ==PATH_UP ) || (pp->state ==PATH_GHOST ))){ -condlog (1 ,"%s: %s path not up. Skip" , mpp->wwid, pp->dev); +condlog (1 ,"%s: %s path not up. Skip" , + mpp->wwid, pp->dev);continue ; }condlog (3 ,"%s: sending pr out command to %s" , mpp->wwid, pp->dev); - ret =send_prout_activepath (pp->dev, rq_servact, rq_scope, rq_type, - paramp, noisy); + ret =send_prout_activepath (pp->dev, rq_servact, + rq_scope, rq_type, + paramp, noisy);return ret ; } }
@@ -585,7 +590,7 @@ int send_prout_activepath(char * dev, int rq_servact, int rq_scope, rc =pthread_create (&thread, &attr, mpath_prout_pthread_fn, (void *)(¶m));if (rc){condlog (3 ,"%s: failed to create thread %d" , dev, rc); -exit (-1 ); +return MPATH_PR_OTHER ; }/* Free attribute and wait for the other threads */ pthread_attr_destroy (&attr);
@@ -723,16 +728,21 @@ int mpath_prout_rel(struct multipath *mpp,int rq_servact, int rq_scope,condlog (3 ,"%s: reservation key set." , mpp->wwid); } -mpath_prout_common (mpp,MPATH_PROUT_CLEAR_SA , rq_scope, rq_type, pamp, - noisy); + status =mpath_prout_common (mpp,MPATH_PROUT_CLEAR_SA , + rq_scope, rq_type, pamp, noisy); + +if (status) { +condlog (0 ,"%s: failed to send CLEAR_SA" , mpp->wwid); + goto out1; + } pamp->num_transportid =1 ; pptr=pamp->trnptid_list[0 ];for (i =0 ; i < num; i++){ -if (mpp->reservation_key && +if (mpp->reservation_key &&memcmp (pr_buff->prin_descriptor.prin_readfd.descriptors[i]->key, - mpp->reservation_key,8 )){ + mpp->reservation_key,8 )){/*register with tarnsport id*/ memset (pamp,0 , length); pamp->trnptid_list[0 ] = pptr;
Signature · 1 line
1.8.4.5
PATCH 3/78 libmultipath: filter for missing property in get_refwwid()
In reply to Hannes Reineckeget_refwwid() needs to filter for missing udev properties, otherwise it'll accept just about any SCSI device.
--- libmultipath/configure.c | 6 ++++++ 1 file changed, 6 insertions(+)
libmultipath/configure.c +6 -0
@@ -962,6 +962,9 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid)return ret; } } +if (pp->udev &&filter_property (conf, pp->udev) >0 ) +return 2 ; + refwwid = pp->wwid; goto out; }
@@ -986,6 +989,9 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid)return ret; } } +if (pp->udev &&filter_property (conf, pp->udev) >0 ) +return 2 ; + refwwid = pp->wwid; goto out; }
Signature · 1 line
1.8.4.5
PATCH 4/78 Double uevent stacksize yet again
In reply to Hannes ReineckeAnother report indicates that the uevent stacksize might still be too small.
--- libmultipath/defaults.h | 1 + multipathd/main.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
libmultipath/defaults.h +1 -0
@@ -19,6 +19,7 @@#define DEFAULT_DETECT_PRIO DETECT_PRIO_OFF#define DEFAULT_DEFERRED_REMOVE DEFERRED_REMOVE_OFF#define DEFAULT_DELAY_CHECKS DELAY_CHECKS_OFF +#define DEFAULT_UEVENT_STACKSIZE 256#define DEFAULT_CHECKINT 5#define MAX_CHECKINT (a) (a << 2)
multipathd/main.c +1 -1
@@ -1710,7 +1710,7 @@ child (void * param) udev =udev_new ();setup_thread_attr (&misc_attr,64 *1024 ,1 ); -setup_thread_attr (&uevent_attr,128 *1024 ,1 ); +setup_thread_attr (&uevent_attr,DEFAULT_UEVENT_STACKSIZE *1024 ,1 );setup_thread_attr (&waiter_attr,32 *1024 ,1 );if (logsink ==1 ) {
Signature · 1 line
1.8.4.5
PATCH 5/78 discovery: do not fail discovery on individual devices
In reply to Hannes ReineckeAny errors returned from individual devices during discovery do not invalidate the entire discovery. So instead of returning the pointless accumulated error count we should be returning the number of devices which which could not be discovered or the negative error number if the discovery itself failed.
--- libmpathpersist/mpath_persist.c | 14 ++++++------- libmultipath/discovery.c | 46 ++++++++++++++++++++--------------------- libmultipath/discovery.h | 4 ++++ multipath/main.c | 2 +- multipathd/main.c | 6 ++++-- 5 files changed, 38 insertions(+), 34 deletions(-)
libmpathpersist/mpath_persist.c +7 -7
@@ -178,7 +178,7 @@ int mpath_persistent_reserve_in (int fd, int rq_servact, struct prin_resp *resp, goto out; } -if (path_discovery (pathvec, conf,DI_SYSFS |DI_CHECKER )) { +if (path_discovery (pathvec, conf,DI_SYSFS |DI_CHECKER ) <0 ) { ret =MPATH_PR_DMMP_ERROR ; goto out1; }
@@ -262,13 +262,13 @@ int mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope, curmp =vector_alloc (); pathvec =vector_alloc (); -if (!curmp || !pathvec){ -condlog (0 ,"%s: vector allocation failed." , alias); - ret =MPATH_PR_DMMP_ERROR ; - goto out; - } +if (!curmp || !pathvec){ +condlog (0 ,"%s: vector allocation failed." , alias); + ret =MPATH_PR_DMMP_ERROR ; + goto out; + } -if (path_discovery (pathvec, conf,DI_SYSFS |DI_CHECKER )) { +if (path_discovery (pathvec, conf,DI_SYSFS |DI_CHECKER ) <0 ) { ret =MPATH_PR_DMMP_ERROR ; goto out1; }
libmultipath/discovery.c +22 -24
133 lines not on this page · raw message
libmultipath/discovery.h +4 -0
@@ -24,6 +24,10 @@#define SCSI_COMMAND_TERMINATED 0x22#define SG_ERR_DRIVER_SENSE 0x08 +#define PATHINFO_OK 0 +#define PATHINFO_FAILED 1 +#define PATHINFO_SKIPPED 2 +struct config ;ssize_t sysfs_get_dev (struct udev_device *udev,char * buff,size_t len);
multipath/main.c +1 -1
@@ -343,7 +343,7 @@ configure (void)/* maximum info */ di_flag =DI_ALL ; -if (path_discovery (pathvec, conf, di_flag)) +if (path_discovery (pathvec, conf, di_flag) <0 ) goto out;if (conf->verbosity >2 )
multipathd/main.c +4 -2
@@ -1409,7 +1409,7 @@ configure (struct vectors * vecs, int start_waiters)struct multipath * mpp;struct path * pp;vector mpvec; -int i; +int i, ret;if (!vecs->pathvec && !(vecs->pathvec =vector_alloc ()))return 1 ;
@@ -1423,7 +1423,9 @@ configure (struct vectors * vecs, int start_waiters)/* * probe for current path (from sysfs) and map (from dm) sets */ -path_discovery (vecs->pathvec, conf,DI_ALL ); + ret =path_discovery (vecs->pathvec, conf,DI_ALL ); +if (ret <0 ) +return 1 ; vector_foreach_slot (vecs->pathvec, pp, i){if (filter_path (conf, pp) >0 ){
Signature · 1 line
1.8.4.5
PATCH 6/78 libmultipath: Prefer deprecated 'getuid' callout
In reply to Hannes ReineckeWe need first to check if the deprecated 'getuid' callout should be used, otherwise it might never be called.
--- libmultipath/propsel.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
libmultipath/propsel.c +3 -3
@@ -351,12 +351,12 @@ select_getuid (struct path * pp) {char *origin; -pp_set_ovr (uid_attribute);pp_set_ovr (getuid); -pp_set_hwe (uid_attribute); +pp_set_ovr (uid_attribute);pp_set_hwe (getuid); -pp_set_conf (uid_attribute); +pp_set_hwe (uid_attribute);pp_set_conf (getuid); +pp_set_conf (uid_attribute);pp_set_default (uid_attribute,DEFAULT_UID_ATTRIBUTE ); out:if (pp->uid_attribute)
Signature · 1 line
1.8.4.5
PATCH 7/78 libmultipath: Skip paths with empty wwid
In reply to Hannes ReineckeIf the WWID is empty this is pretty much a terminal error, as we don't have anything to identify the multipath device. So skip those devices during discovery.
--- libmultipath/discovery.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
libmultipath/discovery.c +4 -1
@@ -1230,7 +1230,8 @@ pathinfo (struct path *pp, vector hwtable, int mask)if ((mask &DI_WWID ) && !strlen (pp->wwid))get_uid (pp);if (mask &DI_BLACKLIST && mask &DI_WWID ) { -if (filter_wwid (conf->blist_wwid, conf->elist_wwid, +if (!strlen (pp->wwid) || +filter_wwid (conf->blist_wwid, conf->elist_wwid, pp->wwid) >0 ) {return PATHINFO_SKIPPED ; }
@@ -1244,6 +1245,8 @@ pathinfo (struct path *pp, vector hwtable, int mask)if (pp->state !=PATH_DOWN || pp->priority ==PRIO_UNDEF ) {if (!strlen (pp->wwid))get_uid (pp); +if (!strlen (pp->wwid)) +return PATHINFO_SKIPPED ;get_prio (pp); } }
Signature · 1 line
1.8.4.5
PATCH 8/78 Make systemd installation path configurable
In reply to Hannes ReineckeOlder systems might have systemd installed in /lib, so make the path for systemd and udev configurable.
--- Makefile.inc | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
Makefile.inc +6 -2
@@ -27,10 +27,14 @@ ifndef SYSTEMD endif endif +ifndef SYSTEMDPATH + SYSTEMDPATH=usr/lib +endif + prefix = exec_prefix = $(prefix) bindir = $(exec_prefix)/sbin -libudevdir = ${prefix}/lib/udev +libudevdir = $(prefix)/$(SYSTEMDPATH)/udev udevrulesdir = $(libudevdir)/rules.d multipathdir = $(TOPDIR)/libmultipath mandir = $(prefix)/usr/share/man/man8
@@ -39,7 +43,7 @@ man3dir = $(prefix)/usr/share/man/man3 rcdir = $(prefix)/etc/init.d syslibdir = $(prefix)/$(LIB) libdir = $(prefix)/$(LIB)/multipath -unitdir = $(prefix)/usr/lib/systemd/system +unitdir = $(prefix)/$(SYSTEMDPATH)/systemd/system mpathpersistdir = $(TOPDIR)/libmpathpersist GZIP = gzip -9 -c
Signature · 1 line
1.8.4.5
PATCH 9/78 Add multipath rules for systemd support
In reply to Hannes ReineckeSystemd and dracut rely on the DM_MULTIPATH_DEVICE_PATH setting to mark the devices as used by multipathing. So add a ruleset providing this setting.
--- multipath/Makefile | 2 ++ multipath/multipath.rules | 13 +++++++++++++ 2 files changed, 15 insertions(+) create mode 100644 multipath/multipath.rules
multipath/Makefile +2 -0
@@ -23,6 +23,7 @@ install: $(INSTALL_PROGRAM) -m 755 $(EXEC) $(DESTDIR)$(bindir)/ $(INSTALL_PROGRAM) -d $(DESTDIR)$(udevrulesdir) $(INSTALL_PROGRAM) -m 644 11-dm-mpath.rules $(DESTDIR)$(udevrulesdir) + $(INSTALL_PROGRAM) -m 644 $(EXEC).rules $(DESTDIR)$(libudevdir)/rules.d/40-multipath.rules $(INSTALL_PROGRAM) -d $(DESTDIR)$(mandir) $(INSTALL_PROGRAM) -m 644 $(EXEC).8.gz $(DESTDIR)$(mandir) $(INSTALL_PROGRAM) -d $(DESTDIR)$(man5dir)
@@ -31,6 +32,7 @@ install: uninstall: rm $(DESTDIR)$(bindir)/$(EXEC) rm $(DESTDIR)$(udevrulesdir)/11-dm-mpath.rules + rm $(DESTDIR)$(libudevdir)/rules.d/40-multipath.rules rm $(DESTDIR)$(mandir)/$(EXEC).8.gz rm $(DESTDIR)$(man5dir)/$(EXEC).conf.5.gz
multipath/multipath.rules new file +13 -0
@@ -0,0 +1,13 @@ +# Set DM_MULTIPATH_DEVICE_PATH if the device should be handled by multipath +SUBSYSTEM!="block", GOTO="end_mpath" + +ENV{MPATH_SBIN_PATH}="/sbin" +TEST!="$env{MPATH_SBIN_PATH}/multipath", ENV{MPATH_SBIN_PATH}="/usr/sbin" + +SUBSYSTEM=="block", ACTION=="add|change", KERNEL!="dm-*", \ + ENV{DM_MULTIPATH_DEVICE_PATH}!="1", \ + PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -v 0 -c $tempnode", \ + ENV{DM_MULTIPATH_DEVICE_PATH}="1" \ + ENV{SYSTEMD_READY}="0" + +LABEL="end_mpath"
Signature · 1 line
1.8.4.5
PATCH 10/78 Fixup multipathd.socket to resolve ordering dependeny
In reply to Hannes ReineckePer default any socket file is started after /run is available, as the socket might be placed there.
This results in a dependency cycle as multipathd requires the socket, but the daemon should be started as early as possible, and certainly before /run is mounted.
--- multipathd/multipathd.service | 3 ++- multipathd/multipathd.socket | 8 +++++--- 2 files changed, 7 insertions(+), 4 deletions(-)
multipathd/multipathd.service +2 -1
@@ -1,8 +1,9 @@ [Unit] Description=Device-Mapper Multipath Device Controller Before=iscsi.service iscsid.service lvm2-activation-early.service -After=syslog.target +Before=local-fs-pre.target DefaultDependencies=no +Wants=multipathd.socket Conflicts=shutdown.target [Service]
multipathd/multipathd.socket +5 -3
@@ -1,5 +1,7 @@ +[Unit] +Description=multipathd control socket +DefaultDependencies=no +Before=sockets.target + [Socket] ListenStream=@/org/kernel/linux/storage/multipathd - -[Install] -WantedBy=sockets.target
Signature · 1 line
1.8.4.5
PATCH 11/78 Fixup dependencies in multipathd.service
In reply to Hannes ReineckeThe multipathd.service should be started after the .socket, and it requires local-fs-pre to be reached. Otherwise it won't be able to access /sys etc.
References: bnc#872712
--- multipathd/multipathd.service | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
multipathd/multipathd.service +2 -1
@@ -2,8 +2,9 @@ Description=Device-Mapper Multipath Device Controller Before=iscsi.service iscsid.service lvm2-activation-early.service Before=local-fs-pre.target +After=multipathd.socket DefaultDependencies=no -Wants=multipathd.socket +Wants=local-fs-pre.target multipathd.socket Conflicts=shutdown.target [Service]
Signature · 1 line
1.8.4.5
PATCH 12/78 multipathd: set correct PID when running in debug mode
In reply to Hannes ReineckeWhen running in debug mode the PID is not set, causing the PID in the pidfile to always be set to '0'.
--- multipathd/main.c | 1 + 1 file changed, 1 insertion(+)
multipathd/main.c +1 -0
@@ -2056,6 +2056,7 @@ main (int argc, char *argv[])if (!isatty (fileno (stdout)))setbuf (stdout,NULL ); err =0 ; + daemon_pid =getpid (); }else err =daemonize ();
Signature · 1 line
1.8.4.5
PATCH 13/78 Do not print empty device strings during discovery
In reply to Hannes ReineckeIf the path provides no information we should not print out empty strings, but rather an instructive message telling us so.
References: bnc#872712
--- libmultipath/structs_vec.c | 6 ++++++ 1 file changed, 6 insertions(+)
libmultipath/structs_vec.c +6 -0
@@ -226,6 +226,12 @@ extract_hwe_from_path(struct multipath * mpp) }if (pp) { +if (!strlen (pp->vendor_id) || + !strlen (pp->product_id) || + !strlen (pp->rev)) { +condlog (3 ,"%s: no device details available" , pp->dev); +return NULL ; + }condlog (3 ,"%s: vendor = %s" , pp->dev, pp->vendor_id);condlog (3 ,"%s: product = %s" , pp->dev, pp->product_id);condlog (3 ,"%s: rev = %s" , pp->dev, pp->rev);
Signature · 1 line
1.8.4.5
PATCH 14/78 kpartx.rules: do not call blkid
In reply to Hannes Reineckeblkid is already called by the device-mapper rules, no need to do it ourselves.
--- kpartx/kpartx.rules | 6 ------ 1 file changed, 6 deletions(-)
kpartx/kpartx.rules +0 -6
@@ -12,12 +12,6 @@ ENV{DM_DEPS}=="0", GOTO="kpartx_end" ENV{DM_UUID}=="?*", IMPORT{program}=="kpartx_id %M %m $env{DM_UUID}" -ENV{DM_ACTION}=="PATH_FAILED", ENV{DM_NR_VALID_PATHS}=="0", \ - GOTO="blkid_end" -ENV{DM_UUID}=="mpath-*", IMPORT{program}="/sbin/blkid -o udev -p $tempnode" -ENV{DM_PART}=="?*", IMPORT{program}="/sbin/blkid -o udev -p $tempnode" -LABEL="blkid_end" - OPTIONS="link_priority=50" # Create persistent links for multipath tables
Signature · 1 line
1.8.4.5
PATCH 15/78 Use 'SCSI_IDENT_.*' as the default property whitelist
In reply to Hannes Reinecke59-scsi-sg_utils.rules export the VPD pages as SCSI_IDENT_<association>_<type>.
So whenever we have a SCSI_IDENT_* property we know it has come from VPD pages and we have a legit device.
--- libmultipath/blacklist.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
libmultipath/blacklist.c +1 -1
@@ -196,7 +196,7 @@ setup_default_blist (struct config * conf)if (store_ble (conf->blist_devnode, str,ORIGIN_DEFAULT ))return 1 ; - str =STRDUP ("(ID_SCSI_VPD|ID_WWN)" ); + str =STRDUP ("(SCSI_IDENT_.*|ID_WWN)" );if (!str)return 1 ;if (store_ble (conf->elist_property, str,ORIGIN_DEFAULT ))
Signature · 1 line
1.8.4.5
PATCH 16/78 Fixup wwid blacklist printing
In reply to Hannes ReineckeWhen a wwid is blacklisted the log message looks like '(null): wwid XXX blacklisted'.
Fix it up by printing either the corresponding device or avoid printing the '(null)' entry if no corresponding device is given.
--- libmultipath/blacklist.c | 6 ++++-- libmultipath/blacklist.h | 2 +- libmultipath/configure.c | 2 +- libmultipath/discovery.c | 2 +- multipathd/cli_handlers.c | 2 +- 5 files changed, 8 insertions(+), 6 deletions(-)
libmultipath/blacklist.c +4 -2
@@ -228,6 +228,8 @@ setup_default_blist (struct config * conf)if (vendor && product) \condlog (3 ,"%s: (%s:%s) %s %s" , \ dev, vendor, product, (M ), (S )); \ +else if (wwid && !dev) \ +condlog (3 ,"%s: %s %s" , wwid, (M ), (S )); \else if (wwid) \condlog (3 ,"%s: %s %s %s" , dev, (M ), wwid, (S )); \else if (env) \
@@ -328,10 +330,10 @@ _filter_wwid (vector blist, vector elist, char * wwid) }int -filter_wwid (vector blist,vector elist,char * wwid) +filter_wwid (vector blist,vector elist,char * wwid,char * dev) {int r =_filter_wwid (blist, elist, wwid); -log_filter (NULL ,NULL ,NULL , wwid,NULL , r); +log_filter (dev,NULL ,NULL , wwid,NULL , r);return r; }
libmultipath/blacklist.h +1 -1
@@ -32,7 +32,7 @@ struct blentry_device {int setup_default_blist (struct config *);int alloc_ble_device (vector );int filter_devnode (vector ,vector ,char *); -int filter_wwid (vector ,vector ,char *); +int filter_wwid (vector ,vector ,char *,char *);int filter_device (vector ,vector ,char *,char *);int filter_path (struct config *,struct path *);int filter_property (struct config *,struct udev_device *);
libmultipath/configure.c +1 -1
@@ -1025,7 +1025,7 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid) check:if (refwwid &&strlen (refwwid)) {if (filter_wwid (conf->blist_wwid, conf->elist_wwid, - refwwid) >0 ) + refwwid,NULL ) >0 )return 2 ; } }
libmultipath/discovery.c +1 -1
@@ -1232,7 +1232,7 @@ pathinfo (struct path *pp, vector hwtable, int mask)if (mask &DI_BLACKLIST && mask &DI_WWID ) {if (!strlen (pp->wwid) ||filter_wwid (conf->blist_wwid, conf->elist_wwid, - pp->wwid) >0 ) { + pp->wwid, pp->dev) >0 ) {return PATHINFO_SKIPPED ; } }
multipathd/cli_handlers.c +1 -1
@@ -494,7 +494,7 @@ cli_add_map (void * v, char ** reply, int * len, void * data) param =convert_dev (param,0 );condlog (2 ,"%s: add map (operator)" , param); -if (filter_wwid (conf->blist_wwid, conf->elist_wwid, param) >0 ) { +if (filter_wwid (conf->blist_wwid, conf->elist_wwid, param,NULL ) >0 ) { *reply =strdup ("blacklisted\n" ); *len =strlen (*reply) +1 ;condlog (2 ,"%s: map blacklisted" , param);
Signature · 1 line
1.8.4.5
PATCH 17/78 Allow for empty path argument when printing information
In reply to Hannes ReineckeWhen calling the various print functions we should be allowing for empty path argument.
--- libmultipath/print.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)
libmultipath/print.c +16 -6
@@ -287,7 +287,7 @@ snprint_path_uuid (char * buff, size_t len, struct path * pp)static int snprint_hcil (char * buff,size_t len,struct path * pp) { -if (pp->sg_id.host_no <0 ) +if (!pp || pp->sg_id.host_no <0 )return snprintf (buff, len,"#:#:#:#" ); return snprintf(buff, len, "%i:%i:%i:%i",
@@ -300,7 +300,7 @@ snprint_hcil (char * buff, size_t len, struct path * pp)static int snprint_dev (char * buff,size_t len,struct path * pp) { -if (!strlen (pp->dev)) +if (!pp || !strlen (pp->dev))return snprintf (buff, len,"-" );else return snprint_str (buff, len, pp->dev);
@@ -309,7 +309,7 @@ snprint_dev (char * buff, size_t len, struct path * pp)static int snprint_dev_t (char * buff,size_t len,struct path * pp) { -if (!strlen (pp->dev)) +if (!pp || !strlen (pp->dev))return snprintf (buff, len,"#:#" );else return snprint_str (buff, len, pp->dev_t);
@@ -318,8 +318,12 @@ snprint_dev_t (char * buff, size_t len, struct path * pp)static int snprint_offline (char * buff,size_t len,struct path * pp) { -if (pp->offline) +if (!pp) +return snprintf (buff, len,"unknown" ); +else if (pp->offline)return snprintf (buff, len,"offline" ); +else if (!pp->mpp) +return snprintf (buff, len,"orphan" );else return snprintf (buff, len,"running" ); }
@@ -327,6 +331,9 @@ snprint_offline (char * buff, size_t len, struct path * pp)static int snprint_chk_state (char * buff,size_t len,struct path * pp) { +if (!pp) +return snprintf (buff, len,"undef" ); +switch (pp->state) {case PATH_UP :return snprintf (buff, len,"ready" );
@@ -350,6 +357,9 @@ snprint_chk_state (char * buff, size_t len, struct path * pp)static int snprint_dm_path_state (char * buff,size_t len,struct path * pp) { +if (!pp) +return snprintf (buff, len,"undef" ); +switch (pp->dmstate) {case PSTATE_ACTIVE :return snprintf (buff, len,"active" );
@@ -370,7 +380,7 @@ snprint_vpr (char * buff, size_t len, struct path * pp)static int snprint_next_check (char * buff,size_t len,struct path * pp) { -if (!pp->mpp) +if (!pp || !pp->mpp)return snprintf (buff, len,"orphan" );return snprint_progress (buff, len, pp->tick, pp->checkint);
@@ -379,7 +389,7 @@ snprint_next_check (char * buff, size_t len, struct path * pp)static int snprint_pri (char * buff,size_t len,struct path * pp) { -return snprint_int (buff, len, pp->priority); +return snprint_int (buff, len, pp ? pp->priority :-1 ); } static int
Signature · 1 line
1.8.4.5
PATCH 18/78 Disable reassign maps per default
In reply to Hannes Reineckelvm has undergone various updates, and should now check for correct device assignment. So disable the 'reassign_maps' feature per default.
--- libmultipath/defaults.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
libmultipath/defaults.h +1 -1
@@ -12,7 +12,7 @@#define DEFAULT_RR_WEIGHT RR_WEIGHT_NONE#define DEFAULT_NO_PATH_RETRY NO_PATH_RETRY_UNDEF#define DEFAULT_VERBOSITY 2 -#define DEFAULT_REASSIGN_MAPS 1 +#define DEFAULT_REASSIGN_MAPS 0#define DEFAULT_FIND_MULTIPATHS 0#define DEFAULT_FAST_IO_FAIL 5#define DEFAULT_RETAIN_HWHANDLER RETAIN_HWHANDLER_OFF
Signature · 1 line
1.8.4.5
PATCH 19/78 multipathd: implement 'list path <path>' cli command
In reply to Hannes ReineckeImplement CLI command for printing the status of a single path.
--- multipathd/cli.c | 1 + multipathd/cli_handlers.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ multipathd/cli_handlers.h | 1 + multipathd/main.c | 1 + 4 files changed, 47 insertions(+)
multipathd/cli.c +1 -0
@@ -425,6 +425,7 @@ cli_init (void) {add_handler (LIST +PATHS ,NULL );add_handler (LIST +PATHS +FMT ,NULL ); +add_handler (LIST +PATH ,NULL );add_handler (LIST +STATUS ,NULL );add_handler (LIST +DAEMON ,NULL );add_handler (LIST +MAPS ,NULL );
multipathd/cli_handlers.c +44 -0
56 lines not on this page · raw message
multipathd/cli_handlers.h +1 -0
@@ -1,5 +1,6 @@int cli_list_paths (void * v,char ** reply,int * len,void * data);int cli_list_paths_fmt (void * v,char ** reply,int * len,void * data); +int cli_list_path (void * v,char ** reply,int * len,void * data);int cli_list_status (void * v,char ** reply,int * len,void * data);int cli_list_daemon (void * v,char ** reply,int * len,void * data);int cli_list_maps (void * v,char ** reply,int * len,void * data);
multipathd/main.c +1 -0
@@ -886,6 +886,7 @@ uxlsnrloop (void * ap)set_handler_callback (LIST +PATHS , cli_list_paths);set_handler_callback (LIST +PATHS +FMT , cli_list_paths_fmt); +set_handler_callback (LIST +PATH , cli_list_path);set_handler_callback (LIST +MAPS , cli_list_maps);set_handler_callback (LIST +STATUS , cli_list_status);set_handler_callback (LIST +DAEMON , cli_list_daemon);
Signature · 1 line
1.8.4.5
PATCH 20/78 Make checker_put() and prio_put() idempotent
In reply to Hannes Reineckechecker_put() and prio_put() should check whether an instance was selected prior to dropping a reference to it. Otherwise they'll be dropping a reference which they haven't increased in the first place.
--- libmultipath/checkers.c | 2 +- libmultipath/prio.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
libmultipath/checkers.c +1 -1
@@ -195,7 +195,7 @@ void checker_put (struct checker * dst) {struct checker * src; -if (!dst) +if (!dst || !dst->check)return ; src =checker_lookup (dst->name);if (dst->free)
libmultipath/prio.c +2 -2
@@ -132,7 +132,7 @@ int prio_getprio (struct prio * p, struct path * pp)int prio_selected (struct prio * p) { -if (!p || !p->getprio) +if (!p)return 0 ;return (p->getprio) ?1 :0 ; }
@@ -169,7 +169,7 @@ void prio_put (struct prio * dst) {struct prio * src; -if (!dst) +if (!dst || !dst->getprio)return ; src =prio_lookup (dst->name);
Signature · 1 line
1.8.4.5
PATCH 21/78 Remove trailing linefeed from sysfs attributes
In reply to Hannes ReineckeWhen reading from sysfs attributes might have a trailing linefeed, which will then show up in the log messages as an additional linefeed.
--- libmultipath/sysfs.c | 2 ++ 1 file changed, 2 insertions(+)
libmultipath/sysfs.c +2 -0
@@ -93,6 +93,8 @@ ssize_t sysfs_attr_get_value(struct udev_device *dev, const char *attr_name, }close (fd); +if (size >0 ) + size =strchop (value);return size; }
Signature · 1 line
1.8.4.5
PATCH 22/78 multipath: implement option '-u' for uevents
In reply to Hannes ReineckeWhen calling 'multipath' from an uevent udev will not have all information for that device, as it's being written into the database _after_ the event has been processed. This patch implements an option '-u' which uses the information from the program environment when checking the device.
--- libmultipath/config.c | 1 + libmultipath/config.h | 3 ++- libmultipath/configure.c | 28 ++++++++++++++++++++++++++-- libmultipath/log_pthread.c | 2 +- multipath/main.c | 21 ++++++++++++++++++--- multipath/multipath.8 | 6 +++++- multipath/multipath.rules | 2 +- 7 files changed, 54 insertions(+), 9 deletions(-)
libmultipath/config.c +1 -0
@@ -617,6 +617,7 @@ load_config (char * file, struct udev *udev) conf->partition_delim =NULL ; conf->processed_main_config =0 ; conf->find_multipaths =DEFAULT_FIND_MULTIPATHS ; + conf->uid_attribute =set_default (DEFAULT_UID_ATTRIBUTE );/* * preload default hwtable
libmultipath/config.h +2 -1
@@ -19,7 +19,8 @@ enum devtypes {DEV_NONE ,DEV_DEVT ,DEV_DEVNODE , -DEV_DEVMAP +DEV_DEVMAP , +DEV_UEVENT }; enum mpath_cmds {
libmultipath/configure.c +26 -2
@@ -957,8 +957,8 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid)udev_device_unref (udevice);if (!pp) {if (ret ==1 ) -condlog (0 ,"%s can't store path info" , - buff); +condlog (0 ,"%s: can't store path info" , + dev);return ret; } }
@@ -995,6 +995,30 @@ get_refwwid (char * dev, enum devtypes dev_type, vector pathvec, char **wwid) refwwid = pp->wwid; goto out; } + +if (dev_type ==DEV_UEVENT ) { +struct udev_device *udevice =udev_device_new_from_environment (conf->udev); + +if (!udevice) { +condlog (2 ,"%s: can't get udev device" , dev); +return 1 ; + } + ret =store_pathinfo (pathvec, conf->hwtable, udevice, +DI_SYSFS |DI_WWID , &pp); +udev_device_unref (udevice); +if (!pp) { +if (ret ==1 ) +condlog (0 ,"%s: can't store path info" , + dev); +return ret; + } +if (pp->udev &&filter_property (conf, pp->udev) >0 ) +return 2 ; + + refwwid = pp->wwid; + goto out; + } +if (dev_type ==DEV_DEVMAP ) {if (((dm_get_uuid (dev, tmpwwid)) ==0 ) && (strlen (tmpwwid))) {
libmultipath/log_pthread.c +1 -1
@@ -25,7 +25,7 @@ int logq_running;void log_safe (int prio,const char * fmt,va_list ap) {if (log_thr == (pthread_t )0 ) { -syslog (prio, fmt, ap); +vsyslog (prio, fmt, ap);return ; }
multipath/main.c +18 -3
@@ -28,6 +28,7 @@#include <unistd.h> #include <ctype.h> #include <libudev.h> +#include <syslog.h> #include <checkers.h> #include <prio.h>
@@ -283,6 +284,7 @@ configure (void)int failed =get_refwwid (conf->dev, conf->dev_type, pathvec, &refwwid);if (!refwwid) { +condlog (3 ,"%s: failed to get wwid" , conf->dev);if (failed ==2 && conf->cmd ==CMD_VALID_PATH )printf ("%s is not a valid multipath device path\n" , conf->dev);else
@@ -471,11 +473,11 @@ main (int argc, char *argv[])int r =1 ; udev =udev_new (); - + logsink =0 ;if (load_config (DEFAULT_CONFIGFILE , udev))exit (1 ); -while ((arg =getopt (argc, argv,":adchl::FfM:v:p:b:BritqwW" )) !=EOF ) { +while ((arg =getopt (argc, argv,":adchl::FfM:v:p:b:BritquwW" )) !=EOF ) {switch (arg) {case 1 :printf ("optarg : %s\n" ,optarg);break ;
@@ -542,6 +544,10 @@ main (int argc, char *argv[])case 'h' :usage (argv[0 ]);exit (0 ); +case 'u' : + conf->cmd =CMD_VALID_PATH ; + conf->dev_type =DEV_UEVENT ; +break ;case 'w' : conf->cmd =CMD_REMOVE_WWID ;break ;
@@ -581,9 +587,15 @@ main (int argc, char *argv[]) goto out;strncpy (conf->dev, argv[optind],FILE_NAME_SIZE ); - conf->dev_type =get_dev_type (conf->dev); +if (conf->dev_type !=DEV_UEVENT ) + conf->dev_type =get_dev_type (conf->dev); } conf->daemon =0 ; +if (conf->dev_type ==DEV_UEVENT ) { +openlog ("multipath" ,0 ,LOG_DAEMON ); +setlogmask (LOG_UPTO (conf->verbosity +3 )); + logsink =1 ; + }if (conf->max_fds) {struct rlimit fd_limit;
@@ -659,6 +671,9 @@ out:cleanup_prio ();cleanup_checkers (); +if (conf->dev_type ==DEV_UEVENT ) +closelog (); + out_free_config:/* * Freeing config must be done after dm_lib_exit(), because
multipath/multipath.8 +5 -1
@@ -8,7 +8,7 @@ multipath \- Device mapper target autoconfig .RB [\| \-b\ \c .IR bindings_file \|] .RB [\| \-d \|] -.RB [\| \-h | \-l | \-ll | \-f | \-t | \-F | \-B | \-c | \-q | \|-r | \|-i | \-a | \-w | \-W \|] +.RB [\| \-h | \-l | \-ll | \-f | \-t | \-F | \-B | \-c | \-q | \|-r | \|-i | \-a | \|-u | \-w | \-W \|] .RB [\| \-p\ \c .BR failover | multibus | group_by_serial | group_by_prio | group_by_node_name \|] .RB [\| device \|]
@@ -74,6 +74,10 @@ allow device tables with queue_if_no_path when multipathd is not running .B \-a add the wwid for the specified device to the wwids file .TP +.B \-u +check if the device specified in the program environment should be +a path in a multipath device. +.TP .B \-w remove the wwid for the specified device from the wwids file .TP
multipath/multipath.rules +1 -1
@@ -6,7 +6,7 @@ TEST!="$env{MPATH_SBIN_PATH}/multipath", ENV{MPATH_SBIN_PATH}="/usr/sbin" SUBSYSTEM=="block", ACTION=="add|change", KERNEL!="dm-*", \ ENV{DM_MULTIPATH_DEVICE_PATH}!="1", \ - PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -v 0 -c $tempnode", \ + PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -u %k", \ ENV{DM_MULTIPATH_DEVICE_PATH}="1" \ ENV{SYSTEMD_READY}="0"
Signature · 1 line
1.8.4.5
PATCH 23/78 Install multipath rule under '56-multipath.rules'
In reply to Hannes ReineckeThe multipath rule needs to be inserted between the sg3_utils rules for it to be able to intercept symlink generation.
References: bnc#873151
--- multipath/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
multipath/Makefile +2 -2
@@ -23,7 +23,7 @@ install: $(INSTALL_PROGRAM) -m 755 $(EXEC) $(DESTDIR)$(bindir)/ $(INSTALL_PROGRAM) -d $(DESTDIR)$(udevrulesdir) $(INSTALL_PROGRAM) -m 644 11-dm-mpath.rules $(DESTDIR)$(udevrulesdir) - $(INSTALL_PROGRAM) -m 644 $(EXEC).rules $(DESTDIR)$(libudevdir)/rules.d/40-multipath.rules + $(INSTALL_PROGRAM) -m 644 $(EXEC).rules $(DESTDIR)$(libudevdir)/rules.d/56-multipath.rules $(INSTALL_PROGRAM) -d $(DESTDIR)$(mandir) $(INSTALL_PROGRAM) -m 644 $(EXEC).8.gz $(DESTDIR)$(mandir) $(INSTALL_PROGRAM) -d $(DESTDIR)$(man5dir)
@@ -32,7 +32,7 @@ install: uninstall: rm $(DESTDIR)$(bindir)/$(EXEC) rm $(DESTDIR)$(udevrulesdir)/11-dm-mpath.rules - rm $(DESTDIR)$(libudevdir)/rules.d/40-multipath.rules + rm $(DESTDIR)$(libudevdir)/rules.d/56-multipath.rules rm $(DESTDIR)$(mandir)/$(EXEC).8.gz rm $(DESTDIR)$(man5dir)/$(EXEC).conf.5.gz
Signature · 1 line
1.8.4.5
PATCH 24/78 multipath.rules: Whitelist devices
In reply to Hannes ReineckeCurrently multipath runs only on SCSI and DASD devices (and cciss, but they are no longer supported). So we can as well whitelist them and avoid the curious warning about 'multipath -u /dev/loop failed'.
--- multipath/multipath.rules | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
multipath/multipath.rules +1 -1
@@ -4,7 +4,7 @@ SUBSYSTEM!="block", GOTO="end_mpath" ENV{MPATH_SBIN_PATH}="/sbin" TEST!="$env{MPATH_SBIN_PATH}/multipath", ENV{MPATH_SBIN_PATH}="/usr/sbin" -SUBSYSTEM=="block", ACTION=="add|change", KERNEL!="dm-*", \ +SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="sd*[!0-9]|dasd*[!0-9]", \ ENV{DM_MULTIPATH_DEVICE_PATH}!="1", \ PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -u %k", \ ENV{DM_MULTIPATH_DEVICE_PATH}="1" \
Signature · 1 line
1.8.4.5
PATCH 25/78 multipath.rules: fixup race condition with systemd
In reply to Hannes Reineckesystemd will call 'fsck' on any device, even if no device type is found. This might cause a race between the mount call and any fsck call on the device-mapper device.
So for multipath we should arbitrary set the device type to 'none', as then systemd with detect a missing 'fsck.none' command and skip fsck here.
--- multipath/multipath.rules | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
multipath/multipath.rules +2 -1
@@ -4,10 +4,11 @@ SUBSYSTEM!="block", GOTO="end_mpath" ENV{MPATH_SBIN_PATH}="/sbin" TEST!="$env{MPATH_SBIN_PATH}/multipath", ENV{MPATH_SBIN_PATH}="/usr/sbin" -SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="sd*[!0-9]|dasd*[!0-9]", \ +SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="sd*|dasd*", \ ENV{DM_MULTIPATH_DEVICE_PATH}!="1", \ PROGRAM=="$env{MPATH_SBIN_PATH}/multipath -u %k", \ ENV{DM_MULTIPATH_DEVICE_PATH}="1" \ + ENV{ID_FS_TYPE}="none" \ ENV{SYSTEMD_READY}="0" LABEL="end_mpath"
Signature · 1 line
1.8.4.5
PATCH 26/78 11-dm-mpath.rules: Import blkid values if all paths are down
In reply to Hannes ReineckeWhen all paths are failed we should not try to run any programs requiring disk access. However, we still need to create the symlinks so as not to confuse systemd.
So import the blkid values from the database in these cases.
--- multipath/11-dm-mpath.rules | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
multipath/11-dm-mpath.rules +7 -1
@@ -6,7 +6,13 @@ ENV{DM_UUID}!="mpath-?*", GOTO="mpath_end" # otherwise there would be a hang or IO error on access. # We'd like to avoid this, especially within udev processing. ENV{DM_NR_VALID_PATHS}!="?*", IMPORT{db}="DM_NR_VALID_PATHS" -ENV{DM_NR_VALID_PATHS}=="0", ENV{DM_NOSCAN}="1" +ENV{DM_NR_VALID_PATHS}!="0", GOTO="mpath_blkid_end" +IMPORT{db}="ID_FS_TYPE" +IMPORT{db}="ID_FS_USAGE" +IMPORT{db}="ID_FS_UUID" +IMPORT{db}="ID_FS_UUID_ENC" +IMPORT{db}="ID_FS_VERSION" +LABEL="mpath_blkid_end" # Also skip all foreign rules if no path is available. # Remember the original value of DM_DISABLE_OTHER_RULES_FLAG
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 26/78] 11-dm-mpath.rules: Import blkid values if all paths are down
Quoted 4 lines · Hannes Reinecke
When all paths are failed we should not try to run any programs requiring disk access. However, we still need to create the symlinks so as not to confuse systemd.
So import the blkid values from the database in these cases.
Don't we still need to set DM_NOSCAN here?
-Ben
Quoted 27 lines
Signed-off-by Hannes Reinecke <[email protected]>--- multipath/11-dm-mpath.rules | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)multipath/11-dm-mpath.rules +7
-1@@ -6,7 +6,13 @@ ENV{DM_UUID}!="mpath-?*", GOTO="mpath_end" # otherwise there would be a hang or IO error on access. # We'd like to avoid this, especially within udev processing. ENV{DM_NR_VALID_PATHS}!="?*", IMPORT{db}="DM_NR_VALID_PATHS" -ENV{DM_NR_VALID_PATHS}=="0", ENV{DM_NOSCAN}="1" +ENV{DM_NR_VALID_PATHS}!="0", GOTO="mpath_blkid_end" +IMPORT{db}="ID_FS_TYPE" +IMPORT{db}="ID_FS_USAGE" +IMPORT{db}="ID_FS_UUID" +IMPORT{db}="ID_FS_UUID_ENC" +IMPORT{db}="ID_FS_VERSION" +LABEL="mpath_blkid_end" # Also skip all foreign rules if no path is available. # Remember the original value of DM_DISABLE_OTHER_RULES_FLAGSignature · 1 line
1.8.4.5
Re: [PATCH 26/78] 11-dm-mpath.rules: Import blkid values if all paths are down
Quoted 8 lines · Benjamin Marzinski
Quoted 4 lines · Hannes Reinecke
When all paths are failed we should not try to run any programs requiring disk access. However, we still need to create the symlinks so as not to confuse systemd.
So import the blkid values from the database in these cases.Don't we still need to set
DM_NOSCANhere?
I have no idea at all. I just found the setting somewhere and thought it'd be a good idea.
But don't ask me what's for; device-mapper/LVM udev variables are an arcane subject which I've yet to grasp.
Incidentally, a similar fix is needed for 10-dm.rules. But I'll prepare a separate patch for this.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
Benjamin Marzinski
maintainerRe: [PATCH 26/78] 11-dm-mpath.rules: Import blkid values if all paths are down
Quoted 7 lines · Hannes Reinecke
Quoted 6 lines · Benjamin Marzinski
Quoted 4 lines · Hannes Reinecke
When all paths are failed we should not try to run any programs requiring disk access. However, we still need to create the symlinks so as not to confuse systemd.
So import the blkid values from the database in these cases.
Quoted 6 lines
Quoted 2 lines
Don't we still need to set
DM_NOSCANhere?I have no idea at all. I just found the setting somewhere and thought it'd be a good idea.
But don't ask me what's for; device-mapper/LVM udev variables are an arcane subject which I've yet to grasp.
But what I mean is that your patch removes DM_NOSCAN
-ENV{DM_NR_VALID_PATHS}=="0", ENV{DM_NOSCAN}="1"
I assume that this is an oversight, since as you mentioned, we don't want to be running blkid on a device with no valid paths.
-Ben
Quoted 13 lines
Incidentally, a similar fix is needed for 10-dm.rules. But I'll prepare a separate patch for this.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
PATCH 27/78 kpartx.rules: Skip kpartx for multipath events
In reply to Hannes ReineckeWhenever multipath signals a PATH_FAILED or PATH_REINSTATED event we don't have to call 'kpartx'; the contents on the disk haven't changed.
And we should never call kpartx if all paths are down, irrespective of the uevent.
--- kpartx/kpartx.rules | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kpartx/kpartx.rules +2 -2
@@ -40,8 +40,8 @@ ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_ENC}=="?*", \ SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}" # Create dm tables for partitions -ENV{DM_ACTION}=="PATH_FAILED", ENV{DM_NR_VALID_PATHS}=="0", \ - GOTO="kpartx_end" +ENV{DM_ACTION}=="PATH_FAILED|PATH_REINSTATED", GOTO="kpartx_end" +ENV{DM_NR_VALID_PATHS}=="0", GOTO="kpartx_end" ENV{DM_STATE}!="SUSPENDED", ENV{DM_UUID}=="mpath-*", \ RUN+="/sbin/kpartx -u -p -part /dev/$name"
Signature · 1 line
1.8.4.5
PATCH 28/78 multipathd: handle DOMAP_RETRY
In reply to Hannes ReineckeCalling domap() will first check if all paths are available by using 'flock' on each path. This might fail if other processes are holding the lock already. In these cases we should retry domap() as the processes should terminate eventually.
--- multipathd/main.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
multipathd/main.c +10 -1
@@ -467,6 +467,7 @@ ev_add_path (struct path * pp, struct vectors * vecs)char params[PARAMS_SIZE ] = {0 };int retries =3 ;int start_waiter =0 ; +int ret;/* * need path UID to go any further
@@ -546,7 +547,15 @@ rescan:/* * reload the map for the multipath mapped device */ -if (domap (mpp, params) <=0 ) { +retry: + ret =domap (mpp, params); +if (ret <=0 ) { +if (ret <0 && retries-- >0 ) { +condlog (0 ,"%s: retry domap for addition of new " +"path %s" , mpp->alias, pp->dev); +sleep (1 ); + goto retry; + }condlog (0 ,"%s: failed in domap for addition of new " "path %s" , mpp->alias, pp->dev);/*
Signature · 1 line
1.8.4.5
PATCH 29/78 multipathd: cleanup foreground operation
In reply to Hannes ReineckeWhen specifying '-d' it actually does two things, namely do not daemonize the program and prints logging messages to stdout instead of syslog.
This partially clashes with '-s' which suppresses the timestamp printing. So separate both use cases and ensure that '-d' doesn't overwrite '-s' if specified later.
--- multipathd/main.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
multipathd/main.c +5 -2
@@ -1998,6 +1998,7 @@ main (int argc, char *argv[])extern int optind;int arg;int err; +int foreground =0 ; logsink =1 ; running_state =DAEMON_INIT ;
@@ -2022,7 +2023,9 @@ main (int argc, char *argv[])while ((arg =getopt (argc, argv,":dsv:k::B" )) !=EOF ) {switch (arg) {case 'd' : - logsink =0 ; + foreground =1 ; +if (logsink >0 ) + logsink =0 ;//debug=1; /* ### comment me out ### */ break ;case 'v' :
@@ -2062,7 +2065,7 @@ main (int argc, char *argv[])exit (0 ); } -if (logsink <1 ) { +if (foreground) {if (!isatty (fileno (stdout)))setbuf (stdout,NULL ); err =0 ;
Signature · 1 line
1.8.4.5
PATCH 30/78 Update hwtable for EMC XtremIO
In reply to Hannes Reinecke--- libmultipath/hwtable.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
libmultipath/hwtable.c +13 -0
@@ -289,6 +289,19 @@ static struct hwentry default_hw[] = { .prio_name = DEFAULT_PRIO, .prio_args = NULL, }, + { + .vendor = "XtremIO", + .product = "XtremApp", + .features = DEFAULT_FEATURES, + .hwhandler = DEFAULT_HWHANDLER, + .pgpolicy = MULTIBUS, + .pgfailback = FAILBACK_UNDEF, + .rr_weight = RR_WEIGHT_NONE, + .no_path_retry = 5, + .checker_name = TUR, + .prio_name = DEFAULT_PRIO, + .prio_args = NULL, + },/* * Fujitsu controller family *
Signature · 1 line
1.8.4.5
PATCH 31/78 multipath: check for running daemon when called with '-u'
In reply to Hannes ReineckeWhen multipath is called to check for a valid device it really should check if the daemon is running. Otherwise it'll blindly assume the device to be eligible for multipathing even though the daemon isn't running.
--- multipath/main.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
multipath/main.c +13 -0
@@ -56,6 +56,7 @@#include <sys/time.h> #include <sys/resource.h> #include <wwids.h> +#include <uxsock.h> #include "dev_t.h" int logsink;
@@ -622,6 +623,18 @@ main (int argc, char *argv[])condlog (0 ,"the -c option requires a path to check" ); goto out; } +if (conf->cmd ==CMD_VALID_PATH && + conf->dev_type ==DEV_UEVENT ) { +int fd; + + fd =ux_socket_connect (DEFAULT_SOCKET ); +if (fd ==-1 ) { +printf ("%s is not a valid multipath device path\n" , + conf->dev); + goto out; + } +close (fd); + }if (conf->cmd ==CMD_REMOVE_WWID && !conf->dev) {condlog (0 ,"the -w option requires a device" ); goto out;
Signature · 1 line
1.8.4.5
PATCH 32/78 Revert 'return PATH_DOWN for quiesced paths'
In reply to Hannes ReineckeHas been superseded by commit 00a2549a42707e5e1ef51ef6f638dc5ceaa15740
--- libmultipath/discovery.c | 1 - 1 file changed, 1 deletion(-)
libmultipath/discovery.c +0 -1
@@ -958,7 +958,6 @@ path_offline (struct path * pp)condlog (3 ,"%s: path state = %s" , pp->dev, buff);if (!strncmp (buff,"offline" ,7 ) || - !strncmp (buff,"quiesce" ,7 ) || !strncmp (buff,"transport-offline" ,17 )) { pp->offline =1 ;return PATH_DOWN ;
Signature · 1 line
1.8.4.5
PATCH 33/78 Do not treat 'transport-offline' paths as 'offline'
In reply to Hannes Reinecke'transport-offline' status is entered whenever the transport has detected an offline device, but the SCSI midlayer still has references to it.
Once the last reference drops the SCSI midlayer will properly deregister the device and multipath will be updating the status. But until then we should treat this device as simply 'path down', and not marking it offline.
After all, some transports like FC will be entering the 'transport-offline' state after fast_io_fail triggered, and might revert back to 'running' after reconfiguration.
References: bnc#888378
--- libmultipath/discovery.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
libmultipath/discovery.c +1 -2
@@ -957,8 +957,7 @@ path_offline (struct path * pp)condlog (3 ,"%s: path state = %s" , pp->dev, buff); -if (!strncmp (buff,"offline" ,7 ) || - !strncmp (buff,"transport-offline" ,17 )) { +if (!strncmp (buff,"offline" ,7 )) { pp->offline =1 ;return PATH_DOWN ; }
Signature · 1 line
1.8.4.5
PATCH 34/78 Check for valid DM_DEVICE_INFO before proceeding
In reply to Hannes ReineckeCalling 'DM_DEVICE_INFO' might succeed but the returned context might refer to an invalid device. So one needs to check the 'exists' field to avoid this.
--- libmultipath/devmapper.c | 9 +++++++++ 1 file changed, 9 insertions(+)
libmultipath/devmapper.c +9 -0
@@ -615,6 +615,9 @@ dm_get_opencount (const char * mapname)if (!dm_task_get_info (dmt, &info)) goto out; +if (!info.exists) + goto out; + r = info.open_count; out:dm_task_destroy (dmt);
@@ -640,6 +643,9 @@ dm_get_major (char * mapname)if (!dm_task_get_info (dmt, &info)) goto out; +if (!info.exists) + goto out; + r = info.major; out:dm_task_destroy (dmt);
@@ -665,6 +671,9 @@ dm_get_minor (char * mapname)if (!dm_task_get_info (dmt, &info)) goto out; +if (!info.exists) + goto out; + r = info.minor; out:dm_task_destroy (dmt);
Signature · 1 line
1.8.4.5
PATCH 35/78 Separate out uevent parsing functions
In reply to Hannes ReineckeMove the uevent parsing into separate functions.
No functional change.
--- libmultipath/uevent.c | 254 +++++++++++++++++++++++++++----------------------- 1 file changed, 136 insertions(+), 118 deletions(-)
libmultipath/uevent.c +136 -118
@@ -182,6 +182,80 @@ int uevent_dispatch(int (*uev_trigger)(struct uevent *, void * trigger_data),return 0 ; } +struct uevent *uevent_from_buffer (char *buf,ssize_t buflen) +{ +struct uevent *uev; +char *buffer; +size_t bufpos; +int i; +char *pos; + + uev =alloc_uevent (); +if (!uev) { +condlog (1 ,"lost uevent, oom" ); +return NULL ; + } + +if ((size_t )buflen >sizeof (buf)-1 ) + buflen =sizeof (buf)-1 ; + +/* +* Copy the shared receive buffer contents to buffer private +* to this uevent so we can immediately reuse the shared buffer. +*/ +memcpy (uev->buffer, buf,HOTPLUG_BUFFER_SIZE +OBJECT_SIZE ); + buffer = uev->buffer; + buffer[buflen] ='\0' ; + +/* save start of payload */ + bufpos =strlen (buffer) +1 ; + +/* action string */ + uev->action = buffer; + pos =strchr (buffer,'@' ); +if (!pos) { +condlog (3 ,"bad action string '%s'" , buffer); +FREE (uev); +return NULL ; + } + pos[0 ] ='\0' ; + +/* sysfs path */ + uev->devpath = &pos[1 ]; + +/* hotplug events have the environment attached - reconstruct envp[] */ +for (i =0 ; (bufpos < (size_t )buflen) && (i <HOTPLUG_NUM_ENVP -1 ); i++) { +int keylen; +char *key; + + key = &buffer[bufpos]; + keylen =strlen (key); + uev->envp[i] = key; +/* Filter out sequence number */ +if (strncmp (key,"SEQNUM=" ,7 ) ==0 ) { +char *eptr; + + uev->seqnum =strtoul (key +7 , &eptr,10 ); +if (eptr == key +7 ) + uev->seqnum =-1 ; + } + bufpos += keylen +1 ; + } + uev->envp[i] =NULL ; + +condlog (3 ,"uevent %ld '%s' from '%s'" , uev->seqnum, + uev->action, uev->devpath); + uev->kernel =strrchr (uev->devpath,'/' ); +if (uev->kernel) + uev->kernel++; + +/* print payload environment */ +for (i =0 ; uev->envp[i] !=NULL ; i++) +condlog (5 ,"%s" , uev->envp[i]); + +return uev; +} +int failback_listen (void ) {int sock;
@@ -266,12 +340,9 @@ int failback_listen(void) }while (1 ) { -int i; -char *pos;size_t bufpos;ssize_t buflen;struct uevent *uev; -char *buffer;struct msghdr smsg;struct iovec iov;char cred_msg[CMSG_SPACE (sizeof (struct ucred ))];
@@ -324,69 +395,9 @@ int failback_listen(void) buflen =sizeof (buf)-1 ; } - uev =alloc_uevent (); - -if (!uev) { -condlog (1 ,"lost uevent, oom" ); -continue ; - } - -if ((size_t )buflen >sizeof (buf)-1 ) - buflen =sizeof (buf)-1 ; - -/* -* Copy the shared receive buffer contents to buffer private -* to this uevent so we can immediately reuse the shared buffer. -*/ -memcpy (uev->buffer, buf,HOTPLUG_BUFFER_SIZE +OBJECT_SIZE ); - buffer = uev->buffer; - buffer[buflen] ='\0' ; - -/* save start of payload */ - bufpos =strlen (buffer) +1 ; - -/* action string */ - uev->action = buffer; - pos =strchr (buffer,'@' ); -if (!pos) { -condlog (3 ,"bad action string '%s'" , buffer); + uev =uevent_from_buffer (buf, buflen); +if (!uev)continue ; - } - pos[0 ] ='\0' ; - -/* sysfs path */ - uev->devpath = &pos[1 ]; - -/* hotplug events have the environment attached - reconstruct envp[] */ -for (i =0 ; (bufpos < (size_t )buflen) && (i <HOTPLUG_NUM_ENVP -1 ); i++) { -int keylen; -char *key; - - key = &buffer[bufpos]; - keylen =strlen (key); - uev->envp[i] = key; -/* Filter out sequence number */ -if (strncmp (key,"SEQNUM=" ,7 ) ==0 ) { -char *eptr; - - uev->seqnum =strtoul (key +7 , &eptr,10 ); -if (eptr == key +7 ) - uev->seqnum =-1 ; - } - bufpos += keylen +1 ; - } - uev->envp[i] =NULL ; - -condlog (3 ,"uevent %ld '%s' from '%s'" , uev->seqnum, - uev->action, uev->devpath); - uev->kernel =strrchr (uev->devpath,'/' ); -if (uev->kernel) - uev->kernel++; - -/* print payload environment */ -for (i =0 ; uev->envp[i] !=NULL ; i++) -condlog (5 ,"%s" , uev->envp[i]); -/* * Queue uevent and poke service pthread. */
@@ -401,6 +412,62 @@ exit:return 1 ; } +struct uevent *uevent_from_udev_device (struct udev_device *dev) +{ +struct uevent *uev; +int i =0 ; +char *pos, *end; +struct udev_list_entry *list_entry; + + uev =alloc_uevent (); +if (!uev) { +udev_device_unref (dev); +condlog (1 ,"lost uevent, oom" ); +return NULL ; + } + pos = uev->buffer; + end = pos +HOTPLUG_BUFFER_SIZE +OBJECT_SIZE -1 ; + udev_list_entry_foreach(list_entry, udev_device_get_properties_list_entry(dev)) { +const char *name, *value; +int bytes; + + name =udev_list_entry_get_name (list_entry); +if (!name) + name ="(null)" ; + value =udev_list_entry_get_value (list_entry); +if (!value) + value ="(null)" ; + bytes =snprintf (pos, end - pos,"%s=%s" , name, value); +if (pos + bytes >= end) { +condlog (2 ,"buffer overflow for uevent" ); +break ; + } + uev->envp[i] = pos; + pos += bytes; + *pos ='\0' ; + pos++; +if (strcmp (name,"DEVPATH" ) ==0 ) + uev->devpath = uev->envp[i] +8 ; +if (strcmp (name,"ACTION" ) ==0 ) + uev->action = uev->envp[i] +7 ; + i++; +if (i ==HOTPLUG_NUM_ENVP -1 ) +break ; + } + uev->udev = dev; + uev->envp[i] =NULL ; + +condlog (3 ,"uevent '%s' from '%s'" , uev->action, uev->devpath); + uev->kernel =strrchr (uev->devpath,'/' ); +if (uev->kernel) + uev->kernel++; + +/* print payload environment */ +for (i =0 ; uev->envp[i] !=NULL ; i++) +condlog (5 ,"%s" , uev->envp[i]); +return uev; +} +int uevent_listen (struct udev *udev) {int err;
@@ -456,69 +523,20 @@ int uevent_listen(struct udev *udev) goto out; }while (1 ) { -int i =0 ; -char *pos, *end;struct uevent *uev;struct udev_device *dev; -struct udev_list_entry *list_entry; dev =udev_monitor_receive_device (monitor);if (!dev) {condlog (0 ,"failed getting udev device" );continue ; } - - uev =alloc_uevent (); -if (!uev) { -udev_device_unref (dev); -condlog (1 ,"lost uevent, oom" ); + uev =uevent_from_udev_device (dev); +if (!uev)continue ; - } - pos = uev->buffer; - end = pos +HOTPLUG_BUFFER_SIZE +OBJECT_SIZE -1 ; - udev_list_entry_foreach(list_entry, udev_device_get_properties_list_entry(dev)) { -const char *name, *value; -int bytes; - - name =udev_list_entry_get_name (list_entry); -if (!name) - name ="(null)" ; - value =udev_list_entry_get_value (list_entry); -if (!value) - value ="(null)" ; - bytes =snprintf (pos, end - pos,"%s=%s" , name, - value); -if (pos + bytes >= end) { -condlog (2 ,"buffer overflow for uevent" ); -break ; - } - uev->envp[i] = pos; - pos += bytes; - *pos ='\0' ; - pos++; -if (strcmp (name,"DEVPATH" ) ==0 ) - uev->devpath = uev->envp[i] +8 ; -if (strcmp (name,"ACTION" ) ==0 ) - uev->action = uev->envp[i] +7 ; - i++; -if (i ==HOTPLUG_NUM_ENVP -1 ) -break ; - } - uev->udev = dev; - uev->envp[i] =NULL ; - -condlog (3 ,"uevent '%s' from '%s'" , uev->action, uev->devpath); - uev->kernel =strrchr (uev->devpath,'/' ); -if (uev->kernel) - uev->kernel++; - -/* print payload environment */ -for (i =0 ; uev->envp[i] !=NULL ; i++) -condlog (5 ,"%s" , uev->envp[i]); -/* -* Queue uevent and poke service pthread. -*/ +* Queue uevent and poke service pthread. +*/ pthread_mutex_lock (uevq_lockp);list_add_tail (&uev->node, &uevq);pthread_cond_signal (uev_condp);
Signature · 1 line
1.8.4.5
PATCH 36/78 Use poll() when receiving uevents
In reply to Hannes ReineckeCurrently uevent_listen() will be adding each event individually to the internal queue. This leads to a lock contention on high load, as uevent_listen() has to grab the shared lock before doing so.
This patch batches the uevent reception so that uevents will only ever be added to the internal queue if there are no more events pending.
--- libmultipath/list.h | 47 +++++++++++++++++++++++++------- libmultipath/uevent.c | 74 +++++++++++++++++++++++++++++++++++++++------------ 2 files changed, 95 insertions(+), 26 deletions(-)
libmultipath/list.h +38 -9
@@ -161,18 +161,18 @@ static inline int list_empty(struct list_head *head)return head->next == head; } -static inline void __list_splice (struct list_head *list, -struct list_head *head) +static inline void __list_splice (const struct list_head *list, +struct list_head *prev, +struct list_head *next) {struct list_head *first = list->next;struct list_head *last = list->prev; -struct list_head *at = head->next; - first->prev = head; - head->next = first; + first->prev = prev; + prev->next = first; - last->next = at; - at->prev = last; + last->next = next; + next->prev = last; }/**
@@ -183,7 +183,19 @@ static inline void __list_splice(struct list_head *list,static inline void list_splice (struct list_head *list,struct list_head *head) {if (!list_empty (list)) -__list_splice (list, head); +__list_splice (list, head, head->next); +} + +/** +* list_splice_tail - join two lists, each list being a queue +* @list: the new list to add. +* @head: the place to add it in the first list. +*/ +static inline void list_splice_tail (struct list_head *list, +struct list_head *head) +{ +if (!list_empty (list)) +__list_splice (list, head->prev, head); }/**
@@ -197,7 +209,24 @@ static inline void list_splice_init(struct list_head *list, struct list_head *head) {if (!list_empty (list)) { -__list_splice (list, head); +__list_splice (list, head, head->next); +INIT_LIST_HEAD (list); + } +} + +/** +* list_splice_tail_init - join two lists and reinitialise the emptied list +* @list: the new list to add. +* @head: the place to add it in the first list. +* +* Each of the lists is a queue. +* The list at @list is reinitialised +*/ +static inline void list_splice_tail_init (struct list_head *list, +struct list_head *head) +{ +if (!list_empty (list)) { +__list_splice (list, head->prev, head);INIT_LIST_HEAD (list); } }
libmultipath/uevent.c +57 -17
111 lines not on this page · raw message
Signature · 1 line
1.8.4.5
PATCH 37/78 mpath_persist: cleanup
In reply to Hannes ReineckeCleanup whitespace issues and use the standard 'major' and 'minor' macros.
--- libmpathpersist/mpath_persist.c | 106 +++++++++++++++++++++------------------- libmpathpersist/mpathpr.h | 3 +- 2 files changed, 58 insertions(+), 51 deletions(-)
libmpathpersist/mpath_persist.c +57 -49
@@ -1,7 +1,7 @@#include <libdevmapper.h> #include <defaults.h> #include <sys/stat.h> -#include <linux/kdev_t.h> +#include <sys/types.h> #include <fcntl.h> #include <vector.h> #include <checkers.h>
@@ -19,6 +19,7 @@#include <dmparser.h> #include <ctype.h> #include <propsel.h> +#include <util.h> #include "mpath_persist.h" #include "mpathpr.h"
@@ -71,7 +72,8 @@ updatepaths (struct multipath * mpp) vector_foreach_slot (pgp->paths, pp, j){if (!strlen (pp->dev)){ -if (devt2devname (pp->dev, pp->dev_t)){ +if (devt2devname (pp->dev,PATH_SIZE , + pp->dev_t)){/* * path is not in sysfs anymore */
@@ -94,7 +96,7 @@ updatepaths (struct multipath * mpp)return 0 ; } -int +int mpath_prin_activepath (struct multipath *mpp,int rq_servact,struct prin_resp * resp,int noisy) {
@@ -104,14 +106,19 @@ mpath_prin_activepath (struct multipath *mpp, int rq_servact, vector_foreach_slot (mpp->pg, pgp, j){ vector_foreach_slot (pgp->paths, pp, i){ -if (!((pp->state ==PATH_UP ) || (pp->state ==PATH_GHOST ))){ -condlog (2 ,"%s: %s not available. Skip." , mpp->wwid, pp->dev); -condlog (3 ,"%s: status = %d." , mpp->wwid, pp->state); +if (!((pp->state ==PATH_UP ) || + (pp->state ==PATH_GHOST ))){ +condlog (2 ,"%s: %s not available. Skip." , + mpp->wwid, pp->dev); +condlog (3 ,"%s: status = %d." , + mpp->wwid, pp->state);continue ; } -condlog (3 ,"%s: sending pr in command to %s " , mpp->wwid, pp->dev); - ret =mpath_send_prin_activepath (pp->dev, rq_servact, resp, noisy); +condlog (3 ,"%s: sending pr in command to %s " , + mpp->wwid, pp->dev); + ret =mpath_send_prin_activepath (pp->dev, rq_servact, + resp, noisy);switch (ret) {case MPATH_PR_SUCCESS :
@@ -122,10 +129,11 @@ mpath_prin_activepath (struct multipath *mpp, int rq_servact, } } } -return ret; +return ret; } -int mpath_persistent_reserve_in (int fd,int rq_servact,struct prin_resp *resp,int noisy,int verbose) +int mpath_persistent_reserve_in (int fd,int rq_servact, +struct prin_resp *resp,int noisy,int verbose) {struct stat info;vector curmp =NULL ;
@@ -141,14 +149,14 @@ int mpath_persistent_reserve_in (int fd, int rq_servact, struct prin_resp *resp,if (fstat ( fd, &info) !=0 ){condlog (0 ,"stat error %d" , fd);return MPATH_PR_FILE_ERROR ; - } + }if (!S_ISBLK (info.st_mode)){condlog (0 ,"Failed to get major:minor. fd = %d" , fd);return MPATH_PR_FILE_ERROR ; } - major = (int )MAJOR (info.st_rdev); - minor = (int )MINOR (info.st_rdev); + major =major (info.st_rdev); + minor =minor (info.st_rdev);condlog (4 ,"Device %d:%d: " , major, minor);/* get alias from major:minor*/
@@ -201,14 +209,14 @@ int mpath_persistent_reserve_in (int fd, int rq_servact, struct prin_resp *resp, out1:free_multipathvec (curmp,KEEP_PATHS ); -free_pathvec (pathvec,FREE_PATHS ); +free_pathvec (pathvec,FREE_PATHS ); out:FREE (alias); -return ret; +return ret; }int mpath_persistent_reserve_out (int fd,int rq_servact,int rq_scope, -unsigned int rq_type,struct prout_param_descriptor *paramp,int noisy,int verbose) +unsigned int rq_type,struct prout_param_descriptor *paramp,int noisy,int verbose) {struct stat info;
@@ -223,7 +231,7 @@ int mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope,int ret;int j;unsigned char *keyp; -uint64_t prkey; +uint64_t prkey; conf->verbosity = verbose;
@@ -234,11 +242,11 @@ int mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope,if (!S_ISBLK (info.st_mode)){condlog (3 ,"Failed to get major:minor. fd=%d" , fd); -return MPATH_PR_FILE_ERROR ; - } +return MPATH_PR_FILE_ERROR ; + } - major = (int )MAJOR (info.st_rdev); - minor = (int )MINOR (info.st_rdev); + major =major (info.st_rdev); + minor =minor (info.st_rdev);condlog (4 ,"Device %d:%d" , major, minor);/* get WWN of the device from major:minor*/
@@ -292,22 +300,22 @@ int mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope,switch (rq_servact) { -case MPATH_PROUT_REG_SA : -case MPATH_PROUT_REG_IGN_SA : - ret=mpath_prout_reg (mpp, rq_servact, rq_scope, rq_type, paramp, noisy); -break ; -case MPATH_PROUT_RES_SA : -case MPATH_PROUT_PREE_SA : -case MPATH_PROUT_PREE_AB_SA : -case MPATH_PROUT_CLEAR_SA : - ret =mpath_prout_common (mpp, rq_servact, rq_scope, rq_type, paramp, noisy); -break ; -case MPATH_PROUT_REL_SA : - ret =mpath_prout_rel (mpp, rq_servact, rq_scope, rq_type, paramp, noisy); -break ; -default : - ret =MPATH_PR_OTHER ; - goto out1; +case MPATH_PROUT_REG_SA : +case MPATH_PROUT_REG_IGN_SA : + ret=mpath_prout_reg (mpp, rq_servact, rq_scope, rq_type, paramp, noisy); +break ; +case MPATH_PROUT_RES_SA : +case MPATH_PROUT_PREE_SA : +case MPATH_PROUT_PREE_AB_SA : +case MPATH_PROUT_CLEAR_SA : + ret =mpath_prout_common (mpp, rq_servact, rq_scope, rq_type, paramp, noisy); +break ; +case MPATH_PROUT_REL_SA : + ret =mpath_prout_rel (mpp, rq_servact, rq_scope, rq_type, paramp, noisy); +break ; +default : + ret =MPATH_PR_OTHER ; + goto out1; } if ((ret == MPATH_PR_SUCCESS) && ((rq_servact == MPATH_PROUT_REG_SA) ||
@@ -326,7 +334,7 @@ int mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope,else update_prflag (alias,"set" , noisy); }else { -if ((ret ==MPATH_PR_SUCCESS ) && ((rq_servact ==MPATH_PROUT_CLEAR_SA ) || +if ((ret ==MPATH_PR_SUCCESS ) && ((rq_servact ==MPATH_PROUT_CLEAR_SA ) || (rq_servact ==MPATH_PROUT_PREE_AB_SA ))){update_prflag (alias,"unset" , noisy); }
@@ -337,7 +345,7 @@ out1: out:FREE (alias); -return ret; +return ret; }int
@@ -365,9 +373,9 @@ get_mpvec (vector curmp, vector pathvec, char * refwwid)dm_get_map (mpp->alias, &mpp->size, params);condlog (3 ,"params = %s" , params);dm_get_status (mpp->alias, status); -condlog (3 ,"status = %s" , status); +condlog (3 ,"status = %s" , status);disassemble_map (pathvec, params, mpp); - +/* * disassemble_map() can add new paths to pathvec. * If not in "fast list mode", we need to fetch information
@@ -542,7 +550,7 @@ void * mpath_prout_pthread_fn(void *p) }int mpath_prout_common (struct multipath *mpp,int rq_servact,int rq_scope, -unsigned int rq_type,struct prout_param_descriptor * paramp,int noisy) +unsigned int rq_type,struct prout_param_descriptor * paramp,int noisy) {int i,j, ret;struct pathgroup *pgp =NULL ;
@@ -600,7 +608,7 @@ int send_prout_activepath(char * dev, int rq_servact, int rq_scope, }int mpath_prout_rel (struct multipath *mpp,int rq_servact,int rq_scope, -unsigned int rq_type,struct prout_param_descriptor * paramp,int noisy) +unsigned int rq_type,struct prout_param_descriptor * paramp,int noisy) {int i, j;int num =0 ;
@@ -615,7 +623,7 @@ int mpath_prout_rel(struct multipath *mpp,int rq_servact, int rq_scope,struct prout_param_descriptor *pamp;struct prin_resp *pr_buff;int length; -struct transportid *pptr; +struct transportid *pptr;if (!mpp)return MPATH_PR_DMMP_ERROR ;
@@ -649,7 +657,7 @@ int mpath_prout_rel(struct multipath *mpp,int rq_servact, int rq_scope,condlog (1 ,"%s: %s path not up." , mpp->wwid, pp->dev);continue ; } - +strncpy (thread[count].param.dev, pp->dev,FILE_NAME_SIZE );condlog (3 ,"%s: sending pr out command to %s" , mpp->wwid, pp->dev); rc = pthread_create (&thread[count].id, &attr, mpath_prout_pthread_fn,
@@ -686,13 +694,13 @@ int mpath_prout_rel(struct multipath *mpp,int rq_servact, int rq_scope, num = resp.prin_descriptor.prin_readresv.additional_length /8 ;if (num ==0 ){condlog (2 ,"%s: Path holding reservation is released." , mpp->wwid); -return MPATH_PR_SUCCESS ; +return MPATH_PR_SUCCESS ; }condlog (2 ,"%s: Path holding reservation is not avialable." , mpp->wwid); pr_buff =mpath_alloc_prin_response (MPATH_PRIN_RFSTAT_SA );if (!pr_buff){ -condlog (0 ,"%s: failed to alloc pr in response buffer." , mpp->wwid); +condlog (0 ,"%s: failed to alloc pr in response buffer." , mpp->wwid);return MPATH_PR_OTHER ; }
@@ -778,7 +786,7 @@ int mpath_prout_rel(struct multipath *mpp,int rq_servact, int rq_scope,memset (pamp,0 , length);memcpy (pamp->sa_key, mpp->reservation_key,8 );memset (pamp->key,0 ,8 ); - status =mpath_prout_reg (mpp,MPATH_PROUT_REG_SA , rq_scope, rq_type, pamp, noisy); + status =mpath_prout_reg (mpp,MPATH_PROUT_REG_SA , rq_scope, rq_type, pamp, noisy); }
@@ -812,7 +820,7 @@ void * mpath_alloc_prin_response(int prin_sa)memset (ptr,0 , size);break ;case MPATH_PRIN_RFSTAT_SA : - size =sizeof (struct print_fulldescr_list ) + + size =sizeof (struct print_fulldescr_list ) +sizeof (struct prin_fulldescr *)*MPATH_MX_TIDS ; ptr =malloc (size);memset (ptr,0 , size);
libmpathpersist/mpathpr.h +1 -2
@@ -50,6 +50,5 @@ int send_prout_activepath(char * dev, int rq_servact, int rq_scope,int update_prflag (char * arg1,char * arg2,int noisy);void *mpath_alloc_prin_response (int prin_sa);int update_map_pr (struct multipath *mpp); -int devt2devname (char *devname,char *devt); -#endif +#endif
Signature · 1 line
1.8.4.5
PATCH 38/78 kpartx: use standard 'major' and 'minor' macros
In reply to Hannes ReineckeInstead of hand-coding our own versions we should be using the standard 'major' and 'minor' macros.
--- kpartx/devmapper.h | 7 ++++--- kpartx/kpartx.c | 13 +++++-------- 2 files changed, 9 insertions(+), 11 deletions(-)
kpartx/devmapper.h +4 -3
@@ -1,6 +1,5 @@ -#define MAJOR (dev) ((dev & 0xfff00) >> 8) -#define MINOR (dev) ((dev & 0xff) | ((dev >> 12) & 0xfff00)) -#define MKDEV (ma,mi) ((mi & 0xff) | (ma << 8) | ((mi & ~0xff) << 12)) +#ifndef _KPARTX_DEVMAPPER_H +#define _KPARTX_DEVMAPPER_H#ifdef DM_SUBSYSTEM_UDEV_FLAG0 #define MPATH_UDEV_RELOAD_FLAG DM_SUBSYSTEM_UDEV_FLAG0
@@ -20,3 +19,5 @@ dev_t dm_get_first_dep(char *devname);char *dm_mapuuid (int major,int minor);int dm_devn (char * mapname,int *major,int *minor);int dm_no_partitions (int major,int minor); + +#endif /* _KPARTX_DEVMAPPER_H */
kpartx/kpartx.c +5 -8
@@ -168,8 +168,8 @@ get_hotplug_device(void)if (stat (devname, &buf))return NULL ; - major = (unsigned int )MAJOR (buf.st_rdev); - minor = (unsigned int )MINOR (buf.st_rdev); + major =major (buf.st_rdev); + minor =minor (buf.st_rdev);if (!(mapname =dm_mapname (major, minor)))/* Not dm device. */ return NULL ;
@@ -327,10 +327,8 @@ main(int argc, char **argv){ off =find_devname_offset (device);if (!loopdev) { - uuid =dm_mapuuid ((unsigned int )MAJOR (buf.st_rdev), - (unsigned int )MINOR (buf.st_rdev)); - mapname =dm_mapname ((unsigned int )MAJOR (buf.st_rdev), - (unsigned int )MINOR (buf.st_rdev)); + uuid =dm_mapuuid (major (buf.st_rdev),minor (buf.st_rdev)); + mapname =dm_mapname (major (buf.st_rdev),minor (buf.st_rdev)); }if (!uuid)
@@ -339,8 +337,7 @@ main(int argc, char **argv){if (!mapname) mapname = device + off;else if (!force_devmap && -dm_no_partitions ((unsigned int )MAJOR (buf.st_rdev), - (unsigned int )MINOR (buf.st_rdev))) { +dm_no_partitions (major (buf.st_rdev),minor (buf.st_rdev))) {/* Feature 'no_partitions' is set, return */ return 0 ; }
Signature · 1 line
1.8.4.5
PATCH 39/78 multipath: Use standard 'major' macro
In reply to Hannes ReineckeInstead of carrying our own hand-crafted macros we should be using the system-provided 'major' macro.
--- multipath/dev_t.h | 3 --- multipath/main.c | 3 +-- 2 files changed, 1 insertion(+), 5 deletions(-) delete mode 100644 multipath/dev_t.h
multipath/dev_t.h deleted +0 -3
@@ -1,3 +0,0 @@ -#define MAJOR (dev) ((dev & 0xfff00) >> 8) -#define MINOR (dev) ((dev & 0xff) | ((dev >> 12) & 0xfff00)) -#define MKDEV (ma,mi) ((mi & 0xff) | (ma << 8) | ((mi & ~0xff) << 12))
multipath/main.c +1 -2
@@ -57,7 +57,6 @@#include <sys/resource.h> #include <wwids.h> #include <uxsock.h> -#include "dev_t.h" int logsink;
@@ -454,7 +453,7 @@ get_dev_type(char *dev) {int i;if (stat (dev, &buf) ==0 &&S_ISBLK (buf.st_mode)) { -if (dm_is_dm_major (MAJOR (buf.st_rdev))) +if (dm_is_dm_major (major (buf.st_rdev)))return DEV_DEVMAP ;return DEV_DEVNODE ; }
Signature · 1 line
1.8.4.5
PATCH 40/78 Remove sysfs_get_dev
In reply to Hannes ReineckePointless, and can be replaced by udev_device_get_devnum().
--- libmultipath/discovery.c | 9 ++++----- libmultipath/discovery.h | 1 - libmultipath/structs_vec.c | 4 +++- 3 files changed, 7 insertions(+), 7 deletions(-)
libmultipath/discovery.c +4 -5
@@ -172,7 +172,6 @@ declare_sysfs_get_str(devtype);declare_sysfs_get_str (vendor);declare_sysfs_get_str (model);declare_sysfs_get_str (rev); -declare_sysfs_get_str (dev);int sysfs_get_timeout (struct path *pp,unsigned int *timeout)
@@ -900,6 +899,8 @@ cciss_sysfs_pathinfo (struct path * pp)static int common_sysfs_pathinfo (struct path * pp) { +dev_t devt; +if (!pp)return 1 ;
@@ -907,10 +908,8 @@ common_sysfs_pathinfo (struct path * pp)condlog (4 ,"%s: udev not initialised" , pp->dev);return 1 ; } -if (sysfs_get_dev (pp->udev, pp->dev_t,BLK_DEV_SIZE ) <=0 ) { -condlog (3 ,"%s: no 'dev' attribute in sysfs" , pp->dev); -return 1 ; - } + devt =udev_device_get_devnum (pp->udev); +snprintf (pp->dev_t,BLK_DEV_SIZE ,"%d:%d" ,major (devt),minor (devt));condlog (3 ,"%s: dev_t = %s" , pp->dev, pp->dev_t);
libmultipath/discovery.h +0 -1
@@ -30,7 +30,6 @@struct config ; -ssize_t sysfs_get_dev (struct udev_device *udev,char * buff,size_t len);int path_discovery (vector pathvec,struct config * conf,int flag);int do_tur (char *);
libmultipath/structs_vec.c +3 -1
@@ -8,6 +8,7 @@#include "debug.h" #include "structs.h" #include "structs_vec.h" +#include "sysfs.h" #include "waiter.h" #include "devmapper.h" #include "dmparser.h"
@@ -501,7 +502,8 @@ verify_paths(struct multipath * mpp, struct vectors * vecs, vector rpvec)/* * see if path is in sysfs */ -if (sysfs_get_dev (pp->udev, pp->dev_t,BLK_DEV_SIZE ) <=0 ) { +if (sysfs_attr_get_value (pp->udev,"dev" , + pp->dev_t,BLK_DEV_SIZE ) <0 ) {if (pp->state !=PATH_DOWN ) {condlog (1 ,"%s: removing valid path %s in state %d" , mpp->alias, pp->dev, pp->state);
Signature · 1 line
1.8.4.5
PATCH 41/78 Add paths with a size of '0' as 'ghost' paths
In reply to Hannes ReineckeSPC-3 does not require that a port in 'standby' has to support the 'READ CAPACITY' command. As such we should not reject those paths, but rather add them as 'ghost' paths.
--- libmultipath/discovery.c | 5 +++++ libmultipath/structs_vec.c | 2 +- multipathd/main.c | 26 +++++--------------------- 3 files changed, 11 insertions(+), 22 deletions(-)
libmultipath/discovery.c +5 -0
@@ -1215,6 +1215,11 @@ pathinfo (struct path *pp, vector hwtable, int mask) goto blank;if (pp->state ==PATH_TIMEOUT ) pp->state =PATH_DOWN ; +if (pp->state ==PATH_UP && !pp->size) { +condlog (3 ,"%s: device size is 0, " +"path unuseable" , pp->dev); + pp->state =PATH_GHOST ; + } }else {condlog (3 ,"%s: path inaccessible" , pp->dev); pp->chkrstate = pp->state = path_state;
libmultipath/structs_vec.c +1 -1
@@ -558,7 +558,7 @@ int update_multipath (struct vectors *vecs, char *mapname, int reset)if (pp->state !=PATH_DOWN ) {int oldstate = pp->state; -condlog (2 ,"%s: mark as failed" , pp->dev_t); +condlog (2 ,"%s: mark as failed" , pp->dev); mpp->stat_path_failures++; pp->state =PATH_DOWN ; if (oldstate == PATH_UP ||
multipathd/main.c +5 -21
@@ -479,15 +479,10 @@ ev_add_path (struct path * pp, struct vectors * vecs) mpp = pp->mpp =find_mp_by_wwid (vecs->mpvec, pp->wwid); rescan:if (mpp) { -if ((!pp->size) || (mpp->size != pp->size)) { -if (!pp->size) -condlog (0 ,"%s: failed to add new path %s, " -"device size is 0" , - mpp->alias, pp->dev); -else -condlog (0 ,"%s: failed to add new path %s, " -"device size mismatch" , - mpp->alias, pp->dev); +if (mpp->size != pp->size) { +condlog (0 ,"%s: failed to add new path %s, " +"device size mismatch" , + mpp->alias, pp->dev);int i =find_slot (vecs->pathvec, (void *)pp);if (i !=-1 )vector_del_slot (vecs->pathvec, i);
@@ -503,18 +498,7 @@ rescan:verify_paths (mpp, vecs,NULL ); mpp->flush_on_last_del =FLUSH_UNDEF ; mpp->action =ACT_RELOAD ; - } -else { -if (!pp->size) { -condlog (0 ,"%s: failed to create new map," -" device size is 0 " , pp->dev); -int i =find_slot (vecs->pathvec, (void *)pp); -if (i !=-1 ) -vector_del_slot (vecs->pathvec, i); -free_path (pp); -return 1 ; - } - + }else {if (conf->find_multipaths && !should_multipath (pp, vecs->pathvec)) {orphan_path (pp,"only one path" );
Signature · 1 line
1.8.4.5
PATCH 42/78 Remove last argument from verify_paths()
In reply to Hannes ReineckeArgument is always NULL, so remove it.
--- libmultipath/configure.c | 2 +- libmultipath/structs_vec.c | 14 +++++--------- libmultipath/structs_vec.h | 2 +- multipathd/main.c | 2 +- 4 files changed, 8 insertions(+), 12 deletions(-)
libmultipath/configure.c +1 -1
@@ -820,7 +820,7 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_rif (pp2->priority ==PRIO_UNDEF ) mpp->action =ACT_REJECT ; } -verify_paths (mpp, vecs,NULL ); +verify_paths (mpp, vecs); params[0 ] ='\0' ;if (setup_map (mpp, params,PARAMS_SIZE )) {
libmultipath/structs_vec.c +5 -9
@@ -489,7 +489,7 @@ out: }extern int -verify_paths (struct multipath * mpp,struct vectors * vecs,vector rpvec) +verify_paths (struct multipath * mpp,struct vectors * vecs) {struct path * pp;int count =0 ;
@@ -515,14 +515,10 @@ verify_paths(struct multipath * mpp, struct vectors * vecs, vector rpvec)vector_del_slot (mpp->paths, i); i--; -if (rpvec) -store_path (rpvec, pp); -else { -if ((j =find_slot (vecs->pathvec, - (void *)pp)) !=-1 ) -vector_del_slot (vecs->pathvec, j); -free_path (pp); - } +if ((j =find_slot (vecs->pathvec, + (void *)pp)) !=-1 ) +vector_del_slot (vecs->pathvec, j); +free_path (pp); }else {condlog (4 ,"%s: verified path %s dev_t %s" , mpp->alias, pp->dev, pp->dev_t);
libmultipath/structs_vec.h +1 -1
@@ -19,7 +19,7 @@ int adopt_paths (vector pathvec, struct multipath * mpp, int get_info);void orphan_paths (vector pathvec,struct multipath * mpp);void orphan_path (struct path * pp,const char *reason); -int verify_paths (struct multipath * mpp,struct vectors * vecs,vector rpvec); +int verify_paths (struct multipath * mpp,struct vectors * vecs);int update_mpp_paths (struct multipath * mpp,vector pathvec);int __setup_multipath (struct vectors * vecs,struct multipath * mpp,int reset);
multipathd/main.c +1 -1
@@ -495,7 +495,7 @@ rescan:if (adopt_paths (vecs->pathvec, mpp,1 )) goto fail;/* leave path added to pathvec */ -verify_paths (mpp, vecs,NULL ); +verify_paths (mpp, vecs); mpp->flush_on_last_del =FLUSH_UNDEF ; mpp->action =ACT_RELOAD ; }else {
Signature · 1 line
1.8.4.5
PATCH 43/78 Fixup device-mapper 'cookie' handling
In reply to Hannes Reineckedevice-mapper has a 'cookie', which is inserted with the ioctl for modifying device-mapper devices.
It is used as a synchronization point between udev and any other applications to notify the latter when udev has finished processing the event.
Originally multipath would only use a single cookie for every transaction, and wait for that cookie at the end of the program. Which works well if you only have one transaction, but for several (like calling 'multipath') it will actually overwrite the cookie and fail to wait for earlier events.
This causes libdevmapper to create the device nodes on its own, and the device nodes not being handled by udev.
--- kpartx/devmapper.c | 53 +++++++++++++++++++++++++++++++++++------------ kpartx/devmapper.h | 4 ++-- kpartx/kpartx.c | 22 +++++++++----------- libmultipath/config.h | 1 - libmultipath/configure.c | 5 +++-- libmultipath/devmapper.c | 48 +++++++++++++++++++++++++++++++----------- libmultipath/devmapper.h | 2 +- multipath/main.c | 2 -- multipathd/cli_handlers.c | 4 ++-- 9 files changed, 94 insertions(+), 47 deletions(-)
kpartx/devmapper.c +40 -13
@@ -14,13 +14,6 @@#define MAX_PREFIX_LEN 8#define PARAMS_SIZE 1024 -#ifndef LIBDM_API_COOKIE -static inline int dm_task_set_cookie (struct dm_task *dmt,uint32_t *c,int a) -{ -return 1 ; -} -#endif -extern int dm_prereq (char * str,int x,int y,int z) {
@@ -60,10 +53,13 @@ dm_prereq (char * str, int x, int y, int z) }extern int -dm_simplecmd (int task,const char *name,int no_flush,uint32_t *cookie,uint16_t udev_flags) { +dm_simplecmd (int task,const char *name,int no_flush,uint16_t udev_flags) {int r =0 ;int udev_wait_flag = (task ==DM_DEVICE_RESUME || task ==DM_DEVICE_REMOVE ); +#ifdef LIBDM_API_COOKIE +uint32_t cookie =0 ; +#endif struct dm_task *dmt;if (!(dmt =dm_task_create (task)))
@@ -78,10 +74,23 @@ dm_simplecmd (int task, const char *name, int no_flush, uint32_t *cookie, uint16if (no_flush)dm_task_no_flush (dmt); -if (udev_wait_flag && !dm_task_set_cookie (dmt, cookie, ((udev_sync)?0 :DM_UDEV_DISABLE_LIBRARY_FALLBACK ) | udev_flags)) +#ifdef LIBDM_API_COOKIE +if (!udev_sync) + udev_flags |=DM_UDEV_DISABLE_LIBRARY_FALLBACK ; +if (udev_wait_flag && !dm_task_set_cookie (dmt, &cookie, udev_flags)) { +dm_udev_complete (cookie); goto out; + } +#endif r =dm_task_run (dmt); - +#ifdef LIBDM_API_COOKIE +if (udev_wait_flag) { +if (!r) +dm_udev_complete (cookie); +else +dm_udev_wait (cookie); + } +#endif out:dm_task_destroy (dmt);return r;
@@ -90,10 +99,14 @@ dm_simplecmd (int task, const char *name, int no_flush, uint32_t *cookie, uint16extern int dm_addmap (int task,const char *name,const char *target,const char *params,uint64_t size,int ro,const char *uuid,int part, -mode_t mode,uid_t uid,gid_t gid,uint32_t *cookie) { +mode_t mode,uid_t uid,gid_t gid) {int r =0 ;struct dm_task *dmt;char *prefixed_uuid =NULL ; +#ifdef LIBDM_API_COOKIE +uint32_t cookie =0 ; +uint16_t udev_flags =0 ; +#endif if (!(dmt =dm_task_create (task)))return 0 ;
@@ -128,10 +141,24 @@ dm_addmap (int task, const char *name, const char *target,dm_task_no_open_count (dmt); -if (task ==DM_DEVICE_CREATE && !dm_task_set_cookie (dmt, cookie, (udev_sync)?0 :DM_UDEV_DISABLE_LIBRARY_FALLBACK )) +#ifdef LIBDM_API_COOKIE +if (!udev_sync) + udev_flags =DM_UDEV_DISABLE_LIBRARY_FALLBACK ; +if (task ==DM_DEVICE_CREATE && + !dm_task_set_cookie (dmt, &cookie, udev_flags)) { +dm_udev_complete (cookie); goto addout; + } +#endif r =dm_task_run (dmt); - +#ifdef LIBDM_API_COOKIE +if (task ==DM_DEVICE_CREATE ) { +if (!r) +dm_udev_complete (cookie); +else +dm_udev_wait (cookie); + } +#endif addout:dm_task_destroy (dmt);free (prefixed_uuid);
kpartx/devmapper.h +2 -2
@@ -10,9 +10,9 @@extern int udev_sync;int dm_prereq (char *,int ,int ,int ); -int dm_simplecmd (int ,const char *,int ,uint32_t *,uint16_t ); +int dm_simplecmd (int ,const char *,int ,uint16_t );int dm_addmap (int ,const char *,const char *,const char *,uint64_t , -int ,const char *,int ,mode_t ,uid_t ,gid_t ,uint32_t *); +int ,const char *,int ,mode_t ,uid_t ,gid_t );int dm_map_present (char *);char *dm_mapname (int major,int minor);dev_t dm_get_first_dep (char *devname);
kpartx/kpartx.c +10 -12
@@ -208,7 +208,6 @@ main(int argc, char **argv){int hotplug =0 ;int loopcreated =0 ;struct stat buf; -uint32_t cookie =0 ;initpts ();init_crc32 ();
@@ -281,6 +280,8 @@ main(int argc, char **argv){#ifdef LIBDM_API_COOKIE if (!udev_sync)dm_udev_set_sync_support (0 ); +else +dm_udev_set_sync_support (1 );#endif if (dm_prereq (DM_TARGET ,0 ,0 ,0 ) && (what ==ADD || what ==DELETE || what ==UPDATE )) {
@@ -437,7 +438,7 @@ main(int argc, char **argv){continue ;if (!dm_simplecmd (DM_DEVICE_REMOVE , partname, -0 , &cookie,0 )) { +0 ,0 )) { r++;continue ; }
@@ -488,18 +489,19 @@ main(int argc, char **argv){if (!dm_addmap (op, partname,DM_TARGET , params, slices[j].size, ro, uuid, j+1 , buf.st_mode &0777 , buf.st_uid, - buf.st_gid, &cookie)) { + buf.st_gid)) {fprintf (stderr,"create/reload failed on %s\n" , partname); r++; }if (op ==DM_DEVICE_RELOAD && !dm_simplecmd (DM_DEVICE_RESUME , partname, -1 , &cookie,MPATH_UDEV_RELOAD_FLAG )) { +1 ,MPATH_UDEV_RELOAD_FLAG )) {fprintf (stderr,"resume failed on %s\n" , partname); r++; } +dm_devn (partname, &slices[j].major, &slices[j].minor);
@@ -551,14 +553,12 @@ main(int argc, char **argv){dm_addmap (op, partname,DM_TARGET , params, slices[j].size, ro, uuid, j+1 , buf.st_mode &0777 , - buf.st_uid, buf.st_gid, - &cookie); + buf.st_uid, buf.st_gid);if (op ==DM_DEVICE_RELOAD )dm_simplecmd (DM_DEVICE_RESUME , partname,1 , - &cookie,MPATH_UDEV_RELOAD_FLAG ); - +MPATH_UDEV_RELOAD_FLAG );dm_devn (partname, &slices[j].major, &slices[j].minor);
@@ -590,7 +590,7 @@ main(int argc, char **argv){continue ;if (!dm_simplecmd (DM_DEVICE_REMOVE , - partname,1 , &cookie,0 )) { + partname,1 ,0 )) { r++;continue ; }
@@ -616,9 +616,7 @@ main(int argc, char **argv){ }printf ("loop deleted : %s\n" , device); } -#ifdef LIBDM_API_COOKIE -dm_udev_wait (cookie); -#endif +dm_lib_release ();dm_lib_exit ();
libmultipath/config.h +0 -1
@@ -127,7 +127,6 @@ struct config {uid_t uid;gid_t gid;mode_t mode; -uint32_t cookie;int reassign_maps;int retain_hwhandler;int detect_prio;
libmultipath/configure.c +3 -2
@@ -623,7 +623,8 @@ domap (struct multipath * mpp, char * params)case ACT_RELOAD : r =dm_addmap_reload (mpp, params);if (r) - r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,MPATH_UDEV_RELOAD_FLAG ); + r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias, +0 ,MPATH_UDEV_RELOAD_FLAG );break ;case ACT_RESIZE :
@@ -641,7 +642,7 @@ domap (struct multipath * mpp, char * params)if (r) { r =dm_addmap_reload (mpp, params);if (r) - r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,MPATH_UDEV_RELOAD_FLAG ); + r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,0 ,MPATH_UDEV_RELOAD_FLAG ); }break ;
libmultipath/devmapper.c +36 -12
124 lines not on this page · raw message
libmultipath/devmapper.h +1 -1
@@ -16,7 +16,7 @@ void dm_init(void);int dm_prereq (void );int dm_drv_version (unsigned int * version,char * str);int dm_simplecmd_flush (int ,const char *,int ,uint16_t ); -int dm_simplecmd_noflush (int ,const char *,uint16_t ); +int dm_simplecmd_noflush (int ,const char *,int ,uint16_t );int dm_addmap_create (struct multipath *mpp,char *params);int dm_addmap_reload (struct multipath *mpp,char *params);int dm_map_present (const char *);
multipath/main.c +0 -2
@@ -675,8 +675,6 @@ main (int argc, char *argv[])condlog (3 ,"restart multipath configuration process" ); out: -udev_wait (conf->cookie); -dm_lib_release ();dm_lib_exit ();
multipathd/cli_handlers.c +2 -2
@@ -839,7 +839,7 @@ cli_suspend(void * v, char ** reply, int * len, void * data) {struct vectors * vecs = (struct vectors *)data;char * param =get_keyparam (v,MAP ); -int r =dm_simplecmd_noflush (DM_DEVICE_SUSPEND , param,0 ); +int r =dm_simplecmd_noflush (DM_DEVICE_SUSPEND , param,0 ,0 ); param =convert_dev (param,0 );condlog (2 ,"%s: suspend (operator)" , param);
@@ -861,7 +861,7 @@ cli_resume(void * v, char ** reply, int * len, void * data) {struct vectors * vecs = (struct vectors *)data;char * param =get_keyparam (v,MAP ); -int r =dm_simplecmd_noflush (DM_DEVICE_RESUME , param,0 ); +int r =dm_simplecmd_noflush (DM_DEVICE_RESUME , param,0 ,0 ); param =convert_dev (param,0 );condlog (2 ,"%s: resume (operator)" , param);
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 43/78] Fixup device-mapper 'cookie' handling
Quoted 10 lines · Hannes Reinecke
device-mapper has a 'cookie', which is inserted with the ioctl for modifying device-mapper devices.
It is used as a synchronization point between udev and any other applications to notify the latter when udev has finished processing the event.
Originally multipath would only use a single cookie for every transaction, and wait for that cookie at the end of the program. Which works well if you only have one transaction, but for several (like calling 'multipath') it will actually overwrite the cookie and fail to wait for earlier events.
That shouldn't be happening. Looking at the dm_task_set_cookie code
if (*cookie) {
if (!_get_cookie_sem(*cookie, &semid))
goto_bad;
} else if (!_udev_notify_sem_create(cookie, &semid))
goto_bad;The first time we use that cookie, it should be zero, so a new semaphore will be created, and the id will be returned to us. Subsequent calls to dm_task_set_cookie with the same cookie (which is now non-zero) should just be using the same semaphore, allowing you you wait just one time on all the actions linked to that cookie. This should be more efficient than having to wait on each action (since udev can complete them in the background as we go on).
If there really are cookies that are not waited for, you should be able to see that by running
# dmsetup udevcookies
After running the multipath command. Cookies won't go away until someone waits for them. If there are cookies listed there, then my guess would be that something is resetting conf->cookie to 0 after our first call to dm_task_set_cookie. If there aren't, then it would appear that something else, instead of not waiting for the cookies, is causing device mapper to fail back to manual device node creation.
If you can verify that you are passing the cookie value that you get back from the first call to dm_task_set_cookie() into sebsequent calls, and you still are seeing non-watited for cookies with "dmsetup udevcookies" then that sounds to me like a problem in device-mapper, and we should check that out.
-Ben
Quoted 446 lines
This causes libdevmapper to create the device nodes on its own, and the device nodes not being handled by udev.
Signed-off-by Hannes Reinecke <[email protected]>--- kpartx/devmapper.c | 53 +++++++++++++++++++++++++++++++++++------------ kpartx/devmapper.h | 4 ++-- kpartx/kpartx.c | 22 +++++++++----------- libmultipath/config.h | 1 - libmultipath/configure.c | 5 +++-- libmultipath/devmapper.c | 48 +++++++++++++++++++++++++++++++----------- libmultipath/devmapper.h | 2 +- multipath/main.c | 2 -- multipathd/cli_handlers.c | 4 ++-- 9 files changed, 94 insertions(+), 47 deletions(-)kpartx/devmapper.c +40
-13@@ -14,13 +14,6 @@#define MAX_PREFIX_LEN 8#define PARAMS_SIZE 1024 -#ifndef LIBDM_API_COOKIE -static inline int dm_task_set_cookie (struct dm_task *dmt,uint32_t *c,int a) -{ -return 1 ; -} -#endif -extern int dm_prereq (char * str,int x,int y,int z) {@@ -60,10 +53,13 @@ dm_prereq (char * str, int x, int y, int z) }extern int -dm_simplecmd (int task,const char *name,int no_flush,uint32_t *cookie,uint16_t udev_flags) { +dm_simplecmd (int task,const char *name,int no_flush,uint16_t udev_flags) {int r =0 ;int udev_wait_flag = (task ==DM_DEVICE_RESUME || task ==DM_DEVICE_REMOVE ); +#ifdef LIBDM_API_COOKIE +uint32_t cookie =0 ; +#endif struct dm_task *dmt;if (!(dmt =dm_task_create (task)))@@ -78,10 +74,23 @@ dm_simplecmd (int task, const char *name, int no_flush, uint32_t *cookie, uint16if (no_flush)dm_task_no_flush (dmt); -if (udev_wait_flag && !dm_task_set_cookie (dmt, cookie, ((udev_sync)?0 :DM_UDEV_DISABLE_LIBRARY_FALLBACK ) | udev_flags)) +#ifdef LIBDM_API_COOKIE +if (!udev_sync) + udev_flags |=DM_UDEV_DISABLE_LIBRARY_FALLBACK ; +if (udev_wait_flag && !dm_task_set_cookie (dmt, &cookie, udev_flags)) { +dm_udev_complete (cookie); goto out; + } +#endif r =dm_task_run (dmt); - +#ifdef LIBDM_API_COOKIE +if (udev_wait_flag) { +if (!r) +dm_udev_complete (cookie); +else +dm_udev_wait (cookie); + } +#endif out:dm_task_destroy (dmt);return r;@@ -90,10 +99,14 @@ dm_simplecmd (int task, const char *name, int no_flush, uint32_t *cookie, uint16extern int dm_addmap (int task,const char *name,const char *target,const char *params,uint64_t size,int ro,const char *uuid,int part, -mode_t mode,uid_t uid,gid_t gid,uint32_t *cookie) { +mode_t mode,uid_t uid,gid_t gid) {int r =0 ;struct dm_task *dmt;char *prefixed_uuid =NULL ; +#ifdef LIBDM_API_COOKIE +uint32_t cookie =0 ; +uint16_t udev_flags =0 ; +#endif if (!(dmt =dm_task_create (task)))return 0 ;@@ -128,10 +141,24 @@ dm_addmap (int task, const char *name, const char *target,dm_task_no_open_count (dmt); -if (task ==DM_DEVICE_CREATE && !dm_task_set_cookie (dmt, cookie, (udev_sync)?0 :DM_UDEV_DISABLE_LIBRARY_FALLBACK )) +#ifdef LIBDM_API_COOKIE +if (!udev_sync) + udev_flags =DM_UDEV_DISABLE_LIBRARY_FALLBACK ; +if (task ==DM_DEVICE_CREATE && + !dm_task_set_cookie (dmt, &cookie, udev_flags)) { +dm_udev_complete (cookie); goto addout; + } +#endif r =dm_task_run (dmt); - +#ifdef LIBDM_API_COOKIE +if (task ==DM_DEVICE_CREATE ) { +if (!r) +dm_udev_complete (cookie); +else +dm_udev_wait (cookie); + } +#endif addout:dm_task_destroy (dmt);free (prefixed_uuid);kpartx/devmapper.h +2
-2@@ -10,9 +10,9 @@extern int udev_sync;int dm_prereq (char *,int ,int ,int ); -int dm_simplecmd (int ,const char *,int ,uint32_t *,uint16_t ); +int dm_simplecmd (int ,const char *,int ,uint16_t );int dm_addmap (int ,const char *,const char *,const char *,uint64_t , -int ,const char *,int ,mode_t ,uid_t ,gid_t ,uint32_t *); +int ,const char *,int ,mode_t ,uid_t ,gid_t );int dm_map_present (char *);char *dm_mapname (int major,int minor);dev_t dm_get_first_dep (char *devname);kpartx/kpartx.c +10
-12@@ -208,7 +208,6 @@ main(int argc, char **argv){int hotplug =0 ;int loopcreated =0 ;struct stat buf; -uint32_t cookie =0 ;initpts ();init_crc32 ();@@ -281,6 +280,8 @@ main(int argc, char **argv){#ifdef LIBDM_API_COOKIE if (!udev_sync)dm_udev_set_sync_support (0 ); +else +dm_udev_set_sync_support (1 );#endif if (dm_prereq (DM_TARGET ,0 ,0 ,0 ) && (what ==ADD || what ==DELETE || what ==UPDATE )) {@@ -437,7 +438,7 @@ main(int argc, char **argv){continue ;if (!dm_simplecmd (DM_DEVICE_REMOVE , partname, -0 , &cookie,0 )) { +0 ,0 )) { r++;continue ; }@@ -488,18 +489,19 @@ main(int argc, char **argv){if (!dm_addmap (op, partname,DM_TARGET , params, slices[j].size, ro, uuid, j+1 , buf.st_mode &0777 , buf.st_uid, - buf.st_gid, &cookie)) { + buf.st_gid)) {fprintf (stderr,"create/reload failed on %s\n" , partname); r++; }if (op ==DM_DEVICE_RELOAD && !dm_simplecmd (DM_DEVICE_RESUME , partname, -1 , &cookie,MPATH_UDEV_RELOAD_FLAG )) { +1 ,MPATH_UDEV_RELOAD_FLAG )) {fprintf (stderr,"resume failed on %s\n" , partname); r++; } +dm_devn (partname, &slices[j].major, &slices[j].minor);@@ -551,14 +553,12 @@ main(int argc, char **argv){dm_addmap (op, partname,DM_TARGET , params, slices[j].size, ro, uuid, j+1 , buf.st_mode &0777 , - buf.st_uid, buf.st_gid, - &cookie); + buf.st_uid, buf.st_gid);if (op ==DM_DEVICE_RELOAD )dm_simplecmd (DM_DEVICE_RESUME , partname,1 , - &cookie,MPATH_UDEV_RELOAD_FLAG ); - +MPATH_UDEV_RELOAD_FLAG );dm_devn (partname, &slices[j].major, &slices[j].minor);@@ -590,7 +590,7 @@ main(int argc, char **argv){continue ;if (!dm_simplecmd (DM_DEVICE_REMOVE , - partname,1 , &cookie,0 )) { + partname,1 ,0 )) { r++;continue ; }@@ -616,9 +616,7 @@ main(int argc, char **argv){ }printf ("loop deleted : %s\n" , device); } -#ifdef LIBDM_API_COOKIE -dm_udev_wait (cookie); -#endif +dm_lib_release ();dm_lib_exit ();libmultipath/config.h +0
-1@@ -127,7 +127,6 @@ struct config {uid_t uid;gid_t gid;mode_t mode; -uint32_t cookie;int reassign_maps;int retain_hwhandler;int detect_prio;libmultipath/configure.c +3
-2@@ -623,7 +623,8 @@ domap (struct multipath * mpp, char * params)case ACT_RELOAD : r =dm_addmap_reload (mpp, params);if (r) - r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,MPATH_UDEV_RELOAD_FLAG ); + r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias, +0 ,MPATH_UDEV_RELOAD_FLAG );break ;case ACT_RESIZE :@@ -641,7 +642,7 @@ domap (struct multipath * mpp, char * params)if (r) { r =dm_addmap_reload (mpp, params);if (r) - r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,MPATH_UDEV_RELOAD_FLAG ); + r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,0 ,MPATH_UDEV_RELOAD_FLAG ); }break ;libmultipath/devmapper.c +36
-12@@ -216,6 +216,7 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_tint r =0 ;int udev_wait_flag = (need_sync && (task ==DM_DEVICE_RESUME || task ==DM_DEVICE_REMOVE )); +uint32_t cookie =0 ;struct dm_task *dmt;if (!(dmt =dm_task_create (task)))@@ -234,10 +235,18 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_tif (do_deferred (deferred_remove))dm_task_deferred_remove (dmt);#endif -if (udev_wait_flag && !dm_task_set_cookie (dmt, &conf->cookie, ((conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 ) | udev_flags)) +if (udev_wait_flag && !dm_task_set_cookie (dmt, &cookie, ((conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 ) | udev_flags)) { +dm_udev_complete (cookie); goto out; + } r =dm_task_run (dmt); +if (udev_wait_flag) { +if (!r) +dm_udev_complete (cookie); +else +udev_wait (cookie); + } out:dm_task_destroy (dmt);return r;@@ -249,8 +258,8 @@ dm_simplecmd_flush (int task, const char *name, int needsync, uint16_t udev_flag }extern int -dm_simplecmd_noflush (int task,const char *name,uint16_t udev_flags) { -return dm_simplecmd (task, name,1 ,1 , udev_flags,0 ); +dm_simplecmd_noflush (int task,const char *name,int needsync,uint16_t udev_flags) { +return dm_simplecmd (task, name,1 , needsync, udev_flags,0 ); } static int@@ -265,6 +274,7 @@ dm_addmap (int task, const char *target, struct multipath *mpp, char * params,int r =0 ;struct dm_task *dmt;char *prefixed_uuid =NULL ; +uint32_t cookie =0 ;if (!(dmt =dm_task_create (task)))return 0 ;@@ -305,10 +315,18 @@ dm_addmap (int task, const char *target, struct multipath *mpp, char * params,dm_task_no_open_count (dmt);if (task ==DM_DEVICE_CREATE && - !dm_task_set_cookie (dmt, &conf->cookie, (conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 )) + !dm_task_set_cookie (dmt, &cookie, (conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 )) { +dm_udev_complete (cookie); goto freeout; + } r =dm_task_run (dmt); +if (task ==DM_DEVICE_CREATE ) { +if (!r) +dm_udev_complete (cookie); +else +udev_wait (cookie); + } freeout:if (prefixed_uuid)FREE (prefixed_uuid);@@ -326,7 +344,8 @@ dm_addmap_create (struct multipath *mpp, char * params) {for (ro =0 ; ro <=1 ; ro++) {int err; -if (dm_addmap (DM_DEVICE_CREATE ,TGT_MPATH , mpp, params,1 , ro)) +if (dm_addmap (DM_DEVICE_CREATE ,TGT_MPATH , + mpp, params,1 , ro))return 1 ;/* * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD. @@ -755,14 +774,14 @@ dm_suspend_and_flush_map (const char * mapname)if (s) queue_if_no_path =0 ;else - s =dm_simplecmd_flush (DM_DEVICE_SUSPEND , mapname,0 ,0 ); + s =dm_simplecmd_flush (DM_DEVICE_SUSPEND , mapname,1 ,0 );if (!dm_flush_map (mapname)) {condlog (4 ,"multipath map %s removed" , mapname);return 0 ; }condlog (2 ,"failed to remove multipath map %s" , mapname); -dm_simplecmd_noflush (DM_DEVICE_RESUME , mapname,0 ); +dm_simplecmd_noflush (DM_DEVICE_RESUME , mapname,1 ,0 );if (queue_if_no_path) s =dm_queue_if_no_path ((char *)mapname,1 );return 1 ;@@ -1312,6 +1331,7 @@ dm_rename (char * old, char * new) {int r =0 ;struct dm_task *dmt; +uint32_t cookie;if (dm_rename_partmaps (old, new))return r;@@ -1327,14 +1347,18 @@ dm_rename (char * old, char * new)dm_task_no_open_count (dmt); -if (!dm_task_set_cookie (dmt, &conf->cookie, (conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 )) - goto out; -if (!dm_task_run (dmt)) +if (!dm_task_set_cookie (dmt, &cookie, (conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 )) goto out; + r =dm_task_run (dmt); + +if (!r) +dm_udev_complete (cookie); +else +udev_wait (cookie); - r =1 ; out:dm_task_destroy (dmt); +return r; }@@ -1399,7 +1423,7 @@ int dm_reassign_table(const char *name, char *old, char *new)condlog (3 ,"%s: failed to reassign targets" , name); goto out_reload; } -dm_simplecmd_noflush (DM_DEVICE_RESUME , name,MPATH_UDEV_RELOAD_FLAG ); +dm_simplecmd_noflush (DM_DEVICE_RESUME , name,1 ,MPATH_UDEV_RELOAD_FLAG ); } r =1 ;libmultipath/devmapper.h +1
-1@@ -16,7 +16,7 @@ void dm_init(void);int dm_prereq (void );int dm_drv_version (unsigned int * version,char * str);int dm_simplecmd_flush (int ,const char *,int ,uint16_t ); -int dm_simplecmd_noflush (int ,const char *,uint16_t ); +int dm_simplecmd_noflush (int ,const char *,int ,uint16_t );int dm_addmap_create (struct multipath *mpp,char *params);int dm_addmap_reload (struct multipath *mpp,char *params);int dm_map_present (const char *);multipath/main.c +0
-2@@ -675,8 +675,6 @@ main (int argc, char *argv[])condlog (3 ,"restart multipath configuration process" ); out: -udev_wait (conf->cookie); -dm_lib_release ();dm_lib_exit ();multipathd/cli_handlers.c +2
-2@@ -839,7 +839,7 @@ cli_suspend(void * v, char ** reply, int * len, void * data) {struct vectors * vecs = (struct vectors *)data;char * param =get_keyparam (v,MAP ); -int r =dm_simplecmd_noflush (DM_DEVICE_SUSPEND , param,0 ); +int r =dm_simplecmd_noflush (DM_DEVICE_SUSPEND , param,0 ,0 ); param =convert_dev (param,0 );condlog (2 ,"%s: suspend (operator)" , param);@@ -861,7 +861,7 @@ cli_resume(void * v, char ** reply, int * len, void * data) {struct vectors * vecs = (struct vectors *)data;char * param =get_keyparam (v,MAP ); -int r =dm_simplecmd_noflush (DM_DEVICE_RESUME , param,0 ); +int r =dm_simplecmd_noflush (DM_DEVICE_RESUME , param,0 ,0 ); param =convert_dev (param,0 );condlog (2 ,"%s: resume (operator)" , param);Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 43/78] Fixup device-mapper 'cookie' handling
Quoted 11 lines · Benjamin Marzinski
Quoted 10 lines · Hannes Reinecke
device-mapper has a 'cookie', which is inserted with the ioctl for modifying device-mapper devices.
It is used as a synchronization point between udev and any other applications to notify the latter when udev has finished processing the event.
Originally multipath would only use a single cookie for every transaction, and wait for that cookie at the end of the program. Which works well if you only have one transaction, but for several (like calling 'multipath') it will actually overwrite the cookie and fail to wait for earlier events.
Another thing that might be interesting to try for debugging purposes:
If you enable DM_UDEV_DISABLE_LIBRARY_FALLBACK for multipath as well as multipathd, then multipath shouldn't failback to manual device node creation. Even if multipath isn't waiting on it, udev should create the device node eventually. If you are always getting all of your device nodes when you do this, then it seems that there is a race going on. If nodes occasionally aren't getting created at all when DM_UDEV_DISABLE_LIBRARY_FALLBACK is enabled, then it would seem to point to an issue in udev.
just a thought.
Quoted 488 lines
That shouldn't be happening. Looking at the
dm_task_set_cookiecodeif (*cookie) { if (!_get_cookie_sem(*cookie, &semid)) goto_bad; } else if (!_udev_notify_sem_create(cookie, &semid)) goto_bad;The first time we use that cookie, it should be zero, so a new semaphore will be created, and the id will be returned to us. Subsequent calls to
dm_task_set_cookiewith the same cookie (which is now non-zero) should just be using the same semaphore, allowing you you wait just one time on all the actions linked to that cookie. This should be more efficient than having to wait on each action (since udev can complete them in the background as we go on).If there really are cookies that are not waited for, you should be able to see that by running
# dmsetup udevcookiesAfter running the multipath command. Cookies won't go away until someone waits for them. If there are cookies listed there, then my guess would be that something is resetting
conf->cookieto 0 after our first call todm_task_set_cookie. If there aren't, then it would appear that something else, instead of not waiting for the cookies, is causing device mapper to fail back to manual device node creation.If you can verify that you are passing the cookie value that you get back from the first call to
dm_task_set_cookie()into sebsequent calls, and you still are seeing non-watited for cookies with "dmsetup udevcookies" then that sounds to me like a problem in device-mapper, and we should check that out.-Ben
Quoted 446 lines
This causes libdevmapper to create the device nodes on its own, and the device nodes not being handled by udev.
Signed-off-by Hannes Reinecke <[email protected]>--- kpartx/devmapper.c | 53 +++++++++++++++++++++++++++++++++++------------ kpartx/devmapper.h | 4 ++-- kpartx/kpartx.c | 22 +++++++++----------- libmultipath/config.h | 1 - libmultipath/configure.c | 5 +++-- libmultipath/devmapper.c | 48 +++++++++++++++++++++++++++++++----------- libmultipath/devmapper.h | 2 +- multipath/main.c | 2 -- multipathd/cli_handlers.c | 4 ++-- 9 files changed, 94 insertions(+), 47 deletions(-)kpartx/devmapper.c +40
-13@@ -14,13 +14,6 @@#define MAX_PREFIX_LEN 8#define PARAMS_SIZE 1024 -#ifndef LIBDM_API_COOKIE -static inline int dm_task_set_cookie (struct dm_task *dmt,uint32_t *c,int a) -{ -return 1 ; -} -#endif -extern int dm_prereq (char * str,int x,int y,int z) {@@ -60,10 +53,13 @@ dm_prereq (char * str, int x, int y, int z) }extern int -dm_simplecmd (int task,const char *name,int no_flush,uint32_t *cookie,uint16_t udev_flags) { +dm_simplecmd (int task,const char *name,int no_flush,uint16_t udev_flags) {int r =0 ;int udev_wait_flag = (task ==DM_DEVICE_RESUME || task ==DM_DEVICE_REMOVE ); +#ifdef LIBDM_API_COOKIE +uint32_t cookie =0 ; +#endif struct dm_task *dmt;if (!(dmt =dm_task_create (task)))@@ -78,10 +74,23 @@ dm_simplecmd (int task, const char *name, int no_flush, uint32_t *cookie, uint16if (no_flush)dm_task_no_flush (dmt); -if (udev_wait_flag && !dm_task_set_cookie (dmt, cookie, ((udev_sync)?0 :DM_UDEV_DISABLE_LIBRARY_FALLBACK ) | udev_flags)) +#ifdef LIBDM_API_COOKIE +if (!udev_sync) + udev_flags |=DM_UDEV_DISABLE_LIBRARY_FALLBACK ; +if (udev_wait_flag && !dm_task_set_cookie (dmt, &cookie, udev_flags)) { +dm_udev_complete (cookie); goto out; + } +#endif r =dm_task_run (dmt); - +#ifdef LIBDM_API_COOKIE +if (udev_wait_flag) { +if (!r) +dm_udev_complete (cookie); +else +dm_udev_wait (cookie); + } +#endif out:dm_task_destroy (dmt);return r;@@ -90,10 +99,14 @@ dm_simplecmd (int task, const char *name, int no_flush, uint32_t *cookie, uint16extern int dm_addmap (int task,const char *name,const char *target,const char *params,uint64_t size,int ro,const char *uuid,int part, -mode_t mode,uid_t uid,gid_t gid,uint32_t *cookie) { +mode_t mode,uid_t uid,gid_t gid) {int r =0 ;struct dm_task *dmt;char *prefixed_uuid =NULL ; +#ifdef LIBDM_API_COOKIE +uint32_t cookie =0 ; +uint16_t udev_flags =0 ; +#endif if (!(dmt =dm_task_create (task)))return 0 ;@@ -128,10 +141,24 @@ dm_addmap (int task, const char *name, const char *target,dm_task_no_open_count (dmt); -if (task ==DM_DEVICE_CREATE && !dm_task_set_cookie (dmt, cookie, (udev_sync)?0 :DM_UDEV_DISABLE_LIBRARY_FALLBACK )) +#ifdef LIBDM_API_COOKIE +if (!udev_sync) + udev_flags =DM_UDEV_DISABLE_LIBRARY_FALLBACK ; +if (task ==DM_DEVICE_CREATE && + !dm_task_set_cookie (dmt, &cookie, udev_flags)) { +dm_udev_complete (cookie); goto addout; + } +#endif r =dm_task_run (dmt); - +#ifdef LIBDM_API_COOKIE +if (task ==DM_DEVICE_CREATE ) { +if (!r) +dm_udev_complete (cookie); +else +dm_udev_wait (cookie); + } +#endif addout:dm_task_destroy (dmt);free (prefixed_uuid);kpartx/devmapper.h +2
-2@@ -10,9 +10,9 @@extern int udev_sync;int dm_prereq (char *,int ,int ,int ); -int dm_simplecmd (int ,const char *,int ,uint32_t *,uint16_t ); +int dm_simplecmd (int ,const char *,int ,uint16_t );int dm_addmap (int ,const char *,const char *,const char *,uint64_t , -int ,const char *,int ,mode_t ,uid_t ,gid_t ,uint32_t *); +int ,const char *,int ,mode_t ,uid_t ,gid_t );int dm_map_present (char *);char *dm_mapname (int major,int minor);dev_t dm_get_first_dep (char *devname);kpartx/kpartx.c +10
-12@@ -208,7 +208,6 @@ main(int argc, char **argv){int hotplug =0 ;int loopcreated =0 ;struct stat buf; -uint32_t cookie =0 ;initpts ();init_crc32 ();@@ -281,6 +280,8 @@ main(int argc, char **argv){#ifdef LIBDM_API_COOKIE if (!udev_sync)dm_udev_set_sync_support (0 ); +else +dm_udev_set_sync_support (1 );#endif if (dm_prereq (DM_TARGET ,0 ,0 ,0 ) && (what ==ADD || what ==DELETE || what ==UPDATE )) {@@ -437,7 +438,7 @@ main(int argc, char **argv){continue ;if (!dm_simplecmd (DM_DEVICE_REMOVE , partname, -0 , &cookie,0 )) { +0 ,0 )) { r++;continue ; }@@ -488,18 +489,19 @@ main(int argc, char **argv){if (!dm_addmap (op, partname,DM_TARGET , params, slices[j].size, ro, uuid, j+1 , buf.st_mode &0777 , buf.st_uid, - buf.st_gid, &cookie)) { + buf.st_gid)) {fprintf (stderr,"create/reload failed on %s\n" , partname); r++; }if (op ==DM_DEVICE_RELOAD && !dm_simplecmd (DM_DEVICE_RESUME , partname, -1 , &cookie,MPATH_UDEV_RELOAD_FLAG )) { +1 ,MPATH_UDEV_RELOAD_FLAG )) {fprintf (stderr,"resume failed on %s\n" , partname); r++; } +dm_devn (partname, &slices[j].major, &slices[j].minor);@@ -551,14 +553,12 @@ main(int argc, char **argv){dm_addmap (op, partname,DM_TARGET , params, slices[j].size, ro, uuid, j+1 , buf.st_mode &0777 , - buf.st_uid, buf.st_gid, - &cookie); + buf.st_uid, buf.st_gid);if (op ==DM_DEVICE_RELOAD )dm_simplecmd (DM_DEVICE_RESUME , partname,1 , - &cookie,MPATH_UDEV_RELOAD_FLAG ); - +MPATH_UDEV_RELOAD_FLAG );dm_devn (partname, &slices[j].major, &slices[j].minor);@@ -590,7 +590,7 @@ main(int argc, char **argv){continue ;if (!dm_simplecmd (DM_DEVICE_REMOVE , - partname,1 , &cookie,0 )) { + partname,1 ,0 )) { r++;continue ; }@@ -616,9 +616,7 @@ main(int argc, char **argv){ }printf ("loop deleted : %s\n" , device); } -#ifdef LIBDM_API_COOKIE -dm_udev_wait (cookie); -#endif +dm_lib_release ();dm_lib_exit ();libmultipath/config.h +0
-1@@ -127,7 +127,6 @@ struct config {uid_t uid;gid_t gid;mode_t mode; -uint32_t cookie;int reassign_maps;int retain_hwhandler;int detect_prio;libmultipath/configure.c +3
-2@@ -623,7 +623,8 @@ domap (struct multipath * mpp, char * params)case ACT_RELOAD : r =dm_addmap_reload (mpp, params);if (r) - r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,MPATH_UDEV_RELOAD_FLAG ); + r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias, +0 ,MPATH_UDEV_RELOAD_FLAG );break ;case ACT_RESIZE :@@ -641,7 +642,7 @@ domap (struct multipath * mpp, char * params)if (r) { r =dm_addmap_reload (mpp, params);if (r) - r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,MPATH_UDEV_RELOAD_FLAG ); + r =dm_simplecmd_noflush (DM_DEVICE_RESUME , mpp->alias,0 ,MPATH_UDEV_RELOAD_FLAG ); }break ;libmultipath/devmapper.c +36
-12@@ -216,6 +216,7 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_tint r =0 ;int udev_wait_flag = (need_sync && (task ==DM_DEVICE_RESUME || task ==DM_DEVICE_REMOVE )); +uint32_t cookie =0 ;struct dm_task *dmt;if (!(dmt =dm_task_create (task)))@@ -234,10 +235,18 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_tif (do_deferred (deferred_remove))dm_task_deferred_remove (dmt);#endif -if (udev_wait_flag && !dm_task_set_cookie (dmt, &conf->cookie, ((conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 ) | udev_flags)) +if (udev_wait_flag && !dm_task_set_cookie (dmt, &cookie, ((conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 ) | udev_flags)) { +dm_udev_complete (cookie); goto out; + } r =dm_task_run (dmt); +if (udev_wait_flag) { +if (!r) +dm_udev_complete (cookie); +else +udev_wait (cookie); + } out:dm_task_destroy (dmt);return r;@@ -249,8 +258,8 @@ dm_simplecmd_flush (int task, const char *name, int needsync, uint16_t udev_flag }extern int -dm_simplecmd_noflush (int task,const char *name,uint16_t udev_flags) { -return dm_simplecmd (task, name,1 ,1 , udev_flags,0 ); +dm_simplecmd_noflush (int task,const char *name,int needsync,uint16_t udev_flags) { +return dm_simplecmd (task, name,1 , needsync, udev_flags,0 ); } static int@@ -265,6 +274,7 @@ dm_addmap (int task, const char *target, struct multipath *mpp, char * params,int r =0 ;struct dm_task *dmt;char *prefixed_uuid =NULL ; +uint32_t cookie =0 ;if (!(dmt =dm_task_create (task)))return 0 ;@@ -305,10 +315,18 @@ dm_addmap (int task, const char *target, struct multipath *mpp, char * params,dm_task_no_open_count (dmt);if (task ==DM_DEVICE_CREATE && - !dm_task_set_cookie (dmt, &conf->cookie, (conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 )) + !dm_task_set_cookie (dmt, &cookie, (conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 )) { +dm_udev_complete (cookie); goto freeout; + } r =dm_task_run (dmt); +if (task ==DM_DEVICE_CREATE ) { +if (!r) +dm_udev_complete (cookie); +else +udev_wait (cookie); + } freeout:if (prefixed_uuid)FREE (prefixed_uuid);@@ -326,7 +344,8 @@ dm_addmap_create (struct multipath *mpp, char * params) {for (ro =0 ; ro <=1 ; ro++) {int err; -if (dm_addmap (DM_DEVICE_CREATE ,TGT_MPATH , mpp, params,1 , ro)) +if (dm_addmap (DM_DEVICE_CREATE ,TGT_MPATH , + mpp, params,1 , ro))return 1 ;/* * DM_DEVICE_CREATE is actually DM_DEV_CREATE + DM_TABLE_LOAD. @@ -755,14 +774,14 @@ dm_suspend_and_flush_map (const char * mapname)if (s) queue_if_no_path =0 ;else - s =dm_simplecmd_flush (DM_DEVICE_SUSPEND , mapname,0 ,0 ); + s =dm_simplecmd_flush (DM_DEVICE_SUSPEND , mapname,1 ,0 );if (!dm_flush_map (mapname)) {condlog (4 ,"multipath map %s removed" , mapname);return 0 ; }condlog (2 ,"failed to remove multipath map %s" , mapname); -dm_simplecmd_noflush (DM_DEVICE_RESUME , mapname,0 ); +dm_simplecmd_noflush (DM_DEVICE_RESUME , mapname,1 ,0 );if (queue_if_no_path) s =dm_queue_if_no_path ((char *)mapname,1 );return 1 ;@@ -1312,6 +1331,7 @@ dm_rename (char * old, char * new) {int r =0 ;struct dm_task *dmt; +uint32_t cookie;if (dm_rename_partmaps (old, new))return r;@@ -1327,14 +1347,18 @@ dm_rename (char * old, char * new)dm_task_no_open_count (dmt); -if (!dm_task_set_cookie (dmt, &conf->cookie, (conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 )) - goto out; -if (!dm_task_run (dmt)) +if (!dm_task_set_cookie (dmt, &cookie, (conf->daemon)?DM_UDEV_DISABLE_LIBRARY_FALLBACK :0 )) goto out; + r =dm_task_run (dmt); + +if (!r) +dm_udev_complete (cookie); +else +udev_wait (cookie); - r =1 ; out:dm_task_destroy (dmt); +return r; }@@ -1399,7 +1423,7 @@ int dm_reassign_table(const char *name, char *old, char *new)condlog (3 ,"%s: failed to reassign targets" , name); goto out_reload; } -dm_simplecmd_noflush (DM_DEVICE_RESUME , name,MPATH_UDEV_RELOAD_FLAG ); +dm_simplecmd_noflush (DM_DEVICE_RESUME , name,1 ,MPATH_UDEV_RELOAD_FLAG ); } r =1 ;libmultipath/devmapper.h +1
-1@@ -16,7 +16,7 @@ void dm_init(void);int dm_prereq (void );int dm_drv_version (unsigned int * version,char * str);int dm_simplecmd_flush (int ,const char *,int ,uint16_t ); -int dm_simplecmd_noflush (int ,const char *,uint16_t ); +int dm_simplecmd_noflush (int ,const char *,int ,uint16_t );int dm_addmap_create (struct multipath *mpp,char *params);int dm_addmap_reload (struct multipath *mpp,char *params);int dm_map_present (const char *);multipath/main.c +0
-2@@ -675,8 +675,6 @@ main (int argc, char *argv[])condlog (3 ,"restart multipath configuration process" ); out: -udev_wait (conf->cookie); -dm_lib_release ();dm_lib_exit ();multipathd/cli_handlers.c +2
-2@@ -839,7 +839,7 @@ cli_suspend(void * v, char ** reply, int * len, void * data) {struct vectors * vecs = (struct vectors *)data;char * param =get_keyparam (v,MAP ); -int r =dm_simplecmd_noflush (DM_DEVICE_SUSPEND , param,0 ); +int r =dm_simplecmd_noflush (DM_DEVICE_SUSPEND , param,0 ,0 ); param =convert_dev (param,0 );condlog (2 ,"%s: suspend (operator)" , param);@@ -861,7 +861,7 @@ cli_resume(void * v, char ** reply, int * len, void * data) {struct vectors * vecs = (struct vectors *)data;char * param =get_keyparam (v,MAP ); -int r =dm_simplecmd_noflush (DM_DEVICE_RESUME , param,0 ); +int r =dm_simplecmd_noflush (DM_DEVICE_RESUME , param,0 ,0 ); param =convert_dev (param,0 );condlog (2 ,"%s: resume (operator)" , param);Signature · 1 line
1.8.4.5--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Re: [PATCH 43/78] Fixup device-mapper 'cookie' handling
In reply to Benjamin MarzinskiQuoted 40 lines · Benjamin Marzinski
Quoted 10 lines · Hannes Reinecke
device-mapper has a 'cookie', which is inserted with the ioctl for modifying device-mapper devices.
It is used as a synchronization point between udev and any other applications to notify the latter when udev has finished processing the event.
Originally multipath would only use a single cookie for every transaction, and wait for that cookie at the end of the program. Which works well if you only have one transaction, but for several (like calling 'multipath') it will actually overwrite the cookie and fail to wait for earlier events.That shouldn't be happening. Looking at the
dm_task_set_cookiecodeif (*cookie) { if (!_get_cookie_sem(*cookie, &semid)) goto_bad; } else if (!_udev_notify_sem_create(cookie, &semid)) goto_bad;The first time we use that cookie, it should be zero, so a new semaphore will be created, and the id will be returned to us. Subsequent calls to
dm_task_set_cookiewith the same cookie (which is now non-zero) should just be using the same semaphore, allowing you you wait just one time on all the actions linked to that cookie. This should be more efficient than having to wait on each action (since udev can complete them in the background as we go on).If there really are cookies that are not waited for, you should be able to see that by running
# dmsetup udevcookiesAfter running the multipath command. Cookies won't go away until someone waits for them. If there are cookies listed there, then my guess would be that something is resetting
conf->cookieto 0 after our first call todm_task_set_cookie. If there aren't, then it would appear that something else, instead of not waiting for the cookies, is causing device mapper to fail back to manual device node creation.
Quoted 6 lines
If you can verify that you are passing the cookie value that you get back from the first call to
dm_task_set_cookie()into sebsequent calls, and you still are seeing non-watited for cookies with "dmsetup udevcookies" then that sounds to me like a problem in device-mapper, and we should check that out.
Hmm. It _might_ have been caused by multipathd not properly synchronized with udev (the original service files didn't have a dependency on udev), hence the call to dm_udev_set_sync_support() would cause device-mapper to switch off the udev fallback.
Speaking of nasty side-effects ...
Can't we have a proper warning in dm_udev_set_sync_support(), alerting us that the system will not behave as expected?
But with that resolved it might be that indeed the patch is not required. I'll check.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
PATCH 44/78 multipath: do not print state 'orphan' for option '-l'
In reply to Hannes ReineckeWhen called with '-l' multipath would mistakenly print a state of 'orphan' instead off 'running' of 'offline'.
This is a regression introduced by commit 7023d320.
--- libmultipath/print.c | 2 -- 1 file changed, 2 deletions(-)
libmultipath/print.c +0 -2
@@ -322,8 +322,6 @@ snprint_offline (char * buff, size_t len, struct path * pp)return snprintf (buff, len,"unknown" ); else if (pp->offline) return snprintf(buff, len, "offline"); - else if (!pp->mpp) - return snprintf(buff, len, "orphan"); else return snprintf(buff, len, "running"); }
Signature · 1 line
1.8.4.5
PATCH 45/78 Return error when receiving CLI packet
In reply to Hannes ReineckeWhen receiving a packet from the CLI we should be returning an error code.
--- libmpathpersist/mpath_updatepr.c | 20 ++++++++++++-------- libmultipath/configure.c | 2 +- libmultipath/defaults.h | 1 + libmultipath/uxsock.c | 30 +++++++++++++++++++----------- libmultipath/uxsock.h | 4 ++-- multipathd/main.c | 4 ++-- multipathd/uxclnt.c | 26 +++++++++++++++++--------- multipathd/uxclnt.h | 2 +- multipathd/uxlsnr.c | 3 ++- 9 files changed, 57 insertions(+), 35 deletions(-)
libmpathpersist/mpath_updatepr.c +12 -8
@@ -35,15 +35,19 @@ int update_prflag(char * arg1, char * arg2, int noisy)snprintf (str,sizeof (str),"map %s %s" , arg1, arg2);condlog (2 ,"%s: pr flag message=%s" , arg1, str);send_packet (fd, str,strlen (str) +1 ); -recv_packet (fd, &reply, &len); - -condlog (2 ,"%s: message=%s reply=%s" , arg1, str, reply); -if (!reply ||strncmp (reply,"ok" ,2 ) ==0 ) - ret =-1 ; -else if (strncmp (reply,"fail" ,4 ) ==0 ) + ret =recv_packet (fd, &reply, &len,DEFAULT_UXSOCK_TIMEOUT ); +if (ret <0 ) { +condlog (2 ,"%s: message=%s error=%d" , arg1, str, -ret); ret =-2 ; -else { - ret =atoi (reply); + }else { +condlog (2 ,"%s: message=%s reply=%s" , arg1, str, reply); +if (!reply ||strncmp (reply,"ok" ,2 ) ==0 ) + ret =-1 ; +else if (strncmp (reply,"fail" ,4 ) ==0 ) + ret =-2 ; +else { + ret =atoi (reply); + } }free (reply);
libmultipath/configure.c +1 -1
@@ -714,7 +714,7 @@ int check_daemon(void)if (send_packet (fd,"show daemon" ,12 ) !=0 ) goto out; -if (recv_packet (fd, &reply, &len) !=0 ) +if (recv_packet (fd, &reply, &len,DEFAULT_UXSOCK_TIMEOUT ) !=0 ) goto out;if (strstr (reply,"shutdown" ))
libmultipath/defaults.h +1 -0
@@ -20,6 +20,7 @@#define DEFAULT_DEFERRED_REMOVE DEFERRED_REMOVE_OFF#define DEFAULT_DELAY_CHECKS DELAY_CHECKS_OFF#define DEFAULT_UEVENT_STACKSIZE 256 +#define DEFAULT_UXSOCK_TIMEOUT 1000#define DEFAULT_CHECKINT 5#define MAX_CHECKINT (a) (a << 2)
libmultipath/uxsock.c +19 -11
@@ -128,7 +128,7 @@ size_t write_all(int fd, const void *buf, size_t len)/* * keep reading until its all read */ -size_t read_all (int fd,void *buf,size_t len) +ssize_t read_all (int fd,void *buf,size_t len,unsigned int timeout) {size_t total =0 ;ssize_t n;
@@ -138,21 +138,20 @@ size_t read_all(int fd, void *buf, size_t len)while (len) { pfd.fd = fd; pfd.events =POLLIN ; - ret =poll (&pfd,1 ,1000 ); + ret =poll (&pfd,1 , timeout);if (!ret) { - errno =ETIMEDOUT ; -return total; +return -ETIMEDOUT ; }else if (ret <0 ) {if (errno ==EINTR )continue ; -return total; +return -errno; }else if (!pfd.revents &POLLIN )continue ; n =read (fd, buf, len);if (n <0 ) {if ((errno ==EINTR ) || (errno ==EAGAIN ))continue ; -return total; +return -errno; }if (!n)return total;
@@ -190,12 +189,20 @@ int send_packet(int fd, const char *buf, size_t len)/* * receive a packet in length prefix format */ -int recv_packet (int fd,char **buf,size_t *len) +int recv_packet (int fd,char **buf,size_t *len,unsigned int timeout) { -if (read_all (fd, len,sizeof (*len)) !=sizeof (*len)) { +ssize_t ret; + + ret =read_all (fd, len,sizeof (*len), timeout); +if (ret <0 ) { (*buf) =NULL ; *len =0 ; -return -1 ; +return ret; + } +if (ret <sizeof (*len)) { + (*buf) =NULL ; + *len =0 ; +return -EIO ; }if (len ==0 ) { (*buf) =NULL ;
@@ -204,11 +211,12 @@ int recv_packet(int fd, char **buf, size_t *len) (*buf) =MALLOC (*len);if (!*buf)return -1 ; -if (read_all (fd, *buf, *len) != *len) { + ret =read_all (fd, *buf, *len, timeout); +if (ret != *len) {FREE (*buf); (*buf) =NULL ; *len =0 ; -return -1 ; +return ret <0 ? ret : -EIO ; }return 0 ; }
libmultipath/uxsock.h +2 -2
@@ -2,6 +2,6 @@int ux_socket_connect (const char *name);int ux_socket_listen (const char *name);int send_packet (int fd,const char *buf,size_t len); -int recv_packet (int fd,char **buf,size_t *len); +int recv_packet (int fd,char **buf,size_t *len,unsigned int timeout);size_t write_all (int fd,const void *buf,size_t len); -size_t read_all (int fd,void *buf,size_t len); +ssize_t read_all (int fd,void *buf,size_t len,unsigned int timeout);
multipathd/main.c +2 -2
@@ -2023,7 +2023,7 @@ main (int argc, char *argv[]) logsink =-1 ;break ;case 'k' : -uxclnt (optarg); +uxclnt (optarg,DEFAULT_UXSOCK_TIMEOUT );exit (0 );case 'B' : conf->bindings_read_only =1 ;
@@ -2045,7 +2045,7 @@ main (int argc, char *argv[]) optind++; } c +=snprintf (c, s +CMDSIZE - c,"\n" ); -uxclnt (s); +uxclnt (s,DEFAULT_UXSOCK_TIMEOUT );exit (0 ); }
multipathd/uxclnt.c +17 -9
@@ -9,6 +9,7 @@#include <unistd.h> #include <stdarg.h> #include <fcntl.h> +#include <errno.h> #include <sys/ioctl.h> #include <sys/types.h> #include <sys/socket.h>
@@ -63,10 +64,11 @@ static int need_quit(char *str, size_t len)/* * process the client */ -static void process (int fd) +static void process (int fd,unsigned int timeout) {char *line;char *reply; +int ret;cli_init (); rl_readline_name ="multipathd" ;
@@ -84,7 +86,8 @@ static void process(int fd)break ;if (send_packet (fd, line, llen +1 ) !=0 )break ; -if (recv_packet (fd, &reply, &len) !=0 )break ; + ret =recv_packet (fd, &reply, &len, timeout); +if (ret !=0 )break ;print_reply (reply);
@@ -96,18 +99,23 @@ static void process(int fd) } } -static void process_req (int fd,char * inbuf) +static void process_req (int fd,char * inbuf,unsigned int timeout) {char *reply;size_t len; +int ret;if (send_packet (fd, inbuf,strlen (inbuf) +1 ) !=0 ) {printf ("cannot send packet\n" );return ; } -if (recv_packet (fd, &reply, &len) !=0 ) -printf ("error receiving packet\n" ); -else { + ret =recv_packet (fd, &reply, &len, timeout); +if (ret <0 ) { +if (ret == -ETIMEDOUT ) +printf ("timeout receiving packet\n" ); +else +printf ("error %d receiving packet\n" , ret); + }else {printf ("%s" , reply);FREE (reply); }
@@ -116,7 +124,7 @@ static void process_req(int fd, char * inbuf)/* * entry point */ -int uxclnt (char * inbuf) +int uxclnt (char * inbuf,unsigned int timeout) {int fd;
@@ -125,9 +133,9 @@ int uxclnt(char * inbuf)exit (1 );if (inbuf) -process_req (fd, inbuf); +process_req (fd, inbuf, timeout);else -process (fd); +process (fd, timeout);return 0 ; }
multipathd/uxclnt.h +1 -1
@@ -1 +1 @@ -int uxclnt (char * inbuf); +int uxclnt (char * inbuf,unsigned int timeout);
multipathd/uxlsnr.c +2 -1
@@ -158,7 +158,8 @@ void * uxsock_listen(int (*uxsock_trigger)(char *, char **, int *, void *),struct client *next = c->next;if (polls[i].revents &POLLIN ) { -if (recv_packet (c->fd, &inbuf, &len) !=0 ) { +if (recv_packet (c->fd, &inbuf, &len, +DEFAULT_UXSOCK_TIMEOUT ) !=0 ) {dead_client (c); }else { inbuf[len -1 ] =0 ;
Signature · 1 line
1.8.4.5
PATCH 46/78 Implement 'uxsock_timeout' keyword
In reply to Hannes ReineckeOn larger systems the default CLI receive timeout of 1 second might be too small, and the CLI commands might fail before they can be processed. For these cases I've implemented a new keywork 'uxsock_timeout' which can be increased to avoid these issues.
--- libmultipath/config.c | 1 + libmultipath/config.h | 1 + libmultipath/configure.c | 2 +- libmultipath/dict.c | 30 ++++++++++++++++++++++++++++++ multipath/multipath.conf.5 | 9 +++++++++ multipathd/main.c | 8 ++++++-- 6 files changed, 48 insertions(+), 3 deletions(-)
libmultipath/config.c +1 -0
@@ -617,6 +617,7 @@ load_config (char * file, struct udev *udev) conf->partition_delim =NULL ; conf->processed_main_config =0 ; conf->find_multipaths =DEFAULT_FIND_MULTIPATHS ; + conf->uxsock_timeout =DEFAULT_UXSOCK_TIMEOUT ; conf->uid_attribute =set_default (DEFAULT_UID_ATTRIBUTE );/*
libmultipath/config.h +1 -0
@@ -135,6 +135,7 @@ struct config {int processed_main_config;int delay_watch_checks;int delay_wait_checks; +int uxsock_timeout;unsigned int version[3 ];char * dev;
libmultipath/configure.c +1 -1
@@ -714,7 +714,7 @@ int check_daemon(void)if (send_packet (fd,"show daemon" ,12 ) !=0 ) goto out; -if (recv_packet (fd, &reply, &len,DEFAULT_UXSOCK_TIMEOUT ) !=0 ) +if (recv_packet (fd, &reply, &len, conf->uxsock_timeout) !=0 ) goto out;if (strstr (reply,"shutdown" ))
libmultipath/dict.c +30 -0
@@ -1034,6 +1034,26 @@ declare_hw_snprint(delay_wait_checks, print_delay_checks) declare_mp_handler(delay_wait_checks, set_delay_checks) declare_mp_snprint(delay_wait_checks, print_delay_checks) +static int +def_uxsock_timeout_handler (vector strvec) +{ +unsigned int uxsock_timeout; +char *buff; + + buff =set_value (strvec); +if (!buff) +return 1 ; + +if (sscanf (buff,"%u" , &uxsock_timeout) ==1 && + uxsock_timeout >DEFAULT_UXSOCK_TIMEOUT ) + conf->uxsock_timeout = uxsock_timeout; +else + conf->uxsock_timeout =DEFAULT_UXSOCK_TIMEOUT ; + +free (buff); +return 0 ; +} +/* * blacklist block handlers */
@@ -1115,6 +1135,15 @@ declare_ble_handler(blist_property)declare_ble_handler (elist_property) static int +snprint_def_uxsock_timeout (char * buff,int len,void * data) +{ +if (conf->uxsock_timeout ==DEFAULT_UXSOCK_TIMEOUT ) +return 0 ; + +return snprintf (buff, len,"%u" , conf->uxsock_timeout); +} + +static int snprint_ble_simple (char * buff,int len,void * data) {struct blentry * ble = (struct blentry *)data;
@@ -1335,6 +1364,7 @@ init_keywords(void)install_keyword ("delay_watch_checks" , &def_delay_watch_checks_handler, &snprint_def_delay_watch_checks);install_keyword ("delay_wait_checks" , &def_delay_wait_checks_handler, &snprint_def_delay_wait_checks);install_keyword ("find_multipaths" , &def_find_multipaths_handler, &snprint_def_find_multipaths); +install_keyword ("uxsock_timeout" , &def_uxsock_timeout_handler, &snprint_def_uxsock_timeout);__deprecated install_keyword ("default_selector" , &def_selector_handler,NULL );__deprecated install_keyword ("default_path_grouping_policy" , &def_pgpolicy_handler,NULL );__deprecated install_keyword ("default_uid_attribute" , &def_uid_attribute_handler,NULL );
multipath/multipath.conf.5 +9 -0
@@ -480,6 +480,15 @@ used until it has passed .I delay_wait_checks checks. Default is .I no +.TP +.B uxsock_timeout +CLI receive timeout in milliseconds. For larger systems CLI commands +might timeout before the multipathd lock is released and the CLI command +can be processed. This will result in errors like +'timeout receiving packet' to be returned from CLI commands. +In these cases it is recommended to increase the CLI timeout to avoid +those issues. The default is +.I 1000 . .SH "blacklist section" The
multipathd/main.c +6 -2
@@ -2023,7 +2023,9 @@ main (int argc, char *argv[]) logsink =-1 ;break ;case 'k' : -uxclnt (optarg,DEFAULT_UXSOCK_TIMEOUT ); +if (load_config (DEFAULT_CONFIGFILE ,udev_new ())) +exit (1 ); +uxclnt (optarg, conf->uxsock_timeout);exit (0 );case 'B' : conf->bindings_read_only =1 ;
@@ -2037,6 +2039,8 @@ main (int argc, char *argv[])char * s = cmd;char * c = s; +if (load_config (DEFAULT_CONFIGFILE ,udev_new ())) +exit (1 );while (optind < argc) {if (strchr (argv[optind],' ' )) c +=snprintf (c, s +CMDSIZE - c,"\"%s\" " , argv[optind]);
@@ -2045,7 +2049,7 @@ main (int argc, char *argv[]) optind++; } c +=snprintf (c, s +CMDSIZE - c,"\n" ); -uxclnt (s,DEFAULT_UXSOCK_TIMEOUT ); +uxclnt (s, conf->uxsock_timeout);exit (0 ); }
Signature · 1 line
1.8.4.5
PATCH 47/78 Do not print empty multipaths section
In reply to Hannes ReineckeThe CLI command 'show config' or the 'multipath -t' command should not print an empty 'multipaths' section.
--- multipath/main.c | 11 +++++++---- multipathd/cli_handlers.c | 9 ++++++--- 2 files changed, 13 insertions(+), 7 deletions(-)
multipath/main.c +7 -4
@@ -436,10 +436,13 @@ dump_config (void) reply =REALLOC (reply, maxlen *=2 );continue ; } - c +=snprint_mptable (c, reply + maxlen - c, conf->mptable); - again = ((c - reply) == maxlen); -if (again) - reply =REALLOC (reply, maxlen *=2 ); +if (VECTOR_SIZE (conf->mptable) >0 ) { + c +=snprint_mptable (c, reply + maxlen - c, + conf->mptable); + again = ((c - reply) == maxlen); +if (again) + reply =REALLOC (reply, maxlen *=2 ); + } }printf ("%s" , reply);
multipathd/cli_handlers.c +6 -3
@@ -229,9 +229,12 @@ show_config (char ** r, int * len) maxlen *=2 ;continue ; } - c +=snprint_mptable (c, reply + maxlen - c, conf->mptable); - again = ((c - reply) == maxlen); -REALLOC_REPLY (reply, again, maxlen); +if (VECTOR_SIZE (conf->mptable) >0 ) { + c +=snprint_mptable (c, reply + maxlen - c, + conf->mptable); + again = ((c - reply) == maxlen); +REALLOC_REPLY (reply, again, maxlen); + } } *r = reply; *len = (int )(c - reply +1 );
Signature · 1 line
1.8.4.5
PATCH 48/78 multipathd: reload map if reinstate failed
In reply to Hannes ReineckeThe kernel might fail the 'reinstate' device-mapper message if the path is disabled. In these cases we need to reload the map to give device-mapper a chance to add the correct devices to the table.
--- multipathd/main.c | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-)
multipathd/main.c +23 -7
@@ -965,19 +965,23 @@ fail_path (struct path * pp, int del_active)/* * caller must have locked the path list before calling that function */ -static void +static int reinstate_path (struct path * pp,int add_active) { +int ret =0 ; +if (!pp->mpp) -return ; +return 0 ; -if (dm_reinstate_path (pp->mpp->alias, pp->dev_t)) +if (dm_reinstate_path (pp->mpp->alias, pp->dev_t)) {condlog (0 ,"%s: reinstate failed" , pp->dev_t); -else { + ret =1 ; + }else {condlog (2 ,"%s: reinstated" , pp->dev_t);if (add_active)update_queue_mode_add_path (pp->mpp); } +return ret; } static void
@@ -1132,6 +1136,7 @@ check_path (struct vectors * vecs, struct path * pp)int newstate;int new_path_up =0 ;int chkr_new_path_up =0 ; +int add_active;int oldchkrstate = pp->chkrstate;if (pp->initialized && !pp->mpp)
@@ -1259,11 +1264,17 @@ check_path (struct vectors * vecs, struct path * pp) oldstate != PATH_GHOST) { if (pp->mpp->delay_watch_checks > 0) pp->watch_checks = pp->mpp->delay_watch_checks; - reinstate_path(pp, 1); + add_active = 1; } else {if (pp->watch_checks >0 ) pp->watch_checks--; -reinstate_path (pp,0 ); + add_active =0 ; + } +if (reinstate_path (pp, add_active)) { +condlog (3 ,"%s: reload map" , pp->dev); +ev_add_path (pp, vecs); + pp->tick =1 ; +return 0 ; } new_path_up =1 ;
@@ -1281,7 +1292,12 @@ check_path (struct vectors * vecs, struct path * pp)if (pp->dmstate ==PSTATE_FAILED || pp->dmstate ==PSTATE_UNDEF ) {/* Clear IO errors */ -reinstate_path (pp,0 ); +if (reinstate_path (pp,0 )) { +condlog (3 ,"%s: reload map" , pp->dev); +ev_add_path (pp, vecs); + pp->tick =1 ; +return 0 ; + } }else {LOG_MSG (4 ,checker_message (&pp->checker));if (pp->checkint != conf->max_checkint) {
Signature · 1 line
1.8.4.5
PATCH 49/78 multipathd: do not remove paths without uevents
In reply to Hannes Reineckemultipathd should not remove any paths without the corresponding uevent. Otherwise the daemon will never be able to reinstate that path as it wouldn't get any uevents.
--- multipathd/main.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
multipathd/main.c +8 -5
@@ -1152,11 +1152,14 @@ check_path (struct vectors * vecs, struct path * pp) pp->tick = conf->checkint; newstate =path_offline (pp); -if (newstate ==PATH_REMOVED ) { -condlog (2 ,"%s: remove path (checker)" , pp->dev); -ev_remove_path (pp, vecs); -return 0 ; - } +/* +* Wait for uevent for removed paths; +* some LLDDs like zfcp keep paths unavailable +* without sending uevents. +*/ +if (newstate ==PATH_REMOVED ) + newstate =PATH_DOWN ; +if (newstate ==PATH_UP ) newstate =get_state (pp,1 );else
Signature · 1 line
1.8.4.5
PATCH 50/78 Allow zero-sized devices during configuration
In reply to Hannes ReineckeA size of '0' doesn't indicate an invalid device; other paths might end up with a correct size.
--- libmultipath/configure.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
libmultipath/configure.c +6 -5
@@ -768,8 +768,8 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_rcontinue ;/* 3. if path has disappeared */ -if (!pp1->size) { -orphan_path (pp1,"invalid size" ); +if (pp1->state ==PATH_REMOVED ) { +orphan_path (pp1,"path removed" );continue ; }
@@ -806,10 +806,11 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_rif (strcmp (pp1->wwid, pp2->wwid))continue ; -if (!pp2->size) -continue ; +if (!mpp->size && pp2->size) + mpp->size = pp2->size; -if (pp2->size != mpp->size) { +if (mpp->size && pp2->size && + pp2->size != mpp->size) {/* * ouch, avoid feeding that to the DM */
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 50/78] Allow zero-sized devices during configuration
Quoted 2 lines · Hannes Reinecke
A size of '0' doesn't indicate an invalid device; other paths might end up with a correct size.
don't we also need to change this is ev_add_path, then?
-Ben
Quoted 38 lines
Signed-off-by Hannes Reinecke <[email protected]>--- libmultipath/configure.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)libmultipath/configure.c +6
-5@@ -768,8 +768,8 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_rcontinue ;/* 3. if path has disappeared */ -if (!pp1->size) { -orphan_path (pp1,"invalid size" ); +if (pp1->state ==PATH_REMOVED ) { +orphan_path (pp1,"path removed" );continue ; }@@ -806,10 +806,11 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_rif (strcmp (pp1->wwid, pp2->wwid))continue ; -if (!pp2->size) -continue ; +if (!mpp->size && pp2->size) + mpp->size = pp2->size; -if (pp2->size != mpp->size) { +if (mpp->size && pp2->size && + pp2->size != mpp->size) {/* * ouch, avoid feeding that to the DM */ Signature · 1 line
1.8.4.5
Re: [PATCH 50/78] Allow zero-sized devices during configuration
Quoted 6 lines · Benjamin Marzinski
Quoted 2 lines · Hannes Reinecke
A size of '0' doesn't indicate an invalid device; other paths might end up with a correct size.
don't we also need to change this is
ev_add_path, then?
Probably. I'll check.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
PATCH 51/78 Rework uev_add_path()
In reply to Hannes ReineckeRework uev_add_path() to handle failed and blacklisted paths properly.
--- libmultipath/discovery.c | 34 +++++++++++++++++++++++ libmultipath/discovery.h | 2 ++ libmultipath/structs_vec.c | 3 +++ multipathd/main.c | 67 +++++++++++++++++++++++++++++----------------- 4 files changed, 82 insertions(+), 24 deletions(-)
libmultipath/discovery.c +34 -0
@@ -31,6 +31,40 @@#include "defaults.h" int +alloc_path_with_pathinfo (vector hwtable,struct udev_device *udevice, +int flag,struct path **pp_ptr) +{ +int err =PATHINFO_FAILED ; +struct path * pp; +const char * devname; + +if (pp_ptr) + *pp_ptr =NULL ; + + devname =udev_device_get_sysname (udevice); +if (!devname) +return PATHINFO_FAILED ; + + pp =alloc_path (); + +if (!pp) +return PATHINFO_FAILED ; + +if (safe_sprintf (pp->dev,"%s" , devname)) { +condlog (0 ,"pp->dev too small" ); + }else { + pp->udev =udev_device_ref (udevice); + err =pathinfo (pp, hwtable, flag |DI_BLACKLIST ); + } + +if (err) +free_path (pp); +else if (pp_ptr) + *pp_ptr = pp; +return err; +} + +int store_pathinfo (vector pathvec,vector hwtable,struct udev_device *udevice,int flag,struct path **pp_ptr) {
libmultipath/discovery.h +2 -0
@@ -36,6 +36,8 @@ int do_tur (char *);int path_offline (struct path *);int get_state (struct path * pp,int daemon);int pathinfo (struct path *,vector hwtable,int mask); +int alloc_path_with_pathinfo (vector hwtable,struct udev_device *udevice, +int flag,struct path **pp_ptr);int store_pathinfo (vector pathvec,vector hwtable,struct udev_device *udevice,int flag,struct path **pp_ptr);
libmultipath/structs_vec.c +3 -0
@@ -459,6 +459,9 @@ add_map_with_path (struct vectors * vecs, {struct multipath * mpp; +if (!strlen (pp->wwid)) +return NULL ; +if (!(mpp =alloc_multipath ()))return NULL ;
multipathd/main.c +43 -24
94 lines not on this page · raw message
Signature · 1 line
1.8.4.5
PATCH 52/78 multipathd: Issue warning on CLI command timeout
In reply to Hannes ReineckeSome CLI commands like 'reconfigure' could take longer than the socket timeout, causing the CLI to report 'timeout receiving packet'. In these cases the multipath daemon should issue a warning with the actual time spent in the CLI call to give the administator some hints on the best socket timeout.
--- multipathd/main.c | 4 ++++ multipathd/uxlsnr.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 52 insertions(+), 4 deletions(-)
multipathd/main.c +4 -0
@@ -1522,6 +1522,8 @@ reconfigure (struct vectors * vecs)struct config * old = conf;int retval =1 ; + running_state =DAEMON_CONFIGURE ; +/* * free old map and path vectors ... they use old conf state */
@@ -1546,6 +1548,8 @@ reconfigure (struct vectors * vecs) retval =0 ; } + running_state =DAEMON_RUNNING ; +return retval; }
multipathd/uxlsnr.c +48 -4
101 lines not on this page · raw message
Signature · 1 line
1.8.4.5
PATCH 53/78 Use strlen() when checking for valid wwid
In reply to Hannes ReineckeThe wwid is required to be a string, so we should be using strlen() when checking for a valid WWID.
--- libmultipath/configure.c | 5 +---- multipathd/main.c | 3 +-- 2 files changed, 2 insertions(+), 6 deletions(-)
libmultipath/configure.c +1 -4
@@ -734,7 +734,6 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_r {int r =1 ;int k, i; -char empty_buff[WWID_SIZE ];char params[PARAMS_SIZE ];struct multipath * mpp;struct path * pp1;
@@ -742,8 +741,6 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_rvector curmp = vecs->mpvec;vector pathvec = vecs->pathvec; -memset (empty_buff,0 ,WWID_SIZE ); -/* ignore refwwid if it's empty */ if (refwwid && !strlen (refwwid)) refwwid =NULL ;
@@ -757,7 +754,7 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_r/* skip this path for some reason */ /* 1. if path has no unique id or wwid blacklisted */ -if (memcmp (empty_buff, pp1->wwid,WWID_SIZE ) ==0 || +if (strlen (pp1->wwid) ==0 ||filter_path (conf, pp1) >0 ) {orphan_path (pp1,"wwid blacklisted" );continue ;
multipathd/main.c +1 -2
@@ -482,7 +482,6 @@ int ev_add_path (struct path * pp, struct vectors * vecs) { struct multipath * mpp; -char empty_buff[WWID_SIZE ] = {0 };char params[PARAMS_SIZE ] = {0 };int retries =3 ;int start_waiter =0 ;
@@ -491,7 +490,7 @@ ev_add_path (struct path * pp, struct vectors * vecs)/* * need path UID to go any further */ -if (memcmp (empty_buff, pp->wwid,WWID_SIZE ) ==0 ) { +if (strlen (pp->wwid) ==0 ) {condlog (0 ,"%s: failed to get path uid" , pp->dev); goto fail;/* leave path added to pathvec */ }
Signature · 1 line
1.8.4.5
PATCH 54/78 multipathd: Use standard lists for CLI handling
In reply to Hannes ReineckeWe already have standard list handling macros in list.h, so we should be using them for CLI, too.
--- libmultipath/list.h | 2 ++ libmultipath/uxsock.c | 2 +- multipathd/uxlsnr.c | 40 ++++++++++++++++++++++++++-------------- 3 files changed, 29 insertions(+), 15 deletions(-)
libmultipath/list.h +2 -0
@@ -8,6 +8,8 @@#ifndef _LIST_H#define _LIST_H +#include <stddef.h> +/** * container_of - cast a member of a structure out to the containing structure *
libmultipath/uxsock.c +1 -1
@@ -210,7 +210,7 @@ int recv_packet(int fd, char **buf, size_t *len, unsigned int timeout) } (*buf) =MALLOC (*len);if (!*buf) -return -1 ; +return -ENOMEM ; ret =read_all (fd, *buf, *len, timeout);if (ret != *len) {FREE (*buf);
multipathd/uxlsnr.c +26 -14
@@ -39,11 +39,11 @@struct timespec sleep_time = {5 ,0 };struct client { +struct list_head node;int fd; -struct client *next, *prev; }; -static struct client *clients; +LIST_HEAD (clients);static unsigned num_clients;struct pollfd *polls;volatile sig_atomic_t reconfig_sig =0 ;
@@ -67,10 +67,10 @@ static void new_client(int ux_sock)/* put it in our linked list */ c = (struct client *)MALLOC (sizeof (*c));memset (c,0 ,sizeof (*c)); +INIT_LIST_HEAD (&c->node); c->fd = fd; - c->next = clients; -if (c->next) c->next->prev = c; - clients = c; + +list_add_tail (&c->node, &clients); num_clients++; }
@@ -80,9 +80,8 @@ static void new_client(int ux_sock)static void dead_client (struct client *c) {close (c->fd); -if (c->prev) c->prev->next = c->next; -if (c->next) c->next->prev = c->prev; -if (c == clients) clients = c->next; + c->fd =-1 ; +list_del_init (&c->node);FREE (c); num_clients--; }
@@ -151,7 +150,7 @@ void * uxsock_listen(int (*uxsock_trigger)(char *, char **, int *, void *),sigdelset (&mask,SIGHUP );sigdelset (&mask,SIGUSR1 );while (1 ) { -struct client *c; +struct client *c, *tmp;int i, poll_count;/*
@@ -168,9 +167,13 @@ void * uxsock_listen(int (*uxsock_trigger)(char *, char **, int *, void *), polls[0 ].events =POLLIN ;/* setup the clients */ -for (i=1 , c = clients; c; i++, c = c->next) { + i =1 ; +list_for_each_entry (c, &clients, node) { polls[i].fd = c->fd; polls[i].events =POLLIN ; +if (i > num_clients) +break ; + i++; }/* most of our life is spent in this call */
@@ -191,12 +194,22 @@ void * uxsock_listen(int (*uxsock_trigger)(char *, char **, int *, void *),continue ;/* see if a client wants to speak to us */ -for (i=1 , c = clients; c; i++) { -struct client *next = c->next; - +for (i =1 ; i < num_clients +1 ; i++) {if (polls[i].revents &POLLIN ) {struct timeval start_time; + c =NULL ; +list_for_each_entry (tmp, &clients, node) { +if (tmp->fd == polls[i].fd) { + c = tmp; +break ; + } + } +if (!c) { +condlog (3 ,"cli%d: invalid fd %d" , + i, polls[i].fd); +continue ; + }if (gettimeofday (&start_time,NULL ) !=0 ) start_time.tv_sec =0 ;
@@ -223,7 +236,6 @@ void * uxsock_listen(int (*uxsock_trigger)(char *, char **, int *, void *),FREE (inbuf); } } - c = next; }/* see if we got a new client */
Signature · 1 line
1.8.4.5
PATCH 55/78 uxlsnr: use typedef for trigger function
In reply to Hannes Reinecke--- multipathd/uxlsnr.c | 3 +-- multipathd/uxlsnr.h | 8 +++++--- 2 files changed, 6 insertions(+), 5 deletions(-)
multipathd/uxlsnr.c +1 -2
@@ -119,8 +119,7 @@ void uxsock_cleanup(void *arg)/* * entry point */ -void *uxsock_listen (int (*uxsock_trigger)(char *,char **,int *,void *), -void * trigger_data) +void *uxsock_listen (uxsock_trigger_fn uxsock_trigger,void * trigger_data) {int ux_sock;size_t len;
multipathd/uxlsnr.h +5 -3
@@ -1,11 +1,13 @@#ifndef _UXLSNR_H#define _UXLSNR_H -void *uxsock_listen (int (*uxsock_trigger) - (char *,char **,int *,void *), -void * trigger_data); +typedef int (uxsock_trigger_fn )(char *,char **,int *,void *); + +void *uxsock_listen (uxsock_trigger_fn uxsock_trigger, +void * trigger_data);extern volatile sig_atomic_t reconfig_sig;extern volatile sig_atomic_t log_reset_sig; +#endif
Signature · 1 line
1.8.4.5
PATCH 56/78 multipathd: lock cli client list
In reply to Hannes Reinecke--- multipathd/uxlsnr.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-)
multipathd/uxlsnr.c +16 -8
@@ -44,7 +44,7 @@ struct client { };LIST_HEAD (clients); -static unsigned num_clients; +pthread_mutex_t client_lock =PTHREAD_MUTEX_INITIALIZER ;struct pollfd *polls;volatile sig_atomic_t reconfig_sig =0 ;volatile sig_atomic_t log_reset_sig =0 ;
@@ -64,14 +64,15 @@ static void new_client(int ux_sock)if (fd ==-1 )return ; -/* put it in our linked list */ c = (struct client *)MALLOC (sizeof (*c));memset (c,0 ,sizeof (*c));INIT_LIST_HEAD (&c->node); c->fd = fd; +/* put it in our linked list */ +pthread_mutex_lock (&client_lock);list_add_tail (&c->node, &clients); - num_clients++; +pthread_mutex_unlock (&client_lock); }/*
@@ -79,11 +80,12 @@ static void new_client(int ux_sock)*/ static void dead_client (struct client *c) { +pthread_mutex_lock (&client_lock); +list_del_init (&c->node); +pthread_mutex_unlock (&client_lock);close (c->fd); c->fd =-1 ; -list_del_init (&c->node);FREE (c); - num_clients--; }void free_polls (void )
@@ -150,7 +152,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)sigdelset (&mask,SIGUSR1 );while (1 ) {struct client *c, *tmp; -int i, poll_count; +int i, poll_count, num_clients;/* * Store configuration timeout;
@@ -161,6 +163,11 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data) timeout = conf->uxsock_timeout;/* setup for a poll */ +pthread_mutex_lock (&client_lock); + num_clients =0 ; +list_for_each_entry (c, &clients, node) { + num_clients++; + } polls =REALLOC (polls, (1 +num_clients) *sizeof (*polls)); polls[0 ].fd = ux_sock; polls[0 ].events =POLLIN ;
@@ -170,10 +177,9 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)list_for_each_entry (c, &clients, node) { polls[i].fd = c->fd; polls[i].events =POLLIN ; -if (i > num_clients) -break ; i++; } +pthread_mutex_unlock (&client_lock);/* most of our life is spent in this call */ poll_count =ppoll (polls, i, &sleep_time, &mask);
@@ -198,12 +204,14 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)struct timeval start_time; c =NULL ; +pthread_mutex_lock (&client_lock);list_for_each_entry (tmp, &clients, node) {if (tmp->fd == polls[i].fd) { c = tmp;break ; } } +pthread_mutex_unlock (&client_lock);if (!c) {condlog (3 ,"cli%d: invalid fd %d" , i, polls[i].fd);
Signature · 1 line
1.8.4.5
PATCH 57/78 multipath: enable sync support
In reply to Hannes Reineckemultipath didn't enable 'sync support' for device-mapper, causing device-mapper to create the device nodes internally rather than relying on udev here.
--- multipath/main.c | 1 + 1 file changed, 1 insertion(+)
multipath/main.c +1 -0
@@ -582,6 +582,7 @@ main (int argc, char *argv[])if (dm_prereq ())exit (1 );dm_drv_version (conf->version,TGT_MPATH ); +udev_set_sync_support (1 );if (optind < argc) { conf->dev =MALLOC (FILE_NAME_SIZE );
Signature · 1 line
1.8.4.5
PATCH 58/78 Remove dm_udev_XXX wrapper functions
In reply to Hannes ReineckeAs the definitions are encapsulated with the correct definitions anyway there is no need to have private wrappers here.
--- libmultipath/devmapper.c | 19 +++++-------------- libmultipath/devmapper.h | 2 -- multipath/main.c | 2 +- multipathd/main.c | 2 +- 4 files changed, 7 insertions(+), 18 deletions(-)
libmultipath/devmapper.c +5 -14
@@ -42,23 +42,14 @@ static inline int dm_task_set_cookie(struct dm_task *dmt, uint32_t *c, int a)return 1 ; } -void udev_wait (unsigned int c) +void dm_udev_wait (unsigned int c) { } -void udev_set_sync_support (int c) +void dm_udev_set_sync_support (int c) { } -#else -void udev_wait (unsigned int c) -{ -dm_udev_wait (c); -} -void udev_set_sync_support (int c) -{ -dm_udev_set_sync_support (c); -}#endif static void
@@ -245,7 +236,7 @@ dm_simplecmd (int task, const char *name, int no_flush, int need_sync, uint16_tif (!r)dm_udev_complete (cookie);else -udev_wait (cookie); +dm_udev_wait (cookie); } out:dm_task_destroy (dmt);
@@ -325,7 +316,7 @@ dm_addmap (int task, const char *target, struct multipath *mpp, char * params,if (!r)dm_udev_complete (cookie);else -udev_wait (cookie); +dm_udev_wait (cookie); } freeout:if (prefixed_uuid)
@@ -1354,7 +1345,7 @@ dm_rename (char * old, char * new)if (!r)dm_udev_complete (cookie);else -udev_wait (cookie); +dm_udev_wait (cookie); out:dm_task_destroy (dmt);
libmultipath/devmapper.h +0 -2
@@ -49,8 +49,6 @@ int dm_rename (char * old, char * new);int dm_reassign (const char * mapname);int dm_reassign_table (const char *name,char *old,char *new);int dm_setgeometry (struct multipath *mpp); -void udev_wait (unsigned int c); -void udev_set_sync_support (int c);#define VERSION_GE (v, minv) ( \ (v[0] > minv[0]) || \
multipath/main.c +1 -1
@@ -582,7 +582,7 @@ main (int argc, char *argv[])if (dm_prereq ())exit (1 );dm_drv_version (conf->version,TGT_MPATH ); -udev_set_sync_support (1 ); +dm_udev_set_sync_support (1 );if (optind < argc) { conf->dev =MALLOC (FILE_NAME_SIZE );
multipathd/main.c +1 -1
@@ -1817,7 +1817,7 @@ child (void * param)set_oom_adj (); conf->daemon =1 ; -udev_set_sync_support (0 ); +dm_udev_set_sync_support (0 );#ifdef USE_SYSTEMD envp =getenv ("WATCHDOG_USEC" );if (envp &&sscanf (envp,"%lu" , &checkint) ==1 ) {
Signature · 1 line
1.8.4.5
PATCH 59/78 Revert to ACT_RELOAD in domap if the map exists
In reply to Hannes ReineckeACT_CREATE checks for an existing map already, so we might as well revert to ACT_RELOAD in these cases and avoid an error during configuration.
--- libmultipath/configure.c | 16 ++++++++-------- libmultipath/devmapper.c | 5 ++++- 2 files changed, 12 insertions(+), 9 deletions(-)
libmultipath/configure.c +8 -8
@@ -577,7 +577,7 @@ fail:extern int domap (struct multipath * mpp,char * params) { -int r =0 ; +int r =DOMAP_FAIL ;/* * last chance to quit before touching the devmaps
@@ -587,6 +587,12 @@ domap (struct multipath * mpp, char * params)return DOMAP_DRY ; } +if (mpp->action ==ACT_CREATE && +dm_map_present (mpp->alias)) { +condlog (3 ,"%s: map already present" , mpp->alias); + mpp->action =ACT_RELOAD ; + } +switch (mpp->action) {case ACT_REJECT :case ACT_NOTHING :
@@ -609,12 +615,6 @@ domap (struct multipath * mpp, char * params)return DOMAP_RETRY ; } -if (dm_map_present (mpp->alias)) { -condlog (3 ,"%s: map already present" , mpp->alias); -lock_multipath (mpp,0 ); -break ; - } - r =dm_addmap_create (mpp, params);lock_multipath (mpp,0 );
@@ -650,7 +650,7 @@ domap (struct multipath * mpp, char * params)break ; } -if (r) { +if (r ==DOMAP_OK ) {/* * DM_DEVICE_CREATE, DM_DEVICE_RENAME, or DM_DEVICE_RELOAD * succeeded
libmultipath/devmapper.c +4 -1
@@ -347,8 +347,11 @@ dm_addmap_create (struct multipath *mpp, char * params) {condlog (3 ,"%s: failed to load map (a path might be in use)" , mpp->alias);dm_flush_map_nosync (mpp->alias); } -if (err !=EROFS ) +if (err !=EROFS ) { +condlog (3 ,"%s: failed to load map, error %d" , + mpp->alias, err);break ; + } }return 0 ; }
Signature · 1 line
1.8.4.5
PATCH 60/78 multipathd: use local variable for watchdog configuration
In reply to Hannes ReineckeThe 'watchdog' setting in struct config needs to be a local variable in multipathd/main.c, as the config will be freed during reconfiguration and the checkerloop might then access an invalid structure.
--- libmultipath/config.h | 1 - multipathd/main.c | 8 ++++++-- 2 files changed, 6 insertions(+), 3 deletions(-)
libmultipath/config.h +0 -1
@@ -116,7 +116,6 @@ struct config {int ignore_wwids;int checker_timeout;int daemon; -int watchdog;int flush_on_last_del;int attribute_flags;int fast_io_fail;
multipathd/main.c +6 -2
@@ -29,6 +29,10 @@*/ #include <checkers.h> +#ifdef USE_SYSTEMD +static int use_watchdog; +#endif +/* * libmultipath */
@@ -1400,7 +1404,7 @@ checkerloop (void *ap)pthread_testcancel ();condlog (4 ,"tick" );#ifdef USE_SYSTEMD -if (conf->watchdog) +if (use_watchdog)sd_notify (0 ,"WATCHDOG=1" );#endif if (vecs->pathvec) {
@@ -1830,7 +1834,7 @@ child (void * param) conf->checkint = conf->max_checkint /4 ;condlog (3 ,"enabling watchdog, interval %d max %d" , conf->checkint, conf->max_checkint); - conf->watchdog = conf->checkint; + use_watchdog = conf->checkint; }#endif /*
Signature · 1 line
1.8.4.5
PATCH 61/78 Ignore devices when sysfs_get_tgt_nodename fails
In reply to Hannes ReineckeIf sysfs_get_tgt_nodename fails we should not try to use this device.
--- libmultipath/discovery.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
libmultipath/discovery.c +7 -5
@@ -349,8 +349,9 @@ sysfs_get_tgt_nodename (struct path *pp, char * node)snprintf (node,NODE_NAME_SIZE ,"ata-%d.00" , tgtid);return 0 ; } +/* Unknown SCSI transport. Keep fingers crossed */ pp->sg_id.proto_id =SCSI_PROTOCOL_UNSPEC ; -return 1 ; +return 0 ; }int sysfs_get_host_adapter_name (struct path *pp,char *adapter_name)
@@ -807,10 +808,11 @@ scsi_sysfs_pathinfo (struct path * pp)/* * target node name */ -if (!sysfs_get_tgt_nodename (pp, pp->tgt_node_name)) { -condlog (3 ,"%s: tgt_node_name = %s" , - pp->dev, pp->tgt_node_name); - } +if (sysfs_get_tgt_nodename (pp, pp->tgt_node_name)) +return 1 ; + +condlog (3 ,"%s: tgt_node_name = %s" , + pp->dev, pp->tgt_node_name);return 0 ; }
Signature · 1 line
1.8.4.5
PATCH 62/78 Skip USB devices during discovery
In reply to Hannes ReineckeSome USB devices even present VPD pages, but they still remain USB devices, and we shouldn't run multipath on those. So detect USB devices properly and skip them during discovery.
--- libmultipath/discovery.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
libmultipath/discovery.c +13 -0
@@ -275,6 +275,19 @@ sysfs_get_tgt_nodename (struct path *pp, char * node) } } +/* Check for USB */ + tgtdev =udev_device_get_parent (parent); +while (tgtdev) { + value =udev_device_get_subsystem (tgtdev); +if (value && !strcmp (value,"usb" )) { + pp->sg_id.proto_id =SCSI_PROTOCOL_UNSPEC ; + tgtname =udev_device_get_sysname (tgtdev); +strncpy (node, tgtname,strlen (tgtname)); +condlog (3 ,"%s: skip USB device %s" , pp->dev, node); +return 1 ; + } + tgtdev =udev_device_get_parent (tgtdev); + } parent =udev_device_get_parent_with_subsystem_devtype (pp->udev,"scsi" ,"scsi_target" );if (!parent)return 1 ;
Signature · 1 line
1.8.4.5
PATCH 63/78 Read wwid from sysfs vpg_pg83 attribute
In reply to Hannes ReineckeUsing 'uid_attribute' per default has the problem that udev might not be able to retrieve the device ID in time as the device might be (temporarily) blocked.
It also has the problem that the 'ID_SERIAL' attribute is not well defined and might have been overridden by other udev rules.
As recent kernels have a 'vpd_pg83' sysfs attribute multipath should be reading this one directly and extract the uid from there. With that multipath does not need to do any I/O to generate the device wwid, eliminating one common error cause during failover.
--- libmultipath/discovery.c | 305 ++++++++++++++++++++++++++++++++++++++++++----- libmultipath/sysfs.c | 51 ++++++++ libmultipath/sysfs.h | 2 + 3 files changed, 331 insertions(+), 27 deletions(-)
libmultipath/discovery.c +278 -27
@@ -207,6 +207,30 @@ declare_sysfs_get_str(vendor);declare_sysfs_get_str (model);declare_sysfs_get_str (rev); +ssize_t +sysfs_get_vpd (struct udev_device * udev,int pg, +unsigned char * buff,size_t len) +{ +ssize_t attr_len; +char attrname[9 ]; +const char * devname; + +if (!udev) { +condlog (3 ,"No udev device given\n" ); +return -ENOSYS ; + } + + devname =udev_device_get_sysname (udev); +sprintf (attrname,"vpd_pg%02x" , pg); + attr_len =sysfs_bin_attr_get_value (udev, attrname, buff, len); +if (attr_len <0 ) { +condlog (3 ,"%s: attribute %s not found in sysfs" , + devname, attrname); +return attr_len; + } +return attr_len; +} +int sysfs_get_timeout (struct path *pp,unsigned int *timeout) {
@@ -764,6 +788,183 @@ get_geometry(struct path *pp) }static int +get_vpd (struct udev_device *parent,int pg,char * str,int maxlen) +{ +int len = -ENODATA , buff_len; +unsigned char buff[4096 ]; + +memset (buff,0x0 ,4096 ); +if (sysfs_get_vpd (parent, pg, buff,4096 ) <=0 ) { +condlog (3 ,"failed to get vpd pg%02x" , pg); +return -EIO ; + } + +if (buff[1 ] != pg) { +condlog (3 ,"vpd pg%02x error, invalid vpd page %02x" , + pg, buff[1 ]); +return -ENODATA ; + } + buff_len = (buff[2 ] <<8 ) + buff[3 ] +4 ; +if (buff_len >4096 ) +condlog (3 ,"vpd pg%02x page truncated" , pg); + +if (pg ==0x80 ) { +char *p =NULL ; + len = buff[3 ] + (buff[2 ] <<8 ); +if (len >= maxlen) { +condlog (3 ,"vpd pg%02x overflow, %d/%d bytes required" , + pg, len, maxlen); +return -EINVAL ; + } +if (len >0 ) { +memcpy (str, buff +4 , len); + str[len] ='\0' ; + } + p = str + len -1 ; +while (p > str && *p ==' ' ) { + *p ='\0' ; + p--; + len --; + } + }else if (pg ==0x83 ) { +unsigned char *d; +unsigned char *vpd =NULL ; +int vpd_type, vpd_len, prio =-1 , i; + + d = (unsigned char *)buff +4 ; +while (d < (unsigned char *)buff + buff_len) { +/* Select 'association: LUN' */ +if ((d[1 ] &0x30 ) !=0 ) { + d += d[3 ] +4 ; +continue ; + } +switch (d[1 ] &0xf ) { +case 0x3 : +/* NAA: Prio 5 */ +if (prio <5 ) { + prio =5 ; + vpd = d; + } +break ; +case 0x8 : +/* SCSI Name: Prio 4 */ +if (memcmp (d +4 ,"eui." ,4 ) && +memcmp (d +4 ,"naa." ,4 ) && +memcmp (d +4 ,"iqn." ,4 )) +continue ; +if (prio <4 ) { + prio =4 ; + vpd = d; + } +break ; +case 0x2 : +/* EUI-64: Prio 3 */ +if (prio <3 ) { + prio =3 ; + vpd = d; + } +break ; +case 0x1 : +/* T-10 Vendor ID: Prio 2 */ +if (prio <2 ) { + prio =2 ; + vpd = d; + } +break ; + } + d += d[3 ] +4 ; + } +if (prio >0 ) { + vpd_type = vpd[1 ] &0xf ; + vpd_len = vpd[3 ]; + vpd +=4 ; +if (vpd_type ==0x2 || vpd_type ==0x3 ) { +int i; + + len =sprintf (str,"%d" , vpd_type); +for (i =0 ; i < vpd_len; i++) { + len +=sprintf (str + len, +"%02x" , vpd[i]); +if (len >= maxlen) +break ; + } + }else if (vpd_type ==0x8 ) { +if (!memcmp ("eui." , vpd,4 )) { + str[0 ] ='2' ; + len =1 ; + vpd +=4 ; + vpd_len -=4 ; +for (i =0 ; i < vpd_len; i++) { + len +=sprintf (str + len,"%c" , +tolower (vpd[i])); +if (len >= maxlen) +break ; + } + len = vpd_len +1 ; + str[len] ='\0' ; + }else if (!memcmp ("naa." , vpd,4 )) { + str[0 ] ='3' ; + len =1 ; + vpd +=4 ; + vpd_len -=4 ; +for (i =0 ; i < vpd_len; i++) { + len +=sprintf (str + len,"%c" , +tolower (vpd[i])); +if (len >= maxlen) +break ; + } + len = vpd_len +1 ; + str[len] ='\0' ; + }else { + str[0 ] ='8' ; + len =1 ; + vpd +=4 ; + vpd_len -=4 ; +if (vpd_len > maxlen +2 ) + vpd_len = maxlen -2 ; +memcpy (str, vpd, vpd_len); + len = vpd_len +1 ; + str[len] ='\0' ; + } + }else if (vpd_type ==0x1 ) { +unsigned char *p; +int p_len; + + str[0 ] ='1' ; + len =1 ; + p = vpd; +while ((p =memchr (vpd,' ' , vpd_len))) { + p_len = p - vpd; +if (len + p_len > maxlen -1 ) + p_len = maxlen - len -2 ; +memcpy (str + len, vpd, p_len); + len += p_len; +if (len >= maxlen -1 ) { + str[len] ='\0' ; +break ; + } + str[len] ='_' ; + len ++; + vpd = p; + vpd_len -= p_len; +while (vpd && *vpd ==' ' ) { + vpd++; + vpd_len --; + } + } +if (len >1 && str[len -1 ] =='_' ) { + str[len -1 ] ='\0' ; + len--; + } + } + } + }else + len = -ENOSYS ; + +return len; +} + +static int scsi_sysfs_pathinfo (struct path * pp) {struct udev_device *parent;
@@ -803,6 +1004,9 @@ scsi_sysfs_pathinfo (struct path * pp)condlog (3 ,"%s: rev = %s" , pp->dev, pp->rev); +if (get_vpd (parent,0x80 , pp->serial,SERIAL_SIZE ) >=0 ) +condlog (3 ,"%s: serial = %s" , pp->dev, pp->serial); +/* * set the hwe configlet pointer */
@@ -1051,7 +1255,8 @@ static int scsi_ioctl_pathinfo (struct path * pp, int mask) {if (mask &DI_SERIAL ) { -get_serial (pp->serial,SERIAL_SIZE , pp->fd); +if (strlen (pp->serial) ==0 ) +get_serial (pp->serial,SERIAL_SIZE , pp->fd);condlog (3 ,"%s: serial = %s" , pp->dev, pp->serial); }
@@ -1144,10 +1349,56 @@ get_prio (struct path * pp) }static int +get_udev_uid (struct path * pp,char *uid_attribute) +{ +ssize_t len; +const char *value; + + value =udev_device_get_property_value (pp->udev, + uid_attribute); +if ((!value ||strlen (value) ==0 ) && conf->cmd ==CMD_VALID_PATH ) + value =getenv (uid_attribute); +if (value &&strlen (value)) { +if (strlen (value) +1 >WWID_SIZE ) { +condlog (0 ,"%s: wwid overflow" , pp->dev); + len =WWID_SIZE ; + }else { + len =strlen (value); + } +strncpy (pp->wwid, value, len); + }else { +condlog (3 ,"%s: no %s attribute" , pp->dev, + uid_attribute); + len = -EINVAL ; + } +return len; +} + +static int +get_vpd_uid (struct path * pp) +{ +struct udev_device *parent = pp->udev; + +while (parent) { +const char *subsys =udev_device_get_subsystem (parent); +if (subsys && !strncmp (subsys,"scsi" ,4 )) +break ; + parent =udev_device_get_parent (parent); + } + +if (!parent) { +condlog (3 ,"%s: no scsi device found in sysfs" , pp->dev); +return -ENXIO ; + } +return get_vpd (parent,0x83 , pp->wwid,WWID_SIZE ); +} + +static int get_uid (struct path * pp) {char *c; -const char *origin; +const char *origin ="none" ; +ssize_t len =0 ;if (!pp->uid_attribute && !pp->getuid)select_getuid (pp);
@@ -1166,40 +1417,40 @@ get_uid (struct path * pp)if (apply_format (pp->getuid, &buff[0 ], pp)) {condlog (0 ,"error formatting uid callout command" );memset (pp->wwid,0 ,WWID_SIZE ); + len = -EINVAL ; }else if (execute_program (buff, pp->wwid,WWID_SIZE )) {condlog (3 ,"error calling out %s" , buff);memset (pp->wwid,0 ,WWID_SIZE ); - } + len = -EIO ; + }else + len =strlen (pp->wwid); origin ="callout" ; }else { -const char *value; - - value =udev_device_get_property_value (pp->udev, - pp->uid_attribute); -if ((!value ||strlen (value) ==0 ) && - conf->cmd ==CMD_VALID_PATH ) - value =getenv (pp->uid_attribute); -if (value &&strlen (value)) { -size_t len =WWID_SIZE ; - -if (strlen (value) +1 >WWID_SIZE ) { -condlog (0 ,"%s: wwid overflow" , pp->dev); - }else { - len =strlen (value); - } -strncpy (pp->wwid, value, len); +if (pp->uid_attribute) { + len =get_udev_uid (pp, pp->uid_attribute); + origin ="udev" ; }else { -condlog (3 ,"%s: no %s attribute" , pp->dev, - pp->uid_attribute); + len =get_vpd_uid (pp); +if (len >0 ) + origin ="sysfs" ; +else { + len =get_udev_uid (pp,DEFAULT_UID_ATTRIBUTE ); + origin ="udev" ; + } } - origin ="udev" ; } -/* Strip any trailing blanks */ - c =strchr (pp->wwid,'\0' ); - c--; -while (c && c >= pp->wwid && *c ==' ' ) { - *c ='\0' ; +if ( len <0 ) { +condlog (1 ,"%s: failed to get uid: %s" , + pp->dev,strerror (-len)); +memset (pp->wwid,0x0 ,WWID_SIZE ); + }else { +/* Strip any trailing blanks */ + c =strchr (pp->wwid,'\0' ); c--; +while (c && c >= pp->wwid && *c ==' ' ) { + *c ='\0' ; + c--; + } }condlog (3 ,"%s: uid = %s (%s)" , pp->dev, *pp->wwid =='\0' ?"<empty>" : pp->wwid, origin);
libmultipath/sysfs.c +51 -0
57 lines not on this page · raw message
libmultipath/sysfs.h +2 -0
@@ -9,6 +9,8 @@ ssize_t sysfs_attr_set_value(struct udev_device *dev, const char *attr_name, char * value, size_t value_len);ssize_t sysfs_attr_get_value (struct udev_device *dev,const char *attr_name,char * value,size_t value_len); +ssize_t sysfs_bin_attr_get_value (struct udev_device *dev,const char *attr_name, +unsigned char * value,size_t value_len);int sysfs_get_size (struct path *pp,unsigned long long * size);int sysfs_check_holders (char * check_devt,char * new_devt);#endif
Signature · 1 line
1.8.4.5
PATCH 64/78 Assign local priority for NAA VPD descriptor
In reply to Hannes ReineckeSome devices assign several NAA VPD descriptors. So implement an internal priority to select the 'best' NAA descriptor.
--- libmultipath/discovery.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-)
libmultipath/discovery.c +25 -3
@@ -829,7 +829,7 @@ get_vpd (struct udev_device *parent, int pg, char * str, int maxlen) }else if (pg ==0x83 ) {unsigned char *d;unsigned char *vpd =NULL ; -int vpd_type, vpd_len, prio =-1 , i; +int vpd_type, vpd_len, prio =-1 , i, naa_prio; d = (unsigned char *)buff +4 ;while (d < (unsigned char *)buff + buff_len) {
@@ -841,8 +841,30 @@ get_vpd (struct udev_device *parent, int pg, char * str, int maxlen)switch (d[1 ] &0xf ) {case 0x3 :/* NAA: Prio 5 */ -if (prio <5 ) { - prio =5 ; +switch (d[4 ] >>4 ) { +case 6 : +/* IEEE Registered Extended: Prio 8 */ + naa_prio =8 ; +break ; +case 5 : +/* IEEE Registered: Prio 7 */ + naa_prio =7 ; +break ; +case 2 : +/* IEEE Extended: Prio 6 */ + naa_prio =6 ; +break ; +case 3 : +/* IEEE Locally assigned: Prio 1 */ + naa_prio =1 ; +break ; +default : +/* Default: no priority */ + naa_prio =-1 ; +break ; + } +if (prio < naa_prio) { + prio = naa_prio; vpd = d; }break ;
Signature · 1 line
1.8.4.5
PATCH 65/78 Use sysfs attribute vpd_pg80 to read serial number
In reply to Hannes ReineckeThere is no need to issue an ioctl to read the serial number as it's already present in sysfs. And we should not print out the serial number unconditionally but rather check the discovery mask.
--- libmultipath/discovery.c | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-)
libmultipath/discovery.c +28 -8
@@ -1026,9 +1026,6 @@ scsi_sysfs_pathinfo (struct path * pp)condlog (3 ,"%s: rev = %s" , pp->dev, pp->rev); -if (get_vpd (parent,0x80 , pp->serial,SERIAL_SIZE ) >=0 ) -condlog (3 ,"%s: serial = %s" , pp->dev, pp->serial); -/* * set the hwe configlet pointer */
@@ -1276,13 +1273,36 @@ sysfs_pathinfo(struct path * pp)static int scsi_ioctl_pathinfo (struct path * pp,int mask) { -if (mask &DI_SERIAL ) { -if (strlen (pp->serial) ==0 ) -get_serial (pp->serial,SERIAL_SIZE , pp->fd); -condlog (3 ,"%s: serial = %s" , pp->dev, pp->serial); +struct udev_device *parent; +const char *attr_path =NULL ; + +if (!(mask &DI_SERIAL )) +return 0 ; + + parent = pp->udev; +while (parent) { +const char *subsys =udev_device_get_subsystem (parent); +if (subsys && !strncmp (subsys,"scsi" ,4 )) { + attr_path =udev_device_get_sysname (parent); +if (!attr_path) +break ; +if (sscanf (attr_path,"%i:%i:%i:%i" , + &pp->sg_id.host_no, + &pp->sg_id.channel, + &pp->sg_id.scsi_id, + &pp->sg_id.lun) ==4 ) +break ; + } + parent =udev_device_get_parent (parent); } +if (!attr_path || pp->sg_id.host_no ==-1 ) +return -ENODEV ; -return 0 ; +if (get_vpd (parent,0x80 , pp->serial,SERIAL_SIZE ) >0 ) +condlog (3 ,"%s: serial = %s" , + pp->dev, pp->serial); + +return strlen (pp->serial) ?0 : -EIO ; } static int
Signature · 1 line
1.8.4.5
PATCH 66/78 Update multipath.conf.5 to clarify wwid generation
In reply to Hannes ReineckeAdd a new section in multipath.conf.5 to clarify wwid generation and update wrong defaults for the property whitelist.
--- multipath.conf.defaults | 2 +- multipath/multipath.conf.5 | 33 ++++++++++++++++++++++++++++----- 2 files changed, 29 insertions(+), 6 deletions(-)
multipath.conf.defaults +1 -1
@@ -73,7 +73,7 @@ # } #} #blacklist_exceptions { -# property "(ID_SCSI_VPD|ID_WWN)" +# property "(SCSI_IDENT_.*|ID_WWN)" #} #devices { # device {
multipath/multipath.conf.5 +28 -5
@@ -55,9 +55,8 @@ section. .TP .B multipaths This section defines the multipath topologies. They are indexed by a -\fIWorld Wide Identifier\fR(wwid), which is taken to be the value of -the udev attribute given by the -\fIuid_attribute\fR keyword. +\fIWorld Wide Identifier\fR(wwid). For details on the wwid generation +see section \fBWWID generation\fR below. .TP .B devices This section defines the device-specific settings.
@@ -176,7 +175,7 @@ identifier. Default value is .B getuid_callout The default program and args to callout to obtain a unique path identifier. Should be specified with an absolute path. -This parameter is deprecated; \fIuid_attribute\fR should be used instead. +This parameter is deprecated. .TP .B prio The name of the path priority routine. The specified routine
@@ -534,7 +533,7 @@ The \fIWorld Wide Identification\fR of a device. .TP .B property Regular expression of the udev property to be whitelisted. Defaults to -.I (ID_WWN|ID_SCSI_VPD) +.I (ID_WWN|SCSI_IDENT_.*) .TP .B devnode Regular expression of the device nodes to be whitelisted.
@@ -771,6 +770,30 @@ sections: .RE .PD .LP +.SH "WWID generation" +Multipath uses a \fIWorld Wide Identification\fR (wwid) to determine +which paths belong to the same device. Each path presenting the same +wwid is assumed to point to the same device. +.LP +The wwid is generated by three methods (in the order of preference): +.TP 17 +.B getuid_callout +Use the specified external program; cf \fIgetuid_callout\fR above. +Care should be taken when using this method; the external program +needs to be loaded from disk for execution, which might lead to +deadlock situations in an all-paths-down scenario. +.TP +.B uid_attribute +Use the value of the specified udev attribute; cf \fIuid_attribute\fR +above. This method is preferred to \fIgetuid_callout\fR as multipath +does not need to call any external programs here. However, under +certain circumstances udev might not be able to generate the requested +variable. +.TP +.B vpd_pg83 +If none of the \fIgetuid_callout\fR or \fIuid_attribute\fR parameters +are present multipath will try to use the sysfs attribute +\fIvpd_pg83\fR to generate the wwid. .SH "KNOWN ISSUES" The usage of .B queue_if_no_path
Signature · 1 line
1.8.4.5
PATCH 67/78 libmultipath: Fall back to SG_IO if no UID could be assigned
In reply to Hannes ReineckeRelying on udev attributes or sysfs vpd pages might fail, in which case we're unable to assign an UID and multipath will fail to start up.
Implement a fallback to read VPD page 0x83 directly in these cases.
--- libmultipath/discovery.c | 53 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 39 insertions(+), 14 deletions(-)
libmultipath/discovery.c +39 -14
@@ -770,6 +770,29 @@ get_serial (char * str, int maxlen, int fd)return 1 ; } +#define DEFAULT_SGIO_LEN 254 + +static int +sgio_get_vpd (unsigned char * buff,int maxlen,int fd) +{ +int len =DEFAULT_SGIO_LEN ; + +if (fd <0 ) { + errno =EBADF ; +return -1 ; + } +retry: +if (0 ==do_inq (fd,0 ,1 ,0x83 , buff, len)) { + len = buff[3 ] + (buff[2 ] <<8 ); +if (len >= maxlen) +return len; +if (len >DEFAULT_SGIO_LEN ) + goto retry; +return 0 ; + } +return -1 ; +} +static int get_geometry (struct path *pp) {
@@ -788,15 +811,19 @@ get_geometry(struct path *pp) }static int -get_vpd (struct udev_device *parent,int pg,char * str,int maxlen) +get_vpd (struct udev_device *parent,int fd,int pg,char * str,int maxlen) {int len = -ENODATA , buff_len;unsigned char buff[4096 ];memset (buff,0x0 ,4096 ); -if (sysfs_get_vpd (parent, pg, buff,4096 ) <=0 ) { -condlog (3 ,"failed to get vpd pg%02x" , pg); -return -EIO ; +if (!parent ||sysfs_get_vpd (parent, pg, buff,4096 ) <=0 ) { +condlog (3 ,"failed to read sysfs vpd pg%02x" , pg); +if (sgio_get_vpd (buff,4096 , fd) <=0 ) { +condlog (3 ,"failed to issue vpd inquiry for pg%02x" , + pg); +return -errno; + } }if (buff[1 ] != pg) {
@@ -1298,7 +1325,7 @@ scsi_ioctl_pathinfo (struct path * pp, int mask)if (!attr_path || pp->sg_id.host_no ==-1 )return -ENODEV ; -if (get_vpd (parent,0x80 , pp->serial,SERIAL_SIZE ) >0 ) +if (get_vpd (parent, pp->fd,0x80 , pp->serial,SERIAL_SIZE ) >0 )condlog (3 ,"%s: serial = %s" , pp->dev, pp->serial);
@@ -1428,11 +1455,7 @@ get_vpd_uid(struct path * pp) parent =udev_device_get_parent (parent); } -if (!parent) { -condlog (3 ,"%s: no scsi device found in sysfs" , pp->dev); -return -ENXIO ; - } -return get_vpd (parent,0x83 , pp->wwid,WWID_SIZE ); +return get_vpd (parent, pp->fd,0x83 , pp->wwid,WWID_SIZE ); } static int
@@ -1471,14 +1494,16 @@ get_uid (struct path * pp)if (pp->uid_attribute) { len =get_udev_uid (pp, pp->uid_attribute); origin ="udev" ; - }else { + } +if (len <=0 ) { len =get_vpd_uid (pp);if (len >0 ) origin ="sysfs" ; -else { - len =get_udev_uid (pp,DEFAULT_UID_ATTRIBUTE ); + } +if (len <=0 ) { + len =get_udev_uid (pp,DEFAULT_UID_ATTRIBUTE ); +if (len >0 ) origin ="udev" ; - } } }if ( len <0 ) {
Signature · 1 line
1.8.4.5
PATCH 68/78 libmultipath: unset 'uid_attribute' on failure
In reply to Hannes ReineckeDue to a race condition within udev the 'uid_attribute' might not always be set. So we should be zeroing the 'uid_attribute' when retrieving the uid by other means, otherwise the discovery process will blacklist the device.
--- libmultipath/discovery.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
libmultipath/discovery.c +7 -3
@@ -1497,13 +1497,17 @@ get_uid (struct path * pp) }if (len <=0 ) { len =get_vpd_uid (pp); -if (len >0 ) +if (len >0 ) { origin ="sysfs" ; + pp->uid_attribute =NULL ; + } }if (len <=0 ) { len =get_udev_uid (pp,DEFAULT_UID_ATTRIBUTE ); -if (len >0 ) - origin ="udev" ; +if (len >0 ) { + origin ="default" ; + pp->uid_attribute =DEFAULT_UID_ATTRIBUTE ; + } } }if ( len <0 ) {
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 68/78] libmultipath: unset 'uid_attribute' on failure
Quoted 4 lines · Hannes Reinecke
Due to a race condition within udev the '
uid_attribute' might not always be set. So we should be zeroing the 'uid_attribute' when retrieving the uid by other means, otherwise the discovery process will blacklist the device.
Possibly I'm just missing obvious here, but I don't get the point of zeroing out the uid_attribute. Won't it just get reset on the next call to get_uid?
-Ben
Quoted 33 lines
Signed-off-by Hannes Reinecke <[email protected]>--- libmultipath/discovery.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)libmultipath/discovery.c +7
-3@@ -1497,13 +1497,17 @@ get_uid (struct path * pp) }if (len <=0 ) { len =get_vpd_uid (pp); -if (len >0 ) +if (len >0 ) { origin ="sysfs" ; + pp->uid_attribute =NULL ; + } }if (len <=0 ) { len =get_udev_uid (pp,DEFAULT_UID_ATTRIBUTE ); -if (len >0 ) - origin ="udev" ; +if (len >0 ) { + origin ="default" ; + pp->uid_attribute =DEFAULT_UID_ATTRIBUTE ; + } } }if ( len <0 ) {Signature · 1 line
1.8.4.5
Re: [PATCH 68/78] libmultipath: unset 'uid_attribute' on failure
Quoted 10 lines · Benjamin Marzinski
Quoted 4 lines · Hannes Reinecke
Due to a race condition within udev the '
uid_attribute' might not always be set. So we should be zeroing the 'uid_attribute' when retrieving the uid by other means, otherwise the discovery process will blacklist the device.Possibly I'm just missing obvious here, but I don't get the point of zeroing out the
uid_attribute. Won't it just get reset on the next call toget_uid?
The uid_attribute setting is evaluated later on, so when we're not zeroing it the code assumes we've got the UID from the udev attribute, which is untrue.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
PATCH 69/78 Separate out vpd parsing functions
In reply to Hannes Reinecke--- libmultipath/discovery.c | 367 +++++++++++++++++++++++++---------------------- 1 file changed, 193 insertions(+), 174 deletions(-)
libmultipath/discovery.c +193 -174
@@ -811,9 +811,196 @@ get_geometry(struct path *pp) }static int +parse_vpd_pg80 (const unsigned char *in,char *out,size_t out_len) +{ +char *p =NULL ; +int len = in[3 ] + (in[2 ] <<8 ); + +if (len >= out_len) { +condlog (2 ,"vpd pg80 overflow, %d/%d bytes required" , + len, (int )out_len); + len = out_len; + } +if (len >0 ) { +memcpy (out, in +4 , len); + out[len] ='\0' ; + } +/* +* Strip trailing whitspaces +*/ + p = out + len -1 ; +while (p > out && *p ==' ' ) { + *p ='\0' ; + p--; + len --; + } +return len; +} + +static int +parse_vpd_pg83 (const unsigned char *in,size_t in_len, +char *out,size_t out_len) +{ +unsigned char *d; +unsigned char *vpd =NULL ; +int len = -ENODATA , vpd_type, vpd_len, prio =-1 , i, naa_prio; + + d = (unsigned char *)in +4 ; +while (d < (unsigned char *)in + in_len) { +/* Select 'association: LUN' */ +if ((d[1 ] &0x30 ) !=0 ) { + d += d[3 ] +4 ; +continue ; + } +switch (d[1 ] &0xf ) { +case 0x3 : +/* NAA: Prio 5 */ +switch (d[4 ] >>4 ) { +case 6 : +/* IEEE Registered Extended: Prio 8 */ + naa_prio =8 ; +break ; +case 5 : +/* IEEE Registered: Prio 7 */ + naa_prio =7 ; +break ; +case 2 : +/* IEEE Extended: Prio 6 */ + naa_prio =6 ; +break ; +case 3 : +/* IEEE Locally assigned: Prio 1 */ + naa_prio =1 ; +break ; +default : +/* Default: no priority */ + naa_prio =-1 ; +break ; + } +if (prio < naa_prio) { + prio = naa_prio; + vpd = d; + } +break ; +case 0x8 : +/* SCSI Name: Prio 4 */ +if (memcmp (d +4 ,"eui." ,4 ) && +memcmp (d +4 ,"naa." ,4 ) && +memcmp (d +4 ,"iqn." ,4 )) +continue ; +if (prio <4 ) { + prio =4 ; + vpd = d; + } +break ; +case 0x2 : +/* EUI-64: Prio 3 */ +if (prio <3 ) { + prio =3 ; + vpd = d; + } +break ; +case 0x1 : +/* T-10 Vendor ID: Prio 2 */ +if (prio <2 ) { + prio =2 ; + vpd = d; + } +break ; + } + d += d[3 ] +4 ; + } +if (prio >0 ) { + vpd_type = vpd[1 ] &0xf ; + vpd_len = vpd[3 ]; + vpd +=4 ; +if (vpd_type ==0x2 || vpd_type ==0x3 ) { +int i; + + len =sprintf (out,"%d" , vpd_type); +for (i =0 ; i < vpd_len; i++) { + len +=sprintf (out + len, +"%02x" , vpd[i]); +if (len >= out_len) +break ; + } + }else if (vpd_type ==0x8 ) { +if (!memcmp ("eui." , vpd,4 )) { + out[0 ] ='2' ; + len =1 ; + vpd +=4 ; + vpd_len -=4 ; +for (i =0 ; i < vpd_len; i++) { + len +=sprintf (out + len,"%c" , +tolower (vpd[i])); +if (len >= out_len) +break ; + } + len = vpd_len +1 ; + out[len] ='\0' ; + }else if (!memcmp ("naa." , vpd,4 )) { + out[0 ] ='3' ; + len =1 ; + vpd +=4 ; + vpd_len -=4 ; +for (i =0 ; i < vpd_len; i++) { + len +=sprintf (out + len,"%c" , +tolower (vpd[i])); +if (len >= out_len) +break ; + } + len = vpd_len +1 ; + out[len] ='\0' ; + }else { + out[0 ] ='8' ; + len =1 ; + vpd +=4 ; + vpd_len -=4 ; +if (vpd_len > out_len +2 ) + vpd_len = out_len -2 ; +memcpy (out, vpd, vpd_len); + len = vpd_len +1 ; + out[len] ='\0' ; + } + }else if (vpd_type ==0x1 ) { +unsigned char *p; +int p_len; + + out[0 ] ='1' ; + len =1 ; + p = vpd; +while ((p =memchr (vpd,' ' , vpd_len))) { + p_len = p - vpd; +if (len + p_len > out_len -1 ) + p_len = out_len - len -2 ; +memcpy (out + len, vpd, p_len); + len += p_len; +if (len >= out_len -1 ) { + out[len] ='\0' ; +break ; + } + out[len] ='_' ; + len ++; + vpd = p; + vpd_len -= p_len; +while (vpd && *vpd ==' ' ) { + vpd++; + vpd_len --; + } + } +if (len >1 && out[len -1 ] =='_' ) { + out[len -1 ] ='\0' ; + len--; + } + } + } +return len; +} + +static int get_vpd (struct udev_device *parent,int fd,int pg,char * str,int maxlen) { -int len = -ENODATA , buff_len; +int len, buff_len;unsigned char buff[4096 ];memset (buff,0x0 ,4096 );
@@ -835,179 +1022,11 @@ get_vpd (struct udev_device *parent, int fd, int pg, char * str, int maxlen)if (buff_len >4096 )condlog (3 ,"vpd pg%02x page truncated" , pg); -if (pg ==0x80 ) { -char *p =NULL ; - len = buff[3 ] + (buff[2 ] <<8 ); -if (len >= maxlen) { -condlog (3 ,"vpd pg%02x overflow, %d/%d bytes required" , - pg, len, maxlen); -return -EINVAL ; - } -if (len >0 ) { -memcpy (str, buff +4 , len); - str[len] ='\0' ; - } - p = str + len -1 ; -while (p > str && *p ==' ' ) { - *p ='\0' ; - p--; - len --; - } - }else if (pg ==0x83 ) { -unsigned char *d; -unsigned char *vpd =NULL ; -int vpd_type, vpd_len, prio =-1 , i, naa_prio; - - d = (unsigned char *)buff +4 ; -while (d < (unsigned char *)buff + buff_len) { -/* Select 'association: LUN' */ -if ((d[1 ] &0x30 ) !=0 ) { - d += d[3 ] +4 ; -continue ; - } -switch (d[1 ] &0xf ) { -case 0x3 : -/* NAA: Prio 5 */ -switch (d[4 ] >>4 ) { -case 6 : -/* IEEE Registered Extended: Prio 8 */ - naa_prio =8 ; -break ; -case 5 : -/* IEEE Registered: Prio 7 */ - naa_prio =7 ; -break ; -case 2 : -/* IEEE Extended: Prio 6 */ - naa_prio =6 ; -break ; -case 3 : -/* IEEE Locally assigned: Prio 1 */ - naa_prio =1 ; -break ; -default : -/* Default: no priority */ - naa_prio =-1 ; -break ; - } -if (prio < naa_prio) { - prio = naa_prio; - vpd = d; - } -break ; -case 0x8 : -/* SCSI Name: Prio 4 */ -if (memcmp (d +4 ,"eui." ,4 ) && -memcmp (d +4 ,"naa." ,4 ) && -memcmp (d +4 ,"iqn." ,4 )) -continue ; -if (prio <4 ) { - prio =4 ; - vpd = d; - } -break ; -case 0x2 : -/* EUI-64: Prio 3 */ -if (prio <3 ) { - prio =3 ; - vpd = d; - } -break ; -case 0x1 : -/* T-10 Vendor ID: Prio 2 */ -if (prio <2 ) { - prio =2 ; - vpd = d; - } -break ; - } - d += d[3 ] +4 ; - } -if (prio >0 ) { - vpd_type = vpd[1 ] &0xf ; - vpd_len = vpd[3 ]; - vpd +=4 ; -if (vpd_type ==0x2 || vpd_type ==0x3 ) { -int i; - - len =sprintf (str,"%d" , vpd_type); -for (i =0 ; i < vpd_len; i++) { - len +=sprintf (str + len, -"%02x" , vpd[i]); -if (len >= maxlen) -break ; - } - }else if (vpd_type ==0x8 ) { -if (!memcmp ("eui." , vpd,4 )) { - str[0 ] ='2' ; - len =1 ; - vpd +=4 ; - vpd_len -=4 ; -for (i =0 ; i < vpd_len; i++) { - len +=sprintf (str + len,"%c" , -tolower (vpd[i])); -if (len >= maxlen) -break ; - } - len = vpd_len +1 ; - str[len] ='\0' ; - }else if (!memcmp ("naa." , vpd,4 )) { - str[0 ] ='3' ; - len =1 ; - vpd +=4 ; - vpd_len -=4 ; -for (i =0 ; i < vpd_len; i++) { - len +=sprintf (str + len,"%c" , -tolower (vpd[i])); -if (len >= maxlen) -break ; - } - len = vpd_len +1 ; - str[len] ='\0' ; - }else { - str[0 ] ='8' ; - len =1 ; - vpd +=4 ; - vpd_len -=4 ; -if (vpd_len > maxlen +2 ) - vpd_len = maxlen -2 ; -memcpy (str, vpd, vpd_len); - len = vpd_len +1 ; - str[len] ='\0' ; - } - }else if (vpd_type ==0x1 ) { -unsigned char *p; -int p_len; - - str[0 ] ='1' ; - len =1 ; - p = vpd; -while ((p =memchr (vpd,' ' , vpd_len))) { - p_len = p - vpd; -if (len + p_len > maxlen -1 ) - p_len = maxlen - len -2 ; -memcpy (str + len, vpd, p_len); - len += p_len; -if (len >= maxlen -1 ) { - str[len] ='\0' ; -break ; - } - str[len] ='_' ; - len ++; - vpd = p; - vpd_len -= p_len; -while (vpd && *vpd ==' ' ) { - vpd++; - vpd_len --; - } - } -if (len >1 && str[len -1 ] =='_' ) { - str[len -1 ] ='\0' ; - len--; - } - } - } - }else +if (pg ==0x80 ) + len =parse_vpd_pg80 (buff, str, maxlen); +else if (pg ==0x83 ) + len =parse_vpd_pg83 (buff, buff_len, str, maxlen); +else len = -ENOSYS ;return len;
Signature · 1 line
1.8.4.5
PATCH 70/78 multipathd: use SG_IO as fallback to generate uid
In reply to Hannes ReineckeIf both udev attribute and sysfs vpd fails to generate a valid WWID fallback to use SG_IO directly.
--- libmultipath/discovery.c | 61 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 50 insertions(+), 11 deletions(-)
libmultipath/discovery.c +50 -11
@@ -998,7 +998,7 @@ parse_vpd_pg83(const unsigned char *in, size_t in_len, }static int -get_vpd (struct udev_device *parent,int fd,int pg,char * str,int maxlen) +get_vpd_sysfs (struct udev_device *parent,int pg,char * str,int maxlen) {int len, buff_len;unsigned char buff[4096 ];
@@ -1006,11 +1006,39 @@ get_vpd (struct udev_device *parent, int fd, int pg, char * str, int maxlen)memset (buff,0x0 ,4096 );if (!parent ||sysfs_get_vpd (parent, pg, buff,4096 ) <=0 ) {condlog (3 ,"failed to read sysfs vpd pg%02x" , pg); -if (sgio_get_vpd (buff,4096 , fd) <=0 ) { -condlog (3 ,"failed to issue vpd inquiry for pg%02x" , - pg); -return -errno; - } +return -EINVAL ; + } + +if (buff[1 ] != pg) { +condlog (3 ,"vpd pg%02x error, invalid vpd page %02x" , + pg, buff[1 ]); +return -ENODATA ; + } + buff_len = (buff[2 ] <<8 ) + buff[3 ] +4 ; +if (buff_len >4096 ) +condlog (3 ,"vpd pg%02x page truncated" , pg); + +if (pg ==0x80 ) + len =parse_vpd_pg80 (buff, str, maxlen); +else if (pg ==0x83 ) + len =parse_vpd_pg83 (buff, buff_len, str, maxlen); +else + len = -ENOSYS ; + +return len; +} + +static int +get_vpd_sgio (int fd,int pg,char * str,int maxlen) +{ +int len, buff_len; +unsigned char buff[4096 ]; + +memset (buff,0x0 ,4096 ); +if (sgio_get_vpd (buff,4096 , fd) <=0 ) { +condlog (3 ,"failed to issue vpd inquiry for pg%02x" , + pg); +return -errno; }if (buff[1 ] != pg) {
@@ -1344,7 +1372,7 @@ scsi_ioctl_pathinfo (struct path * pp, int mask)if (!attr_path || pp->sg_id.host_no ==-1 )return -ENODEV ; -if (get_vpd (parent, pp->fd,0x80 , pp->serial,SERIAL_SIZE ) >0 ) +if (get_vpd_sysfs (parent,0x80 , pp->serial,SERIAL_SIZE ) >0 )condlog (3 ,"%s: serial = %s" , pp->dev, pp->serial);
@@ -1474,7 +1502,7 @@ get_vpd_uid(struct path * pp) parent =udev_device_get_parent (parent); } -return get_vpd (parent, pp->fd,0x83 , pp->wwid,WWID_SIZE ); +return get_vpd_sysfs (parent,0x83 , pp->wwid,WWID_SIZE ); } static int
@@ -1513,19 +1541,30 @@ get_uid (struct path * pp)if (pp->uid_attribute) { len =get_udev_uid (pp, pp->uid_attribute); origin ="udev" ; +if (len <=0 ) +condlog (2 , +"%s: failed to get UID attribute '%s'" , + pp->dev, pp->uid_attribute); }if (len <=0 ) { len =get_vpd_uid (pp);if (len >0 ) { origin ="sysfs" ; pp->uid_attribute =NULL ; + }else { +condlog (2 ,"%s: failed to get sysfs vpd pg83" , + pp->dev); } }if (len <=0 ) { - len =get_udev_uid (pp,DEFAULT_UID_ATTRIBUTE ); + len =get_vpd_sgio (pp->fd,0x83 , pp->wwid, +WWID_SIZE );if (len >0 ) { - origin ="default" ; - pp->uid_attribute =DEFAULT_UID_ATTRIBUTE ; + origin ="sgio" ; + pp->uid_attribute =NULL ; + }else { +condlog (2 ,"%s: failed to get sgio vpd pg83" , + pp->dev); } } }
Signature · 1 line
1.8.4.5
PATCH 71/78 Do not automatically fall back to vpd uid generation
In reply to Hannes ReineckeWhen a wwid via uid_attribute cannot be generated we should not fall back to using sysfs vpd attributes automatically. The uid_attribute can be set to virtually anything and an automatic fallback will then generated a different wwid, leading to an incorrect setup.
A fallback is only valid if the uid_attribute value is the default setting of 'ID_SERIAL', for which the format is fixed.
--- libmultipath/discovery.c | 39 ++++++++++++++++----------------------- 1 file changed, 16 insertions(+), 23 deletions(-)
libmultipath/discovery.c +16 -23
@@ -1509,7 +1509,7 @@ static int get_uid (struct path * pp) {char *c; -const char *origin ="none" ; +const char *origin ="unknown" ;ssize_t len =0 ;if (!pp->uid_attribute && !pp->getuid)
@@ -1542,35 +1542,28 @@ get_uid (struct path * pp) len =get_udev_uid (pp, pp->uid_attribute); origin ="udev" ;if (len <=0 ) -condlog (2 , -"%s: failed to get UID attribute '%s'" , - pp->dev, pp->uid_attribute); +condlog (1 , +"%s: failed to get udev uid: %s" , + pp->dev,strerror (-len)); + } -if (len <=0 ) { +if (len <=0 && + !strcmp (pp->uid_attribute,DEFAULT_UID_ATTRIBUTE )) { len =get_vpd_uid (pp); -if (len >0 ) { - origin ="sysfs" ; - pp->uid_attribute =NULL ; - }else { -condlog (2 ,"%s: failed to get sysfs vpd pg83" , - pp->dev); - } - } -if (len <=0 ) { - len =get_vpd_sgio (pp->fd,0x83 , pp->wwid, -WWID_SIZE ); -if (len >0 ) { + origin ="sysfs" ; + pp->uid_attribute =NULL ; +if (len <0 ) { +condlog (1 ,"%s: failed to get sysfs uid: %s" , + pp->dev,strerror (-len)); + len =get_vpd_sgio (pp->fd,0x83 , pp->wwid, +WWID_SIZE ); origin ="sgio" ; - pp->uid_attribute =NULL ; - }else { -condlog (2 ,"%s: failed to get sgio vpd pg83" , - pp->dev); } } }if ( len <0 ) { -condlog (1 ,"%s: failed to get uid: %s" , - pp->dev,strerror (-len)); +condlog (1 ,"%s: failed to get %s uid: %s" , + pp->dev, origin,strerror (-len));memset (pp->wwid,0x0 ,WWID_SIZE ); }else {/* Strip any trailing blanks */
Signature · 1 line
1.8.4.5
PATCH 72/78 libmultipath: make vpd page 0x80 optional
In reply to Hannes ReineckeVPD page 0x80 is optional, so we shouldn't return an error if the page is not present.
--- libmultipath/discovery.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
libmultipath/discovery.c +2 -2
@@ -1370,13 +1370,13 @@ scsi_ioctl_pathinfo (struct path * pp, int mask) parent =udev_device_get_parent (parent); }if (!attr_path || pp->sg_id.host_no ==-1 ) -return -ENODEV ; +return 0 ;if (get_vpd_sysfs (parent,0x80 , pp->serial,SERIAL_SIZE ) >0 )condlog (3 ,"%s: serial = %s" , pp->dev, pp->serial); -return strlen (pp->serial) ?0 : -EIO ; +return 0 ; } static int
Signature · 1 line
1.8.4.5
PATCH 73/78 multipathd: push down lock in checkerloop()
In reply to Hannes ReineckeInstead of grabbing the lock at the start of the checkerloop and releasing it at the end we should be holding it only during the time when we actually need it.
--- multipathd/main.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
multipathd/main.c +12 -4
@@ -1399,32 +1399,40 @@ checkerloop (void *ap)if (gettimeofday (&start_time,NULL ) !=0 ) start_time.tv_sec =0 ; -pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (vecs->lock); -pthread_testcancel ();condlog (4 ,"tick" );#ifdef USE_SYSTEMD if (use_watchdog)sd_notify (0 ,"WATCHDOG=1" );#endif if (vecs->pathvec) { +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); vector_foreach_slot (vecs->pathvec, pp, i) { num_paths +=check_path (vecs, pp); } +lock_cleanup_pop (vecs->lock); }if (vecs->mpvec) { +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel ();defered_failback_tick (vecs->mpvec);retry_count_tick (vecs->mpvec); +lock_cleanup_pop (vecs->lock); }if (count) count--;else { +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel ();condlog (4 ,"map garbage collection" );mpvec_garbage_collector (vecs); count =MAPGCINT ; +lock_cleanup_pop (vecs->lock); } -lock_cleanup_pop (vecs->lock);if (start_time.tv_sec &&gettimeofday (&end_time,NULL ) ==0 && num_paths) {
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 73/78] multipathd: push down lock in checkerloop()
Quoted 3 lines · Hannes Reinecke
Instead of grabbing the lock at the start of the checkerloop and releasing it at the end we should be holding it only during the time when we actually need it.
the vecs lock is designed to protect the vecs vectors (and conf), so checking vecs->pathvec and vecs->mpvec outside of the vecs lock doesn't guarantee us that they will still be there once we've grabbed the lock. I don't think this could actually cause a memory issue, since all the vecs code already checks if the vector is NULL, and the check before locking will probably be right most of the time. But perhaps we should check again after we do the locking, or at least add a comment saying that we know that the pre-lock check isn't guaranteed to be correct, so as not to confuse people.
-Ben
Quoted 57 lines
Signed-off-by Hannes Reinecke <[email protected]>--- multipathd/main.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)multipathd/main.c +12
-4@@ -1399,32 +1399,40 @@ checkerloop (void *ap)if (gettimeofday (&start_time,NULL ) !=0 ) start_time.tv_sec =0 ; -pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (vecs->lock); -pthread_testcancel ();condlog (4 ,"tick" );#ifdef USE_SYSTEMD if (use_watchdog)sd_notify (0 ,"WATCHDOG=1" );#endif if (vecs->pathvec) { +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); vector_foreach_slot (vecs->pathvec, pp, i) { num_paths +=check_path (vecs, pp); } +lock_cleanup_pop (vecs->lock); }if (vecs->mpvec) { +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel ();defered_failback_tick (vecs->mpvec);retry_count_tick (vecs->mpvec); +lock_cleanup_pop (vecs->lock); }if (count) count--;else { +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel ();condlog (4 ,"map garbage collection" );mpvec_garbage_collector (vecs); count =MAPGCINT ; +lock_cleanup_pop (vecs->lock); } -lock_cleanup_pop (vecs->lock);if (start_time.tv_sec &&gettimeofday (&end_time,NULL ) ==0 && num_paths) {Signature · 1 line
1.8.4.5
PATCH 74/78 Allow specific CLI commands to run unlocked
In reply to Hannes ReineckeWhen multipath is busy with checking paths or processing udev events it'll take the vector lock, causing the CLI
to become unresponsive.
This patch allows certain CLI commands to not wait for the vector lock, so that those commands will always succeed.
--- multipathd/cli.c | 26 +++++++++++++++++++++++++- multipathd/cli.h | 2 ++ multipathd/main.c | 17 ++++++----------- 3 files changed, 33 insertions(+), 12 deletions(-)
multipathd/cli.c +25 -1
@@ -1,8 +1,11 @@/* * Copyright (c) 2005 Christophe Varoqui */ +#include <pthread.h> #include <memory.h> #include <vector.h> +#include <structs.h> +#include <structs_vec.h> #include <parser.h> #include <util.h> #include <version.h>
@@ -99,6 +102,19 @@ set_handler_callback (unsigned long fp, int (*fn)(void *, char **, int *, void *if (!h)return 1 ; h->fn = fn; + h->locked =1 ; +return 0 ; +} + +int +set_unlocked_handler_callback (unsigned long fp,int (*fn)(void *,char **,int *,void *)) +{ +struct handler * h =find_handler (fp); + +if (!h) +return 1 ; + h->fn = fn; + h->locked =0 ;return 0 ; }
@@ -396,7 +412,15 @@ parse_cmd (char * cmd, char ** reply, int * len, void * data)/* * execute handler */ - r = h->fn (cmdvec, reply, len, data); +if (h->locked) { +struct vectors * vecs = (struct vectors *)data; +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); + r = h->fn (cmdvec, reply, len, data); +lock_cleanup_pop (vecs->lock); + }else + r = h->fn (cmdvec, reply, len, data);free_keys (cmdvec);return r;
multipathd/cli.h +2 -0
@@ -82,12 +82,14 @@ struct key {struct handler {unsigned long fingerprint; +int locked;int (*fn)(void *,char **,int *,void *); };int alloc_handlers (void );int add_handler (unsigned long fp,int (*fn)(void *,char **,int *,void *));int set_handler_callback (unsigned long fp,int (*fn)(void *,char **,int *,void *)); +int set_unlocked_handler_callback (unsigned long fp,int (*fn)(void *,char **,int *,void *));int parse_cmd (char * cmd,char ** reply,int * len,void *);int load_keys (void );char *get_keyparam (vector v,unsigned long code);
multipathd/main.c +6 -11
@@ -773,10 +773,6 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data) *len =0 ; vecs = (struct vectors *)trigger_data; -pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (vecs->lock); -pthread_testcancel (); - r =parse_cmd (str, reply, len, vecs);if (r >0 ) {
@@ -791,7 +787,6 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data) }/* else if (r < 0) leave *reply alone */ -lock_cleanup_pop (vecs->lock);return r; }
@@ -903,16 +898,16 @@ uxlsnrloop (void * ap)set_handler_callback (LIST +PATHS +FMT , cli_list_paths_fmt);set_handler_callback (LIST +PATH , cli_list_path);set_handler_callback (LIST +MAPS , cli_list_maps); -set_handler_callback (LIST +STATUS , cli_list_status); -set_handler_callback (LIST +DAEMON , cli_list_daemon); +set_unlocked_handler_callback (LIST +STATUS , cli_list_status); +set_unlocked_handler_callback (LIST +DAEMON , cli_list_daemon);set_handler_callback (LIST +MAPS +STATUS , cli_list_maps_status);set_handler_callback (LIST +MAPS +STATS , cli_list_maps_stats);set_handler_callback (LIST +MAPS +FMT , cli_list_maps_fmt);set_handler_callback (LIST +MAPS +TOPOLOGY , cli_list_maps_topology);set_handler_callback (LIST +TOPOLOGY , cli_list_maps_topology);set_handler_callback (LIST +MAP +TOPOLOGY , cli_list_map_topology); -set_handler_callback (LIST +CONFIG , cli_list_config); -set_handler_callback (LIST +BLACKLIST , cli_list_blacklist); +set_unlocked_handler_callback (LIST +CONFIG , cli_list_config); +set_unlocked_handler_callback (LIST +BLACKLIST , cli_list_blacklist);set_handler_callback (LIST +DEVICES , cli_list_devices);set_handler_callback (LIST +WILDCARDS , cli_list_wildcards);set_handler_callback (ADD +PATH , cli_add_path);
@@ -932,8 +927,8 @@ uxlsnrloop (void * ap)set_handler_callback (RESTOREQ +MAP , cli_restore_queueing);set_handler_callback (DISABLEQ +MAPS , cli_disable_all_queueing);set_handler_callback (RESTOREQ +MAPS , cli_restore_all_queueing); -set_handler_callback (QUIT , cli_quit); -set_handler_callback (SHUTDOWN , cli_shutdown); +set_unlocked_handler_callback (QUIT , cli_quit); +set_unlocked_handler_callback (SHUTDOWN , cli_shutdown);set_handler_callback (GETPRSTATUS +MAP , cli_getprstatus);set_handler_callback (SETPRSTATUS +MAP , cli_setprstatus);set_handler_callback (UNSETPRSTATUS +MAP , cli_unsetprstatus);
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 74/78] Allow specific CLI commands to run unlocked
Quoted 5 lines · Hannes Reinecke
When multipath is busy with checking paths or processing udev events it'll take the vector lock, causing the CLI
to become unresponsive.
This patch allows certain CLI commands to not wait for the vector lock, so that those commands will always succeed.
This one looks like it can actually cause memory corruption.
Holding the vecs lock is necessary to protect the vecs vectors and conf, and the only unlocked handler that doesn't touch anything that vecs protects is cli_quit.
All the others at least call condlog, which calls dlog, which includes this line
thres = (conf) ? conf->verbosity : 0;
During a reconfigure, conf will temporarily be null, and it can happen
after the check. Now as the code stands at this patch, the cli
handlers are protected from concurrent reconfigure calls because reconfigure() is also called by uxsock_listen, so it can't run at the same time as a cli handler. But your later patch "multipathd:
asynchronous configuration" moves this handling to child(), and I don't see anything that will keep it from running reconfigure() while a cli_handler is running.
cli_reconfigure and cli_list_blacklist make use of conf for much more than the initial condlog, so they are even more at risk.
But the biggest problem is cli_list_status, which traverses vecs->pathvec without the vecs lock held. I don't see anything to stop, for instance, ev_add_path from running at the same time. This means that it's possible that vecs->pathvec will get realloced out from under cli_list_status.
-Ben
Quoted 137 lines
Signed-off-by Hannes Reinecke <[email protected]>--- multipathd/cli.c | 26 +++++++++++++++++++++++++- multipathd/cli.h | 2 ++ multipathd/main.c | 17 ++++++----------- 3 files changed, 33 insertions(+), 12 deletions(-)multipathd/cli.c +25
-1@@ -1,8 +1,11 @@/* * Copyright (c) 2005 Christophe Varoqui */ +#include <pthread.h> #include <memory.h> #include <vector.h> +#include <structs.h> +#include <structs_vec.h> #include <parser.h> #include <util.h> #include <version.h> @@ -99,6 +102,19 @@ set_handler_callback (unsigned long fp, int (*fn)(void *, char **, int *, void *if (!h)return 1 ; h->fn = fn; + h->locked =1 ; +return 0 ; +} + +int +set_unlocked_handler_callback (unsigned long fp,int (*fn)(void *,char **,int *,void *)) +{ +struct handler * h =find_handler (fp); + +if (!h) +return 1 ; + h->fn = fn; + h->locked =0 ;return 0 ; }@@ -396,7 +412,15 @@ parse_cmd (char * cmd, char ** reply, int * len, void * data)/* * execute handler */ - r = h->fn (cmdvec, reply, len, data); +if (h->locked) { +struct vectors * vecs = (struct vectors *)data; +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); + r = h->fn (cmdvec, reply, len, data); +lock_cleanup_pop (vecs->lock); + }else + r = h->fn (cmdvec, reply, len, data);free_keys (cmdvec);return r;multipathd/cli.h +2
-0@@ -82,12 +82,14 @@ struct key {struct handler {unsigned long fingerprint; +int locked;int (*fn)(void *,char **,int *,void *); };int alloc_handlers (void );int add_handler (unsigned long fp,int (*fn)(void *,char **,int *,void *));int set_handler_callback (unsigned long fp,int (*fn)(void *,char **,int *,void *)); +int set_unlocked_handler_callback (unsigned long fp,int (*fn)(void *,char **,int *,void *));int parse_cmd (char * cmd,char ** reply,int * len,void *);int load_keys (void );char *get_keyparam (vector v,unsigned long code);multipathd/main.c +6
-11@@ -773,10 +773,6 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data) *len =0 ; vecs = (struct vectors *)trigger_data; -pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (vecs->lock); -pthread_testcancel (); - r =parse_cmd (str, reply, len, vecs);if (r >0 ) {@@ -791,7 +787,6 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data) }/* else if (r < 0) leave *reply alone */ -lock_cleanup_pop (vecs->lock);return r; }@@ -903,16 +898,16 @@ uxlsnrloop (void * ap)set_handler_callback (LIST +PATHS +FMT , cli_list_paths_fmt);set_handler_callback (LIST +PATH , cli_list_path);set_handler_callback (LIST +MAPS , cli_list_maps); -set_handler_callback (LIST +STATUS , cli_list_status); -set_handler_callback (LIST +DAEMON , cli_list_daemon); +set_unlocked_handler_callback (LIST +STATUS , cli_list_status); +set_unlocked_handler_callback (LIST +DAEMON , cli_list_daemon);set_handler_callback (LIST +MAPS +STATUS , cli_list_maps_status);set_handler_callback (LIST +MAPS +STATS , cli_list_maps_stats);set_handler_callback (LIST +MAPS +FMT , cli_list_maps_fmt);set_handler_callback (LIST +MAPS +TOPOLOGY , cli_list_maps_topology);set_handler_callback (LIST +TOPOLOGY , cli_list_maps_topology);set_handler_callback (LIST +MAP +TOPOLOGY , cli_list_map_topology); -set_handler_callback (LIST +CONFIG , cli_list_config); -set_handler_callback (LIST +BLACKLIST , cli_list_blacklist); +set_unlocked_handler_callback (LIST +CONFIG , cli_list_config); +set_unlocked_handler_callback (LIST +BLACKLIST , cli_list_blacklist);set_handler_callback (LIST +DEVICES , cli_list_devices);set_handler_callback (LIST +WILDCARDS , cli_list_wildcards);set_handler_callback (ADD +PATH , cli_add_path);@@ -932,8 +927,8 @@ uxlsnrloop (void * ap)set_handler_callback (RESTOREQ +MAP , cli_restore_queueing);set_handler_callback (DISABLEQ +MAPS , cli_disable_all_queueing);set_handler_callback (RESTOREQ +MAPS , cli_restore_all_queueing); -set_handler_callback (QUIT , cli_quit); -set_handler_callback (SHUTDOWN , cli_shutdown); +set_unlocked_handler_callback (QUIT , cli_quit); +set_unlocked_handler_callback (SHUTDOWN , cli_shutdown);set_handler_callback (GETPRSTATUS +MAP , cli_getprstatus);set_handler_callback (SETPRSTATUS +MAP , cli_setprstatus);set_handler_callback (UNSETPRSTATUS +MAP , cli_unsetprstatus);Signature · 1 line
1.8.4.5
PATCH 75/78 Push down vector lock during uevent processing
In reply to Hannes ReineckeWhen adding lots of paths the vector lock which is taken at the start of the uevent handler will become a bottleneck as it'll compete with the checkerloop.
So move the vector handling down into the individual event handler.
--- multipathd/main.c | 67 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 48 insertions(+), 19 deletions(-)
multipathd/main.c +48 -19
@@ -279,7 +279,11 @@ uev_add_map (struct uevent * uev, struct vectors * vecs)return 1 ; } } +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); rc =ev_add_map (uev->kernel, alias, vecs); +lock_cleanup_pop (vecs->lock);FREE (alias);return rc; }
@@ -361,6 +365,10 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)return 0 ; } minor =uevent_get_minor (uev); + +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); mpp =find_mp_by_minor (vecs->mpvec, minor);if (!mpp) {
@@ -377,10 +385,12 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)orphan_paths (vecs->pathvec, mpp);remove_map_and_stop_waiter (mpp, vecs,1 ); out: +lock_cleanup_pop (vecs->lock);FREE (alias);return 0 ; } +/* Called from CLI handler */ int ev_remove_map (char * devname,char * alias,int minor,struct vectors * vecs) {
@@ -416,6 +426,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)return 1 ; } +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); pp =find_path_by_dev (vecs->pathvec, uev->kernel);if (pp) {int r;
@@ -443,8 +456,15 @@ uev_add_path (struct uevent *uev, struct vectors * vecs) ret =1 ; } } -return ret; } +/* +* The linux implementation of pthread_lock() and pthread_unlock() +* requires that both must be at the same indentation level, +* hence the slightly odd coding. +*/ +lock_cleanup_pop (vecs->lock); +if (pp) +return ret;/* * get path vital state
@@ -462,6 +482,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)free_path (pp);return 1 ; } +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); ret =store_path (vecs->pathvec, pp);if (!ret) { pp->checkint = conf->checkint;
@@ -473,7 +496,7 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)free_path (pp); ret =1 ; } - +lock_cleanup_pop (vecs->lock);return ret; }
@@ -535,12 +558,12 @@ rescan:*/ start_waiter =1 ; } -else +if (!start_waiter) goto fail;/* leave path added to pathvec */ } -/* persistent reseravtion check*/ -mpath_pr_event_handle (pp); +/* persistent reservation check*/ +mpath_pr_event_handle (pp);/* * push the map to the device-mapper
@@ -568,7 +591,7 @@ retry:* deal with asynchronous uevents :(( */ if (mpp->action ==ACT_RELOAD && retries-- >0 ) { -condlog (0 ,"%s: uev_add_path sleep" , mpp->alias); +condlog (0 ,"%s: ev_add_path sleep" , mpp->alias);sleep (1 );update_mpp_paths (mpp, vecs->pathvec); goto rescan;
@@ -597,8 +620,7 @@ retry:condlog (2 ,"%s [%s]: path added to devmap %s" , pp->dev, pp->dev_t, mpp->alias);return 0 ; - } -else + }else goto fail; fail_map:
@@ -612,17 +634,22 @@ static int uev_remove_path (struct uevent *uev, struct vectors * vecs) { struct path *pp; +int ret;condlog (2 ,"%s: remove path (uevent)" , uev->kernel); +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); pp =find_path_by_dev (vecs->pathvec, uev->kernel); - +if (pp) + ret =ev_remove_path (pp, vecs); +lock_cleanup_pop (vecs->lock);if (!pp) {/* Not an error; path might have been purged earlier */ condlog (0 ,"%s: path already removed" , uev->kernel);return 0 ; } - -return ev_remove_path (pp, vecs); +return ret; }int
@@ -726,20 +753,27 @@ uev_update_path (struct uevent *uev, struct vectors * vecs)if (ro >=0 ) {struct path * pp; +struct multipath *mpp =NULL ;condlog (2 ,"%s: update path write_protect to '%d' (uevent)" , uev->kernel, ro); +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); pp =find_path_by_dev (vecs->pathvec, uev->kernel); +if (pp) + mpp = pp->mpp; +lock_cleanup_pop (vecs->lock);if (!pp) {condlog (0 ,"%s: spurious uevent, path not found" , uev->kernel);return 1 ; } -if (pp->mpp) { - retval =reload_map (vecs, pp->mpp,0 ); +if (mpp) { + retval =reload_map (vecs, mpp,0 );condlog (2 ,"%s: map %s reloaded (retval %d)" , - uev->kernel, pp->mpp->alias, retval); + uev->kernel, mpp->alias, retval); } }
@@ -823,10 +857,6 @@ uev_trigger (struct uevent * uev, void * trigger_data)if (uev_discard (uev->devpath))return 0 ; -pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (vecs->lock); -pthread_testcancel (); -/* * device map event * Add events are ignored here as the tables
@@ -865,7 +895,6 @@ uev_trigger (struct uevent * uev, void * trigger_data) } out: -lock_cleanup_pop (vecs->lock);return r; }
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 75/78] Push down vector lock during uevent processing
Quoted 5 lines · Hannes Reinecke
When adding lots of paths the vector lock which is taken at the start of the uevent handler will become a bottleneck as it'll compete with the checkerloop.
So move the vector handling down into the individual event handler.
Again, with these changes, there are places in these code paths where conf is dereferenced without vecs being held, and there's nothing to keep reconfigure from running at the same time as these functions.
-Ben
Quoted 205 lines
Signed-off-by Hannes Reinecke <[email protected]>--- multipathd/main.c | 67 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 48 insertions(+), 19 deletions(-)multipathd/main.c +48
-19@@ -279,7 +279,11 @@ uev_add_map (struct uevent * uev, struct vectors * vecs)return 1 ; } } +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); rc =ev_add_map (uev->kernel, alias, vecs); +lock_cleanup_pop (vecs->lock);FREE (alias);return rc; }@@ -361,6 +365,10 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)return 0 ; } minor =uevent_get_minor (uev); + +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); mpp =find_mp_by_minor (vecs->mpvec, minor);if (!mpp) {@@ -377,10 +385,12 @@ uev_remove_map (struct uevent * uev, struct vectors * vecs)orphan_paths (vecs->pathvec, mpp);remove_map_and_stop_waiter (mpp, vecs,1 ); out: +lock_cleanup_pop (vecs->lock);FREE (alias);return 0 ; } +/* Called from CLI handler */ int ev_remove_map (char * devname,char * alias,int minor,struct vectors * vecs) {@@ -416,6 +426,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)return 1 ; } +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); pp =find_path_by_dev (vecs->pathvec, uev->kernel);if (pp) {int r;@@ -443,8 +456,15 @@ uev_add_path (struct uevent *uev, struct vectors * vecs) ret =1 ; } } -return ret; } +/* +* The linux implementation of pthread_lock() and pthread_unlock() +* requires that both must be at the same indentation level, +* hence the slightly odd coding. +*/ +lock_cleanup_pop (vecs->lock); +if (pp) +return ret;/* * get path vital state @@ -462,6 +482,9 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)free_path (pp);return 1 ; } +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); ret =store_path (vecs->pathvec, pp);if (!ret) { pp->checkint = conf->checkint;@@ -473,7 +496,7 @@ uev_add_path (struct uevent *uev, struct vectors * vecs)free_path (pp); ret =1 ; } - +lock_cleanup_pop (vecs->lock);return ret; }@@ -535,12 +558,12 @@ rescan:*/ start_waiter =1 ; } -else +if (!start_waiter) goto fail;/* leave path added to pathvec */ } -/* persistent reseravtion check*/ -mpath_pr_event_handle (pp); +/* persistent reservation check*/ +mpath_pr_event_handle (pp);/* * push the map to the device-mapper @@ -568,7 +591,7 @@ retry:* deal with asynchronous uevents :(( */ if (mpp->action ==ACT_RELOAD && retries-- >0 ) { -condlog (0 ,"%s: uev_add_path sleep" , mpp->alias); +condlog (0 ,"%s: ev_add_path sleep" , mpp->alias);sleep (1 );update_mpp_paths (mpp, vecs->pathvec); goto rescan;@@ -597,8 +620,7 @@ retry:condlog (2 ,"%s [%s]: path added to devmap %s" , pp->dev, pp->dev_t, mpp->alias);return 0 ; - } -else + }else goto fail; fail_map:@@ -612,17 +634,22 @@ static int uev_remove_path (struct uevent *uev, struct vectors * vecs) { struct path *pp; +int ret;condlog (2 ,"%s: remove path (uevent)" , uev->kernel); +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); pp =find_path_by_dev (vecs->pathvec, uev->kernel); - +if (pp) + ret =ev_remove_path (pp, vecs); +lock_cleanup_pop (vecs->lock);if (!pp) {/* Not an error; path might have been purged earlier */ condlog (0 ,"%s: path already removed" , uev->kernel);return 0 ; } - -return ev_remove_path (pp, vecs); +return ret; }int @@ -726,20 +753,27 @@ uev_update_path (struct uevent *uev, struct vectors * vecs)if (ro >=0 ) {struct path * pp; +struct multipath *mpp =NULL ;condlog (2 ,"%s: update path write_protect to '%d' (uevent)" , uev->kernel, ro); +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); pp =find_path_by_dev (vecs->pathvec, uev->kernel); +if (pp) + mpp = pp->mpp; +lock_cleanup_pop (vecs->lock);if (!pp) {condlog (0 ,"%s: spurious uevent, path not found" , uev->kernel);return 1 ; } -if (pp->mpp) { - retval =reload_map (vecs, pp->mpp,0 ); +if (mpp) { + retval =reload_map (vecs, mpp,0 );condlog (2 ,"%s: map %s reloaded (retval %d)" , - uev->kernel, pp->mpp->alias, retval); + uev->kernel, mpp->alias, retval); } }@@ -823,10 +857,6 @@ uev_trigger (struct uevent * uev, void * trigger_data)if (uev_discard (uev->devpath))return 0 ; -pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (vecs->lock); -pthread_testcancel (); -/* * device map event * Add events are ignored here as the tables @@ -865,7 +895,6 @@ uev_trigger (struct uevent * uev, void * trigger_data) } out: -lock_cleanup_pop (vecs->lock);return r; }Signature · 1 line
1.8.4.5
PATCH 76/78 multipathd: timeout CLI commands when waiting for lock
In reply to Hannes ReineckeWhen a CLI command is waiting for the vector lock it might timeout, but the command itself remains queued. This causes some irritation as no other commands can be send until the original command is processed.
--- libmultipath/config.h | 2 ++ libmultipath/configure.c | 7 ++++--- libmultipath/dict.c | 11 +++++++---- libmultipath/lock.h | 4 ++-- multipathd/cli.c | 26 ++++++++++++++++++++++---- multipathd/cli.h | 2 +- multipathd/main.c | 12 ++++++++---- multipathd/main.h | 1 + multipathd/uxlsnr.c | 16 +++------------- 9 files changed, 50 insertions(+), 31 deletions(-)
libmultipath/config.h +2 -0
@@ -170,6 +170,8 @@ struct config {vector elist_property; }; +extern int uxsock_timeout; +struct config * conf;struct hwentry *find_hwe (vector hwtable,char * vendor,char * product,char *revision);
libmultipath/configure.c +4 -3
@@ -701,7 +701,7 @@ deadmap (struct multipath * mpp)return 1 ;/* dead */ } -int check_daemon (void ) +int check_daemon (int timeout) {int fd;char *reply;
@@ -714,7 +714,7 @@ int check_daemon(void)if (send_packet (fd,"show daemon" ,12 ) !=0 ) goto out; -if (recv_packet (fd, &reply, &len, conf->uxsock_timeout) !=0 ) +if (recv_packet (fd, &reply, &len, timeout) !=0 ) goto out;if (strstr (reply,"shutdown" ))
@@ -848,7 +848,8 @@ coalesce_paths (struct vectors * vecs, vector newmp, char * refwwid, int force_rif (r ==DOMAP_DRY )continue ; -if (!conf->daemon && !conf->allow_queueing && !check_daemon ()) { +if (!conf->daemon && !conf->allow_queueing && + !check_daemon (uxsock_timeout)) {if (mpp->no_path_retry !=NO_PATH_RETRY_UNDEF && mpp->no_path_retry !=NO_PATH_RETRY_FAIL ) condlog(3, "%s: multipathd not running, unset "
libmultipath/dict.c +7 -4
@@ -1037,23 +1037,26 @@ declare_mp_snprint(delay_wait_checks, print_delay_checks)static int def_uxsock_timeout_handler (vector strvec) { -unsigned int uxsock_timeout; +unsigned int timeout;char *buff; buff =set_value (strvec);if (!buff)return 1 ; -if (sscanf (buff,"%u" , &uxsock_timeout) ==1 && - uxsock_timeout >DEFAULT_UXSOCK_TIMEOUT ) - conf->uxsock_timeout = uxsock_timeout; +if (sscanf (buff,"%u" , &timeout) ==1 && + timeout >DEFAULT_UXSOCK_TIMEOUT ) + conf->uxsock_timeout = timeout;else conf->uxsock_timeout =DEFAULT_UXSOCK_TIMEOUT ;free (buff); + uxsock_timeout = conf->uxsock_timeout;return 0 ; } +int uxsock_timeout =DEFAULT_UXSOCK_TIMEOUT ; +/* * blacklist block handlers */
libmultipath/lock.h +2 -2
@@ -21,11 +21,11 @@ struct mutex_lock { a.depth--; pthread_mutex_unlock(a.mutex)#define lock_cleanup_pop (a) \ fprintf(stderr, "%s:%s(%i) unlock %p depth: %d (%ld)\n", __FILE__, __FUNCTION__, __LINE__, a.mutex, a.depth, pthread_self()); \ - pthread_cleanup_pop(1); + pthread_cleanup_pop(1)#else #define lock (a) a.depth++; pthread_mutex_lock(a.mutex)#define unlock (a) a.depth--; pthread_mutex_unlock(a.mutex) -#define lock_cleanup_pop (a) pthread_cleanup_pop(1); +#define lock_cleanup_pop (a) pthread_cleanup_pop(1)#endif void cleanup_lock (void * data);
multipathd/cli.c +22 -4
@@ -1,6 +1,7 @@/* * Copyright (c) 2005 Christophe Varoqui */ +#include <sys/time.h> #include <pthread.h> #include <memory.h> #include <vector.h>
@@ -386,11 +387,13 @@ genhelp_handler (void) }int -parse_cmd (char * cmd,char ** reply,int * len,void * data) +parse_cmd (char * cmd,char ** reply,int * len,void * data,int timeout ) {int r;struct handler * h;vector cmdvec =NULL ; +struct timespec tmo; +struct timeval now; r =get_cmdvec (cmd, &cmdvec);
@@ -412,12 +415,27 @@ parse_cmd (char * cmd, char ** reply, int * len, void * data)/* * execute handler */ +if (gettimeofday (&now,NULL ) ==0 ) { + tmo.tv_sec = now.tv_sec + timeout; + tmo.tv_nsec = now.tv_usec *1000 ; + }else { + tmo.tv_sec =0 ; + }if (h->locked) {struct vectors * vecs = (struct vectors *)data; +pthread_cleanup_push (cleanup_lock, &vecs->lock); -lock (vecs->lock); -pthread_testcancel (); - r = h->fn (cmdvec, reply, len, data); +if (tmo.tv_sec) { + vecs->lock.depth++; + r =pthread_mutex_timedlock (vecs->lock.mutex, &tmo); + }else { +lock (vecs->lock); + r =0 ; + } +if (r ==0 ) { +pthread_testcancel (); + r = h->fn (cmdvec, reply, len, data); + }lock_cleanup_pop (vecs->lock); }else r = h->fn (cmdvec, reply, len, data);
multipathd/cli.h +1 -1
@@ -90,7 +90,7 @@ int alloc_handlers (void);int add_handler (unsigned long fp,int (*fn)(void *,char **,int *,void *));int set_handler_callback (unsigned long fp,int (*fn)(void *,char **,int *,void *));int set_unlocked_handler_callback (unsigned long fp,int (*fn)(void *,char **,int *,void *)); -int parse_cmd (char * cmd,char ** reply,int * len,void *); +int parse_cmd (char * cmd,char ** reply,int * len,void *,int );int load_keys (void );char *get_keyparam (vector v,unsigned long code);void free_keys (vector vec);
multipathd/main.c +8 -4
@@ -807,10 +807,13 @@ uxsock_trigger (char * str, char ** reply, int * len, void * trigger_data) *len =0 ; vecs = (struct vectors *)trigger_data; - r =parse_cmd (str, reply, len, vecs); + r =parse_cmd (str, reply, len, vecs, uxsock_timeout /1000 );if (r >0 ) { - *reply =STRDUP ("fail\n" ); +if (r ==ETIMEDOUT ) + *reply =STRDUP ("timeout\n" ); +else + *reply =STRDUP ("fail\n" ); *len =strlen (*reply) +1 ; r =1 ; }
@@ -917,6 +920,7 @@ uevqloop (void * ap)return NULL ; } +static void *uxlsnrloop (void * ap) {
@@ -2102,7 +2106,7 @@ main (int argc, char *argv[])case 'k' :if (load_config (DEFAULT_CONFIGFILE ,udev_new ()))exit (1 ); -uxclnt (optarg, conf->uxsock_timeout); +uxclnt (optarg, uxsock_timeout +100 );exit (0 );case 'B' : conf->bindings_read_only =1 ;
@@ -2126,7 +2130,7 @@ main (int argc, char *argv[]) optind++; } c +=snprintf (c, s +CMDSIZE - c,"\n" ); -uxclnt (s, conf->uxsock_timeout); +uxclnt (s, uxsock_timeout +100 );exit (0 ); }
multipathd/main.h +1 -0
@@ -15,6 +15,7 @@ struct prout_param_descriptor;struct prin_resp ;extern pid_t daemon_pid; +extern int uxsock_timeout;void exit_daemon (void );const char *daemon_status (void );
multipathd/uxlsnr.c +3 -13
@@ -125,7 +125,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data) {int ux_sock;size_t len; -int rlen, timeout; +int rlen;char *inbuf;char *reply;sigset_t mask;
@@ -142,8 +142,6 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)return NULL ; } - timeout = conf->uxsock_timeout; -pthread_cleanup_push (uxsock_cleanup,NULL ); polls = (struct pollfd *)MALLOC (0 );
@@ -154,14 +152,6 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)struct client *c, *tmp;int i, poll_count, num_clients; -/* -* Store configuration timeout; -* configuration might change during -* the call to 'reconfigure'. -*/ -if (conf) - timeout = conf->uxsock_timeout; -/* setup for a poll */ pthread_mutex_lock (&client_lock); num_clients =0 ;
@@ -221,7 +211,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data) start_time.tv_sec =0 ;if (recv_packet (c->fd, &inbuf, &len, - timeout) !=0 ) { + uxsock_timeout) !=0 ) {dead_client (c); }else { inbuf[len -1 ] =0 ;
@@ -239,7 +229,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data) reply =NULL ; }check_timeout (start_time, inbuf, - timeout); + uxsock_timeout);FREE (inbuf); } }
Signature · 1 line
1.8.4.5
PATCH 77/78 multipathd: asynchronous configuration
In reply to Hannes ReineckeFor initial configuration multipathd waits it has synchronized with the existing setup. On larger systems this takes up quite some time (I've measured 80 seconds on a system with 1024 paths) causing systemd to stall and the system to fail booting. This patch makes the initial configuration asynchronous, and using the same codepath as the existing 'reconfigure' CLI command.
--- multipathd/cli_handlers.c | 5 ++- multipathd/main.c | 79 ++++++++++++++++++++++++++++------------------- multipathd/main.h | 1 + 3 files changed, 51 insertions(+), 34 deletions(-)
multipathd/cli_handlers.c +2 -3
@@ -830,11 +830,10 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)int cli_reconfigure (void * v,char ** reply,int * len,void * data) { -struct vectors * vecs = (struct vectors *)data; -condlog (2 ,"reconfigure (operator)" ); -return reconfigure (vecs); +post_config_state (DAEMON_CONFIGURE ); +return 0 ; }int
multipathd/main.c +48 -31
167 lines not on this page · raw message
multipathd/main.h +1 -0
@@ -25,6 +25,7 @@ int ev_remove_path (struct path *, struct vectors *);int ev_add_map (char *,char *,struct vectors *);int ev_remove_map (char *,char *,int ,struct vectors *);void sync_map_state (struct multipath *); +void post_config_state (enum daemon_status );void *mpath_alloc_prin_response (int prin_sa);int prin_do_scsi_ioctl (char *,int rq_servact,struct prin_resp * resp,int noisy);
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 77/78] multipathd: asynchronous configuration
Quoted 7 lines · Hannes Reinecke
For initial configuration multipathd waits it has synchronized with the existing setup. On larger systems this takes up quite some time (I've measured 80 seconds on a system with 1024 paths) causing systemd to stall and the system to fail booting. This patch makes the initial configuration asynchronous, and using the same codepath as the existing 'reconfigure' CLI command.
If the issue is that configure takes too long, and is keeping multipath from resetting the watchdog timer, can't this problem still happen?
reconfigure still is called under the vecs lock, and checkerloop still locks the vecs lock, so a call to reconfigure can still stall the checker loop for just as long.
Or am I missing something?
-Ben
Quoted 223 lines
Signed-off-by Hannes Reinecke <[email protected]>--- multipathd/cli_handlers.c | 5 ++- multipathd/main.c | 79 ++++++++++++++++++++++++++++------------------- multipathd/main.h | 1 + 3 files changed, 51 insertions(+), 34 deletions(-)multipathd/cli_handlers.c +2
-3@@ -830,11 +830,10 @@ cli_switch_group(void * v, char ** reply, int * len, void * data)int cli_reconfigure (void * v,char ** reply,int * len,void * data) { -struct vectors * vecs = (struct vectors *)data; -condlog (2 ,"reconfigure (operator)" ); -return reconfigure (vecs); +post_config_state (DAEMON_CONFIGURE ); +return 0 ; }int multipathd/main.c +48
-31@@ -93,10 +93,11 @@ struct mpath_event_paramunsigned int mpath_mx_alloc_len;int logsink; -enum daemon_status running_state; +enum daemon_status running_state =DAEMON_INIT ;pid_t daemon_pid; +pthread_mutex_t config_lock =PTHREAD_MUTEX_INITIALIZER ; +pthread_cond_t config_cond =PTHREAD_COND_INITIALIZER ; -static sem_t exit_sem;/* * global copy of vecs for use in sig handlers */ @@ -104,6 +105,21 @@ struct vectors * gvecs;struct udev * udev; +static void config_cleanup (void *arg) +{ +pthread_mutex_unlock (&config_lock); +} + +void post_config_state (enum daemon_status state) +{ +pthread_mutex_lock (&config_lock); +if (state != running_state) { + running_state = state; +pthread_cond_broadcast (&config_cond); + } +pthread_mutex_unlock (&config_lock); +} +static int need_switch_pathgroup (struct multipath * mpp,int refresh) {@@ -860,6 +876,15 @@ uev_trigger (struct uevent * uev, void * trigger_data)if (uev_discard (uev->devpath))return 0 ; +pthread_cleanup_push (config_cleanup,NULL ); +pthread_mutex_lock (&config_lock); +if (running_state !=DAEMON_RUNNING ) +pthread_cond_wait (&config_cond, &config_lock); +pthread_cleanup_pop (1 ); + +if (running_state ==DAEMON_SHUTDOWN ) +return 0 ; +/* * device map event * Add events are ignored here as the tables @@ -948,7 +973,7 @@ uxlsnrloop (void * ap)set_handler_callback (ADD +MAP , cli_add_map);set_handler_callback (DEL +MAP , cli_del_map);set_handler_callback (SWITCH +MAP +GROUP , cli_switch_group); -set_handler_callback (RECONFIGURE , cli_reconfigure); +set_unlocked_handler_callback (RECONFIGURE , cli_reconfigure);set_handler_callback (SUSPEND +MAP , cli_suspend);set_handler_callback (RESUME +MAP , cli_resume);set_handler_callback (RESIZE +MAP , cli_resize);@@ -977,7 +1002,7 @@ uxlsnrloop (void * ap)void exit_daemon (void ) { -sem_post (&exit_sem); +post_config_state (DAEMON_SHUTDOWN ); } const char *@@ -1561,8 +1586,6 @@ reconfigure (struct vectors * vecs)struct config * old = conf;int retval =1 ; - running_state =DAEMON_CONFIGURE ; -/* * free old map and path vectors ... they use old conf state */ @@ -1587,7 +1610,7 @@ reconfigure (struct vectors * vecs) retval =0 ; } - running_state =DAEMON_RUNNING ; +post_config_state (DAEMON_RUNNING );return retval; }@@ -1643,12 +1666,7 @@ handle_signals(void) {if (reconfig_sig && running_state ==DAEMON_RUNNING ) {condlog (2 ,"reconfigure (signal)" ); -pthread_cleanup_push (cleanup_lock, - &gvecs->lock); -lock (gvecs->lock); -pthread_testcancel (); -reconfigure (gvecs); -lock_cleanup_pop (gvecs->lock); +post_config_state (DAEMON_CONFIGURE ); }if (log_reset_sig) {condlog (2 ,"reset log (signal)" );@@ -1781,7 +1799,6 @@ child (void * param)char *envp;mlockall (MCL_CURRENT |MCL_FUTURE ); -sem_init (&exit_sem,0 ,0 );signal_init (); udev =udev_new ();@@ -1796,7 +1813,7 @@ child (void * param)pthread_attr_destroy (&log_attr); } - running_state =DAEMON_START ; +post_config_state (DAEMON_START );#ifdef USE_SYSTEMD sd_notify (0 ,"STATUS=startup" );@@ -1891,15 +1908,7 @@ child (void * param)#ifdef USE_SYSTEMD sd_notify (0 ,"STATUS=configure" );#endif - running_state =DAEMON_CONFIGURE ; - -lock (vecs->lock); -if (configure (vecs,1 )) { -unlock (vecs->lock); -condlog (0 ,"failure during configuration" ); - goto failed; - } -unlock (vecs->lock); +post_config_state (DAEMON_CONFIGURE );/* * start threads @@ -1918,20 +1927,29 @@ child (void * param) pid_rc =pidfile_create (DEFAULT_PIDFILE , daemon_pid);/* Ignore errors, we can live without */ - running_state =DAEMON_RUNNING ;#ifdef USE_SYSTEMD sd_notify (0 ,"READY=1\nSTATUS=running" );#endif -/* -* exit path -*/ -while (sem_wait (&exit_sem) !=0 );/* Do nothing */ +while (running_state !=DAEMON_SHUTDOWN ) { +pthread_cleanup_push (config_cleanup,NULL ); +pthread_mutex_lock (&config_lock); +if (running_state ==DAEMON_RUNNING ) { +pthread_cond_wait (&config_cond, &config_lock); + } +pthread_cleanup_pop (1 ); +if (running_state ==DAEMON_CONFIGURE ) { +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); +reconfigure (vecs); +lock_cleanup_pop (vecs->lock); + } + }#ifdef USE_SYSTEMD sd_notify (0 ,"STATUS=shutdown" );#endif - running_state =DAEMON_SHUTDOWN ;lock (vecs->lock);if (conf->queue_without_daemon ==QUE_NO_DAEMON_OFF )vector_foreach_slot (vecs->mpvec, mpp, i)@@ -2066,7 +2084,6 @@ main (int argc, char *argv[])int foreground =0 ; logsink =1 ; - running_state =DAEMON_INIT ;dm_init ();if (getuid () !=0 ) {multipathd/main.h +1
-0@@ -25,6 +25,7 @@ int ev_remove_path (struct path *, struct vectors *);int ev_add_map (char *,char *,struct vectors *);int ev_remove_map (char *,char *,int ,struct vectors *);void sync_map_state (struct multipath *); +void post_config_state (enum daemon_status );void *mpath_alloc_prin_response (int prin_sa);int prin_do_scsi_ioctl (char *,int rq_servact,struct prin_resp * resp,int noisy);Signature · 1 line
1.8.4.5
Re: [PATCH 77/78] multipathd: asynchronous configuration
Quoted 9 lines · Benjamin Marzinski
Quoted 7 lines · Hannes Reinecke
For initial configuration multipathd waits it has synchronized with the existing setup. On larger systems this takes up quite some time (I've measured 80 seconds on a system with 1024 paths) causing systemd to stall and the system to fail booting. This patch makes the initial configuration asynchronous, and using the same codepath as the existing 'reconfigure' CLI command.
Quoted 8 lines
If the issue is that configure takes too long, and is keeping multipath from resetting the watchdog timer, can't this problem still happen?
reconfigure still is called under the vecs lock, and checkerloop still locks the vecs lock, so a call to reconfigure can still stall the checker loop for just as long.
Or am I missing something?
The problem here is that on large configurations multipathd might spend too much time in configure(), so the sd_notify() call with 'RUNNING' is would be later than the default systemd service timeout. Hence multipathd will be killed by systemd with a job timeout, with no chance to get it to run, ever.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
PATCH 78/78 multipathd: trigger all devices on startup
In reply to Hannes ReineckeDuring startup multipathd might race with udev and device discovery. During that time any information from libudev might not be fully available, leading to spurious multipathd failures during startup. So instead of scanning all devices on our own we should just re-trigger the existing devices; with that we'll read _all_ devices via uevents during startup and avoid the race condition.
--- libmultipath/discovery.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ libmultipath/discovery.h | 1 + multipathd/main.c | 36 +++++++++++++++++++++++++++++---- 3 files changed, 85 insertions(+), 4 deletions(-)
libmultipath/discovery.c +52 -0
@@ -134,6 +134,58 @@ path_discover (vector pathvec, struct config * conf, }int +path_trigger (struct config * conf,int flag) +{ +struct udev_enumerate *udev_iter; +struct udev_list_entry *entry; +int num_paths =0 ; + + udev_iter =udev_enumerate_new (conf->udev); +if (!udev_iter) +return -ENOMEM ; + +udev_enumerate_add_match_subsystem (udev_iter,"block" ); +udev_enumerate_scan_devices (udev_iter); + +udev_list_entry_foreach (entry, + udev_enumerate_get_list_entry(udev_iter)) { +const char *devpath; +char *devname; +char filename[PATH_MAX ]; +int fd; + + devpath =udev_list_entry_get_name (entry); +condlog (3 ,"Trigger device %s" , devpath); + devname =strrchr (devpath,'/' ); +if (!devname) { +condlog (3 ,"%s: invalid devpath" , devpath); +continue ; + } + devname++; +if (filter_devnode (conf->blist_devnode, + conf->elist_devnode, devname) >0 ) { +condlog (3 ,"%s: blacklisted" , devname); +continue ; + } +strncpy (filename, devpath,strlen (devpath) +1 ); +strncat (filename,"/uevent" ,8 ); + fd =open (filename,O_WRONLY |O_CLOEXEC ); +if (fd <0 ) +continue ; +if (write (fd,"add" ,3 ) <0 ) { +condlog (3 ,"%s: Failed to trigger 'add' uevent: %m" , + devpath); + }else { + num_paths++; + } +close (fd); + } +udev_enumerate_unref (udev_iter); +condlog (4 ,"Triggered %d paths" , num_paths); +return num_paths; +} + +int path_discovery (vector pathvec,struct config * conf,int flag) {struct udev_enumerate *udev_iter;
libmultipath/discovery.h +1 -0
@@ -31,6 +31,7 @@struct config ;int path_discovery (vector pathvec,struct config * conf,int flag); +int path_trigger (struct config * conf,int flag);int do_tur (char *);int path_offline (struct path *);
multipathd/main.c +32 -4
@@ -1581,6 +1581,24 @@ configure (struct vectors * vecs, int start_waiters) }int +trigger_devices (struct vectors * vecs) +{ + +if (!vecs->pathvec && !(vecs->pathvec =vector_alloc ())) +return 1 ; + +if (!vecs->mpvec && !(vecs->mpvec =vector_alloc ())) +return 1 ; + +/* +* Trigger all non-blacklisted block devices +*/ +path_trigger (conf,DI_ALL ); + +return 0 ; +} + +int reconfigure (struct vectors * vecs) {struct config * old = conf;
@@ -1902,15 +1920,21 @@ child (void * param)condlog (0 ,"failed to create cli listener: %d" , rc); goto failed; } -/* -* fetch and configure both paths and multipaths -*/ #ifdef USE_SYSTEMD sd_notify (0 ,"STATUS=configure" );#endif post_config_state (DAEMON_CONFIGURE );/* +* Trigger all paths to force reconfiguration +*/ +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); +trigger_devices (vecs); +lock_cleanup_pop (vecs->lock); + +/* * start threads */ if ((rc =pthread_create (&check_thr, &misc_attr, checkerloop, vecs))) {
@@ -1927,6 +1951,8 @@ child (void * param) pid_rc =pidfile_create (DEFAULT_PIDFILE , daemon_pid);/* Ignore errors, we can live without */ +post_config_state (DAEMON_RUNNING ); +#ifdef USE_SYSTEMD sd_notify (0 ,"READY=1\nSTATUS=running" );#endif
@@ -2129,7 +2155,9 @@ main (int argc, char *argv[]) conf->bindings_read_only =1 ;break ;default : - ; +fprintf (stderr,"Invalid argument '-%c'\n" , + optopt); +exit (1 ); } }if (optind < argc) {
Signature · 1 line
1.8.4.5
Benjamin Marzinski
maintainerRe: [PATCH 78/78] multipathd: trigger all devices on startup
Quoted 6 lines · Hannes Reinecke
During startup multipathd might race with udev and device discovery. During that time any information from libudev might not be fully available, leading to spurious multipathd failures during startup. So instead of scanning all devices on our own we should just re-trigger the existing devices; with that we'll read _all_ devices via uevents during startup and avoid the race condition.
Should we also be sending change uevents for the dm devices?
-Ben
Quoted 159 lines
Signed-off-by Hannes Reinecke <[email protected]>--- libmultipath/discovery.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ libmultipath/discovery.h | 1 + multipathd/main.c | 36 +++++++++++++++++++++++++++++---- 3 files changed, 85 insertions(+), 4 deletions(-)libmultipath/discovery.c +52
-0@@ -134,6 +134,58 @@ path_discover (vector pathvec, struct config * conf, }int +path_trigger (struct config * conf,int flag) +{ +struct udev_enumerate *udev_iter; +struct udev_list_entry *entry; +int num_paths =0 ; + + udev_iter =udev_enumerate_new (conf->udev); +if (!udev_iter) +return -ENOMEM ; + +udev_enumerate_add_match_subsystem (udev_iter,"block" ); +udev_enumerate_scan_devices (udev_iter); + +udev_list_entry_foreach (entry, + udev_enumerate_get_list_entry(udev_iter)) { +const char *devpath; +char *devname; +char filename[PATH_MAX ]; +int fd; + + devpath =udev_list_entry_get_name (entry); +condlog (3 ,"Trigger device %s" , devpath); + devname =strrchr (devpath,'/' ); +if (!devname) { +condlog (3 ,"%s: invalid devpath" , devpath); +continue ; + } + devname++; +if (filter_devnode (conf->blist_devnode, + conf->elist_devnode, devname) >0 ) { +condlog (3 ,"%s: blacklisted" , devname); +continue ; + } +strncpy (filename, devpath,strlen (devpath) +1 ); +strncat (filename,"/uevent" ,8 ); + fd =open (filename,O_WRONLY |O_CLOEXEC ); +if (fd <0 ) +continue ; +if (write (fd,"add" ,3 ) <0 ) { +condlog (3 ,"%s: Failed to trigger 'add' uevent: %m" , + devpath); + }else { + num_paths++; + } +close (fd); + } +udev_enumerate_unref (udev_iter); +condlog (4 ,"Triggered %d paths" , num_paths); +return num_paths; +} + +int path_discovery (vector pathvec,struct config * conf,int flag) {struct udev_enumerate *udev_iter;libmultipath/discovery.h +1
-0@@ -31,6 +31,7 @@struct config ;int path_discovery (vector pathvec,struct config * conf,int flag); +int path_trigger (struct config * conf,int flag);int do_tur (char *);int path_offline (struct path *);multipathd/main.c +32
-4@@ -1581,6 +1581,24 @@ configure (struct vectors * vecs, int start_waiters) }int +trigger_devices (struct vectors * vecs) +{ + +if (!vecs->pathvec && !(vecs->pathvec =vector_alloc ())) +return 1 ; + +if (!vecs->mpvec && !(vecs->mpvec =vector_alloc ())) +return 1 ; + +/* +* Trigger all non-blacklisted block devices +*/ +path_trigger (conf,DI_ALL ); + +return 0 ; +} + +int reconfigure (struct vectors * vecs) {struct config * old = conf;@@ -1902,15 +1920,21 @@ child (void * param)condlog (0 ,"failed to create cli listener: %d" , rc); goto failed; } -/* -* fetch and configure both paths and multipaths -*/ #ifdef USE_SYSTEMD sd_notify (0 ,"STATUS=configure" );#endif post_config_state (DAEMON_CONFIGURE );/* +* Trigger all paths to force reconfiguration +*/ +pthread_cleanup_push (cleanup_lock, &vecs->lock); +lock (vecs->lock); +pthread_testcancel (); +trigger_devices (vecs); +lock_cleanup_pop (vecs->lock); + +/* * start threads */ if ((rc =pthread_create (&check_thr, &misc_attr, checkerloop, vecs))) {@@ -1927,6 +1951,8 @@ child (void * param) pid_rc =pidfile_create (DEFAULT_PIDFILE , daemon_pid);/* Ignore errors, we can live without */ +post_config_state (DAEMON_RUNNING ); +#ifdef USE_SYSTEMD sd_notify (0 ,"READY=1\nSTATUS=running" );#endif @@ -2129,7 +2155,9 @@ main (int argc, char *argv[]) conf->bindings_read_only =1 ;break ;default : - ; +fprintf (stderr,"Invalid argument '-%c'\n" , + optopt); +exit (1 ); } }if (optind < argc) {Signature · 1 line
1.8.4.5
Re: [PATCH 78/78] multipathd: trigger all devices on startup
Quoted 10 lines · Benjamin Marzinski
Quoted 6 lines · Hannes Reinecke
During startup multipathd might race with udev and device discovery. During that time any information from libudev might not be fully available, leading to spurious multipathd failures during startup. So instead of scanning all devices on our own we should just re-trigger the existing devices; with that we'll read _all_ devices via uevents during startup and avoid the race condition.
Should we also be sending change uevents for the dm devices?
No. We just need to retrigger the 'sd' devices, as with them we'll be reconfiguring the dm devices anyway.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
Hi,
This patchset is merged up to 75734af37267100778468319813cecc050befb1a : multipathd: push down lock in checkerloop().
Ben,
incidentally the find_multipath patches are in.
Best regards,
Christophe Varoqui
Quoted 202 lines · Hannes Reinecke
Hi Christophe,
here are the patches which I've queued up during SLES12 development. Most of them are (more or less) obvious bugfixes, but there are some things which would warrant a deeper review:
- device-mapper cookie handling: This _really_ is an arcane topic, and I would love to have someone from the device-mapper team to review it. I'm still facing some issues where udev emits a warning 'conflicting device node/dev/mapper/XXX..', so there might be some issue with it.
- Asynchronous configuration: On large systemsconfigure()as being called during multipathd startup might exceed the systemd job timeout, resulting in multipathd being killed by systemd. For that I've been implementing an udev retrigger method, which just issues an 'add' event for all available block devices.
- Related to the above: I've implemented an asynchronous 'reconfigure' CLI call, as for the same reason systemd might kill the multipathd during a 'restart' operation
- systemd/udev integration: This is _apain_. It took me months to come up with a working solution. _Especially_ as apparently RH is shipping an '11-dm-multipath.rules' udev ruleset for multipathd, which supposedly should be handling this.
It didn't do it for me, though, so I've attached my own version here. The udev integration I've been settling for works like this:
- 11-dm-mpath.rules checks if the device is eligible for multipathing For that I've implemented a new option '-u' which looks at the uevent variables _only_.
- if a device should be handled by multipathing the 'by-id' and 'by-uuid' persistent symlinks are _not_ generated, as this would trigger systemd to start scanning/mounting/lvm activation/whatnot; the 'by-id' and 'by-uuid' symlinks are only generated for the multipath devices themselves.
- I've added a new multipath.rules file which should be inserted between 55-scsi-sg3_id.rules and 58-scsi_sg3_symlink.rules. This will intercept the SCSI IDENT variables and blank them out if the (block) device is handled by multipathing.
- sysfs VPD page handling. I've planned to implement this as a direct replacement of the originalscsi_idprogram, but then it turned out that our WWID handling is actually too generic for that: Both getuid anduid_attributecan basically take _any_ value, so it's impossible to tell if the system uses the original (scsi_idcompatible) IDs or not. And as the VPD parsing code can only generatescsi_idcompatible IDs we might run into compability issues.
So it's disabled per default. I'd really would like to clean it up, as I suspect that about 95% of the installations are usingscsi_idcompatible IDs. But for this to happen we need to clarify/restrict the ID generation mechanism.As usual, comments and reviews are welcome.
The patchset is based upon the '
find_multipaths' patchset from Ben Marzinski; it can be retrieved fromgithub.com:/hreinecke/multipath-tools.git branch suse-resync
Cheers,
Hannes
Hannes Reinecke (78):
libmultipath: remove compilation warning indevmapper.c
mpath_persist: Do not callexit()from a shared library
libmultipath: filter for missing property inget_refwwid()Double uevent stacksize yet again
discovery: do not fail discovery on individual devices
libmultipath: Prefer deprecated 'getuid' callout
libmultipath: Skip paths with empty wwid
Make systemd installation path configurable
Add multipath rules for systemd support
Fixup multipathd.socket to resolve ordering dependeny Fixup dependencies in multipathd.service
multipathd: set correct PID when running in debug mode Do not print empty device strings during discovery kpartx.rules: do not call blkid
Use 'SCSI_IDENT_.*' as the default property whitelist Fixup wwid blacklist printing
Allow for empty path argument when printing information Disable reassign maps per default
multipathd: implement 'list path <path>' cli command Makechecker_put()andprio_put()idempotent Remove trailing linefeed from sysfs attributes
multipath: implement option '-u' for uevents Install multipath rule under '56-multipath.rules' multipath.rules: Whitelist devices multipath.rules: fixup race condition with systemd 11-dm-mpath.rules: Import blkid values if all paths are down kpartx.rules: Skip kpartx for multipath events
multipathd: handleDOMAP_RETRY
multipathd: cleanup foreground operation
Update hwtable for EMC XtremIO
multipath: check for running daemon when called with '-u' Revert 'returnPATH_DOWNfor quiesced paths'
Do not treat 'transport-offline' paths as 'offline' Check for validDM_DEVICE_INFObefore proceeding Separate out uevent parsing functions
Usepoll()when receiving uevents
mpath_persist: cleanup
kpartx: use standard 'major' and 'minor' macros
multipath: Use standard 'major' macro
Removesysfs_get_dev
Add paths with a size of '0' as 'ghost' paths Remove last argument fromverify_paths()
Fixup device-mapper 'cookie' handling
multipath: do not print state 'orphan' for option '-l' Return error when receiving CLI packet
Implement 'uxsock_timeout' keyword
Do not print empty multipaths section
multipathd: reload map if reinstate failed
multipathd: do not remove paths without uevents Allow zero-sized devices during configuration Reworkuev_add_path()
multipathd: Issue warning on CLI command timeout Usestrlen()when checking for valid wwid
multipathd: Use standard lists for CLI handling
uxlsnr: use typedef for trigger function
multipathd: lock cli client list
multipath: enable sync support
Removedm_udev_XXXwrapper functions
Revert toACT_RELOADin domap if the map exists
multipathd: use local variable for watchdog configuration Ignore devices whensysfs_get_tgt_nodenamefails Skip USB devices during discovery
Read wwid from sysfsvpg_pg83attribute
Assign local priority for NAA VPD descriptor
Use sysfs attributevpd_pg80to read serial number Update multipath.conf.5 to clarify wwid generation
libmultipath: Fall back toSG_IOif no UID could be assigned
libmultipath: unset 'uid_attribute' on failure Separate out vpd parsing functions
multipathd: useSG_IOas fallback to generate uid Do not automatically fall back to vpd uid generation
libmultipath: make vpd page0x80optional
multipathd: push down lock incheckerloop()
Allow specific CLI commands to run unlocked
Push down vector lock during uevent processing
multipathd: timeout CLI commands when waiting for lock
multipathd: asynchronous configuration
multipathd: trigger all devices on startupMakefile.inc | 8 +- kpartx/devmapper.c | 53 +++- kpartx/devmapper.h | 11 +- kpartx/kpartx.c | 35 +-- kpartx/kpartx.rules | 10 +- libmpathpersist/mpath_persist.c | 172 ++++++----- libmpathpersist/mpath_updatepr.c | 20 +- libmpathpersist/mpathpr.h | 3 +- libmultipath/blacklist.c | 8 +- libmultipath/blacklist.h | 2 +- libmultipath/checkers.c | 2 +- libmultipath/config.c | 2 + libmultipath/config.h | 8 +- libmultipath/configure.c | 82 ++++-- libmultipath/defaults.h | 4 +- libmultipath/devmapper.c | 77 +++-- libmultipath/devmapper.h | 4 +- libmultipath/dict.c | 33 +++ libmultipath/discovery.c | 613 ++++++++++++++++++++++++++++++++++----- libmultipath/discovery.h | 8 +- libmultipath/hwtable.c | 13 + libmultipath/list.h | 49 +++- libmultipath/lock.h | 4 +- libmultipath/log_pthread.c | 2 +- libmultipath/print.c | 20 +- libmultipath/prio.c | 4 +- libmultipath/propsel.c | 6 +- libmultipath/structs_vec.c | 29 +- libmultipath/structs_vec.h | 2 +- libmultipath/sysfs.c | 53 ++++ libmultipath/sysfs.h | 2 + libmultipath/uevent.c | 316 ++++++++++++-------- libmultipath/uxsock.c | 32 +- libmultipath/uxsock.h | 4 +- multipath.conf.defaults | 2 +- multipath/11-dm-mpath.rules | 8 +- multipath/Makefile | 2 + multipath/dev_t.h | 3 - multipath/main.c | 53 +++- multipath/multipath.8 | 6 +- multipath/multipath.conf.5 | 42 ++- multipath/multipath.rules | 14 + multipathd/cli.c | 47 ++- multipathd/cli.h | 4 +- multipathd/cli_handlers.c | 64 +++- multipathd/cli_handlers.h | 1 + multipathd/main.c | 400 ++++++++++++++++--------- multipathd/main.h | 2 + multipathd/multipathd.service | 4 +- multipathd/multipathd.socket | 8 +- multipathd/uxclnt.c | 26 +- multipathd/uxclnt.h | 2 +- multipathd/uxlsnr.c | 102 +++++-- multipathd/uxlsnr.h | 8 +- 54 files changed, 1834 insertions(+), 655 deletions(-) delete mode 100644 multipath/dev_t.h create mode 100644 multipath/multipath.rules--
1.8.4.5
Quoted 9 lines · Christophe Varoqui
Hi,
This patchset is merged up
to75734af37267100778468319813cecc050befb1a: multipathd: push down lock incheckerloop().Ben,
incidentally thefind_multipathpatches are in.
Thanks. I'll be working on the remaining ones to address the objections Ben raised.
Cheers,
Hannes
Signature · 10 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg) -- dm-devel mailing list [email protected] https://www.redhat.com/mailman/listinfo/dm-devel