Skip to contentSkip to conversation

libmultipath: fallback to const prio if getprio() fails

PATCH In review Martin Wilck 3 participants 6 messages Mar 20, 2018 – Mar 22, 2018 Device mapper

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 libmultipath: fallback to const prio if getprio() fails

Some storage controllers don't support ALUA even though they have hwtable entries telling so. One such example is the IBM IPR SAS controller found on certain IBM Power systems. IBM has confirmed that reliable detection of ALUA support for this controller by vendor/model/rev is impossible.

For such cases, instead of simply failing to setup multipath, fall back to const prio.

---
 libmultipath/discovery.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
libmultipath/discovery.c +20 -3
index 9f2a9c907914..a51eb0b82f53 100644
@@ -1637,9 +1637,26 @@ get_prio (struct path * pp)
 	pp->priority = prio_getprio(p, pp, conf->checker_timeout);
 	put_multipath_config(conf);
 	if (pp->priority < 0) {
-		condlog(3, "%s: %s prio error", pp->dev, prio_name(p));
-		pp->priority = PRIO_UNDEF;
-		return 1;
+		const char origin[] = "(fallback after error)";
+
+		if (strcmp(prio_name(p), DEFAULT_PRIO)) {
+			conf = get_multipath_config();
+			prio_get(conf->multipath_dir, p, DEFAULT_PRIO,
+				 DEFAULT_PRIO_ARGS);
+			pp->priority = prio_getprio(p, pp,
+						    conf->checker_timeout);
+			put_multipath_config(conf);
+		}
+
+		if (pp->priority < 0) {
+			condlog(1, "%s: %s prio error", pp->dev, prio_name(p));
+			pp->priority = PRIO_UNDEF;
+			return 1;
+		}
+
+		condlog(3, "%s: new prio = %s %s", pp->dev, prio_name(p), origin);
+		condlog(3, "%s: new prio args = \"%s\" %s", pp->dev, prio_args(p),
+			origin);
 	}
 	condlog(3, "%s: %s prio = %u",
 		pp->dev, prio_name(p), pp->priority);
Signature · 1 line
2.16.1
#1 lore raw

PATCH 2/2 multipathd: handle errors in uxlsnr as fatal

The ppoll() calls of the uxlsnr thread are vital for proper functioning of multipathd. If the uxlsnr thread can't open the socket or fails to call ppoll() for other reasons, quit the daemon. If we don't do that, multipathd may hang in a state where it can't be terminated any more, because the uxlsnr thread is responsible for handling all signals. This happens e.g. if systemd's multipathd.socket is running in and multipathd is started from outside systemd.

24f2844 "multipathd: fix signal blocking logic" has made this problem more severe. Before that patch, the signals weren't actually blocked in any thread. That's not to say 24f2844 was wrong. I still think it's correct, we just need this one on top.

---
 multipathd/uxlsnr.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
multipathd/uxlsnr.c +3 -2
index cdafd82943e7..6f666663fc6f 100644
@@ -178,7 +178,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 
 	if (ux_sock == -1) {
 		condlog(1, "could not create uxsock: %d", errno);
-		return NULL;
+		exit_daemon();
 	}
 
 	pthread_cleanup_push(uxsock_cleanup, (void *)ux_sock);
@@ -187,7 +187,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 	polls = (struct pollfd *)MALLOC((MIN_POLLS + 1) * sizeof(struct pollfd));
 	if (!polls) {
 		condlog(0, "uxsock: failed to allocate poll fds");
-		return NULL;
+		exit_daemon();
 	}
 	sigfillset(&mask);
 	sigdelset(&mask, SIGINT);
@@ -249,6 +249,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 
 			/* something went badly wrong! */
 			condlog(0, "uxsock: poll failed with %d", errno);
+			exit_daemon();
 			break;
 		}
 
Signature · 1 line
2.16.1
#2 lore raw

Re: [PATCH 2/2] multipathd: handle errors in uxlsnr as fatal

Quoted 41 lines · Martin Wilck

The ppoll() calls of the uxlsnr thread are vital for proper functioning of multipathd. If the uxlsnr thread can't open the socket or fails to call ppoll() for other reasons, quit the daemon. If we don't do that, multipathd may hang in a state where it can't be terminated any more, because the uxlsnr thread is responsible for handling all signals. This happens e.g. if systemd's multipathd.socket is running in and multipathd is started from outside systemd.

24f2844 "multipathd: fix signal blocking logic" has made this problem more severe. Before that patch, the signals weren't actually blocked in any thread. That's not to say 24f2844 was wrong. I still think it's correct, we just need this one on top.

---
  multipathd/uxlsnr.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
multipathd/uxlsnr.c +2 -2
index cdafd82943e7..6f666663fc6f 100644
@@ -178,7 +178,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 
  	if (ux_sock == -1) {
  		condlog(1, "could not create uxsock: %d", errno);
-		return NULL;
+		exit_daemon();
  	}
 
  	pthread_cleanup_push(uxsock_cleanup, (void *)ux_sock);
@@ -187,7 +187,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
  	polls = (struct pollfd *)MALLOC((MIN_POLLS + 1) * sizeof(struct pollfd));
  	if (!polls) {
  		condlog(0, "uxsock: failed to allocate poll fds");
-		return NULL;
+		exit_daemon();
  	}
  	sigfillset(&mask);
  	sigdelset(&mask, SIGINT);

multipathd/uxlsnr.c void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data) · line 254

Quoted 6 lines
multipathd/uxlsnr.c +1 -0
@@ -249,6 +249,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 
  			/* something went badly wrong! */
  			condlog(0, "uxsock: poll failed with %d", errno);
+			exit_daemon();
  			break;
  		}

Hi Martin,

Your analysis is reasonable. It is necessary to deal with fatal error not only to return, if not doing this multipathd can't exit normally and multipathd commands can't work any more. I think your patch is OK, but I have some ideas inspired by your patch.
Calling exit_daemon() is to shut down the multipathd, relay on the outside to pull multipathd again. Is there a function can be use to deal with fatal error? Its function are close the socket(if create successfully before) and create a new socket to make uxlsnr thread work properly again or continue to create uxsocket? This function actually is try to repair those errors.
It just an idea, maybe not quite right.

Regards,
Chongyun

#3 lore raw

Re: [PATCH 2/2] multipathd: handle errors in uxlsnr as fatal

Quoted 66 lines · Chongyun Wu
Quoted 61 lines · Martin Wilck

The ppoll() calls of the uxlsnr thread are vital for proper functioning of
multipathd. If the uxlsnr thread can't open the socket or fails to call ppoll()
for other reasons, quit the daemon. If we don't do that, multipathd may
hang in a state where it can't be terminated any more, because the uxlsnr
thread is responsible for handling all signals. This happens e.g. if
systemd's multipathd.socket is running in and multipathd is started from
outside systemd.

24f2844 "multipathd: fix signal blocking logic" has made this problem more
severe. Before that patch, the signals weren't actually blocked in any thread.
That's not to say 24f2844 was wrong. I still think it's correct, we just
need this one on top.

---
  multipathd/uxlsnr.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
multipathd/uxlsnr.c +1 -1
index cdafd82943e7..6f666663fc6f 100644
@@ -178,7 +178,7 @@ void * uxsock_listen(uxsock_trigger_fn
 uxsock_trigger, void * trigger_data)
 
  	if (ux_sock == -1) {
  		condlog(1, "could not create uxsock: %d", errno);
-		return NULL;
+		exit_daemon();
  	}
 
  	pthread_cleanup_push(uxsock_cleanup, (void *)ux_sock);
multipathd/uxlsnr.c +1 -1
@@ -187,7 +187,7 @@ void * uxsock_listen(uxsock_trigger_fn
 uxsock_trigger, void * trigger_data)
  	polls = (struct pollfd *)MALLOC((MIN_POLLS + 1) *
 sizeof(struct pollfd));
  	if (!polls) {
  		condlog(0, "uxsock: failed to allocate poll
 fds");
-		return NULL;
+		exit_daemon();
  	}
  	sigfillset(&mask);
  	sigdelset(&mask, SIGINT);
multipathd/uxlsnr.c +1 -0
@@ -249,6 +249,7 @@ void * uxsock_listen(uxsock_trigger_fn
 uxsock_trigger, void * trigger_data)
 
  			/* something went badly wrong! */
  			condlog(0, "uxsock: poll failed with %d",
 errno);
+			exit_daemon();
  			break;
Quoted 1 line
  		}

Hi Martin,

Quoted 11 lines

Your analysis is reasonable. It is necessary to deal with fatal error
not only to return, if not doing this multipathd can't exit normally and
multipathd commands can't work any more. I think your patch is OK, but I
have some ideas inspired by your patch.
Calling exit_daemon() is to shut down the multipathd, relay on the outside to pull multipathd again. Is there a function can be use to deal
with fatal error?

I don't think so, that's why I call the error "fatal". But feel free to come up with a more intelligent solution. In the only case with practical relevance I've seen so far (socket can't be opened because it's open in systemd), it's sufficient that multipathd is started via systemd (this should be done in production anyway), or that the admin runs "systemctl stop multipathd.socket" before starting the daemon manually.

In general, I don't think it makes sense to try and be overly smart. Errors that prevent the ux socket from being opened (with the mentioned exception) are likely to be so severe that any attempts to work around them will probably fail as well.

Martin

Signature · 8 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
maintainer Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] multipathd: handle errors in uxlsnr as fatal

In reply to Martin Wilck
Quoted 8 lines · Martin Wilck

The ppoll() calls of the uxlsnr thread are vital for proper functioning of multipathd. If the uxlsnr thread can't open the socket or fails to call ppoll() for other reasons, quit the daemon. If we don't do that, multipathd may hang in a state where it can't be terminated any more, because the uxlsnr thread is responsible for handling all signals. This happens e.g. if systemd's multipathd.socket is running in and multipathd is started from outside systemd.

Quoted 5 lines

24f2844 "multipathd: fix signal blocking logic" has made this problem more severe. Before that patch, the signals weren't actually blocked in any thread. That's not to say 24f2844 was wrong. I still think it's correct, we just need this one on top.

Quoted 37 lines
---
 multipathd/uxlsnr.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
multipathd/uxlsnr.c +3 -2
index cdafd82943e7..6f666663fc6f 100644
@@ -178,7 +178,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 
 	if (ux_sock == -1) {
 		condlog(1, "could not create uxsock: %d", errno);
-		return NULL;
+		exit_daemon();
 	}
 
 	pthread_cleanup_push(uxsock_cleanup, (void *)ux_sock);
@@ -187,7 +187,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 	polls = (struct pollfd *)MALLOC((MIN_POLLS + 1) * sizeof(struct pollfd));
 	if (!polls) {
 		condlog(0, "uxsock: failed to allocate poll fds");
-		return NULL;
+		exit_daemon();
 	}
 	sigfillset(&mask);
 	sigdelset(&mask, SIGINT);
@@ -249,6 +249,7 @@ void * uxsock_listen(uxsock_trigger_fn uxsock_trigger, void * trigger_data)
 
 			/* something went badly wrong! */
 			condlog(0, "uxsock: poll failed with %d", errno);
+			exit_daemon();
 			break;
 		}
 
Signature · 1 line
2.16.1
#5 lore raw
In reply to Martin Wilck
Quoted 7 lines · Martin Wilck

Some storage controllers don't support ALUA even though they have hwtable entries telling so. One such example is the IBM IPR SAS controller
found on certain IBM Power systems. IBM has confirmed that reliable detection of ALUA support for this controller by vendor/model/rev is impossible.

Quoted 5 lines

For such cases, instead of simply failing to setup multipath, fall back
to const prio.

NACK to self. I just realized that this won't work, because we might be resetting e.g. alua prioritizer to "const" because of a temporary problem. So this needs more thought.

Martin

Signature · 8 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel