Skip to contentSkip to conversation

More misc multipath patches

PATCH In review Benjamin Marzinski 2 participants 18 messages Dec 18, 2020 – Jan 27, 2021 Device mapper

lore
Share & tools
lore

Benjamin Marzinski

author maintainer
#0 lore raw

cover More misc multipath patches

The first two patches fix issues with mpathpersist. The second two deal with minor configuration file issues. The fifth patch deals with a problem with multipathd.service the could cause multipathd to not get stopped until after final.target was reached, causing shutdown errors. The last patch is just a doc clarification, inspired by customer questions.

Benjamin Marzinski (6):
mpathpersist: Fix Register and Ignore with 0x00 SARK
mpathpersist: update prkeys file on changing registrations
libmultipath: warn about missing braces at end of multipath.conf
libmultipath: ignore multipaths sections without wwid option
multipathd: Fix multipathd stopping on shutdown multipath.conf.5: Improve checker_timeout description

 libmpathpersist/mpath_persist.c | 10 ++++++----
 libmultipath/config.c           |  7 +++++++
 libmultipath/parser.c           |  5 +++--
 multipath/multipath.conf.5      |  9 +++++++--
 multipathd/multipathd.service   |  2 +-
 5 files changed, 24 insertions(+), 9 deletions(-)
Signature · 6 lines
2.17.2

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
author maintainer
#1 lore raw

PATCH 1/6 mpathpersist: Fix Register and Ignore with 0x00 SARK

When the Register and Ignore command is run with sg_persist, if a 0x00 Service Action Reservation Key is given or the --param-sark option is not used at all, sg_persist will clear the registration. mpathpersist will fail with an error. This patch fixes mpathpersist to work like sg_persist in this case.

---
 libmpathpersist/mpath_persist.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
libmpathpersist/mpath_persist.c +2 -1
index 79322e86..41789c46 100644
@@ -304,7 +304,8 @@ int __mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope,
 	}
 
 	if (memcmp(paramp->key, &mpp->reservation_key, 8) &&
-	    memcmp(paramp->sa_key, &mpp->reservation_key, 8)) {
+	    memcmp(paramp->sa_key, &mpp->reservation_key, 8) &&
+	    (prkey || rq_servact != MPATH_PROUT_REG_IGN_SA)) {
 		condlog(0, "%s: configured reservation key doesn't match: 0x%" PRIx64, alias, get_be64(mpp->reservation_key));
 		ret = MPATH_PR_SYNTAX_ERROR;
 		goto out1;
Signature · 6 lines
2.17.2

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Reviewed-by on v1 1/6
#2 lore raw

Re: [dm-devel] [PATCH 1/6] mpathpersist: Fix Register and Ignore with 0x00 SARK

Quoted 9 lines · Benjamin Marzinski

When the Register and Ignore command is run with sg_persist, if a 0x00
Service Action Reservation Key is given or the --param-sark option is not used at all, sg_persist will clear the
registration. mpathpersist
will fail with an error. This patch fixes mpathpersist to work like sg_persist in this case.

Signature · 11 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer



--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
author maintainer
#3 lore raw

PATCH 2/6 mpathpersist: update prkeys file on changing registrations

In reply to Benjamin Marzinski

When the "reservation_key" option is set to "file" and Register command is run with both the current Reservation Key and a new Service Action Reservation Key, mpathpersist will change the registration, but will not update the prkeys file. This means that future paths that come online will not be able to register, since multipathd is still using the old reservation key. Fix this.

---
 libmpathpersist/mpath_persist.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
libmpathpersist/mpath_persist.c +4 -3
index 41789c46..08077936 100644
@@ -290,9 +290,10 @@ int __mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope,
 
 	memcpy(&prkey, paramp->sa_key, 8);
 	if (mpp->prkey_source == PRKEY_SOURCE_FILE && prkey &&
-	    ((!get_be64(mpp->reservation_key) &&
-	      rq_servact == MPATH_PROUT_REG_SA) ||
-	     rq_servact == MPATH_PROUT_REG_IGN_SA)) {
+	    (rq_servact == MPATH_PROUT_REG_IGN_SA ||
+	     (rq_servact == MPATH_PROUT_REG_SA &&
+	      (!get_be64(mpp->reservation_key) ||
+	       memcmp(paramp->key, &mpp->reservation_key, 8) == 0)))) {
 		memcpy(&mpp->reservation_key, paramp->sa_key, 8);
 		if (update_prkey_flags(alias, get_be64(mpp->reservation_key),
 				       paramp->sa_flags)) {
Signature · 6 lines
2.17.2

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Reviewed-by on v1 2/6
#4 lore raw

Re: [dm-devel] [PATCH 2/6] mpathpersist: update prkeys file on changing registrations

Quoted 10 lines · Benjamin Marzinski

When the "reservation_key" option is set to "file" and Register command
is run with both the current Reservation Key and a new Service Action Reservation Key, mpathpersist will change the registration, but will not
update the prkeys file. This means that future paths that come online will not be able to register, since multipathd is still using the old reservation key. Fix this.

Signature · 11 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer



--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
author maintainer
#5 lore raw

PATCH 3/6 libmultipath: warn about missing braces at end of multipath.conf

In reply to Benjamin Marzinski

Multipath doesn't warn when multipath.conf is missing closing braces at the end of the file. This has confused people about the correct config file syntax, so add a warning.

---
 libmultipath/parser.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
libmultipath/parser.c +3 -2
index 163ffbc9..c70243c3 100644
@@ -537,7 +537,7 @@ process_stream(struct config *conf, FILE *stream, vector keywords,
 		if (!strcmp(str, EOB)) {
 			if (kw_level > 0) {
 				free_strvec(strvec);
-				break;
+				goto out;
 			}
 			condlog(0, "unmatched '%s' at line %d of %s",
 				EOB, line_nr, file);
@@ -576,7 +576,8 @@ process_stream(struct config *conf, FILE *stream, vector keywords,
 
 		free_strvec(strvec);
 	}
-
+	if (kw_level == 1)
+		condlog(1, "missing '%s' at end of %s", EOB, file);
 out:
 	FREE(buf);
 	free_uniques(uniques);
Signature · 6 lines
2.17.2

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Reviewed-by on v1 3/6
#6 lore raw

Re: [dm-devel] [PATCH 3/6] libmultipath: warn about missing braces at end of multipath.conf

Quoted 7 lines · Benjamin Marzinski

Multipath doesn't warn when multipath.conf is missing closing braces at
the end of the file. This has confused people about the correct config
file syntax, so add a warning.

Signature · 11 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer



--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
author maintainer
#7 lore raw

PATCH 4/6 libmultipath: ignore multipaths sections without wwid option

In reply to Benjamin Marzinski

"multipathd show config local" was crashing in find_mp_by_wwid() if the multipath configuration included a multipaths section that did not set a wwid option. There is no reason to keep a mpentry that didn't set its wwid. Remove it in merge_mptable().

---
 libmultipath/config.c | 7 +++++++
 1 file changed, 7 insertions(+)
libmultipath/config.c +7 -0
index 9f3cb38d..a643703e 100644
@@ -509,6 +509,13 @@ void merge_mptable(vector mptable)
 	int i, j;
 
 	vector_foreach_slot(mptable, mp1, i) {
+		/* drop invalid multipath configs */
+		if (!mp1->wwid) {
+			condlog(0, "multipaths config section missing wwid");
+			vector_del_slot(mptable, i--);
+			free_mpe(mp1);
+			continue;
+		}
 		j = i + 1;
 		vector_foreach_slot_after(mptable, mp2, j) {
 			if (strcmp(mp1->wwid, mp2->wwid))
Signature · 6 lines
2.17.2

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
#8 lore raw

Re: [dm-devel] [PATCH 4/6] libmultipath: ignore multipaths sections without wwid option

Quoted 28 lines · Benjamin Marzinski

"multipathd show config local" was crashing in find_mp_by_wwid() if the multipath configuration included a multipaths section that did not set a wwid option. There is no reason to keep a mpentry that didn't set its wwid. Remove it in merge_mptable().

---
 libmultipath/config.c | 7 +++++++
 1 file changed, 7 insertions(+)
libmultipath/config.c +7 -0
index 9f3cb38d..a643703e 100644
@@ -509,6 +509,13 @@ void merge_mptable(vector mptable)
 	int i, j;
 
 	vector_foreach_slot(mptable, mp1, i) {
+		/* drop invalid multipath configs */
+		if (!mp1->wwid) {
+			condlog(0, "multipaths config section missing
 wwid");
+			vector_del_slot(mptable, i--);
+			free_mpe(mp1);
+			continue;
+		}
 		j = i + 1;
 		vector_foreach_slot_after(mptable, mp2, j) {
Quoted 1 line

if (strcmp(mp1->wwid, mp2->wwid))

Wouldn't you need to check mp2->wwid, too?

Martin

Signature · 11 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer



--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
author maintainer
#9 lore raw

Re: [dm-devel] [PATCH 4/6] libmultipath: ignore multipaths sections without wwid option

Quoted 18 lines · Martin Wilck
Quoted 17 lines · Benjamin Marzinski

"multipathd show config local" was crashing in find_mp_by_wwid() if the multipath configuration included a multipaths section that did not set a wwid option. There is no reason to keep a mpentry that didn't set its wwid. Remove it in merge_mptable().

---
 libmultipath/config.c | 7 +++++++
 1 file changed, 7 insertions(+)
libmultipath/config.c +0 -0
index 9f3cb38d..a643703e 100644
@@ -509,6 +509,13 @@ void merge_mptable(vector mptable)
 	int i, j;
 
Quoted 14 lines
Quoted 11 lines
libmultipath/config.c +7 -0
 	vector_foreach_slot(mptable, mp1, i) {
+		/* drop invalid multipath configs */
+		if (!mp1->wwid) {
+			condlog(0, "multipaths config section missing
 wwid");
+			vector_del_slot(mptable, i--);
+			free_mpe(mp1);
+			continue;
+		}
 		j = i + 1;
 		vector_foreach_slot_after(mptable, mp2, j) {
Quoted 1 line

if (strcmp(mp1->wwid, mp2->wwid))

Wouldn't you need to check mp2->wwid, too?

Oops. Yeah, strcmp() is undefined with NULL, so that needs a check. BTW, I noticed that added this one to you upstream-queue branch, even though you didn't ack it, while not adding patch 5/6 (multipathd: Fix multipathd stopping on shutdown), even though you did ack it. Was this intentional?

At any rate, since this is already in upstream-queue, I'll just post a follow-up that checks mp2->wwid, before calling strcmp()

-Ben

Quoted 8 lines

Martin

Signature · 4 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

author maintainer
#10 lore raw

PATCH 5/6 multipathd: Fix multipathd stopping on shutdown

In reply to Benjamin Marzinski

According to man "systemd.special"

"shutdown.target: ... Services that shall be terminated on system shutdown shall add Conflicts= and Before= dependencies to this unit for their service unit, which is implicitly done when DefaultDependencies=yes is set (the default)."

multipathd.service sets DefaultDependencies=no and includes the Conflits= dependency, but not the Before= one. This can cause multipathd to continue running past when it is supposed to during shutdown.

---
 multipathd/multipathd.service | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
multipathd/multipathd.service +1 -1
index ba24983e..7d547fa7 100644
@@ -2,7 +2,7 @@
 Description=Device-Mapper Multipath Device Controller
 Wants=systemd-udev-trigger.service systemd-udev-settle.service
 Before=iscsi.service iscsid.service lvm2-activation-early.service
-Before=local-fs-pre.target blk-availability.service
+Before=local-fs-pre.target blk-availability.service shutdown.target
 After=multipathd.socket systemd-udev-trigger.service systemd-udev-settle.service
 DefaultDependencies=no
 Conflicts=shutdown.target
Signature · 6 lines
2.17.2

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Reviewed-by on v1 5/6
#11 lore raw

Re: [dm-devel] [PATCH 5/6] multipathd: Fix multipathd stopping on shutdown

Quoted 8 lines · Benjamin Marzinski

According to man "systemd.special"

"shutdown.target: ... Services that shall be terminated on system shutdown shall add Conflicts= and Before= dependencies to this unit for
their service unit, which is implicitly done when DefaultDependencies=yes is set (the default)."

Quoted 6 lines

multipathd.service sets DefaultDependencies=no and includes the Conflits= dependency, but not the Before= one. This can cause multipathd
to continue running past when it is supposed to during shutdown.

Signature · 11 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer



--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
author maintainer
#12 lore raw

Re: [dm-devel] [PATCH 5/6] multipathd: Fix multipathd stopping on shutdown

Quoted 9 lines · Martin Wilck
Quoted 8 lines · Benjamin Marzinski

According to man "systemd.special"

"shutdown.target: ... Services that shall be terminated on system shutdown shall add Conflicts= and Before= dependencies to this unit for
their service unit, which is implicitly done when DefaultDependencies=yes is set (the default)."

Quoted 8 lines
Quoted 6 lines

multipathd.service sets DefaultDependencies=no and includes the Conflits= dependency, but not the Before= one. This can cause multipathd
to continue running past when it is supposed to during shutdown.

Martin, I noticed that this commit didn't make it into your queue branch. I assume it's just an oversight. Right?

-Ben

Quoted 7 lines
Signature · 5 lines
-- 
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

#13 lore raw

Re: [dm-devel] [PATCH 5/6] multipathd: Fix multipathd stopping on shutdown

Quoted 3 lines · Benjamin Marzinski

Martin, I noticed that this commit didn't make it into your queue branch. I assume it's just an oversight. Right?

Yes, I'm sorry. Fixed now.

Cheers,
Martin

Signature · 10 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix Imendörffer



--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
author maintainer
#14 lore raw

PATCH 6/6 multipath.conf.5: Improve checker_timeout description

In reply to Benjamin Marzinski

I was asked to explain how checker_timeout works for checkers like directio, that don't issue scsi commands with an explicit timeout

---
 multipath/multipath.conf.5 | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
multipath/multipath.conf.5 +7 -2
index ea66a01e..c7c4184b 100644
@@ -639,8 +639,13 @@ The default is: \fBno\fR
 .
 .TP
 .B checker_timeout
-Specify the timeout to use for path checkers and prioritizers that issue SCSI
-commands with an explicit timeout, in seconds.
+Specify the timeout to use for path checkers and prioritizers, in seconds.
+Only prioritizers that issue scsi commands use checker_timeout. Checkers
+that support an asynchronous mode (\fItur\fR and \fIdirectio\fR), will
+return shortly after being called by multipathd, regardless of whether the
+storage array responds. If the storage array hasn't responded, mulitpathd will
+check for a response every second, until \fIchecker_timeout\fR seconds have
+elapsed.
 .RS
 .TP
 The default is: in \fB/sys/block/sd<x>/device/timeout\fR
Signature · 6 lines
2.17.2

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
#15 lore raw

Re: [dm-devel] [PATCH 6/6] multipath.conf.5: Improve checker_timeout description

Quoted 21 lines · Benjamin Marzinski

I was asked to explain how checker_timeout works for checkers like directio, that don't issue scsi commands with an explicit timeout

---
 multipath/multipath.conf.5 | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
multipath/multipath.conf.5 +0 -1
index ea66a01e..c7c4184b 100644
@@ -639,8 +639,13 @@ The default is: \fBno\fR
 .
 .TP
 .B checker_timeout
-Specify the timeout to use for path checkers and prioritizers that

issue SCSI

multipath/multipath.conf.5 +1 -1
-commands with an explicit timeout, in seconds.
+Specify the timeout to use for path checkers and prioritizers, in

seconds.

Quoted 8 lines

+Only prioritizers that issue scsi commands use checker_timeout. Checkers
+that support an asynchronous mode (\fItur\fR and \fIdirectio\fR), will
+return shortly after being called by multipathd, regardless of whether the
+storage array responds. If the storage array hasn't responded, mulitpathd will

typo

Quoted 3 lines

+check for a response every second, until \fIchecker_timeout\fR seconds have
+elapsed.

This is a bit confusing IMHO. Most importantly, checkers will consider a path down if it doesn't respond to the checker command after the given timeout. The async behavior doesn't fit too well into this section. Maybe we should mention and explain the async property in the path_checkers section, and only refer to that here.

(Btw is the directio checker still deprecated after all the work you recently put into it? The man page still says so).

Regards,
Martin

Signature · 11 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer



--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
author maintainer
#16 lore raw

Re: [dm-devel] [PATCH 6/6] multipath.conf.5: Improve checker_timeout description

Quoted 33 lines · Martin Wilck
Quoted 21 lines · Benjamin Marzinski

I was asked to explain how checker_timeout works for checkers like directio, that don't issue scsi commands with an explicit timeout

---
 multipath/multipath.conf.5 | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
multipath/multipath.conf.5 +0 -1
index ea66a01e..c7c4184b 100644
@@ -639,8 +639,13 @@ The default is: \fBno\fR
 .
 .TP
 .B checker_timeout
-Specify the timeout to use for path checkers and prioritizers that

issue SCSI

multipath/multipath.conf.5 +1 -1
-commands with an explicit timeout, in seconds.
+Specify the timeout to use for path checkers and prioritizers, in

seconds.

Quoted 8 lines

+Only prioritizers that issue scsi commands use checker_timeout. Checkers
+that support an asynchronous mode (\fItur\fR and \fIdirectio\fR), will
+return shortly after being called by multipathd, regardless of whether the
+storage array responds. If the storage array hasn't responded, mulitpathd will

typo

Quoted 9 lines
Quoted 3 lines

+check for a response every second, until \fIchecker_timeout\fR seconds have
+elapsed.

This is a bit confusing IMHO. Most importantly, checkers will consider a path down if it doesn't respond to the checker command after the given timeout. The async behavior doesn't fit too well into this section. Maybe we should mention and explain the async property in the path_checkers section, and only refer to that here.

Sounds reasonable.

Quoted 2 lines

(Btw is the directio checker still deprecated after all the work you recently put into it? The man page still says so).

No. I'll change that. There are times when devices claim to be ready with the tur checker, when in truth, IO to them will fail. In these cases, the directio checker is the best way to avoid having paths bouncing up and down.

-Ben

Quoted 10 lines

Regards,
Martin

Signature · 4 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE  Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix
Imendörffer

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

#17 lore raw

Re: [dm-devel] [PATCH 6/6] multipath.conf.5: Improve checker_timeout description

Quoted 53 lines · Benjamin Marzinski
Quoted 38 lines · Martin Wilck
Quoted 26 lines · Benjamin Marzinski

I was asked to explain how checker_timeout works for checkers like
directio, that don't issue scsi commands with an explicit timeout

---
 multipath/multipath.conf.5 | 9 +++++++--

 1 file changed, 7 insertions(+), 2 deletions(-)
multipath/multipath.conf.5 +0 -0
index ea66a01e..c7c4184b 100644
@@ -639,8 +639,13 @@ The default is: \fBno\fR

 .
 .TP
 .B checker_timeout
-Specify the timeout to use for path checkers and prioritizers that
issue SCSI

multipath/multipath.conf.5 +1 -1
-commands with an explicit timeout, in seconds.
+Specify the timeout to use for path checkers and prioritizers,

in
seconds.
+Only prioritizers that issue scsi commands use checker_timeout.

Quoted 8 lines

Checkers
+that support an asynchronous mode (\fItur\fR and \fIdirectio\fR),
will
+return shortly after being called by multipathd, regardless of whether the
+storage array responds. If the storage array hasn't responded, mulitpathd will

typo

Quoted 11 lines
Quoted 3 lines

+check for a response every second, until \fIchecker_timeout\fR seconds have
+elapsed.

This is a bit confusing IMHO. Most importantly, checkers will consider
a path down if it doesn't respond to the checker command after the given timeout. The async behavior doesn't fit too well into this section. Maybe we should mention and explain the async property in the
path_checkers section, and only refer to that here.

Sounds reasonable.

Quoted 8 lines
Quoted 3 lines

(Btw is the directio checker still deprecated after all the work you
recently put into it? The man page still says so).

No. I'll change that.  There are times when devices claim to be ready with the tur checker, when in truth, IO to them will fail.  In these cases, the directio checker is the best way to avoid having paths bouncing up and down.

Right. I recently had one such case with persistent reservations. SPC-4 mandates that the status of TUR commands is independent of PR status (TUR is always "allowed"), while obviously ordinary IO would fail if the active PR exclude the current host. That basically makes the TUR checker inappropriate as soon as PRs on SPC-4 compliant devices are in use.

As we have support for PR already, I wonder if we could/should extend the TUR checker to take this into account.

Cheers,
Martin

Signature · 10 lines
Dr. Martin Wilck <[email protected]>, Tel. +49 (0)911 74053 2107
SUSE Software Solutions Germany GmbH
HRB 36809, AG Nürnberg GF: Felix Imendörffer



--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel