nvme_fc auto-connect scripts

From: James Smart <[email protected]>

Sagi recently posted a "one-shot" call to nvme-connect via a systemd script. FC has no problems operating with it assuming all the required connect fields are present (host_traddr required).

However FC is a bit more dynamic, and we've built up scripts that can be triggered whenever a new FC remote port with a discovery subsytem is detected. Whenever such a port is detected (by registration of the remoteport with the nvme_fc transport by the lldd), the transport generates a udev event. A udev rule is created that parses the event information into the FC remote port (traddr) that supports a discovery subsystem and the FC host port that is connected to the FC remote port (host_traddr). The rule then invokes an instance-based systemd service to perform a "nvme connect-all" with the traddr and host-traddr info.

To resolve some issues seen by discovery of remote ports before the udev service was running we've added an additional system service nvmefc-boot-connections.service which will retrigger the event by writint 'add' into the 'nvme_discovery' sysfs attribute.

This patch has been adapted from an earlier patch from James Smart.

---
 Makefile                                |  9 +++++++++
 nvme-fc/70-nvmefc-autoconnect.rules     |  8 ++++++++
 nvme-fc/nvmefc-boot-connections.service |  9 +++++++++
 nvme-fc/nvmefc-connect at .service         | 14 ++++++++++++++
 4 files changed, 40 insertions(+)
 create mode 100644 nvme-fc/70-nvmefc-autoconnect.rules
 create mode 100644 nvme-fc/nvmefc-boot-connections.service
 create mode 100644 nvme-fc/nvmefc-connect at .service

index 70541cb..8b9d0e1 100644
@@ -8,6 +8,8 @@ DESTDIR =
 PREFIX ?= /usr/local
 SYSCONFDIR = /etc
 SBINDIR = $(PREFIX)/sbin
+SYSTEMDDIR = /usr/lib/systemd
+UDEVDIR = /usr/lib/udev
 LIB_DEPENDS =
 
 ifeq ($(LIBUUID),0)

@@ -86,6 +88,13 @@ install-bash-completion:
 	$(INSTALL) -d $(DESTDIR)$(PREFIX)/share/bash-completion/completions
 	$(INSTALL) -m 644 -T ./completions/bash-nvme-completion.sh $(DESTDIR)$(PREFIX)/share/bash-completion/completions/nvme
 
+install-udev:
+	$(INSTALL) -d $(DESTDIR)$(SYSTEMDDIR)/system
+	$(INSTALL) -m 644 ./nvme-fc/nvme-fc-connect at .service $(DESTDIR)$(SYSTEMDDIR)/system
+	$(INSTALL) -m 644 ./nvme-fc/nvmefc-boot-connections.service $(DESTDIR)$(SYSTEMDIR)/system
+	$(INSTALL) -d $(DESTDIR)$(UDEVDIR)/rules.d
+	$(INSTALL) -m 644 ./nvme-fc/70-nvme-fc-autoconnect.rules $(DESTDIR)$(UDEVDIR)/rules.d
+
 install: install-bin install-man install-bash-completion
 
 nvme.spec: nvme.spec.in NVME-VERSION-FILE

new file mode 100644
index 0000000..aa4f9bf
@@ -0,0 +1,8 @@
+#
+# nvme_fc: udev event to automatically scan (via discovery controller)
+#   new FC nvme remote ports and auto-connect to the subsystems they report.
+#
+
+ACTION=="change", SUBSYSTEM=="fc", ENV{FC_EVENT}=="nvmediscovery", \
+  ENV{NVMEFC_HOST_TRADDR}=="*",  ENV{NVMEFC_TRADDR}=="*", \
+  RUN+="/usr/bin/systemctl --no-block start nvmefc-connect at --host-traddr=$env{NVMEFC_HOST_TRADDR}\\x20--traddr=$env{NVMEFC_TRADDR}.service"

new file mode 100644
index 0000000..aa35c15
@@ -0,0 +1,9 @@
+[Unit]
+Description=Auto-connect to subsystems on FC-NVME devices during boot
+
+[Service]
+Type=oneshot
+ExecStart=/bin/sh -c "echo add > /sys/class/fc/fc_udev_device/nvme_discovery"
+
+[Install]
+WantedBy=default.target

new file mode 100644
index 0000000..c2370d6
@@ -0,0 +1,14 @@
+#
+# Unit file used by 70-nvmefc-autoconnect.rules.
+#
+
+[Unit]
+Description=Auto-connect to subsystems on FC-NVME devices
+After=syslog.target
+
+[Service]
+Type=oneshot
+ExecStart=/bin/sh -c "/usr/sbin/nvme connect-all --transport=fc `/usr/bin/echo -e '%i'`"
+
+[Install]
+WantedBy=default.target

2.13.7

I think that this is a good approach. Can we add a preparatory patch that will allow nvme-cli to accept nil/none arguments? This way we can have the kernel shoot consistent udev events where parameters can stay don't-care and we can have the same rule...

If nvme-cli can accept a command in the form:
nvme connect-all --traddr=<a> --host-traddr=<b> --trsvcid=none

I think we can consolidate a bit more (it will be a different device because for IP stuff the device will be the controller char device).

Perhaps we should change the name 70-nvmefc-autoconnect.rules to 70-nvmf-autoconnect.rules that will contain rules fc and IP in the same rule file...

Good point, ans something which I had been thinking about, too. But really it only makes sense if these scripts are supposed to work on different fabrics.

Of course we can make it more general.

It was just that buy-in from other subsystems into doing autoconnect was rather reluctant.

I'll be resending the patches.

Cheers,

Hannes

where do we sit with this ?

-- james

I'll be working on updating nvme-cli to accept 'none' parameters, and then I'll update the patch to be transport-agnostic.

Cheers,

Hannes

Dr. Hannes Reinecke		               zSeries & Storage
hare at suse.com			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)