Skip to contentSkip to conversation

nvme-cli: NVMf auto discovery systemd service

RFC PATCH New Sagi Grimberg 3 participants 5 messages Sep 24, 2017 – Sep 25, 2017 NVMe

lore
Share & tools
lore

Sagi Grimberg

author maintainer
#0 lore raw

RFC PATCH 1/1 nvme-cli: NVMf auto discovery systemd service

Run a one-shot discovery (and connect) at system boot time. This allows to auto discover and establish nvmf sessions.

If the user is interested in using this, it should fill in one or more discovery endpoints, for example:
-t rdma -a 192.168.1.1 -s 4420

This service relies on nvme-fabrics module being loaded, hence we add it to modules load at boot list.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>

---
Don't have a fedora station near-by, manually tested auto discover during boot with systemd service, and it seem to work (although I needed to manually get mlx5_ib modules to auto-load as for some reason it fails to auto load at boot time).

We can also add some documentation into discovery.conf to help the user add discovery endpoints.

 nvme.spec.in                   | 19 +++++++++++++++++++
 systemd/nvmf-discovery.service | 12 ++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 systemd/nvmf-discovery.service
nvme.spec.in +19 -0
index 0be61e398fd0..c22bdeec9138 100644
@@ -43,12 +43,31 @@ if [ $1 = 1 ]; then # 1 : This package is being installed for the first time
         if [ ! -f /etc/nvme/hostid ]; then
                 uuidgen > /etc/nvme/hostid
         fi
+        if [ ! -f /etc/nvme/discovery.conf ]; then
+		touch /etc/nvme/discovery.conf
+        fi
 fi
 
+# NVMf auto-discovery service
+install -m 0755 systemd/nvmf-discovery.service /lib/systemd/system/
+systemctl enable nvmf-discovery.service
+
+# we rely on /dev/nvme-fabrics misc device, so auto load nvme-fabrics
+# at boot time
+echo nvme-fabrics >> /etc/modules-load.d/modules.conf
+
 %preun
 if [ "$1" = "remove" ]; then
+	sed -i 's/nvme-fabrics//e' /etc/modules-load.d/modules.conf
+
+	if [ -f /lib/systemd/system/nvmf-discovery.service]; then
+		systemctl disable nvmf-discovery.service
+		rm -f /lib/systemd/system/nvmf-discovery.service
+	fi
+
 	if [ -d /etc/nvme ]; then
 		rm -f /etc/nvme/hostnqn
+		rm -f /etc/nvme/discovery.conf
 		if [ ! -n "`ls -A /etc/nvme`" ]; then
 			rm -rf /etc/nvme
 		fi
systemd/nvmf-discovery.service new file +12 -0
new file mode 100644
index 000000000000..25f7bf1f077b
@@ -0,0 +1,12 @@
+[Unit]
+Description=NVMf auto discovery service
+After=systemd-modules-load.service network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=nvme connect-all
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
Signature · 1 line
2.7.4
maintainer
#1 lore raw

RFC PATCH 1/1 nvme-cli: NVMf auto discovery systemd service

This looks reasoanable to me, although I'm really not the expert for systemd stuff.

Does it handle the case where /etc/nvme/discovery.conf is non-existent (yes, you created in the specfile, but not everyone uses rpms).

Quoted 2 lines

We can also add some documentation into discovery.conf to help the user add discovery endpoints.

That would be great for the default one, e.g. some example RDMA and FC configfs.

author maintainer
#2 lore raw

RFC PATCH 1/1 nvme-cli: NVMf auto discovery systemd service

Quoted 2 lines

This looks reasoanable to me, although I'm really not the expert for systemd stuff.

Me neither, Hence the RFC...

Quoted 2 lines

Does it handle the case where /etc/nvme/discovery.conf is non-existent (yes, you created in the specfile, but not everyone uses rpms).

Well, it will fail the nvme connect-all command, which would appear in the journal and disable the service, the user is expected to add discovery.conf and re-enable the service at that point...

Quoted 5 lines
Quoted 2 lines

We can also add some documentation into discovery.conf to help the user add discovery endpoints.

That would be great for the default one, e.g. some example RDMA and FC configfs.

I think James was pushing udev stuff for auto connection as FC discovery is in-band, so not sure how this will apply to FC...

#3 lore raw

RFC PATCH 1/1 nvme-cli: NVMf auto discovery systemd service

In reply to Sagi Grimberg
Quoted 30 lines · Sagi Grimberg

Run a one-shot discovery (and connect) at system boot time. This allows to auto discover and establish nvmf sessions.

If the user is interested in using this, it should fill in one or more discovery endpoints, for example:
-t rdma -a 192.168.1.1 -s 4420

This service relies on nvme-fabrics module being loaded, hence we add it to modules load at boot list.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>

---
Don't have a fedora station near-by, manually tested auto discover during boot with systemd service, and it seem to work (although I needed to manually get mlx5_ib modules to auto-load as for some reason it fails to auto load at boot time).

We can also add some documentation into discovery.conf to help the user add discovery endpoints.

 nvme.spec.in                   | 19 +++++++++++++++++++
 systemd/nvmf-discovery.service | 12 ++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 systemd/nvmf-discovery.service
nvme.spec.in +0 -0
index 0be61e398fd0..c22bdeec9138 100644

nvme.spec.in if [ $1 = 1 ]; then # 1 : This package is being installed for the first time · line 48

Quoted 6 lines
nvme.spec.in +3 -0
@@ -43,12 +43,31 @@ if [ $1 = 1 ]; then # 1 : This package is being installed for the first time
         if [ ! -f /etc/nvme/hostid ]; then
                 uuidgen > /etc/nvme/hostid
         fi
+        if [ ! -f /etc/nvme/discovery.conf ]; then
+		touch /etc/nvme/discovery.conf
+        fi

I don't think having an empty discovery.conf is a particularly good idea, the contents should be either added by hand or by some configuration/installer tool (YAST, Anaconda, Chef, Puppet, yada yada).

nvme.spec.in if [ $1 = 1 ]; then # 1 : This package is being installed for the first time · line 53

Quoted 5 lines
nvme.spec.in +3 -0
 fi
 
+# NVMf auto-discovery service
+install -m 0755 systemd/nvmf-discovery.service /lib/systemd/system/
+systemctl enable nvmf-discovery.service

Unconditionally starting services will is (at least for SUSE RPMs) a no-go.

nvme.spec.in if [ $1 = 1 ]; then # 1 : This package is being installed for the first time · line 57

Quoted 4 lines
nvme.spec.in +4 -0
+
+# we rely on /dev/nvme-fabrics misc device, so auto load nvme-fabrics
+# at boot time
+echo nvme-fabrics >> /etc/modules-load.d/modules.conf

This should be done in the systemd unit file I think. Something like this:

systemd/nvmf-discovery.service line 6

Quoted 12 lines
systemd/nvmf-discovery.service new file +6 -0
new file mode 100644
index 000000000000..25f7bf1f077b
@@ -0,0 +1,12 @@
+[Unit]
+Description=NVMf auto discovery service
+After=systemd-modules-load.service network-online.target
+
+[Service]
+Type=oneshot

ExecStartPre=-/sbin/modprobe nvme-fabrics

Quoted 9 lines
systemd/nvmf-discovery.service new file +6 -0
+ExecStart=nvme connect-all
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
Signature · 1 line
2.7.4
Signature · 6 lines
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
author maintainer
#4 lore raw

RFC PATCH 1/1 nvme-cli: NVMf auto discovery systemd service

Quoted 8 lines

nvme.spec.in if [ $1 = 1 ]; then # 1 : This package is being installed for the first time · line 48

Quoted 4 lines
nvme.spec.in +3 -0
+        if [ ! -f /etc/nvme/discovery.conf ]; then
+		touch /etc/nvme/discovery.conf
+        fi

I don't think having an empty discovery.conf is a particularly good idea, the contents should be either added by hand or by some configuration/installer tool (YAST, Anaconda, Chef, Puppet, yada yada).

Its just placing the file there to not fail the service by definition of the conf not being there - if its empty it just silently exits. We can skip it if you think its not a good idea.

Quoted 5 lines

nvme.spec.in if [ $1 = 1 ]; then # 1 : This package is being installed for the first time · line 53

Quoted 3 lines
nvme.spec.in +3 -0
+# NVMf auto-discovery service
+install -m 0755 systemd/nvmf-discovery.service /lib/systemd/system/
+systemctl enable nvmf-discovery.service

Unconditionally starting services will is (at least for SUSE RPMs) a no-go.

OK, we can lose it.

Quoted 7 lines

nvme.spec.in if [ $1 = 1 ]; then # 1 : This package is being installed for the first time · line 57

Quoted 4 lines
nvme.spec.in +4 -0
+
+# we rely on /dev/nvme-fabrics misc device, so auto load nvme-fabrics
+# at boot time
+echo nvme-fabrics >> /etc/modules-load.d/modules.conf

This should be done in the systemd unit file I think. Something like this:

Quoted 14 lines

systemd/nvmf-discovery.service line 6

Quoted 12 lines
systemd/nvmf-discovery.service new file +6 -0
new file mode 100644
index 000000000000..25f7bf1f077b
@@ -0,0 +1,12 @@
+[Unit]
+Description=NVMf auto discovery service
+After=systemd-modules-load.service network-online.target
+
+[Service]
+Type=oneshot

ExecStartPre=-/sbin/modprobe nvme-fabrics

looks better!