nvme_fc auto-connect scripts

cp nvmefc_boot_connections.sh /usr/sbin/lpfc/

+ExecStart=/bin/sh -c "/usr/sbin/lpfc/nvmefc_boot_connections.sh"

Quoted 2 lines

cp nvmefc_boot_connections.sh /usr/sbin/lpfc/

[...]

Quoted 1 line

+ExecStart=/bin/sh -c "/usr/sbin/lpfc/nvmefc_boot_connections.sh"

/usr/sbin/lpfc wont get past our package reviewers, %_sbindir (for rpm, I'm sure there is an equivalent for deb as well) should be used there.

Also wouldn't it make sense to have the scripts/services in nvme-cli?

Sagi recently posted a "one-shot" call to nvme-connect via a systemd script. FC has no problems operating with it assuming all the required connect fields are present (host_traddr required).

However FC is a bit more dynamic, and we've built up scripts that can be triggered whenever a new FC remote port with a discovery subsytem is detected. Whenever such a port is detected (by registration of the remoteport with the nvme_fc transport by the lldd), the transport generates a udev event. A udev rule is created that parses the event information into the FC remote port (traddr) that supports a discovery subsystem and the FC host port that is connected to the FC remote port (host_traddr). The rule then invokes an instance-based systemd service to perform a "nvme connect-all" with the traddr and host-traddr info.

To resolve some issues seen by discovery of remote ports before the udev service was running and where the events were not replayed, there is an additional systemd service that runs once and invokes a script which locates the FC host ports on the system, looks for any connected FC remote ports with discovery subsystems, and invokes a connect-all for them. The FC port information is specific to lpfc attributes currently, but when the shared transport is in place, they will be common.

I don't claim to be a systemd expert, so comments are certainly welcome.

Quoted 22 lines · James Smart

Sagi recently posted a "one-shot" call to nvme-connect via a systemd script. FC has no problems operating with it assuming all the required connect fields are present (host_traddr required).

However FC is a bit more dynamic, and we've built up scripts that can be triggered whenever a new FC remote port with a discovery subsytem is detected. Whenever such a port is detected (by registration of the remoteport with the nvme_fc transport by the lldd), the transport generates a udev event. A udev rule is created that parses the event information into the FC remote port (traddr) that supports a discovery subsystem and the FC host port that is connected to the FC remote port (host_traddr). The rule then invokes an instance-based systemd service to perform a "nvme connect-all" with the traddr and host-traddr info.

To resolve some issues seen by discovery of remote ports before the udev service was running and where the events were not replayed, there is an additional systemd service that runs once and invokes a script which locates the FC host ports on the system, looks for any connected FC remote ports with discovery subsystems, and invokes a connect-all for them. The FC port information is specific to lpfc attributes currently, but when the shared transport is in place, they will be common.

Quoted 2 lines

I don't claim to be a systemd expert, so comments are certainly welcome.

In general I'm not in favour of calling 'systemctl start' from a udev rule; this typically will cause systemd to rearrange the internal dependency chain, and might trigger various services to be enabled or disabled.

The 'normal' flow of events is:

kernel device -> uevent -> udev rules -> systemd services

ie for the autoconnect it would be _far_ better if we could have an event per NVMe lport device, as then we could call 'connect' on the lport device.

So a better workflow would be to have the 'discovery' event trigger a 'change' (or 'add'?) event on the lports, and then we could have per-lport udev rules triggering the actualy connect.

Let's see if I can come up with something along these lines.

Cheers,

Hannes

Quoted 42 lines · Hannes Reinecke

Quoted 22 lines · James Smart

Sagi recently posted a "one-shot" call to nvme-connect via a systemd script. FC has no problems operating with it assuming all the required connect fields are present (host_traddr required).

However FC is a bit more dynamic, and we've built up scripts that can be triggered whenever a new FC remote port with a discovery subsytem is detected. Whenever such a port is detected (by registration of the remoteport with the nvme_fc transport by the lldd), the transport generates a udev event.? A udev rule is created that parses the event information into the FC remote port (traddr) that supports a discovery subsystem and the FC host port that is connected to the FC remote port (host_traddr). The rule then invokes an instance-based systemd service to perform a "nvme connect-all" with the traddr and host-traddr info.

To resolve some issues seen by discovery of remote ports before the udev service was running and where the events were not replayed, there is an additional systemd service that runs once and invokes a script which locates the FC host ports on the system, looks for any connected FC remote ports with discovery subsystems, and invokes a connect-all for them.? The FC port information is specific to lpfc attributes currently, but when the shared transport is in place, they will be common.

Quoted 2 lines

I don't claim to be a systemd expert, so comments are certainly welcome.

In general I'm not in favour of calling 'systemctl start' from a udev rule; this typically will cause systemd to rearrange the internal dependency chain, and might trigger various services to be enabled or disabled.

The 'normal' flow of events is:

kernel device -> uevent -> udev rules -> systemd services

ie for the autoconnect it would be _far_ better if we could have an event per NVMe lport device, as then we could call 'connect' on the lport device.

So a better workflow would be to have the 'discovery' event trigger a 'change' (or 'add'?) event on the lports, and then we could have per-lport udev rules triggering the actualy connect.

Quoted 5 lines

Let's see if I can come up with something along these lines.

Cheers,

Hannes

I'm certainly open to better options. udev/systemd is far from nice.? However, what I did has quite a bit of precedence.

I have no idea what you would scan on by just an lport. It's not enough for nvme-cli. You must have the lport and the rport information, which is passed via the udev event info and which must also be passed to nvme connect-all

Oh, I know that.
What I would prefer is to have a single udev rule per nvme connection, containing both lport and rport information.
That would relieve us of having additional configuration, and we can enable/disable connection by just having that udev rule. This works pretty well; we're doing it exactly the same way on S/390 for the network and zfcp connections.
So if we have that then we only need to know when the lport/rport combination is ready; once that is done we can call the udev rule. Ideally we'll be getting an event containing both of these informations, then udev will be selecting the correct rule automatically.

Cheers,

Hannes

Quoted 13 lines · Hannes Reinecke

Oh, I know that.
What I would prefer is to have a single udev rule per nvme connection, containing both lport and rport information.
That would relieve us of having additional configuration, and we can enable/disable connection by just having that udev rule. This works pretty well; we're doing it exactly the same way on S/390 for the network and zfcp connections.
So if we have that then we only need to know when the lport/rport combination is ready; once that is done we can call the udev rule. Ideally we'll be getting an event containing both of these informations, then udev will be selecting the correct rule automatically.

Quoted 3 lines

Cheers,

Hannes

The main reason for the systemd use was the potential duration of the resulting nvme connect-all. The scripts are simple if udev makes the nvme cli call and don't require anything other than a single generic rule. But given its potential stalls, very bad.

If I understand you correctly, what you describe requires apriori knowledge of the connections in order to add the udevs rule. This is a step backward from current FC behavior and what users are used to.

On 10/28/2017 06:08 PM, James Smart wrote:

Quoted 10 lines

The main reason for the systemd use was the potential duration of the resulting nvme connect-all. The scripts are simple if udev makes the nvme cli call and don't require anything other than a single generic rule. But given its potential stalls, very bad.

If I understand you correctly, what you describe requires apriori knowledge of the connections in order to add the udevs rule. This is a step backward from current FC behavior and what users are used to.

So, here's my take for this:

# cat 65-nvme-fc-connect-nvmf-test-subsys1.rules
SUBSYSTEM!="block, GOTO="nvme_autoconnect_end"
ACTION!="add|change", GOTO="nvme_autoconnect_end"
ENV{FC_EVENT}!="nvmediscovery", GOTO="nvme_autoconnect_end"
ENV{NVMEFC_HOST_TRADDR}!="?*", GOTO="nvme_autoconnect_end"
ENV{NVMEFC_TRADDR}!="?*", GOTO="nvme_autoconnect_end"
RUN+="/usr/sbin/nvme connect -t fc \
--host-traddr=$env{NVMEFC_HOST_TRADDR} \
-n nvmf-test-subsys1 -a $env{NVMEFC_TRADDR}"
LABEL="nvme_autoconnect_end"

Obviously, we would need to filter for the appropriate values of traddr and host_traddr to not attempt connections to non-existing namespaces.

However, I do see the problem of nvme connect stalls; if we were having a systemd service we would solve this issue nicely, indeed. But then we shouldn't activate the service, it should rather be activated per default, but depend on a specifice NVMe-FC _target_, which should be presented by the discovery event.

Let's see what I can cobble together here.