SAS domain layout for Linux sysfs

Hi,

This is an RFC about a SAS domain layout for Linux sysfs.

The idea is to represent "what is out there" and "what we see from this host adapter" in sysfs, so that a process can show a picture of the storage network.

This gives a close representation of what the SAS spec describes, so that more tools can be built on that, like storage applications, user applications, etc.

The SAS LLDD registers with the SAS class, giving it information about itself: phys, WWNs, etc. The SAS class does SAS domain discovery, representing its findings in the syfs host domain and in the sysfs SAS domain (defined in the RFC below).

SAS domain layout for Linux sysfs
=================================

0. Introduction

The use of SAS address and WWN are used interchangeably.

There are two domains which we want to represent in sysfs, in order to eliminate redundancies.

            |     /-------------------\
+-------+   |    /  SAS_ADDR0          \
|ha0 [] =---|---(                       )
+---||||+   |    \                     /
            |     |       SAS_ADDR2   |
+-------+   |    /                     \
|ha1 [] =---|---(      SAS_ADDR1        )
+---||||+   |    \                     /
            |     \___________________/
            |
Host domain |         SAS Domain

Figure 1. Domains represented by sysfs

The host domain (/sys/class/sas_ha/ha0/, etc) shows the SAS domain as seen by the Host Adapter. The sysfs SAS domain (/sys/bus/sas/ ), shows the SAS domain as it exists irrespectively of which HA (Host Adapter) you use to connect to it (to a device).

           |                        |
+-------+  |  +-----+               |
|ha0 [] =--|--= ex0 =--.            |
+---||||+  |  +-----+   \  +-----+  |
           |             `-= ex2 =--|--> ta0
           |               |     =--|--> in2
           |             .-=     =--|--> ta2
+-------+  |  +-----+   /  +-----+  |
|ha1 [] =--|--= ex1 =--'            |
+---||||+  |  +-----+               |
           |                        |
Host domain| Sysfs SAS domain only  | Both domains

Figure 2. Breakdown of sample storage setup

Host domain: /sys/class/sas_ha/haX
Since its point of view is from the host where ha0 and ha1 are.

Sysfs SAS domain only: /sys/bus/sas/...
Since it is part of the SDS and irrespective of the host. I.e. another host could connect to ex1 or even to ex2, and see the same devices.

Both domains:
Since those end devices are visible by all host SAS processes.

1 Host Domain
-------------

The host domain is a SAS domain as seen by a particular SAS Host Adapter. It lives in /sys/class/sas_ha .

/sys/class/sas_ha/

Holds all Host Adapters.

/sys/class/sas_ha/ha0
/sys/class/sas_ha/ha1
..., etc.

Host adapter directories hold the attributes of the host adapter, the phys and the ports. E.g.

/sys/class/sas_ha/ha0/device --> <PCI device directory> /sys/class/sas_ha/ha0/driver --> <PCI driver directory> /sys/class/sas_ha/ha0/phys/ /sys/class/sas_ha/ha0/ports/ /sys/class/sas_ha/ha0/device_name

device_name is a sysfs text file holding the SAS address of the SAS host adapter.

phys/ lists the phys which are part of the HA (Host
Adapter).

/sys/class/sas_ha/ha0/phys/0/
/sys/class/sas_ha/ha0/phys/1/
..., etc.

A phy has attributes which are stored in its directory, e.g.:

/sys/class/sas_ha/ha0/phys/0/ /sys/class/sas_ha/ha0/phys/0/port --> ../../ports/<port id>
/sys/class/sas_ha/ha0/phys/0/id (same as .)
/sys/class/sas_ha/ha0/phys/0/enabled /sys/class/sas_ha/ha0/phys/0/class /sys/class/sas_ha/ha0/phys/0/proto /sys/class/sas_ha/ha0/phys/0/type /sys/class/sas_ha/ha0/phys/0/role /sys/class/sas_ha/ha0/phys/0/linkrate /sys/class/sas_ha/ha0/phys/0/sas_address
(sas address as transmitted in IDENTIFY)

Those are standard attributes, SAS 1.1, chapter 4.

ports/ lists the ports which are part of the HA.

1.1 Directly attached end devices
---------------------------------

/sys/class/sas_ha/ha0/ports/<port id>/
/sys/class/sas_ha/ha0/ports/<port id>/class
/sys/class/sas_ha/ha0/ports/<port id>/port_identifier
/sys/class/sas_ha/ha0/ports/<port id>/attached_port_identifier -> ../../../../../bus/sas/<WWN_ta0>/ports/<port id>/port_identifier /sys/class/sas_ha/ha0/ports/<port id>/phys/
/sys/class/sas_ha/ha0/ports/<port id>/phys/0 -> ../../../phys/0
/sys/class/sas_ha/ha0/ports/<port id>/phys/1 -> ../../../phys/1
/sys/class/sas_ha/ha0/ports/<port id>/devices/
/sys/class/sas_ha/ha0/ports/<port id>/devices/ta0 -> ../../../devices/ta0

<port id> is a positive integer starting from 0, which is just a local port identifier.

port_identifier is a text file, which is the SAS port identifier.

attached_port_identifier is a link to the other port's WWN. It is a link to the port inside a device's directory structure in the sysfs SAS domain.

phys/ is a directory with symlinks to phys of that adapter which participate in this port.

devices/ is a directory of devices visible from this port, In this case a single device directly attached.

1.2 Devices past expanders
--------------------------

/sys/class/sas_ha/ha0/ports/<port id>/
/sys/class/sas_ha/ha0/ports/<port id>/class
/sys/class/sas_ha/ha0/ports/<port id>/port_identifier
/sys/class/sas_ha/ha0/ports/<port id>/attached_port_identifier -> ../../../../../bus/sas/<WWN_ex0>/ports/<port id>/port_identifier /sys/class/sas_ha/ha0/ports/<port id>/phys/
/sys/class/sas_ha/ha0/ports/<port id>/phys/0 -> ../../../phys/0
/sys/class/sas_ha/ha0/ports/<port id>/phys/1 -> ../../../phys/1
/sys/class/sas_ha/ha0/ports/<port id>/ex0 -> ../../../../../bus/sas/<WWN_ex0> /sys/class/sas_ha/ha0/ports/<port id>/devices/
/sys/class/sas_ha/ha0/ports/<port id>/devices/ta1 -> ../../../devices/ta1
/sys/class/sas_ha/ha0/ports/<port id>/devices/in3 -> ../../../devices/in3

Using SMP DISCOVER for each phy of the expander
we find out ports (matching attached WWNs from the expander's view point) and report them. See sysfs SAS domain below.

1.3 Host adapter directory
--------------------------

Host adapter directories would also hold a "cheat sheet" of

/sys/class/sas_ha/ha0/devices/

The contents of this directory would be symbolic links to the sysfs SAS domain. The name used, two letters and an integer, is per HA unique.

/sys/class/sas_ha/ha0/devices/ta0 -> ../../../../sas/bus/<WWN_ta0> /sys/class/sas_ha/ha0/devices/ta1 -> ../../../../sas/bus/<WWN_ta1> /sys/class/sas_ha/ha0/devices/in3 -> ../../../../sas/bus/<WWN_in3>

All those point to directories in the sysfs SAS domain, where attributes of those devices are held.

2. Sysfs SAS Domain
-------------------

Represent everything which "sits out there" in the SAS domain, irrespective of how you connect to it.

/sys/bus/sas/
/sys/bus/sas/<WWN_ta0>/
/sys/bus/sas/<WWN_ta0>/phys/

The ports/ directory is populated when different initiators discover the device. That is, for each HA which can make a connection to the target, there's a port on the target, we record those ports here.

/sys/bus/sas/<WWN_ta0>/ports/
/sys/bus/sas/<WWN_ta0>/ports/<port id>/port_identifer
/sys/bus/sas/<WWN_ta0>/ports/<port id>/attached_port_identifer -> ../../../../../class/sas_ha/ha0/ports/<port id>/port_identifier

port_identifier is a text file whose contents is the WWN of the port.

/sys/bus/sas/<WWN_ta1>/
/sys/bus/sas/<WWN_in3>/
/sys/bus/sas/<WWN_ex2>/

/sys/bus/sas/<WWN_ex0>/
/sys/bus/sas/<WWN_ex0>/phys/
/sys/bus/sas/<WWN_ex0>/ports/
/sys/bus/sas/<WWN_ex0>/ports/<port id>/
/sys/bus/sas/<WWN_ex0>/ports/<port id>/port_identifier
/sys/bus/sas/<WWN_ex0>/ports/<port id>/attached_port_identifier -> ../../../../../class/sas_ha/ha0/ports/<port id>/port_identifier /sys/bus/sas/<WWN_ex0>/devices/
/sys/bus/sas/<WWN_ex0>/devices/<WWN_ta1> -> ../../<WWN_ta1>
/sys/bus/sas/<WWN_ex0>/devices/<WWN_ta0> -> ../../<WWN_ta0>
(in case connection to ta0 through ex0 is also
possible)

Thanks,
Luben

This is contrary to any sysfs topology I know about, especially any existing transport class (SPI, FC, iSCSI). We only ever care about what's seen from a HA, e.g. if you have muliple SPI cards that are on a single parallel bus you'll have the same bus represented twice, similarly if you have two fibre channel HBAs connected to the same SAN you'll have the SAN topology duplicated in both sub-topologies. This matches the internal data structure of the scsi subsystem and the transport class, e.g. we have a scsi_device object for every lun that's seen from a hba, linked to the HBAs Scsi_Host object and not one shared by multiple HBAs. Dito for fibre channel remote ports.

One note to this proposal: it probably doesn't make a lot of sense to try to flesh out the sysfs topology before doing the kernel internal object model as the former pretty much follows the latter.

This RFC is about SAS.

Imagine you could connect to the same device via two different PCI controllers on the same host.

Hmm, this proposal is for SAS only, Christoph.

If you have multiple SAS host adapters connected to the same SAS domain, the _path_ they connect to a SAS device may be _different_. But what is the same is the SAS domain (topology) itself *regardless of how you connect to it.*

In order to eliminate duplication of sysfs entries (directories and files) to describe the same SAS device, we split up the representation into a "flat" directory with just a bunch of SAS devices, this is /sys/bus/sas/. And the way you _connect_ to those SAS devices is represented in sys/class/sas_ha/.

See this (new) picture:

           |                        |
+-------+  |                        |
|ha0 [] =--|-----------.            |
+---||||+  |            \  +-----+  |
           |             `-= ex2 =--|--> ta0
           |               |     =--|--> in2
           |             .-=     =--|--> ta2
+-------+  |  +-----+   /  +-----+  |
|ha1 [] =--|--= ex1 =--'            |
+---||||+  |  +-----+               |
           |                        |
Host domain| Sysfs SAS domain only  | Both domains

Anything *but* "Host domain" is "out there" and *doesn't change* regardless of how you connect to it.

What changes is *how you connect* to to SAS devices from your host (the host domain).

In effect ta0, in2, ta2, ex1, ex2 are represented only *once*, in /sys/bus/sas/. But the way your host adapter connects to them is described in /sys/class/sas_ha/, with the appropriate symbolic links as described in the RFC.

Luben

But our SAS transport class should be consistant with the other transport classes.

Which is exactly the case in the examples I've mentioned.

Please read the previous mail again, you're not getting it at all.

If you don't understand the problems it's not worth talking about more.

Here is your original email:

You're are stating that:
1) You "only ever care about what's seen from a HA",
2) "if you have muliple SPI cards that are on a single parallel bus you'll have the same bus represented twice"
3) "we have a scsi_device object for every lun
that's seen from a hba, linked to the HBAs Scsi_Host object and not one shared by multiple HBAs"

So in effect, you _want_ duplication, right? This is perfectly OK. In fact it is desirable (and has never been under question).

This isn't directly related to the RFC. The RFC basically outlines the result of *a SAS discovery process*. The discovery process/LLDD can register the LU many times. This has *never* been an issue of discussion.

Your concern is satisfied in that, /sys/bus/scsi/devices/... would point to the LU. And a link from /sys/class/sas_ha/ha0/devices/ to the LU could be placed (outside the scope of this RFC). The RFC doesn't mention LUs, but they could be included if you wish so.

Luben

Yes.

The point is that discovery must happen for each HBA separately because we absolutely do not want to have global state.

OK.

Can you please define "global state"?

In SAS, when the domain changes, the controller(s) connected to the domain get notification and pass it to the discovery process, which will run again.

Overall, since the discovery process gets (internally) a "picture" of the domain out there, it would be appropriate to show this "picture" to the user.

Luben

The SAS/sysfs discussion seems to have run aground so I'll try a different example in order to find some common ground.

Below is a diagram with one SAS HBA (also known as HA) with two phys in use. One is connected directly to a SAS dual ported disk and the other phy goes via a SAS expander to the other "side" of the same disk:

+---------+  |                     |  +-----------+
|scsi0 [] =--|---------------------|--= sda       |
|         |  |                     |  |           |
|         |  |    +-----+          |  |           |
|scsi1 [] =--|----= ex1 =----------|--= sdb       |
+-----||||+  |    +-----+          |  +-----------+
             |                     |
1 HBA(HA)    | 2 service delivery  |   1 dual ported
               subsystems              SAS disk

(2 SAS domains)

SCSI analysis
-------------
Even though there is one HBA and one SCSI initiator we have two SCSI initiator ports. In Linux we name initiator ports (according to scsi_mid_low_api.txt) rather than an initiator device, hence scsi0 and scsi1. Each SCSI initiator port is connected to an independent service delivery subsystem. There is one target device which contains two ports and one lu (lun=0). Those target ports are connected to separate service delivery subsystems.

Since we name by path, the dual ported disk has two linux device names: sda and sdb. The tuple for sda is <scsi0,0,upper_target_port_sas_address,0>. The third element is a 64 bit SAS address which may need to be mapped to small integer to fit the linux scsi subsystem.

The VPD device identification page will show that the lu name of both sda and sdb is the same. This becomes the responsibility of multipath-tools to sort out.

SAS analysis
------------
The two phys in the HBA do _not_ form a wide port since the other ends have different SAS addresses (sda's port address is different from the ex1's address). Hence we have 2 narrow ports (where 1 phy corresponds to 1 initiator port).

Dual ported SAS disks have different SAS addresses on each phy so that each phy will be a (narrow) target port.

With or without the expander (ex1) there would be two SAS domains. From the initiator's point of view the difference between its two ports is that the silicon state machines can see sda but not sdb (scsi1's state machine sees ex1 instead).

sysfs and SAS discovery
-----------------------
It seems reasonable that sda should be visible in sysfs since the silicon in the HBA (for scsi0) knows about it. However the silicon for scsi1 knows about ex1 (and nothing beyond that). SAS defines the Serial Management Protocol (SMP) for the purpose of probing what (else) is connected to an expander. SMP is similar to a SCSI command set and SMP frames need to be sent via scsi1 to ex1. Note that ex1 is _not_ a SCSI device (it is
part of the service delivery subsystem) and we have no representation currently for it in sysfs. [Fibre channel switches are architecturally similar to SAS expanders.]

It has been stated that the SAS discovery algorithm (i.e. the recursive use of SMP) should be implemented once in the SAS transport layer so that all SAS LLDs can use it. Putting the SAS discovery algorithm in the user space may be even more politically correct.

Once the SAS discovery algorithm has been run should we show its results in sysfs?? We probably want to know about SCSI target devices (like we do for other transports). The SAS discovery algorithm may have found other interesting things:
- other expanders (beyond what the silicon has seen)
- other initiators (implies a multi initiator environment)
- miswired SAS domains (since SAS expander routing rules have restrictions)

Other tools may want to access SMP (and SCSI log pages in SCSI target devices) to identify bottlenecks and access vendor extensions.

Are there disagreements with the example and its analysis? Other comments?

Doug Gilbert

Yes, I think we should know about those other devices, part of SAS SDS.

Yes, very true. I can imagine user space apps sending SMP and what not to expanders/RAID devices/enclosures past expanders, to control the storage network.

A sysfs representation of the discovery result could make this easy, since as you pointed out expanders are not SAS devices, and thus do not fit the linux-scsi HCTL space.

Luben

Note that the current scsi code allows to create custom per-transport objects below the target object. We're using that in the fibre channel transport class for the concept of remote ports.

I think so, yes. Similar to how we have all fibre channel remote ports in sysfs, even if they are not scsi targets.

Yes, I've been meaning to ask about this... since a SAS host adapter (class) has ports (class?) which has phys (class), so there's this hierarchy between those to-be transport classes.

Is this doable given the current infrastructure?

(I guess when it comes down to it one can represent a sub-class as
an "attribute" or even as a linked list...)

Ok, we can show this from each HA (/sys/class/sas_ha/...) but you do not want the existence of /sys/bus/sas/... where things are consolidated. I guess that's ok, and if needed, it can always be implemented later.

Luben

Sure, quoting your initial mail:

/----------------------------------------------------------------------

| 2. Sysfs SAS Domain
| -------------------
|
| Represent everything which "sits out there" in the SAS
| domain, irrespective of how you connect to it.
|
| /sys/bus/sas/
| /sys/bus/sas/<WWN_ta0>/
| /sys/bus/sas/<WWN_ta0>/phys/
|
| ...

\----------------------------------------------------------------------

Here you have a global hiearchy. We are not interested in that, though - we only care for what's visible from a certain HBA.

Same as with fibre channel, and that's fine and expected.

Absolutely.

True, it is a "global" hierarchy. But visiblity from a single HBA is also present in /sys/class/sas_ha/... .

Ok, so then we want some kind of "global" representation? Since the discover process (in its attempt to discover topology errors) would have to know little or less about the "global" outlook.

Luben