: Generic SCSI Target Mid-level For Linux (SCST), target drivers for iSCSI and QLogic Fibre Channel cards released

I'm glad to announce that version 1.0.0 of Generic SCSI Target Middle Level for Linux (SCST) was released and available for download from
http://scst.sourceforge.net/downloads.html

SCST is a subsystem of the Linux kernel that provides a standard framework for SCSI target drivers development. It is designed to provide unified, consistent interface between SCSI target drivers and Linux kernel and simplify target drivers development as much as possible. It has the following main features:

* Simple, easy to use interface with target drivers. Particularly,
SCST core performs required pre- and post- processing of incoming requests as well as necessary error recovery.

* Undertakes most problems, related to execution contexts, thus
practically eliminating one of the most complicated problem in the kernel drivers development. For example, a target driver for QLogic 22xx/23xx cards, which has all necessary features, is only about 2000 lines of code long.

* Very low overhead, fine-grained locks and simplest commands
processing path, which allow to reach maximum possible performance and scalability. Particularly, incoming requests can be processed in the caller's context or in one of the internal SCST core's tasklets, therefore no extra context switches required.

* Device handlers, i.e. plugins, architecture provides extra
flexibility by allowing various I/O modes in backstorage handling. For example, pass-through device handlers allow to use real SCSI hardware and vdisk device handler allows to use files as virtual disks.

* Provides advanced per-initiator devices visibility management
(LUN masking), which allows different initiators to see different set of devices with different access permissions. For instance, initiator A could see exported from target T devices X and Y read-writable, and initiator B from the same target T could see devices Y read-only and Z read-writable.

* Emulates necessary functionality of SCSI host adapter, because
from remote initiators point of view SCST acts as a SCSI host with its own devices.

The following I/O modes are supported by SCST:

* Pass-through mode with one to many relationship, i.e. when multiple
initiators can connect to the exported pass-through devices, for virtually all SCSI devices types: disks (type 0), tapes (type 1), processors (type 3), CDROMs (type 5), MO disks (type 7), medium changers (type 8) and RAID controllers (type 0xC)

* FILEIO mode, which allows to use files on file systems or block
devices as virtual remotely available SCSI disks or CDROMs with benefits of the Linux page cache

* BLOCKIO mode, which performs direct block I/O with a block device,
bypassing page-cache for all operations. This mode works ideally with high-end storage HBAs and for applications that either do not need caching between application and disk or need the large block throughput.

* User space mode using scst_user device handler, which allows to
implement in the user space virtual SCSI devices in the SCST environment.

Detail description of SCST, its drivers and utilities you can find on SCST home page http://scst.sourceforge.net.

Comparison with the mainstream target middle level STGT you can find on the SCST vs STGT page http://scst.sourceforge.net/scstvsstgt.html. In short, SCST has the following main advantages over STGT:

- Better performance (in many cases tens of %% and more) with
potential for further improvement, for example, by implementing zero-copy cache I/O.

- Monolithic in-kernel architecture, which follows standard Linux
kernel paradigm to eliminate distributed processing. It is simpler, hence more reliable and maintainable.

SCST is being prepared in form of patch for review and inclusion to the kernel.

Vlad
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Vlad,

Quoted 1 line · Vladislav Bolkhovitin

I'm glad to announce that version 1.0.0 of Generic SCSI Target Middle Level for Linux (SCST) was released and available for download from http://scst.sourceforge.net/downloads.html

Congratulations on reaching your v1.0.0 release!

Quoted 1 line

Comparison with the mainstream target middle level STGT you can find on the SCST vs STGT page http://scst.sourceforge.net/scstvsstgt.html. In short, SCST has the following main advantages over STGT:

I noticed that you included a reference to my presentation at LSF 08' on your SCST vs. STGT page liked above, and took my description of your work (you are more than welcome to come and present your own case at LSF '09) very much out of context.

If you wish to reference my presentation, please at least make the comparision between LIO-Core+LIO-Target vs. SCST vs. STGT, and NOT JUST SCST vs. STGT so that the community are large can understand the differences and technical challenges.

The more in fighting between the
leaders in our community, the less the community benefits.

Many thanks for your most valuable of time,

--nab

Hi Nicholas,

Quoted 8 lines · Nicholas A. Bellinger

Hi Vlad,

Quoted 3 lines · Vladislav Bolkhovitin

I'm glad to announce that version 1.0.0 of Generic SCSI Target Middle Level for Linux (SCST) was released and available for download from
http://scst.sourceforge.net/downloads.html

Congratulations on reaching your v1.0.0 release!

Thanks!

Quoted 4 lines

Quoted 3 lines

Comparison with the mainstream target middle level STGT you can find on the SCST vs STGT page http://scst.sourceforge.net/scstvsstgt.html. In short, SCST has the following main advantages over STGT:

Quoted 4 lines

I noticed that you included a reference to my presentation at LSF 08' on your SCST vs. STGT page liked above, and took my description of your work (you are more than welcome to come and present your own case at LSF '09) very much out of context.

I wasn't on the presentation, so on it might have looked as out of context.

I have only documents, which I referenced. In them, especially
in "2008 Linux Storage & Filesystem Workshop" summary, it doesn't look as I took it out of context. You put emphasis on "older" vs "current"/"new", didn't you ;)?

Plus, Mike Christie is also listed as an
author.

BTW, there are another inaccuracies on your slides:

- STGT doesn't support "hardware accelerated traditional iSCSI
(Qlogic)", at least I have not found any signs of it.

- For SCST you wrote "Only project to support PSCSI, FC and SAS Target
mode (with out of tree hardware drivers)". This is ambiguous statement, but looks like you meant that SCST is intended and limited to support only the listed transport. This is incorrect. SCST is intended to support ALL possible transports and types backstorage.

Quoted 4 lines

If you wish to reference my presentation, please at least make the comparision between LIO-Core+LIO-Target vs. SCST vs. STGT, and NOT JUST SCST vs. STGT so that the community are large can understand the differences and technical challenges.

The SCST vs STGT page was written a long ago, before I ever looked at LIO. I wasn't actually going to refer to your presentation, just added a small note to your funny, from my POV, "older" vs "new" architecture comparison ;)

But, when I have time for careful look, I'm going to write some LIO critics. So far, at the first glance:

- It is too iSCSI-centric. ISCSI is a very special transport, so looks
like when you decide to add in LIO drivers for other transports, especially for parallel SCSI and SAS, you are going to have big troubles and major redesign.

And this is a real showstopper for making LIO-Core
the default and the only SCSI target framework. SCST is SCSI-centric,

just because there's no way to make *SCSI* target framework not being SCSI-centric. Nobody blames Linux SCSI (initiator) mid-layer for being SCSI-centric, correct?

- Seems, it's a bit overcomplicated, because it has too many abstract
interfaces where there's not much need it them. Having too many abstract interfaces makes code analyze a lot more complicated. For comparison, SCST has only 2 such interfaces: for target drivers and for backstorage dev handlers. Plus, there is half-abstract interface for memory allocator (sgv_pool_set_allocator()) to allow scst_user to allocate user space supplied pages. And they cover all needs.

- Pass-through mode (PSCSI) also provides non-enforced 1-to-1
relationship, as it used to be in STGT (now in STGT support for pass-through mode seems to be removed), which isn't mentioned anywhere.

- There is some confusion in the code in the function and variable
names between persistent and SAM-2 reservations.

- There is at least one SCSI standard violation: target and LUN resets
don't clear the reservation.

Again, it is the first impression, without deep analyze, so I might be wrong somewhere.

Quoted 2 lines

The more in fighting between the
leaders in our community, the less the community benefits.

Sure. If my note hurts you, I can remove it. But you should also remove from your presentation and the summary paper those psychological arguments to not confuse people.

Quoted 2 lines

I have only documents, which I referenced. In them, especially
in "2008 Linux Storage & Filesystem Workshop" summary, it doesn't look as I took it out of context. You put emphasis on "older" vs "current"/"new", didn't you ;)?

Well, my job was to catch everyone up to speed on the status of the 4 (four) different (insert your favorite SAM capable transport name here) Linux v2.6 based target projects. With all of the acroynms for the standards+implementations+linux-kernel being extremly confusing to anyone who does know all of them by heart. Even those people in the room, who where fimilar with storage, but not necessarly with target mode engine design, its hard to follow.

Quoted 5 lines

BTW, there are another inaccuracies on your slides:

- STGT doesn't support "hardware accelerated traditional iSCSI
(Qlogic)", at least I have not found any signs of it.

<nod>, that is correct. It does it's hardware acceleration generically using OFA VERBS for hardware that do wire protocol that implements fabric dependent direct data placement. iSER does this with 504[0-4],
and I don't recall exactly how IB does it. Anyways, the point is that
they use a single interface so that hardware vendors do not have to implement their own APIs, which are very complex, and usually very buggy when coming from a company who is trying to get a design into ASIC.

Quoted 4 lines

But, when I have time for careful look, I'm going to write some LIO critics. So far, at the first glance:

- It is too iSCSI-centric. ISCSI is a very special transport, so looks
like when you decide to add in LIO drivers for other transports, especially for parallel SCSI and SAS, you are going to have big troubles and major redesign.

Not true. Because LIO-Core subsystem API is battle hardened (you could say it is the 2nd oldest, behind UNH's :), allocating LIO-Core SE tasks (that then get issued to LIO-Core subsystem plugins) from a SCSI CDB with sectors+offset for ICF_SCSI_DATA_SG_IO_CDB, or a generically emulated SCSI control CDB or logic in LIO-Core, or using LIO-Core/PSCSI to let the underlying hardware do its thing, but still fill in the holes so that *ANY* SCSI subsystem, including from different OSes, can talk with storage objects behind LIO-Core when running in initiator mode
amoungst the possible fabrics. Some of the classic examples here are:

*) Because the Solaris 10 SCSI subsystem requiring all iSCSI devices to have EVPD information, otherwise LUN registration would fail. This means that suddently struct block_device and struct file need to have WWN information, which may be DIFFERENT based upon if said object was a Linux/MD or LVM block device, for example.

*) Every cluster design that required block level shared storage needs to have at least SAM-2 Reservations.

*) Exporting via LIO-Core Hardware RAID adapters on OSes where max_sectors cannot be easily changed. This is because some Hardware RAID requires a smaller struct scsi_device->max_sector to handle smaller stripe sizes for their arrays.

*) Some adapters in drivers/scsi which are not REAL SCSI devices emulate none/some WWN or control logic mentioned above. I have had to do a couple of hacks over the years in LIO-Core/PSCSI to make everything place nice going to the client side of the cloud, check out iscsi_target_pscsi.c:pscsi_transport_complete() to see what I mean.

Quoted 2 lines

And this is a real showstopper for making LIO-Core
the default and the only SCSI target framework. SCST is SCSI-centric,

Well, one needs to understand that LIO-Core subsystem API is more than a SCSI target framework. Its a generic method of accessing any possible storage object of the storage stack, and having said engine handle the hardware restrictions (be they physical or virtual) for the underlying storage object. It can run as a SCSI engine to real (or emualted) SCSI hardware from linux/drivers/scsi, but the real strength is that it sits above the SCSI/BLOCK/FILE layers and uses a single codepath for all underlying storage objects. For example in the lio-core-2.6.git tree, I chose the location linux/drivers/lio-core, because LIO-Core uses 'struct file' from fs/, 'struct block_device' from block/ and struct scsi_device from drivers/scsi.

Its worth to note that I am still doing the re-org of LIO-Core and LIO-Target v3.0.0, but this will be coming soon along with the first non traditional iSCSI packets to run across LIO-Core.

Quoted 1 line

just because there's no way to make *SCSI* target framework not being SCSI-centric. Nobody blames Linux SCSI (initiator) mid-layer for being SCSI-centric, correct?

Well, as we have discussed before, the emulation of the SCSI control path is really a whole different monster, and I am certainly not interested in having to emulate all of the t10.org standards myself. :-)

Quoted 2 lines

- Seems, it's a bit overcomplicated, because it has too many abstract
interfaces where there's not much need it them. Having too many abstract interfaces makes code analyze a lot more complicated. For comparison, SCST has only 2 such interfaces: for target drivers and for backstorage dev handlers. Plus, there is half-abstract interface for memory allocator (sgv_pool_set_allocator()) to allow scst_user to allocate user space supplied pages. And they cover all needs.

Well, I have discussed why I think the LIO-Core design (which was more neccessity at the start) has been able to work with for all kernel subsystems/storage objects on all architectures for v2.2, v2.4 and v2.6 kernels. I also mention these at the 10,000 ft level in my LSF 08' pres.

Quoted 3 lines

- Pass-through mode (PSCSI) also provides non-enforced 1-to-1
relationship, as it used to be in STGT (now in STGT support for pass-through mode seems to be removed), which isn't mentioned anywhere.

Please be more specific by what you mean here. Also, note that because PSCSI is an LIO-Core subsystem plugin, LIO-Core handles the limitations of the storage object through the LIO-Core subsystem API. This means that things like (received initiator CDB sectors > LIO-Core storage object max_sectors) are handled generically by LIO-Core, using a single set of algoritims for all I/O interaction with Linux storage systems. These algoritims are also the same for DIFFERENT types of transport fabrics, both those that expect LIO-Core to allocate memory, OR that hardware will have preallocated memory and possible restrictions from the CPU/BUS architecture (take non-cache coherent MIPS for example) of how the memory gets DMA'ed or PIO'ed down to the packet's intended storage object.

Quoted 2 lines

- There is some confusion in the code in the function and variable
names between persistent and SAM-2 reservations.

Well, that would be because persistent reservations are not emulated generally for all of the subsystem plugins just yet. Obviously with LIO-Core/PSCSI if the underlying hardware supports it, it will work.

Quoted 4 lines

Quoted 2 lines

The more in fighting between the
leaders in our community, the less the community benefits.

Sure. If my note hurts you, I can remove it. But you should also remove from your presentation and the summary paper those psychological arguments to not confuse people.

Its not about removing, it is about updating the page to better reflect the bigger picture so folks coming to the sight can get the latest information from last update.

Many thanks for your most valuable of time,

--nab

Quoted 5 lines · Nicholas A. Bellinger

Quoted 4 lines

I have only documents, which I referenced. In them, especially
in "2008 Linux Storage & Filesystem Workshop" summary, it doesn't look as I took it out of context. You put emphasis on "older" vs "current"/"new", didn't you ;)?

Quoted 7 lines

Well, my job was to catch everyone up to speed on the status of the 4 (four) different (insert your favorite SAM capable transport name here) Linux v2.6 based target projects. With all of the acroynms for the standards+implementations+linux-kernel being extremly confusing to anyone who does know all of them by heart. Even those people in the room, who where fimilar with storage, but not necessarly with target mode engine design, its hard to follow.

Yes, this is a problem. Even storage experts are not too familiar with SCSI internals and not willing much to get better familiarity. Hence, almost nobody really understands for what is all those SCSI processing in SCST..

Quoted 6 lines

Quoted 5 lines

BTW, there are another inaccuracies on your slides:

- STGT doesn't support "hardware accelerated traditional iSCSI
(Qlogic)", at least I have not found any signs of it.

Quoted 7 lines

<nod>, that is correct. It does it's hardware acceleration generically using OFA VERBS for hardware that do wire protocol that implements fabric dependent direct data placement. iSER does this with 504[0-4],
and I don't recall exactly how IB does it. Anyways, the point is that
they use a single interface so that hardware vendors do not have to implement their own APIs, which are very complex, and usually very buggy when coming from a company who is trying to get a design into ASIC.

ISER is "iSCSI Extensions for RDMA", while usually under "hardware accelerated traditional iSCSI" people mean regular hardware iSCSI cards, like QLogic 4xxx. Hence, your sentence for most people, including myself, was incorrect and confusing.

Quoted 32 lines

Quoted 7 lines

But, when I have time for careful look, I'm going to write some LIO critics. So far, at the first glance:

- It is too iSCSI-centric. ISCSI is a very special transport, so looks
like when you decide to add in LIO drivers for other transports, especially for parallel SCSI and SAS, you are going to have big troubles and major redesign.

Not true. Because LIO-Core subsystem API is battle hardened (you could say it is the 2nd oldest, behind UNH's :), allocating LIO-Core SE tasks (that then get issued to LIO-Core subsystem plugins) from a SCSI CDB with sectors+offset for ICF_SCSI_DATA_SG_IO_CDB, or a generically emulated SCSI control CDB or logic in LIO-Core, or using LIO-Core/PSCSI to let the underlying hardware do its thing, but still fill in the holes so that *ANY* SCSI subsystem, including from different OSes, can talk with storage objects behind LIO-Core when running in initiator mode
amoungst the possible fabrics. Some of the classic examples here are:

*) Because the Solaris 10 SCSI subsystem requiring all iSCSI devices to have EVPD information, otherwise LUN registration would fail. This means that suddently struct block_device and struct file need to have WWN information, which may be DIFFERENT based upon if said object was a Linux/MD or LVM block device, for example.

*) Every cluster design that required block level shared storage needs to have at least SAM-2 Reservations.

*) Exporting via LIO-Core Hardware RAID adapters on OSes where max_sectors cannot be easily changed. This is because some Hardware RAID requires a smaller struct scsi_device->max_sector to handle smaller stripe sizes for their arrays.

Quoted 5 lines

*) Some adapters in drivers/scsi which are not REAL SCSI devices emulate none/some WWN or control logic mentioned above. I have had to do a couple of hacks over the years in LIO-Core/PSCSI to make everything place nice going to the client side of the cloud, check out iscsi_target_pscsi.c:pscsi_transport_complete() to see what I mean.

I meant something different: interface between target drivers and SCSI target core. Here (seems) you are going to have big troubles when you try to add not-iSCSI transport, like FC, for instance.

Quoted 14 lines

Quoted 2 lines

And this is a real showstopper for making LIO-Core
the default and the only SCSI target framework. SCST is SCSI-centric,

Well, one needs to understand that LIO-Core subsystem API is more than a SCSI target framework. Its a generic method of accessing any possible storage object of the storage stack, and having said engine handle the hardware restrictions (be they physical or virtual) for the underlying storage object. It can run as a SCSI engine to real (or emualted) SCSI hardware from linux/drivers/scsi, but the real strength is that it sits above the SCSI/BLOCK/FILE layers and uses a single codepath for all underlying storage objects. For example in the lio-core-2.6.git tree, I chose the location linux/drivers/lio-core, because LIO-Core uses 'struct file' from fs/, 'struct block_device' from block/ and struct scsi_device from drivers/scsi.

SCST and iSCSI-SCST, basically, do the same things, except iSCSI MC/S and related, + something more, like 1-to-many pass-through and scst_user, which need a big chunks of code, correct? And they are together about 2 times smaller:

$ find core-iscsi/svn/trunk/target/target -type f -name "*.[ch]"|xargs wc
59764  163202 1625877 total
+
$ find core-iscsi/svn/trunk/target/include -type f -name "*.[ch]"|xargs
2981  9316 91930 total
=
62745      1717807

vs

$ find svn/trunk/scst -type f -name "*.[ch]"|xargs wc
28327  77878 734625 total
+
$ find svn/trunk/iscsi-scst/kernel -type f -name "*.[ch]"|xargs wc
7857  20394 194693 total
=
36184        929318

Or did I count incorrectly?

Quoted 8 lines

Its worth to note that I am still doing the re-org of LIO-Core and LIO-Target v3.0.0, but this will be coming soon along with the first non traditional iSCSI packets to run across LIO-Core.

Quoted 3 lines

just because there's no way to make *SCSI* target framework not being SCSI-centric. Nobody blames Linux SCSI (initiator) mid-layer for being SCSI-centric, correct?

Quoted 4 lines

Well, as we have discussed before, the emulation of the SCSI control path is really a whole different monster, and I am certainly not interested in having to emulate all of the t10.org standards myself. :-)

Sure, there optional things. But there are also requirements, which must be followed. So, this isn't about interested or not, this is about must do or don't do at all.

Quoted 8 lines

Quoted 7 lines

- Seems, it's a bit overcomplicated, because it has too many abstract
interfaces where there's not much need it them. Having too many abstract interfaces makes code analyze a lot more complicated. For comparison, SCST has only 2 such interfaces: for target drivers and for backstorage dev handlers. Plus, there is half-abstract interface for memory allocator (sgv_pool_set_allocator()) to allow scst_user to allocate user space supplied pages. And they cover all needs.

Quoted 5 lines

Well, I have discussed why I think the LIO-Core design (which was more neccessity at the start) has been able to work with for all kernel subsystems/storage objects on all architectures for v2.2, v2.4 and v2.6 kernels. I also mention these at the 10,000 ft level in my LSF 08' pres.

Nobody in the Linux kernel community is interested to have obsolete or unneeded for the current kernel version code in the kernel, so if you want LIO core be in the kernel, you will have to make a major cleanup.

Also, see the above LIO vs SCST size comparison. Is the additional code all about the obsolete/currently unneeded features?

Quoted 5 lines

Quoted 4 lines

- Pass-through mode (PSCSI) also provides non-enforced 1-to-1
relationship, as it used to be in STGT (now in STGT support for pass-through mode seems to be removed), which isn't mentioned anywhere.

Quoted 12 lines

Please be more specific by what you mean here. Also, note that because PSCSI is an LIO-Core subsystem plugin, LIO-Core handles the limitations of the storage object through the LIO-Core subsystem API. This means that things like (received initiator CDB sectors > LIO-Core storage object max_sectors) are handled generically by LIO-Core, using a single set of algoritims for all I/O interaction with Linux storage systems. These algoritims are also the same for DIFFERENT types of transport fabrics, both those that expect LIO-Core to allocate memory, OR that hardware will have preallocated memory and possible restrictions from the CPU/BUS architecture (take non-cache coherent MIPS for example) of how the memory gets DMA'ed or PIO'ed down to the packet's intended storage object.

See here:
http://www.mail-archive.com/[email protected]/msg06911.html

Quoted 6 lines

Quoted 2 lines

- There is some confusion in the code in the function and variable
names between persistent and SAM-2 reservations.

Well, that would be because persistent reservations are not emulated generally for all of the subsystem plugins just yet. Obviously with LIO-Core/PSCSI if the underlying hardware supports it, it will work.

What you did (passing reservation commands directly to devices and nothing more) will work only with a single initiator per device, where reservations in the majority of cases are not needed at all.

With
multiple initiators, as it is in clusters and where reservations are really needed, it will sooner or later lead to data corruption. See the referenced above message as well as the whole thread.

Quoted 7 lines

Quoted 6 lines

Quoted 2 lines

The more in fighting between the
leaders in our community, the less the community benefits.

Sure. If my note hurts you, I can remove it. But you should also remove from your presentation and the summary paper those psychological arguments to not confuse people.

Quoted 3 lines

Its not about removing, it is about updating the page to better reflect the bigger picture so folks coming to the sight can get the latest information from last update.

Your suggestions?

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

And this is a real showstopper for making LIO-Core
the default and the only SCSI target framework. SCST is SCSI-centric,

Well, one needs to understand that LIO-Core subsystem API is more than a SCSI target framework. Its a generic method of accessing any possible storage object of the storage stack, and having said engine handle the hardware restrictions (be they physical or virtual) for the underlying storage object. It can run as a SCSI engine to real (or emualted) SCSI hardware from linux/drivers/scsi, but the real strength is that it sits above the SCSI/BLOCK/FILE layers and uses a single codepath for all underlying storage objects. For example in the lio-core-2.6.git tree, I chose the location linux/drivers/lio-core, because LIO-Core uses 'struct file' from fs/, 'struct block_device' from block/ and struct scsi_device from drivers/scsi.

Quoted 1 line

SCST and iSCSI-SCST, basically, do the same things, except iSCSI MC/S and related, + something more, like 1-to-many pass-through and scst_user, which need a big chunks of code, correct? And they are together about 2 times smaller:

Yes, something much more. A complete implementation of traditional iSCSI/TCP (known as RFC-3720), iSCSI/SCTP (which will be important in the future), and IPv6 (also important) is a significant amount of logic. When I say a 'complete implementation' I mean:

I) Active-Active connection layer recovery (known as ErrorRecoveryLevel=2). (We are going to use the same code for iSER for inter-nexus OS independent (eg: below the SCSI Initiator level) recovery. Again, the important part here is that recovery and outstanding task migration happens transparently to the host OS SCSI subsystem. This means (at least with iSCSI and iSER): not having to register multiple LUNs and depend (at least completely) on SCSI WWN information, and OS dependent SCSI level multipath.

II) MC/S for multiplexing (same as I), as well as being able to multiplex across multiple cards and subnets (using TCP, SCTP has multi-homing). Also being able to bring iSCSI connections up/down on the fly, until we all have iSCSI/SCTP, is very important too.

III) Every possible combination of RFC-3720 defined parameter keys (and provide the apparatis to prove it). And yes, anyone can do this today against their own Target. I created core-iscsi-dv specifically for testing LIO-Target <-> LIO-Core back in 2005. Core-iSCSI-DV is the _ONLY_ _PUBLIC_ RFC-3720 domain validation tool that will actually demonstrate, using ANY data integrity tool complete domain validation of user defined keys. Please have a look at:

http://linux-iscsi.org/index.php/Core-iscsi-dv

http://www.linux-iscsi.org/files/core-iscsi-dv/README

Any traditional iSCSI target mode implementation + Storage Engine + Subsystem Plugin that thinks its ready to go into the kernel will have to pass at LEAST the 8k test loop interations, the simplest being:

HeaderDigest, DataDigest, MaxRecvDataSegmentLength (512 -> 262144, in 512 byte increments)

Core-iSCSI-DV is also a great indication of stability and data integrity of hardware/software of an iSCSI Target + Engine, espically when you have multiple core-iscsi-dv nodes hitting multiple VHACS clouds on physical machines within the cluster. I have never run IET against core-iscsi-dv personally, and I don't think Ming or Ross has either. So until SOMEONE actually does this first, I think that iSCSI-SCST is more of an experiment for your our devel that a strong contender for Linux/iSCSI Target Mode.

Obviously not. Also, what I was talking about there was the strength and flexibility of the LIO-Core design (it even ran on the Playstation 2 at one point, http://linux-iscsi.org/index.php/Playstation2/iSCSI, when MIPS r5900 boots modern v2.6, then we will do it again with LIO :-)

Quoted 15 lines

Quoted 15 lines

Quoted 3 lines

- Pass-through mode (PSCSI) also provides non-enforced 1-to-1
relationship, as it used to be in STGT (now in STGT support for pass-through mode seems to be removed), which isn't mentioned anywhere.

Please be more specific by what you mean here. Also, note that because PSCSI is an LIO-Core subsystem plugin, LIO-Core handles the limitations of the storage object through the LIO-Core subsystem API. This means that things like (received initiator CDB sectors > LIO-Core storage object max_sectors) are handled generically by LIO-Core, using a single set of algoritims for all I/O interaction with Linux storage systems. These algoritims are also the same for DIFFERENT types of transport fabrics, both those that expect LIO-Core to allocate memory, OR that hardware will have preallocated memory and possible restrictions from the CPU/BUS architecture (take non-cache coherent MIPS for example) of how the memory gets DMA'ed or PIO'ed down to the packet's intended storage object.

Quoted 2 lines

See here: http://www.mail-archive.com/[email protected]/msg06911.html

<nod>

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

- There is some confusion in the code in the function and variable
names between persistent and SAM-2 reservations.

Well, that would be because persistent reservations are not emulated generally for all of the subsystem plugins just yet. Obviously with LIO-Core/PSCSI if the underlying hardware supports it, it will work.

Quoted 1 line

What you did (passing reservation commands directly to devices and nothing more) will work only with a single initiator per device, where reservations in the majority of cases are not needed at all.

I know, like I said, implementing Persistent Reservations for stuff besides real SCSI hardware with LIO-Core/PSCSI is a TODO item. Note that the VHACS cloud (see below) will need this for DRBD objects at some point.

Quoted 9 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

The more in fighting between the
leaders in our community, the less the community benefits.

Sure. If my note hurts you, I can remove it. But you should also remove from your presentation and the summary paper those psychological arguments to not confuse people.

Its not about removing, it is about updating the page to better reflect the bigger picture so folks coming to the sight can get the latest information from last update.

Your suggestions?

I would consider helping with this at some point, but as you can see, I am extremly busy ATM. I have looked at SCST quite a bit over the years, but I am not the one making a public comparision page, at least not yet. :-) So until then, at least explain how there are 3 projects on your page, with the updated 10,000 ft overviews, and mabye even add some links to LIO-Target and a bit about VHACS cloud. I would be willing to include info about SCST into the Linux-iSCSI.org wiki. Also, please feel free to open an account and start adding stuff about SCST yourself to the site.

For Linux-iSCSI.org and VHACS (which is really where everything is going now), please have a look at:

http://linux-iscsi.org/index.php/VHACS-VM
http://linux-iscsi.org/index.php/VHACS

Btw, the VHACS and LIO-Core design will allow for other fabrics to be used inside our cloud, and between other virtualized client setups who speak the wire protocol presented by the server side of VHACS cloud.

Many thanks for your most valuable of time,

--nab

Quoted 53 lines · Nicholas A. Bellinger

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

And this is a real showstopper for making LIO-Core
the default and the only SCSI target framework. SCST is SCSI-centric,

Well, one needs to understand that LIO-Core subsystem API is more than a SCSI target framework. Its a generic method of accessing any possible storage object of the storage stack, and having said engine handle the hardware restrictions (be they physical or virtual) for the underlying storage object. It can run as a SCSI engine to real (or emualted) SCSI hardware from linux/drivers/scsi, but the real strength is that it sits above the SCSI/BLOCK/FILE layers and uses a single codepath for all underlying storage objects. For example in the lio-core-2.6.git tree, I chose the location linux/drivers/lio-core, because LIO-Core uses 'struct file' from fs/, 'struct block_device' from block/ and struct scsi_device from drivers/scsi.

Quoted 4 lines

SCST and iSCSI-SCST, basically, do the same things, except iSCSI MC/S and related, + something more, like 1-to-many pass-through and scst_user, which need a big chunks of code, correct? And they are together about 2 times smaller:

Yes, something much more. A complete implementation of traditional iSCSI/TCP (known as RFC-3720), iSCSI/SCTP (which will be important in the future), and IPv6 (also important) is a significant amount of logic. When I say a 'complete implementation' I mean:

I) Active-Active connection layer recovery (known as ErrorRecoveryLevel=2). (We are going to use the same code for iSER for inter-nexus OS independent (eg: below the SCSI Initiator level) recovery. Again, the important part here is that recovery and outstanding task migration happens transparently to the host OS SCSI subsystem. This means (at least with iSCSI and iSER): not having to register multiple LUNs and depend (at least completely) on SCSI WWN information, and OS dependent SCSI level multipath.

II) MC/S for multiplexing (same as I), as well as being able to multiplex across multiple cards and subnets (using TCP, SCTP has multi-homing). Also being able to bring iSCSI connections up/down on the fly, until we all have iSCSI/SCTP, is very important too.

III) Every possible combination of RFC-3720 defined parameter keys (and provide the apparatis to prove it). And yes, anyone can do this today against their own Target. I created core-iscsi-dv specifically for testing LIO-Target <-> LIO-Core back in 2005. Core-iSCSI-DV is the _ONLY_ _PUBLIC_ RFC-3720 domain validation tool that will actually demonstrate, using ANY data integrity tool complete domain validation of user defined keys. Please have a look at:

http://linux-iscsi.org/index.php/Core-iscsi-dv

http://www.linux-iscsi.org/files/core-iscsi-dv/README

Any traditional iSCSI target mode implementation + Storage Engine + Subsystem Plugin that thinks its ready to go into the kernel will have to pass at LEAST the 8k test loop interations, the simplest being:

Quoted 8 lines

HeaderDigest, DataDigest, MaxRecvDataSegmentLength (512 -> 262144, in 512 byte increments)

Core-iSCSI-DV is also a great indication of stability and data integrity of hardware/software of an iSCSI Target + Engine, espically when you have multiple core-iscsi-dv nodes hitting multiple VHACS clouds on physical machines within the cluster. I have never run IET against core-iscsi-dv personally, and I don't think Ming or Ross has either.

So

Quoted 3 lines

until SOMEONE actually does this first, I think that iSCSI-SCST is more of an experiment for your our devel that a strong contender for Linux/iSCSI Target Mode.

There are big doubts among storage experts if features I and II are needed at all, see, e.g. http://lkml.org/lkml/2008/2/5/331.

I also tend
to agree, that for block storage on practice MC/S is not needed or, at least, definitely doesn't worth the effort, because:

1. It is useless for sync. untagged operation (regular reads in most cases over a single stream), when always there is only one command being executed at any time, because of the commands connection allegiance, which forbids transferring data for a command over multiple connections.

2. The only advantage it has over traditional OS multi-pathing is keeping commands execution order, but on practice at the moment there is no demand for this feature, because all OS'es I know don't rely on commands order to protect data integrity. They use other techniques, like queue draining. A good target should be able itself to scheduler coming commands for execution in the correct from performance POV order and not rely for that on the commands order as they came from initiators.

From other side, devices bonding also preserves commands execution
order, but doesn't suffer from the connection allegiance limitation of MC/S, so can boost performance ever for sync untagged operations. Plus, it's pretty simple, easy to use and doesn't need any additional code. I don't have the exact numbers of MC/S vs bonding performance comparison (mostly, because open-iscsi doesn't support MC/S, but very curious to see them), but have very strong suspicious that on modern OS'es, which do TCP frames reorder in zero-copy manner, there shouldn't be much performance difference between MC/S vs bonding in the maximum possible throughput, but bonding should outperform MC/S a lot in case of sync untagged operations.

Anyway, I think features I and II, if added, would increase iSCSI-SCST kernel side code not more than on 5K lines, because most of the code is already there, the most important part which missed is fixes of locking problems, which almost never add a lot of code.

Relating Core-iSCSI-DV,
I'm sure iSCSI-SCST will pass it without problems among the required set of iSCSI features, although still there are some limitations, derived from IET, for instance, support for multu-PDU commands in discovery sessions, which isn't implemented. But for adding to iSCSI-SCST optional iSCSI features there should be good *practical* reasons, which at the moment don't exist. And unused features are bad features, because they overcomplicate the code and make its maintainance harder for no gain.

So, current SCST+iSCSI-SCST 36K lines + 5K new lines = 41K lines, which still a lot less than LIO's 63K lines. I downloaded the cleanuped lio-core-2.6.git tree and:

$ find lio-core-2.6/drivers/lio-core -type f -name "*.[ch]"|xargs wc
57064  156617 1548344 total

Still much bigger.

Quoted 4 lines

Obviously not. Also, what I was talking about there was the strength and flexibility of the LIO-Core design (it even ran on the Playstation 2 at one point, http://linux-iscsi.org/index.php/Playstation2/iSCSI, when MIPS r5900 boots modern v2.6, then we will do it again with LIO :-)

SCST and the target drivers have been successfully ran on PPC and Sparc64, so I don't see any reasons, why it can't be ran on Playstation 2 as well.

Quoted 27 lines

Quoted 16 lines

Quoted 4 lines

Quoted 4 lines

- Pass-through mode (PSCSI) also provides non-enforced 1-to-1
relationship, as it used to be in STGT (now in STGT support for pass-through mode seems to be removed), which isn't mentioned anywhere.

Quoted 12 lines

Please be more specific by what you mean here. Also, note that because PSCSI is an LIO-Core subsystem plugin, LIO-Core handles the limitations of the storage object through the LIO-Core subsystem API. This means that things like (received initiator CDB sectors > LIO-Core storage object max_sectors) are handled generically by LIO-Core, using a single set of algoritims for all I/O interaction with Linux storage systems. These algoritims are also the same for DIFFERENT types of transport fabrics, both those that expect LIO-Core to allocate memory, OR that hardware will have preallocated memory and possible restrictions from the CPU/BUS architecture (take non-cache coherent MIPS for example) of how the memory gets DMA'ed or PIO'ed down to the packet's intended storage object.

Quoted 3 lines

See here:
http://www.mail-archive.com/[email protected]/msg06911.html

<nod>

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

- There is some confusion in the code in the function and variable
names between persistent and SAM-2 reservations.

Well, that would be because persistent reservations are not emulated generally for all of the subsystem plugins just yet. Obviously with LIO-Core/PSCSI if the underlying hardware supports it, it will work.

Quoted 8 lines

Quoted 3 lines

What you did (passing reservation commands directly to devices and nothing more) will work only with a single initiator per device, where reservations in the majority of cases are not needed at all.

I know, like I said, implementing Persistent Reservations for stuff besides real SCSI hardware with LIO-Core/PSCSI is a TODO item. Note that the VHACS cloud (see below) will need this for DRBD objects at some point.

The problem is that persistent reservations don't work for multiple initiators even for real SCSI hardware with LIO-Core/PSCSI and I clearly described why in the referenced e-mail. Nicholas, why don't you want to see it?

Quoted 28 lines

Quoted 11 lines

Quoted 9 lines

Quoted 6 lines

Quoted 2 lines

The more in fighting between the
leaders in our community, the less the community benefits.

Sure. If my note hurts you, I can remove it. But you should also remove from your presentation and the summary paper those psychological arguments to not confuse people.

Its not about removing, it is about updating the page to better reflect the bigger picture so folks coming to the sight can get the latest information from last update.

Your suggestions?

I would consider helping with this at some point, but as you can see, I am extremly busy ATM. I have looked at SCST quite a bit over the years, but I am not the one making a public comparision page, at least not yet. :-) So until then, at least explain how there are 3 projects on your page, with the updated 10,000 ft overviews, and mabye even add some links to LIO-Target and a bit about VHACS cloud. I would be willing to include info about SCST into the Linux-iSCSI.org wiki. Also, please feel free to open an account and start adding stuff about SCST yourself to the site.

For Linux-iSCSI.org and VHACS (which is really where everything is going now), please have a look at:

http://linux-iscsi.org/index.php/VHACS-VM
http://linux-iscsi.org/index.php/VHACS

Quoted 6 lines

Btw, the VHACS and LIO-Core design will allow for other fabrics to be used inside our cloud, and between other virtualized client setups who speak the wire protocol presented by the server side of VHACS cloud.

Many thanks for your most valuable of time,

Using MC/S with a modern iSCSI implementation to take advantage of lots of cores and hardware threads is something that allows one to multiplex across multiple vendor's NIC ports, with the least possible overhead, in the OS INDEPENDENT manner.

Quoted 57 lines · Vladislav Bolkhovitin

Quoted 56 lines · Nicholas A. Bellinger

Quoted 13 lines

Quoted 13 lines

Quoted 2 lines

And this is a real showstopper for making LIO-Core
the default and the only SCSI target framework. SCST is SCSI-centric,

Well, one needs to understand that LIO-Core subsystem API is more than a SCSI target framework. Its a generic method of accessing any possible storage object of the storage stack, and having said engine handle the hardware restrictions (be they physical or virtual) for the underlying storage object. It can run as a SCSI engine to real (or emualted) SCSI hardware from linux/drivers/scsi, but the real strength is that it sits above the SCSI/BLOCK/FILE layers and uses a single codepath for all underlying storage objects. For example in the lio-core-2.6.git tree, I chose the location linux/drivers/lio-core, because LIO-Core uses 'struct file' from fs/, 'struct block_device' from block/ and struct scsi_device from drivers/scsi.

Quoted 4 lines

SCST and iSCSI-SCST, basically, do the same things, except iSCSI MC/S and related, + something more, like 1-to-many pass-through and scst_user, which need a big chunks of code, correct? And they are together about 2 times smaller:

Yes, something much more. A complete implementation of traditional iSCSI/TCP (known as RFC-3720), iSCSI/SCTP (which will be important in the future), and IPv6 (also important) is a significant amount of logic. When I say a 'complete implementation' I mean:

I) Active-Active connection layer recovery (known as ErrorRecoveryLevel=2). (We are going to use the same code for iSER for inter-nexus OS independent (eg: below the SCSI Initiator level) recovery. Again, the important part here is that recovery and outstanding task migration happens transparently to the host OS SCSI subsystem. This means (at least with iSCSI and iSER): not having to register multiple LUNs and depend (at least completely) on SCSI WWN information, and OS dependent SCSI level multipath.

II) MC/S for multiplexing (same as I), as well as being able to multiplex across multiple cards and subnets (using TCP, SCTP has multi-homing). Also being able to bring iSCSI connections up/down on the fly, until we all have iSCSI/SCTP, is very important too.

III) Every possible combination of RFC-3720 defined parameter keys (and provide the apparatis to prove it). And yes, anyone can do this today against their own Target. I created core-iscsi-dv specifically for testing LIO-Target <-> LIO-Core back in 2005. Core-iSCSI-DV is the _ONLY_ _PUBLIC_ RFC-3720 domain validation tool that will actually demonstrate, using ANY data integrity tool complete domain validation of user defined keys. Please have a look at:

http://linux-iscsi.org/index.php/Core-iscsi-dv

http://www.linux-iscsi.org/files/core-iscsi-dv/README

Any traditional iSCSI target mode implementation + Storage Engine + Subsystem Plugin that thinks its ready to go into the kernel will have to pass at LEAST the 8k test loop interations, the simplest being:

HeaderDigest, DataDigest, MaxRecvDataSegmentLength (512 -> 262144, in 512 byte increments)

Quoted 5 lines

Quoted 5 lines

Core-iSCSI-DV is also a great indication of stability and data integrity of hardware/software of an iSCSI Target + Engine, espically when you have multiple core-iscsi-dv nodes hitting multiple VHACS clouds on physical machines within the cluster. I have never run IET against core-iscsi-dv personally, and I don't think Ming or Ross has either.

Ming or Ross, would you like to make a comment on this, considering after it, it is your work..?

Quoted 8 lines

So

Quoted 3 lines

until SOMEONE actually does this first, I think that iSCSI-SCST is more of an experiment for your our devel that a strong contender for Linux/iSCSI Target Mode.

There are big doubts among storage experts if features I and II are needed at all, see, e.g. http://lkml.org/lkml/2008/2/5/331.

Well, jgarzik is both a NETWORKING and STORAGE (he was a networking guy first, mind you) expert!

Quoted 4 lines

I also tend
to agree, that for block storage on practice MC/S is not needed or, at least, definitely doesn't worth the effort, because:

Trying to agrue against MC/S (or against any other major part of RFC-3720, including ERL=2) is saying that Linux/iSCSI should be BEHIND what the greatest minds in the IETF have produced (and learned) from iSCSI. Considering so many people are interested in seeing Linux/iSCSI be best and most complete implementation possible, surely one would not be foolish enough to try to debate that Linux should be BEHIND what others have figured out, be it with RFCs or running code.

Also, you should understand that MC/S is more than about just moving data I/O across multiple TCP connections, its about being able to bring those paths up/down on the fly without having to actually STOP/PAUSE anything. Then you then add the ERL=2 pixie dust, which you should understand, is the result of over a decade of work creating RFC-3720 within the IETF IPS TWG. What you have is a fabric that does not STOP/PAUSE from an OS INDEPENDENT LEVEL (below the OS dependent SCSI subsystem layer) perspective, on every possible T/I node, big and small, open or closed platform. Even as we move towards more logic in the network layer (a la Stream Control Transmission Protocol), we will still benefit from RFC-3720 as the years roll on. Quite a powerful thing..

Quoted 5 lines

1. It is useless for sync. untagged operation (regular reads in most cases over a single stream), when always there is only one command being executed at any time, because of the commands connection allegiance, which forbids transferring data for a command over multiple connections.

This is a very Parallel SCSI centric way of looking at design of SAM. Since SAM allows the transport fabric to enforce its own ordering rules (it does offer some of its own SCSI level ones of course). Obviously each fabric (PSCSI, FC, SAS, iSCSI) are very different from the bus phase perspective. But, if you look back into the history of iSCSI, you will see that an asymmetric design with seperate CONTROL/DATA TCP connections was considered originally BEFORE the Command Sequence Number (CmdSN) ordering algoritim was adopted that allows both SINGLE and MULTIPLE TCP connections to move both CONTROL/DATA packets across a iSCSI Nexus.

Using MC/S with a modern iSCSI implementation to take advantage of lots of cores and hardware threads is something that allows one to multiplex across multiple vendor's NIC ports, with the least possible overhead, in the OS INDEPENDENT manner. Keep in mind that you can do the allocation and RX of WRITE data OOO, but the actual *EXECUTION* down via the subsystem API (which is what LIO-Target <-> LIO-Core does, in a generic way) MUST BE in the same over as the CDBs came from the iSCSI Initiator port. This is the only requirement for iSCSI CmdSN order rules wrt the SCSI Architecture Model.

Quoted 8 lines

2. The only advantage it has over traditional OS multi-pathing is keeping commands execution order, but on practice at the moment there is no demand for this feature, because all OS'es I know don't rely on commands order to protect data integrity. They use other techniques, like queue draining. A good target should be able itself to scheduler coming commands for execution in the correct from performance POV order and not rely for that on the commands order as they came from initiators.

Ok, you are completely missing the point of MC/S and ERL=2. Notice how it works in both iSCSI *AND* iSER (even across DDP fabrics!). I discussed the significant benefit of ERL=2 in numerious previous threads. But they can all be neatly summerized in:

http://linux-iscsi.org/builds/user/nab/Inter.vs.OuterNexus.Multiplexing.pdf

Internexus Multiplexing is DESIGNED to work with OS dependent multipath transparently, and as a matter of fact, it complements it quite well, in a OSI (independent) method. Its completely up to the admin to determine the benefit and configure the knobs.

So, the bit: "We should not implement this important part of the RFC just because I want some code in the kernel" is not going to get your design very far.

Quoted 12 lines

From other side, devices bonding also preserves commands execution
order, but doesn't suffer from the connection allegiance limitation of MC/S, so can boost performance ever for sync untagged operations. Plus, it's pretty simple, easy to use and doesn't need any additional code. I don't have the exact numbers of MC/S vs bonding performance comparison (mostly, because open-iscsi doesn't support MC/S, but very curious to see them), but have very strong suspicious that on modern OS'es, which do TCP frames reorder in zero-copy manner, there shouldn't be much performance difference between MC/S vs bonding in the maximum possible throughput, but bonding should outperform MC/S a lot in case of sync untagged operations.

Simple case here for you to get your feet wet with MC/S. Try doing bonding across 4x GB/sec ports on 2x socket 2x core x86_64 and compare MC/S vs. OS dependent networking bonding and see what you find. There about two iSCSI initiators for two OSes that implementing MC/S and LIO-Target <-> LIO-Target. Anyone interested in the CPU overhead on this setup between MC/S and Link Layer bonding across 2x 2x 1 Gb/sec port chips on 4 core x86_64..?

Quoted 4 lines

Anyway, I think features I and II, if added, would increase iSCSI-SCST kernel side code not more than on 5K lines, because most of the code is already there, the most important part which missed is fixes of locking problems, which almost never add a lot of code.

You can think whatever you want. Why don't you have a look at lio-core-2.6.git and see how big they are for yourself.

Quoted 9 lines

Relating Core-iSCSI-DV,
I'm sure iSCSI-SCST will pass it without problems among the required set of iSCSI features, although still there are some limitations, derived from IET, for instance, support for multu-PDU commands in discovery sessions, which isn't implemented. But for adding to iSCSI-SCST optional iSCSI features there should be good *practical* reasons, which at the moment don't exist. And unused features are bad features, because they overcomplicate the code and make its maintainance harder for no gain.

Again, you can think whatever you want. But since you did not implement the majority of the iSCSI-SCST code yourself, (or implement your own iSCSI Initiator in parallel with your own iSCSI Target), I do not believe you are in a position to say. Any IET devs want to comment on this..?

Quoted 4 lines

So, current SCST+iSCSI-SCST 36K lines + 5K new lines = 41K lines, which still a lot less than LIO's 63K lines. I downloaded the cleanuped lio-core-2.6.git tree and:

Blindly comparing lines of code with no context is usually dumb. But, since that is what you seem to be stuck on, how about this:

LIO 63k +
SCST (minus iSCSI) ??k +
iSER from STGT ??k ==

For the complete LIO-Core engine on fabrics, and which includes what Rafiu from Openfiler has been so kind to call LIO-Target, "arguably the most feature complete and mature implementation out there (on any platform) "

Quoted 5 lines

$ find lio-core-2.6/drivers/lio-core -type f -name "*.[ch]"|xargs wc
57064  156617 1548344 total

Still much bigger.

Quoted 9 lines

Quoted 4 lines

Obviously not. Also, what I was talking about there was the strength and flexibility of the LIO-Core design (it even ran on the Playstation 2 at one point, http://linux-iscsi.org/index.php/Playstation2/iSCSI, when MIPS r5900 boots modern v2.6, then we will do it again with LIO :-)

SCST and the target drivers have been successfully ran on PPC and Sparc64, so I don't see any reasons, why it can't be ran on Playstation 2 as well.

Oh it can, can it..? Does your engine memory allocation algoritim provide for a SINGLE method for allocating linked list scatterlists containing page links of ANY (not just PAGE_SIZE) size handled generically across both internal or preregistered memory allocation acases, or coming from say, a software RNIC moving DDP packets for iSCSI in a single code path..?

And then it needs to be able to go down to the PS2-Linux PATA driver,
that does not show up under the SCSI subsystem mind you. Surely you
understand that because the MIPS r5900 is a non cache coherent architecture that you simply cannot allocate out multiple page contigious scatterlists for your I/Os, and simply expect it to work when we are sending blocks down to the 32-bit MIPS r3000 IOP..?

Quoted 36 lines

Quoted 27 lines

Quoted 16 lines

Quoted 4 lines

Quoted 4 lines

- Pass-through mode (PSCSI) also provides non-enforced 1-to-1
relationship, as it used to be in STGT (now in STGT support for pass-through mode seems to be removed), which isn't mentioned anywhere.

Quoted 12 lines

Please be more specific by what you mean here. Also, note that because PSCSI is an LIO-Core subsystem plugin, LIO-Core handles the limitations of the storage object through the LIO-Core subsystem API. This means that things like (received initiator CDB sectors > LIO-Core storage object max_sectors) are handled generically by LIO-Core, using a single set of algoritims for all I/O interaction with Linux storage systems. These algoritims are also the same for DIFFERENT types of transport fabrics, both those that expect LIO-Core to allocate memory, OR that hardware will have preallocated memory and possible restrictions from the CPU/BUS architecture (take non-cache coherent MIPS for example) of how the memory gets DMA'ed or PIO'ed down to the packet's intended storage object.

Quoted 3 lines

See here:
http://www.mail-archive.com/[email protected]/msg06911.html

<nod>

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

- There is some confusion in the code in the function and variable
names between persistent and SAM-2 reservations.

Well, that would be because persistent reservations are not emulated generally for all of the subsystem plugins just yet. Obviously with LIO-Core/PSCSI if the underlying hardware supports it, it will work.

Quoted 8 lines

Quoted 3 lines

What you did (passing reservation commands directly to devices and nothing more) will work only with a single initiator per device, where reservations in the majority of cases are not needed at all.

I know, like I said, implementing Persistent Reservations for stuff besides real SCSI hardware with LIO-Core/PSCSI is a TODO item. Note that the VHACS cloud (see below) will need this for DRBD objects at some point.

Quoted 5 lines

The problem is that persistent reservations don't work for multiple initiators even for real SCSI hardware with LIO-Core/PSCSI and I clearly described why in the referenced e-mail. Nicholas, why don't you want to see it?

Why don't you provide a reference in the code to where you think the problem is, and/or problem case using Linux iSCSI Initiators VMs to demonstrate the bug..?

Quoted 28 lines

Quoted 28 lines

Quoted 11 lines

Quoted 9 lines

Quoted 6 lines

Quoted 2 lines

The more in fighting between the
leaders in our community, the less the community benefits.

Sure. If my note hurts you, I can remove it. But you should also remove from your presentation and the summary paper those psychological arguments to not confuse people.

Its not about removing, it is about updating the page to better reflect the bigger picture so folks coming to the sight can get the latest information from last update.

Your suggestions?

I would consider helping with this at some point, but as you can see, I am extremly busy ATM. I have looked at SCST quite a bit over the years, but I am not the one making a public comparision page, at least not yet. :-) So until then, at least explain how there are 3 projects on your page, with the updated 10,000 ft overviews, and mabye even add some links to LIO-Target and a bit about VHACS cloud. I would be willing to include info about SCST into the Linux-iSCSI.org wiki. Also, please feel free to open an account and start adding stuff about SCST yourself to the site.

For Linux-iSCSI.org and VHACS (which is really where everything is going now), please have a look at:

http://linux-iscsi.org/index.php/VHACS-VM
http://linux-iscsi.org/index.php/VHACS

Quoted 6 lines

Quoted 6 lines

Btw, the VHACS and LIO-Core design will allow for other fabrics to be used inside our cloud, and between other virtualized client setups who speak the wire protocol presented by the server side of VHACS cloud.

Many thanks for your most valuable of time,

New v0.8.15 VHACS-VM images online btw. Keep checking the site for more details.

Many thanks for your most valuable of time,

--nab

Quoted 5 lines

So

Quoted 3 lines

until SOMEONE actually does this first, I think that iSCSI-SCST is more of an experiment for your our devel that a strong contender for Linux/iSCSI Target Mode.

There are big doubts among storage experts if features I and II are needed at all, see, e.g. http://lkml.org/lkml/2008/2/5/331.

Well, jgarzik is both a NETWORKING and STORAGE (he was a networking guy first, mind you) expert!

Quoted 2 lines

I also tend
to agree, that for block storage on practice MC/S is not needed or, at least, definitely doesn't worth the effort, because:

Trying to agrue against MC/S (or against any other major part of RFC-3720, including ERL=2) is saying that Linux/iSCSI should be BEHIND what the greatest minds in the IETF have produced (and learned) from iSCSI. Considering so many people are interested in seeing Linux/iSCSI be best and most complete implementation possible, surely one would not be foolish enough to try to debate that Linux should be BEHIND what others have figured out, be it with RFCs or running code.

Also, you should understand that MC/S is more than about just moving data I/O across multiple TCP connections, its about being able to bring those paths up/down on the fly without having to actually STOP/PAUSE anything. Then you then add the ERL=2 pixie dust, which you should understand, is the result of over a decade of work creating RFC-3720 within the IETF IPS TWG. What you have is a fabric that does not STOP/PAUSE from an OS INDEPENDENT LEVEL (below the OS dependent SCSI subsystem layer) perspective, on every possible T/I node, big and small, open or closed platform. Even as we move towards more logic in the network layer (a la Stream Control Transmission Protocol), we will still benefit from RFC-3720 as the years roll on. Quite a powerful thing..

Quoted 1 line

1. It is useless for sync. untagged operation (regular reads in most cases over a single stream), when always there is only one command being executed at any time, because of the commands connection allegiance, which forbids transferring data for a command over multiple connections.

This is a very Parallel SCSI centric way of looking at design of SAM. Since SAM allows the transport fabric to enforce its own ordering rules (it does offer some of its own SCSI level ones of course). Obviously each fabric (PSCSI, FC, SAS, iSCSI) are very different from the bus phase perspective. But, if you look back into the history of iSCSI, you will see that an asymmetric design with seperate CONTROL/DATA TCP connections was considered originally BEFORE the Command Sequence Number (CmdSN) ordering algoritim was adopted that allows both SINGLE and MULTIPLE TCP connections to move both CONTROL/DATA packets across a iSCSI Nexus.

Using MC/S with a modern iSCSI implementation to take advantage of lots of cores and hardware threads is something that allows one to multiplex across multiple vendor's NIC ports, with the least possible overhead, in the OS INDEPENDENT manner. Keep in mind that you can do the allocation and RX of WRITE data OOO, but the actual *EXECUTION* down via the subsystem API (which is what LIO-Target <-> LIO-Core does, in a generic way) MUST BE in the same over as the CDBs came from the iSCSI Initiator port. This is the only requirement for iSCSI CmdSN order rules wrt the SCSI Architecture Model.

Quoted 2 lines

2. The only advantage it has over traditional OS multi-pathing is keeping commands execution order, but on practice at the moment there is no demand for this feature, because all OS'es I know don't rely on commands order to protect data integrity. They use other techniques, like queue draining. A good target should be able itself to scheduler coming commands for execution in the correct from performance POV order and not rely for that on the commands order as they came from initiators.

Ok, you are completely missing the point of MC/S and ERL=2. Notice how it works in both iSCSI *AND* iSER (even across DDP fabrics!). I discussed the significant benefit of ERL=2 in numerious previous threads. But they can all be neatly summerized in:

http://linux-iscsi.org/builds/user/nab/Inter.vs.OuterNexus.Multiplexing.pdf

Internexus Multiplexing is DESIGNED to work with OS dependent multipath transparently, and as a matter of fact, it complements it quite well, in a OSI (independent) method. Its completely up to the admin to determine the benefit and configure the knobs.

Quoted 2 lines

From other side, devices bonding also preserves commands execution
order, but doesn't suffer from the connection allegiance limitation of MC/S, so can boost performance ever for sync untagged operations. Plus, it's pretty simple, easy to use and doesn't need any additional code. I don't have the exact numbers of MC/S vs bonding performance comparison (mostly, because open-iscsi doesn't support MC/S, but very curious to see them), but have very strong suspicious that on modern OS'es, which do TCP frames reorder in zero-copy manner, there shouldn't be much performance difference between MC/S vs bonding in the maximum possible throughput, but bonding should outperform MC/S a lot in case of sync untagged operations.

Simple case here for you to get your feet wet with MC/S. Try doing bonding across 4x GB/sec ports on 2x socket 2x core x86_64 and compare MC/S vs. OS dependent networking bonding and see what you find. There about two iSCSI initiators for two OSes that implementing MC/S and LIO-Target <-> LIO-Target. Anyone interested in the CPU overhead on this setup between MC/S and Link Layer bonding across 2x 2x 1 Gb/sec port chips on 4 core x86_64..?

Quoted 1 line

Anyway, I think features I and II, if added, would increase iSCSI-SCST kernel side code not more than on 5K lines, because most of the code is already there, the most important part which missed is fixes of locking problems, which almost never add a lot of code.

You can think whatever you want. Why don't you have a look at lio-core-2.6.git and see how big they are for yourself.

Quoted 2 lines

Relating Core-iSCSI-DV,
I'm sure iSCSI-SCST will pass it without problems among the required set of iSCSI features, although still there are some limitations, derived from IET, for instance, support for multu-PDU commands in discovery sessions, which isn't implemented. But for adding to iSCSI-SCST optional iSCSI features there should be good *practical* reasons, which at the moment don't exist. And unused features are bad features, because they overcomplicate the code and make its maintainance harder for no gain.

Again, you can think whatever you want. But since you did not implement the majority of the iSCSI-SCST code yourself, (or implement your own iSCSI Initiator in parallel with your own iSCSI Target), I do not believe you are in a position to say. Any IET devs want to comment on this..?

Quoted 29 lines

Quoted 29 lines

Quoted 15 lines

Quoted 15 lines

Quoted 3 lines

- Pass-through mode (PSCSI) also provides non-enforced 1-to-1
relationship, as it used to be in STGT (now in STGT support for pass-through mode seems to be removed), which isn't mentioned anywhere.

Please be more specific by what you mean here. Also, note that because PSCSI is an LIO-Core subsystem plugin, LIO-Core handles the limitations of the storage object through the LIO-Core subsystem API. This means that things like (received initiator CDB sectors > LIO-Core storage object max_sectors) are handled generically by LIO-Core, using a single set of algoritims for all I/O interaction with Linux storage systems. These algoritims are also the same for DIFFERENT types of transport fabrics, both those that expect LIO-Core to allocate memory, OR that hardware will have preallocated memory and possible restrictions from the CPU/BUS architecture (take non-cache coherent MIPS for example) of how the memory gets DMA'ed or PIO'ed down to the packet's intended storage object.

Quoted 2 lines

See here: http://www.mail-archive.com/[email protected]/msg06911.html

<nod>

Quoted 6 lines

Quoted 5 lines

Quoted 2 lines

- There is some confusion in the code in the function and variable
names between persistent and SAM-2 reservations.

Well, that would be because persistent reservations are not emulated generally for all of the subsystem plugins just yet. Obviously with LIO-Core/PSCSI if the underlying hardware supports it, it will work.

What you did (passing reservation commands directly to devices and nothing more) will work only with a single initiator per device, where reservations in the majority of cases are not needed at all.

I know, like I said, implementing Persistent Reservations for stuff besides real SCSI hardware with LIO-Core/PSCSI is a TODO item. Note that the VHACS cloud (see below) will need this for DRBD objects at some point.

Quoted 1 line

The problem is that persistent reservations don't work for multiple initiators even for real SCSI hardware with LIO-Core/PSCSI and I clearly described why in the referenced e-mail. Nicholas, why don't you want to see it?

Why don't you provide a reference in the code to where you think the problem is, and/or problem case using Linux iSCSI Initiators VMs to demonstrate the bug..?

Quoted 25 lines

Quoted 25 lines

Quoted 9 lines

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

The more in fighting between the
leaders in our community, the less the community benefits.

Sure. If my note hurts you, I can remove it. But you should also remove from your presentation and the summary paper those psychological arguments to not confuse people.

Its not about removing, it is about updating the page to better reflect the bigger picture so folks coming to the sight can get the latest information from last update.

Your suggestions?

I would consider helping with this at some point, but as you can see, I am extremly busy ATM. I have looked at SCST quite a bit over the years, but I am not the one making a public comparision page, at least not yet. :-) So until then, at least explain how there are 3 projects on your page, with the updated 10,000 ft overviews, and mabye even add some links to LIO-Target and a bit about VHACS cloud. I would be willing to include info about SCST into the Linux-iSCSI.org wiki. Also, please feel free to open an account and start adding stuff about SCST yourself to the site.

For Linux-iSCSI.org and VHACS (which is really where everything is going now), please have a look at:

http://linux-iscsi.org/index.php/VHACS-VM
http://linux-iscsi.org/index.php/VHACS

Quoted 6 lines

Quoted 6 lines

Btw, the VHACS and LIO-Core design will allow for other fabrics to be used inside our cloud, and between other virtualized client setups who speak the wire protocol presented by the server side of VHACS cloud.

Many thanks for your most valuable of time,

New v0.8.15 VHACS-VM images online btw. Keep checking the site for more details.

Many thanks for your most valuable of time,

--nab