: Comparison of features between different SCSI targets (SCST, STGT, IET, LIO) updated
Hello,
I updated the Linux SCSI targets comparison page http://scst.sourceforge.net/comparison.html, which compares features of the existing Linux SCSI target subsystems. The comparison includes SCST, STGT, IET and LIO. I added IET there, because it is the most used Linux iSCSI target at the moment.
This page is intended to be a complete and fair feature-by-feature comparison between the targets. The only thing I have not added there is quality of various corner cases recovery, because I don't know how to measure that. SCST is really good in this area, while some other targets are often not too much, but I have no idea how to summarize it in an objective view.
If you see I'm wrong somewhere or forgot something, you are welcome to correct me and I will fix that.
Vlad
Quoted 7 lines · Vladislav Bolkhovitin
Hello,
I updated the Linux SCSI targets comparison page
http://scst.sourceforge.net/comparison.html, which compares features of the existing Linux SCSI target subsystems. The comparison includes SCST, STGT, IET and LIO. I added IET there, because it is the most used Linux iSCSI target at the moment.
.
.
Quoted 6 lines
If you see I'm wrong somewhere or forgot something, you are welcome to correct me and I will fix that.
Vlad
as a user following the potential inclusion of a kernel-space target, iscsi specifically, I would be very interested in seeing what other pluses the other frameworks have over scst, because if this chart is accurate, all the other targets have quite a ways to go to catch up.
To me, lacking correct reserve and task management means "not ready for public consumption". Making any kernel change that does not have RESERVE/RELEASE and full TM command support is only going to make Linux look buggy and amateur-ish in the storage world. Best to not change anything than to implement something without them. These 2 features alone are what helped me choose SCST in the first place.
Quoted 14 lines · Chris Weiss
Quoted 7 lines · Vladislav Bolkhovitin
Hello,
I updated the Linux SCSI targets comparison page
http://scst.sourceforge.net/comparison.html, which compares features of the existing Linux SCSI target subsystems. The comparison includes SCST, STGT, IET and LIO. I added IET there, because it is the most used Linux iSCSI target at the moment..
.Quoted 4 lines
If you see I'm wrong somewhere or forgot something, you are welcome to correct me and I will fix that.
Quoted 8 lines
Quoted 2 lines
Vlad
as a user following the potential inclusion of a kernel-space target, iscsi specifically, I would be very interested in seeing what other pluses the other frameworks have over scst, because if this chart is accurate, all the other targets have quite a ways to go to catch up.
Actually sorry, anyone who has spent more than 30 minutes looking at the TCM v4 code that has already been posted to linux-scsi on monday knows this list just more handwaving. I suggest you start doing the same (actually discussion specific source file + line refrences) unless you actually want to trust this hopelessly out-of-date list on blind princaple.
Quoted 4 lines
To me, lacking correct reserve and task management means "not ready for public consumption". Making any kernel change that does not have RESERVE/RELEASE and full TM command support is only going to make Linux look buggy and amateur-ish in the storage world.
First, understand that Vlad has been asked to produce a problem use case for his CRH=1 (Compatibility Reservation Handling) concerns using the SPC-3 RESERVE/RELEASE methods with the TCM v4 code. He has been never been able to produce a use case, ever. Also, just for reference, does SCST's SPC-3 persisent reservation handling actually properly support CRH=1 emulation from spc4r17..? Last time I checked, it most certainly did *not*.
Second, in terms of TM emulation / passthrough support in the TCM v4 code, we follow what is implemented in drivers/scsi ML and LLDs, primarly to properly for Linux SCSI Initiators. I honestly don't have alot of interesting currently in implementing all of the ancient TM emulation that none of the mainline SCSI LLDs in Linux implement today, or plan to do the future. As for specific TM concerns, I am happy to address then on a case by case basis with the appropiate use case.
Best,
--nab
Quoted 6 lines · Nicholas A. Bellinger
Quoted 5 lines
as a user following the potential inclusion of a kernel-space target, iscsi specifically, I would be very interested in seeing what other pluses the other frameworks have over scst, because if this chart is accurate, all the other targets have quite a ways to go to catch up.
Quoted 6 lines
Actually sorry, anyone who has spent more than 30 minutes looking at the TCM v4 code that has already been posted to linux-scsi on monday knows this list just more handwaving. I suggest you start doing the same (actually discussion specific source file + line refrences) unless you actually want to trust this hopelessly out-of-date list on blind princaple.
I'm just an end user, I have no interesting in line-by-line comparisons or specs details. I do, however, have interest in bullet points, and the lack of bullet points, that throw up red flags to 3rd party support contracts. So, did the kernel team, or some other impartial source, compile a similar but not out of date list, and is it available for viewing?
Quoted 7 lines
First, understand that Vlad has been asked to produce a problem use case for his CRH=1 (Compatibility Reservation Handling) concerns using the SPC-3 RESERVE/RELEASE methods with the TCM v4 code. He has been never been able to produce a use case, ever. Also, just for reference, does SCST's SPC-3 persisent reservation handling actually properly support CRH=1 emulation from spc4r17..? Last time I checked, it most certainly did *not*.
I haven't tested this specifically, and I'm not even sure that the use case is right, but ESX (the only initiator i actually use) uses reservations when expanding vmdk files for size, thin provisioning, and snapshots. If I'm running big expands on 2 separate ESX nodes on the same iscsi or FC LUN, can you be sure that I won't have a collision without reservations? And I mean 100% absolute sure.
Quoted 7 lines
Second, in terms of TM emulation / passthrough support in the TCM v4 code, we follow what is implemented in
drivers/scsiML and LLDs, primarly to properly for Linux SCSI Initiators. I honestly don't have alot of interesting currently in implementing all of the ancient TM emulation that none of the mainline SCSI LLDs in Linux implement today, or plan to do the future. As for specific TM concerns, I am happy to address then on a case by case basis with the appropiate use case.
Vmware's ESX uses ABORT extensively with network errors and when the targets backend goes belly up, and just plain goes nuts if it doens't get the kind of response that completely makes sense to it, typically requiring the ESX node to be rebooted to get out of a confused state. Maybe this is a failure in ESX's initiator, but ESX is following the specs, or at least they claim to be.
And a question that never got answered in another thread: how many OSS and commercial products are using LIO? and how many of those are certified and supported by other commercial products? This is directly related to impartial testing as well as standards implementation, stability, and "plays nice with others". Or in other words, where I can look at an example implementation of LIO that's considered by someone who is not you to be production ready?
Quoted 15 lines · Chris Weiss
Quoted 6 lines · Nicholas A. Bellinger
Quoted 5 lines
as a user following the potential inclusion of a kernel-space target, iscsi specifically, I would be very interested in seeing what other pluses the other frameworks have over scst, because if this chart is accurate, all the other targets have quite a ways to go to catch up.
Quoted 6 lines
Actually sorry, anyone who has spent more than 30 minutes looking at the TCM v4 code that has already been posted to linux-scsi on monday knows this list just more handwaving. I suggest you start doing the same (actually discussion specific source file + line refrences) unless you actually want to trust this hopelessly out-of-date list on blind princaple.
Quoted 6 lines
I'm just an end user, I have no interesting in line-by-line comparisons or specs details. I do, however, have interest in bullet points, and the lack of bullet points, that throw up red flags to 3rd party support contracts. So, did the kernel team, or some other impartial source, compile a similar but not out of date list, and is it available for viewing?
Ok, but please understand that this is a technical list where we discuss specifics of kernel code, and produce and review patches to reach a desired goal.
One of the reason that TCM/LIO has been selected over SCST is because our patch workflow over the recent yeras demonstrates exactly this point the mainline folks who are following along. For the new persistent reservations and ALUA multipath features that have been developed in recent times, the code was added feature bit by feature bit in a reviewablle manner, with each patch series going to linux-scsi for comments. This allowed the code to benefit in a number of cases where different people (including Tomo-san, mnc, and others) noticed something strange in their own testing, and in each case we where able to get the item resolved. The scoreboard for these features is on the wiki here:
http://linux-iscsi.org/index.php/Persistent_Reservations
http://linux-iscsi.org/index.php/ALUA
In terms of some feature list about who supports what for the mainline discussion, this is quite pointless at this point. The best way for mainline people to know what your stack supports is to actually add support for said feature in reviewable and digestable pieces that are presented in much a manner that they are accessable to a larger audicence. Just developing code some-where else and posting the entire thing means the code in question does not benefit by public review during the development phase.
Quoted 9 lines
Quoted 8 lines
First, understand that Vlad has been asked to produce a problem use case for his CRH=1 (Compatibility Reservation Handling) concerns using the SPC-3 RESERVE/RELEASE methods with the TCM v4 code. He has been never been able to produce a use case, ever. Also, just for reference, does SCST's SPC-3 persisent reservation handling actually properly support CRH=1 emulation from spc4r17..? Last time I checked, it most certainly did *not*.
Quoted 7 lines
I haven't tested this specifically, and I'm not even sure that the use case is right, but ESX (the only initiator i actually use) uses reservations when expanding vmdk files for size, thin provisioning, and snapshots. If I'm running big expands on 2 separate ESX nodes on the same iscsi or FC LUN, can you be sure that I won't have a collision without reservations? And I mean 100% absolute sure.
We have some people happly using ESX with LUN_RESET emulation with CDB level emulation with TCM/FILEIO , TCM/IBLOCK and with TCM/pSCSI in passthrough mode.
Quoted 8 lines
Quoted 7 lines
Second, in terms of TM emulation / passthrough support in the TCM v4 code, we follow what is implemented in
drivers/scsiML and LLDs, primarly to properly for Linux SCSI Initiators. I honestly don't have alot of interesting currently in implementing all of the ancient TM emulation that none of the mainline SCSI LLDs in Linux implement today, or plan to do the future. As for specific TM concerns, I am happy to address then on a case by case basis with the appropiate use case.
Quoted 6 lines
Vmware's ESX uses ABORT extensively with network errors and when the targets backend goes belly up, and just plain goes nuts if it doens't get the kind of response that completely makes sense to it, typically requiring the ESX node to be rebooted to get out of a confused state. Maybe this is a failure in ESX's initiator, but ESX is following the specs, or at least they claim to be.
Yes, it needs to fall back to LUN_RESET, or reset the iSCSI I_T nexus. I am happy to add proper TASK_ABORT support into TCM v4 at some point down the line. However if you can demonstrate that this is a real issue to me for your particular setup/version of ESX with LUNs reporting SPC-2 or SPC-3 support, I would be happy to consider bumping this up the priority list.
Quoted 8 lines
And a question that never got answered in another thread: how many OSS and commercial products are using LIO? and how many of those are certified and supported by other commercial products? This is directly related to impartial testing as well as standards implementation, stability, and "plays nice with others". Or in other words, where I can look at an example implementation of LIO that's considered by someone who is not you to be production ready?
TCM/LIO v3 has been validated for the highest level of Vsphere 4 compliance (eg: above ESX which only uses SPC-2) with >= SPC-3 compatiable Persistent Reservations and ALUA multipath.
http://linux-iscsi.org/index.php/Ecosystem
The vendors on this list including, but not limited to QNAP, Data Robotics, and Netgear have all received independent SPC-3 SCSI LUN verification with VMware using TCM/LIO v3.x directly over the last 12 months. Which means that here in North America, you can walk into just about any major electronic store that carries these products, and purchase TCM/LIO v3.x in shrinkwrap form from multiple vendors on multiple hardware platforms. In some cases, these products using a custom set of userspace /sys/kernel/config/target/ scripts code (of varing degrees of quality) with a propritary web-interface (also of varing degrees of quality) on top.
Best,
--nab
Quoted 20 lines · Nicholas A. Bellinger, on 09/03/2010 12:12 AM wrote:
Quoted 11 lines
Quoted 5 lines
I updated the Linux SCSI targets comparison page
http://scst.sourceforge.net/comparison.html, which compares features of the existing Linux SCSI target subsystems. The comparison includes SCST, STGT, IET and LIO. I added IET there, because it is the most used Linux iSCSI target at the moment..
.Quoted 4 lines
If you see I'm wrong somewhere or forgot something, you are welcome to correct me and I will fix that.
Quoted 8 lines
Quoted 2 lines
Vlad
as a user following the potential inclusion of a kernel-space target, iscsi specifically, I would be very interested in seeing what other pluses the other frameworks have over scst, because if this chart is accurate, all the other targets have quite a ways to go to catch up.
Quoted 6 lines
Actually sorry, anyone who has spent more than 30 minutes looking at the TCM v4 code that has already been posted to linux-scsi on monday knows this list just more handwaving. I suggest you start doing the same (actually discussion specific source file + line refrences) unless you actually want to trust this hopelessly out-of-date list on blind princaple.
Well, I expected your reaction like that. You have nothing to say, so (as usually) prefer to call all "handwaving" and pretend it doesn't exist.
Quoted 8 lines
Quoted 4 lines
To me, lacking correct reserve and task management means "not ready for public consumption". Making any kernel change that does not have RESERVE/RELEASE and full TM command support is only going to make Linux look buggy and amateur-ish in the storage world.
First, understand that Vlad has been asked to produce a problem use case for his CRH=1 (Compatibility Reservation Handling) concerns using the SPC-3 RESERVE/RELEASE methods with the TCM v4 code.
For what is CRH here? It has no relation to the problem.
Quoted 2 lines
He has been never
been able to produce a use case, ever.
I did it at least twice and can do again if somebody ask. I'd suggest you to check this list archive. After that STGT and IET teams fixed that problem in their implementations.
Quoted 4 lines
Also, just for reference, does
SCST's SPC-3 persisent reservation handling actually properly support CRH=1 emulation from spc4r17..? Last time I checked, it most certainly did *not*.
It does. I'd suggest to check again, starting from scst_reserve_local().
Quoted 7 lines
Second, in terms of TM emulation / passthrough support in the TCM v4 code, we follow what is implemented in
drivers/scsiML and LLDs, primarly to properly for Linux SCSI Initiators. I honestly don't have alot of interesting currently in implementing all of the ancient TM emulation that none of the mainline SCSI LLDs in Linux implement today, or plan to do the future. As for specific TM concerns, I am happy to address then on a case by case basis with the appropiate use case.
Interesting, from which times ABORT TASK is ancient?
Vlad