qla2xxx: v3.4 target mode LLD changes + tcm_qla2xxx fabric module
Quoted 10 lines · On Sunday December 18, 2011 Nicholas Bellinger wrote:
So to get the ball rolling on remaining items, one question is still how to resolve mixed target/initiator mode operation on a HW per port context basis..?
This is currently done with a qla2xxx module parameter, but to do mixed mode properly we will need something smarter between scsi-core and target-core ports. Note we currently set
qlini_mode=QLA2XXX_INI_MODE_STR_DISABLED, so by default patch #1 will effectively disable initiator mode by skippingscsi_scan_host()from being called in to avoid scsi-core timeouts when performing immediate transition from initiator mode -> target mode via ISP reset.
Quoted 6 lines
What we would like to eventually do is run qla2xxx LLD to allow both initiator and target mode access based on the physical HW port. We tried some simple
qla_target.cchanges this make this work, but to really do it properly and address currentqlini_mode=QLA2XXX_INI_MODE_STR_DISABLEDusage it will need to involve scsi-core so that individual HW port can be configured and dynamically changed across different access modes.
(sorry for the late reply)
Dynamically switching between initiator mode, target mode or even using both modes simultaneously is already possible for iSCSI over Ethernet and for SRP over InfiniBand with the current SCSI core. Why would the SCSI core have to be modified in order to make the same possible for FCP ? Am I missing something ? If SCSI initiator timeouts during initiator-to-target mode transitions are an issue, why not to abort pending SCSI commands before performing the transition ?
Bart.
Quoted 18 lines · Bart Van Assche
Quoted 10 lines · On Sunday December 18, 2011 Nicholas Bellinger wrote:
So to get the ball rolling on remaining items, one question is still how to resolve mixed target/initiator mode operation on a HW per port context basis..?
This is currently done with a qla2xxx module parameter, but to do mixed mode properly we will need something smarter between scsi-core and target-core ports. Note we currently set
qlini_mode=QLA2XXX_INI_MODE_STR_DISABLED, so by default patch #1 will effectively disable initiator mode by skippingscsi_scan_host()from being called in to avoid scsi-core timeouts when performing immediate transition from initiator mode -> target mode via ISP reset.Quoted 6 lines
What we would like to eventually do is run qla2xxx LLD to allow both initiator and target mode access based on the physical HW port. We tried some simple
qla_target.cchanges this make this work, but to really do it properly and address currentqlini_mode=QLA2XXX_INI_MODE_STR_DISABLEDusage it will need to involve scsi-core so that individual HW port can be configured and dynamically changed across different access modes.
Quoted 5 lines
(sorry for the late reply)
Dynamically switching between initiator mode, target mode or even using both modes simultaneously is already possible for iSCSI over Ethernet and for SRP over InfiniBand with the current SCSI core.
Correct, because the implementation of these two examples is not dependent upon the registering of any LLD logic within scsi-core in order to function in target mode. We've always been able to avoid these types of conflicts between scsi-core and target-core subsystems up until now when initiator + target have been implemented as logically separate drivers.
Quoted 2 lines
Why would the SCSI core have to be modified in order to make the same possible for FCP ? Am I missing something ?
So it's nothing fabric specific to FCP of course, look at software + Intel 82599 HW offload for example in tcm_fc(FCoE) where initiator and target mode are logically separate drivers based upon common libfc code..
But the underlying issue with the current implementation of HW FC HBAs such as qla2xxx is that only a small subset of code is actually made common (at least in the fabric I/O codepath sense) amongst HW FC LLDs within libfc. It's my understanding that since this type of code has historically been very driver/vendor specific, trying to make this (more) common for individual HW FC HBAs hasn't been interesting enough for vendors to pursue benefit wise given the amount of re-factoring + regressions involved to existing initiator-mode operation.
That being the case, qla_target.c ends up depending upon alot of LLD specific code (some of which is not initiator specific or made common in libfc), so at least in the current implementation it really doesn't make sense to try to push this logic back out into tcm_qla2xxx code..
Quoted 3 lines
If SCSI initiator timeouts during initiator-to-target mode transitions are an issue, why not to abort pending SCSI commands before performing the transition ?
That would probably work, but I think it's still just a band-aid on the underlying issue of SCSI LLDs always kicking off initiator mode enable / disable operations from within PCI *_probe_one() / *_remove_code() code.
--nab
Quoted 3 lines · Nicholas A. Bellinger
That would probably work, but I think it's still just a band-aid on the underlying issue of SCSI LLDs always kicking off initiator mode enable / disable operations from within PCI
*_probe_one()/*_remove_code()code.
Maybe the following makes sense (I'm not familiar with the qla2xxx driver nor with FC): split the qla2xxx driver in three kernel modules - a kernel module with the code that is shared by initiator and target mode, a kernel module with the initiator functionality and a kernel module with the target functionality. This will allow users to choose which functionality to enable by loading the proper kernel module(s).
Note: in the kernel module with the shared code an interface will have to be added that allows the initiator and the target module to enumerate qla2xxx HBA ports. Maybe it's a good idea to add an interface similar to the add_one() / remove_one() callback functions present in the Linux InfiniBand stack.
Bart.
Quoted 7 lines · Bart Van Assche
Quoted 3 lines · Nicholas A. Bellinger
That would probably work, but I think it's still just a band-aid on the underlying issue of SCSI LLDs always kicking off initiator mode enable / disable operations from within PCI
*_probe_one()/*_remove_code()code.
Quoted 6 lines
Maybe the following makes sense (I'm not familiar with the qla2xxx driver nor with FC): split the qla2xxx driver in three kernel modules - a kernel module with the code that is shared by initiator and target mode, a kernel module with the initiator functionality and a kernel module with the target functionality. This will allow users to choose which functionality to enable by loading the proper kernel module(s).
Seriously, this involves high-level re-factoring of FC HBA drivers with 10x the moving parts and 100x the mission critical install base of your other HW example.
So yes, wanting to re-factor logic out into common code for initiator / target mode is good starting point, but without getting into the real-deal FC HW specifics of how this would work for running code along with tangible HBA vendor interest backing it, this type of statement quickly falls into the realm of wishful thinking.
Quoted 6 lines
Note: in the kernel module with the shared code an interface will have to be added that allows the initiator and the target module to enumerate qla2xxx HBA ports. Maybe it's a good idea to add an interface similar to the
add_one()/remove_one()callback functions present in the Linux InfiniBand stack.
Roland has had a few thoughts to this end about incremental improvements post 3.5 merge, but the target team has been primarily interested in fixing tcm_qla2xxx bugs and ensuring the qla2xxx LLD changes for target mode do not cause FC initiator mode regressions.
Considering the time scales involved with doing this type of testing (esp the latter, which can take months) plus given the amount of LLD code involved (40K LOC), a large re-org that effects existing FC initiator mode operation is something we'd like to avoid for now.
Thanks,
--nab
Quoted 11 lines · Nicholas A. Bellinger
Quoted 5 lines · Bart Van Assche
Note: in the kernel module with the shared code an interface will have to be added that allows the initiator and the target module to enumerate qla2xxx HBA ports. Maybe it's a good idea to add an interface similar to the
add_one()/remove_one()callback functions present in the Linux InfiniBand stack.Considering the time scales involved with doing this type of testing (esp the latter, which can take months) plus given the amount of LLD code involved (40K LOC), a large re-org that effects existing FC initiator mode operation is something we'd like to avoid for now.
Let me summarize what I've noticed after I had a (very) short look at the proposed qla2xxx driver changes:
- The proposed changes do not allow to enable initiator and target mode simultaneously on a single FC port.
- Most of the initiator-mode SCSI host sysfs attributes are relevant in target mode too (e.g. NV-RAM access) and hence should be moved from the SCSI host to somewhere else. A good example is e.g. the Linux IB stack - in the IB stack there is a clean separation between HCA-specific sysfs attributes and sysfs attributes associated with higher-level services.
- The posted patch does not allow to add more FC functionality in a clean way, e.g. IP over FC or FC-VI. (Note: IP over FC has been standardized through RFC 4338 - http://tools.ietf.org/html/rfc4338.)
So I'm not sure whether the proposed approach is acceptable for a mainline kernel driver. However, I'm not a SCSI maintainer and hence I do not have a decisive voice in this matter.
Bart.