IO transfer limits

DISCUSSION john clyne 7 participants 17 messages Jan 10, 2007 – Jan 17, 2007 1 message not in the archive SCSI

Share & tools

john clyne

author

2007-01-10 16:40 UTC #0 lore raw

Can anyone give me some guidance on where in the IO stack I might be running into a 512KB limit on IO transfer sizes to an external FC device? I've checked IO scheduler parameter (/sys/block/<dev>/queue/{max_sectors_kb,max_hw_sectors_kb}. Both are set to 32767. I'm using Qlogic HBAs (qla2312), but I don't see any relevent parameters. I'm running RHEL 4.0 with a 2.6.9-34 kernel. Any pointers would be greatly appreciated.

cheers - jc

Signature · 2 lines

View this message in context: http://www.nabble.com/IO-transfer-limits-tf2953389.html#a8260533
Sent from the linux-scsi mailing list archive at Nabble.com.

Mike Christie

2007-01-10 17:40 UTC #1 lore raw

Quoted 8 lines · john clyne

Can anyone give me some guidance on where in the IO stack I might be running into a 512KB limit on IO transfer sizes to an external FC device? I've checked IO scheduler parameter (/sys/block/<dev>/queue/{max_sectors_kb,max_hw_sectors_kb}. Both are set to 32767. I'm using Qlogic HBAs (qla2312), but I don't see any relevent parameters. I'm running RHEL 4.0 with a 2.6.9-34 kernel. Any pointers would be greatly appreciated.

There are also scatterlist limits.

/sys/class/scsi_host/hostX/sg_tablesize is a limit for the number of scatter list entries. For qla2xxx it is 255.

The scsi layer sets the queue's max_phys_segments to 128 by default. I thought there was ia scsi compile time option to increase this, but maybe you have to just modify the SCSI_MAX_PHYS_SEGMENTS define by hand.

So with the default value and with 4 K pages if you end up getting pages that cannot be clustered you will end up with 4K * 128.

Douglas Gilbert

2007-01-10 18:38 UTC #8 lore raw

In reply to john clyne

Quoted 7 lines · john clyne

Can anyone give me some guidance on where in the IO stack I might be running into a 512KB limit on IO transfer sizes to an external FC device? I've checked IO scheduler parameter (/sys/block/<dev>/queue/{max_sectors_kb,max_hw_sectors_kb}. Both are set to 32767. I'm using Qlogic HBAs (qla2312), but I don't see any relevent parameters. I'm running RHEL 4.0 with a 2.6.9-34 kernel. Any pointers would be greatly appreciated.

John,
I discuss the subject in this page:
http://www.torque.net/sg/sg_io.html
in the section titled:
Maximum transfer size per command

Mike C. has given you the answer for the block device interface (e.g. via /dev/sda); you should be able to do about 8 times better via the scsi generic interface (e.g. /dev/sg0).

Doug Gilbert

john clyne

author

2007-01-10 23:07 UTC #9 lore raw

So given that the HBA max_sectors_kb looks sufficiently large, my interpretation of your article is the next place I need to look is at ajusting SCSI_MAX_PHYS_SEGMENTS. Sounds like i can only increase this to
256. Does that than give me a max transfer size of 256 * page_size (4k) = 1MB? Is 1MB the limit or am I missing something?

thanks - jc

Quoted 33 lines · Douglas Gilbert

Quoted 10 lines · john clyne
Can anyone give me some guidance on where in the IO stack I might be running
into a 512KB limit on IO transfer sizes to an external FC device? I've checked IO scheduler parameter (/sys/block/<dev>/queue/{max_sectors_kb,max_hw_sectors_kb}. Both are set to
32767. I'm using Qlogic HBAs (qla2312), but I don't see any relevent parameters. I'm running RHEL 4.0 with a 2.6.9-34 kernel. Any pointers would
be greatly appreciated.
John,
I discuss the subject in this page:
   http://www.torque.net/sg/sg_io.html
in the section titled:
Maximum transfer size per command
Mike C. has given you the answer for the block device interface (e.g. via /dev/sda); you should be able to do about 8 times better via the scsi generic interface (e.g. /dev/sg0).
Doug Gilbert
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

View this message in context: http://www.nabble.com/IO-transfer-limits-tf2953389.html#a8268682
Sent from the linux-scsi mailing list archive at Nabble.com.

john clyne

author

2007-01-10 23:47 UTC #2 lore raw

In reply to Mike Christie

What do the different hostX in /sys/class/scsi_host corespond to? There are seven hostX directories, 5 with sg_tablesize set to 128 and two set to 255.

Is the implication that the hard limit is 255 * page_size, or is page_size simply the default?

Quoted 31 lines · Mike Christie

Quoted 11 lines · john clyne
Can anyone give me some guidance on where in the IO stack I might be running
into a 512KB limit on IO transfer sizes to an external FC device? I've checked IO scheduler parameter (/sys/block/<dev>/queue/{max_sectors_kb,max_hw_sectors_kb}. Both are set to
32767. I'm using Qlogic HBAs (qla2312), but I don't see any relevent parameters. I'm running RHEL 4.0 with a 2.6.9-34 kernel. Any pointers would
be greatly appreciated.
There are also scatterlist limits.
/sys/class/scsi_host/hostX/sg_tablesize is a limit for the number of scatter list entries. For qla2xxx it is 255.
The scsi layer sets the queue's max_phys_segments to 128 by default. I thought there was ia scsi compile time option to increase this, but maybe you have to just modify the SCSI_MAX_PHYS_SEGMENTS define by hand.
So with the default value and with 4 K pages if you end up getting pages that cannot be clustered you will end up with 4K * 128.
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

View this message in context: http://www.nabble.com/IO-transfer-limits-tf2953389.html#a8269227
Sent from the linux-scsi mailing list archive at Nabble.com.

Douglas Gilbert

2007-01-11 01:15 UTC #3 lore raw

Quoted 2 lines · john clyne

What do the different hostX in /sys/class/scsi_host corespond to? There are seven hostX directories, 5 with sg_tablesize set to 128 and two set to 255.

A Linux "host" is a SCSI initiator port (e.g. FC) or a SCSI initiator device (e.g. SAS). Another way of looking at a "host" is as a bridge between a computer bus (e.g. PCI) and a storage transport. There is usually one (low level) driver (LLD) controlling all hosts associated with a specific class of hardware.

If you fetch the lsscsi utility and load it then you can try 'lsscsi --hosts' to list the active hosts on a system (numbered on the left) and see the names of the various LLDs associated with them. Here is an example:

# lsscsi --hosts
[0]    sata_nv
[1]    sata_nv
[2]    sata_nv
[3]    sata_nv
[4]    mptsas
[5]    aic94xx
[6]    sbp2

The first four are SATA ports (connectors) on the motherboard, all controlled by the same driver. Then there is a LSI SAS HBA (whose driver is mptsas), an Adaptec SAS HBA (48300) and finally an Adaptec IEEE 1394 controller.

Quoted 2 lines

Is the implication that the hard limit is 255 * page_size, or is page_size simply the default?

There are "big" pages (around 1 MB in size) but I'm unaware that anything in the SCSI subsystem uses them. Otherwise the kernel page size is typically 4 KB. When the scsi generic driver builds its scatter gather lists, then it attempts to place 8 contiguous pages in each scatter gather element.

"Arm waving" was a term used when I tried to explain to several kernel people that there were users out there that needed larger IO transfer limits. So I suggest that you talk the the management and tell them why you need higher limits. Linux is retarded in this area.

Doug Gilbert

Quoted 31 lines

Quoted 29 lines · Mike Christie
Quoted 11 lines · john clyne
Can anyone give me some guidance on where in the IO stack I might be running
into a 512KB limit on IO transfer sizes to an external FC device? I've checked IO scheduler parameter (/sys/block/<dev>/queue/{max_sectors_kb,max_hw_sectors_kb}. Both are set to
32767. I'm using Qlogic HBAs (qla2312), but I don't see any relevent parameters. I'm running RHEL 4.0 with a 2.6.9-34 kernel. Any pointers would
be greatly appreciated.
There are also scatterlist limits.
/sys/class/scsi_host/hostX/sg_tablesize is a limit for the number of scatter list entries. For qla2xxx it is 255.
The scsi layer sets the queue's max_phys_segments to 128 by default. I thought there was ia scsi compile time option to increase this, but maybe you have to just modify the SCSI_MAX_PHYS_SEGMENTS define by hand.
So with the default value and with 4 K pages if you end up getting pages that cannot be clustered you will end up with 4K * 128.
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Stefan Richter

2007-01-11 16:13 UTC #4 lore raw

Quoted 25 lines · Douglas Gilbert

Quoted 2 lines · john clyne
What do the different hostX in /sys/class/scsi_host corespond to? There are seven hostX directories, 5 with sg_tablesize set to 128 and two set to 255.
A Linux "host" is a SCSI initiator port (e.g. FC) or a SCSI initiator device (e.g. SAS). Another way of looking at a "host" is as a bridge between a computer bus (e.g. PCI) and a storage transport. There is usually one (low level) driver (LLD) controlling all hosts associated with a specific class of hardware.
If you fetch the lsscsi utility and load it then you can try 'lsscsi --hosts' to list the active hosts on a system (numbered on the left) and see the names of the various LLDs associated with them. Here is an example:
# lsscsi --hosts
[0]    sata_nv
[1]    sata_nv
[2]    sata_nv
[3]    sata_nv
[4]    mptsas
[5]    aic94xx
[6]    sbp2

Quoted 4 lines

The first four are SATA ports (connectors) on the motherboard, all controlled by the same driver. Then there is a LSI SAS HBA (whose driver is mptsas), an Adaptec SAS HBA (48300) and finally an Adaptec IEEE 1394 controller.

A side note:
I don't think a Scsi_Host has a well-defined meaning beyond "the kernel-internal resource which LLDs use to connect to the Linux SCSI mid layer". It may have further meaning for many LLDs, but not for all. Specifically, the host6 in your example is in the current implementation
indeed nothing more than an internal resource. lsscsi is nevertheless
able to determine the actual initiator port by means of knowledge of the implementation.

Signature · 3 lines

Stefan Richter
-=====-=-=== ---= -=-==
http://arcgraph.de/sr/

James Bottomley

2007-01-11 18:23 UTC #10 lore raw

In reply to john clyne

Quoted 5 lines · john clyne

So given that the HBA max_sectors_kb looks sufficiently large, my interpretation of your article is the next place I need to look is at ajusting SCSI_MAX_PHYS_SEGMENTS. Sounds like i can only increase this to
256. Does that than give me a max transfer size of 256 * page_size (4k) = 1MB? Is 1MB the limit or am I missing something?

Well ... that's not the maximum possible ... just the maximum number of SG elements. If the buffer is completely fragmented in physical memory, then it will be 1MB ... however, linux does a reasonable job of getting physically contiguous pages into a large memory allocation, so you could get transfers that are larger than the 1MB limit by quite a way. And, of course, any system that has a larger page size or uses an iommu will go over this 1MB as well.

Jmaes

Thayne Harmon

2007-01-11 19:43 UTC #13 lore raw

no utility / method to show association between host bus adapter and non-sg BLOCK devices

In reply to Douglas Gilbert

Gentlemen,

hwinfo, lshal, sysfs do not show the relationship for non-sg BLOCK devices with there associated Host Bus Adapter.

Do you know of a utility or method that can show this?

Example is the HP/Compaq CCISS block driver.

The HBA and devices are listed, but no association is given or can be determine, only by the user knowing which is which.

The kernel certainly knows, surely the above apps could be made to determine this or some utility exits that will show this?

Thanks for any help,

Thayne Harmon

john clyne

author

2007-01-11 19:48 UTC #11 lore raw

In reply to James Bottomley

So what's the best course of action for increasing the 512kb IO transfer size limit, changing SCSI_MAX_PHYS_SEGMENTS (currently 128 and apparently can be set no greater than 256), or increasing the page size? Or is there another option.

thanks - jc

Quoted 25 lines · James Bottomley

Quoted 5 lines · john clyne
So given that the HBA max_sectors_kb looks sufficiently large, my interpretation of your article is the next place I need to look is at ajusting SCSI_MAX_PHYS_SEGMENTS. Sounds like i can only increase this to
256. Does that than give me a max transfer size of 256 * page_size (4k) = 1MB? Is 1MB the limit or am I missing something?
Well ... that's not the maximum possible ... just the maximum number of SG elements. If the buffer is completely fragmented in physical memory, then it will be 1MB ... however, linux does a reasonable job of getting physically contiguous pages into a large memory allocation, so you could get transfers that are larger than the 1MB limit by quite a way. And, of course, any system that has a larger page size or uses an iommu will go over this 1MB as well.
Jmaes
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

View this message in context: http://www.nabble.com/IO-transfer-limits-tf2953389.html#a8284792
Sent from the linux-scsi mailing list archive at Nabble.com.

Douglas Gilbert

2007-01-11 20:15 UTC #14 lore raw

Re: no utility / method to show association between host bus adapter and non-sg BLOCK devices

In reply to Thayne Harmon

Quoted 4 lines · Thayne Harmon

Gentlemen,
hwinfo, lshal, sysfs do not show the relationship for non-sg BLOCK devices with there associated Host Bus Adapter.

All devices (i.e. logical units) have a 4 element tuple associated with them and the first element is the host number. A HBA contains one or more hosts. Then you can datamine in /sys/class/scsi_host/host<n> for whatever information you want.

Quoted 1 line

Do you know of a utility or method that can show this?

May I suggest lsscsi. That won't help you in the lk 2.4 series and earlier though There are other methods by which the sg device corresponding to a "non-sg" block device (e.g. /dev/sdc) can be found.

Quoted 7 lines

Example is the HP/Compaq CCISS block driver.
The HBA and devices are listed, but no association is given or can be determine, only by the user knowing which is which.
The kernel certainly knows, surely the above apps could be made to determine this or some utility exits that will show this?

See http://www.torque.net/scsi/lsscsi.html

Doug Gilbert

Douglas Gilbert

2007-01-11 21:50 UTC #5 lore raw

In reply to Stefan Richter

Quoted 30 lines · Stefan Richter

Quoted 24 lines · Douglas Gilbert
Quoted 2 lines · john clyne
What do the different hostX in /sys/class/scsi_host corespond to? There are seven hostX directories, 5 with sg_tablesize set to 128 and two set to 255.
A Linux "host" is a SCSI initiator port (e.g. FC) or a SCSI initiator device (e.g. SAS). Another way of looking at a "host" is as a bridge between a computer bus (e.g. PCI) and a storage transport. There is usually one (low level) driver (LLD) controlling all hosts associated with a specific class of hardware.
If you fetch the lsscsi utility and load it then you can try 'lsscsi --hosts' to list the active hosts on a system (numbered on the left) and see the names of the various LLDs associated with them. Here is an example:
# lsscsi --hosts
[0]    sata_nv
[1]    sata_nv
[2]    sata_nv
[3]    sata_nv
[4]    mptsas
[5]    aic94xx
[6]    sbp2
Quoted 4 lines
The first four are SATA ports (connectors) on the motherboard, all controlled by the same driver. Then there is a LSI SAS HBA (whose driver is mptsas), an Adaptec SAS HBA (48300) and finally an Adaptec IEEE 1394 controller.

Quoted 8 lines

A side note:
I don't think a Scsi_Host has a well-defined meaning beyond "the kernel-internal resource which LLDs use to connect to the Linux SCSI mid layer". It may have further meaning for many LLDs, but not for all. Specifically, the host6 in your example is in the current implementation
indeed nothing more than an internal resource. lsscsi is nevertheless
able to determine the actual initiator port by means of knowledge of the implementation.

There are two sides to a "host": a kernel side (e.g. a PCI device or virtual) and a storage transport side. A host can be seen as a bridge between the two sides.

SCSI command sets need the concept of an initiator. For queuing, mode page policy and reservations (i.e. in multi-initiator environments) those initiators (actually initiator ports) need domain unique identifiers, preferably world wide unique. The identifier is an attribute of the external side (i.e. the storage transport side) of a linux "host".

So even if you consider the kernel side of a host is a kludge, there is still the storage transport side to consider.

In the case of sbp, the initiator (device and port) has a EUI-64 wwn. SBP, USB mass storage, and iSCSI all set up SCSI hosts in Linux on a session basis (just-in-time if you like). As long as the initiator port identifiers are stable (predictable from one session to the next) it seems to me little different to SAS, SPI and FC which maintain their hosts for as long as their HBAs are present.

There are cheap external boxes out there that have 1394, USB and (e)SATA interfaces. I wonder what would happen if one tried to use two interfaces connected to different machines at the same time :-)

Doug Gilbert

Stefan Richter

2007-01-12 00:54 UTC #6 lore raw

Quoted 4 lines · Douglas Gilbert

There are cheap external boxes out there that have 1394, USB and (e)SATA interfaces. I wonder what would happen if one tried to use two interfaces connected to different machines at the same time :-)

AFAIK these are all based on the OXUF924 series of bridge chips. Their FireWire side supports multiple initiators, at least if OxSemi's firmware is used, like all but the very earliest OxSemi SBP-2 target devices. However my _guess_ is the different interconnects---FireWire, USB, eSATA---cannot be used together at once. (John, sorry for straying off the subject.)

Signature · 3 lines

Stefan Richter
-=====-=-=== ---= -==--
http://arcgraph.de/sr/

James Bottomley

2007-01-12 01:17 UTC #12 lore raw

In reply to john clyne

Quoted 4 lines · john clyne

So what's the best course of action for increasing the 512kb IO transfer size limit, changing SCSI_MAX_PHYS_SEGMENTS (currently 128 and apparently can be set no greater than 256), or increasing the page size? Or is there another option.

Well, without knowing the details of what you want to do it's hard to say. Doug offered you a method via SG_IO (albeit involving buffer copying); you can just increase the SCSI_MAX_PHYS_SEGMENTS, or you could increase the PAGE_SIZE or use an IOMMU. For a given workload, you'll just have to see which works out best in practice.

James

john clyne

author

2007-01-12 22:27 UTC #7 lore raw

In reply to Mike Christie

Increasing SCSI_MAX_PHYS_SEGMENTS to 256 has doubled the IO size to 1MB. Thanks to all!

cheers - jc

Quoted 31 lines · Mike Christie

Quoted 11 lines · john clyne
Can anyone give me some guidance on where in the IO stack I might be running
into a 512KB limit on IO transfer sizes to an external FC device? I've checked IO scheduler parameter (/sys/block/<dev>/queue/{max_sectors_kb,max_hw_sectors_kb}. Both are set to
32767. I'm using Qlogic HBAs (qla2312), but I don't see any relevent parameters. I'm running RHEL 4.0 with a 2.6.9-34 kernel. Any pointers would
be greatly appreciated.
There are also scatterlist limits.
/sys/class/scsi_host/hostX/sg_tablesize is a limit for the number of scatter list entries. For qla2xxx it is 255.
The scsi layer sets the queue's max_phys_segments to 128 by default. I thought there was ia scsi compile time option to increase this, but maybe you have to just modify the SCSI_MAX_PHYS_SEGMENTS define by hand.
So with the default value and with 4 K pages if you end up getting pages that cannot be clustered you will end up with 4K * 128.
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

View this message in context: http://www.nabble.com/IO-transfer-limits-tf2953389.html#a8307532
Sent from the linux-scsi mailing list archive at Nabble.com.

Douglas Gilbert

2007-01-17 21:04 UTC #16 lore raw

Re: no utility / method to show association between HBA & non-sg BLOCK (scsi) devices - register_blkdev()

In reply to a message not in the archive

Quoted 20 lines · Thayne Harmon

Quoted 1 line
Quoted 1 line
Quoted 1 line
On Thu, Jan 11, 2007 at 1:15 PM, in message <[email protected]>,
Quoted 12 lines · Douglas Gilbert
Quoted 3 lines · Thayne Harmon
Gentlemen,
hwinfo, lshal, sysfs do not show the relationship for non- sg BLOCK devices
with there
Quoted 1 line
associated Host Bus Adapter.
All devices (i.e. logical units) have a 4 element tuple associated with them and the first element is the host number. A HBA contains one or more hosts. Then you can datamine in /sys/class/scsi_host/host<n> for whatever information you want.
Quoted 5 lines
Quoted 1 line
Do you know of a utility or method that can show this?
May I suggest lsscsi. That won't help you in the lk 2.4 series and earlier though There are other methods by which the sg device corresponding to a "non- sg" block device (e.g. /dev/sdc) can be found.

Quoted 4 lines

[context - Linux testserver 2.6.16.21-0.8-smp i586]
There is no corresponding sg device. The device file is /dev/cciss/c0d1.

Ok, I'm not familiar with the cciss driver. It looks like it lives outside the linux scsi subsystem but according to Documentation/cciss.txt it can subsequently "engage" the scsi subsystem??

If it is outside the scsi subsystem then it doesn't get corresponding sg devices. However as part of the block subsystem it might accept the SG_IO ioctl (if it accepts SCSI commands and it is implemented).

Quoted 5 lines

I tried lsscsi, however it would not print out the non-sg block devices.
I have attached the output of tree /sys and the output of lsscsi and uname. One can search for cciss to find the devices and the driver.
I still cannot see a relationship.

Quoted 4 lines

[0:0:0:0]    storage COMPAQ   MSA1000          4.32  -       
[0:0:0:3]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sda
[0:0:0:4]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdb
[0:0:0:5]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdc

Quoted 8 lines

[0:0:0:6]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdd
[0:0:0:7]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sde
[1:0:0:0]    storage COMPAQ   MSA1000          4.32  -       
[1:0:0:3]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdf
[1:0:0:4]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdg
[1:0:0:5]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdh
[1:0:0:6]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdi
[1:0:0:7]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdj

Well this looks like output from lsscsi. And those devices look like they could be associated with cciss, especially the compaq "storage" devices. These devices should have corresponding sg device nodes. Try "lsscsi -g".

Still a bit unclear as hosts 0 and 1 are Fibre Channel judging from the sysfs output for them.

Doug Gilbert

Andrew Patterson

2007-01-17 22:18 UTC #17 lore raw

Re: no utility / method to show association between HBA & non-sg BLOCK (scsi) devices - register_blkdev()

Quoted 31 lines · Douglas Gilbert

Quoted 20 lines · Thayne Harmon
Quoted 1 line
Quoted 1 line
Quoted 1 line
On Thu, Jan 11, 2007 at 1:15 PM, in message <[email protected]>,
Quoted 12 lines · Douglas Gilbert
Quoted 3 lines · Thayne Harmon
Gentlemen,
hwinfo, lshal, sysfs do not show the relationship for non- sg BLOCK devices
with there
Quoted 1 line
associated Host Bus Adapter.
All devices (i.e. logical units) have a 4 element tuple associated with them and the first element is the host number. A HBA contains one or more hosts. Then you can datamine in /sys/class/scsi_host/host<n> for whatever information you want.
Quoted 5 lines
Quoted 1 line
Do you know of a utility or method that can show this?
May I suggest lsscsi. That won't help you in the lk 2.4 series and earlier though There are other methods by which the sg device corresponding to a "non- sg" block device (e.g. /dev/sdc) can be found.
Quoted 4 lines
[context - Linux testserver 2.6.16.21-0.8-smp i586]
There is no corresponding sg device. The device file is /dev/cciss/c0d1.
Ok, I'm not familiar with the cciss driver. It looks like it lives outside the linux scsi subsystem but according to Documentation/cciss.txt it can subsequently "engage" the scsi subsystem??

Quoted 5 lines

If it is outside the scsi subsystem then it doesn't get corresponding sg devices. However as part of the block subsystem it might accept the SG_IO ioctl (if it accepts SCSI commands and it is implemented).

Cciss is a block driver mostly outside of SCSI. It does interface into the SCSI system for tape drives.

Quoted 21 lines

Quoted 5 lines

I tried lsscsi, however it would not print out the non-sg block devices.
I have attached the output of tree /sys and the output of lsscsi and uname. One can search for cciss to find the devices and the driver.
I still cannot see a relationship.

Quoted 4 lines

[0:0:0:0]    storage COMPAQ   MSA1000          4.32  -       
[0:0:0:3]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sda
[0:0:0:4]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdb
[0:0:0:5]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdc

Quoted 8 lines

[0:0:0:6]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdd
[0:0:0:7]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sde
[1:0:0:0]    storage COMPAQ   MSA1000          4.32  -       
[1:0:0:3]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdf
[1:0:0:4]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdg
[1:0:0:5]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdh
[1:0:0:6]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdi
[1:0:0:7]    disk    COMPAQ   MSA1000 VOLUME   4.32  /dev/sdj

Quoted 4 lines

Well this looks like output from lsscsi. And those devices look like they could be associated with cciss, especially the compaq "storage" devices. These devices should have corresponding sg device nodes. Try "lsscsi -g".

These are Fibre-Channel Storage arrays, probably going through a QLogic HBA.

Quoted 11 lines

Still a bit unclear as hosts 0 and 1 are Fibre Channel judging from the sysfs output for them.
Doug Gilbert
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

Andrew Patterson
Hewlett-Packard Company