libsas: support NCQ for SATA disks
Subsystems (2)
PATCH 1/1 libsas: support NCQ for SATA disks
This patch adds SATAII NCQ support to libsas. Both the use_ncq and the dma_xfer flags in ata_task must be set for NCQ to work correctly on the Adaptec SAS controller. The rest of the patch adds ATA_FLAG_NCQ to sata_port_info and sets up ap->scsi_host so that ata_setup_ncq doesn't crash. Please note that this patch is against the aic94xx-sas git tree, not scsi-misc. Thanks also to James Bottomley for providing an earlier version of this patch from which to work.
I've tested this patch on a x206m with a ST380819AS SATA2 disk plugged into the Adaptec SAS controller. The drive came up with a queue depth of 31, and I successfully ran an I/O flood test to coerce libata into sending multiple commands simultaneously. A kernel probe recorded the maximum tag number that had been seen before and after the flood test; before the test it was 2 and after it was 30, as I expected.
--
drivers/scsi/libsas/sas_scsi_host.c +9 -2
@@ -673,8 +673,14 @@ static unsigned int sas_ata_qc_issue(str task->ata_task.retry_count =1 ; task->task_state_flags =SAS_TASK_STATE_PENDING ; -if (qc->tf.protocol ==ATA_PROT_DMA ) +switch (qc->tf.protocol) { +case ATA_PROT_NCQ : + task->ata_task.use_ncq =1 ; +/* fall through */ +case ATA_PROT_DMA : task->ata_task.dma_xfer =1 ; +break ; + }if (sas_ha->lldd_max_execute_num <2 ) res = i->dft->lldd_execute_task (task,1 ,GFP_ATOMIC );
@@ -807,7 +813,7 @@ static struct ata_port_operations sas_sastatic struct ata_port_info sata_port_info = { .flags =ATA_FLAG_SATA |ATA_FLAG_NO_LEGACY |ATA_FLAG_SATA_RESET | -ATA_FLAG_MMIO |ATA_FLAG_PIO_DMA , +ATA_FLAG_MMIO |ATA_FLAG_PIO_DMA |ATA_FLAG_NCQ , .pio_mask =0x1f ,/* PIO0-4 */ .mwdma_mask =0x07 ,/* MWDMA0-2 */ .udma_mask =ATA_UDMA6 ,
@@ -875,6 +881,7 @@ int sas_target_alloc(struct scsi_target ap->private_data = found_dev; ap->cbl =ATA_CBL_SATA ; + ap->scsi_host = shost; found_dev->sata_dev.ap = ap; }
Quoted 6 lines · Darrick J. Wong
I've tested this patch on a x206m with a ST380819AS SATA2 disk plugged into the Adaptec SAS controller. The drive came up with a queue depth of 31, and I successfully ran an I/O flood test to coerce libata into sending multiple commands simultaneously. A kernel probe recorded the maximum tag number that had been seen before and after the flood test; before the test it was 2 and after it was 30, as I expected.
This doesn't seem to quite work for me on a SATA-1 disc:
sas: DOING DISCOVERY on port 1, pid:1897
sas: sas_ata_phy_reset: Found ATA device.
ata1.00: ATA-7, max UDMA/133, 781422768 sectors: LBA48 NCQ (depth 31/32) ata1.00: configured for UDMA/133
scsi 2:0:1:0: Direct-Access ATA ST3400832AS 3.03 PQ: 0
ANSI: 5
SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
sdc: unknown partition table
sd 2:0:1:0: Attached scsi disk sdc
sas: DONE DISCOVERY on port 1, pid:1897, result:0
sas: command 0xf785f3c0, task 0x00000000, timed out: EH_HANDLED
sas: command 0xf785f3c0, task 0x00000000, timed out: EH_HANDLED [...]
It looks like the first few commands get through (read capacity, ATA IDENTIFY etc) and it hangs up on the read for the partition table.
James
Quoted 21 lines · James Bottomley
This doesn't seem to quite work for me on a SATA-1 disc:
sas: DOING DISCOVERY on port 1, pid:1897
sas:sas_ata_phy_reset: Found ATA device.
ata1.00: ATA-7, max UDMA/133, 781422768 sectors: LBA48 NCQ (depth 31/32) ata1.00: configured for UDMA/133
scsi 2:0:1:0: Direct-Access ATA ST3400832AS 3.03 PQ: 0
ANSI: 5
SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
sdc: unknown partition table
sd 2:0:1:0: Attached scsi disk sdc
sas: DONE DISCOVERY on port 1, pid:1897, result:0
sas: command0xf785f3c0, task0x00000000, timed out:EH_HANDLED
sas: command0xf785f3c0, task0x00000000, timed out:EH_HANDLED[...]
Quoted 2 lines
It looks like the first few commands get through (read capacity, ATA IDENTIFY etc) and it hangs up on the read for the partition table.
Hm... if I put in some debug printks in the qc_issue code, I get the same symptoms. I've observed that once again we get hung up on ATA commands where the tag number > 0. I also noticed this pattern:
1. ATA command w/ tag 0 (command A) issued.
2. Command A goes out to sas-ata.
2. ATA command w/ tag 1 (command B) issued.
3. Command A completes
4. Command B goes out to sas-ata.
[...]
5. Command B times out.
Very odd that this all works if there are no printks. I don't see anything obvious that would suggest why this apparent race seems to happen--unless there's some conflict between issuing an ATA command while completing another one.
--D
Quoted 38 lines · --- "Darrick J. Wong
Quoted 21 lines · James Bottomley
This doesn't seem to quite work for me on a SATA-1 disc:
sas: DOING DISCOVERY on port 1, pid:1897
sas:sas_ata_phy_reset: Found ATA device.
ata1.00: ATA-7, max UDMA/133, 781422768 sectors: LBA48 NCQ (depth 31/32) ata1.00: configured for UDMA/133
scsi 2:0:1:0: Direct-Access ATA ST3400832AS 3.03 PQ: 0
ANSI: 5
SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
sdc: unknown partition table
sd 2:0:1:0: Attached scsi disk sdc
sas: DONE DISCOVERY on port 1, pid:1897, result:0
sas: command0xf785f3c0, task0x00000000, timed out:EH_HANDLED
sas: command0xf785f3c0, task0x00000000, timed out:EH_HANDLED[...]Quoted 2 lines
It looks like the first few commands get through (read capacity, ATA IDENTIFY etc) and it hangs up on the read for the partition table.
Hm... if I put in some debug printks in the
qc_issuecode, I get the same symptoms. I've observed that once again we get hung up on ATA commands where the tag number > 0. I also noticed this pattern:1. ATA command w/ tag 0 (command A) issued.
2. Command A goes out to sas-ata.
2. ATA command w/ tag 1 (command B) issued.
3. Command A completes
4. Command B goes out to sas-ata.
[...]
5. Command B times out.
Quoted 4 lines
Very odd that this all works if there are no printks. I don't see anything obvious that would suggest why this apparent race seems to happen--unless there's some conflict between issuing an ATA command while completing another one.
Keep debugging.
The GPL open sourced SCSI/ATA Translation Layer (SATL) I maintain, works perfectly for any SATA drive w/ NCQ (through the aic94xx SDS/interconnect), and I haven't heard any complaints from people using it.
Luben
Quoted 38 lines · Darrick J. Wong
Quoted 21 lines · James Bottomley
This doesn't seem to quite work for me on a SATA-1 disc:
sas: DOING DISCOVERY on port 1, pid:1897
sas:sas_ata_phy_reset: Found ATA device.
ata1.00: ATA-7, max UDMA/133, 781422768 sectors: LBA48 NCQ (depth 31/32) ata1.00: configured for UDMA/133
scsi 2:0:1:0: Direct-Access ATA ST3400832AS 3.03 PQ: 0
ANSI: 5
SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
SCSI device sdc: 781422768 512-byte hdwr sectors (400088 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
sdc: unknown partition table
sd 2:0:1:0: Attached scsi disk sdc
sas: DONE DISCOVERY on port 1, pid:1897, result:0
sas: command0xf785f3c0, task0x00000000, timed out:EH_HANDLED
sas: command0xf785f3c0, task0x00000000, timed out:EH_HANDLED[...]Quoted 2 lines
It looks like the first few commands get through (read capacity, ATA IDENTIFY etc) and it hangs up on the read for the partition table.
Hm... if I put in some debug printks in the
qc_issuecode, I get the same symptoms. I've observed that once again we get hung up on ATA commands where the tag number > 0. I also noticed this pattern:1. ATA command w/ tag 0 (command A) issued.
2. Command A goes out to sas-ata.
2. ATA command w/ tag 1 (command B) issued.
3. Command A completes
4. Command B goes out to sas-ata.
[...]
5. Command B times out.
Quoted 4 lines
Very odd that this all works if there are no printks. I don't see anything obvious that would suggest why this apparent race seems to happen--unless there's some conflict between issuing an ATA command while completing another one.
We have found in our system, using libata with Luben's driver, that the aic94xx sequencer expects the SATA commands to always have a tag of 0. The sequencer assigns its own tag before passing it to the drive. Interestingly, it seems to OR-in its generated tag value, so if a non-zero tag value is passed in, a real mess ensues.
Andy Warner can provide a lot more details on this.
Signature · 1 line
Mark Rustad, [email protected]
Quoted 18 lines · Mark Rustad
[...]
Quoted 12 lines
Hm... if I put in some debug printks in the
qc_issuecode, I get the same symptoms. I've observed that once again we get hung up on ATA commands where the tag number > 0. I also noticed this pattern:1. ATA command w/ tag 0 (command A) issued.
2. Command A goes out to sas-ata.
2. ATA command w/ tag 1 (command B) issued.
3. Command A completes
4. Command B goes out to sas-ata.
[...]
5. Command B times out.Quoted 4 lines
Very odd that this all works if there are no printks. I don't see anything obvious that would suggest why this apparent race seems to happen--unless there's some conflict between issuing an ATA command while completing another one.
Quoted 5 lines
We have found in our system, using libata with Luben's driver, that the aic94xx sequencer expects the SATA commands to always have a tag of 0. The sequencer assigns its own tag before passing it to the drive. Interestingly, it seems to OR-in its generated tag value, so if a non-zero tag value is passed in, a real mess ensues.
Mark speaks the truth. If you're going to use libata as a SATL, then you have to force the sector count to 0 before feeding the taskfile to the aic94xx code. There may be a role for a flag to libata that lets it totally skip the tag allocation, when used with smart adapters like the 9410. There's little point maintaining the bitfields if they are going to be ignored anyway.
You're also in for a bunch of fun servicing the SIGNAL_NCQ_ERROR escbs, suspending/clearing all outstanding I/O, issuing a read log page 10 command, mapping the tag back onto the original scb via the sister ddb, erroring that one, and retrying the rest. I'm working with mostly frozen code, so patches to the current de-Lubenised code are difficult (to put it mildly.) Sorry.
Signature · 1 line
Andy
Quoted 25 lines · --- Andy Warner
Quoted 18 lines · Mark Rustad
[...]
Quoted 12 lines
Hm... if I put in some debug printks in the
qc_issuecode, I get the same symptoms. I've observed that once again we get hung up on ATA commands where the tag number > 0. I also noticed this pattern:1. ATA command w/ tag 0 (command A) issued.
2. Command A goes out to sas-ata.
2. ATA command w/ tag 1 (command B) issued.
3. Command A completes
4. Command B goes out to sas-ata.
[...]
5. Command B times out.Quoted 4 lines
Very odd that this all works if there are no printks. I don't see anything obvious that would suggest why this apparent race seems to happen--unless there's some conflict between issuing an ATA command while completing another one.
Quoted 5 lines
We have found in our system, using libata with Luben's driver, that the aic94xx sequencer expects the SATA commands to always have a tag of 0. The sequencer assigns its own tag before passing it to the drive. Interestingly, it seems to OR-in its generated tag value, so if a non-zero tag value is passed in, a real mess ensues.
Quoted 6 lines
Mark speaks the truth. If you're going to use libata as a SATL, then you have to force the sector count to 0 before feeding the taskfile to the aic94xx code. There may be a role for a flag to libata that lets it totally skip the tag allocation, when used with smart adapters like the 9410. There's little point maintaining the bitfields if they are going to be ignored anyway.
Justin Gibbs explained many years ago the why and how of SCSI tags right here in linux-scsi in a thread. Their "domain", who is reponsible of allocating them and why, and why they are needed for (i.e. it is impossible to) managing SCSI Domains. Although not as detailed as I had mentioned above, his explanation is still true and holds today, as it did back then.
Quoted 6 lines
You're also in for a bunch of fun servicing the
SIGNAL_NCQ_ERRORescbs, suspending/clearing all outstanding I/O, issuing a read log page 10 command, mapping the tag back onto the original scb via the sister ddb, erroring that one, and retrying the rest. I'm working with mostly frozen code, so patches to the current de-Lubenised code are difficult (to put it mildly.) Sorry.
Andy, Mark, SAS/SATL is now at 1.2.7, both having been updated as SAS and SAT had been updated.
Luben
Quoted 32 lines · Andy Warner
Quoted 18 lines · Mark Rustad
[...]
Quoted 12 lines
Hm... if I put in some debug printks in the
qc_issuecode, I get the same symptoms. I've observed that once again we get hung up on ATA commands where the tag number > 0. I also noticed this pattern:1. ATA command w/ tag 0 (command A) issued.
2. Command A goes out to sas-ata.
2. ATA command w/ tag 1 (command B) issued.
3. Command A completes
4. Command B goes out to sas-ata.
[...]
5. Command B times out.Quoted 4 lines
Very odd that this all works if there are no printks. I don't see anything obvious that would suggest why this apparent race seems to happen--unless there's some conflict between issuing an ATA command while completing another one.
Quoted 5 lines
We have found in our system, using libata with Luben's driver, that the aic94xx sequencer expects the SATA commands to always have a tag of 0. The sequencer assigns its own tag before passing it to the drive. Interestingly, it seems to OR-in its generated tag value, so if a non-zero tag value is passed in, a real mess ensues.
Mark speaks the truth. If you're going to use libata as a SATL, then you have to force the sector count to 0 before feeding the taskfile to the aic94xx code. There may be a role for a flag to libata that lets it totally skip the tag allocation, when used with smart adapters like the 9410. There's little point maintaining the bitfields if they are going to be ignored anyway.
Quoted 6 lines
You're also in for a bunch of fun servicing the
SIGNAL_NCQ_ERRORescbs, suspending/clearing all outstanding I/O, issuing a read log page 10 command, mapping the tag back onto the original scb via the sister ddb, erroring that one, and retrying the rest. I'm working with mostly frozen code, so patches to the current de-Lubenised code are difficult (to put it mildly.) Sorry.
Thanks. We actually had two problems: the one with the tag type but the other with the fact that we sent our first NCQ command to the device before the sequencer had been informed of the NCQ tagging capabilities. I fixed the latter by moving the rphy_add() to the correct point in the code after the NCQ capabilities are set up.
This is my suggested fix (and now NCQ is working nicely with my seagate SATA disc). Someone else gets to fix the error handling, I think
James
drivers/scsi/aic94xx/aic94xx_task.c +0 -1
@@ -390,7 +390,6 @@ static int asd_build_ata_ascb(struct asd scb->ata_task.total_xfer_len =cpu_to_le32 (task->total_xfer_len); scb->ata_task.fis = task->ata_task.fis; - scb->ata_task.fis.fis_type =0x27 ;if (likely (!task->ata_task.device_control_reg_update)) scb->ata_task.fis.flags |=0x80 ;/* C=1: update ATA cmd reg */ scb->ata_task.fis.flags &=0xF0 ;/* PM_PORT field shall be 0 */
drivers/scsi/libsas/sas_scsi_host.c +6 -0
@@ -657,6 +657,12 @@ static unsigned int sas_ata_qc_issue(str task->task_proto =SAS_PROTOCOL_STP ; task->task_done = sas_ata_task_done; +if (qc->tf.command ==ATA_CMD_FPDMA_WRITE || + qc->tf.command ==ATA_CMD_FPDMA_READ ) { +/* Need to zero out the tag libata assigned us */ + qc->tf.nsect =0 ; + } +ata_tf_to_fis (&qc->tf, (u8 *)&task->ata_task.fis,0 ); task->uldd_task = qc;if (is_atapi_taskfile (&qc->tf)) {
drivers/scsi/libsas/sas_discover.c +3 -5
@@ -376,6 +376,7 @@ static int sas_issue_ata_cmd(struct doma task->dev = dev; + task->ata_task.fis.fis_type =0x27 ; task->ata_task.fis.command = command; task->ata_task.fis.features = features; task->ata_task.fis.device = d2h_fis->device;
@@ -488,11 +489,7 @@ cont1:sas_fill_in_rphy (dev, dev->rphy); - res =sas_rphy_add (dev->rphy); -if (res) - goto out_err; - -return res; +return 0 ; out_err: dev->sata_dev.identify_packet_device =NULL ; dev->sata_dev.identify_device =NULL ;
@@ -576,6 +573,7 @@ int sas_discover_sata(struct domain_devisas_notify_lldd_dev_gone (dev);if (!res) {sas_notify_lldd_dev_found (dev); + res =sas_rphy_add (dev->rphy); }return res; }
drivers/scsi/libsas/sas_discover.c static int sas_issue_ata_cmd(struct doma · line 382
Quoted 10 lines · James Bottomley
drivers/scsi/libsas/sas_discover.c +1
-0@@ -376,6 +376,7 @@ static int sas_issue_ata_cmd(struct doma task->dev = dev; + task->ata_task.fis.fis_type =0x27 ; task->ata_task.fis.command = command; task->ata_task.fis.features = features; task->ata_task.fis.device = d2h_fis->device;
As an aside, please clean up that magic number by assigning it a named constant (suggest SATA_FIS_H2D_REG) in include/linux/ata.h.
Jeff
Quoted 2 lines · Jeff Garzik
As an aside, please clean up that magic number by assigning it a named constant (suggest
SATA_FIS_H2D_REG) ininclude/linux/ata.h.
Fine by me ... I thought there would be one, but libata-core also uses the hex value. Why don't you put it in and I'll convert aic94xx over ... that way we don't get into any cross tree dependencies?
James
Quoted 10 lines · Darrick J. Wong
This patch adds SATAII NCQ support to libsas. Both the
use_ncqand thedma_xferflags inata_taskmust be set for NCQ to work correctly on the Adaptec SAS controller. The rest of the patch addsATA_FLAG_NCQtosata_port_infoand sets upap->scsi_hostso thatata_setup_ncqdoesn't crash. Please note that this patch is against the aic94xx-sas git tree, not scsi-misc. Thanks also to James Bottomley for providing an earlier version of this patch from which to work.
drivers/scsi/libsas/sas_scsi_host.c int sas_target_alloc(struct scsi_target · line 887
Quoted 7 lines
drivers/scsi/libsas/sas_scsi_host.c +1
-0@@ -875,6 +881,7 @@ int sas_target_alloc(struct scsi_target ap->private_data = found_dev; ap->cbl =ATA_CBL_SATA ; + ap->scsi_host = shost; found_dev->sata_dev.ap = ap; }
This doesn't look like the right fix for the oops you were seeing. The SAS usage of libata has ap->scsi_host as NULL, which indicates that libata does not own the associated scsi_host. I'm concerned you may have broken some other code path by making this change. I think the correct fix may require removing the dependence of ap->scsi_host from ata_dev_config_ncq.
Brian
Signature · 3 lines
Brian King eServer Storage I/O IBM Linux Technology Center
Quoted 6 lines · Brian King
This doesn't look like the right fix for the oops you were seeing. The SAS usage of libata has
ap->scsi_hostas NULL, which indicates that libata does not own the associatedscsi_host. I'm concerned you may have broken some other code path by making this change. I think the correct fix may require removing the dependence ofap->scsi_hostfromata_dev_config_ncq.
Yep. I had already mentioned this on IRC.
Jeff
Quoted 9 lines · Jeff Garzik
Quoted 6 lines · Brian King
This doesn't look like the right fix for the oops you were seeing. The SAS usage of libata has
ap->scsi_hostas NULL, which indicates that libata does not own the associatedscsi_host. I'm concerned you may have broken some other code path by making this change. I think the correct fix may require removing the dependence ofap->scsi_hostfromata_dev_config_ncq.Yep. I had already mentioned this on IRC.
I understand, but right at the moment, my priority is sorting out the aic94xx driver so that it works with SATA devices. It has become apparent that there's some need for a bit of code sorting out in libata to drive intelligent sas controllers, so we can take a look at bugs in ata_dev_config_ncq() when someone's time frees up to look into the libata issues.
James
Quoted 9 lines · --- James Bottomley
Quoted 8 lines · Jeff Garzik
Quoted 6 lines · Brian King
This doesn't look like the right fix for the oops you were seeing. The SAS usage of libata has
ap->scsi_hostas NULL, which indicates that libata does not own the associatedscsi_host. I'm concerned you may have broken some other code path by making this change. I think the correct fix may require removing the dependence ofap->scsi_hostfromata_dev_config_ncq.
Quoted 8 lines
Quoted 1 line
Yep. I had already mentioned this on IRC.
I understand, but right at the moment, my priority is sorting out the aic94xx driver so that it works with SATA devices. It has become apparent that there's some need for a bit of code sorting out in libata to drive intelligent sas controllers, so we can take a look at bugs in
ata_dev_config_ncq()when someone's time frees up to look into the libata issues.
Just to give people some freedom of choice:
I feel the need to mention that my code as I support it, the SAS Stack, does include fully T10-complient SATL, which does SAT on per device basis, leaving the transport be whatever it need be. I.e. no messing around with "ata ports" or "hosts" or what not.
A provider (SAS Stack, software ATA emulation, whatever) simply _registers_ with SATL, declaring in a structure of method stubs what it can do, and then SATL drives the device. This is similar to how my SAS Stack works with SAS LLDD (the bottomleyised version of my code uses a libsas "on the side" instead of being a fully fledged layer as intended and originally written).
The SATL also does SATA feature configuration as per the declared capabilities of the LLDD (aic94xx, or whatever) _and_ as per the device declared capabilities. I.e. if one or the other doesn't support it, then it is turned off, if both support it it is on.
The SATL also supports NCQ, S-ATAPI, full error recovery (NCQ, etc).
It also supports MODE SENSE, MODE SELECT, ATA Pass-through (all protocols), S-ATAPI, etc, etc, as per SAT.*
Bottomley has simply made a decision for you, what kind of code you should be using, what it supports and how it supports it.
Good luck everyone!
Luben
* For example, I exclusively cut CDs/DVDs using a S-ATAPI DVD-RW device attached to an expander using the SAS Stack and SATL.