New: iSCSI does not implement ordering guarantees required by e.g. journaling filesystems
A message that is not in the archive (sent off-list or not yet received): [email protected]/
Quoted 31 lines · On Mon, 19 Nov 2007 05:44:01 -0800 (PST) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=9405
Summary: iSCSI does not implement ordering guarantees required by
e.g. journaling filesystems
Product: IO/Storage
Version: 2.5
KernelVersion: 2.6.23.1
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: SCSI
AssignedTo: [email protected]
ReportedBy: [email protected]Most recent kernel where this bug did not occur: (new issue)
Distribution: any
Hardware Environment: (does not apply)
Software Environment: (does not apply)
Problem Description: The sd (SCSI disk) driver ignores block device barriers (REQ_HARDBARRIER). The iSCSI code in the kernel sends all iSCSI commands with flagISCSI_ATTR_SIMPLEto the iSCSI target. This means that the target may reorder these commands. Since a.o. correct operation of journaling filesystems depends on being able to enforce the order of certain block write operations, not enforcing write ordering is a bug. This can be solved by either adding support forREQ_HARDBARRIERin the sd device or by replacingISCSI_ATTR_SIMPLEbyISCSI_ATTR_ORDERED.
Quoted 6 lines
Steps to reproduce: Source reading of
drivers/scsi/sd.canddrivers/scsi/libiscsi.c.References: SCSI Architecture Model - 3, paragraph 8.6
(http://www.t10.org/ftp/t10/drafts/sam3/sam3r14.pdf).
(does iscsi have a maintainer?)
James Bottomley
maintainerQuoted 34 lines · Andrew Morton
Quoted 31 lines · On Mon, 19 Nov 2007 05:44:01 -0800 (PST) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=9405
Summary: iSCSI does not implement ordering guarantees required by
e.g. journaling filesystems
Product: IO/Storage
Version: 2.5
KernelVersion: 2.6.23.1
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: SCSI
AssignedTo: [email protected]
ReportedBy: [email protected]Most recent kernel where this bug did not occur: (new issue)
Distribution: any
Hardware Environment: (does not apply)
Software Environment: (does not apply)
Problem Description: The sd (SCSI disk) driver ignores block device barriers (REQ_HARDBARRIER). The iSCSI code in the kernel sends all iSCSI commands with flagISCSI_ATTR_SIMPLEto the iSCSI target. This means that the target may reorder these commands. Since a.o. correct operation of journaling filesystems depends on being able to enforce the order of certain block write operations, not enforcing write ordering is a bug. This can be solved by either adding support forREQ_HARDBARRIERin the sd device or by replacingISCSI_ATTR_SIMPLEbyISCSI_ATTR_ORDERED.
Quoted 8 lines
Quoted 6 lines
Steps to reproduce: Source reading of
drivers/scsi/sd.canddrivers/scsi/libiscsi.c.References: SCSI Architecture Model - 3, paragraph 8.6
(http://www.t10.org/ftp/t10/drafts/sam3/sam3r14.pdf).(does iscsi have a maintainer?)
Yes, mike christie
And please close this as invalid. FS ordering guarantees in linux aren't done via ordered tags.
James
Quoted 40 lines · James Bottomley
Quoted 39 lines · Andrew Morton
Quoted 36 lines · On Mon, 19 Nov 2007 05:44:01 -0800 (PST) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=9405
Summary: iSCSI does not implement ordering guarantees required by
e.g. journaling filesystems
Product: IO/Storage
Version: 2.5
KernelVersion: 2.6.23.1
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: SCSI
AssignedTo: [email protected]
ReportedBy: [email protected]Most recent kernel where this bug did not occur: (new issue)
Distribution: any
Hardware Environment: (does not apply)
Software Environment: (does not apply) Problem Description: The sd (SCSI disk) driver ignores block device barriers (REQ_HARDBARRIER). The iSCSI code in the kernel sends all iSCSI commands with
flagISCSI_ATTR_SIMPLEto the iSCSI target. This means that the target may
reorder these commands. Since a.o. correct operation of journaling filesystems
depends on being able to enforce the order of certain block write operations,
not enforcing write ordering is a bug. This can be solved by either adding
support forREQ_HARDBARRIERin the sd device or by replacingISCSI_ATTR_SIMPLE
byISCSI_ATTR_ORDERED.Steps to reproduce: Source reading of
drivers/scsi/sd.canddrivers/scsi/libiscsi.c.References: SCSI Architecture Model - 3, paragraph 8.6
(http://www.t10.org/ftp/t10/drafts/sam3/sam3r14.pdf).
Quoted 7 lines
Quoted 1 line
(does iscsi have a maintainer?)
Yes, mike christie
And please close this as invalid. FS ordering guarantees in linux aren't done via ordered tags.
I had a related question. I was working on the attached patch for soe other testing (patch made against scsi-rc-fixes, but is not stable so do not apply), which does the scsi_populate_tag_msg conversion from MSG_* to ISCSI_ATTR and sets the proper iscsi bits.
If I do this patch where I call scsi_activate_tcq on a device and that concertsion, does this require that my driver not reorder commands? I was just a little worried on some of the error handling paths where we requeue commands to the mid layer.
drivers/scsi/iscsi_tcp.c +10 -0
@@ -37,6 +37,7 @@#include <linux/scatterlist.h> #include <net/tcp.h> #include <scsi/scsi_cmnd.h> +#include <scsi/scsi_tcq.h> #include <scsi/scsi_device.h> #include <scsi/scsi_host.h> #include <scsi/scsi.h>
@@ -2211,8 +2212,17 @@ static void iscsi_tcp_session_destroy(struct iscsi_cls_session *cls_session)static int iscsi_tcp_slave_configure (struct scsi_device *sdev) { +int depth =1 , tag =0 ; +blk_queue_bounce_limit (sdev->request_queue,BLK_BOUNCE_ANY );blk_queue_dma_alignment (sdev->request_queue,0 ); + +if (sdev->tagged_supported) { +scsi_activate_tcq (sdev,ISCSI_DEF_CMD_PER_LUN ); + depth =ISCSI_DEF_CMD_PER_LUN ; + tag =MSG_ORDERED_TAG ; + } +scsi_adjust_queue_depth (sdev, tag, depth);return 0 ; }
drivers/scsi/libiscsi.c +23 -1
@@ -122,6 +122,27 @@ void iscsi_prep_unsolicit_data_pdu(struct iscsi_cmd_task *ctask, }EXPORT_SYMBOL_GPL (iscsi_prep_unsolicit_data_pdu); +static uint32_t iscsi_command_attr (struct scsi_cmnd *cmd) +{ +unsigned int attr =ISCSI_ATTR_UNTAGGED ; +char msg[2 ]; + +if (scsi_populate_tag_msg (cmd, msg) ==2 ) { +switch (msg[0 ]) { +case MSG_SIMPLE_TAG : + attr =ISCSI_ATTR_SIMPLE ; +break ; +case MSG_HEAD_TAG : + attr =ISCSI_ATTR_HEAD_OF_QUEUE ; +break ; +case MSG_ORDERED_TAG : + attr =ISCSI_ATTR_ORDERED ; +break ; + } + } +return attr; +} +/** * iscsi_prep_scsi_cmd_pdu - prep iscsi scsi cmd pdu * @ctask: iscsi cmd task
@@ -137,7 +158,8 @@ static void iscsi_prep_scsi_cmd_pdu(struct iscsi_cmd_task *ctask)struct scsi_cmnd *sc = ctask->sc; hdr->opcode =ISCSI_OP_SCSI_CMD ; - hdr->flags =ISCSI_ATTR_SIMPLE ; + hdr->flags = hdr->flags = (iscsi_command_attr (sc) & +ISCSI_FLAG_CMD_ATTR_MASK );int_to_scsilun (sc->device->lun, (struct scsi_lun *)hdr->lun); hdr->itt =build_itt (ctask->itt, conn->id, session->age); hdr->data_length =cpu_to_be32 (scsi_bufflen (sc));
James Bottomley
maintainerQuoted 55 lines · Mike Christie
Quoted 41 lines · James Bottomley
Quoted 40 lines · Andrew Morton
Quoted 37 lines · On Mon, 19 Nov 2007 05:44:01 -0800 (PST) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=9405
Summary: iSCSI does not implement ordering guarantees required by
e.g. journaling filesystems
Product: IO/Storage
Version: 2.5
KernelVersion: 2.6.23.1
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: SCSI
AssignedTo: [email protected]
ReportedBy: [email protected]Most recent kernel where this bug did not occur: (new issue)
Distribution: any
Hardware Environment: (does not apply)
Software Environment: (does not apply)
Problem Description: The sd (SCSI disk) driver ignores block device barriers (REQ_HARDBARRIER). The iSCSI code in the kernel sends all iSCSI commands with flagISCSI_ATTR_SIMPLEto the iSCSI target. This means that the target may reorder these commands. Since a.o. correct operation of journaling filesystems depends on being able to enforce the order of certain block write operations, not enforcing write ordering is a bug. This can be solved by either adding support forREQ_HARDBARRIERin the sd device or by replacingISCSI_ATTR_SIMPLEbyISCSI_ATTR_ORDERED.Steps to reproduce: Source reading of
drivers/scsi/sd.canddrivers/scsi/libiscsi.c.References: SCSI Architecture Model - 3, paragraph 8.6
(http://www.t10.org/ftp/t10/drafts/sam3/sam3r14.pdf).Quoted 7 lines
Quoted 1 line
(does iscsi have a maintainer?)
Yes, mike christie
And please close this as invalid. FS ordering guarantees in linux aren't done via ordered tags.
I had a related question. I was working on the attached patch for soe other testing (patch made against scsi-rc-fixes, but is not stable so do not apply), which does the
scsi_populate_tag_msgconversion from MSG_* toISCSI_ATTRand sets the proper iscsi bits.
Quoted 4 lines
If I do this patch where I call
scsi_activate_tcqon a device and that concertsion, does this require that my driver not reorder commands? I was just a little worried on some of the error handling paths where we requeue commands to the mid layer.
Right, there's no way of guaranteeing that commands aren't reordered in the error path (or even the queue full submission path) which is why we don't use ordered tags to enforce barriers.
James
Quoted 10 lines · James Bottomley
Quoted 6 lines
Quoted 2 lines
And please close this as invalid. FS ordering guarantees in linux aren't done via ordered tags.
I had a related question. I was working on the attached patch for soe other testing (patch made against scsi-rc-fixes, but is not stable so do not apply), which does the
scsi_populate_tag_msgconversion from MSG_* toISCSI_ATTRand sets the proper iscsi bits.If I do this patch where I call
scsi_activate_tcqon a device and that concertsion, does this require that my driver not reorder commands? I was just a little worried on some of the error handling paths where we requeue commands to the mid layer.Right, there's no way of guaranteeing that commands aren't reordered in the error path (or even the queue full submission path) which is why we don't use ordered tags to enforce barriers.
May I make your answer more precise? SCSI for non-caching and write-through caching devices provides a way to guarantee order of commands on the error path via ACA and UA_INTLCK facilities, if they are supported by device. For write-back caching devices it's different, because cache may reorder commands after they are reported as completed to the initiator as well as there is a possibility for deferred errors.
So, there is no way to guarantee commands order in case of errors, because Linux doesn't implement that.
BTW, there is still something wrong in the SCSI/block/FS layers error processing. Playing with my SCSI target I've noticed that if it returns pretty valid TASK ABORTED status for some SCSI command, FS on initiator (ext3) immediately gets corrupted and journal replay on remount doesn't repair it, only manual e2fsck helps. So, apparently:
1. SCSI ML handles well not all status codes, which it should.
2. Block/FS levels (sometimes) don't handle I/O errors well enough without corrupting file systems.
I don't have time for further investigations, but, if somebody prepare a patch to fix that, I'm willing to assist in testing.
Vlad
James Bottomley
maintainerQuoted 18 lines · Vladislav Bolkhovitin
Quoted 8 lines · James Bottomley
Quoted 8 lines
Quoted 2 lines
And please close this as invalid. FS ordering guarantees in linux aren't done via ordered tags.
I had a related question. I was working on the attached patch for soe other testing (patch made against scsi-rc-fixes, but is not stable so do not apply), which does the
scsi_populate_tag_msgconversion from MSG_* toISCSI_ATTRand sets the proper iscsi bits.Quoted 8 lines
Quoted 4 lines
If I do this patch where I call
scsi_activate_tcqon a device and that concertsion, does this require that my driver not reorder commands? I was just a little worried on some of the error handling paths where we requeue commands to the mid layer.Right, there's no way of guaranteeing that commands aren't reordered in the error path (or even the queue full submission path) which is why we don't use ordered tags to enforce barriers.
Quoted 6 lines
May I make your answer more precise? SCSI for non-caching and write-through caching devices provides a way to guarantee order of commands on the error path via ACA and
UA_INTLCKfacilities, if they are supported by device. For write-back caching devices it's different, because cache may reorder commands after they are reported as completed to the initiator as well as there is a possibility for deferred errors.
Yes, I know this. The problem is that because we can't rely on the ordering guarantees in *every* situation, it's unsafe to rely on them for barrier support (the case you most need them is the one where the guarantees have likely failed). Thus, linux fs on SCSI implement barriers by waiting for completions. The only case we could implement flush barriers in SCSI, as they do in IDE is in the single outstanding command case where we don't have any reordering to worry about (i.e. queue depth of one).
Quoted 10 lines
So, there is no way to guarantee commands order in case of errors, because Linux doesn't implement that.
BTW, there is still something wrong in the SCSI/block/FS layers error processing. Playing with my SCSI target I've noticed that if it returns pretty valid TASK ABORTED status for some SCSI command, FS on initiator (ext3) immediately gets corrupted and journal replay on remount doesn't repair it, only manual e2fsck helps. So, apparently:
1. SCSI ML handles well not all status codes, which it should.
It certainly handles TASK ABORTED.
Quoted 2 lines
2. Block/FS levels (sometimes) don't handle I/O errors well enough without corrupting file systems.
I'm not sure your conclusions necessarily follow your data. What was the reason for the TASK ABORTED (I'd guess QErr settings, right)?
Journals can fail to recover in cases where the underlying medium is corrupted. If TASK ABORTED was because of QErr, what was the original failure?
Also, what was going on in the system (and what device was this ... iSCSI I guess) ... I assume nothing powered down, so it's not a caching problem (and that, since you seem to be using TCQ you do have your caches set to write through).
Quoted 2 lines
I don't have time for further investigations, but, if somebody prepare a patch to fix that, I'm willing to assist in testing.
We'll need a bit more data to identify an actual root cause for this problem before anyone can prepare a patch to fix it.
James
Quoted 17 lines · James Bottomley
Quoted 7 lines · Vladislav Bolkhovitin
Quoted 5 lines · James Bottomley
Quoted 5 lines
Quoted 2 lines
And please close this as invalid. FS ordering guarantees in linux aren't done via ordered tags.
I had a related question. I was working on the attached patch for soe other testing (patch made against scsi-rc-fixes, but is not stable so do not apply), which does the
scsi_populate_tag_msgconversion from MSG_* toISCSI_ATTRand sets the proper iscsi bits.Quoted 7 lines
Quoted 5 lines
Quoted 1 line
If I do this patch where I call
scsi_activate_tcqon a device and that concertsion, does this require that my driver not reorder commands? I was just a little worried on some of the error handling paths where we requeue commands to the mid layer.Right, there's no way of guaranteeing that commands aren't reordered in the error path (or even the queue full submission path) which is why we don't use ordered tags to enforce barriers.
May I make your answer more precise? SCSI for non-caching and write-through caching devices provides a way to guarantee order of commands on the error path via ACA and
UA_INTLCKfacilities, if they are supported by device. For write-back caching devices it's different, because cache may reorder commands after they are reported as completed to the initiator as well as there is a possibility for deferred errors.
Quoted 8 lines
Yes, I know this. The problem is that because we can't rely on the ordering guarantees in *every* situation, it's unsafe to rely on them for barrier support (the case you most need them is the one where the guarantees have likely failed). Thus, linux fs on SCSI implement barriers by waiting for completions. The only case we could implement flush barriers in SCSI, as they do in IDE is in the single outstanding command case where we don't have any reordering to worry about (i.e. queue depth of one).
...if we are going to work only with devices with write-back cache only or not supporting ACA/UA_INTLCK facilities. It might be well possible that some hypothetic SCSI device with write-through cache (WCE bit is 0 or set to 0), ACA/UA_INTLCK and ORDERED commands support would perform considerebly better with barriers by ORDERED tags, than with barriers by waiting for completions and write-back cache, especially for file systems like XFS, because with barriers by ORDERED tags it is possible to keep SCSI tarnsport wire pipe full, where it has to be drained with barriers by waiting for completions. But, since AFAIK the majority of SCSI disks don't support ACA/UA_INTLCK, I have to agree with you, there is not much point currently to implement barriers by ORDERED tags in the SCSI ML.
Quoted 7 lines
Quoted 5 lines
So, there is no way to guarantee commands order in case of errors, because Linux doesn't implement that.
BTW, there is still something wrong in the SCSI/block/FS layers error processing. Playing with my SCSI target I've noticed that if it returns pretty valid TASK ABORTED status for some SCSI command, FS on initiator (ext3) immediately gets corrupted and journal replay on remount doesn't repair it, only manual e2fsck helps. So, apparently:
1. SCSI ML handles well not all status codes, which it should.
Quoted 8 lines
It certainly handles TASK ABORTED.
Quoted 1 line
2. Block/FS levels (sometimes) don't handle I/O errors well enough without corrupting file systems.
I'm not sure your conclusions necessarily follow your data. What was the reason for the TASK ABORTED (I'd guess QErr settings, right)?
It was my desire/curiosity during tests of SCST (http://scst.sf.net), when it working with several initiators with different transports over the same set of devices, each of them having with TAS bit in the control mode page set. According to SAM, in this case TASK ABORTED status can be returned at any time, similarly to QUEUE FULL, i.e. IMHO such command just should be retried. But QUEUE FULL status handled well, but TASK ABORTED leads to filesystem corruption.
Quoted 3 lines
Journals can fail to recover in cases where the underlying medium is corrupted. If TASK ABORTED was because of QErr, what was the original failure?
See above. No "medium" corruption happened.
Quoted 2 lines
Also, what was going on in the system (and what device was this ... iSCSI I guess) ...
It doesn't matter. It happens with FC transport as well.
Quoted 3 lines
I assume nothing powered down, so it's not a caching problem (and that, since you seem to be using TCQ you do have your caches set to write through).
The target stays pretty well and healthy.
Quoted 13 lines
Quoted 1 line
I don't have time for further investigations, but, if somebody prepare a patch to fix that, I'm willing to assist in testing.
We'll need a bit more data to identify an actual root cause for this problem before anyone can prepare a patch to fix it.
James
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
James Bottomley
maintainerQuoted 11 lines · Vladislav Bolkhovitin
Quoted 2 lines · James Bottomley
I'm not sure your conclusions necessarily follow your data. What was the reason for the TASK ABORTED (I'd guess QErr settings, right)?
It was my desire/curiosity during tests of SCST (http://scst.sf.net),
when it working with several initiators with different transports over the same set of devices, each of them having with TAS bit in the control mode page set. According to SAM, in this case TASK ABORTED status can be returned at any time, similarly to QUEUE FULL, i.e. IMHO such command just should be retried. But QUEUE FULL status handled well, but TASK ABORTED leads to filesystem corruption.
So this is with a soft target implementation ... so it could be an ordering issue inside the target that's causing the filesystem corruption on error.
if you specifically set TAS=1 you're giving up the right to know what caused the command termination. With insufficient information, it's really unsafe to simply retry, which is why the mid layer just returns TASK ABORTED as an error. If you set TAS=0 we'll get a check condition/unit attention explaining what happened (usually commands cleared by another initiator) and we'll explicitly do the right thing based on the sense data.
One of my test suites has an initiator which randomly spits errors. I've yet to see it cause an error that an ext3 journal can't recover from. So, if there's a genuine problem we need a nice test case to pass to the filesystem people.
James
Quoted 4 lines · James Bottomley
Quoted 2 lines · Vladislav Bolkhovitin
James Bottomley wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure your conclusions necessarily follow your data. What was the reason for the TASK ABORTED (I'd guess QErr settings, right)?
It was my desire/curiosity during tests of SCST (http://scst.sf.net), when it working with several initiators with different transports over the same set of devices, each of them having with TAS bit in the control mode page set. According to SAM, in this case TASK ABORTED status can be returned at any time, similarly to QUEUE FULL, i.e. IMHO such command just should be retried. But QUEUE FULL status handled well, but TASK ABORTED leads to filesystem corruption.
So this is with a soft target implementation ... so it could be an ordering issue inside the target that's causing the filesystem corruption on error.
Target offers no ordering guarantees for SIMPLE commands and frankly says that to initiator via QUEUE ALGORITHM MODIFIER value 1 in the control mode page. As we know, initiator doesn't use ORDERED tags (and it really doesn't use them according to the logs), so if it's an ordering issue, it's at the initiator's side.
Quoted 7 lines
if you specifically set TAS=1 you're giving up the right to know what caused the command termination. With insufficient information, it's really unsafe to simply retry, which is why the mid layer just returns TASK ABORTED as an error. If you set TAS=0 we'll get a check condition/unit attention explaining what happened (usually commands cleared by another initiator) and we'll explicitly do the right thing based on the sense data.
But having TAS=1 is legal, right? So it should be handled well. If TAS=0, TASK ABORTED can't be returned, it would be illegal. So, TASK ABORTED status can only be returned with TAS=1.
Quoted 4 lines
One of my test suites has an initiator which randomly spits errors. I've yet to see it cause an error that an ext3 journal can't recover from. So, if there's a genuine problem we need a nice test case to pass to the filesystem people.
If you need a clear testcase (IMHO, in this case it isn't needed, because it's clear without it), I can prepare a patch for SCST to randomly return TASK ABORTED status.
You can get the latest version of SCST and the target drivers using SVN:
$ svn co https://scst.svn.sourceforge.net/svnroot/scst
Quoted 8 lines
James
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
James Bottomley
maintainerQuoted 34 lines · Vladislav Bolkhovitin
Quoted 15 lines · James Bottomley
Quoted 5 lines · Vladislav Bolkhovitin
Quoted 2 lines · James Bottomley
I'm not sure your conclusions necessarily follow your data. What was the reason for the TASK ABORTED (I'd guess QErr settings, right)?
Quoted 7 lines
It was my desire/curiosity during tests of SCST (http://scst.sf.net),
when it working with several initiators with different transports over the same set of devices, each of them having with TAS bit in the control mode page set. According to SAM, in this case TASK ABORTED status can be returned at any time, similarly to QUEUE FULL, i.e. IMHO such command just should be retried. But QUEUE FULL status handled well, but TASK ABORTED leads to filesystem corruption.Quoted 3 lines
So this is with a soft target implementation ... so it could be an ordering issue inside the target that's causing the filesystem corruption on error.
Target offers no ordering guarantees for SIMPLE commands and frankly says that to initiator via QUEUE ALGORITHM MODIFIER value 1 in the control mode page. As we know, initiator doesn't use ORDERED tags (and it really doesn't use them according to the logs), so if it's an ordering issue, it's at the initiator's side.
Quoted 7 lines
if you specifically set TAS=1 you're giving up the right to know what caused the command termination. With insufficient information, it's really unsafe to simply retry, which is why the mid layer just returns TASK ABORTED as an error. If you set TAS=0 we'll get a check condition/unit attention explaining what happened (usually commands cleared by another initiator) and we'll explicitly do the right thing based on the sense data.
Quoted 3 lines
But having TAS=1 is legal, right? So it should be handled well. If TAS=0, TASK ABORTED can't be returned, it would be illegal. So, TASK ABORTED status can only be returned with TAS=1.
Driving with your handbrake on is legal too ... that doesn't mean you should do it ... and it certainly doesn't give you a legitimate complaint against the manufacturer of your car for excessive brake pad wear.
We handle TASK ABORTED as well as we can (by failing it). For better handling set TAS=0 and we'll handle the individual cases according to the sense codes.
Quoted 5 lines
Quoted 4 lines
One of my test suites has an initiator which randomly spits errors. I've yet to see it cause an error that an ext3 journal can't recover from. So, if there's a genuine problem we need a nice test case to pass to the filesystem people.
Quoted 7 lines
If you need a clear testcase (IMHO, in this case it isn't needed, because it's clear without it), I can prepare a patch for SCST to randomly return TASK ABORTED status.
You can get the latest version of SCST and the target drivers using SVN:
$ svn co https://scst.svn.sourceforge.net/svnroot/scst
There's no real need to bother with setting all this up ... a simple initiator modification randomly to return TASK ABORTED should suffice.
James
Quoted 22 lines · James Bottomley
Quoted 20 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure your conclusions necessarily follow your data. What was the reason for the TASK ABORTED (I'd guess QErr settings, right)?
It was my desire/curiosity during tests of SCST (http://scst.sf.net), when it working with several initiators with different transports over the same set of devices, each of them having with TAS bit in the control mode page set. According to SAM, in this case TASK ABORTED status can be returned at any time, similarly to QUEUE FULL, i.e. IMHO such command just should be retried. But QUEUE FULL status handled well, but TASK ABORTED leads to filesystem corruption.
So this is with a soft target implementation ... so it could be an ordering issue inside the target that's causing the filesystem corruption on error.
Target offers no ordering guarantees for SIMPLE commands and frankly says that to initiator via QUEUE ALGORITHM MODIFIER value 1 in the control mode page. As we know, initiator doesn't use ORDERED tags (and it really doesn't use them according to the logs), so if it's an ordering issue, it's at the initiator's side.
Quoted 7 lines
if you specifically set TAS=1 you're giving up the right to know what caused the command termination. With insufficient information, it's really unsafe to simply retry, which is why the mid layer just returns TASK ABORTED as an error. If you set TAS=0 we'll get a check condition/unit attention explaining what happened (usually commands cleared by another initiator) and we'll explicitly do the right thing based on the sense data.
Quoted 1 line
But having TAS=1 is legal, right? So it should be handled well. If TAS=0, TASK ABORTED can't be returned, it would be illegal. So, TASK ABORTED status can only be returned with TAS=1.
Quoted 8 lines
Driving with your handbrake on is legal too ... that doesn't mean you should do it ... and it certainly doesn't give you a legitimate complaint against the manufacturer of your car for excessive brake pad wear.
We handle TASK ABORTED as well as we can (by failing it). For better handling set TAS=0 and we'll handle the individual cases according to the sense codes.
So, should I consider your words as you think that it's perfectly fine to corrupt file system for devices with TAS=1? Absolutely legal devices, repeat. Hence, in your opinion, no further investigation should be done?
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
One of my test suites has an initiator which randomly spits errors. I've yet to see it cause an error that an ext3 journal can't recover from. So, if there's a genuine problem we need a nice test case to pass to the filesystem people.
Quoted 8 lines
Quoted 5 lines
If you need a clear testcase (IMHO, in this case it isn't needed, because it's clear without it), I can prepare a patch for SCST to randomly return TASK ABORTED status.
You can get the latest version of SCST and the target drivers using SVN:
$ svn co https://scst.svn.sourceforge.net/svnroot/scstThere's no real need to bother with setting all this up ... a simple initiator modification randomly to return TASK ABORTED should suffice.
Yes, you're right. Then, I suppose, Mike Christie should be the best person to do it?
Vlad
Quoted 8 lines · Vladislav Bolkhovitin
Quoted 3 lines · James Bottomley
We handle TASK ABORTED as well as we can (by failing it). For better handling set TAS=0 and we'll handle the individual cases according to the sense codes.
So, should I consider your words as you think that it's perfectly fine to corrupt file system for devices with TAS=1? Absolutely legal devices, repeat. Hence, in your opinion, no further investigation should be done?
I don't know how you manage to read his words this way. I understand him to mean that the SCSI subsystem is doing the best in can under the somewhat misconfigured circumstances, and the problem lies in the FS not handling errors correctly.
Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
James Bottomley
maintainerQuoted 40 lines · Vladislav Bolkhovitin
Quoted 39 lines · James Bottomley
Quoted 30 lines
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
I'm not sure your conclusions necessarily follow your data. What was the reason for the TASK ABORTED (I'd guess QErr settings, right)?
It was my desire/curiosity during tests of SCST (http://scst.sf.net),
when it working with several initiators with different transports over the same set of devices, each of them having with TAS bit in the control mode page set. According to SAM, in this case TASK ABORTED status can be returned at any time, similarly to QUEUE FULL, i.e. IMHO such command just should be retried. But QUEUE FULL status handled well, but TASK ABORTED leads to filesystem corruption.Quoted 3 lines
So this is with a soft target implementation ... so it could be an ordering issue inside the target that's causing the filesystem corruption on error.
Target offers no ordering guarantees for SIMPLE commands and frankly says that to initiator via QUEUE ALGORITHM MODIFIER value 1 in the control mode page. As we know, initiator doesn't use ORDERED tags (and it really doesn't use them according to the logs), so if it's an ordering issue, it's at the initiator's side.
Quoted 7 lines
if you specifically set TAS=1 you're giving up the right to know what caused the command termination. With insufficient information, it's really unsafe to simply retry, which is why the mid layer just returns TASK ABORTED as an error. If you set TAS=0 we'll get a check condition/unit attention explaining what happened (usually commands cleared by another initiator) and we'll explicitly do the right thing based on the sense data.
Quoted 3 lines
But having TAS=1 is legal, right? So it should be handled well. If TAS=0, TASK ABORTED can't be returned, it would be illegal. So, TASK ABORTED status can only be returned with TAS=1.
Driving with your handbrake on is legal too ... that doesn't mean you should do it ... and it certainly doesn't give you a legitimate complaint against the manufacturer of your car for excessive brake pad wear.
Quoted 7 lines
Quoted 3 lines
We handle TASK ABORTED as well as we can (by failing it). For better handling set TAS=0 and we'll handle the individual cases according to the sense codes.
So, should I consider your words as you think that it's perfectly fine to corrupt file system for devices with TAS=1? Absolutely legal devices, repeat. Hence, in your opinion, no further investigation should be done?
Logic wouldn't support such a conclusion.
You have intertwined two issues
1. How should the mid layer handle TASK ABORTED. I think we've reached the point where returning I/O error is the best we can do, but if TAS=0 we could have used the sense data to do better.
2. Should a request I/O error cause corruption in ext3 that can't be recovered by a journal replay. I think the answer here is no, so there needs to be an easily reproducible test case to pass to the filesystem people.
James
Quoted 26 lines · James Bottomley
Quoted 26 lines
Quoted 26 lines
Quoted 19 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I'm not sure your conclusions necessarily follow your data. What was the reason for the TASK ABORTED (I'd guess QErr settings, right)?
It was my desire/curiosity during tests of SCST (http://scst.sf.net), when it working with several initiators with different transports over the same set of devices, each of them having with TAS bit in the control mode page set. According to SAM, in this case TASK ABORTED status can be returned at any time, similarly to QUEUE FULL, i.e. IMHO such command just should be retried. But QUEUE FULL status handled well, but TASK ABORTED leads to filesystem corruption.
So this is with a soft target implementation ... so it could be an ordering issue inside the target that's causing the filesystem corruption on error.
Target offers no ordering guarantees for SIMPLE commands and frankly says that to initiator via QUEUE ALGORITHM MODIFIER value 1 in the control mode page. As we know, initiator doesn't use ORDERED tags (and it really doesn't use them according to the logs), so if it's an ordering issue, it's at the initiator's side.
Quoted 7 lines
if you specifically set TAS=1 you're giving up the right to know what caused the command termination. With insufficient information, it's really unsafe to simply retry, which is why the mid layer just returns TASK ABORTED as an error. If you set TAS=0 we'll get a check condition/unit attention explaining what happened (usually commands cleared by another initiator) and we'll explicitly do the right thing based on the sense data.
Quoted 1 line
But having TAS=1 is legal, right? So it should be handled well. If TAS=0, TASK ABORTED can't be returned, it would be illegal. So, TASK ABORTED status can only be returned with TAS=1.
Driving with your handbrake on is legal too ... that doesn't mean you should do it ... and it certainly doesn't give you a legitimate complaint against the manufacturer of your car for excessive brake pad wear.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
We handle TASK ABORTED as well as we can (by failing it). For better handling set TAS=0 and we'll handle the individual cases according to the sense codes.
So, should I consider your words as you think that it's perfectly fine to corrupt file system for devices with TAS=1? Absolutely legal devices, repeat. Hence, in your opinion, no further investigation should be done?
Logic wouldn't support such a conclusion.
Sorry, lately I've got too many "I won't bother, this is your problem" style answers
Quoted 9 lines
You have intertwined two issues
1. How should the mid layer handle TASK ABORTED. I think we've reached the point where returning I/O error is the best we can do, but if TAS=0 we could have used the sense data to do better.
2. Should a request I/O error cause corruption in ext3 that can't be recovered by a journal replay. I think the answer here is no, so there needs to be an easily reproducible test case to pass to the filesystem people.
OK, I see you point. As I already wrote, I can assist only in testing here.
Quoted 8 lines
James
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · James Bottomley
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
if you specifically set TAS=1 you're giving up the right to know what caused the command termination. With insufficient information, it's really unsafe to simply retry, which is why the mid layer just returns TASK ABORTED as an error. If you set TAS=0 we'll get a check condition/unit attention explaining what happened (usually commands cleared by another initiator) and we'll explicitly do the right thing based on the sense data.
Actually, having TAS=1 has a considerably advantage over TAS=0 from error recovery point of view. With TAS=1 all aborted commands are supposed to be returned immediately to all affected initiators. With TAS=0 affected initiators will not receive any notification about aborted commands, only COMMANDS CLEARED BY ANOTHER INITIATOR UA will be established. So, they will know about that only after there will be timeout for their commands.
Thus, with TAS=1 almost immediate error recovery is possible, but with TAS=0 error recovery is possible after timeout, which for SSC devices can be hours.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
But having TAS=1 is legal, right? So it should be handled well. If TAS=0, TASK ABORTED can't be returned, it would be illegal. So, TASK ABORTED status can only be returned with TAS=1.
Driving with your handbrake on is legal too ... that doesn't mean you should do it ... and it certainly doesn't give you a legitimate complaint against the manufacturer of your car for excessive brake pad wear.
We handle TASK ABORTED as well as we can (by failing it). For better handling set TAS=0 and we'll handle the individual cases according to the sense codes.
After some digging in SAM/SPC I've figured out that TASK ABORTED status can be returned exactly in the same circumstances as COMMANDS CLEARED BY ANOTHER INITIATOR UA, it only depends from TAS bit, which way of the notification is used. So, TASK ABORTED status carries the same information as COMMANDS CLEARED BY ANOTHER INITIATOR UA and should be handled at the same way. I.e., if for COMMANDS CLEARED BY ANOTHER INITIATOR UA the affected commands are restarted, they should be restarted for TASK ABORTED status as well.
Vlad
Quoted 33 lines · Andrew Morton
Quoted 30 lines · On Mon, 19 Nov 2007 05:44:01 -0800 (PST) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=9405
Summary: iSCSI does not implement ordering guarantees required by
e.g. journaling filesystems
Product: IO/Storage
Version: 2.5
KernelVersion: 2.6.23.1
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: SCSI
AssignedTo: [email protected]
ReportedBy: [email protected]Most recent kernel where this bug did not occur: (new issue)
Distribution: any
Hardware Environment: (does not apply)
Software Environment: (does not apply) Problem Description: The sd (SCSI disk) driver ignores block device barriers (REQ_HARDBARRIER). The iSCSI code in the kernel sends all iSCSI commands with
flagISCSI_ATTR_SIMPLEto the iSCSI target. This means that the target may
reorder these commands. Since a.o. correct operation of journaling filesystems
depends on being able to enforce the order of certain block write operations,
not enforcing write ordering is a bug. This can be solved by either adding
support forREQ_HARDBARRIERin the sd device or by replacingISCSI_ATTR_SIMPLE
byISCSI_ATTR_ORDERED.
Quoted 8 lines
Quoted 6 lines
Steps to reproduce: Source reading of
drivers/scsi/sd.canddrivers/scsi/libiscsi.c.References: SCSI Architecture Model - 3, paragraph 8.6
(http://www.t10.org/ftp/t10/drafts/sam3/sam3r14.pdf).(does iscsi have a maintainer?)
Attached is a patch to add me to the maintainers file so it will be easier to hunt me down in the future. It was made over 2.6.24-rc2.
Add Mike Christie to MAINTAINERS file.
MAINTAINERS +8 -0
@@ -2111,6 +2111,14 @@ L: [email protected] (subscribers-only) W: http://irda.sourceforge.net/ S: Maintained +iSCSI +P: Mike Christie +M: [email protected] +L: [email protected] +W: www.open-iscsi.org +T: git kernel.org:/pub/scm/linux/kernel/mnc/linux-2.6-iscsi.git +S: Maintained + ISAPNP P: Jaroslav Kysela M: [email protected]
Quoted 1 line · Mike Christie
+iSCSI
It's traditional to use all-caps here, even when the normal capitalisation is different. so I think this should be "ISCSI". Damned if I know why we have this convention, though.
Quoted 6 lines
MAINTAINERS +6
-0+P: Mike Christie +M: [email protected] +L: [email protected] +W: www.open-iscsi.org +T: git kernel.org:/pub/scm/linux/kernel/mnc/linux-2.6-iscsi.git +S: Maintained
Signature · 4 lines
Intel are signing my paycheques ... these opinions are still mine "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step."
Quoted 7 lines · Matthew Wilcox
Quoted 1 line · Mike Christie
+iSCSI
It's traditional to use all-caps here, even when the normal capitalisation is different. so I think this should be "ISCSI". Damned if I know why we have this convention, though.
Thanks. Here is a updated patch.
Add Mike Christie to MAINTAINERS file.
v2 - used all caps for system
MAINTAINERS +8 -0
@@ -2111,6 +2111,14 @@ L: [email protected] (subscribers-only) W: http://irda.sourceforge.net/ S: Maintained +ISCSI +P: Mike Christie +M: [email protected] +L: [email protected] +W: www.open-iscsi.org +T: git kernel.org:/pub/scm/linux/kernel/mnc/linux-2.6-iscsi.git +S: Maintained + ISAPNP P: Jaroslav Kysela M: [email protected]