Skip to contentSkip to conversation

fc transport: extensions for fast fail and dev loss

RFC In review James Smart 4 participants 15 messages Jun 20, 2006 – Aug 10, 2006 SCSI

lore
Share & tools
lore
#0 lore raw

RFC 1/1 fc transport: extensions for fast fail and dev loss

Folks,

The following addresses some long standing todo items I've had in the FC transport. They primarily arise when considering multipathing, or trying to marry driver internal state to transport state. It is intended that this same type of functionality would be usable in other transports as well.

Here's what is contained:

- dev_loss_tmo LLDD callback :
Currently, there is no notification to the LLDD of when the transport gives up on the device returning and starts to return DID_NO_CONNECT in the queuecommand helper function. This callback notifies the LLDD that the transport has now given up on the rport, thereby acknowledging the prior fc_remote_port_delete() call. The callback also expects the LLDD to initiate the termination of any outstanding i/o on the rport.

- fast_io_fail_tmo and LLD callback:
There are some cases where it may take a long while to truly determine device loss, but the system is in a multipathing configuration that if the i/o was failed quickly (faster than dev_loss_tmo), it could be redirected to a different path and completed sooner (assuming the multipath thing knew that the sdev was blocked).

iSCSI is one of the transports that may vary dev_loss_tmo values per session, and you would like fast io failure.

- fast_loss_time recommendation:
In discussing how a admin should set dev_loss_tmo in a multipathing environment, it became apparent that we expected the admin to know a lot. They had to know the transport type, what the minimum setting can be that still survives normal link bouncing, and they may even have to know about device specifics. For iSCSI, the proper loss time may vary widely from session to session.

This attribute is an exported "recommendation" by the LLDD and transport on what the lowest setting for dev_loss_tmo should be for a multipathing environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

I have one criticism of these changes. The callbacks are calling into the LLDD with an rport post the driver's rport_delete call. What it means is that we are essentially extending the lifetime of an rport until the dev_loss_tmo call occurs.

-- james s

include/scsi/scsi_transport_fc.h +7 -0
diff -upNr a/include/scsi/scsi_transport_fc.h b/include/scsi/scsi_transport_fc.h
@@ -187,6 +187,8 @@ struct fc_rport {	/* aka fc_starget_attr
 
 	/* Dynamic Attributes */
 	u32 dev_loss_tmo;	/* Remote Port loss timeout in seconds. */
+	u32 fast_loss_time;	/* Fastest setting for dev_loss_tmo to
+				 * detect a path failure. */
 
 	/* Private (Transport-managed) Attributes */
 	u64 node_name;
@@ -195,6 +197,7 @@ struct fc_rport {	/* aka fc_starget_attr
 	u32 roles;
 	enum fc_port_state port_state;	/* Will only be ONLINE or UNKNOWN */
 	u32 scsi_target_id;
+	u32 fast_io_fail_tmo;
 
 	/* exported data */
 	void *dd_data;			/* Used for driver-specific storage */
@@ -399,6 +402,7 @@ struct fc_host_attrs {
 struct fc_function_template {
 	void    (*get_rport_dev_loss_tmo)(struct fc_rport *);
 	void	(*set_rport_dev_loss_tmo)(struct fc_rport *, u32);
+	void    (*get_rport_fast_loss_time)(struct fc_rport *);
 
 	void	(*get_starget_node_name)(struct scsi_target *);
 	void	(*get_starget_port_name)(struct scsi_target *);
@@ -416,6 +420,9 @@ struct fc_function_template {
 
 	int	(*issue_fc_host_lip)(struct Scsi_Host *);
 
+	void    (*dev_loss_tmo_callbk)(struct fc_rport *);
+	void	(*terminate_rport_io)(struct fc_rport *);
+
 	/* allocation lengths for host-specific data */
 	u32	 			dd_fcrport_size;
 
drivers/scsi/scsi_transport_fc.c +127 -4

224 lines not on this page · raw message

#1 lore raw
Quoted 8 lines · James Smart

Folks,

The following addresses some long standing todo items I've had in the FC transport. They primarily arise when considering multipathing, or trying to marry driver internal state to transport state. It is intended that this same type of functionality would be usable in other transports as well.

I agree we need something like this. iSCSI is going to move to something closer to FC in 2.6.19 to better integrate qla4xxx and give FC and iSCSI a similar interface when it makes sense.

Quoted 10 lines

Here's what is contained:

- dev_loss_tmo LLDD callback :
Currently, there is no notification to the LLDD of when the transport gives up on the device returning and starts to return DID_NO_CONNECT in the queuecommand helper function. This callback notifies the LLDD that the transport has now given up on the rport, thereby acknowledging the prior fc_remote_port_delete() call. The callback also expects the LLDD to initiate the termination of any outstanding i/o on the rport.

iSCSI does something like this at the lower level right now. For the common lower level iscsi layer that software drivers share we have a callback that allows drivers to do the same thing as your dev_loss_tmo callback. When we move to the new model we will need something like this.

Quoted 10 lines

- fast_io_fail_tmo and LLD callback:
There are some cases where it may take a long while to truly determine device loss, but the system is in a multipathing configuration that if the i/o was failed quickly (faster than dev_loss_tmo), it could be redirected to a different path and completed sooner (assuming the multipath thing knew that the sdev was blocked).

iSCSI is one of the transports that may vary dev_loss_tmo values per session, and you would like fast io failure.

Agree. Currently we are sort of doing this in userspace, but since qla4xxx does more in the kernel we would like to move it there so qla4xxxx and other HW iscsi cards do not have to jump to so many hoops to use the functionality.

Quoted 13 lines

- fast_loss_time recommendation:
In discussing how a admin should set dev_loss_tmo in a multipathing environment, it became apparent that we expected the admin to know a lot. They had to know the transport type, what the minimum setting can be that still survives normal link bouncing, and they may even have to know about device specifics. For iSCSI, the proper loss time may vary widely from session to session.

This attribute is an exported "recommendation" by the LLDD and transport on what the lowest setting for dev_loss_tmo should be for a multipathing environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

Quoted 5 lines

I have one criticism of these changes. The callbacks are calling into the LLDD with an rport post the driver's rport_delete call. What it means is that we are essentially extending the lifetime of an rport until the dev_loss_tmo call occurs.

So is the fast_io_fail_tmo callback the terminate_rport_io callback? If so, are we supposed to unblock the rport/session/target from fc_timeout_fail_rport_io and call into the LLD and the LLD will set some bit (or maybe check some rport/session/target/scsi_device bit) so that incoming IO and IO sitting in the driver will be failed with something like DID_BUS_BUSY so it goes to the upper layers? I think I only the unblock happen on success or fc_starget_delete, so IO in the driver looks like it can get failed upwards but IO sitting in the queue sits there until fc_rport_final_delete or success.

If that is correct, what about a new device state? When the fail fast tmo expires we can set the device to the new state, run the queue and incoming IO or IO in the request_queue marked with FAILFAST can be failed upwards by scsi-ml.

I just woke up though :)

#2 lore raw
Quoted 2 lines · Mike Christie

So is the fast_io_fail_tmo callback the terminate_rport_io callback?

Yes.
When fast_io_fail_tmo expires, it calls the terminate_rport_io() callback.

Quoted 3 lines

If
so, are we supposed to unblock the rport/session/target from fc_timeout_fail_rport_io

No... don't unblock.

Quoted 4 lines

and call into the LLD and the LLD will set some
bit (or maybe check some rport/session/target/scsi_device bit) so that incoming IO and IO sitting in the driver will be failed with something like DID_BUS_BUSY so it goes to the upper layers?

The way this is managed in the fc transport is - the LLD calls the transport when it establishes connectivity (an "add" call), and when it loses connectivity (a "delete" call). When the transport receives the delete call, it changs the rport state, blocks the rport, and starts the dev_loss timeout (and potentially the fast_io_fail_tmo if < dev_loss). If the LLD makes the add call prior to dev_loss expiring, then updates the state, and unblocks the rport. If dev_loss expires, it updates state again (essentially the true deleted state) and tears down the target tree.

To deal with requests being received while blocked, etc - the LLD's use a helper routine (fc_remote_port_chkready()), which validates the rport state, and if not valid (e.g. blocked or removed) returns the appropriate status to return to the midlayer. If blocked, it returns DID_IMM_RETRY. If deleted, it returns DID_NO_CONNECT.

What the above never dealt with was the i/o already in the driver. The driver always had the option to terminate the active i/o when the loss of connectivity occured, or it could just wait for it to timeout, etc and be killed that way. This patch added the callback at dev_loss_tmo to guarantee i/o is killed, and added the fast_io_fail_tmo if you wanted a faster guarantee. If fast_io_fail_tmo expires and the callback is called - it just kills the outstanding i/o and does nothing to the rport's blocked state.

Quoted 4 lines

I think I only the
unblock happen on success or fc_starget_delete, so IO in the driver looks like it can get failed upwards but IO sitting in the queue sits there until fc_rport_final_delete or success.

Yeah - essentially this is correct. I hope the above read that way. I'm also hoping the iSER folks are reading this to get the general feel of what's happening with block, dev_loss, etc.

Quoted 7 lines

If that is correct, what about a new device state? When the fail fast tmo expires we can set the device to the new state, run the queue and incoming IO or IO in the request_queue marked with FAILFAST can be failed upwards by scsi-ml.

I just woke up though :)

Sounds reasonable. It is adding a new semantic to what was meant by fast_fail - but it's in line with our goal. The goal was to terminate i/o so that they could be quick rescheduled on a different path rather than wait (what may be a long time) for the dev_loss connectivity timer to fire. Makes sense you would want to make new i/o requests bound by that same window.

-- james s

#3 lore raw
Quoted 36 lines · James Smart
Quoted 2 lines · Mike Christie

So is the fast_io_fail_tmo callback the terminate_rport_io callback?

Yes.
When fast_io_fail_tmo expires, it calls the terminate_rport_io() callback.

Quoted 3 lines

If
so, are we supposed to unblock the rport/session/target from fc_timeout_fail_rport_io

No... don't unblock.

Quoted 4 lines

and call into the LLD and the LLD will set some
bit (or maybe check some rport/session/target/scsi_device bit) so that incoming IO and IO sitting in the driver will be failed with something like DID_BUS_BUSY so it goes to the upper layers?

The way this is managed in the fc transport is - the LLD calls the transport when it establishes connectivity (an "add" call), and when it loses connectivity (a "delete" call). When the transport receives the delete call, it changes the rport state, blocks the rport, and starts the dev_loss timeout (and potentially the fast_io_fail_tmo if < dev_loss). If the LLD makes the add call prior to dev_loss expiring, then updates the state, and unblocks the rport. If dev_loss expires, it updates state again (essentially the true deleted state) and tears down the target tree.

To deal with requests being received while blocked, etc - the LLD's use a helper routine (fc_remote_port_chkready()), which validates the rport state, and if not valid (e.g. blocked or removed) returns the appropriate status to return to the midlayer. If blocked, it returns DID_IMM_RETRY. If deleted, it returns DID_NO_CONNECT.

Quoted 8 lines

What the above never dealt with was the i/o already in the driver. The driver always had the option to terminate the active i/o when the loss of connectivity occurred, or it could just wait for it to timeout, etc and be killed that way. This patch added the callback at dev_loss_tmo to guarantee i/o is killed, and added the fast_io_fail_tmo if you wanted a faster guarantee. If fast_io_fail_tmo expires and the callback is called - it just kills the outstanding i/o and does nothing to the rport's blocked state.

Haven't most drivers / board firmware generally cleaned up any outstanding i/o at the time (or shortly thereafter) of the fc_remote_port_delete() call? I would think it reasonable to just require that the driver clean up the i/o after calling fc_remote_port_delete(). Is there a significant reason to keep the i/o alive in the driver? The rport has just been deleted.... Would this eliminate the need for the callback? If the driver implements this, could it just have a NULL callback routine?

Mike

Quoted 32 lines
Quoted 5 lines

I think I only the
unblock happen on success or fc_starget_delete, so IO in the driver looks like it can get failed upwards but IO sitting in the queue sits there until fc_rport_final_delete or success.

Yeah - essentially this is correct. I hope the above read that way. I'm also hoping the iSER folks are reading this to get the general feel of what's happening with block, dev_loss, etc.

Quoted 7 lines

If that is correct, what about a new device state? When the fail fast tmo expires we can set the device to the new state, run the queue and incoming IO or IO in the request_queue marked with FAILFAST can be failed upwards by scsi-ml.

I just woke up though :)

Sounds reasonable. It is adding a new semantic to what was meant by fast_fail - but it's in line with our goal. The goal was to terminate i/o so that they could be quick rescheduled on a different path rather than wait (what may be a long time) for the dev_loss connectivity timer to fire. Makes sense you would want to make new i/o requests bound by that same window.

-- james s
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
Quoted 7 lines · Michael Reed

Haven't most drivers / board firmware generally cleaned up any outstanding i/o at the time (or shortly thereafter) of the fc_remote_port_delete() call? I would think it reasonable to just require that the driver clean up the i/o after calling fc_remote_port_delete(). Is there a significant reason to keep the i/o alive in the driver? The rport has just been deleted.... Would this eliminate the need for the callback? If the driver implements this, could it just have a NULL callback routine?

You don't want to kill i/o at the time that the rport loses connectivity (e.g. when we call delete). The biggest reason is for devices that support FCP Recovery (Tape being a prime example, but it's not limited to them). There are also several topology glitches, which can be seconds in length, which are abberant, and really didn't reflect true device loss. Why kill and reissue these io's as well ? it really just adds more load to the system to abort and retry the i/o's. Also, FC-DA compliance is recommending that the login and the exchanges stay around as long as possible (you could argue the diagram mandates it). So, for the best "FC behavior" you really shouldn't kill the i/o until dev_loss_tmo fires.

-- james s

PS: I think you're highlighting my complaint with the patch. I should
be renaming the "delete" function as it really isn't delete. The rport isn't truly "deleted" until dev_loss_tmo fires. We didn't consider this when we dropped the block/unblock_rport routines in favor of the more straight-forward add/delete calls. At this point, I'd rather document the lifetime in the api/header than deal with namechanges and its affects on drivers/distros.

#5 lore raw
In reply to James Smart
Quoted 10 lines · James Smart

Folks,

The following addresses some long standing todo items I've had in the FC transport. They primarily arise when considering multipathing, or trying to marry driver internal state to transport state. It is intended that this same type of functionality would be usable in other transports as well.

Here's what is contained:

Quoted 7 lines

- dev_loss_tmo LLDD callback :
Currently, there is no notification to the LLDD of when the transport gives up on the device returning and starts to return DID_NO_CONNECT in the queuecommand helper function. This callback notifies the LLDD that the transport has now given up on the rport, thereby acknowledging the prior fc_remote_port_delete() call. The callback also expects the LLDD to initiate the termination of any outstanding i/o on the rport.

I think this is fine.

Quoted 6 lines

- fast_io_fail_tmo and LLD callback:
There are some cases where it may take a long while to truly determine device loss, but the system is in a multipathing configuration that if the i/o was failed quickly (faster than dev_loss_tmo), it could be redirected to a different path and completed sooner (assuming the multipath thing knew that the sdev was blocked).

shouldn't we just always fail REQ_FAILFAST requests ASAP and totally ignore any kind of devloss timeout for them?

Quoted 4 lines

This attribute is an exported "recommendation" by the LLDD and transport on what the lowest setting for dev_loss_tmo should be for a multipathing environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

This kind of policy really doesn't belong into the kernel. I'd rather see a nice userspace command to get this right for the user as part of sg_utils or Jeffs infamous blktool.

Quoted 4 lines

I have one criticism of these changes. The callbacks are calling into the LLDD with an rport post the driver's rport_delete call. What it means is that we are essentially extending the lifetime of an rport until the dev_loss_tmo call occurs.

Which is okay as long as it's documented well enough.

#6 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines

- fast_io_fail_tmo and LLD callback:
There are some cases where it may take a long while to truly determine device loss, but the system is in a multipathing configuration that if the i/o was failed quickly (faster than dev_loss_tmo), it could be redirected to a different path and completed sooner (assuming the multipath thing knew that the sdev was blocked).

shouldn't we just always fail REQ_FAILFAST requests ASAP and totally ignore any kind of devloss timeout for them?

A couple of questions....
- This implies 1 by 1 implicit i/o aborts. Keep in mind that the connectivity to the device/target has been lost, so you can't send transport-level single-io abort requests, nor Target-level TMF's. So.. how much are you trying to guarantee this behavior to the upper layers ?

Please note that you may get differing behavior from different adapter/driver's. Some may support cancelling the i/o within the adapter (and properly protect against later link-side references), thus it works as desired. Others may not, and would then have to resort to implicit logouts - which will abort non-REQ_FAILFAST i/o's as well. This is ok if those i/o's are retryable (like on disks), but bad if they aren't (what if one of the luns were a tape?). Instead of implicit logouts, the driver may just ignore the REQ_FAILFAST flags all together and wait for dev_loss_tmo to kill things.

- Do you want a SCSI LLD looking at more than the scsi_cmnd ? (e.g. is it proper for it to be looking at the block request structure ?) Would this mean we want to reflect the block flag via a scsi_cmnd flag ?

- There's an argument on whether we're FC-DA compliant. Yes, Linux doesn't care and the above would be good for the system, but vendor selection still grades based on OS-ignorant transport standard compliance.

- Are we sure all the meaningful i/o will have REQ_FAILFAST set ?

Quoted 8 lines
Quoted 4 lines

This attribute is an exported "recommendation" by the LLDD and transport on what the lowest setting for dev_loss_tmo should be for a multipathing environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

This kind of policy really doesn't belong into the kernel. I'd rather see a nice userspace command to get this right for the user as part of sg_utils or Jeffs infamous blktool.

Makes sense. However, the tool may still need to get input from the transport/LLD - so something like this may still be needed. Actually, it would probably be this - we'd just change it to "a recommendation to a tool" instead of the admin.

-- james s

#7 lore raw
In reply to James Smart

Closing Statements:

I've attached the original RFC below. See
http://marc.theaimsgroup.com/?l=linux-scsi&m=115082917628466&w=2

I've updated it with what I perceive to be the position and resolution based on comments. Keep in mind that we're trying to lay the groundwork for common behavior and tunables between the transports.

Please let me know if I've mis-represented anything, or if there is a dissention in the resolution. I'd like to close on this.

Quoted 10 lines · James Smart

Folks,

The following addresses some long standing todo items I've had in the FC transport. They primarily arise when considering multipathing, or trying to marry driver internal state to transport state. It is intended that this same type of functionality would be usable in other transports as well.

Here's what is contained:

Quoted 7 lines

- dev_loss_tmo LLDD callback :
Currently, there is no notification to the LLDD of when the transport gives up on the device returning and starts to return DID_NO_CONNECT in the queuecommand helper function. This callback notifies the LLDD that the transport has now given up on the rport, thereby acknowledging the prior fc_remote_port_delete() call. The callback also expects the LLDD to initiate the termination of any outstanding i/o on the rport.

I believe there is no dissention on this change.
Please note: this is essentially a confirmation from the transport to the
LLD that the rport is fully deleted. Thus, the LLD must expect to see these callbacks as a normal part of the rport being terminated (even if it is not blocked).

I'll move forward with this.

Quoted 10 lines

- fast_io_fail_tmo and LLD callback:
There are some cases where it may take a long while to truly determine device loss, but the system is in a multipathing configuration that if the i/o was failed quickly (faster than dev_loss_tmo), it could be redirected to a different path and completed sooner (assuming the multipath thing knew that the sdev was blocked).

iSCSI is one of the transports that may vary dev_loss_tmo values per session, and you would like fast io failure.

The current transport implementation did not specify what happened to
active i/o (given to the driver, in the adapter, but not yet completed back to the midlayer) when a device was blocked, nor during the block-to->dev_loss transition period. It was up to the driver. Many assumed active i/o was immediately terminated, which is semi-consistent with the behavior of most drivers for most "connectivity loss" scenarios.

The conversations then started to jump around, considering what i/o's you
may want to have fail quickly, etc.

Here's my opinion:
We have the following points in time to look at:
(a) the device is blocked by the transport
(b) there is a time T, usually in a multipathing environment, where it would be useful to error the i/o early rather than wait for dev_loss It is assumed that any such i/o request would be marked REQ_FASTFAIL
(c) the dev_loss_tmo fires - we're to assume the device is gone
and at any time post (a), the device may return, unblock and never encounter points (b) and (c).

As for what happens to active i/o :

always: the driver can fail an i/o at any point in time if it deems
it appropriate.

at (a): There are scenarios where a short link perturbation may occur,
which may not disrupt the i/o. Therefore, we should not force io to be terminated.

at (b): Minimally, we should terminate all active i/o requests marked
as type REQ_FASTFAIL. From an api perspective, driver support for this is optional. And we must also assume that there will be implementations which have to abort all i/o in order to terminate those marked REQ_FASTFAIL. Is this acceptable ? (it meets the "always" condition above)

Q: so far we've limited the io to those w/ REQ_FASTFAIL.
Would we ever want to allow a user to fast fail all i/o regardless of the request flags ? (in case they flags weren't getting set on all the i/o the user wanted to see fail ?)

There's a desire to address pending i/o (those on the block request queue or new requests going there) so that if we've crossed point (b) that we also fail them. The proposal is to add a new state (device ? or queue ?), which would occur as of point (b). All REQ_FASTFAIL io on the queue, as well as on a new io, will be failed with a new i/o status if in this state. Non-REQ_FASTFAIL i/o would continue to enter/sit on the request queue until dev_loss_tmo fires.

at (c): per the dev_loss_tmo callback, all i/o should be terminated.
Their completions do not have to be synchronous to the return from the callback - they can occur afterward.

Comments ?

Assuming that folks agree, I'd like to do this in 2 patches:
- one that puts in the transport fast_io_fail_tmo and LLD callback
- another that adds the new state, io completion status, and does the handling of the request queue REQ_FASTFAIL i/o.

Quoted 9 lines

- fast_loss_time recommendation:
In discussing how a admin should set dev_loss_tmo in a multipathing environment, it became apparent that we expected the admin to know a lot. They had to know the transport type, what the minimum setting can be that still survives normal link bouncing, and they may even have to know about device specifics. For iSCSI, the proper loss time may vary widely from session to session.

Quoted 4 lines

This attribute is an exported "recommendation" by the LLDD and transport on what the lowest setting for dev_loss_tmo should be for a multipathing environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

The only objection was from Christoph - wanting a utility to get/set this stuff. However, the counter was this attribute was still meaningful, as it was the conduit to obtain a recommendation from the transport/LLD.

So - I assume this proceeds as is - with a change in it's description.

Quoted 5 lines

I have one criticism of these changes. The callbacks are calling into the LLDD with an rport post the driver's rport_delete call. What it means is that we are essentially extending the lifetime of an rport until the dev_loss_tmo call occurs.

It's ok - and adding the appropriate comments are fine.

Thanks.

-- james s

#8 lore raw
Quoted 31 lines · James Smart

Closing Statements:

I've attached the original RFC below. See
http://marc.theaimsgroup.com/?l=linux-scsi&m=115082917628466&w=2

I've updated it with what I perceive to be the position and resolution based on comments. Keep in mind that we're trying to lay the groundwork for common behavior and tunables between the transports.

Please let me know if I've mis-represented anything, or if there is a dissention in the resolution. I'd like to close on this.

Quoted 10 lines · James Smart

Folks,

The following addresses some long standing todo items I've had in the FC transport. They primarily arise when considering multipathing, or trying to marry driver internal state to transport state. It is intended that this same type of functionality would be usable in other transports as well.

Here's what is contained:

Quoted 7 lines

- dev_loss_tmo LLDD callback :
Currently, there is no notification to the LLDD of when the transport gives up on the device returning and starts to return DID_NO_CONNECT in the queuecommand helper function. This callback notifies the LLDD that the transport has now given up on the rport, thereby acknowledging the prior fc_remote_port_delete() call. The callback also expects the LLDD to initiate the termination of any outstanding i/o on the rport.

Quoted 7 lines

I believe there is no dissention on this change.
Please note: this is essentially a confirmation from the transport to the
LLD that the rport is fully deleted. Thus, the LLD must expect to see these callbacks as a normal part of the rport being terminated (even if it is not blocked).

I'll move forward with this.

Concur.

Quoted 22 lines
Quoted 10 lines

- fast_io_fail_tmo and LLD callback:
There are some cases where it may take a long while to truly determine device loss, but the system is in a multipathing configuration that if the i/o was failed quickly (faster than dev_loss_tmo), it could be redirected to a different path and completed sooner (assuming the
multipath thing knew that the sdev was blocked).
iSCSI is one of the transports that may vary dev_loss_tmo values
per session, and you would like fast io failure.

The current transport implementation did not specify what happened to
active i/o (given to the driver, in the adapter, but not yet completed back to the midlayer) when a device was blocked, nor during the block-to->dev_loss transition period. It was up to the driver. Many assumed active i/o was immediately terminated, which is semi-consistent with the behavior of most drivers for most "connectivity loss" scenarios.

The conversations then started to jump around, considering what i/o's you
may want to have fail quickly, etc.

Quoted 9 lines

Here's my opinion:
We have the following points in time to look at:
(a) the device is blocked by the transport
(b) there is a time T, usually in a multipathing environment, where it would be useful to error the i/o early rather than wait for dev_loss It is assumed that any such i/o request would be marked REQ_FASTFAIL
(c) the dev_loss_tmo fires - we're to assume the device is gone
and at any time post (a), the device may return, unblock and never encounter points (b) and (c).

REQ_FAILFAST is stored in the request structure. Are there "issues" with using scsi_cmnd.request in the lldd?

Quoted 17 lines

As for what happens to active i/o :

always: the driver can fail an i/o at any point in time if it deems
it appropriate.

at (a): There are scenarios where a short link perturbation may occur,
which may not disrupt the i/o. Therefore, we should not force io to be terminated.

at (b): Minimally, we should terminate all active i/o requests marked
as type REQ_FASTFAIL. From an api perspective, driver support for this is optional. And we must also assume that there will be implementations which have to abort all i/o in order to terminate those marked REQ_FASTFAIL. Is this acceptable ? (it meets the "always" condition above)

Quoted 5 lines

Q: so far we've limited the io to those w/ REQ_FASTFAIL.
Would we ever want to allow a user to fast fail all i/o regardless of the request flags ? (in case they flags weren't getting set on all the i/o the user wanted to see fail ?)

REQ_FAILFAST appears to influence retries during error recovery so there may be unexpected side effects of doing this. But, that said, I'd say yes. From my perspective, I'd make this the default behavior.

In talking with our volume manager people, the question raised was "Why would you want some i/o to fail quickly and some not?" They even considered non-i/o scsi commands.

I think that if the default behavior is to have fast_io_fail_tmo enabled, then it should be controlled by REQ_FAILFAST in the request. If the default is to have the timer disabled, i.e., an admin has to enable it (or define when it's enabled) then when enabled it should apply to every i/o. In examining the patch, it appears to be disabled by default, so our conclusion is that all i/o should fast fail when enabled. We also concur with having fast_io_fail_tmo disabled by default.

I guess this implies leave REQ_FAILFAST to error recovery. :)

Mike

Quoted 63 lines

There's a desire to address pending i/o (those on the block request queue or new requests going there) so that if we've crossed point (b) that we also fail them. The proposal is to add a new state (device ? or queue ?), which would occur as of point (b). All REQ_FASTFAIL io on the queue, as well as on a new io, will be failed with a new i/o status if in this state. Non-REQ_FASTFAIL i/o would continue to enter/sit on the request queue until dev_loss_tmo fires.

at (c): per the dev_loss_tmo callback, all i/o should be terminated.
Their completions do not have to be synchronous to the return from the callback - they can occur afterward.

Comments ?

Assuming that folks agree, I'd like to do this in 2 patches:
- one that puts in the transport fast_io_fail_tmo and LLD callback
- another that adds the new state, io completion status, and does the handling of the request queue REQ_FASTFAIL i/o.

Quoted 9 lines

- fast_loss_time recommendation:
In discussing how a admin should set dev_loss_tmo in a multipathing environment, it became apparent that we expected the admin to know a lot. They had to know the transport type, what the minimum setting can be that still survives normal link bouncing, and they may even have to know about device specifics. For iSCSI, the proper loss time may vary widely from session to session.

Quoted 6 lines

This attribute is an exported "recommendation" by the LLDD and
transport
on what the lowest setting for dev_loss_tmo should be for a
multipathing
environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

The only objection was from Christoph - wanting a utility to get/set this stuff. However, the counter was this attribute was still meaningful, as it was the conduit to obtain a recommendation from the transport/LLD.

So - I assume this proceeds as is - with a change in it's description.

Quoted 5 lines

I have one criticism of these changes. The callbacks are calling into the LLDD with an rport post the driver's rport_delete call. What it means is that we are essentially extending the lifetime of an rport until the dev_loss_tmo call occurs.

It's ok - and adding the appropriate comments are fine.

Thanks.

-- james s

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
Quoted 104 lines · Michael Reed
Quoted 30 lines · James Smart

Closing Statements:

I've attached the original RFC below. See
http://marc.theaimsgroup.com/?l=linux-scsi&m=115082917628466&w=2

I've updated it with what I perceive to be the position and resolution based on comments. Keep in mind that we're trying to lay the groundwork for common behavior and tunables between the transports.

Please let me know if I've mis-represented anything, or if there is a dissention in the resolution. I'd like to close on this.

Quoted 10 lines · James Smart

Folks,

The following addresses some long standing todo items I've had in the FC transport. They primarily arise when considering multipathing, or trying to marry driver internal state to transport state. It is intended that this same type of functionality would be usable in other transports as well.

Here's what is contained:

Quoted 7 lines

- dev_loss_tmo LLDD callback :
Currently, there is no notification to the LLDD of when the transport gives up on the device returning and starts to return DID_NO_CONNECT in the queuecommand helper function. This callback notifies the LLDD that the transport has now given up on the rport, thereby acknowledging the prior fc_remote_port_delete() call. The callback also expects the LLDD to initiate the termination of any outstanding i/o on the rport.

Quoted 7 lines

I believe there is no dissention on this change.
Please note: this is essentially a confirmation from the transport to the
LLD that the rport is fully deleted. Thus, the LLD must expect to see these callbacks as a normal part of the rport being terminated (even if it is not blocked).

I'll move forward with this.

Concur.

Quoted 19 lines
Quoted 8 lines

- fast_io_fail_tmo and LLD callback:
There are some cases where it may take a long while to truly determine device loss, but the system is in a multipathing configuration that if the i/o was failed quickly (faster than dev_loss_tmo), it could be redirected to a different path and completed sooner (assuming the
multipath thing knew that the sdev was blocked).
iSCSI is one of the transports that may vary dev_loss_tmo values
per session, and you would like fast io failure.

The current transport implementation did not specify what happened to
active i/o (given to the driver, in the adapter, but not yet completed back to the midlayer) when a device was blocked, nor during the block-to->dev_loss transition period. It was up to the driver. Many assumed active i/o was immediately terminated, which is semi-consistent with the behavior of most drivers for most "connectivity loss" scenarios.

The conversations then started to jump around, considering what i/o's you
may want to have fail quickly, etc.

Quoted 9 lines

Here's my opinion:
We have the following points in time to look at:
(a) the device is blocked by the transport
(b) there is a time T, usually in a multipathing environment, where it would be useful to error the i/o early rather than wait for dev_loss It is assumed that any such i/o request would be marked REQ_FASTFAIL
(c) the dev_loss_tmo fires - we're to assume the device is gone
and at any time post (a), the device may return, unblock and never encounter points (b) and (c).

REQ_FAILFAST is stored in the request structure. Are there "issues" with using scsi_cmnd.request in the lldd?

Quoted 16 lines

As for what happens to active i/o :

always: the driver can fail an i/o at any point in time if it deems
it appropriate.

at (a): There are scenarios where a short link perturbation may occur,
which may not disrupt the i/o. Therefore, we should not force io to be terminated.

at (b): Minimally, we should terminate all active i/o requests marked
as type REQ_FASTFAIL. From an api perspective, driver support for this is optional. And we must also assume that there will be implementations which have to abort all i/o in order to terminate those marked REQ_FASTFAIL. Is this acceptable ? (it meets the "always" condition above)

Quoted 5 lines

Q: so far we've limited the io to those w/ REQ_FASTFAIL.
Would we ever want to allow a user to fast fail all i/o regardless of the request flags ? (in case they flags weren't getting set on all the i/o the user wanted to see fail ?)

REQ_FAILFAST appears to influence retries during error recovery so there may be unexpected side effects of doing this. But, that said, I'd say yes. From my perspective, I'd make this the default behavior.

In talking with our volume manager people, the question raised was "Why would you want some i/o to fail quickly and some not?" They even considered non-i/o scsi commands.

Quoted 8 lines

I think that if the default behavior is to have fast_io_fail_tmo enabled, then it should be controlled by REQ_FAILFAST in the request. If the default is to have the timer disabled, i.e., an admin has to enable it (or define when it's enabled) then when enabled it should apply to every i/o. In examining the patch, it appears to be disabled by default, so our conclusion is that all i/o should fast fail when enabled. We also concur with having fast_io_fail_tmo disabled by default.

I think this also implies that our volume manager guys would be just as happy setting dev_loss_tmo to a small value and not use fast_io_fail_tmo.

Mike

Quoted 72 lines

I guess this implies leave REQ_FAILFAST to error recovery. :)

Mike

Quoted 49 lines

There's a desire to address pending i/o (those on the block request queue or new requests going there) so that if we've crossed point (b) that we also fail them. The proposal is to add a new state (device ? or queue ?), which would occur as of point (b). All REQ_FASTFAIL io on the queue, as well as on a new io, will be failed with a new i/o status if in this state. Non-REQ_FASTFAIL i/o would continue to enter/sit on the request queue until dev_loss_tmo fires.

at (c): per the dev_loss_tmo callback, all i/o should be terminated.
Their completions do not have to be synchronous to the return from the callback - they can occur afterward.

Comments ?

Assuming that folks agree, I'd like to do this in 2 patches:
- one that puts in the transport fast_io_fail_tmo and LLD callback
- another that adds the new state, io completion status, and does the handling of the request queue REQ_FASTFAIL i/o.

Quoted 8 lines

- fast_loss_time recommendation:
In discussing how a admin should set dev_loss_tmo in a multipathing environment, it became apparent that we expected the admin to know a lot. They had to know the transport type, what the minimum setting can be that still survives normal link bouncing, and they may even have to know about device specifics. For iSCSI, the proper loss time may vary widely from session to session.

Quoted 6 lines

This attribute is an exported "recommendation" by the LLDD and
transport
on what the lowest setting for dev_loss_tmo should be for a
multipathing
environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

The only objection was from Christoph - wanting a utility to get/set this stuff. However, the counter was this attribute was still meaningful, as it was the conduit to obtain a recommendation from the transport/LLD.

So - I assume this proceeds as is - with a change in it's description.

Quoted 5 lines

I have one criticism of these changes. The callbacks are calling into the LLDD with an rport post the driver's rport_delete call. What it means is that we are essentially extending the lifetime of an rport until the dev_loss_tmo call occurs.

It's ok - and adding the appropriate comments are fine.

Quoted 10 lines

Thanks.

-- james s

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw

Michael Reed wrote:

...snip...

Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
Quoted 1 line

- fast_io_fail_tmo and LLD callback:

at (b): Minimally, we should terminate all active i/o requests marked
as type REQ_FASTFAIL. From an api perspective, driver support for this is optional. And we must also assume that there will be implementations which have to abort all i/o in order to terminate those marked REQ_FASTFAIL. Is this acceptable ? (it meets the "always" condition above)

Quoted 5 lines

Q: so far we've limited the io to those w/ REQ_FASTFAIL.
Would we ever want to allow a user to fast fail all i/o regardless of the request flags ? (in case they flags weren't getting set on all the i/o the user wanted to see fail ?)

REQ_FAILFAST appears to influence retries during error recovery so there may be unexpected side effects of doing this. But, that said, I'd say yes. From my perspective, I'd make this the default behavior.

In talking with our volume manager people, the question raised was "Why would you want some i/o to fail quickly and some not?" They even considered non-i/o scsi commands.

...snip...

Quoted 1 line

In thinking about this over night, I would like to withdraw my previous comments. (Hence the snip!)

Let's take the case of real time capture from a device and post processing of that data. The capture operation would likely want a fast fail to avoid dropping data. The post processing of that data would like to wait for the device to return to avoid disruption and potential premature termination of the job.

Under the above scenario, assuming it's a valid scenario, are there mechanisms in place to allow an application to tag an i/o stream for fast fail?

Mike

#11 lore raw
Quoted 10 lines · Michael Reed

In thinking about this over night, I would like to withdraw my previous comments. (Hence the snip!)

Let's take the case of real time capture from a device and post processing of that data. The capture operation would likely want a fast fail to avoid dropping data. The post processing of that data would like to wait for the device to return to avoid disruption and potential premature termination of the job.

Quoted 3 lines

Under the above scenario, assuming it's a valid scenario, are there mechanisms in place to allow an application to tag an i/o stream for fast fail?

Two comments:
- This conflicts with Christoph's last comment about fast_fail failing all i/o's. I prefer the fail all, as its the easier, more straight-forward approach. Easiest to explain behavior for too.
- As for tagging i/o from an application, the only places I see in the kernel setting the flags that make REQ_FAILFAST get set are strictly in-the-kernel items (multipath). I don't see any way for an application to mark this. It implies that to do what you stated above, requires the app has to serially perform each mode - and to do so at a target level. E.g. set fast fail, perform capture, set no fast fail, perform post processing.

-- james s

#12 lore raw
In reply to James Smart
Quoted 10 lines · James Smart
Quoted 9 lines

Here's what is contained:

- dev_loss_tmo LLDD callback :
Currently, there is no notification to the LLDD of when the transport gives up on the device returning and starts to return DID_NO_CONNECT in the queuecommand helper function. This callback notifies the LLDD that the transport has now given up on the rport, thereby acknowledging the prior fc_remote_port_delete() call. The callback also expects the LLDD to initiate the termination of any outstanding i/o on the rport.

Quoted 7 lines

I believe there is no dissention on this change.
Please note: this is essentially a confirmation from the transport to the
LLD that the rport is fully deleted. Thus, the LLD must expect to see these callbacks as a normal part of the rport being terminated (even if it is not blocked).

I'll move forward with this.

ACK.

Quoted 31 lines
Quoted 9 lines

- fast_io_fail_tmo and LLD callback:
There are some cases where it may take a long while to truly determine device loss, but the system is in a multipathing configuration that if the i/o was failed quickly (faster than dev_loss_tmo), it could be redirected to a different path and completed sooner (assuming the multipath thing knew that the sdev was blocked).

iSCSI is one of the transports that may vary dev_loss_tmo values per session, and you would like fast io failure.

The current transport implementation did not specify what happened to
active i/o (given to the driver, in the adapter, but not yet completed back to the midlayer) when a device was blocked, nor during the block-to->dev_loss transition period. It was up to the driver. Many assumed active i/o was immediately terminated, which is semi-consistent with the behavior of most drivers for most "connectivity loss" scenarios.

The conversations then started to jump around, considering what i/o's you
may want to have fail quickly, etc.

Here's my opinion:
We have the following points in time to look at:
(a) the device is blocked by the transport
(b) there is a time T, usually in a multipathing environment, where it would be useful to error the i/o early rather than wait for dev_loss It is assumed that any such i/o request would be marked REQ_FASTFAIL
(c) the dev_loss_tmo fires - we're to assume the device is gone
and at any time post (a), the device may return, unblock and never encounter points (b) and (c).

Quoted 8 lines

As for what happens to active i/o :

always: the driver can fail an i/o at any point in time if it deems
it appropriate.

at (a): There are scenarios where a short link perturbation may occur,
which may not disrupt the i/o. Therefore, we should not force io to be terminated.

Ok..

Quoted 8 lines

at (b): Minimally, we should terminate all active i/o requests marked
as type REQ_FASTFAIL. From an api perspective, driver support for this is optional. And we must also assume that there will be implementations which have to abort all i/o in order to terminate those marked REQ_FASTFAIL. Is this acceptable ? (it meets the "always" condition above)

Quoted 5 lines

Q: so far we've limited the io to those w/ REQ_FASTFAIL.
Would we ever want to allow a user to fast fail all i/o regardless of the request flags ? (in case they flags weren't getting set on all the i/o the user wanted to see fail ?)

I think we should fail all. It's not like an unprivilegued process could request FASTFAIL. The administrator should know what she/he is doing.

Quoted 8 lines

There's a desire to address pending i/o (those on the block request queue or new requests going there) so that if we've crossed point (b) that we also fail them. The proposal is to add a new state (device ? or queue ?), which would occur as of point (b). All REQ_FASTFAIL io on the queue, as well as on a new io, will be failed with a new i/o status if in this state. Non-REQ_FASTFAIL i/o would continue to enter/sit on the request queue until dev_loss_tmo fires.

We have a queue per device, so adding another scsi_device state sound like the right way to go aheade.

Quoted 3 lines

at (c): per the dev_loss_tmo callback, all i/o should be terminated.
Their completions do not have to be synchronous to the return from the callback - they can occur afterward.

ACK.

Quoted 13 lines
Quoted 8 lines

- fast_loss_time recommendation:
In discussing how a admin should set dev_loss_tmo in a multipathing environment, it became apparent that we expected the admin to know a lot. They had to know the transport type, what the minimum setting can be that still survives normal link bouncing, and they may even have to know about device specifics. For iSCSI, the proper loss time may vary widely from session to session.

Quoted 4 lines

This attribute is an exported "recommendation" by the LLDD and transport on what the lowest setting for dev_loss_tmo should be for a multipathing environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

Quoted 5 lines

The only objection was from Christoph - wanting a utility to get/set this stuff. However, the counter was this attribute was still meaningful, as it was the conduit to obtain a recommendation from the transport/LLD.

So - I assume this proceeds as is - with a change in it's description.

I must say I'm still not happy with this. It's really policy that we try to keep out of the kernel.

#13 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 7 lines

at (b): Minimally, we should terminate all active i/o requests marked
as type REQ_FASTFAIL. From an api perspective, driver support for this is optional. And we must also assume that there will be implementations which have to abort all i/o in order to terminate those marked REQ_FASTFAIL. Is this acceptable ? (it meets the "always" condition above)

Quoted 8 lines
Quoted 5 lines

Q: so far we've limited the io to those w/ REQ_FASTFAIL.
Would we ever want to allow a user to fast fail all i/o regardless of the request flags ? (in case they flags weren't getting set on all the i/o the user wanted to see fail ?)

I think we should fail all. It's not like an unprivilegued process could request FASTFAIL. The administrator should know what she/he is doing.

Good. All it is.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

- fast_loss_time recommendation:
In discussing how a admin should set dev_loss_tmo in a multipathing environment, it became apparent that we expected the admin to know a lot. They had to know the transport type, what the minimum setting can be that still survives normal link bouncing, and they may even have to know about device specifics. For iSCSI, the proper loss time may vary widely from session to session.

This attribute is an exported "recommendation" by the LLDD and transport on what the lowest setting for dev_loss_tmo should be for a multipathing environment. Thus, the admin only needs to cat this attribute to obtain the value to echo into dev_loss_tmo.

Quoted 8 lines
Quoted 5 lines

The only objection was from Christoph - wanting a utility to get/set this stuff. However, the counter was this attribute was still meaningful, as it was the conduit to obtain a recommendation from the transport/LLD.

So - I assume this proceeds as is - with a change in it's description.

I must say I'm still not happy with this. It's really policy that we try to keep out of the kernel.

Ok. I'll drop this. I don't think it was that important for FC. Mike Christie had some better arguments for iSCSI.

-- james

#14 lore raw
Quoted 42 lines · James Smart
Quoted 7 lines · Christoph Hellwig
Quoted 7 lines

at (b): Minimally, we should terminate all active i/o requests marked
as type REQ_FASTFAIL. From an api perspective, driver support for this is optional. And we must also assume that there will be implementations which have to abort all i/o in order to terminate those marked REQ_FASTFAIL. Is this acceptable ? (it meets the "always" condition above)

Quoted 8 lines
Quoted 5 lines

Q: so far we've limited the io to those w/ REQ_FASTFAIL.
Would we ever want to allow a user to fast fail all i/o regardless of the request flags ? (in case they flags weren't getting set on all the i/o the user wanted to see fail ?)

I think we should fail all. It's not like an unprivilegued process could request FASTFAIL. The administrator should know what she/he is doing.

Good. All it is.

Quoted 21 lines
Quoted 21 lines
Quoted 8 lines

- fast_loss_time recommendation:
In discussing how a admin should set dev_loss_tmo in a multipathing environment, it became apparent that we expected the admin to know a lot. They had to know the transport type, what the minimum setting can be that still survives normal link bouncing, and they may even have to know about device specifics. For iSCSI, the proper loss time may vary widely from session to session.

Quoted 7 lines

This attribute is an exported "recommendation" by the LLDD and
transport
on what the lowest setting for dev_loss_tmo should be for a
multipathing
environment. Thus, the admin only needs to cat this attribute to
obtain
the value to echo into dev_loss_tmo.

The only objection was from Christoph - wanting a utility to get/set this
stuff. However, the counter was this attribute was still meaningful, as it
was the conduit to obtain a recommendation from the transport/LLD.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

So - I assume this proceeds as is - with a change in it's description.

I must say I'm still not happy with this. It's really policy that we try to keep out of the kernel.

Ok. I'll drop this. I don't think it was that important for FC. Mike Christie
had some better arguments for iSCSI.

For softwware iscsi we probably do not need kernel support since we do so much setup in userspace we can easily have the admin pass in a different argument for different defaults and we also have a way to set different values and store them in userspace. So no big deal. For Hw iscsi, we can work similar to FC so we should have no problem there too.