ALUA device handler update, part II
v8 cover ALUA device handler update, part II
As promised here is now the second part of my ALUA device handler update. This contains a major rework of the ALUA device handler as execution is moved onto a workqueue. This has the advantage that we avoid having to do multiple calls to the same LUN (as happens frequently when failing over a LUN with several paths) and finally retries are handled correctly. As some arrays are only capable of handling one STPG at a time I've added a blacklist flag which then uses a singlethreaded workqueue, thereby effectively synchronize STPG handling.
Thanks to Bart for this suggestion.
The entire patchset can be found at:
git.kernel.org:/hare/scsi-devel/h/alua-2.v8
As usual, comments and reviews are welcome.
Changes to v7:
- Remove SCSI_ACCESS_STATE_UNKNOWN as suggested by Bart
- Fixup typo in pg->flags as suggested by Bart
- Return -EINVAL when reading from access_state and no device handler is attached.
Changes to v6:
- Fixup mutex lockdep issue noticed by Bart
- Reshuffle locks in alua_check_vpd
Changes to v5:
- Fixup lock imbalance noticed by Bart
- use #define instead of enum for access_state as suggested by Ewan
- Updated patch description as suggested by Ewan
- Changed occurrences to 'SET TARGET PORT GROUP' as suggested by Bart
Changes to v4:
- use kfree_rcu() as suggested by hch
- Use 'IS_ERR' instead of 'PTR_ERR' when checking for validity of a pointer
- Simplify pg assignment as suggested by hch
- Use separate WARN_ON statements a suggested by hch
- Fixes to avoid I/O stall on failover
Changes to v3:
- Use scsi_device flag for blacklisting as suggested by hch
- Add Arrays for synchronous ALUA handling
- Move synchronize_rcu() into release_port_group()
- Add remaining reviewed tags
Changes to v2:
- Use a SCSI blacklist flag instead of a hardware handler parameter for switching to synchronous ALUA handling
- Move scsi_get_device_flags{,_keyed} to scsi_devinfo.h
- Move flush_delayed_work() into release_port_group()
- Rename alua_lookup_pg() into alua_find_get_pg()
- Add __rcu annotations to keep sparse happy
Changes to v1:
- Include reviews from hch
- Switch to hardware handler parameter instead of module option
Hannes Reinecke (23):
scsi_dh_alua: Pass buffer as function argument
scsi_dh_alua: separate out alua_stpg()
scsi_dh_alua: Make stpg synchronous
scsi_dh_alua: call alua_rtpg() if stpg fails
scsi_dh_alua: switch to scsi_execute_req_flags()
scsi_dh_alua: allocate RTPG buffer separately
scsi_dh_alua: Use separate alua_port_group structure
scsi_dh_alua: use unique device id
scsi_dh_alua: simplify alua_initialize()
revert commit a8e5a2d593cb ("[SCSI] scsi_dh_alua: ALUA handler attach
should succeed while TPG is transitioning")
scsi_dh_alua: move optimize_stpg evaluation
scsi_dh_alua: remove 'rel_port' from alua_dh_data structure
scsi_dh_alua: Use workqueue for RTPG
scsi_dh_alua: Allow workqueue to run synchronously
scsi_dh_alua: Add new blacklist flag 'BLIST_SYNC_ALUA'
scsi_dh_alua: Recheck state on unit attention
scsi_dh_alua: update all port states
scsi_dh_alua: Send TEST UNIT READY to poll for transitioning
scsi_dh: add 'rescan' callback
scsi: Add 'access_state' attribute
scsi_dh_alua: use common definitions for ALUA state
scsi_dh_alua: update 'access_state' field
scsi_dh_alua: Update version to 2.0
drivers/scsi/device_handler/scsi_dh_alua.c | 979 ++++++++++++++++++++--------- drivers/scsi/scsi_devinfo.c | 2 + drivers/scsi/scsi_lib.c | 1 + drivers/scsi/scsi_scan.c | 11 +- drivers/scsi/scsi_sysfs.c | 53 ++ include/scsi/scsi_device.h | 2 + include/scsi/scsi_devinfo.h | 1 + include/scsi/scsi_dh.h | 2 + include/scsi/scsi_proto.h | 12 + 9 files changed, 756 insertions(+), 307 deletions(-)
Signature · 1 line
1.8.5.6
PATCH v8 1/23 scsi_dh_alua: Pass buffer as function argument
Pass in the buffer as a function argument for submit_rtpg().
--- drivers/scsi/device_handler/scsi_dh_alua.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +7 -6
@@ -135,12 +135,13 @@ static struct request *get_alua_req(struct scsi_device *sdev,* submit_rtpg - Issue a REPORT TARGET GROUP STATES command * @sdev: sdev the command should be sent to */ -static unsigned submit_rtpg (struct scsi_device *sdev,struct alua_dh_data *h) +static unsigned submit_rtpg (struct scsi_device *sdev,unsigned char *buff, +int bufflen,unsigned char *sense,int flags) {struct request *rq;int err =0 ; - rq =get_alua_req (sdev, h->buff, h->bufflen,READ ); + rq =get_alua_req (sdev, buff, bufflen,READ );if (!rq) { err =DRIVER_BUSY <<24 ; goto done;
@@ -148,14 +149,14 @@ static unsigned submit_rtpg(struct scsi_device *sdev, struct alua_dh_data *h)/* Prepare the command. */ rq->cmd[0 ] =MAINTENANCE_IN ; -if (!(h->flags &ALUA_RTPG_EXT_HDR_UNSUPP )) +if (!(flags &ALUA_RTPG_EXT_HDR_UNSUPP )) rq->cmd[1 ] =MI_REPORT_TARGET_PGS |MI_EXT_HDR_PARAM_FMT ;else rq->cmd[1 ] =MI_REPORT_TARGET_PGS ; -put_unaligned_be32 (h->bufflen, &rq->cmd[6 ]); +put_unaligned_be32 (bufflen, &rq->cmd[6 ]); rq->cmd_len =COMMAND_SIZE (MAINTENANCE_IN ); - rq->sense = h->sense; + rq->sense = sense;memset (rq->sense,0 ,SCSI_SENSE_BUFFERSIZE ); rq->sense_len =0 ;
@@ -446,7 +447,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ expiry =round_jiffies_up (jiffies + h->transition_tmo *HZ ); retry: - retval =submit_rtpg (sdev, h); + retval =submit_rtpg (sdev, h->buff, h->bufflen, h->sense, h->flags);if (retval) {if (!scsi_normalize_sense (h->sense,SCSI_SENSE_BUFFERSIZE , &sense_hdr)) {
Signature · 1 line
1.8.5.6
PATCH v8 2/23 scsi_dh_alua: separate out alua_stpg()
In reply to Hannes ReineckeSeparate out SET TARGET PORT GROUP functionality into a separate function alua_stpg().
--- drivers/scsi/device_handler/scsi_dh_alua.c | 95 +++++++++++++++++++----------- 1 file changed, 61 insertions(+), 34 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +61 -34
@@ -579,6 +579,65 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ }/* +* alua_stpg - Issue a SET TARGET PORT GROUP command +* +* Issue a SET TARGET PORT GROUP command and evaluate the +* response. Returns SCSI_DH_RETRY if the target port group +* state is found to be in 'transitioning'. +* If SCSI_DH_OK is returned the passed-in 'fn' function +* this function will take care of executing 'fn'. +* Otherwise 'fn' should be executed by the caller with the +* returned error code. +*/ +static unsigned alua_stpg (struct scsi_device *sdev,struct alua_dh_data *h, +activate_complete fn,void *data) +{ +int err =SCSI_DH_OK ; +int stpg =0 ; + +if (!(h->tpgs &TPGS_MODE_EXPLICIT )) +/* Only implicit ALUA supported */ + goto out; + +switch (h->state) { +case TPGS_STATE_NONOPTIMIZED : + stpg =1 ; +if ((h->flags &ALUA_OPTIMIZE_STPG ) && + !h->pref && + (h->tpgs &TPGS_MODE_IMPLICIT )) + stpg =0 ; +break ; +case TPGS_STATE_STANDBY : +case TPGS_STATE_UNAVAILABLE : + stpg =1 ; +break ; +case TPGS_STATE_OFFLINE : + err =SCSI_DH_IO ; +break ; +case TPGS_STATE_TRANSITIONING : + err =SCSI_DH_RETRY ; +break ; +default : +break ; + } + +if (stpg) { + h->callback_fn = fn; + h->callback_data = data; + err =submit_stpg (h); +if (err !=SCSI_DH_OK ) + h->callback_fn = h->callback_data =NULL ; +else + fn =NULL ; + } +out: +if (fn) +fn (data, err); + +return err; +} + +/* * alua_initialize - Initialize ALUA state * @sdev: the device to be initialized *
@@ -655,7 +714,6 @@ static int alua_activate(struct scsi_device *sdev, {struct alua_dh_data *h = sdev->handler_data;int err =SCSI_DH_OK ; -int stpg =0 ; err =alua_rtpg (sdev, h,1 );if (err !=SCSI_DH_OK )
@@ -664,41 +722,10 @@ static int alua_activate(struct scsi_device *sdev,if (optimize_stpg) h->flags |=ALUA_OPTIMIZE_STPG ; -if (h->tpgs &TPGS_MODE_EXPLICIT ) { -switch (h->state) { -case TPGS_STATE_NONOPTIMIZED : - stpg =1 ; -if ((h->flags &ALUA_OPTIMIZE_STPG ) && - (!h->pref) && - (h->tpgs &TPGS_MODE_IMPLICIT )) - stpg =0 ; -break ; -case TPGS_STATE_STANDBY : -case TPGS_STATE_UNAVAILABLE : - stpg =1 ; -break ; -case TPGS_STATE_OFFLINE : - err =SCSI_DH_IO ; -break ; -case TPGS_STATE_TRANSITIONING : - err =SCSI_DH_RETRY ; -break ; -default : -break ; - } - } - -if (stpg) { - h->callback_fn = fn; - h->callback_data = data; - err =submit_stpg (h); -if (err ==SCSI_DH_OK ) -return 0 ; - h->callback_fn = h->callback_data =NULL ; - } + err =alua_stpg (sdev, h, fn, data); out: -if (fn) +if (err !=SCSI_DH_OK && fn)fn (data, err);return 0 ; }
Signature · 1 line
1.8.5.6
PATCH v8 3/23 scsi_dh_alua: Make stpg synchronous
In reply to Hannes ReineckeThe 'activate_complete' function needs to be executed after stpg has finished, so we can as well execute stpg synchronously and call the function directly.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 158 ++++++++++------------------- 1 file changed, 54 insertions(+), 104 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +54 -104
@@ -169,81 +169,28 @@ done: }/* -* stpg_endio - Evaluate SET TARGET GROUP STATES -* @sdev: the device to be evaluated -* @state: the new target group state -* -* Evaluate a SET TARGET GROUP STATES command response. -*/ -static void stpg_endio (struct request *req,int error) -{ -struct alua_dh_data *h = req->end_io_data; -struct scsi_sense_hdr sense_hdr; -unsigned err =SCSI_DH_OK ; - -if (host_byte (req->errors) !=DID_OK || -msg_byte (req->errors) !=COMMAND_COMPLETE ) { - err =SCSI_DH_IO ; - goto done; - } - -if (scsi_normalize_sense (h->sense,SCSI_SENSE_BUFFERSIZE , - &sense_hdr)) { -if (sense_hdr.sense_key ==NOT_READY && - sense_hdr.asc ==0x04 && sense_hdr.ascq ==0x0a ) { -/* ALUA state transition already in progress */ - err =SCSI_DH_OK ; - goto done; - } -if (sense_hdr.sense_key ==UNIT_ATTENTION ) { - err =SCSI_DH_RETRY ; - goto done; - } -sdev_printk (KERN_INFO , h->sdev,"%s: stpg failed\n" , -ALUA_DH_NAME ); -scsi_print_sense_hdr (h->sdev,ALUA_DH_NAME , &sense_hdr); - err =SCSI_DH_IO ; - }else if (error) - err =SCSI_DH_IO ; - -if (err ==SCSI_DH_OK ) { - h->state =TPGS_STATE_OPTIMIZED ; -sdev_printk (KERN_INFO , h->sdev, -"%s: port group %02x switched to state %c\n" , -ALUA_DH_NAME , h->group_id, -print_alua_state (h->state)); - } -done: - req->end_io_data =NULL ; -__blk_put_request (req->q, req); -if (h->callback_fn) { - h->callback_fn (h->callback_data, err); - h->callback_fn = h->callback_data =NULL ; - } -return ; -} - -/* -* submit_stpg - Issue a SET TARGET GROUP STATES command +* submit_stpg - Issue a SET TARGET PORT GROUP command * * Currently we're only setting the current target port group state * to 'active/optimized' and let the array firmware figure out * the states of the remaining groups. */ -static unsigned submit_stpg (struct alua_dh_data *h) +static unsigned submit_stpg (struct scsi_device *sdev,int group_id, +unsigned char *sense) {struct request *rq; +unsigned char stpg_data[8 ];int stpg_len =8 ; -struct scsi_device *sdev = h->sdev; +int err =0 ;/* Prepare the data buffer */ -memset (h->buff,0 , stpg_len); - h->buff[4 ] =TPGS_STATE_OPTIMIZED &0x0f ; -put_unaligned_be16 (h->group_id, &h->buff[6 ]); +memset (stpg_data,0 , stpg_len); + stpg_data[4 ] =TPGS_STATE_OPTIMIZED &0x0f ; +put_unaligned_be16 (group_id, &stpg_data[6 ]); - rq =get_alua_req (sdev, h->buff, stpg_len,WRITE ); + rq =get_alua_req (sdev, stpg_data, stpg_len,WRITE );if (!rq) -return SCSI_DH_RES_TEMP_UNAVAIL ; +return DRIVER_BUSY <<24 ;/* Prepare the command. */ rq->cmd[0 ] =MAINTENANCE_OUT ;
@@ -251,13 +198,17 @@ static unsigned submit_stpg(struct alua_dh_data *h)put_unaligned_be32 (stpg_len, &rq->cmd[6 ]); rq->cmd_len =COMMAND_SIZE (MAINTENANCE_OUT ); - rq->sense = h->sense; + rq->sense = sense;memset (rq->sense,0 ,SCSI_SENSE_BUFFERSIZE ); rq->sense_len =0 ; - rq->end_io_data = h; -blk_execute_rq_nowait (rq->q,NULL , rq,1 , stpg_endio); -return SCSI_DH_OK ; +blk_execute_rq (rq->q,NULL , rq,1 ); +if (rq->errors) + err = rq->errors; + +blk_put_request (rq); + +return err; }/*
@@ -582,59 +533,59 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_* alua_stpg - Issue a SET TARGET PORT GROUP command * * Issue a SET TARGET PORT GROUP command and evaluate the -* response. Returns SCSI_DH_RETRY if the target port group -* state is found to be in 'transitioning'. -* If SCSI_DH_OK is returned the passed-in 'fn' function -* this function will take care of executing 'fn'. -* Otherwise 'fn' should be executed by the caller with the -* returned error code. +* response. Returns SCSI_DH_RETRY per default to trigger +* a re-evaluation of the target group state or SCSI_DH_OK +* if no further action needs to be taken. */ -static unsigned alua_stpg (struct scsi_device *sdev,struct alua_dh_data *h, -activate_complete fn,void *data) +static unsigned alua_stpg (struct scsi_device *sdev,struct alua_dh_data *h) { -int err =SCSI_DH_OK ; -int stpg =0 ; - -if (!(h->tpgs &TPGS_MODE_EXPLICIT )) -/* Only implicit ALUA supported */ - goto out; +int retval; +struct scsi_sense_hdr sense_hdr; +if (!(h->tpgs &TPGS_MODE_EXPLICIT )) { +/* Only implicit ALUA supported, retry */ +return SCSI_DH_RETRY ; + }switch (h->state) { +case TPGS_STATE_OPTIMIZED : +return SCSI_DH_OK ;case TPGS_STATE_NONOPTIMIZED : - stpg =1 ;if ((h->flags &ALUA_OPTIMIZE_STPG ) && !h->pref && (h->tpgs &TPGS_MODE_IMPLICIT )) - stpg =0 ; +return SCSI_DH_OK ;break ;case TPGS_STATE_STANDBY :case TPGS_STATE_UNAVAILABLE : - stpg =1 ;break ;case TPGS_STATE_OFFLINE : - err =SCSI_DH_IO ; -break ; +return SCSI_DH_IO ;case TPGS_STATE_TRANSITIONING : - err =SCSI_DH_RETRY ;break ;default : -break ; +sdev_printk (KERN_INFO , sdev, +"%s: stpg failed, unhandled TPGS state %d" , +ALUA_DH_NAME , h->state); +return SCSI_DH_NOSYS ; } + retval =submit_stpg (sdev, h->group_id, h->sense); -if (stpg) { - h->callback_fn = fn; - h->callback_data = data; - err =submit_stpg (h); -if (err !=SCSI_DH_OK ) - h->callback_fn = h->callback_data =NULL ; -else - fn =NULL ; +if (retval) { +if (!scsi_normalize_sense (h->sense,SCSI_SENSE_BUFFERSIZE , + &sense_hdr)) { +sdev_printk (KERN_INFO , sdev, +"%s: stpg failed, result %d" , +ALUA_DH_NAME , retval); +if (driver_byte (retval) ==DRIVER_BUSY ) +return SCSI_DH_DEV_TEMP_BUSY ; + }else { +sdev_printk (KERN_INFO , h->sdev,"%s: stpg failed\n" , +ALUA_DH_NAME ); +scsi_print_sense_hdr (sdev,ALUA_DH_NAME , &sense_hdr); + } } -out: -if (fn) -fn (data, err); - -return err; +/* Retry RTPG */ +return SCSI_DH_RETRY ; }/*
@@ -722,10 +673,9 @@ static int alua_activate(struct scsi_device *sdev,if (optimize_stpg) h->flags |=ALUA_OPTIMIZE_STPG ; - err =alua_stpg (sdev, h, fn, data); - + err =alua_stpg (sdev, h); out: -if (err !=SCSI_DH_OK && fn) +if (fn)fn (data, err);return 0 ; }
Signature · 1 line
1.8.5.6
PATCH v8 4/23 scsi_dh_alua: call alua_rtpg() if stpg fails
In reply to Hannes ReineckeIf the call to SET TARGET PORT GROUPS fails we have no idea what state the array is left in, so we need to issue a call to REPORT TARGET PORT GROUPS in these cases.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/scsi/device_handler/scsi_dh_alua.c +2 -0
@@ -674,6 +674,8 @@ static int alua_activate(struct scsi_device *sdev, h->flags |=ALUA_OPTIMIZE_STPG ; err =alua_stpg (sdev, h); +if (err ==SCSI_DH_RETRY ) + err =alua_rtpg (sdev, h,1 ); out:if (fn)fn (data, err);
Signature · 1 line
1.8.5.6
PATCH v8 5/23 scsi_dh_alua: switch to scsi_execute_req_flags()
In reply to Hannes ReineckeAll commands are issued synchronously, so no need to open-code scsi_execute_req_flags() anymore. And we can get rid of the static sense code structure element. scsi_execute_req_flags() will be setting REQ_QUIET and REQ_PREEMPT, but that is perfectly fine as we're evaluating and logging any errors ourselves and we really need to send the command even if the device is quiesced.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 125 +++++++++-------------------- 1 file changed, 36 insertions(+), 89 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +36 -89
@@ -75,7 +75,6 @@ struct alua_dh_data {unsigned char *buff;int bufflen;unsigned char transition_tmo; -unsigned char sense[SCSI_SENSE_BUFFERSIZE ];struct scsi_device *sdev;activate_complete callback_fn;void *callback_data;
@@ -101,71 +100,30 @@ static int realloc_buffer(struct alua_dh_data *h, unsigned len)return 0 ; } -static struct request *get_alua_req (struct scsi_device *sdev, -void *buffer,unsigned buflen,int rw) -{ -struct request *rq; -struct request_queue *q = sdev->request_queue; - - rq =blk_get_request (q, rw,GFP_NOIO ); - -if (IS_ERR (rq)) { -sdev_printk (KERN_INFO , sdev, -"%s: blk_get_request failed\n" , __func__); -return NULL ; - } -blk_rq_set_block_pc (rq); - -if (buflen &&blk_rq_map_kern (q, rq, buffer, buflen,GFP_NOIO )) { -blk_put_request (rq); -sdev_printk (KERN_INFO , sdev, -"%s: blk_rq_map_kern failed\n" , __func__); -return NULL ; - } - - rq->cmd_flags |=REQ_FAILFAST_DEV |REQ_FAILFAST_TRANSPORT | -REQ_FAILFAST_DRIVER ; - rq->retries =ALUA_FAILOVER_RETRIES ; - rq->timeout =ALUA_FAILOVER_TIMEOUT *HZ ; - -return rq; -} -/* * submit_rtpg - Issue a REPORT TARGET GROUP STATES command * @sdev: sdev the command should be sent to */ -static unsigned submit_rtpg (struct scsi_device *sdev,unsigned char *buff, -int bufflen,unsigned char *sense,int flags) +static int submit_rtpg (struct scsi_device *sdev,unsigned char *buff, +int bufflen,struct scsi_sense_hdr *sshdr,int flags) { -struct request *rq; -int err =0 ; - - rq =get_alua_req (sdev, buff, bufflen,READ ); -if (!rq) { - err =DRIVER_BUSY <<24 ; - goto done; - } +u8 cdb[COMMAND_SIZE (MAINTENANCE_IN )]; +int req_flags =REQ_FAILFAST_DEV |REQ_FAILFAST_TRANSPORT | +REQ_FAILFAST_DRIVER ;/* Prepare the command. */ - rq->cmd[0 ] =MAINTENANCE_IN ; +memset (cdb,0x0 ,COMMAND_SIZE (MAINTENANCE_IN )); + cdb[0 ] =MAINTENANCE_IN ;if (!(flags &ALUA_RTPG_EXT_HDR_UNSUPP )) - rq->cmd[1 ] =MI_REPORT_TARGET_PGS |MI_EXT_HDR_PARAM_FMT ; + cdb[1 ] =MI_REPORT_TARGET_PGS |MI_EXT_HDR_PARAM_FMT ;else - rq->cmd[1 ] =MI_REPORT_TARGET_PGS ; -put_unaligned_be32 (bufflen, &rq->cmd[6 ]); - rq->cmd_len =COMMAND_SIZE (MAINTENANCE_IN ); - - rq->sense = sense; -memset (rq->sense,0 ,SCSI_SENSE_BUFFERSIZE ); - rq->sense_len =0 ; - -blk_execute_rq (rq->q,NULL , rq,1 ); -if (rq->errors) - err = rq->errors; -blk_put_request (rq); -done: -return err; + cdb[1 ] =MI_REPORT_TARGET_PGS ; +put_unaligned_be32 (bufflen, &cdb[6 ]); + +return scsi_execute_req_flags (sdev, cdb,DMA_FROM_DEVICE , + buff, bufflen, sshdr, +ALUA_FAILOVER_TIMEOUT *HZ , +ALUA_FAILOVER_RETRIES ,NULL , req_flags); }/*
@@ -175,40 +133,30 @@ done:* to 'active/optimized' and let the array firmware figure out * the states of the remaining groups. */ -static unsigned submit_stpg (struct scsi_device *sdev,int group_id, -unsigned char *sense) +static int submit_stpg (struct scsi_device *sdev,int group_id, +struct scsi_sense_hdr *sshdr) { -struct request *rq; +u8 cdb[COMMAND_SIZE (MAINTENANCE_OUT )];unsigned char stpg_data[8 ];int stpg_len =8 ; -int err =0 ; +int req_flags =REQ_FAILFAST_DEV |REQ_FAILFAST_TRANSPORT | +REQ_FAILFAST_DRIVER ;/* Prepare the data buffer */ memset (stpg_data,0 , stpg_len); stpg_data[4 ] =TPGS_STATE_OPTIMIZED &0x0f ;put_unaligned_be16 (group_id, &stpg_data[6 ]); - rq =get_alua_req (sdev, stpg_data, stpg_len,WRITE ); -if (!rq) -return DRIVER_BUSY <<24 ; -/* Prepare the command. */ - rq->cmd[0 ] =MAINTENANCE_OUT ; - rq->cmd[1 ] =MO_SET_TARGET_PGS ; -put_unaligned_be32 (stpg_len, &rq->cmd[6 ]); - rq->cmd_len =COMMAND_SIZE (MAINTENANCE_OUT ); - - rq->sense = sense; -memset (rq->sense,0 ,SCSI_SENSE_BUFFERSIZE ); - rq->sense_len =0 ; - -blk_execute_rq (rq->q,NULL , rq,1 ); -if (rq->errors) - err = rq->errors; - -blk_put_request (rq); - -return err; +memset (cdb,0x0 ,COMMAND_SIZE (MAINTENANCE_OUT )); + cdb[0 ] =MAINTENANCE_OUT ; + cdb[1 ] =MO_SET_TARGET_PGS ; +put_unaligned_be32 (stpg_len, &cdb[6 ]); + +return scsi_execute_req_flags (sdev, cdb,DMA_TO_DEVICE , + stpg_data, stpg_len, + sshdr,ALUA_FAILOVER_TIMEOUT *HZ , +ALUA_FAILOVER_RETRIES ,NULL , req_flags); }/*
@@ -398,14 +346,14 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ expiry =round_jiffies_up (jiffies + h->transition_tmo *HZ ); retry: - retval =submit_rtpg (sdev, h->buff, h->bufflen, h->sense, h->flags); + retval =submit_rtpg (sdev, h->buff, h->bufflen, &sense_hdr, h->flags); +if (retval) { -if (!scsi_normalize_sense (h->sense,SCSI_SENSE_BUFFERSIZE , - &sense_hdr)) { +if (!scsi_sense_valid (&sense_hdr)) {sdev_printk (KERN_INFO , sdev,"%s: rtpg failed, result %d\n" ,ALUA_DH_NAME , retval); -if (driver_byte (retval) ==DRIVER_BUSY ) +if (driver_byte (retval) ==DRIVER_ERROR )return SCSI_DH_DEV_TEMP_BUSY ;return SCSI_DH_IO ; }
@@ -568,15 +516,14 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h) ALUA_DH_NAME, h->state);return SCSI_DH_NOSYS ; } - retval =submit_stpg (sdev, h->group_id, h->sense); + retval =submit_stpg (sdev, h->group_id, &sense_hdr);if (retval) { -if (!scsi_normalize_sense (h->sense,SCSI_SENSE_BUFFERSIZE , - &sense_hdr)) { +if (!scsi_sense_valid (&sense_hdr)) {sdev_printk (KERN_INFO , sdev,"%s: stpg failed, result %d" ,ALUA_DH_NAME , retval); -if (driver_byte (retval) ==DRIVER_BUSY ) +if (driver_byte (retval) ==DRIVER_ERROR )return SCSI_DH_DEV_TEMP_BUSY ; }else { sdev_printk(KERN_INFO, h->sdev, "%s: stpg failed\n",
Signature · 1 line
1.8.5.6
PATCH v8 6/23 scsi_dh_alua: allocate RTPG buffer separately
In reply to Hannes ReineckeThe RTPG buffer will only evaluated within alua_rtpg(), so we can allocate it locally there and avoid having to put it into the global structure.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 57 ++++++++++++------------------ 1 file changed, 23 insertions(+), 34 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +23 -34
@@ -56,7 +56,7 @@#define TPGS_MODE_IMPLICIT 0x1#define TPGS_MODE_EXPLICIT 0x2 -#define ALUA_INQUIRY_SIZE 36 +#define ALUA_RTPG_SIZE 128#define ALUA_FAILOVER_TIMEOUT 60#define ALUA_FAILOVER_RETRIES 5
@@ -71,9 +71,6 @@ struct alua_dh_data {int state;int pref;unsigned flags;/* used for optimizing STPG */ -unsigned char inq[ALUA_INQUIRY_SIZE ]; -unsigned char *buff; -int bufflen;unsigned char transition_tmo;struct scsi_device *sdev;activate_complete callback_fn;
@@ -85,21 +82,6 @@ struct alua_dh_data {static char print_alua_state (int ); -static int realloc_buffer (struct alua_dh_data *h,unsigned len) -{ -if (h->buff && h->buff != h->inq) -kfree (h->buff); - - h->buff =kmalloc (len,GFP_NOIO ); -if (!h->buff) { - h->buff = h->inq; - h->bufflen =ALUA_INQUIRY_SIZE ; -return 1 ; - } - h->bufflen = len; -return 0 ; -} -/* * submit_rtpg - Issue a REPORT TARGET GROUP STATES command * @sdev: sdev the command should be sent to
@@ -333,8 +315,8 @@ static int alua_check_sense(struct scsi_device *sdev,static int alua_rtpg (struct scsi_device *sdev,struct alua_dh_data *h,int wait_for_transition) {struct scsi_sense_hdr sense_hdr; -int len, k, off, valid_states =0 ; -unsigned char *ucp; +int len, k, off, valid_states =0 , bufflen =ALUA_RTPG_SIZE ; +unsigned char *ucp, *buff;unsigned err, retval;unsigned long expiry, interval =0 ;unsigned int tpg_desc_tbl_off;
@@ -345,14 +327,19 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_else expiry =round_jiffies_up (jiffies + h->transition_tmo *HZ ); + buff =kzalloc (bufflen,GFP_KERNEL ); +if (!buff) +return SCSI_DH_DEV_TEMP_BUSY ; + retry: - retval =submit_rtpg (sdev, h->buff, h->bufflen, &sense_hdr, h->flags); + retval =submit_rtpg (sdev, buff, bufflen, &sense_hdr, h->flags);if (retval) {if (!scsi_sense_valid (&sense_hdr)) {sdev_printk (KERN_INFO , sdev,"%s: rtpg failed, result %d\n" ,ALUA_DH_NAME , retval); +kfree (buff);if (driver_byte (retval) ==DRIVER_ERROR )return SCSI_DH_DEV_TEMP_BUSY ;return SCSI_DH_IO ;
@@ -390,14 +377,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_sdev_printk (KERN_ERR , sdev,"%s: rtpg failed\n" ,ALUA_DH_NAME );scsi_print_sense_hdr (sdev,ALUA_DH_NAME , &sense_hdr); +kfree (buff);return SCSI_DH_IO ; } - len =get_unaligned_be32 (&h->buff[0 ]) +4 ; + len =get_unaligned_be32 (&buff[0 ]) +4 ; -if (len > h->bufflen) { +if (len > bufflen) {/* Resubmit with the correct length */ -if (realloc_buffer (h, len)) { +kfree (buff); + bufflen = len; + buff =kmalloc (bufflen,GFP_KERNEL ); +if (!buff) {sdev_printk (KERN_WARNING , sdev,"%s: kmalloc buffer failed\n" ,__func__);/* Temporary failure, bypass */
@@ -407,24 +398,25 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ } orig_transition_tmo = h->transition_tmo; -if ((h->buff[4 ] &RTPG_FMT_MASK ) ==RTPG_FMT_EXT_HDR && h->buff[5 ] !=0 ) - h->transition_tmo = h->buff[5 ]; +if ((buff[4 ] &RTPG_FMT_MASK ) ==RTPG_FMT_EXT_HDR && buff[5 ] !=0 ) + h->transition_tmo = buff[5 ];else h->transition_tmo =ALUA_FAILOVER_TIMEOUT ; -if (wait_for_transition && (orig_transition_tmo != h->transition_tmo)) { +if (wait_for_transition && + (orig_transition_tmo != h->transition_tmo)) {sdev_printk (KERN_INFO , sdev,"%s: transition timeout set to %d seconds\n" ,ALUA_DH_NAME , h->transition_tmo); expiry = jiffies + h->transition_tmo *HZ ; } -if ((h->buff[4 ] &RTPG_FMT_MASK ) ==RTPG_FMT_EXT_HDR ) +if ((buff[4 ] &RTPG_FMT_MASK ) ==RTPG_FMT_EXT_HDR ) tpg_desc_tbl_off =8 ;else tpg_desc_tbl_off =4 ; -for (k = tpg_desc_tbl_off, ucp = h->buff + tpg_desc_tbl_off; +for (k = tpg_desc_tbl_off, ucp = buff + tpg_desc_tbl_off; k < len; k += off, ucp += off) {
@@ -474,6 +466,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ err =SCSI_DH_OK ;break ; } +kfree (buff);return err; }
@@ -668,8 +661,6 @@ static int alua_bus_attach(struct scsi_device *sdev) h->state =TPGS_STATE_OPTIMIZED ; h->group_id =-1 ; h->rel_port =-1 ; - h->buff = h->inq; - h->bufflen =ALUA_INQUIRY_SIZE ; h->sdev = sdev; err =alua_initialize (sdev, h);
@@ -691,8 +682,6 @@ static void alua_bus_detach(struct scsi_device *sdev) {struct alua_dh_data *h = sdev->handler_data; -if (h->buff && h->inq != h->buff) -kfree (h->buff); sdev->handler_data =NULL ;kfree (h); }
Signature · 1 line
1.8.5.6
PATCH v8 7/23 scsi_dh_alua: Use separate alua_port_group structure
In reply to Hannes ReineckeThe port group needs to be a separate structure as several LUNs might belong to the same group.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 181 ++++++++++++++++++++--------- include/scsi/scsi_dh.h | 1 + 2 files changed, 129 insertions(+), 53 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +128 -53
@@ -64,14 +64,24 @@#define ALUA_OPTIMIZE_STPG 1#define ALUA_RTPG_EXT_HDR_UNSUPP 2 -struct alua_dh_data { +static LIST_HEAD (port_group_list); +static DEFINE_SPINLOCK (port_group_lock); + +struct alua_port_group { +struct kref kref; +struct list_head node;int group_id; -int rel_port;int tpgs;int state;int pref;unsigned flags;/* used for optimizing STPG */ unsigned char transition_tmo; +}; + +struct alua_dh_data { +struct alua_port_group *pg; +int group_id; +int rel_port;struct scsi_device *sdev;activate_complete callback_fn;void *callback_data;
@@ -82,6 +92,17 @@ struct alua_dh_data {static char print_alua_state (int ); +static void release_port_group (struct kref *kref) +{ +struct alua_port_group *pg; + + pg = container_of(kref, struct alua_port_group, kref); +spin_lock (&port_group_lock); +list_del (&pg->node); +spin_unlock (&port_group_lock); +kfree (pg); +} +/* * submit_rtpg - Issue a REPORT TARGET GROUP STATES command * @sdev: sdev the command should be sent to
@@ -142,6 +163,35 @@ static int submit_stpg(struct scsi_device *sdev, int group_id, }/* +* alua_alloc_pg - Allocate a new port_group structure +* @sdev: scsi device +* @h: alua device_handler data +* @group_id: port group id +* +* Allocate a new port_group structure for a given +* device. +*/ +struct alua_port_group *alua_alloc_pg (struct scsi_device *sdev, +int group_id,int tpgs) +{ +struct alua_port_group *pg; + + pg =kzalloc (sizeof (struct alua_port_group ),GFP_KERNEL ); +if (!pg) +return NULL ; + + pg->group_id = group_id; + pg->tpgs = tpgs; + pg->state =TPGS_STATE_OPTIMIZED ; +kref_init (&pg->kref); +spin_lock (&port_group_lock); +list_add (&pg->node, &port_group_list); +spin_unlock (&port_group_lock); + +return pg; +} + +/* * alua_check_tpgs - Evaluate TPGS setting * @sdev: device to be checked *
@@ -216,7 +266,6 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h) ALUA_DH_NAME);return SCSI_DH_DEV_UNSUPP ; } - h->state =TPGS_STATE_OPTIMIZED ; h->group_id = group_id; sdev_printk(KERN_INFO, sdev,
@@ -312,7 +361,7 @@ static int alua_check_sense(struct scsi_device *sdev,* Returns SCSI_DH_DEV_OFFLINED if the path is * found to be unusable. */ -static int alua_rtpg (struct scsi_device *sdev,struct alua_dh_data *h,int wait_for_transition) +static int alua_rtpg (struct scsi_device *sdev,struct alua_port_group *pg,int wait_for_transition) {struct scsi_sense_hdr sense_hdr;int len, k, off, valid_states =0 , bufflen =ALUA_RTPG_SIZE ;
@@ -322,17 +371,17 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_unsigned int tpg_desc_tbl_off;unsigned char orig_transition_tmo; -if (!h->transition_tmo) +if (!pg->transition_tmo) expiry =round_jiffies_up (jiffies +ALUA_FAILOVER_TIMEOUT *HZ );else - expiry =round_jiffies_up (jiffies + h->transition_tmo *HZ ); + expiry =round_jiffies_up (jiffies + pg->transition_tmo *HZ ); buff =kzalloc (bufflen,GFP_KERNEL );if (!buff)return SCSI_DH_DEV_TEMP_BUSY ; retry: - retval =submit_rtpg (sdev, buff, bufflen, &sense_hdr, h->flags); + retval =submit_rtpg (sdev, buff, bufflen, &sense_hdr, pg->flags);if (retval) {if (!scsi_sense_valid (&sense_hdr)) {
@@ -353,10 +402,10 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_* The retry without rtpg_ext_hdr_req set * handles this. */ -if (!(h->flags &ALUA_RTPG_EXT_HDR_UNSUPP ) && +if (!(pg->flags &ALUA_RTPG_EXT_HDR_UNSUPP ) && sense_hdr.sense_key ==ILLEGAL_REQUEST && sense_hdr.asc ==0x24 && sense_hdr.ascq ==0 ) { - h->flags |=ALUA_RTPG_EXT_HDR_UNSUPP ; + pg->flags |=ALUA_RTPG_EXT_HDR_UNSUPP ; goto retry; }/*
@@ -397,18 +446,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ goto retry; } - orig_transition_tmo = h->transition_tmo; + orig_transition_tmo = pg->transition_tmo;if ((buff[4 ] &RTPG_FMT_MASK ) ==RTPG_FMT_EXT_HDR && buff[5 ] !=0 ) - h->transition_tmo = buff[5 ]; + pg->transition_tmo = buff[5 ];else - h->transition_tmo =ALUA_FAILOVER_TIMEOUT ; + pg->transition_tmo =ALUA_FAILOVER_TIMEOUT ;if (wait_for_transition && - (orig_transition_tmo != h->transition_tmo)) { + (orig_transition_tmo != pg->transition_tmo)) {sdev_printk (KERN_INFO , sdev,"%s: transition timeout set to %d seconds\n" , -ALUA_DH_NAME , h->transition_tmo); - expiry = jiffies + h->transition_tmo *HZ ; +ALUA_DH_NAME , pg->transition_tmo); + expiry = jiffies + pg->transition_tmo *HZ ; }if ((buff[4 ] &RTPG_FMT_MASK ) ==RTPG_FMT_EXT_HDR )
@@ -420,9 +469,9 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ k < len; k += off, ucp += off) { -if (h->group_id ==get_unaligned_be16 (&ucp[2 ])) { - h->state = ucp[0 ] &0x0f ; - h->pref = ucp[0 ] >>7 ; +if (pg->group_id ==get_unaligned_be16 (&ucp[2 ])) { + pg->state = ucp[0 ] &0x0f ; + pg->pref = ucp[0 ] >>7 ; valid_states = ucp[1 ]; } off =8 + (ucp[7 ] *4 );
@@ -430,8 +479,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ sdev_printk(KERN_INFO, sdev, "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n", - ALUA_DH_NAME, h->group_id, print_alua_state(h->state), - h->pref ? "preferred" : "non-preferred", + ALUA_DH_NAME, pg->group_id, print_alua_state(pg->state), + pg->pref ? "preferred" : "non-preferred", valid_states&TPGS_SUPPORT_TRANSITION?'T':'t', valid_states&TPGS_SUPPORT_OFFLINE?'O':'o', valid_states&TPGS_SUPPORT_LBA_DEPENDENT?'L':'l',
@@ -440,7 +489,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ valid_states&TPGS_SUPPORT_NONOPTIMIZED ?'N' :'n' , valid_states&TPGS_SUPPORT_OPTIMIZED?'A':'a'); -switch (h->state) { +switch (pg->state) {case TPGS_STATE_TRANSITIONING :if (wait_for_transition) {if (time_before (jiffies, expiry)) {
@@ -455,7 +504,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_ }/* Transitioning time exceeded, set port to standby */ - h->state =TPGS_STATE_STANDBY ; + pg->state =TPGS_STATE_STANDBY ;break ;case TPGS_STATE_OFFLINE :/* Path unusable */
@@ -478,22 +527,22 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_dh_data *h, int wait_* a re-evaluation of the target group state or SCSI_DH_OK * if no further action needs to be taken. */ -static unsigned alua_stpg (struct scsi_device *sdev,struct alua_dh_data *h) +static unsigned alua_stpg (struct scsi_device *sdev,struct alua_port_group *pg) {int retval;struct scsi_sense_hdr sense_hdr; -if (!(h->tpgs &TPGS_MODE_EXPLICIT )) { +if (!(pg->tpgs &TPGS_MODE_EXPLICIT )) {/* Only implicit ALUA supported, retry */ return SCSI_DH_RETRY ; } -switch (h->state) { +switch (pg->state) {case TPGS_STATE_OPTIMIZED :return SCSI_DH_OK ;case TPGS_STATE_NONOPTIMIZED : -if ((h->flags &ALUA_OPTIMIZE_STPG ) && - !h->pref && - (h->tpgs &TPGS_MODE_IMPLICIT )) +if ((pg->flags &ALUA_OPTIMIZE_STPG ) && + !pg->pref && + (pg->tpgs &TPGS_MODE_IMPLICIT ))return SCSI_DH_OK ;break ;case TPGS_STATE_STANDBY :
@@ -506,10 +555,10 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)default :sdev_printk (KERN_INFO , sdev,"%s: stpg failed, unhandled TPGS state %d" , -ALUA_DH_NAME , h->state); +ALUA_DH_NAME , pg->state);return SCSI_DH_NOSYS ; } - retval =submit_stpg (sdev, h->group_id, &sense_hdr); + retval =submit_stpg (sdev, pg->group_id, &sense_hdr);if (retval) {if (!scsi_sense_valid (&sense_hdr)) {
@@ -519,7 +568,7 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)if (driver_byte (retval) ==DRIVER_ERROR )return SCSI_DH_DEV_TEMP_BUSY ; }else { -sdev_printk (KERN_INFO , h->sdev,"%s: stpg failed\n" , +sdev_printk (KERN_INFO , sdev,"%s: stpg failed\n" ,ALUA_DH_NAME );scsi_print_sense_hdr (sdev,ALUA_DH_NAME , &sense_hdr); }
@@ -537,20 +586,24 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_dh_data *h)*/ static int alua_initialize (struct scsi_device *sdev,struct alua_dh_data *h) { -int err =SCSI_DH_DEV_UNSUPP ; +int err =SCSI_DH_DEV_UNSUPP , tpgs; - h->tpgs =alua_check_tpgs (sdev); -if (h->tpgs ==TPGS_MODE_NONE ) + tpgs =alua_check_tpgs (sdev); +if (tpgs ==TPGS_MODE_NONE ) goto out; err =alua_check_vpd (sdev, h);if (err !=SCSI_DH_OK ) goto out; - err =alua_rtpg (sdev, h,0 ); -if (err !=SCSI_DH_OK ) + h->pg =alua_alloc_pg (sdev, h->group_id, tpgs); +if (!h->pg) { + err =SCSI_DH_NOMEM ; goto out; - + } +kref_get (&h->pg->kref); + err =alua_rtpg (sdev, h->pg,0 ); +kref_put (&h->pg->kref, release_port_group); out:return err; }
@@ -566,6 +619,7 @@ out:static int alua_set_params (struct scsi_device *sdev,const char *params) {struct alua_dh_data *h = sdev->handler_data; +struct alua_port_group *pg =NULL ;unsigned int optimize =0 , argc;const char *p = params;int result =SCSI_DH_OK ;
@@ -578,10 +632,14 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)if ((sscanf (p,"%u" , &optimize) !=1 ) || (optimize >1 ))return -EINVAL ; + pg = h->pg; +if (!pg) +return -ENXIO ; +if (optimize) - h->flags |=ALUA_OPTIMIZE_STPG ; + pg->flags |=ALUA_OPTIMIZE_STPG ;else - h->flags &= ~ALUA_OPTIMIZE_STPG ; + pg->flags &= ~ALUA_OPTIMIZE_STPG ;return result; }
@@ -606,16 +664,23 @@ static int alua_activate(struct scsi_device *sdev,struct alua_dh_data *h = sdev->handler_data;int err =SCSI_DH_OK ; - err =alua_rtpg (sdev, h,1 ); -if (err !=SCSI_DH_OK ) +if (!h->pg) goto out; +kref_get (&h->pg->kref); +if (optimize_stpg) - h->flags |=ALUA_OPTIMIZE_STPG ; + h->pg->flags |=ALUA_OPTIMIZE_STPG ; - err =alua_stpg (sdev, h); + err =alua_rtpg (sdev, h->pg,1 ); +if (err !=SCSI_DH_OK ) { +kref_put (&h->pg->kref, release_port_group); + goto out; + } + err =alua_stpg (sdev, h->pg);if (err ==SCSI_DH_RETRY ) - err =alua_rtpg (sdev, h,1 ); + err =alua_rtpg (sdev, h->pg,1 ); +kref_put (&h->pg->kref, release_port_group); out:if (fn)fn (data, err);
@@ -631,13 +696,19 @@ out:static int alua_prep_fn (struct scsi_device *sdev,struct request *req) {struct alua_dh_data *h = sdev->handler_data; +int state;int ret =BLKPREP_OK ; -if (h->state ==TPGS_STATE_TRANSITIONING ) +if (!h->pg) +return ret; +kref_get (&h->pg->kref); + state = h->pg->state; +kref_put (&h->pg->kref, release_port_group); +if (state ==TPGS_STATE_TRANSITIONING ) ret =BLKPREP_DEFER ; -else if (h->state !=TPGS_STATE_OPTIMIZED && - h->state !=TPGS_STATE_NONOPTIMIZED && - h->state !=TPGS_STATE_LBA_DEPENDENT ) { +else if (state !=TPGS_STATE_OPTIMIZED && + state !=TPGS_STATE_NONOPTIMIZED && + state !=TPGS_STATE_LBA_DEPENDENT ) { ret =BLKPREP_KILL ; req->cmd_flags |=REQ_QUIET ; }
@@ -652,18 +723,18 @@ static int alua_prep_fn(struct scsi_device *sdev, struct request *req)static int alua_bus_attach (struct scsi_device *sdev) {struct alua_dh_data *h; -int err; +int err, ret = -EINVAL ; h =kzalloc (sizeof (*h) ,GFP_KERNEL );if (!h)return -ENOMEM ; - h->tpgs =TPGS_MODE_UNINITIALIZED ; - h->state =TPGS_STATE_OPTIMIZED ; - h->group_id =-1 ; + h->pg =NULL ; h->rel_port =-1 ; h->sdev = sdev; err =alua_initialize (sdev, h); +if (err ==SCSI_DH_NOMEM ) + ret = -ENOMEM ;if (err !=SCSI_DH_OK && err !=SCSI_DH_DEV_OFFLINED ) goto failed;
@@ -671,7 +742,7 @@ static int alua_bus_attach(struct scsi_device *sdev)return 0 ; failed:kfree (h); -return -EINVAL ; +return ret; }/*
@@ -682,6 +753,10 @@ static void alua_bus_detach(struct scsi_device *sdev) {struct alua_dh_data *h = sdev->handler_data; +if (h->pg) { +kref_put (&h->pg->kref, release_port_group); + h->pg =NULL ; + } sdev->handler_data =NULL ;kfree (h); }
include/scsi/scsi_dh.h +1 -0
@@ -52,6 +52,7 @@ enum {SCSI_DH_TIMED_OUT ,SCSI_DH_RES_TEMP_UNAVAIL ,SCSI_DH_DEV_OFFLINED , +SCSI_DH_NOMEM ,SCSI_DH_NOSYS , SCSI_DH_DRIVER_MAX, };
Signature · 1 line
1.8.5.6
PATCH v8 8/23 scsi_dh_alua: use unique device id
In reply to Hannes ReineckeUse scsi_vpd_lun_id() to assign a unique device identification to the alua port group structure.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 55 +++++++++++++++++++++++++++--- 1 file changed, 50 insertions(+), 5 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +50 -5
@@ -70,6 +70,8 @@ static DEFINE_SPINLOCK(port_group_lock);struct alua_port_group {struct kref kref;struct list_head node; +unsigned char device_id_str[256 ]; +int device_id_len;int group_id;int tpgs;int state;
@@ -162,6 +164,26 @@ static int submit_stpg(struct scsi_device *sdev, int group_id, ALUA_FAILOVER_RETRIES, NULL, req_flags); } +struct alua_port_group *alua_find_get_pg (char *id_str,size_t id_size, +int group_id) +{ +struct alua_port_group *pg; + +list_for_each_entry (pg, &port_group_list, node) { +if (pg->group_id != group_id) +continue ; +if (pg->device_id_len != id_size) +continue ; +if (strncmp (pg->device_id_str, id_str, id_size)) +continue ; +if (!kref_get_unless_zero (&pg->kref)) +continue ; +return pg; + } + +return NULL ; +} +/* * alua_alloc_pg - Allocate a new port_group structure * @sdev: scsi device
@@ -174,17 +196,39 @@ static int submit_stpg(struct scsi_device *sdev, int group_id,struct alua_port_group *alua_alloc_pg (struct scsi_device *sdev,int group_id,int tpgs) { -struct alua_port_group *pg; +struct alua_port_group *pg, *tmp_pg; pg =kzalloc (sizeof (struct alua_port_group ),GFP_KERNEL );if (!pg) -return NULL ; +return ERR_PTR (-ENOMEM ); + pg->device_id_len =scsi_vpd_lun_id (sdev, pg->device_id_str, +sizeof (pg->device_id_str)); +if (pg->device_id_len <=0 ) { +/* +* Internal error: TPGS supported but no device +* identifcation found. Disable ALUA support. +*/ +kfree (pg); +sdev_printk (KERN_INFO , sdev, +"%s: No device descriptors found\n" , +ALUA_DH_NAME ); +return ERR_PTR (-ENXIO ); + } pg->group_id = group_id; pg->tpgs = tpgs; pg->state =TPGS_STATE_OPTIMIZED ;kref_init (&pg->kref); +spin_lock (&port_group_lock); + tmp_pg =alua_find_get_pg (pg->device_id_str, pg->device_id_len, + group_id); +if (tmp_pg) { +spin_unlock (&port_group_lock); +kfree (pg); +return tmp_pg; + } +list_add (&pg->node, &port_group_list);spin_unlock (&port_group_lock);
@@ -269,7 +313,7 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h) h->group_id = group_id;sdev_printk (KERN_INFO , sdev, -"%s: port group %02x rel port %02x\n" , +"%s: port group %x rel port %x\n" ,ALUA_DH_NAME , h->group_id, h->rel_port);return 0 ;
@@ -597,8 +641,9 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h) goto out; h->pg =alua_alloc_pg (sdev, h->group_id, tpgs); -if (!h->pg) { - err =SCSI_DH_NOMEM ; +if (IS_ERR (h->pg)) { +if (PTR_ERR (h->pg) == -ENOMEM ) + err =SCSI_DH_NOMEM ; goto out; }kref_get (&h->pg->kref);
Signature · 1 line
1.8.5.6
PATCH v8 9/23 scsi_dh_alua: simplify alua_initialize()
In reply to Hannes ReineckeRework alua_check_vpd() to use scsi_vpd_get_tpg() and move the port group selection into the function, too. With that we can simplify alua_initialize() to just call alua_check_tpgs() and alua_check_vpd();
--- drivers/scsi/device_handler/scsi_dh_alua.c | 38 +++++++++++++----------------- 1 file changed, 17 insertions(+), 21 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +17 -21
@@ -92,6 +92,7 @@ struct alua_dh_data {#define ALUA_POLICY_SWITCH_CURRENT 0#define ALUA_POLICY_SWITCH_ALL 1 +static int alua_rtpg (struct scsi_device *,struct alua_port_group *,int );static char print_alua_state (int );static void release_port_group (struct kref *kref)
@@ -294,7 +295,8 @@ static int alua_check_tpgs(struct scsi_device *sdev)* Extract the relative target port and the target port group * descriptor from the list of identificators. */ -static int alua_check_vpd (struct scsi_device *sdev,struct alua_dh_data *h) +static int alua_check_vpd (struct scsi_device *sdev,struct alua_dh_data *h, +int tpgs) {int rel_port =-1 , group_id;
@@ -310,13 +312,21 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h) ALUA_DH_NAME);return SCSI_DH_DEV_UNSUPP ; } - h->group_id = group_id; + + h->pg =alua_alloc_pg (sdev, group_id, tpgs); +if (IS_ERR (h->pg)) { +if (PTR_ERR (h->pg) == -ENOMEM ) +return SCSI_DH_NOMEM ; +return SCSI_DH_DEV_UNSUPP ; + } + h->rel_port = rel_port;sdev_printk (KERN_INFO , sdev, -"%s: port group %x rel port %x\n" , -ALUA_DH_NAME , h->group_id, h->rel_port); +"%s: device %s port group %x rel port %x\n" , +ALUA_DH_NAME , h->pg->device_id_str, + h->group_id, h->rel_port); -return 0 ; +return alua_rtpg (sdev, h->pg,0 ); }static char print_alua_state (int state)
@@ -633,23 +643,9 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)int err =SCSI_DH_DEV_UNSUPP , tpgs; tpgs =alua_check_tpgs (sdev); -if (tpgs ==TPGS_MODE_NONE ) - goto out; - - err =alua_check_vpd (sdev, h); -if (err !=SCSI_DH_OK ) - goto out; +if (tpgs !=TPGS_MODE_NONE ) + err =alua_check_vpd (sdev, h, tpgs); - h->pg =alua_alloc_pg (sdev, h->group_id, tpgs); -if (IS_ERR (h->pg)) { -if (PTR_ERR (h->pg) == -ENOMEM ) - err =SCSI_DH_NOMEM ; - goto out; - } -kref_get (&h->pg->kref); - err =alua_rtpg (sdev, h->pg,0 ); -kref_put (&h->pg->kref, release_port_group); -out:return err; }/*
Signature · 1 line
1.8.5.6
PATCH v8 10/23 revert commit a8e5a2d593cb ("[SCSI] scsi_dh_alua: ALUA handler attach should succeed while TPG is transitioning")
In reply to Hannes ReineckeThis reverts commit a8e5a2d593cbfccf530c3382c2c328d2edaa7b66
Obsoleted by the next patch.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 31 ++++++++++++------------------ 1 file changed, 12 insertions(+), 19 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +12 -19
@@ -92,7 +92,7 @@ struct alua_dh_data {#define ALUA_POLICY_SWITCH_CURRENT 0#define ALUA_POLICY_SWITCH_ALL 1 -static int alua_rtpg (struct scsi_device *,struct alua_port_group *,int ); +static int alua_rtpg (struct scsi_device *,struct alua_port_group *);static char print_alua_state (int );static void release_port_group (struct kref *kref)
@@ -326,7 +326,7 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h,ALUA_DH_NAME , h->pg->device_id_str, h->group_id, h->rel_port); -return alua_rtpg (sdev, h->pg,0 ); +return alua_rtpg (sdev, h->pg); }static char print_alua_state (int state)
@@ -409,13 +409,12 @@ static int alua_check_sense(struct scsi_device *sdev,/* * alua_rtpg - Evaluate REPORT TARGET GROUP STATES * @sdev: the device to be evaluated. -* @wait_for_transition: if nonzero, wait ALUA_FAILOVER_TIMEOUT seconds for device to exit transitioning state * * Evaluate the Target Port Group State. * Returns SCSI_DH_DEV_OFFLINED if the path is * found to be unusable. */ -static int alua_rtpg (struct scsi_device *sdev,struct alua_port_group *pg,int wait_for_transition) +static int alua_rtpg (struct scsi_device *sdev,struct alua_port_group *pg) {struct scsi_sense_hdr sense_hdr;int len, k, off, valid_states =0 , bufflen =ALUA_RTPG_SIZE ;
@@ -506,8 +505,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int welse pg->transition_tmo =ALUA_FAILOVER_TIMEOUT ; -if (wait_for_transition && - (orig_transition_tmo != pg->transition_tmo)) { +if (orig_transition_tmo != pg->transition_tmo) {sdev_printk (KERN_INFO , sdev,"%s: transition timeout set to %d seconds\n" ,ALUA_DH_NAME , pg->transition_tmo);
@@ -545,19 +543,14 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg, int wswitch (pg->state) {case TPGS_STATE_TRANSITIONING : -if (wait_for_transition) { -if (time_before (jiffies, expiry)) { -/* State transition, retry */ - interval +=2000 ; -msleep (interval); - goto retry; - } - err =SCSI_DH_RETRY ; - }else { - err =SCSI_DH_OK ; +if (time_before (jiffies, expiry)) { +/* State transition, retry */ + interval +=2000 ; +msleep (interval); + goto retry; } -/* Transitioning time exceeded, set port to standby */ + err =SCSI_DH_RETRY ; pg->state =TPGS_STATE_STANDBY ;break ;case TPGS_STATE_OFFLINE :
@@ -713,14 +706,14 @@ static int alua_activate(struct scsi_device *sdev,if (optimize_stpg) h->pg->flags |=ALUA_OPTIMIZE_STPG ; - err =alua_rtpg (sdev, h->pg,1 ); + err =alua_rtpg (sdev, h->pg);if (err !=SCSI_DH_OK ) {kref_put (&h->pg->kref, release_port_group); goto out; } err =alua_stpg (sdev, h->pg);if (err ==SCSI_DH_RETRY ) - err =alua_rtpg (sdev, h->pg,1 ); + err =alua_rtpg (sdev, h->pg);kref_put (&h->pg->kref, release_port_group); out:if (fn)
Signature · 1 line
1.8.5.6
PATCH v8 11/23 scsi_dh_alua: move optimize_stpg evaluation
In reply to Hannes ReineckeWhen the optimize_stpg module option is set we should just set it once during port_group allocation. Doing so allows us to override it later with device specific settings.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +6 -7
@@ -64,6 +64,10 @@#define ALUA_OPTIMIZE_STPG 1#define ALUA_RTPG_EXT_HDR_UNSUPP 2 +static uint optimize_stpg; +module_param (optimize_stpg, uint,S_IRUGO |S_IWUSR ); +MODULE_PARM_DESC (optimize_stpg,"Allow use of a non-optimized path, rather than sending a STPG, when implicit TPGS is supported (0=No,1=Yes). Default is 0." ); +static LIST_HEAD (port_group_list);static DEFINE_SPINLOCK (port_group_lock);
@@ -219,6 +223,8 @@ struct alua_port_group *alua_alloc_pg(struct scsi_device *sdev, pg->group_id = group_id; pg->tpgs = tpgs; pg->state =TPGS_STATE_OPTIMIZED ; +if (optimize_stpg) + pg->flags |=ALUA_OPTIMIZE_STPG ;kref_init (&pg->kref);spin_lock (&port_group_lock);
@@ -678,10 +684,6 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)return result; } -static uint optimize_stpg; -module_param (optimize_stpg, uint,S_IRUGO |S_IWUSR ); -MODULE_PARM_DESC (optimize_stpg,"Allow use of a non-optimized path, rather than sending a STPG, when implicit TPGS is supported (0=No,1=Yes). Default is 0." ); -/* * alua_activate - activate a path * @sdev: device on the path to be activated
@@ -703,9 +705,6 @@ static int alua_activate(struct scsi_device *sdev,kref_get (&h->pg->kref); -if (optimize_stpg) - h->pg->flags |=ALUA_OPTIMIZE_STPG ; - err =alua_rtpg (sdev, h->pg);if (err !=SCSI_DH_OK ) {kref_put (&h->pg->kref, release_port_group);
Signature · 1 line
1.8.5.6
PATCH v8 12/23 scsi_dh_alua: remove 'rel_port' from alua_dh_data structure
In reply to Hannes ReineckeThe 'relative port' field is not used, and might get stale when the port group changes. So remove the field altogether.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +1 -5
@@ -87,7 +87,6 @@ struct alua_port_group {struct alua_dh_data {struct alua_port_group *pg;int group_id; -int rel_port;struct scsi_device *sdev;activate_complete callback_fn;void *callback_data;
@@ -325,12 +324,10 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h,return SCSI_DH_NOMEM ;return SCSI_DH_DEV_UNSUPP ; } - h->rel_port = rel_port; -sdev_printk (KERN_INFO , sdev,"%s: device %s port group %x rel port %x\n" ,ALUA_DH_NAME , h->pg->device_id_str, - h->group_id, h->rel_port); + h->group_id, rel_port);return alua_rtpg (sdev, h->pg); }
@@ -762,7 +759,6 @@ static int alua_bus_attach(struct scsi_device *sdev)if (!h)return -ENOMEM ; h->pg =NULL ; - h->rel_port =-1 ; h->sdev = sdev; err =alua_initialize (sdev, h);
Signature · 1 line
1.8.5.6
PATCH v8 13/23 scsi_dh_alua: Use workqueue for RTPG
In reply to Hannes ReineckeThe current ALUA device_handler has two drawbacks:
- We're sending a 'SET TARGET PORT GROUP' command to every LUN, disregarding the fact that several LUNs might be in a port group and will be automatically switched whenever _any_ LUN within that port group receives the command.
- Whenever a LUN is in 'transitioning' mode we cannot block I/O to that LUN, instead the controller has to abort the command. This leads to increased traffic across the wire and heavy load on the controller during switchover.
With this patch the RTPG handling is moved to a per-portgroup workqueue. This reduces the number of 'REPORT TARGET PORT GROUP' and 'SET TARGET PORT GROUPS' sent to the controller as we're sending them now per port group, and not per device as previously. It also allows us to block I/O to any LUN / port group found to be in 'transitioning' ALUA mode, as the workqueue item will be requeued until the controller moves out of transitioning.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 296 +++++++++++++++++++++++------ 1 file changed, 242 insertions(+), 54 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +242 -54
@@ -59,10 +59,15 @@#define ALUA_RTPG_SIZE 128#define ALUA_FAILOVER_TIMEOUT 60#define ALUA_FAILOVER_RETRIES 5 +#define ALUA_RTPG_DELAY_MSECS 5/* device handler flags */ -#define ALUA_OPTIMIZE_STPG 1 -#define ALUA_RTPG_EXT_HDR_UNSUPP 2 +#define ALUA_OPTIMIZE_STPG 0x01 +#define ALUA_RTPG_EXT_HDR_UNSUPP 0x02 +/* State machine flags */ +#define ALUA_PG_RUN_RTPG 0x10 +#define ALUA_PG_RUN_STPG 0x20 +#define ALUA_PG_RUNNING 0x40static uint optimize_stpg;module_param (optimize_stpg, uint,S_IRUGO |S_IWUSR );
@@ -70,9 +75,11 @@ MODULE_PARM_DESC(optimize_stpg, "Allow use of a non-optimized path, rather thanstatic LIST_HEAD (port_group_list);static DEFINE_SPINLOCK (port_group_lock); +static struct workqueue_struct *kaluad_wq;struct alua_port_group {struct kref kref; +struct rcu_head rcu;struct list_head node;unsigned char device_id_str[256 ];int device_id_len;
@@ -82,12 +89,25 @@ struct alua_port_group {int pref;unsigned flags;/* used for optimizing STPG */ unsigned char transition_tmo; +unsigned long expiry; +unsigned long interval; +struct delayed_work rtpg_work; +spinlock_t lock; +struct list_head rtpg_list; +struct scsi_device *rtpg_sdev; };struct alua_dh_data {struct alua_port_group *pg;int group_id; +spinlock_t pg_lock;struct scsi_device *sdev; +int init_error; +struct mutex init_mutex; +}; + +struct alua_queue_data { +struct list_head entry;activate_complete callback_fn;void *callback_data; };
@@ -95,18 +115,22 @@ struct alua_dh_data {#define ALUA_POLICY_SWITCH_CURRENT 0#define ALUA_POLICY_SWITCH_ALL 1 -static int alua_rtpg (struct scsi_device *,struct alua_port_group *); -static char print_alua_state (int ); +static void alua_rtpg_work (struct work_struct *work); +static void alua_rtpg_queue (struct alua_port_group *pg, +struct scsi_device *sdev, +struct alua_queue_data *qdata);static void release_port_group (struct kref *kref) {struct alua_port_group *pg; pg = container_of(kref, struct alua_port_group, kref); +if (pg->rtpg_sdev) +flush_delayed_work (&pg->rtpg_work);spin_lock (&port_group_lock);list_del (&pg->node);spin_unlock (&port_group_lock); -kfree (pg); +kfree_rcu (pg, rcu); }/*
@@ -225,6 +249,10 @@ struct alua_port_group *alua_alloc_pg(struct scsi_device *sdev,if (optimize_stpg) pg->flags |=ALUA_OPTIMIZE_STPG ;kref_init (&pg->kref); +INIT_DELAYED_WORK (&pg->rtpg_work, alua_rtpg_work); +INIT_LIST_HEAD (&pg->rtpg_list); +INIT_LIST_HEAD (&pg->node); +spin_lock_init (&pg->lock);spin_lock (&port_group_lock); tmp_pg = alua_find_get_pg(pg->device_id_str, pg->device_id_len,
@@ -304,6 +332,7 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h, int tpgs) {int rel_port =-1 , group_id; +struct alua_port_group *pg, *old_pg =NULL ; group_id =scsi_vpd_tpg_id (sdev, &rel_port);if (group_id <0 ) {
@@ -318,18 +347,30 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h,return SCSI_DH_DEV_UNSUPP ; } - h->pg =alua_alloc_pg (sdev, group_id, tpgs); -if (IS_ERR (h->pg)) { -if (PTR_ERR (h->pg) == -ENOMEM ) + pg =alua_alloc_pg (sdev, group_id, tpgs); +if (IS_ERR (pg)) { +if (PTR_ERR (pg) == -ENOMEM )return SCSI_DH_NOMEM ;return SCSI_DH_DEV_UNSUPP ; }sdev_printk (KERN_INFO , sdev,"%s: device %s port group %x rel port %x\n" , -ALUA_DH_NAME , h->pg->device_id_str, - h->group_id, rel_port); +ALUA_DH_NAME , pg->device_id_str, group_id, rel_port); + +/* Check for existing port group references */ +spin_lock (&h->pg_lock); + old_pg = h->pg; +if (old_pg != pg) { +/* port group has changed. Update to new port group */ +rcu_assign_pointer (h->pg, pg); + } +alua_rtpg_queue (h->pg, sdev,NULL ); +spin_unlock (&h->pg_lock); + +if (old_pg) +kref_put (&old_pg->kref, release_port_group); -return alua_rtpg (sdev, h->pg); +return SCSI_DH_OK ; }static char print_alua_state (int state)
@@ -423,14 +464,17 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)int len, k, off, valid_states =0 , bufflen =ALUA_RTPG_SIZE ;unsigned char *ucp, *buff;unsigned err, retval; -unsigned long expiry, interval =0 ;unsigned int tpg_desc_tbl_off;unsigned char orig_transition_tmo; -if (!pg->transition_tmo) - expiry =round_jiffies_up (jiffies +ALUA_FAILOVER_TIMEOUT *HZ ); -else - expiry =round_jiffies_up (jiffies + pg->transition_tmo *HZ ); +if (!pg->expiry) { +unsigned long transition_tmo =ALUA_FAILOVER_TIMEOUT *HZ ; + +if (pg->transition_tmo) + transition_tmo = pg->transition_tmo *HZ ; + + pg->expiry =round_jiffies_up (jiffies + transition_tmo); + } buff =kzalloc (bufflen,GFP_KERNEL );if (!buff)
@@ -473,16 +517,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg) err =SCSI_DH_RETRY ; else if (sense_hdr.sense_key == UNIT_ATTENTION) err =SCSI_DH_RETRY ; -if (err ==SCSI_DH_RETRY &&time_before (jiffies, expiry)) { +if (err ==SCSI_DH_RETRY && + pg->expiry !=0 &&time_before (jiffies, pg->expiry)) {sdev_printk (KERN_ERR , sdev,"%s: rtpg retry\n" ,ALUA_DH_NAME );scsi_print_sense_hdr (sdev,ALUA_DH_NAME , &sense_hdr); - goto retry; +return err; }sdev_printk (KERN_ERR , sdev,"%s: rtpg failed\n" ,ALUA_DH_NAME );scsi_print_sense_hdr (sdev,ALUA_DH_NAME , &sense_hdr);kfree (buff); + pg->expiry =0 ;return SCSI_DH_IO ; }
@@ -497,6 +543,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)sdev_printk (KERN_WARNING , sdev,"%s: kmalloc buffer failed\n" ,__func__);/* Temporary failure, bypass */ + pg->expiry =0 ;return SCSI_DH_DEV_TEMP_BUSY ; } goto retry;
@@ -512,7 +559,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)sdev_printk (KERN_INFO , sdev,"%s: transition timeout set to %d seconds\n" ,ALUA_DH_NAME , pg->transition_tmo); - expiry = jiffies + pg->transition_tmo *HZ ; + pg->expiry = jiffies + pg->transition_tmo *HZ ; }if ((buff[4 ] &RTPG_FMT_MASK ) ==RTPG_FMT_EXT_HDR )
@@ -546,23 +593,26 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)switch (pg->state) {case TPGS_STATE_TRANSITIONING : -if (time_before (jiffies, expiry)) { +if (time_before (jiffies, pg->expiry)) {/* State transition, retry */ - interval +=2000 ; -msleep (interval); - goto retry; + pg->interval =2 ; + err =SCSI_DH_RETRY ; + }else { +/* Transitioning time exceeded, set port to standby */ + err =SCSI_DH_IO ; + pg->state =TPGS_STATE_STANDBY ; + pg->expiry =0 ; } -/* Transitioning time exceeded, set port to standby */ - err =SCSI_DH_RETRY ; - pg->state =TPGS_STATE_STANDBY ;break ;case TPGS_STATE_OFFLINE :/* Path unusable */ err =SCSI_DH_DEV_OFFLINED ; + pg->expiry =0 ;break ;default :/* Useable path if active */ err =SCSI_DH_OK ; + pg->expiry =0 ;break ; }kfree (buff);
@@ -627,6 +677,107 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)return SCSI_DH_RETRY ; } +static void alua_rtpg_work (struct work_struct *work) +{ +struct alua_port_group *pg = + container_of(work, struct alua_port_group, rtpg_work.work); +struct scsi_device *sdev; +LIST_HEAD (qdata_list); +int err =SCSI_DH_OK ; +struct alua_queue_data *qdata, *tmp; +unsigned long flags; + +spin_lock_irqsave (&pg->lock, flags); + sdev = pg->rtpg_sdev; +if (!sdev) { +WARN_ON (pg->flags &ALUA_PG_RUN_RTPG ); +WARN_ON (pg->flags &ALUA_PG_RUN_STPG ); +spin_unlock_irqrestore (&pg->lock, flags); +return ; + } + pg->flags |=ALUA_PG_RUNNING ; +if (pg->flags &ALUA_PG_RUN_RTPG ) { + pg->flags &= ~ALUA_PG_RUN_RTPG ; +spin_unlock_irqrestore (&pg->lock, flags); + err =alua_rtpg (sdev, pg); +spin_lock_irqsave (&pg->lock, flags); +if (err ==SCSI_DH_RETRY ) { + pg->flags &= ~ALUA_PG_RUNNING ; + pg->flags |=ALUA_PG_RUN_RTPG ; +spin_unlock_irqrestore (&pg->lock, flags); +queue_delayed_work (kaluad_wq, &pg->rtpg_work, + pg->interval *HZ ); +return ; + } +if (err !=SCSI_DH_OK ) + pg->flags &= ~ALUA_PG_RUN_STPG ; + } +if (pg->flags &ALUA_PG_RUN_STPG ) { + pg->flags &= ~ALUA_PG_RUN_STPG ; +spin_unlock_irqrestore (&pg->lock, flags); + err =alua_stpg (sdev, pg); +spin_lock_irqsave (&pg->lock, flags); +if (err ==SCSI_DH_RETRY ) { + pg->flags |=ALUA_PG_RUN_RTPG ; + pg->interval =0 ; + pg->flags &= ~ALUA_PG_RUNNING ; +spin_unlock_irqrestore (&pg->lock, flags); +queue_delayed_work (kaluad_wq, &pg->rtpg_work, + pg->interval *HZ ); +return ; + } + } + +list_splice_init (&pg->rtpg_list, &qdata_list); + pg->rtpg_sdev =NULL ; +spin_unlock_irqrestore (&pg->lock, flags); + +list_for_each_entry_safe (qdata, tmp, &qdata_list, entry) { +list_del (&qdata->entry); +if (qdata->callback_fn) + qdata->callback_fn (qdata->callback_data, err); +kfree (qdata); + } +spin_lock_irqsave (&pg->lock, flags); + pg->flags &= ~ALUA_PG_RUNNING ; +spin_unlock_irqrestore (&pg->lock, flags); +scsi_device_put (sdev); +kref_put (&pg->kref, release_port_group); +} + +static void alua_rtpg_queue (struct alua_port_group *pg, +struct scsi_device *sdev, +struct alua_queue_data *qdata) +{ +int start_queue =0 ; +unsigned long flags; + +if (!pg) +return ; + +spin_lock_irqsave (&pg->lock, flags); +if (qdata) { +list_add_tail (&qdata->entry, &pg->rtpg_list); + pg->flags |=ALUA_PG_RUN_STPG ; + } +if (pg->rtpg_sdev ==NULL ) { + pg->interval =0 ; + pg->flags |=ALUA_PG_RUN_RTPG ; +kref_get (&pg->kref); + pg->rtpg_sdev = sdev; +scsi_device_get (sdev); + start_queue =1 ; + } +spin_unlock_irqrestore (&pg->lock, flags); + +if (start_queue && + !queue_delayed_work (kaluad_wq, &pg->rtpg_work, +msecs_to_jiffies (ALUA_RTPG_DELAY_MSECS ))) { +scsi_device_put (sdev); +kref_put (&pg->kref, release_port_group); + } +} +/* * alua_initialize - Initialize ALUA state * @sdev: the device to be initialized
@@ -638,10 +789,12 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h) {int err =SCSI_DH_DEV_UNSUPP , tpgs; +mutex_lock (&h->init_mutex); tpgs =alua_check_tpgs (sdev);if (tpgs !=TPGS_MODE_NONE ) err =alua_check_vpd (sdev, h, tpgs); - + h->init_error = err; +mutex_unlock (&h->init_mutex);return err; }/*
@@ -656,10 +809,11 @@ static int alua_initialize(struct scsi_device *sdev, struct alua_dh_data *h)static int alua_set_params (struct scsi_device *sdev,const char *params) {struct alua_dh_data *h = sdev->handler_data; -struct alua_port_group *pg =NULL ; +struct alua_port_group __rcu *pg =NULL ;unsigned int optimize =0 , argc;const char *p = params;int result =SCSI_DH_OK ; +unsigned long flags;if ((sscanf (params,"%u" , &argc) !=1 ) || (argc !=1 ))return -EINVAL ;
@@ -669,14 +823,19 @@ static int alua_set_params(struct scsi_device *sdev, const char *params)if ((sscanf (p,"%u" , &optimize) !=1 ) || (optimize >1 ))return -EINVAL ; - pg = h->pg; -if (!pg) +rcu_read_lock (); + pg =rcu_dereference (h->pg); +if (!pg) { +rcu_read_unlock ();return -ENXIO ; - + } +spin_lock_irqsave (&pg->lock, flags);if (optimize) pg->flags |=ALUA_OPTIMIZE_STPG ;else pg->flags &= ~ALUA_OPTIMIZE_STPG ; +spin_unlock_irqrestore (&pg->lock, flags); +rcu_read_unlock ();return result; }
@@ -696,21 +855,33 @@ static int alua_activate(struct scsi_device *sdev, {struct alua_dh_data *h = sdev->handler_data;int err =SCSI_DH_OK ; +struct alua_queue_data *qdata; +struct alua_port_group __rcu *pg; -if (!h->pg) + qdata =kzalloc (sizeof (*qdata),GFP_KERNEL ); +if (!qdata) { + err =SCSI_DH_RES_TEMP_UNAVAIL ; goto out; - -kref_get (&h->pg->kref); - - err =alua_rtpg (sdev, h->pg); -if (err !=SCSI_DH_OK ) { -kref_put (&h->pg->kref, release_port_group); + } + qdata->callback_fn = fn; + qdata->callback_data = data; + +mutex_lock (&h->init_mutex); +rcu_read_lock (); + pg =rcu_dereference (h->pg); +if (!pg || !kref_get_unless_zero (&pg->kref)) { +rcu_read_unlock (); +kfree (qdata); + err = h->init_error; +mutex_unlock (&h->init_mutex); goto out; } - err =alua_stpg (sdev, h->pg); -if (err ==SCSI_DH_RETRY ) - err =alua_rtpg (sdev, h->pg); -kref_put (&h->pg->kref, release_port_group); + fn =NULL ; +rcu_read_unlock (); +mutex_unlock (&h->init_mutex); + +alua_rtpg_queue (pg, sdev, qdata); +kref_put (&pg->kref, release_port_group); out:if (fn)fn (data, err);
@@ -726,14 +897,15 @@ out:static int alua_prep_fn (struct scsi_device *sdev,struct request *req) {struct alua_dh_data *h = sdev->handler_data; -int state; +struct alua_port_group __rcu *pg; +int state =TPGS_STATE_OPTIMIZED ;int ret =BLKPREP_OK ; -if (!h->pg) -return ret; -kref_get (&h->pg->kref); - state = h->pg->state; -kref_put (&h->pg->kref, release_port_group); +rcu_read_lock (); + pg =rcu_dereference (h->pg); +if (pg) + state = pg->state; +rcu_read_unlock ();if (state ==TPGS_STATE_TRANSITIONING ) ret =BLKPREP_DEFER ; else if (state != TPGS_STATE_OPTIMIZED &&
@@ -758,9 +930,12 @@ static int alua_bus_attach(struct scsi_device *sdev) h =kzalloc (sizeof (*h) ,GFP_KERNEL );if (!h)return -ENOMEM ; - h->pg =NULL ; +spin_lock_init (&h->pg_lock); +rcu_assign_pointer (h->pg,NULL ); + h->init_error =SCSI_DH_OK ; h->sdev = sdev; +mutex_init (&h->init_mutex); err =alua_initialize (sdev, h);if (err ==SCSI_DH_NOMEM ) ret = -ENOMEM ;
@@ -781,11 +956,16 @@ failed:static void alua_bus_detach (struct scsi_device *sdev) {struct alua_dh_data *h = sdev->handler_data; +struct alua_port_group *pg; + +spin_lock (&h->pg_lock); + pg = h->pg; +rcu_assign_pointer (h->pg,NULL ); + h->sdev =NULL ; +spin_unlock (&h->pg_lock); +if (pg) +kref_put (&pg->kref, release_port_group); -if (h->pg) { -kref_put (&h->pg->kref, release_port_group); - h->pg =NULL ; - } sdev->handler_data =NULL ;kfree (h); }
@@ -805,16 +985,24 @@ static int __init alua_init(void) {int r; + kaluad_wq =alloc_workqueue ("kaluad" ,WQ_MEM_RECLAIM ,0 ); +if (!kaluad_wq) { +/* Temporary failure, bypass */ +return SCSI_DH_DEV_TEMP_BUSY ; + } r =scsi_register_device_handler (&alua_dh); -if (r !=0 ) +if (r !=0 ) {printk (KERN_ERR "%s: Failed to register scsi device handler" ,ALUA_DH_NAME ); +destroy_workqueue (kaluad_wq); + }return r; } static void __exit alua_exit(void) {scsi_unregister_device_handler (&alua_dh); +destroy_workqueue (kaluad_wq); }module_init (alua_init);
Signature · 1 line
1.8.5.6
PATCH v8 14/23 scsi_dh_alua: Allow workqueue to run synchronously
In reply to Hannes ReineckeSome arrays may only capable of handling one STPG at a time, so this patch adds a singlethreaded workqueue for STPGs to be submitted synchronously.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +18 -3
@@ -64,6 +64,7 @@/* device handler flags */ #define ALUA_OPTIMIZE_STPG 0x01#define ALUA_RTPG_EXT_HDR_UNSUPP 0x02 +#define ALUA_SYNC_STPG 0x04/* State machine flags */ #define ALUA_PG_RUN_RTPG 0x10#define ALUA_PG_RUN_STPG 0x20
@@ -76,6 +77,7 @@ MODULE_PARM_DESC(optimize_stpg, "Allow use of a non-optimized path, rather thanstatic LIST_HEAD (port_group_list);static DEFINE_SPINLOCK (port_group_lock);static struct workqueue_struct *kaluad_wq; +static struct workqueue_struct *kaluad_sync_wq;struct alua_port_group {struct kref kref;
@@ -686,6 +688,7 @@ static void alua_rtpg_work(struct work_struct *work)int err =SCSI_DH_OK ;struct alua_queue_data *qdata, *tmp;unsigned long flags; +struct workqueue_struct *alua_wq = kaluad_wq;spin_lock_irqsave (&pg->lock, flags); sdev = pg->rtpg_sdev;
@@ -695,6 +698,8 @@ static void alua_rtpg_work(struct work_struct *work)spin_unlock_irqrestore (&pg->lock, flags);return ; } +if (pg->flags &ALUA_SYNC_STPG ) + alua_wq = kaluad_sync_wq; pg->flags |=ALUA_PG_RUNNING ;if (pg->flags &ALUA_PG_RUN_RTPG ) { pg->flags &= ~ALUA_PG_RUN_RTPG ;
@@ -705,7 +710,7 @@ static void alua_rtpg_work(struct work_struct *work) pg->flags &= ~ALUA_PG_RUNNING ; pg->flags |=ALUA_PG_RUN_RTPG ;spin_unlock_irqrestore (&pg->lock, flags); -queue_delayed_work (kaluad_wq, &pg->rtpg_work, +queue_delayed_work (alua_wq, &pg->rtpg_work, pg->interval *HZ );return ; }
@@ -722,7 +727,7 @@ static void alua_rtpg_work(struct work_struct *work) pg->interval =0 ; pg->flags &= ~ALUA_PG_RUNNING ;spin_unlock_irqrestore (&pg->lock, flags); -queue_delayed_work (kaluad_wq, &pg->rtpg_work, +queue_delayed_work (alua_wq, &pg->rtpg_work, pg->interval *HZ );return ; }
@@ -751,6 +756,7 @@ static void alua_rtpg_queue(struct alua_port_group *pg, {int start_queue =0 ;unsigned long flags; +struct workqueue_struct *alua_wq = kaluad_wq;if (!pg)return ;
@@ -768,10 +774,12 @@ static void alua_rtpg_queue(struct alua_port_group *pg,scsi_device_get (sdev); start_queue =1 ; } +if (pg->flags &ALUA_SYNC_STPG ) + alua_wq = kaluad_sync_wq;spin_unlock_irqrestore (&pg->lock, flags);if (start_queue && - !queue_delayed_work (kaluad_wq, &pg->rtpg_work, + !queue_delayed_work (alua_wq, &pg->rtpg_work,msecs_to_jiffies (ALUA_RTPG_DELAY_MSECS ))) {scsi_device_put (sdev);kref_put (&pg->kref, release_port_group);
@@ -990,10 +998,16 @@ static int __init alua_init(void)/* Temporary failure, bypass */ return SCSI_DH_DEV_TEMP_BUSY ; } + kaluad_sync_wq =create_workqueue ("kaluad_sync" ); +if (!kaluad_sync_wq) { +destroy_workqueue (kaluad_wq); +return SCSI_DH_DEV_TEMP_BUSY ; + } r =scsi_register_device_handler (&alua_dh);if (r !=0 ) {printk (KERN_ERR "%s: Failed to register scsi device handler" ,ALUA_DH_NAME ); +destroy_workqueue (kaluad_sync_wq);destroy_workqueue (kaluad_wq); }return r;
@@ -1002,6 +1016,7 @@ static int __init alua_init(void) static void __exit alua_exit(void) {scsi_unregister_device_handler (&alua_dh); +destroy_workqueue (kaluad_sync_wq);destroy_workqueue (kaluad_wq); }
Signature · 1 line
1.8.5.6
PATCH v8 15/23 scsi_dh_alua: Add new blacklist flag 'BLIST_SYNC_ALUA'
In reply to Hannes ReineckeAdd a new blacklist flag BLIST_SYNC_ALUA to instruct the alua device handler to use synchronous command submission for ALUA commands.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 2 ++ drivers/scsi/scsi_devinfo.c | 2 ++ drivers/scsi/scsi_scan.c | 3 +++ include/scsi/scsi_device.h | 1 + include/scsi/scsi_devinfo.h | 1 + 5 files changed, 9 insertions(+)
drivers/scsi/device_handler/scsi_dh_alua.c +2 -0
@@ -366,6 +366,8 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h,/* port group has changed. Update to new port group */ rcu_assign_pointer (h->pg, pg); } +if (sdev->synchronous_alua) + pg->flags |=ALUA_SYNC_STPG ;alua_rtpg_queue (h->pg, sdev,NULL );spin_unlock (&h->pg_lock);
drivers/scsi/scsi_devinfo.c +2 -0
@@ -218,6 +218,8 @@ static struct { {"NAKAMICH", "MJ-5.16S", NULL, BLIST_FORCELUN | BLIST_SINGLELUN}, {"NEC", "PD-1 ODX654P", NULL, BLIST_FORCELUN | BLIST_SINGLELUN}, {"NEC", "iStorage", NULL, BLIST_REPORTLUN2}, + {"NETAPP", "LUN C-Mode", NULL, BLIST_SYNC_ALUA}, + {"NETAPP", "INF-01-00", NULL, BLIST_SYNC_ALUA}, {"NRC", "MBR-7", NULL, BLIST_FORCELUN | BLIST_SINGLELUN}, {"NRC", "MBR-7.4", NULL, BLIST_FORCELUN | BLIST_SINGLELUN}, {"PIONEER", "CD-ROM DRM-600", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
drivers/scsi/scsi_scan.c +3 -0
@@ -964,6 +964,9 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,if (*bflags &BLIST_NO_DIF ) sdev->no_dif =1 ; +if (*bflags &BLIST_SYNC_ALUA ) + sdev->synchronous_alua =1 ; + sdev->eh_timeout =SCSI_DEFAULT_EH_TIMEOUT ;if (*bflags &BLIST_TRY_VPD_PAGES )
include/scsi/scsi_device.h +1 -0
@@ -176,6 +176,7 @@ struct scsi_device { unsigned no_dif:1; /* T10 PI (DIF) should be disabled */ unsigned broken_fua:1; /* Don't set FUA bit */ unsigned lun_in_cdb:1; /* Store LUN bits in CDB[1] */ + unsigned synchronous_alua:1; /* Synchronous ALUA commands */atomic_t disk_events_disable_depth;/* disable depth for disk events */
include/scsi/scsi_devinfo.h +1 -0
@@ -37,5 +37,6 @@#define BLIST_TRY_VPD_PAGES 0x10000000/* Attempt to read VPD pages */ #define BLIST_NO_RSOC 0x20000000/* don't try to issue RSOC */ #define BLIST_MAX_1024 0x40000000/* maximum 1024 sector cdb length */ +#define BLIST_SYNC_ALUA 0x80000000/* Synchronous ALUA commands */ #endif
Signature · 1 line
1.8.5.6
PATCH v8 16/23 scsi_dh_alua: Recheck state on unit attention
In reply to Hannes ReineckeWhen we receive a unit attention code of 'ALUA state changed' we should recheck the state, as it might be due to an implicit ALUA state transition. This allows us to return NEEDS_RETRY instead of ADD_TO_MLQUEUE, allowing to terminate the retries after a certain time.
At the same time a workqueue item might already be queued, which should be started immediately to avoid any delays.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 67 ++++++++++++++++++++++++------ 1 file changed, 55 insertions(+), 12 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +55 -12
@@ -120,7 +120,8 @@ struct alua_queue_data {static void alua_rtpg_work (struct work_struct *work);static void alua_rtpg_queue (struct alua_port_group *pg,struct scsi_device *sdev, -struct alua_queue_data *qdata); +struct alua_queue_data *qdata,bool force); +static void alua_check (struct scsi_device *sdev,bool force);static void release_port_group (struct kref *kref) {
@@ -368,7 +369,7 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h, }if (sdev->synchronous_alua) pg->flags |=ALUA_SYNC_STPG ; -alua_rtpg_queue (h->pg, sdev,NULL ); +alua_rtpg_queue (h->pg, sdev,NULL , true);spin_unlock (&h->pg_lock);if (old_pg)
@@ -404,18 +405,24 @@ static int alua_check_sense(struct scsi_device *sdev, {switch (sense_hdr->sense_key) {case NOT_READY : -if (sense_hdr->asc ==0x04 && sense_hdr->ascq ==0x0a ) +if (sense_hdr->asc ==0x04 && sense_hdr->ascq ==0x0a ) {/* * LUN Not Accessible - ALUA state transition */ -return ADD_TO_MLQUEUE ; +alua_check (sdev, false); +return NEEDS_RETRY ; + }break ;case UNIT_ATTENTION : -if (sense_hdr->asc ==0x29 && sense_hdr->ascq ==0x00 ) +if (sense_hdr->asc ==0x29 && sense_hdr->ascq ==0x00 ) {/* -* Power On, Reset, or Bus Device Reset, just retry. +* Power On, Reset, or Bus Device Reset. +* Might have obscured a state transition, +* so schedule a recheck. */ +alua_check (sdev, true);return ADD_TO_MLQUEUE ; + }if (sense_hdr->asc ==0x29 && sense_hdr->ascq ==0x04 )/* * Device internal reset
@@ -426,16 +433,20 @@ static int alua_check_sense(struct scsi_device *sdev,* Mode Parameters Changed */ return ADD_TO_MLQUEUE ; -if (sense_hdr->asc ==0x2a && sense_hdr->ascq ==0x06 ) +if (sense_hdr->asc ==0x2a && sense_hdr->ascq ==0x06 ) {/* * ALUA state changed */ +alua_check (sdev, true);return ADD_TO_MLQUEUE ; -if (sense_hdr->asc ==0x2a && sense_hdr->ascq ==0x07 ) + } +if (sense_hdr->asc ==0x2a && sense_hdr->ascq ==0x07 ) {/* * Implicit ALUA state transition failed */ +alua_check (sdev, true);return ADD_TO_MLQUEUE ; + }if (sense_hdr->asc ==0x3f && sense_hdr->ascq ==0x03 )/* * Inquiry data has changed
@@ -708,7 +719,7 @@ static void alua_rtpg_work(struct work_struct *work)spin_unlock_irqrestore (&pg->lock, flags); err =alua_rtpg (sdev, pg);spin_lock_irqsave (&pg->lock, flags); -if (err ==SCSI_DH_RETRY ) { +if (err ==SCSI_DH_RETRY || pg->flags &ALUA_PG_RUN_RTPG ) { pg->flags &= ~ALUA_PG_RUNNING ; pg->flags |=ALUA_PG_RUN_RTPG ;spin_unlock_irqrestore (&pg->lock, flags);
@@ -724,7 +735,7 @@ static void alua_rtpg_work(struct work_struct *work)spin_unlock_irqrestore (&pg->lock, flags); err =alua_stpg (sdev, pg);spin_lock_irqsave (&pg->lock, flags); -if (err ==SCSI_DH_RETRY ) { +if (err ==SCSI_DH_RETRY || pg->flags &ALUA_PG_RUN_RTPG ) { pg->flags |=ALUA_PG_RUN_RTPG ; pg->interval =0 ; pg->flags &= ~ALUA_PG_RUNNING ;
@@ -754,7 +765,7 @@ static void alua_rtpg_work(struct work_struct *work)static void alua_rtpg_queue (struct alua_port_group *pg,struct scsi_device *sdev, -struct alua_queue_data *qdata) +struct alua_queue_data *qdata,bool force) {int start_queue =0 ;unsigned long flags;
@@ -767,6 +778,7 @@ static void alua_rtpg_queue(struct alua_port_group *pg,if (qdata) {list_add_tail (&qdata->entry, &pg->rtpg_list); pg->flags |=ALUA_PG_RUN_STPG ; + force = true; }if (pg->rtpg_sdev ==NULL ) { pg->interval =0 ;
@@ -775,7 +787,15 @@ static void alua_rtpg_queue(struct alua_port_group *pg, pg->rtpg_sdev = sdev;scsi_device_get (sdev); start_queue =1 ; + }else if (!(pg->flags &ALUA_PG_RUN_RTPG ) && force) { + pg->flags |=ALUA_PG_RUN_RTPG ; +/* Do not queue if the worker is already running */ +if (!(pg->flags &ALUA_PG_RUNNING )) { +kref_get (&pg->kref); + start_queue =1 ; + } } +if (pg->flags &ALUA_SYNC_STPG ) alua_wq = kaluad_sync_wq;spin_unlock_irqrestore (&pg->lock, flags);
@@ -890,7 +910,7 @@ static int alua_activate(struct scsi_device *sdev,rcu_read_unlock ();mutex_unlock (&h->init_mutex); -alua_rtpg_queue (pg, sdev, qdata); +alua_rtpg_queue (pg, sdev, qdata, true);kref_put (&pg->kref, release_port_group); out:if (fn)
@@ -899,6 +919,29 @@ out: }/* +* alua_check - check path status +* @sdev: device on the path to be checked +* +* Check the device status +*/ +static void alua_check (struct scsi_device *sdev,bool force) +{ +struct alua_dh_data *h = sdev->handler_data; +struct alua_port_group *pg; + +rcu_read_lock (); + pg =rcu_dereference (h->pg); +if (!pg || !kref_get_unless_zero (&pg->kref)) { +rcu_read_unlock (); +return ; + } +rcu_read_unlock (); + +alua_rtpg_queue (pg, sdev,NULL , force); +kref_put (&pg->kref, release_port_group); +} + +/* * alua_prep_fn - request callback * * Fail I/O to all paths not in state
Signature · 1 line
1.8.5.6
PATCH v8 17/23 scsi_dh_alua: update all port states
In reply to Hannes ReineckeWhen we read in the target port group state we should be updating all affected port groups, otherwise we risk running out of sync.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 35 ++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 9 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +26 -9
@@ -476,11 +476,13 @@ static int alua_check_sense(struct scsi_device *sdev,static int alua_rtpg (struct scsi_device *sdev,struct alua_port_group *pg) {struct scsi_sense_hdr sense_hdr; +struct alua_port_group *tmp_pg;int len, k, off, valid_states =0 , bufflen =ALUA_RTPG_SIZE ; -unsigned char *ucp, *buff; +unsigned char *desc, *buff;unsigned err, retval;unsigned int tpg_desc_tbl_off;unsigned char orig_transition_tmo; +unsigned long flags;if (!pg->expiry) {unsigned long transition_tmo =ALUA_FAILOVER_TIMEOUT *HZ ;
@@ -582,18 +584,32 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)else tpg_desc_tbl_off =4 ; -for (k = tpg_desc_tbl_off, ucp = buff + tpg_desc_tbl_off; +for (k = tpg_desc_tbl_off, desc = buff + tpg_desc_tbl_off; k < len; - k += off, ucp += off) { - -if (pg->group_id ==get_unaligned_be16 (&ucp[2 ])) { - pg->state = ucp[0 ] &0x0f ; - pg->pref = ucp[0 ] >>7 ; - valid_states = ucp[1 ]; + k += off, desc += off) { +u16 group_id =get_unaligned_be16 (&desc[2 ]); + +spin_lock_irqsave (&port_group_lock, flags); + tmp_pg =alua_find_get_pg (pg->device_id_str, pg->device_id_len, + group_id); +spin_unlock_irqrestore (&port_group_lock, flags); +if (tmp_pg) { +if (spin_trylock_irqsave (&tmp_pg->lock, flags)) { +if ((tmp_pg == pg) || + !(tmp_pg->flags &ALUA_PG_RUNNING )) { + tmp_pg->state = desc[0 ] &0x0f ; + tmp_pg->pref = desc[0 ] >>7 ; + } +if (tmp_pg == pg) + valid_states = desc[1 ]; +spin_unlock_irqrestore (&tmp_pg->lock, flags); + } +kref_put (&tmp_pg->kref, release_port_group); } - off =8 + (ucp[7 ] *4 ); + off =8 + (desc[7 ] *4 ); } +spin_lock_irqsave (&pg->lock, flags); sdev_printk(KERN_INFO, sdev, "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n", ALUA_DH_NAME, pg->group_id, print_alua_state(pg->state),
@@ -630,6 +646,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg) pg->expiry =0 ;break ; } +spin_unlock_irqrestore (&pg->lock, flags);kfree (buff);return err; }
Signature · 1 line
1.8.5.6
PATCH v8 18/23 scsi_dh_alua: Send TEST UNIT READY to poll for transitioning
In reply to Hannes ReineckeSending a 'REPORT TARGET PORT GROUP' command is a costly operation, as the array has to gather information about all ports. So instead of using RTPG to poll for a status update when a port is in transitioning we should be sending a TEST UNIT READY, and wait for the sense code to report success.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 38 ++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
drivers/scsi/device_handler/scsi_dh_alua.c +38 -0
@@ -466,6 +466,30 @@ static int alua_check_sense(struct scsi_device *sdev, }/* +* alua_tur - Send a TEST UNIT READY +* @sdev: device to which the TEST UNIT READY command should be send +* +* Send a TEST UNIT READY to @sdev to figure out the device state +* Returns SCSI_DH_RETRY if the sense code is NOT READY/ALUA TRANSITIONING, +* SCSI_DH_OK if no error occurred, and SCSI_DH_IO otherwise. +*/ +static int alua_tur (struct scsi_device *sdev) +{ +struct scsi_sense_hdr sense_hdr; +int retval; + + retval =scsi_test_unit_ready (sdev,ALUA_FAILOVER_TIMEOUT *HZ , +ALUA_FAILOVER_RETRIES , &sense_hdr); +if (sense_hdr.sense_key ==NOT_READY && + sense_hdr.asc ==0x04 && sense_hdr.ascq ==0x0a ) +return SCSI_DH_RETRY ; +else if (retval) +return SCSI_DH_IO ; +else +return SCSI_DH_OK ; +} + +/* * alua_rtpg - Evaluate REPORT TARGET GROUP STATES * @sdev: the device to be evaluated. *
@@ -732,8 +756,22 @@ static void alua_rtpg_work(struct work_struct *work) alua_wq = kaluad_sync_wq; pg->flags |=ALUA_PG_RUNNING ;if (pg->flags &ALUA_PG_RUN_RTPG ) { +int state = pg->state; + pg->flags &= ~ALUA_PG_RUN_RTPG ;spin_unlock_irqrestore (&pg->lock, flags); +if (state ==TPGS_STATE_TRANSITIONING ) { +if (alua_tur (sdev) ==SCSI_DH_RETRY ) { +spin_lock_irqsave (&pg->lock, flags); + pg->flags &= ~ALUA_PG_RUNNING ; + pg->flags |=ALUA_PG_RUN_RTPG ; +spin_unlock_irqrestore (&pg->lock, flags); +queue_delayed_work (alua_wq, &pg->rtpg_work, + pg->interval *HZ ); +return ; + } +/* Send RTPG on failure or if TUR indicates SUCCESS */ + } err =alua_rtpg (sdev, pg);spin_lock_irqsave (&pg->lock, flags);if (err ==SCSI_DH_RETRY || pg->flags &ALUA_PG_RUN_RTPG ) {
Signature · 1 line
1.8.5.6
PATCH v8 19/23 scsi_dh: add 'rescan' callback
In reply to Hannes ReineckeIf a device needs to be rescanned the device_handler might need to be rechecked, too.
So add a 'rescan' callback to the device handler and call it upon scsi_rescan_device(). The rescan callback will be invoked from the Unit Attention handling of ASC/ASCQ 3F 03 (INQUIRY DATA HAS CHANGED).
--- drivers/scsi/device_handler/scsi_dh_alua.c | 8 ++++++++ drivers/scsi/scsi_lib.c | 1 + drivers/scsi/scsi_scan.c | 8 +++++++- include/scsi/scsi_dh.h | 1 + 4 files changed, 17 insertions(+), 1 deletion(-)
drivers/scsi/device_handler/scsi_dh_alua.c +8 -0
@@ -1026,6 +1026,13 @@ static int alua_prep_fn(struct scsi_device *sdev, struct request *req) } +static void alua_rescan (struct scsi_device *sdev) +{ +struct alua_dh_data *h = sdev->handler_data; + +alua_initialize (sdev, h); +} +/* * alua_bus_attach - Attach device handler * @sdev: device to be attached to
@@ -1086,6 +1093,7 @@ static struct scsi_device_handler alua_dh = { .prep_fn = alua_prep_fn, .check_sense = alua_check_sense, .activate = alua_activate, + .rescan = alua_rescan, .set_params = alua_set_params, };
drivers/scsi/scsi_lib.c +1 -0
@@ -2699,6 +2699,7 @@ static void scsi_evt_emit(struct scsi_device *sdev, struct scsi_event *evt) envp[idx++] ="SDEV_MEDIA_CHANGE=1" ;break ;case SDEV_EVT_INQUIRY_CHANGE_REPORTED : +scsi_rescan_device (&sdev->sdev_gendev); envp[idx++] ="SDEV_UA=INQUIRY_DATA_HAS_CHANGED" ;break ;case SDEV_EVT_CAPACITY_CHANGE_REPORTED :
drivers/scsi/scsi_scan.c +7 -1
@@ -43,6 +43,7 @@#include <scsi/scsi_devinfo.h> #include <scsi/scsi_host.h> #include <scsi/scsi_transport.h> +#include <scsi/scsi_dh.h> #include <scsi/scsi_eh.h> #include "scsi_priv.h"
@@ -1524,9 +1525,14 @@ EXPORT_SYMBOL(scsi_add_device);void scsi_rescan_device (struct device *dev) { +struct scsi_device *sdev =to_scsi_device (dev); +device_lock (dev); -scsi_attach_vpd (to_scsi_device (dev)); +scsi_attach_vpd (sdev); + +if (sdev->handler && sdev->handler->rescan) + sdev->handler->rescan (sdev);if (dev->driver &&try_module_get (dev->driver->owner)) {struct scsi_driver *drv =to_scsi_driver (dev->driver);
include/scsi/scsi_dh.h +1 -0
@@ -71,6 +71,7 @@ struct scsi_device_handler {int (*activate)(struct scsi_device *,activate_complete ,void *);int (*prep_fn)(struct scsi_device *,struct request *);int (*set_params)(struct scsi_device *,const char *); +void (*rescan)(struct scsi_device *); };#ifdef CONFIG_SCSI_DH
Signature · 1 line
1.8.5.6
PATCH v8 20/23 scsi: Add 'access_state' attribute
In reply to Hannes ReineckeAdd an 'access_state' attribute to struct scsi_device to display the asymmetric LUN access state.
--- drivers/scsi/scsi_sysfs.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++ include/scsi/scsi_device.h | 1 + include/scsi/scsi_proto.h | 12 +++++++++++ 3 files changed, 66 insertions(+)
drivers/scsi/scsi_sysfs.c +53 -0
@@ -81,6 +81,33 @@ const char *scsi_host_state_name(enum scsi_host_state state)return name; } +static const struct { +unsigned char value; +char *name; +} sdev_access_states[] = { + {SCSI_ACCESS_STATE_OPTIMAL ,"active/optimized" }, + {SCSI_ACCESS_STATE_ACTIVE ,"active/non-optimized" }, + {SCSI_ACCESS_STATE_STANDBY ,"standby" }, + {SCSI_ACCESS_STATE_UNAVAILABLE ,"unavailable" }, + {SCSI_ACCESS_STATE_LBA ,"lba-dependent" }, + {SCSI_ACCESS_STATE_OFFLINE ,"offline" }, + {SCSI_ACCESS_STATE_TRANSITIONING ,"transitioning" }, +}; + +const char *scsi_access_state_name (unsigned char state) +{ +int i; +char *name =NULL ; + +for (i =0 ; i <ARRAY_SIZE (sdev_access_states); i++) { +if (sdev_access_states[i].value == state) { + name = sdev_access_states[i].name; +break ; + } + } +return name; +} +static int check_set (unsigned long long *val,char *src) {char *last;
@@ -973,6 +1000,31 @@ sdev_store_dh_state(struct device *dev, struct device_attribute *attr, static DEVICE_ATTR(dh_state, S_IRUGO | S_IWUSR, sdev_show_dh_state, sdev_store_dh_state); + +static ssize_t +sdev_show_access_state (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct scsi_device *sdev =to_scsi_device (dev); +unsigned char access_state; +const char *access_state_name; +bool pref = false; + +if (!sdev->handler) +return -EINVAL ; + +if (sdev->access_state &SCSI_ACCESS_STATE_PREFERRED ) + pref = true; + + access_state = (sdev->access_state &SCSI_ACCESS_STATE_MASK ); + access_state_name =scsi_access_state_name (access_state); + +return snprintf (buf,32 ,"%s%s\n" , + access_state_name ? access_state_name :"unknown" , + pref ?" preferred" :"" ); +} +static DEVICE_ATTR(access_state, S_IRUGO, sdev_show_access_state, NULL);#endif static ssize_t
@@ -1047,6 +1099,7 @@ static struct attribute *scsi_sdev_attrs[] = { &dev_attr_wwid.attr, #ifdef CONFIG_SCSI_DH &dev_attr_dh_state.attr, + &dev_attr_access_state.attr, #endif &dev_attr_queue_ramp_up_period.attr, REF_EVT(media_change),
include/scsi/scsi_device.h +1 -0
@@ -201,6 +201,7 @@ struct scsi_device {struct scsi_device_handler *handler;void *handler_data; +unsigned char access_state;enum scsi_device_state sdev_state;unsigned long sdev_data[0 ]; } __attribute__((aligned(sizeof(unsigned long))));
include/scsi/scsi_proto.h +12 -0
@@ -277,5 +277,17 @@ struct scsi_lun { __u8 scsi_lun[8]; }; +/* SPC asymmetric access states */ +#define SCSI_ACCESS_STATE_OPTIMAL 0x00 +#define SCSI_ACCESS_STATE_ACTIVE 0x01 +#define SCSI_ACCESS_STATE_STANDBY 0x02 +#define SCSI_ACCESS_STATE_UNAVAILABLE 0x03 +#define SCSI_ACCESS_STATE_LBA 0x04 +#define SCSI_ACCESS_STATE_OFFLINE 0x0e +#define SCSI_ACCESS_STATE_TRANSITIONING 0x0f + +/* Values for REPORT TARGET GROUP STATES */ +#define SCSI_ACCESS_STATE_MASK 0x0f +#define SCSI_ACCESS_STATE_PREFERRED 0x80#endif /* _SCSI_PROTO_H_ */
Signature · 1 line
1.8.5.6
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 13 lines · Hannes Reinecke
(quoted diff) +13
-0+static ssize_t +sdev_show_access_state(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct scsi_device *sdev = to_scsi_device(dev); + unsigned char access_state; + const char *access_state_name; + bool pref = false; + + if (!sdev->handler) + return -EINVAL; +
Review
Quoted 10 lines
(quoted diff) +10
-0+ if (sdev->access_state & SCSI_ACCESS_STATE_PREFERRED) + pref = true; + + access_state = (sdev->access_state & SCSI_ACCESS_STATE_MASK); + access_state_name = scsi_access_state_name(access_state); + + return snprintf(buf, 32, "%s%s\n", + access_state_name ? access_state_name : "unknown", + pref ? " preferred" : ""); +}
Hello Hannes,
What is inelegant about the above approach is that the access_state attribute is added to all SCSI devices, whether or not a device handler has been attached, and that reading that attribute doesn't fail for devices to which another handler than the ALUA handler has been attached. How about the patch below, which moves the access_state show handler to the scsi_dh_alua.c source file, where it belongs?
[PATCH] Restrict access_state attribute to devices that support ALUA
--- drivers/scsi/device_handler/scsi_dh_alua.c | 52 +++++++++++++++++++++++++++++ drivers/scsi/scsi_dh.c | 14 ++++++++ drivers/scsi/scsi_sysfs.c | 53 ------------------------------ include/scsi/scsi_dh.h | 1 + 4 files changed, 67 insertions(+), 53 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +52 -0
@@ -1121,6 +1121,57 @@ static void alua_bus_detach(struct scsi_device *sdev)kfree (h); } +static const struct { +unsigned char value; +const char *name; +} sdev_access_states[] = { + {SCSI_ACCESS_STATE_OPTIMAL ,"active/optimized" }, + {SCSI_ACCESS_STATE_ACTIVE ,"active/non-optimized" }, + {SCSI_ACCESS_STATE_STANDBY ,"standby" }, + {SCSI_ACCESS_STATE_UNAVAILABLE ,"unavailable" }, + {SCSI_ACCESS_STATE_LBA ,"lba-dependent" }, + {SCSI_ACCESS_STATE_OFFLINE ,"offline" }, + {SCSI_ACCESS_STATE_TRANSITIONING ,"transitioning" }, +}; + +const char *scsi_access_state_name (unsigned char state) +{ +int i; +const char *name =NULL ; + +for (i =0 ; i <ARRAY_SIZE (sdev_access_states); i++) { +if (sdev_access_states[i].value == state) { + name = sdev_access_states[i].name; +break ; + } + } +return name; +} + +static ssize_t +sdev_show_access_state (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct scsi_device *sdev =to_scsi_device (dev); +unsigned char access_state; +const char *access_state_name; +bool pref = sdev->access_state &SCSI_ACCESS_STATE_PREFERRED ; + + access_state = sdev->access_state &SCSI_ACCESS_STATE_MASK ; + access_state_name =scsi_access_state_name (access_state); + +return snprintf (buf,32 ,"%s%s\n" , + access_state_name ? access_state_name :"unknown" , + pref ?" preferred" :"" ); +} +static DEVICE_ATTR(access_state, S_IRUGO, sdev_show_access_state, NULL); + +static const struct attribute *alua_attrs[] = { + &dev_attr_access_state.attr, +NULL +}; +static struct scsi_device_handler alua_dh = { .name =ALUA_DH_NAME , .module =THIS_MODULE ,
@@ -1131,6 +1182,7 @@ static struct scsi_device_handler alua_dh = { .activate = alua_activate, .rescan = alua_rescan, .set_params = alua_set_params, + .attrs = alua_attrs, };static int __init alua_init (void )
drivers/scsi/scsi_dh.c +11 -0
@@ -164,6 +167,17 @@ int scsi_dh_add_device(struct scsi_device *sdev) devinfo =__scsi_dh_lookup (drv);if (devinfo) err =scsi_dh_handler_attach (sdev, devinfo); +if (err ==0 && sdev->handler && sdev->handler->attrs) { + err =sysfs_create_files (&sdev->sdev_gendev.kobj, + sdev->handler->attrs); +if (err) { +sdev_printk (KERN_INFO , sdev, +"failed to add device handler sysfs attributes: %d\n" , + err); +scsi_dh_handler_detach (sdev); + } + } +return err; }
drivers/scsi/scsi_sysfs.c +0 -53
71 lines not on this page · raw message
include/scsi/scsi_dh.h +1 -0
@@ -72,6 +72,7 @@ struct scsi_device_handler {int (*prep_fn)(struct scsi_device *,struct request *);int (*set_params)(struct scsi_device *,const char *);void (*rescan)(struct scsi_device *); +const struct attribute **attrs; };#ifdef CONFIG_SCSI_DH
Signature · 1 line
2.7.1
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 27 lines · Bart Van Assche
Quoted 13 lines · Hannes Reinecke
drivers/scsi/device_handler/scsi_dh_alua.c +13
-0+static ssize_t +sdev_show_access_state (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct scsi_device *sdev =to_scsi_device (dev); +unsigned char access_state; +const char *access_state_name; +bool pref = false; + +if (!sdev->handler) +return -EINVAL ; +drivers/scsi/device_handler/scsi_dh_alua.c static void alua_bus_detach(struct scsi_device *sdev)
Quoted 10 lines
drivers/scsi/device_handler/scsi_dh_alua.c +10
-0+if (sdev->access_state &SCSI_ACCESS_STATE_PREFERRED ) + pref = true; + + access_state = (sdev->access_state &SCSI_ACCESS_STATE_MASK ); + access_state_name =scsi_access_state_name (access_state); + +return snprintf (buf,32 ,"%s%s\n" , + access_state_name ? access_state_name :"unknown" , + pref ?" preferred" :"" ); +}Hello Hannes,
Quoted 6 lines
What is inelegant about the above approach is that the
access_stateattribute is added to all SCSI devices, whether or not a device handler has been attached, and that reading that attribute doesn't fail for devices to which another handler than the ALUA handler has been attached. How about the patch below, which moves theaccess_stateshow handler to thescsi_dh_alua.csource file, where it belongs?
Actually, this was my first approach, but hch suggested moving it to the generic code :-(
Also when using your suggestion the 'access_state' attribute will only be created _after_ the 'ADD' uevent, making it impossible to use it from udev events.
However, I'm currently working on using the 'is_visible' callback for SCSI sysfs attributes for this; in theory it should be possible to use 'sysfs_update_groups()' during scsi_add_lun() to blank out the
unsupported attributes.
Which I think might be the best approach here.
But probably not for this patch series; I'd rather have the alua update in first as it took long enough.
If you're absolutely against it we can drop the 'access_state' patches (ie patch 20-22) and have them folded into a separate patchset.
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 3 lines · Hannes Reinecke
Also when using your suggestion the '
access_state' attribute will only be created _after_ the 'ADD' uevent, making it impossible to use it from udev events.
Can you give an example in which it would be useful to read the ALUA state from a udev handler ? I'm not sure such an example exists.
Quoted 2 lines
If you're absolutely against it we can drop the '
access_state' patches (ie patch 20-22) and have them folded into a separate patchset.
That sounds like a good idea to me.
Thanks,
Bart.
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 8 lines · Bart Van Assche
Quoted 3 lines · Hannes Reinecke
Also when using your suggestion the '
access_state' attribute will only be created _after_ the 'ADD' uevent, making it impossible to use it from udev events.Can you give an example in which it would be useful to read the ALUA state from a udev handler ? I'm not sure such an example exists.
When evaluating the 'access_state' from an uevent we can avoid sending I/O if the path is unavailable; eg if the path is in 'transitioning' I/O will be queued until that path becomes available again. Which means that the uevent will be delayed during udev processing, so that the event will never be read by multipathing (as it's being invoked
only after udev event processing has finished).
And in extreme cases (like OnTap takeover/giveback) it will even drop the event completely due to a timeout.
Quoted 5 lines
Quoted 2 lines
If you're absolutely against it we can drop the '
access_state' patches (ie patch 20-22) and have them folded into a separate patchset.That sounds like a good idea to me.
Ok, let's do this.
Martin, can you please ignore patches 20-22 when pulling the patchset? I'll be resubmitting a new patchset with them once these issues are sorted out. Or do you need a new patch submission?
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Hannes" == Hannes Reinecke <[email protected]> writes:
Hannes> Martin, can you please ignore patches 20-22 when pulling the Hannes> patchset? I'll be resubmitting a new patchset with them once Hannes> these issues are sorted out. Or do you need a new patch Hannes> submission?
No, it's OK. I'll merge the first part of the series in the morning.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
In reply to Hannes ReineckeQuoted 9 lines · Hannes Reinecke
Quoted 8 lines · Bart Van Assche
Quoted 3 lines · Hannes Reinecke
Also when using your suggestion the '
access_state' attribute will only be created _after_ the 'ADD' uevent, making it impossible to use it from udev events.Can you give an example in which it would be useful to read the ALUA state from a udev handler ? I'm not sure such an example exists.
Quoted 8 lines
When evaluating the '
access_state' from an uevent we can avoid sending I/O if the path is unavailable; eg if the path is in 'transitioning' I/O will be queued until that path becomes available again. Which means that the uevent will be delayed during udev processing, so that the event will never be read by multipathing (as it's being invoked only after udev event processing has finished).
And in extreme cases (like OnTap takeover/giveback) it will even drop the event completely due to a timeout.
Hello Hannes,
Thanks for the feedback.
Please split the access_state attribute in two sysfs attributes - one for the ALUA state and one for the "preferred" state. This will make it easier for shell scripts to process these sysfs attributes.
I agree with using the 'is_visible' callback. But since using that callback will cause the ALUA sysfs attributes to become only visible after the ALUA handler has been attached I don't see why not to move the implementation of the ALUA sysfs attributes into the ALUA device handler source file.
BTW, since the ALUA state can change at any time after the access_state sysfs attribute has been read and before I/O is submitted my preference is to avoid triggering any SCSI commands from a udev rule that depend on the ALUA state.
Bart.
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 20 lines · Bart Van Assche
Quoted 10 lines · Hannes Reinecke
Quoted 9 lines · Bart Van Assche
Quoted 4 lines · Hannes Reinecke
Also when using your suggestion the '
access_state' attribute will only be created _after_ the 'ADD' uevent, making it impossible to use it from
udev events.Can you give an example in which it would be useful to read the ALUA state from a udev handler ? I'm not sure such an example exists.
Quoted 8 lines
When evaluating the '
access_state' from an uevent we can avoid sending I/O if the path is unavailable; eg if the path is in 'transitioning' I/O will be queued until that path becomes available again. Which means that the uevent will be delayed during udev processing, so that the event will never be read by multipathing (as it's being invoked only after udev event processing has finished).
And in extreme cases (like OnTap takeover/giveback) it will even drop the event completely due to a timeout.
Quoted 8 lines
Hello Hannes,
Thanks for the feedback.
Please split the
access_stateattribute in two sysfs attributes - one for the ALUA state and one for the "preferred" state. This will make it easier for shell scripts to process these sysfs attributes.
Okay, agreed.
Quoted 6 lines
I agree with using the '
is_visible' callback. But since using that callback will cause the ALUA sysfs attributes to become only visible after the ALUA handler has been attached I don't see why not to move the implementation of the ALUA sysfs attributes into the ALUA device handler source file.
Thing is, there are others (like the vpg_pg attributes) which will benefit from using the 'is_visible' callback. And move the 'access_state' attribute into the generic functions makes the footprint smaller, as the required infrastructure is already present. Plus I would really like to have the 'access_state' variable part of the scsi_device, to avoid the intrinsics of having to look it up from the device_handler data.
Quoted 5 lines
BTW, since the ALUA state can change at any time after the
access_statesysfs attribute has been read and before I/O is submitted my preference is to avoid triggering any SCSI commands from a udev rule that depend on the ALUA state.
Yes, I thought about it some more, too.
The main reason why I need the 'access_state' attribute is to decouple the multipath daemon; at the moment the multipath daemon has to issue REPORT TARGET PORT GROUPS frequently to figure out the status, which is causing quite some load on the target. When using the 'access_state' attribute we would avoid doing I/O for that and have a consistent view, both on the kernel and the multipath daemon side.
But it's actually a good thing to have the 'access_state' patch in a different series; I've got some more patches converting the remaining device_handler to also supply the 'access_state' values. Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 10 lines · Hannes Reinecke
The main reason why I need the '
access_state' attribute is to decouple the multipath daemon; at the moment the multipath daemon has to issue REPORT TARGET PORT GROUPS frequently to figure out the status, which is causing quite some load on the target. When using the 'access_state' attribute we would avoid doing I/O for that and have a consistent view, both on the kernel and the multipath daemon side.But it's actually a good thing to have the '
access_state' patch in a different series; I've got some more patches converting the remainingdevice_handlerto also supply the 'access_state' values.
Hello Hannes,
The above sounds very interesting to me. Will multipathd recognize at run-time whether or not the kernel supports the sysfs ALUA state attribute ? Will ALUA state changes be reported through udev or will multipathd poll the sysfs ALUA state attributes ? And if the netlink buffer that is used in multipathd to receive udev events overflows (ENOBUFS), will multipathd resynchronize its state ? As far as I can see in source file libmultipath/uevent.c today multipathd ignores netlink buffer overflows.
Thanks,
Bart.
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Hannes Reinecke
The main reason why I need the '
access_state' attribute is to decouple the multipath daemon; at the moment the multipath daemon has to issue REPORT TARGET PORT GROUPS frequently to figure out the status, which is causing quite some load on the target. When using the 'access_state' attribute we would avoid doing I/O for that and have a consistent view, both on the kernel and the multipath daemon side.But it's actually a good thing to have the '
access_state' patch in a different series; I've got some more patches converting the remainingdevice_handlerto also supply the 'access_state' values.
Quoted 5 lines
Hello Hannes,
The above sounds very interesting to me. Will multipathd recognize at run-time whether or not the kernel supports the sysfs ALUA state attribute ?
That was the plan.
(Not that I've got any idea _how_ to do this ATM :-)
Quoted 2 lines
Will ALUA state changes be reported through udev or will multipathd poll the sysfs ALUA state attributes ?
Polling. udev events will be too unreliable here, as each uevent potentially causes I/O during uevent processing which might stall as the path might be down.
So we might never get events for failed/transitioning paths, however, these are precisely the cases which we're interested in ...
Quoted 6 lines
And if the netlink
buffer that is used in multipathd to receive udev events overflows (ENOBUFS), will multipathd resynchronize its state ? As far as I can see in source filelibmultipath/uevent.ctoday multipathd ignores netlink buffer overflows.
Which, thankfully, doesn't apply as multipathd will be polling the sysfs attribute directly :-)
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv8 20/23] scsi: Add 'access_state' attribute
Quoted 21 lines · Hannes Reinecke
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Hannes Reinecke
The main reason why I need the '
access_state' attribute is to decouple the multipath daemon; at the moment the multipath daemon has to issue REPORT TARGET PORT GROUPS frequently to figure out the status, which is causing quite some load on the target. When using the 'access_state' attribute we would avoid doing I/O for that and have a consistent view, both on the kernel and the multipath daemon side.But it's actually a good thing to have the '
access_state' patch in a different series; I've got some more patches converting the remainingdevice_handlerto also supply the 'access_state' values.Quoted 5 lines
Hello Hannes,
The above sounds very interesting to me. Will multipathd recognize at run-time whether or not the kernel supports the sysfs ALUA state attribute ?
That was the plan.
(Not that I've got any idea _how_ to do this ATM :-)
Quoted 7 lines
Quoted 2 lines
Will ALUA state changes be reported through udev or will multipathd poll the sysfs ALUA state attributes ?
Polling. udev events will be too unreliable here, as each uevent potentially causes I/O during uevent processing which might stall as the path might be down.
So we might never get events for failed/transitioning paths, however, these are precisely the cases which we're interested in ...
And, in fact, if paths go down due to e.g. an FC switch failure, it is common for a large number of paths to go down at once, causing a large number of uevents to be queued.
Quoted 13 lines
Quoted 7 lines
And if the netlink
buffer that is used in multipathd to receive udev events overflows (ENOBUFS), will multipathd resynchronize its state ? As far as I can see in source filelibmultipath/uevent.ctoday multipathd ignores netlink buffer overflows.Which, thankfully, doesn't apply as multipathd will be polling the sysfs attribute directly :-)
Cheers,
Hannes
PATCH v8 21/23 scsi_dh_alua: use common definitions for ALUA state
In reply to Hannes Reineckescsi_proto.h now contains definitions for the ALUA state, so we don't have to carry them in the device handler.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 58 +++++++++++++----------------- 1 file changed, 25 insertions(+), 33 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +25 -33
@@ -31,14 +31,6 @@#define ALUA_DH_NAME "alua"#define ALUA_DH_VER "1.3" -#define TPGS_STATE_OPTIMIZED 0x0 -#define TPGS_STATE_NONOPTIMIZED 0x1 -#define TPGS_STATE_STANDBY 0x2 -#define TPGS_STATE_UNAVAILABLE 0x3 -#define TPGS_STATE_LBA_DEPENDENT 0x4 -#define TPGS_STATE_OFFLINE 0xe -#define TPGS_STATE_TRANSITIONING 0xf -#define TPGS_SUPPORT_NONE 0x00#define TPGS_SUPPORT_OPTIMIZED 0x01#define TPGS_SUPPORT_NONOPTIMIZED 0x02
@@ -180,7 +172,7 @@ static int submit_stpg(struct scsi_device *sdev, int group_id,/* Prepare the data buffer */ memset (stpg_data,0 , stpg_len); - stpg_data[4 ] =TPGS_STATE_OPTIMIZED &0x0f ; + stpg_data[4 ] =SCSI_ACCESS_STATE_OPTIMAL ;put_unaligned_be16 (group_id, &stpg_data[6 ]);/* Prepare the command. */
@@ -248,7 +240,7 @@ struct alua_port_group *alua_alloc_pg(struct scsi_device *sdev, } pg->group_id = group_id; pg->tpgs = tpgs; - pg->state =TPGS_STATE_OPTIMIZED ; + pg->state =SCSI_ACCESS_STATE_OPTIMAL ;if (optimize_stpg) pg->flags |=ALUA_OPTIMIZE_STPG ;kref_init (&pg->kref);
@@ -378,22 +370,22 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h,return SCSI_DH_OK ; } -static char print_alua_state (int state) +static char print_alua_state (unsigned char state) {switch (state) { -case TPGS_STATE_OPTIMIZED : +case SCSI_ACCESS_STATE_OPTIMAL :return 'A' ; -case TPGS_STATE_NONOPTIMIZED : +case SCSI_ACCESS_STATE_ACTIVE :return 'N' ; -case TPGS_STATE_STANDBY : +case SCSI_ACCESS_STATE_STANDBY :return 'S' ; -case TPGS_STATE_UNAVAILABLE : +case SCSI_ACCESS_STATE_UNAVAILABLE :return 'U' ; -case TPGS_STATE_LBA_DEPENDENT : +case SCSI_ACCESS_STATE_LBA :return 'L' ; -case TPGS_STATE_OFFLINE : +case SCSI_ACCESS_STATE_OFFLINE :return 'O' ; -case TPGS_STATE_TRANSITIONING : +case SCSI_ACCESS_STATE_TRANSITIONING :return 'T' ;default :return 'X' ;
@@ -647,7 +639,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg) valid_states&TPGS_SUPPORT_OPTIMIZED?'A':'a');switch (pg->state) { -case TPGS_STATE_TRANSITIONING : +case SCSI_ACCESS_STATE_TRANSITIONING :if (time_before (jiffies, pg->expiry)) {/* State transition, retry */ pg->interval =2 ;
@@ -655,11 +647,11 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg) }else {/* Transitioning time exceeded, set port to standby */ err =SCSI_DH_IO ; - pg->state =TPGS_STATE_STANDBY ; + pg->state =SCSI_ACCESS_STATE_STANDBY ; pg->expiry =0 ; }break ; -case TPGS_STATE_OFFLINE : +case SCSI_ACCESS_STATE_OFFLINE :/* Path unusable */ err =SCSI_DH_DEV_OFFLINED ; pg->expiry =0 ;
@@ -693,20 +685,20 @@ static unsigned alua_stpg(struct scsi_device *sdev, struct alua_port_group *pg)return SCSI_DH_RETRY ; }switch (pg->state) { -case TPGS_STATE_OPTIMIZED : +case SCSI_ACCESS_STATE_OPTIMAL :return SCSI_DH_OK ; -case TPGS_STATE_NONOPTIMIZED : +case SCSI_ACCESS_STATE_ACTIVE :if ((pg->flags &ALUA_OPTIMIZE_STPG ) && !pg->pref && (pg->tpgs &TPGS_MODE_IMPLICIT ))return SCSI_DH_OK ;break ; -case TPGS_STATE_STANDBY : -case TPGS_STATE_UNAVAILABLE : +case SCSI_ACCESS_STATE_STANDBY : +case SCSI_ACCESS_STATE_UNAVAILABLE :break ; -case TPGS_STATE_OFFLINE : +case SCSI_ACCESS_STATE_OFFLINE :return SCSI_DH_IO ; -case TPGS_STATE_TRANSITIONING : +case SCSI_ACCESS_STATE_TRANSITIONING :break ;default : sdev_printk(KERN_INFO, sdev,
@@ -760,7 +752,7 @@ static void alua_rtpg_work(struct work_struct *work) pg->flags &= ~ALUA_PG_RUN_RTPG ;spin_unlock_irqrestore (&pg->lock, flags); -if (state ==TPGS_STATE_TRANSITIONING ) { +if (state ==SCSI_ACCESS_STATE_TRANSITIONING ) {if (alua_tur (sdev) ==SCSI_DH_RETRY ) {spin_lock_irqsave (&pg->lock, flags); pg->flags &= ~ALUA_PG_RUNNING ;
@@ -1006,7 +998,7 @@ static int alua_prep_fn(struct scsi_device *sdev, struct request *req) {struct alua_dh_data *h = sdev->handler_data;struct alua_port_group __rcu *pg; -int state =TPGS_STATE_OPTIMIZED ; +unsigned char state =SCSI_ACCESS_STATE_OPTIMAL ;int ret =BLKPREP_OK ;rcu_read_lock ();
@@ -1014,11 +1006,11 @@ static int alua_prep_fn(struct scsi_device *sdev, struct request *req)if (pg) state = pg->state;rcu_read_unlock (); -if (state ==TPGS_STATE_TRANSITIONING ) +if (state ==SCSI_ACCESS_STATE_TRANSITIONING ) ret =BLKPREP_DEFER ; -else if (state !=TPGS_STATE_OPTIMIZED && - state !=TPGS_STATE_NONOPTIMIZED && - state !=TPGS_STATE_LBA_DEPENDENT ) { +else if (state !=SCSI_ACCESS_STATE_OPTIMAL && + state !=SCSI_ACCESS_STATE_ACTIVE && + state !=SCSI_ACCESS_STATE_LBA ) { ret =BLKPREP_KILL ; req->cmd_flags |=REQ_QUIET ; }
Signature · 1 line
1.8.5.6
PATCH v8 22/23 scsi_dh_alua: update 'access_state' field
In reply to Hannes ReineckeTrack attached SCSI devices and update the 'access_state' field whenever an ALUA state change has been detected.
--- drivers/scsi/device_handler/scsi_dh_alua.c | 48 ++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +46 -2
@@ -24,6 +24,7 @@#include <linux/module.h> #include <asm/unaligned.h> #include <scsi/scsi.h> +#include <scsi/scsi_proto.h> #include <scsi/scsi_dbg.h> #include <scsi/scsi_eh.h> #include <scsi/scsi_dh.h>
@@ -75,6 +76,7 @@ struct alua_port_group {struct kref kref;struct rcu_head rcu;struct list_head node; +struct list_head dh_list;unsigned char device_id_str[256 ];int device_id_len;int group_id;
@@ -92,6 +94,7 @@ struct alua_port_group { };struct alua_dh_data { +struct list_head node;struct alua_port_group *pg;int group_id;spinlock_t pg_lock;
@@ -247,6 +250,7 @@ struct alua_port_group *alua_alloc_pg(struct scsi_device *sdev,INIT_DELAYED_WORK (&pg->rtpg_work, alua_rtpg_work);INIT_LIST_HEAD (&pg->rtpg_list);INIT_LIST_HEAD (&pg->node); +INIT_LIST_HEAD (&pg->dh_list);spin_lock_init (&pg->lock);spin_lock (&port_group_lock);
@@ -328,6 +332,8 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h, {int rel_port =-1 , group_id;struct alua_port_group *pg, *old_pg =NULL ; +bool pg_updated; +unsigned long flags; group_id =scsi_vpd_tpg_id (sdev, &rel_port);if (group_id <0 ) {
@@ -357,10 +363,22 @@ static int alua_check_vpd(struct scsi_device *sdev, struct alua_dh_data *h, old_pg = h->pg;if (old_pg != pg) {/* port group has changed. Update to new port group */ +if (h->pg) { +spin_lock_irqsave (&old_pg->lock, flags); +list_del_rcu (&h->node); +spin_unlock_irqrestore (&old_pg->lock, flags); + }rcu_assign_pointer (h->pg, pg); + pg_updated = true; } + +spin_lock_irqsave (&pg->lock, flags);if (sdev->synchronous_alua) pg->flags |=ALUA_SYNC_STPG ; +if (pg_updated) +list_add_rcu (&h->node, &pg->dh_list); +spin_unlock_irqrestore (&pg->lock, flags); +alua_rtpg_queue (h->pg, sdev,NULL , true);spin_unlock (&h->pg_lock);
@@ -613,8 +631,18 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)if (spin_trylock_irqsave (&tmp_pg->lock, flags)) {if ((tmp_pg == pg) || !(tmp_pg->flags &ALUA_PG_RUNNING )) { +struct alua_dh_data *h; + tmp_pg->state = desc[0 ] &0x0f ; tmp_pg->pref = desc[0 ] >>7 ; +rcu_read_lock (); +list_for_each_entry_rcu (h, + &tmp_pg->dh_list, node) { +/* h->sdev should always be valid */ +BUG_ON (!h->sdev); + h->sdev->access_state = desc[0 ]; + } +rcu_read_unlock (); }if (tmp_pg == pg) valid_states = desc[1 ];
@@ -645,10 +673,22 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg) pg->interval =2 ; err =SCSI_DH_RETRY ; }else { +struct alua_dh_data *h; +/* Transitioning time exceeded, set port to standby */ err =SCSI_DH_IO ; pg->state =SCSI_ACCESS_STATE_STANDBY ; pg->expiry =0 ; +rcu_read_lock (); +list_for_each_entry_rcu (h, &pg->dh_list, node) { +BUG_ON (!h->sdev); + h->sdev->access_state = + (pg->state &SCSI_ACCESS_STATE_MASK ); +if (pg->pref) + h->sdev->access_state |= +SCSI_ACCESS_STATE_PREFERRED ; + } +rcu_read_unlock (); }break ;case SCSI_ACCESS_STATE_OFFLINE :
@@ -1041,6 +1081,7 @@ static int alua_bus_attach(struct scsi_device *sdev)rcu_assign_pointer (h->pg,NULL ); h->init_error =SCSI_DH_OK ; h->sdev = sdev; +INIT_LIST_HEAD (&h->node);mutex_init (&h->init_mutex); err =alua_initialize (sdev, h);
@@ -1070,9 +1111,12 @@ static void alua_bus_detach(struct scsi_device *sdev)rcu_assign_pointer (h->pg,NULL ); h->sdev =NULL ;spin_unlock (&h->pg_lock); -if (pg) +if (pg) { +spin_lock (&pg->lock); +list_del_rcu (&h->node); +spin_unlock (&pg->lock);kref_put (&pg->kref, release_port_group); - + } sdev->handler_data =NULL ;kfree (h); }
Signature · 1 line
1.8.5.6
PATCH v8 23/23 scsi_dh_alua: Update version to 2.0
In reply to Hannes Reinecke--- drivers/scsi/device_handler/scsi_dh_alua.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/scsi/device_handler/scsi_dh_alua.c +1 -1
@@ -30,7 +30,7 @@#include <scsi/scsi_dh.h> #define ALUA_DH_NAME "alua" -#define ALUA_DH_VER "1.3" +#define ALUA_DH_VER "2.0"#define TPGS_SUPPORT_NONE 0x00#define TPGS_SUPPORT_OPTIMIZED 0x01
Signature · 1 line
1.8.5.6
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Hannes" == Hannes Reinecke <[email protected]> writes:
Hannes> As promised here is now the second part of my ALUA device Hannes> handler update. This contains a major rework of the ALUA device Hannes> handler as execution is moved onto a workqueue. This has the Hannes> advantage that we avoid having to do multiple calls to the same Hannes> LUN (as happens frequently when failing over a LUN with several Hannes> paths) and finally retries are handled correctly. As some Hannes> arrays are only capable of handling one STPG at a time I've Hannes> added a blacklist flag which then uses a singlethreaded Hannes> workqueue, thereby effectively synchronize STPG handling. Hannes> Thanks to Bart for this suggestion.
Applied 1-19 and 23 to 4.6/scsi-queue.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering