Skip to contentSkip to conversation

ieee1394: sbp2: misc updates

PATCH New Stefan Richter 1 participant 7 messages Jul 23, 2006

lore
Share & tools
lore
#0 lore raw

cover ieee1394: sbp2: misc updates

The following patches include minor fixes and cleanups:

1/6 ieee1394: sbp2: safer last_orb and next_ORB handling 2/6 ieee1394: sbp2: discard return value of sbp2_link_orb_command 3/6 ieee1394: sbp2: optimize DMA direction of command ORBs 4/6 ieee1394: sbp2: safer initialization of status fifo 5/6 ieee1394: sbp2: more checks of status block
6/6 ieee1394: sbp2: convert sbp2util_down_timeout to waitqueue

 drivers/ieee1394/sbp2.c |  306 +++++++++++++++++-----------------------
 drivers/ieee1394/sbp2.h |   27 +--
 2 files changed, 143 insertions(+), 190 deletions(-)

I hope to follow up with further fixes that deal with command ORB queueing during the next few weeks.

Signature · 3 lines
Stefan Richter
-=====-=-==- -=== =-===
http://arcgraph.de/sr/
#1 lore raw

PATCH 1/6 ieee1394: sbp2: safer last_orb and next_ORB handling

The sbp2 initiator has two ways to tell a target's fetch agent about new command ORBs:
- Write the ORB's address to the ORB_POINTER register. This must not be done while the fetch agent is active.
- Put the ORB's address into the previously submitted ORB's next_ORB field and write to the DOORBELL register. This may be done while the fetch agent is active or suspended. It must not be done while the fetch agent is in reset state.
Sbp2 has a last_orb pointer which indicates in what way a new command should be announced. That pointer is concurrently accessed at various occasions. Furthermore, initiator and target are accessing the next_ORB field of ORBs concurrently and asynchronously.

This patch does:
- Protect all initiator accesses to last_orb by sbp2_command_orb_lock.
- Add pci_dma_sync_single_for_device before a previously submitted ORB's next_ORB field is overwritten.
- Insert a memory barrier between when next_ORB_lo and next_ORB_hi are overwritten. Next_ORB_hi must not be updated before next_ORB_lo.
- Remove the rather unspecific and now superfluous qualifier "volatile" from the next_ORB fields.
- Add comments on how last_orb is connected with what is known about the target's fetch agent's state.

---
 drivers/ieee1394/sbp2.c |  101 +++++++++++++++++++---------------------
 drivers/ieee1394/sbp2.h |    4 -
 2 files changed, 50 insertions(+), 55 deletions(-)

TODO:
There are further issues with how the ORB lists are maintained which I hope to address RSN.

drivers/ieee1394/sbp2.h +2 -2
Index: linux/drivers/ieee1394/sbp2.h
===================================================================
@@ -46,8 +46,8 @@
 #define ORB_SET_DIRECTION(value)		((value & 0x1) << 27)
 
 struct sbp2_command_orb {
-	volatile u32 next_ORB_hi;
-	volatile u32 next_ORB_lo;
+	u32 next_ORB_hi;
+	u32 next_ORB_lo;
 	u32 data_descriptor_hi;
 	u32 data_descriptor_lo;
 	u32 misc;
drivers/ieee1394/sbp2.c +48 -53

147 lines not on this page · raw message

#2 lore raw

PATCH 2/6 ieee1394: sbp2: discard return value of sbp2_link_orb_command

In reply to Stefan Richter

Since sbp2 is at the moment unable to do anything with the return value of sbp2_link_orb_command, just discard it.

---
 drivers/ieee1394/sbp2.c |    8 +++-----
 drivers/ieee1394/sbp2.h |    5 -----
 2 files changed, 3 insertions(+), 10 deletions(-)

TODO:
Alas sbp2 initiates 1394 write requests to the target's ORB_POINTER and DOORBELL registers from atomic context. These write transactions should be moved into non-atomic context, i.e. a kthread or workqueue. That way typical failures of sbp2util_node_write_no_wait can be avoided in the first place, and there are better possibilities to recover from remaining failures.

drivers/ieee1394/sbp2.c +3 -5
Index: linux/drivers/ieee1394/sbp2.c
===================================================================
@@ -1964,7 +1964,7 @@ static void sbp2_create_command_orb(stru
 /*
  * This function is called in order to begin a regular SBP-2 command.
  */
-static int sbp2_link_orb_command(struct scsi_id_instance_data *scsi_id,
+static void sbp2_link_orb_command(struct scsi_id_instance_data *scsi_id,
 				 struct sbp2_command_info *command)
 {
 	struct sbp2scsi_host_info *hi = scsi_id->hi;
@@ -2030,11 +2030,9 @@ static int sbp2_link_orb_command(struct 
 
 	SBP2_ORB_DEBUG("write to %s register, command orb %p",
 			last_orb ? "DOORBELL" : "ORB_POINTER", command_orb);
-	if (sbp2util_node_write_no_wait(scsi_id->ne, addr, data, length) < 0) {
+	if (sbp2util_node_write_no_wait(scsi_id->ne, addr, data, length))
 		SBP2_ERR("sbp2util_node_write_no_wait failed.\n");
-		return -EIO;
-	}
-	return 0;
+	/* We rely on SCSI EH to deal with _node_write_ failures. */
 }
 
 /*
drivers/ieee1394/sbp2.h +0 -5
Index: linux/drivers/ieee1394/sbp2.h
===================================================================
@@ -390,11 +390,6 @@ static int sbp2_logout_device(struct scs
 static int sbp2_handle_status_write(struct hpsb_host *host, int nodeid, int destid,
 				    quadlet_t *data, u64 addr, size_t length, u16 flags);
 static int sbp2_agent_reset(struct scsi_id_instance_data *scsi_id, int wait);
-static int sbp2_link_orb_command(struct scsi_id_instance_data *scsi_id,
-				 struct sbp2_command_info *command);
-static int sbp2_send_command(struct scsi_id_instance_data *scsi_id,
-			     struct scsi_cmnd *SCpnt,
-			     void (*done)(struct scsi_cmnd *));
 static unsigned int sbp2_status_to_sense_data(unchar *sbp2_status,
 					      unchar *sense_data);
 static void sbp2_parse_unit_directory(struct scsi_id_instance_data *scsi_id,
#3 lore raw

PATCH 3/6 ieee1394: sbp2: optimize DMA direction of command ORBs

In reply to Stefan Richter

Only the driver writes ORBs, the device just reads them. Therefore PCI_DMA_BIDIRECTIONAL can be replaced by PCI_DMA_TODEVICE which may be cheaper on some architectures.

---
 drivers/ieee1394/sbp2.c |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

TODO:
Convert sbp2 from PCI DMA to generic DMA mapping.

drivers/ieee1394/sbp2.c +8 -8
Index: linux/drivers/ieee1394/sbp2.c
===================================================================
@@ -492,7 +492,7 @@ static int sbp2util_create_command_orb_p
 		command->command_orb_dma =
 		    pci_map_single(hi->host->pdev, &command->command_orb,
 				   sizeof(struct sbp2_command_orb),
-				   PCI_DMA_BIDIRECTIONAL);
+				   PCI_DMA_TODEVICE);
 		SBP2_DMA_ALLOC("single command orb DMA");
 		command->sge_dma =
 		    pci_map_single(hi->host->pdev,
@@ -525,7 +525,7 @@ static void sbp2util_remove_command_orb_
 			/* Release our generic DMA's */
 			pci_unmap_single(host->pdev, command->command_orb_dma,
 					 sizeof(struct sbp2_command_orb),
-					 PCI_DMA_BIDIRECTIONAL);
+					 PCI_DMA_TODEVICE);
 			SBP2_DMA_FREE("single command orb DMA");
 			pci_unmap_single(host->pdev, command->sge_dma,
 					 sizeof(command->scatter_gather_element),
@@ -1982,7 +1982,7 @@ static void sbp2_link_orb_command(struct
 
 	pci_dma_sync_single_for_device(hi->host->pdev, command->command_orb_dma,
 				       sizeof(struct sbp2_command_orb),
-				       PCI_DMA_BIDIRECTIONAL);
+				       PCI_DMA_TODEVICE);
 	pci_dma_sync_single_for_device(hi->host->pdev, command->sge_dma,
 				       sizeof(command->scatter_gather_element),
 				       PCI_DMA_BIDIRECTIONAL);
@@ -2012,14 +2012,14 @@ static void sbp2_link_orb_command(struct
 		 */
 		pci_dma_sync_single_for_cpu(hi->host->pdev, last_orb_dma,
 					    sizeof(struct sbp2_command_orb),
-					    PCI_DMA_BIDIRECTIONAL);
+					    PCI_DMA_TODEVICE);
 		last_orb->next_ORB_lo = cpu_to_be32(command->command_orb_dma);
 		wmb();
 		/* Tells hardware that this pointer is valid */
 		last_orb->next_ORB_hi = 0;
 		pci_dma_sync_single_for_device(hi->host->pdev, last_orb_dma,
 					       sizeof(struct sbp2_command_orb),
-					       PCI_DMA_BIDIRECTIONAL);
+					       PCI_DMA_TODEVICE);
 		addr += SBP2_DOORBELL_OFFSET;
 		data[0] = 0;
 		length = 4;
@@ -2176,7 +2176,7 @@ static int sbp2_handle_status_write(stru
 		SBP2_DEBUG("Found status for command ORB");
 		pci_dma_sync_single_for_cpu(hi->host->pdev, command->command_orb_dma,
 					    sizeof(struct sbp2_command_orb),
-					    PCI_DMA_BIDIRECTIONAL);
+					    PCI_DMA_TODEVICE);
 		pci_dma_sync_single_for_cpu(hi->host->pdev, command->sge_dma,
 					    sizeof(command->scatter_gather_element),
 					    PCI_DMA_BIDIRECTIONAL);
@@ -2365,7 +2365,7 @@ static void sbp2scsi_complete_all_comman
 		command = list_entry(lh, struct sbp2_command_info, list);
 		pci_dma_sync_single_for_cpu(hi->host->pdev, command->command_orb_dma,
 					    sizeof(struct sbp2_command_orb),
-					    PCI_DMA_BIDIRECTIONAL);
+					    PCI_DMA_TODEVICE);
 		pci_dma_sync_single_for_cpu(hi->host->pdev, command->sge_dma,
 					    sizeof(command->scatter_gather_element),
 					    PCI_DMA_BIDIRECTIONAL);
@@ -2548,7 +2548,7 @@ static int sbp2scsi_abort(struct scsi_cm
 			pci_dma_sync_single_for_cpu(hi->host->pdev,
 						    command->command_orb_dma,
 						    sizeof(struct sbp2_command_orb),
-						    PCI_DMA_BIDIRECTIONAL);
+						    PCI_DMA_TODEVICE);
 			pci_dma_sync_single_for_cpu(hi->host->pdev,
 						    command->sge_dma,
 						    sizeof(command->scatter_gather_element),
#4 lore raw

PATCH 4/6 ieee1394: sbp2: safer initialization of status fifo

In reply to Stefan Richter

Sbp2's copy of the status fifo was cleared when management ORBs or new command ORBs were prepared. The latter had potential for a race condition if the block layer's soft IRQ and the 1394 LLD's interrupt handler ran on different CPUs. It would also yield wrong status if a command was completed with non-zero completion status before other commands that had zero completion status, and no new command was enqueued in the meantime.

Now, the status buffer is cleared right before it is written. Thus it ends up in the following simpler and safer access pattern:
- sbp2_alloc_device: allocates and implicitly clears once,
- sbp2_handle_status_write: clears, writes, and reads,
- sbp2_query_logins, sbp2_login_device, sbp2_reconnect_device: read.
The latter three do not race with sbp2_handle_status_write because of how the protocol works.

As a tiny optimization, the first two quadlets of the status never need to be cleared.

---
 drivers/ieee1394/sbp2.c |   80 +++++++++++++++-------------------------
 1 files changed, 30 insertions(+), 50 deletions(-)
drivers/ieee1394/sbp2.c +30 -50
Index: linux/drivers/ieee1394/sbp2.c
===================================================================
@@ -1182,7 +1182,6 @@ static int sbp2_query_logins(struct scsi
 			     "sbp2 query logins orb", scsi_id->query_logins_orb_dma);
 
 	memset(scsi_id->query_logins_response, 0, sizeof(struct sbp2_query_logins_response));
-	memset(&scsi_id->status_block, 0, sizeof(struct sbp2_status_block));
 
 	data[0] = ORB_SET_NODE_ID(hi->host->node_id);
 	data[1] = scsi_id->query_logins_orb_dma;
@@ -1278,7 +1277,6 @@ static int sbp2_login_device(struct scsi
 			     "sbp2 login orb", scsi_id->login_orb_dma);
 
 	memset(scsi_id->login_response, 0, sizeof(struct sbp2_login_response));
-	memset(&scsi_id->status_block, 0, sizeof(struct sbp2_status_block));
 
 	data[0] = ORB_SET_NODE_ID(hi->host->node_id);
 	data[1] = scsi_id->login_orb_dma;
@@ -1445,14 +1443,6 @@ static int sbp2_reconnect_device(struct 
 	sbp2util_packet_dump(scsi_id->reconnect_orb, sizeof(struct sbp2_reconnect_orb),
 			     "sbp2 reconnect orb", scsi_id->reconnect_orb_dma);
 
-	/*
-	 * Initialize status fifo
-	 */
-	memset(&scsi_id->status_block, 0, sizeof(struct sbp2_status_block));
-
-	/*
-	 * Ok, let's write to the target's management agent register
-	 */
 	data[0] = ORB_SET_NODE_ID(hi->host->node_id);
 	data[1] = scsi_id->reconnect_orb_dma;
 	sbp2util_cpu_to_be32_buffer(data, 8);
@@ -2069,11 +2059,6 @@ static int sbp2_send_command(struct scsi
 			     "sbp2 command orb", command->command_orb_dma);
 
 	/*
-	 * Initialize status fifo
-	 */
-	memset(&scsi_id->status_block, 0, sizeof(struct sbp2_status_block));
-
-	/*
 	 * Link up the orb, and ring the doorbell if needed
 	 */
 	sbp2_link_orb_command(scsi_id, command);
@@ -2114,12 +2099,14 @@ static unsigned int sbp2_status_to_sense
 /*
  * This function deals with status writes from the SBP-2 device
  */
-static int sbp2_handle_status_write(struct hpsb_host *host, int nodeid, int destid,
-				    quadlet_t *data, u64 addr, size_t length, u16 fl)
+static int sbp2_handle_status_write(struct hpsb_host *host, int nodeid,
+				    int destid, quadlet_t *data, u64 addr,
+				    size_t length, u16 fl)
 {
 	struct sbp2scsi_host_info *hi;
 	struct scsi_id_instance_data *scsi_id = NULL, *scsi_id_tmp;
 	struct scsi_cmnd *SCpnt = NULL;
+	struct sbp2_status_block *sb;
 	u32 scsi_status = SBP2_SCSI_STATUS_GOOD;
 	struct sbp2_command_info *command;
 	unsigned long flags;
@@ -2158,19 +2145,21 @@ static int sbp2_handle_status_write(stru
 	}
 
 	/*
-	 * Put response into scsi_id status fifo...
+	 * Put response into scsi_id status fifo buffer. The first two bytes
+	 * come in big endian bit order. Often the target writes only a
+	 * truncated status block, minimally the first two quadlets. The rest
+	 * is implied to be zeros.
 	 */
-	memcpy(&scsi_id->status_block, data, length);
+	sb = &scsi_id->status_block;
+	memset(sb->command_set_dependent, 0, sizeof(sb->command_set_dependent));
+	memcpy(sb, data, length);
+	sbp2util_be32_to_cpu_buffer(sb, 8);
 
 	/*
-	 * Byte swap first two quadlets (8 bytes) of status for processing
+	 * Handle command ORB status here if necessary. First, need to match
+	 * status with command.
 	 */
-	sbp2util_be32_to_cpu_buffer(&scsi_id->status_block, 8);
-
-	/*
-	 * Handle command ORB status here if necessary. First, need to match status with command.
-	 */
-	command = sbp2util_find_command_for_orb(scsi_id, scsi_id->status_block.ORB_offset_lo);
+	command = sbp2util_find_command_for_orb(scsi_id, sb->ORB_offset_lo);
 	if (command) {
 
 		SBP2_DEBUG("Found status for command ORB");
@@ -2185,7 +2174,8 @@ static int sbp2_handle_status_write(stru
 		outstanding_orb_decr;
 
 		/*
-		 * Matched status with command, now grab scsi command pointers and check status
+		 * Matched status with command, now grab scsi command pointers
+		 * and check status.
 		 */
 		SCpnt = command->Current_SCpnt;
 		spin_lock_irqsave(&scsi_id->sbp2_command_orb_lock, flags);
@@ -2193,28 +2183,22 @@ static int sbp2_handle_status_write(stru
 		spin_unlock_irqrestore(&scsi_id->sbp2_command_orb_lock, flags);
 
 		if (SCpnt) {
-
 			/*
-			 * See if the target stored any scsi status information
+			 * See if the target stored any scsi status information.
 			 */
-			if (STATUS_GET_LENGTH(scsi_id->status_block.ORB_offset_hi_misc) > 1) {
-				/*
-				 * Translate SBP-2 status to SCSI sense data
-				 */
+			if (STATUS_GET_LENGTH(sb->ORB_offset_hi_misc) > 1) {
 				SBP2_DEBUG("CHECK CONDITION");
-				scsi_status = sbp2_status_to_sense_data((unchar *)&scsi_id->status_block, SCpnt->sense_buffer);
+				scsi_status = sbp2_status_to_sense_data(
+					(unchar *)sb, SCpnt->sense_buffer);
 			}
 
 			/*
-			 * Check to see if the dead bit is set. If so, we'll have to initiate
-			 * a fetch agent reset.
+			 * Check to see if the dead bit is set. If so, we'll
+			 * have to initiate a fetch agent reset.
 			 */
-			if (STATUS_GET_DEAD_BIT(scsi_id->status_block.ORB_offset_hi_misc)) {
-
-				/*
-				 * Initiate a fetch agent reset.
-				 */
-				SBP2_DEBUG("Dead bit set - initiating fetch agent reset");
+			if (STATUS_GET_DEAD_BIT(sb->ORB_offset_hi_misc)) {
+				SBP2_DEBUG("Dead bit set - "
+					   "initiating fetch agent reset");
                                 sbp2_agent_reset(scsi_id, 0);
 			}
 
@@ -2235,21 +2219,17 @@ static int sbp2_handle_status_write(stru
 		spin_unlock_irqrestore(&scsi_id->sbp2_command_orb_lock, flags);
 
 	} else {
-
 		/*
 		 * It's probably a login/logout/reconnect status.
 		 */
-		if ((scsi_id->login_orb_dma == scsi_id->status_block.ORB_offset_lo) ||
-		    (scsi_id->query_logins_orb_dma == scsi_id->status_block.ORB_offset_lo) ||
-		    (scsi_id->reconnect_orb_dma == scsi_id->status_block.ORB_offset_lo) ||
-		    (scsi_id->logout_orb_dma == scsi_id->status_block.ORB_offset_lo)) {
+		if ((sb->ORB_offset_lo == scsi_id->reconnect_orb_dma) ||
+		    (sb->ORB_offset_lo == scsi_id->login_orb_dma) ||
+		    (sb->ORB_offset_lo == scsi_id->query_logins_orb_dma) ||
+		    (sb->ORB_offset_lo == scsi_id->logout_orb_dma))
 			atomic_set(&scsi_id->sbp2_login_complete, 1);
-		}
 	}
 
 	if (SCpnt) {
-
-		/* Complete the SCSI command. */
 		SBP2_DEBUG("Completing SCSI command");
 		sbp2scsi_complete_command(scsi_id, scsi_status, SCpnt,
 					  command->Current_done);
#5 lore raw

PATCH 5/6 ieee1394: sbp2: more checks of status block

In reply to Stefan Richter

- Add checks for the (very unlikely) cases that the target writes too little or too much status data or writes unsolicited status.
- Indicate that these and similar conditions are unlikely().
- Check the 'resp' and 'sbp_status' fields for possible failure status.
- Slightly optimize access macros for the status block bitfields.
- Unify a few related log messages.

---
 drivers/ieee1394/sbp2.c |   70 +++++++++++++++++-----------------------
 drivers/ieee1394/sbp2.h |   14 ++++----
 2 files changed, 38 insertions(+), 46 deletions(-)

TODO:
Check if status' 'src'==1, then withhold the respective ORB from reuse until status for any subsequent ORB was received.

drivers/ieee1394/sbp2.h +8 -6
Index: linux/drivers/ieee1394/sbp2.h
===================================================================
@@ -180,12 +180,14 @@ struct sbp2_unrestricted_page_table {
 
 #define SBP2_SCSI_STATUS_SELECTION_TIMEOUT	0xff
 
-#define STATUS_GET_ORB_OFFSET_HI(value)         (value & 0xffff)
-#define STATUS_GET_SBP_STATUS(value)            ((value >> 16) & 0xff)
-#define STATUS_GET_LENGTH(value)                ((value >> 24) & 0x7)
-#define STATUS_GET_DEAD_BIT(value)              ((value >> 27) & 0x1)
-#define STATUS_GET_RESP(value)                  ((value >> 28) & 0x3)
-#define STATUS_GET_SRC(value)                   ((value >> 30) & 0x3)
+#define STATUS_GET_SRC(value)			(((value) >> 30) & 0x3)
+#define STATUS_GET_LEN(value)			(((value) >> 24) & 0x7)
+#define STATUS_GET_ORB_OFFSET_HI(value)		((value) & 0x0000ffff)
+#define STATUS_TEST_D(value)			((value) & 0x08000000)
+/* test 'resp' | 'sbp2_status' */
+#define STATUS_TEST_RS(value)			((value) & 0x30ff0000)
+/* test 'resp' | 'dead' | 'sbp2_status' */
+#define STATUS_TEST_RDS(value)			((value) & 0x38ff0000)
 
 struct sbp2_status_block {
 	u32 ORB_offset_hi_misc;
drivers/ieee1394/sbp2.c +30 -40

154 lines not on this page · raw message

#6 lore raw

PATCH 6/6 ieee1394: sbp2: convert sbp2util_down_timeout to waitqueue

In reply to Stefan Richter

The waitqueue API is used to replace a custom wait mechanism. Only one global waitqueue (instead of per-device waitqueues or completions) is added because there is usually just one waiter.

---
 drivers/ieee1394/sbp2.c |   51 +++++++++++++++++-----------------------
 drivers/ieee1394/sbp2.h |    4 +--
 2 files changed, 24 insertions(+), 31 deletions(-)
drivers/ieee1394/sbp2.h +2 -2
Index: linux/drivers/ieee1394/sbp2.h
===================================================================
@@ -320,9 +320,9 @@ struct scsi_id_instance_data {
 	u64 status_fifo_addr;
 
 	/*
-	 * Variable used for logins, reconnects, logouts, query logins
+	 * Waitqueue flag for logins, reconnects, logouts, query logins
 	 */
-	atomic_t sbp2_login_complete;
+	int access_complete:1;
 
 	/*
 	 * Pool of command orbs, so we can have more than overlapped command per id
drivers/ieee1394/sbp2.c +22 -29

133 lines not on this page · raw message