Advanced TCA SCSI Disk Hotswap

linux-scsi,

Attached is an update of the Advanced TCA disk hotswap driver to provide disk hotswap support for the Linux Kernel 2.5.43. Hotswap targets include both SCSI and FibreChannel. Note this support is generic in that it will work in a typical PC system with SCSI or FibreChannel disks.

I've not yet ported or made suggested changes to the ga mapping driver yet. I'll work on that next.

As always, the latest Advanced TCA hotswap patches are available for Linux 2.4 and Linux 2.5 at the sourceforge site: (please note ga mapper hasn't yet been ported to 2.5)
http://sourceforge.net/project/showfiles.php?group_id=64580

Changes from kernel 2.4.19 to 2.5.44 release:
* Complete port to linux kernel 2.5.44
* Locking of scsi_host->host_queue structure when traversing scsi device list (suggested by Alan Cox)
- This keeps multiple hotswap commands from corrupting the SCSI device
list and
causing system failure
* ioctls deleted and replaced by ramfs scsi_hotswap_fs (suggested by Greg KH)
- removes need for character device node
- usage information available by read() call to files
- available over nfs exported directories
* local file-scope functions declared as static instead of global (suggested by Greg KH)
* code reformatted to fit on 80 character wide screen and to follow linux general kernel instead of scsi subsystem format
* KERNEL_SYSCALLS define removed from header (suggested by Greg KH)
* list of include files optimized to only include files as needed
* Removed direct call from scsi.c into scsi_hotswap_init and used module_init instead
* Made modular so can be compiled as a module or in the kernel directly
* Fixed bug where scsi_hotswap_insert_by_scsi_id didn't properly add a device in some cases

diff -uNr linux-2.5.44-qla/MAINTAINERS linux-2.5.44-scsi-hotswap/MAINTAINERS
@@ -1418,6 +1418,12 @@
 W:	http://www.kernel.dk
 S:	Maintained
 
+SCSI HOTSWAP DRIVER
+P:	Steven Dake
+M:	[email protected]
+L:	[email protected]
+S:	Maintained
+
 SCSI SG DRIVER
 P:	Doug Gilbert
 M:	[email protected]

diff -uNr linux-2.5.44-qla/drivers/scsi/Config.help linux-2.5.44-scsi-hotswap/drivers/scsi/Config.help
@@ -34,6 +34,17 @@
   is located on a SCSI disk. In this case, do not compile the driver
   for your SCSI host adapter (below) as a module either.
 
+CONFIG_SCSIFCHOTSWAP
+ If you want to support the ability to include the hotswap FibreChannel
+ and SCSI driver, please say yes here.  Hotswap can then be executed
+ through a ramfs interface which provides better error reporting.
+
+ Further, this interface supports insertion and removal by WWN for
+ FibreChannel drivers which support this feature.
+
+ The only FibreChannel driver that supports this feature is Qlogic V6
+ with a specific support patch.
+
 CONFIG_SD_EXTRA_DEVS
   This controls the amount of additional space allocated in tables for
   drivers that are loaded as modules after the kernel is booted.  In

diff -uNr linux-2.5.44-qla/drivers/scsi/Config.in linux-2.5.44-scsi-hotswap/drivers/scsi/Config.in
@@ -1,5 +1,6 @@
 comment 'SCSI support type (disk, tape, CD-ROM)'
 
+dep_tristate '  SCSI hotswap support' CONFIG_SCSIFCHOTSWAP $CONFIG_SCSI
 dep_tristate '  SCSI disk support' CONFIG_BLK_DEV_SD $CONFIG_SCSI
 
 if [ "$CONFIG_BLK_DEV_SD" != "n" ]; then

diff -uNr linux-2.5.44-qla/drivers/scsi/Makefile linux-2.5.44-scsi-hotswap/drivers/scsi/Makefile
@@ -117,6 +117,7 @@
 
 obj-$(CONFIG_ARCH_ACORN)	+= ../acorn/scsi/
 
+obj-$(CONFIG_SCSIFCHOTSWAP)	+= hotswap.o
 obj-$(CONFIG_CHR_DEV_ST)	+= st.o
 obj-$(CONFIG_CHR_DEV_OSST)	+= osst.o
 obj-$(CONFIG_BLK_DEV_SD)	+= sd_mod.o

diff -uNr linux-2.5.44-qla/drivers/scsi/hosts.c linux-2.5.44-scsi-hotswap/drivers/scsi/hosts.c
@@ -371,6 +371,7 @@
 	scsi_hosts_registered++;
 
 	spin_lock_init(&shost->default_lock);
+	spin_lock_init(&shost->host_queue_lock);
 	scsi_assign_lock(shost, &shost->default_lock);
 	atomic_set(&shost->host_active,0);
 

diff -uNr linux-2.5.44-qla/drivers/scsi/hosts.h linux-2.5.44-scsi-hotswap/drivers/scsi/hosts.h
@@ -264,6 +264,14 @@
      */
     int (* bios_param)(Disk *, struct block_device *, int []);
 
+#ifdef CONFIG_SCSIFCHOTSWAP
+	/*
+	 * Used to determine the id to send the inquiry command to during
+	 * hot additions of FibreChannel targets
+	 */
+	int (*get_scsi_info_from_wwn)(int mode, unsigned long long wwn, int *host, int *channel, int *lun, int *id);
+#endif
+
     /*
      * This determines if we will use a non-interrupt driven
      * or an interrupt driven scheme,  It is set to the maximum number

@@ -374,6 +382,7 @@
      */
     struct list_head      sh_list;
     Scsi_Device           * host_queue;
+    spinlock_t		  host_queue_lock;
     struct list_head	  all_scsi_hosts;
     struct list_head	  my_devices;
 

diff -uNr linux-2.5.44-qla/drivers/scsi/scsi.c linux-2.5.44-scsi-hotswap/drivers/scsi/scsi.c
@@ -413,6 +413,9 @@
 				 * allow us to more easily figure out whether we should
 				 * do anything here or not.
 				 */
+
+				spin_lock (&host->host_queue_lock);
+
 				for (SDpnt = host->host_queue;
 				     SDpnt;
 				     SDpnt = SDpnt->next) {

@@ -430,6 +433,9 @@
                                                 break;
                                         }
 				}
+
+				spin_unlock (&host->host_queue_lock);
+
 				if (SDpnt) {
 					/*
 					 * Some other device in this cluster is busy.

@@ -1694,6 +1700,8 @@
 		len += size;
 		pos = begin + len;
 #endif
+		spin_lock (&HBA_ptr->host_queue_lock);
+
 		for (scd = HBA_ptr->host_queue; scd; scd = scd->next) {
 			proc_print_scsidevice(scd, buffer, &size, len);
 			len += size;

@@ -1706,6 +1714,8 @@
 			if (pos > offset + length)
 				goto stop_output;
 		}
+
+		spin_unlock (&HBA_ptr->host_queue_lock);
 	}
 
 stop_output:

@@ -1864,6 +1874,8 @@
 		if (!HBA_ptr)
 			goto out;
 
+		spin_lock (&HBA_ptr->host_queue_lock);
+
 		for (scd = HBA_ptr->host_queue; scd; scd = scd->next) {
 			if ((scd->channel == channel
 			     && scd->id == id

@@ -1872,6 +1884,8 @@
 			}
 		}
 
+		spin_unlock (&HBA_ptr->host_queue_lock);
+
 		err = -ENOSYS;
 		if (scd)
 			goto out;	/* We do not yet support unplugging */

@@ -1910,6 +1924,8 @@
 		if (!HBA_ptr)
 			goto out;
 
+		spin_lock (&HBA_ptr->host_queue_lock);
+
 		for (scd = HBA_ptr->host_queue; scd; scd = scd->next) {
 			if ((scd->channel == channel
 			     && scd->id == id

@@ -1918,6 +1934,8 @@
 			}
 		}
 
+		spin_unlock (&HBA_ptr->host_queue_lock);
+
 		if (scd == NULL)
 			goto out;	/* there is no such device attached */
 

@@ -1951,9 +1969,14 @@
 			if (scd->prev != NULL)
 				scd->prev->next = scd->next;
 
+			spin_lock (&HBA_ptr->host_queue_lock);
+
 			if (HBA_ptr->host_queue == scd) {
 				HBA_ptr->host_queue = scd->next;
 			}
+
+			spin_unlock (&HBA_ptr->host_queue_lock);
+
 			blk_cleanup_queue(&scd->request_queue);
 			if (scd->inquiry)
 				kfree(scd->inquiry);

@@ -1997,11 +2020,15 @@
 
 	for (shpnt = scsi_host_get_next(NULL); shpnt;
 	     shpnt = scsi_host_get_next(shpnt)) {
+		spin_lock (&shpnt->host_queue_lock);
+
 		for (SDpnt = shpnt->host_queue; SDpnt;
 		     SDpnt = SDpnt->next) {
 			if (tpnt->detect)
 				SDpnt->attached += (*tpnt->detect) (SDpnt);
 		}
+
+		spin_unlock (&shpnt->host_queue_lock);
 	}
 
 	/*

@@ -2017,6 +2044,8 @@
 	 */
 	for (shpnt = scsi_host_get_next(NULL); shpnt;
 	     shpnt = scsi_host_get_next(shpnt)) {
+		spin_lock (&shpnt->host_queue_lock);
+
 		for (SDpnt = shpnt->host_queue; SDpnt;
 		     SDpnt = SDpnt->next) {
 			if (tpnt->attach)

@@ -2032,6 +2061,8 @@
 					out_of_space = 1;
 			}
 		}
+
+		spin_unlock (&shpnt->host_queue_lock);
 	}
 
 	/*

@@ -2068,6 +2099,8 @@
 
 	for (shpnt = scsi_host_get_next(NULL); shpnt;
 	     shpnt = scsi_host_get_next(shpnt)) {
+		spin_lock (&shpnt->host_queue_lock);
+
 		for (SDpnt = shpnt->host_queue; SDpnt;
 		     SDpnt = SDpnt->next) {
 			if (tpnt->detach)

@@ -2084,6 +2117,8 @@
 				scsi_release_commandblocks(SDpnt);
 			}
 		}
+
+		spin_unlock (&shpnt->host_queue_lock);
 	}
 	/*
 	 * Extract the template from the linked list.

@@ -2154,6 +2189,8 @@
 	for (shpnt = scsi_host_get_next(NULL); shpnt;
 	     shpnt = scsi_host_get_next(shpnt)) {
 		printk(KERN_INFO "h:c:t:l (dev sect nsect cnumsec sg) (ret all flg) (to/cmd to ito) cmd snse result\n");
+		spin_lock (&shpnt->host_queue_lock);
+
 		for (SDpnt = shpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {
 			for (SCpnt = SDpnt->device_queue; SCpnt; SCpnt = SCpnt->next) {
 				/*  (0) h:c:t:l (dev sect nsect cnumsec sg) (ret all flg) (to/cmd to ito) cmd snse result %d %x      */

@@ -2185,6 +2222,8 @@
 				       SCpnt->result);
 			}
 		}
+
+		spin_unlock (&shpnt->host_queue_lock);
 	}
 #endif	/* CONFIG_SCSI_LOGGING */ /* } */
 }

diff -uNr linux-2.5.44-qla/drivers/scsi/scsi_error.c linux-2.5.44-scsi-hotswap/drivers/scsi/scsi_error.c
@@ -202,6 +202,8 @@
 	int devices_failed = 0;
 
 
+	spin_lock (&shost->host_queue_lock);
+
 	for (sdev = shost->host_queue; sdev; sdev = sdev->next) {
 		for (scmd = sc_list; scmd; scmd = scmd->bh_next) {
 			if (scmd->device == sdev) {

@@ -227,6 +229,8 @@
 		}
 	}
 
+	spin_unlock (&shost->host_queue_lock);
+
 	SCSI_LOG_ERROR_RECOVERY(2, printk("Total of %d commands on %d"
 					  " devices require eh work\n",
 				  total_failures, devices_failed));

@@ -247,6 +251,8 @@
 	Scsi_Device *sdev;
 	Scsi_Cmnd *scmd;
 
+	spin_lock (&shost->host_queue_lock);
+
 	for (found = 0, sdev = shost->host_queue; sdev; sdev = sdev->next) {
 		for (scmd = sdev->device_queue; scmd; scmd = scmd->next) {
 			if (scsi_eh_eflags_chk(scmd, SCSI_EH_CMD_ERR)) {

@@ -283,6 +289,8 @@
 		}
 	}
 
+	spin_unlock (&shost->host_queue_lock);
+
 	SCSI_LOG_ERROR_RECOVERY(1, scsi_eh_prt_fail_stats(*sc_list, shost));
 
 	if (shost->host_failed != found)

@@ -962,6 +970,8 @@
 
 	SCSI_LOG_ERROR_RECOVERY(3, printk("%s: Trying BDR\n", __FUNCTION__));
 
+	spin_lock (&shost->host_queue_lock);
+
 	for (sdev = shost->host_queue; sdev; sdev = sdev->next) {
 		for (scmd = sc_todo; scmd; scmd = scmd->bh_next)
 			if ((scmd->device == sdev) &&

@@ -985,6 +995,7 @@
 							scsi_eh_finish_cmd(scmd, shost);
 					}
 	}
+	spin_unlock (&shost->host_queue_lock);
 
 	return shost->host_failed;
 }

@@ -1016,11 +1027,15 @@
 		/*
 		 * Mark all affected devices to expect a unit attention.
 		 */
+		spin_lock (&scmd->host->host_queue_lock);
+
 		for (sdev = scmd->host->host_queue; sdev; sdev = sdev->next)
 			if (scmd->channel == sdev->channel) {
 				sdev->was_reset = 1;
 				sdev->expecting_cc_ua = 1;
 			}
+
+		spin_unlock (&scmd->host->host_queue_lock);
 	}
 	return rtn;
 }

@@ -1052,11 +1067,13 @@
 		/*
 		 * Mark all affected devices to expect a unit attention.
 		 */
+		spin_lock (&scmd->host->host_queue_lock);
 		for (sdev = scmd->host->host_queue; sdev; sdev = sdev->next)
 			if (scmd->channel == sdev->channel) {
 				sdev->was_reset = 1;
 				sdev->expecting_cc_ua = 1;
 			}
+		spin_unlock (&scmd->host->host_queue_lock);
 	}
 	return rtn;
 }

@@ -1475,6 +1492,8 @@
 	 * requests are started.
 	 */
 	spin_lock_irqsave(shost->host_lock, flags);
+	spin_lock (&shost->host_queue_lock);
+
 	for (sdev = shost->host_queue; sdev; sdev = sdev->next) {
 		request_queue_t *q = &sdev->request_queue;
 

@@ -1487,6 +1506,8 @@
 
 		q->request_fn(q);
 	}
+
+	spin_unlock (&shost->host_queue_lock);
 	spin_unlock_irqrestore(shost->host_lock, flags);
 }
 

diff -uNr linux-2.5.44-qla/drivers/scsi/scsi_lib.c linux-2.5.44-scsi-hotswap/drivers/scsi/scsi_lib.c
@@ -261,6 +261,8 @@
 	if (SDpnt->single_lun && blk_queue_empty(q) && SDpnt->device_busy ==0) {
 		request_queue_t *q;
 
+		spin_lock (&SHpnt->host_queue_lock);
+
 		for (SDpnt = SHpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {
 			if (((SHpnt->can_queue > 0)
 			     && (SHpnt->host_busy >= SHpnt->can_queue))

@@ -273,6 +275,8 @@
 			q = &SDpnt->request_queue;
 			q->request_fn(q);
 		}
+
+		spin_unlock (&SHpnt->host_queue_lock);
 	}
 
 	/*

@@ -285,6 +289,7 @@
 	 */
 	all_clear = 1;
 	if (SHpnt->some_device_starved) {
+		spin_lock (&SHpnt->host_queue_lock);
 		for (SDpnt = SHpnt->host_queue; SDpnt; SDpnt = SDpnt->next) {
 			request_queue_t *q;
 			if ((SHpnt->can_queue > 0 && (SHpnt->host_busy >= SHpnt->can_queue))

@@ -299,6 +304,7 @@
 			q->request_fn(q);
 			all_clear = 0;
 		}
+		spin_unlock (&SHpnt->host_queue_lock);
 		if (SDpnt == NULL && all_clear) {
 			SHpnt->some_device_starved = 0;
 		}

@@ -1038,8 +1044,13 @@
 
 	SHpnt->host_self_blocked = FALSE;
 	/* Now that we are unblocked, try to start the queues. */
+
+	spin_lock (&SHpnt->host_queue_lock);
+
 	for (SDloop = SHpnt->host_queue; SDloop; SDloop = SDloop->next)
 		scsi_queue_next_request(&SDloop->request_queue, NULL);
+
+	spin_unlock (&SHpnt->host_queue_lock);
 }
 
 /*

@@ -1066,12 +1077,17 @@
 void scsi_report_bus_reset(struct Scsi_Host * SHpnt, int channel)
 {
 	Scsi_Device *SDloop;
+
+	spin_lock (&SHpnt->host_queue_lock);
+
 	for (SDloop = SHpnt->host_queue; SDloop; SDloop = SDloop->next) {
 		if (channel == SDloop->channel) {
 			SDloop->was_reset = 1;
 			SDloop->expecting_cc_ua = 1;
 		}
 	}
+
+	spin_unlock (&SHpnt->host_queue_lock);
 }
 
 /*

diff -uNr linux-2.5.44-qla/drivers/scsi/scsi_scan.c linux-2.5.44-scsi-hotswap/drivers/scsi/scsi_scan.c
@@ -530,6 +530,8 @@
 		/*
 		 * Add it to the end of the shost->host_queue list.
 		 */
+		spin_lock (&shost->host_queue_lock);
+
 		if (shost->host_queue != NULL) {
 			sdev->prev = shost->host_queue;
 			while (sdev->prev->next != NULL)

@@ -538,6 +540,8 @@
 		} else
 			shost->host_queue = sdev;
 
+		spin_unlock (&shost->host_queue_lock);
+
 	}
 	return (sdev);
 }

@@ -554,8 +558,12 @@
 {
 	if (sdev->prev != NULL)
 		sdev->prev->next = sdev->next;
-	else
+	else {
+		spin_lock (&sdev->host->host_queue_lock);
 		sdev->host->host_queue = sdev->next;
+		spin_unlock (&sdev->host->host_queue_lock);
+	}
+
 	if (sdev->next != NULL)
 		sdev->next->prev = sdev->prev;
 

I looked at this briefly last night. The patch seems to come in three pieces:

1) Add list locking to manipulations of the host device list
2) Add a new filesystem type for exposing SCSI information
3) add a lot more methods for adding and removing SCSI devices.

1) seems to be something we should have

I'm dubious about 2). Could you explain why you need a new fs interface as opposed to using driverfs?

3) duplicates existing functionality and the new bits it does add could be done using a hotplug framework.

For all the removes, if you exposed the information you're looking for (FC wwn) in driverfs, you could use the existing remove interface.

For the adds, all of this would be finessed by having the FC event that detected a new device on the fibre trigger a hotplug event. You could then have the hotplug trigger the existing add device interface to make the component configure automatically (hotplug on fibre events like this has been a hot item for a while, so having a general interface would be most welcome).

If you must delay configuration and do it by wwn later, you could capture the wwns to host,channel,id,lun mappings in a table on the hotplug event (and not configure the device) and use the information for later configuration using the existing interfaces.

James

James,
Responses below:

The main reason I use a new fs interface is this was a port of a 2.4 driver (and as you know 2.4 doesn't have driverfs:) Since my customers that are using this software are currently on 2.4, that is my main concern. I had wanted to see this code reach the 2.5 mainline so I could maintain it in the main kernel instead of as seperate patches for each kernel rev.

I plan to produce a now patch that dumps the filesystem interface and replaces it with driverfs files in /sys/bus/scsi. These things take time, but I hope to be finished by October 25th.

The current remove interface is unmaintained, doesn't contain locking, and requires laborious string processing resulting in slow results. Further there is no usage information (which means the usage must come by looking at drivers/scsi/scsi.c which is beyond most typical users).

Another problem is performance:

Imagine scanning each disk in driverfs looking at its WWN attribute (if it has one) until a match is found. Assume there are 16 FC devices. That is several hundred syscalls just to complete one hotswap operation.

The method in the patch takes only two syscalls (open and write) to complete a hotswap operation, independent of the number of FibreChannel devices. Of course, the scanning still occurs, but through a very fast function call interface (1 call per adaptor) as opposed to a slower syscall interface.

This requires the adaptor to maintain a mapping of WWNs to SCSI IDs, however, this is already required by most FibreChannel firmware I've seen (and hence is available in the driver database already).

Hotplugs on FibreChannel don't trigger "events". What they can do is LIP (loop initialization procedure) if the device has been configured in it's SCSI code pages to do such a thing. Since this is device specific I'd hate to rely on it for hotswap.

I do like your idea, though, and if I have a chance this year, I'll implement automatic hotplug on new devices found during a LIP.

I think this would be too slow. 10 msec for my entire hotswap is available. If you calculate 2msec for the actual hotswap disk operation, that leaves 8 msec for the rest of the mess. Scanning through tables or scanning tens or hundreds of files through hundreds of syscalls may betoo slow.

Thanks
-steve

OK, that's good, thanks.

It is maintained (well, I was planning on looking after it). The locking can be added (the 1st part of your patch). It does two in kernel strncmps. That's not really slow by most definitions.

I don't really think it's the job of the kernel to conatin usage information. That's the job of the user level documentation.

Why is speed so important?

There will be a point where for a large number of drivers, a linear scan even in the kernel will be slower than a good DB lookup in userspace.

They don't now, but they should. The LIP protocol makes the FC driver aware of the gain or loss of devices. This should be communicated to the mid-layer and then trigger a hotplug event. Someone needs to write this, I was just wondering if you might.

Where does the 10ms figure come from?

James

James
Some responses below:

The locking most definately needs to be added to the kernel. I'm surprised the original patch didn't contain any locking, but then again, my first patch didn't either :)

I've gotten mixed feedback on this. I'll add you to the list that doesn't like this.

perhaps it should be removed (even though it takes up minimal memory).

Telecoms and Datacoms have told me in numerous conversations that a hotswap operation should occur in 20msec. I've arbitrarily set 10msec as my target to ensure that I meet the worse-case bus-is-loaded responses during scans, etc.

I can't mention the names of the telecoms, but several with 10000+ employees have mentioned it.

This may be true, but most systems will only have at most 4-5 devices. Theres only so much room on PCI for FC devices :)

I like the idea and it was something I was considering for early next year. Its driver dependent and until a FC driver is in the kernel, theres not much point yet :)

Keep in mind also that a LIP is not always generated on an insertion and isn't generated on a removal at all. This makes insertion easy but removal still requires user intervention.

In Advanced TCA (what spawned this work) a button is pressed to indicate hotswap removal which makes for easy detection of hotswap events. This is why there are kernel interfaces for removal and insertion (so a kernel driver can be written to detect the button press and remove the devices from the os data structures and then light a blue led indicating safe for removal).

See above

Thanks James for reading the code and giving comments!

On Thu, 24 Oct 2002, Steven Dake wrote:

| James
| Some responses below:
|
| James Bottomley wrote:
|
| >[email protected] said:

| >I don't really think it's the job of the kernel to conatin usage information.
| >That's the job of the user level documentation.
| >
| >
| I've gotten mixed feedback on this.  I'll add you to the list that
| doesn't like this.

add me to that list also.

| perhaps it should be removed (even though it takes up minimal memory). yes, i agree.

| >>Imagine scanning each disk in driverfs looking at its WWN attribute
| >>(if  it has one) until a match is found.  Assume there are 16 FC
| >>devices.  That is  several hundred syscalls just to complete one
| >>hotswap operation.
| >>
| >>
| >
| >Why is speed so important?
| >
| >
| Telecoms and Datacoms have told me in numerous conversations that a hotswap
| operation should occur in 20msec.  I've arbitrarily set 10msec as my
| target to
| ensure that I meet the worse-case bus-is-loaded responses during scans, etc.
|
| I can't mention the names of the telecoms, but several with 10000+ employees
| have mentioned it.

| >>I think this would be too slow.  10 msec for my entire hotswap is
| >>available.  If you calculate 2msec for the actual hotswap disk
| >>operation, that leaves 8 msec for the rest of the mess.  Scanning
| >>through tables or scanning tens or hundreds of files through hundreds
| >>of  syscalls may betoo slow.
| >>
| >
| >Where does the 10ms figure come from?
| >
| See above

I've already ask Steve about this and received his answers. Can't say that I agree with them though, so I asked someone from a Telecom Equipment Mfr. about this. He said that it's just for equipment testing, where technicians verify that hotswap works, and they are impatient to wait, so they practice surprise removal instead of coordinated removal. He doesn't think that's how it's actually done out in the field, just in test labs.

Preface question: does cPCI support surprise removal (in the PICMG specs, not in some implementation)? I know that PCI hotplug doesn't support surprise removal, only "coordinated" removal.

So the question that has to be answered IMO is: do we want to support surprise removal for something like manufacturing test, which doesn't abide by the coordinated removal protocol?

or: Do we have to support surprise removal, only because it can't be prevented? I expect that this is the case, but I still don't see or understand the 20 ms time requirement.

~Randy

PICMG 2.12 doesn't support surprise removal (the hardware does, the software doesn't). The latch must first be popped, then the user must wait for the blue led. If the blue led isn't lit, the operating system isn't ready for the board to be removed.

This said, operators are paid 10 bucks an hour to replace boards and you know how that goes. :)

For Compact PCI, the surprise removal rate is about 100 msec. This is as fast as the user can rip the board out of a chassis, meaning if you can light the blue led in less then 100 msec it doesn't matter if the extraction is a surprise or not.

For Advanced TCA, there isn't a "latch" required to unpop before removing the board. For Compact PCI, the latch must be popped, allowing a signal to be sent to the board. For ATCA, a button is pressed (which is a major complaint of Advanced TCA, boards can be removed without any signaling to the OS that the board is being removed). I'm not sure what the PICMG3 foks are going to do about that problem. I'm assuming they are going to rework the enumeration of the hotswap event to be driven by extracting the board instead of by a button.

In this case, extremely fast hotswap times are required, because the board can be removed very fast in Advanced TCA (vs the latched method of Compact PCI).

Perhaps this is where the timing constraints originate.

No, according to PICMG 2.1 R2.0, suprise removal is "non-compliant".

I've not implemented it yet, but I'm pretty sure I can detect surprise extractions in my cPCI driver. The only thing holding me back at the moment is that there's no clear way to report this status change via pcihpfs without doing something a bit funky like reporting "-1" in the "adapter" node.

Scott

Scott Murray
SOMA Networks, Inc.
Toronto, Ontario
e-mail: [email protected]

Why would you need to report anything other than if the card is present or not? What would a "supprise" removal cause you to do differently? Hm, well I guess we should be extra careful in trying to shut down any driver bound to that card...

thanks,

greg k-h

Montavista has discussed at length Compact PCI hotswap using surprise removal events.

The key feature of any hotswap operation that happens in a surprise fashion is that the device driver might want a hint that the hardware is no longer present so it can immediatly dump its buffers/io maps/etc and totally stop accessing the device. An expected removal, on the other hand, would give the device driver time to flush its buffers (for example a scsi driver could dump its outstanding queued scsi messages). Once the driver is done accessing the device, the blue led on the CompactPCI board can be lit and it can be removed.

This is the main difference. Since the driver model of Linux doesn't support a surprise extract method call for drivers, I don't think its been implemented here. Further the drivers must be modified to actually use the hint instead of doing its normal shutdown operation.

Surprise extraction is not a simple problem especially to ensure the device drivers exit cleanly without dumping more data on the PCI bus to a PCI device that may not exist.

Thanks!
-steve

Wrong. The _only_ supported method so far has been surprise removal. For years now. This happens every day in the land of CardBus, which was the first "PCI" hotplug implementation in the Linux kernel. PCI HotPlug introduces a new, non-surprise removal.

Thus, the current model should be assumed to be surprise removal, and you need an additional notification from the system if a "nice" removal is about to occur.

PCI is electrically safe. Reads to non-existent areas return 0xffffffff, etc. Take a look at net drivers some day, we have been handling this for years.

Surprise removal is actually easier from many perspectives -- you don't have to worry about quiescing the hardware, you simply have to error out all I/Os, and clean up the kernel structures that are left behind (host info, device info, etc.). The non-surprise removal is more annoying, in that you could potentially have an indefinite wait (and must actively avoid such a situation) while shutting down the hardware, completing I/Os, etc.

Jeff

Thats primarily about resource handling orders. Making sure we don't release the claimed pci resources in the driver until the driver itself is sure it has shut up.

I'm doing suprise removal ok with the thinkpad 600 (in 2.4 with some limits due to the 2.4 pci layer). Network stuff seems to be fine. The block layer has major issues.

Thinking about it a bit more, my idea to use -1 is indeed unnecessary, since userspace code can check if the adapter node changes to 0 before it itself writes a 0 to it. If multiple users/software play around with the nodes at the same time, they'll get what's coming to them...

Yeah, as Steven mentions in his reply, Linux drivers don't handle this well at the moment.

Scott

Scott Murray
SOMA Networks, Inc.
Toronto, Ontario
e-mail: [email protected]

When I first read the SCSI code many years ago, I found surprise wasn't adequate and I was forced to resort to astonishment. It's being cleaned up slowly.

Originally hot removal/insertion was the exception, so nobody tripped over the locking issue. Now it's fast becoming the rule.

I have to think about other SCSI systems as well. Some IBM beasties have > 256 PCI slots. Infiniband is threatening direct bus-fibre attachment.

OK, I understand what's going on now. It's no different from those hotplug PCI busses where you press the button and a second or so later the LED goes out and you can remove the card. 10ms sounds rather a short maximum time for a technician to wait for a light to go out....I suppose Telco technicians are rather impatient.

I really think you need to lengthen this interval. The kernel is moving towards this type of hotplug infrastructure which you can easily leverage (or even help build), but it's definitely going to be mainly in user space.

James

Caveat coder -- you also have to handle the case where the device is already gone, by the time you are notified of the hot-unplug event. Some ejections are less friendly than others... though from a SCSI standpoint, hopefully that case is easier -- error out all I/Os in flight, and unregister the host and device structures associated with the recently-removed host. The devil, of course, is in the details ;-)

Jeff

Hmmm... Not being familiar with the SCSI layer but sticking my nose in anyway on general block device/mount point hotplug issues:

How hard would it be to write a simple debugging function to lobotomize a block device? (So that all further I/O to that sucker immediately returns an error.) Not just simulating an a hot extraction (or catastrophic failure) of a block device, but also something you could use to see how gracefully filesystems react.

The reason I ask is there was a discussion a while back about the new lazy unmount (umount -l /blah/foo) not always being quite enough, and that sometimes what what you want is basically "umount -9 /blah/foo" (ala kill -9). Close all files, reparent all process home directories and chroot mount points to a dummy inode, flush all I/O, drive a stake through the superblock's heart, and scatter the ashes at sea. Somebody posted a patch to actually do this. (Against 2.4, i think.) I could probably dig it up if you were curious. Let's see...

http://marc.theaimsgroup.com/?l=linux-kernel&m=103443466225915&q=raw

The eject command should certainly have an "umount with shotgun" option, so zombie processes can't pin your CD in the drive. (Your average end-user is NOT going to be able to grovel through /proc to figure out which processes have an open filehandle or home directory under the cdrom mount point so it can kill them and get the disk out. They're going to power cycle the machine and eject it while the bios is in charge. I've done this myself a couple of times when I'm in a hurry.)

Anyway, if the block device under the filesystem honestly does go away for hotplug eject reasons, the obvious thing to do is umount -9 the sucker immediately so userspace can collapse gracefully (or even conceivably recover). The main difference here is that the flushing would all error out and get discarded, and this wouldn't always get reported to the user, but thanks to write cacheing that's the case anyway. (Use some variant of O_DIRECT or fsync if you care.) The errors userspace does see switch from "all my I/O failed with a media error" to "all my filehandles closed out from under me" (and the directory I'm in has been deleted), but that's still relatively logical behavior.

Does this sound like it's off in left field?

Rob

http://penguicon.sf.net - Terry Pratchett, Eric Raymond, Pete Abrams, Illiad, 
CmdrTaco, liquid nitrogen ice cream, and caffienated jello.  Well why not?

On Sun, 27 Oct 2002, Rob Landley wrote:

(maybe wrap lines around column 70 ? :)

...

Stephen Tweedie did something like this already (for 2.4.19-pre10), called "testdrive". It uses loopback over a block device. He says that it will need modifications to use bio in 2.5.

See here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102457399020069&w=2

~Randy

| Hmmm...  Not being familiar with the SCSI layer but sticking my nose in anyway
| on general block device/mount point hotplug issues:
|
| How hard would it be to write a simple debugging function to lobotomize a
| block device?  (So that all further I/O to that sucker immediately returns an
| error.)  Not just simulating an a hot extraction (or catastrophic failure) of
| a block device, but also something you could use to see how gracefully
| filesystems react.
|
| The reason I ask is there was a discussion a while back about the new lazy
| unmount (umount -l /blah/foo) not always being quite enough, and that
| sometimes what what you want is basically "umount -9 /blah/foo" (ala kill
| -9).  Close all files, reparent all process home directories and chroot mount
| points to a dummy inode, flush all I/O, drive a stake through the
| superblock's heart, and scatter the ashes at sea.  Somebody posted a patch to
| actually do this.  (Against 2.4, i think.)  I could probably dig it up if you
| were curious.  Let's see...
|
| http://marc.theaimsgroup.com/?l=linux-kernel&m=103443466225915&q=raw
|
| The eject command should certainly have an "umount with shotgun" option, so
| zombie processes can't pin your CD in the drive.  (Your average end-user is
| NOT going to be able to grovel through /proc to figure out which processes
| have an open filehandle or home directory under the cdrom mount point so it
| can kill them and get the disk out.  They're going to power cycle the machine
| and eject it while the bios is in charge.  I've done this myself a couple of
| times when I'm in a hurry.)
|
| Anyway, if the block device under the filesystem honestly does go away for
| hotplug eject reasons, the obvious thing to do is umount -9 the sucker
| immediately so userspace can collapse gracefully (or even conceivably
| recover).  The main difference here is that the flushing would all error out
| and get discarded, and this wouldn't always get reported to the user, but
| thanks to write cacheing that's the case anyway.  (Use some variant of
| O_DIRECT or fsync if you care.)  The errors userspace does see switch from
| "all my I/O failed with a media error" to "all my filehandles closed out from
| under me" (and the directory I'm in has been deleted), but that's still
| relatively logical behavior.
|
| Does this sound like it's off in left field?
|
| Rob

A note here is that the SCSI device model still may need to be altered. I am having issues of trying to fill out support and in looks like we might have to restructure a few things.

If the nodes are structure under your own directory they should be ok.

It makes a asymmetric interface, but some parameters can be applied on removal by a cd to the correct leaf node.

Lock operations are somewhat simplified if we convert some of these operations to the device model.

In switched FC fabrics you do receive SCNs if you have registered for them which most adapters do.

We already have interface support from the device model we just need to make the calls.

-andmike
--
Michael Anderson
[email protected]

Thanks for pointing that out. I fixed this once but must have used the wrong set of code when starting this patch. I'll fix.

Thanks again!
-steve