Skip to contentSkip to conversation

ide: Break ide_lock to per-hwgroup lock

PATCH In review Ravikiran G Thirumalai 4 participants 20 messages Sep 6, 2005 – Sep 27, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover ide: Break ide_lock to per-hwgroup lock

The following patchset breaks down the global ide_lock to per-hwgroup lock. We have taken the following approach.

1. Move the hwif tuning code from probe_hwif to ideprobe_init, after hwif_init so that hwgroups are present for all the hwifs when the tune routines for the hwifs are invoked (patch 1)

2. Change the core ide code to use hwgroup->lock instead of ide_lock. Deprecate ide_lock (patch 2)

3. Change the host controllers to use hwgroup->lock (patch 3)

4. Change host controller drivers to use per driver lock instead of ide_lock where needed or hwgroup->lock on case by case basis. This can be done incrementally for various controllers and we will have working code between patches -- this is done now for piix controller only. Eventually, we can change all controllers to remove ide_lock

Thanks to Bartlomiej for comments and suggestions.

Patchset follows. Patchset tested on a smp box with a piix controller.

Thanks,
Kiran

#1 lore raw

PATCH 1/4 ide: Break ide_lock -- Move hwif tuning code after hwif_init

Following patch moves the hwif tuning code from probe_hwif to ideprobe_init after ideprobe_init calls hwif_init so that all hwif's have associated hwgroups. With this patch, we should always have hwgroups for hwifs during calls the drive tune routines.

drivers/ide/ide-probe.c +13 -3
Index: linux-2.6.13/drivers/ide/ide-probe.c
===================================================================
@@ -852,8 +852,15 @@
 
 	if (!hwif->present) {
 		ide_hwif_release_regions(hwif);
-		return;
 	}
+}
+
+static void ide_tune_drives(ide_hwif_t * hwif)
+{
+	unsigned int unit;
+
+	if (!hwif->present)
+		return;
 
 	for (unit = 0; unit < MAX_DRIVES; ++unit) {
 		ide_drive_t *drive = &hwif->drives[unit];
@@ -1443,9 +1450,12 @@
 	for (index = 0; index < MAX_HWIFS; ++index)
 		if (probe[index])
 			probe_hwif(&ide_hwifs[index]);
-	for (index = 0; index < MAX_HWIFS; ++index)
-		if (probe[index])
+	for (index = 0; index < MAX_HWIFS; ++index) {
+		if (probe[index]) {
 			hwif_init(&ide_hwifs[index]);
+			ide_tune_drives(&ide_hwifs[index]);
+		}
+	}
 	for (index = 0; index < MAX_HWIFS; ++index) {
 		if (probe[index]) {
 			ide_hwif_t *hwif = &ide_hwifs[index];
#2 lore raw

PATCH 2/4 ide: Break ide_lock -- replace ide_lock with hwgroup->lock in core ide

In reply to Ravikiran G Thirumalai

Patch to convert ide core code to use hwgroup lock instead of a global ide_lock.

drivers/ide/ide-cd.c +17 -13
Index: linux-2.6.13/drivers/ide/ide-cd.c
===================================================================
@@ -590,7 +590,8 @@
 
 static void cdrom_end_request (ide_drive_t *drive, int uptodate)
 {
-	struct request *rq = HWGROUP(drive)->rq;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	struct request *rq = hwgroup->rq;
 	int nsectors = rq->hard_cur_sectors;
 
 	if ((rq->flags & REQ_SENSE) && uptodate) {
@@ -612,10 +613,10 @@
 			/*
 			 * now end failed request
 			 */
-			spin_lock_irqsave(&ide_lock, flags);
+			spin_lock_irqsave(&hwgroup->lock, flags);
 			end_that_request_chunk(failed, 0, failed->data_len);
 			end_that_request_last(failed);
-			spin_unlock_irqrestore(&ide_lock, flags);
+			spin_unlock_irqrestore(&hwgroup->lock, flags);
 		}
 
 		cdrom_analyze_sense_data(drive, failed, sense);
@@ -636,7 +637,8 @@
    Returns 1 if the request was ended. */
 static int cdrom_decode_status(ide_drive_t *drive, int good_stat, int *stat_ret)
 {
-	struct request *rq = HWGROUP(drive)->rq;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	struct request *rq = hwgroup->rq;
 	int stat, err, sense_key;
 	
 	/* Check for errors. */
@@ -698,10 +700,10 @@
 		 * request sense has completed
 		 */
 		if (stat & ERR_STAT) {
-			spin_lock_irqsave(&ide_lock, flags);
+			spin_lock_irqsave(&hwgroup->lock, flags);
 			blkdev_dequeue_request(rq);
 			HWGROUP(drive)->rq = NULL;
-			spin_unlock_irqrestore(&ide_lock, flags);
+			spin_unlock_irqrestore(&hwgroup->lock, flags);
 
 			cdrom_queue_request_sense(drive, rq->sense, rq);
 		} else
@@ -741,9 +743,9 @@
 					 * take a breather relying on the
 					 * unplug timer to kick us again
 					 */
-					spin_lock_irqsave(&ide_lock, flags);
+					spin_lock_irqsave(&hwgroup->lock, flags);
 					blk_plug_device(drive->queue);
-					spin_unlock_irqrestore(&ide_lock,flags);
+					spin_unlock_irqrestore(&hwgroup->lock, flags);
 					return 1;
 				}
 			}
@@ -839,6 +841,7 @@
 	ide_startstop_t startstop;
 	struct cdrom_info *info = drive->driver_data;
 	ide_hwif_t *hwif = drive->hwif;
+	ide_hwgroup_t *hwgroup = hwif->hwgroup;
 
 	/* Wait for the controller to be idle. */
 	if (ide_wait_stat(&startstop, drive, 0, BUSY_STAT, WAIT_READY))
@@ -866,10 +869,10 @@
 		unsigned long flags;
 
 		/* packet command */
-		spin_lock_irqsave(&ide_lock, flags);
+		spin_lock_irqsave(&hwgroup->lock, flags);
 		hwif->OUTBSYNC(drive, WIN_PACKETCMD, IDE_COMMAND_REG);
 		ndelay(400);
-		spin_unlock_irqrestore(&ide_lock, flags);
+		spin_unlock_irqrestore(&hwgroup->lock, flags);
 
 		return (*handler) (drive);
 	}
@@ -1609,7 +1612,8 @@
 static ide_startstop_t cdrom_newpc_intr(ide_drive_t *drive)
 {
 	struct cdrom_info *info = drive->driver_data;
-	struct request *rq = HWGROUP(drive)->rq;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	struct request *rq = hwgroup->rq;
 	int dma_error, dma, stat, ireason, len, thislen;
 	u8 lowcyl, highcyl;
 	xfer_func_t *xferfunc;
@@ -1737,11 +1741,11 @@
 	if (!rq->data_len)
 		post_transform_command(rq);
 
-	spin_lock_irqsave(&ide_lock, flags);
+	spin_lock_irqsave(&hwgroup->lock, flags);
 	blkdev_dequeue_request(rq);
 	end_that_request_last(rq);
 	HWGROUP(drive)->rq = NULL;
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 	return ide_stopped;
 }
 
drivers/ide/ide-disk.c +2 -1
Index: linux-2.6.13/drivers/ide/ide-disk.c
===================================================================
@@ -786,11 +786,12 @@
 
 static int set_nowerr(ide_drive_t *drive, int arg)
 {
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 	if (ide_spin_wait_hwgroup(drive))
 		return -EBUSY;
 	drive->nowerr = arg;
 	drive->bad_wstat = arg ? BAD_R_STAT : BAD_W_STAT;
-	spin_unlock_irq(&ide_lock);
+	spin_unlock_irq(&hwgroup->lock);
 	return 0;
 }
 
drivers/ide/ide-io.c +35 -32

232 lines not on this page · raw message

drivers/ide/ide-iops.c +10 -8
Index: linux-2.6.13/drivers/ide/ide-iops.c
===================================================================
@@ -945,9 +945,10 @@
 		      unsigned int timeout, ide_expiry_t *expiry)
 {
 	unsigned long flags;
-	spin_lock_irqsave(&ide_lock, flags);
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	spin_lock_irqsave(&hwgroup->lock, flags);
 	__ide_set_handler(drive, handler, timeout, expiry);
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 }
 
 EXPORT_SYMBOL(ide_set_handler);
@@ -972,7 +973,7 @@
 	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 	ide_hwif_t *hwif = HWIF(drive);
 	
-	spin_lock_irqsave(&ide_lock, flags);
+	spin_lock_irqsave(&hwgroup->lock, flags);
 	
 	if(hwgroup->handler)
 		BUG();
@@ -988,7 +989,7 @@
 	   devices 
 	*/
 	ndelay(400);
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 }
 
 EXPORT_SYMBOL(ide_execute_command);
@@ -1168,10 +1169,11 @@
 	ide_hwif_t *hwif;
 	ide_hwgroup_t *hwgroup;
 	
-	spin_lock_irqsave(&ide_lock, flags);
 	hwif = HWIF(drive);
 	hwgroup = HWGROUP(drive);
 
+	spin_lock_irqsave(&hwgroup->lock, flags);
+
 	/* We must not reset with running handlers */
 	if(hwgroup->handler != NULL)
 		BUG();
@@ -1186,7 +1188,7 @@
 		hwgroup->poll_timeout = jiffies + WAIT_WORSTCASE;
 		hwgroup->polling = 1;
 		__ide_set_handler(drive, &atapi_reset_pollfunc, HZ/20, NULL);
-		spin_unlock_irqrestore(&ide_lock, flags);
+		spin_unlock_irqrestore(&hwgroup->lock, flags);
 		return ide_started;
 	}
 
@@ -1199,7 +1201,7 @@
 
 #if OK_TO_RESET_CONTROLLER
 	if (!IDE_CONTROL_REG) {
-		spin_unlock_irqrestore(&ide_lock, flags);
+		spin_unlock_irqrestore(&hwgroup->lock, flags);
 		return ide_stopped;
 	}
 
@@ -1239,7 +1241,7 @@
 	
 #endif	/* OK_TO_RESET_CONTROLLER */
 
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 	return ide_started;
 }
 
drivers/ide/ide-lib.c +4 -5
Index: linux-2.6.13/drivers/ide/ide-lib.c
===================================================================
@@ -450,12 +450,11 @@
 	struct request *rq;
 	u8 opcode = 0;
 	int found = 0;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 
-	spin_lock(&ide_lock);
-	rq = NULL;
-	if (HWGROUP(drive))
-		rq = HWGROUP(drive)->rq;
-	spin_unlock(&ide_lock);
+	spin_lock(&hwgroup->lock);
+	rq = hwgroup->rq;
+	spin_unlock(&hwgroup->lock);
 	if (!rq)
 		return;
 	if (rq->flags & (REQ_DRIVE_CMD | REQ_DRIVE_TASK)) {
drivers/ide/ide-probe.c +14 -11
Index: linux-2.6.13/drivers/ide/ide-probe.c
===================================================================
@@ -985,7 +985,8 @@
 	 *	do not.
 	 */
 
-	q = blk_init_queue_node(do_ide_request, &ide_lock, hwif_to_node(hwif));
+	q = blk_init_queue_node(do_ide_request, &(HWGROUP(drive)->lock), 
+				hwif_to_node(hwif));
 	if (!q)
 		return 1;
 
@@ -1099,10 +1100,10 @@
 		 * linked list, the first entry is the hwif that owns
 		 * hwgroup->handler - do not change that.
 		 */
-		spin_lock_irq(&ide_lock);
+		spin_lock_irq(&hwgroup->lock);
 		hwif->next = hwgroup->hwif->next;
 		hwgroup->hwif->next = hwif;
-		spin_unlock_irq(&ide_lock);
+		spin_unlock_irq(&hwgroup->lock);
 	} else {
 		hwgroup = kmalloc_node(sizeof(ide_hwgroup_t), GFP_KERNEL,
 					hwif_to_node(hwif->drives[0].hwif));
@@ -1112,6 +1113,7 @@
 		hwif->hwgroup = hwgroup;
 
 		memset(hwgroup, 0, sizeof(ide_hwgroup_t));
+		spin_lock_init(&hwgroup->lock);
 		hwgroup->hwif     = hwif->next = hwif;
 		hwgroup->rq       = NULL;
 		hwgroup->handler  = NULL;
@@ -1159,7 +1161,7 @@
 			printk(KERN_ERR "ide: failed to init %s\n",drive->name);
 			continue;
 		}
-		spin_lock_irq(&ide_lock);
+		spin_lock_irq(&hwgroup->lock);
 		if (!hwgroup->drive) {
 			/* first drive for hwgroup. */
 			drive->next = drive;
@@ -1169,7 +1171,7 @@
 			drive->next = hwgroup->drive->next;
 			hwgroup->drive->next = drive;
 		}
-		spin_unlock_irq(&ide_lock);
+		spin_unlock_irq(&hwgroup->lock);
 	}
 
 #if !defined(__mc68000__) && !defined(CONFIG_APUS) && !defined(__sparc__)
@@ -1193,13 +1195,13 @@
 	up(&ide_cfg_sem);
 	return 0;
 out_unlink:
-	spin_lock_irq(&ide_lock);
 	if (hwif->next == hwif) {
 		BUG_ON(match);
 		BUG_ON(hwgroup->hwif != hwif);
 		kfree(hwgroup);
 	} else {
 		ide_hwif_t *g;
+		spin_lock_irq(&hwgroup->lock);
 		g = hwgroup->hwif;
 		while (g->next != hwif)
 			g = g->next;
@@ -1210,8 +1212,8 @@
 			hwgroup->hwif = g;
 		}
 		BUG_ON(hwgroup->hwif == hwif);
+		spin_unlock_irq(&hwgroup->lock);
 	}
-	spin_unlock_irq(&ide_lock);
 out_up:
 	up(&ide_cfg_sem);
 	return 1;
@@ -1316,8 +1318,9 @@
 static void drive_release_dev (struct device *dev)
 {
 	ide_drive_t *drive = container_of(dev, ide_drive_t, gendev);
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 
-	spin_lock_irq(&ide_lock);
+	spin_lock_irq(&hwgroup->lock);
 	if (drive->devfs_name[0] != '\0') {
 		devfs_remove(drive->devfs_name);
 		drive->devfs_name[0] = '\0';
@@ -1329,11 +1332,11 @@
 	}
 	drive->present = 0;
 	/* Messed up locking ... */
-	spin_unlock_irq(&ide_lock);
+	spin_unlock_irq(&hwgroup->lock);
 	blk_cleanup_queue(drive->queue);
-	spin_lock_irq(&ide_lock);
+	spin_lock_irq(&hwgroup->lock);
 	drive->queue = NULL;
-	spin_unlock_irq(&ide_lock);
+	spin_unlock_irq(&hwgroup->lock);
 
 	up(&drive->gendev_rel_sem);
 }
drivers/ide/ide.c +39 -29

212 lines not on this page · raw message

drivers/scsi/ide-scsi.c +11 -9
Index: linux-2.6.13/drivers/scsi/ide-scsi.c
===================================================================
@@ -956,6 +956,7 @@
 	ide_drive_t    *drive = scsi->drive;
 	int		busy;
 	int             ret   = FAILED;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 
 	/* In idescsi_eh_abort we try to gently pry our command from the ide subsystem */
 
@@ -974,7 +975,7 @@
 	if (test_bit(IDESCSI_LOG_CMD, &scsi->log))
 		printk (KERN_WARNING "ide-scsi: drive did%s become ready\n", busy?" not":"");
 
-	spin_lock_irq(&ide_lock);
+	spin_lock_irq(&hwgroup->lock);
 
 	/* If there is no pc running we're done (our interrupt took care of it) */
 	if (!scsi->pc) {
@@ -1000,7 +1001,7 @@
 	}
 
 ide_unlock:
-	spin_unlock_irq(&ide_lock);
+	spin_unlock_irq(&hwgroup->lock);
 no_drive:
 	if (test_bit(IDESCSI_LOG_CMD, &scsi->log))
 		printk (KERN_WARNING "ide-scsi: abort returns %s\n", ret == SUCCESS?"success":"failed");
@@ -1015,6 +1016,7 @@
 	ide_drive_t    *drive = scsi->drive;
 	int             ready = 0;
 	int             ret   = SUCCESS;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 
 	/* In idescsi_eh_reset we forcefully remove the command from the ide subsystem and reset the device. */
 
@@ -1028,11 +1030,11 @@
 	}
 
 	spin_lock_irq(cmd->device->host->host_lock);
-	spin_lock(&ide_lock);
+	spin_lock(&hwgroup->lock);
 
-	if (!scsi->pc || (req = scsi->pc->rq) != HWGROUP(drive)->rq || !HWGROUP(drive)->handler) {
+	if (!scsi->pc || (req = scsi->pc->rq) != hwgroup->rq || !hwgroup->handler) {
 		printk (KERN_WARNING "ide-scsi: No active request in idescsi_eh_reset\n");
-		spin_unlock(&ide_lock);
+		spin_unlock(&hwgroup->lock);
 		spin_unlock_irq(cmd->device->host->host_lock);
 		return FAILED;
 	}
@@ -1052,10 +1054,10 @@
 		end_that_request_last(req);
 	}
 
-	HWGROUP(drive)->rq = NULL;
-	HWGROUP(drive)->handler = NULL;
-	HWGROUP(drive)->busy = 1;		/* will set this to zero when ide reset finished */
-	spin_unlock(&ide_lock);
+	hwgroup->rq = NULL;
+	hwgroup->handler = NULL;
+	hwgroup->busy = 1;		/* will set this to zero when ide reset finished */
+	spin_unlock(&hwgroup->lock);
 
 	ide_do_reset(drive);
 
include/linux/ide.h +4 -3
Index: linux-2.6.13/include/linux/ide.h
===================================================================
@@ -961,6 +961,7 @@
 	int pio_clock;
 
 	unsigned char cmd_buf[4];
+	spinlock_t lock;
 } ide_hwgroup_t;
 
 /* structure attached to the request for IDE_TASK_CMDS */
@@ -1480,12 +1481,12 @@
 /*
  * Structure locking:
  *
- * ide_cfg_sem and ide_lock together protect changes to
+ * ide_cfg_sem and hwgroup->lock together protect changes to
  * ide_hwif_t->{next,hwgroup}
  * ide_drive_t->next
  *
- * ide_hwgroup_t->busy: ide_lock
- * ide_hwgroup_t->hwif: ide_lock
+ * ide_hwgroup_t->busy: hwgroup->lock
+ * ide_hwgroup_t->hwif: hwgroup->lock
  * ide_hwif_t->mate: constant, no locking
  * ide_drive_t->hwif: constant, no locking
  */
#3 lore raw

Re: [patch 2/4] ide: Break ide_lock -- replace ide_lock with hwgroup->lock in core ide

Quoted 7 lines · Ravikiran G Thirumalai

Patch to convert ide core code to use hwgroup lock instead of a global ide_lock.

Index: linux-2.6.13/drivers/ide/ide-io.c
===================================================================
--- linux-2.6.13.orig/drivers/ide/ide-io.c 2005-09-06 11:22:29.000000000 -0700

Review line 1221

Quoted 10 lines
(quoted diff) +2 -2
@@ -1211,11 +1214,11 @@
 		 */
 		if (masked_irq != IDE_NO_IRQ && hwif->irq != masked_irq)
 			disable_irq_nosync(hwif->irq);
-		spin_unlock(&ide_lock);
+		spin_unlock(&hwgroup->lock);
 		local_irq_enable();
 			/* allow other IRQs while we start this request */
 		startstop = start_request(drive, rq);
-		spin_lock_irq(&ide_lock);
+		spin_unlock_irq(&hwgroup->lock);
		^^^^^^^^^^^^^^^^^^^^^^^

My bad,
Fixed patch attached.

Thanks,
Kiran

Patch to convert ide core code to use hwgroup lock instead of a global ide_lock.

drivers/ide/ide-cd.c +17 -13
Index: linux-2.6.13/drivers/ide/ide-cd.c
===================================================================
@@ -590,7 +590,8 @@
 
 static void cdrom_end_request (ide_drive_t *drive, int uptodate)
 {
-	struct request *rq = HWGROUP(drive)->rq;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	struct request *rq = hwgroup->rq;
 	int nsectors = rq->hard_cur_sectors;
 
 	if ((rq->flags & REQ_SENSE) && uptodate) {
@@ -612,10 +613,10 @@
 			/*
 			 * now end failed request
 			 */
-			spin_lock_irqsave(&ide_lock, flags);
+			spin_lock_irqsave(&hwgroup->lock, flags);
 			end_that_request_chunk(failed, 0, failed->data_len);
 			end_that_request_last(failed);
-			spin_unlock_irqrestore(&ide_lock, flags);
+			spin_unlock_irqrestore(&hwgroup->lock, flags);
 		}
 
 		cdrom_analyze_sense_data(drive, failed, sense);
@@ -636,7 +637,8 @@
    Returns 1 if the request was ended. */
 static int cdrom_decode_status(ide_drive_t *drive, int good_stat, int *stat_ret)
 {
-	struct request *rq = HWGROUP(drive)->rq;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	struct request *rq = hwgroup->rq;
 	int stat, err, sense_key;
 	
 	/* Check for errors. */
@@ -698,10 +700,10 @@
 		 * request sense has completed
 		 */
 		if (stat & ERR_STAT) {
-			spin_lock_irqsave(&ide_lock, flags);
+			spin_lock_irqsave(&hwgroup->lock, flags);
 			blkdev_dequeue_request(rq);
 			HWGROUP(drive)->rq = NULL;
-			spin_unlock_irqrestore(&ide_lock, flags);
+			spin_unlock_irqrestore(&hwgroup->lock, flags);
 
 			cdrom_queue_request_sense(drive, rq->sense, rq);
 		} else
@@ -741,9 +743,9 @@
 					 * take a breather relying on the
 					 * unplug timer to kick us again
 					 */
-					spin_lock_irqsave(&ide_lock, flags);
+					spin_lock_irqsave(&hwgroup->lock, flags);
 					blk_plug_device(drive->queue);
-					spin_unlock_irqrestore(&ide_lock,flags);
+					spin_unlock_irqrestore(&hwgroup->lock, flags);
 					return 1;
 				}
 			}
@@ -839,6 +841,7 @@
 	ide_startstop_t startstop;
 	struct cdrom_info *info = drive->driver_data;
 	ide_hwif_t *hwif = drive->hwif;
+	ide_hwgroup_t *hwgroup = hwif->hwgroup;
 
 	/* Wait for the controller to be idle. */
 	if (ide_wait_stat(&startstop, drive, 0, BUSY_STAT, WAIT_READY))
@@ -866,10 +869,10 @@
 		unsigned long flags;
 
 		/* packet command */
-		spin_lock_irqsave(&ide_lock, flags);
+		spin_lock_irqsave(&hwgroup->lock, flags);
 		hwif->OUTBSYNC(drive, WIN_PACKETCMD, IDE_COMMAND_REG);
 		ndelay(400);
-		spin_unlock_irqrestore(&ide_lock, flags);
+		spin_unlock_irqrestore(&hwgroup->lock, flags);
 
 		return (*handler) (drive);
 	}
@@ -1609,7 +1612,8 @@
 static ide_startstop_t cdrom_newpc_intr(ide_drive_t *drive)
 {
 	struct cdrom_info *info = drive->driver_data;
-	struct request *rq = HWGROUP(drive)->rq;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	struct request *rq = hwgroup->rq;
 	int dma_error, dma, stat, ireason, len, thislen;
 	u8 lowcyl, highcyl;
 	xfer_func_t *xferfunc;
@@ -1737,11 +1741,11 @@
 	if (!rq->data_len)
 		post_transform_command(rq);
 
-	spin_lock_irqsave(&ide_lock, flags);
+	spin_lock_irqsave(&hwgroup->lock, flags);
 	blkdev_dequeue_request(rq);
 	end_that_request_last(rq);
 	HWGROUP(drive)->rq = NULL;
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 	return ide_stopped;
 }
 
drivers/ide/ide-disk.c +2 -1
Index: linux-2.6.13/drivers/ide/ide-disk.c
===================================================================
@@ -786,11 +786,12 @@
 
 static int set_nowerr(ide_drive_t *drive, int arg)
 {
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 	if (ide_spin_wait_hwgroup(drive))
 		return -EBUSY;
 	drive->nowerr = arg;
 	drive->bad_wstat = arg ? BAD_R_STAT : BAD_W_STAT;
-	spin_unlock_irq(&ide_lock);
+	spin_unlock_irq(&hwgroup->lock);
 	return 0;
 }
 
drivers/ide/ide-io.c +35 -32

232 lines not on this page · raw message

drivers/ide/ide-iops.c +10 -8
Index: linux-2.6.13/drivers/ide/ide-iops.c
===================================================================
@@ -945,9 +945,10 @@
 		      unsigned int timeout, ide_expiry_t *expiry)
 {
 	unsigned long flags;
-	spin_lock_irqsave(&ide_lock, flags);
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+	spin_lock_irqsave(&hwgroup->lock, flags);
 	__ide_set_handler(drive, handler, timeout, expiry);
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 }
 
 EXPORT_SYMBOL(ide_set_handler);
@@ -972,7 +973,7 @@
 	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 	ide_hwif_t *hwif = HWIF(drive);
 	
-	spin_lock_irqsave(&ide_lock, flags);
+	spin_lock_irqsave(&hwgroup->lock, flags);
 	
 	if(hwgroup->handler)
 		BUG();
@@ -988,7 +989,7 @@
 	   devices 
 	*/
 	ndelay(400);
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 }
 
 EXPORT_SYMBOL(ide_execute_command);
@@ -1168,10 +1169,11 @@
 	ide_hwif_t *hwif;
 	ide_hwgroup_t *hwgroup;
 	
-	spin_lock_irqsave(&ide_lock, flags);
 	hwif = HWIF(drive);
 	hwgroup = HWGROUP(drive);
 
+	spin_lock_irqsave(&hwgroup->lock, flags);
+
 	/* We must not reset with running handlers */
 	if(hwgroup->handler != NULL)
 		BUG();
@@ -1186,7 +1188,7 @@
 		hwgroup->poll_timeout = jiffies + WAIT_WORSTCASE;
 		hwgroup->polling = 1;
 		__ide_set_handler(drive, &atapi_reset_pollfunc, HZ/20, NULL);
-		spin_unlock_irqrestore(&ide_lock, flags);
+		spin_unlock_irqrestore(&hwgroup->lock, flags);
 		return ide_started;
 	}
 
@@ -1199,7 +1201,7 @@
 
 #if OK_TO_RESET_CONTROLLER
 	if (!IDE_CONTROL_REG) {
-		spin_unlock_irqrestore(&ide_lock, flags);
+		spin_unlock_irqrestore(&hwgroup->lock, flags);
 		return ide_stopped;
 	}
 
@@ -1239,7 +1241,7 @@
 	
 #endif	/* OK_TO_RESET_CONTROLLER */
 
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 	return ide_started;
 }
 
drivers/ide/ide-lib.c +4 -5
Index: linux-2.6.13/drivers/ide/ide-lib.c
===================================================================
@@ -450,12 +450,11 @@
 	struct request *rq;
 	u8 opcode = 0;
 	int found = 0;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 
-	spin_lock(&ide_lock);
-	rq = NULL;
-	if (HWGROUP(drive))
-		rq = HWGROUP(drive)->rq;
-	spin_unlock(&ide_lock);
+	spin_lock(&hwgroup->lock);
+	rq = hwgroup->rq;
+	spin_unlock(&hwgroup->lock);
 	if (!rq)
 		return;
 	if (rq->flags & (REQ_DRIVE_CMD | REQ_DRIVE_TASK)) {
drivers/ide/ide-probe.c +14 -11
Index: linux-2.6.13/drivers/ide/ide-probe.c
===================================================================
@@ -985,7 +985,8 @@
 	 *	do not.
 	 */
 
-	q = blk_init_queue_node(do_ide_request, &ide_lock, hwif_to_node(hwif));
+	q = blk_init_queue_node(do_ide_request, &(HWGROUP(drive)->lock), 
+				hwif_to_node(hwif));
 	if (!q)
 		return 1;
 
@@ -1099,10 +1100,10 @@
 		 * linked list, the first entry is the hwif that owns
 		 * hwgroup->handler - do not change that.
 		 */
-		spin_lock_irq(&ide_lock);
+		spin_lock_irq(&hwgroup->lock);
 		hwif->next = hwgroup->hwif->next;
 		hwgroup->hwif->next = hwif;
-		spin_unlock_irq(&ide_lock);
+		spin_unlock_irq(&hwgroup->lock);
 	} else {
 		hwgroup = kmalloc_node(sizeof(ide_hwgroup_t), GFP_KERNEL,
 					hwif_to_node(hwif->drives[0].hwif));
@@ -1112,6 +1113,7 @@
 		hwif->hwgroup = hwgroup;
 
 		memset(hwgroup, 0, sizeof(ide_hwgroup_t));
+		spin_lock_init(&hwgroup->lock);
 		hwgroup->hwif     = hwif->next = hwif;
 		hwgroup->rq       = NULL;
 		hwgroup->handler  = NULL;
@@ -1159,7 +1161,7 @@
 			printk(KERN_ERR "ide: failed to init %s\n",drive->name);
 			continue;
 		}
-		spin_lock_irq(&ide_lock);
+		spin_lock_irq(&hwgroup->lock);
 		if (!hwgroup->drive) {
 			/* first drive for hwgroup. */
 			drive->next = drive;
@@ -1169,7 +1171,7 @@
 			drive->next = hwgroup->drive->next;
 			hwgroup->drive->next = drive;
 		}
-		spin_unlock_irq(&ide_lock);
+		spin_unlock_irq(&hwgroup->lock);
 	}
 
 #if !defined(__mc68000__) && !defined(CONFIG_APUS) && !defined(__sparc__)
@@ -1193,13 +1195,13 @@
 	up(&ide_cfg_sem);
 	return 0;
 out_unlink:
-	spin_lock_irq(&ide_lock);
 	if (hwif->next == hwif) {
 		BUG_ON(match);
 		BUG_ON(hwgroup->hwif != hwif);
 		kfree(hwgroup);
 	} else {
 		ide_hwif_t *g;
+		spin_lock_irq(&hwgroup->lock);
 		g = hwgroup->hwif;
 		while (g->next != hwif)
 			g = g->next;
@@ -1210,8 +1212,8 @@
 			hwgroup->hwif = g;
 		}
 		BUG_ON(hwgroup->hwif == hwif);
+		spin_unlock_irq(&hwgroup->lock);
 	}
-	spin_unlock_irq(&ide_lock);
 out_up:
 	up(&ide_cfg_sem);
 	return 1;
@@ -1316,8 +1318,9 @@
 static void drive_release_dev (struct device *dev)
 {
 	ide_drive_t *drive = container_of(dev, ide_drive_t, gendev);
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 
-	spin_lock_irq(&ide_lock);
+	spin_lock_irq(&hwgroup->lock);
 	if (drive->devfs_name[0] != '\0') {
 		devfs_remove(drive->devfs_name);
 		drive->devfs_name[0] = '\0';
@@ -1329,11 +1332,11 @@
 	}
 	drive->present = 0;
 	/* Messed up locking ... */
-	spin_unlock_irq(&ide_lock);
+	spin_unlock_irq(&hwgroup->lock);
 	blk_cleanup_queue(drive->queue);
-	spin_lock_irq(&ide_lock);
+	spin_lock_irq(&hwgroup->lock);
 	drive->queue = NULL;
-	spin_unlock_irq(&ide_lock);
+	spin_unlock_irq(&hwgroup->lock);
 
 	up(&drive->gendev_rel_sem);
 }
drivers/ide/ide.c +39 -29

212 lines not on this page · raw message

drivers/scsi/ide-scsi.c +11 -9
Index: linux-2.6.13/drivers/scsi/ide-scsi.c
===================================================================
@@ -956,6 +956,7 @@
 	ide_drive_t    *drive = scsi->drive;
 	int		busy;
 	int             ret   = FAILED;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 
 	/* In idescsi_eh_abort we try to gently pry our command from the ide subsystem */
 
@@ -974,7 +975,7 @@
 	if (test_bit(IDESCSI_LOG_CMD, &scsi->log))
 		printk (KERN_WARNING "ide-scsi: drive did%s become ready\n", busy?" not":"");
 
-	spin_lock_irq(&ide_lock);
+	spin_lock_irq(&hwgroup->lock);
 
 	/* If there is no pc running we're done (our interrupt took care of it) */
 	if (!scsi->pc) {
@@ -1000,7 +1001,7 @@
 	}
 
 ide_unlock:
-	spin_unlock_irq(&ide_lock);
+	spin_unlock_irq(&hwgroup->lock);
 no_drive:
 	if (test_bit(IDESCSI_LOG_CMD, &scsi->log))
 		printk (KERN_WARNING "ide-scsi: abort returns %s\n", ret == SUCCESS?"success":"failed");
@@ -1015,6 +1016,7 @@
 	ide_drive_t    *drive = scsi->drive;
 	int             ready = 0;
 	int             ret   = SUCCESS;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 
 	/* In idescsi_eh_reset we forcefully remove the command from the ide subsystem and reset the device. */
 
@@ -1028,11 +1030,11 @@
 	}
 
 	spin_lock_irq(cmd->device->host->host_lock);
-	spin_lock(&ide_lock);
+	spin_lock(&hwgroup->lock);
 
-	if (!scsi->pc || (req = scsi->pc->rq) != HWGROUP(drive)->rq || !HWGROUP(drive)->handler) {
+	if (!scsi->pc || (req = scsi->pc->rq) != hwgroup->rq || !hwgroup->handler) {
 		printk (KERN_WARNING "ide-scsi: No active request in idescsi_eh_reset\n");
-		spin_unlock(&ide_lock);
+		spin_unlock(&hwgroup->lock);
 		spin_unlock_irq(cmd->device->host->host_lock);
 		return FAILED;
 	}
@@ -1052,10 +1054,10 @@
 		end_that_request_last(req);
 	}
 
-	HWGROUP(drive)->rq = NULL;
-	HWGROUP(drive)->handler = NULL;
-	HWGROUP(drive)->busy = 1;		/* will set this to zero when ide reset finished */
-	spin_unlock(&ide_lock);
+	hwgroup->rq = NULL;
+	hwgroup->handler = NULL;
+	hwgroup->busy = 1;		/* will set this to zero when ide reset finished */
+	spin_unlock(&hwgroup->lock);
 
 	ide_do_reset(drive);
 
include/linux/ide.h +4 -3
Index: linux-2.6.13/include/linux/ide.h
===================================================================
@@ -961,6 +961,7 @@
 	int pio_clock;
 
 	unsigned char cmd_buf[4];
+	spinlock_t lock;
 } ide_hwgroup_t;
 
 /* structure attached to the request for IDE_TASK_CMDS */
@@ -1480,12 +1481,12 @@
 /*
  * Structure locking:
  *
- * ide_cfg_sem and ide_lock together protect changes to
+ * ide_cfg_sem and hwgroup->lock together protect changes to
  * ide_hwif_t->{next,hwgroup}
  * ide_drive_t->next
  *
- * ide_hwgroup_t->busy: ide_lock
- * ide_hwgroup_t->hwif: ide_lock
+ * ide_hwgroup_t->busy: hwgroup->lock
+ * ide_hwgroup_t->hwif: hwgroup->lock
  * ide_hwif_t->mate: constant, no locking
  * ide_drive_t->hwif: constant, no locking
  */
#4 lore raw

PATCH 3/4 ide: Break ide_lock -- change controller drivers

In reply to Ravikiran G Thirumalai

Patch to make ide-host controllers use hwgroup lock where serialization with hwgroup->lock is necessary

drivers/ide/legacy/ht6560b.c +15 -10
Index: linux-2.6.13/drivers/ide/legacy/ht6560b.c
===================================================================
@@ -256,9 +256,11 @@
 {
 	unsigned long flags;
 	int t = HT_PREFETCH_MODE << 8;
-	
-	spin_lock_irqsave(&ide_lock, flags);
-	
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
+
+	spin_lock_irqsave(&hwgroup->lock, flags);
+	spin_lock(&ide_lock);
+
 	/*
 	 *  Prefetch mode and unmask irq seems to conflict
 	 */
@@ -270,9 +272,10 @@
 		drive->drive_data &= ~t;  /* disable prefetch mode */
 		drive->no_unmask = 0;
 	}
-	
-	spin_unlock_irqrestore(&ide_lock, flags);
-	
+
+	spin_unlock(&ide_lock);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
+
 #ifdef DEBUG
 	printk("ht6560b: drive %s prefetch mode %sabled\n", drive->name, (state ? "en" : "dis"));
 #endif
@@ -282,6 +285,7 @@
 {
 	unsigned long flags;
 	u8 timing;
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 	
 	switch (pio) {
 	case 8:         /* set prefetch off */
@@ -291,14 +295,15 @@
 	}
 	
 	timing = ht_pio2timings(drive, pio);
-	
-	spin_lock_irqsave(&ide_lock, flags);
+
+	spin_lock_irqsave(&hwgroup->lock, flags);
+	spin_lock(&ide_lock);
 	
 	drive->drive_data &= 0xff00;
 	drive->drive_data |= timing;
 	
-	spin_unlock_irqrestore(&ide_lock, flags);
-	
+	spin_unlock(&ide_lock);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 #ifdef DEBUG
 	printk("ht6560b: drive %s tuned to pio mode %#x timing=%#x\n", drive->name, pio, timing);
 #endif
drivers/ide/pci/cmd640.c +7 -2
Index: linux-2.6.13/drivers/ide/pci/cmd640.c
===================================================================
@@ -442,11 +442,14 @@
 static void set_prefetch_mode (unsigned int index, int mode)
 {
 	ide_drive_t *drive = cmd_drives[index];
+	ide_hwgroup_t *hwgroup = HWGROUP(drive);
 	int reg = prefetch_regs[index];
 	u8 b;
 	unsigned long flags;
 
-	spin_lock_irqsave(&ide_lock, flags);
+	spin_lock_irqsave(&hwgroup->lock, flags);
+	spin_lock(&ide_lock);
+
 	b = __get_cmd640_reg(reg);
 	if (mode) {	/* want prefetch on? */
 #if CMD640_PREFETCH_MASKS
@@ -462,7 +465,9 @@
 		b |= prefetch_masks[index];	/* disable prefetch */
 	}
 	__put_cmd640_reg(reg, b);
-	spin_unlock_irqrestore(&ide_lock, flags);
+
+	spin_unlock(&ide_lock);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 }
 
 /*
drivers/ide/pci/piix.c +5 -2
Index: linux-2.6.13/drivers/ide/pci/piix.c
===================================================================
@@ -215,6 +215,7 @@
 {
 	ide_hwif_t *hwif	= HWIF(drive);
 	struct pci_dev *dev	= hwif->pci_dev;
+	ide_hwgroup_t *hwgroup  = HWGROUP(drive);
 	int is_slave		= (&hwif->drives[1] == drive);
 	int master_port		= hwif->channel ? 0x42 : 0x40;
 	int slave_port		= 0x44;
@@ -229,7 +230,8 @@
 			    { 2, 3 }, };
 
 	pio = ide_get_best_pio_mode(drive, pio, 5, NULL);
-	spin_lock_irqsave(&ide_lock, flags);
+	spin_lock_irqsave(&hwgroup->lock, flags);
+	spin_lock(&ide_lock);
 	pci_read_config_word(dev, master_port, &master_data);
 	if (is_slave) {
 		master_data = master_data | 0x4000;
@@ -249,7 +251,8 @@
 	pci_write_config_word(dev, master_port, master_data);
 	if (is_slave)
 		pci_write_config_byte(dev, slave_port, slave_data);
-	spin_unlock_irqrestore(&ide_lock, flags);
+	spin_unlock(&ide_lock);
+	spin_unlock_irqrestore(&hwgroup->lock, flags);
 }
 
 /**
#5 lore raw

PATCH 4/4 ide: Break ide_lock -- remove ide_lock from piix driver

In reply to Ravikiran G Thirumalai

Patch to convert piix driver to use per-driver/hwgroup lock and kill ide_lock. In the case of piix, hwgroup->lock should be sufficient.

drivers/ide/pci/piix.c +0 -2
Index: linux-2.6.13/drivers/ide/pci/piix.c
===================================================================
@@ -231,7 +231,6 @@
 
 	pio = ide_get_best_pio_mode(drive, pio, 5, NULL);
 	spin_lock_irqsave(&hwgroup->lock, flags);
-	spin_lock(&ide_lock);
 	pci_read_config_word(dev, master_port, &master_data);
 	if (is_slave) {
 		master_data = master_data | 0x4000;
@@ -251,7 +250,6 @@
 	pci_write_config_word(dev, master_port, master_data);
 	if (is_slave)
 		pci_write_config_byte(dev, slave_port, slave_data);
-	spin_unlock(&ide_lock);
 	spin_unlock_irqrestore(&hwgroup->lock, flags);
 }
 
#6 lore raw

Re: [patch 4/4] ide: Break ide_lock -- remove ide_lock from piix driver

Quoted 2 lines · Ravikiran G Thirumalai

Patch to convert piix driver to use per-driver/hwgroup lock and kill ide_lock. In the case of piix, hwgroup->lock should be sufficient.

PIIX requires that both channels are quiescent when retuning in some cases. It wasn't totally safe before, its now totally broken. Start by fixing the IDE layer locking properly (or forward porting my patches and then fixing them for all the refcounting changes and other stuff done since).

Better yet move to the SATA driver - it can drive never PIIX chips in PATA mode and its in my experience already more reliable.

Alan

#7 lore raw

Re: [patch 4/4] ide: Break ide_lock -- remove ide_lock from piix driver

Quoted 6 lines · Alan Cox
Quoted 2 lines · Ravikiran G Thirumalai

Patch to convert piix driver to use per-driver/hwgroup lock and kill ide_lock. In the case of piix, hwgroup->lock should be sufficient.

PIIX requires that both channels are quiescent when retuning in some cases. It wasn't totally safe before, its now totally broken. Start by

Then the change to piix controller in my patchset is bad, How about changing the ide_lock to per-driver lock in this case? Locking for rest of the controllers in the system is left equivalent to what ide_lock did earlier..

Quoted 3 lines

fixing the IDE layer locking properly (or forward porting my patches and then fixing them for all the refcounting changes and other stuff done since).

Can you please point me to the patchset...

Thanks,
Kiran

#8 lore raw

Re: [patch 4/4] ide: Break ide_lock -- remove ide_lock from piix driver

Quoted 3 lines · Ravikiran G Thirumalai

Then the change to piix controller in my patchset is bad, How about changing the ide_lock to per-driver lock in this case? Locking for rest of the controllers in the system is left equivalent to what ide_lock did earlier..

Thats what we did in Fedora Core. Its neccessary because the state of ide_lock is undefined when the tuning functions get called and the base kernel didn't handle this well - we had various reports from users of hangs as a result.

Its not a full fix, really PIIX needs to wrap the entire command issue with a semaphore or similar to avoid the error handling (CRC error) breaking the driver. Unfortunately we do the CRC error handling and speed change in an IRQ handler polled so I've never seen how we could fix that without rewriting the entire error recovery code to work like the scsi layer (in a thread).

Quoted 5 lines
Quoted 3 lines

fixing the IDE layer locking properly (or forward porting my patches and then fixing them for all the refcounting changes and other stuff done since).

Can you please point me to the patchset...

2.6.11-ac kernels. I will send you (off list) the last full version of the fixes. Part of what it fixes (especially the proc file locking changes) are mostly handled now by the reworking of the IDE code to use refcounts that Bartlomiej did, and which is definitely the better way to fix it.

Alan

#9 lore raw
In reply to Ravikiran G Thirumalai
Quoted 2 lines · Ravikiran G Thirumalai

The following patchset breaks down the global ide_lock to per-hwgroup lock. We have taken the following approach.

Curious, what is the point of this?

Signature · 1 line
Jens Axboe
#10 lore raw
Quoted 6 lines · Jens Axboe
Quoted 2 lines · Ravikiran G Thirumalai

The following patchset breaks down the global ide_lock to per-hwgroup lock. We have taken the following approach.

Curious, what is the point of this?

On smp machines with multiple ide interfaces, we take per-group lock instead of a global lock, there by breaking the lock to per-irq hwgroups.

Thanks,
Kiran

#11 lore raw
Quoted 10 lines · Ravikiran G Thirumalai
Quoted 6 lines · Jens Axboe
Quoted 2 lines · Ravikiran G Thirumalai

The following patchset breaks down the global ide_lock to per-hwgroup lock. We have taken the following approach.

Curious, what is the point of this?

On smp machines with multiple ide interfaces, we take per-group lock instead of a global lock, there by breaking the lock to per-irq hwgroups.

I realize the theory behind breaking up locks, I'm just wondering about this specific case. Please show actual contention data promoting this specific case, we don't break up locks "just because".

I'm asking because I've never heard anyone complain about IDE lock contention and a proper patch usually comes with analysis of why it is needed.

Signature · 1 line
Jens Axboe
#12 lore raw
Quoted 12 lines · Jens Axboe
Quoted 10 lines · Ravikiran G Thirumalai
Quoted 6 lines · Jens Axboe
Quoted 2 lines · Ravikiran G Thirumalai

The following patchset breaks down the global ide_lock to per-hwgroup lock. We have taken the following approach.

Curious, what is the point of this?

On smp machines with multiple ide interfaces, we take per-group lock instead of a global lock, there by breaking the lock to per-irq hwgroups.

Quoted 7 lines

I realize the theory behind breaking up locks, I'm just wondering about this specific case. Please show actual contention data promoting this specific case, we don't break up locks "just because".

I'm asking because I've never heard anyone complain about IDE lock contention and a proper patch usually comes with analysis of why it is needed.

Since ide_lock spinlock is used for all drives as queue lock and for all controllers as IDE lock I guess that with multiple controllers there is a lot contention on it...

Breaking ide_lock is fine with me however seeing numbers would greatly help in getting wider acceptance for this change, Ravikiran?

Bartlomiej

#13 lore raw
Quoted 15 lines · Bartlomiej Zolnierkiewicz
Quoted 6 lines · Jens Axboe
Quoted 5 lines · Ravikiran G Thirumalai
Quoted 4 lines · Jens Axboe
Quoted 2 lines · Ravikiran G Thirumalai

The following patchset breaks down the global ide_lock to per-hwgroup lock. We have taken the following approach.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Curious, what is the point of this?

I'm asking because I've never heard anyone complain about IDE lock contention and a proper patch usually comes with analysis of why it is needed.

Quoted 7 lines

Since ide_lock spinlock is used for all drives as queue lock and for all controllers as IDE lock I guess that with multiple controllers there is a lot contention on it...

Breaking ide_lock is fine with me however seeing numbers would greatly help in getting wider acceptance for this change, Ravikiran?

With the lock breakdown, Iozone read performance went up by 5.5% on a 2 way x86 xeon box, with two hwifs and two processes reading from disks connected to each hwif. There seems to be a small regression on Iozone write tests -- about 1.5%. Maybe this is measurement noise as there is no apparent reason for this. We plan to run more tests to see if the regression for writes is statistically significant.

Btw, there was some problem with the moving of tuning code in this patchset causing disks not to work in DMA mode. We were fixing that; will publish a newer patchset with test results soon.

Thanks,
Kiran

#14 lore raw
Quoted 24 lines · Ravikiran G Thirumalai
Quoted 15 lines · Bartlomiej Zolnierkiewicz
Quoted 6 lines · Jens Axboe
Quoted 5 lines · Ravikiran G Thirumalai
Quoted 4 lines · Jens Axboe
Quoted 2 lines · Ravikiran G Thirumalai

The following patchset breaks down the global ide_lock to per-hwgroup lock. We have taken the following approach.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Curious, what is the point of this?

I'm asking because I've never heard anyone complain about IDE lock contention and a proper patch usually comes with analysis of why it is needed.

Quoted 7 lines

Since ide_lock spinlock is used for all drives as queue lock and for all controllers as IDE lock I guess that with multiple controllers there is a lot contention on it...

Breaking ide_lock is fine with me however seeing numbers would greatly help in getting wider acceptance for this change, Ravikiran?

Quoted 6 lines

With the lock breakdown, Iozone read performance went up by 5.5% on a 2 way x86 xeon box, with two hwifs and two processes reading from disks connected to each hwif. There seems to be a small regression on Iozone write tests -- about 1.5%. Maybe this is measurement noise as there is no apparent reason for this. We plan to run more tests to see if the regression for writes is statistically significant.

You should run it eg 10 times on each kernel to get a feel for the variance of the results. Were you testing 2 or 4 disks?

Some profile information for both kernels would also be paramount.

Quoted 3 lines

Btw, there was some problem with the moving of tuning code in this patchset causing disks not to work in DMA mode. We were fixing that; will publish a newer patchset with test results soon.

I take it the numbers posted were for DMA enabled on all disks?

Signature · 1 line
Jens Axboe
#15 lore raw
Quoted 5 lines · Jens Axboe

On Tue, Sep 27 2005, Ravikiran G Thirumalai wrote:

You should run it eg 10 times on each kernel to get a feel for the variance of the results. Were you testing 2 or 4 disks?

Yes, I was planning to do that, We were testing with 2 disks.

Quoted 2 lines

...
I take it the numbers posted were for DMA enabled on all disks?

Yes.

Thanks,
Kiran

#16 lore raw
Quoted 8 lines · Ravikiran G Thirumalai
Quoted 5 lines · Jens Axboe

On Tue, Sep 27 2005, Ravikiran G Thirumalai wrote:

You should run it eg 10 times on each kernel to get a feel for the variance of the results. Were you testing 2 or 4 disks?

Yes, I was planning to do that, We were testing with 2 disks.

With 2 disks, the 5.5% sounds like an aweful lot. What block size were the tests run with? To get the file system out of the way, you could run something as simple as a dd on each device. That should show queueing contention (which I guess is the prime area of contention) more isolated. And profiles would still be very interesting to see :-)

Signature · 1 line
Jens Axboe
#17 lore raw
In reply to Ravikiran G Thirumalai
Quoted 2 lines · Ravikiran G Thirumalai

2. Change the core ide code to use hwgroup->lock instead of ide_lock. Deprecate ide_lock (patch 2)

hwgroups and IDE locking requirements are frequently completely unrelated. Its clear from the changes proposed you've not tested on real hardware for each case and you have not studied the documented errata even for the controllers that are publically specified. This is peoples data you are endangering, not just risking the odd crash.

Quoted 2 lines

patches -- this is done now for piix controller only. Eventually, we can change all controllers to remove ide_lock

You could start by fixing the real locking bugs. Or at least trying to get known fixes past the maintainer and going from there.

Alan

#18 lore raw
Quoted 7 lines · Alan Cox
Quoted 2 lines · Ravikiran G Thirumalai

2. Change the core ide code to use hwgroup->lock instead of ide_lock. Deprecate ide_lock (patch 2)

hwgroups and IDE locking requirements are frequently completely unrelated. Its clear from the changes proposed you've not tested on real hardware for each case and you have not studied the documented errata

I tested it on a 2way box with a piix controller. It got through Bonnie++. I have access to piix controllers only, so that was the only controller I changed. I did not read the errata though... :(

Do you think the approach is unsafe, even if the piix tune routine is serialized with a per-driver lock? Bartlomiej?

Thanks,
Kiran

#19 lore raw
Quoted 3 lines · Ravikiran G Thirumalai

I tested it on a 2way box with a piix controller. It got through Bonnie++. I have access to piix controllers only, so that was the only controller I changed. I did not read the errata though... :(

Errata matter a lot unfortunately - older IDE controllers commonly have bugs where changing the set up of one interface affects the other even if a transfer is occuring.

Quoted 2 lines

Do you think the approach is unsafe, even if the piix tune routine is serialized with a per-driver lock? Bartlomiej?

If the documents/errata for the chips are checked then I can see it making sense. I do worry that it may introduce other races because the IDE locking is such a mess but that would be true of any change and eventually has to be addressed.

I think it makes sense, with corrections.