Skip to contentSkip to conversation

Hardware Accelerated MD RAID5: Introduction

PATCH In review Dan Williams 9 participants 55 messages Sep 11, 2006 – Oct 11, 2006

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover Hardware Accelerated MD RAID5: Introduction

Neil,

The following patches implement hardware accelerated raid5 for the Intel Xscale® series of I/O Processors. The MD changes allow stripe operations to run outside the spin lock in a work queue. Hardware acceleration is achieved by using a dma-engine-aware work queue routine instead of the default software only routine.

Since the last release of the raid5 changes many bug fixes and other improvements have been made as a result of stress testing. See the per patch change logs for more information about what was fixed. This release is the first release of the full dma implementation.

The patches touch 3 areas, the md-raid5 driver, the generic dmaengine interface, and a platform device driver for IOPs. The raid5 changes follow your comments concerning making the acceleration implementation similar to how the stripe cache handles I/O requests. The dmaengine changes are the second release of this code. They expand the interface to handle more than memcpy operations, and add a generic raid5-dma client. The iop-adma driver supports dma memcpy, xor, xor zero sum, and memset across all IOP architectures (32x, 33x, and 13xx).

Concerning the context switching performance concerns raised at the previous release, I have observed the following. For the hardware accelerated case it appears that performance is always better with the work queue than without since it allows multiple stripes to be operated on simultaneously. I expect the same for an SMP platform, but so far my testing has been limited to IOPs. For a single-processor non-accelerated configuration I have not observed performance degradation with work queue support enabled, but in the Kconfig option help text I recommend disabling it (CONFIG_MD_RAID456_WORKQUEUE).

Please consider the patches for -mm.

-Dan

[PATCH 01/19] raid5: raid5_do_soft_block_ops
[PATCH 02/19] raid5: move write operations to a workqueue
[PATCH 03/19] raid5: move check parity operations to a workqueue
[PATCH 04/19] raid5: move compute block operations to a workqueue
[PATCH 05/19] raid5: move read completion copies to a workqueue
[PATCH 06/19] raid5: move the reconstruct write expansion operation to a workqueue [PATCH 07/19] raid5: remove compute_block and compute_parity5
[PATCH 08/19] dmaengine: enable multiple clients and operations
[PATCH 09/19] dmaengine: reduce backend address permutations
[PATCH 10/19] dmaengine: expose per channel dma mapping characteristics to clients [PATCH 11/19] dmaengine: add memset as an asynchronous dma operation [PATCH 12/19] dmaengine: dma_async_memcpy_err for DMA engines that do not support memcpy [PATCH 13/19] dmaengine: add support for dma xor zero sum operations [PATCH 14/19] dmaengine: add dma_sync_wait
[PATCH 15/19] dmaengine: raid5 dma client
[PATCH 16/19] dmaengine: Driver for the Intel IOP 32x, 33x, and 13xx RAID engines [PATCH 17/19] iop3xx: define IOP3XX_REG_ADDR[32|16|8] and clean up DMA/AAU defs [PATCH 18/19] iop3xx: Give Linux control over PCI (ATU) initialization [PATCH 19/19] iop3xx: IOP 32x and 33x support for the iop-adma driver

Note, the iop3xx patches apply against the iop3xx platform code re-factoring done by Lennert Buytenhek. His patches are reproduced, with permission, on the Xscale IOP SourceForge site.

Also available on SourceForge:

Linux Symposium Paper: MD RAID Acceleration Support for Asynchronous DMA/XOR Engines
http://prdownloads.sourceforge.net/xscaleiop/ols_paper_2006.pdf?download

Tar archive of the patch set
http://prdownloads.sourceforge.net/xscaleiop/md_raid_accel-2.6.18-rc6.tar.gz?download

[PATCH 01/19] http://prdownloads.sourceforge.net/xscaleiop/md-add-raid5-do-soft-block-ops.patch?download
[PATCH 02/19] http://prdownloads.sourceforge.net/xscaleiop/md-move-write-operations-to-a-workqueue.patch?download
[PATCH 03/19] http://prdownloads.sourceforge.net/xscaleiop/md-move-check-parity-operations-to-a-workqueue.patch?download
[PATCH 04/19] http://prdownloads.sourceforge.net/xscaleiop/md-move-compute-block-operations-to-a-workqueue.patch?download
[PATCH 05/19] http://prdownloads.sourceforge.net/xscaleiop/md-move-read-completion-copies-to-a-workqueue.patch?download
[PATCH 06/19] http://prdownloads.sourceforge.net/xscaleiop/md-move-expansion-operations-to-a-workqueue.patch?download
[PATCH 07/19] http://prdownloads.sourceforge.net/xscaleiop/md-remove-compute_block-and-compute_parity5.patch?download
[PATCH 08/19] http://prdownloads.sourceforge.net/xscaleiop/dmaengine-multiple-clients-and-multiple-operations.patch?download
[PATCH 09/19] http://prdownloads.sourceforge.net/xscaleiop/dmaengine-unite-backend-address-types.patch?download
[PATCH 10/19] http://prdownloads.sourceforge.net/xscaleiop/dmaengine-dma-async-map-page.patch?download
[PATCH 11/19] http://prdownloads.sourceforge.net/xscaleiop/dmaengine-dma-async-memset.patch?download
[PATCH 12/19] http://prdownloads.sourceforge.net/xscaleiop/dmaengine-dma-async-memcpy-err.patch?download
[PATCH 13/19] http://prdownloads.sourceforge.net/xscaleiop/dmaengine-dma-async-zero-sum.patch?download
[PATCH 14/19] http://prdownloads.sourceforge.net/xscaleiop/dmaengine-dma-sync-wait.patch?download
[PATCH 15/19] http://prdownloads.sourceforge.net/xscaleiop/md-raid5-dma-client.patch?download
[PATCH 16/19] http://prdownloads.sourceforge.net/xscaleiop/iop-adma-device-driver.patch?download
[PATCH 17/19] http://prdownloads.sourceforge.net/xscaleiop/iop3xx-register-macro-cleanup.patch?download
[PATCH 18/19] http://prdownloads.sourceforge.net/xscaleiop/iop3xx-pci-initialization.patch?download
[PATCH 19/19] http://prdownloads.sourceforge.net/xscaleiop/iop3xx-adma-support.patch?download

Optimal performance on IOPs is obtained with:
CONFIG_MD_RAID456_WORKQUEUE=y
CONFIG_MD_RAID5_HW_OFFLOAD=y
CONFIG_RAID5_DMA=y
CONFIG_INTEL_IOP_ADMA=y
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#1 lore raw

PATCH 1/19 raid5: raid5_do_soft_block_ops

From: Dan Williams <[email protected]>

raid5_do_soft_block_ops consolidates all the stripe cache maintenance operations into a single routine. The stripe operations are:
* copying data between the stripe cache and user application buffers
* computing blocks to save a disk access, or to recover a missing block
* updating the parity on a write operation (reconstruct write and read-modify-write)
* checking parity correctness

---

 drivers/md/raid5.c         |  289 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/raid/raid5.h |  129 +++++++++++++++++++-
 2 files changed, 415 insertions(+), 3 deletions(-)
drivers/md/raid5.c +289 -0
index 4500660..8fde62b 100644
@@ -1362,6 +1362,295 @@ static int stripe_to_pdidx(sector_t stri
 	return pd_idx;
 }
 
+/*
+ * raid5_do_soft_block_ops - perform block memory operations on stripe data
+ * outside the spin lock.
+ */
+static void raid5_do_soft_block_ops(void *stripe_head_ref)
+{
+	struct stripe_head *sh = stripe_head_ref;
+	int i, pd_idx = sh->pd_idx, disks = sh->disks;
+	void *ptr[MAX_XOR_BLOCKS];
+	int overlap=0, work=0, written=0, compute=0, dd_idx=0;
+	int pd_uptodate=0;
+	unsigned long state, ops_state, ops_state_orig;
+	raid5_conf_t *conf = sh->raid_conf;
+
+	/* take a snapshot of what needs to be done at this point in time */
+	spin_lock(&sh->lock);
+	state = sh->state;
+	ops_state_orig = ops_state = sh->ops.state;
+	spin_unlock(&sh->lock);
+
+	if (test_bit(STRIPE_OP_BIOFILL, &state)) {
+		struct bio *return_bi=NULL;
+
+		for (i=disks ; i-- ;) {
+			struct r5dev *dev = &sh->dev[i];
+			if (test_bit(R5_ReadReq, &dev->flags)) {
+				struct bio *rbi, *rbi2;
+				PRINTK("%s: stripe %llu STRIPE_OP_BIOFILL op_state: %lx disk: %d\n",
+					__FUNCTION__, (unsigned long long)sh->sector,
+					ops_state, i);
+				spin_lock_irq(&conf->device_lock);
+				rbi = dev->toread;
+				dev->toread = NULL;
+				spin_unlock_irq(&conf->device_lock);
+				overlap++;
+				while (rbi && rbi->bi_sector < dev->sector + STRIPE_SECTORS) {
+					copy_data(0, rbi, dev->page, dev->sector);
+					rbi2 = r5_next_bio(rbi, dev->sector);
+					spin_lock_irq(&conf->device_lock);
+					if (--rbi->bi_phys_segments == 0) {
+						rbi->bi_next = return_bi;
+						return_bi = rbi;
+					}
+					spin_unlock_irq(&conf->device_lock);
+					rbi = rbi2;
+				}
+				dev->read = return_bi;
+			}
+		}
+		if (overlap) {
+			set_bit(STRIPE_OP_BIOFILL_Done, &ops_state);
+			work++;
+		}
+	}
+
+	if (test_bit(STRIPE_OP_COMPUTE, &state)) {
+		for (i=disks ; i-- ;) {
+			struct r5dev *dev = &sh->dev[i];
+			if (test_bit(R5_ComputeReq, &dev->flags)) {
+				dd_idx = i;
+				i = -1;
+				break;
+			}
+		}
+		BUG_ON(i >= 0);
+		PRINTK("%s: stripe %llu STRIPE_OP_COMPUTE op_state: %lx block: %d\n",
+			__FUNCTION__, (unsigned long long)sh->sector,
+			ops_state, dd_idx);
+		ptr[0] = page_address(sh->dev[dd_idx].page);
+
+		if (test_and_clear_bit(STRIPE_OP_COMPUTE_Prep, &ops_state)) {
+			memset(ptr[0], 0, STRIPE_SIZE);
+			set_bit(STRIPE_OP_COMPUTE_Parity, &ops_state);
+		}
+
+		if (test_and_clear_bit(STRIPE_OP_COMPUTE_Parity, &ops_state)) {
+			int count = 1;
+			for (i = disks ; i--; ) {
+				struct r5dev *dev = &sh->dev[i];
+				void *p;
+				if (i == dd_idx)
+					continue;
+				p = page_address(dev->page);
+				ptr[count++] = p;
+
+				check_xor();
+			}
+			if (count != 1)
+				xor_block(count, STRIPE_SIZE, ptr);
+
+			work++;
+			compute++;
+			set_bit(STRIPE_OP_COMPUTE_Done, &ops_state);
+		}
+	}
+
+	if (test_bit(STRIPE_OP_RMW, &state)) {
+		BUG_ON(test_bit(STRIPE_OP_RCW, &state));
+
+		PRINTK("%s: stripe %llu STRIPE_OP_RMW op_state: %lx\n",
+			__FUNCTION__, (unsigned long long)sh->sector,
+			ops_state);
+
+		ptr[0] = page_address(sh->dev[pd_idx].page);
+
+		if (test_and_clear_bit(STRIPE_OP_RMW_ParityPre, &ops_state)) {
+			int count = 1;
+
+			for (i=disks ; i-- ;) {
+				struct r5dev *dev = &sh->dev[i];
+				struct bio *chosen;
+
+				/* Only process blocks that are known to be uptodate */
+				if (dev->towrite && test_bit(R5_RMWReq, &dev->flags)) {
+					ptr[count++] = page_address(dev->page);
+
+					spin_lock(&sh->lock);
+					chosen = dev->towrite;
+					dev->towrite = NULL;
+					BUG_ON(dev->written);
+					dev->written = chosen;
+					spin_unlock(&sh->lock);
+
+					overlap++;
+
+					check_xor();
+				}
+			}
+			if (count != 1)
+				xor_block(count, STRIPE_SIZE, ptr);
+			set_bit(STRIPE_OP_RMW_Drain, &ops_state);
+		}
+		if (test_and_clear_bit(STRIPE_OP_RMW_Drain, &ops_state)) {
+			for (i=disks ; i-- ;) {
+				struct r5dev *dev = &sh->dev[i];
+				struct bio *wbi = dev->written;
+
+				if (dev->written)
+					written++;
+
+				while (wbi && wbi->bi_sector < dev->sector + STRIPE_SECTORS) {
+					copy_data(1, wbi, dev->page, dev->sector);
+					wbi = r5_next_bio(wbi, dev->sector);
+				}
+			}
+			set_bit(STRIPE_OP_RMW_ParityPost, &ops_state);
+		}
+		if (test_and_clear_bit(STRIPE_OP_RMW_ParityPost, &ops_state)) {
+			int count = 1;
+			for (i=disks ; i-- ;) {
+				struct r5dev *dev = &sh->dev[i];
+				if (dev->written) {
+					ptr[count++] = page_address(dev->page);
+					check_xor();
+				}
+			}
+			if (count != 1)
+				xor_block(count, STRIPE_SIZE, ptr);
+
+			work++;
+			pd_uptodate++;
+			set_bit(STRIPE_OP_RMW_Done, &ops_state);
+		}
+
+	}
+
+	if (test_bit(STRIPE_OP_RCW, &state)) {
+		BUG_ON(test_bit(STRIPE_OP_RMW, &state));
+
+		PRINTK("%s: stripe %llu STRIPE_OP_RCW op_state: %lx\n",
+			__FUNCTION__, (unsigned long long)sh->sector,
+			ops_state);
+
+		ptr[0] = page_address(sh->dev[pd_idx].page);
+
+		if (test_and_clear_bit(STRIPE_OP_RCW_Drain, &ops_state)) {
+			for (i=disks ; i-- ;) {
+				struct r5dev *dev = &sh->dev[i];
+				struct bio *chosen;
+				struct bio *wbi;
+
+				if (i!=pd_idx && dev->towrite &&
+					test_bit(R5_LOCKED, &dev->flags)) {
+
+					spin_lock(&sh->lock);
+					chosen = dev->towrite;
+					dev->towrite = NULL;
+					spin_unlock(&sh->lock);
+
+					BUG_ON(dev->written);
+					wbi = dev->written = chosen;
+
+					overlap++;
+					written++;
+
+					while (wbi && wbi->bi_sector < dev->sector + STRIPE_SECTORS) {
+						copy_data(1, wbi, dev->page, dev->sector);
+						wbi = r5_next_bio(wbi, dev->sector);
+					}
+				} else if (i==pd_idx)
+					memset(ptr[0], 0, STRIPE_SIZE);
+			}
+			set_bit(STRIPE_OP_RCW_Parity, &ops_state);
+		}
+		if (test_and_clear_bit(STRIPE_OP_RCW_Parity, &ops_state)) {
+			int count = 1;
+			for (i=disks; i--;)
+				if (i != pd_idx) {
+					ptr[count++] = page_address(sh->dev[i].page);
+					check_xor();
+				}
+			if (count != 1)
+				xor_block(count, STRIPE_SIZE, ptr);
+
+			work++;
+			pd_uptodate++;
+			set_bit(STRIPE_OP_RCW_Done, &ops_state);
+
+		}
+	}
+
+	if (test_bit(STRIPE_OP_CHECK, &state)) {
+		PRINTK("%s: stripe %llu STRIPE_OP_CHECK op_state: %lx\n",
+		__FUNCTION__, (unsigned long long)sh->sector,
+		ops_state);
+
+		ptr[0] = page_address(sh->dev[pd_idx].page);
+
+		if (test_and_clear_bit(STRIPE_OP_CHECK_Gen, &ops_state)) {
+			int count = 1;
+			for (i=disks; i--;)
+				if (i != pd_idx) {
+					ptr[count++] = page_address(sh->dev[i].page);
+					check_xor();
+				}
+			if (count != 1)
+				xor_block(count, STRIPE_SIZE, ptr);
+
+			set_bit(STRIPE_OP_CHECK_Verify, &ops_state);
+		}
+		if (test_and_clear_bit(STRIPE_OP_CHECK_Verify, &ops_state)) {
+			if (page_is_zero(sh->dev[pd_idx].page))
+				set_bit(STRIPE_OP_CHECK_IsZero, &ops_state);
+
+			work++;
+			set_bit(STRIPE_OP_CHECK_Done, &ops_state);
+		}
+	}
+
+	spin_lock(&sh->lock);
+	/* Update the state of operations:
+	 * -clear incoming requests
+	 * -preserve output status (i.e. done status / check result)
+	 * -preserve requests added since 'ops_state_orig' was set
+	 */
+	sh->ops.state ^= (ops_state_orig & ~STRIPE_OP_COMPLETION_MASK);
+	sh->ops.state |= ops_state;
+
+	if (pd_uptodate)
+		set_bit(R5_UPTODATE, &sh->dev[sh->pd_idx].flags);
+
+	if (written)
+		for (i=disks ; i-- ;) {
+			struct r5dev *dev = &sh->dev[i];
+			if (dev->written)
+				set_bit(R5_UPTODATE, &dev->flags);
+		}
+
+	if (overlap)
+		for (i= disks; i-- ;) {
+			struct r5dev *dev = &sh->dev[i];
+			if (test_and_clear_bit(R5_Overlap, &dev->flags))
+				wake_up(&sh->raid_conf->wait_for_overlap);
+		}
+
+	if (compute) {
+		clear_bit(R5_ComputeReq, &sh->dev[dd_idx].flags);
+		set_bit(R5_UPTODATE, &sh->dev[dd_idx].flags);
+	}
+
+	sh->ops.pending -= work;
+	BUG_ON(sh->ops.pending < 0);
+	clear_bit(STRIPE_OP_QUEUED, &sh->state);
+	set_bit(STRIPE_HANDLE, &sh->state);
+	queue_raid_work(sh);
+	spin_unlock(&sh->lock);
+
+	release_stripe(sh);
+}
 
 /*
  * handle_stripe - do things to a stripe.
include/linux/raid/raid5.h +126 -3

179 lines not on this page · raw message

#2 lore raw

Re: [PATCH 01/19] raid5: raid5_do_soft_block_ops

Quoted 22 lines · Dan Williams

From: Dan Williams <[email protected]>

raid5_do_soft_block_ops consolidates all the stripe cache maintenance operations into a single routine. The stripe operations are:
* copying data between the stripe cache and user application buffers
* computing blocks to save a disk access, or to recover a missing block
* updating the parity on a write operation (reconstruct write and read-modify-write)
* checking parity correctness

---

 drivers/md/raid5.c         |  289 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/raid/raid5.h |  129 +++++++++++++++++++-
 2 files changed, 415 insertions(+), 3 deletions(-)
drivers/md/raid5.c +0 -0
index 4500660..8fde62b 100644

drivers/md/raid5.c static int stripe_to_pdidx(sector_t stri · line 1369

Quoted 8 lines
drivers/md/raid5.c +5 -0
@@ -1362,6 +1362,295 @@ static int stripe_to_pdidx(sector_t stri
 	return pd_idx;
 }
 
+/*
+ * raid5_do_soft_block_ops - perform block memory operations on stripe data
+ * outside the spin lock.
+ */
+static void raid5_do_soft_block_ops(void *stripe_head_ref)

This function absolutely must be broken up into multiple functions, presumably one per operation.

Jeff

#3 lore raw

PATCH 2/19 raid5: move write operations to a workqueue

In reply to Dan Williams

From: Dan Williams <[email protected]>

Enable handle_stripe5 to pass off write operations to raid5_do_soft_blocks_ops (which can be run as a workqueue). The operations moved are reconstruct-writes and read-modify-writes formerly handled by compute_parity5.

Changelog:
* moved raid5_do_soft_block_ops changes into a separate patch
* changed handle_write_operations5 to only initiate write operations, which prevents new writes from being requested while the current one is in flight
* all blocks undergoing a write are now marked locked and !uptodate at the beginning of the write operation
* blocks undergoing a read-modify-write need a request flag to distinguish them from blocks that are locked for reading. Reconstruct-writes still use the R5_LOCKED bit to select blocks for the operation
* integrated the work queue Kconfig option

---

 drivers/md/Kconfig         |   21 +++++
 drivers/md/raid5.c         |  192 ++++++++++++++++++++++++++++++++++++++------
 include/linux/raid/raid5.h |    3 +
 3 files changed, 190 insertions(+), 26 deletions(-)
drivers/md/Kconfig +21 -0
index bf869ed..2a16b3b 100644
@@ -162,6 +162,27 @@ config MD_RAID5_RESHAPE
 	  There should be enough spares already present to make the new
 	  array workable.
 
+config MD_RAID456_WORKQUEUE
+	depends on MD_RAID456
+	bool "Offload raid work to a workqueue from raid5d"
+	---help---
+	  This option enables raid work (block copy and xor operations)
+	  to run in a workqueue.  If your platform has a high context
+	  switch penalty say N.  If you are using hardware offload or
+	  are running on an SMP platform say Y.
+
+	  If unsure say, Y.
+
+config MD_RAID456_WORKQUEUE_MULTITHREAD
+	depends on MD_RAID456_WORKQUEUE && SMP
+	bool "Enable multi-threaded raid processing"
+	default y
+	---help---
+	  This option controls whether the raid workqueue will be multi-
+	  threaded or single threaded.
+
+	  If unsure say, Y.
+
 config MD_MULTIPATH
 	tristate "Multipath I/O support"
 	depends on BLK_DEV_MD
drivers/md/raid5.c +166 -26

264 lines not on this page · raw message

include/linux/raid/raid5.h +3 -0
index c8a315b..31ae55c 100644
@@ -3,6 +3,7 @@ #define _RAID5_H
 
 #include <linux/raid/md.h>
 #include <linux/raid/xor.h>
+#include <linux/workqueue.h>
 
 /*
  *
@@ -333,6 +334,7 @@ struct raid5_private_data {
 	atomic_t		preread_active_stripes; /* stripes with scheduled io */
 
 	atomic_t		reshape_stripes; /* stripes with pending writes for reshape */
+
 	#ifdef CONFIG_MD_RAID456_WORKQUEUE
 	struct workqueue_struct *block_ops_queue;
 	#endif
@@ -376,6 +378,7 @@ struct raid5_private_data {
 typedef struct raid5_private_data raid5_conf_t;
 
 #define mddev_to_conf(mddev) ((raid5_conf_t *) mddev->private)
+
 /* must be called under the stripe lock */
 static inline void queue_raid_work(struct stripe_head *sh)
 {
#4 lore raw

Re: [PATCH 02/19] raid5: move write operations to a workqueue

Quoted 44 lines · Dan Williams

From: Dan Williams <[email protected]>

Enable handle_stripe5 to pass off write operations to raid5_do_soft_blocks_ops (which can be run as a workqueue). The operations moved are reconstruct-writes and read-modify-writes formerly handled by compute_parity5.

Changelog:
* moved raid5_do_soft_block_ops changes into a separate patch
* changed handle_write_operations5 to only initiate write operations, which prevents new writes from being requested while the current one is in flight
* all blocks undergoing a write are now marked locked and !uptodate at the beginning of the write operation
* blocks undergoing a read-modify-write need a request flag to distinguish them from blocks that are locked for reading. Reconstruct-writes still use the R5_LOCKED bit to select blocks for the operation
* integrated the work queue Kconfig option

---

 drivers/md/Kconfig         |   21 +++++
 drivers/md/raid5.c         |  192 ++++++++++++++++++++++++++++++++++++++------
 include/linux/raid/raid5.h |    3 +
 3 files changed, 190 insertions(+), 26 deletions(-)
drivers/md/Kconfig +10 -0
index bf869ed..2a16b3b 100644
@@ -162,6 +162,27 @@ config MD_RAID5_RESHAPE
 	  There should be enough spares already present to make the new
 	  array workable.
 
+config MD_RAID456_WORKQUEUE
+	depends on MD_RAID456
+	bool "Offload raid work to a workqueue from raid5d"
+	---help---
+	  This option enables raid work (block copy and xor operations)
+	  to run in a workqueue.  If your platform has a high context
+	  switch penalty say N.  If you are using hardware offload or
+	  are running on an SMP platform say Y.
+
+	  If unsure say, Y.

drivers/md/Kconfig config MD_RAID5_RESHAPE · line 184

Quoted 10 lines
drivers/md/Kconfig +10 -0
+
+config MD_RAID456_WORKQUEUE_MULTITHREAD
+	depends on MD_RAID456_WORKQUEUE && SMP
+	bool "Enable multi-threaded raid processing"
+	default y
+	---help---
+	  This option controls whether the raid workqueue will be multi-
+	  threaded or single threaded.
+
+	  If unsure say, Y.

In the final patch that gets merged, these configuration options should go away. We are very anti-#ifdef in Linux, for a variety of reasons. In this particular instance, code complexity increases and maintainability decreases as the #ifdef forest grows.

Jeff

#5 lore raw

PATCH 3/19 raid5: move check parity operations to a workqueue

In reply to Dan Williams

From: Dan Williams <[email protected]>

Enable handle_stripe5 to pass off check parity operations to raid5_do_soft_block_ops formerly handled by compute_parity5.

Changelog:
* removed handle_check_operations5. All logic moved into handle_stripe5 so that we do not need to go through the initiation logic to end the operation.
* clear the uptodate bit on the parity block
* hold off check operations if a parity dependent operation is in flight like a write

---

 drivers/md/raid5.c |   60 ++++++++++++++++++++++++++++++++++++----------------
 1 files changed, 42 insertions(+), 18 deletions(-)
drivers/md/raid5.c +42 -18
index e39d248..24ed4d8 100644
@@ -2121,35 +2121,59 @@ #endif
 			locked += handle_write_operations5(sh, rcw);
 	}
 
-	/* maybe we need to check and possibly fix the parity for this stripe
-	 * Any reads will already have been scheduled, so we just see if enough data
-	 * is available
+	/* 1/ Maybe we need to check and possibly fix the parity for this stripe.
+	 *    Any reads will already have been scheduled, so we just see if enough data
+	 *    is available.
+	 * 2/ Hold off parity checks while parity dependent operations are in flight
+	 *    (RCW and RMW are protected by 'locked')
 	 */
-	if (syncing && locked == 0 &&
-	    !test_bit(STRIPE_INSYNC, &sh->state)) {
+	if ((syncing && locked == 0 &&
+	    !test_bit(STRIPE_INSYNC, &sh->state)) ||
+	    	test_bit(STRIPE_OP_CHECK, &sh->state)) {
+
 		set_bit(STRIPE_HANDLE, &sh->state);
+		/* Take one of the following actions:
+		 * 1/ start a check parity operation if (uptodate == disks)
+		 * 2/ finish a check parity operation and act on the result
+		 */
 		if (failed == 0) {
-			BUG_ON(uptodate != disks);
-			compute_parity5(sh, CHECK_PARITY);
-			uptodate--;
-			if (page_is_zero(sh->dev[sh->pd_idx].page)) {
-				/* parity is correct (on disc, not in buffer any more) */
-				set_bit(STRIPE_INSYNC, &sh->state);
-			} else {
-				conf->mddev->resync_mismatches += STRIPE_SECTORS;
-				if (test_bit(MD_RECOVERY_CHECK, &conf->mddev->recovery))
-					/* don't try to repair!! */
+			if (!test_bit(STRIPE_OP_CHECK, &sh->state)) {
+				BUG_ON(uptodate != disks);
+				set_bit(STRIPE_OP_CHECK, &sh->state);
+				set_bit(STRIPE_OP_CHECK_Gen, &sh->ops.state);
+				clear_bit(R5_UPTODATE, &sh->dev[sh->pd_idx].flags);
+				sh->ops.pending++;
+				uptodate--;
+			} else if (test_and_clear_bit(STRIPE_OP_CHECK_Done, &sh->ops.state)) {
+				clear_bit(STRIPE_OP_CHECK, &sh->state);
+
+				if (test_and_clear_bit(STRIPE_OP_CHECK_IsZero,
+							&sh->ops.state))
+					/* parity is correct (on disc, not in buffer any more) */
 					set_bit(STRIPE_INSYNC, &sh->state);
 				else {
-					compute_block(sh, sh->pd_idx);
-					uptodate++;
+					conf->mddev->resync_mismatches += STRIPE_SECTORS;
+					if (test_bit(MD_RECOVERY_CHECK, &conf->mddev->recovery))
+						/* don't try to repair!! */
+						set_bit(STRIPE_INSYNC, &sh->state);
+					else {
+						compute_block(sh, sh->pd_idx);
+						uptodate++;
+					}
 				}
 			}
 		}
-		if (!test_bit(STRIPE_INSYNC, &sh->state)) {
+
+		/* Wait for check parity operations to complete
+		 * before write-back
+		 */
+		if (!test_bit(STRIPE_INSYNC, &sh->state) &&
+			!test_bit(STRIPE_OP_CHECK, &sh->state)) {
+
 			/* either failed parity check, or recovery is happening */
 			if (failed==0)
 				failed_num = sh->pd_idx;
+
 			dev = &sh->dev[failed_num];
 			BUG_ON(!test_bit(R5_UPTODATE, &dev->flags));
 			BUG_ON(uptodate != disks);
#6 lore raw

PATCH 4/19 raid5: move compute block operations to a workqueue

In reply to Dan Williams

From: Dan Williams <[email protected]>

Enable handle_stripe5 to pass off compute block operations to raid5_do_soft_block_ops, formerly handled by compute_block.

Here are a few notes about the new flags R5_ComputeReq and STRIPE_OP_COMPUTE_Recover:

Previously, when handle_stripe5 found a block that needed to be computed it updated it in the same step. Now that these operations are separated (across multiple calls to handle_stripe5), a R5_ComputeReq flag is needed to tell other parts of handle_stripe5 to treat the block under computation as if it were up to date. The order of events in the work queue ensures that the block is indeed up to date before performing further operations.

STRIPE_OP_COMPUTE_Recover_pd was added to track when the parity block is being computed due to a failed parity check. This allows the code in handle_stripe5 that produces requests for check_parity and compute_block operations to be separate from the code that consumes the result.

Changelog:
* count blocks under computation as uptodate
* removed handle_compute_operations5. All logic moved into handle_stripe5 so that we do not need to go through the initiation logic to end the operation.
* since the write operations mark blocks !uptodate we hold off the code to compute/read blocks until it completes.
* new compute block operations and reads are held off while a compute is in flight
* do not compute a block while a check parity operation is pending, and do not start a new check parity operation while a compute operation is pending
* STRIPE_OP_Recover_pd holds off the clearing of the STRIPE_OP_COMPUTE state. This allows the transition to be handled by the check parity logic that writes recomputed parity to disk.

---

 drivers/md/raid5.c |  153 ++++++++++++++++++++++++++++++++++++----------------
 1 files changed, 107 insertions(+), 46 deletions(-)
drivers/md/raid5.c +107 -46
index 24ed4d8..0c39203 100644
@@ -1300,7 +1300,8 @@ static int handle_write_operations5(stru
 		}
 	} else {
 		/* enter stage 1 of read modify write operation */
-		BUG_ON(!test_bit(R5_UPTODATE, &sh->dev[pd_idx].flags));
+		BUG_ON(!(test_bit(R5_UPTODATE, &sh->dev[pd_idx].flags) ||
+			test_bit(R5_ComputeReq, &sh->dev[pd_idx].flags)));
 
 		set_bit(STRIPE_OP_RMW, &sh->state);
 		set_bit(STRIPE_OP_RMW_ParityPre, &sh->ops.state);
@@ -1314,7 +1315,8 @@ static int handle_write_operations5(stru
 			 * so we distinguish these blocks by the RMWReq bit
 			 */
 			if (dev->towrite &&
-			    test_bit(R5_UPTODATE, &dev->flags)) {
+			    (test_bit(R5_UPTODATE, &dev->flags) ||
+			    test_bit(R5_ComputeReq, &dev->flags))) {
 				set_bit(R5_RMWReq, &dev->flags);
 				set_bit(R5_LOCKED, &dev->flags);
 				clear_bit(R5_UPTODATE, &dev->flags);
@@ -1748,7 +1750,7 @@ static void handle_stripe5(struct stripe
 	int i;
 	int syncing, expanding, expanded;
 	int locked=0, uptodate=0, to_read=0, to_write=0, failed=0, written=0;
-	int non_overwrite=0, write_complete=0;
+	int compute=0, non_overwrite=0, write_complete=0;
 	int failed_num=0;
 	struct r5dev *dev;
 
@@ -1799,7 +1801,7 @@ static void handle_stripe5(struct stripe
 		/* now count some things */
 		if (test_bit(R5_LOCKED, &dev->flags)) locked++;
 		if (test_bit(R5_UPTODATE, &dev->flags)) uptodate++;
-
+		if (test_bit(R5_ComputeReq, &dev->flags)) BUG_ON(++compute > 1);
 		
 		if (dev->toread) to_read++;
 		if (dev->towrite) {
@@ -1955,40 +1957,83 @@ static void handle_stripe5(struct stripe
 	 * parity, or to satisfy requests
 	 * or to load a block that is being partially written.
 	 */
-	if (to_read || non_overwrite || (syncing && (uptodate < disks)) || expanding) {
-		for (i=disks; i--;) {
-			dev = &sh->dev[i];
-			if (!test_bit(R5_LOCKED, &dev->flags) && !test_bit(R5_UPTODATE, &dev->flags) &&
-			    (dev->toread ||
-			     (dev->towrite && !test_bit(R5_OVERWRITE, &dev->flags)) ||
-			     syncing ||
-			     expanding ||
-			     (failed && (sh->dev[failed_num].toread ||
-					 (sh->dev[failed_num].towrite && !test_bit(R5_OVERWRITE, &sh->dev[failed_num].flags))))
-				    )
-				) {
-				/* we would like to get this block, possibly
-				 * by computing it, but we might not be able to
+	if (to_read || non_overwrite || (syncing && (uptodate + compute < disks)) || expanding ||
+		test_bit(STRIPE_OP_COMPUTE, &sh->state)) {
+		/* Finish any pending compute operations.  Parity recovery implies
+		 * a write-back which is handled later on in this routine
+		 */
+		if (test_bit(STRIPE_OP_COMPUTE, &sh->state) &&
+			test_bit(STRIPE_OP_COMPUTE_Done, &sh->ops.state) &&
+			!test_bit(STRIPE_OP_COMPUTE_Recover_pd, &sh->ops.state)) {
+			clear_bit(STRIPE_OP_COMPUTE, &sh->state);
+			clear_bit(STRIPE_OP_COMPUTE_Done, &sh->ops.state);
+		}
+		
+		/* blocks being written are temporarily !UPTODATE */
+		if (!test_bit(STRIPE_OP_COMPUTE, &sh->state) &&
+			!test_bit(STRIPE_OP_RCW, &sh->state) &&
+			!test_bit(STRIPE_OP_RMW, &sh->state)) {
+			for (i=disks; i--;) {
+				dev = &sh->dev[i];
+
+				/* don't schedule compute operations or reads on
+				 * the parity block while a check is in flight
 				 */
-				if (uptodate == disks-1) {
-					PRINTK("Computing block %d\n", i);
-					compute_block(sh, i);
-					uptodate++;
-				} else if (test_bit(R5_Insync, &dev->flags)) {
-					set_bit(R5_LOCKED, &dev->flags);
-					set_bit(R5_Wantread, &dev->flags);
+				if ((i == sh->pd_idx) && test_bit(STRIPE_OP_CHECK, &sh->state))
+					continue;
+
+				if (!test_bit(R5_LOCKED, &dev->flags) && !test_bit(R5_UPTODATE, &dev->flags) &&
+				     (dev->toread ||
+				     (dev->towrite && !test_bit(R5_OVERWRITE, &dev->flags)) ||
+				     syncing ||
+				     expanding ||
+				     (failed && (sh->dev[failed_num].toread ||
+						 (sh->dev[failed_num].towrite &&
+						 	!test_bit(R5_OVERWRITE, &sh->dev[failed_num].flags))))
+					    )
+					) {
+					/* 1/ We would like to get this block, possibly
+					 * by computing it, but we might not be able to.
+					 *
+					 * 2/ Since parity check operations make the parity
+					 * block !uptodate it will need to be refreshed
+					 * before any compute operations on data disks are
+					 * scheduled.
+					 *
+					 * 3/ We hold off parity block re-reads until check
+					 * operations have quiesced.
+					 */
+					if ((uptodate == disks-1) && !test_bit(STRIPE_OP_CHECK, &sh->state)) {
+						set_bit(STRIPE_OP_COMPUTE, &sh->state);
+						set_bit(STRIPE_OP_COMPUTE_Prep, &sh->ops.state);
+						set_bit(R5_ComputeReq, &dev->flags);
+						sh->ops.pending++;
+						/* Careful: from this point on 'uptodate' is in the eye of the
+						 * workqueue which services 'compute' operations before writes.
+						 * R5_ComputeReq flags blocks that will be R5_UPTODATE
+						 * in the work queue.
+						 */
+						uptodate++;
+					} else if ((uptodate < disks-1) && test_bit(R5_Insync, &dev->flags)) {
+						/* Note: we hold off compute operations while checks are in flight,
+						 * but we still prefer 'compute' over 'read' hence we only read if
+						 * (uptodate < disks-1)
+						 */
+						set_bit(R5_LOCKED, &dev->flags);
+						set_bit(R5_Wantread, &dev->flags);
 #if 0
-					/* if I am just reading this block and we don't have
-					   a failed drive, or any pending writes then sidestep the cache */
-					if (sh->bh_read[i] && !sh->bh_read[i]->b_reqnext &&
-					    ! syncing && !failed && !to_write) {
-						sh->bh_cache[i]->b_page =  sh->bh_read[i]->b_page;
-						sh->bh_cache[i]->b_data =  sh->bh_read[i]->b_data;
-					}
+						/* if I am just reading this block and we don't have
+						   a failed drive, or any pending writes then sidestep the cache */
+						if (sh->bh_read[i] && !sh->bh_read[i]->b_reqnext &&
+						    ! syncing && !failed && !to_write) {
+							sh->bh_cache[i]->b_page =  sh->bh_read[i]->b_page;
+							sh->bh_cache[i]->b_data =  sh->bh_read[i]->b_data;
+						}
 #endif
-					locked++;
-					PRINTK("Reading block %d (sync=%d)\n", 
-						i, syncing);
+						locked++;
+						PRINTK("Reading block %d (sync=%d)\n", 
+							i, syncing);
+					}
 				}
 			}
 		}
@@ -2055,7 +2100,7 @@ #if 0
 || sh->bh_page[i]!=bh->b_page
 #endif
 				    ) &&
-			    !test_bit(R5_UPTODATE, &dev->flags)) {
+			    !(test_bit(R5_UPTODATE, &dev->flags) || test_bit(R5_ComputeReq, &dev->flags))) {
 				if (test_bit(R5_Insync, &dev->flags)
 /*				    && !(!mddev->insync && i == sh->pd_idx) */
 					)
@@ -2069,7 +2114,7 @@ #if 0
 || sh->bh_page[i] != bh->b_page
 #endif
 				    ) &&
-			    !test_bit(R5_UPTODATE, &dev->flags)) {
+			    !(test_bit(R5_UPTODATE, &dev->flags) || test_bit(R5_ComputeReq, &dev->flags))) {
 				if (test_bit(R5_Insync, &dev->flags)) rcw++;
 				else rcw += 2*disks;
 			}
@@ -2082,7 +2127,8 @@ #endif
 			for (i=disks; i--;) {
 				dev = &sh->dev[i];
 				if ((dev->towrite || i == sh->pd_idx) &&
-				    !test_bit(R5_LOCKED, &dev->flags) && !test_bit(R5_UPTODATE, &dev->flags) &&
+				    !test_bit(R5_LOCKED, &dev->flags) &&
+				    !(test_bit(R5_UPTODATE, &dev->flags) || test_bit(R5_ComputeReq, &dev->flags)) &&
 				    test_bit(R5_Insync, &dev->flags)) {
 					if (test_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
 					{
@@ -2101,7 +2147,8 @@ #endif
 			for (i=disks; i--;) {
 				dev = &sh->dev[i];
 				if (!test_bit(R5_OVERWRITE, &dev->flags) && i != sh->pd_idx &&
-				    !test_bit(R5_LOCKED, &dev->flags) && !test_bit(R5_UPTODATE, &dev->flags) &&
+				    !test_bit(R5_LOCKED, &dev->flags) &&
+				    !(test_bit(R5_UPTODATE, &dev->flags) || test_bit(R5_ComputeReq, &dev->flags)) &&
 				    test_bit(R5_Insync, &dev->flags)) {
 					if (test_bit(STRIPE_PREREAD_ACTIVE, &sh->state))
 					{
@@ -2127,16 +2174,19 @@ #endif
 	 * 2/ Hold off parity checks while parity dependent operations are in flight
 	 *    (RCW and RMW are protected by 'locked')
 	 */
-	if ((syncing && locked == 0 &&
-	    !test_bit(STRIPE_INSYNC, &sh->state)) ||
-	    	test_bit(STRIPE_OP_CHECK, &sh->state)) {
+	if ((syncing && locked == 0 && !test_bit(STRIPE_OP_COMPUTE, &sh->state) &&
+		!test_bit(STRIPE_INSYNC, &sh->state)) ||
+	    	test_bit(STRIPE_OP_CHECK, &sh->state) ||
+	    	test_bit(STRIPE_OP_COMPUTE_Recover_pd, &sh->ops.state)) {
 
 		set_bit(STRIPE_HANDLE, &sh->state);
 		/* Take one of the following actions:
 		 * 1/ start a check parity operation if (uptodate == disks)
 		 * 2/ finish a check parity operation and act on the result
+		 * 3/ skip to the writeback section if we previously 
+		 *    initiated a recovery operation
 		 */
-		if (failed == 0) {
+		if (failed == 0 && !test_bit(STRIPE_OP_COMPUTE_Recover_pd, &sh->ops.state)) {
 			if (!test_bit(STRIPE_OP_CHECK, &sh->state)) {
 				BUG_ON(uptodate != disks);
 				set_bit(STRIPE_OP_CHECK, &sh->state);
@@ -2157,18 +2207,29 @@ #endif
 						/* don't try to repair!! */
 						set_bit(STRIPE_INSYNC, &sh->state);
 					else {
-						compute_block(sh, sh->pd_idx);
+						set_bit(STRIPE_OP_COMPUTE, &sh->state);
+						set_bit(STRIPE_OP_COMPUTE_Recover_pd, &sh->ops.state);
+						set_bit(STRIPE_OP_COMPUTE_Prep, &sh->ops.state);
+						set_bit(R5_ComputeReq, &sh->dev[sh->pd_idx].flags);
+						sh->ops.pending++;
 						uptodate++;
 					}
 				}
 			}
 		}
+		if (test_bit(STRIPE_OP_COMPUTE_Done, &sh->ops.state) &&
+			test_bit(STRIPE_OP_COMPUTE_Recover_pd, &sh->ops.state)) {
+			clear_bit(STRIPE_OP_COMPUTE, &sh->state);
+			clear_bit(STRIPE_OP_COMPUTE_Done, &sh->ops.state);
+			clear_bit(STRIPE_OP_COMPUTE_Recover_pd, &sh->ops.state);
+		}
 
-		/* Wait for check parity operations to complete
+		/* Wait for check parity and compute block operations to complete
 		 * before write-back
 		 */
 		if (!test_bit(STRIPE_INSYNC, &sh->state) &&
-			!test_bit(STRIPE_OP_CHECK, &sh->state)) {
+			!test_bit(STRIPE_OP_CHECK, &sh->state) &&
+			!test_bit(STRIPE_OP_COMPUTE, &sh->state)) {
 
 			/* either failed parity check, or recovery is happening */
 			if (failed==0)
#7 lore raw

PATCH 5/19 raid5: move read completion copies to a workqueue

In reply to Dan Williams

From: Dan Williams <[email protected]>

Enable handle_stripe5 to hand off the memory copy operations that satisfy read requests to raid5_do_soft_blocks_ops, formerly this was handled in line within handle_stripe5.

It adds a 'read' (past tense) pointer to the r5dev structure
to to track reads that have been offloaded to the workqueue. When the copy operation is complete the 'read' pointer is reused as the return_bi for the bi_end_io() call.

Changelog:
* dev->read only holds reads that have been satisfied, previously it doubled as a request queue to the operations routine
* added R5_ReadReq to mark the blocks that belong to a given bio fill operation
* requested reads no longer count towards the 'to_read' count, 'to_fill' tracks the number of requested reads

---

 drivers/md/raid5.c |   67 +++++++++++++++++++++++++++++-----------------------
 1 files changed, 38 insertions(+), 29 deletions(-)
drivers/md/raid5.c +38 -29
index 0c39203..1a8dfd2 100644
@@ -240,11 +240,11 @@ static void init_stripe(struct stripe_he
 	for (i = sh->disks; i--; ) {
 		struct r5dev *dev = &sh->dev[i];
 
-		if (dev->toread || dev->towrite || dev->written ||
+		if (dev->toread || dev->read || dev->towrite || dev->written ||
 		    test_bit(R5_LOCKED, &dev->flags)) {
-			printk("sector=%llx i=%d %p %p %p %d\n",
+			printk("sector=%llx i=%d %p %p %p %p %d\n",
 			       (unsigned long long)sh->sector, i, dev->toread,
-			       dev->towrite, dev->written,
+			       dev->read, dev->towrite, dev->written,
 			       test_bit(R5_LOCKED, &dev->flags));
 			BUG();
 		}
@@ -1749,7 +1749,7 @@ static void handle_stripe5(struct stripe
 	struct bio *bi;
 	int i;
 	int syncing, expanding, expanded;
-	int locked=0, uptodate=0, to_read=0, to_write=0, failed=0, written=0;
+	int locked=0, uptodate=0, to_read=0, to_write=0, failed=0, written=0, to_fill=0;
 	int compute=0, non_overwrite=0, write_complete=0;
 	int failed_num=0;
 	struct r5dev *dev;
@@ -1765,44 +1765,47 @@ static void handle_stripe5(struct stripe
 	syncing = test_bit(STRIPE_SYNCING, &sh->state);
 	expanding = test_bit(STRIPE_EXPAND_SOURCE, &sh->state);
 	expanded = test_bit(STRIPE_EXPAND_READY, &sh->state);
-	/* Now to look around and see what can be done */
 
+	if (test_bit(STRIPE_OP_BIOFILL, &sh->state) &&
+		test_bit(STRIPE_OP_BIOFILL_Done, &sh->ops.state)) {
+		clear_bit(STRIPE_OP_BIOFILL, &sh->state);
+		clear_bit(STRIPE_OP_BIOFILL_Done, &sh->ops.state);
+	}
+
+	/* Now to look around and see what can be done */
 	rcu_read_lock();
 	for (i=disks; i--; ) {
 		mdk_rdev_t *rdev;
 		dev = &sh->dev[i];
 		clear_bit(R5_Insync, &dev->flags);
 
-		PRINTK("check %d: state 0x%lx read %p write %p written %p\n",
-			i, dev->flags, dev->toread, dev->towrite, dev->written);
+		PRINTK("check %d: state 0x%lx toread %p read %p write %p written %p\n",
+		i, dev->flags, dev->toread, dev->read, dev->towrite, dev->written);
+
+		/* maybe we can acknowledge completion of a biofill operation */
+		if (test_bit(R5_ReadReq, &dev->flags) && !dev->toread)
+			clear_bit(R5_ReadReq, &dev->flags);
+
 		/* maybe we can reply to a read */
+		if (dev->read && !test_bit(R5_ReadReq, &dev->flags) &&
+			!test_bit(STRIPE_OP_BIOFILL, &sh->state)) {
+			return_bi = dev->read;
+			dev->read = NULL;
+		}
+
+		/* maybe we can start a biofill operation */
 		if (test_bit(R5_UPTODATE, &dev->flags) && dev->toread) {
-			struct bio *rbi, *rbi2;
-			PRINTK("Return read for disc %d\n", i);
-			spin_lock_irq(&conf->device_lock);
-			rbi = dev->toread;
-			dev->toread = NULL;
-			if (test_and_clear_bit(R5_Overlap, &dev->flags))
-				wake_up(&conf->wait_for_overlap);
-			spin_unlock_irq(&conf->device_lock);
-			while (rbi && rbi->bi_sector < dev->sector + STRIPE_SECTORS) {
-				copy_data(0, rbi, dev->page, dev->sector);
-				rbi2 = r5_next_bio(rbi, dev->sector);
-				spin_lock_irq(&conf->device_lock);
-				if (--rbi->bi_phys_segments == 0) {
-					rbi->bi_next = return_bi;
-					return_bi = rbi;
-				}
-				spin_unlock_irq(&conf->device_lock);
-				rbi = rbi2;
-			}
+			to_read--;
+			if (!test_bit(STRIPE_OP_BIOFILL, &sh->state))
+				set_bit(R5_ReadReq, &dev->flags);
 		}
 
 		/* now count some things */
 		if (test_bit(R5_LOCKED, &dev->flags)) locked++;
 		if (test_bit(R5_UPTODATE, &dev->flags)) uptodate++;
+		if (test_bit(R5_ReadReq, &dev->flags)) to_fill++;
 		if (test_bit(R5_ComputeReq, &dev->flags)) BUG_ON(++compute > 1);
-		
+
 		if (dev->toread) to_read++;
 		if (dev->towrite) {
 			to_write++;
@@ -1824,9 +1827,15 @@ static void handle_stripe5(struct stripe
 			set_bit(R5_Insync, &dev->flags);
 	}
 	rcu_read_unlock();
+
+	if (to_fill && !test_bit(STRIPE_OP_BIOFILL, &sh->state)) {
+		set_bit(STRIPE_OP_BIOFILL, &sh->state);
+		sh->ops.pending++;
+	}
+
 	PRINTK("locked=%d uptodate=%d to_read=%d"
-		" to_write=%d failed=%d failed_num=%d\n",
-		locked, uptodate, to_read, to_write, failed, failed_num);
+		" to_write=%d to_fill=%d failed=%d failed_num=%d\n",
+		locked, uptodate, to_read, to_write, to_fill, failed, failed_num);
 	/* check if the array has lost two devices and, if so, some requests might
 	 * need to be failed
 	 */
#8 lore raw

PATCH 6/19 raid5: move the reconstruct write expansion operation to a workqueue

In reply to Dan Williams

From: Dan Williams <[email protected]>

Enable handle_stripe5 to use the reconstruct write operations capability for expansion operations.

However this does not move the copy operation associated with an expand to the workqueue. First, it was difficult to find a clean way to pass the parameters of this operation to the queue. Second, this section of code is a good candidate for performing the copies with inline calls to the dma routines.

---

 drivers/md/raid5.c |   36 +++++++++++++++++++++++++++---------
 1 files changed, 27 insertions(+), 9 deletions(-)
drivers/md/raid5.c +27 -9
index 1a8dfd2..a07b52b 100644
@@ -2053,6 +2053,7 @@ #endif
 	 * completed
 	 */
 	if (test_bit(STRIPE_OP_RCW, &sh->state) &&
+		!test_bit(STRIPE_OP_RCW_Expand, &sh->ops.state) &&
 		test_bit(STRIPE_OP_RCW_Done, &sh->ops.state)) {
 		clear_bit(STRIPE_OP_RCW, &sh->state);
 		clear_bit(STRIPE_OP_RCW_Done, &sh->ops.state);
@@ -2226,6 +2227,7 @@ #endif
 				}
 			}
 		}
+
 		if (test_bit(STRIPE_OP_COMPUTE_Done, &sh->ops.state) &&
 			test_bit(STRIPE_OP_COMPUTE_Recover_pd, &sh->ops.state)) {
 			clear_bit(STRIPE_OP_COMPUTE, &sh->state);
@@ -2282,18 +2284,28 @@ #endif
 		}
 	}
 
-	if (expanded && test_bit(STRIPE_EXPANDING, &sh->state)) {
+	/* Finish 'rcw' operations initiated by the expansion
+	 * process
+	 */
+	if (test_bit(STRIPE_OP_RCW, &sh->state) &&
+		test_bit(STRIPE_OP_RCW_Expand, &sh->ops.state) &&
+		test_bit(STRIPE_OP_RCW_Done, &sh->ops.state)) {
+		clear_bit(STRIPE_OP_RCW, &sh->state);
+		clear_bit(STRIPE_OP_RCW_Done, &sh->ops.state);
+		clear_bit(STRIPE_OP_RCW_Expand, &sh->ops.state);
+		clear_bit(STRIPE_EXPANDING, &sh->state);
+		for (i= conf->raid_disks; i--;)
+			set_bit(R5_Wantwrite, &sh->dev[i].flags);
+	}
+
+	if (expanded && test_bit(STRIPE_EXPANDING, &sh->state) &&
+		!test_bit(STRIPE_OP_RCW, &sh->state)) {
 		/* Need to write out all blocks after computing parity */
 		sh->disks = conf->raid_disks;
 		sh->pd_idx = stripe_to_pdidx(sh->sector, conf, conf->raid_disks);
-		compute_parity5(sh, RECONSTRUCT_WRITE);
-		for (i= conf->raid_disks; i--;) {
-			set_bit(R5_LOCKED, &sh->dev[i].flags);
-			locked++;
-			set_bit(R5_Wantwrite, &sh->dev[i].flags);
-		}
-		clear_bit(STRIPE_EXPANDING, &sh->state);
-	} else if (expanded) {
+		set_bit(STRIPE_OP_RCW_Expand, &sh->ops.state);
+		locked += handle_write_operations5(sh, 0);
+	} else if (expanded && !test_bit(STRIPE_OP_RCW, &sh->state)) {
 		clear_bit(STRIPE_EXPAND_READY, &sh->state);
 		atomic_dec(&conf->reshape_stripes);
 		wake_up(&conf->wait_for_overlap);
@@ -2327,9 +2339,15 @@ #endif
 					release_stripe(sh2);
 					continue;
 				}
+				/* to do: perform these operations with a dma engine
+				 * inline (rather than pushing to the workqueue)
+				 */
+				/*#ifdef CONFIG_RAID5_DMA*/
+				/*#else*/
 				memcpy(page_address(sh2->dev[dd_idx].page),
 				       page_address(sh->dev[i].page),
 				       STRIPE_SIZE);
+				/*#endif*/
 				set_bit(R5_Expanded, &sh2->dev[dd_idx].flags);
 				set_bit(R5_UPTODATE, &sh2->dev[dd_idx].flags);
 				for (j=0; j<conf->raid_disks; j++)
#9 lore raw

PATCH 7/19 raid5: remove compute_block and compute_parity5

In reply to Dan Williams

From: Dan Williams <[email protected]>

replaced by the workqueue implementation

---

 drivers/md/raid5.c |  123 ----------------------------------------------------
 1 files changed, 0 insertions(+), 123 deletions(-)
drivers/md/raid5.c +0 -123
index a07b52b..ad6883b 100644
@@ -964,129 +964,6 @@ #define check_xor() 	do { 						\
 			} while(0)
 
 
-static void compute_block(struct stripe_head *sh, int dd_idx)
-{
-	int i, count, disks = sh->disks;
-	void *ptr[MAX_XOR_BLOCKS], *p;
-
-	PRINTK("compute_block, stripe %llu, idx %d\n", 
-		(unsigned long long)sh->sector, dd_idx);
-
-	ptr[0] = page_address(sh->dev[dd_idx].page);
-	memset(ptr[0], 0, STRIPE_SIZE);
-	count = 1;
-	for (i = disks ; i--; ) {
-		if (i == dd_idx)
-			continue;
-		p = page_address(sh->dev[i].page);
-		if (test_bit(R5_UPTODATE, &sh->dev[i].flags))
-			ptr[count++] = p;
-		else
-			printk(KERN_ERR "compute_block() %d, stripe %llu, %d"
-				" not present\n", dd_idx,
-				(unsigned long long)sh->sector, i);
-
-		check_xor();
-	}
-	if (count != 1)
-		xor_block(count, STRIPE_SIZE, ptr);
-	set_bit(R5_UPTODATE, &sh->dev[dd_idx].flags);
-}
-
-static void compute_parity5(struct stripe_head *sh, int method)
-{
-	raid5_conf_t *conf = sh->raid_conf;
-	int i, pd_idx = sh->pd_idx, disks = sh->disks, count;
-	void *ptr[MAX_XOR_BLOCKS];
-	struct bio *chosen;
-
-	PRINTK("compute_parity5, stripe %llu, method %d\n",
-		(unsigned long long)sh->sector, method);
-
-	count = 1;
-	ptr[0] = page_address(sh->dev[pd_idx].page);
-	switch(method) {
-	case READ_MODIFY_WRITE:
-		BUG_ON(!test_bit(R5_UPTODATE, &sh->dev[pd_idx].flags));
-		for (i=disks ; i-- ;) {
-			if (i==pd_idx)
-				continue;
-			if (sh->dev[i].towrite &&
-			    test_bit(R5_UPTODATE, &sh->dev[i].flags)) {
-				ptr[count++] = page_address(sh->dev[i].page);
-				chosen = sh->dev[i].towrite;
-				sh->dev[i].towrite = NULL;
-
-				if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
-					wake_up(&conf->wait_for_overlap);
-
-				BUG_ON(sh->dev[i].written);
-				sh->dev[i].written = chosen;
-				check_xor();
-			}
-		}
-		break;
-	case RECONSTRUCT_WRITE:
-		memset(ptr[0], 0, STRIPE_SIZE);
-		for (i= disks; i-- ;)
-			if (i!=pd_idx && sh->dev[i].towrite) {
-				chosen = sh->dev[i].towrite;
-				sh->dev[i].towrite = NULL;
-
-				if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
-					wake_up(&conf->wait_for_overlap);
-
-				BUG_ON(sh->dev[i].written);
-				sh->dev[i].written = chosen;
-			}
-		break;
-	case CHECK_PARITY:
-		break;
-	}
-	if (count>1) {
-		xor_block(count, STRIPE_SIZE, ptr);
-		count = 1;
-	}
-	
-	for (i = disks; i--;)
-		if (sh->dev[i].written) {
-			sector_t sector = sh->dev[i].sector;
-			struct bio *wbi = sh->dev[i].written;
-			while (wbi && wbi->bi_sector < sector + STRIPE_SECTORS) {
-				copy_data(1, wbi, sh->dev[i].page, sector);
-				wbi = r5_next_bio(wbi, sector);
-			}
-
-			set_bit(R5_LOCKED, &sh->dev[i].flags);
-			set_bit(R5_UPTODATE, &sh->dev[i].flags);
-		}
-
-	switch(method) {
-	case RECONSTRUCT_WRITE:
-	case CHECK_PARITY:
-		for (i=disks; i--;)
-			if (i != pd_idx) {
-				ptr[count++] = page_address(sh->dev[i].page);
-				check_xor();
-			}
-		break;
-	case READ_MODIFY_WRITE:
-		for (i = disks; i--;)
-			if (sh->dev[i].written) {
-				ptr[count++] = page_address(sh->dev[i].page);
-				check_xor();
-			}
-	}
-	if (count != 1)
-		xor_block(count, STRIPE_SIZE, ptr);
-	
-	if (method != CHECK_PARITY) {
-		set_bit(R5_UPTODATE, &sh->dev[pd_idx].flags);
-		set_bit(R5_LOCKED,   &sh->dev[pd_idx].flags);
-	} else
-		clear_bit(R5_UPTODATE, &sh->dev[pd_idx].flags);
-}
-
 static void compute_parity6(struct stripe_head *sh, int method)
 {
 	raid6_conf_t *conf = sh->raid_conf;
#10 lore raw

PATCH 8/19 dmaengine: enable multiple clients and operations

In reply to Dan Williams

From: Dan Williams <[email protected]>

Enable the dmaengine interface to allow multiple clients to share a channel, and enable clients to request channels based on an operations capability mask. This prepares the interface for use with the RAID5 client and the future RAID6 client.

Multi-client support is achieved by modifying channels to maintain a list of peer clients.

Multi-operation support is achieved by modifying clients to maintain lists of channel references. Channel references in a given request list satisfy a client specified capability mask.

Changelog:
* make the dmaengine api EXPORT_SYMBOL_GPL
* zero sum support should be standalone, not integrated into xor

---

 drivers/dma/dmaengine.c   |  357 ++++++++++++++++++++++++++++++++++++---------
 drivers/dma/ioatdma.c     |   12 +-
 include/linux/dmaengine.h |  164 ++++++++++++++++++---
 net/core/dev.c            |   21 +--
 net/ipv4/tcp.c            |    4 -
 5 files changed, 443 insertions(+), 115 deletions(-)
drivers/dma/dmaengine.c +286 -71
index 1527804..e10f19d 100644
@@ -37,8 +37,13 @@
  * Each device has a channels list, which runs unlocked but is never modified
  * once the device is registered, it's just setup by the driver.
  *
- * Each client has a channels list, it's only modified under the client->lock
- * and in an RCU callback, so it's safe to read under rcu_read_lock().
+ * Each client has 'n' lists of channel references where
+ * n == DMA_MAX_CHAN_TYPE_REQ.  These lists are only modified under the
+ * client->lock and in an RCU callback, so they are safe to read under
+ * rcu_read_lock().
+ *
+ * Each channel has a list of peer clients, it's only modified under the
+ * chan->lock.  This allows a channel to be shared amongst several clients
  *
  * Each device has a kref, which is initialized to 1 when the device is
  * registered. A kref_put is done for each class_device registered.  When the
@@ -85,6 +90,18 @@ static ssize_t show_memcpy_count(struct 
 	return sprintf(buf, "%lu\n", count);
 }
 
+static ssize_t show_xor_count(struct class_device *cd, char *buf)
+{
+	struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
+	unsigned long count = 0;
+	int i;
+
+	for_each_possible_cpu(i)
+		count += per_cpu_ptr(chan->local, i)->xor_count;
+
+	return sprintf(buf, "%lu\n", count);
+}
+
 static ssize_t show_bytes_transferred(struct class_device *cd, char *buf)
 {
 	struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
@@ -97,16 +114,37 @@ static ssize_t show_bytes_transferred(st
 	return sprintf(buf, "%lu\n", count);
 }
 
+static ssize_t show_bytes_xor(struct class_device *cd, char *buf)
+{
+	struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
+	unsigned long count = 0;
+	int i;
+
+	for_each_possible_cpu(i)
+		count += per_cpu_ptr(chan->local, i)->bytes_xor;
+
+	return sprintf(buf, "%lu\n", count);
+}
+
 static ssize_t show_in_use(struct class_device *cd, char *buf)
 {
+	unsigned int clients = 0;
+	struct list_head *peer;
 	struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
 
-	return sprintf(buf, "%d\n", (chan->client ? 1 : 0));
+	rcu_read_lock();
+	list_for_each_rcu(peer, &chan->peers)
+		clients++;
+	rcu_read_unlock();
+
+	return sprintf(buf, "%d\n", clients);
 }
 
 static struct class_device_attribute dma_class_attrs[] = {
 	__ATTR(memcpy_count, S_IRUGO, show_memcpy_count, NULL),
+	__ATTR(xor_count, S_IRUGO, show_xor_count, NULL),
 	__ATTR(bytes_transferred, S_IRUGO, show_bytes_transferred, NULL),
+	__ATTR(bytes_xor, S_IRUGO, show_bytes_xor, NULL),
 	__ATTR(in_use, S_IRUGO, show_in_use, NULL),
 	__ATTR_NULL
 };
@@ -130,34 +168,79 @@ static struct class dma_devclass = {
 /**
  * dma_client_chan_alloc - try to allocate a channel to a client
  * @client: &dma_client
+ * @req: request descriptor
  *
  * Called with dma_list_mutex held.
  */
-static struct dma_chan *dma_client_chan_alloc(struct dma_client *client)
+static struct dma_chan *dma_client_chan_alloc(struct dma_client *client,
+	struct dma_req *req)
 {
 	struct dma_device *device;
 	struct dma_chan *chan;
+	struct dma_client_chan_peer *peer;
+	struct dma_chan_client_ref *chan_ref;
 	unsigned long flags;
 	int desc;	/* allocated descriptor count */
+	int allocated;	/* flag re-allocations */
 
-	/* Find a channel, any DMA engine will do */
+	/* Find a channel */
 	list_for_each_entry(device, &dma_device_list, global_node) {
+		if ((req->cap_mask & device->capabilities)
+			!= req->cap_mask)
+			continue;
 		list_for_each_entry(chan, &device->channels, device_node) {
-			if (chan->client)
+			allocated = 0;
+			rcu_read_lock();
+			list_for_each_entry_rcu(chan_ref, &req->channels, req_node) {
+				if (chan_ref->chan == chan) {
+					allocated = 1;
+					break;
+				}
+			}
+			rcu_read_unlock();
+
+			if (allocated)
 				continue;
 
+			/* can the channel be shared between multiple clients */
+			if ((req->exclusive && !list_empty(&chan->peers)) ||
+				chan->exclusive)
+				continue;
+
+			chan_ref = kmalloc(sizeof(*chan_ref), GFP_KERNEL);
+			if (!chan_ref)
+				continue;
+
+			peer = kmalloc(sizeof(*peer), GFP_KERNEL);
+			if (!peer) {
+				kfree(chan_ref);
+				continue;
+			}
+
 			desc = chan->device->device_alloc_chan_resources(chan);
-			if (desc >= 0) {
+			if (desc) {
 				kref_get(&device->refcount);
-				kref_init(&chan->refcount);
-				chan->slow_ref = 0;
-				INIT_RCU_HEAD(&chan->rcu);
-				chan->client = client;
+				kref_get(&chan->refcount);
+				INIT_RCU_HEAD(&peer->rcu);
+				INIT_RCU_HEAD(&chan_ref->rcu);
+				INIT_LIST_HEAD(&peer->peer_node);
+				INIT_LIST_HEAD(&chan_ref->req_node);
+				peer->client = client;
+				chan_ref->chan = chan;
+
+				spin_lock_irqsave(&chan->lock, flags);
+				list_add_tail_rcu(&peer->peer_node, &chan->peers);
+				spin_unlock_irqrestore(&chan->lock, flags);
+
 				spin_lock_irqsave(&client->lock, flags);
-				list_add_tail_rcu(&chan->client_node,
-				                  &client->channels);
+				chan->exclusive = req->exclusive ? client : NULL;
+				list_add_tail_rcu(&chan_ref->req_node,
+						&req->channels);
 				spin_unlock_irqrestore(&client->lock, flags);
 				return chan;
+			} else {
+				kfree(peer);
+				kfree(chan_ref);
 			}
 		}
 	}
@@ -173,7 +256,6 @@ void dma_chan_cleanup(struct kref *kref)
 {
 	struct dma_chan *chan = container_of(kref, struct dma_chan, refcount);
 	chan->device->device_free_chan_resources(chan);
-	chan->client = NULL;
 	kref_put(&chan->device->refcount, dma_async_device_cleanup);
 }
 
@@ -186,51 +268,93 @@ static void dma_chan_free_rcu(struct rcu
 		bias -= local_read(&per_cpu_ptr(chan->local, i)->refcount);
 	atomic_sub(bias, &chan->refcount.refcount);
 	kref_put(&chan->refcount, dma_chan_cleanup);
+	kref_put(&chan->device->refcount, dma_async_device_cleanup);
+}
+
+static void dma_peer_free_rcu(struct rcu_head *rcu)
+{
+	struct dma_client_chan_peer *peer =
+		container_of(rcu, struct dma_client_chan_peer, rcu);
+
+	kfree(peer);
+}
+
+static void dma_chan_ref_free_rcu(struct rcu_head *rcu)
+{
+	struct dma_chan_client_ref *chan_ref =
+		container_of(rcu, struct dma_chan_client_ref, rcu);
+
+	kfree(chan_ref);
 }
 
-static void dma_client_chan_free(struct dma_chan *chan)
+static void dma_client_chan_free(struct dma_client *client,
+				struct dma_chan_client_ref *chan_ref)
 {
+	struct dma_client_chan_peer *peer;
+	struct dma_chan *chan = chan_ref->chan;
 	atomic_add(0x7FFFFFFF, &chan->refcount.refcount);
 	chan->slow_ref = 1;
-	call_rcu(&chan->rcu, dma_chan_free_rcu);
+	rcu_read_lock();
+	list_for_each_entry_rcu(peer, &chan->peers, peer_node)
+		if (peer->client == client) {
+			unsigned long flags;
+
+			spin_lock_irqsave(&chan->lock, flags);
+			list_del_rcu(&peer->peer_node);
+			if (list_empty(&chan->peers))
+				chan->exclusive = NULL;
+			spin_unlock_irqrestore(&chan->lock, flags);
+			call_rcu(&peer->rcu, dma_peer_free_rcu);
+			call_rcu(&chan_ref->rcu, dma_chan_ref_free_rcu);
+			call_rcu(&chan->rcu, dma_chan_free_rcu);
+			break;
+		}
+	rcu_read_unlock();
 }
 
 /**
  * dma_chans_rebalance - reallocate channels to clients
  *
- * When the number of DMA channel in the system changes,
- * channels need to be rebalanced among clients.
+ * When the number of DMA channels in the system changes,
+ * channels need to be rebalanced among clients
  */
 static void dma_chans_rebalance(void)
 {
 	struct dma_client *client;
 	struct dma_chan *chan;
+	struct dma_chan_client_ref *chan_ref;
+
 	unsigned long flags;
+	int i;
 
 	mutex_lock(&dma_list_mutex);
 
 	list_for_each_entry(client, &dma_client_list, global_node) {
-		while (client->chans_desired > client->chan_count) {
-			chan = dma_client_chan_alloc(client);
-			if (!chan)
-				break;
-			client->chan_count++;
-			client->event_callback(client,
-	                                       chan,
-	                                       DMA_RESOURCE_ADDED);
-		}
-		while (client->chans_desired < client->chan_count) {
-			spin_lock_irqsave(&client->lock, flags);
-			chan = list_entry(client->channels.next,
-			                  struct dma_chan,
-			                  client_node);
-			list_del_rcu(&chan->client_node);
-			spin_unlock_irqrestore(&client->lock, flags);
-			client->chan_count--;
-			client->event_callback(client,
-			                       chan,
-			                       DMA_RESOURCE_REMOVED);
-			dma_client_chan_free(chan);
+		for (i = 0; i < DMA_MAX_CHAN_TYPE_REQ; i++) {
+			struct dma_req *req = &client->req[i];
+			while (req->chans_desired > atomic_read(&req->chan_count)) {
+				chan = dma_client_chan_alloc(client, req);
+				if (!chan)
+					break;
+				atomic_inc(&req->chan_count);
+				client->event_callback(client,
+		                                       chan,
+		                                       DMA_RESOURCE_ADDED);
+			}
+			while (req->chans_desired < atomic_read(&req->chan_count)) {
+				spin_lock_irqsave(&client->lock, flags);
+				chan_ref = list_entry(req->channels.next,
+				                  struct dma_chan_client_ref,
+				                  req_node);
+				list_del_rcu(&chan_ref->req_node);
+				spin_unlock_irqrestore(&client->lock, flags);
+				atomic_dec(&req->chan_count);
+
+				client->event_callback(client,
+				                       chan_ref->chan,
+				                       DMA_RESOURCE_REMOVED);
+				dma_client_chan_free(client, chan_ref);
+			}
 		}
 	}
 
@@ -244,15 +368,18 @@ static void dma_chans_rebalance(void)
 struct dma_client *dma_async_client_register(dma_event_callback event_callback)
 {
 	struct dma_client *client;
+	int i;
 
 	client = kzalloc(sizeof(*client), GFP_KERNEL);
 	if (!client)
 		return NULL;
 
-	INIT_LIST_HEAD(&client->channels);
+	for (i = 0; i < DMA_MAX_CHAN_TYPE_REQ; i++) {
+		INIT_LIST_HEAD(&client->req[i].channels);
+		atomic_set(&client->req[i].chan_count, 0);
+	}
+
 	spin_lock_init(&client->lock);
-	client->chans_desired = 0;
-	client->chan_count = 0;
 	client->event_callback = event_callback;
 
 	mutex_lock(&dma_list_mutex);
@@ -270,14 +397,16 @@ struct dma_client *dma_async_client_regi
  */
 void dma_async_client_unregister(struct dma_client *client)
 {
-	struct dma_chan *chan;
+	struct dma_chan_client_ref *chan_ref;
+	int i;
 
 	if (!client)
 		return;
 
 	rcu_read_lock();
-	list_for_each_entry_rcu(chan, &client->channels, client_node)
-		dma_client_chan_free(chan);
+	for (i = 0; i < DMA_MAX_CHAN_TYPE_REQ; i++)
+		list_for_each_entry_rcu(chan_ref, &client->req[i].channels, req_node)
+			dma_client_chan_free(client, chan_ref);
 	rcu_read_unlock();
 
 	mutex_lock(&dma_list_mutex);
@@ -292,17 +421,46 @@ void dma_async_client_unregister(struct 
  * dma_async_client_chan_request - request DMA channels
  * @client: &dma_client
  * @number: count of DMA channels requested
+ * @mask: limits the DMA channels returned to those that
+ *	have the requisite capabilities
  *
  * Clients call dma_async_client_chan_request() to specify how many
  * DMA channels they need, 0 to free all currently allocated.
  * The resulting allocations/frees are indicated to the client via the
- * event callback.
+ * event callback.  If the client has exhausted the number of distinct
+ * requests allowed (DMA_MAX_CHAN_TYPE_REQ) this function will return 0.
  */
-void dma_async_client_chan_request(struct dma_client *client,
-			unsigned int number)
+int dma_async_client_chan_request(struct dma_client *client,
+			unsigned int number, unsigned int mask)
 {
-	client->chans_desired = number;
-	dma_chans_rebalance();
+	int request_slot_found = 0, i;
+
+	/* adjust an outstanding request */
+	for (i = 0; i < DMA_MAX_CHAN_TYPE_REQ; i++) {
+		struct dma_req *req = &client->req[i];
+		if (req->cap_mask == mask) {
+			req->chans_desired = number;
+			request_slot_found = 1;
+			break;
+		}
+	}
+
+	/* start a new request */
+	if (!request_slot_found)
+		for (i = 0; i < DMA_MAX_CHAN_TYPE_REQ; i++) {
+			struct dma_req *req = &client->req[i];
+			if (!req->chans_desired) {
+				req->chans_desired = number;
+				req->cap_mask = mask;
+				request_slot_found = 1;
+				break;
+			}
+		}
+
+	if (request_slot_found)
+		dma_chans_rebalance();
+
+	return request_slot_found;
 }
 
 /**
@@ -335,6 +493,7 @@ int dma_async_device_register(struct dma
 		         device->dev_id, chan->chan_id);
 
 		kref_get(&device->refcount);
+		kref_init(&chan->refcount);
 		class_device_register(&chan->class_dev);
 	}
 
@@ -348,6 +507,20 @@ int dma_async_device_register(struct dma
 }
 
 /**
+ * dma_async_chan_init - common channel initialization
+ * @chan: &dma_chan
+ * @device: &dma_device
+ */
+void dma_async_chan_init(struct dma_chan *chan, struct dma_device *device)
+{
+	INIT_LIST_HEAD(&chan->peers);
+	INIT_RCU_HEAD(&chan->rcu);
+	spin_lock_init(&chan->lock);
+	chan->device = device;
+	list_add_tail(&chan->device_node, &device->channels);
+}
+
+/**
  * dma_async_device_cleanup - function called when all references are released
  * @kref: kernel reference object
  */
@@ -366,31 +539,70 @@ static void dma_async_device_cleanup(str
 void dma_async_device_unregister(struct dma_device *device)
 {
 	struct dma_chan *chan;
+	struct dma_client_chan_peer *peer;
+	struct dma_req *req;
+	struct dma_chan_client_ref *chan_ref;
+	struct dma_client *client;
+	int i;
 	unsigned long flags;
 
 	mutex_lock(&dma_list_mutex);
 	list_del(&device->global_node);
 	mutex_unlock(&dma_list_mutex);
 
+	/* look up and free each reference to a channel
+	 * note: a channel can be allocated to a client once per
+	 * request type (DMA_MAX_CHAN_TYPE_REQ)
+	 */
 	list_for_each_entry(chan, &device->channels, device_node) {
-		if (chan->client) {
-			spin_lock_irqsave(&chan->client->lock, flags);
-			list_del(&chan->client_node);
-			chan->client->chan_count--;
-			spin_unlock_irqrestore(&chan->client->lock, flags);
-			chan->client->event_callback(chan->client,
-			                             chan,
-			                             DMA_RESOURCE_REMOVED);
-			dma_client_chan_free(chan);
+		rcu_read_lock();
+		list_for_each_entry_rcu(peer, &chan->peers, peer_node) {
+			client = peer->client;
+			for (i = 0; i < DMA_MAX_CHAN_TYPE_REQ; i++) {
+				req = &client->req[i];
+				list_for_each_entry_rcu(chan_ref,
+							&req->channels,
+							req_node) {
+					if (chan_ref->chan == chan) {
+						spin_lock_irqsave(&client->lock, flags);
+						list_del_rcu(&chan_ref->req_node);
+						spin_unlock_irqrestore(&client->lock, flags);
+						atomic_dec(&req->chan_count);
+						client->event_callback(
+						client,
+						chan,
+						DMA_RESOURCE_REMOVED);
+						dma_client_chan_free(client,
+								     chan_ref);
+						break;
+					}
+				}
+			}
 		}
-		class_device_unregister(&chan->class_dev);
+		rcu_read_unlock();
+		kref_put(&chan->refcount, dma_chan_cleanup);
+		kref_put(&device->refcount, dma_async_device_cleanup);
 	}
+
+	class_device_unregister(&chan->class_dev);
+
 	dma_chans_rebalance();
 
 	kref_put(&device->refcount, dma_async_device_cleanup);
 	wait_for_completion(&device->done);
 }
 
+/**
+ * dma_async_xor_pgs_to_pg_err - default function for dma devices that
+ *	do not support xor
+ */
+dma_cookie_t dma_async_xor_pgs_to_pg_err(struct dma_chan *chan,
+	struct page *dest_pg, unsigned int dest_off, struct page *src_pgs,
+	unsigned int src_cnt, unsigned int src_off, size_t len)
+{
+	return -ENXIO;
+}
+
 static int __init dma_bus_init(void)
 {
 	mutex_init(&dma_list_mutex);
@@ -399,14 +611,17 @@ static int __init dma_bus_init(void)
 
 subsys_initcall(dma_bus_init);
 
-EXPORT_SYMBOL(dma_async_client_register);
-EXPORT_SYMBOL(dma_async_client_unregister);
-EXPORT_SYMBOL(dma_async_client_chan_request);
-EXPORT_SYMBOL(dma_async_memcpy_buf_to_buf);
-EXPORT_SYMBOL(dma_async_memcpy_buf_to_pg);
-EXPORT_SYMBOL(dma_async_memcpy_pg_to_pg);
-EXPORT_SYMBOL(dma_async_memcpy_complete);
-EXPORT_SYMBOL(dma_async_memcpy_issue_pending);
-EXPORT_SYMBOL(dma_async_device_register);
-EXPORT_SYMBOL(dma_async_device_unregister);
-EXPORT_SYMBOL(dma_chan_cleanup);
+EXPORT_SYMBOL_GPL(dma_async_client_register);
+EXPORT_SYMBOL_GPL(dma_async_client_unregister);
+EXPORT_SYMBOL_GPL(dma_async_client_chan_request);
+EXPORT_SYMBOL_GPL(dma_async_memcpy_buf_to_buf);
+EXPORT_SYMBOL_GPL(dma_async_memcpy_buf_to_pg);
+EXPORT_SYMBOL_GPL(dma_async_memcpy_pg_to_pg);
+EXPORT_SYMBOL_GPL(dma_async_xor_pgs_to_pg);
+EXPORT_SYMBOL_GPL(dma_async_operation_complete);
+EXPORT_SYMBOL_GPL(dma_async_issue_pending);
+EXPORT_SYMBOL_GPL(dma_async_device_register);
+EXPORT_SYMBOL_GPL(dma_async_device_unregister);
+EXPORT_SYMBOL_GPL(dma_chan_cleanup);
+EXPORT_SYMBOL_GPL(dma_async_xor_pgs_to_pg_err);
+EXPORT_SYMBOL_GPL(dma_async_chan_init);
drivers/dma/ioatdma.c +5 -7
index dbd4d6c..415de03 100644
@@ -69,11 +69,7 @@ static int enumerate_dma_channels(struct
 		spin_lock_init(&ioat_chan->desc_lock);
 		INIT_LIST_HEAD(&ioat_chan->free_desc);
 		INIT_LIST_HEAD(&ioat_chan->used_desc);
-		/* This should be made common somewhere in dmaengine.c */
-		ioat_chan->common.device = &device->common;
-		ioat_chan->common.client = NULL;
-		list_add_tail(&ioat_chan->common.device_node,
-		              &device->common.channels);
+		dma_async_chan_init(&ioat_chan->common, &device->common);
 	}
 	return device->common.chancnt;
 }
@@ -759,8 +755,10 @@ #endif
 	device->common.device_memcpy_buf_to_buf = ioat_dma_memcpy_buf_to_buf;
 	device->common.device_memcpy_buf_to_pg = ioat_dma_memcpy_buf_to_pg;
 	device->common.device_memcpy_pg_to_pg = ioat_dma_memcpy_pg_to_pg;
-	device->common.device_memcpy_complete = ioat_dma_is_complete;
-	device->common.device_memcpy_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.device_operation_complete = ioat_dma_is_complete;
+	device->common.device_xor_pgs_to_pg = dma_async_xor_pgs_to_pg_err;
+	device->common.device_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.capabilities = DMA_MEMCPY;
 	printk(KERN_INFO "Intel(R) I/OAT DMA Engine found, %d channels\n",
 		device->common.chancnt);
 
include/linux/dmaengine.h +140 -24

283 lines not on this page · raw message

net/core/dev.c +10 -11
index d4a1ec3..9447f94 100644
@@ -1941,13 +1941,8 @@ #ifdef CONFIG_NET_DMA
 	 * There may not be any more sk_buffs coming right now, so push
 	 * any pending DMA copies to hardware
 	 */
-	if (net_dma_client) {
-		struct dma_chan *chan;
-		rcu_read_lock();
-		list_for_each_entry_rcu(chan, &net_dma_client->channels, client_node)
-			dma_async_memcpy_issue_pending(chan);
-		rcu_read_unlock();
-	}
+	if (net_dma_client)
+		dma_async_issue_all(net_dma_client);
 #endif
 	local_irq_enable();
 	return;
@@ -3410,7 +3405,8 @@ #ifdef CONFIG_NET_DMA
 static void net_dma_rebalance(void)
 {
 	unsigned int cpu, i, n;
-	struct dma_chan *chan;
+	struct dma_chan_client_ref *chan_ref;
+	struct dma_req *req;
 
 	if (net_dma_count == 0) {
 		for_each_online_cpu(cpu)
@@ -3421,13 +3417,16 @@ static void net_dma_rebalance(void)
 	i = 0;
 	cpu = first_cpu(cpu_online_map);
 
+	/* NET_DMA only requests one type of dma channel (memcpy) */
+	req = &net_dma_client->req[0];
+
 	rcu_read_lock();
-	list_for_each_entry(chan, &net_dma_client->channels, client_node) {
+	list_for_each_entry(chan_ref, &req->channels, req_node) {
 		n = ((num_online_cpus() / net_dma_count)
 		   + (i < (num_online_cpus() % net_dma_count) ? 1 : 0));
 
 		while(n) {
-			per_cpu(softnet_data, cpu).net_dma = chan;
+			per_cpu(softnet_data, cpu).net_dma = chan_ref->chan;
 			cpu = next_cpu(cpu, cpu_online_map);
 			n--;
 		}
@@ -3471,7 +3470,7 @@ static int __init netdev_dma_register(vo
 	if (net_dma_client == NULL)
 		return -ENOMEM;
 
-	dma_async_client_chan_request(net_dma_client, num_online_cpus());
+	dma_async_client_chan_request(net_dma_client, num_online_cpus(), DMA_MEMCPY);
 	return 0;
 }
 
net/ipv4/tcp.c +2 -2
index 934396b..cd8ad41 100644
@@ -1431,9 +1431,9 @@ #ifdef CONFIG_NET_DMA
 		struct sk_buff *skb;
 		dma_cookie_t done, used;
 
-		dma_async_memcpy_issue_pending(tp->ucopy.dma_chan);
+		dma_async_issue_pending(tp->ucopy.dma_chan);
 
-		while (dma_async_memcpy_complete(tp->ucopy.dma_chan,
+		while (dma_async_operation_complete(tp->ucopy.dma_chan,
 		                                 tp->ucopy.dma_cookie, &done,
 		                                 &used) == DMA_IN_PROGRESS) {
 			/* do partial cleanup of sk_async_wait_queue */
#11 lore raw

Re: [PATCH 08/19] dmaengine: enable multiple clients and operations

drivers/dma/ioatdma.c #endif · line 761

Quoted 10 lines · Dan Williams
drivers/dma/ioatdma.c +4 -2
@@ -759,8 +755,10 @@ #endif
 	device->common.device_memcpy_buf_to_buf = ioat_dma_memcpy_buf_to_buf;
 	device->common.device_memcpy_buf_to_pg = ioat_dma_memcpy_buf_to_pg;
 	device->common.device_memcpy_pg_to_pg = ioat_dma_memcpy_pg_to_pg;
-	device->common.device_memcpy_complete = ioat_dma_is_complete;
-	device->common.device_memcpy_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.device_operation_complete = ioat_dma_is_complete;
+	device->common.device_xor_pgs_to_pg = dma_async_xor_pgs_to_pg_err;
+	device->common.device_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.capabilities = DMA_MEMCPY;

Are we really going to add a set of hooks for each DMA engine whizbang feature?

That will get ugly when DMA engines support memcpy, xor, crc32, sha1, aes, and a dozen other transforms.

Quoted 25 lines
include/linux/dmaengine.h +9 -1
index c94d8f1..3599472 100644
@@ -20,7 +20,7 @@
  */
 #ifndef DMAENGINE_H
 #define DMAENGINE_H
-
+#include <linux/config.h>
 #ifdef CONFIG_DMA_ENGINE
 
 #include <linux/device.h>
@@ -65,6 +65,27 @@ enum dma_status {
 };
 
 /**
+ * enum dma_capabilities - DMA operational capabilities
+ * @DMA_MEMCPY: src to dest copy
+ * @DMA_XOR: src*n to dest xor
+ * @DMA_DUAL_XOR: src*n to dest_diag and dest_horiz xor
+ * @DMA_PQ_XOR: src*n to dest_q and dest_p gf/xor
+ * @DMA_MEMCPY_CRC32C: src to dest copy and crc-32c sum
+ * @DMA_SHARE: multiple clients can use this channel
+ */

include/linux/dmaengine.h enum dma_status { · line 85

Quoted 10 lines
include/linux/dmaengine.h +10 -0
+enum dma_capabilities {
+	DMA_MEMCPY		= 0x1,
+	DMA_XOR			= 0x2,
+	DMA_PQ_XOR		= 0x4,
+	DMA_DUAL_XOR		= 0x8,
+	DMA_PQ_UPDATE		= 0x10,
+	DMA_ZERO_SUM		= 0x20,
+	DMA_PQ_ZERO_SUM		= 0x40,
+	DMA_MEMSET		= 0x80,
+	DMA_MEMCPY_CRC32C	= 0x100,

Please use the more readable style that explicitly lists bits:

	DMA_MEMCPY		= (1 << 0),
	DMA_XOR			= (1 << 1),
	...

include/linux/dmaengine.h struct dma_chan_percpu { · line 101

Quoted 11 lines
include/linux/dmaengine.h +3 -0
+/**
  * struct dma_chan_percpu - the per-CPU part of struct dma_chan
  * @refcount: local_t used for open-coded "bigref" counting
  * @memcpy_count: transaction counter
@@ -75,27 +96,32 @@ struct dma_chan_percpu {
 	local_t refcount;
 	/* stats */
 	unsigned long memcpy_count;
+	unsigned long xor_count;
 	unsigned long bytes_transferred;
+	unsigned long bytes_xor;

Clearly, each operation needs to be more compartmentalized.

This just isn't scalable, when you consider all the possible transforms.

Jeff

#12 lore raw

Re: [PATCH 08/19] dmaengine: enable multiple clients and operations

Quoted 15 lines · Jeff Garzik

drivers/dma/ioatdma.c #endif · line 761

Quoted 10 lines · Dan Williams
drivers/dma/ioatdma.c +4 -2
@@ -759,8 +755,10 @@ #endif
      device->common.device_memcpy_buf_to_buf = ioat_dma_memcpy_buf_to_buf;
      device->common.device_memcpy_buf_to_pg = ioat_dma_memcpy_buf_to_pg;
      device->common.device_memcpy_pg_to_pg = ioat_dma_memcpy_pg_to_pg;
-     device->common.device_memcpy_complete = ioat_dma_is_complete;
-     device->common.device_memcpy_issue_pending = ioat_dma_memcpy_issue_pending;
+     device->common.device_operation_complete = ioat_dma_is_complete;
+     device->common.device_xor_pgs_to_pg = dma_async_xor_pgs_to_pg_err;
+     device->common.device_issue_pending = ioat_dma_memcpy_issue_pending;
+     device->common.capabilities = DMA_MEMCPY;

Are we really going to add a set of hooks for each DMA engine whizbang feature?

What's the alternative? But, also see patch 9 "dmaengine: reduce backend address permutations" it relieves some of this pain.

Quoted 43 lines

That will get ugly when DMA engines support memcpy, xor, crc32, sha1, aes, and a dozen other transforms.

Quoted 25 lines
include/linux/dmaengine.h +9 -1
index c94d8f1..3599472 100644
@@ -20,7 +20,7 @@
  */
 #ifndef DMAENGINE_H
 #define DMAENGINE_H
-
+#include <linux/config.h>
 #ifdef CONFIG_DMA_ENGINE
 
 #include <linux/device.h>
@@ -65,6 +65,27 @@ enum dma_status {
 };
 
 /**
+ * enum dma_capabilities - DMA operational capabilities
+ * @DMA_MEMCPY: src to dest copy
+ * @DMA_XOR: src*n to dest xor
+ * @DMA_DUAL_XOR: src*n to dest_diag and dest_horiz xor
+ * @DMA_PQ_XOR: src*n to dest_q and dest_p gf/xor
+ * @DMA_MEMCPY_CRC32C: src to dest copy and crc-32c sum
+ * @DMA_SHARE: multiple clients can use this channel
+ */

include/linux/dmaengine.h enum dma_status { · line 85

Quoted 10 lines
include/linux/dmaengine.h +10 -0
+enum dma_capabilities {
+     DMA_MEMCPY              = 0x1,
+     DMA_XOR                 = 0x2,
+     DMA_PQ_XOR              = 0x4,
+     DMA_DUAL_XOR            = 0x8,
+     DMA_PQ_UPDATE           = 0x10,
+     DMA_ZERO_SUM            = 0x20,
+     DMA_PQ_ZERO_SUM         = 0x40,
+     DMA_MEMSET              = 0x80,
+     DMA_MEMCPY_CRC32C       = 0x100,

Please use the more readable style that explicitly lists bits:

Quoted 3 lines
        DMA_MEMCPY              = (1 << 0),
        DMA_XOR                 = (1 << 1),
        ...

I prefer this as well, although at one point I was told (not by you) the absolute number was preferred when I was making changes to drivers/scsi/sata_vsc.c. In any event I'll change it...

Quoted 15 lines

include/linux/dmaengine.h struct dma_chan_percpu { · line 101

Quoted 12 lines
include/linux/dmaengine.h +3 -0
+/**
  * struct dma_chan_percpu - the per-CPU part of struct dma_chan
  * @refcount: local_t used for open-coded "bigref" counting
  * @memcpy_count: transaction counter
@@ -75,27 +96,32 @@ struct dma_chan_percpu {
      local_t refcount;
      /* stats */
      unsigned long memcpy_count;
+     unsigned long xor_count;
      unsigned long bytes_transferred;
+     unsigned long bytes_xor;

Clearly, each operation needs to be more compartmentalized.

Quoted 1 line

This just isn't scalable, when you consider all the possible transforms.

Ok, one set of counters per op is probably overkill what about lumping operations into groups and just tracking at the group level? i.e.

memcpy, memset -> string_count, string_bytes_transferred crc, sha1, aes -> hash_count, hash_transferred xor, pq_xor -> sum_count, sum_transferred

Quoted 2 lines

Jeff

Dan

#13 lore raw

Re: [PATCH 08/19] dmaengine: enable multiple clients and operations

Jeff> Are we really going to add a set of hooks for each DMA Jeff> engine whizbang feature?

Dan> What's the alternative? But, also see patch 9 "dmaengine:
Dan> reduce backend address permutations" it relieves some of this Dan> pain.

I guess you can pass an opcode into a common "start operation" function.

With all the memcpy / xor / crypto / etc. hardware out there already, we definitely have to get this interface right.

- R.

#14 lore raw

Re: [PATCH 08/19] dmaengine: enable multiple clients and operations

Quoted 2 lines · Roland Dreier

Jeff> Are we really going to add a set of hooks for each DMA Jeff> engine whizbang feature?

...ok, but at some level we are going to need a file that has:
EXPORT_SYMBOL_GPL(dma_whizbang_op1)
. . .
EXPORT_SYMBOL_GPL(dma_whizbang_opX)
correct?

Quoted 5 lines

Dan> What's the alternative? But, also see patch 9 "dmaengine:
Dan> reduce backend address permutations" it relieves some of this Dan> pain.

I guess you can pass an opcode into a common "start operation" function.

But then we still have the problem of being able to request a memory copy operation of a channel that only understands xor, a la Jeff's comment to patch 12:

"Further illustration of how this API growth is going wrong. You should create an API such that it is impossible for an XOR transform to ever call non-XOR-transform hooks."

Quoted 4 lines

With all the memcpy / xor / crypto / etc. hardware out there already, we definitely have to get this interface right.

- R.

I understand what you are saying Jeff, the implementation can be made better, but something I think is valuable is the ability to write clients once like NET_DMA and RAID5_DMA and have them run without modification on any platform that can provide the engine interface rather than needing a client per architecture IOP_RAID5_DMA...FOO_X_RAID5_DMA.

Or is this an example of the where "Do What You Must, And No More" comes in, i.e. don't worry about making a generic RAID5_DMA while there is only one implementation existence?

I also want to pose the question of whether the dmaengine interface should handle cryptographic transforms? We already have Acrypto:
http://tservice.net.ru/~s0mbre/blog/devel/acrypto/index.html. At the same time since IOPs can do Galois Field multiplication and XOR it would be nice to take advantage of that for crypto acceleration, but this does not fit the model of a device that Acrypto supports.

Dan

#15 lore raw

Re: [PATCH 08/19] dmaengine: enable multiple clients and operations

Quoted 10 lines · Dan Williams

Or is this an example of the where "Do What You Must, And No More" comes in, i.e. don't worry about making a generic RAID5_DMA while there is only one implementation existence?

I also want to pose the question of whether the dmaengine interface should handle cryptographic transforms? We already have Acrypto:
http://tservice.net.ru/~s0mbre/blog/devel/acrypto/index.html. At the same time since IOPs can do Galois Field multiplication and XOR it would be nice to take advantage of that for crypto acceleration, but this does not fit the model of a device that Acrypto supports.

Each acrypto crypto device provides set of capabilities it supports, and when user requests some operation, acrypto core selects device with the maximum speed for given capabilities, so one can easily add there GF multiplication devices. Acrypto supports "sync" mode too in case your hardware is synchronous (i.e. it does not provide interrupt or other async event when operation is completed).

P.S. acrypto homepage with some design notes and supported features can be found here:
http://tservice.net.ru/~s0mbre/old/?section=projects&item=acrypto

Quoted 1 line

Dan

Signature · 1 line
	Evgeniy Polyakov
#16 lore raw

Re: [PATCH 08/19] dmaengine: enable multiple clients and operations

In reply to Dan Williams
Quoted 8 lines · Dan Williams
Quoted 2 lines · Roland Dreier

Jeff> Are we really going to add a set of hooks for each DMA Jeff> engine whizbang feature?

...ok, but at some level we are going to need a file that has:
EXPORT_SYMBOL_GPL(dma_whizbang_op1)
. . .
EXPORT_SYMBOL_GPL(dma_whizbang_opX)
correct?

If properly modularized, you'll have multiple files with such exports.

Or perhaps you won't have such exports at all, if it is hidden inside a module-specific struct-of-hooks.

Quoted 6 lines

I understand what you are saying Jeff, the implementation can be made better, but something I think is valuable is the ability to write clients once like NET_DMA and RAID5_DMA and have them run without modification on any platform that can provide the engine interface rather than needing a client per architecture IOP_RAID5_DMA...FOO_X_RAID5_DMA.

It depends on the situation.

The hardware capabilities exported by each platform[or device] vary greatly, not only in the raw capabilities provided, but also in the level of offload.

In general, we don't want to see hardware-specific stuff in generic code, though...

Quoted 10 lines

Or is this an example of the where "Do What You Must, And No More" comes in, i.e. don't worry about making a generic RAID5_DMA while there is only one implementation existence?

I also want to pose the question of whether the dmaengine interface should handle cryptographic transforms? We already have Acrypto:
http://tservice.net.ru/~s0mbre/blog/devel/acrypto/index.html. At the same time since IOPs can do Galois Field multiplication and XOR it would be nice to take advantage of that for crypto acceleration, but this does not fit the model of a device that Acrypto supports.

It would be quite interesting to see where the synergies are between the two, at the very least. "async [transform|sum]" is a superset of "async crypto" after all.

Jeff

#17 lore raw

Re: [PATCH 08/19] dmaengine: enable multiple clients and operations

In reply to Jeff Garzik
Quoted 16 lines · On Mon, 11 Sep 2006 19:44:16 -0400 Jeff Garzik <[email protected]> wrote:

Review #endif · line 761

Quoted 10 lines · Dan Williams
(quoted diff) +4 -2
@@ -759,8 +755,10 @@ #endif
 	device->common.device_memcpy_buf_to_buf = ioat_dma_memcpy_buf_to_buf;
 	device->common.device_memcpy_buf_to_pg = ioat_dma_memcpy_buf_to_pg;
 	device->common.device_memcpy_pg_to_pg = ioat_dma_memcpy_pg_to_pg;
-	device->common.device_memcpy_complete = ioat_dma_is_complete;
-	device->common.device_memcpy_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.device_operation_complete = ioat_dma_is_complete;
+	device->common.device_xor_pgs_to_pg = dma_async_xor_pgs_to_pg_err;
+	device->common.device_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.capabilities = DMA_MEMCPY;

Are we really going to add a set of hooks for each DMA engine whizbang feature?

Quoted 2 lines

That will get ugly when DMA engines support memcpy, xor, crc32, sha1, aes, and a dozen other transforms.

Yes, it will be unmaintainable. We need some sort of multiplexing with per-function registrations.

Here's a first cut at it, just very quick. It could be improved further but it shows that we could exorcise most of the hardcoded things pretty easily.

Dan, would this fit with your added XOR stuff as well? If so, would you mind rebasing on top of something like this (with your further cleanups going in before added function, please. :-)

(Build tested only, since I lack Intel hardware).

It would be nice if we could move the type specification to only be needed in the channel allocation. I don't know how well that fits the model for some of the hardware platforms though, since a single channel might be shared for different types of functions. Maybe we need a different level of abstraction there instead, i.e. divorce the hardware channel and software channel model and have several software channels map onto a hardware one.

Clean up the DMA API a bit, allowing each engine to register an array of supported functions instead of allocating static names for each possible function.

drivers/dma/dmaengine.c +7 -7
index 1527804..282ce85 100644
@@ -80,7 +80,7 @@ static ssize_t show_memcpy_count(struct 
 	int i;
 
 	for_each_possible_cpu(i)
-		count += per_cpu_ptr(chan->local, i)->memcpy_count;
+		count += per_cpu_ptr(chan->local, i)->count;
 
 	return sprintf(buf, "%lu\n", count);
 }
@@ -105,7 +105,7 @@ static ssize_t show_in_use(struct class_
 }
 
 static struct class_device_attribute dma_class_attrs[] = {
-	__ATTR(memcpy_count, S_IRUGO, show_memcpy_count, NULL),
+	__ATTR(count, S_IRUGO, show_memcpy_count, NULL),
 	__ATTR(bytes_transferred, S_IRUGO, show_bytes_transferred, NULL),
 	__ATTR(in_use, S_IRUGO, show_in_use, NULL),
 	__ATTR_NULL
@@ -402,11 +402,11 @@ subsys_initcall(dma_bus_init);
 EXPORT_SYMBOL(dma_async_client_register);
 EXPORT_SYMBOL(dma_async_client_unregister);
 EXPORT_SYMBOL(dma_async_client_chan_request);
-EXPORT_SYMBOL(dma_async_memcpy_buf_to_buf);
-EXPORT_SYMBOL(dma_async_memcpy_buf_to_pg);
-EXPORT_SYMBOL(dma_async_memcpy_pg_to_pg);
-EXPORT_SYMBOL(dma_async_memcpy_complete);
-EXPORT_SYMBOL(dma_async_memcpy_issue_pending);
+EXPORT_SYMBOL(dma_async_buf_to_buf);
+EXPORT_SYMBOL(dma_async_buf_to_pg);
+EXPORT_SYMBOL(dma_async_pg_to_pg);
+EXPORT_SYMBOL(dma_async_complete);
+EXPORT_SYMBOL(dma_async_issue_pending);
 EXPORT_SYMBOL(dma_async_device_register);
 EXPORT_SYMBOL(dma_async_device_unregister);
 EXPORT_SYMBOL(dma_chan_cleanup);
drivers/dma/ioatdma.c +11 -5
index dbd4d6c..6cbed42 100644
@@ -40,6 +40,7 @@
 #define to_ioat_device(dev) container_of(dev, struct ioat_device, common)
 #define to_ioat_desc(lh) container_of(lh, struct ioat_desc_sw, node)
 
+
 /* internal functions */
 static int __devinit ioat_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
 static void __devexit ioat_remove(struct pci_dev *pdev);
@@ -681,6 +682,14 @@ out:
 	return err;
 }
 
+struct dma_function ioat_memcpy_functions = {
+	.buf_to_buf = ioat_dma_memcpy_buf_to_buf,
+	.buf_to_pg = ioat_dma_memcpy_buf_to_pg,
+	.pg_to_pg = ioat_dma_memcpy_pg_to_pg,
+	.complete = ioat_dma_is_complete,
+	.issue_pending = ioat_dma_memcpy_issue_pending,
+};
+
 static int __devinit ioat_probe(struct pci_dev *pdev,
                                 const struct pci_device_id *ent)
 {
@@ -756,11 +765,8 @@ static int __devinit ioat_probe(struct p
 
 	device->common.device_alloc_chan_resources = ioat_dma_alloc_chan_resources;
 	device->common.device_free_chan_resources = ioat_dma_free_chan_resources;
-	device->common.device_memcpy_buf_to_buf = ioat_dma_memcpy_buf_to_buf;
-	device->common.device_memcpy_buf_to_pg = ioat_dma_memcpy_buf_to_pg;
-	device->common.device_memcpy_pg_to_pg = ioat_dma_memcpy_pg_to_pg;
-	device->common.device_memcpy_complete = ioat_dma_is_complete;
-	device->common.device_memcpy_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.funcs[DMAFUNC_MEMCPY] = &ioat_memcpy_functions;
+
 	printk(KERN_INFO "Intel(R) I/OAT DMA Engine found, %d channels\n",
 		device->common.chancnt);
 
drivers/dma/iovlock.c +4 -7
index d637555..8a2f642 100644
@@ -151,11 +151,8 @@ static dma_cookie_t dma_memcpy_to_kernel
 	while (len > 0) {
 		if (iov->iov_len) {
 			int copy = min_t(unsigned int, iov->iov_len, len);
-			dma_cookie = dma_async_memcpy_buf_to_buf(
-					chan,
-					iov->iov_base,
-					kdata,
-					copy);
+			dma_cookie = dma_async_buf_to_buf(DMAFUNC_MEMCPY, chan,
+					iov->iov_base, kdata, copy);
 			kdata += copy;
 			len -= copy;
 			iov->iov_len -= copy;
@@ -210,7 +207,7 @@ dma_cookie_t dma_memcpy_to_iovec(struct 
 			copy = min_t(int, PAGE_SIZE - iov_byte_offset, len);
 			copy = min_t(int, copy, iov[iovec_idx].iov_len);
 
-			dma_cookie = dma_async_memcpy_buf_to_pg(chan,
+			dma_cookie = dma_async_buf_to_pg(DMAFUNC_MEMCPY, chan,
 					page_list->pages[page_idx],
 					iov_byte_offset,
 					kdata,
@@ -274,7 +271,7 @@ dma_cookie_t dma_memcpy_pg_to_iovec(stru
 			copy = min_t(int, PAGE_SIZE - iov_byte_offset, len);
 			copy = min_t(int, copy, iov[iovec_idx].iov_len);
 
-			dma_cookie = dma_async_memcpy_pg_to_pg(chan,
+			dma_cookie = dma_async_pg_to_pg(DMAFUNC_MEMCPY, chan,
 					page_list->pages[page_idx],
 					iov_byte_offset,
 					page,
include/linux/dmaengine.h +83 -49

233 lines not on this page · raw message

net/core/dev.c +1 -1
index d4a1ec3..e8a8ee9 100644
@@ -1945,7 +1945,7 @@ out:
 		struct dma_chan *chan;
 		rcu_read_lock();
 		list_for_each_entry_rcu(chan, &net_dma_client->channels, client_node)
-			dma_async_memcpy_issue_pending(chan);
+			dma_async_issue_pending(DMAFUNC_MEMCPY, chan);
 		rcu_read_unlock();
 	}
 #endif
net/ipv4/tcp.c +2 -2
index 934396b..c270837 100644
@@ -1431,9 +1431,9 @@ skip_copy:
 		struct sk_buff *skb;
 		dma_cookie_t done, used;
 
-		dma_async_memcpy_issue_pending(tp->ucopy.dma_chan);
+		dma_async_issue_pending(DMAFUNC_MEMCPY, tp->ucopy.dma_chan);
 
-		while (dma_async_memcpy_complete(tp->ucopy.dma_chan,
+		while (dma_async_complete(DMAFUNC_MEMCPY, tp->ucopy.dma_chan,
 		                                 tp->ucopy.dma_cookie, &done,
 		                                 &used) == DMA_IN_PROGRESS) {
 			/* do partial cleanup of sk_async_wait_queue */
#18 lore raw

PATCH 1/1 dmaengine: clean up and abstract function types (was Re: [PATCH 08/19] dmaengine: enable multiple clients and operations)

Quoted 9 lines · On Fri, 15 Sep 2006 11:38:17 -0500 Olof Johansson <[email protected]> wrote:
Quoted 5 lines · On Mon, 11 Sep 2006 19:44:16 -0400 Jeff Garzik <[email protected]> wrote:

Are we really going to add a set of hooks for each DMA engine whizbang feature?

That will get ugly when DMA engines support memcpy, xor, crc32, sha1, aes, and a dozen other transforms.

Quoted 6 lines

Yes, it will be unmaintainable. We need some sort of multiplexing with per-function registrations.

Here's a first cut at it, just very quick. It could be improved further but it shows that we could exorcise most of the hardcoded things pretty easily.

Ok, that was obviously a naive and not so nice first attempt, but I figured it was worth it to show how it can be done.

This is a little more proper: Specify at client registration time what the function the client will use is, and make the channel use it. This way most of the error checking per call can be removed too.

Chris/Dan: Please consider picking this up as a base for the added functionality and cleanups.

Clean up dmaengine a bit. Make the client registration specify which channel functions ("type") the client will use. Also, make devices register which functions they will provide.

Also exorcise most of the memcpy-specific references from the generic dma engine code. There's still some left in the iov stuff.

drivers/dma/dmaengine.c +15 -9
Index: linux-2.6/drivers/dma/dmaengine.c
===================================================================
@@ -73,14 +73,14 @@ static LIST_HEAD(dma_client_list);
 
 /* --- sysfs implementation --- */
 
-static ssize_t show_memcpy_count(struct class_device *cd, char *buf)
+static ssize_t show_count(struct class_device *cd, char *buf)
 {
 	struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
 	unsigned long count = 0;
 	int i;
 
 	for_each_possible_cpu(i)
-		count += per_cpu_ptr(chan->local, i)->memcpy_count;
+		count += per_cpu_ptr(chan->local, i)->count;
 
 	return sprintf(buf, "%lu\n", count);
 }
@@ -105,7 +105,7 @@ static ssize_t show_in_use(struct class_
 }
 
 static struct class_device_attribute dma_class_attrs[] = {
-	__ATTR(memcpy_count, S_IRUGO, show_memcpy_count, NULL),
+	__ATTR(count, S_IRUGO, show_count, NULL),
 	__ATTR(bytes_transferred, S_IRUGO, show_bytes_transferred, NULL),
 	__ATTR(in_use, S_IRUGO, show_in_use, NULL),
 	__ATTR_NULL
@@ -142,6 +142,10 @@ static struct dma_chan *dma_client_chan_
 
 	/* Find a channel, any DMA engine will do */
 	list_for_each_entry(device, &dma_device_list, global_node) {
+		/* Skip devices that don't provide the right function */
+		if (!device->funcs[client->type])
+			continue;
+
 		list_for_each_entry(chan, &device->channels, device_node) {
 			if (chan->client)
 				continue;
@@ -241,7 +245,8 @@ static void dma_chans_rebalance(void)
  * dma_async_client_register - allocate and register a &dma_client
  * @event_callback: callback for notification of channel addition/removal
  */
-struct dma_client *dma_async_client_register(dma_event_callback event_callback)
+struct dma_client *dma_async_client_register(enum dma_function_type type,
+		dma_event_callback event_callback)
 {
 	struct dma_client *client;
 
@@ -254,6 +259,7 @@ struct dma_client *dma_async_client_regi
 	client->chans_desired = 0;
 	client->chan_count = 0;
 	client->event_callback = event_callback;
+	client->type = type;
 
 	mutex_lock(&dma_list_mutex);
 	list_add_tail(&client->global_node, &dma_client_list);
@@ -402,11 +408,11 @@ subsys_initcall(dma_bus_init);
 EXPORT_SYMBOL(dma_async_client_register);
 EXPORT_SYMBOL(dma_async_client_unregister);
 EXPORT_SYMBOL(dma_async_client_chan_request);
-EXPORT_SYMBOL(dma_async_memcpy_buf_to_buf);
-EXPORT_SYMBOL(dma_async_memcpy_buf_to_pg);
-EXPORT_SYMBOL(dma_async_memcpy_pg_to_pg);
-EXPORT_SYMBOL(dma_async_memcpy_complete);
-EXPORT_SYMBOL(dma_async_memcpy_issue_pending);
+EXPORT_SYMBOL(dma_async_buf_to_buf);
+EXPORT_SYMBOL(dma_async_buf_to_pg);
+EXPORT_SYMBOL(dma_async_pg_to_pg);
+EXPORT_SYMBOL(dma_async_complete);
+EXPORT_SYMBOL(dma_async_issue_pending);
 EXPORT_SYMBOL(dma_async_device_register);
 EXPORT_SYMBOL(dma_async_device_unregister);
 EXPORT_SYMBOL(dma_chan_cleanup);
drivers/dma/ioatdma.c +10 -5
Index: linux-2.6/drivers/dma/ioatdma.c
===================================================================
@@ -681,6 +682,14 @@ out:
 	return err;
 }
 
+struct dma_function ioat_memcpy_functions = {
+	.buf_to_buf = ioat_dma_memcpy_buf_to_buf,
+	.buf_to_pg = ioat_dma_memcpy_buf_to_pg,
+	.pg_to_pg = ioat_dma_memcpy_pg_to_pg,
+	.complete = ioat_dma_is_complete,
+	.issue_pending = ioat_dma_memcpy_issue_pending,
+};
+
 static int __devinit ioat_probe(struct pci_dev *pdev,
                                 const struct pci_device_id *ent)
 {
@@ -756,11 +765,8 @@ static int __devinit ioat_probe(struct p
 
 	device->common.device_alloc_chan_resources = ioat_dma_alloc_chan_resources;
 	device->common.device_free_chan_resources = ioat_dma_free_chan_resources;
-	device->common.device_memcpy_buf_to_buf = ioat_dma_memcpy_buf_to_buf;
-	device->common.device_memcpy_buf_to_pg = ioat_dma_memcpy_buf_to_pg;
-	device->common.device_memcpy_pg_to_pg = ioat_dma_memcpy_pg_to_pg;
-	device->common.device_memcpy_complete = ioat_dma_is_complete;
-	device->common.device_memcpy_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.funcs[DMAFUNC_MEMCPY] = &ioat_memcpy_functions;
+
 	printk(KERN_INFO "Intel(R) I/OAT DMA Engine found, %d channels\n",
 		device->common.chancnt);
 
include/linux/dmaengine.h +69 -47

234 lines not on this page · raw message

net/core/dev.c +2 -2
Index: linux-2.6/net/core/dev.c
===================================================================
@@ -1945,7 +1945,7 @@ out:
 		struct dma_chan *chan;
 		rcu_read_lock();
 		list_for_each_entry_rcu(chan, &net_dma_client->channels, client_node)
-			dma_async_memcpy_issue_pending(chan);
+			dma_async_issue_pending(chan);
 		rcu_read_unlock();
 	}
 #endif
@@ -3467,7 +3467,7 @@ static void netdev_dma_event(struct dma_
 static int __init netdev_dma_register(void)
 {
 	spin_lock_init(&net_dma_event_lock);
-	net_dma_client = dma_async_client_register(netdev_dma_event);
+	net_dma_client = dma_async_client_register(DMAFUNC_MEMCPY, netdev_dma_event);
 	if (net_dma_client == NULL)
 		return -ENOMEM;
 
#19 lore raw

PATCH v2 1/1 dmaengine: clean up and abstract function types (was Re: [PATCH 08/19] dmaengine: enable multiple clients and operations)

[Bad day, forgot a quilt refresh.]

Clean up dmaengine a bit. Make the client registration specify which channel functions ("type") the client will use. Also, make devices register which functions they will provide.

Also exorcise most of the memcpy-specific references from the generic dma engine code. There's still some left in the iov stuff.

drivers/dma/dmaengine.c +15 -9
Index: linux-2.6/drivers/dma/dmaengine.c
===================================================================
@@ -73,14 +73,14 @@ static LIST_HEAD(dma_client_list);
 
 /* --- sysfs implementation --- */
 
-static ssize_t show_memcpy_count(struct class_device *cd, char *buf)
+static ssize_t show_count(struct class_device *cd, char *buf)
 {
 	struct dma_chan *chan = container_of(cd, struct dma_chan, class_dev);
 	unsigned long count = 0;
 	int i;
 
 	for_each_possible_cpu(i)
-		count += per_cpu_ptr(chan->local, i)->memcpy_count;
+		count += per_cpu_ptr(chan->local, i)->count;
 
 	return sprintf(buf, "%lu\n", count);
 }
@@ -105,7 +105,7 @@ static ssize_t show_in_use(struct class_
 }
 
 static struct class_device_attribute dma_class_attrs[] = {
-	__ATTR(memcpy_count, S_IRUGO, show_memcpy_count, NULL),
+	__ATTR(count, S_IRUGO, show_count, NULL),
 	__ATTR(bytes_transferred, S_IRUGO, show_bytes_transferred, NULL),
 	__ATTR(in_use, S_IRUGO, show_in_use, NULL),
 	__ATTR_NULL
@@ -142,6 +142,10 @@ static struct dma_chan *dma_client_chan_
 
 	/* Find a channel, any DMA engine will do */
 	list_for_each_entry(device, &dma_device_list, global_node) {
+		/* Skip devices that don't provide the right function */
+		if (!device->funcs[client->type])
+			continue;
+
 		list_for_each_entry(chan, &device->channels, device_node) {
 			if (chan->client)
 				continue;
@@ -241,7 +245,8 @@ static void dma_chans_rebalance(void)
  * dma_async_client_register - allocate and register a &dma_client
  * @event_callback: callback for notification of channel addition/removal
  */
-struct dma_client *dma_async_client_register(dma_event_callback event_callback)
+struct dma_client *dma_async_client_register(enum dma_function_type type,
+		dma_event_callback event_callback)
 {
 	struct dma_client *client;
 
@@ -254,6 +259,7 @@ struct dma_client *dma_async_client_regi
 	client->chans_desired = 0;
 	client->chan_count = 0;
 	client->event_callback = event_callback;
+	client->type = type;
 
 	mutex_lock(&dma_list_mutex);
 	list_add_tail(&client->global_node, &dma_client_list);
@@ -402,11 +408,11 @@ subsys_initcall(dma_bus_init);
 EXPORT_SYMBOL(dma_async_client_register);
 EXPORT_SYMBOL(dma_async_client_unregister);
 EXPORT_SYMBOL(dma_async_client_chan_request);
-EXPORT_SYMBOL(dma_async_memcpy_buf_to_buf);
-EXPORT_SYMBOL(dma_async_memcpy_buf_to_pg);
-EXPORT_SYMBOL(dma_async_memcpy_pg_to_pg);
-EXPORT_SYMBOL(dma_async_memcpy_complete);
-EXPORT_SYMBOL(dma_async_memcpy_issue_pending);
+EXPORT_SYMBOL(dma_async_buf_to_buf);
+EXPORT_SYMBOL(dma_async_buf_to_pg);
+EXPORT_SYMBOL(dma_async_pg_to_pg);
+EXPORT_SYMBOL(dma_async_complete);
+EXPORT_SYMBOL(dma_async_issue_pending);
 EXPORT_SYMBOL(dma_async_device_register);
 EXPORT_SYMBOL(dma_async_device_unregister);
 EXPORT_SYMBOL(dma_chan_cleanup);
drivers/dma/ioatdma.c +11 -5
Index: linux-2.6/drivers/dma/ioatdma.c
===================================================================
@@ -40,6 +40,7 @@
 #define to_ioat_device(dev) container_of(dev, struct ioat_device, common)
 #define to_ioat_desc(lh) container_of(lh, struct ioat_desc_sw, node)
 
+
 /* internal functions */
 static int __devinit ioat_probe(struct pci_dev *pdev, const struct pci_device_id *ent);
 static void __devexit ioat_remove(struct pci_dev *pdev);
@@ -681,6 +682,14 @@ out:
 	return err;
 }
 
+struct dma_function ioat_memcpy_functions = {
+	.buf_to_buf = ioat_dma_memcpy_buf_to_buf,
+	.buf_to_pg = ioat_dma_memcpy_buf_to_pg,
+	.pg_to_pg = ioat_dma_memcpy_pg_to_pg,
+	.complete = ioat_dma_is_complete,
+	.issue_pending = ioat_dma_memcpy_issue_pending,
+};
+
 static int __devinit ioat_probe(struct pci_dev *pdev,
                                 const struct pci_device_id *ent)
 {
@@ -756,11 +765,8 @@ static int __devinit ioat_probe(struct p
 
 	device->common.device_alloc_chan_resources = ioat_dma_alloc_chan_resources;
 	device->common.device_free_chan_resources = ioat_dma_free_chan_resources;
-	device->common.device_memcpy_buf_to_buf = ioat_dma_memcpy_buf_to_buf;
-	device->common.device_memcpy_buf_to_pg = ioat_dma_memcpy_buf_to_pg;
-	device->common.device_memcpy_pg_to_pg = ioat_dma_memcpy_pg_to_pg;
-	device->common.device_memcpy_complete = ioat_dma_is_complete;
-	device->common.device_memcpy_issue_pending = ioat_dma_memcpy_issue_pending;
+	device->common.funcs[DMAFUNC_MEMCPY] = &ioat_memcpy_functions;
+
 	printk(KERN_INFO "Intel(R) I/OAT DMA Engine found, %d channels\n",
 		device->common.chancnt);
 
include/linux/dmaengine.h +69 -47

234 lines not on this page · raw message

net/core/dev.c +2 -2
Index: linux-2.6/net/core/dev.c
===================================================================
@@ -1945,7 +1945,7 @@ out:
 		struct dma_chan *chan;
 		rcu_read_lock();
 		list_for_each_entry_rcu(chan, &net_dma_client->channels, client_node)
-			dma_async_memcpy_issue_pending(chan);
+			dma_async_issue_pending(chan);
 		rcu_read_unlock();
 	}
 #endif
@@ -3467,7 +3467,7 @@ static void netdev_dma_event(struct dma_
 static int __init netdev_dma_register(void)
 {
 	spin_lock_init(&net_dma_event_lock);
-	net_dma_client = dma_async_client_register(netdev_dma_event);
+	net_dma_client = dma_async_client_register(DMAFUNC_MEMCPY, netdev_dma_event);
 	if (net_dma_client == NULL)
 		return -ENOMEM;
 
drivers/dma/iovlock.c +3 -3
Index: linux-2.6/drivers/dma/iovlock.c
===================================================================
@@ -151,7 +151,7 @@ static dma_cookie_t dma_memcpy_to_kernel
 	while (len > 0) {
 		if (iov->iov_len) {
 			int copy = min_t(unsigned int, iov->iov_len, len);
-			dma_cookie = dma_async_memcpy_buf_to_buf(
+			dma_cookie = dma_async_buf_to_buf(
 					chan,
 					iov->iov_base,
 					kdata,
@@ -210,7 +210,7 @@ dma_cookie_t dma_memcpy_to_iovec(struct 
 			copy = min_t(int, PAGE_SIZE - iov_byte_offset, len);
 			copy = min_t(int, copy, iov[iovec_idx].iov_len);
 
-			dma_cookie = dma_async_memcpy_buf_to_pg(chan,
+			dma_cookie = dma_async_buf_to_pg(chan,
 					page_list->pages[page_idx],
 					iov_byte_offset,
 					kdata,
@@ -274,7 +274,7 @@ dma_cookie_t dma_memcpy_pg_to_iovec(stru
 			copy = min_t(int, PAGE_SIZE - iov_byte_offset, len);
 			copy = min_t(int, copy, iov[iovec_idx].iov_len);
 
-			dma_cookie = dma_async_memcpy_pg_to_pg(chan,
+			dma_cookie = dma_async_pg_to_pg(chan,
 					page_list->pages[page_idx],
 					iov_byte_offset,
 					page,
net/ipv4/tcp.c +4 -4
Index: linux-2.6/net/ipv4/tcp.c
===================================================================
@@ -1431,11 +1431,11 @@ skip_copy:
 		struct sk_buff *skb;
 		dma_cookie_t done, used;
 
-		dma_async_memcpy_issue_pending(tp->ucopy.dma_chan);
+		dma_async_issue_pending(tp->ucopy.dma_chan);
 
-		while (dma_async_memcpy_complete(tp->ucopy.dma_chan,
-		                                 tp->ucopy.dma_cookie, &done,
-		                                 &used) == DMA_IN_PROGRESS) {
+		while (dma_async_complete(tp->ucopy.dma_chan,
+		                          tp->ucopy.dma_cookie, &done,
+		                          &used) == DMA_IN_PROGRESS) {
 			/* do partial cleanup of sk_async_wait_queue */
 			while ((skb = skb_peek(&sk->sk_async_wait_queue)) &&
 			       (dma_async_is_complete(skb->dma_cookie, done,
#20 lore raw

Re: [PATCH] dmaengine: clean up and abstract function types (was Re: [PATCH 08/19] dmaengine: enable multiple clients and operations)

In reply to Olof Johansson
Quoted 21 lines · Olof Johansson
Quoted 9 lines · On Fri, 15 Sep 2006 11:38:17 -0500 Olof Johansson <[email protected]> wrote:
Quoted 5 lines · On Mon, 11 Sep 2006 19:44:16 -0400 Jeff Garzik <[email protected]> wrote:

Are we really going to add a set of hooks for each DMA engine whizbang feature?

That will get ugly when DMA engines support memcpy, xor, crc32, sha1, aes, and a dozen other transforms.

Quoted 6 lines

Yes, it will be unmaintainable. We need some sort of multiplexing with per-function registrations.

Here's a first cut at it, just very quick. It could be improved further but it shows that we could exorcise most of the hardcoded things pretty easily.

Ok, that was obviously a naive and not so nice first attempt, but I figured it was worth it to show how it can be done.

Quoted 7 lines

This is a little more proper: Specify at client registration time what the function the client will use is, and make the channel use it. This way most of the error checking per call can be removed too.

Chris/Dan: Please consider picking this up as a base for the added functionality and cleanups.

Thanks for this Olof it has sparked some ideas about how to redo support for multiple operations.

Quoted 10 lines

Clean up dmaengine a bit. Make the client registration specify which channel functions ("type") the client will use. Also, make devices register which functions they will provide.

Also exorcise most of the memcpy-specific references from the generic dma engine code. There's still some left in the iov stuff.

I think we should keep the operation type in the function name but drop all the [buf|pg|dma]_to_[buf|pg|dma] permutations. The buffer type can be handled generically across all operation types. Something like the following for a pg_to_buf memcpy.

struct dma_async_op_memcpy *op;
struct page *pg;
void *buf;
size_t len;

dma_async_op_init_src_pg(op, pg);
dma_async_op_init_dest_buf(op, buf);
dma_async_memcpy(chan, op, len);

-Dan

#21 lore raw

Re: [PATCH] dmaengine: clean up and abstract function types (was Re: [PATCH 08/19] dmaengine: enable multiple clients and operations)

Quoted 8 lines · On Mon, 18 Sep 2006 15:56:37 -0700 "Dan Williams" <[email protected]>…
Quoted 5 lines · Olof Johansson

On Fri, 15 Sep 2006 11:38:17 -0500 Olof Johansson <[email protected]> wrote:

Chris/Dan: Please consider picking this up as a base for the added functionality and cleanups.

Thanks for this Olof it has sparked some ideas about how to redo support for multiple operations.

Good. :)

Quoted 5 lines

I think we should keep the operation type in the function name but drop all the [buf|pg|dma]_to_[buf|pg|dma] permutations. The buffer type can be handled generically across all operation types. Something like the following for a pg_to_buf memcpy.

Quoted 8 lines

struct dma_async_op_memcpy *op;
struct page *pg;
void *buf;
size_t len;

dma_async_op_init_src_pg(op, pg);
dma_async_op_init_dest_buf(op, buf);
dma_async_memcpy(chan, op, len);

I'm generally for a more generic interface, especially in the address permutation cases like above. However, I think it'll be a mistake to keep the association between the API and the function names and types so close.

What's the benefit of keeping a memcpy-specific dma_async_memcpy() instead of a more generic dma_async_commit() (or similar)? We'll know based on how the client/channel was allocated what kind of function is requested, won't we?

Same goes for the dma_async_op_memcpy. Make it an union that has a type field if you need per-operation settings. But as before, we'll know what kind of op structure gets passed in since we'll know what kind of operation is to be performed on it.

Finally, yet again the same goes for the op_init settings. I would even prefer it to not be function-based, instead just direct union/struct assignments.

struct dma_async_op op;
...

op.src_type = PG; op.src = pg;
op.dest_type = BUF; op.dest = buf;
op.len = len;
dma_async_commit(chan, op);

op might have to be dynamically allocated, since it'll outlive the scope of this function. But the idea would be the same.

-Olof

#22 lore raw

Re: [PATCH] dmaengine: clean up and abstract function types (was Re: [PATCH 08/19] dmaengine: enable multiple clients and operations)

Ar Llu, 2006-09-18 am 20:05 -0500, ysgrifennodd Olof Johansson:

Quoted 6 lines · On Mon, 18 Sep 2006 15:56:37 -0700 "Dan Williams" <[email protected]>…

op.src_type = PG; op.src = pg;
op.dest_type = BUF; op.dest = buf;
op.len = len;
dma_async_commit(chan, op);

At OLS Linus suggested it should distinguish between sync and async events for locking reasons.

	if(dma_async_commit(foo) == SYNC_COMPLETE) {
		finalise_stuff();
	}

	else		/* will call foo->callback(foo->dev_id) */

because otherwise you have locking complexities - the callback wants to take locks to guard the object it works on but if it is called synchronously - eg if hardware is busy and we fall back - it might deadlock with the caller of dmaa_async_foo() who also needs to hold the lock.

Alan

#23 lore raw

Re: [PATCH] dmaengine: clean up and abstract function types (was Re: [PATCH 08/19] dmaengine: enable multiple clients and operations)

Quoted 17 lines · On Tue, 19 Sep 2006 12:20:09 +0100 Alan Cox <[email protected]> wrote:

Ar Llu, 2006-09-18 am 20:05 -0500, ysgrifennodd Olof Johansson:

Quoted 6 lines · On Mon, 18 Sep 2006 15:56:37 -0700 "Dan Williams" <[email protected]>…

op.src_type = PG; op.src = pg;
op.dest_type = BUF; op.dest = buf;
op.len = len;
dma_async_commit(chan, op);

At OLS Linus suggested it should distinguish between sync and async events for locking reasons.

	if(dma_async_commit(foo) == SYNC_COMPLETE) {
		finalise_stuff();
	}

	else		/* will call foo->callback(foo->dev_id) */
Quoted 5 lines

because otherwise you have locking complexities - the callback wants to take locks to guard the object it works on but if it is called synchronously - eg if hardware is busy and we fall back - it might deadlock with the caller of dmaa_async_foo() who also needs to hold the lock.

Good point, sounds very reasonable to me.

-Olof

#24 lore raw

PATCH 9/19 dmaengine: reduce backend address permutations

In reply to Dan Williams

From: Dan Williams <[email protected]>

Change the backend dma driver API to accept a 'union dmaengine_addr'. The intent is to be able to support a wide range of frontend address type permutations without needing an equal number of function type permutations on the backend.

Changelog:
* make the dmaengine api EXPORT_SYMBOL_GPL
* zero sum support should be standalone, not integrated into xor

---

 drivers/dma/dmaengine.c   |   15 ++-
 drivers/dma/ioatdma.c     |  186 +++++++++++++++++--------------------------
 include/linux/dmaengine.h |  193 +++++++++++++++++++++++++++++++++++++++------
 3 files changed, 249 insertions(+), 145 deletions(-)
drivers/dma/dmaengine.c +10 -5
index e10f19d..9b02afa 100644
@@ -593,12 +593,13 @@ void dma_async_device_unregister(struct 
 }
 
 /**
- * dma_async_xor_pgs_to_pg_err - default function for dma devices that
+ * dma_async_do_xor_err - default function for dma devices that
  *	do not support xor
  */
-dma_cookie_t dma_async_xor_pgs_to_pg_err(struct dma_chan *chan,
-	struct page *dest_pg, unsigned int dest_off, struct page *src_pgs,
-	unsigned int src_cnt, unsigned int src_off, size_t len)
+dma_cookie_t dma_async_do_xor_err(struct dma_chan *chan,
+		union dmaengine_addr dest, unsigned int dest_off,
+		union dmaengine_addr src, unsigned int src_cnt,
+		unsigned int src_off, size_t len, unsigned long flags)
 {
 	return -ENXIO;
 }
@@ -617,11 +618,15 @@ EXPORT_SYMBOL_GPL(dma_async_client_chan_
 EXPORT_SYMBOL_GPL(dma_async_memcpy_buf_to_buf);
 EXPORT_SYMBOL_GPL(dma_async_memcpy_buf_to_pg);
 EXPORT_SYMBOL_GPL(dma_async_memcpy_pg_to_pg);
+EXPORT_SYMBOL_GPL(dma_async_memcpy_dma_to_dma);
+EXPORT_SYMBOL_GPL(dma_async_memcpy_pg_to_dma);
+EXPORT_SYMBOL_GPL(dma_async_memcpy_dma_to_pg);
 EXPORT_SYMBOL_GPL(dma_async_xor_pgs_to_pg);
+EXPORT_SYMBOL_GPL(dma_async_xor_dma_list_to_dma);
 EXPORT_SYMBOL_GPL(dma_async_operation_complete);
 EXPORT_SYMBOL_GPL(dma_async_issue_pending);
 EXPORT_SYMBOL_GPL(dma_async_device_register);
 EXPORT_SYMBOL_GPL(dma_async_device_unregister);
 EXPORT_SYMBOL_GPL(dma_chan_cleanup);
-EXPORT_SYMBOL_GPL(dma_async_xor_pgs_to_pg_err);
+EXPORT_SYMBOL_GPL(dma_async_do_xor_err);
 EXPORT_SYMBOL_GPL(dma_async_chan_init);
drivers/dma/ioatdma.c +72 -114

275 lines not on this page · raw message

include/linux/dmaengine.h +167 -26

262 lines not on this page · raw message

#25 lore raw

Re: [PATCH 09/19] dmaengine: reduce backend address permutations

Hi,

Quoted 6 lines · On Mon, 11 Sep 2006 16:18:23 -0700 Dan Williams <[email protected]>…

From: Dan Williams <[email protected]>

Change the backend dma driver API to accept a 'union dmaengine_addr'. The intent is to be able to support a wide range of frontend address type permutations without needing an equal number of function type permutations on the backend.

Please do the cleanup of existing code before you apply new function. Earlier patches in this series added code that you're modifying here. If you modify the existing code first it's less churn for everyone to review.

Thanks,

Olof

#26 lore raw

PATCH 10/19 dmaengine: expose per channel dma mapping characteristics to clients

In reply to Dan Williams

From: Dan Williams <[email protected]>

Allow a client to ensure that the dma channel it has selected can dma to the specified buffer or page address. Also allow the client to pre-map address ranges to be passed to the operations API.

Changelog:
* make the dmaengine api EXPORT_SYMBOL_GPL
* zero sum support should be standalone, not integrated into xor

---

 drivers/dma/dmaengine.c   |    4 ++++
 drivers/dma/ioatdma.c     |   35 +++++++++++++++++++++++++++++++++++
 include/linux/dmaengine.h |   34 ++++++++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+), 0 deletions(-)
drivers/dma/dmaengine.c +4 -0
index 9b02afa..e78ce89 100644
@@ -630,3 +630,7 @@ EXPORT_SYMBOL_GPL(dma_async_device_unreg
 EXPORT_SYMBOL_GPL(dma_chan_cleanup);
 EXPORT_SYMBOL_GPL(dma_async_do_xor_err);
 EXPORT_SYMBOL_GPL(dma_async_chan_init);
+EXPORT_SYMBOL_GPL(dma_async_map_page);
+EXPORT_SYMBOL_GPL(dma_async_map_single);
+EXPORT_SYMBOL_GPL(dma_async_unmap_page);
+EXPORT_SYMBOL_GPL(dma_async_unmap_single);
drivers/dma/ioatdma.c +35 -0
index dd5b9f0..0159d14 100644
@@ -637,6 +637,37 @@ extern dma_cookie_t dma_async_do_xor_err
 	union dmaengine_addr src, unsigned int src_cnt,
 	unsigned int src_off, size_t len, unsigned long flags);
 
+static dma_addr_t ioat_map_page(struct dma_chan *chan, struct page *page,
+					unsigned long offset, size_t size,
+					int direction)
+{
+	struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+	return pci_map_page(ioat_chan->device->pdev, page, offset, size,
+			direction);
+}
+
+static dma_addr_t ioat_map_single(struct dma_chan *chan, void *cpu_addr,
+					size_t size, int direction)
+{
+	struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+	return pci_map_single(ioat_chan->device->pdev, cpu_addr, size,
+			direction);
+}
+
+static void ioat_unmap_page(struct dma_chan *chan, dma_addr_t handle,
+				size_t size, int direction)
+{
+	struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+	pci_unmap_page(ioat_chan->device->pdev, handle, size, direction);
+}
+
+static void ioat_unmap_single(struct dma_chan *chan, dma_addr_t handle,
+				size_t size, int direction)
+{
+	struct ioat_dma_chan *ioat_chan = to_ioat_chan(chan);
+	pci_unmap_single(ioat_chan->device->pdev, handle, size,	direction);
+}
+
 static int __devinit ioat_probe(struct pci_dev *pdev,
                                 const struct pci_device_id *ent)
 {
@@ -717,6 +748,10 @@ #endif
 	device->common.capabilities = DMA_MEMCPY;
 	device->common.device_do_dma_memcpy = do_ioat_dma_memcpy;
 	device->common.device_do_dma_xor = dma_async_do_xor_err;
+	device->common.map_page = ioat_map_page;
+	device->common.map_single = ioat_map_single;
+	device->common.unmap_page = ioat_unmap_page;
+	device->common.unmap_single = ioat_unmap_single;
 	printk(KERN_INFO "Intel(R) I/OAT DMA Engine found, %d channels\n",
 		device->common.chancnt);
 
include/linux/dmaengine.h +34 -0
index df055cc..cb4cfcf 100644
@@ -287,6 +287,15 @@ struct dma_device {
 	enum dma_status (*device_operation_complete)(struct dma_chan *chan,
 			dma_cookie_t cookie, dma_cookie_t *last,
 			dma_cookie_t *used);
+	dma_addr_t (*map_page)(struct dma_chan *chan, struct page *page,
+				unsigned long offset, size_t size,
+				int direction);
+	dma_addr_t (*map_single)(struct dma_chan *chan, void *cpu_addr,
+				size_t size, int direction);
+	void (*unmap_page)(struct dma_chan *chan, dma_addr_t handle,
+				size_t size, int direction);
+	void (*unmap_single)(struct dma_chan *chan, dma_addr_t handle,
+				size_t size, int direction);
 	void (*device_issue_pending)(struct dma_chan *chan);
 };
 
@@ -592,6 +601,31 @@ static inline enum dma_status dma_async_
 	return DMA_IN_PROGRESS;
 }
 
+static inline dma_addr_t dma_async_map_page(struct dma_chan *chan,
+			struct page *page, unsigned long offset, size_t size,
+			int direction)
+{
+	return chan->device->map_page(chan, page, offset, size, direction);
+}
+
+static inline dma_addr_t dma_async_map_single(struct dma_chan *chan,
+			void *cpu_addr,	size_t size, int direction)
+{
+	return chan->device->map_single(chan, cpu_addr, size, direction);
+}
+
+static inline void dma_async_unmap_page(struct dma_chan *chan,
+			dma_addr_t handle, size_t size, int direction)
+{
+	chan->device->unmap_page(chan, handle, size, direction);
+}
+
+static inline void dma_async_unmap_single(struct dma_chan *chan,
+			dma_addr_t handle, size_t size, int direction)
+{
+	chan->device->unmap_single(chan, handle, size, direction);
+}
+
 /* --- DMA device --- */
 
 int dma_async_device_register(struct dma_device *device);
#27 lore raw

PATCH 11/19 dmaengine: add memset as an asynchronous dma operation

In reply to Dan Williams

From: Dan Williams <[email protected]>

Changelog:
* make the dmaengine api EXPORT_SYMBOL_GPL
* zero sum support should be standalone, not integrated into xor

---

 drivers/dma/dmaengine.c   |   15 ++++++++++
 drivers/dma/ioatdma.c     |    5 +++
 include/linux/dmaengine.h |   68 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 88 insertions(+), 0 deletions(-)
drivers/dma/dmaengine.c +15 -0
index e78ce89..fe62237 100644
@@ -604,6 +604,17 @@ dma_cookie_t dma_async_do_xor_err(struct
 	return -ENXIO;
 }
 
+/**
+ * dma_async_do_memset_err - default function for dma devices that
+ *      do not support memset
+ */
+dma_cookie_t dma_async_do_memset_err(struct dma_chan *chan,
+                union dmaengine_addr dest, unsigned int dest_off,
+                int val, size_t len, unsigned long flags)
+{
+        return -ENXIO;
+}
+
 static int __init dma_bus_init(void)
 {
 	mutex_init(&dma_list_mutex);
@@ -621,6 +632,9 @@ EXPORT_SYMBOL_GPL(dma_async_memcpy_pg_to
 EXPORT_SYMBOL_GPL(dma_async_memcpy_dma_to_dma);
 EXPORT_SYMBOL_GPL(dma_async_memcpy_pg_to_dma);
 EXPORT_SYMBOL_GPL(dma_async_memcpy_dma_to_pg);
+EXPORT_SYMBOL_GPL(dma_async_memset_buf);
+EXPORT_SYMBOL_GPL(dma_async_memset_page);
+EXPORT_SYMBOL_GPL(dma_async_memset_dma);
 EXPORT_SYMBOL_GPL(dma_async_xor_pgs_to_pg);
 EXPORT_SYMBOL_GPL(dma_async_xor_dma_list_to_dma);
 EXPORT_SYMBOL_GPL(dma_async_operation_complete);
@@ -629,6 +643,7 @@ EXPORT_SYMBOL_GPL(dma_async_device_regis
 EXPORT_SYMBOL_GPL(dma_async_device_unregister);
 EXPORT_SYMBOL_GPL(dma_chan_cleanup);
 EXPORT_SYMBOL_GPL(dma_async_do_xor_err);
+EXPORT_SYMBOL_GPL(dma_async_do_memset_err);
 EXPORT_SYMBOL_GPL(dma_async_chan_init);
 EXPORT_SYMBOL_GPL(dma_async_map_page);
 EXPORT_SYMBOL_GPL(dma_async_map_single);
drivers/dma/ioatdma.c +5 -0
index 0159d14..231247c 100644
@@ -637,6 +637,10 @@ extern dma_cookie_t dma_async_do_xor_err
 	union dmaengine_addr src, unsigned int src_cnt,
 	unsigned int src_off, size_t len, unsigned long flags);
 
+extern dma_cookie_t dma_async_do_memset_err(struct dma_chan *chan,
+	union dmaengine_addr dest, unsigned int dest_off,
+	int val, size_t size, unsigned long flags);
+
 static dma_addr_t ioat_map_page(struct dma_chan *chan, struct page *page,
 					unsigned long offset, size_t size,
 					int direction)
@@ -748,6 +752,7 @@ #endif
 	device->common.capabilities = DMA_MEMCPY;
 	device->common.device_do_dma_memcpy = do_ioat_dma_memcpy;
 	device->common.device_do_dma_xor = dma_async_do_xor_err;
+	device->common.device_do_dma_memset = dma_async_do_memset_err;
 	device->common.map_page = ioat_map_page;
 	device->common.map_single = ioat_map_single;
 	device->common.unmap_page = ioat_unmap_page;
include/linux/dmaengine.h +68 -0

86 lines not on this page · raw message

#28 lore raw

Re: [PATCH 11/19] dmaengine: add memset as an asynchronous dma operation

Quoted 90 lines · Dan Williams

From: Dan Williams <[email protected]>

Changelog:
* make the dmaengine api EXPORT_SYMBOL_GPL
* zero sum support should be standalone, not integrated into xor

---

 drivers/dma/dmaengine.c   |   15 ++++++++++
 drivers/dma/ioatdma.c     |    5 +++
 include/linux/dmaengine.h |   68 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 88 insertions(+), 0 deletions(-)
drivers/dma/dmaengine.c +15 -0
index e78ce89..fe62237 100644
@@ -604,6 +604,17 @@ dma_cookie_t dma_async_do_xor_err(struct
 	return -ENXIO;
 }
 
+/**
+ * dma_async_do_memset_err - default function for dma devices that
+ *      do not support memset
+ */
+dma_cookie_t dma_async_do_memset_err(struct dma_chan *chan,
+                union dmaengine_addr dest, unsigned int dest_off,
+                int val, size_t len, unsigned long flags)
+{
+        return -ENXIO;
+}
+
 static int __init dma_bus_init(void)
 {
 	mutex_init(&dma_list_mutex);
@@ -621,6 +632,9 @@ EXPORT_SYMBOL_GPL(dma_async_memcpy_pg_to
 EXPORT_SYMBOL_GPL(dma_async_memcpy_dma_to_dma);
 EXPORT_SYMBOL_GPL(dma_async_memcpy_pg_to_dma);
 EXPORT_SYMBOL_GPL(dma_async_memcpy_dma_to_pg);
+EXPORT_SYMBOL_GPL(dma_async_memset_buf);
+EXPORT_SYMBOL_GPL(dma_async_memset_page);
+EXPORT_SYMBOL_GPL(dma_async_memset_dma);
 EXPORT_SYMBOL_GPL(dma_async_xor_pgs_to_pg);
 EXPORT_SYMBOL_GPL(dma_async_xor_dma_list_to_dma);
 EXPORT_SYMBOL_GPL(dma_async_operation_complete);
@@ -629,6 +643,7 @@ EXPORT_SYMBOL_GPL(dma_async_device_regis
 EXPORT_SYMBOL_GPL(dma_async_device_unregister);
 EXPORT_SYMBOL_GPL(dma_chan_cleanup);
 EXPORT_SYMBOL_GPL(dma_async_do_xor_err);
+EXPORT_SYMBOL_GPL(dma_async_do_memset_err);
 EXPORT_SYMBOL_GPL(dma_async_chan_init);
 EXPORT_SYMBOL_GPL(dma_async_map_page);
 EXPORT_SYMBOL_GPL(dma_async_map_single);
drivers/dma/ioatdma.c +5 -0
index 0159d14..231247c 100644
@@ -637,6 +637,10 @@ extern dma_cookie_t dma_async_do_xor_err
 	union dmaengine_addr src, unsigned int src_cnt,
 	unsigned int src_off, size_t len, unsigned long flags);
 
+extern dma_cookie_t dma_async_do_memset_err(struct dma_chan *chan,
+	union dmaengine_addr dest, unsigned int dest_off,
+	int val, size_t size, unsigned long flags);
+
 static dma_addr_t ioat_map_page(struct dma_chan *chan, struct page *page,
 					unsigned long offset, size_t size,
 					int direction)
@@ -748,6 +752,7 @@ #endif
 	device->common.capabilities = DMA_MEMCPY;
 	device->common.device_do_dma_memcpy = do_ioat_dma_memcpy;
 	device->common.device_do_dma_xor = dma_async_do_xor_err;
+	device->common.device_do_dma_memset = dma_async_do_memset_err;
 	device->common.map_page = ioat_map_page;
 	device->common.map_single = ioat_map_single;
 	device->common.unmap_page = ioat_unmap_page;
include/linux/dmaengine.h +1 -0
index cb4cfcf..8d53b08 100644
@@ -260,6 +260,7 @@ struct dma_chan_client_ref {
  * @device_issue_pending: push appended descriptors to hardware
  * @device_do_dma_memcpy: perform memcpy with a dma engine
  * @device_do_dma_xor: perform block xor with a dma engine
+ * @device_do_dma_memset: perform block fill with a dma engine
  */
 struct dma_device {
 

include/linux/dmaengine.h struct dma_device { · line 290

Quoted 6 lines
include/linux/dmaengine.h +3 -0
@@ -284,6 +285,9 @@ struct dma_device {
 			union dmaengine_addr src, unsigned int src_cnt,
 			unsigned int src_off, size_t len,
 			unsigned long flags);
+	dma_cookie_t (*device_do_dma_memset)(struct dma_chan *chan,
+			union dmaengine_addr dest, unsigned int dest_off,
+			int value, size_t len, unsigned long flags);

Same comment as for XOR: adding operations in this way just isn't scalable.

Operations need to be more compartmentalized.

Maybe a client could do:

	struct adma_transaction adma_xact;

	/* fill in hooks with XOR-specific info */
	init_XScale_xor(adma_device, &adma_xact, my_completion_func);

	/* initiate transaction */	
	adma_go(&adma_xact);

	/* callback signals completion asynchronously */
#29 lore raw

PATCH 12/19 dmaengine: dma_async_memcpy_err for DMA engines that do not support memcpy

In reply to Dan Williams

From: Dan Williams <[email protected]>

Default virtual function that returns an error if the user attempts a memcpy operation. An XOR engine is an example of a DMA engine that does not support memcpy.

---

 drivers/dma/dmaengine.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/dma/dmaengine.c +13 -0
index fe62237..33ad690 100644
@@ -593,6 +593,18 @@ void dma_async_device_unregister(struct 
 }
 
 /**
+ * dma_async_do_memcpy_err - default function for dma devices that
+ *	do not support memcpy
+ */
+dma_cookie_t dma_async_do_memcpy_err(struct dma_chan *chan,
+		union dmaengine_addr dest, unsigned int dest_off,
+		union dmaengine_addr src, unsigned int src_off,
+                size_t len, unsigned long flags)
+{
+	return -ENXIO;
+}
+
+/**
  * dma_async_do_xor_err - default function for dma devices that
  *	do not support xor
  */
@@ -642,6 +654,7 @@ EXPORT_SYMBOL_GPL(dma_async_issue_pendin
 EXPORT_SYMBOL_GPL(dma_async_device_register);
 EXPORT_SYMBOL_GPL(dma_async_device_unregister);
 EXPORT_SYMBOL_GPL(dma_chan_cleanup);
+EXPORT_SYMBOL_GPL(dma_async_do_memcpy_err);
 EXPORT_SYMBOL_GPL(dma_async_do_xor_err);
 EXPORT_SYMBOL_GPL(dma_async_do_memset_err);
 EXPORT_SYMBOL_GPL(dma_async_chan_init);
#30 lore raw

Re: [PATCH 12/19] dmaengine: dma_async_memcpy_err for DMA engines that do not support memcpy

Quoted 20 lines · Dan Williams

From: Dan Williams <[email protected]>

Default virtual function that returns an error if the user attempts a memcpy operation. An XOR engine is an example of a DMA engine that does not support memcpy.

---

 drivers/dma/dmaengine.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/dma/dmaengine.c +0 -0
index fe62237..33ad690 100644
@@ -593,6 +593,18 @@ void dma_async_device_unregister(struct 
 }
 
 /**

drivers/dma/dmaengine.c void dma_async_device_unregister(struct · line 605

Quoted 10 lines
drivers/dma/dmaengine.c +10 -0
+ * dma_async_do_memcpy_err - default function for dma devices that
+ *	do not support memcpy
+ */
+dma_cookie_t dma_async_do_memcpy_err(struct dma_chan *chan,
+		union dmaengine_addr dest, unsigned int dest_off,
+		union dmaengine_addr src, unsigned int src_off,
+                size_t len, unsigned long flags)
+{
+	return -ENXIO;
+}

Further illustration of how this API growth is going wrong. You should create an API such that it is impossible for an XOR transform to ever call non-XOR-transform hooks.

Jeff

#31 lore raw

PATCH 13/19 dmaengine: add support for dma xor zero sum operations

In reply to Dan Williams

From: Dan Williams <[email protected]>

---

 drivers/dma/dmaengine.c   |   15 ++++++++++++
 drivers/dma/ioatdma.c     |    6 +++++
 include/linux/dmaengine.h |   56 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 77 insertions(+), 0 deletions(-)
drivers/dma/dmaengine.c +15 -0
index 33ad690..190c612 100644
@@ -617,6 +617,18 @@ dma_cookie_t dma_async_do_xor_err(struct
 }
 
 /**
+ * dma_async_do_zero_sum_err - default function for dma devices that
+ *	do not support xor zero sum
+ */
+dma_cookie_t dma_async_do_zero_sum_err(struct dma_chan *chan,
+		union dmaengine_addr src, unsigned int src_cnt,
+		unsigned int src_off, size_t len, u32 *result,
+		unsigned long flags)
+{
+	return -ENXIO;
+}
+
+/**
  * dma_async_do_memset_err - default function for dma devices that
  *      do not support memset
  */
@@ -649,6 +661,8 @@ EXPORT_SYMBOL_GPL(dma_async_memset_page)
 EXPORT_SYMBOL_GPL(dma_async_memset_dma);
 EXPORT_SYMBOL_GPL(dma_async_xor_pgs_to_pg);
 EXPORT_SYMBOL_GPL(dma_async_xor_dma_list_to_dma);
+EXPORT_SYMBOL_GPL(dma_async_zero_sum_pgs);
+EXPORT_SYMBOL_GPL(dma_async_zero_sum_dma_list);
 EXPORT_SYMBOL_GPL(dma_async_operation_complete);
 EXPORT_SYMBOL_GPL(dma_async_issue_pending);
 EXPORT_SYMBOL_GPL(dma_async_device_register);
@@ -656,6 +670,7 @@ EXPORT_SYMBOL_GPL(dma_async_device_unreg
 EXPORT_SYMBOL_GPL(dma_chan_cleanup);
 EXPORT_SYMBOL_GPL(dma_async_do_memcpy_err);
 EXPORT_SYMBOL_GPL(dma_async_do_xor_err);
+EXPORT_SYMBOL_GPL(dma_async_do_zero_sum_err);
 EXPORT_SYMBOL_GPL(dma_async_do_memset_err);
 EXPORT_SYMBOL_GPL(dma_async_chan_init);
 EXPORT_SYMBOL_GPL(dma_async_map_page);
drivers/dma/ioatdma.c +6 -0
index 231247c..4e90b02 100644
@@ -637,6 +637,11 @@ extern dma_cookie_t dma_async_do_xor_err
 	union dmaengine_addr src, unsigned int src_cnt,
 	unsigned int src_off, size_t len, unsigned long flags);
 
+extern dma_cookie_t dma_async_do_zero_sum_err(struct dma_chan *chan,
+        union dmaengine_addr src, unsigned int src_cnt,
+        unsigned int src_off, size_t len, u32 *result, 
+	unsigned long flags);
+
 extern dma_cookie_t dma_async_do_memset_err(struct dma_chan *chan,
 	union dmaengine_addr dest, unsigned int dest_off,
 	int val, size_t size, unsigned long flags);
@@ -752,6 +757,7 @@ #endif
 	device->common.capabilities = DMA_MEMCPY;
 	device->common.device_do_dma_memcpy = do_ioat_dma_memcpy;
 	device->common.device_do_dma_xor = dma_async_do_xor_err;
+	device->common.device_do_dma_zero_sum = dma_async_do_zero_sum_err;
 	device->common.device_do_dma_memset = dma_async_do_memset_err;
 	device->common.map_page = ioat_map_page;
 	device->common.map_single = ioat_map_single;
include/linux/dmaengine.h +56 -0

74 lines not on this page · raw message

#32 lore raw

PATCH 14/19 dmaengine: add dma_sync_wait

In reply to Dan Williams

From: Dan Williams <[email protected]>

dma_sync_wait is a common routine to live wait for a dma operation to complete.

---

 include/linux/dmaengine.h |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)
include/linux/dmaengine.h +12 -0
index 9fd6cbd..0a70c9e 100644
@@ -750,6 +750,18 @@ static inline void dma_async_unmap_singl
 	chan->device->unmap_single(chan, handle, size, direction);
 }
 
+static inline enum dma_status dma_sync_wait(struct dma_chan *chan,
+						dma_cookie_t cookie)
+{
+	enum dma_status status;
+	dma_async_issue_pending(chan);
+	do {
+		status = dma_async_operation_complete(chan, cookie, NULL, NULL);
+	} while (status == DMA_IN_PROGRESS);
+
+	return status;
+}
+
 /* --- DMA device --- */
 
 int dma_async_device_register(struct dma_device *device);
#33 lore raw

Re: [PATCH 14/19] dmaengine: add dma_sync_wait

Quoted 19 lines · Dan Williams

From: Dan Williams <[email protected]>

dma_sync_wait is a common routine to live wait for a dma operation to complete.

---

 include/linux/dmaengine.h |   12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)
include/linux/dmaengine.h +0 -0
index 9fd6cbd..0a70c9e 100644
@@ -750,6 +750,18 @@ static inline void dma_async_unmap_singl
 	chan->device->unmap_single(chan, handle, size, direction);
 }
 

include/linux/dmaengine.h static inline void dma_async_unmap_singl · line 762

Quoted 10 lines
include/linux/dmaengine.h +10 -0
+static inline enum dma_status dma_sync_wait(struct dma_chan *chan,
+						dma_cookie_t cookie)
+{
+	enum dma_status status;
+	dma_async_issue_pending(chan);
+	do {
+		status = dma_async_operation_complete(chan, cookie, NULL, NULL);
+	} while (status == DMA_IN_PROGRESS);
+
+	return status;

Where are the timeouts, etc.? Looks like an infinite loop to me, in the worst case.

Jeff

#34 lore raw

PATCH 15/19 dmaengine: raid5 dma client

In reply to Dan Williams

From: Dan Williams <[email protected]>

Adds a dmaengine client that is the hardware accelerated version of raid5_do_soft_block_ops. It utilizes the raid5 workqueue implementation to operate on multiple stripes simultaneously. See the iop-adma.c driver for an example of a driver that enables hardware accelerated raid5.

Changelog:
* mark operations as _Dma rather than _Done until all outstanding operations have completed. Once all operations have completed update the state and return it to the handle list
* add a helper routine to retrieve the last used cookie
* use dma_async_zero_sum_dma_list for checking parity which optionally allows parity check operations to not dirty the parity block in the cache (if 'disks' is less than 'MAX_ADMA_XOR_SOURCES')
* remove dependencies on iop13xx
* take into account the fact that dma engines have a staging buffer so we can perform 1 less block operation compared to software xor
* added __arch_raid5_dma_chan_request __arch_raid5_dma_next_channel and __arch_raid5_dma_check_channel to make the driver architecture independent
* added channel switching capability for architectures that implement different operations (i.e. copy & xor) on individual channels
* added initial support for "non-blocking" channel switching

---

 drivers/dma/Kconfig        |    9 +
 drivers/dma/Makefile       |    1 
 drivers/dma/raid5-dma.c    |  730 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/md/Kconfig         |   11 +
 drivers/md/raid5.c         |   66 ++++
 include/linux/dmaengine.h  |    5 
 include/linux/raid/raid5.h |   24 +
 7 files changed, 839 insertions(+), 7 deletions(-)
drivers/dma/Kconfig +9 -0
index 30d021d..fced8c3 100644
@@ -22,6 +22,15 @@ config NET_DMA
 	  Since this is the main user of the DMA engine, it should be enabled;
 	  say Y here.
 
+config RAID5_DMA
+        tristate "MD raid5: block operations offload"
+	depends on INTEL_IOP_ADMA && MD_RAID456
+	default y
+	---help---
+	  This enables the use of DMA engines in the MD-RAID5 driver to
+	  offload stripe cache operations, freeing CPU cycles.
+	  say Y here
+
 comment "DMA Devices"
 
 config INTEL_IOATDMA
drivers/dma/Makefile +1 -0
index bdcfdbd..4e36d6e 100644
@@ -1,3 +1,4 @@
 obj-$(CONFIG_DMA_ENGINE) += dmaengine.o
 obj-$(CONFIG_NET_DMA) += iovlock.o
+obj-$(CONFIG_RAID5_DMA) += raid5-dma.o
 obj-$(CONFIG_INTEL_IOATDMA) += ioatdma.o
drivers/dma/raid5-dma.c new file +730 -0

730 lines not on this page · raw message

drivers/md/Kconfig +11 -0
index 2a16b3b..dbd3ddc 100644
@@ -183,6 +183,17 @@ config MD_RAID456_WORKQUEUE_MULTITHREAD
 
 	  If unsure say, Y.
 
+config MD_RAID5_HW_OFFLOAD
+	depends on MD_RAID456 && RAID5_DMA
+	bool "Execute raid5 xor/copy operations with hardware engines"
+	default y
+	---help---
+	  On platforms with the requisite hardware capabilities MD
+	  can offload  RAID5 stripe cache operations (i.e. parity
+	  maintenance and bio buffer copies)
+
+	  If unsure say, Y.
+
 config MD_MULTIPATH
 	tristate "Multipath I/O support"
 	depends on BLK_DEV_MD
drivers/md/raid5.c +60 -6

127 lines not on this page · raw message

include/linux/dmaengine.h +5 -0
index 0a70c9e..7fd5aaf 100644
@@ -762,6 +762,11 @@ static inline enum dma_status dma_sync_w
 	return status;
 }
 
+static inline dma_cookie_t dma_async_get_last_cookie(struct dma_chan *chan)
+{
+	return chan->cookie;
+}
+
 /* --- DMA device --- */
 
 int dma_async_device_register(struct dma_device *device);
include/linux/raid/raid5.h +23 -1
index 31ae55c..f5b021d 100644
@@ -4,6 +4,9 @@ #define _RAID5_H
 #include <linux/raid/md.h>
 #include <linux/raid/xor.h>
 #include <linux/workqueue.h>
+#ifdef CONFIG_MD_RAID5_HW_OFFLOAD
+#include <linux/dmaengine.h>
+#endif
 
 /*
  *
@@ -169,16 +172,28 @@ struct stripe_head {
 		#ifdef CONFIG_MD_RAID456_WORKQUEUE
 		struct work_struct	work;		/* work queue descriptor */
 		#endif
+		#ifdef CONFIG_MD_RAID5_HW_OFFLOAD
+		u32 			dma_result;	/* storage for dma engine zero sum results */
+		dma_cookie_t		dma_cookie;	/* last issued dma operation */
+		struct dma_chan		*dma_chan;	/* dma channel for ops offload */
+		#ifdef CONFIG_RAID5_DMA_WAIT_VIA_REQUEUE
+		void			*ops_bookmark;	/* place holder for requeued stripes */
+		#endif /* CONFIG_RAID5_DMA_WAIT_VIA_REQUEUE */
+		#endif /* CONFIG_MD_RAID5_HW_OFFLOAD */
 	} ops;
 	struct r5dev {
 		struct bio	req;
 		struct bio_vec	vec;
 		struct page	*page;
+		#ifdef CONFIG_MD_RAID5_HW_OFFLOAD
+		dma_addr_t	dma;
+		#endif
 		struct bio	*toread, *read, *towrite, *written;
 		sector_t	sector;			/* sector of this page */
 		unsigned long	flags;
 	} dev[1]; /* allocated with extra space depending of RAID geometry */
 };
+
 /* Flags */
 #define	R5_UPTODATE	0	/* page contains current data */
 #define	R5_LOCKED	1	/* IO has been submitted on "req" */
@@ -190,7 +205,6 @@ #define	R5_Wantwrite	5
 #define	R5_Overlap	7	/* There is a pending overlapping request on this block */
 #define	R5_ReadError	8	/* seen a read error here recently */
 #define	R5_ReWrite	9	/* have tried to over-write the readerror */
-
 #define	R5_Expanded	10	/* This block now has post-expand data */
 #define	R5_Consistent	11	/* Block is HW DMA-able without a cache flush */
 #define	R5_ComputeReq	12	/* compute_block in progress treat as uptodate */
@@ -373,6 +387,14 @@ struct raid5_private_data {
 	int			pool_size; /* number of disks in stripeheads in pool */
 	spinlock_t		device_lock;
 	struct disk_info	*disks;
+#ifdef CONFIG_MD_RAID5_HW_OFFLOAD
+	struct raid5_dma {
+		struct module *owner;
+		void (*do_block_ops)(void *stripe_ref);
+		struct dma_chan * (*channel_iterate)(void);
+	} dma;
+#endif
+
 };
 
 typedef struct raid5_private_data raid5_conf_t;
#35 lore raw

Re: [PATCH 15/19] dmaengine: raid5 dma client

Quoted 249 lines · Dan Williams

From: Dan Williams <[email protected]>

Adds a dmaengine client that is the hardware accelerated version of raid5_do_soft_block_ops. It utilizes the raid5 workqueue implementation to operate on multiple stripes simultaneously. See the iop-adma.c driver for an example of a driver that enables hardware accelerated raid5.

Changelog:
* mark operations as _Dma rather than _Done until all outstanding operations have completed. Once all operations have completed update the state and return it to the handle list
* add a helper routine to retrieve the last used cookie
* use dma_async_zero_sum_dma_list for checking parity which optionally allows parity check operations to not dirty the parity block in the cache (if 'disks' is less than 'MAX_ADMA_XOR_SOURCES')
* remove dependencies on iop13xx
* take into account the fact that dma engines have a staging buffer so we can perform 1 less block operation compared to software xor
* added __arch_raid5_dma_chan_request __arch_raid5_dma_next_channel and __arch_raid5_dma_check_channel to make the driver architecture independent
* added channel switching capability for architectures that implement different operations (i.e. copy & xor) on individual channels
* added initial support for "non-blocking" channel switching

---

 drivers/dma/Kconfig        |    9 +
 drivers/dma/Makefile       |    1 
 drivers/dma/raid5-dma.c    |  730 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/md/Kconfig         |   11 +
 drivers/md/raid5.c         |   66 ++++
 include/linux/dmaengine.h  |    5 
 include/linux/raid/raid5.h |   24 +
 7 files changed, 839 insertions(+), 7 deletions(-)
drivers/dma/Kconfig +9 -0
index 30d021d..fced8c3 100644
@@ -22,6 +22,15 @@ config NET_DMA
 	  Since this is the main user of the DMA engine, it should be enabled;
 	  say Y here.
 
+config RAID5_DMA
+        tristate "MD raid5: block operations offload"
+	depends on INTEL_IOP_ADMA && MD_RAID456
+	default y
+	---help---
+	  This enables the use of DMA engines in the MD-RAID5 driver to
+	  offload stripe cache operations, freeing CPU cycles.
+	  say Y here
+
 comment "DMA Devices"
 
 config INTEL_IOATDMA
drivers/dma/Makefile +1 -0
index bdcfdbd..4e36d6e 100644
@@ -1,3 +1,4 @@
 obj-$(CONFIG_DMA_ENGINE) += dmaengine.o
 obj-$(CONFIG_NET_DMA) += iovlock.o
+obj-$(CONFIG_RAID5_DMA) += raid5-dma.o
 obj-$(CONFIG_INTEL_IOATDMA) += ioatdma.o
drivers/dma/raid5-dma.c new file +178 -0
new file mode 100644
index 0000000..04a1790
@@ -0,0 +1,730 @@
+/*
+ * Offload raid5 operations to hardware RAID engines
+ * Copyright(c) 2006 Intel Corporation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59
+ * Temple Place - Suite 330, Boston, MA  02111-1307, USA.
+ *
+ * The full GNU General Public License is included in this distribution in the
+ * file called COPYING.
+ */
+
+#include <linux/raid/raid5.h>
+#include <linux/dmaengine.h>
+
+static struct dma_client *raid5_dma_client;
+static atomic_t raid5_count;
+extern void release_stripe(struct stripe_head *sh);
+extern void __arch_raid5_dma_chan_request(struct dma_client *client);
+extern struct dma_chan *__arch_raid5_dma_next_channel(struct dma_client *client);
+
+#define MAX_HW_XOR_SRCS 16
+
+#ifndef STRIPE_SIZE
+#define STRIPE_SIZE PAGE_SIZE
+#endif
+
+#ifndef STRIPE_SECTORS
+#define STRIPE_SECTORS		(STRIPE_SIZE>>9)
+#endif
+
+#ifndef r5_next_bio
+#define r5_next_bio(bio, sect) ( ( (bio)->bi_sector + ((bio)->bi_size>>9) < sect + STRIPE_SECTORS) ? (bio)->bi_next : NULL)
+#endif
+
+#define DMA_RAID5_DEBUG 0
+#define PRINTK(x...) ((void)(DMA_RAID5_DEBUG && printk(x)))
+
+/*
+ * Copy data between a page in the stripe cache, and one or more bion
+ * The page could align with the middle of the bio, or there could be
+ * several bion, each with several bio_vecs, which cover part of the page
+ * Multiple bion are linked together on bi_next.  There may be extras
+ * at the end of this list.  We ignore them.
+ */
+static dma_cookie_t dma_raid_copy_data(int frombio, struct bio *bio,
+		     dma_addr_t dma, sector_t sector, struct dma_chan *chan,
+		     dma_cookie_t cookie)
+{
+	struct bio_vec *bvl;
+	struct page *bio_page;
+	int i;
+	int dma_offset;
+	dma_cookie_t last_cookie = cookie;
+
+	if (bio->bi_sector >= sector)
+		dma_offset = (signed)(bio->bi_sector - sector) * 512;
+	else
+		dma_offset = (signed)(sector - bio->bi_sector) * -512;
+	bio_for_each_segment(bvl, bio, i) {
+		int len = bio_iovec_idx(bio,i)->bv_len;
+		int clen;
+		int b_offset = 0;
+
+		if (dma_offset < 0) {
+			b_offset = -dma_offset;
+			dma_offset += b_offset;
+			len -= b_offset;
+		}
+
+		if (len > 0 && dma_offset + len > STRIPE_SIZE)
+			clen = STRIPE_SIZE - dma_offset;
+		else clen = len;
+
+		if (clen > 0) {
+			b_offset += bio_iovec_idx(bio,i)->bv_offset;
+			bio_page = bio_iovec_idx(bio,i)->bv_page;
+			if (frombio)
+				do {
+					cookie = dma_async_memcpy_pg_to_dma(chan,
+								dma + dma_offset,
+								bio_page,
+								b_offset,
+								clen);
+					if (cookie == -ENOMEM)
+						dma_sync_wait(chan, last_cookie);
+					else
+						WARN_ON(cookie <= 0);
+				} while (cookie == -ENOMEM);
+			else
+				do {
+					cookie = dma_async_memcpy_dma_to_pg(chan,
+								bio_page,
+								b_offset,
+								dma + dma_offset,
+								clen);
+					if (cookie == -ENOMEM)
+						dma_sync_wait(chan, last_cookie);
+					else
+						WARN_ON(cookie <= 0);
+				} while (cookie == -ENOMEM);
+		}
+		last_cookie = cookie;
+		if (clen < len) /* hit end of page */
+			break;
+		dma_offset +=  len;
+	}
+
+	return last_cookie;
+}
+
+#define issue_xor() do {					          \
+			 do {					          \
+			 	cookie = dma_async_xor_dma_list_to_dma(   \
+			 		sh->ops.dma_chan,	          \
+			 		xor_destination_addr,	          \
+			 		dma,			          \
+			 		count,			          \
+			 		STRIPE_SIZE);		          \
+			 	if (cookie == -ENOMEM)		          \
+			 		dma_sync_wait(sh->ops.dma_chan,	  \
+			 			sh->ops.dma_cookie);      \
+			 	else				          \
+			 		WARN_ON(cookie <= 0);	          \
+			 } while (cookie == -ENOMEM);		          \
+			 sh->ops.dma_cookie = cookie;		          \
+			 dma[0] = xor_destination_addr;			  \
+			 count = 1;					  \
+			} while(0)
+#define check_xor() do {						\
+			if (count == MAX_HW_XOR_SRCS)			\
+				issue_xor();				\
+		     } while (0)
+
+#ifdef CONFIG_RAID5_DMA_ARCH_NEEDS_CHAN_SWITCH
+extern struct dma_chan *__arch_raid5_dma_check_channel(struct dma_chan *chan,
+						dma_cookie_t cookie,
+						struct dma_client *client,
+						unsigned long capabilities);
+
+#ifdef CONFIG_RAID5_DMA_WAIT_VIA_REQUEUE
+#define check_channel(cap, bookmark) do {			     \
+bookmark:							     \
+	next_chan = __arch_raid5_dma_check_channel(sh->ops.dma_chan, \
+						sh->ops.dma_cookie,  \
+						raid5_dma_client,    \
+						(cap));		     \
+	if (!next_chan) {					     \
+		BUG_ON(sh->ops.ops_bookmark);			     \
+		sh->ops.ops_bookmark = &&bookmark;		     \
+		goto raid5_dma_retry;				     \
+	} else {						     \
+		sh->ops.dma_chan = next_chan;			     \
+		sh->ops.dma_cookie = dma_async_get_last_cookie(	     \
+							next_chan);  \
+		sh->ops.ops_bookmark = NULL;			     \
+	}							     \
+} while (0)
+#else
+#define check_channel(cap, bookmark) do {			     \
+bookmark:							     \
+	next_chan = __arch_raid5_dma_check_channel(sh->ops.dma_chan, \
+						sh->ops.dma_cookie,  \
+						raid5_dma_client,    \
+						(cap));		     \
+	if (!next_chan) {					     \
+		dma_sync_wait(sh->ops.dma_chan, sh->ops.dma_cookie); \
+		goto bookmark;					     \

drivers/dma/raid5-dma.c line 188

Quoted 10 lines
drivers/dma/raid5-dma.c new file +10 -0
+	} else {						     \
+		sh->ops.dma_chan = next_chan;			     \
+		sh->ops.dma_cookie = dma_async_get_last_cookie(	     \
+							next_chan);  \
+	}							     \
+} while (0)
+#endif /* CONFIG_RAID5_DMA_WAIT_VIA_REQUEUE */
+#else
+#define check_channel(cap, bookmark) do { } while (0)
+#endif /* CONFIG_RAID5_DMA_ARCH_NEEDS_CHAN_SWITCH */

The above seems a bit questionable and overengineered.

Linux mantra: Do What You Must, And No More.

In this case, just code and note that it's IOP-specific. Don't bother to support cases that doesn't exist yet.

Quoted 8 lines
drivers/dma/raid5-dma.c new file +8 -0
+ * dma_do_raid5_block_ops - perform block memory operations on stripe data
+ * outside the spin lock with dma engines
+ *
+ * A note about the need for __arch_raid5_dma_check_channel:
+ * This function is only needed to support architectures where a single raid
+ * operation spans multiple hardware channels.  For example on a reconstruct
+ * write, memory copy operations are submitted to a memcpy channel and then
+ * the routine must switch to the xor channel to complete the raid operation.

drivers/dma/raid5-dma.c line 208

Quoted 10 lines
drivers/dma/raid5-dma.c new file +10 -0
+ * __arch_raid5_dma_check_channel makes sure the previous operation has
+ * completed before returning the new channel.
+ * Some efficiency can be gained by putting the stripe back on the work
+ * queue rather than spin waiting.  This code is a work in progress and is
+ * available via the 'broken' option CONFIG_RAID5_DMA_WAIT_VIA_REQUEUE.
+ * If 'wait via requeue' is not defined the check_channel macro live waits
+ * for the next channel.
+ */
+static void dma_do_raid5_block_ops(void *stripe_head_ref)
+{

Another way-too-big function that should be split up.

#36 lore raw

PATCH 16/19 dmaengine: Driver for the Intel IOP 32x, 33x, and 13xx RAID engines

In reply to Dan Williams

From: Dan Williams <[email protected]>

This is a driver for the iop DMA/AAU/ADMA units which are capable of pq_xor, pq_update, pq_zero_sum, xor, dual_xor, xor_zero_sum, fill, copy+crc, and copy operations.

Changelog:
* fixed a slot allocation bug in do_iop13xx_adma_xor that caused too few slots to be requested eventually leading to data corruption
* enabled the slot allocation routine to attempt to free slots before returning -ENOMEM
* switched the cleanup routine to solely use the software chain and the status register to determine if a descriptor is complete. This is necessary to support other IOP engines that do not have status writeback capability
* make the driver iop generic
* modified the allocation routines to understand allocating a group of slots for a single operation
* added a null xor initialization operation for the xor only channel on iop3xx
* add software emulation of zero sum on iop32x
* support xor operations on buffers larger than the hardware maximum
* add architecture specific raid5-dma support functions

---

 drivers/dma/Kconfig                 |   27 +
 drivers/dma/Makefile                |    1 
 drivers/dma/iop-adma.c              | 1501 +++++++++++++++++++++++++++++++++++
 include/asm-arm/hardware/iop_adma.h |   98 ++
 4 files changed, 1624 insertions(+), 3 deletions(-)
drivers/dma/Kconfig +24 -3
index fced8c3..3556143 100644
@@ -7,8 +7,8 @@ menu "DMA Engine support"
 config DMA_ENGINE
 	bool "Support for DMA engines"
 	---help---
-	  DMA engines offload copy operations from the CPU to dedicated
-	  hardware, allowing the copies to happen asynchronously.
+          DMA engines offload block memory operations from the CPU to dedicated
+          hardware, allowing the operations to happen asynchronously.
 
 comment "DMA Clients"
 
@@ -28,9 +28,19 @@ config RAID5_DMA
 	default y
 	---help---
 	  This enables the use of DMA engines in the MD-RAID5 driver to
-	  offload stripe cache operations, freeing CPU cycles.
+	  offload stripe cache operations (i.e. xor, memcpy), freeing CPU cycles.
 	  say Y here
 
+config RAID5_DMA_WAIT_VIA_REQUEUE
+	bool "raid5-dma: Non-blocking channel switching"
+	depends on RAID5_DMA_ARCH_NEEDS_CHAN_SWITCH && RAID5_DMA && BROKEN
+	default n
+	---help---
+	  This enables the raid5-dma driver to continue to operate on incoming
+	  stripes when it determines that the current stripe must wait for a
+	  a hardware channel to finish operations.  This code is a work in
+	  progress, only say Y to debug the implementation, otherwise say N.
+
 comment "DMA Devices"
 
 config INTEL_IOATDMA
@@ -40,4 +50,15 @@ config INTEL_IOATDMA
 	---help---
 	  Enable support for the Intel(R) I/OAT DMA engine.
 
+config INTEL_IOP_ADMA
+        tristate "Intel IOP ADMA support"
+        depends on DMA_ENGINE && (ARCH_IOP32X || ARCH_IOP33X || ARCH_IOP13XX)
+	select RAID5_DMA_ARCH_NEEDS_CHAN_SWITCH if (ARCH_IOP32X || ARCH_IOP33X)
+        default m
+        ---help---
+          Enable support for the Intel(R) IOP Series RAID engines.
+
+config RAID5_DMA_ARCH_NEEDS_CHAN_SWITCH
+	bool
+
 endmenu
drivers/dma/Makefile +1 -0
index 4e36d6e..233eae7 100644
@@ -2,3 +2,4 @@ obj-$(CONFIG_DMA_ENGINE) += dmaengine.o
 obj-$(CONFIG_NET_DMA) += iovlock.o
 obj-$(CONFIG_RAID5_DMA) += raid5-dma.o
 obj-$(CONFIG_INTEL_IOATDMA) += ioatdma.o
+obj-$(CONFIG_INTEL_IOP_ADMA) += iop-adma.o
drivers/dma/iop-adma.c new file +1501 -0

1,501 lines not on this page · raw message

include/asm-arm/hardware/iop_adma.h new file +98 -0

98 lines not on this page · raw message

#37 lore raw

Re: [PATCH 16/19] dmaengine: Driver for the Intel IOP 32x, 33x, and 13xx RAID engines

Hi,

Quoted 5 lines · On Mon, 11 Sep 2006 16:19:00 -0700 Dan Williams <[email protected]>…

From: Dan Williams <[email protected]>

This is a driver for the iop DMA/AAU/ADMA units which are capable of pq_xor, pq_update, pq_zero_sum, xor, dual_xor, xor_zero_sum, fill, copy+crc, and copy operations.

You implement a bunch of different functions here. I agree with Jeff's feedback related to the lack of scalability the way the API is going right now.

Another example of this is that the driver is doing it's own self-test of the functions. This means that every backend driver will need to duplicate this code. Wouldn't it be easier for everyone if the common infrastructure did a test call at the time of registration of a function instead, and return failure if it doesn't pass?

Quoted 4 lines
 drivers/dma/Kconfig                 |   27 +
 drivers/dma/Makefile                |    1 
 drivers/dma/iop-adma.c              | 1501 +++++++++++++++++++++++++++++++++++
 include/asm-arm/hardware/iop_adma.h |   98 ++

ioatdma.h is currently under drivers/dma/. If the contents is strictly device-related please add them under drivers/dma.

-Olof

#38 lore raw

PATCH 17/19 iop3xx: define IOP3XX_REG_ADDR[32|16|8] and clean up DMA/AAU defs

In reply to Dan Williams

From: Dan Williams <[email protected]>

Also brings the iop3xx registers in line with the format of the iop13xx register definitions.

---

 include/asm-arm/arch-iop32x/entry-macro.S |    2 
 include/asm-arm/arch-iop32x/iop32x.h      |   14 +
 include/asm-arm/arch-iop33x/entry-macro.S |    2 
 include/asm-arm/arch-iop33x/iop33x.h      |   38 ++-
 include/asm-arm/hardware/iop3xx.h         |  347 +++++++++++++----------------
 5 files changed, 188 insertions(+), 215 deletions(-)
include/asm-arm/arch-iop32x/entry-macro.S +1 -1
index 1500cbb..f357be4 100644
@@ -13,7 +13,7 @@ #include <asm/arch/iop32x.h>
 		.endm
 
 		.macro	get_irqnr_and_base, irqnr, irqstat, base, tmp
-		ldr	\base, =IOP3XX_REG_ADDR(0x07D8)
+		ldr	\base, =0xfeffe7d8
 		ldr	\irqstat, [\base]		@ Read IINTSRC
 		cmp	\irqstat, #0
 		clzne	\irqnr, \irqstat
include/asm-arm/arch-iop32x/iop32x.h +8 -6
index 15b4d6a..904a14d 100644
@@ -19,16 +19,18 @@ #define __IOP32X_H
  * Peripherals that are shared between the iop32x and iop33x but
  * located at different addresses.
  */
-#define IOP3XX_GPIO_REG(reg)	(IOP3XX_PERIPHERAL_VIRT_BASE + 0x07c0 + (reg))
-#define IOP3XX_TIMER_REG(reg)	(IOP3XX_PERIPHERAL_VIRT_BASE + 0x07e0 + (reg))
+#define IOP3XX_GPIO_REG32(reg)	 (volatile u32 *)(IOP3XX_PERIPHERAL_VIRT_BASE +\
+						  0x07c0 + (reg))
+#define IOP3XX_TIMER_REG32(reg) (volatile u32 *)(IOP3XX_PERIPHERAL_VIRT_BASE +\
+						  0x07e0 + (reg))
 
 #include <asm/hardware/iop3xx.h>
 
 /* Interrupt Controller  */
-#define IOP32X_INTCTL		(volatile u32 *)IOP3XX_REG_ADDR(0x07d0)
-#define IOP32X_INTSTR		(volatile u32 *)IOP3XX_REG_ADDR(0x07d4)
-#define IOP32X_IINTSRC		(volatile u32 *)IOP3XX_REG_ADDR(0x07d8)
-#define IOP32X_FINTSRC		(volatile u32 *)IOP3XX_REG_ADDR(0x07dc)
+#define IOP32X_INTCTL		IOP3XX_REG_ADDR32(0x07d0)
+#define IOP32X_INTSTR		IOP3XX_REG_ADDR32(0x07d4)
+#define IOP32X_IINTSRC		IOP3XX_REG_ADDR32(0x07d8)
+#define IOP32X_FINTSRC		IOP3XX_REG_ADDR32(0x07dc)
 
 
 #endif
include/asm-arm/arch-iop33x/entry-macro.S +1 -1
index 92b7917..eb207d2 100644
@@ -13,7 +13,7 @@ #include <asm/arch/iop33x.h>
 		.endm
 
 		.macro	get_irqnr_and_base, irqnr, irqstat, base, tmp
-		ldr	\base, =IOP3XX_REG_ADDR(0x07C8)
+		ldr	\base, =0xfeffe7c8
 		ldr	\irqstat, [\base]		@ Read IINTVEC
 		cmp	\irqstat, #0
 		ldreq	\irqstat, [\base]		@ erratum 63 workaround
include/asm-arm/arch-iop33x/iop33x.h +20 -18
index 9b38fde..c171383 100644
@@ -18,28 +18,30 @@ #define __IOP33X_H
  * Peripherals that are shared between the iop32x and iop33x but
  * located at different addresses.
  */
-#define IOP3XX_GPIO_REG(reg)	(IOP3XX_PERIPHERAL_VIRT_BASE + 0x1780 + (reg))
-#define IOP3XX_TIMER_REG(reg)	(IOP3XX_PERIPHERAL_VIRT_BASE + 0x07d0 + (reg))
+#define IOP3XX_GPIO_REG32(reg)	 (volatile u32 *)(IOP3XX_PERIPHERAL_VIRT_BASE +\
+						  0x1780 + (reg))
+#define IOP3XX_TIMER_REG32(reg) (volatile u32 *)(IOP3XX_PERIPHERAL_VIRT_BASE +\
+						  0x07d0 + (reg))
 
 #include <asm/hardware/iop3xx.h>
 
 /* Interrupt Controller  */
-#define IOP33X_INTCTL0		(volatile u32 *)IOP3XX_REG_ADDR(0x0790)
-#define IOP33X_INTCTL1		(volatile u32 *)IOP3XX_REG_ADDR(0x0794)
-#define IOP33X_INTSTR0		(volatile u32 *)IOP3XX_REG_ADDR(0x0798)
-#define IOP33X_INTSTR1		(volatile u32 *)IOP3XX_REG_ADDR(0x079c)
-#define IOP33X_IINTSRC0		(volatile u32 *)IOP3XX_REG_ADDR(0x07a0)
-#define IOP33X_IINTSRC1		(volatile u32 *)IOP3XX_REG_ADDR(0x07a4)
-#define IOP33X_FINTSRC0		(volatile u32 *)IOP3XX_REG_ADDR(0x07a8)
-#define IOP33X_FINTSRC1		(volatile u32 *)IOP3XX_REG_ADDR(0x07ac)
-#define IOP33X_IPR0		(volatile u32 *)IOP3XX_REG_ADDR(0x07b0)
-#define IOP33X_IPR1		(volatile u32 *)IOP3XX_REG_ADDR(0x07b4)
-#define IOP33X_IPR2		(volatile u32 *)IOP3XX_REG_ADDR(0x07b8)
-#define IOP33X_IPR3		(volatile u32 *)IOP3XX_REG_ADDR(0x07bc)
-#define IOP33X_INTBASE		(volatile u32 *)IOP3XX_REG_ADDR(0x07c0)
-#define IOP33X_INTSIZE		(volatile u32 *)IOP3XX_REG_ADDR(0x07c4)
-#define IOP33X_IINTVEC		(volatile u32 *)IOP3XX_REG_ADDR(0x07c8)
-#define IOP33X_FINTVEC		(volatile u32 *)IOP3XX_REG_ADDR(0x07cc)
+#define IOP33X_INTCTL0		IOP3XX_REG_ADDR32(0x0790)
+#define IOP33X_INTCTL1		IOP3XX_REG_ADDR32(0x0794)
+#define IOP33X_INTSTR0		IOP3XX_REG_ADDR32(0x0798)
+#define IOP33X_INTSTR1		IOP3XX_REG_ADDR32(0x079c)
+#define IOP33X_IINTSRC0	IOP3XX_REG_ADDR32(0x07a0)
+#define IOP33X_IINTSRC1	IOP3XX_REG_ADDR32(0x07a4)
+#define IOP33X_FINTSRC0	IOP3XX_REG_ADDR32(0x07a8)
+#define IOP33X_FINTSRC1	IOP3XX_REG_ADDR32(0x07ac)
+#define IOP33X_IPR0		IOP3XX_REG_ADDR32(0x07b0)
+#define IOP33X_IPR1		IOP3XX_REG_ADDR32(0x07b4)
+#define IOP33X_IPR2		IOP3XX_REG_ADDR32(0x07b8)
+#define IOP33X_IPR3		IOP3XX_REG_ADDR32(0x07bc)
+#define IOP33X_INTBASE		IOP3XX_REG_ADDR32(0x07c0)
+#define IOP33X_INTSIZE		IOP3XX_REG_ADDR32(0x07c4)
+#define IOP33X_IINTVEC		IOP3XX_REG_ADDR32(0x07c8)
+#define IOP33X_FINTVEC		IOP3XX_REG_ADDR32(0x07cc)
 
 /* UARTs  */
 #define IOP33X_UART0_PHYS	(IOP3XX_PERIPHERAL_PHYS_BASE + 0x1700)
include/asm-arm/hardware/iop3xx.h +158 -189

380 lines not on this page · raw message

#39 lore raw

Re: [PATCH 17/19] iop3xx: define IOP3XX_REG_ADDR[32|16|8] and clean up DMA/AAU defs

Quoted 6 lines · Dan Williams

From: Dan Williams <[email protected]>

Also brings the iop3xx registers in line with the format of the iop13xx register definitions.

Quoted 8 lines

---

 include/asm-arm/arch-iop32x/entry-macro.S |    2 
 include/asm-arm/arch-iop32x/iop32x.h      |   14 +
 include/asm-arm/arch-iop33x/entry-macro.S |    2 
 include/asm-arm/arch-iop33x/iop33x.h      |   38 ++-
 include/asm-arm/hardware/iop3xx.h         |  347 +++++++++++++----------------
 5 files changed, 188 insertions(+), 215 deletions(-)

Another Linux mantra: "volatile" == hiding a bug. Avoid, please.

Jeff

#40 lore raw

PATCH 18/19 iop3xx: Give Linux control over PCI (ATU) initialization

In reply to Dan Williams

From: Dan Williams <[email protected]>

Currently the iop3xx platform support code assumes that RedBoot is the bootloader and has already initialized the ATU. Linux should handle this initialization for three reasons:

1/ The memory map that RedBoot sets up is not optimal (page_to_dma and virt_to_phys return different addresses). The effect of this is that using the dma mapping API for the internal bus dma units generates pci bus addresses that are incorrect for the internal bus.

2/ Not all iop platforms use RedBoot

3/ If the ATU is already initialized it indicates that the iop is an add-in card in another host, it does not own the PCI bus, and should not be re-initialized.

---

 arch/arm/mach-iop32x/Kconfig         |    8 ++
 arch/arm/mach-iop32x/ep80219.c       |    4 +
 arch/arm/mach-iop32x/iq31244.c       |    5 +
 arch/arm/mach-iop32x/iq80321.c       |    5 +
 arch/arm/mach-iop33x/Kconfig         |    8 ++
 arch/arm/mach-iop33x/iq80331.c       |    5 +
 arch/arm/mach-iop33x/iq80332.c       |    4 +
 arch/arm/plat-iop/pci.c              |  140 ++++++++++++++++++++++++++++++++++
 include/asm-arm/arch-iop32x/iop32x.h |    9 ++
 include/asm-arm/arch-iop32x/memory.h |    4 -
 include/asm-arm/arch-iop33x/iop33x.h |   10 ++
 include/asm-arm/arch-iop33x/memory.h |    4 -
 include/asm-arm/hardware/iop3xx.h    |   20 ++++-
 13 files changed, 214 insertions(+), 12 deletions(-)
arch/arm/mach-iop32x/Kconfig +8 -0
index 05549a5..b2788e3 100644
@@ -22,6 +22,14 @@ config ARCH_IQ80321
 	  Say Y here if you want to run your kernel on the Intel IQ80321
 	  evaluation kit for the IOP321 processor.
 
+config IOP3XX_ATU
+        bool "Enable the PCI Controller"
+        default y
+        help
+          Say Y here if you want the IOP to initialize its PCI Controller.
+          Say N if the IOP is an add in card, the host system owns the PCI
+          bus in this case.
+
 endmenu
 
 endif
arch/arm/mach-iop32x/ep80219.c +3 -1
index f616d3e..1a5c586 100644
@@ -100,7 +100,7 @@ ep80219_pci_map_irq(struct pci_dev *dev,
 
 static struct hw_pci ep80219_pci __initdata = {
 	.swizzle	= pci_std_swizzle,
-	.nr_controllers = 1,
+	.nr_controllers = 0,
 	.setup		= iop3xx_pci_setup,
 	.preinit	= iop3xx_pci_preinit,
 	.scan		= iop3xx_pci_scan_bus,
@@ -109,6 +109,8 @@ static struct hw_pci ep80219_pci __initd
 
 static int __init ep80219_pci_init(void)
 {
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		ep80219_pci.nr_controllers = 1;
 #if 0
 	if (machine_is_ep80219())
 		pci_common_init(&ep80219_pci);
arch/arm/mach-iop32x/iq31244.c +4 -1
index 967a696..25d5d62 100644
@@ -97,7 +97,7 @@ iq31244_pci_map_irq(struct pci_dev *dev,
 
 static struct hw_pci iq31244_pci __initdata = {
 	.swizzle	= pci_std_swizzle,
-	.nr_controllers = 1,
+	.nr_controllers = 0,
 	.setup		= iop3xx_pci_setup,
 	.preinit	= iop3xx_pci_preinit,
 	.scan		= iop3xx_pci_scan_bus,
@@ -106,6 +106,9 @@ static struct hw_pci iq31244_pci __initd
 
 static int __init iq31244_pci_init(void)
 {
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		iq31244_pci.nr_controllers = 1;
+
 	if (machine_is_iq31244())
 		pci_common_init(&iq31244_pci);
 
arch/arm/mach-iop32x/iq80321.c +4 -1
index ef4388c..cdd2265 100644
@@ -97,7 +97,7 @@ iq80321_pci_map_irq(struct pci_dev *dev,
 
 static struct hw_pci iq80321_pci __initdata = {
 	.swizzle	= pci_std_swizzle,
-	.nr_controllers = 1,
+	.nr_controllers = 0,
 	.setup		= iop3xx_pci_setup,
 	.preinit	= iop3xx_pci_preinit,
 	.scan		= iop3xx_pci_scan_bus,
@@ -106,6 +106,9 @@ static struct hw_pci iq80321_pci __initd
 
 static int __init iq80321_pci_init(void)
 {
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		iq80321_pci.nr_controllers = 1;
+
 	if (machine_is_iq80321())
 		pci_common_init(&iq80321_pci);
 
arch/arm/mach-iop33x/Kconfig +8 -0
index 9aa016b..45598e0 100644
@@ -16,6 +16,14 @@ config MACH_IQ80332
 	  Say Y here if you want to run your kernel on the Intel IQ80332
 	  evaluation kit for the IOP332 chipset.
 
+config IOP3XX_ATU
+	bool "Enable the PCI Controller"
+	default y
+	help
+	  Say Y here if you want the IOP to initialize its PCI Controller.
+	  Say N if the IOP is an add in card, the host system owns the PCI
+	  bus in this case.
+
 endmenu
 
 endif
arch/arm/mach-iop33x/iq80331.c +4 -1
index 7714c94..3807000 100644
@@ -78,7 +78,7 @@ iq80331_pci_map_irq(struct pci_dev *dev,
 
 static struct hw_pci iq80331_pci __initdata = {
 	.swizzle	= pci_std_swizzle,
-	.nr_controllers = 1,
+	.nr_controllers = 0,
 	.setup		= iop3xx_pci_setup,
 	.preinit	= iop3xx_pci_preinit,
 	.scan		= iop3xx_pci_scan_bus,
@@ -87,6 +87,9 @@ static struct hw_pci iq80331_pci __initd
 
 static int __init iq80331_pci_init(void)
 {
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		iq80331_pci.nr_controllers = 1;
+
 	if (machine_is_iq80331())
 		pci_common_init(&iq80331_pci);
 
arch/arm/mach-iop33x/iq80332.c +4 -0
index a3fa7f8..8780d55 100644
@@ -93,6 +93,10 @@ static struct hw_pci iq80332_pci __initd
 
 static int __init iq80332_pci_init(void)
 {
+
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		iq80332_pci.nr_controllers = 1;
+
 	if (machine_is_iq80332())
 		pci_common_init(&iq80332_pci);
 
arch/arm/plat-iop/pci.c +139 -1

157 lines not on this page · raw message

include/asm-arm/arch-iop32x/iop32x.h +9 -0
index 904a14d..93209c7 100644
@@ -32,5 +32,14 @@ #define IOP32X_INTSTR		IOP3XX_REG_ADDR32
 #define IOP32X_IINTSRC		IOP3XX_REG_ADDR32(0x07d8)
 #define IOP32X_FINTSRC		IOP3XX_REG_ADDR32(0x07dc)
 
+/* ATU Parameters
+ * set up a 1:1 bus to physical ram relationship
+ * w/ physical ram on top of pci in the memory map
+ */
+#define IOP32X_MAX_RAM_SIZE            0x40000000UL
+#define IOP3XX_MAX_RAM_SIZE            IOP32X_MAX_RAM_SIZE
+#define IOP3XX_PCI_LOWER_MEM_BA        0x80000000
+#define IOP32X_PCI_MEM_WINDOW_SIZE     0x04000000
+#define IOP3XX_PCI_MEM_WINDOW_SIZE     IOP32X_PCI_MEM_WINDOW_SIZE
 
 #endif
include/asm-arm/arch-iop32x/memory.h +2 -2
index 764cd3f..c51072a 100644
@@ -19,8 +19,8 @@ #define PHYS_OFFSET	UL(0xa0000000)
  * bus_to_virt: Used to convert an address for DMA operations
  *		to an address that the kernel can use.
  */
-#define __virt_to_bus(x)	(((__virt_to_phys(x)) & ~(*IOP3XX_IATVR2)) | ((*IOP3XX_IABAR2) & 0xfffffff0))
-#define __bus_to_virt(x)	(__phys_to_virt(((x) & ~(*IOP3XX_IALR2)) | ( *IOP3XX_IATVR2)))
+#define __virt_to_bus(x)	(__virt_to_phys(x))
+#define __bus_to_virt(x)	(__phys_to_virt(x))
 
 
 #endif
include/asm-arm/arch-iop33x/iop33x.h +10 -0
index c171383..e106b80 100644
@@ -49,5 +49,15 @@ #define IOP33X_UART0_VIRT	(IOP3XX_PERIPH
 #define IOP33X_UART1_PHYS	(IOP3XX_PERIPHERAL_PHYS_BASE + 0x1740)
 #define IOP33X_UART1_VIRT	(IOP3XX_PERIPHERAL_VIRT_BASE + 0x1740)
 
+/* ATU Parameters
+ * set up a 1:1 bus to physical ram relationship
+ * w/ pci on top of physical ram in memory map
+ */
+#define IOP33X_MAX_RAM_SIZE		0x80000000UL
+#define IOP3XX_MAX_RAM_SIZE		IOP33X_MAX_RAM_SIZE
+#define IOP3XX_PCI_LOWER_MEM_BA	(PHYS_OFFSET + IOP33X_MAX_RAM_SIZE)
+#define IOP33X_PCI_MEM_WINDOW_SIZE	0x08000000
+#define IOP3XX_PCI_MEM_WINDOW_SIZE	IOP33X_PCI_MEM_WINDOW_SIZE
+
 
 #endif
include/asm-arm/arch-iop33x/memory.h +2 -2
index 0d39139..c874912 100644
@@ -19,8 +19,8 @@ #define PHYS_OFFSET	UL(0x00000000)
  * bus_to_virt: Used to convert an address for DMA operations
  *		to an address that the kernel can use.
  */
-#define __virt_to_bus(x)	(((__virt_to_phys(x)) & ~(*IOP3XX_IATVR2)) | ((*IOP3XX_IABAR2) & 0xfffffff0))
-#define __bus_to_virt(x)	(__phys_to_virt(((x) & ~(*IOP3XX_IALR2)) | ( *IOP3XX_IATVR2)))
+#define __virt_to_bus(x)	(__virt_to_phys(x))
+#define __bus_to_virt(x)	(__phys_to_virt(x))
 
 
 #endif
include/asm-arm/hardware/iop3xx.h +17 -3
index 295789a..5a084c8 100644
@@ -28,6 +28,7 @@ #ifndef __ASSEMBLY__
 extern void gpio_line_config(int line, int direction);
 extern int  gpio_line_get(int line);
 extern void gpio_line_set(int line, int value);
+extern int init_atu;
 #endif
 
 
@@ -98,6 +99,21 @@ #define IOP3XX_PCIXNEXT	IOP3XX_REG_ADDR8
 #define IOP3XX_PCIXCMD		IOP3XX_REG_ADDR16(0x01e2)
 #define IOP3XX_PCIXSR		IOP3XX_REG_ADDR32(0x01e4)
 #define IOP3XX_PCIIRSR		IOP3XX_REG_ADDR32(0x01ec)
+#define IOP3XX_PCSR_OUT_Q_BUSY (1 << 15)
+#define IOP3XX_PCSR_IN_Q_BUSY	(1 << 14)
+#define IOP3XX_ATUCR_OUT_EN	(1 << 1)
+
+#define IOP3XX_INIT_ATU_DEFAULT 0
+#define IOP3XX_INIT_ATU_DISABLE -1
+#define IOP3XX_INIT_ATU_ENABLE	 1
+
+#ifdef CONFIG_IOP3XX_ATU
+#define iop3xx_get_init_atu(x) (init_atu == IOP3XX_INIT_ATU_DEFAULT ?\
+				IOP3XX_INIT_ATU_ENABLE : init_atu)
+#else
+#define iop3xx_get_init_atu(x) (init_atu == IOP3XX_INIT_ATU_DEFAULT ?\
+				IOP3XX_INIT_ATU_DISABLE : init_atu)
+#endif
 
 /* Messaging Unit  */
 #define IOP3XX_IMR0		IOP3XX_REG_ADDR32(0x0310)
@@ -219,14 +235,12 @@ #define IOP3XX_IBMR1		IOP3XX_REG_ADDR32(
 /*
  * IOP3XX I/O and Mem space regions for PCI autoconfiguration
  */
-#define IOP3XX_PCI_MEM_WINDOW_SIZE	0x04000000
 #define IOP3XX_PCI_LOWER_MEM_PA	0x80000000
-#define IOP3XX_PCI_LOWER_MEM_BA	(*IOP3XX_OMWTVR0)
 
 #define IOP3XX_PCI_IO_WINDOW_SIZE	0x00010000
 #define IOP3XX_PCI_LOWER_IO_PA		0x90000000
 #define IOP3XX_PCI_LOWER_IO_VA		0xfe000000
-#define IOP3XX_PCI_LOWER_IO_BA		(*IOP3XX_OIOWTVR)
+#define IOP3XX_PCI_LOWER_IO_BA		0x90000000
 
 
 #ifndef __ASSEMBLY__
#41 lore raw

Re: [PATCH 18/19] iop3xx: Give Linux control over PCI (ATU) initialization

Quoted 279 lines · Dan Williams

From: Dan Williams <[email protected]>

Currently the iop3xx platform support code assumes that RedBoot is the bootloader and has already initialized the ATU. Linux should handle this initialization for three reasons:

1/ The memory map that RedBoot sets up is not optimal (page_to_dma and virt_to_phys return different addresses). The effect of this is that using the dma mapping API for the internal bus dma units generates pci bus addresses that are incorrect for the internal bus.

2/ Not all iop platforms use RedBoot

3/ If the ATU is already initialized it indicates that the iop is an add-in card in another host, it does not own the PCI bus, and should not be re-initialized.

---

 arch/arm/mach-iop32x/Kconfig         |    8 ++
 arch/arm/mach-iop32x/ep80219.c       |    4 +
 arch/arm/mach-iop32x/iq31244.c       |    5 +
 arch/arm/mach-iop32x/iq80321.c       |    5 +
 arch/arm/mach-iop33x/Kconfig         |    8 ++
 arch/arm/mach-iop33x/iq80331.c       |    5 +
 arch/arm/mach-iop33x/iq80332.c       |    4 +
 arch/arm/plat-iop/pci.c              |  140 ++++++++++++++++++++++++++++++++++
 include/asm-arm/arch-iop32x/iop32x.h |    9 ++
 include/asm-arm/arch-iop32x/memory.h |    4 -
 include/asm-arm/arch-iop33x/iop33x.h |   10 ++
 include/asm-arm/arch-iop33x/memory.h |    4 -
 include/asm-arm/hardware/iop3xx.h    |   20 ++++-
 13 files changed, 214 insertions(+), 12 deletions(-)
arch/arm/mach-iop32x/Kconfig +8 -0
index 05549a5..b2788e3 100644
@@ -22,6 +22,14 @@ config ARCH_IQ80321
 	  Say Y here if you want to run your kernel on the Intel IQ80321
 	  evaluation kit for the IOP321 processor.
 
+config IOP3XX_ATU
+        bool "Enable the PCI Controller"
+        default y
+        help
+          Say Y here if you want the IOP to initialize its PCI Controller.
+          Say N if the IOP is an add in card, the host system owns the PCI
+          bus in this case.
+
 endmenu
 
 endif
arch/arm/mach-iop32x/ep80219.c +3 -1
index f616d3e..1a5c586 100644
@@ -100,7 +100,7 @@ ep80219_pci_map_irq(struct pci_dev *dev,
 
 static struct hw_pci ep80219_pci __initdata = {
 	.swizzle	= pci_std_swizzle,
-	.nr_controllers = 1,
+	.nr_controllers = 0,
 	.setup		= iop3xx_pci_setup,
 	.preinit	= iop3xx_pci_preinit,
 	.scan		= iop3xx_pci_scan_bus,
@@ -109,6 +109,8 @@ static struct hw_pci ep80219_pci __initd
 
 static int __init ep80219_pci_init(void)
 {
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		ep80219_pci.nr_controllers = 1;
 #if 0
 	if (machine_is_ep80219())
 		pci_common_init(&ep80219_pci);
arch/arm/mach-iop32x/iq31244.c +4 -1
index 967a696..25d5d62 100644
@@ -97,7 +97,7 @@ iq31244_pci_map_irq(struct pci_dev *dev,
 
 static struct hw_pci iq31244_pci __initdata = {
 	.swizzle	= pci_std_swizzle,
-	.nr_controllers = 1,
+	.nr_controllers = 0,
 	.setup		= iop3xx_pci_setup,
 	.preinit	= iop3xx_pci_preinit,
 	.scan		= iop3xx_pci_scan_bus,
@@ -106,6 +106,9 @@ static struct hw_pci iq31244_pci __initd
 
 static int __init iq31244_pci_init(void)
 {
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		iq31244_pci.nr_controllers = 1;
+
 	if (machine_is_iq31244())
 		pci_common_init(&iq31244_pci);
 
arch/arm/mach-iop32x/iq80321.c +4 -1
index ef4388c..cdd2265 100644
@@ -97,7 +97,7 @@ iq80321_pci_map_irq(struct pci_dev *dev,
 
 static struct hw_pci iq80321_pci __initdata = {
 	.swizzle	= pci_std_swizzle,
-	.nr_controllers = 1,
+	.nr_controllers = 0,
 	.setup		= iop3xx_pci_setup,
 	.preinit	= iop3xx_pci_preinit,
 	.scan		= iop3xx_pci_scan_bus,
@@ -106,6 +106,9 @@ static struct hw_pci iq80321_pci __initd
 
 static int __init iq80321_pci_init(void)
 {
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		iq80321_pci.nr_controllers = 1;
+
 	if (machine_is_iq80321())
 		pci_common_init(&iq80321_pci);
 
arch/arm/mach-iop33x/Kconfig +8 -0
index 9aa016b..45598e0 100644
@@ -16,6 +16,14 @@ config MACH_IQ80332
 	  Say Y here if you want to run your kernel on the Intel IQ80332
 	  evaluation kit for the IOP332 chipset.
 
+config IOP3XX_ATU
+	bool "Enable the PCI Controller"
+	default y
+	help
+	  Say Y here if you want the IOP to initialize its PCI Controller.
+	  Say N if the IOP is an add in card, the host system owns the PCI
+	  bus in this case.
+
 endmenu
 
 endif
arch/arm/mach-iop33x/iq80331.c +4 -1
index 7714c94..3807000 100644
@@ -78,7 +78,7 @@ iq80331_pci_map_irq(struct pci_dev *dev,
 
 static struct hw_pci iq80331_pci __initdata = {
 	.swizzle	= pci_std_swizzle,
-	.nr_controllers = 1,
+	.nr_controllers = 0,
 	.setup		= iop3xx_pci_setup,
 	.preinit	= iop3xx_pci_preinit,
 	.scan		= iop3xx_pci_scan_bus,
@@ -87,6 +87,9 @@ static struct hw_pci iq80331_pci __initd
 
 static int __init iq80331_pci_init(void)
 {
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		iq80331_pci.nr_controllers = 1;
+
 	if (machine_is_iq80331())
 		pci_common_init(&iq80331_pci);
 
arch/arm/mach-iop33x/iq80332.c +4 -0
index a3fa7f8..8780d55 100644
@@ -93,6 +93,10 @@ static struct hw_pci iq80332_pci __initd
 
 static int __init iq80332_pci_init(void)
 {
+
+	if (iop3xx_get_init_atu() == IOP3XX_INIT_ATU_ENABLE)
+		iq80332_pci.nr_controllers = 1;
+
 	if (machine_is_iq80332())
 		pci_common_init(&iq80332_pci);
 
arch/arm/plat-iop/pci.c +84 -1
index e647812..19aace9 100644
@@ -55,7 +55,7 @@ static u32 iop3xx_cfg_address(struct pci
  * This routine checks the status of the last configuration cycle.  If an error
  * was detected it returns a 1, else it returns a 0.  The errors being checked
  * are parity, master abort, target abort (master and target).  These types of
- * errors occure during a config cycle where there is no device, like during
+ * errors occur during a config cycle where there is no device, like during
  * the discovery stage.
  */
 static int iop3xx_pci_status(void)
@@ -223,8 +223,111 @@ struct pci_bus *iop3xx_pci_scan_bus(int 
 	return pci_scan_bus(sys->busnr, &iop3xx_ops, sys);
 }
 
+void __init iop3xx_atu_setup(void)
+{
+	/* BAR 0 ( Disabled ) */
+	*IOP3XX_IAUBAR0 = 0x0;
+	*IOP3XX_IABAR0  = 0x0;
+	*IOP3XX_IATVR0  = 0x0;
+	*IOP3XX_IALR0   = 0x0;
+
+	/* BAR 1 ( Disabled ) */
+	*IOP3XX_IAUBAR1 = 0x0;
+	*IOP3XX_IABAR1  = 0x0;
+	*IOP3XX_IALR1   = 0x0;
+
+	/* BAR 2 (1:1 mapping with Physical RAM) */
+	/* Set limit and enable */
+	*IOP3XX_IALR2 = ~((u32)IOP3XX_MAX_RAM_SIZE - 1) & ~0x1;
+	*IOP3XX_IAUBAR2 = 0x0;
+
+	/* Align the inbound bar with the base of memory */
+	*IOP3XX_IABAR2 = PHYS_OFFSET |
+			       PCI_BASE_ADDRESS_MEM_TYPE_64 |
+			       PCI_BASE_ADDRESS_MEM_PREFETCH;
+
+	*IOP3XX_IATVR2 = PHYS_OFFSET;
+
+	/* Outbound window 0 */
+	*IOP3XX_OMWTVR0 = IOP3XX_PCI_LOWER_MEM_PA;
+	*IOP3XX_OUMWTVR0 = 0;
+
+	/* Outbound window 1 */
+	*IOP3XX_OMWTVR1 = IOP3XX_PCI_LOWER_MEM_PA + IOP3XX_PCI_MEM_WINDOW_SIZE;
+	*IOP3XX_OUMWTVR1 = 0;
+
+	/* BAR 3 ( Disabled ) */
+	*IOP3XX_IAUBAR3 = 0x0;
+	*IOP3XX_IABAR3  = 0x0;
+	*IOP3XX_IATVR3  = 0x0;
+	*IOP3XX_IALR3   = 0x0;
+
+	/* Setup the I/O Bar
+	 */
+	*IOP3XX_OIOWTVR = IOP3XX_PCI_LOWER_IO_PA;;
+
+	/* Enable inbound and outbound cycles
+	 */
+	*IOP3XX_ATUCMD |= PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER |
+			       PCI_COMMAND_PARITY | PCI_COMMAND_SERR;
+	*IOP3XX_ATUCR |= IOP3XX_ATUCR_OUT_EN;
+}
+
+void __init iop3xx_atu_disable(void)
+{
+	*IOP3XX_ATUCMD = 0;
+	*IOP3XX_ATUCR = 0;
+
+	/* wait for cycles to quiesce */
+	while (*IOP3XX_PCSR & (IOP3XX_PCSR_OUT_Q_BUSY |
+				     IOP3XX_PCSR_IN_Q_BUSY))
+		cpu_relax();
+
+	/* BAR 0 ( Disabled ) */
+	*IOP3XX_IAUBAR0 = 0x0;
+	*IOP3XX_IABAR0  = 0x0;
+	*IOP3XX_IATVR0  = 0x0;
+	*IOP3XX_IALR0   = 0x0;
+
+	/* BAR 1 ( Disabled ) */
+	*IOP3XX_IAUBAR1 = 0x0;
+	*IOP3XX_IABAR1  = 0x0;
+	*IOP3XX_IALR1   = 0x0;
+
+	/* BAR 2 ( Disabled ) */
+	*IOP3XX_IAUBAR2 = 0x0;
+	*IOP3XX_IABAR2  = 0x0;
+	*IOP3XX_IATVR2  = 0x0;
+	*IOP3XX_IALR2   = 0x0;
+
+	/* BAR 3 ( Disabled ) */
+	*IOP3XX_IAUBAR3 = 0x0;
+	*IOP3XX_IABAR3  = 0x0;
+	*IOP3XX_IATVR3  = 0x0;
+	*IOP3XX_IALR3   = 0x0;
+

arch/arm/plat-iop/pci.c struct pci_bus *iop3xx_pci_scan_bus(int · line 318

Quoted 10 lines
arch/arm/plat-iop/pci.c +10 -0
+	/* Clear the outbound windows */
+	*IOP3XX_OIOWTVR  = 0;
+
+	/* Outbound window 0 */
+	*IOP3XX_OMWTVR0 = 0;
+	*IOP3XX_OUMWTVR0 = 0;
+
+	/* Outbound window 1 */
+	*IOP3XX_OMWTVR1 = 0;
+	*IOP3XX_OUMWTVR1 = 0;

You should be using readl(), writel() variants rather than writing C code that appears to be normal, but in reality has hardware side-effects.

Jeff

#42 lore raw

PATCH 19/19 iop3xx: IOP 32x and 33x support for the iop-adma driver

In reply to Dan Williams

From: Dan Williams <[email protected]>

Adds the platform device definitions and the architecture specific support routines (i.e. register initialization and descriptor formats) for the iop-adma driver.

Changelog:
* add support for > 1k zero sum buffer sizes
* added dma/aau platform devices to iq80321 and iq80332 setup
* fixed the calculation in iop_desc_is_aligned
* support xor buffer sizes larger than 16MB
* fix places where software descriptors are assumed to be contiguous, only hardware descriptors are contiguous
* iop32x does not support hardware zero sum, add software emulation support for up to a PAGE_SIZE buffer size
* added raid5 dma driver support functions

---

 arch/arm/mach-iop32x/iq80321.c         |  141 +++++
 arch/arm/mach-iop33x/iq80331.c         |    9 
 arch/arm/mach-iop33x/iq80332.c         |    8 
 arch/arm/mach-iop33x/setup.c           |  132 +++++
 include/asm-arm/arch-iop32x/adma.h     |    5 
 include/asm-arm/arch-iop33x/adma.h     |    5 
 include/asm-arm/hardware/iop3xx-adma.h |  901 ++++++++++++++++++++++++++++++++
 7 files changed, 1201 insertions(+), 0 deletions(-)
arch/arm/mach-iop32x/iq80321.c +141 -0
index cdd2265..79d6514 100644
@@ -33,6 +33,9 @@ #include <asm/mach/time.h>
 #include <asm/mach-types.h>
 #include <asm/page.h>
 #include <asm/pgtable.h>
+#ifdef CONFIG_DMA_ENGINE
+#include <asm/hardware/iop_adma.h>
+#endif
 
 /*
  * IQ80321 timer tick configuration.
@@ -170,12 +173,150 @@ static struct platform_device iq80321_se
 	.resource	= &iq80321_uart_resource,
 };
 
+#ifdef CONFIG_DMA_ENGINE
+/* AAU and DMA Channels */
+static struct resource iop3xx_dma_0_resources[] = {
+	[0] = {
+		.start = (unsigned long) IOP3XX_DMA_CCR(0),
+		.end = ((unsigned long) IOP3XX_DMA_DCR(0)) + 4,
+		.flags = IORESOURCE_MEM,
+	},
+	[1] = {
+		.start = IRQ_IOP32X_DMA0_EOT,
+		.end = IRQ_IOP32X_DMA0_EOT,
+		.flags = IORESOURCE_IRQ
+	},
+	[2] = {
+		.start = IRQ_IOP32X_DMA0_EOC,
+		.end = IRQ_IOP32X_DMA0_EOC,
+		.flags = IORESOURCE_IRQ
+	},
+	[3] = {
+		.start = IRQ_IOP32X_DMA0_ERR,
+		.end = IRQ_IOP32X_DMA0_ERR,
+		.flags = IORESOURCE_IRQ
+	}
+};
+
+static struct resource iop3xx_dma_1_resources[] = {
+	[0] = {
+		.start = (unsigned long) IOP3XX_DMA_CCR(1),
+		.end = ((unsigned long) IOP3XX_DMA_DCR(1)) + 4,
+		.flags = IORESOURCE_MEM,
+	},
+	[1] = {
+		.start = IRQ_IOP32X_DMA1_EOT,
+		.end = IRQ_IOP32X_DMA1_EOT,
+		.flags = IORESOURCE_IRQ
+	},
+	[2] = {
+		.start = IRQ_IOP32X_DMA1_EOC,
+		.end = IRQ_IOP32X_DMA1_EOC,
+		.flags = IORESOURCE_IRQ
+	},
+	[3] = {
+		.start = IRQ_IOP32X_DMA1_ERR,
+		.end = IRQ_IOP32X_DMA1_ERR,
+		.flags = IORESOURCE_IRQ
+	}
+};
+
+
+static struct resource iop3xx_aau_resources[] = {
+	[0] = {
+		.start = (unsigned long) IOP3XX_AAU_ACR,
+		.end = (unsigned long) IOP3XX_AAU_SAR_EDCR(32),
+		.flags = IORESOURCE_MEM,
+	},
+	[1] = {
+		.start = IRQ_IOP32X_AA_EOT,
+		.end = IRQ_IOP32X_AA_EOT,
+		.flags = IORESOURCE_IRQ
+	},
+	[2] = {
+		.start = IRQ_IOP32X_AA_EOC,
+		.end = IRQ_IOP32X_AA_EOC,
+		.flags = IORESOURCE_IRQ
+	},
+	[3] = {
+		.start = IRQ_IOP32X_AA_ERR,
+		.end = IRQ_IOP32X_AA_ERR,
+		.flags = IORESOURCE_IRQ
+	}
+};
+
+static u64 iop3xx_adma_dmamask = DMA_32BIT_MASK;
+
+static struct iop_adma_platform_data iop3xx_dma_0_data = {
+	.hw_id = IOP3XX_DMA0_ID,
+	.capabilities =	DMA_MEMCPY | DMA_MEMCPY_CRC32C,
+	.pool_size = PAGE_SIZE,
+};
+
+static struct iop_adma_platform_data iop3xx_dma_1_data = {
+	.hw_id = IOP3XX_DMA1_ID,
+	.capabilities =	DMA_MEMCPY | DMA_MEMCPY_CRC32C,
+	.pool_size = PAGE_SIZE,
+};
+
+static struct iop_adma_platform_data iop3xx_aau_data = {
+	.hw_id = IOP3XX_AAU_ID,
+	.capabilities =	DMA_XOR | DMA_ZERO_SUM | DMA_MEMSET,
+	.pool_size = 3 * PAGE_SIZE,
+};
+
+struct platform_device iop3xx_dma_0_channel = {
+	.name = "IOP-ADMA",
+	.id = 0,
+	.num_resources = 4,
+	.resource = iop3xx_dma_0_resources,
+	.dev = {
+		.dma_mask = &iop3xx_adma_dmamask,
+		.coherent_dma_mask = DMA_64BIT_MASK,
+		.platform_data = (void *) &iop3xx_dma_0_data,
+	},
+};
+
+struct platform_device iop3xx_dma_1_channel = {
+	.name = "IOP-ADMA",
+	.id = 1,
+	.num_resources = 4,
+	.resource = iop3xx_dma_1_resources,
+	.dev = {
+		.dma_mask = &iop3xx_adma_dmamask,
+		.coherent_dma_mask = DMA_64BIT_MASK,
+		.platform_data = (void *) &iop3xx_dma_1_data,
+	},
+};
+
+struct platform_device iop3xx_aau_channel = {
+	.name = "IOP-ADMA",
+	.id = 2,
+	.num_resources = 4,
+	.resource = iop3xx_aau_resources,
+	.dev = {
+		.dma_mask = &iop3xx_adma_dmamask,
+		.coherent_dma_mask = DMA_64BIT_MASK,
+		.platform_data = (void *) &iop3xx_aau_data,
+	},
+};
+#endif /* CONFIG_DMA_ENGINE */
+
+extern struct platform_device iop3xx_dma_0_channel;
+extern struct platform_device iop3xx_dma_1_channel;
+extern struct platform_device iop3xx_aau_channel;
 static void __init iq80321_init_machine(void)
 {
 	platform_device_register(&iop3xx_i2c0_device);
 	platform_device_register(&iop3xx_i2c1_device);
 	platform_device_register(&iq80321_flash_device);
 	platform_device_register(&iq80321_serial_device);
+	#ifdef CONFIG_DMA_ENGINE
+	platform_device_register(&iop3xx_dma_0_channel);
+	platform_device_register(&iop3xx_dma_1_channel);
+	platform_device_register(&iop3xx_aau_channel);
+	#endif
+
 }
 
 MACHINE_START(IQ80321, "Intel IQ80321")
arch/arm/mach-iop33x/iq80331.c +9 -0
index 3807000..34bedc6 100644
@@ -122,6 +122,10 @@ static struct platform_device iq80331_fl
 	.resource	= &iq80331_flash_resource,
 };
 
+
+extern struct platform_device iop3xx_dma_0_channel;
+extern struct platform_device iop3xx_dma_1_channel;
+extern struct platform_device iop3xx_aau_channel;
 static void __init iq80331_init_machine(void)
 {
 	platform_device_register(&iop3xx_i2c0_device);
@@ -129,6 +133,11 @@ static void __init iq80331_init_machine(
 	platform_device_register(&iop33x_uart0_device);
 	platform_device_register(&iop33x_uart1_device);
 	platform_device_register(&iq80331_flash_device);
+	#ifdef CONFIG_DMA_ENGINE
+	platform_device_register(&iop3xx_dma_0_channel);
+	platform_device_register(&iop3xx_dma_1_channel);
+	platform_device_register(&iop3xx_aau_channel);
+	#endif
 }
 
 MACHINE_START(IQ80331, "Intel IQ80331")
arch/arm/mach-iop33x/iq80332.c +8 -0
index 8780d55..ed36016 100644
@@ -129,6 +129,9 @@ static struct platform_device iq80332_fl
 	.resource	= &iq80332_flash_resource,
 };
 
+extern struct platform_device iop3xx_dma_0_channel;
+extern struct platform_device iop3xx_dma_1_channel;
+extern struct platform_device iop3xx_aau_channel;
 static void __init iq80332_init_machine(void)
 {
 	platform_device_register(&iop3xx_i2c0_device);
@@ -136,6 +139,11 @@ static void __init iq80332_init_machine(
 	platform_device_register(&iop33x_uart0_device);
 	platform_device_register(&iop33x_uart1_device);
 	platform_device_register(&iq80332_flash_device);
+	#ifdef CONFIG_DMA_ENGINE
+	platform_device_register(&iop3xx_dma_0_channel);
+	platform_device_register(&iop3xx_dma_1_channel);
+	platform_device_register(&iop3xx_aau_channel);
+	#endif
 }
 
 MACHINE_START(IQ80332, "Intel IQ80332")
arch/arm/mach-iop33x/setup.c +132 -0

141 lines not on this page · raw message

include/asm-arm/arch-iop32x/adma.h new file +5 -0
new file mode 100644
index 0000000..5ed9203
@@ -0,0 +1,5 @@
+#ifndef IOP32X_ADMA_H
+#define IOP32X_ADMA_H
+#include <asm/hardware/iop3xx-adma.h>
+#endif
+
include/asm-arm/arch-iop33x/adma.h new file +5 -0
new file mode 100644
index 0000000..4b92f79
@@ -0,0 +1,5 @@
+#ifndef IOP33X_ADMA_H
+#define IOP33X_ADMA_H
+#include <asm/hardware/iop3xx-adma.h>
+#endif
+
include/asm-arm/hardware/iop3xx-adma.h new file +901 -0

901 lines not on this page · raw message

#43 lore raw
In reply to Dan Williams
Quoted 47 lines · Dan Williams

Neil,

The following patches implement hardware accelerated raid5 for the Intel Xscale® series of I/O Processors. The MD changes allow stripe operations to run outside the spin lock in a work queue. Hardware acceleration is achieved by using a dma-engine-aware work queue routine instead of the default software only routine.

Since the last release of the raid5 changes many bug fixes and other improvements have been made as a result of stress testing. See the per patch change logs for more information about what was fixed. This release is the first release of the full dma implementation.

The patches touch 3 areas, the md-raid5 driver, the generic dmaengine interface, and a platform device driver for IOPs. The raid5 changes follow your comments concerning making the acceleration implementation similar to how the stripe cache handles I/O requests. The dmaengine changes are the second release of this code. They expand the interface to handle more than memcpy operations, and add a generic raid5-dma client. The iop-adma driver supports dma memcpy, xor, xor zero sum, and memset across all IOP architectures (32x, 33x, and 13xx).

Concerning the context switching performance concerns raised at the previous release, I have observed the following. For the hardware accelerated case it appears that performance is always better with the work queue than without since it allows multiple stripes to be operated on simultaneously. I expect the same for an SMP platform, but so far my testing has been limited to IOPs. For a single-processor non-accelerated configuration I have not observed performance degradation with work queue support enabled, but in the Kconfig option help text I recommend disabling it (CONFIG_MD_RAID456_WORKQUEUE).

Please consider the patches for -mm.

-Dan

[PATCH 01/19] raid5: raid5_do_soft_block_ops
[PATCH 02/19] raid5: move write operations to a workqueue
[PATCH 03/19] raid5: move check parity operations to a workqueue [PATCH 04/19] raid5: move compute block operations to a workqueue [PATCH 05/19] raid5: move read completion copies to a workqueue [PATCH 06/19] raid5: move the reconstruct write expansion operation to a workqueue [PATCH 07/19] raid5: remove compute_block and compute_parity5
[PATCH 08/19] dmaengine: enable multiple clients and operations [PATCH 09/19] dmaengine: reduce backend address permutations
[PATCH 10/19] dmaengine: expose per channel dma mapping characteristics to clients [PATCH 11/19] dmaengine: add memset as an asynchronous dma operation

Quoted 8 lines

[PATCH 12/19] dmaengine: dma_async_memcpy_err for DMA engines that do not support memcpy [PATCH 13/19] dmaengine: add support for dma xor zero sum operations [PATCH 14/19] dmaengine: add dma_sync_wait
[PATCH 15/19] dmaengine: raid5 dma client
[PATCH 16/19] dmaengine: Driver for the Intel IOP 32x, 33x, and 13xx RAID engines [PATCH 17/19] iop3xx: define IOP3XX_REG_ADDR[32|16|8] and clean up DMA/AAU defs [PATCH 18/19] iop3xx: Give Linux control over PCI (ATU) initialization [PATCH 19/19] iop3xx: IOP 32x and 33x support for the iop-adma driver

Can devices like drivers/scsi/sata_sx4.c or drivers/scsi/sata_promise.c take advantage of this? Promise silicon supports RAID5 XOR offload.

If so, how? If not, why not? :)

Jeff

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#44 lore raw
Quoted 57 lines · Jeff Garzik
Quoted 47 lines · Dan Williams

Neil,

The following patches implement hardware accelerated raid5 for the Intel Xscale(r) series of I/O Processors. The MD changes allow stripe operations to run outside the spin lock in a work queue. Hardware acceleration is achieved by using a dma-engine-aware work queue routine instead of the default software only routine.

Since the last release of the raid5 changes many bug fixes and other improvements have been made as a result of stress testing. See the per patch change logs for more information about what was fixed. This release is the first release of the full dma implementation.

The patches touch 3 areas, the md-raid5 driver, the generic dmaengine interface, and a platform device driver for IOPs. The raid5 changes follow your comments concerning making the acceleration implementation similar to how the stripe cache handles I/O requests. The dmaengine changes are the second release of this code. They expand the interface to handle more than memcpy operations, and add a generic raid5-dma client. The iop-adma driver supports dma memcpy, xor, xor zero sum, and memset across all IOP architectures (32x, 33x, and 13xx).

Concerning the context switching performance concerns raised at the previous release, I have observed the following. For the hardware accelerated case it appears that performance is always better with the work queue than without since it allows multiple stripes to be operated on simultaneously. I expect the same for an SMP platform, but so far my testing has been limited to IOPs. For a single-processor non-accelerated configuration I have not observed performance degradation with work queue support enabled, but in the Kconfig option help text I recommend disabling it (CONFIG_MD_RAID456_WORKQUEUE).

Please consider the patches for -mm.

-Dan

[PATCH 01/19] raid5: raid5_do_soft_block_ops
[PATCH 02/19] raid5: move write operations to a workqueue
[PATCH 03/19] raid5: move check parity operations to a workqueue [PATCH 04/19] raid5: move compute block operations to a workqueue [PATCH 05/19] raid5: move read completion copies to a workqueue [PATCH 06/19] raid5: move the reconstruct write expansion operation to a workqueue [PATCH 07/19] raid5: remove compute_block and compute_parity5
[PATCH 08/19] dmaengine: enable multiple clients and operations [PATCH 09/19] dmaengine: reduce backend address permutations
[PATCH 10/19] dmaengine: expose per channel dma mapping characteristics to clients [PATCH 11/19] dmaengine: add memset as an asynchronous dma operation

Quoted 8 lines

[PATCH 12/19] dmaengine: dma_async_memcpy_err for DMA engines that do not support memcpy [PATCH 13/19] dmaengine: add support for dma xor zero sum operations [PATCH 14/19] dmaengine: add dma_sync_wait
[PATCH 15/19] dmaengine: raid5 dma client
[PATCH 16/19] dmaengine: Driver for the Intel IOP 32x, 33x, and 13xx RAID engines [PATCH 17/19] iop3xx: define IOP3XX_REG_ADDR[32|16|8] and clean up DMA/AAU defs [PATCH 18/19] iop3xx: Give Linux control over PCI (ATU) initialization [PATCH 19/19] iop3xx: IOP 32x and 33x support for the iop-adma driver

Quoted 4 lines

Can devices like drivers/scsi/sata_sx4.c or drivers/scsi/sata_promise.c take advantage of this? Promise silicon supports RAID5 XOR offload.

If so, how? If not, why not? :)

This is a frequently asked question, Alan Cox had the same one at OLS. The answer is "probably." The only complication I currently see is where/how the stripe cache is maintained. With the IOPs its easy because the DMA engines operate directly on kernel memory. With the Promise card I believe they have memory on the card and it's not clear to me if the XOR engines on the card can deal with host memory. Also, MD would need to be modified to handle a stripe cache located on a device, or somehow synchronize its local cache with card in a manner that is still able to beat software only MD.

Quoted 1 line

Jeff

Dan

#45 lore raw
Quoted 9 lines · Dan Williams

This is a frequently asked question, Alan Cox had the same one at OLS. The answer is "probably." The only complication I currently see is where/how the stripe cache is maintained. With the IOPs its easy because the DMA engines operate directly on kernel memory. With the Promise card I believe they have memory on the card and it's not clear to me if the XOR engines on the card can deal with host memory. Also, MD would need to be modified to handle a stripe cache located on a device, or somehow synchronize its local cache with card in a manner that is still able to beat software only MD.

sata_sx4 operates through [standard PC] memory on the card, and you use a DMA engine to copy memory to/from the card.

[select chipsets supported by] sata_promise operates directly on host memory.

So, while sata_sx4 is farther away from your direct-host-memory model, it also has much more potential for RAID acceleration: ideally, RAID1 just copies data to the card once, then copies the data to multiple drives from there. Similarly with RAID5, you can eliminate copies and offload XOR, presuming the drives are all connected to the same card.

Jeff

#46 lore raw
Quoted 14 lines · Jeff Garzik
Quoted 9 lines · Dan Williams

This is a frequently asked question, Alan Cox had the same one at OLS. The answer is "probably." The only complication I currently see is where/how the stripe cache is maintained. With the IOPs its easy because the DMA engines operate directly on kernel memory. With the Promise card I believe they have memory on the card and it's not clear to me if the XOR engines on the card can deal with host memory. Also, MD would need to be modified to handle a stripe cache located on a device, or somehow synchronize its local cache with card in a manner that is still able to beat software only MD.

sata_sx4 operates through [standard PC] memory on the card, and you use a DMA engine to copy memory to/from the card.

Quoted 8 lines

[select chipsets supported by] sata_promise operates directly on host memory.

So, while sata_sx4 is farther away from your direct-host-memory model, it also has much more potential for RAID acceleration: ideally, RAID1 just copies data to the card once, then copies the data to multiple drives from there. Similarly with RAID5, you can eliminate copies and offload XOR, presuming the drives are all connected to the same card.

In the sata_promise case its straight forward, all that is needed is dmaengine drivers for the xor and memcpy engines. This would be similar to the current I/OAT model where dma resources are provided by a PCI function. The sata_sx4 case would need a different flavor of the dma_do_raid5_block_ops routine, one that understands where the cache is located. MD would also need the capability to bypass the block layer since the data will have already been transferred to the card by a stripe cache operation

The RAID1 case give me pause because it seems any work along these lines requires that the implementation work for both MD and DM, which then eventually leads to being tasked with merging the two.

Quoted 1 line

Jeff

Dan

#47 lore raw
Quoted 32 lines · Dan Williams
Quoted 14 lines · Jeff Garzik
Quoted 9 lines · Dan Williams

This is a frequently asked question, Alan Cox had the same one at OLS. The answer is "probably." The only complication I currently see is where/how the stripe cache is maintained. With the IOPs its easy because the DMA engines operate directly on kernel memory. With the Promise card I believe they have memory on the card and it's not clear to me if the XOR engines on the card can deal with host memory. Also, MD would need to be modified to handle a stripe cache located on a device, or somehow synchronize its local cache with card in a manner that is still able to beat software only MD.

sata_sx4 operates through [standard PC] memory on the card, and you use a DMA engine to copy memory to/from the card.

Quoted 8 lines

[select chipsets supported by] sata_promise operates directly on host memory.

So, while sata_sx4 is farther away from your direct-host-memory model, it also has much more potential for RAID acceleration: ideally, RAID1 just copies data to the card once, then copies the data to multiple drives from there. Similarly with RAID5, you can eliminate copies and offload XOR, presuming the drives are all connected to the same card.

In the sata_promise case its straight forward, all that is needed is dmaengine drivers for the xor and memcpy engines. This would be similar to the current I/OAT model where dma resources are provided by a PCI function. The sata_sx4 case would need a different flavor of the dma_do_raid5_block_ops routine, one that understands where the cache is located. MD would also need the capability to bypass the block layer since the data will have already been transferred to the card by a stripe cache operation

Quoted 3 lines

The RAID1 case give me pause because it seems any work along these lines requires that the implementation work for both MD and DM, which then eventually leads to being tasked with merging the two.

RAID5 has similar properties. If all devices in a RAID5 array are attached to a single SX4 card, then a high level write to the RAID5 array is passed directly to the card, which then performs XOR, striping, etc.

Jeff

#48 lore raw
In reply to Dan Williams
Quoted 2 lines · Dan Williams

Neil,

...

Quoted 10 lines

Concerning the context switching performance concerns raised at the previous release, I have observed the following. For the hardware accelerated case it appears that performance is always better with the work queue than without since it allows multiple stripes to be operated on simultaneously. I expect the same for an SMP platform, but so far my testing has been limited to IOPs. For a single-processor non-accelerated configuration I have not observed performance degradation with work queue support enabled, but in the Kconfig option help text I recommend disabling it (CONFIG_MD_RAID456_WORKQUEUE).

Out of curiosity; how does accelerated compare to non-accelerated?

Signature · 1 line
 / jakob
#49 lore raw
Quoted 15 lines · Jakob Oestergaard
Quoted 2 lines · Dan Williams

Neil,

...

Quoted 10 lines

Concerning the context switching performance concerns raised at the previous release, I have observed the following. For the hardware accelerated case it appears that performance is always better with the work queue than without since it allows multiple stripes to be operated on simultaneously. I expect the same for an SMP platform, but so far my testing has been limited to IOPs. For a single-processor non-accelerated configuration I have not observed performance degradation with work queue support enabled, but in the Kconfig option help text I recommend disabling it (CONFIG_MD_RAID456_WORKQUEUE).

Quoted 1 line

Out of curiosity; how does accelerated compare to non-accelerated?

One quick example:
4-disk SATA array rebuild on iop321 without acceleration - 'top' reports md0_resync and md0_raid5 dueling for the CPU each at ~50% utilization.

With acceleration - 'top' reports md0_resync cpu utilization at ~90% with the rest split between md0_raid5 and md0_raid5_ops.

The sync speed reported by /proc/mdstat is ~40% higher in the accelerated case.

That being said, array resync is a special case, so your mileage may vary with other applications.

I will put together some data from bonnie++, iozone, maybe contest, and post it on SourceForge.

Quoted 1 line

/ jakob

Dan

#50 lore raw

On Wed, Sep 13, 2006 at 12:17:55PM -0700, Dan Williams wrote:
...

Quoted 7 lines
Quoted 1 line

Out of curiosity; how does accelerated compare to non-accelerated?

One quick example:
4-disk SATA array rebuild on iop321 without acceleration - 'top' reports md0_resync and md0_raid5 dueling for the CPU each at ~50% utilization.

Quoted 5 lines

With acceleration - 'top' reports md0_resync cpu utilization at ~90% with the rest split between md0_raid5 and md0_raid5_ops.

The sync speed reported by /proc/mdstat is ~40% higher in the accelerated case.

Ok, nice :)

Quoted 3 lines

That being said, array resync is a special case, so your mileage may vary with other applications.

Every-day usage I/O performance data would be nice indeed :)

Quoted 2 lines

I will put together some data from bonnie++, iozone, maybe contest, and post it on SourceForge.

Great!

Signature · 1 line
 / jakob
#51 lore raw
Quoted 21 lines · Jakob Oestergaard

On Wed, Sep 13, 2006 at 12:17:55PM -0700, Dan Williams wrote:
...

Quoted 7 lines
Quoted 1 line

Out of curiosity; how does accelerated compare to non-accelerated?

One quick example:
4-disk SATA array rebuild on iop321 without acceleration - 'top' reports md0_resync and md0_raid5 dueling for the CPU each at ~50% utilization.

Quoted 5 lines

With acceleration - 'top' reports md0_resync cpu utilization at ~90% with the rest split between md0_raid5 and md0_raid5_ops.

The sync speed reported by /proc/mdstat is ~40% higher in the accelerated case.

Ok, nice :)

Quoted 3 lines

That being said, array resync is a special case, so your mileage may vary with other applications.

Quoted 7 lines

Every-day usage I/O performance data would be nice indeed :)

Quoted 2 lines

I will put together some data from bonnie++, iozone, maybe contest, and post it on SourceForge.

Great!

I have posted some Iozone data and graphs showing the performance impact of the patches across the three iop processors iop321, iop331, and iop341. The general take away from the data is that using dma engines extends the region that Iozone calls the "buffer cache effect". Write performance benefited the most as expected, but read performance showed some modest gains as well. There are some regions (smaller file size and record length) that show a performance disadvantage but it is typically less than 5%.

The graphs map the relative performance multiplier that the raid patches generate ('2.6.18-rc6 performance' x 'performance multiplier' = '2.6.18-rc6-raid performance') . A value of '1' designates equal performance. The large cliff that drops to zero is a "not measured" region, i.e. the record length is larger than the file size. Iozone outputs to Excel, but I have also made pdf's of the graphs available.
Note: Openoffice-calc can view the data but it does not support the 3D surface graphs that Iozone uses.

Excel:
http://prdownloads.sourceforge.net/xscaleiop/iozone_raid_accel.xls?download

PDF Graphs:
http://prdownloads.sourceforge.net/xscaleiop/iop-iozone-graphs-20061010.tar.bz2?download

Regards,
Dan

#52 lore raw
In reply to Dan Williams
Quoted 7 lines · On Monday September 11, [email protected] wrote:

Neil,

The following patches implement hardware accelerated raid5 for the Intel Xscale® series of I/O Processors. The MD changes allow stripe operations to run outside the spin lock in a work queue. Hardware acceleration is achieved by using a dma-engine-aware work queue routine instead of the default software only routine.

Hi Dan,
Sorry for the delay in replying.
I've looked through these patches at last (mostly the raid-specific bits) and while there is clearly a lot of good stuff here, it does 'feel' right - it just seems too complex.

The particular issues that stand out to me are:
- 33 new STRIPE_OP_* flags. I'm sure there doesn't need to be that many new flags.
- the "raid5 dma client" patch moves far too much internal knowledge about raid5 into drivers/dma.

Clearly there are some complex issues being dealt with and some complexity is to be expected, but I feel there must be room for some serious simplification.

Let me try to describe how I envisage it might work.

As you know, the theory-of-operation of handle_stripe is that it assesses the state of a stripe deciding what actions to perform and then performs them. Synchronous actions (e.g. current parity calcs) are performed 'in-line'. Async actions (reads, writes) and actions that cannot be performed under a spinlock (->b_end_io) are recorded as being needed and then are initiated at the end of handle_stripe outside of the sh->lock.

The proposal is to bring the parity and other bulk-memory operations out of the spinlock and make them optionally asynchronous.

The set of tasks that might be needed to be performed on a stripe are:
Clear a target cache block
pre-xor various cache blocks into a target
copy data out of bios into cache blocks. (drain) post-xor various cache blocks into a target copy data into bios out of cache blocks (fill) test if a cache block is all zeros
start a read on a cache block
start a write on a cache block

(There is also a memcpy when expanding raid5. I think I would try to
simply avoid that copy and move pointers around instead).

Some of these steps require sequencing. e.g.
clear, pre-xor, copy, post-xor, write
for a rwm cycle.
We could require handle_stripe to be called again for each step. i.e. first call just clears the target and flags it as clear. Next call initiates the pre-xor and flags that as done. Etc. However I think that would make the non-offloaded case too slow, or at least too clumsy.

So instead we set flags to say what needs to be done and have a workqueue system that does it.

(so far this is all quite similar to what you have done.)

So handle_stripe would set various flag and other things (like identify which block was the 'target' block) and run the following in a workqueue:

raid5_do_stuff(struct stripe_head *sh)

{
	raid5_cont_t *conf = sh->raid_conf;

	if (test_bit(CLEAR_TARGET, &sh->ops.pending)) {
		struct page = *p->sh->dev[sh->ops.target].page;
		rv = async_memset(p, 0, 0, PAGE_SIZE, ops_done, sh);
		if (rv != BUSY)
			clear_bit(CLEAR_TARGET, &sh->ops.pending);
		if (rv != COMPLETE)
			goto out;
	}

	while (test_bit(PRE_XOR, &sh->ops.pending)) {
		struct page *plist[XOR_MAX];
		int offset[XOR_MAX];
		int pos = 0;
		int d;

		for (d = sh->ops.nextdev;
		     d < conf->raid_disks && pos < XOR_MAX ;
		     d++) {
			if (sh->ops.nextdev == sh->ops.target)
				continue;
			if (!test_bit(R5_WantPreXor, &sh->dev[d].flags))
				continue;
			plist[pos] = sh->dev[d].page;
			offset[pos++] = 0;
		}
		if (pos) {
			struct page *p = sh->dev[sh->ops.target].page;
			rv = async_xor(p, 0, plist, offset, pos, PAGE_SIZE,
				       ops_done, sh);
			if (rv != BUSY)
				sh->ops.nextdev = d;
			if (rv != COMPLETE)
				goto out;
		} else {
			clear_bit(PRE_XOR, &sh->ops.pending);
			sh->ops.nextdev = 0;
		}
	}
		
	while (test_bit(COPY_IN, &sh0>ops.pending)) {
		...
	}
	....

	if (test_bit(START_IO, &sh->ops.pending)) {
		int d;
		for (d = 0 ; d < conf->raid_disk ; d++) {
			/* all that code from the end of handle_stripe */
		}

	release_stripe(conf, sh);
	return;

out:

	if (rv == BUSY) {
		/* wait on something and try again ???*/
	}
	return;
}

ops_done(struct stripe_head *sh)

{
	queue_work(....whatever..);
}

Things to note:
- we keep track of where we are up to in sh->ops. .pending is flags saying what is left to be done .next_dev is the next device to process for operations that work on several devices
.next_bio, .next_iov will be needed for copy operations that
cross multiple bios and iovecs.

- Each sh->dev has R5_Want flags reflecting which multi-device operations are wanted on each device.

- async bulk-memory operations take pages, offsets, and lengths, and can return COMPLETE (if the operation was performed synchronously) IN_PROGRESS (if it has been started, or at least queued) or BUSY if it couldn't even be queued. Exactly what to do in that case I'm not sure. Probably we need a waitqueue to wait on.

- The interface between the client and the ADMA hardware is a collection of async_ functions. async_memcpy, async_xor, async_memset etc.

I gather there needs to be some understanding
about whether the pages are already appropriately mapped for DMA or whether a mapping is needed. Maybe an extra flag argument should be passed.

I imagine that any piece of ADMA hardware would register with the 'async_*' subsystem, and a call to async_X would be routed as appropriate, or be run in-line.

This approach introduces 8 flags for sh->ops.pending and maybe two or three new R5_Want* flags. It also keeps the raid5 knowledge firmly in the raid5 code base. So it seems to keep the complexity under control

Would this approach make sense to you? Is there something really important I have missed?

(I'll try and be more responsive next time).

Thanks,
NeilBrown

#53 lore raw
Quoted 23 lines · Neil Brown
Quoted 7 lines · On Monday September 11, [email protected] wrote:

Neil,

The following patches implement hardware accelerated raid5 for the Intel Xscale(r) series of I/O Processors. The MD changes allow stripe operations to run outside the spin lock in a work queue. Hardware acceleration is achieved by using a dma-engine-aware work queue routine instead of the default software only routine.

Hi Dan,
Sorry for the delay in replying.
I've looked through these patches at last (mostly the raid-specific bits) and while there is clearly a lot of good stuff here, it does 'feel' right - it just seems too complex.

The particular issues that stand out to me are:
- 33 new STRIPE_OP_* flags. I'm sure there doesn't need to be that many new flags.
- the "raid5 dma client" patch moves far too much internal knowledge about raid5 into drivers/dma.

Quoted 3 lines

Clearly there are some complex issues being dealt with and some complexity is to be expected, but I feel there must be room for some serious simplification.

A valid criticism. There was definitely a push to just get it functional, so I can now see how the complexity crept into the implementation. The primary cause was the choice to explicitly handle channel switching in raid5-dma. However, relieving "client" code from this responsibility is something I am taking care of in the async api changes.

Quoted 147 lines

Let me try to describe how I envisage it might work.

As you know, the theory-of-operation of handle_stripe is that it assesses the state of a stripe deciding what actions to perform and then performs them. Synchronous actions (e.g. current parity calcs) are performed 'in-line'. Async actions (reads, writes) and actions that cannot be performed under a spinlock (->b_end_io) are recorded as being needed and then are initiated at the end of handle_stripe outside of the sh->lock.

The proposal is to bring the parity and other bulk-memory operations out of the spinlock and make them optionally asynchronous.

The set of tasks that might be needed to be performed on a stripe are:
Clear a target cache block
pre-xor various cache blocks into a target
copy data out of bios into cache blocks. (drain) post-xor various cache blocks into a target copy data into bios out of cache blocks (fill) test if a cache block is all zeros
start a read on a cache block
start a write on a cache block

(There is also a memcpy when expanding raid5. I think I would try to
simply avoid that copy and move pointers around instead).

Some of these steps require sequencing. e.g.
clear, pre-xor, copy, post-xor, write
for a rwm cycle.
We could require handle_stripe to be called again for each step. i.e. first call just clears the target and flags it as clear. Next call initiates the pre-xor and flags that as done. Etc. However I think that would make the non-offloaded case too slow, or at least too clumsy.

So instead we set flags to say what needs to be done and have a workqueue system that does it.

(so far this is all quite similar to what you have done.)

So handle_stripe would set various flag and other things (like identify which block was the 'target' block) and run the following in a workqueue:

raid5_do_stuff(struct stripe_head *sh)

{
        raid5_cont_t *conf = sh->raid_conf;

        if (test_bit(CLEAR_TARGET, &sh->ops.pending)) {
                struct page = *p->sh->dev[sh->ops.target].page;
                rv = async_memset(p, 0, 0, PAGE_SIZE, ops_done, sh);
                if (rv != BUSY)
                        clear_bit(CLEAR_TARGET, &sh->ops.pending);
                if (rv != COMPLETE)
                        goto out;
        }

        while (test_bit(PRE_XOR, &sh->ops.pending)) {
                struct page *plist[XOR_MAX];
                int offset[XOR_MAX];
                int pos = 0;
                int d;

                for (d = sh->ops.nextdev;
                     d < conf->raid_disks && pos < XOR_MAX ;
                     d++) {
                        if (sh->ops.nextdev == sh->ops.target)
                                continue;
                        if (!test_bit(R5_WantPreXor, &sh->dev[d].flags))
                                continue;
                        plist[pos] = sh->dev[d].page;
                        offset[pos++] = 0;
                }
                if (pos) {
                        struct page *p = sh->dev[sh->ops.target].page;
                        rv = async_xor(p, 0, plist, offset, pos, PAGE_SIZE,
                                       ops_done, sh);
                        if (rv != BUSY)
                                sh->ops.nextdev = d;
                        if (rv != COMPLETE)
                                goto out;
                } else {
                        clear_bit(PRE_XOR, &sh->ops.pending);
                        sh->ops.nextdev = 0;
                }
        }

        while (test_bit(COPY_IN, &sh0>ops.pending)) {
                ...
        }
        ....

        if (test_bit(START_IO, &sh->ops.pending)) {
                int d;
                for (d = 0 ; d < conf->raid_disk ; d++) {
                        /* all that code from the end of handle_stripe */
                }

        release_stripe(conf, sh);
        return;

out:

        if (rv == BUSY) {
                /* wait on something and try again ???*/
        }
        return;
}

ops_done(struct stripe_head *sh)

{
        queue_work(....whatever..);
}

Things to note:
- we keep track of where we are up to in sh->ops. .pending is flags saying what is left to be done .next_dev is the next device to process for operations that work on several devices
.next_bio, .next_iov will be needed for copy operations that
cross multiple bios and iovecs.

- Each sh->dev has R5_Want flags reflecting which multi-device operations are wanted on each device.

- async bulk-memory operations take pages, offsets, and lengths, and can return COMPLETE (if the operation was performed synchronously) IN_PROGRESS (if it has been started, or at least queued) or BUSY if it couldn't even be queued. Exactly what to do in that case I'm not sure. Probably we need a waitqueue to wait on.

- The interface between the client and the ADMA hardware is a collection of async_ functions. async_memcpy, async_xor, async_memset etc.

I gather there needs to be some understanding
about whether the pages are already appropriately mapped for DMA or whether a mapping is needed. Maybe an extra flag argument should be passed.

I imagine that any piece of ADMA hardware would register with the 'async_*' subsystem, and a call to async_X would be routed as appropriate, or be run in-line.

Quoted 5 lines

This approach introduces 8 flags for sh->ops.pending and maybe two or three new R5_Want* flags. It also keeps the raid5 knowledge firmly in the raid5 code base. So it seems to keep the complexity under control

Would this approach make sense to you?

Definitely.

Quoted 1 line

Is there something really important I have missed?

No, nothing important jumps out. Just a follow up question/note about the details.

You imply that the async path and the sync path are unified in this implementation. I think it is doable but it will add some complexity since the sync case is not a distinct subset of the async case. For example "Clear a target cache block" is required for the sync case, but it can go away when using hardware engines. Engines typically have their own accumulator buffer to store the temporary result, whereas software only operates on memory.

What do you think of adding async tests for these situations? test_bit(XOR, &conf->async)

Where a flag is set if calls to async_<operation> may be routed to hardware engine? Otherwise skip any async specific details.

Quoted 2 lines

(I'll try and be more responsive next time).

Thanks for shepherding this along.

Quoted 3 lines

Thanks,
NeilBrown

Regards,
Dan

#54 lore raw

[dropped akpm from the Cc: as current discussion isn't directly relevant to him]

Quoted 14 lines · On Tuesday October 10, [email protected] wrote:
Quoted 1 line · Neil Brown

Is there something really important I have missed?

No, nothing important jumps out. Just a follow up question/note about the details.

You imply that the async path and the sync path are unified in this implementation. I think it is doable but it will add some complexity since the sync case is not a distinct subset of the async case. For example "Clear a target cache block" is required for the sync case, but it can go away when using hardware engines. Engines typically have their own accumulator buffer to store the temporary result, whereas software only operates on memory.

Quoted 5 lines

What do you think of adding async tests for these situations? test_bit(XOR, &conf->async)

Where a flag is set if calls to async_<operation> may be routed to hardware engine? Otherwise skip any async specific details.

I'd rather try to come up with an interface that was equally appropriate to both offload and inline. I appreciate that it might not be possible to get an interface that gets best performance out of both, but I'd like to explore that direction first.

I'd guess from what you say that the dma engine is given a bunch of sources and a destination and it xor's all the sources together into an accumulation buffer, and then writes the accum buffer to the destination. Would that be right? Can you use the destination as one of the sources?

That can obviously be done inline too with some changes to the xor code, and avoiding the initial memset might be good for performance too.

So I would suggest we drop the memset idea, and define the async_xor interface to xor a number of sources into a destination, where the destination is allowed to be the same as the first source, but doesn't need to be.
Then the inline version could use a memset followed by the current xor operations, or could use newly written xor operations, and the offload version could equally do whatever is appropriate.

Another place where combining operations might make sense is copy-in and post-xor. In some cases it might be more efficient to only read the source once, and both write it to the destination and xor it into the target. Would your DMA engine be able to optimise this combination? I think current processors could certainly do better if the two were combined.

So there is definitely room to move, but would rather avoid flags if I could.

NeilBrown