Skip to contentSkip to conversation

ioatdma: ring buffer management updates

PATCH In review Dan Williams 2 participants 4 messages May 11, 2010 – May 13, 2010 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover ioatdma: ring buffer management updates

Two patches targeted at the next merge window affecting the ioatdma driver (used when NET_DMA and/or ASYNC_TX_DMA+MD_RAID456 are enabled). According to perf the split locking update improves cpu utilization by a few percentage points.

---

Dan Williams (2):
ioat2,3: convert to producer/consumer locking
ioat: convert to circ_buf

 drivers/dma/ioat/dma.h    |    1 
 drivers/dma/ioat/dma_v2.c |  184 +++++++++++++++++++++++----------------------
 drivers/dma/ioat/dma_v2.h |   33 +++-----
 drivers/dma/ioat/dma_v3.c |  117 +++++++++--------------------
 4 files changed, 142 insertions(+), 193 deletions(-)
#1 lore raw

PATCH 1/2 ioat: convert to circ_buf

Use the common power-of-2 circular buffer macros.

---

 drivers/dma/ioat/dma_v2.c |    2 +-
 drivers/dma/ioat/dma_v2.h |   18 +++++++-----------
 2 files changed, 8 insertions(+), 12 deletions(-)
drivers/dma/ioat/dma_v2.c +1 -1
index b5ae56c..b6699a3 100644
@@ -553,7 +553,7 @@ bool reshape_ring(struct ioat2_dma_chan *ioat, int order)
 	 */
 	struct ioat_chan_common *chan = &ioat->base;
 	struct dma_chan *c = &chan->common;
-	const u16 curr_size = ioat2_ring_mask(ioat) + 1;
+	const u16 curr_size = ioat2_ring_size(ioat);
 	const u16 active = ioat2_ring_active(ioat);
 	const u16 new_size = 1 << order;
 	struct ioat_ring_ent **ring;
drivers/dma/ioat/dma_v2.h +7 -11
index ef2871f..d7b64f1 100644
@@ -22,6 +22,7 @@
 #define IOATDMA_V2_H
 
 #include <linux/dmaengine.h>
+#include <linux/circ_buf.h>
 #include "dma.h"
 #include "hw.h"
 
@@ -71,31 +72,26 @@ static inline struct ioat2_dma_chan *to_ioat2_chan(struct dma_chan *c)
 	return container_of(chan, struct ioat2_dma_chan, base);
 }
 
-static inline u16 ioat2_ring_mask(struct ioat2_dma_chan *ioat)
+static inline u16 ioat2_ring_size(struct ioat2_dma_chan *ioat)
 {
-	return (1 << ioat->alloc_order) - 1;
+	return 1 << ioat->alloc_order;
 }
 
 /* count of descriptors in flight with the engine */
 static inline u16 ioat2_ring_active(struct ioat2_dma_chan *ioat)
 {
-	return (ioat->head - ioat->tail) & ioat2_ring_mask(ioat);
+	return CIRC_CNT(ioat->head, ioat->tail, ioat2_ring_size(ioat));
 }
 
 /* count of descriptors pending submission to hardware */
 static inline u16 ioat2_ring_pending(struct ioat2_dma_chan *ioat)
 {
-	return (ioat->head - ioat->issued) & ioat2_ring_mask(ioat);
+	return CIRC_CNT(ioat->head, ioat->issued, ioat2_ring_size(ioat));
 }
 
 static inline u16 ioat2_ring_space(struct ioat2_dma_chan *ioat)
 {
-	u16 num_descs = ioat2_ring_mask(ioat) + 1;
-	u16 active = ioat2_ring_active(ioat);
-
-	BUG_ON(active > num_descs);
-
-	return num_descs - active;
+	return ioat2_ring_size(ioat) - ioat2_ring_active(ioat);
 }
 
 /* assumes caller already checked space */
@@ -151,7 +147,7 @@ struct ioat_ring_ent {
 static inline struct ioat_ring_ent *
 ioat2_get_ring_ent(struct ioat2_dma_chan *ioat, u16 idx)
 {
-	return ioat->ring[idx & ioat2_ring_mask(ioat)];
+	return ioat->ring[idx & (ioat2_ring_size(ioat) - 1)];
 }
 
 static inline void ioat2_set_chainaddr(struct ioat2_dma_chan *ioat, u64 addr)

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

Re: [PATCH 2/2] ioat2,3: convert to producer/consumer locking

Out of interest, does it make the code smaller if you mark ioat2_get_ring_ent() and ioat2_ring_mask() with __attribute_const__?

I'm not sure whether it'll affect how long gcc is willing to cache these, but once computed, I would guess they won't change within the calling function.

Also, is the device you're driving watching the ring and its indices? If so, does it modify the indices? If that is the case, you might need to use read_barrier_depends() rather than smp_read_barrier_depends().

Review

Quoted 5 lines
(quoted diff) +2 -0
+		prefetch(ioat2_get_ring_ent(ioat, idx + i + 1));
+		desc = ioat2_get_ring_ent(ioat, idx + i);
 		dump_desc_dbg(ioat, desc);
 		tx = &desc->txd;
 		if (tx->cookie) {

Is this right, I wonder? You're prefetching [i+1] before reading [i]? Doesn't this mean that you might have to wait for [i+1] to be retrieved from RAM before
[i] can be read? Should you instead read tx->cookie before issuing the prefetch? Admittedly, this is only likely to affect the reading of the head of the queue - subsequent reads in the same loop will, of course, have been prefetched.

David

#4 lore raw

Re: [PATCH 2/2] ioat2,3: convert to producer/consumer locking

Quoted 6 lines · David Howells

Out of interest, does it make the code smaller if you mark ioat2_get_ring_ent() and ioat2_ring_mask() with __attribute_const__?

I'm not sure whether it'll affect how long gcc is willing to cache these, but once computed, I would guess they won't change within the calling function.

Unfortunately, it does not make a difference, but I'll keep this in mind if ioat2_get_ring_ent() ever gets more complicated (which it might in the future).

Quoted 3 lines

Also, is the device you're driving watching the ring and its indices?  If so, does it modify the indices?  If that is the case, you might need to use read_barrier_depends() rather than smp_read_barrier_depends().

The device does not observe the indices directly. Instead we increment a free running 'count' register by the distance between ioat->pending and ioat->head.

Quoted 7 lines
Quoted 6 lines

+             prefetch(ioat2_get_ring_ent(ioat, idx + i + 1));
+             desc = ioat2_get_ring_ent(ioat, idx + i); dump_desc_dbg(ioat, desc);
              tx = &desc->txd;
              if (tx->cookie) {

Quoted 6 lines

Is this right, I wonder?  You're prefetching [i+1] before reading [i]?  Doesn't this mean that you might have to wait for [i+1] to be retrieved from RAM before
[i] can be read?  Should you instead read tx->cookie before issuing the prefetch?  Admittedly, this is only likely to affect the reading of the head of the queue - subsequent reads in the same loop will, of course, have been prefetched.

Yes, it should be the other way around.

Thanks!

--
Dan