backport of sba sg list management to ccio-dma

A long time ago (a couple months?) I merged large swaths of ccio and sba code... unfortunately, it caused an almost measurable (~2-5%) loss of performance on networking and scsi throughput...

I probably lost the code when my j6000 disk failed, but I'll probably have another crack at it someday.

Anyway, the patch looks good, but it needs a bit of testing, so I think we'll wait for 2.6.25 to merge it?

Regards,
Kyle
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Hello Kyle,

Hppay that you finaly receive it ;-)
(btw sorry for boucing)

On network it's sure this one is a measarable improvement ;-)
Otoh for scsi I mainly record a improvement of stability (but I will try more measure)

too bad

hope so.

I hope too more testing, I can just test on my d380 (my c110 seems to be in bad status: after each reboot it makes a fsck and lost many files :_( ???)

Tx,

	J.

_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

I've looked over the code, and it looks fine to me. Will wait for an ack

Cheers and thanks,
Kyle
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

We both have access to the FtC test ring - but I think we are also both too lazy to build a kernel and test it on a remote box. :)

In general I'm ok with this code since I trust Joel is testing it with both NIC and storage devices and I'm too lazy to.

Some comments/requests/ideas below but nothing critical.

Apologies that I'm cut/pasting from the mailing list archive:

    http://lists.parisc-linux.org/pipermail/parisc-linux/2007-October/031998.html

I didn't have the original mail in my inbox anymore.

+static CCIO_INLINE int
+get_iovp_order(unsigned long size)
+{
+ int order;
+
+ size = (size - 1) >> (IOVP_SHIFT - 1);
+ order = -1;
+ do {
+ size >>= 1;
+ order++;
+ } while (size);
+ return order;
+}

Would this be faster if written to use "ffs()"?

...
+#define ROUNDUP(x, y) ((x + ((y)-1)) & ~((y)-1))

ROUNDUP got replace in SBA with ALIGN(). Please use ALIGN instead.

...
+ } else {
+ /*
+ ** Search the resource bit map on well-aligned values.
+ ** "o" is the alignment.
+ ** We need the alignment to invalidate I/O TLB using
+ ** xxx HW features in the unmap path.
+ */
+ unsigned long o = 1 << get_iovp_order(pages_wanted << IOVP_SHIFT);
+ uint bitshiftcnt = ROUNDUP(ioc->res_bitshift, o);
+ unsigned long mask;

CCIO doesn't have this feature AFAIK. The comment only applies to SBA. Please remove the second sentence of the comment.
Searching on "well-aligned" values is still a good idea.

NOTE: Each 8 consecutive CCIO IO PDIR mappings land on the same IO TLB entry.

To avoid thrashing the IO TLB, ideally CCIO code will do the following:

o map_sg (ie storage devices) use 8 bit alignment (ie each mapping start on a new byte boundary). Traditional block storage can complete the IOs out of order and in parallel (via different controllers). This would thrash one IO TLB entry if 8 seperate 4K DMA mappings are inflight at the same time.

o map_single() (ie NICs) to just use the next available bit for a given PCI device. NIC DMA patterns (at least for a given stream) are linear. The ccio_map_single() should track the last bitmap octet used for both DMA_TO_DEVICE and FROM_DEVICE for each PCI device.
Once that octet is "used", find the next 8 bit field in the bitmap that is available.

For a single NIC and one or two SCSI controllers, this probably doesn't make much difference unless you start measuring latency. Throughput should be nearly the same.

My guess is the biggest perf gain will come from reducing the number of "sync" ops by implementing DELAYED_RESOURCE_CNT.

...
#ifdef CCIO_SEARCH_TIME
+ udelay(100);

Did you want to keep udelay() call?

...
+#else /* DELAYED_RESOURCE_CNT == 0 */
+
+ ccio_free_range(ioc, iova, size);
+
+ /* force fdc's to be visible now */
+ asm volatile("sync" :: );
+
+#endif /* DELAYED_RESOURCE_CNT == 0 */

Does the asm("sync") need to be outside the #endif ? Please make sure "sync" is called exactly once IFF the IO PDIR is modified.

...
+#ifdef CCIO_MAP_STATS
+ ioc->usg_pages += sg_dma_len(sglist) >> IOVP_SHIFT;
+ ioc->usingle_calls--; /* kluge since call is unmap_sg() */
+#endif

I wouldn't add MAP_STATS here.
They aren't enabled in SBA becuase they impact performance too much. I expect that will always be true and would seriously consider removing MAP_STATS code from SBA as well. Timing the bitmap search is probably the only critical bit of info that really matters. And that's for developement/testing only.

hth,
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Grant,

Just come back here because I don't understand this strange stuff:

Grant Grundler wrote:
[snip]

here is short test:
without udelay:

# scp vmlinux  jso@patst007:/var/tmp/vmlinux-2.6.23-pa-d32up3
Password:
vmlinux                                                                                    100% 4364KB 101.5KB/s   00:43

with udelay:

# scp vmlinux  jso@patst007:/var/tmp/vmlinux-2.6.23-pa-d32up3
Password:
vmlinux                                                                                    100% 4364KB 436.4KB/s   00:10

well still far from what we could expect (about 800KB/s) but fwiw something supposing slowdown the kernel improve nic perf?

_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

For testing NIC perf, please don't start with scp. Use ftp or something else that doesn't use encryption. See www.netperf.org and/or iperf.

Yeah, that happens and it's clearly something wrong with the code. Suggests there is contention either between CPU and IOTLB or between CPU and IO device for a cacheline(s).

BTW, I'm assuming you have DELAYED_RESOURCE_CNT set to 16 or more.

thanks,
grant
_______________________________________________
parisc-linux mailing list
[email protected]
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux