Skip to contentSkip to conversation

ixgbe: Reduce memory consumption with larger page sizes

PATCH In review Anton Blanchard 2 participants 2 messages Oct 21, 2013 – Oct 22, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ixgbe: Reduce memory consumption with larger page sizes

The ixgbe driver allocates pages for its receive rings. It currently uses 512 pages, regardless of page size. During receive handling it adds the unused part of the page back into the rx ring, avoiding the need for a new allocation.

On a ppc64 box with 64 threads and 64kB pages, we end up with 512 entries * 64 rx queues * 64kB = 2GB memory used. Even more of a concern is that we use up 2GB of IOMMU space in order to map all this memory.

The driver makes a number of decisions based on if PAGE_SIZE is less than 8kB, so use this as the breakpoint and only allocate 128 entries on 8kB or larger page sizes.

---

Jeff: The breakpoint and the ring size I chose was pretty arbitrary, feel free to adjust as you see fit. Our main concern is we get that 2GB consumption down to something more reasonable :)

drivers/net/ethernet/intel/ixgbe/ixgbe.h +4 -0
Index: b/drivers/net/ethernet/intel/ixgbe/ixgbe.h
===================================================================
@@ -67,7 +67,11 @@
 #define IXGBE_MAX_TXD			   4096
 #define IXGBE_MIN_TXD			     64
 
+#if (PAGE_SIZE < 8192)
 #define IXGBE_DEFAULT_RXD		    512
+#else
+#define IXGBE_DEFAULT_RXD		    128
+#endif
 #define IXGBE_MAX_RXD			   4096
 #define IXGBE_MIN_RXD			     64
 
#1 lore raw
Quoted 14 lines · Anton Blanchard

The ixgbe driver allocates pages for its receive rings. It currently uses 512 pages, regardless of page size. During receive handling it adds the unused part of the page back into the rx ring, avoiding the need for a new allocation.

On a ppc64 box with 64 threads and 64kB pages, we end up with 512 entries * 64 rx queues * 64kB = 2GB memory used. Even more of a concern is that we use up 2GB of IOMMU space in order to map all this memory.

The driver makes a number of decisions based on if PAGE_SIZE is less than 8kB, so use this as the breakpoint and only allocate 128 entries on 8kB or larger page sizes.

Quoted 7 lines

---

Jeff: The breakpoint and the ring size I chose was pretty arbitrary, feel free to adjust as you see fit. Our main concern is we get that 2GB
consumption down to something more reasonable :)

Thanks Anton, I will add your patch to my queue.