Skip to contentSkip to conversation

scatterlist: add sg_alloc_table_append function

PATCH Superseded Leon Romanovsky 2 participants 11 messages Sep 10, 2020 – Sep 16, 2020

Subsystems (2)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover scatterlist: add sg_alloc_table_append function

From: Leon Romanovsky <[email protected]>

Changelog:
v1:
* Changed _sg_chain to be __sg_chain
* Added dependency on ARCH_NO_SG_CHAIN
* Removed struct sg_append
v0:
* https://lore.kernel.org/lkml/[email protected]

--------------------------------------------------------------------------
From Maor:

This series adds a new constructor for a scatter gather table. Like sg_alloc_table_from_pages function, this function merges all contiguous chunks of the pages a into single scatter gather entry.

In contrast to sg_alloc_table_from_pages, the new API allows chaining of new pages to already initialized SG table.

This allows drivers to utilize the optimization of merging contiguous pages without a need to pre allocate all the pages and hold them in a very large temporary buffer prior to the call to SG table initialization.

The first two patches refactor the code of sg_alloc_table_from_pages in order to have code sharing and add sg_alloc_next function to allow dynamic allocation of more entries in the SG table.

The third patch introduces the new API.

The last patch changes the Infiniband driver to use the new API. It removes duplicate functionality from the code and benefits the optimization of allocating dynamic SG table from pages.

In huge pages system of 2MB page size, without this change, the SG table would contain x512 SG entries.
E.g. for 100GB memory registration:

Number of entries Size
Before 26214400 600.0MB
After 51200 1.2MB

Thanks

Maor Gottlieb (4):
lib/scatterlist: Refactor sg_alloc_table_from_pages lib/scatterlist: Add support in dynamically allocation of SG entries lib/scatterlist: Add support in dynamic allocation of SG table from
pages
RDMA/umem: Move to allocate SG table from pages

 drivers/infiniband/Kconfig     |   1 +
 drivers/infiniband/core/umem.c |  90 ++--------
 include/linux/scatterlist.h    |  35 ++--
 lib/scatterlist.c              | 292 +++++++++++++++++++++++++--------
 4 files changed, 255 insertions(+), 163 deletions(-)

--
2.26.2

#1 lore raw

PATCH 1/4 lib/scatterlist: Refactor sg_alloc_table_from_pages

From: Maor Gottlieb <[email protected]>

Currently, sg_alloc_table_from_pages doesn't support dynamic chaining of SG entries. Therefore it requires from user to allocate all the pages in advance and hold them in a large buffer. Such a buffer consumes a lot of temporary memory in HPC systems which do a very large memory registration.

The next patches introduce API for dynamically allocation from pages and it requires us to do the following:
* Extract the code to alloc_from_pages_common.
* Change the build of the table to iterate on the chunks and not on the SGEs. It will allow dynamic allocation of more SGEs.

Since sg_alloc_table_from_pages allocate exactly the number of chunks, therefore chunks are equal to the number of SG entries.

---
 lib/scatterlist.c | 75 ++++++++++++++++++++++++++++-------------------
 1 file changed, 45 insertions(+), 30 deletions(-)
lib/scatterlist.c +45 -30
index 5d63a8857f36..292e785d21ee 100644
@@ -365,38 +365,18 @@ int sg_alloc_table(struct sg_table *table, unsigned int nents, gfp_t gfp_mask)
 }
 EXPORT_SYMBOL(sg_alloc_table);
 
-/**
- * __sg_alloc_table_from_pages - Allocate and initialize an sg table from
- *			         an array of pages
- * @sgt:	 The sg table header to use
- * @pages:	 Pointer to an array of page pointers
- * @n_pages:	 Number of pages in the pages array
- * @offset:      Offset from start of the first page to the start of a buffer
- * @size:        Number of valid bytes in the buffer (after offset)
- * @max_segment: Maximum size of a scatterlist node in bytes (page aligned)
- * @gfp_mask:	 GFP allocation mask
- *
- *  Description:
- *    Allocate and initialize an sg table from a list of pages. Contiguous
- *    ranges of the pages are squashed into a single scatterlist node up to the
- *    maximum size specified in @max_segment. An user may provide an offset at a
- *    start and a size of valid data in a buffer specified by the page array.
- *    The returned sg table is released by sg_free_table.
- *
- * Returns:
- *   0 on success, negative error on failure
- */
-int __sg_alloc_table_from_pages(struct sg_table *sgt, struct page **pages,
-				unsigned int n_pages, unsigned int offset,
-				unsigned long size, unsigned int max_segment,
-				gfp_t gfp_mask)
+static struct scatterlist *
+alloc_from_pages_common(struct sg_table *sgt, struct page **pages,
+			unsigned int n_pages, unsigned int offset,
+			unsigned long size, unsigned int max_segment,
+			gfp_t gfp_mask)
 {
 	unsigned int chunks, cur_page, seg_len, i;
+	struct scatterlist *prv, *s = NULL;
 	int ret;
-	struct scatterlist *s;
 
 	if (WARN_ON(!max_segment || offset_in_page(max_segment)))
-		return -EINVAL;
+		return ERR_PTR(-EINVAL);
 
 	/* compute number of contiguous chunks */
 	chunks = 1;
@@ -412,11 +392,12 @@ int __sg_alloc_table_from_pages(struct sg_table *sgt, struct page **pages,
 
 	ret = sg_alloc_table(sgt, chunks, gfp_mask);
 	if (unlikely(ret))
-		return ret;
+		return ERR_PTR(ret);
 
 	/* merging chunks and putting them into the scatterlist */
 	cur_page = 0;
-	for_each_sg(sgt->sgl, s, sgt->orig_nents, i) {
+	s = sgt->sgl;
+	for (i = 0; i < chunks; i++) {
 		unsigned int j, chunk_size;
 
 		/* look for the end of the current chunk */
@@ -435,9 +416,43 @@ int __sg_alloc_table_from_pages(struct sg_table *sgt, struct page **pages,
 		size -= chunk_size;
 		offset = 0;
 		cur_page = j;
+		prv = s;
+		s = sg_next(s);
 	}
+	return prv;
+}
 
-	return 0;
+/**
+ * __sg_alloc_table_from_pages - Allocate and initialize an sg table from
+ *			         an array of pages
+ * @sgt:	 The sg table header to use
+ * @pages:	 Pointer to an array of page pointers
+ * @n_pages:	 Number of pages in the pages array
+ * @offset:      Offset from start of the first page to the start of a buffer
+ * @size:        Number of valid bytes in the buffer (after offset)
+ * @max_segment: Maximum size of a scatterlist node in bytes (page aligned)
+ * @gfp_mask:	 GFP allocation mask
+ *
+ *  Description:
+ *    Allocate and initialize an sg table from a list of pages. Contiguous
+ *    ranges of the pages are squashed into a single scatterlist node up to the
+ *    maximum size specified in @max_segment. A user may provide an offset at a
+ *    start and a size of valid data in a buffer specified by the page array.
+ *    The returned sg table is released by sg_free_table.
+ *
+ * Returns:
+ *   0 on success, negative error on failure
+ */
+int __sg_alloc_table_from_pages(struct sg_table *sgt, struct page **pages,
+				unsigned int n_pages, unsigned int offset,
+				unsigned long size, unsigned int max_segment,
+				gfp_t gfp_mask)
+{
+	struct scatterlist *sg;
+
+	sg = alloc_from_pages_common(sgt, pages, n_pages, offset, size,
+				     max_segment, gfp_mask);
+	return PTR_ERR_OR_ZERO(sg);
 }
 EXPORT_SYMBOL(__sg_alloc_table_from_pages);
 

--
2.26.2

#2 lore raw

Re: [PATCH rdma-next v1 1/4] lib/scatterlist: Refactor sg_alloc_table_from_pages

Quoted 13 lines · Leon Romanovsky

From: Maor Gottlieb <[email protected]>

Currently, sg_alloc_table_from_pages doesn't support dynamic chaining of SG entries. Therefore it requires from user to allocate all the pages in advance and hold them in a large buffer. Such a buffer consumes a lot of temporary memory in HPC systems which do a very large memory registration.

The next patches introduce API for dynamically allocation from pages and it requires us to do the following:
* Extract the code to alloc_from_pages_common.
* Change the build of the table to iterate on the chunks and not on the SGEs. It will allow dynamic allocation of more SGEs.

Quoted 5 lines

Since sg_alloc_table_from_pages allocate exactly the number of chunks, therefore chunks are equal to the number of SG entries.

I really don't think this refactoring on its own adds any value, it just makes reading the rest of the series harder.

(functionally it looks correct, though)

#3 lore raw

Re: [PATCH rdma-next v1 1/4] lib/scatterlist: Refactor sg_alloc_table_from_pages

Quoted 14 lines · Christoph Hellwig
Quoted 13 lines · Leon Romanovsky

From: Maor Gottlieb <[email protected]>

Currently, sg_alloc_table_from_pages doesn't support dynamic chaining of SG entries. Therefore it requires from user to allocate all the pages in advance and hold them in a large buffer. Such a buffer consumes a lot of temporary memory in HPC systems which do a very large memory registration.

The next patches introduce API for dynamically allocation from pages and it requires us to do the following:
* Extract the code to alloc_from_pages_common.
* Change the build of the table to iterate on the chunks and not on the SGEs. It will allow dynamic allocation of more SGEs.

Quoted 8 lines
Quoted 5 lines

Since sg_alloc_table_from_pages allocate exactly the number of chunks, therefore chunks are equal to the number of SG entries.

I really don't think this refactoring on its own adds any value, it just makes reading the rest of the series harder.

We expected this type of the comment, but decided to send "splitted" version because it is much easier to squash patches instead of splitting them later.

We will squash and resubmit.

Quoted 2 lines

(functionally it looks correct, though)

Thanks

#4 lore raw

PATCH 2/4 lib/scatterlist: Add support in dynamically allocation of SG entries

In reply to Leon Romanovsky

From: Maor Gottlieb <[email protected]>

In order to support dynamic allocation of SG table, this patch introduces sg_alloc_next. This function should be called to add more entries to the table. In order to share the code, we will do the following:
* Extract the allocation code from __sg_alloc_table to sg_alloc.
* Add a function to chain SGE to the next page.

---
 include/linux/scatterlist.h |  29 ++++++----
 lib/scatterlist.c           | 110 ++++++++++++++++++++++++------------
 2 files changed, 91 insertions(+), 48 deletions(-)
include/linux/scatterlist.h +17 -12
index 45cf7b69d852..9d13004334aa 100644
@@ -165,6 +165,22 @@ static inline void sg_set_buf(struct scatterlist *sg, const void *buf,
 #define for_each_sgtable_dma_sg(sgt, sg, i)	\
 	for_each_sg((sgt)->sgl, sg, (sgt)->nents, i)
 
+static inline void __sg_chain(struct scatterlist *chain_sg,
+			      struct scatterlist *sgl)
+{
+	/*
+	 * offset and length are unused for chain entry. Clear them.
+	 */
+	chain_sg->offset = 0;
+	chain_sg->length = 0;
+
+	/*
+	 * Set lowest bit to indicate a link pointer, and make sure to clear
+	 * the termination bit if it happens to be set.
+	 */
+	chain_sg->page_link = ((unsigned long) sgl | SG_CHAIN) & ~SG_END;
+}
+
 /**
  * sg_chain - Chain two sglists together
  * @prv:	First scatterlist
@@ -178,18 +194,7 @@ static inline void sg_set_buf(struct scatterlist *sg, const void *buf,
 static inline void sg_chain(struct scatterlist *prv, unsigned int prv_nents,
 			    struct scatterlist *sgl)
 {
-	/*
-	 * offset and length are unused for chain entry.  Clear them.
-	 */
-	prv[prv_nents - 1].offset = 0;
-	prv[prv_nents - 1].length = 0;
-
-	/*
-	 * Set lowest bit to indicate a link pointer, and make sure to clear
-	 * the termination bit if it happens to be set.
-	 */
-	prv[prv_nents - 1].page_link = ((unsigned long) sgl | SG_CHAIN)
-					& ~SG_END;
+	__sg_chain(&prv[prv_nents - 1], sgl);
 }
 
 /**
lib/scatterlist.c +74 -36

142 lines not on this page · raw message

--
2.26.2

#5 lore raw

Re: [PATCH rdma-next v1 2/4] lib/scatterlist: Add support in dynamically allocation of SG entries

lib/scatterlist.c int __sg_alloc_table(struct sg_table *table, unsigned int nents, · line 308

Quoted 3 lines
lib/scatterlist.c +3 -0
+			/* We decrease one since the prvious last sge in used to
+			 * chainning.
+			 */

The normal style would be:

			/*
			 * We decrease one since the prvious last sge in used to
			 * chain the chunks together.
			 */

(also fixing up what I think it should be saying while I'm at it)

lib/scatterlist.c int __sg_alloc_table(struct sg_table *table, unsigned int nents, · line 348

Quoted 11 lines

+ * Thus if @nents is bigger than @max_ents, the scatterlists will be
+ * chained in units of @max_ents.
+ *
+ **/

lib/scatterlist.c +7 -0
+static int sg_alloc_next(struct sg_table *table, struct scatterlist *last,
+			 unsigned int nents, unsigned int max_ents,
+			 gfp_t gfp_mask)
+{
+	return sg_alloc(table, last, nents, max_ents, NULL, 0, gfp_mask,
+			sg_kmalloc);
+}

This helper seems unused in this patch. For bisection you probably want to move it into the next patch with the user.

In fact I'm not even sure there is much of a point in splitting out this patch either.

#6 lore raw

Re: [PATCH rdma-next v1 2/4] lib/scatterlist: Add support in dynamically allocation of SG entries

Quoted 5 lines · Christoph Hellwig

lib/scatterlist.c int __sg_alloc_table(struct sg_table *table, unsigned int nents, · line 308

Quoted 3 lines
lib/scatterlist.c +3 -0
+			/* We decrease one since the prvious last sge in used to
+			 * chainning.
+			 */

The normal style would be:

It is netdev style of formatting, will change.

Quoted 23 lines
			/*
			 * We decrease one since the prvious last sge in used to
			 * chain the chunks together.
			 */

(also fixing up what I think it should be saying while I'm at it)

lib/scatterlist.c int __sg_alloc_table(struct sg_table *table, unsigned int nents, · line 348

Quoted 11 lines

+ * Thus if @nents is bigger than @max_ents, the scatterlists will be
+ * chained in units of @max_ents.
+ *
+ **/

lib/scatterlist.c +7 -0
+static int sg_alloc_next(struct sg_table *table, struct scatterlist *last,
+			 unsigned int nents, unsigned int max_ents,
+			 gfp_t gfp_mask)
+{
+	return sg_alloc(table, last, nents, max_ents, NULL, 0, gfp_mask,
+			sg_kmalloc);
+}

This helper seems unused in this patch. For bisection you probably want to move it into the next patch with the user.

Quoted 2 lines

In fact I'm not even sure there is much of a point in splitting out this patch either.

We will squash.

#7 lore raw

PATCH 3/4 lib/scatterlist: Add support in dynamic allocation of SG table from pages

In reply to Leon Romanovsky

From: Maor Gottlieb <[email protected]>

Add an API that supports dynamic allocation of the SG table from pages, such function should be used by drivers that can't supply all the pages at one time.

This function returns the last populated sge in the table. Users should pass it as an argument to the function from the second call and forward. As for sg_alloc_table_from_pages, nents will be equal to the number of populated SGEs (chunks).

With this new API, drivers can benefit the optimization of merging contiguous pages without a need to allocate all pages in advance and hold them in a large buffer.

E.g. with the Infiniband driver that allocates a single page for hold the pages. For 1TB memory registration, the temporary buffer would consume only 4KB, instead of 2GB.

---
 include/linux/scatterlist.h |   6 ++
 lib/scatterlist.c           | 131 +++++++++++++++++++++++++++++++-----
 2 files changed, 119 insertions(+), 18 deletions(-)
include/linux/scatterlist.h +6 -0
index 9d13004334aa..9fcc5783dfbc 100644
@@ -291,6 +291,12 @@ void sg_free_table(struct sg_table *);
 int __sg_alloc_table(struct sg_table *, unsigned int, unsigned int,
 		     struct scatterlist *, unsigned int, gfp_t, sg_alloc_fn *);
 int sg_alloc_table(struct sg_table *, unsigned int, gfp_t);
+#ifndef CONFIG_ARCH_NO_SG_CHAIN
+struct scatterlist *sg_alloc_table_append(
+	struct sg_table *sgt, struct page **pages, unsigned int n_pages,
+	unsigned int offset, unsigned long size, unsigned int max_segment,
+	gfp_t gfp_mask, struct scatterlist *prv, unsigned int left_pages);
+#endif
 int __sg_alloc_table_from_pages(struct sg_table *sgt, struct page **pages,
 				unsigned int n_pages, unsigned int offset,
 				unsigned long size, unsigned int max_segment,
lib/scatterlist.c +113 -18

174 lines not on this page · raw message

--
2.26.2

#8 lore raw

Re: [PATCH rdma-next v1 3/4] lib/scatterlist: Add support in dynamic allocation of SG table from pages

include/linux/scatterlist.h void sg_free_table(struct sg_table *); · line 299

Quoted 6 lines
include/linux/scatterlist.h +6 -0
+#ifndef CONFIG_ARCH_NO_SG_CHAIN
+struct scatterlist *sg_alloc_table_append(
+	struct sg_table *sgt, struct page **pages, unsigned int n_pages,
+	unsigned int offset, unsigned long size, unsigned int max_segment,
+	gfp_t gfp_mask, struct scatterlist *prv, unsigned int left_pages);
+#endif

Odd indentation here, we either do two tabs (my preference) or aligned to the opening brace (what you seem to be doing elsewhere in the series).

lib/scatterlist.c int sg_alloc_table(struct sg_table *table, unsigned int nents, gfp_t gfp_mask) · line 426

Quoted 9 lines
lib/scatterlist.c +9 -0
+	/* Check if last entry should be keeped for chainning */
+	next_sg = sg_next(prv);
+	if (!sg_is_last(next_sg) || left_npages == 1)
+		return next_sg;
+
+	ret = sg_alloc_next(table, next_sg,
+			    min_t(unsigned long, left_npages,
+				  SG_MAX_SINGLE_ALLOC),
+			    SG_MAX_SINGLE_ALLOC, gfp_mask);

Do we even need the sg_alloc_next helper added in the last patch, given that this fairly simple function is the only caller?

lib/scatterlist.c int sg_alloc_table(struct sg_table *table, unsigned int nents, gfp_t gfp_mask) · line 435

Quoted 4 lines
lib/scatterlist.c +4 -0
+static struct scatterlist *alloc_from_pages_common(
+	struct sg_table *sgt, struct page **pages, unsigned int n_pages,
+	unsigned int offset, unsigned long size, unsigned int max_segment,
+	gfp_t gfp_mask, struct scatterlist *prv, unsigned int left_pages)

Same strange one tab indent as above.

Quoted 23 lines
lib/scatterlist.c +23 -0
+#ifndef CONFIG_ARCH_NO_SG_CHAIN
+/**
+ * sg_alloc_table_append - Allocate and initialize an sg table from
+ *                         an array of pages
+ * @sgt:	 The sg table header to use
+ * @pages:	 Pointer to an array of page pointers
+ * @n_pages:	 Number of pages in the pages array
+ * @offset:      Offset from start of the first page to the start of a buffer
+ * @size:        Number of valid bytes in the buffer (after offset)
+ * @max_segment: Maximum size of a scatterlist node in bytes (page aligned)
+ * @gfp_mask:	 GFP allocation mask
+ * @prv:	 Last populated sge in sgt
+ * @left_pages:  Left pages caller have to set after this call
+ *
+ *  Description:
+ *    If @prv is NULL, it allocates and initialize an sg table from a list of
+ *    pages. Contiguous ranges of the pages are squashed into a single
+ *    scatterlist node up to the maximum size specified in @max_segment. A user
+ *    may provide an offset at a start and a size of valid data in a buffer
+ *    specified by the page array. A user may provide @append to chain pages
+ *    to last entry in sgt. The returned sg table is released by sg_free_table.
+ *
+ * Returns:

lib/scatterlist.c alloc_from_pages_common(struct sg_table *sgt, struct page **pages, · line 550

Quoted 10 lines
lib/scatterlist.c +10 -0
+ *   Last SGE in sgt on success, negative error on failure.
+ *
+ * Notes:
+ *   If this function returns non-0 (eg failure), the caller must call
+ *   sg_free_table() to cleanup any leftover allocations.
+ */
+struct scatterlist *sg_alloc_table_append(
+	struct sg_table *sgt, struct page **pages, unsigned int n_pages,
+	unsigned int offset, unsigned long size, unsigned int max_segment,
+	gfp_t gfp_mask, struct scatterlist *prv, unsigned int left_pages)

One-tab indent again.

lib/scatterlist.c alloc_from_pages_common(struct sg_table *sgt, struct page **pages, · line 556

Quoted 6 lines
lib/scatterlist.c +6 -0
+{
+	return alloc_from_pages_common(sgt, pages, n_pages, offset, size,
+				       max_segment, gfp_mask, prv, left_pages);
+}
+EXPORT_SYMBOL_GPL(sg_alloc_table_append);
+#endif

So there reason I suggested to not provide sg_alloc_table_append if CONFIG_ARCH_NO_SG_CHAIN was set was to avoid the extra alloc_from_pages_common helper. It might be better to move to your run-time check and just make it conditіtional on a non-NULL prv pointer, which would allow us to merge alloc_from_pages_common into sg_alloc_table_append. Sorry for leading you down this path.

#9 lore raw

Re: [PATCH rdma-next v1 3/4] lib/scatterlist: Add support in dynamic allocation of SG table from pages

Quoted 9 lines · Christoph Hellwig

include/linux/scatterlist.h void sg_free_table(struct sg_table *); · line 299

Quoted 6 lines
include/linux/scatterlist.h +6 -0
+#ifndef CONFIG_ARCH_NO_SG_CHAIN
+struct scatterlist *sg_alloc_table_append(
+	struct sg_table *sgt, struct page **pages, unsigned int n_pages,
+	unsigned int offset, unsigned long size, unsigned int max_segment,
+	gfp_t gfp_mask, struct scatterlist *prv, unsigned int left_pages);
+#endif

Odd indentation here, we either do two tabs (my preference) or aligned to the opening brace (what you seem to be doing elsewhere in the series).

All indentation came from clang-formatter, we will change.

Thanks

#10 lore raw

PATCH 4/4 RDMA/umem: Move to allocate SG table from pages

In reply to Leon Romanovsky

From: Maor Gottlieb <[email protected]>

Remove the implementation of ib_umem_add_sg_table and instead call to sg_alloc_table_append which already has the logic to merge contiguous pages.

Besides that it removes duplicated functionality, it reduces the memory consumption of the SG table significantly. Prior to this patch, the SG table was allocated in advance regardless consideration of contiguous pages.

In huge pages system of 2MB page size, without this change, the SG table would contain x512 SG entries.
E.g. for 100GB memory registration:

Number of entries Size
Before 26214400 600.0MB
After 51200 1.2MB

---
 drivers/infiniband/Kconfig     |  1 +
 drivers/infiniband/core/umem.c | 90 +++++-----------------------------
 2 files changed, 12 insertions(+), 79 deletions(-)
drivers/infiniband/Kconfig +1 -0
index 91b023341b77..2f299c0be2e5 100644
@@ -6,6 +6,7 @@ menuconfig INFINIBAND
 	depends on INET
 	depends on m || IPV6 != m
 	depends on !ALPHA
+	depends on !ARCH_NO_SG_CHAIN
 	select IRQ_POLL
 	select DIMLIB
 	help
drivers/infiniband/core/umem.c +11 -79

123 lines not on this page · raw message

--
2.26.2