Skip to contentSkip to conversation

docs: net: page_pool: sync dev and kdoc

PATCH Applied Jakub Kicinski 6 participants 10 messages Aug 2, 2023 – Aug 3, 2023

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

v2 cover docs: net: page_pool: sync dev and kdoc

Document PP_FLAG_DMA_SYNC_DEV based on recent conversation. Use kdoc to document structs and functions, to avoid duplication.

Olek, this will conflict with your work, but I think that trying to make progress in parallel is the best course of action... Retargetting at net-next to make it a little less bad.

Jakub Kicinski (2):
docs: net: page_pool: document PP_FLAG_DMA_SYNC_DEV parameters
docs: net: page_pool: use kdoc to avoid duplicating the information

 Documentation/networking/page_pool.rst | 110 +++++++++-----------
 include/net/page_pool.h                | 134 +++++++++++++++++++------
 net/core/page_pool.c                   |  31 +++++-
 3 files changed, 181 insertions(+), 94 deletions(-)
Signature · 1 line
2.41.0
author maintainer
#1 lore raw

PATCH v2 1/2 docs: net: page_pool: document PP_FLAG_DMA_SYNC_DEV parameters

Using PP_FLAG_DMA_SYNC_DEV is a bit confusing. It was perhaps more obvious when it was introduced but the page pool use has grown beyond XDP and beyond packet-per-page so now making the heads and tails out of this feature is not trivial.

Obviously making the API more user friendly would be a better fix, but until someone steps up to do that let's at least document what the parameters are.

Relevant discussion in the first Link.

---
v2:
- s/sync'/sync/ and other fixes from Randy
v1: https://lore.kernel.org/all/[email protected]/

---
 Documentation/networking/page_pool.rst | 34 ++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
Documentation/networking/page_pool.rst +34 -0
index 0aa850cf4447..eb96a592ec6b 100644
@@ -109,6 +109,40 @@ a page will cause no race conditions is enough.
   caller can then report those stats to the user (perhaps via ethtool,
   debugfs, etc.). See below for an example usage of this API.
 
+DMA sync
+--------
+Driver is always responsible for syncing the pages for the CPU.
+Drivers may choose to take care of syncing for the device as well
+or set the ``PP_FLAG_DMA_SYNC_DEV`` flag to request that pages
+allocated from the page pool are already synced for the device.
+
+If ``PP_FLAG_DMA_SYNC_DEV`` is set, the driver must inform the core what portion
+of the buffer has to be synced. This allows the core to avoid syncing the entire
+page when the drivers knows that the device only accessed a portion of the page.
+
+Most drivers will reserve headroom in front of the frame. This part
+of the buffer is not touched by the device, so to avoid syncing
+it drivers can set the ``offset`` field in struct page_pool_params
+appropriately.
+
+For pages recycled on the XDP xmit and skb paths the page pool will
+use the ``max_len`` member of struct page_pool_params to decide how
+much of the page needs to be synced (starting at ``offset``).
+When directly freeing pages in the driver (page_pool_put_page())
+the ``dma_sync_size`` argument specifies how much of the buffer needs
+to be synced.
+
+If in doubt set ``offset`` to 0, ``max_len`` to ``PAGE_SIZE`` and
+pass -1 as ``dma_sync_size``. That combination of arguments is always
+correct.
+
+Note that the syncing parameters are for the entire page.
+This is important to remember when using fragments (``PP_FLAG_PAGE_FRAG``),
+where allocated buffers may be smaller than a full page.
+Unless the driver author really understands page pool internals
+it's recommended to always use ``offset = 0``, ``max_len = PAGE_SIZE``
+with fragmented page pools.
+
 Stats API and structures
 ------------------------
 If the kernel is configured with ``CONFIG_PAGE_POOL_STATS=y``, the API
Signature · 1 line
2.41.0
reviewer Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH net-next v2 1/2] docs: net: page_pool: document PP_FLAG_DMA_SYNC_DEV parameters

Quoted 6 lines · Jakub Kicinski

Using PP_FLAG_DMA_SYNC_DEV is a bit confusing. It was perhaps more obvious when it was introduced but the page pool use has grown beyond XDP and beyond packet-per-page so now making the heads and tails out of this feature is not trivial.

Quoted 8 lines

Obviously making the API more user friendly would be a better fix, but until someone steps up to do that let's at least document what the parameters are.

Relevant discussion in the first Link.

Thanks.

Quoted 15 lines

---
v2:
- s/sync'/sync/ and other fixes from Randy
v1: https://lore.kernel.org/all/[email protected]/

---
 Documentation/networking/page_pool.rst | 34 ++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
Signature · 1 line
~Randy
Reviewed-by on v2 1/2
#3 lore raw

Re: [PATCH net-next v2 1/2] docs: net: page_pool: document PP_FLAG_DMA_SYNC_DEV parameters

In reply to Jakub Kicinski
Quoted 7 lines · Jakub Kicinski

Using PP_FLAG_DMA_SYNC_DEV is a bit confusing. It was perhaps more obvious when it was introduced but the page pool use has grown beyond XDP and beyond packet-per-page so now making the heads and tails out of this feature is not trivial.

Quoted 8 lines

Obviously making the API more user friendly would be a better fix, but until someone steps up to do that let's at least document what the parameters are.

Relevant discussion in the first Link.

maintainer Acked-by on v2 1/2
#4 lore raw

Re: [PATCH net-next v2 1/2] docs: net: page_pool: document PP_FLAG_DMA_SYNC_DEV parameters

In reply to Jakub Kicinski
Quoted 10 lines · Jakub Kicinski

Using PP_FLAG_DMA_SYNC_DEV is a bit confusing. It was perhaps more obvious when it was introduced but the page pool use has grown beyond XDP and beyond packet-per-page so now making the heads and tails out of this feature is not trivial.

Obviously making the API more user friendly would be a better fix, but until someone steps up to do that let's at least document what the parameters are.

Quoted 5 lines

Relevant discussion in the first Link.

---

Thanks for updating the documentation much appreciated.

Reviewed-by on v2 1/2
#5 lore raw

Re: [PATCH net-next v2 1/2] docs: net: page_pool: document PP_FLAG_DMA_SYNC_DEV parameters

In reply to Jakub Kicinski
Quoted 77 lines · Jakub Kicinski

Using PP_FLAG_DMA_SYNC_DEV is a bit confusing. It was perhaps more obvious when it was introduced but the page pool use has grown beyond XDP and beyond packet-per-page so now making the heads and tails out of this feature is not trivial.

Obviously making the API more user friendly would be a better fix, but until someone steps up to do that let's at least document what the parameters are.

Relevant discussion in the first Link.

---
v2:
- s/sync'/sync/ and other fixes from Randy
v1: https://lore.kernel.org/all/[email protected]/

---
 Documentation/networking/page_pool.rst | 34 ++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
Documentation/networking/page_pool.rst +34 -1
index 0aa850cf4447..eb96a592ec6b 100644
@@ -109,6 +109,40 @@ a page will cause no race conditions is enough.
   caller can then report those stats to the user (perhaps via ethtool,
   debugfs, etc.). See below for an example usage of this API.
 
+DMA sync
+--------
+Driver is always responsible for syncing the pages for the CPU.
+Drivers may choose to take care of syncing for the device as well
+or set the ``PP_FLAG_DMA_SYNC_DEV`` flag to request that pages
+allocated from the page pool are already synced for the device.
+
+If ``PP_FLAG_DMA_SYNC_DEV`` is set, the driver must inform the core what portion
+of the buffer has to be synced. This allows the core to avoid syncing the entire
+page when the drivers knows that the device only accessed a portion of the page.
+
+Most drivers will reserve headroom in front of the frame. This part
+of the buffer is not touched by the device, so to avoid syncing
+it drivers can set the ``offset`` field in struct page_pool_params
+appropriately.
+
+For pages recycled on the XDP xmit and skb paths the page pool will
+use the ``max_len`` member of struct page_pool_params to decide how
+much of the page needs to be synced (starting at ``offset``).
+When directly freeing pages in the driver (page_pool_put_page())
+the ``dma_sync_size`` argument specifies how much of the buffer needs
+to be synced.
+
+If in doubt set ``offset`` to 0, ``max_len`` to ``PAGE_SIZE`` and
+pass -1 as ``dma_sync_size``. That combination of arguments is always
+correct.
+
+Note that the syncing parameters are for the entire page.
+This is important to remember when using fragments (``PP_FLAG_PAGE_FRAG``),
+where allocated buffers may be smaller than a full page.
+Unless the driver author really understands page pool internals
+it's recommended to always use ``offset = 0``, ``max_len = PAGE_SIZE``
+with fragmented page pools.
+
 Stats API and structures
 ------------------------
 If the kernel is configured with ``CONFIG_PAGE_POOL_STATS=y``, the API
--
Quoted 3 lines

2.41.0

author maintainer
#6 lore raw

PATCH v2 2/2 docs: net: page_pool: use kdoc to avoid duplicating the information

In reply to Jakub Kicinski

All struct members of the driver-facing APIs are documented twice, in the code and under Documentation. This is a bit tedious.

I also get the feeling that a lot of developers will read the header when coding, rather than the doc. Bring the two a little closer together by using kdoc for structs and functions.

Using kdoc also gives us links (mentioning a function or struct in the text gets replaced by a link to its doc).

--
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: Lorenzo Bianconi <[email protected]>
CC: Randy Dunlap <[email protected]>
CC: Alexander Lobakin <[email protected]>

---
 Documentation/networking/page_pool.rst |  86 ++++------------
 include/net/page_pool.h                | 134 +++++++++++++++++++------
 net/core/page_pool.c                   |  31 +++++-
 3 files changed, 152 insertions(+), 99 deletions(-)
Documentation/networking/page_pool.rst +18 -68
index eb96a592ec6b..53b5448cc0f1 100644
@@ -64,50 +64,19 @@ This lockless guarantee naturally comes from running under a NAPI softirq.
 The protection doesn't strictly have to be NAPI, any guarantee that allocating
 a page will cause no race conditions is enough.
 
-* page_pool_create(): Create a pool.
-    * flags:      PP_FLAG_DMA_MAP, PP_FLAG_DMA_SYNC_DEV
-    * order:      2^order pages on allocation
-    * pool_size:  size of the ptr_ring
-    * nid:        preferred NUMA node for allocation
-    * dev:        struct device. Used on DMA operations
-    * dma_dir:    DMA direction
-    * max_len:    max DMA sync memory size
-    * offset:     DMA address offset
+.. kernel-doc:: net/core/page_pool.c
+   :identifiers: page_pool_create
 
-* page_pool_put_page(): The outcome of this depends on the page refcnt. If the
-  driver bumps the refcnt > 1 this will unmap the page. If the page refcnt is 1
-  the allocator owns the page and will try to recycle it in one of the pool
-  caches. If PP_FLAG_DMA_SYNC_DEV is set, the page will be synced for_device
-  using dma_sync_single_range_for_device().
+.. kernel-doc:: include/net/page_pool.h
+   :identifiers: struct page_pool_params
 
-* page_pool_put_full_page(): Similar to page_pool_put_page(), but will DMA sync
-  for the entire memory area configured in area pool->max_len.
+.. kernel-doc:: include/net/page_pool.h
+   :identifiers: page_pool_put_page page_pool_put_full_page
+		 page_pool_recycle_direct page_pool_dev_alloc_pages
+		 page_pool_get_dma_addr page_pool_get_dma_dir
 
-* page_pool_recycle_direct(): Similar to page_pool_put_full_page() but caller
-  must guarantee safe context (e.g NAPI), since it will recycle the page
-  directly into the pool fast cache.
-
-* page_pool_dev_alloc_pages(): Get a page from the page allocator or page_pool
-  caches.
-
-* page_pool_get_dma_addr(): Retrieve the stored DMA address.
-
-* page_pool_get_dma_dir(): Retrieve the stored DMA direction.
-
-* page_pool_put_page_bulk(): Tries to refill a number of pages into the
-  ptr_ring cache holding ptr_ring producer lock. If the ptr_ring is full,
-  page_pool_put_page_bulk() will release leftover pages to the page allocator.
-  page_pool_put_page_bulk() is suitable to be run inside the driver NAPI tx
-  completion loop for the XDP_REDIRECT use case.
-  Please note the caller must not use data area after running
-  page_pool_put_page_bulk(), as this function overwrites it.
-
-* page_pool_get_stats(): Retrieve statistics about the page_pool. This API
-  is only available if the kernel has been configured with
-  ``CONFIG_PAGE_POOL_STATS=y``. A pointer to a caller allocated ``struct
-  page_pool_stats`` structure is passed to this API which is filled in. The
-  caller can then report those stats to the user (perhaps via ethtool,
-  debugfs, etc.). See below for an example usage of this API.
+.. kernel-doc:: net/core/page_pool.c
+   :identifiers: page_pool_put_page_bulk page_pool_get_stats
 
 DMA sync
 --------
@@ -146,36 +115,17 @@ with fragmented page pools.
 Stats API and structures
 ------------------------
 If the kernel is configured with ``CONFIG_PAGE_POOL_STATS=y``, the API
-``page_pool_get_stats()`` and structures described below are available. It
-takes a  pointer to a ``struct page_pool`` and a pointer to a ``struct
-page_pool_stats`` allocated by the caller.
+page_pool_get_stats() and structures described below are available.
+It takes a  pointer to a ``struct page_pool`` and a pointer to a struct
+page_pool_stats allocated by the caller.
 
-The API will fill in the provided ``struct page_pool_stats`` with
+The API will fill in the provided struct page_pool_stats with
 statistics about the page_pool.
 
-The stats structure has the following fields::
-
-    struct page_pool_stats {
-        struct page_pool_alloc_stats alloc_stats;
-        struct page_pool_recycle_stats recycle_stats;
-    };
-
-
-The ``struct page_pool_alloc_stats`` has the following fields:
-  * ``fast``: successful fast path allocations
-  * ``slow``: slow path order-0 allocations
-  * ``slow_high_order``: slow path high order allocations
-  * ``empty``: ptr ring is empty, so a slow path allocation was forced.
-  * ``refill``: an allocation which triggered a refill of the cache
-  * ``waive``: pages obtained from the ptr ring that cannot be added to
-    the cache due to a NUMA mismatch.
-
-The ``struct page_pool_recycle_stats`` has the following fields:
-  * ``cached``: recycling placed page in the page pool cache
-  * ``cache_full``: page pool cache was full
-  * ``ring``: page placed into the ptr ring
-  * ``ring_full``: page released from page pool because the ptr ring was full
-  * ``released_refcnt``: page released (and not recycled) because refcnt > 1
+.. kernel-doc:: include/net/page_pool.h
+   :identifiers: struct page_pool_recycle_stats
+		 struct page_pool_alloc_stats
+		 struct page_pool_stats
 
 Coding examples
 ===============
include/net/page_pool.h +104 -30

191 lines not on this page · raw message

net/core/page_pool.c +30 -1
index 8704127646b3..b3e8ed5241ea 100644
@@ -58,6 +58,17 @@ static const char pp_stats[][ETH_GSTRING_LEN] = {
 	"rx_pp_recycle_released_ref",
 };
 
+/**
+ * page_pool_get_stats() - fetch page pool stats
+ * @pool:	pool from which page was allocated
+ * @stats:	struct page_pool_stats to fill in
+ *
+ * Retrieve statistics about the page_pool. This API is only available
+ * if the kernel has been configured with ``CONFIG_PAGE_POOL_STATS=y``.
+ * A pointer to a caller allocated struct page_pool_stats structure
+ * is passed to this API which is filled in. The caller can then report
+ * those stats to the user (perhaps via ethtool, debugfs, etc.).
+ */
 bool page_pool_get_stats(struct page_pool *pool,
 			 struct page_pool_stats *stats)
 {
@@ -224,6 +235,10 @@ static int page_pool_init(struct page_pool *pool,
 	return 0;
 }
 
+/**
+ * page_pool_create() - create a page pool.
+ * @params: parameters, see struct page_pool_params
+ */
 struct page_pool *page_pool_create(const struct page_pool_params *params)
 {
 	struct page_pool *pool;
@@ -628,7 +643,21 @@ void page_pool_put_defragged_page(struct page_pool *pool, struct page *page,
 }
 EXPORT_SYMBOL(page_pool_put_defragged_page);
 
-/* Caller must not use data area after call, as this function overwrites it */
+/**
+ * page_pool_put_page_bulk() - release references on multiple pages
+ * @pool:	pool from which pages were allocated
+ * @data:	array holding page pointers
+ * @count:	number of pages in @data
+ *
+ * Tries to refill a number of pages into the ptr_ring cache holding ptr_ring
+ * producer lock. If the ptr_ring is full, page_pool_put_page_bulk()
+ * will release leftover pages to the page allocator.
+ * page_pool_put_page_bulk() is suitable to be run inside the driver NAPI tx
+ * completion loop for the XDP_REDIRECT use case.
+ *
+ * Please note the caller must not use data area after running
+ * page_pool_put_page_bulk(), as this function overwrites it.
+ */
 void page_pool_put_page_bulk(struct page_pool *pool, void **data,
 			     int count)
 {
Signature · 1 line
2.41.0
reviewer Reviewed-by Tested-by on v2 2/2
#7 lore raw

Re: [PATCH net-next v2 2/2] docs: net: page_pool: use kdoc to avoid duplicating the information

Quoted 3 lines · Jakub Kicinski

All struct members of the driver-facing APIs are documented twice, in the code and under Documentation. This is a bit tedious.

so easier to maintain...

Quoted 3 lines

I also get the feeling that a lot of developers will read the header when coding, rather than the doc. Bring the two a little closer together by using kdoc for structs and functions.

Ack that.

Quoted 5 lines

Using kdoc also gives us links (mentioning a function or struct in the text gets replaced by a link to its doc).

Thanks.

Quoted 14 lines

--
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: Lorenzo Bianconi <[email protected]>
CC: Randy Dunlap <[email protected]>
CC: Alexander Lobakin <[email protected]>

---
 Documentation/networking/page_pool.rst |  86 ++++------------
 include/net/page_pool.h                | 134 +++++++++++++++++++------
 net/core/page_pool.c                   |  31 +++++-
 3 files changed, 152 insertions(+), 99 deletions(-)
Signature · 1 line
~Randy
maintainer Acked-by on v2 2/2
#8 lore raw

Re: [PATCH net-next v2 2/2] docs: net: page_pool: use kdoc to avoid duplicating the information

In reply to Jakub Kicinski
Quoted 11 lines · Jakub Kicinski

All struct members of the driver-facing APIs are documented twice, in the code and under Documentation. This is a bit tedious.

I also get the feeling that a lot of developers will read the header when coding, rather than the doc. Bring the two a little closer together by using kdoc for structs and functions.

Using kdoc also gives us links (mentioning a function or struct in the text gets replaced by a link to its doc).

This is great! :-)

#9 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 8 lines · On Wed, 2 Aug 2023 09:18:19 -0700 you wrote:

Document PP_FLAG_DMA_SYNC_DEV based on recent conversation. Use kdoc to document structs and functions, to avoid duplication.

Olek, this will conflict with your work, but I think that trying to make progress in parallel is the best course of action... Retargetting at net-next to make it a little less bad.

[...]

Here is the summary with links:
- [net-next,v2,1/2] docs: net: page_pool: document PP_FLAG_DMA_SYNC_DEV parameters

    https://git.kernel.org/netdev/net-next/c/e70380650a32
  - [net-next,v2,2/2] docs: net: page_pool: use kdoc to avoid duplicating the information
    https://git.kernel.org/netdev/net-next/c/82e896d992fa

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html