gpudev: introduce memory API
Subsystems (3)
PATCH 1/1 gpudev: introduce memory API
From: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
The infrastructure is prepared to welcome drivers in drivers/gpu/ as the upcoming NVIDIA one, implementing the gpudev API. Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.
The next step should focus on GPU processing task control.
--- .gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 ++ doc/guides/gpus/features/default.ini | 13 ++ doc/guides/gpus/index.rst | 11 ++ doc/guides/gpus/overview.rst | 7 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpu.rst | 5 + doc/guides/prog_guide/index.rst | 1 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpu_driver.h | 44 +++++++ lib/gpudev/meson.build | 9 ++ lib/gpudev/rte_gpudev.h | 183 +++++++++++++++++++++++++++ lib/gpudev/version.map | 11 ++ lib/meson.build | 1 + 18 files changed, 308 insertions(+) create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644 doc/guides/gpus/overview.rst create mode 100644 doc/guides/prog_guide/gpu.rst create mode 100644 drivers/gpu/meson.build create mode 100644 lib/gpudev/gpu_driver.h create mode 100644 lib/gpudev/meson.build create mode 100644 lib/gpudev/rte_gpudev.h create mode 100644 lib/gpudev/version.map
.gitignore +1 -0
@@ -14,6 +14,7 @@ doc/guides/compressdevs/overview_feature_table.txt doc/guides/regexdevs/overview_feature_table.txt doc/guides/vdpadevs/overview_feature_table.txt doc/guides/bbdevs/overview_feature_table.txt +doc/guides/gpus/overview_feature_table.txt # ignore generated ctags/cscope files cscope.out.po
MAINTAINERS +6 -0
@@ -452,6 +452,12 @@ F: app/test-regex/ F: doc/guides/prog_guide/regexdev.rst F: doc/guides/regexdevs/features/default.ini +GPU API - EXPERIMENTAL +M: Elena Agostini <[email protected]> +F: lib/gpudev/ +F: doc/guides/prog_guide/gpu.rst +F: doc/guides/gpus/features/default.ini + Eventdev API M: Jerin Jacob <[email protected]> T: git://dpdk.org/next/dpdk-next-eventdev
doc/api/doxy-api-index.md +1 -0
@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h), [eventdev] (@ref rte_eventdev.h), [event_eth_rx_adapter] (@ref rte_event_eth_rx_adapter.h), [event_eth_tx_adapter] (@ref rte_event_eth_tx_adapter.h),
doc/api/doxy-api.conf.in +1 -0
@@ -40,6 +40,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/eventdev \ @TOPDIR@/lib/fib \ @TOPDIR@/lib/flow_classify \ + @TOPDIR@/lib/gpudev \ @TOPDIR@/lib/graph \ @TOPDIR@/lib/gro \ @TOPDIR@/lib/gso \
doc/guides/conf.py +8 -0
@@ -152,6 +152,9 @@ def generate_overview_table(output_filename, table_id, section, table_name, titl name = ini_filename[:-4] name = name.replace('_vf', 'vf') pmd_names.append(name) + if not pmd_names: + # Add an empty column if table is empty (required by RST syntax) + pmd_names.append(' ') # Pad the table header names. max_header_len = len(max(pmd_names, key=len))
@@ -388,6 +391,11 @@ def setup(app): 'Features', 'Features availability in bbdev drivers', 'Feature') + table_file = dirname(__file__) + '/gpus/overview_feature_table.txt' + generate_overview_table(table_file, 1, + 'Features', + 'Features availability in GPU drivers', + 'Feature') if LooseVersion(sphinx_version) < LooseVersion('1.3.1'): print('Upgrade sphinx to version >= 1.3.1 for '
doc/guides/gpus/features/default.ini new file +13 -0
@@ -0,0 +1,13 @@ +; +; Features of a GPU driver. +; +; This file defines the features that are valid for inclusion in +; the other driver files and also the order that they appear in +; the features table in the documentation. The feature description +; string should not exceed feature_str_len defined in conf.py. +; +[Features] +Get device info = +Share CPU memory with GPU = +Allocate GPU memory = +Free memory =
doc/guides/gpus/index.rst new file +11 -0
@@ -0,0 +1,11 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +GPU Drivers +=========== + +.. toctree:: + :maxdepth: 2 + :numbered: + + overview
doc/guides/gpus/overview.rst new file +7 -0
@@ -0,0 +1,7 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +Overview of GPU Drivers +======================= + +.. include:: overview_feature_table.txt
doc/guides/index.rst +1 -0
@@ -21,6 +21,7 @@ DPDK documentation compressdevs/index vdpadevs/index regexdevs/index + gpus/index eventdevs/index rawdevs/index mempool/index
doc/guides/prog_guide/gpu.rst new file +5 -0
@@ -0,0 +1,5 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +GPU Library +===========
doc/guides/prog_guide/index.rst +1 -0
@@ -27,6 +27,7 @@ Programmer's Guide cryptodev_lib compressdev regexdev + gpu rte_security rawdev link_bonding_poll_mode_drv_lib
drivers/gpu/meson.build new file +4 -0
@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 NVIDIA Corporation & Affiliates + +drivers = []
drivers/meson.build +1 -0
@@ -18,6 +18,7 @@ subdirs = [ 'vdpa', # depends on common, bus and mempool. 'event', # depends on common, bus, mempool and net. 'baseband', # depends on common and bus. + 'gpu', # depends on common and bus. ] if meson.is_cross_build()
lib/gpudev/gpu_driver.h new file +44 -0
44 lines not on this page · raw message
lib/gpudev/meson.build new file +9 -0
@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 NVIDIA Corporation & Affiliates + +headers = files( + 'rte_gpudev.h', +) + +sources = files( +)
lib/gpudev/rte_gpudev.h new file +183 -0
183 lines not on this page · raw message
lib/gpudev/version.map new file +11 -0
@@ -0,0 +1,11 @@ +EXPERIMENTAL { + global: + + rte_gpu_dev_count_avail; + rte_gpu_dev_find_next; + rte_gpu_dev_info_get; + rte_gpu_dev_is_valid; + rte_gpu_free; + rte_gpu_malloc; + rte_gpu_malloc_visible; +};
lib/meson.build +1 -0
@@ -33,6 +33,7 @@ libraries = [ 'distributor', 'efd', 'eventdev', + 'gpudev', 'gro', 'gso', 'ip_frag',
Signature · 1 line
2.31.1
lib/gpudev/rte_gpudev.h line 45
Quoted 10 lines · On Wed, 2 Jun 2021 22:35:31 +0200 Thomas Monjalon <[email protected]> wrote:
lib/gpudev/rte_gpudev.h new file +10
-0+/** Store a list of info for a given GPU. */ +struct rte_gpu_info { +/** GPU device ID. */ +uint16_t gpu_id; +/** Unique identifier name. */ +char name[RTE_GPU_NAME_MAX_LEN ]; +/** Total memory available on device. */ +size_t total_memory; +/** Total processors available on device. */ +int processor_count;
Nit: shouldn't processor_count be unsigned.
02/06/2021 22:46, Stephen Hemminger:
Quoted 15 lines
lib/gpudev/rte_gpudev.h line 45
Quoted 10 lines · On Wed, 2 Jun 2021 22:35:31 +0200 Thomas Monjalon <[email protected]> wrote:
lib/gpudev/rte_gpudev.h new file +10
-0+/** Store a list of info for a given GPU. */ +struct rte_gpu_info { +/** GPU device ID. */ +uint16_t gpu_id; +/** Unique identifier name. */ +char name[RTE_GPU_NAME_MAX_LEN ]; +/** Total memory available on device. */ +size_t total_memory; +/** Total processors available on device. */ +int processor_count;Nit: shouldn't
processor_countbe unsigned.
Absolutely yes, thanks for the catch.
Quoted 17 lines · Thomas Monjalon
From: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
The infrastructure is prepared to welcome drivers in
drivers/gpu/as the upcoming NVIDIA one, implementing the gpudev API. Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.
Quoted 4 lines
The next step should focus on GPU processing task control.
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>
LGTM as an RFC. It is definitely to a patch to apply since implementation is missing. See my notes below.
Quoted 228 lines
--- .gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 ++ doc/guides/gpus/features/default.ini | 13 ++ doc/guides/gpus/index.rst | 11 ++ doc/guides/gpus/overview.rst | 7 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpu.rst | 5 + doc/guides/prog_guide/index.rst | 1 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpu_driver.h | 44 +++++++ lib/gpudev/meson.build | 9 ++ lib/gpudev/rte_gpudev.h | 183 +++++++++++++++++++++++++++ lib/gpudev/version.map | 11 ++ lib/meson.build | 1 + 18 files changed, 308 insertions(+) create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644 doc/guides/gpus/overview.rst create mode 100644 doc/guides/prog_guide/gpu.rst create mode 100644 drivers/gpu/meson.build create mode 100644 lib/gpudev/gpu_driver.h create mode 100644 lib/gpudev/meson.build create mode 100644 lib/gpudev/rte_gpudev.h create mode 100644 lib/gpudev/version.map.gitignore +1
-0@@ -14,6 +14,7 @@ doc/guides/compressdevs/overview_feature_table.txt doc/guides/regexdevs/overview_feature_table.txt doc/guides/vdpadevs/overview_feature_table.txt doc/guides/bbdevs/overview_feature_table.txt +doc/guides/gpus/overview_feature_table.txt # ignore generated ctags/cscope files cscope.out.poMAINTAINERS +6
-0@@ -452,6 +452,12 @@ F: app/test-regex/ F: doc/guides/prog_guide/regexdev.rst F: doc/guides/regexdevs/features/default.ini +GPU API - EXPERIMENTAL +M: Elena Agostini <[email protected]> +F: lib/gpudev/ +F: doc/guides/prog_guide/gpu.rst +F: doc/guides/gpus/features/default.ini + Eventdev API M: Jerin Jacob <[email protected]> T: git://dpdk.org/next/dpdk-next-eventdevdoc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h), [eventdev] (@ref rte_eventdev.h), [event_eth_rx_adapter] (@ref rte_event_eth_rx_adapter.h), [event_eth_tx_adapter] (@ref rte_event_eth_tx_adapter.h),doc/api/doxy-api.conf.in +1
-0@@ -40,6 +40,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/eventdev \ @TOPDIR@/lib/fib \ @TOPDIR@/lib/flow_classify \ + @TOPDIR@/lib/gpudev \ @TOPDIR@/lib/graph \ @TOPDIR@/lib/gro \ @TOPDIR@/lib/gso \doc/guides/conf.py +8
-0@@ -152,6 +152,9 @@ def generate_overview_table(output_filename, table_id, section, table_name, titl name = ini_filename[:-4] name = name.replace('_vf', 'vf') pmd_names.append(name) + if not pmd_names: + # Add an empty column if table is empty (required by RST syntax) + pmd_names.append(' ') # Pad the table header names. max_header_len = len(max(pmd_names, key=len))@@ -388,6 +391,11 @@ def setup(app): 'Features', 'Features availability in bbdev drivers', 'Feature') + table_file = dirname(__file__) + '/gpus/overview_feature_table.txt' + generate_overview_table(table_file, 1, + 'Features', + 'Features availability in GPU drivers', + 'Feature') if LooseVersion(sphinx_version) < LooseVersion('1.3.1'): print('Upgrade sphinx to version >= 1.3.1 for 'doc/guides/gpus/features/default.ini new file +13
-0@@ -0,0 +1,13 @@ +; +; Features of a GPU driver. +; +; This file defines the features that are valid for inclusion in +; the other driver files and also the order that they appear in +; the features table in the documentation. The feature description +; string should not exceed feature_str_len defined in conf.py. +; +[Features] +Get device info = +Share CPU memory with GPU = +Allocate GPU memory = +Free memory =doc/guides/gpus/index.rst new file +11
-0@@ -0,0 +1,11 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +GPU Drivers +=========== + +.. toctree:: + :maxdepth: 2 + :numbered: + + overviewdoc/guides/gpus/overview.rst new file +7
-0@@ -0,0 +1,7 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +Overview of GPU Drivers +======================= + +.. include:: overview_feature_table.txtdoc/guides/index.rst +1
-0@@ -21,6 +21,7 @@ DPDK documentation compressdevs/index vdpadevs/index regexdevs/index + gpus/index eventdevs/index rawdevs/index mempool/indexdoc/guides/prog_guide/gpu.rst new file +5
-0@@ -0,0 +1,5 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +GPU Library +===========doc/guides/prog_guide/index.rst +1
-0@@ -27,6 +27,7 @@ Programmer's Guide cryptodev_lib compressdev regexdev + gpu rte_security rawdev link_bonding_poll_mode_drv_libdrivers/gpu/meson.build new file +4
-0@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 NVIDIA Corporation & Affiliates + +drivers = []drivers/meson.build +1
-0@@ -18,6 +18,7 @@ subdirs = [ 'vdpa', # depends on common, bus and mempool. 'event', # depends on common, bus, mempool and net. 'baseband', # depends on common and bus. + 'gpu', # depends on common and bus. ] if meson.is_cross_build()lib/gpudev/gpu_driver.h new file +7
-0@@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright 2021 NVIDIA Corporation & Affiliates +*/ + +#ifndef GPU_DRIVER_H +#define GPU_DRIVER_H +
lib/gpudev/gpu_driver.h line 17
Quoted 10 lines
lib/gpudev/gpu_driver.h new file +10
-0+#include <stdint.h> + +#include <rte_common.h> + +#include "rte_gpudev.h" + +struct rte_gpu_dev ; + +typedef int (*gpu_malloc_t )(struct rte_gpu_dev *dev,size_t size,void **ptr); +typedef int (*gpu_free_t )(struct rte_gpu_dev *dev,void *ptr);
Not that important but I always prefer to typedef function prototypes w/o pointer and use pointer in the structure below. I.e.
typedef int (gpu_malloc_t)(struct rte_gpu_dev *dev, size_t size, void **ptr);
It allows to specify that corresponding callback must comply to the prototype and produce build error otherwise (and do not rely on warnings), e.g.
static gpu_malloc_t mlx5_gpu_malloc;
static intmlx5_gpu_malloc(struct rte_gpu_dev *dev, size_t size, void **ptr)
{
...
}May be a new library should go this way.
Quoted 6 lines
lib/gpudev/gpu_driver.h new file +6
-0+ +struct rte_gpu_dev { +/* Backing device. */ +struct rte_device *device; +/* GPU info structure. */ +struct rte_gpu_info info;
lib/gpudev/gpu_driver.h line 33
Quoted 10 lines
lib/gpudev/gpu_driver.h new file +10
-0+/* Counter of processes using the device. */ +uint16_t process_cnt; +/* If device is currently used or not. */ +enum rte_gpu_state state; +/* FUNCTION: Allocate memory on the GPU. */ +gpu_malloc_t gpu_malloc; +/* FUNCTION: Allocate memory on the CPU visible from the GPU. */ +gpu_malloc_t gpu_malloc_visible; +/* FUNCTION: Free allocated memory on the GPU. */ +gpu_free_t gpu_free;
Don't we need a callback to get dev_info?
Quoted 144 lines
lib/gpudev/gpu_driver.h new file +11
-0+/* Device interrupt handle. */ +struct rte_intr_handle *intr_handle; +/* Driver-specific private data. */ + void *dev_private; +} __rte_cache_aligned; + +struct rte_gpu_dev *rte_gpu_dev_allocate (const char *name); +struct rte_gpu_dev *rte_gpu_dev_get_by_name (const char *name); +int rte_gpu_dev_release (struct rte_gpu_dev *gpudev); + +#endif /* GPU_DRIVER_H */ lib/gpudev/meson.build new file +9
-0@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021 NVIDIA Corporation & Affiliates + +headers = files( + 'rte_gpudev.h', +) + +sources = files( +)lib/gpudev/rte_gpudev.h new file +112
-0@@ -0,0 +1,183 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright 2021 NVIDIA Corporation & Affiliates +*/ + +#ifndef RTE_GPUDEV_H +#define RTE_GPUDEV_H + +#include <stdint.h> +#include <stdbool.h> + +#include <rte_common.h> + +/** + * @file + * Generic library to interact with a GPU. + * + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + */ + +#ifdef __cplusplus +extern "C" { +#endif + +/** Maximum number of GPU engines. */ +#define RTE_GPU_MAX_DEVS UINT16_C(32) +/** Maximum length of device name. */ +#define RTE_GPU_NAME_MAX_LEN 128 + +/** Flags indicate current state of GPU device. */ +enum rte_gpu_state { + RTE_GPU_STATE_UNUSED, /**< not initialized */ + RTE_GPU_STATE_INITIALIZED, /**< initialized */ +}; + +/** Store a list of info for a given GPU. */ +struct rte_gpu_info { + /** GPU device ID. */ + uint16_t gpu_id; + /** Unique identifier name. */ + char name[RTE_GPU_NAME_MAX_LEN]; + /** Total memory available on device. */ + size_t total_memory; + /** Total processors available on device. */ + int processor_count; +}; + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Returns the number of GPUs detected and associated to DPDK. + * + * @return + * The number of available GPUs. + */ +__rte_experimental +uint16_t rte_gpu_dev_count_avail(void); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Check if the device is valid and initialized in DPDK. + * + * @param gpu_id + * The input GPU ID. + * + * @return + * - True if gpu_id is a valid and initialized GPU. + * - False otherwise. + */ +__rte_experimental +bool rte_gpu_dev_is_valid(uint16_t gpu_id); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Get the GPU ID of the next valid GPU initialized in DPDK. + * + * @param gpu_id + * The initial GPU ID to start the research. + * + * @return + * Next GPU ID corresponding to a valid and initialized GPU device. + */ +__rte_experimental +uint16_t rte_gpu_dev_find_next(uint16_t gpu_id); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Macro to iterate over all valid GPUs. + * + * @param gpu_id + * The ID of the next possible valid GPU. + * @return + * Next valid GPU ID, RTE_GPU_MAX_DEVS if there is none. + */ +#define RTE_GPU_FOREACH_DEV(gpu_id) \ + for (gpu_id = rte_gpu_find_next(0); \ + gpu_id < RTE_GPU_MAX_DEVS; \ + gpu_id = rte_gpu_find_next(gpu_id + 1)) + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Return GPU specific info. + *
lib/gpudev/rte_gpudev.h line 122
Quoted 10 lines
lib/gpudev/rte_gpudev.h new file +10
-0+* @param gpu_id +* GPU ID to get info. +* @param info +* Memory structure to fill with the info. +* +* @return +* 0 on success, -1 otherwise. +*/ +__rte_experimental +int rte_gpu_dev_info_get(uint16_t gpu_id, struct rte_gpu_info **info);
Hm, I think it is better to have 'struct rte_gpu_info *info'. Why should it allocate and return memory to be freed by caller?
lib/gpudev/rte_gpudev.h line 128
Quoted 6 lines
lib/gpudev/rte_gpudev.h new file +6
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Allocate a chunk of memory on the GPU.
Looking a below function it is required to clarify here if the memory is visible or invisible to GPU (or both allowed).
lib/gpudev/rte_gpudev.h line 133
Quoted 5 lines
lib/gpudev/rte_gpudev.h new file +5
-0+* +* @param gpu_id +* GPU ID to allocate memory. +* @param size +* Number of bytes to allocate.
Is behaviour defined if zero size is requested? IMHO, it would be good to define.
lib/gpudev/rte_gpudev.h line 138
Quoted 5 lines
lib/gpudev/rte_gpudev.h new file +5
-0+* @param ptr +* Pointer to store the address of the allocated memory. +* +* @return +* 0 on success, -1 otherwise.
Don't we want to differentiate various errors using negative errno as it is done in many DPDK libraries?
lib/gpudev/rte_gpudev.h line 141
Quoted 3 lines
lib/gpudev/rte_gpudev.h new file +3
-0+*/ +__rte_experimental +int rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr);
May be *malloc() should return a pointer and "negative" values used to report various errnos?
The problem with the approach that comparison vs NULL will not work in this case and we need special macro or small inline function to check error condition.
Returned pointer is definitely more convenient, but above not may result in bugs.
Quoted 6 lines
lib/gpudev/rte_gpudev.h new file +6
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Allocate a chunk of memory on the CPU that is visible from the GPU.
lib/gpudev/rte_gpudev.h line 157
Quoted 10 lines
lib/gpudev/rte_gpudev.h new file +10
-0+* +* @param gpu_id +* Reference GPU ID. +* @param size +* Number of bytes to allocate. +* @param ptr +* Pointer to store the address of the allocated memory. +* +* @return +* 0 on success, -1 otherwise.
Same here
Quoted 4 lines
lib/gpudev/rte_gpudev.h new file +4
-0+*/ +__rte_experimental +int rte_gpu_malloc_visible(uint16_t gpu_id, size_t size, void **ptr); +
lib/gpudev/rte_gpudev.h line 171
Quoted 10 lines
lib/gpudev/rte_gpudev.h new file +10
-0+/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Deallocate a chunk of memory allocated with rte_gpu_malloc*. +* +* @param gpu_id +* Reference GPU ID. +* @param ptr +* Pointer to the memory area to be deallocated.
I think it should be NOP in the case of NULL pointer and it should be documented. If not, it must be documented as well.
lib/gpudev/rte_gpudev.h line 119
Quoted 3 lines
lib/gpudev/rte_gpudev.h new file +3
-0+* +* @return +* 0 on success, -1 otherwise.
Same here
Quoted 39 lines
lib/gpudev/rte_gpudev.h new file +9
-0+*/ +__rte_experimental +int rte_gpu_free(uint16_t gpu_id, void *ptr); + +#ifdef __cplusplus +} +#endif + +#endif /* RTE_GPUDEV_H */ lib/gpudev/version.map new file +11
-0@@ -0,0 +1,11 @@ +EXPERIMENTAL { + global: + + rte_gpu_dev_count_avail; + rte_gpu_dev_find_next; + rte_gpu_dev_info_get; + rte_gpu_dev_is_valid; + rte_gpu_free; + rte_gpu_malloc; + rte_gpu_malloc_visible; +};lib/meson.build +1
-0@@ -33,6 +33,7 @@ libraries = [ 'distributor', 'efd', 'eventdev', + 'gpudev', 'gro', 'gso', 'ip_frag',
03/06/2021 09:06, Andrew Rybchenko:
Quoted 18 lines · Thomas Monjalon
Quoted 17 lines
From: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
The infrastructure is prepared to welcome drivers in
drivers/gpu/as the upcoming NVIDIA one, implementing the gpudev API. Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.
Quoted 8 lines
Quoted 4 lines
The next step should focus on GPU processing task control.
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>LGTM as an RFC. It is definitely to a patch to apply since implementation is missing. See my notes below.
Yes sorry I forgot the RFC tag when sending.
[...]
Quoted 14 lines
lib/gpudev/gpu_driver.h line 17
Quoted 2 lines
lib/gpudev/gpu_driver.h new file +2
-0+typedef int (*gpu_malloc_t )(struct rte_gpu_dev *dev,size_t size,void **ptr); +typedef int (*gpu_free_t )(struct rte_gpu_dev *dev,void *ptr);Not that important but I always prefer to typedef function prototypes w/o pointer and use pointer in the structure below. I.e.
typedef int (gpu_malloc_t)(struct
rte_gpu_dev*dev,size_tsize, void **ptr);It allows to specify that corresponding callback must comply to the prototype and produce build error otherwise (and do not rely on warnings), e.g.
Quoted 8 lines
static
gpu_malloc_tmlx5_gpu_malloc;
static intmlx5_gpu_malloc(struct rte_gpu_dev *dev, size_t size, void **ptr){ ... }May be a new library should go this way.
I agree.
Quoted 11 lines
Quoted 11 lines
lib/gpudev/gpu_driver.h new file +10
-0+ +struct rte_gpu_dev { +/* Backing device. */ +struct rte_device *device; +/* GPU info structure. */ +struct rte_gpu_info info; +/* Counter of processes using the device. */ +uint16_t process_cnt; +/* If device is currently used or not. */ +enum rte_gpu_state state;
Quoted 8 lines
lib/gpudev/gpu_driver.h line 33
Quoted 6 lines
lib/gpudev/gpu_driver.h new file +6
-0+/* FUNCTION: Allocate memory on the GPU. */ +gpu_malloc_t gpu_malloc; +/* FUNCTION: Allocate memory on the CPU visible from the GPU. */ +gpu_malloc_t gpu_malloc_visible; +/* FUNCTION: Free allocated memory on the GPU. */ +gpu_free_t gpu_free;Don't we need a callback to get
dev_info?
Yes it's my miss.
[...]
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+__rte_experimental +int rte_gpu_dev_info_get(uint16_t gpu_id, struct rte_gpu_info **info);Hm, I think it is better to have 'struct
rte_gpu_info*info'. Why should it allocate and return memory to be freed by caller?
No you're right, I overlooked it.
[...]
Quoted 4 lines
Quoted 1 line
+ * Allocate a chunk of memory on the GPU.
Looking a below function it is required to clarify here if the memory is visible or invisible to GPU (or both allowed).
This function allocates on the GPU so it is visible by the GPU. I feel I misunderstand your question.
Quoted 8 lines
Quoted 5 lines
+ *
+ * @paramgpu_id
+ * GPU ID to allocate memory.
+ * @param size
+ * Number of bytes to allocate.Is behaviour defined if zero size is requested? IMHO, it would be good to define.
OK
Quoted 8 lines
Quoted 5 lines
+ * @param ptr
+ * Pointer to store the address of the allocated memory.
+ *
+ * @return
+ * 0 on success, -1 otherwise.Don't we want to differentiate various errors using negative errno as it is done in many DPDK libraries?
Yes I think so, I was just too much lazy to do it in this RFC.
Quoted 6 lines
Review
Quoted 3 lines
+ */
(quoted diff) +2
-0+__rte_experimental +int rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr);May be
*malloc()should return a pointer and "negative" values used to report various errnos?
I don't understand what you mean by negative values if it is a pointer. We could return a pointer and use rte_errno.
Quoted 6 lines
The problem with the approach that comparison vs NULL will not work in this case and we need special macro or small inline function to check error condition.
Returned pointer is definitely more convenient, but above not may result in bugs.
I don't know what is better.
[...]
Quoted 9 lines
Quoted 6 lines
+ * Deallocate a chunk of memory allocated with rte_gpu_malloc*.
+ *
+ * @paramgpu_id
+ * Reference GPU ID.
+ * @param ptr
+ * Pointer to the memory area to be deallocated.I think it should be NOP in the case of NULL pointer and it should be documented. If not, it must be documented as well.
OK for NOP.
Quoted 9 lines · Thomas Monjalon
03/06/2021 09:06, Andrew Rybchenko:
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ * Allocate a chunk of memory on the GPU.
Looking a below function it is required to clarify here if the memory is visible or invisible to GPU (or both allowed).
This function allocates on the GPU so it is visible by the GPU. I feel I misunderstand your question.
Below function says rte_gpu_malloc_visible() and its description highlights that allocated memory is visible to GPU. My problem that I don't understand what's the difference between these two functions.
Quoted 9 lines
Quoted 6 lines
Review
Quoted 3 lines
+ */
(quoted diff) +2
-0+__rte_experimental +int rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr);May be
*malloc()should return a pointer and "negative" values used to report various errnos?I don't understand what you mean by negative values if it is a pointer. We could return a pointer and use
rte_errno.
I was talking about something like (void *)(-ENOMEM), but it is a bad idea. NULL + rte_errno is much better.
However, may be I'd kept callback as is set rte_error in lib if negative value is returned by the callback. This way we'll be safe against lost rte_errno update in drivers.
03/06/2021 09:49, Andrew Rybchenko:
Quoted 8 lines · Thomas Monjalon
Quoted 7 lines
03/06/2021 09:06, Andrew Rybchenko:
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ * Allocate a chunk of memory on the GPU.
Looking a below function it is required to clarify here if the memory is visible or invisible to GPU (or both allowed).
Quoted 7 lines
Quoted 2 lines
This function allocates on the GPU so it is visible by the GPU. I feel I misunderstand your question.
Below function says
rte_gpu_malloc_visible()and its description highlights that allocated memory is visible to GPU. My problem that I don't understand what's the difference between these two functions.
One function allocates in GPU mem, the other allows the GPU to use CPU mem.
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Review
Quoted 3 lines
+ */
(quoted diff) +2
-0+__rte_experimental +int rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr);May be
*malloc()should return a pointer and "negative" values used to report various errnos?I don't understand what you mean by negative values if it is a pointer. We could return a pointer and use
rte_errno.
Quoted 6 lines
I was talking about something like (void *)(-ENOMEM), but it is a bad idea. NULL +
rte_errnois much better.However, may be I'd kept callback as is set
rte_errorin lib if negative value is returned by the callback. This way we'll be safe against lostrte_errnoupdate in drivers.
OK
Quoted 12 lines · Thomas Monjalon
03/06/2021 09:49, Andrew Rybchenko:
Quoted 11 lines · Thomas Monjalon
Quoted 9 lines
03/06/2021 09:06, Andrew Rybchenko:
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ * Allocate a chunk of memory on the GPU.
Looking a below function it is required to clarify here if the memory is visible or invisible to GPU (or both allowed).
This function allocates on the GPU so it is visible by the GPU. I feel I misunderstand your question.
Quoted 6 lines
Quoted 4 lines
Below function says
rte_gpu_malloc_visible()and its description highlights that allocated memory is visible to GPU. My problem that I don't understand what's the difference between these two functions.One function allocates in GPU mem, the other allows the GPU to use CPU mem.
Ah, I see no. G looks like C and I've not noticed the difference. Now it is clear. My bad.
Quick pass:
Quoted 14 lines · Thomas Monjalon
lib/gpudev/gpu_driver.h new file +8
-0@@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright 2021 NVIDIA Corporation & Affiliates +*/ + +#ifndef GPU_DRIVER_H +#define GPU_DRIVER_H + +#include <stdint.h>
lib/gpudev/gpu_driver.h line 18
Quoted 10 lines
lib/gpudev/gpu_driver.h new file +10
-0+ +#include <rte_common.h> + +#include "rte_gpudev.h" + +struct rte_gpu_dev ; + +typedef int (*gpu_malloc_t )(struct rte_gpu_dev *dev,size_t size,void **ptr); +typedef int (*gpu_free_t )(struct rte_gpu_dev *dev,void *ptr); +
Great to see this structure hidden in a driver-only header.
lib/gpudev/gpu_driver.h line 19
Quoted 1 line
lib/gpudev/gpu_driver.h new file +1
-0+struct rte_gpu_dev {
We could have a name[] field here, that will be later pointed at, in rte_gpu_info.
Who is responsible for deciding of the device name?
Quoted 12 lines
lib/gpudev/gpu_driver.h new file +12
-0+/* Backing device. */ +struct rte_device *device; +/* GPU info structure. */ +struct rte_gpu_info info; +/* Counter of processes using the device. */ +uint16_t process_cnt; +/* If device is currently used or not. */ +enum rte_gpu_state state; +/* FUNCTION: Allocate memory on the GPU. */ +gpu_malloc_t gpu_malloc; +/* FUNCTION: Allocate memory on the CPU visible from the GPU. */ +gpu_malloc_t gpu_malloc_visible;
lib/gpudev/gpu_driver.h line 41
Quoted 10 lines
lib/gpudev/gpu_driver.h new file +10
-0+/* FUNCTION: Free allocated memory on the GPU. */ +gpu_free_t gpu_free; +/* Device interrupt handle. */ +struct rte_intr_handle *intr_handle; +/* Driver-specific private data. */ +void *dev_private; +} __rte_cache_aligned; + +struct rte_gpu_dev *rte_gpu_dev_allocate (const char *name); +struct rte_gpu_dev *rte_gpu_dev_get_by_name (const char *name);
Those symbols will have to be marked internal (__rte_internal + version.map) for drivers to see them.
Quoted 27 lines
lib/gpudev/gpu_driver.h new file +3
-0+int rte_gpu_dev_release (struct rte_gpu_dev *gpudev); + +#endif /* GPU_DRIVER_H */ lib/gpudev/rte_gpudev.h new file +16
-0@@ -0,0 +1,183 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright 2021 NVIDIA Corporation & Affiliates +*/ + +#ifndef RTE_GPUDEV_H +#define RTE_GPUDEV_H + +#include <stdint.h> +#include <stdbool.h> + +#include <rte_common.h> + +/** +* @file +* Generic library to interact with a GPU. +*
lib/gpudev/rte_gpudev.h line 26
Quoted 10 lines
lib/gpudev/rte_gpudev.h new file +10
-0+* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +*/ + +#ifdef __cplusplus +extern "C" { +#endif + +/** Maximum number of GPU engines. */ +#define RTE_GPU_MAX_DEVS UINT16_C(32)
Bleh.
Let's stop with max values.
The iterator _next should return a special value indicating there is no more devs to list.
Quoted 5 lines
lib/gpudev/rte_gpudev.h new file +5
-0+/** Maximum length of device name. */ +#define RTE_GPU_NAME_MAX_LEN 128 + +/** Flags indicate current state of GPU device. */ +enum rte_gpu_state {
lib/gpudev/rte_gpudev.h line 41
Quoted 10 lines
lib/gpudev/rte_gpudev.h new file +10
-0+RTE_GPU_STATE_UNUSED ,/**< not initialized */ + RTE_GPU_STATE_INITIALIZED, /**< initialized */ +}; + +/** Store a list of info for a given GPU. */ +struct rte_gpu_info { +/** GPU device ID. */ +uint16_t gpu_id; +/** Unique identifier name. */ +char name[RTE_GPU_NAME_MAX_LEN ];
const char *name;
Then the gpu generic layer simply fills this with the rte_gpu_dev->name field I proposed above.
Quoted 5 lines
lib/gpudev/rte_gpudev.h new file +5
-0+/** Total memory available on device. */ +size_t total_memory; +/** Total processors available on device. */ +int processor_count; +};
Signature · 1 line
David Marchand
03/06/2021 09:18, David Marchand:
Quoted 8 lines
Quick pass:
Quoted 5 lines · Thomas Monjalon
lib/gpudev/gpu_driver.h new file +0
-0
[...]
Quoted 9 lines
lib/gpudev/gpu_driver.h line 18
Quoted 5 lines
lib/gpudev/gpu_driver.h new file +5
-0+struct rte_gpu_dev ; + +typedef int (*gpu_malloc_t )(struct rte_gpu_dev *dev,size_t size,void **ptr); +typedef int (*gpu_free_t )(struct rte_gpu_dev *dev,void *ptr); +Great to see this structure hidden in a driver-only header.
Quoted 5 lines
Quoted 1 line
+struct
rte_gpu_dev{We could have a name[] field here, that will be later pointed at, in
rte_gpu_info.
Who is responsible for deciding of the device name?
The driver is responsible for the name of the device. Yes I agree to store the name here.
Quoted 17 lines
Quoted 17 lines
lib/gpudev/gpu_driver.h new file +17
-0+/* Backing device. */ +struct rte_device *device; +/* GPU info structure. */ +struct rte_gpu_info info; +/* Counter of processes using the device. */ +uint16_t process_cnt; +/* If device is currently used or not. */ +enum rte_gpu_state state; +/* FUNCTION: Allocate memory on the GPU. */ +gpu_malloc_t gpu_malloc; +/* FUNCTION: Allocate memory on the CPU visible from the GPU. */ +gpu_malloc_t gpu_malloc_visible; +/* FUNCTION: Free allocated memory on the GPU. */ +gpu_free_t gpu_free; +/* Device interrupt handle. */ +struct rte_intr_handle *intr_handle; +/* Driver-specific private data. */
Quoted 8 lines
lib/gpudev/gpu_driver.h line 41
Quoted 5 lines
lib/gpudev/gpu_driver.h new file +5
-0+void *dev_private; +} __rte_cache_aligned; + +struct rte_gpu_dev *rte_gpu_dev_allocate (const char *name); +struct rte_gpu_dev *rte_gpu_dev_get_by_name (const char *name);Those symbols will have to be marked internal (
__rte_internal+ version.map) for drivers to see them.
OK, good catch.
[...]
Quoted 7 lines
lib/gpudev/gpu_driver.h
Quoted 2 lines
lib/gpudev/gpu_driver.h new file +2
-0+/** Maximum number of GPU engines. */ +#define RTE_GPU_MAX_DEVS UINT16_C(32)Bleh.
Let's stop with max values.
The iterator_nextshould return a special value indicating there is no more devs to list.
I fully agree.
I would like to define gpu_id as an int to simplify comparisons with error value. int or int16_t ?
Quoted 2 lines
lib/gpudev/gpu_driver.h
Quoted 2 lines
lib/gpudev/gpu_driver.h new file +2
-0+/** Maximum length of device name. */ +#define RTE_GPU_NAME_MAX_LEN 128
Will be dropped as well.
Quoted 16 lines
lib/gpudev/gpu_driver.h
Quoted 13 lines
lib/gpudev/gpu_driver.h new file +13
-0+ +/** Flags indicate current state of GPU device. */ +enum rte_gpu_state { +RTE_GPU_STATE_UNUSED ,/**< not initialized */ +RTE_GPU_STATE_INITIALIZED ,/**< initialized */ +}; + +/** Store a list of info for a given GPU. */ +struct rte_gpu_info { +/** GPU device ID. */ +uint16_t gpu_id; +/** Unique identifier name. */ +char name[RTE_GPU_NAME_MAX_LEN ];const char *name;
Quoted 2 lines
Then the gpu generic layer simply fills this with the
rte_gpu_dev->namefield I proposed above.
Yes.
Quoted 5 lines
Quoted 5 lines
lib/gpudev/gpu_driver.h new file +5
-0+/** Total memory available on device. */ +size_t total_memory; +/** Total processors available on device. */ +int processor_count; +};
Quoted 86 lines · Thomas Monjalon
From: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
The infrastructure is prepared to welcome drivers in
drivers/gpu/as the upcoming NVIDIA one, implementing the gpudev API. Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.The next step should focus on GPU processing task control.
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>--- .gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 ++ doc/guides/gpus/features/default.ini | 13 ++ doc/guides/gpus/index.rst | 11 ++ doc/guides/gpus/overview.rst | 7 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpu.rst | 5 + doc/guides/prog_guide/index.rst | 1 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpu_driver.h | 44 +++++++ lib/gpudev/meson.build | 9 ++ lib/gpudev/rte_gpudev.h | 183 +++++++++++++++++++++++++++ lib/gpudev/version.map | 11 ++ lib/meson.build | 1 + 18 files changed, 308 insertions(+) create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644 doc/guides/gpus/overview.rst create mode 100644 doc/guides/prog_guide/gpu.rst create mode 100644 drivers/gpu/meson.build create mode 100644 lib/gpudev/gpu_driver.h create mode 100644 lib/gpudev/meson.build create mode 100644 lib/gpudev/rte_gpudev.h create mode 100644 lib/gpudev/version.map.gitignore +1
-0@@ -14,6 +14,7 @@ doc/guides/compressdevs/overview_feature_table.txt doc/guides/regexdevs/overview_feature_table.txt doc/guides/vdpadevs/overview_feature_table.txt doc/guides/bbdevs/overview_feature_table.txt +doc/guides/gpus/overview_feature_table.txt # ignore generated ctags/cscope files cscope.out.poMAINTAINERS +6
-0@@ -452,6 +452,12 @@ F: app/test-regex/ F: doc/guides/prog_guide/regexdev.rst F: doc/guides/regexdevs/features/default.ini +GPU API - EXPERIMENTAL +M: Elena Agostini <[email protected]> +F: lib/gpudev/ +F: doc/guides/prog_guide/gpu.rst +F: doc/guides/gpus/features/default.ini + Eventdev API M: Jerin Jacob <[email protected]> T: git://dpdk.org/next/dpdk-next-eventdevdoc/api/doxy-api-index.md +0
-0
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 4 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),
Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;
See above?
Quoted 1 line
+ /* GPU info structure. */
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.
Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
There is a PCI device probed.
I don't understand why it would not be represented as a device.
Quoted 25 lines · Thomas Monjalon
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.
All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).
Quoted 2 lines
03/06/2021 10:41, Jerin Jacob:
Quoted 29 lines · Thomas Monjalon
Quoted 25 lines
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.
Quoted 8 lines
All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).
I don't understand what you mean by "symmetry".
Quoted 40 lines · Thomas Monjalon
03/06/2021 10:41, Jerin Jacob:
Quoted 29 lines · Thomas Monjalon
Quoted 25 lines
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.Quoted 8 lines
All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).
Quoted 1 line
I don't understand what you mean by "symmetry".
The common attributes. or similarity
Quoted 3 lines
03/06/2021 10:47, Jerin Jacob:
Quoted 41 lines · Thomas Monjalon
Quoted 40 lines
03/06/2021 10:41, Jerin Jacob:
Quoted 37 lines · Thomas Monjalon
Quoted 25 lines
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).
Quoted 3 lines
Quoted 1 line
I don't understand what you mean by "symmetry".
The common attributes. or similarity
The common attributes of a device are:
- driver
- bus
- devargs
We have these attributes for a GPU.
About configure/start/stop usual functions,
I think we'll have something similar in the second step for running tasks.
Quoted 45 lines · Thomas Monjalon
03/06/2021 10:47, Jerin Jacob:
Quoted 43 lines · Thomas Monjalon
Quoted 41 lines
03/06/2021 10:41, Jerin Jacob:
Quoted 29 lines · Thomas Monjalon
Quoted 25 lines
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.Quoted 8 lines
All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).I don't understand what you mean by "symmetry".
Quoted 7 lines
Quoted 1 line
The common attributes. or similarity
The common attributes of a device are:
- driver
- bus
- devargs
We have these attributes for a GPU.
Yes. Those are attributes of rte_device. That does not mean and library can not use rte_device.(mempool library driver is using rte_device which is backed by PCI)
In terms of similarity, all other device libraries(not devices) have queue, enqueue() and dequeue() kind of scheme
in ethdev, cryptodev, compressdev, eventdev, bbdev, rawdev. regexdev. i.e existing DPDK device libraries,
This one des not have have that, So question why to call it libgpudev vs libgpu.
The functions you have are memory allocation etc. That's more of a library candidate.
Quoted 3 lines
About configure/start/stop usual functions,
I think we'll have something similar in the second step
Do you think or it will be there?. I think, it is import decision. The device needs have a queue kind of structure
and it is mapping to core to have a notion of configure. queue_setup, start and stop etc
Something similar to
http://code.dpdk.org/dpdk/v21.05/source/lib/regexdev/rte_regexdev.h#L27
Could you share how "running tasks" translates to the above scheme like other her dpdk device libraries?
Quoted 3 lines
for running tasks.
03/06/2021 11:20, Jerin Jacob:
Quoted 63 lines · Thomas Monjalon
Quoted 45 lines
03/06/2021 10:47, Jerin Jacob:
Quoted 43 lines · Thomas Monjalon
Quoted 41 lines
03/06/2021 10:41, Jerin Jacob:
Quoted 29 lines · Thomas Monjalon
Quoted 25 lines
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.Quoted 8 lines
All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).I don't understand what you mean by "symmetry".
Quoted 7 lines
Quoted 1 line
The common attributes. or similarity
The common attributes of a device are:
- driver
- bus
- devargs
We have these attributes for a GPU.Yes. Those are attributes of
rte_device. That does not mean and library can not use rte_device.(mempool library driver is usingrte_devicewhich is backed by PCI)
In terms of similarity, all other device libraries(not devices) have queue,enqueue()anddequeue()kind of scheme
in ethdev, cryptodev, compressdev, eventdev, bbdev, rawdev. regexdev. i.e existing DPDK device libraries,
This one des not have have that, So question why to call it libgpudev vs libgpu.
Quoted 7 lines
The functions you have are memory allocation etc. That's more of a library candidate.
Quoted 2 lines
About configure/start/stop usual functions,
I think we'll have something similar in the second stepDo you think or it will be there?. I think, it is import decision.
That's an important discussion we need to have.
We are preparing a proposal.
Quoted 3 lines
The device needs have a queue kind of structure
and it is mapping to core to have a notion of configure.queue_setup, start and stop etc
Why is it a requirement to call it a device API?
Quoted 4 lines
Something similar to
http://code.dpdk.org/dpdk/v21.05/source/lib/regexdev/rte_regexdev.h#L27
Could you share how "running tasks" translates to the above scheme like other her dpdk device libraries?
We will share our view soon but what to control in GPU execution must be a community discussed requirement.
Quoted 64 lines · Thomas Monjalon
03/06/2021 11:20, Jerin Jacob:
Quoted 62 lines · Thomas Monjalon
Quoted 45 lines
03/06/2021 10:47, Jerin Jacob:
Quoted 43 lines · Thomas Monjalon
Quoted 41 lines
03/06/2021 10:41, Jerin Jacob:
Quoted 29 lines · Thomas Monjalon
Quoted 25 lines
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.Quoted 8 lines
All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).I don't understand what you mean by "symmetry".
Quoted 7 lines
Quoted 1 line
The common attributes. or similarity
The common attributes of a device are:
- driver
- bus
- devargs
We have these attributes for a GPU.Yes. Those are attributes of
rte_device. That does not mean and library can not use rte_device.(mempool library driver is usingrte_devicewhich is backed by PCI)
In terms of similarity, all other device libraries(not devices) have queue,enqueue()anddequeue()kind of scheme
in ethdev, cryptodev, compressdev, eventdev, bbdev, rawdev. regexdev. i.e existing DPDK device libraries,
This one des not have have that, So question why to call it libgpudev vs libgpu.
See below[1]
Quoted 11 lines
Quoted 8 lines
The functions you have are memory allocation etc. That's more of a library candidate.
Quoted 2 lines
About configure/start/stop usual functions,
I think we'll have something similar in the second stepDo you think or it will be there?. I think, it is import decision.
That's an important discussion we need to have.
We are preparing a proposal.
Ack.
Quoted 6 lines
Quoted 4 lines
The device needs have a queue kind of structure
and it is mapping to core to have a notion of configure.queue_setup, start and stop etcWhy is it a requirement to call it a device API?
Then we need to define what needs to call as device library vs library and how? Why mempool is not called a device library vs library? and why all other device library has a common structure like queues and
it binding core etc. I tried to explain above the similar attributes for dpdk device libraries[1] which I think, it a requirement so that the end user will have familiarity with device libraries rather than each one has separate General guidelines and principles.
I think, it is more TB discussion topic and decides on this because I don't see in technical issue in calling it a library.
Quoted 8 lines
Quoted 5 lines
Something similar to
http://code.dpdk.org/dpdk/v21.05/source/lib/regexdev/rte_regexdev.h#L27
Could you share how "running tasks" translates to the above scheme like other her dpdk device libraries?We will share our view soon but what to control in GPU execution must be a community discussed requirement.
Makes sense.
Quoted 3 lines
03/06/2021 12:04, Jerin Jacob:
Quoted 83 lines · Thomas Monjalon
Quoted 64 lines
03/06/2021 11:20, Jerin Jacob:
Quoted 62 lines · Thomas Monjalon
Quoted 45 lines
03/06/2021 10:47, Jerin Jacob:
Quoted 43 lines · Thomas Monjalon
Quoted 41 lines
03/06/2021 10:41, Jerin Jacob:
Quoted 29 lines · Thomas Monjalon
Quoted 25 lines
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.Quoted 8 lines
All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).I don't understand what you mean by "symmetry".
Quoted 7 lines
Quoted 1 line
The common attributes. or similarity
The common attributes of a device are:
- driver
- bus
- devargs
We have these attributes for a GPU.Yes. Those are attributes of
rte_device. That does not mean and library can not use rte_device.(mempool library driver is usingrte_devicewhich is backed by PCI)
In terms of similarity, all other device libraries(not devices) have queue,enqueue()anddequeue()kind of scheme
in ethdev, cryptodev, compressdev, eventdev, bbdev, rawdev. regexdev. i.e existing DPDK device libraries,
This one des not have have that, So question why to call it libgpudev vs libgpu.See below[1]
Quoted 11 lines
Quoted 8 lines
The functions you have are memory allocation etc. That's more of a library candidate.
Quoted 2 lines
About configure/start/stop usual functions,
I think we'll have something similar in the second stepDo you think or it will be there?. I think, it is import decision.
That's an important discussion we need to have.
We are preparing a proposal.Ack.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
The device needs have a queue kind of structure
and it is mapping to core to have a notion of configure.queue_setup, start and stop etcWhy is it a requirement to call it a device API?
Then we need to define what needs to call as device library vs library and how? Why mempool is not called a device library vs library?
My view is simple:
if it has drivers, it is a device API, except bus and mempool libs. About mempool, it started as a standard lib and got extended for HW support.
Quoted 9 lines
and why all
other device library has a common structure like queues and it binding core etc. I tried to explain above the similar attributes for dpdk device libraries[1] which I think, it a requirement so that the end user will have familiarity with device libraries rather than each one has separate General guidelines and principles.I think, it is more TB discussion topic and decides on this because I don't see in technical issue in calling it a library.
The naming is just a choice.
Yesterday morning it was called lib/gpu/
and in the evening it was renamed lib/gpudev/
so no technical issue :)
But the design of the API with queues or other paradigm is something I would like to discuss here.
Note: there was no intent to publish GPU processing control in DPDK 21.08. We want to focus on GPU memory in 21.08, but I understand it is a key decision in the big picture. What would be your need and would you design such API?
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Something similar to
http://code.dpdk.org/dpdk/v21.05/source/lib/regexdev/rte_regexdev.h#L27
Could you share how "running tasks" translates to the above scheme like other her dpdk device libraries?We will share our view soon but what to control in GPU execution must be a community discussed requirement.
Makes sense.
Quoted 89 lines · Thomas Monjalon
03/06/2021 12:04, Jerin Jacob:
Quoted 87 lines · Thomas Monjalon
Quoted 64 lines
03/06/2021 11:20, Jerin Jacob:
Quoted 62 lines · Thomas Monjalon
Quoted 45 lines
03/06/2021 10:47, Jerin Jacob:
Quoted 43 lines · Thomas Monjalon
Quoted 41 lines
03/06/2021 10:41, Jerin Jacob:
Quoted 29 lines · Thomas Monjalon
Quoted 25 lines
03/06/2021 09:47, Jerin Jacob:
Quoted 16 lines · Thomas Monjalon
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 7 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.Quoted 6 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 2 lines
There is a PCI device probed.
I don't understand why it would not be represented as a device.Quoted 8 lines
All other DPDK device has symmetry in structures like queue and symmetry in operation like it has configure, start, stop etc. This one seems more like mempool to me all we want set of vendor-defined ops. So any justification on
make it a device ? why not like mempool library? (driver/mempool/octeontx2 Mempool HW is also PCI device, but we don't take device path for mempool. So I would like to understand any technical reason for making it a device).I don't understand what you mean by "symmetry".
Quoted 7 lines
Quoted 1 line
The common attributes. or similarity
The common attributes of a device are:
- driver
- bus
- devargs
We have these attributes for a GPU.Yes. Those are attributes of
rte_device. That does not mean and library can not use rte_device.(mempool library driver is usingrte_devicewhich is backed by PCI)
In terms of similarity, all other device libraries(not devices) have queue,enqueue()anddequeue()kind of scheme
in ethdev, cryptodev, compressdev, eventdev, bbdev, rawdev. regexdev. i.e existing DPDK device libraries,
This one des not have have that, So question why to call it libgpudev vs libgpu.See below[1]
Quoted 11 lines
Quoted 8 lines
The functions you have are memory allocation etc. That's more of a library candidate.
Quoted 2 lines
About configure/start/stop usual functions,
I think we'll have something similar in the second stepDo you think or it will be there?. I think, it is import decision.
That's an important discussion we need to have.
We are preparing a proposal.Ack.
Quoted 5 lines
Quoted 4 lines
The device needs have a queue kind of structure
and it is mapping to core to have a notion of configure.queue_setup, start and stop etc
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Why is it a requirement to call it a device API?
Then we need to define what needs to call as device library vs library and how? Why mempool is not called a device library vs library?
My view is simple:
if it has drivers, it is a device API, except bus and mempool libs.
rte_secuity has drivers but it is not called a device library.
Quoted 1 line
About mempool, it started as a standard lib and got extended for HW support.
Yes. We did not change to device library as it was fundamentally different than another DPDK deices
when we added the device support.
Quoted 8 lines
Quoted 8 lines
and why all
other device library has a common structure like queues and it binding core etc. I tried to explain above the similar attributes for dpdk device libraries[1] which I think, it a requirement so that the end user will have familiarity with device libraries rather than each one has separate General guidelines and principles.
Quoted 4 lines
Quoted 2 lines
I think, it is more TB discussion topic and decides on this because I don't see in technical issue in calling it a library.
The naming is just a choice.
Not sure.
Quoted 6 lines
Yesterday morning it was called
lib/gpu/
and in the evening it was renamedlib/gpudev/
so no technical issue :)But the design of the API with queues or other paradigm is something I would like to discuss here.
Yeah, That is important. IMO, That defines what needs to be a device library.
Quoted 3 lines
Note: there was no intent to publish GPU processing control in DPDK 21.08. We want to focus on GPU memory in 21.08, but I understand it is a key decision in the big picture.
if the scope is only memory allocation, IMO, it is better to make a library.
Quoted 1 line
What would be your need and would you design such API?
For me, there is no need for gpu library(as of now). May GPU consumers can define what
they need to control using the library.
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
Something similar to
http://code.dpdk.org/dpdk/v21.05/source/lib/regexdev/rte_regexdev.h#L27
Could you share how "running tasks" translates to the above scheme like other her dpdk device libraries?We will share our view soon but what to control in GPU execution must be a community discussed requirement.
Makes sense.
03/06/2021 13:38, Jerin Jacob:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
03/06/2021 12:04, Jerin Jacob:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
03/06/2021 11:20, Jerin Jacob:
Quoted 3 lines
The device needs have a queue kind of structure
and it is mapping to core to have a notion of configure.queue_setup, start and stop etcWhy is it a requirement to call it a device API?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Then we need to define what needs to call as device library vs library and how? Why mempool is not called a device library vs library?
My view is simple:
if it has drivers, it is a device API, except bus and mempool libs.
rte_secuityhas drivers but it is not called a device library.
rte_security is a monster beast :)
Yes it has rte_security_ops implemented in net and crypto drivers, but it is an API extension only, there is no driver dedicated to security.
Quoted 27 lines
Quoted 1 line
About mempool, it started as a standard lib and got extended for HW support.
Yes. We did not change to device library as it was fundamentally different than another DPDK deices
when we added the device support.Quoted 11 lines
Quoted 9 lines
and why all
other device library has a common structure like queues and it binding core etc. I tried to explain above the similar attributes for dpdk device libraries[1] which I think, it a requirement so that the end user will have familiarity with device libraries rather than each one has separate General guidelines and principles.I think, it is more TB discussion topic and decides on this because I don't see in technical issue in calling it a library.
The naming is just a choice.
Not sure.
Quoted 6 lines
Yesterday morning it was called
lib/gpu/
and in the evening it was renamedlib/gpudev/
so no technical issue :)But the design of the API with queues or other paradigm is something I would like to discuss here.
Quoted 7 lines
Yeah, That is important. IMO, That defines what needs to be a device library.
Quoted 3 lines
Note: there was no intent to publish GPU processing control in DPDK 21.08. We want to focus on GPU memory in 21.08, but I understand it is a key decision in the big picture.
if the scope is only memory allocation, IMO, it is better to make a library.
No it is only the first step.
Quoted 4 lines
Quoted 1 line
What would be your need and would you design such API?
For me, there is no need for gpu library(as of now). May GPU consumers can define what they need to control using the library.
We need to integrate GPU processing workload in the DPDK workflow as a generic API.
There could be 2 modes:
- queue of tasks
- tasks in an infinite loop
In both modes, we could get completion notifications with an interrupt/callback or by polling a shared memory.
Quoted 66 lines · Thomas Monjalon
03/06/2021 13:38, Jerin Jacob:
Quoted 10 lines · Thomas Monjalon
Quoted 9 lines
03/06/2021 12:04, Jerin Jacob:
Quoted 8 lines · Thomas Monjalon
Quoted 6 lines
03/06/2021 11:20, Jerin Jacob:
Quoted 3 lines
The device needs have a queue kind of structure
and it is mapping to core to have a notion of configure.queue_setup, start and stop etcWhy is it a requirement to call it a device API?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Then we need to define what needs to call as device library vs library and how? Why mempool is not called a device library vs library?
My view is simple:
if it has drivers, it is a device API, except bus and mempool libs.
rte_secuityhas drivers but it is not called a device library.
rte_securityis a monster beast :)
Yes it hasrte_security_opsimplemented in net and crypto drivers, but it is an API extension only, there is no driver dedicated to security.Quoted 27 lines
Quoted 1 line
About mempool, it started as a standard lib and got extended for HW support.
Yes. We did not change to device library as it was fundamentally different than another DPDK deices
when we added the device support.Quoted 11 lines
Quoted 9 lines
and why all
other device library has a common structure like queues and it binding core etc. I tried to explain above the similar attributes for dpdk device libraries[1] which I think, it a requirement so that the end user will have familiarity with device libraries rather than each one has separate General guidelines and principles.I think, it is more TB discussion topic and decides on this because I don't see in technical issue in calling it a library.
The naming is just a choice.
Not sure.
Quoted 6 lines
Yesterday morning it was called
lib/gpu/
and in the evening it was renamedlib/gpudev/
so no technical issue :)But the design of the API with queues or other paradigm is something I would like to discuss here.
Quoted 7 lines
Yeah, That is important. IMO, That defines what needs to be a device library.
Quoted 3 lines
Note: there was no intent to publish GPU processing control in DPDK 21.08. We want to focus on GPU memory in 21.08, but I understand it is a key decision in the big picture.
if the scope is only memory allocation, IMO, it is better to make a library.
No it is only the first step.
Quoted 4 lines
Quoted 1 line
What would be your need and would you design such API?
For me, there is no need for gpu library(as of now). May GPU consumers can define what they need to control using the library.
Quoted 7 lines
We need to integrate GPU processing workload in the DPDK workflow as a generic API.
There could be 2 modes:
- queue of tasks
- tasks in an infinite loop
In both modes, we could get completion notifications with an interrupt/callback or by polling a shared memory.
OK. If we have enqeue/dequeue kind operation and with queue model then it makes sense to have a device model. It was not there in your initial patch, but if we are adding
in the future then it OK.
Quoted 3 lines
Quoted 86 lines · Jerin Jacob
Quoted 85 lines · Thomas Monjalon
From: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
The infrastructure is prepared to welcome drivers in
drivers/gpu/as the upcoming NVIDIA one, implementing the gpudev API. Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.The next step should focus on GPU processing task control.
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>--- .gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 ++ doc/guides/gpus/features/default.ini | 13 ++ doc/guides/gpus/index.rst | 11 ++ doc/guides/gpus/overview.rst | 7 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpu.rst | 5 + doc/guides/prog_guide/index.rst | 1 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpu_driver.h | 44 +++++++ lib/gpudev/meson.build | 9 ++ lib/gpudev/rte_gpudev.h | 183 +++++++++++++++++++++++++++ lib/gpudev/version.map | 11 ++ lib/meson.build | 1 + 18 files changed, 308 insertions(+) create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644 doc/guides/gpus/overview.rst create mode 100644 doc/guides/prog_guide/gpu.rst create mode 100644 drivers/gpu/meson.build create mode 100644 lib/gpudev/gpu_driver.h create mode 100644 lib/gpudev/meson.build create mode 100644 lib/gpudev/rte_gpudev.h create mode 100644 lib/gpudev/version.map.gitignore +1
-0@@ -14,6 +14,7 @@ doc/guides/compressdevs/overview_feature_table.txt doc/guides/regexdevs/overview_feature_table.txt doc/guides/vdpadevs/overview_feature_table.txt doc/guides/bbdevs/overview_feature_table.txt +doc/guides/gpus/overview_feature_table.txt # ignore generated ctags/cscope files cscope.out.poMAINTAINERS +6
-0@@ -452,6 +452,12 @@ F: app/test-regex/ F: doc/guides/prog_guide/regexdev.rst F: doc/guides/regexdevs/features/default.ini +GPU API - EXPERIMENTAL +M: Elena Agostini <[email protected]> +F: lib/gpudev/ +F: doc/guides/prog_guide/gpu.rst +F: doc/guides/gpus/features/default.ini + Eventdev API M: Jerin Jacob <[email protected]> T: git://dpdk.org/next/dpdk-next-eventdevdoc/api/doxy-api-index.md +0
-0
Quoted 8 lines
doc/api/doxy-api-index.md The public API headers are grouped by topics: · line 24
Quoted 5 lines
doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 13 lines
Any specific reason for making it a device? The reason why I am asking this is, as other DPDK devices as symmetry in queue(s), configure, start, stop operation etc.
doc/api/doxy-api-index.md The public API headers are grouped by topics:
Quoted 4 lines
doc/api/doxy-api-index.md +4
-0+ +struct rte_gpu_dev { + /* Backing device. */ + struct rte_device *device;See above?
Quoted 1 line
+ /* GPU info structure. */
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.
The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.
Quoted 15 lines · Thomas Monjalon
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs with rte_gpu_ instead of rte_gpu_dev as it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.
Quoted 2 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.
Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.
I would like to read more opinions.
Quoted 33 lines · Thomas Monjalon
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.
Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
What I don't understand where is GPU specifics here? I.e. why GPU? NIC can have own memory and provide corresponding API.
What's the difference of "the memory on the CPU that is visible from the GPU" from existing memzones which are DMA mapped?
04/06/2021 15:05, Andrew Rybchenko:
Quoted 41 lines · Thomas Monjalon
Quoted 33 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Quoted 8 lines
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.
Quoted 1 line
I.e. why GPU? NIC can have own memory and provide corresponding API.
So far we don't need to explicitly allocate memory on the NIC. The packets are received or copied to the CPU memory. In the GPU case, the NIC could save the packets directly in the GPU memory, thus the need to manage the GPU memory.
Also, because the GPU program is dynamically loaded,
there is no fixed API to interact with the GPU workload except via memory.
Quoted 2 lines
What's the difference of "the memory on the CPU that is visible from the GPU" from existing memzones which are DMA mapped?
The only difference is that the GPU must map the CPU memory in its program logic.
Quoted 49 lines · Thomas Monjalon
04/06/2021 15:05, Andrew Rybchenko:
Quoted 48 lines · Thomas Monjalon
Quoted 33 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required. No strong opinion on the topic, but unbinding from "Graphics" would be nice.
Quoted 7 lines
Quoted 1 line
I.e. why GPU? NIC can have own memory and provide corresponding API.
So far we don't need to explicitly allocate memory on the NIC. The packets are received or copied to the CPU memory. In the GPU case, the NIC could save the packets directly in the GPU memory, thus the need to manage the GPU memory.
Quoted 8 lines
Also, because the GPU program is dynamically loaded,
there is no fixed API to interact with the GPU workload except via memory.Quoted 2 lines
What's the difference of "the memory on the CPU that is visible from the GPU" from existing memzones which are DMA mapped?
The only difference is that the GPU must map the CPU memory in its program logic.
I see. Thanks for the explanations.
04/06/2021 15:59, Andrew Rybchenko:
Quoted 57 lines · Thomas Monjalon
Quoted 49 lines
04/06/2021 15:05, Andrew Rybchenko:
Quoted 48 lines · Thomas Monjalon
Quoted 33 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.
Quoted 5 lines
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required. No strong opinion on the topic, but unbinding from "Graphics" would be nice.
That's a question I ask myself for months now.
I am not able to find a better name,
and I start thinking that "GPU" is famous enough in high-load computing to convey the idea of what we can expect.
Quoted 65 lines · Thomas Monjalon
04/06/2021 15:59, Andrew Rybchenko:
Quoted 57 lines · Thomas Monjalon
Quoted 49 lines
04/06/2021 15:05, Andrew Rybchenko:
Quoted 48 lines · Thomas Monjalon
Quoted 33 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.Quoted 5 lines
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required. No strong opinion on the topic, but unbinding from "Graphics" would be nice.
Quoted 4 lines
That's a question I ask myself for months now.
I am not able to find a better name,
and I start thinking that "GPU" is famous enough in high-load computing to convey the idea of what we can expect.
The closest I can think of is big-little architecture in ARM SoC.
https://www.arm.com/why-arm/technologies/big-little
We do have similar architecture, Where the "coprocessor" is part of the main CPU.
It is operations are:
- Download firmware
- Memory mapping for Main CPU memory by the co-processor
- Enq/Deq Jobs from/to Main CPU/Coprocessor CPU.
If your scope is something similar and No Graphics involved here then we can remove G.
Coincidentally, Yesterday, I had an interaction with Elena for the same for BaseBand related work in ORAN where
GPU used as Baseband processing instead of Graphics.(So I can understand the big picture of this library)
I can think of "coprocessor-dev" as one of the name. We do have similar machine learning co-processors(for compute)
if we can keep a generic name and it is for the above functions we may use this subsystem as well in the future.
Quoted 3 lines
04/06/2021 17:20, Jerin Jacob:
Quoted 74 lines · Thomas Monjalon
Quoted 64 lines
04/06/2021 15:59, Andrew Rybchenko:
Quoted 57 lines · Thomas Monjalon
Quoted 49 lines
04/06/2021 15:05, Andrew Rybchenko:
Quoted 48 lines · Thomas Monjalon
Quoted 33 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.Quoted 5 lines
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required. No strong opinion on the topic, but unbinding from "Graphics" would be nice.
Quoted 4 lines
That's a question I ask myself for months now.
I am not able to find a better name,
and I start thinking that "GPU" is famous enough in high-load computing to convey the idea of what we can expect.The closest I can think of is big-little architecture in ARM SoC.
https://www.arm.com/why-arm/technologies/big-little
Quoted 6 lines
We do have similar architecture, Where the "coprocessor" is part of the main CPU.
It is operations are:
- Download firmware
- Memory mapping for Main CPU memory by the co-processor
- Enq/Deq Jobs from/to Main CPU/Coprocessor CPU.
Yes it looks like the exact same scope.
I like the word "co-processor" in this context.
Quoted 2 lines
If your scope is something similar and No Graphics involved here then we can remove G.
Indeed no graphics in DPDK :)
By removing the G, you mean keeping only PU? like "pudev"? We could also define the G as "General".
Quoted 4 lines
Coincidentally, Yesterday, I had an interaction with Elena for the same for BaseBand related work in ORAN where
GPU used as Baseband processing instead of Graphics.(So I can understand the big picture of this library)
Yes baseband processing is one possible usage of GPU with DPDK. We could also imagine some security analysis, or any machine learning...
Quoted 1 line
I can think of "coprocessor-dev" as one of the name.
"coprocessor" looks too long as prefix of the functions.
Quoted 3 lines
We do have similar machine learning co-processors(for compute) if we can keep a generic name and it is for the above functions we may use this subsystem as well in the future.
Yes that's the idea to share a common synchronization mechanism with different HW.
That's cool to have such a big interest in the community for this patch.
Quoted 106 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Thomas Monjalon
Sent: Friday, June 4, 2021 23:51
To: Jerin Jacob <[email protected]>
Cc: Honnappa Nagarahalli <[email protected]>; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; dpdk-dev <[email protected]>; Elena Agostini <[email protected]>; David Marchand <[email protected]>
Subject: Re: [dpdk-dev] [PATCH] gpudev: introduce memory API04/06/2021 17:20, Jerin Jacob:
Quoted 74 lines · Thomas Monjalon
Quoted 64 lines
04/06/2021 15:59, Andrew Rybchenko:
Quoted 57 lines · Thomas Monjalon
Quoted 49 lines
04/06/2021 15:05, Andrew Rybchenko:
Quoted 48 lines · Thomas Monjalon
Quoted 33 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref
rte_gpudev.h),Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.Quoted 5 lines
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required. No strong opinion on the topic, but unbinding from "Graphics" would be nice.
Quoted 4 lines
That's a question I ask myself for months now.
I am not able to find a better name,
and I start thinking that "GPU" is famous enough in high-load computing to convey the idea of what we can expect.The closest I can think of is big-little architecture in ARM SoC.
https://www.arm.com/why-arm/technologies/big-littleQuoted 6 lines
We do have similar architecture, Where the "coprocessor" is part of the main CPU.
It is operations are:
- Download firmware
- Memory mapping for Main CPU memory by the co-processor
- Enq/Deq Jobs from/to Main CPU/Coprocessor CPU.Yes it looks like the exact same scope.
I like the word "co-processor" in this context.Quoted 2 lines
If your scope is something similar and No Graphics involved here then we can remove G.
Indeed no graphics in DPDK :)
By removing the G, you mean keeping only PU? like "pudev"? We could also define the G as "General".Quoted 4 lines
Coincidentally, Yesterday, I had an interaction with Elena for the same for BaseBand related work in ORAN where
GPU used as Baseband processing instead of Graphics.(So I can understand the big picture of this library)
Quoted 11 lines
Yes baseband processing is one possible usage of GPU with DPDK. We could also imagine some security analysis, or any machine learning...
Quoted 1 line
I can think of "coprocessor-dev" as one of the name.
"coprocessor" looks too long as prefix of the functions.
Quoted 3 lines
We do have similar machine learning co-processors(for compute) if we can keep a generic name and it is for the above functions we may use this subsystem as well in the future.
Accelerator, 'acce_dev' ? ;-)
Quoted 5 lines
Yes that's the idea to share a common synchronization mechanism with different HW.
That's cool to have such a big interest in the community for this patch.
Quoted 107 lines · Haiyue
Quoted 107 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Thomas Monjalon
Sent: Friday, June 4, 2021 23:51
To: Jerin Jacob <[email protected]>
Cc: Honnappa Nagarahalli <[email protected]>; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; dpdk-dev <[email protected]>; Elena Agostini <[email protected]>; David Marchand <[email protected]>
Subject: Re: [dpdk-dev] [PATCH] gpudev: introduce memory API04/06/2021 17:20, Jerin Jacob:
Quoted 74 lines · Thomas Monjalon
Quoted 64 lines
04/06/2021 15:59, Andrew Rybchenko:
Quoted 57 lines · Thomas Monjalon
Quoted 49 lines
04/06/2021 15:05, Andrew Rybchenko:
Quoted 48 lines · Thomas Monjalon
Quoted 33 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 14 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 12 lines · Jerin Jacob
Quoted 5 lines · Thomas Monjalon
Quoted 1 line
+ [gpudev] (@ref rte_gpudev.h),
Since this device does not have a queue etc? Shouldn't make it a library like mempool with vendor-defined ops?
+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device abstraction library.
Quoted 8 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that
it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet processing pipeline.Quoted 7 lines
I might have confused you. My intention is not to make to fit under mempool API.
I agree that we need a separate library for this. My objection is only to not call libgpudev and
call it libgpu. And have APIs withrte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and
it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.Quoted 5 lines
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required. No strong opinion on the topic, but unbinding from "Graphics" would be nice.
Quoted 4 lines
That's a question I ask myself for months now.
I am not able to find a better name,
and I start thinking that "GPU" is famous enough in high-load computing to convey the idea of what we can expect.The closest I can think of is big-little architecture in ARM SoC.
https://www.arm.com/why-arm/technologies/big-littleQuoted 6 lines
We do have similar architecture, Where the "coprocessor" is part of the main CPU.
It is operations are:
- Download firmware
- Memory mapping for Main CPU memory by the co-processor
- Enq/Deq Jobs from/to Main CPU/Coprocessor CPU.Yes it looks like the exact same scope.
I like the word "co-processor" in this context.Quoted 2 lines
If your scope is something similar and No Graphics involved here then we can remove G.
Indeed no graphics in DPDK :)
By removing the G, you mean keeping only PU? like "pudev"? We could also define the G as "General".Quoted 4 lines
Coincidentally, Yesterday, I had an interaction with Elena for the same for BaseBand related work in ORAN where
GPU used as Baseband processing instead of Graphics.(So I can understand the big picture of this library)
Quoted 6 lines
Quoted 6 lines
Yes baseband processing is one possible usage of GPU with DPDK. We could also imagine some security analysis, or any machine learning...
Quoted 1 line
I can think of "coprocessor-dev" as one of the name.
"coprocessor" looks too long as prefix of the functions.
Yes. Libray name can be lengthy, but API prefix should be 3 letters kind short form will be required.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
We do have similar machine learning co-processors(for compute) if we can keep a generic name and it is for the above functions we may use this subsystem as well in the future.
Accelerator, '
acce_dev' ? ;-)
It may get confused with HW accelerators.
Some of the options I can think of. Sorting in my preference.
library name, API prefix
1) libhpc-dev, rte_hpc_ (hpc-> Heterogeneous processor compute)
2) libhc-dev, rte_hc_
(https://en.wikipedia.org/wiki/Heterogeneous_computing see: Example hardware)
3) libpu-dev, rte_pu_ (pu -> processing unit)
4) libhp-dev, rte_hp_ (hp->heterogeneous processor)
5) libcoprocessor-dev, rte_cps_ ?
6) libcompute-dev, rte_cpt_ ?
7) libgpu-dev, rte_gpu_
Quoted 7 lines
Quoted 6 lines
Yes that's the idea to share a common synchronization mechanism with different HW.
That's cool to have such a big interest in the community for this patch.
<snip>
Quoted 32 lines
Quoted 3 lines
Quoted 3 lines
04/06/2021 17:20, Jerin Jacob:
Quoted 1 line
On Fri, Jun 4, 2021 at 7:39 PM Thomas Monjalon
Quoted 6 lines · <[email protected]> wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
04/06/2021 15:59, Andrew Rybchenko:
Quoted 5 lines · Thomas Monjalon
Quoted 4 lines
04/06/2021 15:05, Andrew Rybchenko:
Quoted 3 lines · Thomas Monjalon
Quoted 2 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 1 line
On Fri, Jun 4, 2021 at 3:58 PM Thomas Monjalon
Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 2 lines · Jerin Jacob
Quoted 1 line
On Thu, Jun 3, 2021 at 2:05 AM Thomas Monjalon
Quoted 10 lines · <[email protected]> wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
+ [gpudev] (@ref rte_gpudev.h),
Since this device does not have a queue etc? Shouldn't
make it a library like mempool with vendor-defined ops?+1
Current RFC announces additional memory allocation capabilities, which can suits better as extension to existing memory related library instead of a new device
abstraction library.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.
Quoted 49 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The other reason it needs to be a full library is that it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet
processing pipeline.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I might have confused you. My intention is not to make to fit
under mempool API.
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 34 lines
Quoted 28 lines
Quoted 21 lines
Quoted 21 lines
Quoted 7 lines
Quoted 6 lines
I agree that we need a separate library for this. My objection is only to not call libgpudev and call it libgpu. And have APIs with
rte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.Quoted 5 lines
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required.
No strong opinion on the topic, but unbinding from "Graphics" would be nice.Quoted 4 lines
That's a question I ask myself for months now.
I am not able to find a better name, and I start thinking that "GPU" is famous enough in high-load computing to convey the idea of what we can expect.The closest I can think of is big-little architecture in ARM SoC.
https://www.arm.com/why-arm/technologies/big-little
From the application pov, big-little arch is nothing but SMT. Not sure how it is similar to another device on PCIe.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
We do have similar architecture, Where the "coprocessor" is part of the main CPU.
It is operations are:
- Download firmware
- Memory mapping for Main CPU memory by the co-processor
- Enq/Deq Jobs from/to Main CPU/Coprocessor CPU.Yes it looks like the exact same scope.
I like the word "co-processor" in this context.Quoted 2 lines
If your scope is something similar and No Graphics involved here then we can remove G.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Indeed no graphics in DPDK :)
By removing the G, you mean keeping only PU? like "pudev"? We could also define the G as "General".Quoted 4 lines
Coincidentally, Yesterday, I had an interaction with Elena for the same for BaseBand related work in ORAN where GPU used as Baseband processing instead of Graphics.(So I can understand the big picture of this library)
This patch does not provide the big picture view of what the processing looks like using GPU. It would be good to explain that.
For ex:
1) Will the notion of GPU hidden from the application? i.e. is the application allowed to launch kernels?
1a) Will DPDK provide abstract APIs to launch kernels?
This would require us to have the notion of GPU in DPDK and the application would depend on the availability of GPU in the system.
2) Is launching kernels hidden? i.e. the application still calls DPDK abstract APIs (such as encryption/decryption APIs) without knowing that the encryption/decryption is happening on GPU. This does not require us to have a notion of GPU in DPDK at the API level
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.
Quoted 25 lines
Quoted 7 lines
Quoted 7 lines
Yes baseband processing is one possible usage of GPU with DPDK. We could also imagine some security analysis, or any machine learning...
Quoted 1 line
I can think of "coprocessor-dev" as one of the name.
"coprocessor" looks too long as prefix of the functions.
Yes. Libray name can be lengthy, but API prefix should be 3 letters kind short form will be required.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
We do have similar machine learning co-processors(for compute) if we can keep a generic name and it is for the above functions we may use this subsystem as well in the future.
Accelerator, '
acce_dev' ? ;-)It may get confused with HW accelerators.
Some of the options I can think of. Sorting in my preference.
Quoted 10 lines
library name, API prefix
1) libhpc-dev,rte_hpc_(hpc-> Heterogeneous processor compute)
2) libhc-dev,rte_hc_
(https://en.wikipedia.org/wiki/Heterogeneous_computing see: Example hardware)
3) libpu-dev,rte_pu_(pu -> processing unit)
4) libhp-dev,rte_hp_(hp->heterogeneousprocessor)
5) libcoprocessor-dev,rte_cps_?
6) libcompute-dev,rte_cpt_?
7) libgpu-dev,rte_gpu_
These seem to assume that the application can launch its own workload on the device? Does DPDK need to provide abstract APIs for launching work on a device?
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Yes that's the idea to share a common synchronization mechanism with different HW.
That's cool to have such a big interest in the community for this patch.
Quoted 82 lines · Honnappa Nagarahalli
<snip>
Quoted 79 lines
Quoted 3 lines
Quoted 3 lines
04/06/2021 17:20, Jerin Jacob:
Quoted 1 line
On Fri, Jun 4, 2021 at 7:39 PM Thomas Monjalon
Quoted 6 lines · <[email protected]> wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
04/06/2021 15:59, Andrew Rybchenko:
Quoted 5 lines · Thomas Monjalon
Quoted 4 lines
04/06/2021 15:05, Andrew Rybchenko:
Quoted 3 lines · Thomas Monjalon
Quoted 2 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 1 line
On Fri, Jun 4, 2021 at 3:58 PM Thomas Monjalon
Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 2 lines · Jerin Jacob
On Thu, Jun 3, 2021 at 2:05 AM Thomas Monjalon
Quoted 10 lines · <[email protected]> wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
+ [gpudev] (@ref rte_gpudev.h),
Since this device does not have a queue etc? Shouldn't
make it a library like mempool with vendor-defined ops?+1
Current RFC announces additional memory allocation
capabilities, which can suits better as extension to
existing memory related library instead of a new deviceabstraction library.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.The other reason it needs to be a full library is that it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet
processing pipeline.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I might have confused you. My intention is not to make to fit
under mempool API.
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 38 lines
Quoted 28 lines
Quoted 21 lines
Quoted 21 lines
Quoted 7 lines
Quoted 6 lines
I agree that we need a separate library for this. My objection is only to not call libgpudev and call it libgpu. And have APIs with
rte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.Quoted 5 lines
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required.
No strong opinion on the topic, but unbinding from "Graphics" would be nice.That's a question I ask myself for months now.
I am not able to find a better name, and I start thinking that "GPU" is famous enough in high-load computing to convey the idea of what we can expect.
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The closest I can think of is big-little architecture in ARM SoC.
https://www.arm.com/why-arm/technologies/big-littleFrom the application pov, big-little arch is nothing but SMT. Not sure how it is similar to another device on PCIe.
Yes. It may not be a device sitting on a PCIe bus, However, It can access it via some bus from the main CPU.
Quoted 23 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 8 lines
We do have similar architecture, Where the "coprocessor" is part of the main CPU.
It is operations are:
- Download firmware
- Memory mapping for Main CPU memory by the co-processor
- Enq/Deq Jobs from/to Main CPU/Coprocessor CPU.Yes it looks like the exact same scope.
I like the word "co-processor" in this context.Quoted 2 lines
If your scope is something similar and No Graphics involved here then we can remove G.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Indeed no graphics in DPDK :)
By removing the G, you mean keeping only PU? like "pudev"? We could also define the G as "General".Quoted 4 lines
Coincidentally, Yesterday, I had an interaction with Elena for the same for BaseBand related work in ORAN where GPU used as Baseband processing instead of Graphics.(So I can understand the big picture of this library)
Quoted 7 lines
This patch does not provide the big picture view of what the processing looks like using GPU. It would be good to explain that.
For ex:
1) Will the notion of GPU hidden from the application? i.e. is the application allowed to launch kernels?
1a) Will DPDK provide abstract APIs to launch kernels?
This would require us to have the notion of GPU in DPDK and the application would depend on the availability of GPU in the system.
2) Is launching kernels hidden? i.e. the application still calls DPDK abstract APIs (such as encryption/decryption APIs) without knowing that the encryption/decryption is happening on GPU. This does not require us to have a notion of GPU in DPDK at the API level
I will leave this to Thomas.
Quoted 2 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.
That is a good thought. it is possible to hook the download firmware, memory management, Job management(as messages to/from device) to rte_device itself.
I think, one needs to consider, how to integrate with the existing DPDK subsystem, for example: If one decided to implement bbdev or regexdev with such computing device,
Need to consider, Is it better to have bbdev driver has depended gpudev or rte_device has this callback and use with bbdev driver.
Quoted 51 lines
Quoted 27 lines
Quoted 9 lines
Quoted 9 lines
Yes baseband processing is one possible usage of GPU with DPDK. We could also imagine some security analysis, or any machine learning...
Quoted 1 line
I can think of "coprocessor-dev" as one of the name.
"coprocessor" looks too long as prefix of the functions.
Yes. Libray name can be lengthy, but API prefix should be 3 letters kind short form will be required.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
We do have similar machine learning co-processors(for compute) if we can keep a generic name and it is for the above functions we may use this subsystem as well in the future.
Accelerator, '
acce_dev' ? ;-)It may get confused with HW accelerators.
Some of the options I can think of. Sorting in my preference.
Quoted 10 lines
library name, API prefix
1) libhpc-dev,rte_hpc_(hpc-> Heterogeneous processor compute)
2) libhc-dev,rte_hc_
(https://en.wikipedia.org/wiki/Heterogeneous_computing see: Example hardware)
3) libpu-dev,rte_pu_(pu -> processing unit)
4) libhp-dev,rte_hp_(hp->heterogeneousprocessor)
5) libcoprocessor-dev,rte_cps_?
6) libcompute-dev,rte_cpt_?
7) libgpu-dev,rte_gpu_These seem to assume that the application can launch its own workload on the device? Does DPDK need to provide abstract APIs for launching work on a device?
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Yes that's the idea to share a common synchronization mechanism with different HW.
That's cool to have such a big interest in the community for this patch.
06/06/2021 07:28, Jerin Jacob:
Quoted 6 lines
On Sun, Jun 6, 2021 at 6:44 AM Honnappa Nagarahalli
Quoted 5 lines
This patch does not provide the big picture view of what the processing looks like using GPU. It would be good to explain that. For ex:
1) Will the notion of GPU hidden from the application? i.e. is the application allowed to launch kernels?
1a) Will DPDK provide abstract APIs to launch kernels?
This would require us to have the notion of GPU in DPDK and the application would depend on the availability of GPU in the system.
Not sure "kernels" is a well known word in this context. I propose talking about computing tasks.
The DPDK application running on the CPU must be synchronized with the tasks running on devices, so yes we need a way to decide what to launch and when from the DPDK application.
Quoted 4 lines
Quoted 2 lines
2) Is launching kernels hidden? i.e. the application still calls DPDK abstract APIs (such as encryption/decryption APIs) without knowing that the encryption/decryption is happening on GPU. This does not require us to have a notion of GPU in DPDK at the API level
I will leave this to Thomas.
The general need is to allow running any kind of processing on devices. Some processing may be very specific, others could fit in the existing class API like crypto and regex.
I think implementing such specific class drivers based on tasks dynamically loaded on the device may be done as a second step.
Thank you for the questions, it helps defining the big picture for the next revision of the patch.
Quoted 10 lines
Quoted 1 line
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.That is a good thought. it is possible to hook the download firmware, memory management, Job management(as messages to/from device) to
rte_deviceitself.
I think, one needs to consider, how to integrate with the existing DPDK subsystem, for example: If one decided to implement bbdev or regexdev with such computing device,
Need to consider, Is it better to have bbdev driver has depended gpudev orrte_devicehas this callback and use with bbdev driver.
Absolutely. If a specialized driver class fits with a workload, it is best handled with a driver in its specific class.
Quoted 35 lines
Quoted 35 lines
Quoted 34 lines
Quoted 6 lines
Quoted 6 lines
Yes baseband processing is one possible usage of GPU with DPDK. We could also imagine some security analysis, or any machine learning...
Quoted 1 line
I can think of "coprocessor-dev" as one of the name.
"coprocessor" looks too long as prefix of the functions.
Yes. Libray name can be lengthy, but API prefix should be 3 letters kind short form will be required.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
We do have similar machine learning co-processors(for compute) if we can keep a generic name and it is for the above functions we may use this subsystem as well in the future.
Accelerator, '
acce_dev' ? ;-)It may get confused with HW accelerators.
Some of the options I can think of. Sorting in my preference.
library name, API prefix
1) libhpc-dev,rte_hpc_(hpc-> Heterogeneous processor compute)
2) libhc-dev,rte_hc_
(https://en.wikipedia.org/wiki/Heterogeneous_computing see: Example hardware)
3) libpu-dev,rte_pu_(pu -> processing unit)
4) libhp-dev,rte_hp_(hp->heterogeneousprocessor)
5) libcoprocessor-dev,rte_cps_?
6) libcompute-dev,rte_cpt_?
7) libgpu-dev,rte_gpu_
Quoted 1 line
Quoted 1 line
These seem to assume that the application can launch its own workload on the device? Does DPDK need to provide abstract APIs for launching work on a device?
That's the difficult part.
We should not try to re-invent CUDA or OpenCL.
I think this part should not be in DPDK.
We only need to synchronize with dynamic nature of the device workload. We will be more specific in the v2.
Quoted 130 lines
-----Original Message-----
From: Honnappa Nagarahalli <[email protected]>
Sent: Sunday, June 6, 2021 09:14
To: Jerin Jacob <[email protected]>; Wang, Haiyue <[email protected]>
Cc: [email protected]; Andrew Rybchenko <[email protected]>; Yigit, Ferruh <[email protected]>; dpdk-dev <[email protected]>; Elena Agostini <[email protected]>; David Marchand <[email protected]>; nd <[email protected]>; Honnappa Nagarahalli <[email protected]>; nd <[email protected]>
Subject: RE: [dpdk-dev] [PATCH] gpudev: introduce memory API<snip>
Quoted 32 lines
Quoted 3 lines
Quoted 3 lines
04/06/2021 17:20, Jerin Jacob:
Quoted 1 line
On Fri, Jun 4, 2021 at 7:39 PM Thomas Monjalon
Quoted 6 lines · <[email protected]> wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
04/06/2021 15:59, Andrew Rybchenko:
Quoted 5 lines · Thomas Monjalon
Quoted 4 lines
04/06/2021 15:05, Andrew Rybchenko:
Quoted 3 lines · Thomas Monjalon
Quoted 2 lines
04/06/2021 13:09, Jerin Jacob:
Quoted 1 line
On Fri, Jun 4, 2021 at 3:58 PM Thomas Monjalon
Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
03/06/2021 11:33, Ferruh Yigit:
Quoted 2 lines · Jerin Jacob
On Thu, Jun 3, 2021 at 2:05 AM Thomas Monjalon
Quoted 10 lines · <[email protected]> wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
+ [gpudev] (@ref rte_gpudev.h),
Since this device does not have a queue etc? Shouldn't
make it a library like mempool with vendor-defined ops?+1
Current RFC announces additional memory allocation
capabilities, which can suits better as extension to
existing memory related library instead of a new deviceabstraction library.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
It is not replacing mempool.
It is more at the same level as EAL memory management:
allocate simple buffer, but with the exception it is done on a specific device, so it requires a device ID.Quoted 49 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The other reason it needs to be a full library is that it will start a workload on the GPU and get completion notification so we can integrate the GPU workload in a packet
processing pipeline.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I might have confused you. My intention is not to make to fit
under mempool API.
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 34 lines
Quoted 28 lines
Quoted 21 lines
Quoted 21 lines
Quoted 7 lines
Quoted 6 lines
I agree that we need a separate library for this. My objection is only to not call libgpudev and call it libgpu. And have APIs with
rte_gpu_instead ofrte_gpu_devas it not like existing "device libraries" in DPDK and it like other "libraries" in DPDK.Quoted 6 lines
I think we should define a queue of processing actions, so it looks like other device libraries.
And anyway I think a library managing a device class, and having some device drivers deserves the name of device library.I would like to read more opinions.
Since the library is an unified interface to GPU device drivers I think it should be named as in the patch - gpudev.
Mempool looks like an exception here - initially it was pure SW library, but not there are HW backends and corresponding device drivers.
Quoted 6 lines
Quoted 1 line
What I don't understand where is GPU specifics here?
That's an interesting question.
Let's ask first what is a GPU for DPDK?
I think it is like a sub-CPU with high parallel execution capabilities, and it is controlled by the CPU.Quoted 5 lines
I have no good ideas how to name it in accordance with above description to avoid "G" which for "Graphics" if understand correctly. However, may be it is not required.
No strong opinion on the topic, but unbinding from "Graphics" would be nice.Quoted 4 lines
That's a question I ask myself for months now.
I am not able to find a better name, and I start thinking that "GPU" is famous enough in high-load computing to convey the idea of what we can expect.The closest I can think of is big-little architecture in ARM SoC.
https://www.arm.com/why-arm/technologies/big-littleFrom the application pov, big-little arch is nothing but SMT. Not sure how it is similar to another device on PCIe.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
We do have similar architecture, Where the "coprocessor" is part of the main CPU.
It is operations are:
- Download firmware
- Memory mapping for Main CPU memory by the co-processor
- Enq/Deq Jobs from/to Main CPU/Coprocessor CPU.Yes it looks like the exact same scope.
I like the word "co-processor" in this context.Quoted 2 lines
If your scope is something similar and No Graphics involved here then we can remove G.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Indeed no graphics in DPDK :)
By removing the G, you mean keeping only PU? like "pudev"? We could also define the G as "General".Quoted 4 lines
Coincidentally, Yesterday, I had an interaction with Elena for the same for BaseBand related work in ORAN where GPU used as Baseband processing instead of Graphics.(So I can understand the big picture of this library)
This patch does not provide the big picture view of what the processing looks like using GPU. It would be good to explain that.
For ex:
1) Will the notion of GPU hidden from the application? i.e. is the application allowed to launch kernels?
1a) Will DPDK provide abstract APIs to launch kernels?
This would require us to have the notion of GPU in DPDK and the application would depend on the
availability of GPU in the system.
2) Is launching kernels hidden? i.e. the application still calls DPDK abstract APIs (such as encryption/decryption APIs) without knowing that the encryption/decryption is happening on GPU. This does not require us to have a notion of GPU in DPDK at the API level
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.
"rte_dev_malloc" sounds a good name, then looks like we need to enhance the 'struct rte_device' with some new ops as:
eal: move DMA mapping from bus-specific to generic driver
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Quoted 51 lines
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Yes baseband processing is one possible usage of GPU with DPDK. We could also imagine some security analysis, or any machine learning...
Quoted 1 line
I can think of "coprocessor-dev" as one of the name.
"coprocessor" looks too long as prefix of the functions.
Yes. Libray name can be lengthy, but API prefix should be 3 letters kind short form will be required.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
We do have similar machine learning co-processors(for compute) if we can keep a generic name and it is for the above functions we may use this subsystem as well in the future.
Accelerator, '
acce_dev' ? ;-)It may get confused with HW accelerators.
Some of the options I can think of. Sorting in my preference.
Quoted 10 lines
library name, API prefix
1) libhpc-dev,rte_hpc_(hpc-> Heterogeneous processor compute)
2) libhc-dev,rte_hc_
(https://en.wikipedia.org/wiki/Heterogeneous_computing see: Example hardware)
3) libpu-dev,rte_pu_(pu -> processing unit)
4) libhp-dev,rte_hp_(hp->heterogeneousprocessor)
5) libcoprocessor-dev,rte_cps_?
6) libcompute-dev,rte_cpt_?
7) libgpu-dev,rte_gpu_These seem to assume that the application can launch its own workload on the device? Does DPDK need to provide abstract APIs for launching work on a device?
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Yes that's the idea to share a common synchronization mechanism with different HW.
That's cool to have such a big interest in the community for this patch.
07/06/2021 09:20, Wang, Haiyue:
Quoted 8 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs."
rte_dev_malloc" sounds a good name,
Yes I like the idea.
2 concerns:
1/ Device memory allocation requires a device handle. So far we avoided exposing rte_device to the application. How should we get a device handle from a DPDK application?
2/ Implementation must be done in a driver. Should it be a callback defined at rte_device level?
Quoted 6 lines
then looks like we need to enhance the
'structrte_device' with some new ops as:eal: move DMA mapping from bus-specific to generic driver
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Not sure the above patch is a good idea.
Let's discuss this DMA detail later :)
Quoted 9 lines · Thomas Monjalon
07/06/2021 09:20, Wang, Haiyue:
Quoted 7 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.
Quoted 8 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?
Each device behaves differently at this level. In the view of the generic application, the architecture should like
< Use DPDK subsystem as rte_ethdev, rte_bbdev etc for SPECIFIC function > ^
|
< DPDK driver>
^
|
<rte_device with this new callbacks >
An implementation may decide to have "in tree" or "out of tree" drivers or rte_device implementaion.
But generic DPDK applications should not use devices directly. i.e rte_device need to have this callback and
mlx ethdev/crypto driver use this driver to implement public API. Otherwise, it is the same as rawdev in DPDK.
So not sure what it brings other than raw dev here if we are not taking the above architecture.
Quoted 3 lines
2/ Implementation must be done in a driver. Should it be a callback defined at
rte_devicelevel?
IMO, Yes and DPDK subsystem drivers to use it.
Quoted 13 lines
Quoted 7 lines
then looks like we need to enhance the
'structrte_device' with some new ops as:eal: move DMA mapping from bus-specific to generic driver
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Not sure the above patch is a good idea.
Let's discuss this DMA detail later :)
07/06/2021 15:54, Jerin Jacob:
Quoted 21 lines · Thomas Monjalon
Quoted 8 lines
07/06/2021 09:20, Wang, Haiyue:
Quoted 7 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.Quoted 8 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?Each device behaves differently at this level. In the view of the generic application, the architecture should like
Quoted 7 lines
< Use DPDK subsystem as
rte_ethdev,rte_bbdevetc for SPECIFIC function > ^
|
< DPDK driver>
^
|
<rte_devicewith this new callbacks >
I think the formatting went wrong above.
I would add more to the block diagram:
class device API - computing device API
| | |
class device driver - computing device driver
| |EAL device with memory callback
The idea above is that the class device driver can use services of the new computing device library.
One basic API service is to provide a device ID for the memory callback. Other services are for execution control.
Quoted 10 lines
An implementation may decide to have "in tree" or "out of tree" drivers or
rte_deviceimplementaion.
But generic DPDK applications should not use devices directly. i.erte_deviceneed to have this callback and
mlx ethdev/crypto driver use this driver to implement public API. Otherwise, it is the same as rawdev in DPDK.
So not sure what it brings other than raw dev here if we are not taking the above architecture.Quoted 1 line
Quoted 4 lines
Quoted 2 lines
2/ Implementation must be done in a driver. Should it be a callback defined at
rte_devicelevel?IMO, Yes and DPDK subsystem drivers to use it.
I'm not sure subsystems should bypass the API for device memory. We could do some generic work in the API function and call the driver callback only for device-specific stuff.
In such case the callback and the API would be
in the library computing device library.
On the other hand, having the callback and API in EAL would allow having a common function for memory allocation in EAL.
Another thought: I would like to unify memory allocation in DPDK with the same set of flags in an unique function.
A flag could be used to target devices instead of the running CPU, and the same parameter could be shared for the device ID or NUMA node.
Quoted 35 lines · Thomas Monjalon
07/06/2021 15:54, Jerin Jacob:
Quoted 21 lines · Thomas Monjalon
Quoted 8 lines
07/06/2021 09:20, Wang, Haiyue:
Quoted 7 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.Quoted 8 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?Each device behaves differently at this level. In the view of the generic application, the architecture should like
Quoted 7 lines
< Use DPDK subsystem as
rte_ethdev,rte_bbdevetc for SPECIFIC function > ^
|
< DPDK driver>
^
|
<rte_devicewith this new callbacks >I think the formatting went wrong above.
I would add more to the block diagram:
Quoted 8 lines
class device API - computing device API
| | | class device driver - computing device driver | |EAL device with memory callback
The idea above is that the class device driver can use services of the new computing device library.
Yes. The question is, do we need any public DPDK _application_ APIs for that? If it is public API then the scope is much bigger than that as the application can use it directly and it makes it non portable.
if the scope is only, the class driver consumption then the existing "bus" _kind of_
abstraction/API makes sense to me.
Where it abstracts,
-FW download of device
-Memory management of device
-Opaque way to enq/deque jobs to the device.
And above should be consumed by "class driver" not "application".
If the application doing do that, we are in rte_raw device territory.
Quoted 32 lines
One basic API service is to provide a device ID for the memory callback. Other services are for execution control.
Quoted 10 lines
An implementation may decide to have "in tree" or "out of tree" drivers or
rte_deviceimplementaion.
But generic DPDK applications should not use devices directly. i.erte_deviceneed to have this callback and
mlx ethdev/crypto driver use this driver to implement public API. Otherwise, it is the same as rawdev in DPDK.
So not sure what it brings other than raw dev here if we are not taking the above architecture.Quoted 1 line
Quoted 4 lines
Quoted 2 lines
2/ Implementation must be done in a driver. Should it be a callback defined at
rte_devicelevel?IMO, Yes and DPDK subsystem drivers to use it.
I'm not sure subsystems should bypass the API for device memory. We could do some generic work in the API function and call the driver callback only for device-specific stuff.
In such case the callback and the API would be
in the library computing device library.
On the other hand, having the callback and API in EAL would allow having a common function for memory allocation in EAL.Another thought: I would like to unify memory allocation in DPDK with the same set of flags in an unique function.
A flag could be used to target devices instead of the running CPU, and the same parameter could be shared for the device ID or NUMA node.
08/06/2021 06:10, Jerin Jacob:
Quoted 42 lines · Thomas Monjalon
Quoted 41 lines
07/06/2021 15:54, Jerin Jacob:
Quoted 21 lines · Thomas Monjalon
Quoted 8 lines
07/06/2021 09:20, Wang, Haiyue:
Quoted 7 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.Quoted 8 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?Each device behaves differently at this level. In the view of the generic application, the architecture should like
Quoted 7 lines
< Use DPDK subsystem as
rte_ethdev,rte_bbdevetc for SPECIFIC function > ^
|
< DPDK driver>
^
|
<rte_devicewith this new callbacks >I think the formatting went wrong above.
I would add more to the block diagram:
class device API - computing device API
| | | class device driver - computing device driver | |EAL device with memory callback
Quoted 4 lines
Quoted 2 lines
The idea above is that the class device driver can use services of the new computing device library.
Yes. The question is, do we need any public DPDK _application_ APIs for that?
To have something generic!
Quoted 2 lines
If it is public API then the scope is much bigger than that as the application can use it directly and it makes it non portable.
It is a non-sense. If we make an API, it will be better portable. The only part which is non-portable is the program on the device which may be different per computing device.
The synchronization with the DPDK application should be portable if we define some good API.
Quoted 9 lines
if the scope is only, the class driver consumption then the existing "bus"
_kindof_
abstraction/API makes sense to me.Where it abstracts,
(quoted diff) +0
-3-FW download of device -Memory management of device -Opaque way to enq/deque jobs to the device.
Quoted 3 lines
And above should be consumed by "class driver" not "application".
If the application doing do that, we are in
rte_rawdevice territory.
I'm sorry I don't understand what you make such assertion. It seems you don't want generic API (which is the purpose of DPDK).
Quoted 47 lines · Thomas Monjalon
08/06/2021 06:10, Jerin Jacob:
Quoted 45 lines · Thomas Monjalon
Quoted 43 lines
07/06/2021 15:54, Jerin Jacob:
Quoted 21 lines · Thomas Monjalon
Quoted 8 lines
07/06/2021 09:20, Wang, Haiyue:
Quoted 7 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.Quoted 8 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?Each device behaves differently at this level. In the view of the generic application, the architecture should like
Quoted 7 lines
< Use DPDK subsystem as
rte_ethdev,rte_bbdevetc for SPECIFIC function > ^
|
< DPDK driver>
^
|
<rte_devicewith this new callbacks >I think the formatting went wrong above.
I would add more to the block diagram:
class device API - computing device API
| | | class device driver - computing device driver | |EAL device with memory callback
The idea above is that the class device driver can use services of the new computing device library.
Quoted 8 lines
Quoted 1 line
Yes. The question is, do we need any public DPDK _application_ APIs for that?
To have something generic!
Quoted 2 lines
If it is public API then the scope is much bigger than that as the application can use it directly and it makes it non portable.
It is a non-sense. If we make an API, it will be better portable.
The portal application will be using class device API.
For example, when application needs to call rte_gpu_malloc() vs rte_malloc() ? Is it better the use of drivers specific functions used in "class device driver" not exposed?
Quoted 14 lines
The only part which is non-portable is the program on the device which may be different per computing device.
The synchronization with the DPDK application should be portable if we define some good API.Quoted 9 lines
if the scope is only, the class driver consumption then the existing "bus"
_kindof_
abstraction/API makes sense to me.Where it abstracts,
(quoted diff) +0
-3-FW download of device -Memory management of device -Opaque way to enq/deque jobs to the device.
Quoted 6 lines
Quoted 3 lines
And above should be consumed by "class driver" not "application".
If the application doing do that, we are in
rte_rawdevice territory.I'm sorry I don't understand what you make such assertion. It seems you don't want generic API (which is the purpose of DPDK).
I would like to have a generic _application_ API if the application _needs_ to use it.
The v1 nowhere close to any compute device description.
It has a memory allocation API. It is the device attribute, not strictly tied to ONLY TO computing device.
So at least, I am asking to have concrete
proposal on "compute device" schematic rather than start with memory API and rubber stamp as new device adds anything in future.
When we added any all the class devices to DPDK, Everyone had a complete view of it is function(at RFC of each subsystem had enough API to express the "basic" usage)
and purpose from the _application_ PoV. I see that is missing here.
Quoted 2 lines
08/06/2021 09:09, Jerin Jacob:
Quoted 85 lines · Thomas Monjalon
Quoted 47 lines
08/06/2021 06:10, Jerin Jacob:
Quoted 45 lines · Thomas Monjalon
Quoted 43 lines
07/06/2021 15:54, Jerin Jacob:
Quoted 21 lines · Thomas Monjalon
Quoted 8 lines
07/06/2021 09:20, Wang, Haiyue:
Quoted 7 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.Quoted 8 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?Each device behaves differently at this level. In the view of the generic application, the architecture should like
Quoted 7 lines
< Use DPDK subsystem as
rte_ethdev,rte_bbdevetc for SPECIFIC function > ^
|
< DPDK driver>
^
|
<rte_devicewith this new callbacks >I think the formatting went wrong above.
I would add more to the block diagram:
class device API - computing device API
| | | class device driver - computing device driver | |EAL device with memory callback
The idea above is that the class device driver can use services of the new computing device library.
Quoted 8 lines
Quoted 1 line
Yes. The question is, do we need any public DPDK _application_ APIs for that?
To have something generic!
Quoted 2 lines
If it is public API then the scope is much bigger than that as the application can use it directly and it makes it non portable.
It is a non-sense. If we make an API, it will be better portable.
The portal application will be using class device API.
For example, when application needs to callrte_gpu_malloc()vsrte_malloc()? Is it better the use of drivers specific functions used in "class device driver" not exposed?Quoted 14 lines
The only part which is non-portable is the program on the device which may be different per computing device.
The synchronization with the DPDK application should be portable if we define some good API.Quoted 9 lines
if the scope is only, the class driver consumption then the existing "bus"
_kindof_
abstraction/API makes sense to me.Where it abstracts,
(quoted diff) +0
-3-FW download of device -Memory management of device -Opaque way to enq/deque jobs to the device.Quoted 6 lines
Quoted 3 lines
And above should be consumed by "class driver" not "application".
If the application doing do that, we are in
rte_rawdevice territory.I'm sorry I don't understand what you make such assertion. It seems you don't want generic API (which is the purpose of DPDK).
Quoted 4 lines
I would like to have a generic _application_ API if the application _needs_ to use it.
The v1 nowhere close to any compute device description.
As I said, I forgot the RFC tag.
I just wanted to start the discussion and it was fruitful, no regret.
Quoted 11 lines
It has a memory allocation API. It is the device attribute, not strictly tied to ONLY TO computing device.
So at least, I am asking to have concrete
proposal on "compute device" schematic rather than start with memory API and rubber stamp as new device adds anything in future.When we added any all the class devices to DPDK, Everyone had a complete view of it is
function(at RFC of each subsystem had enough API to express the "basic" usage)
and purpose from the _application_ PoV. I see that is missing here.
I keep explaining in emails while preparing a v2.
Now that we go into circles, let's wait the v2 which will address a lot of comments.
Quoted 68 lines · Thomas Monjalon
08/06/2021 06:10, Jerin Jacob:
Quoted 42 lines · Thomas Monjalon
Quoted 41 lines
07/06/2021 15:54, Jerin Jacob:
Quoted 21 lines · Thomas Monjalon
Quoted 8 lines
07/06/2021 09:20, Wang, Haiyue:
Quoted 7 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.Quoted 8 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?Each device behaves differently at this level. In the view of the generic application, the architecture should like
Quoted 7 lines
< Use DPDK subsystem as
rte_ethdev,rte_bbdevetc for SPECIFIC function > ^
|
< DPDK driver>
^
|
<rte_devicewith this new callbacks >I think the formatting went wrong above.
I would add more to the block diagram:
class device API - computing device API
| | | class device driver - computing device driver | |EAL device with memory callback
Quoted 4 lines
Quoted 2 lines
The idea above is that the class device driver can use services of the new computing device library.
Yes. The question is, do we need any public DPDK _application_ APIs for that?
To have something generic!
Quoted 2 lines
If it is public API then the scope is much bigger than that as the application can use it directly and it makes it non portable.
It is a non-sense. If we make an API, it will be better portable. The only part which is non-portable is the program on the device which may be different per computing device.
The synchronization with the DPDK application should be portable if we define some good API.Quoted 9 lines
if the scope is only, the class driver consumption then the existing "bus"
_kindof_
abstraction/API makes sense to me.Where it abstracts,
(quoted diff) +0
-3-FW download of device -Memory management of device -Opaque way to enq/deque jobs to the device.
Quoted 7 lines
Quoted 3 lines
And above should be consumed by "class driver" not "application".
If the application doing do that, we are in
rte_rawdevice territory.I'm sorry I don't understand what you make such assertion. It seems you don't want generic API (which is the purpose of DPDK).
The FW/kernel/"computing tasks" in the co-processor can be doing anything, as it has been in FPGA/rawdev.
If there is no defined input & output of that computing task, an application developed using it will be specific to that computing task, this is not portable and feels like how rawdev works.
It is possible to have a generic API for control, to start the task and get completion notification, but not having common input/output interface with computing task still has same problem I think.
If the application is strictly depends to what computing task does, why not extending rawdev to have the control APIs? Instead of new library. And as you already said for memory, generic APIs can be used with additional flags and using rawdev handler.
Or another option can be defining computing task a little more, have a common interface, like mbuf, and add some capabilities/flags to let application know more about computing task and give decision based on it, is this the intention?
15/06/2021 20:24, Ferruh Yigit:
Quoted 88 lines · Thomas Monjalon
Quoted 68 lines
08/06/2021 06:10, Jerin Jacob:
Quoted 42 lines · Thomas Monjalon
Quoted 41 lines
07/06/2021 15:54, Jerin Jacob:
Quoted 21 lines · Thomas Monjalon
Quoted 8 lines
07/06/2021 09:20, Wang, Haiyue:
Quoted 7 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 5 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts its own memory that need to be managed by the application, can use these APIs.Quoted 8 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?Each device behaves differently at this level. In the view of the generic application, the architecture should like
Quoted 7 lines
< Use DPDK subsystem as
rte_ethdev,rte_bbdevetc for SPECIFIC function > ^
|
< DPDK driver>
^
|
<rte_devicewith this new callbacks >I think the formatting went wrong above.
I would add more to the block diagram:
class device API - computing device API
| | | class device driver - computing device driver | |EAL device with memory callback
Quoted 4 lines
Quoted 2 lines
The idea above is that the class device driver can use services of the new computing device library.
Yes. The question is, do we need any public DPDK _application_ APIs for that?
To have something generic!
Quoted 2 lines
If it is public API then the scope is much bigger than that as the application can use it directly and it makes it non portable.
It is a non-sense. If we make an API, it will be better portable. The only part which is non-portable is the program on the device which may be different per computing device.
The synchronization with the DPDK application should be portable if we define some good API.Quoted 9 lines
if the scope is only, the class driver consumption then the existing "bus"
_kindof_
abstraction/API makes sense to me.Where it abstracts,
(quoted diff) +0
-3-FW download of device -Memory management of device -Opaque way to enq/deque jobs to the device.Quoted 7 lines
Quoted 3 lines
And above should be consumed by "class driver" not "application".
If the application doing do that, we are in
rte_rawdevice territory.I'm sorry I don't understand what you make such assertion. It seems you don't want generic API (which is the purpose of DPDK).
The FW/kernel/"computing tasks" in the co-processor can be doing anything, as it has been in FPGA/rawdev.
If there is no defined input & output of that computing task, an application developed using it will be specific to that computing task, this is not portable and feels like how rawdev works.
It is possible to have a generic API for control, to start the task and get completion notification, but not having common input/output interface with computing task still has same problem I think.
Quoted 8 lines
If the application is strictly depends to what computing task does, why not extending rawdev to have the control APIs? Instead of new library. And as you already said for memory, generic APIs can be used with additional flags and using rawdev handler.
Or another option can be defining computing task a little more, have a common interface, like mbuf, and add some capabilities/flags to let application know more about computing task and give decision based on it, is this the intention?
I think we'll propose a thin layer to allow device memory management with generic API in EAL and mbuf.
The task should be defined and controlled by the application, and there is not much DPDK can do generically.
Stay tuned, and thanks for all the feedbacks.
Quoted 36 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>Quoted 7 lines · Thomas Monjalon
07/06/2021 09:20, Wang, Haiyue:
Quoted 5 lines
From: Honnappa Nagarahalli <[email protected]>
Quoted 4 lines
If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own local memory. May be some of the APIs could use generic names. For ex: instead of calling it as "
rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This wayany future device which hosts its own memory that need to be managed by the application, can use these APIs.
Quoted 10 lines
Quoted 3 lines
Quoted 1 line
"
rte_dev_malloc" sounds a good name,Yes I like the idea.
2 concerns:1/ Device memory allocation requires a device handle. So far we avoided exposing
rte_deviceto the application. How should we get a device handle from a DPDK application?Each device behaves differently at this level. In the view of the generic application, the architecture should like
< Use DPDK subsystem as
rte_ethdev,rte_bbdevetc for SPECIFIC function > ^ |
< DPDK driver>
^
|
<rte_devicewith this new callbacks >
Quoted 8 lines
An implementation may decide to have "in tree" or "out of tree" drivers or
rte_deviceimplementaion.
But generic DPDK applications should not use devices directly. i.erte_deviceneed to have this callback and mlx ethdev/crypto driver use this driver to implement public API.
Otherwise, it is the same as rawdev in DPDK.
So not sure what it brings other than raw dev here if we are not taking the above architecture.
Agree, I think it is important to hide the device under the APIs for the application to benefit.
Quoted 22 lines
Quoted 4 lines
2/ Implementation must be done in a driver. Should it be a callback defined at
rte_devicelevel?IMO, Yes and DPDK subsystem drivers to use it.
Quoted 7 lines
Quoted 7 lines
then looks like we need to enhance the 'struct
rte_device' with some new ops as:eal: move DMA mapping from bus-specific to generic driver
https://patchwork.dpdk.org/project/dpdk/patch/20210331224547.2217759
Quoted 7 lines
Quoted 1 line
Not sure the above patch is a good idea.
Let's discuss this DMA detail later :)
Quoted 24 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Thomas Monjalon
Sent: Thursday, June 3, 2021 04:36
To: [email protected]
Cc: Elena Agostini <[email protected]>
Subject: [dpdk-dev] [PATCH] gpudev: introduce memory APIFrom: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
The infrastructure is prepared to welcome drivers in
drivers/gpu/as the upcoming NVIDIA one, implementing the gpudev API. Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.
Quoted 2 lines
The next step should focus on GPU processing task control.
Is this patch for 'L2FWD-NV Workload on GPU' on P26 ?
https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9730-packet-processing-on-gpu-at-100gbe-line-rate.pdf
Quoted 36 lines
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>--- .gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 ++ doc/guides/gpus/features/default.ini | 13 ++ doc/guides/gpus/index.rst | 11 ++ doc/guides/gpus/overview.rst | 7 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpu.rst | 5 + doc/guides/prog_guide/index.rst | 1 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpu_driver.h | 44 +++++++ lib/gpudev/meson.build | 9 ++ lib/gpudev/rte_gpudev.h | 183 +++++++++++++++++++++++++++ lib/gpudev/version.map | 11 ++ lib/meson.build | 1 + 18 files changed, 308 insertions(+) create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644 doc/guides/gpus/overview.rst create mode 100644 doc/guides/prog_guide/gpu.rst create mode 100644 drivers/gpu/meson.build create mode 100644 lib/gpudev/gpu_driver.h create mode 100644 lib/gpudev/meson.build create mode 100644 lib/gpudev/rte_gpudev.h create mode 100644 lib/gpudev/version.map--
2.31.1
04/06/2021 07:51, Wang, Haiyue:
Quoted 17 lines
Quoted 17 lines
From: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
The infrastructure is prepared to welcome drivers in
drivers/gpu/as the upcoming NVIDIA one, implementing the gpudev API. Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.
Quoted 5 lines
Quoted 2 lines
The next step should focus on GPU processing task control.
Is this patch for 'L2FWD-NV Workload on GPU' on P26 ?
https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9730-packet-processing-on-gpu-at-100gbe-line-rate.pdf
Yes this is the same project: use GPU in DPDK workload.
Quoted 76 lines
-----Original Message-----
From: dev <[email protected]> On Behalf Of Thomas Monjalon
Sent: Thursday, June 3, 2021 04:36
To: [email protected]
Cc: Elena Agostini <[email protected]>
Subject: [dpdk-dev] [PATCH] gpudev: introduce memory APIFrom: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
The infrastructure is prepared to welcome drivers in
drivers/gpu/as the upcoming NVIDIA one, implementing the gpudev API. Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.The next step should focus on GPU processing task control.
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>--- .gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 ++ doc/guides/gpus/features/default.ini | 13 ++ doc/guides/gpus/index.rst | 11 ++ doc/guides/gpus/overview.rst | 7 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpu.rst | 5 + doc/guides/prog_guide/index.rst | 1 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpu_driver.h | 44 +++++++ lib/gpudev/meson.build | 9 ++ lib/gpudev/rte_gpudev.h | 183 +++++++++++++++++++++++++++ lib/gpudev/version.map | 11 ++ lib/meson.build | 1 + 18 files changed, 308 insertions(+) create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644 doc/guides/gpus/overview.rst create mode 100644 doc/guides/prog_guide/gpu.rst create mode 100644 drivers/gpu/meson.build create mode 100644 lib/gpudev/gpu_driver.h create mode 100644 lib/gpudev/meson.build create mode 100644 lib/gpudev/rte_gpudev.h create mode 100644 lib/gpudev/version.maplib/gpudev/gpu_driver.h new file +16
-0+#include <stdint.h> + +#include <rte_common.h> + +#include "rte_gpudev.h" + +struct rte_gpu_dev ; + +typedef int (*gpu_malloc_t )(struct rte_gpu_dev *dev,size_t size,void **ptr); +typedef int (*gpu_free_t )(struct rte_gpu_dev *dev,void *ptr); + +struct rte_gpu_dev { +/* Backing device. */ +struct rte_device *device; +/* GPU info structure. */ +struct rte_gpu_info info;
lib/gpudev/gpu_driver.h line 33
Quoted 10 lines
lib/gpudev/gpu_driver.h new file +10
-0+/* Counter of processes using the device. */ +uint16_t process_cnt; +/* If device is currently used or not. */ +enum rte_gpu_state state; +/* FUNCTION: Allocate memory on the GPU. */ +gpu_malloc_t gpu_malloc; +/* FUNCTION: Allocate memory on the CPU visible from the GPU. */ +gpu_malloc_t gpu_malloc_visible; +/* FUNCTION: Free allocated memory on the GPU. */ +gpu_free_t gpu_free;
I'm wondering that we can define the malloc type as:
typedef int (*gpu_malloc_t)(struct rte_gpu_dev *dev, size_t size, void **ptr,
unsigned int flags)
#define RTE_GPU_MALLOC_F_CPU_VISIBLE 0x01u --> gpu_malloc_visible
Then only one malloc function member is needed, paired with 'gpu_free'.
Quoted 35 lines
(quoted diff) +33
-0+ /* Device interrupt handle. */ + struct rte_intr_handle *intr_handle; + /* Driver-specific private data. */ + void *dev_private; +} __rte_cache_aligned; + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Allocate a chunk of memory on the GPU. + * + * @param gpu_id + * GPU ID to allocate memory. + * @param size + * Number of bytes to allocate. + * @param ptr + * Pointer to store the address of the allocated memory. + * + * @return + * 0 on success, -1 otherwise. + */ +__rte_experimental +int rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice. + * + * Allocate a chunk of memory on the CPU that is visible from the GPU. + * + * @param gpu_id + * Reference GPU ID.
Review
Quoted 10 lines
(quoted diff) +10
-0+ * @param size + * Number of bytes to allocate. + * @param ptr + * Pointer to store the address of the allocated memory. + * + * @return + * 0 on success, -1 otherwise. + */ +__rte_experimental +int rte_gpu_malloc_visible(uint16_t gpu_id, size_t size, void **ptr);
Then 'rte_gpu_malloc_visible' is no needed, and the new call is:
rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr, RTE_GPU_MALLOC_F_CPU_VISIBLE).
Also, we can define more flags for feature extension. ;-)
Quoted 5 lines
lib/gpudev/rte_gpudev.h new file +3
-1+ +#ifdef __cplusplus +} --2.31.1
04/06/2021 13:07, Wang, Haiyue:
Quoted 4 lines
Quoted 4 lines
From: Elena Agostini <[email protected]>
lib/gpudev/gpu_driver.h new file +3
-0+typedef int (*gpu_malloc_t )(struct rte_gpu_dev *dev,size_t size,void **ptr); +typedef int (*gpu_free_t )(struct rte_gpu_dev *dev,void *ptr); +
[...]
Quoted 16 lines
lib/gpudev/gpu_driver.h line 33
Quoted 6 lines
lib/gpudev/gpu_driver.h new file +6
-0+/* FUNCTION: Allocate memory on the GPU. */ +gpu_malloc_t gpu_malloc; +/* FUNCTION: Allocate memory on the CPU visible from the GPU. */ +gpu_malloc_t gpu_malloc_visible; +/* FUNCTION: Free allocated memory on the GPU. */ +gpu_free_t gpu_free;I'm wondering that we can define the malloc type as:
typedef int (*gpu_malloc_t)(struct
rte_gpu_dev*dev,size_tsize, void **ptr,
unsigned int flags)#define
RTE_GPU_MALLOC_F_CPU_VISIBLE0x01u-->gpu_malloc_visibleThen only one malloc function member is needed, paired with '
gpu_free'.
[...]
Quoted 1 line
Quoted 1 line
+int
rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr);
[...]
Quoted 7 lines
Quoted 1 line
+int
rte_gpu_malloc_visible(uint16_t gpu_id, size_t size, void **ptr);Then '
rte_gpu_malloc_visible' is no needed, and the new call is:
rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr, RTE_GPU_MALLOC_F_CPU_VISIBLE).Also, we can define more flags for feature extension. ;-)
Yes it is a good idea.
Another question is about the function rte_gpu_free().
How do we recognize that a memory chunk is from the CPU and GPU visible, or just from GPU?
Quoted 38 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Friday, June 4, 2021 20:44
To: Wang, Haiyue <[email protected]>
Cc: [email protected]; Elena Agostini <[email protected]>
Subject: Re: [dpdk-dev] [PATCH] gpudev: introduce memory API04/06/2021 13:07, Wang, Haiyue:
Quoted 4 lines
Quoted 4 lines
From: Elena Agostini <[email protected]>
lib/gpudev/gpu_driver.h new file +3
-0+typedef int (*gpu_malloc_t )(struct rte_gpu_dev *dev,size_t size,void **ptr); +typedef int (*gpu_free_t )(struct rte_gpu_dev *dev,void *ptr); +[...]
Quoted 16 lines
lib/gpudev/gpu_driver.h line 33
Quoted 6 lines
lib/gpudev/gpu_driver.h new file +6
-0+/* FUNCTION: Allocate memory on the GPU. */ +gpu_malloc_t gpu_malloc; +/* FUNCTION: Allocate memory on the CPU visible from the GPU. */ +gpu_malloc_t gpu_malloc_visible; +/* FUNCTION: Free allocated memory on the GPU. */ +gpu_free_t gpu_free;I'm wondering that we can define the malloc type as:
typedef int (*gpu_malloc_t)(struct
rte_gpu_dev*dev,size_tsize, void **ptr,
unsigned int flags)#define
RTE_GPU_MALLOC_F_CPU_VISIBLE0x01u-->gpu_malloc_visibleThen only one malloc function member is needed, paired with '
gpu_free'.[...]
Quoted 1 line
Quoted 1 line
+int
rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr);[...]
Quoted 6 lines
Quoted 1 line
+int
rte_gpu_malloc_visible(uint16_t gpu_id, size_t size, void **ptr);Then '
rte_gpu_malloc_visible' is no needed, and the new call is:
rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr, RTE_GPU_MALLOC_F_CPU_VISIBLE).
Quoted 8 lines
Quoted 1 line
Also, we can define more flags for feature extension. ;-)
Yes it is a good idea.
Another question is about the function
rte_gpu_free().
How do we recognize that a memory chunk is from the CPU and GPU visible, or just from GPU?
I didn't find the rte_gpu_free_visible definition, and the rte_gpu_free's comment just says: deallocate a chunk of memory allocated with rte_gpu_malloc*
Looks like the rte_gpu_free can handle this case ?
And from the definition "rte_gpu_free(uint16_t gpu_id, void *ptr)", the free needs to check whether this memory belong to the GPU or not, so it also can recognize the memory type, I think.
04/06/2021 15:25, Wang, Haiyue:
Quoted 10 lines
From: Thomas Monjalon <[email protected]>
Quoted 4 lines
Another question is about the function
rte_gpu_free().
How do we recognize that a memory chunk is from the CPU and GPU visible, or just from GPU?I didn't find the
rte_gpu_free_visibledefinition, and the rte_gpu_free's comment just says: deallocate a chunk of memory allocated with rte_gpu_malloc*Looks like the
rte_gpu_freecan handle this case ?
This is the proposal, yes.
Quoted 3 lines
And from the definition "
rte_gpu_free(uint16_t gpu_id, void *ptr)", the free needs to check whether this memory belong to the GPU or not, so it also can recognize the memory type, I think.
Yes that's the idea behind having a single free function.
We could have some metadata in front of the memory chunk.
My question is to confirm whether it is a good design or not, and whether it should be driver specific or have a common struct in the lib.
Opinions?
Quoted 26 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Friday, June 4, 2021 22:06
To: Wang, Haiyue <[email protected]>
Cc: [email protected]; Elena Agostini <[email protected]>; [email protected]; Yigit, Ferruh <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] gpudev: introduce memory API04/06/2021 15:25, Wang, Haiyue:
Quoted 10 lines
From: Thomas Monjalon <[email protected]>
Quoted 4 lines
Another question is about the function
rte_gpu_free().
How do we recognize that a memory chunk is from the CPU and GPU visible, or just from GPU?I didn't find the
rte_gpu_free_visibledefinition, and the rte_gpu_free's comment just says: deallocate a chunk of memory allocated with rte_gpu_malloc*Looks like the
rte_gpu_freecan handle this case ?This is the proposal, yes.
Quoted 3 lines
And from the definition "
rte_gpu_free(uint16_t gpu_id, void *ptr)", the free needs to check whether this memory belong to the GPU or not, so it also can recognize the memory type, I think.
Quoted 7 lines
Yes that's the idea behind having a single free function.
We could have some metadata in front of the memory chunk.
My question is to confirm whether it is a good design or not, and whether it should be driver specific or have a common struct in the lib.Opinions?
Make the GPU memory to be registered into the common lib API with the metadata like address, size etc, and also some GPU specific callbacks like to handle how to make GPU memory visible to CPU ?
And the memory register can be like the exist external memory function:
int
rte_extmem_register(void *va_addr, size_t len, rte_iova_t iova_addrs[],
unsigned int n_pages, size_t page_sz)
04/06/2021 20:04, Wang, Haiyue:
Quoted 31 lines
From: Thomas Monjalon <[email protected]>
Quoted 18 lines
04/06/2021 15:25, Wang, Haiyue:
Quoted 10 lines
From: Thomas Monjalon <[email protected]>
Quoted 4 lines
Another question is about the function
rte_gpu_free().
How do we recognize that a memory chunk is from the CPU and GPU visible, or just from GPU?I didn't find the
rte_gpu_free_visibledefinition, and the rte_gpu_free's comment just says: deallocate a chunk of memory allocated with rte_gpu_malloc*Looks like the
rte_gpu_freecan handle this case ?This is the proposal, yes.
Quoted 3 lines
And from the definition "
rte_gpu_free(uint16_t gpu_id, void *ptr)", the free needs to check whether this memory belong to the GPU or not, so it also can recognize the memory type, I think.Quoted 7 lines
Yes that's the idea behind having a single free function.
We could have some metadata in front of the memory chunk.
My question is to confirm whether it is a good design or not, and whether it should be driver specific or have a common struct in the lib.Opinions?
Make the GPU memory to be registered into the common lib API with the metadata like address, size etc, and also some GPU specific callbacks like to handle how to make GPU memory visible to CPU ?
Quoted 5 lines
And the memory register can be like the exist external memory function:
int
rte_extmem_register(void*va_addr,size_tlen,rte_iova_tiova_addrs[],
unsigned intn_pages,size_tpage_sz)
How do you specify the device ID?
I may have missed something.
Quoted 40 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Saturday, June 5, 2021 15:49
To: Wang, Haiyue <[email protected]>
Cc: [email protected]; Elena Agostini <[email protected]>; [email protected]; Yigit, Ferruh <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] gpudev: introduce memory API04/06/2021 20:04, Wang, Haiyue:
Quoted 31 lines
From: Thomas Monjalon <[email protected]>
Quoted 18 lines
04/06/2021 15:25, Wang, Haiyue:
Quoted 10 lines
From: Thomas Monjalon <[email protected]>
Quoted 4 lines
Another question is about the function
rte_gpu_free().
How do we recognize that a memory chunk is from the CPU and GPU visible, or just from GPU?I didn't find the
rte_gpu_free_visibledefinition, and the rte_gpu_free's comment just says: deallocate a chunk of memory allocated with rte_gpu_malloc*Looks like the
rte_gpu_freecan handle this case ?This is the proposal, yes.
Quoted 3 lines
And from the definition "
rte_gpu_free(uint16_t gpu_id, void *ptr)", the free needs to check whether this memory belong to the GPU or not, so it also can recognize the memory type, I think.Quoted 7 lines
Yes that's the idea behind having a single free function.
We could have some metadata in front of the memory chunk.
My question is to confirm whether it is a good design or not, and whether it should be driver specific or have a common struct in the lib.Opinions?
Make the GPU memory to be registered into the common lib API with the metadata like address, size etc, and also some GPU specific callbacks like to handle how to make GPU memory visible to CPU ?
Quoted 7 lines
Quoted 5 lines
And the memory register can be like the exist external memory function:
int
rte_extmem_register(void*va_addr,size_tlen,rte_iova_tiova_addrs[],
unsigned intn_pages,size_tpage_sz)How do you specify the device ID
I mean that take the current external memory register as an example, it is not a real proto-type.
The GPU memory management library can provide the this kind of API for GPU driver to register its memory at probe time or start time ?
Quoted 6 lines
I may have missed something.
<snip>
Quoted 5 lines
From: Elena Agostini <[email protected]>
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
It would be good to explain how the application using GPU+DPDK would look like.
Which parts of the workload need DPDK's support?
Any requirements on co-existence of GPU with other accelerators?
Quoted 4 lines
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
Is this memory for packet buffers or something else?
Quoted 501 lines
The infrastructure is prepared to welcome drivers in
drivers/gpu/as the upcoming NVIDIA one, implementing the gpudev API.
Other additions planned for next revisions:
- C implementation file
- guide documentation
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.The next step should focus on GPU processing task control.
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>--- .gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 ++ doc/guides/gpus/features/default.ini | 13 ++ doc/guides/gpus/index.rst | 11 ++ doc/guides/gpus/overview.rst | 7 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpu.rst | 5 + doc/guides/prog_guide/index.rst | 1 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpu_driver.h | 44 +++++++ lib/gpudev/meson.build | 9 ++ lib/gpudev/rte_gpudev.h | 183 +++++++++++++++++++++++++++ lib/gpudev/version.map | 11 ++ lib/meson.build | 1 + 18 files changed, 308 insertions(+) create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644
doc/guides/gpus/overview.rstcreate mode 100644doc/guides/prog_guide/gpu.rstcreate mode 100644drivers/gpu/meson.buildcreate mode 100644lib/gpudev/gpu_driver.hcreate mode 100644lib/gpudev/meson.buildcreate mode 100644lib/gpudev/rte_gpudev.hcreate mode 100644lib/gpudev/version.map.gitignore +1
-0@@ -14,6 +14,7 @@ doc/guides/compressdevs/overview_feature_table.txt doc/guides/regexdevs/overview_feature_table.txt doc/guides/vdpadevs/overview_feature_table.txt doc/guides/bbdevs/overview_feature_table.txt +doc/guides/gpus/overview_feature_table.txt # ignore generated ctags/cscope files cscope.out.poMAINTAINERS +6
-0@@ -452,6 +452,12 @@ F: app/test-regex/ F: doc/guides/prog_guide/regexdev.rst F: doc/guides/regexdevs/features/default.ini +GPU API - EXPERIMENTAL +M: Elena Agostini <[email protected]> +F: lib/gpudev/ +F: doc/guides/prog_guide/gpu.rst +F: doc/guides/gpus/features/default.ini + Eventdev API M: Jerin Jacob <[email protected]> T: git://dpdk.org/next/dpdk-next-eventdevdoc/api/doxy-api-index.md index renamed from doc/api/doxy-api-index.md +0
-0
1992107a03..bd10342ca2100644doc/api/doxy-api-index.md +1
-0@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h), [eventdev] (@ref rte_eventdev.h), [event_eth_rx_adapter] (@ref rte_event_eth_rx_adapter.h), [event_eth_tx_adapter] (@ref rte_event_eth_tx_adapter.h),doc/api/doxy-api.conf.in index renamed from doc/api/doxy-api.conf.in +0
-0
325a0195c6..831b9a6b33100644doc/api/doxy-api.conf.in +1
-0@@ -40,6 +40,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api- index.md \ @TOPDIR@/lib/eventdev \ @TOPDIR@/lib/fib \ @TOPDIR@/lib/flow_classify \ + @TOPDIR@/lib/gpudev \ @TOPDIR@/lib/graph \ @TOPDIR@/lib/gro \@TOPDIR@/lib/gso \ diff --git a/doc/guides/conf.py
b/doc/guides/conf.pyindex67d2dd62c7..7930da9ceb100644doc/guides/conf.py +0
-0@@ -152,6 +152,9 @@ def generate_overview_table(output_filename,
table_id, section,table_name, titlname = ini_filename[:-4] name = name.replace('_vf', 'vf') pmd_names.append(name)doc/guides/conf.py +3
-0+ if not pmd_names: + # Add an empty column if table is empty (required by RST syntax) + pmd_names.append(' ') # Pad the table header names. max_header_len = len(max(pmd_names, key=len)) @@ -388,6 +391,11@@ def setup(app):
'Features', 'Features availability in bbdev drivers', 'Feature')doc/guides/conf.py +5
-0+ table_file = dirname(__file__) + '/gpus/overview_feature_table.txt' + generate_overview_table(table_file, 1, + 'Features', + 'Features availability in GPU drivers', + 'Feature') if LooseVersion(sphinx_version) < LooseVersion('1.3.1'): print('Upgrade sphinx to version >= 1.3.1 for 'doc/guides/gpus/features/default.ini new file +13
-0@@ -0,0 +1,13 @@ +; +; Features of a GPU driver. +; +; This file defines the features that are valid for inclusion in ; the +other driver files and also the order that they appear in ; the +features table in the documentation. The feature description ; string +should not exceed feature_str_len defined in conf.py. +; +[Features] +Get device info = +Share CPU memory with GPU = +Allocate GPU memory = +Free memory =doc/guides/gpus/index.rst new file renamed from doc/guides/gpus/index.rst +0
-0mode 100644 index 0000000000..f9c62aeb36
doc/guides/gpus/index.rst new file +11
-0@@ -0,0 +1,11 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +GPU Drivers +=========== + +.. toctree:: + :maxdepth: 2 + :numbered: + + overviewdoc/guides/gpus/overview.rst new file +7
-0@@ -0,0 +1,7 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +Overview of GPU Drivers +======================= + +.. include:: overview_feature_table.txtdoc/guides/index.rst index renamed from doc/guides/index.rst +0
-0
857f0363d3..ee4d79a4eb100644doc/guides/index.rst +1
-0@@ -21,6 +21,7 @@ DPDK documentation compressdevs/index vdpadevs/index regexdevs/index + gpus/index eventdevs/index rawdevs/index mempool/indexdoc/guides/prog_guide/gpu.rst new file +5
-0@@ -0,0 +1,5 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2021 NVIDIA Corporation & Affiliates + +GPU Library +===========(quoted diff) +0
-0
b/doc/guides/prog_guide/index.rstindex2dce507f46..dfddf90b51100644doc/guides/prog_guide/index.rst +1
-0@@ -27,6 +27,7 @@ Programmer's Guide cryptodev_lib compressdev regexdev + gpu rte_security rawdev link_bonding_poll_mode_drv_libdrivers/gpu/meson.build new file mode renamed from drivers/gpu/meson.build +0
-0100644 index 0000000000..5189950616
drivers/gpu/meson.build new file +4
-0@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright 2021 NVIDIA +Corporation & Affiliates + +drivers = []drivers/meson.build index renamed from drivers/meson.build +0
-0
bc6f4f567f..f607040d79100644drivers/meson.build +1
-0@@ -18,6 +18,7 @@ subdirs = [ 'vdpa', # depends on common, bus and mempool. 'event', # depends on common, bus, mempool and net. 'baseband', # depends on common and bus. + 'gpu', # depends on common and bus. ] if meson.is_cross_build()lib/gpudev/gpu_driver.h new file mode renamed from lib/gpudev/gpu_driver.h +0
-0100644 index 0000000000..5ff609e49d
lib/gpudev/gpu_driver.h new file +43
-0@@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright 2021 NVIDIA Corporation & Affiliates */ + +#ifndef GPU_DRIVER_H +#define GPU_DRIVER_H + +#include <stdint.h> + +#include <rte_common.h> + +#include "rte_gpudev.h" + +struct rte_gpu_dev ; + +typedef int (*gpu_malloc_t )(struct rte_gpu_dev *dev,size_t size,void +**ptr);typedef int (*gpu_free_t )(struct rte_gpu_dev *dev,void *ptr); + +struct rte_gpu_dev { +/* Backing device. */ +struct rte_device *device; +/* GPU info structure. */ +struct rte_gpu_info info; +/* Counter of processes using the device. */ +uint16_t process_cnt; +/* If device is currently used or not. */ +enum rte_gpu_state state; +/* FUNCTION: Allocate memory on the GPU. */ +gpu_malloc_t gpu_malloc; +/* FUNCTION: Allocate memory on the CPU visible from the GPU. */ +gpu_malloc_t gpu_malloc_visible; +/* FUNCTION: Free allocated memory on the GPU. */ +gpu_free_t gpu_free; +/* Device interrupt handle. */ +struct rte_intr_handle *intr_handle; +/* Driver-specific private data. */ +void *dev_private; +} __rte_cache_aligned; + +struct rte_gpu_dev *rte_gpu_dev_allocate (const char *name);struct +rte_gpu_dev *rte_gpu_dev_get_by_name (const char *name);int +rte_gpu_dev_release (struct rte_gpu_dev *gpudev); + +#endif /* GPU_DRIVER_H */ lib/gpudev/meson.build new file mode renamed from lib/gpudev/meson.build +0
-0100644 index 0000000000..f05459e18d
lib/gpudev/meson.build new file +9
-0@@ -0,0 +1,9 @@ +# SPDX-License-Identifier: BSD-3-Clause # Copyright 2021 NVIDIA +Corporation & Affiliates + +headers = files( + 'rte_gpudev.h', +) + +sources = files( +)lib/gpudev/rte_gpudev.h new file mode renamed from lib/gpudev/rte_gpudev.h +0
-0100644 index 0000000000..b12f35c17e
lib/gpudev/rte_gpudev.h new file +180
-0@@ -0,0 +1,183 @@ +/* SPDX-License-Identifier: BSD-3-Clause +* Copyright 2021 NVIDIA Corporation & Affiliates */ + +#ifndef RTE_GPUDEV_H +#define RTE_GPUDEV_H + +#include <stdint.h> +#include <stdbool.h> + +#include <rte_common.h> + +/** +* @file +* Generic library to interact with a GPU. +* +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +*/ + +#ifdef __cplusplus +extern "C" { +#endif + +/** Maximum number of GPU engines. */ +#define RTE_GPU_MAX_DEVS UINT16_C(32) +/** Maximum length of device name. */ +#define RTE_GPU_NAME_MAX_LEN 128 + +/** Flags indicate current state of GPU device. */ enum rte_gpu_state { +RTE_GPU_STATE_UNUSED ,/**< not initialized */ +RTE_GPU_STATE_INITIALIZED ,/**< initialized */ +}; + +/** Store a list of info for a given GPU. */ struct rte_gpu_info { +/** GPU device ID. */ +uint16_t gpu_id; +/** Unique identifier name. */ +char name[RTE_GPU_NAME_MAX_LEN ]; +/** Total memory available on device. */ +size_t total_memory; +/** Total processors available on device. */ +int processor_count; +}; + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Returns the number of GPUs detected and associated to DPDK. +* +* @return +* The number of available GPUs. +*/ +__rte_experimental +uint16_t rte_gpu_dev_count_avail(void); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Check if the device is valid and initialized in DPDK. +* +* @param gpu_id +* The input GPU ID. +* +* @return +* - True if gpu_id is a valid and initialized GPU. +* - False otherwise. +*/ +__rte_experimental +bool rte_gpu_dev_is_valid(uint16_t gpu_id); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Get the GPU ID of the next valid GPU initialized in DPDK. +* +* @param gpu_id +* The initial GPU ID to start the research. +* +* @return +* Next GPU ID corresponding to a valid and initialized GPU device. +*/ +__rte_experimental +uint16_t rte_gpu_dev_find_next(uint16_t gpu_id); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Macro to iterate over all valid GPUs. +* +* @param gpu_id +* The ID of the next possible valid GPU. +* @return +* Next valid GPU ID, RTE_GPU_MAX_DEVS if there is none. +*/ +#define RTE_GPU_FOREACH_DEV (gpu_id) \ + for (gpu_id = rte_gpu_find_next(0); \ + gpu_id < RTE_GPU_MAX_DEVS; \ + gpu_id = rte_gpu_find_next(gpu_id + 1)) + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Return GPU specific info. +* +* @param gpu_id +* GPU ID to get info. +* @param info +* Memory structure to fill with the info. +* +* @return +* 0 on success, -1 otherwise. +*/ +__rte_experimental +int rte_gpu_dev_info_get(uint16_t gpu_id, struct rte_gpu_info **info); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Allocate a chunk of memory on the GPU. +* +* @param gpu_id +* GPU ID to allocate memory. +* @param size +* Number of bytes to allocate. +* @param ptr +* Pointer to store the address of the allocated memory. +* +* @return +* 0 on success, -1 otherwise. +*/ +__rte_experimental +int rte_gpu_malloc(uint16_t gpu_id, size_t size, void **ptr); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Allocate a chunk of memory on the CPU that is visible from the GPU. +* +* @param gpu_id +* Reference GPU ID. +* @param size +* Number of bytes to allocate. +* @param ptr +* Pointer to store the address of the allocated memory. +* +* @return +* 0 on success, -1 otherwise. +*/ +__rte_experimental +int rte_gpu_malloc_visible(uint16_t gpu_id, size_t size, void **ptr); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Deallocate a chunk of memory allocated with rte_gpu_malloc*. +* +* @param gpu_id +* Reference GPU ID. +* @param ptr +* Pointer to the memory area to be deallocated. +* +* @return +* 0 on success, -1 otherwise. +*/ +__rte_experimental +int rte_gpu_free(uint16_t gpu_id, void *ptr); + +#ifdef __cplusplus +} +#endif + +#endif /* RTE_GPUDEV_H */ lib/gpudev/version.map new file mode renamed from lib/gpudev/version.map +0
-0100644 index 0000000000..9e0f218e8b
lib/gpudev/version.map new file +11
-0@@ -0,0 +1,11 @@ +EXPERIMENTAL { + global: + + rte_gpu_dev_count_avail; + rte_gpu_dev_find_next; + rte_gpu_dev_info_get; + rte_gpu_dev_is_valid; + rte_gpu_free; + rte_gpu_malloc; + rte_gpu_malloc_visible; +};lib/meson.build index 4a64756a68..ffefc64c69 renamed from lib/meson.build +0
-0100644
lib/meson.build +1
-1@@ -33,6 +33,7 @@ libraries = [ 'distributor', 'efd', 'eventdev', + 'gpudev', 'gro', 'gso', 'ip_frag', --2.31.1
06/06/2021 03:10, Honnappa Nagarahalli:
Quoted 4 lines
Quoted 2 lines
The new library gpudev is for dealing with GPU from a DPDK application in a vendor-agnostic way.
It would be good to explain how the application using GPU+DPDK would look like.
It can be anything to invent.
We can give few more words saying it is a computing done in parallel, being either a loop or sequence of tasks. In both cases, the DPDK application running on the CPU must be in control.
Quoted 1 line
Which parts of the workload need DPDK's support?
Control:
- start task (by function or shared memory variable)
- get completion notification (most probably by shared memory)
We will add such requirements in v2.
Quoted 1 line
Any requirements on co-existence of GPU with other accelerators?
It must be generic to allow any workload with any device.
Quoted 5 lines
Quoted 3 lines
As a first step, the features are focused on memory management. A function allows to allocate memory inside the GPU, while another one allows to use main (CPU) memory from the GPU.
Is this memory for packet buffers or something else?
Can be for packets or for control variables.
v2 cover heterogeneous computing library
In reply to Thomas MonjalonFrom: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices like GPUs.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible from the device
- Communication functions to enhance the dialog between the CPU and the device
The infrastructure is prepared to welcome drivers in drivers/hc/ as the upcoming NVIDIA one, implementing the hcdev API.
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration in devtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.
Below is a pseudo-code to give an example about how to use functions in this library in case of a CUDA application.
Elena Agostini (4):
hcdev: introduce heterogeneous computing device library
hcdev: add memory API
hcdev: add communication flag
hcdev: add communication list
Thomas Monjalon (3):
hcdev: add event notification
hcdev: add child device representing a device context
hcdev: support multi-process
.gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 + doc/guides/hcdevs/features/default.ini | 13 + doc/guides/hcdevs/index.rst | 11 + doc/guides/hcdevs/overview.rst | 11 + doc/guides/index.rst | 1 + doc/guides/prog_guide/hcdev.rst | 5 + doc/guides/prog_guide/index.rst | 1 + doc/guides/rel_notes/release_21_08.rst | 5 + drivers/hc/meson.build | 4 + drivers/meson.build | 1 + lib/hcdev/hcdev.c | 789 +++++++++++++++++++++++++ lib/hcdev/hcdev_driver.h | 96 +++ lib/hcdev/meson.build | 12 + lib/hcdev/rte_hcdev.h | 592 +++++++++++++++++++ lib/hcdev/version.map | 35 ++ lib/meson.build | 1 + 20 files changed, 1594 insertions(+) create mode 100644 doc/guides/hcdevs/features/default.ini create mode 100644 doc/guides/hcdevs/index.rst create mode 100644 doc/guides/hcdevs/overview.rst create mode 100644 doc/guides/prog_guide/hcdev.rst create mode 100644 drivers/hc/meson.build create mode 100644 lib/hcdev/hcdev.c create mode 100644 lib/hcdev/hcdev_driver.h create mode 100644 lib/hcdev/meson.build create mode 100644 lib/hcdev/rte_hcdev.h create mode 100644 lib/hcdev/version.map
//////////////////////////////////////////////////////////////////////// ///// HCDEV library + CUDA functions //////////////////////////////////////////////////////////////////////// #define GPU_PAGE_SHIFT 16
#define GPU_PAGE_SIZE (1UL << GPU_PAGE_SHIFT)
int main() {
struct rte_hcdev_flag quit_flag;
struct rte_hcdev_comm_list *comm_list;
int nb_rx = 0;
int comm_list_entry = 0;
struct rte_mbuf * rx_mbufs[max_rx_mbufs];
cudaStream_t cstream;
struct rte_mempool *mpool_payload, *mpool_header;
struct rte_pktmbuf_extmem ext_mem;
int16_t dev_id;
/* Initialize CUDA objects (cstream, context, etc..). */
/* Use hcdev library to register a new CUDA context if any */
/* Let's assume the application wants to use the default context of the GPU device 0 */
dev_id = 0;
/* Create an external memory mempool using memory allocated on the GPU. */
ext_mem.elt_size = mbufs_headroom_size;
ext_mem.buf_len = RTE_ALIGN_CEIL(mbufs_num * ext_mem.elt_size, GPU_PAGE_SIZE);
ext_mem.buf_iova = RTE_BAD_IOVA;
ext_mem.buf_ptr = rte_hcdev_malloc(dev_id, ext_mem.buf_len, 0);
rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len, NULL, ext_mem.buf_iova, GPU_PAGE_SIZE);
rte_dev_dma_map(rte_eth_devices[l2fwd_port_id].device, ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len);
mpool_payload = rte_pktmbuf_pool_create_extbuf("gpu_mempool", mbufs_num,
0, 0, ext_mem.elt_size,
rte_socket_id(), &ext_mem, 1);
/*
* Create CPU - device communication flag. With this flag, the CPU can tell to the CUDA kernel
* to exit from the main loop.
*/
rte_hcdev_comm_create_flag(dev_id, &quit_flag, RTE_HCDEV_COMM_FLAG_CPU);
rte_hcdev_comm_set_flag(&quit_flag, 0);
/*
* Create CPU - device communication list. Each entry of this list will be populated by the CPU
* with a new set of received mbufs that the CUDA kernel has to process.
*/
comm_list = rte_hcdev_comm_create_list(dev_id, num_entries);
/* A very simple CUDA kernel with just 1 CUDA block and RTE_HCDEV_COMM_LIST_PKTS_MAX CUDA threads. */
cuda_kernel_packet_processing<<<1, RTE_HCDEV_COMM_LIST_PKTS_MAX, 0, cstream>>>(quit_flag->ptr, comm_list, num_entries, ...);
/*
* For simplicity, the CPU here receives only 2 bursts of mbufs.
* In a real application, network activity and device processing should overlap.
*/
nb_rx = rte_eth_rx_burst(port_id, queue_id, &(rx_mbufs[0]), max_rx_mbufs);
rte_hcdev_comm_populate_list_pkts(comm_list[0], rx_mbufs, nb_rx);
nb_rx = rte_eth_rx_burst(port_id, queue_id, &(rx_mbufs[0]), max_rx_mbufs);
rte_hcdev_comm_populate_list_pkts(comm_list[1], rx_mbufs, nb_rx);
/*
* CPU waits for the completion of the packets' processing on the CUDA kernel
* and then it does a cleanup of the received mbufs.
*/
while (rte_hcdev_comm_cleanup_list(comm_list[0]));
while (rte_hcdev_comm_cleanup_list(comm_list[1]));
/* CPU notifies the CUDA kernel that it has to terminate */
rte_hcdev_comm_set_flag(&quit_flag, 1);
/* hcdev objects cleanup/destruction */
/* CUDA cleanup */
/* DPDK cleanup */
return 0;
}//////////////////////////////////////////////////////////////////////// ///// CUDA kernel ////////////////////////////////////////////////////////////////////////
void cuda_kernel(uint32_t * quit_flag_ptr, struct rte_hcdev_comm_list *comm_list, int comm_list_entries) {
int comm_list_index = 0;
struct rte_hcdev_comm_pkt *pkt_list = NULL;
/* Do some pre-processing operations. */
/* GPU kernel keeps checking this flag to know if it has to quit or wait for more packets. */
while (*quit_flag_ptr == 0)
{
if (comm_list[comm_list_index]->status != RTE_HCDEV_COMM_LIST_READY)
continue;
if (threadIdx.x < comm_list[comm_list_index]->num_pkts)
{
/* Each CUDA thread processes a different packet. */
packet_processing(comm_list[comm_list_index]->addr, comm_list[comm_list_index]->size, ..);
}
__threadfence();
__syncthreads();
/* Wait for new packets on the next communication list entry. */
comm_list_index = (comm_list_index+1) % comm_list_entries;
}
/* Do some post-processing operations. */
}Signature · 1 line
2.31.1
RFC PATCH v2 1/7 hcdev: introduce heterogeneous computing device library
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The new library hcdev is for dealing with computing devices from a DPDK application running on the CPU.
The infrastructure is prepared to welcome drivers in drivers/hc/.
--- .gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 + doc/guides/hcdevs/features/default.ini | 10 + doc/guides/hcdevs/index.rst | 11 ++ doc/guides/hcdevs/overview.rst | 11 ++ doc/guides/index.rst | 1 + doc/guides/prog_guide/hcdev.rst | 5 + doc/guides/prog_guide/index.rst | 1 + doc/guides/rel_notes/release_21_08.rst | 4 + drivers/hc/meson.build | 4 + drivers/meson.build | 1 + lib/hcdev/hcdev.c | 249 +++++++++++++++++++++++++ lib/hcdev/hcdev_driver.h | 67 +++++++ lib/hcdev/meson.build | 10 + lib/hcdev/rte_hcdev.h | 169 +++++++++++++++++ lib/hcdev/version.map | 20 ++ lib/meson.build | 1 + 20 files changed, 581 insertions(+) create mode 100644 doc/guides/hcdevs/features/default.ini create mode 100644 doc/guides/hcdevs/index.rst create mode 100644 doc/guides/hcdevs/overview.rst create mode 100644 doc/guides/prog_guide/hcdev.rst create mode 100644 drivers/hc/meson.build create mode 100644 lib/hcdev/hcdev.c create mode 100644 lib/hcdev/hcdev_driver.h create mode 100644 lib/hcdev/meson.build create mode 100644 lib/hcdev/rte_hcdev.h create mode 100644 lib/hcdev/version.map
.gitignore +1 -0
@@ -14,6 +14,7 @@ doc/guides/compressdevs/overview_feature_table.txt doc/guides/regexdevs/overview_feature_table.txt doc/guides/vdpadevs/overview_feature_table.txt doc/guides/bbdevs/overview_feature_table.txt +doc/guides/hcdevs/overview_feature_table.txt # ignore generated ctags/cscope files cscope.out.po
MAINTAINERS +6 -0
@@ -452,6 +452,12 @@ F: app/test-regex/ F: doc/guides/prog_guide/regexdev.rst F: doc/guides/regexdevs/features/default.ini +Heterogeneous Computing API - EXPERIMENTAL +M: Elena Agostini <[email protected]> +F: lib/hcdev/ +F: doc/guides/prog_guide/hcdev.rst +F: doc/guides/hcdevs/features/default.ini + Eventdev API M: Jerin Jacob <[email protected]> T: git://dpdk.org/next/dpdk-next-eventdev
doc/api/doxy-api-index.md +1 -0
@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [hcdev] (@ref rte_hcdev.h), [eventdev] (@ref rte_eventdev.h), [event_eth_rx_adapter] (@ref rte_event_eth_rx_adapter.h), [event_eth_tx_adapter] (@ref rte_event_eth_tx_adapter.h),
doc/api/doxy-api.conf.in +1 -0
@@ -44,6 +44,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/gro \ @TOPDIR@/lib/gso \ @TOPDIR@/lib/hash \ + @TOPDIR@/lib/hcdev \ @TOPDIR@/lib/ip_frag \ @TOPDIR@/lib/ipsec \ @TOPDIR@/lib/jobstats \
doc/guides/conf.py +8 -0
@@ -152,6 +152,9 @@ def generate_overview_table(output_filename, table_id, section, table_name, titl name = ini_filename[:-4] name = name.replace('_vf', 'vf') pmd_names.append(name) + if not pmd_names: + # Add an empty column if table is empty (required by RST syntax) + pmd_names.append(' ') # Pad the table header names. max_header_len = len(max(pmd_names, key=len))
@@ -388,6 +391,11 @@ def setup(app): 'Features', 'Features availability in bbdev drivers', 'Feature') + table_file = dirname(__file__) + '/hcdevs/overview_feature_table.txt' + generate_overview_table(table_file, 1, + 'Features', + 'Features availability in hcdev drivers', + 'Feature') if LooseVersion(sphinx_version) < LooseVersion('1.3.1'): print('Upgrade sphinx to version >= 1.3.1 for '
doc/guides/hcdevs/features/default.ini new file +10 -0
@@ -0,0 +1,10 @@ +; +; Features of heterogeneous device driver. +; +; This file defines the features that are valid for inclusion in +; the other driver files and also the order that they appear in +; the features table in the documentation. The feature description +; string should not exceed feature_str_len defined in conf.py. +; +[Features] +Get device info =
doc/guides/hcdevs/index.rst new file +11 -0
@@ -0,0 +1,11 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2021 NVIDIA Corporation & Affiliates + +Heterogeneous Computing Device Drivers +====================================== + +.. toctree:: + :maxdepth: 2 + :numbered: + + overview
doc/guides/hcdevs/overview.rst new file +11 -0
@@ -0,0 +1,11 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2021 NVIDIA Corporation & Affiliates + +Overview of Heterogeneous Computing Drivers +=========================================== + +Heterogeneous computing device may refer to any computing unit +able to process data and to share some memory with the CPU. +Examples are GPU or specialized processor in a SoC. + +.. include:: overview_feature_table.txt
doc/guides/index.rst +1 -0
@@ -21,6 +21,7 @@ DPDK documentation compressdevs/index vdpadevs/index regexdevs/index + hcdevs/index eventdevs/index rawdevs/index mempool/index
doc/guides/prog_guide/hcdev.rst new file +5 -0
@@ -0,0 +1,5 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2021 NVIDIA Corporation & Affiliates + +Heterogeneous Computing Device Library +======================================
doc/guides/prog_guide/index.rst +1 -0
@@ -27,6 +27,7 @@ Programmer's Guide cryptodev_lib compressdev regexdev + hcdev rte_security rawdev link_bonding_poll_mode_drv_lib
doc/guides/rel_notes/release_21_08.rst +4 -0
@@ -55,6 +55,10 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Introduced Heterogeneous Computing Device library with first features:** + + * Device information + * **Added auxiliary bus support.** Auxiliary bus provides a way to split function into child-devices
drivers/hc/meson.build new file +4 -0
@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2021 NVIDIA Corporation & Affiliates + +drivers = []
drivers/meson.build +1 -0
@@ -18,6 +18,7 @@ subdirs = [ 'vdpa', # depends on common, bus and mempool. 'event', # depends on common, bus, mempool and net. 'baseband', # depends on common and bus. + 'hc', # depends on common and bus. ] if meson.is_cross_build()
lib/hcdev/hcdev.c new file +249 -0
249 lines not on this page · raw message
lib/hcdev/hcdev_driver.h new file +67 -0
67 lines not on this page · raw message
lib/hcdev/meson.build new file +10 -0
@@ -0,0 +1,10 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2021 NVIDIA Corporation & Affiliates + +headers = files( + 'rte_hcdev.h', +) + +sources = files( + 'hcdev.c', +)
lib/hcdev/rte_hcdev.h new file +169 -0
169 lines not on this page · raw message
lib/hcdev/version.map new file +20 -0
@@ -0,0 +1,20 @@ +EXPERIMENTAL { + global: + + # added in 21.11 + rte_hcdev_close; + rte_hcdev_count_avail; + rte_hcdev_find_next; + rte_hcdev_info_get; + rte_hcdev_init; + rte_hcdev_is_valid; +}; + +INTERNAL { + global: + + rte_hcdev_allocate; + rte_hcdev_complete_new; + rte_hcdev_get_by_name; + rte_hcdev_release; +};
lib/meson.build +1 -0
@@ -35,6 +35,7 @@ libraries = [ 'eventdev', 'gro', 'gso', + 'hcdev', 'ip_frag', 'jobstats', 'kni',
Signature · 1 line
2.31.1
RFC PATCH v2 2/7 hcdev: add event notification
In reply to Thomas MonjalonCallback functions may be registered for a device event. Callback management is per-process and not thread-safe.
The events RTE_HCDEV_EVENT_NEW and RTE_HCDEV_EVENT_DEL are notified respectively after creation and before removal of a device, as part of the library functions. Some future events may be emitted from drivers.
--- lib/hcdev/hcdev.c | 137 +++++++++++++++++++++++++++++++++++++++ lib/hcdev/hcdev_driver.h | 7 ++ lib/hcdev/rte_hcdev.h | 71 ++++++++++++++++++++ lib/hcdev/version.map | 3 + 4 files changed, 218 insertions(+)
lib/hcdev/hcdev.c +137 -0
@@ -3,6 +3,7 @@*/ #include <rte_eal.h> +#include <rte_tailq.h> #include <rte_string_fns.h> #include <rte_errno.h> #include <rte_log.h>
@@ -27,6 +28,15 @@ static int16_t hcdev_max;/* Number of currently valid devices */ static int16_t hcdev_count; +/* Event callback object */ +struct rte_hcdev_callback { +TAILQ_ENTRY (rte_hcdev_callback) next; +rte_hcdev_callback_t *function; +void *user_data; +enum rte_hcdev_event event; +}; +static void hcdev_free_callbacks (struct rte_hcdev *dev); +int rte_hcdev_init (size_t dev_max) {
@@ -166,6 +176,7 @@ rte_hcdev_allocate(const char *name) dev->info.name = dev->name; dev->info.dev_id = dev_id; dev->info.numa_node =-1 ; +TAILQ_INIT (&dev->callbacks); hcdev_count++; HCDEV_LOG(DEBUG, "new device %s (id %d) of total %d",
@@ -180,6 +191,7 @@ rte_hcdev_complete_new(struct rte_hcdev *dev)return ; dev->state =RTE_HCDEV_STATE_INITIALIZED ; +rte_hcdev_notify (dev,RTE_HCDEV_EVENT_NEW ); }int
@@ -192,6 +204,9 @@ rte_hcdev_release(struct rte_hcdev *dev)HCDEV_LOG (DEBUG ,"free device %s (id %d)" , dev->info.name, dev->info.dev_id); +rte_hcdev_notify (dev,RTE_HCDEV_EVENT_DEL ); + +hcdev_free_callbacks (dev); dev->state =RTE_HCDEV_STATE_UNUSED ; hcdev_count--;
@@ -224,6 +239,128 @@ rte_hcdev_close(int16_t dev_id)return firsterr; } +int +rte_hcdev_callback_register (int16_t dev_id,enum rte_hcdev_event event, +rte_hcdev_callback_t *function,void *user_data) +{ +int16_t next_dev, last_dev; +struct rte_hcdev_callback_list *callbacks; +struct rte_hcdev_callback *callback; + +if (!rte_hcdev_is_valid (dev_id) && dev_id !=RTE_HCDEV_ID_ANY ) { +HCDEV_LOG (ERR ,"register callback of invalid ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } +if (function ==NULL ) { +HCDEV_LOG (ERR ,"cannot register callback without function" ); + rte_errno =EINVAL ; +return -rte_errno; + } + +if (dev_id ==RTE_HCDEV_ID_ANY ) { + next_dev =0 ; + last_dev = hcdev_max -1 ; + }else { + next_dev = last_dev = dev_id; + } +do { + callbacks = &hcdevs[next_dev].callbacks; + +/* check if not already registered */ + TAILQ_FOREACH(callback, callbacks, next) { +if (callback->event == event && + callback->function == function && + callback->user_data == user_data) { +HCDEV_LOG (INFO ,"callback already registered" ); +return 0 ; + } + } + + callback =malloc (sizeof (*callback)); +if (callback ==NULL ) { +HCDEV_LOG (ERR ,"cannot allocate callback" ); +return -ENOMEM ; + } + callback->function = function; + callback->user_data = user_data; + callback->event = event; +TAILQ_INSERT_TAIL (callbacks, callback, next); + + }while (++next_dev <= last_dev); + +return 0 ; +} + +int +rte_hcdev_callback_unregister (int16_t dev_id,enum rte_hcdev_event event, +rte_hcdev_callback_t *function,void *user_data) +{ +int16_t next_dev, last_dev; +struct rte_hcdev_callback_list *callbacks; +struct rte_hcdev_callback *callback, *next_callback; + +if (!rte_hcdev_is_valid (dev_id) && dev_id !=RTE_HCDEV_ID_ANY ) { +HCDEV_LOG (ERR ,"unregister callback of invalid ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } +if (function ==NULL ) { +HCDEV_LOG (ERR ,"cannot unregister callback without function" ); + rte_errno =EINVAL ; +return -rte_errno; + } + +if (dev_id ==RTE_HCDEV_ID_ANY ) { + next_dev =0 ; + last_dev = hcdev_max -1 ; + }else { + next_dev = last_dev = dev_id; + } + +do { + callbacks = &hcdevs[next_dev].callbacks; + TAILQ_FOREACH_SAFE(callback, callbacks, next, next_callback) { +if (callback->event != event || + callback->function != function || + (callback->user_data != user_data && + user_data != (void *)-1 )) +continue ; +TAILQ_REMOVE (callbacks, callback, next); +free (callback); + } + }while (++next_dev <= last_dev); + +return 0 ; +} + +static void +hcdev_free_callbacks (struct rte_hcdev *dev) +{ +struct rte_hcdev_callback_list *callbacks; +struct rte_hcdev_callback *callback, *next_callback; + + callbacks = &dev->callbacks; + TAILQ_FOREACH_SAFE(callback, callbacks, next, next_callback) { +TAILQ_REMOVE (callbacks, callback, next); +free (callback); + } +} + +void +rte_hcdev_notify (struct rte_hcdev *dev,enum rte_hcdev_event event) +{ +int16_t dev_id; +struct rte_hcdev_callback *callback; + + dev_id = dev->info.dev_id; + TAILQ_FOREACH(callback, &dev->callbacks, next) { +if (callback->event != event || callback->function ==NULL ) +continue ; + callback->function (dev_id, event, callback->user_data); + } +} +int rte_hcdev_info_get (int16_t dev_id,struct rte_hcdev_info *info) {
lib/hcdev/hcdev_driver.h +7 -0
@@ -12,6 +12,7 @@#define RTE_HCDEV_DRIVER_H #include <stdint.h> +#include <sys/queue.h> #include <rte_dev.h>
@@ -43,6 +44,8 @@ struct rte_hcdev {struct rte_hcdev_info info;/* Driver functions. */ struct rte_hcdev_ops ops; +/* Event callback list. */ + TAILQ_HEAD(rte_hcdev_callback_list, rte_hcdev_callback) callbacks;/* Current state (used or not) in the running process. */ enum rte_hcdev_state state;/* Updated by this library. */ /* Driver-specific private data for the running process. */
@@ -64,4 +67,8 @@ void rte_hcdev_complete_new(struct rte_hcdev *dev);__rte_internal int rte_hcdev_release(struct rte_hcdev *dev); +/* Call registered callbacks. No multi-process event. */ +__rte_internal +void rte_hcdev_notify(struct rte_hcdev *dev, enum rte_hcdev_event); +#endif /* RTE_HCDEV_DRIVER_H */
lib/hcdev/rte_hcdev.h +71 -0
95 lines not on this page · raw message
lib/hcdev/version.map +3 -0
@@ -2,6 +2,8 @@ EXPERIMENTAL { global: # added in 21.11 + rte_hcdev_callback_register; + rte_hcdev_callback_unregister; rte_hcdev_close; rte_hcdev_count_avail; rte_hcdev_find_next;
@@ -16,5 +18,6 @@ INTERNAL { rte_hcdev_allocate; rte_hcdev_complete_new; rte_hcdev_get_by_name; + rte_hcdev_notify; rte_hcdev_release; };
Signature · 1 line
2.31.1
RFC PATCH v2 3/7 hcdev: add child device representing a device context
In reply to Thomas MonjalonThe computing device may operate in some isolated contexts. Memory and processing are isolated in a silo represented by a child device.
The context is provided as an opaque by the caller of rte_hcdev_add_child().
--- lib/hcdev/hcdev.c | 45 ++++++++++++++++++++++++-- lib/hcdev/hcdev_driver.h | 2 +- lib/hcdev/rte_hcdev.h | 69 +++++++++++++++++++++++++++++++++++++--- lib/hcdev/version.map | 1 + 4 files changed, 110 insertions(+), 7 deletions(-)
lib/hcdev/hcdev.c +43 -2
@@ -79,13 +79,22 @@ rte_hcdev_is_valid(int16_t dev_id)return false; } +static bool +hcdev_match_parent (int16_t dev_id,int16_t parent) +{ +if (parent ==RTE_HCDEV_ID_ANY ) +return true; +return hcdevs[dev_id].info.parent == parent; +} +int16_t -rte_hcdev_find_next (int16_t dev_id) +rte_hcdev_find_next (int16_t dev_id,int16_t parent) {if (dev_id <0 ) dev_id =0 ;while (dev_id < hcdev_max && - hcdevs[dev_id].state ==RTE_HCDEV_STATE_UNUSED ) + (hcdevs[dev_id].state ==RTE_HCDEV_STATE_UNUSED || + !hcdev_match_parent (dev_id, parent))) dev_id++;if (dev_id >= hcdev_max)
@@ -176,6 +185,7 @@ rte_hcdev_allocate(const char *name) dev->info.name = dev->name; dev->info.dev_id = dev_id; dev->info.numa_node =-1 ; + dev->info.parent =RTE_HCDEV_ID_NONE ;TAILQ_INIT (&dev->callbacks); hcdev_count++;
@@ -184,6 +194,28 @@ rte_hcdev_allocate(const char *name)return dev; } +int16_t +rte_hcdev_add_child (const char *name,int16_t parent,uint64_t child_context) +{ +struct rte_hcdev *dev; + +if (!rte_hcdev_is_valid (parent)) { +HCDEV_LOG (ERR ,"add child to invalid parent ID %d" , parent); + rte_errno =ENODEV ; +return -rte_errno; + } + + dev =rte_hcdev_allocate (name); +if (dev ==NULL ) +return -rte_errno; + + dev->info.parent = parent; + dev->info.context = child_context; + +rte_hcdev_complete_new (dev); +return dev->info.dev_id; +} +void rte_hcdev_complete_new (struct rte_hcdev *dev) {
@@ -197,10 +229,19 @@ rte_hcdev_complete_new(struct rte_hcdev *dev)int rte_hcdev_release (struct rte_hcdev *dev) { +int16_t dev_id, child; +if (dev ==NULL ) { rte_errno =ENODEV ;return -rte_errno; } + dev_id = dev->info.dev_id; + RTE_HCDEV_FOREACH_CHILD(child, dev_id) { +HCDEV_LOG (ERR ,"cannot release device %d with child %d" , + dev_id, child); + rte_errno =EBUSY ; +return -rte_errno; + }HCDEV_LOG (DEBUG ,"free device %s (id %d)" , dev->info.name, dev->info.dev_id);
lib/hcdev/hcdev_driver.h +1 -1
@@ -31,7 +31,7 @@ typedef int (rte_hcdev_info_get_t)(struct rte_hcdev *dev, struct rte_hcdev_infostruct rte_hcdev_ops {/* Get device info. If NULL, info is just copied. */ rte_hcdev_info_get_t *dev_info_get; -/* Close device. */ +/* Close device or child context. */ rte_hcdev_close_t *dev_close; };
lib/hcdev/rte_hcdev.h +65 -4
107 lines not on this page · raw message
lib/hcdev/version.map +1 -0
@@ -2,6 +2,7 @@ EXPERIMENTAL { global: # added in 21.11 + rte_hcdev_add_child; rte_hcdev_callback_register; rte_hcdev_callback_unregister; rte_hcdev_close;
Signature · 1 line
2.31.1
RFC PATCH v2 4/7 hcdev: support multi-process
In reply to Thomas MonjalonThe device data shared between processes are moved in a struct allocated in a shared memory (a new memzone for all hcdevs). The main struct rte_hcdev references the shared memory via the pointer mpshared.
The API function rte_hcdev_attach() is added to attach a device from the secondary process.
The function rte_hcdev_allocate() can be used only by primary process.
--- lib/hcdev/hcdev.c | 114 ++++++++++++++++++++++++++++++++------- lib/hcdev/hcdev_driver.h | 23 ++++++-- lib/hcdev/rte_hcdev.h | 3 +- lib/hcdev/version.map | 1 + 4 files changed, 115 insertions(+), 26 deletions(-)
lib/hcdev/hcdev.c +95 -19
@@ -5,6 +5,7 @@#include <rte_eal.h> #include <rte_tailq.h> #include <rte_string_fns.h> +#include <rte_memzone.h> #include <rte_errno.h> #include <rte_log.h>
@@ -28,6 +29,12 @@ static int16_t hcdev_max;/* Number of currently valid devices */ static int16_t hcdev_count; +/* Shared memory between processes. */ +static const char *HCDEV_MEMZONE ="rte_hcdev_shared" ; +static struct { + __extension__struct rte_hcdev_mpshared hcdevs[0 ]; +} *hcdev_shared_mem; +/* Event callback object */ struct rte_hcdev_callback {TAILQ_ENTRY (rte_hcdev_callback) next;
@@ -40,6 +47,8 @@ static void hcdev_free_callbacks(struct rte_hcdev *dev);int rte_hcdev_init (size_t dev_max) { +const struct rte_memzone *memzone; +if (dev_max ==0 || dev_max >INT16_MAX ) {HCDEV_LOG (ERR ,"invalid array size" ); rte_errno =EINVAL ;
@@ -60,6 +69,23 @@ rte_hcdev_init(size_t dev_max)return -rte_errno; } +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { + memzone =rte_memzone_reserve (HCDEV_MEMZONE , +sizeof (*hcdev_shared_mem) + +sizeof (*hcdev_shared_mem->hcdevs) * dev_max, +SOCKET_ID_ANY ,0 ); + }else { + memzone =rte_memzone_lookup (HCDEV_MEMZONE ); + } +if (memzone ==NULL ) { +HCDEV_LOG (ERR ,"cannot initialize shared memory" ); +free (hcdevs); + hcdevs =NULL ; + rte_errno =ENOMEM ; +return -rte_errno; + } + hcdev_shared_mem = memzone->addr; + hcdev_max = dev_max;return 0 ; }
@@ -74,7 +100,7 @@ bool rte_hcdev_is_valid(int16_t dev_id) {if (dev_id >=0 && dev_id < hcdev_max && - hcdevs[dev_id].state ==RTE_HCDEV_STATE_INITIALIZED ) + hcdevs[dev_id].process_state ==RTE_HCDEV_STATE_INITIALIZED )return true;return false; }
@@ -84,7 +110,7 @@ hcdev_match_parent(int16_t dev_id, int16_t parent) {if (parent ==RTE_HCDEV_ID_ANY )return true; -return hcdevs[dev_id].info.parent == parent; +return hcdevs[dev_id].mpshared->info.parent == parent; }int16_t
@@ -93,7 +119,7 @@ rte_hcdev_find_next(int16_t dev_id, int16_t parent)if (dev_id <0 ) dev_id =0 ;while (dev_id < hcdev_max && - (hcdevs[dev_id].state ==RTE_HCDEV_STATE_UNUSED || + (hcdevs[dev_id].process_state ==RTE_HCDEV_STATE_UNUSED || !hcdev_match_parent (dev_id, parent))) dev_id++;
@@ -108,7 +134,7 @@ hcdev_find_free_id(void)int16_t dev_id;for (dev_id =0 ; dev_id < hcdev_max; dev_id++) { -if (hcdevs[dev_id].state ==RTE_HCDEV_STATE_UNUSED ) +if (hcdevs[dev_id].process_state ==RTE_HCDEV_STATE_UNUSED )return dev_id; }return RTE_HCDEV_ID_NONE ;
@@ -135,7 +161,7 @@ rte_hcdev_get_by_name(const char *name)RTE_HCDEV_FOREACH (dev_id) { dev = &hcdevs[dev_id]; -if (strncmp (name, dev->name,RTE_DEV_NAME_MAX_LEN ) ==0 ) +if (strncmp (name, dev->mpshared->name,RTE_DEV_NAME_MAX_LEN ) ==0 )return dev; }return NULL ;
@@ -177,16 +203,20 @@ rte_hcdev_allocate(const char *name) dev = &hcdevs[dev_id];memset (dev,0 ,sizeof (*dev)); -if (rte_strscpy (dev->name, name,RTE_DEV_NAME_MAX_LEN ) <0 ) { + dev->mpshared = &hcdev_shared_mem->hcdevs[dev_id]; +memset (dev->mpshared,0 ,sizeof (*dev->mpshared)); + +if (rte_strscpy (dev->mpshared->name, name,RTE_DEV_NAME_MAX_LEN ) <0 ) {HCDEV_LOG (ERR ,"device name too long: %s" , name); rte_errno =ENAMETOOLONG ;return NULL ; } - dev->info.name = dev->name; - dev->info.dev_id = dev_id; - dev->info.numa_node =-1 ; - dev->info.parent =RTE_HCDEV_ID_NONE ; + dev->mpshared->info.name = dev->mpshared->name; + dev->mpshared->info.dev_id = dev_id; + dev->mpshared->info.numa_node =-1 ; + dev->mpshared->info.parent =RTE_HCDEV_ID_NONE ;TAILQ_INIT (&dev->callbacks); +__atomic_fetch_add (&dev->mpshared->process_refcnt,1 , __ATOMIC_RELAXED); hcdev_count++; HCDEV_LOG(DEBUG, "new device %s (id %d) of total %d",
@@ -194,6 +224,51 @@ rte_hcdev_allocate(const char *name)return dev; } +struct rte_hcdev * +rte_hcdev_attach (const char *name) +{ +int16_t dev_id; +struct rte_hcdev *dev; +struct rte_hcdev_mpshared *shared_dev; + +if (rte_eal_process_type () !=RTE_PROC_SECONDARY ) { +HCDEV_LOG (ERR ,"only secondary process can attach device" ); + rte_errno =EPERM ; +return NULL ; + } +if (name ==NULL ) { +HCDEV_LOG (ERR ,"attach device without a name" ); + rte_errno =EINVAL ; +return NULL ; + } + +/* implicit initialization of library before adding first device */ +if (hcdevs ==NULL &&rte_hcdev_init (RTE_HCDEV_DEFAULT_MAX ) <0 ) +return NULL ; + +for (dev_id =0 ; dev_id < hcdev_max; dev_id++) { + shared_dev = &hcdev_shared_mem->hcdevs[dev_id]; +if (strncmp (name, shared_dev->name,RTE_DEV_NAME_MAX_LEN ) ==0 ) +break ; + } +if (dev_id >= hcdev_max) { +HCDEV_LOG (ERR ,"device with name %s not found" , name); + rte_errno =ENOENT ; +return NULL ; + } + dev = &hcdevs[dev_id]; +memset (dev,0 ,sizeof (*dev)); + +TAILQ_INIT (&dev->callbacks); + dev->mpshared = shared_dev; +__atomic_fetch_add (&dev->mpshared->process_refcnt,1 , __ATOMIC_RELAXED); + + hcdev_count++; +HCDEV_LOG (DEBUG ,"attached device %s (id %d) of total %d" , + name, dev_id, hcdev_count); +return dev; +} +int16_t rte_hcdev_add_child (const char *name,int16_t parent,uint64_t child_context) {
@@ -209,11 +284,11 @@ rte_hcdev_add_child(const char *name, int16_t parent, uint64_t child_context)if (dev ==NULL )return -rte_errno; - dev->info.parent = parent; - dev->info.context = child_context; + dev->mpshared->info.parent = parent; + dev->mpshared->info.context = child_context;rte_hcdev_complete_new (dev); -return dev->info.dev_id; +return dev->mpshared->info.dev_id; }void
@@ -222,7 +297,7 @@ rte_hcdev_complete_new(struct rte_hcdev *dev)if (dev ==NULL )return ; - dev->state =RTE_HCDEV_STATE_INITIALIZED ; + dev->process_state =RTE_HCDEV_STATE_INITIALIZED ;rte_hcdev_notify (dev,RTE_HCDEV_EVENT_NEW ); }
@@ -235,7 +310,7 @@ rte_hcdev_release(struct rte_hcdev *dev) rte_errno =ENODEV ;return -rte_errno; } - dev_id = dev->info.dev_id; + dev_id = dev->mpshared->info.dev_id; RTE_HCDEV_FOREACH_CHILD(child, dev_id) {HCDEV_LOG (ERR ,"cannot release device %d with child %d" , dev_id, child);
@@ -244,11 +319,12 @@ rte_hcdev_release(struct rte_hcdev *dev) }HCDEV_LOG (DEBUG ,"free device %s (id %d)" , - dev->info.name, dev->info.dev_id); + dev->mpshared->info.name, dev->mpshared->info.dev_id);rte_hcdev_notify (dev,RTE_HCDEV_EVENT_DEL );hcdev_free_callbacks (dev); - dev->state =RTE_HCDEV_STATE_UNUSED ; + dev->process_state =RTE_HCDEV_STATE_UNUSED ; +__atomic_fetch_sub (&dev->mpshared->process_refcnt,1 , __ATOMIC_RELAXED); hcdev_count--;return 0 ;
@@ -394,7 +470,7 @@ rte_hcdev_notify(struct rte_hcdev *dev, enum rte_hcdev_event event)int16_t dev_id;struct rte_hcdev_callback *callback; - dev_id = dev->info.dev_id; + dev_id = dev->mpshared->info.dev_id; TAILQ_FOREACH(callback, &dev->callbacks, next) {if (callback->event != event || callback->function ==NULL )continue ;
@@ -420,7 +496,7 @@ rte_hcdev_info_get(int16_t dev_id, struct rte_hcdev_info *info) }if (dev->ops.dev_info_get ==NULL ) { - *info = dev->info; + *info = dev->mpshared->info;return 0 ; }return HCDEV_DRV_RET (dev->ops.dev_info_get (dev, info));
lib/hcdev/hcdev_driver.h +18 -5
@@ -35,19 +35,28 @@ struct rte_hcdev_ops {rte_hcdev_close_t *dev_close; }; -struct rte_hcdev { -/* Backing device. */ -struct rte_device *device; +struct rte_hcdev_mpshared {/* Unique identifier name. */ char name[RTE_DEV_NAME_MAX_LEN ];/* Updated by this library. */ +/* Driver-specific private data shared in multi-process. */ +void *dev_private;/* Device info structure. */ struct rte_hcdev_info info; +/* Counter of processes using the device. */ +uint16_t process_refcnt;/* Updated by this library. */ +}; + +struct rte_hcdev { +/* Backing device. */ +struct rte_device *device; +/* Data shared between processes. */ +struct rte_hcdev_mpshared *mpshared;/* Driver functions. */ struct rte_hcdev_ops ops;/* Event callback list. */ TAILQ_HEAD(rte_hcdev_callback_list, rte_hcdev_callback) callbacks;/* Current state (used or not) in the running process. */ -enum rte_hcdev_state state;/* Updated by this library. */ +enum rte_hcdev_state process_state;/* Updated by this library. */ /* Driver-specific private data for the running process. */ void *process_private; } __rte_cache_aligned;
@@ -55,10 +64,14 @@ struct rte_hcdev {__rte_internal struct rte_hcdev *rte_hcdev_get_by_name(const char *name); -/* First step of initialization */ +/* First step of initialization in primary process. */ __rte_internal struct rte_hcdev *rte_hcdev_allocate(const char *name); +/* First step of initialization in secondary process. */ +__rte_internal +struct rte_hcdev *rte_hcdev_attach(const char *name); +/* Last step of initialization. */ __rte_internal void rte_hcdev_complete_new(struct rte_hcdev *dev);
lib/hcdev/rte_hcdev.h +1 -2
@@ -15,9 +15,8 @@* @file * Generic library to interact with heterogeneous computing device. * -* The API is not thread-safe. -* Device management must be done by a single thread. * TODO device rwlock for callback list +* TODO mp shared rwlock for device array * * @warning * @b EXPERIMENTAL: this API may change without prior notice.
lib/hcdev/version.map +1 -0
@@ -17,6 +17,7 @@ INTERNAL { global: rte_hcdev_allocate; + rte_hcdev_attach; rte_hcdev_complete_new; rte_hcdev_get_by_name; rte_hcdev_notify;
Signature · 1 line
2.31.1
RFC PATCH v2 5/7 hcdev: add memory API
In reply to Thomas MonjalonFrom: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. Such workload distribution can be achieved by sharing some memory.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the device, or in the main (CPU) memory while making it visible for the device. This memory may be used to save packets or for synchronization data.
The next step should focus on GPU processing task control.
--- doc/guides/hcdevs/features/default.ini | 3 + doc/guides/rel_notes/release_21_08.rst | 1 + lib/hcdev/hcdev.c | 88 ++++++++++++++++++++++++++ lib/hcdev/hcdev_driver.h | 9 +++ lib/hcdev/rte_hcdev.h | 53 ++++++++++++++++ lib/hcdev/version.map | 2 + 6 files changed, 156 insertions(+)
doc/guides/hcdevs/features/default.ini +3 -0
@@ -8,3 +8,6 @@ ; [Features] Get device info = +Share CPU memory with device = +Allocate device memory = +Free memory =
doc/guides/rel_notes/release_21_08.rst +1 -0
@@ -58,6 +58,7 @@ New Features * **Introduced Heterogeneous Computing Device library with first features:** * Device information + * Memory management * **Added auxiliary bus support.**
lib/hcdev/hcdev.c +88 -0
97 lines not on this page · raw message
lib/hcdev/hcdev_driver.h +9 -0
@@ -27,12 +27,21 @@ enum rte_hcdev_state {struct rte_hcdev ;typedef int (rte_hcdev_close_t )(struct rte_hcdev *dev);typedef int (rte_hcdev_info_get_t )(struct rte_hcdev *dev,struct rte_hcdev_info *info); +typedef int (rte_hcdev_mem_alloc_t )(struct rte_hcdev *dev,size_t size,void **ptr); +typedef int (rte_hcdev_mem_register_t )(struct rte_hcdev *dev,size_t size,void *ptr); +typedef int (rte_hcdev_free_t )(struct rte_hcdev *dev,void *ptr);struct rte_hcdev_ops {/* Get device info. If NULL, info is just copied. */ rte_hcdev_info_get_t *dev_info_get;/* Close device or child context. */ rte_hcdev_close_t *dev_close; +/* Allocate memory in device. */ +rte_hcdev_mem_alloc_t *mem_alloc; +/* Register CPU memory in device. */ +rte_hcdev_mem_register_t *mem_register; +/* Free memory allocated or registered in device. */ +rte_hcdev_free_t *mem_free; };struct rte_hcdev_mpshared {
lib/hcdev/rte_hcdev.h +53 -0
65 lines not on this page · raw message
lib/hcdev/version.map +2 -0
@@ -8,9 +8,11 @@ EXPERIMENTAL { rte_hcdev_close; rte_hcdev_count_avail; rte_hcdev_find_next; + rte_hcdev_free; rte_hcdev_info_get; rte_hcdev_init; rte_hcdev_is_valid; + rte_hcdev_malloc; }; INTERNAL {
Signature · 1 line
2.31.1
RFC PATCH v2 6/7 hcdev: add communication flag
In reply to Thomas MonjalonFrom: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. When mixing network activity with task processing there may be the need to put in communication the CPU with the device in order to synchronize operations.
The purpose of this flag is to allow the CPU and the device to exchange ACKs. A possible use-case is described below.
CPU:
- Trigger some task on the device
- Prepare some data
- Signal to the device the data is ready updating the communication flag
Device:
- Do some pre-processing
- Wait for more data from the CPU polling on the communication flag
- Consume the data prepared by the CPU
--- lib/hcdev/hcdev.c | 71 ++++++++++++++++++++++++++++ lib/hcdev/rte_hcdev.h | 107 ++++++++++++++++++++++++++++++++++++++++++ lib/hcdev/version.map | 4 ++ 3 files changed, 182 insertions(+)
lib/hcdev/hcdev.c +71 -0
@@ -589,3 +589,74 @@ rte_hcdev_free(int16_t dev_id, void *ptr)/* TODO unregister callback */ /* TODO rte_free CPU memory */ } + +int +rte_hcdev_comm_create_flag (uint16_t dev_id,struct rte_hcdev_comm_flag *hcflag, +enum rte_hcdev_comm_flag_type mtype) +{ +size_t flag_size; + +if (hcflag ==NULL ) { + rte_errno =EINVAL ; +return -rte_errno; + } +if (mtype !=RTE_HCDEV_COMM_FLAG_CPU ) { + rte_errno =EINVAL ; +return -rte_errno; + } + + flag_size =sizeof (uint32_t ); + + hcflag->ptr =rte_hcdev_malloc (dev_id, flag_size, +RTE_HCDEV_MALLOC_REGISTER_FROM_CPU ); +if (hcflag->ptr ==NULL ) +return -rte_errno; + + hcflag->mtype = mtype; +return 0 ; +} + +int +rte_hcdev_comm_destroy_flag (uint16_t dev_id,struct rte_hcdev_comm_flag *hcflag) +{ +if (hcflag ==NULL ) { + rte_errno =EINVAL ; +return -rte_errno; + } + +return rte_hcdev_free (dev_id, hcflag->ptr); +} + +int +rte_hcdev_comm_set_flag (struct rte_hcdev_comm_flag *hcflag,uint32_t val) +{ +if (hcflag ==NULL ) { + rte_errno =EINVAL ; +return -rte_errno; + } +if (hcflag->mtype !=RTE_HCDEV_COMM_FLAG_CPU ) { + rte_errno =EINVAL ; +return -rte_errno; + } + +RTE_HCDEV_VOLATILE (*hcflag->ptr) = val; + +return 0 ; +} + +int +rte_hcdev_comm_get_flag_value (struct rte_hcdev_comm_flag *hcflag,uint32_t *val) +{ +if (hcflag ==NULL ) { + rte_errno =EINVAL ; +return -rte_errno; + } +if (hcflag->mtype !=RTE_HCDEV_COMM_FLAG_CPU ) { + rte_errno =EINVAL ; +return -rte_errno; + } + + *val =RTE_HCDEV_VOLATILE (*hcflag->ptr); + +return 0 ; +}
lib/hcdev/rte_hcdev.h +107 -0
125 lines not on this page · raw message
lib/hcdev/version.map +4 -0
@@ -6,6 +6,10 @@ EXPERIMENTAL { rte_hcdev_callback_register; rte_hcdev_callback_unregister; rte_hcdev_close; + rte_hcdev_comm_create_flag; + rte_hcdev_comm_destroy_flag; + rte_hcdev_comm_get_flag_value; + rte_hcdev_comm_set_flag; rte_hcdev_count_avail; rte_hcdev_find_next; rte_hcdev_free;
Signature · 1 line
2.31.1
RFC PATCH v2 7/7 hcdev: add communication list
In reply to Thomas MonjalonFrom: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. When mixing network activity with task processing there may be the need to put in communication the CPU with the device in order to synchronize operations.
An example could be a receive-and-process application
where CPU is responsible for receiving packets in multiple mbufs and the device is responsible for processing the content of those packets.
The purpose of this list is to provide a buffer in CPU memory visible from the device that can be treated as a circular buffer
to let the CPU provide fondamental info of received packets to the device.
A possible use-case is described below.
CPU:
- Trigger some task on the device
- in a loop:
- receive a number of packets
- provide packets info to the device
Device:
- Do some pre-processing
- Wait to receive a new set of packet to be processed
Layout of a communication list would be:
-------
| 0 | => pkt_list
| status |
| #pkts |
-------
| 1 | => pkt_list
| status |
| #pkts |
-------
| 2 | => pkt_list
| status |
| #pkts |
-------
| .... | => pkt_list
---------- lib/hcdev/hcdev.c | 127 ++++++++++++++++++++++++++++++++++++++++ lib/hcdev/meson.build | 2 + lib/hcdev/rte_hcdev.h | 132 ++++++++++++++++++++++++++++++++++++++++++ lib/hcdev/version.map | 4 ++ 4 files changed, 265 insertions(+)
lib/hcdev/hcdev.c +127 -0
@@ -660,3 +660,130 @@ rte_hcdev_comm_get_flag_value(struct rte_hcdev_comm_flag *hcflag, uint32_t *val)return 0 ; } + +struct rte_hcdev_comm_list * +rte_hcdev_comm_create_list (uint16_t dev_id, +uint32_t num_comm_items) +{ +struct rte_hcdev_comm_list *comm_list; +uint32_t idx_l; + +if (num_comm_items ==0 ) { + rte_errno =EINVAL ; +return NULL ; + } + + comm_list =rte_hcdev_malloc (dev_id, +sizeof (struct rte_hcdev_comm_list ) * num_comm_items, +RTE_HCDEV_MALLOC_REGISTER_FROM_CPU ); +if (comm_list ==NULL ) { + rte_errno =ENOMEM ; +return NULL ; + } + +for (idx_l =0 ; idx_l < num_comm_items; idx_l++) { + comm_list[idx_l].pkt_list = +rte_hcdev_malloc (dev_id, +sizeof (struct rte_hcdev_comm_pkt ) * +RTE_HCDEV_COMM_LIST_PKTS_MAX , +RTE_HCDEV_MALLOC_REGISTER_FROM_CPU ); +if (comm_list[idx_l].pkt_list ==NULL ) { + rte_errno =ENOMEM ; +return NULL ; + } + +RTE_HCDEV_VOLATILE (comm_list[idx_l].status) = +RTE_HCDEV_COMM_LIST_FREE ; + comm_list[idx_l].num_pkts =0 ; + } + +return comm_list; +} + +int +rte_hcdev_comm_destroy_list (uint16_t dev_id, +struct rte_hcdev_comm_list *comm_list, +uint32_t num_comm_items) +{ +uint32_t idx_l; + +if (comm_list ==NULL ) { + rte_errno =EINVAL ; +return -rte_errno; + } + +for (idx_l =0 ; idx_l < num_comm_items; idx_l++) +rte_hcdev_free (dev_id, comm_list[idx_l].pkt_list); +rte_hcdev_free (dev_id, comm_list); + +return 0 ; +} + +int +rte_hcdev_comm_populate_list_pkts (struct rte_hcdev_comm_list *comm_list_item, +struct rte_mbuf **mbufs,uint32_t num_mbufs) +{ +uint32_t idx; + +if (comm_list_item ==NULL || comm_list_item->pkt_list ==NULL || + mbufs ==NULL || num_mbufs >RTE_HCDEV_COMM_LIST_PKTS_MAX ) { + rte_errno =EINVAL ; +return -rte_errno; + } + +for (idx =0 ; idx < num_mbufs; idx++) { +/* support only unchained mbufs */ +if (unlikely ((mbufs[idx]->nb_segs >1 ) || + (mbufs[idx]->next !=NULL ) || + (mbufs[idx]->data_len != mbufs[idx]->pkt_len))) { + rte_errno =ENOTSUP ; +return -rte_errno; + } + comm_list_item->pkt_list[idx].addr = +rte_pktmbuf_mtod_offset (mbufs[idx], uintptr_t,0 ); + comm_list_item->pkt_list[idx].size = mbufs[idx]->pkt_len; + comm_list_item->pkt_list[idx].opaque = mbufs[idx]; + } + +RTE_HCDEV_VOLATILE (comm_list_item->num_pkts) = num_mbufs; +rte_mb (); +RTE_HCDEV_VOLATILE (comm_list_item->status) =RTE_HCDEV_COMM_LIST_READY ; + +return 0 ; +} + +int +rte_hcdev_comm_cleanup_list (struct rte_hcdev_comm_list *comm_list_item) +{ +struct rte_mbuf *mbufs[RTE_HCDEV_COMM_LIST_PKTS_MAX ]; +uint32_t idx =0 ; + +if (comm_list_item ==NULL ) { + rte_errno =EINVAL ; +return -rte_errno; + } + +if (RTE_HCDEV_VOLATILE (comm_list_item->status) == +RTE_HCDEV_COMM_LIST_READY ) { +HCDEV_LOG (ERR ,"packet list is still in progress" ); + rte_errno =EINVAL ; +return -rte_errno; + } + +for (idx =0 ; idx <RTE_HCDEV_COMM_LIST_PKTS_MAX ; idx++) { +if (comm_list_item->pkt_list[idx].addr ==0 ) +break ; + + comm_list_item->pkt_list[idx].addr =0 ; + comm_list_item->pkt_list[idx].size =0 ; + mbufs[idx] = (struct rte_mbuf *) comm_list_item->pkt_list[idx].opaque; + } + +rte_pktmbuf_free_bulk (mbufs, idx); + +RTE_HCDEV_VOLATILE (comm_list_item->status) =RTE_HCDEV_COMM_LIST_FREE ; +RTE_HCDEV_VOLATILE (comm_list_item->num_pkts) =0 ; +rte_mb (); + +return 0 ; +}
lib/hcdev/meson.build +2 -0
@@ -8,3 +8,5 @@ headers = files( sources = files( 'hcdev.c', ) + +deps += ['mbuf']
lib/hcdev/rte_hcdev.h +132 -0
160 lines not on this page · raw message
lib/hcdev/version.map +4 -0
@@ -6,9 +6,13 @@ EXPERIMENTAL { rte_hcdev_callback_register; rte_hcdev_callback_unregister; rte_hcdev_close; + rte_hcdev_comm_cleanup_list; rte_hcdev_comm_create_flag; + rte_hcdev_comm_create_list; rte_hcdev_comm_destroy_flag; + rte_hcdev_comm_destroy_list; rte_hcdev_comm_get_flag_value; + rte_hcdev_comm_populate_list_pkts; rte_hcdev_comm_set_flag; rte_hcdev_count_avail; rte_hcdev_find_next;
Signature · 1 line
2.31.1
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
In reply to Thomas MonjalonQuoted 22 lines · Thomas Monjalon
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices like GPUs.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible from the device
- Communication functions to enhance the dialog between the CPU and the deviceThe infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing the hcdev API.
Quoted 8 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.
Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload
on a different class of CPU.
In terms of RFC patch features
1) memory API - Use cases are aligned
2) communication flag and communication list
Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailing list.
Now the bigger question is why need to Tx and then Rx something to compute the device
Isn't ot offload something? If so, why not add the those offload in respective subsystem
to improve the subsystem(ethdev, cryptiodev etc) features set to adapt new features or
introduce new subsystem (like ML, Inline Baseband processing) so that it will be an opportunity to
implement the same in HW or compute device. For example, if we take this path, ML offloading will
be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev)
to deal with specific computing device offload "COMMANDS" (The commands will be specific to offload device, the same code wont run on other compute device)
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of
path. If we decide another path as a community it is _fine_ too(as a _project manager_ point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).
Quoted 160 lines
Below is a pseudo-code to give an example about how to use functions in this library in case of a CUDA application.
Elena Agostini (4):
hcdev: introduce heterogeneous computing device library
hcdev: add memory API
hcdev: add communication flag
hcdev: add communication listThomas Monjalon (3):
hcdev: add event notification
hcdev: add child device representing a device context
hcdev: support multi-process.gitignore | 1 + MAINTAINERS | 6 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 + doc/guides/hcdevs/features/default.ini | 13 + doc/guides/hcdevs/index.rst | 11 + doc/guides/hcdevs/overview.rst | 11 + doc/guides/index.rst | 1 + doc/guides/prog_guide/hcdev.rst | 5 + doc/guides/prog_guide/index.rst | 1 + doc/guides/rel_notes/release_21_08.rst | 5 + drivers/hc/meson.build | 4 + drivers/meson.build | 1 + lib/hcdev/hcdev.c | 789 +++++++++++++++++++++++++ lib/hcdev/hcdev_driver.h | 96 +++ lib/hcdev/meson.build | 12 + lib/hcdev/rte_hcdev.h | 592 +++++++++++++++++++ lib/hcdev/version.map | 35 ++ lib/meson.build | 1 + 20 files changed, 1594 insertions(+) create mode 100644 doc/guides/hcdevs/features/default.ini create mode 100644 doc/guides/hcdevs/index.rst create mode 100644 doc/guides/hcdevs/overview.rst create mode 100644 doc/guides/prog_guide/hcdev.rst create mode 100644 drivers/hc/meson.build create mode 100644 lib/hcdev/hcdev.c create mode 100644 lib/hcdev/hcdev_driver.h create mode 100644 lib/hcdev/meson.build create mode 100644 lib/hcdev/rte_hcdev.h create mode 100644 lib/hcdev/version.map//////////////////////////////////////////////////////////////////////// ///// HCDEV library + CUDA functions //////////////////////////////////////////////////////////////////////// #define
GPU_PAGE_SHIFT16
#defineGPU_PAGE_SIZE(1UL <<GPU_PAGE_SHIFT)int main() { struct rte_hcdev_flag quit_flag; struct rte_hcdev_comm_list *comm_list; int nb_rx = 0; int comm_list_entry = 0; struct rte_mbuf * rx_mbufs[max_rx_mbufs]; cudaStream_t cstream; struct rte_mempool *mpool_payload, *mpool_header; struct rte_pktmbuf_extmem ext_mem; int16_t dev_id; /* Initialize CUDA objects (cstream, context, etc..). */ /* Use hcdev library to register a new CUDA context if any */ /* Let's assume the application wants to use the default context of the GPU device 0 */ dev_id = 0; /* Create an external memory mempool using memory allocated on the GPU. */ ext_mem.elt_size = mbufs_headroom_size; ext_mem.buf_len = RTE_ALIGN_CEIL(mbufs_num * ext_mem.elt_size, GPU_PAGE_SIZE); ext_mem.buf_iova = RTE_BAD_IOVA; ext_mem.buf_ptr = rte_hcdev_malloc(dev_id, ext_mem.buf_len, 0); rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len, NULL, ext_mem.buf_iova, GPU_PAGE_SIZE); rte_dev_dma_map(rte_eth_devices[l2fwd_port_id].device, ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len); mpool_payload = rte_pktmbuf_pool_create_extbuf("gpu_mempool", mbufs_num, 0, 0, ext_mem.elt_size, rte_socket_id(), &ext_mem, 1); /* * Create CPU - device communication flag. With this flag, the CPU can tell to the CUDA kernel * to exit from the main loop. */ rte_hcdev_comm_create_flag(dev_id, &quit_flag, RTE_HCDEV_COMM_FLAG_CPU); rte_hcdev_comm_set_flag(&quit_flag, 0); /* * Create CPU - device communication list. Each entry of this list will be populated by the CPU * with a new set of received mbufs that the CUDA kernel has to process. */ comm_list = rte_hcdev_comm_create_list(dev_id, num_entries); /* A very simple CUDA kernel with just 1 CUDA block and RTE_HCDEV_COMM_LIST_PKTS_MAX CUDA threads. */ cuda_kernel_packet_processing<<<1, RTE_HCDEV_COMM_LIST_PKTS_MAX, 0, cstream>>>(quit_flag->ptr, comm_list, num_entries, ...); /* * For simplicity, the CPU here receives only 2 bursts of mbufs. * In a real application, network activity and device processing should overlap. */ nb_rx = rte_eth_rx_burst(port_id, queue_id, &(rx_mbufs[0]), max_rx_mbufs); rte_hcdev_comm_populate_list_pkts(comm_list[0], rx_mbufs, nb_rx); nb_rx = rte_eth_rx_burst(port_id, queue_id, &(rx_mbufs[0]), max_rx_mbufs); rte_hcdev_comm_populate_list_pkts(comm_list[1], rx_mbufs, nb_rx); /* * CPU waits for the completion of the packets' processing on the CUDA kernel * and then it does a cleanup of the received mbufs. */ while (rte_hcdev_comm_cleanup_list(comm_list[0])); while (rte_hcdev_comm_cleanup_list(comm_list[1])); /* CPU notifies the CUDA kernel that it has to terminate */ rte_hcdev_comm_set_flag(&quit_flag, 1); /* hcdev objects cleanup/destruction */ /* CUDA cleanup */ /* DPDK cleanup */ return 0; }//////////////////////////////////////////////////////////////////////// ///// CUDA kernel ////////////////////////////////////////////////////////////////////////
void cuda_kernel(uint32_t * quit_flag_ptr, struct rte_hcdev_comm_list *comm_list, int comm_list_entries) { int comm_list_index = 0; struct rte_hcdev_comm_pkt *pkt_list = NULL; /* Do some pre-processing operations. */ /* GPU kernel keeps checking this flag to know if it has to quit or wait for more packets. */ while (*quit_flag_ptr == 0) { if (comm_list[comm_list_index]->status != RTE_HCDEV_COMM_LIST_READY) continue; if (threadIdx.x < comm_list[comm_list_index]->num_pkts) { /* Each CUDA thread processes a different packet. */ packet_processing(comm_list[comm_list_index]->addr, comm_list[comm_list_index]->size, ..); } __threadfence(); __syncthreads(); /* Wait for new packets on the next communication list entry. */ comm_list_index = (comm_list_index+1) % comm_list_entries; } /* Do some post-processing operations. */ }--
2.31.1
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
31/07/2021 09:06, Jerin Jacob:
Quoted 43 lines · Thomas Monjalon
Quoted 22 lines
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices like GPUs.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible from the device
- Communication functions to enhance the dialog between the CPU and the deviceThe infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing the hcdev API.Quoted 8 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload
on a different class of CPU.
Quoted 8 lines
In terms of RFC patch features
1) memory API - Use cases are aligned
2) communication flag and communication list
Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailing list.
Interesting.
Quoted 7 lines
Now the bigger question is why need to Tx and then Rx something to compute the device
Isn't ot offload something? If so, why not add the those offload in respective subsystem
to improve thesubsystem(ethdev, cryptiodev etc)features set to adapt new features or
introduce new subsystem (like ML, Inline Baseband processing) so that
Quoted 8 lines
it will be an opportunity to
implement the same in HW or compute device. For example, if we take this path, ML offloading will
be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev)
to deal with specific computing device offload "COMMANDS" (The commands will be specific to offload device, the same code wont run on other compute device)
Having specific features API is convenient for compatibility between devices, yes, for the set of defined features. Our approach is to start with a flexible API that the application can use to implement any processing because with GPU programming, there is no restriction on what can be achieved.
This approach does not contradict what you propose,
it does not prevent extending existing classes.
Quoted 6 lines
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of
path. If we decide another path as a community it is _fine_ too(as a_projectmanager_point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).
Adding a new class API is also improving DPDK.
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 47 lines · Thomas Monjalon
31/07/2021 09:06, Jerin Jacob:
Quoted 45 lines · Thomas Monjalon
Quoted 22 lines
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices like GPUs.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible from the device
- Communication functions to enhance the dialog between the CPU and the deviceThe infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing the hcdev API.Quoted 8 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload
on a different class of CPU.In terms of RFC patch features
Quoted 8 lines
Quoted 6 lines
1) memory API - Use cases are aligned
2) communication flag and communication list
Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailing list.Interesting.
It is hard to generalize the communication mechanism.
Is other GPU vendors have a similar communication mechanism? AMD, Intel ??
Quoted 17 lines
Quoted 16 lines
Now the bigger question is why need to Tx and then Rx something to compute the device
Isn't ot offload something? If so, why not add the those offload in respective subsystem
to improve thesubsystem(ethdev, cryptiodev etc)features set to adapt new features or
introduce new subsystem (like ML, Inline Baseband processing) so that it will be an opportunity to
implement the same in HW or compute device. For example, if we take this path, ML offloading will
be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev)
to deal with specific computing device offload "COMMANDS" (The commands will be specific to offload device, the same code wont run on other compute device)
Quoted 7 lines
Having specific features API is convenient for compatibility between devices, yes, for the set of defined features. Our approach is to start with a flexible API that the application can use to implement any processing because with GPU programming, there is no restriction on what can be achieved.
This approach does not contradict what you propose,
it does not prevent extending existing classes.
It does prevent extending the existing classes as no one is going to extent it there is the path of not doing do.
If an application can run only on a specific device, it is similar to a raw device,
where the device definition is not defined. (i.e JOB metadata is not defined and it is specific to the device).
Quoted 9 lines
Quoted 7 lines
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of
path. If we decide another path as a community it is _fine_ too(as a_projectmanager_point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).Adding a new class API is also improving DPDK.
But the class is similar as raw dev class. The reason I say, Job submission and response is can be abstracted as queue/dequeue APIs. Taks/Job metadata is specific to compute devices (and it can not be generalized).
If we generalize it makes sense to have a new class that does "specific function".
Quoted 2 lines
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
31/07/2021 15:42, Jerin Jacob:
Quoted 53 lines · Thomas Monjalon
Quoted 52 lines
31/07/2021 09:06, Jerin Jacob:
Quoted 50 lines · Thomas Monjalon
Quoted 21 lines
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices like GPUs.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible from the device
- Communication functions to enhance the dialog between the CPU and the deviceThe infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing the hcdev API.Quoted 8 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload
on a different class of CPU.In terms of RFC patch features
1) memory API - Use cases are aligned
2) communication flag and communication list
Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailing list.
Quoted 4 lines
Quoted 1 line
Interesting.
It is hard to generalize the communication mechanism.
Is other GPU vendors have a similar communication mechanism? AMD, Intel ??
I don't know who to ask in AMD & Intel. Any ideas?
Quoted 24 lines
Quoted 16 lines
Quoted 7 lines
Now the bigger question is why need to Tx and then Rx something to compute the device
Isn't ot offload something? If so, why not add the those offload in respective subsystem
to improve thesubsystem(ethdev, cryptiodev etc)features set to adapt new features or
introduce new subsystem (like ML, Inline Baseband processing) so thatQuoted 8 lines
it will be an opportunity to
implement the same in HW or compute device. For example, if we take this path, ML offloading will
be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev)
to deal with specific computing device offload "COMMANDS" (The commands will be specific to offload device, the same code wont run on other compute device)Quoted 7 lines
Having specific features API is convenient for compatibility between devices, yes, for the set of defined features. Our approach is to start with a flexible API that the application can use to implement any processing because with GPU programming, there is no restriction on what can be achieved.
This approach does not contradict what you propose,
it does not prevent extending existing classes.
Quoted 2 lines
It does prevent extending the existing classes as no one is going to extent it there is the path of not doing do.
I disagree. Specific API is more convenient for some tasks, so there is an incentive to define or extend specific device class APIs. But it should not forbid doing custom processing.
Quoted 12 lines
If an application can run only on a specific device, it is similar to a raw device,
where the device definition is not defined. (i.e JOB metadata is not defined and it is specific to the device).Quoted 7 lines
Quoted 6 lines
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of
path. If we decide another path as a community it is _fine_ too(as a_projectmanager_point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).
Quoted 8 lines
Quoted 1 line
Adding a new class API is also improving DPDK.
But the class is similar as raw dev class. The reason I say, Job submission and response is can be abstracted as queue/dequeue APIs. Taks/Job metadata is specific to compute devices (and it can not be generalized).
If we generalize it makes sense to have a new class that does "specific function".
Computing device programming is already generalized with languages like OpenCL. We should not try to reinvent the same.
We are just trying to properly integrate the concept in DPDK
and allow building on top of it.
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 55 lines · Thomas Monjalon
31/07/2021 15:42, Jerin Jacob:
Quoted 53 lines · Thomas Monjalon
Quoted 52 lines
31/07/2021 09:06, Jerin Jacob:
Quoted 50 lines · Thomas Monjalon
Quoted 21 lines
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices like GPUs.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible from the device
- Communication functions to enhance the dialog between the CPU and the deviceThe infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing the hcdev API.Quoted 8 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload
on a different class of CPU.In terms of RFC patch features
1) memory API - Use cases are aligned
2) communication flag and communication list
Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailing list.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Interesting.
It is hard to generalize the communication mechanism.
Is other GPU vendors have a similar communication mechanism? AMD, Intel ??I don't know who to ask in AMD & Intel. Any ideas?
Good question.
At least in Marvell HW, the communication flag and communication list is our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. kind of similar to the DMA API that has been discussed on the mailing list.
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 16 lines
Now the bigger question is why need to Tx and then Rx something to compute the device
Isn't ot offload something? If so, why not add the those offload in respective subsystem
to improve thesubsystem(ethdev, cryptiodev etc)features set to adapt new features or
introduce new subsystem (like ML, Inline Baseband processing) so that it will be an opportunity to
implement the same in HW or compute device. For example, if we take this path, ML offloading will
be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev)
to deal with specific computing device offload "COMMANDS" (The commands will be specific to offload device, the same code wont run on other compute device)Having specific features API is convenient for compatibility between devices, yes, for the set of defined features. Our approach is to start with a flexible API that the application can use to implement any processing because with GPU programming, there is no restriction on what can be achieved.
This approach does not contradict what you propose,
it does not prevent extending existing classes.
Quoted 6 lines
Quoted 2 lines
It does prevent extending the existing classes as no one is going to extent it there is the path of not doing do.
I disagree. Specific API is more convenient for some tasks, so there is an incentive to define or extend specific device class APIs. But it should not forbid doing custom processing.
This is the same as the raw device is in DPDK where the device personality is not defined.
Even if define another API and if the personality is not defined, it comes similar to the raw device as similar
to rawdev enqueue and dequeue.
To summarize,
1) My _personal_ preference is to have specific subsystems
to improve the DPDK instead of the raw device kind of path.
2) If the device personality is not defined, use rawdev
3) All computing devices do not use "communication flag" and "communication list"
kind of structure. If are targeting a generic computing device then that is not a portable scheme.
For GPU abstraction if "communication flag" and "communication list" is the right kind of mechanism
then we can have a separate library for GPU communication specific to GPU <-> DPDK communication needs and explicit for GPU.
I think generic DPDK applications like testpmd should not pollute with device-specific functions. Like, call device-specific messages from the application
which makes the application runs only one device. I don't have a strong opinion(expect
standardizing "communication flag" and "communication list" as generic computing device
communication mechanism) of others think it is OK to do that way in DPDK.
Quoted 22 lines
Quoted 13 lines
If an application can run only on a specific device, it is similar to a raw device,
where the device definition is not defined. (i.e JOB metadata is not defined and it is specific to the device).Quoted 7 lines
Quoted 6 lines
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of
path. If we decide another path as a community it is _fine_ too(as a_projectmanager_point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).Quoted 8 lines
Quoted 1 line
Adding a new class API is also improving DPDK.
But the class is similar as raw dev class. The reason I say, Job submission and response is can be abstracted as queue/dequeue APIs. Taks/Job metadata is specific to compute devices (and it can not be generalized).
If we generalize it makes sense to have a new class that does "specific function".
Quoted 4 lines
Computing device programming is already generalized with languages like OpenCL. We should not try to reinvent the same.
We are just trying to properly integrate the concept in DPDK
and allow building on top of it.
See above.
Quoted 2 lines
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 117 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Friday, August 27, 2021 20:19
To: Thomas Monjalon <[email protected]>
Cc: Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Wang, Haiyue <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Elena Agostini <[email protected]>
Subject: Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 55 lines · Thomas Monjalon
31/07/2021 15:42, Jerin Jacob:
Quoted 53 lines · Thomas Monjalon
Quoted 52 lines
31/07/2021 09:06, Jerin Jacob:
Quoted 50 lines · Thomas Monjalon
Quoted 21 lines
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices like GPUs.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible from the device
- Communication functions to enhance the dialog between the CPU and the deviceThe infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing the hcdev API.Quoted 8 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload
on a different class of CPU.In terms of RFC patch features
1) memory API - Use cases are aligned
2) communication flag and communication list
Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailing list.Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Interesting.
It is hard to generalize the communication mechanism.
Is other GPU vendors have a similar communication mechanism? AMD, Intel ??I don't know who to ask in AMD & Intel. Any ideas?
Good question.
At least in Marvell HW, the communication flag and communication list is our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. kind of similar to the DMA API that has been discussed on the mailing list.Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 16 lines
Now the bigger question is why need to Tx and then Rx something to compute the device
Isn't ot offload something? If so, why not add the those offload in respective subsystem
to improve thesubsystem(ethdev, cryptiodev etc)features set to adapt new features or
introduce new subsystem (like ML, Inline Baseband processing) so that it will be an opportunity to
implement the same in HW or compute device. For example, if we take this path, ML offloading will
be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev)
to deal with specific computing device offload "COMMANDS" (The commands will be specific to offload device, the same code wont run on other compute device)Having specific features API is convenient for compatibility between devices, yes, for the set of defined features. Our approach is to start with a flexible API that the application can use to implement any processing because with GPU programming, there is no restriction on what can be achieved.
This approach does not contradict what you propose,
it does not prevent extending existing classes.Quoted 6 lines
Quoted 2 lines
It does prevent extending the existing classes as no one is going to extent it there is the path of not doing do.
I disagree. Specific API is more convenient for some tasks, so there is an incentive to define or extend specific device class APIs. But it should not forbid doing custom processing.
This is the same as the raw device is in DPDK where the device personality is not defined.
Quoted 8 lines
Even if define another API and if the personality is not defined, it comes similar to the raw device as similar
to rawdev enqueue and dequeue.To summarize,
1) My _personal_ preference is to have specific subsystems to improve the DPDK instead of the raw device kind of path.
Something like rte_memdev to focus on device (GPU) memory management ?
The new DPDK auxiliary bus maybe make life easier to solve the complex heterogeneous computing library. ;-)
Quoted 50 lines
2) If the device personality is not defined, use rawdev
3) All computing devices do not use "communication flag" and "communication list"
kind of structure. If are targeting a generic computing device then that is not a portable scheme.
For GPU abstraction if "communication flag" and "communication list" is the right kind of mechanism
then we can have a separate library for GPU communication specific to GPU <-> DPDK communication needs and explicit for GPU.I think generic DPDK applications like testpmd should not pollute with device-specific functions. Like, call device-specific messages from the application
which makes the application runs only one device. I don't have a strong opinion(expect
standardizing "communication flag" and "communication list" as generic computing device
communication mechanism) of others think it is OK to do that way in DPDK.Quoted 22 lines
Quoted 13 lines
If an application can run only on a specific device, it is similar to a raw device,
where the device definition is not defined. (i.e JOB metadata is not defined and it is specific to the device).Quoted 7 lines
Quoted 6 lines
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of
path. If we decide another path as a community it is _fine_ too(as a_projectmanager_point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).Quoted 8 lines
Quoted 1 line
Adding a new class API is also improving DPDK.
But the class is similar as raw dev class. The reason I say, Job submission and response is can be abstracted as queue/dequeue APIs. Taks/Job metadata is specific to compute devices (and it can not be generalized).
If we generalize it makes sense to have a new class that does "specific function".Quoted 4 lines
Computing device programming is already generalized with languages like OpenCL. We should not try to reinvent the same.
We are just trying to properly integrate the concept in DPDK
and allow building on top of it.See above.
Quoted 2 lines
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 96 lines
-----Original Message-----
From: Wang, Haiyue <[email protected]>
Sent: Sunday, August 29, 2021 7:33 AM
To: Jerin Jacob <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>
Cc: Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Elena Agostini <[email protected]>
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 5 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Friday, August 27, 2021 20:19
To: Thomas Monjalon <[email protected]>
Cc: Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; StephenHemminger
Quoted 1 line
<[email protected]>; David Marchand
<[email protected]>; Andrew Rybchenko
Quoted 1 line
<[email protected]>; Wang, Haiyue <[email protected]>;
Honnappa Nagarahalli
Quoted 1 line
<[email protected]>; Yigit, Ferruh <[email protected]>;
[email protected]; Elena
Quoted 4 lines
Agostini <[email protected]>
Subject: Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryOn Fri, Aug 27, 2021 at 3:14 PM Thomas Monjalon <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines
31/07/2021 15:42, Jerin Jacob:
Quoted 1 line
On Sat, Jul 31, 2021 at 1:51 PM Thomas Monjalon
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
31/07/2021 09:06, Jerin Jacob:
Quoted 1 line
On Fri, Jul 30, 2021 at 7:25 PM Thomas Monjalon
Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the
CPU.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices
like GPUs.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
When mixing network activity with task processing on a non-CPU
device,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
there may be the need to put in communication the CPU with the
device
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in order to manage the memory, synchronize operations, exchange
info, etc..
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible fromthe device
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- Communication functions to enhance the dialog between the CPU
and the device
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 4 lines
The infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing the hcdev API.Quoted 8 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload
on a different class of CPU.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
In terms of RFC patch features
1) memory API - Use cases are aligned
2) communication flag and communication list
Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailing
Quoted 17 lines
list.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Interesting.
It is hard to generalize the communication mechanism. Is other GPU vendors have a similar communication mechanism? AMD,
Intel ??
Quoted 10 lines
Quoted 2 lines
I don't know who to ask in AMD & Intel. Any ideas?
Good question.
At least in Marvell HW, the communication flag and communication list is our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. kind of similar to the DMA API that has been discussed on the mailing list.
Please correct me if I'm wrong but what you are describing is a specific way to submit work on the device. Communication flag/list here is a direct data communication between the CPU and some kind of workload (e.g. GPU kernel) that's already running on the device.
The rationale here is that:
- some work has been already submitted on the device and it's running
- CPU needs a real-time direct interaction through memory with the device
- the workload on the device needs some info from the CPU it can't get at submission time
This is good enough for NVIDIA and AMD GPU.
Need to double check for Intel GPU.
Quoted 42 lines
Quoted 42 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 17 lines
Now the bigger question is why need to Tx and then Rx something to compute the device
Isn't ot offload something? If so, why not add the those offload in respective subsystem
to improve thesubsystem(ethdev, cryptiodev etc)features set to adapt new features or
introduce new subsystem (like ML, Inline Baseband processing) so that it will be an opportunity to
implement the same in HW or compute device. For example, if we take this path, ML offloading will
be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev)
to deal with specific computing device offload "COMMANDS" (The commands will be specific to offload device, the same code wont run on other compute device)Having specific features API is convenient for compatibility between devices, yes, for the set of defined features. Our approach is to start with a flexible API that the application can use to implement any processing because with GPU programming, there is no restriction on what can be achieved.
This approach does not contradict what you propose,
it does not prevent extending existing classes.Quoted 6 lines
Quoted 2 lines
It does prevent extending the existing classes as no one is going to extent it there is the path of not doing do.
I disagree. Specific API is more convenient for some tasks, so there is an incentive to define or extend specific device class APIs. But it should not forbid doing custom processing.
This is the same as the raw device is in DPDK where the device personality is not defined.
Even if define another API and if the personality is not defined, it comes similar to the raw device as similar
to rawdev enqueue and dequeue.To summarize,
Quoted 7 lines
Quoted 2 lines
1) My _personal_ preference is to have specific subsystems to improve the DPDK instead of the raw device kind of path.
Something like
rte_memdevto focus on device (GPU) memory management ?The new DPDK auxiliary bus maybe make life easier to solve the complex heterogeneous computing library. ;-)
To get a concrete idea about what's the best and most comprehensive approach we should start with something that's flexible and simple enough.
A dedicated library it's a good starting point: easy to implement and embed in DPDK applications, isolated from other components and users can play with it learning from the code. As a second step we can think to embed the functionality in some other way
within DPDK (e.g. split memory management and communication features).
Quoted 12 lines
Quoted 12 lines
2) If the device personality is not defined, use rawdev
3) All computing devices do not use "communication flag" and "communication list"
kind of structure. If are targeting a generic computing device then that is not a portable scheme.
For GPU abstraction if "communication flag" and "communication list" is the right kind of mechanism
then we can have a separate library for GPU communication specific to GPU <-DPDK communication needs and explicit for GPU.
Quoted 8 lines
Quoted 8 lines
I think generic DPDK applications like testpmd should not pollute with device-specific functions. Like, call device-specific messages from the application
which makes the application runs only one device. I don't have a strong opinion(expect
standardizing "communication flag" and "communication list" as generic computing device
communication mechanism) of others think it is OK to do that way in DPDK.
I'd like to introduce (with a dedicated option) the memory API in testpmd to provide an example of how to TX/RX packets using device memory.
I agree to not embed communication flag/list features.
Quoted 24 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
If an application can run only on a specific device, it is similar to a raw device,
where the device definition is not defined. (i.e JOB metadata is not definedand
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
it is specific to the device).
Quoted 8 lines
Quoted 6 lines
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of
path. If we decide another path as a community it is _fine_ too(as a_projectmanager_point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).Adding a new class API is also improving DPDK.
But the class is similar as raw dev class. The reason I say, Job submission and response is can be abstracted as queue/dequeue APIs. Taks/Job metadata is specific to compute devices (and it can not be generalized).
If we generalize it makes sense to have a new class that does "specific function".
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Computing device programming is already generalized with languages like
OpenCL.
Quoted 3 lines
Quoted 3 lines
We should not try to reinvent the same.
We are just trying to properly integrate the concept in DPDK and allow building on top of it.
Agree.
Quoted 5 lines
Quoted 5 lines
See above.
Quoted 2 lines
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 115 lines · Elena Agostini
Quoted 97 lines
-----Original Message-----
From: Wang, Haiyue <[email protected]>
Sent: Sunday, August 29, 2021 7:33 AM
To: Jerin Jacob <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>
Cc: Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Elena Agostini <[email protected]>
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 5 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Friday, August 27, 2021 20:19
To: Thomas Monjalon <[email protected]>
Cc: Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; StephenHemminger
Quoted 1 line
<[email protected]>; David Marchand
<[email protected]>; Andrew Rybchenko
Quoted 1 line
<[email protected]>; Wang, Haiyue <[email protected]>;
Honnappa Nagarahalli
Quoted 1 line
<[email protected]>; Yigit, Ferruh <[email protected]>;
[email protected]; Elena
Quoted 4 lines
Agostini <[email protected]>
Subject: Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryOn Fri, Aug 27, 2021 at 3:14 PM Thomas Monjalon <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines
31/07/2021 15:42, Jerin Jacob:
Quoted 1 line
On Sat, Jul 31, 2021 at 1:51 PM Thomas Monjalon
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
31/07/2021 09:06, Jerin Jacob:
Quoted 1 line
On Fri, Jul 30, 2021 at 7:25 PM Thomas Monjalon
Quoted 3 lines · <[email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the
CPU.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices
like GPUs.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
When mixing network activity with task processing on a non-CPU
device,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
there may be the need to put in communication the CPU with the
device
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in order to manage the memory, synchronize operations, exchange
info, etc..
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible fromthe device
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
- Communication functions to enhance the dialog between the CPU
and the device
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 4 lines
The infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing the hcdev API.Quoted 8 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPU memory.Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload
on a different class of CPU.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
In terms of RFC patch features
1) memory API - Use cases are aligned
2) communication flag and communication list
Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailingQuoted 17 lines
list.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Interesting.
It is hard to generalize the communication mechanism. Is other GPU vendors have a similar communication mechanism? AMD,
Intel ??
Quoted 10 lines
Quoted 2 lines
I don't know who to ask in AMD & Intel. Any ideas?
Good question.
At least in Marvell HW, the communication flag and communication list is our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and
the job completion result happens through the event device. kind of similar to the DMA API that has been discussed on the mailing list.
Quoted 4 lines
Please correct me if I'm wrong but what you are describing is a specific way to submit work on the device. Communication flag/list here is a direct data communication between the CPU and some kind of workload (e.g. GPU kernel) that's already running on the device.
Exactly. What I meant is Communication flag/list is not generic enough to express
and generic compute device. If all GPU works in this way, we could make the library
name as GPU specific and add GPU specific communication mechanism.
Quoted 88 lines
The rationale here is that:
- some work has been already submitted on the device and it's running
- CPU needs a real-time direct interaction through memory with the device
- the workload on the device needs some info from the CPU it can't get at submission timeThis is good enough for NVIDIA and AMD GPU.
Need to double check for Intel GPU.Quoted 42 lines
Quoted 42 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 17 lines
Now the bigger question is why need to Tx and then Rx something to compute the device
Isn't ot offload something? If so, why not add the those offload in respective subsystem
to improve thesubsystem(ethdev, cryptiodev etc)features set to adapt new features or
introduce new subsystem (like ML, Inline Baseband processing) so that it will be an opportunity to
implement the same in HW or compute device. For example, if we take this path, ML offloading will
be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev)
to deal with specific computing device offload "COMMANDS" (The commands will be specific to offload device, the same code wont run on other compute device)Having specific features API is convenient for compatibility between devices, yes, for the set of defined features. Our approach is to start with a flexible API that the application can use to implement any processing because with GPU programming, there is no restriction on what can be achieved.
This approach does not contradict what you propose,
it does not prevent extending existing classes.Quoted 6 lines
Quoted 2 lines
It does prevent extending the existing classes as no one is going to extent it there is the path of not doing do.
I disagree. Specific API is more convenient for some tasks, so there is an incentive to define or extend specific device class APIs. But it should not forbid doing custom processing.
This is the same as the raw device is in DPDK where the device personality is not defined.
Even if define another API and if the personality is not defined, it comes similar to the raw device as similar
to rawdev enqueue and dequeue.To summarize,
Quoted 7 lines
Quoted 2 lines
1) My _personal_ preference is to have specific subsystems to improve the DPDK instead of the raw device kind of path.
Something like
rte_memdevto focus on device (GPU) memory management ?The new DPDK auxiliary bus maybe make life easier to solve the complex heterogeneous computing library. ;-)
To get a concrete idea about what's the best and most comprehensive approach we should start with something that's flexible and simple enough.
A dedicated library it's a good starting point: easy to implement and embed in DPDK applications, isolated from other components and users can play with it learning from the code. As a second step we can think to embed the functionality in some other way
within DPDK (e.g. split memory management and communication features).Quoted 12 lines
Quoted 12 lines
2) If the device personality is not defined, use rawdev
3) All computing devices do not use "communication flag" and "communication list"
kind of structure. If are targeting a generic computing device then that is not a portable scheme.
For GPU abstraction if "communication flag" and "communication list" is the right kind of mechanism
then we can have a separate library for GPU communication specific to GPU <-DPDK communication needs and explicit for GPU.
Quoted 8 lines
Quoted 8 lines
I think generic DPDK applications like testpmd should not pollute with device-specific functions. Like, call device-specific messages from the application
which makes the application runs only one device. I don't have a strong opinion(expect
standardizing "communication flag" and "communication list" as generic computing device
communication mechanism) of others think it is OK to do that way in DPDK.
Quoted 2 lines
I'd like to introduce (with a dedicated option) the memory API in testpmd to provide an example of how to TX/RX packets using device memory.
Not sure without embedding sideband communication mechanism how it can notify to GPU and back to CPU. If you could share the example API sequence that helps to us understand the level of coupling with testpmd.
Quoted 40 lines
I agree to not embed communication flag/list features.
Quoted 24 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
If an application can run only on a specific device, it is similar to a raw device,
where the device definition is not defined. (i.e JOB metadata is not definedand
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
it is specific to the device).
Quoted 8 lines
Quoted 6 lines
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of
path. If we decide another path as a community it is _fine_ too(as a_projectmanager_point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).Adding a new class API is also improving DPDK.
But the class is similar as raw dev class. The reason I say, Job submission and response is can be abstracted as queue/dequeue APIs. Taks/Job metadata is specific to compute devices (and it can not be generalized).
If we generalize it makes sense to have a new class that does "specific function".Quoted 5 lines
Quoted 1 line
Quoted 1 line
Computing device programming is already generalized with languages like
OpenCL.
Quoted 3 lines
Quoted 3 lines
We should not try to reinvent the same.
We are just trying to properly integrate the concept in DPDK and allow building on top of it.Agree.
Quoted 5 lines
Quoted 5 lines
See above.
Quoted 2 lines
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 161 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Thursday, September 2, 2021 3:12 PM
To: Elena Agostini <[email protected]>
Cc: Wang, Haiyue <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 6 lines · Elena Agostini
Quoted 6 lines
-----Original Message-----
From: Wang, Haiyue <[email protected]>
Sent: Sunday, August 29, 2021 7:33 AM
To: Jerin Jacob <[email protected]>; NBU-Contact-ThomasMonjalon
Quoted 70 lines
Quoted 70 lines
<[email protected]>
Cc: Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]; Elena Agostini <[email protected]>
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 6 lines
-----Original Message-----
From: Jerin Jacob <[email protected]>
Sent: Friday, August 27, 2021 20:19
To: Thomas Monjalon <[email protected]>
Cc: Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; StephenHemminger
Quoted 1 line
<[email protected]>; David Marchand
<[email protected]>; Andrew Rybchenko
Quoted 2 lines
<[email protected]>; Wang, Haiyue
<[email protected]>;Honnappa Nagarahalli
Quoted 2 lines
<[email protected]>; Yigit, Ferruh
<[email protected]>;[email protected]; Elena
Quoted 6 lines
Agostini <[email protected]>
Subject: Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryOn Fri, Aug 27, 2021 at 3:14 PM Thomas Monjalon
<[email protected]>Quoted 3 lines · wrote:
Quoted 3 lines
31/07/2021 15:42, Jerin Jacob:
Quoted 1 line
On Sat, Jul 31, 2021 at 1:51 PM Thomas Monjalon
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
31/07/2021 09:06, Jerin Jacob:
Quoted 1 line
On Fri, Jul 30, 2021 at 7:25 PM Thomas Monjalon
Quoted 4 lines · <[email protected]> wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the
CPU.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and other type of devices
like GPUs.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
When mixing network activity with task processing on a non-CPU
device,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
there may be the need to put in communication the CPU with the
device
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
in order to manage the memory, synchronize operations, exchange
info, etc..
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
This library provides a number of new features:
- Interoperability with device specific library with generic handlers
- Possibility to allocate and free memory on the device
- Possibility to allocate and free memory on the CPU but visible fromQuoted 7 lines
Quoted 7 lines
the device
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
- Communication functions to enhance the dialog between the CPU
and the device
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The infrastructure is prepared to welcome drivers in
drivers/hc/as the upcoming NVIDIA one, implementing thehcdev API.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Some parts are not complete:
- locks
- memory allocation table
- memory freeing
- guide documentation
- integration indevtools/check-doc-vs-code.sh
- unit tests
- integration in testpmd to enable Rx/Tx to/from GPUmemory.
Quoted 32 lines
Quoted 32 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Since the above line is the crux of the following text, I will start from this point.
+ Techboard
I can give my honest feedback on this.
I can map similar stuff in Marvell HW, where we do machine learning as compute offload on a different class of CPU.
In terms of RFC patch features
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
1) memory API - Use cases are aligned
2) communication flag and communication list Our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and the job completion result happens through the event device. Kind of similar to the DMA API that has been discussed on the mailinglist.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Interesting.
It is hard to generalize the communication mechanism. Is other GPU vendors have a similar communication mechanism? AMD,
Intel ??
Quoted 3 lines
Quoted 2 lines
I don't know who to ask in AMD & Intel. Any ideas?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Good question.
At least in Marvell HW, the communication flag and communication list is our structure is completely different and we are using HW ring kind of interface to post the job to compute interface and the job completion result happens through the event device. kind of similar to the DMA API that has been discussed on the mailing
list.
Quoted 5 lines
Please correct me if I'm wrong but what you are describing is a specific way to submit work on the device. Communication flag/list here is a direct data communication between the CPU and some kind of workload (e.g. GPU kernel) that's already running on the device.
Quoted 4 lines
Exactly. What I meant is Communication flag/list is not generic enough to express and generic compute device. If all GPU works in this way, we could make the library name as GPU specific and add GPU specific communication mechanism.
I'm in favor of reverting the name of the library with a more specific gpudev name instead of hcdev. This library (both memory allocations and fancy features like communication lists) can be tested on various GPUs but I'm not sure about other type of devices.
Again, as initial step, I would not complicate things Let's have a GPU oriented library for now.
Quoted 97 lines
Quoted 10 lines
The rationale here is that:
- some work has been already submitted on the device and it's running
- CPU needs a real-time direct interaction through memory with the device
- the workload on the device needs some info from the CPU it can't get at submission timeQuoted 25 lines
This is good enough for NVIDIA and AMD GPU.
Need to double check for Intel GPU.Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 16 lines
Now the bigger question is why need to Tx and then Rx something to compute the device Isn't ot offload something? If so, why not add the those offload in respective subsystem to improve the
subsystem(ethdev, cryptiodev etc)features set to adapt new features or introduce new subsystem (like ML, Inline Baseband processing) so that it will be an opportunity to implement the same in HW or compute device. For example, if we take this path, ML offloading will be application code like testpmd, which deals with "specific" device commands(aka glorified rawdev) to deal with specific computing device offload "COMMANDS"
(The commands will be specific to offload device, the same code wont run on other compute device)Having specific features API is convenient for compatibility between devices, yes, for the set of defined features. Our approach is to start with a flexible API that the application can use to implement any processing because with GPU programming, there is no restriction on what can be
achieved.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
This approach does not contradict what you propose, it does not prevent extending existing classes.
It does prevent extending the existing classes as no one is going to extent it there is the path of not doing do.
I disagree. Specific API is more convenient for some tasks, so there is an incentive to define or extend specific device class APIs. But it should not forbid doing custom processing.
This is the same as the raw device is in DPDK where the device personality is not defined.
Even if define another API and if the personality is not defined, it comes similar to the raw device as similar to rawdev enqueue and dequeue.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
To summarize,
1) My _personal_ preference is to have specific subsystems to improve the DPDK instead of the raw device kind of path.
Something like
rte_memdevto focus on device (GPU) memorymanagement ?
Quoted 6 lines
Quoted 3 lines
The new DPDK auxiliary bus maybe make life easier to solve the complex heterogeneous computing library. ;-)
To get a concrete idea about what's the best and most comprehensive approach we should start with something that's flexible and simple
enough.
Quoted 3 lines
A dedicated library it's a good starting point: easy to implement and embed in DPDK applications, isolated from other components and users
can play with it learning from the code.
Quoted 2 lines
As a second step we can think to embed the functionality in some other way within DPDK (e.g. split memory management and communication
features).
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
2) If the device personality is not defined, use rawdev
3) All computing devices do not use "communication flag" and "communication list"
kind of structure. If are targeting a generic computing device then that is not a portable scheme.
For GPU abstraction if "communication flag" and "communicationlist"
Quoted 14 lines
Quoted 13 lines
Quoted 13 lines
is the right kind of mechanism
then we can have a separate library for GPU communication specific to GPU <-DPDK communication needs and explicit for GPU.
I think generic DPDK applications like testpmd should not pollute with device-specific functions. Like, call device-specific messages from the application which makes the application runs only one device. I don't have a strong opinion(expect standardizing "communication flag" and "communication list" as generic computing device communication mechanism) of others think it is OK to do that way in DPDK.
Quoted 8 lines
Quoted 2 lines
I'd like to introduce (with a dedicated option) the memory API in testpmd to provide an example of how to TX/RX packets using device
memory.
Not sure without embedding sideband communication mechanism how it can notify to GPU and back to CPU. If you could share the example API sequence that helps to us understand the level of coupling with testpmd.
There is no need of communication mechanism here.
Assuming there is not workload to process network packets (to not complicate things), the steps are:
1) Create a DPDK mempool with device external memory using the hcdev (or gpudev) library
2) Use that mempool to tx/rx/fwd packets
As an example, you look at my l2fwd-nv application here: https://github.com/NVIDIA/l2fwd-nv
Quoted 43 lines
Quoted 23 lines
I agree to not embed communication flag/list features.
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
If an application can run only on a specific device, it is similar to a raw device, where the device definition is not defined. (i.e JOB metadata is not defined
and
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
it is specific to the device).
Quoted 8 lines
Quoted 6 lines
Just my _personal_ preference is to have specific subsystems to improve the DPDK instead of raw device kind of path. If we decide another path as a community it is _fine_ too(as a
_projectmanager_point of view it will be an easy path to dump SDK stuff to DPDK without introducing the pain of the subsystem nor improving the DPDK).Adding a new class API is also improving DPDK.
But the class is similar as raw dev class. The reason I say, Job submission and response is can be abstracted as
queue/dequeue APIs.
Quoted 19 lines
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Taks/Job metadata is specific to compute devices (and it can not be generalized).
If we generalize it makes sense to have a new class that does "specific function".Computing device programming is already generalized with languages like
OpenCL.
Quoted 3 lines
Quoted 3 lines
We should not try to reinvent the same.
We are just trying to properly integrate the concept in DPDK and allow building on top of it.Agree.
Quoted 5 lines
Quoted 5 lines
See above.
Quoted 2 lines
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 24 lines
-----Original Message-----
From: Elena Agostini <[email protected]>
Sent: Tuesday, September 7, 2021 00:11
To: Jerin Jacob <[email protected]>
Cc: Wang, Haiyue <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 9 lines
Quoted 3 lines
I'd like to introduce (with a dedicated option) the memory API in testpmd to provide an example of how to TX/RX packets using device
memory.
Not sure without embedding sideband communication mechanism how it can notify to GPU and back to CPU. If you could share the example API sequence that helps to us understand the level of coupling with testpmd.
Quoted 8 lines
There is no need of communication mechanism here.
Assuming there is not workload to process network packets (to not complicate things), the steps are:
1) Create a DPDK mempool with device external memory using the hcdev (or gpudev) library
2) Use that mempool to tx/rx/fwd packetsAs an example, you look at my l2fwd-nv application here: https://github.com/NVIDIA/l2fwd-nv
To enhance the 'rte_extmem_register' / 'rte_pktmbuf_pool_create_extbuf' ?
if (l2fwd_mem_type == MEM_HOST_PINNED) {
ext_mem.buf_ptr = rte_malloc("extmem", ext_mem.buf_len, 0);
CUDA_CHECK(cudaHostRegister(ext_mem.buf_ptr, ext_mem.buf_len, cudaHostRegisterMapped));
void *pDevice;
CUDA_CHECK(cudaHostGetDevicePointer(&pDevice, ext_mem.buf_ptr, 0));
if (pDevice != ext_mem.buf_ptr)
rte_exit(EXIT_FAILURE, "GPU pointer does not match CPU pointer\n");
} else {
ext_mem.buf_iova = RTE_BAD_IOVA;
CUDA_CHECK(cudaMalloc(&ext_mem.buf_ptr, ext_mem.buf_len));
if (ext_mem.buf_ptr == NULL)
rte_exit(EXIT_FAILURE, "Could not allocate GPU memory\n");
unsigned int flag = 1;
CUresult status = cuPointerSetAttribute(&flag, CU_POINTER_ATTRIBUTE_SYNC_MEMOPS, (CUdeviceptr)ext_mem.buf_ptr);
if (CUDA_SUCCESS != status) {
rte_exit(EXIT_FAILURE, "Could not set SYNC MEMOP attribute for GPU memory at %llx\n", (CUdeviceptr)ext_mem.buf_ptr);
}
ret = rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len, NULL, ext_mem.buf_iova, GPU_PAGE_SIZE);
if (ret)
rte_exit(EXIT_FAILURE, "Could not register GPU memory\n");
}
ret = rte_dev_dma_map(rte_eth_devices[l2fwd_port_id].device, ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len);
if (ret)
rte_exit(EXIT_FAILURE, "Could not DMA map EXT memory\n");
mpool_payload = rte_pktmbuf_pool_create_extbuf("payload_mpool", l2fwd_nb_mbufs,
0, 0, ext_mem.elt_size,
rte_socket_id(), &ext_mem, 1);
if (mpool_payload == NULL)
rte_exit(EXIT_FAILURE, "Could not create EXT memory mempool\n");Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 57 lines
-----Original Message-----
From: Wang, Haiyue <[email protected]>
Sent: Monday, September 6, 2021 7:15 PM
To: Elena Agostini <[email protected]>; Jerin Jacob <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>; Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 5 lines
-----Original Message-----
From: Elena Agostini <[email protected]>
Sent: Tuesday, September 7, 2021 00:11
To: Jerin Jacob <[email protected]>
Cc: Wang, Haiyue <[email protected]>; NBU-Contact-ThomasMonjalon
Quoted 2 lines
<[email protected]>; Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>;
David
Quoted 17 lines
Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 6 lines
Quoted 3 lines
I'd like to introduce (with a dedicated option) the memory API in testpmd to provide an example of how to TX/RX packets using device
memory.
Not sure without embedding sideband communication mechanism how
it
Quoted 2 lines
Quoted 2 lines
can notify to GPU and back to CPU. If you could share the example API sequence that helps to us understand the level of coupling with
testpmd.
Quoted 9 lines
Quoted 1 line
There is no need of communication mechanism here. Assuming there is not workload to process network packets (to not complicate things), the steps are:
1) Create a DPDK mempool with device external memory using the hcdev (or gpudev) library
2) Use that mempool to tx/rx/fwd packets
Quoted 7 lines
Quoted 3 lines
As an example, you look at my l2fwd-nv application here:
https://github.com/NVIDIA/l2fwd-nvTo enhance the '
rte_extmem_register' / 'rte_pktmbuf_pool_create_extbuf' ?
The purpose of these two functions is different.
Here DPDK allows the user to use any kind of memory to rx/tx packets. It's not about allocating memory.
Maybe I'm missing the point here: what's the main objection in having a GPU library?
Quoted 43 lines
if (l2fwd_mem_type == MEM_HOST_PINNED) { ext_mem.buf_ptr = rte_malloc("extmem", ext_mem.buf_len, 0); CUDA_CHECK(cudaHostRegister(ext_mem.buf_ptr, ext_mem.buf_len, cudaHostRegisterMapped)); void *pDevice; CUDA_CHECK(cudaHostGetDevicePointer(&pDevice, ext_mem.buf_ptr, 0)); if (pDevice != ext_mem.buf_ptr) rte_exit(EXIT_FAILURE, "GPU pointer does not match CPU pointer\n"); } else { ext_mem.buf_iova = RTE_BAD_IOVA; CUDA_CHECK(cudaMalloc(&ext_mem.buf_ptr, ext_mem.buf_len)); if (ext_mem.buf_ptr == NULL) rte_exit(EXIT_FAILURE, "Could not allocate GPU memory\n"); unsigned int flag = 1; CUresult status = cuPointerSetAttribute(&flag,
CU_POINTER_ATTRIBUTE_SYNC_MEMOPS, (CUdeviceptr)ext_mem.buf_ptr);
if (CUDA_SUCCESS!= status) {
rte_exit(EXIT_FAILURE, "Could not set SYNC MEMOP attribute
for GPU memory at %llx\n", (CUdeviceptr)ext_mem.buf_ptr);
}
ret = rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len,NULL, ext_mem.buf_iova, GPU_PAGE_SIZE); if (ret) rte_exit(EXIT_FAILURE, "Could not register GPU memory\n"); } ret = rte_dev_dma_map(rte_eth_devices[l2fwd_port_id].device,ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len);
if (ret)
rte_exit(EXIT_FAILURE, "Could not DMA map EXT memory\n");
mpool_payload= rte_pktmbuf_pool_create_extbuf("payload_mpool",l2fwd_nb_mbufs,0, 0, ext_mem.elt_size, rte_socket_id(),
&ext_mem, 1);
if (mpool_payload== NULL)
rte_exit(EXIT_FAILURE, "Could not create EXT memory
mempool\n");
Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library
Quoted 75 lines
-----Original Message-----
From: Elena Agostini <[email protected]>
Sent: Tuesday, September 7, 2021 01:23
To: Wang, Haiyue <[email protected]>; Jerin Jacob <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>; Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 55 lines
-----Original Message-----
From: Wang, Haiyue <[email protected]>
Sent: Monday, September 6, 2021 7:15 PM
To: Elena Agostini <[email protected]>; Jerin Jacob <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>; Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>; David Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 5 lines
-----Original Message-----
From: Elena Agostini <[email protected]>
Sent: Tuesday, September 7, 2021 00:11
To: Jerin Jacob <[email protected]>
Cc: Wang, Haiyue <[email protected]>; NBU-Contact-ThomasMonjalon
Quoted 2 lines
<[email protected]>; Jerin Jacob <[email protected]>; dpdk-dev <[email protected]>; Stephen Hemminger <[email protected]>;
David
Quoted 17 lines
Marchand <[email protected]>; Andrew Rybchenko <[email protected]>; Honnappa Nagarahalli <[email protected]>; Yigit, Ferruh <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing libraryQuoted 6 lines
Quoted 3 lines
I'd like to introduce (with a dedicated option) the memory API in testpmd to provide an example of how to TX/RX packets using device
memory.
Not sure without embedding sideband communication mechanism how
it
Quoted 2 lines
Quoted 2 lines
can notify to GPU and back to CPU. If you could share the example API sequence that helps to us understand the level of coupling with
testpmd.
Quoted 9 lines
Quoted 1 line
There is no need of communication mechanism here. Assuming there is not workload to process network packets (to not complicate things), the steps are:
1) Create a DPDK mempool with device external memory using the hcdev (or gpudev) library
2) Use that mempool to tx/rx/fwd packetsQuoted 7 lines
Quoted 3 lines
As an example, you look at my l2fwd-nv application here:
https://github.com/NVIDIA/l2fwd-nvTo enhance the '
rte_extmem_register' / 'rte_pktmbuf_pool_create_extbuf' ?
Quoted 6 lines
The purpose of these two functions is different.
Here DPDK allows the user to use any kind of memory to rx/tx packets. It's not about allocating memory.Maybe I'm missing the point here: what's the main objection in having a GPU library?
Exactly. ;-)
Maybe a real device code is worth for people to get the whole picture.
Quoted 46 lines
Quoted 31 lines
if (l2fwd_mem_type == MEM_HOST_PINNED) { ext_mem.buf_ptr = rte_malloc("extmem", ext_mem.buf_len, 0); CUDA_CHECK(cudaHostRegister(ext_mem.buf_ptr, ext_mem.buf_len, cudaHostRegisterMapped)); void *pDevice; CUDA_CHECK(cudaHostGetDevicePointer(&pDevice, ext_mem.buf_ptr, 0)); if (pDevice != ext_mem.buf_ptr) rte_exit(EXIT_FAILURE, "GPU pointer does not match CPU pointer\n"); } else { ext_mem.buf_iova = RTE_BAD_IOVA; CUDA_CHECK(cudaMalloc(&ext_mem.buf_ptr, ext_mem.buf_len)); if (ext_mem.buf_ptr == NULL) rte_exit(EXIT_FAILURE, "Could not allocate GPU memory\n"); unsigned int flag = 1; CUresult status = cuPointerSetAttribute(&flag,
CU_POINTER_ATTRIBUTE_SYNC_MEMOPS, (CUdeviceptr)ext_mem.buf_ptr);
if (CUDA_SUCCESS!= status) {
rte_exit(EXIT_FAILURE, "Could not set SYNC MEMOP attribute
for GPU memory at %llx\n", (CUdeviceptr)ext_mem.buf_ptr);
}
ret = rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len,NULL, ext_mem.buf_iova, GPU_PAGE_SIZE); if (ret) rte_exit(EXIT_FAILURE, "Could not register GPU memory\n"); } ret = rte_dev_dma_map(rte_eth_devices[l2fwd_port_id].device,Quoted 6 lines
ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len);
if (ret)
rte_exit(EXIT_FAILURE, "Could not DMA map EXT memory\n");
mpool_payload= rte_pktmbuf_pool_create_extbuf("payload_mpool",l2fwd_nb_mbufs,0, 0,ext_mem.elt_size,
Quoted 1 line
rte_socket_id(),Quoted 6 lines
&ext_mem, 1);
if (mpool_payload== NULL)
rte_exit(EXIT_FAILURE, "Could not create EXT memory
mempool\n");
v3 cover GPU library
In reply to Thomas MonjalonFrom: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPU
The infrastructure is prepared to welcome drivers in drivers/gpu/ as the CUDA one, sent as draft:
https://patches.dpdk.org/project/dpdk/patch/[email protected]/
Elena Agostini (6):
gpudev: introduce GPU device class library
gpudev: add memory API
gpudev: add memory barrier
gpudev: add communication flag
gpudev: add communication list
doc: add CUDA example in GPU guide
Thomas Monjalon (3):
gpudev: add event notification
gpudev: add child device representing a device context
gpudev: support multi-process
.gitignore | 1 + MAINTAINERS | 6 + app/meson.build | 1 + app/test-gpudev/main.c | 394 +++++++++++ app/test-gpudev/meson.build | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 + doc/guides/gpus/features/default.ini | 13 + doc/guides/gpus/index.rst | 11 + doc/guides/gpus/overview.rst | 10 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpudev.rst | 226 +++++++ doc/guides/prog_guide/index.rst | 1 + doc/guides/rel_notes/release_21_11.rst | 6 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpudev.c | 904 +++++++++++++++++++++++++ lib/gpudev/gpudev_driver.h | 102 +++ lib/gpudev/meson.build | 12 + lib/gpudev/rte_gpudev.h | 649 ++++++++++++++++++ lib/gpudev/version.map | 38 ++ lib/meson.build | 1 + 23 files changed, 2396 insertions(+) create mode 100644 app/test-gpudev/main.c create mode 100644 app/test-gpudev/meson.build create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644 doc/guides/gpus/overview.rst create mode 100644 doc/guides/prog_guide/gpudev.rst create mode 100644 drivers/gpu/meson.build create mode 100644 lib/gpudev/gpudev.c create mode 100644 lib/gpudev/gpudev_driver.h create mode 100644 lib/gpudev/meson.build create mode 100644 lib/gpudev/rte_gpudev.h create mode 100644 lib/gpudev/version.map
Signature · 1 line
2.17.1
PATCH v3 1/9 gpudev: introduce GPU device class library
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The new library gpudev is for dealing with GPGPU computing devices from a DPDK application running on the CPU.
The infrastructure is prepared to welcome drivers in drivers/gpu/.
--- .gitignore | 1 + MAINTAINERS | 6 + app/meson.build | 1 + app/test-gpudev/main.c | 107 +++++++++++ app/test-gpudev/meson.build | 5 + doc/api/doxy-api-index.md | 1 + doc/api/doxy-api.conf.in | 1 + doc/guides/conf.py | 8 + doc/guides/gpus/features/default.ini | 10 + doc/guides/gpus/index.rst | 11 ++ doc/guides/gpus/overview.rst | 10 + doc/guides/index.rst | 1 + doc/guides/prog_guide/gpudev.rst | 36 ++++ doc/guides/prog_guide/index.rst | 1 + doc/guides/rel_notes/release_21_11.rst | 4 + drivers/gpu/meson.build | 4 + drivers/meson.build | 1 + lib/gpudev/gpudev.c | 249 +++++++++++++++++++++++++ lib/gpudev/gpudev_driver.h | 67 +++++++ lib/gpudev/meson.build | 10 + lib/gpudev/rte_gpudev.h | 168 +++++++++++++++++ lib/gpudev/version.map | 20 ++ lib/meson.build | 1 + 23 files changed, 723 insertions(+) create mode 100644 app/test-gpudev/main.c create mode 100644 app/test-gpudev/meson.build create mode 100644 doc/guides/gpus/features/default.ini create mode 100644 doc/guides/gpus/index.rst create mode 100644 doc/guides/gpus/overview.rst create mode 100644 doc/guides/prog_guide/gpudev.rst create mode 100644 drivers/gpu/meson.build create mode 100644 lib/gpudev/gpudev.c create mode 100644 lib/gpudev/gpudev_driver.h create mode 100644 lib/gpudev/meson.build create mode 100644 lib/gpudev/rte_gpudev.h create mode 100644 lib/gpudev/version.map
.gitignore +1 -0
@@ -14,6 +14,7 @@ doc/guides/compressdevs/overview_feature_table.txt doc/guides/regexdevs/overview_feature_table.txt doc/guides/vdpadevs/overview_feature_table.txt doc/guides/bbdevs/overview_feature_table.txt +doc/guides/gpus/overview_feature_table.txt # ignore generated ctags/cscope files cscope.out.po
MAINTAINERS +6 -0
@@ -454,6 +454,12 @@ F: app/test-regex/ F: doc/guides/prog_guide/regexdev.rst F: doc/guides/regexdevs/features/default.ini +General-Purpose Graphics Processing Unit (GPU) API - EXPERIMENTAL +M: Elena Agostini <[email protected]> +F: lib/gpudev/ +F: doc/guides/prog_guide/gpudev.rst +F: doc/guides/gpus/features/default.ini + Eventdev API M: Jerin Jacob <[email protected]> T: git://dpdk.org/next/dpdk-next-eventdev
app/meson.build +1 -0
@@ -12,6 +12,7 @@ apps = [ 'test-eventdev', 'test-fib', 'test-flow-perf', + 'test-gpudev', 'test-pipeline', 'test-pmd', 'test-regex',
app/test-gpudev/main.c new file +107 -0
107 lines not on this page · raw message
app/test-gpudev/meson.build new file +5 -0
@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2021 NVIDIA Corporation & Affiliates + +sources = files('main.c') +deps = ['gpudev', 'ethdev']
doc/api/doxy-api-index.md +1 -0
@@ -21,6 +21,7 @@ The public API headers are grouped by topics: [compressdev] (@ref rte_compressdev.h), [compress] (@ref rte_comp.h), [regexdev] (@ref rte_regexdev.h), + [gpudev] (@ref rte_gpudev.h), [eventdev] (@ref rte_eventdev.h), [event_eth_rx_adapter] (@ref rte_event_eth_rx_adapter.h), [event_eth_tx_adapter] (@ref rte_event_eth_tx_adapter.h),
doc/api/doxy-api.conf.in +1 -0
@@ -40,6 +40,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \ @TOPDIR@/lib/eventdev \ @TOPDIR@/lib/fib \ @TOPDIR@/lib/flow_classify \ + @TOPDIR@/lib/gpudev \ @TOPDIR@/lib/graph \ @TOPDIR@/lib/gro \ @TOPDIR@/lib/gso \
doc/guides/conf.py +8 -0
@@ -152,6 +152,9 @@ def generate_overview_table(output_filename, table_id, section, table_name, titl name = ini_filename[:-4] name = name.replace('_vf', 'vf') pmd_names.append(name) + if not pmd_names: + # Add an empty column if table is empty (required by RST syntax) + pmd_names.append(' ') # Pad the table header names. max_header_len = len(max(pmd_names, key=len))
@@ -388,6 +391,11 @@ def setup(app): 'Features', 'Features availability in bbdev drivers', 'Feature') + table_file = dirname(__file__) + '/gpus/overview_feature_table.txt' + generate_overview_table(table_file, 1, + 'Features', + 'Features availability in GPU drivers', + 'Feature') if LooseVersion(sphinx_version) < LooseVersion('1.3.1'): print('Upgrade sphinx to version >= 1.3.1 for '
doc/guides/gpus/features/default.ini new file +10 -0
@@ -0,0 +1,10 @@ +; +; Features of GPU drivers. +; +; This file defines the features that are valid for inclusion in +; the other driver files and also the order that they appear in +; the features table in the documentation. The feature description +; string should not exceed feature_str_len defined in conf.py. +; +[Features] +Get device info =
doc/guides/gpus/index.rst new file +11 -0
@@ -0,0 +1,11 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2021 NVIDIA Corporation & Affiliates + +General-Purpose Graphics Processing Unit Drivers +================================================ + +.. toctree:: + :maxdepth: 2 + :numbered: + + overview
doc/guides/gpus/overview.rst new file +10 -0
@@ -0,0 +1,10 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2021 NVIDIA Corporation & Affiliates + +Overview of GPU Drivers +======================= + +General-Purpose computing on Graphics Processing Unit (GPGPU) +is the use of GPU to perform parallel computation. + +.. include:: overview_feature_table.txt
doc/guides/index.rst +1 -0
@@ -21,6 +21,7 @@ DPDK documentation compressdevs/index vdpadevs/index regexdevs/index + gpus/index eventdevs/index rawdevs/index mempool/index
doc/guides/prog_guide/gpudev.rst new file +36 -0
@@ -0,0 +1,36 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2021 NVIDIA Corporation & Affiliates + +General-Purpose Graphics Processing Unit Library +================================================ + +When mixing networking activity with task processing on a GPU device, +there may be the need to put in communication the CPU with the device +in order to manage the memory, synchronize operations, exchange info, etc.. + +By means of the generic GPU interface provided by this library, +it is possible to allocate a chunk of GPU memory and use it +to create a DPDK mempool with external mbufs having the payload +on the GPU memory, enabling any network interface card +(which support this feature like Mellanox NIC) +to directly transmit and receive packets using GPU memory. + +Additionally, this library provides a number of functions +to enhance the dialog between CPU and GPU. + +Out of scope of this library is to provide a wrapper for GPU specific libraries +(e.g. CUDA Toolkit or OpenCL), thus it is not possible to launch workload +on the device or create GPU specific objects +(e.g. CUDA Driver context or CUDA Streams in case of NVIDIA GPUs). + + +Features +-------- + +This library provides a number of features: + +- Interoperability with device-specific library through generic handlers. + + +API Overview +------------
doc/guides/prog_guide/index.rst +1 -0
@@ -27,6 +27,7 @@ Programmer's Guide cryptodev_lib compressdev regexdev + gpudev rte_security rawdev link_bonding_poll_mode_drv_lib
doc/guides/rel_notes/release_21_11.rst +4 -0
@@ -62,6 +62,10 @@ New Features * Added bus-level parsing of the devargs syntax. * Kept compatibility with the legacy syntax as parsing fallback. +* **Introduced GPU device class with first features:** + + * Device information + * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.** Added macros ETH_RSS_IPV4_CHKSUM and ETH_RSS_L4_CHKSUM, now IPv4 and
drivers/gpu/meson.build new file +4 -0
@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2021 NVIDIA Corporation & Affiliates + +drivers = []
drivers/meson.build +1 -0
@@ -18,6 +18,7 @@ subdirs = [ 'vdpa', # depends on common, bus and mempool. 'event', # depends on common, bus, mempool and net. 'baseband', # depends on common and bus. + 'gpu', # depends on common and bus. ] if meson.is_cross_build()
lib/gpudev/gpudev.c new file +249 -0
249 lines not on this page · raw message
lib/gpudev/gpudev_driver.h new file +67 -0
67 lines not on this page · raw message
lib/gpudev/meson.build new file +10 -0
@@ -0,0 +1,10 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright (c) 2021 NVIDIA Corporation & Affiliates + +headers = files( + 'rte_gpudev.h', +) + +sources = files( + 'gpudev.c', +)
lib/gpudev/rte_gpudev.h new file +168 -0
168 lines not on this page · raw message
lib/gpudev/version.map new file +20 -0
@@ -0,0 +1,20 @@ +EXPERIMENTAL { + global: + + # added in 21.11 + rte_gpu_close; + rte_gpu_count_avail; + rte_gpu_find_next; + rte_gpu_info_get; + rte_gpu_init; + rte_gpu_is_valid; +}; + +INTERNAL { + global: + + rte_gpu_allocate; + rte_gpu_complete_new; + rte_gpu_get_by_name; + rte_gpu_release; +};
lib/meson.build +1 -0
@@ -33,6 +33,7 @@ libraries = [ 'distributor', 'efd', 'eventdev', + 'gpudev', 'gro', 'gso', 'ip_frag',
Signature · 1 line
2.17.1
PATCH v3 2/9 gpudev: add event notification
In reply to [email protected]From: Thomas Monjalon <[email protected]>
Callback functions may be registered for a device event. Callback management is per-process and not thread-safe.
The events RTE_GPU_EVENT_NEW and RTE_GPU_EVENT_DEL are notified respectively after creation and before removal of a device, as part of the library functions. Some future events may be emitted from drivers.
--- lib/gpudev/gpudev.c | 148 +++++++++++++++++++++++++++++++++++++ lib/gpudev/gpudev_driver.h | 7 ++ lib/gpudev/rte_gpudev.h | 70 ++++++++++++++++++ lib/gpudev/version.map | 3 + 4 files changed, 228 insertions(+)
lib/gpudev/gpudev.c +148 -0
@@ -3,6 +3,7 @@*/ #include <rte_eal.h> +#include <rte_tailq.h> #include <rte_string_fns.h> #include <rte_errno.h> #include <rte_log.h>
@@ -27,6 +28,16 @@ static int16_t gpu_max;/* Number of currently valid devices */ static int16_t gpu_count; +/* Event callback object */ +struct rte_gpu_callback { +TAILQ_ENTRY (rte_gpu_callback) next; +rte_gpu_callback_t *function; +void *user_data; +enum rte_gpu_event event; +}; +static rte_rwlock_t gpu_callback_lock =RTE_RWLOCK_INITIALIZER ; +static void gpu_free_callbacks (struct rte_gpu *dev); +int rte_gpu_init (size_t dev_max) {
@@ -166,6 +177,7 @@ rte_gpu_allocate(const char *name) dev->info.name = dev->name; dev->info.dev_id = dev_id; dev->info.numa_node =-1 ; +TAILQ_INIT (&dev->callbacks); gpu_count++; GPU_LOG(DEBUG, "new device %s (id %d) of total %d",
@@ -180,6 +192,8 @@ rte_gpu_complete_new(struct rte_gpu *dev)return ; dev->state =RTE_GPU_STATE_INITIALIZED ; + dev->state =RTE_GPU_STATE_INITIALIZED ; +rte_gpu_notify (dev,RTE_GPU_EVENT_NEW ); }int
@@ -192,6 +206,9 @@ rte_gpu_release(struct rte_gpu *dev)GPU_LOG (DEBUG ,"free device %s (id %d)" , dev->info.name, dev->info.dev_id); +rte_gpu_notify (dev,RTE_GPU_EVENT_DEL ); + +gpu_free_callbacks (dev); dev->state =RTE_GPU_STATE_UNUSED ; gpu_count--;
@@ -224,6 +241,137 @@ rte_gpu_close(int16_t dev_id)return firsterr; } +int +rte_gpu_callback_register (int16_t dev_id,enum rte_gpu_event event, +rte_gpu_callback_t *function,void *user_data) +{ +int16_t next_dev, last_dev; +struct rte_gpu_callback_list *callbacks; +struct rte_gpu_callback *callback; + +if (!rte_gpu_is_valid (dev_id) && dev_id !=RTE_GPU_ID_ANY ) { +GPU_LOG (ERR ,"register callback of invalid ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } +if (function ==NULL ) { +GPU_LOG (ERR ,"cannot register callback without function" ); + rte_errno =EINVAL ; +return -rte_errno; + } + +if (dev_id ==RTE_GPU_ID_ANY ) { + next_dev =0 ; + last_dev = gpu_max -1 ; + }else { + next_dev = last_dev = dev_id; + } + +rte_rwlock_write_lock (&gpu_callback_lock); +do { + callbacks = &gpus[next_dev].callbacks; + +/* check if not already registered */ + TAILQ_FOREACH(callback, callbacks, next) { +if (callback->event == event && + callback->function == function && + callback->user_data == user_data) { +GPU_LOG (INFO ,"callback already registered" ); +return 0 ; + } + } + + callback =malloc (sizeof (*callback)); +if (callback ==NULL ) { +GPU_LOG (ERR ,"cannot allocate callback" ); +return -ENOMEM ; + } + callback->function = function; + callback->user_data = user_data; + callback->event = event; +TAILQ_INSERT_TAIL (callbacks, callback, next); + + }while (++next_dev <= last_dev); +rte_rwlock_write_unlock (&gpu_callback_lock); + +return 0 ; +} + +int +rte_gpu_callback_unregister (int16_t dev_id,enum rte_gpu_event event, +rte_gpu_callback_t *function,void *user_data) +{ +int16_t next_dev, last_dev; +struct rte_gpu_callback_list *callbacks; +struct rte_gpu_callback *callback, *nextcb; + +if (!rte_gpu_is_valid (dev_id) && dev_id !=RTE_GPU_ID_ANY ) { +GPU_LOG (ERR ,"unregister callback of invalid ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } +if (function ==NULL ) { +GPU_LOG (ERR ,"cannot unregister callback without function" ); + rte_errno =EINVAL ; +return -rte_errno; + } + +if (dev_id ==RTE_GPU_ID_ANY ) { + next_dev =0 ; + last_dev = gpu_max -1 ; + }else { + next_dev = last_dev = dev_id; + } + +rte_rwlock_write_lock (&gpu_callback_lock); +do { + callbacks = &gpus[next_dev].callbacks; + RTE_TAILQ_FOREACH_SAFE(callback, callbacks, next, nextcb) { +if (callback->event != event || + callback->function != function || + (callback->user_data != user_data && + user_data != (void *)-1 )) +continue ; +TAILQ_REMOVE (callbacks, callback, next); +free (callback); + } + }while (++next_dev <= last_dev); +rte_rwlock_write_unlock (&gpu_callback_lock); + +return 0 ; +} + +static void +gpu_free_callbacks (struct rte_gpu *dev) +{ +struct rte_gpu_callback_list *callbacks; +struct rte_gpu_callback *callback, *nextcb; + + callbacks = &dev->callbacks; +rte_rwlock_write_lock (&gpu_callback_lock); + RTE_TAILQ_FOREACH_SAFE(callback, callbacks, next, nextcb) { +TAILQ_REMOVE (callbacks, callback, next); +free (callback); + } +rte_rwlock_write_unlock (&gpu_callback_lock); +} + +void +rte_gpu_notify (struct rte_gpu *dev,enum rte_gpu_event event) +{ +int16_t dev_id; +struct rte_gpu_callback *callback; + + dev_id = dev->info.dev_id; +rte_rwlock_read_lock (&gpu_callback_lock); + TAILQ_FOREACH(callback, &dev->callbacks, next) { +if (callback->event != event || callback->function ==NULL ) +continue ; + callback->function (dev_id, event, callback->user_data); + } +rte_rwlock_read_unlock (&gpu_callback_lock); +} +int rte_gpu_info_get (int16_t dev_id,struct rte_gpu_info *info) {
lib/gpudev/gpudev_driver.h +7 -0
@@ -12,6 +12,7 @@#define RTE_GPUDEV_DRIVER_H #include <stdint.h> +#include <sys/queue.h> #include <rte_dev.h>
@@ -43,6 +44,8 @@ struct rte_gpu {struct rte_gpu_info info;/* Driver functions. */ struct rte_gpu_ops ops; +/* Event callback list. */ + TAILQ_HEAD(rte_gpu_callback_list, rte_gpu_callback) callbacks;/* Current state (used or not) in the running process. */ enum rte_gpu_state state;/* Updated by this library. */ /* Driver-specific private data for the running process. */
@@ -64,4 +67,8 @@ void rte_gpu_complete_new(struct rte_gpu *dev);__rte_internal int rte_gpu_release(struct rte_gpu *dev); +/* Call registered callbacks. No multi-process event. */ +__rte_internal +void rte_gpu_notify(struct rte_gpu *dev, enum rte_gpu_event); +#endif /* RTE_GPUDEV_DRIVER_H */
lib/gpudev/rte_gpudev.h +70 -0
88 lines not on this page · raw message
lib/gpudev/version.map +3 -0
@@ -2,6 +2,8 @@ EXPERIMENTAL { global: # added in 21.11 + rte_gpu_callback_register; + rte_gpu_callback_unregister; rte_gpu_close; rte_gpu_count_avail; rte_gpu_find_next;
@@ -16,5 +18,6 @@ INTERNAL { rte_gpu_allocate; rte_gpu_complete_new; rte_gpu_get_by_name; + rte_gpu_notify; rte_gpu_release; };
Signature · 1 line
2.17.1
PATCH v3 3/9 gpudev: add child device representing a device context
In reply to [email protected]From: Thomas Monjalon <[email protected]>
The computing device may operate in some isolated contexts. Memory and processing are isolated in a silo represented by a child device.
The context is provided as an opaque by the caller of rte_gpu_add_child().
--- doc/guides/prog_guide/gpudev.rst | 12 ++++++ lib/gpudev/gpudev.c | 45 +++++++++++++++++++- lib/gpudev/gpudev_driver.h | 2 +- lib/gpudev/rte_gpudev.h | 71 +++++++++++++++++++++++++++++--- lib/gpudev/version.map | 1 + 5 files changed, 123 insertions(+), 8 deletions(-)
doc/guides/prog_guide/gpudev.rst +12 -0
@@ -34,3 +34,15 @@ This library provides a number of features: API Overview ------------ + +Child Device +~~~~~~~~~~~~ + +By default, DPDK PCIe module detects and registers physical GPU devices +in the system. +With the gpudev library is also possible to add additional non-physical devices +through an ``uint64_t`` generic handler (e.g. CUDA Driver context) +that will be registered internally by the driver as an additional device (child) +connected to a physical device (parent). +Each device (parent or child) is represented through a ID +required to indicate which device a given operation should be executed on.
lib/gpudev/gpudev.c +43 -2
78 lines not on this page · raw message
lib/gpudev/gpudev_driver.h +1 -1
@@ -31,7 +31,7 @@ typedef int (rte_gpu_info_get_t)(struct rte_gpu *dev, struct rte_gpu_info *info)struct rte_gpu_ops {/* Get device info. If NULL, info is just copied. */ rte_gpu_info_get_t *dev_info_get; -/* Close device. */ +/* Close device or child context. */ rte_gpu_close_t *dev_close; };
lib/gpudev/rte_gpudev.h +66 -5
108 lines not on this page · raw message
lib/gpudev/version.map +1 -0
@@ -2,6 +2,7 @@ EXPERIMENTAL { global: # added in 21.11 + rte_gpu_add_child; rte_gpu_callback_register; rte_gpu_callback_unregister; rte_gpu_close;
Signature · 1 line
2.17.1
PATCH v3 4/9 gpudev: support multi-process
In reply to [email protected]From: Thomas Monjalon <[email protected]>
The device data shared between processes are moved in a struct allocated in a shared memory (a new memzone for all GPUs). The main struct rte_gpu references the shared memory via the pointer mpshared.
The API function rte_gpu_attach() is added to attach a device from the secondary process.
The function rte_gpu_allocate() can be used only by primary process.
--- lib/gpudev/gpudev.c | 127 +++++++++++++++++++++++++++++++------ lib/gpudev/gpudev_driver.h | 25 ++++++-- lib/gpudev/version.map | 1 + 3 files changed, 127 insertions(+), 26 deletions(-)
lib/gpudev/gpudev.c +107 -20
@@ -5,6 +5,7 @@#include <rte_eal.h> #include <rte_tailq.h> #include <rte_string_fns.h> +#include <rte_memzone.h> #include <rte_errno.h> #include <rte_log.h>
@@ -28,6 +29,12 @@ static int16_t gpu_max;/* Number of currently valid devices */ static int16_t gpu_count; +/* Shared memory between processes. */ +static const char *GPU_MEMZONE ="rte_gpu_shared" ; +static struct { + __extension__struct rte_gpu_mpshared gpus[0 ]; +} *gpu_shared_mem; +/* Event callback object */ struct rte_gpu_callback {TAILQ_ENTRY (rte_gpu_callback) next;
@@ -75,7 +82,7 @@ bool rte_gpu_is_valid(int16_t dev_id) {if (dev_id >=0 && dev_id < gpu_max && - gpus[dev_id].state ==RTE_GPU_STATE_INITIALIZED ) + gpus[dev_id].process_state ==RTE_GPU_STATE_INITIALIZED )return true;return false; }
@@ -85,7 +92,7 @@ gpu_match_parent(int16_t dev_id, int16_t parent) {if (parent ==RTE_GPU_ID_ANY )return true; -return gpus[dev_id].info.parent == parent; +return gpus[dev_id].mpshared->info.parent == parent; }int16_t
@@ -94,7 +101,7 @@ rte_gpu_find_next(int16_t dev_id, int16_t parent)if (dev_id <0 ) dev_id =0 ;while (dev_id < gpu_max && - (gpus[dev_id].state ==RTE_GPU_STATE_UNUSED || + (gpus[dev_id].process_state ==RTE_GPU_STATE_UNUSED || !gpu_match_parent (dev_id, parent))) dev_id++;
@@ -109,7 +116,7 @@ gpu_find_free_id(void)int16_t dev_id;for (dev_id =0 ; dev_id < gpu_max; dev_id++) { -if (gpus[dev_id].state ==RTE_GPU_STATE_UNUSED ) +if (gpus[dev_id].process_state ==RTE_GPU_STATE_UNUSED )return dev_id; }return RTE_GPU_ID_NONE ;
@@ -136,12 +143,35 @@ rte_gpu_get_by_name(const char *name)RTE_GPU_FOREACH (dev_id) { dev = &gpus[dev_id]; -if (strncmp (name, dev->name,RTE_DEV_NAME_MAX_LEN ) ==0 ) +if (strncmp (name, dev->mpshared->name,RTE_DEV_NAME_MAX_LEN ) ==0 )return dev; }return NULL ; } +static int +gpu_shared_mem_init (void ) +{ +const struct rte_memzone *memzone; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { + memzone =rte_memzone_reserve (GPU_MEMZONE , +sizeof (*gpu_shared_mem) + +sizeof (*gpu_shared_mem->gpus) * gpu_max, +SOCKET_ID_ANY ,0 ); + }else { + memzone =rte_memzone_lookup (GPU_MEMZONE ); + } +if (memzone ==NULL ) { +GPU_LOG (ERR ,"cannot initialize shared memory" ); + rte_errno =ENOMEM ; +return -rte_errno; + } + + gpu_shared_mem = memzone->addr; +return 0 ; +} +struct rte_gpu *rte_gpu_allocate (const char *name) {
@@ -163,6 +193,10 @@ rte_gpu_allocate(const char *name)if (gpus ==NULL &&rte_gpu_init (RTE_GPU_DEFAULT_MAX ) <0 )return NULL ; +/* initialize shared memory before adding first device */ +if (gpu_shared_mem ==NULL &&gpu_shared_mem_init () <0 ) +return NULL ; +if (rte_gpu_get_by_name (name) !=NULL ) {GPU_LOG (ERR ,"device with name %s already exists" , name); rte_errno =EEXIST ;
@@ -178,16 +212,20 @@ rte_gpu_allocate(const char *name) dev = &gpus[dev_id];memset (dev,0 ,sizeof (*dev)); -if (rte_strscpy (dev->name, name,RTE_DEV_NAME_MAX_LEN ) <0 ) { + dev->mpshared = &gpu_shared_mem->gpus[dev_id]; +memset (dev->mpshared,0 ,sizeof (*dev->mpshared)); + +if (rte_strscpy (dev->mpshared->name, name,RTE_DEV_NAME_MAX_LEN ) <0 ) {GPU_LOG (ERR ,"device name too long: %s" , name); rte_errno =ENAMETOOLONG ;return NULL ; } - dev->info.name = dev->name; - dev->info.dev_id = dev_id; - dev->info.numa_node =-1 ; - dev->info.parent =RTE_GPU_ID_NONE ; + dev->mpshared->info.name = dev->mpshared->name; + dev->mpshared->info.dev_id = dev_id; + dev->mpshared->info.numa_node =-1 ; + dev->mpshared->info.parent =RTE_GPU_ID_NONE ;TAILQ_INIT (&dev->callbacks); +__atomic_fetch_add (&dev->mpshared->process_refcnt,1 , __ATOMIC_RELAXED); gpu_count++; GPU_LOG(DEBUG, "new device %s (id %d) of total %d",
@@ -195,6 +233,55 @@ rte_gpu_allocate(const char *name)return dev; } +struct rte_gpu * +rte_gpu_attach (const char *name) +{ +int16_t dev_id; +struct rte_gpu *dev; +struct rte_gpu_mpshared *shared_dev; + +if (rte_eal_process_type () !=RTE_PROC_SECONDARY ) { +GPU_LOG (ERR ,"only secondary process can attach device" ); + rte_errno =EPERM ; +return NULL ; + } +if (name ==NULL ) { +GPU_LOG (ERR ,"attach device without a name" ); + rte_errno =EINVAL ; +return NULL ; + } + +/* implicit initialization of library before adding first device */ +if (gpus ==NULL &&rte_gpu_init (RTE_GPU_DEFAULT_MAX ) <0 ) +return NULL ; + +/* initialize shared memory before adding first device */ +if (gpu_shared_mem ==NULL &&gpu_shared_mem_init () <0 ) +return NULL ; + +for (dev_id =0 ; dev_id < gpu_max; dev_id++) { + shared_dev = &gpu_shared_mem->gpus[dev_id]; +if (strncmp (name, shared_dev->name,RTE_DEV_NAME_MAX_LEN ) ==0 ) +break ; + } +if (dev_id >= gpu_max) { +GPU_LOG (ERR ,"device with name %s not found" , name); + rte_errno =ENOENT ; +return NULL ; + } + dev = &gpus[dev_id]; +memset (dev,0 ,sizeof (*dev)); + +TAILQ_INIT (&dev->callbacks); + dev->mpshared = shared_dev; +__atomic_fetch_add (&dev->mpshared->process_refcnt,1 , __ATOMIC_RELAXED); + + gpu_count++; +GPU_LOG (DEBUG ,"attached device %s (id %d) of total %d" , + name, dev_id, gpu_count); +return dev; +} +int16_t rte_gpu_add_child (const char *name,int16_t parent,uint64_t child_context) {
@@ -210,11 +297,11 @@ rte_gpu_add_child(const char *name, int16_t parent, uint64_t child_context)if (dev ==NULL )return -rte_errno; - dev->info.parent = parent; - dev->info.context = child_context; + dev->mpshared->info.parent = parent; + dev->mpshared->info.context = child_context;rte_gpu_complete_new (dev); -return dev->info.dev_id; +return dev->mpshared->info.dev_id; }void
@@ -223,8 +310,7 @@ rte_gpu_complete_new(struct rte_gpu *dev)if (dev ==NULL )return ; - dev->state =RTE_GPU_STATE_INITIALIZED ; - dev->state =RTE_GPU_STATE_INITIALIZED ; + dev->process_state =RTE_GPU_STATE_INITIALIZED ;rte_gpu_notify (dev,RTE_GPU_EVENT_NEW ); }
@@ -237,7 +323,7 @@ rte_gpu_release(struct rte_gpu *dev) rte_errno =ENODEV ;return -rte_errno; } - dev_id = dev->info.dev_id; + dev_id = dev->mpshared->info.dev_id; RTE_GPU_FOREACH_CHILD(child, dev_id) {GPU_LOG (ERR ,"cannot release device %d with child %d" , dev_id, child);
@@ -246,11 +332,12 @@ rte_gpu_release(struct rte_gpu *dev) }GPU_LOG (DEBUG ,"free device %s (id %d)" , - dev->info.name, dev->info.dev_id); + dev->mpshared->info.name, dev->mpshared->info.dev_id);rte_gpu_notify (dev,RTE_GPU_EVENT_DEL );gpu_free_callbacks (dev); - dev->state =RTE_GPU_STATE_UNUSED ; + dev->process_state =RTE_GPU_STATE_UNUSED ; +__atomic_fetch_sub (&dev->mpshared->process_refcnt,1 , __ATOMIC_RELAXED); gpu_count--;return 0 ;
@@ -403,7 +490,7 @@ rte_gpu_notify(struct rte_gpu *dev, enum rte_gpu_event event)int16_t dev_id;struct rte_gpu_callback *callback; - dev_id = dev->info.dev_id; + dev_id = dev->mpshared->info.dev_id;rte_rwlock_read_lock (&gpu_callback_lock); TAILQ_FOREACH(callback, &dev->callbacks, next) {if (callback->event != event || callback->function ==NULL )
@@ -431,7 +518,7 @@ rte_gpu_info_get(int16_t dev_id, struct rte_gpu_info *info) }if (dev->ops.dev_info_get ==NULL ) { - *info = dev->info; + *info = dev->mpshared->info;return 0 ; }return GPU_DRV_RET (dev->ops.dev_info_get (dev, info));
lib/gpudev/gpudev_driver.h +19 -6
@@ -35,19 +35,28 @@ struct rte_gpu_ops {rte_gpu_close_t *dev_close; }; -struct rte_gpu { -/* Backing device. */ -struct rte_device *device; +struct rte_gpu_mpshared {/* Unique identifier name. */ char name[RTE_DEV_NAME_MAX_LEN ];/* Updated by this library. */ +/* Driver-specific private data shared in multi-process. */ +void *dev_private;/* Device info structure. */ struct rte_gpu_info info; +/* Counter of processes using the device. */ +uint16_t process_refcnt;/* Updated by this library. */ +}; + +struct rte_gpu { +/* Backing device. */ +struct rte_device *device; +/* Data shared between processes. */ +struct rte_gpu_mpshared *mpshared;/* Driver functions. */ struct rte_gpu_ops ops;/* Event callback list. */ TAILQ_HEAD(rte_gpu_callback_list, rte_gpu_callback) callbacks;/* Current state (used or not) in the running process. */ -enum rte_gpu_state state;/* Updated by this library. */ +enum rte_gpu_state process_state;/* Updated by this library. */ /* Driver-specific private data for the running process. */ void *process_private; } __rte_cache_aligned;
@@ -55,15 +64,19 @@ struct rte_gpu {__rte_internal struct rte_gpu *rte_gpu_get_by_name(const char *name); -/* First step of initialization */ +/* First step of initialization in primary process. */ __rte_internal struct rte_gpu *rte_gpu_allocate(const char *name); +/* First step of initialization in secondary process. */ +__rte_internal +struct rte_gpu *rte_gpu_attach(const char *name); +/* Last step of initialization. */ __rte_internal void rte_gpu_complete_new(struct rte_gpu *dev); -/* Last step of removal. */ +/* Last step of removal (primary or secondary process). */ __rte_internal int rte_gpu_release(struct rte_gpu *dev);
lib/gpudev/version.map +1 -0
@@ -17,6 +17,7 @@ INTERNAL { global: rte_gpu_allocate; + rte_gpu_attach; rte_gpu_complete_new; rte_gpu_get_by_name; rte_gpu_notify;
Signature · 1 line
2.17.1
PATCH v3 5/9 gpudev: add memory API
In reply to [email protected]From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. Such workload distribution can be achieved by sharing some memory.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the device, or in the main (CPU) memory while making it visible for the device. This memory may be used to save packets or for synchronization data.
The next step should focus on GPU processing task control.
--- app/test-gpudev/main.c | 118 +++++++++++++++++++++++++ doc/guides/gpus/features/default.ini | 3 + doc/guides/prog_guide/gpudev.rst | 19 ++++ doc/guides/rel_notes/release_21_11.rst | 1 + lib/gpudev/gpudev.c | 101 +++++++++++++++++++++ lib/gpudev/gpudev_driver.h | 12 +++ lib/gpudev/rte_gpudev.h | 95 ++++++++++++++++++++ lib/gpudev/version.map | 4 + 8 files changed, 353 insertions(+)
app/test-gpudev/main.c +118 -0
@@ -62,6 +62,110 @@ args_parse(int argc, char **argv) } } +static int +alloc_gpu_memory (uint16_t gpu_id) +{ +void * ptr_1 =NULL ; +void * ptr_2 =NULL ; +size_t buf_bytes =1024 ; +int ret =0 ; + +printf ("\n=======> TEST: Allocate GPU memory\n" ); + +/* Alloc memory on GPU 0 */ + ptr_1 =rte_gpu_malloc (gpu_id, buf_bytes); +if (ptr_1 ==NULL ) + { +fprintf (stderr,"rte_gpu_malloc GPU memory returned error\n" ); +return -1 ; + } +printf ("GPU memory allocated at 0x%p %zdB\n" , ptr_1, buf_bytes); + + ptr_2 =rte_gpu_malloc (gpu_id, buf_bytes); +if (ptr_2 ==NULL ) + { +fprintf (stderr,"rte_gpu_malloc GPU memory returned error\n" ); +return -1 ; + } +printf ("GPU memory allocated at 0x%p %zdB\n" , ptr_2, buf_bytes); + + ret =rte_gpu_free (gpu_id, (uint8_t *)(ptr_1)+0x700 ); +if (ret <0 ) + { +printf ("GPU memory 0x%p + 0x700 NOT freed because of memory address not recognized by driver\n" , ptr_1); + } +else + { +fprintf (stderr,"rte_gpu_free erroneusly freed GPU memory 0x%p + 0x700\n" , ptr_1); +return -1 ; + } + + ret =rte_gpu_free (gpu_id, ptr_2); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_free returned error %d\n" , ret); +return -1 ; + } +printf ("GPU memory 0x%p freed\n" , ptr_2); + + ret =rte_gpu_free (gpu_id, ptr_1); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_free returned error %d\n" , ret); +return -1 ; + } +printf ("GPU memory 0x%p freed\n" , ptr_1); + +return 0 ; +} + +static int +register_cpu_memory (uint16_t gpu_id) +{ +void * ptr =NULL ; +size_t buf_bytes =1024 ; +int ret =0 ; + +printf ("\n=======> TEST: Register CPU memory\n" ); + +/* Alloc memory on CPU visible from GPU 0 */ + ptr =rte_zmalloc (NULL , buf_bytes,0 ); +if (ptr ==NULL ) { +fprintf (stderr,"Failed to allocate CPU memory.\n" ); +return -1 ; + } + + ret =rte_gpu_register (gpu_id, buf_bytes, ptr); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_register CPU memory returned error %d\n" , ret); +return -1 ; + } +printf ("CPU memory registered at 0x%p %zdB\n" , ptr, buf_bytes); + + ret =rte_gpu_unregister (gpu_id, (uint8_t *)(ptr)+0x700 ); +if (ret <0 ) + { +printf ("CPU memory 0x%p + 0x700 NOT unregistered because of memory address not recognized by driver\n" , ptr); + } +else + { +fprintf (stderr,"rte_gpu_free erroneusly freed GPU memory 0x%p + 0x700\n" , ptr); +return -1 ; + } +printf ("CPU memory 0x%p unregistered\n" , ptr); + + ret =rte_gpu_unregister (gpu_id, ptr); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_unregister returned error %d\n" , ret); +return -1 ; + } +printf ("CPU memory 0x%p unregistered\n" , ptr); + +return 0 ; +} +int main (int argc,char **argv) {
@@ -99,6 +203,20 @@ main(int argc, char **argv) }printf ("\n\n" ); +if (nb_gpus ==0 ) + { +fprintf (stderr,"Need at least one GPU on the system to run the example\n" ); +return EXIT_FAILURE ; + } + + gpu_id =0 ; + +/** +* Memory tests +*/ +alloc_gpu_memory (gpu_id); +register_cpu_memory (gpu_id); +/* clean up the EAL */ rte_eal_cleanup ();printf ("Bye...\n" );
doc/guides/gpus/features/default.ini +3 -0
@@ -8,3 +8,6 @@ ; [Features] Get device info = +Share CPU memory with device = +Allocate device memory = +Free memory =
doc/guides/prog_guide/gpudev.rst +19 -0
@@ -30,6 +30,8 @@ Features This library provides a number of features: - Interoperability with device-specific library through generic handlers. +- Allocate and free memory on the device. +- Register CPU memory to make it visible from the device. API Overview
@@ -46,3 +48,20 @@ that will be registered internally by the driver as an additional device (child) connected to a physical device (parent). Each device (parent or child) is represented through a ID required to indicate which device a given operation should be executed on. + +Memory Allocation +~~~~~~~~~~~~~~~~~ + +gpudev can allocate on an input given GPU device a memory area +returning the pointer to that memory. +Later, it's also possible to free that memory with gpudev. +GPU memory allocated outside of the gpudev library +(e.g. with GPU-specific library) cannot be freed by the gpudev library. + +Memory Registration +~~~~~~~~~~~~~~~~~~~ + +gpudev can register a CPU memory area to make it visible from a GPU device. +Later, it's also possible to unregister that memory with gpudev. +CPU memory registered outside of the gpudev library +(e.g. with GPU specific library) cannot be unregistered by the gpudev library.
doc/guides/rel_notes/release_21_11.rst +1 -0
@@ -65,6 +65,7 @@ New Features * **Introduced GPU device class with first features:** * Device information + * Memory management * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.**
lib/gpudev/gpudev.c +101 -0
110 lines not on this page · raw message
lib/gpudev/gpudev_driver.h +12 -0
@@ -27,12 +27,24 @@ enum rte_gpu_state {struct rte_gpu ;typedef int (rte_gpu_close_t )(struct rte_gpu *dev);typedef int (rte_gpu_info_get_t )(struct rte_gpu *dev,struct rte_gpu_info *info); +typedef int (rte_gpu_mem_alloc_t )(struct rte_gpu *dev,size_t size,void **ptr); +typedef int (rte_gpu_free_t )(struct rte_gpu *dev,void *ptr); +typedef int (rte_gpu_mem_register_t )(struct rte_gpu *dev,size_t size,void *ptr); +typedef int (rte_gpu_mem_unregister_t )(struct rte_gpu *dev,void *ptr);struct rte_gpu_ops {/* Get device info. If NULL, info is just copied. */ rte_gpu_info_get_t *dev_info_get;/* Close device or child context. */ rte_gpu_close_t *dev_close; +/* Allocate memory in device. */ +rte_gpu_mem_alloc_t *mem_alloc; +/* Register CPU memory in device. */ +rte_gpu_mem_register_t *mem_register; +/* Free memory allocated or registered in device. */ +rte_gpu_free_t *mem_free; +/* Unregister CPU memory in device. */ +rte_gpu_mem_unregister_t *mem_unregister; };struct rte_gpu_mpshared {
lib/gpudev/rte_gpudev.h +95 -0
107 lines not on this page · raw message
lib/gpudev/version.map +4 -0
@@ -8,9 +8,13 @@ EXPERIMENTAL { rte_gpu_close; rte_gpu_count_avail; rte_gpu_find_next; + rte_gpu_free; rte_gpu_info_get; rte_gpu_init; rte_gpu_is_valid; + rte_gpu_malloc; + rte_gpu_register; + rte_gpu_unregister; }; INTERNAL {
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 5/9] gpudev: add memory API
09/10/2021 03:53, [email protected]:
lib/gpudev/version.map EXPERIMENTAL { · line 18
Quoted 10 lines
lib/gpudev/version.map +4
-0+ rte_gpu_free; rte_gpu_info_get; rte_gpu_init; rte_gpu_is_valid; + rte_gpu_malloc; + rte_gpu_register; + rte_gpu_unregister; };
Should we insert _mem_ in each function name?
Proposal:
rte_gpu_mem_alloc
rte_gpu_mem_free
rte_gpu_mem_register
rte_gpu_mem_unregister
Re: [dpdk-dev] [PATCH v3 5/9] gpudev: add memory API
In reply to [email protected]Quoted 31 lines · On 2021-10-09 03:53, [email protected] wrote:
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. Such workload distribution can be achieved by sharing some memory.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the device, or in the main (CPU) memory while making it visible for the device. This memory may be used to save packets or for synchronization data.
The next step should focus on GPU processing task control.
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>--- app/test-gpudev/main.c | 118 +++++++++++++++++++++++++ doc/guides/gpus/features/default.ini | 3 + doc/guides/prog_guide/gpudev.rst | 19 ++++ doc/guides/rel_notes/release_21_11.rst | 1 + lib/gpudev/gpudev.c | 101 +++++++++++++++++++++ lib/gpudev/gpudev_driver.h | 12 +++ lib/gpudev/rte_gpudev.h | 95 ++++++++++++++++++++ lib/gpudev/version.map | 4 + 8 files changed, 353 insertions(+)app/test-gpudev/main.c +0
-0
app/test-gpudev/main.c args_parse(int argc, char **argv) · line 68
Quoted 7 lines
app/test-gpudev/main.c +4
-0@@ -62,6 +62,110 @@ args_parse(int argc, char **argv) } } +static int +alloc_gpu_memory (uint16_t gpu_id) +{ +void * ptr_1 =NULL ;
Delete space between '*' and 'p'.
app/test-gpudev/main.c args_parse(int argc, char **argv) · line 71
Quoted 3 lines
app/test-gpudev/main.c +3
-0+void * ptr_2 =NULL ; +size_t buf_bytes =1024 ; +int ret =0 ;
This initialization is redundant.
app/test-gpudev/main.c args_parse(int argc, char **argv) · line 78
Quoted 7 lines
app/test-gpudev/main.c +7
-0+ +printf ("\n=======> TEST: Allocate GPU memory\n" ); + +/* Alloc memory on GPU 0 */ + ptr_1 =rte_gpu_malloc (gpu_id, buf_bytes); +if (ptr_1 ==NULL ) + {
Misplaced braces.
"if (" rather than "if(".
app/test-gpudev/main.c args_parse(int argc, char **argv) · line 86
Quoted 8 lines
app/test-gpudev/main.c +8
-0+fprintf (stderr,"rte_gpu_malloc GPU memory returned error\n" ); +return -1 ; + } +printf ("GPU memory allocated at 0x%p %zdB\n" , ptr_1, buf_bytes); + + ptr_2 =rte_gpu_malloc (gpu_id, buf_bytes); +if (ptr_2 ==NULL ) + {
Again, and throughout this file.
Quoted 352 lines
app/test-gpudev/main.c +95
-0+fprintf (stderr,"rte_gpu_malloc GPU memory returned error\n" ); +return -1 ; + } +printf ("GPU memory allocated at 0x%p %zdB\n" , ptr_2, buf_bytes); + + ret =rte_gpu_free (gpu_id, (uint8_t *)(ptr_1)+0x700 ); +if (ret <0 ) + { +printf ("GPU memory 0x%p + 0x700 NOT freed because of memory address not recognized by driver\n" , ptr_1); + } +else + { +fprintf (stderr,"rte_gpu_free erroneusly freed GPU memory 0x%p + 0x700\n" , ptr_1); +return -1 ; + } + + ret =rte_gpu_free (gpu_id, ptr_2); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_free returned error %d\n" , ret); +return -1 ; + } +printf ("GPU memory 0x%p freed\n" , ptr_2); + + ret =rte_gpu_free (gpu_id, ptr_1); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_free returned error %d\n" , ret); +return -1 ; + } +printf ("GPU memory 0x%p freed\n" , ptr_1); + +return 0 ; +} + +static int +register_cpu_memory (uint16_t gpu_id) +{ +void * ptr =NULL ; +size_t buf_bytes =1024 ; +int ret =0 ; + +printf ("\n=======> TEST: Register CPU memory\n" ); + +/* Alloc memory on CPU visible from GPU 0 */ + ptr =rte_zmalloc (NULL , buf_bytes,0 ); +if (ptr ==NULL ) { +fprintf (stderr,"Failed to allocate CPU memory.\n" ); +return -1 ; + } + + ret =rte_gpu_register (gpu_id, buf_bytes, ptr); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_register CPU memory returned error %d\n" , ret); +return -1 ; + } +printf ("CPU memory registered at 0x%p %zdB\n" , ptr, buf_bytes); + + ret =rte_gpu_unregister (gpu_id, (uint8_t *)(ptr)+0x700 ); +if (ret <0 ) + { +printf ("CPU memory 0x%p + 0x700 NOT unregistered because of memory address not recognized by driver\n" , ptr); + } +else + { +fprintf (stderr,"rte_gpu_free erroneusly freed GPU memory 0x%p + 0x700\n" , ptr); +return -1 ; + } +printf ("CPU memory 0x%p unregistered\n" , ptr); + + ret =rte_gpu_unregister (gpu_id, ptr); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_unregister returned error %d\n" , ret); +return -1 ; + } +printf ("CPU memory 0x%p unregistered\n" , ptr); + +return 0 ; +} +int main (int argc,char **argv) {@@ -99,6 +203,20 @@ main(int argc, char **argv) }printf ("\n\n" ); + if(nb_gpus == 0 > + { + fprintf(stderr, "Need at least one GPU on the system to run the example\n"); +return EXIT_FAILURE ; + } + + gpu_id =0 ; + +/** +* Memory tests +*/ +alloc_gpu_memory (gpu_id); +register_cpu_memory (gpu_id); +/* clean up the EAL */ rte_eal_cleanup ();printf ("Bye...\n" );doc/guides/gpus/features/default.ini +3
-0@@ -8,3 +8,6 @@ ; [Features] Get device info = +Share CPU memory with device = +Allocate device memory = +Free memory =doc/guides/prog_guide/gpudev.rst +19
-0@@ -30,6 +30,8 @@ Features This library provides a number of features: - Interoperability with device-specific library through generic handlers. +- Allocate and free memory on the device. +- Register CPU memory to make it visible from the device. API Overview@@ -46,3 +48,20 @@ that will be registered internally by the driver as an additional device (child) connected to a physical device (parent). Each device (parent or child) is represented through a ID required to indicate which device a given operation should be executed on. + +Memory Allocation +~~~~~~~~~~~~~~~~~ + +gpudev can allocate on an input given GPU device a memory area +returning the pointer to that memory. +Later, it's also possible to free that memory with gpudev. +GPU memory allocated outside of the gpudev library +(e.g. with GPU-specific library) cannot be freed by the gpudev library. + +Memory Registration +~~~~~~~~~~~~~~~~~~~ + +gpudev can register a CPU memory area to make it visible from a GPU device. +Later, it's also possible to unregister that memory with gpudev. +CPU memory registered outside of the gpudev library +(e.g. with GPU specific library) cannot be unregistered by the gpudev library.doc/guides/rel_notes/release_21_11.rst +1
-0@@ -65,6 +65,7 @@ New Features * **Introduced GPU device class with first features:** * Device information + * Memory management * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.**lib/gpudev/gpudev.c +101
-0@@ -6,6 +6,7 @@#include <rte_tailq.h> #include <rte_string_fns.h> #include <rte_memzone.h> +#include <rte_malloc.h> #include <rte_errno.h> #include <rte_log.h> @@ -523,3 +524,103 @@ rte_gpu_info_get(int16_t dev_id, struct rte_gpu_info *info) }return GPU_DRV_RET (dev->ops.dev_info_get (dev, info)); } + +void * +rte_gpu_malloc (int16_t dev_id,size_t size) +{ +struct rte_gpu *dev; +void *ptr; +int ret; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"alloc mem for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return NULL ; + } + +if (dev->ops.mem_alloc ==NULL ) { +GPU_LOG (ERR ,"mem allocation not supported" ); + rte_errno =ENOTSUP ; +return NULL ; + } + +if (size ==0 )/* dry-run */ +return NULL ; + + ret = dev->ops.mem_alloc (dev, size, &ptr); + +switch (ret) { +case 0 : +return ptr; +case -ENOMEM : +case -E2BIG : + rte_errno = -ret; +return NULL ; +default : + rte_errno = -EPERM ; +return NULL ; + } +} + +int +rte_gpu_register (int16_t dev_id,size_t size,void * ptr) +{ +struct rte_gpu *dev; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"alloc mem for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } + +if (dev->ops.mem_register ==NULL ) { +GPU_LOG (ERR ,"mem registration not supported" ); + rte_errno =ENOTSUP ; +return -rte_errno; + } + +if (size ==0 || ptr ==NULL )/* dry-run */ +return -EINVAL ; + +return GPU_DRV_RET (dev->ops.mem_register (dev, size, ptr)); +} + +int +rte_gpu_unregister (int16_t dev_id,void * ptr) +{ +struct rte_gpu *dev; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"unregister mem for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } + +if (dev->ops.mem_unregister ==NULL ) { + rte_errno =ENOTSUP ; +return -rte_errno; + } +return GPU_DRV_RET (dev->ops.mem_unregister (dev, ptr)); +} + +int +rte_gpu_free (int16_t dev_id,void *ptr) +{ +struct rte_gpu *dev; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"free mem for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } + +if (dev->ops.mem_free ==NULL ) { + rte_errno =ENOTSUP ; +return -rte_errno; + } +return GPU_DRV_RET (dev->ops.mem_free (dev, ptr)); +}lib/gpudev/gpudev_driver.h +12
-0@@ -27,12 +27,24 @@ enum rte_gpu_state {struct rte_gpu ;typedef int (rte_gpu_close_t )(struct rte_gpu *dev);typedef int (rte_gpu_info_get_t )(struct rte_gpu *dev,struct rte_gpu_info *info); +typedef int (rte_gpu_mem_alloc_t )(struct rte_gpu *dev,size_t size,void **ptr); +typedef int (rte_gpu_free_t )(struct rte_gpu *dev,void *ptr); +typedef int (rte_gpu_mem_register_t )(struct rte_gpu *dev,size_t size,void *ptr); +typedef int (rte_gpu_mem_unregister_t )(struct rte_gpu *dev,void *ptr);struct rte_gpu_ops {/* Get device info. If NULL, info is just copied. */ rte_gpu_info_get_t *dev_info_get;/* Close device or child context. */ rte_gpu_close_t *dev_close; +/* Allocate memory in device. */ +rte_gpu_mem_alloc_t *mem_alloc; +/* Register CPU memory in device. */ +rte_gpu_mem_register_t *mem_register; +/* Free memory allocated or registered in device. */ +rte_gpu_free_t *mem_free; +/* Unregister CPU memory in device. */ +rte_gpu_mem_unregister_t *mem_unregister; };struct rte_gpu_mpshared {lib/gpudev/rte_gpudev.h +30
-0@@ -9,6 +9,7 @@#include <stdint.h> #include <stdbool.h> +#include <rte_bitops.h> #include <rte_compat.h> /** @@ -292,6 +293,100 @@ int rte_gpu_callback_unregister(int16_t dev_id, enum rte_gpu_event event,__rte_experimental int rte_gpu_info_get(int16_t dev_id, struct rte_gpu_info *info); +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Allocate a chunk of memory usable by the device. +* +* @param dev_id +* Device ID requiring allocated memory. +* @param size +* Number of bytes to allocate. +* Requesting 0 will do nothing. +* +* @return +* A pointer to the allocated memory, otherwise NULL and rte_errno is set: +* - ENODEV if invalid dev_id +* - EINVAL if reserved flags +* - ENOTSUP if operation not supported by the driver +* - E2BIG if size is higher than limit +* - ENOMEM if out of space +* - EPERM if driver error +*/ +__rte_experimental +void *rte_gpu_malloc(int16_t dev_id, size_t size) +__rte_alloc_size (2 ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +*
lib/gpudev/rte_gpudev.h int rte_gpu_callback_unregister(int16_t dev_id, enum rte_gpu_event event, · line 334
Quoted 10 lines
lib/gpudev/rte_gpudev.h +10
-0+* Deallocate a chunk of memory allocated with rte_gpu_malloc(). +* +* @param dev_id +* Reference device ID. +* @param ptr +* Pointer to the memory area to be deallocated. +* NULL is a no-op accepted value. +* +* @return +* 0 on success, -rte_errno otherwise:
I don't think you are supposed to set rte_errno if it's not needed, which is not the case here (since you return the error code).
Quoted 77 lines
lib/gpudev/rte_gpudev.h +55
-0+* - ENODEV if invalid dev_id +* - ENOTSUP if operation not supported by the driver +* - EPERM if driver error +*/ +__rte_experimental +int rte_gpu_free(int16_t dev_id, void *ptr); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Register a chunk of memory on the CPU usable by the device. +* +* @param dev_id +* Device ID requiring allocated memory. +* @param size +* Number of bytes to allocate. +* Requesting 0 will do nothing. +* @param ptr +* Pointer to the memory area to be registered. +* NULL is a no-op accepted value. + +* @return +* A pointer to the allocated memory, otherwise NULL and rte_errno is set: +* - ENODEV if invalid dev_id +* - EINVAL if reserved flags +* - ENOTSUP if operation not supported by the driver +* - E2BIG if size is higher than limit +* - ENOMEM if out of space +* - EPERM if driver error +*/ +__rte_experimental +int rte_gpu_register(int16_t dev_id, size_t size, void * ptr); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Deregister a chunk of memory previusly registered with rte_gpu_mem_register() +* +* @param dev_id +* Reference device ID. +* @param ptr +* Pointer to the memory area to be unregistered. +* NULL is a no-op accepted value. +* +* @return +* 0 on success, -rte_errno otherwise: +* - ENODEV if invalid dev_id +* - ENOTSUP if operation not supported by the driver +* - EPERM if driver error +*/ +__rte_experimental +int rte_gpu_unregister(int16_t dev_id, void *ptr); + #ifdef __cplusplus }#endif lib/gpudev/version.map +4
-0@@ -8,9 +8,13 @@ EXPERIMENTAL { rte_gpu_close; rte_gpu_count_avail; rte_gpu_find_next; + rte_gpu_free; rte_gpu_info_get; rte_gpu_init; rte_gpu_is_valid; + rte_gpu_malloc; + rte_gpu_register; + rte_gpu_unregister; }; INTERNAL {
Re: [dpdk-dev] [PATCH v3 5/9] gpudev: add memory API
Quoted 41 lines
From: Mattias Rönnblom <[email protected]>
Date: Friday, 29 October 2021 at 21:38
To: Elena Agostini <[email protected]>, [email protected] <[email protected]>
Cc: NBU-Contact-Thomas Monjalon <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 5/9] gpudev: add memory API>Quoted 34 lines · On 2021-10-09 03:53, [email protected] wrote:
From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. Such workload distribution can be achieved by sharing some memory.
As a first step, the features are focused on memory management. A function allows to allocate memory inside the device, or in the main (CPU) memory while making it visible for the device. This memory may be used to save packets or for synchronization data.
The next step should focus on GPU processing task control.
Signed-off-by Elena Agostini <[email protected]>Signed-off-by Thomas Monjalon <[email protected]>--- app/test-gpudev/main.c | 118 +++++++++++++++++++++++++ doc/guides/gpus/features/default.ini | 3 + doc/guides/prog_guide/gpudev.rst | 19 ++++ doc/guides/rel_notes/release_21_11.rst | 1 + lib/gpudev/gpudev.c | 101 +++++++++++++++++++++ lib/gpudev/gpudev_driver.h | 12 +++ lib/gpudev/rte_gpudev.h | 95 ++++++++++++++++++++ lib/gpudev/version.map | 4 + 8 files changed, 353 insertions(+)app/test-gpudev/main.c +0
-0@@ -62,6 +62,110 @@ args_parse(int argc, char **argv) } }
Quoted 6 lines
app/test-gpudev/main.c args_parse(int argc, char **argv)
Quoted 4 lines
app/test-gpudev/main.c +4
-0+static int +alloc_gpu_memory (uint16_t gpu_id) +{ + void * ptr_1 = NULL;>Delete space between '*' and 'p'.>
Thanks Mattias, I addressed all of your comments and I re-run checkpatch script In all the gpudev related files.
Quoted 472 lines
app/test-gpudev/main.c args_parse(int argc, char **argv)
Quoted 3 lines
app/test-gpudev/main.c +3
-0+void * ptr_2 =NULL ; +size_t buf_bytes =1024 ; + int ret = 0;>This initialization is redundant.>
app/test-gpudev/main.c args_parse(int argc, char **argv)
Quoted 7 lines
app/test-gpudev/main.c +7
-0+ +printf ("\n=======> TEST: Allocate GPU memory\n" ); + +/* Alloc memory on GPU 0 */ + ptr_1 =rte_gpu_malloc (gpu_id, buf_bytes); +if (ptr_1 ==NULL ) + {>Misplaced braces.>
"if (" rather than "if(".>
app/test-gpudev/main.c args_parse(int argc, char **argv)
Quoted 8 lines
app/test-gpudev/main.c +8
-0+fprintf (stderr,"rte_gpu_malloc GPU memory returned error\n" ); +return -1 ; + } +printf ("GPU memory allocated at 0x%p %zdB\n" , ptr_1, buf_bytes); + + ptr_2 =rte_gpu_malloc (gpu_id, buf_bytes); +if (ptr_2 ==NULL ) + {>Again, and throughout this file.>
Quoted 352 lines
app/test-gpudev/main.c +95
-0+fprintf (stderr,"rte_gpu_malloc GPU memory returned error\n" ); +return -1 ; + } +printf ("GPU memory allocated at 0x%p %zdB\n" , ptr_2, buf_bytes); + + ret =rte_gpu_free (gpu_id, (uint8_t *)(ptr_1)+0x700 ); +if (ret <0 ) + { +printf ("GPU memory 0x%p + 0x700 NOT freed because of memory address not recognized by driver\n" , ptr_1); + } +else + { +fprintf (stderr,"rte_gpu_free erroneusly freed GPU memory 0x%p + 0x700\n" , ptr_1); +return -1 ; + } + + ret =rte_gpu_free (gpu_id, ptr_2); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_free returned error %d\n" , ret); +return -1 ; + } +printf ("GPU memory 0x%p freed\n" , ptr_2); + + ret =rte_gpu_free (gpu_id, ptr_1); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_free returned error %d\n" , ret); +return -1 ; + } +printf ("GPU memory 0x%p freed\n" , ptr_1); + +return 0 ; +} + +static int +register_cpu_memory (uint16_t gpu_id) +{ +void * ptr =NULL ; +size_t buf_bytes =1024 ; +int ret =0 ; + +printf ("\n=======> TEST: Register CPU memory\n" ); + +/* Alloc memory on CPU visible from GPU 0 */ + ptr =rte_zmalloc (NULL , buf_bytes,0 ); +if (ptr ==NULL ) { +fprintf (stderr,"Failed to allocate CPU memory.\n" ); +return -1 ; + } + + ret =rte_gpu_register (gpu_id, buf_bytes, ptr); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_register CPU memory returned error %d\n" , ret); +return -1 ; + } +printf ("CPU memory registered at 0x%p %zdB\n" , ptr, buf_bytes); + + ret =rte_gpu_unregister (gpu_id, (uint8_t *)(ptr)+0x700 ); +if (ret <0 ) + { +printf ("CPU memory 0x%p + 0x700 NOT unregistered because of memory address not recognized by driver\n" , ptr); + } +else + { +fprintf (stderr,"rte_gpu_free erroneusly freed GPU memory 0x%p + 0x700\n" , ptr); +return -1 ; + } +printf ("CPU memory 0x%p unregistered\n" , ptr); + + ret =rte_gpu_unregister (gpu_id, ptr); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_unregister returned error %d\n" , ret); +return -1 ; + } +printf ("CPU memory 0x%p unregistered\n" , ptr); + +return 0 ; +} +int main (int argc,char **argv) {@@ -99,6 +203,20 @@ main(int argc, char **argv) }printf ("\n\n" ); + if(nb_gpus == 0 > + { + fprintf(stderr, "Need at least one GPU on the system to run the example\n"); +return EXIT_FAILURE ; + } + + gpu_id =0 ; + +/** +* Memory tests +*/ +alloc_gpu_memory (gpu_id); +register_cpu_memory (gpu_id); +/* clean up the EAL */ rte_eal_cleanup ();printf ("Bye...\n" );doc/guides/gpus/features/default.ini +3
-0@@ -8,3 +8,6 @@ ; [Features] Get device info = +Share CPU memory with device = +Allocate device memory = +Free memory =doc/guides/prog_guide/gpudev.rst +19
-0@@ -30,6 +30,8 @@ Features This library provides a number of features: - Interoperability with device-specific library through generic handlers. +- Allocate and free memory on the device. +- Register CPU memory to make it visible from the device. API Overview@@ -46,3 +48,20 @@ that will be registered internally by the driver as an additional device (child) connected to a physical device (parent). Each device (parent or child) is represented through a ID required to indicate which device a given operation should be executed on. + +Memory Allocation +~~~~~~~~~~~~~~~~~ + +gpudev can allocate on an input given GPU device a memory area +returning the pointer to that memory. +Later, it's also possible to free that memory with gpudev. +GPU memory allocated outside of the gpudev library +(e.g. with GPU-specific library) cannot be freed by the gpudev library. + +Memory Registration +~~~~~~~~~~~~~~~~~~~ + +gpudev can register a CPU memory area to make it visible from a GPU device. +Later, it's also possible to unregister that memory with gpudev. +CPU memory registered outside of the gpudev library +(e.g. with GPU specific library) cannot be unregistered by the gpudev library.doc/guides/rel_notes/release_21_11.rst +1
-0@@ -65,6 +65,7 @@ New Features * **Introduced GPU device class with first features:** * Device information + * Memory management * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.**lib/gpudev/gpudev.c +101
-0@@ -6,6 +6,7 @@#include <rte_tailq.h> #include <rte_string_fns.h> #include <rte_memzone.h> +#include <rte_malloc.h> #include <rte_errno.h> #include <rte_log.h> @@ -523,3 +524,103 @@ rte_gpu_info_get(int16_t dev_id, struct rte_gpu_info *info) }return GPU_DRV_RET (dev->ops.dev_info_get (dev, info)); } + +void * +rte_gpu_malloc (int16_t dev_id,size_t size) +{ +struct rte_gpu *dev; +void *ptr; +int ret; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"alloc mem for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return NULL ; + } + +if (dev->ops.mem_alloc ==NULL ) { +GPU_LOG (ERR ,"mem allocation not supported" ); + rte_errno =ENOTSUP ; +return NULL ; + } + +if (size ==0 )/* dry-run */ +return NULL ; + + ret = dev->ops.mem_alloc (dev, size, &ptr); + +switch (ret) { +case 0 : +return ptr; +case -ENOMEM : +case -E2BIG : + rte_errno = -ret; +return NULL ; +default : + rte_errno = -EPERM ; +return NULL ; + } +} + +int +rte_gpu_register (int16_t dev_id,size_t size,void * ptr) +{ +struct rte_gpu *dev; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"alloc mem for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } + +if (dev->ops.mem_register ==NULL ) { +GPU_LOG (ERR ,"mem registration not supported" ); + rte_errno =ENOTSUP ; +return -rte_errno; + } + +if (size ==0 || ptr ==NULL )/* dry-run */ +return -EINVAL ; + +return GPU_DRV_RET (dev->ops.mem_register (dev, size, ptr)); +} + +int +rte_gpu_unregister (int16_t dev_id,void * ptr) +{ +struct rte_gpu *dev; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"unregister mem for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } + +if (dev->ops.mem_unregister ==NULL ) { + rte_errno =ENOTSUP ; +return -rte_errno; + } +return GPU_DRV_RET (dev->ops.mem_unregister (dev, ptr)); +} + +int +rte_gpu_free (int16_t dev_id,void *ptr) +{ +struct rte_gpu *dev; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"free mem for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } + +if (dev->ops.mem_free ==NULL ) { + rte_errno =ENOTSUP ; +return -rte_errno; + } +return GPU_DRV_RET (dev->ops.mem_free (dev, ptr)); +}lib/gpudev/gpudev_driver.h +12
-0@@ -27,12 +27,24 @@ enum rte_gpu_state {struct rte_gpu ;typedef int (rte_gpu_close_t )(struct rte_gpu *dev);typedef int (rte_gpu_info_get_t )(struct rte_gpu *dev,struct rte_gpu_info *info); +typedef int (rte_gpu_mem_alloc_t )(struct rte_gpu *dev,size_t size,void **ptr); +typedef int (rte_gpu_free_t )(struct rte_gpu *dev,void *ptr); +typedef int (rte_gpu_mem_register_t )(struct rte_gpu *dev,size_t size,void *ptr); +typedef int (rte_gpu_mem_unregister_t )(struct rte_gpu *dev,void *ptr);struct rte_gpu_ops {/* Get device info. If NULL, info is just copied. */ rte_gpu_info_get_t *dev_info_get;/* Close device or child context. */ rte_gpu_close_t *dev_close; +/* Allocate memory in device. */ +rte_gpu_mem_alloc_t *mem_alloc; +/* Register CPU memory in device. */ +rte_gpu_mem_register_t *mem_register; +/* Free memory allocated or registered in device. */ +rte_gpu_free_t *mem_free; +/* Unregister CPU memory in device. */ +rte_gpu_mem_unregister_t *mem_unregister; };struct rte_gpu_mpshared {lib/gpudev/rte_gpudev.h +30
-0@@ -9,6 +9,7 @@#include <stdint.h> #include <stdbool.h> +#include <rte_bitops.h> #include <rte_compat.h> /** @@ -292,6 +293,100 @@ int rte_gpu_callback_unregister(int16_t dev_id, enum rte_gpu_event event,__rte_experimental int rte_gpu_info_get(int16_t dev_id, struct rte_gpu_info *info); +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Allocate a chunk of memory usable by the device. +* +* @param dev_id +* Device ID requiring allocated memory. +* @param size +* Number of bytes to allocate. +* Requesting 0 will do nothing. +* +* @return +* A pointer to the allocated memory, otherwise NULL and rte_errno is set: +* - ENODEV if invalid dev_id +* - EINVAL if reserved flags +* - ENOTSUP if operation not supported by the driver +* - E2BIG if size is higher than limit +* - ENOMEM if out of space +* - EPERM if driver error +*/ +__rte_experimental +void *rte_gpu_malloc(int16_t dev_id, size_t size) +__rte_alloc_size (2 ); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* lib/gpudev/rte_gpudev.h int rte_gpu_callback_unregister(int16_t dev_id, enum rte_gpu_event event, · line 334
Quoted 10 lines
lib/gpudev/rte_gpudev.h +10
-0+* Deallocate a chunk of memory allocated with rte_gpu_malloc(). +* +* @param dev_id +* Reference device ID. +* @param ptr +* Pointer to the memory area to be deallocated. +* NULL is a no-op accepted value. +* +* @return +* 0 on success, -rte_errno otherwise:> I don't think you are supposed to set
rte_errnoif it's not needed, which is not the case here (since you return the error code).>Quoted 77 lines
lib/gpudev/rte_gpudev.h +55
-0+* - ENODEV if invalid dev_id +* - ENOTSUP if operation not supported by the driver +* - EPERM if driver error +*/ +__rte_experimental +int rte_gpu_free(int16_t dev_id, void *ptr); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Register a chunk of memory on the CPU usable by the device. +* +* @param dev_id +* Device ID requiring allocated memory. +* @param size +* Number of bytes to allocate. +* Requesting 0 will do nothing. +* @param ptr +* Pointer to the memory area to be registered. +* NULL is a no-op accepted value. + +* @return +* A pointer to the allocated memory, otherwise NULL and rte_errno is set: +* - ENODEV if invalid dev_id +* - EINVAL if reserved flags +* - ENOTSUP if operation not supported by the driver +* - E2BIG if size is higher than limit +* - ENOMEM if out of space +* - EPERM if driver error +*/ +__rte_experimental +int rte_gpu_register(int16_t dev_id, size_t size, void * ptr); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Deregister a chunk of memory previusly registered with rte_gpu_mem_register() +* +* @param dev_id +* Reference device ID. +* @param ptr +* Pointer to the memory area to be unregistered. +* NULL is a no-op accepted value. +* +* @return +* 0 on success, -rte_errno otherwise: +* - ENODEV if invalid dev_id +* - ENOTSUP if operation not supported by the driver +* - EPERM if driver error +*/ +__rte_experimental +int rte_gpu_unregister(int16_t dev_id, void *ptr); + #ifdef __cplusplus }#endif lib/gpudev/version.map +4
-0@@ -8,9 +8,13 @@ EXPERIMENTAL { rte_gpu_close; rte_gpu_count_avail; rte_gpu_find_next; + rte_gpu_free; rte_gpu_info_get; rte_gpu_init; rte_gpu_is_valid; + rte_gpu_malloc; + rte_gpu_register; + rte_gpu_unregister; }; INTERNAL {
PATCH v3 6/9 gpudev: add memory barrier
In reply to [email protected]From: Elena Agostini <[email protected]>
Add a function for the application to ensure the coherency of the writes executed by another device into the GPU memory.
--- doc/guides/prog_guide/gpudev.rst | 8 ++++++++ lib/gpudev/gpudev.c | 19 +++++++++++++++++++ lib/gpudev/gpudev_driver.h | 3 +++ lib/gpudev/rte_gpudev.h | 18 ++++++++++++++++++ lib/gpudev/version.map | 1 + 5 files changed, 49 insertions(+)
doc/guides/prog_guide/gpudev.rst +8 -0
@@ -65,3 +65,11 @@ gpudev can register a CPU memory area to make it visible from a GPU device. Later, it's also possible to unregister that memory with gpudev. CPU memory registered outside of the gpudev library (e.g. with GPU specific library) cannot be unregistered by the gpudev library. + +Memory Barrier +~~~~~~~~~~~~~~ + +Some GPU drivers may need, under certain conditions, +to enforce the coherency of external devices writes (e.g. NIC receiving packets) +into the GPU memory. +gpudev abstracts and exposes this capability.
lib/gpudev/gpudev.c +19 -0
@@ -624,3 +624,22 @@ rte_gpu_free(int16_t dev_id, void *ptr) }return GPU_DRV_RET (dev->ops.mem_free (dev, ptr)); } + +int +rte_gpu_mbw (int16_t dev_id) +{ +struct rte_gpu *dev; + + dev =gpu_get_by_id (dev_id); +if (dev ==NULL ) { +GPU_LOG (ERR ,"memory barrier for invalid device ID %d" , dev_id); + rte_errno =ENODEV ; +return -rte_errno; + } + +if (dev->ops.mbw ==NULL ) { + rte_errno =ENOTSUP ; +return -rte_errno; + } +return GPU_DRV_RET (dev->ops.mbw (dev)); +}
lib/gpudev/gpudev_driver.h +3 -0
@@ -31,6 +31,7 @@ typedef int (rte_gpu_mem_alloc_t)(struct rte_gpu *dev, size_t size, void **ptr);typedef int (rte_gpu_free_t )(struct rte_gpu *dev,void *ptr);typedef int (rte_gpu_mem_register_t )(struct rte_gpu *dev,size_t size,void *ptr);typedef int (rte_gpu_mem_unregister_t )(struct rte_gpu *dev,void *ptr); +typedef int (rte_gpu_mbw_t )(struct rte_gpu *dev);struct rte_gpu_ops {/* Get device info. If NULL, info is just copied. */
@@ -45,6 +46,8 @@ struct rte_gpu_ops {rte_gpu_free_t *mem_free;/* Unregister CPU memory in device. */ rte_gpu_mem_unregister_t *mem_unregister; +/* Enforce GPU memory write barrier. */ +rte_gpu_mbw_t *mbw; };struct rte_gpu_mpshared {
lib/gpudev/rte_gpudev.h +18 -0
@@ -387,6 +387,24 @@ int rte_gpu_register(int16_t dev_id, size_t size, void * ptr);__rte_experimental int rte_gpu_unregister(int16_t dev_id, void *ptr); +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Enforce a GPU memory write barrier. +* +* @param dev_id +* Reference device ID. +* +* @return +* 0 on success, -rte_errno otherwise: +* - ENODEV if invalid dev_id +* - ENOTSUP if operation not supported by the driver +* - EPERM if driver error +*/ +__rte_experimental +int rte_gpu_mbw(int16_t dev_id); + #ifdef __cplusplus }#endif
lib/gpudev/version.map +1 -0
@@ -13,6 +13,7 @@ EXPERIMENTAL { rte_gpu_init; rte_gpu_is_valid; rte_gpu_malloc; + rte_gpu_mbw; rte_gpu_register; rte_gpu_unregister; };
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 6/9] gpudev: add memory barrier
09/10/2021 03:53, [email protected]:
Quoted 14 lines
From: Elena Agostini <[email protected]>
Add a function for the application to ensure the coherency of the writes executed by another device into the GPU memory.
Signed-off-by Elena Agostini <[email protected]>---
lib/gpudev/rte_gpudev.h +7
-0+/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice. +* +* Enforce a GPU memory write barrier. +* +* @param dev_id
lib/gpudev/rte_gpudev.h int rte_gpu_register(int16_t dev_id, size_t size, void * ptr); · line 406
Quoted 10 lines
lib/gpudev/rte_gpudev.h +10
-0+* Reference device ID. +* +* @return +* 0 on success, -rte_errno otherwise: +* - ENODEV if invalid dev_id +* - ENOTSUP if operation not supported by the driver +* - EPERM if driver error +*/ +__rte_experimental +int rte_gpu_mbw(int16_t dev_id);
I would replace mbw with wmb.
Also it may be worth adding few more words about the goal:
ensure that previous writes in GPU memory are complete? Does it work for writes done from CPU? from GPU?
PATCH v3 7/9 gpudev: add communication flag
In reply to [email protected]From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. When mixing network activity with task processing there may be the need to put in communication the CPU with the device in order to synchronize operations.
The purpose of this flag is to allow the CPU and the GPU to exchange ACKs. A possible use-case is described below.
CPU:
- Trigger some task on the GPU
- Prepare some data
- Signal to the GPU the data is ready updating the communication flag
GPU:
- Do some pre-processing
- Wait for more data from the CPU polling on the communication flag
- Consume the data prepared by the CPU
--- app/test-gpudev/main.c | 66 +++++++++++++++ doc/guides/prog_guide/gpudev.rst | 13 +++ doc/guides/rel_notes/release_21_11.rst | 1 + lib/gpudev/gpudev.c | 94 +++++++++++++++++++++ lib/gpudev/rte_gpudev.h | 108 +++++++++++++++++++++++++ lib/gpudev/version.map | 4 + 6 files changed, 286 insertions(+)
app/test-gpudev/main.c +66 -0
@@ -166,6 +166,67 @@ register_cpu_memory(uint16_t gpu_id)return 0 ; } +static int +create_update_comm_flag (uint16_t gpu_id) +{ +struct rte_gpu_comm_flag devflag; +int ret =0 ; +uint32_t set_val; +uint32_t get_val; + +printf ("\n=======> TEST: Communication flag\n" ); + + ret =rte_gpu_comm_create_flag (gpu_id, &devflag,RTE_GPU_COMM_FLAG_CPU ); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_create_flag returned error %d\n" , ret); +return -1 ; + } + + set_val =25 ; + ret =rte_gpu_comm_set_flag (&devflag, set_val); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_set_flag returned error %d\n" , ret); +return -1 ; + } + + ret =rte_gpu_comm_get_flag_value (&devflag, &get_val); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_get_flag_value returned error %d\n" , ret); +return -1 ; + } + +printf ("Communication flag value at 0x%p was set to %d and current value is %d\n" , devflag.ptr, set_val, get_val); + + set_val =38 ; + ret =rte_gpu_comm_set_flag (&devflag, set_val); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_set_flag returned error %d\n" , ret); +return -1 ; + } + + ret =rte_gpu_comm_get_flag_value (&devflag, &get_val); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_get_flag_value returned error %d\n" , ret); +return -1 ; + } + +printf ("Communication flag value at 0x%p was set to %d and current value is %d\n" , devflag.ptr, set_val, get_val); + + ret =rte_gpu_comm_destroy_flag (&devflag); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_destroy_flags returned error %d\n" , ret); +return -1 ; + } + +return 0 ; +} +int main (int argc,char **argv) {
@@ -217,6 +278,11 @@ main(int argc, char **argv)alloc_gpu_memory (gpu_id);register_cpu_memory (gpu_id); +/** +* Communication items test +*/ +create_update_comm_flag (gpu_id); +/* clean up the EAL */ rte_eal_cleanup ();printf ("Bye...\n" );
doc/guides/prog_guide/gpudev.rst +13 -0
@@ -32,6 +32,10 @@ This library provides a number of features: - Interoperability with device-specific library through generic handlers. - Allocate and free memory on the device. - Register CPU memory to make it visible from the device. +- Communication between the CPU and the device. + +The whole CPU - GPU communication is implemented +using CPU memory visible from the GPU. API Overview
@@ -73,3 +77,12 @@ Some GPU drivers may need, under certain conditions, to enforce the coherency of external devices writes (e.g. NIC receiving packets) into the GPU memory. gpudev abstracts and exposes this capability. + +Communication Flag +~~~~~~~~~~~~~~~~~~ + +Considering an application with some GPU task +that's waiting to receive a signal from the CPU +to move forward with the execution. +The communication flag allocates a CPU memory GPU-visible ``uint32_t`` flag +that can be used by the CPU to communicate with a GPU task.
doc/guides/rel_notes/release_21_11.rst +1 -0
@@ -66,6 +66,7 @@ New Features * Device information * Memory management + * Communication flag * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.**
lib/gpudev/gpudev.c +94 -0
97 lines not on this page · raw message
lib/gpudev/rte_gpudev.h +108 -0
126 lines not on this page · raw message
lib/gpudev/version.map +4 -0
@@ -6,6 +6,10 @@ EXPERIMENTAL { rte_gpu_callback_register; rte_gpu_callback_unregister; rte_gpu_close; + rte_gpu_comm_create_flag; + rte_gpu_comm_destroy_flag; + rte_gpu_comm_get_flag_value; + rte_gpu_comm_set_flag; rte_gpu_count_avail; rte_gpu_find_next; rte_gpu_free;
Signature · 1 line
2.17.1
PATCH v3 8/9 gpudev: add communication list
In reply to [email protected]From: Elena Agostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel. When mixing network activity with task processing there may be the need to put in communication the CPU with the device in order to synchronize operations.
An example could be a receive-and-process application
where CPU is responsible for receiving packets in multiple mbufs and the GPU is responsible for processing the content of those packets.
The purpose of this list is to provide a buffer in CPU memory visible from the GPU that can be treated as a circular buffer
to let the CPU provide fondamental info of received packets to the GPU.
A possible use-case is described below.
CPU:
- Trigger some task on the GPU
- in a loop:
- receive a number of packets
- provide packets info to the GPU
GPU:
- Do some pre-processing
- Wait to receive a new set of packet to be processed
Layout of a communication list would be:
-------
| 0 | => pkt_list
| status |
| #pkts |
-------
| 1 | => pkt_list
| status |
| #pkts |
-------
| 2 | => pkt_list
| status |
| #pkts |
-------
| .... | => pkt_list
---------- app/test-gpudev/main.c | 103 +++++++++++++++ doc/guides/prog_guide/gpudev.rst | 16 +++ doc/guides/rel_notes/release_21_11.rst | 2 +- lib/gpudev/gpudev.c | 165 +++++++++++++++++++++++++ lib/gpudev/meson.build | 2 + lib/gpudev/rte_gpudev.h | 129 +++++++++++++++++++ lib/gpudev/version.map | 4 + 7 files changed, 420 insertions(+), 1 deletion(-)
app/test-gpudev/main.c +103 -0
@@ -227,6 +227,108 @@ create_update_comm_flag(uint16_t gpu_id)return 0 ; } +static int +simulate_gpu_task (struct rte_gpu_comm_list *comm_list_item,int num_pkts) +{ +int idx; + +if (comm_list_item ==NULL ) +return -1 ; + +for (idx =0 ; idx < num_pkts; idx++) { +/** +* consume(comm_list_item->pkt_list[idx].addr); +*/ + } + comm_list_item->status =RTE_GPU_COMM_LIST_DONE ; + +return 0 ; +} + +static int +create_update_comm_list (uint16_t gpu_id) +{ +int ret =0 ; +int i =0 ; +struct rte_gpu_comm_list * comm_list; +uint32_t num_comm_items =1024 ; +struct rte_mbuf * mbufs[10 ]; + +printf ("\n=======> TEST: Communication list\n" ); + + comm_list =rte_gpu_comm_create_list (gpu_id, num_comm_items); +if (comm_list ==NULL ) + { +fprintf (stderr,"rte_gpu_comm_create_list returned error %d\n" , ret); +return -1 ; + } + +/** +* Simulate DPDK receive functions like rte_eth_rx_burst() +*/ +for (i =0 ; i <10 ; i++) + { + mbufs[i] =rte_zmalloc (NULL ,sizeof (struct rte_mbuf ),0 ); +if (mbufs[i] ==NULL ) { +fprintf (stderr,"Failed to allocate fake mbufs in CPU memory.\n" ); +return -1 ; + } + +memset (mbufs[i],0 ,sizeof (struct rte_mbuf )); + } + +/** +* Populate just the first item of the list +*/ + ret =rte_gpu_comm_populate_list_pkts (&(comm_list[0 ]), mbufs,10 ); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_populate_list_pkts returned error %d\n" , ret); +return -1 ; + } + + ret =rte_gpu_comm_cleanup_list (&(comm_list[0 ])); +if (ret ==0 ) + { +fprintf (stderr,"rte_gpu_comm_cleanup_list erroneusly cleaned the list even if packets have not beeing consumed yet\n" ); +return -1 ; + } +else + { +fprintf (stderr,"rte_gpu_comm_cleanup_list correctly didn't clean up the packets because they have not beeing consumed yet\n" ); + } + +/** +* Simulate a GPU tasks going through the packet list to consume +* mbufs packets and release them +*/ +simulate_gpu_task (&(comm_list[0 ]),10 ); + +/** +* Packets have been consumed, now the communication item +* and the related mbufs can be all released +*/ + ret =rte_gpu_comm_cleanup_list (&(comm_list[0 ])); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_cleanup_list returned error %d\n" , ret); +return -1 ; + } + + ret =rte_gpu_comm_destroy_list (comm_list, num_comm_items); +if (ret <0 ) + { +fprintf (stderr,"rte_gpu_comm_destroy_list returned error %d\n" , ret); +return -1 ; + } + +for (i =0 ; i <10 ; i++) +rte_free (mbufs[i]); + +printf ("\nCommunication list test passed!\n" ); +return 0 ; +} +int main (int argc,char **argv) {
@@ -282,6 +384,7 @@ main(int argc, char **argv)* Communication items test */ create_update_comm_flag (gpu_id); +create_update_comm_list (gpu_id);/* clean up the EAL */ rte_eal_cleanup ();
doc/guides/prog_guide/gpudev.rst +16 -0
@@ -86,3 +86,19 @@ that's waiting to receive a signal from the CPU to move forward with the execution. The communication flag allocates a CPU memory GPU-visible ``uint32_t`` flag that can be used by the CPU to communicate with a GPU task. + +Communication list +~~~~~~~~~~~~~~~~~~ + +By default, DPDK pulls free mbufs from a mempool to receive packets. +Best practice, expecially in a multithreaded application, +is to no make any assumption on which mbufs will be used +to receive the next bursts of packets. +Considering an application with a GPU memory mempool +attached to a receive queue having some task waiting on the GPU +to receive a new burst of packets to be processed, +there is the need to communicate from the CPU +the list of mbuf payload addresses where received packet have been stored. +The ``rte_gpu_comm_*()`` functions are responsible to create a list of packets +that can be populated with receive mbuf payload addresses +and communicated to the task running on the GPU.
doc/guides/rel_notes/release_21_11.rst +1 -1
@@ -66,7 +66,7 @@ New Features * Device information * Memory management - * Communication flag + * Communication flag & list * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.**
lib/gpudev/gpudev.c +165 -0
168 lines not on this page · raw message
lib/gpudev/meson.build +2 -0
@@ -8,3 +8,5 @@ headers = files( sources = files( 'gpudev.c', ) + +deps += ['mbuf']
lib/gpudev/rte_gpudev.h +129 -0
153 lines not on this page · raw message
lib/gpudev/version.map +4 -0
@@ -6,9 +6,13 @@ EXPERIMENTAL { rte_gpu_callback_register; rte_gpu_callback_unregister; rte_gpu_close; + rte_gpu_comm_cleanup_list; rte_gpu_comm_create_flag; + rte_gpu_comm_create_list; rte_gpu_comm_destroy_flag; + rte_gpu_comm_destroy_list; rte_gpu_comm_get_flag_value; + rte_gpu_comm_populate_list_pkts; rte_gpu_comm_set_flag; rte_gpu_count_avail; rte_gpu_find_next;
Signature · 1 line
2.17.1
PATCH v3 9/9 doc: add CUDA example in GPU guide
In reply to [email protected]From: Elena Agostini <[email protected]>
--- doc/guides/prog_guide/gpudev.rst | 122 +++++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+)
doc/guides/prog_guide/gpudev.rst +122 -0
@@ -102,3 +102,125 @@ the list of mbuf payload addresses where received packet have been stored. The ``rte_gpu_comm_*()`` functions are responsible to create a list of packets that can be populated with receive mbuf payload addresses and communicated to the task running on the GPU. + + +CUDA Example +------------ + +In the example below, there is a pseudo-code to give an example +about how to use functions in this library in case of a CUDA application. + +.. code-block:: c + + ////////////////////////////////////////////////////////////////////////// + ///// gpudev library + CUDA functions + ////////////////////////////////////////////////////////////////////////// + #define GPU_PAGE_SHIFT 16 + #define GPU_PAGE_SIZE (1UL << GPU_PAGE_SHIFT) + + int main() { + struct rte_gpu_flag quit_flag; + struct rte_gpu_comm_list *comm_list; + int nb_rx = 0; + int comm_list_entry = 0; + struct rte_mbuf * rx_mbufs[max_rx_mbufs]; + cudaStream_t cstream; + struct rte_mempool *mpool_payload, *mpool_header; + struct rte_pktmbuf_extmem ext_mem; + int16_t dev_id; + int16_t port_id = 0; + + /** Initialize CUDA objects (cstream, context, etc..). */ + /** Use gpudev library to register a new CUDA context if any */ + /** Let's assume the application wants to use the default context of the GPU device 0 */ + + dev_id = 0; + + /** + * Create an external memory mempool using memory allocated on the GPU. + */ + ext_mem.elt_size = mbufs_headroom_size; + ext_mem.buf_len = RTE_ALIGN_CEIL(mbufs_num * ext_mem.elt_size, GPU_PAGE_SIZE); + ext_mem.buf_iova = RTE_BAD_IOVA; + ext_mem.buf_ptr = rte_gpu_malloc(dev_id, ext_mem.buf_len, 0); + rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len, NULL, ext_mem.buf_iova, GPU_PAGE_SIZE); + rte_dev_dma_map(rte_eth_devices[port_id].device, ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len); + mpool_payload = rte_pktmbuf_pool_create_extbuf("gpu_mempool", mbufs_num, + 0, 0, ext_mem.elt_size, + rte_socket_id(), &ext_mem, 1); + + /** + * Create CPU - device communication flag. With this flag, the CPU can tell to the CUDA kernel + * to exit from the main loop. + */ + rte_gpu_comm_create_flag(dev_id, &quit_flag, RTE_GPU_COMM_FLAG_CPU); + rte_gpu_comm_set_flag(&quit_flag , 0); + + /** + * Create CPU - device communication list. Each entry of this list will be populated by the CPU + * with a new set of received mbufs that the CUDA kernel has to process. + */ + comm_list = rte_gpu_comm_create_list(dev_id, num_entries); + + /** A very simple CUDA kernel with just 1 CUDA block and RTE_GPU_COMM_LIST_PKTS_MAX CUDA threads. */ + cuda_kernel_packet_processing<<<1, RTE_GPU_COMM_LIST_PKTS_MAX, 0, cstream>>>(quit_flag->ptr, comm_list, num_entries, ...); + + /** + * For simplicity, the CPU here receives only 2 bursts of mbufs. + * In a real application, network activity and device processing should overlap. + */ + nb_rx = rte_eth_rx_burst(port_id, queue_id, &(rx_mbufs[0]), max_rx_mbufs); + rte_gpu_comm_populate_list_pkts(comm_list[0], rx_mbufs, nb_rx); + nb_rx = rte_eth_rx_burst(port_id, queue_id, &(rx_mbufs[0]), max_rx_mbufs); + rte_gpu_comm_populate_list_pkts(comm_list[1], rx_mbufs, nb_rx); + + /** + * CPU waits for the completion of the packets' processing on the CUDA kernel + * and then it does a cleanup of the received mbufs. + */ + while(rte_gpu_comm_cleanup_list(comm_list[0])); + while(rte_gpu_comm_cleanup_list(comm_list[1])); + + /** CPU notifies the CUDA kernel that it has to terminate */ + rte_gpu_comm_set_flag(&quit_flag, 1); + + /** gpudev objects cleanup/destruction */ + /** CUDA cleanup */ + + rte_gpu_free(dev_id, ext_mem.buf_len); + + /** DPDK cleanup */ + + return 0; + } + + ////////////////////////////////////////////////////////////////////////// + ///// CUDA kernel + ////////////////////////////////////////////////////////////////////////// + + void cuda_kernel(uint32_t * quit_flag_ptr, struct rte_gpu_comm_list *comm_list, int comm_list_entries) { + int comm_list_index = 0; + struct rte_gpu_comm_pkt *pkt_list = NULL; + + /** Do some pre-processing operations. */ + + /** GPU kernel keeps checking this flag to know if it has to quit or wait for more packets. */ + while(*quit_flag_ptr == 0) + { + if(comm_list[comm_list_index]->status != RTE_GPU_COMM_LIST_READY) + continue; + + if(threadIdx.x < comm_list[comm_list_index]->num_pkts) + { + /** Each CUDA thread processes a different packet. */ + packet_processing(comm_list[comm_list_index]->addr, comm_list[comm_list_index]->size, ..); + } + __threadfence(); + __syncthreads(); + + /** Wait for new packets on the next communication list entry. */ + comm_list_index = (comm_list_index+1) % comm_list_entries; + } + + /** Do some post-processing operations. */ + }
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
In reply to [email protected]Quoted 13 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
Quoted 5 lines
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPU
In the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.
Irrespective of the RFC I am planning to send and since the new library needs techboard approval, You may
request that the techboard decide approval for this library. Also, As far as I remember a minimum a SW driver in
additional to HW driver to accept a new driver class.
Just my 2c to save your cycles.
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
10/10/2021 12:16, Jerin Jacob:
Quoted 20 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 13 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
Quoted 5 lines
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPU
Quoted 5 lines
In the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?
I don't see these features as driver-specific.
Quoted 4 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.
The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.
gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.
Quoted 5 lines
Irrespective of the RFC I am planning to send and since the new library needs techboard approval, You may
request that the techboard decide approval for this library. Also, As far as I remember a minimum a SW driver in
additional to HW driver to accept a new driver class.
No, only one driver is required:
"When introducing a new device API, at least one driver should implement it." Anyway, a SW driver doesn't make sense for gpudev.
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
Quoted 22 lines · Thomas Monjalon
10/10/2021 12:16, Jerin Jacob:
Quoted 20 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 18 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPU
Quoted 7 lines
Quoted 5 lines
In the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?I don't see these features as driver-specific.
That is the disconnect. I see this as more driver-specific details which are not required
to implement an "application" facing API.
For example, If we need to implement application facing" subsystems like bbdev, If we make all this driver interface, you can still implement the bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how memory is allocated etc
to "application".
Quoted 9 lines
Quoted 5 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 7 lines
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.
See above.
Quoted 8 lines
Quoted 6 lines
Irrespective of the RFC I am planning to send and since the new library needs techboard approval, You may
request that the techboard decide approval for this library. Also, As far as I remember a minimum a SW driver in
additional to HW driver to accept a new driver class.No, only one driver is required:
That is good news.
Quoted 2 lines
"When introducing a new device API, at least one driver should implement it." Anyway, a SW driver doesn't make sense for gpudev.
OK
Quoted 3 lines
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
11/10/2021 10:43, Jerin Jacob:
Quoted 24 lines · Thomas Monjalon
Quoted 23 lines
10/10/2021 12:16, Jerin Jacob:
Quoted 22 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 13 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
Quoted 5 lines
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPUIn the RFC thread, There was one outstanding non technical issues on this,
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?I don't see these features as driver-specific.
That is the disconnect. I see this as more driver-specific details which are not required to implement an "application" facing API.
Indeed this is the disconnect.
I already answered but it seems you don't accept the answer.
First, this is not driver-specific. It is a low-level API.
Quoted 6 lines
For example, If we need to implement application facing" subsystems like bbdev, If we make all this driver interface, you can still implement the bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how memory is allocated etc
to "application".
There are 2 things to understand here.
First we want to allow the application using the GPU for needs which are not exposed by any other DPDK API.
Second, if we want to implement another DPDK API like bbdev, then the GPU implementation would be exposed as a vdev in bbdev, using the HW GPU device being a PCI in gpudev.
They are two different levels, got it?
Quoted 17 lines
Quoted 8 lines
Quoted 4 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 7 lines
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.See above.
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
Quoted 30 lines · Thomas Monjalon
11/10/2021 10:43, Jerin Jacob:
Quoted 28 lines · Thomas Monjalon
Quoted 27 lines
10/10/2021 12:16, Jerin Jacob:
Quoted 25 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 13 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
Quoted 5 lines
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPUIn the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I don't see these features as driver-specific.
That is the disconnect. I see this as more driver-specific details which are not required to implement an "application" facing API.
Indeed this is the disconnect.
I already answered but it seems you don't accept the answer.
Same with you. That is why I requested, we need to get opinions from others. Some of them already provided opinions in RFC.
Quoted 2 lines
First, this is not driver-specific. It is a low-level API.
What is the difference between low-level API and driver-level API.
Quoted 10 lines
Quoted 7 lines
For example, If we need to implement application facing" subsystems like bbdev, If we make all this driver interface, you can still implement the bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how memory is allocated etc
to "application".There are 2 things to understand here.
Quoted 7 lines
First we want to allow the application using the GPU for needs which are not exposed by any other DPDK API.
Second, if we want to implement another DPDK API like bbdev, then the GPU implementation would be exposed as a vdev in bbdev, using the HW GPU device being a PCI in gpudev.
They are two different levels, got it?
Exactly. So what is the point of exposing low-level driver API to "application",
why not it is part of the internal driver API. My point is, why the application needs to worry
about, How the CPU <-> Device communicated? CPU < -> Device memory visibility etc.
Quoted 21 lines
Quoted 18 lines
Quoted 9 lines
Quoted 5 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 7 lines
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.See above.
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
11/10/2021 11:29, Jerin Jacob:
Quoted 32 lines · Thomas Monjalon
Quoted 31 lines
11/10/2021 10:43, Jerin Jacob:
Quoted 30 lines · Thomas Monjalon
Quoted 28 lines
10/10/2021 12:16, Jerin Jacob:
Quoted 20 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 13 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
Quoted 5 lines
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPUQuoted 5 lines
In the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?I don't see these features as driver-specific.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
That is the disconnect. I see this as more driver-specific details which are not required to implement an "application" facing API.
Indeed this is the disconnect.
I already answered but it seems you don't accept the answer.Same with you. That is why I requested, we need to get opinions from others. Some of them already provided opinions in RFC.
This is why I Cc'ed techboard.
Quoted 3 lines
Quoted 1 line
First, this is not driver-specific. It is a low-level API.
What is the difference between low-level API and driver-level API.
The low-level API provides tools to build a feature, but no specific feature.
Quoted 17 lines
Quoted 9 lines
Quoted 6 lines
For example, If we need to implement application facing" subsystems like bbdev, If we make all this driver interface, you can still implement the bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how memory is allocated etc
to "application".There are 2 things to understand here.
Quoted 7 lines
First we want to allow the application using the GPU for needs which are not exposed by any other DPDK API.
Second, if we want to implement another DPDK API like bbdev, then the GPU implementation would be exposed as a vdev in bbdev, using the HW GPU device being a PCI in gpudev.
They are two different levels, got it?
Quoted 6 lines
Exactly. So what is the point of exposing low-level driver API to "application",
why not it is part of the internal driver API. My point is, why the application needs to worry
about, How the CPU <-> Device communicated? CPU < -> Device memory visibility etc.
There are two reasons.
1/ The application may want to use the GPU for some application-specific needs which are not abstracted in DPDK API.
2/ This API may also be used by some feature implementation internally in some DPDK libs or drivers.
We cannot skip the gpudev layer because this is what allows generic probing of the HW, and gpudev allows to share the GPU with multiple features implemented in different libs or drivers, thanks to the "child" concept.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
Quoted 4 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 7 lines
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.See above.
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
Quoted 37 lines · Thomas Monjalon
11/10/2021 11:29, Jerin Jacob:
Quoted 35 lines · Thomas Monjalon
Quoted 34 lines
11/10/2021 10:43, Jerin Jacob:
Quoted 32 lines · Thomas Monjalon
Quoted 21 lines
10/10/2021 12:16, Jerin Jacob:
Quoted 20 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 18 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPUQuoted 7 lines
Quoted 5 lines
In the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?I don't see these features as driver-specific.
That is the disconnect. I see this as more driver-specific details which are not required to implement an "application" facing API.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Indeed this is the disconnect.
I already answered but it seems you don't accept the answer.Same with you. That is why I requested, we need to get opinions from others. Some of them already provided opinions in RFC.
This is why I Cc'ed techboard.
Yes. Indeed.
Quoted 32 lines
Quoted 4 lines
Quoted 2 lines
First, this is not driver-specific. It is a low-level API.
What is the difference between low-level API and driver-level API.
The low-level API provides tools to build a feature, but no specific feature.
Quoted 17 lines
Quoted 9 lines
Quoted 6 lines
For example, If we need to implement application facing" subsystems like bbdev, If we make all this driver interface, you can still implement the bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how memory is allocated etc
to "application".There are 2 things to understand here.
Quoted 7 lines
First we want to allow the application using the GPU for needs which are not exposed by any other DPDK API.
Second, if we want to implement another DPDK API like bbdev, then the GPU implementation would be exposed as a vdev in bbdev, using the HW GPU device being a PCI in gpudev.
They are two different levels, got it?Quoted 6 lines
Exactly. So what is the point of exposing low-level driver API to "application",
why not it is part of the internal driver API. My point is, why the application needs to worry
about, How the CPU <-> Device communicated? CPU < -> Device memory visibility etc.
Quoted 4 lines
There are two reasons.
1/ The application may want to use the GPU for some application-specific needs which are not abstracted in DPDK API.
Yes. Exactly, That's where my concern, If we take this path, What is the motivation to contribute to DPDK abstracted subsystem APIs which make sense for multiple vendors and every
Similar stuff applicable for DPU,
Otherway to put, if GPU is doing some ethdev offload, why not making as ethdev offload in ethdev spec so that
another type of device can be used and make sense for application writters.
For example, In the future, If someone needs to add ML(Machine learning) subsystem and enable a proper subsystem
interface that is good for DPDK. If this path is open, there is no motivation for contribution and the application
will not have a standard interface doing the ML job across multiple vendors.
That's is the only reason why saying it should not APPLICATION interface it can be DRIVER interface.
Quoted 6 lines
2/ This API may also be used by some feature implementation internally in some DPDK libs or drivers.
We cannot skip the gpudev layer because this is what allows generic probing of the HW, and gpudev allows to share the GPU with multiple features implemented in different libs or drivers, thanks to the "child" concept.
Again, why do applications need to know it? It is similar to bus kind of this where it
sharing the physical resouces.
Quoted 22 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
Quoted 6 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 7 lines
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.See above.
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
11/10/2021 13:41, Jerin Jacob:
Quoted 81 lines · Thomas Monjalon
Quoted 36 lines
11/10/2021 11:29, Jerin Jacob:
Quoted 35 lines · Thomas Monjalon
Quoted 34 lines
11/10/2021 10:43, Jerin Jacob:
Quoted 32 lines · Thomas Monjalon
Quoted 21 lines
10/10/2021 12:16, Jerin Jacob:
Quoted 20 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 18 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPUQuoted 7 lines
Quoted 5 lines
In the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?I don't see these features as driver-specific.
That is the disconnect. I see this as more driver-specific details which are not required to implement an "application" facing API.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Indeed this is the disconnect.
I already answered but it seems you don't accept the answer.Same with you. That is why I requested, we need to get opinions from others. Some of them already provided opinions in RFC.
This is why I Cc'ed techboard.
Yes. Indeed.
Quoted 34 lines
Quoted 4 lines
Quoted 2 lines
First, this is not driver-specific. It is a low-level API.
What is the difference between low-level API and driver-level API.
The low-level API provides tools to build a feature, but no specific feature.
Quoted 17 lines
Quoted 9 lines
Quoted 6 lines
For example, If we need to implement application facing" subsystems like bbdev, If we make all this driver interface, you can still implement the bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how memory is allocated etc
to "application".There are 2 things to understand here.
Quoted 7 lines
First we want to allow the application using the GPU for needs which are not exposed by any other DPDK API.
Second, if we want to implement another DPDK API like bbdev, then the GPU implementation would be exposed as a vdev in bbdev, using the HW GPU device being a PCI in gpudev.
They are two different levels, got it?Quoted 6 lines
Exactly. So what is the point of exposing low-level driver API to "application",
why not it is part of the internal driver API. My point is, why the application needs to worry
about, How the CPU <-> Device communicated? CPU < -> Device memory visibility etc.There are two reasons.
Quoted 7 lines
Quoted 2 lines
1/ The application may want to use the GPU for some application-specific needs which are not abstracted in DPDK API.
Yes. Exactly, That's where my concern, If we take this path, What is the motivation to contribute to DPDK abstracted subsystem APIs which make sense for multiple vendors and every
Similar stuff applicable for DPU,
A feature-specific API is better of course, there is no lose of motivation. But you cannot forbid applications to have their own features on GPU.
Quoted 3 lines
Otherway to put, if GPU is doing some ethdev offload, why not making as ethdev offload in ethdev spec so that
another type of device can be used and make sense for application writters.
If we do ethdev offload, yes we'll implement it.
And we'll do it on top of gpudev, which is the only way to share the CPU.
Quoted 5 lines
For example, In the future, If someone needs to add ML(Machine learning) subsystem and enable a proper subsystem
interface that is good for DPDK. If this path is open, there is no motivation for contribution and the application
will not have a standard interface doing the ML job across multiple vendors.
Wrong. It does remove the motivation, it is a first step to build on top of it.
Quoted 4 lines
That's is the only reason why saying it should not APPLICATION interface it can be DRIVER interface.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
2/ This API may also be used by some feature implementation internally in some DPDK libs or drivers.
We cannot skip the gpudev layer because this is what allows generic probing of the HW, and gpudev allows to share the GPU with multiple features implemented in different libs or drivers, thanks to the "child" concept.Again, why do applications need to know it? It is similar to
buskind of this where it sharing the physical resouces.
No it's not a bus, it is a device that we need to share.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 4 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.See above.
We are going into circles.
In short, Jerin wants to forbid the generic use of GPU in DPDK. He wants only feature-specific API.
It is like restricting the functions we can run on a CPU.
And anyway we need this layer to share the GPU between multiple features.
Techboard please vote.
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
Quoted 86 lines · Thomas Monjalon
11/10/2021 13:41, Jerin Jacob:
Quoted 84 lines · Thomas Monjalon
Quoted 36 lines
11/10/2021 11:29, Jerin Jacob:
Quoted 35 lines · Thomas Monjalon
Quoted 34 lines
11/10/2021 10:43, Jerin Jacob:
Quoted 32 lines · Thomas Monjalon
Quoted 21 lines
10/10/2021 12:16, Jerin Jacob:
Quoted 20 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 18 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPUQuoted 7 lines
Quoted 5 lines
In the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?I don't see these features as driver-specific.
That is the disconnect. I see this as more driver-specific details which are not required to implement an "application" facing API.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Indeed this is the disconnect.
I already answered but it seems you don't accept the answer.Same with you. That is why I requested, we need to get opinions from others. Some of them already provided opinions in RFC.
This is why I Cc'ed techboard.
Yes. Indeed.
Quoted 36 lines
Quoted 4 lines
Quoted 2 lines
First, this is not driver-specific. It is a low-level API.
What is the difference between low-level API and driver-level API.
The low-level API provides tools to build a feature, but no specific feature.
Quoted 17 lines
Quoted 9 lines
Quoted 6 lines
For example, If we need to implement application facing" subsystems like bbdev, If we make all this driver interface, you can still implement the bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how memory is allocated etc
to "application".There are 2 things to understand here.
Quoted 7 lines
First we want to allow the application using the GPU for needs which are not exposed by any other DPDK API.
Second, if we want to implement another DPDK API like bbdev, then the GPU implementation would be exposed as a vdev in bbdev, using the HW GPU device being a PCI in gpudev.
They are two different levels, got it?Quoted 6 lines
Exactly. So what is the point of exposing low-level driver API to "application",
why not it is part of the internal driver API. My point is, why the application needs to worry
about, How the CPU <-> Device communicated? CPU < -> Device memory visibility etc.There are two reasons.
1/ The application may want to use the GPU for some application-specific needs which are not abstracted in DPDK API.
Quoted 7 lines
Quoted 4 lines
Yes. Exactly, That's where my concern, If we take this path, What is the motivation to contribute to DPDK abstracted subsystem APIs which make sense for multiple vendors and every
Similar stuff applicable for DPU,A feature-specific API is better of course, there is no lose of motivation. But you cannot forbid applications to have their own features on GPU.
it still can use it. We don't need DPDK APIs for that.
Quoted 8 lines
Quoted 4 lines
Otherway to put, if GPU is doing some ethdev offload, why not making as ethdev offload in ethdev spec so that
another type of device can be used and make sense for application writters.If we do ethdev offload, yes we'll implement it.
And we'll do it on top of gpudev, which is the only way to share the CPU.
Quoted 7 lines
Quoted 5 lines
For example, In the future, If someone needs to add ML(Machine learning) subsystem and enable a proper subsystem
interface that is good for DPDK. If this path is open, there is no motivation for contribution and the application
will not have a standard interface doing the ML job across multiple vendors.Wrong. It does remove the motivation, it is a first step to build on top of it.
IMO, No need to make driver API to the public to feature API.
Quoted 29 lines
Quoted 5 lines
That's is the only reason why saying it should not APPLICATION interface it can be DRIVER interface.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
2/ This API may also be used by some feature implementation internally in some DPDK libs or drivers.
We cannot skip the gpudev layer because this is what allows generic probing of the HW, and gpudev allows to share the GPU with multiple features implemented in different libs or drivers, thanks to the "child" concept.Again, why do applications need to know it? It is similar to
buskind of this where it sharing the physical resouces.No it's not a bus, it is a device that we need to share.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 4 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.See above.
We are going into circles.
Yes.
Quoted 1 line
In short, Jerin wants to forbid the generic use of GPU in DPDK.
See below.
Quoted 1 line
He wants only feature-specific API.
To re-reiterate, feature-specific "application" API. A device-specific bit can be
driver API and accessible to the out-of-tree driver if needed.
IMO, if we take this path, DPU, XPU, GPU, etc we need N different libraries to get the job done for a specific feature for the dataplane. Instead, Enabling public feature APIs will make the application portable and does not
need to worry about which type of *PU it runs.
Quoted 3 lines
It is like restricting the functions we can run on a CPU.
And anyway we need this layer to share the GPU between multiple features.
No disagreement there. Is that layer public application API or not is the question.
it is like PCI device API calls over of the application and makes the application device specific.
Quoted 2 lines
Techboard please vote.
Yes.
Quoted 2 lines
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
From: Jerin Jacob <[email protected]>
Date: Monday, 11 October 2021 at 15:30
To: NBU-Contact-Thomas Monjalon <[email protected]>
Cc: [email protected] <[email protected]>, Elena Agostini <[email protected]>, dpdk-dev <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3 0/9] GPU library
Quoted 149 lines · Thomas Monjalon
Quoted 86 lines
11/10/2021 13:41, Jerin Jacob:
Quoted 84 lines · Thomas Monjalon
Quoted 36 lines
11/10/2021 11:29, Jerin Jacob:
Quoted 35 lines · Thomas Monjalon
Quoted 34 lines
11/10/2021 10:43, Jerin Jacob:
Quoted 32 lines · Thomas Monjalon
Quoted 21 lines
10/10/2021 12:16, Jerin Jacob:
Quoted 20 lines · On Fri, Oct 8, 2021 at 11:13 PM <[email protected]> wrote:
Quoted 18 lines
From: eagostini <[email protected]>
In heterogeneous computing system, processing is not only in the CPU. Some tasks can be delegated to devices working in parallel.
The goal of this new library is to enhance the collaboration between DPDK, that's primarily a CPU framework, and GPU devices.
When mixing network activity with task processing on a non-CPU device, there may be the need to put in communication the CPU with the device in order to manage the memory, synchronize operations, exchange info, etc..
This library provides a number of new features:
- Interoperability with GPU-specific library with generic handlers
- Possibility to allocate and free memory on the GPU
- Possibility to allocate and free memory on the CPU but visible from the GPU
- Communication functions to enhance the dialog between the CPU and the GPUQuoted 7 lines
Quoted 5 lines
In the RFC thread, There was one outstanding non technical issues on this,
i.e
The above features are driver specific details. Does the DPDK _application_ need to be aware of this?I don't see these features as driver-specific.
That is the disconnect. I see this as more driver-specific details which are not required to implement an "application" facing API.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Indeed this is the disconnect.
I already answered but it seems you don't accept the answer.Same with you. That is why I requested, we need to get opinions from others. Some of them already provided opinions in RFC.
This is why I Cc'ed techboard.
Yes. Indeed.
Quoted 36 lines
Quoted 4 lines
Quoted 2 lines
First, this is not driver-specific. It is a low-level API.
What is the difference between low-level API and driver-level API.
The low-level API provides tools to build a feature, but no specific feature.
Quoted 17 lines
Quoted 9 lines
Quoted 6 lines
For example, If we need to implement application facing" subsystems like bbdev, If we make all this driver interface, you can still implement the bbdev API as a driver without
exposing HW specific details like how devices communicate to CPU, how memory is allocated etc
to "application".There are 2 things to understand here.
Quoted 7 lines
First we want to allow the application using the GPU for needs which are not exposed by any other DPDK API.
Second, if we want to implement another DPDK API like bbdev, then the GPU implementation would be exposed as a vdev in bbdev, using the HW GPU device being a PCI in gpudev.
They are two different levels, got it?Quoted 6 lines
Exactly. So what is the point of exposing low-level driver API to "application",
why not it is part of the internal driver API. My point is, why the application needs to worry
about, How the CPU <-> Device communicated? CPU < -> Device memory visibility etc.There are two reasons.
1/ The application may want to use the GPU for some application-specific needs which are not abstracted in DPDK API.
Quoted 7 lines
Quoted 4 lines
Yes. Exactly, That's where my concern, If we take this path, What is the motivation to contribute to DPDK abstracted subsystem APIs which make sense for multiple vendors and every
Similar stuff applicable for DPU,A feature-specific API is better of course, there is no lose of motivation. But you cannot forbid applications to have their own features on GPU.
it still can use it. We don't need DPDK APIs for that.
Quoted 8 lines
Quoted 4 lines
Otherway to put, if GPU is doing some ethdev offload, why not making as ethdev offload in ethdev spec so that
another type of device can be used and make sense for application writters.If we do ethdev offload, yes we'll implement it.
And we'll do it on top of gpudev, which is the only way to share the CPU.Quoted 7 lines
Quoted 5 lines
For example, In the future, If someone needs to add ML(Machine learning) subsystem and enable a proper subsystem
interface that is good for DPDK. If this path is open, there is no motivation for contribution and the application
will not have a standard interface doing the ML job across multiple vendors.Wrong. It does remove the motivation, it is a first step to build on top of it.
IMO, No need to make driver API to the public to feature API.
Quoted 34 lines
Quoted 5 lines
That's is the only reason why saying it should not APPLICATION interface it can be DRIVER interface.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
2/ This API may also be used by some feature implementation internally in some DPDK libs or drivers.
We cannot skip the gpudev layer because this is what allows generic probing of the HW, and gpudev allows to share the GPU with multiple features implemented in different libs or drivers, thanks to the "child" concept.Again, why do applications need to know it? It is similar to
buskind of this where it sharing the physical resouces.No it's not a bus, it is a device that we need to share.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
Quoted 4 lines
aka DPDK device class has a fixed personality and it has API to deal with abstracting specific application specific
end user functionality like ethdev, cryptodev, eventdev irrespective of underlying bus/device properties.The goal of the lib is to allow anyone to invent any feature which is not already available in DPDK.
Quoted 7 lines
Quoted 4 lines
Even similar semantics are required for DPU(Smart NIC) communitication. I am planning to
send RFC in coming days to address the issue without the application knowing the Bus/HW/Driver details.gpudev is not exposing bus/hw/driver details.
I don't understand what you mean.See above.
Quoted 7 lines
Quoted 1 line
We are going into circles.
Yes.
Quoted 1 line
In short, Jerin wants to forbid the generic use of GPU in DPDK.
See below.
Honestly I don’t see a real problem releasing the library at application level. It doesn’t prevent to use it internally by other DPDK libraries/drivers if needed.
Applications can benefit of this library for a number of reasons:
* Enhance the interaction between GPU specific library and DPDK * Hide GPU specific implementation details to the “final” user that wants to build a GPU + DPDK application * Measure network throughput with common tools like testpmd using GPU memory
Please be aware that this is just a starting point.
I’m planning to expose a number of features (at memory and processing levels) that can be useful to enhance the communication among GPU, CPU and NIC hiding the implementation details within the library/driver.
Quoted 7 lines
Quoted 2 lines
He wants only feature-specific API.
To re-reiterate, feature-specific "application" API. A device-specific bit can be
driver API and accessible to the out-of-tree driver if needed.
Quoted 6 lines
IMO, if we take this path, DPU, XPU, GPU, etc we need N different libraries to get the job done for a specific feature for the dataplane. Instead, Enabling public feature APIs will make the application portable and does not
need to worry about which type of *PU it runs.
As I stated multiple times, let’s start with something simple that works and then think about how to enhance the library/driver.
IMHO it doesn’t make sense to address all the use-cases now.
This is a completely new scenario we’re opening in the DPDK context, let’s start from the basis.
Quoted 17 lines
Quoted 4 lines
It is like restricting the functions we can run on a CPU.
And anyway we need this layer to share the GPU between multiple features.
No disagreement there. Is that layer public application API or not is the question.
it is like PCI device API calls over of the application and makes the application device specific.Quoted 2 lines
Techboard please vote.
Yes.
Quoted 2 lines
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
Quoted 29 lines
Quoted 17 lines
Quoted 4 lines
He wants only feature-specific API.
To re-reiterate, feature-specific "application" API. A device-specific
bit can be
driver API and accessible to the out-of-tree driver if needed.
IMO, if we take this path, DPU, XPU, GPU, etc we need N different libraries to
Quoted 9 lines
get the job done for a specific feature for the dataplane.
Instead, Enabling public feature APIs will make the application
portable and does not
need to worry about which type of *PU it runs.
Quoted 3 lines
As I stated multiple times, let’s start with something simple that works and
then think about how to enhance the library/driver.
I have submitted RFC[1] to abstract in a generic way so that workload-specific _driver_ aspects are NOT EXPOSED to the application.
The RFC explains with application code, how any workload can be used by the application without exposing the driver details. Also, shows how to enable a framework to abstract a different form of workload acceletor(DPU, GPU, XPU, IPU....)
In order to map to this discussion with RFC:
You may need to add a new host port for GPU which as lib/dwa/rte_dwa_port_host_xxxxx.h and adding new workload as lib/dwa/rte_dwa_profile_xxxx.h(Which can be reused by all dataplane workload accelerator)
[1]
http://mails.dpdk.org/archives/dev/2021-October/226070.html
Quoted 6 lines
IMHO it doesn’t make sense to address all the use-cases now.
This is a completely new scenario we’re opening in the DPDK context, let’s start
from the basis.
Re: [dpdk-dev] [PATCH v3 0/9] GPU library
19/10/2021 20:47, Jerin Jacob:
Quoted 11 lines
Quoted 2 lines
As I stated multiple times, let’s start with something simple that works and then think about how to enhance the library/driver.
I have submitted RFC[1] to abstract in a generic way so that workload-specific _driver_ aspects are NOT EXPOSED to the application.
The RFC explains with application code, how any workload can be used by the application without exposing the driver details. Also, shows how to enable a framework to abstract a different form of workload
acceletor(DPU, GPU, XPU, IPU....)
Quoted 7 lines
In order to map to this discussion with RFC:
You may need to add a new host port for GPU which aslib/dwa/rte_dwa_port_host_xxxxx.hand adding new workload as lib/dwa/rte_dwa_profile_xxxx.h(Which can be reused by all dataplane workload accelerator)[1] http://mails.dpdk.org/archives/dev/2021-October/226070.html
That's really 2 different points of view:
- expose only some predefined dataplane workloads
- or allow to build any custom worload