RCU integration with hash library
Subsystems (3)
cover RCU integration with hash library
This patchset integrates RCU QSBR support with hash library.
Note: This patchset has dependency on another patchset:
https://patchwork.dpdk.org/cover/57813/
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.
Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.
Add functional and performance test cases.
Fix 'ixgbe_ethdev.h' to avoid multiple definitions of 'bool'. (Please note that this fix is temporary. Recommend suggesting better solution)
Dharmik Thakkar (3):
net/ixgbe: avoid multpile definitions of 'bool' lib/hash: integrate RCU QSBR
test/hash: add tests for integrated RCU QSBR
app/test/test_hash_readwrite_lf.c | 700 ++++++++++++++++++++++++++- drivers/net/ixgbe/ixgbe_ethdev.h | 1 + lib/librte_hash/Makefile | 2 +- lib/librte_hash/meson.build | 2 + lib/librte_hash/rte_cuckoo_hash.c | 354 +++++++++++++- lib/librte_hash/rte_cuckoo_hash.h | 3 + lib/librte_hash/rte_hash.h | 38 +- lib/librte_hash/rte_hash_version.map | 2 +- 8 files changed, 1065 insertions(+), 37 deletions(-)
Signature · 1 line
2.17.1
RFC 1/3 net/ixgbe: avoid multpile definitions of 'bool'
Compilation issue arises due to multiple definitions of 'bool' in 'ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This is a temporary fix. Recommend suggesting better solution.)
--- drivers/net/ixgbe/ixgbe_ethdev.h | 1 + 1 file changed, 1 insertion(+)
drivers/net/ixgbe/ixgbe_ethdev.h +1 -0
@@ -18,6 +18,7 @@#include <rte_flow.h> #include <rte_time.h> #include <rte_hash.h> +#undef bool#include <rte_pci.h> #include <rte_bus_pci.h> #include <rte_tm_driver.h>
Signature · 1 line
2.17.1
Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'
Quoted 16 lines · On Sun, 1 Sep 2019 06:58:08 +0000 Dharmik Thakkar <[email protected]>…
Compilation issue arises due to multiple definitions of 'bool' in '
ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This is a temporary fix. Recommend suggesting better solution.)Signed-off-by Dharmik Thakkar <[email protected]>--- drivers/net/ixgbe/ixgbe_ethdev.h | 1 + 1 file changed, 1 insertion(+)drivers/net/ixgbe/ixgbe_ethdev.h +0
-0
drivers/net/ixgbe/ixgbe_ethdev.h line 24
Quoted 7 lines
drivers/net/ixgbe/ixgbe_ethdev.h +1
-0@@ -18,6 +18,7 @@#include <rte_flow.h> #include <rte_time.h> #include <rte_hash.h> +#undef bool#include <rte_pci.h> #include <rte_bus_pci.h> #include <rte_tm_driver.h>
Just fix ixgbe_ethdev.h to include stdbool itself?
drivers/net/ixgbe/ixgbe_ethdev.h +1 -0
@@ -6,6 +6,7 @@#define _IXGBE_ETHDEV_H_#include <stdint.h> +#include <stdbool.h> #include "base/ixgbe_type.h" #include "base/ixgbe_dcb.h"
Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'
Quoted 18 lines · Stephen Hemminger
Quoted 11 lines · On Sun, 1 Sep 2019 06:58:08 +0000 Dharmik Thakkar <[email protected]>…
Compilation issue arises due to multiple definitions of 'bool' in '
ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This is a temporary fix. Recommend suggesting better solution.)Signed-off-by Dharmik Thakkar <[email protected]>---
drivers/net/ixgbe/ixgbe_ethdev.h| 1 +1 file changed, 1 insertion(+)
Quoted 14 lines
drivers/net/ixgbe/ixgbe_ethdev.h line 24
Quoted 12 lines
drivers/net/ixgbe/ixgbe_ethdev.h +1
-0@@ -18,6 +18,7 @@#include <rte_flow.h> #include <rte_time.h> #include <rte_hash.h> +#undef bool#include <rte_pci.h> #include <rte_bus_pci.h> #include <rte_tm_driver.h> Just fix
ixgbe_ethdev.hto include stdbool itself?
Conflicts with 'typedef int bool;’ in ixgbe_osdep.h:86
Quoted 13 lines
drivers/net/ixgbe/ixgbe_ethdev.h +1
-0@@ -6,6 +6,7 @@#define _IXGBE_ETHDEV_H_#include <stdint.h> +#include <stdbool.h> #include "base/ixgbe_type.h" #include "base/ixgbe_dcb.h"
Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'
Quoted 30 lines · On Thu, 5 Sep 2019 20:10:05 +0000 Dharmik Thakkar <[email protected]>…
Quoted 16 lines · Stephen Hemminger
Quoted 11 lines · On Sun, 1 Sep 2019 06:58:08 +0000 Dharmik Thakkar <[email protected]>…
Compilation issue arises due to multiple definitions of 'bool' in '
ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This is a temporary fix. Recommend suggesting better solution.)Signed-off-by Dharmik Thakkar <[email protected]>---
drivers/net/ixgbe/ixgbe_ethdev.h| 1 +1 file changed, 1 insertion(+)Quoted 14 lines
drivers/net/ixgbe/ixgbe_ethdev.h line 24
Quoted 12 lines
drivers/net/ixgbe/ixgbe_ethdev.h +1
-0@@ -18,6 +18,7 @@#include <rte_flow.h> #include <rte_time.h> #include <rte_hash.h> +#undef bool#include <rte_pci.h> #include <rte_bus_pci.h> #include <rte_tm_driver.h> Just fix
ixgbe_ethdev.hto include stdbool itself?
Quoted 1 line
Conflicts with 'typedef int bool;’ in ixgbe_osdep.h:86
Then fix the ixgbe_osdep.h. It is the problem.
RFC 2/3 lib/hash: integrate RCU QSBR
In reply to Dharmik ThakkarIntegrate RCU QSBR process.
(Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.)
--- lib/librte_hash/Makefile | 2 +- lib/librte_hash/meson.build | 2 + lib/librte_hash/rte_cuckoo_hash.c | 352 +++++++++++++++++++++++++-- lib/librte_hash/rte_cuckoo_hash.h | 3 + lib/librte_hash/rte_hash.h | 38 ++- lib/librte_hash/rte_hash_version.map | 2 +- 6 files changed, 363 insertions(+), 36 deletions(-)
lib/librte_hash/Makefile +1 -1
@@ -8,7 +8,7 @@ LIB = librte_hash.a CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -LDLIBS += -lrte_eal -lrte_ring +LDLIBS += -lrte_eal -lrte_ring -lrte_rcu EXPORT_MAP := rte_hash_version.map
lib/librte_hash/meson.build +2 -0
@@ -2,6 +2,7 @@ # Copyright(c) 2017 Intel Corporation version = 2 +allow_experimental_apis = true headers = files('rte_cmp_arm64.h', 'rte_cmp_x86.h', 'rte_crc_arm64.h',
@@ -14,6 +15,7 @@ headers = files('rte_cmp_arm64.h', sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c') deps += ['ring'] +deps += ['rcu'] # rte ring reset is not yet part of stable API allow_experimental_apis = true
lib/librte_hash/rte_cuckoo_hash.c +326 -26
469 lines not on this page · raw message
lib/librte_hash/rte_cuckoo_hash.h +3 -0
@@ -168,6 +168,9 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +struct rte_rcu_qsbr *qsv; +struct rte_ring *qs_fifo; +rte_hash_free_key_data free_key_data_func;/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;
lib/librte_hash/rte_hash.h +30 -8
@@ -15,6 +15,7 @@#include <stddef.h> #include <rte_compat.h> +#include <rte_rcu_qsbr.h> #ifdef __cplusplusextern "C" {
@@ -44,8 +45,6 @@ extern "C" {/** Flag to disable freeing of key index on hash delete. * Refer to rte_hash_del_xxx APIs for more details. -* This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF -* is enabled. */ #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
@@ -69,6 +68,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,/** Type of function used to compare the hash key. */ typedef int (*rte_hash_cmp_eq_t )(const void *key1,const void *key2,size_t key_len); +/** +* Type of function used to free data stored in the key. +* Required when using internal RCU to allow appication to free key-data once +* the key is returned to the the ring of free key-slots. +*/ +typedef void (*rte_hash_free_key_data )(void *key_data); +/** * Parameters used when creating the hash table. */
@@ -267,8 +273,7 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t* and should only be called from one thread by default. * Thread safety can be enabled by setting flag during * table creation. -* If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or -* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, +* If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called * additionally to free the index associated with the key.
@@ -296,8 +301,7 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);* and should only be called from one thread by default. * Thread safety can be enabled by setting flag during * table creation. -* If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or -* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, +* If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called * additionally to free the index associated with the key.
@@ -350,8 +354,7 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,* of the key. This operation is not multi-thread safe and should * only be called from one thread by default. Thread safety * can be enabled by setting flag during table creation. -* If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or -* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, +* If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled, * the key index returned by rte_hash_del_key_xxx APIs must be freed * using this API. This API should be called after all the readers * have stopped referencing the entry corresponding to this key.
@@ -545,6 +548,25 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,*/ int32_t rte_hash_iterate (const struct rte_hash *h,const void **key,void **data,uint32_t *next); + +/** +* Configure the RCU QSBR variable to use in hash structure. +* +* @param h +* Hash table +* @param v +* RCU QSBR variable +* @return +* On success - 0 +* On error - 1 with error code set in rte_errno. +* Possible rte_errno codes are: +* - EINVAL if invalid pointer +* - EEXIST if already added QSBR +* - ENOMEM if memory allocation failure +*/ +__rte_experimental +int rte_hash_rcu_qsbr_add(struct rte_hash *h, struct rte_rcu_qsbr *v, +rte_hash_free_key_data free_key_data_func); #ifdef __cplusplus }#endif
lib/librte_hash/rte_hash_version.map +1 -1
@@ -58,5 +58,5 @@ EXPERIMENTAL { global: rte_hash_free_key_with_position; - + rte_hash_rcu_qsbr_add; };
Signature · 1 line
2.17.1
RFC 3/3 test/hash: add tests for integrated RCU QSBR
In reply to Dharmik ThakkarAdd functional and performance tests for the integrated RCU QSBR.
--- app/test/test_hash_readwrite_lf.c | 694 +++++++++++++++++++++++++++++- 1 file changed, 693 insertions(+), 1 deletion(-)
app/test/test_hash_readwrite_lf.c +693 -1
@@ -13,6 +13,7 @@#include <rte_malloc.h> #include <rte_random.h> #include <rte_spinlock.h> +#include <rte_rcu_qsbr.h> #include "test.h"
@@ -58,6 +59,7 @@ struct rwc_perf {uint32_t w_ks_r_miss[2 ][NUM_TEST ];uint32_t multi_rw[NUM_TEST -1 ][2 ][NUM_TEST ];uint32_t w_ks_r_hit_extbkt[2 ][NUM_TEST ]; +uint32_t writer_add_del[NUM_TEST ]; };static struct rwc_perf rwc_lf_results, rwc_non_lf_results;
@@ -84,12 +86,15 @@ struct {static rte_atomic64_t gread_cycles;static rte_atomic64_t greads; +static rte_atomic64_t gwrite_cycles; +static rte_atomic64_t gwrites;static volatile uint8_t writer_done;uint16_t enabled_core_ids[RTE_MAX_LCORE ];uint8_t *scanned_bkts; +uint8_t no_free_on_del;static inline uint16_t get_short_sig (const hash_sig_t hash)
@@ -173,6 +178,9 @@ init_params(int rwc_lf, int use_jhash, int htm, int ext_bkt)if (ext_bkt) hash_params.extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; +if (no_free_on_del) + hash_params.extra_flag |=RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL ; + hash_params.name ="tests" ; handle =rte_hash_create (&hash_params);
@@ -1240,6 +1248,666 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,return -1 ; } +static uint32_t *hash_data[TOTAL_ENTRY ]; +static struct rte_rcu_qsbr *t; +#define COUNTER_VALUE 4096 +#define TEST_RCU_MAX_LCORE 128 +static int +test_rwc_rcu_reader (__attribute__((unused))void *arg) +{ +struct rte_rcu_qsbr *temp; +unsigned int i; +uint32_t lcore_id =rte_lcore_id (); +uint32_t *pdata; + temp = t; + +rte_rcu_qsbr_thread_register (temp, lcore_id); +do { +rte_rcu_qsbr_thread_online (temp, lcore_id); +for (i =0 ; i < tbl_rwc_test_param.count_keys_no_ks; i++) { +rte_rcu_qsbr_lock (temp, lcore_id); +if (rte_hash_lookup_data (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i, + (void **)&pdata) != -ENOENT ) { +if (pdata !=NULL ) { + pdata[lcore_id] =0 ; +while (pdata[lcore_id] <COUNTER_VALUE ) + pdata[lcore_id]++; + } + } +rte_rcu_qsbr_unlock (temp, lcore_id); + } +/* Update quiescent state counter */ +rte_rcu_qsbr_quiescent (temp, lcore_id); +rte_rcu_qsbr_thread_offline (temp, lcore_id); + }while (!writer_done); +rte_rcu_qsbr_thread_unregister (temp, lcore_id); + +return 0 ; +} + +static int +test_rwc_rcu_multi_writer (void *arg) +{ +int32_t pos; + +uint32_t i, offset; +uint8_t pos_core = (uint32_t )((uintptr_t )arg); + offset = pos_core * tbl_rwc_test_param.single_insert; + +for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) { +/* Delete element from the shared data structure */ + pos =rte_hash_del_key (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i); +if (no_free_on_del) { +if (rte_hash_free_key_with_position (tbl_rwc_test_param.h, + pos) <0 ) +return -1 ; + } +if (pos <0 ) { +printf ("Delete key failed #%u at #%d\n" , + tbl_rwc_test_param.keys_no_ks[i], i); +return -1 ; + } +rte_hash_add_key (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i); + } +return 0 ; +} + +uint8_t error_state; +static void +free_hash_data (void *pdata) +{ +uint32_t i; +uint32_t *h_data = pdata; +if (h_data ==NULL ) +return ; +for (i =0 ; i < rwc_core_cnt[NUM_TEST -1 ]; i++) { +if (h_data[i] !=COUNTER_VALUE && h_data[i] !=0 ) { +printf ("Reader did not complete\n" ); +__atomic_store_n (&error_state,1 , __ATOMIC_RELAXED); +return ; + } + } +rte_free (pdata); +return ; +} + +/* +* Test integrated RCU with 'free' on hash_delete +* and with extended bkt feature disabled +*/ +static int +test_hash_rcu_free_on_del (__attribute__((unused))struct rwc_perf + *rwc_perf_results,int rwc_lf,int htm, +int ext_bkt) +{ +unsigned int n, m; +uint64_t i; +uint8_t write_type; +int use_jhash =0 ; + no_free_on_del =0 ; + +if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) + goto err; + +uint32_t sz; + sz =rte_rcu_qsbr_get_memsize (TEST_RCU_MAX_LCORE ); + t = (struct rte_rcu_qsbr *)rte_zmalloc (NULL , sz,RTE_CACHE_LINE_SIZE ); +if (rte_hash_rcu_qsbr_add (tbl_rwc_test_param.h, t, + free_hash_data) <0 ) { +printf ("RCU init in hash failed\n" ); +return -1 ; + } +printf ("\nTest: RCU - free on hash del - ext bkt disabled\n" ); +uint8_t pos_core; + +for (m =1 ; m <NUM_TEST ; m++) { +/* Calculate keys added by each writer */ + tbl_rwc_test_param.single_insert = + tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m]; +for (n =0 ; n <NUM_TEST ; n++) { +unsigned int tot_lcore =rte_lcore_count (); +if (tot_lcore < (rwc_core_cnt[n] + + rwc_core_cnt[m] +1 )) + goto finish; + +printf ("\nNumber of writers: %u" , rwc_core_cnt[m]); +printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); + +rte_hash_reset (tbl_rwc_test_param.h); + writer_done =0 ; + write_type =WRITE_NO_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; + write_type =WRITE_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; +for (i =0 ; i < tbl_rwc_test_param.count_keys_no_ks; + i++) { + hash_data[i] = +rte_zmalloc (NULL ,sizeof (uint32_t ) + * rwc_core_cnt[NUM_TEST -1 ], +0 ); +if (hash_data[i] ==NULL ) { +printf ("No memory\n" ); +return -1 ; + } + } +for (i =0 ; + i < tbl_rwc_test_param.count_keys_no_ks; + i++) { +if (rte_hash_add_key_data (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i, + (void *)((uintptr_t )hash_data[i])) + <0 ) { +printf ("Hash key add Failed #%d\n" , + tbl_rwc_test_param.keys_no_ks[i]); +return -1 ; + } + } + +rte_rcu_qsbr_init (t,TEST_RCU_MAX_LCORE ); +/* Launch reader(s) */ +for (i =1 ; i <= rwc_core_cnt[n]; i++) +rte_eal_remote_launch (test_rwc_rcu_reader, + (void *)(uintptr_t )rwc_core_cnt[m], + enabled_core_ids[i]); + pos_core =0 ; +/* Launch writers */ +for (; i <= rwc_core_cnt[m] + + rwc_core_cnt[n]; i++) { +rte_eal_remote_launch + (test_rwc_rcu_multi_writer, + (void *)(uintptr_t )pos_core, + enabled_core_ids[i]); + pos_core++; + } + +/* Wait for writers to complete */ +for (i = rwc_core_cnt[n] +1 ; + i <= rwc_core_cnt[m] + rwc_core_cnt[n]; + i++) +rte_eal_wait_lcore (enabled_core_ids[i]); + + writer_done =1 ; + +for (i =1 ; i <= rwc_core_cnt[n]; i++) +if (rte_eal_wait_lcore (enabled_core_ids[i]) <0 ) + goto err; + + } + } + +finish: +rte_hash_free (tbl_rwc_test_param.h); +rte_free (t); +return 0 ; + +err: +rte_hash_free (tbl_rwc_test_param.h); +rte_free (t); +return -1 ; +} + +/* +* Test integrated RCU with NO 'free' on hash_delete +* and with extended bkt feature disabled +*/ +static int +test_hash_rcu_no_free_on_del (__attribute__((unused))struct rwc_perf + *rwc_perf_results,int rwc_lf,int htm, +int ext_bkt) +{ +unsigned int n, m; +uint64_t i; +uint8_t write_type; +int use_jhash =0 ; + no_free_on_del =1 ; + +if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) + goto err; + +uint32_t sz; + sz =rte_rcu_qsbr_get_memsize (TEST_RCU_MAX_LCORE ); + t = (struct rte_rcu_qsbr *)rte_zmalloc (NULL , sz,RTE_CACHE_LINE_SIZE ); +if (rte_hash_rcu_qsbr_add (tbl_rwc_test_param.h, t, + free_hash_data) <0 ) { +printf ("RCU init in hash failed\n" ); +return -1 ; + } + +printf ("\nTest: RCU - no free on hash del - ext bkt disabled\n" ); +uint8_t pos_core; + +for (m =1 ; m <NUM_TEST ; m++) { +/* Calculate keys added by each writer */ + tbl_rwc_test_param.single_insert = + tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m]; +for (n =0 ; n <NUM_TEST ; n++) { +unsigned int tot_lcore =rte_lcore_count (); +if (tot_lcore < (rwc_core_cnt[n] + + rwc_core_cnt[m] +1 )) + goto finish; + +printf ("\nNumber of writers: %u" , rwc_core_cnt[m]); +printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); + +rte_hash_reset (tbl_rwc_test_param.h); + writer_done =0 ; + write_type =WRITE_NO_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; + write_type =WRITE_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; +for (i =0 ; i < tbl_rwc_test_param.count_keys_no_ks; + i++) { + hash_data[i] = +rte_zmalloc (NULL ,sizeof (uint32_t ) + * rwc_core_cnt[NUM_TEST -1 ], +0 ); +if (hash_data[i] ==NULL ) { +printf ("No memory\n" ); +return -1 ; + } + } +for (i =0 ; + i < tbl_rwc_test_param.count_keys_no_ks; + i++) { +if (rte_hash_add_key_data (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i, + (void *)((uintptr_t )hash_data[i])) + <0 ) { +printf ("Hash key add Failed #%d\n" , + tbl_rwc_test_param.keys_no_ks[i]); +return -1 ; + } + } + +rte_rcu_qsbr_init (t,TEST_RCU_MAX_LCORE ); +/* Launch reader(s) */ +for (i =1 ; i <= rwc_core_cnt[n]; i++) +rte_eal_remote_launch (test_rwc_rcu_reader, + (void *)(uintptr_t )rwc_core_cnt[m], + enabled_core_ids[i]); + pos_core =0 ; +/* Launch writers */ +for (; i <= rwc_core_cnt[m] + + rwc_core_cnt[n]; i++) { +rte_eal_remote_launch + (test_rwc_rcu_multi_writer, + (void *)(uintptr_t )pos_core, + enabled_core_ids[i]); + pos_core++; + } +/* Wait for writers to complete */ +for (i = rwc_core_cnt[n] +1 ; + i <= rwc_core_cnt[m] + rwc_core_cnt[n]; + i++) +rte_eal_wait_lcore (enabled_core_ids[i]); + + writer_done =1 ; + +for (i =1 ; i <= rwc_core_cnt[n]; i++) +if (rte_eal_wait_lcore (enabled_core_ids[i]) <0 ) + goto err; + } + } + +finish: +rte_hash_free (tbl_rwc_test_param.h); +rte_free (t); +return 0 ; + +err: +rte_hash_free (tbl_rwc_test_param.h); +rte_free (t); +return -1 ; +} + +/* +* Test integrated RCU with 'free' on hash_delete +* and with extended bkt feature enabled +*/ +static int +test_hash_rcu_ext_bkt_free_on_del (__attribute__((unused))struct rwc_perf + *rwc_perf_results,int rwc_lf,int htm, +int ext_bkt) +{ +unsigned int n, m; +uint64_t i; +uint8_t write_type; +int use_jhash =0 ; + no_free_on_del =0 ; + +if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) + goto err; + +uint32_t sz; + sz =rte_rcu_qsbr_get_memsize (TEST_RCU_MAX_LCORE ); + t = (struct rte_rcu_qsbr *)rte_zmalloc (NULL , sz,RTE_CACHE_LINE_SIZE ); +if (rte_hash_rcu_qsbr_add (tbl_rwc_test_param.h, t, + free_hash_data) <0 ) { +printf ("RCU init in hash failed\n" ); +return -1 ; + } + +printf ("\nTest: RCU - free on hash del - ext bkt enabled\n" ); +uint8_t pos_core; + +for (m =1 ; m <NUM_TEST ; m++) { +/* Calculate keys added by each writer */ + tbl_rwc_test_param.single_insert = + tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m]; +for (n =0 ; n <NUM_TEST ; n++) { +unsigned int tot_lcore =rte_lcore_count (); +if (tot_lcore < (rwc_core_cnt[n] + + rwc_core_cnt[m] +1 )) + goto finish; + +printf ("\nNumber of writers: %u" , rwc_core_cnt[m]); +printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); + +rte_hash_reset (tbl_rwc_test_param.h); + writer_done =0 ; + write_type =WRITE_NO_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; + write_type =WRITE_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; +for (i =0 ; i < tbl_rwc_test_param.count_keys_no_ks; + i++) { + hash_data[i] = +rte_zmalloc (NULL ,sizeof (uint32_t ) + * rwc_core_cnt[NUM_TEST -1 ], +0 ); +if (hash_data[i] ==NULL ) { +printf ("No memory\n" ); +return -1 ; + } + } +for (i =0 ; + i < tbl_rwc_test_param.count_keys_no_ks; + i++) { +if (rte_hash_add_key_data (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i, + (void *)((uintptr_t )hash_data[i])) + <0 ) { +printf ("Hash key add Failed #%d\n" , + tbl_rwc_test_param.keys_no_ks[i]); +return -1 ; + } + } + +rte_rcu_qsbr_init (t,TEST_RCU_MAX_LCORE ); +/* Launch reader(s) */ +for (i =1 ; i <= rwc_core_cnt[n]; i++) +rte_eal_remote_launch (test_rwc_rcu_reader, + (void *)(uintptr_t )rwc_core_cnt[m], + enabled_core_ids[i]); + pos_core =0 ; +/* Launch writers */ +for (; i <= rwc_core_cnt[m] + + rwc_core_cnt[n]; i++) { +rte_eal_remote_launch + (test_rwc_rcu_multi_writer, + (void *)(uintptr_t )pos_core, + enabled_core_ids[i]); + pos_core++; + } +/* Wait for writers to complete */ +for (i = rwc_core_cnt[n] +1 ; + i <= rwc_core_cnt[m] + rwc_core_cnt[n]; + i++) +rte_eal_wait_lcore (enabled_core_ids[i]); + + writer_done =1 ; + +for (i =1 ; i <= rwc_core_cnt[n]; i++) +if (rte_eal_wait_lcore (enabled_core_ids[i]) <0 ) + goto err; + } + } + +finish: +rte_hash_free (tbl_rwc_test_param.h); +rte_free (t); +return 0 ; + +err: +rte_hash_free (tbl_rwc_test_param.h); +rte_free (t); +return -1 ; +} + +/* +* Test integrated RCU with NO 'free' on hash_delete +* and with extended bkt feature enabled +*/ +static int +test_hash_rcu_ext_bkt_no_free_on_del (__attribute__((unused))struct rwc_perf + *rwc_perf_results,int rwc_lf,int htm, +int ext_bkt) +{ +unsigned int n, m; +uint64_t i; +uint8_t write_type; +int use_jhash =0 ; + no_free_on_del =1 ; + +if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) + goto err; + +uint32_t sz; + sz =rte_rcu_qsbr_get_memsize (TEST_RCU_MAX_LCORE ); + t = (struct rte_rcu_qsbr *)rte_zmalloc (NULL , sz,RTE_CACHE_LINE_SIZE ); +if (rte_hash_rcu_qsbr_add (tbl_rwc_test_param.h, t, + free_hash_data) <0 ) { +printf ("RCU init in hash failed\n" ); +return -1 ; + } + +printf ("\nTest: RCU - no free on hash del - ext bkt enabled\n" ); +uint8_t pos_core; + +for (m =1 ; m <NUM_TEST ; m++) { +/* Calculate keys added by each writer */ + tbl_rwc_test_param.single_insert = + tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m]; +for (n =0 ; n <NUM_TEST ; n++) { +unsigned int tot_lcore =rte_lcore_count (); +if (tot_lcore < (rwc_core_cnt[n] + + rwc_core_cnt[m] +1 )) + goto finish; + +printf ("\nNumber of writers: %u" , rwc_core_cnt[m]); +printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); + +rte_hash_reset (tbl_rwc_test_param.h); + writer_done =0 ; + write_type =WRITE_NO_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; + write_type =WRITE_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; +for (i =0 ; i < tbl_rwc_test_param.count_keys_no_ks; + i++) { + hash_data[i] = +rte_zmalloc (NULL ,sizeof (uint32_t ) + * rwc_core_cnt[NUM_TEST -1 ], +0 ); +if (hash_data[i] ==NULL ) { +printf ("No memory\n" ); +return -1 ; + } + } +for (i =0 ; + i < tbl_rwc_test_param.count_keys_no_ks; + i++) { +if (rte_hash_add_key_data (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i, + (void *)((uintptr_t )hash_data[i])) + <0 ) { +printf ("Hash key add Failed #%d\n" , + tbl_rwc_test_param.keys_no_ks[i]); +return -1 ; + } + } + +rte_rcu_qsbr_init (t,TEST_RCU_MAX_LCORE ); +/* Launch reader(s) */ +for (i =1 ; i <= rwc_core_cnt[n]; i++) +rte_eal_remote_launch (test_rwc_rcu_reader, + (void *)(uintptr_t )rwc_core_cnt[m], + enabled_core_ids[i]); + pos_core =0 ; +/* Launch writers */ +for (; i <= rwc_core_cnt[m] + + rwc_core_cnt[n]; i++) { +rte_eal_remote_launch + (test_rwc_rcu_multi_writer, + (void *)(uintptr_t )pos_core, + enabled_core_ids[i]); + pos_core++; + } +/* Wait for writers to complete */ +for (i = rwc_core_cnt[n] +1 ; + i <= rwc_core_cnt[m] + rwc_core_cnt[n]; + i++) +rte_eal_wait_lcore (enabled_core_ids[i]); + + writer_done =1 ; + +for (i =1 ; i <= rwc_core_cnt[n]; i++) +if (rte_eal_wait_lcore (enabled_core_ids[i]) <0 ) + goto err; + } + } + +finish: +rte_hash_free (tbl_rwc_test_param.h); +rte_free (t); +return 0 ; + +err: +rte_hash_free (tbl_rwc_test_param.h); +rte_free (t); +return -1 ; +} + +static int +test_multiwriter_perf (void *arg) +{ +uint32_t i, offset; +uint64_t begin, cycles; +uint8_t pos_core = (uint32_t )((uintptr_t )arg); + offset = pos_core * tbl_rwc_test_param.single_insert; + + begin =rte_rdtsc_precise (); +for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) { +/* Delete element from the shared data structure */ +rte_hash_del_key (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i); +rte_hash_add_key (tbl_rwc_test_param.h, + tbl_rwc_test_param.keys_no_ks + i); + } + cycles =rte_rdtsc_precise () - begin; +rte_atomic64_add (&gwrite_cycles, cycles); +rte_atomic64_add (&gwrites, tbl_rwc_test_param.single_insert); +return 0 ; +} + +/* +* Test RCU perf: +* Writer(s) delete and add keys in the table. +*/ +static int +test_hash_writer_perf (struct rwc_perf *rwc_perf_results,int rwc_lf, +int htm,int ext_bkt) +{ +unsigned int n; +uint64_t i; +int use_jhash =0 ; +uint8_t write_type; + + no_free_on_del =0 ; + +rte_atomic64_init (&gwrites); +rte_atomic64_init (&gwrite_cycles); + +if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) + goto err; + +if (rwc_lf) { +uint32_t sz; + sz =rte_rcu_qsbr_get_memsize (TEST_RCU_MAX_LCORE ); + t = (struct rte_rcu_qsbr *)rte_zmalloc (NULL , sz,RTE_CACHE_LINE_SIZE ); +if (rte_hash_rcu_qsbr_add (tbl_rwc_test_param.h, t, + free_hash_data) <0 ) { +printf ("RCU init in hash failed\n" ); +return -1 ; + } +printf ("\nTest: Writer perf with integrated RCU\n" ); + }else +printf ("\nTest: Writer perf without integrated RCU\n" ); + +for (n =0 ; n <NUM_TEST ; n++) { +unsigned int tot_lcore =rte_lcore_count (); +if (tot_lcore < rwc_core_cnt[n] +1 ) + goto finish; + +/* Calculate keys added by each writer */ + tbl_rwc_test_param.single_insert = + tbl_rwc_test_param.count_keys_no_ks / + rwc_core_cnt[n]; +printf ("\nNumber of writers: %u\n" , rwc_core_cnt[n]); + +rte_atomic64_clear (&gwrites); +rte_atomic64_clear (&gwrite_cycles); + +rte_hash_reset (tbl_rwc_test_param.h); +if (rwc_lf) +rte_rcu_qsbr_init (t,TEST_RCU_MAX_LCORE ); + write_type =WRITE_NO_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; + write_type =WRITE_KEY_SHIFT ; +if (write_keys (write_type) <0 ) + goto err; +for (i =1 ; i <= rwc_core_cnt[n]; i++) +rte_eal_remote_launch (test_multiwriter_perf, + (void *)(uintptr_t )(i -1 ), + enabled_core_ids[i]); + +rte_eal_mp_wait_lcore (); + +unsigned long long cycles_per_write_operation = +rte_atomic64_read (&gwrite_cycles) / +rte_atomic64_read (&gwrites); + rwc_perf_results->writer_add_del[n] + = cycles_per_write_operation; +printf ("Cycles per write operation: %llu\n" , + cycles_per_write_operation); + } + +finish: +rte_hash_free (tbl_rwc_test_param.h); +if (rwc_lf) +rte_free (t); +return 0 ; + +err: +rte_eal_mp_wait_lcore (); +rte_hash_free (tbl_rwc_test_param.h); +if (rwc_lf) +rte_free (t); +return -1 ; +} +static int test_hash_readwrite_lf_main (void ) {
@@ -1272,7 +1940,6 @@ test_hash_readwrite_lf_main(void)return -1 ;if (get_enabled_cores_list () !=0 )return -1 ; -if (RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ) { rwc_lf =1 ; ext_bkt =1 ;
@@ -1299,6 +1966,23 @@ test_hash_readwrite_lf_main(void)if (test_hash_add_ks_lookup_hit_extbkt (&rwc_lf_results, rwc_lf, htm, ext_bkt) <0 )return -1 ; + ext_bkt =0 ; +if (test_hash_rcu_free_on_del (&rwc_lf_results, rwc_lf, htm, + ext_bkt) <0 ) +return -1 ; +if (test_hash_rcu_no_free_on_del (&rwc_lf_results, rwc_lf, htm, + ext_bkt) <0 ) +return -1 ; + ext_bkt =1 ; +if (test_hash_rcu_ext_bkt_free_on_del (&rwc_lf_results, rwc_lf, + htm, ext_bkt) <0 ) +return -1 ; +if (test_hash_rcu_ext_bkt_no_free_on_del (&rwc_lf_results, rwc_lf + , htm, ext_bkt) <0 ) +return -1 ; +if (test_hash_writer_perf (&rwc_lf_results, rwc_lf + , htm, ext_bkt) <0 ) +return -1 ; }printf ("\nTest lookup with read-write concurrency lock free support" " disabled\n" );
@@ -1330,9 +2014,17 @@ test_hash_readwrite_lf_main(void)if (test_hash_multi_add_lookup (&rwc_non_lf_results, rwc_lf, htm, ext_bkt) <0 )return -1 ; + no_free_on_del =0 ;if (test_hash_add_ks_lookup_hit_extbkt (&rwc_non_lf_results, rwc_lf, htm, ext_bkt) <0 )return -1 ; +if (test_hash_writer_perf (&rwc_non_lf_results, rwc_lf, + htm, ext_bkt) <0 ) +return -1 ; + +if (error_state !=0 ) +return -1 ; + results:printf ("\n\t\t\t\t\t\t********** Results summary **********\n\n" );int i, j, k;
Signature · 1 line
2.17.1
Quoted 38 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Dharmik Thakkar
Sent: Saturday, August 31, 2019 11:58 PM
Cc: [email protected]; [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [dpdk-dev] [RFC 0/3] RCU integration with hash libraryThis patchset integrates RCU QSBR support with hash library.
Note: This patchset has dependency on another patchset:
https://patchwork.dpdk.org/cover/57813/Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Add functional and performance test cases.
Fix '
ixgbe_ethdev.h' to avoid multiple definitions of 'bool'. (Please note that this fix is temporary. Recommend suggesting better solution)Dharmik Thakkar (3):
net/ixgbe: avoid multpile definitions of 'bool'
lib/hash: integrate RCU QSBR
test/hash: add tests for integrated RCU QSBRapp/test/test_hash_readwrite_lf.c | 700 ++++++++++++++++++++++++++- drivers/net/ixgbe/ixgbe_ethdev.h | 1 + lib/librte_hash/Makefile | 2 +- lib/librte_hash/meson.build | 2 + lib/librte_hash/rte_cuckoo_hash.c | 354 +++++++++++++- lib/librte_hash/rte_cuckoo_hash.h | 3 + lib/librte_hash/rte_hash.h | 38 +- lib/librte_hash/rte_hash_version.map | 2 +- 8 files changed, 1065 insertions(+), 37 deletions(-)
Quoted 2 lines
--
2.17.1
[Wang, Yipeng]
Hi Dharmik, thanks for the code!
I agree with Stephen that many people may not use RCU,
and when they use, they may want some control in the application level. I can see the benefit to have certain RCU helper functions inside the libraries when people indeed uses QSBR as you designed,
but I am not sure if the benefits justify the effort to put it in many DPDK libraries. It adds new APIs, requires future maintenance, and so on, given the feature can be totally realized by the user application with small cost I assume.
I believe your plan is to add similar helper functions to many other libraries. I think we could consider them as a single proposal, and it makes sense to bring it up to the technical committee for a broader discussion.
RFC v2 1/1 lib/hash: integrate RCU QSBR
In reply to Dharmik ThakkarIntegrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.
---
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead
- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted
--- lib/Makefile | 2 +- lib/librte_hash/Makefile | 2 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 291 +++++++++++++++++++++------ lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 75 ++++++- lib/librte_hash/rte_hash_version.map | 2 +- 7 files changed, 308 insertions(+), 73 deletions(-)
lib/Makefile +1 -1
@@ -51,7 +51,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += librte_vhost DEPDIRS-librte_vhost := librte_eal librte_mempool librte_mbuf librte_ethdev \ librte_net librte_hash librte_cryptodev DIRS-$(CONFIG_RTE_LIBRTE_HASH) += librte_hash -DEPDIRS-librte_hash := librte_eal librte_ring +DEPDIRS-librte_hash := librte_eal librte_ring librte_rcu DIRS-$(CONFIG_RTE_LIBRTE_EFD) += librte_efd DEPDIRS-librte_efd := librte_eal librte_ring librte_hash DIRS-$(CONFIG_RTE_LIBRTE_RIB) += librte_rib
lib/librte_hash/Makefile +1 -1
@@ -8,7 +8,7 @@ LIB = librte_hash.a CFLAGS += -O3 CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -LDLIBS += -lrte_eal -lrte_ring +LDLIBS += -lrte_eal -lrte_ring -lrte_rcu EXPORT_MAP := rte_hash_version.map
lib/librte_hash/meson.build +1 -0
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h', sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c') deps += ['ring'] +deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +225 -66
404 lines not on this page · raw message
lib/librte_hash/rte_cuckoo_hash.h +8 -0
@@ -168,6 +168,11 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +/* RCU config */ +struct rte_hash_rcu_config *hash_rcu_cfg; +/**< HASH RCU QSBR configuration structure */ +struct rte_rcu_qsbr_dq *dq;/**< RCU QSBR defer queue. */ +/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node { int prev_slot; /* Parent(slot) in search path */ }; +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_HASH_RCU_DQ_RECLAIM_MAX 16 +#endif
lib/librte_hash/rte_hash.h +71 -4
123 lines not on this page · raw message
lib/librte_hash/rte_hash_version.map +1 -1
@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; };
Signature · 1 line
2.17.1
Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
CI has reported some compilation issues for this patch. I will resolve these issues once the RFC patch is approved.
Thank you!
Quoted 671 lines · Dharmik Thakkar
Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>---
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted
---
lib/Makefile | 2 +- lib/librte_hash/Makefile | 2 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 291 +++++++++++++++++++++------ lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 75 ++++++- lib/librte_hash/rte_hash_version.map | 2 +- 7 files changed, 308 insertions(+), 73 deletions(-)lib/Makefile +0
-0@@ -51,7 +51,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += librte_vhost DEPDIRS-librte_vhost := librte_eal librte_mempool librte_mbuf librte_ethdev \ librte_net librte_hash librte_cryptodevDIRS-$(CONFIG_RTE_LIBRTE_HASH) +=
librte_hashlib/Makefile +1
-1-DEPDIRS-librte_hash := librte_eal librte_ring +DEPDIRS-librte_hash := librte_eal librte_ring librte_rcuDIRS-$(CONFIG_RTE_LIBRTE_EFD) +=
librte_efdDEPDIRS-librte_efd :=librte_eallibrte_ringlibrte_hashDIRS-$(CONFIG_RTE_LIBRTE_RIB) +=librte_riblib/librte_hash/Makefile +0
-0@@ -8,7 +8,7 @@ LIB = librte_hash.aCFLAGS += -O3
CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)lib/librte_hash/Makefile +1
-1-LDLIBS += -lrte_eal -lrte_ring +LDLIBS += -lrte_eal -lrte_ring -lrte_rcu
EXPORT_MAP:= rte_hash_version.maplib/librte_hash/meson.build +0
-0@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h', sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')deps += ['ring']
lib/librte_hash/meson.build +1
-0+deps += ['rcu']lib/librte_hash/rte_cuckoo_hash.c +5
-0@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = { };EAL_REGISTER_TAILQ (rte_hash_tailq) +struct __rte_hash_rcu_dq_entry { +uint32_t key_idx; +uint32_t ext_bkt_idx;/**< Extended bkt index */ +}; +struct
rte_hash*rte_hash_find_existing(const char *name)
{lib/librte_hash/rte_cuckoo_hash.c +49
-1@@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)if (params->extra_flag &RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ) { readwrite_concur_lf_support =1 ; -/* Enable not freeing internal memory/index on delete */ +/* Enable not freeing internal memory/index on delete. +* If internal RCU is enabled, freeing of internal memory/index +* is done on delete +*/ no_free_on_del =1 ; }@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)rte_mcfg_tailq_write_unlock (); +/* RCU clean up. */ +if (h->dq) +rte_rcu_qsbr_dq_delete (h->dq); +if (h->use_local_cache)rte_free (h->local_free_slots);if (h->writer_takes_lock)@@ -612,6 +624,16 @@ rte_hash_reset(struct rte_hash *h)return ;__hash_rw_writer_lock (h); + +/* RCU clean up. */ +if (h->hash_rcu_cfg->v) { +rte_rcu_qsbr_dq_delete (h->dq); + h->dq =NULL ; +if (rte_hash_rcu_qsbr_add (h, h->hash_rcu_cfg) <0 ) { +RTE_LOG (ERR ,HASH ,"RCU init failed\n" ); +return ; + } + }memset (h->buckets,0 , h->num_buckets *sizeof (struct rte_hash_bucket ));memset (h->key_store,0 , h->key_entry_size * (h->entries +1 )); *h->tbl_chng_cnt =0 ;@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,return -ENOSPC ; } +static inline uint32_t +alloc_slot (const struct rte_hash *h,struct lcore_cache *cached_free_slots) +{ +unsigned int n_slots; +uint32_t slot_id; +if (h->use_local_cache) { +/* Try to get a free slot from the local cache */ +if (cached_free_slots->len ==0 ) { +/* Need to get another burst of free slots from global ring */ + n_slots =rte_ring_mc_dequeue_burst_elem (h->free_slots, + cached_free_slots->objs, +sizeof (uint32_t ), +LCORE_CACHE_SIZE ,NULL ); +if (n_slots ==0 ) +return EMPTY_SLOT ; + + cached_free_slots->len += n_slots; + } + +/* Get a free slot from the local cache */ + cached_free_slots->len--; + slot_id = cached_free_slots->objs[cached_free_slots->len]; + }else { +if (rte_ring_sc_dequeue_elem (h->free_slots, &slot_id, +sizeof (uint32_t )) !=0 ) +return EMPTY_SLOT ; + } + +return slot_id; +} +static inline
int32_t
__rte_hash_add_key_with_hash(const structrte_hash*h, const void *key,
hash_sig_tsig, void *data)lib/librte_hash/rte_cuckoo_hash.c +142
-34@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,uint32_t ext_bkt_id =0 ;uint32_t slot_id;int ret; -unsigned n_slots;unsigned lcore_id;unsigned int i;struct lcore_cache *cached_free_slots =NULL ;@@ -1001,28 +1053,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,if (h->use_local_cache) { lcore_id =rte_lcore_id (); cached_free_slots = &h->local_free_slots[lcore_id]; -/* Try to get a free slot from the local cache */ -if (cached_free_slots->len ==0 ) { -/* Need to get another burst of free slots from global ring */ - n_slots =rte_ring_mc_dequeue_burst_elem (h->free_slots, - cached_free_slots->objs, -sizeof (uint32_t ), -LCORE_CACHE_SIZE ,NULL ); -if (n_slots ==0 ) { -return -ENOSPC ; + } + slot_id =alloc_slot (h, cached_free_slots); +if (slot_id ==EMPTY_SLOT ) { +if (h->hash_rcu_cfg->v) { +if (rte_rcu_qsbr_dq_reclaim (h->dq, + h->hash_rcu_cfg->reclaim_max, +NULL ,NULL ,NULL ) + ==0 ) { + slot_id =alloc_slot (h, cached_free_slots); } - - cached_free_slots->len += n_slots; } - -/* Get a free slot from the local cache */ - cached_free_slots->len--; - slot_id = cached_free_slots->objs[cached_free_slots->len]; - }else { -if (rte_ring_sc_dequeue_elem (h->free_slots, &slot_id, -sizeof (uint32_t )) !=0 ) { +if (slot_id ==EMPTY_SLOT )return -ENOSPC ; - } } new_k =RTE_PTR_ADD (keys, slot_id * h->key_entry_size);@@ -1118,8 +1161,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,if (rte_ring_sc_dequeue_elem (h->free_ext_bkts, &ext_bkt_id,sizeof (uint32_t )) !=0 || ext_bkt_id ==0 ) { - ret = -ENOSPC ; - goto failure; +if (h->hash_rcu_cfg->v) { +if (rte_rcu_qsbr_dq_reclaim (h->dq, + h->hash_rcu_cfg->reclaim_max, +NULL ,NULL ,NULL ) + ==0 ) { +rte_ring_sc_dequeue_elem (h->free_ext_bkts, + &ext_bkt_id, +sizeof (uint32_t )); + } + } +if (ext_bkt_id ==0 ) { + ret = -ENOSPC ; + goto failure; + } }/* Use the first location of the new bucket */ @@ -1395,12 +1450,12 @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)return __rte_hash_lookup_with_hash (h, key,rte_hash_hash (h, key), data); } -static inline void -remove_entry (const struct rte_hash *h,struct rte_hash_bucket *bkt,unsigned i) +static int +free_slot (const struct rte_hash *h,uint32_t slot_id) {unsigned lcore_id, n_slots;struct lcore_cache *cached_free_slots; - +/* Return key indexes to free slot ring */ if (h->use_local_cache) { lcore_id =rte_lcore_id (); cached_free_slots = &h->local_free_slots[lcore_id];@@ -1411,18 +1466,122 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i) cached_free_slots->objs,sizeof (uint32_t ), LCORE_CACHE_SIZE, NULL); -ERR_IF_TRUE ((n_slots ==0 ), -"%s: could not enqueue free slots in global ring\n" , - __func__); +RETURN_IF_TRUE ((n_slots ==0 ), -EFAULT ); cached_free_slots->len -= n_slots; } -/* Put index of new free slot in cache. */ - cached_free_slots->objs[cached_free_slots->len] = - bkt->key_idx[i]; - cached_free_slots->len++; + } + +enqueue_slot_back (h, cached_free_slots, slot_id); +return 0 ; +} + +static void +__hash_rcu_qsbr_free_resource (void *p,void *e,unsigned int n) +{ +void *key_data =NULL ; +int ret; +struct rte_hash_key *keys, *k; +struct rte_hash *h = (struct rte_hash *)p; +struct __rte_hash_rcu_dq_entry rcu_dq_entry = + *((struct __rte_hash_rcu_dq_entry *)e); + +RTE_SET_USED (n); + keys = h->key_store; + + k = (struct rte_hash_key *) ((char *)keys + + rcu_dq_entry.key_idx * h->key_entry_size); + key_data = k->pdata; +if (h->hash_rcu_cfg->free_key_data_func) + h->hash_rcu_cfg->free_key_data_func (h->hash_rcu_cfg->key_data_ptr, + key_data); + +if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx !=EMPTY_SLOT ) +/* Recycle empty ext bkt to free list. */ +rte_ring_sp_enqueue_elem (h->free_ext_bkts, + &rcu_dq_entry.ext_bkt_idx,sizeof (uint32_t )); + +/* Return key indexes to free slot ring */ + ret =free_slot (h, rcu_dq_entry.key_idx); +if (ret <0 ) { +RTE_LOG (ERR ,HASH , +"%s: could not enqueue free slots in global ring\n" , + __func__); + } +} + +int +rte_hash_rcu_qsbr_add (struct rte_hash *h, +struct rte_hash_rcu_config *cfg) +{ +struct rte_rcu_qsbr_dq_parameters params = {0 }; +char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE ]; +struct rte_hash_rcu_config *hash_rcu_cfg =NULL ; + +const uint32_t total_entries = h->use_local_cache ? + h->entries + (RTE_MAX_LCORE -1 ) * (LCORE_CACHE_SIZE -1 ) +1 + : h->entries +1 ; + +if ((h ==NULL ) || cfg ==NULL ) { + rte_errno =EINVAL ; +return 1 ; + } + + hash_rcu_cfg =rte_zmalloc (NULL ,sizeof (struct rte_hash_rcu_config ),0 ); +if (hash_rcu_cfg ==NULL ) { +RTE_LOG (ERR ,HASH ,"memory allocation failed\n" ); +return 1 ; + } + +if (cfg->mode ==RTE_HASH_QSBR_MODE_SYNC ) { +/* No other things to do. */ + }else if (cfg->mode ==RTE_HASH_QSBR_MODE_DQ ) { +/* Init QSBR defer queue. */ +snprintf (rcu_dq_name,sizeof (rcu_dq_name), +"HASH_RCU_%s" , h->name); + params.name = rcu_dq_name; + params.size = cfg->dq_size; +if (params.size ==0 ) + params.size = total_entries; + params.trigger_reclaim_limit = cfg->reclaim_thd; +if (params.max_reclaim_size ==0 ) + params.max_reclaim_size =RTE_HASH_RCU_DQ_RECLAIM_MAX ; + params.esize =sizeof (struct __rte_hash_rcu_dq_entry ); + params.free_fn = __hash_rcu_qsbr_free_resource; + params.p = h; + params.v = cfg->v; + h->dq =rte_rcu_qsbr_dq_create (¶ms); +if (h->dq ==NULL ) { +rte_free (hash_rcu_cfg); +RTE_LOG (ERR ,HASH ,"HASH defer queue creation failed\n" ); +return 1 ; + } }else { -rte_ring_sp_enqueue_elem (h->free_slots, - &bkt->key_idx[i],sizeof (uint32_t )); +rte_free (hash_rcu_cfg); + rte_errno =EINVAL ; +return 1 ; + } + + hash_rcu_cfg->v = cfg->v; + hash_rcu_cfg->mode = cfg->mode; + hash_rcu_cfg->dq_size = cfg->dq_size; + hash_rcu_cfg->reclaim_thd = cfg->reclaim_thd; + hash_rcu_cfg->reclaim_max = cfg->reclaim_max; + hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func; + hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr; + + h->hash_rcu_cfg = hash_rcu_cfg; + +return 0 ; +} + +static inline void +remove_entry (const struct rte_hash *h,struct rte_hash_bucket *bkt,unsigned i) +{ +int ret =free_slot (h, bkt->key_idx[i]); +if (ret <0 ) { +RTE_LOG (ERR ,HASH , +"%s: could not enqueue free slots in global ring\n" , + __func__); } }@@ -1521,6 +1680,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,int pos;int32_t ret, i;uint16_t short_sig; +uint32_t index =EMPTY_SLOT ; +struct __rte_hash_rcu_dq_entry rcu_dq_entry; short_sig =get_short_sig (sig); prim_bucket_idx =get_prim_bucket_index (h, sig);@@ -1555,10 +1716,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,/* Search last bucket to see if empty to be recycled */
return_bkt:lib/librte_hash/rte_cuckoo_hash.c +3
-4-if (!last_bkt) { -__hash_rw_writer_unlock (h); -return ret; - } +if (!last_bkt) + goto return_key; +while (last_bkt->next) { prev_bkt = last_bkt; last_bkt = last_bkt->next;lib/librte_hash/rte_cuckoo_hash.c +26
-27@@ -1571,11 +1731,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,/* found empty bucket and recycle */ if (i ==RTE_HASH_BUCKET_ENTRIES ) { prev_bkt->next =NULL ; -uint32_t index = last_bkt - h->buckets_ext +1 ; + index = last_bkt - h->buckets_ext +1 ;/* Recycle the empty bkt if * no_free_on_del is disabled. */ -if (h->no_free_on_del) +if (h->no_free_on_del) {/* Store index of an empty ext bkt to be recycled * on calling rte_hash_del_xxx APIs. * When lock free read-write concurrency is enabled, @@ -1583,12 +1743,32 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,* immediately (as readers might be using it still). * Hence freeing of the ext bkt is piggy-backed to * freeing of the key index. +* If using external RCU, store this index in an array. */ - h->ext_bkt_to_free[ret] = index; - else +if (h->hash_rcu_cfg->v ==NULL ) + h->ext_bkt_to_free[ret] = index; + }else rte_ring_sp_enqueue_elem (h->free_ext_bkts, &index,sizeof (uint32_t )); } + +return_key: +/* Using internal RCU QSBR */ +if (h->hash_rcu_cfg->v) { +/* Key index where key is stored, adding the first dummy index */ + rcu_dq_entry.key_idx = ret +1 ; + rcu_dq_entry.ext_bkt_idx = index; +if (h->hash_rcu_cfg->mode ==RTE_HASH_QSBR_MODE_SYNC ) { +/* Wait for quiescent state change. */ +rte_rcu_qsbr_synchronize (h->hash_rcu_cfg->v, +RTE_QSBR_THRID_INVALID ); +__hash_rcu_qsbr_free_resource ((void *)((uintptr_t )h), + &rcu_dq_entry,1 ); + }else if (h->hash_rcu_cfg->mode ==RTE_HASH_QSBR_MODE_DQ ) +/* Push into QSBR FIFO. */ +if (rte_rcu_qsbr_dq_enqueue (h->dq, &rcu_dq_entry) !=0 ) +RTE_LOG (ERR ,HASH ,"Failed to push QSBR FIFO\n" ); + }__hash_rw_writer_unlock (h);return ret; }@@ -1637,8 +1817,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,RETURN_IF_TRUE (((h ==NULL ) || (key_idx ==EMPTY_SLOT )), -EINVAL ); -unsigned int lcore_id, n_slots; -struct lcore_cache *cached_free_slots;const uint32_t total_entries = h->use_local_cache ? h->entries + (RTE_MAX_LCORE -1 ) * (LCORE_CACHE_SIZE -1 ) +1 : h->entries +1 ;@@ -1656,28 +1834,9 @@ rte_hash_free_key_with_position(const struct rte_hash *h, } } -if (h->use_local_cache) { - lcore_id =rte_lcore_id (); - cached_free_slots = &h->local_free_slots[lcore_id]; -/* Cache full, need to free it. */ -if (cached_free_slots->len ==LCORE_CACHE_SIZE ) { -/* Need to enqueue the free slots in global ring. */ - n_slots =rte_ring_mp_enqueue_burst_elem (h->free_slots, - cached_free_slots->objs, -sizeof (uint32_t ), -LCORE_CACHE_SIZE ,NULL ); -RETURN_IF_TRUE ((n_slots ==0 ), -EFAULT ); - cached_free_slots->len -= n_slots; - } -/* Put index of new free slot in cache. */ - cached_free_slots->objs[cached_free_slots->len] = key_idx; - cached_free_slots->len++; - }else { -rte_ring_sp_enqueue_elem (h->free_slots, &key_idx, -sizeof (uint32_t )); - } +/* Enqueue slot to cache/ring of free slots. */ +return free_slot (h, key_idx); -return 0 ; }static inline void
lib/librte_hash/rte_cuckoo_hash.h +8
-0@@ -168,6 +168,11 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +/* RCU config */ +struct rte_hash_rcu_config *hash_rcu_cfg; +/**< HASH RCU QSBR configuration structure */ +struct rte_rcu_qsbr_dq *dq;/**< RCU QSBR defer queue. */ +/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;@@ -230,4 +235,7 @@ struct queue_node { int prev_slot; /* Parent(slot) in search path */ }; +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_HASH_RCU_DQ_RECLAIM_MAX 16 +#endif lib/librte_hash/rte_hash.h +49
-4@@ -15,6 +15,7 @@#include <stddef.h> #include <rte_compat.h> +#include <rte_rcu_qsbr.h> #ifdef __cplusplusextern "C" {@@ -45,7 +46,8 @@ extern "C" {/** Flag to disable freeing of key index on hash delete. * Refer to rte_hash_del_xxx APIs for more details. * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF -* is enabled. +* is enabled. However, if internal RCU is enabled, freeing of internal +* memory/index is done on delete */ #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,/** Type of function used to compare the hash key. */ typedef int (*rte_hash_cmp_eq_t )(const void *key1,const void *key2,size_t key_len); +/** +* Type of function used to free data stored in the key. +* Required when using internal RCU to allow application to free key-data once +* the key is returned to the the ring of free key-slots. +*/ +typedef void (*rte_hash_free_key_data )(void *p,void *key_data); +/** * Parameters used when creating the hash table. */ @@ -81,6 +90,39 @@ struct rte_hash_parameters {uint8_t extra_flag;/**< Indicate if additional parameters are present. */ }; +/** RCU reclamation modes */ +enum rte_hash_qsbr_mode { +/** Create defer queue for reclaim. */ +RTE_HASH_QSBR_MODE_DQ =0 , +/** Use blocking mode reclaim. No defer queue created. */ +RTE_HASH_QSBR_MODE_SYNC +}; + +/** HASH RCU QSBR configuration structure. */ +struct rte_hash_rcu_config { +struct rte_rcu_qsbr *v;/**< RCU QSBR variable. */ +enum rte_hash_qsbr_mode mode; +/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx +* '0' for default: create defer queue for reclaim. +*/ +uint32_t dq_size; +/**< RCU defer queue size. +* default: total hash table entries. +*/ +uint32_t reclaim_thd;/**< Threshold to trigger auto reclaim. */ +uint32_t reclaim_max; +/**< Max entries to reclaim in one go. +* default: RTE_HASH_RCU_DQ_RECLAIM_MAX. +*/ +void *key_data_ptr; +/**< Pointer passed to the free function. Typically, this is the +* pointer to the data structure to which the resource to free +* (key-data) belongs. This can be NULL. +*/ +rte_hash_free_key_data free_key_data_func; +/**< Function to call to free the resource (key-data). */ +}; +/** @internal A hash table structure. */ struct rte_hash ;@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t* Thread safety can be enabled by setting flag during * table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or -* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, +* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and +* internal RCU is NOT enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called * additionally to free the index associated with the key. @@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);* Thread safety can be enabled by setting flag during * table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or -* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, +* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and +* internal RCU is NOT enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called * additionally to free the index associated with the key. @@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,* only be called from one thread by default. Thread safety * can be enabled by setting flag during table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or -* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, +* RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and +* internal RCU is NOT enabled, * the key index returned by rte_hash_del_key_xxx APIs must be freed * using this API. This API should be called after all the readers * have stopped referencing the entry corresponding to this key. @@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,*/
int32_trte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);lib/librte_hash/rte_hash.h +22
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Associate RCU QSBR variable with an Hash object. +* +* @param h +* the hash object to add RCU QSBR +* @param cfg +* RCU QSBR configuration +* @return +* On success - 0 +* On error - 1 with error code set in rte_errno. +* Possible rte_errno codes are: +* - EINVAL - invalid pointer +* - EEXIST - already added QSBR +* - ENOMEM - memory allocation failure +*/ +__rte_experimental +int rte_hash_rcu_qsbr_add(struct rte_hash *h, +struct rte_hash_rcu_config *cfg);#ifdef
__cplusplus
}
#endiflib/librte_hash/rte_hash_version.map +1
-1@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; };Signature · 1 line
2.17.1
Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
In reply to Dharmik ThakkarQuoted 46 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, August 18, 2020 9:06 PM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [RFC v2] lib/hash: integrate RCU QSBRIntegrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>---
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted
--- lib/Makefile | 2 +- lib/librte_hash/Makefile | 2 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 291 +++++++++++++++++++++------ lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 75 ++++++- lib/librte_hash/rte_hash_version.map | 2 +- 7 files changed, 308 insertions(+), 73 deletions(-)
<......>
Quoted 14 lines
lib/librte_hash/rte_hash.h +14
-0+/** HASH RCU QSBR configuration structure. */ struct +rte_hash_rcu_config { +struct rte_rcu_qsbr *v;/**< RCU QSBR variable. */ +enum rte_hash_qsbr_mode mode; +/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx +* '0' for default: create defer queue for reclaim. +*/ +uint32_t dq_size; +/**< RCU defer queue size. +* default: total hash table entries. +*/ +uint32_t reclaim_thd;/**< Threshold to trigger auto reclaim. */ +uint32_t reclaim_max; +/**< Max entries to reclaim in one go.
lib/librte_hash/rte_hash.h struct rte_hash_parameters { · line 125
Quoted 10 lines
lib/librte_hash/rte_hash.h +10
-0+* default: RTE_HASH_RCU_DQ_RECLAIM_MAX. +*/ +void *key_data_ptr; +/**< Pointer passed to the free function. Typically, this is the +* pointer to the data structure to which the resource to free +* (key-data) belongs. This can be NULL. +*/ +rte_hash_free_key_data free_key_data_func; +/**< Function to call to free the resource (key-data). */ }; +
[Wang, Yipeng]
I guess this is mostly a wrapper of rte_rcu_qsbr_dq_parameters.
Personally, I incline to use variable names that match the existing qsbr parameters better. For example, you could still call reclaim_thd as reclaim_limit. And _max to be _size. Thus, people who are already familiar with qsbr can match the meanings better.
Quoted 37 lines
/** @internal A hash table structure. */ struct
rte_hash;(quoted diff) +0
-0@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h,const void *key,
hash_sig_t
* Thread safety can be enabled by setting flag during
* table creation.
* IfRTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DELor
- *RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LFis enabled,
+ *RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LFis enabled and
+ * internal RCU is NOT enabled,
* the key index returned byrte_hash_add_key_xxxAPIs will not be
* freed by this API.rte_hash_free_key_with_positionAPI must be called
* additionally to free the index associated with the key.lib/librte_hash/rte_hash.h +2
-1@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key); * Thread safety can be enabled by setting flag during * table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called* additionally to free the index associated with the key.
lib/librte_hash/rte_hash.h +2
-1@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position, * only be called from one thread by default. Thread safety * can be enabled by setting flag during table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_del_key_xxx APIs must be freed * using this API. This API should be called after all the readers* have stopped referencing the entry corresponding to this key.
lib/librte_hash/rte_hash.h +0
-0@@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, const · line 680
Quoted 10 lines
lib/librte_hash/rte_hash.h +6
-0*/ int32_t rte_hash_iterate (const struct rte_hash *h,const void **key,void **data,uint32_t *next); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Associate RCU QSBR variable with an Hash object.
[Wang, Yipeng] To enable RCU we need to call this func.
I think you can be more explicit, e.g. "This API should be called to enable the RCU support"
lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, · line 691
Quoted 13 lines
lib/librte_hash/rte_hash.h +13
-0+* +* @param h +* the hash object to add RCU QSBR +* @param cfg +* RCU QSBR configuration +* @return +* On success - 0 +* On error - 1 with error code set in rte_errno. +* Possible rte_errno codes are: +* - EINVAL - invalid pointer +* - EEXIST - already added QSBR +* - ENOMEM - memory allocation failure +*/
[Wang, Yipeng] Is there any further requirement for when to call this API? E.g. you could say "this API should be called immediately after rte_hash_create()"
Quoted 19 lines
lib/librte_hash/rte_hash.h +3
-0+__rte_experimental +int rte_hash_rcu_qsbr_add(struct rte_hash *h, +struct rte_hash_rcu_config *cfg);#ifdef __cplusplus}
#endiflib/librte_hash/rte_hash_version.map +1
-2@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; }; --
Quoted 1 line
2.17.1
[Wang, Yipeng]
Hi, Dharmik,
Thanks for the patch. It generally looks good to me.
I guess you will revise documentation and the unit test as well after the RFC. That is helpful for users to understand how to use hash appropriately with the RCU lib.
Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
Hi Yipeng,
Thank you for the comments!
Quoted 73 lines · On Aug 31, 2020, at 3:47 PM, Wang, Yipeng1 <[email protected]> wrote:
Quoted 47 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, August 18, 2020 9:06 PM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [RFC v2] lib/hash: integrate RCU QSBRIntegrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>---
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted
---
lib/Makefile | 2 +- lib/librte_hash/Makefile | 2 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 291 +++++++++++++++++++++------ lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 75 ++++++- lib/librte_hash/rte_hash_version.map | 2 +- 7 files changed, 308 insertions(+), 73 deletions(-)<......>
Quoted 21 lines
lib/librte_hash/rte_hash.h +21
-0+/** HASH RCU QSBR configuration structure. */ struct +rte_hash_rcu_config { +struct rte_rcu_qsbr *v;/**< RCU QSBR variable. */ +enum rte_hash_qsbr_mode mode; +/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx +* '0' for default: create defer queue for reclaim. +*/ +uint32_t dq_size; +/**< RCU defer queue size. +* default: total hash table entries. +*/ +uint32_t reclaim_thd;/**< Threshold to trigger auto reclaim. */ +uint32_t reclaim_max; +/**< Max entries to reclaim in one go. +* default: RTE_HASH_RCU_DQ_RECLAIM_MAX. +*/ +void *key_data_ptr; +/**< Pointer passed to the free function. Typically, this is the +* pointer to the data structure to which the resource to free +* (key-data) belongs. This can be NULL. +*/
Quoted 9 lines
lib/librte_hash/rte_hash.h struct rte_hash_parameters { · line 125
Quoted 3 lines
lib/librte_hash/rte_hash.h +3
-0+rte_hash_free_key_data free_key_data_func; +/**< Function to call to free the resource (key-data). */ }; +[Wang, Yipeng]
I guess this is mostly a wrapper ofrte_rcu_qsbr_dq_parameters.
Personally, I incline to use variable names that match the existing qsbr parameters better. For example, you could still callreclaim_thdasreclaim_limit. And_maxto be_size. Thus, people who are already familiar with qsbr can match the meanings better.
Makes sense. I will update it.
Quoted 42 lines
Quoted 42 lines
/** @internal A hash table structure. */ struct
rte_hash;(quoted diff) +0
-0@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h,const void *key,
hash_sig_t
* Thread safety can be enabled by setting flag during
* table creation.
* IfRTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DELor
- *RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LFis enabled,
+ *RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LFis enabled and
+ * internal RCU is NOT enabled,
* the key index returned byrte_hash_add_key_xxxAPIs will not be
* freed by this API.rte_hash_free_key_with_positionAPI must be called
* additionally to free the index associated with the key.lib/librte_hash/rte_hash.h +2
-1@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key); * Thread safety can be enabled by setting flag during * table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called* additionally to free the index associated with the key.
lib/librte_hash/rte_hash.h +2
-1@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position, * only be called from one thread by default. Thread safety * can be enabled by setting flag during table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_del_key_xxx APIs must be freed * using this API. This API should be called after all the readers* have stopped referencing the entry corresponding to this key.
(quoted diff) +0
-0@@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, */
int32_trte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
Quoted 9 lines
lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, · line 678
Quoted 6 lines
lib/librte_hash/rte_hash.h +6
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Associate RCU QSBR variable with an Hash object. [Wang, Yipeng] To enable RCU we need to call this func.
I think you can be more explicit, e.g. "This API should be called to enable the RCU support"
Yes.
Quoted 13 lines
Quoted 6 lines
+ *
+ * @param h
+ * the hash object to add RCU QSBR
+ * @param cfg
+ * RCU QSBR configuration
+ * @returnQuoted 7 lines
+ * On success - 0 + * On error - 1 with error code set in rte_errno. + * Possible rte_errno codes are: + * - EINVAL - invalid pointer + * - EEXIST - already added QSBR + * - ENOMEM - memory allocation failure+ */
Quoted 3 lines
[Wang, Yipeng] Is there any further requirement for when to call this API? E.g. you could say "this API should be called immediately after
rte_hash_create()"
Sure, I will add further guidelines/requirements.
Quoted 15 lines
Quoted 15 lines
lib/librte_hash/rte_hash.h +3
-0+__rte_experimental +int rte_hash_rcu_qsbr_add(struct rte_hash *h, +struct rte_hash_rcu_config *cfg);#ifdef
__cplusplus
}
#endiflib/librte_hash/rte_hash_version.map +0
-0@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id;
Quoted 8 lines
Quoted 5 lines
lib/librte_hash/rte_hash_version.map +1
-2- + rte_hash_rcu_qsbr_add; }; --2.17.1
[Wang, Yipeng]
Hi, Dharmik,
Thanks for the patch. It generally looks good to me.
That’s great. I will convert it to a patch.
Quoted 2 lines
I guess you will revise documentation and the unit test as well after the RFC. That is helpful for users to understand how to use hash appropriately with the RCU lib.
Yes, I will add the documentation and unit test patches.
Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
Hello Dharmik,
Quoted 10 lines · Dharmik Thakkar
Quoted 3 lines
[Wang, Yipeng]
Hi, Dharmik,
Thanks for the patch. It generally looks good to me.That’s great. I will convert it to a patch.
Quoted 2 lines
I guess you will revise documentation and the unit test as well after the RFC. That is helpful for users to understand how to use hash appropriately with the RCU lib.
Yes, I will add the documentation and unit test patches.
I did not see a non RFC patch.
Is this still targeting 20.11?
Signature · 1 line
David Marchand
Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR
Hi David,
Quoted 11 lines · David Marchand
Hello Dharmik,
Quoted 6 lines · Dharmik Thakkar
Quoted 3 lines
[Wang, Yipeng]
Hi, Dharmik,
Thanks for the patch. It generally looks good to me.That’s great. I will convert it to a patch.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I guess you will revise documentation and the unit test as well after the RFC. That is helpful for users to understand how to use hash appropriately with the RCU lib.
Yes, I will add the documentation and unit test patches.
I did not see a non RFC patch.
Is this still targeting 20.11?
Yes, I will send out the patch series today. Sorry, it got delayed!
Quoted 4 lines
Signature · 2 lines
-- David Marchand
cover hash: integrate RCU QSBR
In reply to Dharmik ThakkarIntegrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.
Functional tests and performance tests are added to cover the integration with RCU.
---
v3:
- Add documentation
- Add unit tests
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead
- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted
Dharmik Thakkar (3):
lib/hash: integrate RCU QSBR
test/hash: replace rte atomic with C11 atomic APIs test/hash: add tests for integrated RCU QSBR
app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 260 +++++++++++++---- doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 300 ++++++++++++++----- lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 ++++- lib/librte_hash/rte_hash_version.map | 2 +- 8 files changed, 917 insertions(+), 132 deletions(-)
Signature · 1 line
2.17.1
PATCH v3 1/3 lib/hash: integrate RCU QSBR
Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.
--- doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 300 +++++++++++++++++++++------ lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 ++++++- lib/librte_hash/rte_hash_version.map | 2 +- 6 files changed, 324 insertions(+), 75 deletions(-)
doc/guides/prog_guide/hash_lib.rst +8 -3
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values * If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position. Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position. + RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature. + Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details. + Extendable Bucket Functionality support ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default). in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the hash table size and can't tolerate any key insertion failure (even if very few). -Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and -deleted keys, to maintain the 100% capacity guarantee. +Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR +(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee. Implementation Details (non Extendable Bucket Case) ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion: Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency', the position of the key is not freed. It is the responsibility of the user to free the position after -readers are not referencing the position anymore. +readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below: * Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set, wait till the readers are not referencing the position returned during add/delete flow and then free the position. RCU mechanisms can be used to find out when the readers are not referencing the position anymore. + RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature. + Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details. * Lookup flow: Lookup for the flow key in the hash. If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
lib/librte_hash/meson.build +1 -0
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h', sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c') deps += ['ring'] +deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +233 -67
417 lines not on this page · raw message
lib/librte_hash/rte_cuckoo_hash.h +8 -0
@@ -168,6 +168,11 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +/* RCU config */ +struct rte_hash_rcu_config *hash_rcu_cfg; +/**< HASH RCU QSBR configuration structure */ +struct rte_rcu_qsbr_dq *dq;/**< RCU QSBR defer queue. */ +/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node { int prev_slot; /* Parent(slot) in search path */ }; +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_HASH_RCU_DQ_RECLAIM_MAX 16 +#endif
lib/librte_hash/rte_hash.h +73 -4
125 lines not on this page · raw message
lib/librte_hash/rte_hash_version.map +1 -1
@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; };
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v3 1/3] lib/hash: integrate RCU QSBR
Quoted 11 lines · Dharmik Thakkar
Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>
Quoted 9 lines
--- doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 300 +++++++++++++++++++++------ lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 ++++++- lib/librte_hash/rte_hash_version.map | 2 +- 6 files changed, 324 insertions(+), 75 deletions(-)
PATCH v3 2/3 test/hash: replace rte atomic with C11 atomic APIs
In reply to Dharmik ThakkarReplace rte_atomic APIs with C11 atomic APIs intest_hash_readwrite_lf_perf.c
--- app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++--------------- 1 file changed, 36 insertions(+), 53 deletions(-)
app/test/test_hash_readwrite_lf_perf.c +36 -53
@@ -82,8 +82,8 @@ static struct {struct rte_hash *h; } tbl_rwc_test_param; -static rte_atomic64_t gread_cycles; -static rte_atomic64_t greads; +static uint64_t gread_cycles; +static uint64_t greads;static volatile uint8_t writer_done;
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg) }while (!writer_done); cycles =rte_rdtsc_precise () - begin; -rte_atomic64_add (&gread_cycles, cycles); -rte_atomic64_add (&greads, read_cnt*loop_cnt); +__atomic_fetch_add (&gread_cycles, cycles, __ATOMIC_RELAXED); +__atomic_fetch_add (&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);return 0 ; }
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type =WRITE_NO_KEY_SHIFT ;uint8_t read_type =READ_PASS_NO_KEY_SHIFTS ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - no key-shifts, read - hit\n" );
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_no_ks_r_hit[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t read_type =READ_FAIL ;int ret; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - no key-shifts, Hash lookup - miss\n" );
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_no_ks_r_miss[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,uint8_t write_type;uint8_t read_type =READ_PASS_NON_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err; printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_nsp[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type;uint8_t read_type =READ_PASS_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err; printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf, goto finish;printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); + +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_sp[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, intuint8_t write_type;uint8_t read_type =READ_FAIL ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - key shift, Hash lookup - miss\n" );
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, intprintf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup); }
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type;uint8_t read_type =READ_PASS_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Multi-add-lookup\n" );
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , + __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) - /rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, + __ATOMIC_RELAXED) / +__atomic_load_n (&greads, + __ATOMIC_RELAXED); rwc_perf_results->multi_rw[m][k][n] = cycles_per_lookup; printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,uint8_t write_type;uint8_t read_type =READ_PASS_KEY_SHIFTS_EXTBKT ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n" );
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); write_type =WRITE_NO_KEY_SHIFT ;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_extbkt[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
Signature · 1 line
2.17.1
PATCH v3 3/3 test/hash: add tests for integrated RCU QSBR
In reply to Dharmik ThakkarAdd functional and performance tests for the integrated RCU QSBR.
--- app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 171 ++++++++++- 2 files changed, 557 insertions(+), 4 deletions(-)
app/test/test_hash.c +389 -1
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) -#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ +#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ if (cond) { \ printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ if (handle) rte_fbk_hash_free(handle); \
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) +#define RETURN_IF_ERROR_RCU_QSBR (cond, str, ...) do { \ + if (cond) { \ + printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ + if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) { \ + writer_done = 1; \ +/* Wait until reader exited. */ \ +rte_eal_mp_wait_lcore (); \ + } \ +if (g_handle)rte_hash_free (g_handle); \ +if (g_qsv)rte_free (g_qsv); \ +return -1 ; \ + } \ +} while(0) +/* 5-tuple key type */ struct flow_key {uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)return ret; } +static struct rte_hash *g_handle; +static struct rte_rcu_qsbr *g_qsv; +static volatile uint8_t writer_done; +struct flow_key g_rand_keys[9 ]; +/* +* rte_hash_rcu_qsbr_add positive and negative tests. +* - Add RCU QSBR variable to Hash +* - Add another RCU QSBR variable to Hash +* - Check returns +*/ +static int +test_hash_rcu_qsbr_add (void ) +{ +size_t sz; +struct rte_rcu_qsbr *qsv2 =NULL ; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + +struct rte_hash_parameters params; + +printf ("\n# Running RCU QSBR add tests\n" ); +memcpy (¶ms, &ut_params,sizeof (params)); + params.name ="test_hash_rcu_qsbr_add" ; + params.extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF | +RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD ; + g_handle =rte_hash_create (¶ms); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; +/* Invalid QSBR mode */ + rcu_cfg.mode =2 ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status ==0 ,"Invalid QSBR mode test failed" ); + + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Create and attach another RCU QSBR to hash table */ + qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (qsv2 ==NULL , +"RCU QSBR variable creation failed" ); + + rcu_cfg.v = qsv2; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +rte_free (qsv2); +RETURN_IF_ERROR_RCU_QSBR (status ==0 , +"Attach RCU QSBR to hash table succeeded where failure" +" is expected" ); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; +} + +/* +* rte_hash_rcu_qsbr_add DQ mode functional test. +* Reader and writer are in the same thread in this test. +* - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries +* - Add RCU QSBR variable to hash +* - Add 8 hash entries and fill the bucket +* - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt +* - Register a reader thread (not a real thread) +* - Reader lookup existing entry +* - Writer deletes the entry +* - Reader lookup the entry +* - Writer re-add the entry (no available free index) +* - Reader report quiescent state and unregister +* - Writer re-add the entry +* - Reader lookup the entry +*/ +static int +test_hash_rcu_qsbr_dq_mode (uint8_t ext_bkt) +{ +uint32_t total_entries = (ext_bkt ==0 ) ?8 :9 ; + +uint8_t hash_extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ; + +if (ext_bkt) + hash_extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; + +struct rte_hash_parameters params_pseudo_hash = { + .name ="test_hash_rcu_qsbr_dq_mode" , + .entries = total_entries, + .key_len =sizeof (struct flow_key ),/* 13 */ + .hash_func = pseudo_hash, + .hash_func_init_val =0 , + .socket_id =0 , + .extra_flag = hash_extra_flag, + }; +int pos[total_entries]; +int expected_pos[total_entries]; +unsigned i; +size_t sz; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + + g_qsv =NULL ; + g_handle =NULL ; + +for (i =0 ; i < total_entries; i++) { + g_rand_keys[i].port_dst = i; + g_rand_keys[i].port_src = i+1 ; + } + +if (ext_bkt) +printf ("\n# Running RCU QSBR DQ mode functional test with" +" ext bkt\n" ); +else +printf ("\n# Running RCU QSBR DQ mode functional test\n" ); + + g_handle =rte_hash_create (¶ms_pseudo_hash); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Fill bucket */ +for (i =0 ; i < total_entries; i++) { + pos[i] =rte_hash_add_key (g_handle, &g_rand_keys[i]); +print_key_info ("Add" , &g_rand_keys[i], pos[i]); +RETURN_IF_ERROR_RCU_QSBR (pos[i] <0 , +"failed to add key (pos[%u]=%d)" , i, + pos[i]); + expected_pos[i] = pos[i]; + } + +/* Register pseudo reader */ + status =rte_rcu_qsbr_thread_register (g_qsv,0 ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR thread registration failed" ); +rte_rcu_qsbr_thread_online (g_qsv,0 ); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to find correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +/* Writer update */ + pos[0 ] =rte_hash_del_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Del" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to del correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != -ENOENT , +"found deleted key (pos[%u]=%d)" ,0 , pos[0 ]); + +/* Fill bucket */ + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != -ENOSPC , +"Added key successfully (pos[%u]=%d)" ,0 , pos[0 ]); + +/* Reader quiescent */ +rte_rcu_qsbr_quiescent (g_qsv,0 ); + +/* Fill bucket */ + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] <0 , +"failed to add key (pos[%u]=%d)" ,0 , pos[0 ]); + expected_pos[0 ] = pos[0 ]; + +rte_rcu_qsbr_thread_offline (g_qsv,0 ); + (void )rte_rcu_qsbr_thread_unregister (g_qsv,0 ); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to find correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +rte_hash_free (g_handle); +rte_free (g_qsv); +return 0 ; + +} + +/* Report quiescent state interval every 1024 lookups. Larger critical +* sections in reader will result in writer polling multiple times. +*/ +#define QSBR_REPORTING_INTERVAL 1024 +#define WRITER_ITERATIONS 512 + +/* +* Reader thread using rte_hash data structure with RCU. +*/ +static int +test_hash_rcu_qsbr_reader (void *arg) +{ +int i; + +RTE_SET_USED (arg); +/* Register this thread to report quiescent state */ + (void )rte_rcu_qsbr_thread_register (g_qsv,0 ); +rte_rcu_qsbr_thread_online (g_qsv,0 ); + +do { +for (i =0 ; i <QSBR_REPORTING_INTERVAL ; i++) +rte_hash_lookup (g_handle, &g_rand_keys[0 ]); + +/* Update quiescent state */ +rte_rcu_qsbr_quiescent (g_qsv,0 ); + }while (!writer_done); + +rte_rcu_qsbr_thread_offline (g_qsv,0 ); + (void )rte_rcu_qsbr_thread_unregister (g_qsv,0 ); + +return 0 ; +} + +/* +* rte_hash_rcu_qsbr_add sync mode functional test. +* 1 Reader and 1 writer. They cannot be in the same thread in this test. +* - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries +* - Add RCU QSBR variable to hash +* - Register a reader thread. Reader keeps looking up a specific key. +* - Writer keeps adding and deleting a specific key. +*/ +static int +test_hash_rcu_qsbr_sync_mode (uint8_t ext_bkt) +{ +uint32_t total_entries = (ext_bkt ==0 ) ?8 :9 ; + +uint8_t hash_extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ; + +if (ext_bkt) + hash_extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; + +struct rte_hash_parameters params_pseudo_hash = { + .name ="test_hash_rcu_qsbr_sync_mode" , + .entries = total_entries, + .key_len =sizeof (struct flow_key ),/* 13 */ + .hash_func = pseudo_hash, + .hash_func_init_val =0 , + .socket_id =0 , + .extra_flag = hash_extra_flag, + }; +int pos[total_entries]; +int expected_pos[total_entries]; +unsigned i; +size_t sz; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + + g_qsv =NULL ; + g_handle =NULL ; + +for (i =0 ; i < total_entries; i++) { + g_rand_keys[i].port_dst = i; + g_rand_keys[i].port_src = i+1 ; + } + +if (ext_bkt) +printf ("\n# Running RCU QSBR sync mode functional test with" +" ext bkt\n" ); +else +printf ("\n# Running RCU QSBR sync mode functional test\n" ); + + g_handle =rte_hash_create (¶ms_pseudo_hash); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Launch reader thread */ +rte_eal_remote_launch (test_hash_rcu_qsbr_reader,NULL , +rte_get_next_lcore (-1 ,1 ,0 )); + +/* Fill bucket */ +for (i =0 ; i < total_entries; i++) { + pos[i] =rte_hash_add_key (g_handle, &g_rand_keys[i]); +print_key_info ("Add" , &g_rand_keys[i], pos[i]); +RETURN_IF_ERROR_RCU_QSBR (pos[i] <0 , +"failed to add key (pos[%u]=%d)" , i, pos[i]); + expected_pos[i] = pos[i]; + } + writer_done =0 ; + +/* Writer Update */ +for (i =0 ; i <WRITER_ITERATIONS ; i++) { + expected_pos[0 ] = pos[0 ]; + pos[0 ] =rte_hash_del_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Del" , &g_rand_keys[0 ], status); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to del correct key (pos[%u]=%d)" + ,0 , pos[0 ]); + + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] <0 , +"failed to add key (pos[%u]=%d)" ,0 , + pos[0 ]); + } + + writer_done =1 ; +/* Wait until reader exited. */ +rte_eal_mp_wait_lcore (); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; + +} +/* * Do all unit and performance tests. */
@@ -1862,6 +2235,21 @@ test_hash(void)if (test_crc32_hash_alg_equiv () <0 )return -1 ; +if (test_hash_rcu_qsbr_add () <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_dq_mode (0 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_dq_mode (1 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_sync_mode (0 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_sync_mode (1 ) <0 ) +return -1 ; +return 0 ; }
app/test/test_hash_readwrite_lf_perf.c +168 -3
225 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH 0/3] hash: integrate RCU QSBR
In reply to Dharmik ThakkarQuoted 12 lines · Dharmik Thakkar
Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
Quoted 7 lines
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Functional tests and performance tests are added to cover the integration with RCU.
clang builds are broken.
http://mails.dpdk.org/archives/test-report/2020-October/160043.html
https://travis-ci.com/github/ovsrobot/dpdk/builds/190521500
Signature · 1 line
David Marchand
v4 cover hash: integrate RCU QSBR
In reply to Dharmik ThakkarIntegrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.
Functional tests and performance tests are added to cover the integration with RCU.
---
v4:
- Fix clang compilation issues
v3:
- Add documentation
- Add unit tests
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead
- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted
Dharmik Thakkar (3):
lib/hash: integrate RCU QSBR
test/hash: replace rte atomic with C11 atomic APIs test/hash: add tests for integrated RCU QSBR
app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 260 +++++++++++++---- doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 302 ++++++++++++++----- lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 ++++- lib/librte_hash/rte_hash_version.map | 2 +- 8 files changed, 918 insertions(+), 133 deletions(-)
Signature · 1 line
2.17.1
PATCH v4 1/3 lib/hash: integrate RCU QSBR
Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.
--- doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 302 +++++++++++++++++++++------ lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 ++++++- lib/librte_hash/rte_hash_version.map | 2 +- 6 files changed, 325 insertions(+), 76 deletions(-)
doc/guides/prog_guide/hash_lib.rst +8 -3
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values * If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position. Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position. + RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature. + Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details. + Extendable Bucket Functionality support ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default). in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the hash table size and can't tolerate any key insertion failure (even if very few). -Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and -deleted keys, to maintain the 100% capacity guarantee. +Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR +(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee. Implementation Details (non Extendable Bucket Case) ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion: Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency', the position of the key is not freed. It is the responsibility of the user to free the position after -readers are not referencing the position anymore. +readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below: * Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set, wait till the readers are not referencing the position returned during add/delete flow and then free the position. RCU mechanisms can be used to find out when the readers are not referencing the position anymore. + RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature. + Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details. * Lookup flow: Lookup for the flow key in the hash. If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
lib/librte_hash/meson.build +1 -0
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h', sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c') deps += ['ring'] +deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +234 -68
418 lines not on this page · raw message
lib/librte_hash/rte_cuckoo_hash.h +8 -0
@@ -168,6 +168,11 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +/* RCU config */ +struct rte_hash_rcu_config *hash_rcu_cfg; +/**< HASH RCU QSBR configuration structure */ +struct rte_rcu_qsbr_dq *dq;/**< RCU QSBR defer queue. */ +/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node { int prev_slot; /* Parent(slot) in search path */ }; +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_HASH_RCU_DQ_RECLAIM_MAX 16 +#endif
lib/librte_hash/rte_hash.h +73 -4
125 lines not on this page · raw message
lib/librte_hash/rte_hash_version.map +1 -1
@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; };
Signature · 1 line
2.17.1
PATCH v4 2/3 test/hash: replace rte atomic with C11 atomic APIs
In reply to Dharmik ThakkarReplace rte_atomic APIs with C11 atomic APIs intest_hash_readwrite_lf_perf.c
--- app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++--------------- 1 file changed, 36 insertions(+), 53 deletions(-)
app/test/test_hash_readwrite_lf_perf.c +36 -53
@@ -82,8 +82,8 @@ static struct {struct rte_hash *h; } tbl_rwc_test_param; -static rte_atomic64_t gread_cycles; -static rte_atomic64_t greads; +static uint64_t gread_cycles; +static uint64_t greads;static volatile uint8_t writer_done;
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg) }while (!writer_done); cycles =rte_rdtsc_precise () - begin; -rte_atomic64_add (&gread_cycles, cycles); -rte_atomic64_add (&greads, read_cnt*loop_cnt); +__atomic_fetch_add (&gread_cycles, cycles, __ATOMIC_RELAXED); +__atomic_fetch_add (&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);return 0 ; }
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type =WRITE_NO_KEY_SHIFT ;uint8_t read_type =READ_PASS_NO_KEY_SHIFTS ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - no key-shifts, read - hit\n" );
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_no_ks_r_hit[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t read_type =READ_FAIL ;int ret; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - no key-shifts, Hash lookup - miss\n" );
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_no_ks_r_miss[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,uint8_t write_type;uint8_t read_type =READ_PASS_NON_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err; printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_nsp[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type;uint8_t read_type =READ_PASS_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err; printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf, goto finish;printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); + +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_sp[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, intuint8_t write_type;uint8_t read_type =READ_FAIL ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - key shift, Hash lookup - miss\n" );
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, intprintf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup); }
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type;uint8_t read_type =READ_PASS_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Multi-add-lookup\n" );
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , + __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) - /rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, + __ATOMIC_RELAXED) / +__atomic_load_n (&greads, + __ATOMIC_RELAXED); rwc_perf_results->multi_rw[m][k][n] = cycles_per_lookup; printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,uint8_t write_type;uint8_t read_type =READ_PASS_KEY_SHIFTS_EXTBKT ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n" );
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); write_type =WRITE_NO_KEY_SHIFT ;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_extbkt[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
Signature · 1 line
2.17.1
PATCH v4 3/3 test/hash: add tests for integrated RCU QSBR
In reply to Dharmik ThakkarAdd functional and performance tests for the integrated RCU QSBR.
--- app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 171 ++++++++++- 2 files changed, 557 insertions(+), 4 deletions(-)
app/test/test_hash.c +389 -1
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) -#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ +#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ if (cond) { \ printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ if (handle) rte_fbk_hash_free(handle); \
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) +#define RETURN_IF_ERROR_RCU_QSBR (cond, str, ...) do { \ + if (cond) { \ + printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ + if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) { \ + writer_done = 1; \ +/* Wait until reader exited. */ \ +rte_eal_mp_wait_lcore (); \ + } \ +if (g_handle)rte_hash_free (g_handle); \ +if (g_qsv)rte_free (g_qsv); \ +return -1 ; \ + } \ +} while(0) +/* 5-tuple key type */ struct flow_key {uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)return ret; } +static struct rte_hash *g_handle; +static struct rte_rcu_qsbr *g_qsv; +static volatile uint8_t writer_done; +struct flow_key g_rand_keys[9 ]; +/* +* rte_hash_rcu_qsbr_add positive and negative tests. +* - Add RCU QSBR variable to Hash +* - Add another RCU QSBR variable to Hash +* - Check returns +*/ +static int +test_hash_rcu_qsbr_add (void ) +{ +size_t sz; +struct rte_rcu_qsbr *qsv2 =NULL ; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + +struct rte_hash_parameters params; + +printf ("\n# Running RCU QSBR add tests\n" ); +memcpy (¶ms, &ut_params,sizeof (params)); + params.name ="test_hash_rcu_qsbr_add" ; + params.extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF | +RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD ; + g_handle =rte_hash_create (¶ms); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; +/* Invalid QSBR mode */ + rcu_cfg.mode =2 ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status ==0 ,"Invalid QSBR mode test failed" ); + + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Create and attach another RCU QSBR to hash table */ + qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (qsv2 ==NULL , +"RCU QSBR variable creation failed" ); + + rcu_cfg.v = qsv2; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +rte_free (qsv2); +RETURN_IF_ERROR_RCU_QSBR (status ==0 , +"Attach RCU QSBR to hash table succeeded where failure" +" is expected" ); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; +} + +/* +* rte_hash_rcu_qsbr_add DQ mode functional test. +* Reader and writer are in the same thread in this test. +* - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries +* - Add RCU QSBR variable to hash +* - Add 8 hash entries and fill the bucket +* - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt +* - Register a reader thread (not a real thread) +* - Reader lookup existing entry +* - Writer deletes the entry +* - Reader lookup the entry +* - Writer re-add the entry (no available free index) +* - Reader report quiescent state and unregister +* - Writer re-add the entry +* - Reader lookup the entry +*/ +static int +test_hash_rcu_qsbr_dq_mode (uint8_t ext_bkt) +{ +uint32_t total_entries = (ext_bkt ==0 ) ?8 :9 ; + +uint8_t hash_extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ; + +if (ext_bkt) + hash_extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; + +struct rte_hash_parameters params_pseudo_hash = { + .name ="test_hash_rcu_qsbr_dq_mode" , + .entries = total_entries, + .key_len =sizeof (struct flow_key ),/* 13 */ + .hash_func = pseudo_hash, + .hash_func_init_val =0 , + .socket_id =0 , + .extra_flag = hash_extra_flag, + }; +int pos[total_entries]; +int expected_pos[total_entries]; +unsigned i; +size_t sz; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + + g_qsv =NULL ; + g_handle =NULL ; + +for (i =0 ; i < total_entries; i++) { + g_rand_keys[i].port_dst = i; + g_rand_keys[i].port_src = i+1 ; + } + +if (ext_bkt) +printf ("\n# Running RCU QSBR DQ mode functional test with" +" ext bkt\n" ); +else +printf ("\n# Running RCU QSBR DQ mode functional test\n" ); + + g_handle =rte_hash_create (¶ms_pseudo_hash); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Fill bucket */ +for (i =0 ; i < total_entries; i++) { + pos[i] =rte_hash_add_key (g_handle, &g_rand_keys[i]); +print_key_info ("Add" , &g_rand_keys[i], pos[i]); +RETURN_IF_ERROR_RCU_QSBR (pos[i] <0 , +"failed to add key (pos[%u]=%d)" , i, + pos[i]); + expected_pos[i] = pos[i]; + } + +/* Register pseudo reader */ + status =rte_rcu_qsbr_thread_register (g_qsv,0 ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR thread registration failed" ); +rte_rcu_qsbr_thread_online (g_qsv,0 ); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to find correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +/* Writer update */ + pos[0 ] =rte_hash_del_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Del" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to del correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != -ENOENT , +"found deleted key (pos[%u]=%d)" ,0 , pos[0 ]); + +/* Fill bucket */ + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != -ENOSPC , +"Added key successfully (pos[%u]=%d)" ,0 , pos[0 ]); + +/* Reader quiescent */ +rte_rcu_qsbr_quiescent (g_qsv,0 ); + +/* Fill bucket */ + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] <0 , +"failed to add key (pos[%u]=%d)" ,0 , pos[0 ]); + expected_pos[0 ] = pos[0 ]; + +rte_rcu_qsbr_thread_offline (g_qsv,0 ); + (void )rte_rcu_qsbr_thread_unregister (g_qsv,0 ); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to find correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +rte_hash_free (g_handle); +rte_free (g_qsv); +return 0 ; + +} + +/* Report quiescent state interval every 1024 lookups. Larger critical +* sections in reader will result in writer polling multiple times. +*/ +#define QSBR_REPORTING_INTERVAL 1024 +#define WRITER_ITERATIONS 512 + +/* +* Reader thread using rte_hash data structure with RCU. +*/ +static int +test_hash_rcu_qsbr_reader (void *arg) +{ +int i; + +RTE_SET_USED (arg); +/* Register this thread to report quiescent state */ + (void )rte_rcu_qsbr_thread_register (g_qsv,0 ); +rte_rcu_qsbr_thread_online (g_qsv,0 ); + +do { +for (i =0 ; i <QSBR_REPORTING_INTERVAL ; i++) +rte_hash_lookup (g_handle, &g_rand_keys[0 ]); + +/* Update quiescent state */ +rte_rcu_qsbr_quiescent (g_qsv,0 ); + }while (!writer_done); + +rte_rcu_qsbr_thread_offline (g_qsv,0 ); + (void )rte_rcu_qsbr_thread_unregister (g_qsv,0 ); + +return 0 ; +} + +/* +* rte_hash_rcu_qsbr_add sync mode functional test. +* 1 Reader and 1 writer. They cannot be in the same thread in this test. +* - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries +* - Add RCU QSBR variable to hash +* - Register a reader thread. Reader keeps looking up a specific key. +* - Writer keeps adding and deleting a specific key. +*/ +static int +test_hash_rcu_qsbr_sync_mode (uint8_t ext_bkt) +{ +uint32_t total_entries = (ext_bkt ==0 ) ?8 :9 ; + +uint8_t hash_extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ; + +if (ext_bkt) + hash_extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; + +struct rte_hash_parameters params_pseudo_hash = { + .name ="test_hash_rcu_qsbr_sync_mode" , + .entries = total_entries, + .key_len =sizeof (struct flow_key ),/* 13 */ + .hash_func = pseudo_hash, + .hash_func_init_val =0 , + .socket_id =0 , + .extra_flag = hash_extra_flag, + }; +int pos[total_entries]; +int expected_pos[total_entries]; +unsigned i; +size_t sz; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + + g_qsv =NULL ; + g_handle =NULL ; + +for (i =0 ; i < total_entries; i++) { + g_rand_keys[i].port_dst = i; + g_rand_keys[i].port_src = i+1 ; + } + +if (ext_bkt) +printf ("\n# Running RCU QSBR sync mode functional test with" +" ext bkt\n" ); +else +printf ("\n# Running RCU QSBR sync mode functional test\n" ); + + g_handle =rte_hash_create (¶ms_pseudo_hash); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Launch reader thread */ +rte_eal_remote_launch (test_hash_rcu_qsbr_reader,NULL , +rte_get_next_lcore (-1 ,1 ,0 )); + +/* Fill bucket */ +for (i =0 ; i < total_entries; i++) { + pos[i] =rte_hash_add_key (g_handle, &g_rand_keys[i]); +print_key_info ("Add" , &g_rand_keys[i], pos[i]); +RETURN_IF_ERROR_RCU_QSBR (pos[i] <0 , +"failed to add key (pos[%u]=%d)" , i, pos[i]); + expected_pos[i] = pos[i]; + } + writer_done =0 ; + +/* Writer Update */ +for (i =0 ; i <WRITER_ITERATIONS ; i++) { + expected_pos[0 ] = pos[0 ]; + pos[0 ] =rte_hash_del_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Del" , &g_rand_keys[0 ], status); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to del correct key (pos[%u]=%d)" + ,0 , pos[0 ]); + + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] <0 , +"failed to add key (pos[%u]=%d)" ,0 , + pos[0 ]); + } + + writer_done =1 ; +/* Wait until reader exited. */ +rte_eal_mp_wait_lcore (); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; + +} +/* * Do all unit and performance tests. */
@@ -1862,6 +2235,21 @@ test_hash(void)if (test_crc32_hash_alg_equiv () <0 )return -1 ; +if (test_hash_rcu_qsbr_add () <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_dq_mode (0 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_dq_mode (1 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_sync_mode (0 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_sync_mode (1 ) <0 ) +return -1 ; +return 0 ; }
app/test/test_hash_readwrite_lf_perf.c +168 -3
225 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v4 0/3] hash: integrate RCU QSBR
In reply to Dharmik ThakkarQuoted 12 lines · Dharmik Thakkar
Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
Quoted 7 lines
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Functional tests and performance tests are added to cover the integration with RCU.
This patch breaks Windows build: hash library was building fine so far, adding a dependency on rcu broke it as rcu is not ready (yet?).
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH v4 0/3] hash: integrate RCU QSBR
Quoted 19 lines · David Marchand
Quoted 14 lines · Dharmik Thakkar
Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
Quoted 9 lines
Quoted 5 lines
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Functional tests and performance tests are added to cover the integration with RCU.
This patch breaks Windows build: hash library was building fine so far, adding a dependency on rcu broke it as rcu is not ready (yet?).
Yes, ‘rcu’ is not currently enabled for Windows. I have submitted a RFC patch [1] to enable ‘rcu’ for Windows. I will update this patch, once ‘rcu’ support is enabled and the compilation succeeds on Windows.
[1] https://patches.dpdk.org/patch/81405/
Quoted 3 lines
Signature · 2 lines
-- David Marchand
v5 cover hash: integrate RCU QSBR
In reply to Dharmik ThakkarIntegrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.
Functional tests and performance tests are added to cover the integration with RCU.
---
v5:
- Enable 'rcu' for Windows
v4:
- Fix clang compilation issues
v3:
- Add documentation
- Add unit tests
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead
- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted
Dharmik Thakkar (4):
rcu: build on Windows
lib/hash: integrate RCU QSBR
test/hash: replace rte atomic with C11 atomic APIs test/hash: add tests for integrated RCU QSBR
app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 259 ++++++++++++---- doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 302 ++++++++++++++----- lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 ++++- lib/librte_hash/version.map | 2 +- lib/meson.build | 1 + 9 files changed, 918 insertions(+), 133 deletions(-)
Signature · 1 line
2.17.1
PATCH v5 1/4 rcu: build on Windows
Build the lib for Windows.
--- lib/meson.build | 1 + 1 file changed, 1 insertion(+)
lib/meson.build +1 -0
@@ -41,6 +41,7 @@ if is_windows 'telemetry', 'eal', 'ring', + 'rcu', 'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci', 'cmdline', 'hash',
Signature · 1 line
2.17.1
PATCH v5 2/4 lib/hash: integrate RCU QSBR
In reply to Dharmik ThakkarCurrently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.
--- doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 302 ++++++++++++++++++++++------- lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 +++++++- lib/librte_hash/version.map | 2 +- 6 files changed, 325 insertions(+), 76 deletions(-)
doc/guides/prog_guide/hash_lib.rst +8 -3
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values * If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position. Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position. + RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature. + Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details. + Extendable Bucket Functionality support ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default). in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the hash table size and can't tolerate any key insertion failure (even if very few). -Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and -deleted keys, to maintain the 100% capacity guarantee. +Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR +(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee. Implementation Details (non Extendable Bucket Case) ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion: Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency', the position of the key is not freed. It is the responsibility of the user to free the position after -readers are not referencing the position anymore. +readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below: * Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set, wait till the readers are not referencing the position returned during add/delete flow and then free the position. RCU mechanisms can be used to find out when the readers are not referencing the position anymore. + RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature. + Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details. * Lookup flow: Lookup for the flow key in the hash. If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
lib/librte_hash/meson.build +1 -0
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h', sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c') deps += ['ring'] +deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +234 -68
418 lines not on this page · raw message
lib/librte_hash/rte_cuckoo_hash.h +8 -0
@@ -168,6 +168,11 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +/* RCU config */ +struct rte_hash_rcu_config *hash_rcu_cfg; +/**< HASH RCU QSBR configuration structure */ +struct rte_rcu_qsbr_dq *dq;/**< RCU QSBR defer queue. */ +/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node { int prev_slot; /* Parent(slot) in search path */ }; +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_HASH_RCU_DQ_RECLAIM_MAX 16 +#endif
lib/librte_hash/rte_hash.h +73 -4
125 lines not on this page · raw message
lib/librte_hash/version.map +1 -1
@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; };
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR
Quoted 43 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 2/4] lib/hash: integrate RCU QSBRCurrently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>--- doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 302 ++++++++++++++++++++++------- lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 +++++++- lib/librte_hash/version.map | 2 +- 6 files changed, 325 insertions(+), 76 deletions(-)doc/guides/prog_guide/hash_lib.rst +0
-0@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads andwrites the following flag values
* If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL)
flag is set, the position of the entry in the hash table is not freed upon callingdelete(). This flag is enabled
by default when the lock free read/write concurrency flag is set. The
Quoted 8 lines
application should free the position after all the readers have stopped referencing the position.
Where required, the application can make use of RCU mechanisms to
determine when the readers have stopped referencing the position.
+ RCU QSBR process is integrated within the Hash library for safe freeing of
the position. Application has certain responsibilities while using this feature.
+ Please refer to resource reclamation framework of :ref:`RCU library
<RCU_Library>` for more details.
[Yipeng]: Maybe also add: rte_hash_rcu_qsbr_add() need to be called to use the embedded RCU mechanism. Just to give user a pointer to which API to look.
Quoted 120 lines
doc/guides/prog_guide/hash_lib.rst +1
-0+ Extendable Bucket Functionality support ----------------------------------------@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flagis not set by default).
in the very unlikely case due to excessive hash collisions that a key has failed
to be inserted, the hash table bucket is extended with a linked list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and -deleted keys, to maintain the 100% capacity guarantee.doc/guides/prog_guide/hash_lib.rst +2
-0+Please note that with the 'lock free read/write concurrency' flag +enabled, users need to call 'rte_hash_free_key_with_position' API orconfigure integrated RCU QSBR (or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
Implementation Details (non Extendable Bucket Case)
---------------------------------------------------doc/guides/prog_guide/hash_lib.rst +0
-0@@ -172,7 +175,7 @@ Example of deletion: Similar to lookup, the key is searched in its primary and secondary buckets. Ifthe key is found, the entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency', the position of the key is not freed. It is the responsibility of the user to free the position after -readers are not referencing the position anymore.
doc/guides/prog_guide/hash_lib.rst +3
-0+readers are not referencing the position anymore. User can configure +integrated RCU QSBR or use external RCU mechanisms to safely free the +position on delete Implementation Details (with Extendable Bucket) @@ -286,6 +289,8 @@ Theflow table operations on the application side are described below:
* Free flow: Free flow key position. If 'no free on delete' or 'lock-free
read/write concurrency' flags are set,
wait till the readers are not referencing the position returned during
add/delete flow and then free the position.
RCU mechanisms can be used to find out when the readers are not
referencing the position anymore.
+ RCU QSBR process is integrated within the Hash library for safe freeing of
the position. Application has certain responsibilities while using this feature.
+ Please refer to resource reclamation framework of :ref:`RCU library
<RCU_Library>` for more details.* Lookup flow: Lookup for the flow key in the hash.
If the returned position is valid (flow lookup hit), use the returned position
to access the flow entry in the flow table.lib/librte_hash/meson.build index renamed from lib/librte_hash/meson.build +0
-0
6ab46ae9d768..0977a63fd279100644lib/librte_hash/meson.build +1
-0@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h', sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c') deps += ['ring'] +deps += ['rcu']lib/librte_hash/rte_cuckoo_hash.c +4
-0@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = { };EAL_REGISTER_TAILQ (rte_hash_tailq) +struct __rte_hash_rcu_dq_entry { +uint32_t key_idx; +uint32_t ext_bkt_idx;/**< Extended bkt index */ }; +struct rte_hash * rte_hash_find_existing(const char *name) { @@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params) if (params->extra_flag &
RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {lib/librte_hash/rte_cuckoo_hash.c +18
-1readwrite_concur_lf_support =1 ; -/* Enable not freeing internal memory/index on delete */ +/* Enable not freeing internal memory/index on delete. +* If internal RCU is enabled, freeing of internal memory/index +* is done on delete +*/ no_free_on_del =1 ; }@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)rte_mcfg_tailq_write_unlock (); +/* RCU clean up. */ +if (h->dq) +rte_rcu_qsbr_dq_delete (h->dq); +if (h->use_local_cache)rte_free (h->local_free_slots);if (h->writer_takes_lock)@@ -607,11 +619,21 @@ void rte_hash_reset(struct rte_hash *h) {uint32_t tot_ring_cnt, i; +unsigned int pending;if (h ==NULL )return ;__hash_rw_writer_lock (h); + +/* RCU QSBR clean up. */ +if (h->dq) { +/* Reclaim all the resources */ +rte_rcu_qsbr_dq_reclaim (h->dq, ~0 ,NULL , &pending,NULL ); +if (pending !=0 ) +RTE_LOG (ERR ,HASH ,"RCU reclaim all resources failed\n"); + } +memset (h->buckets,0 , h->num_buckets *sizeof (struct rte_hash_bucket ));lib/librte_hash/rte_cuckoo_hash.c +0
-0memset (h->key_store,0 , h->key_entry_size * (h->entries +1 )); *h->tbl_chng_cnt =0 ;
lib/librte_hash/rte_cuckoo_hash.c rte_hash_cuckoo_make_space_mw(const struct · line 982
Quoted 9 lines
lib/librte_hash/rte_cuckoo_hash.c +5
-0@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h, return -ENOSPC ; } +static inline uint32_t +alloc_slot (const struct rte_hash *h,struct lcore_cache +*cached_free_slots) { +unsigned int n_slots; +uint32_t slot_id;
[Yipeng]: Blank line after variable declaration.
Quoted 312 lines
lib/librte_hash/rte_cuckoo_hash.c +4
-0+if (h->use_local_cache) { +/* Try to get a free slot from the local cache */ +if (cached_free_slots->len ==0 ) { +/* Need to get another burst of free slots from global ring */
lib/librte_hash/rte_cuckoo_hash.c +1
-0+ n_slots = rte_ring_mc_dequeue_burst_elem(h-Quoted 1 line
free_slots,lib/librte_hash/rte_cuckoo_hash.c +21
-1+ cached_free_slots->objs, +sizeof (uint32_t ), + LCORE_CACHE_SIZE, NULL); +if (n_slots ==0 ) +return EMPTY_SLOT ; + + cached_free_slots->len += n_slots; + } + +/* Get a free slot from the local cache */ + cached_free_slots->len--; + slot_id = cached_free_slots->objs[cached_free_slots->len]; + }else { +if (rte_ring_sc_dequeue_elem (h->free_slots, &slot_id, +sizeof (uint32_t )) !=0 ) +return EMPTY_SLOT ; + } + +return slot_id; +} +static inline int32_t __rte_hash_add_key_with_hash (const struct rte_hash *h,const void *key,hash_sig_t sig,void *data)@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, uint32_t ext_bkt_id = 0;uint32_t slot_id;int ret; -unsigned n_slots;unsigned lcore_id;unsigned int i;struct
lcore_cache*cached_free_slots= NULL; @@ -1001,28
+1053,20 @@ __rte_hash_add_key_with_hash(const structrte_hash*h, const void *key,lib/librte_hash/rte_cuckoo_hash.c +0
-3if (h->use_local_cache) { lcore_id =rte_lcore_id (); cached_free_slots = &h->local_free_slots[lcore_id]; -/* Try to get a free slot from the local cache */ -if (cached_free_slots->len ==0 ) { -/* Need to get another burst of free slots from global ring */
lib/librte_hash/rte_cuckoo_hash.c +0
-1- n_slots = rte_ring_mc_dequeue_burst_elem(h-Quoted 1 line
free_slots,lib/librte_hash/rte_cuckoo_hash.c +12
-16- cached_free_slots->objs, -sizeof (uint32_t ), - LCORE_CACHE_SIZE, NULL); -if (n_slots ==0 ) { -return -ENOSPC ; - } - - cached_free_slots->len += n_slots; + } + slot_id =alloc_slot (h, cached_free_slots); +if (slot_id ==EMPTY_SLOT ) { +if (h->dq) { +__hash_rw_writer_lock (h); + ret =rte_rcu_qsbr_dq_reclaim (h->dq, + h->hash_rcu_cfg->max_reclaim_size, +NULL ,NULL ,NULL ); +__hash_rw_writer_unlock (h); +if (ret ==0 ) + slot_id =alloc_slot (h, cached_free_slots); } - -/* Get a free slot from the local cache */ - cached_free_slots->len--; - slot_id = cached_free_slots->objs[cached_free_slots->len]; - }else { -if (rte_ring_sc_dequeue_elem (h->free_slots, &slot_id, -sizeof (uint32_t )) !=0 ) { +if (slot_id ==EMPTY_SLOT )return -ENOSPC ; - } } new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size); @@ -1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct
rte_hash*h, const void *key,lib/librte_hash/rte_cuckoo_hash.c +5
-2if (rte_ring_sc_dequeue_elem (h->free_ext_bkts, &ext_bkt_id,sizeof (uint32_t )) !=0 || ext_bkt_id ==0 ) { - ret = -ENOSPC ; - goto failure; +if (h->dq) { +if (rte_rcu_qsbr_dq_reclaim (h->dq, + h->hash_rcu_cfg->max_reclaim_size, +NULL ,NULL ,NULL ) ==0 ) { + rte_ring_sc_dequeue_elem(h-Quoted 1 line
free_ext_bkts,lib/librte_hash/rte_cuckoo_hash.c +8
-0+ &ext_bkt_id, + sizeof(uint32_t)); + } + } +if (ext_bkt_id ==0 ) { + ret = -ENOSPC ; + goto failure; + } } /* Use the first location of the new bucket */ @@ -1395,12 +1450,12@@
rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
return __rte_hash_lookup_with_hash(h, key,rte_hash_hash(h, key),
data); }lib/librte_hash/rte_cuckoo_hash.c +0
-2-static inline void -remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,unsigned i)
lib/librte_hash/rte_cuckoo_hash.c +4
-2+static int +free_slot (const struct rte_hash *h,uint32_t slot_id) {unsigned lcore_id, n_slots; -struct lcore_cache *cached_free_slots; - +struct lcore_cache *cached_free_slots =NULL ; +/* Return key indexes to free slot ring */ if (h->use_local_cache) { lcore_id =rte_lcore_id (); cached_free_slots = &h->local_free_slots[lcore_id]; @@ -1411,18 +1466,127 @@
remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)lib/librte_hash/rte_cuckoo_hash.c +0
-2cached_free_slots->objs,sizeof (uint32_t ), LCORE_CACHE_SIZE, NULL); - ERR_IF_TRUE((n_slots == 0), - "%s: could not enqueue free slots in globalring\n",
lib/librte_hash/rte_cuckoo_hash.c +24
-5- __func__); +RETURN_IF_TRUE ((n_slots ==0 ), -EFAULT ); cached_free_slots->len -= n_slots; } -/* Put index of new free slot in cache. */ - cached_free_slots->objs[cached_free_slots->len] = - bkt->key_idx[i]; - cached_free_slots->len++; + } + +enqueue_slot_back (h, cached_free_slots, slot_id); +return 0 ; +} + +static void +__hash_rcu_qsbr_free_resource (void *p,void *e,unsigned int n) { +void *key_data =NULL ; +int ret; +struct rte_hash_key *keys, *k; +struct rte_hash *h = (struct rte_hash *)p; +struct __rte_hash_rcu_dq_entry rcu_dq_entry = + *((struct __rte_hash_rcu_dq_entry *)e); + +RTE_SET_USED (n); + keys = h->key_store; + + k = (struct rte_hash_key *) ((char *)keys + + rcu_dq_entry.key_idx * h->key_entry_size); + key_data = k->pdata; +if (h->hash_rcu_cfg->free_key_data_func) + h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg-Quoted 1 line
key_data_ptr,lib/librte_hash/rte_cuckoo_hash.c +3
-0+ key_data); + + if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx !=
EMPTY_SLOT)lib/librte_hash/rte_cuckoo_hash.c +37
-0+/* Recycle empty ext bkt to free list. */ +rte_ring_sp_enqueue_elem (h->free_ext_bkts, + &rcu_dq_entry.ext_bkt_idx,sizeof (uint32_t )); + +/* Return key indexes to free slot ring */ + ret =free_slot (h, rcu_dq_entry.key_idx); +if (ret <0 ) { +RTE_LOG (ERR ,HASH , +"%s: could not enqueue free slots in global ring\n" , + __func__); + } +} + +int +rte_hash_rcu_qsbr_add (struct rte_hash *h, +struct rte_hash_rcu_config *cfg) +{ +struct rte_rcu_qsbr_dq_parameters params = {0 }; +char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE ]; +struct rte_hash_rcu_config *hash_rcu_cfg =NULL ; + +const uint32_t total_entries = h->use_local_cache ? + h->entries + (RTE_MAX_LCORE -1 ) * (LCORE_CACHE_SIZE -1 ) + 1 + : h->entries +1 ; + +if ((h ==NULL ) || cfg ==NULL || cfg->v ==NULL ) { + rte_errno =EINVAL ; +return 1 ; + } + +if (h->hash_rcu_cfg) { + rte_errno =EEXIST ; +return 1 ; + } + + hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config),0);
lib/librte_hash/rte_cuckoo_hash.c +18
-0+if (hash_rcu_cfg ==NULL ) { +RTE_LOG (ERR ,HASH ,"memory allocation failed\n" ); +return 1 ; + } + +if (cfg->mode ==RTE_HASH_QSBR_MODE_SYNC ) { +/* No other things to do. */ + }else if (cfg->mode ==RTE_HASH_QSBR_MODE_DQ ) { +/* Init QSBR defer queue. */ +snprintf (rcu_dq_name,sizeof (rcu_dq_name), +"HASH_RCU_%s" , h->name); + params.name = rcu_dq_name; + params.size = cfg->dq_size; +if (params.size ==0 ) + params.size = total_entries; + params.trigger_reclaim_limit = cfg->trigger_reclaim_limit; +if (params.max_reclaim_size ==0 ) + params.max_reclaim_size =
RTE_HASH_RCU_DQ_RECLAIM_MAX;lib/librte_hash/rte_cuckoo_hash.c +8
-0+ params.esize =sizeof (struct __rte_hash_rcu_dq_entry ); + params.free_fn = __hash_rcu_qsbr_free_resource; + params.p = h; + params.v = cfg->v; + h->dq =rte_rcu_qsbr_dq_create (¶ms); +if (h->dq ==NULL ) { +rte_free (hash_rcu_cfg); + RTE_LOG(ERR, HASH, "HASH defer queue creationfailed\n");
lib/librte_hash/rte_cuckoo_hash.c +30
-2+return 1 ; + } }else { -rte_ring_sp_enqueue_elem (h->free_slots, - &bkt->key_idx[i],sizeof (uint32_t )); +rte_free (hash_rcu_cfg); + rte_errno =EINVAL ; +return 1 ; + } + + hash_rcu_cfg->v = cfg->v; + hash_rcu_cfg->mode = cfg->mode; + hash_rcu_cfg->dq_size = params.size; + hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit; + hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size; + hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func; + hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr; + + h->hash_rcu_cfg = hash_rcu_cfg; + +return 0 ; +} + +static inline void +remove_entry (const struct rte_hash *h,struct rte_hash_bucket *bkt, +unsigned i) { +int ret =free_slot (h, bkt->key_idx[i]); +if (ret <0 ) { +RTE_LOG (ERR ,HASH , +"%s: could not enqueue free slots in global ring\n" , + __func__); } }@@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, int pos;int32_t ret, i;uint16_t short_sig; +uint32_t index =EMPTY_SLOT ; +struct __rte_hash_rcu_dq_entry rcu_dq_entry; short_sig =get_short_sig (sig);
prim_bucket_idx=get_prim_bucket_index(h, sig); @@ -1555,10
+1721,9 @@ __rte_hash_del_key_with_hash(const structrte_hash*h, const void *key,/* Search last bucket to see if empty to be recycled */
return_bkt:lib/librte_hash/rte_cuckoo_hash.c +5
-6-if (!last_bkt) { -__hash_rw_writer_unlock (h); -return ret; - } +if (!last_bkt) + goto return_key; +while (last_bkt->next) { prev_bkt = last_bkt; last_bkt = last_bkt->next;@@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,/* found empty bucket and recycle */ if (i == RTE_HASH_BUCKET_ENTRIES) { prev_bkt->next = NULL; -uint32_t index = last_bkt - h->buckets_ext +1 ; + index = last_bkt - h->buckets_ext +1 ;/* Recycle the empty bkt if * no_free_on_del is disabled. */ -if (h->no_free_on_del) +if (h->no_free_on_del) {/* Store index of an empty ext bkt to be recycled * on calling rte_hash_del_xxx APIs. lib/librte_hash/rte_cuckoo_hash.c +0
-0* When lock free read-write concurrency is enabled,
lib/librte_hash/rte_cuckoo_hash.c __rte_hash_del_key_with_hash(const struct · line 1755
Quoted 10 lines
lib/librte_hash/rte_cuckoo_hash.c +3
-2@@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const structrte_hash *h, const void *key, * immediately (as readers might be using it still). * Hence freeing of the ext bkt is piggy-backed to * freeing of the key index. +* If using external RCU, store this index in an array. */ - h->ext_bkt_to_free[ret] = index; - else +if (h->hash_rcu_cfg ==NULL ) + h->ext_bkt_to_free[ret] = index;
[Yipeng]: If using embedded qsbr (not NULL), how did you recycle the ext bkt?
Quoted 216 lines
lib/librte_hash/rte_cuckoo_hash.c +6
-0+ }else rte_ring_sp_enqueue_elem (h->free_ext_bkts, &index,sizeof (uint32_t )); } + +return_key: +/* Using internal RCU QSBR */ +if (h->hash_rcu_cfg) { +/* Key index where key is stored, adding the first dummy index */
lib/librte_hash/rte_cuckoo_hash.c +9
-0+ rcu_dq_entry.key_idx = ret +1 ; + rcu_dq_entry.ext_bkt_idx = index; +if (h->dq ==NULL ) { +/* Wait for quiescent state change if using +* RTE_HASH_QSBR_MODE_SYNC +*/ +rte_rcu_qsbr_synchronize (h->hash_rcu_cfg->v, +RTE_QSBR_THRID_INVALID ); + __hash_rcu_qsbr_free_resource((void*)((uintptr_t)h),
lib/librte_hash/rte_cuckoo_hash.c +3
-0+ &rcu_dq_entry, 1); + }else if (h->dq) +/* Push into QSBR FIFO if using
RTE_HASH_QSBR_MODE_DQ*/lib/librte_hash/rte_cuckoo_hash.c +1
-0+ if (rte_rcu_qsbr_dq_enqueue(h->dq,
&rcu_dq_entry) != 0)lib/librte_hash/rte_cuckoo_hash.c +1
-0+ RTE_LOG(ERR, HASH, "Failed to push QSBRFIFO\n");
lib/librte_hash/rte_cuckoo_hash.c +4
-23+ }__hash_rw_writer_unlock (h);return ret; }@@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,RETURN_IF_TRUE (((h ==NULL ) || (key_idx ==EMPTY_SLOT )), -EINVAL ); -unsigned int lcore_id, n_slots; -struct lcore_cache *cached_free_slots;const uint32_t total_entries = h->use_local_cache ? h->entries + (RTE_MAX_LCORE -1 ) * (LCORE_CACHE_SIZE -1 ) + 1 : h->entries +1 ;@@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const structrte_hash *h, } } -if (h->use_local_cache) { - lcore_id =rte_lcore_id (); - cached_free_slots = &h->local_free_slots[lcore_id]; -/* Cache full, need to free it. */ -if (cached_free_slots->len ==LCORE_CACHE_SIZE ) { -/* Need to enqueue the free slots in global ring. */ - n_slots =rte_ring_mp_enqueue_burst_elem (h- >free_slots, - cached_free_slots->objs, -sizeof (uint32_t ), -LCORE_CACHE_SIZE ,NULL ); -RETURN_IF_TRUE ((n_slots ==0 ), -EFAULT ); - cached_free_slots->len -= n_slots; - } -/* Put index of new free slot in cache. */ - cached_free_slots->objs[cached_free_slots->len] = key_idx; - cached_free_slots->len++; - }else { -rte_ring_sp_enqueue_elem (h->free_slots, &key_idx, -sizeof (uint32_t )); - } +/* Enqueue slot to cache/ring of free slots. */ + return free_slot(h, key_idx); -return 0 ; }static inline void
lib/librte_hash/rte_cuckoo_hash.h +8
-0@@ -168,6 +168,11 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +/* RCU config */ +struct rte_hash_rcu_config *hash_rcu_cfg; +/**< HASH RCU QSBR configuration structure */ +struct rte_rcu_qsbr_dq *dq;/**< RCU QSBR defer queue. */ +/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;@@ -230,4 +235,7 @@ struct queue_node { int prev_slot; /* Parent(slot) in search path */ }; +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_HASH_RCU_DQ_RECLAIM_MAX 16 +#endif lib/librte_hash/rte_hash.h index renamed from lib/librte_hash/rte_hash.h +0
-0
bff40251bc98..3d28f177f14a100644lib/librte_hash/rte_hash.h +1
-0@@ -15,6 +15,7 @@#include <stddef.h> #include <rte_compat.h> +#include <rte_rcu_qsbr.h> #ifdef __cplusplusextern "C" {@@ -45,7 +46,8 @@ extern "C" {/** Flag to disable freeing of key index on hash delete. * Refer to rte_hash_del_xxx APIs for more details. * This is enabled by default when
RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
*/
#defineRTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL0x10lib/librte_hash/rte_hash.h +8
-0@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len, /** Type of function used to compare the hash key. */ typedef int (*rte_hash_cmp_eq_t)(const void *key1,const void *key2,size_t key_len); +/** +* Type of function used to free data stored in the key. +* Required when using internal RCU to allow application to free +key-data once +* the key is returned to the the ring of free key-slots. +*/ +typedef void (*rte_hash_free_key_data )(void *p,void *key_data); +/** * Parameters used when creating the hash table. */ @@ -81,6 +90,39 @@ struct rte_hash_parameters {uint8_t extra_flag;/**< Indicate if additional parameters are present. */
};lib/librte_hash/rte_hash.h +20
-0+/** RCU reclamation modes */ +enum rte_hash_qsbr_mode { +/** Create defer queue for reclaim. */ +RTE_HASH_QSBR_MODE_DQ =0 , +/** Use blocking mode reclaim. No defer queue created. */ +RTE_HASH_QSBR_MODE_SYNC +}; + +/** HASH RCU QSBR configuration structure. */ struct +rte_hash_rcu_config { +struct rte_rcu_qsbr *v;/**< RCU QSBR variable. */ +enum rte_hash_qsbr_mode mode; +/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx +* '0' for default: create defer queue for reclaim. +*/ +uint32_t dq_size; +/**< RCU defer queue size. +* default: total hash table entries. +*/ +uint32_t trigger_reclaim_limit;/**< Threshold to trigger auto reclaim. */
lib/librte_hash/rte_hash.h +14
-1+uint32_t max_reclaim_size; +/**< Max entries to reclaim in one go. +* default: RTE_HASH_RCU_DQ_RECLAIM_MAX. +*/ +void *key_data_ptr; +/**< Pointer passed to the free function. Typically, this is the +* pointer to the data structure to which the resource to free +* (key-data) belongs. This can be NULL. +*/ +rte_hash_free_key_data free_key_data_func; +/**< Function to call to free the resource (key-data). */ }; +/** @internal A hash table structure. */ struct rte_hash ;@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t * Thread safety can be enabled by setting flag during * table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called* additionally to free the index associated with the key.
lib/librte_hash/rte_hash.h +2
-1@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key); * Thread safety can be enabled by setting flag during * table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called* additionally to free the index associated with the key.
lib/librte_hash/rte_hash.h +2
-1@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position, * only be called from one thread by default. Thread safety * can be enabled by setting flag during table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_del_key_xxx APIs must be freed * using this API. This API should be called after all the readers* have stopped referencing the entry corresponding to this key.
lib/librte_hash/rte_hash.h +0
-0@@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, · line 680
Quoted 10 lines
lib/librte_hash/rte_hash.h +6
-0*/ int32_t rte_hash_iterate (const struct rte_hash *h,const void **key,void **data,uint32_t *next); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Associate RCU QSBR variable with an Hash object.
[Yipeng]: a Hash object
Quoted 34 lines
lib/librte_hash/rte_hash.h +19
-0+* This API should be called to enable the integrated RCU QSBR support +and +* should be called immediately after creating the Hash object. +* +* @param h +* the hash object to add RCU QSBR +* @param cfg +* RCU QSBR configuration +* @return +* On success - 0 +* On error - 1 with error code set in rte_errno. +* Possible rte_errno codes are: +* - EINVAL - invalid pointer +* - EEXIST - already added QSBR +* - ENOMEM - memory allocation failure +*/ +__rte_experimental +int rte_hash_rcu_qsbr_add(struct rte_hash *h, +struct rte_hash_rcu_config *cfg); #ifdef __cplusplus }#endif lib/librte_hash/version.map index renamed from lib/librte_hash/version.map +0
-0
c0db81014ff9..c6d73080f478100644lib/librte_hash/version.map +1
-2@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; }; --
Quoted 1 line
2.17.1
[Yipeng]:
Hi, Dharmik, thanks for the work! It generally looks good.
Just some minor issues to address and one question for the ext table inlined.
Thanks!
Re: [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR
Quoted 50 lines · On Oct 20, 2020, at 9:42 PM, Wang, Yipeng1 <[email protected]> wrote:
Quoted 47 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 2/4] lib/hash: integrate RCU QSBRCurrently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>---
doc/guides/prog_guide/hash_lib.rst | 11 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 302 ++++++++++++++++++++++------- lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 +++++++- lib/librte_hash/version.map | 2 +- 6 files changed, 325 insertions(+), 76 deletions(-)doc/guides/prog_guide/hash_lib.rst +0
-0@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads andwrites the following flag values
* If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon callingdelete(). This flag is enabled
by default when the lock free read/write concurrency flag is set. The
application should free the position after all the readers have stopped referencing the position.
Where required, the application can make use of RCU mechanisms to
Quoted 8 lines
Quoted 5 lines
determine when the readers have stopped referencing the position.
+ RCU QSBR process is integrated within the Hash library for safe freeing of
the position. Application has certain responsibilities while using this feature.
+ Please refer to resource reclamation framework of :ref:`RCU library
<RCU_Library>` for more details.[Yipeng]: Maybe also add:
rte_hash_rcu_qsbr_add()need to be called to use the embedded RCU mechanism. Just to give user a pointer to which API to look.
Copy.
Quoted 124 lines
Quoted 124 lines
+
Extendable Bucket Functionality support
----------------------------------------doc/guides/prog_guide/hash_lib.rst +0
-0@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flagis not set by default).
in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and -deleted keys, to maintain the 100% capacity guarantee.doc/guides/prog_guide/hash_lib.rst +2
-0+Please note that with the 'lock free read/write concurrency' flag +enabled, users need to call 'rte_hash_free_key_with_position' API orconfigure integrated RCU QSBR (or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
Implementation Details (non Extendable Bucket Case)
---------------------------------------------------doc/guides/prog_guide/hash_lib.rst +0
-0@@ -172,7 +175,7 @@ Example of deletion:Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency', the position of the key is not freed. It is the responsibility of the user to free the position after -readers are not referencing the position anymore.
doc/guides/prog_guide/hash_lib.rst +3
-0+readers are not referencing the position anymore. User can configure +integrated RCU QSBR or use external RCU mechanisms to safely free the +position on deleteImplementation Details (with Extendable Bucket) @@ -286,6 +289,8 @@ The flow table operations on the application side are described below:
* Free flow: Free flow key position. If 'no free on delete' or 'lock-free
read/write concurrency' flags are set,
wait till the readers are not referencing the position returned during
add/delete flow and then free the position.
RCU mechanisms can be used to find out when the readers are not
referencing the position anymore.
+ RCU QSBR process is integrated within the Hash library for safe freeing of
the position. Application has certain responsibilities while using this feature.
+ Please refer to resource reclamation framework of :ref:`RCU library
<RCU_Library>` for more details.* Lookup flow: Lookup for the flow key in the hash.
If the returned position is valid (flow lookup hit), use the returned position
to access the flow entry in the flow table.lib/librte_hash/meson.build index renamed from lib/librte_hash/meson.build +0
-0
6ab46ae9d768..0977a63fd279100644lib/librte_hash/meson.build +0
-0@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',sources =
files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')deps += ['ring']lib/librte_hash/meson.build +1
-0+deps += ['rcu']lib/librte_hash/rte_cuckoo_hash.c +4
-0@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = { };EAL_REGISTER_TAILQ (rte_hash_tailq) +struct __rte_hash_rcu_dq_entry { +uint32_t key_idx; +uint32_t ext_bkt_idx;/**< Extended bkt index */ }; +struct
rte_hash*rte_hash_find_existing(const char *name){ @@ -210,7 +215,10 @@rte_hash_create(const struct rte_hash_parameters *params)if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) { readwrite_concur_lf_support = 1;lib/librte_hash/rte_cuckoo_hash.c +4
-1-/* Enable not freeing internal memory/index on delete */ +/* Enable not freeing internal memory/index on delete. +* If internal RCU is enabled, freeing of internal memory/index +* is done on delete +*/ no_free_on_del = 1; }lib/librte_hash/rte_cuckoo_hash.c +14
-0@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)rte_mcfg_tailq_write_unlock (); +/* RCU clean up. */ +if (h->dq) +rte_rcu_qsbr_dq_delete (h->dq); +if (h->use_local_cache)rte_free (h->local_free_slots);if (h->writer_takes_lock)@@ -607,11 +619,21 @@ void rte_hash_reset(struct rte_hash *h) {uint32_t tot_ring_cnt, i; +unsigned int pending;if (h ==NULL )return ;__hash_rw_writer_lock (h); + +/* RCU QSBR clean up. */ +if (h->dq) { +/* Reclaim all the resources */ +rte_rcu_qsbr_dq_reclaim (h->dq, ~0 ,NULL , &pending,NULL ); +if (pending !=0 ) +RTE_LOG (ERR ,HASH ,"RCU reclaim all resources failed\n"); + } +memset (h->buckets,0 , h->num_buckets *sizeof (struct rte_hash_bucket ));memset(h->key_store, 0, h->key_entry_size * (h->entries + 1)); *h->tbl_chng_cnt = 0;lib/librte_hash/rte_cuckoo_hash.c +0
-0@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h, return -ENOSPC ; }
Quoted 7 lines
lib/librte_hash/rte_cuckoo_hash.c rte_hash_cuckoo_make_space_mw(const struct rte_hash *h, · line 981
Quoted 5 lines
lib/librte_hash/rte_cuckoo_hash.c +5
-0+static inline uint32_t +alloc_slot (const struct rte_hash *h,struct lcore_cache +*cached_free_slots) { +unsigned int n_slots; +uint32_t slot_id;[Yipeng]: Blank line after variable declaration.
Copy.
Quoted 318 lines
Quoted 318 lines
lib/librte_hash/rte_cuckoo_hash.c +4
-0+if (h->use_local_cache) { +/* Try to get a free slot from the local cache */ +if (cached_free_slots->len ==0 ) { +/* Need to get another burst of free slots from global ring */
lib/librte_hash/rte_cuckoo_hash.c +1
-0+ n_slots = rte_ring_mc_dequeue_burst_elem(h-Quoted 1 line
free_slots,lib/librte_hash/rte_cuckoo_hash.c +21
-0+ cached_free_slots->objs, +sizeof (uint32_t ), + LCORE_CACHE_SIZE, NULL); +if (n_slots ==0 ) +return EMPTY_SLOT ; + + cached_free_slots->len += n_slots; + } + +/* Get a free slot from the local cache */ + cached_free_slots->len--; + slot_id = cached_free_slots->objs[cached_free_slots->len]; + }else { +if (rte_ring_sc_dequeue_elem (h->free_slots, &slot_id, +sizeof (uint32_t )) !=0 ) +return EMPTY_SLOT ; + } + +return slot_id; +} +static inline
int32_t
__rte_hash_add_key_with_hash(const structrte_hash*h, const void *key,
hash_sig_tsig, void *data)lib/librte_hash/rte_cuckoo_hash.c +0
-1@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, uint32_t ext_bkt_id = 0;uint32_t slot_id;int ret; -unsigned n_slots;unsigned lcore_id;unsigned int i;struct lcore_cache *cached_free_slots = NULL; @@ -1001,28 +1053,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, if (h->use_local_cache) { lcore_id = rte_lcore_id(); cached_free_slots = &h->local_free_slots[lcore_id];lib/librte_hash/rte_cuckoo_hash.c +0
-3-/* Try to get a free slot from the local cache */ -if (cached_free_slots->len ==0 ) { -/* Need to get another burst of free slots from global ring */
lib/librte_hash/rte_cuckoo_hash.c +0
-1- n_slots = rte_ring_mc_dequeue_burst_elem(h-Quoted 1 line
free_slots,lib/librte_hash/rte_cuckoo_hash.c +11
-8- cached_free_slots->objs, -sizeof (uint32_t ), - LCORE_CACHE_SIZE, NULL); -if (n_slots ==0 ) { -return -ENOSPC ; - } - - cached_free_slots->len += n_slots; + } + slot_id =alloc_slot (h, cached_free_slots); +if (slot_id ==EMPTY_SLOT ) { +if (h->dq) { +__hash_rw_writer_lock (h); + ret =rte_rcu_qsbr_dq_reclaim (h->dq, + h->hash_rcu_cfg->max_reclaim_size, +NULL ,NULL ,NULL ); +__hash_rw_writer_unlock (h); +if (ret ==0 ) + slot_id =alloc_slot (h, cached_free_slots);}lib/librte_hash/rte_cuckoo_hash.c +1
-7- -/* Get a free slot from the local cache */ - cached_free_slots->len--; - slot_id = cached_free_slots->objs[cached_free_slots->len]; - }else { -if (rte_ring_sc_dequeue_elem (h->free_slots, &slot_id, -sizeof (uint32_t )) !=0 ) { +if (slot_id ==EMPTY_SLOT )return -ENOSPC;lib/librte_hash/rte_cuckoo_hash.c +0
-1- }} new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size); @@ - 1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id, sizeof(uint32_t)) != 0 || ext_bkt_id == 0) {lib/librte_hash/rte_cuckoo_hash.c +5
-2- ret = -ENOSPC ; - goto failure; +if (h->dq) { +if (rte_rcu_qsbr_dq_reclaim (h->dq, + h->hash_rcu_cfg->max_reclaim_size, +NULL ,NULL ,NULL ) ==0 ) { + rte_ring_sc_dequeue_elem(h-Quoted 1 line
free_ext_bkts,lib/librte_hash/rte_cuckoo_hash.c +8
-0+ &ext_bkt_id, + sizeof(uint32_t)); + } + } +if (ext_bkt_id ==0 ) { + ret = -ENOSPC ; + goto failure; + }} /* Use the first location of the new bucket */ @@ -1395,12 +1450,12@@
rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
return __rte_hash_lookup_with_hash(h, key,rte_hash_hash(h, key),
data); }lib/librte_hash/rte_cuckoo_hash.c +0
-2-static inline void -remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,unsigned i)
lib/librte_hash/rte_cuckoo_hash.c +2
-0+static int +free_slot (const struct rte_hash *h,uint32_t slot_id){ unsigned lcore_id, n_slots;lib/librte_hash/rte_cuckoo_hash.c +2
-2-struct lcore_cache *cached_free_slots; - +struct lcore_cache *cached_free_slots =NULL ; +/* Return key indexes to free slot ring */ if (h->use_local_cache) { lcore_id = rte_lcore_id(); cached_free_slots = &h->local_free_slots[lcore_id]; @@ - 1411,18 +1466,127 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i) cached_free_slots->objs, sizeof(uint32_t), LCORE_CACHE_SIZE, NULL);lib/librte_hash/rte_cuckoo_hash.c +0
-2- ERR_IF_TRUE((n_slots == 0), - "%s: could not enqueue free slots in globalring\n",
lib/librte_hash/rte_cuckoo_hash.c +1
-1- __func__); +RETURN_IF_TRUE ((n_slots ==0 ), -EFAULT );cached_free_slots->len -= n_slots; }lib/librte_hash/rte_cuckoo_hash.c +23
-4-/* Put index of new free slot in cache. */ - cached_free_slots->objs[cached_free_slots->len] = - bkt->key_idx[i]; - cached_free_slots->len++; + } + +enqueue_slot_back (h, cached_free_slots, slot_id); +return 0 ; +} + +static void +__hash_rcu_qsbr_free_resource (void *p,void *e,unsigned int n) { +void *key_data =NULL ; +int ret; +struct rte_hash_key *keys, *k; +struct rte_hash *h = (struct rte_hash *)p; +struct __rte_hash_rcu_dq_entry rcu_dq_entry = + *((struct __rte_hash_rcu_dq_entry *)e); + +RTE_SET_USED (n); + keys = h->key_store; + + k = (struct rte_hash_key *) ((char *)keys + + rcu_dq_entry.key_idx * h->key_entry_size); + key_data = k->pdata; +if (h->hash_rcu_cfg->free_key_data_func) + h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg-Quoted 1 line
key_data_ptr,lib/librte_hash/rte_cuckoo_hash.c +3
-0+ key_data); + + if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx !=
EMPTY_SLOT)lib/librte_hash/rte_cuckoo_hash.c +37
-0+/* Recycle empty ext bkt to free list. */ +rte_ring_sp_enqueue_elem (h->free_ext_bkts, + &rcu_dq_entry.ext_bkt_idx,sizeof (uint32_t )); + +/* Return key indexes to free slot ring */ + ret =free_slot (h, rcu_dq_entry.key_idx); +if (ret <0 ) { +RTE_LOG (ERR ,HASH , +"%s: could not enqueue free slots in global ring\n" , + __func__); + } +} + +int +rte_hash_rcu_qsbr_add (struct rte_hash *h, +struct rte_hash_rcu_config *cfg) +{ +struct rte_rcu_qsbr_dq_parameters params = {0 }; +char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE ]; +struct rte_hash_rcu_config *hash_rcu_cfg =NULL ; + +const uint32_t total_entries = h->use_local_cache ? + h->entries + (RTE_MAX_LCORE -1 ) * (LCORE_CACHE_SIZE -1 ) + 1 + : h->entries +1 ; + +if ((h ==NULL ) || cfg ==NULL || cfg->v ==NULL ) { + rte_errno =EINVAL ; +return 1 ; + } + +if (h->hash_rcu_cfg) { + rte_errno =EEXIST ; +return 1 ; + } + + hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config),0);
lib/librte_hash/rte_cuckoo_hash.c +18
-0+if (hash_rcu_cfg ==NULL ) { +RTE_LOG (ERR ,HASH ,"memory allocation failed\n" ); +return 1 ; + } + +if (cfg->mode ==RTE_HASH_QSBR_MODE_SYNC ) { +/* No other things to do. */ + }else if (cfg->mode ==RTE_HASH_QSBR_MODE_DQ ) { +/* Init QSBR defer queue. */ +snprintf (rcu_dq_name,sizeof (rcu_dq_name), +"HASH_RCU_%s" , h->name); + params.name = rcu_dq_name; + params.size = cfg->dq_size; +if (params.size ==0 ) + params.size = total_entries; + params.trigger_reclaim_limit = cfg->trigger_reclaim_limit; +if (params.max_reclaim_size ==0 ) + params.max_reclaim_size =
RTE_HASH_RCU_DQ_RECLAIM_MAX;lib/librte_hash/rte_cuckoo_hash.c +8
-0+ params.esize =sizeof (struct __rte_hash_rcu_dq_entry ); + params.free_fn = __hash_rcu_qsbr_free_resource; + params.p = h; + params.v = cfg->v; + h->dq =rte_rcu_qsbr_dq_create (¶ms); +if (h->dq ==NULL ) { +rte_free (hash_rcu_cfg); + RTE_LOG(ERR, HASH, "HASH defer queue creationfailed\n");
lib/librte_hash/rte_cuckoo_hash.c +2
-0+return 1 ; + }} else {lib/librte_hash/rte_cuckoo_hash.c +26
-2-rte_ring_sp_enqueue_elem (h->free_slots, - &bkt->key_idx[i],sizeof (uint32_t )); +rte_free (hash_rcu_cfg); + rte_errno =EINVAL ; +return 1 ; + } + + hash_rcu_cfg->v = cfg->v; + hash_rcu_cfg->mode = cfg->mode; + hash_rcu_cfg->dq_size = params.size; + hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit; + hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size; + hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func; + hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr; + + h->hash_rcu_cfg = hash_rcu_cfg; + +return 0 ; +} + +static inline void +remove_entry (const struct rte_hash *h,struct rte_hash_bucket *bkt, +unsigned i) { +int ret =free_slot (h, bkt->key_idx[i]); +if (ret <0 ) { +RTE_LOG (ERR ,HASH , +"%s: could not enqueue free slots in global ring\n" , + __func__);} }lib/librte_hash/rte_cuckoo_hash.c +2
-0@@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, int pos;int32_t ret, i;uint16_t short_sig; +uint32_t index =EMPTY_SLOT ; +struct __rte_hash_rcu_dq_entry rcu_dq_entry; short_sig =get_short_sig (sig);prim_bucket_idx = get_prim_bucket_index(h, sig); @@ -1555,10+1721,9 @@ __rte_hash_del_key_with_hash(const struct
rte_hash*h, const void *key,/* Search last bucket to see if empty to be recycled */
return_bkt:lib/librte_hash/rte_cuckoo_hash.c +3
-4-if (!last_bkt) { -__hash_rw_writer_unlock (h); -return ret; - } +if (!last_bkt) + goto return_key; +while (last_bkt->next) { prev_bkt = last_bkt; last_bkt = last_bkt->next;lib/librte_hash/rte_cuckoo_hash.c +2
-2@@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,/* found empty bucket and recycle */ if (i == RTE_HASH_BUCKET_ENTRIES) { prev_bkt->next = NULL; -uint32_t index = last_bkt - h->buckets_ext +1 ; + index = last_bkt - h->buckets_ext +1 ;/* Recycle the empty bkt if * no_free_on_del is disabled. */ -if (h->no_free_on_del) +if (h->no_free_on_del) {/* Store index of an empty ext bkt to be recycled * on calling rte_hash_del_xxx APIs. * When lock free read-write concurrency is enabled,
lib/librte_hash/rte_cuckoo_hash.c +1
-0@@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, * immediately (as readers might be using it still). * Hence freeing of the ext bkt is piggy-backed to * freeing of the key index. + * If using external RCU, store this index in an array.
Quoted 8 lines
lib/librte_hash/rte_cuckoo_hash.c __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, · line 1754
Quoted 5 lines
lib/librte_hash/rte_cuckoo_hash.c +2
-2*/ - h->ext_bkt_to_free[ret] = index; - else +if (h->hash_rcu_cfg ==NULL ) + h->ext_bkt_to_free[ret] = index;[Yipeng]: If using embedded qsbr (not NULL), how did you recycle the ext bkt?
In DQ mode, ext_bkt_idx is enqueued to the defer queue along with the key_idx and recycled as part of rte_rcu_qsbr_dq_reclaim(). In SYNC mode, it is recycled after rte_rcu_qsbr_synchronize() succeeds.
Quoted 220 lines
Quoted 220 lines
lib/librte_hash/rte_cuckoo_hash.c +6
-0+ }else rte_ring_sp_enqueue_elem (h->free_ext_bkts, &index,sizeof (uint32_t )); } + +return_key: +/* Using internal RCU QSBR */ +if (h->hash_rcu_cfg) { +/* Key index where key is stored, adding the first dummy index */
lib/librte_hash/rte_cuckoo_hash.c +9
-0+ rcu_dq_entry.key_idx = ret +1 ; + rcu_dq_entry.ext_bkt_idx = index; +if (h->dq ==NULL ) { +/* Wait for quiescent state change if using +* RTE_HASH_QSBR_MODE_SYNC +*/ +rte_rcu_qsbr_synchronize (h->hash_rcu_cfg->v, +RTE_QSBR_THRID_INVALID ); + __hash_rcu_qsbr_free_resource((void*)((uintptr_t)h),
lib/librte_hash/rte_cuckoo_hash.c +3
-0+ &rcu_dq_entry, 1); + }else if (h->dq) +/* Push into QSBR FIFO if using
RTE_HASH_QSBR_MODE_DQ*/lib/librte_hash/rte_cuckoo_hash.c +1
-0+ if (rte_rcu_qsbr_dq_enqueue(h->dq,
&rcu_dq_entry) != 0)lib/librte_hash/rte_cuckoo_hash.c +1
-0+ RTE_LOG(ERR, HASH, "Failed to push QSBRFIFO\n");
lib/librte_hash/rte_cuckoo_hash.c +1
-0+ }__hash_rw_writer_unlock(h); return ret; }lib/librte_hash/rte_cuckoo_hash.c +1
-2@@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const structrte_hash *h, RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL); -unsigned int lcore_id, n_slots; -struct lcore_cache *cached_free_slots; const uint32_t total_entries = h->use_local_cache ? h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1: h->entries + 1;lib/librte_hash/rte_cuckoo_hash.c +2
-21@@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const structrte_hash *h, } } -if (h->use_local_cache) { - lcore_id =rte_lcore_id (); - cached_free_slots = &h->local_free_slots[lcore_id]; -/* Cache full, need to free it. */ -if (cached_free_slots->len ==LCORE_CACHE_SIZE ) { -/* Need to enqueue the free slots in global ring. */ - n_slots =rte_ring_mp_enqueue_burst_elem (h- > free_slots, - cached_free_slots->objs, -sizeof (uint32_t ), -LCORE_CACHE_SIZE ,NULL ); -RETURN_IF_TRUE ((n_slots ==0 ), -EFAULT ); - cached_free_slots->len -= n_slots; - } -/* Put index of new free slot in cache. */ - cached_free_slots->objs[cached_free_slots->len] = key_idx; - cached_free_slots->len++; - }else { -rte_ring_sp_enqueue_elem (h->free_slots, &key_idx, -sizeof (uint32_t )); - } +/* Enqueue slot to cache/ring of free slots. */ + return free_slot(h, key_idx); -return 0 ; }static inline void
lib/librte_hash/rte_cuckoo_hash.h +8
-0@@ -168,6 +168,11 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +/* RCU config */ +struct rte_hash_rcu_config *hash_rcu_cfg; +/**< HASH RCU QSBR configuration structure */ +struct rte_rcu_qsbr_dq *dq;/**< RCU QSBR defer queue. */ +/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;@@ -230,4 +235,7 @@ struct queue_node { int prev_slot; /* Parent(slot) in search path */ }; +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_HASH_RCU_DQ_RECLAIM_MAX 16 +#endif lib/librte_hash/rte_hash.h index renamed from lib/librte_hash/rte_hash.h +0
-0
bff40251bc98..3d28f177f14a100644lib/librte_hash/rte_hash.h +1
-0@@ -15,6 +15,7 @@#include <stddef.h> #include <rte_compat.h> +#include <rte_rcu_qsbr.h> #ifdef __cplusplusextern "C" {@@ -45,7 +46,8 @@ extern "C" {/** Flag to disable freeing of key index on hash delete. * Refer to rte_hash_del_xxx APIs for more details. * This is enabled by default when
RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
*/
#defineRTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL0x10lib/librte_hash/rte_hash.h +8
-0@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len, /** Type of function used to compare the hash key. */ typedef int (*rte_hash_cmp_eq_t)(const void *key1,const void *key2,size_t key_len); +/** +* Type of function used to free data stored in the key. +* Required when using internal RCU to allow application to free +key-data once +* the key is returned to the the ring of free key-slots. +*/ +typedef void (*rte_hash_free_key_data )(void *p,void *key_data); +/** * Parameters used when creating the hash table. */ @@ -81,6 +90,39 @@ struct rte_hash_parameters {uint8_t extra_flag;/**< Indicate if additional parameters are present. */
};lib/librte_hash/rte_hash.h +20
-0+/** RCU reclamation modes */ +enum rte_hash_qsbr_mode { +/** Create defer queue for reclaim. */ +RTE_HASH_QSBR_MODE_DQ =0 , +/** Use blocking mode reclaim. No defer queue created. */ +RTE_HASH_QSBR_MODE_SYNC +}; + +/** HASH RCU QSBR configuration structure. */ struct +rte_hash_rcu_config { +struct rte_rcu_qsbr *v;/**< RCU QSBR variable. */ +enum rte_hash_qsbr_mode mode; +/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx +* '0' for default: create defer queue for reclaim. +*/ +uint32_t dq_size; +/**< RCU defer queue size. +* default: total hash table entries. +*/ +uint32_t trigger_reclaim_limit;/**< Threshold to trigger auto reclaim. */
lib/librte_hash/rte_hash.h +12
-0+uint32_t max_reclaim_size; +/**< Max entries to reclaim in one go. +* default: RTE_HASH_RCU_DQ_RECLAIM_MAX. +*/ +void *key_data_ptr; +/**< Pointer passed to the free function. Typically, this is the +* pointer to the data structure to which the resource to free +* (key-data) belongs. This can be NULL. +*/ +rte_hash_free_key_data free_key_data_func; +/**< Function to call to free the resource (key-data). */ }; +/** @internal A hash table structure. */ struct
rte_hash;lib/librte_hash/rte_hash.h +2
-1@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t * Thread safety can be enabled by setting flag during * table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called* additionally to free the index associated with the key.
lib/librte_hash/rte_hash.h +2
-1@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key); * Thread safety can be enabled by setting flag during * table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_add_key_xxx APIs will not be * freed by this API. rte_hash_free_key_with_position API must be called* additionally to free the index associated with the key.
lib/librte_hash/rte_hash.h +2
-1@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position, * only be called from one thread by default. Thread safety * can be enabled by setting flag during table creation. * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or - * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled, + * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and + * internal RCU is NOT enabled, * the key index returned by rte_hash_del_key_xxx APIs must be freed * using this API. This API should be called after all the readers* have stopped referencing the entry corresponding to this key.
lib/librte_hash/rte_hash.h +0
-0@@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h,const void **keys, */
int32_t
rte_hash_iterate(const structrte_hash*h, const void **key, void **data,
Quoted 8 lines
lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, · line 678
Quoted 7 lines
uint32_t*next);lib/librte_hash/rte_hash.h +6
-0+ +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Associate RCU QSBR variable with an Hash object. [Yipeng]: a Hash object
Copy.
Quoted 38 lines
Quoted 35 lines
+ * This API should be called to enable the integrated RCU QSBR support
lib/librte_hash/rte_hash.h +18
-0+and +* should be called immediately after creating the Hash object. +* +* @param h +* the hash object to add RCU QSBR +* @param cfg +* RCU QSBR configuration +* @return +* On success - 0 +* On error - 1 with error code set in rte_errno. +* Possible rte_errno codes are: +* - EINVAL - invalid pointer +* - EEXIST - already added QSBR +* - ENOMEM - memory allocation failure +*/ +__rte_experimental +int rte_hash_rcu_qsbr_add(struct rte_hash *h, +struct rte_hash_rcu_config *cfg);#ifdef
__cplusplus
}
#endiflib/librte_hash/version.map index renamed from lib/librte_hash/version.map +0
-0
c0db81014ff9..c6d73080f478100644lib/librte_hash/version.map +1
-2@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; }; --Quoted 1 line
2.17.1
[Yipeng]:
Quoted 2 lines
Hi, Dharmik, thanks for the work! It generally looks good.
Just some minor issues to address and one question for the ext table inlined.
Thank you for the comments, Yipeng! I will update the patches.
Quoted 2 lines
Thanks!
PATCH v5 3/4 test/hash: replace rte atomic with C11 atomic APIs
In reply to Dharmik ThakkarReplace rte_atomic APIs with C11 atomic APIs intest_hash_readwrite_lf_perf.c
--- app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++--------------- 1 file changed, 36 insertions(+), 53 deletions(-)
app/test/test_hash_readwrite_lf_perf.c +36 -53
@@ -82,8 +82,8 @@ static struct {struct rte_hash *h; } tbl_rwc_test_param; -static rte_atomic64_t gread_cycles; -static rte_atomic64_t greads; +static uint64_t gread_cycles; +static uint64_t greads;static volatile uint8_t writer_done;
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg) }while (!writer_done); cycles =rte_rdtsc_precise () - begin; -rte_atomic64_add (&gread_cycles, cycles); -rte_atomic64_add (&greads, read_cnt*loop_cnt); +__atomic_fetch_add (&gread_cycles, cycles, __ATOMIC_RELAXED); +__atomic_fetch_add (&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);return 0 ; }
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type =WRITE_NO_KEY_SHIFT ;uint8_t read_type =READ_PASS_NO_KEY_SHIFTS ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - no key-shifts, read - hit\n" );
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_no_ks_r_hit[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t read_type =READ_FAIL ;int ret; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - no key-shifts, Hash lookup - miss\n" );
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_no_ks_r_miss[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,uint8_t write_type;uint8_t read_type =READ_PASS_NON_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err; printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_nsp[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type;uint8_t read_type =READ_PASS_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err; printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf, goto finish;printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); + +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_sp[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, intuint8_t write_type;uint8_t read_type =READ_FAIL ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - key shift, Hash lookup - miss\n" );
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, intprintf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup); }
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type;uint8_t read_type =READ_PASS_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Multi-add-lookup\n" );
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , + __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) - /rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, + __ATOMIC_RELAXED) / +__atomic_load_n (&greads, + __ATOMIC_RELAXED); rwc_perf_results->multi_rw[m][k][n] = cycles_per_lookup; printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,uint8_t write_type;uint8_t read_type =READ_PASS_KEY_SHIFTS_EXTBKT ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n" );
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); write_type =WRITE_NO_KEY_SHIFT ;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_extbkt[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs
Quoted 9 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs
Quoted 5 lines
Replace
rte_atomicAPIs with C11 atomic APIs intest_hash_readwrite_lf_perf.cSigned-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>---
PATCH v5 4/4 test/hash: add tests for integrated RCU QSBR
In reply to Dharmik ThakkarAdd functional and performance tests for the integrated RCU QSBR.
--- app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++- 2 files changed, 556 insertions(+), 4 deletions(-)
app/test/test_hash.c +389 -1
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) -#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ +#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ if (cond) { \ printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ if (handle) rte_fbk_hash_free(handle); \
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) +#define RETURN_IF_ERROR_RCU_QSBR (cond, str, ...) do { \ + if (cond) { \ + printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ + if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) { \ + writer_done = 1; \ +/* Wait until reader exited. */ \ +rte_eal_mp_wait_lcore (); \ + } \ +if (g_handle)rte_hash_free (g_handle); \ +if (g_qsv)rte_free (g_qsv); \ +return -1 ; \ + } \ +} while(0) +/* 5-tuple key type */ struct flow_key {uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)return ret; } +static struct rte_hash *g_handle; +static struct rte_rcu_qsbr *g_qsv; +static volatile uint8_t writer_done; +struct flow_key g_rand_keys[9 ]; +/* +* rte_hash_rcu_qsbr_add positive and negative tests. +* - Add RCU QSBR variable to Hash +* - Add another RCU QSBR variable to Hash +* - Check returns +*/ +static int +test_hash_rcu_qsbr_add (void ) +{ +size_t sz; +struct rte_rcu_qsbr *qsv2 =NULL ; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + +struct rte_hash_parameters params; + +printf ("\n# Running RCU QSBR add tests\n" ); +memcpy (¶ms, &ut_params,sizeof (params)); + params.name ="test_hash_rcu_qsbr_add" ; + params.extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF | +RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD ; + g_handle =rte_hash_create (¶ms); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; +/* Invalid QSBR mode */ + rcu_cfg.mode =2 ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status ==0 ,"Invalid QSBR mode test failed" ); + + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Create and attach another RCU QSBR to hash table */ + qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (qsv2 ==NULL , +"RCU QSBR variable creation failed" ); + + rcu_cfg.v = qsv2; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +rte_free (qsv2); +RETURN_IF_ERROR_RCU_QSBR (status ==0 , +"Attach RCU QSBR to hash table succeeded where failure" +" is expected" ); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; +} + +/* +* rte_hash_rcu_qsbr_add DQ mode functional test. +* Reader and writer are in the same thread in this test. +* - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries +* - Add RCU QSBR variable to hash +* - Add 8 hash entries and fill the bucket +* - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt +* - Register a reader thread (not a real thread) +* - Reader lookup existing entry +* - Writer deletes the entry +* - Reader lookup the entry +* - Writer re-add the entry (no available free index) +* - Reader report quiescent state and unregister +* - Writer re-add the entry +* - Reader lookup the entry +*/ +static int +test_hash_rcu_qsbr_dq_mode (uint8_t ext_bkt) +{ +uint32_t total_entries = (ext_bkt ==0 ) ?8 :9 ; + +uint8_t hash_extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ; + +if (ext_bkt) + hash_extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; + +struct rte_hash_parameters params_pseudo_hash = { + .name ="test_hash_rcu_qsbr_dq_mode" , + .entries = total_entries, + .key_len =sizeof (struct flow_key ),/* 13 */ + .hash_func = pseudo_hash, + .hash_func_init_val =0 , + .socket_id =0 , + .extra_flag = hash_extra_flag, + }; +int pos[total_entries]; +int expected_pos[total_entries]; +unsigned i; +size_t sz; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + + g_qsv =NULL ; + g_handle =NULL ; + +for (i =0 ; i < total_entries; i++) { + g_rand_keys[i].port_dst = i; + g_rand_keys[i].port_src = i+1 ; + } + +if (ext_bkt) +printf ("\n# Running RCU QSBR DQ mode functional test with" +" ext bkt\n" ); +else +printf ("\n# Running RCU QSBR DQ mode functional test\n" ); + + g_handle =rte_hash_create (¶ms_pseudo_hash); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Fill bucket */ +for (i =0 ; i < total_entries; i++) { + pos[i] =rte_hash_add_key (g_handle, &g_rand_keys[i]); +print_key_info ("Add" , &g_rand_keys[i], pos[i]); +RETURN_IF_ERROR_RCU_QSBR (pos[i] <0 , +"failed to add key (pos[%u]=%d)" , i, + pos[i]); + expected_pos[i] = pos[i]; + } + +/* Register pseudo reader */ + status =rte_rcu_qsbr_thread_register (g_qsv,0 ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR thread registration failed" ); +rte_rcu_qsbr_thread_online (g_qsv,0 ); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to find correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +/* Writer update */ + pos[0 ] =rte_hash_del_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Del" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to del correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != -ENOENT , +"found deleted key (pos[%u]=%d)" ,0 , pos[0 ]); + +/* Fill bucket */ + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != -ENOSPC , +"Added key successfully (pos[%u]=%d)" ,0 , pos[0 ]); + +/* Reader quiescent */ +rte_rcu_qsbr_quiescent (g_qsv,0 ); + +/* Fill bucket */ + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] <0 , +"failed to add key (pos[%u]=%d)" ,0 , pos[0 ]); + expected_pos[0 ] = pos[0 ]; + +rte_rcu_qsbr_thread_offline (g_qsv,0 ); + (void )rte_rcu_qsbr_thread_unregister (g_qsv,0 ); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to find correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +rte_hash_free (g_handle); +rte_free (g_qsv); +return 0 ; + +} + +/* Report quiescent state interval every 1024 lookups. Larger critical +* sections in reader will result in writer polling multiple times. +*/ +#define QSBR_REPORTING_INTERVAL 1024 +#define WRITER_ITERATIONS 512 + +/* +* Reader thread using rte_hash data structure with RCU. +*/ +static int +test_hash_rcu_qsbr_reader (void *arg) +{ +int i; + +RTE_SET_USED (arg); +/* Register this thread to report quiescent state */ + (void )rte_rcu_qsbr_thread_register (g_qsv,0 ); +rte_rcu_qsbr_thread_online (g_qsv,0 ); + +do { +for (i =0 ; i <QSBR_REPORTING_INTERVAL ; i++) +rte_hash_lookup (g_handle, &g_rand_keys[0 ]); + +/* Update quiescent state */ +rte_rcu_qsbr_quiescent (g_qsv,0 ); + }while (!writer_done); + +rte_rcu_qsbr_thread_offline (g_qsv,0 ); + (void )rte_rcu_qsbr_thread_unregister (g_qsv,0 ); + +return 0 ; +} + +/* +* rte_hash_rcu_qsbr_add sync mode functional test. +* 1 Reader and 1 writer. They cannot be in the same thread in this test. +* - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries +* - Add RCU QSBR variable to hash +* - Register a reader thread. Reader keeps looking up a specific key. +* - Writer keeps adding and deleting a specific key. +*/ +static int +test_hash_rcu_qsbr_sync_mode (uint8_t ext_bkt) +{ +uint32_t total_entries = (ext_bkt ==0 ) ?8 :9 ; + +uint8_t hash_extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ; + +if (ext_bkt) + hash_extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; + +struct rte_hash_parameters params_pseudo_hash = { + .name ="test_hash_rcu_qsbr_sync_mode" , + .entries = total_entries, + .key_len =sizeof (struct flow_key ),/* 13 */ + .hash_func = pseudo_hash, + .hash_func_init_val =0 , + .socket_id =0 , + .extra_flag = hash_extra_flag, + }; +int pos[total_entries]; +int expected_pos[total_entries]; +unsigned i; +size_t sz; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + + g_qsv =NULL ; + g_handle =NULL ; + +for (i =0 ; i < total_entries; i++) { + g_rand_keys[i].port_dst = i; + g_rand_keys[i].port_src = i+1 ; + } + +if (ext_bkt) +printf ("\n# Running RCU QSBR sync mode functional test with" +" ext bkt\n" ); +else +printf ("\n# Running RCU QSBR sync mode functional test\n" ); + + g_handle =rte_hash_create (¶ms_pseudo_hash); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Launch reader thread */ +rte_eal_remote_launch (test_hash_rcu_qsbr_reader,NULL , +rte_get_next_lcore (-1 ,1 ,0 )); + +/* Fill bucket */ +for (i =0 ; i < total_entries; i++) { + pos[i] =rte_hash_add_key (g_handle, &g_rand_keys[i]); +print_key_info ("Add" , &g_rand_keys[i], pos[i]); +RETURN_IF_ERROR_RCU_QSBR (pos[i] <0 , +"failed to add key (pos[%u]=%d)" , i, pos[i]); + expected_pos[i] = pos[i]; + } + writer_done =0 ; + +/* Writer Update */ +for (i =0 ; i <WRITER_ITERATIONS ; i++) { + expected_pos[0 ] = pos[0 ]; + pos[0 ] =rte_hash_del_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Del" , &g_rand_keys[0 ], status); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to del correct key (pos[%u]=%d)" + ,0 , pos[0 ]); + + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] <0 , +"failed to add key (pos[%u]=%d)" ,0 , + pos[0 ]); + } + + writer_done =1 ; +/* Wait until reader exited. */ +rte_eal_mp_wait_lcore (); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; + +} +/* * Do all unit and performance tests. */
@@ -1862,6 +2235,21 @@ test_hash(void)if (test_crc32_hash_alg_equiv () <0 )return -1 ; +if (test_hash_rcu_qsbr_add () <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_dq_mode (0 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_dq_mode (1 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_sync_mode (0 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_sync_mode (1 ) <0 ) +return -1 ; +return 0 ; }
app/test/test_hash_readwrite_lf_perf.c +167 -3
218 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
Quoted 100 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBRAdd functional and performance tests for the integrated RCU QSBR.
Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>--- app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++- 2 files changed, 556 insertions(+), 4 deletions(-)app/test/test_hash.c index renamed from app/test/test_hash.c +0
-0
990a1815f893..22b47b3e7728100644app/test/test_hash.c +1
-1@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) -#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ +#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ if (cond) { \
printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
\app/test/test_hash.c +4
-0if (handle)rte_fbk_hash_free (handle); \@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) +#define RETURN_IF_ERROR_RCU_QSBR (cond, str, ...) do { \ +if (cond) { \ + printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ +if (rcu_cfg.mode ==RTE_HASH_QSBR_MODE_SYNC ) {\app/test/test_hash.c +34
-0+ writer_done =1 ; \ +/* Wait until reader exited. */ \ +rte_eal_mp_wait_lcore (); \ + } \ +if (g_handle)rte_hash_free (g_handle); \ +if (g_qsv)rte_free (g_qsv); \ +return -1 ; \ + } \ +}while (0 ) +/* 5-tuple key type */ struct flow_key {uint32_t ip_src;@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)return ret; } +static struct rte_hash *g_handle; +static struct rte_rcu_qsbr *g_qsv; +static volatile uint8_t writer_done; +struct flow_key g_rand_keys[9 ]; +/* +* rte_hash_rcu_qsbr_add positive and negative tests. +* - Add RCU QSBR variable to Hash +* - Add another RCU QSBR variable to Hash +* - Check returns +*/ +static int +test_hash_rcu_qsbr_add (void ) +{ +size_t sz; +struct rte_rcu_qsbr *qsv2 =NULL ; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + +struct rte_hash_parameters params; + +printf ("\n# Running RCU QSBR add tests\n" ); +memcpy (¶ms, &ut_params,sizeof (params)); + params.name ="test_hash_rcu_qsbr_add" ; + params.extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;app/test/test_hash.c +2
-0+ g_handle =rte_hash_create (¶ms); + RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creationfailed");
app/test/test_hash.c +5
-0+ +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE,
SOCKET_ID_ANY);
app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1853
Quoted 4 lines
app/test/test_hash.c +4
-0+RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE );
[Wang, Yipeng] It reminds me that could we hide this function in the rte_cuckoo_hash.c as well?
I saw most of the rcu related functions are hidden in the hash implementation, it would be less confusing if we hide this one as well.
app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1859
Quoted 6 lines
app/test/test_hash.c +6
-0+RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; +/* Invalid QSBR mode */ + rcu_cfg.mode =2 ;
[Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large number like 0xff?
app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1863
Quoted 5 lines
app/test/test_hash.c +5
-0+ status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status ==0 ,"Invalid QSBR mode test +failed"); + + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ;
[Wang, Yipeng] This reminds me that if there is an explanation on the difference of the two modes for users to easy to choose?
Quoted 26 lines
app/test/test_hash.c +8
-0+/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Create and attach another RCU QSBR to hash table */ + qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE,
SOCKET_ID_ANY);app/test/test_hash.c +9
-0+RETURN_IF_ERROR_RCU_QSBR (qsv2 ==NULL , +"RCU QSBR variable creation failed" ); + + rcu_cfg.v = qsv2; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +rte_free (qsv2); + RETURN_IF_ERROR_RCU_QSBR(status == 0, + "Attach RCU QSBR to hash table succeeded wherefailure"
app/test/test_hash.c +7
-0+ " is expected"); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; +}
<...>
Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
Quoted 102 lines · On Oct 20, 2020, at 10:54 PM, Wang, Yipeng1 <[email protected]> wrote:
Quoted 99 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBRAdd functional and performance tests for the integrated RCU QSBR.
Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>---
app/test/test_hash.c| 390 ++++++++++++++++++++++++-app/test/test_hash_readwrite_lf_perf.c| 170 ++++++++++-2 files changed, 556 insertions(+), 4 deletions(-)app/test/test_hash.c index renamed from app/test/test_hash.c +0
-0
990a1815f893..22b47b3e7728100644app/test/test_hash.c +1
-1@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) -#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ +#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ if (cond) { \printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ if (handle) rte_fbk_hash_free(handle); \app/test/test_hash.c +4
-0@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) +#define RETURN_IF_ERROR_RCU_QSBR (cond, str, ...) do { \ +if (cond) { \ + printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ +if (rcu_cfg.mode ==RTE_HASH_QSBR_MODE_SYNC ) {\app/test/test_hash.c +10
-0+ writer_done =1 ; \ +/* Wait until reader exited. */ \ +rte_eal_mp_wait_lcore (); \ + } \ +if (g_handle)rte_hash_free (g_handle); \ +if (g_qsv)rte_free (g_qsv); \ +return -1 ; \ + } \ +}while (0 ) +/* 5-tuple key type */
struct flow_key { uint32_t ip_src;app/test/test_hash.c +24
-0@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)return ret; } +static struct rte_hash *g_handle; +static struct rte_rcu_qsbr *g_qsv; +static volatile uint8_t writer_done; +struct flow_key g_rand_keys[9 ]; +/* +* rte_hash_rcu_qsbr_add positive and negative tests. +* - Add RCU QSBR variable to Hash +* - Add another RCU QSBR variable to Hash +* - Check returns +*/ +static int +test_hash_rcu_qsbr_add (void ) +{ +size_t sz; +struct rte_rcu_qsbr *qsv2 =NULL ; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + +struct rte_hash_parameters params; + +printf ("\n# Running RCU QSBR add tests\n" ); +memcpy (¶ms, &ut_params,sizeof (params)); + params.name ="test_hash_rcu_qsbr_add" ; + params.extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;app/test/test_hash.c +2
-0+ g_handle =rte_hash_create (¶ms); + RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creationfailed");
app/test/test_hash.c +4
-0+ +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
Quoted 8 lines
app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1853
Quoted 6 lines
app/test/test_hash.c +1
-0+ RTE_CACHE_LINE_SIZE,
SOCKET_ID_ANY);app/test/test_hash.c +4
-0+RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE );[Wang, Yipeng] It reminds me that could we hide this function in the
rte_cuckoo_hash.cas well?
I saw most of the rcu related functions are hidden in the hash implementation, it would be less confusing if we hide this one as well.
Yes, I think this can be hidden within rte_hash_reset() and rte_hash_rcu_qsbr_add()
Quoted 8 lines
app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1859
Quoted 7 lines
app/test/test_hash.c +6
-0+RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; +/* Invalid QSBR mode */ + rcu_cfg.mode =2 ;[Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large number like
0xff?
Yes, I can use 0xxff
Quoted 7 lines
app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1863
Quoted 6 lines
app/test/test_hash.c +5
-0+ status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status ==0 ,"Invalid QSBR mode test +failed"); + + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ;[Wang, Yipeng] This reminds me that if there is an explanation on the difference of the two modes for users to easy to choose?
It is available within rte_hash.h
Quoted 29 lines
Quoted 28 lines
app/test/test_hash.c +8
-0+/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Create and attach another RCU QSBR to hash table */ + qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE,
SOCKET_ID_ANY);app/test/test_hash.c +9
-0+RETURN_IF_ERROR_RCU_QSBR (qsv2 ==NULL , +"RCU QSBR variable creation failed" ); + + rcu_cfg.v = qsv2; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +rte_free (qsv2); + RETURN_IF_ERROR_RCU_QSBR(status == 0, + "Attach RCU QSBR to hash table succeeded wherefailure"
app/test/test_hash.c +7
-0+ " is expected"); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; +}<...>
Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR
In reply to Wang, Yipeng1<snip>
Quoted 91 lines
Quoted 91 lines
Add functional and performance tests for the integrated RCU QSBR.
Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>--- app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++- 2 files changed, 556 insertions(+), 4 deletions(-)app/test/test_hash.c index renamed from app/test/test_hash.c +0
-0
990a1815f893..22b47b3e7728100644app/test/test_hash.c +1
-1@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) -#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ +#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ if (cond) { \app/test/test_hash.c +4
-0printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \if (handle)rte_fbk_hash_free (handle); \@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) +#define RETURN_IF_ERROR_RCU_QSBR (cond, str, ...) do { \ +if (cond) { \ + printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ +if (rcu_cfg.mode ==RTE_HASH_QSBR_MODE_SYNC ) {\app/test/test_hash.c +34
-0+ writer_done =1 ; \ +/* Wait until reader exited. */ \ +rte_eal_mp_wait_lcore (); \ + } \ +if (g_handle)rte_hash_free (g_handle); \ +if (g_qsv)rte_free (g_qsv); \ +return -1 ; \ + } \ +}while (0 ) +/* 5-tuple key type */ struct flow_key {uint32_t ip_src;@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)return ret; } +static struct rte_hash *g_handle; +static struct rte_rcu_qsbr *g_qsv; +static volatile uint8_t writer_done; +struct flow_key g_rand_keys[9 ]; +/* +* rte_hash_rcu_qsbr_add positive and negative tests. +* - Add RCU QSBR variable to Hash +* - Add another RCU QSBR variable to Hash +* - Check returns +*/ +static int +test_hash_rcu_qsbr_add (void ) +{ +size_t sz; +struct rte_rcu_qsbr *qsv2 =NULL ; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + +struct rte_hash_parameters params; + +printf ("\n# Running RCU QSBR add tests\n" ); +memcpy (¶ms, &ut_params,sizeof (params)); + params.name ="test_hash_rcu_qsbr_add" ; + params.extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;app/test/test_hash.c +2
-0+ g_handle =rte_hash_create (¶ms); + RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creationfailed");
app/test/test_hash.c +5
-0+ +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE,
SOCKET_ID_ANY);
Quoted 8 lines
app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1853
Quoted 4 lines
app/test/test_hash.c +4
-0+RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE );[Wang, Yipeng] It reminds me that could we hide this function in the
rte_cuckoo_hash.cas well?
I saw most of the rcu related functions are hidden in the hash implementation, it would be less confusing if we hide this one as well.
The functionality required for resource reclamation is split into various parts [1]. The functionality of creating the RCU variable, initializing it etc is left to the application. The application has better knowledge of the data plane threads, which ones are using the (hash) library etc. Hence it is better to keep this in the application.
[1] http://doc.dpdk.org/guides/prog_guide/rcu_lib.html#resource-reclamation-framework-for-dpdk
Quoted 45 lines
app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1859
Quoted 7 lines
app/test/test_hash.c +6
-0+RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; +/* Invalid QSBR mode */ + rcu_cfg.mode =2 ;[Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large number like
0xff?app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1863
Quoted 5 lines
app/test/test_hash.c +5
-0+ status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status ==0 ,"Invalid QSBR mode test +failed"); + + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ;[Wang, Yipeng] This reminds me that if there is an explanation on the difference of the two modes for users to easy to choose?
Quoted 26 lines
app/test/test_hash.c +8
-0+/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Create and attach another RCU QSBR to hash table */ + qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz, + RTE_CACHE_LINE_SIZE,
SOCKET_ID_ANY);app/test/test_hash.c +9
-0+RETURN_IF_ERROR_RCU_QSBR (qsv2 ==NULL , +"RCU QSBR variable creation failed" ); + + rcu_cfg.v = qsv2; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +rte_free (qsv2); + RETURN_IF_ERROR_RCU_QSBR(status == 0, + "Attach RCU QSBR to hash table succeeded wherefailure"
app/test/test_hash.c +7
-0+ " is expected"); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; +}<...>
v6 cover hash: integrate RCU QSBR
In reply to Dharmik ThakkarIntegrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.
Functional tests and performance tests are added to cover the integration with RCU.
---
v6:
- Update documentation
- Remove hard-coded value
v5:
- Enable 'rcu' for Windows
v4:
- Fix clang compilation issues
v3:
- Add documentation
- Add unit tests
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead
- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted
Dharmik Thakkar (4):
rcu: build on Windows
lib/hash: integrate RCU QSBR
test/hash: replace rte atomic with C11 atomic APIs test/hash: add tests for integrated RCU QSBR
app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 259 ++++++++++++---- doc/guides/prog_guide/hash_lib.rst | 12 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 303 ++++++++++++++----- lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 ++++- lib/librte_hash/version.map | 2 +- lib/meson.build | 1 + 9 files changed, 920 insertions(+), 133 deletions(-)
Signature · 1 line
2.17.1
PATCH v6 1/4 rcu: build on Windows
Build the lib for Windows.
--- lib/meson.build | 1 + 1 file changed, 1 insertion(+)
lib/meson.build +1 -0
@@ -41,6 +41,7 @@ if is_windows 'telemetry', 'eal', 'ring', + 'rcu', 'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci', 'cmdline', 'hash',
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows
Quoted 22 lines · Dharmik Thakkar
Build the lib for Windows.
Signed-off-by Dharmik Thakkar <[email protected]>Tested-by Dmitry Kozlyuk <[email protected]>--- lib/meson.build | 1 + 1 file changed, 1 insertion(+)lib/meson.build +1
-1@@ -41,6 +41,7 @@ if is_windows 'telemetry', 'eal', 'ring', + 'rcu', 'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci', 'cmdline', 'hash', --
Quoted 2 lines
2.17.1
I only saw a test from Dmitry, pinging other Windows maintainers to get a clear ack.
Thanks.
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows
In reply to Dharmik ThakkarQuoted 5 lines · Dharmik Thakkar
Build the lib for Windows.
Signed-off-by Dharmik Thakkar <[email protected]>Tested-by Dmitry Kozlyuk <[email protected]>---
Builds successfully on Windows.
PATCH v6 2/4 lib/hash: integrate RCU QSBR
In reply to Dharmik ThakkarCurrently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.
--- doc/guides/prog_guide/hash_lib.rst | 12 +- lib/librte_hash/meson.build | 1 + lib/librte_hash/rte_cuckoo_hash.c | 303 ++++++++++++++++++++++------- lib/librte_hash/rte_cuckoo_hash.h | 8 + lib/librte_hash/rte_hash.h | 77 +++++++- lib/librte_hash/version.map | 2 +- 6 files changed, 327 insertions(+), 76 deletions(-)
doc/guides/prog_guide/hash_lib.rst +9 -3
@@ -102,6 +102,10 @@ For concurrent writes, and concurrent reads and writes the following flag values * If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position. Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position. + RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature. + For example, rte_hash_rcu_qsbr_add() need to be called to use the integrated RCU mechanism. + Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details. + Extendable Bucket Functionality support ----------------------------------------
@@ -109,8 +113,8 @@ An extra flag is used to enable this functionality (flag is not set by default). in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the hash table size and can't tolerate any key insertion failure (even if very few). -Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and -deleted keys, to maintain the 100% capacity guarantee. +Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR +(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee. Implementation Details (non Extendable Bucket Case) ---------------------------------------------------
@@ -172,7 +176,7 @@ Example of deletion: Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency', the position of the key is not freed. It is the responsibility of the user to free the position after -readers are not referencing the position anymore. +readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete Implementation Details (with Extendable Bucket)
@@ -286,6 +290,8 @@ The flow table operations on the application side are described below: * Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set, wait till the readers are not referencing the position returned during add/delete flow and then free the position. RCU mechanisms can be used to find out when the readers are not referencing the position anymore. + RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature. + Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details. * Lookup flow: Lookup for the flow key in the hash. If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
lib/librte_hash/meson.build +1 -0
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h', sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c') deps += ['ring'] +deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +235 -68
419 lines not on this page · raw message
lib/librte_hash/rte_cuckoo_hash.h +8 -0
@@ -168,6 +168,11 @@ struct rte_hash {struct lcore_cache *local_free_slots;/**< Local cache per lcore, storing some indexes of the free slots */ +/* RCU config */ +struct rte_hash_rcu_config *hash_rcu_cfg; +/**< HASH RCU QSBR configuration structure */ +struct rte_rcu_qsbr_dq *dq;/**< RCU QSBR defer queue. */ +/* Fields used in lookup */ uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node { int prev_slot; /* Parent(slot) in search path */ }; +/** @internal Default RCU defer queue entries to reclaim in one go. */ +#define RTE_HASH_RCU_DQ_RECLAIM_MAX 16 +#endif
lib/librte_hash/rte_hash.h +73 -4
125 lines not on this page · raw message
lib/librte_hash/version.map +1 -1
@@ -36,5 +36,5 @@ EXPERIMENTAL { rte_hash_lookup_with_hash_bulk; rte_hash_lookup_with_hash_bulk_data; rte_hash_max_key_id; - + rte_hash_rcu_qsbr_add; };
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR
Hi Yipeng,
I have addressed the comments. Can you please take a look?
Thank you!
Quoted 15 lines · Dharmik Thakkar
Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>---
Re: [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR
In reply to Dharmik ThakkarQuoted 18 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Wednesday, October 21, 2020 3:50 PM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v6 2/4] lib/hash: integrate RCU QSBRCurrently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.
Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.
Quoted 5 lines
Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>Acked-by Ray Kinsella <[email protected]>---
[Wang, Yipeng]
Thanks for revising the code.
Please check the checkpatch reported coding style issues to fix. After that
PATCH v6 3/4 test/hash: replace rte atomic with C11 atomic APIs
In reply to Dharmik ThakkarReplace rte_atomic APIs with C11 atomic APIs intest_hash_readwrite_lf_perf.c
--- app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++--------------- 1 file changed, 36 insertions(+), 53 deletions(-)
app/test/test_hash_readwrite_lf_perf.c +36 -53
@@ -82,8 +82,8 @@ static struct {struct rte_hash *h; } tbl_rwc_test_param; -static rte_atomic64_t gread_cycles; -static rte_atomic64_t greads; +static uint64_t gread_cycles; +static uint64_t greads;static volatile uint8_t writer_done;
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg) }while (!writer_done); cycles =rte_rdtsc_precise () - begin; -rte_atomic64_add (&gread_cycles, cycles); -rte_atomic64_add (&greads, read_cnt*loop_cnt); +__atomic_fetch_add (&gread_cycles, cycles, __ATOMIC_RELAXED); +__atomic_fetch_add (&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);return 0 ; }
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type =WRITE_NO_KEY_SHIFT ;uint8_t read_type =READ_PASS_NO_KEY_SHIFTS ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - no key-shifts, read - hit\n" );
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_no_ks_r_hit[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t read_type =READ_FAIL ;int ret; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - no key-shifts, Hash lookup - miss\n" );
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_no_ks_r_miss[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,uint8_t write_type;uint8_t read_type =READ_PASS_NON_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err; printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_nsp[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type;uint8_t read_type =READ_PASS_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err; printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf, goto finish;printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); + +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_sp[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, intuint8_t write_type;uint8_t read_type =READ_FAIL ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - key shift, Hash lookup - miss\n" );
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, intprintf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup); }
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,uint8_t write_type;uint8_t read_type =READ_PASS_SHIFT_PATH ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Multi-add-lookup\n" );
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , + __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); writer_done =0 ;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) - /rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, + __ATOMIC_RELAXED) / +__atomic_load_n (&greads, + __ATOMIC_RELAXED); rwc_perf_results->multi_rw[m][k][n] = cycles_per_lookup; printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,uint8_t write_type;uint8_t read_type =READ_PASS_KEY_SHIFTS_EXTBKT ; -rte_atomic64_init (&greads); -rte_atomic64_init (&gread_cycles); -if (init_params (rwc_lf, use_jhash, htm, ext_bkt) !=0 ) goto err;printf ("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n" );
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,printf ("\nNumber of readers: %u\n" , rwc_core_cnt[n]); -rte_atomic64_clear (&greads); -rte_atomic64_clear (&gread_cycles); +__atomic_store_n (&greads,0 , __ATOMIC_RELAXED); +__atomic_store_n (&gread_cycles,0 , __ATOMIC_RELAXED);rte_hash_reset (tbl_rwc_test_param.h); write_type =WRITE_NO_KEY_SHIFT ;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results, goto err;unsigned long long cycles_per_lookup = -rte_atomic64_read (&gread_cycles) / -rte_atomic64_read (&greads); +__atomic_load_n (&gread_cycles, __ATOMIC_RELAXED) + /__atomic_load_n (&greads, __ATOMIC_RELAXED); rwc_perf_results->w_ks_r_hit_extbkt[m][n] = cycles_per_lookup;printf ("Cycles per lookup: %llu\n" , cycles_per_lookup);
Signature · 1 line
2.17.1
PATCH v6 4/4 test/hash: add tests for integrated RCU QSBR
In reply to Dharmik ThakkarAdd functional and performance tests for the integrated RCU QSBR.
--- app/test/test_hash.c | 390 ++++++++++++++++++++++++- app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++- 2 files changed, 556 insertions(+), 4 deletions(-)
app/test/test_hash.c +389 -1
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) -#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ +#define RETURN_IF_ERROR_FBK (cond, str, ...) do { \ if (cond) { \ printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ if (handle) rte_fbk_hash_free(handle); \
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21, } \ } while(0) +#define RETURN_IF_ERROR_RCU_QSBR (cond, str, ...) do { \ + if (cond) { \ + printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \ + if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) { \ + writer_done = 1; \ +/* Wait until reader exited. */ \ +rte_eal_mp_wait_lcore (); \ + } \ +if (g_handle)rte_hash_free (g_handle); \ +if (g_qsv)rte_free (g_qsv); \ +return -1 ; \ + } \ +} while(0) +/* 5-tuple key type */ struct flow_key {uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)return ret; } +static struct rte_hash *g_handle; +static struct rte_rcu_qsbr *g_qsv; +static volatile uint8_t writer_done; +struct flow_key g_rand_keys[9 ]; +/* +* rte_hash_rcu_qsbr_add positive and negative tests. +* - Add RCU QSBR variable to Hash +* - Add another RCU QSBR variable to Hash +* - Check returns +*/ +static int +test_hash_rcu_qsbr_add (void ) +{ +size_t sz; +struct rte_rcu_qsbr *qsv2 =NULL ; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + +struct rte_hash_parameters params; + +printf ("\n# Running RCU QSBR add tests\n" ); +memcpy (¶ms, &ut_params,sizeof (params)); + params.name ="test_hash_rcu_qsbr_add" ; + params.extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF | +RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD ; + g_handle =rte_hash_create (¶ms); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; +/* Invalid QSBR mode */ + rcu_cfg.mode =0xff ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status ==0 ,"Invalid QSBR mode test failed" ); + + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Create and attach another RCU QSBR to hash table */ + qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (qsv2 ==NULL , +"RCU QSBR variable creation failed" ); + + rcu_cfg.v = qsv2; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +rte_free (qsv2); +RETURN_IF_ERROR_RCU_QSBR (status ==0 , +"Attach RCU QSBR to hash table succeeded where failure" +" is expected" ); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; +} + +/* +* rte_hash_rcu_qsbr_add DQ mode functional test. +* Reader and writer are in the same thread in this test. +* - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries +* - Add RCU QSBR variable to hash +* - Add 8 hash entries and fill the bucket +* - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt +* - Register a reader thread (not a real thread) +* - Reader lookup existing entry +* - Writer deletes the entry +* - Reader lookup the entry +* - Writer re-add the entry (no available free index) +* - Reader report quiescent state and unregister +* - Writer re-add the entry +* - Reader lookup the entry +*/ +static int +test_hash_rcu_qsbr_dq_mode (uint8_t ext_bkt) +{ +uint32_t total_entries = (ext_bkt ==0 ) ?8 :9 ; + +uint8_t hash_extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ; + +if (ext_bkt) + hash_extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; + +struct rte_hash_parameters params_pseudo_hash = { + .name ="test_hash_rcu_qsbr_dq_mode" , + .entries = total_entries, + .key_len =sizeof (struct flow_key ),/* 13 */ + .hash_func = pseudo_hash, + .hash_func_init_val =0 , + .socket_id =0 , + .extra_flag = hash_extra_flag, + }; +int pos[total_entries]; +int expected_pos[total_entries]; +unsigned i; +size_t sz; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + + g_qsv =NULL ; + g_handle =NULL ; + +for (i =0 ; i < total_entries; i++) { + g_rand_keys[i].port_dst = i; + g_rand_keys[i].port_src = i+1 ; + } + +if (ext_bkt) +printf ("\n# Running RCU QSBR DQ mode functional test with" +" ext bkt\n" ); +else +printf ("\n# Running RCU QSBR DQ mode functional test\n" ); + + g_handle =rte_hash_create (¶ms_pseudo_hash); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_DQ ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Fill bucket */ +for (i =0 ; i < total_entries; i++) { + pos[i] =rte_hash_add_key (g_handle, &g_rand_keys[i]); +print_key_info ("Add" , &g_rand_keys[i], pos[i]); +RETURN_IF_ERROR_RCU_QSBR (pos[i] <0 , +"failed to add key (pos[%u]=%d)" , i, + pos[i]); + expected_pos[i] = pos[i]; + } + +/* Register pseudo reader */ + status =rte_rcu_qsbr_thread_register (g_qsv,0 ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR thread registration failed" ); +rte_rcu_qsbr_thread_online (g_qsv,0 ); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to find correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +/* Writer update */ + pos[0 ] =rte_hash_del_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Del" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to del correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != -ENOENT , +"found deleted key (pos[%u]=%d)" ,0 , pos[0 ]); + +/* Fill bucket */ + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != -ENOSPC , +"Added key successfully (pos[%u]=%d)" ,0 , pos[0 ]); + +/* Reader quiescent */ +rte_rcu_qsbr_quiescent (g_qsv,0 ); + +/* Fill bucket */ + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] <0 , +"failed to add key (pos[%u]=%d)" ,0 , pos[0 ]); + expected_pos[0 ] = pos[0 ]; + +rte_rcu_qsbr_thread_offline (g_qsv,0 ); + (void )rte_rcu_qsbr_thread_unregister (g_qsv,0 ); + +/* Lookup */ + pos[0 ] =rte_hash_lookup (g_handle, &g_rand_keys[0 ]); +print_key_info ("Lkp" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to find correct key (pos[%u]=%d)" ,0 , + pos[0 ]); + +rte_hash_free (g_handle); +rte_free (g_qsv); +return 0 ; + +} + +/* Report quiescent state interval every 1024 lookups. Larger critical +* sections in reader will result in writer polling multiple times. +*/ +#define QSBR_REPORTING_INTERVAL 1024 +#define WRITER_ITERATIONS 512 + +/* +* Reader thread using rte_hash data structure with RCU. +*/ +static int +test_hash_rcu_qsbr_reader (void *arg) +{ +int i; + +RTE_SET_USED (arg); +/* Register this thread to report quiescent state */ + (void )rte_rcu_qsbr_thread_register (g_qsv,0 ); +rte_rcu_qsbr_thread_online (g_qsv,0 ); + +do { +for (i =0 ; i <QSBR_REPORTING_INTERVAL ; i++) +rte_hash_lookup (g_handle, &g_rand_keys[0 ]); + +/* Update quiescent state */ +rte_rcu_qsbr_quiescent (g_qsv,0 ); + }while (!writer_done); + +rte_rcu_qsbr_thread_offline (g_qsv,0 ); + (void )rte_rcu_qsbr_thread_unregister (g_qsv,0 ); + +return 0 ; +} + +/* +* rte_hash_rcu_qsbr_add sync mode functional test. +* 1 Reader and 1 writer. They cannot be in the same thread in this test. +* - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries +* - Add RCU QSBR variable to hash +* - Register a reader thread. Reader keeps looking up a specific key. +* - Writer keeps adding and deleting a specific key. +*/ +static int +test_hash_rcu_qsbr_sync_mode (uint8_t ext_bkt) +{ +uint32_t total_entries = (ext_bkt ==0 ) ?8 :9 ; + +uint8_t hash_extra_flag =RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF ; + +if (ext_bkt) + hash_extra_flag |=RTE_HASH_EXTRA_FLAGS_EXT_TABLE ; + +struct rte_hash_parameters params_pseudo_hash = { + .name ="test_hash_rcu_qsbr_sync_mode" , + .entries = total_entries, + .key_len =sizeof (struct flow_key ),/* 13 */ + .hash_func = pseudo_hash, + .hash_func_init_val =0 , + .socket_id =0 , + .extra_flag = hash_extra_flag, + }; +int pos[total_entries]; +int expected_pos[total_entries]; +unsigned i; +size_t sz; +int32_t status; +struct rte_hash_rcu_config rcu_cfg = {0 }; + + g_qsv =NULL ; + g_handle =NULL ; + +for (i =0 ; i < total_entries; i++) { + g_rand_keys[i].port_dst = i; + g_rand_keys[i].port_src = i+1 ; + } + +if (ext_bkt) +printf ("\n# Running RCU QSBR sync mode functional test with" +" ext bkt\n" ); +else +printf ("\n# Running RCU QSBR sync mode functional test\n" ); + + g_handle =rte_hash_create (¶ms_pseudo_hash); +RETURN_IF_ERROR_RCU_QSBR (g_handle ==NULL ,"Hash creation failed" ); + +/* Create RCU QSBR variable */ + sz =rte_rcu_qsbr_get_memsize (RTE_MAX_LCORE ); + g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket (NULL , sz, +RTE_CACHE_LINE_SIZE ,SOCKET_ID_ANY ); +RETURN_IF_ERROR_RCU_QSBR (g_qsv ==NULL , +"RCU QSBR variable creation failed" ); + + status =rte_rcu_qsbr_init (g_qsv,RTE_MAX_LCORE ); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"RCU QSBR variable initialization failed" ); + + rcu_cfg.v = g_qsv; + rcu_cfg.mode =RTE_HASH_QSBR_MODE_SYNC ; +/* Attach RCU QSBR to hash table */ + status =rte_hash_rcu_qsbr_add (g_handle, &rcu_cfg); +RETURN_IF_ERROR_RCU_QSBR (status !=0 , +"Attach RCU QSBR to hash table failed" ); + +/* Launch reader thread */ +rte_eal_remote_launch (test_hash_rcu_qsbr_reader,NULL , +rte_get_next_lcore (-1 ,1 ,0 )); + +/* Fill bucket */ +for (i =0 ; i < total_entries; i++) { + pos[i] =rte_hash_add_key (g_handle, &g_rand_keys[i]); +print_key_info ("Add" , &g_rand_keys[i], pos[i]); +RETURN_IF_ERROR_RCU_QSBR (pos[i] <0 , +"failed to add key (pos[%u]=%d)" , i, pos[i]); + expected_pos[i] = pos[i]; + } + writer_done =0 ; + +/* Writer Update */ +for (i =0 ; i <WRITER_ITERATIONS ; i++) { + expected_pos[0 ] = pos[0 ]; + pos[0 ] =rte_hash_del_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Del" , &g_rand_keys[0 ], status); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] != expected_pos[0 ], +"failed to del correct key (pos[%u]=%d)" + ,0 , pos[0 ]); + + pos[0 ] =rte_hash_add_key (g_handle, &g_rand_keys[0 ]); +print_key_info ("Add" , &g_rand_keys[0 ], pos[0 ]); +RETURN_IF_ERROR_RCU_QSBR (pos[0 ] <0 , +"failed to add key (pos[%u]=%d)" ,0 , + pos[0 ]); + } + + writer_done =1 ; +/* Wait until reader exited. */ +rte_eal_mp_wait_lcore (); + +rte_hash_free (g_handle); +rte_free (g_qsv); + +return 0 ; + +} +/* * Do all unit and performance tests. */
@@ -1862,6 +2235,21 @@ test_hash(void)if (test_crc32_hash_alg_equiv () <0 )return -1 ; +if (test_hash_rcu_qsbr_add () <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_dq_mode (0 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_dq_mode (1 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_sync_mode (0 ) <0 ) +return -1 ; + +if (test_hash_rcu_qsbr_sync_mode (1 ) <0 ) +return -1 ; +return 0 ; }
app/test/test_hash_readwrite_lf_perf.c +167 -3
218 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [dpdk-dev] [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR
Quoted 9 lines
-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Wednesday, October 21, 2020 3:50 PM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR
Quoted 6 lines
Add functional and performance tests for the integrated RCU QSBR.
Suggested-by Honnappa Nagarahalli <[email protected]>Signed-off-by Dharmik Thakkar <[email protected]>Reviewed-by Ruifeng Wang <[email protected]>---
[Wang, Yipeng]
Also please confirm the checkpatch report see if it is not a false-positive. Then:
Re: [dpdk-dev] [PATCH v6 0/4] hash: integrate RCU QSBR
In reply to Dharmik ThakkarQuoted 12 lines · Dharmik Thakkar
Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
Quoted 7 lines
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Functional tests and performance tests are added to cover the integration with RCU.
Fixed some style issues/checkpatch warnings.
The unit test code especially is not really checkpatch compliant, maybe worth a separate cleanup later.
Removed unrelated changes (like removing empty lines, fixing typos in unrelated parts).
Series applied, thanks.
Little note: checkpatch now has a check on repeated words better than a script of mine:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/scripts/checkpatch.pl?id=1a3dcf2e6b35faa1176b9cd8200094fbce16ba19
Probably worth it for people to update their checkpatch.pl.
Signature · 1 line
David Marchand
Re: [dpdk-dev] [PATCH v6 0/4] hash: integrate RCU QSBR
Quoted 33 lines · David Marchand
Quoted 12 lines · Dharmik Thakkar
Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.
Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.
Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.
Quoted 7 lines
[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html
Introduce a new API
rte_hash_rcu_qsbr_addfor application to register a RCU variable that hash library will use.Functional tests and performance tests are added to cover the integration with RCU.
Fixed some style issues/checkpatch warnings.
The unit test code especially is not really checkpatch compliant, maybe worth a separate cleanup later.Removed unrelated changes (like removing empty lines, fixing typos in unrelated parts).
Quoted 8 lines
Series applied, thanks.
Little note: checkpatch now has a check on repeated words better than a script of mine:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/scripts/checkpatch.pl?id=1a3dcf2e6b35faa1176b9cd8200094fbce16ba19
Probably worth it for people to update their checkpatch.pl.
Thank you, David! Appreciate your help.
Quoted 3 lines
Signature · 2 lines
-- David Marchand