Skip to contentSkip to conversation

RCU integration with hash library

RFC In review Dharmik Thakkar 7 participants 49 messages Sep 1, 2019 – Oct 26, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover RCU integration with hash library

This patchset integrates RCU QSBR support with hash library.

Note: This patchset has dependency on another patchset:

      https://patchwork.dpdk.org/cover/57813/

Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Add functional and performance test cases.

Fix 'ixgbe_ethdev.h' to avoid multiple definitions of 'bool'. (Please note that this fix is temporary. Recommend suggesting better solution)

Dharmik Thakkar (3):
net/ixgbe: avoid multpile definitions of 'bool' lib/hash: integrate RCU QSBR
test/hash: add tests for integrated RCU QSBR

 app/test/test_hash_readwrite_lf.c    | 700 ++++++++++++++++++++++++++-
 drivers/net/ixgbe/ixgbe_ethdev.h     |   1 +
 lib/librte_hash/Makefile             |   2 +-
 lib/librte_hash/meson.build          |   2 +
 lib/librte_hash/rte_cuckoo_hash.c    | 354 +++++++++++++-
 lib/librte_hash/rte_cuckoo_hash.h    |   3 +
 lib/librte_hash/rte_hash.h           |  38 +-
 lib/librte_hash/rte_hash_version.map |   2 +-
 8 files changed, 1065 insertions(+), 37 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

RFC 1/3 net/ixgbe: avoid multpile definitions of 'bool'

Compilation issue arises due to multiple definitions of 'bool' in 'ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This is a temporary fix. Recommend suggesting better solution.)

---
 drivers/net/ixgbe/ixgbe_ethdev.h | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ixgbe/ixgbe_ethdev.h +1 -0
index 6e9ed2e10f3c..dd4d6c022545 100644
@@ -18,6 +18,7 @@
 #include <rte_flow.h>
 #include <rte_time.h>
 #include <rte_hash.h>
+#undef bool
 #include <rte_pci.h>
 #include <rte_bus_pci.h>
 #include <rte_tm_driver.h>
Signature · 1 line
2.17.1
#2 lore raw

Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'

Quoted 16 lines · On Sun, 1 Sep 2019 06:58:08 +0000 Dharmik Thakkar <[email protected]>…

Compilation issue arises due to multiple definitions of 'bool' in 'ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This is a temporary fix. Recommend suggesting better solution.)

---
 drivers/net/ixgbe/ixgbe_ethdev.h | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ixgbe/ixgbe_ethdev.h +0 -0
index 6e9ed2e10f3c..dd4d6c022545 100644

drivers/net/ixgbe/ixgbe_ethdev.h line 24

Quoted 7 lines
drivers/net/ixgbe/ixgbe_ethdev.h +1 -0
@@ -18,6 +18,7 @@
 #include <rte_flow.h>
 #include <rte_time.h>
 #include <rte_hash.h>
+#undef bool
 #include <rte_pci.h>
 #include <rte_bus_pci.h>
 #include <rte_tm_driver.h>

Just fix ixgbe_ethdev.h to include stdbool itself?

drivers/net/ixgbe/ixgbe_ethdev.h +1 -0
index 6e9ed2e10f3c..cf32b1761acb 100644
@@ -6,6 +6,7 @@
 #define _IXGBE_ETHDEV_H_
 
 #include <stdint.h>
+#include <stdbool.h>
 
 #include "base/ixgbe_type.h"
 #include "base/ixgbe_dcb.h"
#3 lore raw

Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'

Quoted 18 lines · Stephen Hemminger
Quoted 11 lines · On Sun, 1 Sep 2019 06:58:08 +0000 Dharmik Thakkar <[email protected]>…

Compilation issue arises due to multiple definitions of 'bool' in 'ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This is a temporary fix. Recommend suggesting better solution.)

---
drivers/net/ixgbe/ixgbe_ethdev.h | 1 +

1 file changed, 1 insertion(+)
Quoted 14 lines

drivers/net/ixgbe/ixgbe_ethdev.h line 24

Quoted 12 lines
drivers/net/ixgbe/ixgbe_ethdev.h +1 -0
index 6e9ed2e10f3c..dd4d6c022545 100644
@@ -18,6 +18,7 @@
 #include <rte_flow.h>
 #include <rte_time.h>
 #include <rte_hash.h>
+#undef bool
 #include <rte_pci.h>
 #include <rte_bus_pci.h>
 #include <rte_tm_driver.h>

Just fix ixgbe_ethdev.h to include stdbool itself?

Conflicts with 'typedef int bool;’ in ixgbe_osdep.h:86

Quoted 13 lines
drivers/net/ixgbe/ixgbe_ethdev.h +1 -0
index 6e9ed2e10f3c..cf32b1761acb 100644
@@ -6,6 +6,7 @@
 #define _IXGBE_ETHDEV_H_
 
 #include <stdint.h>
+#include <stdbool.h>
 
 #include "base/ixgbe_type.h"
 #include "base/ixgbe_dcb.h"
#4 lore raw

Re: [dpdk-dev] [RFC 1/3] net/ixgbe: avoid multpile definitions of 'bool'

Quoted 30 lines · On Thu, 5 Sep 2019 20:10:05 +0000 Dharmik Thakkar <[email protected]>…
Quoted 16 lines · Stephen Hemminger
Quoted 11 lines · On Sun, 1 Sep 2019 06:58:08 +0000 Dharmik Thakkar <[email protected]>…

Compilation issue arises due to multiple definitions of 'bool' in 'ixgbe_ethdev.h'.
(Please note that this issue showed up when 'rte_rcu_qsbr.h' got included within 'rte_hash.h', because 'rte_rcu_qsbr.h' includes 'stdbool.h'. This is a temporary fix. Recommend suggesting better solution.)

---
drivers/net/ixgbe/ixgbe_ethdev.h | 1 +

1 file changed, 1 insertion(+)
Quoted 14 lines

drivers/net/ixgbe/ixgbe_ethdev.h line 24

Quoted 12 lines
drivers/net/ixgbe/ixgbe_ethdev.h +1 -0
index 6e9ed2e10f3c..dd4d6c022545 100644
@@ -18,6 +18,7 @@
 #include <rte_flow.h>
 #include <rte_time.h>
 #include <rte_hash.h>
+#undef bool
 #include <rte_pci.h>
 #include <rte_bus_pci.h>
 #include <rte_tm_driver.h>  

Just fix ixgbe_ethdev.h to include stdbool itself?

Quoted 1 line

Conflicts with 'typedef int bool;’ in ixgbe_osdep.h:86

Then fix the ixgbe_osdep.h. It is the problem.

#5 lore raw

RFC 2/3 lib/hash: integrate RCU QSBR

In reply to Dharmik Thakkar

Integrate RCU QSBR process.
(Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.)

---
 lib/librte_hash/Makefile             |   2 +-
 lib/librte_hash/meson.build          |   2 +
 lib/librte_hash/rte_cuckoo_hash.c    | 352 +++++++++++++++++++++++++--
 lib/librte_hash/rte_cuckoo_hash.h    |   3 +
 lib/librte_hash/rte_hash.h           |  38 ++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 6 files changed, 363 insertions(+), 36 deletions(-)
lib/librte_hash/Makefile +1 -1
index 5669d83f454f..3edc9c96beaa 100644
@@ -8,7 +8,7 @@ LIB = librte_hash.a
 
 CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
 CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)
-LDLIBS += -lrte_eal -lrte_ring
+LDLIBS += -lrte_eal -lrte_ring -lrte_rcu
 
 EXPORT_MAP := rte_hash_version.map
 
lib/librte_hash/meson.build +2 -0
index ebf70de89014..468668ea2488 100644
@@ -2,6 +2,7 @@
 # Copyright(c) 2017 Intel Corporation
 
 version = 2
+allow_experimental_apis = true
 headers = files('rte_cmp_arm64.h',
 	'rte_cmp_x86.h',
 	'rte_crc_arm64.h',
@@ -14,6 +15,7 @@ headers = files('rte_cmp_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
 
 # rte ring reset is not yet part of stable API
 allow_experimental_apis = true
lib/librte_hash/rte_cuckoo_hash.c +326 -26

469 lines not on this page · raw message

lib/librte_hash/rte_cuckoo_hash.h +3 -0
index fb19bb27dfef..c64494ee8b85 100644
@@ -168,6 +168,9 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	struct rte_rcu_qsbr *qsv;
+	struct rte_ring *qs_fifo;
+	rte_hash_free_key_data free_key_data_func;
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
lib/librte_hash/rte_hash.h +30 -8
index 0d73370dc483..20db00784d0b 100644
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -44,8 +45,6 @@ extern "C" {
 
 /** Flag to disable freeing of key index on hash delete.
  * Refer to rte_hash_del_xxx APIs for more details.
- * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
  */
 #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
 
@@ -69,6 +68,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */
 typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow appication to free key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *key_data);
+
 /**
  * Parameters used when creating the hash table.
  */
@@ -267,8 +273,7 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
  * and should only be called from one thread by default.
  * Thread safety can be enabled by setting flag during
  * table creation.
- * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -296,8 +301,7 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
  * and should only be called from one thread by default.
  * Thread safety can be enabled by setting flag during
  * table creation.
- * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called
  * additionally to free the index associated with the key.
@@ -350,8 +354,7 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
  * of the key. This operation is not multi-thread safe and should
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
- * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL is enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers
  * have stopped referencing the entry corresponding to this key.
@@ -545,6 +548,25 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);
+
+/**
+ * Configure the RCU QSBR variable to use in hash structure.
+ *
+ * @param h
+ *   Hash table
+ * @param v
+ *   RCU QSBR variable
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL if invalid pointer
+ *   - EEXIST if already added QSBR
+ *   - ENOMEM if memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h, struct rte_rcu_qsbr *v,
+			rte_hash_free_key_data free_key_data_func);
 #ifdef __cplusplus
 }
 #endif
lib/librte_hash/rte_hash_version.map +1 -1
index 734ae28b0408..8c3b21891de0 100644
@@ -58,5 +58,5 @@ EXPERIMENTAL {
 	global:
 
 	rte_hash_free_key_with_position;
-
+	rte_hash_rcu_qsbr_add;
 };
Signature · 1 line
2.17.1
#6 lore raw

RFC 3/3 test/hash: add tests for integrated RCU QSBR

In reply to Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

---
 app/test/test_hash_readwrite_lf.c | 694 +++++++++++++++++++++++++++++-
 1 file changed, 693 insertions(+), 1 deletion(-)
app/test/test_hash_readwrite_lf.c +693 -1
index 1f2fba41f1b8..0846ed46ae52 100644
@@ -13,6 +13,7 @@
 #include <rte_malloc.h>
 #include <rte_random.h>
 #include <rte_spinlock.h>
+#include <rte_rcu_qsbr.h>
 
 #include "test.h"
 
@@ -58,6 +59,7 @@ struct rwc_perf {
 	uint32_t w_ks_r_miss[2][NUM_TEST];
 	uint32_t multi_rw[NUM_TEST - 1][2][NUM_TEST];
 	uint32_t w_ks_r_hit_extbkt[2][NUM_TEST];
+	uint32_t writer_add_del[NUM_TEST];
 };
 
 static struct rwc_perf rwc_lf_results, rwc_non_lf_results;
@@ -84,12 +86,15 @@ struct {
 
 static rte_atomic64_t gread_cycles;
 static rte_atomic64_t greads;
+static rte_atomic64_t gwrite_cycles;
+static rte_atomic64_t gwrites;
 
 static volatile uint8_t writer_done;
 
 uint16_t enabled_core_ids[RTE_MAX_LCORE];
 
 uint8_t *scanned_bkts;
+uint8_t no_free_on_del;
 
 static inline uint16_t
 get_short_sig(const hash_sig_t hash)
@@ -173,6 +178,9 @@ init_params(int rwc_lf, int use_jhash, int htm, int ext_bkt)
 	if (ext_bkt)
 		hash_params.extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
 
+	if (no_free_on_del)
+		hash_params.extra_flag |= RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL;
+
 	hash_params.name = "tests";
 
 	handle = rte_hash_create(&hash_params);
@@ -1240,6 +1248,666 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	return -1;
 }
 
+static uint32_t *hash_data[TOTAL_ENTRY];
+static struct rte_rcu_qsbr *t;
+#define COUNTER_VALUE 4096
+#define TEST_RCU_MAX_LCORE 128
+static int
+test_rwc_rcu_reader(__attribute__((unused))void *arg)
+{
+	struct rte_rcu_qsbr *temp;
+	unsigned int i;
+	uint32_t lcore_id = rte_lcore_id();
+	uint32_t *pdata;
+	temp = t;
+
+	rte_rcu_qsbr_thread_register(temp, lcore_id);
+	do {
+		rte_rcu_qsbr_thread_online(temp, lcore_id);
+		for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks; i++) {
+			rte_rcu_qsbr_lock(temp, lcore_id);
+			if (rte_hash_lookup_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void **)&pdata) != -ENOENT) {
+				if (pdata != NULL) {
+					pdata[lcore_id] = 0;
+					while (pdata[lcore_id] < COUNTER_VALUE)
+						pdata[lcore_id]++;
+				}
+			}
+			rte_rcu_qsbr_unlock(temp, lcore_id);
+		}
+		/* Update quiescent state counter */
+		rte_rcu_qsbr_quiescent(temp, lcore_id);
+		rte_rcu_qsbr_thread_offline(temp, lcore_id);
+	} while (!writer_done);
+	rte_rcu_qsbr_thread_unregister(temp, lcore_id);
+
+	return 0;
+}
+
+static int
+test_rwc_rcu_multi_writer(void *arg)
+{
+	int32_t pos;
+
+	uint32_t i, offset;
+	uint8_t pos_core = (uint32_t)((uintptr_t)arg);
+	offset = pos_core * tbl_rwc_test_param.single_insert;
+
+	for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) {
+		/* Delete element from the shared data structure */
+		pos = rte_hash_del_key(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i);
+		if (no_free_on_del) {
+			if (rte_hash_free_key_with_position(tbl_rwc_test_param.h,
+								pos) < 0)
+				return -1;
+		}
+		if (pos < 0) {
+			printf("Delete key failed #%u at #%d\n",
+				tbl_rwc_test_param.keys_no_ks[i], i);
+			return -1;
+		}
+		rte_hash_add_key(tbl_rwc_test_param.h,
+				tbl_rwc_test_param.keys_no_ks + i);
+	}
+	return 0;
+}
+
+uint8_t error_state;
+static void
+free_hash_data(void *pdata)
+{
+	uint32_t i;
+	uint32_t *h_data = pdata;
+	if (h_data == NULL)
+		return;
+	for (i = 0; i < rwc_core_cnt[NUM_TEST - 1]; i++) {
+		if (h_data[i] != COUNTER_VALUE && h_data[i] != 0) {
+			printf("Reader did not complete\n");
+			__atomic_store_n(&error_state, 1, __ATOMIC_RELAXED);
+			return;
+		}
+	}
+	rte_free(pdata);
+	return;
+}
+
+/*
+ * Test integrated RCU with 'free' on hash_delete
+ * and with extended bkt feature disabled
+ */
+static int
+test_hash_rcu_free_on_del(__attribute__((unused))struct rwc_perf
+				*rwc_perf_results, int rwc_lf, int htm,
+				int ext_bkt)
+{
+	unsigned int n, m;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	no_free_on_del = 0;
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	uint32_t sz;
+	sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+	t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+				free_hash_data) < 0) {
+		printf("RCU init in hash failed\n");
+		return -1;
+	}
+	printf("\nTest: RCU - free on hash del - ext bkt disabled\n");
+	uint8_t pos_core;
+
+	for (m = 1; m < NUM_TEST; m++) {
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m];
+		for (n = 0; n < NUM_TEST; n++) {
+			unsigned int tot_lcore = rte_lcore_count();
+			if (tot_lcore < (rwc_core_cnt[n] +
+			     rwc_core_cnt[m] + 1))
+				goto finish;
+
+			printf("\nNumber of writers: %u", rwc_core_cnt[m]);
+			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
+
+			rte_hash_reset(tbl_rwc_test_param.h);
+			writer_done = 0;
+			write_type = WRITE_NO_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			write_type = WRITE_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				hash_data[i] =
+					rte_zmalloc(NULL, sizeof(uint32_t)
+						* rwc_core_cnt[NUM_TEST - 1],
+						0);
+				if (hash_data[i] == NULL) {
+					printf("No memory\n");
+					return -1;
+				}
+			}
+			for (i = 0;
+				i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				if (rte_hash_add_key_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void *)((uintptr_t)hash_data[i]))
+					< 0) {
+					printf("Hash key add Failed #%d\n",
+						tbl_rwc_test_param.keys_no_ks[i]);
+					return -1;
+				}
+			}
+
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+			/* Launch reader(s) */
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				rte_eal_remote_launch(test_rwc_rcu_reader,
+					(void *)(uintptr_t)rwc_core_cnt[m],
+					enabled_core_ids[i]);
+			pos_core = 0;
+			/* Launch writers */
+			for (; i <= rwc_core_cnt[m]
+			     + rwc_core_cnt[n];	i++) {
+				rte_eal_remote_launch
+					(test_rwc_rcu_multi_writer,
+					(void *)(uintptr_t)pos_core,
+					enabled_core_ids[i]);
+				pos_core++;
+			}
+
+			/* Wait for writers to complete */
+			for (i = rwc_core_cnt[n] + 1;
+			     i <= rwc_core_cnt[m] + rwc_core_cnt[n];
+			     i++)
+				rte_eal_wait_lcore(enabled_core_ids[i]);
+
+			writer_done = 1;
+
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0)
+					goto err;
+
+		}
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return 0;
+
+err:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return -1;
+}
+
+/*
+ * Test integrated RCU with NO 'free' on hash_delete
+ * and with extended bkt feature disabled
+ */
+static int
+test_hash_rcu_no_free_on_del(__attribute__((unused))struct rwc_perf
+				*rwc_perf_results, int rwc_lf, int htm,
+				int ext_bkt)
+{
+	unsigned int n, m;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	no_free_on_del = 1;
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	uint32_t sz;
+	sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+	t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+				free_hash_data) < 0) {
+		printf("RCU init in hash failed\n");
+		return -1;
+	}
+
+	printf("\nTest: RCU - no free on hash del - ext bkt disabled\n");
+	uint8_t pos_core;
+
+	for (m = 1; m < NUM_TEST; m++) {
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m];
+		for (n = 0; n < NUM_TEST; n++) {
+			unsigned int tot_lcore = rte_lcore_count();
+			if (tot_lcore < (rwc_core_cnt[n] +
+			     rwc_core_cnt[m] + 1))
+				goto finish;
+
+			printf("\nNumber of writers: %u", rwc_core_cnt[m]);
+			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
+
+			rte_hash_reset(tbl_rwc_test_param.h);
+			writer_done = 0;
+			write_type = WRITE_NO_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			write_type = WRITE_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				hash_data[i] =
+					rte_zmalloc(NULL, sizeof(uint32_t)
+						* rwc_core_cnt[NUM_TEST - 1],
+						0);
+				if (hash_data[i] == NULL) {
+					printf("No memory\n");
+					return -1;
+				}
+			}
+			for (i = 0;
+				i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				if (rte_hash_add_key_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void *)((uintptr_t)hash_data[i]))
+					< 0) {
+					printf("Hash key add Failed #%d\n",
+						tbl_rwc_test_param.keys_no_ks[i]);
+					return -1;
+				}
+			}
+
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+			/* Launch reader(s) */
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				rte_eal_remote_launch(test_rwc_rcu_reader,
+					(void *)(uintptr_t)rwc_core_cnt[m],
+					enabled_core_ids[i]);
+			pos_core = 0;
+			/* Launch writers */
+			for (; i <= rwc_core_cnt[m]
+			     + rwc_core_cnt[n];	i++) {
+				rte_eal_remote_launch
+					(test_rwc_rcu_multi_writer,
+					(void *)(uintptr_t)pos_core,
+					enabled_core_ids[i]);
+				pos_core++;
+			}
+			/* Wait for writers to complete */
+			for (i = rwc_core_cnt[n] + 1;
+			     i <= rwc_core_cnt[m] + rwc_core_cnt[n];
+			     i++)
+				rte_eal_wait_lcore(enabled_core_ids[i]);
+
+			writer_done = 1;
+
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0)
+					goto err;
+		}
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return 0;
+
+err:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return -1;
+}
+
+/*
+ * Test integrated RCU with 'free' on hash_delete
+ * and with extended bkt feature enabled
+ */
+static int
+test_hash_rcu_ext_bkt_free_on_del(__attribute__((unused))struct rwc_perf
+					*rwc_perf_results, int rwc_lf, int htm,
+					int ext_bkt)
+{
+	unsigned int n, m;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	no_free_on_del = 0;
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	uint32_t sz;
+	sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+	t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+				free_hash_data) < 0) {
+		printf("RCU init in hash failed\n");
+		return -1;
+	}
+
+	printf("\nTest: RCU - free on hash del - ext bkt enabled\n");
+	uint8_t pos_core;
+
+	for (m = 1; m < NUM_TEST; m++) {
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m];
+		for (n = 0; n < NUM_TEST; n++) {
+			unsigned int tot_lcore = rte_lcore_count();
+			if (tot_lcore < (rwc_core_cnt[n] +
+			     rwc_core_cnt[m] + 1))
+				goto finish;
+
+			printf("\nNumber of writers: %u", rwc_core_cnt[m]);
+			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
+
+			rte_hash_reset(tbl_rwc_test_param.h);
+			writer_done = 0;
+			write_type = WRITE_NO_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			write_type = WRITE_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				hash_data[i] =
+					rte_zmalloc(NULL, sizeof(uint32_t)
+						* rwc_core_cnt[NUM_TEST - 1],
+						0);
+				if (hash_data[i] == NULL) {
+					printf("No memory\n");
+					return -1;
+				}
+			}
+			for (i = 0;
+				i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				if (rte_hash_add_key_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void *)((uintptr_t)hash_data[i]))
+					< 0) {
+					printf("Hash key add Failed #%d\n",
+						tbl_rwc_test_param.keys_no_ks[i]);
+					return -1;
+				}
+			}
+
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+			/* Launch reader(s) */
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				rte_eal_remote_launch(test_rwc_rcu_reader,
+					(void *)(uintptr_t)rwc_core_cnt[m],
+					enabled_core_ids[i]);
+			pos_core = 0;
+			/* Launch writers */
+			for (; i <= rwc_core_cnt[m]
+			     + rwc_core_cnt[n];	i++) {
+				rte_eal_remote_launch
+					(test_rwc_rcu_multi_writer,
+					(void *)(uintptr_t)pos_core,
+					enabled_core_ids[i]);
+				pos_core++;
+			}
+			/* Wait for writers to complete */
+			for (i = rwc_core_cnt[n] + 1;
+			     i <= rwc_core_cnt[m] + rwc_core_cnt[n];
+			     i++)
+				rte_eal_wait_lcore(enabled_core_ids[i]);
+
+			writer_done = 1;
+
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0)
+					goto err;
+		}
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return 0;
+
+err:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return -1;
+}
+
+/*
+ * Test integrated RCU with NO 'free' on hash_delete
+ * and with extended bkt feature enabled
+ */
+static int
+test_hash_rcu_ext_bkt_no_free_on_del(__attribute__((unused))struct rwc_perf
+					*rwc_perf_results, int rwc_lf, int htm,
+					int ext_bkt)
+{
+	unsigned int n, m;
+	uint64_t i;
+	uint8_t write_type;
+	int use_jhash = 0;
+	no_free_on_del = 1;
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	uint32_t sz;
+	sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+	t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+	if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+				free_hash_data) < 0) {
+		printf("RCU init in hash failed\n");
+		return -1;
+	}
+
+	printf("\nTest: RCU - no free on hash del - ext bkt enabled\n");
+	uint8_t pos_core;
+
+	for (m = 1; m < NUM_TEST; m++) {
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks / rwc_core_cnt[m];
+		for (n = 0; n < NUM_TEST; n++) {
+			unsigned int tot_lcore = rte_lcore_count();
+			if (tot_lcore < (rwc_core_cnt[n] +
+			     rwc_core_cnt[m] + 1))
+				goto finish;
+
+			printf("\nNumber of writers: %u", rwc_core_cnt[m]);
+			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
+
+			rte_hash_reset(tbl_rwc_test_param.h);
+			writer_done = 0;
+			write_type = WRITE_NO_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			write_type = WRITE_KEY_SHIFT;
+			if (write_keys(write_type) < 0)
+				goto err;
+			for (i = 0; i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				hash_data[i] =
+					rte_zmalloc(NULL, sizeof(uint32_t)
+						* rwc_core_cnt[NUM_TEST - 1],
+						0);
+				if (hash_data[i] == NULL) {
+					printf("No memory\n");
+					return -1;
+				}
+			}
+			for (i = 0;
+				i < tbl_rwc_test_param.count_keys_no_ks;
+				i++) {
+				if (rte_hash_add_key_data(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i,
+					(void *)((uintptr_t)hash_data[i]))
+					< 0) {
+					printf("Hash key add Failed #%d\n",
+						tbl_rwc_test_param.keys_no_ks[i]);
+					return -1;
+				}
+			}
+
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+			/* Launch reader(s) */
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				rte_eal_remote_launch(test_rwc_rcu_reader,
+					(void *)(uintptr_t)rwc_core_cnt[m],
+					enabled_core_ids[i]);
+			pos_core = 0;
+			/* Launch writers */
+			for (; i <= rwc_core_cnt[m]
+			     + rwc_core_cnt[n];	i++) {
+				rte_eal_remote_launch
+					(test_rwc_rcu_multi_writer,
+					(void *)(uintptr_t)pos_core,
+					enabled_core_ids[i]);
+				pos_core++;
+			}
+			/* Wait for writers to complete */
+			for (i = rwc_core_cnt[n] + 1;
+			     i <= rwc_core_cnt[m] + rwc_core_cnt[n];
+			     i++)
+				rte_eal_wait_lcore(enabled_core_ids[i]);
+
+			writer_done = 1;
+
+			for (i = 1; i <= rwc_core_cnt[n]; i++)
+				if (rte_eal_wait_lcore(enabled_core_ids[i]) < 0)
+					goto err;
+		}
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return 0;
+
+err:
+	rte_hash_free(tbl_rwc_test_param.h);
+	rte_free(t);
+	return -1;
+}
+
+static int
+test_multiwriter_perf(void *arg)
+{
+	uint32_t i, offset;
+	uint64_t begin, cycles;
+	uint8_t pos_core = (uint32_t)((uintptr_t)arg);
+	offset = pos_core * tbl_rwc_test_param.single_insert;
+
+	begin = rte_rdtsc_precise();
+	for (i = offset; i < offset + tbl_rwc_test_param.single_insert; i++) {
+		/* Delete element from the shared data structure */
+		rte_hash_del_key(tbl_rwc_test_param.h,
+					tbl_rwc_test_param.keys_no_ks + i);
+		rte_hash_add_key(tbl_rwc_test_param.h,
+				tbl_rwc_test_param.keys_no_ks + i);
+	}
+	cycles = rte_rdtsc_precise() - begin;
+	rte_atomic64_add(&gwrite_cycles, cycles);
+	rte_atomic64_add(&gwrites, tbl_rwc_test_param.single_insert);
+	return 0;
+}
+
+/*
+ * Test RCU perf:
+ * Writer(s) delete and add keys in the table.
+ */
+static int
+test_hash_writer_perf(struct rwc_perf *rwc_perf_results, int rwc_lf,
+				int htm, int ext_bkt)
+{
+	unsigned int n;
+	uint64_t i;
+	int use_jhash = 0;
+	uint8_t write_type;
+
+	no_free_on_del = 0;
+
+	rte_atomic64_init(&gwrites);
+	rte_atomic64_init(&gwrite_cycles);
+
+	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
+		goto err;
+
+	if (rwc_lf) {
+		uint32_t sz;
+		sz = rte_rcu_qsbr_get_memsize(TEST_RCU_MAX_LCORE);
+		t = (struct rte_rcu_qsbr *)rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
+		if (rte_hash_rcu_qsbr_add(tbl_rwc_test_param.h, t,
+					free_hash_data) < 0) {
+			printf("RCU init in hash failed\n");
+			return -1;
+		}
+		printf("\nTest: Writer perf with integrated RCU\n");
+	} else
+		printf("\nTest: Writer perf without integrated RCU\n");
+
+	for (n = 0; n < NUM_TEST; n++) {
+		unsigned int tot_lcore = rte_lcore_count();
+		if (tot_lcore < rwc_core_cnt[n] + 1)
+			goto finish;
+
+		/* Calculate keys added by each writer */
+		tbl_rwc_test_param.single_insert =
+			tbl_rwc_test_param.count_keys_no_ks /
+				rwc_core_cnt[n];
+		printf("\nNumber of writers: %u\n", rwc_core_cnt[n]);
+
+		rte_atomic64_clear(&gwrites);
+		rte_atomic64_clear(&gwrite_cycles);
+
+		rte_hash_reset(tbl_rwc_test_param.h);
+		if (rwc_lf)
+			rte_rcu_qsbr_init(t, TEST_RCU_MAX_LCORE);
+		write_type = WRITE_NO_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+		write_type = WRITE_KEY_SHIFT;
+		if (write_keys(write_type) < 0)
+			goto err;
+		for (i = 1; i <= rwc_core_cnt[n]; i++)
+			rte_eal_remote_launch(test_multiwriter_perf,
+					(void *)(uintptr_t)(i - 1),
+						enabled_core_ids[i]);
+
+		rte_eal_mp_wait_lcore();
+
+		unsigned long long cycles_per_write_operation =
+			rte_atomic64_read(&gwrite_cycles) /
+			rte_atomic64_read(&gwrites);
+		rwc_perf_results->writer_add_del[n]
+					= cycles_per_write_operation;
+		printf("Cycles per write operation: %llu\n",
+				cycles_per_write_operation);
+	}
+
+finish:
+	rte_hash_free(tbl_rwc_test_param.h);
+	if (rwc_lf)
+		rte_free(t);
+	return 0;
+
+err:
+	rte_eal_mp_wait_lcore();
+	rte_hash_free(tbl_rwc_test_param.h);
+	if (rwc_lf)
+		rte_free(t);
+	return -1;
+}
+
 static int
 test_hash_readwrite_lf_main(void)
 {
@@ -1272,7 +1940,6 @@ test_hash_readwrite_lf_main(void)
 		return -1;
 	if (get_enabled_cores_list() != 0)
 		return -1;
-
 	if (RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		rwc_lf = 1;
 		ext_bkt = 1;
@@ -1299,6 +1966,23 @@ test_hash_readwrite_lf_main(void)
 		if (test_hash_add_ks_lookup_hit_extbkt(&rwc_lf_results, rwc_lf,
 							htm, ext_bkt) < 0)
 			return -1;
+		ext_bkt = 0;
+		if (test_hash_rcu_free_on_del(&rwc_lf_results, rwc_lf, htm,
+						ext_bkt) < 0)
+			return -1;
+		if (test_hash_rcu_no_free_on_del(&rwc_lf_results, rwc_lf, htm,
+						ext_bkt) < 0)
+			return -1;
+		ext_bkt = 1;
+		if (test_hash_rcu_ext_bkt_free_on_del(&rwc_lf_results, rwc_lf,
+							htm, ext_bkt) < 0)
+			return -1;
+		if (test_hash_rcu_ext_bkt_no_free_on_del(&rwc_lf_results, rwc_lf
+							, htm, ext_bkt) < 0)
+			return -1;
+		if (test_hash_writer_perf(&rwc_lf_results, rwc_lf
+							, htm, ext_bkt) < 0)
+			return -1;
 	}
 	printf("\nTest lookup with read-write concurrency lock free support"
 	       " disabled\n");
@@ -1330,9 +2014,17 @@ test_hash_readwrite_lf_main(void)
 	if (test_hash_multi_add_lookup(&rwc_non_lf_results, rwc_lf, htm,
 							ext_bkt) < 0)
 		return -1;
+	no_free_on_del = 0;
 	if (test_hash_add_ks_lookup_hit_extbkt(&rwc_non_lf_results, rwc_lf,
 						htm, ext_bkt) < 0)
 		return -1;
+	if (test_hash_writer_perf(&rwc_non_lf_results, rwc_lf,
+						htm, ext_bkt) < 0)
+		return -1;
+
+	if (error_state != 0)
+		return -1;
+
 results:
 	printf("\n\t\t\t\t\t\t********** Results summary **********\n\n");
 	int i, j, k;
Signature · 1 line
2.17.1
#7 lore raw
In reply to Dharmik Thakkar
Quoted 38 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Dharmik Thakkar
Sent: Saturday, August 31, 2019 11:58 PM
Cc: [email protected]; [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [dpdk-dev] [RFC 0/3] RCU integration with hash library

This patchset integrates RCU QSBR support with hash library.

Note: This patchset has dependency on another patchset:

     https://patchwork.dpdk.org/cover/57813/

Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Add functional and performance test cases.

Fix 'ixgbe_ethdev.h' to avoid multiple definitions of 'bool'. (Please note that this fix is temporary. Recommend suggesting better solution)

Dharmik Thakkar (3):
net/ixgbe: avoid multpile definitions of 'bool'
lib/hash: integrate RCU QSBR
test/hash: add tests for integrated RCU QSBR

app/test/test_hash_readwrite_lf.c    | 700 ++++++++++++++++++++++++++-
drivers/net/ixgbe/ixgbe_ethdev.h     |   1 +
lib/librte_hash/Makefile             |   2 +-
lib/librte_hash/meson.build          |   2 +
lib/librte_hash/rte_cuckoo_hash.c    | 354 +++++++++++++-
lib/librte_hash/rte_cuckoo_hash.h    |   3 +
lib/librte_hash/rte_hash.h           |  38 +-
lib/librte_hash/rte_hash_version.map |   2 +-
8 files changed, 1065 insertions(+), 37 deletions(-)
Quoted 2 lines

--
2.17.1

[Wang, Yipeng]
Hi Dharmik, thanks for the code!

I agree with Stephen that many people may not use RCU,
and when they use, they may want some control in the application level. I can see the benefit to have certain RCU helper functions inside the libraries when people indeed uses QSBR as you designed,
but I am not sure if the benefits justify the effort to put it in many DPDK libraries. It adds new APIs, requires future maintenance, and so on, given the feature can be totally realized by the user application with small cost I assume.

I believe your plan is to add similar helper functions to many other libraries. I think we could consider them as a single proposal, and it makes sense to bring it up to the technical committee for a broader discussion.

#8 lore raw

RFC v2 1/1 lib/hash: integrate RCU QSBR

In reply to Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

---
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead

- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted

---
 lib/Makefile                         |   2 +-
 lib/librte_hash/Makefile             |   2 +-
 lib/librte_hash/meson.build          |   1 +
 lib/librte_hash/rte_cuckoo_hash.c    | 291 +++++++++++++++++++++------
 lib/librte_hash/rte_cuckoo_hash.h    |   8 +
 lib/librte_hash/rte_hash.h           |  75 ++++++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 7 files changed, 308 insertions(+), 73 deletions(-)
lib/Makefile +1 -1
index 8f5b68a2d469..dffe31c829f0 100644
@@ -51,7 +51,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += librte_vhost
 DEPDIRS-librte_vhost := librte_eal librte_mempool librte_mbuf librte_ethdev \
 			librte_net librte_hash librte_cryptodev
 DIRS-$(CONFIG_RTE_LIBRTE_HASH) += librte_hash
-DEPDIRS-librte_hash := librte_eal librte_ring
+DEPDIRS-librte_hash := librte_eal librte_ring librte_rcu
 DIRS-$(CONFIG_RTE_LIBRTE_EFD) += librte_efd
 DEPDIRS-librte_efd := librte_eal librte_ring librte_hash
 DIRS-$(CONFIG_RTE_LIBRTE_RIB) += librte_rib
lib/librte_hash/Makefile +1 -1
index ec9f86499262..10e697f48652 100644
@@ -8,7 +8,7 @@ LIB = librte_hash.a
 
 CFLAGS += -O3
 CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)
-LDLIBS += -lrte_eal -lrte_ring
+LDLIBS += -lrte_eal -lrte_ring -lrte_rcu
 
 EXPORT_MAP := rte_hash_version.map
 
lib/librte_hash/meson.build +1 -0
index 6ab46ae9d768..0977a63fd279 100644
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +225 -66

404 lines not on this page · raw message

lib/librte_hash/rte_cuckoo_hash.h +8 -0
index 345de6bf9cfd..85be49d3bbe7 100644
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
lib/librte_hash/rte_hash.h +71 -4

123 lines not on this page · raw message

lib/librte_hash/rte_hash_version.map +1 -1
index c0db81014ff9..c6d73080f478 100644
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
Signature · 1 line
2.17.1
#9 lore raw

Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR

CI has reported some compilation issues for this patch. I will resolve these issues once the RFC patch is approved.

Thank you!

Quoted 671 lines · Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

---
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead

- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted

---

lib/Makefile                         |   2 +-
lib/librte_hash/Makefile             |   2 +-
lib/librte_hash/meson.build          |   1 +
lib/librte_hash/rte_cuckoo_hash.c    | 291 +++++++++++++++++++++------
lib/librte_hash/rte_cuckoo_hash.h    |   8 +
lib/librte_hash/rte_hash.h           |  75 ++++++-
lib/librte_hash/rte_hash_version.map |   2 +-
7 files changed, 308 insertions(+), 73 deletions(-)
lib/Makefile +0 -0
index 8f5b68a2d469..dffe31c829f0 100644
@@ -51,7 +51,7 @@ DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += librte_vhost
 DEPDIRS-librte_vhost := librte_eal librte_mempool librte_mbuf librte_ethdev \
 			librte_net librte_hash librte_cryptodev

DIRS-$(CONFIG_RTE_LIBRTE_HASH) += librte_hash

lib/Makefile +1 -1
-DEPDIRS-librte_hash := librte_eal librte_ring
+DEPDIRS-librte_hash := librte_eal librte_ring librte_rcu

DIRS-$(CONFIG_RTE_LIBRTE_EFD) += librte_efd DEPDIRS-librte_efd := librte_eal librte_ring librte_hash DIRS-$(CONFIG_RTE_LIBRTE_RIB) += librte_rib

lib/librte_hash/Makefile +0 -0
index ec9f86499262..10e697f48652 100644
@@ -8,7 +8,7 @@ LIB = librte_hash.a
 

CFLAGS += -O3
CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR)

lib/librte_hash/Makefile +1 -1
-LDLIBS += -lrte_eal -lrte_ring
+LDLIBS += -lrte_eal -lrte_ring -lrte_rcu

EXPORT_MAP := rte_hash_version.map

lib/librte_hash/meson.build +0 -0
index 6ab46ae9d768..0977a63fd279 100644
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')

deps += ['ring']

lib/librte_hash/meson.build +1 -0
+deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +5 -0
index 0a6d474713a2..01c2cbe0e38b 100644
@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_rcu_dq_entry {
+	uint32_t key_idx;
+	uint32_t ext_bkt_idx; /**< Extended bkt index */
+};
+

struct rte_hash *
rte_hash_find_existing(const char *name)
{

lib/librte_hash/rte_cuckoo_hash.c +49 -1
@@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)
 
 	if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
 		readwrite_concur_lf_support = 1;
-		/* Enable not freeing internal memory/index on delete */
+		/* Enable not freeing internal memory/index on delete.
+		 * If internal RCU is enabled, freeing of internal memory/index
+		 * is done on delete
+		 */
 		no_free_on_del = 1;
 	}
 
@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up. */
+	if (h->dq)
+		rte_rcu_qsbr_dq_delete(h->dq);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -612,6 +624,16 @@ rte_hash_reset(struct rte_hash *h)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU clean up. */
+	if (h->hash_rcu_cfg->v) {
+		rte_rcu_qsbr_dq_delete(h->dq);
+		h->dq = NULL;
+		if (rte_hash_rcu_qsbr_add(h, h->hash_rcu_cfg) < 0) {
+			RTE_LOG(ERR, HASH, "RCU init failed\n");
+			return;
+		}
+	}
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket));
 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
 	*h->tbl_chng_cnt = 0;
@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct rte_hash *h,
 	return -ENOSPC;
 }
 
+static inline uint32_t
+alloc_slot(const struct rte_hash *h, struct lcore_cache *cached_free_slots)
+{
+	unsigned int  n_slots;
+	uint32_t slot_id;
+	if (h->use_local_cache) {
+		/* Try to get a free slot from the local cache */
+		if (cached_free_slots->len == 0) {
+			/* Need to get another burst of free slots from global ring */
+			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
+					cached_free_slots->objs,
+					sizeof(uint32_t),
+					LCORE_CACHE_SIZE, NULL);
+			if (n_slots == 0)
+				return EMPTY_SLOT;
+
+			cached_free_slots->len += n_slots;
+		}
+
+		/* Get a free slot from the local cache */
+		cached_free_slots->len--;
+		slot_id = cached_free_slots->objs[cached_free_slots->len];
+	} else {
+		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
+						sizeof(uint32_t)) != 0)
+			return EMPTY_SLOT;
+	}
+
+	return slot_id;
+}
+

static inline int32_t
__rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
hash_sig_t sig, void *data)

lib/librte_hash/rte_cuckoo_hash.c +142 -34
@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	uint32_t ext_bkt_id = 0;
 	uint32_t slot_id;
 	int ret;
-	unsigned n_slots;
 	unsigned lcore_id;
 	unsigned int i;
 	struct lcore_cache *cached_free_slots = NULL;
@@ -1001,28 +1053,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Try to get a free slot from the local cache */
-		if (cached_free_slots->len == 0) {
-			/* Need to get another burst of free slots from global ring */
-			n_slots = rte_ring_mc_dequeue_burst_elem(h->free_slots,
-					cached_free_slots->objs,
-					sizeof(uint32_t),
-					LCORE_CACHE_SIZE, NULL);
-			if (n_slots == 0) {
-				return -ENOSPC;
+	}
+	slot_id = alloc_slot(h, cached_free_slots);
+	if (slot_id == EMPTY_SLOT) {
+		if (h->hash_rcu_cfg->v) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+						    h->hash_rcu_cfg->reclaim_max,
+						    NULL, NULL, NULL)
+					== 0) {
+				slot_id = alloc_slot(h, cached_free_slots);
 			}
-
-			cached_free_slots->len += n_slots;
 		}
-
-		/* Get a free slot from the local cache */
-		cached_free_slots->len--;
-		slot_id = cached_free_slots->objs[cached_free_slots->len];
-	} else {
-		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
-						sizeof(uint32_t)) != 0) {
+		if (slot_id == EMPTY_SLOT)
 			return -ENOSPC;
-		}
 	}
 
 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size);
@@ -1118,8 +1161,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
 						sizeof(uint32_t)) != 0 ||
 					ext_bkt_id == 0) {
-		ret = -ENOSPC;
-		goto failure;
+		if (h->hash_rcu_cfg->v) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+						    h->hash_rcu_cfg->reclaim_max,
+						    NULL, NULL, NULL)
+					== 0) {
+				rte_ring_sc_dequeue_elem(h->free_ext_bkts,
+							 &ext_bkt_id,
+							 sizeof(uint32_t));
+			}
+		}
+		if (ext_bkt_id == 0) {
+			ret = -ENOSPC;
+			goto failure;
+		}
 	}
 
 	/* Use the first location of the new bucket */
@@ -1395,12 +1450,12 @@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
 	return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key), data);
 }
 
-static inline void
-remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+static int
+free_slot(const struct rte_hash *h, uint32_t slot_id)
 {
 	unsigned lcore_id, n_slots;
 	struct lcore_cache *cached_free_slots;
-
+	/* Return key indexes to free slot ring */
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
@@ -1411,18 +1466,122 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
 						cached_free_slots->objs,
 						sizeof(uint32_t),
 						LCORE_CACHE_SIZE, NULL);
-			ERR_IF_TRUE((n_slots == 0),
-				"%s: could not enqueue free slots in global ring\n",
-				__func__);
+			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
 			cached_free_slots->len -= n_slots;
 		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] =
-							bkt->key_idx[i];
-		cached_free_slots->len++;
+	}
+
+	enqueue_slot_back(h, cached_free_slots, slot_id);
+	return 0;
+}
+
+static void
+__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n)
+{
+	void *key_data = NULL;
+	int ret;
+	struct rte_hash_key *keys, *k;
+	struct rte_hash *h = (struct rte_hash *)p;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
+			*((struct __rte_hash_rcu_dq_entry *)e);
+
+	RTE_SET_USED(n);
+	keys = h->key_store;
+
+	k = (struct rte_hash_key *) ((char *)keys +
+				rcu_dq_entry.key_idx * h->key_entry_size);
+	key_data = k->pdata;
+	if (h->hash_rcu_cfg->free_key_data_func)
+		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg->key_data_ptr,
+						    key_data);
+
+	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx != EMPTY_SLOT)
+		/* Recycle empty ext bkt to free list. */
+		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
+			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
+
+	/* Return key indexes to free slot ring */
+	ret = free_slot(h, rcu_dq_entry.key_idx);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
+	}
+}
+
+int
+rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg)
+{
+	struct rte_rcu_qsbr_dq_parameters params = {0};
+	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
+
+	const uint32_t total_entries = h->use_local_cache ?
+		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
+							: h->entries + 1;
+
+	if ((h == NULL) || cfg == NULL) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config), 0);
+	if (hash_rcu_cfg == NULL) {
+		RTE_LOG(ERR, HASH, "memory allocation failed\n");
+		return 1;
+	}
+
+	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+		/* No other things to do. */
+	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
+		/* Init QSBR defer queue. */
+		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+					"HASH_RCU_%s", h->name);
+		params.name = rcu_dq_name;
+		params.size = cfg->dq_size;
+		if (params.size == 0)
+			params.size = total_entries;
+		params.trigger_reclaim_limit = cfg->reclaim_thd;
+		if (params.max_reclaim_size == 0)
+			params.max_reclaim_size = RTE_HASH_RCU_DQ_RECLAIM_MAX;
+		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
+		params.free_fn = __hash_rcu_qsbr_free_resource;
+		params.p = h;
+		params.v = cfg->v;
+		h->dq = rte_rcu_qsbr_dq_create(&params);
+		if (h->dq == NULL) {
+			rte_free(hash_rcu_cfg);
+			RTE_LOG(ERR, HASH, "HASH defer queue creation failed\n");
+			return 1;
+		}
 	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots,
-				&bkt->key_idx[i], sizeof(uint32_t));
+		rte_free(hash_rcu_cfg);
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg->v = cfg->v;
+	hash_rcu_cfg->mode = cfg->mode;
+	hash_rcu_cfg->dq_size = cfg->dq_size;
+	hash_rcu_cfg->reclaim_thd = cfg->reclaim_thd;
+	hash_rcu_cfg->reclaim_max = cfg->reclaim_max;
+	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
+	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
+
+	h->hash_rcu_cfg = hash_rcu_cfg;
+
+	return 0;
+}
+
+static inline void
+remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)
+{
+	int ret = free_slot(h, bkt->key_idx[i]);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
 	}
 }
 
@@ -1521,6 +1680,8 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t index = EMPTY_SLOT;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
 
 	short_sig = get_short_sig(sig);
 	prim_bucket_idx = get_prim_bucket_index(h, sig);
@@ -1555,10 +1716,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 
 /* Search last bucket to see if empty to be recycled */

return_bkt:

lib/librte_hash/rte_cuckoo_hash.c +3 -4
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
	while (last_bkt->next) {
		prev_bkt = last_bkt;
		last_bkt = last_bkt->next;
lib/librte_hash/rte_cuckoo_hash.c +26 -27
@@ -1571,11 +1731,11 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 	/* found empty bucket and recycle */
 	if (i == RTE_HASH_BUCKET_ENTRIES) {
 		prev_bkt->next = NULL;
-		uint32_t index = last_bkt - h->buckets_ext + 1;
+		index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
 		 * no_free_on_del is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->no_free_on_del) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.
 			 * When lock free read-write concurrency is enabled,
@@ -1583,12 +1743,32 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
+			if (h->hash_rcu_cfg->v == NULL)
+				h->ext_bkt_to_free[ret] = index;
+		} else
 			rte_ring_sp_enqueue_elem(h->free_ext_bkts, &index,
 							sizeof(uint32_t));
 	}
+
+return_key:
+	/* Using internal RCU QSBR */
+	if (h->hash_rcu_cfg->v) {
+		/* Key index where key is stored, adding the first dummy index */
+		rcu_dq_entry.key_idx = ret + 1;
+		rcu_dq_entry.ext_bkt_idx = index;
+		if (h->hash_rcu_cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+			/* Wait for quiescent state change. */
+			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
+						 RTE_QSBR_THRID_INVALID);
+			__hash_rcu_qsbr_free_resource((void *)((uintptr_t)h),
+						      &rcu_dq_entry, 1);
+		} else if (h->hash_rcu_cfg->mode == RTE_HASH_QSBR_MODE_DQ)
+			/* Push into QSBR FIFO. */
+			if (rte_rcu_qsbr_dq_enqueue(h->dq, &rcu_dq_entry) != 0)
+				RTE_LOG(ERR, HASH, "Failed to push QSBR FIFO\n");
+	}
 	__hash_rw_writer_unlock(h);
 	return ret;
 }
@@ -1637,8 +1817,6 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 
 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
 
-	unsigned int lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
 	const uint32_t total_entries = h->use_local_cache ?
 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1) + 1
 							: h->entries + 1;
@@ -1656,28 +1834,9 @@ rte_hash_free_key_with_position(const struct rte_hash *h,
 		}
 	}
 
-	if (h->use_local_cache) {
-		lcore_id = rte_lcore_id();
-		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Cache full, need to free it. */
-		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
-			/* Need to enqueue the free slots in global ring. */
-			n_slots = rte_ring_mp_enqueue_burst_elem(h->free_slots,
-						cached_free_slots->objs,
-						sizeof(uint32_t),
-						LCORE_CACHE_SIZE, NULL);
-			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
-			cached_free_slots->len -= n_slots;
-		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] = key_idx;
-		cached_free_slots->len++;
-	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
-						sizeof(uint32_t));
-	}
+	/* Enqueue slot to cache/ring of free slots. */
+	return free_slot(h, key_idx);
 
-	return 0;
 }
 

static inline void

lib/librte_hash/rte_cuckoo_hash.h +8 -0
index 345de6bf9cfd..85be49d3bbe7 100644
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
lib/librte_hash/rte_hash.h +49 -4
index bff40251bc98..5431bcf4aeb1 100644
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -45,7 +46,8 @@ extern "C" {
 /** Flag to disable freeing of key index on hash delete.
 * Refer to rte_hash_del_xxx APIs for more details.
 * This is enabled by default when RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
 */
 #define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10
 
@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */
 typedef int (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow application to free key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
+
 /**
 * Parameters used when creating the hash table.
 */
@@ -81,6 +90,39 @@ struct rte_hash_parameters {
 	uint8_t extra_flag;		/**< Indicate if additional parameters are present. */
 };
 
+/** RCU reclamation modes */
+enum rte_hash_qsbr_mode {
+	/** Create defer queue for reclaim. */
+	RTE_HASH_QSBR_MODE_DQ = 0,
+	/** Use blocking mode reclaim. No defer queue created. */
+	RTE_HASH_QSBR_MODE_SYNC
+};
+
+/** HASH RCU QSBR configuration structure. */
+struct rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t reclaim_thd;	/**< Threshold to trigger auto reclaim. */
+	uint32_t reclaim_max;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */
+};
+
 /** @internal A hash table structure. */
 struct rte_hash;
 
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t
 * Thread safety can be enabled by setting flag during
 * table creation.
 * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
 * the key index returned by rte_hash_add_key_xxx APIs will not be
 * freed by this API. rte_hash_free_key_with_position API must be called
 * additionally to free the index associated with the key.
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void *key);
 * Thread safety can be enabled by setting flag during
 * table creation.
 * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
 * the key index returned by rte_hash_add_key_xxx APIs will not be
 * freed by this API. rte_hash_free_key_with_position API must be called
 * additionally to free the index associated with the key.
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash *h, const int32_t position,
 * only be called from one thread by default. Thread safety
 * can be enabled by setting flag during table creation.
 * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
 * the key index returned by rte_hash_del_key_xxx APIs must be freed
 * using this API. This API should be called after all the readers
 * have stopped referencing the entry corresponding to this key.
@@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys,
 */

int32_t
rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);

lib/librte_hash/rte_hash.h +22 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);

#ifdef __cplusplus
}
#endif

lib/librte_hash/rte_hash_version.map +1 -1
index c0db81014ff9..c6d73080f478 100644
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
Signature · 1 line
2.17.1
#10 lore raw

Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR

In reply to Dharmik Thakkar
Quoted 46 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, August 18, 2020 9:06 PM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [RFC v2] lib/hash: integrate RCU QSBR

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

---
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead

- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted

---
 lib/Makefile                         |   2 +-
 lib/librte_hash/Makefile             |   2 +-
 lib/librte_hash/meson.build          |   1 +
 lib/librte_hash/rte_cuckoo_hash.c    | 291 +++++++++++++++++++++------
 lib/librte_hash/rte_cuckoo_hash.h    |   8 +
 lib/librte_hash/rte_hash.h           |  75 ++++++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 7 files changed, 308 insertions(+), 73 deletions(-)

<......>

Quoted 14 lines
lib/librte_hash/rte_hash.h +14 -0
+/** HASH RCU QSBR configuration structure. */ struct
+rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t reclaim_thd;	/**< Threshold to trigger auto reclaim. */
+	uint32_t reclaim_max;
+	/**< Max entries to reclaim in one go.

lib/librte_hash/rte_hash.h struct rte_hash_parameters { · line 125

Quoted 10 lines
lib/librte_hash/rte_hash.h +10 -0
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */ };
+

[Wang, Yipeng]
I guess this is mostly a wrapper of rte_rcu_qsbr_dq_parameters.
Personally, I incline to use variable names that match the existing qsbr parameters better. For example, you could still call reclaim_thd as reclaim_limit. And _max to be _size. Thus, people who are already familiar with qsbr can match the meanings better.

Quoted 37 lines

/** @internal A hash table structure. */ struct rte_hash;

(quoted diff) +0 -0
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h,

const void *key, hash_sig_t
* Thread safety can be enabled by setting flag during
* table creation.
* If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
* the key index returned by rte_hash_add_key_xxx APIs will not be
* freed by this API. rte_hash_free_key_with_position API must be called
* additionally to free the index associated with the key.

lib/librte_hash/rte_hash.h +2 -1
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void
 *key);
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called

* additionally to free the index associated with the key.

lib/librte_hash/rte_hash.h +2 -1
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash
 *h, const int32_t position,
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers

* have stopped referencing the entry corresponding to this key.

lib/librte_hash/rte_hash.h +0 -0
@@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const
 void **keys,

lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, const · line 680

Quoted 10 lines
lib/librte_hash/rte_hash.h +6 -0
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data,
 uint32_t *next);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.

[Wang, Yipeng] To enable RCU we need to call this func.
I think you can be more explicit, e.g. "This API should be called to enable the RCU support"

lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, · line 691

Quoted 13 lines
lib/librte_hash/rte_hash.h +13 -0
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */

[Wang, Yipeng] Is there any further requirement for when to call this API? E.g. you could say "this API should be called immediately after rte_hash_create()"

Quoted 19 lines
lib/librte_hash/rte_hash.h +3 -0
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);
 #ifdef __cplusplus

}
#endif

lib/librte_hash/rte_hash_version.map +1 -2
index c0db81014ff9..c6d73080f478 100644
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
--
Quoted 1 line

2.17.1

[Wang, Yipeng]
Hi, Dharmik,
Thanks for the patch. It generally looks good to me.
I guess you will revise documentation and the unit test as well after the RFC. That is helpful for users to understand how to use hash appropriately with the RCU lib.

#11 lore raw

Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR

Hi Yipeng,
Thank you for the comments!

Quoted 73 lines · On Aug 31, 2020, at 3:47 PM, Wang, Yipeng1 <[email protected]> wrote:
Quoted 47 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, August 18, 2020 9:06 PM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [RFC v2] lib/hash: integrate RCU QSBR

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

---
v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead

- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted

---

lib/Makefile                         |   2 +-
lib/librte_hash/Makefile             |   2 +-
lib/librte_hash/meson.build          |   1 +
lib/librte_hash/rte_cuckoo_hash.c    | 291 +++++++++++++++++++++------
lib/librte_hash/rte_cuckoo_hash.h    |   8 +
lib/librte_hash/rte_hash.h           |  75 ++++++-
lib/librte_hash/rte_hash_version.map |   2 +-
7 files changed, 308 insertions(+), 73 deletions(-)

<......>

Quoted 21 lines
lib/librte_hash/rte_hash.h +21 -0
+/** HASH RCU QSBR configuration structure. */ struct
+rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t reclaim_thd;	/**< Threshold to trigger auto reclaim. */
+	uint32_t reclaim_max;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
Quoted 9 lines

lib/librte_hash/rte_hash.h struct rte_hash_parameters { · line 125

Quoted 3 lines
lib/librte_hash/rte_hash.h +3 -0
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */ };
+

[Wang, Yipeng]
I guess this is mostly a wrapper of rte_rcu_qsbr_dq_parameters.
Personally, I incline to use variable names that match the existing qsbr parameters better. For example, you could still call reclaim_thd as reclaim_limit. And _max to be _size. Thus, people who are already familiar with qsbr can match the meanings better.

Makes sense. I will update it.

Quoted 42 lines
Quoted 42 lines

/** @internal A hash table structure. */ struct rte_hash;

(quoted diff) +0 -0
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash *h,

const void *key, hash_sig_t
* Thread safety can be enabled by setting flag during
* table creation.
* If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
* the key index returned by rte_hash_add_key_xxx APIs will not be
* freed by this API. rte_hash_free_key_with_position API must be called
* additionally to free the index associated with the key.

lib/librte_hash/rte_hash.h +2 -1
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const void
 *key);
 * Thread safety can be enabled by setting flag during
 * table creation.
 * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
 * the key index returned by rte_hash_add_key_xxx APIs will not be
 * freed by this API. rte_hash_free_key_with_position API must be called

* additionally to free the index associated with the key.

lib/librte_hash/rte_hash.h +2 -1
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct rte_hash
 *h, const int32_t position,
 * only be called from one thread by default. Thread safety
 * can be enabled by setting flag during table creation.
 * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
 * the key index returned by rte_hash_del_key_xxx APIs must be freed
 * using this API. This API should be called after all the readers

* have stopped referencing the entry corresponding to this key.

(quoted diff) +0 -0
@@ -625,6 +670,28 @@ rte_hash_lookup_bulk(const struct rte_hash *h, const
 void **keys,
 */

int32_t
rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32_t *next);

Quoted 9 lines

lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, · line 678

Quoted 6 lines
lib/librte_hash/rte_hash.h +6 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.

[Wang, Yipeng] To enable RCU we need to call this func.
I think you can be more explicit, e.g. "This API should be called to enable the RCU support"

Yes.

Quoted 13 lines
Quoted 6 lines

+ *
+ * @param h
+ * the hash object to add RCU QSBR
+ * @param cfg
+ * RCU QSBR configuration
+ * @return

Quoted 7 lines
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure

+ */

Quoted 3 lines

[Wang, Yipeng] Is there any further requirement for when to call this API? E.g. you could say "this API should be called immediately after rte_hash_create()"

Sure, I will add further guidelines/requirements.

Quoted 15 lines
Quoted 15 lines
lib/librte_hash/rte_hash.h +3 -0
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);

#ifdef __cplusplus
}
#endif

lib/librte_hash/rte_hash_version.map +0 -0
index c0db81014ff9..c6d73080f478 100644
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
Quoted 8 lines
Quoted 5 lines
lib/librte_hash/rte_hash_version.map +1 -2
-
+	rte_hash_rcu_qsbr_add;
 };
--

2.17.1

[Wang, Yipeng]
Hi, Dharmik,
Thanks for the patch. It generally looks good to me.

That’s great. I will convert it to a patch.

Quoted 2 lines

I guess you will revise documentation and the unit test as well after the RFC. That is helpful for users to understand how to use hash appropriately with the RCU lib.

Yes, I will add the documentation and unit test patches.

#12 lore raw

Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR

Hello Dharmik,

Quoted 10 lines · Dharmik Thakkar
Quoted 3 lines

[Wang, Yipeng]
Hi, Dharmik,
Thanks for the patch. It generally looks good to me.

That’s great. I will convert it to a patch.

Quoted 2 lines

I guess you will revise documentation and the unit test as well after the RFC. That is helpful for users to understand how to use hash appropriately with the RCU lib.

Yes, I will add the documentation and unit test patches.

I did not see a non RFC patch.
Is this still targeting 20.11?

Signature · 1 line
David Marchand
#13 lore raw

Re: [dpdk-dev] [RFC v2] lib/hash: integrate RCU QSBR

Hi David,

Quoted 11 lines · David Marchand

Hello Dharmik,

Quoted 6 lines · Dharmik Thakkar
Quoted 3 lines

[Wang, Yipeng]
Hi, Dharmik,
Thanks for the patch. It generally looks good to me.

That’s great. I will convert it to a patch.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

I guess you will revise documentation and the unit test as well after the RFC. That is helpful for users to understand how to use hash appropriately with the RCU lib.

Yes, I will add the documentation and unit test patches.

I did not see a non RFC patch.
Is this still targeting 20.11?

Yes, I will send out the patch series today. Sorry, it got delayed!

Quoted 4 lines
Signature · 2 lines
-- 
David Marchand
#14 lore raw

cover hash: integrate RCU QSBR

In reply to Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.
---
v3:
- Add documentation
- Add unit tests

v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead

- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted

Dharmik Thakkar (3):
lib/hash: integrate RCU QSBR
test/hash: replace rte atomic with C11 atomic APIs test/hash: add tests for integrated RCU QSBR

 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 260 +++++++++++++----
 doc/guides/prog_guide/hash_lib.rst     |  11 +-
 lib/librte_hash/meson.build            |   1 +
 lib/librte_hash/rte_cuckoo_hash.c      | 300 ++++++++++++++-----
 lib/librte_hash/rte_cuckoo_hash.h      |   8 +
 lib/librte_hash/rte_hash.h             |  77 ++++-
 lib/librte_hash/rte_hash_version.map   |   2 +-
 8 files changed, 917 insertions(+), 132 deletions(-)
Signature · 1 line
2.17.1
#15 lore raw

PATCH v3 1/3 lib/hash: integrate RCU QSBR

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

---
 doc/guides/prog_guide/hash_lib.rst   |  11 +-
 lib/librte_hash/meson.build          |   1 +
 lib/librte_hash/rte_cuckoo_hash.c    | 300 +++++++++++++++++++++------
 lib/librte_hash/rte_cuckoo_hash.h    |   8 +
 lib/librte_hash/rte_hash.h           |  77 ++++++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 6 files changed, 324 insertions(+), 75 deletions(-)
doc/guides/prog_guide/hash_lib.rst +8 -3
index d06c7de2ead1..63e183ed1f08 100644
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values
 *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
    by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position.
    Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position.
+   RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+   Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default).
 in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked
 list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the
 hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and
-deleted keys, to maintain the 100% capacity guarantee.
+Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR
+(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
 
 Implementation Details (non Extendable Bucket Case)
 ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the
 entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency',
 the position of the key is not freed. It is the responsibility of the user to free the position after
-readers are not referencing the position anymore.
+readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete
 
 
 Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below:
 *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set,
     wait till the readers are not referencing the position returned during add/delete flow and then free the position.
     RCU mechanisms can be used to find out when the readers are not referencing the position anymore.
+    RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+    Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
 
 *   Lookup flow: Lookup for the flow key in the hash.
     If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
lib/librte_hash/meson.build +1 -0
index 6ab46ae9d768..0977a63fd279 100644
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +233 -67

417 lines not on this page · raw message

lib/librte_hash/rte_cuckoo_hash.h +8 -0
index 345de6bf9cfd..85be49d3bbe7 100644
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
lib/librte_hash/rte_hash.h +73 -4

125 lines not on this page · raw message

lib/librte_hash/rte_hash_version.map +1 -1
index c0db81014ff9..c6d73080f478 100644
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
Signature · 1 line
2.17.1
Acked-by on v3 1/3
#16 lore raw

Re: [dpdk-dev] [PATCH v3 1/3] lib/hash: integrate RCU QSBR

Quoted 11 lines · Dharmik Thakkar

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

Quoted 9 lines
---
 doc/guides/prog_guide/hash_lib.rst   |  11 +-
 lib/librte_hash/meson.build          |   1 +
 lib/librte_hash/rte_cuckoo_hash.c    | 300 +++++++++++++++++++++------
 lib/librte_hash/rte_cuckoo_hash.h    |   8 +
 lib/librte_hash/rte_hash.h           |  77 ++++++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 6 files changed, 324 insertions(+), 75 deletions(-)
#17 lore raw

PATCH v3 2/3 test/hash: replace rte atomic with C11 atomic APIs

In reply to Dharmik Thakkar

Replace rte_atomic APIs with C11 atomic APIs in
test_hash_readwrite_lf_perf.c

---
 app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++---------------
 1 file changed, 36 insertions(+), 53 deletions(-)
app/test/test_hash_readwrite_lf_perf.c +36 -53
index 889799865c7b..328fa5116f65 100644
@@ -82,8 +82,8 @@ static struct {
 	struct rte_hash *h;
 } tbl_rwc_test_param;
 
-static rte_atomic64_t gread_cycles;
-static rte_atomic64_t greads;
+static uint64_t gread_cycles;
+static uint64_t greads;
 
 static volatile uint8_t writer_done;
 
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg)
 	} while (!writer_done);
 
 	cycles = rte_rdtsc_precise() - begin;
-	rte_atomic64_add(&gread_cycles, cycles);
-	rte_atomic64_add(&greads, read_cnt*loop_cnt);
+	__atomic_fetch_add(&gread_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);
 	return 0;
 }
 
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type = WRITE_NO_KEY_SHIFT;
 	uint8_t read_type = READ_PASS_NO_KEY_SHIFTS;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, read - hit\n");
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_hit[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t read_type = READ_FAIL;
 	int ret;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, Hash lookup - miss\n");
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_miss[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_NON_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_nsp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				goto finish;
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_sp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 	uint8_t write_type;
 	uint8_t read_type = READ_FAIL;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - miss\n");
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
 		}
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Multi-add-lookup\n");
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				printf("\nNumber of readers: %u\n",
 				       rwc_core_cnt[n]);
 
-				rte_atomic64_clear(&greads);
-				rte_atomic64_clear(&gread_cycles);
+				__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+				__atomic_store_n(&gread_cycles, 0,
+						 __ATOMIC_RELAXED);
 
 				rte_hash_reset(tbl_rwc_test_param.h);
 				writer_done = 0;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 						goto err;
 
 				unsigned long long cycles_per_lookup =
-					rte_atomic64_read(&gread_cycles)
-					/ rte_atomic64_read(&greads);
+					__atomic_load_n(&gread_cycles,
+							__ATOMIC_RELAXED) /
+					__atomic_load_n(&greads,
+							  __ATOMIC_RELAXED);
 				rwc_perf_results->multi_rw[m][k][n]
 					= cycles_per_lookup;
 				printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_KEY_SHIFTS_EXTBKT;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n");
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			write_type = WRITE_NO_KEY_SHIFT;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_extbkt[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
Signature · 1 line
2.17.1
#18 lore raw

PATCH v3 3/3 test/hash: add tests for integrated RCU QSBR

In reply to Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 171 ++++++++++-
 2 files changed, 557 insertions(+), 4 deletions(-)
app/test/test_hash.c +389 -1
index 990a1815f893..22b47b3e7728 100644
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {				\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {			\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {		\
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+				RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where failure"
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add DQ mode functional test.
+ * Reader and writer are in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Add 8 hash entries and fill the bucket
+ *  - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt
+ *  - Register a reader thread (not a real thread)
+ *  - Reader lookup existing entry
+ *  - Writer deletes the entry
+ *  - Reader lookup the entry
+ *  - Writer re-add the entry (no available free index)
+ *  - Reader report quiescent state and unregister
+ *  - Writer re-add the entry
+ *  - Reader lookup the entry
+ */
+static int
+test_hash_rcu_qsbr_dq_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_dq_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR DQ mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR DQ mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+					 "failed to add key (pos[%u]=%d)", i,
+					 pos[i]);
+		expected_pos[i] = pos[i];
+	}
+
+	/* Register pseudo reader */
+	status = rte_rcu_qsbr_thread_register(g_qsv, 0);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR thread registration failed");
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Writer update */
+	pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Del", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to del correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOENT,
+				 "found deleted key (pos[%u]=%d)", 0, pos[0]);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOSPC,
+				 "Added key successfully (pos[%u]=%d)", 0, pos[0]);
+
+	/* Reader quiescent */
+	rte_rcu_qsbr_quiescent(g_qsv, 0);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+				 "failed to add key (pos[%u]=%d)", 0, pos[0]);
+	expected_pos[0] = pos[0];
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+	return 0;
+
+}
+
+/* Report quiescent state interval every 1024 lookups. Larger critical
+ * sections in reader will result in writer polling multiple times.
+ */
+#define QSBR_REPORTING_INTERVAL 1024
+#define WRITER_ITERATIONS	512
+
+/*
+ * Reader thread using rte_hash data structure with RCU.
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	int i;
+
+	RTE_SET_USED(arg);
+	/* Register this thread to report quiescent state */
+	(void)rte_rcu_qsbr_thread_register(g_qsv, 0);
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	do {
+		for (i = 0; i < QSBR_REPORTING_INTERVAL; i++)
+			rte_hash_lookup(g_handle, &g_rand_keys[0]);
+
+		/* Update quiescent state */
+		rte_rcu_qsbr_quiescent(g_qsv, 0);
+	} while (!writer_done);
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add sync mode functional test.
+ * 1 Reader and 1 writer. They cannot be in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Register a reader thread. Reader keeps looking up a specific key.
+ *  - Writer keeps adding and deleting a specific key.
+ */
+static int
+test_hash_rcu_qsbr_sync_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_sync_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR sync mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR sync mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Launch reader thread */
+	rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+				rte_get_next_lcore(-1, 1, 0));
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+				"failed to add key (pos[%u]=%d)", i, pos[i]);
+		expected_pos[i] = pos[i];
+	}
+	writer_done = 0;
+
+	/* Writer Update */
+	for (i = 0; i < WRITER_ITERATIONS; i++) {
+		expected_pos[0] = pos[0];
+		pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Del", &g_rand_keys[0], status);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+					 "failed to del correct key (pos[%u]=%d)"
+					 , 0, pos[0]);
+
+		pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Add", &g_rand_keys[0], pos[0]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+					 "failed to add key (pos[%u]=%d)", 0,
+					 pos[0]);
+	}
+
+	writer_done = 1;
+	/* Wait until reader exited. */
+	rte_eal_mp_wait_lcore();
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return  0;
+
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1862,6 +2235,21 @@ test_hash(void)
 	if (test_crc32_hash_alg_equiv() < 0)
 		return -1;
 
+	if (test_hash_rcu_qsbr_add() < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(1) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(1) < 0)
+		return -1;
+
 	return 0;
 }
 
app/test/test_hash_readwrite_lf_perf.c +168 -3

225 lines not on this page · raw message

Signature · 1 line
2.17.1
#19 lore raw

Re: [dpdk-dev] [PATCH 0/3] hash: integrate RCU QSBR

In reply to Dharmik Thakkar
Quoted 12 lines · Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

Quoted 7 lines

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.

clang builds are broken.

http://mails.dpdk.org/archives/test-report/2020-October/160043.html
https://travis-ci.com/github/ovsrobot/dpdk/builds/190521500

Signature · 1 line
David Marchand
#20 lore raw

v4 cover hash: integrate RCU QSBR

In reply to Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.
---
v4:
- Fix clang compilation issues

v3:
- Add documentation
- Add unit tests

v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead

- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted

Dharmik Thakkar (3):
lib/hash: integrate RCU QSBR
test/hash: replace rte atomic with C11 atomic APIs test/hash: add tests for integrated RCU QSBR

 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 260 +++++++++++++----
 doc/guides/prog_guide/hash_lib.rst     |  11 +-
 lib/librte_hash/meson.build            |   1 +
 lib/librte_hash/rte_cuckoo_hash.c      | 302 ++++++++++++++-----
 lib/librte_hash/rte_cuckoo_hash.h      |   8 +
 lib/librte_hash/rte_hash.h             |  77 ++++-
 lib/librte_hash/rte_hash_version.map   |   2 +-
 8 files changed, 918 insertions(+), 133 deletions(-)
Signature · 1 line
2.17.1
#21 lore raw

PATCH v4 1/3 lib/hash: integrate RCU QSBR

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

---
 doc/guides/prog_guide/hash_lib.rst   |  11 +-
 lib/librte_hash/meson.build          |   1 +
 lib/librte_hash/rte_cuckoo_hash.c    | 302 +++++++++++++++++++++------
 lib/librte_hash/rte_cuckoo_hash.h    |   8 +
 lib/librte_hash/rte_hash.h           |  77 ++++++-
 lib/librte_hash/rte_hash_version.map |   2 +-
 6 files changed, 325 insertions(+), 76 deletions(-)
doc/guides/prog_guide/hash_lib.rst +8 -3
index d06c7de2ead1..63e183ed1f08 100644
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values
 *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
    by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position.
    Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position.
+   RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+   Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default).
 in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked
 list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the
 hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and
-deleted keys, to maintain the 100% capacity guarantee.
+Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR
+(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
 
 Implementation Details (non Extendable Bucket Case)
 ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the
 entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency',
 the position of the key is not freed. It is the responsibility of the user to free the position after
-readers are not referencing the position anymore.
+readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete
 
 
 Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below:
 *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set,
     wait till the readers are not referencing the position returned during add/delete flow and then free the position.
     RCU mechanisms can be used to find out when the readers are not referencing the position anymore.
+    RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+    Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
 
 *   Lookup flow: Lookup for the flow key in the hash.
     If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
lib/librte_hash/meson.build +1 -0
index 6ab46ae9d768..0977a63fd279 100644
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +234 -68

418 lines not on this page · raw message

lib/librte_hash/rte_cuckoo_hash.h +8 -0
index 345de6bf9cfd..85be49d3bbe7 100644
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
lib/librte_hash/rte_hash.h +73 -4

125 lines not on this page · raw message

lib/librte_hash/rte_hash_version.map +1 -1
index c0db81014ff9..c6d73080f478 100644
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
Signature · 1 line
2.17.1
#22 lore raw

PATCH v4 2/3 test/hash: replace rte atomic with C11 atomic APIs

In reply to Dharmik Thakkar

Replace rte_atomic APIs with C11 atomic APIs in
test_hash_readwrite_lf_perf.c

---
 app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++---------------
 1 file changed, 36 insertions(+), 53 deletions(-)
app/test/test_hash_readwrite_lf_perf.c +36 -53
index 889799865c7b..328fa5116f65 100644
@@ -82,8 +82,8 @@ static struct {
 	struct rte_hash *h;
 } tbl_rwc_test_param;
 
-static rte_atomic64_t gread_cycles;
-static rte_atomic64_t greads;
+static uint64_t gread_cycles;
+static uint64_t greads;
 
 static volatile uint8_t writer_done;
 
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg)
 	} while (!writer_done);
 
 	cycles = rte_rdtsc_precise() - begin;
-	rte_atomic64_add(&gread_cycles, cycles);
-	rte_atomic64_add(&greads, read_cnt*loop_cnt);
+	__atomic_fetch_add(&gread_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);
 	return 0;
 }
 
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type = WRITE_NO_KEY_SHIFT;
 	uint8_t read_type = READ_PASS_NO_KEY_SHIFTS;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, read - hit\n");
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_hit[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t read_type = READ_FAIL;
 	int ret;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, Hash lookup - miss\n");
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_miss[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_NON_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_nsp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				goto finish;
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_sp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 	uint8_t write_type;
 	uint8_t read_type = READ_FAIL;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - miss\n");
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
 		}
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Multi-add-lookup\n");
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				printf("\nNumber of readers: %u\n",
 				       rwc_core_cnt[n]);
 
-				rte_atomic64_clear(&greads);
-				rte_atomic64_clear(&gread_cycles);
+				__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+				__atomic_store_n(&gread_cycles, 0,
+						 __ATOMIC_RELAXED);
 
 				rte_hash_reset(tbl_rwc_test_param.h);
 				writer_done = 0;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 						goto err;
 
 				unsigned long long cycles_per_lookup =
-					rte_atomic64_read(&gread_cycles)
-					/ rte_atomic64_read(&greads);
+					__atomic_load_n(&gread_cycles,
+							__ATOMIC_RELAXED) /
+					__atomic_load_n(&greads,
+							  __ATOMIC_RELAXED);
 				rwc_perf_results->multi_rw[m][k][n]
 					= cycles_per_lookup;
 				printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_KEY_SHIFTS_EXTBKT;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n");
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			write_type = WRITE_NO_KEY_SHIFT;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_extbkt[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
Signature · 1 line
2.17.1
#23 lore raw

PATCH v4 3/3 test/hash: add tests for integrated RCU QSBR

In reply to Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 171 ++++++++++-
 2 files changed, 557 insertions(+), 4 deletions(-)
app/test/test_hash.c +389 -1
index 990a1815f893..22b47b3e7728 100644
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {				\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {			\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {		\
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+				RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where failure"
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add DQ mode functional test.
+ * Reader and writer are in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Add 8 hash entries and fill the bucket
+ *  - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt
+ *  - Register a reader thread (not a real thread)
+ *  - Reader lookup existing entry
+ *  - Writer deletes the entry
+ *  - Reader lookup the entry
+ *  - Writer re-add the entry (no available free index)
+ *  - Reader report quiescent state and unregister
+ *  - Writer re-add the entry
+ *  - Reader lookup the entry
+ */
+static int
+test_hash_rcu_qsbr_dq_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_dq_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR DQ mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR DQ mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+					 "failed to add key (pos[%u]=%d)", i,
+					 pos[i]);
+		expected_pos[i] = pos[i];
+	}
+
+	/* Register pseudo reader */
+	status = rte_rcu_qsbr_thread_register(g_qsv, 0);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR thread registration failed");
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Writer update */
+	pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Del", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to del correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOENT,
+				 "found deleted key (pos[%u]=%d)", 0, pos[0]);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOSPC,
+				 "Added key successfully (pos[%u]=%d)", 0, pos[0]);
+
+	/* Reader quiescent */
+	rte_rcu_qsbr_quiescent(g_qsv, 0);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+				 "failed to add key (pos[%u]=%d)", 0, pos[0]);
+	expected_pos[0] = pos[0];
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+	return 0;
+
+}
+
+/* Report quiescent state interval every 1024 lookups. Larger critical
+ * sections in reader will result in writer polling multiple times.
+ */
+#define QSBR_REPORTING_INTERVAL 1024
+#define WRITER_ITERATIONS	512
+
+/*
+ * Reader thread using rte_hash data structure with RCU.
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	int i;
+
+	RTE_SET_USED(arg);
+	/* Register this thread to report quiescent state */
+	(void)rte_rcu_qsbr_thread_register(g_qsv, 0);
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	do {
+		for (i = 0; i < QSBR_REPORTING_INTERVAL; i++)
+			rte_hash_lookup(g_handle, &g_rand_keys[0]);
+
+		/* Update quiescent state */
+		rte_rcu_qsbr_quiescent(g_qsv, 0);
+	} while (!writer_done);
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add sync mode functional test.
+ * 1 Reader and 1 writer. They cannot be in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Register a reader thread. Reader keeps looking up a specific key.
+ *  - Writer keeps adding and deleting a specific key.
+ */
+static int
+test_hash_rcu_qsbr_sync_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_sync_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR sync mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR sync mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Launch reader thread */
+	rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+				rte_get_next_lcore(-1, 1, 0));
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+				"failed to add key (pos[%u]=%d)", i, pos[i]);
+		expected_pos[i] = pos[i];
+	}
+	writer_done = 0;
+
+	/* Writer Update */
+	for (i = 0; i < WRITER_ITERATIONS; i++) {
+		expected_pos[0] = pos[0];
+		pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Del", &g_rand_keys[0], status);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+					 "failed to del correct key (pos[%u]=%d)"
+					 , 0, pos[0]);
+
+		pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Add", &g_rand_keys[0], pos[0]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+					 "failed to add key (pos[%u]=%d)", 0,
+					 pos[0]);
+	}
+
+	writer_done = 1;
+	/* Wait until reader exited. */
+	rte_eal_mp_wait_lcore();
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return  0;
+
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1862,6 +2235,21 @@ test_hash(void)
 	if (test_crc32_hash_alg_equiv() < 0)
 		return -1;
 
+	if (test_hash_rcu_qsbr_add() < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(1) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(1) < 0)
+		return -1;
+
 	return 0;
 }
 
app/test/test_hash_readwrite_lf_perf.c +168 -3

225 lines not on this page · raw message

Signature · 1 line
2.17.1
#24 lore raw

Re: [dpdk-dev] [PATCH v4 0/3] hash: integrate RCU QSBR

In reply to Dharmik Thakkar
Quoted 12 lines · Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

Quoted 7 lines

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.

This patch breaks Windows build: hash library was building fine so far, adding a dependency on rcu broke it as rcu is not ready (yet?).

Signature · 1 line
David Marchand
#25 lore raw

Re: [dpdk-dev] [PATCH v4 0/3] hash: integrate RCU QSBR

Quoted 19 lines · David Marchand
Quoted 14 lines · Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Quoted 9 lines
Quoted 5 lines

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.

This patch breaks Windows build: hash library was building fine so far, adding a dependency on rcu broke it as rcu is not ready (yet?).

Yes, ‘rcu’ is not currently enabled for Windows. I have submitted a RFC patch [1] to enable ‘rcu’ for Windows. I will update this patch, once ‘rcu’ support is enabled and the compilation succeeds on Windows.

[1] https://patches.dpdk.org/patch/81405/

Quoted 3 lines
Signature · 2 lines
-- 
David Marchand
#26 lore raw

v5 cover hash: integrate RCU QSBR

In reply to Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.
---
v5:
- Enable 'rcu' for Windows

v4:
- Fix clang compilation issues

v3:
- Add documentation
- Add unit tests

v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead

- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted

Dharmik Thakkar (4):
rcu: build on Windows
lib/hash: integrate RCU QSBR
test/hash: replace rte atomic with C11 atomic APIs test/hash: add tests for integrated RCU QSBR

 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 259 ++++++++++++----
 doc/guides/prog_guide/hash_lib.rst     |  11 +-
 lib/librte_hash/meson.build            |   1 +
 lib/librte_hash/rte_cuckoo_hash.c      | 302 ++++++++++++++-----
 lib/librte_hash/rte_cuckoo_hash.h      |   8 +
 lib/librte_hash/rte_hash.h             |  77 ++++-
 lib/librte_hash/version.map            |   2 +-
 lib/meson.build                        |   1 +
 9 files changed, 918 insertions(+), 133 deletions(-)
Signature · 1 line
2.17.1
#27 lore raw

PATCH v5 1/4 rcu: build on Windows

Build the lib for Windows.

---
 lib/meson.build | 1 +
 1 file changed, 1 insertion(+)
lib/meson.build +1 -0
index dd55b5cb53e4..1bb019720c6a 100644
@@ -41,6 +41,7 @@ if is_windows
 		'telemetry',
 		'eal',
 		'ring',
+		'rcu',
 		'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
 		'cmdline',
 		'hash',
Signature · 1 line
2.17.1
#28 lore raw

PATCH v5 2/4 lib/hash: integrate RCU QSBR

In reply to Dharmik Thakkar

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

---
 doc/guides/prog_guide/hash_lib.rst |  11 +-
 lib/librte_hash/meson.build        |   1 +
 lib/librte_hash/rte_cuckoo_hash.c  | 302 ++++++++++++++++++++++-------
 lib/librte_hash/rte_cuckoo_hash.h  |   8 +
 lib/librte_hash/rte_hash.h         |  77 +++++++-
 lib/librte_hash/version.map        |   2 +-
 6 files changed, 325 insertions(+), 76 deletions(-)
doc/guides/prog_guide/hash_lib.rst +8 -3
index d06c7de2ead1..63e183ed1f08 100644
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and writes the following flag values
 *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
    by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position.
    Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position.
+   RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+   Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag is not set by default).
 in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked
 list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the
 hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and
-deleted keys, to maintain the 100% capacity guarantee.
+Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR
+(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
 
 Implementation Details (non Extendable Bucket Case)
 ---------------------------------------------------
@@ -172,7 +175,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the
 entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency',
 the position of the key is not freed. It is the responsibility of the user to free the position after
-readers are not referencing the position anymore.
+readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete
 
 
 Implementation Details (with Extendable Bucket)
@@ -286,6 +289,8 @@ The flow table operations on the application side are described below:
 *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set,
     wait till the readers are not referencing the position returned during add/delete flow and then free the position.
     RCU mechanisms can be used to find out when the readers are not referencing the position anymore.
+    RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+    Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
 
 *   Lookup flow: Lookup for the flow key in the hash.
     If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
lib/librte_hash/meson.build +1 -0
index 6ab46ae9d768..0977a63fd279 100644
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +234 -68

418 lines not on this page · raw message

lib/librte_hash/rte_cuckoo_hash.h +8 -0
index 345de6bf9cfd..85be49d3bbe7 100644
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
lib/librte_hash/rte_hash.h +73 -4

125 lines not on this page · raw message

lib/librte_hash/version.map +1 -1
index c0db81014ff9..c6d73080f478 100644
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
Signature · 1 line
2.17.1
#29 lore raw

Re: [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR

Quoted 43 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 2/4] lib/hash: integrate RCU QSBR

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

---
 doc/guides/prog_guide/hash_lib.rst |  11 +-
 lib/librte_hash/meson.build        |   1 +
 lib/librte_hash/rte_cuckoo_hash.c  | 302 ++++++++++++++++++++++-------
 lib/librte_hash/rte_cuckoo_hash.h  |   8 +
 lib/librte_hash/rte_hash.h         |  77 +++++++-
 lib/librte_hash/version.map        |   2 +-
 6 files changed, 325 insertions(+), 76 deletions(-)
doc/guides/prog_guide/hash_lib.rst +0 -0
index d06c7de2ead1..63e183ed1f08 100644
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and

writes the following flag values
* If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL)
flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
by default when the lock free read/write concurrency flag is set. The

Quoted 8 lines

application should free the position after all the readers have stopped referencing the position.
Where required, the application can make use of RCU mechanisms to
determine when the readers have stopped referencing the position.
+ RCU QSBR process is integrated within the Hash library for safe freeing of
the position. Application has certain responsibilities while using this feature.
+ Please refer to resource reclamation framework of :ref:`RCU library
<RCU_Library>` for more details.

[Yipeng]: Maybe also add: rte_hash_rcu_qsbr_add() need to be called to use the embedded RCU mechanism. Just to give user a pointer to which API to look.

Quoted 120 lines
doc/guides/prog_guide/hash_lib.rst +1 -0
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag

is not set by default).
in the very unlikely case due to excessive hash collisions that a key has failed
to be inserted, the hash table bucket is extended with a linked list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and -deleted keys, to maintain the 100% capacity guarantee.

doc/guides/prog_guide/hash_lib.rst +2 -0
+Please note that with the 'lock free read/write concurrency' flag
+enabled, users need to call 'rte_hash_free_key_with_position' API or

configure integrated RCU QSBR (or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.

Implementation Details (non Extendable Bucket Case)
---------------------------------------------------

doc/guides/prog_guide/hash_lib.rst +0 -0
@@ -172,7 +175,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If

the key is found, the entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency', the position of the key is not freed. It is the responsibility of the user to free the position after -readers are not referencing the position anymore.

doc/guides/prog_guide/hash_lib.rst +3 -0
+readers are not referencing the position anymore. User can configure
+integrated RCU QSBR or use external RCU mechanisms to safely free the
+position on delete
 
 
 Implementation Details (with Extendable Bucket) @@ -286,6 +289,8 @@ The

flow table operations on the application side are described below:
* Free flow: Free flow key position. If 'no free on delete' or 'lock-free
read/write concurrency' flags are set,
wait till the readers are not referencing the position returned during
add/delete flow and then free the position.
RCU mechanisms can be used to find out when the readers are not
referencing the position anymore.
+ RCU QSBR process is integrated within the Hash library for safe freeing of
the position. Application has certain responsibilities while using this feature.
+ Please refer to resource reclamation framework of :ref:`RCU library
<RCU_Library>` for more details.

* Lookup flow: Lookup for the flow key in the hash.
If the returned position is valid (flow lookup hit), use the returned position
to access the flow entry in the flow table.

lib/librte_hash/meson.build index renamed from lib/librte_hash/meson.build +0 -0

6ab46ae9d768..0977a63fd279 100644

lib/librte_hash/meson.build +1 -0
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')  deps += ['ring']
+deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +4 -0
index aad0c965be5e..b9e4d82a0c14 100644
@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {  };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_rcu_dq_entry {
+	uint32_t key_idx;
+	uint32_t ext_bkt_idx; /**< Extended bkt index */ };
+
 struct rte_hash *
 rte_hash_find_existing(const char *name)  { @@ -210,7 +215,10 @@
 rte_hash_create(const struct rte_hash_parameters *params)
 
 	if (params->extra_flag &

RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {

lib/librte_hash/rte_cuckoo_hash.c +18 -1
 		readwrite_concur_lf_support = 1;
-		/* Enable not freeing internal memory/index on delete */
+		/* Enable not freeing internal memory/index on delete.
+		 * If internal RCU is enabled, freeing of internal memory/index
+		 * is done on delete
+		 */
 		no_free_on_del = 1;
 	}
@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up. */
+	if (h->dq)
+		rte_rcu_qsbr_dq_delete(h->dq);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -607,11 +619,21 @@ void
 rte_hash_reset(struct rte_hash *h)
 {
 	uint32_t tot_ring_cnt, i;
+	unsigned int pending;
 
 	if (h == NULL)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU QSBR clean up. */
+	if (h->dq) {
+		/* Reclaim all the resources */
+		rte_rcu_qsbr_dq_reclaim(h->dq, ~0, NULL, &pending, NULL);
+		if (pending != 0)
+			RTE_LOG(ERR, HASH, "RCU reclaim all resources
 failed\n");
+	}
+
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct
 rte_hash_bucket));
lib/librte_hash/rte_cuckoo_hash.c +0 -0
 	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
 	*h->tbl_chng_cnt = 0;

lib/librte_hash/rte_cuckoo_hash.c rte_hash_cuckoo_make_space_mw(const struct · line 982

Quoted 9 lines
lib/librte_hash/rte_cuckoo_hash.c +5 -0
@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct
 rte_hash *h,
 	return -ENOSPC;
 }
 
+static inline uint32_t
+alloc_slot(const struct rte_hash *h, struct lcore_cache
+*cached_free_slots) {
+	unsigned int  n_slots;
+	uint32_t slot_id;

[Yipeng]: Blank line after variable declaration.

Quoted 312 lines
lib/librte_hash/rte_cuckoo_hash.c +4 -0
+	if (h->use_local_cache) {
+		/* Try to get a free slot from the local cache */
+		if (cached_free_slots->len == 0) {
+			/* Need to get another burst of free slots from global

ring */

lib/librte_hash/rte_cuckoo_hash.c +1 -0
+			n_slots = rte_ring_mc_dequeue_burst_elem(h-
Quoted 1 line

free_slots,

lib/librte_hash/rte_cuckoo_hash.c +21 -1
+					cached_free_slots->objs,
+					sizeof(uint32_t),
+					LCORE_CACHE_SIZE, NULL);
+			if (n_slots == 0)
+				return EMPTY_SLOT;
+
+			cached_free_slots->len += n_slots;
+		}
+
+		/* Get a free slot from the local cache */
+		cached_free_slots->len--;
+		slot_id = cached_free_slots->objs[cached_free_slots->len];
+	} else {
+		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
+						sizeof(uint32_t)) != 0)
+			return EMPTY_SLOT;
+	}
+
+	return slot_id;
+}
+
 static inline int32_t
 __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
 						hash_sig_t sig, void *data)
@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct
 rte_hash *h, const void *key,
 	uint32_t ext_bkt_id = 0;
 	uint32_t slot_id;
 	int ret;
-	unsigned n_slots;
 	unsigned lcore_id;
 	unsigned int i;

struct lcore_cache *cached_free_slots = NULL; @@ -1001,28
+1053,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,

lib/librte_hash/rte_cuckoo_hash.c +0 -3
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Try to get a free slot from the local cache */
-		if (cached_free_slots->len == 0) {
-			/* Need to get another burst of free slots from global

ring */

lib/librte_hash/rte_cuckoo_hash.c +0 -1
-			n_slots = rte_ring_mc_dequeue_burst_elem(h-
Quoted 1 line

free_slots,

lib/librte_hash/rte_cuckoo_hash.c +12 -16
-					cached_free_slots->objs,
-					sizeof(uint32_t),
-					LCORE_CACHE_SIZE, NULL);
-			if (n_slots == 0) {
-				return -ENOSPC;
-			}
-
-			cached_free_slots->len += n_slots;
+	}
+	slot_id = alloc_slot(h, cached_free_slots);
+	if (slot_id == EMPTY_SLOT) {
+		if (h->dq) {
+			__hash_rw_writer_lock(h);
+			ret = rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL);
+			__hash_rw_writer_unlock(h);
+			if (ret == 0)
+				slot_id = alloc_slot(h, cached_free_slots);
 		}
-
-		/* Get a free slot from the local cache */
-		cached_free_slots->len--;
-		slot_id = cached_free_slots->objs[cached_free_slots->len];
-	} else {
-		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
-						sizeof(uint32_t)) != 0) {
+		if (slot_id == EMPTY_SLOT)
 			return -ENOSPC;
-		}
 	}
 
 	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size); @@ -

1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,

lib/librte_hash/rte_cuckoo_hash.c +5 -2
 	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
 						sizeof(uint32_t)) != 0 ||
 					ext_bkt_id == 0) {
-		ret = -ENOSPC;
-		goto failure;
+		if (h->dq) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL) == 0) {
+				rte_ring_sc_dequeue_elem(h-
Quoted 1 line

free_ext_bkts,

lib/librte_hash/rte_cuckoo_hash.c +8 -0
+							 &ext_bkt_id,
+							 sizeof(uint32_t));
+			}
+		}
+		if (ext_bkt_id == 0) {
+			ret = -ENOSPC;
+			goto failure;
+		}
 	}
 
 	/* Use the first location of the new bucket */ @@ -1395,12 +1450,12

@@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key),
data); }

lib/librte_hash/rte_cuckoo_hash.c +0 -2
-static inline void
-remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,

unsigned i)

lib/librte_hash/rte_cuckoo_hash.c +4 -2
+static int
+free_slot(const struct rte_hash *h, uint32_t slot_id)
 {
 	unsigned lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
-
+	struct lcore_cache *cached_free_slots = NULL;
+	/* Return key indexes to free slot ring */
 	if (h->use_local_cache) {
 		lcore_id = rte_lcore_id();
 		cached_free_slots = &h->local_free_slots[lcore_id]; @@ -

1411,18 +1466,127 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i)

lib/librte_hash/rte_cuckoo_hash.c +0 -2
 						cached_free_slots->objs,
 						sizeof(uint32_t),
 						LCORE_CACHE_SIZE, NULL);
-			ERR_IF_TRUE((n_slots == 0),
-				"%s: could not enqueue free slots in global

ring\n",

lib/librte_hash/rte_cuckoo_hash.c +24 -5
-				__func__);
+			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
 			cached_free_slots->len -= n_slots;
 		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] =
-							bkt->key_idx[i];
-		cached_free_slots->len++;
+	}
+
+	enqueue_slot_back(h, cached_free_slots, slot_id);
+	return 0;
+}
+
+static void
+__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n) {
+	void *key_data = NULL;
+	int ret;
+	struct rte_hash_key *keys, *k;
+	struct rte_hash *h = (struct rte_hash *)p;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
+			*((struct __rte_hash_rcu_dq_entry *)e);
+
+	RTE_SET_USED(n);
+	keys = h->key_store;
+
+	k = (struct rte_hash_key *) ((char *)keys +
+				rcu_dq_entry.key_idx * h->key_entry_size);
+	key_data = k->pdata;
+	if (h->hash_rcu_cfg->free_key_data_func)
+		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg-
Quoted 1 line

key_data_ptr,

lib/librte_hash/rte_cuckoo_hash.c +3 -0
+						    key_data);
+
+	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx !=

EMPTY_SLOT)

lib/librte_hash/rte_cuckoo_hash.c +37 -0
+		/* Recycle empty ext bkt to free list. */
+		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
+			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
+
+	/* Return key indexes to free slot ring */
+	ret = free_slot(h, rcu_dq_entry.key_idx);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
+	}
+}
+
+int
+rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg)
+{
+	struct rte_rcu_qsbr_dq_parameters params = {0};
+	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
+
+	const uint32_t total_entries = h->use_local_cache ?
+		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1)
+ 1
+							: h->entries + 1;
+
+	if ((h == NULL) || cfg == NULL || cfg->v == NULL) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	if (h->hash_rcu_cfg) {
+		rte_errno = EEXIST;
+		return 1;
+	}
+
+	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config),

0);

lib/librte_hash/rte_cuckoo_hash.c +18 -0
+	if (hash_rcu_cfg == NULL) {
+		RTE_LOG(ERR, HASH, "memory allocation failed\n");
+		return 1;
+	}
+
+	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+		/* No other things to do. */
+	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
+		/* Init QSBR defer queue. */
+		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+					"HASH_RCU_%s", h->name);
+		params.name = rcu_dq_name;
+		params.size = cfg->dq_size;
+		if (params.size == 0)
+			params.size = total_entries;
+		params.trigger_reclaim_limit = cfg->trigger_reclaim_limit;
+		if (params.max_reclaim_size == 0)
+			params.max_reclaim_size =

RTE_HASH_RCU_DQ_RECLAIM_MAX;

lib/librte_hash/rte_cuckoo_hash.c +8 -0
+		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
+		params.free_fn = __hash_rcu_qsbr_free_resource;
+		params.p = h;
+		params.v = cfg->v;
+		h->dq = rte_rcu_qsbr_dq_create(&params);
+		if (h->dq == NULL) {
+			rte_free(hash_rcu_cfg);
+			RTE_LOG(ERR, HASH, "HASH defer queue creation

failed\n");

lib/librte_hash/rte_cuckoo_hash.c +30 -2
+			return 1;
+		}
 	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots,
-				&bkt->key_idx[i], sizeof(uint32_t));
+		rte_free(hash_rcu_cfg);
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg->v = cfg->v;
+	hash_rcu_cfg->mode = cfg->mode;
+	hash_rcu_cfg->dq_size = params.size;
+	hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit;
+	hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size;
+	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
+	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
+
+	h->hash_rcu_cfg = hash_rcu_cfg;
+
+	return 0;
+}
+
+static inline void
+remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,
+unsigned i) {
+	int ret = free_slot(h, bkt->key_idx[i]);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
 	}
 }
@@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct
 rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t index = EMPTY_SLOT;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
 
 	short_sig = get_short_sig(sig);

prim_bucket_idx = get_prim_bucket_index(h, sig); @@ -1555,10
+1721,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,

/* Search last bucket to see if empty to be recycled */ return_bkt:

lib/librte_hash/rte_cuckoo_hash.c +5 -6
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
 	while (last_bkt->next) {
 		prev_bkt = last_bkt;
 		last_bkt = last_bkt->next;
@@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct
 rte_hash *h, const void *key,
 	/* found empty bucket and recycle */
 	if (i == RTE_HASH_BUCKET_ENTRIES) {
 		prev_bkt->next = NULL;
-		uint32_t index = last_bkt - h->buckets_ext + 1;
+		index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
 		 * no_free_on_del is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->no_free_on_del) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.
lib/librte_hash/rte_cuckoo_hash.c +0 -0
 			 * When lock free read-write concurrency is enabled,

lib/librte_hash/rte_cuckoo_hash.c __rte_hash_del_key_with_hash(const struct · line 1755

Quoted 10 lines
lib/librte_hash/rte_cuckoo_hash.c +3 -2
@@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const struct
 rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
+			if (h->hash_rcu_cfg == NULL)
+				h->ext_bkt_to_free[ret] = index;

[Yipeng]: If using embedded qsbr (not NULL), how did you recycle the ext bkt?

Quoted 216 lines
lib/librte_hash/rte_cuckoo_hash.c +6 -0
+		} else
 			rte_ring_sp_enqueue_elem(h->free_ext_bkts,
 &index,
 							sizeof(uint32_t));
 	}
+
+return_key:
+	/* Using internal RCU QSBR */
+	if (h->hash_rcu_cfg) {
+		/* Key index where key is stored, adding the first dummy

index */

lib/librte_hash/rte_cuckoo_hash.c +9 -0
+		rcu_dq_entry.key_idx = ret + 1;
+		rcu_dq_entry.ext_bkt_idx = index;
+		if (h->dq == NULL) {
+			/* Wait for quiescent state change if using
+			 * RTE_HASH_QSBR_MODE_SYNC
+			 */
+			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
+						 RTE_QSBR_THRID_INVALID);
+			__hash_rcu_qsbr_free_resource((void

*)((uintptr_t)h),

lib/librte_hash/rte_cuckoo_hash.c +3 -0
+						      &rcu_dq_entry, 1);
+		} else if (h->dq)
+			/* Push into QSBR FIFO if using

RTE_HASH_QSBR_MODE_DQ */

lib/librte_hash/rte_cuckoo_hash.c +1 -0
+			if (rte_rcu_qsbr_dq_enqueue(h->dq,

&rcu_dq_entry) != 0)

lib/librte_hash/rte_cuckoo_hash.c +1 -0
+				RTE_LOG(ERR, HASH, "Failed to push QSBR

FIFO\n");

lib/librte_hash/rte_cuckoo_hash.c +4 -23
+	}
 	__hash_rw_writer_unlock(h);
 	return ret;
 }
@@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const struct
 rte_hash *h,
 
 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
 
-	unsigned int lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
 	const uint32_t total_entries = h->use_local_cache ?
 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1)
+ 1
 							: h->entries + 1;
@@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const struct
 rte_hash *h,
 		}
 	}
 
-	if (h->use_local_cache) {
-		lcore_id = rte_lcore_id();
-		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Cache full, need to free it. */
-		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
-			/* Need to enqueue the free slots in global ring. */
-			n_slots = rte_ring_mp_enqueue_burst_elem(h-
 >free_slots,
-						cached_free_slots->objs,
-						sizeof(uint32_t),
-						LCORE_CACHE_SIZE, NULL);
-			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
-			cached_free_slots->len -= n_slots;
-		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] = key_idx;
-		cached_free_slots->len++;
-	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
-						sizeof(uint32_t));
-	}
+	/* Enqueue slot to cache/ring of free slots. */
+	return free_slot(h, key_idx);
 
-	return 0;
 }

static inline void

lib/librte_hash/rte_cuckoo_hash.h +8 -0
index 345de6bf9cfd..85be49d3bbe7 100644
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
lib/librte_hash/rte_hash.h index renamed from lib/librte_hash/rte_hash.h +0 -0

bff40251bc98..3d28f177f14a 100644

lib/librte_hash/rte_hash.h +1 -0
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -45,7 +46,8 @@ extern "C" {
 /** Flag to disable freeing of key index on hash delete.
  * Refer to rte_hash_del_xxx APIs for more details.
  * This is enabled by default when

RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
*/
#define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10

lib/librte_hash/rte_hash.h +8 -0
@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void
 *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */  typedef int
 (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow application to free
+key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
+
 /**
  * Parameters used when creating the hash table.
  */
@@ -81,6 +90,39 @@ struct rte_hash_parameters {
 	uint8_t extra_flag;		/**< Indicate if additional parameters

are present. */
};

lib/librte_hash/rte_hash.h +20 -0
+/** RCU reclamation modes */
+enum rte_hash_qsbr_mode {
+	/** Create defer queue for reclaim. */
+	RTE_HASH_QSBR_MODE_DQ = 0,
+	/** Use blocking mode reclaim. No defer queue created. */
+	RTE_HASH_QSBR_MODE_SYNC
+};
+
+/** HASH RCU QSBR configuration structure. */ struct
+rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t trigger_reclaim_limit;	/**< Threshold to trigger auto reclaim.

*/

lib/librte_hash/rte_hash.h +14 -1
+	uint32_t max_reclaim_size;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */ };
+
 /** @internal A hash table structure. */  struct rte_hash;
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash
 *h, const void *key, hash_sig_t
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called

* additionally to free the index associated with the key.

lib/librte_hash/rte_hash.h +2 -1
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const
 void *key);
  * Thread safety can be enabled by setting flag during
  * table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_add_key_xxx APIs will not be
  * freed by this API. rte_hash_free_key_with_position API must be called

* additionally to free the index associated with the key.

lib/librte_hash/rte_hash.h +2 -1
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct
 rte_hash *h, const int32_t position,
  * only be called from one thread by default. Thread safety
  * can be enabled by setting flag during table creation.
  * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
  * the key index returned by rte_hash_del_key_xxx APIs must be freed
  * using this API. This API should be called after all the readers

* have stopped referencing the entry corresponding to this key.

lib/librte_hash/rte_hash.h +0 -0
@@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h,
 const void **keys,

lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, · line 680

Quoted 10 lines
lib/librte_hash/rte_hash.h +6 -0
  */
 int32_t
 rte_hash_iterate(const struct rte_hash *h, const void **key, void **data,
 uint32_t *next);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.

[Yipeng]: a Hash object

Quoted 34 lines
lib/librte_hash/rte_hash.h +19 -0
+ * This API should be called to enable the integrated RCU QSBR support
+and
+ * should be called immediately after creating the Hash object.
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);
 #ifdef __cplusplus
 }
 #endif
lib/librte_hash/version.map index renamed from lib/librte_hash/version.map +0 -0

c0db81014ff9..c6d73080f478 100644

lib/librte_hash/version.map +1 -2
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
--
Quoted 1 line

2.17.1

[Yipeng]:

Hi, Dharmik, thanks for the work! It generally looks good.
Just some minor issues to address and one question for the ext table inlined.

Thanks!

#30 lore raw

Re: [dpdk-dev] [PATCH v5 2/4] lib/hash: integrate RCU QSBR

Quoted 50 lines · On Oct 20, 2020, at 9:42 PM, Wang, Yipeng1 <[email protected]> wrote:
Quoted 47 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 2/4] lib/hash: integrate RCU QSBR

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

---

doc/guides/prog_guide/hash_lib.rst |  11 +-
lib/librte_hash/meson.build        |   1 +
lib/librte_hash/rte_cuckoo_hash.c  | 302 ++++++++++++++++++++++-------
lib/librte_hash/rte_cuckoo_hash.h  |   8 +
lib/librte_hash/rte_hash.h         |  77 +++++++-
lib/librte_hash/version.map        |   2 +-
6 files changed, 325 insertions(+), 76 deletions(-)
doc/guides/prog_guide/hash_lib.rst +0 -0
index d06c7de2ead1..63e183ed1f08 100644
@@ -102,6 +102,9 @@ For concurrent writes, and concurrent reads and

writes the following flag values
* If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
by default when the lock free read/write concurrency flag is set. The
application should free the position after all the readers have stopped referencing the position.
Where required, the application can make use of RCU mechanisms to

Quoted 8 lines
Quoted 5 lines

determine when the readers have stopped referencing the position.
+ RCU QSBR process is integrated within the Hash library for safe freeing of
the position. Application has certain responsibilities while using this feature.
+ Please refer to resource reclamation framework of :ref:`RCU library
<RCU_Library>` for more details.

[Yipeng]: Maybe also add: rte_hash_rcu_qsbr_add() need to be called to use the embedded RCU mechanism. Just to give user a pointer to which API to look.

Copy.

Quoted 124 lines
Quoted 124 lines

+

Extendable Bucket Functionality support
----------------------------------------

doc/guides/prog_guide/hash_lib.rst +0 -0
@@ -109,8 +112,8 @@ An extra flag is used to enable this functionality (flag

is not set by default).
in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and -deleted keys, to maintain the 100% capacity guarantee.

doc/guides/prog_guide/hash_lib.rst +2 -0
+Please note that with the 'lock free read/write concurrency' flag
+enabled, users need to call 'rte_hash_free_key_with_position' API or

configure integrated RCU QSBR (or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.

Implementation Details (non Extendable Bucket Case)
---------------------------------------------------

doc/guides/prog_guide/hash_lib.rst +0 -0
@@ -172,7 +175,7 @@ Example of deletion:

Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency', the position of the key is not freed. It is the responsibility of the user to free the position after -readers are not referencing the position anymore.

doc/guides/prog_guide/hash_lib.rst +3 -0
+readers are not referencing the position anymore. User can configure
+integrated RCU QSBR or use external RCU mechanisms to safely free the
+position on delete

Implementation Details (with Extendable Bucket) @@ -286,6 +289,8 @@ The flow table operations on the application side are described below:
* Free flow: Free flow key position. If 'no free on delete' or 'lock-free
read/write concurrency' flags are set,
wait till the readers are not referencing the position returned during
add/delete flow and then free the position.
RCU mechanisms can be used to find out when the readers are not
referencing the position anymore.
+ RCU QSBR process is integrated within the Hash library for safe freeing of
the position. Application has certain responsibilities while using this feature.
+ Please refer to resource reclamation framework of :ref:`RCU library
<RCU_Library>` for more details.

* Lookup flow: Lookup for the flow key in the hash.
If the returned position is valid (flow lookup hit), use the returned position
to access the flow entry in the flow table.

lib/librte_hash/meson.build index renamed from lib/librte_hash/meson.build +0 -0

6ab46ae9d768..0977a63fd279 100644

lib/librte_hash/meson.build +0 -0
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 

sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c') deps += ['ring']

lib/librte_hash/meson.build +1 -0
+deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +4 -0
index aad0c965be5e..b9e4d82a0c14 100644
@@ -52,6 +52,11 @@ static struct rte_tailq_elem rte_hash_tailq = {  };
 EAL_REGISTER_TAILQ(rte_hash_tailq)
 
+struct __rte_hash_rcu_dq_entry {
+	uint32_t key_idx;
+	uint32_t ext_bkt_idx; /**< Extended bkt index */ };
+

struct rte_hash *
rte_hash_find_existing(const char *name) { @@ -210,7 +215,10 @@ rte_hash_create(const struct rte_hash_parameters *params)

	if (params->extra_flag &
RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF) {
		readwrite_concur_lf_support = 1;
lib/librte_hash/rte_cuckoo_hash.c +4 -1
-		/* Enable not freeing internal memory/index on delete */
+		/* Enable not freeing internal memory/index on delete.
+		 * If internal RCU is enabled, freeing of internal memory/index
+		 * is done on delete
+		 */
		no_free_on_del = 1;
	}
lib/librte_hash/rte_cuckoo_hash.c +14 -0
@@ -505,6 +513,10 @@ rte_hash_free(struct rte_hash *h)
 
 	rte_mcfg_tailq_write_unlock();
 
+	/* RCU clean up. */
+	if (h->dq)
+		rte_rcu_qsbr_dq_delete(h->dq);
+
 	if (h->use_local_cache)
 		rte_free(h->local_free_slots);
 	if (h->writer_takes_lock)
@@ -607,11 +619,21 @@ void
 rte_hash_reset(struct rte_hash *h)
 {
 	uint32_t tot_ring_cnt, i;
+	unsigned int pending;
 
 	if (h == NULL)
 		return;
 
 	__hash_rw_writer_lock(h);
+
+	/* RCU QSBR clean up. */
+	if (h->dq) {
+		/* Reclaim all the resources */
+		rte_rcu_qsbr_dq_reclaim(h->dq, ~0, NULL, &pending, NULL);
+		if (pending != 0)
+			RTE_LOG(ERR, HASH, "RCU reclaim all resources
 failed\n");
+	}
+
 	memset(h->buckets, 0, h->num_buckets * sizeof(struct
 rte_hash_bucket));
	memset(h->key_store, 0, h->key_entry_size * (h->entries + 1));
	*h->tbl_chng_cnt = 0;
lib/librte_hash/rte_cuckoo_hash.c +0 -0
@@ -952,6 +974,37 @@ rte_hash_cuckoo_make_space_mw(const struct
 rte_hash *h,
 	return -ENOSPC;
 }
 
Quoted 7 lines

lib/librte_hash/rte_cuckoo_hash.c rte_hash_cuckoo_make_space_mw(const struct rte_hash *h, · line 981

Quoted 5 lines
lib/librte_hash/rte_cuckoo_hash.c +5 -0
+static inline uint32_t
+alloc_slot(const struct rte_hash *h, struct lcore_cache
+*cached_free_slots) {
+	unsigned int  n_slots;
+	uint32_t slot_id;

[Yipeng]: Blank line after variable declaration.

Copy.

Quoted 318 lines
Quoted 318 lines
lib/librte_hash/rte_cuckoo_hash.c +4 -0
+	if (h->use_local_cache) {
+		/* Try to get a free slot from the local cache */
+		if (cached_free_slots->len == 0) {
+			/* Need to get another burst of free slots from global

ring */

lib/librte_hash/rte_cuckoo_hash.c +1 -0
+			n_slots = rte_ring_mc_dequeue_burst_elem(h-
Quoted 1 line

free_slots,

lib/librte_hash/rte_cuckoo_hash.c +21 -0
+					cached_free_slots->objs,
+					sizeof(uint32_t),
+					LCORE_CACHE_SIZE, NULL);
+			if (n_slots == 0)
+				return EMPTY_SLOT;
+
+			cached_free_slots->len += n_slots;
+		}
+
+		/* Get a free slot from the local cache */
+		cached_free_slots->len--;
+		slot_id = cached_free_slots->objs[cached_free_slots->len];
+	} else {
+		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
+						sizeof(uint32_t)) != 0)
+			return EMPTY_SLOT;
+	}
+
+	return slot_id;
+}
+

static inline int32_t
__rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key,
hash_sig_t sig, void *data)

lib/librte_hash/rte_cuckoo_hash.c +0 -1
@@ -963,7 +1016,6 @@ __rte_hash_add_key_with_hash(const struct
 rte_hash *h, const void *key,
 	uint32_t ext_bkt_id = 0;
 	uint32_t slot_id;
 	int ret;
-	unsigned n_slots;
 	unsigned lcore_id;
 	unsigned int i;
	struct lcore_cache *cached_free_slots = NULL; @@ -1001,28
+1053,20 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h,
const void *key,
	if (h->use_local_cache) {
		lcore_id = rte_lcore_id();
		cached_free_slots = &h->local_free_slots[lcore_id];
lib/librte_hash/rte_cuckoo_hash.c +0 -3
-		/* Try to get a free slot from the local cache */
-		if (cached_free_slots->len == 0) {
-			/* Need to get another burst of free slots from global

ring */

lib/librte_hash/rte_cuckoo_hash.c +0 -1
-			n_slots = rte_ring_mc_dequeue_burst_elem(h-
Quoted 1 line

free_slots,

lib/librte_hash/rte_cuckoo_hash.c +11 -8
-					cached_free_slots->objs,
-					sizeof(uint32_t),
-					LCORE_CACHE_SIZE, NULL);
-			if (n_slots == 0) {
-				return -ENOSPC;
-			}
-
-			cached_free_slots->len += n_slots;
+	}
+	slot_id = alloc_slot(h, cached_free_slots);
+	if (slot_id == EMPTY_SLOT) {
+		if (h->dq) {
+			__hash_rw_writer_lock(h);
+			ret = rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL);
+			__hash_rw_writer_unlock(h);
+			if (ret == 0)
+				slot_id = alloc_slot(h, cached_free_slots);
		}
lib/librte_hash/rte_cuckoo_hash.c +1 -7
-
-		/* Get a free slot from the local cache */
-		cached_free_slots->len--;
-		slot_id = cached_free_slots->objs[cached_free_slots->len];
-	} else {
-		if (rte_ring_sc_dequeue_elem(h->free_slots, &slot_id,
-						sizeof(uint32_t)) != 0) {
+		if (slot_id == EMPTY_SLOT)
			return -ENOSPC;
lib/librte_hash/rte_cuckoo_hash.c +0 -1
-		}
	}

	new_k = RTE_PTR_ADD(keys, slot_id * h->key_entry_size); @@ -
1118,8 +1162,19 @@ __rte_hash_add_key_with_hash(const struct rte_hash
*h, const void *key,
	if (rte_ring_sc_dequeue_elem(h->free_ext_bkts, &ext_bkt_id,
						sizeof(uint32_t)) != 0 ||
					ext_bkt_id == 0) {
lib/librte_hash/rte_cuckoo_hash.c +5 -2
-		ret = -ENOSPC;
-		goto failure;
+		if (h->dq) {
+			if (rte_rcu_qsbr_dq_reclaim(h->dq,
+					h->hash_rcu_cfg->max_reclaim_size,
+					NULL, NULL, NULL) == 0) {
+				rte_ring_sc_dequeue_elem(h-
Quoted 1 line

free_ext_bkts,

lib/librte_hash/rte_cuckoo_hash.c +8 -0
+							 &ext_bkt_id,
+							 sizeof(uint32_t));
+			}
+		}
+		if (ext_bkt_id == 0) {
+			ret = -ENOSPC;
+			goto failure;
+		}
	}

	/* Use the first location of the new bucket */ @@ -1395,12 +1450,12

@@ rte_hash_lookup_data(const struct rte_hash *h, const void *key, void **data)
return __rte_hash_lookup_with_hash(h, key, rte_hash_hash(h, key),
data); }

lib/librte_hash/rte_cuckoo_hash.c +0 -2
-static inline void
-remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,

unsigned i)

lib/librte_hash/rte_cuckoo_hash.c +2 -0
+static int
+free_slot(const struct rte_hash *h, uint32_t slot_id)
{
	unsigned lcore_id, n_slots;
lib/librte_hash/rte_cuckoo_hash.c +2 -2
-	struct lcore_cache *cached_free_slots;
-
+	struct lcore_cache *cached_free_slots = NULL;
+	/* Return key indexes to free slot ring */
	if (h->use_local_cache) {
		lcore_id = rte_lcore_id();
		cached_free_slots = &h->local_free_slots[lcore_id]; @@ -
1411,18 +1466,127 @@ remove_entry(const struct rte_hash *h, struct
rte_hash_bucket *bkt, unsigned i)
						cached_free_slots->objs,
						sizeof(uint32_t),
						LCORE_CACHE_SIZE, NULL);
lib/librte_hash/rte_cuckoo_hash.c +0 -2
-			ERR_IF_TRUE((n_slots == 0),
-				"%s: could not enqueue free slots in global

ring\n",

lib/librte_hash/rte_cuckoo_hash.c +1 -1
-				__func__);
+			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
			cached_free_slots->len -= n_slots;
		}
lib/librte_hash/rte_cuckoo_hash.c +23 -4
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] =
-							bkt->key_idx[i];
-		cached_free_slots->len++;
+	}
+
+	enqueue_slot_back(h, cached_free_slots, slot_id);
+	return 0;
+}
+
+static void
+__hash_rcu_qsbr_free_resource(void *p, void *e, unsigned int n) {
+	void *key_data = NULL;
+	int ret;
+	struct rte_hash_key *keys, *k;
+	struct rte_hash *h = (struct rte_hash *)p;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry =
+			*((struct __rte_hash_rcu_dq_entry *)e);
+
+	RTE_SET_USED(n);
+	keys = h->key_store;
+
+	k = (struct rte_hash_key *) ((char *)keys +
+				rcu_dq_entry.key_idx * h->key_entry_size);
+	key_data = k->pdata;
+	if (h->hash_rcu_cfg->free_key_data_func)
+		h->hash_rcu_cfg->free_key_data_func(h->hash_rcu_cfg-
Quoted 1 line

key_data_ptr,

lib/librte_hash/rte_cuckoo_hash.c +3 -0
+						    key_data);
+
+	if (h->ext_table_support && rcu_dq_entry.ext_bkt_idx !=

EMPTY_SLOT)

lib/librte_hash/rte_cuckoo_hash.c +37 -0
+		/* Recycle empty ext bkt to free list. */
+		rte_ring_sp_enqueue_elem(h->free_ext_bkts,
+			&rcu_dq_entry.ext_bkt_idx, sizeof(uint32_t));
+
+	/* Return key indexes to free slot ring */
+	ret = free_slot(h, rcu_dq_entry.key_idx);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
+	}
+}
+
+int
+rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg)
+{
+	struct rte_rcu_qsbr_dq_parameters params = {0};
+	char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+	struct rte_hash_rcu_config *hash_rcu_cfg = NULL;
+
+	const uint32_t total_entries = h->use_local_cache ?
+		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1)
+ 1
+							: h->entries + 1;
+
+	if ((h == NULL) || cfg == NULL || cfg->v == NULL) {
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	if (h->hash_rcu_cfg) {
+		rte_errno = EEXIST;
+		return 1;
+	}
+
+	hash_rcu_cfg = rte_zmalloc(NULL, sizeof(struct rte_hash_rcu_config),

0);

lib/librte_hash/rte_cuckoo_hash.c +18 -0
+	if (hash_rcu_cfg == NULL) {
+		RTE_LOG(ERR, HASH, "memory allocation failed\n");
+		return 1;
+	}
+
+	if (cfg->mode == RTE_HASH_QSBR_MODE_SYNC) {
+		/* No other things to do. */
+	} else if (cfg->mode == RTE_HASH_QSBR_MODE_DQ) {
+		/* Init QSBR defer queue. */
+		snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+					"HASH_RCU_%s", h->name);
+		params.name = rcu_dq_name;
+		params.size = cfg->dq_size;
+		if (params.size == 0)
+			params.size = total_entries;
+		params.trigger_reclaim_limit = cfg->trigger_reclaim_limit;
+		if (params.max_reclaim_size == 0)
+			params.max_reclaim_size =

RTE_HASH_RCU_DQ_RECLAIM_MAX;

lib/librte_hash/rte_cuckoo_hash.c +8 -0
+		params.esize = sizeof(struct __rte_hash_rcu_dq_entry);
+		params.free_fn = __hash_rcu_qsbr_free_resource;
+		params.p = h;
+		params.v = cfg->v;
+		h->dq = rte_rcu_qsbr_dq_create(&params);
+		if (h->dq == NULL) {
+			rte_free(hash_rcu_cfg);
+			RTE_LOG(ERR, HASH, "HASH defer queue creation

failed\n");

lib/librte_hash/rte_cuckoo_hash.c +2 -0
+			return 1;
+		}
	} else {
lib/librte_hash/rte_cuckoo_hash.c +26 -2
-		rte_ring_sp_enqueue_elem(h->free_slots,
-				&bkt->key_idx[i], sizeof(uint32_t));
+		rte_free(hash_rcu_cfg);
+		rte_errno = EINVAL;
+		return 1;
+	}
+
+	hash_rcu_cfg->v = cfg->v;
+	hash_rcu_cfg->mode = cfg->mode;
+	hash_rcu_cfg->dq_size = params.size;
+	hash_rcu_cfg->trigger_reclaim_limit = params.trigger_reclaim_limit;
+	hash_rcu_cfg->max_reclaim_size = params.max_reclaim_size;
+	hash_rcu_cfg->free_key_data_func = cfg->free_key_data_func;
+	hash_rcu_cfg->key_data_ptr = cfg->key_data_ptr;
+
+	h->hash_rcu_cfg = hash_rcu_cfg;
+
+	return 0;
+}
+
+static inline void
+remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt,
+unsigned i) {
+	int ret = free_slot(h, bkt->key_idx[i]);
+	if (ret < 0) {
+		RTE_LOG(ERR, HASH,
+			"%s: could not enqueue free slots in global ring\n",
+				__func__);
	}
}
lib/librte_hash/rte_cuckoo_hash.c +2 -0
@@ -1521,6 +1685,8 @@ __rte_hash_del_key_with_hash(const struct
 rte_hash *h, const void *key,
 	int pos;
 	int32_t ret, i;
 	uint16_t short_sig;
+	uint32_t index = EMPTY_SLOT;
+	struct __rte_hash_rcu_dq_entry rcu_dq_entry;
 
 	short_sig = get_short_sig(sig);
	prim_bucket_idx = get_prim_bucket_index(h, sig); @@ -1555,10

+1721,9 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key,

/* Search last bucket to see if empty to be recycled */ return_bkt:

lib/librte_hash/rte_cuckoo_hash.c +3 -4
-	if (!last_bkt) {
-		__hash_rw_writer_unlock(h);
-		return ret;
-	}
+	if (!last_bkt)
+		goto return_key;
+
	while (last_bkt->next) {
		prev_bkt = last_bkt;
		last_bkt = last_bkt->next;
lib/librte_hash/rte_cuckoo_hash.c +2 -2
@@ -1571,11 +1736,11 @@ __rte_hash_del_key_with_hash(const struct
 rte_hash *h, const void *key,
 	/* found empty bucket and recycle */
 	if (i == RTE_HASH_BUCKET_ENTRIES) {
 		prev_bkt->next = NULL;
-		uint32_t index = last_bkt - h->buckets_ext + 1;
+		index = last_bkt - h->buckets_ext + 1;
 		/* Recycle the empty bkt if
 		 * no_free_on_del is disabled.
 		 */
-		if (h->no_free_on_del)
+		if (h->no_free_on_del) {
 			/* Store index of an empty ext bkt to be recycled
 			 * on calling rte_hash_del_xxx APIs.

* When lock free read-write concurrency is enabled,

lib/librte_hash/rte_cuckoo_hash.c +1 -0
@@ -1583,12 +1748,34 @@ __rte_hash_del_key_with_hash(const struct
 rte_hash *h, const void *key,
 			 * immediately (as readers might be using it still).
 			 * Hence freeing of the ext bkt is piggy-backed to
 			 * freeing of the key index.
+			 * If using external RCU, store this index in an array.
Quoted 8 lines

lib/librte_hash/rte_cuckoo_hash.c __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, · line 1754

Quoted 5 lines
lib/librte_hash/rte_cuckoo_hash.c +2 -2
 			 */
-			h->ext_bkt_to_free[ret] = index;
-		else
+			if (h->hash_rcu_cfg == NULL)
+				h->ext_bkt_to_free[ret] = index;

[Yipeng]: If using embedded qsbr (not NULL), how did you recycle the ext bkt?

In DQ mode, ext_bkt_idx is enqueued to the defer queue along with the key_idx and recycled as part of rte_rcu_qsbr_dq_reclaim(). In SYNC mode, it is recycled after rte_rcu_qsbr_synchronize() succeeds.

Quoted 220 lines
Quoted 220 lines
lib/librte_hash/rte_cuckoo_hash.c +6 -0
+		} else
 			rte_ring_sp_enqueue_elem(h->free_ext_bkts,
 &index,
 							sizeof(uint32_t));
 	}
+
+return_key:
+	/* Using internal RCU QSBR */
+	if (h->hash_rcu_cfg) {
+		/* Key index where key is stored, adding the first dummy

index */

lib/librte_hash/rte_cuckoo_hash.c +9 -0
+		rcu_dq_entry.key_idx = ret + 1;
+		rcu_dq_entry.ext_bkt_idx = index;
+		if (h->dq == NULL) {
+			/* Wait for quiescent state change if using
+			 * RTE_HASH_QSBR_MODE_SYNC
+			 */
+			rte_rcu_qsbr_synchronize(h->hash_rcu_cfg->v,
+						 RTE_QSBR_THRID_INVALID);
+			__hash_rcu_qsbr_free_resource((void

*)((uintptr_t)h),

lib/librte_hash/rte_cuckoo_hash.c +3 -0
+						      &rcu_dq_entry, 1);
+		} else if (h->dq)
+			/* Push into QSBR FIFO if using

RTE_HASH_QSBR_MODE_DQ */

lib/librte_hash/rte_cuckoo_hash.c +1 -0
+			if (rte_rcu_qsbr_dq_enqueue(h->dq,

&rcu_dq_entry) != 0)

lib/librte_hash/rte_cuckoo_hash.c +1 -0
+				RTE_LOG(ERR, HASH, "Failed to push QSBR

FIFO\n");

lib/librte_hash/rte_cuckoo_hash.c +1 -0
+	}
	__hash_rw_writer_unlock(h);
	return ret;
}
lib/librte_hash/rte_cuckoo_hash.c +1 -2
@@ -1637,8 +1824,6 @@ rte_hash_free_key_with_position(const struct
 rte_hash *h,
 
 	RETURN_IF_TRUE(((h == NULL) || (key_idx == EMPTY_SLOT)), -EINVAL);
 
-	unsigned int lcore_id, n_slots;
-	struct lcore_cache *cached_free_slots;
 	const uint32_t total_entries = h->use_local_cache ?
 		h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1)
+ 1
							: h->entries + 1;
lib/librte_hash/rte_cuckoo_hash.c +2 -21
@@ -1656,28 +1841,9 @@ rte_hash_free_key_with_position(const struct
 rte_hash *h,
 		}
 	}
 
-	if (h->use_local_cache) {
-		lcore_id = rte_lcore_id();
-		cached_free_slots = &h->local_free_slots[lcore_id];
-		/* Cache full, need to free it. */
-		if (cached_free_slots->len == LCORE_CACHE_SIZE) {
-			/* Need to enqueue the free slots in global ring. */
-			n_slots = rte_ring_mp_enqueue_burst_elem(h-
 > free_slots,
-						cached_free_slots->objs,
-						sizeof(uint32_t),
-						LCORE_CACHE_SIZE, NULL);
-			RETURN_IF_TRUE((n_slots == 0), -EFAULT);
-			cached_free_slots->len -= n_slots;
-		}
-		/* Put index of new free slot in cache. */
-		cached_free_slots->objs[cached_free_slots->len] = key_idx;
-		cached_free_slots->len++;
-	} else {
-		rte_ring_sp_enqueue_elem(h->free_slots, &key_idx,
-						sizeof(uint32_t));
-	}
+	/* Enqueue slot to cache/ring of free slots. */
+	return free_slot(h, key_idx);
 
-	return 0;
 }

static inline void

lib/librte_hash/rte_cuckoo_hash.h +8 -0
index 345de6bf9cfd..85be49d3bbe7 100644
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
lib/librte_hash/rte_hash.h index renamed from lib/librte_hash/rte_hash.h +0 -0

bff40251bc98..3d28f177f14a 100644

lib/librte_hash/rte_hash.h +1 -0
@@ -15,6 +15,7 @@
 #include <stddef.h>
 
 #include <rte_compat.h>
+#include <rte_rcu_qsbr.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -45,7 +46,8 @@ extern "C" {
 /** Flag to disable freeing of key index on hash delete.
 * Refer to rte_hash_del_xxx APIs for more details.
 * This is enabled by default when

RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF
- * is enabled.
+ * is enabled. However, if internal RCU is enabled, freeing of internal
+ * memory/index is done on delete
*/
#define RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL 0x10

lib/librte_hash/rte_hash.h +8 -0
@@ -67,6 +69,13 @@ typedef uint32_t (*rte_hash_function)(const void
 *key, uint32_t key_len,
 /** Type of function used to compare the hash key. */  typedef int
 (*rte_hash_cmp_eq_t)(const void *key1, const void *key2, size_t key_len);
 
+/**
+ * Type of function used to free data stored in the key.
+ * Required when using internal RCU to allow application to free
+key-data once
+ * the key is returned to the the ring of free key-slots.
+ */
+typedef void (*rte_hash_free_key_data)(void *p, void *key_data);
+
 /**
 * Parameters used when creating the hash table.
 */
@@ -81,6 +90,39 @@ struct rte_hash_parameters {
 	uint8_t extra_flag;		/**< Indicate if additional parameters

are present. */
};

lib/librte_hash/rte_hash.h +20 -0
+/** RCU reclamation modes */
+enum rte_hash_qsbr_mode {
+	/** Create defer queue for reclaim. */
+	RTE_HASH_QSBR_MODE_DQ = 0,
+	/** Use blocking mode reclaim. No defer queue created. */
+	RTE_HASH_QSBR_MODE_SYNC
+};
+
+/** HASH RCU QSBR configuration structure. */ struct
+rte_hash_rcu_config {
+	struct rte_rcu_qsbr *v;		/**< RCU QSBR variable. */
+	enum rte_hash_qsbr_mode mode;
+	/**< Mode of RCU QSBR. RTE_HASH_QSBR_MODE_xxx
+	 * '0' for default: create defer queue for reclaim.
+	 */
+	uint32_t dq_size;
+	/**< RCU defer queue size.
+	 * default: total hash table entries.
+	 */
+	uint32_t trigger_reclaim_limit;	/**< Threshold to trigger auto reclaim.

*/

lib/librte_hash/rte_hash.h +12 -0
+	uint32_t max_reclaim_size;
+	/**< Max entries to reclaim in one go.
+	 * default: RTE_HASH_RCU_DQ_RECLAIM_MAX.
+	 */
+	void *key_data_ptr;
+	/**< Pointer passed to the free function. Typically, this is the
+	 * pointer to the data structure to which the resource to free
+	 * (key-data) belongs. This can be NULL.
+	 */
+	rte_hash_free_key_data free_key_data_func;
+	/**< Function to call to free the resource (key-data). */ };
+

/** @internal A hash table structure. */ struct rte_hash;

lib/librte_hash/rte_hash.h +2 -1
@@ -287,7 +329,8 @@ rte_hash_add_key_with_hash(const struct rte_hash
 *h, const void *key, hash_sig_t
 * Thread safety can be enabled by setting flag during
 * table creation.
 * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
 * the key index returned by rte_hash_add_key_xxx APIs will not be
 * freed by this API. rte_hash_free_key_with_position API must be called

* additionally to free the index associated with the key.

lib/librte_hash/rte_hash.h +2 -1
@@ -316,7 +359,8 @@ rte_hash_del_key(const struct rte_hash *h, const
 void *key);
 * Thread safety can be enabled by setting flag during
 * table creation.
 * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
 * the key index returned by rte_hash_add_key_xxx APIs will not be
 * freed by this API. rte_hash_free_key_with_position API must be called

* additionally to free the index associated with the key.

lib/librte_hash/rte_hash.h +2 -1
@@ -370,7 +414,8 @@ rte_hash_get_key_with_position(const struct
 rte_hash *h, const int32_t position,
 * only be called from one thread by default. Thread safety
 * can be enabled by setting flag during table creation.
 * If RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL or
- * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled,
+ * RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF is enabled and
+ * internal RCU is NOT enabled,
 * the key index returned by rte_hash_del_key_xxx APIs must be freed
 * using this API. This API should be called after all the readers

* have stopped referencing the entry corresponding to this key.

lib/librte_hash/rte_hash.h +0 -0
@@ -625,6 +670,30 @@ rte_hash_lookup_bulk(const struct rte_hash *h,
 const void **keys,
 */

int32_t
rte_hash_iterate(const struct rte_hash *h, const void **key, void **data,

Quoted 8 lines

lib/librte_hash/rte_hash.h rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, · line 678

Quoted 7 lines

uint32_t *next);

lib/librte_hash/rte_hash.h +6 -0
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Associate RCU QSBR variable with an Hash object.

[Yipeng]: a Hash object

Copy.

Quoted 38 lines
Quoted 35 lines

+ * This API should be called to enable the integrated RCU QSBR support

lib/librte_hash/rte_hash.h +18 -0
+and
+ * should be called immediately after creating the Hash object.
+ *
+ * @param h
+ *   the hash object to add RCU QSBR
+ * @param cfg
+ *   RCU QSBR configuration
+ * @return
+ *   On success - 0
+ *   On error - 1 with error code set in rte_errno.
+ *   Possible rte_errno codes are:
+ *   - EINVAL - invalid pointer
+ *   - EEXIST - already added QSBR
+ *   - ENOMEM - memory allocation failure
+ */
+__rte_experimental
+int rte_hash_rcu_qsbr_add(struct rte_hash *h,
+				struct rte_hash_rcu_config *cfg);

#ifdef __cplusplus
}
#endif

lib/librte_hash/version.map index renamed from lib/librte_hash/version.map +0 -0

c0db81014ff9..c6d73080f478 100644

lib/librte_hash/version.map +1 -2
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
--
Quoted 1 line

2.17.1

[Yipeng]:

Quoted 2 lines

Hi, Dharmik, thanks for the work! It generally looks good.
Just some minor issues to address and one question for the ext table inlined.

Thank you for the comments, Yipeng! I will update the patches.

Quoted 2 lines

Thanks!

#31 lore raw

PATCH v5 3/4 test/hash: replace rte atomic with C11 atomic APIs

In reply to Dharmik Thakkar

Replace rte_atomic APIs with C11 atomic APIs in
test_hash_readwrite_lf_perf.c

---
 app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++---------------
 1 file changed, 36 insertions(+), 53 deletions(-)
app/test/test_hash_readwrite_lf_perf.c +36 -53
index 889799865c7b..328fa5116f65 100644
@@ -82,8 +82,8 @@ static struct {
 	struct rte_hash *h;
 } tbl_rwc_test_param;
 
-static rte_atomic64_t gread_cycles;
-static rte_atomic64_t greads;
+static uint64_t gread_cycles;
+static uint64_t greads;
 
 static volatile uint8_t writer_done;
 
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg)
 	} while (!writer_done);
 
 	cycles = rte_rdtsc_precise() - begin;
-	rte_atomic64_add(&gread_cycles, cycles);
-	rte_atomic64_add(&greads, read_cnt*loop_cnt);
+	__atomic_fetch_add(&gread_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);
 	return 0;
 }
 
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type = WRITE_NO_KEY_SHIFT;
 	uint8_t read_type = READ_PASS_NO_KEY_SHIFTS;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, read - hit\n");
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_hit[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t read_type = READ_FAIL;
 	int ret;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, Hash lookup - miss\n");
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_miss[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_NON_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_nsp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				goto finish;
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_sp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 	uint8_t write_type;
 	uint8_t read_type = READ_FAIL;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - miss\n");
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
 		}
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Multi-add-lookup\n");
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				printf("\nNumber of readers: %u\n",
 				       rwc_core_cnt[n]);
 
-				rte_atomic64_clear(&greads);
-				rte_atomic64_clear(&gread_cycles);
+				__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+				__atomic_store_n(&gread_cycles, 0,
+						 __ATOMIC_RELAXED);
 
 				rte_hash_reset(tbl_rwc_test_param.h);
 				writer_done = 0;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 						goto err;
 
 				unsigned long long cycles_per_lookup =
-					rte_atomic64_read(&gread_cycles)
-					/ rte_atomic64_read(&greads);
+					__atomic_load_n(&gread_cycles,
+							__ATOMIC_RELAXED) /
+					__atomic_load_n(&greads,
+							  __ATOMIC_RELAXED);
 				rwc_perf_results->multi_rw[m][k][n]
 					= cycles_per_lookup;
 				printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_KEY_SHIFTS_EXTBKT;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n");
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			write_type = WRITE_NO_KEY_SHIFT;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_extbkt[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
Signature · 1 line
2.17.1
Acked-by on v5 3/4
#32 lore raw

Re: [dpdk-dev] [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs

Quoted 9 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 3/4] test/hash: replace rte atomic with C11 atomic APIs

Quoted 5 lines

Replace rte_atomic APIs with C11 atomic APIs in test_hash_readwrite_lf_perf.c

---

#33 lore raw

PATCH v5 4/4 test/hash: add tests for integrated RCU QSBR

In reply to Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
 2 files changed, 556 insertions(+), 4 deletions(-)
app/test/test_hash.c +389 -1
index 990a1815f893..22b47b3e7728 100644
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {				\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {			\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {		\
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+				RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where failure"
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add DQ mode functional test.
+ * Reader and writer are in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Add 8 hash entries and fill the bucket
+ *  - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt
+ *  - Register a reader thread (not a real thread)
+ *  - Reader lookup existing entry
+ *  - Writer deletes the entry
+ *  - Reader lookup the entry
+ *  - Writer re-add the entry (no available free index)
+ *  - Reader report quiescent state and unregister
+ *  - Writer re-add the entry
+ *  - Reader lookup the entry
+ */
+static int
+test_hash_rcu_qsbr_dq_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_dq_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR DQ mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR DQ mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+					 "failed to add key (pos[%u]=%d)", i,
+					 pos[i]);
+		expected_pos[i] = pos[i];
+	}
+
+	/* Register pseudo reader */
+	status = rte_rcu_qsbr_thread_register(g_qsv, 0);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR thread registration failed");
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Writer update */
+	pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Del", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to del correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOENT,
+				 "found deleted key (pos[%u]=%d)", 0, pos[0]);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOSPC,
+				 "Added key successfully (pos[%u]=%d)", 0, pos[0]);
+
+	/* Reader quiescent */
+	rte_rcu_qsbr_quiescent(g_qsv, 0);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+				 "failed to add key (pos[%u]=%d)", 0, pos[0]);
+	expected_pos[0] = pos[0];
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+	return 0;
+
+}
+
+/* Report quiescent state interval every 1024 lookups. Larger critical
+ * sections in reader will result in writer polling multiple times.
+ */
+#define QSBR_REPORTING_INTERVAL 1024
+#define WRITER_ITERATIONS	512
+
+/*
+ * Reader thread using rte_hash data structure with RCU.
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	int i;
+
+	RTE_SET_USED(arg);
+	/* Register this thread to report quiescent state */
+	(void)rte_rcu_qsbr_thread_register(g_qsv, 0);
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	do {
+		for (i = 0; i < QSBR_REPORTING_INTERVAL; i++)
+			rte_hash_lookup(g_handle, &g_rand_keys[0]);
+
+		/* Update quiescent state */
+		rte_rcu_qsbr_quiescent(g_qsv, 0);
+	} while (!writer_done);
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add sync mode functional test.
+ * 1 Reader and 1 writer. They cannot be in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Register a reader thread. Reader keeps looking up a specific key.
+ *  - Writer keeps adding and deleting a specific key.
+ */
+static int
+test_hash_rcu_qsbr_sync_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_sync_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR sync mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR sync mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Launch reader thread */
+	rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+				rte_get_next_lcore(-1, 1, 0));
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+				"failed to add key (pos[%u]=%d)", i, pos[i]);
+		expected_pos[i] = pos[i];
+	}
+	writer_done = 0;
+
+	/* Writer Update */
+	for (i = 0; i < WRITER_ITERATIONS; i++) {
+		expected_pos[0] = pos[0];
+		pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Del", &g_rand_keys[0], status);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+					 "failed to del correct key (pos[%u]=%d)"
+					 , 0, pos[0]);
+
+		pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Add", &g_rand_keys[0], pos[0]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+					 "failed to add key (pos[%u]=%d)", 0,
+					 pos[0]);
+	}
+
+	writer_done = 1;
+	/* Wait until reader exited. */
+	rte_eal_mp_wait_lcore();
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return  0;
+
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1862,6 +2235,21 @@ test_hash(void)
 	if (test_crc32_hash_alg_equiv() < 0)
 		return -1;
 
+	if (test_hash_rcu_qsbr_add() < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(1) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(1) < 0)
+		return -1;
+
 	return 0;
 }
 
app/test/test_hash_readwrite_lf_perf.c +167 -3

218 lines not on this page · raw message

Signature · 1 line
2.17.1
#34 lore raw

Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR

Quoted 100 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR

Add functional and performance tests for the integrated RCU QSBR.

---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
 2 files changed, 556 insertions(+), 4 deletions(-)
app/test/test_hash.c index renamed from app/test/test_hash.c +0 -0

990a1815f893..22b47b3e7728 100644

app/test/test_hash.c +1 -1
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10,
 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {
 	\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\

printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
\

app/test/test_hash.c +4 -0
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8,
 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {
 	\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
 \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {
	\
app/test/test_hash.c +34 -0
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF

|
+

	RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
app/test/test_hash.c +2 -0
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation

failed");

app/test/test_hash.c +5 -0
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE,

SOCKET_ID_ANY);

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1853

Quoted 4 lines
app/test/test_hash.c +4 -0
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);

[Wang, Yipeng] It reminds me that could we hide this function in the rte_cuckoo_hash.c as well?
I saw most of the rcu related functions are hidden in the hash implementation, it would be less confusing if we hide this one as well.

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1859

Quoted 6 lines
app/test/test_hash.c +6 -0
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;

[Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large number like 0xff?

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1863

Quoted 5 lines
app/test/test_hash.c +5 -0
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test
+failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;

[Wang, Yipeng] This reminds me that if there is an explanation on the difference of the two modes for users to easy to choose?

Quoted 26 lines
app/test/test_hash.c +8 -0
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE,

SOCKET_ID_ANY);

app/test/test_hash.c +9 -0
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where

failure"

app/test/test_hash.c +7 -0
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}

<...>

#35 lore raw

Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR

Quoted 102 lines · On Oct 20, 2020, at 10:54 PM, Wang, Yipeng1 <[email protected]> wrote:
Quoted 99 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Tuesday, October 20, 2020 9:13 AM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR

Add functional and performance tests for the integrated RCU QSBR.

---
app/test/test_hash.c | 390 ++++++++++++++++++++++++-
app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-

2 files changed, 556 insertions(+), 4 deletions(-)
app/test/test_hash.c index renamed from app/test/test_hash.c +0 -0

990a1815f893..22b47b3e7728 100644

app/test/test_hash.c +1 -1
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10,
 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {
 	\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
\
		if (handle) rte_fbk_hash_free(handle);			\
app/test/test_hash.c +4 -0
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8,
 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {
 	\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
 \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {
	\
app/test/test_hash.c +10 -0
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+

/* 5-tuple key type */

struct flow_key {
	uint32_t ip_src;
app/test/test_hash.c +24 -0
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF

|
+

	RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
app/test/test_hash.c +2 -0
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation

failed");

app/test/test_hash.c +4 -0
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
Quoted 8 lines

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1853

Quoted 6 lines
app/test/test_hash.c +1 -0
+					RTE_CACHE_LINE_SIZE,

SOCKET_ID_ANY);

app/test/test_hash.c +4 -0
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);

[Wang, Yipeng] It reminds me that could we hide this function in the rte_cuckoo_hash.c as well?
I saw most of the rcu related functions are hidden in the hash implementation, it would be less confusing if we hide this one as well.

Yes, I think this can be hidden within rte_hash_reset() and rte_hash_rcu_qsbr_add()

Quoted 8 lines

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1859

Quoted 7 lines
app/test/test_hash.c +6 -0
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;

[Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large number like 0xff?

Yes, I can use 0xxff

Quoted 7 lines

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1863

Quoted 6 lines
app/test/test_hash.c +5 -0
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test
+failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;

[Wang, Yipeng] This reminds me that if there is an explanation on the difference of the two modes for users to easy to choose?

It is available within rte_hash.h

Quoted 29 lines
Quoted 28 lines
app/test/test_hash.c +8 -0
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE,

SOCKET_ID_ANY);

app/test/test_hash.c +9 -0
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where

failure"

app/test/test_hash.c +7 -0
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}

<...>

#36 lore raw

Re: [dpdk-dev] [PATCH v5 4/4] test/hash: add tests for integrated RCU QSBR

In reply to Wang, Yipeng1

<snip>

Quoted 91 lines
Quoted 91 lines

Add functional and performance tests for the integrated RCU QSBR.

---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
 2 files changed, 556 insertions(+), 4 deletions(-)
app/test/test_hash.c index renamed from app/test/test_hash.c +0 -0

990a1815f893..22b47b3e7728 100644

app/test/test_hash.c +1 -1
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5,
 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {
 	\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
app/test/test_hash.c +4 -0
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5,
 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {
 	\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__);
 \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {
	\
app/test/test_hash.c +34 -0
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF

|
+

	RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
app/test/test_hash.c +2 -0
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation

failed");

app/test/test_hash.c +5 -0
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE,

SOCKET_ID_ANY);

Quoted 8 lines

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1853

Quoted 4 lines
app/test/test_hash.c +4 -0
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);

[Wang, Yipeng] It reminds me that could we hide this function in the rte_cuckoo_hash.c as well?
I saw most of the rcu related functions are hidden in the hash implementation, it would be less confusing if we hide this one as well.

The functionality required for resource reclamation is split into various parts [1]. The functionality of creating the RCU variable, initializing it etc is left to the application. The application has better knowledge of the data plane threads, which ones are using the (hash) library etc. Hence it is better to keep this in the application.

[1] http://doc.dpdk.org/guides/prog_guide/rcu_lib.html#resource-reclamation-framework-for-dpdk

Quoted 45 lines

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1859

Quoted 7 lines
app/test/test_hash.c +6 -0
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 2;

[Wang, Yipeng] Any other way rather than hardcode 2 here? Maybe just a large number like 0xff?

app/test/test_hash.c test_hash_add_delete_jhash_3word(void) · line 1863

Quoted 5 lines
app/test/test_hash.c +5 -0
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test
+failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;

[Wang, Yipeng] This reminds me that if there is an explanation on the difference of the two modes for users to easy to choose?

Quoted 26 lines
app/test/test_hash.c +8 -0
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE,

SOCKET_ID_ANY);

app/test/test_hash.c +9 -0
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where

failure"

app/test/test_hash.c +7 -0
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}

<...>

#37 lore raw

v6 cover hash: integrate RCU QSBR

In reply to Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.
---
v6:
- Update documentation
- Remove hard-coded value

v5:
- Enable 'rcu' for Windows

v4:
- Fix clang compilation issues

v3:
- Add documentation
- Add unit tests

v2:
- Remove defer queue related functions and use resource reclamation APIs from the RCU QSBR library instead

- Remove patch (net/ixgbe: avoid multpile definitions of 'bool') from the series as it is already accepted

Dharmik Thakkar (4):
rcu: build on Windows
lib/hash: integrate RCU QSBR
test/hash: replace rte atomic with C11 atomic APIs test/hash: add tests for integrated RCU QSBR

 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 259 ++++++++++++----
 doc/guides/prog_guide/hash_lib.rst     |  12 +-
 lib/librte_hash/meson.build            |   1 +
 lib/librte_hash/rte_cuckoo_hash.c      | 303 ++++++++++++++-----
 lib/librte_hash/rte_cuckoo_hash.h      |   8 +
 lib/librte_hash/rte_hash.h             |  77 ++++-
 lib/librte_hash/version.map            |   2 +-
 lib/meson.build                        |   1 +
 9 files changed, 920 insertions(+), 133 deletions(-)
Signature · 1 line
2.17.1
#38 lore raw

PATCH v6 1/4 rcu: build on Windows

Build the lib for Windows.

---
 lib/meson.build | 1 +
 1 file changed, 1 insertion(+)
lib/meson.build +1 -0
index dd55b5cb53e4..1bb019720c6a 100644
@@ -41,6 +41,7 @@ if is_windows
 		'telemetry',
 		'eal',
 		'ring',
+		'rcu',
 		'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
 		'cmdline',
 		'hash',
Signature · 1 line
2.17.1
#39 lore raw

Re: [dpdk-dev] [PATCH v6 1/4] rcu: build on Windows

Quoted 22 lines · Dharmik Thakkar

Build the lib for Windows.

---
 lib/meson.build | 1 +
 1 file changed, 1 insertion(+)
lib/meson.build +1 -1
index dd55b5cb53e4..1bb019720c6a 100644
@@ -41,6 +41,7 @@ if is_windows
                'telemetry',
                'eal',
                'ring',
+               'rcu',
                'mempool', 'mbuf', 'net', 'meter', 'ethdev', 'pci',
                'cmdline',
                'hash',
--
Quoted 2 lines

2.17.1

I only saw a test from Dmitry, pinging other Windows maintainers to get a clear ack.
Thanks.

Signature · 1 line
David Marchand
#41 lore raw

PATCH v6 2/4 lib/hash: integrate RCU QSBR

In reply to Dharmik Thakkar

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

---
 doc/guides/prog_guide/hash_lib.rst |  12 +-
 lib/librte_hash/meson.build        |   1 +
 lib/librte_hash/rte_cuckoo_hash.c  | 303 ++++++++++++++++++++++-------
 lib/librte_hash/rte_cuckoo_hash.h  |   8 +
 lib/librte_hash/rte_hash.h         |  77 +++++++-
 lib/librte_hash/version.map        |   2 +-
 6 files changed, 327 insertions(+), 76 deletions(-)
doc/guides/prog_guide/hash_lib.rst +9 -3
index d06c7de2ead1..fcdc3fa7a9ab 100644
@@ -102,6 +102,10 @@ For concurrent writes, and concurrent reads and writes the following flag values
 *  If the 'do not free on delete' (RTE_HASH_EXTRA_FLAGS_NO_FREE_ON_DEL) flag is set, the position of the entry in the hash table is not freed upon calling delete(). This flag is enabled
    by default when the lock free read/write concurrency flag is set. The application should free the position after all the readers have stopped referencing the position.
    Where required, the application can make use of RCU mechanisms to determine when the readers have stopped referencing the position.
+   RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+   For example, rte_hash_rcu_qsbr_add() need to be called to use the integrated RCU mechanism.
+   Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
+
 
 Extendable Bucket Functionality support
 ----------------------------------------
@@ -109,8 +113,8 @@ An extra flag is used to enable this functionality (flag is not set by default).
 in the very unlikely case due to excessive hash collisions that a key has failed to be inserted, the hash table bucket is extended with a linked
 list to insert these failed keys. This feature is important for the workloads (e.g. telco workloads) that need to insert up to 100% of the
 hash table size and can't tolerate any key insertion failure (even if very few).
-Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API in order to free the empty buckets and
-deleted keys, to maintain the 100% capacity guarantee.
+Please note that with the 'lock free read/write concurrency' flag enabled, users need to call 'rte_hash_free_key_with_position' API or configure integrated RCU QSBR
+(or use external RCU mechanisms) in order to free the empty buckets and deleted keys, to maintain the 100% capacity guarantee.
 
 Implementation Details (non Extendable Bucket Case)
 ---------------------------------------------------
@@ -172,7 +176,7 @@ Example of deletion:
 Similar to lookup, the key is searched in its primary and secondary buckets. If the key is found, the
 entry is marked as empty. If the hash table was configured with 'no free on delete' or 'lock free read/write concurrency',
 the position of the key is not freed. It is the responsibility of the user to free the position after
-readers are not referencing the position anymore.
+readers are not referencing the position anymore. User can configure integrated RCU QSBR or use external RCU mechanisms to safely free the position on delete
 
 
 Implementation Details (with Extendable Bucket)
@@ -286,6 +290,8 @@ The flow table operations on the application side are described below:
 *   Free flow: Free flow key position. If 'no free on delete' or 'lock-free read/write concurrency' flags are set,
     wait till the readers are not referencing the position returned during add/delete flow and then free the position.
     RCU mechanisms can be used to find out when the readers are not referencing the position anymore.
+    RCU QSBR process is integrated within the Hash library for safe freeing of the position. Application has certain responsibilities while using this feature.
+    Please refer to resource reclamation framework of :ref:`RCU library <RCU_Library>` for more details.
 
 *   Lookup flow: Lookup for the flow key in the hash.
     If the returned position is valid (flow lookup hit), use the returned position to access the flow entry in the flow table.
lib/librte_hash/meson.build +1 -0
index 6ab46ae9d768..0977a63fd279 100644
@@ -10,3 +10,4 @@ headers = files('rte_crc_arm64.h',
 
 sources = files('rte_cuckoo_hash.c', 'rte_fbk_hash.c')
 deps += ['ring']
+deps += ['rcu']
lib/librte_hash/rte_cuckoo_hash.c +235 -68

419 lines not on this page · raw message

lib/librte_hash/rte_cuckoo_hash.h +8 -0
index 345de6bf9cfd..85be49d3bbe7 100644
@@ -168,6 +168,11 @@ struct rte_hash {
 	struct lcore_cache *local_free_slots;
 	/**< Local cache per lcore, storing some indexes of the free slots */
 
+	/* RCU config */
+	struct rte_hash_rcu_config *hash_rcu_cfg;
+	/**< HASH RCU QSBR configuration structure */
+	struct rte_rcu_qsbr_dq *dq;	/**< RCU QSBR defer queue. */
+
 	/* Fields used in lookup */
 
 	uint32_t key_len __rte_cache_aligned;
@@ -230,4 +235,7 @@ struct queue_node {
 	int prev_slot;               /* Parent(slot) in search path */
 };
 
+/** @internal Default RCU defer queue entries to reclaim in one go. */
+#define RTE_HASH_RCU_DQ_RECLAIM_MAX	16
+
 #endif
lib/librte_hash/rte_hash.h +73 -4

125 lines not on this page · raw message

lib/librte_hash/version.map +1 -1
index c0db81014ff9..c6d73080f478 100644
@@ -36,5 +36,5 @@ EXPERIMENTAL {
 	rte_hash_lookup_with_hash_bulk;
 	rte_hash_lookup_with_hash_bulk_data;
 	rte_hash_max_key_id;
-
+	rte_hash_rcu_qsbr_add;
 };
Signature · 1 line
2.17.1
#42 lore raw

Re: [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR

Hi Yipeng,

I have addressed the comments. Can you please take a look?

Thank you!

Quoted 15 lines · Dharmik Thakkar

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

---

Acked-by on v6 2/4
#43 lore raw

Re: [dpdk-dev] [PATCH v6 2/4] lib/hash: integrate RCU QSBR

In reply to Dharmik Thakkar
Quoted 18 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Wednesday, October 21, 2020 3:50 PM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>; Ray Kinsella <[email protected]>; Neil Horman <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v6 2/4] lib/hash: integrate RCU QSBR

Currently, users have to use external RCU mechanisms to free resources when using lock free hash algorithm.

Integrate RCU QSBR process to make it easier for the applications to use lock free algorithm.
Refer to RCU documentation to understand various aspects of integrating RCU library into other libraries.

Quoted 5 lines

---

[Wang, Yipeng]
Thanks for revising the code.
Please check the checkpatch reported coding style issues to fix. After that

#44 lore raw

PATCH v6 3/4 test/hash: replace rte atomic with C11 atomic APIs

In reply to Dharmik Thakkar

Replace rte_atomic APIs with C11 atomic APIs in
test_hash_readwrite_lf_perf.c

---
 app/test/test_hash_readwrite_lf_perf.c | 89 +++++++++++---------------
 1 file changed, 36 insertions(+), 53 deletions(-)
app/test/test_hash_readwrite_lf_perf.c +36 -53
index 889799865c7b..328fa5116f65 100644
@@ -82,8 +82,8 @@ static struct {
 	struct rte_hash *h;
 } tbl_rwc_test_param;
 
-static rte_atomic64_t gread_cycles;
-static rte_atomic64_t greads;
+static uint64_t gread_cycles;
+static uint64_t greads;
 
 static volatile uint8_t writer_done;
 
@@ -645,8 +645,8 @@ test_rwc_reader(__rte_unused void *arg)
 	} while (!writer_done);
 
 	cycles = rte_rdtsc_precise() - begin;
-	rte_atomic64_add(&gread_cycles, cycles);
-	rte_atomic64_add(&greads, read_cnt*loop_cnt);
+	__atomic_fetch_add(&gread_cycles, cycles, __ATOMIC_RELAXED);
+	__atomic_fetch_add(&greads, read_cnt*loop_cnt, __ATOMIC_RELAXED);
 	return 0;
 }
 
@@ -703,9 +703,6 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type = WRITE_NO_KEY_SHIFT;
 	uint8_t read_type = READ_PASS_NO_KEY_SHIFTS;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, read - hit\n");
@@ -721,8 +718,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -739,8 +736,8 @@ test_hash_add_no_ks_lookup_hit(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_hit[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -773,9 +770,6 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t read_type = READ_FAIL;
 	int ret;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - no key-shifts, Hash lookup - miss\n");
@@ -791,8 +785,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -811,8 +805,8 @@ test_hash_add_no_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_no_ks_r_miss[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -845,9 +839,6 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_NON_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit"
@@ -864,8 +855,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -887,8 +878,8 @@ test_hash_add_ks_lookup_hit_non_sp(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_nsp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -921,9 +912,6 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - hit (shift-path)"
@@ -940,8 +928,9 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				goto finish;
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -963,8 +952,8 @@ test_hash_add_ks_lookup_hit_sp(struct rwc_perf *rwc_perf_results, int rwc_lf,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_sp[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
@@ -997,9 +986,6 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 	uint8_t write_type;
 	uint8_t read_type = READ_FAIL;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key shift, Hash lookup - miss\n");
@@ -1015,8 +1001,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			writer_done = 0;
@@ -1038,8 +1024,8 @@ test_hash_add_ks_lookup_miss(struct rwc_perf *rwc_perf_results, int rwc_lf, int
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_miss[m][n] = cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
 		}
@@ -1071,9 +1057,6 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_SHIFT_PATH;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Multi-add-lookup\n");
@@ -1098,8 +1081,9 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 				printf("\nNumber of readers: %u\n",
 				       rwc_core_cnt[n]);
 
-				rte_atomic64_clear(&greads);
-				rte_atomic64_clear(&gread_cycles);
+				__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+				__atomic_store_n(&gread_cycles, 0,
+						 __ATOMIC_RELAXED);
 
 				rte_hash_reset(tbl_rwc_test_param.h);
 				writer_done = 0;
@@ -1138,8 +1122,10 @@ test_hash_multi_add_lookup(struct rwc_perf *rwc_perf_results, int rwc_lf,
 						goto err;
 
 				unsigned long long cycles_per_lookup =
-					rte_atomic64_read(&gread_cycles)
-					/ rte_atomic64_read(&greads);
+					__atomic_load_n(&gread_cycles,
+							__ATOMIC_RELAXED) /
+					__atomic_load_n(&greads,
+							  __ATOMIC_RELAXED);
 				rwc_perf_results->multi_rw[m][k][n]
 					= cycles_per_lookup;
 				printf("Cycles per lookup: %llu\n",
@@ -1172,9 +1158,6 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 	uint8_t write_type;
 	uint8_t read_type = READ_PASS_KEY_SHIFTS_EXTBKT;
 
-	rte_atomic64_init(&greads);
-	rte_atomic64_init(&gread_cycles);
-
 	if (init_params(rwc_lf, use_jhash, htm, ext_bkt) != 0)
 		goto err;
 	printf("\nTest: Hash add - key-shifts, read - hit (ext_bkt)\n");
@@ -1190,8 +1173,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 
 			printf("\nNumber of readers: %u\n", rwc_core_cnt[n]);
 
-			rte_atomic64_clear(&greads);
-			rte_atomic64_clear(&gread_cycles);
+			__atomic_store_n(&greads, 0, __ATOMIC_RELAXED);
+			__atomic_store_n(&gread_cycles, 0, __ATOMIC_RELAXED);
 
 			rte_hash_reset(tbl_rwc_test_param.h);
 			write_type = WRITE_NO_KEY_SHIFT;
@@ -1222,8 +1205,8 @@ test_hash_add_ks_lookup_hit_extbkt(struct rwc_perf *rwc_perf_results,
 					goto err;
 
 			unsigned long long cycles_per_lookup =
-				rte_atomic64_read(&gread_cycles) /
-				rte_atomic64_read(&greads);
+				__atomic_load_n(&gread_cycles, __ATOMIC_RELAXED)
+				/ __atomic_load_n(&greads, __ATOMIC_RELAXED);
 			rwc_perf_results->w_ks_r_hit_extbkt[m][n]
 						= cycles_per_lookup;
 			printf("Cycles per lookup: %llu\n", cycles_per_lookup);
Signature · 1 line
2.17.1
#45 lore raw

PATCH v6 4/4 test/hash: add tests for integrated RCU QSBR

In reply to Dharmik Thakkar

Add functional and performance tests for the integrated RCU QSBR.

---
 app/test/test_hash.c                   | 390 ++++++++++++++++++++++++-
 app/test/test_hash_readwrite_lf_perf.c | 170 ++++++++++-
 2 files changed, 556 insertions(+), 4 deletions(-)
app/test/test_hash.c +389 -1
index 990a1815f893..cf299982aedc 100644
@@ -52,7 +52,7 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
-#define RETURN_IF_ERROR_FBK(cond, str, ...) do {				\
+#define RETURN_IF_ERROR_FBK(cond, str, ...) do {			\
 	if (cond) {							\
 		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
 		if (handle) rte_fbk_hash_free(handle);			\
@@ -60,6 +60,20 @@ static uint32_t hashtest_key_lens[] = {0, 2, 4, 5, 6, 7, 8, 10, 11, 15, 16, 21,
 	}								\
 } while(0)
 
+#define RETURN_IF_ERROR_RCU_QSBR(cond, str, ...) do {			\
+	if (cond) {							\
+		printf("ERROR line %d: " str "\n", __LINE__, ##__VA_ARGS__); \
+		if (rcu_cfg.mode == RTE_HASH_QSBR_MODE_SYNC) {		\
+			writer_done = 1;				\
+			/* Wait until reader exited. */			\
+			rte_eal_mp_wait_lcore();			\
+		}							\
+		if (g_handle) rte_hash_free(g_handle);			\
+		if (g_qsv) rte_free(g_qsv);				\
+		return -1;						\
+	}								\
+} while(0)
+
 /* 5-tuple key type */
 struct flow_key {
 	uint32_t ip_src;
@@ -1801,6 +1815,365 @@ test_hash_add_delete_jhash_3word(void)
 	return ret;
 }
 
+static struct rte_hash *g_handle;
+static struct rte_rcu_qsbr *g_qsv;
+static volatile uint8_t writer_done;
+struct flow_key g_rand_keys[9];
+/*
+ * rte_hash_rcu_qsbr_add positive and negative tests.
+ *  - Add RCU QSBR variable to Hash
+ *  - Add another RCU QSBR variable to Hash
+ *  - Check returns
+ */
+static int
+test_hash_rcu_qsbr_add(void)
+{
+	size_t sz;
+	struct rte_rcu_qsbr *qsv2 = NULL;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	struct rte_hash_parameters params;
+
+	printf("\n# Running RCU QSBR add tests\n");
+	memcpy(&params, &ut_params, sizeof(params));
+	params.name = "test_hash_rcu_qsbr_add";
+	params.extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF |
+				RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD;
+	g_handle = rte_hash_create(&params);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	/* Invalid QSBR mode */
+	rcu_cfg.mode = 0xff;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0, "Invalid QSBR mode test failed");
+
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Create and attach another RCU QSBR to hash table */
+	qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(qsv2 == NULL,
+				 "RCU QSBR variable creation failed");
+
+	rcu_cfg.v = qsv2;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	rte_free(qsv2);
+	RETURN_IF_ERROR_RCU_QSBR(status == 0,
+			"Attach RCU QSBR to hash table succeeded where failure"
+			" is expected");
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add DQ mode functional test.
+ * Reader and writer are in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Add 8 hash entries and fill the bucket
+ *  - If ext bkt is enabled, add 1 extra entry that is available in the ext bkt
+ *  - Register a reader thread (not a real thread)
+ *  - Reader lookup existing entry
+ *  - Writer deletes the entry
+ *  - Reader lookup the entry
+ *  - Writer re-add the entry (no available free index)
+ *  - Reader report quiescent state and unregister
+ *  - Writer re-add the entry
+ *  - Reader lookup the entry
+ */
+static int
+test_hash_rcu_qsbr_dq_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_dq_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR DQ mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR DQ mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_DQ;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+					 "failed to add key (pos[%u]=%d)", i,
+					 pos[i]);
+		expected_pos[i] = pos[i];
+	}
+
+	/* Register pseudo reader */
+	status = rte_rcu_qsbr_thread_register(g_qsv, 0);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR thread registration failed");
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Writer update */
+	pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Del", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to del correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOENT,
+				 "found deleted key (pos[%u]=%d)", 0, pos[0]);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != -ENOSPC,
+				 "Added key successfully (pos[%u]=%d)", 0, pos[0]);
+
+	/* Reader quiescent */
+	rte_rcu_qsbr_quiescent(g_qsv, 0);
+
+	/* Fill bucket */
+	pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+	print_key_info("Add", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+				 "failed to add key (pos[%u]=%d)", 0, pos[0]);
+	expected_pos[0] = pos[0];
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	/* Lookup */
+	pos[0] = rte_hash_lookup(g_handle, &g_rand_keys[0]);
+	print_key_info("Lkp", &g_rand_keys[0], pos[0]);
+	RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+				 "failed to find correct key (pos[%u]=%d)", 0,
+				 pos[0]);
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+	return 0;
+
+}
+
+/* Report quiescent state interval every 1024 lookups. Larger critical
+ * sections in reader will result in writer polling multiple times.
+ */
+#define QSBR_REPORTING_INTERVAL 1024
+#define WRITER_ITERATIONS	512
+
+/*
+ * Reader thread using rte_hash data structure with RCU.
+ */
+static int
+test_hash_rcu_qsbr_reader(void *arg)
+{
+	int i;
+
+	RTE_SET_USED(arg);
+	/* Register this thread to report quiescent state */
+	(void)rte_rcu_qsbr_thread_register(g_qsv, 0);
+	rte_rcu_qsbr_thread_online(g_qsv, 0);
+
+	do {
+		for (i = 0; i < QSBR_REPORTING_INTERVAL; i++)
+			rte_hash_lookup(g_handle, &g_rand_keys[0]);
+
+		/* Update quiescent state */
+		rte_rcu_qsbr_quiescent(g_qsv, 0);
+	} while (!writer_done);
+
+	rte_rcu_qsbr_thread_offline(g_qsv, 0);
+	(void)rte_rcu_qsbr_thread_unregister(g_qsv, 0);
+
+	return 0;
+}
+
+/*
+ * rte_hash_rcu_qsbr_add sync mode functional test.
+ * 1 Reader and 1 writer. They cannot be in the same thread in this test.
+ *  - Create hash which supports maximum 8 (9 if ext bkt is enabled) entries
+ *  - Add RCU QSBR variable to hash
+ *  - Register a reader thread. Reader keeps looking up a specific key.
+ *  - Writer keeps adding and deleting a specific key.
+ */
+static int
+test_hash_rcu_qsbr_sync_mode(uint8_t ext_bkt)
+{
+	uint32_t total_entries = (ext_bkt == 0) ? 8 : 9;
+
+	uint8_t hash_extra_flag = RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF;
+
+	if (ext_bkt)
+		hash_extra_flag |= RTE_HASH_EXTRA_FLAGS_EXT_TABLE;
+
+	struct rte_hash_parameters params_pseudo_hash = {
+		.name = "test_hash_rcu_qsbr_sync_mode",
+		.entries = total_entries,
+		.key_len = sizeof(struct flow_key), /* 13 */
+		.hash_func = pseudo_hash,
+		.hash_func_init_val = 0,
+		.socket_id = 0,
+		.extra_flag = hash_extra_flag,
+	};
+	int pos[total_entries];
+	int expected_pos[total_entries];
+	unsigned i;
+	size_t sz;
+	int32_t status;
+	struct rte_hash_rcu_config rcu_cfg = {0};
+
+	g_qsv = NULL;
+	g_handle = NULL;
+
+	for (i = 0; i < total_entries; i++) {
+		g_rand_keys[i].port_dst = i;
+		g_rand_keys[i].port_src = i+1;
+	}
+
+	if (ext_bkt)
+		printf("\n# Running RCU QSBR sync mode functional test with"
+		       " ext bkt\n");
+	else
+		printf("\n# Running RCU QSBR sync mode functional test\n");
+
+	g_handle = rte_hash_create(&params_pseudo_hash);
+	RETURN_IF_ERROR_RCU_QSBR(g_handle == NULL, "Hash creation failed");
+
+	/* Create RCU QSBR variable */
+	sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+	g_qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+					RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	RETURN_IF_ERROR_RCU_QSBR(g_qsv == NULL,
+				 "RCU QSBR variable creation failed");
+
+	status = rte_rcu_qsbr_init(g_qsv, RTE_MAX_LCORE);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "RCU QSBR variable initialization failed");
+
+	rcu_cfg.v = g_qsv;
+	rcu_cfg.mode = RTE_HASH_QSBR_MODE_SYNC;
+	/* Attach RCU QSBR to hash table */
+	status = rte_hash_rcu_qsbr_add(g_handle, &rcu_cfg);
+	RETURN_IF_ERROR_RCU_QSBR(status != 0,
+				 "Attach RCU QSBR to hash table failed");
+
+	/* Launch reader thread */
+	rte_eal_remote_launch(test_hash_rcu_qsbr_reader, NULL,
+				rte_get_next_lcore(-1, 1, 0));
+
+	/* Fill bucket */
+	for (i = 0; i < total_entries; i++) {
+		pos[i] = rte_hash_add_key(g_handle, &g_rand_keys[i]);
+		print_key_info("Add", &g_rand_keys[i], pos[i]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[i] < 0,
+				"failed to add key (pos[%u]=%d)", i, pos[i]);
+		expected_pos[i] = pos[i];
+	}
+	writer_done = 0;
+
+	/* Writer Update */
+	for (i = 0; i < WRITER_ITERATIONS; i++) {
+		expected_pos[0] = pos[0];
+		pos[0] = rte_hash_del_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Del", &g_rand_keys[0], status);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] != expected_pos[0],
+					 "failed to del correct key (pos[%u]=%d)"
+					 , 0, pos[0]);
+
+		pos[0] = rte_hash_add_key(g_handle, &g_rand_keys[0]);
+		print_key_info("Add", &g_rand_keys[0], pos[0]);
+		RETURN_IF_ERROR_RCU_QSBR(pos[0] < 0,
+					 "failed to add key (pos[%u]=%d)", 0,
+					 pos[0]);
+	}
+
+	writer_done = 1;
+	/* Wait until reader exited. */
+	rte_eal_mp_wait_lcore();
+
+	rte_hash_free(g_handle);
+	rte_free(g_qsv);
+
+	return  0;
+
+}
+
 /*
  * Do all unit and performance tests.
  */
@@ -1862,6 +2235,21 @@ test_hash(void)
 	if (test_crc32_hash_alg_equiv() < 0)
 		return -1;
 
+	if (test_hash_rcu_qsbr_add() < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_dq_mode(1) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(0) < 0)
+		return -1;
+
+	if (test_hash_rcu_qsbr_sync_mode(1) < 0)
+		return -1;
+
 	return 0;
 }
 
app/test/test_hash_readwrite_lf_perf.c +167 -3

218 lines not on this page · raw message

Signature · 1 line
2.17.1
Acked-by on v6 4/4
#46 lore raw

Re: [dpdk-dev] [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR

Quoted 9 lines

-----Original Message-----
From: Dharmik Thakkar <[email protected]>
Sent: Wednesday, October 21, 2020 3:50 PM
To: Wang, Yipeng1 <[email protected]>; Gobriel, Sameh <[email protected]>; Richardson, Bruce <[email protected]>
Cc: [email protected]; [email protected]; Dharmik Thakkar <[email protected]>
Subject: [PATCH v6 4/4] test/hash: add tests for integrated RCU QSBR

Quoted 6 lines

Add functional and performance tests for the integrated RCU QSBR.

---

[Wang, Yipeng]
Also please confirm the checkpatch report see if it is not a false-positive. Then:

#47 lore raw

Re: [dpdk-dev] [PATCH v6 0/4] hash: integrate RCU QSBR

In reply to Dharmik Thakkar
Quoted 12 lines · Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

Quoted 7 lines

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.

Fixed some style issues/checkpatch warnings.
The unit test code especially is not really checkpatch compliant, maybe worth a separate cleanup later.

Removed unrelated changes (like removing empty lines, fixing typos in unrelated parts).

Series applied, thanks.

Little note: checkpatch now has a check on repeated words better than a script of mine:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/scripts/checkpatch.pl?id=1a3dcf2e6b35faa1176b9cd8200094fbce16ba19
Probably worth it for people to update their checkpatch.pl.

Signature · 1 line
David Marchand
#48 lore raw

Re: [dpdk-dev] [PATCH v6 0/4] hash: integrate RCU QSBR

Quoted 33 lines · David Marchand
Quoted 12 lines · Dharmik Thakkar

Integrate RCU QSBR to make it easier for the applications to use lock free algorithm.

Resource reclamation implementation was split from the original series, and has already been part of RCU library. Rework the series to base hash integration on RCU reclamation APIs.

Refer 'Resource reclamation framework for DPDK' available at [1] to understand various aspects of integrating RCU library into other libraries.

Quoted 7 lines

[1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html

Introduce a new API rte_hash_rcu_qsbr_add for application to register a RCU variable that hash library will use.

Functional tests and performance tests are added to cover the integration with RCU.

Fixed some style issues/checkpatch warnings.
The unit test code especially is not really checkpatch compliant, maybe worth a separate cleanup later.

Removed unrelated changes (like removing empty lines, fixing typos in unrelated parts).

Quoted 8 lines

Series applied, thanks.

Little note: checkpatch now has a check on repeated words better than a script of mine:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/scripts/checkpatch.pl?id=1a3dcf2e6b35faa1176b9cd8200094fbce16ba19
Probably worth it for people to update their checkpatch.pl.

Thank you, David! Appreciate your help.

Quoted 3 lines
Signature · 2 lines
-- 
David Marchand