Skip to contentSkip to conversation

Add Membership Library

PATCH Superseded Yipeng Wang 7 participants 81 messages Aug 22, 2017 – Oct 8, 2017 Library code

lore
Share & tools
lore
#0 lore raw

cover Add Membership Library

DPDP Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.

A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.

We sent out RFC previously:
http://dpdk.org/ml/archives/dev/2017-May/066599.html

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

 MAINTAINERS                              |    7 +
 config/common_base                       |    5 +
 doc/api/doxy-api-index.md                |    3 +-
 doc/api/doxy-api.conf                    |    1 +
 doc/guides/prog_guide/img/member_i1.svg  | 1613 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg  |   36 +
 doc/guides/prog_guide/img/member_i3.svg  |  148 +++
 doc/guides/prog_guide/img/member_i4.svg  |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg  |  163 +++
 doc/guides/prog_guide/img/member_i6.svg  |  332 ++++++
 doc/guides/prog_guide/img/member_i7.svg  |  399 ++++++++
 doc/guides/prog_guide/index.rst          |   14 +
 doc/guides/prog_guide/member_lib.rst     |  432 ++++++++
 doc/guides/rel_notes/release_17_11.rst   |   17 +
 lib/Makefile                             |    2 +
 lib/librte_eal/common/eal_common_log.c   |    1 +
 lib/librte_eal/common/include/rte_log.h  |    1 +
 lib/librte_member/Makefile               |   50 +
 lib/librte_member/rte_member.c           |  357 +++++++
 lib/librte_member/rte_member.h           |  518 ++++++++++
 lib/librte_member/rte_member_ht.c        |  567 +++++++++++
 lib/librte_member/rte_member_ht.h        |  115 +++
 lib/librte_member/rte_member_vbf.c       |  309 ++++++
 lib/librte_member/rte_member_vbf.h       |   85 ++
 lib/librte_member/rte_member_version.map |   15 +
 lib/librte_member/rte_member_x86.h       |  111 ++
 mk/rte.app.mk                            |    2 +
 test/test/Makefile                       |    3 +
 test/test/test_member.c                  |  550 ++++++++++
 test/test/test_member_perf.c             |  643 ++++++++++++
 30 files changed, 6948 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
 create mode 100644 lib/librte_member/rte_member_version.map
 create mode 100644 lib/librte_member/rte_member_x86.h
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/7 member: implement main API

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  48 +++
 lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518 +++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +
 7 files changed, 942 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
index 86caba1..c82033a 100644
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
 DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
 DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_eal/common/eal_common_log.c +1 -0
index 0e3b932..90db6a3 100644
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = {
 	{RTE_LOGTYPE_CRYPTODEV,  "cryptodev"},
 	{RTE_LOGTYPE_EFD,        "efd"},
 	{RTE_LOGTYPE_EVENTDEV,   "eventdev"},
+	{RTE_LOGTYPE_MEMBER,     "member"},
 	{RTE_LOGTYPE_USER1,      "user1"},
 	{RTE_LOGTYPE_USER2,      "user2"},
 	{RTE_LOGTYPE_USER3,      "user3"},
lib/librte_eal/common/include/rte_log.h +1 -0
index ec8dba7..ce1a3d0 100644
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;
 #define RTE_LOGTYPE_CRYPTODEV 17 /**< Log related to cryptodev. */
 #define RTE_LOGTYPE_EFD       18 /**< Log related to EFD. */
 #define RTE_LOGTYPE_EVENTDEV  19 /**< Log related to eventdev. */
+#define RTE_LOGTYPE_MEMBER    20 /**< Log related to membership. */
 
 /* these log types can be used in an application */
 #define RTE_LOGTYPE_USER1     24 /**< User-defined log type 1. */
lib/librte_member/Makefile new file +48 -0

48 lines not on this page · raw message

lib/librte_member/rte_member.c new file +357 -0

357 lines not on this page · raw message

lib/librte_member/rte_member.h new file +518 -0

518 lines not on this page · raw message

lib/librte_member/rte_member_version.map new file +15 -0
new file mode 100644
index 0000000..b0977e1
@@ -0,0 +1,15 @@
+DPDK_17.11 {
+	global:
+
+	rte_member_create;
+	rte_member_lookup;
+	rte_member_lookup_bulk;
+	rte_member_lookup_multi;
+	rte_member_lookup_multi_bulk;
+	rte_member_add;
+	rte_member_free;
+	rte_member_reset;
+	rte_member_delete;
+
+	local: *;
+};
Signature · 1 line
2.7.4
#2 lore raw

Re: [PATCH 1/7] member: implement main API

lib/librte_member/rte_member.c line 231

Quoted 9 lines · On Mon, 21 Aug 2017 17:19:47 -0700 Yipeng Wang <[email protected]> wrote:
lib/librte_member/rte_member.c new file +9 -0
+int
+rte_member_lookup(const void *ss, const void *key,
+		MEMBER_SET_TYPE *set_id)
+{
+	const struct rte_member_setsum *setsum = ss;
+	if (setsum == NULL || key == NULL || set_id == NULL)
+		return -EINVAL;
+
+	int ret = 0;

Don't needlessly initialize all variable.
In all code paths, ret is set.

In fact you could just make all cases of the switch into return and compiler could optimize it as a tail call.

#3 lore raw

Re: [PATCH 1/7] member: implement main API

In reply to Yipeng Wang
Quoted 91 lines · Yipeng Wang

Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 + lib/librte_eal/common/eal_common_log.c   |   1 + lib/librte_eal/common/include/rte_log.h  |   1 + lib/librte_member/Makefile               |  48 +++ lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++ lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +

 7 files changed, 942 insertions(+)

 create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

lib/librte_member/Makefile new file +6 -0
new file mode 100644
index 0000000..997c825
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or

without
+#   modification, are permitted provided that the following conditions

lib/librte_member/Makefile new file +3 -0
+#   are met:
+#
+#     * Redistributions of source code must retain the above

copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in

lib/librte_member/Makefile new file +4 -0
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products

derived
+#       from this software without specific prior written permission.

lib/librte_member/Makefile new file +2 -0
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,

lib/librte_member/Makefile new file +2 -0
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF

USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

lib/librte_member/Makefile line 38

Quoted 8 lines
lib/librte_member/Makefile new file +8 -0
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+

This breaks reproducibility as the output directory will be included before the source directory, causing a race - please do something like:

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

Quoted 10 lines
lib/librte_member/Makefile new file +10 -0
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
Signature · 2 lines
Kind regards,
Luca Boccassi
#4 lore raw

Re: [PATCH 1/7] member: implement main API

Quoted 104 lines · Luca Boccassi
Quoted 91 lines · Yipeng Wang

Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 + lib/librte_eal/common/eal_common_log.c   |   1 + lib/librte_eal/common/include/rte_log.h  |   1 + lib/librte_member/Makefile               |  48 +++ lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++ lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +

 7 files changed, 942 insertions(+)

 create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

lib/librte_member/Makefile new file +6 -0
new file mode 100644
index 0000000..997c825
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or

without
+#   modification, are permitted provided that the following conditions

lib/librte_member/Makefile new file +3 -0
+#   are met:
+#
+#     * Redistributions of source code must retain the above

copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in

lib/librte_member/Makefile new file +4 -0
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products

derived
+#       from this software without specific prior written permission.

lib/librte_member/Makefile new file +2 -0
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,

lib/librte_member/Makefile new file +2 -0
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF

USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

lib/librte_member/Makefile line 38

Quoted 8 lines
lib/librte_member/Makefile new file +8 -0
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+

This breaks reproducibility as the output directory will be included before the source directory, causing a race - please do something like:

Quoted 2 lines

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

Can we remove "-I$(SRCDIR)" completely by first installing headers and later compiling objects, all using $(RTE_OUT) only?

Do you think can this work?

Quoted 12 lines
Quoted 11 lines
lib/librte_member/Makefile new file +10 -0
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
#5 lore raw

Re: [PATCH 1/7] member: implement main API

Quoted 119 lines · Ferruh Yigit
Quoted 118 lines · Luca Boccassi
Quoted 103 lines · Yipeng Wang

Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 + lib/librte_eal/common/eal_common_log.c   |   1 + lib/librte_eal/common/include/rte_log.h  |   1 + lib/librte_member/Makefile               |  48 +++ lib/librte_member/rte_member.c           | 357
+++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +

 7 files changed, 942 insertions(+)

 create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

lib/librte_member/Makefile new file +6 -0
new file mode 100644
index 0000000..997c825
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or

without
+#   modification, are permitted provided that the following conditions

lib/librte_member/Makefile new file +3 -0
+#   are met:
+#
+#     * Redistributions of source code must retain the above

copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the

lib/librte_member/Makefile new file +2 -0
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of

its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.

lib/librte_member/Makefile new file +2 -0
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

lib/librte_member/Makefile line 38

Quoted 8 lines
lib/librte_member/Makefile new file +8 -0
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+

This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something like:

Quoted 8 lines
Quoted 2 lines

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

Can we remove "-I$(SRCDIR)" completely by first installing headers and
later compiling objects, all using $(RTE_OUT) only?

Do you think can this work?

I'm not sure, it might - but given Bruce's effort to port to Meson I'm not sure it's worth spending a lot of time doing big refactoring of the existing build system

Quoted 12 lines
Quoted 10 lines
Quoted 10 lines
lib/librte_member/Makefile new file +10 -0
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
Signature · 2 lines
Kind regards,
Luca Boccassi
#6 lore raw

Re: [PATCH 1/7] member: implement main API

Quoted 127 lines · Luca Boccassi
Quoted 126 lines · Ferruh Yigit
Quoted 118 lines · Luca Boccassi
Quoted 103 lines · Yipeng Wang

Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 + lib/librte_eal/common/eal_common_log.c   |   1 + lib/librte_eal/common/include/rte_log.h  |   1 + lib/librte_member/Makefile               |  48 +++ lib/librte_member/rte_member.c           | 357
+++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +

 7 files changed, 942 insertions(+)

 create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

lib/librte_member/Makefile new file +6 -0
new file mode 100644
index 0000000..997c825
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or

without
+#   modification, are permitted provided that the following conditions

lib/librte_member/Makefile new file +3 -0
+#   are met:
+#
+#     * Redistributions of source code must retain the above

copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the

lib/librte_member/Makefile new file +2 -0
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of

its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.

lib/librte_member/Makefile new file +2 -0
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

lib/librte_member/Makefile

Quoted 8 lines
lib/librte_member/Makefile new file +8 -0
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+

This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something like:

Quoted 2 lines

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

Can we remove "-I$(SRCDIR)" completely by first installing headers and
later compiling objects, all using $(RTE_OUT) only?

Quoted 5 lines
Quoted 1 line

Do you think can this work?

I'm not sure, it might - but given Bruce's effort to port to Meson I'm not sure it's worth spending a lot of time doing big refactoring of the existing build system

Not big refactoring, following seems worked for me, if you would like to test:

mk/rte.lib.mk +6 -1
index 13115d146..643da47da 100644
@@ -59,14 +59,19 @@ endif
 
 
 _BUILD = $(LIB)
-_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
+PREINSTALL = $(SYMLINK-FILES-y)
+_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)
 _CLEAN = doclean
 
 .PHONY: all
 all: install
 
 .PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
 install: build _postinstall
+else
+install: _preinstall build _postinstall
+endif
 
 _postinstall: build
 
Quoted 14 lines
Quoted 13 lines
Quoted 11 lines
Quoted 11 lines
lib/librte_member/Makefile new file +10 -0
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
#7 lore raw

Re: [PATCH 1/7] member: implement main API

Quoted 186 lines · Ferruh Yigit
Quoted 149 lines · Luca Boccassi
Quoted 148 lines · Ferruh Yigit
Quoted 139 lines · Luca Boccassi
Quoted 123 lines · Yipeng Wang

Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to
a
set(s). A membership test for an element will return the set this
element
belongs to or not-found if the element is never inserted into the
set-summary.

The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 + lib/librte_eal/common/eal_common_log.c   |   1 + lib/librte_eal/common/include/rte_log.h  |   1 + lib/librte_member/Makefile               |  48 +++ lib/librte_member/rte_member.c           | 357
+++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +

 7 files changed, 942 insertions(+)

 create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

lib/librte_member/Makefile new file +3 -0
new file mode 100644
index 0000000..997c825
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights

reserved.

lib/librte_member/Makefile new file +3 -0
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with

or
without
+#   modification, are permitted provided that the following conditions

lib/librte_member/Makefile new file +3 -0
+#   are met:
+#
+#     * Redistributions of source code must retain the above

copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with
the

lib/librte_member/Makefile new file +2 -0
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names

of
its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written permission.

lib/librte_member/Makefile new file +2 -0
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.

lib/librte_member/Makefile

Quoted 8 lines
lib/librte_member/Makefile new file +8 -0
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+

This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something
like:

Quoted 2 lines

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

Can we remove "-I$(SRCDIR)" completely by first installing headers
and
later compiling objects, all using $(RTE_OUT) only?

Quoted 7 lines
Quoted 1 line

Do you think can this work?

I'm not sure, it might - but given Bruce's effort to port to Meson I'm
not sure it's worth spending a lot of time doing big refactoring of the
existing build system

Not big refactoring, following seems worked for me, if you would like to
test:

mk/rte.lib.mk +2 -1
index 13115d146..643da47da 100644
@@ -59,14 +59,19 @@ endif
 
 
  _BUILD = $(LIB)
-_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y)
 $(RTE_OUTPUT)/lib/$(LIB)
+PREINSTALL = $(SYMLINK-FILES-y)
+_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)

 _CLEAN = doclean

 .PHONY: all
 all: install

 .PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
 install: build _postinstall

mk/rte.lib.mk +3 -0
+else
+install: _preinstall build _postinstall
+endif
Quoted 1 line

 _postinstall: build

Easier than I would have thought :-)

I'll give that patch a spin through the repro build grinder next week, thanks.

Signature · 2 lines
Kind regards,
Luca Boccassi
#8 lore raw

Re: [PATCH 1/7] member: implement main API

Quoted 202 lines · Luca Boccassi
Quoted 201 lines · Ferruh Yigit
Quoted 163 lines · Luca Boccassi
Quoted 155 lines · Ferruh Yigit
Quoted 151 lines · Luca Boccassi
Quoted 135 lines · Yipeng Wang

Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the
Membership
library is a data structure that provides a "set-summary" and
responds
to set-membership queries of whether a certain element belongs to
a
set(s). A membership test for an element will return the set
this
element
belongs to or not-found if the element is never inserted into
the
set-summary.

The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 + lib/librte_eal/common/eal_common_log.c   |   1 + lib/librte_eal/common/include/rte_log.h  |   1 + lib/librte_member/Makefile               |  48 +++ lib/librte_member/rte_member.c           | 357
+++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +

 7 files changed, 942 insertions(+)

 create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

lib/librte_member/Makefile new file +3 -0
new file mode 100644
index 0000000..997c825
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights

reserved.

lib/librte_member/Makefile new file +3 -0
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms,

with
or
without
+#   modification, are permitted provided that the following
conditions

lib/librte_member/Makefile new file +3 -0
+#   are met:
+#
+#     * Redistributions of source code must retain the

above
copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above
copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with
the

lib/librte_member/Makefile new file +2 -0
+#       distribution.
+#     * Neither the name of Intel Corporation nor the

names
of
its
+#       contributors may be used to endorse or promote products
derived
+#       from this software without specific prior written permission.

lib/librte_member/Makefile new file +2 -0
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT
NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT
NOT
+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF
USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND
ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF
THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH
DAMAGE.

lib/librte_member/Makefile

Quoted 8 lines
lib/librte_member/Makefile new file +8 -0
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+

This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something
like:

Quoted 2 lines

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

Quoted 6 lines

Can we remove "-I$(SRCDIR)" completely by first installing headers
and
later compiling objects, all using $(RTE_OUT) only?

Do you think can this work?

Quoted 7 lines

I'm not sure, it might - but given Bruce's effort to port to Meson
I'm
not sure it's worth spending a lot of time doing big refactoring of
the
existing build system

Not big refactoring, following seems worked for me, if you would like
to
test:

mk/rte.lib.mk +2 -1
index 13115d146..643da47da 100644
@@ -59,14 +59,19 @@ endif
 
 
  _BUILD = $(LIB)
-_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y)
 $(RTE_OUTPUT)/lib/$(LIB)
+PREINSTALL = $(SYMLINK-FILES-y)
+_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)

 _CLEAN = doclean

 .PHONY: all
 all: install

 .PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
 install: build _postinstall

mk/rte.lib.mk +3 -0
+else
+install: _preinstall build _postinstall
+endif
Quoted 7 lines
Quoted 1 line

 _postinstall: build

Easier than I would have thought :-)

I'll give that patch a spin through the repro build grinder next week,
thanks.

Thanks Ferruh, this works in making the build reproducible with the current CI as an alternative with a fixed build path.

But on the other hand, it will cause the compiler to always embed the custom build directory in the path to the header in the DWARF .debug files, rather than the "repository" path:

readelf --debug-dump=line /tmp/69c02db5050024b530a14555b49528c419419f.debug

<cut>
 The Directory Table (offset 0x1b):
  1     dpdk/lib/librte_eal/linuxapp/eal
  2     /usr/include/x86_64-linux-gnu/bits
  3     dpdk/lib/librte_eal/common
  4     dpdk/debian/build/shared-root/include 5     dpdk/debian/build/shared-root/include/generic 6     /usr/lib/gcc/x86_64-linux-gnu/7/include 7     /usr/include/x86_64-linux-gnu/sys
  8     /usr/include

 The File Name Table (offset 0x15a):
  Entry Dir     Time    Size    Name
  1     1       0       0       eal.c
  2     2       0       0       stdio2.h
  3     3       0       0       eal_filesystem.h 4     2       0       0       fcntl2.h
  5     4       0       0       rte_atomic.h 6     5       0       0       rte_atomic.h 7     2       0       0       string3.h
  8     4       0       0       rte_cycles.h 9     4       0       0       rte_random.h 10    6       0       0       xmmintrin.h 11    4       0       0       rte_eal_memconfig.h 12    4       0       0       rte_lcore.h

<cut>

note 4 and 5 items in the directory tables, they point to debian/build- shared-root which is the output directory we use in the package build.

This means that 2 builds that are otherwise exactly the same (same machine type, same build dependencies) will not be binary identical due to that difference.

On one hand it's a much smaller and nicer patch and there's the Meson work which will make both obsolete, on the other hand it's not very reproducibl-y. What do you think?

Signature · 2 lines
Kind regards,
Luca Boccassi
#9 lore raw

Re: [PATCH 1/7] member: implement main API

In reply to Luca Boccassi
Quoted 116 lines

-----Original Message-----
From: Luca Boccassi [mailto:[email protected]]
Sent: Tuesday, August 22, 2017 3:02 AM
To: Wang, Yipeng1 <[email protected]>
Cc: [email protected]
Subject: Re: [dpdk-dev] [PATCH 1/7] member: implement main API

Quoted 73 lines · Yipeng Wang

Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 + lib/librte_eal/common/eal_common_log.c   |   1 + lib/librte_eal/common/include/rte_log.h  |   1 + lib/librte_member/Makefile               |  48 +++ lib/librte_member/rte_member.c           | 357 +++++++++++++++++++++ lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +

 7 files changed, 942 insertions(+)

 create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

lib/librte_member/Makefile new file +6 -0
new file mode 100644
index 0000000..997c825
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or

without
+#   modification, are permitted provided that the following conditions

lib/librte_member/Makefile new file +3 -0
+#   are met:
+#
+#     * Redistributions of source code must retain the above

copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer in

lib/librte_member/Makefile new file +4 -0
+#       the documentation and/or other materials provided with the
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of its
+#       contributors may be used to endorse or promote products

derived
+#       from this software without specific prior written permission.

lib/librte_member/Makefile new file +2 -0
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT

Quoted 8 lines

NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,

BUT NOT

Quoted 1 line

+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;

LOSS OF

Quoted 2 lines

USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED

AND

Quoted 4 lines

ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT

OF

lib/librte_member/Makefile line 38

Quoted 11 lines

THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

lib/librte_member/Makefile new file +8 -0
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+

This breaks reproducibility as the output directory will be included before the source directory, causing a race - please do something like:

Quoted 3 lines

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

[Wang, Yipeng]
Thanks Luca, I will reverse the include order in next version.

I am not an expert on the build system, to be more clear, the race condition you mentioned is because the library may search build directory first that some symlinks may have not been created yet, right?

Thanks

Quoted 14 lines

lib/librte_member/Makefile line 48

Quoted 10 lines
lib/librte_member/Makefile new file +10 -0
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk

--
Kind regards,
Luca Boccassi

#10 lore raw

Re: [PATCH 1/7] member: implement main API

Quoted 136 lines · On Thu, 2017-08-24 at 18:38 +0000, Wang, Yipeng1 wrote:
Quoted 129 lines

-----Original Message-----
From: Luca Boccassi [mailto:[email protected]]
Sent: Tuesday, August 22, 2017 3:02 AM
To: Wang, Yipeng1 <[email protected]>
Cc: [email protected]
Subject: Re: [dpdk-dev] [PATCH 1/7] member: implement main API

Quoted 80 lines · Yipeng Wang

Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 + lib/librte_eal/common/eal_common_log.c   |   1 + lib/librte_eal/common/include/rte_log.h  |   1 + lib/librte_member/Makefile               |  48 +++ lib/librte_member/rte_member.c           | 357
+++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  15 +

 7 files changed, 942 insertions(+)

 create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

lib/librte_member/Makefile new file +6 -0
new file mode 100644
index 0000000..997c825
@@ -0,0 +1,48 @@
+#   BSD LICENSE
+#
+#   Copyright(c) 2017 Intel Corporation. All rights reserved.
+#   All rights reserved.
+#
+#   Redistribution and use in source and binary forms, with or

without
+#   modification, are permitted provided that the following conditions

lib/librte_member/Makefile new file +3 -0
+#   are met:
+#
+#     * Redistributions of source code must retain the above

copyright
+#       notice, this list of conditions and the following disclaimer.
+#     * Redistributions in binary form must reproduce the above copyright
+#       notice, this list of conditions and the following disclaimer
in
+#       the documentation and/or other materials provided with the

lib/librte_member/Makefile new file +2 -0
+#       distribution.
+#     * Neither the name of Intel Corporation nor the names of

its
+#       contributors may be used to endorse or promote products derived
+#       from this software without specific prior written permission.

lib/librte_member/Makefile new file +2 -0
+#
+#   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

CONTRIBUTORS
+#   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT

Quoted 8 lines

NOT
+#   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+#   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+#   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,

BUT NOT

Quoted 1 line

+#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;

LOSS OF

Quoted 2 lines

USE,
+#   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED

AND

Quoted 5 lines

ON ANY
+#   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+#   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT

OF

lib/librte_member/Makefile line 38

Quoted 11 lines

THE USE
+#   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

lib/librte_member/Makefile new file +8 -0
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_member.a
+
+CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
+

This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something like:

Quoted 3 lines

CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3

[Wang, Yipeng]
Thanks Luca, I will reverse the include order in next version.

Quoted 7 lines

I am not an expert on the build system, to be more clear, the race condition you
mentioned is because the library may search build directory first that some symlinks
may have not been created yet, right?

Thanks

Yes, and as a consequence on some builds the DWARF symbols list lib/libfoo/foo.h as the source file path in the .debug, and on some builds they list build-dir/include/foo.h

Quoted 14 lines
Quoted 14 lines

lib/librte_member/Makefile line 48

Quoted 10 lines
lib/librte_member/Makefile new file +10 -0
+EXPORT_MAP := rte_member_version.map
+
+LIBABIVER := 1
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+# install includes
+SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
+
+include $(RTE_SDK)/mk/rte.lib.mk

--
Kind regards,
Luca Boccassi

Signature · 2 lines
Kind regards,
Luca Boccassi
#11 lore raw

PATCH 2/7 member: implement HT mode

In reply to Yipeng Wang

One of the set-summray structure is hash-table based set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is more similar to traditional cuckoo filter. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. So the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.

This patch add the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

---
 lib/librte_member/Makefile        |   2 +-
 lib/librte_member/rte_member_ht.c | 488 ++++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_ht.h | 115 +++++++++
 3 files changed, 604 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
index 997c825..110e4be 100644
@@ -41,7 +41,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_ht.c new file +488 -0

488 lines not on this page · raw message

lib/librte_member/rte_member_ht.h new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
2.7.4
#12 lore raw

PATCH 3/7 member: implement vBF mode

In reply to Yipeng Wang

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set arbitrarily so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

---
 lib/librte_member/Makefile         |   2 +-
 lib/librte_member/rte_member_vbf.c | 309 +++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_vbf.h |  85 ++++++++++
 3 files changed, 395 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
index 110e4be..90bdfe8 100644
@@ -41,7 +41,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c rte_member_vbf.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_vbf.c new file +309 -0

309 lines not on this page · raw message

lib/librte_member/rte_member_vbf.h new file +85 -0

85 lines not on this page · raw message

Signature · 1 line
2.7.4
#13 lore raw

PATCH 4/7 member: add AVX for HT mode

In reply to Yipeng Wang

For key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

---
 lib/librte_member/rte_member_ht.c  | 143 ++++++++++++++++++++++++++++---------
 lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++
 2 files changed, 222 insertions(+), 32 deletions(-)
 create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -32
index 3e411ed..2f52220 100644
@@ -40,6 +40,10 @@
 #include "rte_member.h"
 #include "rte_member_ht.h"
 
+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+
 
 static inline int
 insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -133,6 +137,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
 		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
 			buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
 	}
+#if defined(RTE_ARCH_X86)
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+			RTE_MEMBER_BUCKET_ENTRIES == 16)
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+	else
+#endif
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;
 
 
 	RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -172,11 +183,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
 	*set_id = RTE_MEMBER_NO_MATCH;
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	if (search_bucket_single(prim_bucket, tmp_sig, buckets,
-			set_id) ||
-			search_bucket_single(sec_bucket, tmp_sig,
-				buckets, set_id))
-		return 1;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single_avx(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+		break;
+#endif
+	default:
+		if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+	}
 
 	return 0;
 }
@@ -201,13 +224,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
 	}
 
 	for (i = 0; i < num_keys; i++) {
-		if (search_bucket_single(prim_buckets[i], tmp_sig[i],
-				buckets, &set_id[i]) ||
-				search_bucket_single(sec_buckets[i],
-				tmp_sig[i], buckets, &set_id[i]))
-			ret++;
-		else
-			set_id[i] = RTE_MEMBER_NO_MATCH;
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			if (search_bucket_single_avx(prim_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]) ||
+				search_bucket_single_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				ret++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+			break;
+#endif
+		default:
+			if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+					buckets, &set_id[i]) ||
+					search_bucket_single(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				ret++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+		}
 	}
 	return ret;
 }
@@ -225,12 +262,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
 
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
-			 match_per_key, set_id);
-	if (ret < match_per_key)
-		search_bucket_multi(sec_bucket, tmp_sig,
-			buckets, &ret, match_per_key, set_id);
-	return ret;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+			&ret, match_per_key, set_id);
+		if (ret < match_per_key)
+			search_bucket_multi_avx(sec_bucket, tmp_sig,
+				buckets, &ret, match_per_key, set_id);
+		return ret;
+#endif
+	default:
+		search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+				 match_per_key, set_id);
+		if (ret < match_per_key)
+			search_bucket_multi(sec_bucket, tmp_sig,
+				buckets, &ret, match_per_key, set_id);
+		return ret;
+	}
 }
 
 
@@ -257,16 +306,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
 	for (i = 0; i < num_keys; i++) {
 		match_cnt_t = 0;
 
-		search_bucket_multi(prim_buckets[i], tmp_sig[i],
-			buckets, &match_cnt_t, match_per_key,
-			&set_ids[i*match_per_key]);
-		if (match_cnt_t < match_per_key)
-			search_bucket_multi(sec_buckets[i], tmp_sig[i],
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
 				buckets, &match_cnt_t, match_per_key,
 				&set_ids[i*match_per_key]);
-		match_count[i] = match_cnt_t;
-		if (match_cnt_t != 0)
-			ret++;
+			if (match_cnt_t < match_per_key)
+				search_bucket_multi_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &match_cnt_t,
+					match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_t;
+			if (match_cnt_t != 0)
+				ret++;
+			break;
+#endif
+		default:
+			search_bucket_multi(prim_buckets[i], tmp_sig[i],
+				buckets, &match_cnt_t, match_per_key,
+				&set_ids[i*match_per_key]);
+			if (match_cnt_t < match_per_key)
+				search_bucket_multi(sec_buckets[i], tmp_sig[i],
+					buckets, &match_cnt_t, match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_t;
+			if (match_cnt_t != 0)
+				ret++;
+		}
 	}
 	return ret;
 }
@@ -298,12 +365,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
 
 static inline int
 try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
-		SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+		SIG_TYPE sig, MEMBER_SET_TYPE set_id,
+		enum rte_member_sig_compare_function cmp_fn)
 {
-	if (insert_overwrite_search(prim, sig, buckets, set_id) ||
-			insert_overwrite_search(sec, sig, buckets,
-				set_id))
-		return 0;
+	switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (insert_overwrite_search_avx(prim, sig, buckets, set_id) ||
+				insert_overwrite_search_avx(sec, sig, buckets,
+					set_id))
+			return 0;
+		break;
+#endif
+	default:
+		if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+				insert_overwrite_search(sec, sig, buckets,
+					set_id))
+			return 0;
+	}
 	return -1;
 }
 
@@ -409,7 +488,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
 	/* if it is cache based filter, we try overwriting existing entry */
 	if (ss->cache) {
 		ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
-					set_id);
+					set_id, ss->sig_cmp_fn);
 		if (ret != -1)
 			return ret;
 	}
lib/librte_member/rte_member_x86.h new file +111 -0

111 lines not on this page · raw message

Signature · 1 line
2.7.4
#14 lore raw

PATCH 5/7 member: enable the library

In reply to Yipeng Wang

This patch enables the Membership library.

---
 MAINTAINERS                | 7 +++++++
 config/common_base         | 5 +++++
 lib/librte_member/Makefile | 2 ++
 mk/rte.app.mk              | 2 ++
 4 files changed, 16 insertions(+)
MAINTAINERS +7 -0
index a0cd75e..e372edf 100644
@@ -734,6 +734,13 @@ F: test/test/test_meter.c
 F: examples/qos_meter/
 F: doc/guides/sample_app_ug/qos_metering.rst
 
+Membership - EXPERIMENTAL
+M: Yipeng Wang <[email protected]>
+M: Sameh Gobriel <[email protected]>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
+
 
 Other libraries
 ---------------
config/common_base +5 -0
index 5e97a08..5e31ced 100644
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
 CONFIG_RTE_LIBRTE_EFD=y
 
 #
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
 # Compile librte_jobstats
 #
 CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
index 90bdfe8..3c2c4e0 100644
@@ -36,6 +36,8 @@ LIB = librte_member.a
 
 CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3
 
+LDLIBS += -lm
+
 EXPORT_MAP := rte_member_version.map
 
 LIBABIVER := 1
mk/rte.app.mk +2 -0
index c25fdd9..c79acf0 100644
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE)        += -lrte_cfgfile
 _LDLIBS-y += --whole-archive
 
 _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH)           += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)        += -lrte_member
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lrte_vhost
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS)         += -lrte_kvargs
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lm
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrt
 _LDLIBS-$(CONFIG_RTE_LIBRTE_METER)          += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)          += -lm
 ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lnuma
 endif
Signature · 1 line
2.7.4
#15 lore raw

PATCH 6/7 test/member: add functional and perf tests

In reply to Yipeng Wang

This patch adds functional and performance tests for membership library.

---
 test/test/Makefile           |   3 +
 test/test/test_member.c      | 550 ++++++++++++++++++++++++++++++++++++
 test/test/test_member_perf.c | 643 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1196 insertions(+)
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
test/test/Makefile +3 -0
index 42d9a49..b61dde3 100644
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
 SRCS-y += test_memcpy.c
 SRCS-y += test_memcpy_perf.c
 
+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
 
test/test/test_member.c new file +550 -0

550 lines not on this page · raw message

test/test/test_member_perf.c new file +643 -0

643 lines not on this page · raw message

Signature · 1 line
2.7.4
#16 lore raw

PATCH 7/7 doc: add membership documentation

In reply to Yipeng Wang

This patch adds the documentation for membership library.

---
 doc/api/doxy-api-index.md               |    3 +-
 doc/api/doxy-api.conf                   |    1 +
 doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg |   36 +
 doc/guides/prog_guide/img/member_i3.svg |  148 +++
 doc/guides/prog_guide/img/member_i4.svg |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg |  163 ++++
 doc/guides/prog_guide/img/member_i6.svg |  332 +++++++
 doc/guides/prog_guide/img/member_i7.svg |  399 ++++++++
 doc/guides/prog_guide/index.rst         |   14 +
 doc/guides/prog_guide/member_lib.rst    |  432 +++++++++
 doc/guides/rel_notes/release_17_11.rst  |   17 +
 12 files changed, 3607 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
doc/api/doxy-api-index.md +2 -1
index 19e0d4f..fe87e09 100644
@@ -105,7 +105,8 @@ The public API headers are grouped by topics:
   [LPM IPv4 route]     (@ref rte_lpm.h),
   [LPM IPv6 route]     (@ref rte_lpm6.h),
   [ACL]                (@ref rte_acl.h),
-  [EFD]                (@ref rte_efd.h)
+  [EFD]                (@ref rte_efd.h),
+  [member]             (@ref rte_member.h)
 
 - **QoS**:
   [metering]           (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
index 823554f..b792d6d 100644
@@ -58,6 +58,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_mempool \
                           lib/librte_meter \
                           lib/librte_metrics \
+                          lib/librte_member \
                           lib/librte_net \
                           lib/librte_pdump \
                           lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0

1,613 lines not on this page · raw message

doc/guides/prog_guide/img/member_i2.svg new file +36 -0
new file mode 100644
index 0000000..759c654
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in"
+		viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+	<v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {fill:none;stroke:none;stroke-width:0.25}
+		.st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+		.st3 {baseline-shift:32.4943%;font-size:0.649886em}
+		.st4 {font-size:1em}
+		.st5 {font-family:Cambria Math;font-size:1em}
+		.st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+		<g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)">
+			<title>Sheet.3</title>
+			<desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc>
+			<v:textBlock v:margins="rect(4,4,4,4)"/>
+			<v:textRect cx="157.5" cy="45.5" width="315" height="90"/>
+			<rect x="0" y="0.5" width="315" height="90" class="st1"/>
+			<text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan
+						class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3"
+						v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em"
+						class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text>		</g>
+	</g>
+</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0

148 lines not on this page · raw message

doc/guides/prog_guide/img/member_i4.svg new file +450 -0

450 lines not on this page · raw message

doc/guides/prog_guide/img/member_i5.svg new file +163 -0

163 lines not on this page · raw message

doc/guides/prog_guide/img/member_i6.svg new file +332 -0

332 lines not on this page · raw message

doc/guides/prog_guide/img/member_i7.svg new file +399 -0

242 lines not on this page · raw message

doc/guides/prog_guide/index.rst +14 -0
index 40f04a1..7ff5144 100644

Lines of this file are not shown. View the raw message.

doc/guides/prog_guide/member_lib.rst new file +432 -0

432 lines not on this page · raw message

doc/guides/rel_notes/release_17_11.rst +17 -0
index 170f4f9..4002df3 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.7.4

Part of the diff is not shown here. View the raw message.

#17 lore raw
In reply to Yipeng Wang
Quoted 11 lines · On Mon, 21 Aug 2017 17:19:46 -0700 Yipeng Wang <[email protected]> wrote:

This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.

I don't think it makes sense to merge two different types of tables in one API. Especially in DPDK where every cycle counts. You are taking an extra branch on each lookup. The user of this API is likely to know exactly what type of objects and look are desired.

#18 lore raw

Hi Stephen,

Thanks for the comments. We will remove the needless return initialization in next version.

For the unified API concern, we think the current implementation has two benefits: 1. It is easier to extend the library with new types of filters without adding a lot of top level APIs every time. 2. When users switch between different types in their application, they do not need to switch between function calls. They can reuse same code for different types.

However, we agree with you that a switch case in the library affects the performance. We did a quick test and here is the results (cycles/operation):
lookup lookup_bulk(16 batch) lookup_multi lookup_multi_bulk
switch_case 50 36 57 45
direct call 47 35 54 45

There will be 3 cycle difference for non-bulk version lookup. I guess if users usually use bulk version, it maybe not a big concern, but single key lookup indeed slower. If you think the benefit we mentioned does not outweigh the performance slowdown, we would like to make the change.

We also considered using function pointers to get rid of switch case, but it will be unfriendly to the multi-process usages.

Please comments.

Thanks
Yipeng

Quoted 31 lines

-----Original Message-----
From: Stephen Hemminger [mailto:[email protected]]
Sent: Monday, August 21, 2017 9:02 PM
To: Wang, Yipeng1 <[email protected]>
Cc: [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>
Subject: Re: [PATCH 0/7] Add Membership Library

Quoted 9 lines · On Mon, 21 Aug 2017 17:19:46 -0700 Yipeng Wang <[email protected]> wrote:

This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF

only has false-positives similar to the non-cache HTSS.

Quoted 2 lines

However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.

I don't think it makes sense to merge two different types of tables in one API. Especially in DPDK where every cycle counts. You are taking an extra branch on each lookup. The user of this API is likely to know exactly what type of objects and look are desired.

#19 lore raw

v2 cover Add Membership Library

In reply to Yipeng Wang

DPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.

A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitilize issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment
- documentation: update the programmer guide for membership library

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

 MAINTAINERS                              |    7 +
 config/common_base                       |    5 +
 doc/api/doxy-api-index.md                |    3 +-
 doc/api/doxy-api.conf                    |    1 +
 doc/guides/prog_guide/img/member_i1.svg  | 1613 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg  |   36 +
 doc/guides/prog_guide/img/member_i3.svg  |  148 +++
 doc/guides/prog_guide/img/member_i4.svg  |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg  |  163 +++
 doc/guides/prog_guide/img/member_i6.svg  |  332 ++++++
 doc/guides/prog_guide/img/member_i7.svg  |  399 ++++++++
 doc/guides/prog_guide/index.rst          |   14 +
 doc/guides/prog_guide/member_lib.rst     |  440 ++++++++
 doc/guides/rel_notes/release_17_11.rst   |   17 +
 lib/Makefile                             |    2 +
 lib/librte_eal/common/eal_common_log.c   |    1 +
 lib/librte_eal/common/include/rte_log.h  |    1 +
 lib/librte_member/Makefile               |   51 +
 lib/librte_member/rte_member.c           |  342 +++++++
 lib/librte_member/rte_member.h           |  518 ++++++++++
 lib/librte_member/rte_member_ht.c        |  569 +++++++++++
 lib/librte_member/rte_member_ht.h        |  115 +++
 lib/librte_member/rte_member_vbf.c       |  350 +++++++
 lib/librte_member/rte_member_vbf.h       |   85 ++
 lib/librte_member/rte_member_version.map |   16 +
 lib/librte_member/rte_member_x86.h       |  111 ++
 mk/rte.app.mk                            |    2 +
 test/test/Makefile                       |    3 +
 test/test/test_member.c                  |  682 +++++++++++++
 test/test/test_member_perf.c             |  643 ++++++++++++
 30 files changed, 7118 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
 create mode 100644 lib/librte_member/rte_member_version.map
 create mode 100644 lib/librte_member/rte_member_x86.h
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
#20 lore raw

PATCH v2 1/7 member: implement main API

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  49 +++
 lib/librte_member/rte_member.c           | 342 ++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518 +++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  16 +
 7 files changed, 929 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
index 86caba1..c82033a 100644
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
 DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
 DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_eal/common/eal_common_log.c +1 -0
index 0e3b932..90db6a3 100644
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = {
 	{RTE_LOGTYPE_CRYPTODEV,  "cryptodev"},
 	{RTE_LOGTYPE_EFD,        "efd"},
 	{RTE_LOGTYPE_EVENTDEV,   "eventdev"},
+	{RTE_LOGTYPE_MEMBER,     "member"},
 	{RTE_LOGTYPE_USER1,      "user1"},
 	{RTE_LOGTYPE_USER2,      "user2"},
 	{RTE_LOGTYPE_USER3,      "user3"},
lib/librte_eal/common/include/rte_log.h +1 -0
index ec8dba7..ce1a3d0 100644
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;
 #define RTE_LOGTYPE_CRYPTODEV 17 /**< Log related to cryptodev. */
 #define RTE_LOGTYPE_EFD       18 /**< Log related to EFD. */
 #define RTE_LOGTYPE_EVENTDEV  19 /**< Log related to eventdev. */
+#define RTE_LOGTYPE_MEMBER    20 /**< Log related to membership. */
 
 /* these log types can be used in an application */
 #define RTE_LOGTYPE_USER1     24 /**< User-defined log type 1. */
lib/librte_member/Makefile new file +49 -0

49 lines not on this page · raw message

lib/librte_member/rte_member.c new file +342 -0

342 lines not on this page · raw message

lib/librte_member/rte_member.h new file +518 -0

518 lines not on this page · raw message

lib/librte_member/rte_member_version.map new file +16 -0
new file mode 100644
index 0000000..a5877c9
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+	global:
+
+	rte_member_create;
+	rte_member_find_existing;
+	rte_member_lookup;
+	rte_member_lookup_bulk;
+	rte_member_lookup_multi;
+	rte_member_lookup_multi_bulk;
+	rte_member_add;
+	rte_member_free;
+	rte_member_reset;
+	rte_member_delete;
+
+	local: *;
+};
Signature · 1 line
2.7.4
#21 lore raw

PATCH v2 2/7 member: implement HT mode

In reply to Yipeng Wang

One of the set-summray structure is hash-table based set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is more similar to traditional cuckoo filter. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. So the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.

This patch add the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

---
 lib/librte_member/Makefile        |   2 +-
 lib/librte_member/rte_member_ht.c | 490 ++++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_ht.h | 115 +++++++++
 3 files changed, 606 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
index 1a79eaa..ad26548 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_ht.c new file +490 -0

490 lines not on this page · raw message

lib/librte_member/rte_member_ht.h new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
2.7.4
#22 lore raw

PATCH v2 3/7 member: implement vBF mode

In reply to Yipeng Wang

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set arbitrarily so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

---
 lib/librte_member/Makefile         |   2 +-
 lib/librte_member/rte_member_vbf.c | 350 +++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_vbf.h |  85 +++++++++
 3 files changed, 436 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
index ad26548..50275ed 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c rte_member_vbf.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_vbf.c new file +350 -0

350 lines not on this page · raw message

lib/librte_member/rte_member_vbf.h new file +85 -0

85 lines not on this page · raw message

Signature · 1 line
2.7.4
#23 lore raw

PATCH v2 4/7 member: add AVX for HT mode

In reply to Yipeng Wang

For key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

---
 lib/librte_member/rte_member_ht.c  | 143 ++++++++++++++++++++++++++++---------
 lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++
 2 files changed, 222 insertions(+), 32 deletions(-)
 create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -32
index b2ae6d0..15e2534 100644
@@ -40,6 +40,10 @@
 #include "rte_member.h"
 #include "rte_member_ht.h"
 
+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+
 
 static inline int
 insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -135,6 +139,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
 		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
 			buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
 	}
+#if defined(RTE_ARCH_X86)
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+			RTE_MEMBER_BUCKET_ENTRIES == 16)
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+	else
+#endif
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;
 
 
 	RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -174,11 +185,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
 	*set_id = RTE_MEMBER_NO_MATCH;
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	if (search_bucket_single(prim_bucket, tmp_sig, buckets,
-			set_id) ||
-			search_bucket_single(sec_bucket, tmp_sig,
-				buckets, set_id))
-		return 1;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single_avx(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+		break;
+#endif
+	default:
+		if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+	}
 
 	return 0;
 }
@@ -203,13 +226,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
 	}
 
 	for (i = 0; i < num_keys; i++) {
-		if (search_bucket_single(prim_buckets[i], tmp_sig[i],
-				buckets, &set_id[i]) ||
-				search_bucket_single(sec_buckets[i],
-				tmp_sig[i], buckets, &set_id[i]))
-			ret++;
-		else
-			set_id[i] = RTE_MEMBER_NO_MATCH;
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			if (search_bucket_single_avx(prim_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]) ||
+				search_bucket_single_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				ret++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+			break;
+#endif
+		default:
+			if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+					buckets, &set_id[i]) ||
+					search_bucket_single(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				ret++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+		}
 	}
 	return ret;
 }
@@ -227,12 +264,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
 
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
-			 match_per_key, set_id);
-	if (ret < match_per_key)
-		search_bucket_multi(sec_bucket, tmp_sig,
-			buckets, &ret, match_per_key, set_id);
-	return ret;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+			&ret, match_per_key, set_id);
+		if (ret < match_per_key)
+			search_bucket_multi_avx(sec_bucket, tmp_sig,
+				buckets, &ret, match_per_key, set_id);
+		return ret;
+#endif
+	default:
+		search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+				 match_per_key, set_id);
+		if (ret < match_per_key)
+			search_bucket_multi(sec_bucket, tmp_sig,
+				buckets, &ret, match_per_key, set_id);
+		return ret;
+	}
 }
 
 
@@ -259,16 +308,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
 	for (i = 0; i < num_keys; i++) {
 		match_cnt_t = 0;
 
-		search_bucket_multi(prim_buckets[i], tmp_sig[i],
-			buckets, &match_cnt_t, match_per_key,
-			&set_ids[i*match_per_key]);
-		if (match_cnt_t < match_per_key)
-			search_bucket_multi(sec_buckets[i], tmp_sig[i],
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
 				buckets, &match_cnt_t, match_per_key,
 				&set_ids[i*match_per_key]);
-		match_count[i] = match_cnt_t;
-		if (match_cnt_t != 0)
-			ret++;
+			if (match_cnt_t < match_per_key)
+				search_bucket_multi_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &match_cnt_t,
+					match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_t;
+			if (match_cnt_t != 0)
+				ret++;
+			break;
+#endif
+		default:
+			search_bucket_multi(prim_buckets[i], tmp_sig[i],
+				buckets, &match_cnt_t, match_per_key,
+				&set_ids[i*match_per_key]);
+			if (match_cnt_t < match_per_key)
+				search_bucket_multi(sec_buckets[i], tmp_sig[i],
+					buckets, &match_cnt_t, match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_t;
+			if (match_cnt_t != 0)
+				ret++;
+		}
 	}
 	return ret;
 }
@@ -300,12 +367,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
 
 static inline int
 try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
-		SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+		SIG_TYPE sig, MEMBER_SET_TYPE set_id,
+		enum rte_member_sig_compare_function cmp_fn)
 {
-	if (insert_overwrite_search(prim, sig, buckets, set_id) ||
-			insert_overwrite_search(sec, sig, buckets,
-				set_id))
-		return 0;
+	switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (insert_overwrite_search_avx(prim, sig, buckets, set_id) ||
+				insert_overwrite_search_avx(sec, sig, buckets,
+					set_id))
+			return 0;
+		break;
+#endif
+	default:
+		if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+				insert_overwrite_search(sec, sig, buckets,
+					set_id))
+			return 0;
+	}
 	return -1;
 }
 
@@ -411,7 +490,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
 	/* if it is cache based filter, we try overwriting existing entry */
 	if (ss->cache) {
 		ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
-					set_id);
+					set_id, ss->sig_cmp_fn);
 		if (ret != -1)
 			return ret;
 	}
lib/librte_member/rte_member_x86.h new file +111 -0

111 lines not on this page · raw message

Signature · 1 line
2.7.4
#24 lore raw

PATCH v2 5/7 member: enable the library

In reply to Yipeng Wang

This patch enables the Membership library.

---
 MAINTAINERS                | 7 +++++++
 config/common_base         | 5 +++++
 lib/librte_member/Makefile | 2 ++
 mk/rte.app.mk              | 2 ++
 4 files changed, 16 insertions(+)
MAINTAINERS +7 -0
index a0cd75e..e372edf 100644
@@ -734,6 +734,13 @@ F: test/test/test_meter.c
 F: examples/qos_meter/
 F: doc/guides/sample_app_ug/qos_metering.rst
 
+Membership - EXPERIMENTAL
+M: Yipeng Wang <[email protected]>
+M: Sameh Gobriel <[email protected]>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
+
 
 Other libraries
 ---------------
config/common_base +5 -0
index 5e97a08..5e31ced 100644
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
 CONFIG_RTE_LIBRTE_EFD=y
 
 #
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
 # Compile librte_jobstats
 #
 CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
index 50275ed..3bac1d0 100644
@@ -37,6 +37,8 @@ LIB = librte_member.a
 CFLAGS := -I$(SRCDIR) $(CFLAGS)
 CFLAGS += $(WERROR_FLAGS) -O3
 
+LDLIBS += -lm
+
 EXPORT_MAP := rte_member_version.map
 
 LIBABIVER := 1
mk/rte.app.mk +2 -0
index c25fdd9..c79acf0 100644
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE)        += -lrte_cfgfile
 _LDLIBS-y += --whole-archive
 
 _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH)           += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)        += -lrte_member
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lrte_vhost
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS)         += -lrte_kvargs
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lm
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrt
 _LDLIBS-$(CONFIG_RTE_LIBRTE_METER)          += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)          += -lm
 ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lnuma
 endif
Signature · 1 line
2.7.4
#25 lore raw

PATCH v2 6/7 test/member: add functional and perf tests

In reply to Yipeng Wang

This patch adds functional and performance tests for membership library.

---
 test/test/Makefile           |   3 +
 test/test/test_member.c      | 682 +++++++++++++++++++++++++++++++++++++++++++
 test/test/test_member_perf.c | 643 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1328 insertions(+)
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
test/test/Makefile +3 -0
index 42d9a49..b61dde3 100644
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
 SRCS-y += test_memcpy.c
 SRCS-y += test_memcpy_perf.c
 
+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
 
test/test/test_member.c new file +682 -0

682 lines not on this page · raw message

test/test/test_member_perf.c new file +643 -0

643 lines not on this page · raw message

Signature · 1 line
2.7.4
#26 lore raw

PATCH v2 7/7 doc: add membership documentation

In reply to Yipeng Wang

This patch adds the documentation for membership library.

---
 doc/api/doxy-api-index.md               |    3 +-
 doc/api/doxy-api.conf                   |    1 +
 doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg |   36 +
 doc/guides/prog_guide/img/member_i3.svg |  148 +++
 doc/guides/prog_guide/img/member_i4.svg |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg |  163 ++++
 doc/guides/prog_guide/img/member_i6.svg |  332 +++++++
 doc/guides/prog_guide/img/member_i7.svg |  399 ++++++++
 doc/guides/prog_guide/index.rst         |   14 +
 doc/guides/prog_guide/member_lib.rst    |  440 +++++++++
 doc/guides/rel_notes/release_17_11.rst  |   17 +
 12 files changed, 3615 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
doc/api/doxy-api-index.md +2 -1
index 19e0d4f..fe87e09 100644
@@ -105,7 +105,8 @@ The public API headers are grouped by topics:
   [LPM IPv4 route]     (@ref rte_lpm.h),
   [LPM IPv6 route]     (@ref rte_lpm6.h),
   [ACL]                (@ref rte_acl.h),
-  [EFD]                (@ref rte_efd.h)
+  [EFD]                (@ref rte_efd.h),
+  [member]             (@ref rte_member.h)
 
 - **QoS**:
   [metering]           (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
index 823554f..b792d6d 100644
@@ -58,6 +58,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_mempool \
                           lib/librte_meter \
                           lib/librte_metrics \
+                          lib/librte_member \
                           lib/librte_net \
                           lib/librte_pdump \
                           lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0

1,613 lines not on this page · raw message

doc/guides/prog_guide/img/member_i2.svg new file +36 -0
new file mode 100644
index 0000000..759c654
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in"
+		viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+	<v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {fill:none;stroke:none;stroke-width:0.25}
+		.st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+		.st3 {baseline-shift:32.4943%;font-size:0.649886em}
+		.st4 {font-size:1em}
+		.st5 {font-family:Cambria Math;font-size:1em}
+		.st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+		<g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)">
+			<title>Sheet.3</title>
+			<desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc>
+			<v:textBlock v:margins="rect(4,4,4,4)"/>
+			<v:textRect cx="157.5" cy="45.5" width="315" height="90"/>
+			<rect x="0" y="0.5" width="315" height="90" class="st1"/>
+			<text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan
+						class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3"
+						v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em"
+						class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text>		</g>
+	</g>
+</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0

148 lines not on this page · raw message

doc/guides/prog_guide/img/member_i4.svg new file +450 -0

450 lines not on this page · raw message

doc/guides/prog_guide/img/member_i5.svg new file +163 -0

163 lines not on this page · raw message

doc/guides/prog_guide/img/member_i6.svg new file +332 -0

332 lines not on this page · raw message

doc/guides/prog_guide/img/member_i7.svg new file +399 -0

242 lines not on this page · raw message

doc/guides/prog_guide/index.rst +14 -0
index 40f04a1..7ff5144 100644

Lines of this file are not shown. View the raw message.

doc/guides/prog_guide/member_lib.rst new file +440 -0

440 lines not on this page · raw message

doc/guides/rel_notes/release_17_11.rst +17 -0
index 170f4f9..4002df3 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.7.4

Part of the diff is not shown here. View the raw message.

Reviewed-by on v2 7/7
#27 lore raw

Re: [PATCH v2 7/7] doc: add membership documentation

Quoted 10 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Yipeng Wang
Sent: Saturday, September 2, 2017 2:25 AM
To: [email protected]
Cc: [email protected]; [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [dpdk-dev] [PATCH v2 7/7] doc: add membership documentation

Quoted 4 lines

This patch adds the documentation for membership library.

---

Thanks for the detailed docs and the nice images. Some comments below.

doc/guides/prog_guide/member_lib.rst line 35

Quoted 5 lines
doc/guides/prog_guide/member_lib.rst new file +5 -0
+
+.. _Member_Library:
+
+Membership Library
+=======================

The underline should match the length of the title and should be followed by a blank line. Make this change throughout the doc.

Quoted 18 lines
doc/guides/prog_guide/member_lib.rst new file +4 -0
+
+Introduction
+------------
+The DPDK Membership Library provides an API for DPDK applications to

insert a
+new member, delete an existing member, or query the existence of a member in a
+given set, or a group of sets. For the case of a group of sets the library
+will return not only whether the element has been inserted before in one of

(quoted diff) +2 -0
+the sets but also which set it belongs to.  The Membership Library is an
+extension and generalization of a traditional filter structure (for

example
+Bloom Filter [Member-bloom]) that has multiple usages in a wide variety of

(quoted diff) +2 -0
+workloads and applications. In general, the Membership Library is a data
+structure that provides a “set-summary” on whether a member belongs to a
Quoted 1 line

set,

Use standard quotes instead of smart quotes. The doc renderers will convert them to smart quotes.

Quoted 8 lines
doc/guides/prog_guide/member_lib.rst new file +8 -0
+probability.
+
+.. _figure_membership1:
+.. figure:: img/member_i1.*
+
+  Example Usages of Membership Library
+
+We believe that there are various usages for a Membership Library in a
Quoted 6 lines

very
+large set of applications and workloads. Interested readers can refer to +[Member-survey] for a survey of possible networking usages. The following figure
+provide a small set of examples of using the Membership Library. Sub- figure(a)

These sub-figures would be better as separate bullet sections.

There are a lot of other small fixes that I will send on to you off-line.

#28 lore raw

v3 cover Add Membership Library

In reply to Yipeng Wang

DPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.

A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

v3:
- documentation: update documentation to incorperate John's review.

v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitilize issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment.
- documentation: update the programmer guide for membership library.

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

 MAINTAINERS                              |    7 +
 config/common_base                       |    5 +
 doc/api/doxy-api-index.md                |    3 +-
 doc/api/doxy-api.conf                    |    1 +
 doc/guides/prog_guide/img/member_i1.svg  | 1613 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg  |   36 +
 doc/guides/prog_guide/img/member_i3.svg  |  148 +++
 doc/guides/prog_guide/img/member_i4.svg  |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg  |  163 +++
 doc/guides/prog_guide/img/member_i6.svg  |  332 ++++++
 doc/guides/prog_guide/img/member_i7.svg  |  399 ++++++++
 doc/guides/prog_guide/index.rst          |   14 +
 doc/guides/prog_guide/member_lib.rst     |  421 ++++++++
 doc/guides/rel_notes/release_17_11.rst   |   17 +
 lib/Makefile                             |    2 +
 lib/librte_eal/common/eal_common_log.c   |    1 +
 lib/librte_eal/common/include/rte_log.h  |    1 +
 lib/librte_member/Makefile               |   51 +
 lib/librte_member/rte_member.c           |  342 +++++++
 lib/librte_member/rte_member.h           |  518 ++++++++++
 lib/librte_member/rte_member_ht.c        |  569 +++++++++++
 lib/librte_member/rte_member_ht.h        |  115 +++
 lib/librte_member/rte_member_vbf.c       |  350 +++++++
 lib/librte_member/rte_member_vbf.h       |   85 ++
 lib/librte_member/rte_member_version.map |   16 +
 lib/librte_member/rte_member_x86.h       |  111 ++
 mk/rte.app.mk                            |    2 +
 test/test/Makefile                       |    3 +
 test/test/test_member.c                  |  682 +++++++++++++
 test/test/test_member_perf.c             |  643 ++++++++++++
 30 files changed, 7099 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
 create mode 100644 lib/librte_member/rte_member_version.map
 create mode 100644 lib/librte_member/rte_member_x86.h
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
#29 lore raw

PATCH v3 1/7 member: implement main API

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  49 +++
 lib/librte_member/rte_member.c           | 342 ++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518 +++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  16 +
 7 files changed, 929 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
index 86caba1..c82033a 100644
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
 DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
 DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_eal/common/eal_common_log.c +1 -0
index 0e3b932..90db6a3 100644
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = {
 	{RTE_LOGTYPE_CRYPTODEV,  "cryptodev"},
 	{RTE_LOGTYPE_EFD,        "efd"},
 	{RTE_LOGTYPE_EVENTDEV,   "eventdev"},
+	{RTE_LOGTYPE_MEMBER,     "member"},
 	{RTE_LOGTYPE_USER1,      "user1"},
 	{RTE_LOGTYPE_USER2,      "user2"},
 	{RTE_LOGTYPE_USER3,      "user3"},
lib/librte_eal/common/include/rte_log.h +1 -0
index ec8dba7..ce1a3d0 100644
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;
 #define RTE_LOGTYPE_CRYPTODEV 17 /**< Log related to cryptodev. */
 #define RTE_LOGTYPE_EFD       18 /**< Log related to EFD. */
 #define RTE_LOGTYPE_EVENTDEV  19 /**< Log related to eventdev. */
+#define RTE_LOGTYPE_MEMBER    20 /**< Log related to membership. */
 
 /* these log types can be used in an application */
 #define RTE_LOGTYPE_USER1     24 /**< User-defined log type 1. */
lib/librte_member/Makefile new file +49 -0

49 lines not on this page · raw message

lib/librte_member/rte_member.c new file +342 -0

342 lines not on this page · raw message

lib/librte_member/rte_member.h new file +518 -0

518 lines not on this page · raw message

lib/librte_member/rte_member_version.map new file +16 -0
new file mode 100644
index 0000000..a5877c9
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+	global:
+
+	rte_member_create;
+	rte_member_find_existing;
+	rte_member_lookup;
+	rte_member_lookup_bulk;
+	rte_member_lookup_multi;
+	rte_member_lookup_multi_bulk;
+	rte_member_add;
+	rte_member_free;
+	rte_member_reset;
+	rte_member_delete;
+
+	local: *;
+};
Signature · 1 line
2.7.4
#30 lore raw

Re: [PATCH v3 1/7] member: implement main API

06/09/2017 01:59, Yipeng Wang:

lib/librte_eal/common/eal_common_log.c static const struct logtype logtype_strings[] = { · line 285

Quoted 10 lines
lib/librte_eal/common/eal_common_log.c +1 -0
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = {
        {RTE_LOGTYPE_CRYPTODEV,  "cryptodev"},
        {RTE_LOGTYPE_EFD,        "efd"},
        {RTE_LOGTYPE_EVENTDEV,   "eventdev"},
+       {RTE_LOGTYPE_MEMBER,     "member"},
        {RTE_LOGTYPE_USER1,      "user1"},
        {RTE_LOGTYPE_USER2,      "user2"},
        {RTE_LOGTYPE_USER3,      "user3"},

The static logtypes should be removed.
Please use the new dynamic log types.

#31 lore raw

Re: [PATCH v3 1/7] member: implement main API

In reply to Yipeng Wang
Quoted 12 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Yipeng Wang
Sent: Wednesday, September 6, 2017 1:00 AM
To: [email protected]
Cc: Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>; Wang, Yipeng1 <[email protected]>; Mcnamara, John <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/7] member: implement main API

Quoted 8 lines

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set- membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test is not 100% accurate. Certain false

Is -> are.

Quoted 22 lines

positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.

This patch add the main API definition.

---
 lib/Makefile                             |   2 +
 lib/librte_eal/common/eal_common_log.c   |   1 +
 lib/librte_eal/common/include/rte_log.h  |   1 +
 lib/librte_member/Makefile               |  49 +++
 lib/librte_member/rte_member.c           | 342 ++++++++++++++++++++
 lib/librte_member/rte_member.h           | 518
+++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  16 +
 7 files changed, 929 insertions(+)
 create mode 100644 lib/librte_member/Makefile  create mode 100644

lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map

...

Quoted 5 lines
(quoted diff) +0 -0

b/lib/librte_member/rte_member.c new file mode 100644 index 0000000..71b066d

lib/librte_member/rte_member.c new file +0 -0

...

Quoted 38 lines
lib/librte_member/rte_member.c new file +28 -0
+
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_eal_memconfig.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+
+#include "rte_member.h"
+#include "rte_member_ht.h"
+#include "rte_member_vbf.h"
+
+TAILQ_HEAD(rte_member_list, rte_tailq_entry); static struct
+rte_tailq_elem rte_member_tailq = {
+	.name = "RTE_MEMBER",
+};
+EAL_REGISTER_TAILQ(rte_member_tailq)
+
+
+void *
+rte_member_find_existing(const char *name) {
+	struct rte_member_setsum *setsum = NULL;
+	struct rte_tailq_entry *te;
+	struct rte_member_list *member_list;
+
+	member_list = RTE_TAILQ_CAST(rte_member_tailq.head,

rte_member_list);

lib/librte_member/rte_member.c new file +5 -0
+
+	rte_rwlock_read_lock(RTE_EAL_TAILQ_RWLOCK);
+	TAILQ_FOREACH(te, member_list, next) {
+		setsum = (struct rte_member_setsum *) te->data;
+		if (strncmp(name, setsum->name,

RTE_MEMBER_NAMESIZE) == 0)

lib/librte_member/rte_member.c new file +3 -0
+			break;
+	}
+	rte_rwlock_read_unlock(RTE_EAL_TAILQ_RWLOCK);

lib/librte_member/rte_member.c line 79

Quoted 10 lines
lib/librte_member/rte_member.c new file +10 -0
+
+	if (te == NULL) {
+		rte_errno = ENOENT;
+		return NULL;
+	}
+	return setsum;
+}
+
+void
+rte_member_free(void *ss)

Why not using directly "struct rte_member_setsum", so you can use it directly?
This applies to other functions that are using "void *".
I see that the content of this structure is internal, but you can still use a pointer to that structure.

...

Quoted 6 lines
lib/librte_member/rte_member.c new file +6 -0
+void *
+rte_member_create(const struct rte_member_parameters *params) {
+	struct rte_tailq_entry *te;
+	struct rte_member_list *member_list = NULL;
+	struct rte_member_setsum *setsum = NULL;
+	int ret;

lib/librte_member/rte_member.c line 132

Quoted 10 lines
lib/librte_member/rte_member.c new file +10 -0
+
+	if (params == NULL) {
+		rte_errno = EINVAL;
+		return NULL;
+	}
+
+	if ((params->key_len == 0)) {
+		rte_errno = EINVAL;
+		RTE_LOG(ERR, MEMBER,
+			"Memship create with invalid parameters\n");

rte_member_create has invalid parameters?

Quoted 6 lines
(quoted diff) +0 -0

b/lib/librte_member/rte_member.h new file mode 100644 index 0000000..de44b1b

lib/librte_member/rte_member.h new file +0 -0
@@ -0,0 +1,518 @@

...

lib/librte_member/rte_member.h line 129

Quoted 4 lines
lib/librte_member/rte_member.h new file +2 -0
+enum rte_member_setsum_type {
+	RTE_MEMBER_TYPE_HT = 0,  /**< Hash table based set summary.

*/

lib/librte_member/rte_member.h new file +1 -0
+	RTE_MEMBER_TYPE_VBF,     /**< vector of bloom filters. */

"vector" -> "Vector".

Quoted 6 lines
lib/librte_member/rte_member.h new file +6 -0
+	RTE_MEMBER_NUM_TYPE
+};
+
+/** @internal compare function for different arch. */ enum
+rte_member_sig_compare_function {
+	RTE_MEMBER_COMPARE_SCALAR = 0,

lib/librte_member/rte_member.h line 145

Quoted 10 lines
lib/librte_member/rte_member.h new file +10 -0
+	RTE_MEMBER_COMPARE_AVX2,
+	RTE_MEMBER_COMPARE_NUM
+};
+
+/** @internal setsummary structure. */
+struct rte_member_setsum {
+	enum rte_member_setsum_type type;
+	const char *name;
+	uint32_t key_len;
+	uint32_t socket_id;          /* NUMA Socket ID for memory. */

Either comment all the fields or do not do it (for this case, because the structure is internal, otherwise, all fields would require a comment). Also, remember that they should all start with capital letters.

Quoted 9 lines
lib/librte_member/rte_member.h new file +9 -0
+	uint32_t prim_hash_seed;
+	uint32_t sec_hash_seed;
+
+
+	/* hash table based */
+	uint32_t bucket_cnt;
+	uint32_t bucket_mask;
+	uint8_t cache;
+	enum rte_member_sig_compare_function sig_cmp_fn;

...

Quoted 1 line

+struct rte_member_parameters {

...

lib/librte_member/rte_member.h line 205

Quoted 1 line
lib/librte_member/rte_member.h new file +1 -0
+	uint8_t iscache;

As said in the documentation patch, change to "is_cache".

Quoted 6 lines
lib/librte_member/rte_member.h new file +3 -0
+
+	/**
+	 * For HT setsummary, num_keys equals to the number of entries

of the

lib/librte_member/rte_member.h new file +1 -0
+	 * table. When the number of keys that inserted to the HT

setsummary

"number of keys inserted in the HT summary".

Quoted 17 lines
lib/librte_member/rte_member.h new file +2 -0
+	 * approaches this number, eviction could happen. For cache mode,
+	 * keys could be evicted out of the table. For non-cache mode, keys

will

lib/librte_member/rte_member.h new file +2 -0
+	 * be evicted to other buckets like cuckoo hash. The table will also
+	 * likely to become full before the number of inserted keys equal to

the

lib/librte_member/rte_member.h new file +3 -0
+	 * total number of entries.
+	 *
+	 * For vBF, num_keys equal to the expected number of keys that

will

lib/librte_member/rte_member.h new file +1 -0
+	 * be inserted into the vBF. The implementation assumes the keys

are

lib/librte_member/rte_member.h new file +2 -0
+	 * evenly distributed to each BF in vBF. This is used to calculate the
+	 * number of bits we need for each BF. User does not specify the

size of

lib/librte_member/rte_member.h new file +1 -0
+	 * each BF directly because the optimal size depends on the

num_keys

lib/librte_member/rte_member.h line 225

Quoted 5 lines
lib/librte_member/rte_member.h new file +5 -0
+	 * and false positive rate.
+	 */
+	uint32_t num_keys;
+
+

Leave just one blank line between fields.

Quoted 38 lines
lib/librte_member/rte_member.h new file +3 -0
+	/**
+	 * The length of key is used for hash calculation. Since key is not
+	 * stored in set-summary, large key does not require more memory

space.

lib/librte_member/rte_member.h new file +9 -0
+	 */
+	uint32_t key_len;
+
+
+	/**
+	 * num_set is only relevant to vBF based setsummary.
+	 * num_set is equal to the number of BFs in vBF. For current
+	 * implementation, it only supports 1,2,4,8,16,32 BFs in one vBF set
+	 * summary. If other number of sets are needed, for example 5, the

user

lib/librte_member/rte_member.h new file +12 -0
+	 * should allocate the minimum available value that larger than 5,
+	 * which is 8.
+	 */
+	uint32_t num_set;
+
+	/**
+	 * false_postive_rate is only relevant to vBF based setsummary.
+	 * false_postivie_rate is the user-defined false positive rate
+	 * given expected number of inserted keys (num_keys). It is used to
+	 * calculate the total number of bits for each BF, and the number of
+	 * hash values used during lookup and insertion. For details please
+	 * refer to vBF implementation and membership library

documentation.

lib/librte_member/rte_member.h new file +3 -0
+	 *
+	 * HT setsummary's false positive rate is in the order of:
+	 * false_pos = (1/bucket_count)*(1/2^16), since we use 16-bit

signature.

lib/librte_member/rte_member.h new file +2 -0
+	 * This is because two keys needs to map to same bucket and same
+	 * signature to have a collision (false positive). bucket_count is

equal

lib/librte_member/rte_member.h new file +1 -0
+	 * to number of entries (num_keys) divided by entry count per

bucket

lib/librte_member/rte_member.h new file +1 -0
+	 * (RTE_MEMBER_BUCKET_ENTRIES). Thus, the false_postivie_rate

is not

lib/librte_member/rte_member.h

Quoted 2 lines
lib/librte_member/rte_member.h new file +2 -0
+	 * directly set by users.
+	 */

Typo in "positive" in several places in the comment above.
Also, clarify that "false_positive_rate" is not directly set by users for HT mode.

Quoted 1 line
lib/librte_member/rte_member.h new file +1 -0
+	float false_positive_rate;

...

Quoted 2 lines

+ * Lookup key in set-summary (SS).
+ * Single key lookup and return as soon as the first match found

Leave a blank line before parameters.

Quoted 4 lines

+ * @param setsum
+ * pointer of a setsummary
+ * @param key
+ * pointer of the key that needs to lookup

Better something like this "Pointer of the key to be looked up"

Quoted 6 lines

+ * @param set_id
+ * output the set id matches the key
+ * @return
+ * return 1 for found a match and 0 for not found a match
+ */
+

Definitions of the fields should start with capital letter.

lib/librte_member/rte_member.h line 338

Quoted 10 lines
lib/librte_member/rte_member.h new file +10 -0
+int
+rte_member_lookup(const void *setsum,
+		const void *key, MEMBER_SET_TYPE *set_id);
+
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * lookup bulk of keys in set-summary (SS).

"Lookup bulk"

Quoted 5 lines

+ * Each key lookup returns as soon as the first match found
+ * @param setsum
+ * Pointer of a setsummary
+ * @param keys
+ * Pointer of bulk of keys that to be lookup

"Pointer of bulk of keys to be looked up".
Check for this in other parts of the code.

Quoted 9 lines

+ * @param num_keys
+ * Number of keys that will be lookup
+ * @param set_ids

+ *   Output set ids for all the keys to this array.
+ *   User should preallocate array that can contain all results, which size is
+ *   the num_keys.

+ * @return
+ * The number of keys that found a match
+ */

lib/librte_member/rte_member.h line 362

Quoted 10 lines
lib/librte_member/rte_member.h new file +10 -0
+
+
+int
+rte_member_lookup_bulk(const void *setsum,
+		const void **keys, uint32_t num_keys,
+		MEMBER_SET_TYPE *set_ids);
+
+
+
+

Too many blank spaces. Generally leave one between functions. Check for this in the rest of the files.

Quoted 14 lines

+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a bulk of keys in set-summary (SS) for multiple matches each key.
+ * Each key lookup will find all matched entries (multiple match).
+ * Note that for cache mode HT, each key can have at most one match. So
+ * multi-match function is mainly used for vBF and non-cache mode HT.
+ * @param setsum
+ * pointer of a setsummary
+ * @param keys
+ * The keys that to be lookup
+ * @param num_keys

Quoted 8 lines

+ * The number of keys that will be lookup
+ * @param max_match_per_key
+ * The possible maximum number of matches for each key
+ * @param match_count
+ * Output the number of matches for each key in an array
+ * @param set_ids
+ * Return set ids for all the matches of all keys. User pass in a
preallocated

"User passes in"

Quoted 13 lines

+ * 2D array with first dimension as key index and second dimension as
match
+ * index. For example set_ids[bulk_size][max_match_per_key]
+ * @return
+ * The number of keys that found one or more matches in the set-
summary
+ */

lib/librte_member/rte_member.h new file +6 -0
+
+int
+rte_member_lookup_multi_bulk(const void *setsum,
+		const void **keys, uint32_t num_keys,
+		uint32_t max_match_per_key,
+		uint32_t *match_count,

lib/librte_member/rte_member.h line 435

Quoted 10 lines
lib/librte_member/rte_member.h new file +10 -0
+		MEMBER_SET_TYPE *set_ids);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Insert key into set-summary (SS).
+ *
+ * @param setsum
+ *   pointer of a set-summary

Start with capital letter.

Quoted 17 lines

+ * @param key
+ * the key that needs to be added
+ * @param set_id
+ * The set id associated with the key that needs to be added. Different
mode

+ *   supports different set_id ranges. 0 cannot be used as set_id since
+ *   RTE_MEMBER_NO_MATCH by default is set as 0.
+ *   For HT mode, the set_id has range as [1, 0x7FFF], MSB is reserved.
+ *   For vBF mode the set id is limited by the num_set parameter when

create
+ * the set-summary.
+ * @return
+ * HT (cache mode) and vBF should never fail unless the set_id is not in
the
+ * valid range. In such case -EINVAL is returned.
+ * For HT (non-cache mode) it could fail with -ENOSPC error code when
table is

Quoted 5 lines
+ *   full.
+ *   For success it returns different values for different modes to provide
+ *   extra information for users.
+ *   Return 0 for HT (cache mode) if the add does not cause
+ *   eviction, return 1 otherwise. Return 0 for HT mode if success, -ENOSPC

For HT non-cache mode?

Quoted 9 lines

for
+ * full, and 1 if cuckoo eviction happens. Always return 0 for vBF mode.
+ */

lib/librte_member/rte_member.h new file +6 -0
+
+int
+rte_member_add(const void *setsum, const void *key,
+		MEMBER_SET_TYPE set_id);
+
+

...

Quoted 10 lines
lib/librte_member/rte_member_version.map new file +3 -0
new file mode 100644
index 0000000..a5877c9
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+	global:
+

lib/librte_member/rte_member_version.map line 13

Quoted 10 lines
lib/librte_member/rte_member_version.map new file +10 -0
+	rte_member_create;
+	rte_member_find_existing;
+	rte_member_lookup;
+	rte_member_lookup_bulk;
+	rte_member_lookup_multi;
+	rte_member_lookup_multi_bulk;
+	rte_member_add;
+	rte_member_free;
+	rte_member_reset;
+	rte_member_delete;

This list must be in alphabetical order.

Quoted 5 lines
lib/librte_member/rte_member_version.map new file +3 -1
+
+	local: *;
+};
--

2.7.4

#32 lore raw

PATCH v3 2/7 member: implement HT mode

In reply to Yipeng Wang

One of the set-summray structure is hash-table based set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is more similar to traditional cuckoo filter. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. So the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.

This patch add the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

---
 lib/librte_member/Makefile        |   2 +-
 lib/librte_member/rte_member_ht.c | 490 ++++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_ht.h | 115 +++++++++
 3 files changed, 606 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
index 1a79eaa..ad26548 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_ht.c new file +490 -0

490 lines not on this page · raw message

lib/librte_member/rte_member_ht.h new file +115 -0

115 lines not on this page · raw message

Signature · 1 line
2.7.4
#33 lore raw

PATCH v3 3/7 member: implement vBF mode

In reply to Yipeng Wang

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set arbitrarily so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

---
 lib/librte_member/Makefile         |   2 +-
 lib/librte_member/rte_member_vbf.c | 350 +++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_vbf.h |  85 +++++++++
 3 files changed, 436 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
index ad26548..50275ed 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c rte_member_vbf.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_vbf.c new file +350 -0

350 lines not on this page · raw message

lib/librte_member/rte_member_vbf.h new file +85 -0

85 lines not on this page · raw message

Signature · 1 line
2.7.4
#34 lore raw

PATCH v3 4/7 member: add AVX for HT mode

In reply to Yipeng Wang

For key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

---
 lib/librte_member/rte_member_ht.c  | 143 ++++++++++++++++++++++++++++---------
 lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++
 2 files changed, 222 insertions(+), 32 deletions(-)
 create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -32
index b2ae6d0..15e2534 100644
@@ -40,6 +40,10 @@
 #include "rte_member.h"
 #include "rte_member_ht.h"
 
+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+
 
 static inline int
 insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -135,6 +139,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
 		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
 			buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
 	}
+#if defined(RTE_ARCH_X86)
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+			RTE_MEMBER_BUCKET_ENTRIES == 16)
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+	else
+#endif
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;
 
 
 	RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -174,11 +185,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
 	*set_id = RTE_MEMBER_NO_MATCH;
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	if (search_bucket_single(prim_bucket, tmp_sig, buckets,
-			set_id) ||
-			search_bucket_single(sec_bucket, tmp_sig,
-				buckets, set_id))
-		return 1;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single_avx(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+		break;
+#endif
+	default:
+		if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+	}
 
 	return 0;
 }
@@ -203,13 +226,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
 	}
 
 	for (i = 0; i < num_keys; i++) {
-		if (search_bucket_single(prim_buckets[i], tmp_sig[i],
-				buckets, &set_id[i]) ||
-				search_bucket_single(sec_buckets[i],
-				tmp_sig[i], buckets, &set_id[i]))
-			ret++;
-		else
-			set_id[i] = RTE_MEMBER_NO_MATCH;
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			if (search_bucket_single_avx(prim_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]) ||
+				search_bucket_single_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				ret++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+			break;
+#endif
+		default:
+			if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+					buckets, &set_id[i]) ||
+					search_bucket_single(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				ret++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+		}
 	}
 	return ret;
 }
@@ -227,12 +264,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
 
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
-			 match_per_key, set_id);
-	if (ret < match_per_key)
-		search_bucket_multi(sec_bucket, tmp_sig,
-			buckets, &ret, match_per_key, set_id);
-	return ret;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+			&ret, match_per_key, set_id);
+		if (ret < match_per_key)
+			search_bucket_multi_avx(sec_bucket, tmp_sig,
+				buckets, &ret, match_per_key, set_id);
+		return ret;
+#endif
+	default:
+		search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+				 match_per_key, set_id);
+		if (ret < match_per_key)
+			search_bucket_multi(sec_bucket, tmp_sig,
+				buckets, &ret, match_per_key, set_id);
+		return ret;
+	}
 }
 
 
@@ -259,16 +308,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
 	for (i = 0; i < num_keys; i++) {
 		match_cnt_t = 0;
 
-		search_bucket_multi(prim_buckets[i], tmp_sig[i],
-			buckets, &match_cnt_t, match_per_key,
-			&set_ids[i*match_per_key]);
-		if (match_cnt_t < match_per_key)
-			search_bucket_multi(sec_buckets[i], tmp_sig[i],
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
 				buckets, &match_cnt_t, match_per_key,
 				&set_ids[i*match_per_key]);
-		match_count[i] = match_cnt_t;
-		if (match_cnt_t != 0)
-			ret++;
+			if (match_cnt_t < match_per_key)
+				search_bucket_multi_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &match_cnt_t,
+					match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_t;
+			if (match_cnt_t != 0)
+				ret++;
+			break;
+#endif
+		default:
+			search_bucket_multi(prim_buckets[i], tmp_sig[i],
+				buckets, &match_cnt_t, match_per_key,
+				&set_ids[i*match_per_key]);
+			if (match_cnt_t < match_per_key)
+				search_bucket_multi(sec_buckets[i], tmp_sig[i],
+					buckets, &match_cnt_t, match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_t;
+			if (match_cnt_t != 0)
+				ret++;
+		}
 	}
 	return ret;
 }
@@ -300,12 +367,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
 
 static inline int
 try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
-		SIG_TYPE sig, MEMBER_SET_TYPE set_id)
+		SIG_TYPE sig, MEMBER_SET_TYPE set_id,
+		enum rte_member_sig_compare_function cmp_fn)
 {
-	if (insert_overwrite_search(prim, sig, buckets, set_id) ||
-			insert_overwrite_search(sec, sig, buckets,
-				set_id))
-		return 0;
+	switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (insert_overwrite_search_avx(prim, sig, buckets, set_id) ||
+				insert_overwrite_search_avx(sec, sig, buckets,
+					set_id))
+			return 0;
+		break;
+#endif
+	default:
+		if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+				insert_overwrite_search(sec, sig, buckets,
+					set_id))
+			return 0;
+	}
 	return -1;
 }
 
@@ -411,7 +490,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
 	/* if it is cache based filter, we try overwriting existing entry */
 	if (ss->cache) {
 		ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
-					set_id);
+					set_id, ss->sig_cmp_fn);
 		if (ret != -1)
 			return ret;
 	}
lib/librte_member/rte_member_x86.h new file +111 -0

111 lines not on this page · raw message

Signature · 1 line
2.7.4
#35 lore raw

PATCH v3 5/7 member: enable the library

In reply to Yipeng Wang

This patch enables the Membership library.

---
 MAINTAINERS                | 7 +++++++
 config/common_base         | 5 +++++
 lib/librte_member/Makefile | 2 ++
 mk/rte.app.mk              | 2 ++
 4 files changed, 16 insertions(+)
MAINTAINERS +7 -0
index a0cd75e..e372edf 100644
@@ -734,6 +734,13 @@ F: test/test/test_meter.c
 F: examples/qos_meter/
 F: doc/guides/sample_app_ug/qos_metering.rst
 
+Membership - EXPERIMENTAL
+M: Yipeng Wang <[email protected]>
+M: Sameh Gobriel <[email protected]>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
+
 
 Other libraries
 ---------------
config/common_base +5 -0
index 5e97a08..5e31ced 100644
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
 CONFIG_RTE_LIBRTE_EFD=y
 
 #
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
 # Compile librte_jobstats
 #
 CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
index 50275ed..3bac1d0 100644
@@ -37,6 +37,8 @@ LIB = librte_member.a
 CFLAGS := -I$(SRCDIR) $(CFLAGS)
 CFLAGS += $(WERROR_FLAGS) -O3
 
+LDLIBS += -lm
+
 EXPORT_MAP := rte_member_version.map
 
 LIBABIVER := 1
mk/rte.app.mk +2 -0
index c25fdd9..c79acf0 100644
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE)        += -lrte_cfgfile
 _LDLIBS-y += --whole-archive
 
 _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH)           += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)        += -lrte_member
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lrte_vhost
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS)         += -lrte_kvargs
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lm
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrt
 _LDLIBS-$(CONFIG_RTE_LIBRTE_METER)          += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)          += -lm
 ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lnuma
 endif
Signature · 1 line
2.7.4
#36 lore raw

Re: [PATCH v3 5/7] member: enable the library

06/09/2017 01:59, Yipeng Wang:

MAINTAINERS F: test/test/test_meter.c · line 742

Quoted 12 lines
MAINTAINERS +6 -0
@@ -734,6 +734,13 @@ F: test/test/test_meter.c
 F: examples/qos_meter/
 F: doc/guides/sample_app_ug/qos_metering.rst
 
+Membership - EXPERIMENTAL
+M: Yipeng Wang <[email protected]>
+M: Sameh Gobriel <[email protected]>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*

I know it is not obvious but this section is sorted in alphabetical order. Please move between Hashes and LPM. Thanks

#37 lore raw

PATCH v3 6/7 test/member: add functional and perf tests

In reply to Yipeng Wang

This patch adds functional and performance tests for membership library.

---
 test/test/Makefile           |   3 +
 test/test/test_member.c      | 682 +++++++++++++++++++++++++++++++++++++++++++
 test/test/test_member_perf.c | 643 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1328 insertions(+)
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
test/test/Makefile +3 -0
index 42d9a49..b61dde3 100644
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
 SRCS-y += test_memcpy.c
 SRCS-y += test_memcpy_perf.c
 
+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
 
test/test/test_member.c new file +682 -0

682 lines not on this page · raw message

test/test/test_member_perf.c new file +643 -0

643 lines not on this page · raw message

Signature · 1 line
2.7.4
#38 lore raw

PATCH v3 7/7 doc: add membership documentation

In reply to Yipeng Wang

This patch adds the documentation for membership library.

---
 doc/api/doxy-api-index.md               |    3 +-
 doc/api/doxy-api.conf                   |    1 +
 doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg |   36 +
 doc/guides/prog_guide/img/member_i3.svg |  148 +++
 doc/guides/prog_guide/img/member_i4.svg |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg |  163 ++++
 doc/guides/prog_guide/img/member_i6.svg |  332 +++++++
 doc/guides/prog_guide/img/member_i7.svg |  399 ++++++++
 doc/guides/prog_guide/index.rst         |   14 +
 doc/guides/prog_guide/member_lib.rst    |  421 ++++++++
 doc/guides/rel_notes/release_17_11.rst  |   17 +
 12 files changed, 3596 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
doc/api/doxy-api-index.md +2 -1
index 19e0d4f..fe87e09 100644
@@ -105,7 +105,8 @@ The public API headers are grouped by topics:
   [LPM IPv4 route]     (@ref rte_lpm.h),
   [LPM IPv6 route]     (@ref rte_lpm6.h),
   [ACL]                (@ref rte_acl.h),
-  [EFD]                (@ref rte_efd.h)
+  [EFD]                (@ref rte_efd.h),
+  [member]             (@ref rte_member.h)
 
 - **QoS**:
   [metering]           (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
index 823554f..b792d6d 100644
@@ -58,6 +58,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_mempool \
                           lib/librte_meter \
                           lib/librte_metrics \
+                          lib/librte_member \
                           lib/librte_net \
                           lib/librte_pdump \
                           lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0

1,613 lines not on this page · raw message

doc/guides/prog_guide/img/member_i2.svg new file +36 -0
new file mode 100644
index 0000000..759c654
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in"
+		viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+	<v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {fill:none;stroke:none;stroke-width:0.25}
+		.st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+		.st3 {baseline-shift:32.4943%;font-size:0.649886em}
+		.st4 {font-size:1em}
+		.st5 {font-family:Cambria Math;font-size:1em}
+		.st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+		<g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)">
+			<title>Sheet.3</title>
+			<desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc>
+			<v:textBlock v:margins="rect(4,4,4,4)"/>
+			<v:textRect cx="157.5" cy="45.5" width="315" height="90"/>
+			<rect x="0" y="0.5" width="315" height="90" class="st1"/>
+			<text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan
+						class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3"
+						v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em"
+						class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text>		</g>
+	</g>
+</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0

148 lines not on this page · raw message

doc/guides/prog_guide/img/member_i4.svg new file +450 -0

450 lines not on this page · raw message

doc/guides/prog_guide/img/member_i5.svg new file +163 -0

163 lines not on this page · raw message

doc/guides/prog_guide/img/member_i6.svg new file +332 -0

332 lines not on this page · raw message

doc/guides/prog_guide/img/member_i7.svg new file +399 -0

242 lines not on this page · raw message

doc/guides/prog_guide/index.rst +14 -0
index 40f04a1..7ff5144 100644

Lines of this file are not shown. View the raw message.

doc/guides/prog_guide/member_lib.rst new file +421 -0

421 lines not on this page · raw message

doc/guides/rel_notes/release_17_11.rst +17 -0
index 170f4f9..4002df3 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.7.4

Part of the diff is not shown here. View the raw message.

Acked-by on v3 7/7
#39 lore raw

Re: [PATCH v3 7/7] doc: add membership documentation

Quoted 23 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 6, 2017 1:00 AM
To: [email protected]
Cc: Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>; Wang, Yipeng1 <[email protected]>; Mcnamara, John <[email protected]>
Subject: [PATCH v3 7/7] doc: add membership documentation

This patch adds the documentation for membership library.

doc/guides/prog_guide/member_lib.rst new file +4 -0
+Set-summary Element Insertion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``rte_member_add()`` function is use to insert an element/key into a
Quoted 2 lines

set-summary structure. If it fails an
+error is returned. For success the returned value is dpendednt on the

Minor typo: s/dpendednt/dependent/

Apart from that:

#40 lore raw

Re: [PATCH v3 7/7] doc: add membership documentation

In reply to Yipeng Wang
Quoted 16 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Yipeng Wang
Sent: Wednesday, September 6, 2017 1:00 AM
To: [email protected]
Cc: Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>; Wang, Yipeng1 <[email protected]>; Mcnamara, John <[email protected]>
Subject: [dpdk-dev] [PATCH v3 7/7] doc: add membership documentation

This patch adds the documentation for membership library.

...

Quoted 15 lines
doc/guides/prog_guide/member_lib.rst new file +7 -0
new file mode 100644
index 0000000..2b32535
 
 
+Membership Library
+==================
+
+Introduction
+------------
+
+The DPDK Membership Library provides an API for DPDK applications to
Quoted 4 lines

insert a
+new member, delete an existing member, or query the existence of a member in a
+given set, or a group of sets. For the case of a group of sets the library

"group of sets, the library".

...

Quoted 1 line

+We are including a configurable Membership Library in DPDK to cover set

This looks like more like a cover letter than part of programmer's guide. Reword this loke "This Membership Library covers set membership...", and avoid "include... in DPDK", as it is implicit.

Quoted 23 lines

+membership functionality for both a single set and multi-set scenarios. The
+library is optimized to support the customer network applications which require
+membership checking functionality. In this guide, we will cover two set- summary

doc/guides/prog_guide/member_lib.rst new file +2 -0
+schemes including a vector of Bloom Filters and Hash-Table based
+set-summary schemes with and without false negative probability,

followed by

doc/guides/prog_guide/member_lib.rst new file +7 -0
+a brief discussion of the Membership Library API.
+
+Vector of Bloom Filters
+-----------------------
+
+Bloom Filter (BF) [Member-bloom] is a well-known space-efficient
+probabilistic data structure that answers set membership queries (test

whether
+an element is a member of a set) with some probability of false positives and

doc/guides/prog_guide/member_lib.rst new file +4 -0
+zero false negatives; a query for an element returns either it is "possibly in
+a set" (with very high probability) or "definitely not in a set".
+
+The BF is a method for representing a set of ``n`` elements (for example
Quoted 8 lines

flow keys
+in network applications domain) to support membership queries. The idea of BF is
+to allocate a bit-vector ``v`` with ``m`` bits, which are initially all set to 0. Then
+it chooses ``k`` independent hash functions ``h1``, ``h2``, ... ``hk`` with hash values range from
+``1`` to ``m`` to perform hashing calculations on each element.

From "0" to "m-1"?

...

Quoted 21 lines
doc/guides/prog_guide/member_lib.rst new file +8 -0
+
+  Vector Bloom Filter (vBF) Overview
+
+To support membership test for both multiple sets and a single set,
+the library implements a Vector Bloom Filter (vBF) scheme.
+vBF basically composes multiple bloom filters into a vector of bloom filers.
+The membership test is conducted on all of the
+bloom filters concurrently to determine which set(s) it belongs to or none

of
+them. The basic idea of vBF is shown in the above figure where an element is

doc/guides/prog_guide/member_lib.rst new file +9 -0
+used to address multiple bloom filters concurrently and the bloom filter
+index(es) with a hit is returned.
+
+.. _figure_membership5:
+.. figure:: img/member_i5.*
+
+  vBF for Flow Scheduling to Worker Thread
+
+As previously mentioned, there are many usages of such structures. vBF is

used

doc/guides/prog_guide/member_lib.rst

Quoted 2 lines
doc/guides/prog_guide/member_lib.rst new file +2 -0
+for applications that needs to check membership against multiple sets
+simultaneously. 

"needs" -> "need".

...

doc/guides/prog_guide/member_lib.rst line 267

Quoted 6 lines
doc/guides/prog_guide/member_lib.rst new file +6 -0
+The major usage of HTSS with false negative is to use it as a cache for
+distributing elements to different target sets. By allowing HTSS to evict old
+elements, the set-summary can keep track of the most recent elements
+(i.e. active) as a cache typically does. Old inactive elements (infrequently
+used elements) will automatically and eventually get evicted from the
+set-summary. It worth noting that the set-summary still has false positive

"It is worth".

...

Quoted 6 lines

+We also pass two seeds: ``prim_hash_seed`` and +``sec_hash_seed`` for the primary and secondary hash functions to calculate two independent hash values.
+``socket_id`` parameter is the NUMA socket ID for the memory used to create the
+set-summary. For HTSS, another parameter ``iscache`` is used to indicate

In order to keep consistency, I would use "is_cache".

Quoted 30 lines
doc/guides/prog_guide/member_lib.rst new file +2 -0
+if this set-summary is a cache (i.e. with false negative probability) or not.
+For vBF, extra parameters are needed. For example, ``num_set`` is the

number of
+sets needed to initialize the vector bloom filters. This number is equal to the
+number of bloom filters will be created.
+``false_pos_rate`` is the false positive rate. num_keys and false_pos_rate will be used to determine

doc/guides/prog_guide/member_lib.rst new file +7 -0
+the number of hash functions and the bloom filter size.
+
+
+Set-summary Element Insertion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``rte_member_add()`` function is use to insert an element/key into a

set-summary structure. If it fails an

doc/guides/prog_guide/member_lib.rst new file +3 -0
+error is returned. For success the returned value is dpendednt on the
+set-summary mode to provide extra information for the users. For vBF
+mode, a return value of 0 means a successful insert. For HTSS mode

without false negative, the insert
+could fail with ``-ENOSPC`` if the table is full. With false negative (i.e. cache mode),
+for insert that does not cause any eviction (i.e. no overwriting happens to an
+existing entry) the return value is 0. For insertion that causes eviction, the return

doc/guides/prog_guide/member_lib.rst new file +3 -0
+value is 1 to indicate such situation, but it is not an error.
+
+The input arguments for the function should include the ``key`` which is a

pointer to the element/key that needs to

doc/guides/prog_guide/member_lib.rst line 350

Quoted 6 lines
doc/guides/prog_guide/member_lib.rst new file +6 -0
+be added to the set-summary, and ``set_id`` which is the set id associated
+with the key that needs to be added.
+
+
+Set-summary Element Lookup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Remove unneeded last "~" (I see 4 extra at the end).

Quoted 29 lines
doc/guides/prog_guide/member_lib.rst new file +2 -0
+
+The ``rte_member_lookup()`` function looks up a single key/element in

the set-summary structure. It

doc/guides/prog_guide/member_lib.rst new file +2 -0
+returns as soon as the first match is found. The return value is 1 if a
+match is found and 0 otherwise. The arguments for the function include

``key`` which is a pointer to the
+element/key that needs to be looked up, and ``set_id`` which is used to return the

doc/guides/prog_guide/member_lib.rst new file +3 -0
+first target set id where the key has matched, if any.
+
+The ``rte_member_lookup_bulk()`` function is used to look up a bulk of

keys/elements in the
+set-summary structure for their first match. Each key lookup returns as soon as the first match is found. The

doc/guides/prog_guide/member_lib.rst new file +2 -0
+return value is the number of keys that find a match. The arguments of the
+function include ``keys`` which is a pointer to a bulk of keys that are to be

looked up,
+``num_keys`` is the number

doc/guides/prog_guide/member_lib.rst new file +2 -0
+of keys that will be looked up, and ``set_ids`` are the return target set
+ids for the first match found for each of the input keys. ``set_ids`` is an

array
+needs to be sized according to the ``num_keys``. If there is no match, the set id

doc/guides/prog_guide/member_lib.rst new file +3 -0
+for that key will be set to RTE_MEMBER_NO_MATCH.
+
+The ``rte_member_lookup_multi()`` function looks up a single

key/element in the

doc/guides/prog_guide/member_lib.rst new file +2 -0
+set-summary structure for multiple matches. It
+returns ALL the matches (possibly more than one) found for this key when
Quoted 3 lines

it
+is matched against all target sets (it worth noting that for cache mode HTSS,

"it is worth"

Quoted 7 lines

+the current implementation matches at most one target set). The return value is

doc/guides/prog_guide/member_lib.rst new file +3 -0
+the number of matches
+that was found for this key (for cache mode HTSS the return value
+should be at most 1). The arguments for the function include ``key`` which

is a pointer to the
+element/key that needs to be looked up, ``match_per_key`` which is to

Better to use "max_match_per_key". Will comment more in other patches.

Quoted 3 lines

indicate maximum number of matches
+the user expect to find for each key, and ``set_id`` which is used to return all

"user expects to".

Quoted 4 lines

+target set ids where the key has matched, if any. The ``set_id`` array should be sized
+according to ``match_per_key``. For vBF, maximum number of matches per key is equal

"For vBF, the maximum number".

Quoted 3 lines

+to the number of sets. For HTSS, maximum number of matches per key is equal to two times
+entry count per bucket. ``match_per_key`` should be equal or smaller than

"two time the entry count".

Quoted 15 lines

maximum number of

doc/guides/prog_guide/member_lib.rst new file +3 -0
+possible matches.
+
+The ``rte_membership_lookup_multi_bulk()`` function looks up a bulk of

keys/elements elements in the
+set-summary structure for multiple matches, each key lookup returns ALL the matches (possibly more
+than one) found for this key when it is matched against all target sets (cache mode HTSS

doc/guides/prog_guide/member_lib.rst new file +5 -0
+matches at most one target set). The
+return value is the number of keys that find one or more matches in the
+set-summary structure. The arguments of the
+function include ``keys`` which is
+a pointer to a bulk of keys that are to be looked up, ``num_keys`` is the

number

doc/guides/prog_guide/member_lib.rst

Quoted 2 lines
doc/guides/prog_guide/member_lib.rst new file +2 -0
+of keys that will be looked up, ``match_per_key`` is the possible
+max number of matches for each key, ``match_count`` which is the

Use "maximum number", no need to abbreviate here.

Quoted 5 lines

returned number

doc/guides/prog_guide/member_lib.rst new file +3 -0
+of matches for each key, and ``set_ids`` are the returned target set
+ids for all matches found for each keys. ``set_ids`` is 2-D array
+that for each key, a 1-D array should be sized according to

``match_per_key``.

I would reword it a bit, to improve readability. For instance:
"set_ids" is a 2-D array, containing a 1-D array per key, which should be sized according to "match_per_key".

Quoted 8 lines

+``match_per_key`` should be equal or smaller than maximum number of

doc/guides/prog_guide/member_lib.rst new file +7 -0
+possible matches, similar to ``rte_member_lookup_multi``.
+
+
+Set-summary Element Delete
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``rte_membership_delete()`` function deletes an element/key from a
Quoted 7 lines

set-summary structure, if it fails
+an error is returned. The input arguments should include ``key`` which is a pointer to the
+element/key that needs to be deleted from the set-summary, and ``set_id``
+which is the set id associated with the key to delete. It worth noting that current

"It is worth".

Quoted 2 lines

+implementation of vBF does not support deletion [1]_. An error code ``- EINVAL`` will be returned.

#41 lore raw

v4 cover Add Membership Library

In reply to Yipeng Wang

DPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.

A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

v4:
- member lib: change two hash function macros to be one. crc or jhash will be chosen depending on the architecture.
- member lib: use dynamic log type instead of static one by Thomas' comment.
- member lib: code cleanups (remove unnecessary code, change variable name, coding style, etc).
- member lib: reorder members in setsummary struct to be more cache friendly.
- member lib: setsummary struct member "name" should be char array rather than char pointer. Fixed.
- member lib: add check for two hash seeds to make sure they are not equal.
- member lib: vBF lookup speed optimization.
- member lib: revise comments in various places as Pablo suggested.
- member lib: change "void *" to "struct rte_member_setsum *" in public API as Pablo suggested. test case is modified accordingly to directly use the struct.
- member lib: change order in rte_member_version.map as Pablo suggested.
- MAINTAINERS: change the maintainer block order according to Thomas' comment.
- test: add printf to show which section expects error messages. Change set count to 16 from 32 for performance test.
- documentation: revise several places according to Pablo's and John's comments.

v3:
- documentation: update documentation to incorporate John's review.

v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitialized issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment.
- documentation: update the programmer guide for membership library.

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

 MAINTAINERS                              |    8 +-
 config/common_base                       |    5 +
 doc/api/doxy-api-index.md                |    3 +-
 doc/api/doxy-api.conf                    |    1 +
 doc/guides/prog_guide/img/member_i1.svg  | 1613 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg  |   36 +
 doc/guides/prog_guide/img/member_i3.svg  |  148 +++
 doc/guides/prog_guide/img/member_i4.svg  |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg  |  163 +++
 doc/guides/prog_guide/img/member_i6.svg  |  332 ++++++
 doc/guides/prog_guide/img/member_i7.svg  |  399 ++++++++
 doc/guides/prog_guide/index.rst          |   14 +
 doc/guides/prog_guide/member_lib.rst     |  420 ++++++++
 doc/guides/rel_notes/release_17_11.rst   |   17 +
 lib/Makefile                             |    2 +
 lib/librte_member/Makefile               |   51 +
 lib/librte_member/rte_member.c           |  336 +++++++
 lib/librte_member/rte_member.h           |  507 ++++++++++
 lib/librte_member/rte_member_ht.c        |  553 ++++++++++
 lib/librte_member/rte_member_ht.h        |   94 ++
 lib/librte_member/rte_member_vbf.c       |  349 +++++++
 lib/librte_member/rte_member_vbf.h       |   79 ++
 lib/librte_member/rte_member_version.map |   16 +
 lib/librte_member/rte_member_x86.h       |  107 ++
 mk/rte.app.mk                            |    2 +
 test/test/Makefile                       |    3 +
 test/test/test_member.c                  |  676 +++++++++++++
 test/test/test_member_perf.c             |  630 ++++++++++++
 28 files changed, 7012 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
 create mode 100644 lib/librte_member/rte_member_version.map
 create mode 100644 lib/librte_member/rte_member_x86.h
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
#42 lore raw

PATCH v4 1/7 member: implement main API

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.

This patch adds the main API definition.

---
 lib/Makefile                             |   2 +
 lib/librte_member/Makefile               |  49 +++
 lib/librte_member/rte_member.c           | 336 ++++++++++++++++++++
 lib/librte_member/rte_member.h           | 507 +++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  16 +
 5 files changed, 910 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
index 86caba1..c82033a 100644
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
 DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
 DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_member/Makefile new file +49 -0

49 lines not on this page · raw message

lib/librte_member/rte_member.c new file +336 -0

336 lines not on this page · raw message

lib/librte_member/rte_member.h new file +507 -0

507 lines not on this page · raw message

lib/librte_member/rte_member_version.map new file +16 -0
new file mode 100644
index 0000000..019e4cd
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+	global:
+
+	rte_member_add;
+	rte_member_create;
+	rte_member_delete;
+	rte_member_find_existing;
+	rte_member_free;
+	rte_member_lookup;
+	rte_member_lookup_bulk;
+	rte_member_lookup_multi;
+	rte_member_lookup_multi_bulk;
+	rte_member_reset;
+
+	local: *;
+};
Signature · 1 line
2.7.4
#43 lore raw

Re: [PATCH v4 1/7] member: implement main API

Hi Yipeng,

Quoted 17 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:40 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 1/7] member: implement main API

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set- membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

Quoted 8 lines

The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.

This patch adds the main API definition.

A few comments on changes that you didn't make in the v4.

Thanks,
Pablo

...

Quoted 18 lines
lib/librte_member/rte_member.c new file +15 -0
+
+struct rte_member_setsum *
+rte_member_create(const struct rte_member_parameters *params) {
+	struct rte_tailq_entry *te;
+	struct rte_member_list *member_list;
+	struct rte_member_setsum *setsum;
+	int ret;
+
+	if (params == NULL) {
+		rte_errno = EINVAL;
+		return NULL;
+	}
+
+	if (params->key_len == 0 ||
+			params->prim_hash_seed == params-
Quoted 1 line

sec_hash_seed) {

(quoted diff) +2 -0
+		rte_errno = EINVAL;
+		RTE_MEMBER_LOG(ERR, "Memship create with invalid
Quoted 1 line

parameters\n");

Do not use "Memship". Change to " rte_member_create has invalid parameters"? Or something else that you want, but not Memship.

Quoted 3 lines
lib/librte_member/rte_member.c new file +3 -0
+		return NULL;
+	}
+

...

Quoted 13 lines
lib/librte_member/rte_member.h new file +8 -0
+struct rte_member_parameters {
+	const char *name;			/**< Name of the hash. */
+
+	/**
+	 * User to specify the type of the setsummary from one of
+	 * rte_member_setsum_type.
+	 *
+	 * HT based setsummary is implemented like a hash table. User

should use

lib/librte_member/rte_member.h new file +3 -0
+	 * this type when there are many sets.
+	 *
+	 * vBF setsummary is a vector of bloom filters. It is used when

number

lib/librte_member/rte_member.h line 198

Quoted 8 lines
lib/librte_member/rte_member.h new file +8 -0
+	 * of sets is not big (less than 32 for current implementation).
+	 */
+	enum rte_member_setsum_type type;
+
+	/**
+	 * If it is HT based setsummary, user to specify the subtype or mode
+	 * of the setsummary. It could be cache, or non-cache mode.
+	 * Set iscache to be 1 if to use as cache mode.

Change to "is_cache".

Quoted 23 lines
(quoted diff) +2 -0
+	 *
+	 * For cache mode, keys can be evicted out of the HT setsummary.

Keys

lib/librte_member/rte_member.h new file +7 -0
+	 * with the same signature and map to the same bucket
+	 * will overwrite each other in the setsummary table.
+	 * This mode is useful for the case that the set-summary only
+	 * needs to keep record of the recently inserted keys. Both
+	 * false-negative and false-positive could happen.
+	 *
+	 * For non-cache mode, keys cannot be evicted out of the cache. So

for

(quoted diff) +1 -0
+	 * this mode the setsummary will become full eventually. Keys with

the

(quoted diff) +1 -0
+	 * same signature but map to the same bucket will still occupy

multiple

lib/librte_member/rte_member.h new file +6 -0
+	 * entries. This mode does not give false-negative result.
+	 */
+	uint8_t is_cache;
+
+	/**
+	 * For HT setsummary, num_keys equals to the number of entries

of the

(quoted diff) +1 -0
+	 * table. When the number of keys that inserted in the HT
Quoted 1 line

setsummary

"number of keys inserted in the HT summary". Or "were inserted".

Quoted 3 lines
(quoted diff) +2 -0
+	 * approaches this number, eviction could happen. For cache mode,
+	 * keys could be evicted out of the table. For non-cache mode, keys

will

...

Quoted 20 lines
lib/librte_member/rte_member.h new file +7 -0
+	/**
+	 * false_positive_rate is only relevant to vBF based setsummary.
+	 * false_positive_rate is the user-defined false positive rate
+	 * given expected number of inserted keys (num_keys). It is used to
+	 * calculate the total number of bits for each BF, and the number of
+	 * hash values used during lookup and insertion. For details please
+	 * refer to vBF implementation and membership library

documentation.

lib/librte_member/rte_member.h new file +4 -0
+	 * Note that this parameter is not directly set by users for HT mode.
+	 *
+	 * HT setsummary's false positive rate is in the order of:
+	 * false_pos = (1/bucket_count)*(1/2^16), since we use 16-bit

signature.

(quoted diff) +2 -0
+	 * This is because two keys needs to map to same bucket and same
+	 * signature to have a collision (false positive). bucket_count is

equal

(quoted diff) +1 -0
+	 * to number of entries (num_keys) divided by entry count per

bucket

(quoted diff) +1 -0
+	 * (RTE_MEMBER_BUCKET_ENTRIES). Thus, the false_positive_rate

is not

lib/librte_member/rte_member.h line 263

Quoted 1 line
lib/librte_member/rte_member.h new file +1 -0
+	 * directly set by users.

Unless I understood it incorrectly, I think you should clarify that this field is not set for HT mode, but it is for vBF, right?

Quoted 75 lines
lib/librte_member/rte_member.h new file +5 -0
+	 */
+	float false_positive_rate;
+
+	/**
+	 * We use two seeds to calculate two independent hashes for each

key.

lib/librte_member/rte_member.h new file +4 -0
+	 *
+	 * For HT type, one hash is used as signature, and the other is used
+	 * for bucket location.
+	 * For vBF type, these two hashes and their combinations are used

as

lib/librte_member/rte_member.h new file +6 -0
+	 * hash locations to index the bit array.
+	 */
+	uint32_t prim_hash_seed;
+
+	/**
+	 * The secondary seed should be a different value from the primary

seed.

lib/librte_member/rte_member.h new file +4 -0
+	 */
+	uint32_t sec_hash_seed;
+
+	int socket_id;			/**< NUMA Socket ID for memory.

*/

lib/librte_member/rte_member.h new file +51 -0
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Find an existing set-summary and return a pointer to it.
+ *
+ * @param name
+ *   Name of the set-summary.
+ * @return
+ *   Pointer to the set-summary or NULL if object not found
+ *   with rte_errno set appropriately. Possible rte_errno values include:
+ *    - ENOENT - value not available for return
+ */
+struct rte_member_setsum *
+rte_member_find_existing(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create set-summary (SS).
+ *
+ * @param params
+ *   Parameters to initialize the setsummary.
+ * @return
+ *   Return the pointer to the setsummary.
+ *   Return value is NULL if the creation failed.
+ */
+struct rte_member_setsum *
+rte_member_create(const struct rte_member_parameters *params);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup key in set-summary (SS).
+ * Single key lookup and return as soon as the first match found
+ *
+ * @param setsum
+ *   Pointer of a setsummary.
+ * @param key
+ *   Pointer of the key to be looked up.
+ * @param set_id
+ *   Output the set id matches the key.
+ * @return
+ *   Return 1 for found a match and 0 for not found a match.
+ */
+int
+rte_member_lookup(const struct rte_member_setsum *setsum, const

void *key,

lib/librte_member/rte_member.h line 340

Quoted 7 lines
lib/librte_member/rte_member.h new file +7 -0
+			member_set_t *set_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * lookup bulk of keys in set-summary (SS).

"Lookup bulk"

Quoted 13 lines

+ * Each key lookup returns as soon as the first match found
+ *
+ * @param setsum
+ * Pointer of a setsummary.
+ * @param keys
+ * Pointer of the bulk of keys to be looked up.
+ * @param num_keys
+ * Number of keys that will be lookup.
+ * @param set_ids

+ *   Output set ids for all the keys to this array.
+ *   User should preallocate array that can contain all results, which size is
+ *   the num_keys.

+ * @return

#44 lore raw

PATCH v4 2/7 member: implement HT mode

In reply to Yipeng Wang

One of the set-summary structures is hash-table based set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is more similar to traditional cuckoo filter. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. Thus, the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.

This patch adds the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

---
 lib/librte_member/Makefile        |   2 +-
 lib/librte_member/rte_member_ht.c | 474 ++++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_ht.h |  94 ++++++++
 3 files changed, 569 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
index 1a79eaa..ad26548 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_ht.c new file +474 -0

474 lines not on this page · raw message

lib/librte_member/rte_member_ht.h new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
2.7.4
#45 lore raw

Re: [PATCH v4 2/7] member: implement HT mode

Quoted 12 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:40 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 2/7] member: implement HT mode

...

Quoted 20 lines
lib/librte_member/Makefile index renamed from lib/librte_member/Makefile +0 -0

1a79eaa..ad26548 100644

lib/librte_member/Makefile +0 -0
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map

LIBABIVER := 1

 # all source are stored in SRCS-y
lib/librte_member/Makefile +1 -1
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c

rte_member_ht.c

 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_ht.c new file +0 -0
new file mode 100644
index 0000000..55672a4

...

lib/librte_member/rte_member_ht.c line 46

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+
+static inline int
+insert_overwrite_search(uint32_t bucket, member_sig_t tmp_sig,
+		struct member_ht_bucket *buckets,
+		member_set_t set_id)

I would call "bucket", "bucket_id", for better understanding. This comment also applies to other parts of the code (e.g. prim_buckets).

lib/librte_member/rte_member_ht.c line 53

Quoted 7 lines
lib/librte_member/rte_member_ht.c new file +7 -0
+{
+	int i;
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		if (buckets[bucket].sigs[i] == tmp_sig) {
+			buckets[bucket].sets[i] = set_id;
+			return 1;
+		}

Is this function used to update an existing entry?
At first, I thought that this was evicting another entry, when the bucket was full, but it looks like it is updating the set_id of an existing entry.
If this is the case, I would change the function name and add a comment explaining this.

lib/librte_member/rte_member_ht.c line 63

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+	}
+	return 0;
+}
+
+static inline int
+search_bucket_single(uint32_t bucket, member_sig_t tmp_sig,
+		struct member_ht_bucket *buckets,
+		member_set_t *set_id)
+{
+	int iter;

Iter should be "unsigned int" (or similar, maybe "uint8_t")

Quoted 6 lines
lib/librte_member/rte_member_ht.c new file +6 -0
+	for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+		if (tmp_sig == buckets[bucket].sigs[iter] &&
+				buckets[bucket].sets[iter] !=
+				RTE_MEMBER_NO_MATCH) {
+			*set_id = buckets[bucket].sets[iter];
+			return 1;

lib/librte_member/rte_member_ht.c line 79

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+		}
+	}
+	return 0;
+}
+
+static inline void
+search_bucket_multi(uint32_t bucket, member_sig_t tmp_sig,
+		struct member_ht_bucket *buckets,
+		uint32_t *counter,
+		uint32_t match_per_key,

Better change to "matches_per_key".

Quoted 18 lines
lib/librte_member/rte_member_ht.c new file +18 -0
+		member_set_t *set_id)
+{
+	int iter;
+	for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+		if (tmp_sig == buckets[bucket].sigs[iter] &&
+				buckets[bucket].sets[iter] !=
+				RTE_MEMBER_NO_MATCH) {
+			set_id[*counter] = buckets[bucket].sets[iter];
+			(*counter)++;
+			if (*counter >= match_per_key)
+				return;
+		}
+	}
+}
+
+int
+rte_member_create_ht(struct rte_member_setsum *ss,
+		const struct rte_member_parameters *params) {

...

lib/librte_member/rte_member_ht.c line 139

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+
+	RTE_MEMBER_LOG(DEBUG, "Hash table based filter created, "
+			"the table has %u entries, %u buckets\n",
+		num_buckets,
+		num_buckets / RTE_MEMBER_BUCKET_ENTRIES);

Shouldn't this be "num_buckets * RTE_MEMBER_BUCKET_ENTRIES" and "num_buckets"?

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+	return 0;
+}
+
+static inline
+void get_buckets_index(const struct rte_member_setsum *ss, const void

*key,

lib/librte_member/rte_member_ht.c new file +1 -0
+		uint32_t *prim_bkt, uint32_t *sec_bkt, member_sig_t

*sig) {

"static inline void" should be in the same line.

lib/librte_member/rte_member_ht.c line 153

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +3 -0
+	uint32_t first_hash = MEMBER_HASH_FUNC(key, ss->key_len,
+						ss->prim_hash_seed);
+	uint32_t sec_hash = MEMBER_HASH_FUNC(&first_hash,

sizeof(uint32_t),

lib/librte_member/rte_member_ht.c new file +4 -0
+						ss->sec_hash_seed);
+	*sig = first_hash;
+	if (ss->cache) {
+		*prim_bkt = sec_hash & ss->bucket_mask;

Is this correct? Using the secondary hash to acces the primary bucket? Why is the cache case different from the non-cache case?
I think this function deserves some comments to explain all this calculations.

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+		*sec_bkt =  (sec_hash >> 16) & ss->bucket_mask;
+	} else {
+		*prim_bkt = sec_hash & ss->bucket_mask;
+		*sec_bkt =  (*prim_bkt ^ *sig) & ss->bucket_mask;
+	}

lib/librte_member/rte_member_ht.c line 169

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+}
+
+int
+rte_member_lookup_ht(const struct rte_member_setsum *ss,
+		const void *key, member_set_t *set_id) {
+	uint32_t prim_bucket, sec_bucket;
+	member_sig_t tmp_sig;
+	struct member_ht_bucket *buckets = ss->table;
+
+

Remove extra blank line.

Quoted 17 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+	*set_id = RTE_MEMBER_NO_MATCH;
+	get_buckets_index(ss, key, &prim_bucket, &sec_bucket,

&tmp_sig);

lib/librte_member/rte_member_ht.c new file +13 -0
+
+	if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+			set_id) ||
+			search_bucket_single(sec_bucket, tmp_sig,
+				buckets, set_id))
+		return 1;
+
+	return 0;
+}
+
+uint32_t
+rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
+		const void **keys, uint32_t num_keys, member_set_t

*set_id) {

lib/librte_member/rte_member_ht.c

Quoted 2 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+	uint32_t i;
+	uint32_t ret = 0;

Better change to something more meaningful, like "nr_matches".

Quoted 20 lines
lib/librte_member/rte_member_ht.c new file +20 -0
+	struct member_ht_bucket *buckets = ss->table;
+	member_sig_t tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX];
+	uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX];
+	uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX];
+
+	for (i = 0; i < num_keys; i++) {
+		get_buckets_index(ss, keys[i], &prim_buckets[i],
+				&sec_buckets[i], &tmp_sig[i]);
+		rte_prefetch0(&buckets[prim_buckets[i]]);
+		rte_prefetch0(&buckets[sec_buckets[i]]);
+	}
+
+	for (i = 0; i < num_keys; i++) {
+		if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+				buckets, &set_id[i]) ||
+				search_bucket_single(sec_buckets[i],
+				tmp_sig[i], buckets, &set_id[i]))
+			ret++;
+		else
+			set_id[i] = RTE_MEMBER_NO_MATCH;

lib/librte_member/rte_member_ht.c line 217

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+	}
+	return ret;
+}
+
+uint32_t
+rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
+		const void *key, uint32_t match_per_key,
+		member_set_t *set_id)
+{
+	uint32_t ret = 0;

Better change to something more meaningful, like "nr_matches". This applies to the following functions.

Quoted 15 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+	uint32_t prim_bucket, sec_bucket;
+	member_sig_t tmp_sig;
+	struct member_ht_bucket *buckets = ss->table;
+
+	get_buckets_index(ss, key, &prim_bucket, &sec_bucket,

&tmp_sig);

lib/librte_member/rte_member_ht.c new file +9 -0
+
+	search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+			 match_per_key, set_id);
+	if (ret < match_per_key)
+		search_bucket_multi(sec_bucket, tmp_sig,
+			buckets, &ret, match_per_key, set_id);
+	return ret;
+}
+

...

lib/librte_member/rte_member_ht.c line 274

Quoted 6 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+static inline int
+try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t

sec,

lib/librte_member/rte_member_ht.c new file +3 -0
+		member_sig_t sig, member_set_t set_id) {
+	int i;
+	/* If not full then insert into one slot*/

Extra space at the end of the comment, before */

lib/librte_member/rte_member_ht.c line 282

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +8 -0
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		if (buckets[prim].sets[i] == RTE_MEMBER_NO_MATCH) {
+			buckets[prim].sigs[i] = sig;
+			buckets[prim].sets[i] = set_id;
+			return 0;
+		}
+	}
+	/* if prim failed, we need to access second cache line */

Second bucket, instead of second cache line? Also, check that all comments start with capital letters.

Quoted 35 lines
lib/librte_member/rte_member_ht.c new file +12 -0
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		if (buckets[sec].sets[i] == RTE_MEMBER_NO_MATCH) {
+			buckets[sec].sigs[i] = sig;
+			buckets[sec].sets[i] = set_id;
+			return 0;
+		}
+	}
+	return -1;
+}
+
+static inline int
+try_overwrite(struct member_ht_bucket *buckets, uint32_t prim,

uint32_t sec,

lib/librte_member/rte_member_ht.c new file +21 -0
+		member_sig_t sig, member_set_t set_id) {
+	if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+			insert_overwrite_search(sec, sig, buckets,
+				set_id))
+		return 0;
+	return -1;
+}
+
+static inline int
+evict_from_bucket(void)
+{
+	/* for now, we randomly pick one entry to evict */
+	return rte_rand() & (RTE_MEMBER_BUCKET_ENTRIES - 1); }
+
+/*
+ * This function is similar to the cuckoo hash make_space function in
+hash
+ * library
+ */
+static inline int
+make_space_bucket(const struct rte_member_setsum *ss, uint32_t

bkt_num)

lib/librte_member/rte_member_ht.c

Quoted 2 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+{
+	static unsigned int nr_pushes;

A patch has been sent to change this static variable to be non-static, in the hash library. Since this is following a similar implementation, it will need the same change:
http://dpdk.org/dev/patchwork/patch/29104/

lib/librte_member/rte_member_ht.c line 325

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+	unsigned int i, j;
+	int ret;
+	struct member_ht_bucket *buckets = ss->table;
+	uint32_t next_bucket_idx;
+	struct member_ht_bucket

*next_bkt[RTE_MEMBER_BUCKET_ENTRIES];

lib/librte_member/rte_member_ht.c new file +2 -0
+	struct member_ht_bucket *bkt = &buckets[bkt_num];
+	member_set_t flag_mask = 1U << (sizeof(member_set_t) * 8 - 1);

Include a comment about this flag_mask
(MSB is set to indicate if an entry has been already pushed).

Quoted 21 lines
lib/librte_member/rte_member_ht.c new file +7 -0
+	/*
+	 * Push existing item (search for bucket with space in
+	 * alternative locations) to its alternative location
+	 */
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		/* Search for space in alternative locations */
+		next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss-
Quoted 1 line

bucket_mask;

lib/librte_member/rte_member_ht.c new file +3 -0
+		next_bkt[i] = &buckets[next_bucket_idx];
+		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++) {
+			if (next_bkt[i]->sets[j] ==

RTE_MEMBER_NO_MATCH)

lib/librte_member/rte_member_ht.c new file +8 -0
+				break;
+		}
+
+		if (j != RTE_MEMBER_BUCKET_ENTRIES)
+			break;
+	}
+
+	/* Alternative location has spare room (end of recursive function)

*/

lib/librte_member/rte_member_ht.c line 347

Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +4 -0
+	if (i != RTE_MEMBER_BUCKET_ENTRIES) {
+		next_bkt[i]->sigs[j] = bkt->sigs[i];
+		next_bkt[i]->sets[j] = bkt->sets[i];
+		return i;

Since you want to return 1 when there is an eviction, I think you can return 1 here, no need to return the index in the bucket.

Quoted 21 lines
lib/librte_member/rte_member_ht.c new file +21 -0
+	}
+
+	/* Pick entry that has not been pushed yet */
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++)
+		if ((bkt->sets[i] & flag_mask) == 0)
+			break;
+
+	/* All entries have been pushed, so entry cannot be added */
+	if (i == RTE_MEMBER_BUCKET_ENTRIES ||
+			nr_pushes > RTE_MEMBER_MAX_PUSHES)
+		return -ENOSPC;
+
+	next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask;
+	/* Set flag to indicate that this entry is going to be pushed */
+	bkt->sets[i] |= flag_mask;
+
+	nr_pushes++;
+	/* Need room in alternative bucket to insert the pushed entry */
+	ret = make_space_bucket(ss, next_bucket_idx);
+	/*
+	 * After recursive function.

lib/librte_member/rte_member_ht.c line 378

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+	 * Clear flags and insert the pushed entry
+	 * in its alternative location if successful,
+	 * or return error
+	 */
+	bkt->sets[i] &= ~flag_mask;
+	nr_pushes = 0;
+	if (ret >= 0) {
+		next_bkt[i]->sigs[ret] = bkt->sigs[i];
+		next_bkt[i]->sets[ret] = bkt->sets[i];
+		return i;

Same about return 1 here.

Quoted 24 lines
lib/librte_member/rte_member_ht.c new file +14 -0
+	} else
+		return ret;
+}
+
+int
+rte_member_add_ht(const struct rte_member_setsum *ss,
+		const void *key, member_set_t set_id) {
+	int ret;
+	uint32_t prim_bucket, sec_bucket;
+	member_sig_t tmp_sig;
+	struct member_ht_bucket *buckets = ss->table;
+	member_set_t flag_mask = 1U << (sizeof(member_set_t) * 8 - 1);
+
+	if (set_id == RTE_MEMBER_NO_MATCH || (set_id & flag_mask) !=

0)

lib/librte_member/rte_member_ht.c new file +3 -0
+		return -EINVAL;
+
+	get_buckets_index(ss, key, &prim_bucket, &sec_bucket,

&tmp_sig);

lib/librte_member/rte_member_ht.c new file +4 -0
+
+	/* if it is cache based filter, we try overwriting existing entry */
+	if (ss->cache) {
+		ret = try_overwrite(buckets, prim_bucket, sec_bucket,

tmp_sig,

lib/librte_member/rte_member_ht.c line 401

Quoted 1 line
lib/librte_member/rte_member_ht.c new file +1 -0
+					set_id);

If the comment that I made above (in the insert_overwrite_search function) is true and this function is used to update the set_id of an entry, can this be used also in non-cache mode?

lib/librte_member/rte_member_ht.c line 405

Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +4 -0
+		if (ret != -1)
+			return ret;
+	}
+	/* If not full then insert into one slot*/

Extra space before */.

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +7 -0
+	ret = try_insert(buckets, prim_bucket, sec_bucket, tmp_sig, set_id);
+	if (ret != -1)
+		return ret;
+
+	/* random pick prim or sec for recursive displacement */
+
+	uint32_t select_bucket = (tmp_sig && 1U) ? prim_bucket :

sec_bucket;

lib/librte_member/rte_member_ht.c line 420

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +8 -0
+	if (ss->cache) {
+		ret = evict_from_bucket();
+		buckets[select_bucket].sigs[ret] = tmp_sig;
+		buckets[select_bucket].sets[ret] = set_id;
+		return 1;
+	}
+
+	ret = make_space_bucket(ss, select_bucket);

If this only return a negative value when failure or 1 when success, you can check for ret == 1 and not set ret = 1.

Quoted 14 lines
lib/librte_member/rte_member_ht.c new file +14 -0
+	if (ret >= 0) {
+		buckets[select_bucket].sigs[ret] = tmp_sig;
+		buckets[select_bucket].sets[ret] = set_id;
+		ret = 1;
+	}
+
+	return ret;
+}
+
+void
+rte_member_free_ht(struct rte_member_setsum *ss) {
+	rte_free(ss->table);
+}
+

...

lib/librte_member/rte_member_ht.c line 469

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+
+void
+rte_member_reset_ht(const struct rte_member_setsum *ss) {
+	uint32_t i, j;
+	struct member_ht_bucket *buckets = ss->table;

To keep to consistency, I would leave a blank line between the variables declarations and the rest of the function implementation.

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+	for (i = 0; i < ss->bucket_cnt; i++) {
+		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
+			buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
+	}
+}
#46 lore raw

Re: [PATCH v4 2/7] member: implement HT mode

Thank you very much Pablo.

I agree with you on most of your comments and I will address them soon.
But please find some explanations inlined for certain questions you raised.

Quoted 145 lines

-----Original Message-----
From: De Lara Guarch, Pablo
Sent: Monday, October 2, 2017 6:31 AM
To: Wang, Yipeng1 <[email protected]>; [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>
Subject: RE: [PATCH v4 2/7] member: implement HT mode

Quoted 10 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:40 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 2/7] member: implement HT mode

...

lib/librte_member/Makefile

Quoted 1 line
lib/librte_member/Makefile +0 -0

index

Quoted 19 lines

1a79eaa..ad26548 100644

lib/librte_member/Makefile +0 -0
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map

LIBABIVER := 1

 # all source are stored in SRCS-y
lib/librte_member/Makefile +1 -1
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c

rte_member_ht.c

 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_ht.c new file +0 -0
new file mode 100644
index 0000000..55672a4

...

lib/librte_member/rte_member_ht.c line 46

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+
+static inline int
+insert_overwrite_search(uint32_t bucket, member_sig_t tmp_sig,
+		struct member_ht_bucket *buckets,
+		member_set_t set_id)

I would call "bucket", "bucket_id", for better understanding. This comment also applies to other parts of the code (e.g. prim_buckets).

lib/librte_member/rte_member_ht.c line 53

Quoted 7 lines
lib/librte_member/rte_member_ht.c new file +7 -0
+{
+	int i;
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		if (buckets[bucket].sigs[i] == tmp_sig) {
+			buckets[bucket].sets[i] = set_id;
+			return 1;
+		}

Is this function used to update an existing entry?
At first, I thought that this was evicting another entry, when the bucket was full,
but it looks like it is updating the set_id of an existing entry. If this is the case, I would change the function name and add a comment explaining this.

lib/librte_member/rte_member_ht.c line 63

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+	}
+	return 0;
+}
+
+static inline int
+search_bucket_single(uint32_t bucket, member_sig_t tmp_sig,
+		struct member_ht_bucket *buckets,
+		member_set_t *set_id)
+{
+	int iter;

Iter should be "unsigned int" (or similar, maybe "uint8_t")

Quoted 6 lines
lib/librte_member/rte_member_ht.c new file +6 -0
+	for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+		if (tmp_sig == buckets[bucket].sigs[iter] &&
+				buckets[bucket].sets[iter] !=
+				RTE_MEMBER_NO_MATCH) {
+			*set_id = buckets[bucket].sets[iter];
+			return 1;

lib/librte_member/rte_member_ht.c line 79

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+		}
+	}
+	return 0;
+}
+
+static inline void
+search_bucket_multi(uint32_t bucket, member_sig_t tmp_sig,
+		struct member_ht_bucket *buckets,
+		uint32_t *counter,
+		uint32_t match_per_key,

Better change to "matches_per_key".

Quoted 18 lines
lib/librte_member/rte_member_ht.c new file +18 -0
+		member_set_t *set_id)
+{
+	int iter;
+	for (iter = 0; iter < RTE_MEMBER_BUCKET_ENTRIES; iter++) {
+		if (tmp_sig == buckets[bucket].sigs[iter] &&
+				buckets[bucket].sets[iter] !=
+				RTE_MEMBER_NO_MATCH) {
+			set_id[*counter] = buckets[bucket].sets[iter];
+			(*counter)++;
+			if (*counter >= match_per_key)
+				return;
+		}
+	}
+}
+
+int
+rte_member_create_ht(struct rte_member_setsum *ss,
+		const struct rte_member_parameters *params) {

...

lib/librte_member/rte_member_ht.c line 139

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+
+	RTE_MEMBER_LOG(DEBUG, "Hash table based filter created, "
+			"the table has %u entries, %u buckets\n",
+		num_buckets,
+		num_buckets / RTE_MEMBER_BUCKET_ENTRIES);

Shouldn't this be "num_buckets * RTE_MEMBER_BUCKET_ENTRIES" and "num_buckets"?

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+	return 0;
+}
+
+static inline
+void get_buckets_index(const struct rte_member_setsum *ss, const void

*key,

lib/librte_member/rte_member_ht.c new file +1 -0
+		uint32_t *prim_bkt, uint32_t *sec_bkt, member_sig_t

*sig) {

"static inline void" should be in the same line.

Quoted 13 lines

lib/librte_member/rte_member_ht.c line 153

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +3 -0
+	uint32_t first_hash = MEMBER_HASH_FUNC(key, ss->key_len,
+						ss->prim_hash_seed);
+	uint32_t sec_hash = MEMBER_HASH_FUNC(&first_hash,

sizeof(uint32_t),

lib/librte_member/rte_member_ht.c new file +4 -0
+						ss->sec_hash_seed);
+	*sig = first_hash;
+	if (ss->cache) {
+		*prim_bkt = sec_hash & ss->bucket_mask;

Is this correct? Using the secondary hash to acces the primary bucket? Why is the cache case different from the non-cache case?
I think this function deserves some comments to explain all this calculations.

[Wang, Yipeng]
We use the first hash value for the signature, and the second hash value to derive the primary and secondary bucket locations.

For the non-cache mode, we use xor hashing which is called partial-key cuckoo hashing proposed by B. Fan, et al's paper
"Cuckoo Filter: Practically Better Than Bloom". Since the non-cache mode
is similar to cuckoo filter, the partial key hash enables the derivation of alternative bucket location by only using signatures without full key or storing alternative signature.

For cache mode, we do not need to get alternative bucket (just overwritten), so we use higher/lower bits for more independent hash values.

I will add a comment to explain in next version.

Quoted 188 lines
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+		*sec_bkt =  (sec_hash >> 16) & ss->bucket_mask;
+	} else {
+		*prim_bkt = sec_hash & ss->bucket_mask;
+		*sec_bkt =  (*prim_bkt ^ *sig) & ss->bucket_mask;
+	}

lib/librte_member/rte_member_ht.c line 169

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+}
+
+int
+rte_member_lookup_ht(const struct rte_member_setsum *ss,
+		const void *key, member_set_t *set_id) {
+	uint32_t prim_bucket, sec_bucket;
+	member_sig_t tmp_sig;
+	struct member_ht_bucket *buckets = ss->table;
+
+

Remove extra blank line.

Quoted 17 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+	*set_id = RTE_MEMBER_NO_MATCH;
+	get_buckets_index(ss, key, &prim_bucket, &sec_bucket,

&tmp_sig);

lib/librte_member/rte_member_ht.c new file +13 -0
+
+	if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+			set_id) ||
+			search_bucket_single(sec_bucket, tmp_sig,
+				buckets, set_id))
+		return 1;
+
+	return 0;
+}
+
+uint32_t
+rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
+		const void **keys, uint32_t num_keys, member_set_t

*set_id) {

lib/librte_member/rte_member_ht.c

Quoted 2 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+	uint32_t i;
+	uint32_t ret = 0;

Better change to something more meaningful, like "nr_matches".

Quoted 20 lines
lib/librte_member/rte_member_ht.c new file +20 -0
+	struct member_ht_bucket *buckets = ss->table;
+	member_sig_t tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX];
+	uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX];
+	uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX];
+
+	for (i = 0; i < num_keys; i++) {
+		get_buckets_index(ss, keys[i], &prim_buckets[i],
+				&sec_buckets[i], &tmp_sig[i]);
+		rte_prefetch0(&buckets[prim_buckets[i]]);
+		rte_prefetch0(&buckets[sec_buckets[i]]);
+	}
+
+	for (i = 0; i < num_keys; i++) {
+		if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+				buckets, &set_id[i]) ||
+				search_bucket_single(sec_buckets[i],
+				tmp_sig[i], buckets, &set_id[i]))
+			ret++;
+		else
+			set_id[i] = RTE_MEMBER_NO_MATCH;

lib/librte_member/rte_member_ht.c line 217

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+	}
+	return ret;
+}
+
+uint32_t
+rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
+		const void *key, uint32_t match_per_key,
+		member_set_t *set_id)
+{
+	uint32_t ret = 0;

Better change to something more meaningful, like "nr_matches". This applies to the following functions.

Quoted 15 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+	uint32_t prim_bucket, sec_bucket;
+	member_sig_t tmp_sig;
+	struct member_ht_bucket *buckets = ss->table;
+
+	get_buckets_index(ss, key, &prim_bucket, &sec_bucket,

&tmp_sig);

lib/librte_member/rte_member_ht.c new file +9 -0
+
+	search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+			 match_per_key, set_id);
+	if (ret < match_per_key)
+		search_bucket_multi(sec_bucket, tmp_sig,
+			buckets, &ret, match_per_key, set_id);
+	return ret;
+}
+

...

lib/librte_member/rte_member_ht.c line 274

Quoted 6 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+static inline int
+try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t

sec,

lib/librte_member/rte_member_ht.c new file +3 -0
+		member_sig_t sig, member_set_t set_id) {
+	int i;
+	/* If not full then insert into one slot*/

Extra space at the end of the comment, before */

lib/librte_member/rte_member_ht.c line 282

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +8 -0
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		if (buckets[prim].sets[i] == RTE_MEMBER_NO_MATCH) {
+			buckets[prim].sigs[i] = sig;
+			buckets[prim].sets[i] = set_id;
+			return 0;
+		}
+	}
+	/* if prim failed, we need to access second cache line */

Second bucket, instead of second cache line? Also, check that all comments start with capital letters.

Quoted 35 lines
lib/librte_member/rte_member_ht.c new file +12 -0
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		if (buckets[sec].sets[i] == RTE_MEMBER_NO_MATCH) {
+			buckets[sec].sigs[i] = sig;
+			buckets[sec].sets[i] = set_id;
+			return 0;
+		}
+	}
+	return -1;
+}
+
+static inline int
+try_overwrite(struct member_ht_bucket *buckets, uint32_t prim,

uint32_t sec,

lib/librte_member/rte_member_ht.c new file +21 -0
+		member_sig_t sig, member_set_t set_id) {
+	if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+			insert_overwrite_search(sec, sig, buckets,
+				set_id))
+		return 0;
+	return -1;
+}
+
+static inline int
+evict_from_bucket(void)
+{
+	/* for now, we randomly pick one entry to evict */
+	return rte_rand() & (RTE_MEMBER_BUCKET_ENTRIES - 1); }
+
+/*
+ * This function is similar to the cuckoo hash make_space function in
+hash
+ * library
+ */
+static inline int
+make_space_bucket(const struct rte_member_setsum *ss, uint32_t

bkt_num)

lib/librte_member/rte_member_ht.c

Quoted 2 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+{
+	static unsigned int nr_pushes;

A patch has been sent to change this static variable to be non-static, in the hash library. Since this is following a similar implementation, it will need the same change:
http://dpdk.org/dev/patchwork/patch/29104/

lib/librte_member/rte_member_ht.c line 325

Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+	unsigned int i, j;
+	int ret;
+	struct member_ht_bucket *buckets = ss->table;
+	uint32_t next_bucket_idx;
+	struct member_ht_bucket

*next_bkt[RTE_MEMBER_BUCKET_ENTRIES];

lib/librte_member/rte_member_ht.c new file +2 -0
+	struct member_ht_bucket *bkt = &buckets[bkt_num];
+	member_set_t flag_mask = 1U << (sizeof(member_set_t) * 8 - 1);

Include a comment about this flag_mask
(MSB is set to indicate if an entry has been already pushed).

Quoted 21 lines
lib/librte_member/rte_member_ht.c new file +7 -0
+	/*
+	 * Push existing item (search for bucket with space in
+	 * alternative locations) to its alternative location
+	 */
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		/* Search for space in alternative locations */
+		next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss-
Quoted 1 line

bucket_mask;

lib/librte_member/rte_member_ht.c new file +3 -0
+		next_bkt[i] = &buckets[next_bucket_idx];
+		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++) {
+			if (next_bkt[i]->sets[j] ==

RTE_MEMBER_NO_MATCH)

lib/librte_member/rte_member_ht.c new file +8 -0
+				break;
+		}
+
+		if (j != RTE_MEMBER_BUCKET_ENTRIES)
+			break;
+	}
+
+	/* Alternative location has spare room (end of recursive function)

*/

Quoted 8 lines

lib/librte_member/rte_member_ht.c line 347

Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +4 -0
+	if (i != RTE_MEMBER_BUCKET_ENTRIES) {
+		next_bkt[i]->sigs[j] = bkt->sigs[i];
+		next_bkt[i]->sets[j] = bkt->sets[i];
+		return i;

Since you want to return 1 when there is an eviction, I think you can return 1 here,
no need to return the index in the bucket.

[Wang, Yipeng]
We return the index in the bucket to keep the recursive function going right? I.e. the make_space needs the returned index to displace the entry.

Quoted 56 lines
Quoted 22 lines
lib/librte_member/rte_member_ht.c new file +21 -0
+	}
+
+	/* Pick entry that has not been pushed yet */
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++)
+		if ((bkt->sets[i] & flag_mask) == 0)
+			break;
+
+	/* All entries have been pushed, so entry cannot be added */
+	if (i == RTE_MEMBER_BUCKET_ENTRIES ||
+			nr_pushes > RTE_MEMBER_MAX_PUSHES)
+		return -ENOSPC;
+
+	next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask;
+	/* Set flag to indicate that this entry is going to be pushed */
+	bkt->sets[i] |= flag_mask;
+
+	nr_pushes++;
+	/* Need room in alternative bucket to insert the pushed entry */
+	ret = make_space_bucket(ss, next_bucket_idx);
+	/*
+	 * After recursive function.

lib/librte_member/rte_member_ht.c line 378

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+	 * Clear flags and insert the pushed entry
+	 * in its alternative location if successful,
+	 * or return error
+	 */
+	bkt->sets[i] &= ~flag_mask;
+	nr_pushes = 0;
+	if (ret >= 0) {
+		next_bkt[i]->sigs[ret] = bkt->sigs[i];
+		next_bkt[i]->sets[ret] = bkt->sets[i];
+		return i;

Same about return 1 here.

Quoted 21 lines
lib/librte_member/rte_member_ht.c new file +14 -0
+	} else
+		return ret;
+}
+
+int
+rte_member_add_ht(const struct rte_member_setsum *ss,
+		const void *key, member_set_t set_id) {
+	int ret;
+	uint32_t prim_bucket, sec_bucket;
+	member_sig_t tmp_sig;
+	struct member_ht_bucket *buckets = ss->table;
+	member_set_t flag_mask = 1U << (sizeof(member_set_t) * 8 - 1);
+
+	if (set_id == RTE_MEMBER_NO_MATCH || (set_id & flag_mask) !=

0)

lib/librte_member/rte_member_ht.c new file +3 -0
+		return -EINVAL;
+
+	get_buckets_index(ss, key, &prim_bucket, &sec_bucket,

&tmp_sig);

lib/librte_member/rte_member_ht.c new file +2 -0
+
+	/* if it is cache based filter, we try overwriting existing entry */
Quoted 9 lines

lib/librte_member/rte_member_ht.c line 401

Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +2 -0
+	if (ss->cache) {
+		ret = try_overwrite(buckets, prim_bucket, sec_bucket,

tmp_sig,

lib/librte_member/rte_member_ht.c new file +1 -0
+					set_id);

If the comment that I made above (in the insert_overwrite_search function) is true and this function is used to update the set_id of an entry, can this be used also in non-cache mode?

[Wang, Yipeng]
I changed the name to update_entry_search. For non-cache mode, we do not do update even if the key finds an entry with the same signature during add. This is because for non-cache mode, we do not allow false negative, keys Overwriting each other may incur false negative.

I will add more explanation in comments.

Quoted 18 lines

lib/librte_member/rte_member_ht.c line 405

Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +4 -0
+		if (ret != -1)
+			return ret;
+	}
+	/* If not full then insert into one slot*/

Extra space before */.

Quoted 11 lines
lib/librte_member/rte_member_ht.c new file +7 -0
+	ret = try_insert(buckets, prim_bucket, sec_bucket, tmp_sig, set_id);
+	if (ret != -1)
+		return ret;
+
+	/* random pick prim or sec for recursive displacement */
+
+	uint32_t select_bucket = (tmp_sig && 1U) ? prim_bucket :

sec_bucket;

lib/librte_member/rte_member_ht.c new file +3 -0
+	if (ss->cache) {
+		ret = evict_from_bucket();
+		buckets[select_bucket].sigs[ret] = tmp_sig;
Quoted 9 lines

lib/librte_member/rte_member_ht.c line 420

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+		buckets[select_bucket].sets[ret] = set_id;
+		return 1;
+	}
+
+	ret = make_space_bucket(ss, select_bucket);

If this only return a negative value when failure or 1 when success, you can check for ret == 1 and not set ret = 1.

[Wang, Yipeng]
Please see my comments above, if we return 1 for make_space, will the recursive process be broken?

Quoted 32 lines
Quoted 14 lines
lib/librte_member/rte_member_ht.c new file +14 -0
+	if (ret >= 0) {
+		buckets[select_bucket].sigs[ret] = tmp_sig;
+		buckets[select_bucket].sets[ret] = set_id;
+		ret = 1;
+	}
+
+	return ret;
+}
+
+void
+rte_member_free_ht(struct rte_member_setsum *ss) {
+	rte_free(ss->table);
+}
+

...

lib/librte_member/rte_member_ht.c line 469

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+
+void
+rte_member_reset_ht(const struct rte_member_setsum *ss) {
+	uint32_t i, j;
+	struct member_ht_bucket *buckets = ss->table;

To keep to consistency, I would leave a blank line between the variables declarations
and the rest of the function implementation.

Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5 -0
+	for (i = 0; i < ss->bucket_cnt; i++) {
+		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
+			buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
+	}
+}
#47 lore raw

PATCH v4 3/7 member: implement vBF mode

In reply to Yipeng Wang

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

---
 lib/librte_member/Makefile         |   2 +-
 lib/librte_member/rte_member_vbf.c | 349 +++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_vbf.h |  79 +++++++++
 3 files changed, 429 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
index ad26548..50275ed 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c rte_member_vbf.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_vbf.c new file +349 -0

349 lines not on this page · raw message

lib/librte_member/rte_member_vbf.h new file +79 -0

79 lines not on this page · raw message

Signature · 1 line
2.7.4
#48 lore raw

Re: [PATCH v4 3/7] member: implement vBF mode

Quoted 29 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:41 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 3/7] member: implement vBF mode

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

...

Quoted 2 lines
lib/librte_member/rte_member_vbf.c +0 -0

...

Quoted 6 lines
lib/librte_member/rte_member_vbf.c +5 -0
+int
+rte_member_create_vbf(struct rte_member_setsum *ss,
+		const struct rte_member_parameters *params) {
+
+	if (params->num_set > 32 || !rte_is_power_of_2(params-
Quoted 1 line

num_set) ||

Magic number. Define a macro instead.

Quoted 7 lines
lib/librte_member/rte_member_vbf.c +5 -0
+			params->num_keys == 0 ||
+			params->false_positive_rate == 0 ||
+			params->false_positive_rate > 1) {
+		rte_errno = EINVAL;
+		RTE_MEMBER_LOG(ERR, "vBF create with invalid

parameters\n");

lib/librte_member/rte_member_vbf.c +1 -0
+		return -EINVAL;

...

lib/librte_member/rte_member_vbf.c line 112

Quoted 4 lines
lib/librte_member/rte_member_vbf.c +4 -0
+
+	/*
+	 * reduce hash function count, until we approach the user specified
+	 * false-positive rate. otherwise it is too conservative

Watch out for capital letters at the start of the comment and after a full stop.

Quoted 24 lines
lib/librte_member/rte_member_vbf.c +8 -0
+	 */
+	int tmp_num_hash = ss->num_hashes;
+
+	while (tmp_num_hash > 1) {
+		float tmp_fp = new_fp;
+
+		tmp_num_hash--;
+		new_fp = pow((1 - pow((1 - 1.0 / ss->bits),

num_keys_per_bf *

lib/librte_member/rte_member_vbf.c +14 -0
+					tmp_num_hash)), tmp_num_hash);
+		new_fp = 1 - pow((1 - new_fp), ss->num_set);
+
+		if (new_fp > params->false_positive_rate) {
+			new_fp = tmp_fp;
+			tmp_num_hash++;
+			break;
+		}
+	}
+
+	ss->num_hashes = tmp_num_hash;
+
+	RTE_MEMBER_LOG(DEBUG, "vector bloom filter created, "
+		"each bloom filter expects %u keys, needs %u bits, %u

hashes, "

lib/librte_member/rte_member_vbf.c line 137

Quoted 3 lines
lib/librte_member/rte_member_vbf.c +3 -0
+		"with false positive rate set as %.5f, "
+		"The new calculated vBF false positive rate is %.5f\n",
+		num_keys_per_bf, ss->bits, ss->num_hashes, x, new_fp);

Use a more descriptive variable name for "x".

Quoted 6 lines
lib/librte_member/rte_member_vbf.c +3 -0
+
+	ss->table = rte_zmalloc_socket(NULL, ss->num_set * (ss->bits >> 3),
+					RTE_CACHE_LINE_SIZE, ss-
Quoted 1 line

socket_id);

lib/librte_member/rte_member_vbf.c +2 -0
+
+	/*

lib/librte_member/rte_member_vbf.c line 152

Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10 -0
+	 * To avoid multiplication and division:
+	 * mul_shift is used for multiplication shift during bit test
+	 * div_shift is used for division shift, to be divided by number of bits
+	 * represented by a uint32_t variable
+	 */
+	ss->mul_shift = __builtin_ctzl(ss->num_set);
+	ss->div_shift = __builtin_ctzl(32 >> ss->mul_shift);
+
+	if (ss->table == NULL)
+		return -ENOMEM;

I would move this check just after the malloc call.

Quoted 18 lines
lib/librte_member/rte_member_vbf.c +17 -0
+
+	return 0;
+}
+
+static inline uint32_t
+test_bit(uint32_t bit_loc, const struct rte_member_setsum *ss) {
+	uint32_t *vbf = ss->table;
+	uint32_t n = ss->num_set;
+	uint32_t div_shift = ss->div_shift;
+	uint32_t mul_shift = ss->mul_shift;
+	/*
+	 * a is how many bits in one BF are represented by one 32bit
+	 * variable.
+	 */
+	uint32_t a = 32 >> mul_shift;
+	/*
+	 * x>>b is the divide, x & (a-1) is the mod, & (1<<n-1) to mask out

bits

lib/librte_member/rte_member_vbf.c

Quoted 3 lines
lib/librte_member/rte_member_vbf.c +3 -0
+	 * we do not need
+	 */
+	return (vbf[bit_loc>>div_shift] >> ((bit_loc & (a - 1)) << mul_shift))

Add spaces around ">>".

Quoted 4 lines

&

lib/librte_member/rte_member_vbf.c +3 -0
+							((1ULL << n) - 1);
+}
+

lib/librte_member/rte_member_vbf.c line 186

Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10 -0
+static inline void
+set_bit(uint32_t bit_loc, const struct rte_member_setsum *ss, int32_t
+set) {
+	uint32_t *vbf = ss->table;
+	uint32_t div_shift = ss->div_shift;
+	uint32_t mul_shift = ss->mul_shift;
+	uint32_t a = 32 >> mul_shift;
+
+	vbf[bit_loc>>div_shift] |= 1U << (((bit_loc & (a - 1)) << mul_shift) +
+								set - 1);

Same as above.

Quoted 13 lines
lib/librte_member/rte_member_vbf.c +4 -0
+}
+
+int
+rte_member_lookup_vbf(const struct rte_member_setsum *ss, const

void *key,

lib/librte_member/rte_member_vbf.c +4 -0
+		member_set_t *set_id)
+{
+	uint32_t j;
+	uint32_t h1 = MEMBER_HASH_FUNC(key, ss->key_len, ss-
Quoted 1 line

prim_hash_seed);

lib/librte_member/rte_member_vbf.c +3 -0
+	uint32_t h2 = MEMBER_HASH_FUNC(&h1, sizeof(uint32_t),
+						ss->sec_hash_seed);
+	uint32_t mask = ~0;

lib/librte_member/rte_member_vbf.c line 207

Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10 -0
+	uint32_t bit_loc;
+
+	for (j = 0; j < ss->num_hashes; j++) {
+		bit_loc = (h1 + j * h2) & ss->bit_mask;
+		mask &= test_bit(bit_loc, ss);
+	}
+
+	if (mask) {
+		*set_id = __builtin_ctzl(mask) + 1;
+		return 1;

Wouldn't it be better to return 0 when there is a hit and -ENOENT when there is not?

lib/librte_member/rte_member_vbf.c

Quoted 12 lines
lib/librte_member/rte_member_vbf.c +9 -0
+	}
+
+	*set_id = RTE_MEMBER_NO_MATCH;
+	return 0;
+}
+
+uint32_t
+rte_member_lookup_bulk_vbf(const struct rte_member_setsum *ss,
+		const void **keys, uint32_t num_keys, member_set_t

*set_ids) {

lib/librte_member/rte_member_vbf.c +2 -0
+	uint32_t i, k;
+	uint32_t ret = 0;

Change variable name to "nr_matches/hits" or similar. Same in the next functions.

#49 lore raw

Re: [PATCH v4 3/7] member: implement vBF mode

Please see one comment inlined:

Quoted 172 lines

-----Original Message-----
From: De Lara Guarch, Pablo
Sent: Monday, October 2, 2017 8:44 AM
To: Wang, Yipeng1 <[email protected]>; [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>
Subject: RE: [PATCH v4 3/7] member: implement vBF mode

Quoted 27 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:41 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 3/7] member: implement vBF mode

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

...

Quoted 2 lines
lib/librte_member/rte_member_vbf.c +0 -0

...

Quoted 6 lines
lib/librte_member/rte_member_vbf.c +5 -0
+int
+rte_member_create_vbf(struct rte_member_setsum *ss,
+		const struct rte_member_parameters *params) {
+
+	if (params->num_set > 32 || !rte_is_power_of_2(params-
Quoted 1 line

num_set) ||

Magic number. Define a macro instead.

Quoted 7 lines
lib/librte_member/rte_member_vbf.c +5 -0
+			params->num_keys == 0 ||
+			params->false_positive_rate == 0 ||
+			params->false_positive_rate > 1) {
+		rte_errno = EINVAL;
+		RTE_MEMBER_LOG(ERR, "vBF create with invalid

parameters\n");

lib/librte_member/rte_member_vbf.c +1 -0
+		return -EINVAL;

...

lib/librte_member/rte_member_vbf.c line 112

Quoted 4 lines
lib/librte_member/rte_member_vbf.c +4 -0
+
+	/*
+	 * reduce hash function count, until we approach the user specified
+	 * false-positive rate. otherwise it is too conservative

Watch out for capital letters at the start of the comment and after a full stop.

Quoted 24 lines
lib/librte_member/rte_member_vbf.c +8 -0
+	 */
+	int tmp_num_hash = ss->num_hashes;
+
+	while (tmp_num_hash > 1) {
+		float tmp_fp = new_fp;
+
+		tmp_num_hash--;
+		new_fp = pow((1 - pow((1 - 1.0 / ss->bits),

num_keys_per_bf *

lib/librte_member/rte_member_vbf.c +14 -0
+					tmp_num_hash)), tmp_num_hash);
+		new_fp = 1 - pow((1 - new_fp), ss->num_set);
+
+		if (new_fp > params->false_positive_rate) {
+			new_fp = tmp_fp;
+			tmp_num_hash++;
+			break;
+		}
+	}
+
+	ss->num_hashes = tmp_num_hash;
+
+	RTE_MEMBER_LOG(DEBUG, "vector bloom filter created, "
+		"each bloom filter expects %u keys, needs %u bits, %u

hashes, "

lib/librte_member/rte_member_vbf.c line 137

Quoted 3 lines
lib/librte_member/rte_member_vbf.c +3 -0
+		"with false positive rate set as %.5f, "
+		"The new calculated vBF false positive rate is %.5f\n",
+		num_keys_per_bf, ss->bits, ss->num_hashes, x, new_fp);

Use a more descriptive variable name for "x".

Quoted 6 lines
lib/librte_member/rte_member_vbf.c +3 -0
+
+	ss->table = rte_zmalloc_socket(NULL, ss->num_set * (ss->bits >> 3),
+					RTE_CACHE_LINE_SIZE, ss-
Quoted 1 line

socket_id);

lib/librte_member/rte_member_vbf.c +2 -0
+
+	/*

lib/librte_member/rte_member_vbf.c line 152

Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10 -0
+	 * To avoid multiplication and division:
+	 * mul_shift is used for multiplication shift during bit test
+	 * div_shift is used for division shift, to be divided by number of bits
+	 * represented by a uint32_t variable
+	 */
+	ss->mul_shift = __builtin_ctzl(ss->num_set);
+	ss->div_shift = __builtin_ctzl(32 >> ss->mul_shift);
+
+	if (ss->table == NULL)
+		return -ENOMEM;

I would move this check just after the malloc call.

Quoted 18 lines
lib/librte_member/rte_member_vbf.c +17 -0
+
+	return 0;
+}
+
+static inline uint32_t
+test_bit(uint32_t bit_loc, const struct rte_member_setsum *ss) {
+	uint32_t *vbf = ss->table;
+	uint32_t n = ss->num_set;
+	uint32_t div_shift = ss->div_shift;
+	uint32_t mul_shift = ss->mul_shift;
+	/*
+	 * a is how many bits in one BF are represented by one 32bit
+	 * variable.
+	 */
+	uint32_t a = 32 >> mul_shift;
+	/*
+	 * x>>b is the divide, x & (a-1) is the mod, & (1<<n-1) to mask out

bits

lib/librte_member/rte_member_vbf.c

Quoted 3 lines
lib/librte_member/rte_member_vbf.c +3 -0
+	 * we do not need
+	 */
+	return (vbf[bit_loc>>div_shift] >> ((bit_loc & (a - 1)) << mul_shift))

Add spaces around ">>".

Quoted 4 lines

&

lib/librte_member/rte_member_vbf.c +3 -0
+							((1ULL << n) - 1);
+}
+

lib/librte_member/rte_member_vbf.c line 186

Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10 -0
+static inline void
+set_bit(uint32_t bit_loc, const struct rte_member_setsum *ss, int32_t
+set) {
+	uint32_t *vbf = ss->table;
+	uint32_t div_shift = ss->div_shift;
+	uint32_t mul_shift = ss->mul_shift;
+	uint32_t a = 32 >> mul_shift;
+
+	vbf[bit_loc>>div_shift] |= 1U << (((bit_loc & (a - 1)) << mul_shift) +
+								set - 1);

Same as above.

Quoted 10 lines
lib/librte_member/rte_member_vbf.c +4 -0
+}
+
+int
+rte_member_lookup_vbf(const struct rte_member_setsum *ss, const

void *key,

lib/librte_member/rte_member_vbf.c +4 -0
+		member_set_t *set_id)
+{
+	uint32_t j;
+	uint32_t h1 = MEMBER_HASH_FUNC(key, ss->key_len, ss-
Quoted 1 line

prim_hash_seed);

Quoted 16 lines

lib/librte_member/rte_member_vbf.c line 207

Quoted 13 lines
lib/librte_member/rte_member_vbf.c +13 -0
+	uint32_t h2 = MEMBER_HASH_FUNC(&h1, sizeof(uint32_t),
+						ss->sec_hash_seed);
+	uint32_t mask = ~0;
+	uint32_t bit_loc;
+
+	for (j = 0; j < ss->num_hashes; j++) {
+		bit_loc = (h1 + j * h2) & ss->bit_mask;
+		mask &= test_bit(bit_loc, ss);
+	}
+
+	if (mask) {
+		*set_id = __builtin_ctzl(mask) + 1;
+		return 1;

Wouldn't it be better to return 0 when there is a hit and -ENOENT when there is not?

[Wang, Yipeng]
I tried to follow the convention of other lookup functions that the return value Is kind of the number of matches that found. So 1 means found 1 and 0 means not Found. I defined the public API rte_member_lookup to also returns 1 for found.

Quoted 15 lines

lib/librte_member/rte_member_vbf.c

Quoted 12 lines
lib/librte_member/rte_member_vbf.c +9 -0
+	}
+
+	*set_id = RTE_MEMBER_NO_MATCH;
+	return 0;
+}
+
+uint32_t
+rte_member_lookup_bulk_vbf(const struct rte_member_setsum *ss,
+		const void **keys, uint32_t num_keys, member_set_t

*set_ids) {

lib/librte_member/rte_member_vbf.c +2 -0
+	uint32_t i, k;
+	uint32_t ret = 0;

Change variable name to "nr_matches/hits" or similar. Same in the next functions.

#50 lore raw

PATCH v4 4/7 member: add AVX for HT mode

In reply to Yipeng Wang

For key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contiguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

---
 lib/librte_member/rte_member_ht.c  | 145 ++++++++++++++++++++++++++++---------
 lib/librte_member/rte_member_x86.h | 107 +++++++++++++++++++++++++++
 2 files changed, 219 insertions(+), 33 deletions(-)
 create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +112 -33
index 55672a4..ef1541a 100644
@@ -40,6 +40,10 @@
 #include "rte_member.h"
 #include "rte_member_ht.h"
 
+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+
 static inline int
 insert_overwrite_search(uint32_t bucket, member_sig_t tmp_sig,
 		struct member_ht_bucket *buckets,
@@ -132,6 +136,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
 		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
 			buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
 	}
+#if defined(RTE_ARCH_X86)
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+			RTE_MEMBER_BUCKET_ENTRIES == 16)
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+	else
+#endif
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;
 
 	RTE_MEMBER_LOG(DEBUG, "Hash table based filter created, "
 			"the table has %u entries, %u buckets\n",
@@ -166,15 +177,26 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
 	member_sig_t tmp_sig;
 	struct member_ht_bucket *buckets = ss->table;
 
-
 	*set_id = RTE_MEMBER_NO_MATCH;
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	if (search_bucket_single(prim_bucket, tmp_sig, buckets,
-			set_id) ||
-			search_bucket_single(sec_bucket, tmp_sig,
-				buckets, set_id))
-		return 1;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single_avx(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+		break;
+#endif
+	default:
+		if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+	}
 
 	return 0;
 }
@@ -198,13 +220,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
 	}
 
 	for (i = 0; i < num_keys; i++) {
-		if (search_bucket_single(prim_buckets[i], tmp_sig[i],
-				buckets, &set_id[i]) ||
-				search_bucket_single(sec_buckets[i],
-				tmp_sig[i], buckets, &set_id[i]))
-			ret++;
-		else
-			set_id[i] = RTE_MEMBER_NO_MATCH;
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			if (search_bucket_single_avx(prim_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]) ||
+				search_bucket_single_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				ret++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+			break;
+#endif
+		default:
+			if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+					buckets, &set_id[i]) ||
+					search_bucket_single(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				ret++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+		}
 	}
 	return ret;
 }
@@ -221,12 +257,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
 
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
-			 match_per_key, set_id);
-	if (ret < match_per_key)
-		search_bucket_multi(sec_bucket, tmp_sig,
-			buckets, &ret, match_per_key, set_id);
-	return ret;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+			&ret, match_per_key, set_id);
+		if (ret < match_per_key)
+			search_bucket_multi_avx(sec_bucket, tmp_sig,
+				buckets, &ret, match_per_key, set_id);
+		return ret;
+#endif
+	default:
+		search_bucket_multi(prim_bucket, tmp_sig, buckets, &ret,
+				 match_per_key, set_id);
+		if (ret < match_per_key)
+			search_bucket_multi(sec_bucket, tmp_sig,
+				buckets, &ret, match_per_key, set_id);
+		return ret;
+	}
 }
 
 uint32_t
@@ -252,16 +300,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
 	for (i = 0; i < num_keys; i++) {
 		match_cnt_tmp = 0;
 
-		search_bucket_multi(prim_buckets[i], tmp_sig[i],
-			buckets, &match_cnt_tmp, match_per_key,
-			&set_ids[i*match_per_key]);
-		if (match_cnt_tmp < match_per_key)
-			search_bucket_multi(sec_buckets[i], tmp_sig[i],
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
 				buckets, &match_cnt_tmp, match_per_key,
 				&set_ids[i*match_per_key]);
-		match_count[i] = match_cnt_tmp;
-		if (match_cnt_tmp != 0)
-			ret++;
+			if (match_cnt_tmp < match_per_key)
+				search_bucket_multi_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &match_cnt_tmp,
+					match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_tmp;
+			if (match_cnt_tmp != 0)
+				ret++;
+			break;
+#endif
+		default:
+			search_bucket_multi(prim_buckets[i], tmp_sig[i],
+				buckets, &match_cnt_tmp, match_per_key,
+				&set_ids[i*match_per_key]);
+			if (match_cnt_tmp < match_per_key)
+				search_bucket_multi(sec_buckets[i], tmp_sig[i],
+					buckets, &match_cnt_tmp, match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_tmp;
+			if (match_cnt_tmp != 0)
+				ret++;
+		}
 	}
 	return ret;
 }
@@ -271,6 +337,7 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
 		member_sig_t sig, member_set_t set_id)
 {
 	int i;
+
 	/* If not full then insert into one slot*/
 	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
 		if (buckets[prim].sets[i] == RTE_MEMBER_NO_MATCH) {
@@ -292,12 +359,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
 
 static inline int
 try_overwrite(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
-		member_sig_t sig, member_set_t set_id)
+		member_sig_t sig, member_set_t set_id,
+		enum rte_member_sig_compare_function cmp_fn)
 {
-	if (insert_overwrite_search(prim, sig, buckets, set_id) ||
-			insert_overwrite_search(sec, sig, buckets,
-				set_id))
-		return 0;
+	switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (insert_overwrite_search_avx(prim, sig, buckets, set_id) ||
+				insert_overwrite_search_avx(sec, sig, buckets,
+					set_id))
+			return 0;
+		break;
+#endif
+	default:
+		if (insert_overwrite_search(prim, sig, buckets, set_id) ||
+				insert_overwrite_search(sec, sig, buckets,
+					set_id))
+			return 0;
+	}
 	return -1;
 }
 
@@ -398,7 +477,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
 	/* if it is cache based filter, we try overwriting existing entry */
 	if (ss->cache) {
 		ret = try_overwrite(buckets, prim_bucket, sec_bucket, tmp_sig,
-					set_id);
+					set_id, ss->sig_cmp_fn);
 		if (ret != -1)
 			return ret;
 	}
lib/librte_member/rte_member_x86.h new file +107 -0

107 lines not on this page · raw message

Signature · 1 line
2.7.4
#51 lore raw

PATCH v4 5/7 member: enable the library

In reply to Yipeng Wang

This patch enables the Membership library.

---
 MAINTAINERS                | 8 +++++++-
 config/common_base         | 5 +++++
 lib/librte_member/Makefile | 2 ++
 mk/rte.app.mk              | 2 ++
 4 files changed, 16 insertions(+), 1 deletion(-)
MAINTAINERS +7 -1
index a0cd75e..adb8e2c 100644
@@ -726,6 +726,13 @@ F: test/test/test_lpm*
 F: test/test/test_func_reentrancy.c
 F: test/test/test_xmmt_ops.h
 
+Membership - EXPERIMENTAL
+M: Yipeng Wang <[email protected]>
+M: Sameh Gobriel <[email protected]>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
+
 Traffic metering
 M: Cristian Dumitrescu <[email protected]>
 F: lib/librte_meter/
@@ -734,7 +741,6 @@ F: test/test/test_meter.c
 F: examples/qos_meter/
 F: doc/guides/sample_app_ug/qos_metering.rst
 
-
 Other libraries
 ---------------
 
config/common_base +5 -0
index 5e97a08..5e31ced 100644
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
 CONFIG_RTE_LIBRTE_EFD=y
 
 #
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
 # Compile librte_jobstats
 #
 CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
index 50275ed..3bac1d0 100644
@@ -37,6 +37,8 @@ LIB = librte_member.a
 CFLAGS := -I$(SRCDIR) $(CFLAGS)
 CFLAGS += $(WERROR_FLAGS) -O3
 
+LDLIBS += -lm
+
 EXPORT_MAP := rte_member_version.map
 
 LIBABIVER := 1
mk/rte.app.mk +2 -0
index c25fdd9..c79acf0 100644
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE)        += -lrte_cfgfile
 _LDLIBS-y += --whole-archive
 
 _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH)           += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)        += -lrte_member
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lrte_vhost
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS)         += -lrte_kvargs
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lm
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrt
 _LDLIBS-$(CONFIG_RTE_LIBRTE_METER)          += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)          += -lm
 ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lnuma
 endif
Signature · 1 line
2.7.4
#52 lore raw

Re: [PATCH v4 5/7] member: enable the library

Quoted 27 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:41 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 5/7] member: enable the library

This patch enables the Membership library.

---
 MAINTAINERS                | 8 +++++++-
 config/common_base         | 5 +++++
 lib/librte_member/Makefile | 2 ++
 mk/rte.app.mk              | 2 ++
 4 files changed, 16 insertions(+), 1 deletion(-)
MAINTAINERS +0 -0
index a0cd75e..adb8e2c 100644

MAINTAINERS F: test/test/test_lpm* · line 735

Quoted 10 lines
MAINTAINERS +7 -0
@@ -726,6 +726,13 @@ F: test/test/test_lpm*
 F: test/test/test_func_reentrancy.c
 F: test/test/test_xmmt_ops.h
 
+Membership - EXPERIMENTAL
+M: Yipeng Wang <[email protected]>
+M: Sameh Gobriel <[email protected]>
+F: lib/librte_member/
+F: doc/guides/prog_guide/member_lib.rst
+F: test/test/test_member*
+

Add these last two items in patches 6 and 7, where you are adding the files.

#53 lore raw

PATCH v4 6/7 test/member: add functional and perf tests

In reply to Yipeng Wang

This patch adds functional and performance tests for membership library.

---
 test/test/Makefile           |   3 +
 test/test/test_member.c      | 676 +++++++++++++++++++++++++++++++++++++++++++
 test/test/test_member_perf.c | 630 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1309 insertions(+)
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
test/test/Makefile +3 -0
index 42d9a49..b61dde3 100644
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
 SRCS-y += test_memcpy.c
 SRCS-y += test_memcpy_perf.c
 
+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
 
test/test/test_member.c new file +676 -0

676 lines not on this page · raw message

test/test/test_member_perf.c new file +630 -0

630 lines not on this page · raw message

Signature · 1 line
2.7.4
#54 lore raw

Re: [PATCH v4 6/7] test/member: add functional and perf tests

Quoted 15 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:41 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 6/7] test/member: add functional and perf tests

This patch adds functional and performance tests for membership library.

...

Quoted 1 line

+++ b/test/test/test_member.c

...

test/test/test_member.c line 104

Quoted 3 lines
test/test/test_member.c new file +3 -0
+
+#define MAX_ENTRIES (1 << 16)
+uint8_t gened_keys[MAX_ENTRIES][KEY_SIZE];

Gened? Is this "generated"? Maybe it is worth using the full word.

Review

Quoted 5 lines
test/test/test_member.c new file +3 -0
+
+static struct rte_member_parameters params = {
+		.num_keys = MAX_ENTRIES,       /* Total hash table entries.

*/

(quoted diff) +1 -0
+		.key_len = KEY_SIZE,       /* Length of hash key. */

Align comments with tabs.

Quoted 3 lines
(quoted diff) +2 -0
+
+		/*num_set and false_positive_rate only relevant to vBF

setsum*/

Add whitespaces around comment.

test/test/test_member.c

Quoted 13 lines
test/test/test_member.c new file +11 -0
+		.num_set = 32,
+		.false_positive_rate = 0.03,
+		.prim_hash_seed = 1,
+		.sec_hash_seed = 11,
+		.socket_id = 0          /* NUMA Socket ID for memory. */
+};
 
 
+static int test_member_insert(void)
+{
+	int ret_ht, ret_cache, ret_vbf, i;
+
+	for (i = 0; i < 5; i++) {

Use macro for the value 5, used here and other functions.

Quoted 12 lines
test/test/test_member.c new file +4 -0
+		ret_ht = rte_member_add(setsum_ht, &keys[i], test_set[i]);
+		ret_cache = rte_member_add(setsum_cache, &keys[i],
+						test_set[i]);
+		ret_vbf = rte_member_add(setsum_vbf, &keys[i],

test_set[i]);

(quoted diff) +1 -0
+		TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0 && ret_vbf >=

0,

test/test/test_member.c new file +5 -0
+				"insert error");
+	}
+	printf("insert key success\n");
+	return 0;
+}

...

test/test/test_member.c line 385

Quoted 3 lines
test/test/test_member.c new file +3 -0
+static int test_member_multimatch(void) {
+	int ret_ht, ret_vbf, ret_cache;
+	member_set_t set_ids_ht[32] = {0};

Same comment about the value 5 applies here, for the value 32.

Quoted 21 lines
test/test/test_member.c new file +20 -0
+	member_set_t set_ids_vbf[32] = {0};
+	member_set_t set_ids_cache[32] = {0};
+
+	member_set_t set_ids_ht_m[5][32] = {{0} };
+	member_set_t set_ids_vbf_m[5][32] = {{0} };
+	member_set_t set_ids_cache_m[5][32] = {{0} };
+
+	uint32_t match_count_ht[5];
+	uint32_t match_count_vbf[5];
+	uint32_t match_count_cache[5];
+
+	uint32_t num_key_ht = 5;
+	uint32_t num_key_vbf = 5;
+	uint32_t num_key_cache = 5;
+
+	const void *key_array[5];
+
+	uint32_t i, j;
+
+	/* same key at most inserted 2*entry_per_bucket times for HT

mode */

Review

Quoted 1 line
(quoted diff) +1 -0
+	for (i = 1; i < 33; i++) {

This 33 can be expressed as 32 + 1 (using macro for 32). Also, add a comment explaining why you are skipping value 0.

...

Quoted 3 lines
test/test/test_member_perf.c new file +0 -0
@@ -0,0 +1,630 @@

...

Quoted 18 lines
test/test/test_member_perf.c new file +17 -0
+static int
+timed_lookups_bulk(struct member_perf_params *params, int type) {
+	unsigned int i, j, k;
+	member_set_t result[BURST_SIZE] = {0};
+	const void *keys_burst[BURST_SIZE];
+	int ret;
+
+	false_data_bulk[type][params->cycle] = 0;
+
+	const uint64_t start_tsc = rte_rdtsc();
+
+	for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+		for (j = 0; j < KEYS_TO_ADD / BURST_SIZE; j++) {
+			for (k = 0; k < BURST_SIZE; k++)
+				keys_burst[k] = keys[j * BURST_SIZE + k];
+
+			ret = rte_member_lookup_bulk(params-
Quoted 1 line

setsum[type],

test/test/test_member_perf.c line 314

Quoted 1 line
test/test/test_member_perf.c new file +1 -0
+				&keys_burst[0],

Using keys_burst directly is equivalent to this, right?

...

Quoted 19 lines
test/test/test_member_perf.c new file +2 -0
+static int
+timed_lookups_multimatch(struct member_perf_params *params, int

type) {

test/test/test_member_perf.c new file +10 -0
+	unsigned int i, j;
+	member_set_t result[RTE_MEMBER_BUCKET_ENTRIES] = {0};
+	int ret;
+	false_data_multi[type][params->cycle] = 0;
+
+	const uint64_t start_tsc = rte_rdtsc();
+
+	for (i = 0; i < NUM_LOOKUPS / KEYS_TO_ADD; i++) {
+		for (j = 0; j < KEYS_TO_ADD; j++) {
+			ret = rte_member_lookup_multi(params-
Quoted 1 line

setsum[type],

test/test/test_member_perf.c new file +1 -0
+				&keys[j], RTE_MEMBER_BUCKET_ENTRIES,

result);

test/test/test_member_perf.c new file +2 -0
+			if (type != CACHE && ret <= 0) {
+				printf("lookup multi has wrong return value

%d,"

test/test/test_member_perf.c line 355

Quoted 3 lines
test/test/test_member_perf.c new file +3 -0
+					"type %d\n", ret, type);
+			}
+			if (result[0] != data[type][j])

Why using always result[0]? A comment would be good.

#55 lore raw

PATCH v4 7/7 doc: add membership documentation

In reply to Yipeng Wang

This patch adds the documentation for membership library.

---
 doc/api/doxy-api-index.md               |    3 +-
 doc/api/doxy-api.conf                   |    1 +
 doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg |   36 +
 doc/guides/prog_guide/img/member_i3.svg |  148 +++
 doc/guides/prog_guide/img/member_i4.svg |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg |  163 ++++
 doc/guides/prog_guide/img/member_i6.svg |  332 +++++++
 doc/guides/prog_guide/img/member_i7.svg |  399 ++++++++
 doc/guides/prog_guide/index.rst         |   14 +
 doc/guides/prog_guide/member_lib.rst    |  420 ++++++++
 doc/guides/rel_notes/release_17_11.rst  |   17 +
 12 files changed, 3595 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
doc/api/doxy-api-index.md +2 -1
index 19e0d4f..fe87e09 100644
@@ -105,7 +105,8 @@ The public API headers are grouped by topics:
   [LPM IPv4 route]     (@ref rte_lpm.h),
   [LPM IPv6 route]     (@ref rte_lpm6.h),
   [ACL]                (@ref rte_acl.h),
-  [EFD]                (@ref rte_efd.h)
+  [EFD]                (@ref rte_efd.h),
+  [member]             (@ref rte_member.h)
 
 - **QoS**:
   [metering]           (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
index 823554f..b792d6d 100644
@@ -58,6 +58,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_mempool \
                           lib/librte_meter \
                           lib/librte_metrics \
+                          lib/librte_member \
                           lib/librte_net \
                           lib/librte_pdump \
                           lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0

1,613 lines not on this page · raw message

doc/guides/prog_guide/img/member_i2.svg new file +36 -0
new file mode 100644
index 0000000..759c654
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in"
+		viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+	<v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {fill:none;stroke:none;stroke-width:0.25}
+		.st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+		.st3 {baseline-shift:32.4943%;font-size:0.649886em}
+		.st4 {font-size:1em}
+		.st5 {font-family:Cambria Math;font-size:1em}
+		.st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+		<g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)">
+			<title>Sheet.3</title>
+			<desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc>
+			<v:textBlock v:margins="rect(4,4,4,4)"/>
+			<v:textRect cx="157.5" cy="45.5" width="315" height="90"/>
+			<rect x="0" y="0.5" width="315" height="90" class="st1"/>
+			<text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan
+						class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3"
+						v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em"
+						class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text>		</g>
+	</g>
+</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0

148 lines not on this page · raw message

doc/guides/prog_guide/img/member_i4.svg new file +450 -0

450 lines not on this page · raw message

doc/guides/prog_guide/img/member_i5.svg new file +163 -0

163 lines not on this page · raw message

doc/guides/prog_guide/img/member_i6.svg new file +332 -0

332 lines not on this page · raw message

doc/guides/prog_guide/img/member_i7.svg new file +399 -0

242 lines not on this page · raw message

doc/guides/prog_guide/index.rst +14 -0
index 40f04a1..7ff5144 100644

Lines of this file are not shown. View the raw message.

doc/guides/prog_guide/member_lib.rst new file +420 -0

420 lines not on this page · raw message

doc/guides/rel_notes/release_17_11.rst +17 -0
index 8bf91bd..ba068ec 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.7.4

Part of the diff is not shown here. View the raw message.

#56 lore raw

v5 cover Add Membership Library

In reply to Yipeng Wang

DPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.

A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

v5:
- member lib: incorperate Pablo's review comments, including: multiple places with coding style issues, change variable/function names to be more clear, reorder code blocks, add descriptions, use macros for constant numbers, fix bugs in log message, etc.
- test: modify the deletion function test to cover more cases.
- test: add check in performance test to check false negatvie for HT non-cache mode.
- MAINTAINERS: complete in multiple commits instead of one as Pablo suggested.

v4:
- member lib: change two hash function macros to be one. crc or jhash will be chosen depending on the architecture.
- member lib: use dynamic log type instead of static one by Thomas' comment.
- member lib: code cleanups (remove unnecessary code, change variable name, coding style, etc).
- member lib: reorder members in setsummary struct to be more cache friendly.
- member lib: setsummary struct member "name" should be char array rather than char pointer. Fixed.
- member lib: add check for two hash seeds to make sure they are not equal.
- member lib: vBF lookup speed optimization.
- member lib: revise comments in various places as Pablo suggested.
- member lib: change "void *" to "struct rte_member_setsum *" in public API as Pablo suggested. test case is modified accordingly to directly use the struct.
- member lib: change order in rte_member_version.map as Pablo suggested.
- MAINTAINERS: change the maintainer block order according to Thomas' comment.
- test: add printf to show which section expects error messages. Change set count to 16 from 32 for performance test.
- documentation: revise several places according to Pablo's and John's comments.

v3:
- documentation: update documentation to incorporate John's review.

v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitialized issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment.
- documentation: update the programmer guide for membership library.

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

 MAINTAINERS                              |    8 +-
 config/common_base                       |    5 +
 doc/api/doxy-api-index.md                |    3 +-
 doc/api/doxy-api.conf                    |    1 +
 doc/guides/prog_guide/img/member_i1.svg  | 1613 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg  |   36 +
 doc/guides/prog_guide/img/member_i3.svg  |  148 +++
 doc/guides/prog_guide/img/member_i4.svg  |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg  |  163 +++
 doc/guides/prog_guide/img/member_i6.svg  |  332 ++++++
 doc/guides/prog_guide/img/member_i7.svg  |  399 ++++++++
 doc/guides/prog_guide/index.rst          |   14 +
 doc/guides/prog_guide/member_lib.rst     |  420 ++++++++
 doc/guides/rel_notes/release_17_11.rst   |   17 +
 lib/Makefile                             |    2 +
 lib/librte_member/Makefile               |   51 +
 lib/librte_member/rte_member.c           |  337 +++++++
 lib/librte_member/rte_member.h           |  513 ++++++++++
 lib/librte_member/rte_member_ht.c        |  586 +++++++++++
 lib/librte_member/rte_member_ht.h        |   94 ++
 lib/librte_member/rte_member_vbf.c       |  351 +++++++
 lib/librte_member/rte_member_vbf.h       |   82 ++
 lib/librte_member/rte_member_version.map |   16 +
 lib/librte_member/rte_member_x86.h       |  107 ++
 mk/rte.app.mk                            |    2 +
 test/test/Makefile                       |    3 +
 test/test/test_member.c                  |  744 ++++++++++++++
 test/test/test_member_perf.c             |  654 ++++++++++++
 28 files changed, 7149 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
 create mode 100644 lib/librte_member/rte_member_version.map
 create mode 100644 lib/librte_member/rte_member_x86.h
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
#57 lore raw

PATCH v5 1/7 member: implement main API

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter and cuckoo filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.

This patch adds the main API definition.

---
 lib/Makefile                             |   2 +
 lib/librte_member/Makefile               |  49 +++
 lib/librte_member/rte_member.c           | 337 ++++++++++++++++++++
 lib/librte_member/rte_member.h           | 513 +++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  16 +
 5 files changed, 917 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
index 86caba1..c82033a 100644
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
 DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
 DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_member/Makefile new file +49 -0

49 lines not on this page · raw message

lib/librte_member/rte_member.c new file +337 -0

337 lines not on this page · raw message

lib/librte_member/rte_member.h new file +513 -0

513 lines not on this page · raw message

lib/librte_member/rte_member_version.map new file +16 -0
new file mode 100644
index 0000000..019e4cd
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+	global:
+
+	rte_member_add;
+	rte_member_create;
+	rte_member_delete;
+	rte_member_find_existing;
+	rte_member_free;
+	rte_member_lookup;
+	rte_member_lookup_bulk;
+	rte_member_lookup_multi;
+	rte_member_lookup_multi_bulk;
+	rte_member_reset;
+
+	local: *;
+};
Signature · 1 line
2.7.4
Reviewed-by on v5 1/7
#58 lore raw

Re: [PATCH v5 1/7] member: implement main API

Quoted 20 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 1/7] member: implement main API

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter and cuckoo filter) structure.
In general, the Membership library is a data structure that provides a "set- summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

Quoted 8 lines

The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.

This patch adds the main API definition.

#59 lore raw

PATCH v5 2/7 member: implement HT mode

In reply to Yipeng Wang

One of the set-summary structures is hash-table based set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is similar to the cuckoo filter [1]. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. Thus, the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.

This patch adds the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

---
 lib/librte_member/Makefile        |   2 +-
 lib/librte_member/rte_member_ht.c | 506 ++++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_ht.h |  94 +++++++
 3 files changed, 601 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
index 1a79eaa..ad26548 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_ht.c new file +506 -0

506 lines not on this page · raw message

lib/librte_member/rte_member_ht.h new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
2.7.4
Reviewed-by on v5 2/7
#60 lore raw

Re: [PATCH v5 2/7] member: implement HT mode

Hi Yipeng,

Quoted 32 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 2/7] member: implement HT mode

One of the set-summary structures is hash-table based set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is similar to the cuckoo filter [1].
When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. Thus, the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.

Quoted 7 lines

This patch adds the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

...

Just a comment below. The rest looks fine to me (and thanks for the explanations in the v4 patch, all clear), so keep my "Reviewed-by" in next version.

Quoted 1 line

+++ b/lib/librte_member/rte_member_ht.c

...

lib/librte_member/rte_member_ht.c line 49

Quoted 7 lines
lib/librte_member/rte_member_ht.c new file +7 -0
+/* Search bucket for entry with tmp_sig and update set_id */ static
+inline int update_entry_search(uint32_t bucket_id, member_sig_t
+tmp_sig,
+		struct member_ht_bucket *buckets,
+		member_set_t set_id)
+{
+	int32_t i;

"i" cannot take negative values, so use uint32_t (same in "search_bucket_single" and "search_bucket_multi").

Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10 -0
+
+	for (i = 0; i < RTE_MEMBER_BUCKET_ENTRIES; i++) {
+		if (buckets[bucket_id].sigs[i] == tmp_sig) {
+			buckets[bucket_id].sets[i] = set_id;
+			return 1;
+		}
+	}
+	return 0;
+}
+
#61 lore raw

PATCH v5 3/7 member: implement vBF mode

In reply to Yipeng Wang

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

---
 lib/librte_member/Makefile         |   2 +-
 lib/librte_member/rte_member_vbf.c | 351 +++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_vbf.h |  82 +++++++++
 3 files changed, 434 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
index ad26548..50275ed 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c rte_member_vbf.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_vbf.c new file +351 -0

351 lines not on this page · raw message

lib/librte_member/rte_member_vbf.h new file +82 -0

82 lines not on this page · raw message

Signature · 1 line
2.7.4
Reviewed-by on v5 3/7
#62 lore raw

Re: [PATCH v5 3/7] member: implement vBF mode

Hi Yipeng,

Quoted 21 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 3/7] member: implement vBF mode

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

Quoted 6 lines

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

Just a comment below. The rest looks fine to me (and thanks for the explanations in the v4 patch, all clear), so keep my "Reviewed-by" in next version.

...

Quoted 1 line

+++ b/lib/librte_member/rte_member_vbf.c

...

lib/librte_member/rte_member_vbf.c line 143

Quoted 8 lines
lib/librte_member/rte_member_vbf.c new file +8 -0
+	/*
+	 * To avoid multiplication and division:
+	 * mul_shift is used for multiplication shift during bit test
+	 * div_shift is used for division shift, to be divided by number of bits
+	 * represented by a uint32_t variable
+	 */
+	 ss->mul_shift = __builtin_ctzl(ss->num_set);
+	 ss->div_shift = __builtin_ctzl(32 >> ss->mul_shift);

Remove the whitespace added after the tab in the two lines above.

#63 lore raw

PATCH v5 4/7 member: add AVX for HT mode

In reply to Yipeng Wang

For key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contiguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

---
 lib/librte_member/rte_member_ht.c  | 142 +++++++++++++++++++++++++++++--------
 lib/librte_member/rte_member_x86.h | 107 ++++++++++++++++++++++++++++
 2 files changed, 218 insertions(+), 31 deletions(-)
 create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -31
index 08cce02..2b37bda 100644
@@ -40,6 +40,10 @@
 #include "rte_member.h"
 #include "rte_member_ht.h"
 
+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+
 /* Search bucket for entry with tmp_sig and update set_id */
 static inline int
 update_entry_search(uint32_t bucket_id, member_sig_t tmp_sig,
@@ -136,6 +140,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
 		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
 			buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
 	}
+#if defined(RTE_ARCH_X86)
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+			RTE_MEMBER_BUCKET_ENTRIES == 16)
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+	else
+#endif
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;
 
 	RTE_MEMBER_LOG(DEBUG, "Hash table based filter created, "
 			"the table has %u entries, %u buckets\n",
@@ -193,11 +204,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
 	*set_id = RTE_MEMBER_NO_MATCH;
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	if (search_bucket_single(prim_bucket, tmp_sig, buckets,
-			set_id) ||
-			search_bucket_single(sec_bucket, tmp_sig,
-				buckets, set_id))
-		return 1;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single_avx(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+		break;
+#endif
+	default:
+		if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+	}
 
 	return 0;
 }
@@ -221,13 +244,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
 	}
 
 	for (i = 0; i < num_keys; i++) {
-		if (search_bucket_single(prim_buckets[i], tmp_sig[i],
-				buckets, &set_id[i]) ||
-				search_bucket_single(sec_buckets[i],
-				tmp_sig[i], buckets, &set_id[i]))
-			num_matches++;
-		else
-			set_id[i] = RTE_MEMBER_NO_MATCH;
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			if (search_bucket_single_avx(prim_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]) ||
+				search_bucket_single_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				num_matches++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+			break;
+#endif
+		default:
+			if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+					buckets, &set_id[i]) ||
+					search_bucket_single(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				num_matches++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+		}
 	}
 	return num_matches;
 }
@@ -244,12 +281,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
 
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	search_bucket_multi(prim_bucket, tmp_sig, buckets, &num_matches,
-			 match_per_key, set_id);
-	if (num_matches < match_per_key)
-		search_bucket_multi(sec_bucket, tmp_sig,
-			buckets, &num_matches, match_per_key, set_id);
-	return num_matches;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+			&num_matches, match_per_key, set_id);
+		if (num_matches < match_per_key)
+			search_bucket_multi_avx(sec_bucket, tmp_sig,
+				buckets, &num_matches, match_per_key, set_id);
+		return num_matches;
+#endif
+	default:
+		search_bucket_multi(prim_bucket, tmp_sig, buckets, &num_matches,
+				 match_per_key, set_id);
+		if (num_matches < match_per_key)
+			search_bucket_multi(sec_bucket, tmp_sig,
+				buckets, &num_matches, match_per_key, set_id);
+		return num_matches;
+	}
 }
 
 uint32_t
@@ -275,16 +324,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
 	for (i = 0; i < num_keys; i++) {
 		match_cnt_tmp = 0;
 
-		search_bucket_multi(prim_buckets[i], tmp_sig[i],
-			buckets, &match_cnt_tmp, match_per_key,
-			&set_ids[i*match_per_key]);
-		if (match_cnt_tmp < match_per_key)
-			search_bucket_multi(sec_buckets[i], tmp_sig[i],
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
 				buckets, &match_cnt_tmp, match_per_key,
 				&set_ids[i*match_per_key]);
-		match_count[i] = match_cnt_tmp;
-		if (match_cnt_tmp != 0)
-			num_matches++;
+			if (match_cnt_tmp < match_per_key)
+				search_bucket_multi_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &match_cnt_tmp,
+					match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_tmp;
+			if (match_cnt_tmp != 0)
+				num_matches++;
+			break;
+#endif
+		default:
+			search_bucket_multi(prim_buckets[i], tmp_sig[i],
+				buckets, &match_cnt_tmp, match_per_key,
+				&set_ids[i*match_per_key]);
+			if (match_cnt_tmp < match_per_key)
+				search_bucket_multi(sec_buckets[i], tmp_sig[i],
+					buckets, &match_cnt_tmp, match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_tmp;
+			if (match_cnt_tmp != 0)
+				num_matches++;
+		}
 	}
 	return num_matches;
 }
@@ -315,11 +382,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
 
 static inline int
 try_update(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
-		member_sig_t sig, member_set_t set_id)
+		member_sig_t sig, member_set_t set_id,
+		enum rte_member_sig_compare_function cmp_fn)
 {
-	if (update_entry_search(prim, sig, buckets, set_id) ||
-		update_entry_search(sec, sig, buckets, set_id))
-		return 0;
+	switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (update_entry_search_avx(prim, sig, buckets, set_id) ||
+				update_entry_search_avx(sec, sig, buckets,
+					set_id))
+			return 0;
+		break;
+#endif
+	default:
+		if (update_entry_search(prim, sig, buckets, set_id) ||
+				update_entry_search(sec, sig, buckets,
+					set_id))
+			return 0;
+	}
 	return -1;
 }
 
@@ -430,7 +510,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
 	 */
 	if (ss->cache) {
 		ret = try_update(buckets, prim_bucket, sec_bucket, tmp_sig,
-					set_id);
+					set_id, ss->sig_cmp_fn);
 		if (ret != -1)
 			return ret;
 	}
lib/librte_member/rte_member_x86.h new file +107 -0

107 lines not on this page · raw message

Signature · 1 line
2.7.4
Reviewed-by on v5 4/7
#64 lore raw

Re: [PATCH v5 4/7] member: add AVX for HT mode

Quoted 12 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 4/7] member: add AVX for HT mode

Quoted 8 lines

For key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contiguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

#65 lore raw

PATCH v5 5/7 member: enable the library

In reply to Yipeng Wang

This patch enables the Membership library.

---
 MAINTAINERS                | 6 +++++-
 config/common_base         | 5 +++++
 lib/librte_member/Makefile | 2 ++
 mk/rte.app.mk              | 2 ++
 4 files changed, 14 insertions(+), 1 deletion(-)
MAINTAINERS +5 -1
index a0cd75e..318d892 100644
@@ -726,6 +726,11 @@ F: test/test/test_lpm*
 F: test/test/test_func_reentrancy.c
 F: test/test/test_xmmt_ops.h
 
+Membership - EXPERIMENTAL
+M: Yipeng Wang <[email protected]>
+M: Sameh Gobriel <[email protected]>
+F: lib/librte_member/
+
 Traffic metering
 M: Cristian Dumitrescu <[email protected]>
 F: lib/librte_meter/
@@ -734,7 +739,6 @@ F: test/test/test_meter.c
 F: examples/qos_meter/
 F: doc/guides/sample_app_ug/qos_metering.rst
 
-
 Other libraries
 ---------------
 
config/common_base +5 -0
index 5e97a08..5e31ced 100644
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
 CONFIG_RTE_LIBRTE_EFD=y
 
 #
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
 # Compile librte_jobstats
 #
 CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
index 50275ed..3bac1d0 100644
@@ -37,6 +37,8 @@ LIB = librte_member.a
 CFLAGS := -I$(SRCDIR) $(CFLAGS)
 CFLAGS += $(WERROR_FLAGS) -O3
 
+LDLIBS += -lm
+
 EXPORT_MAP := rte_member_version.map
 
 LIBABIVER := 1
mk/rte.app.mk +2 -0
index c25fdd9..c79acf0 100644
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE)        += -lrte_cfgfile
 _LDLIBS-y += --whole-archive
 
 _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH)           += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)        += -lrte_member
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lrte_vhost
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS)         += -lrte_kvargs
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lm
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrt
 _LDLIBS-$(CONFIG_RTE_LIBRTE_METER)          += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)          += -lm
 ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lnuma
 endif
Signature · 1 line
2.7.4
Reviewed-by on v5 5/7
#66 lore raw

Re: [PATCH v5 5/7] member: enable the library

Quoted 12 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 5/7] member: enable the library

Quoted 3 lines

This patch enables the Membership library.

#67 lore raw

PATCH v5 6/7 test/member: add functional and perf tests

In reply to Yipeng Wang

This patch adds functional and performance tests for membership library.

---
 MAINTAINERS                  |   1 +
 test/test/Makefile           |   3 +
 test/test/test_member.c      | 744 +++++++++++++++++++++++++++++++++++++++++++
 test/test/test_member_perf.c | 654 +++++++++++++++++++++++++++++++++++++
 4 files changed, 1402 insertions(+)
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
MAINTAINERS +1 -0
index 318d892..26b2374 100644
@@ -730,6 +730,7 @@ Membership - EXPERIMENTAL
 M: Yipeng Wang <[email protected]>
 M: Sameh Gobriel <[email protected]>
 F: lib/librte_member/
+F: test/test/test_member*
 
 Traffic metering
 M: Cristian Dumitrescu <[email protected]>
test/test/Makefile +3 -0
index 42d9a49..b61dde3 100644
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
 SRCS-y += test_memcpy.c
 SRCS-y += test_memcpy_perf.c
 
+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
 
test/test/test_member.c new file +744 -0

744 lines not on this page · raw message

test/test/test_member_perf.c new file +654 -0

654 lines not on this page · raw message

Signature · 1 line
2.7.4
Reviewed-by on v5 6/7
#68 lore raw

Re: [PATCH v5 6/7] test/member: add functional and perf tests

Quoted 10 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 6/7] test/member: add functional and perf tests

Quoted 3 lines

This patch adds functional and performance tests for membership library.

#69 lore raw

PATCH v5 7/7 doc: add membership documentation

In reply to Yipeng Wang

This patch adds the documentation for membership library.

---
 MAINTAINERS                             |    1 +
 doc/api/doxy-api-index.md               |    3 +-
 doc/api/doxy-api.conf                   |    1 +
 doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg |   36 +
 doc/guides/prog_guide/img/member_i3.svg |  148 +++
 doc/guides/prog_guide/img/member_i4.svg |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg |  163 ++++
 doc/guides/prog_guide/img/member_i6.svg |  332 +++++++
 doc/guides/prog_guide/img/member_i7.svg |  399 ++++++++
 doc/guides/prog_guide/index.rst         |   14 +
 doc/guides/prog_guide/member_lib.rst    |  420 ++++++++
 doc/guides/rel_notes/release_17_11.rst  |   17 +
 13 files changed, 3596 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
MAINTAINERS +1 -0
index 26b2374..2d57d3b 100644
@@ -731,6 +731,7 @@ M: Yipeng Wang <[email protected]>
 M: Sameh Gobriel <[email protected]>
 F: lib/librte_member/
 F: test/test/test_member*
+F: doc/guides/prog_guide/member_lib.rst
 
 Traffic metering
 M: Cristian Dumitrescu <[email protected]>
doc/api/doxy-api-index.md +2 -1
index 19e0d4f..fe87e09 100644
@@ -105,7 +105,8 @@ The public API headers are grouped by topics:
   [LPM IPv4 route]     (@ref rte_lpm.h),
   [LPM IPv6 route]     (@ref rte_lpm6.h),
   [ACL]                (@ref rte_acl.h),
-  [EFD]                (@ref rte_efd.h)
+  [EFD]                (@ref rte_efd.h),
+  [member]             (@ref rte_member.h)
 
 - **QoS**:
   [metering]           (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
index 823554f..b792d6d 100644
@@ -58,6 +58,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_mempool \
                           lib/librte_meter \
                           lib/librte_metrics \
+                          lib/librte_member \
                           lib/librte_net \
                           lib/librte_pdump \
                           lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0

1,613 lines not on this page · raw message

doc/guides/prog_guide/img/member_i2.svg new file +36 -0
new file mode 100644
index 0000000..759c654
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in"
+		viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+	<v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {fill:none;stroke:none;stroke-width:0.25}
+		.st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+		.st3 {baseline-shift:32.4943%;font-size:0.649886em}
+		.st4 {font-size:1em}
+		.st5 {font-family:Cambria Math;font-size:1em}
+		.st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+		<g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)">
+			<title>Sheet.3</title>
+			<desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc>
+			<v:textBlock v:margins="rect(4,4,4,4)"/>
+			<v:textRect cx="157.5" cy="45.5" width="315" height="90"/>
+			<rect x="0" y="0.5" width="315" height="90" class="st1"/>
+			<text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan
+						class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3"
+						v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em"
+						class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text>		</g>
+	</g>
+</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0

148 lines not on this page · raw message

doc/guides/prog_guide/img/member_i4.svg new file +450 -0

450 lines not on this page · raw message

doc/guides/prog_guide/img/member_i5.svg new file +163 -0

163 lines not on this page · raw message

doc/guides/prog_guide/img/member_i6.svg new file +332 -0

332 lines not on this page · raw message

doc/guides/prog_guide/img/member_i7.svg new file +399 -0

235 lines not on this page · raw message

doc/guides/prog_guide/index.rst +14 -0
index 40f04a1..7ff5144 100644

Lines of this file are not shown. View the raw message.

doc/guides/prog_guide/member_lib.rst new file +420 -0

420 lines not on this page · raw message

doc/guides/rel_notes/release_17_11.rst +17 -0
index 8bf91bd..ba068ec 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.7.4

Part of the diff is not shown here. View the raw message.

Reviewed-by on v5 7/7
#70 lore raw

Re: [PATCH v5 7/7] doc: add membership documentation

Quoted 10 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 7/7] doc: add membership documentation

Quoted 4 lines

This patch adds the documentation for membership library.

#71 lore raw

v6 cover Add Membership Library

In reply to Yipeng Wang

DPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).

The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.

The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.

A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.

This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.

[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.

[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

v6:
- member lib: minor coding style changes from Pablo's comment and checkpatch report.
- member lib: minor log message changes.

v5:
- member lib: incorperate Pablo's review comments, including: multiple places with coding style issues, change variable/function names to be more clear, reorder code blocks, add descriptions, use macros for constant numbers, fix bugs in log message, etc.
- test: modify the deletion function test to cover more cases.
- test: add check in performance test to check false negatvie for HT non-cache mode.
- MAINTAINERS: complete in multiple commits instead of one as Pablo suggested.

v4:
- member lib: change two hash function macros to be one. crc or jhash will be chosen depending on the architecture.
- member lib: use dynamic log type instead of static one by Thomas' comment.
- member lib: code cleanups (remove unnecessary code, change variable name, coding style, etc).
- member lib: reorder members in setsummary struct to be more cache friendly.
- member lib: setsummary struct member "name" should be char array rather than char pointer. Fixed.
- member lib: add check for two hash seeds to make sure they are not equal.
- member lib: vBF lookup speed optimization.
- member lib: revise comments in various places as Pablo suggested.
- member lib: change "void *" to "struct rte_member_setsum *" in public API as Pablo suggested. test case is modified accordingly to directly use the struct.
- member lib: change order in rte_member_version.map as Pablo suggested.
- MAINTAINERS: change the maintainer block order according to Thomas' comment.
- test: add printf to show which section expects error messages. Change set count to 16 from 32 for performance test.
- documentation: revise several places according to Pablo's and John's comments.

v3:
- documentation: update documentation to incorporate John's review.

v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitialized issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment.
- documentation: update the programmer guide for membership library.

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

 MAINTAINERS                              |    8 +-
 config/common_base                       |    5 +
 doc/api/doxy-api-index.md                |    3 +-
 doc/api/doxy-api.conf                    |    1 +
 doc/guides/prog_guide/img/member_i1.svg  | 1613 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg  |   36 +
 doc/guides/prog_guide/img/member_i3.svg  |  148 +++
 doc/guides/prog_guide/img/member_i4.svg  |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg  |  163 +++
 doc/guides/prog_guide/img/member_i6.svg  |  332 ++++++
 doc/guides/prog_guide/img/member_i7.svg  |  399 ++++++++
 doc/guides/prog_guide/index.rst          |   14 +
 doc/guides/prog_guide/member_lib.rst     |  420 ++++++++
 doc/guides/rel_notes/release_17_11.rst   |   17 +
 lib/Makefile                             |    2 +
 lib/librte_member/Makefile               |   51 +
 lib/librte_member/rte_member.c           |  337 +++++++
 lib/librte_member/rte_member.h           |  513 ++++++++++
 lib/librte_member/rte_member_ht.c        |  586 +++++++++++
 lib/librte_member/rte_member_ht.h        |   94 ++
 lib/librte_member/rte_member_vbf.c       |  350 +++++++
 lib/librte_member/rte_member_vbf.h       |   82 ++
 lib/librte_member/rte_member_version.map |   16 +
 lib/librte_member/rte_member_x86.h       |  107 ++
 mk/rte.app.mk                            |    2 +
 test/test/Makefile                       |    3 +
 test/test/test_member.c                  |  744 ++++++++++++++
 test/test/test_member_perf.c             |  654 ++++++++++++
 28 files changed, 7148 insertions(+), 2 deletions(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
 create mode 100644 lib/librte_member/rte_member_version.map
 create mode 100644 lib/librte_member/rte_member_x86.h
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
#72 lore raw

PATCH v6 1/7 member: implement main API

Membership library is an extension and generalization of a traditional filter (for example Bloom Filter and cuckoo filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.

The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.

This patch adds the main API definition.

---
 lib/Makefile                             |   2 +
 lib/librte_member/Makefile               |  49 +++
 lib/librte_member/rte_member.c           | 337 ++++++++++++++++++++
 lib/librte_member/rte_member.h           | 513 +++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_version.map |  16 +
 5 files changed, 917 insertions(+)
 create mode 100644 lib/librte_member/Makefile
 create mode 100644 lib/librte_member/rte_member.c
 create mode 100644 lib/librte_member/rte_member.h
 create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
index 86caba1..c82033a 100644
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder
 DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf
 DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump
 DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether
+DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member
+DEPDIRS-librte_member := librte_eal librte_hash
 
 ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y)
 DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_member/Makefile new file +49 -0

49 lines not on this page · raw message

lib/librte_member/rte_member.c new file +337 -0

337 lines not on this page · raw message

lib/librte_member/rte_member.h new file +513 -0

513 lines not on this page · raw message

lib/librte_member/rte_member_version.map new file +16 -0
new file mode 100644
index 0000000..019e4cd
@@ -0,0 +1,16 @@
+DPDK_17.11 {
+	global:
+
+	rte_member_add;
+	rte_member_create;
+	rte_member_delete;
+	rte_member_find_existing;
+	rte_member_free;
+	rte_member_lookup;
+	rte_member_lookup_bulk;
+	rte_member_lookup_multi;
+	rte_member_lookup_multi_bulk;
+	rte_member_reset;
+
+	local: *;
+};
Signature · 1 line
2.7.4
#73 lore raw

PATCH v6 2/7 member: implement HT mode

In reply to Yipeng Wang

One of the set-summary structures is hash-table based set-summary (HTSS). One example is cuckoo filter [1].

Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.

Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is similar to the cuckoo filter [1]. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.

The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. Thus, the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.

This patch adds the implementation of HTSS.

[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.

---
 lib/librte_member/Makefile        |   2 +-
 lib/librte_member/rte_member_ht.c | 506 ++++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_ht.h |  94 +++++++
 3 files changed, 601 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_ht.c
 create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
index 1a79eaa..ad26548 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_ht.c new file +506 -0

506 lines not on this page · raw message

lib/librte_member/rte_member_ht.h new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
2.7.4
#74 lore raw

PATCH v6 3/7 member: implement vBF mode

In reply to Yipeng Wang

Bloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.

Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.

This patch adds the vBF implementation.

[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.

---
 lib/librte_member/Makefile         |   2 +-
 lib/librte_member/rte_member_vbf.c | 350 +++++++++++++++++++++++++++++++++++++
 lib/librte_member/rte_member_vbf.h |  82 +++++++++
 3 files changed, 433 insertions(+), 1 deletion(-)
 create mode 100644 lib/librte_member/rte_member_vbf.c
 create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
index ad26548..50275ed 100644
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map
 LIBABIVER := 1
 
 # all source are stored in SRCS-y
-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) +=  rte_member.c rte_member_ht.c rte_member_vbf.c
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
 
lib/librte_member/rte_member_vbf.c new file +350 -0

350 lines not on this page · raw message

lib/librte_member/rte_member_vbf.h new file +82 -0

82 lines not on this page · raw message

Signature · 1 line
2.7.4
#75 lore raw

PATCH v6 4/7 member: add AVX for HT mode

In reply to Yipeng Wang

For key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contiguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.

This patch adds AVX2 implementation in a separate header file.

---
 lib/librte_member/rte_member_ht.c  | 142 +++++++++++++++++++++++++++++--------
 lib/librte_member/rte_member_x86.h | 107 ++++++++++++++++++++++++++++
 2 files changed, 218 insertions(+), 31 deletions(-)
 create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -31
index e038987..59332d5 100644
@@ -40,6 +40,10 @@
 #include "rte_member.h"
 #include "rte_member_ht.h"
 
+#if defined(RTE_ARCH_X86)
+#include "rte_member_x86.h"
+#endif
+
 /* Search bucket for entry with tmp_sig and update set_id */
 static inline int
 update_entry_search(uint32_t bucket_id, member_sig_t tmp_sig,
@@ -136,6 +140,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,
 		for (j = 0; j < RTE_MEMBER_BUCKET_ENTRIES; j++)
 			buckets[i].sets[j] = RTE_MEMBER_NO_MATCH;
 	}
+#if defined(RTE_ARCH_X86)
+	if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2) &&
+			RTE_MEMBER_BUCKET_ENTRIES == 16)
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_AVX2;
+	else
+#endif
+		ss->sig_cmp_fn = RTE_MEMBER_COMPARE_SCALAR;
 
 	RTE_MEMBER_LOG(DEBUG, "Hash table based filter created, "
 			"the table has %u entries, %u buckets\n",
@@ -193,11 +204,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,
 	*set_id = RTE_MEMBER_NO_MATCH;
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	if (search_bucket_single(prim_bucket, tmp_sig, buckets,
-			set_id) ||
-			search_bucket_single(sec_bucket, tmp_sig,
-				buckets, set_id))
-		return 1;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (search_bucket_single_avx(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single_avx(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+		break;
+#endif
+	default:
+		if (search_bucket_single(prim_bucket, tmp_sig, buckets,
+				set_id) ||
+				search_bucket_single(sec_bucket, tmp_sig,
+					buckets, set_id))
+			return 1;
+	}
 
 	return 0;
 }
@@ -221,13 +244,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss,
 	}
 
 	for (i = 0; i < num_keys; i++) {
-		if (search_bucket_single(prim_buckets[i], tmp_sig[i],
-				buckets, &set_id[i]) ||
-				search_bucket_single(sec_buckets[i],
-				tmp_sig[i], buckets, &set_id[i]))
-			num_matches++;
-		else
-			set_id[i] = RTE_MEMBER_NO_MATCH;
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			if (search_bucket_single_avx(prim_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]) ||
+				search_bucket_single_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				num_matches++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+			break;
+#endif
+		default:
+			if (search_bucket_single(prim_buckets[i], tmp_sig[i],
+					buckets, &set_id[i]) ||
+					search_bucket_single(sec_buckets[i],
+					tmp_sig[i], buckets, &set_id[i]))
+				num_matches++;
+			else
+				set_id[i] = RTE_MEMBER_NO_MATCH;
+		}
 	}
 	return num_matches;
 }
@@ -244,12 +281,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,
 
 	get_buckets_index(ss, key, &prim_bucket, &sec_bucket, &tmp_sig);
 
-	search_bucket_multi(prim_bucket, tmp_sig, buckets, &num_matches,
-			 match_per_key, set_id);
-	if (num_matches < match_per_key)
-		search_bucket_multi(sec_bucket, tmp_sig,
-			buckets, &num_matches, match_per_key, set_id);
-	return num_matches;
+	switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		search_bucket_multi_avx(prim_bucket, tmp_sig, buckets,
+			&num_matches, match_per_key, set_id);
+		if (num_matches < match_per_key)
+			search_bucket_multi_avx(sec_bucket, tmp_sig,
+				buckets, &num_matches, match_per_key, set_id);
+		return num_matches;
+#endif
+	default:
+		search_bucket_multi(prim_bucket, tmp_sig, buckets, &num_matches,
+				 match_per_key, set_id);
+		if (num_matches < match_per_key)
+			search_bucket_multi(sec_bucket, tmp_sig,
+				buckets, &num_matches, match_per_key, set_id);
+		return num_matches;
+	}
 }
 
 uint32_t
@@ -275,16 +324,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,
 	for (i = 0; i < num_keys; i++) {
 		match_cnt_tmp = 0;
 
-		search_bucket_multi(prim_buckets[i], tmp_sig[i],
-			buckets, &match_cnt_tmp, match_per_key,
-			&set_ids[i*match_per_key]);
-		if (match_cnt_tmp < match_per_key)
-			search_bucket_multi(sec_buckets[i], tmp_sig[i],
+		switch (ss->sig_cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+		case RTE_MEMBER_COMPARE_AVX2:
+			search_bucket_multi_avx(prim_buckets[i], tmp_sig[i],
 				buckets, &match_cnt_tmp, match_per_key,
 				&set_ids[i*match_per_key]);
-		match_count[i] = match_cnt_tmp;
-		if (match_cnt_tmp != 0)
-			num_matches++;
+			if (match_cnt_tmp < match_per_key)
+				search_bucket_multi_avx(sec_buckets[i],
+					tmp_sig[i], buckets, &match_cnt_tmp,
+					match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_tmp;
+			if (match_cnt_tmp != 0)
+				num_matches++;
+			break;
+#endif
+		default:
+			search_bucket_multi(prim_buckets[i], tmp_sig[i],
+				buckets, &match_cnt_tmp, match_per_key,
+				&set_ids[i*match_per_key]);
+			if (match_cnt_tmp < match_per_key)
+				search_bucket_multi(sec_buckets[i], tmp_sig[i],
+					buckets, &match_cnt_tmp, match_per_key,
+					&set_ids[i*match_per_key]);
+			match_count[i] = match_cnt_tmp;
+			if (match_cnt_tmp != 0)
+				num_matches++;
+		}
 	}
 	return num_matches;
 }
@@ -315,11 +382,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
 
 static inline int
 try_update(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,
-		member_sig_t sig, member_set_t set_id)
+		member_sig_t sig, member_set_t set_id,
+		enum rte_member_sig_compare_function cmp_fn)
 {
-	if (update_entry_search(prim, sig, buckets, set_id) ||
-		update_entry_search(sec, sig, buckets, set_id))
-		return 0;
+	switch (cmp_fn) {
+#if defined(RTE_ARCH_X86) && defined(RTE_MACHINE_CPUFLAG_AVX2)
+	case RTE_MEMBER_COMPARE_AVX2:
+		if (update_entry_search_avx(prim, sig, buckets, set_id) ||
+				update_entry_search_avx(sec, sig, buckets,
+					set_id))
+			return 0;
+		break;
+#endif
+	default:
+		if (update_entry_search(prim, sig, buckets, set_id) ||
+				update_entry_search(sec, sig, buckets,
+					set_id))
+			return 0;
+	}
 	return -1;
 }
 
@@ -430,7 +510,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,
 	 */
 	if (ss->cache) {
 		ret = try_update(buckets, prim_bucket, sec_bucket, tmp_sig,
-					set_id);
+					set_id, ss->sig_cmp_fn);
 		if (ret != -1)
 			return ret;
 	}
lib/librte_member/rte_member_x86.h new file +107 -0

107 lines not on this page · raw message

Signature · 1 line
2.7.4
#76 lore raw

PATCH v6 5/7 member: enable the library

In reply to Yipeng Wang

This patch enables the Membership library.

---
 MAINTAINERS                | 6 +++++-
 config/common_base         | 5 +++++
 lib/librte_member/Makefile | 2 ++
 mk/rte.app.mk              | 2 ++
 4 files changed, 14 insertions(+), 1 deletion(-)
MAINTAINERS +5 -1
index 8df2a7f..1358e47 100644
@@ -729,6 +729,11 @@ F: test/test/test_lpm*
 F: test/test/test_func_reentrancy.c
 F: test/test/test_xmmt_ops.h
 
+Membership - EXPERIMENTAL
+M: Yipeng Wang <[email protected]>
+M: Sameh Gobriel <[email protected]>
+F: lib/librte_member/
+
 Traffic metering
 M: Cristian Dumitrescu <[email protected]>
 F: lib/librte_meter/
@@ -737,7 +742,6 @@ F: test/test/test_meter.c
 F: examples/qos_meter/
 F: doc/guides/sample_app_ug/qos_metering.rst
 
-
 Other libraries
 ---------------
 
config/common_base +5 -0
index 12f6be9..e305e9f 100644
@@ -596,6 +596,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n
 CONFIG_RTE_LIBRTE_EFD=y
 
 #
+# Compile librte_member
+#
+CONFIG_RTE_LIBRTE_MEMBER=y
+
+#
 # Compile librte_jobstats
 #
 CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
index 50275ed..3bac1d0 100644
@@ -37,6 +37,8 @@ LIB = librte_member.a
 CFLAGS := -I$(SRCDIR) $(CFLAGS)
 CFLAGS += $(WERROR_FLAGS) -O3
 
+LDLIBS += -lm
+
 EXPORT_MAP := rte_member_version.map
 
 LIBABIVER := 1
mk/rte.app.mk +2 -0
index c25fdd9..c79acf0 100644
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE)        += -lrte_cfgfile
 _LDLIBS-y += --whole-archive
 
 _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH)           += -lrte_hash
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)        += -lrte_member
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lrte_vhost
 _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS)         += -lrte_kvargs
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF)           += -lrte_mbuf
@@ -196,6 +197,7 @@ endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lm
 _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED)          += -lrt
 _LDLIBS-$(CONFIG_RTE_LIBRTE_METER)          += -lm
+_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER)          += -lm
 ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
 _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST)          += -lnuma
 endif
Signature · 1 line
2.7.4
#77 lore raw

PATCH v6 6/7 test/member: add functional and perf tests

In reply to Yipeng Wang

This patch adds functional and performance tests for membership library.

---
 MAINTAINERS                  |   1 +
 test/test/Makefile           |   3 +
 test/test/test_member.c      | 744 +++++++++++++++++++++++++++++++++++++++++++
 test/test/test_member_perf.c | 654 +++++++++++++++++++++++++++++++++++++
 4 files changed, 1402 insertions(+)
 create mode 100644 test/test/test_member.c
 create mode 100644 test/test/test_member_perf.c
MAINTAINERS +1 -0
index 1358e47..afd36cf 100644
@@ -733,6 +733,7 @@ Membership - EXPERIMENTAL
 M: Yipeng Wang <[email protected]>
 M: Sameh Gobriel <[email protected]>
 F: lib/librte_member/
+F: test/test/test_member*
 
 Traffic metering
 M: Cristian Dumitrescu <[email protected]>
test/test/Makefile +3 -0
index 42d9a49..b61dde3 100644
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c
 SRCS-y += test_memcpy.c
 SRCS-y += test_memcpy_perf.c
 
+
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c
+SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c
 SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
 
test/test/test_member.c new file +744 -0

744 lines not on this page · raw message

test/test/test_member_perf.c new file +654 -0

654 lines not on this page · raw message

Signature · 1 line
2.7.4
#78 lore raw

PATCH v6 7/7 doc: add membership documentation

In reply to Yipeng Wang

This patch adds the documentation for membership library.

---
 MAINTAINERS                             |    1 +
 doc/api/doxy-api-index.md               |    3 +-
 doc/api/doxy-api.conf                   |    1 +
 doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++
 doc/guides/prog_guide/img/member_i2.svg |   36 +
 doc/guides/prog_guide/img/member_i3.svg |  148 +++
 doc/guides/prog_guide/img/member_i4.svg |  450 +++++++++
 doc/guides/prog_guide/img/member_i5.svg |  163 ++++
 doc/guides/prog_guide/img/member_i6.svg |  332 +++++++
 doc/guides/prog_guide/img/member_i7.svg |  399 ++++++++
 doc/guides/prog_guide/index.rst         |   14 +
 doc/guides/prog_guide/member_lib.rst    |  420 ++++++++
 doc/guides/rel_notes/release_17_11.rst  |   17 +
 13 files changed, 3596 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/prog_guide/img/member_i1.svg
 create mode 100644 doc/guides/prog_guide/img/member_i2.svg
 create mode 100644 doc/guides/prog_guide/img/member_i3.svg
 create mode 100644 doc/guides/prog_guide/img/member_i4.svg
 create mode 100644 doc/guides/prog_guide/img/member_i5.svg
 create mode 100644 doc/guides/prog_guide/img/member_i6.svg
 create mode 100644 doc/guides/prog_guide/img/member_i7.svg
 create mode 100644 doc/guides/prog_guide/member_lib.rst
MAINTAINERS +1 -0
index afd36cf..51ef958 100644
@@ -734,6 +734,7 @@ M: Yipeng Wang <[email protected]>
 M: Sameh Gobriel <[email protected]>
 F: lib/librte_member/
 F: test/test/test_member*
+F: doc/guides/prog_guide/member_lib.rst
 
 Traffic metering
 M: Cristian Dumitrescu <[email protected]>
doc/api/doxy-api-index.md +2 -1
index 19e0d4f..fe87e09 100644
@@ -105,7 +105,8 @@ The public API headers are grouped by topics:
   [LPM IPv4 route]     (@ref rte_lpm.h),
   [LPM IPv6 route]     (@ref rte_lpm6.h),
   [ACL]                (@ref rte_acl.h),
-  [EFD]                (@ref rte_efd.h)
+  [EFD]                (@ref rte_efd.h),
+  [member]             (@ref rte_member.h)
 
 - **QoS**:
   [metering]           (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
index 823554f..b792d6d 100644
@@ -58,6 +58,7 @@ INPUT                   = doc/api/doxy-api-index.md \
                           lib/librte_mempool \
                           lib/librte_meter \
                           lib/librte_metrics \
+                          lib/librte_member \
                           lib/librte_net \
                           lib/librte_pdump \
                           lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0

1,613 lines not on this page · raw message

doc/guides/prog_guide/img/member_i2.svg new file +36 -0
new file mode 100644
index 0000000..759c654
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 -->
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events"
+		xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in"
+		viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6">
+	<v:documentProperties v:langID="1033" v:viewMarkup="false"/>
+
+	<style type="text/css">
+	<![CDATA[
+		.st1 {fill:none;stroke:none;stroke-width:0.25}
+		.st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em}
+		.st3 {baseline-shift:32.4943%;font-size:0.649886em}
+		.st4 {font-size:1em}
+		.st5 {font-family:Cambria Math;font-size:1em}
+		.st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+	]]>
+	</style>
+
+	<g v:mID="0" v:index="1" v:groupContext="foregroundPage">
+		<title>Page-1</title>
+		<v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/>
+		<g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)">
+			<title>Sheet.3</title>
+			<desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc>
+			<v:textBlock v:margins="rect(4,4,4,4)"/>
+			<v:textRect cx="157.5" cy="45.5" width="315" height="90"/>
+			<rect x="0" y="0.5" width="315" height="90" class="st1"/>
+			<text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan
+						dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan
+						class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3"
+						v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em"
+						class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text>		</g>
+	</g>
+</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0

148 lines not on this page · raw message

doc/guides/prog_guide/img/member_i4.svg new file +450 -0

450 lines not on this page · raw message

doc/guides/prog_guide/img/member_i5.svg new file +163 -0

163 lines not on this page · raw message

doc/guides/prog_guide/img/member_i6.svg new file +332 -0

332 lines not on this page · raw message

doc/guides/prog_guide/img/member_i7.svg new file +399 -0

235 lines not on this page · raw message

doc/guides/prog_guide/index.rst +14 -0
index 40f04a1..7ff5144 100644

Lines of this file are not shown. View the raw message.

doc/guides/prog_guide/member_lib.rst new file +420 -0

420 lines not on this page · raw message

doc/guides/rel_notes/release_17_11.rst +17 -0
index 8bf91bd..ba068ec 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.7.4

Part of the diff is not shown here. View the raw message.

Acked-by on v6 7/7
#79 lore raw

Re: [PATCH v6 7/7] doc: add membership documentation

Quoted 12 lines

-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, October 4, 2017 4:12 AM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v6 7/7] doc: add membership documentation

Quoted 5 lines

This patch adds the documentation for membership library.

#80 lore raw
In reply to Yipeng Wang
Quoted 8 lines

Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation

It looks good.

I have fixed some minor details (alphabetical sorting, alignments), and I have enabled the compilation from the first patch.

Applied, thanks