Add Membership Library
cover Add Membership Library
DPDP Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).
The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.
The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.
A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.
This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.
We sent out RFC previously:
http://dpdk.org/ml/archives/dev/2017-May/066599.html
[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.
[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation
MAINTAINERS | 7 + config/common_base | 5 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 +++ doc/guides/prog_guide/img/member_i6.svg | 332 ++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 432 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 50 + lib/librte_member/rte_member.c | 357 +++++++ lib/librte_member/rte_member.h | 518 ++++++++++ lib/librte_member/rte_member_ht.c | 567 +++++++++++ lib/librte_member/rte_member_ht.h | 115 +++ lib/librte_member/rte_member_vbf.c | 309 ++++++ lib/librte_member/rte_member_vbf.h | 85 ++ lib/librte_member/rte_member_version.map | 15 + lib/librte_member/rte_member_x86.h | 111 ++ mk/rte.app.mk | 2 + test/test/Makefile | 3 + test/test/test_member.c | 550 ++++++++++ test/test/test_member_perf.c | 643 ++++++++++++ 30 files changed, 6948 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h create mode 100644 lib/librte_member/rte_member_version.map create mode 100644 lib/librte_member/rte_member_x86.h create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
PATCH 1/7 member: implement main API
Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.
This patch add the main API definition.
--- lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357 +++++++++++++++++++++ lib/librte_member/rte_member.h | 518 +++++++++++++++++++++++++++++++ lib/librte_member/rte_member_version.map | 15 + 7 files changed, 942 insertions(+) create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether +DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member +DEPDIRS-librte_member := librte_eal librte_hash ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_eal/common/eal_common_log.c +1 -0
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = { {RTE_LOGTYPE_CRYPTODEV, "cryptodev"}, {RTE_LOGTYPE_EFD, "efd"}, {RTE_LOGTYPE_EVENTDEV, "eventdev"}, + {RTE_LOGTYPE_MEMBER, "member"}, {RTE_LOGTYPE_USER1, "user1"}, {RTE_LOGTYPE_USER2, "user2"}, {RTE_LOGTYPE_USER3, "user3"},
lib/librte_eal/common/include/rte_log.h +1 -0
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;#define RTE_LOGTYPE_CRYPTODEV 17/**< Log related to cryptodev. */ #define RTE_LOGTYPE_EFD 18/**< Log related to EFD. */ #define RTE_LOGTYPE_EVENTDEV 19/**< Log related to eventdev. */ +#define RTE_LOGTYPE_MEMBER 20/**< Log related to membership. */ /* these log types can be used in an application */ #define RTE_LOGTYPE_USER1 24/**< User-defined log type 1. */
lib/librte_member/Makefile new file +48 -0
48 lines not on this page · raw message
lib/librte_member/rte_member.c new file +357 -0
357 lines not on this page · raw message
lib/librte_member/rte_member.h new file +518 -0
518 lines not on this page · raw message
lib/librte_member/rte_member_version.map new file +15 -0
@@ -0,0 +1,15 @@ +DPDK_17.11 { + global: + + rte_member_create; + rte_member_lookup; + rte_member_lookup_bulk; + rte_member_lookup_multi; + rte_member_lookup_multi_bulk; + rte_member_add; + rte_member_free; + rte_member_reset; + rte_member_delete; + + local: *; +};
Signature · 1 line
2.7.4
Re: [PATCH 1/7] member: implement main API
lib/librte_member/rte_member.c line 231
Quoted 9 lines · On Mon, 21 Aug 2017 17:19:47 -0700 Yipeng Wang <[email protected]> wrote:
lib/librte_member/rte_member.c new file +9
-0+int +rte_member_lookup (const void *ss,const void *key, +MEMBER_SET_TYPE *set_id) +{ +const struct rte_member_setsum *setsum = ss; +if (setsum ==NULL || key ==NULL || set_id ==NULL ) +return -EINVAL ; + +int ret =0 ;
Don't needlessly initialize all variable.
In all code paths, ret is set.
In fact you could just make all cases of the switch into return and compiler could optimize it as a tail call.
Re: [PATCH 1/7] member: implement main API
In reply to Yipeng WangQuoted 91 lines · Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>---
lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357 +++++++++++++++++++++ lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +7 files changed, 942 insertions(+)create mode 100644
lib/librte_member/Makefilecreate mode 100644lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.maplib/librte_member/Makefile new file +6
-0@@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with orwithout
+# modification, are permitted provided that the following conditionslib/librte_member/Makefile new file +3
-0+# are met: +# +# * Redistributions of source code must retain the abovecopyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer inlib/librte_member/Makefile new file +4
-0+# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote productsderived
+# from this software without specific prior written permission.lib/librte_member/Makefile new file +2
-0+# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ANDCONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,lib/librte_member/Makefile new file +2
-0+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OFUSE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
lib/librte_member/Makefile line 38
Quoted 8 lines
lib/librte_member/Makefile new file +8
-0+ +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_member.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +
This breaks reproducibility as the output directory will be included before the source directory, causing a race - please do something like:
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
Quoted 10 lines
lib/librte_member/Makefile new file +10
-0+EXPORT_MAP := rte_member_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h + +include $(RTE_SDK)/mk/rte.lib.mk
Signature · 2 lines
Kind regards, Luca Boccassi
Re: [PATCH 1/7] member: implement main API
Quoted 104 lines · Luca Boccassi
Quoted 91 lines · Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>---
lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357 +++++++++++++++++++++ lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +7 files changed, 942 insertions(+)create mode 100644
lib/librte_member/Makefilecreate mode 100644lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.maplib/librte_member/Makefile new file +6
-0@@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with orwithout
+# modification, are permitted provided that the following conditionslib/librte_member/Makefile new file +3
-0+# are met: +# +# * Redistributions of source code must retain the abovecopyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer inlib/librte_member/Makefile new file +4
-0+# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote productsderived
+# from this software without specific prior written permission.lib/librte_member/Makefile new file +2
-0+# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ANDCONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,lib/librte_member/Makefile new file +2
-0+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OFUSE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.lib/librte_member/Makefile line 38
Quoted 8 lines
lib/librte_member/Makefile new file +8
-0+ +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_member.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +This breaks reproducibility as the output directory will be included before the source directory, causing a race - please do something like:
Quoted 2 lines
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
Can we remove "-I$(SRCDIR)" completely by first installing headers and later compiling objects, all using $(RTE_OUT) only?
Do you think can this work?
Quoted 12 lines
Quoted 11 lines
lib/librte_member/Makefile new file +10
-0+EXPORT_MAP := rte_member_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h + +include $(RTE_SDK)/mk/rte.lib.mk
Re: [PATCH 1/7] member: implement main API
Quoted 119 lines · Ferruh Yigit
Quoted 118 lines · Luca Boccassi
Quoted 103 lines · Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>---
lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357
+++++++++++++++++++++
lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +7 files changed, 942 insertions(+)create mode 100644
lib/librte_member/Makefilecreate mode 100644lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.maplib/librte_member/Makefile new file +6
-0@@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with orwithout
+# modification, are permitted provided that the following conditionslib/librte_member/Makefile new file +3
-0+# are met: +# +# * Redistributions of source code must retain the abovecopyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer
in
+# the documentation and/or other materials provided with thelib/librte_member/Makefile new file +2
-0+# distribution. +# * Neither the name of Intel Corporation nor the names ofits
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.lib/librte_member/Makefile new file +2
-0+# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ANDCONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.lib/librte_member/Makefile line 38
Quoted 8 lines
lib/librte_member/Makefile new file +8
-0+ +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_member.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something like:
Quoted 8 lines
Quoted 2 lines
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3Can we remove "-I$(SRCDIR)" completely by first installing headers and
later compiling objects, all using $(RTE_OUT) only?Do you think can this work?
I'm not sure, it might - but given Bruce's effort to port to Meson I'm not sure it's worth spending a lot of time doing big refactoring of the existing build system
Quoted 12 lines
Quoted 10 lines
Quoted 10 lines
lib/librte_member/Makefile new file +10
-0+EXPORT_MAP := rte_member_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h + +include $(RTE_SDK)/mk/rte.lib.mk
Signature · 2 lines
Kind regards, Luca Boccassi
Re: [PATCH 1/7] member: implement main API
Quoted 127 lines · Luca Boccassi
Quoted 126 lines · Ferruh Yigit
Quoted 118 lines · Luca Boccassi
Quoted 103 lines · Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>---
lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357
+++++++++++++++++++++
lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +7 files changed, 942 insertions(+)create mode 100644
lib/librte_member/Makefilecreate mode 100644lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.maplib/librte_member/Makefile new file +6
-0@@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with orwithout
+# modification, are permitted provided that the following conditionslib/librte_member/Makefile new file +3
-0+# are met: +# +# * Redistributions of source code must retain the abovecopyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer
in
+# the documentation and/or other materials provided with thelib/librte_member/Makefile new file +2
-0+# distribution. +# * Neither the name of Intel Corporation nor the names ofits
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.lib/librte_member/Makefile new file +2
-0+# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ANDCONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.lib/librte_member/Makefile
Quoted 8 lines
lib/librte_member/Makefile new file +8
-0+ +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_member.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something like:Quoted 2 lines
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3Can we remove "-I$(SRCDIR)" completely by first installing headers and
later compiling objects, all using $(RTE_OUT) only?
Quoted 5 lines
Quoted 1 line
Do you think can this work?
I'm not sure, it might - but given Bruce's effort to port to Meson I'm not sure it's worth spending a lot of time doing big refactoring of the existing build system
Not big refactoring, following seems worked for me, if you would like to test:
mk/rte.lib.mk +6 -1
@@ -59,14 +59,19 @@ endif _BUILD = $(LIB) -_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y) $(RTE_OUTPUT)/lib/$(LIB) +PREINSTALL = $(SYMLINK-FILES-y) +_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB) _CLEAN = doclean .PHONY: all all: install .PHONY: install +ifeq ($(SYMLINK-FILES-y),) install: build _postinstall +else +install: _preinstall build _postinstall +endif _postinstall: build
Quoted 14 lines
Quoted 13 lines
Quoted 11 lines
Quoted 11 lines
lib/librte_member/Makefile new file +10
-0+EXPORT_MAP := rte_member_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h + +include $(RTE_SDK)/mk/rte.lib.mk
Re: [PATCH 1/7] member: implement main API
Quoted 186 lines · Ferruh Yigit
Quoted 149 lines · Luca Boccassi
Quoted 148 lines · Ferruh Yigit
Quoted 139 lines · Luca Boccassi
Quoted 123 lines · Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to
a
set(s). A membership test for an element will return the set this
element
belongs to or not-found if the element is never inserted into the
set-summary.The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>---
lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357
+++++++++++++++++++++
lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +7 files changed, 942 insertions(+)create mode 100644
lib/librte_member/Makefilecreate mode 100644lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.maplib/librte_member/Makefile new file +3
-0@@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rightsreserved.
lib/librte_member/Makefile new file +3
-0+# All rights reserved. +# +# Redistribution and use in source and binary forms, withor
without
+# modification, are permitted provided that the following conditionslib/librte_member/Makefile new file +3
-0+# are met: +# +# * Redistributions of source code must retain the abovecopyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above
copyright
+# notice, this list of conditions and the following disclaimer
in
+# the documentation and/or other materials provided with
thelib/librte_member/Makefile new file +2
-0+# distribution. +# * Neither the name of Intel Corporation nor the namesof
its
+# contributors may be used to endorse or promote products
derived
+# from this software without specific prior written permission.lib/librte_member/Makefile new file +2
-0+# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ANDCONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT
NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND
ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR
TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF
THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.lib/librte_member/Makefile
Quoted 8 lines
lib/librte_member/Makefile new file +8
-0+ +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_member.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something
like:Quoted 2 lines
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3Can we remove "-I$(SRCDIR)" completely by first installing headers
and
later compiling objects, all using $(RTE_OUT) only?Quoted 7 lines
Quoted 1 line
Do you think can this work?
I'm not sure, it might - but given Bruce's effort to port to Meson I'm
not sure it's worth spending a lot of time doing big refactoring of the
existing build systemNot big refactoring, following seems worked for me, if you would like to
test:mk/rte.lib.mk +2
-1@@ -59,14 +59,19 @@ endif _BUILD = $(LIB) -_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y) $(RTE_OUTPUT)/lib/$(LIB) +PREINSTALL = $(SYMLINK-FILES-y) +_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)_CLEAN = doclean
.PHONY: all
all: install.PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
install: build_postinstallmk/rte.lib.mk +3
-0+else +install: _preinstall build _postinstall +endif
Quoted 1 line
_postinstall: build
Easier than I would have thought :-)
I'll give that patch a spin through the repro build grinder next week, thanks.
Signature · 2 lines
Kind regards, Luca Boccassi
Re: [PATCH 1/7] member: implement main API
Quoted 202 lines · Luca Boccassi
Quoted 201 lines · Ferruh Yigit
Quoted 163 lines · Luca Boccassi
Quoted 155 lines · Ferruh Yigit
Quoted 151 lines · Luca Boccassi
Quoted 135 lines · Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the
Membership
library is a data structure that provides a "set-summary" and
responds
to set-membership queries of whether a certain element belongs to
a
set(s). A membership test for an element will return the set
this
element
belongs to or not-found if the element is never inserted into
the
set-summary.The results of the membership test is not 100% accurate. Certain
false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set-
summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>---
lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357
+++++++++++++++++++++
lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +7 files changed, 942 insertions(+)create mode 100644
lib/librte_member/Makefilecreate mode 100644lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.maplib/librte_member/Makefile new file +3
-0@@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rightsreserved.
lib/librte_member/Makefile new file +3
-0+# All rights reserved. +# +# Redistribution and use in source and binary forms,with
or
without
+# modification, are permitted provided that the following
conditionslib/librte_member/Makefile new file +3
-0+# are met: +# +# * Redistributions of source code must retain theabove
copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above
copyright
+# notice, this list of conditions and the following disclaimer
in
+# the documentation and/or other materials provided with
thelib/librte_member/Makefile new file +2
-0+# distribution. +# * Neither the name of Intel Corporation nor thenames
of
its
+# contributors may be used to endorse or promote products
derived
+# from this software without specific prior written permission.lib/librte_member/Makefile new file +2
-0+# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ANDCONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING,
BUT
NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT
NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF
USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND
ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR
TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF
THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH
DAMAGE.lib/librte_member/Makefile
Quoted 8 lines
lib/librte_member/Makefile new file +8
-0+ +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_member.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something
like:Quoted 2 lines
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3Quoted 6 lines
Can we remove "-I$(SRCDIR)" completely by first installing headers
and
later compiling objects, all using $(RTE_OUT) only?Do you think can this work?
Quoted 7 lines
I'm not sure, it might - but given Bruce's effort to port to Meson
I'm
not sure it's worth spending a lot of time doing big refactoring of
the
existing build systemNot big refactoring, following seems worked for me, if you would like
to
test:mk/rte.lib.mk +2
-1@@ -59,14 +59,19 @@ endif _BUILD = $(LIB) -_INSTALL = $(INSTALL-FILES-y) $(SYMLINK-FILES-y) $(RTE_OUTPUT)/lib/$(LIB) +PREINSTALL = $(SYMLINK-FILES-y) +_INSTALL = $(INSTALL-FILES-y) $(RTE_OUTPUT)/lib/$(LIB)_CLEAN = doclean
.PHONY: all
all: install.PHONY: install
+ifeq ($(SYMLINK-FILES-y),)
install: build_postinstallmk/rte.lib.mk +3
-0+else +install: _preinstall build _postinstall +endif
Quoted 7 lines
Quoted 1 line
_postinstall: build
Easier than I would have thought :-)
I'll give that patch a spin through the repro build grinder next week,
thanks.
Thanks Ferruh, this works in making the build reproducible with the current CI as an alternative with a fixed build path.
But on the other hand, it will cause the compiler to always embed the custom build directory in the path to the header in the DWARF .debug files, rather than the "repository" path:
readelf --debug-dump=line /tmp/69c02db5050024b530a14555b49528c419419f.debug
<cut>
The Directory Table (offset 0x1b):
1 dpdk/lib/librte_eal/linuxapp/eal
2 /usr/include/x86_64-linux-gnu/bits
3 dpdk/lib/librte_eal/common
4 dpdk/debian/build/shared-root/include 5 dpdk/debian/build/shared-root/include/generic 6 /usr/lib/gcc/x86_64-linux-gnu/7/include 7 /usr/include/x86_64-linux-gnu/sys
8 /usr/include
The File Name Table (offset 0x15a):
Entry Dir Time Size Name
1 1 0 0 eal.c
2 2 0 0 stdio2.h
3 3 0 0 eal_filesystem.h 4 2 0 0 fcntl2.h
5 4 0 0 rte_atomic.h 6 5 0 0 rte_atomic.h 7 2 0 0 string3.h
8 4 0 0 rte_cycles.h 9 4 0 0 rte_random.h 10 6 0 0 xmmintrin.h 11 4 0 0 rte_eal_memconfig.h 12 4 0 0 rte_lcore.h
<cut>
note 4 and 5 items in the directory tables, they point to debian/build- shared-root which is the output directory we use in the package build.
This means that 2 builds that are otherwise exactly the same (same machine type, same build dependencies) will not be binary identical due to that difference.
On one hand it's a much smaller and nicer patch and there's the Meson work which will make both obsolete, on the other hand it's not very reproducibl-y. What do you think?
Signature · 2 lines
Kind regards, Luca Boccassi
Re: [PATCH 1/7] member: implement main API
In reply to Luca BoccassiQuoted 116 lines
-----Original Message-----
From: Luca Boccassi [mailto:[email protected]]
Sent: Tuesday, August 22, 2017 3:02 AM
To: Wang, Yipeng1 <[email protected]>
Cc: [email protected]
Subject: Re: [dpdk-dev] [PATCH 1/7] member: implement main APIQuoted 73 lines · Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>---
lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357 +++++++++++++++++++++ lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +7 files changed, 942 insertions(+)create mode 100644
lib/librte_member/Makefilecreate mode 100644lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.maplib/librte_member/Makefile new file +6
-0@@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with orwithout
+# modification, are permitted provided that the following conditionslib/librte_member/Makefile new file +3
-0+# are met: +# +# * Redistributions of source code must retain the abovecopyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer inlib/librte_member/Makefile new file +4
-0+# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote productsderived
+# from this software without specific prior written permission.lib/librte_member/Makefile new file +2
-0+# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ANDCONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUTQuoted 8 lines
NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,BUT NOT
Quoted 1 line
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF
Quoted 2 lines
USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSEDAND
Quoted 4 lines
ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUTOF
lib/librte_member/Makefile line 38
Quoted 11 lines
THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.lib/librte_member/Makefile new file +8
-0+ +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_member.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +This breaks reproducibility as the output directory will be included before the source directory, causing a race - please do something like:
Quoted 3 lines
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3
[Wang, Yipeng]
Thanks Luca, I will reverse the include order in next version.
I am not an expert on the build system, to be more clear, the race condition you mentioned is because the library may search build directory first that some symlinks may have not been created yet, right?
Thanks
Quoted 14 lines
lib/librte_member/Makefile line 48
Quoted 10 lines
lib/librte_member/Makefile new file +10
-0+EXPORT_MAP := rte_member_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h + +include $(RTE_SDK)/mk/rte.lib.mk--
Kind regards,
Luca Boccassi
Re: [PATCH 1/7] member: implement main API
Quoted 136 lines · On Thu, 2017-08-24 at 18:38 +0000, Wang, Yipeng1 wrote:
Quoted 129 lines
-----Original Message-----
From: Luca Boccassi [mailto:[email protected]]
Sent: Tuesday, August 22, 2017 3:02 AM
To: Wang, Yipeng1 <[email protected]>
Cc: [email protected]
Subject: Re: [dpdk-dev] [PATCH 1/7] member: implement main APIQuoted 80 lines · Yipeng Wang
Membership library is an extension and generalization of a traditional
filter (for example Bloom Filter) structure. In general, the Membership
library is a data structure that provides a "set-summary" and responds
to set-membership queries of whether a certain element belongs to a
set(s). A membership test for an element will return the set this element
belongs to or not-found if the element is never inserted into the set-summary.The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However,
comparing to a "full-blown" complete list of elements, a "set- summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>---
lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 48 +++ lib/librte_member/rte_member.c | 357
+++++++++++++++++++++
lib/librte_member/rte_member.h | 518
+++++++++++++++++++++++++++++++
lib/librte_member/rte_member_version.map | 15 +7 files changed, 942 insertions(+)create mode 100644
lib/librte_member/Makefilecreate mode 100644lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.maplib/librte_member/Makefile new file +6
-0@@ -0,0 +1,48 @@ +# BSD LICENSE +# +# Copyright(c) 2017 Intel Corporation. All rights reserved. +# All rights reserved. +# +# Redistribution and use in source and binary forms, with orwithout
+# modification, are permitted provided that the following conditionslib/librte_member/Makefile new file +3
-0+# are met: +# +# * Redistributions of source code must retain the abovecopyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer
in
+# the documentation and/or other materials provided with thelib/librte_member/Makefile new file +2
-0+# distribution. +# * Neither the name of Intel Corporation nor the names ofits
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.lib/librte_member/Makefile new file +2
-0+# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ANDCONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUTQuoted 8 lines
NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,BUT NOT
Quoted 1 line
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF
Quoted 2 lines
USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSEDAND
Quoted 5 lines
ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUTOF
lib/librte_member/Makefile line 38
Quoted 11 lines
THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.lib/librte_member/Makefile new file +8
-0+ +include $(RTE_SDK)/mk/rte.vars.mk + +# library name +LIB = librte_member.a + +CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +This breaks reproducibility as the output directory will be included
before the source directory, causing a race - please do something like:Quoted 3 lines
CFLAGS := -I$(SRCDIR) $(CFLAGS)
CFLAGS += $(WERROR_FLAGS) -O3[Wang, Yipeng]
Thanks Luca, I will reverse the include order in next version.
Quoted 7 lines
I am not an expert on the build system, to be more clear, the race condition you
mentioned is because the library may search build directory first that some symlinks
may have not been created yet, right?Thanks
Yes, and as a consequence on some builds the DWARF symbols list lib/libfoo/foo.h as the source file path in the .debug, and on some builds they list build-dir/include/foo.h
Quoted 14 lines
Quoted 14 lines
lib/librte_member/Makefile line 48
Quoted 10 lines
lib/librte_member/Makefile new file +10
-0+EXPORT_MAP := rte_member_version.map + +LIBABIVER := 1 + +# all source are stored in SRCS-y +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +# install includes +SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h + +include $(RTE_SDK)/mk/rte.lib.mk--
Kind regards,
Luca Boccassi
Signature · 2 lines
Kind regards, Luca Boccassi
PATCH 2/7 member: implement HT mode
In reply to Yipeng WangOne of the set-summray structure is hash-table based set-summary (HTSS). One example is cuckoo filter [1].
Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.
Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is more similar to traditional cuckoo filter. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.
The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. So the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.
This patch add the implementation of HTSS.
[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_ht.c | 488 ++++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_ht.h | 115 +++++++++ 3 files changed, 604 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
@@ -41,7 +41,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_ht.c new file +488 -0
488 lines not on this page · raw message
lib/librte_member/rte_member_ht.h new file +115 -0
115 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH 3/7 member: implement vBF mode
In reply to Yipeng WangBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.
Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set arbitrarily so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_vbf.c | 309 +++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_vbf.h | 85 ++++++++++ 3 files changed, 395 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
@@ -41,7 +41,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_vbf.c new file +309 -0
309 lines not on this page · raw message
lib/librte_member/rte_member_vbf.h new file +85 -0
85 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH 4/7 member: add AVX for HT mode
In reply to Yipeng WangFor key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.
This patch adds AVX2 implementation in a separate header file.
--- lib/librte_member/rte_member_ht.c | 143 ++++++++++++++++++++++++++++--------- lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+), 32 deletions(-) create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -32
@@ -40,6 +40,10 @@#include "rte_member.h" #include "rte_member_ht.h" +#if defined(RTE_ARCH_X86 ) +#include "rte_member_x86.h" +#endif + static inline int insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -133,6 +137,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) buckets[i].sets[j] =RTE_MEMBER_NO_MATCH ; } +#if defined(RTE_ARCH_X86 ) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +RTE_MEMBER_BUCKET_ENTRIES ==16 ) + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_AVX2 ; + else +#endif + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_SCALAR ; RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -172,11 +183,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss, *set_id =RTE_MEMBER_NO_MATCH ;get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -if (search_bucket_single (prim_bucket, tmp_sig, buckets, - set_id) || -search_bucket_single (sec_bucket, tmp_sig, - buckets, set_id)) -return 1 ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single_avx (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; +break ; +#endif +default : +if (search_bucket_single (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; + }return 0 ; }
@@ -201,13 +224,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss, }for (i =0 ; i < num_keys; i++) { -if (search_bucket_single (prim_buckets[i], tmp_sig[i], - buckets, &set_id[i]) || -search_bucket_single (sec_buckets[i], - tmp_sig[i], buckets, &set_id[i])) - ret++; -else - set_id[i] =RTE_MEMBER_NO_MATCH ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_buckets[i], + tmp_sig[i], buckets, &set_id[i]) || +search_bucket_single_avx (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; +break ; +#endif +default : +if (search_bucket_single (prim_buckets[i], tmp_sig[i], + buckets, &set_id[i]) || +search_bucket_single (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; + } }return ret; }
@@ -225,12 +262,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, - match_per_key, set_id); -if (ret < match_per_key) -search_bucket_multi (sec_bucket, tmp_sig, - buckets, &ret, match_per_key, set_id); -return ret; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_bucket, tmp_sig, buckets, + &ret, match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi_avx (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; +#endif +default : +search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, + match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; + } }
@@ -257,16 +306,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,for (i =0 ; i < num_keys; i++) { match_cnt_t =0 ; -search_bucket_multi (prim_buckets[i], tmp_sig[i], - buckets, &match_cnt_t, match_per_key, - &set_ids[i*match_per_key]); -if (match_cnt_t < match_per_key) -search_bucket_multi (sec_buckets[i], tmp_sig[i], +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_buckets[i], tmp_sig[i], buckets, &match_cnt_t, match_per_key, &set_ids[i*match_per_key]); - match_count[i] = match_cnt_t; -if (match_cnt_t !=0 ) - ret++; +if (match_cnt_t < match_per_key) +search_bucket_multi_avx (sec_buckets[i], + tmp_sig[i], buckets, &match_cnt_t, + match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_t; +if (match_cnt_t !=0 ) + ret++; +break ; +#endif +default : +search_bucket_multi (prim_buckets[i], tmp_sig[i], + buckets, &match_cnt_t, match_per_key, + &set_ids[i*match_per_key]); +if (match_cnt_t < match_per_key) +search_bucket_multi (sec_buckets[i], tmp_sig[i], + buckets, &match_cnt_t, match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_t; +if (match_cnt_t !=0 ) + ret++; + } }return ret; }
@@ -298,12 +365,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,static inline int try_overwrite (struct member_ht_bucket *buckets,uint32_t prim,uint32_t sec, -SIG_TYPE sig,MEMBER_SET_TYPE set_id) +SIG_TYPE sig,MEMBER_SET_TYPE set_id, +enum rte_member_sig_compare_function cmp_fn) { -if (insert_overwrite_search (prim, sig, buckets, set_id) || -insert_overwrite_search (sec, sig, buckets, - set_id)) -return 0 ; +switch (cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (insert_overwrite_search_avx (prim, sig, buckets, set_id) || +insert_overwrite_search_avx (sec, sig, buckets, + set_id)) +return 0 ; +break ; +#endif +default : +if (insert_overwrite_search (prim, sig, buckets, set_id) || +insert_overwrite_search (sec, sig, buckets, + set_id)) +return 0 ; + }return -1 ; }
@@ -409,7 +488,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,/* if it is cache based filter, we try overwriting existing entry */ if (ss->cache) { ret =try_overwrite (buckets, prim_bucket, sec_bucket, tmp_sig, - set_id); + set_id, ss->sig_cmp_fn);if (ret !=-1 )return ret; }
lib/librte_member/rte_member_x86.h new file +111 -0
111 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH 5/7 member: enable the library
In reply to Yipeng WangThis patch enables the Membership library.
--- MAINTAINERS | 7 +++++++ config/common_base | 5 +++++ lib/librte_member/Makefile | 2 ++ mk/rte.app.mk | 2 ++ 4 files changed, 16 insertions(+)
MAINTAINERS +7 -0
@@ -734,6 +734,13 @@ F: test/test/test_meter.c F: examples/qos_meter/ F: doc/guides/sample_app_ug/qos_metering.rst +Membership - EXPERIMENTAL +M: Yipeng Wang <[email protected]> +M: Sameh Gobriel <[email protected]> +F: lib/librte_member/ +F: doc/guides/prog_guide/member_lib.rst +F: test/test/test_member* + Other libraries ---------------
config/common_base +5 -0
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n CONFIG_RTE_LIBRTE_EFD=y # +# Compile librte_member +# +CONFIG_RTE_LIBRTE_MEMBER=y + +# # Compile librte_jobstats # CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
@@ -36,6 +36,8 @@ LIB = librte_member.a CFLAGS += $(WERROR_FLAGS) -I$(SRCDIR) -O3 +LDLIBS += -lm + EXPORT_MAP := rte_member_version.map LIBABIVER := 1
mk/rte.app.mk +2 -0
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile _LDLIBS-y += --whole-archive _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt _LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma endif
Signature · 1 line
2.7.4
PATCH 6/7 test/member: add functional and perf tests
In reply to Yipeng WangThis patch adds functional and performance tests for membership library.
--- test/test/Makefile | 3 + test/test/test_member.c | 550 ++++++++++++++++++++++++++++++++++++ test/test/test_member_perf.c | 643 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1196 insertions(+) create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
test/test/Makefile +3 -0
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c SRCS-y += test_memcpy.c SRCS-y += test_memcpy_perf.c + +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
test/test/test_member.c new file +550 -0
550 lines not on this page · raw message
test/test/test_member_perf.c new file +643 -0
643 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH 7/7 doc: add membership documentation
In reply to Yipeng WangThis patch adds the documentation for membership library.
--- doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 ++++ doc/guides/prog_guide/img/member_i6.svg | 332 +++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 432 +++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + 12 files changed, 3607 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst
doc/api/doxy-api-index.md +2 -1
@@ -105,7 +105,8 @@ The public API headers are grouped by topics: [LPM IPv4 route] (@ref rte_lpm.h), [LPM IPv6 route] (@ref rte_lpm6.h), [ACL] (@ref rte_acl.h), - [EFD] (@ref rte_efd.h) + [EFD] (@ref rte_efd.h), + [member] (@ref rte_member.h) - **QoS**: [metering] (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
@@ -58,6 +58,7 @@ INPUT = doc/api/doxy-api-index.md \ lib/librte_mempool \ lib/librte_meter \ lib/librte_metrics \ + lib/librte_member \ lib/librte_net \ lib/librte_pdump \ lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0
1,613 lines not on this page · raw message
doc/guides/prog_guide/img/member_i2.svg new file +36 -0
@@ -0,0 +1,36 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in" + viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6"> + <v:documentProperties v:langID="1033" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {fill:none;stroke:none;stroke-width:0.25} + .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em} + .st3 {baseline-shift:32.4943%;font-size:0.649886em} + .st4 {font-size:1em} + .st5 {font-family:Cambria Math;font-size:1em} + .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/> + <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)"> + <title>Sheet.3</title> + <desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="157.5" cy="45.5" width="315" height="90"/> + <rect x="0" y="0.5" width="315" height="90" class="st1"/> + <text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan + dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan + dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan + class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3" + v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em" + class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text> </g> + </g> +</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0
148 lines not on this page · raw message
doc/guides/prog_guide/img/member_i4.svg new file +450 -0
450 lines not on this page · raw message
doc/guides/prog_guide/img/member_i5.svg new file +163 -0
163 lines not on this page · raw message
doc/guides/prog_guide/img/member_i6.svg new file +332 -0
332 lines not on this page · raw message
doc/guides/prog_guide/img/member_i7.svg new file +399 -0
242 lines not on this page · raw message
doc/guides/prog_guide/member_lib.rst new file +432 -0
432 lines not on this page · raw message
doc/guides/rel_notes/release_17_11.rst +17 -0
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.7.4
Part of the diff is not shown here. View the raw message.
Quoted 11 lines · On Mon, 21 Aug 2017 17:19:46 -0700 Yipeng Wang <[email protected]> wrote:
This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.
I don't think it makes sense to merge two different types of tables in one API. Especially in DPDK where every cycle counts. You are taking an extra branch on each lookup. The user of this API is likely to know exactly what type of objects and look are desired.
Hi Stephen,
Thanks for the comments. We will remove the needless return initialization in next version.
For the unified API concern, we think the current implementation has two benefits: 1. It is easier to extend the library with new types of filters without adding a lot of top level APIs every time. 2. When users switch between different types in their application, they do not need to switch between function calls. They can reuse same code for different types.
However, we agree with you that a switch case in the library affects the performance. We did a quick test and here is the results (cycles/operation):
lookup lookup_bulk(16 batch) lookup_multi lookup_multi_bulkswitch_case 50 36 57 45
direct call 47 35 54 45
There will be 3 cycle difference for non-bulk version lookup. I guess if users usually use bulk version, it maybe not a big concern, but single key lookup indeed slower. If you think the benefit we mentioned does not outweigh the performance slowdown, we would like to make the change.
We also considered using function pointers to get rid of switch case, but it will be unfriendly to the multi-process usages.
Please comments.
Thanks
Yipeng
Quoted 31 lines
-----Original Message-----
From: Stephen Hemminger [mailto:[email protected]]
Sent: Monday, August 21, 2017 9:02 PM
To: Wang, Yipeng1 <[email protected]>
Cc: [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>
Subject: Re: [PATCH 0/7] Add Membership LibraryQuoted 9 lines · On Mon, 21 Aug 2017 17:19:46 -0700 Yipeng Wang <[email protected]> wrote:
This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF
only has false-positives similar to the non-cache HTSS.
Quoted 2 lines
However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.
I don't think it makes sense to merge two different types of tables in one API. Especially in DPDK where every cycle counts. You are taking an extra branch on each lookup. The user of this API is likely to know exactly what type of objects and look are desired.
v2 cover Add Membership Library
In reply to Yipeng WangDPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).
The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.
The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.
A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.
This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.
[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.
[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitilize issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment
- documentation: update the programmer guide for membership library
Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation
MAINTAINERS | 7 + config/common_base | 5 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 +++ doc/guides/prog_guide/img/member_i6.svg | 332 ++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 440 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 51 + lib/librte_member/rte_member.c | 342 +++++++ lib/librte_member/rte_member.h | 518 ++++++++++ lib/librte_member/rte_member_ht.c | 569 +++++++++++ lib/librte_member/rte_member_ht.h | 115 +++ lib/librte_member/rte_member_vbf.c | 350 +++++++ lib/librte_member/rte_member_vbf.h | 85 ++ lib/librte_member/rte_member_version.map | 16 + lib/librte_member/rte_member_x86.h | 111 ++ mk/rte.app.mk | 2 + test/test/Makefile | 3 + test/test/test_member.c | 682 +++++++++++++ test/test/test_member_perf.c | 643 ++++++++++++ 30 files changed, 7118 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h create mode 100644 lib/librte_member/rte_member_version.map create mode 100644 lib/librte_member/rte_member_x86.h create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
PATCH v2 1/7 member: implement main API
Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.
This patch add the main API definition.
--- lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 49 +++ lib/librte_member/rte_member.c | 342 ++++++++++++++++++++ lib/librte_member/rte_member.h | 518 +++++++++++++++++++++++++++++++ lib/librte_member/rte_member_version.map | 16 + 7 files changed, 929 insertions(+) create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether +DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member +DEPDIRS-librte_member := librte_eal librte_hash ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_eal/common/eal_common_log.c +1 -0
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = { {RTE_LOGTYPE_CRYPTODEV, "cryptodev"}, {RTE_LOGTYPE_EFD, "efd"}, {RTE_LOGTYPE_EVENTDEV, "eventdev"}, + {RTE_LOGTYPE_MEMBER, "member"}, {RTE_LOGTYPE_USER1, "user1"}, {RTE_LOGTYPE_USER2, "user2"}, {RTE_LOGTYPE_USER3, "user3"},
lib/librte_eal/common/include/rte_log.h +1 -0
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;#define RTE_LOGTYPE_CRYPTODEV 17/**< Log related to cryptodev. */ #define RTE_LOGTYPE_EFD 18/**< Log related to EFD. */ #define RTE_LOGTYPE_EVENTDEV 19/**< Log related to eventdev. */ +#define RTE_LOGTYPE_MEMBER 20/**< Log related to membership. */ /* these log types can be used in an application */ #define RTE_LOGTYPE_USER1 24/**< User-defined log type 1. */
lib/librte_member/Makefile new file +49 -0
49 lines not on this page · raw message
lib/librte_member/rte_member.c new file +342 -0
342 lines not on this page · raw message
lib/librte_member/rte_member.h new file +518 -0
518 lines not on this page · raw message
lib/librte_member/rte_member_version.map new file +16 -0
@@ -0,0 +1,16 @@ +DPDK_17.11 { + global: + + rte_member_create; + rte_member_find_existing; + rte_member_lookup; + rte_member_lookup_bulk; + rte_member_lookup_multi; + rte_member_lookup_multi_bulk; + rte_member_add; + rte_member_free; + rte_member_reset; + rte_member_delete; + + local: *; +};
Signature · 1 line
2.7.4
PATCH v2 2/7 member: implement HT mode
In reply to Yipeng WangOne of the set-summray structure is hash-table based set-summary (HTSS). One example is cuckoo filter [1].
Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.
Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is more similar to traditional cuckoo filter. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.
The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. So the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.
This patch add the implementation of HTSS.
[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_ht.c | 490 ++++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_ht.h | 115 +++++++++ 3 files changed, 606 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_ht.c new file +490 -0
490 lines not on this page · raw message
lib/librte_member/rte_member_ht.h new file +115 -0
115 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v2 3/7 member: implement vBF mode
In reply to Yipeng WangBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.
Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set arbitrarily so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_vbf.c | 350 +++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_vbf.h | 85 +++++++++ 3 files changed, 436 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_vbf.c new file +350 -0
350 lines not on this page · raw message
lib/librte_member/rte_member_vbf.h new file +85 -0
85 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v2 4/7 member: add AVX for HT mode
In reply to Yipeng WangFor key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.
This patch adds AVX2 implementation in a separate header file.
--- lib/librte_member/rte_member_ht.c | 143 ++++++++++++++++++++++++++++--------- lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+), 32 deletions(-) create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -32
@@ -40,6 +40,10 @@#include "rte_member.h" #include "rte_member_ht.h" +#if defined(RTE_ARCH_X86 ) +#include "rte_member_x86.h" +#endif + static inline int insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -135,6 +139,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) buckets[i].sets[j] =RTE_MEMBER_NO_MATCH ; } +#if defined(RTE_ARCH_X86 ) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +RTE_MEMBER_BUCKET_ENTRIES ==16 ) + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_AVX2 ; + else +#endif + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_SCALAR ; RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -174,11 +185,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss, *set_id =RTE_MEMBER_NO_MATCH ;get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -if (search_bucket_single (prim_bucket, tmp_sig, buckets, - set_id) || -search_bucket_single (sec_bucket, tmp_sig, - buckets, set_id)) -return 1 ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single_avx (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; +break ; +#endif +default : +if (search_bucket_single (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; + }return 0 ; }
@@ -203,13 +226,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss, }for (i =0 ; i < num_keys; i++) { -if (search_bucket_single (prim_buckets[i], tmp_sig[i], - buckets, &set_id[i]) || -search_bucket_single (sec_buckets[i], - tmp_sig[i], buckets, &set_id[i])) - ret++; -else - set_id[i] =RTE_MEMBER_NO_MATCH ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_buckets[i], + tmp_sig[i], buckets, &set_id[i]) || +search_bucket_single_avx (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; +break ; +#endif +default : +if (search_bucket_single (prim_buckets[i], tmp_sig[i], + buckets, &set_id[i]) || +search_bucket_single (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; + } }return ret; }
@@ -227,12 +264,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, - match_per_key, set_id); -if (ret < match_per_key) -search_bucket_multi (sec_bucket, tmp_sig, - buckets, &ret, match_per_key, set_id); -return ret; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_bucket, tmp_sig, buckets, + &ret, match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi_avx (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; +#endif +default : +search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, + match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; + } }
@@ -259,16 +308,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,for (i =0 ; i < num_keys; i++) { match_cnt_t =0 ; -search_bucket_multi (prim_buckets[i], tmp_sig[i], - buckets, &match_cnt_t, match_per_key, - &set_ids[i*match_per_key]); -if (match_cnt_t < match_per_key) -search_bucket_multi (sec_buckets[i], tmp_sig[i], +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_buckets[i], tmp_sig[i], buckets, &match_cnt_t, match_per_key, &set_ids[i*match_per_key]); - match_count[i] = match_cnt_t; -if (match_cnt_t !=0 ) - ret++; +if (match_cnt_t < match_per_key) +search_bucket_multi_avx (sec_buckets[i], + tmp_sig[i], buckets, &match_cnt_t, + match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_t; +if (match_cnt_t !=0 ) + ret++; +break ; +#endif +default : +search_bucket_multi (prim_buckets[i], tmp_sig[i], + buckets, &match_cnt_t, match_per_key, + &set_ids[i*match_per_key]); +if (match_cnt_t < match_per_key) +search_bucket_multi (sec_buckets[i], tmp_sig[i], + buckets, &match_cnt_t, match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_t; +if (match_cnt_t !=0 ) + ret++; + } }return ret; }
@@ -300,12 +367,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,static inline int try_overwrite (struct member_ht_bucket *buckets,uint32_t prim,uint32_t sec, -SIG_TYPE sig,MEMBER_SET_TYPE set_id) +SIG_TYPE sig,MEMBER_SET_TYPE set_id, +enum rte_member_sig_compare_function cmp_fn) { -if (insert_overwrite_search (prim, sig, buckets, set_id) || -insert_overwrite_search (sec, sig, buckets, - set_id)) -return 0 ; +switch (cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (insert_overwrite_search_avx (prim, sig, buckets, set_id) || +insert_overwrite_search_avx (sec, sig, buckets, + set_id)) +return 0 ; +break ; +#endif +default : +if (insert_overwrite_search (prim, sig, buckets, set_id) || +insert_overwrite_search (sec, sig, buckets, + set_id)) +return 0 ; + }return -1 ; }
@@ -411,7 +490,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,/* if it is cache based filter, we try overwriting existing entry */ if (ss->cache) { ret =try_overwrite (buckets, prim_bucket, sec_bucket, tmp_sig, - set_id); + set_id, ss->sig_cmp_fn);if (ret !=-1 )return ret; }
lib/librte_member/rte_member_x86.h new file +111 -0
111 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v2 5/7 member: enable the library
In reply to Yipeng WangThis patch enables the Membership library.
--- MAINTAINERS | 7 +++++++ config/common_base | 5 +++++ lib/librte_member/Makefile | 2 ++ mk/rte.app.mk | 2 ++ 4 files changed, 16 insertions(+)
MAINTAINERS +7 -0
@@ -734,6 +734,13 @@ F: test/test/test_meter.c F: examples/qos_meter/ F: doc/guides/sample_app_ug/qos_metering.rst +Membership - EXPERIMENTAL +M: Yipeng Wang <[email protected]> +M: Sameh Gobriel <[email protected]> +F: lib/librte_member/ +F: doc/guides/prog_guide/member_lib.rst +F: test/test/test_member* + Other libraries ---------------
config/common_base +5 -0
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n CONFIG_RTE_LIBRTE_EFD=y # +# Compile librte_member +# +CONFIG_RTE_LIBRTE_MEMBER=y + +# # Compile librte_jobstats # CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
@@ -37,6 +37,8 @@ LIB = librte_member.a CFLAGS := -I$(SRCDIR) $(CFLAGS) CFLAGS += $(WERROR_FLAGS) -O3 +LDLIBS += -lm + EXPORT_MAP := rte_member_version.map LIBABIVER := 1
mk/rte.app.mk +2 -0
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile _LDLIBS-y += --whole-archive _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt _LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma endif
Signature · 1 line
2.7.4
PATCH v2 6/7 test/member: add functional and perf tests
In reply to Yipeng WangThis patch adds functional and performance tests for membership library.
--- test/test/Makefile | 3 + test/test/test_member.c | 682 +++++++++++++++++++++++++++++++++++++++++++ test/test/test_member_perf.c | 643 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1328 insertions(+) create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
test/test/Makefile +3 -0
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c SRCS-y += test_memcpy.c SRCS-y += test_memcpy_perf.c + +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
test/test/test_member.c new file +682 -0
682 lines not on this page · raw message
test/test/test_member_perf.c new file +643 -0
643 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v2 7/7 doc: add membership documentation
In reply to Yipeng WangThis patch adds the documentation for membership library.
--- doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 ++++ doc/guides/prog_guide/img/member_i6.svg | 332 +++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 440 +++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + 12 files changed, 3615 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst
doc/api/doxy-api-index.md +2 -1
@@ -105,7 +105,8 @@ The public API headers are grouped by topics: [LPM IPv4 route] (@ref rte_lpm.h), [LPM IPv6 route] (@ref rte_lpm6.h), [ACL] (@ref rte_acl.h), - [EFD] (@ref rte_efd.h) + [EFD] (@ref rte_efd.h), + [member] (@ref rte_member.h) - **QoS**: [metering] (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
@@ -58,6 +58,7 @@ INPUT = doc/api/doxy-api-index.md \ lib/librte_mempool \ lib/librte_meter \ lib/librte_metrics \ + lib/librte_member \ lib/librte_net \ lib/librte_pdump \ lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0
1,613 lines not on this page · raw message
doc/guides/prog_guide/img/member_i2.svg new file +36 -0
@@ -0,0 +1,36 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in" + viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6"> + <v:documentProperties v:langID="1033" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {fill:none;stroke:none;stroke-width:0.25} + .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em} + .st3 {baseline-shift:32.4943%;font-size:0.649886em} + .st4 {font-size:1em} + .st5 {font-family:Cambria Math;font-size:1em} + .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/> + <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)"> + <title>Sheet.3</title> + <desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="157.5" cy="45.5" width="315" height="90"/> + <rect x="0" y="0.5" width="315" height="90" class="st1"/> + <text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan + dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan + dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan + class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3" + v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em" + class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text> </g> + </g> +</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0
148 lines not on this page · raw message
doc/guides/prog_guide/img/member_i4.svg new file +450 -0
450 lines not on this page · raw message
doc/guides/prog_guide/img/member_i5.svg new file +163 -0
163 lines not on this page · raw message
doc/guides/prog_guide/img/member_i6.svg new file +332 -0
332 lines not on this page · raw message
doc/guides/prog_guide/img/member_i7.svg new file +399 -0
242 lines not on this page · raw message
doc/guides/prog_guide/member_lib.rst new file +440 -0
440 lines not on this page · raw message
doc/guides/rel_notes/release_17_11.rst +17 -0
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.7.4
Part of the diff is not shown here. View the raw message.
Re: [PATCH v2 7/7] doc: add membership documentation
Quoted 10 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Yipeng Wang
Sent: Saturday, September 2, 2017 2:25 AM
To: [email protected]
Cc: [email protected]; [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [dpdk-dev] [PATCH v2 7/7] doc: add membership documentation
Quoted 4 lines
This patch adds the documentation for membership library.
Signed-off-by Yipeng Wang <[email protected]>---
Thanks for the detailed docs and the nice images. Some comments below.
doc/guides/prog_guide/member_lib.rst line 35
Quoted 5 lines
doc/guides/prog_guide/member_lib.rst new file +5
-0+ +.. _Member_Library: + +Membership Library +=======================
The underline should match the length of the title and should be followed by a blank line. Make this change throughout the doc.
Quoted 18 lines
doc/guides/prog_guide/member_lib.rst new file +4
-0+ +Introduction +------------ +The DPDK Membership Library provides an API for DPDK applications toinsert a
+new member, delete an existing member, or query the existence of a member in a
+given set, or a group of sets. For the case of a group of sets the library
+will return not only whether the element has been inserted before in one of(quoted diff) +2
-0+the sets but also which set it belongs to. The Membership Library is an +extension and generalization of a traditional filter structure (forexample
+Bloom Filter [Member-bloom]) that has multiple usages in a wide variety of(quoted diff) +2
-0+workloads and applications. In general, the Membership Library is a data +structure that provides a “set-summary” on whether a member belongs to a
Quoted 1 line
set,
Use standard quotes instead of smart quotes. The doc renderers will convert them to smart quotes.
Quoted 8 lines
doc/guides/prog_guide/member_lib.rst new file +8
-0+probability. + +.. _figure_membership1: +.. figure:: img/member_i1.* + + Example Usages of Membership Library + +We believe that there are various usages for a Membership Library in a
Quoted 6 lines
very
+large set of applications and workloads. Interested readers can refer to +[Member-survey] for a survey of possible networking usages. The following figure
+provide a small set of examples of using the Membership Library. Sub- figure(a)
These sub-figures would be better as separate bullet sections.
There are a lot of other small fixes that I will send on to you off-line.
v3 cover Add Membership Library
In reply to Yipeng WangDPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).
The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.
The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.
A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.
This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.
[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.
[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
v3:
- documentation: update documentation to incorperate John's review.
v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitilize issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment.
- documentation: update the programmer guide for membership library.
Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation
MAINTAINERS | 7 + config/common_base | 5 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 +++ doc/guides/prog_guide/img/member_i6.svg | 332 ++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 421 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 51 + lib/librte_member/rte_member.c | 342 +++++++ lib/librte_member/rte_member.h | 518 ++++++++++ lib/librte_member/rte_member_ht.c | 569 +++++++++++ lib/librte_member/rte_member_ht.h | 115 +++ lib/librte_member/rte_member_vbf.c | 350 +++++++ lib/librte_member/rte_member_vbf.h | 85 ++ lib/librte_member/rte_member_version.map | 16 + lib/librte_member/rte_member_x86.h | 111 ++ mk/rte.app.mk | 2 + test/test/Makefile | 3 + test/test/test_member.c | 682 +++++++++++++ test/test/test_member_perf.c | 643 ++++++++++++ 30 files changed, 7099 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h create mode 100644 lib/librte_member/rte_member_version.map create mode 100644 lib/librte_member/rte_member_x86.h create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
PATCH v3 1/7 member: implement main API
Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
The results of the membership test is not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.
This patch add the main API definition.
--- lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 49 +++ lib/librte_member/rte_member.c | 342 ++++++++++++++++++++ lib/librte_member/rte_member.h | 518 +++++++++++++++++++++++++++++++ lib/librte_member/rte_member_version.map | 16 + 7 files changed, 929 insertions(+) create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether +DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member +DEPDIRS-librte_member := librte_eal librte_hash ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_eal/common/eal_common_log.c +1 -0
@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = { {RTE_LOGTYPE_CRYPTODEV, "cryptodev"}, {RTE_LOGTYPE_EFD, "efd"}, {RTE_LOGTYPE_EVENTDEV, "eventdev"}, + {RTE_LOGTYPE_MEMBER, "member"}, {RTE_LOGTYPE_USER1, "user1"}, {RTE_LOGTYPE_USER2, "user2"}, {RTE_LOGTYPE_USER3, "user3"},
lib/librte_eal/common/include/rte_log.h +1 -0
@@ -87,6 +87,7 @@ extern struct rte_logs rte_logs;#define RTE_LOGTYPE_CRYPTODEV 17/**< Log related to cryptodev. */ #define RTE_LOGTYPE_EFD 18/**< Log related to EFD. */ #define RTE_LOGTYPE_EVENTDEV 19/**< Log related to eventdev. */ +#define RTE_LOGTYPE_MEMBER 20/**< Log related to membership. */ /* these log types can be used in an application */ #define RTE_LOGTYPE_USER1 24/**< User-defined log type 1. */
lib/librte_member/Makefile new file +49 -0
49 lines not on this page · raw message
lib/librte_member/rte_member.c new file +342 -0
342 lines not on this page · raw message
lib/librte_member/rte_member.h new file +518 -0
518 lines not on this page · raw message
lib/librte_member/rte_member_version.map new file +16 -0
@@ -0,0 +1,16 @@ +DPDK_17.11 { + global: + + rte_member_create; + rte_member_find_existing; + rte_member_lookup; + rte_member_lookup_bulk; + rte_member_lookup_multi; + rte_member_lookup_multi_bulk; + rte_member_add; + rte_member_free; + rte_member_reset; + rte_member_delete; + + local: *; +};
Signature · 1 line
2.7.4
Re: [PATCH v3 1/7] member: implement main API
06/09/2017 01:59, Yipeng Wang:
lib/librte_eal/common/eal_common_log.c static const struct logtype logtype_strings[] = { · line 285
Quoted 10 lines
lib/librte_eal/common/eal_common_log.c +1
-0@@ -279,6 +279,7 @@ static const struct logtype logtype_strings[] = { {RTE_LOGTYPE_CRYPTODEV, "cryptodev"}, {RTE_LOGTYPE_EFD, "efd"}, {RTE_LOGTYPE_EVENTDEV, "eventdev"}, + {RTE_LOGTYPE_MEMBER, "member"}, {RTE_LOGTYPE_USER1, "user1"}, {RTE_LOGTYPE_USER2, "user2"}, {RTE_LOGTYPE_USER3, "user3"},
The static logtypes should be removed.
Please use the new dynamic log types.
Re: [PATCH v3 1/7] member: implement main API
In reply to Yipeng WangQuoted 12 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Yipeng Wang
Sent: Wednesday, September 6, 2017 1:00 AM
To: [email protected]
Cc: Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>; Wang, Yipeng1 <[email protected]>; Mcnamara, John <[email protected]>
Subject: [dpdk-dev] [PATCH v3 1/7] member: implement main API
Quoted 8 lines
Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set- membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
The results of the membership test is not 100% accurate. Certain false
Is -> are.
Quoted 22 lines
positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.This patch add the main API definition.
Signed-off-by Yipeng Wang <[email protected]>--- lib/Makefile | 2 + lib/librte_eal/common/eal_common_log.c | 1 + lib/librte_eal/common/include/rte_log.h | 1 + lib/librte_member/Makefile | 49 +++ lib/librte_member/rte_member.c | 342 ++++++++++++++++++++ lib/librte_member/rte_member.h | 518 +++++++++++++++++++++++++++++++ lib/librte_member/rte_member_version.map | 16 + 7 files changed, 929 insertions(+) create mode 100644 lib/librte_member/Makefile create mode 100644
lib/librte_member/rte_member.ccreate mode 100644lib/librte_member/rte_member.hcreate mode 100644lib/librte_member/rte_member_version.map
...
Quoted 5 lines
(quoted diff) +0
-0
b/lib/librte_member/rte_member.cnew file mode 100644 index 0000000..71b066dlib/librte_member/rte_member.c new file +0
-0
...
Quoted 38 lines
lib/librte_member/rte_member.c new file +28
-0+ +#include <string.h> + +#include <rte_eal.h> +#include <rte_eal_memconfig.h> +#include <rte_memory.h> +#include <rte_memzone.h> +#include <rte_malloc.h> +#include <rte_errno.h> + +#include "rte_member.h" +#include "rte_member_ht.h" +#include "rte_member_vbf.h" + +TAILQ_HEAD (rte_member_list, rte_tailq_entry);static struct +rte_tailq_elem rte_member_tailq = { + .name ="RTE_MEMBER" , +}; +EAL_REGISTER_TAILQ (rte_member_tailq) + + +void * +rte_member_find_existing (const char *name) { +struct rte_member_setsum *setsum =NULL ; +struct rte_tailq_entry *te; +struct rte_member_list *member_list; + + member_list = RTE_TAILQ_CAST(rte_member_tailq.head,
rte_member_list);lib/librte_member/rte_member.c new file +5
-0+ +rte_rwlock_read_lock (RTE_EAL_TAILQ_RWLOCK ); + TAILQ_FOREACH(te, member_list, next) { + setsum = (struct rte_member_setsum *) te->data; + if (strncmp(name, setsum->name,
RTE_MEMBER_NAMESIZE) == 0)lib/librte_member/rte_member.c new file +3
-0+break ; + } +rte_rwlock_read_unlock (RTE_EAL_TAILQ_RWLOCK );
lib/librte_member/rte_member.c line 79
Quoted 10 lines
lib/librte_member/rte_member.c new file +10
-0+ +if (te ==NULL ) { + rte_errno =ENOENT ; +return NULL ; + } +return setsum; +} + +void +rte_member_free (void *ss)
Why not using directly "struct rte_member_setsum", so you can use it directly?
This applies to other functions that are using "void *".
I see that the content of this structure is internal, but you can still use a pointer to that structure.
...
Quoted 6 lines
lib/librte_member/rte_member.c new file +6
-0+void * +rte_member_create (const struct rte_member_parameters *params) { +struct rte_tailq_entry *te; +struct rte_member_list *member_list =NULL ; +struct rte_member_setsum *setsum =NULL ; +int ret;
lib/librte_member/rte_member.c line 132
Quoted 10 lines
lib/librte_member/rte_member.c new file +10
-0+ +if (params ==NULL ) { + rte_errno =EINVAL ; +return NULL ; + } + +if ((params->key_len ==0 )) { + rte_errno =EINVAL ; +RTE_LOG (ERR ,MEMBER , +"Memship create with invalid parameters\n" );
rte_member_create has invalid parameters?
Quoted 6 lines
(quoted diff) +0
-0
b/lib/librte_member/rte_member.hnew file mode 100644 index 0000000..de44b1blib/librte_member/rte_member.h new file +0
-0@@ -0,0 +1,518 @@
...
lib/librte_member/rte_member.h line 129
Quoted 4 lines
lib/librte_member/rte_member.h new file +2
-0+enum rte_member_setsum_type { +RTE_MEMBER_TYPE_HT =0 ,/**< Hash table based set summary. */
lib/librte_member/rte_member.h new file +1
-0+ RTE_MEMBER_TYPE_VBF, /**< vector of bloom filters. */
"vector" -> "Vector".
Quoted 6 lines
lib/librte_member/rte_member.h new file +6
-0+RTE_MEMBER_NUM_TYPE +}; + +/** @internal compare function for different arch. */ enum +rte_member_sig_compare_function { +RTE_MEMBER_COMPARE_SCALAR =0 ,
lib/librte_member/rte_member.h line 145
Quoted 10 lines
lib/librte_member/rte_member.h new file +10
-0+RTE_MEMBER_COMPARE_AVX2 , +RTE_MEMBER_COMPARE_NUM +}; + +/** @internal setsummary structure. */ +struct rte_member_setsum { +enum rte_member_setsum_type type; +const char *name; +uint32_t key_len; +uint32_t socket_id;/* NUMA Socket ID for memory. */
Either comment all the fields or do not do it (for this case, because the structure is internal, otherwise, all fields would require a comment). Also, remember that they should all start with capital letters.
Quoted 9 lines
lib/librte_member/rte_member.h new file +9
-0+uint32_t prim_hash_seed; +uint32_t sec_hash_seed; + + +/* hash table based */ +uint32_t bucket_cnt; +uint32_t bucket_mask; +uint8_t cache; +enum rte_member_sig_compare_function sig_cmp_fn;
...
Quoted 1 line
+struct
rte_member_parameters{
...
lib/librte_member/rte_member.h line 205
Quoted 1 line
lib/librte_member/rte_member.h new file +1
-0+uint8_t iscache;
As said in the documentation patch, change to "is_cache".
Quoted 6 lines
lib/librte_member/rte_member.h new file +3
-0+ +/** +* For HT setsummary, num_keys equals to the number of entries of the
lib/librte_member/rte_member.h new file +1
-0+* table. When the number of keys that inserted to the HT setsummary
"number of keys inserted in the HT summary".
Quoted 17 lines
lib/librte_member/rte_member.h new file +2
-0+* approaches this number, eviction could happen. For cache mode, +* keys could be evicted out of the table. For non-cache mode, keys will
lib/librte_member/rte_member.h new file +2
-0+* be evicted to other buckets like cuckoo hash. The table will also +* likely to become full before the number of inserted keys equal to the
lib/librte_member/rte_member.h new file +3
-0+* total number of entries. +* +* For vBF, num_keys equal to the expected number of keys that will
lib/librte_member/rte_member.h new file +1
-0+* be inserted into the vBF. The implementation assumes the keys are
lib/librte_member/rte_member.h new file +2
-0+* evenly distributed to each BF in vBF. This is used to calculate the +* number of bits we need for each BF. User does not specify the size of
lib/librte_member/rte_member.h new file +1
-0+* each BF directly because the optimal size depends on the
num_keys
lib/librte_member/rte_member.h line 225
Quoted 5 lines
lib/librte_member/rte_member.h new file +5
-0+* and false positive rate. +*/ +uint32_t num_keys; + +
Leave just one blank line between fields.
Quoted 38 lines
lib/librte_member/rte_member.h new file +3
-0+/** +* The length of key is used for hash calculation. Since key is not +* stored in set-summary, large key does not require more memory space.
lib/librte_member/rte_member.h new file +9
-0+*/ +uint32_t key_len; + + +/** +* num_set is only relevant to vBF based setsummary. +* num_set is equal to the number of BFs in vBF. For current +* implementation, it only supports 1,2,4,8,16,32 BFs in one vBF set +* summary. If other number of sets are needed, for example 5, the user
lib/librte_member/rte_member.h new file +12
-0+* should allocate the minimum available value that larger than 5, +* which is 8. +*/ +uint32_t num_set; + +/** +* false_postive_rate is only relevant to vBF based setsummary. +* false_postivie_rate is the user-defined false positive rate +* given expected number of inserted keys (num_keys). It is used to +* calculate the total number of bits for each BF, and the number of +* hash values used during lookup and insertion. For details please +* refer to vBF implementation and membership library documentation.
lib/librte_member/rte_member.h new file +3
-0+* +* HT setsummary's false positive rate is in the order of: +* false_pos = (1/bucket_count)*(1/2^16), since we use 16-bit signature.
lib/librte_member/rte_member.h new file +2
-0+* This is because two keys needs to map to same bucket and same +* signature to have a collision (false positive). bucket_count is equal
lib/librte_member/rte_member.h new file +1
-0+* to number of entries (num_keys) divided by entry count per bucket
lib/librte_member/rte_member.h new file +1
-0+* (RTE_MEMBER_BUCKET_ENTRIES). Thus, the false_postivie_rate is not
lib/librte_member/rte_member.h
Quoted 2 lines
lib/librte_member/rte_member.h new file +2
-0+* directly set by users. +*/
Typo in "positive" in several places in the comment above.
Also, clarify that "false_positive_rate" is not directly set by users for HT mode.
Quoted 1 line
lib/librte_member/rte_member.h new file +1
-0+float false_positive_rate;
...
Quoted 2 lines
+ * Lookup key in set-summary (SS).
+ * Single key lookup and return as soon as the first match found
Leave a blank line before parameters.
Quoted 4 lines
+ * @param setsum
+ * pointer of a setsummary
+ * @param key
+ * pointer of the key that needs to lookup
Better something like this "Pointer of the key to be looked up"
Quoted 6 lines
+ * @param
set_id
+ * output the set id matches the key
+ * @return
+ * return 1 for found a match and 0 for not found a match
+ */
+
Definitions of the fields should start with capital letter.
lib/librte_member/rte_member.h line 338
Quoted 10 lines
lib/librte_member/rte_member.h new file +10
-0+int +rte_member_lookup (const void *setsum, +const void *key,MEMBER_SET_TYPE *set_id); + + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* lookup bulk of keys in set-summary (SS).
"Lookup bulk"
Quoted 5 lines
+ * Each key lookup returns as soon as the first match found
+ * @param setsum
+ * Pointer of a setsummary
+ * @param keys
+ * Pointer of bulk of keys that to be lookup
"Pointer of bulk of keys to be looked up".
Check for this in other parts of the code.
Quoted 9 lines
+ * @param
num_keys
+ * Number of keys that will be lookup
+ * @paramset_ids+ * Output set ids for all the keys to this array. + * User should preallocate array that can contain all results, which size is + * the num_keys.+ * @return
+ * The number of keys that found a match
+ */
lib/librte_member/rte_member.h line 362
Quoted 10 lines
lib/librte_member/rte_member.h new file +10
-0+ + +int +rte_member_lookup_bulk (const void *setsum, +const void **keys,uint32_t num_keys, +MEMBER_SET_TYPE *set_ids); + + + +
Too many blank spaces. Generally leave one between functions. Check for this in the rest of the files.
Quoted 14 lines
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Lookup a bulk of keys in set-summary (SS) for multiple matches each key.
+ * Each key lookup will find all matched entries (multiple match).
+ * Note that for cache mode HT, each key can have at most one match. So
+ * multi-match function is mainly used for vBF and non-cache mode HT.
+ * @param setsum
+ * pointer of a setsummary
+ * @param keys
+ * The keys that to be lookup
+ * @paramnum_keys
Quoted 8 lines
+ * The number of keys that will be lookup
+ * @parammax_match_per_key
+ * The possible maximum number of matches for each key
+ * @parammatch_count
+ * Output the number of matches for each key in an array
+ * @paramset_ids
+ * Return set ids for all the matches of all keys. User pass in a
preallocated
"User passes in"
Quoted 13 lines
+ * 2D array with first dimension as key index and second dimension as
match
+ * index. For example set_ids[bulk_size][max_match_per_key]
+ * @return
+ * The number of keys that found one or more matches in the set-
summary
+ */lib/librte_member/rte_member.h new file +6
-0+ +int +rte_member_lookup_multi_bulk(const void *setsum, + const void **keys, uint32_t num_keys, + uint32_t max_match_per_key, + uint32_t *match_count,
lib/librte_member/rte_member.h line 435
Quoted 10 lines
lib/librte_member/rte_member.h new file +10
-0+ MEMBER_SET_TYPE *set_ids); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Insert key into set-summary (SS). +* +* @param setsum +* pointer of a set-summary
Start with capital letter.
Quoted 17 lines
+ * @param key
+ * the key that needs to be added
+ * @paramset_id
+ * The set id associated with the key that needs to be added. Different
mode+ * supports different set_id ranges. 0 cannot be used as set_id since + * RTE_MEMBER_NO_MATCH by default is set as 0. + * For HT mode, the set_id has range as [1, 0x7FFF], MSB is reserved. + * For vBF mode the set id is limited by the num_set parameter whencreate
+ * the set-summary.
+ * @return
+ * HT (cache mode) and vBF should never fail unless theset_idis not in
the
+ * valid range. In such case-EINVALis returned.
+ * For HT (non-cache mode) it could fail with-ENOSPCerror code when
table is
Quoted 5 lines
+ * full. + * For success it returns different values for different modes to provide + * extra information for users. + * Return 0 for HT (cache mode) if the add does not cause + * eviction, return 1 otherwise. Return 0 for HT mode if success, -ENOSPC
For HT non-cache mode?
Quoted 9 lines
for
+ * full, and 1 if cuckoo eviction happens. Always return 0 for vBF mode.
+ */lib/librte_member/rte_member.h new file +6
-0+ +int +rte_member_add (const void *setsum,const void *key, +MEMBER_SET_TYPE set_id); + +
...
Quoted 10 lines
lib/librte_member/rte_member_version.map new file +3
-0@@ -0,0 +1,16 @@ +DPDK_17.11 { + global: +
lib/librte_member/rte_member_version.map line 13
Quoted 10 lines
lib/librte_member/rte_member_version.map new file +10
-0+ rte_member_create; + rte_member_find_existing; + rte_member_lookup; + rte_member_lookup_bulk; + rte_member_lookup_multi; + rte_member_lookup_multi_bulk; + rte_member_add; + rte_member_free; + rte_member_reset; + rte_member_delete;
This list must be in alphabetical order.
Quoted 5 lines
lib/librte_member/rte_member_version.map new file +3
-1+ + local: *; +}; --2.7.4
PATCH v3 2/7 member: implement HT mode
In reply to Yipeng WangOne of the set-summray structure is hash-table based set-summary (HTSS). One example is cuckoo filter [1].
Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.
Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is more similar to traditional cuckoo filter. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.
The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. So the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.
This patch add the implementation of HTSS.
[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_ht.c | 490 ++++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_ht.h | 115 +++++++++ 3 files changed, 606 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_ht.c new file +490 -0
490 lines not on this page · raw message
lib/librte_member/rte_member_ht.h new file +115 -0
115 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v3 3/7 member: implement vBF mode
In reply to Yipeng WangBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.
Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set arbitrarily so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_vbf.c | 350 +++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_vbf.h | 85 +++++++++ 3 files changed, 436 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_vbf.c new file +350 -0
350 lines not on this page · raw message
lib/librte_member/rte_member_vbf.h new file +85 -0
85 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v3 4/7 member: add AVX for HT mode
In reply to Yipeng WangFor key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.
This patch adds AVX2 implementation in a separate header file.
--- lib/librte_member/rte_member_ht.c | 143 ++++++++++++++++++++++++++++--------- lib/librte_member/rte_member_x86.h | 111 ++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+), 32 deletions(-) create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -32
@@ -40,6 +40,10 @@#include "rte_member.h" #include "rte_member_ht.h" +#if defined(RTE_ARCH_X86 ) +#include "rte_member_x86.h" +#endif + static inline int insert_overwrite_search(uint32_t bucket, SIG_TYPE tmp_sig,
@@ -135,6 +139,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) buckets[i].sets[j] =RTE_MEMBER_NO_MATCH ; } +#if defined(RTE_ARCH_X86 ) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +RTE_MEMBER_BUCKET_ENTRIES ==16 ) + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_AVX2 ; + else +#endif + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_SCALAR ; RTE_LOG(DEBUG, MEMBER, "Hash table based filter created, "
@@ -174,11 +185,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss, *set_id =RTE_MEMBER_NO_MATCH ;get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -if (search_bucket_single (prim_bucket, tmp_sig, buckets, - set_id) || -search_bucket_single (sec_bucket, tmp_sig, - buckets, set_id)) -return 1 ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single_avx (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; +break ; +#endif +default : +if (search_bucket_single (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; + }return 0 ; }
@@ -203,13 +226,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss, }for (i =0 ; i < num_keys; i++) { -if (search_bucket_single (prim_buckets[i], tmp_sig[i], - buckets, &set_id[i]) || -search_bucket_single (sec_buckets[i], - tmp_sig[i], buckets, &set_id[i])) - ret++; -else - set_id[i] =RTE_MEMBER_NO_MATCH ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_buckets[i], + tmp_sig[i], buckets, &set_id[i]) || +search_bucket_single_avx (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; +break ; +#endif +default : +if (search_bucket_single (prim_buckets[i], tmp_sig[i], + buckets, &set_id[i]) || +search_bucket_single (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; + } }return ret; }
@@ -227,12 +264,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, - match_per_key, set_id); -if (ret < match_per_key) -search_bucket_multi (sec_bucket, tmp_sig, - buckets, &ret, match_per_key, set_id); -return ret; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_bucket, tmp_sig, buckets, + &ret, match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi_avx (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; +#endif +default : +search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, + match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; + } }
@@ -259,16 +308,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,for (i =0 ; i < num_keys; i++) { match_cnt_t =0 ; -search_bucket_multi (prim_buckets[i], tmp_sig[i], - buckets, &match_cnt_t, match_per_key, - &set_ids[i*match_per_key]); -if (match_cnt_t < match_per_key) -search_bucket_multi (sec_buckets[i], tmp_sig[i], +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_buckets[i], tmp_sig[i], buckets, &match_cnt_t, match_per_key, &set_ids[i*match_per_key]); - match_count[i] = match_cnt_t; -if (match_cnt_t !=0 ) - ret++; +if (match_cnt_t < match_per_key) +search_bucket_multi_avx (sec_buckets[i], + tmp_sig[i], buckets, &match_cnt_t, + match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_t; +if (match_cnt_t !=0 ) + ret++; +break ; +#endif +default : +search_bucket_multi (prim_buckets[i], tmp_sig[i], + buckets, &match_cnt_t, match_per_key, + &set_ids[i*match_per_key]); +if (match_cnt_t < match_per_key) +search_bucket_multi (sec_buckets[i], tmp_sig[i], + buckets, &match_cnt_t, match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_t; +if (match_cnt_t !=0 ) + ret++; + } }return ret; }
@@ -300,12 +367,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,static inline int try_overwrite (struct member_ht_bucket *buckets,uint32_t prim,uint32_t sec, -SIG_TYPE sig,MEMBER_SET_TYPE set_id) +SIG_TYPE sig,MEMBER_SET_TYPE set_id, +enum rte_member_sig_compare_function cmp_fn) { -if (insert_overwrite_search (prim, sig, buckets, set_id) || -insert_overwrite_search (sec, sig, buckets, - set_id)) -return 0 ; +switch (cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (insert_overwrite_search_avx (prim, sig, buckets, set_id) || +insert_overwrite_search_avx (sec, sig, buckets, + set_id)) +return 0 ; +break ; +#endif +default : +if (insert_overwrite_search (prim, sig, buckets, set_id) || +insert_overwrite_search (sec, sig, buckets, + set_id)) +return 0 ; + }return -1 ; }
@@ -411,7 +490,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,/* if it is cache based filter, we try overwriting existing entry */ if (ss->cache) { ret =try_overwrite (buckets, prim_bucket, sec_bucket, tmp_sig, - set_id); + set_id, ss->sig_cmp_fn);if (ret !=-1 )return ret; }
lib/librte_member/rte_member_x86.h new file +111 -0
111 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v3 5/7 member: enable the library
In reply to Yipeng WangThis patch enables the Membership library.
--- MAINTAINERS | 7 +++++++ config/common_base | 5 +++++ lib/librte_member/Makefile | 2 ++ mk/rte.app.mk | 2 ++ 4 files changed, 16 insertions(+)
MAINTAINERS +7 -0
@@ -734,6 +734,13 @@ F: test/test/test_meter.c F: examples/qos_meter/ F: doc/guides/sample_app_ug/qos_metering.rst +Membership - EXPERIMENTAL +M: Yipeng Wang <[email protected]> +M: Sameh Gobriel <[email protected]> +F: lib/librte_member/ +F: doc/guides/prog_guide/member_lib.rst +F: test/test/test_member* + Other libraries ---------------
config/common_base +5 -0
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n CONFIG_RTE_LIBRTE_EFD=y # +# Compile librte_member +# +CONFIG_RTE_LIBRTE_MEMBER=y + +# # Compile librte_jobstats # CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
@@ -37,6 +37,8 @@ LIB = librte_member.a CFLAGS := -I$(SRCDIR) $(CFLAGS) CFLAGS += $(WERROR_FLAGS) -O3 +LDLIBS += -lm + EXPORT_MAP := rte_member_version.map LIBABIVER := 1
mk/rte.app.mk +2 -0
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile _LDLIBS-y += --whole-archive _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt _LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma endif
Signature · 1 line
2.7.4
Re: [PATCH v3 5/7] member: enable the library
06/09/2017 01:59, Yipeng Wang:
MAINTAINERS F: test/test/test_meter.c · line 742
Quoted 12 lines
MAINTAINERS +6
-0@@ -734,6 +734,13 @@ F: test/test/test_meter.c F: examples/qos_meter/ F: doc/guides/sample_app_ug/qos_metering.rst +Membership - EXPERIMENTAL +M: Yipeng Wang <[email protected]> +M: Sameh Gobriel <[email protected]> +F: lib/librte_member/ +F: doc/guides/prog_guide/member_lib.rst +F: test/test/test_member*
I know it is not obvious but this section is sorted in alphabetical order. Please move between Hashes and LPM. Thanks
PATCH v3 6/7 test/member: add functional and perf tests
In reply to Yipeng WangThis patch adds functional and performance tests for membership library.
--- test/test/Makefile | 3 + test/test/test_member.c | 682 +++++++++++++++++++++++++++++++++++++++++++ test/test/test_member_perf.c | 643 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1328 insertions(+) create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
test/test/Makefile +3 -0
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c SRCS-y += test_memcpy.c SRCS-y += test_memcpy_perf.c + +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
test/test/test_member.c new file +682 -0
682 lines not on this page · raw message
test/test/test_member_perf.c new file +643 -0
643 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v3 7/7 doc: add membership documentation
In reply to Yipeng WangThis patch adds the documentation for membership library.
--- doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 ++++ doc/guides/prog_guide/img/member_i6.svg | 332 +++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 421 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + 12 files changed, 3596 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst
doc/api/doxy-api-index.md +2 -1
@@ -105,7 +105,8 @@ The public API headers are grouped by topics: [LPM IPv4 route] (@ref rte_lpm.h), [LPM IPv6 route] (@ref rte_lpm6.h), [ACL] (@ref rte_acl.h), - [EFD] (@ref rte_efd.h) + [EFD] (@ref rte_efd.h), + [member] (@ref rte_member.h) - **QoS**: [metering] (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
@@ -58,6 +58,7 @@ INPUT = doc/api/doxy-api-index.md \ lib/librte_mempool \ lib/librte_meter \ lib/librte_metrics \ + lib/librte_member \ lib/librte_net \ lib/librte_pdump \ lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0
1,613 lines not on this page · raw message
doc/guides/prog_guide/img/member_i2.svg new file +36 -0
@@ -0,0 +1,36 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in" + viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6"> + <v:documentProperties v:langID="1033" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {fill:none;stroke:none;stroke-width:0.25} + .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em} + .st3 {baseline-shift:32.4943%;font-size:0.649886em} + .st4 {font-size:1em} + .st5 {font-family:Cambria Math;font-size:1em} + .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/> + <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)"> + <title>Sheet.3</title> + <desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="157.5" cy="45.5" width="315" height="90"/> + <rect x="0" y="0.5" width="315" height="90" class="st1"/> + <text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan + dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan + dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan + class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3" + v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em" + class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text> </g> + </g> +</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0
148 lines not on this page · raw message
doc/guides/prog_guide/img/member_i4.svg new file +450 -0
450 lines not on this page · raw message
doc/guides/prog_guide/img/member_i5.svg new file +163 -0
163 lines not on this page · raw message
doc/guides/prog_guide/img/member_i6.svg new file +332 -0
332 lines not on this page · raw message
doc/guides/prog_guide/img/member_i7.svg new file +399 -0
242 lines not on this page · raw message
doc/guides/prog_guide/member_lib.rst new file +421 -0
421 lines not on this page · raw message
doc/guides/rel_notes/release_17_11.rst +17 -0
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.7.4
Part of the diff is not shown here. View the raw message.
Re: [PATCH v3 7/7] doc: add membership documentation
Quoted 23 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 6, 2017 1:00 AM
To: [email protected]
Cc: Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>; Wang, Yipeng1 <[email protected]>; Mcnamara, John <[email protected]>
Subject: [PATCH v3 7/7] doc: add membership documentationThis patch adds the documentation for membership library.
Signed-off-by Yipeng Wang <[email protected]>Reviewed-by John McNamara <[email protected]>doc/guides/prog_guide/member_lib.rst new file +4
-0+Set-summary Element Insertion +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``rte_member_add()`` function is use to insert an element/key into a
Quoted 2 lines
set-summary structure. If it fails an
+error is returned. For success the returned value is dpendednt on the
Minor typo: s/dpendednt/dependent/
Apart from that:
Re: [PATCH v3 7/7] doc: add membership documentation
In reply to Yipeng WangQuoted 16 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Yipeng Wang
Sent: Wednesday, September 6, 2017 1:00 AM
To: [email protected]
Cc: Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Wang, Ren <[email protected]>; Wang, Yipeng1 <[email protected]>; Mcnamara, John <[email protected]>
Subject: [dpdk-dev] [PATCH v3 7/7] doc: add membership documentationThis patch adds the documentation for membership library.
Signed-off-by Yipeng Wang <[email protected]>Reviewed-by John McNamara <[email protected]>
...
Quoted 15 lines
doc/guides/prog_guide/member_lib.rst new file +7
-0+Membership Library +================== + +Introduction +------------ + +The DPDK Membership Library provides an API for DPDK applications to
Quoted 4 lines
insert a
+new member, delete an existing member, or query the existence of a member in a
+given set, or a group of sets. For the case of a group of sets the library
"group of sets, the library".
...
Quoted 1 line
+We are including a configurable Membership Library in DPDK to cover set
This looks like more like a cover letter than part of programmer's guide. Reword this loke "This Membership Library covers set membership...", and avoid "include... in DPDK", as it is implicit.
Quoted 23 lines
+membership functionality for both a single set and multi-set scenarios. The
+library is optimized to support the customer network applications which require
+membership checking functionality. In this guide, we will cover two set- summarydoc/guides/prog_guide/member_lib.rst new file +2
-0+schemes including a vector of Bloom Filters and Hash-Table based +set-summary schemes with and without false negative probability,followed by
doc/guides/prog_guide/member_lib.rst new file +7
-0+a brief discussion of the Membership Library API. + +Vector of Bloom Filters +----------------------- + +Bloom Filter (BF) [Member-bloom] is a well-known space-efficient +probabilistic data structure that answers set membership queries (testwhether
+an element is a member of a set) with some probability of false positives anddoc/guides/prog_guide/member_lib.rst new file +4
-0+zero false negatives; a query for an element returns either it is "possibly in +a set" (with very high probability) or "definitely not in a set". + +The BF is a method for representing a set of ``n`` elements (for example
Quoted 8 lines
flow keys
+in network applications domain) to support membership queries. The idea of BF is
+to allocate a bit-vector ``v`` with ``m`` bits, which are initially all set to 0. Then
+it chooses ``k`` independent hash functions ``h1``, ``h2``, ... ``hk`` with hash values range from
+``1`` to ``m`` to perform hashing calculations on each element.
From "0" to "m-1"?
...
Quoted 21 lines
doc/guides/prog_guide/member_lib.rst new file +8
-0+ + Vector Bloom Filter (vBF) Overview + +To support membership test for both multiple sets and a single set, +the library implements a Vector Bloom Filter (vBF) scheme. +vBF basically composes multiple bloom filters into a vector of bloom filers. +The membership test is conducted on all of the +bloom filters concurrently to determine which set(s) it belongs to or noneof
+them. The basic idea of vBF is shown in the above figure where an element isdoc/guides/prog_guide/member_lib.rst new file +9
-0+used to address multiple bloom filters concurrently and the bloom filter +index(es) with a hit is returned. + +.. _figure_membership5: +.. figure:: img/member_i5.* + + vBF for Flow Scheduling to Worker Thread + +As previously mentioned, there are many usages of such structures. vBF isused
doc/guides/prog_guide/member_lib.rst
Quoted 2 lines
doc/guides/prog_guide/member_lib.rst new file +2
-0+for applications that needs to check membership against multiple sets +simultaneously.
"needs" -> "need".
...
doc/guides/prog_guide/member_lib.rst line 267
Quoted 6 lines
doc/guides/prog_guide/member_lib.rst new file +6
-0+The major usage of HTSS with false negative is to use it as a cache for +distributing elements to different target sets. By allowing HTSS to evict old +elements, the set-summary can keep track of the most recent elements +(i.e. active) as a cache typically does. Old inactive elements (infrequently +used elements) will automatically and eventually get evicted from the +set-summary. It worth noting that the set-summary still has false positive
"It is worth".
...
Quoted 6 lines
+We also pass two seeds: ``prim_hash_seed`` and +``sec_hash_seed`` for the primary and secondary hash functions to calculate two independent hash values.
+``socket_id`` parameter is the NUMA socket ID for the memory used to create the
+set-summary. For HTSS, another parameter ``iscache`` is used to indicate
In order to keep consistency, I would use "is_cache".
Quoted 30 lines
doc/guides/prog_guide/member_lib.rst new file +2
-0+if this set-summary is a cache (i.e. with false negative probability) or not. +For vBF, extra parameters are needed. For example, ``num_set`` is thenumber of
+sets needed to initialize the vector bloom filters. This number is equal to the
+number of bloom filters will be created.
+``false_pos_rate`` is the false positive rate.num_keysandfalse_pos_ratewill be used to determinedoc/guides/prog_guide/member_lib.rst new file +7
-0+the number of hash functions and the bloom filter size. + + +Set-summary Element Insertion +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``rte_member_add()`` function is use to insert an element/key into aset-summary structure. If it fails an
doc/guides/prog_guide/member_lib.rst new file +3
-0+error is returned. For success the returned value is dpendednt on the +set-summary mode to provide extra information for the users. For vBF +mode, a return value of 0 means a successful insert. For HTSS modewithout false negative, the insert
+could fail with ``-ENOSPC`` if the table is full. With false negative (i.e. cache mode),
+for insert that does not cause any eviction (i.e. no overwriting happens to an
+existing entry) the return value is 0. For insertion that causes eviction, the returndoc/guides/prog_guide/member_lib.rst new file +3
-0+value is 1 to indicate such situation, but it is not an error. + +The input arguments for the function should include the ``key`` which is apointer to the element/key that needs to
doc/guides/prog_guide/member_lib.rst line 350
Quoted 6 lines
doc/guides/prog_guide/member_lib.rst new file +6
-0+be added to the set-summary, and ``set_id`` which is the set id associated +with the key that needs to be added. + + +Set-summary Element Lookup +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Remove unneeded last "~" (I see 4 extra at the end).
Quoted 29 lines
doc/guides/prog_guide/member_lib.rst new file +2
-0+ +The ``rte_member_lookup()`` function looks up a single key/element inthe set-summary structure. It
doc/guides/prog_guide/member_lib.rst new file +2
-0+returns as soon as the first match is found. The return value is 1 if a +match is found and 0 otherwise. The arguments for the function include``key`` which is a pointer to the
+element/key that needs to be looked up, and ``set_id`` which is used to return thedoc/guides/prog_guide/member_lib.rst new file +3
-0+first target set id where the key has matched, if any. + +The ``rte_member_lookup_bulk()`` function is used to look up a bulk ofkeys/elements in the
+set-summary structure for their first match. Each key lookup returns as soon as the first match is found. Thedoc/guides/prog_guide/member_lib.rst new file +2
-0+return value is the number of keys that find a match. The arguments of the +function include ``keys`` which is a pointer to a bulk of keys that are to belooked up,
+``num_keys`` is the numberdoc/guides/prog_guide/member_lib.rst new file +2
-0+of keys that will be looked up, and ``set_ids`` are the return target set +ids for the first match found for each of the input keys. ``set_ids`` is anarray
+needs to be sized according to the ``num_keys``. If there is no match, the set iddoc/guides/prog_guide/member_lib.rst new file +3
-0+for that key will be set to RTE_MEMBER_NO_MATCH. + +The ``rte_member_lookup_multi()`` function looks up a singlekey/element in the
doc/guides/prog_guide/member_lib.rst new file +2
-0+set-summary structure for multiple matches. It +returns ALL the matches (possibly more than one) found for this key when
Quoted 3 lines
it
+is matched against all target sets (it worth noting that for cache mode HTSS,
"it is worth"
Quoted 7 lines
+the current implementation matches at most one target set). The return value is
doc/guides/prog_guide/member_lib.rst new file +3
-0+the number of matches +that was found for this key (for cache mode HTSS the return value +should be at most 1). The arguments for the function include ``key`` whichis a pointer to the
+element/key that needs to be looked up, ``match_per_key`` which is to
Better to use "max_match_per_key". Will comment more in other patches.
Quoted 3 lines
indicate maximum number of matches
+the user expect to find for each key, and ``set_id`` which is used to return all
"user expects to".
Quoted 4 lines
+target set ids where the key has matched, if any. The ``set_id`` array should be sized
+according to ``match_per_key``. For vBF, maximum number of matches per key is equal
"For vBF, the maximum number".
Quoted 3 lines
+to the number of sets. For HTSS, maximum number of matches per key is equal to two times
+entry count per bucket. ``match_per_key`` should be equal or smaller than
"two time the entry count".
Quoted 15 lines
maximum number of
doc/guides/prog_guide/member_lib.rst new file +3
-0+possible matches. + +The ``rte_membership_lookup_multi_bulk()`` function looks up a bulk ofkeys/elements elements in the
+set-summary structure for multiple matches, each key lookup returns ALL the matches (possibly more
+than one) found for this key when it is matched against all target sets (cache mode HTSSdoc/guides/prog_guide/member_lib.rst new file +5
-0+matches at most one target set). The +return value is the number of keys that find one or more matches in the +set-summary structure. The arguments of the +function include ``keys`` which is +a pointer to a bulk of keys that are to be looked up, ``num_keys`` is thenumber
doc/guides/prog_guide/member_lib.rst
Quoted 2 lines
doc/guides/prog_guide/member_lib.rst new file +2
-0+of keys that will be looked up, ``match_per_key`` is the possible +max number of matches for each key, ``match_count`` which is the
Use "maximum number", no need to abbreviate here.
Quoted 5 lines
returned number
doc/guides/prog_guide/member_lib.rst new file +3
-0+of matches for each key, and ``set_ids`` are the returned target set +ids for all matches found for each keys. ``set_ids`` is 2-D array +that for each key, a 1-D array should be sized according to``match_per_key``.
I would reword it a bit, to improve readability. For instance:
"set_ids" is a 2-D array, containing a 1-D array per key, which should be sized according to "match_per_key".
Quoted 8 lines
+``match_per_key`` should be equal or smaller than maximum number of
doc/guides/prog_guide/member_lib.rst new file +7
-0+possible matches, similar to ``rte_member_lookup_multi``. + + +Set-summary Element Delete +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``rte_membership_delete()`` function deletes an element/key from a
Quoted 7 lines
set-summary structure, if it fails
+an error is returned. The input arguments should include ``key`` which is a pointer to the
+element/key that needs to be deleted from the set-summary, and ``set_id``
+which is the set id associated with the key to delete. It worth noting that current
"It is worth".
Quoted 2 lines
+implementation of vBF does not support deletion [1]_. An error code ``- EINVAL`` will be returned.
v4 cover Add Membership Library
In reply to Yipeng WangDPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).
The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.
The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.
A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.
This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.
[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.
[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
v4:
- member lib: change two hash function macros to be one. crc or jhash will be chosen depending on the architecture.
- member lib: use dynamic log type instead of static one by Thomas' comment.
- member lib: code cleanups (remove unnecessary code, change variable name, coding style, etc).
- member lib: reorder members in setsummary struct to be more cache friendly.
- member lib: setsummary struct member "name" should be char array rather than char pointer. Fixed.
- member lib: add check for two hash seeds to make sure they are not equal.
- member lib: vBF lookup speed optimization.
- member lib: revise comments in various places as Pablo suggested.
- member lib: change "void *" to "struct rte_member_setsum *" in public API as Pablo suggested. test case is modified accordingly to directly use the struct.
- member lib: change order in rte_member_version.map as Pablo suggested.
- MAINTAINERS: change the maintainer block order according to Thomas' comment.
- test: add printf to show which section expects error messages. Change set count to 16 from 32 for performance test.
- documentation: revise several places according to Pablo's and John's comments.
v3:
- documentation: update documentation to incorporate John's review.
v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitialized issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment.
- documentation: update the programmer guide for membership library.
Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation
MAINTAINERS | 8 +- config/common_base | 5 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 +++ doc/guides/prog_guide/img/member_i6.svg | 332 ++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 420 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + lib/Makefile | 2 + lib/librte_member/Makefile | 51 + lib/librte_member/rte_member.c | 336 +++++++ lib/librte_member/rte_member.h | 507 ++++++++++ lib/librte_member/rte_member_ht.c | 553 ++++++++++ lib/librte_member/rte_member_ht.h | 94 ++ lib/librte_member/rte_member_vbf.c | 349 +++++++ lib/librte_member/rte_member_vbf.h | 79 ++ lib/librte_member/rte_member_version.map | 16 + lib/librte_member/rte_member_x86.h | 107 ++ mk/rte.app.mk | 2 + test/test/Makefile | 3 + test/test/test_member.c | 676 +++++++++++++ test/test/test_member_perf.c | 630 ++++++++++++ 28 files changed, 7012 insertions(+), 2 deletions(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h create mode 100644 lib/librte_member/rte_member_version.map create mode 100644 lib/librte_member/rte_member_x86.h create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
PATCH v4 1/7 member: implement main API
Membership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.
This patch adds the main API definition.
--- lib/Makefile | 2 + lib/librte_member/Makefile | 49 +++ lib/librte_member/rte_member.c | 336 ++++++++++++++++++++ lib/librte_member/rte_member.h | 507 +++++++++++++++++++++++++++++++ lib/librte_member/rte_member_version.map | 16 + 5 files changed, 910 insertions(+) create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether +DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member +DEPDIRS-librte_member := librte_eal librte_hash ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_member/Makefile new file +49 -0
49 lines not on this page · raw message
lib/librte_member/rte_member.c new file +336 -0
336 lines not on this page · raw message
lib/librte_member/rte_member.h new file +507 -0
507 lines not on this page · raw message
lib/librte_member/rte_member_version.map new file +16 -0
@@ -0,0 +1,16 @@ +DPDK_17.11 { + global: + + rte_member_add; + rte_member_create; + rte_member_delete; + rte_member_find_existing; + rte_member_free; + rte_member_lookup; + rte_member_lookup_bulk; + rte_member_lookup_multi; + rte_member_lookup_multi_bulk; + rte_member_reset; + + local: *; +};
Signature · 1 line
2.7.4
Re: [PATCH v4 1/7] member: implement main API
Hi Yipeng,
Quoted 17 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:40 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 1/7] member: implement main APIMembership library is an extension and generalization of a traditional filter (for example Bloom Filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set- membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
Quoted 8 lines
The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.This patch adds the main API definition.
Signed-off-by Yipeng Wang <[email protected]>
A few comments on changes that you didn't make in the v4.
Thanks,
Pablo
...
Quoted 18 lines
lib/librte_member/rte_member.c new file +15
-0+ +struct rte_member_setsum * +rte_member_create(const struct rte_member_parameters *params) { + struct rte_tailq_entry *te; + struct rte_member_list *member_list; + struct rte_member_setsum *setsum; + int ret; + + if (params == NULL) { + rte_errno = EINVAL; + return NULL; + } + + if (params->key_len == 0 || + params->prim_hash_seed == params-Quoted 1 line
sec_hash_seed) {(quoted diff) +2
-0+ rte_errno = EINVAL; + RTE_MEMBER_LOG(ERR, "Memship create with invalid
Quoted 1 line
parameters\n");
Do not use "Memship". Change to " rte_member_create has invalid parameters"? Or something else that you want, but not Memship.
Quoted 3 lines
lib/librte_member/rte_member.c new file +3
-0+return NULL ; + } +
...
Quoted 13 lines
lib/librte_member/rte_member.h new file +8
-0+struct rte_member_parameters { +const char *name;/**< Name of the hash. */ + +/** +* User to specify the type of the setsummary from one of +* rte_member_setsum_type. +* +* HT based setsummary is implemented like a hash table. User should use
lib/librte_member/rte_member.h new file +3
-0+* this type when there are many sets. +* +* vBF setsummary is a vector of bloom filters. It is used when number
lib/librte_member/rte_member.h line 198
Quoted 8 lines
lib/librte_member/rte_member.h new file +8
-0+* of sets is not big (less than 32 for current implementation). +*/ +enum rte_member_setsum_type type; + +/** +* If it is HT based setsummary, user to specify the subtype or mode +* of the setsummary. It could be cache, or non-cache mode. +* Set iscache to be 1 if to use as cache mode.
Change to "is_cache".
Quoted 23 lines
(quoted diff) +2
-0+ * + * For cache mode, keys can be evicted out of the HT setsummary.Keys
lib/librte_member/rte_member.h new file +7
-0+* with the same signature and map to the same bucket +* will overwrite each other in the setsummary table. +* This mode is useful for the case that the set-summary only +* needs to keep record of the recently inserted keys. Both +* false-negative and false-positive could happen. +* +* For non-cache mode, keys cannot be evicted out of the cache. So for
(quoted diff) +1
-0+ * this mode the setsummary will become full eventually. Keys withthe
(quoted diff) +1
-0+ * same signature but map to the same bucket will still occupymultiple
lib/librte_member/rte_member.h new file +6
-0+* entries. This mode does not give false-negative result. +*/ +uint8_t is_cache; + +/** +* For HT setsummary, num_keys equals to the number of entries of the
(quoted diff) +1
-0+ * table. When the number of keys that inserted in the HT
Quoted 1 line
setsummary
"number of keys inserted in the HT summary". Or "were inserted".
Quoted 3 lines
(quoted diff) +2
-0+ * approaches this number, eviction could happen. For cache mode, + * keys could be evicted out of the table. For non-cache mode, keyswill
...
Quoted 20 lines
lib/librte_member/rte_member.h new file +7
-0+/** +* false_positive_rate is only relevant to vBF based setsummary. +* false_positive_rate is the user-defined false positive rate +* given expected number of inserted keys (num_keys). It is used to +* calculate the total number of bits for each BF, and the number of +* hash values used during lookup and insertion. For details please +* refer to vBF implementation and membership library documentation.
lib/librte_member/rte_member.h new file +4
-0+* Note that this parameter is not directly set by users for HT mode. +* +* HT setsummary's false positive rate is in the order of: +* false_pos = (1/bucket_count)*(1/2^16), since we use 16-bit signature.
(quoted diff) +2
-0+ * This is because two keys needs to map to same bucket and same + * signature to have a collision (false positive). bucket_count isequal
(quoted diff) +1
-0+ * to number of entries (num_keys) divided by entry count perbucket
(quoted diff) +1
-0+ * (RTE_MEMBER_BUCKET_ENTRIES). Thus, the false_positive_rateis not
lib/librte_member/rte_member.h line 263
Quoted 1 line
lib/librte_member/rte_member.h new file +1
-0+* directly set by users.
Unless I understood it incorrectly, I think you should clarify that this field is not set for HT mode, but it is for vBF, right?
Quoted 75 lines
lib/librte_member/rte_member.h new file +5
-0+*/ +float false_positive_rate; + +/** +* We use two seeds to calculate two independent hashes for each key.
lib/librte_member/rte_member.h new file +4
-0+* +* For HT type, one hash is used as signature, and the other is used +* for bucket location. +* For vBF type, these two hashes and their combinations are used as
lib/librte_member/rte_member.h new file +6
-0+* hash locations to index the bit array. +*/ +uint32_t prim_hash_seed; + +/** +* The secondary seed should be a different value from the primary seed.
lib/librte_member/rte_member.h new file +4
-0+*/ +uint32_t sec_hash_seed; + +int socket_id;/**< NUMA Socket ID for memory. */
lib/librte_member/rte_member.h new file +51
-0+}; + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Find an existing set-summary and return a pointer to it. +* +* @param name +* Name of the set-summary. +* @return +* Pointer to the set-summary or NULL if object not found +* with rte_errno set appropriately. Possible rte_errno values include: +* - ENOENT - value not available for return +*/ +struct rte_member_setsum * +rte_member_find_existing (const char *name); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Create set-summary (SS). +* +* @param params +* Parameters to initialize the setsummary. +* @return +* Return the pointer to the setsummary. +* Return value is NULL if the creation failed. +*/ +struct rte_member_setsum * +rte_member_create (const struct rte_member_parameters *params); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* Lookup key in set-summary (SS). +* Single key lookup and return as soon as the first match found +* +* @param setsum +* Pointer of a setsummary. +* @param key +* Pointer of the key to be looked up. +* @param set_id +* Output the set id matches the key. +* @return +* Return 1 for found a match and 0 for not found a match. +*/ +int +rte_member_lookup(const struct rte_member_setsum *setsum, constvoid *key,
lib/librte_member/rte_member.h line 340
Quoted 7 lines
lib/librte_member/rte_member.h new file +7
-0+ member_set_t *set_id); + +/** +* @warning +* @b EXPERIMENTAL: this API may change without prior notice +* +* lookup bulk of keys in set-summary (SS).
"Lookup bulk"
Quoted 13 lines
+ * Each key lookup returns as soon as the first match found
+ *
+ * @param setsum
+ * Pointer of a setsummary.
+ * @param keys
+ * Pointer of the bulk of keys to be looked up.
+ * @paramnum_keys
+ * Number of keys that will be lookup.
+ * @paramset_ids+ * Output set ids for all the keys to this array. + * User should preallocate array that can contain all results, which size is + * the num_keys.+ * @return
PATCH v4 2/7 member: implement HT mode
In reply to Yipeng WangOne of the set-summary structures is hash-table based set-summary (HTSS). One example is cuckoo filter [1].
Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.
Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is more similar to traditional cuckoo filter. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.
The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. Thus, the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.
This patch adds the implementation of HTSS.
[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_ht.c | 474 ++++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_ht.h | 94 ++++++++ 3 files changed, 569 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_ht.c new file +474 -0
474 lines not on this page · raw message
lib/librte_member/rte_member_ht.h new file +94 -0
94 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH v4 2/7] member: implement HT mode
Quoted 12 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:40 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 2/7] member: implement HT mode
...
Quoted 20 lines
lib/librte_member/Makefile index renamed from lib/librte_member/Makefile +0
-01a79eaa..ad26548 100644
lib/librte_member/Makefile +0
-0@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.mapLIBABIVER := 1
# all source are stored in SRCS-ylib/librte_member/Makefile +1
-1-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c
rte_member_ht.c# install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.hlib/librte_member/rte_member_ht.c new file +0
-0
...
lib/librte_member/rte_member_ht.c line 46
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ +static inline int +insert_overwrite_search (uint32_t bucket,member_sig_t tmp_sig, +struct member_ht_bucket *buckets, +member_set_t set_id)
I would call "bucket", "bucket_id", for better understanding. This comment also applies to other parts of the code (e.g. prim_buckets).
lib/librte_member/rte_member_ht.c line 53
Quoted 7 lines
lib/librte_member/rte_member_ht.c new file +7
-0+{ +int i; +for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +if (buckets[bucket].sigs[i] == tmp_sig) { + buckets[bucket].sets[i] = set_id; +return 1 ; + }
Is this function used to update an existing entry?
At first, I thought that this was evicting another entry, when the bucket was full, but it looks like it is updating the set_id of an existing entry.
If this is the case, I would change the function name and add a comment explaining this.
lib/librte_member/rte_member_ht.c line 63
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+ } +return 0 ; +} + +static inline int +search_bucket_single (uint32_t bucket,member_sig_t tmp_sig, +struct member_ht_bucket *buckets, +member_set_t *set_id) +{ +int iter;
Iter should be "unsigned int" (or similar, maybe "uint8_t")
Quoted 6 lines
lib/librte_member/rte_member_ht.c new file +6
-0+for (iter =0 ; iter <RTE_MEMBER_BUCKET_ENTRIES ; iter++) { +if (tmp_sig == buckets[bucket].sigs[iter] && + buckets[bucket].sets[iter] != +RTE_MEMBER_NO_MATCH ) { + *set_id = buckets[bucket].sets[iter]; +return 1 ;
lib/librte_member/rte_member_ht.c line 79
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+ } + } +return 0 ; +} + +static inline void +search_bucket_multi(uint32_t bucket, member_sig_t tmp_sig, + struct member_ht_bucket *buckets, + uint32_t *counter, + uint32_t match_per_key,
Better change to "matches_per_key".
Quoted 18 lines
lib/librte_member/rte_member_ht.c new file +18
-0+ member_set_t *set_id) +{ +int iter; +for (iter =0 ; iter <RTE_MEMBER_BUCKET_ENTRIES ; iter++) { +if (tmp_sig == buckets[bucket].sigs[iter] && + buckets[bucket].sets[iter] != +RTE_MEMBER_NO_MATCH ) { + set_id[*counter] = buckets[bucket].sets[iter]; + (*counter)++; +if (*counter >= match_per_key) +return ; + } + } +} + +int +rte_member_create_ht (struct rte_member_setsum *ss, +const struct rte_member_parameters *params) {
...
lib/librte_member/rte_member_ht.c line 139
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ +RTE_MEMBER_LOG (DEBUG ,"Hash table based filter created, " +"the table has %u entries, %u buckets\n" , + num_buckets, + num_buckets /RTE_MEMBER_BUCKET_ENTRIES );
Shouldn't this be "num_buckets * RTE_MEMBER_BUCKET_ENTRIES" and "num_buckets"?
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +5
-0+return 0 ; +} + +static inline +void get_buckets_index(const struct rte_member_setsum *ss, const void*key,
lib/librte_member/rte_member_ht.c new file +1
-0+ uint32_t *prim_bkt, uint32_t *sec_bkt, member_sig_t*sig) {
"static inline void" should be in the same line.
lib/librte_member/rte_member_ht.c line 153
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +3
-0+uint32_t first_hash =MEMBER_HASH_FUNC (key, ss->key_len, + ss->prim_hash_seed); + uint32_t sec_hash = MEMBER_HASH_FUNC(&first_hash,
sizeof(uint32_t),lib/librte_member/rte_member_ht.c new file +4
-0+ ss->sec_hash_seed); + *sig = first_hash; +if (ss->cache) { + *prim_bkt = sec_hash & ss->bucket_mask;
Is this correct? Using the secondary hash to acces the primary bucket? Why is the cache case different from the non-cache case?
I think this function deserves some comments to explain all this calculations.
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ *sec_bkt = (sec_hash >>16 ) & ss->bucket_mask; + }else { + *prim_bkt = sec_hash & ss->bucket_mask; + *sec_bkt = (*prim_bkt ^ *sig) & ss->bucket_mask; + }
lib/librte_member/rte_member_ht.c line 169
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+} + +int +rte_member_lookup_ht (const struct rte_member_setsum *ss, +const void *key,member_set_t *set_id) { +uint32_t prim_bucket, sec_bucket; +member_sig_t tmp_sig; +struct member_ht_bucket *buckets = ss->table; + +
Remove extra blank line.
Quoted 17 lines
lib/librte_member/rte_member_ht.c new file +2
-0+ *set_id =RTE_MEMBER_NO_MATCH ; + get_buckets_index(ss, key, &prim_bucket, &sec_bucket,
&tmp_sig);lib/librte_member/rte_member_ht.c new file +13
-0+ +if (search_bucket_single (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; + +return 0 ; +} + +uint32_t +rte_member_lookup_bulk_ht (const struct rte_member_setsum *ss, + const void **keys, uint32_t num_keys, member_set_t
*set_id) {
lib/librte_member/rte_member_ht.c
Quoted 2 lines
lib/librte_member/rte_member_ht.c new file +2
-0+uint32_t i; +uint32_t ret =0 ;
Better change to something more meaningful, like "nr_matches".
Quoted 20 lines
lib/librte_member/rte_member_ht.c new file +20
-0+struct member_ht_bucket *buckets = ss->table; +member_sig_t tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX ]; +uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX ]; +uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX ]; + +for (i =0 ; i < num_keys; i++) { +get_buckets_index (ss, keys[i], &prim_buckets[i], + &sec_buckets[i], &tmp_sig[i]); +rte_prefetch0 (&buckets[prim_buckets[i]]); +rte_prefetch0 (&buckets[sec_buckets[i]]); + } + +for (i =0 ; i < num_keys; i++) { +if (search_bucket_single (prim_buckets[i], tmp_sig[i], + buckets, &set_id[i]) || +search_bucket_single (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ;
lib/librte_member/rte_member_ht.c line 217
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+ } +return ret; +} + +uint32_t +rte_member_lookup_multi_ht (const struct rte_member_setsum *ss, +const void *key,uint32_t match_per_key, +member_set_t *set_id) +{ +uint32_t ret =0 ;
Better change to something more meaningful, like "nr_matches". This applies to the following functions.
Quoted 15 lines
lib/librte_member/rte_member_ht.c new file +5
-0+uint32_t prim_bucket, sec_bucket; +member_sig_t tmp_sig; +struct member_ht_bucket *buckets = ss->table; + + get_buckets_index(ss, key, &prim_bucket, &sec_bucket,
&tmp_sig);lib/librte_member/rte_member_ht.c new file +9
-0+ +search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, + match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; +} +
...
lib/librte_member/rte_member_ht.c line 274
Quoted 6 lines
lib/librte_member/rte_member_ht.c new file +2
-0+static inline int +try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_tsec,
lib/librte_member/rte_member_ht.c new file +3
-0+ member_sig_t sig, member_set_t set_id) { +int i; +/* If not full then insert into one slot*/
Extra space at the end of the comment, before */
lib/librte_member/rte_member_ht.c line 282
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +8
-0+for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +if (buckets[prim].sets[i] ==RTE_MEMBER_NO_MATCH ) { + buckets[prim].sigs[i] = sig; + buckets[prim].sets[i] = set_id; +return 0 ; + } + } +/* if prim failed, we need to access second cache line */
Second bucket, instead of second cache line? Also, check that all comments start with capital letters.
Quoted 35 lines
lib/librte_member/rte_member_ht.c new file +12
-0+for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +if (buckets[sec].sets[i] ==RTE_MEMBER_NO_MATCH ) { + buckets[sec].sigs[i] = sig; + buckets[sec].sets[i] = set_id; +return 0 ; + } + } +return -1 ; +} + +static inline int +try_overwrite(struct member_ht_bucket *buckets, uint32_t prim,
uint32_tsec,lib/librte_member/rte_member_ht.c new file +21
-0+ member_sig_t sig, member_set_t set_id) { + if (insert_overwrite_search(prim, sig, buckets, set_id) || + insert_overwrite_search(sec, sig, buckets, + set_id)) + return 0; +return -1 ; +} + +static inline int +evict_from_bucket (void ) +{ +/* for now, we randomly pick one entry to evict */ +return rte_rand () & (RTE_MEMBER_BUCKET_ENTRIES -1 ); } + +/* +* This function is similar to the cuckoo hash make_space function in +hash +* library +*/ +static inline int +make_space_bucket(const struct rte_member_setsum *ss, uint32_t
bkt_num)
lib/librte_member/rte_member_ht.c
Quoted 2 lines
lib/librte_member/rte_member_ht.c new file +2
-0+{ +static unsigned int nr_pushes;
A patch has been sent to change this static variable to be non-static, in the hash library. Since this is following a similar implementation, it will need the same change:
http://dpdk.org/dev/patchwork/patch/29104/
lib/librte_member/rte_member_ht.c line 325
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +5
-0+unsigned int i, j; +int ret; +struct member_ht_bucket *buckets = ss->table; +uint32_t next_bucket_idx; +struct member_ht_bucket *next_bkt[RTE_MEMBER_BUCKET_ENTRIES];
lib/librte_member/rte_member_ht.c new file +2
-0+struct member_ht_bucket *bkt = &buckets[bkt_num]; +member_set_t flag_mask =1U << (sizeof (member_set_t) *8 -1 );
Include a comment about this flag_mask
(MSB is set to indicate if an entry has been already pushed).
Quoted 21 lines
lib/librte_member/rte_member_ht.c new file +7
-0+/* +* Push existing item (search for bucket with space in +* alternative locations) to its alternative location +*/ +for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +/* Search for space in alternative locations */ + next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss-Quoted 1 line
bucket_mask;lib/librte_member/rte_member_ht.c new file +3
-0+ next_bkt[i] = &buckets[next_bucket_idx]; +for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) { + if (next_bkt[i]->sets[j] ==
RTE_MEMBER_NO_MATCH)lib/librte_member/rte_member_ht.c new file +8
-0+break ; + } + +if (j !=RTE_MEMBER_BUCKET_ENTRIES ) +break ; + } + +/* Alternative location has spare room (end of recursive function) */
lib/librte_member/rte_member_ht.c line 347
Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +4
-0+if (i !=RTE_MEMBER_BUCKET_ENTRIES ) { + next_bkt[i]->sigs[j] = bkt->sigs[i]; + next_bkt[i]->sets[j] = bkt->sets[i]; +return i;
Since you want to return 1 when there is an eviction, I think you can return 1 here, no need to return the index in the bucket.
Quoted 21 lines
lib/librte_member/rte_member_ht.c new file +21
-0+ } + +/* Pick entry that has not been pushed yet */ +for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) +if ((bkt->sets[i] & flag_mask) ==0 ) +break ; + +/* All entries have been pushed, so entry cannot be added */ +if (i ==RTE_MEMBER_BUCKET_ENTRIES || + nr_pushes >RTE_MEMBER_MAX_PUSHES ) +return -ENOSPC ; + + next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask; +/* Set flag to indicate that this entry is going to be pushed */ + bkt->sets[i] |= flag_mask; + + nr_pushes++; +/* Need room in alternative bucket to insert the pushed entry */ + ret =make_space_bucket (ss, next_bucket_idx); +/* +* After recursive function.
lib/librte_member/rte_member_ht.c line 378
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+* Clear flags and insert the pushed entry +* in its alternative location if successful, +* or return error +*/ + bkt->sets[i] &= ~flag_mask; + nr_pushes =0 ; +if (ret >=0 ) { + next_bkt[i]->sigs[ret] = bkt->sigs[i]; + next_bkt[i]->sets[ret] = bkt->sets[i]; +return i;
Same about return 1 here.
Quoted 24 lines
lib/librte_member/rte_member_ht.c new file +14
-0+ } else + return ret; +} + +int +rte_member_add_ht(const struct rte_member_setsum *ss, + const void *key, member_set_t set_id) { + int ret; + uint32_t prim_bucket, sec_bucket; + member_sig_t tmp_sig; + struct member_ht_bucket *buckets = ss->table; + member_set_t flag_mask = 1U << (sizeof(member_set_t) * 8 - 1); + + if (set_id == RTE_MEMBER_NO_MATCH || (set_id & flag_mask) !=0)
lib/librte_member/rte_member_ht.c new file +3
-0+return -EINVAL ; + + get_buckets_index(ss, key, &prim_bucket, &sec_bucket,
&tmp_sig);lib/librte_member/rte_member_ht.c new file +4
-0+ +/* if it is cache based filter, we try overwriting existing entry */ +if (ss->cache) { + ret = try_overwrite(buckets, prim_bucket, sec_bucket,
tmp_sig,
lib/librte_member/rte_member_ht.c line 401
Quoted 1 line
lib/librte_member/rte_member_ht.c new file +1
-0+ set_id);
If the comment that I made above (in the insert_overwrite_search function) is true and this function is used to update the set_id of an entry, can this be used also in non-cache mode?
lib/librte_member/rte_member_ht.c line 405
Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +4
-0+if (ret !=-1 ) +return ret; + } +/* If not full then insert into one slot*/
Extra space before */.
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +7
-0+ ret =try_insert (buckets, prim_bucket, sec_bucket, tmp_sig, set_id); +if (ret !=-1 ) +return ret; + +/* random pick prim or sec for recursive displacement */ + + uint32_t select_bucket = (tmp_sig && 1U) ? prim_bucket :
sec_bucket;
lib/librte_member/rte_member_ht.c line 420
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +8
-0+if (ss->cache) { + ret =evict_from_bucket (); + buckets[select_bucket].sigs[ret] = tmp_sig; + buckets[select_bucket].sets[ret] = set_id; +return 1 ; + } + + ret =make_space_bucket (ss, select_bucket);
If this only return a negative value when failure or 1 when success, you can check for ret == 1 and not set ret = 1.
Quoted 14 lines
lib/librte_member/rte_member_ht.c new file +14
-0+if (ret >=0 ) { + buckets[select_bucket].sigs[ret] = tmp_sig; + buckets[select_bucket].sets[ret] = set_id; + ret =1 ; + } + +return ret; +} + +void +rte_member_free_ht (struct rte_member_setsum *ss) { +rte_free (ss->table); +} +
...
lib/librte_member/rte_member_ht.c line 469
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ +void +rte_member_reset_ht (const struct rte_member_setsum *ss) { +uint32_t i, j; +struct member_ht_bucket *buckets = ss->table;
To keep to consistency, I would leave a blank line between the variables declarations and the rest of the function implementation.
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+for (i =0 ; i < ss->bucket_cnt; i++) { +for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) + buckets[i].sets[j] =RTE_MEMBER_NO_MATCH ; + } +}
Re: [PATCH v4 2/7] member: implement HT mode
Thank you very much Pablo.
I agree with you on most of your comments and I will address them soon.
But please find some explanations inlined for certain questions you raised.
Quoted 145 lines
-----Original Message-----
From: De Lara Guarch, Pablo
Sent: Monday, October 2, 2017 6:31 AM
To: Wang, Yipeng1 <[email protected]>; [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>
Subject: RE: [PATCH v4 2/7] member: implement HT modeQuoted 10 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:40 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 2/7] member: implement HT mode...
lib/librte_member/Makefile
Quoted 1 line
lib/librte_member/Makefile +0
-0index
Quoted 19 lines
1a79eaa..ad26548 100644
lib/librte_member/Makefile +0
-0@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.mapLIBABIVER := 1
# all source are stored in SRCS-ylib/librte_member/Makefile +1
-1-SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c
rte_member_ht.c# install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.hlib/librte_member/rte_member_ht.c new file +0
-0...
lib/librte_member/rte_member_ht.c line 46
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ +static inline int +insert_overwrite_search (uint32_t bucket,member_sig_t tmp_sig, +struct member_ht_bucket *buckets, +member_set_t set_id)I would call "bucket", "
bucket_id", for better understanding. This comment also applies to other parts of the code (e.g.prim_buckets).lib/librte_member/rte_member_ht.c line 53
Quoted 7 lines
lib/librte_member/rte_member_ht.c new file +7
-0+{ +int i; +for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +if (buckets[bucket].sigs[i] == tmp_sig) { + buckets[bucket].sets[i] = set_id; +return 1 ; + }Is this function used to update an existing entry?
At first, I thought that this was evicting another entry, when the bucket was full,
but it looks like it is updating theset_idof an existing entry. If this is the case, I would change the function name and add a comment explaining this.lib/librte_member/rte_member_ht.c line 63
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+ } +return 0 ; +} + +static inline int +search_bucket_single (uint32_t bucket,member_sig_t tmp_sig, +struct member_ht_bucket *buckets, +member_set_t *set_id) +{ +int iter;Iter should be "unsigned int" (or similar, maybe "
uint8_t")Quoted 6 lines
lib/librte_member/rte_member_ht.c new file +6
-0+for (iter =0 ; iter <RTE_MEMBER_BUCKET_ENTRIES ; iter++) { +if (tmp_sig == buckets[bucket].sigs[iter] && + buckets[bucket].sets[iter] != +RTE_MEMBER_NO_MATCH ) { + *set_id = buckets[bucket].sets[iter]; +return 1 ;lib/librte_member/rte_member_ht.c line 79
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+ } + } +return 0 ; +} + +static inline void +search_bucket_multi(uint32_t bucket, member_sig_t tmp_sig, + struct member_ht_bucket *buckets, + uint32_t *counter, + uint32_t match_per_key,Better change to "
matches_per_key".Quoted 18 lines
lib/librte_member/rte_member_ht.c new file +18
-0+ member_set_t *set_id) +{ +int iter; +for (iter =0 ; iter <RTE_MEMBER_BUCKET_ENTRIES ; iter++) { +if (tmp_sig == buckets[bucket].sigs[iter] && + buckets[bucket].sets[iter] != +RTE_MEMBER_NO_MATCH ) { + set_id[*counter] = buckets[bucket].sets[iter]; + (*counter)++; +if (*counter >= match_per_key) +return ; + } + } +} + +int +rte_member_create_ht (struct rte_member_setsum *ss, +const struct rte_member_parameters *params) {...
lib/librte_member/rte_member_ht.c line 139
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ +RTE_MEMBER_LOG (DEBUG ,"Hash table based filter created, " +"the table has %u entries, %u buckets\n" , + num_buckets, + num_buckets /RTE_MEMBER_BUCKET_ENTRIES );Shouldn't this be "
num_buckets*RTE_MEMBER_BUCKET_ENTRIES" and "num_buckets"?Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +5
-0+return 0 ; +} + +static inline +void get_buckets_index(const struct rte_member_setsum *ss, const void*key,
lib/librte_member/rte_member_ht.c new file +1
-0+ uint32_t *prim_bkt, uint32_t *sec_bkt, member_sig_t*sig) {
"static inline void" should be in the same line.
Quoted 13 lines
lib/librte_member/rte_member_ht.c line 153
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +3
-0+uint32_t first_hash =MEMBER_HASH_FUNC (key, ss->key_len, + ss->prim_hash_seed); + uint32_t sec_hash = MEMBER_HASH_FUNC(&first_hash,
sizeof(uint32_t),lib/librte_member/rte_member_ht.c new file +4
-0+ ss->sec_hash_seed); + *sig = first_hash; +if (ss->cache) { + *prim_bkt = sec_hash & ss->bucket_mask;Is this correct? Using the secondary hash to acces the primary bucket? Why is the cache case different from the non-cache case?
I think this function deserves some comments to explain all this calculations.
[Wang, Yipeng]
We use the first hash value for the signature, and the second hash value to derive the primary and secondary bucket locations.
For the non-cache mode, we use xor hashing which is called partial-key cuckoo hashing proposed by B. Fan, et al's paper
"Cuckoo Filter: Practically Better Than Bloom". Since the non-cache mode
is similar to cuckoo filter, the partial key hash enables the derivation of alternative bucket location by only using signatures without full key or storing alternative signature.
For cache mode, we do not need to get alternative bucket (just overwritten), so we use higher/lower bits for more independent hash values.
I will add a comment to explain in next version.
Quoted 188 lines
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ *sec_bkt = (sec_hash >>16 ) & ss->bucket_mask; + }else { + *prim_bkt = sec_hash & ss->bucket_mask; + *sec_bkt = (*prim_bkt ^ *sig) & ss->bucket_mask; + }lib/librte_member/rte_member_ht.c line 169
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+} + +int +rte_member_lookup_ht (const struct rte_member_setsum *ss, +const void *key,member_set_t *set_id) { +uint32_t prim_bucket, sec_bucket; +member_sig_t tmp_sig; +struct member_ht_bucket *buckets = ss->table; + +Remove extra blank line.
Quoted 17 lines
lib/librte_member/rte_member_ht.c new file +2
-0+ *set_id =RTE_MEMBER_NO_MATCH ; + get_buckets_index(ss, key, &prim_bucket, &sec_bucket,
&tmp_sig);lib/librte_member/rte_member_ht.c new file +13
-0+ +if (search_bucket_single (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; + +return 0 ; +} + +uint32_t +rte_member_lookup_bulk_ht (const struct rte_member_setsum *ss, + const void **keys, uint32_t num_keys, member_set_t
*set_id) {lib/librte_member/rte_member_ht.c
Quoted 2 lines
lib/librte_member/rte_member_ht.c new file +2
-0+uint32_t i; +uint32_t ret =0 ;Better change to something more meaningful, like "
nr_matches".Quoted 20 lines
lib/librte_member/rte_member_ht.c new file +20
-0+struct member_ht_bucket *buckets = ss->table; +member_sig_t tmp_sig[RTE_MEMBER_LOOKUP_BULK_MAX ]; +uint32_t prim_buckets[RTE_MEMBER_LOOKUP_BULK_MAX ]; +uint32_t sec_buckets[RTE_MEMBER_LOOKUP_BULK_MAX ]; + +for (i =0 ; i < num_keys; i++) { +get_buckets_index (ss, keys[i], &prim_buckets[i], + &sec_buckets[i], &tmp_sig[i]); +rte_prefetch0 (&buckets[prim_buckets[i]]); +rte_prefetch0 (&buckets[sec_buckets[i]]); + } + +for (i =0 ; i < num_keys; i++) { +if (search_bucket_single (prim_buckets[i], tmp_sig[i], + buckets, &set_id[i]) || +search_bucket_single (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ;lib/librte_member/rte_member_ht.c line 217
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+ } +return ret; +} + +uint32_t +rte_member_lookup_multi_ht (const struct rte_member_setsum *ss, +const void *key,uint32_t match_per_key, +member_set_t *set_id) +{ +uint32_t ret =0 ;Better change to something more meaningful, like "
nr_matches". This applies to the following functions.Quoted 15 lines
lib/librte_member/rte_member_ht.c new file +5
-0+uint32_t prim_bucket, sec_bucket; +member_sig_t tmp_sig; +struct member_ht_bucket *buckets = ss->table; + + get_buckets_index(ss, key, &prim_bucket, &sec_bucket,
&tmp_sig);lib/librte_member/rte_member_ht.c new file +9
-0+ +search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, + match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; +} +...
lib/librte_member/rte_member_ht.c line 274
Quoted 6 lines
lib/librte_member/rte_member_ht.c new file +2
-0+static inline int +try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_tsec,
lib/librte_member/rte_member_ht.c new file +3
-0+ member_sig_t sig, member_set_t set_id) { +int i; +/* If not full then insert into one slot*/ Extra space at the end of the comment, before */
lib/librte_member/rte_member_ht.c line 282
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +8
-0+for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +if (buckets[prim].sets[i] ==RTE_MEMBER_NO_MATCH ) { + buckets[prim].sigs[i] = sig; + buckets[prim].sets[i] = set_id; +return 0 ; + } + } +/* if prim failed, we need to access second cache line */ Second bucket, instead of second cache line? Also, check that all comments start with capital letters.
Quoted 35 lines
lib/librte_member/rte_member_ht.c new file +12
-0+for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +if (buckets[sec].sets[i] ==RTE_MEMBER_NO_MATCH ) { + buckets[sec].sigs[i] = sig; + buckets[sec].sets[i] = set_id; +return 0 ; + } + } +return -1 ; +} + +static inline int +try_overwrite(struct member_ht_bucket *buckets, uint32_t prim,
uint32_tsec,lib/librte_member/rte_member_ht.c new file +21
-0+ member_sig_t sig, member_set_t set_id) { + if (insert_overwrite_search(prim, sig, buckets, set_id) || + insert_overwrite_search(sec, sig, buckets, + set_id)) + return 0; +return -1 ; +} + +static inline int +evict_from_bucket (void ) +{ +/* for now, we randomly pick one entry to evict */ +return rte_rand () & (RTE_MEMBER_BUCKET_ENTRIES -1 ); } + +/* +* This function is similar to the cuckoo hash make_space function in +hash +* library +*/ +static inline int +make_space_bucket(const struct rte_member_setsum *ss, uint32_t
bkt_num)lib/librte_member/rte_member_ht.c
Quoted 2 lines
lib/librte_member/rte_member_ht.c new file +2
-0+{ +static unsigned int nr_pushes;A patch has been sent to change this static variable to be non-static, in the hash library. Since this is following a similar implementation, it will need the same change:
http://dpdk.org/dev/patchwork/patch/29104/lib/librte_member/rte_member_ht.c line 325
Quoted 8 lines
lib/librte_member/rte_member_ht.c new file +5
-0+unsigned int i, j; +int ret; +struct member_ht_bucket *buckets = ss->table; +uint32_t next_bucket_idx; +struct member_ht_bucket *next_bkt[RTE_MEMBER_BUCKET_ENTRIES];
lib/librte_member/rte_member_ht.c new file +2
-0+struct member_ht_bucket *bkt = &buckets[bkt_num]; +member_set_t flag_mask =1U << (sizeof (member_set_t) *8 -1 );Include a comment about this
flag_mask
(MSB is set to indicate if an entry has been already pushed).Quoted 21 lines
lib/librte_member/rte_member_ht.c new file +7
-0+/* +* Push existing item (search for bucket with space in +* alternative locations) to its alternative location +*/ +for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +/* Search for space in alternative locations */ + next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss-Quoted 1 line
bucket_mask;lib/librte_member/rte_member_ht.c new file +3
-0+ next_bkt[i] = &buckets[next_bucket_idx]; +for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) { + if (next_bkt[i]->sets[j] ==
RTE_MEMBER_NO_MATCH)lib/librte_member/rte_member_ht.c new file +8
-0+break ; + } + +if (j !=RTE_MEMBER_BUCKET_ENTRIES ) +break ; + } + +/* Alternative location has spare room (end of recursive function) */
Quoted 8 lines
lib/librte_member/rte_member_ht.c line 347
Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +4
-0+if (i !=RTE_MEMBER_BUCKET_ENTRIES ) { + next_bkt[i]->sigs[j] = bkt->sigs[i]; + next_bkt[i]->sets[j] = bkt->sets[i]; +return i;Since you want to return 1 when there is an eviction, I think you can return 1 here,
no need to return the index in the bucket.
[Wang, Yipeng]
We return the index in the bucket to keep the recursive function going right? I.e. the make_space needs the returned index to displace the entry.
Quoted 56 lines
Quoted 22 lines
lib/librte_member/rte_member_ht.c new file +21
-0+ } + +/* Pick entry that has not been pushed yet */ +for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) +if ((bkt->sets[i] & flag_mask) ==0 ) +break ; + +/* All entries have been pushed, so entry cannot be added */ +if (i ==RTE_MEMBER_BUCKET_ENTRIES || + nr_pushes >RTE_MEMBER_MAX_PUSHES ) +return -ENOSPC ; + + next_bucket_idx = (bkt->sigs[i] ^ bkt_num) & ss->bucket_mask; +/* Set flag to indicate that this entry is going to be pushed */ + bkt->sets[i] |= flag_mask; + + nr_pushes++; +/* Need room in alternative bucket to insert the pushed entry */ + ret =make_space_bucket (ss, next_bucket_idx); +/* +* After recursive function. lib/librte_member/rte_member_ht.c line 378
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+* Clear flags and insert the pushed entry +* in its alternative location if successful, +* or return error +*/ + bkt->sets[i] &= ~flag_mask; + nr_pushes =0 ; +if (ret >=0 ) { + next_bkt[i]->sigs[ret] = bkt->sigs[i]; + next_bkt[i]->sets[ret] = bkt->sets[i]; +return i;Same about return 1 here.
Quoted 21 lines
lib/librte_member/rte_member_ht.c new file +14
-0+ } else + return ret; +} + +int +rte_member_add_ht(const struct rte_member_setsum *ss, + const void *key, member_set_t set_id) { + int ret; + uint32_t prim_bucket, sec_bucket; + member_sig_t tmp_sig; + struct member_ht_bucket *buckets = ss->table; + member_set_t flag_mask = 1U << (sizeof(member_set_t) * 8 - 1); + + if (set_id == RTE_MEMBER_NO_MATCH || (set_id & flag_mask) !=0)
lib/librte_member/rte_member_ht.c new file +3
-0+return -EINVAL ; + + get_buckets_index(ss, key, &prim_bucket, &sec_bucket,
&tmp_sig);lib/librte_member/rte_member_ht.c new file +2
-0+ +/* if it is cache based filter, we try overwriting existing entry */
Quoted 9 lines
lib/librte_member/rte_member_ht.c line 401
Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +2
-0+if (ss->cache) { + ret = try_overwrite(buckets, prim_bucket, sec_bucket,
tmp_sig,lib/librte_member/rte_member_ht.c new file +1
-0+ set_id);If the comment that I made above (in the
insert_overwrite_searchfunction) is true and this function is used to update theset_idof an entry, can this be used also in non-cache mode?
[Wang, Yipeng]
I changed the name to update_entry_search. For non-cache mode, we do not do update even if the key finds an entry with the same signature during add. This is because for non-cache mode, we do not allow false negative, keys Overwriting each other may incur false negative.
I will add more explanation in comments.
Quoted 18 lines
lib/librte_member/rte_member_ht.c line 405
Quoted 4 lines
lib/librte_member/rte_member_ht.c new file +4
-0+if (ret !=-1 ) +return ret; + } +/* If not full then insert into one slot*/ Extra space before */.
Quoted 11 lines
lib/librte_member/rte_member_ht.c new file +7
-0+ ret =try_insert (buckets, prim_bucket, sec_bucket, tmp_sig, set_id); +if (ret !=-1 ) +return ret; + +/* random pick prim or sec for recursive displacement */ + + uint32_t select_bucket = (tmp_sig && 1U) ? prim_bucket :
sec_bucket;lib/librte_member/rte_member_ht.c new file +3
-0+if (ss->cache) { + ret =evict_from_bucket (); + buckets[select_bucket].sigs[ret] = tmp_sig;
Quoted 9 lines
lib/librte_member/rte_member_ht.c line 420
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ buckets[select_bucket].sets[ret] = set_id; +return 1 ; + } + + ret =make_space_bucket (ss, select_bucket);If this only return a negative value when failure or 1 when success, you can check for ret == 1 and not set ret = 1.
[Wang, Yipeng]
Please see my comments above, if we return 1 for make_space, will the recursive process be broken?
Quoted 32 lines
Quoted 14 lines
lib/librte_member/rte_member_ht.c new file +14
-0+if (ret >=0 ) { + buckets[select_bucket].sigs[ret] = tmp_sig; + buckets[select_bucket].sets[ret] = set_id; + ret =1 ; + } + +return ret; +} + +void +rte_member_free_ht (struct rte_member_setsum *ss) { +rte_free (ss->table); +} +...
lib/librte_member/rte_member_ht.c line 469
Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+ +void +rte_member_reset_ht (const struct rte_member_setsum *ss) { +uint32_t i, j; +struct member_ht_bucket *buckets = ss->table;To keep to consistency, I would leave a blank line between the variables declarations
and the rest of the function implementation.Quoted 5 lines
lib/librte_member/rte_member_ht.c new file +5
-0+for (i =0 ; i < ss->bucket_cnt; i++) { +for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) + buckets[i].sets[j] =RTE_MEMBER_NO_MATCH ; + } +}
PATCH v4 3/7 member: implement vBF mode
In reply to Yipeng WangBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.
Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_vbf.c | 349 +++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_vbf.h | 79 +++++++++ 3 files changed, 429 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_vbf.c new file +349 -0
349 lines not on this page · raw message
lib/librte_member/rte_member_vbf.h new file +79 -0
79 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH v4 3/7] member: implement vBF mode
Quoted 29 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:41 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 3/7] member: implement vBF modeBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
Signed-off-by Yipeng Wang <[email protected]>
...
Quoted 2 lines
lib/librte_member/rte_member_vbf.c +0
-0
...
Quoted 6 lines
lib/librte_member/rte_member_vbf.c +5
-0+int +rte_member_create_vbf(struct rte_member_setsum *ss, + const struct rte_member_parameters *params) { + + if (params->num_set > 32 || !rte_is_power_of_2(params-Quoted 1 line
num_set) ||
Magic number. Define a macro instead.
Quoted 7 lines
lib/librte_member/rte_member_vbf.c +5
-0+ params->num_keys == 0 || + params->false_positive_rate == 0 || + params->false_positive_rate > 1) { + rte_errno =EINVAL ; + RTE_MEMBER_LOG(ERR, "vBF create with invalidparameters\n");
lib/librte_member/rte_member_vbf.c +1
-0+return -EINVAL ;
...
lib/librte_member/rte_member_vbf.c line 112
Quoted 4 lines
lib/librte_member/rte_member_vbf.c +4
-0+ +/* +* reduce hash function count, until we approach the user specified +* false-positive rate. otherwise it is too conservative
Watch out for capital letters at the start of the comment and after a full stop.
Quoted 24 lines
lib/librte_member/rte_member_vbf.c +8
-0+*/ +int tmp_num_hash = ss->num_hashes; + + while (tmp_num_hash > 1) { + float tmp_fp = new_fp; + + tmp_num_hash--; + new_fp = pow((1 - pow((1 - 1.0 / ss->bits),
num_keys_per_bf*lib/librte_member/rte_member_vbf.c +14
-0+ tmp_num_hash)), tmp_num_hash); + new_fp =1 -pow ((1 - new_fp), ss->num_set); + +if (new_fp > params->false_positive_rate) { + new_fp = tmp_fp; + tmp_num_hash++; +break ; + } + } + + ss->num_hashes = tmp_num_hash; + +RTE_MEMBER_LOG (DEBUG ,"vector bloom filter created, " + "each bloom filter expects %u keys, needs %u bits, %uhashes, "
lib/librte_member/rte_member_vbf.c line 137
Quoted 3 lines
lib/librte_member/rte_member_vbf.c +3
-0+"with false positive rate set as %.5f, " +"The new calculated vBF false positive rate is %.5f\n" , + num_keys_per_bf, ss->bits, ss->num_hashes, x, new_fp);
Use a more descriptive variable name for "x".
Quoted 6 lines
lib/librte_member/rte_member_vbf.c +3
-0+ + ss->table = rte_zmalloc_socket(NULL, ss->num_set * (ss->bits >> 3), + RTE_CACHE_LINE_SIZE, ss-Quoted 1 line
socket_id);lib/librte_member/rte_member_vbf.c +2
-0+ +/*
lib/librte_member/rte_member_vbf.c line 152
Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10
-0+* To avoid multiplication and division: +* mul_shift is used for multiplication shift during bit test +* div_shift is used for division shift, to be divided by number of bits +* represented by a uint32_t variable +*/ + ss->mul_shift =__builtin_ctzl (ss->num_set); + ss->div_shift =__builtin_ctzl (32 >> ss->mul_shift); + +if (ss->table ==NULL ) +return -ENOMEM ;
I would move this check just after the malloc call.
Quoted 18 lines
lib/librte_member/rte_member_vbf.c +17
-0+ +return 0 ; +} + +static inline uint32_t +test_bit (uint32_t bit_loc,const struct rte_member_setsum *ss) { +uint32_t *vbf = ss->table; +uint32_t n = ss->num_set; +uint32_t div_shift = ss->div_shift; +uint32_t mul_shift = ss->mul_shift; +/* +* a is how many bits in one BF are represented by one 32bit +* variable. +*/ +uint32_t a =32 >> mul_shift; +/* +* x>>b is the divide, x & (a-1) is the mod, & (1<<n-1) to mask out bits
lib/librte_member/rte_member_vbf.c
Quoted 3 lines
lib/librte_member/rte_member_vbf.c +3
-0+* we do not need +*/ +return (vbf[bit_loc>>div_shift] >> ((bit_loc & (a -1 )) << mul_shift))
Add spaces around ">>".
Quoted 4 lines
&
lib/librte_member/rte_member_vbf.c +3
-0+ ((1ULL << n) -1 ); +} +
lib/librte_member/rte_member_vbf.c line 186
Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10
-0+static inline void +set_bit (uint32_t bit_loc,const struct rte_member_setsum *ss,int32_t +set) { +uint32_t *vbf = ss->table; +uint32_t div_shift = ss->div_shift; +uint32_t mul_shift = ss->mul_shift; +uint32_t a =32 >> mul_shift; + + vbf[bit_loc>>div_shift] |=1U << (((bit_loc & (a -1 )) << mul_shift) + + set -1 );
Same as above.
Quoted 13 lines
lib/librte_member/rte_member_vbf.c +4
-0+} + +int +rte_member_lookup_vbf(const struct rte_member_setsum *ss, constvoid *key,
lib/librte_member/rte_member_vbf.c +4
-0+ member_set_t *set_id) +{ +uint32_t j; + uint32_t h1 = MEMBER_HASH_FUNC(key, ss->key_len, ss-Quoted 1 line
prim_hash_seed);lib/librte_member/rte_member_vbf.c +3
-0+uint32_t h2 =MEMBER_HASH_FUNC (&h1,sizeof (uint32_t ), + ss->sec_hash_seed); +uint32_t mask = ~0 ;
lib/librte_member/rte_member_vbf.c line 207
Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10
-0+uint32_t bit_loc; + +for (j =0 ; j < ss->num_hashes; j++) { + bit_loc = (h1 + j * h2) & ss->bit_mask; + mask &=test_bit (bit_loc, ss); + } + +if (mask) { + *set_id =__builtin_ctzl (mask) +1 ; +return 1 ;
Wouldn't it be better to return 0 when there is a hit and -ENOENT when there is not?
lib/librte_member/rte_member_vbf.c
Quoted 12 lines
lib/librte_member/rte_member_vbf.c +9
-0+ } + + *set_id =RTE_MEMBER_NO_MATCH ; +return 0 ; +} + +uint32_t +rte_member_lookup_bulk_vbf (const struct rte_member_setsum *ss, + const void **keys, uint32_t num_keys, member_set_t
*set_ids) {lib/librte_member/rte_member_vbf.c +2
-0+uint32_t i, k; +uint32_t ret =0 ;
Change variable name to "nr_matches/hits" or similar. Same in the next functions.
Re: [PATCH v4 3/7] member: implement vBF mode
Please see one comment inlined:
Quoted 172 lines
-----Original Message-----
From: De Lara Guarch, Pablo
Sent: Monday, October 2, 2017 8:44 AM
To: Wang, Yipeng1 <[email protected]>; [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>
Subject: RE: [PATCH v4 3/7] member: implement vBF modeQuoted 27 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:41 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 3/7] member: implement vBF modeBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
Signed-off-by Yipeng Wang <[email protected]>...
Quoted 2 lines
lib/librte_member/rte_member_vbf.c +0
-0...
Quoted 6 lines
lib/librte_member/rte_member_vbf.c +5
-0+int +rte_member_create_vbf(struct rte_member_setsum *ss, + const struct rte_member_parameters *params) { + + if (params->num_set > 32 || !rte_is_power_of_2(params-Quoted 1 line
num_set) ||Magic number. Define a macro instead.
Quoted 7 lines
lib/librte_member/rte_member_vbf.c +5
-0+ params->num_keys == 0 || + params->false_positive_rate == 0 || + params->false_positive_rate > 1) { + rte_errno =EINVAL ; + RTE_MEMBER_LOG(ERR, "vBF create with invalidparameters\n");
lib/librte_member/rte_member_vbf.c +1
-0+return -EINVAL ;...
lib/librte_member/rte_member_vbf.c line 112
Quoted 4 lines
lib/librte_member/rte_member_vbf.c +4
-0+ +/* +* reduce hash function count, until we approach the user specified +* false-positive rate. otherwise it is too conservative Watch out for capital letters at the start of the comment and after a full stop.
Quoted 24 lines
lib/librte_member/rte_member_vbf.c +8
-0+*/ +int tmp_num_hash = ss->num_hashes; + + while (tmp_num_hash > 1) { + float tmp_fp = new_fp; + + tmp_num_hash--; + new_fp = pow((1 - pow((1 - 1.0 / ss->bits),
num_keys_per_bf*lib/librte_member/rte_member_vbf.c +14
-0+ tmp_num_hash)), tmp_num_hash); + new_fp =1 -pow ((1 - new_fp), ss->num_set); + +if (new_fp > params->false_positive_rate) { + new_fp = tmp_fp; + tmp_num_hash++; +break ; + } + } + + ss->num_hashes = tmp_num_hash; + +RTE_MEMBER_LOG (DEBUG ,"vector bloom filter created, " + "each bloom filter expects %u keys, needs %u bits, %uhashes, "
lib/librte_member/rte_member_vbf.c line 137
Quoted 3 lines
lib/librte_member/rte_member_vbf.c +3
-0+"with false positive rate set as %.5f, " +"The new calculated vBF false positive rate is %.5f\n" , + num_keys_per_bf, ss->bits, ss->num_hashes, x, new_fp);Use a more descriptive variable name for "x".
Quoted 6 lines
lib/librte_member/rte_member_vbf.c +3
-0+ + ss->table = rte_zmalloc_socket(NULL, ss->num_set * (ss->bits >> 3), + RTE_CACHE_LINE_SIZE, ss-Quoted 1 line
socket_id);lib/librte_member/rte_member_vbf.c +2
-0+ +/* lib/librte_member/rte_member_vbf.c line 152
Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10
-0+* To avoid multiplication and division: +* mul_shift is used for multiplication shift during bit test +* div_shift is used for division shift, to be divided by number of bits +* represented by a uint32_t variable +*/ + ss->mul_shift =__builtin_ctzl (ss->num_set); + ss->div_shift =__builtin_ctzl (32 >> ss->mul_shift); + +if (ss->table ==NULL ) +return -ENOMEM ;I would move this check just after the malloc call.
Quoted 18 lines
lib/librte_member/rte_member_vbf.c +17
-0+ +return 0 ; +} + +static inline uint32_t +test_bit (uint32_t bit_loc,const struct rte_member_setsum *ss) { +uint32_t *vbf = ss->table; +uint32_t n = ss->num_set; +uint32_t div_shift = ss->div_shift; +uint32_t mul_shift = ss->mul_shift; +/* +* a is how many bits in one BF are represented by one 32bit +* variable. +*/ +uint32_t a =32 >> mul_shift; +/* +* x>>b is the divide, x & (a-1) is the mod, & (1<<n-1) to mask out bits
lib/librte_member/rte_member_vbf.c
Quoted 3 lines
lib/librte_member/rte_member_vbf.c +3
-0+* we do not need +*/ +return (vbf[bit_loc>>div_shift] >> ((bit_loc & (a -1 )) << mul_shift))Add spaces around ">>".
Quoted 4 lines
&
lib/librte_member/rte_member_vbf.c +3
-0+ ((1ULL << n) -1 ); +} +lib/librte_member/rte_member_vbf.c line 186
Quoted 10 lines
lib/librte_member/rte_member_vbf.c +10
-0+static inline void +set_bit (uint32_t bit_loc,const struct rte_member_setsum *ss,int32_t +set) { +uint32_t *vbf = ss->table; +uint32_t div_shift = ss->div_shift; +uint32_t mul_shift = ss->mul_shift; +uint32_t a =32 >> mul_shift; + + vbf[bit_loc>>div_shift] |=1U << (((bit_loc & (a -1 )) << mul_shift) + + set -1 );Same as above.
Quoted 10 lines
lib/librte_member/rte_member_vbf.c +4
-0+} + +int +rte_member_lookup_vbf(const struct rte_member_setsum *ss, constvoid *key,
lib/librte_member/rte_member_vbf.c +4
-0+ member_set_t *set_id) +{ +uint32_t j; + uint32_t h1 = MEMBER_HASH_FUNC(key, ss->key_len, ss-Quoted 1 line
prim_hash_seed);
Quoted 16 lines
lib/librte_member/rte_member_vbf.c line 207
Quoted 13 lines
lib/librte_member/rte_member_vbf.c +13
-0+uint32_t h2 =MEMBER_HASH_FUNC (&h1,sizeof (uint32_t ), + ss->sec_hash_seed); +uint32_t mask = ~0 ; +uint32_t bit_loc; + +for (j =0 ; j < ss->num_hashes; j++) { + bit_loc = (h1 + j * h2) & ss->bit_mask; + mask &=test_bit (bit_loc, ss); + } + +if (mask) { + *set_id =__builtin_ctzl (mask) +1 ; +return 1 ;Wouldn't it be better to return 0 when there is a hit and
-ENOENTwhen there is not?
[Wang, Yipeng]
I tried to follow the convention of other lookup functions that the return value Is kind of the number of matches that found. So 1 means found 1 and 0 means not Found. I defined the public API rte_member_lookup to also returns 1 for found.
Quoted 15 lines
lib/librte_member/rte_member_vbf.c
Quoted 12 lines
lib/librte_member/rte_member_vbf.c +9
-0+ } + + *set_id =RTE_MEMBER_NO_MATCH ; +return 0 ; +} + +uint32_t +rte_member_lookup_bulk_vbf (const struct rte_member_setsum *ss, + const void **keys, uint32_t num_keys, member_set_t
*set_ids) {lib/librte_member/rte_member_vbf.c +2
-0+uint32_t i, k; +uint32_t ret =0 ;Change variable name to "nr_matches/hits" or similar. Same in the next functions.
PATCH v4 4/7 member: add AVX for HT mode
In reply to Yipeng WangFor key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contiguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.
This patch adds AVX2 implementation in a separate header file.
--- lib/librte_member/rte_member_ht.c | 145 ++++++++++++++++++++++++++++--------- lib/librte_member/rte_member_x86.h | 107 +++++++++++++++++++++++++++ 2 files changed, 219 insertions(+), 33 deletions(-) create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +112 -33
@@ -40,6 +40,10 @@#include "rte_member.h" #include "rte_member_ht.h" +#if defined(RTE_ARCH_X86 ) +#include "rte_member_x86.h" +#endif + static inline int insert_overwrite_search(uint32_t bucket, member_sig_t tmp_sig, struct member_ht_bucket *buckets,
@@ -132,6 +136,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) buckets[i].sets[j] =RTE_MEMBER_NO_MATCH ; } +#if defined(RTE_ARCH_X86 ) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +RTE_MEMBER_BUCKET_ENTRIES ==16 ) + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_AVX2 ; + else +#endif + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_SCALAR ; RTE_MEMBER_LOG(DEBUG, "Hash table based filter created, " "the table has %u entries, %u buckets\n",
@@ -166,15 +177,26 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss,member_sig_t tmp_sig;struct member_ht_bucket *buckets = ss->table; - *set_id =RTE_MEMBER_NO_MATCH ;get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -if (search_bucket_single (prim_bucket, tmp_sig, buckets, - set_id) || -search_bucket_single (sec_bucket, tmp_sig, - buckets, set_id)) -return 1 ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single_avx (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; +break ; +#endif +default : +if (search_bucket_single (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; + }return 0 ; }
@@ -198,13 +220,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss, }for (i =0 ; i < num_keys; i++) { -if (search_bucket_single (prim_buckets[i], tmp_sig[i], - buckets, &set_id[i]) || -search_bucket_single (sec_buckets[i], - tmp_sig[i], buckets, &set_id[i])) - ret++; -else - set_id[i] =RTE_MEMBER_NO_MATCH ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_buckets[i], + tmp_sig[i], buckets, &set_id[i]) || +search_bucket_single_avx (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; +break ; +#endif +default : +if (search_bucket_single (prim_buckets[i], tmp_sig[i], + buckets, &set_id[i]) || +search_bucket_single (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + ret++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; + } }return ret; }
@@ -221,12 +257,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, - match_per_key, set_id); -if (ret < match_per_key) -search_bucket_multi (sec_bucket, tmp_sig, - buckets, &ret, match_per_key, set_id); -return ret; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_bucket, tmp_sig, buckets, + &ret, match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi_avx (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; +#endif +default : +search_bucket_multi (prim_bucket, tmp_sig, buckets, &ret, + match_per_key, set_id); +if (ret < match_per_key) +search_bucket_multi (sec_bucket, tmp_sig, + buckets, &ret, match_per_key, set_id); +return ret; + } }uint32_t
@@ -252,16 +300,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,for (i =0 ; i < num_keys; i++) { match_cnt_tmp =0 ; -search_bucket_multi (prim_buckets[i], tmp_sig[i], - buckets, &match_cnt_tmp, match_per_key, - &set_ids[i*match_per_key]); -if (match_cnt_tmp < match_per_key) -search_bucket_multi (sec_buckets[i], tmp_sig[i], +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_buckets[i], tmp_sig[i], buckets, &match_cnt_tmp, match_per_key, &set_ids[i*match_per_key]); - match_count[i] = match_cnt_tmp; -if (match_cnt_tmp !=0 ) - ret++; +if (match_cnt_tmp < match_per_key) +search_bucket_multi_avx (sec_buckets[i], + tmp_sig[i], buckets, &match_cnt_tmp, + match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_tmp; +if (match_cnt_tmp !=0 ) + ret++; +break ; +#endif +default : +search_bucket_multi (prim_buckets[i], tmp_sig[i], + buckets, &match_cnt_tmp, match_per_key, + &set_ids[i*match_per_key]); +if (match_cnt_tmp < match_per_key) +search_bucket_multi (sec_buckets[i], tmp_sig[i], + buckets, &match_cnt_tmp, match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_tmp; +if (match_cnt_tmp !=0 ) + ret++; + } }return ret; }
@@ -271,6 +337,7 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec, member_sig_t sig, member_set_t set_id) { int i; +/* If not full then insert into one slot*/ for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) {if (buckets[prim].sets[i] ==RTE_MEMBER_NO_MATCH ) {
@@ -292,12 +359,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,static inline int try_overwrite (struct member_ht_bucket *buckets,uint32_t prim,uint32_t sec, -member_sig_t sig,member_set_t set_id) +member_sig_t sig,member_set_t set_id, +enum rte_member_sig_compare_function cmp_fn) { -if (insert_overwrite_search (prim, sig, buckets, set_id) || -insert_overwrite_search (sec, sig, buckets, - set_id)) -return 0 ; +switch (cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (insert_overwrite_search_avx (prim, sig, buckets, set_id) || +insert_overwrite_search_avx (sec, sig, buckets, + set_id)) +return 0 ; +break ; +#endif +default : +if (insert_overwrite_search (prim, sig, buckets, set_id) || +insert_overwrite_search (sec, sig, buckets, + set_id)) +return 0 ; + }return -1 ; }
@@ -398,7 +477,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,/* if it is cache based filter, we try overwriting existing entry */ if (ss->cache) { ret =try_overwrite (buckets, prim_bucket, sec_bucket, tmp_sig, - set_id); + set_id, ss->sig_cmp_fn);if (ret !=-1 )return ret; }
lib/librte_member/rte_member_x86.h new file +107 -0
107 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v4 5/7 member: enable the library
In reply to Yipeng WangThis patch enables the Membership library.
--- MAINTAINERS | 8 +++++++- config/common_base | 5 +++++ lib/librte_member/Makefile | 2 ++ mk/rte.app.mk | 2 ++ 4 files changed, 16 insertions(+), 1 deletion(-)
MAINTAINERS +7 -1
@@ -726,6 +726,13 @@ F: test/test/test_lpm* F: test/test/test_func_reentrancy.c F: test/test/test_xmmt_ops.h +Membership - EXPERIMENTAL +M: Yipeng Wang <[email protected]> +M: Sameh Gobriel <[email protected]> +F: lib/librte_member/ +F: doc/guides/prog_guide/member_lib.rst +F: test/test/test_member* + Traffic metering M: Cristian Dumitrescu <[email protected]> F: lib/librte_meter/
@@ -734,7 +741,6 @@ F: test/test/test_meter.c F: examples/qos_meter/ F: doc/guides/sample_app_ug/qos_metering.rst - Other libraries ---------------
config/common_base +5 -0
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n CONFIG_RTE_LIBRTE_EFD=y # +# Compile librte_member +# +CONFIG_RTE_LIBRTE_MEMBER=y + +# # Compile librte_jobstats # CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
@@ -37,6 +37,8 @@ LIB = librte_member.a CFLAGS := -I$(SRCDIR) $(CFLAGS) CFLAGS += $(WERROR_FLAGS) -O3 +LDLIBS += -lm + EXPORT_MAP := rte_member_version.map LIBABIVER := 1
mk/rte.app.mk +2 -0
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile _LDLIBS-y += --whole-archive _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt _LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma endif
Signature · 1 line
2.7.4
Re: [PATCH v4 5/7] member: enable the library
Quoted 27 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:41 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 5/7] member: enable the libraryThis patch enables the Membership library.
Signed-off-by Yipeng Wang <[email protected]>--- MAINTAINERS | 8 +++++++- config/common_base | 5 +++++ lib/librte_member/Makefile | 2 ++ mk/rte.app.mk | 2 ++ 4 files changed, 16 insertions(+), 1 deletion(-)MAINTAINERS +0
-0
MAINTAINERS F: test/test/test_lpm* · line 735
Quoted 10 lines
MAINTAINERS +7
-0@@ -726,6 +726,13 @@ F: test/test/test_lpm* F: test/test/test_func_reentrancy.c F: test/test/test_xmmt_ops.h +Membership - EXPERIMENTAL +M: Yipeng Wang <[email protected]> +M: Sameh Gobriel <[email protected]> +F: lib/librte_member/ +F: doc/guides/prog_guide/member_lib.rst +F: test/test/test_member* +
Add these last two items in patches 6 and 7, where you are adding the files.
PATCH v4 6/7 test/member: add functional and perf tests
In reply to Yipeng WangThis patch adds functional and performance tests for membership library.
--- test/test/Makefile | 3 + test/test/test_member.c | 676 +++++++++++++++++++++++++++++++++++++++++++ test/test/test_member_perf.c | 630 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1309 insertions(+) create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
test/test/Makefile +3 -0
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c SRCS-y += test_memcpy.c SRCS-y += test_memcpy_perf.c + +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
test/test/test_member.c new file +676 -0
676 lines not on this page · raw message
test/test/test_member_perf.c new file +630 -0
630 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH v4 6/7] test/member: add functional and perf tests
Quoted 15 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, September 27, 2017 6:41 PM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v4 6/7] test/member: add functional and perf testsThis patch adds functional and performance tests for membership library.
Signed-off-by Yipeng Wang <[email protected]>
...
Quoted 1 line
+++
b/test/test/test_member.c
...
test/test/test_member.c line 104
Quoted 3 lines
test/test/test_member.c new file +3
-0+ +#define MAX_ENTRIES (1 << 16) +uint8_t gened_keys[MAX_ENTRIES ][KEY_SIZE ];
Gened? Is this "generated"? Maybe it is worth using the full word.
Review
Quoted 5 lines
test/test/test_member.c new file +3
-0+ +static struct rte_member_parameters params = { + .num_keys =MAX_ENTRIES ,/* Total hash table entries. */
(quoted diff) +1
-0+ .key_len = KEY_SIZE, /* Length of hash key. */
Align comments with tabs.
Quoted 3 lines
(quoted diff) +2
-0+ + /*num_set and false_positive_rate only relevant to vBFsetsum*/
Add whitespaces around comment.
test/test/test_member.c
Quoted 13 lines
test/test/test_member.c new file +11
-0+ .num_set = 32, + .false_positive_rate = 0.03, + .prim_hash_seed = 1, + .sec_hash_seed = 11, + .socket_id = 0 /* NUMA Socket ID for memory. */ +}; +static int test_member_insert (void ) +{ +int ret_ht, ret_cache, ret_vbf, i; + +for (i =0 ; i <5 ; i++) {
Use macro for the value 5, used here and other functions.
Quoted 12 lines
test/test/test_member.c new file +4
-0+ ret_ht =rte_member_add (setsum_ht, &keys[i], test_set[i]); + ret_cache =rte_member_add (setsum_cache, &keys[i], + test_set[i]); + ret_vbf = rte_member_add(setsum_vbf, &keys[i],test_set[i]);
(quoted diff) +1
-0+ TEST_ASSERT(ret_ht >= 0 && ret_cache >= 0 && ret_vbf >=0,
test/test/test_member.c new file +5
-0+ "insert error"); + } +printf ("insert key success\n" ); +return 0 ; +}
...
test/test/test_member.c line 385
Quoted 3 lines
test/test/test_member.c new file +3
-0+static int test_member_multimatch (void ) { +int ret_ht, ret_vbf, ret_cache; +member_set_t set_ids_ht[32 ] = {0 };
Same comment about the value 5 applies here, for the value 32.
Quoted 21 lines
test/test/test_member.c new file +20
-0+member_set_t set_ids_vbf[32 ] = {0 }; +member_set_t set_ids_cache[32 ] = {0 }; + +member_set_t set_ids_ht_m[5 ][32 ] = {{0 } }; +member_set_t set_ids_vbf_m[5 ][32 ] = {{0 } }; +member_set_t set_ids_cache_m[5 ][32 ] = {{0 } }; + +uint32_t match_count_ht[5 ]; +uint32_t match_count_vbf[5 ]; +uint32_t match_count_cache[5 ]; + +uint32_t num_key_ht =5 ; +uint32_t num_key_vbf =5 ; +uint32_t num_key_cache =5 ; + +const void *key_array[5 ]; + +uint32_t i, j; + +/* same key at most inserted 2*entry_per_bucket times for HT mode */
Review
Quoted 1 line
(quoted diff) +1
-0+ for (i = 1; i < 33; i++) {
This 33 can be expressed as 32 + 1 (using macro for 32). Also, add a comment explaining why you are skipping value 0.
...
Quoted 3 lines
test/test/test_member_perf.c new file +0
-0@@ -0,0 +1,630 @@
...
Quoted 18 lines
test/test/test_member_perf.c new file +17
-0+static int +timed_lookups_bulk (struct member_perf_params *params,int type) { +unsigned int i, j, k; +member_set_t result[BURST_SIZE ] = {0 }; +const void *keys_burst[BURST_SIZE ]; +int ret; + + false_data_bulk[type][params->cycle] =0 ; + +const uint64_t start_tsc =rte_rdtsc (); + +for (i =0 ; i <NUM_LOOKUPS /KEYS_TO_ADD ; i++) { +for (j =0 ; j <KEYS_TO_ADD /BURST_SIZE ; j++) { +for (k =0 ; k <BURST_SIZE ; k++) + keys_burst[k] = keys[j *BURST_SIZE + k]; + + ret = rte_member_lookup_bulk(params-Quoted 1 line
setsum[type],
test/test/test_member_perf.c line 314
Quoted 1 line
test/test/test_member_perf.c new file +1
-0+ &keys_burst[0],
Using keys_burst directly is equivalent to this, right?
...
Quoted 19 lines
test/test/test_member_perf.c new file +2
-0+static int +timed_lookups_multimatch(struct member_perf_params *params, inttype) {
test/test/test_member_perf.c new file +10
-0+unsigned int i, j; +member_set_t result[RTE_MEMBER_BUCKET_ENTRIES ] = {0 }; +int ret; + false_data_multi[type][params->cycle] =0 ; + +const uint64_t start_tsc =rte_rdtsc (); + +for (i =0 ; i <NUM_LOOKUPS /KEYS_TO_ADD ; i++) { +for (j =0 ; j <KEYS_TO_ADD ; j++) { + ret = rte_member_lookup_multi(params-Quoted 1 line
setsum[type],
test/test/test_member_perf.c new file +1
-0+ &keys[j], RTE_MEMBER_BUCKET_ENTRIES,result);
test/test/test_member_perf.c new file +2
-0+if (type !=CACHE && ret <=0 ) { + printf("lookup multi has wrong return value%d,"
test/test/test_member_perf.c line 355
Quoted 3 lines
test/test/test_member_perf.c new file +3
-0+ "type %d\n", ret, type); + } +if (result[0 ] != data[type][j])
Why using always result[0]? A comment would be good.
PATCH v4 7/7 doc: add membership documentation
In reply to Yipeng WangThis patch adds the documentation for membership library.
--- doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 ++++ doc/guides/prog_guide/img/member_i6.svg | 332 +++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 420 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + 12 files changed, 3595 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst
doc/api/doxy-api-index.md +2 -1
@@ -105,7 +105,8 @@ The public API headers are grouped by topics: [LPM IPv4 route] (@ref rte_lpm.h), [LPM IPv6 route] (@ref rte_lpm6.h), [ACL] (@ref rte_acl.h), - [EFD] (@ref rte_efd.h) + [EFD] (@ref rte_efd.h), + [member] (@ref rte_member.h) - **QoS**: [metering] (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
@@ -58,6 +58,7 @@ INPUT = doc/api/doxy-api-index.md \ lib/librte_mempool \ lib/librte_meter \ lib/librte_metrics \ + lib/librte_member \ lib/librte_net \ lib/librte_pdump \ lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0
1,613 lines not on this page · raw message
doc/guides/prog_guide/img/member_i2.svg new file +36 -0
@@ -0,0 +1,36 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in" + viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6"> + <v:documentProperties v:langID="1033" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {fill:none;stroke:none;stroke-width:0.25} + .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em} + .st3 {baseline-shift:32.4943%;font-size:0.649886em} + .st4 {font-size:1em} + .st5 {font-family:Cambria Math;font-size:1em} + .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/> + <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)"> + <title>Sheet.3</title> + <desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="157.5" cy="45.5" width="315" height="90"/> + <rect x="0" y="0.5" width="315" height="90" class="st1"/> + <text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan + dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan + dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan + class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3" + v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em" + class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text> </g> + </g> +</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0
148 lines not on this page · raw message
doc/guides/prog_guide/img/member_i4.svg new file +450 -0
450 lines not on this page · raw message
doc/guides/prog_guide/img/member_i5.svg new file +163 -0
163 lines not on this page · raw message
doc/guides/prog_guide/img/member_i6.svg new file +332 -0
332 lines not on this page · raw message
doc/guides/prog_guide/img/member_i7.svg new file +399 -0
242 lines not on this page · raw message
doc/guides/prog_guide/member_lib.rst new file +420 -0
420 lines not on this page · raw message
doc/guides/rel_notes/release_17_11.rst +17 -0
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.7.4
Part of the diff is not shown here. View the raw message.
v5 cover Add Membership Library
In reply to Yipeng WangDPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).
The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.
The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.
A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.
This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.
[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.
[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
v5:
- member lib: incorperate Pablo's review comments, including: multiple places with coding style issues, change variable/function names to be more clear, reorder code blocks, add descriptions, use macros for constant numbers, fix bugs in log message, etc.
- test: modify the deletion function test to cover more cases.
- test: add check in performance test to check false negatvie for HT non-cache mode.
- MAINTAINERS: complete in multiple commits instead of one as Pablo suggested.
v4:
- member lib: change two hash function macros to be one. crc or jhash will be chosen depending on the architecture.
- member lib: use dynamic log type instead of static one by Thomas' comment.
- member lib: code cleanups (remove unnecessary code, change variable name, coding style, etc).
- member lib: reorder members in setsummary struct to be more cache friendly.
- member lib: setsummary struct member "name" should be char array rather than char pointer. Fixed.
- member lib: add check for two hash seeds to make sure they are not equal.
- member lib: vBF lookup speed optimization.
- member lib: revise comments in various places as Pablo suggested.
- member lib: change "void *" to "struct rte_member_setsum *" in public API as Pablo suggested. test case is modified accordingly to directly use the struct.
- member lib: change order in rte_member_version.map as Pablo suggested.
- MAINTAINERS: change the maintainer block order according to Thomas' comment.
- test: add printf to show which section expects error messages. Change set count to 16 from 32 for performance test.
- documentation: revise several places according to Pablo's and John's comments.
v3:
- documentation: update documentation to incorporate John's review.
v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitialized issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment.
- documentation: update the programmer guide for membership library.
Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation
MAINTAINERS | 8 +- config/common_base | 5 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 +++ doc/guides/prog_guide/img/member_i6.svg | 332 ++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 420 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + lib/Makefile | 2 + lib/librte_member/Makefile | 51 + lib/librte_member/rte_member.c | 337 +++++++ lib/librte_member/rte_member.h | 513 ++++++++++ lib/librte_member/rte_member_ht.c | 586 +++++++++++ lib/librte_member/rte_member_ht.h | 94 ++ lib/librte_member/rte_member_vbf.c | 351 +++++++ lib/librte_member/rte_member_vbf.h | 82 ++ lib/librte_member/rte_member_version.map | 16 + lib/librte_member/rte_member_x86.h | 107 ++ mk/rte.app.mk | 2 + test/test/Makefile | 3 + test/test/test_member.c | 744 ++++++++++++++ test/test/test_member_perf.c | 654 ++++++++++++ 28 files changed, 7149 insertions(+), 2 deletions(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h create mode 100644 lib/librte_member/rte_member_version.map create mode 100644 lib/librte_member/rte_member_x86.h create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
PATCH v5 1/7 member: implement main API
Membership library is an extension and generalization of a traditional filter (for example Bloom Filter and cuckoo filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.
This patch adds the main API definition.
--- lib/Makefile | 2 + lib/librte_member/Makefile | 49 +++ lib/librte_member/rte_member.c | 337 ++++++++++++++++++++ lib/librte_member/rte_member.h | 513 +++++++++++++++++++++++++++++++ lib/librte_member/rte_member_version.map | 16 + 5 files changed, 917 insertions(+) create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether +DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member +DEPDIRS-librte_member := librte_eal librte_hash ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_member/Makefile new file +49 -0
49 lines not on this page · raw message
lib/librte_member/rte_member.c new file +337 -0
337 lines not on this page · raw message
lib/librte_member/rte_member.h new file +513 -0
513 lines not on this page · raw message
lib/librte_member/rte_member_version.map new file +16 -0
@@ -0,0 +1,16 @@ +DPDK_17.11 { + global: + + rte_member_add; + rte_member_create; + rte_member_delete; + rte_member_find_existing; + rte_member_free; + rte_member_lookup; + rte_member_lookup_bulk; + rte_member_lookup_multi; + rte_member_lookup_multi_bulk; + rte_member_reset; + + local: *; +};
Signature · 1 line
2.7.4
Re: [PATCH v5 1/7] member: implement main API
Quoted 20 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 1/7] member: implement main APIMembership library is an extension and generalization of a traditional filter (for example Bloom Filter and cuckoo filter) structure.
In general, the Membership library is a data structure that provides a "set- summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
Quoted 8 lines
The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary"
is memory efficient and fast on lookup.This patch adds the main API definition.
Signed-off-by Yipeng Wang <[email protected]>
PATCH v5 2/7 member: implement HT mode
In reply to Yipeng WangOne of the set-summary structures is hash-table based set-summary (HTSS). One example is cuckoo filter [1].
Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.
Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is similar to the cuckoo filter [1]. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.
The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. Thus, the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.
This patch adds the implementation of HTSS.
[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_ht.c | 506 ++++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_ht.h | 94 +++++++ 3 files changed, 601 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_ht.c new file +506 -0
506 lines not on this page · raw message
lib/librte_member/rte_member_ht.h new file +94 -0
94 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH v5 2/7] member: implement HT mode
Hi Yipeng,
Quoted 32 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 2/7] member: implement HT modeOne of the set-summary structures is hash-table based set-summary (HTSS). One example is cuckoo filter [1].
Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.
Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is similar to the cuckoo filter [1].
When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. Thus, the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.
Quoted 7 lines
This patch adds the implementation of HTSS.
[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
Signed-off-by Yipeng Wang <[email protected]>
...
Just a comment below. The rest looks fine to me (and thanks for the explanations in the v4 patch, all clear), so keep my "Reviewed-by" in next version.
Quoted 1 line
+++
b/lib/librte_member/rte_member_ht.c
...
lib/librte_member/rte_member_ht.c line 49
Quoted 7 lines
lib/librte_member/rte_member_ht.c new file +7
-0+/* Search bucket for entry with tmp_sig and update set_id */ static +inline int update_entry_search (uint32_t bucket_id,member_sig_t +tmp_sig, +struct member_ht_bucket *buckets, +member_set_t set_id) +{ +int32_t i;
"i" cannot take negative values, so use uint32_t (same in "search_bucket_single" and "search_bucket_multi").
Quoted 10 lines
lib/librte_member/rte_member_ht.c new file +10
-0+ +for (i =0 ; i <RTE_MEMBER_BUCKET_ENTRIES ; i++) { +if (buckets[bucket_id].sigs[i] == tmp_sig) { + buckets[bucket_id].sets[i] = set_id; +return 1 ; + } + } +return 0 ; +} +
PATCH v5 3/7 member: implement vBF mode
In reply to Yipeng WangBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.
Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_vbf.c | 351 +++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_vbf.h | 82 +++++++++ 3 files changed, 434 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_vbf.c new file +351 -0
351 lines not on this page · raw message
lib/librte_member/rte_member_vbf.h new file +82 -0
82 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH v5 3/7] member: implement vBF mode
Hi Yipeng,
Quoted 21 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 3/7] member: implement vBF modeBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries.
Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
Quoted 6 lines
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
Signed-off-by Yipeng Wang <[email protected]>
Just a comment below. The rest looks fine to me (and thanks for the explanations in the v4 patch, all clear), so keep my "Reviewed-by" in next version.
...
Quoted 1 line
+++
b/lib/librte_member/rte_member_vbf.c
...
lib/librte_member/rte_member_vbf.c line 143
Quoted 8 lines
lib/librte_member/rte_member_vbf.c new file +8
-0+/* +* To avoid multiplication and division: +* mul_shift is used for multiplication shift during bit test +* div_shift is used for division shift, to be divided by number of bits +* represented by a uint32_t variable +*/ + ss->mul_shift =__builtin_ctzl (ss->num_set); + ss->div_shift =__builtin_ctzl (32 >> ss->mul_shift);
Remove the whitespace added after the tab in the two lines above.
PATCH v5 4/7 member: add AVX for HT mode
In reply to Yipeng WangFor key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contiguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.
This patch adds AVX2 implementation in a separate header file.
--- lib/librte_member/rte_member_ht.c | 142 +++++++++++++++++++++++++++++-------- lib/librte_member/rte_member_x86.h | 107 ++++++++++++++++++++++++++++ 2 files changed, 218 insertions(+), 31 deletions(-) create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -31
@@ -40,6 +40,10 @@#include "rte_member.h" #include "rte_member_ht.h" +#if defined(RTE_ARCH_X86 ) +#include "rte_member_x86.h" +#endif +/* Search bucket for entry with tmp_sig and update set_id */ static inline int update_entry_search(uint32_t bucket_id, member_sig_t tmp_sig,
@@ -136,6 +140,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) buckets[i].sets[j] =RTE_MEMBER_NO_MATCH ; } +#if defined(RTE_ARCH_X86 ) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +RTE_MEMBER_BUCKET_ENTRIES ==16 ) + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_AVX2 ; + else +#endif + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_SCALAR ; RTE_MEMBER_LOG(DEBUG, "Hash table based filter created, " "the table has %u entries, %u buckets\n",
@@ -193,11 +204,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss, *set_id =RTE_MEMBER_NO_MATCH ;get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -if (search_bucket_single (prim_bucket, tmp_sig, buckets, - set_id) || -search_bucket_single (sec_bucket, tmp_sig, - buckets, set_id)) -return 1 ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single_avx (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; +break ; +#endif +default : +if (search_bucket_single (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; + }return 0 ; }
@@ -221,13 +244,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss, }for (i =0 ; i < num_keys; i++) { -if (search_bucket_single (prim_buckets[i], tmp_sig[i], - buckets, &set_id[i]) || -search_bucket_single (sec_buckets[i], - tmp_sig[i], buckets, &set_id[i])) - num_matches++; -else - set_id[i] =RTE_MEMBER_NO_MATCH ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_buckets[i], + tmp_sig[i], buckets, &set_id[i]) || +search_bucket_single_avx (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + num_matches++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; +break ; +#endif +default : +if (search_bucket_single (prim_buckets[i], tmp_sig[i], + buckets, &set_id[i]) || +search_bucket_single (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + num_matches++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; + } }return num_matches; }
@@ -244,12 +281,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -search_bucket_multi (prim_bucket, tmp_sig, buckets, &num_matches, - match_per_key, set_id); -if (num_matches < match_per_key) -search_bucket_multi (sec_bucket, tmp_sig, - buckets, &num_matches, match_per_key, set_id); -return num_matches; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_bucket, tmp_sig, buckets, + &num_matches, match_per_key, set_id); +if (num_matches < match_per_key) +search_bucket_multi_avx (sec_bucket, tmp_sig, + buckets, &num_matches, match_per_key, set_id); +return num_matches; +#endif +default : +search_bucket_multi (prim_bucket, tmp_sig, buckets, &num_matches, + match_per_key, set_id); +if (num_matches < match_per_key) +search_bucket_multi (sec_bucket, tmp_sig, + buckets, &num_matches, match_per_key, set_id); +return num_matches; + } }uint32_t
@@ -275,16 +324,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,for (i =0 ; i < num_keys; i++) { match_cnt_tmp =0 ; -search_bucket_multi (prim_buckets[i], tmp_sig[i], - buckets, &match_cnt_tmp, match_per_key, - &set_ids[i*match_per_key]); -if (match_cnt_tmp < match_per_key) -search_bucket_multi (sec_buckets[i], tmp_sig[i], +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_buckets[i], tmp_sig[i], buckets, &match_cnt_tmp, match_per_key, &set_ids[i*match_per_key]); - match_count[i] = match_cnt_tmp; -if (match_cnt_tmp !=0 ) - num_matches++; +if (match_cnt_tmp < match_per_key) +search_bucket_multi_avx (sec_buckets[i], + tmp_sig[i], buckets, &match_cnt_tmp, + match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_tmp; +if (match_cnt_tmp !=0 ) + num_matches++; +break ; +#endif +default : +search_bucket_multi (prim_buckets[i], tmp_sig[i], + buckets, &match_cnt_tmp, match_per_key, + &set_ids[i*match_per_key]); +if (match_cnt_tmp < match_per_key) +search_bucket_multi (sec_buckets[i], tmp_sig[i], + buckets, &match_cnt_tmp, match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_tmp; +if (match_cnt_tmp !=0 ) + num_matches++; + } }return num_matches; }
@@ -315,11 +382,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,static inline int try_update (struct member_ht_bucket *buckets,uint32_t prim,uint32_t sec, -member_sig_t sig,member_set_t set_id) +member_sig_t sig,member_set_t set_id, +enum rte_member_sig_compare_function cmp_fn) { -if (update_entry_search (prim, sig, buckets, set_id) || -update_entry_search (sec, sig, buckets, set_id)) -return 0 ; +switch (cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (update_entry_search_avx (prim, sig, buckets, set_id) || +update_entry_search_avx (sec, sig, buckets, + set_id)) +return 0 ; +break ; +#endif +default : +if (update_entry_search (prim, sig, buckets, set_id) || +update_entry_search (sec, sig, buckets, + set_id)) +return 0 ; + }return -1 ; }
@@ -430,7 +510,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,*/ if (ss->cache) { ret =try_update (buckets, prim_bucket, sec_bucket, tmp_sig, - set_id); + set_id, ss->sig_cmp_fn);if (ret !=-1 )return ret; }
lib/librte_member/rte_member_x86.h new file +107 -0
107 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH v5 4/7] member: add AVX for HT mode
Quoted 12 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 4/7] member: add AVX for HT mode
Quoted 8 lines
For key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contiguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.
This patch adds AVX2 implementation in a separate header file.
Signed-off-by Yipeng Wang <[email protected]>
PATCH v5 5/7 member: enable the library
In reply to Yipeng WangThis patch enables the Membership library.
--- MAINTAINERS | 6 +++++- config/common_base | 5 +++++ lib/librte_member/Makefile | 2 ++ mk/rte.app.mk | 2 ++ 4 files changed, 14 insertions(+), 1 deletion(-)
MAINTAINERS +5 -1
@@ -726,6 +726,11 @@ F: test/test/test_lpm* F: test/test/test_func_reentrancy.c F: test/test/test_xmmt_ops.h +Membership - EXPERIMENTAL +M: Yipeng Wang <[email protected]> +M: Sameh Gobriel <[email protected]> +F: lib/librte_member/ + Traffic metering M: Cristian Dumitrescu <[email protected]> F: lib/librte_meter/
@@ -734,7 +739,6 @@ F: test/test/test_meter.c F: examples/qos_meter/ F: doc/guides/sample_app_ug/qos_metering.rst - Other libraries ---------------
config/common_base +5 -0
@@ -595,6 +595,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n CONFIG_RTE_LIBRTE_EFD=y # +# Compile librte_member +# +CONFIG_RTE_LIBRTE_MEMBER=y + +# # Compile librte_jobstats # CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
@@ -37,6 +37,8 @@ LIB = librte_member.a CFLAGS := -I$(SRCDIR) $(CFLAGS) CFLAGS += $(WERROR_FLAGS) -O3 +LDLIBS += -lm + EXPORT_MAP := rte_member_version.map LIBABIVER := 1
mk/rte.app.mk +2 -0
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile _LDLIBS-y += --whole-archive _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt _LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma endif
Signature · 1 line
2.7.4
Re: [PATCH v5 5/7] member: enable the library
Quoted 12 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 5/7] member: enable the library
Quoted 3 lines
This patch enables the Membership library.
Signed-off-by Yipeng Wang <[email protected]>
PATCH v5 6/7 test/member: add functional and perf tests
In reply to Yipeng WangThis patch adds functional and performance tests for membership library.
--- MAINTAINERS | 1 + test/test/Makefile | 3 + test/test/test_member.c | 744 +++++++++++++++++++++++++++++++++++++++++++ test/test/test_member_perf.c | 654 +++++++++++++++++++++++++++++++++++++ 4 files changed, 1402 insertions(+) create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
MAINTAINERS +1 -0
@@ -730,6 +730,7 @@ Membership - EXPERIMENTAL M: Yipeng Wang <[email protected]> M: Sameh Gobriel <[email protected]> F: lib/librte_member/ +F: test/test/test_member* Traffic metering M: Cristian Dumitrescu <[email protected]>
test/test/Makefile +3 -0
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c SRCS-y += test_memcpy.c SRCS-y += test_memcpy_perf.c + +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
test/test/test_member.c new file +744 -0
744 lines not on this page · raw message
test/test/test_member_perf.c new file +654 -0
654 lines not on this page · raw message
Signature · 1 line
2.7.4
Re: [PATCH v5 6/7] test/member: add functional and perf tests
Quoted 10 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 6/7] test/member: add functional and perf tests
Quoted 3 lines
This patch adds functional and performance tests for membership library.
Signed-off-by Yipeng Wang <[email protected]>
PATCH v5 7/7 doc: add membership documentation
In reply to Yipeng WangThis patch adds the documentation for membership library.
--- MAINTAINERS | 1 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 ++++ doc/guides/prog_guide/img/member_i6.svg | 332 +++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 420 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + 13 files changed, 3596 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst
MAINTAINERS +1 -0
@@ -731,6 +731,7 @@ M: Yipeng Wang <[email protected]> M: Sameh Gobriel <[email protected]> F: lib/librte_member/ F: test/test/test_member* +F: doc/guides/prog_guide/member_lib.rst Traffic metering M: Cristian Dumitrescu <[email protected]>
doc/api/doxy-api-index.md +2 -1
@@ -105,7 +105,8 @@ The public API headers are grouped by topics: [LPM IPv4 route] (@ref rte_lpm.h), [LPM IPv6 route] (@ref rte_lpm6.h), [ACL] (@ref rte_acl.h), - [EFD] (@ref rte_efd.h) + [EFD] (@ref rte_efd.h), + [member] (@ref rte_member.h) - **QoS**: [metering] (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
@@ -58,6 +58,7 @@ INPUT = doc/api/doxy-api-index.md \ lib/librte_mempool \ lib/librte_meter \ lib/librte_metrics \ + lib/librte_member \ lib/librte_net \ lib/librte_pdump \ lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0
1,613 lines not on this page · raw message
doc/guides/prog_guide/img/member_i2.svg new file +36 -0
@@ -0,0 +1,36 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in" + viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6"> + <v:documentProperties v:langID="1033" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {fill:none;stroke:none;stroke-width:0.25} + .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em} + .st3 {baseline-shift:32.4943%;font-size:0.649886em} + .st4 {font-size:1em} + .st5 {font-family:Cambria Math;font-size:1em} + .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/> + <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)"> + <title>Sheet.3</title> + <desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="157.5" cy="45.5" width="315" height="90"/> + <rect x="0" y="0.5" width="315" height="90" class="st1"/> + <text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan + dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan + dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan + class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3" + v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em" + class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text> </g> + </g> +</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0
148 lines not on this page · raw message
doc/guides/prog_guide/img/member_i4.svg new file +450 -0
450 lines not on this page · raw message
doc/guides/prog_guide/img/member_i5.svg new file +163 -0
163 lines not on this page · raw message
doc/guides/prog_guide/img/member_i6.svg new file +332 -0
332 lines not on this page · raw message
doc/guides/prog_guide/img/member_i7.svg new file +399 -0
235 lines not on this page · raw message
doc/guides/prog_guide/member_lib.rst new file +420 -0
420 lines not on this page · raw message
doc/guides/rel_notes/release_17_11.rst +17 -0
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.7.4
Part of the diff is not shown here. View the raw message.
Re: [PATCH v5 7/7] doc: add membership documentation
Quoted 10 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Tuesday, October 3, 2017 5:32 AM
To: [email protected]; De Lara Guarch, Pablo <[email protected]>
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v5 7/7] doc: add membership documentation
Quoted 4 lines
This patch adds the documentation for membership library.
Signed-off-by Yipeng Wang <[email protected]>Reviewed-by John McNamara <[email protected]>
v6 cover Add Membership Library
In reply to Yipeng WangDPDK Membership Library provides an API that can be used by many DPDK applications to conduct one or more set-membership tests (we mention some possible use cases below, but interested readers can refer to
[1] for a wider survey of use cases).
The basic functionalities of the Membership Library include inserting a new member, deleting an existing member, and querying the existence of a member. The query result would indicate with high accuracy which specific set this member belongs to among a group of sets.
The Membership Library is an extension and generalization of traditional filter data structures [2,3], which maintain a space-efficient “set-summary”. There are two advantages of using such a set-summary rather than operating on a “full-blown” complete list of elements: firstly it has a much smaller storage requirement than storing the whole list of elements, and secondly set membership tests (or other operations) is much more efficient than searching through the complete list of elements.
A membership test for an element will return the set this element belongs to if found (or return "not-found") with high accuracy. If needed, the accuracy of the membership tests could be further increased with larger storage space. Set-summary is a fundamental data aggregation component that can be used in many network applications. It is a crucial structure to address performance and scalability issues of diverse applications including overlay networks, wild card flow classification, web-caches, load balancing, connection tracking, data-centric networks, flow table summaries, network statistics and traffic monitoring. Our Proof of Concept (PoC) using set-summary to optimize flow lookup in Open vSwitch (OvS) shows a speedup of about 2-3X.
This patch set implements two types of set-summaries, i.e., hash-table based set-summary (HTSS) and Vector Bloom Filter (vBF). HTSS supports both the non-cache and cache modes. The non-cache mode can incur a small chance of false-positives which is the case when the set-summary indicates a key belongs to a given set while actually it is not. The cache mode can also have false-negatives in addition to false-positives. False-negatives means the case when the set-summary indicates a key does not belong to a given set while actually it does. This happens because cache mode allows new key to evict existing keys. vBF only has false-positives similar to the non-cache HTSS. However, one can set the false-positive rate arbitrarily. HTSS's false-positive rate is determined by the hash-table size and the signature size.
[1] A Broder and M Mitzenmacher, “Network Applications of Bloom Filters: A Survey,” in Internet Mathematics, 2005.
[2] B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
[3] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
v6:
- member lib: minor coding style changes from Pablo's comment and checkpatch report.
- member lib: minor log message changes.
v5:
- member lib: incorperate Pablo's review comments, including: multiple places with coding style issues, change variable/function names to be more clear, reorder code blocks, add descriptions, use macros for constant numbers, fix bugs in log message, etc.
- test: modify the deletion function test to cover more cases.
- test: add check in performance test to check false negatvie for HT non-cache mode.
- MAINTAINERS: complete in multiple commits instead of one as Pablo suggested.
v4:
- member lib: change two hash function macros to be one. crc or jhash will be chosen depending on the architecture.
- member lib: use dynamic log type instead of static one by Thomas' comment.
- member lib: code cleanups (remove unnecessary code, change variable name, coding style, etc).
- member lib: reorder members in setsummary struct to be more cache friendly.
- member lib: setsummary struct member "name" should be char array rather than char pointer. Fixed.
- member lib: add check for two hash seeds to make sure they are not equal.
- member lib: vBF lookup speed optimization.
- member lib: revise comments in various places as Pablo suggested.
- member lib: change "void *" to "struct rte_member_setsum *" in public API as Pablo suggested. test case is modified accordingly to directly use the struct.
- member lib: change order in rte_member_version.map as Pablo suggested.
- MAINTAINERS: change the maintainer block order according to Thomas' comment.
- test: add printf to show which section expects error messages. Change set count to 16 from 32 for performance test.
- documentation: revise several places according to Pablo's and John's comments.
v3:
- documentation: update documentation to incorporate John's review.
v2:
- test: add bad parameter test functions to test the creation fail case.
- test: add find existing test.
- member lib: Add num_entries check in rte_member_create_ht.
- member lib: Add multiple checks for rte_member_create_vbf to avoid divide-by-0.
- member lib: remove unnecessary ret from rte_member.c according to Stephen's comment.
- member lib: change the vBF creation function to be not too conservative. Previous algorithm is too conservative on false positive.
- member lib: fix uninitialized issue fail gcc-4.8 in rte_member_find_existing.
- member lib: add rte_member_find_existing to version map.
- makefile: lib/librte_member/Makefile: change include order according to Luca's comment.
- documentation: update the programmer guide for membership library.
Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation
MAINTAINERS | 8 +- config/common_base | 5 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 ++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 +++ doc/guides/prog_guide/img/member_i6.svg | 332 ++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 420 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + lib/Makefile | 2 + lib/librte_member/Makefile | 51 + lib/librte_member/rte_member.c | 337 +++++++ lib/librte_member/rte_member.h | 513 ++++++++++ lib/librte_member/rte_member_ht.c | 586 +++++++++++ lib/librte_member/rte_member_ht.h | 94 ++ lib/librte_member/rte_member_vbf.c | 350 +++++++ lib/librte_member/rte_member_vbf.h | 82 ++ lib/librte_member/rte_member_version.map | 16 + lib/librte_member/rte_member_x86.h | 107 ++ mk/rte.app.mk | 2 + test/test/Makefile | 3 + test/test/test_member.c | 744 ++++++++++++++ test/test/test_member_perf.c | 654 ++++++++++++ 28 files changed, 7148 insertions(+), 2 deletions(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h create mode 100644 lib/librte_member/rte_member_version.map create mode 100644 lib/librte_member/rte_member_x86.h create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
Signature · 1 line
2.7.4
PATCH v6 1/7 member: implement main API
Membership library is an extension and generalization of a traditional filter (for example Bloom Filter and cuckoo filter) structure. In general, the Membership library is a data structure that provides a "set-summary" and responds to set-membership queries of whether a certain element belongs to a set(s). A membership test for an element will return the set this element belongs to or not-found if the element is never inserted into the set-summary.
The results of the membership test are not 100% accurate. Certain false positive or false negative probability could exist. However, comparing to a "full-blown" complete list of elements, a "set-summary" is memory efficient and fast on lookup.
This patch adds the main API definition.
--- lib/Makefile | 2 + lib/librte_member/Makefile | 49 +++ lib/librte_member/rte_member.c | 337 ++++++++++++++++++++ lib/librte_member/rte_member.h | 513 +++++++++++++++++++++++++++++++ lib/librte_member/rte_member_version.map | 16 + 5 files changed, 917 insertions(+) create mode 100644 lib/librte_member/Makefile create mode 100644 lib/librte_member/rte_member.c create mode 100644 lib/librte_member/rte_member.h create mode 100644 lib/librte_member/rte_member_version.map
lib/Makefile +2 -0
@@ -108,6 +108,8 @@ DIRS-$(CONFIG_RTE_LIBRTE_REORDER) += librte_reorder DEPDIRS-librte_reorder := librte_eal librte_mempool librte_mbuf DIRS-$(CONFIG_RTE_LIBRTE_PDUMP) += librte_pdump DEPDIRS-librte_pdump := librte_eal librte_mempool librte_mbuf librte_ether +DIRS-$(CONFIG_RTE_LIBRTE_MEMBER) += librte_member +DEPDIRS-librte_member := librte_eal librte_hash ifeq ($(CONFIG_RTE_EXEC_ENV_LINUXAPP),y) DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
lib/librte_member/Makefile new file +49 -0
49 lines not on this page · raw message
lib/librte_member/rte_member.c new file +337 -0
337 lines not on this page · raw message
lib/librte_member/rte_member.h new file +513 -0
513 lines not on this page · raw message
lib/librte_member/rte_member_version.map new file +16 -0
@@ -0,0 +1,16 @@ +DPDK_17.11 { + global: + + rte_member_add; + rte_member_create; + rte_member_delete; + rte_member_find_existing; + rte_member_free; + rte_member_lookup; + rte_member_lookup_bulk; + rte_member_lookup_multi; + rte_member_lookup_multi_bulk; + rte_member_reset; + + local: *; +};
Signature · 1 line
2.7.4
PATCH v6 2/7 member: implement HT mode
In reply to Yipeng WangOne of the set-summary structures is hash-table based set-summary (HTSS). One example is cuckoo filter [1].
Comparing to a traditional hash table, HTSS has a much more compact structure. For each element, only one signature and its corresponding set ID is stored. No key comparison is required during lookup. For the table structure, there are multiple entries in each bucket, and the table is composed of many buckets.
Two modes are supported for HTSS, "cache" and "none-cache" modes. The non-cache mode is similar to the cuckoo filter [1]. When a bucket is full, one entry will be evicted to its alternative bucket to make space for the new key. The table could be full and then no more keys could be inserted. This mode has false-positive rate but no false-negative. Multiple entries with same signature could stay in the same bucket.
The "cache" mode does not evict key to its alternative bucket when a bucket is full, an existing key will be evicted out of the table like a cache. Thus, the table will never reject keys when it is full. Another property is in each bucket, there cannot be multiple entries with same signature. The mode could have both false-positive and false-negative probability.
This patch adds the implementation of HTSS.
[1] B Fan, D G Andersen and M Kaminsky, “Cuckoo Filter: Practically Better Than Bloom,” in Conference on emerging Networking Experiments and Technologies, 2014.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_ht.c | 506 ++++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_ht.h | 94 +++++++ 3 files changed, 601 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_ht.c create mode 100644 lib/librte_member/rte_member_ht.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_ht.c new file +506 -0
506 lines not on this page · raw message
lib/librte_member/rte_member_ht.h new file +94 -0
94 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v6 3/7 member: implement vBF mode
In reply to Yipeng WangBloom Filter (BF) [1] is a well-known space-efficient probabilistic data structure that answers set membership queries. Vector of Bloom Filters (vBF) is an extension to traditional BF that supports multi-set membership testing. Traditional BF will return found or not-found for each key. vBF will also return which set the key belongs to if it is found.
Since each set requires a BF, vBF should be used when set count is small. vBF's false positive rate could be set appropriately so that its memory requirement and lookup speed is better in certain cases comparing to HT based set-summary.
This patch adds the vBF implementation.
[1]B H Bloom, “Space/Time Trade-offs in Hash Coding with Allowable Errors,” Communications of the ACM, 1970.
--- lib/librte_member/Makefile | 2 +- lib/librte_member/rte_member_vbf.c | 350 +++++++++++++++++++++++++++++++++++++ lib/librte_member/rte_member_vbf.h | 82 +++++++++ 3 files changed, 433 insertions(+), 1 deletion(-) create mode 100644 lib/librte_member/rte_member_vbf.c create mode 100644 lib/librte_member/rte_member_vbf.h
lib/librte_member/Makefile +1 -1
@@ -42,7 +42,7 @@ EXPORT_MAP := rte_member_version.map LIBABIVER := 1 # all source are stored in SRCS-y -SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += rte_member.c rte_member_ht.c rte_member_vbf.c # install includes SYMLINK-$(CONFIG_RTE_LIBRTE_MEMBER)-include := rte_member.h
lib/librte_member/rte_member_vbf.c new file +350 -0
350 lines not on this page · raw message
lib/librte_member/rte_member_vbf.h new file +82 -0
82 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v6 4/7 member: add AVX for HT mode
In reply to Yipeng WangFor key search, the signatures of all entries are compared against the signature of the key that is being looked up. Since all signatures are contiguously put in a bucket, they can be compared with vector instructions (AVX2), achieving higher lookup performance.
This patch adds AVX2 implementation in a separate header file.
--- lib/librte_member/rte_member_ht.c | 142 +++++++++++++++++++++++++++++-------- lib/librte_member/rte_member_x86.h | 107 ++++++++++++++++++++++++++++ 2 files changed, 218 insertions(+), 31 deletions(-) create mode 100644 lib/librte_member/rte_member_x86.h
lib/librte_member/rte_member_ht.c +111 -31
@@ -40,6 +40,10 @@#include "rte_member.h" #include "rte_member_ht.h" +#if defined(RTE_ARCH_X86 ) +#include "rte_member_x86.h" +#endif +/* Search bucket for entry with tmp_sig and update set_id */ static inline int update_entry_search(uint32_t bucket_id, member_sig_t tmp_sig,
@@ -136,6 +140,13 @@ rte_member_create_ht(struct rte_member_setsum *ss,for (j =0 ; j <RTE_MEMBER_BUCKET_ENTRIES ; j++) buckets[i].sets[j] =RTE_MEMBER_NO_MATCH ; } +#if defined(RTE_ARCH_X86 ) +if (rte_cpu_get_flag_enabled (RTE_CPUFLAG_AVX2 ) && +RTE_MEMBER_BUCKET_ENTRIES ==16 ) + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_AVX2 ; + else +#endif + ss->sig_cmp_fn =RTE_MEMBER_COMPARE_SCALAR ; RTE_MEMBER_LOG(DEBUG, "Hash table based filter created, " "the table has %u entries, %u buckets\n",
@@ -193,11 +204,23 @@ rte_member_lookup_ht(const struct rte_member_setsum *ss, *set_id =RTE_MEMBER_NO_MATCH ;get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -if (search_bucket_single (prim_bucket, tmp_sig, buckets, - set_id) || -search_bucket_single (sec_bucket, tmp_sig, - buckets, set_id)) -return 1 ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single_avx (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; +break ; +#endif +default : +if (search_bucket_single (prim_bucket, tmp_sig, buckets, + set_id) || +search_bucket_single (sec_bucket, tmp_sig, + buckets, set_id)) +return 1 ; + }return 0 ; }
@@ -221,13 +244,27 @@ rte_member_lookup_bulk_ht(const struct rte_member_setsum *ss, }for (i =0 ; i < num_keys; i++) { -if (search_bucket_single (prim_buckets[i], tmp_sig[i], - buckets, &set_id[i]) || -search_bucket_single (sec_buckets[i], - tmp_sig[i], buckets, &set_id[i])) - num_matches++; -else - set_id[i] =RTE_MEMBER_NO_MATCH ; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (search_bucket_single_avx (prim_buckets[i], + tmp_sig[i], buckets, &set_id[i]) || +search_bucket_single_avx (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + num_matches++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; +break ; +#endif +default : +if (search_bucket_single (prim_buckets[i], tmp_sig[i], + buckets, &set_id[i]) || +search_bucket_single (sec_buckets[i], + tmp_sig[i], buckets, &set_id[i])) + num_matches++; +else + set_id[i] =RTE_MEMBER_NO_MATCH ; + } }return num_matches; }
@@ -244,12 +281,24 @@ rte_member_lookup_multi_ht(const struct rte_member_setsum *ss,get_buckets_index (ss, key, &prim_bucket, &sec_bucket, &tmp_sig); -search_bucket_multi (prim_bucket, tmp_sig, buckets, &num_matches, - match_per_key, set_id); -if (num_matches < match_per_key) -search_bucket_multi (sec_bucket, tmp_sig, - buckets, &num_matches, match_per_key, set_id); -return num_matches; +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_bucket, tmp_sig, buckets, + &num_matches, match_per_key, set_id); +if (num_matches < match_per_key) +search_bucket_multi_avx (sec_bucket, tmp_sig, + buckets, &num_matches, match_per_key, set_id); +return num_matches; +#endif +default : +search_bucket_multi (prim_bucket, tmp_sig, buckets, &num_matches, + match_per_key, set_id); +if (num_matches < match_per_key) +search_bucket_multi (sec_bucket, tmp_sig, + buckets, &num_matches, match_per_key, set_id); +return num_matches; + } }uint32_t
@@ -275,16 +324,34 @@ rte_member_lookup_multi_bulk_ht(const struct rte_member_setsum *ss,for (i =0 ; i < num_keys; i++) { match_cnt_tmp =0 ; -search_bucket_multi (prim_buckets[i], tmp_sig[i], - buckets, &match_cnt_tmp, match_per_key, - &set_ids[i*match_per_key]); -if (match_cnt_tmp < match_per_key) -search_bucket_multi (sec_buckets[i], tmp_sig[i], +switch (ss->sig_cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +search_bucket_multi_avx (prim_buckets[i], tmp_sig[i], buckets, &match_cnt_tmp, match_per_key, &set_ids[i*match_per_key]); - match_count[i] = match_cnt_tmp; -if (match_cnt_tmp !=0 ) - num_matches++; +if (match_cnt_tmp < match_per_key) +search_bucket_multi_avx (sec_buckets[i], + tmp_sig[i], buckets, &match_cnt_tmp, + match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_tmp; +if (match_cnt_tmp !=0 ) + num_matches++; +break ; +#endif +default : +search_bucket_multi (prim_buckets[i], tmp_sig[i], + buckets, &match_cnt_tmp, match_per_key, + &set_ids[i*match_per_key]); +if (match_cnt_tmp < match_per_key) +search_bucket_multi (sec_buckets[i], tmp_sig[i], + buckets, &match_cnt_tmp, match_per_key, + &set_ids[i*match_per_key]); + match_count[i] = match_cnt_tmp; +if (match_cnt_tmp !=0 ) + num_matches++; + } }return num_matches; }
@@ -315,11 +382,24 @@ try_insert(struct member_ht_bucket *buckets, uint32_t prim, uint32_t sec,static inline int try_update (struct member_ht_bucket *buckets,uint32_t prim,uint32_t sec, -member_sig_t sig,member_set_t set_id) +member_sig_t sig,member_set_t set_id, +enum rte_member_sig_compare_function cmp_fn) { -if (update_entry_search (prim, sig, buckets, set_id) || -update_entry_search (sec, sig, buckets, set_id)) -return 0 ; +switch (cmp_fn) { +#if defined(RTE_ARCH_X86 ) && defined(RTE_MACHINE_CPUFLAG_AVX2 ) +case RTE_MEMBER_COMPARE_AVX2 : +if (update_entry_search_avx (prim, sig, buckets, set_id) || +update_entry_search_avx (sec, sig, buckets, + set_id)) +return 0 ; +break ; +#endif +default : +if (update_entry_search (prim, sig, buckets, set_id) || +update_entry_search (sec, sig, buckets, + set_id)) +return 0 ; + }return -1 ; }
@@ -430,7 +510,7 @@ rte_member_add_ht(const struct rte_member_setsum *ss,*/ if (ss->cache) { ret =try_update (buckets, prim_bucket, sec_bucket, tmp_sig, - set_id); + set_id, ss->sig_cmp_fn);if (ret !=-1 )return ret; }
lib/librte_member/rte_member_x86.h new file +107 -0
107 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v6 5/7 member: enable the library
In reply to Yipeng WangThis patch enables the Membership library.
--- MAINTAINERS | 6 +++++- config/common_base | 5 +++++ lib/librte_member/Makefile | 2 ++ mk/rte.app.mk | 2 ++ 4 files changed, 14 insertions(+), 1 deletion(-)
MAINTAINERS +5 -1
@@ -729,6 +729,11 @@ F: test/test/test_lpm* F: test/test/test_func_reentrancy.c F: test/test/test_xmmt_ops.h +Membership - EXPERIMENTAL +M: Yipeng Wang <[email protected]> +M: Sameh Gobriel <[email protected]> +F: lib/librte_member/ + Traffic metering M: Cristian Dumitrescu <[email protected]> F: lib/librte_meter/
@@ -737,7 +742,6 @@ F: test/test/test_meter.c F: examples/qos_meter/ F: doc/guides/sample_app_ug/qos_metering.rst - Other libraries ---------------
config/common_base +5 -0
@@ -596,6 +596,11 @@ CONFIG_RTE_LIBRTE_HASH_DEBUG=n CONFIG_RTE_LIBRTE_EFD=y # +# Compile librte_member +# +CONFIG_RTE_LIBRTE_MEMBER=y + +# # Compile librte_jobstats # CONFIG_RTE_LIBRTE_JOBSTATS=y
lib/librte_member/Makefile +2 -0
@@ -37,6 +37,8 @@ LIB = librte_member.a CFLAGS := -I$(SRCDIR) $(CFLAGS) CFLAGS += $(WERROR_FLAGS) -O3 +LDLIBS += -lm + EXPORT_MAP := rte_member_version.map LIBABIVER := 1
mk/rte.app.mk +2 -0
@@ -86,6 +86,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CFGFILE) += -lrte_cfgfile _LDLIBS-y += --whole-archive _LDLIBS-$(CONFIG_RTE_LIBRTE_HASH) += -lrte_hash +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lrte_member _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost _LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs _LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
@@ -196,6 +197,7 @@ endif _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lm _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt _LDLIBS-$(CONFIG_RTE_LIBRTE_METER) += -lm +_LDLIBS-$(CONFIG_RTE_LIBRTE_MEMBER) += -lm ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y) _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma endif
Signature · 1 line
2.7.4
PATCH v6 6/7 test/member: add functional and perf tests
In reply to Yipeng WangThis patch adds functional and performance tests for membership library.
--- MAINTAINERS | 1 + test/test/Makefile | 3 + test/test/test_member.c | 744 +++++++++++++++++++++++++++++++++++++++++++ test/test/test_member_perf.c | 654 +++++++++++++++++++++++++++++++++++++ 4 files changed, 1402 insertions(+) create mode 100644 test/test/test_member.c create mode 100644 test/test/test_member_perf.c
MAINTAINERS +1 -0
@@ -733,6 +733,7 @@ Membership - EXPERIMENTAL M: Yipeng Wang <[email protected]> M: Sameh Gobriel <[email protected]> F: lib/librte_member/ +F: test/test/test_member* Traffic metering M: Cristian Dumitrescu <[email protected]>
test/test/Makefile +3 -0
@@ -123,6 +123,9 @@ SRCS-y += test_logs.c SRCS-y += test_memcpy.c SRCS-y += test_memcpy_perf.c + +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member.c +SRCS-$(CONFIG_RTE_LIBRTE_MEMBER) += test_member_perf.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd.c SRCS-$(CONFIG_RTE_LIBRTE_EFD) += test_efd_perf.c
test/test/test_member.c new file +744 -0
744 lines not on this page · raw message
test/test/test_member_perf.c new file +654 -0
654 lines not on this page · raw message
Signature · 1 line
2.7.4
PATCH v6 7/7 doc: add membership documentation
In reply to Yipeng WangThis patch adds the documentation for membership library.
--- MAINTAINERS | 1 + doc/api/doxy-api-index.md | 3 +- doc/api/doxy-api.conf | 1 + doc/guides/prog_guide/img/member_i1.svg | 1613 +++++++++++++++++++++++++++++++ doc/guides/prog_guide/img/member_i2.svg | 36 + doc/guides/prog_guide/img/member_i3.svg | 148 +++ doc/guides/prog_guide/img/member_i4.svg | 450 +++++++++ doc/guides/prog_guide/img/member_i5.svg | 163 ++++ doc/guides/prog_guide/img/member_i6.svg | 332 +++++++ doc/guides/prog_guide/img/member_i7.svg | 399 ++++++++ doc/guides/prog_guide/index.rst | 14 + doc/guides/prog_guide/member_lib.rst | 420 ++++++++ doc/guides/rel_notes/release_17_11.rst | 17 + 13 files changed, 3596 insertions(+), 1 deletion(-) create mode 100644 doc/guides/prog_guide/img/member_i1.svg create mode 100644 doc/guides/prog_guide/img/member_i2.svg create mode 100644 doc/guides/prog_guide/img/member_i3.svg create mode 100644 doc/guides/prog_guide/img/member_i4.svg create mode 100644 doc/guides/prog_guide/img/member_i5.svg create mode 100644 doc/guides/prog_guide/img/member_i6.svg create mode 100644 doc/guides/prog_guide/img/member_i7.svg create mode 100644 doc/guides/prog_guide/member_lib.rst
MAINTAINERS +1 -0
@@ -734,6 +734,7 @@ M: Yipeng Wang <[email protected]> M: Sameh Gobriel <[email protected]> F: lib/librte_member/ F: test/test/test_member* +F: doc/guides/prog_guide/member_lib.rst Traffic metering M: Cristian Dumitrescu <[email protected]>
doc/api/doxy-api-index.md +2 -1
@@ -105,7 +105,8 @@ The public API headers are grouped by topics: [LPM IPv4 route] (@ref rte_lpm.h), [LPM IPv6 route] (@ref rte_lpm6.h), [ACL] (@ref rte_acl.h), - [EFD] (@ref rte_efd.h) + [EFD] (@ref rte_efd.h), + [member] (@ref rte_member.h) - **QoS**: [metering] (@ref rte_meter.h),
doc/api/doxy-api.conf +1 -0
@@ -58,6 +58,7 @@ INPUT = doc/api/doxy-api-index.md \ lib/librte_mempool \ lib/librte_meter \ lib/librte_metrics \ + lib/librte_member \ lib/librte_net \ lib/librte_pdump \ lib/librte_pipeline \
doc/guides/prog_guide/img/member_i1.svg new file +1613 -0
1,613 lines not on this page · raw message
doc/guides/prog_guide/img/member_i2.svg new file +36 -0
@@ -0,0 +1,36 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export memship_i2.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="4.38194in" height="1.25694in" + viewBox="0 0 315.5 90.5" xml:space="preserve" color-interpolation-filters="sRGB" class="st6"> + <v:documentProperties v:langID="1033" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {fill:none;stroke:none;stroke-width:0.25} + .st2 {fill:#5b9bd5;font-family:Calibri;font-size:1.16666em} + .st3 {baseline-shift:32.4943%;font-size:0.649886em} + .st4 {font-size:1em} + .st5 {font-family:Cambria Math;font-size:1em} + .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/> + <g id="shape3-1" v:mID="3" v:groupContext="shape" transform="translate(0.25,-0.25)"> + <title>Sheet.3</title> + <desc>False Positive Probability = (1-(1-1/m)kn)k ≃ (1-ekn/m)k</desc> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="157.5" cy="45.5" width="315" height="90"/> + <rect x="0" y="0.5" width="315" height="90" class="st1"/> + <text x="8.28" y="49.82" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>False Positive Probability = (1-(1-1/m)<tspan + dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan dy="0.152em" class="st4">)</tspan><tspan + dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan><tspan dy="0.152em" class="st4"> </tspan><tspan + class="st5">≃</tspan> (1-e<tspan dy="-0.234em" class="st3" v:baseFontSize="14">kn</tspan><tspan class="st3" + v:baseFontSize="14">/</tspan><tspan class="st3" v:baseFontSize="14">m</tspan><tspan dy="0.152em" + class="st4">)</tspan><tspan dy="-0.234em" class="st3" v:baseFontSize="14">k</tspan></text> </g> + </g> +</svg>
doc/guides/prog_guide/img/member_i3.svg new file +148 -0
148 lines not on this page · raw message
doc/guides/prog_guide/img/member_i4.svg new file +450 -0
450 lines not on this page · raw message
doc/guides/prog_guide/img/member_i5.svg new file +163 -0
163 lines not on this page · raw message
doc/guides/prog_guide/img/member_i6.svg new file +332 -0
332 lines not on this page · raw message
doc/guides/prog_guide/img/member_i7.svg new file +399 -0
235 lines not on this page · raw message
doc/guides/prog_guide/member_lib.rst new file +420 -0
420 lines not on this page · raw message
doc/guides/rel_notes/release_17_11.rst +17 -0
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.7.4
Part of the diff is not shown here. View the raw message.
Re: [PATCH v6 7/7] doc: add membership documentation
Quoted 12 lines
-----Original Message-----
From: Wang, Yipeng1
Sent: Wednesday, October 4, 2017 4:12 AM
To: [email protected]
Cc: [email protected]; Tai, Charlie <[email protected]>; Gobriel, Sameh <[email protected]>; De Lara Guarch, Pablo <[email protected]>; Mcnamara, John <[email protected]>; Wang, Yipeng1 <[email protected]>
Subject: [PATCH v6 7/7] doc: add membership documentation
Quoted 5 lines
This patch adds the documentation for membership library.
Signed-off-by Yipeng Wang <[email protected]>Reviewed-by John McNamara <[email protected]>Reviewed-by Pablo de Lara <[email protected]>
Thomas Monjalon
AppliedQuoted 8 lines
Yipeng Wang (7):
member: implement main API
member: implement HT mode
member: implement vBF mode
member: add AVX for HT mode
member: enable the library
test/member: add functional and perf tests
doc: add membership documentation
It looks good.
I have fixed some minor details (alphabetical sorting, alignments), and I have enabled the compilation from the first patch.
Applied, thanks