package/libmdbx: new package (library/database).
v2 cover package/libmdbx: new package (library/database).
This patch adds libmdbx v0.9.1 and below is a brief overview of libmdbx.
Please merge.
Regards,
Leonid.
--
libmdbx is an extremely fast, compact, powerful, embedded, transactional key-value database, with permissive license. libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
Historically, libmdbx (MDBX) is a deeply revised and extended descendant of the legendary LMDB (Lightning Memory-Mapped Database). libmdbx inherits all benefits from LMDB, but resolves some issues and adds a set of improvements.
According to developers, for now libmdbx surpasses the LMDB in terms of reliability, features and performance.
The most important differences MDBX from LMDB:
==============================================
1) More attention is paid to the quality of the code, to an "unbreakability" of the API, to testing and automatic checks (i.e. sanitizers, etc). So there:
- more control during operation;
- more checking parameters, internal audit of database structures;
- no warnings from compiler;
- no issues from ASAN, UBSAN, Valgrind, Coverity;
- etc.
2) Keys could be more than 2 times longer than LMDB.
3) Up to 20% faster than??LMDB??in??CRUD??benchmarks.
4) Automatic on-the-fly database size adjustment,
both increment and reduction.
5) Automatic continuous zero-overhead database compactification.
6) The same database format for 32- and 64-bit builds.
7) LIFO policy for Garbage Collection recycling (this can significantly increase write performance due write-back disk cache up to several times in a best case scenario).
8) Range query estimation.
9) Utility for checking the integrity of the database structure with some recovery capabilities.
For more info please refer:
- https://github.com/erthink/libmdbx for source code and README.
- https://erthink.github.io/libmdbx for API description.
Leonid Yuriev (1):
package/libmdbx: new package (library/database).
DEVELOPERS | 3 ++ package/Config.in | 1 + package/libmdbx/Config.in | 35 ++++++++++++++++++++++ package/libmdbx/libmdbx.hash | 2 ++ package/libmdbx/libmdbx.mk | 57 ++++++++++++++++++++++++++++++++++++ 5 files changed, 98 insertions(+) create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mk
--
MDBX is a Btree-based database management library modeled loosely on the BerkeleyDB API, but much simplified. The entire database (aka "environment") is exposed in a memory map, and all data fetches return data directly from the mapped memory, so no malloc's or memcpy's occur during data fetches. As such, the library is extremely simple because it requires no page caching layer of its own, and it is extremely high performance and memory-efficient. It is also fully transactional with full ACID semantics, and when the memory map is read-only, the database integrity cannot be corrupted by stray pointer writes from application code.
The library is fully thread-aware and supports concurrent read/write access from multiple processes and threads. Data pages use a copy-on-write strategy so no active data pages are ever overwritten, which also provides resistance to corruption and eliminates the need of any special recovery procedures after a system crash. Writes are fully serialized; only one write transaction may be active at a time, which guarantees that writers can never deadlock. The database structure is multi-versioned so readers run with no locks; writers cannot block readers, and readers don't block writers.
Unlike other well-known database mechanisms which use either write-ahead transaction logs or append-only data writes, MDBX requires no maintenance during operation. Both write-ahead loggers and append-only databases require periodic checkpointing and/or compaction of their log or database files otherwise they grow without bound. MDBX tracks retired/freed pages within the database and re-uses them for new write operations, so the database size does not grow without bound in normal use.
The memory map can be used as a read-only or read-write map. It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database.
Features
========
- Key-value data model, keys are always sorted.
- Fully ACID-compliant, through to MVCC and CoW.
- Multiple key-value sub-databases within a single datafile.
- Range lookups, including range query estimation.
- Efficient support for short fixed length keys, including native 32/64-bit integers.
- Ultra-efficient support for multimaps. Multi-values sorted, searchable and iterable. Keys stored without duplication.
- Data is memory-mapped and accessible directly/zero-copy. Traversal of database records is extremely-fast.
- Transactions for readers and writers, ones do not block others.
- Writes are strongly serialized. No transaction conflicts nor deadlocks.
- Readers are non-blocking, notwithstanding snapshot isolation.
- Nested write transactions.
- Reads scale linearly across CPUs.
- Continuous zero-overhead database compactification.
- Automatic on-the-fly database size adjustment.
- Customizable database page size.
- Olog(N) cost of lookup, insert, update, and delete operations by virtue of B+ tree characteristics.
- Online hot backup.
- Append operation for efficient bulk insertion of pre-sorted data.
- No WAL nor any transaction journal. No crash recovery needed. No maintenance is required.
- No internal cache and/or memory management, all done by basic OS services.
Limitations
===========
- Page size: a power of 2, maximum 65536 bytes, default 4096 bytes.
- Key size: minimum 0, maximum ????? pagesize (1300 bytes for default 4K pagesize, 21780 bytes for 64K pagesize).
- Value size: minimum 0, maximum 2146435072 (0x7FF00000) bytes for maps, ????? pagesize for multimaps (1348 bytes default 4K pagesize, 21828 bytes for 64K pagesize).
- Write transaction size: up to 4194301 (0x3FFFFD) pages (16 GiB for default 4K pagesize, 256 GiB for 64K pagesize).
- Database size: up to 2147483648 pages (8 TiB for default 4K pagesize, 128 TiB for 64K pagesize).
- Maximum sub-databases: 32765.
Gotchas
=======
- There cannot be more than one writer at a time, i.e. no more than one write transaction at a time.
- libmdbx is based on B+ tree, so access to database pages is mostly random. Thus SSDs provide a significant performance boost over spinning disks for large databases.
- libmdbx uses shadow paging instead of WAL. Thus syncing data to disk might be a bottleneck for write intensive workload.
- libmdbx uses copy-on-write for snapshot isolation during updates, but read transactions prevents recycling an old retired/freed pages, since it read ones. Thus altering of data during a parallel long-lived read operation will increase the process work set, may exhaust entire free database space, the database can grow quickly, and result in performance degradation. Try to avoid long running read transactions.
- libmdbx is extraordinarily fast and provides minimal overhead for data access, so you should reconsider using brute force techniques and double check your code. On the one hand, in the case of libmdbx, a simple linear search may be more profitable than complex indexes. On the other hand, if you make something suboptimally, you can notice detrimentally only on sufficiently large data.
PATCH v2 1/1 package/libmdbx: new package (library/database).
This patch adds libmdbx v0.9.1:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.
https://github.com/erthink/libmdbx
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)
--- DEVELOPERS | 3 ++ package/Config.in | 1 + package/libmdbx/Config.in | 35 ++++++++++++++++++++++ package/libmdbx/libmdbx.hash | 2 ++ package/libmdbx/libmdbx.mk | 57 ++++++++++++++++++++++++++++++++++++ 5 files changed, 98 insertions(+) create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mk
DEVELOPERS +3 -0
@@ -1544,6 +1544,9 @@ N: Leon Anavi <[email protected]> F: board/olimex/a10_olinuxino F: configs/olimex_a10_olinuxino_lime_defconfig +N: Leonid Yuriev <[email protected]> +F: package/libmdbx + N: Lionel Flandrin <[email protected]> F: package/python-babel/ F: package/python-daemonize/
package/Config.in +1 -0
@@ -1370,6 +1370,7 @@ menu "Database" source "package/kompexsqlite/Config.in" source "package/leveldb/Config.in" source "package/libgit2/Config.in" + source "package/libmdbx/Config.in" source "package/libodb/Config.in" source "package/libodb-boost/Config.in" source "package/libodb-mysql/Config.in"
package/libmdbx/Config.in new file +35 -0
@@ -0,0 +1,35 @@ +config BR2_PACKAGE_LIBMDBX + bool "libmdbx" + depends on BR2_USE_MMU + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + depends on BR2_TOOLCHAIN_HAS_SYNC_4 + help + One of the fastest compact key-value ACID database + without WAL. libmdbx has a specific set of properties + and capabilities, focused on creating unique lightweight + solutions. + + libmdbx surpasses the legendary legendary LMDB in terms of + reliability, features and performance. + + https://github.com/erthink/libmdbx + +if BR2_PACKAGE_LIBMDBX + +config BR2_PACKAGE_LIBMDBX_TOOLS + bool "install tools" + help + Install libmdbx tools for checking, dump, restore + and show statistics of databases. + +# C++ API is preliminary for now +#config BR2_PACKAGE_LIBMDBX_CXX +# bool "C++ API" +# select BR2_INSTALL_LIBSTDCPP +# help +# Enable modern C++ API for libmdbx. + +endif + +comment "libmdbx needs a toolchain w/ threads" + depends on !BR2_TOOLCHAIN_HAS_THREADS
package/libmdbx/libmdbx.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE +sha256 c7fb24381eb4d92f2e2edc17e577cb721269683c816c6cca307c58f6f346e786 libmdbx-amalgamated-0.9.1.tar.gz
package/libmdbx/libmdbx.mk new file +57 -0
57 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH v2 1/1 package/libmdbx: new package (library/database).
Hi Leonid,
Thank you for this contribution.
Below are some comments.
Am Fr., 2. Okt. 2020 um 13:22 Uhr schrieb Leonid Yuriev <[email protected]>:
Quoted 80 lines
This patch adds libmdbx v0.9.1:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.https://github.com/erthink/libmdbx
Signed-off-by Leonid Yuriev <[email protected]>---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)Signed-off-by Leonid Yuriev <[email protected]>--- DEVELOPERS | 3 ++ package/Config.in | 1 + package/libmdbx/Config.in | 35 ++++++++++++++++++++++ package/libmdbx/libmdbx.hash | 2 ++ package/libmdbx/libmdbx.mk | 57 ++++++++++++++++++++++++++++++++++++ 5 files changed, 98 insertions(+) create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mkDEVELOPERS +3
-0@@ -1544,6 +1544,9 @@ N: Leon Anavi <[email protected]> F: board/olimex/a10_olinuxino F: configs/olimex_a10_olinuxino_lime_defconfig +N: Leonid Yuriev <[email protected]> +F: package/libmdbx + N: Lionel Flandrin <[email protected]> F: package/python-babel/ F: package/python-daemonize/package/Config.in +1
-0@@ -1370,6 +1370,7 @@ menu "Database" source "package/kompexsqlite/Config.in" source "package/leveldb/Config.in" source "package/libgit2/Config.in" + source "package/libmdbx/Config.in" source "package/libodb/Config.in" source "package/libodb-boost/Config.in" source "package/libodb-mysql/Config.in"package/libmdbx/Config.in new file +20
-0@@ -0,0 +1,35 @@ +config BR2_PACKAGE_LIBMDBX + bool "libmdbx" + depends on BR2_USE_MMU + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + depends on BR2_TOOLCHAIN_HAS_SYNC_4 + help + One of the fastest compact key-value ACID database + without WAL. libmdbx has a specific set of properties + and capabilities, focused on creating unique lightweight + solutions. + + libmdbx surpasses the legendary legendary LMDB in terms of + reliability, features and performance. + + https://github.com/erthink/libmdbx + +if BR2_PACKAGE_LIBMDBX + +config BR2_PACKAGE_LIBMDBX_TOOLS + bool "install tools"
package/libmdbx/Config.in line 30
Quoted 10 lines
package/libmdbx/Config.in new file +10
-0+ help + Install libmdbx tools for checking, dump, restore + and show statistics of databases. + +# C++ API is preliminary for now +#config BR2_PACKAGE_LIBMDBX_CXX +# bool "C++ API" +# select BR2_INSTALL_LIBSTDCPP +# help +# Enable modern C++ API for libmdbx.
You should not keep this outcommented stuff here.
Quoted 19 lines
package/libmdbx/Config.in new file +5
-0+ +endif + +comment "libmdbx needs a toolchain w/ threads" + depends on !BR2_TOOLCHAIN_HAS_THREADSpackage/libmdbx/libmdbx.hash new file +2
-0@@ -0,0 +1,2 @@ +sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE +sha256 c7fb24381eb4d92f2e2edc17e577cb721269683c816c6cca307c58f6f346e786 libmdbx-amalgamated-0.9.1.tar.gzpackage/libmdbx/libmdbx.mk new file +0
-0
package/libmdbx/libmdbx.mk line 8
Quoted 8 lines
package/libmdbx/libmdbx.mk new file +8
-0@@ -0,0 +1,57 @@ +################################################################################ +# +# libmdbx +# +# One of the fastest embeddable key-value ACID database without WAL. libmdbx is +# superior to legendary LMDB in terms of reliability, features and performance. +# +################################################################################
The header of the makefile should only hold the name of the package without the description.
package/libmdbx/libmdbx.mk line 20
Quoted 12 lines
package/libmdbx/libmdbx.mk new file +12
-0+ +LIBMDBX_VERSION = 0.9.1 +LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz +LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION) +LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO +LIBMDBX_LICENSE = OLDAP-2.8 +LIBMDBX_LICENSE_FILES = LICENSE +LIBMDBX_REDISTRIBUTE = YES +LIBMDBX_STRIP_COMPONENTS = 0 +LIBMDBX_INSTALL_STAGING = YES + +LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES:BOOL=OFF -DBUILD_FOR_NATIVE_CPU:BOOL=OFF
You should omit the type stuff for the cmake config, like ":BOOL".
package/libmdbx/libmdbx.mk line 27
Quoted 7 lines
package/libmdbx/libmdbx.mk new file +7
-0+ +# C++ API is preliminary for now +#ifeq ($(BR2_PACKAGE_LIBMDBX_CXX),y) +#LIBMDBX_CONF_OPTS += -DMDBX_BUILD_CXX:BOOL=ON +#else +#LIBMDBX_CONF_OPTS += -DMDBX_BUILD_CXX:BOOL=OFF +#endif
Same here .. no out commented stuff.
package/libmdbx/libmdbx.mk line 33
Quoted 6 lines
package/libmdbx/libmdbx.mk new file +6
-0+ +ifeq ($(BR2_STATIC_LIBS),y) +LIBMDBX_CONF_OPTS += -DMDBX_INSTALL_STATIC:BOOL=ON +else +LIBMDBX_CONF_OPTS += -DMDBX_INSTALL_STATIC:BOOL=OFF +endif
This can be:
LIBMDBX_CONF_OPTS += \
-DMDBX_INSTALL_STATIC=$(if $(BR2_STATIC_LIBS),ON,OFF)The same pattern also for DMDBX_BUILD_TOOLS
LIBMDBX_CONF_OPTS += \
-DMDBX_BUILD_TOOLS=$(if $(BR2_PACKAGE_LIBMDBX_TOOLS),ON,OFF)package/libmdbx/libmdbx.mk line 37
Quoted 4 lines
package/libmdbx/libmdbx.mk new file +4
-0+ +ifeq ($(BR2_SHARED_LIBS),y) +LIBMDBX_CONF_OPTS += -DMDBX_BUILD_SHARED_LIBRARY:BOOL=ON +LIBMDBX_INSTALL_TARGET = YES
'YES' is the default value for <pkg>_INSGTALL_TARGET. so you do not need to set it.
Quoted 10 lines
package/libmdbx/libmdbx.mk new file +10
-0+ifeq ($(BR2_PACKAGE_LIBMDBX_TOOLS),y) +LIBMDBX_CONF_OPTS += -DMDBX_BUILD_TOOLS:BOOL=ON -DMDBX_LINK_TOOLS_NONSTATIC:BOOL=ON +else +LIBMDBX_CONF_OPTS += -DMDBX_BUILD_TOOLS:BOOL=OFF +endif + +else + +LIBMDBX_CONF_OPTS += -DMDBX_BUILD_SHARED_LIBRARY:BOOL=OFF +ifeq ($(BR2_PACKAGE_LIBMDBX_TOOLS),y)
package/libmdbx/libmdbx.mk line 57
Quoted 10 lines
package/libmdbx/libmdbx.mk new file +10
-0+LIBMDBX_CONF_OPTS += -DMDBX_BUILD_TOOLS:BOOL=ON -DMDBX_LINK_TOOLS_NONSTATIC:BOOL=OFF +LIBMDBX_INSTALL_TARGET = YES +else +LIBMDBX_CONF_OPTS += -DMDBX_BUILD_TOOLS:BOOL=OFF +LIBMDBX_INSTALL_TARGET = NO +endif + +endif + +$(eval $(cmake-package))
By the way .. there are two tools that help validating new package:
utils/tes-pkg -p libmdbx
utils/check-package package/libmdbx/*
Please check and fix the findings of these tests.
Thank you
Signature · 1 line
Heiko
v3 cover package/libmdbx: new package (library/database)
This patch adds libmdbx v0.9.1 and below is a brief overview of libmdbx.
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)
Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch for pthread_yield().
- passed utils/check-package package/libmdbx/* (suggested by Heiko Thiery).
- passed utils/test-pkg -a -p libmdbx (suggested by Heiko Thiery).
Please merge.
Regards,
Leonid.
--
libmdbx is an extremely fast, compact, powerful, embedded, transactional key-value database, with permissive license. libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
Historically, libmdbx (MDBX) is a deeply revised and extended descendant of the legendary LMDB (Lightning Memory-Mapped Database). libmdbx inherits all benefits from LMDB, but resolves some issues and adds a set of improvements.
According to developers, for now libmdbx surpasses the LMDB in terms of reliability, features and performance.
The most important differences MDBX from LMDB:
==============================================
1. More attention is paid to the quality of the code, to an "unbreakability" of the API, to testing and automatic checks (i.e. sanitizers, etc). So there:
- more control during operation;
- more checking parameters, internal audit of database structures;
- no warnings from compiler;
- no issues from ASAN, UBSAN, Valgrind, Coverity;
- etc.
2. Keys could be more than 2 times longer than LMDB.
3. Up to 20% faster than LMDB in CRUD benchmarks.
4. Automatic on-the-fly database size adjustment,
both increment and reduction.
5. Automatic continuous zero-overhead database compactification.
6. The same database format for 32- and 64-bit builds.
7. LIFO policy for Garbage Collection recycling (this can significantly increase write performance due write-back disk cache up to several times in a best case scenario).
8. Range query estimation.
9. Utility for checking the integrity of the database structure with some recovery capabilities.
For more info please refer:
- https://github.com/erthink/libmdbx for source code and README.
- https://erthink.github.io/libmdbx for API description.
--
MDBX is a Btree-based database management library modeled loosely on the BerkeleyDB API, but much simplified. The entire database (aka "environment") is exposed in a memory map, and all data fetches return data directly from the mapped memory, so no malloc's or memcpy's occur during data fetches. As such, the library is extremely simple because it requires no page caching layer of its own, and it is extremely high performance and memory-efficient. It is also fully transactional with full ACID semantics, and when the memory map is read-only, the database integrity cannot be corrupted by stray pointer writes from application code.
The library is fully thread-aware and supports concurrent read/write access from multiple processes and threads. Data pages use a copy-on-write strategy so no active data pages are ever overwritten, which also provides resistance to corruption and eliminates the need of any special recovery procedures after a system crash. Writes are fully serialized; only one write transaction may be active at a time, which guarantees that writers can never deadlock. The database structure is multi-versioned so readers run with no locks; writers cannot block readers, and readers don't block writers.
Unlike other well-known database mechanisms which use either write-ahead transaction logs or append-only data writes, MDBX requires no maintenance during operation. Both write-ahead loggers and append-only databases require periodic checkpointing and/or compaction of their log or database files otherwise they grow without bound. MDBX tracks retired/freed pages within the database and re-uses them for new write operations, so the database size does not grow without bound in normal use.
The memory map can be used as a read-only or read-write map. It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database.
Features
========
- Key-value data model, keys are always sorted.
- Fully ACID-compliant, through to MVCC and CoW.
- Multiple key-value sub-databases within a single datafile.
- Range lookups, including range query estimation.
- Efficient support for short fixed length keys, including native 32/64-bit integers.
- Ultra-efficient support for multimaps. Multi-values sorted, searchable and iterable. Keys stored without duplication.
- Data is memory-mapped and accessible directly/zero-copy. Traversal of database records is extremely-fast.
- Transactions for readers and writers, ones do not block others.
- Writes are strongly serialized. No transaction conflicts nor deadlocks.
- Readers are non-blocking, notwithstanding snapshot isolation.
- Nested write transactions.
- Reads scale linearly across CPUs.
- Continuous zero-overhead database compactification.
- Automatic on-the-fly database size adjustment.
- Customizable database page size.
- Olog(N) cost of lookup, insert, update, and delete operations by virtue of B+ tree characteristics.
- Online hot backup.
- Append operation for efficient bulk insertion of pre-sorted data.
- No WAL nor any transaction journal. No crash recovery needed. No maintenance is required.
- No internal cache and/or memory management, all done by basic OS services.
Limitations
===========
- Page size: a power of 2, maximum 65536 bytes, default 4096 bytes.
- Key size: minimum 0, maximum ?? pagesize (1300 bytes for default 4K pagesize, 21780 bytes for 64K pagesize).
- Value size: minimum 0, maximum 2146435072 (0x7FF00000) bytes for maps, ?? pagesize for multimaps (1348 bytes default 4K pagesize, 21828 bytes for 64K pagesize).
- Write transaction size: up to 4194301 (0x3FFFFD) pages (16 GiB for default 4K pagesize, 256 GiB for 64K pagesize).
- Database size: up to 2147483648 pages (8 TiB for default 4K pagesize, 128 TiB for 64K pagesize).
- Maximum sub-databases: 32765.
Gotchas
=======
- There cannot be more than one writer at a time, i.e. no more than one write transaction at a time.
- libmdbx is based on B+ tree, so access to database pages is mostly random. Thus SSDs provide a significant performance boost over spinning disks for large databases.
- libmdbx uses shadow paging instead of WAL. Thus syncing data to disk might be a bottleneck for write intensive workload.
- libmdbx uses copy-on-write for snapshot isolation during updates, but read transactions prevents recycling an old retired/freed pages, since it read ones. Thus altering of data during a parallel long-lived read operation will increase the process work set, may exhaust entire free database space, the database can grow quickly, and result in performance degradation. Try to avoid long running read transactions.
- libmdbx is extraordinarily fast and provides minimal overhead for data access, so you should reconsider using brute force techniques and double check your code. On the one hand, in the case of libmdbx, a simple linear search may be more profitable than complex indexes. On the other hand, if you make something suboptimally, you can notice detrimentally only on sufficiently large data.
--
Leonid Yuriev (1):
package/libmdbx: new package (library/database).
DEVELOPERS | 3 + package/Config.in | 1 + .../0001-mdbx-fix-minor-copy-paste-typo.patch | 27 ++++++++ ...e-fix-missing-installation-of-mdbx.h.patch | 54 +++++++++++++++ .../0003-mdbx-fix-obsolete-__noreturn.patch | 27 ++++++++ ...ield-instruction-on-ARM-if-unsupport.patch | 28 ++++++++ ...fix-minor-false-positive-GCC-warning.patch | 27 ++++++++ ...ad-workaround-for-buggy-toolchain-cm.patch | 65 +++++++++++++++++++ ...mdbx-fix-pthread_yield-for-non-GLIBC.patch | 42 ++++++++++++ package/libmdbx/Config.in | 35 ++++++++++ package/libmdbx/libmdbx.hash | 2 + package/libmdbx/libmdbx.mk | 23 +++++++ 12 files changed, 334 insertions(+) create mode 100644 package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch create mode 100644 package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch create mode 100644 package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch create mode 100644 package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch create mode 100644 package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch create mode 100644 package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch create mode 100644 package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mk
Signature · 1 line
2.28.0
PATCH v3 1/1 package/libmdbx: new package (library/database).
In reply to Heiko ThieryThis patch adds libmdbx v0.9.1:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.
https://github.com/erthink/libmdbx
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)
Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch for pthread_yield().
- passed utils/check-package package/libmdbx/* (suggested by Heiko Thiery).
- passed utils/test-pkg -a -p libmdbx (suggested by Heiko Thiery).
--- DEVELOPERS | 3 + package/Config.in | 1 + .../0001-mdbx-fix-minor-copy-paste-typo.patch | 27 ++++++++ ...e-fix-missing-installation-of-mdbx.h.patch | 54 +++++++++++++++ .../0003-mdbx-fix-obsolete-__noreturn.patch | 27 ++++++++ ...ield-instruction-on-ARM-if-unsupport.patch | 28 ++++++++ ...fix-minor-false-positive-GCC-warning.patch | 27 ++++++++ ...ad-workaround-for-buggy-toolchain-cm.patch | 65 +++++++++++++++++++ ...mdbx-fix-pthread_yield-for-non-GLIBC.patch | 42 ++++++++++++ package/libmdbx/Config.in | 35 ++++++++++ package/libmdbx/libmdbx.hash | 2 + package/libmdbx/libmdbx.mk | 23 +++++++ 12 files changed, 334 insertions(+) create mode 100644 package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch create mode 100644 package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch create mode 100644 package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch create mode 100644 package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch create mode 100644 package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch create mode 100644 package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch create mode 100644 package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mk
DEVELOPERS +3 -0
@@ -1545,6 +1545,9 @@ N: Leon Anavi <[email protected]> F: board/olimex/a10_olinuxino F: configs/olimex_a10_olinuxino_lime_defconfig +N: Leonid Yuriev <[email protected]> +F: package/libmdbx + N: Lionel Flandrin <[email protected]> F: package/python-babel/ F: package/python-daemonize/
package/Config.in +1 -0
@@ -1370,6 +1370,7 @@ menu "Database" source "package/kompexsqlite/Config.in" source "package/leveldb/Config.in" source "package/libgit2/Config.in" + source "package/libmdbx/Config.in" source "package/libodb/Config.in" source "package/libodb-boost/Config.in" source "package/libodb-mysql/Config.in"
package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 5807e2eda046bff35946972289236bbef5c0c597 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Thu, 1 Oct 2020 02:27:29 +0300 +Subject: [PATCH] mdbx++: fix minor copy&paste typo. + +Change-Id: I0af3e7ffbbd1231069a60f9f48880df3df2141d7 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.h++ | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx.h++ b/mdbx.h++ +index 095ff9abb..2050f4b92 100644 +--- a/mdbx.h++ ++++ b/mdbx.h++ +@@ -538,7 +538,7 @@ struct LIBMDBX_API_TYPE slice : public ::MDBX_val { + byte *from_base58(byte *dest, size_t dest_size, + bool ignore_spaces = false) const; + +- /// vReturns the buffer size in bytes needed for conversion ++ /// \brief Returns the buffer size in bytes needed for conversion + /// [Base58](https://en.wikipedia.org/wiki/Base58) dump to data. + MDBX_CXX11_CONSTEXPR size_t from_base58_bytes() const noexcept { + return length() / 11 * 8 + length() % 11 * 32 / 43; +-- +2.28.0 +
package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch new file +54 -0
54 lines not on this page · raw message
package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 280ed17ea2dff464c9688fcc92079f36d8b74621 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Sat, 3 Oct 2020 11:35:26 +0300 +Subject: [PATCH] mdbx: fix obsolete `__noreturn`. + +Change-Id: Ic78843d6f16de2a409c16ceecc7acb2ed8aa3e68 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index 04ea5140d..9212640fe 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -141,7 +141,7 @@ __extern_C void __assert(const char *, const char *, unsigned int, const char *) + #else + __nothrow + #endif /* __THROW */ +- __noreturn; ++ MDBX_NORETURN; + #define __assert_fail(assertion, file, line, function) \ + __assert(assertion, file, line, function) + +-- +2.28.0 +
package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch new file +28 -0
@@ -0,0 +1,28 @@ +From 70b615e8d4d10cda2d961a815dd15eb87a1f3925 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Sun, 4 Oct 2020 14:54:11 +0300 +Subject: [PATCH] mdbx: don't use `yield` instruction on ARM if unsupported. + +Change-Id: I0b01d783fe4336b089f4b051fb61c203b5879aa5 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index aba445424..a9ba1ee85 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -763,7 +763,8 @@ static __always_inline void atomic_yield(void) { + #else + __asm__ __volatile__("hint @pause"); + #endif +-#elif defined(__arm__) || defined(__aarch64__) ++#elif defined(__aarch64__) || (defined(__ARM_ARCH) && __ARM_ARCH > 6) || \ ++ defined(__ARM_ARCH_6K__) + #ifdef __CC_ARM + __yield(); + #else +-- +2.28.0 +
package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 8d4e7994c0c5c0cf69326139717e324f90bff65b Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Mon, 5 Oct 2020 15:09:10 +0300 +Subject: [PATCH] mdbx-load: fix minor false-positive GCC warning. + +Change-Id: Ie75c793712d050e8d3da76a4d0a8df9b81dc5275 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx_load.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx_load.c b/mdbx_load.c +index e5e72fc8f..54e179553 100644 +--- a/mdbx_load.c ++++ b/mdbx_load.c +@@ -318,7 +318,7 @@ static int readhdr(void) { + } + + for (int i = 0; dbflags[i].bit; i++) { +- bool value; ++ bool value = false; + if (valbool(dbuf.iov_base, dbflags[i].name, &value)) { + if (value) + dbi_flags |= dbflags[i].bit; +-- +2.28.0 +
package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch new file +65 -0
65 lines not on this page · raw message
package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch new file +42 -0
42 lines not on this page · raw message
package/libmdbx/Config.in new file +35 -0
@@ -0,0 +1,35 @@ +config BR2_PACKAGE_LIBMDBX + bool "libmdbx" + depends on BR2_USE_MMU + depends on BR2_TOOLCHAIN_HAS_THREADS + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + depends on BR2_TOOLCHAIN_HAS_SYNC_4 + help + One of the fastest compact key-value ACID database + without WAL. libmdbx has a specific set of properties + and capabilities, focused on creating unique lightweight + solutions. + + libmdbx surpasses the legendary legendary LMDB in terms of + reliability, features and performance. + + https://github.com/erthink/libmdbx + +if BR2_PACKAGE_LIBMDBX + +config BR2_PACKAGE_LIBMDBX_TOOLS + bool "install tools" + help + Install libmdbx tools for checking, dump, restore + and show statistics of databases. + +config BR2_PACKAGE_LIBMDBX_CXX + bool "C++ API" + select BR2_INSTALL_LIBSTDCPP + help + Enable modern C++ API for libmdbx. + +endif + +comment "libmdbx needs a toolchain w/ threads" + depends on !BR2_TOOLCHAIN_HAS_THREADS
package/libmdbx/libmdbx.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE +sha256 c7fb24381eb4d92f2e2edc17e577cb721269683c816c6cca307c58f6f346e786 libmdbx-amalgamated-0.9.1.tar.gz
package/libmdbx/libmdbx.mk new file +23 -0
@@ -0,0 +1,23 @@ +################################################################################ +# +# libmdbx +# +################################################################################ + +LIBMDBX_VERSION = 0.9.1 +LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz +LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION) +LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO +LIBMDBX_LICENSE = OLDAP-2.8 +LIBMDBX_LICENSE_FILES = LICENSE +LIBMDBX_REDISTRIBUTE = YES +LIBMDBX_STRIP_COMPONENTS = 0 +LIBMDBX_INSTALL_STAGING = YES + +LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES=OFF -DBUILD_FOR_NATIVE_CPU=OFF \ + -DMDBX_INSTALL_STATIC=$(if $(BR2_STATIC_LIBS),ON,OFF) \ + -DMDBX_BUILD_SHARED_LIBRARY=$(if $(BR2_SHARED_LIBS),ON,OFF) \ + -DMDBX_BUILD_CXX=$(if $(BR2_PACKAGE_LIBMDBX_CXX),ON,OFF) \ + -DMDBX_LINK_TOOLS_NONSTATIC=$(if $(BR2_SHARED_LIBS),ON,OFF) + +$(eval $(cmake-package))
Signature · 1 line
2.28.0
PATCH v3 1/1 package/libmdbx: new package (library/database).
Hi Leonid,
thank you for the new version. Some comments below.
Am Mo., 5. Okt. 2020 um 22:16 Uhr schrieb Leonid Yuriev <[email protected]>:
Quoted 38 lines
This patch adds libmdbx v0.9.1:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.https://github.com/erthink/libmdbx
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch forpthread_yield().
- passedutils/check-package package/libmdbx/*(suggested by Heiko Thiery).
- passedutils/test-pkg -a -p libmdbx(suggested by Heiko Thiery).Signed-off-by Leonid Yuriev <[email protected]>--- DEVELOPERS | 3 + package/Config.in | 1 + .../0001-mdbx-fix-minor-copy-paste-typo.patch | 27 ++++++++ ...e-fix-missing-installation-of-mdbx.h.patch | 54 +++++++++++++++ .../0003-mdbx-fix-obsolete-__noreturn.patch | 27 ++++++++ ...ield-instruction-on-ARM-if-unsupport.patch | 28 ++++++++ ...fix-minor-false-positive-GCC-warning.patch | 27 ++++++++ ...ad-workaround-for-buggy-toolchain-cm.patch | 65 +++++++++++++++++++ ...mdbx-fix-pthread_yield-for-non-GLIBC.patch | 42 ++++++++++++ package/libmdbx/Config.in | 35 ++++++++++ package/libmdbx/libmdbx.hash | 2 + package/libmdbx/libmdbx.mk | 23 +++++++
Quoted 8 lines
12 files changed, 334 insertions(+) create mode 100644 package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch create mode 100644 package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch create mode 100644 package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch create mode 100644 package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch create mode 100644 package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch create mode 100644 package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch create mode 100644 package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch
Is it required to add all these patches? They seem all to be in the upstream repo. So you could also set the version to the last commit id that contains all these changes.
Quoted 362 lines
create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mkDEVELOPERS +3
-0@@ -1545,6 +1545,9 @@ N: Leon Anavi <[email protected]> F: board/olimex/a10_olinuxino F: configs/olimex_a10_olinuxino_lime_defconfig +N: Leonid Yuriev <[email protected]> +F: package/libmdbx + N: Lionel Flandrin <[email protected]> F: package/python-babel/ F: package/python-daemonize/package/Config.in +1
-0@@ -1370,6 +1370,7 @@ menu "Database" source "package/kompexsqlite/Config.in" source "package/leveldb/Config.in" source "package/libgit2/Config.in" + source "package/libmdbx/Config.in" source "package/libodb/Config.in" source "package/libodb-boost/Config.in" source "package/libodb-mysql/Config.in"package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch new file +27
-0@@ -0,0 +1,27 @@ +From 5807e2eda046bff35946972289236bbef5c0c597 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Thu, 1 Oct 2020 02:27:29 +0300 +Subject: [PATCH] mdbx++: fix minor copy&paste typo. + +Change-Id: I0af3e7ffbbd1231069a60f9f48880df3df2141d7 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.h++ | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx.h++ b/mdbx.h++ +index 095ff9abb..2050f4b92 100644 +--- a/mdbx.h++ ++++ b/mdbx.h++ +@@ -538,7 +538,7 @@ struct LIBMDBX_API_TYPE slice : public ::MDBX_val { + byte *from_base58(byte *dest, size_t dest_size, + bool ignore_spaces = false) const; + +- /// vReturns the buffer size in bytes needed for conversion ++ /// \brief Returns the buffer size in bytes needed for conversion + /// [Base58](https://en.wikipedia.org/wiki/Base58) dump to data. + MDBX_CXX11_CONSTEXPR size_t from_base58_bytes() const noexcept { + return length() / 11 * 8 + length() % 11 * 32 / 43; +-- +2.28.0 +package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch new file +54
-0@@ -0,0 +1,54 @@ +From dc2cd19d566e84451ee47cf04277f97bc22771c5 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Fri, 2 Oct 2020 00:05:02 +0300 +Subject: [PATCH] mdbx-cmake: fix missing installation of `mdbx.h++` + +Change-Id: I41975e4eeff6583a266b273b9a4f8486982ede90 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + CMakeLists.txt | 8 +++++--- + 1 file changed, 5 insertions(+), 3 deletions(-) + +diff --git a/CMakeLists.txt b/CMakeLists.txt +index 28db425c5..95f116521 100644 +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -485,7 +485,8 @@ fetch_version(MDBX "${CMAKE_CURRENT_SOURCE_DIR}" FALSE) + message(STATUS "libmdbx version is ${MDBX_VERSION}") + + # sources list +-set(LIBMDBX_SOURCES mdbx.h "${CMAKE_CURRENT_BINARY_DIR}/config.h" ) ++set(LIBMDBX_PUBLIC_HEADERS mdbx.h) ++set(LIBMDBX_SOURCES mdbx.h "${CMAKE_CURRENT_BINARY_DIR}/config.h") + if(MDBX_AMALGAMATED_SOURCE) + list(APPEND LIBMDBX_SOURCES mdbx.c) + else() +@@ -511,6 +512,7 @@ else() + endif(MDBX_AMALGAMATED_SOURCE) + if(MDBX_BUILD_CXX) + message(STATUS "Use C${MDBX_C_STANDARD} and C++${MDBX_CXX_STANDARD} for libmdbx") ++ list(APPEND LIBMDBX_PUBLIC_HEADERS mdbx.h++) + list(APPEND LIBMDBX_SOURCES "${MDBX_SOURCE_DIR}/mdbx.c++" mdbx.h++) + else() + message(STATUS "Use C${MDBX_C_STANDARD} for libmdbx but C++ portion is disabled") +@@ -561,7 +563,7 @@ if(MDBX_INSTALL_STATIC) + else() + add_library(mdbx-static STATIC EXCLUDE_FROM_ALL ${LIBMDBX_SOURCES}) + endif() +-set_target_properties(mdbx-static PROPERTIES PUBLIC_HEADER mdbx.h) ++set_target_properties(mdbx-static PROPERTIES PUBLIC_HEADER "${LIBMDBX_PUBLIC_HEADERS}") + target_compile_definitions(mdbx-static PRIVATE MDBX_BUILD_SHARED_LIBRARY=0) + target_setup_options(mdbx-static) + libmdbx_setup_libs(mdbx-static INTERFACE) +@@ -576,7 +578,7 @@ endif() + # build shared library + if(MDBX_BUILD_SHARED_LIBRARY) + add_library(mdbx SHARED ${LIBMDBX_SOURCES}) +- set_target_properties(mdbx PROPERTIES PUBLIC_HEADER mdbx.h) ++ set_target_properties(mdbx PROPERTIES PUBLIC_HEADER "${LIBMDBX_PUBLIC_HEADERS}") + target_compile_definitions(mdbx PRIVATE LIBMDBX_EXPORTS MDBX_BUILD_SHARED_LIBRARY=1 INTERFACE LIBMDBX_IMPORTS) + target_setup_options(mdbx) + libmdbx_setup_libs(mdbx PRIVATE) +-- +2.28.0 +package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch new file +27
-0@@ -0,0 +1,27 @@ +From 280ed17ea2dff464c9688fcc92079f36d8b74621 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Sat, 3 Oct 2020 11:35:26 +0300 +Subject: [PATCH] mdbx: fix obsolete `__noreturn`. + +Change-Id: Ic78843d6f16de2a409c16ceecc7acb2ed8aa3e68 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index 04ea5140d..9212640fe 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -141,7 +141,7 @@ __extern_C void __assert(const char *, const char *, unsigned int, const char *) + #else + __nothrow + #endif /* __THROW */ +- __noreturn; ++ MDBX_NORETURN; + #define __assert_fail(assertion, file, line, function) \ + __assert(assertion, file, line, function) + +-- +2.28.0 +package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch new file +28
-0@@ -0,0 +1,28 @@ +From 70b615e8d4d10cda2d961a815dd15eb87a1f3925 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Sun, 4 Oct 2020 14:54:11 +0300 +Subject: [PATCH] mdbx: don't use `yield` instruction on ARM if unsupported. + +Change-Id: I0b01d783fe4336b089f4b051fb61c203b5879aa5 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index aba445424..a9ba1ee85 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -763,7 +763,8 @@ static __always_inline void atomic_yield(void) { + #else + __asm__ __volatile__("hint @pause"); + #endif +-#elif defined(__arm__) || defined(__aarch64__) ++#elif defined(__aarch64__) || (defined(__ARM_ARCH) && __ARM_ARCH > 6) || \ ++ defined(__ARM_ARCH_6K__) + #ifdef __CC_ARM + __yield(); + #else +-- +2.28.0 +package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch new file +27
-0@@ -0,0 +1,27 @@ +From 8d4e7994c0c5c0cf69326139717e324f90bff65b Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Mon, 5 Oct 2020 15:09:10 +0300 +Subject: [PATCH] mdbx-load: fix minor false-positive GCC warning. + +Change-Id: Ie75c793712d050e8d3da76a4d0a8df9b81dc5275 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx_load.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx_load.c b/mdbx_load.c +index e5e72fc8f..54e179553 100644 +--- a/mdbx_load.c ++++ b/mdbx_load.c +@@ -318,7 +318,7 @@ static int readhdr(void) { + } + + for (int i = 0; dbflags[i].bit; i++) { +- bool value; ++ bool value = false; + if (valbool(dbuf.iov_base, dbflags[i].name, &value)) { + if (value) + dbi_flags |= dbflags[i].bit; +-- +2.28.0 +package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch new file +65
-0@@ -0,0 +1,65 @@ +From 787eaaa373073e17f3a53658b085c255bc2c8ff8 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Mon, 5 Oct 2020 19:12:20 +0300 +Subject: [PATCH] mdbx-cmake: pthread workaround for buggy + toolchain/cmake/buildroot. + +Change-Id: I0d95e783abbd10a63cd1595a9de50593e814a967 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + CMakeLists.txt | 21 ++++++++++++++++++--- + 1 file changed, 18 insertions(+), 3 deletions(-) + +diff --git a/CMakeLists.txt b/CMakeLists.txt +index 95f116521..20a50a453 100644 +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -173,10 +173,27 @@ if(NOT CMAKE_BUILD_TYPE) + endif() + string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPERCASE) + ++set(CMAKE_THREAD_PREFER_PTHREAD TRUE) ++set(THREADS_PREFER_PTHREAD_FLAG TRUE) ++find_package(Threads REQUIRED) ++ + include(cmake/utils.cmake) + include(cmake/compiler.cmake) + include(cmake/profile.cmake) + ++# Workaround for `-pthread` toolchain/cmake bug ++if(NOT APPLE AND NOT MSVC ++ AND CMAKE_USE_PTHREADS_INIT AND NOT CMAKE_THREAD_LIBS_INIT ++ AND (CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_CLANG)) ++ check_compiler_flag("-pthread" CC_HAS_PTHREAD) ++ if(CC_HAS_PTHREAD AND NOT CMAKE_EXE_LINKER_FLAGS MATCHES "-pthread") ++ message(STATUS "Force add -pthread for linker flags to avoid troubles") ++ set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -pthread") ++ set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -pthread") ++ set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} -pthread") ++ endif() ++endif() ++ + CHECK_FUNCTION_EXISTS(pow NOT_NEED_LIBM) + if(NOT_NEED_LIBM) + set(LIB_MATH "") +@@ -190,8 +207,6 @@ else() + endif() + endif() + +-find_package(Threads REQUIRED) +- + if(SUBPROJECT) + if(NOT DEFINED BUILD_SHARED_LIBS) + option(BUILD_SHARED_LIBS "Build shared libraries (DLLs)" OFF) +@@ -541,7 +556,7 @@ macro(target_setup_options TARGET) + endmacro() + + macro(libmdbx_setup_libs TARGET MODE) +- target_link_libraries(${TARGET} ${MODE} ${CMAKE_THREAD_LIBS_INIT}) ++ target_link_libraries(${TARGET} ${MODE} Threads::Threads) + if(${CMAKE_SYSTEM_NAME} STREQUAL "Windows") + target_link_libraries(${TARGET} ${MODE} ntdll.lib) + if(MDBX_NTDLL_EXTRA_IMPLIB AND MDBX_AVOID_CRT) +-- +2.28.0 +package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch new file +42
-0@@ -0,0 +1,42 @@ +From 16d43332a9d28ba27aa3d127e8f6b77a87b2f5eb Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Mon, 5 Oct 2020 21:08:09 +0300 +Subject: [PATCH] mdbx: fix `pthread_yield()` for non-GLIBC. + +Change-Id: I080e37a42b62e524896dea8747e9f23e2fcd584f +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 4 +++- + mdbx.c | 1 + + 2 files changed, 4 insertions(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index a9ba1ee85..ace6f2756 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -777,7 +777,9 @@ static __always_inline void atomic_yield(void) { + defined(__mips64__) || defined(_M_MRX000) || defined(_MIPS_) || \ + defined(__MWERKS__) || defined(__sgi) + __asm__ __volatile__(".word 0x00000140"); +-#else ++#elif defined(__linux__) || defined(__gnu_linux__) || defined(_UNIX03_SOURCE) ++ sched_yield(); ++#elif (defined(_GNU_SOURCE) && __GLIBC_PREREQ(2, 1)) || defined(_OPEN_THREADS) + pthread_yield(); + #endif + } +diff --git a/mdbx.c b/mdbx.c +index 3509942dc..3af23ce0b 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -119,6 +119,7 @@ + + #if defined(__linux__) || defined(__gnu_linux__) + #include <linux/sysctl.h> ++#include <sched.h> + #include <sys/sendfile.h> + #include <sys/statfs.h> + #endif /* Linux */ +-- +2.28.0 +package/libmdbx/Config.in new file +14
-0@@ -0,0 +1,35 @@ +config BR2_PACKAGE_LIBMDBX + bool "libmdbx" + depends on BR2_USE_MMU + depends on BR2_TOOLCHAIN_HAS_THREADS + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + depends on BR2_TOOLCHAIN_HAS_SYNC_4 + help + One of the fastest compact key-value ACID database + without WAL. libmdbx has a specific set of properties + and capabilities, focused on creating unique lightweight + solutions. + + libmdbx surpasses the legendary legendary LMDB in terms of + reliability, features and performance.
package/libmdbx/Config.in line 24
Quoted 10 lines
package/libmdbx/Config.in new file +10
-0+ + https://github.com/erthink/libmdbx + +if BR2_PACKAGE_LIBMDBX + +config BR2_PACKAGE_LIBMDBX_TOOLS + bool "install tools" + help + Install libmdbx tools for checking, dump, restore + and show statistics of databases.
This config option is nowhere used? Do you miss to add it in the makefile or is it still a leftover?
package/libmdbx/Config.in line 31
Quoted 7 lines
package/libmdbx/Config.in new file +7
-0+ +config BR2_PACKAGIBMDBX_CXX + bool "C++ API" + select BR2_INSTALL_LIBSTDCPP + help + Enable modern C++ API for libmdbx. +
When enabling CXX support you could add a comment that a toolchain with cpp support is required and not enable it by selecting this option.
config BR2_PACKAGIBMDBX_CXX
bool "C++ API"
help
Enable modern C++ API for libmdbx.
comment "libmdbx c++ support needs a toolchain w/ C++"
depends on !BR2_INSTALL_LIBSTDCPP
See also: https://git.buildroot.net/buildroot/tree/package/rrdtool/Config.in#n31
package/libmdbx/Config.in line 35
Quoted 4 lines
package/libmdbx/Config.in new file +4
-0+endif + +comment "libmdbx needs a toolchain w/ threads" + depends on !BR2_TOOLCHAIN_HAS_THREADS
Here you the comment should look something like:
comment "libmdbx needs a toolchain w/ threads, gcc >= 4.8"
depends on BR2_USE_MMU
depends on !BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 || !BR2_TOOLCHAIN_HAS_THREADS
Quoted 27 lines
package/libmdbx/libmdbx.hash new file +2
-0@@ -0,0 +1,2 @@ +sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE +sha256 c7fb24381eb4d92f2e2edc17e577cb721269683c816c6cca307c58f6f346e786 libmdbx-amalgamated-0.9.1.tar.gzpackage/libmdbx/libmdbx.mk new file +13
-0@@ -0,0 +1,23 @@ +################################################################################ +# +# libmdbx +# +################################################################################ + +LIBMDBX_VERSION = 0.9.1 +LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz +LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION) +LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO +LIBMDBX_LICENSE = OLDAP-2.8 +LIBMDBX_LICENSE_FILES = LICENSE +LIBMDBX_REDISTRIBUTE = YES
package/libmdbx/libmdbx.mk line 23
Quoted 10 lines
package/libmdbx/libmdbx.mk new file +10
-0+LIBMDBX_STRIP_COMPONENTS = 0 +LIBMDBX_INSTALL_STAGING = YES + +LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES=OFF -DBUILD_FOR_NATIVE_CPU=OFF \ + -DMDBX_INSTALL_STATIC=$(if $(BR2_STATIC_LIBS),ON,OFF) \ + -DMDBX_BUILD_SHARED_LIBRARY=$(if $(BR2_SHARED_LIBS),ON,OFF) \ + -DMDBX_BUILD_CXX=$(if $(BR2_PACKAGE_LIBMDBX_CXX),ON,OFF) \ + -DMDBX_LINK_TOOLS_NONSTATIC=$(if $(BR2_SHARED_LIBS),ON,OFF) + +$(eval $(cmake-package))
Thank you
Signature · 1 line
Heiko
PATCH v3 1/1 package/libmdbx: new package (library/database).
Hi, Heiko.
Thanks for comments.
I will send an updated (hope final) version of the patch soon. And here are the answers to some of your questions/comments below.
Quoted 4 lines · Heiko Thiery
Hi Leonid,
thank you for the new version. Some comments below.
[...]
Quoted 11 lines
Quoted 7 lines
create mode 100644 package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch create mode 100644 package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch create mode 100644 package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch create mode 100644 package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch create mode 100644 package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch create mode 100644 package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch create mode 100644 package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patchIs it required to add all these patches? They seem all to be in the upstream repo. So you could also set the version to the last commit id that contains all these changes.
Yes, these patches are required for amalgamated source code which is available only for a releases (i.e. github's tags).
Quoted 5 lines
package/libmdbx/Config.in line 21
Quoted 2 lines
package/libmdbx/Config.in new file +2
-0+config BR2_PACKAGE_LIBMDBX_TOOLS + bool "install tools"This config option is nowhere used? Do you miss to add it in the makefile or is it still a leftover?
This is my mistake.
Now fixed in the makefile.
Quoted 7 lines
package/libmdbx/Config.in line 28
Quoted 3 lines
package/libmdbx/Config.in new file +3
-0+config BR2_PACKAGE_LIBMDBX_CXX + bool "C++ API" + select BR2_INSTALL_LIBSTDCPPWhen enabling CXX support you could add a comment that a toolchain with cpp support is required and not enable it by selecting this option.
[..]
Quoted 2 lines
comment "libmdbx c++ support needs a toolchain w/ C++"
depends on !BR2_INSTALL_LIBSTDCPP
[..]
Quoted 7 lines
package/libmdbx/Config.in line 35
Quoted 2 lines
package/libmdbx/Config.in new file +2
-0+comment "libmdbx needs a toolchain w/ threads" + depends on !BR2_TOOLCHAIN_HAS_THREADSHere you the comment should look something like:
comment "libmdbx needs a toolchain w/ threads, gcc >= 4.8"
depends onBR2_USE_MMU
depends on !BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 || !BR2_TOOLCHAIN_HAS_THREADS
Done.
I also clarified these dependencies.
Regards,
Leonid.
v4 cover package/libmdbx: new package (library/database)
This patch adds libmdbx v0.9.1 and below is a brief overview of libmdbx.
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)
Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch for pthread_yield().
- passed utils/check-package package/libmdbx/* (suggested by Heiko Thiery).
- passed utils/test-pkg -a -p libmdbx (suggested by Heiko Thiery).
Changes v3 -> v4:
- fix passing BR2_PACKAGE_LIBMDBX_TOOLS option to cmake.
- fix using depend on instead of select,
and add comment for C++ toolchain (suggested by Heiko Thiery).
- fix minor help typo.
Please merge.
Regards,
Leonid.
--
libmdbx is an extremely fast, compact, powerful, embedded, transactional key-value database, with permissive license. libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
Historically, libmdbx (MDBX) is a deeply revised and extended descendant of the legendary LMDB (Lightning Memory-Mapped Database). libmdbx inherits all benefits from LMDB, but resolves some issues and adds a set of improvements.
According to developers, for now libmdbx surpasses the LMDB in terms of reliability, features and performance.
The most important differences MDBX from LMDB:
==============================================
1. More attention is paid to the quality of the code, to an "unbreakability" of the API, to testing and automatic checks (i.e. sanitizers, etc). So there:
- more control during operation;
- more checking parameters, internal audit of database structures;
- no warnings from compiler;
- no issues from ASAN, UBSAN, Valgrind, Coverity;
- etc.
2. Keys could be more than 2 times longer than LMDB.
3. Up to 20% faster than LMDB in CRUD benchmarks.
4. Automatic on-the-fly database size adjustment,
both increment and reduction.
5. Automatic continuous zero-overhead database compactification.
6. The same database format for 32- and 64-bit builds.
7. LIFO policy for Garbage Collection recycling (this can significantly increase write performance due write-back disk cache up to several times in a best case scenario).
8. Range query estimation.
9. Utility for checking the integrity of the database structure with some recovery capabilities.
For more info please refer:
- https://github.com/erthink/libmdbx for source code and README.
- https://erthink.github.io/libmdbx for API description.
--
MDBX is a Btree-based database management library modeled loosely on the BerkeleyDB API, but much simplified. The entire database (aka "environment") is exposed in a memory map, and all data fetches return data directly from the mapped memory, so no malloc's or memcpy's occur during data fetches. As such, the library is extremely simple because it requires no page caching layer of its own, and it is extremely high performance and memory-efficient. It is also fully transactional with full ACID semantics, and when the memory map is read-only, the database integrity cannot be corrupted by stray pointer writes from application code.
The library is fully thread-aware and supports concurrent read/write access from multiple processes and threads. Data pages use a copy-on-write strategy so no active data pages are ever overwritten, which also provides resistance to corruption and eliminates the need of any special recovery procedures after a system crash. Writes are fully serialized; only one write transaction may be active at a time, which guarantees that writers can never deadlock. The database structure is multi-versioned so readers run with no locks; writers cannot block readers, and readers don't block writers.
Unlike other well-known database mechanisms which use either write-ahead transaction logs or append-only data writes, MDBX requires no maintenance during operation. Both write-ahead loggers and append-only databases require periodic checkpointing and/or compaction of their log or database files otherwise they grow without bound. MDBX tracks retired/freed pages within the database and re-uses them for new write operations, so the database size does not grow without bound in normal use.
The memory map can be used as a read-only or read-write map. It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database.
Features
========
- Key-value data model, keys are always sorted.
- Fully ACID-compliant, through to MVCC and CoW.
- Multiple key-value sub-databases within a single datafile.
- Range lookups, including range query estimation.
- Efficient support for short fixed length keys, including native 32/64-bit integers.
- Ultra-efficient support for multimaps. Multi-values sorted, searchable and iterable. Keys stored without duplication.
- Data is memory-mapped and accessible directly/zero-copy. Traversal of database records is extremely-fast.
- Transactions for readers and writers, ones do not block others.
- Writes are strongly serialized. No transaction conflicts nor deadlocks.
- Readers are non-blocking, notwithstanding snapshot isolation.
- Nested write transactions.
- Reads scale linearly across CPUs.
- Continuous zero-overhead database compactification.
- Automatic on-the-fly database size adjustment.
- Customizable database page size.
- Olog(N) cost of lookup, insert, update, and delete operations by virtue of B+ tree characteristics.
- Online hot backup.
- Append operation for efficient bulk insertion of pre-sorted data.
- No WAL nor any transaction journal. No crash recovery needed. No maintenance is required.
- No internal cache and/or memory management, all done by basic OS services.
Limitations
===========
- Page size: a power of 2, maximum 65536 bytes, default 4096 bytes.
- Key size: minimum 0, maximum ?? pagesize (1300 bytes for default 4K pagesize, 21780 bytes for 64K pagesize).
- Value size: minimum 0, maximum 2146435072 (0x7FF00000) bytes for maps, ?? pagesize for multimaps (1348 bytes default 4K pagesize, 21828 bytes for 64K pagesize).
- Write transaction size: up to 4194301 (0x3FFFFD) pages (16 GiB for default 4K pagesize, 256 GiB for 64K pagesize).
- Database size: up to 2147483648 pages (8 TiB for default 4K pagesize, 128 TiB for 64K pagesize).
- Maximum sub-databases: 32765.
Gotchas
=======
- There cannot be more than one writer at a time, i.e. no more than one write transaction at a time.
- libmdbx is based on B+ tree, so access to database pages is mostly random. Thus SSDs provide a significant performance boost over spinning disks for large databases.
- libmdbx uses shadow paging instead of WAL. Thus syncing data to disk might be a bottleneck for write intensive workload.
- libmdbx uses copy-on-write for snapshot isolation during updates, but read transactions prevents recycling an old retired/freed pages, since it read ones. Thus altering of data during a parallel long-lived read operation will increase the process work set, may exhaust entire free database space, the database can grow quickly, and result in performance degradation. Try to avoid long running read transactions.
- libmdbx is extraordinarily fast and provides minimal overhead for data access, so you should reconsider using brute force techniques and double check your code. On the one hand, in the case of libmdbx, a simple linear search may be more profitable than complex indexes. On the other hand, if you make something suboptimally, you can notice detrimentally only on sufficiently large data.
--
Leonid Yuriev (1):
package/libmdbx: new package (library/database).
DEVELOPERS | 3 + package/Config.in | 1 + .../0001-mdbx-fix-minor-copy-paste-typo.patch | 27 ++++++++ ...e-fix-missing-installation-of-mdbx.h.patch | 54 +++++++++++++++ .../0003-mdbx-fix-obsolete-__noreturn.patch | 27 ++++++++ ...ield-instruction-on-ARM-if-unsupport.patch | 28 ++++++++ ...fix-minor-false-positive-GCC-warning.patch | 27 ++++++++ ...ad-workaround-for-buggy-toolchain-cm.patch | 65 +++++++++++++++++++ ...mdbx-fix-pthread_yield-for-non-GLIBC.patch | 42 ++++++++++++ package/libmdbx/Config.in | 42 ++++++++++++ package/libmdbx/libmdbx.hash | 2 + package/libmdbx/libmdbx.mk | 24 +++++++ 12 files changed, 342 insertions(+) create mode 100644 package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch create mode 100644 package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch create mode 100644 package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch create mode 100644 package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch create mode 100644 package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch create mode 100644 package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch create mode 100644 package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mk
Signature · 1 line
2.28.0
PATCH v4 1/1 package/libmdbx: new package (library/database).
In reply to Леонид ЮрьевThis patch adds libmdbx v0.9.1:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.
https://github.com/erthink/libmdbx
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)
Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch for pthread_yield().
- passed utils/check-package package/libmdbx/* (suggested by Heiko Thiery)
- passed utils/test-pkg -a -p libmdbx,
except w/o threads & w/o MMU (suggested by Heiko Thiery).
Changes v3 -> v4:
- fix passing BR2_PACKAGE_LIBMDBX_TOOLS option to cmake.
- fix using depend on instead of select,
and add comment for C++ toolchain (suggested by Heiko Thiery).
- fix minor help typo.
--- DEVELOPERS | 3 + package/Config.in | 1 + .../0001-mdbx-fix-minor-copy-paste-typo.patch | 27 ++++++++ ...e-fix-missing-installation-of-mdbx.h.patch | 54 +++++++++++++++ .../0003-mdbx-fix-obsolete-__noreturn.patch | 27 ++++++++ ...ield-instruction-on-ARM-if-unsupport.patch | 28 ++++++++ ...fix-minor-false-positive-GCC-warning.patch | 27 ++++++++ ...ad-workaround-for-buggy-toolchain-cm.patch | 65 +++++++++++++++++++ ...mdbx-fix-pthread_yield-for-non-GLIBC.patch | 42 ++++++++++++ package/libmdbx/Config.in | 42 ++++++++++++ package/libmdbx/libmdbx.hash | 2 + package/libmdbx/libmdbx.mk | 24 +++++++ 12 files changed, 342 insertions(+) create mode 100644 package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch create mode 100644 package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch create mode 100644 package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch create mode 100644 package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch create mode 100644 package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch create mode 100644 package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch create mode 100644 package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mk
DEVELOPERS +3 -0
@@ -1551,6 +1551,9 @@ N: Leon Anavi <[email protected]> F: board/olimex/a10_olinuxino F: configs/olimex_a10_olinuxino_lime_defconfig +N: Leonid Yuriev <[email protected]> +F: package/libmdbx + N: Lionel Flandrin <[email protected]> F: package/python-babel/ F: package/python-daemonize/
package/Config.in +1 -0
@@ -1370,6 +1370,7 @@ menu "Database" source "package/kompexsqlite/Config.in" source "package/leveldb/Config.in" source "package/libgit2/Config.in" + source "package/libmdbx/Config.in" source "package/libodb/Config.in" source "package/libodb-boost/Config.in" source "package/libodb-mysql/Config.in"
package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 5807e2eda046bff35946972289236bbef5c0c597 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Thu, 1 Oct 2020 02:27:29 +0300 +Subject: [PATCH] mdbx++: fix minor copy&paste typo. + +Change-Id: I0af3e7ffbbd1231069a60f9f48880df3df2141d7 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.h++ | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx.h++ b/mdbx.h++ +index 095ff9abb..2050f4b92 100644 +--- a/mdbx.h++ ++++ b/mdbx.h++ +@@ -538,7 +538,7 @@ struct LIBMDBX_API_TYPE slice : public ::MDBX_val { + byte *from_base58(byte *dest, size_t dest_size, + bool ignore_spaces = false) const; + +- /// vReturns the buffer size in bytes needed for conversion ++ /// \brief Returns the buffer size in bytes needed for conversion + /// [Base58](https://en.wikipedia.org/wiki/Base58) dump to data. + MDBX_CXX11_CONSTEXPR size_t from_base58_bytes() const noexcept { + return length() / 11 * 8 + length() % 11 * 32 / 43; +-- +2.28.0 +
package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch new file +54 -0
54 lines not on this page · raw message
package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 280ed17ea2dff464c9688fcc92079f36d8b74621 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Sat, 3 Oct 2020 11:35:26 +0300 +Subject: [PATCH] mdbx: fix obsolete `__noreturn`. + +Change-Id: Ic78843d6f16de2a409c16ceecc7acb2ed8aa3e68 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index 04ea5140d..9212640fe 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -141,7 +141,7 @@ __extern_C void __assert(const char *, const char *, unsigned int, const char *) + #else + __nothrow + #endif /* __THROW */ +- __noreturn; ++ MDBX_NORETURN; + #define __assert_fail(assertion, file, line, function) \ + __assert(assertion, file, line, function) + +-- +2.28.0 +
package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch new file +28 -0
@@ -0,0 +1,28 @@ +From 70b615e8d4d10cda2d961a815dd15eb87a1f3925 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Sun, 4 Oct 2020 14:54:11 +0300 +Subject: [PATCH] mdbx: don't use `yield` instruction on ARM if unsupported. + +Change-Id: I0b01d783fe4336b089f4b051fb61c203b5879aa5 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index aba445424..a9ba1ee85 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -763,7 +763,8 @@ static __always_inline void atomic_yield(void) { + #else + __asm__ __volatile__("hint @pause"); + #endif +-#elif defined(__arm__) || defined(__aarch64__) ++#elif defined(__aarch64__) || (defined(__ARM_ARCH) && __ARM_ARCH > 6) || \ ++ defined(__ARM_ARCH_6K__) + #ifdef __CC_ARM + __yield(); + #else +-- +2.28.0 +
package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch new file +27 -0
@@ -0,0 +1,27 @@ +From 8d4e7994c0c5c0cf69326139717e324f90bff65b Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Mon, 5 Oct 2020 15:09:10 +0300 +Subject: [PATCH] mdbx-load: fix minor false-positive GCC warning. + +Change-Id: Ie75c793712d050e8d3da76a4d0a8df9b81dc5275 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx_load.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx_load.c b/mdbx_load.c +index e5e72fc8f..54e179553 100644 +--- a/mdbx_load.c ++++ b/mdbx_load.c +@@ -318,7 +318,7 @@ static int readhdr(void) { + } + + for (int i = 0; dbflags[i].bit; i++) { +- bool value; ++ bool value = false; + if (valbool(dbuf.iov_base, dbflags[i].name, &value)) { + if (value) + dbi_flags |= dbflags[i].bit; +-- +2.28.0 +
package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch new file +65 -0
65 lines not on this page · raw message
package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch new file +42 -0
42 lines not on this page · raw message
package/libmdbx/Config.in new file +42 -0
42 lines not on this page · raw message
package/libmdbx/libmdbx.hash new file +2 -0
@@ -0,0 +1,2 @@ +sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE +sha256 c7fb24381eb4d92f2e2edc17e577cb721269683c816c6cca307c58f6f346e786 libmdbx-amalgamated-0.9.1.tar.gz
package/libmdbx/libmdbx.mk new file +24 -0
@@ -0,0 +1,24 @@ +################################################################################ +# +# libmdbx +# +################################################################################ + +LIBMDBX_VERSION = 0.9.1 +LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz +LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION) +LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO +LIBMDBX_LICENSE = OLDAP-2.8 +LIBMDBX_LICENSE_FILES = LICENSE +LIBMDBX_REDISTRIBUTE = YES +LIBMDBX_STRIP_COMPONENTS = 0 +LIBMDBX_INSTALL_STAGING = YES + +LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES=OFF -DBUILD_FOR_NATIVE_CPU=OFF \ + -DMDBX_INSTALL_STATIC=$(if $(BR2_STATIC_LIBS),ON,OFF) \ + -DMDBX_BUILD_SHARED_LIBRARY=$(if $(BR2_SHARED_LIBS),ON,OFF) \ + -DMDBX_BUILD_CXX=$(if $(BR2_PACKAGE_LIBMDBX_CXX),ON,OFF) \ + -DMDBX_BUILD_TOOLS=$(if $(BR2_PACKAGE_LIBMDBX_TOOLS),ON,OFF) \ + -DMDBX_LINK_TOOLS_NONSTATIC=$(if $(BR2_SHARED_LIBS),ON,OFF) + +$(eval $(cmake-package))
Signature · 1 line
2.28.0
PATCH v4 1/1 package/libmdbx: new package (library/database).
Hi,
Am Di., 6. Okt. 2020 um 21:27 Uhr schrieb Leonid Yuriev <[email protected]>:
Quoted 25 lines
This patch adds libmdbx v0.9.1:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.https://github.com/erthink/libmdbx
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch forpthread_yield().
- passedutils/check-package package/libmdbx/*(suggested by Heiko Thiery)
- passedutils/test-pkg -a -p libmdbx,
except w/o threads & w/o MMU (suggested by Heiko Thiery).
Quoted 7 lines
Changes v3 -> v4:
- fix passingBR2_PACKAGE_LIBMDBX_TOOLSoption to cmake.
- fix usingdepend oninstead ofselect,
and addcommentfor C++ toolchain (suggested by Heiko Thiery).
- fix minor help typo.Signed-off-by Leonid Yuriev <[email protected]>
Quoted 456 lines
--- DEVELOPERS | 3 + package/Config.in | 1 + .../0001-mdbx-fix-minor-copy-paste-typo.patch | 27 ++++++++ ...e-fix-missing-installation-of-mdbx.h.patch | 54 +++++++++++++++ .../0003-mdbx-fix-obsolete-__noreturn.patch | 27 ++++++++ ...ield-instruction-on-ARM-if-unsupport.patch | 28 ++++++++ ...fix-minor-false-positive-GCC-warning.patch | 27 ++++++++ ...ad-workaround-for-buggy-toolchain-cm.patch | 65 +++++++++++++++++++ ...mdbx-fix-pthread_yield-for-non-GLIBC.patch | 42 ++++++++++++ package/libmdbx/Config.in | 42 ++++++++++++ package/libmdbx/libmdbx.hash | 2 + package/libmdbx/libmdbx.mk | 24 +++++++ 12 files changed, 342 insertions(+) create mode 100644 package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch create mode 100644 package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch create mode 100644 package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch create mode 100644 package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch create mode 100644 package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch create mode 100644 package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch create mode 100644 package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mkDEVELOPERS +3
-0@@ -1551,6 +1551,9 @@ N: Leon Anavi <[email protected]> F: board/olimex/a10_olinuxino F: configs/olimex_a10_olinuxino_lime_defconfig +N: Leonid Yuriev <[email protected]> +F: package/libmdbx + N: Lionel Flandrin <[email protected]> F: package/python-babel/ F: package/python-daemonize/package/Config.in +1
-0@@ -1370,6 +1370,7 @@ menu "Database" source "package/kompexsqlite/Config.in" source "package/leveldb/Config.in" source "package/libgit2/Config.in" + source "package/libmdbx/Config.in" source "package/libodb/Config.in" source "package/libodb-boost/Config.in" source "package/libodb-mysql/Config.in"package/libmdbx/0001-mdbx-fix-minor-copy-paste-typo.patch new file +27
-0@@ -0,0 +1,27 @@ +From 5807e2eda046bff35946972289236bbef5c0c597 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Thu, 1 Oct 2020 02:27:29 +0300 +Subject: [PATCH] mdbx++: fix minor copy&paste typo. + +Change-Id: I0af3e7ffbbd1231069a60f9f48880df3df2141d7 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.h++ | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx.h++ b/mdbx.h++ +index 095ff9abb..2050f4b92 100644 +--- a/mdbx.h++ ++++ b/mdbx.h++ +@@ -538,7 +538,7 @@ struct LIBMDBX_API_TYPE slice : public ::MDBX_val { + byte *from_base58(byte *dest, size_t dest_size, + bool ignore_spaces = false) const; + +- /// vReturns the buffer size in bytes needed for conversion ++ /// \brief Returns the buffer size in bytes needed for conversion + /// [Base58](https://en.wikipedia.org/wiki/Base58) dump to data. + MDBX_CXX11_CONSTEXPR size_t from_base58_bytes() const noexcept { + return length() / 11 * 8 + length() % 11 * 32 / 43; +-- +2.28.0 +package/libmdbx/0002-mdbx-cmake-fix-missing-installation-of-mdbx.h.patch new file +54
-0@@ -0,0 +1,54 @@ +From dc2cd19d566e84451ee47cf04277f97bc22771c5 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Fri, 2 Oct 2020 00:05:02 +0300 +Subject: [PATCH] mdbx-cmake: fix missing installation of `mdbx.h++` + +Change-Id: I41975e4eeff6583a266b273b9a4f8486982ede90 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + CMakeLists.txt | 8 +++++--- + 1 file changed, 5 insertions(+), 3 deletions(-) + +diff --git a/CMakeLists.txt b/CMakeLists.txt +index 28db425c5..95f116521 100644 +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -485,7 +485,8 @@ fetch_version(MDBX "${CMAKE_CURRENT_SOURCE_DIR}" FALSE) + message(STATUS "libmdbx version is ${MDBX_VERSION}") + + # sources list +-set(LIBMDBX_SOURCES mdbx.h "${CMAKE_CURRENT_BINARY_DIR}/config.h" ) ++set(LIBMDBX_PUBLIC_HEADERS mdbx.h) ++set(LIBMDBX_SOURCES mdbx.h "${CMAKE_CURRENT_BINARY_DIR}/config.h") + if(MDBX_AMALGAMATED_SOURCE) + list(APPEND LIBMDBX_SOURCES mdbx.c) + else() +@@ -511,6 +512,7 @@ else() + endif(MDBX_AMALGAMATED_SOURCE) + if(MDBX_BUILD_CXX) + message(STATUS "Use C${MDBX_C_STANDARD} and C++${MDBX_CXX_STANDARD} for libmdbx") ++ list(APPEND LIBMDBX_PUBLIC_HEADERS mdbx.h++) + list(APPEND LIBMDBX_SOURCES "${MDBX_SOURCE_DIR}/mdbx.c++" mdbx.h++) + else() + message(STATUS "Use C${MDBX_C_STANDARD} for libmdbx but C++ portion is disabled") +@@ -561,7 +563,7 @@ if(MDBX_INSTALL_STATIC) + else() + add_library(mdbx-static STATIC EXCLUDE_FROM_ALL ${LIBMDBX_SOURCES}) + endif() +-set_target_properties(mdbx-static PROPERTIES PUBLIC_HEADER mdbx.h) ++set_target_properties(mdbx-static PROPERTIES PUBLIC_HEADER "${LIBMDBX_PUBLIC_HEADERS}") + target_compile_definitions(mdbx-static PRIVATE MDBX_BUILD_SHARED_LIBRARY=0) + target_setup_options(mdbx-static) + libmdbx_setup_libs(mdbx-static INTERFACE) +@@ -576,7 +578,7 @@ endif() + # build shared library + if(MDBX_BUILD_SHARED_LIBRARY) + add_library(mdbx SHARED ${LIBMDBX_SOURCES}) +- set_target_properties(mdbx PROPERTIES PUBLIC_HEADER mdbx.h) ++ set_target_properties(mdbx PROPERTIES PUBLIC_HEADER "${LIBMDBX_PUBLIC_HEADERS}") + target_compile_definitions(mdbx PRIVATE LIBMDBX_EXPORTS MDBX_BUILD_SHARED_LIBRARY=1 INTERFACE LIBMDBX_IMPORTS) + target_setup_options(mdbx) + libmdbx_setup_libs(mdbx PRIVATE) +-- +2.28.0 +package/libmdbx/0003-mdbx-fix-obsolete-__noreturn.patch new file +27
-0@@ -0,0 +1,27 @@ +From 280ed17ea2dff464c9688fcc92079f36d8b74621 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Sat, 3 Oct 2020 11:35:26 +0300 +Subject: [PATCH] mdbx: fix obsolete `__noreturn`. + +Change-Id: Ic78843d6f16de2a409c16ceecc7acb2ed8aa3e68 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index 04ea5140d..9212640fe 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -141,7 +141,7 @@ __extern_C void __assert(const char *, const char *, unsigned int, const char *) + #else + __nothrow + #endif /* __THROW */ +- __noreturn; ++ MDBX_NORETURN; + #define __assert_fail(assertion, file, line, function) \ + __assert(assertion, file, line, function) + +-- +2.28.0 +package/libmdbx/0004-mdbx-don-t-use-yield-instruction-on-ARM-if-unsupport.patch new file +28
-0@@ -0,0 +1,28 @@ +From 70b615e8d4d10cda2d961a815dd15eb87a1f3925 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Sun, 4 Oct 2020 14:54:11 +0300 +Subject: [PATCH] mdbx: don't use `yield` instruction on ARM if unsupported. + +Change-Id: I0b01d783fe4336b089f4b051fb61c203b5879aa5 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index aba445424..a9ba1ee85 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -763,7 +763,8 @@ static __always_inline void atomic_yield(void) { + #else + __asm__ __volatile__("hint @pause"); + #endif +-#elif defined(__arm__) || defined(__aarch64__) ++#elif defined(__aarch64__) || (defined(__ARM_ARCH) && __ARM_ARCH > 6) || \ ++ defined(__ARM_ARCH_6K__) + #ifdef __CC_ARM + __yield(); + #else +-- +2.28.0 +package/libmdbx/0005-mdbx-load-fix-minor-false-positive-GCC-warning.patch new file +27
-0@@ -0,0 +1,27 @@ +From 8d4e7994c0c5c0cf69326139717e324f90bff65b Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Mon, 5 Oct 2020 15:09:10 +0300 +Subject: [PATCH] mdbx-load: fix minor false-positive GCC warning. + +Change-Id: Ie75c793712d050e8d3da76a4d0a8df9b81dc5275 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx_load.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/mdbx_load.c b/mdbx_load.c +index e5e72fc8f..54e179553 100644 +--- a/mdbx_load.c ++++ b/mdbx_load.c +@@ -318,7 +318,7 @@ static int readhdr(void) { + } + + for (int i = 0; dbflags[i].bit; i++) { +- bool value; ++ bool value = false; + if (valbool(dbuf.iov_base, dbflags[i].name, &value)) { + if (value) + dbi_flags |= dbflags[i].bit; +-- +2.28.0 +package/libmdbx/0006-mdbx-cmake-pthread-workaround-for-buggy-toolchain-cm.patch new file +65
-0@@ -0,0 +1,65 @@ +From 787eaaa373073e17f3a53658b085c255bc2c8ff8 Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Mon, 5 Oct 2020 19:12:20 +0300 +Subject: [PATCH] mdbx-cmake: pthread workaround for buggy + toolchain/cmake/buildroot. + +Change-Id: I0d95e783abbd10a63cd1595a9de50593e814a967 +Signed-off-by: Leonid Yuriev <[email protected]> +--- + CMakeLists.txt | 21 ++++++++++++++++++--- + 1 file changed, 18 insertions(+), 3 deletions(-) + +diff --git a/CMakeLists.txt b/CMakeLists.txt +index 95f116521..20a50a453 100644 +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -173,10 +173,27 @@ if(NOT CMAKE_BUILD_TYPE) + endif() + string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPERCASE) + ++set(CMAKE_THREAD_PREFER_PTHREAD TRUE) ++set(THREADS_PREFER_PTHREAD_FLAG TRUE) ++find_package(Threads REQUIRED) ++ + include(cmake/utils.cmake) + include(cmake/compiler.cmake) + include(cmake/profile.cmake) + ++# Workaround for `-pthread` toolchain/cmake bug ++if(NOT APPLE AND NOT MSVC ++ AND CMAKE_USE_PTHREADS_INIT AND NOT CMAKE_THREAD_LIBS_INIT ++ AND (CMAKE_COMPILER_IS_GNUCC OR CMAKE_COMPILER_IS_CLANG)) ++ check_compiler_flag("-pthread" CC_HAS_PTHREAD) ++ if(CC_HAS_PTHREAD AND NOT CMAKE_EXE_LINKER_FLAGS MATCHES "-pthread") ++ message(STATUS "Force add -pthread for linker flags to avoid troubles") ++ set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -pthread") ++ set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -pthread") ++ set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} -pthread") ++ endif() ++endif() ++ + CHECK_FUNCTION_EXISTS(pow NOT_NEED_LIBM) + if(NOT_NEED_LIBM) + set(LIB_MATH "") +@@ -190,8 +207,6 @@ else() + endif() + endif() + +-find_package(Threads REQUIRED) +- + if(SUBPROJECT) + if(NOT DEFINED BUILD_SHARED_LIBS) + option(BUILD_SHARED_LIBS "Build shared libraries (DLLs)" OFF) +@@ -541,7 +556,7 @@ macro(target_setup_options TARGET) + endmacro() + + macro(libmdbx_setup_libs TARGET MODE) +- target_link_libraries(${TARGET} ${MODE} ${CMAKE_THREAD_LIBS_INIT}) ++ target_link_libraries(${TARGET} ${MODE} Threads::Threads) + if(${CMAKE_SYSTEM_NAME} STREQUAL "Windows") + target_link_libraries(${TARGET} ${MODE} ntdll.lib) + if(MDBX_NTDLL_EXTRA_IMPLIB AND MDBX_AVOID_CRT) +-- +2.28.0 +package/libmdbx/0007-mdbx-fix-pthread_yield-for-non-GLIBC.patch new file +42
-0@@ -0,0 +1,42 @@ +From 16d43332a9d28ba27aa3d127e8f6b77a87b2f5eb Mon Sep 17 00:00:00 2001 +From: Leonid Yuriev <[email protected]> +Date: Mon, 5 Oct 2020 21:08:09 +0300 +Subject: [PATCH] mdbx: fix `pthread_yield()` for non-GLIBC. + +Change-Id: I080e37a42b62e524896dea8747e9f23e2fcd584f +Signed-off-by: Leonid Yuriev <[email protected]> +--- + mdbx.c | 4 +++- + mdbx.c | 1 + + 2 files changed, 4 insertions(+), 1 deletion(-) + +diff --git a/mdbx.c b/mdbx.c +index a9ba1ee85..ace6f2756 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -777,7 +777,9 @@ static __always_inline void atomic_yield(void) { + defined(__mips64__) || defined(_M_MRX000) || defined(_MIPS_) || \ + defined(__MWERKS__) || defined(__sgi) + __asm__ __volatile__(".word 0x00000140"); +-#else ++#elif defined(__linux__) || defined(__gnu_linux__) || defined(_UNIX03_SOURCE) ++ sched_yield(); ++#elif (defined(_GNU_SOURCE) && __GLIBC_PREREQ(2, 1)) || defined(_OPEN_THREADS) + pthread_yield(); + #endif + } +diff --git a/mdbx.c b/mdbx.c +index 3509942dc..3af23ce0b 100644 +--- a/mdbx.c ++++ b/mdbx.c +@@ -119,6 +119,7 @@ + + #if defined(__linux__) || defined(__gnu_linux__) + #include <linux/sysctl.h> ++#include <sched.h> + #include <sys/sendfile.h> + #include <sys/statfs.h> + #endif /* Linux */ +-- +2.28.0 +package/libmdbx/Config.in new file +42
-0@@ -0,0 +1,42 @@ +config BR2_PACKAGE_LIBMDBX + bool "libmdbx" + depends on BR2_USE_MMU + depends on BR2_TOOLCHAIN_HAS_THREADS + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4 + depends on BR2_TOOLCHAIN_HAS_SYNC_4 + help + One of the fastest compact key-value ACID database + without WAL. libmdbx has a specific set of properties + and capabilities, focused on creating unique lightweight + solutions. + + libmdbx surpasses the legendary LMDB in terms of + reliability, features and performance. + + https://github.com/erthink/libmdbx + +if BR2_PACKAGE_LIBMDBX + +config BR2_PACKAGE_LIBMDBX_TOOLS + bool "install tools" + help + Install libmdbx tools for checking, dump, restore + and show statistics of databases. + +config BR2_PACKAGE_LIBMDBX_CXX + bool "C++ API" + depends on BR2_INSTALL_LIBSTDCPP + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + help + Enable modern C++11/14/17/20 API for libmdbx. + +comment "libmdbx C++ support needs a toolchain w/ C++11, gcc >= 4.8" + depends on !BR2_INSTALL_LIBSTDCPP || \ + !BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + +endif + +comment "libmdbx needs a toolchain w/ threads, gcc >= 4.4" + depends on BR2_USE_MMU + depends on !BR2_TOOLCHAIN_HAS_THREADS || \ + !BR2_TOOLCHAIN_GCC_AT_LEAST_4_4package/libmdbx/libmdbx.hash new file +2
-0@@ -0,0 +1,2 @@ +sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE +sha256 c7fb24381eb4d92f2e2edc17e577cb721269683c816c6cca307c58f6f346e786 libmdbx-amalgamated-0.9.1.tar.gzpackage/libmdbx/libmdbx.mk new file +24
-1@@ -0,0 +1,24 @@ +################################################################################ +# +# libmdbx +# +################################################################################ + +LIBMDBX_VERSION = 0.9.1 +LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz +LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION) +LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO +LIBMDBX_LICENSE = OLDAP-2.8 +LIBMDBX_LICENSE_FILES = LICENSE +LIBMDBX_REDISTRIBUTE = YES +LIBMDBX_STRIP_COMPONENTS = 0 +LIBMDBX_INSTALL_STAGING = YES + +LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES=OFF -DBUILD_FOR_NATIVE_CPU=OFF \ + -DMDBX_INSTALL_STATIC=$(if $(BR2_STATIC_LIBS),ON,OFF) \ + -DMDBX_BUILD_SHARED_LIBRARY=$(if $(BR2_SHARED_LIBS),ON,OFF) \ + -DMDBX_BUILD_CXX=$(if $(BR2_PACKAGE_LIBMDBX_CXX),ON,OFF) \ + -DMDBX_BUILD_TOOLS=$(if $(BR2_PACKAGE_LIBMDBX_TOOLS),ON,OFF) \ + -DMDBX_LINK_TOOLS_NONSTATIC=$(if $(BR2_SHARED_LIBS),ON,OFF) + +$(eval $(cmake-package)) --2.28.0
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
PATCH v4 1/1 package/libmdbx: new package (library/database).
In reply to Leonid YurievHello Leonid,
Thanks for this contribution and new iteration!
Quoted 9 lines · On Tue, 6 Oct 2020 22:27:09 +0300 Leonid Yuriev <[email protected]> wrote:
This patch adds libmdbx v0.9.1:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.
Your Signed-off-by line should be here... and not after the changelog.
Quoted 24 lines
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch forpthread_yield().
- passedutils/check-package package/libmdbx/*(suggested by Heiko Thiery)
- passedutils/test-pkg -a -p libmdbx,
except w/o threads & w/o MMU (suggested by Heiko Thiery).Changes v3 -> v4:
- fix passingBR2_PACKAGE_LIBMDBX_TOOLSoption to cmake.
- fix usingdepend oninstead ofselect,
and addcommentfor C++ toolchain (suggested by Heiko Thiery).
- fix minor help typo.Signed-off-by Leonid Yuriev <[email protected]>
DEVELOPERS N: Leon Anavi <[email protected]> · line 1555
Quoted 2 lines
DEVELOPERS +2
-0+N: Leonid Yuriev <[email protected]> +F: package/libmdbx
Final / missing.
Quoted 22 lines
package/libmdbx/Config.in new file +16
-0@@ -0,0 +1,42 @@ +config BR2_PACKAGE_LIBMDBX + bool "libmdbx" + depends on BR2_USE_MMU + depends on BR2_TOOLCHAIN_HAS_THREADS + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4 + depends on BR2_TOOLCHAIN_HAS_SYNC_4 + help + One of the fastest compact key-value ACID database + without WAL. libmdbx has a specific set of properties + and capabilities, focused on creating unique lightweight + solutions. + + libmdbx surpasses the legendary LMDB in terms of + reliability, features and performance. + + https://github.com/erthink/libmdbx
package/libmdbx/Config.in line 26
Quoted 10 lines
package/libmdbx/Config.in new file +10
-0+ +if BR2_PACKAGE_LIBMDBX + +config BR2_PACKAGE_LIBMDBX_TOOLS + bool "install tools" + help + Install libmdbx tools for checking, dump, restore + and show statistics of databases. + +config BR2_PACKAGE_LIBMDBX_CXX
I don't think this option is needed, just enable C++ support in the .mk file when the relevant conditions are met.
Quoted 4 lines
package/libmdbx/Config.in new file +4
-0+ bool "C++ API" + depends on BR2_INSTALL_LIBSTDCPP + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + help
package/libmdbx/Config.in line 40
Quoted 10 lines
package/libmdbx/Config.in new file +10
-0+ Enable modern C++11/14/17/20 API for libmdbx. + +comment "libmdbx C++ support needs a toolchain w/ C++11, gcc >= 4.8" + depends on !BR2_INSTALL_LIBSTDCPP || \ + !BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + +endif + +comment "libmdbx needs a toolchain w/ threads, gcc >= 4.4" + depends on BR2_USE_MMU
depends on BR2_TOOLCHAIN_HAS_SYNC_4
should be added here as well.
package/libmdbx/libmdbx.hash
Quoted 8 lines
package/libmdbx/Config.in new file +2
-0+ depends on !BR2_TOOLCHAIN_HAS_THREADS || \ + !BR2_TOOLCHAIN_GCC_AT_LEAST_4_4package/libmdbx/libmdbx.hash new file +0
-0@@ -0,0 +1,2 @@
# Locally calculated
package/libmdbx/libmdbx.hash line 2
Quoted 2 lines
package/libmdbx/libmdbx.hash new file +2
-0+sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE +sha256 c7fb24381eb4d92f2e2edc17e577cb721269683c816c6cca307c58f6f346e786 libmdbx-amalgamated-0.9.1.tar.gz
Please put the tarball hash before the license hash.
Quoted 9 lines
package/libmdbx/libmdbx.mk new file +3
-0@@ -0,0 +1,24 @@ +################################################################################ +# +# libmdbx
package/libmdbx/libmdbx.mk line 13
Quoted 10 lines
package/libmdbx/libmdbx.mk new file +10
-0+# +################################################################################ + +LIBMDBX_VERSION = 0.9.1 +LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz +LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION) +LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO +LIBMDBX_LICENSE = OLDAP-2.8 +LIBMDBX_LICENSE_FILES = LICENSE +LIBMDBX_REDISTRIBUTE = YES
Not needed, this is the default.
package/libmdbx/libmdbx.mk line 14
Quoted 1 line
package/libmdbx/libmdbx.mk new file +1
-0+LIBMDBX_STRIP_COMPONENTS = 0
Quite odd, why do you generate your tarballs like this ?
package/libmdbx/libmdbx.mk line 19
Quoted 5 lines
package/libmdbx/libmdbx.mk new file +5
-0+LIBMDBX_INSTALL_STAGING = YES + +LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES=OFF -DBUILD_FOR_NATIVE_CPU=OFF \ + -DMDBX_INSTALL_STATIC=$(if $(BR2_STATIC_LIBS),ON,OFF) \ + -DMDBX_BUILD_SHARED_LIBRARY=$(if $(BR2_SHARED_LIBS),ON,OFF) \
This is not good as it doesn't account for the BR2_SHARED_STATIC_LIBS situation. So it should be:
ifeq ($(BR2_STATIC_LIBS)$(BR2_SHARED_STATIC_LIBS),y) LIBMDBX_CONF_OPTS += -DMDBX_INSTALL_STATIC=ON elseLIBMDBX_CONF_OPTS += -DMDBX_INSTALL_STATIC=OFF endif
ifeq ($(BR2_SHARED_LIBS)$(BR2_SHARED_STATIC_LIBS),y) LIBMDBX_CONF_OPTS += -DMDBX_BUILD_SHARED_LIBRARY=ON elseLIBMDBX_CONF_OPTS += -DMDBX_BUILD_SHARED_LIBRARY=OFF endif
package/libmdbx/libmdbx.mk line 22
Quoted 3 lines
package/libmdbx/libmdbx.mk new file +3
-0+ -DMDBX_BUILD_CXX=$(if $(BR2_PACKAGE_LIBMDBX_CXX),ON,OFF) \ + -DMDBX_BUILD_TOOLS=$(if $(BR2_PACKAGE_LIBMDBX_TOOLS),ON,OFF) \ + -DMDBX_LINK_TOOLS_NONSTATIC=$(if $(BR2_SHARED_LIBS),ON,OFF)
Ditto here.
However, the bigger problem is that it doesn't build for me. I used the following defconfig:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2020.02.tar.bz2" BR2_TOOLCHAIN_EXTERNAL_GCC_5=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_14=y
BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_LIBMDBX=y
BR2_PACKAGE_LIBMDBX_TOOLS=y
BR2_PACKAGE_LIBMDBX_CXX=y
# BR2_TARGET_ROOTFS_TAR is not set
And it fails like this:
In file included from /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.c++:105:0:/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:223:10: error: ?exception_ptr? in namespace ?std? does not name a type
::std::exception_ptr captured_;
^
/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++: In member function ?bool mdbx::exception_thunk::is_clean() const?:
/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2730:66: error: ?captured_? was not declared in this scope
inline bool exception_thunk::is_clean() const noexcept { return !captured_; }
^/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++: In member function ?void mdbx::exception_thunk::capture()?:/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2734:3: error: ?captured_? was not declared in this scope
captured_ = ::std::current_exception();
^/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2734:15: error: ?current_exception? is not a member of ?std?
captured_ = ::std::current_exception();
^
/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++: In member function ?void mdbx::exception_thunk::rethrow_captured() const?:
/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2738:7: error: ?captured_? was not declared in this scope
if (captured_)
^
/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2739:25: error: ?rethrow_exception? is not a member of ?std?
MDBX_CXX20_UNLIKELY ::std::rethrow_exception(captured_);
^make[3]: *** [CMakeFiles/mdbx.dir/build.make:96: CMakeFiles/mdbx.dir/mdbx.c++.o] Error 1
Some older versions of gcc had an issue with exception_ptr, see BR2_TOOLCHAIN_HAS_GCC_BUG_64735. You might need to depend on !BR2_TOOLCHAIN_HAS_GCC_BUG_64735. See other packages that have this dependency in Buildroot.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v4 1/1 package/libmdbx: new package (library/database).
Hi, Thomas.
Thank you for the review, I will definitely take your comments and suggestions into account when preparing the next version of my patch. The next version/iteration will be based on the next release of libmdbx library, i.e. some internal patches will be dropped.
I am currently working on the next release of the library, which had to be postponed due to a couple of bugs.
This was the reason for the delay in my response, sorry.
Regards,
Leonid.
Quoted 235 lines · Thomas Petazzoni
Hello Leonid,
Thanks for this contribution and new iteration!
Quoted 9 lines · On Tue, 6 Oct 2020 22:27:09 +0300 Leonid Yuriev <[email protected]> wrote:
This patch adds libmdbx v0.9.1:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.Your Signed-off-by line should be here... and not after the changelog.
Quoted 24 lines
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch forpthread_yield().
- passedutils/check-package package/libmdbx/*(suggested by Heiko Thiery)
- passedutils/test-pkg -a -p libmdbx,
except w/o threads & w/o MMU (suggested by Heiko Thiery).Changes v3 -> v4:
- fix passingBR2_PACKAGE_LIBMDBX_TOOLSoption to cmake.
- fix usingdepend oninstead ofselect,
and addcommentfor C++ toolchain (suggested by Heiko Thiery).
- fix minor help typo.Signed-off-by Leonid Yuriev <[email protected]>DEVELOPERS N: Leon Anavi <[email protected]> · line 1555
Quoted 2 lines
DEVELOPERS +2
-0+N: Leonid Yuriev <[email protected]> +F: package/libmdbxFinal / missing.
Quoted 22 lines
package/libmdbx/Config.in new file +16
-0@@ -0,0 +1,42 @@ +config BR2_PACKAGE_LIBMDBX + bool "libmdbx" + depends on BR2_USE_MMU + depends on BR2_TOOLCHAIN_HAS_THREADS + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_4 + depends on BR2_TOOLCHAIN_HAS_SYNC_4 + help + One of the fastest compact key-value ACID database + without WAL. libmdbx has a specific set of properties + and capabilities, focused on creating unique lightweight + solutions. + + libmdbx surpasses the legendary LMDB in terms of + reliability, features and performance. + + https://github.com/erthink/libmdbxpackage/libmdbx/Config.in line 26
Quoted 10 lines
package/libmdbx/Config.in new file +10
-0+ +if BR2_PACKAGE_LIBMDBX + +config BR2_PACKAGE_LIBMDBX_TOOLS + bool "install tools" + help + Install libmdbx tools for checking, dump, restore + and show statistics of databases. + +config BR2_PACKAGE_LIBMDBX_CXXI don't think this option is needed, just enable C++ support in the .mk file when the relevant conditions are met.
Quoted 4 lines
package/libmdbx/Config.in new file +4
-0+ bool "C++ API" + depends on BR2_INSTALL_LIBSTDCPP + depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + helppackage/libmdbx/Config.in line 40
Quoted 10 lines
package/libmdbx/Config.in new file +10
-0+ Enable modern C++11/14/17/20 API for libmdbx. + +comment "libmdbx C++ support needs a toolchain w/ C++11, gcc >= 4.8" + depends on !BR2_INSTALL_LIBSTDCPP || \ + !BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 + +endif + +comment "libmdbx needs a toolchain w/ threads, gcc >= 4.4" + depends on BR2_USE_MMUdepends on
BR2_TOOLCHAIN_HAS_SYNC_4should be added here as well.
package/libmdbx/libmdbx.hash
Quoted 8 lines
package/libmdbx/Config.in new file +2
-0+ depends on !BR2_TOOLCHAIN_HAS_THREADS || \ + !BR2_TOOLCHAIN_GCC_AT_LEAST_4_4package/libmdbx/libmdbx.hash new file +0
-0@@ -0,0 +1,2 @@# Locally calculated
package/libmdbx/libmdbx.hash line 2
Quoted 2 lines
package/libmdbx/libmdbx.hash new file +2
-0+sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE +sha256 c7fb24381eb4d92f2e2edc17e577cb721269683c816c6cca307c58f6f346e786 libmdbx-amalgamated-0.9.1.tar.gzPlease put the tarball hash before the license hash.
Quoted 9 lines
package/libmdbx/libmdbx.mk new file +3
-0@@ -0,0 +1,24 @@ +################################################################################ +# +# libmdbxpackage/libmdbx/libmdbx.mk line 13
Quoted 10 lines
package/libmdbx/libmdbx.mk new file +10
-0+# +################################################################################ + +LIBMDBX_VERSION = 0.9.1 +LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz +LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION) +LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO +LIBMDBX_LICENSE = OLDAP-2.8 +LIBMDBX_LICENSE_FILES = LICENSE +LIBMDBX_REDISTRIBUTE = YESNot needed, this is the default.
package/libmdbx/libmdbx.mk line 14
Quoted 1 line
package/libmdbx/libmdbx.mk new file +1
-0+LIBMDBX_STRIP_COMPONENTS = 0Quite odd, why do you generate your tarballs like this ?
package/libmdbx/libmdbx.mk line 19
Quoted 5 lines
package/libmdbx/libmdbx.mk new file +5
-0+LIBMDBX_INSTALL_STAGING = YES + +LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES=OFF -DBUILD_FOR_NATIVE_CPU=OFF \ + -DMDBX_INSTALL_STATIC=$(if $(BR2_STATIC_LIBS),ON,OFF) \ + -DMDBX_BUILD_SHARED_LIBRARY=$(if $(BR2_SHARED_LIBS),ON,OFF) \This is not good as it doesn't account for the
BR2_SHARED_STATIC_LIBSsituation. So it should be:ifeq ($(BR2_STATIC_LIBS)$(BR2_SHARED_STATIC_LIBS),y)
LIBMDBX_CONF_OPTS+= -DMDBX_INSTALL_STATIC=ON elseLIBMDBX_CONF_OPTS+= -DMDBX_INSTALL_STATIC=OFF endififeq ($(BR2_SHARED_LIBS)$(BR2_SHARED_STATIC_LIBS),y)
LIBMDBX_CONF_OPTS+= -DMDBX_BUILD_SHARED_LIBRARY=ON elseLIBMDBX_CONF_OPTS+= -DMDBX_BUILD_SHARED_LIBRARY=OFF endifpackage/libmdbx/libmdbx.mk line 22
Quoted 3 lines
package/libmdbx/libmdbx.mk new file +3
-0+ -DMDBX_BUILD_CXX=$(if $(BR2_PACKAGE_LIBMDBX_CXX),ON,OFF) \ + -DMDBX_BUILD_TOOLS=$(if $(BR2_PACKAGE_LIBMDBX_TOOLS),ON,OFF) \ + -DMDBX_LINK_TOOLS_NONSTATIC=$(if $(BR2_SHARED_LIBS),ON,OFF)Ditto here.
However, the bigger problem is that it doesn't build for me. I used the following defconfig:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2020.02.tar.bz2" BR2_TOOLCHAIN_EXTERNAL_GCC_5=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_14=y
BR2_TOOLCHAIN_EXTERNAL_LOCALE=y
#BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUGis not set
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
#BR2_PACKAGE_BUSYBOXis not set
BR2_PACKAGE_LIBMDBX=y
BR2_PACKAGE_LIBMDBX_TOOLS=y
BR2_PACKAGE_LIBMDBX_CXX=y
#BR2_TARGET_ROOTFS_TARis not setAnd it fails like this:
In file included from
/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.c++:105:0:/home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:223:10: error: ?exception_ptr? in namespace ?std? does not name a type::std::exception_ptr captured_; ^ /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++: In member function ?bool mdbx::exception_thunk::is_clean() const?: /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2730:66: error: ?captured_? was not declared in this scope inline bool exception_thunk::is_clean() const noexcept { return !captured_; } ^ /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++: In member function ?void mdbx::exception_thunk::capture()?: /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2734:3: error: ?captured_? was not declared in this scope captured_ = ::std::current_exception(); ^ /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2734:15: error: ?current_exception? is not a member of ?std? captured_ = ::std::current_exception(); ^ /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++: In member function ?void mdbx::exception_thunk::rethrow_captured() const?: /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2738:7: error: ?captured_? was not declared in this scope if (captured_) ^ /home/thomas/projets/buildroot/output/build/libmdbx-0.9.1/mdbx.h++:2739:25: error: ?rethrow_exception? is not a member of ?std? MDBX_CXX20_UNLIKELY ::std::rethrow_exception(captured_); ^make[3]: *** [CMakeFiles/mdbx.dir/build.make:96:
CMakeFiles/mdbx.dir/mdbx.c++.o] Error 1Some older versions of gcc had an issue with
exception_ptr, seeBR2_TOOLCHAIN_HAS_GCC_BUG_64735. You might need to depend on !BR2_TOOLCHAIN_HAS_GCC_BUG_64735. See other packages that have this dependency in Buildroot.Thanks!
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
PATCH v2 1/1 package/libmdbx: new package (library/database).
In reply to Heiko ThieryHi, Heiko.
Thanks for your advice.
A few minutes ago, I sent the third version of the patch, where all issues are eliminated.
Thank you again.
Regards,
Leonid.
Quoted 7 lines · Heiko Thiery
Hi Leonid,
Thank you for this contribution.
Below are some comments.
[...]
Quoted 11 lines
By the way .. there are two tools that help validating new package:
utils/tes-pkg -p libmdbx
utils/check-package package/libmdbx/*Please check and fix the findings of these tests.
Thank you
--
Heiko
v5 cover cover letter for package/libmdbx: new package (library/database)
This patch adds libmdbx v0.9.2 and below is a brief overview of libmdbx. Please merge.
Regards,
Leonid.
--
libmdbx is an extremely fast, compact, powerful, embedded, transactional key-value database, with permissive license. libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
Historically, libmdbx (MDBX) is a deeply revised and extended descendant of the legendary LMDB (Lightning Memory-Mapped Database). libmdbx inherits all benefits from LMDB, but resolves some issues and adds a set of improvements.
According to developers, for now libmdbx surpasses the LMDB in terms of reliability, features and performance.
The most important differences MDBX from LMDB:
==============================================
1. More attention is paid to the quality of the code, to an "unbreakability" of the API, to testing and automatic checks (i.e. sanitizers, etc). So there:
- more control during operation;
- more checking parameters, internal audit of database structures;
- no warnings from compiler;
- no issues from ASAN, UBSAN, Valgrind, Coverity;
- etc.
2. Keys could be more than 2 times longer than LMDB.
3. Up to 20% faster than LMDB in CRUD benchmarks.
4. Automatic on-the-fly database size adjustment,
both increment and reduction.
5. Automatic continuous zero-overhead database compactification.
6. The same database format for 32- and 64-bit builds.
7. LIFO policy for Garbage Collection recycling (this can significantly increase write performance due write-back disk cache up to several times in a best case scenario).
8. Range query estimation.
9. Utility for checking the integrity of the database structure with some recovery capabilities.
For more info please refer:
- https://github.com/erthink/libmdbx for source code and README.
- https://erthink.github.io/libmdbx for API description.
--
MDBX is a Btree-based database management library modeled loosely on the BerkeleyDB API, but much simplified. The entire database (aka "environment") is exposed in a memory map, and all data fetches return data directly from the mapped memory, so no malloc's or memcpy's occur during data fetches. As such, the library is extremely simple because it requires no page caching layer of its own, and it is extremely high performance and memory-efficient. It is also fully transactional with full ACID semantics, and when the memory map is read-only, the database integrity cannot be corrupted by stray pointer writes from application code.
The library is fully thread-aware and supports concurrent read/write access from multiple processes and threads. Data pages use a copy-on-write strategy so no active data pages are ever overwritten, which also provides resistance to corruption and eliminates the need of any special recovery procedures after a system crash. Writes are fully serialized; only one write transaction may be active at a time, which guarantees that writers can never deadlock. The database structure is multi-versioned so readers run with no locks; writers cannot block readers, and readers don't block writers.
Unlike other well-known database mechanisms which use either write-ahead transaction logs or append-only data writes, MDBX requires no maintenance during operation. Both write-ahead loggers and append-only databases require periodic checkpointing and/or compaction of their log or database files otherwise they grow without bound. MDBX tracks retired/freed pages within the database and re-uses them for new write operations, so the database size does not grow without bound in normal use.
The memory map can be used as a read-only or read-write map. It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database.
Features
========
- Key-value data model, keys are always sorted.
- Fully ACID-compliant, through to MVCC and CoW.
- Multiple key-value sub-databases within a single datafile.
- Range lookups, including range query estimation.
- Efficient support for short fixed length keys, including native 32/64-bit integers.
- Ultra-efficient support for multimaps. Multi-values sorted, searchable and iterable. Keys stored without duplication.
- Data is memory-mapped and accessible directly/zero-copy. Traversal of database records is extremely-fast.
- Transactions for readers and writers, ones do not block others.
- Writes are strongly serialized. No transaction conflicts nor deadlocks.
- Readers are non-blocking, notwithstanding snapshot isolation.
- Nested write transactions.
- Reads scale linearly across CPUs.
- Continuous zero-overhead database compactification.
- Automatic on-the-fly database size adjustment.
- Customizable database page size.
- Olog(N) cost of lookup, insert, update, and delete operations by virtue of B+ tree characteristics.
- Online hot backup.
- Append operation for efficient bulk insertion of pre-sorted data.
- No WAL nor any transaction journal. No crash recovery needed. No maintenance is required.
- No internal cache and/or memory management, all done by basic OS services.
Limitations
===========
- Page size: a power of 2, maximum 65536 bytes, default 4096 bytes.
- Key size: minimum 0, maximum ?? pagesize (1300 bytes for default 4K pagesize, 21780 bytes for 64K pagesize).
- Value size: minimum 0, maximum 2146435072 (0x7FF00000) bytes for maps, ?? pagesize for multimaps (1348 bytes default 4K pagesize, 21828 bytes for 64K pagesize).
- Write transaction size: up to 4194301 (0x3FFFFD) pages (16 GiB for default 4K pagesize, 256 GiB for 64K pagesize).
- Database size: up to 2147483648 pages (8 TiB for default 4K pagesize, 128 TiB for 64K pagesize).
- Maximum sub-databases: 32765.
Gotchas
=======
- There cannot be more than one writer at a time, i.e. no more than one write transaction at a time.
- libmdbx is based on B+ tree, so access to database pages is mostly random. Thus SSDs provide a significant performance boost over spinning disks for large databases.
- libmdbx uses shadow paging instead of WAL. Thus syncing data to disk might be a bottleneck for write intensive workload.
- libmdbx uses copy-on-write for snapshot isolation during updates, but read transactions prevents recycling an old retired/freed pages, since it read ones. Thus altering of data during a parallel long-lived read operation will increase the process work set, may exhaust entire free database space, the database can grow quickly, and result in performance degradation. Try to avoid long running read transactions.
- libmdbx is extraordinarily fast and provides minimal overhead for data access, so you should reconsider using brute force techniques and double check your code. On the one hand, in the case of libmdbx, a simple linear search may be more profitable than complex indexes. On the other hand, if you make something suboptimally, you can notice detrimentally only on sufficiently large data.
--
Leonid Yuriev (1):
package/libmdbx: new package (library/database).
DEVELOPERS | 3 +++ package/Config.in | 1 + package/libmdbx/Config.in | 45 ++++++++++++++++++++++++++++++++++++ package/libmdbx/libmdbx.hash | 5 ++++ package/libmdbx/libmdbx.mk | 33 ++++++++++++++++++++++++++ 5 files changed, 87 insertions(+) create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mk
Signature · 1 line
2.29.2
PATCH v5 1/1 package/libmdbx: new package (library/database).
This patch adds libmdbx v0.9.2:
- libmdbx is one of the fastest compact embeddable key-value ACID database.
- libmdbx has a specific set of properties and capabilities, focused on creating unique lightweight solutions.
- libmdbx surpasses the legendary LMDB (Lightning Memory-Mapped Database) in terms of reliability, features and performance.
- https://github.com/erthink/libmdbx
---
Changes v1 -> v2:
- libmdbx version v0.8.2 -> v0.9.1 (released 2020-09-30)
Changes v2 -> v3:
- removed outcommented stuff (suggested by Heiko Thiery).
- cleaned up and simplified the makefile (suggested by Heiko Thiery).
- added patch for C++ header installation.
- added patch with fix minor copy&paste typo.
- added patch with pthread workaround for buggy toolchain/cmake/buildroot.
- added patch for pthread_yield().
- passed utils/check-package package/libmdbx/* (suggested by Heiko Thiery)
- passed utils/test-pkg -a -p libmdbx,
except w/o threads & w/o MMU (suggested by Heiko Thiery).
Changes v3 -> v4:
- fix passing BR2_PACKAGE_LIBMDBX_TOOLS option to cmake.
- fix using depend on instead of select,
and add comment for C++ toolchain (suggested by Heiko Thiery).
- fix minor help typo.
Changes v4 -> v5:
- libmdbx version v0.9.1 -> v0.9.2 (released 2020-11-27)
- dropped all patch since not needed.
- cosmetic changes (suggested by Thomas Petazzoni <[email protected]>).
- added dependence of BR2_TOOLCHAIN_HAS_SYNC_4 (suggested by Thomas Petazzoni <[email protected]>).
- added dependence of !BR2_TOOLCHAIN_HAS_GCC_BUG_64735 (suggested by Thomas Petazzoni <[email protected]>).
- take in account the BR2_SHARED_STATIC_LIBS (suggested by Thomas Petazzoni <[email protected]>).
--- DEVELOPERS | 3 +++ package/Config.in | 1 + package/libmdbx/Config.in | 45 ++++++++++++++++++++++++++++++++++++ package/libmdbx/libmdbx.hash | 5 ++++ package/libmdbx/libmdbx.mk | 33 ++++++++++++++++++++++++++ 5 files changed, 87 insertions(+) create mode 100644 package/libmdbx/Config.in create mode 100644 package/libmdbx/libmdbx.hash create mode 100644 package/libmdbx/libmdbx.mk
DEVELOPERS +3 -0
@@ -1482,6 +1482,9 @@ N: Leon Anavi <[email protected]> F: board/olimex/a10_olinuxino F: configs/olimex_a10_olinuxino_lime_defconfig +N: Leonid Yuriev <[email protected]> +F: package/libmdbx/ + N: Lionel Flandrin <[email protected]> F: package/python-babel/ F: package/python-daemonize/
package/Config.in +1 -0
@@ -1372,6 +1372,7 @@ menu "Database" source "package/kompexsqlite/Config.in" source "package/leveldb/Config.in" source "package/libgit2/Config.in" + source "package/libmdbx/Config.in" source "package/libodb/Config.in" source "package/libodb-boost/Config.in" source "package/libodb-mysql/Config.in"
package/libmdbx/Config.in new file +45 -0
45 lines not on this page · raw message
package/libmdbx/libmdbx.hash new file +5 -0
@@ -0,0 +1,5 @@ +# Hashes from: https://github.com/erthink/libmdbx/releases/ +sha256 c35cc53d66d74ebfc86e39441ba26276541ac7892bf91dba1e70c83665a02767 libmdbx-amalgamated-0.9.2.tar.gz + +# Locally calculated +sha256 310fe25c858a9515fc8c8d7d1f24a67c9496f84a91e0a0e41ea9975b1371e569 LICENSE
package/libmdbx/libmdbx.mk new file +33 -0
@@ -0,0 +1,33 @@ +################################################################################ +# +# libmdbx +# +################################################################################ + +LIBMDBX_VERSION = 0.9.2 +LIBMDBX_SOURCE = libmdbx-amalgamated-$(LIBMDBX_VERSION).tar.gz +LIBMDBX_SITE = https://github.com/erthink/libmdbx/releases/download/v$(LIBMDBX_VERSION) +LIBMDBX_SUPPORTS_IN_SOURCE_BUILD = NO +LIBMDBX_LICENSE = OLDAP-2.8 +LIBMDBX_LICENSE_FILES = LICENSE +LIBMDBX_REDISTRIBUTE = YES +LIBMDBX_STRIP_COMPONENTS = 0 +LIBMDBX_INSTALL_STAGING = YES + +LIBMDBX_CONF_OPTS = -DMDBX_INSTALL_MANPAGES=OFF -DBUILD_FOR_NATIVE_CPU=OFF \ + -DMDBX_BUILD_CXX=$(if $(BR2_PACKAGE_LIBMDBX_CXX),ON,OFF) \ + -DMDBX_BUILD_TOOLS=$(if $(BR2_PACKAGE_LIBMDBX_TOOLS),ON,OFF) + +ifeq ($(BR2_STATIC_LIBS)$(BR2_SHARED_STATIC_LIBS),y) +LIBMDBX_CONF_OPTS += -DMDBX_INSTALL_STATIC=ON +else +LIBMDBX_CONF_OPTS += -DMDBX_INSTALL_STATIC=OFF +endif + +ifeq ($(BR2_SHARED_LIBS)$(BR2_SHARED_STATIC_LIBS),y) +LIBMDBX_CONF_OPTS += -DMDBX_BUILD_SHARED_LIBRARY=ON -DMDBX_LINK_TOOLS_NONSTATIC=ON +else +LIBMDBX_CONF_OPTS += -DMDBX_BUILD_SHARED_LIBRARY=OFF -DMDBX_LINK_TOOLS_NONSTATIC=OFF +endif + +$(eval $(cmake-package))
Signature · 1 line
2.29.2