Skip to contentSkip to conversation

qt5webkit: fix build issue using system leveldb

PATCH New Gaël PORTAY 2 participants 6 messages Sep 3, 2018

lore
Share & tools
lore
#0 lore raw

v2 cover qt5webkit: fix build issue using system leveldb

Hi all,

This patchset fixes the build issue reported by autobuilder [0].

/home/naourr/work/instance-2/output/build/qt5webkit-5.9.1/Source/WebCore//.obj/platform/leveldb/LevelDBDatabase.o: In function `WebCore::LevelDBDatabase::openInMemory(WebCore::LevelDBComparator const*)':
LevelDBDatabase.cpp.text._ZN7WebCore15LevelDBDatabase12openInMemoryEPKNS_17LevelDBComparatorE+0x34): undefined reference to `leveldb::NewMemEnv(leveldb::Env*)'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile.api:97: ../lib/libQt5WebKit.so.5.9.1] Error 1

The issue happens when the package leveldb is enabled.

QtWebKit builds its own copy of leveldb [1] (as a third-party) if the system does not provided it (i.e. buildroot). It builds it differently and this is the origin of that issue. Instead of using the Makefile provided by leveldb [2], QtWebKit uses qmake to build that library [3].

The missing symbol issue happens because the symbol leveldb::NewMemEnv is bundled in the static library libmemenv.a (aside libleveldb.so). This static library consists of this single symbol which is like an extra that is built but *NOT* shipped by default at installation in the staging directory. Unfortunatly, that symbol is required later by WebCore [4].

The missing symbol issue happens because the symbol leveldb::NewMemEnv is bundled in the static library libmemenv.a (aside libleveldb.so). This static library consists of this single symbol which is like an extra that is built but *NOT* shipped by default at installation in the staging directory. Unfortunatly, that symbol is required later by WebCore [4].

The copy built by QtWebKit is an all-in-one library including both libleveldb and libmemenv; thus QtWebKit links against libleveldb only. Also, the linker finds the buildroot's copy first (not the third-party):
that explains why it is complaining about a missing symbol. That copy does not have the symbol leveldb::NewMemEnv.

Fortunatly, QtWebKit provides a facility to link against the system leveldb package. The qmake flag WEBKIT_CONFIG+=use_system_leveldb tells Qt5WebKit to link against libleveldb *AND* libmemenv [5].

To fix that issue, this patchset builds the leveldb package and tells QtWebKit to use it instead of building its own copy.

The first two patches concern the leveldb package.

The first one adds installation of headers and the missing static library in the staging directory.

The second one patches leveldb to generate position independant code for the static library libmemenv.a; in order let it linkable by the Qt5WebKit module. Without that patch, the build raises the error below:

/home/gportay/src/buildroot/output/host/opt/ext-toolchain/bin/../lib/gcc/x86_64-amd-linux-gnu/6.2.0/../../../../x86_64-amd-linux-gnu/bin/ld: /home/gportay/src/buildroot/output/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/libmemenv.a(memenv.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC /home/gportay/src/buildroot/output/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/libmemenv.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status

It it not relevant to upstream that patch since the project has already moved to cmake [6] for the (upcoming?) version 1.21. Backporting the upstreamed patches was way too complicated.

The last patch selects the leveldb package in qt5webkit, installs memenv and sets the appropriate qmake flag to tell it to use the package provided by buildroot.

[0]: http://autobuild.buildroot.net/results/8afc8771741f62da3b2117d2124d1c6f2d941903/
[1]: https://github.com/qt/qtwebkit/tree/5.9/Source/ThirdParty/leveldb
[2]: https://github.com/qt/qtwebkit/blob/5.9/Source/ThirdParty/leveldb/Makefile#L167-L169
[3]: https://github.com/qt/qtwebkit/blob/5.9/Source/ThirdParty/leveldb/Target.pri#L80
[4]: https://github.com/qt/qtwebkit/blob/5.9/Source/WebCore/platform/leveldb/LevelDBDatabase.cpp#L185
[5]: https://github.com/qt/qtwebkit/blob/5.9/Source/WebCore/WebCore.pri#L254
[6]: https://github.com/google/leveldb/commit/739c25100e46576cdcdfff2d6f43f9f7008103c7

Regards,
Ga?l PORTAY (3):
leveldb: install memenv static library and header
leveldb: generate pic for static libraries
qt5webkit: select leveldb package and memenv

 ...sition-independant-code-for-static-librar.patch | 52 ++++++++++++++++++++++
 package/leveldb/Config.in                          | 10 +++++
 package/leveldb/leveldb.mk                         |  8 ++++
 package/qt5/qt5webkit/Config.in                    |  4 ++
 package/qt5/qt5webkit/qt5webkit.mk                 |  4 +-
 5 files changed, 76 insertions(+), 2 deletions(-)
 create mode 100644 package/leveldb/0001-Generate-position-independant-code-for-static-librar.patch
Signature · 1 line
2.11.0
#1 lore raw

PATCH v2 1/3 leveldb: install memenv static library and header

The project builds a tiny static library that consists of a single symbol which creates an in-memory LevelDB database.

That library is not installed by default and may be used by other projects.

This commit creates a config option that installs in the staging directory the libmemenv.a static library and the memenv.h header file when the option is set.

---
 package/leveldb/Config.in  | 10 ++++++++++
 package/leveldb/leveldb.mk |  8 ++++++++
 2 files changed, 18 insertions(+)
package/leveldb/Config.in +10 -0
index 163a75e8f2..4ceb2cfb5b 100644
@@ -10,5 +10,15 @@ config BR2_PACKAGE_LEVELDB
 
 	  https://github.com/google/leveldb
 
+if BR2_PACKAGE_LEVELDB
+
+config BR2_PACKAGE_LEVELDB_MEMENV
+	bool "memenv"
+	help
+	  Installs memenv static library and header to create in-memory
+	  LevelDB database.
+
+endif
+
 comment "leveldb needs a toolchain w/ C++, threads"
 	depends on !BR2_INSTALL_LIBSTDCPP || !BR2_TOOLCHAIN_HAS_THREADS
package/leveldb/leveldb.mk +8 -0
index 54942a0f27..c8c2e102c0 100644
@@ -21,10 +21,18 @@ define LEVELDB_BUILD_CMDS
 		$(LEVELDB_MAKE_ARGS) -C $(@D)
 endef
 
+ifdef BR2_PACKAGE_LEVELDB_MEMENV
+define LEVELDB_INSTALL_MEMENV
+	$(INSTALL) -m 0644 $(@D)/out-static/libmemenv.a $(STAGING_DIR)/usr/lib
+	$(INSTALL) -D -m 0644 $(@D)/helpers/memenv/memenv.h $(STAGING_DIR)/usr/include/helpers/memenv/memenv.h
+endef
+endif
+
 define LEVELDB_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) \
 		INSTALL_ROOT=$(STAGING_DIR) INSTALL_PREFIX=/usr \
 		$(LEVELDB_MAKE_ARGS) -C $(@D) install
+	$(LEVELDB_INSTALL_MEMENV)
 endef
 
 define LEVELDB_INSTALL_TARGET_CMDS
Signature · 1 line
2.11.0
#2 lore raw

PATCH v2 1/3 leveldb: install memenv static library and header

Hello,

Quoted 1 line · Ga?l PORTAY

+ifdef BR2_PACKAGE_LEVELDB_MEMENV

Too much vacations here! :-) You have forgotten how Buildroot does this normally:

ifeq ($(BR2_PACKAGE_LEVELDB_MEMENV),y)

package/leveldb/leveldb.mk define LEVELDB_BUILD_CMDS · line 26

Quoted 2 lines
package/leveldb/leveldb.mk +2 -0
+define LEVELDB_INSTALL_MEMENV
+	$(INSTALL) -m 0644 $(@D)/out-static/libmemenv.a $(STAGING_DIR)/usr/lib

-D option and full destination path please.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH v2 1/3 leveldb: install memenv static library and header

Thomas,

Quoted 11 lines · Thomas Petazzoni

Hello,

Quoted 1 line · Ga?l PORTAY

+ifdef BR2_PACKAGE_LEVELDB_MEMENV

Too much vacations here! :-) You have forgotten how Buildroot does this normally:

ifeq ($(BR2_PACKAGE_LEVELDB_MEMENV),y)

... and I also forgot to add the Changes in the coverletter :(

Quoted 5 lines

package/leveldb/leveldb.mk define LEVELDB_BUILD_CMDS · line 26

Quoted 2 lines
package/leveldb/leveldb.mk +2 -0
+define LEVELDB_INSTALL_MEMENV
+	$(INSTALL) -m 0644 $(@D)/out-static/libmemenv.a $(STAGING_DIR)/usr/lib

-D option and full destination path please.

Okay, I will resend a v3 later that day.

Quoted 7 lines

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Regards,
Ga?l

#4 lore raw

PATCH v2 2/3 leveldb: generate pic for static libraries

In reply to Gaël PORTAY

The project's static libraries are not compiled with the -fPIC compiler flag that prevents any libraries to link against those libraries.

This commit adds a patch that sets the -fPIC compiler flag to the list of CFLAGS/CXXFLAGS.

The project now generates position independant code for all of its outputs (i.e. not limited anymore to its shared libraries).

Fixes:

/home/gportay/src/buildroot/output/host/opt/ext-toolchain/bin/../lib/gcc/x86_64-amd-linux-gnu/6.2.0/../../../../x86_64-amd-linux-gnu/bin/ld: /home/gportay/src/buildroot/output/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/libmemenv.a(memenv.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC /home/gportay/src/buildroot/output/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/libmemenv.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status

---
 ...sition-independant-code-for-static-librar.patch | 52 ++++++++++++++++++++++
 1 file changed, 52 insertions(+)
 create mode 100644 package/leveldb/0001-Generate-position-independant-code-for-static-librar.patch
package/leveldb/0001-Generate-position-independant-code-for-static-librar.patch new file +52 -0
new file mode 100644
index 0000000000..dce06ec725
@@ -0,0 +1,52 @@
+From 6ed1b57ef6bcee0d497c181730710b2b0fafbfb3 Mon Sep 17 00:00:00 2001
+From: =?utf-8?q?Ga=C3=ABl=20PORTAY?= <[email protected]>
+Date: Fri, 31 Aug 2018 12:23:46 -0400
+Subject: [PATCH] Generate position independant code for static library
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: 8bit
+
+Currently, only shared libraries are using the PIC flag.
+
+Generalize this flag for static libraries in order to let them linkable
+by dynamic libraries.
+
+Fixes:
+
+	/home/gportay/src/buildroot/output/host/opt/ext-toolchain/bin/../lib/gcc/x86_64-amd-linux-gnu/6.2.0/../../../../x86_64-amd-linux-gnu/bin/ld: /home/gportay/src/buildroot/output/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/libmemenv.a(memenv.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
+	/home/gportay/src/buildroot/output/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/libmemenv.a: error adding symbols: Bad value
+	collect2: error: ld returned 1 exit status
+
+Upstream-Status: Inappropriate [upstream has migrated to cmake]
+Signed-off-by: Ga?l PORTAY <[email protected]>
+---
+ build_detect_platform | 6 +++---
+ 1 file changed, 3 insertions(+), 3 deletions(-)
+
+diff --git a/build_detect_platform b/build_detect_platform
+index d2a20ce..4839444 100755
+--- a/build_detect_platform
++++ b/build_detect_platform
+@@ -55,8 +55,8 @@ fi
+ 
+ COMMON_FLAGS=
+ CROSS_COMPILE=
+-PLATFORM_CCFLAGS=
+-PLATFORM_CXXFLAGS=
++PLATFORM_CCFLAGS="-fPIC"
++PLATFORM_CXXFLAGS="-fPIC"
+ PLATFORM_LDFLAGS=
+ PLATFORM_LIBS=
+ PLATFORM_SHARED_EXT="so"
+@@ -197,7 +197,7 @@ else
+ EOF
+     if [ "$?" = 0 ]; then
+         COMMON_FLAGS="$COMMON_FLAGS -DLEVELDB_PLATFORM_POSIX -DLEVELDB_ATOMIC_PRESENT"
+-        PLATFORM_CXXFLAGS="-std=c++0x"
++        PLATFORM_CXXFLAGS="$PLATFORM_CXXFLAGS -std=c++0x"
+     else
+         COMMON_FLAGS="$COMMON_FLAGS -DLEVELDB_PLATFORM_POSIX"
+     fi
+-- 
+2.18.0
+
Signature · 1 line
2.11.0
#5 lore raw

PATCH v2 3/3 qt5webkit: select leveldb package and memenv

In reply to Gaël PORTAY

This patch fixes the build issue reported by autobuilder [0].

/home/naourr/work/instance-2/output/build/qt5webkit-5.9.1/Source/WebCore//.obj/platform/leveldb/LevelDBDatabase.o: In function `WebCore::LevelDBDatabase::openInMemory(WebCore::LevelDBComparator const*)':
LevelDBDatabase.cpp.text._ZN7WebCore15LevelDBDatabase12openInMemoryEPKNS_17LevelDBComparatorE+0x34): undefined reference to `leveldb::NewMemEnv(leveldb::Env*)'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile.api:97: ../lib/libQt5WebKit.so.5.9.1]
Error 1

The issue happens when both packages leveldb and qt5webkit are enabled.

QtWebKit builds its own copy of leveldb [1] (as a third-party) if the system does not provided it (i.e. buildroot). It builds it differently and this is the origin of that issue. Instead of using the Makefile provided by leveldb [2], QtWebKit uses qmake to build that library [3].

The missing symbol issue happens because the symbol leveldb::NewMemEnv is bundled in the static library libmemenv.a (aside libleveldb.so). This static library consists of this single symbol which is like an extra that is built but *NOT* shipped by default at installation in the staging directory. Unfortunatly, that symbol is required later by WebCore [4].

The copy built by QtWebKit is an all-in-one library including both libleveldb and libmemenv; thus QtWebKit links against libleveldb only. Also, the linker finds the buildroot's copy first (not the third-party):
that explains why it is complaining about a missing symbol. That copy does not have the symbol leveldb::NewMemEnv.

Fortunatly, QtWebKit provides a facility to link against the system leveldb package. The qmake flag WEBKIT_CONFIG+=use_system_leveldb tells Qt5WebKit to link against libleveldb *AND* libmemenv [5].

To fix that issue, this commit selects the package leveldb and its option BR2_PACKAGE_LEVELDB_MEMENV to install libmemenv static library and its header. It ensures that QtWebKit has every thing it needs to be built. It also sets the appropriate qmake configure flags to tells QtWebKit to use the leveldb copy build by the system (instead of the one that can be build internaly as a third-party).

[0]: http://autobuild.buildroot.net/results/8afc8771741f62da3b2117d2124d1c6f2d941903/
[1]: https://github.com/qt/qtwebkit/tree/5.9/Source/ThirdParty/leveldb
[2]: https://github.com/qt/qtwebkit/blob/5.9/Source/ThirdParty/leveldb/Makefile#L167-L169
[3]: https://github.com/qt/qtwebkit/blob/5.9/Source/ThirdParty/leveldb/Target.pri#L80
[4]: https://github.com/qt/qtwebkit/blob/5.9/Source/WebCore/platform/leveldb/LevelDBDatabase.cpp#L185
[5]: https://github.com/qt/qtwebkit/blob/5.9/Source/WebCore/WebCore.pri#L254
[6]: https://github.com/google/leveldb/commit/739c25100e46576cdcdfff2d6f43f9f7008103c7

---
 package/qt5/qt5webkit/Config.in    | 4 ++++
 package/qt5/qt5webkit/qt5webkit.mk | 4 ++--
 2 files changed, 6 insertions(+), 2 deletions(-)
package/qt5/qt5webkit/Config.in +4 -0
index 72c641b9f7..0f5e9f98e7 100644
@@ -3,10 +3,14 @@ config BR2_PACKAGE_QT5WEBKIT
 	depends on !BR2_STATIC_LIBS
 	depends on BR2_PACKAGE_QT5_JSCORE_AVAILABLE
 	depends on BR2_HOST_GCC_AT_LEAST_4_8 # icu
+	depends on BR2_INSTALL_LIBSTDCPP # leveldb
 	depends on BR2_TOOLCHAIN_GCC_AT_LEAST_4_8 # icu
+	depends on BR2_TOOLCHAIN_HAS_THREADS # leveldb
 	depends on !BR2_BINFMT_FLAT # icu
 	# assumes a FPU is available on MIPS
 	depends on !BR2_MIPS_SOFT_FLOAT
+	select BR2_PACKAGE_LEVELDB
+	select BR2_PACKAGE_LEVELDB_MEMENV
 	select BR2_PACKAGE_QT5BASE
 	select BR2_PACKAGE_QT5BASE_ICU
 	select BR2_PACKAGE_QT5BASE_GUI
package/qt5/qt5webkit/qt5webkit.mk +2 -2
index e8d368fb01..a0a0998d12 100644
@@ -16,7 +16,7 @@ endif
 QT5WEBKIT_SOURCE = qtwebkit-opensource-src-$(QT5WEBKIT_VERSION).tar.xz
 QT5WEBKIT_DEPENDENCIES = \
 	host-bison host-flex host-gperf host-python host-ruby \
-	qt5base sqlite
+	leveldb qt5base sqlite
 QT5WEBKIT_INSTALL_STAGING = YES
 
 QT5WEBKIT_LICENSE_FILES = Source/WebCore/LICENSE-LGPL-2 Source/WebCore/LICENSE-LGPL-2.1
@@ -45,7 +45,7 @@ endef
 QT5WEBKIT_PRE_CONFIGURE_HOOKS += QT5WEBKIT_PYTHON2_SYMLINK
 
 define QT5WEBKIT_CONFIGURE_CMDS
-	(cd $(@D); $(TARGET_MAKE_ENV) $(QT5WEBKIT_ENV) $(HOST_DIR)/bin/qmake)
+	(cd $(@D); $(TARGET_MAKE_ENV) $(QT5WEBKIT_ENV) $(HOST_DIR)/bin/qmake WEBKIT_CONFIG+=use_system_leveldb)
 endef
 
 define QT5WEBKIT_BUILD_CMDS
Signature · 1 line
2.11.0