Skip to contentSkip to conversation

cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

PATCH In review Leon Woestenberg 4 participants 6 messages Aug 14, 2020 – Aug 19, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cmake-native: Use cmake-provided zstd library; supported host distro zstd may be too old.

cmake-native uses the system provided libraries due to:

CMAKE_EXTRACONF = "<...> -DCMAKE_USE_SYSTEM_LIBRARIES=1 <...>"

Now, iff the libzstd(-dev) is installed but too old, which can happen on Ubuntu 16.04:

dpkg -l | grep zstd
ii libzstd-dev 0.5.1-1
ii libzstd0 0.5.1-1

cmake configure will use the zstd system library:

cat ./tmp/work/x86_64-linux/cmake-native/3.15.3-r0/temp/log.do_configure | grep -i ZSTD -- Using system-installed ZSTD
-- Found ZSTD: /usr/lib/x86_64-linux-gnu/libzstd.so

but will fail to compile due to:

<...>/tmp/work/x86_64-linux/cmake-native/3.15.3-r0/cmake-3.15.3/Utilities/cmlibarchive/libarchive/archive_read_support_filter_zstd.c:59:2: error: unknown type name ‘ZSTD_DStream’

|   ZSTD_DStream *dstream;
|   ^

The library does not contain that datatype:

grep -rne ZSTD_DStream /usr/include/zstd.h
<no output>

Apparently the cmake-native library check is not checking the library version or features.

This fix is to not depend on the system library and use the zstd library provided with cmake.

CMAKE_EXTRACONF = "\
<...>

    -DCMAKE_USE_SYSTEM_LIBRARY_ZSTD=0 \

<...>

---
 meta/recipes-devtools/cmake/cmake-native_3.16.5.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-devtools/cmake/cmake-native_3.16.5.bb +1 -0
index b2952ee..e0ac3f8 100644
@@ -21,6 +21,7 @@ CMAKE_EXTRACONF = "\
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBARCHIVE=0 \
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBUV=0 \
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBRHASH=0 \
+    -DCMAKE_USE_SYSTEM_LIBRARY_ZSTD=0 \
     -DENABLE_ACL=0 -DHAVE_ACL_LIBACL_H=0 \
     -DHAVE_SYS_ACL_H=0 \
 "
Signature · 1 line
2.7.4
#1 lore raw

This needs to go to master first probably?

Alex

Quoted 64 lines · Leon Woestenberg

cmake-native uses the system provided libraries due to:

CMAKE_EXTRACONF = "<...> -DCMAKE_USE_SYSTEM_LIBRARIES=1 <...>"

Now, iff the libzstd(-dev) is installed but too old, which can happen on Ubuntu 16.04:

dpkg -l | grep zstd
ii libzstd-dev 0.5.1-1
ii libzstd0 0.5.1-1

cmake configure will use the zstd system library:

cat ./tmp/work/x86_64-linux/cmake-native/3.15.3-r0/temp/log.do_configure | grep -i ZSTD
-- Using system-installed ZSTD
-- Found ZSTD: /usr/lib/x86_64-linux-gnu/libzstd.so

but will fail to compile due to:

<...>/tmp/work/x86_64-linux/cmake-native/3.15.3-r0/cmake-3.15.3/Utilities/cmlibarchive/libarchive/archive_read_support_filter_zstd.c:59:2:
error: unknown type name ‘ZSTD_DStream’

|   ZSTD_DStream *dstream;
|   ^

The library does not contain that datatype:

grep -rne ZSTD_DStream /usr/include/zstd.h
<no output>

Apparently the cmake-native library check is not checking the library version or features.

This fix is to not depend on the system library and use the zstd library provided with cmake.

CMAKE_EXTRACONF = "\
<...>

    -DCMAKE_USE_SYSTEM_LIBRARY_ZSTD=0 \

<...>

---
 meta/recipes-devtools/cmake/cmake-native_3.16.5.bb | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-devtools/cmake/cmake-native_3.16.5.bb +1 -1
index b2952ee..e0ac3f8 100644
@@ -21,6 +21,7 @@ CMAKE_EXTRACONF = "\
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBARCHIVE=0 \
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBUV=0 \
     -DCMAKE_USE_SYSTEM_LIBRARY_LIBRHASH=0 \
+    -DCMAKE_USE_SYSTEM_LIBRARY_ZSTD=0 \
     -DENABLE_ACL=0 -DHAVE_ACL_LIBACL_H=0 \
     -DHAVE_SYS_ACL_H=0 \
 "
--

2.7.4

#2 lore raw

Hello Alexander,

Quoted 10 lines · Alexander Kanavin

This needs to go to master first probably?
Alex

Quoted 5 lines · Leon Woestenberg

cmake-native uses the system provided libraries due to:
<...>
This fix is to not depend on the system library and use the zstd library provided with cmake.

Yes most probably, but master did not build for me so was untestable for me within reasonable time span I had for this issue.

Regards,

Leon.

#3 lore raw
In reply to Alexander Kanavin
Quoted 1 line · Alexander Kanavin

This needs to go to master first probably?

Better for master might be moving zstd from meta-openembedded, and then DEPENDS on zstd-native?

IMHO moving target zstd to OE-core is already overdue, so this would not be solely for cmake.

Quoted 2 lines

Alex
...

cu
Adrian

#4 lore raw

Hello Adrian, all,

Quoted 10 lines · Adrian Bunk
Quoted 1 line · Alexander Kanavin

This needs to go to master first probably?

Better for master might be moving zstd from meta-openembedded, and then DEPENDS on zstd-native?

IMHO moving target zstd to OE-core is already overdue, so this would not be solely for cmake.

Good point, makes better sense then my proposed fix for cmake-native only.

More breakage of master against Ubuntu 16.04 (still supported by Yocto) that supports Adrian's suggestion:

| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:
In function ‘int lto_normalized_zstd_level()’:
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:120:36:
error: ‘ZSTD_maxCLevel’ was not declared in this scope

|    else if (level > ZSTD_maxCLevel ())
|                                     ^
|

../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:
In function ‘void lto_uncompression_zstd(lto_compression_stream*)’:
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:160:74:
error: ‘ZSTD_getFrameContentSize’ was not declared in this scope
| unsigned long long const rsize = ZSTD_getFrameContentSize (cursor,
size);
|
^
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:161:16:
error: ‘ZSTD_CONTENTSIZE_ERROR’ was not declared in this scope

|    if (rsize == ZSTD_CONTENTSIZE_ERROR)
|                 ^
|
../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:163:21:
error: ‘ZSTD_CONTENTSIZE_UNKNOWN’ was not declared in this scope
|    else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN)
|                      ^
| Makefile:1118: recipe for target 'lto-compress.o' failed
| make[1]: *** [lto-compress.o] Error 1
| make[1]: *** Waiting for unfinished jobs....

Did not deep-dive into this, nor test with Adrian's suggestion yet.

Regards,

Leon.

#5 lore raw
Quoted 11 lines · Leon Woestenberg

Hello Adrian, all,

Quoted 7 lines · Adrian Bunk
Quoted 1 line · Alexander Kanavin

This needs to go to master first probably?

Better for master might be moving zstd from meta-openembedded, and then DEPENDS on zstd-native?

Quoted 7 lines
Quoted 3 lines

IMHO moving target zstd to OE-core is already overdue, so this would not be solely for cmake.

Good point, makes better sense then my proposed fix for cmake-native only.

More breakage of master against Ubuntu 16.04 (still supported by Yocto) that supports Adrian's suggestion:

If cmake-native is auto detecting zstd support from the host (and therefore building non-deterministically depending on whether or not the host has zstd development libraries) then the build breakage is just a symptom. Adding zstd to oe-core would be a good approach to fixing the bug for master but not for older releases.

Quoted 26 lines
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c: In function ‘int lto_normalized_zstd_level()’:
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:120:36: error: ‘ZSTD_maxCLevel’ was not declared in this scope
|    else if (level > ZSTD_maxCLevel ())
|                                     ^
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c: In function ‘void lto_uncompression_zstd(lto_compression_stream*)’:
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:160:74: error: ‘ZSTD_getFrameContentSize’ was not declared in this scope
|    unsigned long long const rsize = ZSTD_getFrameContentSize (cursor, size);
|                                                                           ^
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:161:16: error: ‘ZSTD_CONTENTSIZE_ERROR’ was not declared in this scope
|    if (rsize == ZSTD_CONTENTSIZE_ERROR)
|                 ^
| ../../../../../../../work-shared/gcc-10.2.0-r0/gcc-10.2.0/gcc/lto-compress.c:163:21: error: ‘ZSTD_CONTENTSIZE_UNKNOWN’ was not declared in this scope
|    else if (rsize == ZSTD_CONTENTSIZE_UNKNOWN)
|                      ^
| Makefile:1118: recipe for target 'lto-compress.o' failed
| make[1]: *** [lto-compress.o] Error 1
| make[1]: *** Waiting for unfinished jobs....

Did not deep-dive into this, nor test with Adrian's suggestion yet.

Regards,

Leon.