lto: Add global LTO distro policy file
PATCH v5 1/11 lto: Add global LTO distro policy file
Distros which want to enable LTO can utilize this file, it only covers packages from OE-Core, other layers should include there own exclusion list for recipe which dont work with LTO
Document the needed changes in local.conf.extended
--- meta/conf/distro/include/lto.inc | 28 ++++++++++++++++++++++++++++ meta/conf/local.conf.sample.extended | 4 ++++ 2 files changed, 32 insertions(+) create mode 100644 meta/conf/distro/include/lto.inc
meta/conf/distro/include/lto.inc new file +28 -0
@@ -0,0 +1,28 @@ +# To enable LTO, add following in local.conf +# require conf/distro/include/lto.inc +# DISTRO_FEATURES_append = " lto" +# + +# Disable LTO for following packages +LTO_pn-glibc = "" +LTO_pn-gcc-runtime = "" +LTO_pn-libgcc-initial = "" +LTO_pn-libgcc = "" +LTO_pn-libpam = "" +LTO_pn-elfutils = "" +LTO_pn-perl = "" +LTO_pn-busybox = "" +LTO_pn-libxcrypt = "" +LTO_pn-curl = "" +LTO_pn-libcap = "" +LTO_pn-libproxy = "" +LTO_pn-libbsd = "" + +# Override it for additional or different options if needed e.g. +# with clang thin-lto might be better for compile speed +LTO ?= "-flto" + +SELECTED_OPTIMIZATION_append = "${@bb.utils.contains('DISTRO_FEATURES', 'lto', ' ${LTO}', '', d)}" +TARGET_LDFLAGS_append_class-target = "${@bb.utils.contains('DISTRO_FEATURES', 'lto', ' ${LTO}', '', d)}" + +SELECTED_OPTIMIZATION[vardeps] += "LTO"
meta/conf/local.conf.sample.extended +4 -0
@@ -417,3 +417,7 @@ # ERR_REPORT_UPLOAD_ALL = "1" # # INHERIT += "report-error" + +# Enable LTO Distro-wide +# require conf/distro/include/lto.inc +# DISTRO_FEATURES_append = " lto"
Signature · 1 line
2.29.2
PATCH v5 2/11 python3: Enable lto if its in DISTRO_FEATURES
python3 configure compiles on object file and then greps for strings in it for endianness for target ,when using LTO the .o files are not nomal ELF onjects so this test fails, since we are using --enable-lto to enable this here we dont need to inject extra paths via bitbake anyway therefore reset LTO variable for target
--- meta/recipes-devtools/python/python3_3.9.0.bb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
meta/recipes-devtools/python/python3_3.9.0.bb +5 -1
@@ -74,6 +74,9 @@ export CROSSPYTHONPATH="${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynl EXTRANATIVEPATH += "python3-native" +# LTO will be enabled via packageconfig depending upong distro features +LTO_class-target = "" + CACHED_CONFIGUREVARS = " \ ac_cv_file__dev_ptmx=yes \ ac_cv_file__dev_ptc=no \
@@ -88,7 +91,7 @@ def possibly_include_pgo(d): return '' -PACKAGECONFIG_class-target ??= "readline ${@possibly_include_pgo(d)} gdbm" +PACKAGECONFIG_class-target ??= "readline ${@possibly_include_pgo(d)} gdbm ${@bb.utils.filter('DISTRO_FEATURES', 'lto', d)}" PACKAGECONFIG_class-native ??= "readline gdbm" PACKAGECONFIG_class-nativesdk ??= "readline gdbm" PACKAGECONFIG[readline] = ",,readline"
@@ -96,6 +99,7 @@ PACKAGECONFIG[readline] = ",,readline" PACKAGECONFIG[pgo] = "--enable-optimizations,,qemu-native" PACKAGECONFIG[tk] = ",,tk" PACKAGECONFIG[gdbm] = ",,gdbm" +PACKAGECONFIG[lto] = "--with-lto,," do_configure_prepend () { mkdir -p ${B}/Modules
Signature · 1 line
2.29.2
PATCH v5 3/11 lto.inc: Add -ffat-lto-objects and -fuse-linker-plugin
In reply to Khem RajThis helps to improve LTO and ensure the libs can be linked with non-LTO objects too
--- meta/conf/distro/include/lto.inc | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
meta/conf/distro/include/lto.inc +10 -1
@@ -20,7 +20,16 @@ LTO_pn-libbsd = "" # Override it for additional or different options if needed e.g. # with clang thin-lto might be better for compile speed -LTO ?= "-flto" +# +# ffat-lto-objects +# object files that contain both the intermediate +# language and the object code. This makes them +# usable for both LTO linking and normal linking +# +# -fuse-linker-plugin +# ensures that libraries participate in LTO by supplying intermediate +# code from .a files to linker +LTO ?= "-flto -ffat-lto-objects -fuse-linker-plugin " SELECTED_OPTIMIZATION_append = "${@bb.utils.contains('DISTRO_FEATURES', 'lto', ' ${LTO}', '', d)}" TARGET_LDFLAGS_append_class-target = "${@bb.utils.contains('DISTRO_FEATURES', 'lto', ' ${LTO}', '', d)}"
Signature · 1 line
2.29.2
PATCH v5 4/11 lto: Introduce LTOEXTRA variable
In reply to Khem RajCertain packages may need additional flags to enable LTO, therefore LTOEXTRA can be used to pass those flags
Add -flto-partition=none for alsa-lib
--- meta/conf/distro/include/lto.inc | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
meta/conf/distro/include/lto.inc +9 -2
@@ -18,6 +18,13 @@ LTO_pn-libcap = "" LTO_pn-libproxy = "" LTO_pn-libbsd = "" +# Custom LTO flags +# disable partitioning/streaming algorithm since its uses ASM +# constructs not compatible with lto +LTOEXTRA_pn-alsa-lib = "-flto-partition=none" + +LTOEXTRA ?= "" + # Override it for additional or different options if needed e.g. # with clang thin-lto might be better for compile speed #
@@ -29,9 +36,9 @@ LTO_pn-libbsd = "" # -fuse-linker-plugin # ensures that libraries participate in LTO by supplying intermediate # code from .a files to linker -LTO ?= "-flto -ffat-lto-objects -fuse-linker-plugin " +LTO ?= "-flto -ffat-lto-objects -fuse-linker-plugin ${LTOEXTRA}" SELECTED_OPTIMIZATION_append = "${@bb.utils.contains('DISTRO_FEATURES', 'lto', ' ${LTO}', '', d)}" TARGET_LDFLAGS_append_class-target = "${@bb.utils.contains('DISTRO_FEATURES', 'lto', ' ${LTO}', '', d)}" -SELECTED_OPTIMIZATION[vardeps] += "LTO" +SELECTED_OPTIMIZATION[vardeps] += "LTO LTOEXTRA"
Signature · 1 line
2.29.2
PATCH v5 5/11 libaio: Disable LTO
In reply to Khem RajLibaio uses ASMs to implement symbol versioning which does not work with LTO
--- meta/conf/distro/include/lto.inc | 1 + 1 file changed, 1 insertion(+)
meta/conf/distro/include/lto.inc +1 -0
@@ -8,6 +8,7 @@ LTO_pn-glibc = "" LTO_pn-gcc-runtime = "" LTO_pn-libgcc-initial = "" LTO_pn-libgcc = "" +LTO_pn-libaio = "" LTO_pn-libpam = "" LTO_pn-elfutils = "" LTO_pn-perl = ""
Signature · 1 line
2.29.2
PATCH v5 6/11 weston: Fix linking with LTO
In reply to Khem Rajweston uses --no-undefined option during linking which does not work with LTO therefore override it with -z undefs when lto is enabled in distro
--- meta/recipes-graphics/wayland/weston_9.0.0.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-graphics/wayland/weston_9.0.0.bb +2 -0
@@ -26,6 +26,8 @@ REQUIRED_DISTRO_FEATURES = "opengl" DEPENDS = "libxkbcommon gdk-pixbuf pixman cairo glib-2.0" DEPENDS += "wayland wayland-protocols libinput virtual/egl pango wayland-native" +LDFLAGS += "${@bb.utils.contains('DISTRO_FEATURES', 'lto', '-Wl,-z,undefs', '', d)}" + WESTON_MAJOR_VERSION = "${@'.'.join(d.getVar('PV').split('.')[0:1])}" EXTRA_OEMESON += "-Dbackend-default=auto -Dbackend-rdp=false -Dpipewire=false"
Signature · 1 line
2.29.2
PATCH v5 7/11 lto.inc: Disable LTO for xserver-xorg
In reply to Khem RajIt ends up with undefined symbol when LTO is enabled
Fixestest/xi2/protocol-xiquerydevice.c:345: undefined reference to `GrabButton'
--- meta/conf/distro/include/lto.inc | 1 + 1 file changed, 1 insertion(+)
meta/conf/distro/include/lto.inc +1 -0
@@ -18,6 +18,7 @@ LTO_pn-curl = "" LTO_pn-libcap = "" LTO_pn-libproxy = "" LTO_pn-libbsd = "" +LTO_pn-xserver-xorg = "" # Custom LTO flags # disable partitioning/streaming algorithm since its uses ASM
Signature · 1 line
2.29.2
PATCH v5 8/11 gcc: Do no parameterize LTO configuration flags
In reply to Khem RajThese flags are redundant as LTO now enabled in GCC by default for sometime, Moreover, dropping LTO variable here avoids a conflict with global LTO variable being added for distrowide LTO support
--- meta/recipes-devtools/gcc/gcc-10.2.inc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
meta/recipes-devtools/gcc/gcc-10.2.inc +1 -2
@@ -81,14 +81,13 @@ S = "${TMPDIR}/work-shared/gcc-${PV}-${PR}/gcc-${PV}" FORTRAN = "" JAVA = "" -LTO = "--enable-lto" SSP ?= "--disable-libssp" SSP_mingw32 = "--enable-libssp" EXTRA_OECONF_BASE = "\ - ${LTO} \ ${SSP} \ --enable-libitm \ + --enable-lto \ --disable-bootstrap \ --with-system-zlib \ ${@'--with-linker-hash-style=${LINKER_HASH_STYLE}' if '${LINKER_HASH_STYLE}' else ''} \
Signature · 1 line
2.29.2
PATCH v5 9/11 puzzles: Check for excessive constant arguments
In reply to Khem RajFixes an issue found with LTO builds
--- ...Check-for-excessive-values-to-malloc.patch | 49 +++++++++++++++++++ meta/recipes-sato/puzzles/puzzles_git.bb | 1 + 2 files changed, 50 insertions(+) create mode 100644 meta/recipes-sato/puzzles/files/0001-malloc-Check-for-excessive-values-to-malloc.patch
meta/recipes-sato/puzzles/files/0001-malloc-Check-for-excessive-values-to-malloc.patch new file +49 -0
@@ -0,0 +1,49 @@ +From 1c01a5bc9ac7f8aaa484b1a8e0e74aa5f8899d0e Mon Sep 17 00:00:00 2001 +From: Khem Raj <[email protected]> +Date: Sun, 8 Nov 2020 11:17:59 -0800 +Subject: [PATCH] malloc: Check for excessive values to malloc + +with whole program optimizers like lto smalloc() +is inlined the excessive constant argument is propagated to +malloc() and ultimately triggers the warning. + +malloc.c:15:9: error: argument 1 range [18446744065119617024, 18446744073709551580] exceeds maximum object size 9223372036854775807 [-Werror=alloc-size-larger-than=] + +therefore add a check before excessive constant argument before calling +malloc + +Note that this will not happen with normal compile since they happen to +be in different translation units and compiler can not semantically +analyze as much + +Upstream-Status: Pending + +Signed-off-by: Khem Raj <[email protected]> +--- + malloc.c | 3 +++ + 1 file changed, 3 insertions(+) + +diff --git a/malloc.c b/malloc.c +index a7fa7c5..520377c 100644 +--- a/malloc.c ++++ b/malloc.c +@@ -2,6 +2,7 @@ + * malloc.c: safe wrappers around malloc, realloc, free, strdup + */ + ++#include <stdint.h> + #include <stdlib.h> + #include <string.h> + #include "puzzles.h" +@@ -12,6 +13,8 @@ + */ + void *smalloc(size_t size) { + void *p; ++ if (size > PTRDIFF_MAX) ++ fatal("exceeds maximum object size"); + p = malloc(size); + if (!p) + fatal("out of memory"); +-- +2.29.2 +
meta/recipes-sato/puzzles/puzzles_git.bb +1 -0
@@ -14,6 +14,7 @@ SRC_URI = "git://git.tartarus.org/simon/puzzles.git \ file://0001-Use-Wno-error-format-overflow-if-the-compiler-suppor.patch \ file://0001-pattern.c-Change-string-lenght-parameter-to-be-size_.patch \ file://fix-ki-uninitialized.patch \ + file://0001-malloc-Check-for-excessive-values-to-malloc.patch \ " UPSTREAM_CHECK_COMMITS = "1"
Signature · 1 line
2.29.2
PATCH v5 10/11 lto.inc: Disable LTO for perf
In reply to Khem Rajperf invokes kernel sources, and currently fails in multiple places for now therefore disable LTO for perf
--- meta/conf/distro/include/lto.inc | 1 + 1 file changed, 1 insertion(+)
meta/conf/distro/include/lto.inc +1 -0
@@ -18,6 +18,7 @@ LTO_pn-curl = "" LTO_pn-libcap = "" LTO_pn-libproxy = "" LTO_pn-libbsd = "" +LTO_pn-perf = "" LTO_pn-xserver-xorg = "" # Custom LTO flags
Signature · 1 line
2.29.2
PATCH v5 11/11 gcc: Handle duplicate names for variables
In reply to Khem RajPREFIXVAR is defined in cppdefaults.c as well which shows up as error when LTO is enabled to build gcc, therefore rename the one instance to PREFIXVAR1
--- ...4-handle-sysroot-support-for-nativesdk-gcc.patch | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
meta/recipes-devtools/gcc/gcc/0024-handle-sysroot-support-for-nativesdk-gcc.patch +8 -5
@@ -1,4 +1,4 @@ -From bb1f359e34649516e61305e9748534cce7e0ee70 Mon Sep 17 00:00:00 2001 +From 702daf2e9cb97337e0e594fcd435b1b61a917d14 Mon Sep 17 00:00:00 2001 From: Khem Raj <[email protected]> Date: Mon, 7 Dec 2015 23:39:54 +0000 Subject: [PATCH] handle sysroot support for nativesdk-gcc
@@ -321,7 +321,7 @@ index 9098ab044ab..bfad4ebe382 100644 str = update_path (ostr, p->component); free (ostr); diff --git a/gcc/prefix.c b/gcc/prefix.c -index 1a403e535bd..3257ed3cd3e 100644 +index 1a403e535bd..c26d07bde12 100644 --- a/gcc/prefix.c +++ b/gcc/prefix.c @@ -72,7 +72,9 @@ License along with GCC; see the file COPYING3. If not see
@@ -329,9 +329,9 @@ index 1a403e535bd..3257ed3cd3e 100644 #include "common/common-target.h" -static const char *std_prefix = PREFIX; -+static const char PREFIXVAR[4096] __attribute__ ((section (".gccrelocprefix"))) = PREFIX; ++char PREFIXVAR1[4096] __attribute__ ((section (".gccrelocprefix"))) = PREFIX; + -+static const char *std_prefix = PREFIXVAR; ++static const char *std_prefix = PREFIXVAR1; static const char *get_key_value (char *); static char *translate_name (char *);
@@ -340,7 +340,10 @@ index 1a403e535bd..3257ed3cd3e 100644 if (prefix == 0) - prefix = PREFIX; -+ prefix = PREFIXVAR; ++ prefix = PREFIXVAR1; /* We used to strip trailing DIR_SEPARATORs here, but that can sometimes yield a result with no separator when one was coded +-- +2.29.2 +
Signature · 1 line
2.29.2
Quoted 7 lines · Khem Raj
Distros which want to enable LTO can utilize this file, it only covers packages from OE-Core, other layers should include there own exclusion list for recipe which dont work with LTO
Document the needed changes in local.conf.extended
Maybe I missed something but I haven't seen a word why I should want link time optimization. And from the patches I see here it seems we are about to create another contributor's burden.
Andreas
Quoted 7 lines · Andreas Müller
Quoted 5 lines · Khem Raj
Distros which want to enable LTO can utilize this file, it only covers packages from OE-Core, other layers should include there own exclusion list for recipe which dont work with LTO
Quoted 6 lines
Quoted 2 lines
Document the needed changes in local.conf.extended
Maybe I missed something but I haven't seen a word why I should want link time optimization. And from the patches I see here it seems we are about to create another contributor's burden.
Thanks for your feedback. LTO is quite beneficial for the kind of images OE is targeting,
It results in smaller binaries and better optimizations as well. LTO has been around for a while
in GCC but it is becoming useful with gcc10+ and more distros e.g. Fedora/SuSE are also starting
to use LTO distro wide which means we will be able to reuse the upstream work that's ongoing in
different packages. You can see there are patches already floating to enable it for poky-tiny already
so OE userbase is already starting to use it. Moreover, its already default for many packages themselves
so you might already be using it at package level e.g. systemd
This is similar to what we did for enabling security flags. As you
can see, these changes are currently optional. This means they won't go into effect for
distros without them choosing to use it via DISTRO_FEATURES. Since it takes a bit to get
such changes stable, we might keep it disabled by default for the next release depending upon
how it pans out. Eventually, the plan is to enable it for poky/yocto distro.
As far as the contribution burden, I think it won't be a blocker until it becomes one of the tested configurations
on yocto auto-builder.
Feel free to express if you have more concerns.
Quoted 1 line
Andreas
Quoted 39 lines · Khem Raj
Quoted 7 lines · Andreas Müller
Quoted 5 lines · Khem Raj
Distros which want to enable LTO can utilize this file, it only covers packages from OE-Core, other layers should include there own exclusion list for recipe which dont work with LTO
Quoted 6 lines
Quoted 2 lines
Document the needed changes in local.conf.extended
Maybe I missed something but I haven't seen a word why I should want link time optimization. And from the patches I see here it seems we are about to create another contributor's burden.
Thanks for your feedback. LTO is quite beneficial for the kind of images OE is targeting,
It results in smaller binaries and better optimizations as well. LTO has been around for a while
in GCC but it is becoming useful with gcc10+ and more distros e.g. Fedora/SuSE are also starting
to use LTO distro wide which means we will be able to reuse the upstream work that's ongoing in
different packages. You can see there are patches already floating to enable it for poky-tiny already
so OE userbase is already starting to use it. Moreover, its already default for many packages themselves
so you might already be using it at package level e.g. systemdThis is similar to what we did for enabling security flags. As you
can see, these changes are currently optional. This means they won't go into effect for
distros without them choosing to use it viaDISTRO_FEATURES. Since it takes a bit to get
such changes stable, we might keep it disabled by default for the next release depending upon
how it pans out. Eventually, the plan is to enable it for poky/yocto distro.
Quoted 6 lines
As far as the contribution burden, I think it won't be a blocker until it becomes one of the tested configurations
on yocto auto-builder.Feel free to express if you have more concerns.
Thanks for explanation. At least I understand better now why you (and others?) are taking efforts on LTO support.
Andreas