toolchain/wrapper: adhere to SOURCE_DATE_EPOCH
cover toolchain/wrapper: adhere to SOURCE_DATE_EPOCH
Hello All!
This series, for the next branch, re-introduces conformance with SOURCE_DATE_EPOCH, with a clean-room re-implementation.
The first patch is cherry-picked from master, to remove the previous implementation, while the second patch actually provides the new implementation.
Regards,
Yann E. MORIN.
The following changes since commit 9a7ac39605e1bf416db033b009e6c09f54f44fdf
keepalived: new package (2017-02-20 22:44:08 +0100)
are available in the git repository at:
git://git.buildroot.org/~ymorin/git/buildroot.git
for you to fetch changes up to 3c4084fc0c3cf518c6fe170bebd83124730eba00
toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc (2017-02-21 21:31:44 +0100)
----------------------------------------------------------------
Yann E. MORIN (2):
Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper" toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc
toolchain/toolchain-wrapper.c | 109 +++++++++++++++++++++--------------------- 1 file changed, 55 insertions(+), 54 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
This reverts commit 76838f63412a30a358210e457dda4b79f7730624.
The commit referenced above explicitly states that the function was copied as-is from the gcc source code at the time. And indeed that is exactly how the function appeared in gcc in commit e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934.
However, our toolchain wrapper is "GPLv2 only", while the file this function was copied from is "GPLv3 or later". As such we can't include that function and still comply to both licenses.
Furthermore, the code is far from optimum.
Since this feature is not release-critical, revert it until we re-implement it.
--- toolchain/toolchain-wrapper.c | 72 +------------------------------------------ 1 file changed, 1 insertion(+), 71 deletions(-)
toolchain/toolchain-wrapper.c +1 -71
@@ -22,15 +22,12 @@#include <unistd.h> #include <stdlib.h> #include <errno.h> -#include <time.h> #ifdef BR_CCACHE static char ccache_path[PATH_MAX ];#endif static char path[PATH_MAX ];static char sysroot[PATH_MAX ]; -static char source_time[sizeof ("-D__TIME__=\"HH:MM:SS\"" )]; -static char source_date[sizeof ("-D__DATE__=\"MMM DD YYYY\"" )];/** * GCC errors out with certain combinations of arguments (examples are
@@ -42,11 +39,8 @@ static char source_date[sizeof("-D__DATE__=\"MMM DD YYYY\"")];* -mfloat-abi= * -march= * -mcpu= -* -D__TIME__= -* -D__DATE__= -* -Wno-builtin-macro-redefined */ -#define EXCLUSIVE_ARGS 6 +#define EXCLUSIVE_ARGS 3 static char *predef_args[] = {#ifdef BR_CCACHE
@@ -163,47 +157,6 @@ static void check_unsafe_path(const char *arg, } } -/* Read SOURCE_DATE_EPOCH from environment to have a deterministic -* timestamp to replace embedded current dates to get reproducible -* results. Returns -1 if SOURCE_DATE_EPOCH is not defined. -*/ -static time_t get_source_date_epoch () -{ -char *source_date_epoch; -long long epoch; -char *endptr; - - source_date_epoch =getenv ("SOURCE_DATE_EPOCH" ); -if (!source_date_epoch) -return (time_t) -1 ; - - errno =0 ; - epoch =strtoll (source_date_epoch, &endptr,10 ); -if ((errno ==ERANGE && (epoch ==LLONG_MAX || epoch ==LLONG_MIN )) - || (errno !=0 && epoch ==0 )) { -fprintf (stderr,"environment variable $SOURCE_DATE_EPOCH: " -"strtoll: %s\n" ,strerror (errno)); -exit (2 ); - } -if (endptr == source_date_epoch) { -fprintf (stderr,"environment variable $SOURCE_DATE_EPOCH: " -"no digits were found: %s\n" , endptr); -exit (2 ); - } -if (*endptr !='\0' ) { -fprintf (stderr,"environment variable $SOURCE_DATE_EPOCH: " -"trailing garbage: %s\n" , endptr); -exit (2 ); - } -if (epoch <0 ) { -fprintf (stderr,"environment variable $SOURCE_DATE_EPOCH: " -"value must be nonnegative: %lld \n" , epoch); -exit (2 ); - } - -return (time_t ) epoch; -} -int main (int argc,char **argv) {char **args, **cur, **exec_args;
@@ -214,7 +167,6 @@ int main(int argc, char **argv)char *paranoid_wrapper;int paranoid;int ret, i, count =0 , debug; -time_t source_date_epoch;/* Calculate the relative paths */ basename =strrchr (progpath,'/' );
@@ -320,28 +272,6 @@ int main(int argc, char **argv) }#endif /* ARCH || CPU */ - source_date_epoch =get_source_date_epoch (); -if (source_date_epoch !=-1 ) { -struct tm *tm =localtime (&source_date_epoch); -if (!tm) { -perror ("__FILE__: localtime" ); -return 3 ; - } - ret =strftime (source_time,sizeof (source_time),"-D__TIME__=\"%T\"" , tm); -if (!ret) { -perror ("__FILE__: overflow" ); -return 3 ; - } - *cur++ = source_time; - ret =strftime (source_date,sizeof (source_date),"-D__DATE__=\"%b %e %Y\"" , tm); -if (!ret) { -perror ("__FILE__: overflow" ); -return 3 ; - } - *cur++ = source_date; - *cur++ ="-Wno-builtin-macro-redefined" ; - } - paranoid_wrapper =getenv ("BR_COMPILER_PARANOID_UNSAFE_PATH" );if (paranoid_wrapper &&strlen (paranoid_wrapper) >0 ) paranoid =1 ;
Signature · 1 line
2.7.4
PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
Hello,
Quoted 16 lines · Yann E. MORIN
This reverts commit
76838f63412a30a358210e457dda4b79f7730624.The commit referenced above explicitly states that the function was copied as-is from the gcc source code at the time. And indeed that is exactly how the function appeared in gcc in commit
e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934.However, our toolchain wrapper is "GPLv2 only", while the file this function was copied from is "GPLv3 or later". As such we can't include that function and still comply to both licenses.
Furthermore, the code is far from optimum.
Since this feature is not release-critical, revert it until we re-implement it.
Quoted 7 lines
Signed-off-by "Yann E. MORIN" <[email protected]>Cc J?r?me Pouiller <[email protected]>Cc Samuel Martin <[email protected]>Cc Peter Korsgaard <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Signed-off-by Peter Korsgaard <[email protected]>
This has already been applied in master:
https://git.buildroot.org/buildroot/commit/?id=1c8c0d91a431fda2c052072af87e0de35bd3b953
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
Thomas, All,
On 2017-02-21 21:47 +0100, Thomas Petazzoni spake thusly:
Quoted 2 lines · Yann E. MORIN
Quoted 1 line
This reverts commit
76838f63412a30a358210e457dda4b79f7730624.
[--SNIP--]
Quoted 2 lines
This has already been applied in master:
https://git.buildroot.org/buildroot/commit/?id=1c8c0d91a431fda2c052072af87e0de35bd3b953
I know.
And as the cover letter stated (and as the 'tag' in the subject tells), this is intended for next. ;-)
It is needed to revert this patch in next before applying the following patch.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
Hello,
Quoted 2 lines · Yann E. MORIN
It is needed to revert this patch in next before applying the following patch.
Since it has already been reverted in master, I believe we will wait for the release, and next to be merged in master before applying.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
Quoted 9 lines · Thomas Petazzoni
Hello,
Quoted 2 lines · Yann E. MORIN
It is needed to revert this patch in next before applying the following patch.
Since it has already been reverted in master, I believe we will wait for the release, and next to be merged in master before applying.
Alternatively, merge master into next now and apply patch 2 to next.
Is there any reason not to regularly merge master into next?
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Quoted 7 lines · Thomas Petazzoni
Hello,
Quoted 2 lines · Yann E. MORIN
It is needed to revert this patch in next before applying the following patch.
Quoted 6 lines
Quoted 2 lines
Since it has already been reverted in master, I believe we will wait for the release, and next to be merged in master before applying.
Alternatively, merge master into next now and apply patch 2 to next.
Is there any reason not to regularly merge master into next?
I have no problem doing that if it is useful, but I haven't personally tried that kind of back/forward merges before. Don't we then end up with a number of merge commits on next? What happens when next is then merged back to master?
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
Quoted 13 lines · Peter Korsgaard
Quoted 12 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Quoted 9 lines · Thomas Petazzoni
Hello,
Quoted 2 lines · Yann E. MORIN
It is needed to revert this patch in next before applying the following patch.
Since it has already been reverted in master, I believe we will wait for the release, and next to be merged in master before applying.
Quoted 7 lines
Quoted 3 lines
Alternatively, merge master into next now and apply patch 2 to next.
Is there any reason not to regularly merge master into next?
I have no problem doing that if it is useful, but I haven't personally tried that kind of back/forward merges before. Don't we then end up with a number of merge commits on next?
You indeed get merge commits on next, but is that a problem? With more merge
commits, it's IMHO easier to understand the history because each merge makes things line up in 'git log --graph'.
Quoted 2 lines
What happens when next is then merged
back to master?
It shouldn't make much of a difference. Except that you probably get less merge
conflicts.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
In reply to Arnout VandecappelleHello,
Quoted 1 line · Arnout Vandecappelle
Is there any reason not to regularly merge master into next?
Because there's usually not reason to do so, and this patch is also not a very good reason to do so. There's still plenty of things to fix in master, and master is really the focus. The release is only in 6 days, so it's not a lot of time to wait to have next merged back into master and the new development cycle opened.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 2/2 toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc
In reply to Yann E. MORINStarting with verison 7, gcc will automatically recognise and enforce the environment variable SOURCE_DATE_EPOCH, and will fake __DATE__ and __TIME__ accordingly, to produce reproducible builds (at least in regards to date and time).
However, older gcc versions (i.e. all so far) do not offer this feature.
So, we use our toolchain wrapper to force-feed __DATE__ and __TIME_ as macros, that will take precedence over those that gcc may compute itself. We compute them according to the specs:
https://reproducible-builds.org/specs/source-date-epoch/
https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html
Since we define macros otherwise internal to gcc, we have to tell it not to warn about that. The -Wno-builtin-macro-redefined flag was introduced in gcc-4.4.0, which is old enough now that we can consider that all toolchains we care about will support that flag.
Note however: the behaviour of gcc-7 when __DATE__ and __TIME__ are user-defined and SOURCE_DATE_EPOCH is also set is as yet unknown. Whether the user-defined macros take precedence over the internal computations in gcc, or whether gcc ignores the user-defined macros and enforce them from the value of SOURCE_DATE_EPOCH is of no consequence:
that would yield the exact same end result, to use the date and time set in SOURCE_DATE_EPOCH.
--- toolchain/toolchain-wrapper.c | 75 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 73 insertions(+), 2 deletions(-)
toolchain/toolchain-wrapper.c +73 -2
@@ -22,12 +22,19 @@#include <unistd.h> #include <stdlib.h> #include <errno.h> +#include <time.h> +#include <stdbool.h> #ifdef BR_CCACHE static char ccache_path[PATH_MAX ];#endif static char path[PATH_MAX ];static char sysroot[PATH_MAX ]; +/* As would be defined by gcc: +* https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html +* sizeof() on string literals includes the terminating \0. */ +static char _time_[sizeof ("-D__TIME__=\"HH:MM:SS\"" )]; +static char _date_[sizeof ("-D__DATE__=\"MMM DD YYYY\"" )];/** * GCC errors out with certain combinations of arguments (examples are
@@ -35,12 +42,15 @@ static char sysroot[PATH_MAX];* that we only pass the predefined one to the real compiler if the inverse * option isn't in the argument list. * This specifies the worst case number of extra arguments we might pass -* Currently, we have: +* Currently, we may have: * -mfloat-abi= * -march= * -mcpu= +* -D__TIME__= +* -D__DATE__= +* -Wno-builtin-macro-redefined */ -#define EXCLUSIVE_ARGS 3 +#define EXCLUSIVE_ARGS 6 static char *predef_args[] = {#ifdef BR_CCACHE
@@ -157,6 +167,60 @@ static void check_unsafe_path(const char *arg, } } +/* Returns false if SOURCE_DATE_EPOCH was not defined in the environment. +* +* Returns true if SOURCE_DATE_EPOCH is in the environment and represent +* a valid timestamp, in which case the timestamp is formatted into the +* global variables _source_ and _date_. +* +* Aborts if SOURCE_DATE_EPOCH was set in the environment but did not +* contain a valid timestamp. +* +* Valid values are defined in the spec: +* https://reproducible-builds.org/specs/source-date-epoch/ +* but we further restrict them to be positive or null. +*/ +bool parse_source_date_epoch_from_env (void ) +{ +char *epoch_env, *endptr; +time_t epoch; +struct tm epoch_tm; + +if ((epoch_env =getenv ("SOURCE_DATE_EPOCH" )) ==NULL ) +return false; + errno =0 ; + epoch = (time_t )strtoll (epoch_env, &endptr,10 ); +/* We just need to test if it is incorrect, but we do not +* care why it is incorrect. +*/ +if ((errno !=0 ) || !*epoch_env || *endptr || (epoch <0 )) { +fprintf (stderr,"%s: invalid SOURCE_DATE_EPOCH='%s'\n" , + program_invocation_short_name, + epoch_env); +exit (1 ); + } +tzset ();/* For localtime_r(), below. */ +if (localtime_r (&epoch, &epoch_tm) ==NULL ) { +fprintf (stderr,"%s: cannot parse SOURCE_DATE_EPOCH=%s\n" , + program_invocation_short_name, +getenv ("SOURCE_DATE_EPOCH" )); +exit (1 ); + } +if (!strftime (_time_,sizeof (_time_),"-D__TIME__=\"%T\"" , &epoch_tm)) { +fprintf (stderr,"%s: cannot set time from SOURCE_DATE_EPOCH=%s\n" , + program_invocation_short_name, +getenv ("SOURCE_DATE_EPOCH" )); +exit (1 ); + } +if (!strftime (_date_,sizeof (_date_),"-D__DATE__=\"%b %e %Y\"" , &epoch_tm)) { +fprintf (stderr,"%s: cannot set date from SOURCE_DATE_EPOCH=%s\n" , + program_invocation_short_name, +getenv ("SOURCE_DATE_EPOCH" )); +exit (1 ); + } +return true; +} +int main (int argc,char **argv) {char **args, **cur, **exec_args;
@@ -272,6 +336,13 @@ int main(int argc, char **argv) }#endif /* ARCH || CPU */ +if (parse_source_date_epoch_from_env ()) { + *cur++ = _time_; + *cur++ = _date_; +/* This has existed since gcc-4.4.0 */ + *cur++ ="-Wno-builtin-macro-redefined" ; + } + paranoid_wrapper =getenv ("BR_COMPILER_PARANOID_UNSAFE_PATH" );if (paranoid_wrapper &&strlen (paranoid_wrapper) >0 ) paranoid =1 ;
Signature · 1 line
2.7.4
PATCH 2/2 toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc
Hello,
Quoted 5 lines · Yann E. MORIN
+/* Returns false if
SOURCE_DATE_EPOCHwas not defined in the environment.
+ *
+ * Returns true ifSOURCE_DATE_EPOCHis in the environment and represent
+ * a valid timestamp, in which case the timestamp is formatted into the
+ * global variables _source_ and _date_.
Shouldn't this be:
"global variables _date_ and _time_"
instead ?
If you confirm, then I can fixup this when applying.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 2/2 toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc
Thomas, All,
On 2017-03-01 23:12 +0100, Thomas Petazzoni spake thusly:
Quoted 7 lines · Yann E. MORIN
Quoted 5 lines
+/* Returns false if
SOURCE_DATE_EPOCHwas not defined in the environment.
+ *
+ * Returns true ifSOURCE_DATE_EPOCHis in the environment and represent
+ * a valid timestamp, in which case the timestamp is formatted into the
+ * global variables _source_ and _date_.
Quoted 7 lines
Shouldn't this be:
"global variables _date_ and _time_"
instead ?
If you confirm, then I can fixup this when applying.
Yes, obviously it's _time_ and _date_.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'