Skip to contentSkip to conversation

toolchain/wrapper: adhere to SOURCE_DATE_EPOCH

PATCH New Yann E. MORIN 4 participants 12 messages Feb 21, 2017 – Mar 2, 2017

lore
Share & tools
lore
#0 lore raw

cover toolchain/wrapper: adhere to SOURCE_DATE_EPOCH

Hello All!

This series, for the next branch, re-introduces conformance with SOURCE_DATE_EPOCH, with a clean-room re-implementation.

The first patch is cherry-picked from master, to remove the previous implementation, while the second patch actually provides the new implementation.

Regards,
Yann E. MORIN.

The following changes since commit 9a7ac39605e1bf416db033b009e6c09f54f44fdf

keepalived: new package (2017-02-20 22:44:08 +0100)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 3c4084fc0c3cf518c6fe170bebd83124730eba00

toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc (2017-02-21 21:31:44 +0100)

----------------------------------------------------------------
Yann E. MORIN (2):
Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper" toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc

 toolchain/toolchain-wrapper.c | 109 +++++++++++++++++++++---------------------
 1 file changed, 55 insertions(+), 54 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

This reverts commit 76838f63412a30a358210e457dda4b79f7730624.

The commit referenced above explicitly states that the function was copied as-is from the gcc source code at the time. And indeed that is exactly how the function appeared in gcc in commit e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934.

However, our toolchain wrapper is "GPLv2 only", while the file this function was copied from is "GPLv3 or later". As such we can't include that function and still comply to both licenses.

Furthermore, the code is far from optimum.

Since this feature is not release-critical, revert it until we re-implement it.

Cc J?r?me Pouiller <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 toolchain/toolchain-wrapper.c | 72 +------------------------------------------
 1 file changed, 1 insertion(+), 71 deletions(-)
toolchain/toolchain-wrapper.c +1 -71
index eb4ee8d..100aa18 100644
@@ -22,15 +22,12 @@
 #include <unistd.h>
 #include <stdlib.h>
 #include <errno.h>
-#include <time.h>
 
 #ifdef BR_CCACHE
 static char ccache_path[PATH_MAX];
 #endif
 static char path[PATH_MAX];
 static char sysroot[PATH_MAX];
-static char source_time[sizeof("-D__TIME__=\"HH:MM:SS\"")];
-static char source_date[sizeof("-D__DATE__=\"MMM DD YYYY\"")];
 
 /**
  * GCC errors out with certain combinations of arguments (examples are
@@ -42,11 +39,8 @@ static char source_date[sizeof("-D__DATE__=\"MMM DD YYYY\"")];
  * 	-mfloat-abi=
  * 	-march=
  * 	-mcpu=
- * 	-D__TIME__=
- * 	-D__DATE__=
- * 	-Wno-builtin-macro-redefined
  */
-#define EXCLUSIVE_ARGS	6
+#define EXCLUSIVE_ARGS	3
 
 static char *predef_args[] = {
 #ifdef BR_CCACHE
@@ -163,47 +157,6 @@ static void check_unsafe_path(const char *arg,
 	}
 }
 
-/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
- * timestamp to replace embedded current dates to get reproducible
- * results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.
- */
-static time_t get_source_date_epoch()
-{
-	char *source_date_epoch;
-	long long epoch;
-	char *endptr;
-
-	source_date_epoch = getenv("SOURCE_DATE_EPOCH");
-	if (!source_date_epoch)
-		return (time_t) -1;
-
-	errno = 0;
-	epoch = strtoll(source_date_epoch, &endptr, 10);
-	if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
-			|| (errno != 0 && epoch == 0)) {
-		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
-				"strtoll: %s\n", strerror(errno));
-		exit(2);
-	}
-	if (endptr == source_date_epoch) {
-		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
-				"no digits were found: %s\n", endptr);
-		exit(2);
-	}
-	if (*endptr != '\0') {
-		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
-				"trailing garbage: %s\n", endptr);
-		exit(2);
-	}
-	if (epoch < 0) {
-		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
-				"value must be nonnegative: %lld \n", epoch);
-		exit(2);
-	}
-
-	return (time_t) epoch;
-}
-
 int main(int argc, char **argv)
 {
 	char **args, **cur, **exec_args;
@@ -214,7 +167,6 @@ int main(int argc, char **argv)
 	char *paranoid_wrapper;
 	int paranoid;
 	int ret, i, count = 0, debug;
-	time_t source_date_epoch;
 
 	/* Calculate the relative paths */
 	basename = strrchr(progpath, '/');
@@ -320,28 +272,6 @@ int main(int argc, char **argv)
 	}
 #endif /* ARCH || CPU */
 
-	source_date_epoch = get_source_date_epoch();
-	if (source_date_epoch != -1) {
-		struct tm *tm = localtime(&source_date_epoch);
-		if (!tm) {
-			perror("__FILE__: localtime");
-			return 3;
-		}
-		ret = strftime(source_time, sizeof(source_time), "-D__TIME__=\"%T\"", tm);
-		if (!ret) {
-			perror("__FILE__: overflow");
-			return 3;
-		}
-		*cur++ = source_time;
-		ret = strftime(source_date, sizeof(source_date), "-D__DATE__=\"%b %e %Y\"", tm);
-		if (!ret) {
-			perror("__FILE__: overflow");
-			return 3;
-		}
-		*cur++ = source_date;
-		*cur++ = "-Wno-builtin-macro-redefined";
-	}
-
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
 		paranoid = 1;
Signature · 1 line
2.7.4
#2 lore raw

PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

Hello,

Quoted 16 lines · Yann E. MORIN

This reverts commit 76838f63412a30a358210e457dda4b79f7730624.

The commit referenced above explicitly states that the function was copied as-is from the gcc source code at the time. And indeed that is exactly how the function appeared in gcc in commit e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934.

However, our toolchain wrapper is "GPLv2 only", while the file this function was copied from is "GPLv3 or later". As such we can't include that function and still comply to both licenses.

Furthermore, the code is far from optimum.

Since this feature is not release-critical, revert it until we re-implement it.

Quoted 7 lines
Cc J?r?me Pouiller <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>

This has already been applied in master:

https://git.buildroot.org/buildroot/commit/?id=1c8c0d91a431fda2c052072af87e0de35bd3b953

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#3 lore raw

PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

Thomas, All,

On 2017-02-21 21:47 +0100, Thomas Petazzoni spake thusly:

Quoted 2 lines · Yann E. MORIN
Quoted 1 line

This reverts commit 76838f63412a30a358210e457dda4b79f7730624.

[--SNIP--]

Quoted 2 lines

This has already been applied in master:
https://git.buildroot.org/buildroot/commit/?id=1c8c0d91a431fda2c052072af87e0de35bd3b953

I know.

And as the cover letter stated (and as the 'tag' in the subject tells), this is intended for next. ;-)

It is needed to revert this patch in next before applying the following patch.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#4 lore raw

PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

Hello,

Quoted 2 lines · Yann E. MORIN

It is needed to revert this patch in next before applying the following patch.

Since it has already been reverted in master, I believe we will wait for the release, and next to be merged in master before applying.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#5 lore raw

PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

Quoted 9 lines · Thomas Petazzoni

Hello,

Quoted 2 lines · Yann E. MORIN

It is needed to revert this patch in next before applying the following patch.

Since it has already been reverted in master, I believe we will wait for the release, and next to be merged in master before applying.

Alternatively, merge master into next now and apply patch 2 to next.

Is there any reason not to regularly merge master into next?

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#6 lore raw

PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Quoted 7 lines · Thomas Petazzoni

Hello,

Quoted 2 lines · Yann E. MORIN

It is needed to revert this patch in next before applying the following patch.

Quoted 6 lines
Quoted 2 lines

Since it has already been reverted in master, I believe we will wait for the release, and next to be merged in master before applying.

Alternatively, merge master into next now and apply patch 2 to next.

Is there any reason not to regularly merge master into next?

I have no problem doing that if it is useful, but I haven't personally tried that kind of back/forward merges before. Don't we then end up with a number of merge commits on next? What happens when next is then merged back to master?

Signature · 1 line
Bye, Peter Korsgaard
#7 lore raw

PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

Quoted 13 lines · Peter Korsgaard
Quoted 12 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Quoted 9 lines · Thomas Petazzoni

Hello,

Quoted 2 lines · Yann E. MORIN

It is needed to revert this patch in next before applying the following patch.

Since it has already been reverted in master, I believe we will wait for the release, and next to be merged in master before applying.

Quoted 7 lines
Quoted 3 lines

Alternatively, merge master into next now and apply patch 2 to next.

Is there any reason not to regularly merge master into next?

I have no problem doing that if it is useful, but I haven't personally tried that kind of back/forward merges before. Don't we then end up with a number of merge commits on next?

You indeed get merge commits on next, but is that a problem? With more merge
commits, it's IMHO easier to understand the history because each merge makes things line up in 'git log --graph'.

Quoted 2 lines

What happens when next is then merged
back to master?

It shouldn't make much of a difference. Except that you probably get less merge
conflicts.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#8 lore raw

PATCH 1/2 Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

In reply to Arnout Vandecappelle

Hello,

Quoted 1 line · Arnout Vandecappelle

Is there any reason not to regularly merge master into next?

Because there's usually not reason to do so, and this patch is also not a very good reason to do so. There's still plenty of things to fix in master, and master is really the focus. The release is only in 6 days, so it's not a lot of time to wait to have next merged back into master and the new development cycle opened.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#9 lore raw

PATCH 2/2 toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc

In reply to Yann E. MORIN

Starting with verison 7, gcc will automatically recognise and enforce the environment variable SOURCE_DATE_EPOCH, and will fake __DATE__ and __TIME__ accordingly, to produce reproducible builds (at least in regards to date and time).

However, older gcc versions (i.e. all so far) do not offer this feature.

So, we use our toolchain wrapper to force-feed __DATE__ and __TIME_ as macros, that will take precedence over those that gcc may compute itself. We compute them according to the specs:
https://reproducible-builds.org/specs/source-date-epoch/
https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html

Since we define macros otherwise internal to gcc, we have to tell it not to warn about that. The -Wno-builtin-macro-redefined flag was introduced in gcc-4.4.0, which is old enough now that we can consider that all toolchains we care about will support that flag.

Note however: the behaviour of gcc-7 when __DATE__ and __TIME__ are user-defined and SOURCE_DATE_EPOCH is also set is as yet unknown. Whether the user-defined macros take precedence over the internal computations in gcc, or whether gcc ignores the user-defined macros and enforce them from the value of SOURCE_DATE_EPOCH is of no consequence:
that would yield the exact same end result, to use the date and time set in SOURCE_DATE_EPOCH.

Cc J?r?me Pouiller <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Peter Korsgaard <[email protected]>
---
 toolchain/toolchain-wrapper.c | 75 +++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 73 insertions(+), 2 deletions(-)
toolchain/toolchain-wrapper.c +73 -2
index 100aa18..a7f01e7 100644
@@ -22,12 +22,19 @@
 #include <unistd.h>
 #include <stdlib.h>
 #include <errno.h>
+#include <time.h>
+#include <stdbool.h>
 
 #ifdef BR_CCACHE
 static char ccache_path[PATH_MAX];
 #endif
 static char path[PATH_MAX];
 static char sysroot[PATH_MAX];
+/* As would be defined by gcc:
+ *   https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html
+ * sizeof() on string literals includes the terminating \0. */
+static char _time_[sizeof("-D__TIME__=\"HH:MM:SS\"")];
+static char _date_[sizeof("-D__DATE__=\"MMM DD YYYY\"")];
 
 /**
  * GCC errors out with certain combinations of arguments (examples are
@@ -35,12 +42,15 @@ static char sysroot[PATH_MAX];
  * that we only pass the predefined one to the real compiler if the inverse
  * option isn't in the argument list.
  * This specifies the worst case number of extra arguments we might pass
- * Currently, we have:
+ * Currently, we may have:
  * 	-mfloat-abi=
  * 	-march=
  * 	-mcpu=
+ * 	-D__TIME__=
+ * 	-D__DATE__=
+ * 	-Wno-builtin-macro-redefined
  */
-#define EXCLUSIVE_ARGS	3
+#define EXCLUSIVE_ARGS	6
 
 static char *predef_args[] = {
 #ifdef BR_CCACHE
@@ -157,6 +167,60 @@ static void check_unsafe_path(const char *arg,
 	}
 }
 
+/* Returns false if SOURCE_DATE_EPOCH was not defined in the environment.
+ *
+ * Returns true if SOURCE_DATE_EPOCH is in the environment and represent
+ * a valid timestamp, in which case the timestamp is formatted into the
+ * global variables _source_ and _date_.
+ *
+ * Aborts if SOURCE_DATE_EPOCH was set in the environment but did not
+ * contain a valid timestamp.
+ *
+ * Valid values are defined in the spec:
+ *     https://reproducible-builds.org/specs/source-date-epoch/
+ * but we further restrict them to be positive or null.
+ */
+bool parse_source_date_epoch_from_env(void)
+{
+	char *epoch_env, *endptr;
+	time_t epoch;
+	struct tm epoch_tm;
+
+	if ((epoch_env = getenv("SOURCE_DATE_EPOCH")) == NULL)
+		return false;
+	errno = 0;
+	epoch = (time_t) strtoll(epoch_env, &endptr, 10);
+	/* We just need to test if it is incorrect, but we do not
+	 * care why it is incorrect.
+	 */
+	if ((errno != 0) || !*epoch_env || *endptr || (epoch < 0)) {
+		fprintf(stderr, "%s: invalid SOURCE_DATE_EPOCH='%s'\n",
+			program_invocation_short_name,
+			epoch_env);
+		exit(1);
+	}
+	tzset(); /* For localtime_r(), below. */
+	if (localtime_r(&epoch, &epoch_tm) == NULL) {
+		fprintf(stderr, "%s: cannot parse SOURCE_DATE_EPOCH=%s\n",
+				program_invocation_short_name,
+				getenv("SOURCE_DATE_EPOCH"));
+		exit(1);
+	}
+	if (!strftime(_time_, sizeof(_time_), "-D__TIME__=\"%T\"", &epoch_tm)) {
+		fprintf(stderr, "%s: cannot set time from SOURCE_DATE_EPOCH=%s\n",
+				program_invocation_short_name,
+				getenv("SOURCE_DATE_EPOCH"));
+		exit(1);
+	}
+	if (!strftime(_date_, sizeof(_date_), "-D__DATE__=\"%b %e %Y\"", &epoch_tm)) {
+		fprintf(stderr, "%s: cannot set date from SOURCE_DATE_EPOCH=%s\n",
+				program_invocation_short_name,
+				getenv("SOURCE_DATE_EPOCH"));
+		exit(1);
+	}
+	return true;
+}
+
 int main(int argc, char **argv)
 {
 	char **args, **cur, **exec_args;
@@ -272,6 +336,13 @@ int main(int argc, char **argv)
 	}
 #endif /* ARCH || CPU */
 
+	if (parse_source_date_epoch_from_env()) {
+		*cur++ = _time_;
+		*cur++ = _date_;
+		/* This has existed since gcc-4.4.0 */
+		*cur++ = "-Wno-builtin-macro-redefined";
+	}
+
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
 		paranoid = 1;
Signature · 1 line
2.7.4
#10 lore raw

PATCH 2/2 toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc

Hello,

Quoted 5 lines · Yann E. MORIN

+/* Returns false if SOURCE_DATE_EPOCH was not defined in the environment.
+ *
+ * Returns true if SOURCE_DATE_EPOCH is in the environment and represent
+ * a valid timestamp, in which case the timestamp is formatted into the
+ * global variables _source_ and _date_.

Shouldn't this be:

"global variables _date_ and _time_"

instead ?

If you confirm, then I can fixup this when applying.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#11 lore raw

PATCH 2/2 toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc

Thomas, All,

On 2017-03-01 23:12 +0100, Thomas Petazzoni spake thusly:

Quoted 7 lines · Yann E. MORIN
Quoted 5 lines

+/* Returns false if SOURCE_DATE_EPOCH was not defined in the environment.
+ *
+ * Returns true if SOURCE_DATE_EPOCH is in the environment and represent
+ * a valid timestamp, in which case the timestamp is formatted into the
+ * global variables _source_ and _date_.

Quoted 7 lines

Shouldn't this be:

"global variables _date_ and _time_"

instead ?

If you confirm, then I can fixup this when applying.

Yes, obviously it's _time_ and _date_.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'