pkg-infra: put optional step_user def and assignment in one block
PATCH v2 1/2 pkg-infra: put optional step_user def and assignment in one block
---
Like the manual says in 'The .mk file' section ...
Changes v1 -> v2
- Add acked-by
--- package/pkg-generic.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/pkg-generic.mk +1 -1
@@ -56,11 +56,11 @@ endef GLOBAL_INSTRUMENTATION_HOOKS += step_time # User-supplied script +ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),) define step_user @$(foreach user_hook, $(BR2_INSTRUMENTATION_SCRIPTS), \ $(EXTRA_ENV) $(user_hook) "$(1)" "$(2)" "$(3)"$(sep)) endef -ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),) GLOBAL_INSTRUMENTATION_HOOKS += step_user endif
Signature · 1 line
1.9.1
PATCH v2 2/2 pkg-infra: make timing of steps optional
Commit 17d4eb1e0261793a9f89e4a2253602c7ab926d2e added a hook to log timing of steps to a build-time.log file, which provides data for the "graph-build" target for examining build time stats. If one uses buildroot on a daily basis as part of a build system, then its conceivable that there might be long periods of time between "make clean" ops. So the log file continues to grow. This patch makes the accumulation of the timing data optional, to avoid having a silent endlessly growing log in the build directory.
---
Changes v1 -> v2:
- Switch test to positive logic, per feedback from Yann Morin.
--- Config.in | 6 ++++++ package/pkg-generic.mk | 2 ++ 2 files changed, 8 insertions(+)
Config.in +6 -0
@@ -574,6 +574,12 @@ config BR2_GLOBAL_PATCH_DIR Otherwise, if the directory <global-patch-dir>/<packagename> exists, then all *.patch files in the directory will be applied. +config BR2_GATHER_BUILD_TIME_STATS + bool "Gather build time statistics" + help + Record the start and end time of each step in the build process, so + that buildroot can generate graphs of the build times. + endmenu source "toolchain/Config.in"
package/pkg-generic.mk +2 -0
@@ -48,12 +48,14 @@ endef # Actual steps hooks # Time steps +ifeq ($(BR2_GATHER_BUILD_TIME_STATS),y) define step_time printf "%s:%-5.5s:%-20.20s: %s\n" \ "$$(date +%s)" "$(1)" "$(2)" "$(3)" \ >>"$(BUILD_DIR)/build-time.log" endef GLOBAL_INSTRUMENTATION_HOOKS += step_time +endif # User-supplied script ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),)
Signature · 1 line
1.9.1
PATCH v2 2/2 pkg-infra: make timing of steps optional
Danomi, All,
On 2014-09-14 21:31 -0400, Danomi Manchego spake thusly:
Quoted 8 lines
Commit
17d4eb1e0261793a9f89e4a2253602c7ab926d2eadded a hook to log timing of steps to a build-time.log file, which provides data for the "graph-build" target for examining build time stats. If one uses buildroot on a daily basis as part of a build system, then its conceivable that there might be long periods of time between "make clean" ops. So the log file continues to grow. This patch makes the accumulation of the timing data optional, to avoid having a silent endlessly growing log in the build directory.
Quoted 6 lines
Signed-off-by Danomi Manchego <[email protected]>---
Changes v1 -> v2:
- Switch test to positive logic, per feedback from Yann Morin.
Thanks.
However, I'm still not convinced.
- if the problem is the size of that file: each step adds around 120 bytes (~60 for start, same for end); so a 1MiB would hold around 8000 steps, and most packages have less than 8 steps, so that is a *complete* build (download, extract, patch, configure, build, install-target, install-staging) of about 1000 packages. 1MiB is very small compared to the size of those 1000 packages, and even when compared to the smallest system (busybox based), that's still one order of magnitude less than the sources of busybox, and you'd get to build it 8000 times in a row...
Lets assume I made a mistake of one order of magnitude in those calculations, and 1MiB is only 800 steps; well, 10MiB are those 8000 steps, is on-par with Buildroot source tree, and it is still very small for today's storage.
- if the problem is the accuracy of the content of that file:
Buildroot never guaranteed (and will probably never guarantee) that a partial build is coherent. Only with a build from scratch do you get a coherent output. That's valid for target/ , staging/ and host/ . It is is also valid for graphs/ .
- the overhead of the tracing? Well, that's mere milliseconds at
worse, for each step. Running: date +%s.%N; date +%s.%N
consistently gives me from 1.6ms to 2.5ms between the two dates. OK, after 8000 steps, that about 20 seconds...
So, I wonder what problem this patch is trying to solve.
Regards,
Yann E. MORIN.
Quoted 48 lines
--- Config.in | 6 ++++++ package/pkg-generic.mk | 2 ++ 2 files changed, 8 insertions(+)Config.in +6
-0@@ -574,6 +574,12 @@ config BR2_GLOBAL_PATCH_DIR Otherwise, if the directory <global-patch-dir>/<packagename> exists, then all *.patch files in the directory will be applied. +config BR2_GATHER_BUILD_TIME_STATS + bool "Gather build time statistics" + help + Record the start and end time of each step in the build process, so + that buildroot can generate graphs of the build times. + endmenu source "toolchain/Config.in"package/pkg-generic.mk +2
-0@@ -48,12 +48,14 @@ endef # Actual steps hooks # Time steps +ifeq ($(BR2_GATHER_BUILD_TIME_STATS),y) define step_time printf "%s:%-5.5s:%-20.20s: %s\n" \ "$$(date +%s)" "$(1)" "$(2)" "$(3)" \ >>"$(BUILD_DIR)/build-time.log" endef GLOBAL_INSTRUMENTATION_HOOKS += step_time +endif # User-supplied script ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),)Signature · 6 lines
1.9.1 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 2/2 pkg-infra: make timing of steps optional
Yann,
Quoted 43 lines · Yann E. MORIN
Danomi, All,
On 2014-09-14 21:31 -0400, Danomi Manchego spake thusly:
Quoted 8 lines
Commit
17d4eb1e0261793a9f89e4a2253602c7ab926d2eadded a hook to log timing of steps to a build-time.log file, which provides data for the "graph-build" target for examining build time stats. If one uses buildroot on a daily basis as part of a build system, then its conceivable that there might be long periods of time between "make clean" ops. So the log file continues to grow. This patch makes the accumulation of the timing data optional, to avoid having a silent endlessly growing log in the build directory.Quoted 6 lines
Signed-off-by Danomi Manchego <[email protected]>---
Changes v1 -> v2:
- Switch test to positive logic, per feedback from Yann Morin.Thanks.
However, I'm still not convinced.
- if the problem is the size of that file: each step adds around 120 bytes (~60 for start, same for end); so a 1MiB would hold around 8000 steps, and most packages have less than 8 steps, so that is a *complete* build (download, extract, patch, configure, build, install-target, install-staging) of about 1000 packages. 1MiB is very small compared to the size of those 1000 packages, and even when compared to the smallest system (busybox based), that's still one order of magnitude less than the sources of busybox, and you'd get to build it 8000 times in a row...
Lets assume I made a mistake of one order of magnitude in those calculations, and 1MiB is only 800 steps; well, 10MiB are those 8000 steps, is on-par with Buildroot source tree, and it is still very small for today's storage.
- if the problem is the accuracy of the content of that file:
Buildroot never guaranteed (and will probably never guarantee) that a partial build is coherent. Only with a build from scratch do you get a coherent output. That's valid for target/ , staging/ and host/ . It is is also valid for graphs/ .
Quoted 6 lines
- the overhead of the tracing? Well, that's mere milliseconds at
worse, for each step. Running: date +%s.%N; date +%s.%N
consistently gives me from 1.6ms to 2.5ms between the two dates. OK, after 8000 steps, that about 20 seconds...So, I wonder what problem this patch is trying to solve.
We use buildroot as the core or the build systems of our projects. While I personally have the habit of periodically doing a build from scratch, I have observed that most of my coworkers *never* do a build from scratch, for the entire 1-2 year project, unless a defconfig change breaks things so completely that it can't be ignored. It's for their sake that I thought it would be good to patch away the file that grows forever.
If there's not appetite for this change, I suppose that I could keep it as a local customization.
Danomi -
PATCH v2 2/2 pkg-infra: make timing of steps optional
Danomi, All,
On 2014-09-21 22:18 -0400, Danomi Manchego spake thusly:
Quoted 49 lines · Yann E. MORIN
Quoted 48 lines
Danomi, All,
On 2014-09-14 21:31 -0400, Danomi Manchego spake thusly:
Quoted 8 lines
Commit
17d4eb1e0261793a9f89e4a2253602c7ab926d2eadded a hook to log timing of steps to a build-time.log file, which provides data for the "graph-build" target for examining build time stats. If one uses buildroot on a daily basis as part of a build system, then its conceivable that there might be long periods of time between "make clean" ops. So the log file continues to grow. This patch makes the accumulation of the timing data optional, to avoid having a silent endlessly growing log in the build directory.Quoted 6 lines
Signed-off-by Danomi Manchego <[email protected]>---
Changes v1 -> v2:
- Switch test to positive logic, per feedback from Yann Morin.Thanks.
However, I'm still not convinced.
- if the problem is the size of that file: each step adds around 120 bytes (~60 for start, same for end); so a 1MiB would hold around 8000 steps, and most packages have less than 8 steps, so that is a *complete* build (download, extract, patch, configure, build, install-target, install-staging) of about 1000 packages. 1MiB is very small compared to the size of those 1000 packages, and even when compared to the smallest system (busybox based), that's still one order of magnitude less than the sources of busybox, and you'd get to build it 8000 times in a row...
Lets assume I made a mistake of one order of magnitude in those calculations, and 1MiB is only 800 steps; well, 10MiB are those 8000 steps, is on-par with Buildroot source tree, and it is still very small for today's storage.
- if the problem is the accuracy of the content of that file:
Buildroot never guaranteed (and will probably never guarantee) that a partial build is coherent. Only with a build from scratch do you get a coherent output. That's valid for target/ , staging/ and host/ . It is is also valid for graphs/ .- the overhead of the tracing? Well, that's mere milliseconds at
worse, for each step. Running: date +%s.%N; date +%s.%N
consistently gives me from 1.6ms to 2.5ms between the two dates. OK, after 8000 steps, that about 20 seconds...
Quoted 6 lines
Quoted 1 line
So, I wonder what problem this patch is trying to solve.
We use buildroot as the core or the build systems of our projects. While I personally have the habit of periodically doing a build from scratch, I have observed that most of my coworkers *never* do a build from scratch, for the entire 1-2 year project,
I was afraid you'd say that! ;-)
Quoted 4 lines
unless a defconfig
change breaks things so completely that it can't be ignored. It's for their sake that I thought it would be good to patch away the file that grows forever.
What about: rm output/build/build-time.log ? ;-]
Quoted 2 lines
If there's not appetite for this change, I suppose that I could keep it as a local customization.
I'm not too fond of this. A maintainer may argue otherwise.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 2/2 pkg-infra: make timing of steps optional
In reply to Danomi ManchegoDear Danomi Manchego,
Quoted 9 lines · Danomi Manchego
Commit
17d4eb1e0261793a9f89e4a2253602c7ab926d2eadded a hook to log timing of steps to a build-time.log file, which provides data for the "graph-build" target for examining build time stats. If one uses buildroot on a daily basis as part of a build system, then its conceivable that there might be long periods of time between "make clean" ops. So the log file continues to grow. This patch makes the accumulation of the timing data optional, to avoid having a silent endlessly growing log in the build directory.Signed-off-by Danomi Manchego <[email protected]>
We had some discussion at the Buildroot meeting today about this patch, and the general opinion is the one already expressed by Yann. The amount of data accumulated in this build-time.log file is very small, so even if you do repeated builds without make clean for several days, the file size will remain really small compared to the overall size of a standard Buildroot build tree. Therefore, we don't think adding yet another configuration option for this is really useful, and consequently we have marked your patch as Rejected in patchwork.
Thanks,
Thomas Petazzoni
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
Thomas Petazzoni
AppliedPATCH v2 1/2 pkg-infra: put optional step_user def and assignment in one block
In reply to Danomi ManchegoDear Danomi Manchego,
Quoted 2 lines · Danomi Manchego
Signed-off-by Danomi Manchego <[email protected]>Acked-by "Yann E. MORIN" <[email protected]>
Applied, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com