staging: Avoid staging the same binaries again and again
PATCH 1/1 staging: Avoid staging the same binaries again and again
It is possible that some or all of ${bindir}, ${base_bindir}, ${sbindir} and ${base_sbindir} might be the same directory. In particular, for micro it is often the case that all four are the same. There is clearly no point in staging the same directory four times, so let's keep track of which paths have already been staged and skip any later duplicates.
--- meta/classes/staging.bbclass | 24 ++++++++++++++++++++---- 1 files changed, 20 insertions(+), 4 deletions(-)
meta/classes/staging.bbclass +20 -4
@@ -21,6 +21,21 @@ sysroot_stage_dir() { fi } +sysroot_stage_dir_once() { + src="$1" + dest="$2" + + src_b64=`echo $src | base64 -w 0` + for d in $staged_dirs; do + if [ $d = $src_b64 ]; then + echo "$src has been staged already" + return + fi + done + sysroot_stage_dir $src $dest + staged_dirs="$staged_dirs $src_b64" +} + sysroot_stage_libdir() { src="$1" dest="$2"
@@ -34,10 +49,11 @@ sysroot_stage_dirs() { sysroot_stage_dir $from${includedir} $to${includedir} if [ "${BUILD_SYS}" = "${HOST_SYS}" ]; then - sysroot_stage_dir $from${bindir} $to${bindir} - sysroot_stage_dir $from${sbindir} $to${sbindir} - sysroot_stage_dir $from${base_bindir} $to${base_bindir} - sysroot_stage_dir $from${base_sbindir} $to${base_sbindir} + staged_dirs= + sysroot_stage_dir_once $from${bindir} $to${bindir} + sysroot_stage_dir_once $from${sbindir} $to${sbindir} + sysroot_stage_dir_once $from${base_bindir} $to${base_bindir} + sysroot_stage_dir_once $from${base_sbindir} $to${base_sbindir} sysroot_stage_dir $from${libexecdir} $to${libexecdir} sysroot_stage_dir $from${sysconfdir} $to${sysconfdir} sysroot_stage_dir $from${localstatedir} $to${localstatedir}
Signature · 1 line
1.7.9
Quoted 24 lines · Phil Blundell
It is possible that some or all of ${bindir}, ${base_bindir}, ${sbindir} and ${base_sbindir} might be the same directory. In particular, for micro it is often the case that all four are the same. There is clearly no point in staging the same directory four times, so let's keep track of which paths have already been staged and skip any later duplicates.
Signed-off-by Phil Blundell <[email protected]>--- meta/classes/staging.bbclass | 24 ++++++++++++++++++++---- 1 files changed, 20 insertions(+), 4 deletions(-)meta/classes/staging.bbclass +4
-0@@ -21,6 +21,21 @@ sysroot_stage_dir() { fi } +sysroot_stage_dir_once() { + src="$1" + dest="$2" +
meta/classes/staging.bbclass sysroot_stage_dir() { · line 37
Quoted 10 lines
meta/classes/staging.bbclass +10
-0+ src_b64=`echo $src | base64 -w 0` + for d in $staged_dirs; do + if [ $d = $src_b64 ]; then + echo "$src has been staged already" + return + fi + done + sysroot_stage_dir $src $dest + staged_dirs="$staged_dirs $src_b64" +}
Wouldn't testing whether $dest already exists work just as well?
Cheers,
Richard
Quoted 19 lines
meta/classes/staging.bbclass +5
-4sysroot_stage_libdir() { src="$1" dest="$2"@@ -34,10 +49,11 @@ sysroot_stage_dirs() { sysroot_stage_dir $from${includedir} $to${includedir} if [ "${BUILD_SYS}" = "${HOST_SYS}" ]; then - sysroot_stage_dir $from${bindir} $to${bindir} - sysroot_stage_dir $from${sbindir} $to${sbindir} - sysroot_stage_dir $from${base_bindir} $to${base_bindir} - sysroot_stage_dir $from${base_sbindir} $to${base_sbindir} + staged_dirs= + sysroot_stage_dir_once $from${bindir} $to${bindir} + sysroot_stage_dir_once $from${sbindir} $to${sbindir} + sysroot_stage_dir_once $from${base_bindir} $to${base_bindir} + sysroot_stage_dir_once $from${base_sbindir} $to${base_sbindir} sysroot_stage_dir $from${libexecdir} $to${libexecdir} sysroot_stage_dir $from${sysconfdir} $to${sysconfdir} sysroot_stage_dir $from${localstatedir} $to${localstatedir}
Quoted 1 line · Richard Purdie
Wouldn't testing whether $dest already exists work just as well?
I wasn't totally confident that $dest was guaranteed never to exist in advance (and that we'd never want to stage multiple $srcs into a single $dest). But yes, if both of those are true then I think testing for the existence of $dest would be fine.
p.
Quoted 5 lines · Phil Blundell
Quoted 1 line · Richard Purdie
Wouldn't testing whether $dest already exists work just as well?
I wasn't totally confident that $dest was guaranteed never to exist in advance
do_populate_sysroot[cleandirs] = "${SYSROOT_DESTDIR}"
(dest is ${SYSROOT_DESTDIR})
Quoted 2 lines
(and that we'd never want to stage multiple $srcs into a single
$dest).
sysroot_stage_* are symmetrical so I can't imagine this happening.
The main worry would be something happening before sysroot_stage_all. SYSROOT_PREPROCESS_FUNCS happen afterwards so there is at least a hook used in most cases that would avoid the issue.
I'm torn whether its better to be simple or less fragile in this case. Or simply do some tests (if ${bindir} != ${sbindir}) and so on.
Cheers,
Richard
Quoted 6 lines
But yes, if both of those are true then I think testing for the
existence of $dest would be fine.p.
Quoted 2 lines · Richard Purdie
I'm torn whether its better to be simple or less fragile in this case. Or simply do some tests (if ${bindir} != ${sbindir}) and so on.
The tests would get a bit ungainly, since you'd end up with 'if ! [ "${sbindir} = "${bindir}" -o "${sbindir}" = "${base_sbindir}" -o "${sbindir}" = "${base_sbindir}" ... ]' kind of things. But yes, that would certainly work as well.
p.
Quoted 8 lines · Richard Purdie
sysroot_stage_* are symmetrical so I can't imagine this happening.
The main worry would be something happening before
sysroot_stage_all.SYSROOT_PREPROCESS_FUNCShappen afterwards so there is at least a hook used in most cases that would avoid the issue.I'm torn whether its better to be simple or less fragile in this case. Or simply do some tests (if ${bindir} != ${sbindir}) and so on.
I'm not quite sure what the conclusion was from this previous discussion. Did you want me to redo the patch to work in some other way?
p.
Quoted 7 lines · Phil Blundell
Quoted 6 lines · Richard Purdie
sysroot_stage_* are symmetrical so I can't imagine this happening.
The main worry would be something happening before
sysroot_stage_all.SYSROOT_PREPROCESS_FUNCShappen afterwards so there is at least a hook used in most cases that would avoid the issue.
Quoted 6 lines
Quoted 2 lines
I'm torn whether its better to be simple or less fragile in this case. Or simply do some tests (if ${bindir} != ${sbindir}) and so on.
I'm not quite sure what the conclusion was from this previous discussion. Did you want me to redo the patch to work in some other way?
The release is now at -rc2 and this patch hasn't reached the threshold of things I'm willing to take. I've tried to at least get some of the patches you're posted in but I wish we'd had them a couple of weeks ago :/.
I also wish we could find some neater/simpler way of doing this since I'm not a fan of "obscuring" the code more than we have to. I'm not coming up with any great ways of doing it though.
Cheers,
Richard
Quoted 4 lines · Richard Purdie
The release is now at -rc2 and this patch hasn't reached the threshold of things I'm willing to take. I've tried to at least get some of the patches you're posted in but I wish we'd had them a couple of weeks ago :/.
Okay, fair enough. There's no particular rush with any of that stuff and it can perfectly well wait until your release is out of the way. Do you want me to resend those patches at some later date?
You might like to consider taking the eglibc patch from a few days ago since it is arguably a regression that eglibc has a build dependency on perl when it didn't use to. But of course this may not matter for the majority of yocto users so it might not be a problem in practice.
p.