Extensible SDK size reduction

PATCH Superseded Randy Witt 2 participants 10 messages Jan 6, 2016 – Jan 11, 2016

lore

Share & tools

lore

Randy Witt

author

2016-01-06 22:15 UTC #0 lore raw

cover Extensible SDK size reduction

This removes some unneeded items from the extensible sdk to reduce size by ~300MB.

The following changes since commit f8185ff9f8ac697733b41f023cb3a4b98b2f823f:

bitbake: ast: Add filename/lineno to mapped functions (2016-01-06 15:27:35 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib rewitt/extsdksize
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/extsdksize

Randy Witt (2):
copy_buildsystem.py: Allow substring matches for excluded_targets
populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

 meta/classes/populate_sdk_ext.bbclass | 6 ++++++
 meta/lib/oe/copy_buildsystem.py       | 7 +++++--
 2 files changed, 11 insertions(+), 2 deletions(-)

Signature · 1 line

2.5.0

Randy Witt

author

2016-01-06 22:15 UTC #1 lore raw

PATCH 1/2 copy_buildsystem.py: Allow substring matches for excluded_targets

This is so that the excluded_targets passed to prune_lockedsigs can take a partial form such as nativesdk to filter based on partial matches.

Signed-off-by Randy Witt <[email protected]>

---
 meta/lib/oe/copy_buildsystem.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

meta/lib/oe/copy_buildsystem.py +5 -2

index 15af4eb..9d8f3a6 100644
@@ -83,8 +83,11 @@ def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs, pruned_output)
             for line in infile:
                 if invalue:
                     if line.endswith('\\\n'):
-                        splitval = line.strip().split(':')
-                        if splitval[1] in allowed_tasks and not splitval[0] in excluded_targets:
+                        target, task = line.strip().split(':')[:2]
+                        badmatch = ([x for x in excluded_targets.split()
+                                     if x in target])
+
+                        if task in allowed_tasks and not badmatch:
                             f.write(line)
                     else:
                         f.write(line)

Signature · 1 line

2.5.0

Randy Witt

author

2016-01-06 22:15 UTC #2 lore raw

PATCH 2/2 populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

The nativesdk and crosssdk targets were only getting pulled into the extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA. However the nativesdk and crosssdk targets are not needed for the extensible sdk to be used.

This patch therefore removes them since it reduces the size of the extensible sdks that include sstate by ~300MB.

Signed-off-by Randy Witt <[email protected]>

---
 meta/classes/populate_sdk_ext.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

meta/classes/populate_sdk_ext.bbclass +6 -0

index c30181a..60467b9 100644
@@ -173,6 +173,12 @@ python copy_buildsystem () {
     # Filter the locked signatures file to just the sstate tasks we are interested in
     allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa', 'do_deploy']
     excluded_targets = d.getVar('SDK_TARGETS', True)
+
+    # Since buildtools is included in the sdk, the nativesdk items and
+    # crosssdk end up in the tasks in locked-sigs.inc. However, the nativesdk
+    # and crosssdk items aren't needed for the extensible sdk to function so
+    # remove them from the list.
+    excluded_targets += 'nativesdk crosssdk'
     lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
     oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
                                          excluded_targets,

Signature · 1 line

2.5.0

Paul Eggleton

2016-01-06 22:42 UTC #3 lore raw

Re: [PATCH 1/2] copy_buildsystem.py: Allow substring matches for excluded_targets

In reply to Randy Witt

Hi Randy,

Quoted 22 lines · On Wed, 06 Jan 2016 14:15:49 Randy Witt wrote:

This is so that the excluded_targets passed to prune_lockedsigs can take a partial form such as nativesdk to filter based on partial matches.
Signed-off-by Randy Witt <[email protected]>
---
 meta/lib/oe/copy_buildsystem.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
(quoted diff) +0 -0
b/meta/lib/oe/copy_buildsystem.py index 15af4eb..9d8f3a6 100644
meta/lib/oe/copy_buildsystem.py +0 -0
@@ -83,8 +83,11 @@ def prune_lockedsigs(allowed_tasks, excluded_targets,
lockedsigs, pruned_output) for line in infile:
                 if invalue:
                     if line.endswith('\\\n'):
meta/lib/oe/copy_buildsystem.py +0 -2
-                        splitval = line.strip().split(':')
-                        if splitval[1] in allowed_tasks and not splitval[0]
in excluded_targets: + target, task =
line.strip().split(':')[:2] + badmatch = ([x for x
in excluded_targets.split() + if x in
target])

meta/lib/oe/copy_buildsystem.py def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs, pruned_output) · line 90

Quoted 2 lines

meta/lib/oe/copy_buildsystem.py +2 -0
+
+                        if task in allowed_tasks and not badmatch:

Actually Mark and I have just noticed excluded_targets is being treated as a list in the original code here but we're passing in a string. I think maybe expecting the caller to supply a list would be a better approach (so we would call .split() on the value of SDK_TARGETS before passing it in). Could you take care of that in a separate patch before doing this?

Thanks,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Randy Witt

author

2016-01-07 19:24 UTC #4 lore raw

v2 cover Extensible SDK size reduction

In reply to Randy Witt

This removes some unneeded items from the extensible sdk to reduce size by ~300MB.

Includes an additional patch

The following changes since commit 0f3ad7c77aa0f8bb7641af1886e1420cdfa8419f:

scripts/yocto-layer: Avoids duplication of "meta-" prefix (2016-01-07 13:52:21 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib rewitt/extsdksize
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/extsdksize

Randy Witt (3):
populate_sdk_ext: Pass excluded_targets as a list to prune_lockedsigs copy_buildsystem.py: Allow substring matches for excluded_targets
populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

 meta/classes/populate_sdk_ext.bbclass | 8 +++++++-
 meta/lib/oe/copy_buildsystem.py       | 7 +++++--
 2 files changed, 12 insertions(+), 3 deletions(-)

Signature · 1 line

2.5.0

Randy Witt

author

2016-01-07 19:24 UTC #5 lore raw

PATCH v2 1/3 populate_sdk_ext: Pass excluded_targets as a list to prune_lockedsigs

prune_lockedsigs expects excluded_targets to be a list, whereas previously it was passed in as a string.

Signed-off-by Randy Witt <[email protected]>

---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

meta/classes/populate_sdk_ext.bbclass +1 -1

index c30181a..5c99bd0 100644
@@ -175,7 +175,7 @@ python copy_buildsystem () {
     excluded_targets = d.getVar('SDK_TARGETS', True)
     lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
     oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
-                                         excluded_targets,
+                                         excluded_targets.split(),
                                          sigfile,
                                          lockedsigs_pruned)

Signature · 1 line

2.5.0

Randy Witt

author

2016-01-07 19:24 UTC #6 lore raw

PATCH v2 2/3 copy_buildsystem.py: Allow substring matches for excluded_targets

This is so that the excluded_targets passed to prune_lockedsigs can take a partial form such as nativesdk to filter based on partial matches.

Signed-off-by Randy Witt <[email protected]>

---
 meta/lib/oe/copy_buildsystem.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

meta/lib/oe/copy_buildsystem.py +5 -2

index 15af4eb..83ab8c5 100644
@@ -83,8 +83,11 @@ def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs, pruned_output)
             for line in infile:
                 if invalue:
                     if line.endswith('\\\n'):
-                        splitval = line.strip().split(':')
-                        if splitval[1] in allowed_tasks and not splitval[0] in excluded_targets:
+                        target, task = line.strip().split(':')[:2]
+                        badmatch = ([x for x in excluded_targets
+                                     if x in target])
+
+                        if task in allowed_tasks and not badmatch:
                             f.write(line)
                     else:
                         f.write(line)

Signature · 1 line

2.5.0

Randy Witt

author

2016-01-07 19:24 UTC #7 lore raw

PATCH v2 3/3 populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

In reply to Randy Witt

This patch therefore removes them since it reduces the size of the extensible sdks that include sstate by ~300MB.

Signed-off-by Randy Witt <[email protected]>

---
 meta/classes/populate_sdk_ext.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

meta/classes/populate_sdk_ext.bbclass +6 -0

index 5c99bd0..ef014b8 100644
@@ -173,6 +173,12 @@ python copy_buildsystem () {
     # Filter the locked signatures file to just the sstate tasks we are interested in
     allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa', 'do_deploy']
     excluded_targets = d.getVar('SDK_TARGETS', True)
+
+    # Since buildtools is included in the sdk, the nativesdk items and
+    # crosssdk end up in the tasks in locked-sigs.inc. However, the nativesdk
+    # and crosssdk items aren't needed for the extensible sdk to function so
+    # remove them from the list.
+    excluded_targets += 'nativesdk crosssdk'
     lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
     oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
                                          excluded_targets.split(),

Signature · 1 line

2.5.0

Paul Eggleton

2016-01-10 22:19 UTC #8 lore raw

Re: [PATCH 3/3] populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

Quoted 29 lines · On Thu, 07 Jan 2016 11:24:27 Randy Witt wrote:

The nativesdk and crosssdk targets were only getting pulled into the extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA. However the nativesdk and crosssdk targets are not needed for the extensible sdk to be used.
This patch therefore removes them since it reduces the size of the extensible sdks that include sstate by ~300MB.
Signed-off-by Randy Witt <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
(quoted diff) +0 -0
b/meta/classes/populate_sdk_ext.bbclass index 5c99bd0..ef014b8 100644
meta/classes/populate_sdk_ext.bbclass +0 -0
@@ -173,6 +173,12 @@ python copy_buildsystem () {
     # Filter the locked signatures file to just the sstate tasks we are
 interested in allowed_tasks = ['do_populate_lic', 'do_populate_sysroot',
 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm',
'do_package_write_deb', 'do_package_qa', 'do_deploy'] excluded_targets = d.getVar('SDK_TARGETS', True)
meta/classes/populate_sdk_ext.bbclass +3 -0
+
+    # Since buildtools is included in the sdk, the nativesdk items and
+    # crosssdk end up in the tasks in locked-sigs.inc. However, the
nativesdk
+ # and crosssdk items aren't needed for the extensible sdk to
function so

meta/classes/populate_sdk_ext.bbclass python copy_buildsystem () { · line 181

Quoted 2 lines

meta/classes/populate_sdk_ext.bbclass +2 -0
+    # remove them from the list.
+    excluded_targets += 'nativesdk crosssdk'

You need a leading space here to make this work (or alternatively split the excluded_targets value in the previous patch when you first set it, and add the items as list items here).

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Paul Eggleton

2016-01-11 05:56 UTC #9 lore raw

Re: [PATCH 3/3] populate_sdk_ext: Don't put nativesdk and crosssdk items in ext sdk

Quoted 23 lines · On Mon, 11 Jan 2016 11:19:37 Paul Eggleton wrote:

Quoted 22 lines · On Thu, 07 Jan 2016 11:24:27 Randy Witt wrote:
The nativesdk and crosssdk targets were only getting pulled into the extensible sdk due to buildtools dependencies being in BB_TASKDEPDATA. However the nativesdk and crosssdk targets are not needed for the extensible sdk to be used.
This patch therefore removes them since it reduces the size of the extensible sdks that include sstate by ~300MB.
Signed-off-by Randy Witt <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
(quoted diff) +0 -0
b/meta/classes/populate_sdk_ext.bbclass index 5c99bd0..ef014b8 100644
meta/classes/populate_sdk_ext.bbclass +0 -0
@@ -173,6 +173,12 @@ python copy_buildsystem () {
 
     # Filter the locked signatures file to just the sstate tasks we are
 

Quoted 16 lines

meta/classes/populate_sdk_ext.bbclass python copy_buildsystem () { · line 181
Quoted 12 lines
meta/classes/populate_sdk_ext.bbclass +0 -0
 interested in allowed_tasks = ['do_populate_lic', 'do_populate_sysroot',
 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm',
'do_package_write_deb', 'do_package_qa', 'do_deploy'] excluded_targets = d.getVar('SDK_TARGETS', True)
meta/classes/populate_sdk_ext.bbclass +3 -0
+
+    # Since buildtools is included in the sdk, the nativesdk items and
+    # crosssdk end up in the tasks in locked-sigs.inc. However, the
nativesdk
+ # and crosssdk items aren't needed for the extensible sdk to
function so
meta/classes/populate_sdk_ext.bbclass +2 -0
+    # remove them from the list.
+    excluded_targets += 'nativesdk crosssdk'
You need a leading space here to make this work (or alternatively split the excluded_targets value in the previous patch when you first set it, and add the items as list items here).

Actually I think I have an alternative fix which we need to do anyway - create a separate task for the recursive dependencies, and then the issue goes away. I'll send out some patches to do this once I've finished testing.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre