Skip to contentSkip to conversation

package.bbclass: improve auto debug package splitting order

PATCH In review Hongxu Jia 2 participants 7 messages Sep 14, 2018 – Sep 17, 2018

lore
Share & tools
lore

Hongxu Jia

author
#0 lore raw

cover package.bbclass: improve auto debug package splitting order

Hi RP,

Is it necessary to split multiple sub dbg packages or only one dbg is allowed? If the former, this fix makes sense.

I know in oe-core, almost one dbg package per recipe, but in oe, 21 recipes have extra sub dbg package and 18 of them did not set NOAUTOPACKAGEDEBUG = "1".

So rather than modify them one by one tediously, how about to improve auto debug package splitting order? If ${PN}-dbg exists, split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not exist, still split them to first dbg package.

//Hongxu

The following changes since commit d180dc758710c7259d45eeb9304e7284a8fd8825:

oeqa/selftest/recipetool: fix non-determinism in cmake test (2018-09-13 16:28:26 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib hongxu/dbg-split
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=hongxu/dbg-split

Hongxu Jia (2):
package.bbclass: improve auto debug package splitting order selftest/package: improve case to test debug package splitting

 .../selftest-hardlink/selftest-hardlink.bb         |  4 +++-
 meta/classes/package.bbclass                       | 10 +++++++-
 meta/lib/oeqa/selftest/cases/package.py            | 27 ++++++++++++++++------
 3 files changed, 32 insertions(+), 9 deletions(-)
Signature · 1 line
2.7.4
author
#1 lore raw

PATCH 1/2 package.bbclass: improve auto debug package splitting order

Since `da5ec06 package: Add auto package splitting of .debug files' applied in oe-core, the first dbg package will have all debug files automatically.

If there are several sub dbg packages, except the first one, the others are empty.

Even though var-NOAUTOPACKAGEDEBUG is provided for manual control, but many recipes (such as in oe) requires to pick up a few debug files into sub dbg packages, and then split the rest to ${PN}-dbg. If set NOAUTOPACKAGEDEBUG = "1", they have to do the extra splitting manually.

So improve splitting order, if ${PN}-dbg exists, split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not exist, still split them to first dbg package.

---
 meta/classes/package.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
meta/classes/package.bbclass +9 -1
index d1e9138..cada46b 100644
@@ -1175,6 +1175,14 @@ python populate_packages () {
             if "/.debug/" in path or path.endswith("/.debug"):
                 debug.append(path)
 
+    pn_dbg = pn + "-dbg"
+    # If ${PN}-dbg does not exist, use first *-dbg to replace
+    if pn_dbg not in package_list:
+        for pkg in package_list:
+            if pkg.endswith("-dbg"):
+                pn_dbg = pkg
+                break
+
     for pkg in package_list:
         root = os.path.join(pkgdest, pkg)
         bb.utils.mkdirhier(root)
@@ -1188,7 +1196,7 @@ python populate_packages () {
         origfiles = filesvar.split()
         files, symlink_paths = files_from_filevars(origfiles)
 
-        if autodebug and pkg.endswith("-dbg"):
+        if autodebug and pn_dbg == pkg:
             files.extend(debug)
 
         for file in files:
Signature · 1 line
2.7.4
author
#2 lore raw

PATCH 2/2 selftest/package: improve case to test debug package splitting

In reply to Hongxu Jia

Tweak recipe selftest-hardlink
- Add ${PN}-bin, ${PN}-bin-dbg

Tweak case test_gdb_hardlink_debug
- Test gdb debug package separately, verify the debug package splitting works correctly.

[Test without commit `package.bbclass: improve auto debug package splitting order']

2018-09-14 15:58:06,233 - oe-selftest - INFO - ======================================================================
2018-09-14 15:58:06,233 - oe-selftest - INFO - FAIL: test_gdb_hardlink_debug (package.PackageTests)
2018-09-14 15:58:06,233 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-09-14 15:58:06,233 - oe-selftest - INFO - Traceback (most recent call last):
  File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 123, in test_gdb_hardlink_debug
    self._test_gdb_debug(packages, binaries)
  File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 162, in _test_gdb_debug
    self.fail('GDB %s failed' % binary)

AssertionError: GDB /usr/libexec/hello3 failed
[Test without commit `package.bbclass: improve auto debug package splitting order']

---
 .../selftest-hardlink/selftest-hardlink.bb         |  4 +++-
 meta/lib/oeqa/selftest/cases/package.py            | 27 ++++++++++++++++------
 2 files changed, 23 insertions(+), 8 deletions(-)
meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb +3 -1
index 842a977..7d544b4 100644
@@ -26,5 +26,7 @@ do_install () {
 }
 
 RDEPENDS_${PN}-gdb += "gdb"
-PACKAGES =+ "${PN}-gdb"
+PACKAGES =+ "${PN}-gdb ${PN}-bin ${PN}-bin-dbg"
 FILES_${PN}-gdb = "${bindir}/gdb.sh"
+FILES_${PN}-bin-dbg = "${bindir}/.debug/*"
+
meta/lib/oeqa/selftest/cases/package.py +20 -7
index 0a88dc2..d00cd49 100644
@@ -115,9 +115,25 @@ class PackageTests(OESelftestTestCase):
 
     # Verify gdb to read symbols from separated debug hardlink file correctly
     def test_gdb_hardlink_debug(self):
-        features = 'IMAGE_INSTALL_append = " selftest-hardlink"\n'
-        features += 'IMAGE_INSTALL_append = " selftest-hardlink-dbg"\n'
-        features += 'IMAGE_INSTALL_append = " selftest-hardlink-gdb"\n'
+        packages = ['selftest-hardlink',
+                    'selftest-hardlink-dbg',
+                    'selftest-hardlink-gdb']
+        binaries = ['/usr/libexec/hello3',
+                    '/usr/libexec/hello4']
+        self._test_gdb_debug(packages, binaries)
+
+        packages = ['selftest-hardlink-bin',
+                    'selftest-hardlink-bin-dbg',
+                    'selftest-hardlink-gdb']
+        binaries = ['/usr/bin/hello1',
+                    '/usr/bin/hello2']
+        self._test_gdb_debug(packages, binaries)
+
+    def _test_gdb_debug(self, packages, binaries):
+        features = ""
+        for pkg in packages:
+            features += 'IMAGE_INSTALL_append = " %s"\n' % pkg
+
         self.write_config(features)
         bitbake("core-image-minimal")
 
@@ -141,9 +157,6 @@ class PackageTests(OESelftestTestCase):
             return False
 
         with runqemu('core-image-minimal') as qemu:
-            for binary in ['/usr/bin/hello1',
-                           '/usr/bin/hello2',
-                           '/usr/libexec/hello3',
-                           '/usr/libexec/hello4']:
+            for binary in binaries:
                 if not gdbtest(qemu, binary):
                     self.fail('GDB %s failed' % binary)
Signature · 1 line
2.7.4
author
#3 lore raw

PATCH v2 2/2 selftest/package: improve case to test debug package splitting

Tweak recipe selftest-hardlink
- Add ${PN}-bin, ${PN}-bin-dbg

Tweak test_preserve_sparse_hardlinks
- s/selftest-hardlink/selftest-hardlink-bin/

Tweak case test_gdb_hardlink_debug
- Test gdb debug package separately, verify the debug package splitting works correctly.

[Test without commit `package.bbclass: improve auto debug package splitting order']

2018-09-14 15:58:06,233 - oe-selftest - INFO - ======================================================================
2018-09-14 15:58:06,233 - oe-selftest - INFO - FAIL: test_gdb_hardlink_debug (package.PackageTests)
2018-09-14 15:58:06,233 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-09-14 15:58:06,233 - oe-selftest - INFO - Traceback (most recent call last):
  File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 123, in test_gdb_hardlink_debug
    self._test_gdb_debug(packages, binaries)
  File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 162, in _test_gdb_debug
    self.fail('GDB %s failed' % binary)

AssertionError: GDB /usr/libexec/hello3 failed
[Test without commit `package.bbclass: improve auto debug package splitting order']

---
 .../selftest-hardlink/selftest-hardlink.bb         |  4 ++-
 meta/lib/oeqa/selftest/cases/package.py            | 31 +++++++++++++++-------
 2 files changed, 25 insertions(+), 10 deletions(-)
meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb +3 -1
index 842a977..7d544b4 100644
@@ -26,5 +26,7 @@ do_install () {
 }
 
 RDEPENDS_${PN}-gdb += "gdb"
-PACKAGES =+ "${PN}-gdb"
+PACKAGES =+ "${PN}-gdb ${PN}-bin ${PN}-bin-dbg"
 FILES_${PN}-gdb = "${bindir}/gdb.sh"
+FILES_${PN}-bin-dbg = "${bindir}/.debug/*"
+
meta/lib/oeqa/selftest/cases/package.py +22 -9
index 0a88dc2..d90297a 100644
@@ -98,10 +98,10 @@ class PackageTests(OESelftestTestCase):
         def checkfiles():
             # Recipe creates 4 hardlinked files, there is a copy in package/ and a copy in packages-split/
             # so expect 8 in total.
-            self.assertEqual(os.stat(dest + "/selftest-hardlink" + bindir + "/hello1").st_nlink, 8)
+            self.assertEqual(os.stat(dest + "/selftest-hardlink-bin" + bindir + "/hello1").st_nlink, 8)
 
             # Test a sparse file remains sparse
-            sparsestat = os.stat(dest + "/selftest-hardlink" + bindir + "/sparsetest")
+            sparsestat = os.stat(dest + "/selftest-hardlink-bin" + bindir + "/sparsetest")
             self.assertEqual(sparsestat.st_blocks, 0)
             self.assertEqual(sparsestat.st_size, 1048576)
 
@@ -115,9 +115,25 @@ class PackageTests(OESelftestTestCase):
 
     # Verify gdb to read symbols from separated debug hardlink file correctly
     def test_gdb_hardlink_debug(self):
-        features = 'IMAGE_INSTALL_append = " selftest-hardlink"\n'
-        features += 'IMAGE_INSTALL_append = " selftest-hardlink-dbg"\n'
-        features += 'IMAGE_INSTALL_append = " selftest-hardlink-gdb"\n'
+        packages = ['selftest-hardlink',
+                    'selftest-hardlink-dbg',
+                    'selftest-hardlink-gdb']
+        binaries = ['/usr/libexec/hello3',
+                    '/usr/libexec/hello4']
+        self._test_gdb_debug(packages, binaries)
+
+        packages = ['selftest-hardlink-bin',
+                    'selftest-hardlink-bin-dbg',
+                    'selftest-hardlink-gdb']
+        binaries = ['/usr/bin/hello1',
+                    '/usr/bin/hello2']
+        self._test_gdb_debug(packages, binaries)
+
+    def _test_gdb_debug(self, packages, binaries):
+        features = ""
+        for pkg in packages:
+            features += 'IMAGE_INSTALL_append = " %s"\n' % pkg
+
         self.write_config(features)
         bitbake("core-image-minimal")
 
@@ -141,9 +157,6 @@ class PackageTests(OESelftestTestCase):
             return False
 
         with runqemu('core-image-minimal') as qemu:
-            for binary in ['/usr/bin/hello1',
-                           '/usr/bin/hello2',
-                           '/usr/libexec/hello3',
-                           '/usr/libexec/hello4']:
+            for binary in binaries:
                 if not gdbtest(qemu, binary):
                     self.fail('GDB %s failed' % binary)
Signature · 1 line
2.7.4
author
#4 lore raw
In reply to Hongxu Jia
Quoted 1 line · Hongxu Jia

Hi RP,

Ping RP,

//Hongxu

Quoted 32 lines

Is it necessary to split multiple sub dbg packages or only one dbg is allowed? If the former, this fix makes sense.

I know in oe-core, almost one dbg package per recipe, but in oe, 21 recipes have extra sub dbg package and 18 of them did not set NOAUTOPACKAGEDEBUG = "1".

So rather than modify them one by one tediously, how about to improve auto debug package splitting order? If ${PN}-dbg exists, split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not exist, still split them to first dbg package.

//Hongxu

The following changes since commit d180dc758710c7259d45eeb9304e7284a8fd8825:

   oeqa/selftest/recipetool: fix non-determinism in cmake test (2018-09-13 16:28:26 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib hongxu/dbg-split
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=hongxu/dbg-split

Hongxu Jia (2):
package.bbclass: improve auto debug package splitting order selftest/package: improve case to test debug package splitting

  .../selftest-hardlink/selftest-hardlink.bb         |  4 +++-
  meta/classes/package.bbclass                       | 10 +++++++-
  meta/lib/oeqa/selftest/cases/package.py            | 27 ++++++++++++++++------
  3 files changed, 32 insertions(+), 9 deletions(-)
#5 lore raw
In reply to Hongxu Jia
Quoted 4 lines · Hongxu Jia

Hi RP,

Is it necessary to split multiple sub dbg packages or only one dbg is allowed? If the former, this fix makes sense.

Only one is recommended and the code only works properly in some cases for one. I'd like to be in a position where we could have that as the only thing we supported but its simply not practical in some cases.

The debug package dependency handling probably needs a rethink but that would be 2.7 development cycle, its too late for 2.6 now.

Quoted 8 lines

I know in oe-core, almost one dbg package per recipe, but in oe, 21 recipes have extra sub dbg package and 18 of them did not set NOAUTOPACKAGEDEBUG = "1".

So rather than modify them one by one tediously, how about to improve auto debug package splitting order? If ${PN}-dbg exists, split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not exist, still split them to first dbg package.

I don't really want to encourage multiple -dbg packages per recipe and I am a little concerned about making changes to the core packaging code with the release coming up. Fixing the recipes, although tedious may be safer/better rather than encouraging people to do things which aren't well supported.

Cheers,

Richard

author
#6 lore raw
Quoted 20 lines · On 2018年09月17日 20:06, [email protected] wrote:
Quoted 4 lines · Hongxu Jia

Hi RP,

Is it necessary to split multiple sub dbg packages or only one dbg is allowed? If the former, this fix makes sense.

Only one is recommended and the code only works properly in some cases for one. I'd like to be in a position where we could have that as the only thing we supported but its simply not practical in some cases.

The debug package dependency handling probably needs a rethink but that would be 2.7 development cycle, its too late for 2.6 now.

Quoted 8 lines

I know in oe-core, almost one dbg package per recipe, but in oe, 21 recipes have extra sub dbg package and 18 of them did not set NOAUTOPACKAGEDEBUG = "1".

So rather than modify them one by one tediously, how about to improve auto debug package splitting order? If ${PN}-dbg exists, split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not exist, still split them to first dbg package.

Quoted 5 lines

I don't really want to encourage multiple -dbg packages per recipe and I am a little concerned about making changes to the core packaging code with the release coming up. Fixing the recipes, although tedious may be safer/better rather than encouraging people to do things which aren't well supported.

Got it, I will fix affected recipes

Thanks for replying

//Hongxu

Quoted 3 lines

Cheers,

Richard