Skip to contentSkip to conversation

Store package files list in results (branch yem/files-list)

PATCH New Yann E. MORIN 2 participants 6 messages Mar 31, 2019 – Apr 4, 2019

lore
Share & tools
lore
#0 lore raw

cover Store package files list in results (branch yem/files-list)

Hello All!

This small series against the autobuilder code, stores the packages files lists in autobuild results, so that it is easier to do a quick analysis about the origin of files in a build.

Regards,
Yann E. MORIN.

The following changes since commit e90dc8753f9deaca9ecb41a5874d7a559ee2557c

utils/daily-mail: sort overall summary in global e-mail by branch name (2019-03-15 15:19:36 +0100)

are available in the git repository at:

/home/ymorin/cache/upstream/buildroot/buildroot-test

for you to fetch changes up to 1414cf2855df133bd02c1a32d8ebe3e632f37734

[AUTOBUILD] autobuild-run: also copy packages files lists if they exist (2019-03-31 19:30:26 +0200)

----------------------------------------------------------------
Yann E. MORIN (2):
[AUTOBUILD] autobuild-run: add helper function to copy-if-exists [AUTOBUILD] autobuild-run: also copy packages files lists if they exist

 scripts/autobuild-run | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH 1/2 autobuild-run: add helper function to copy-if-exists

Cc Thomas Petazzoni <[email protected]>
---
 scripts/autobuild-run | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
scripts/autobuild-run +8 -6
index 3d2e99a..b65d8fb 100755
@@ -478,12 +478,14 @@ def send_results(result, **kwargs):
                     os.path.join(resultdir, "defconfig"))
     shutil.copyfile(os.path.join(outputdir, "branch"),
                     os.path.join(resultdir, "branch"))
-    if os.path.exists(os.path.join(outputdir, "build", "build-time.log")):
-        shutil.copyfile(os.path.join(outputdir, "build", "build-time.log"),
-                        os.path.join(resultdir, "build-time.log"))
-    if os.path.exists(os.path.join(outputdir, "legal-info", "manifest.csv")):
-        shutil.copyfile(os.path.join(outputdir, "legal-info", "manifest.csv"),
-                        os.path.join(resultdir, "licenses-manifest.csv"))
+
+    def copy_if_exists(dir, src, dst=None):
+        if os.path.exists(os.path.join(outputdir, d, src)):
+            shutil.copyfile(os.path.join(outputdir, d, src),
+                            os.path.join(resultdir, src if dst is None else dst))
+
+    copy_if_exists("build", "build-time.log")
+    copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv")
 
     subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \
                      os.path.join(resultdir, "gitid")],
Signature · 1 line
2.14.1
#2 lore raw

PATCH 2/2 autobuild-run: also copy packages files lists if they exist

In reply to Yann E. MORIN
Cc Thomas Petazzoni <[email protected]>
---
 scripts/autobuild-run | 3 +++
 1 file changed, 3 insertions(+)
scripts/autobuild-run +3 -0
index b65d8fb..3523772 100755
@@ -485,6 +485,9 @@ def send_results(result, **kwargs):
                             os.path.join(resultdir, src if dst is None else dst))
 
     copy_if_exists("build", "build-time.log")
+    copy_if_exists("build", "packages-file-list.txt")
+    copy_if_exists("build", "packages-file-list-host.txt")
+    copy_if_exists("build", "packages-file-list-staging.txt")
     copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv")
 
     subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \
Signature · 1 line
2.14.1
#3 lore raw

cover Store package files list in results (branch yem/files-list)

In reply to Yann E. MORIN

Hello Yann,

Quoted 3 lines · On Sun, 31 Mar 2019 19:48:20 +0200 "Yann E. MORIN" <[email protected]>…

Yann E. MORIN (2):
[AUTOBUILD] autobuild-run: add helper function to copy-if-exists [AUTOBUILD] autobuild-run: also copy packages files lists if they exist

Both applied. I hope those additional files will not be too big: we keep all those files for all build results, so it can add up quite a bit over time. The build results currently take 64 GB of disk space, and require 1303600 files.

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

cover Store package files list in results (branch yem/files-list)

Thomas, All,

On 2019-04-03 23:13 +0200, Thomas Petazzoni spake thusly:

Quoted 11 lines
Quoted 3 lines · On Sun, 31 Mar 2019 19:48:20 +0200 "Yann E. MORIN" <[email protected]>…

Yann E. MORIN (2):
[AUTOBUILD] autobuild-run: add helper function to copy-if-exists [AUTOBUILD] autobuild-run: also copy packages files lists if they exist

Both applied. I hope those additional files will not be too big: we keep all those files for all build results, so it can add up quite a bit over time. The build results currently take 64 GB of disk space, and require 1303600 files.

I would expect those files to be of about the same size as the build.log snippet we keep.

If it turns out to be a problem, then we can restrict that to only failed builds.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw

cover Store package files list in results (branch yem/files-list)

Quoted 5 lines · On Wed, 3 Apr 2019 23:30:12 +0200 "Yann E. MORIN" <[email protected]>…

I would expect those files to be of about the same size as the build.log snippet we keep.

If it turns out to be a problem, then we can restrict that to only failed builds.

It is true that for failed builds, not much of the information we keep is interesting. Perhaps just the .config/defconfig are interesting. The build.log and other files are mainly useful to investigate issues.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com