Skip to content

Overwritten file detection, improvements to file listing logic

Thomas Petazzoni buildroot 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest11 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11package/pkg-generic.mk: use $(PKG)_NAME in step_pkg_size_afterpackage/pkg-generic.mk+1 -10 replies
2/11package/pkg-generic.mk: drop useless $(1) argument in step_pkg_size_{before, after}package/pkg-generic.mk+19 -210 replies
3/11package/pkg-generic.mk: introduce final 'install' steppackage/pkg-generic.mk+18 -20 replies
4/11package/pkg-generic.mk: create directories upfront in the configure steppackage/pkg-generic.mk package/pkg-utils.mk+1 -50 replies
5/11package/pkg-generic.mk: rework pkg_size logic with the "installed" steppackage/pkg-generic.mk+14 -281 reply
5/11package/pkg-generic.mk: rework pkg_size logic with the "installed" step+0 -00 replies
6/11package/pkg-generic.mk: exclude the staging sub-directorypackage/pkg-generic.mk+2 -21 reply
7/11package/pkg-generic.mk: move pkg_size_{before, after} and check_bin_arch functionspackage/pkg-generic.mk+33 -331 reply
8/11package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIRpackage/pkg-generic.mk+25 -05 replies
9/11support/testing/infra: add log_file_path() functionsupport/testing/infra/__init__.py+11 -21 reply
10/11support/testing/tests: add test for check_bin_arch.gitlab-ci.yml support/testing/tests/core/br2-external/detect-bad-arch/Config.in support/testing/tests/core/br2-external/detect-bad-arch/external.desc +4 more+42 -01 reply
11/11support/testing/tests: add test for file overwrite detection.gitlab-ci.yml support/testing/tests/core/br2-external/detect-overwrite/Config.in support/testing/tests/core/br2-external/detect-overwrite/external.desc +4 more+76 -01 reply

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [Buildroot] [PATCH 00/11] Overwritten file detection, improvements to file listing logic

Hello,

The initial starting point for this series was that I wanted to detect when a package overwrites files installed by another package, which is something that is not compatible with per-package directory support. I indeed have a complex build configuration which I'm converting to use TLP, and this build configuration does lots of overwrites, which I want to detect and fix.

While doing this overwrite detection, I realized that our existing logic to detect which files are installed by each package is not
reliable: if a target package installs something in $(HOST_DIR), those files installed in $(HOST_DIR) are not tracked. Getting this list of files correct is also needed to properly support the -reinstall,-rebuild,-reconfigure targets in a per-package configuration (which this series does not address yet).

So this series goes on like this (I'm only highlighting the main commits, not all preparation commits and details)

- We create a new "install" step in PATCH 3. It is executed after all installation steps have executed. As of PATCH 3, it does nothing except creating a stamp file, but it will allow us to do some logic once a package is finished installing. Indeed, so far we had no place where we could run some logic after all installation steps of a package have completed.

- In PATCH 5, we rework the pkg_size logic that detects files installed by each package to rely on the new "install" step. Instead of capturing the list of files "before" at the beginning of install-target, install-staging, install-images, install-host, and the list of files "after" at the end of these steps, we capture the list of files "before" at the beginning of the configure step, and the list of files "after" during the new "install" step.

- In PATCH 8, we introduce the logic to detect files overwritten by other packages, using a simple md5sum hash.

- In PATCH 9-11, we add test cases for the "check binary architecture" functionality and the "overwritten files detection" functionality.

This has received some basic testing so far, I'm definitely interested in getting feedback from people, simply by testing if it works for them.

All these changes are also available on the Git branch at:

https://github.com/tpetazzoni/buildroot/tree/detect-overwrite

Best regards,

Thomas Petazzoni

Thomas Petazzoni (11):
package/pkg-generic.mk: use $(PKG)_NAME in step_pkg_size_after package/pkg-generic.mk: drop useless $(1) argument in

    step_pkg_size_{before,after}

package/pkg-generic.mk: introduce final 'install' step package/pkg-generic.mk: create directories upfront in the configure
step
package/pkg-generic.mk: rework pkg_size logic with the "installed"
step
package/pkg-generic.mk: exclude the staging sub-directory package/pkg-generic.mk: move pkg_size_{before,after} and
check_bin_arch functions
package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
HOST_DIR
support/testing/infra: add log_file_path() function support/testing/tests: add test for check_bin_arch support/testing/tests: add test for file overwrite detection

 .gitlab-ci.yml                                |   3 +
 package/pkg-generic.mk                        | 146 ++++++++++--------
 package/pkg-utils.mk                          |   1 -
 support/testing/infra/__init__.py             |  13 +-
 .../br2-external/detect-bad-arch/Config.in    |   1 +
 .../detect-bad-arch/external.desc             |   1 +
 .../br2-external/detect-bad-arch/external.mk  |   1 +
 .../package/detect-bad-arch/Config.in         |   4 +
 .../detect-bad-arch/detect-bad-arch.mk        |  15 ++
 .../br2-external/detect-overwrite/Config.in   |   1 +
 .../detect-overwrite/external.desc            |   1 +
 .../br2-external/detect-overwrite/external.mk |   1 +
 .../package/detect-overwrite/Config.in        |   5 +
 .../detect-overwrite/detect-overwrite.mk      |  19 +++
 support/testing/tests/core/test_bad_arch.py   |  19 +++
 .../testing/tests/core/test_file_overwrite.py |  47 ++++++
 16 files changed, 213 insertions(+), 65 deletions(-)
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
 create mode 100644 support/testing/tests/core/test_bad_arch.py
 create mode 100644 support/testing/tests/core/test_file_overwrite.py
Signature · 1 line
2.25.4