Skip to contentSkip to conversation

support/scripts/pkg-stats: add infra_coinfig to json_output

PATCH In review Heiko Thiery 3 participants 4 messages Jun 24, 2021 – Jul 25, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 support/scripts/pkg-stats: add infra_coinfig to json_output

When we use the statistics output to generate a CVE/CPE customer report showing whether a product is affected by CVEs, we are primarily interested in whether they are relevant to the target system. Currently we cannot see if the package is configured for the build (infra==host) and/or the target system (infra==target).

Therefore, in case of a given configuration, this information is also stored in the JSON output.

---
 support/scripts/pkg-stats | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)
support/scripts/pkg-stats +15 -6
index cc91d13167..f4dd034bd9 100755
@@ -89,6 +89,7 @@ class Package:
         self.path = path
         self.pkg_path = os.path.dirname(path)
         self.infras = None
+        self.infra_config = None
         self.license = None
         self.has_license = False
         self.has_license_files = False
@@ -146,9 +147,9 @@ class Package:
             return False
         return True
 
-    def set_infra(self):
+    def set_infra(self, config_info):
         """
-        Fills in the .infras field
+        Fills in the .infras and ..infra_config fields
         """
         self.infras = list()
         with open(os.path.join(brpath, self.path), 'r') as f:
@@ -163,6 +164,13 @@ class Package:
                 else:
                     self.infras.append(("target", infra))
 
+        if config_info:
+            for pkg_name in config_info:
+                if config_info[pkg_name]['name'] == self.name:
+                    if self.infra_config == None:
+                        self.infra_config = list()
+                    self.infra_config.append(config_info[pkg_name]['type'])
+
     def set_license(self):
         """
         Fills in the .status['license'] and .status['license-files'] fields
@@ -369,8 +377,8 @@ def get_pkglist(npackages, package_list):
 
 def get_config_packages():
     cmd = ["make", "--no-print-directory", "show-info"]
-    js = json.loads(subprocess.check_output(cmd))
-    return set([v["name"] for v in js.values()])
+    config_info = json.loads(subprocess.check_output(cmd))
+    return (config_info, set([v["name"] for v in config_info.values()]))
 
 
 def package_init_make_info():
@@ -1115,10 +1123,11 @@ def __main__():
     if args.nvd_path:
         import cve as cvecheck
 
+    config_info = None
     if args.packages:
         package_list = args.packages.split(",")
     elif args.configpackages:
-        package_list = get_config_packages()
+        (config_info, package_list) = get_config_packages()
     else:
         package_list = None
     date = datetime.datetime.utcnow()
@@ -1137,7 +1146,7 @@ def __main__():
     package_init_make_info()
     print("Getting package details ...")
     for pkg in packages:
-        pkg.set_infra()
+        pkg.set_infra(config_info)
         pkg.set_license()
         pkg.set_hash_info()
         pkg.set_patch_count()
Signature · 1 line
2.30.0
#1 lore raw

PATCH 1/1 support/scripts/pkg-stats: add infra_coinfig to json_output

Quoted 50 lines · On Thu, 24 Jun 2021 14:57:03 +0200 Heiko Thiery <[email protected]> wrote:
support/scripts/pkg-stats +12 -4
index cc91d13167..f4dd034bd9 100755
@@ -89,6 +89,7 @@ class Package:
         self.path = path
         self.pkg_path = os.path.dirname(path)
         self.infras = None
+        self.infra_config = None
         self.license = None
         self.has_license = False
         self.has_license_files = False
@@ -146,9 +147,9 @@ class Package:
             return False
         return True
 
-    def set_infra(self):
+    def set_infra(self, config_info):
         """
-        Fills in the .infras field
+        Fills in the .infras and ..infra_config fields
         """
         self.infras = list()
         with open(os.path.join(brpath, self.path), 'r') as f:
@@ -163,6 +164,13 @@ class Package:
                 else:
                     self.infras.append(("target", infra))
 
+        if config_info:
+            for pkg_name in config_info:
+                if config_info[pkg_name]['name'] == self.name:
+                    if self.infra_config == None:
+                        self.infra_config = list()
+                    self.infra_config.append(config_info[pkg_name]['type'])
+
     def set_license(self):
         """
         Fills in the .status['license'] and .status['license-files'] fields
@@ -369,8 +377,8 @@ def get_pkglist(npackages, package_list):
 
 def get_config_packages():
     cmd = ["make", "--no-print-directory", "show-info"]
-    js = json.loads(subprocess.check_output(cmd))
-    return set([v["name"] for v in js.values()])
+    config_info = json.loads(subprocess.check_output(cmd))
+    return (config_info, set([v["name"] for v in config_info.values()]))
 
 
 def package_init_make_info():

support/scripts/pkg-stats def __main__(): · line 1130

Quoted 9 lines
support/scripts/pkg-stats +2 -1
@@ -1115,10 +1123,11 @@ def __main__():
     if args.nvd_path:
         import cve as cvecheck
 
+    config_info = None
     if args.packages:
         package_list = args.packages.split(",")
     elif args.configpackages:
-        package_list = get_config_packages()
+        (config_info, package_list) = get_config_packages()

Why is this information passed separately, and not within the package_list ? I'm a bit confused here. Why don't have you add a "type" field in the Package class that says whether it's a host, target, or both package ?

Thomas

Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/1 support/scripts/pkg-stats: add infra_coinfig to json_output

Hi Thomas,

Am Do., 24. Juni 2021 um 15:04 Uhr schrieb Thomas Petazzoni <[email protected]>:

Quoted 57 lines
Quoted 53 lines · On Thu, 24 Jun 2021 14:57:03 +0200 Heiko Thiery <[email protected]> wrote:
support/scripts/pkg-stats +12 -4
index cc91d13167..f4dd034bd9 100755
@@ -89,6 +89,7 @@ class Package:
         self.path = path
         self.pkg_path = os.path.dirname(path)
         self.infras = None
+        self.infra_config = None
         self.license = None
         self.has_license = False
         self.has_license_files = False
@@ -146,9 +147,9 @@ class Package:
             return False
         return True
 
-    def set_infra(self):
+    def set_infra(self, config_info):
         """
-        Fills in the .infras field
+        Fills in the .infras and ..infra_config fields
         """
         self.infras = list()
         with open(os.path.join(brpath, self.path), 'r') as f:
@@ -163,6 +164,13 @@ class Package:
                 else:
                     self.infras.append(("target", infra))
 
+        if config_info:
+            for pkg_name in config_info:
+                if config_info[pkg_name]['name'] == self.name:
+                    if self.infra_config == None:
+                        self.infra_config = list()
+                    self.infra_config.append(config_info[pkg_name]['type'])
+
     def set_license(self):
         """
         Fills in the .status['license'] and .status['license-files'] fields
@@ -369,8 +377,8 @@ def get_pkglist(npackages, package_list):
 
 def get_config_packages():
     cmd = ["make", "--no-print-directory", "show-info"]
-    js = json.loads(subprocess.check_output(cmd))
-    return set([v["name"] for v in js.values()])
+    config_info = json.loads(subprocess.check_output(cmd))
+    return (config_info, set([v["name"] for v in config_info.values()]))
 
 
 def package_init_make_info():
@@ -1115,10 +1123,11 @@ def __main__():
     if args.nvd_path:
         import cve as cvecheck
 
Quoted 11 lines

support/scripts/pkg-stats def __main__(): · line 1130

Quoted 6 lines
support/scripts/pkg-stats +2 -1
+    config_info = None
     if args.packages:
         package_list = args.packages.split(",")
     elif args.configpackages:
-        package_list = get_config_packages()
+        (config_info, package_list) = get_config_packages()

Why is this information passed separately, and not within the package_list ? I'm a bit confused here. Why don't have you add a "type" field in the Package class that says whether it's a host, target, or both package ?

Because the package_list is only a list of package names. It is not a class object. And I did not want to change the package_list variable. This is also filled by "packages" command line arguments. Or am I missing something here?

Signature · 1 line
Heiko
#3 lore raw
In reply to Heiko Thiery

Hello Heiko,

Quoted 6 lines · On Thu, 24 Jun 2021 14:57:03 +0200 Heiko Thiery <[email protected]> wrote:

When we use the statistics output to generate a CVE/CPE customer report showing whether a product is affected by CVEs, we are primarily interested in whether they are relevant to the target system. Currently we cannot see if the package is configured for the build (infra==host) and/or the target system (infra==target).

Quoted 7 lines

Therefore, in case of a given configuration, this information is also stored in the JSON output.

---
 support/scripts/pkg-stats | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

We finally had a look with Arnout. We didn't really like this new self.infra_config field. So instead, what we decided is to use the existing self.infras field.

When pkg-stats is run without a configuration (i.e without -c), self.infras lists all infrastructures supported by the package, as it is done today.

When pkg-stats is run with a configuration (i.e with -c), self.infras now only lists the infrastructures that are *used* by the package in the current configuration.

I reworked your code with this idea, and also changed the implementation quite a bit. You can see the result at:

https://git.buildroot.org/buildroot/commit/?id=28973f28acfcee81f2912e7a65dd3704b1d68f6f

When you have the chance, could you give it a test and let us know if it works for you?

Thanks a lot!

Thomas

Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot