Skip to contentSkip to conversation

pkg-stats: query release-monitoring.org with upstream name

PATCH New Francois Perrad 3 participants 10 messages Aug 28, 2019 – Feb 12, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name

By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables *_DISTNAME give the CPAN name which is known by release-monitoring.org

---
 support/scripts/pkg-stats | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +18 -2
index 992c2dd7c..e03f714a6 100755
@@ -47,6 +47,7 @@ class Package:
     all_licenses = list()
     all_license_files = list()
     all_versions = dict()
+    all_upstream_names = dict()
 
     def __init__(self, name, path):
         self.name = name
@@ -58,6 +59,7 @@ class Package:
         self.patch_count = 0
         self.warnings = 0
         self.current_version = None
+        self.upstream_name = name
         self.url = None
         self.url_status = None
         self.url_worker = None
@@ -134,6 +136,14 @@ class Package:
         if var in self.all_versions:
             self.current_version = self.all_versions[var]
 
+    def set_upstream_name(self):
+        """
+        Fills in the .upstream_name field
+        """
+        var = self.pkgvar()
+        if var in self.all_upstream_names:
+            self.upstream_name = self.all_upstream_names[var]
+
     def set_check_package_warnings(self):
         """
         Fills in the .warnings field
@@ -227,7 +237,7 @@ def get_pkglist(npackages, package_list):
 def package_init_make_info():
     # Fetch all variables at once
     variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars",
-                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION"])
+                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME"])
     variable_list = variables.splitlines()
 
     # We process first the host package VERSION, and then the target
@@ -261,6 +271,11 @@ def package_init_make_info():
             pkgvar = pkgvar[:-8]
             Package.all_versions[pkgvar] = value
 
+        elif pkgvar.endswith("_DISTNAME"):
+            # exist only with the perl-package infra
+            pkgvar = pkgvar[:-9]
+            Package.all_upstream_names[pkgvar] = value
+
 
 def check_url_status_worker(url, url_status):
     if url_status != "Missing" and url_status != "No Config.in":
@@ -349,7 +364,7 @@ def check_package_latest_version(packages):
                                     cert_reqs='CERT_REQUIRED', ca_certs=certifi.where(),
                                     timeout=30)
     worker_pool = Pool(processes=64)
-    results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages))
+    results = worker_pool.map(check_package_latest_version_worker, (pkg.upstream_name for pkg in packages))
     for pkg, r in zip(packages, results):
         pkg.latest_version = r
     del http_pool
@@ -742,6 +757,7 @@ def __main__():
         pkg.set_check_package_warnings()
         pkg.set_current_version()
         pkg.set_url()
+        pkg.set_upstream_name()
     print("Checking URL status")
     check_package_urls(packages)
     print("Getting latest versions ...")
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/2 pkg-stats: more upstream names

the python infrastructure populates the variables _NAME_UPSTREAM (like the luarocks one).
and pkg-stats uses them.

---
 package/pkg-python.mk     | 1 +
 support/scripts/pkg-stats | 7 ++++++-
 2 files changed, 7 insertions(+), 1 deletion(-)
package/pkg-python.mk +1 -0
index e90692041..7ec6f215a 100644
@@ -108,6 +108,7 @@ define inner-python-package
 $(2)_ENV         ?=
 $(2)_BUILD_OPTS   ?=
 $(2)_INSTALL_OPTS ?=
+$(2)_NAME_UPSTREAM ?= $(subst python-,,$(1))
 
 ifndef $(2)_SETUP_TYPE
  ifdef $(3)_SETUP_TYPE
support/scripts/pkg-stats +6 -1
index e03f714a6..92159c30f 100755
@@ -237,7 +237,7 @@ def get_pkglist(npackages, package_list):
 def package_init_make_info():
     # Fetch all variables at once
     variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars",
-                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME"])
+                                         "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME %_NAME_UPSTREAM"])
     variable_list = variables.splitlines()
 
     # We process first the host package VERSION, and then the target
@@ -276,6 +276,11 @@ def package_init_make_info():
             pkgvar = pkgvar[:-9]
             Package.all_upstream_names[pkgvar] = value
 
+        elif pkgvar.endswith("_NAME_UPSTREAM"):
+            # populated by some infra: luarocks, python
+            pkgvar = pkgvar[:-14]
+            Package.all_upstream_names[pkgvar] = value
+
 
 def check_url_status_worker(url, url_status):
     if url_status != "Missing" and url_status != "No Config.in":
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/2 pkg-stats: more upstream names

Hello Fran?ois,

+Victor Huesca in Cc.

Quoted 5 lines · On Wed, 28 Aug 2019 09:02:51 +0200 Francois Perrad <[email protected]> wrote:

the python infrastructure populates the variables _NAME_UPSTREAM (like the luarocks one).
and pkg-stats uses them.

I think this should be split into two parts:

- one changing pkg-stats to use NAME_UPSTREAM, as it already helps for luarocks-package. However, could you name a few luarocks packages where their NAME_UPSTREAM matches the name of the project on release-monitoring.org ?

- for the Python case, what should "upstream name" match ? The name registered on PyPi ? Sometimes the name registered on PyPi doesn't always exactly match the upstream name.

In any case, the new NAME_UPSTREAM variable in python-package should be documented in the manual.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH 2/2 pkg-stats: more upstream names

Le mer. 28 ao?t 2019 ? 14:47, Thomas Petazzoni <[email protected]> a ?crit :

Quoted 11 lines

Hello Fran?ois,

+Victor Huesca in Cc.

Quoted 4 lines · On Wed, 28 Aug 2019 09:02:51 +0200 Francois Perrad <[email protected]> wrote:

the python infrastructure populates the variables _NAME_UPSTREAM (like the luarocks one).
and pkg-stats uses them.

Quoted 9 lines
Quoted 1 line

I think this should be split into two parts:

- one changing pkg-stats to use NAME_UPSTREAM, as it already helps for luarocks-package. However, could you name a few luarocks packages where their NAME_UPSTREAM matches the name of the project on release-monitoring.org ?

lua-basexx (basexx) see https://release-monitoring.org/project/15672/
lua-lpeg-patterns (lpeg_patterns) see
https://release-monitoring.org/project/15670/
lua-stdlib (stdlib) see https://release-monitoring.org/project/11735/

There is another issue with LuaRocks packages:
There is already a mismatch between the rock version and the upstream version found by release-monitoring, for example:

    lua-http 0.3-0 vs 0.3
    lua-iconv 7-1 (https://luarocks.org/modules/luarocks/lua-iconv/7-1) vs

7 (https://release-monitoring.org/project/11668/)
This issue could be solve only by adding a specific LuaRocks backend to release-monitoring.

Fran?ois

Quoted 9 lines

- for the Python case, what should "upstream name" match ? The name registered on PyPi ? Sometimes the name registered on PyPi doesn't always exactly match the upstream name.

In any case, the new NAME_UPSTREAM variable in python-package should be documented in the manual.

Best regards,

Quoted 10 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190901/80dd9a5f/attachment.html>

#4 lore raw

PATCH 2/2 pkg-stats: more upstream names

In reply to Francois Perrad
Quoted 9 lines · On Wed, 28 Aug 2019 09:02:51 +0200 Francois Perrad <[email protected]> wrote:

the python infrastructure populates the variables _NAME_UPSTREAM (like the luarocks one).
and pkg-stats uses them.

---
 package/pkg-python.mk     | 1 +
 support/scripts/pkg-stats | 7 ++++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

So we discussed this with Arnout, and since release-monitoring.org provides this mapping functionality, and someone has already filled in the mapping details for almost all Buildroot packages. We don't want to duplicate that information in Buildroot itself.

However, we can improve a bit the logic in pkg-stats when querying by "guess". I'll reply to your PATCH 1/2 with more details about this idea.

In the mean time, I've marked this particular patch as Rejected.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name

In reply to Francois Perrad

Hello,

+Victor Huesca in Cc.

Quoted 5 lines · On Wed, 28 Aug 2019 09:02:50 +0200 Francois Perrad <[email protected]> wrote:

By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables *_DISTNAME give the CPAN name which is known by release-monitoring.org

Thanks for implementing this. It matches what we discussed. The only possible issue that I can see is if a package defines FOO_DISTNAME for some internal reason, but is not a perl-package. Then we will pick up whatever DISTNAME contains.

Perhaps we should standardize on FOO_NAME_UPSTREAM for all package infrastructures including perl-package, and make it a "reserved" variable name, so that it is forbidden to use it for anything else ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw

PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name

Le mer. 28 ao?t 2019 ? 14:44, Thomas Petazzoni <[email protected]> a ?crit :

Quoted 13 lines

Hello,

+Victor Huesca in Cc.

Quoted 5 lines · On Wed, 28 Aug 2019 09:02:50 +0200 Francois Perrad <[email protected]> wrote:

By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables *_DISTNAME give the CPAN name which is known by release-monitoring.org

Quoted 10 lines

Thanks for implementing this. It matches what we discussed. The only possible issue that I can see is if a package defines FOO_DISTNAME for some internal reason, but is not a perl-package. Then we will pick up whatever DISTNAME contains.

Perhaps we should standardize on FOO_NAME_UPSTREAM for all package infrastructures including perl-package, and make it a "reserved" variable name, so that it is forbidden to use it for anything else ?

Sounds great.
If we have our own FOO_NAME_UPSTREAM, that avoids to register "mapping" on release-monitoring.org for any package.

When this addition will be validated, I'll cook one patch which updates (s/DISTNAME/UPSTREAM_NAME/) the 110 Perl/CPAN packages and a V2 of this patch.

Fran?ois

Thomas

Quoted 9 lines

--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190901/73976526/attachment.html>

#7 lore raw

PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name

In reply to Francois Perrad

Hello,

Quoted 5 lines · On Wed, 28 Aug 2019 09:02:50 +0200 Francois Perrad <[email protected]> wrote:

By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables *_DISTNAME give the CPAN name which is known by release-monitoring.org

Since we have this information already in Buildroot, it makes sense to use it. However, the search on release-monitoring.org is done in two steps:

- Using the Buildroot distribution in release-monitoring and its mapping concept. In this case, we really want to use the Buildroot package name unchanged (which is not what your patch does)

- Then, if no result is found through the Buildroot distribution above, we search in the entire namespace. Here, we should search with _DISTNAME when available. We could also improve the logic to strip python- and lua- in the front of packages, to help the guess to work.

Would you be willing to work on this ? It's not too far from the patch you have already implemented here.

Thanks a lot!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw

PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name

Le mer. 5 f?vr. 2020 ? 18:17, Thomas Petazzoni <[email protected]> a ?crit :

Quoted 11 lines

Hello,

Quoted 5 lines · On Wed, 28 Aug 2019 09:02:50 +0200 Francois Perrad <[email protected]> wrote:

By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables *_DISTNAME give the CPAN name which is known by release-monitoring.org

Quoted 9 lines

Since we have this information already in Buildroot, it makes sense to use it. However, the search on release-monitoring.org is done in two steps:

- Using the Buildroot distribution in release-monitoring and its mapping concept. In this case, we really want to use the Buildroot package name unchanged (which is not what your patch does)

It means that a part of Buildroot source (or configuration) is hosted on release-monitoring.org instead of our own git repository.

Quoted 10 lines

- Then, if no result is found through the Buildroot distribution above, we search in the entire namespace. Here, we should search with _DISTNAME when available. We could also improve the logic to strip python- and lua- in the front of packages, to help the guess to work.

Would you be willing to work on this ? It's not too far from the patch you have already implemented here.

Another point about release-monitoring.org,
how to exclude dev branches when the policy of the upstream project is to name them with an odd minor number ?

Here, some false positives:
perl | DISTRO |
https://release-monitoring.org/project/13599 | 5.30.1 | 5.31.8
|
webkitgtk | DISTRO |
https://release-monitoring.org/project/05355 | 2.26.3 | 2.27.4
|
wpewebkit | DISTRO |
https://release-monitoring.org/project/17557 | 2.26.3 | 2.27.4
|

Fran?ois

Thanks a lot!

Quoted 11 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200207/ad86134b/attachment.html>

#9 lore raw

PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name

Hello Fran?ois,

Quoted 6 lines · On Fri, 7 Feb 2020 17:58:57 +0100 Fran?ois Perrad <[email protected]>…
Quoted 3 lines

- Using the Buildroot distribution in release-monitoring and its mapping concept. In this case, we really want to use the Buildroot package name unchanged (which is not what your patch does)

It means that a part of Buildroot source (or configuration) is hosted on release-monitoring.org instead of our own git repository.

Not really. What is stored on release-monitoring side is the mapping of Buildroot package name to release-monitoring.org project names.

If we were to store this information on the Buildroot side, we could say "it's a release-monitoring.org information that is stored inside the Buildroot source code".

Quoted 3 lines

Another point about release-monitoring.org,
how to exclude dev branches when the policy of the upstream project is to name them with an odd minor number ?

This is a known issue, see
https://github.com/fedora-infra/anitya/issues/753. It would be great if someone could work on this.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com