pkg-stats: query release-monitoring.org with upstream name
PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name
By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables *_DISTNAME give the CPAN name which is known by release-monitoring.org
--- support/scripts/pkg-stats | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +18 -2
@@ -47,6 +47,7 @@ class Package: all_licenses = list() all_license_files = list() all_versions = dict() + all_upstream_names = dict() def __init__(self, name, path): self.name = name
@@ -58,6 +59,7 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.upstream_name = name self.url = None self.url_status = None self.url_worker = None
@@ -134,6 +136,14 @@ class Package: if var in self.all_versions: self.current_version = self.all_versions[var] + def set_upstream_name(self): + """ + Fills in the .upstream_name field + """ + var = self.pkgvar() + if var in self.all_upstream_names: + self.upstream_name = self.all_upstream_names[var] + def set_check_package_warnings(self): """ Fills in the .warnings field
@@ -227,7 +237,7 @@ def get_pkglist(npackages, package_list): def package_init_make_info(): # Fetch all variables at once variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars", - "VARS=%_LICENSE %_LICENSE_FILES %_VERSION"]) + "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME"]) variable_list = variables.splitlines() # We process first the host package VERSION, and then the target
@@ -261,6 +271,11 @@ def package_init_make_info(): pkgvar = pkgvar[:-8] Package.all_versions[pkgvar] = value + elif pkgvar.endswith("_DISTNAME"): + # exist only with the perl-package infra + pkgvar = pkgvar[:-9] + Package.all_upstream_names[pkgvar] = value + def check_url_status_worker(url, url_status): if url_status != "Missing" and url_status != "No Config.in":
@@ -349,7 +364,7 @@ def check_package_latest_version(packages): cert_reqs='CERT_REQUIRED', ca_certs=certifi.where(), timeout=30) worker_pool = Pool(processes=64) - results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) + results = worker_pool.map(check_package_latest_version_worker, (pkg.upstream_name for pkg in packages)) for pkg, r in zip(packages, results): pkg.latest_version = r del http_pool
@@ -742,6 +757,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_upstream_name() print("Checking URL status") check_package_urls(packages) print("Getting latest versions ...")
Signature · 1 line
2.20.1
PATCH 2/2 pkg-stats: more upstream names
the python infrastructure populates the variables _NAME_UPSTREAM (like the luarocks one).
and pkg-stats uses them.
--- package/pkg-python.mk | 1 + support/scripts/pkg-stats | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-)
package/pkg-python.mk +1 -0
@@ -108,6 +108,7 @@ define inner-python-package $(2)_ENV ?= $(2)_BUILD_OPTS ?= $(2)_INSTALL_OPTS ?= +$(2)_NAME_UPSTREAM ?= $(subst python-,,$(1)) ifndef $(2)_SETUP_TYPE ifdef $(3)_SETUP_TYPE
support/scripts/pkg-stats +6 -1
@@ -237,7 +237,7 @@ def get_pkglist(npackages, package_list): def package_init_make_info(): # Fetch all variables at once variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars", - "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME"]) + "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DISTNAME %_NAME_UPSTREAM"]) variable_list = variables.splitlines() # We process first the host package VERSION, and then the target
@@ -276,6 +276,11 @@ def package_init_make_info(): pkgvar = pkgvar[:-9] Package.all_upstream_names[pkgvar] = value + elif pkgvar.endswith("_NAME_UPSTREAM"): + # populated by some infra: luarocks, python + pkgvar = pkgvar[:-14] + Package.all_upstream_names[pkgvar] = value + def check_url_status_worker(url, url_status): if url_status != "Missing" and url_status != "No Config.in":
Signature · 1 line
2.20.1
PATCH 2/2 pkg-stats: more upstream names
Hello Fran?ois,
+Victor Huesca in Cc.
Quoted 5 lines · On Wed, 28 Aug 2019 09:02:51 +0200 Francois Perrad <[email protected]> wrote:
the python infrastructure populates the variables
_NAME_UPSTREAM(like the luarocks one).
and pkg-stats uses them.Signed-off-by Francois Perrad <[email protected]>
I think this should be split into two parts:
- one changing pkg-stats to use NAME_UPSTREAM, as it already helps for luarocks-package. However, could you name a few luarocks packages where their NAME_UPSTREAM matches the name of the project on release-monitoring.org ?
- for the Python case, what should "upstream name" match ? The name registered on PyPi ? Sometimes the name registered on PyPi doesn't always exactly match the upstream name.
In any case, the new NAME_UPSTREAM variable in python-package should be documented in the manual.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 2/2 pkg-stats: more upstream names
Le mer. 28 ao?t 2019 ? 14:47, Thomas Petazzoni <[email protected]> a ?crit :
Quoted 11 lines
Hello Fran?ois,
+Victor Huesca in Cc.
Quoted 4 lines · On Wed, 28 Aug 2019 09:02:51 +0200 Francois Perrad <[email protected]> wrote:
the python infrastructure populates the variables
_NAME_UPSTREAM(like the luarocks one).
and pkg-stats uses them.
Quoted 9 lines
Quoted 1 line
Signed-off-by Francois Perrad <[email protected]>I think this should be split into two parts:
- one changing pkg-stats to use
NAME_UPSTREAM, as it already helps for luarocks-package. However, could you name a few luarocks packages where theirNAME_UPSTREAMmatches the name of the project on release-monitoring.org ?
lua-basexx (basexx) see https://release-monitoring.org/project/15672/
lua-lpeg-patterns (lpeg_patterns) see
https://release-monitoring.org/project/15670/
lua-stdlib (stdlib) see https://release-monitoring.org/project/11735/
There is another issue with LuaRocks packages:
There is already a mismatch between the rock version and the upstream version found by release-monitoring, for example:
lua-http 0.3-0 vs 0.3
lua-iconv 7-1 (https://luarocks.org/modules/luarocks/lua-iconv/7-1) vs7 (https://release-monitoring.org/project/11668/)
This issue could be solve only by adding a specific LuaRocks backend to release-monitoring.
Fran?ois
Quoted 9 lines
- for the Python case, what should "upstream name" match ? The name registered on PyPi ? Sometimes the name registered on PyPi doesn't always exactly match the upstream name.
In any case, the new
NAME_UPSTREAMvariable in python-package should be documented in the manual.Best regards,
Quoted 10 lines
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190901/80dd9a5f/attachment.html>
PATCH 2/2 pkg-stats: more upstream names
In reply to Francois PerradQuoted 9 lines · On Wed, 28 Aug 2019 09:02:51 +0200 Francois Perrad <[email protected]> wrote:
the python infrastructure populates the variables
_NAME_UPSTREAM(like the luarocks one).
and pkg-stats uses them.Signed-off-by Francois Perrad <[email protected]>--- package/pkg-python.mk | 1 + support/scripts/pkg-stats | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-)
So we discussed this with Arnout, and since release-monitoring.org provides this mapping functionality, and someone has already filled in the mapping details for almost all Buildroot packages. We don't want to duplicate that information in Buildroot itself.
However, we can improve a bit the logic in pkg-stats when querying by "guess". I'll reply to your PATCH 1/2 with more details about this idea.
In the mean time, I've marked this particular patch as Rejected.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name
In reply to Francois PerradHello,
+Victor Huesca in Cc.
Quoted 5 lines · On Wed, 28 Aug 2019 09:02:50 +0200 Francois Perrad <[email protected]> wrote:
By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables*_DISTNAMEgive the CPAN name which is known by release-monitoring.orgSigned-off-by Francois Perrad <[email protected]>
Thanks for implementing this. It matches what we discussed. The only possible issue that I can see is if a package defines FOO_DISTNAME for some internal reason, but is not a perl-package. Then we will pick up whatever DISTNAME contains.
Perhaps we should standardize on FOO_NAME_UPSTREAM for all package infrastructures including perl-package, and make it a "reserved" variable name, so that it is forbidden to use it for anything else ?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name
Le mer. 28 ao?t 2019 ? 14:44, Thomas Petazzoni <[email protected]> a ?crit :
Quoted 13 lines
Hello,
+Victor Huesca in Cc.
Quoted 5 lines · On Wed, 28 Aug 2019 09:02:50 +0200 Francois Perrad <[email protected]> wrote:
By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables*_DISTNAMEgive the CPAN name which is known by release-monitoring.orgSigned-off-by Francois Perrad <[email protected]>
Quoted 10 lines
Thanks for implementing this. It matches what we discussed. The only possible issue that I can see is if a package defines
FOO_DISTNAMEfor some internal reason, but is not a perl-package. Then we will pick up whatever DISTNAME contains.Perhaps we should standardize on
FOO_NAME_UPSTREAMfor all package infrastructures including perl-package, and make it a "reserved" variable name, so that it is forbidden to use it for anything else ?
Sounds great.
If we have our own FOO_NAME_UPSTREAM, that avoids to register "mapping" on release-monitoring.org for any package.
When this addition will be validated, I'll cook one patch which updates (s/DISTNAME/UPSTREAM_NAME/) the 110 Perl/CPAN packages and a V2 of this patch.
Fran?ois
Thomas
Quoted 9 lines
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190901/73976526/attachment.html>
PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name
In reply to Francois PerradHello,
Quoted 5 lines · On Wed, 28 Aug 2019 09:02:50 +0200 Francois Perrad <[email protected]> wrote:
By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables*_DISTNAMEgive the CPAN name which is known by release-monitoring.orgSigned-off-by Francois Perrad <[email protected]>
Since we have this information already in Buildroot, it makes sense to use it. However, the search on release-monitoring.org is done in two steps:
- Using the Buildroot distribution in release-monitoring and its mapping concept. In this case, we really want to use the Buildroot package name unchanged (which is not what your patch does)
- Then, if no result is found through the Buildroot distribution above, we search in the entire namespace. Here, we should search with _DISTNAME when available. We could also improve the logic to strip python- and lua- in the front of packages, to help the guess to work.
Would you be willing to work on this ? It's not too far from the patch you have already implemented here.
Thanks a lot!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name
Le mer. 5 f?vr. 2020 ? 18:17, Thomas Petazzoni <[email protected]> a ?crit :
Quoted 11 lines
Hello,
Quoted 5 lines · On Wed, 28 Aug 2019 09:02:50 +0200 Francois Perrad <[email protected]> wrote:
By default, this upstream name is our BR name.
But with perl-package infrastructure, the variables*_DISTNAMEgive the CPAN name which is known by release-monitoring.orgSigned-off-by Francois Perrad <[email protected]>
Quoted 9 lines
Since we have this information already in Buildroot, it makes sense to use it. However, the search on release-monitoring.org is done in two steps:
- Using the Buildroot distribution in release-monitoring and its mapping concept. In this case, we really want to use the Buildroot package name unchanged (which is not what your patch does)
It means that a part of Buildroot source (or configuration) is hosted on release-monitoring.org instead of our own git repository.
Quoted 10 lines
- Then, if no result is found through the Buildroot distribution above, we search in the entire namespace. Here, we should search with
_DISTNAMEwhen available. We could also improve the logic to strip python- and lua- in the front of packages, to help the guess to work.Would you be willing to work on this ? It's not too far from the patch you have already implemented here.
Another point about release-monitoring.org,
how to exclude dev branches when the policy of the upstream project is to name them with an odd minor number ?
Here, some false positives:
perl | DISTRO |
https://release-monitoring.org/project/13599 | 5.30.1 | 5.31.8
|
webkitgtk | DISTRO |
https://release-monitoring.org/project/05355 | 2.26.3 | 2.27.4
|
wpewebkit | DISTRO |
https://release-monitoring.org/project/17557 | 2.26.3 | 2.27.4
|
Fran?ois
Thanks a lot!
Quoted 11 lines
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200207/ad86134b/attachment.html>
PATCH 1/2 pkg-stats: query release-monitoring.org with upstream name
Hello Fran?ois,
Quoted 6 lines · On Fri, 7 Feb 2020 17:58:57 +0100 Fran?ois Perrad <[email protected]>…
Quoted 3 lines
- Using the Buildroot distribution in release-monitoring and its mapping concept. In this case, we really want to use the Buildroot package name unchanged (which is not what your patch does)
It means that a part of Buildroot source (or configuration) is hosted on release-monitoring.org instead of our own git repository.
Not really. What is stored on release-monitoring side is the mapping of Buildroot package name to release-monitoring.org project names.
If we were to store this information on the Buildroot side, we could say "it's a release-monitoring.org information that is stored inside the Buildroot source code".
Quoted 3 lines
Another point about release-monitoring.org,
how to excludedevbranches when the policy of the upstream project is to name them with an odd minor number ?
This is a known issue, see
https://github.com/fedora-infra/anitya/issues/753. It would be great if someone could work on this.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com