Skip to contentSkip to conversation

support/scripts/pkg-stats: add latest upstream version information

PATCH Superseded Thomas Petazzoni 5 participants 11 messages Jan 3, 2019 – Jan 4, 2019 1 message not in the archive

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 support/scripts/pkg-stats: add latest upstream version information

This commit adds fetching the latest upstream version of each package from release-monitoring.org.

The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. This mapping mechanism allows to tell release-monitoring.org what is the name of a package in a given distribution/build-system. For example, the package xutil_util-macros in Buildroot is named xorg-util-macros on release-monitoring.org. This mapping can be seen in the section "Mappings" of
https://release-monitoring.org/project/15037/.

If there is no mapping, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name.

Even though fetching from release-monitoring.org is a bit slow, using multiprocessing.Pool has proven to not be reliable, with some requests ending up with an exception. So we keep a serialized approach, but with a single HTTPSConnectionPool() for all queries. Long term, we hope to be able to use a database dump of release-monitoring.org instead.

From an output point of view, the latest version column:

- Is green when the version in Buildroot matches the latest upstream version

- Is orange when the latest upstream version is unknown because the package was not found on release-monitoring.org

- Is red when the version in Buildroot doesn't match the latest upstream version. Note that we are not doing anything smart here:
we are just testing if the strings are equal or not.

- The cell contains the link to the project on release-monitoring.org if found.

- The cell indicates if the match was done using a distro mapping, or through a regular search.

[1] https://release-monitoring.org/distro/Buildroot/

---
Changes since v4:
- Don't use multiprocessing.Pool(), stick to a serialized approach, which is more reliable.
- Handle errors/exceptions properly.
- Improve the layout of the resulting table column.

Changes since v3:
- Use Pool(), like is done for the upstream URL checking added by Matt Weber
- Use the requests Python module instead of the urllib2 Python module, so that we use the same module as the one used for the upstream URL checking
- Adjusted to work with the latest pkg-stats code

Changes since v2:
- Use the "timeout" argument of urllib2.urlopen() in order to make sure that the requests terminate at some point, even if release-monitoring.org is stuck.
- Move a lot of the logic as methods of the Package() class.

Changes since v1:
- Fix flake8 warnings
- Add missing newline in HTML

---
 support/scripts/pkg-stats | 137 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 137 insertions(+)
support/scripts/pkg-stats +137 -0
index d0b06b1e74..d32087cda1 100755
@@ -25,11 +25,19 @@ import re
 import subprocess
 import sys
 import requests  # URL checking
+import json
+import certifi
+from urllib3 import HTTPSConnectionPool
 from multiprocessing import Pool
 
 INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
 URL_RE = re.compile("\s*https?://\S*\s*$")
+RELEASE_MONITORING_API = "http://release-monitoring.org/api"
 
+RM_API_STATUS_ERROR = 1
+RM_API_STATUS_FOUND_BY_DISTRO = 2
+RM_API_STATUS_FOUND_BY_PATTERN = 3
+RM_API_STATUS_NOT_FOUND = 4
 
 class Package:
     all_licenses = list()
@@ -49,6 +57,7 @@ class Package:
         self.url = None
         self.url_status = None
         self.url_worker = None
+        self.latest_version = None
 
     def pkgvar(self):
         return self.name.upper().replace("-", "_")
@@ -297,6 +306,66 @@ def check_package_urls(packages):
     for pkg in packages:
         pkg.url_status = pkg.url_worker.get(timeout=3600)
 
+def release_monitoring_get_latest_version_by_distro(pool, name):
+    try:
+        req = pool.request('GET', "/api/project/Buildroot/%s" % name)
+    except:
+        return (RM_API_STATUS_ERROR, None, None)
+
+    if req.status != 200:
+        return (RM_API_STATUS_NOT_FOUND, None, None)
+
+    data = json.loads(req.data)
+
+    if len(data['versions']) > 0:
+        return (RM_API_STATUS_FOUND_BY_DISTRO, data['versions'][0], data['id'])
+    else:
+        return (RM_API_STATUS_FOUND_BY_DISTRO, None, data['id'])
+
+def release_monitoring_get_latest_version_by_guess(pool, name):
+    try:
+        req = pool.request('GET', "/api/projects/?pattern=%s" % name)
+    except:
+        print("Exception release_monitoring_get_latest_version_by_guess for '%s': %s" % (name, err))
+        return (RM_API_STATUS_ERROR, None, None)
+
+    if req.status != 200:
+        return (RM_API_STATUS_NOT_FOUND, None, None)
+
+    data = json.loads(req.data)
+
+    for p in data['projects']:
+        if p['name'] == name and len(p['versions']) > 0:
+            return (RM_API_STATUS_FOUND_BY_PATTERN, p['versions'][0], p['id'])
+
+    return (RM_API_STATUS_NOT_FOUND, None, None)
+
+def check_package_latest_version(packages):
+    """
+    Fills in the .latest_version field of all Package objects
+
+    This field has a special format:
+      (status, version, id)
+    with:
+    - status: one of RM_API_STATUS_ERROR,
+      RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN,
+      RM_API_STATUS_NOT_FOUND
+    - version: string containing the latest version known by
+      release-monitoring.org for this package
+    - id: string containing the id of the project corresponding to this
+      package, as known by release-monitoring.org
+    """
+    pool = HTTPSConnectionPool('release-monitoring.org', port=443,
+                               cert_reqs='CERT_REQUIRED', ca_certs=certifi.where())
+    count = 0
+    for pkg in packages:
+        v = release_monitoring_get_latest_version_by_distro(pool, pkg.name)
+        if v[0] == RM_API_STATUS_NOT_FOUND:
+            v = release_monitoring_get_latest_version_by_guess(pool, pkg.name)
+
+        pkg.latest_version = v
+        print("[%d/%d] Package %s" % (count, len(packages), pkg.name))
+        count += 1
 
 def calculate_stats(packages):
     stats = defaultdict(int)
@@ -322,6 +391,16 @@ def calculate_stats(packages):
             stats["hash"] += 1
         else:
             stats["no-hash"] += 1
+        if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO:
+            stats["rmo-mapping"] += 1
+        else:
+            stats["rmo-no-mapping"] += 1
+        if not pkg.latest_version[1]:
+            stats["version-unknown"] += 1
+        elif pkg.latest_version[1] == pkg.current_version:
+            stats["version-uptodate"] += 1
+        else:
+            stats["version-not-uptodate"] += 1
         stats["patches"] += pkg.patch_count
     return stats
 
@@ -354,6 +433,7 @@ td.somepatches {
 td.lotsofpatches {
   background: #ff9a69;
 }
+
 td.good_url {
   background: #d2ffc4;
 }
@@ -363,6 +443,20 @@ td.missing_url {
 td.invalid_url {
   background: #ff9a69;
 }
+
+td.version-good {
+  background: #d2ffc4;
+}
+td.version-needs-update {
+  background: #ff9a69;
+}
+td.version-unknown {
+ background: #ffd870;
+}
+td.version-error {
+ background: #ccc;
+}
+
 </style>
 <title>Statistics of Buildroot packages</title>
 </head>
@@ -465,6 +559,36 @@ def dump_html_pkg(f, pkg):
         current_version = pkg.current_version
     f.write("  <td class=\"centered\">%s</td>\n" % current_version)
 
+    # Latest version
+    if pkg.latest_version[0] == RM_API_STATUS_ERROR:
+        td_class.append("version-error")
+    if pkg.latest_version[1] is None:
+        td_class.append("version-unknown")
+    elif pkg.latest_version[1] != pkg.current_version:
+        td_class.append("version-needs-update")
+    else:
+        td_class.append("version-good")
+
+    if pkg.latest_version[0] == RM_API_STATUS_ERROR:
+        latest_version_text = "<b>Error</b>"
+    elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND:
+        latest_version_text = "<b>Not found</b>"
+    else:
+        if pkg.latest_version[1] is None:
+            latest_version_text = "<b>Found, but not version</b>"
+        else:
+            latest_version_text = "<a href=\"https://release-monitoring.org/project/%s\"><b>%s</b></a>" % (pkg.latest_version[2], str(pkg.latest_version[1]))
+
+        latest_version_text += "<br/>"
+
+        if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO:
+            latest_version_text += "found by <a href=\"https://release-monitoring.org/distro/Buildroot/\">distro</a>"
+        else:
+            latest_version_text += "found by guess"
+
+    f.write("  <td class=\"%s\">%s</td>\n" %
+            (" ".join(td_class), latest_version_text))
+
     # Warnings
     td_class = ["centered"]
     if pkg.warnings == 0:
@@ -502,6 +626,7 @@ def dump_html_all_pkgs(f, packages):
 <td class=\"centered\">License files</td>
 <td class=\"centered\">Hash file</td>
 <td class=\"centered\">Current version</td>
+<td class=\"centered\">Latest version</td>
 <td class=\"centered\">Warnings</td>
 <td class=\"centered\">Upstream URL</td>
 </tr>
@@ -532,6 +657,16 @@ def dump_html_stats(f, stats):
             stats["no-hash"])
     f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" %
             stats["patches"])
+    f.write("<tr><td>Packages having a mapping on <i>release-monitoring.org</i></td><td>%s</td></tr>\n" %
+            stats["rmo-mapping"])
+    f.write("<tr><td>Packages lacking a mapping on <i>release-monitoring.org</i></td><td>%s</td></tr>\n" %
+            stats["rmo-no-mapping"])
+    f.write("<tr><td>Packages that are up-to-date</td><td>%s</td></tr>\n" %
+            stats["version-uptodate"])
+    f.write("<tr><td>Packages that are not up-to-date</td><td>%s</td></tr>\n" %
+            stats["version-not-uptodate"])
+    f.write("<tr><td>Packages with no known upstream version</td><td>%s</td></tr>\n" %
+            stats["version-unknown"])
     f.write("</table>\n")
 
 
@@ -587,6 +722,8 @@ def __main__():
         pkg.set_url()
     print("Checking URL status")
     check_package_urls(packages)
+    print("Getting latest versions ...")
+    check_package_latest_version(packages)
     print("Calculate stats")
     stats = calculate_stats(packages)
     print("Write HTML")
Signature · 1 line
2.20.1
#1 lore raw

PATCH v5 1/1 support/scripts/pkg-stats: add latest upstream version information

Hello,

Quoted 2 lines · Thomas Petazzoni

This commit adds fetching the latest upstream version of each package from release-monitoring.org.

If anyone wants to see what the result looks like:

https://bootlin.com/~thomas/stats.html

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH v5 1/1 support/scripts/pkg-stats: add latest upstream version information

In reply to Thomas Petazzoni

This is a useful tool, thanks! I don't have time to test it right now, but an observation.

On Thu, Jan 3, 2019 at 2:38 AM Thomas Petazzoni
<[email protected]> wrote:
...

Quoted 2 lines
support/scripts/pkg-stats +2 -0
+        elif pkg.latest_version[1] == pkg.current_version:
+            stats["version-uptodate"] += 1

...

support/scripts/pkg-stats def dump_html_pkg(f, pkg): · line 568

Quoted 2 lines
support/scripts/pkg-stats +2 -0
+    elif pkg.latest_version[1] != pkg.current_version:
+        td_class.append("version-needs-update")

From https://bootlin.com/~thomas/stats.html, there's a few packages that are up-to-date, but get marked "version-needs-update" because either the Buildroot or release-monitoring version string starts with a "v". For example, the vmtouch and libmodbus packages. It would be nice to change the version comparison to ignore a leading "v". E.g.

def version_uptodate(pkg):
latest = remove_prefix(pkg.latest_version[1], 'v') current = remove_prefix(pkg.current_version, 'v') return latest == current

#3 lore raw

PATCH v5 1/1 support/scripts/pkg-stats: add latest upstream version information

Hello Brandon,

Thanks for the feedback.

Quoted 10 lines · Brandon Maier

From https://bootlin.com/~thomas/stats.html, there's a few packages that are up-to-date, but get marked "version-needs-update" because either the Buildroot or release-monitoring version string starts with a "v". For example, the vmtouch and libmodbus packages. It would be nice to change the version comparison to ignore a leading "v". E.g.

def version_uptodate(pkg):
latest = remove_prefix(pkg.latest_version[1], 'v') current = remove_prefix(pkg.current_version, 'v') return latest == current

I'm aware of this problem, but I'm not sure we should handle this with a hack in the pkg-stats script itself. Indeed sometimes the version prefix is just "v", but sometimes it's the entire name of the package, sometimes something slightly different.

Back when I initially started working on this script, I had a series that changed all the packages that used vX.Y.Z as a version to use just X.Y.Z, and similarly for other packages in similar but slightly different situations. I found (with some effort) the old branch I had with those changes:

https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=fix-versions

It seems like I never posted them. I thought there was some discussion on the list about this issue, but I can't find it. Perhaps I should update this series, submit it for good, and see what the feedback is ?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

:

Quoted 18 lines · Thomas Petazzoni

Hello Brandon,

Thanks for the feedback.

Quoted 10 lines · Brandon Maier

From https://urldefense.proofpoint.com/v2/url?u=https-3A__bootlin.com_-7Ethomas_stats.html&d=DwICAg&c=ilBQI1lupc9Y65XwNblLtw&r=bIwUnEkCqKFQQ0RVQaaY0gBWY7SIAhmiWLyMS82_mSU&m=BLGQQMtjdYeO462A3rnKjL7GVl8jKWnqncRGOzwJuc8&s=nJR7ACo6ote42lIcGZ0XW87x_SqYCdyi5OdeloYNqu4&e=, there's a few packages that are up-to-date, but get marked "version-needs-update" because
either the Buildroot or release-monitoring version string starts with
a "v". For example, the vmtouch and libmodbus packages. It would be
nice to change the version comparison to ignore a leading "v". E.g.

def version_uptodate(pkg):
latest = remove_prefix(pkg.latest_version[1], 'v') current = remove_prefix(pkg.current_version, 'v') return latest == current

Quoted 4 lines

I'm aware of this problem, but I'm not sure we should handle this with a hack in the pkg-stats script itself. Indeed sometimes the version prefix is just "v", but sometimes it's the entire name of the package, sometimes something slightly different.

Agreed it's not a pretty hack. I wouldn't want to handle the other oddities in here, as that would get messy. I suggested it for "v" only because it seems to be very common. It would be a way to get us most of the way there, then it could be reverted if e.g. all the github versions are fixed up.

On those other packages though, perhaps adding an optional variable to package mk files for custom versions? E.g. package/libnfs/libnfs.mk defines version as "libnfs-3.0.0", but the r-m version is "3.0.0". E.g.

package/libnfs/libnfs.mk:
LIBNFS_RM_VERSION = 3.0.0
LIBNFS_VERSION = libnfs-$(LIBNFS_RM_VERSION)

Quoted 7 lines

Back when I initially started working on this script, I had a series that changed all the packages that used vX.Y.Z as a version to use just X.Y.Z, and similarly for other packages in similar but slightly different situations. I found (with some effort) the old branch I had with those changes:

Quoted 5 lines

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.bootlin.com_users_thomas-2Dpetazzoni_buildroot_log_-3Fh-3Dfix-2Dversions&d=DwICAg&c=ilBQI1lupc9Y65XwNblLtw&r=bIwUnEkCqKFQQ0RVQaaY0gBWY7SIAhmiWLyMS82_mSU&m=BLGQQMtjdYeO462A3rnKjL7GVl8jKWnqncRGOzwJuc8&s=YcscSq6riQ_8BeiMfo3XUuVEigc3adPnIJrOJn7hl3w&e=

It seems like I never posted them. I thought there was some discussion on the list about this issue, but I can't find it. Perhaps I should update this series, submit it for good, and see what the feedback is ?

Regardless of this patch series, this sounds like a good idea to me. Especially considering Anitya does this too, as Yann mentioned[1].

[1] https://github.com/release-monitoring/anitya/issues/374

Quoted 8 lines

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://urldefense.proofpoint.com/v2/url?u=https-3A__bootlin.com&d=DwICAg&c=ilBQI1lupc9Y65XwNblLtw&r=bIwUnEkCqKFQQ0RVQaaY0gBWY7SIAhmiWLyMS82_mSU&m=BLGQQMtjdYeO462A3rnKjL7GVl8jKWnqncRGOzwJuc8&s=Pqp_noteyWtmSPPMYtHFZwlZbRE-a5HRmBWfPJcGDV0&e=

#5 lore raw

Hello,

Quoted 6 lines · Brandon Maier

Agreed it's not a pretty hack. I wouldn't want to handle the other oddities in here, as that would get messy. I suggested it for "v" only because it seems to be very common. It would be a way to get us most of the way there, then it could be reverted if e.g. all the github versions are fixed up.

Quoted 8 lines

On those other packages though, perhaps adding an optional variable to package mk files for custom versions? E.g. package/libnfs/libnfs.mk defines version as "libnfs-3.0.0", but the r-m version is "3.0.0". E.g.

package/libnfs/libnfs.mk:
LIBNFS_RM_VERSION = 3.0.0
LIBNFS_VERSION = libnfs-$(LIBNFS_RM_VERSION)

We could do that, but is that really needed/useful ?

Quoted 8 lines
Quoted 3 lines

It seems like I never posted them. I thought there was some discussion on the list about this issue, but I can't find it. Perhaps I should update this series, submit it for good, and see what the feedback is ?

Regardless of this patch series, this sounds like a good idea to me. Especially considering Anitya does this too, as Yann mentioned[1].

[1] https://github.com/release-monitoring/anitya/issues/374

Right. So you say I should submit this "fix-versions" patch series ?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw

On Thu, Jan 3, 2019 at 3:31 PM Thomas Petazzoni
<[email protected]> wrote:
...

Quoted 6 lines
Quoted 3 lines

package/libnfs/libnfs.mk:
LIBNFS_RM_VERSION = 3.0.0
LIBNFS_VERSION = libnfs-$(LIBNFS_RM_VERSION)

We could do that, but is that really needed/useful ?

As discussed on IRC, nevermind this. If all package's _VERSION are changed to match release-monitoring then it won't matter.

Quoted 10 lines
Quoted 8 lines
Quoted 3 lines

It seems like I never posted them. I thought there was some discussion on the list about this issue, but I can't find it. Perhaps I should update this series, submit it for good, and see what the feedback is ?

Regardless of this patch series, this sounds like a good idea to me. Especially considering Anitya does this too, as Yann mentioned[1].

[1] https://github.com/release-monitoring/anitya/issues/374

Right. So you say I should submit this "fix-versions" patch series ?

Yes, good to get the discussion going.

#7 lore raw

PATCH v5 1/1 support/scripts/pkg-stats: add latest upstream version information

In reply to Brandon Maier

Brandon, All,

On 2019-01-03 13:57 -0600, Brandon Maier spake thusly:

Quoted 8 lines

This is a useful tool, thanks! I don't have time to test it right now, but an observation.

On Thu, Jan 3, 2019 at 2:38 AM Thomas Petazzoni
<[email protected]> wrote:
...

support/scripts/pkg-stats def calculate_stats(packages): · line 401

Quoted 2 lines
support/scripts/pkg-stats +2 -0
+        elif pkg.latest_version[1] == pkg.current_version:
+            stats["version-uptodate"] += 1
Quoted 8 lines

...

support/scripts/pkg-stats def dump_html_pkg(f, pkg): · line 568

Quoted 2 lines
support/scripts/pkg-stats +2 -0
+    elif pkg.latest_version[1] != pkg.current_version:
+        td_class.append("version-needs-update")

From https://bootlin.com/~thomas/stats.html, there's a few packages that are up-to-date, but get marked "version-needs-update" because either the Buildroot or release-monitoring version string starts with a "v".

There was an issue raised (long ago) about this in the anitya tool (which generatees the DB at r-m.o):
https://github.com/release-monitoring/anitya/issues/374

So, the version strings we get from r-m.o should no longer have a leading 'v' anymore.

Yet, in our FOO_VERSION, we in Buildroot may still have those leading 'v'. So, we should try to compare the versions as-is, and then remove out leading 'v' and compare again. ot sure of the exact heuristic, though...

Regards,
Yann E. MORIN.

Quoted 11 lines

For example, the vmtouch and libmodbus packages. It would be nice to change the version comparison to ignore a leading "v". E.g.

def version_uptodate(pkg):
latest = remove_prefix(pkg.latest_version[1], 'v') current = remove_prefix(pkg.current_version, 'v') return latest == current
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

A message that is not in the archive (sent off-list or not yet received): CANQCQpZHYrkUVrYUxqANd9xzicMz-9apDWYeUWYmHLqG6YA8wg@mail.gmail.com

#9 lore raw

PATCH v5 1/1 support/scripts/pkg-stats: add latest upstream version information

All,
(sending this one again for patchwork to see it, my email issue should be fixed)

Quoted 181 lines · Matthew Weber

Thomas,

(Sorry about the urls which got reformatted below....)

Quoted 78 lines · Thomas Petazzoni

This commit adds fetching the latest upstream version of each package from release-monitoring.org.

The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. This mapping mechanism allows to tell release-monitoring.org what is the name of a package in a given distribution/build-system. For example, the package xutil_util-macros in Buildroot is named xorg-util-macros on release-monitoring.org. This mapping can be seen in the section "Mappings" of
https://urldefense.proofpoint.com/v2/url?u=https-3A__release-2Dmonitoring.org_project_15037_&d=DwIDAg&c=ilBQI1lupc9Y65XwNblLtw&r=y1sOV0GV8NZUkffv7oCRxs2Sd3nOBS-NxDM3NY8lOgs&m=D29WCvvKfHr85bUA0S0PaBzfYjT7AD5mrqPNN2yGQic&s=MAudgohIjdSwVSLqKKEn3eTk1ZdF8EZZjy91fRk2RK8&e=.

If there is no mapping, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name.

Even though fetching from release-monitoring.org is a bit slow, using multiprocessing.Pool has proven to not be reliable, with some requests ending up with an exception. So we keep a serialized approach, but with a single HTTPSConnectionPool() for all queries. Long term, we hope to be able to use a database dump of release-monitoring.org instead.

From an output point of view, the latest version column:

- Is green when the version in Buildroot matches the latest upstream version

- Is orange when the latest upstream version is unknown because the package was not found on release-monitoring.org

- Is red when the version in Buildroot doesn't match the latest upstream version. Note that we are not doing anything smart here:
we are just testing if the strings are equal or not.

- The cell contains the link to the project on release-monitoring.org if found.

- The cell indicates if the match was done using a distro mapping, or through a regular search.

[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__release-2Dmonitoring.org_distro_Buildroot_&d=DwIDAg&c=ilBQI1lupc9Y65XwNblLtw&r=y1sOV0GV8NZUkffv7oCRxs2Sd3nOBS-NxDM3NY8lOgs&m=D29WCvvKfHr85bUA0S0PaBzfYjT7AD5mrqPNN2yGQic&s=JVlzNLvfXxrlF8HcMalzbDFkuA30zKVTfxaRyA7AVkY&e=

---
Changes since v4:
- Don't use multiprocessing.Pool(), stick to a serialized approach, which is more reliable.
- Handle errors/exceptions properly.
- Improve the layout of the resulting table column.

Changes since v3:
- Use Pool(), like is done for the upstream URL checking added by Matt Weber
- Use the requests Python module instead of the urllib2 Python module, so that we use the same module as the one used for the upstream URL checking
- Adjusted to work with the latest pkg-stats code

Changes since v2:
- Use the "timeout" argument of urllib2.urlopen() in order to make sure that the requests terminate at some point, even if release-monitoring.org is stuck.
- Move a lot of the logic as methods of the Package() class.

Changes since v1:
- Fix flake8 warnings
- Add missing newline in HTML

---
 support/scripts/pkg-stats | 137 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 137 insertions(+)
support/scripts/pkg-stats +0 -0
index d0b06b1e74..d32087cda1 100755
@@ -25,11 +25,19 @@ import re
 import subprocess

support/scripts/pkg-stats import re · line 35

Quoted 10 lines
support/scripts/pkg-stats +4 -0
 import sys
 import requests  # URL checking
+import json
+import certifi
+from urllib3 import HTTPSConnectionPool
 from multiprocessing import Pool
 
 INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
 URL_RE = re.compile("\s*https?://\S*\s*$")
+RELEASE_MONITORING_API = "https://urldefense.proofpoint.com/v2/url?u=http-3A__release-2Dmonitoring.org_api&d=DwIDAg&c=ilBQI1lupc9Y65XwNblLtw&r=y1sOV0GV8NZUkffv7oCRxs2Sd3nOBS-NxDM3NY8lOgs&m=D29WCvvKfHr85bUA0S0PaBzfYjT7AD5mrqPNN2yGQic&s=8CezxghNio2dEk11Egejv806bc2Bt4Vc2IeP_l28tfM&e="

This variable isn't used in the new approach

Quoted 61 lines
support/scripts/pkg-stats +46 -0
+RM_API_STATUS_ERROR = 1
+RM_API_STATUS_FOUND_BY_DISTRO = 2
+RM_API_STATUS_FOUND_BY_PATTERN = 3
+RM_API_STATUS_NOT_FOUND = 4
 
 class Package:
     all_licenses = list()
@@ -49,6 +57,7 @@ class Package:
         self.url = None
         self.url_status = None
         self.url_worker = None
+        self.latest_version = None
 
     def pkgvar(self):
         return self.name.upper().replace("-", "_")
@@ -297,6 +306,66 @@ def check_package_urls(packages):
     for pkg in packages:
         pkg.url_status = pkg.url_worker.get(timeout=3600)
 
+def release_monitoring_get_latest_version_by_distro(pool, name):
+    try:
+        req = pool.request('GET', "/api/project/Buildroot/%s" % name)
+    except:
+        return (RM_API_STATUS_ERROR, None, None)
+
+    if req.status != 200:
+        return (RM_API_STATUS_NOT_FOUND, None, None)
+
+    data = json.loads(req.data)
+
+    if len(data['versions']) > 0:
+        return (RM_API_STATUS_FOUND_BY_DISTRO, data['versions'][0], data['id'])
+    else:
+        return (RM_API_STATUS_FOUND_BY_DISTRO, None, data['id'])
+
+def release_monitoring_get_latest_version_by_guess(pool, name):
+    try:
+        req = pool.request('GET', "/api/projects/?pattern=%s" % name)
+    except:
+        print("Exception release_monitoring_get_latest_version_by_guess for '%s': %s" % (name, err))
+        return (RM_API_STATUS_ERROR, None, None)
+
+    if req.status != 200:
+        return (RM_API_STATUS_NOT_FOUND, None, None)
+
+    data = json.loads(req.data)
+
+    for p in data['projects']:
+        if p['name'] == name and len(p['versions']) > 0:
+            return (RM_API_STATUS_FOUND_BY_PATTERN, p['versions'][0], p['id'])
+
+    return (RM_API_STATUS_NOT_FOUND, None, None)
+
+def check_package_latest_version(packages):
+    """
+    Fills in the .latest_version field of all Package objects
+
+    This field has a special format:
+      (status, version, id)
+    with:

support/scripts/pkg-stats def check_package_urls(packages): · line 359

Quoted 10 lines
support/scripts/pkg-stats +10 -0
+    - status: one of RM_API_STATUS_ERROR,
+      RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN,
+      RM_API_STATUS_NOT_FOUND
+    - version: string containing the latest version known by
+      release-monitoring.org for this package
+    - id: string containing the id of the project corresponding to this
+      package, as known by release-monitoring.org
+    """
+    pool = HTTPSConnectionPool('release-monitoring.org', port=443,
+                               cert_reqs='CERT_REQUIRED', ca_certs=certifi.where())

Suggest adding a timeout value (timeout=#). I had used 30 sec for the url checker logic but with this being serial with one main site, maybe shorter would be better?

I still can't get the proxy and certificate stuff to work right to fully test this patchset at work. It should work fine with a "normal" internet connection.

Functionally I can say I have reviewed the patchset with those minor suggestions above.

Signature · 19 lines
Matthew Weber | Pr. Software Engineer | Commercial Avionics

COLLINS AEROSPACE

400 Collins Road NE, Cedar Rapids, Iowa 52498, USA

Tel: +1 319 295 7349 | FAX: +1 319 263 6099

matthew.weber at collins.com | collinsaerospace.com



CONFIDENTIALITY WARNING: This message may contain proprietary and/or
privileged information of Collins Aerospace and its affiliated
companies. If you are not the intended recipient, please 1) Do not
disclose, copy, distribute or use this message or its contents. 2)
Advise the sender by return email. 3) Delete all copies (including all
attachments) from your computer. Your cooperation is greatly
appreciated.
#10 lore raw

PATCH v5 1/1 support/scripts/pkg-stats: add latest upstream version information

In reply to Thomas Petazzoni

Hello,

In overall looks good. But I think a v6 would be good.

Quoted 11 lines · Thomas Petazzoni

This commit adds fetching the latest upstream version of each package from release-monitoring.org.

The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. This mapping mechanism allows to tell release-monitoring.org what is the name of a package in a given distribution/build-system. For example, the package xutil_util-macros in Buildroot is named xorg-util-macros on release-monitoring.org. This mapping can be seen in the section "Mappings" of
https://release-monitoring.org/project/15037/.

Quoted 8 lines

If there is no mapping, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name.

Even though fetching from release-monitoring.org is a bit slow, using multiprocessing.Pool has proven to not be reliable, with some requests ending up with an exception. So we keep a serialized approach, but with a single HTTPSConnectionPool() for all queries. Long term, we

Looks like using urllib3 improves the speed a lot.
To me it took 25 minutes using v5 instead of 2 hours using v3 (also serialized approach).

[snip]

Quoted 5 lines

---
Changes since v4:
- Don't use multiprocessing.Pool(), stick to a serialized approach, which is more reliable.
- Handle errors/exceptions properly.

Almost!
Could you fix the warnings from flake8?
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/141278172

I never used urllib3 but from the docs it seems there is a base exception, so perhaps this line can be useful to avoid bare except:
from urllib3.exceptions import HTTPError

Quoted 1 line

- Improve the layout of the resulting table column.

[snip]

support/scripts/pkg-stats import re · line 35

Quoted 13 lines

+++ b/support/scripts/pkg-stats

support/scripts/pkg-stats +4 -0
@@ -25,11 +25,19 @@ import re
 import subprocess
 import sys
 import requests  # URL checking
+import json
+import certifi
+from urllib3 import HTTPSConnectionPool
 from multiprocessing import Pool
 
 INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
 URL_RE = re.compile("\s*https?://\S*\s*$")
+RELEASE_MONITORING_API = "http://release-monitoring.org/api"

Not used in v5. Matt already mentioned it, but I highlight it because flake8 does not catch it.

Regards,
Ricardo

#11 lore raw

PATCH v5 1/1 support/scripts/pkg-stats: add latest upstream version information

In reply to Thomas Petazzoni

On Thu, Jan 3, 2019 at 2:38 AM Thomas Petazzoni
<[email protected]> wrote:
...

Quoted 5 lines
support/scripts/pkg-stats +5 -0
+    if len(data['versions']) > 0:
+        return (RM_API_STATUS_FOUND_BY_DISTRO, data['versions'][0], data['id'])
+    else:
+        return (RM_API_STATUS_FOUND_BY_DISTRO, None, data['id'])
+

...

support/scripts/pkg-stats def check_package_urls(packages): · line 340

Quoted 4 lines
support/scripts/pkg-stats +4 -0
+    for p in data['projects']:
+        if p['name'] == name and len(p['versions']) > 0:
+            return (RM_API_STATUS_FOUND_BY_PATTERN, p['versions'][0], p['id'])
+

In these two places, instead of using "versions", could we use "version"? It looks like the first item in "versions" isn't always correct. For example see cdrkit. I did a mock API call with Postman and there's a separate "version" field without the prefix.

Here's what a GET to
https://release-monitoring.org/api/projects/?pattern=cdrkit returns:

{
    "projects": [
        {
            "backend": "GitHub",
            "created_on": 1460976471,
            "ecosystem": "https://github.com/Distrotech/cdrkit",
            "homepage": "https://github.com/Distrotech/cdrkit",
            "id": 10520,
            "name": "cdrkit",
            "regex": null,
            "updated_on": 1533644065,
            "version": "1.1.11",
            "version_url": "Distrotech/cdrkit",
            "versions": [
                "release_1.1.11",
                "1.1.11"
            ]
        }
    ],
    "total": 1
}

Otherwise I ran the script on my setup and everything looks good.