support/scripts/pkg-stats: URL checking support
PATCH 1/14 support/scripts/pkg-stats: URL checking support
- Adds support to check if a package has a URL and if that URL is valid by doing a header request.
- Reports this information as part of the generated html output
This check helps ensure the URLs are valid and can be used for other scripting purposes as the product's home site/URL. CPE XML generation is an example of a case that could use this product URL as part of an automated update generation script.
--- support/scripts/pkg-stats | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-)
support/scripts/pkg-stats +36 -1
@@ -24,6 +24,8 @@ from collections import defaultdict import re import subprocess import sys +import time +import requests # URL checking INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
@@ -43,10 +45,30 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.url = None def pkgvar(self): return self.name.upper().replace("-", "_") + def set_url(self): + """ + Fills in the .url field + """ + in_help_section = False + self.url = "No Config" + for filename in os.listdir(os.path.dirname(self.path)): + if fnmatch.fnmatch(filename, 'Config.*'): + fp = open(os.path.join(os.path.dirname(self.path), filename), "r") + for config_line in fp: + if config_line.strip() == "help": + in_help_section = True + if in_help_section and re.match("(.*)https?://", config_line): + self.url = ''.join(config_line.split()) + fp.close() + return + self.url = "Missing Entry" + fp.close() + def set_infra(self): """ Fills in the .infras field
@@ -356,7 +378,19 @@ def boolean_str(b): def dump_html_pkg(f, pkg): f.write(" <tr>\n") - f.write(" <td>%s</td>\n" % pkg.path[2:]) + url_status = "Ok" + if str(pkg.url) == "Missing Entry": + f.write(" <td>%s<br> (URL: Missing URL)</td>\n" % pkg.path[2:]) + elif str(pkg.url) == "No Config": + f.write(" <td>%s<br> (URL: No Config File)</td>\n" % pkg.path[2:]) + else: + try: + url_status_code = requests.head(pkg.url, verify=False).status_code + if url_status_code > 308: + url_status = "Error(" + str(url_status_code) + ")" + except requests.exceptions.RequestException as e: + url_status = e + f.write(" <td>%s<br> (<a href=%s>URL: %s</a>)</td>\n" % (pkg.path[2:], str(pkg.url), url_status)) # Patch count td_class = ["centered"]
@@ -511,6 +545,7 @@ def __main__(): package_init_make_info() print("Getting package details ...") for pkg in packages: + pkg.set_url() pkg.set_infra() pkg.set_license() pkg.set_hash_info()
Signature · 1 line
1.9.1
PATCH 2/14 boot/at91bootstrap: URL update
--- boot/at91bootstrap/Config.in | 2 ++ 1 file changed, 2 insertions(+)
boot/at91bootstrap/Config.in +2 -0
@@ -10,6 +10,8 @@ config BR2_TARGET_AT91BOOTSTRAP - Physical media algorithm such as DataFlash, NandFlash, NOR Flash... + https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap + if BR2_TARGET_AT91BOOTSTRAP config BR2_TARGET_AT91BOOTSTRAP_CUSTOM_PATCH_DIR
Signature · 1 line
1.9.1
PATCH 2/14 boot/at91bootstrap: URL update
Hello,
Quoted 10 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- boot/at91bootstrap/Config.in | 2 ++ 1 file changed, 2 insertions(+)boot/at91bootstrap/Config.in +0
-0
boot/at91bootstrap/Config.in config BR2_TARGET_AT91BOOTSTRAP · line 13
Quoted 4 lines
boot/at91bootstrap/Config.in +1
-0@@ -10,6 +10,8 @@ config BR2_TARGET_AT91BOOTSTRAP - Physical media algorithm such as DataFlash, NandFlash, NOR Flash... + https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
Actually, this one should use:
https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
which you have used for at91dataflashboot, but at91dataflashboot is a different thing, which no longer has any upstream location.
Perhaps your script should support a way of specifying "yeah, there is no upstream location" in an explicit way.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 2/14 boot/at91bootstrap: URL update
Quoted 17 lines · Thomas Petazzoni
Hello,
Quoted 13 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- boot/at91bootstrap/Config.in | 2 ++ 1 file changed, 2 insertions(+)boot/at91bootstrap/Config.in +0
-0@@ -10,6 +10,8 @@ config BR2_TARGET_AT91BOOTSTRAP - Physical media algorithm such as DataFlash, NandFlash, NOR Flash...
Quoted 6 lines
boot/at91bootstrap/Config.in config BR2_TARGET_AT91BOOTSTRAP · line 13
Quoted 1 line
boot/at91bootstrap/Config.in +1
-0+ https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91BootstrapActually, this one should use:
https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
Ok so this patch is ok then and for the at91dataflashboot I found it mentioned on the same legacy page but it definitely is a weak reference.
Quoted 5 lines
which you have used for at91dataflashboot, but at91dataflashboot is a different thing, which no longer has any upstream location.
Perhaps your script should support a way of specifying "yeah, there is no upstream location" in an explicit way.
Is that maybe a string we put in the end of the help section? "No
Upstream URL"
Matt
PATCH 2/14 boot/at91bootstrap: URL update
Hello,
Quoted 8 lines · Matthew Weber
Quoted 8 lines
Quoted 8 lines
boot/at91bootstrap/Config.in +0
-0@@ -10,6 +10,8 @@ config BR2_TARGET_AT91BOOTSTRAP - Physical media algorithm such as DataFlash, NandFlash, NOR Flash...
Quoted 8 lines
Quoted 6 lines
boot/at91bootstrap/Config.in config BR2_TARGET_AT91BOOTSTRAP · line 13
Quoted 1 line
boot/at91bootstrap/Config.in +1
-0+ https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91BootstrapActually, this one should use:
https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
Ok so this patch is ok then
Ah, yes, my bad, sorry :-/
Quoted 2 lines
and for the at91dataflashboot I found it mentioned on the same legacy page but it definitely is a weak reference.
I sent an e-mail to Atmel^WMicrochip people about at91dataflashboot, we'll see what they have to say.
Quoted 5 lines
Quoted 2 lines
Perhaps your script should support a way of specifying "yeah, there is no upstream location" in an explicit way.
Is that maybe a string we put in the end of the help section? "No
Upstream URL"
Yes, I was thinking about something like that.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/14 boot/at91bootstrap3: URL update
In reply to Matt Weber--- boot/at91bootstrap3/Config.in | 3 +++ 1 file changed, 3 insertions(+)
boot/at91bootstrap3/Config.in +3 -0
@@ -10,6 +10,9 @@ config BR2_TARGET_AT91BOOTSTRAP3 - Physical media algorithm such as DataFlash, NandFlash, NOR Flash... + + https://www.at91.com/linux4sam/bin/view/Linux4SAM/AT91Bootstrap + if BR2_TARGET_AT91BOOTSTRAP3 choice
Signature · 1 line
1.9.1
PATCH 3/14 boot/at91bootstrap3: URL update
Hello,
Quoted 10 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- boot/at91bootstrap3/Config.in | 3 +++ 1 file changed, 3 insertions(+)boot/at91bootstrap3/Config.in +0
-0
boot/at91bootstrap3/Config.in config BR2_TARGET_AT91BOOTSTRAP3 · line 13
Quoted 4 lines
boot/at91bootstrap3/Config.in +1
-0@@ -10,6 +10,9 @@ config BR2_TARGET_AT91BOOTSTRAP3 - Physical media algorithm such as DataFlash, NandFlash, NOR Flash... +
One too many newline.
boot/at91bootstrap3/Config.in config BR2_TARGET_AT91BOOTSTRAP3 · line 14
Quoted 1 line
boot/at91bootstrap3/Config.in +1
-0+ https://www.at91.com/linux4sam/bin/view/Linux4SAM/AT91Bootstrap
And the commit title was wrong: you're not updating the URL, but adding it.
Applied after fixing those issues.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/14 boot/at91bootstrap3: URL update
In reply to Matt WeberQuoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
Signed-off-by Matt Weber <[email protected]>
Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 4/14 boot/at91dataflashboot: URL update
In reply to Matt Weber--- boot/at91dataflashboot/Config.in | 2 ++ 1 file changed, 2 insertions(+)
boot/at91dataflashboot/Config.in +2 -0
@@ -1,3 +1,5 @@ config BR2_TARGET_AT91DATAFLASHBOOT bool "AT91 DataFlashBoot" depends on BR2_arm926t + help + https://www.at91.com/linux4sam/bin/view/Linux4SAM/LegacyAT91Bootstrap
Signature · 1 line
1.9.1
PATCH 5/14 boot/xloader: URL update
In reply to Matt Weber--- boot/xloader/Config.in | 2 ++ 1 file changed, 2 insertions(+)
boot/xloader/Config.in +2 -0
@@ -5,6 +5,8 @@ config BR2_TARGET_XLOADER The x-loader bootloader. It is mainly used on OMAP-based platforms. + http://omappedia.org/wiki/Linux_OMAP_Kernel_Main + if BR2_TARGET_XLOADER config BR2_TARGET_XLOADER_BOARDNAME string "x-loader board name"
Signature · 1 line
1.9.1
PATCH 5/14 boot/xloader: URL update
Hello,
Quoted 4 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- boot/xloader/Config.in | 2 ++ 1 file changed, 2 insertions(+)
I've fixed the commit title: you're adding an upstream URL, not updating it, and I've applied.
However:
- The link isn't really a useful upstream URL. There in fact doesn't seem to be any page that is really relevant for X-Loader.
- I am not sure having a package for xloader still makes sense. It's completely deprecated now that U-Boot has SPL support for TI platforms. Perhaps we should simply remove it ?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 5/14 boot/xloader: URL update
In reply to Matt WeberQuoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
Signed-off-by Matt Weber <[email protected]>
Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 6/14 package/android-tools: URL update
In reply to Matt Weber--- package/android-tools/Config.in | 2 ++ 1 file changed, 2 insertions(+)
package/android-tools/Config.in +2 -0
@@ -15,6 +15,8 @@ config BR2_PACKAGE_ANDROID_TOOLS can be used to interact with target devices using of these protocols. + https://wiki.debian.org/AndroidTools#Original_android-tools_package + if BR2_PACKAGE_ANDROID_TOOLS # We need kernel headers that support the __SANE_USERSPACE_TYPES__
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH 6/14 package/android-tools: URL update
Hello,
Quoted 4 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- package/android-tools/Config.in | 2 ++ 1 file changed, 2 insertions(+)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 6/14 package/android-tools: URL update
In reply to Matt WeberQuoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
Signed-off-by Matt Weber <[email protected]>
Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 7/14 package/arp-scan: URL update
In reply to Matt Weber--- package/arp-scan/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/arp-scan/Config.in +1 -1
@@ -7,4 +7,4 @@ config BR2_PACKAGE_ARP_SCAN arp-scan is a command-line tool that uses the ARP protocol to discover and fingerprint IP hosts on the local network. - http://www.nta-monitor.com/wiki/index.php/Arp-scan_Documentation + https://github.com/royhills/arp-scan
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH 7/14 package/arp-scan: URL update
Hello,
Quoted 4 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- package/arp-scan/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 7/14 package/arp-scan: URL update
In reply to Matt WeberQuoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
Signed-off-by Matt Weber <[email protected]>
Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 8/14 package/bandwidthd: URL move to newline
In reply to Matt WeberAllows scripting of URL checking to be simplier
--- package/bandwidthd/Config.in | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
package/bandwidthd/Config.in +4 -2
@@ -25,8 +25,10 @@ config BR2_PACKAGE_BANDWIDTHD available on github that works on making BandwidthD's build process more compatible with buildroot's. - Upstream: http://bandwidthd.sourceforge.net/ - Github fork: http://github.com/nroach44/bandwidthd + Upstream: + http://bandwidthd.sourceforge.net/ + Github fork: + http://github.com/nroach44/bandwidthd if BR2_PACKAGE_BANDWIDTHD
Signature · 1 line
1.9.1
Thomas Petazzoni
AppliedPATCH 8/14 package/bandwidthd: URL move to newline
Hello,
Quoted 6 lines · Matt Weber
Allows scripting of URL checking to be simplier
Signed-off-by Matt Weber <[email protected]>--- package/bandwidthd/Config.in | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 8/14 package/bandwidthd: URL move to newline
In reply to Matt WeberQuoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
Allows scripting of URL checking to be simplier
Signed-off-by Matt Weber <[email protected]>
Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 9/14 package/connman: URL move to newline
In reply to Matt WeberAllows scripting of URL checking to be simplier
--- package/connman/Config.in | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
package/connman/Config.in +2 -1
@@ -14,7 +14,8 @@ config BR2_PACKAGE_CONNMAN for managing internet connections within embedded devices running the Linux operating system. - For more information, see https://01.org/connman + For more information, see + https://01.org/connman if BR2_PACKAGE_CONNMAN
Signature · 1 line
1.9.1
PATCH 9/14 package/connman: URL move to newline
Hello,
Quoted 12 lines · Matt Weber
Allows scripting of URL checking to be simplier
Signed-off-by Matt Weber <[email protected]>--- package/connman/Config.in | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)package/connman/Config.in +0
-0
package/connman/Config.in config BR2_PACKAGE_CONNMAN · line 18
Quoted 6 lines
package/connman/Config.in +2
-1@@ -14,7 +14,8 @@ config BR2_PACKAGE_CONNMAN for managing internet connections within embedded devices running the Linux operating system. - For more information, see https://01.org/connman + For more information, see + https://01.org/connman
Actually, the "For more information, see" is totally useless, so I've dropped that instead to solve the problem of "we want the URL at the beginning of a line".
Applied with this change. Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 9/14 package/connman: URL move to newline
In reply to Matt WeberQuoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
Allows scripting of URL checking to be simplier
Signed-off-by Matt Weber <[email protected]>
Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 10/14 package/glibc: URL update
In reply to Matt Weber--- package/glibc/Config.in | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
package/glibc/Config.in +2 -1
@@ -5,5 +5,6 @@ config BR2_PACKAGE_GLIBC default y select BR2_PACKAGE_LINUX_HEADERS select BR2_TOOLCHAIN_HAS_SSP - + help + https://www.gnu.org/software/libc/ endif
Signature · 1 line
1.9.1
PATCH 10/14 package/glibc: URL update
Hello,
Quoted 10 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- package/glibc/Config.in | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)package/glibc/Config.in +0
-0
package/glibc/Config.in config BR2_PACKAGE_GLIBC · line 9
Quoted 6 lines
package/glibc/Config.in +2
-1@@ -5,5 +5,6 @@ config BR2_PACKAGE_GLIBC default y select BR2_PACKAGE_LINUX_HEADERS select BR2_TOOLCHAIN_HAS_SSP - + help + https://www.gnu.org/software/libc/
This option is a blind option, so having a help text is not very useful...
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 10/14 package/glibc: URL update
Thomas,
Quoted 9 lines · Thomas Petazzoni
Hello,
Quoted 5 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- package/glibc/Config.in | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 14 lines
package/glibc/Config.in config BR2_PACKAGE_GLIBC · line 9
Quoted 11 lines
package/glibc/Config.in +2
-1@@ -5,5 +5,6 @@ config BR2_PACKAGE_GLIBC default y select BR2_PACKAGE_LINUX_HEADERS select BR2_TOOLCHAIN_HAS_SSP - + help + https://www.gnu.org/software/libc/This option is a blind option, so having a help text is not very useful...
I had been looking at verifying and updating all packages which could end-up with content on target to have a valid URL. This allows auto-generation of CPE XML update requests which need product site urls.
Matt
PATCH 10/14 package/glibc: URL update
Hello,
Quoted 7 lines · Matthew Weber
Quoted 2 lines
This option is a blind option, so having a help text is not very useful...
I had been looking at verifying and updating all packages which could end-up with content on target to have a valid URL. This allows auto-generation of CPE XML update requests which need product site urls.
Then perhaps this should be explained in the commit log, specifically for glibc.
But are you sure your approach is the right one ? Today only target packages have a Config.in file. What are you going to do for host packages ? You don't care because they are host packages and supposedly not relevant for security issues on the target ?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 10/14 package/glibc: URL update
Thomas,
Quoted 8 lines · Thomas Petazzoni
Hello,
Quoted 3 lines · Matthew Weber
Quoted 2 lines
This option is a blind option, so having a help text is not very useful...
Quoted 8 lines
Quoted 4 lines
I had been looking at verifying and updating all packages which could end-up with content on target to have a valid URL. This allows auto-generation of CPE XML update requests which need product site urls.
Then perhaps this should be explained in the commit log, specifically for glibc.
Yes, agree.
Quoted 3 lines
But are you sure your approach is the right one ? Today only target packages have a Config.in file. What are you going to do for host packages ?
Currently the xml generation was a best effort and the cpe reporting was only done against target CPE so far..... Glad to discuss suggestions for a better way vs hand updating the xml with this URL info.
Quoted 2 lines
You don't care because they are host packages and supposedly not relevant for security issues on the target ?
I know better then to say that host packages don't matter :-) We took the most accessible path first and that was to get good coverage on target packages.
Maybe a good topic for next month? I should have an updated CPE patchset out in the next week.
Matt
PATCH 11/14 package/tidsp-binaries: URL update
In reply to Matt Weber--- package/tidsp-binaries/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/tidsp-binaries/Config.in +1 -1
@@ -4,4 +4,4 @@ config BR2_PACKAGE_TIDSP_BINARIES help TI OMAP3 DSP algorithms. - https://gforge.ti.com/gf/project/openmax/ + http://omappedia.org/wiki/OpenMAX_Project
Signature · 1 line
1.9.1
PATCH 11/14 package/tidsp-binaries: URL update
Hello,
Quoted 10 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- package/tidsp-binaries/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/tidsp-binaries/Config.in +0
-0
package/tidsp-binaries/Config.in config BR2_PACKAGE_TIDSP_BINARIES · line 7
Quoted 5 lines
package/tidsp-binaries/Config.in +1
-1@@ -4,4 +4,4 @@ config BR2_PACKAGE_TIDSP_BINARIES help TI OMAP3 DSP algorithms. - https://gforge.ti.com/gf/project/openmax/ + http://omappedia.org/wiki/OpenMAX_Project
Is that really the right URL ? It points to gforge.ti.com, which indeed seems to be down. Should we simply drop this package perhaps ? The upstream site is not available, I doubt anybody is using it: it was introduced in 2011 and never touched again in any useful/meaningful way.
So I would propose to drop this package.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 11/14 package/tidsp-binaries: URL update
Thomas,
Quoted 25 lines · Thomas Petazzoni
Hello,
Quoted 10 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- package/tidsp-binaries/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/tidsp-binaries/Config.in +0
-0package/tidsp-binaries/Config.in config BR2_PACKAGE_TIDSP_BINARIES · line 7
Quoted 5 lines
package/tidsp-binaries/Config.in +1
-1@@ -4,4 +4,4 @@ config BR2_PACKAGE_TIDSP_BINARIES help TI OMAP3 DSP algorithms. - https://gforge.ti.com/gf/project/openmax/ + http://omappedia.org/wiki/OpenMAX_ProjectIs that really the right URL ? It points to gforge.ti.com, which indeed seems to be down. Should we simply drop this package perhaps ? The upstream site is not available, I doubt anybody is using it: it was introduced in 2011 and never touched again in any useful/meaningful way.
Quoted 1 line
So I would propose to drop this package.
This package is used by dsp-tools and gstreamer/gst-dsp so I'd assume those should also be removed.
This capability feels feature complete when looking at the Gstreamer stuff.... I guess we'll see if anyone notices the removal. Should I have 3 seperate patches for the individual packages?
Matt
PATCH 12/14 boot/vexpress-firmware: URL update
In reply to Matt Weber--- boot/vexpress-firmware/Config.in | 2 ++ 1 file changed, 2 insertions(+)
boot/vexpress-firmware/Config.in +2 -0
@@ -4,3 +4,5 @@ config BR2_TARGET_VEXPRESS_FIRMWARE help Versatile Express firmware from ARM, with Linaro mods last change. + + https://git.linaro.org/arm/vexpress-firmware.git
Signature · 1 line
1.9.1
PATCH 12/14 boot/vexpress-firmware: URL update
Hello,
Quoted 4 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- boot/vexpress-firmware/Config.in | 2 ++ 1 file changed, 2 insertions(+)
The title says "URL update", but that's not true: this commit is adding the URL. A better commit title:
boot/vexpress-firmware: add upstream URL in Config.in help text
I fixed that and applied. Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 12/14 boot/vexpress-firmware: URL update
In reply to Matt WeberQuoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Matt" == Matt Weber <[email protected]> writes:
Signed-off-by Matt Weber <[email protected]>
Committed to 2018.02.x, 2018.05.x and 2018.08.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 13/14 package/alljoyn-base: URL update
In reply to Matt Weber--- package/alljoyn-base/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/alljoyn-base/Config.in +1 -1
@@ -17,7 +17,7 @@ config BR2_PACKAGE_ALLJOYN_BASE devices, providing a set of interfaces for different devices to interact and interoperate with one another. - https://allseenalliance.org + https://github.com/alljoyn/alljoyn.github.com/wiki if BR2_PACKAGE_ALLJOYN_BASE
Signature · 1 line
1.9.1
PATCH 13/14 package/alljoyn-base: URL update
Hello,
Quoted 10 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- package/alljoyn-base/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/alljoyn-base/Config.in +0
-0
package/alljoyn-base/Config.in config BR2_PACKAGE_ALLJOYN_BASE · line 20
Quoted 5 lines
package/alljoyn-base/Config.in +1
-1@@ -17,7 +17,7 @@ config BR2_PACKAGE_ALLJOYN_BASE devices, providing a set of interfaces for different devices to interact and interoperate with one another. - https://allseenalliance.org + https://github.com/alljoyn/alljoyn.github.com/wiki
For PATCH 13/14 (this one) and PATCH 14/14, I'd like something a bit better:
- The .mk files still use a mirror of allseenalliance at kernel.org, it should be changed to use the Github location instead.
- You changed alljoyn-base and alljoyn-tcl-base, but not alljoyn-tcl
- The commit title should be "update upstream URL"
- The commit log should contain some details, at least explaining that allseenalliance.org is dead, and that the project was moved to Github, etc.
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 13/14 package/alljoyn-base: URL update
Thomas/Fabrice,
Quoted 17 lines · Thomas Petazzoni
Hello,
Quoted 13 lines · Matt Weber
Signed-off-by Matt Weber <[email protected]>--- package/alljoyn-base/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/alljoyn-base/Config.in +0
-0@@ -17,7 +17,7 @@ config BR2_PACKAGE_ALLJOYN_BASE devices, providing a set of interfaces for different devices to interact and interoperate with one another.
Quoted 8 lines
package/alljoyn-base/Config.in config BR2_PACKAGE_ALLJOYN_BASE · line 20
Quoted 2 lines
package/alljoyn-base/Config.in +1
-1- https://allseenalliance.org + https://github.com/alljoyn/alljoyn.github.com/wikiFor PATCH 13/14 (this one) and PATCH 14/14, I'd like something a bit better:
- The .mk files still use a mirror of allseenalliance at kernel.org, it should be changed to use the Github location instead.
I don't have a good use case to runtime test making this change. I can make it an verified the build is still good.
Quoted 2 lines
- You changed alljoyn-base and alljoyn-tcl-base, but not alljoyn-tcl
I'll have to check, my script may have a bug. My grep is coming back showing I missed the plain alljoyn too.
alljoyn-base https://github.com/alljoyn/alljoyn.github.com/wiki
alljoyn https://allseenalliance.org
alljoyn-tcl-base https://github.com/alljoyn/alljoyn.github.com/wiki
alljoyn-tcl https://allseenalliance.org
Quoted 2 lines
- The commit title should be "update upstream URL"
Noted.
Quoted 4 lines
- The commit log should contain some details, at least explaining that allseenalliance.org is dead, and that the project was moved to Github, etc.
Sure, I'm not to familiar with this one but I can at least state that.
Matt
PATCH 13/14 package/alljoyn-base: URL update
Hello,
Quoted 5 lines · Matthew Weber
Quoted 2 lines
- The .mk files still use a mirror of allseenalliance at kernel.org, it should be changed to use the Github location instead.
I don't have a good use case to runtime test making this change. I can make it an verified the build is still good.
I think we should try to keep using the same versions (we use 16.04, and I saw it was available on Github as well), and preferably try to diff the source code between the old upstream and the new upstream. If the code is reasonably similar, then build testing should be sufficient.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 14/14 package/alljoyn-tcl-base: URL update
In reply to Matt Weber--- package/alljoyn-tcl-base/Config.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/alljoyn-tcl-base/Config.in +1 -1
@@ -12,7 +12,7 @@ config BR2_PACKAGE_ALLJOYN_TCL_BASE AllJoyn distributed programming environment to embedded systems. - https://allseenalliance.org + https://github.com/alljoyn/alljoyn.github.com/wiki comment "alljoyn-tcl-base needs a toolchain w/ threads and dynamic library" depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_STATIC_LIBS
Signature · 1 line
1.9.1
PATCH 1/14 support/scripts/pkg-stats: URL checking support
In reply to Matt WeberHello Matt,
Quoted 52 lines · Matt Weber
support/scripts/pkg-stats +25
-1@@ -24,6 +24,8 @@ from collections import defaultdict import re import subprocess import sys +import time +import requests # URL checking INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")@@ -43,10 +45,30 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.url = None def pkgvar(self): return self.name.upper().replace("-", "_") + def set_url(self): + """ + Fills in the .url field + """ + in_help_section = False + self.url = "No Config" + for filename in os.listdir(os.path.dirname(self.path)): + if fnmatch.fnmatch(filename, 'Config.*'): + fp = open(os.path.join(os.path.dirname(self.path), filename), "r") + for config_line in fp: + if config_line.strip() == "help": + in_help_section = True + if in_help_section and re.match("(.*)https?://", config_line): + self.url = ''.join(config_line.split()) + fp.close() + return + self.url = "Missing Entry" + fp.close() + def set_infra(self): """ Fills in the .infras field@@ -356,7 +378,19 @@ def boolean_str(b): def dump_html_pkg(f, pkg): f.write(" <tr>\n") - f.write(" <td>%s</td>\n" % pkg.path[2:]) + url_status = "Ok" + if str(pkg.url) == "Missing Entry": + f.write(" <td>%s<br> (URL: Missing URL)</td>\n" % pkg.path[2:])
support/scripts/pkg-stats def boolean_str(b): · line 393
Quoted 10 lines
support/scripts/pkg-stats +10
-0+ elif str(pkg.url) == "No Config": + f.write(" <td>%s<br> (URL: No Config File)</td>\n" % pkg.path[2:]) + else: + try: + url_status_code = requests.head(pkg.url, verify=False).status_code + if url_status_code > 308: + url_status = "Error(" + str(url_status_code) + ")" + except requests.exceptions.RequestException as e: + url_status = e + f.write(" <td>%s<br> (<a href=%s>URL: %s</a>)</td>\n" % (pkg.path[2:], str(pkg.url), url_status))
Could you instead add a new column, like we already have for the version, package type, number of patches, etc. and use the existing green/orange/red colors to indicate URL present and working (green), URL not present (orange) and URL present but not working (red) ?
How long does it take to run the script before/after your addition, on all packages ? I'm sure you remember I was working on adding support for using release-monitoring.org to this script to keep track of upstream versions of packages, but doing it sequentially for the 2000+ packages we have was way too slow and I had to use several threads to speed things up and make it reasonable.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/14 support/scripts/pkg-stats: URL checking support
Quoted 64 lines · Thomas Petazzoni
Hello Matt,
Quoted 59 lines · Matt Weber
support/scripts/pkg-stats +32
-1@@ -24,6 +24,8 @@ from collections import defaultdict import re import subprocess import sys +import time +import requests # URL checking INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")@@ -43,10 +45,30 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.url = None def pkgvar(self): return self.name.upper().replace("-", "_") + def set_url(self): + """ + Fills in the .url field + """ + in_help_section = False + self.url = "No Config" + for filename in os.listdir(os.path.dirname(self.path)): + if fnmatch.fnmatch(filename, 'Config.*'): + fp = open(os.path.join(os.path.dirname(self.path), filename), "r") + for config_line in fp: + if config_line.strip() == "help": + in_help_section = True + if in_help_section and re.match("(.*)https?://", config_line): + self.url = ''.join(config_line.split()) + fp.close() + return + self.url = "Missing Entry" + fp.close() + def set_infra(self): """ Fills in the .infras field@@ -356,7 +378,19 @@ def boolean_str(b): def dump_html_pkg(f, pkg): f.write(" <tr>\n") - f.write(" <td>%s</td>\n" % pkg.path[2:]) + url_status = "Ok" + if str(pkg.url) == "Missing Entry": + f.write(" <td>%s<br> (URL: Missing URL)</td>\n" % pkg.path[2:]) + elif str(pkg.url) == "No Config": + f.write(" <td>%s<br> (URL: No Config File)</td>\n" % pkg.path[2:]) + else: + try: + url_status_code = requests.head(pkg.url, verify=False).status_code + if url_status_code > 308: + url_status = "Error(" + str(url_status_code) + ")"
Quoted 9 lines
support/scripts/pkg-stats def boolean_str(b): · line 393
Quoted 3 lines
support/scripts/pkg-stats +3
-0+ except requests.exceptions.RequestException as e: + url_status = e + f.write(" <td>%s<br> (<a href=%s>URL: %s</a>)</td>\n" % (pkg.path[2:], str(pkg.url), url_status))Could you instead add a new column, like we already have for the version, package type, number of patches, etc. and use the existing green/orange/red colors to indicate URL present and working (green), URL not present (orange) and URL present but not working (red) ?
How does this look? (Still some bugs with some of the invalid sites, the webservers are reporting incorrect codes....)
https://pste.eu/p/JpsH.html
Quoted 7 lines
How long does it take to run the script before/after your addition, on all packages ? I'm sure you remember I was working on adding support for using release-monitoring.org to this script to keep track of upstream versions of packages, but doing it sequentially for the 2000+ packages we have was way too slow and I had to use several threads to speed things up and make it reasonable.
Before adding URL checking, it takes ~2mins. I was able to keep the runtime at 2mins 20sec by pulling in your code for threading and using 64 threads.
Matt
PATCH 1/14 support/scripts/pkg-stats: URL checking support
Hello,
Quoted 3 lines · Matthew Weber
How does this look? (Still some bugs with some of the invalid sites, the webservers are reporting incorrect codes....)
https://pste.eu/p/JpsH.html
Looks good!
Quoted 3 lines
Before adding URL checking, it takes ~2mins. I was able to keep the runtime at 2mins 20sec by pulling in your code for threading and using 64 threads.
I think my threading stuff had some issues, which were reported by Ricardo, and are the reason why my "upstream version checking" stuff was not merged.
I'd say, let's merge a first version without threading, and we'll see at adding threading later.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/14 support/scripts/pkg-stats: URL checking support
Thomas,
Quoted 15 lines · Thomas Petazzoni
Hello,
Quoted 3 lines · Matthew Weber
How does this look? (Still some bugs with some of the invalid sites, the webservers are reporting incorrect codes....)
https://pste.eu/p/JpsH.htmlLooks good!
Quoted 3 lines
Before adding URL checking, it takes ~2mins. I was able to keep the runtime at 2mins 20sec by pulling in your code for threading and using 64 threads.
Quoted 7 lines
I think my threading stuff had some issues, which were reported by Ricardo, and are the reason why my "upstream version checking" stuff was not merged.
I'd say, let's merge a first version without threading, and we'll see at adding threading later.
Sounds good. I'll submit it in two pieces, the first without threading and the second, adding the threading as a general feature you can add various tasks to and run them in parallel.
Matt