Skip to contentSkip to conversation

fetch2/wget.py: use default wget user-agent in upstream version checks

PATCH In review Alexander Kanavin 2 participants 3 messages Oct 14, 2024

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 fetch2/wget.py: use default wget user-agent in upstream version checks

From: Alexander Kanavin <[email protected]>

This has been set to various mozilla versions from this point onwards without explanation or specific examples:
https://git.yoctoproject.org/poky/commit/?h=master-next&id=ba06cc8943d021f406c3d292ce415b77319ca5c6

However, sourceforge has recently banned the current 'fake mozilla' version we've been supplying, resulting in broken version checks:

alex@Zen2:/srv/storage/alex/yocto/build-test-sourceforge$ devtool check-upgrade-status gptfdisk
...
--2024-10-12 11:53:23--  https://sourceforge.net/projects/gptfdisk/files/gptfdisk/
Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ...
Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-10-12 11:53:24 ERROR 403: Forbidden.

gptfdisk 1.0.10 UNKNOWN_BROKEN Alexander Kanavin <[email protected]>

On the other hand, using wget's default user-agent works just fine:

--2024-10-12 12:16:06-- https://sourceforge.net/projects/gptfdisk/files/gptfdisk/
Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

---
 bitbake/lib/bb/fetch2/wget.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/bb/fetch2/wget.py +1 -1
index e8b1292433f..493a5b62ee2 100644
@@ -463,7 +463,7 @@ class Wget(FetchMethod):
         f = tempfile.NamedTemporaryFile()
         with tempfile.TemporaryDirectory(prefix="wget-index-") as workdir, tempfile.NamedTemporaryFile(dir=workdir, prefix="wget-listing-") as f:
             fetchcmd = self.basecmd
-            fetchcmd += " -O " + f.name + " --user-agent='" + self.user_agent + "' '" + uri + "'"
+            fetchcmd += " -O " + f.name + " '" + uri + "'"
             try:
                 self._runwget(ud, d, fetchcmd, True, workdir=workdir)
                 fetchresult = f.read()
Signature · 1 line
2.39.5
#1 lore raw

PATCH 2/2 meta: use explicit uri for all sourceforge upstream version checks

From: Alexander Kanavin <[email protected]>

Previously the check for some recipes relied on sourceforge redirecting from downloads.sourceforge.net (SRC_URI is set to that) into the actual project page.

Sourceforge does this for interactive browsers, but not for wget.

With the check no longer mimicking a browser, and being truthful about coming from wget we need to explicitly fetch
the project page in all cases, which is what this commit does. (many recipes already set this explicitly and don't need to be tweaked)

---
 meta/classes-recipe/sourceforge-releases.bbclass            | 2 ++
 meta/recipes-bsp/acpid/acpid.inc                            | 5 +++--
 meta/recipes-bsp/gnu-efi/gnu-efi_3.0.18.bb                  | 2 ++
 meta/recipes-bsp/setserial/setserial_2.17.bb                | 2 +-
 meta/recipes-devtools/cdrtools/cdrtools-native_3.01.bb      | 2 +-
 meta/recipes-extended/cwautomacros/cwautomacros_20110201.bb | 5 ++++-
 meta/recipes-extended/hdparm/hdparm_9.65.bb                 | 1 +
 meta/recipes-extended/mingetty/mingetty_1.08.bb             | 2 +-
 meta/recipes-extended/rpcbind/rpcbind_1.2.7.bb              | 2 +-
 meta/recipes-multimedia/lame/lame_3.100.bb                  | 2 +-
 meta/recipes-support/libfm/libfm-extra_1.3.2.bb             | 5 +++--
 meta/recipes-support/libfm/libfm_1.3.2.bb                   | 5 +++--
 meta/recipes-support/libpcre/libpcre_8.45.bb                | 5 +++--
 13 files changed, 26 insertions(+), 14 deletions(-)
 create mode 100644 meta/classes-recipe/sourceforge-releases.bbclass
meta/classes-recipe/sourceforge-releases.bbclass new file +2 -0
new file mode 100644
index 00000000000..0b5e5d07119
@@ -0,0 +1,2 @@
+SOURCEFORGE_PROJECT ?= "${BPN}"
+UPSTREAM_CHECK_URI = "https://sourceforge.net/projects/${SOURCEFORGE_PROJECT}/files/"
meta/recipes-bsp/acpid/acpid.inc +3 -2
index 3d6f75f90f8..ba954563b67 100644
@@ -10,7 +10,8 @@ BUGTRACKER = "http://sourceforge.net/p/acpid2/tickets/?source=navbar"
 SECTION = "base"
 LICENSE = "GPL-2.0-or-later"
 
-SRC_URI = "${SOURCEFORGE_MIRROR}/acpid2/acpid-${PV}.tar.xz \
+SOURCEFORGE_PROJECT = "acpid2"
+SRC_URI = "${SOURCEFORGE_MIRROR}/${SOURCEFORGE_PROJECT}/acpid-${PV}.tar.xz \
            file://init \
            file://acpid.service \
            file://0001-Replace-stat64-with-stat.patch \
@@ -18,7 +19,7 @@ SRC_URI = "${SOURCEFORGE_MIRROR}/acpid2/acpid-${PV}.tar.xz \
 
 CVE_PRODUCT = "acpid2"
 
-inherit autotools update-rc.d systemd
+inherit autotools update-rc.d systemd sourceforge-releases
 
 INITSCRIPT_NAME = "acpid"
 INITSCRIPT_PARAMS = "defaults"
meta/recipes-bsp/gnu-efi/gnu-efi_3.0.18.bb +2 -0
index a56a85bdeff..8257190568f 100644
@@ -18,6 +18,8 @@ SRC_URI = "${SOURCEFORGE_MIRROR}/${BPN}/files/${BP}.tar.bz2 \
            "
 SRC_URI[sha256sum] = "7f212c96ee66547eeefb531267b641e5473d7d8529f0bd8ccdefd33cf7413f5c"
 
+inherit sourceforge-releases
+
 COMPATIBLE_HOST = "(x86_64.*|i.86.*|aarch64.*|arm.*|riscv64.*)-linux"
 COMPATIBLE_HOST:armv4 = 'null'
 
meta/recipes-bsp/setserial/setserial_2.17.bb +1 -1
index 98ab45145ff..3241b21f82e 100644
@@ -8,7 +8,7 @@ LIC_FILES_CHKSUM = "file://version.h;beginline=1;endline=6;md5=2e7c59cb9e57e356a
 
 DEPENDS += "groff-native"
 
-inherit autotools-brokensep
+inherit autotools-brokensep sourceforge-releases
 
 SRC_URI = "${SOURCEFORGE_MIRROR}/setserial/${BPN}-${PV}.tar.gz \
            file://add_stdlib.patch \
meta/recipes-devtools/cdrtools/cdrtools-native_3.01.bb +1 -1
index 9dc5caf87c3..085384064d0 100644
@@ -25,7 +25,7 @@ EXTRA_OEMAKE = "-e MAKEFLAGS= CPPOPTX='${CPPFLAGS}' COPTX='${CFLAGS}' C++OPTX='$
 # Stop failures when 'cc' can't be found
 export ac_cv_prog_CC = "${CC}"
 
-inherit native
+inherit sourceforge-releases native
 
 # Use -std=gnu89 to build with gcc-14 (https://bugs.gentoo.org/903876)
 # this needs to be after native inherit (which sets CFLAGS to BUILD_CFLAGS)
meta/recipes-extended/cwautomacros/cwautomacros_20110201.bb +4 -1
index 0e155b6c951..84687504349 100644
@@ -5,10 +5,13 @@ DESCRIPTION = "A collection of autoconf macros, plus an autogen.sh script that c
 LICENSE = "GPL-2.0-only"
 LIC_FILES_CHKSUM = "file://LICENSE;md5=eb723b61539feef013de476e68b5c50a"
 
-SRC_URI = "${SOURCEFORGE_MIRROR}/${BPN}.berlios/${BP}.tar.bz2"
+SOURCEFORGE_PROJECT = "${BPN}.berlios"
+SRC_URI = "${SOURCEFORGE_MIRROR}/${SOURCEFORGE_PROJECT}/${BP}.tar.bz2"
 SRC_URI[md5sum] = "074afcb50d0a8bff10786a2954b2b02d"
 SRC_URI[sha256sum] = "3115603b891f3a163c0bbb5fea2f3742113a183fa6745ee5e89e5f6d0e9f6121"
 
+inherit sourceforge-releases
+
 do_configure() {
 	:
 }
meta/recipes-extended/hdparm/hdparm_9.65.bb +1 -0
index 1b6af566a29..34d50054ab5 100644
@@ -26,6 +26,7 @@ SRC_URI = "${SOURCEFORGE_MIRROR}/hdparm/${BP}.tar.gz \
            "
 
 SRC_URI[sha256sum] = "d14929f910d060932e717e9382425d47c2e7144235a53713d55a94f7de535a4b"
+UPSTREAM_CHECK_URI = "https://sourceforge.net/projects/hdparm/files/hdparm/"
 
 EXTRA_OEMAKE = 'STRIP="echo" LDFLAGS="${LDFLAGS}"'
 
meta/recipes-extended/mingetty/mingetty_1.08.bb +1 -1
index aa5a989df2c..00d2564257a 100644
@@ -21,7 +21,7 @@ do_install(){
     oe_runmake install DESTDIR=${D}
 }
 
-inherit update-alternatives
+inherit update-alternatives sourceforge-releases
 
 ALTERNATIVE:${PN} = "getty"
 ALTERNATIVE_LINK_NAME[getty] = "${base_sbindir}/getty"
meta/recipes-extended/rpcbind/rpcbind_1.2.7.bb +1 -1
index d308071d293..e186bc35e10 100644
@@ -20,7 +20,7 @@ SRC_URI = "${SOURCEFORGE_MIRROR}/rpcbind/rpcbind-${PV}.tar.bz2 \
           "
 SRC_URI[sha256sum] = "f6edf8cdf562aedd5d53b8bf93962d61623292bfc4d47eedd3f427d84d06f37e"
 
-inherit autotools update-rc.d systemd pkgconfig update-alternatives
+inherit autotools update-rc.d systemd pkgconfig update-alternatives sourceforge-releases
 
 PACKAGECONFIG ??= "tcp-wrappers"
 PACKAGECONFIG[tcp-wrappers] = "--enable-libwrap,--disable-libwrap,tcp-wrappers"
meta/recipes-multimedia/lame/lame_3.100.bb +1 -1
index 5b42795f4e3..5ace46ace0a 100644
@@ -17,7 +17,7 @@ SRC_URI = "${SOURCEFORGE_MIRROR}/lame/lame-${PV}.tar.gz \
 SRC_URI[md5sum] = "83e260acbe4389b54fe08e0bdbf7cddb"
 SRC_URI[sha256sum] = "ddfe36cab873794038ae2c1210557ad34857a4b6bdc515785d1da9e175b1da1e"
 
-inherit autotools pkgconfig
+inherit autotools pkgconfig sourceforge-releases
 
 PACKAGES += "libmp3lame libmp3lame-dev"
 FILES:${PN} = "${bindir}/lame"
meta/recipes-support/libfm/libfm-extra_1.3.2.bb +3 -2
index eff1b54f734..b7a9b25f862 100644
@@ -8,7 +8,8 @@ LIC_FILES_CHKSUM = "file://src/fm-extra.h;beginline=8;endline=21;md5=ef1f84da64b
 SECTION = "x11/libs"
 DEPENDS = "glib-2.0-native glib-2.0 intltool-native"
 
-SRC_URI = "${SOURCEFORGE_MIRROR}/pcmanfm/libfm-${PV}.tar.xz \
+SOURCEFORGE_PROJECT = "pcmanfm"
+SRC_URI = "${SOURCEFORGE_MIRROR}/${SOURCEFORGE_PROJECT}/libfm-${PV}.tar.xz \
           "
 
 SRC_URI[sha256sum] = "a5042630304cf8e5d8cff9d565c6bd546f228b48c960153ed366a34e87cad1e5"
@@ -17,6 +18,6 @@ S = "${WORKDIR}/libfm-${PV}"
 
 EXTRA_OECONF = "--with-extra-only --with-gtk=no"
 
-inherit autotools pkgconfig gtk-doc gettext
+inherit autotools pkgconfig gtk-doc gettext sourceforge-releases
 
 do_configure[dirs] =+ "${S}/m4"
meta/recipes-support/libfm/libfm_1.3.2.bb +3 -2
index 1d7609165b4..e3cf7517164 100644
@@ -13,14 +13,15 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=4641e94ec96f98fabc56ff9cc48be14b \
 SECTION = "x11/libs"
 DEPENDS = "glib-2.0 glib-2.0-native pango gtk+3 menu-cache intltool-native libexif libfm-extra"
 
-SRC_URI = "${SOURCEFORGE_MIRROR}/pcmanfm/libfm-${PV}.tar.xz \
+SOURCEFORGE_PROJECT = "pcmanfm"
+SRC_URI = "${SOURCEFORGE_MIRROR}/${SOURCEFORGE_PROJECT}/libfm-${PV}.tar.xz \
            file://0001-Correctly-check-the-stamp-file-that-indicates-if-we-.patch \
            file://0001-Do-not-add-library-path-to-avoid-host-contamination.patch \
            "
 
 SRC_URI[sha256sum] = "a5042630304cf8e5d8cff9d565c6bd546f228b48c960153ed366a34e87cad1e5"
 
-inherit autotools pkgconfig gtk-doc gettext features_check mime mime-xdg
+inherit autotools pkgconfig gtk-doc gettext features_check mime mime-xdg sourceforge-releases
 ANY_OF_DISTRO_FEATURES = "${GTK3DISTROFEATURES}"
 
 EXTRA_OECONF = "--with-gtk=3"
meta/recipes-support/libpcre/libpcre_8.45.bb +3 -2
index 17fb6846a52..933bb00de65 100644
@@ -7,7 +7,8 @@ HOMEPAGE = "http://www.pcre.org"
 SECTION = "devel"
 LICENSE = "BSD-3-Clause"
 LIC_FILES_CHKSUM = "file://LICENCE;md5=b5d5d1a69a24ea2718263f1ff85a1c58"
-SRC_URI = "${SOURCEFORGE_MIRROR}/pcre/pcre-${PV}.tar.bz2 \
+SOURCEFORGE_PROJECT = "pcre"
+SRC_URI = "${SOURCEFORGE_MIRROR}/${SOURCEFORGE_PROJECT}/pcre-${PV}.tar.bz2 \
            file://run-ptest \
            file://Makefile \
            "
@@ -32,7 +33,7 @@ PACKAGECONFIG[jit] = "--enable-jit=auto,--disable-jit"
 
 BINCONFIG = "${bindir}/pcre-config"
 
-inherit autotools binconfig-disabled ptest
+inherit autotools binconfig-disabled ptest sourceforge-releases
 
 EXTRA_OECONF = "--enable-utf"
 
Signature · 1 line
2.39.5
#2 lore raw

Patchtest results for [PATCH 1/2] fetch2/wget.py: use default wget user-agent in upstream version checks

In reply to Alexander Kanavin

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/1-2-fetch2-wget.py-use-default-wget-user-agent-in-upstream-version-checks.patch

FAIL: test target mailing list: Series sent to the wrong mailing list or some patches from the series correspond to different mailing lists (test_mbox.TestMbox.test_target_mailing_list)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!