Skip to contentSkip to conversation

archiver.bbclass: Improve the indentation of the documentation

PATCH In review Peter Kjellerstedt 3 participants 4 messages Aug 25 – Aug 28

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 archiver.bbclass: Improve the indentation of the documentation

This makes the class documentation's indentation consistent.

---
 meta/classes/archiver.bbclass | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)
meta/classes/archiver.bbclass +13 -13
index 3124b7040d..f38409774d 100644
@@ -19,12 +19,12 @@
 #  - The recipe (.bb and .inc): ARCHIVER_MODE[recipe] = "1"
 #  - Filter the license, the recipe whose license in
 #    COPYLEFT_LICENSE_INCLUDE will be included, and in
-#    COPYLEFT_LICENSE_EXCLUDE will be excluded.
-#    COPYLEFT_LICENSE_INCLUDE = 'GPL* LGPL*'
-#    COPYLEFT_LICENSE_EXCLUDE = 'CLOSED Proprietary'
-# - The recipe type that will be archived:
-#    COPYLEFT_RECIPE_TYPES = 'target'
-# - The source mirror mode:
+#    COPYLEFT_LICENSE_EXCLUDE will be excluded:
+#     COPYLEFT_LICENSE_INCLUDE = 'GPL* LGPL*'
+#     COPYLEFT_LICENSE_EXCLUDE = 'CLOSED Proprietary'
+#  - The recipe type that will be archived:
+#     COPYLEFT_RECIPE_TYPES = 'target'
+#  - The source mirror mode:
 #    ARCHIVER_MODE[mirror] = "split" (default): Sources are split into
 #    per-recipe directories in a similar way to other archiver modes.
 #    Post-processing may be required to produce a single mirror directory.
@@ -33,13 +33,13 @@
 #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
 #    directory suitable for direct use as a mirror. Duplicate sources are
 #    ignored.
-# - Source mirror exclusions:
-#     ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
-#     This may be used for sources which you are already publishing yourself
-#     (e.g. if the URI starts with 'https://mysite.com/' and your mirror is
-#     going to be published to the same site). It may also be used to exclude
-#     local files (with the prefix 'file://') if these will be provided as part
-#     of an archive of the layers themselves.
+#  - Source mirror exclusions:
+#    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
+#    This may be used for sources which you are already publishing yourself
+#    (e.g. if the URI starts with 'https://mysite.com/' and your mirror is
+#    going to be published to the same site). It may also be used to exclude
+#    local files (with the prefix 'file://') if these will be provided as part
+#    of an archive of the layers themselves.
 #
 
 # Create archive for all the recipe types
#1 lore raw

PATCH 2/2 archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

When using the archiver's mirror mode, ARCHIVER_MIRROR_INCLUDE contains a list of URI prefixes to always include in the mirror. This may be useful if recipes that are included in the mirror (due to their licenses) depend on some recipe that otherwise would not be included in the mirror, and that recipe uses files that are normally only available on a private server.

---
 meta/classes/archiver.bbclass            | 35 ++++++++++++++++++++++--
 meta/lib/oeqa/selftest/cases/archiver.py | 24 ++++++++++++++++
 2 files changed, 57 insertions(+), 2 deletions(-)
meta/classes/archiver.bbclass +33 -2
index f38409774d..db8b60916a 100644
@@ -33,6 +33,12 @@
 #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
 #    directory suitable for direct use as a mirror. Duplicate sources are
 #    ignored.
+#  - Source mirror inclusions:
+#    ARCHIVER_MIRROR_INCLUDE is a list of URI prefixes to always include in
+#    the mirror. This may be useful if recipes that are included in the mirror
+#    (due to their licenses) depend on some recipe that otherwise would not be
+#    included in the mirror, and that recipe uses files that are normally only
+#    available on a private server.
 #  - Source mirror exclusions:
 #    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
 #    This may be used for sources which you are already publishing yourself
@@ -72,8 +78,9 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
 # This is a convenience for the shell script to use it
 
 def include_package(d, pn):
-
-    included, reason = copyleft_should_include(d)
+    included, reason = archiver_should_include(d)
+    if not included:
+        included, reason = copyleft_should_include(d)
     if not included:
         bb.debug(1, 'archiver: %s is excluded: %s' % (pn, reason))
         return False
@@ -94,6 +101,29 @@ def include_package(d, pn):
 
     return True
 
+def archiver_should_include(d):
+    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src')
+    if ar_src == "mirror":
+        src_uri = (d.getVar('SRC_URI') or '').split()
+        if len(src_uri) == 0:
+            return False, None
+
+        mirror_inclusions = (d.getVar('ARCHIVER_MIRROR_INCLUDE') or '').split()
+
+        def is_included(url):
+            for prefix in mirror_inclusions:
+                if url.startswith(prefix):
+                    return True
+            return False
+
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+
+        for ud in fetcher.expanded_urldata():
+            if is_included(ud.url):
+                return True, "URL matches ARCHIVER_MIRROR_INCLUDE"
+
+    return False, None
+
 python () {
     pn = d.getVar('PN')
     assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()
@@ -154,6 +184,7 @@ python () {
 do_ar_prepare[vardeps] += " \
     ARCHIVER_MODE \
     ARCHIVER_MIRROR_EXCLUDE \
+    ARCHIVER_MIRROR_INCLUDE \
     COPYLEFT_LICENSE_EXCLUDE \
     COPYLEFT_LICENSE_INCLUDE \
     COPYLEFT_PN_EXCLUDE \
meta/lib/oeqa/selftest/cases/archiver.py +24 -0
index 8c5aa75e9b..78118db448 100644
@@ -249,6 +249,30 @@ class Archiver(OESelftestTestCase):
         archive_path = os.path.join(glob_result[0], target_file_name)
         self.assertFalse(os.path.exists(archive_path), 'Failed to exclude archive file %s' % (target_file_name))
 
+    def test_archiver_mode_mirror_include(self):
+        """
+        Test that `ARCHIVER_MIRROR_INCLUDE` causes a source URL to be included
+        in the mirror even when the recipe would otherwise be excluded by the
+        copyleft license filter.
+        """
+
+        target = 'selftest-ed'
+        target_file_name = 'ed-1.21.1.tar.lz'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "mirror"\n'
+        features += 'ARCHIVER_MODE[mirror] = "combined"\n'
+        features += 'BB_GENERATE_MIRROR_TARBALLS = "1"\n'
+        features += 'COPYLEFT_LICENSE_INCLUDE = "CLOSED"\n'
+        features += 'ARCHIVER_MIRROR_INCLUDE = "${GNU_MIRROR}"\n'
+        self.write_config(features)
+
+        bitbake('-c deploy_archives %s' % (target))
+
+        bb_vars = get_bb_vars(['DEPLOY_DIR_SRC'])
+        target_path = os.path.join(bb_vars['DEPLOY_DIR_SRC'], 'mirror', target_file_name)
+        self.assertTrue(os.path.exists(target_path), 'Missing archive file %s' % (target_file_name))
+
     def test_archiver_mode_mirror_combined(self):
         """
         Test that the archiver works with `ARCHIVER_MODE[src] = "mirror"`
#2 lore raw

Re: [OE-core] [PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

Quoted 18 lines · Peter Kjellerstedt via lists.openembedded.org

When using the archiver's mirror mode, ARCHIVER_MIRROR_INCLUDE contains a list of URI prefixes to always include in the mirror. This may be useful if recipes that are included in the mirror (due to their licenses) depend on some recipe that otherwise would not be included in the mirror, and that recipe uses files that are normally only available on a private server.

---
 meta/classes/archiver.bbclass            | 35 ++++++++++++++++++++++-- meta/lib/oeqa/selftest/cases/archiver.py | 24 ++++++++++++++++

 2 files changed, 57 insertions(+), 2 deletions(-)
meta/classes/archiver.bbclass +0 -0
index f38409774d..db8b60916a 100644

meta/classes/archiver.bbclass line 41

Quoted 9 lines
meta/classes/archiver.bbclass +6 -0
@@ -33,6 +33,12 @@
  #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
  #    directory suitable for direct use as a mirror. Duplicate sources are
  #    ignored.
+#  - Source mirror inclusions:
+#    ARCHIVER_MIRROR_INCLUDE is a list of URI prefixes to always include in
+#    the mirror. This may be useful if recipes that are included in the mirror
+#    (due to their licenses) depend on some recipe that otherwise would not be
+#    included in the mirror, and that recipe uses files that are normally only
+#    available on a private server.

This wording is a rather ambiguous to me :/.

It reads like things would normally be included but this is a list of extra special things to include. That isn't how it works.

The "depend on some recipe" suggests it somehow also follows dependencies into other recipes, which as far as I know, it doesn't. I don't see where a private server fits into things.

Quoted 5 lines
meta/classes/archiver.bbclass +0 -0
  #  - Source mirror exclusions:
  #    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
  #    This may be used for sources which you are already publishing yourself
@@ -72,8 +78,9 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
  # This is a convenience for the shell script to use it
Quoted 7 lines


 def include_package(d, pn):
-
-    included, reason = copyleft_should_include(d) +    included, reason = archiver_should_include(d) +    if not included:
+        included, reason = copyleft_should_include(d)

It overrides copyleft includes? That should probably be documented. It's relationship with exclude is also not documented. Which wins? I suspect exclude does, but should it?

Quoted 34 lines

     if not included:
         bb.debug(1, 'archiver: %s is excluded: %s' % (pn, reason)) return False

meta/classes/archiver.bbclass +0 -0
@@ -94,6 +101,29 @@ def include_package(d, pn):


     return True

meta/classes/archiver.bbclass +23 -0
+def archiver_should_include(d):
+    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src')
+    if ar_src == "mirror":
+        src_uri = (d.getVar('SRC_URI') or '').split()
+        if len(src_uri) == 0:
+            return False, None
+
+        mirror_inclusions = (d.getVar('ARCHIVER_MIRROR_INCLUDE') or '').split()
+
+        def is_included(url):
+            for prefix in mirror_inclusions:
+                if url.startswith(prefix):
+                    return True
+            return False
+
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+
+        for ud in fetcher.expanded_urldata():
+            if is_included(ud.url):
+                return True, "URL matches ARCHIVER_MIRROR_INCLUDE"
+
+    return False, None
+

 python () {
     pn = d.getVar('PN')
     assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()

meta/classes/archiver.bbclass python () { · line 190

Quoted 7 lines
meta/classes/archiver.bbclass +1 -0
@@ -154,6 +184,7 @@ python () {
  do_ar_prepare[vardeps] += " \
      ARCHIVER_MODE \
      ARCHIVER_MIRROR_EXCLUDE \
+    ARCHIVER_MIRROR_INCLUDE \
      COPYLEFT_LICENSE_EXCLUDE \
      COPYLEFT_LICENSE_INCLUDE \
      COPYLEFT_PN_EXCLUDE \

I get the feeling this was discussed before and one of the concerns was that the include and exclude code are in two different places, working in two quite different ways? That isn't going to help maintainability of this code, which already has a poor reputation amongst developers.

Cheers,

Richard

#3 lore raw

RE: [OE-core] [PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

Quoted 38 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 26 augusti 2026 09:13
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH 2/2] archiver.bbclass: Add ARCHIVER_MIRROR_INCLUDE

Quoted 18 lines · Peter Kjellerstedt via lists.openembedded.org

When using the archiver's mirror mode, ARCHIVER_MIRROR_INCLUDE contains a list of URI prefixes to always include in the mirror. This may be useful if recipes that are included in the mirror (due to their licenses) depend on some recipe that otherwise would not be included in the mirror, and that recipe uses files that are normally only available on a private server.

---
 meta/classes/archiver.bbclass            | 35 ++++++++++++++++++++++-- meta/lib/oeqa/selftest/cases/archiver.py | 24 ++++++++++++++++

 2 files changed, 57 insertions(+), 2 deletions(-)
meta/classes/archiver.bbclass +0 -0
index f38409774d..db8b60916a 100644

meta/classes/archiver.bbclass line 41

Quoted 9 lines
meta/classes/archiver.bbclass +6 -0
@@ -33,6 +33,12 @@
  #    ARCHIVER_MODE[mirror] = "combined": All sources are placed into a single
  #    directory suitable for direct use as a mirror. Duplicate sources are
  #    ignored.
+#  - Source mirror inclusions:
+#    ARCHIVER_MIRROR_INCLUDE is a list of URI prefixes to always include in
+#    the mirror. This may be useful if recipes that are included in the mirror
+#    (due to their licenses) depend on some recipe that otherwise would not be
+#    included in the mirror, and that recipe uses files that are normally only
+#    available on a private server.

This wording is a rather ambiguous to me :/.

Quoted 6 lines

It reads like things would normally be included but this is a list of extra special things to include. That isn't how it works.

The "depend on some recipe" suggests it somehow also follows dependencies into other recipes, which as far as I know, it doesn't. I don't see where a private server fits into things.

What I'm trying to convey is the situation where you have a recipe that is copyleft and thus it will be included in the archives that are produced. If that recipe in turn depends on some recipe that does not use a copyleft license, that second recipe will by default not be included in the saved archives. For such recipes that fetch their code from the Internet, this is not a problem as Bitbake will just fetch the sources as normal. However, if the second recipe is a private one and it has its files on an internal server, those files must be included in the produced artifacts or the copyleft recipe will not be able to be built.

Quoted 7 lines
Quoted 7 lines
meta/classes/archiver.bbclass +0 -0
  #  - Source mirror exclusions:
  #    ARCHIVER_MIRROR_EXCLUDE is a list of prefixes to exclude from the mirror.
  #    This may be used for sources which you are already publishing yourself
@@ -72,8 +78,9 @@ do_ar_original[dirs] = "${ARCHIVER_OUTDIR} ${ARCHIVER_WORKDIR}"
  # This is a convenience for the shell script to use it
 
Quoted 8 lines
Quoted 6 lines

 def include_package(d, pn):
-
-    included, reason = copyleft_should_include(d) +    included, reason = archiver_should_include(d) +    if not included:
+        included, reason = copyleft_should_include(d)

It overrides copyleft includes? That should probably be documented.

You are correct, this should be documented better. The assumption from my side is that if you specify ARCHIVER_MIRROR_INCLUDE, then you know what you are doing and the sources that match the prefix(es) should be included regardless of their licenses.

Quoted 2 lines

It's relationship with exclude is also not documented. Which wins? I suspect exclude does, but should it?

Yes, ARCHIVER_MIRROR_EXCLUDE will win over ARCHIVER_MIRROR_INCLUDE. I think it is a bit of a corner case though. It does not make much sense to specify the same prefixes for both. But it should be documented.

Quoted 38 lines
Quoted 38 lines

     if not included:
         bb.debug(1, 'archiver: %s is excluded: %s' % (pn, reason)) return False

meta/classes/archiver.bbclass +0 -0
@@ -94,6 +101,29 @@ def include_package(d, pn):
 

     return True

meta/classes/archiver.bbclass +23 -0
+def archiver_should_include(d):
+    ar_src = d.getVarFlag('ARCHIVER_MODE', 'src')
+    if ar_src == "mirror":
+        src_uri = (d.getVar('SRC_URI') or '').split()
+        if len(src_uri) == 0:
+            return False, None
+
+        mirror_inclusions = (d.getVar('ARCHIVER_MIRROR_INCLUDE') or '').split()
+
+        def is_included(url):
+            for prefix in mirror_inclusions:
+                if url.startswith(prefix):
+                    return True
+            return False
+
+        fetcher = bb.fetch2.Fetch(src_uri, d)
+
+        for ud in fetcher.expanded_urldata():
+            if is_included(ud.url):
+                return True, "URL matches ARCHIVER_MIRROR_INCLUDE"
+
+    return False, None
+

 python () {
     pn = d.getVar('PN')
     assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()

meta/classes/archiver.bbclass +1 -0
@@ -154,6 +184,7 @@ python () {
  do_ar_prepare[vardeps] += " \
      ARCHIVER_MODE \
      ARCHIVER_MIRROR_EXCLUDE \
+    ARCHIVER_MIRROR_INCLUDE \
Quoted 8 lines

meta/classes/archiver.bbclass python () { · line 190

Quoted 3 lines
meta/classes/archiver.bbclass +0 -0
      COPYLEFT_LICENSE_EXCLUDE \
      COPYLEFT_LICENSE_INCLUDE \
      COPYLEFT_PN_EXCLUDE \

I get the feeling this was discussed before and one of the concerns was that the include and exclude code are in two different places, working in two quite different ways? That isn't going to help maintainability of this code, which already has a poor reputation amongst developers.

If it was discussed before, then that is news to me.

I mimicked ARCHIVER_MIRROR_EXCLUDE when I implemented ARCHIVER_MIRROR_INCLUDE (since we use mirror mode this was enough). However, ARCHIVER_MIRROR_INCLUDE could probably be useful regardless of which mode the archiver is in. So maybe it should be ARCHIVER_INCLUDE instead? And then the fact that the implementations are in totally different parts of the code aren't that much of a concern anymore?

Quoted 4 lines

Cheers,

Richard

//Peter