Skip to contentSkip to conversation

gitsm fetcher fails when `PREMIRRORS_ONLY` is set

DISCUSSION [email protected] 2 participants 2 messages Nov 19 – Nov 20

lore
Share & tools
lore
#0 lore raw

Hi all,

I have encountered a gitsm submodule fetch failure that appears to be related to the following change:

https://lists.openembedded.org/g/bitbake-devel/message/17506

This change prevents Git fetcher to silently contact upstream despite BB_FETCH_PREMIRRORONLY being enabled.

However, for gitsm, this datastore is also passed to the nested submodule Fetch. The submodule therefore inherits BB_NO_NETWORK = "1", blocking its HTTP PREMIRROR download, even though the parent repository was downloaded from the same PREMIRROR:

The resulting error is:

gitsm: submodule download failed: NetworkAccess Network access disabled through BB_NO_NETWORK (or set indirectly due to use of BB_FETCH_PREMIRRORONLY) but access requested with command wget ...

I have considered the following approaches. The snippets below are illustrative examples, not complete patches.

*

Reset BB_NO_NETWORK in gitsm's download_submodule() at:
https://git.openembedded.org/bitbake/tree/lib/bb/fetch/gitsm.py#n212
Before creating the nested Fetch instance:

if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
d.setVar("BB_NO_NETWORK", "0")

try:
newfetch = Fetch([url], d, cache=False)
newfetch.download()
except Exception as e:
logger.error(
'gitsm: submodule download failed: %s %s'
% (type(e).__name__, str(e))
)
raise

This allows the submodule to access the HTTP PREMIRROR.

*

Scope the restriction more narrowly:

https://git.openembedded.org/bitbake/tree/lib/bb/fetch/__init__.py#n1160

Instead of passing the modified BB_NO_NETWORK = "1" value to nested submodule fetches, apply the restriction specifically to the Git operations.

*

Preserve and restore the original BB_NO_NETWORK value at:
https://git.openembedded.org/bitbake/tree/lib/bb/fetch/__init__.py#n1157

In try_mirror_url(), preserve the original value before applying the temporary restriction:

network = ld.getVarFlag("BB_NO_NETWORK", "original_bb_no_network") if network is None:
network = ld.getVar("BB_NO_NETWORK") or "0" ld.setVarFlag(
"BB_NO_NETWORK", "original_bb_no_network", network
)

Restore that value before downloading from the mirror:

if not verify_donestamp(ud, ld, origud) or ud.method.need_update(ud, ld):
if bb.utils.to_boolean(ld.getVar("BB_FETCH_PREMIRRORONLY")):
ld.setVar("BB_NO_NETWORK", network)

ud.method.download(ud, ld)
if hasattr(ud.method, "build_mirror_data"):
ud.method.build_mirror_data(ud, ld)

The intention is to distinguish the user's original network policy from the temporary restriction applied during mirror processing.

The intended behaviour is to allow both the parent repository and its submodules to be fetched from configured PREMIRRORS, without permitting upstream access.

I would appreciate your thoughts on these approaches.

Best regards,
Tiago Vasconcelos

#1 lore raw
Quoted 4 lines · tiago.mlb.vasconcelos via bitbake-devel

I have encountered a gitsm submodule fetch failure that appears to be related to the following change:
https://lists.openembedded.org/g/bitbake-devel/message/17506
This change prevents Git fetcher to silently contact upstream

Quoted 8 lines

despite BB_FETCH_PREMIRRORONLY being enabled.
However, for gitsm, this datastore is also passed to the nested submodule
Fetch. The submodule therefore inherits BB_NO_NETWORK = "1", blocking its
HTTP PREMIRROR download, even though the parent repository was downloaded from
the same PREMIRROR:

The best advice I can give is to write a new test case for bitbake- selftest (in lib/bb/tests/fetch.py) which shows the issue you're seeing.

Once we have that, we can likely more easily understand it and try and come up with a solution.

Cheers,

Richard