bitbake: fetch2/repo: add support for projects to be fetched simultaneously
PATCH 1/3 bitbake: fetch2/repo: add support for projects to be fetched simultaneously
repo tool can fetch projects simultaneously.
As per repo sync documentation:
Usage: repo sync [<project>...]
Options:
-j JOBS, --jobs=JOBS projects to fetch simultaneously (default 1)
By supplying jobs=<some_number> recipe can choose to use the simultaneous fetch feature of repo tool.
Example:
SRC_URI = "repo://REPOROOT;protocol=git;branch=main;manifest=default.xml;jobs=8"
--- bitbake/lib/bb/fetch2/repo.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/repo.py +11 -1
@@ -62,11 +62,21 @@ class Repo(FetchMethod): repodir = os.path.join(codir, "repo") bb.utils.mkdirhier(repodir) if not os.path.exists(os.path.join(repodir, ".repo")): + bb.fetch2.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url) runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d, workdir=repodir) + # Projects to fetch simultaneously at time of 'repo sync'. + # If SRC_URI provides parallel jobs parameter as follows: + # SRC_URI = "repo://REPOROOT;branch=some_branch;jobs=8" + # Sync is run as 'repo sync -j8' + # When jobs parameter is not passed with SRC_URI then + # repo tool's default behavior is to fetch 1 project at a time. + jobs = ud.parm.get('jobs', '') + if jobs: + jobs = '-j' + jobs bb.fetch2.check_network_access(d, "%s sync %s" % (ud.basecmd, ud.url), ud.url) - runfetchcmd("%s sync" % ud.basecmd, d, workdir=repodir) + runfetchcmd("%s sync %s" % (ud.basecmd, jobs), d, workdir=repodir) scmdata = ud.parm.get("scmdata", "") if scmdata == "keep":
Signature · 7 lines
2.7.4 -- Regards, Sourabh
PATCH 2/3 bitbake: fetch2/repo: add support for sync specific projects
repo tool can fetch projects by name. I.e., if there are multiple projects listed in the manifest file.
<manifest>
<project name="project1" path="sync path1" revision="sha256" /> <project name="project2" path="sync path2" revision="sha256" />
</manifest>
Documented Usage: repo sync [<project>...]
By supplying project=<project_name> recipe can choose to sync a specific project.
Example:
SRC_URI = "repo://REPOROOT;protocol=git;branch=main;manifest=default.xml;project=project1"
This is helpful when recipe needs to sync only a specific project from a manifest that lists multiple projects.
--- bitbake/lib/bb/fetch2/repo.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
bitbake/lib/bb/fetch2/repo.py +7 -3
@@ -41,7 +41,10 @@ class Repo(FetchMethod): if not ud.manifest.endswith('.xml'): ud.manifest += '.xml' - ud.localfile = d.expand("repo_%s%s_%s_%s.tar.gz" % (ud.host, ud.path.replace("/", "."), ud.manifest, ud.branch)) + project = ud.parm.get('project', '') + if len(project): + project = '_' + project + ud.localfile = d.expand("repo_%s%s_%s_%s%s.tar.gz" % (ud.host, ud.path.replace("/", "."), ud.manifest, ud.branch, project.replace("/", "."))) def download(self, ud, d): """Fetch url"""
@@ -52,7 +55,8 @@ class Repo(FetchMethod): repodir = d.getVar("REPODIR") or (d.getVar("DL_DIR") + "/repo") gitsrcname = "%s%s" % (ud.host, ud.path.replace("/", ".")) - codir = os.path.join(repodir, gitsrcname, ud.manifest) + project = ud.parm.get('project', '') + codir = os.path.join(repodir, gitsrcname, ud.manifest, project.replace("/", ".")) if ud.user: username = ud.user + "@"
@@ -76,7 +80,7 @@ class Repo(FetchMethod): if jobs: jobs = '-j' + jobs bb.fetch2.check_network_access(d, "%s sync %s" % (ud.basecmd, ud.url), ud.url) - runfetchcmd("%s sync %s" % (ud.basecmd, jobs), d, workdir=repodir) + runfetchcmd("%s sync %s %s" % (ud.basecmd, jobs, project), d, workdir=repodir) scmdata = ud.parm.get("scmdata", "") if scmdata == "keep":
Signature · 7 lines
2.7.4 -- Regards, Sourabh
PATCH 3/3 bitbake: fetch2/repo: add support for --repo-url and --repo-branch
In reply to Sourabh Banerjeerepo tool provides following options to use a stable or preferred version of the tool
repo Version options:
--repo-url=URL repo repository location
--repo-branch=REVISION
repo branch or revision
By supplying repo_url=<uri> and repo_branch=<branch> recipe may choose to sync a specific version of repo tool.
--- bitbake/lib/bb/fetch2/repo.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/repo.py +7 -1
@@ -66,9 +66,15 @@ class Repo(FetchMethod): repodir = os.path.join(codir, "repo") bb.utils.mkdirhier(repodir) if not os.path.exists(os.path.join(repodir, ".repo")): + repo_url = ud.parm.get('repo_url', '') + if repo_url: + repo_url = '--repo-url=' + repo_url + repo_branch = ud.parm.get('repo_branch', '') + if repo_branch: + repo_branch = '--repo-branch=' + repo_branch bb.fetch2.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url) - runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d, workdir=repodir) + runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s %s %s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path, repo_url, repo_branch), d, workdir=repodir) # Projects to fetch simultaneously at time of 'repo sync'. # If SRC_URI provides parallel jobs parameter as follows:
Signature · 7 lines
2.7.4 -- Regards, Sourabh
Re: [OE-core] [bitbake][PATCH 3/3] bitbake: fetch2/repo: add support for --repo-url and --repo-branch
Quoted 36 lines · Sourabh Banerjee
repo tool provides following options to use a stable or preferred version of the tool
repo Version options:
--repo-url=URL repo repository location
--repo-branch=REVISION
repo branch or revisionBy supplying repo_url=<uri> and repo_branch=<branch> recipe may choose to sync a specific version of repo tool.
Signed-off-by Sourabh Banerjee <[email protected]>---
bitbake/lib/bb/fetch2/repo.py | 8 +++++++-1 file changed, 7 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/repo.py +0
-0@@ -66,9 +66,15 @@ class Repo(FetchMethod): repodir = os.path.join(codir, "repo") bb.utils.mkdirhier(repodir)if not
os.path.exists(os.path.join(repodir, ".repo")):bitbake/lib/bb/fetch2/repo.py +6
-0+ repo_url = ud.parm.get('repo_url', '') + if repo_url: + repo_url = '--repo-url=' + repo_url + repo_branch = ud.parm.get('repo_branch', '') + if repo_branch: + repo_branch = '--repo-branch=' + repo_branchbb.fetch2.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
bitbake/lib/bb/fetch2/repo.py +1
-1- runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d, workdir=repodir) + runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s %s %s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path, repo_url, repo_branch), d, workdir=repodir)
Quoted 2 lines
In your other patch you make the new parameters update the mirror tarball name. I'm wondering if we also need to do something like that here as presumably, the fetched output might vary depending on which version of repo was used?
Bitbake patches should go to the bitbake-devel list. I'm also wondering if there should be new fetcher tests these changes should be adding to lib/bb/test/fetch.py, run using bitbake-selftest?
Cheers,
Richard
Re: [OE-core] [bitbake][PATCH 3/3] bitbake: fetch2/repo: add support for --repo-url and --repo-branch
Quoted 34 lines · Richard Purdie
Quoted 33 lines · Sourabh Banerjee
repo tool provides following options to use a stable or preferred version of the tool
repo Version options:
--repo-url=URL repo repository location
--repo-branch=REVISION
repo branch or revisionBy supplying repo_url=<uri> and repo_branch=<branch> recipe may choose to sync a specific version of repo tool.
Signed-off-by Sourabh Banerjee <[email protected]>---
bitbake/lib/bb/fetch2/repo.py | 8 +++++++-1 file changed, 7 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/repo.py +0
-0@@ -66,9 +66,15 @@ class Repo(FetchMethod): repodir = os.path.join(codir, "repo") bb.utils.mkdirhier(repodir)if not
os.path.exists(os.path.join(repodir, ".repo")):bitbake/lib/bb/fetch2/repo.py +6
-0+ repo_url = ud.parm.get('repo_url', '') + if repo_url: + repo_url = '--repo-url=' + repo_url + repo_branch = ud.parm.get('repo_branch', '') + if repo_branch: + repo_branch = '--repo-branch=' + repo_branch
Quoted 12 lines
Quoted 7 lines
bb.fetch2.check_network_access(d, "%s init -m %s -b %s -u %s://%s%s%s" % (ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
- runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s" %
(ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d, workdir=repodir)
+ runfetchcmd("%s init -m %s -b %s -u %s://%s%s%s %s %s" %
(ud.basecmd, ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path,repo_url,repo_branch), d, workdir=repodir)In your other patch you make the new parameters update the mirror tarball name. I'm wondering if we also need to do something like that here as presumably, the fetched output might vary depending on which version of repo was used?
Hi Richard,
Here is what I understand. The 'repo' tool installed on host, let's say at /usr/bin/repo handles the options --repo-url and --repo-branch in 'init' sub-command and downloads the 'repo' tool copy from uri-branch under <working-dir>/.repo/repo/repo. All subsequent sub-commands such as 'sync' will work using the tool copy under <working-dir>/.repo/repo/repo. As a recipe developer/maintainer I find --repo-url and --repo-branch useful when I know my manifest has a XML key-value pair that is only supported certain version (& up) of 'repo' tool. At such a point I will code my SRC_URI asSRC_URI = "\
repo://URI/my-manifest-project.git;\
protocol=git;branch=main;project=platform/my-project;\
repo_url=git://gerrit.googlesource.com/git-repo;repo_branch=stable\
"
Here I thought as my recipe file carries the URI-BRANCH combo to repo tool it needs, shall be good? With this thought I did not code it in mirror tarball name.
Quoted 2 lines
Bitbake patches should go to the bitbake-devel list. I'm also
I have never submitted to bitbake-devel, but I think I will take these there then?
Quoted 2 lines
wondering if there should be new fetcher tests these changes should be adding to
lib/bb/test/fetch.py, run using bitbake-selftest?
Thanks for valuable suggestion, I will look into how tests are developed for fetcher. If possible, please point me to fetcher test, that you consider to be a well written one. I will learn from it.
Quoted 4 lines
Cheers,
Richard
Signature · 2 lines
Regards, Sourabh
Quoted 11 lines · Sourabh Banerjee
repo tool can fetch projects simultaneously.
As per repo sync documentation:
Usage: repo sync [<project>...]
Options:
-j JOBS, --jobs=JOBS projects to fetch simultaneously (default 1)By supplying jobs=<some_number> recipe can choose to use the simultaneous fetch feature of repo tool.
Example:
SRC_URI = "repo://REPOROOT;protocol=git;branch=main;manifest=default.xml;jobs=8"
Quoted 4 lines
Signed-off-by Sourabh Banerjee <[email protected]>---
bitbake/lib/bb/fetch2/repo.py | 12 +++++++++++-1 file changed, 11 insertions(+), 1 deletion(-)
Does this value make sense as a per repository setting or an overall build setting?
Cheers,
Richard
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Sourabh Banerjee
repo tool can fetch projects simultaneously.
As per repo sync documentation:
Usage: repo sync [<project>...]
Options:
-j JOBS, --jobs=JOBS projects to fetch simultaneously (default 1)By supplying jobs=<some_number> recipe can choose to use the simultaneous fetch feature of repo tool.
Example:
SRC_URI = "repo://REPOROOT;protocol=git;branch=main;manifest=default.xml;jobs=8"
Quoted 7 lines
Quoted 4 lines
Signed-off-by Sourabh Banerjee <[email protected]>---
bitbake/lib/bb/fetch2/repo.py | 12 +++++++++++-1 file changed, 11 insertions(+), 1 deletion(-)Does this value make sense as a per repository setting or an overall build setting?
Hi Richard,
The '--jobs' option for the repo tool parallelizes fetch w.r.t manifest.xml. For example, if there are 8 projects listed in a manifest.xml then passing-j4 will run 4 parallel jobs fetching 4 projects simultaneously at a time. So it is related to whole manifest being operated upon by the repo tool.
Quoted 4 lines
Cheers,
Richard
Signature · 2 lines
Regards, Sourabh