externalsrc.bbclass: fix git repo detection
PATCH 1/2 externalsrc.bbclass: fix git repo detection
From: Martin Jansa <[email protected]>
* fix issue introduced in:
https://git.openembedded.org/openembedded-core/commit/?id=95fbac8dcad6c93f4c9737e9fe13e92ab6befa09
* it added check for s_dir + git-dir (typically '.git') isn't the same as ${TOPDIR} + git-dir, but due to copy-paste issue it was just comparing it with s_dir + git-dir again, resulting in most external repos (where git-dir is '.git') to be processed as regular directory (not taking advantage of git write-tree).
* normally this wouldn't be an issue, but for big repo with a lot of files this added a lot of checksums in:
d.setVarFlag('do_compile', 'file-checksums', '${@srctree_hash_files(d)}')
and I mean *a lot, e.g. in chromium build it was 380227 paths which still wouldn't that bad, but the checksum processing in siggen.py isn't trivial and just looping through all these checksums takes very long time (over 1000sec on fast NVME drive with warm cache) and then
https://git.openembedded.org/bitbake/commit/?id=b4975d2ecf615ac4c240808fbc5a3f879a93846b
made the processing a bit more complicated and the loop in get_taskhash() function took 6448sec and to make things worse there was no output from bitbake during that time, so even with -DDD it looks like this:
DEBUG: virtual/libgles2 resolved to: mesa (langdale/oe-core/meta/recipes-graphics/mesa/mesa_22.2.0.bb) Bitbake still alive (no events for 600s). Active tasks:
Bitbake still alive (no events for 1200s). Active tasks:
Bitbake still alive (no events for 1800s). Active tasks:
Bitbake still alive (no events for 2400s). Active tasks:
Bitbake still alive (no events for 3000s). Active tasks:
Bitbake still alive (no events for 3600s). Active tasks:
Bitbake still alive (no events for 4200s). Active tasks:
Bitbake still alive (no events for 4800s). Active tasks:
Bitbake still alive (no events for 5400s). Active tasks:
Bitbake still alive (no events for 6000s). Active tasks:
DEBUG: Starting bitbake-worker
without -DDD it will get stuck for almost 2 hours in:
"Initialising tasks..."
before it finally writes sstate summary like:
"Sstate summary: Wanted 3102 Local 0 Mirrors 0 Missed 3102 Current 1483 (0% match, 32% complete)"
* fix the copy&paste typo to use git work-tree in most cases, but be aware that this issue still exists for huge local source trees not in git
[YOCTO #14942]
--- meta/classes/externalsrc.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/externalsrc.bbclass +2 -2
@@ -212,8 +212,8 @@ def srctree_hash_files(d, srcdir=None): try: git_dir = os.path.join(s_dir, subprocess.check_output(['git', '-C', s_dir, 'rev-parse', '--git-dir'], stderr=subprocess.DEVNULL).decode("utf-8").rstrip()) - top_git_dir = os.path.join(s_dir, subprocess.check_output(['git', '-C', d.getVar("TOPDIR"), 'rev-parse', '--git-dir'], - stderr=subprocess.DEVNULL).decode("utf-8").rstrip()) + top_git_dir = os.path.join(d.getVar("TOPDIR"), + subprocess.check_output(['git', '-C', d.getVar("TOPDIR"), 'rev-parse', '--git-dir'], stderr=subprocess.DEVNULL).decode("utf-8").rstrip()) if git_dir == top_git_dir: git_dir = None except subprocess.CalledProcessError:
PATCH 2/2 externalsrc.bbclass: Remove a trailing slash from ${B}
From: Peter Kjellerstedt <[email protected]>
The trailing slash in ${B} caused -fdebug-prefix-map=${B}=... to not match as intended, resulting in ${TMPDIR} ending up in files in ${PN}-dbg when externalsrc was in use, which in turn triggered buildpath QA warnings.
--- meta/classes/externalsrc.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/externalsrc.bbclass +1 -1
@@ -61,7 +61,7 @@ python () { if externalsrcbuild: d.setVar('B', externalsrcbuild) else: - d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + d.setVar('B', '${WORKDIR}/${BPN}-${PV}') local_srcuri = [] fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d)
Re: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}
I've already got these first two patches in stable/langdale-nut:
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/langdale-nut
And your 3/3 seems to be identical to 2/3. Am I missing something?
Steve
Quoted 37 lines · Peter Kjellerstedt
From: Peter Kjellerstedt <[email protected]>
The trailing slash in ${B} caused -fdebug-prefix-map=${B}=... to not match as intended, resulting in ${TMPDIR} ending up in files in ${PN}-dbg when externalsrc was in use, which in turn triggered buildpath QA warnings.
Signed-off-by Alexandre Belloni <[email protected]>Signed-off-by Richard Purdie <[email protected]>Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/classes/externalsrc.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes/externalsrc.bbclass +1
-2@@ -61,7 +61,7 @@ python () { if externalsrcbuild: d.setVar('B', externalsrcbuild) else: - d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + d.setVar('B', '${WORKDIR}/${BPN}-${PV}') local_srcuri = [] fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d) -=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.
View/Reply Online (#172748): https://lists.openembedded.org/g/openembedded-core/message/172748
Mute This Topic: https://lists.openembedded.org/mt/94828874/3620601
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}
In my inbox it looks differently, 2/3 and 3/3 aren't identical and both are indeed in langdale-nut.
Then there is 1/2 identical to 2/3 (probably from sending last 3 externalsrc.bbclass changes and then re-sending after realizing that only last 2 are needed).
1/3 is already in langdale:
https://git.openembedded.org/openembedded-core/commit/?h=langdale&id=6d9364e5f3535954f65cbbc694ee7933ac1d664f
I think all is fine in langdale-nut.
Cheers,
Quoted 55 lines · Steve Sakoman
I've already got these first two patches in stable/langdale-nut:
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/langdale-nut
And your 3/3 seems to be identical to 2/3. Am I missing something?
Steve
Quoted 8 lines · Peter Kjellerstedt
From: Peter Kjellerstedt <[email protected]>
The trailing slash in ${B} caused -fdebug-prefix-map=${B}=... to not match as intended, resulting in ${TMPDIR} ending up in files in ${PN}-dbg when externalsrc was in use, which in turn triggered buildpath QA warnings.
Review
Quoted 8 lines
Signed-off-by Alexandre Belloni <[email protected]>Signed-off-by Richard Purdie <[email protected]>Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/classes/externalsrc.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)(quoted diff) +0
-0b/meta/classes/externalsrc.bbclass
Quoted 7 lines
index 5905342589..75fb91bcb0 100644
meta/classes/externalsrc.bbclass python () { · line 67
Quoted 8 lines
meta/classes/externalsrc.bbclass +1
-1@@ -61,7 +61,7 @@ python () { if externalsrcbuild: d.setVar('B', externalsrcbuild) else: - d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + d.setVar('B', '${WORKDIR}/${BPN}-${PV}') local_srcuri = [] fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d)-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#172754):
https://lists.openembedded.org/g/openembedded-core/message/172754
Mute This Topic: https://lists.openembedded.org/mt/94828874/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
RE: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}
Argh, the second series (with three patches) is intended for Kirkstone, not Langdale.
//Peter
From: Martin Jansa <[email protected]>
Sent: den 5 november 2022 19:28
To: Steve Sakoman <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}
In my inbox it looks differently, 2/3 and 3/3 aren't identical and both are indeed in langdale-nut.
Then there is 1/2 identical to 2/3 (probably from sending last 3 externalsrc.bbclass changes and then re-sending after realizing that only last 2 are needed).
1/3 is already in langdale:
https://git.openembedded.org/openembedded-core/commit/?h=langdale&id=6d9364e5f3535954f65cbbc694ee7933ac1d664f
I think all is fine in langdale-nut.
Cheers,
On Sat, Nov 5, 2022 at 6:59 PM Steve Sakoman <[email protected]<mailto:[email protected]>> wrote:
I've already got these first two patches in stable/langdale-nut:
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/langdale-nut
And your 3/3 seems to be identical to 2/3. Am I missing something?
Steve
Quoted 23 lines · On Sat, Nov 5, 2022 at 5:42 AM Peter Kjellerstedt…
From: Peter Kjellerstedt <[email protected]<mailto:[email protected]>>
The trailing slash in ${B} caused -fdebug-prefix-map=${B}=... to not match as intended, resulting in ${TMPDIR} ending up in files in ${PN}-dbg when externalsrc was in use, which in turn triggered buildpath QA warnings.
--- meta/classes/externalsrc.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes/externalsrc.bbclass +0
-0
meta/classes/externalsrc.bbclass python () { · line 67
Quoted 8 lines
meta/classes/externalsrc.bbclass +1
-1@@ -61,7 +61,7 @@ python () { if externalsrcbuild: d.setVar('B', externalsrcbuild) else: - d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + d.setVar('B', '${WORKDIR}/${BPN}-${PV}') local_srcuri = [] fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d)
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172754): https://lists.openembedded.org/g/openembedded-core/message/172754
Mute This Topic: https://lists.openembedded.org/mt/94828874/3617156
Group Owner: [email protected]<mailto:openembedded-core%[email protected]>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]<mailto:[email protected]>]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}
Quoted 2 lines · Peter Kjellerstedt
Argh, the second series (with three patches) is intended for Kirkstone, not Langdale.
No worries! I think we are good in kirkstone too:
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/kirkstone-nut
Patch series for both will go out for review in the next day or so once autobuilder testing is done.
Steve
Quoted 81 lines
From: Martin Jansa <[email protected]>
Sent: den 5 november 2022 19:28
To: Steve Sakoman <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}In my inbox it looks differently, 2/3 and 3/3 aren't identical and both are indeed in langdale-nut.
Then there is 1/2 identical to 2/3 (probably from sending last 3 externalsrc.bbclass changes and then re-sending after realizing that only last 2 are needed).
1/3 is already in langdale:
I think all is fine in langdale-nut.
Cheers,
On Sat, Nov 5, 2022 at 6:59 PM Steve Sakoman <[email protected]> wrote:
I've already got these first two patches in stable/langdale-nut:
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/langdale-nut
And your 3/3 seems to be identical to 2/3. Am I missing something?
Steve
Quoted 23 lines · Peter Kjellerstedt
From: Peter Kjellerstedt <[email protected]>
The trailing slash in ${B} caused -fdebug-prefix-map=${B}=... to not match as intended, resulting in ${TMPDIR} ending up in files in ${PN}-dbg when externalsrc was in use, which in turn triggered buildpath QA warnings.
Signed-off-by Alexandre Belloni <[email protected]>Signed-off-by Richard Purdie <[email protected]>Signed-off-by Peter Kjellerstedt <[email protected]>--- meta/classes/externalsrc.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/classes/externalsrc.bbclass +0
-0meta/classes/externalsrc.bbclass python () { · line 67
Quoted 8 lines
meta/classes/externalsrc.bbclass +1
-1@@ -61,7 +61,7 @@ python () { if externalsrcbuild: d.setVar('B', externalsrcbuild) else: - d.setVar('B', '${WORKDIR}/${BPN}-${PV}/') + d.setVar('B', '${WORKDIR}/${BPN}-${PV}') local_srcuri = [] fetch = bb.fetch2.Fetch((d.getVar('SRC_URI') or '').split(), d)-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#172754): https://lists.openembedded.org/g/openembedded-core/message/172754
Mute This Topic: https://lists.openembedded.org/mt/94828874/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
RE: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}
Quoted 12 lines
-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: den 5 november 2022 20:41
To: Peter Kjellerstedt <[email protected]>
Cc: Martin Jansa <[email protected]>; [email protected]
Subject: Re: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}Quoted 3 lines · Peter Kjellerstedt
Argh, the second series (with three patches) is intended for Kirkstone, not Langdale.
Quoted 8 lines
No worries! I think we are good in kirkstone too:
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/kirkstone-nut
Patch series for both will go out for review in the next day or so once autobuilder testing is done.
Steve
Ok, good. Since I typically work with the poky repository, I do not normally see the *-nut branches, and tend to forget they exist in another repository.
//Peter
Re: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}
Quoted 17 lines · Peter Kjellerstedt
Quoted 17 lines
-----Original Message-----
From: Steve Sakoman <[email protected]>
Sent: den 5 november 2022 20:41
To: Peter Kjellerstedt <[email protected]>
Cc: Martin Jansa <[email protected]>; [email protected]
Subject: Re: [OE-core] [langdale][PATCH 2/2] externalsrc.bbclass: Remove a trailing slash from ${B}Quoted 3 lines · Peter Kjellerstedt
Argh, the second series (with three patches) is intended for Kirkstone, not Langdale.
No worries! I think we are good in kirkstone too:
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/kirkstone-nut
Quoted 8 lines
Quoted 4 lines
Patch series for both will go out for review in the next day or so once autobuilder testing is done.
Steve
Ok, good. Since I typically work with the poky repository, I do not normally see the *-nut branches, and tend to forget they exist in another repository.
I've got -nut branches in poky-contrib, so you could look there too :-)
Steve