Skip to contentSkip to conversation

populate_sdk_ext: record METADATA_REVISION

PATCH In review Chen Qi 2 participants 7 messages Mar 5, 2021 – Mar 10, 2021 3 messages not in the archive

lore
Share & tools
lore

Chen Qi

author
#0 lore raw

PATCH 1/1 populate_sdk_ext: record METADATA_REVISION

As we delete the .git/ directory, it's impossible to get METADATA_REVISION inside eSDK. Because of this, we meet the following warning when installing eSDK.

WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64

So we record METADATA_REVISION in eSDK generation time to fix this problem.

---
 meta/classes/populate_sdk_ext.bbclass | 3 +++
 1 file changed, 3 insertions(+)
meta/classes/populate_sdk_ext.bbclass +3 -0
index e6bf27cf38..9112ab6c5e 100644
@@ -364,6 +364,9 @@ python copy_buildsystem () {
             # Hide the config information from bitbake output (since it's fixed within the SDK)
             f.write('BUILDCFG_HEADER = ""\n\n')
 
+            # Write METADATA_REVISION
+            f.write('METADATA_REVISION = "%s"\n\n' % d.getVar('METADATA_REVISION'))
+
             f.write('# Provide a flag to indicate we are in the EXT_SDK Context\n')
             f.write('WITHIN_EXT_SDK = "1"\n\n')
 
Signature · 1 line
2.17.1
#1 lore raw
Quoted 6 lines · Chen Qi

As we delete the .git/ directory, it's impossible to get METADATA_REVISION inside eSDK. Because of this, we meet the following warning when installing eSDK.

  WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
  but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64

Quoted 6 lines

So we record METADATA_REVISION in eSDK generation time to fix this problem.

---
 meta/classes/populate_sdk_ext.bbclass | 3 +++

 1 file changed, 3 insertions(+)

I have to wonder why base-files is depending on the metadata-revision? That implies that base-files and hence anything that depends upon it would rebuild each time you update to a new revision. Is that really what you want?

Cheers,

Richard

author
#2 lore raw
Quoted 13 lines · Richard Purdie
Quoted 6 lines · Chen Qi

As we delete the .git/ directory, it's impossible to get METADATA_REVISION inside eSDK. Because of this, we meet the following warning when installing eSDK.

WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64

Quoted 6 lines

So we record METADATA_REVISION in eSDK generation time to fix this problem.

---
  meta/classes/populate_sdk_ext.bbclass | 3 +++
  1 file changed, 3 insertions(+)
Quoted 11 lines

I have to wonder why base-files is depending on the metadata-revision? That implies that base-files and hence anything that depends upon it would rebuild each time you update to a new revision. Is that really what you want?

Cheers,

Richard

Hi Richard,

I tested it on pure yocto project. No extra layer or configuration.
I'm also wondering why base-files do_install depends on METADATA_REVISION. The strange thing is that things do not rebuild at eSDK installation time. There's only a warning about base-files do_install signature.

In fact, this is a problem I found when I was dealing with another problem with eSDK.
*Yocto's current eSDK cannot be successfully installed on a different distro.* For example, you build esdk on ubunt18 and install the esdk on ubuntu20, then you'll get an error of pseudo do_fetch task re-run unexpectedly. I think it might be related to NATIVELSBSTRING, but I haven't found out why.
Manually setting NATIVELSBSTRING to some fixed string such as 'Linux' will solve the above problem.

I'm still checking the codes. If you have some idea/suggestion, please let me know.

Best Regards,
Chen Qi

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#4 lore raw
Quoted 25 lines · Chen Qi
Quoted 13 lines · Richard Purdie
Quoted 6 lines · Chen Qi

As we delete the .git/ directory, it's impossible to get METADATA_REVISION inside eSDK. Because of this, we meet the following warning when installing eSDK.

WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64

Quoted 6 lines

So we record METADATA_REVISION in eSDK generation time to fix this problem.

---
  meta/classes/populate_sdk_ext.bbclass | 3 +++
  1 file changed, 3 insertions(+)
Quoted 11 lines

I have to wonder why base-files is depending on the metadata-revision? That implies that base-files and hence anything that depends upon it would rebuild each time you update to a new revision. Is that really what you want?

Cheers,

Richard

Quoted 5 lines

Hi Richard,

I tested it on pure yocto project. No extra layer or configuration.
I'm also wondering why base-files do_install depends on METADATA_REVISION. The strange thing is that things do not rebuild at eSDK installation time. There's only a warning about base-files do_install signature.

Just found answer for the above problem.
A previous commit changes poky.conf to make DISTRO_VERSION contain METADATA_REVISION. """

poky.conf: do not write current date into distro version, use git hash instead

"""

I also confirmed that merely updating the git revision will result in a rebuild of base-files.
Rebuild of base-files does not result in rebuild of other recipes because we have 'base-files' in SIGGEN_EXCLUDERECIPES_ABISAFE.

For the problem below, I haven't found the root cause, still working on it.

Best Regards,
Chen Qi

Quoted 13 lines

In fact, this is a problem I found when I was dealing with another problem with eSDK.
*Yocto's current eSDK cannot be successfully installed on a different distro.* For example, you build esdk on ubunt18 and install the esdk on ubuntu20, then you'll get an error of pseudo do_fetch task re-run unexpectedly. I think it might be related to NATIVELSBSTRING, but I haven't found out why.
Manually setting NATIVELSBSTRING to some fixed string such as 'Linux' will solve the above problem.

I'm still checking the codes. If you have some idea/suggestion, please let me know.

Best Regards,
Chen Qi

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#6 lore raw
Quoted 39 lines · Chen Qi
Quoted 25 lines · Chen Qi
Quoted 13 lines · Richard Purdie
Quoted 6 lines · Chen Qi

As we delete the .git/ directory, it's impossible to get METADATA_REVISION inside eSDK. Because of this, we meet the following warning when installing eSDK.

WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64

Quoted 6 lines

So we record METADATA_REVISION in eSDK generation time to fix this problem.

---
  meta/classes/populate_sdk_ext.bbclass | 3 +++
  1 file changed, 3 insertions(+)
Quoted 11 lines

I have to wonder why base-files is depending on the metadata-revision? That implies that base-files and hence anything that depends upon it would rebuild each time you update to a new revision. Is that really what you want?

Cheers,

Richard

Quoted 5 lines

Hi Richard,

I tested it on pure yocto project. No extra layer or configuration.
I'm also wondering why base-files do_install depends on METADATA_REVISION. The strange thing is that things do not rebuild at eSDK installation time. There's only a warning about base-files do_install signature.

Just found answer for the above problem.
A previous commit changes poky.conf to make DISTRO_VERSION contain METADATA_REVISION. """
poky.conf: do not write current date into distro version, use git hash instead

"""

Quoted 15 lines

I also confirmed that merely updating the git revision will result in a rebuild of base-files.
Rebuild of base-files does not result in rebuild of other recipes because we have 'base-files' in SIGGEN_EXCLUDERECIPES_ABISAFE.

For the problem below, I haven't found the root cause, still working on it.

Best Regards,
Chen Qi

Quoted 7 lines

In fact, this is a problem I found when I was dealing with another problem with eSDK.
*Yocto's current eSDK cannot be successfully installed on a different distro.* For example, you build esdk on ubunt18 and install the esdk on ubuntu20, then you'll get an error of pseudo do_fetch task re-run unexpectedly. I think it might be related to NATIVELSBSTRING, but I haven't found out why.
Manually setting NATIVELSBSTRING to some fixed string such as 'Linux' will solve the above problem.

I'm still checking the codes. If you have some idea/suggestion, please let me know.

I think I've found the problem.

"""
commit 3ecf5d9692fec97b37af6a4e6c747a4e2c2ca292

Author: Richard Purdie <[email protected]>
Date:   Sat Nov 21 16:16:40 2020 +0000

uninative: Don't use single sstate for pseudo-native

pseudo-native is a bit special. It conditionally compiles in
support for
xattr, statx and statvfs amongst other options. If a pseudo-native
binary is
used on a system where these functions are present but it wasn't
compiled in
we see hard to debug permissions problems.
[snip]
"""

This commit effectively says that the pseudo-native's sstate cache should not be shared among hosts with different distros, as it's appending ${ORIGLSBSTRING} to SSTATE_PKGARCH for pseudo-native.

I'm not familiar with the pseudo's codes. But I'm wondering if it's possible for pseudo to always build in attr support and behave differently on different hosts/filesystems.

Best Regards,
Chen Qi

Quoted 10 lines
Quoted 5 lines

Best Regards,
Chen Qi

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#8 lore raw
Quoted 75 lines · Chen Qi
Quoted 39 lines · Chen Qi
Quoted 25 lines · Chen Qi
Quoted 13 lines · Richard Purdie
Quoted 6 lines · Chen Qi

As we delete the .git/ directory, it's impossible to get METADATA_REVISION inside eSDK. Because of this, we meet the following warning when installing eSDK.

WARNING: The base-files:do_install sig is computed to be 16b9d96148d45de183cc94667aae016ec7d102d48255456381e718cd4bbd0aa0, \
but the sig is locked to 6eb0dcaed504282becee94662481d79264db920dee1f7deda18230133fff8f36 in SIGGEN_LOCKEDSIGS_t-qemux86-64

Quoted 6 lines

So we record METADATA_REVISION in eSDK generation time to fix this problem.

---
  meta/classes/populate_sdk_ext.bbclass | 3 +++
  1 file changed, 3 insertions(+)
Quoted 11 lines

I have to wonder why base-files is depending on the metadata-revision? That implies that base-files and hence anything that depends upon it would rebuild each time you update to a new revision. Is that really what you want?

Cheers,

Richard

Quoted 5 lines

Hi Richard,

I tested it on pure yocto project. No extra layer or configuration.
I'm also wondering why base-files do_install depends on METADATA_REVISION. The strange thing is that things do not rebuild at eSDK installation time. There's only a warning about base-files do_install signature.

Just found answer for the above problem.
A previous commit changes poky.conf to make DISTRO_VERSION contain METADATA_REVISION. """
poky.conf: do not write current date into distro version, use git hash instead

"""

Quoted 15 lines

I also confirmed that merely updating the git revision will result in a rebuild of base-files.
Rebuild of base-files does not result in rebuild of other recipes because we have 'base-files' in SIGGEN_EXCLUDERECIPES_ABISAFE.

For the problem below, I haven't found the root cause, still working on it.

Best Regards,
Chen Qi

Quoted 7 lines

In fact, this is a problem I found when I was dealing with another problem with eSDK.
*Yocto's current eSDK cannot be successfully installed on a different distro.* For example, you build esdk on ubunt18 and install the esdk on ubuntu20, then you'll get an error of pseudo do_fetch task re-run unexpectedly. I think it might be related to NATIVELSBSTRING, but I haven't found out why.
Manually setting NATIVELSBSTRING to some fixed string such as 'Linux' will solve the above problem.

I'm still checking the codes. If you have some idea/suggestion, please let me know.

I think I've found the problem.

"""
commit 3ecf5d9692fec97b37af6a4e6c747a4e2c2ca292

Author: Richard Purdie <[email protected]>
Date:   Sat Nov 21 16:16:40 2020 +0000

uninative: Don't use single sstate for pseudo-native

pseudo-native is a bit special. It conditionally compiles in
support for
xattr, statx and statvfs amongst other options. If a pseudo-native
binary is
used on a system where these functions are present but it wasn't
compiled in
we see hard to debug permissions problems.
[snip]
"""

Quoted 4 lines

This commit effectively says that the pseudo-native's sstate cache should not be shared among hosts with different distros, as it's appending ${ORIGLSBSTRING} to SSTATE_PKGARCH for pseudo-native.

I'm not familiar with the pseudo's codes. But I'm wondering if it's possible for pseudo to always build in attr support and behave differently on different hosts/filesystems.

Just checked the pseudo recipe. It seems that attr is always enabled.
DEPENDS = "sqlite3 attr"
PSEUDO_EXTRA_OPTS ?= "--enable-force-async --without-passwd-fallback --enable-epoll --enable-xattr"

I think there should be no circumstances where the xattr support is not compiled in, yet the pseudo-native sstate cache is still reused. I'll send out a patch to revert the commit.

Best Regards,
Chen Qi

Quoted 16 lines

Best Regards,
Chen Qi

Quoted 8 lines
Quoted 4 lines

Best Regards,
Chen Qi

#9 lore raw
Quoted 22 lines · ChenQi

 I think I've found the problem.

 """
 commit 3ecf5d9692fec97b37af6a4e6c747a4e2c2ca292
 Author: Richard Purdie <[email protected]>
 Date:   Sat Nov 21 16:16:40 2020 +0000

     uninative: Don't use single sstate for pseudo-native

     pseudo-native is a bit special. It conditionally compiles in support for
     xattr, statx and statvfs amongst other options. If a pseudo-native binary is
     used on a system where these functions are present but it wasn't compiled in
     we see hard to debug permissions problems.
 [snip]
 """

 This commit effectively says that the pseudo-native's sstate cache should not be shared among hosts with different distros, as it's appending ${ORIGLSBSTRING} to SSTATE_PKGARCH for pseudo-native.
 I'm not familiar with the pseudo's codes. But I'm wondering if it's possible for pseudo to always build in attr support and behave differently on different hosts/filesystems.

Quoted 8 lines



 Just checked the pseudo recipe. It seems that attr is always enabled.
 DEPENDS = "sqlite3 attr"
 PSEUDO_EXTRA_OPTS ?= "--enable-force-async --without-passwd-fallback --enable-epoll --enable-xattr"
 I think there should be no circumstances where the xattr support is not compiled in, yet the pseudo-native sstate cache is still reused. I'll send out a patch to revert the commit.

I was very reluctant to add that commit in the first place and from what I remember, it is a lot more tricky that just adding the right dependencies/configuration. I'd need to look at the code to remember but there was code only being enabled on some distros and we'd have to have it enabled in all cases even when headers and library functions were not present.

Cheers,

Richard