Skip to contentSkip to conversation

license.bbclass: only create hardlinks within $TMPDIR

RFC PATCH In review Rasmus Villemoes 2 participants 6 messages May 1 – May 5

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 license.bbclass: only create hardlinks within $TMPDIR

From: Rasmus Villemoes <[email protected]>

This is _not_ meant to be applied, but merely acts as a place to start a discussion.

In our CI setup (which is gitlab-based, but I'm not sure that's too relevant), we sometimes see spurious errors like

touch: setting times of '[...]/rootfs/usr/share/common-licenses/go2rtc/COPYING.MIT': Operation not permitted

Often, the error disappears when the job is re-run, which I don't really claim to understand. But digging into this, it seems that it is related to the use of hard-linking from common-licenses dir in the oe-core repository and into the build directory (and then later steps create further hard links).

While the test done here for whether a hard link can be made is mostly correct (assuming the the sysctl protected_hardlinks is 1 as it is in most distros), those are not the same requirements needed for a subsequent utimensat(2) system call to be applied - that essentially requires one to be the owner of the file, not merely having write access.

We are not the only ones to have observed these spurious failures:
https://stackoverflow.com/questions/77926112/yocto-gitlab-ci-job-sometimes-causes-touch-to-give-operation-not-permitted

But apart from these concrete failures, at a higher level it seems wrong for a build to cause the mtime property of files in the layer directories to change, and it seems even more wrong considering that if one of those files are rewritten (via open(O_TRUNC)+write(), not via "write new file and rename on top"), that affects what is already in build directories. Also, one could imagine two parallel builds using the same layer directories as source, but with different values for the REPRODUCIBLE_TIMESTAMP_ROOTFS, which would result in rather random end results since both builds modify the mtimes of the same inodes.

I do understand that using hard links is quite valuable for saving time and space (I see GPL-2.0-only having 135 links, so that alone is a few MB), which is why I don't think this should be applied as-is.

Could we do something like create a $TMPDIR/license-pool/, and whenever encountering a src not inside TMPDIR, copy that file to $TMPDIR/license-pool/<basename>-<sha256 of source> [if it doesn't already exist], and then use the latter as src in the subsequent "can we hardlink" logic? That would then also improve the case where the meta-layers are (bind)mounted R/O, and we thus always end up copying.

---
 meta/classes-global/license.bbclass | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
meta/classes-global/license.bbclass +5 -4
index af5f1ed41d..61049d76fe 100644
@@ -36,7 +36,7 @@ python do_populate_lic() {
 
     # The base directory we wrangle licenses to
     destdir = os.path.join(d.getVar('LICSSTATEDIR'), d.getVar('LICENSE_DEPLOY_PATHCOMPONENT'), d.getVar('PN'))
-    copy_license_files(lic_files_paths, destdir)
+    copy_license_files(lic_files_paths, destdir, d)
     info = get_recipe_info(d)
     with open(os.path.join(destdir, "recipeinfo"), "w") as f:
         for key in sorted(info.keys()):
@@ -52,7 +52,7 @@ python perform_packagecopy:prepend () {
 
         # LICENSE_FILES_DIRECTORY starts with '/' so os.path.join cannot be used to join D and LICENSE_FILES_DIRECTORY
         destdir = d.getVar('D') + os.path.join(d.getVar('LICENSE_FILES_DIRECTORY'), d.getVar('PN'))
-        copy_license_files(lic_files_paths, destdir)
+        copy_license_files(lic_files_paths, destdir, d)
         add_package_and_files(d)
 }
 perform_packagecopy[vardeps] += "LICENSE_CREATE_PACKAGE"
@@ -76,10 +76,11 @@ def add_package_and_files(d):
         d.setVar('PACKAGES', "%s %s" % (pn_lic, packages))
         d.setVar('FILES:' + pn_lic, files)
 
-def copy_license_files(lic_files_paths, destdir):
+def copy_license_files(lic_files_paths, destdir, d):
     import shutil
     import errno
 
+    tmpdir = d.getVar("TMPDIR")
     bb.utils.mkdirhier(destdir)
     for (basename, path, beginline, endline) in lic_files_paths:
         try:
@@ -89,7 +90,7 @@ def copy_license_files(lic_files_paths, destdir):
                 os.remove(dst)
             if os.path.islink(src):
                 src = os.path.realpath(src)
-            canlink = os.access(src, os.W_OK) and (os.stat(src).st_dev == os.stat(destdir).st_dev) and beginline is None and endline is None
+            canlink = src.startswith(tmpdir) and os.access(src, os.W_OK) and (os.stat(src).st_dev == os.stat(destdir).st_dev) and beginline is None and endline is None
             if canlink:
                 try:
                     os.link(src, dst)
Signature · 1 line
2.53.0
#1 lore raw

On Fri, 2026-05-01 at 20:36 +0200, Rasmus Villemoes via lists.openembedded.org wrote:
From: Rasmus Villemoes <[email protected]>

This is _not_ meant to be applied, but merely acts as a place to start a discussion.

In our CI setup (which is gitlab-based, but I'm not sure that's too relevant), we sometimes see spurious errors like

  touch: setting times of '[...]/rootfs/usr/share/common- licenses/go2rtc/COPYING.MIT': Operation not permitted

Often, the error disappears when the job is re-run, which I don't really claim to understand. But digging into this, it seems that it is related to the use of hard-linking from common-licenses dir in the oe-core repository and into the build directory (and then later steps create further hard links).

While the test done here for whether a hard link can be made is mostly correct (assuming the the sysctl protected_hardlinks is 1 as it is in most distros), those are not the same requirements needed for a subsequent utimensat(2) system call to be applied - that essentially requires one to be the owner of the file, not merely having write access.

We are not the only ones to have observed these spurious failures:
https://stackoverflow.com/questions/77926112/yocto-gitlab-ci-job-sometimes-causes-touch-to-give-operation-not-permitted

But apart from these concrete failures, at a higher level it seems wrong for a build to cause the mtime property of files in the layer directories to change, and it seems even more wrong considering that if one of those files are rewritten (via open(O_TRUNC)+write(), not via "write new file and rename on top"), that affects what is already in build directories. Also, one could imagine two parallel builds using the same layer directories as source, but with different values for the REPRODUCIBLE_TIMESTAMP_ROOTFS, which would result in rather random end results since both builds modify the mtimes of the same inodes.

I do understand that using hard links is quite valuable for saving time and space (I see GPL-2.0-only having 135 links, so that alone is a few MB), which is why I don't think this should be applied as-is.

Could we do something like create a $TMPDIR/license-pool/, and whenever encountering a src not inside TMPDIR, copy that file to $TMPDIR/license-pool/<basename>-<sha256 of source> [if it doesn't already exist], and then use the latter as src in the subsequent "can we hardlink" logic? That would then also improve the case where the meta-layers are (bind)mounted R/O, and we thus always end up copying.

I agree it does look like there is a potential issue here.

I'm not sure we have ever claimed isolation between the build metadata and the builds, I know that for example patch files are linked in, so that when you update patches, it updates the metadata more easily. I can see why you might want that, equally, the ability to update patches that way is a nice usability feature.

Did you track down where the utime of the files was being changed? From the link, it looks like:

    find  ${IMAGE_ROOTFS} -print0 | xargs -0 touch -h  --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS

in reproducible_final_image_task in image.bbclass

I think that function could be changed to clamp to anything newer than REPRODUCIBLE_TIMESTAMP_ROOTFS rather than changing all files, and that might actually avoid the issue.

Cheers,

Richard

#2 lore raw
Quoted 32 lines · Richard Purdie

On Fri, 2026-05-01 at 20:36 +0200, Rasmus Villemoes via lists.openembedded.org wrote:
From: Rasmus Villemoes <[email protected]>

This is _not_ meant to be applied, but merely acts as a place to start a discussion.

Quoted 11 lines

In our CI setup (which is gitlab-based, but I'm not sure that's too relevant), we sometimes see spurious errors like

  touch: setting times of '[...]/rootfs/usr/share/common- licenses/go2rtc/COPYING.MIT': Operation not permitted

I do understand that using hard links is quite valuable for saving time and space (I see GPL-2.0-only having 135 links, so that alone is a few MB), which is why I don't think this should be applied as-is.

Quoted 6 lines

Could we do something like create a $TMPDIR/license-pool/, and whenever encountering a src not inside TMPDIR, copy that file to $TMPDIR/license-pool/<basename>-<sha256 of source> [if it doesn't already exist], and then use the latter as src in the subsequent "can we hardlink" logic? That would then also improve the case where the meta-layers are (bind)mounted R/O, and we thus always end up copying.

I agree it does look like there is a potential issue here.

I'm not sure we have ever claimed isolation between the build metadata and the builds, I know that for example patch files are linked in, so that when you update patches, it updates the metadata more easily. I can see why you might want that, equally, the ability to update patches that way is a nice usability feature.

Quoted 6 lines

Did you track down where the utime of the files was being changed? From the link, it looks like:

    find  ${IMAGE_ROOTFS} -print0 | xargs -0 touch -h  --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS

in reproducible_final_image_task in image.bbclass

Yes, it it precisely that command/function where it triggers.

Quoted 3 lines

I think that function could be changed to clamp to anything newer than REPRODUCIBLE_TIMESTAMP_ROOTFS rather than changing all files, and that might actually avoid the issue.

I don't see how. First, how would that then serve the purpose of producing something reproducible? It seems that the timestamps would or could then depend on when the meta-data repo was cloned. Second, if the mtime is still going to be changed sometimes under some conditions, the problem is not really gone. And third, I think it would make the implementation much more complicated, if one would need to make a decision for each file for whether to touch it or not (i.e. the simple find|xargs would no longer suffice).

Rasmus

#3 lore raw
Quoted 47 lines · Rasmus Villemoes
Quoted 32 lines · Richard Purdie

On Fri, 2026-05-01 at 20:36 +0200, Rasmus Villemoes via lists.openembedded.org wrote:
From: Rasmus Villemoes <[email protected]>

This is _not_ meant to be applied, but merely acts as a place to start a discussion.

Quoted 11 lines

In our CI setup (which is gitlab-based, but I'm not sure that's too relevant), we sometimes see spurious errors like

  touch: setting times of '[...]/rootfs/usr/share/common- licenses/go2rtc/COPYING.MIT': Operation not permitted

I do understand that using hard links is quite valuable for saving time and space (I see GPL-2.0-only having 135 links, so that alone is a few MB), which is why I don't think this should be applied as-is.

Quoted 6 lines

Could we do something like create a $TMPDIR/license-pool/, and whenever encountering a src not inside TMPDIR, copy that file to $TMPDIR/license-pool/<basename>-<sha256 of source> [if it doesn't already exist], and then use the latter as src in the subsequent "can we hardlink" logic? That would then also improve the case where the meta-layers are (bind)mounted R/O, and we thus always end up copying.

I agree it does look like there is a potential issue here.

I'm not sure we have ever claimed isolation between the build metadata and the builds, I know that for example patch files are linked in, so that when you update patches, it updates the metadata more easily. I can see why you might want that, equally, the ability to update patches that way is a nice usability feature.

Quoted 6 lines

Did you track down where the utime of the files was being changed? From the link, it looks like:

    find  ${IMAGE_ROOTFS} -print0 | xargs -0 touch -h  --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS

in reproducible_final_image_task in image.bbclass

Yes, it it precisely that command/function where it triggers.

Quoted 3 lines

I think that function could be changed to clamp to anything newer than REPRODUCIBLE_TIMESTAMP_ROOTFS rather than changing all files, and that might actually avoid the issue.

Quoted 8 lines

I don't see how. First, how would that then serve the purpose of producing something reproducible? It seems that the timestamps would or could then depend on when the meta-data repo was cloned. Second, if the mtime is still going to be changed sometimes under some conditions, the problem is not really gone. And third, I think it would make the implementation much more complicated, if one would need to make a decision for each file for whether to touch it or not (i.e. the simple find|xargs would no longer suffice).

Pretty much all the rest of the system works like this for reproducibility. Whether you make everything REPRODUCIBLE_TIMESTAMP_ROOTFS or just clamp to a max value of REPRODUCIBLE_TIMESTAMP_ROOTFS doesn't really matter since you're using that value regardless.

Basically anything touched as part of the build process (newer than the timestamp) would be reset, everything else (older than that) should be deterministic.

The value of the actual timestamp used is a different discussion but we're not changing that.

Having thought about this a bit, I do think that is the right solution, and it does match how we handle timestamps elsewhere like the packages.

Cheers,

Richard

#4 lore raw
Quoted 67 lines · Richard Purdie
Quoted 47 lines · Rasmus Villemoes
Quoted 32 lines · Richard Purdie

On Fri, 2026-05-01 at 20:36 +0200, Rasmus Villemoes via lists.openembedded.org wrote:
From: Rasmus Villemoes <[email protected]>

This is _not_ meant to be applied, but merely acts as a place to start a discussion.

Quoted 11 lines

In our CI setup (which is gitlab-based, but I'm not sure that's too relevant), we sometimes see spurious errors like

  touch: setting times of '[...]/rootfs/usr/share/common- licenses/go2rtc/COPYING.MIT': Operation not permitted

I do understand that using hard links is quite valuable for saving time and space (I see GPL-2.0-only having 135 links, so that alone is a few MB), which is why I don't think this should be applied as-is.

Quoted 6 lines

Could we do something like create a $TMPDIR/license-pool/, and whenever encountering a src not inside TMPDIR, copy that file to $TMPDIR/license-pool/<basename>-<sha256 of source> [if it doesn't already exist], and then use the latter as src in the subsequent "can we hardlink" logic? That would then also improve the case where the meta-layers are (bind)mounted R/O, and we thus always end up copying.

I agree it does look like there is a potential issue here.

I'm not sure we have ever claimed isolation between the build metadata and the builds, I know that for example patch files are linked in, so that when you update patches, it updates the metadata more easily. I can see why you might want that, equally, the ability to update patches that way is a nice usability feature.

Quoted 6 lines

Did you track down where the utime of the files was being changed? From the link, it looks like:

    find  ${IMAGE_ROOTFS} -print0 | xargs -0 touch -h  --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS

in reproducible_final_image_task in image.bbclass

Yes, it it precisely that command/function where it triggers.

Quoted 3 lines

I think that function could be changed to clamp to anything newer than REPRODUCIBLE_TIMESTAMP_ROOTFS rather than changing all files, and that might actually avoid the issue.

Quoted 8 lines

I don't see how. First, how would that then serve the purpose of producing something reproducible? It seems that the timestamps would or could then depend on when the meta-data repo was cloned. Second, if the mtime is still going to be changed sometimes under some conditions, the problem is not really gone. And third, I think it would make the implementation much more complicated, if one would need to make a decision for each file for whether to touch it or not (i.e. the simple find|xargs would no longer suffice).

Pretty much all the rest of the system works like this for reproducibility. Whether you make everything REPRODUCIBLE_TIMESTAMP_ROOTFS or just clamp to a max value of REPRODUCIBLE_TIMESTAMP_ROOTFS doesn't really matter since you're using that value regardless.

Basically anything touched as part of the build process (newer than the timestamp) would be reset, everything else (older than that) should be deterministic.

Quoted 5 lines

The value of the actual timestamp used is a different discussion but we're not changing that.

Having thought about this a bit, I do think that is the right solution, and it does match how we handle timestamps elsewhere like the packages.

OK, but then that still wouldn't solve the problem we see, where the build has decided that it can create a hard link, but then later it fails to perform the futimensat syscall because the user running bitbake is not (apparently) the user owning the inode.

As I said, I don't really understand how that actually sometimes happens and most of the time doesn't, perhaps it is some weird interaction between how gitlab starts docker containers and perhaps with some pseudo in the mix.

Rasmus

#5 lore raw
Quoted 80 lines · Rasmus Villemoes
Quoted 67 lines · Richard Purdie
Quoted 47 lines · Rasmus Villemoes
Quoted 32 lines · Richard Purdie

On Fri, 2026-05-01 at 20:36 +0200, Rasmus Villemoes via lists.openembedded.org wrote:
From: Rasmus Villemoes <[email protected]>

This is _not_ meant to be applied, but merely acts as a place to start a discussion.

Quoted 11 lines

In our CI setup (which is gitlab-based, but I'm not sure that's too relevant), we sometimes see spurious errors like

  touch: setting times of '[...]/rootfs/usr/share/common- licenses/go2rtc/COPYING.MIT': Operation not permitted

I do understand that using hard links is quite valuable for saving time and space (I see GPL-2.0-only having 135 links, so that alone is a few MB), which is why I don't think this should be applied as-is.

Quoted 6 lines

Could we do something like create a $TMPDIR/license-pool/, and whenever encountering a src not inside TMPDIR, copy that file to $TMPDIR/license-pool/<basename>-<sha256 of source> [if it doesn't already exist], and then use the latter as src in the subsequent "can we hardlink" logic? That would then also improve the case where the meta-layers are (bind)mounted R/O, and we thus always end up copying.

I agree it does look like there is a potential issue here.

I'm not sure we have ever claimed isolation between the build metadata and the builds, I know that for example patch files are linked in, so that when you update patches, it updates the metadata more easily. I can see why you might want that, equally, the ability to update patches that way is a nice usability feature.

Quoted 6 lines

Did you track down where the utime of the files was being changed? From the link, it looks like:

    find  ${IMAGE_ROOTFS} -print0 | xargs -0 touch -h  --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS

in reproducible_final_image_task in image.bbclass

Yes, it it precisely that command/function where it triggers.

Quoted 3 lines

I think that function could be changed to clamp to anything newer than REPRODUCIBLE_TIMESTAMP_ROOTFS rather than changing all files, and that might actually avoid the issue.

Quoted 8 lines

I don't see how. First, how would that then serve the purpose of producing something reproducible? It seems that the timestamps would or could then depend on when the meta-data repo was cloned. Second, if the mtime is still going to be changed sometimes under some conditions, the problem is not really gone. And third, I think it would make the implementation much more complicated, if one would need to make a decision for each file for whether to touch it or not (i.e. the simple find|xargs would no longer suffice).

Pretty much all the rest of the system works like this for reproducibility. Whether you make everything REPRODUCIBLE_TIMESTAMP_ROOTFS or just clamp to a max value of REPRODUCIBLE_TIMESTAMP_ROOTFS doesn't really matter since you're using that value regardless.

Basically anything touched as part of the build process (newer than the timestamp) would be reset, everything else (older than that) should be deterministic.

Quoted 5 lines

The value of the actual timestamp used is a different discussion but we're not changing that.

Having thought about this a bit, I do think that is the right solution, and it does match how we handle timestamps elsewhere like the packages.

OK, but then that still wouldn't solve the problem we see, where the build has decided that it can create a hard link, but then later it fails to perform the futimensat syscall because the user running bitbake is not (apparently) the user owning the inode.

Quoted 4 lines

As I said, I don't really understand how that actually sometimes happens and most of the time doesn't, perhaps it is some weird interaction between how gitlab starts docker containers and perhaps with some pseudo in the mix.

I'm forgetting that git doesn't give consistent timestamps upon checkout. If it did, my suggestion would work however it does not.

I still don't think we should be changing every timestamp in an image as most timestamps should already be consistent. Where we copy data from source control into the image, we are going to need to do something though.

To hit the error, you're probably using license_image.bbclass and hitting the:

oe.path.copyhardlink(pkg_license, rootfs_license)

codepath by having COPY_LIC_MANIFEST set?

Perhaps the answer is just not to hardlink the files into the image in license_image.bbclass? I think the other usages are safe...

Cheers,

Richard