Skip to contentSkip to conversation

sstate: Add fallback when hardlink fails

PATCH In review Michael Arndt 2 participants 5 messages Apr 15

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sstate: Add fallback when hardlink fails

Previously the sstate didn't work on file systems that don't support hardlinks. For example when using WebDAV to share the sstate. This change avoids the problem by adding a fallback in case the hardlink fails.

---
 meta/classes-global/sstate.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
meta/classes-global/sstate.bbclass +10 -2
index 88449d19c7..2b18cafb60 100644
@@ -795,6 +795,14 @@ python sstate_create_and_sign_package () {
         except:
             pass
 
+    # Create hardlink with fallback to rename. Useful for file systems that
+    # don't support hardlinks.
+    def hardlink(src, dst):
+        try:
+            os.link(src, dst)
+        except:
+            src.rename(dst)
+
     def update_file(src, dst, force=False):
         if dst.is_symlink() and not dst.exists():
             force=True
@@ -804,7 +812,7 @@ python sstate_create_and_sign_package () {
             if force:
                 src.rename(dst)
             else:
-                os.link(src, dst)
+                hardlink(src, dst)
             return True
         except:
             pass
@@ -862,7 +870,7 @@ python sstate_create_and_sign_package () {
         with NamedTemporaryFile(prefix=sstate_pkg.name, dir=sstate_pkg.parent) as tmp_pkg_fd:
             tmp_pkg = tmp_pkg_fd.name
             sstate_archive_package(tmp_pkg, d)
-            update_file(tmp_pkg, sstate_pkg)
+            update_file(Path(tmp_pkg), sstate_pkg)
             # update_file() may have renamed tmp_pkg, which must exist when the
             # NamedTemporaryFile() context handler ends.
             touch(Path(tmp_pkg))
#1 lore raw
Quoted 8 lines · Michael Arndt via lists.openembedded.org

Previously the sstate didn't work on file systems that don't support hardlinks. For example when using WebDAV to share the sstate. This change avoids the problem by adding a fallback in case the hardlink fails.

---
 meta/classes-global/sstate.bbclass | 12 ++++++++++--

 1 file changed, 10 insertions(+), 2 deletions(-)

The change makes it unconditionally overwrite the file in the case of any failure. This totally breaks the point of the existing code :(.

The challenge is that the sstate directory is shared, so two builds can write to the same file at the same time. Our policy is "first wins" and anything later is just skipped. The code also needs to make sure that if the sstate file is written, the paired signature matches.

I appreciate the issue you're trying to solve but this isn't the right fix. Generic try/except cases are almost always a red flag in general.

Cheers,

Richard

#2 lore raw

Hello Richard,

Quoted 4 lines

The challenge is that the sstate directory is shared, so two builds can write to the same file at the same time. Our policy is "first wins" and anything later is just skipped. The code also needs to make sure that if the sstate file is written, the paired signature matches.

Thanks for taking a look at my change. I guess then there is no simple fix for the problem, because checking if the file exists and replacing it must be a single atomic operation. Without hardlinks that is a problem.

Quoted 1 line

Generic try/except cases are almost always a red flag in general.

Agreed, that was also the reason why I was surprised that my sstate didn't work. The existing code already swallows the IOException from os.link so you don't see the error about the missing hardlink support.

I guess I will have to use a different file system, but thanks anyway.

Greetings,
Michael

#3 lore raw
Quoted 11 lines · Michael Arndt

Hello Richard,

Quoted 4 lines

The challenge is that the sstate directory is shared, so two builds can write to the same file at the same time. Our policy is "first wins" and anything later is just skipped. The code also needs to make sure that if the sstate file is written, the paired signature matches.

Thanks for taking a look at my change. I guess then there is no simple fix for the problem, because checking if the file exists and replacing it must be a single atomic operation. Without hardlinks that is a problem.

Quoted 7 lines

 > Generic try/except cases are almost always a red flag in general.

Agreed, that was also the reason why I was surprised that my sstate didn't work. The existing code already swallows the IOException from os.link so you don't see the error about the missing hardlink support.

I guess I will have to use a different file system, but thanks anyway.

If you can catch a specific exception about it being not supported, we could at least show that to the user and fail. I noticed there are other generic try/except blocks in there which could probably be improved.

This is pretty core/key code which we've had challenges with in the past though and it does need to work simply/reliably!

Cheers,

Richard

#4 lore raw
Quoted 4 lines

If you can catch a specific exception about it being not supported, we could at least show that to the user and fail. I noticed there are other generic try/except blocks in there which could probably be improved.

Sure I know which errno the exception contains (ENOSYS), but wouldn't it make more sense to only ignore the one error that we want to ignore and fail for everything else? As I understand it we only want to ignore is the FileExistsError, correct?

What would be preferred here?

Greetings,
Michael