Skip to contentSkip to conversation

Use shutil.move when os.rename fails

PATCH In review [email protected] 7 participants 34 messages Mar 29, 2021 – Aug 22, 2022 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Use shutil.move when os.rename fails

---
 meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +22 -4
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break
+        except OSError:
+            shutil.move(src, dest)
 
     return True
 
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break
+        except OSError:
+            shutil.move(plain, pdir)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
Signature · 1 line
2.29.2
#1 lore raw

Can you document the cases that os.rename() is failing ? And also why would we expect the shutil.move() to work in those cases ?

If a change like this cases issues in the future, we need that extra information in the commit head for proper triage.

Bruce

Quoted 83 lines · Devendra Tewari
---
 meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +22 -5
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break
+        except OSError:
+            shutil.move(src, dest)
 
     return True
 
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break
+        except OSError:
+            shutil.move(plain, pdir)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#2 lore raw

Yes please quote a bit from the python manpage [1] - I certainly see the difference (and the edge cases where os.rename might fail), but that should be documented as part of the commit message

[1] https://docs.python.org/3/library/shutil.html#shutil.move

Quoted 99 lines · Bruce Ashfield

Can you document the cases that os.rename() is failing ? And also why would we expect the shutil.move() to work in those cases ?

If a change like this cases issues in the future, we need that extra information in the commit head for proper triage.

Bruce

Quoted 83 lines · Devendra Tewari
---
  meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----
  1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +22 -5
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
  def sstate_installpkgdir(ss, d):
      import oe.path
      import subprocess
+    import shutil
 
      sstateinst = d.getVar("SSTATE_INSTDIR")
      d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
      for state in ss['dirs']:
          prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
      sstate_install(ss, d)
 
      for plain in ss['plaindirs']:
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
          dest = plain
          bb.utils.mkdirhier(src)
          prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break
+        except OSError:
+            shutil.move(src, dest)
 
      return True
 
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 
  def sstate_package(ss, d):
      import oe.path
+    import shutil
 
      tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
                      continue
                  bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
          bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
      workdir = d.getVar('WORKDIR')
      sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
              pdir = plain.replace(sharedworkdir, sstatebuild)
          bb.utils.mkdirhier(plain)
          bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break
+        except OSError:
+            shutil.move(plain, pdir)
 
      d.setVar('SSTATE_BUILDDIR', sstatebuild)
      d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

#3 lore raw

Sure.

Would the following commit message be sufficient?

Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles the error with os.rename and retries with shutil.move.

Thanks,
Devendra

Quoted 98 lines · Konrad Weihmann

Yes please quote a bit from the python manpage [1] - I certainly see the difference (and the edge cases where os.rename might fail), but that should be documented as part of the commit message

[1] https://docs.python.org/3/library/shutil.html#shutil.move

Quoted 91 lines · Bruce Ashfield

Can you document the cases that os.rename() is failing ? And also why would we expect the shutil.move() to work in those cases ? If a change like this cases issues in the future, we need that extra information in the commit head for proper triage.
Bruce

Quoted 83 lines · Devendra Tewari
---
 meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +22 -5
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break
+        except OSError:
+            shutil.move(src, dest)
 
     return True
 
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break
+        except OSError:
+            shutil.move(plain, pdir)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

#4 lore raw

Also, this is due to https://bugzilla.yoctoproject.org/show_bug.cgi?id=1430.

Quoted 118 lines · Devendra Tewari

Sure.

Would the following commit message be sufficient?

Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles the error with os.rename and retries with shutil.move.

Thanks,
Devendra

Quoted 98 lines · Konrad Weihmann

Yes please quote a bit from the python manpage [1] - I certainly see the difference (and the edge cases where os.rename might fail), but that should be documented as part of the commit message

[1] https://docs.python.org/3/library/shutil.html#shutil.move

Quoted 91 lines · Bruce Ashfield

Can you document the cases that os.rename() is failing ? And also why would we expect the shutil.move() to work in those cases ? If a change like this cases issues in the future, we need that extra information in the commit head for proper triage.
Bruce

Quoted 83 lines · Devendra Tewari

---
meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----

1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):

def sstate_installpkgdir(ss, d):
import oe.path
import subprocess

meta/classes/sstate.bbclass +11 -2
+    import shutil
 
    sstateinst = d.getVar("SSTATE_INSTDIR")
    d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
    for state in ss['dirs']:
        prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
    sstate_install(ss, d)
 
    for plain in ss['plaindirs']:
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
        dest = plain
        bb.utils.mkdirhier(src)
        prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break
+        except OSError:
+            shutil.move(src, dest)
 
    return True
 
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 

def sstate_package(ss, d):
import oe.path

meta/classes/sstate.bbclass +11 -3
+    import shutil
 
    tmpdir = d.getVar('TMPDIR')
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
                    continue
                bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
        bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
    workdir = d.getVar('WORKDIR')
    sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
            pdir = plain.replace(sharedworkdir, sstatebuild)
        bb.utils.mkdirhier(plain)
        bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break
+        except OSError:
+            shutil.move(plain, pdir)
 
    d.setVar('SSTATE_BUILDDIR', sstatebuild)
    d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

#8 lore raw
In reply to [email protected]
Quoted 22 lines · Devendra Tewari

---
 meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----

 1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):

 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+ import shutil

     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])

meta/classes/sstate.bbclass +0 -0
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):

     for state in ss['dirs']:
         prepdir(state[1])

meta/classes/sstate.bbclass def sstate_installpkgdir(ss, d): · line 407

Quoted 4 lines
meta/classes/sstate.bbclass +3 -1
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break

That break should definitely not be there, that changes behaviour.

Quoted 10 lines
meta/classes/sstate.bbclass +2 -0
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])

     sstate_install(ss, d)

     for plain in ss['plaindirs']:

meta/classes/sstate.bbclass +0 -0
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):

         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)

meta/classes/sstate.bbclass def sstate_installpkgdir(ss, d): · line 423

Quoted 4 lines
meta/classes/sstate.bbclass +3 -1
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break

Same here...

Quoted 22 lines
meta/classes/sstate.bbclass +2 -0
+        except OSError:
+            shutil.move(src, dest)

     return True

meta/classes/sstate.bbclass +0 -0
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {

 def sstate_package(ss, d):
     import oe.path
+ import shutil

     tmpdir = d.getVar('TMPDIR')

meta/classes/sstate.bbclass +0 -0
@@ -664,7 +674,11 @@ def sstate_package(ss, d):

                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link)) bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))

meta/classes/sstate.bbclass def sstate_package(ss, d): · line 679

Quoted 4 lines
meta/classes/sstate.bbclass +3 -1
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break

and again...

Quoted 7 lines
meta/classes/sstate.bbclass +2 -0
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])

     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")

meta/classes/sstate.bbclass def sstate_package(ss, d): · line 693

Quoted 7 lines
meta/classes/sstate.bbclass +3 -1
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
              pdir = plain.replace(sharedworkdir, sstatebuild)
          bb.utils.mkdirhier(plain)
          bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break

and again...

Quoted 8 lines
meta/classes/sstate.bbclass +2 -0
+        except OSError:
+            shutil.move(plain, pdir)

     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)

#9 lore raw

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

From 9deb390dcdcaef66cec2fae39454c7fb3c81c4e4 Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Mon, 29 Mar 2021 19:41:02 -0300
Subject: [PATCH] Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles the error with os.rename and retries with shutil.move.

---
 meta/classes/sstate.bbclass               | 22 ++++++++++++++++++----
 vscode-bitbake-build/executeBitBakeCmd.sh |  3 +++
 2 files changed, 21 insertions(+), 4 deletions(-)
 create mode 100755 vscode-bitbake-build/executeBitBakeCmd.sh
meta/classes/sstate.bbclass +18 -4
index f579168162..301dfc27db 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,10 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -413,7 +417,10 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+        except OSError:
+            shutil.move(src, dest)
 
     return True
 
@@ -638,6 +645,7 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +672,10 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +685,10 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+        except OSError:
+            shutil.move(plain, pdir)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
vscode-bitbake-build/executeBitBakeCmd.sh new file +3 -0
new file mode 100755
index 0000000000..d7a4c5a5aa
@@ -0,0 +1,3 @@
+#!/bin/bash
+. ./oe-init-build-env vscode-bitbake-build > /dev/null
+bitbake-layers show-layers
\ No newline at end of file
Signature · 1 line
2.29.2
Quoted 103 lines · Richard Purdie
Quoted 18 lines · Devendra Tewari
---
 meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +1 -0
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])

meta/classes/sstate.bbclass def sstate_installpkgdir(ss, d): · line 408

Quoted 8 lines
meta/classes/sstate.bbclass +3 -1
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break

That break should definitely not be there, that changes behaviour.

Quoted 7 lines
meta/classes/sstate.bbclass +2 -0
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 

for plain in ss['plaindirs']:

meta/classes/sstate.bbclass def sstate_installpkgdir(ss, d): · line 423

Quoted 7 lines
meta/classes/sstate.bbclass +3 -1
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break

Same here...

Quoted 19 lines
meta/classes/sstate.bbclass +3 -0
+        except OSError:
+            shutil.move(src, dest)
 
 
     return True
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
 

tmpdir = d.getVar('TMPDIR')

meta/classes/sstate.bbclass def sstate_package(ss, d): · line 679

Quoted 7 lines
meta/classes/sstate.bbclass +3 -1
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break

and again...

Quoted 7 lines
meta/classes/sstate.bbclass +2 -0
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
 
     workdir = d.getVar('WORKDIR')

sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")

meta/classes/sstate.bbclass def sstate_package(ss, d): · line 693

Quoted 7 lines
meta/classes/sstate.bbclass +3 -1
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break

and again...

Quoted 7 lines
meta/classes/sstate.bbclass +2 -0
+        except OSError:
+            shutil.move(plain, pdir)
 
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)

d.setVar('SSTATE_INSTDIR', sstatebuild)

#10 lore raw
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

#11 lore raw

I did try that but got an error that does not happen when we try os.rename first. I'll try to reproduce it again.

I suspect there may be subtle differences in os.rename vs shutil.move with respect to what happens when origin and/or destination do not exist or are invalid.

Quoted 12 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

#12 lore raw
Quoted 4 lines · Devendra Tewari

I did try that but got an error that does not happen when we try os.rename first. I'll try to reproduce it again.

I suspect there may be subtle differences in os.rename vs shutil.move with respect to what happens when origin and/or destination do not exist or are invalid.

If there are subtle issues which you don't yet understand then that's all the more reason for not hiding the new shutil.move code behind a test which will pass for 99.9% of users. Please figure that out before sending another version of the patch.

Quoted 13 lines
Quoted 12 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

#13 lore raw

Will do. Thanks.

Quoted 27 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

I did try that but got an error that does not happen when we try os.rename first. I'll try to reproduce it again.

I suspect there may be subtle differences in os.rename vs shutil.move with respect to what happens when origin and/or destination do not exist or are invalid.

If there are subtle issues which you don't yet understand then that's all the more reason for not hiding the new shutil.move code behind a test which will pass for 99.9% of users. Please figure that out before sending another version of the patch.

Quoted 13 lines
Quoted 12 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

#14 lore raw

With the following patch

From f29ec67239094a256fcfc119fb75be90923d3448 Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Mon, 29 Mar 2021 21:11:56 -0300
Subject: [PATCH] Use shutil.move to rename sstate

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

shutil.move uses os.rename when destination is on the current filesystem, otherwise source is copied to destination and then removed.

---
 meta/classes/sstate.bbclass | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +6 -4
index f579168162..2e87697e3e 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,7 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -413,7 +414,7 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        shutil.move(src, dest)
 
     return True
 
@@ -638,6 +639,7 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +666,7 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        shutil.move(state[1], sstatebuild + state[0])
 
     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +676,7 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        shutil.move(plain, pdir)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
Signature · 40 lines
2.29.2

The build fails with

#12 2734.2 2021-03-30 01:14:29 - INFO     - NOTE: recipe linux-libc-headers-5.10-r0: task do_package_write_rpm: Started
#12 2734.4 2021-03-30 01:14:29 - INFO     - NOTE: recipe lzo-native-2.10-r0: task do_prepare_recipe_sysroot: Started
#12 2734.8 2021-03-30 01:14:30 - INFO     - NOTE: recipe lzo-native-2.10-r0: task do_prepare_recipe_sysroot: Succeeded
#12 2734.9 2021-03-30 01:14:30 - INFO     - NOTE: Running task 2441 of 4982 (virtual:native:/home/pi/docker-meta-raspberrypi/layers/poky/meta/recipes-support/lzo/lzo_2.10.bb:do_configure)
#12 2735.0 2021-03-30 01:14:30 - ERROR    - ERROR: linux-libc-headers-5.10-r0 do_package_write_rpm: Error executing a python function in exec_python_func() autogenerated:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 
#12 2735.0 2021-03-30 01:14:30 - ERROR    - The stack trace of python calls that resulted in this exception/failure was:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0001:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - *** 0002:write_specfile(d)
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0003:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/classes/package_rpm.bbclass', lineno: 342, function: write_specfile
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0338:        localdata.setVar('PKG', pkgname)
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0339:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0340:        localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0341:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - *** 0342:        conffiles = get_conffiles(pkg, d)
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0343:        dirfiles = localdata.getVar('DIRFILES')
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0344:        if dirfiles is not None:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0345:            dirfiles = dirfiles.split()
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0346:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/classes/package.bbclass', lineno: 304, function: get_conffiles
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0300:def get_conffiles(pkg, d):
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0301:    pkgdest = d.getVar('PKGDEST')
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0302:    root = os.path.join(pkgdest, pkg)
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0303:    cwd = os.getcwd()
#12 2735.0 2021-03-30 01:14:30 - ERROR    - *** 0304:    os.chdir(root)
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0305:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0306:    conffiles = d.getVar('CONFFILES_%s' % pkg);
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0307:    if conffiles == None:
#12 2735.0 2021-03-30 01:14:30 - ERROR    - 0308:        conffiles = d.getVar('CONFFILES')
#12 2735.0 2021-03-30 01:14:30 - ERROR    - Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/pi/docker-meta-raspberrypi/build/tmp/work/arm1176jzfshf-vfp-poky-linux-gnueabi/linux-libc-headers/5.10-r0/packages-split/linux-libc-headers-src'

A fully reproducible build can be found at https://github.com/tewarid/docker-meta-raspberrypi/tree/ee083820c565da646b745bc25e439c13f3205dbe. Please see build instructions in the readme.

I can’t seem to find a root cause and don’t have the time to look into this any further.
Quoted 32 lines · Devendra Tewari

Will do. Thanks.

Quoted 27 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

I did try that but got an error that does not happen when we try os.rename first. I'll try to reproduce it again.

I suspect there may be subtle differences in os.rename vs shutil.move with respect to what happens when origin and/or destination do not exist or are invalid.

If there are subtle issues which you don't yet understand then that's all the more reason for not hiding the new shutil.move code behind a test which will pass for 99.9% of users. Please figure that out before sending another version of the patch.

Quoted 13 lines
Quoted 12 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

#15 lore raw
In reply to Andre McCurdy
Quoted 10 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

This is a speed sensitive section of code and shutil has traditionally proved to be slow so I disagree with that, rename is very much preferred here where we can.

Cheers,

Richard

#16 lore raw

I understand that parallel tasks can result in such issues, but would expect some kind of dependency graph that would prevent them from happening. I wish I had more time to understand the issue fully.

Here’s the last version of my patch that still uses os.rename, and on error uses shutil.move, with the reasoning explained in the commit message. There are tons of os.rename in code elsewhere, and I’ve encountered similar issues when doing incremental builds, but I’ll leave fixing them to those with more time at hand. Cheers.

From aeba1f9728f69cdf2ce4ba285be86761d8024f9d Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Tue, 30 Mar 2021 08:27:40 -0300
Subject: [PATCH] Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles error with os.rename and retries with shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code.

---
 meta/classes/sstate.bbclass | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +18 -4
index f579168162..301dfc27db 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,10 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -413,7 +417,10 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+        except OSError:
+            shutil.move(src, dest)
 
     return True
 
@@ -638,6 +645,7 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +672,10 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +685,10 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+        except OSError:
+            shutil.move(plain, pdir)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
Signature · 1 line
2.29.2
Quoted 23 lines · Richard Purdie
Quoted 10 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

This is a speed sensitive section of code and shutil has traditionally proved to be slow so I disagree with that, rename is very much preferred here where we can.

Cheers,

Richard

#5 lore raw
In reply to Devendra Tewari

The link is for a 10-year old bug, probably not what you wanted. Also, if a patch fixes existing bug in bugzilla, it needs to reference it in commit message as well - [YOCTO#1234567]

Quoted 121 lines · Devendra Tewari

Also, this is due to https://bugzilla.yoctoproject.org/show_bug.cgi?id=1430.

Quoted 118 lines · Devendra Tewari

Sure.

Would the following commit message be sufficient?

Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles the error with os.rename and retries with shutil.move.

Thanks,
Devendra

Quoted 98 lines · Konrad Weihmann

Yes please quote a bit from the python manpage [1] - I certainly see the difference (and the edge cases where os.rename might fail), but that should be documented as part of the commit message

[1] https://docs.python.org/3/library/shutil.html#shutil.move

Quoted 91 lines · Bruce Ashfield

Can you document the cases that os.rename() is failing ? And also why would we expect the shutil.move() to work in those cases ? If a change like this cases issues in the future, we need that extra information in the commit head for proper triage.
Bruce

Quoted 83 lines · Devendra Tewari

---
meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----

1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):

def sstate_installpkgdir(ss, d):
import oe.path
import subprocess

meta/classes/sstate.bbclass +11 -2
+    import shutil
 
    sstateinst = d.getVar("SSTATE_INSTDIR")
    d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
    for state in ss['dirs']:
        prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
    sstate_install(ss, d)
 
    for plain in ss['plaindirs']:
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
        dest = plain
        bb.utils.mkdirhier(src)
        prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break
+        except OSError:
+            shutil.move(src, dest)
 
    return True
 
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 

def sstate_package(ss, d):
import oe.path

meta/classes/sstate.bbclass +11 -3
+    import shutil
 
    tmpdir = d.getVar('TMPDIR')
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
                    continue
                bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
        bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
    workdir = d.getVar('WORKDIR')
    sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
            pdir = plain.replace(sharedworkdir, sstatebuild)
        bb.utils.mkdirhier(plain)
        bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break
+        except OSError:
+            shutil.move(plain, pdir)
 
    d.setVar('SSTATE_BUILDDIR', sstatebuild)
    d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

#6 lore raw

Here's the correct link https://bugzilla.yoctoproject.org/show_bug.cgi?id=14301. I'll resubmit the patch referencing the bug in the commit message. Thanks.

Quoted 129 lines

Em 30 de mar. de 2021, à(s) 18:59, Denys Dmytriyenko <[email protected]> escreveu:

The link is for a 10-year old bug, probably not what you wanted. Also, if a patch fixes existing bug in bugzilla, it needs to reference it in commit message as well - [YOCTO#1234567]

Quoted 122 lines · Devendra Tewari

Also, this is due to https://bugzilla.yoctoproject.org/show_bug.cgi?id=1430.

Quoted 118 lines
Quoted 1 line · Devendra Tewari

Sure.

Would the following commit message be sufficient?

Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles the error with os.rename and retries with shutil.move.

Thanks,
Devendra

Quoted 98 lines · Konrad Weihmann

Yes please quote a bit from the python manpage [1] - I certainly see the difference (and the edge cases where os.rename might fail), but that should be documented as part of the commit message

[1] https://docs.python.org/3/library/shutil.html#shutil.move

Quoted 91 lines · Bruce Ashfield

Can you document the cases that os.rename() is failing ? And also why would we expect the shutil.move() to work in those cases ? If a change like this cases issues in the future, we need that extra information in the commit head for proper triage.
Bruce

Quoted 83 lines · Devendra Tewari

---
meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----

1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0
index f579168162..f94aa96d70 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):

def sstate_installpkgdir(ss, d):
import oe.path
import subprocess

meta/classes/sstate.bbclass +11 -2
+    import shutil
 
   sstateinst = d.getVar("SSTATE_INSTDIR")
   d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
   for state in ss['dirs']:
       prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+            break
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
   sstate_install(ss, d)
 
   for plain in ss['plaindirs']:
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
       dest = plain
       bb.utils.mkdirhier(src)
       prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+            break
+        except OSError:
+            shutil.move(src, dest)
 
   return True
 
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 

def sstate_package(ss, d):
import oe.path

meta/classes/sstate.bbclass +11 -3
+    import shutil
 
   tmpdir = d.getVar('TMPDIR')
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
                   continue
               bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
       bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+            break
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
   workdir = d.getVar('WORKDIR')
   sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
           pdir = plain.replace(sharedworkdir, sstatebuild)
       bb.utils.mkdirhier(plain)
       bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+            break
+        except OSError:
+            shutil.move(plain, pdir)
 
   d.setVar('SSTATE_BUILDDIR', sstatebuild)
   d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

#7 lore raw

From 57b633a93bd91b7b1aa21cce5ac7997b958ca917 Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Thu, 1 Apr 2021 16:07:25 -0300
Subject: [PATCH] Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles error with os.rename and retries with shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code.

[Yocto #14301]

---
 meta/classes/sstate.bbclass | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +10 -2
index 8e8efd18d5..60f7a94285 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,10 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -413,7 +417,10 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+        except OSError:
+            shutil.move(src, dest)
 
     return True
 
@@ -638,6 +645,7 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +672,10 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s

pointing at %s. Please replace this with a relative link." % (srcpath, link))
bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1],
sstatebuild + state[0]))
- os.rename(state[1], sstatebuild + state[0])
+ try:
+ os.rename(state[1], sstatebuild + state[0])
+ except OSError:
+ shutil.move(state[1], sstatebuild + state[0])

workdir = d.getVar('WORKDIR')
sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +685,10 @@ def sstate_package(ss, d):
pdir = plain.replace(sharedworkdir, sstatebuild)
bb.utils.mkdirhier(plain) bb.utils.mkdirhier(pdir)
- os.rename(plain, pdir)
+ try:
+ os.rename(plain, pdir)
+ except OSError:
+ shutil.move(plain, pdir)

d.setVar('SSTATE_BUILDDIR', sstatebuild)
d.setVar('SSTATE_INSTDIR', sstatebuild)

Signature · 4 lines
2.29.2


On Tue, Mar 30, 2021 at 7:39 PM Devendra Tewari <[email protected]>
Quoted 233 lines · wrote:

Here's the correct link
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14301. I'll resubmit the patch referencing the bug in the commit message. Thanks.

Em 30 de mar. de 2021, à(s) 18:59, Denys Dmytriyenko <[email protected]> escreveu:

The link is for a 10-year old bug, probably not what you wanted. Also, if a patch fixes existing bug in bugzilla, it needs to reference it in
commit message as well - [YOCTO#1234567]

On Mon, Mar 29, 2021 at 12:37:45PM -0300, Devendra Tewari wrote:

Also, this is due to
https://bugzilla.yoctoproject.org/show_bug.cgi?id=1430.

On 29 Mar 2021, at 12:35, Devendra Tewari <[email protected]> wrote:

Sure.

Would the following commit message be sufficient?

Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles the error with os.rename and retries with
shutil.move.

Thanks,

Devendra

On 29 Mar 2021, at 12:23, Konrad Weihmann <[email protected]> wrote:

Yes please quote a bit from the python manpage [1] - I certainly see the difference (and the edge cases where os.rename might fail), but that should be documented as part of the commit message

[1] https://docs.python.org/3/library/shutil.html#shutil.move

On 29.03.21 17:21, Bruce Ashfield wrote:

Can you document the cases that os.rename() is failing ? And also why

would we expect the shutil.move() to work in those cases ?

If a change like this cases issues in the future, we need that extra

information in the commit head for proper triage.

Bruce

On Mon, Mar 29, 2021 at 11:16 AM Devendra Tewari

<[email protected]> wrote:

---

meta/classes/sstate.bbclass | 26 ++++++++++++++++++++++----

1 file changed, 22 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0

index f579168162..f94aa96d70 100644

--- a/meta/classes/sstate.bbclass

+++ b/meta/classes/sstate.bbclass

meta/classes/sstate.bbclass +0 -0
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 

def sstate_installpkgdir(ss, d):

import oe.path

import subprocess

meta/classes/sstate.bbclass +11 -2
+    import shutil
 
 
   sstateinst = d.getVar("SSTATE_INSTDIR")
 
   d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,11 @@ def sstate_installpkgdir(ss, d):
 
 
   for state in ss['dirs']:
 
       prepdir(state[1])
 
-        os.rename(sstateinst + state[0], state[1])
 
+        try:
 
+            os.rename(sstateinst + state[0], state[1])
 
+            break
 
+        except OSError:
 
+            shutil.move(sstateinst + state[0], state[1])
 
   sstate_install(ss, d)
 
 
   for plain in ss['plaindirs']:
@@ -413,7 +418,11 @@ def sstate_installpkgdir(ss, d):
 
       dest = plain
 
       bb.utils.mkdirhier(src)
 
       prepdir(dest)
 
-        os.rename(src, dest)
 
+        try:
 
+            os.rename(src, dest)
 
+            break
 
+        except OSError:
 
+            shutil.move(src, dest)
 
 
   return True
@@ -638,6 +647,7 @@ python sstate_hardcode_path () {
 
 

def sstate_package(ss, d):

import oe.path

meta/classes/sstate.bbclass +1 -0
+    import shutil
 
 
   tmpdir = d.getVar('TMPDIR')
@@ -664,7 +674,11 @@ def sstate_package(ss, d):
 
                   continue
 
               bb.error("sstate found an absolute path symlink %s pointing
 at %s. Please replace this with a relative link." % (srcpath, link))
 
       bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1],

sstatebuild + state[0]))

meta/classes/sstate.bbclass +10 -3
-        os.rename(state[1], sstatebuild + state[0])
 
+        try:
 
+            os.rename(state[1], sstatebuild + state[0])
 
+            break
 
+        except OSError:
 
+            shutil.move(state[1], sstatebuild + state[0])
 
 
   workdir = d.getVar('WORKDIR')
 
   sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +688,11 @@ def sstate_package(ss, d):
 
           pdir = plain.replace(sharedworkdir, sstatebuild)
 
       bb.utils.mkdirhier(plain)
 
       bb.utils.mkdirhier(pdir)
 
-        os.rename(plain, pdir)
 
+        try:
 
+            os.rename(plain, pdir)
 
+            break
 
+        except OSError:
 
+            shutil.move(plain, pdir)
 
 
   d.setVar('SSTATE_BUILDDIR', sstatebuild)
 
   d.setVar('SSTATE_INSTDIR', sstatebuild)
 
--

2.29.2

#17 lore raw
In reply to Devendra Tewari
Quoted 7 lines · Devendra Tewari

I understand that parallel tasks can result in such issues, but would expect some kind of dependency graph that would prevent them from happening. I wish I had more time to understand the issue fully.

Here’s the last version of my patch that still uses os.rename, and on error uses shutil.move, with the reasoning explained in the commit message. There are tons of os.rename in code elsewhere, and I’ve encountered similar issues when doing incremental builds, but I’ll leave fixing them to those with more time at hand. Cheers.

Here’s a patch that catches error with all os.rename in the code, and retries with shutil.move when that happens.

Quoted 115 lines

From aeba1f9728f69cdf2ce4ba285be86761d8024f9d Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Tue, 30 Mar 2021 08:27:40 -0300
Subject: [PATCH] Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles error with os.rename and retries with shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code.
---
meta/classes/sstate.bbclass | 22 ++++++++++++++++++----

1 file changed, 18 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0
index f579168162..301dfc27db 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):

def sstate_installpkgdir(ss, d):
import oe.path
import subprocess

meta/classes/sstate.bbclass +9 -2
+    import shutil
 
    sstateinst = d.getVar("SSTATE_INSTDIR")
    d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,10 @@ def sstate_installpkgdir(ss, d):
 
    for state in ss['dirs']:
        prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
    sstate_install(ss, d)
 
    for plain in ss['plaindirs']:
@@ -413,7 +417,10 @@ def sstate_installpkgdir(ss, d):
        dest = plain
        bb.utils.mkdirhier(src)
        prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+        except OSError:
+            shutil.move(src, dest)
 
    return True
 
@@ -638,6 +645,7 @@ python sstate_hardcode_path () {
 

def sstate_package(ss, d):
import oe.path

meta/classes/sstate.bbclass +9 -2
+    import shutil
 
    tmpdir = d.getVar('TMPDIR')
@@ -664,7 +672,10 @@ def sstate_package(ss, d):
                    continue
                bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
        bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
    workdir = d.getVar('WORKDIR')
    sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +685,10 @@ def sstate_package(ss, d):
            pdir = plain.replace(sharedworkdir, sstatebuild)
        bb.utils.mkdirhier(plain)
        bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+        except OSError:
+            shutil.move(plain, pdir)
 
    d.setVar('SSTATE_BUILDDIR', sstatebuild)
    d.setVar('SSTATE_INSTDIR', sstatebuild)
Signature · 1 line
2.29.2
Quoted 14 lines · Richard Purdie
Quoted 10 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

Quoted 9 lines

This is a speed sensitive section of code and shutil has traditionally proved to be slow so I disagree with that, rename is very much preferred here where we can.

Cheers,

Richard

From d3472288e80f036dd2f8a8fc72bdeaecd9f386d8 Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Mon, 19 Apr 2021 11:23:58 -0300
Subject: [PATCH] Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles error with os.rename and retries with shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code.

[Yocto #14301]

---
 bitbake/lib/bb/siggen.py                    |  6 +++++-
 bitbake/lib/bb/tests/fetch.py               | 13 ++++++++++--
 bitbake/lib/bb/utils.py                     | 16 ++++++++++++---
 meta/classes/buildhistory.bbclass           |  6 +++++-
 meta/classes/package.bbclass                | 12 +++++++++--
 meta/classes/populate_sdk_ext.bbclass       | 10 ++++++++--
 meta/classes/reproducible_build.bbclass     |  6 +++++-
 meta/classes/sstate.bbclass                 | 22 +++++++++++++++++----
 meta/classes/update-alternatives.bbclass    | 11 +++++++++--
 meta/lib/oe/package_manager/deb/__init__.py | 18 ++++++++++++++---
 meta/lib/oe/package_manager/ipk/__init__.py |  5 ++++-
 meta/lib/oe/rootfs.py                       | 15 +++++++++++---
 meta/lib/oeqa/selftest/cases/wic.py         | 16 ++++++++++++---
 scripts/combo-layer                         |  5 ++++-
 scripts/lib/devtool/standard.py             | 21 ++++++++++++++++----
 scripts/lib/devtool/upgrade.py              |  7 ++++++-
 scripts/lib/wic/plugins/imager/direct.py    |  5 ++++-
 17 files changed, 159 insertions(+), 35 deletions(-)
bitbake/lib/bb/siggen.py +5 -1
index 0d88c6ec68..cf89e21439 100644
@@ -341,6 +341,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
         self.unihash_cache.save(self.unitaskhashes)
 
     def dump_sigtask(self, fn, task, stampbase, runtime):
+        import shutil
 
         tid = fn + ":" + task
         referencestamp = stampbase
@@ -402,7 +403,10 @@ class SignatureGeneratorBasic(SignatureGenerator):
                 p = pickle.dump(data, stream, -1)
                 stream.flush()
             os.chmod(tmpfile, 0o664)
-            os.rename(tmpfile, sigfile)
+            try:
+                os.rename(tmpfile, sigfile)
+            except OSError:
+                shutil.move(tmpfile, sigfile)
         except (OSError, IOError) as err:
             try:
                 os.unlink(tmpfile)
bitbake/lib/bb/tests/fetch.py +11 -2
index ddf6e97439..8cf2eb0ac3 100644
@@ -1770,6 +1770,7 @@ class GitShallowTest(FetcherTest):
         self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
 
     def test_shallow_submodule_mirrors(self):
+        import shutil
         self.add_empty_file('a')
         self.add_empty_file('b')
 
@@ -1794,7 +1795,10 @@ class GitShallowTest(FetcherTest):
 
         # Set up the mirror
         mirrordir = os.path.join(self.tempdir, 'mirror')
-        os.rename(self.dldir, mirrordir)
+        try:
+            os.rename(self.dldir, mirrordir)
+        except OSError:
+            shutil.move(self.dldir, mirrordir)
         self.d.setVar('PREMIRRORS', 'gitsm://.*/.* file://%s/\n' % mirrordir)
 
         # Fetch from the mirror
@@ -1899,6 +1903,7 @@ class GitShallowTest(FetcherTest):
         assert not os.path.exists(os.path.join(self.gitdir, '.git', 'shallow'))
 
     def test_shallow_mirrors(self):
+        import shutil
         self.add_empty_file('a')
         self.add_empty_file('b')
 
@@ -1912,7 +1917,11 @@ class GitShallowTest(FetcherTest):
         bb.utils.mkdirhier(mirrordir)
         self.d.setVar('PREMIRRORS', 'git://.*/.* file://%s/\n' % mirrordir)
 
-        os.rename(os.path.join(self.dldir, mirrortarball),
+        try:
+            os.rename(os.path.join(self.dldir, mirrortarball),
+                  os.path.join(mirrordir, mirrortarball))
+        except OSError:
+            shutil.move(os.path.join(self.dldir, mirrortarball),
                   os.path.join(mirrordir, mirrortarball))
 
         # Fetch from the mirror
bitbake/lib/bb/utils.py +13 -3
index b282d09abf..f289f5664e 100644
@@ -736,6 +736,7 @@ def movefile(src, dest, newmtime = None, sstat = None):
     filesystems.  Returns true on success and false on failure. Move is
     atomic.
     """
+    import shutil
 
     #print "movefile(" + src + "," + dest + "," + str(newmtime) + "," + str(sstat) + ")"
     try:
@@ -782,7 +783,10 @@ def movefile(src, dest, newmtime = None, sstat = None):
 
     if sstat[stat.ST_DEV] == dstat[stat.ST_DEV]:
         try:
-            os.rename(src, destpath)
+            try:
+                os.rename(src, destpath)
+            except OSError:
+                shutil.move(src, destpath)
             renamefailed = 0
         except Exception as e:
             if e.errno != errno.EXDEV:
@@ -796,7 +800,10 @@ def movefile(src, dest, newmtime = None, sstat = None):
         if stat.S_ISREG(sstat[stat.ST_MODE]):
             try: # For safety copy then move it over.
                 shutil.copyfile(src, destpath + "#new")
-                os.rename(destpath + "#new", destpath)
+                try:
+                    os.rename(destpath + "#new", destpath)
+                except OSError:
+                    shutil.move(destpath + "#new", destpath)
                 didcopy = 1
             except Exception as e:
                 print('movefile: copy', src, '->', dest, 'failed.', e)
@@ -874,7 +881,10 @@ def copyfile(src, dest, newmtime = None, sstat = None):
 
             # For safety copy then move it over.
             shutil.copyfile(src, dest + "#new")
-            os.rename(dest + "#new", dest)
+            try:
+                os.rename(dest + "#new", dest)
+            except OSError:
+                shutil.move(dest + "#new", dest)
         except Exception as e:
             logger.warning("copyfile: copy %s to %s failed (%s)" % (src, dest, e))
             return False
meta/classes/buildhistory.bbclass +5 -1
index 117a44eaf3..50279891a3 100644
@@ -872,7 +872,11 @@ python buildhistory_eventhandler() {
                 entries = [ x for x in os.listdir(rootdir) if not x.startswith('.') ]
                 bb.utils.mkdirhier(olddir)
                 for entry in entries:
-                    os.rename(os.path.join(rootdir, entry),
+                    try:
+                        os.rename(os.path.join(rootdir, entry),
+                              os.path.join(olddir, entry))
+                    except OSError:
+                        shutil.move(os.path.join(rootdir, entry),
                               os.path.join(olddir, entry))
         elif isinstance(e, bb.event.BuildCompleted):
             if reset:
meta/classes/package.bbclass +10 -2
index e3f0a7060b..6707c2e9db 100644
@@ -835,6 +835,7 @@ perform_packagecopy[dirs] = "${PKGD}"
 # the fs-perms.txt files
 python fixup_perms () {
     import pwd, grp
+    import shutil
 
     # init using a string with the same format as a line as documented in
     # the fs-perms.txt file
@@ -1049,7 +1050,10 @@ python fixup_perms () {
         # Create path to move directory to, move it, and then setup the symlink
         bb.utils.mkdirhier(os.path.dirname(target))
         #bb.note("Fixup Perms: Rename %s -> %s" % (dir, ptarget))
-        os.rename(origin, target)
+        try:
+            os.rename(origin, target)
+        except OSError:
+            shutil.move(origin, target)
         #bb.note("Fixup Perms: Link %s -> %s" % (dir, link))
         os.symlink(link, origin)
 
@@ -1776,6 +1780,7 @@ python package_do_shlibs() {
     import itertools
     import re, pipes
     import subprocess
+    import shutil
 
     exclude_shlibs = d.getVar('EXCLUDE_FROM_SHLIBS', False)
     if exclude_shlibs:
@@ -1967,7 +1972,10 @@ python package_do_shlibs() {
 
         for (old, new) in renames:
             bb.note("Renaming %s to %s" % (old, new))
-            os.rename(old, new)
+            try:
+                os.rename(old, new)
+            except OSError:
+                shutil.move(old, new)
             pkgfiles[pkg].remove(old)
 
         shlibs_file = os.path.join(shlibswork_dir, pkg + ".list")
meta/classes/populate_sdk_ext.bbclass +8 -2
index 9112ab6c5e..d2a81e7bf3 100644
@@ -165,7 +165,10 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
             shutil.rmtree(temp_sdkbasepath)
         except FileNotFoundError:
             pass
-        os.rename(sdkbasepath, temp_sdkbasepath)
+        try:
+            os.rename(sdkbasepath, temp_sdkbasepath)
+        except OSError:
+            shutil.move(sdkbasepath, temp_sdkbasepath)
         cmdprefix = '. %s .; ' % conf_initpath
         logfile = d.getVar('WORKDIR') + '/tasklist_bb_log.txt'
         try:
@@ -175,7 +178,10 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
             if 'attempted to execute unexpectedly and should have been setscened' in e.stdout:
                 msg += '\n----------\n\nNOTE: "attempted to execute unexpectedly and should have been setscened" errors indicate this may be caused by missing sstate artifacts that were likely produced in earlier builds, but have been subsequently deleted for some reason.\n'
             bb.fatal(msg)
-        os.rename(temp_sdkbasepath, sdkbasepath)
+        try:
+            os.rename(temp_sdkbasepath, sdkbasepath)
+        except OSError:
+            shutil.move(temp_sdkbasepath, sdkbasepath)
         # Clean out residue of running bitbake, which check_sstate_task_list()
         # will effectively do
         clean_esdk_builddir(d, sdkbasepath)
meta/classes/reproducible_build.bbclass +5 -1
index f06e00d70d..480d58f86d 100644
@@ -57,13 +57,17 @@ do_deploy_source_date_epoch () {
 }
 
 python do_deploy_source_date_epoch_setscene () {
+    import shutil
     sstate_setscene(d)
     bb.utils.mkdirhier(d.getVar('SDE_DIR'))
     sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
     if os.path.exists(sde_file):
         target = d.getVar('SDE_FILE')
         bb.debug(1, "Moving setscene SDE file %s -> %s" % (sde_file, target))
-        os.rename(sde_file, target)
+        try:
+            os.rename(sde_file, target)
+        except OSError:
+            shutil.move(sde_file, target)
     else:
         bb.debug(1, "%s not found!" % sde_file)
 }
meta/classes/sstate.bbclass +18 -4
index 8e8efd18d5..60f7a94285 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
+    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,10 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -413,7 +417,10 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+        except OSError:
+            shutil.move(src, dest)
 
     return True
 
@@ -638,6 +645,7 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
+    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -664,7 +672,10 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +685,10 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+        except OSError:
+            shutil.move(plain, pdir)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
meta/classes/update-alternatives.bbclass +9 -2
index 8c2b66e7f1..7e213472e3 100644
@@ -139,6 +139,7 @@ python apply_update_alternative_renames () {
        return
 
     import re
+    import shutil
 
     def update_files(alt_target, alt_target_rename, pkg, d):
         f = d.getVar('FILES_' + pkg)
@@ -184,7 +185,10 @@ python apply_update_alternative_renames () {
                         link_rename.append((alt_target, alt_target_rename))
                     else:
                         bb.note('%s: Rename %s -> %s' % (pn, alt_target, alt_target_rename))
-                        os.rename(src, dest)
+                        try:
+                            os.rename(src, dest)
+                        except OSError:
+                            shutil.move(src, dest)
                         update_files(alt_target, alt_target_rename, pkg, d)
                 else:
                     bb.warn("%s: alternative target (%s or %s) does not exist, skipping..." % (pn, alt_target, alt_target_rename))
@@ -201,7 +205,10 @@ python apply_update_alternative_renames () {
             if os.path.lexists(link_target):
                 # Ok, the link_target exists, we can rename
                 bb.note('%s: Rename (link) %s -> %s' % (pn, alt_target, alt_target_rename))
-                os.rename(src, dest)
+                try:
+                    os.rename(src, dest)
+                except OSError:
+                    shutil.move(src, dest)
             else:
                 # Try to resolve the broken link to link.${BPN}
                 link_maybe = '%s.%s' % (os.readlink(src), pn)
meta/lib/oe/package_manager/deb/__init__.py +15 -3
index 2ee68fefb1..8edc653dff 100644
@@ -5,6 +5,7 @@
 import re
 import subprocess
 from oe.package_manager import *
+import shutil
 
 class DpkgIndexer(Indexer):
     def _create_configs(self):
@@ -214,7 +215,10 @@ class DpkgPM(OpkgDpkgPM):
 
                     tmp_sf.write(status)
 
-        os.rename(status_file + ".tmp", status_file)
+        try:
+            os.rename(status_file + ".tmp", status_file)
+        except OSError:
+            shutil.move(status_file + ".tmp", status_file)
 
     def run_pre_post_installs(self, package_name=None):
         """
@@ -299,13 +303,21 @@ class DpkgPM(OpkgDpkgPM):
             for dir in dirs:
                 new_dir = re.sub(r"\.dpkg-new", "", dir)
                 if dir != new_dir:
-                    os.rename(os.path.join(root, dir),
+                    try:
+                        os.rename(os.path.join(root, dir),
+                              os.path.join(root, new_dir))
+                    except OSError:
+                        shutil.move(os.path.join(root, dir),
                               os.path.join(root, new_dir))
 
             for file in files:
                 new_file = re.sub(r"\.dpkg-new", "", file)
                 if file != new_file:
-                    os.rename(os.path.join(root, file),
+                    try:
+                        os.rename(os.path.join(root, file),
+                              os.path.join(root, new_file))
+                    except OSError:
+                        shutil.move(os.path.join(root, file),
                               os.path.join(root, new_file))
 
 
meta/lib/oe/package_manager/ipk/__init__.py +4 -1
index da488c1c7f..2aa21949f3 100644
@@ -213,7 +213,10 @@ class OpkgPM(OpkgDpkgPM):
 
                     tmp_sf.write(status)
 
-        os.rename(status_file + ".tmp", status_file)
+        try:
+            os.rename(status_file + ".tmp", status_file)
+        except OSError:
+            shutil.move(status_file + ".tmp", status_file)
 
     def _create_custom_config(self):
         bb.note("Building from feeds activated!")
meta/lib/oe/rootfs.py +12 -3
index 249c685dcf..8070fbbf9f 100644
@@ -114,7 +114,10 @@ class Rootfs(object, metaclass=ABCMeta):
             shutil.rmtree(self.image_rootfs + '-orig')
         except:
             pass
-        os.rename(self.image_rootfs, self.image_rootfs + '-orig')
+        try:
+            os.rename(self.image_rootfs, self.image_rootfs + '-orig')
+        except OSError:
+            shutil.move(self.image_rootfs, self.image_rootfs + '-orig')
 
         bb.note("  Creating debug rootfs...")
         bb.utils.mkdirhier(self.image_rootfs)
@@ -165,10 +168,16 @@ class Rootfs(object, metaclass=ABCMeta):
             shutil.rmtree(self.image_rootfs + '-dbg')
         except:
             pass
-        os.rename(self.image_rootfs, self.image_rootfs + '-dbg')
+        try:
+            os.rename(self.image_rootfs, self.image_rootfs + '-dbg')
+        except OSError:
+            shutil.move(self.image_rootfs, self.image_rootfs + '-dbg')
 
         bb.note("  Restoreing original rootfs...")
-        os.rename(self.image_rootfs + '-orig', self.image_rootfs)
+        try:
+            os.rename(self.image_rootfs + '-orig', self.image_rootfs)
+        except OSError:
+            shutil.move(self.image_rootfs + '-orig', self.image_rootfs)
 
     def _exec_shell_cmd(self, cmd):
         fakerootcmd = self.d.getVar('FAKEROOT')
meta/lib/oeqa/selftest/cases/wic.py +13 -3
index fa81584a8c..fa93973087 100644
@@ -1269,6 +1269,7 @@ class Wic2(WicTestCase):
 
     def test_expand_mbr_image(self):
         """Test wic write --expand command for mbr image"""
+        import shutil
         # build an image
         config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "directdisk.wks"\n'
         self.append_config(config)
@@ -1306,8 +1307,14 @@ class Wic2(WicTestCase):
             result = runCmd("%s/usr/sbin/sfdisk -F %s" % (sysroot, new_image_path))
             self.assertTrue("0 B, 0 bytes, 0 sectors" in result.output)
 
-            os.rename(image_path, image_path + '.bak')
-            os.rename(new_image_path, image_path)
+            try:
+                os.rename(image_path, image_path + '.bak')
+            except OSError:
+                shutil.move(image_path, image_path + '.bak')
+            try:
+                os.rename(new_image_path, image_path)
+            except OSError:
+                shutil.move(new_image_path, image_path)
 
             # Check if it boots in qemu
             with runqemu('core-image-minimal', ssh=False) as qemu:
@@ -1318,7 +1325,10 @@ class Wic2(WicTestCase):
             if os.path.exists(new_image_path):
                 os.unlink(new_image_path)
             if os.path.exists(image_path + '.bak'):
-                os.rename(image_path + '.bak', image_path)
+                try:
+                    os.rename(image_path + '.bak', image_path)
+                except OSError:
+                    shutil.move(image_path + '.bak', image_path)
 
     def test_wic_ls_ext(self):
         """Test listing content of the ext partition using 'wic ls'"""
scripts/combo-layer +4 -1
index a634dd69d2..1b88340b6e 100755
@@ -508,7 +508,10 @@ def check_patch(patchfile):
     f.close()
     if of:
         of.close()
-        os.rename(patchfile + '.tmp', patchfile)
+        try:
+            os.rename(patchfile + '.tmp', patchfile)
+        except OSError:
+            shutil.move(patchfile + '.tmp', patchfile)
 
 def drop_to_shell(workdir=None):
     if not sys.stdin.isatty():
scripts/lib/devtool/standard.py +17 -4
index f364a45283..3854dfb3e8 100644
@@ -746,7 +746,10 @@ def _check_preserve(config, recipename):
                         os.remove(removefile)
                 else:
                     tf.write(line)
-    os.rename(newfile, origfile)
+    try:
+        os.rename(newfile, origfile)
+    except OSError:
+        shutil.move(newfile, origfile)
 
 def get_staging_kver(srcdir):
     # Kernel version from work-shared
@@ -1094,10 +1097,16 @@ def rename(args, config, basepath, workspace):
 
     # Rename bbappend
     logger.info('Renaming %s to %s' % (append, newappend))
-    os.rename(append, newappend)
+    try:
+        os.rename(append, newappend)
+    except OSError:
+        shutil.move(append, newappend)
     # Rename recipe file
     logger.info('Renaming %s to %s' % (recipefile, newfile))
-    os.rename(recipefile, newfile)
+    try:
+        os.rename(recipefile, newfile)
+    except OSError:
+        shutil.move(recipefile, newfile)
 
     # Rename source tree if it's the default path
     appendmd5 = None
@@ -1333,7 +1342,11 @@ def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None):
         if match_name:
             # Rename patch files
             if new_patch != match_name:
-                os.rename(os.path.join(destdir, new_patch),
+                try:
+                    os.rename(os.path.join(destdir, new_patch),
+                          os.path.join(destdir, match_name))
+                except OSError:
+                    shutil.move(os.path.join(destdir, new_patch),
                           os.path.join(destdir, match_name))
             # Need to pop it off the list now before checking changed_revs
             oldpath = existing_patches.pop(old_patch)
scripts/lib/devtool/upgrade.py +6 -1
index 5a057e95f5..2bff69f5a2 100644
@@ -71,7 +71,12 @@ def _rename_recipe_dirs(oldpv, newpv, path):
                 if oldfile.find(oldpv) != -1:
                     newfile = oldfile.replace(oldpv, newpv)
                     if oldfile != newfile:
-                        os.rename(os.path.join(path, oldfile), os.path.join(path, newfile))
+                        try:
+                            os.rename(os.path.join(path, oldfile),
+                                  os.path.join(path, newfile))
+                        except OSError:
+                            shutil.move(os.path.join(path, oldfile),
+                                  os.path.join(path, newfile))
 
 def _rename_recipe_file(oldrecipe, bpn, oldpv, newpv, path):
     oldrecipe = os.path.basename(oldrecipe)
scripts/lib/wic/plugins/imager/direct.py +4 -1
index ea709e8c54..fb5e52b2a6 100644
@@ -616,5 +616,8 @@ class PartitionedImage():
                              part.start + part.size_sec - 1, part.size_sec)
 
                 partimage = self.path + '.p%d' % part.num
-                os.rename(source, partimage)
+                try:
+                    os.rename(source, partimage)
+                except OSError:
+                    shutil.move(source, partimage)
                 self.partimages.append(partimage)
Signature · 1 line
2.29.2
#18 lore raw

None of my images are bootable when this patch is applied, because symlinks that busybox should provide arent created in the image. This patch should be looked into a bit further, how was it tested ?

Quoted 131 lines · Devendra Tewari
Quoted 8 lines · Devendra Tewari

I understand that parallel tasks can result in such issues, but would expect some kind of dependency graph that would prevent them from happening. I wish I had more time to understand the issue fully.

Here’s the last version of my patch that still uses os.rename, and on error uses shutil.move, with the reasoning explained in the commit message. There are tons of os.rename in code elsewhere, and I’ve encountered similar issues when doing incremental builds, but I’ll leave fixing them to those with more time at hand. Cheers.

Here’s a patch that catches error with all os.rename in the code, and retries with shutil.move when that happens.

Quoted 115 lines

From aeba1f9728f69cdf2ce4ba285be86761d8024f9d Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Tue, 30 Mar 2021 08:27:40 -0300
Subject: [PATCH] Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles error with os.rename and retries with shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code.
---
meta/classes/sstate.bbclass | 22 ++++++++++++++++++----

1 file changed, 18 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0
index f579168162..301dfc27db 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):

def sstate_installpkgdir(ss, d):
import oe.path
import subprocess

meta/classes/sstate.bbclass +9 -2
+    import shutil
 
    sstateinst = d.getVar("SSTATE_INSTDIR")
    d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,10 @@ def sstate_installpkgdir(ss, d):
 
    for state in ss['dirs']:
        prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
    sstate_install(ss, d)
 
    for plain in ss['plaindirs']:
@@ -413,7 +417,10 @@ def sstate_installpkgdir(ss, d):
        dest = plain
        bb.utils.mkdirhier(src)
        prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+        except OSError:
+            shutil.move(src, dest)
 
    return True
 
@@ -638,6 +645,7 @@ python sstate_hardcode_path () {
 

def sstate_package(ss, d):
import oe.path

meta/classes/sstate.bbclass +9 -3
+    import shutil
 
    tmpdir = d.getVar('TMPDIR')
@@ -664,7 +672,10 @@ def sstate_package(ss, d):
                    continue
                bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
        bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
    workdir = d.getVar('WORKDIR')
    sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +685,10 @@ def sstate_package(ss, d):
            pdir = plain.replace(sharedworkdir, sstatebuild)
        bb.utils.mkdirhier(plain)
        bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+        except OSError:
+            shutil.move(plain, pdir)
 
    d.setVar('SSTATE_BUILDDIR', sstatebuild)
    d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

Quoted 23 lines · Richard Purdie
Quoted 10 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

This is a speed sensitive section of code and shutil has traditionally proved to be slow so I disagree with that, rename is very much preferred here where we can.

Cheers,

Richard

#19 lore raw

I created an image for Raspberry Pi Zero W from code at https://github.com/tewarid/docker-meta-raspberrypi/tree/sstate <https://github.com/tewarid/docker-meta-raspberrypi/tree/sstate>, using Docker for macOS, and on Ubuntu 20.04-LTS, and wrote it out to an SDCard using bmaptool. Booted and checked that busybox symlinks in rootfs look all right.

Quoted 139 lines

Em 21 de abr. de 2021, à(s) 02:22, Khem Raj <[email protected]> escreveu:

None of my images are bootable when this patch is applied, because symlinks that busybox should provide arent created in the image. This patch should be looked into a bit further, how was it tested ?

Quoted 133 lines
Quoted 5 lines · Devendra Tewari
Quoted 1 line · Devendra Tewari

I understand that parallel tasks can result in such issues, but would expect some kind of dependency graph that would prevent them from happening. I wish I had more time to understand the issue fully.

Here’s the last version of my patch that still uses os.rename, and on error uses shutil.move, with the reasoning explained in the commit message. There are tons of os.rename in code elsewhere, and I’ve encountered similar issues when doing incremental builds, but I’ll leave fixing them to those with more time at hand. Cheers.

Here’s a patch that catches error with all os.rename in the code, and retries with shutil.move when that happens.

Quoted 115 lines

From aeba1f9728f69cdf2ce4ba285be86761d8024f9d Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Tue, 30 Mar 2021 08:27:40 -0300
Subject: [PATCH] Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles error with os.rename and retries with shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code.
---
meta/classes/sstate.bbclass | 22 ++++++++++++++++++----

1 file changed, 18 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0
index f579168162..301dfc27db 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):

def sstate_installpkgdir(ss, d):
import oe.path
import subprocess

meta/classes/sstate.bbclass +9 -2
+    import shutil
 
  sstateinst = d.getVar("SSTATE_INSTDIR")
  d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,10 @@ def sstate_installpkgdir(ss, d):
 
  for state in ss['dirs']:
      prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
  sstate_install(ss, d)
 
  for plain in ss['plaindirs']:
@@ -413,7 +417,10 @@ def sstate_installpkgdir(ss, d):
      dest = plain
      bb.utils.mkdirhier(src)
      prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+        except OSError:
+            shutil.move(src, dest)
 
  return True
 
@@ -638,6 +645,7 @@ python sstate_hardcode_path () {
 

def sstate_package(ss, d):
import oe.path

meta/classes/sstate.bbclass +9 -3
+    import shutil
 
  tmpdir = d.getVar('TMPDIR')
@@ -664,7 +672,10 @@ def sstate_package(ss, d):
                  continue
              bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
      bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
  workdir = d.getVar('WORKDIR')
  sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +685,10 @@ def sstate_package(ss, d):
          pdir = plain.replace(sharedworkdir, sstatebuild)
      bb.utils.mkdirhier(plain)
      bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+        except OSError:
+            shutil.move(plain, pdir)
 
  d.setVar('SSTATE_BUILDDIR', sstatebuild)
  d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

Quoted 23 lines · Richard Purdie
Quoted 10 lines · Andre McCurdy
Quoted 4 lines · Devendra Tewari

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

This is a speed sensitive section of code and shutil has traditionally proved to be slow so I disagree with that, rename is very much preferred here where we can.

Cheers,

Richard

#20 lore raw
Quoted 3 lines · Devendra Tewari

I created an image for Raspberry Pi Zero W from code at https://github.com/tewarid/docker-meta-raspberrypi/tree/sstate, using Docker for macOS, and on Ubuntu 20.04-LTS, and wrote it out to an SDCard using bmaptool. Booted and checked that busybox symlinks in rootfs look all right.

I think its not this particular patch per se. But similar patch which is in oe-core/master-next which is causing it
https://git.openembedded.org/openembedded-core/commit/?h=master-next&id=11657970ff8ff89d1a9760b1c75974ba5d149ddd

Quoted 145 lines

Em 21 de abr. de 2021, à(s) 02:22, Khem Raj <[email protected]> escreveu:

None of my images are bootable when this patch is applied, because symlinks that busybox should provide arent created in the image. This patch should be looked into a bit further, how was it tested ?

On Mon, Apr 19, 2021 at 11:43 AM Devendra Tewari
<[email protected]> wrote:

On 30 Mar 2021, at 08:55, Devendra Tewari <[email protected]> wrote:

I understand that parallel tasks can result in such issues, but would expect some kind of dependency graph that would prevent them from happening. I wish I had more time to understand the issue fully.

Here’s the last version of my patch that still uses os.rename, and on error uses shutil.move, with the reasoning explained in the commit message. There are tons of os.rename in code elsewhere, and I’ve encountered similar issues when doing incremental builds, but I’ll leave fixing them to those with more time at hand. Cheers.

Here’s a patch that catches error with all os.rename in the code, and retries with shutil.move when that happens.

From aeba1f9728f69cdf2ce4ba285be86761d8024f9d Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Tue, 30 Mar 2021 08:27:40 -0300
Subject: [PATCH] Use shutil.move when os.rename fails

Incremental build in Docker fails with

OSError: [Errno 18] Invalid cross-device link

When source and destination are on different overlay filesystems.

This change handles error with os.rename and retries with shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code.
---
meta/classes/sstate.bbclass | 22 ++++++++++++++++++----

1 file changed, 18 insertions(+), 4 deletions(-)
meta/classes/sstate.bbclass +0 -0
index f579168162..301dfc27db 100644
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):

def sstate_installpkgdir(ss, d):
import oe.path
import subprocess

meta/classes/sstate.bbclass +9 -2
+    import shutil
 
  sstateinst = d.getVar("SSTATE_INSTDIR")
  d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,10 @@ def sstate_installpkgdir(ss, d):
 
  for state in ss['dirs']:
      prepdir(state[1])
-        os.rename(sstateinst + state[0], state[1])
+        try:
+            os.rename(sstateinst + state[0], state[1])
+        except OSError:
+            shutil.move(sstateinst + state[0], state[1])
  sstate_install(ss, d)
 
  for plain in ss['plaindirs']:
@@ -413,7 +417,10 @@ def sstate_installpkgdir(ss, d):
      dest = plain
      bb.utils.mkdirhier(src)
      prepdir(dest)
-        os.rename(src, dest)
+        try:
+            os.rename(src, dest)
+        except OSError:
+            shutil.move(src, dest)
 
  return True
 
@@ -638,6 +645,7 @@ python sstate_hardcode_path () {
 

def sstate_package(ss, d):
import oe.path

meta/classes/sstate.bbclass +9 -3
+    import shutil
 
  tmpdir = d.getVar('TMPDIR')
@@ -664,7 +672,10 @@ def sstate_package(ss, d):
                  continue
              bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
      bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        os.rename(state[1], sstatebuild + state[0])
+        try:
+            os.rename(state[1], sstatebuild + state[0])
+        except OSError:
+            shutil.move(state[1], sstatebuild + state[0])
 
  workdir = d.getVar('WORKDIR')
  sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +685,10 @@ def sstate_package(ss, d):
          pdir = plain.replace(sharedworkdir, sstatebuild)
      bb.utils.mkdirhier(plain)
      bb.utils.mkdirhier(pdir)
-        os.rename(plain, pdir)
+        try:
+            os.rename(plain, pdir)
+        except OSError:
+            shutil.move(plain, pdir)
 
  d.setVar('SSTATE_BUILDDIR', sstatebuild)
  d.setVar('SSTATE_INSTDIR', sstatebuild)
--

2.29.2

On 30 Mar 2021, at 08:10, Richard Purdie <[email protected]> wrote:

On Mon, 2021-03-29 at 16:00 -0700, Andre McCurdy wrote:

On Mon, Mar 29, 2021 at 3:45 PM Devendra Tewari
<[email protected]> wrote:

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example.

Here’s the updated patch (or should I submit it again via git send-email?)

It would be better to use shutil.move unconditionally in all cases, rather than have a separate shutil.move code path which only gets tested by people doing incremental builds in docker.

This is a speed sensitive section of code and shutil has traditionally proved to be slow so I disagree with that, rename is very much preferred here where we can.

Cheers,

Richard

#21 lore raw
In reply to Devendra Tewari
Quoted 10 lines · Devendra Tewari

I created an image for Raspberry Pi Zero W from code at
https://github.com/tewarid/docker-meta-raspberrypi/tree/sstate, using Docker for macOS, and on Ubuntu 20.04- LTS, and wrote it out to an SDCard using bmaptool. Booted and checked that busybox symlinks in rootfs look all right.

Quoted 5 lines

Em 21 de abr. de 2021, à(s) 02:22, Khem Raj <[email protected]> escreveu:

None of my images are bootable when this patch is applied, because symlinks that busybox should provide arent created in the image. This patch should be looked into a bit further,  how was it tested ?

I think there is some element to this we're not understanding. I'm also worried that the exception is quite generic, OSError covers a lot of things and we probably need to make it more specific.

I'd suggest we add a patch changing all the call sites to use a new wrapper function in bb.utils and then we can adjust that single site to handle issues as they arise and as we understand them.

Cheers,

Richard

#22 lore raw

That's a neat idea - if the exception handler catches OSError with error number 18 (Invalid cross-device link) in os.rename, it should attempt shutil.move, otherwise it should re-throw the exception with raise. A wrapper function will make it easier to handle other situations as they arise. I'll update the patch.

Thanks,
Devendra

Quoted 26 lines

Em 21 de abr. de 2021, à(s) 15:10, Richard Purdie <[email protected]> escreveu:

Quoted 10 lines · Devendra Tewari

I created an image for Raspberry Pi Zero W from code at
https://github.com/tewarid/docker-meta-raspberrypi/tree/sstate, using Docker for macOS, and on Ubuntu 20.04- LTS, and wrote it out to an SDCard using bmaptool. Booted and checked that busybox symlinks in rootfs look all right.

Quoted 5 lines

Em 21 de abr. de 2021, à(s) 02:22, Khem Raj <[email protected]> escreveu:

None of my images are bootable when this patch is applied, because symlinks that busybox should provide arent created in the image. This patch should be looked into a bit further, how was it tested ?

I think there is some element to this we're not understanding. I'm also worried that the exception is quite generic, OSError covers a lot of things and we probably need to make it more specific.

I'd suggest we add a patch changing all the call sites to use a new wrapper function in bb.utils and then we can adjust that single site to handle issues as they arise and as we understand them.

Cheers,

Richard

#23 lore raw
Quoted 4 lines · Devendra Tewari

That's a neat idea - if the exception handler catches OSError with error number 18 (Invalid cross-device link) in os.rename, it should attempt shutil.move, otherwise it should re-throw the exception with raise. A wrapper function will make it easier to handle other situations as they arise. I'll update the patch.

Sounds good. Could you also please split the patch into two, one for bitbake and one for OE-Core since they are two different repositories.

Cheers,

Richard

#24 lore raw

Separate patches for bitbake and oe-core, as requested

I've assumed each will be applied to the master-next branch on top of the previous patch.

Thanks,
Devendra

Quoted 8 lines · Richard Purdie
Quoted 4 lines · Devendra Tewari

That's a neat idea - if the exception handler catches OSError with error number 18 (Invalid cross-device link) in os.rename, it should attempt shutil.move, otherwise it should re-throw the exception with raise. A wrapper function will make it easier to handle other situations as they arise. I'll update the patch.

Quoted 7 lines

Sounds good. Could you also please split the patch into two, one for bitbake and one for OE-Core since they are two different repositories.

Cheers,

Richard

From ec0d5beef8a121b1afeb6e8ae72484ba690ab9e6 Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Wed, 21 Apr 2021 17:57:48 -0300
Subject: [PATCH] use bb.utils.rename to rename file or directory

bb.utils.rename handles cross-device link error with os.rename and retries with shutil.move.

[YOCTO #14301]

---
 lib/bb/siggen.py      |  6 +-----
 lib/bb/tests/fetch.py | 13 ++-----------
 lib/bb/utils.py       | 28 +++++++++++++++-------------
 3 files changed, 18 insertions(+), 29 deletions(-)
lib/bb/siggen.py +1 -5
index cf89e214..574beb30 100644
@@ -341,7 +341,6 @@ class SignatureGeneratorBasic(SignatureGenerator):
         self.unihash_cache.save(self.unitaskhashes)
 
     def dump_sigtask(self, fn, task, stampbase, runtime):
-        import shutil
 
         tid = fn + ":" + task
         referencestamp = stampbase
@@ -403,10 +402,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
                 p = pickle.dump(data, stream, -1)
                 stream.flush()
             os.chmod(tmpfile, 0o664)
-            try:
-                os.rename(tmpfile, sigfile)
-            except OSError:
-                shutil.move(tmpfile, sigfile)
+            bb.utils.rename(tmpfile, sigfile)
         except (OSError, IOError) as err:
             try:
                 os.unlink(tmpfile)
lib/bb/tests/fetch.py +2 -11
index 89207559..9291ce4a 100644
@@ -1772,7 +1772,6 @@ class GitShallowTest(FetcherTest):
         self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
 
     def test_shallow_submodule_mirrors(self):
-        import shutil
         self.add_empty_file('a')
         self.add_empty_file('b')
 
@@ -1799,10 +1798,7 @@ class GitShallowTest(FetcherTest):
 
         # Set up the mirror
         mirrordir = os.path.join(self.tempdir, 'mirror')
-        try:
-            os.rename(self.dldir, mirrordir)
-        except OSError:
-            shutil.move(self.dldir, mirrordir)
+        bb.utils.rename(self.dldir, mirrordir)
         self.d.setVar('PREMIRRORS', 'gitsm://.*/.* file://%s/\n' % mirrordir)
 
         # Fetch from the mirror
@@ -1907,7 +1903,6 @@ class GitShallowTest(FetcherTest):
         assert not os.path.exists(os.path.join(self.gitdir, '.git', 'shallow'))
 
     def test_shallow_mirrors(self):
-        import shutil
         self.add_empty_file('a')
         self.add_empty_file('b')
 
@@ -1921,11 +1916,7 @@ class GitShallowTest(FetcherTest):
         bb.utils.mkdirhier(mirrordir)
         self.d.setVar('PREMIRRORS', 'git://.*/.* file://%s/\n' % mirrordir)
 
-        try:
-            os.rename(os.path.join(self.dldir, mirrortarball),
-                  os.path.join(mirrordir, mirrortarball))
-        except OSError:
-            shutil.move(os.path.join(self.dldir, mirrortarball),
+        bb.utils.rename(os.path.join(self.dldir, mirrortarball),
                   os.path.join(mirrordir, mirrortarball))
 
         # Fetch from the mirror
lib/bb/utils.py +15 -13
index f289f566..0f68b0ae 100644
@@ -736,7 +736,6 @@ def movefile(src, dest, newmtime = None, sstat = None):
     filesystems.  Returns true on success and false on failure. Move is
     atomic.
     """
-    import shutil
 
     #print "movefile(" + src + "," + dest + "," + str(newmtime) + "," + str(sstat) + ")"
     try:
@@ -783,10 +782,7 @@ def movefile(src, dest, newmtime = None, sstat = None):
 
     if sstat[stat.ST_DEV] == dstat[stat.ST_DEV]:
         try:
-            try:
-                os.rename(src, destpath)
-            except OSError:
-                shutil.move(src, destpath)
+            bb.utils.rename(src, destpath)
             renamefailed = 0
         except Exception as e:
             if e.errno != errno.EXDEV:
@@ -800,10 +796,7 @@ def movefile(src, dest, newmtime = None, sstat = None):
         if stat.S_ISREG(sstat[stat.ST_MODE]):
             try: # For safety copy then move it over.
                 shutil.copyfile(src, destpath + "#new")
-                try:
-                    os.rename(destpath + "#new", destpath)
-                except OSError:
-                    shutil.move(destpath + "#new", destpath)
+                bb.utils.rename(destpath + "#new", destpath)
                 didcopy = 1
             except Exception as e:
                 print('movefile: copy', src, '->', dest, 'failed.', e)
@@ -881,10 +874,7 @@ def copyfile(src, dest, newmtime = None, sstat = None):
 
             # For safety copy then move it over.
             shutil.copyfile(src, dest + "#new")
-            try:
-                os.rename(dest + "#new", dest)
-            except OSError:
-                shutil.move(dest + "#new", dest)
+            bb.utils.rename(dest + "#new", dest)
         except Exception as e:
             logger.warning("copyfile: copy %s to %s failed (%s)" % (src, dest, e))
             return False
@@ -1679,3 +1669,15 @@ def is_semver(version):
         return False
 
     return True
+
+def rename(src, dst):
+    import os
+    import shutil
+    try:
+        os.rename(src, dst)
+    except OSError as err:
+        if err.errno == 18:
+            # Invalid cross-device link error
+            shutil.move(src, dst)
+        else:
+            raise err
Signature · 1 line
2.29.2

From e5af200e902edfbec9e2017ff6b008dbb40b6379 Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Wed, 21 Apr 2021 17:49:48 -0300
Subject: [PATCH] use bb.utils.rename to rename file or directory

bb.utils.rename handles cross-device link error with os.rename and retries with shutil.move.

[YOCTO #14301]

---
 meta/classes/buildhistory.bbclass           |  6 +-----
 meta/classes/package.bbclass                | 12 ++---------
 meta/classes/populate_sdk_ext.bbclass       | 10 ++--------
 meta/classes/reproducible_build.bbclass     |  6 +-----
 meta/classes/sstate.bbclass                 | 22 ++++-----------------
 meta/classes/update-alternatives.bbclass    | 11 ++---------
 meta/lib/oe/package_manager/deb/__init__.py | 18 +++--------------
 meta/lib/oe/package_manager/ipk/__init__.py |  5 +----
 meta/lib/oe/rootfs.py                       | 15 +++-----------
 meta/lib/oeqa/selftest/cases/wic.py         | 16 +++------------
 scripts/combo-layer                         |  5 +----
 scripts/lib/devtool/standard.py             | 21 ++++----------------
 scripts/lib/devtool/upgrade.py              |  8 ++------
 scripts/lib/wic/plugins/imager/direct.py    |  5 +----
 14 files changed, 30 insertions(+), 130 deletions(-)
meta/classes/buildhistory.bbclass +1 -5
index 554c8810ac..059de36a5d 100644
@@ -875,11 +875,7 @@ python buildhistory_eventhandler() {
                 entries = [ x for x in os.listdir(rootdir) if not x.startswith('.') ]
                 bb.utils.mkdirhier(olddir)
                 for entry in entries:
-                    try:
-                        os.rename(os.path.join(rootdir, entry),
-                              os.path.join(olddir, entry))
-                    except OSError:
-                        shutil.move(os.path.join(rootdir, entry),
+                    bb.utils.rename(os.path.join(rootdir, entry),
                               os.path.join(olddir, entry))
         elif isinstance(e, bb.event.BuildCompleted):
             if reset:
meta/classes/package.bbclass +2 -10
index 6707c2e9db..cf30f33f3d 100644
@@ -835,7 +835,6 @@ perform_packagecopy[dirs] = "${PKGD}"
 # the fs-perms.txt files
 python fixup_perms () {
     import pwd, grp
-    import shutil
 
     # init using a string with the same format as a line as documented in
     # the fs-perms.txt file
@@ -1050,10 +1049,7 @@ python fixup_perms () {
         # Create path to move directory to, move it, and then setup the symlink
         bb.utils.mkdirhier(os.path.dirname(target))
         #bb.note("Fixup Perms: Rename %s -> %s" % (dir, ptarget))
-        try:
-            os.rename(origin, target)
-        except OSError:
-            shutil.move(origin, target)
+        bb.utils.rename(origin, target)
         #bb.note("Fixup Perms: Link %s -> %s" % (dir, link))
         os.symlink(link, origin)
 
@@ -1780,7 +1776,6 @@ python package_do_shlibs() {
     import itertools
     import re, pipes
     import subprocess
-    import shutil
 
     exclude_shlibs = d.getVar('EXCLUDE_FROM_SHLIBS', False)
     if exclude_shlibs:
@@ -1972,10 +1967,7 @@ python package_do_shlibs() {
 
         for (old, new) in renames:
             bb.note("Renaming %s to %s" % (old, new))
-            try:
-                os.rename(old, new)
-            except OSError:
-                shutil.move(old, new)
+            bb.utils.rename(old, new)
             pkgfiles[pkg].remove(old)
 
         shlibs_file = os.path.join(shlibswork_dir, pkg + ".list")
meta/classes/populate_sdk_ext.bbclass +2 -8
index 12e758c2ad..fe840d9cfb 100644
@@ -165,10 +165,7 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
             shutil.rmtree(temp_sdkbasepath)
         except FileNotFoundError:
             pass
-        try:
-            os.rename(sdkbasepath, temp_sdkbasepath)
-        except OSError:
-            shutil.move(sdkbasepath, temp_sdkbasepath)
+        bb.utils.rename(sdkbasepath, temp_sdkbasepath)
         cmdprefix = '. %s .; ' % conf_initpath
         logfile = d.getVar('WORKDIR') + '/tasklist_bb_log.txt'
         try:
@@ -178,10 +175,7 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
             if 'attempted to execute unexpectedly and should have been setscened' in e.stdout:
                 msg += '\n----------\n\nNOTE: "attempted to execute unexpectedly and should have been setscened" errors indicate this may be caused by missing sstate artifacts that were likely produced in earlier builds, but have been subsequently deleted for some reason.\n'
             bb.fatal(msg)
-        try:
-            os.rename(temp_sdkbasepath, sdkbasepath)
-        except OSError:
-            shutil.move(temp_sdkbasepath, sdkbasepath)
+        bb.utils.rename(temp_sdkbasepath, sdkbasepath)
         # Clean out residue of running bitbake, which check_sstate_task_list()
         # will effectively do
         clean_esdk_builddir(d, sdkbasepath)
meta/classes/reproducible_build.bbclass +1 -5
index 480d58f86d..f2d9324440 100644
@@ -57,17 +57,13 @@ do_deploy_source_date_epoch () {
 }
 
 python do_deploy_source_date_epoch_setscene () {
-    import shutil
     sstate_setscene(d)
     bb.utils.mkdirhier(d.getVar('SDE_DIR'))
     sde_file = os.path.join(d.getVar('SDE_DEPLOYDIR'), '__source_date_epoch.txt')
     if os.path.exists(sde_file):
         target = d.getVar('SDE_FILE')
         bb.debug(1, "Moving setscene SDE file %s -> %s" % (sde_file, target))
-        try:
-            os.rename(sde_file, target)
-        except OSError:
-            shutil.move(sde_file, target)
+        bb.utils.rename(sde_file, target)
     else:
         bb.debug(1, "%s not found!" % sde_file)
 }
meta/classes/sstate.bbclass +4 -18
index 60f7a94285..b1c608dcb1 100644
@@ -384,7 +384,6 @@ def sstate_installpkg(ss, d):
 def sstate_installpkgdir(ss, d):
     import oe.path
     import subprocess
-    import shutil
 
     sstateinst = d.getVar("SSTATE_INSTDIR")
     d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -402,10 +401,7 @@ def sstate_installpkgdir(ss, d):
 
     for state in ss['dirs']:
         prepdir(state[1])
-        try:
-            os.rename(sstateinst + state[0], state[1])
-        except OSError:
-            shutil.move(sstateinst + state[0], state[1])
+        bb.utils.rename(sstateinst + state[0], state[1])
     sstate_install(ss, d)
 
     for plain in ss['plaindirs']:
@@ -417,10 +413,7 @@ def sstate_installpkgdir(ss, d):
         dest = plain
         bb.utils.mkdirhier(src)
         prepdir(dest)
-        try:
-            os.rename(src, dest)
-        except OSError:
-            shutil.move(src, dest)
+        bb.utils.rename(src, dest)
 
     return True
 
@@ -645,7 +638,6 @@ python sstate_hardcode_path () {
 
 def sstate_package(ss, d):
     import oe.path
-    import shutil
 
     tmpdir = d.getVar('TMPDIR')
 
@@ -672,10 +664,7 @@ def sstate_package(ss, d):
                     continue
                 bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
-        try:
-            os.rename(state[1], sstatebuild + state[0])
-        except OSError:
-            shutil.move(state[1], sstatebuild + state[0])
+        bb.utils.rename(state[1], sstatebuild + state[0])
 
     workdir = d.getVar('WORKDIR')
     sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -685,10 +674,7 @@ def sstate_package(ss, d):
             pdir = plain.replace(sharedworkdir, sstatebuild)
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
-        try:
-            os.rename(plain, pdir)
-        except OSError:
-            shutil.move(plain, pdir)
+        bb.utils.rename(plain, pdir)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
meta/classes/update-alternatives.bbclass +2 -9
index 7e213472e3..000e4d5664 100644
@@ -139,7 +139,6 @@ python apply_update_alternative_renames () {
        return
 
     import re
-    import shutil
 
     def update_files(alt_target, alt_target_rename, pkg, d):
         f = d.getVar('FILES_' + pkg)
@@ -185,10 +184,7 @@ python apply_update_alternative_renames () {
                         link_rename.append((alt_target, alt_target_rename))
                     else:
                         bb.note('%s: Rename %s -> %s' % (pn, alt_target, alt_target_rename))
-                        try:
-                            os.rename(src, dest)
-                        except OSError:
-                            shutil.move(src, dest)
+                        bb.utils.rename(src, dest)
                         update_files(alt_target, alt_target_rename, pkg, d)
                 else:
                     bb.warn("%s: alternative target (%s or %s) does not exist, skipping..." % (pn, alt_target, alt_target_rename))
@@ -205,10 +201,7 @@ python apply_update_alternative_renames () {
             if os.path.lexists(link_target):
                 # Ok, the link_target exists, we can rename
                 bb.note('%s: Rename (link) %s -> %s' % (pn, alt_target, alt_target_rename))
-                try:
-                    os.rename(src, dest)
-                except OSError:
-                    shutil.move(src, dest)
+                bb.utils.rename(src, dest)
             else:
                 # Try to resolve the broken link to link.${BPN}
                 link_maybe = '%s.%s' % (os.readlink(src), pn)
meta/lib/oe/package_manager/deb/__init__.py +3 -15
index 8edc653dff..a4b6b6f647 100644
@@ -5,7 +5,6 @@
 import re
 import subprocess
 from oe.package_manager import *
-import shutil
 
 class DpkgIndexer(Indexer):
     def _create_configs(self):
@@ -215,10 +214,7 @@ class DpkgPM(OpkgDpkgPM):
 
                     tmp_sf.write(status)
 
-        try:
-            os.rename(status_file + ".tmp", status_file)
-        except OSError:
-            shutil.move(status_file + ".tmp", status_file)
+        bb.utils.rename(status_file + ".tmp", status_file)
 
     def run_pre_post_installs(self, package_name=None):
         """
@@ -303,21 +299,13 @@ class DpkgPM(OpkgDpkgPM):
             for dir in dirs:
                 new_dir = re.sub(r"\.dpkg-new", "", dir)
                 if dir != new_dir:
-                    try:
-                        os.rename(os.path.join(root, dir),
-                              os.path.join(root, new_dir))
-                    except OSError:
-                        shutil.move(os.path.join(root, dir),
+                    bb.utils.rename(os.path.join(root, dir),
                               os.path.join(root, new_dir))
 
             for file in files:
                 new_file = re.sub(r"\.dpkg-new", "", file)
                 if file != new_file:
-                    try:
-                        os.rename(os.path.join(root, file),
-                              os.path.join(root, new_file))
-                    except OSError:
-                        shutil.move(os.path.join(root, file),
+                    bb.utils.rename(os.path.join(root, file),
                               os.path.join(root, new_file))
 
 
meta/lib/oe/package_manager/ipk/__init__.py +1 -4
index 2aa21949f3..4cd3963111 100644
@@ -213,10 +213,7 @@ class OpkgPM(OpkgDpkgPM):
 
                     tmp_sf.write(status)
 
-        try:
-            os.rename(status_file + ".tmp", status_file)
-        except OSError:
-            shutil.move(status_file + ".tmp", status_file)
+        bb.utils.rename(status_file + ".tmp", status_file)
 
     def _create_custom_config(self):
         bb.note("Building from feeds activated!")
meta/lib/oe/rootfs.py +3 -12
index ef00010047..a7cc4e084a 100644
@@ -114,10 +114,7 @@ class Rootfs(object, metaclass=ABCMeta):
             shutil.rmtree(self.image_rootfs + '-orig')
         except:
             pass
-        try:
-            os.rename(self.image_rootfs, self.image_rootfs + '-orig')
-        except OSError:
-            shutil.move(self.image_rootfs, self.image_rootfs + '-orig')
+        bb.utils.rename(self.image_rootfs, self.image_rootfs + '-orig')
 
         bb.note("  Creating debug rootfs...")
         bb.utils.mkdirhier(self.image_rootfs)
@@ -168,16 +165,10 @@ class Rootfs(object, metaclass=ABCMeta):
             shutil.rmtree(self.image_rootfs + '-dbg')
         except:
             pass
-        try:
-            os.rename(self.image_rootfs, self.image_rootfs + '-dbg')
-        except OSError:
-            shutil.move(self.image_rootfs, self.image_rootfs + '-dbg')
+        bb.utils.rename(self.image_rootfs, self.image_rootfs + '-dbg')
 
         bb.note("  Restoreing original rootfs...")
-        try:
-            os.rename(self.image_rootfs + '-orig', self.image_rootfs)
-        except OSError:
-            shutil.move(self.image_rootfs + '-orig', self.image_rootfs)
+        bb.utils.rename(self.image_rootfs + '-orig', self.image_rootfs)
 
     def _exec_shell_cmd(self, cmd):
         fakerootcmd = self.d.getVar('FAKEROOT')
meta/lib/oeqa/selftest/cases/wic.py +3 -13
index fa93973087..a11e2d0781 100644
@@ -1269,7 +1269,6 @@ class Wic2(WicTestCase):
 
     def test_expand_mbr_image(self):
         """Test wic write --expand command for mbr image"""
-        import shutil
         # build an image
         config = 'IMAGE_FSTYPES = "wic"\nWKS_FILE = "directdisk.wks"\n'
         self.append_config(config)
@@ -1307,14 +1306,8 @@ class Wic2(WicTestCase):
             result = runCmd("%s/usr/sbin/sfdisk -F %s" % (sysroot, new_image_path))
             self.assertTrue("0 B, 0 bytes, 0 sectors" in result.output)
 
-            try:
-                os.rename(image_path, image_path + '.bak')
-            except OSError:
-                shutil.move(image_path, image_path + '.bak')
-            try:
-                os.rename(new_image_path, image_path)
-            except OSError:
-                shutil.move(new_image_path, image_path)
+            bb.utils.rename(image_path, image_path + '.bak')
+            bb.utils.rename(new_image_path, image_path)
 
             # Check if it boots in qemu
             with runqemu('core-image-minimal', ssh=False) as qemu:
@@ -1325,10 +1318,7 @@ class Wic2(WicTestCase):
             if os.path.exists(new_image_path):
                 os.unlink(new_image_path)
             if os.path.exists(image_path + '.bak'):
-                try:
-                    os.rename(image_path + '.bak', image_path)
-                except OSError:
-                    shutil.move(image_path + '.bak', image_path)
+                bb.utils.rename(image_path + '.bak', image_path)
 
     def test_wic_ls_ext(self):
         """Test listing content of the ext partition using 'wic ls'"""
scripts/combo-layer +1 -4
index 1b88340b6e..045de65642 100755
@@ -508,10 +508,7 @@ def check_patch(patchfile):
     f.close()
     if of:
         of.close()
-        try:
-            os.rename(patchfile + '.tmp', patchfile)
-        except OSError:
-            shutil.move(patchfile + '.tmp', patchfile)
+        bb.utils.rename(patchfile + '.tmp', patchfile)
 
 def drop_to_shell(workdir=None):
     if not sys.stdin.isatty():
scripts/lib/devtool/standard.py +4 -17
index 3854dfb3e8..5eba2191d9 100644
@@ -746,10 +746,7 @@ def _check_preserve(config, recipename):
                         os.remove(removefile)
                 else:
                     tf.write(line)
-    try:
-        os.rename(newfile, origfile)
-    except OSError:
-        shutil.move(newfile, origfile)
+    bb.utils.rename(newfile, origfile)
 
 def get_staging_kver(srcdir):
     # Kernel version from work-shared
@@ -1097,16 +1094,10 @@ def rename(args, config, basepath, workspace):
 
     # Rename bbappend
     logger.info('Renaming %s to %s' % (append, newappend))
-    try:
-        os.rename(append, newappend)
-    except OSError:
-        shutil.move(append, newappend)
+    bb.utils.rename(append, newappend)
     # Rename recipe file
     logger.info('Renaming %s to %s' % (recipefile, newfile))
-    try:
-        os.rename(recipefile, newfile)
-    except OSError:
-        shutil.move(recipefile, newfile)
+    bb.utils.rename(recipefile, newfile)
 
     # Rename source tree if it's the default path
     appendmd5 = None
@@ -1342,11 +1333,7 @@ def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None):
         if match_name:
             # Rename patch files
             if new_patch != match_name:
-                try:
-                    os.rename(os.path.join(destdir, new_patch),
-                          os.path.join(destdir, match_name))
-                except OSError:
-                    shutil.move(os.path.join(destdir, new_patch),
+                bb.utils.rename(os.path.join(destdir, new_patch),
                           os.path.join(destdir, match_name))
             # Need to pop it off the list now before checking changed_revs
             oldpath = existing_patches.pop(old_patch)
scripts/lib/devtool/upgrade.py +2 -6
index 2bff69f5a2..24e3700ece 100644
@@ -71,12 +71,8 @@ def _rename_recipe_dirs(oldpv, newpv, path):
                 if oldfile.find(oldpv) != -1:
                     newfile = oldfile.replace(oldpv, newpv)
                     if oldfile != newfile:
-                        try:
-                            os.rename(os.path.join(path, oldfile),
-                                  os.path.join(path, newfile))
-                        except OSError:
-                            shutil.move(os.path.join(path, oldfile),
-                                  os.path.join(path, newfile))
+                        bb.utils.rename(os.path.join(path, oldfile),
+                              os.path.join(path, newfile))
 
 def _rename_recipe_file(oldrecipe, bpn, oldpv, newpv, path):
     oldrecipe = os.path.basename(oldrecipe)
scripts/lib/wic/plugins/imager/direct.py +1 -4
index fb5e52b2a6..96168aadb4 100644
@@ -616,8 +616,5 @@ class PartitionedImage():
                              part.start + part.size_sec - 1, part.size_sec)
 
                 partimage = self.path + '.p%d' % part.num
-                try:
-                    os.rename(source, partimage)
-                except OSError:
-                    shutil.move(source, partimage)
+                bb.utils.rename(source, partimage)
                 self.partimages.append(partimage)
Signature · 1 line
2.29.2
#25 lore raw

These patches are working well at my end, but I’m still seeing the following error in do_rootfs, when building incrementally in a container. I’m not sure if this is due to timing issues alluded to earlier or something else entirely. I do know for sure that this is not caused by the patches because it happens even without them.

ERROR: core-image-base-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_rootfs(d)
     0003:
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/classes/image.bbclass', lineno: 247, function: do_rootfs
     0243:    progress_reporter.next_stage()
     0244:
     0245:    # generate rootfs
     0246:    d.setVarFlag('REPRODUCIBLE_TIMESTAMP_ROOTFS', 'export', '1')
 *** 0247:    create_rootfs(d, progress_reporter=progress_reporter, logcatcher=logcatcher)
     0248:
     0249:    progress_reporter.finish()
     0250:}
     0251:do_rootfs[dirs] = "${TOPDIR}"
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/rootfs.py', lineno: 375, function: create_rootfs
     0371:
     0372:    img_type = d.getVar('IMAGE_PKGTYPE')
     0373:
     0374:    cls = get_class_for_type(img_type)
 *** 0375:    cls(d, manifest_dir, progress_reporter, logcatcher).create()
     0376:    os.environ.clear()
     0377:    os.environ.update(env_bkp)
     0378:
     0379:

File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/rootfs.py', lineno: 203, function: create
0199: if self.progress_reporter:
0200: self.progress_reporter.next_stage()
0201:
0202: # call the package manager dependent create method
*** 0203: self._create()

     0204:
     0205:        sysconfdir = self.image_rootfs + self.d.getVar('sysconfdir')
     0206:        bb.utils.mkdirhier(sysconfdir)
     0207:        with open(sysconfdir + "/version", "w+") as ver:

File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/rootfs.py', lineno: 70, function: _create
0066: rpm_pre_process_cmds = self.d.getVar('RPM_PREPROCESS_COMMANDS')
0067: rpm_post_process_cmds = self.d.getVar('RPM_POSTPROCESS_COMMANDS')
0068:
0069: # update PM index files
*** 0070: self.pm.write_index()
0071:
0072: execute_pre_post_process(self.d, rpm_pre_process_cmds)
0073:
0074: if self.progress_reporter:
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py', lineno: 141, function: write_index

     0137:
     0138:    def write_index(self):
     0139:        lockfilename = self.d.getVar('DEPLOY_DIR_RPM') + "/rpm.lock"
     0140:        lf = bb.utils.lockfile(lockfilename, False)

*** 0141: RpmIndexer(self.d, self.rpm_repo_dir).write_index()
0142: bb.utils.unlockfile(lf)
0143:
0144: def insert_feeds_uris(self, feed_uris, feed_base_paths, feed_archs):
0145: from urllib.parse import urlparse
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py', lineno: 11, function: write_index

     0007:from oe.package_manager import *
     0008:
     0009:class RpmIndexer(Indexer):
     0010:    def write_index(self):
 *** 0011:        self.do_write_index(self.deploy_dir)
     0012:
     0013:    def do_write_index(self, deploy_dir):
     0014:        if self.d.getVar('PACKAGE_FEED_SIGN') == '1':
     0015:            signer = get_signer(self.d, self.d.getVar('PACKAGE_FEED_GPG_BACKEND'))
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py', lineno: 20, function: do_write_index
     0016:        else:
     0017:            signer = None
     0018:
     0019:        createrepo_c = bb.utils.which(os.environ['PATH'], "createrepo_c")
 *** 0020:        result = create_index("%s --update -q %s" % (createrepo_c, deploy_dir))
     0021:        if result:
     0022:            bb.fatal(result)
     0023:
     0024:        # Sign repomd
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/__init__.py', lineno: 26, function: create_index
     0022:def create_index(arg):
     0023:    index_cmd = arg
     0024:
     0025:    bb.note("Executing '%s' ..." % index_cmd)
 *** 0026:    result = subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8")
     0027:    if result:
     0028:        bb.note(result)
     0029:
     0030:def opkg_query(cmd_output):
File: '/usr/lib/python3.8/subprocess.py', lineno: 411, function: check_output
     0407:        # Explicitly passing input=None was previously equivalent to passing an
     0408:        # empty string. That is maintained here for backwards compatibility.
     0409:        kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
     0410:
 *** 0411:    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
     0412:               **kwargs).stdout
     0413:
     0414:
     0415:class CompletedProcess(object):

File: '/usr/lib/python3.8/subprocess.py', lineno: 512, function: run
0508: # We don't call process.wait() as .__exit__ does that for us.
0509: raise
0510: retcode = process.poll()
0511: if check and retcode:
*** 0512: raise CalledProcessError(retcode, process.args,
0513: output=stdout, stderr=stderr)
0514: return CompletedProcess(process.args, retcode, stdout, stderr)
0515:
0516:
Exception: subprocess.CalledProcessError: Command '/home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/recipe-sysroot-native/usr/bin/createrepo_c --update -q /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo' returned non-zero exit status 1.

Subprocess output:
Critical: Cannot rename /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/.repodata/ -> /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/repodata/: Directory not empty

ERROR: Logfile of failure stored in: /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.165072
ERROR: Task (/home/pi/docker-meta-raspberrypi/layers/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs) failed with exit code '1'

Quoted 27 lines · Devendra Tewari

Separate patches for bitbake and oe-core, as requested

<0001-use-bb.utils.rename-to-rename-file-or-directory.patch><0001-use-bb.utils.rename-to-rename-file-or-directory.patch>

I've assumed each will be applied to the master-next branch on top of the previous patch.

Thanks,
Devendra

Quoted 15 lines · Richard Purdie
Quoted 4 lines · Devendra Tewari

That's a neat idea - if the exception handler catches OSError with error number 18 (Invalid cross-device link) in os.rename, it should attempt shutil.move, otherwise it should re-throw the exception with raise. A wrapper function will make it easier to handle other situations as they arise. I'll update the patch.

Sounds good. Could you also please split the patch into two, one for bitbake and one for OE-Core since they are two different repositories.

Cheers,

Richard

#26 lore raw

I suspect this is an issue in upstream createrepo_c tool, which has code that does not work across overlay file systems

    if (g_rename(out_repo, old_repodata_path) == -1) {
        g_debug("Old repodata doesn't exists: Cannot rename %s -> %s: %s",
                out_repo, old_repodata_path, g_strerror(errno));
    } else {
        g_debug("Renamed %s -> %s", out_repo, old_repodata_path);
        old_repodata_renamed = TRUE;
    }

I’ll try patching and report an issue to upstream project.

Quoted 148 lines · Devendra Tewari

These patches are working well at my end, but I’m still seeing the following error in do_rootfs, when building incrementally in a container. I’m not sure if this is due to timing issues alluded to earlier or something else entirely. I do know for sure that this is not caused by the patches because it happens even without them.

ERROR: core-image-base-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_rootfs(d)
     0003:
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/classes/image.bbclass', lineno: 247, function: do_rootfs
     0243:    progress_reporter.next_stage()
     0244:
     0245:    # generate rootfs
     0246:    d.setVarFlag('REPRODUCIBLE_TIMESTAMP_ROOTFS', 'export', '1')
 *** 0247:    create_rootfs(d, progress_reporter=progress_reporter, logcatcher=logcatcher)
     0248:
     0249:    progress_reporter.finish()
     0250:}
     0251:do_rootfs[dirs] = "${TOPDIR}"
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/rootfs.py', lineno: 375, function: create_rootfs
     0371:
     0372:    img_type = d.getVar('IMAGE_PKGTYPE')
     0373:
     0374:    cls = get_class_for_type(img_type)
 *** 0375:    cls(d, manifest_dir, progress_reporter, logcatcher).create()
     0376:    os.environ.clear()
     0377:    os.environ.update(env_bkp)
     0378:
     0379:

File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/rootfs.py', lineno: 203, function: create
0199: if self.progress_reporter:
0200: self.progress_reporter.next_stage()
0201:
0202: # call the package manager dependent create method
*** 0203: self._create()

     0204:
     0205:        sysconfdir = self.image_rootfs + self.d.getVar('sysconfdir')
     0206:        bb.utils.mkdirhier(sysconfdir)
     0207:        with open(sysconfdir + "/version", "w+") as ver:

File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/rootfs.py', lineno: 70, function: _create
0066: rpm_pre_process_cmds = self.d.getVar('RPM_PREPROCESS_COMMANDS')
0067: rpm_post_process_cmds = self.d.getVar('RPM_POSTPROCESS_COMMANDS')
0068:
0069: # update PM index files
*** 0070: self.pm.write_index()
0071:
0072: execute_pre_post_process(self.d, rpm_pre_process_cmds)
0073:
0074: if self.progress_reporter:
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py', lineno: 141, function: write_index

     0137:
     0138:    def write_index(self):
     0139:        lockfilename = self.d.getVar('DEPLOY_DIR_RPM') + "/rpm.lock"
     0140:        lf = bb.utils.lockfile(lockfilename, False)

*** 0141: RpmIndexer(self.d, self.rpm_repo_dir).write_index()
0142: bb.utils.unlockfile(lf)
0143:
0144: def insert_feeds_uris(self, feed_uris, feed_base_paths, feed_archs):
0145: from urllib.parse import urlparse
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py', lineno: 11, function: write_index

     0007:from oe.package_manager import *
     0008:
     0009:class RpmIndexer(Indexer):
     0010:    def write_index(self):
 *** 0011:        self.do_write_index(self.deploy_dir)
     0012:
     0013:    def do_write_index(self, deploy_dir):
     0014:        if self.d.getVar('PACKAGE_FEED_SIGN') == '1':
     0015:            signer = get_signer(self.d, self.d.getVar('PACKAGE_FEED_GPG_BACKEND'))
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py', lineno: 20, function: do_write_index
     0016:        else:
     0017:            signer = None
     0018:
     0019:        createrepo_c = bb.utils.which(os.environ['PATH'], "createrepo_c")
 *** 0020:        result = create_index("%s --update -q %s" % (createrepo_c, deploy_dir))
     0021:        if result:
     0022:            bb.fatal(result)
     0023:
     0024:        # Sign repomd
File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/__init__.py', lineno: 26, function: create_index
     0022:def create_index(arg):
     0023:    index_cmd = arg
     0024:
     0025:    bb.note("Executing '%s' ..." % index_cmd)
 *** 0026:    result = subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True).decode("utf-8")
     0027:    if result:
     0028:        bb.note(result)
     0029:
     0030:def opkg_query(cmd_output):
File: '/usr/lib/python3.8/subprocess.py', lineno: 411, function: check_output
     0407:        # Explicitly passing input=None was previously equivalent to passing an
     0408:        # empty string. That is maintained here for backwards compatibility.
     0409:        kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
     0410:
 *** 0411:    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
     0412:               **kwargs).stdout
     0413:
     0414:
     0415:class CompletedProcess(object):

File: '/usr/lib/python3.8/subprocess.py', lineno: 512, function: run
0508: # We don't call process.wait() as .__exit__ does that for us.
0509: raise
0510: retcode = process.poll()
0511: if check and retcode:
*** 0512: raise CalledProcessError(retcode, process.args,
0513: output=stdout, stderr=stderr)
0514: return CompletedProcess(process.args, retcode, stdout, stderr)
0515:
0516:
Exception: subprocess.CalledProcessError: Command '/home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/recipe-sysroot-native/usr/bin/createrepo_c --update -q /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo' returned non-zero exit status 1.

Subprocess output:
Critical: Cannot rename /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/.repodata/ -> /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/repodata/: Directory not empty

ERROR: Logfile of failure stored in: /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.165072
ERROR: Task (/home/pi/docker-meta-raspberrypi/layers/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs) failed with exit code '1'

Quoted 27 lines · On 21 Apr 2021, at 18:08, Devendra Tewari <[email protected]…

Separate patches for bitbake and oe-core, as requested

<0001-use-bb.utils.rename-to-rename-file-or-directory.patch><0001-use-bb.utils.rename-to-rename-file-or-directory.patch>

I've assumed each will be applied to the master-next branch on top of the previous patch.

Thanks,
Devendra

Quoted 15 lines · On 21 Apr 2021, at 16:21, Richard Purdie <[email protected]…
Quoted 4 lines · Devendra Tewari

That's a neat idea - if the exception handler catches OSError with error number 18 (Invalid cross-device link) in os.rename, it should attempt shutil.move, otherwise it should re-throw the exception with raise. A wrapper function will make it easier to handle other situations as they arise. I'll update the patch.

Sounds good. Could you also please split the patch into two, one for bitbake and one for OE-Core since they are two different repositories.

Cheers,

Richard

#27 lore raw
Quoted 11 lines · Devendra Tewari

I suspect this is an issue in upstream createrepo_c tool, which has code that does not work across overlay file systems

    if (g_rename(out_repo, old_repodata_path) == -1) { g_debug("Old repodata doesn't exists: Cannot rename %s -> %s: %s", out_repo, old_repodata_path, g_strerror(errno)); } else {
        g_debug("Renamed %s -> %s", out_repo, old_repodata_path); old_repodata_renamed = TRUE;
    }

Quoted 1 line

I’ll try patching and report an issue to upstream project.

I'm starting to worry that you're going to have this problem with many tools doing renames. It does sound like something the container filesystem should be handling rather than exposing as a difference in behaviour. I suspect it becomes hard to see predictably since builds writing in the same build instance are probably fine and it only likely happens for files created in one instance and then later renamed in another...

Given the breadth of build systems we run, I'm not sure we want to try and support this?

Cheers,

Richard

#28 lore raw

You’re absolutely right - I’ll investigate whether this is something the container overlay filesystem should be handling. Thanks!

Quoted 30 lines · Richard Purdie
Quoted 11 lines · Devendra Tewari

I suspect this is an issue in upstream createrepo_c tool, which has code that does not work across overlay file systems

    if (g_rename(out_repo, old_repodata_path) == -1) {
        g_debug("Old repodata doesn't exists: Cannot rename %s -> %s: %s",
                out_repo, old_repodata_path, g_strerror(errno));
    } else {
        g_debug("Renamed %s -> %s", out_repo, old_repodata_path);
        old_repodata_renamed = TRUE;
    }
Quoted 1 line

I’ll try patching and report an issue to upstream project.

I'm starting to worry that you're going to have this problem with many tools doing renames. It does sound like something the container filesystem should be handling rather than exposing as a difference in behaviour. I suspect it becomes hard to see predictably since builds writing in the same build instance are probably fine and it only likely happens for files created in one instance and then later renamed in another...

Given the breadth of build systems we run, I'm not sure we want to try and support this?

Cheers,

Richard

#29 lore raw

Looks like an old issue with overlayfs - https://github.com/moby/moby/issues/25409.

Quoted 35 lines · Devendra Tewari

You’re absolutely right - I’ll investigate whether this is something the container overlay filesystem should be handling. Thanks!

Quoted 30 lines · Richard Purdie
Quoted 11 lines · Devendra Tewari

I suspect this is an issue in upstream createrepo_c tool, which has code that does not work across overlay file systems

   if (g_rename(out_repo, old_repodata_path) == -1) {
       g_debug("Old repodata doesn't exists: Cannot rename %s -> %s: %s",
               out_repo, old_repodata_path, g_strerror(errno));
   } else {
       g_debug("Renamed %s -> %s", out_repo, old_repodata_path);
       old_repodata_renamed = TRUE;
   }
Quoted 1 line

I’ll try patching and report an issue to upstream project.

I'm starting to worry that you're going to have this problem with many tools doing renames. It does sound like something the container filesystem should be handling rather than exposing as a difference in behaviour. I suspect it becomes hard to see predictably since builds writing in the same build instance are probably fine and it only likely happens for files created in one instance and then later renamed in another...

Given the breadth of build systems we run, I'm not sure we want to try and support this?

Cheers,

Richard

#30 lore raw

The gist of it is that Docker uses the overlay2 storage driver by default, previously it used aufs. Both drivers have the same limitation in rename, where file rename across layers is handled by moving, but directory rename across layers results in error EXDEV. It is up to user space to handle the error.

Quoted 40 lines · Devendra Tewari

Looks like an old issue with overlayfs - https://github.com/moby/moby/issues/25409.

Quoted 35 lines · Devendra Tewari

You’re absolutely right - I’ll investigate whether this is something the container overlay filesystem should be handling. Thanks!

Quoted 30 lines · Richard Purdie
Quoted 11 lines · Devendra Tewari

I suspect this is an issue in upstream createrepo_c tool, which has code that does not work across overlay file systems

  if (g_rename(out_repo, old_repodata_path) == -1) {
      g_debug("Old repodata doesn't exists: Cannot rename %s -> %s: %s",
              out_repo, old_repodata_path, g_strerror(errno));
  } else {
      g_debug("Renamed %s -> %s", out_repo, old_repodata_path);
      old_repodata_renamed = TRUE;
  }
Quoted 1 line

I’ll try patching and report an issue to upstream project.

I'm starting to worry that you're going to have this problem with many tools doing renames. It does sound like something the container filesystem should be handling rather than exposing as a difference in behaviour. I suspect it becomes hard to see predictably since builds writing in the same build instance are probably fine and it only likely happens for files created in one instance and then later renamed in another...

Given the breadth of build systems we run, I'm not sure we want to try and support this?

Cheers,

Richard

#31 lore raw
Quoted 4 lines · Devendra Tewari

The gist of it is that Docker uses the overlay2 storage driver by default, previously it used aufs. Both drivers have the same limitation in rename, where file rename across layers is handled by moving, but directory rename across layers results in error EXDEV. It is up to user space to handle the error.

I'm still struggling to see how you're going to patch all the various pieces of software out there used during a build to handle EXDEV correctly.

We have handled some cases of this for a long time where we're moving files over to different parts of a filesystem, e.g. /tmp/ to WORKDIR or vice versa.

This new docker case triggers for renames even within a directory, e.g. /some/path/a -> /some/path/b which is not something that would often trigger this.

I need to say clearly now that we are not carrying/taking patches to patch every bit of other software to try and avoid this.

Cheers,

Richard

#32 lore raw

Understood - carrying patches is a burden for sure. I’ll maintain patches for my own narrow use cases at my end, and perhaps submit them upstream.

Thanks,
Devendra

Quoted 27 lines · Richard Purdie
Quoted 4 lines · Devendra Tewari

The gist of it is that Docker uses the overlay2 storage driver by default, previously it used aufs. Both drivers have the same limitation in rename, where file rename across layers is handled by moving, but directory rename across layers results in error EXDEV. It is up to user space to handle the error.

I'm still struggling to see how you're going to patch all the various pieces of software out there used during a build to handle EXDEV correctly.

We have handled some cases of this for a long time where we're moving files over to different parts of a filesystem, e.g. /tmp/ to WORKDIR or vice versa.

This new docker case triggers for renames even within a directory, e.g. /some/path/a -> /some/path/b which is not something that would often trigger this.

I need to say clearly now that we are not carrying/taking patches to patch every bit of other software to try and avoid this.

Cheers,

Richard

#34 lore raw
In reply to a message not in the archive

I was asked about the createrepo_c patch in private. Replying here for closure.

I had reported the issue upstream at https://github.com/rpm-software-management/createrepo_c/issues/266 and it has been fixed. Poky kirkstone and mainline should have the fix.

Kind Regards,
Devendra

Quoted 37 lines · Devendra Tewari

Understood - carrying patches is a burden for sure. I’ll maintain patches for my own narrow use cases at my end, and perhaps submit them upstream.

Thanks,
Devendra

Quoted 27 lines · Richard Purdie
Quoted 4 lines · Devendra Tewari

The gist of it is that Docker uses the overlay2 storage driver by default, previously it used aufs. Both drivers have the same limitation in rename, where file rename across layers is handled by moving, but directory rename across layers results in error EXDEV. It is up to user space to handle the error.

I'm still struggling to see how you're going to patch all the various pieces of software out there used during a build to handle EXDEV correctly.

We have handled some cases of this for a long time where we're moving files over to different parts of a filesystem, e.g. /tmp/ to WORKDIR or vice versa.

This new docker case triggers for renames even within a directory, e.g. /some/path/a -> /some/path/b which is not something that would often trigger this.

I need to say clearly now that we are not carrying/taking patches to patch every bit of other software to try and avoid this.

Cheers,

Richard