oe/copy_buildsystem.py: dereference symlink

PATCH In review Robert Yang 3 participants 6 messages Oct 31, 2016 – Nov 1, 2016

lore

Share & tools

lore

Robert Yang

author

2016-10-31 15:48 UTC #0 lore raw

cover oe/copy_buildsystem.py: dereference symlink

The following changes since commit 42b4fa2f923244bc047874752d2e0381ff6f0a25:

boost: fix the SRC_URI to point to an actual release, and not a master snapshot (2016-10-31 14:23:04 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/cp
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/cp

Robert Yang (1):
oe/copy_buildsystem.py: dereference symlink

 meta/lib/oe/copy_buildsystem.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Signature · 1 line

2.9.0

Robert Yang

author

2016-10-31 15:48 UTC #1 lore raw

PATCH 1/1 oe/copy_buildsystem.py: dereference symlink

When there is a relative symlink in the layer, for example:
symA -> ../out/of/layer/file

symA will be invalid fater copied, it would be invalid from build time if it points to a relative path, and would be invalid after extracted the sdk if it points to a absolute py. Dereference symlink when copy will fix the problem.

Use tar rather than shutil.copytree() to copy is because:
1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks:
https://bugs.python.org/issue21697
And Ubunutu 1404 doesn't upgrade python3 to fix the problem.

2) shutil.copytree(symlinks=False) raises errors when there is a invalid symlink, and tar just prints a warning, tar is preferred here since the real world is unpredicatable

3) tar is faster than shutil.copytree() as said by oe.path.copytree()

So use tar to copy.

Signed-off-by Robert Yang <[email protected]>

---
 meta/lib/oe/copy_buildsystem.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

meta/lib/oe/copy_buildsystem.py +5 -1

index afaff68..29ac6d4 100644
@@ -4,11 +4,15 @@ import stat
 import shutil
 
 def _smart_copy(src, dest):
+    import subprocess
     # smart_copy will choose the correct function depending on whether the
     # source is a file or a directory.
     mode = os.stat(src).st_mode
     if stat.S_ISDIR(mode):
-        shutil.copytree(src, dest, symlinks=True, ignore=shutil.ignore_patterns('.git'))
+        bb.utils.mkdirhier(dest)
+        cmd = "tar --exclude='.git' --xattrs --xattrs-include='*' -chf - -C %s -p . \
+        | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dest)
+        subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
         shutil.copyfile(src, dest)
         shutil.copymode(src, dest)

Signature · 1 line

2.9.0

Richard Purdie

2016-11-01 10:09 UTC #2 lore raw

Quoted 15 lines · Robert Yang

When there is a relative symlink in the layer, for example:
symA -> ../out/of/layer/file
symA will be invalid fater copied, it would be invalid from build time
if it points to a relative path, and would be invalid after extracted the sdk if it points to a absolute py. Dereference symlink when copy will fix the problem.
Use tar rather than shutil.copytree() to copy is because:
1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks:
https://bugs.python.org/issue21697
And Ubunutu 1404 doesn't upgrade python3 to fix the problem.

Quoted 7 lines

2) shutil.copytree(symlinks=False) raises errors when there is a invalid
symlink, and tar just prints a warning, tar is preferred here since
the real world is unpredicatable
3) tar is faster than shutil.copytree() as said by oe.path.copytree()

Could we just use oe.path.copytree() here?

Cheers,

Richard

Robert Yang

author

2016-11-01 10:18 UTC #3 lore raw

Quoted 22 lines · Richard Purdie

Quoted 21 lines · Robert Yang
When there is a relative symlink in the layer, for example:
symA -> ../out/of/layer/file
symA will be invalid fater copied, it would be invalid from build time
if it points to a relative path, and would be invalid after extracted the sdk if it points to a absolute py. Dereference symlink when copy will fix the problem.
Use tar rather than shutil.copytree() to copy is because:
1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks:
https://bugs.python.org/issue21697
And Ubunutu 1404 doesn't upgrade python3 to fix the problem.
2) shutil.copytree(symlinks=False) raises errors when there is a invalid
symlink, and tar just prints a warning, tar is preferred here
since
the real world is unpredicatable

Quoted 3 lines

Quoted 1 line
3) tar is faster than shutil.copytree() as said by oe.path.copytree()
Could we just use oe.path.copytree() here?

I should explain why not use oe.path.copytree(), but it was late last night, so I forgot that. We can't use oe.path.copytree() since:
1) oe.path.copytree() doesn't dereference symlink
2) We need --exclude='.git'

I'd like to add two arguments to oe.path.copytree() to do this if it is worth.

// Robert

Quoted 5 lines

Cheers,
Richard

Burton, Ross

2016-11-01 12:15 UTC #4 lore raw

Quoted 8 lines · Robert Yang

I should explain why not use oe.path.copytree(), but it was late last night, so I forgot that. We can't use oe.path.copytree() since:
1) oe.path.copytree() doesn't dereference symlink
2) We need --exclude='.git'
I'd like to add two arguments to oe.path.copytree() to do this if it is worth.

Adding options to dereference and exclude a specified list of files sounds like a good improvement to me.

Ross

Robert Yang

author

2016-11-01 13:17 UTC #5 lore raw

Quoted 9 lines · Ross

On 1 November 2016 at 10:18, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
I should explain why not use oe.path.copytree(), but it was late last night, so I forgot that. We can't use oe.path.copytree() since:
1) oe.path.copytree() doesn't dereference symlink
2) We need --exclude='.git'

Quoted 5 lines

I'd like to add two arguments to oe.path.copytree() to do this if it is worth.
Adding options to dereference and exclude a specified list of files sounds like a good improvement to me.

Thanks, will send a V2.

// Robert

Quoted 2 lines

Ross