Preserve extended attributes in sstate objects

PATCH Superseded Joshua Lock 6 participants 10 messages Aug 26, 2016 – Sep 6, 2016

lore

Share & tools

lore

Joshua Lock

author

2016-08-26 15:43 UTC #0 lore raw

cover Preserve extended attributes in sstate objects

This small series is part of a larger, on-going, effort to better support extended attributes (xattr).

The goal of sending these two patches before the whole is complete is to improve support for distros with features that rely on xattr that wish to use meta-swupd.

meta-swupd creates sstate objects of update artefacts so that an OS update delta can be generated against a previous OS release without having to have the full build history in TMPDIR — this is especially useful for CI workflows. Without these changes sstate objects don't preserve xattr and thus swupd updates artefacts are incorrect/incomplete.

Regards,

Joshua

The following changes since commit 00027aee12f4bbc9a4ba607c91fcc1e0e8257fa2:

core-image-kernel-dev.bb: Standardize use of _append and leading space. (2016-08-25 23:03:26 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib joshuagl/xattr
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/xattr

Joshua Lock (2):
buildtools-tarball: add tar
oe.path: preserve xattr in copytree() and copyhardlinktree()

 meta/lib/oe/path.py                          | 11 ++++++++---
 meta/recipes-core/meta/buildtools-tarball.bb |  1 +
 2 files changed, 9 insertions(+), 3 deletions(-)

Signature · 1 line

2.7.4

Joshua Lock

author

2016-08-26 15:43 UTC #1 lore raw

PATCH 1/2 buildtools-tarball: add tar

We'd like to make use of features which aren't available in tar versions prior to 1.27, therefore add tar to the buildtools-tarball so that older distributions can still perform builds.

Signed-off-by Joshua Lock <[email protected]>

---
 meta/recipes-core/meta/buildtools-tarball.bb | 1 +
 1 file changed, 1 insertion(+)

meta/recipes-core/meta/buildtools-tarball.bb +1 -0

index 34df531..f28a8c3 100644
@@ -24,6 +24,7 @@ TOOLCHAIN_HOST_TASK ?= "\
     nativesdk-ca-certificates \
     nativesdk-texinfo \
     nativesdk-locale-base-en-us \
+    nativesdk-tar \
     "
 
 SDK_PACKAGE_ARCHS += "buildtools-dummy-${SDKPKGSUFFIX}"

Signature · 1 line

2.7.4

Paul Eggleton

2016-08-30 12:24 UTC #2 lore raw

Re: [PATCH 1/2] buildtools-tarball: add tar

Quoted 15 lines · On Fri, 26 Aug 2016 16:43:05 Joshua Lock wrote:

We'd like to make use of features which aren't available in tar versions prior to 1.27, therefore add tar to the buildtools-tarball so that older distributions can still perform builds.
Signed-off-by Joshua Lock <[email protected]>
---
 meta/recipes-core/meta/buildtools-tarball.bb | 1 +
 1 file changed, 1 insertion(+)
(quoted diff) +0 -0
b/meta/recipes-core/meta/buildtools-tarball.bb index 34df531..f28a8c3 100644

meta/recipes-core/meta/buildtools-tarball.bb TOOLCHAIN_HOST_TASK ?= "\ · line 30

Quoted 7 lines

meta/recipes-core/meta/buildtools-tarball.bb +1 -0

@@ -24,6 +24,7 @@ TOOLCHAIN_HOST_TASK ?= "\
     nativesdk-ca-certificates \
     nativesdk-texinfo \
     nativesdk-locale-base-en-us \
+    nativesdk-tar \
     "
 
 SDK_PACKAGE_ARCHS += "buildtools-dummy-${SDKPKGSUFFIX}"

Er, tar is already in buildtools-tarball... right?

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Joshua Lock

author

2016-08-26 15:43 UTC #3 lore raw

PATCH 2/2 oe.path: preserve xattr in copytree() and copyhardlinktree()

In reply to Joshua Lock

Pass appropriate options to tar invocations in copytree() and copyhardlinktree() to ensure that any extended attributes on the files are preserved during the copy.

We have to drop the use cpio in "Copy-pass" mode in copyhardlinktree() because cpio doesn't support extended attributes on files. Instead we revert back to using cp with different patterns depending on whether or not the directory contains dot files.

[YOCTO #9857]

Signed-off-by Joshua Lock <[email protected]>

---
 meta/lib/oe/path.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

meta/lib/oe/path.py +8 -3

index 3c07df3..631c3b4 100644
@@ -65,7 +65,7 @@ def copytree(src, dst):
     # This way we also preserve hardlinks between files in the tree.
 
     bb.utils.mkdirhier(dst)
-    cmd = 'tar -cf - -C %s -p . | tar -xf - -C %s' % (src, dst)
+    cmd = "tar --xattrs --xattrs-include='*' -cf - -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
     subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
 
 def copyhardlinktree(src, dst):

@@ -77,9 +77,14 @@ def copyhardlinktree(src, dst):
     if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
         # Need to copy directories only with tar first since cp will error if two 
         # writers try and create a directory at the same time
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --no-recursion --files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
-        cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' % (src, dst)
+        if os.path.isdir(src):
+            import glob
+            if len(glob.glob('%s/.??*' % src)) > 0:
+                src = src + '/.??* '
+            src = src + '/*'
+        cmd = 'cp -afl --preserve=xattr %s %s' % (src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
         copytree(src, dst)

Signature · 1 line

2.7.4

Mark Hatle

2016-08-26 15:48 UTC #4 lore raw

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

Quoted 30 lines · Joshua Lock

Pass appropriate options to tar invocations in copytree() and copyhardlinktree() to ensure that any extended attributes on the files are preserved during the copy.
We have to drop the use cpio in "Copy-pass" mode in copyhardlinktree() because cpio doesn't support extended attributes on files. Instead we revert back to using cp with different patterns depending on whether or not the directory contains dot files.
[YOCTO #9857]
Signed-off-by Joshua Lock <[email protected]>
---
 meta/lib/oe/path.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
meta/lib/oe/path.py +1 -1
index 3c07df3..631c3b4 100644
@@ -65,7 +65,7 @@ def copytree(src, dst):
     # This way we also preserve hardlinks between files in the tree.
 
     bb.utils.mkdirhier(dst)
-    cmd = 'tar -cf - -C %s -p . | tar -xf - -C %s' % (src, dst)
+    cmd = "tar --xattrs --xattrs-include='*' -cf - -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
     subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
 
 def copyhardlinktree(src, dst):

meta/lib/oe/path.py def copyhardlinktree(src, dst): · line 82

Quoted 7 lines

meta/lib/oe/path.py +1 -2

@@ -77,9 +77,14 @@ def copyhardlinktree(src, dst):
     if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
         # Need to copy directories only with tar first since cp will error if two 
         # writers try and create a directory at the same time
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --no-recursion --files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
-        cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' % (src, dst)

The reason why cpio was originally used had to do with filenames that had spaces or other special characters in them. (this is why the -print0 was used)

I don't object to the change, but we need to make sure that that will still work properly.

(Also I thought CPIO -could- do xattrs in the current versions...)

--Mark

Quoted 10 lines

meta/lib/oe/path.py +6 -0

+        if os.path.isdir(src):
+            import glob
+            if len(glob.glob('%s/.??*' % src)) > 0:
+                src = src + '/.??* '
+            src = src + '/*'
+        cmd = 'cp -afl --preserve=xattr %s %s' % (src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
         copytree(src, dst)

Joshua Lock

2016-08-30 12:00 UTC #5 lore raw

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

Quoted 58 lines · Mark Hatle

Quoted 41 lines · Joshua Lock
Pass appropriate options to tar invocations in copytree() and copyhardlinktree() to ensure that any extended attributes on the files
are preserved during the copy.
We have to drop the use cpio in "Copy-pass" mode in copyhardlinktree()
because cpio doesn't support extended attributes on files. Instead we
revert back to using cp with different patterns depending on whether
or not the directory contains dot files.
[YOCTO #9857]
Signed-off-by Joshua Lock <[email protected]>
---
meta/lib/oe/path.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
meta/lib/oe/path.py +0 -0
index 3c07df3..631c3b4 100644
@@ -65,7 +65,7 @@ def copytree(src, dst):
      # This way we also preserve hardlinks between files in the
tree.

bb.utils.mkdirhier(dst)
- cmd = 'tar -cf - -C %s -p . | tar -xf - -C %s' % (src, dst) + cmd = "tar --xattrs --xattrs-include='*' -cf - -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst) subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)

def copyhardlinktree(src, dst):
meta/lib/oe/path.py +0 -0
@@ -77,9 +77,14 @@ def copyhardlinktree(src, dst):
if (os.stat(src).st_dev == os.stat(dst).st_dev):
# Need to copy directories only with tar first since cp will error if two
Quoted 10 lines
         # writers try and create a directory at the same time
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
--no-recursion --files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = "cd %s; find . -type d -print | tar --xattrs --
xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - |
tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True,
stderr=subprocess.STDOUT)
-        cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' %
(src, dst)
The reason why cpio was originally used had to do with filenames that had spaces
or other special characters in them. (this is why the -print0 was used)

Quoted 5 lines

I don't object to the change, but we need to make sure that that will still work
properly.
(Also I thought CPIO -could- do xattrs in the current versions...)

It doesn't preserve xattrs the way we currently invoke it and I can't see anything about xattrs in the man pages on Ubuntu or Fedora (which both have different man pages for cpio)

I'm going to submit a v2 of this series that:

a) adds a test to ensure files with spaces and other special characters are correctly copied.
b) adds a test to ensure that xattr on a file are preserved.

Regards,

Joshua

Quoted 16 lines

--Mark
Quoted 12 lines
+        if os.path.isdir(src):
+            import glob
+            if len(glob.glob('%s/.??*' % src)) > 0:
+                src = src + '/.??* '
+            src = src + '/*'
+        cmd = 'cp -afl --preserve=xattr %s %s' % (src, dst) subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
         copytree(src, dst)

Richard Purdie

2016-08-30 12:19 UTC #6 lore raw

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

In reply to Mark Hatle

Quoted 10 lines · Mark Hatle

The reason why cpio was originally used had to do with filenames that had spaces
or other special characters in them. (this is why the -print0 was used)
I don't object to the change, but we need to make sure that that will still work
properly.
(Also I thought CPIO -could- do xattrs in the current versions...)

FWIW cpio itself doesn't appear to support xattrs, nor is it likely to any time soon. It seems people had to use bsdcpio to get that support...

Cheers,

Richard

Nicolas Dechesne

2016-09-06 15:22 UTC #7 lore raw

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

In reply to Joshua Lock

Quoted 21 lines · Joshua Lock

meta/lib/oe/path.py +2 -2

index 3c07df3..631c3b4 100644
@@ -65,7 +65,7 @@ def copytree(src, dst):
     # This way we also preserve hardlinks between files in the tree.
 
     bb.utils.mkdirhier(dst)
-    cmd = 'tar -cf - -C %s -p . | tar -xf - -C %s' % (src, dst)
+    cmd = "tar --xattrs --xattrs-include='*' -cf - -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
     subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
 
 def copyhardlinktree(src, dst):

@@ -77,9 +77,14 @@ def copyhardlinktree(src, dst):
     if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
         # Need to copy directories only with tar first since cp will error if two
         # writers try and create a directory at the same time
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --no-recursion --files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)

meta/lib/oe/path.py def copyhardlinktree(src, dst): · line 90

Quoted 10 lines

meta/lib/oe/path.py +6 -1

-        cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' % (src, dst)
+        if os.path.isdir(src):
+            import glob
+            if len(glob.glob('%s/.??*' % src)) > 0:
+                src = src + '/.??* '
+            src = src + '/*'
+        cmd = 'cp -afl --preserve=xattr %s %s' % (src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
         copytree(src, dst)

Now that this got merged, I am seeing the following error:

==
Exception: subprocess.CalledProcessError: Command 'cp -afl
--preserve=xattr /home/nicolas.dechesne/work/oe-rpb-master/build-rpb/tmp-rpb-glibc/work/x86_64-linux/quilt-native/0.64-r0/license-destdir/* /home/nicolas.dechesne/work/oe-rpb-master/build-rpb/tmp-rpb-glibc/deploy/licenses'
returned non-zero exit status 1

Subprocess output:
cp: cannot preserve extended attributes, cp is built without xattr support ==

What happens is that I am installing coreutils in my images, and i have a copy of 'cp' in my sysroot *and* I don't have xattrs in DISTRO_FEATURES. As a consequence the 'cp' command which is invoked does not have xattrs support, while my host 'cp' command has it...

e.g.

~/work/oe-rpb-master/build-rpb$ tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp -afl --preserve=xattr foo bar
tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp: cannot preserve extended attributes, cp is built without xattr support

while this one works fine:

~/work/oe-rpb-master/build-rpb$ cp -afl --preserve=xattr foo bar

Should we revert these patches or am I missing anything?

thanks

Richard Purdie

2016-09-06 15:26 UTC #8 lore raw

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

Quoted 13 lines · Nicolas Dechesne

What happens is that I am installing coreutils in my images, and i have a copy of 'cp' in my sysroot *and* I don't have xattrs in DISTRO_FEATURES. As a consequence the 'cp' command which is invoked does not have xattrs support, while my host 'cp' command has it...
e.g.
~/work/oe-rpb-master/build-rpb$ tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp -afl --preserve=xattr foo bar
tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp: cannot preserve extended attributes, cp is built without xattr support

Quoted 5 lines

while this one works fine:
~/work/oe-rpb-master/build-rpb$ cp -afl --preserve=xattr foo bar
Should we revert these patches or am I missing anything?

Did you have:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=92fc3ef9738165300a4daed4c90526dbbad43149

Cheers,

Richard

Nicolas Dechesne

2016-09-06 15:34 UTC #9 lore raw

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

Quoted 16 lines · Richard Purdie

Quoted 15 lines · Nicolas Dechesne
What happens is that I am installing coreutils in my images, and i have a copy of 'cp' in my sysroot *and* I don't have xattrs in DISTRO_FEATURES. As a consequence the 'cp' command which is invoked does not have xattrs support, while my host 'cp' command has it...
e.g.
~/work/oe-rpb-master/build-rpb$ tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp -afl --preserve=xattr foo bar
tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp: cannot preserve extended attributes, cp is built without xattr support
while this one works fine:

Quoted 7 lines

Quoted 3 lines
~/work/oe-rpb-master/build-rpb$ cp -afl --preserve=xattr foo bar
Should we revert these patches or am I missing anything?
Did you have:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=92fc3ef9738165300a4daed4c90526dbbad43149

right. I am seeing it. The ERROR above happens when building quilt-native , after I updated my layers. So it's probably a race conditition. we need to rebuild quilt-native, so that we can rebuild coreutils with the change you mention.. but with the --preserve=xattr patch we can't rebuild quilt-native ;-)

so, i have removed coreutils from my sysroot and it is working now.. but we clearly have an 'upgrade' issue.. i am not going to be the only one..