preserve xattrs in images

Installing "tar" into the sysroot leads to race conditions (tasks which do not depend on tar-replacement-native may already call tar while it's installation is incomplete). Avoid those by installing only the tar binary under the name tar-native.

Quoted 4 lines · Patrick Ohly

Installing "tar" into the sysroot leads to race conditions (tasks which do not depend on tar-replacement-native may already call tar while it's installation is incomplete). Avoid those by installing only the tar binary under the name tar-native.

So the point of tar-replacement-native was that it's a *replacement* for the host tar, so it needs the same name. If it was to be installed as an opt-in alternative there's a mechanism for doing that already (see EXTRANATIVEPATH) but that would break the purpose of tar-replacement-native.

I think for this situation you should show how to enable xattrs for images, with the caveat that if your host tar isn't suitable the user should bitbake just tar-replacement-native *first*.

Or install the buildtools perhaps.

On 14 August 2015 at 12:01, Paul Eggleton
<[email protected]> wrote:
Or install the buildtools perhaps.

Yes, I had that in my head but someone failed to write it down. There's a fair chance that a host with a sufficiently old tar also needs other updates, and that's when buildtools is used (which includes tar).

Quoted 5 lines · Ross

On 14 August 2015 at 12:01, Paul Eggleton

<[email protected]> wrote:
Or install the buildtools perhaps.

Quoted 4 lines

Yes, I had that in my head but someone failed to write it down. There's a fair chance that a host with a sufficiently old tar also needs other updates, and that's when buildtools is used (which includes tar).

The hosts we talk about (Debian 7, CentOS 6) work fine for normal usage without having to install buildtools.

Quoted 11 lines · On Friday 14 August 2015 16:52:08 Patrick Ohly wrote:

Quoted 9 lines · Ross

On 14 August 2015 at 12:01, Paul Eggleton

<[email protected]> wrote:
Or install the buildtools perhaps.

Yes, I had that in my head but someone failed to write it down. There's a fair chance that a host with a sufficiently old tar also needs other updates, and that's when buildtools is used (which includes tar).

Quoted 2 lines

The hosts we talk about (Debian 7, CentOS 6) work fine for normal usage without having to install buildtools.

Really? You must have added *something* on top of CentOS 6 because last I checked its git, tar and python versions supplied out of the box were way too old to even get the build to start.

Quoted 15 lines · Paul Eggleton

Quoted 13 lines · On Friday 14 August 2015 16:52:08 Patrick Ohly wrote:

Quoted 9 lines · Ross

On 14 August 2015 at 12:01, Paul Eggleton

<[email protected]> wrote:
Or install the buildtools perhaps.

Yes, I had that in my head but someone failed to write it down. There's a fair chance that a host with a sufficiently old tar also needs other updates, and that's when buildtools is used (which includes tar).

The hosts we talk about (Debian 7, CentOS 6) work fine for normal usage without having to install buildtools.

Quoted 3 lines

Really? You must have added *something* on top of CentOS 6 because last I checked its git, tar and python versions supplied out of the box were way too old to even get the build to start.

I thought someone had said that CentOS is okay out-of-the-box, but I may have misunderstood.

EXTRAINSTALL = "do_install_extra"
EXTRAINSTALL_class-nativesdk = ""
+EXTRAINSTALL_class-native = ""

-EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'" +EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include' 'BUILDDIR=${S}'"

On 11 August 2015 at 09:45, Patrick Ohly <[email protected]> wrote:

        -EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}'
        'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=
        ${S}'"
        +EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}'
        'CFLAGS=${CFLAGS} -I${S}/include' 'BUILDDIR=${S}'"

There's a "xattr" DISTRO_FEATURE that should be respected here for target builds (and explicitly enabled for native I guess).

Does enabling xattr mean adding build dependencies?

Quoted 2 lines

There's a "xattr" DISTRO_FEATURE that should be respected here for target builds (and explicitly enabled for native I guess).

I can add that check, if you want.

Quoted 1 line

Does enabling xattr mean adding build dependencies?

No, the conditional code uses lgetxattr and llistxattr, which are in glibc.

On 12 August 2015 at 10:33, Patrick Ohly <[email protected]> wrote:
> There's a "xattr" DISTRO_FEATURE that should be respected here for
> target builds (and explicitly enabled for native I guess).

I can add that check, if you want.

> Does enabling xattr mean adding build dependencies?

No, the conditional code uses lgetxattr and llistxattr, which are in
glibc.

From five seconds of google it looks like these are in musl and uclibc too, so I've merged this to MUT as-is.

Quoted 11 lines · Ross

On 12 August 2015 at 10:33, Patrick Ohly <[email protected]> wrote:
> There's a "xattr" DISTRO_FEATURE that should be respected here for
> target builds (and explicitly enabled for native I guess).

I can add that check, if you want.

> Does enabling xattr mean adding build dependencies?

Quoted 7 lines

No, the conditional code uses lgetxattr and llistxattr, which are in
glibc.

From five seconds of google it looks like these are in musl and uclibc too, so I've merged this to MUT as-is.

However, it turned out that the XATTR code also introduces a dependency on sys/acl.h:

mkfs.jffs2.c:

#ifndef WITHOUT_XATTR
#include <sys/xattr.h>
#include <sys/acl.h>
#endif

sys/acl.h gets provided by acl, so it may or may not work depending on build order and files on the host.

I hadn't caught that because I had only looked at library dependencies, sorry for that.

What's the right fix? Add the dependency on acl-native in the native case, disable XATTR support for target builds (same state as before) or make it somehow depend on both "xattr" and "acl" distro features?

Or only check for "xattr"? I'm uncertain how the two are related.

Quoted 33 lines · Patrick Ohly

Quoted 11 lines · Ross

On 12 August 2015 at 10:33, Patrick Ohly <[email protected]> wrote:
> There's a "xattr" DISTRO_FEATURE that should be respected here for
> target builds (and explicitly enabled for native I guess).

I can add that check, if you want.

> Does enabling xattr mean adding build dependencies?

Quoted 7 lines

No, the conditional code uses lgetxattr and llistxattr, which are in
glibc.

From five seconds of google it looks like these are in musl and uclibc too, so I've merged this to MUT as-is.

However, it turned out that the XATTR code also introduces a dependency on sys/acl.h:

mkfs.jffs2.c:

#ifndef WITHOUT_XATTR
#include <sys/xattr.h>
#include <sys/acl.h>
#endif

sys/acl.h gets provided by acl, so it may or may not work depending on build order and files on the host.

Quoted 9 lines

I hadn't caught that because I had only looked at library dependencies, sorry for that.

What's the right fix? Add the dependency on acl-native in the native case, disable XATTR support for target builds (same state as before) or make it somehow depend on both "xattr" and "acl" distro features?

Or only check for "xattr"? I'm uncertain how the two are related.

Use PACKAGECONFIG, add xattr as a configuration that in turn adds the 'acl' as a
dependency. Then acl-native will automatically be converted from 'acl' if the
'-native' version is built.

This is typically how I've seen it implemented in other places. (Not necessary
xattr.. but with the push for more Linux Security Models coming from users and
other layers, I think xattr is something we need to make sure really works properly.)

--Mark
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 40 lines · Patrick Ohly

Quoted 11 lines · Ross

On 12 August 2015 at 10:33, Patrick Ohly <[email protected]> wrote:
> There's a "xattr" DISTRO_FEATURE that should be respected here for
> target builds (and explicitly enabled for native I guess).

I can add that check, if you want.

> Does enabling xattr mean adding build dependencies?

Quoted 7 lines

No, the conditional code uses lgetxattr and llistxattr, which are in
glibc.

From five seconds of google it looks like these are in musl and uclibc too, so I've merged this to MUT as-is.

However, it turned out that the XATTR code also introduces a dependency on sys/acl.h:

mkfs.jffs2.c:

#ifndef WITHOUT_XATTR
#include <sys/xattr.h>
#include <sys/acl.h>
#endif

sys/acl.h gets provided by acl, so it may or may not work depending on build order and files on the host.

I hadn't caught that because I had only looked at library dependencies, sorry for that.

What's the right fix? Add the dependency on acl-native in the native case, disable XATTR support for target builds (same state as before) or make it somehow depend on both "xattr" and "acl" distro features?

Quoted 2 lines

Or only check for "xattr"? I'm uncertain how the two are related.

Use PACKAGECONFIG, add xattr as a configuration that in turn adds the 'acl' as a dependency. Then acl-native will automatically be converted from 'acl' if the '-native' version is built.

Quoted 41 lines · Mark Hatle

Quoted 40 lines · Patrick Ohly

Quoted 11 lines · Ross

On 12 August 2015 at 10:33, Patrick Ohly <[email protected]> wrote:
> There's a "xattr" DISTRO_FEATURE that should be respected here for
> target builds (and explicitly enabled for native I guess).

I can add that check, if you want.

> Does enabling xattr mean adding build dependencies?

Quoted 7 lines

No, the conditional code uses lgetxattr and llistxattr, which are in
glibc.

From five seconds of google it looks like these are in musl and uclibc too, so I've merged this to MUT as-is.

However, it turned out that the XATTR code also introduces a dependency on sys/acl.h:

mkfs.jffs2.c:

#ifndef WITHOUT_XATTR
#include <sys/xattr.h>
#include <sys/acl.h>
#endif

sys/acl.h gets provided by acl, so it may or may not work depending on build order and files on the host.

I hadn't caught that because I had only looked at library dependencies, sorry for that.

What's the right fix? Add the dependency on acl-native in the native case, disable XATTR support for target builds (same state as before) or make it somehow depend on both "xattr" and "acl" distro features?

Quoted 6 lines

Quoted 2 lines

Or only check for "xattr"? I'm uncertain how the two are related.

Use PACKAGECONFIG, add xattr as a configuration that in turn adds the 'acl' as a dependency. Then acl-native will automatically be converted from 'acl' if the '-native' version is built.

I agree, a PACKAGECONFIG sounds like the right approach. However, what should the default depend on? The "xattr" or "acl" distro feature? What if someone wants xattr, but not acl? Should xattr support in mtd-utils be enabled in that case even though it pulls in the acl recipe as dependency?

The default does not get changed because supporting xattrs causes a certain overhead (need to build GNU tar, additional system calls when creating the images).

Hi Patrick,

On 11 August 2015 at 09:44, Patrick Ohly <[email protected]> wrote:
The default does not get changed because supporting xattrs causes a
certain overhead (need to build GNU tar, additional system calls when
creating the images).

Two questions:
1) Do enough host distributions not enable xattrs in tar that we need to depend on tar-replacement-native?

2) Have you timed the overhead of enabling xattr archiving on an image that doesn't use xattrs? (subtext: does this need to be an option).

Both Smack and IMA/EVM rely on xattrs in the rootfs. This works for .ext3/.ext4 images, but not for .jffs2 and .tar.bz2. These changes allow optionally building also such images with xattrs without changing the default (which still is to ignore xattrs in .jffs2 and .tar.bz2).

The default does not get changed because supporting xattrs causes a certain overhead (need to build GNU tar, additional system calls when creating the images).

See https://github.com/01org/meta-intel-iot-security/pull/34 for code using these changes.

The following changes since commit 5094354a2811825e6d60963f03959daa349cab23:

bind: upgrade to 9.10.2-p3 (2015-08-09 15:14:32 -0700)

are available in the git repository at:

git://github.com/pohly/openembedded-core xattr
https://github.com/pohly/openembedded-core/tree/xattr

Patrick Ohly (3):
tar-replacement-native: avoid race condition with host tar image_types.bbclass: allow replacing tar command
mtd-utils: keep xattr support enabled

 meta/classes/image_types.bbclass                         | 13 ++++++++++++-
 meta/recipes-devtools/mtd/mtd-utils_git.bb               |  2 +-
 meta/recipes-extended/tar/tar-replacement-native_1.28.bb | 14 +++++++++++++-
 3 files changed, 26 insertions(+), 3 deletions(-)

Is there something here that enables the tar-replacement-native? Or is the user just expected to know they need it?

If you look at meta/classes/sanity.bbclass, there is already a tar check for a specific old version that can't handle overwriting symlinks properly. It should be possible to check for the distro feature of xattr and that the host tar has this. It would be good to prevent the user from having a problem early (sanity) rather then at filesystem generation time.