More fixes for a distro with a merged /usr

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one of the variables.

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where /lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Quoted 19 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one of the variables.

Quoted 8 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where /lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

There seem to be some intresting ideas going around about what can or should be done via fs-perms.txt... AFAICT, fs-perms.txt can't move around files, so moving files form /lib to /usr/lib must be done in the package recipes themselves. (In my opinion, fs-perms.txt is a bad hack for broken recipes that shouldn't exist anyways, but that's another discussion)

I think if a distro config changes any of the base paths ({nonarch_,}base_libdir, base_{,s}bindir), *all* packages should respect this. It's the distro's reponsiblity to create symlinks so everything is found again at the expected paths (other examples for such hardcoded
paths: /bin/sh; the dynamic linker). See also my patchset I submitted to this mailing list, which introduces a distro feature to have such symlinks created by base-files.

Matthias

Quoted 29 lines · Matthias Schiffer

Quoted 19 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one of the variables.

Quoted 8 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where /lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Quoted 6 lines

There seem to be some intresting ideas going around about what can or should be done via fs-perms.txt... AFAICT, fs-perms.txt can't move around files, so moving files form /lib to /usr/lib must be done in the package recipes themselves. (In my opinion, fs-perms.txt is a bad hack for broken recipes that shouldn't exist anyways, but that's another discussion)

Since I wrote fs-perms.txt, I'll explain the purpose. Individual packages don't know if something is a directory, symbolic link, or what owner/group/permissions a system level directory should be set to.

The entire purpose of it is to declare a common set of -system- directories. (Packages/layers can amend and override this as necessary to add their own system directories.)

FYI System directories are things like /usr/bin. Having every package in the system need to define /usr/bin as a directory with an owner/group of root:root and permission of 0755 is a REALLY bad practice.. but putting this knowledge into a single file that synchronizes everything is very practical.

When the system level directories are mapped to symlinks.. the case where everyone is trying to folks /usr -> / or /bin -> /usr/bin.. then it can AUTOMATICALLY map and move the files in these places..

Quoted 8 lines

I think if a distro config changes any of the base paths ({nonarch_,}base_libdir, base_{,s}bindir), *all* packages should respect this. It's the distro's reponsiblity to create symlinks so everything is found again at the expected paths (other examples for such hardcoded
paths: /bin/sh; the dynamic linker). See also my patchset I submitted to this mailing list, which introduces a distro feature to have such symlinks created by base-files.

When this was written it was heavily argued against this knowledge being in base-files or base-dirs (suggested at the time) packages.

Defining a base setup, and then using a synchronization pass using the fs-perms.txt was the way to go.

Note, fs-perms process is absolutely supposed to move files around -if- a symlink is generated.. i.e.:

/lib -> /usr/lib

if you write to /lib/firmware, the code is supposed to see the directory of '/lib', create a new /usr/lib (set perms properly) and move the contents if /lib to /usr/lib, then replace the directory with a symbolic link.

If it's NOT doing that, lets fix it.

--Mark

Quoted 4 lines

Matthias

Quoted 50 lines · Mark Hatle

Quoted 29 lines · Matthias Schiffer

Quoted 19 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one of the variables.

Quoted 8 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where /lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Quoted 6 lines

There seem to be some intresting ideas going around about what can or should be done via fs-perms.txt... AFAICT, fs-perms.txt can't move around files, so moving files form /lib to /usr/lib must be done in the package recipes themselves. (In my opinion, fs-perms.txt is a bad hack for broken recipes that shouldn't exist anyways, but that's another discussion)

Since I wrote fs-perms.txt, I'll explain the purpose. Individual packages don't know if something is a directory, symbolic link, or what owner/group/permissions a system level directory should be set to.

The entire purpose of it is to declare a common set of -system- directories. (Packages/layers can amend and override this as necessary to add their own system directories.)

FYI System directories are things like /usr/bin. Having every package in the system need to define /usr/bin as a directory with an owner/group of root:root and permission of 0755 is a REALLY bad practice.. but putting this knowledge into a single file that synchronizes everything is very practical.

Quoted 3 lines

When the system level directories are mapped to symlinks.. the case where everyone is trying to folks /usr -> / or /bin -> /usr/bin.. then it can AUTOMATICALLY map and move the files in these places..

Thanks for the explanation, I asked a similar question a few weeks ago and didn't get a satisfactory answer about what fs-perms can do. I'll rethink my approach for the merged-usr patchset.

So the remaining issue is how to conditionalize this. I'd like to find a solution which doesn't require distros to define their own fs-perms when they just want to use a merged /usr dir.

Quoted 9 lines

Quoted 8 lines

I think if a distro config changes any of the base paths ({nonarch_,}base_libdir, base_{,s}bindir), *all* packages should respect this. It's the distro's reponsiblity to create symlinks so everything is found again at the expected paths (other examples for such hardcoded
paths: /bin/sh; the dynamic linker). See also my patchset I submitted to this mailing list, which introduces a distro feature to have such symlinks created by base-files.

Quoted 2 lines

When this was written it was heavily argued against this knowledge being in base-files or base-dirs (suggested at the time) packages.

Is that discussion archived somewhere? I'm interested in the argumentation. Do any non-OE distros have a similar feature?

Quoted 9 lines

Defining a base setup, and then using a synchronization pass using the fs-perms.txt was the way to go.

Note, fs-perms process is absolutely supposed to move files around -if- a symlink is generated.. i.e.:

/lib -> /usr/lib

Quoted 5 lines

if you write to /lib/firmware, the code is supposed to see the directory of '/lib', create a new /usr/lib (set perms properly) and move the contents if /lib to /usr/lib, then replace the directory with a symbolic link.

If it's NOT doing that, lets fix it.

I didn't try yet as I didn't now that it is supposed to to that.

Quoted 8 lines

--Mark

Quoted 4 lines

Matthias

Quoted 51 lines · Matthias Schiffer

Quoted 50 lines · Mark Hatle

Quoted 29 lines · Matthias Schiffer

Quoted 19 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one of the variables.

Quoted 8 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where /lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Quoted 6 lines

There seem to be some intresting ideas going around about what can or should be done via fs-perms.txt... AFAICT, fs-perms.txt can't move around files, so moving files form /lib to /usr/lib must be done in the package recipes themselves. (In my opinion, fs-perms.txt is a bad hack for broken recipes that shouldn't exist anyways, but that's another discussion)

Since I wrote fs-perms.txt, I'll explain the purpose. Individual packages don't know if something is a directory, symbolic link, or what owner/group/permissions a system level directory should be set to.

The entire purpose of it is to declare a common set of -system- directories. (Packages/layers can amend and override this as necessary to add their own system directories.)

FYI System directories are things like /usr/bin. Having every package in the system need to define /usr/bin as a directory with an owner/group of root:root and permission of 0755 is a REALLY bad practice.. but putting this knowledge into a single file that synchronizes everything is very practical.

Quoted 7 lines

Quoted 3 lines

When the system level directories are mapped to symlinks.. the case where everyone is trying to folks /usr -> / or /bin -> /usr/bin.. then it can AUTOMATICALLY map and move the files in these places..

Thanks for the explanation, I asked a similar question a few weeks ago and didn't get a satisfactory answer about what fs-perms can do. I'll rethink my approach for the merged-usr patchset.

(I've been out since near the beginning of December, but I'm back now..)

Quoted 3 lines

So the remaining issue is how to conditionalize this. I'd like to find a solution which doesn't require distros to define their own fs-perms when they just want to use a merged /usr dir.

You conditionalize it by providing different default fs-perms files -- OR since more then one file can be loaded -- additional fs-perms with the permutations you desire.

I could easily see having an files/fs-perms-usr-only.txt and an files/fs-perms-no-usr.txt

Where the first would be something like:

# Define symlinks from base directories to their prefix versions
/bin link ${bindir}
/sbin link ${sbindir}
...

fs-perms-no-user would be:

# Define that $prefix as simply pointing back to root for compatibility
/usr link /

Then in your distro.conf file:

FILESYSTEM_PERMS_TABLES = "files/fs-perms.txt" FILESYSTEM_PERMS_TABLES_append_usronly = " files/fs-perms-usr-only.txt" FILESYSTEM_PERMS_TABLES_append_nousr = " files/fs-perms-no-user.txt"

The above will cause packages to produce the symlinks ONLY if the package itself creates a matching directory. If the package is moving all of it's files to the new configured location (from the distro.conf file) then no link is present.

Assuming you want the links, I would add some kind of a change or configuration to the base-files to do this.

(I did argue in the past, unsuccessfully, that base-files or something it calls should process the fs-perms files and simply generate a base configuration based on this setup. Ensuring symlinks and final directories were always created.. but that was rejected at the time..)

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

I think if a distro config changes any of the base paths ({nonarch_,}base_libdir, base_{,s}bindir), *all* packages should respect this. It's the distro's reponsiblity to create symlinks so everything is found again at the expected paths (other examples for such hardcoded
paths: /bin/sh; the dynamic linker). See also my patchset I submitted to this mailing list, which introduces a distro feature to have such symlinks created by base-files.

Quoted 5 lines

Quoted 2 lines

When this was written it was heavily argued against this knowledge being in base-files or base-dirs (suggested at the time) packages.

Is that discussion archived somewhere? I'm interested in the argumentation. Do any non-OE distros have a similar feature?

This would have been discussed back in the original oe-core days. Likely around 2010 on the oe-core list....

Quoted 16 lines

Quoted 15 lines

Defining a base setup, and then using a synchronization pass using the fs-perms.txt was the way to go.

Note, fs-perms process is absolutely supposed to move files around -if- a symlink is generated.. i.e.:

/lib -> /usr/lib

if you write to /lib/firmware, the code is supposed to see the directory of '/lib', create a new /usr/lib (set perms properly) and move the contents if /lib to /usr/lib, then replace the directory with a symbolic link.

If it's NOT doing that, lets fix it.

Quoted 1 line

I didn't try yet as I didn't now that it is supposed to to that.

This is a good example where the upstream software is always going to use '/lib', but you want it to go somewhere else. Without this fs-perms mechanism, you would need to patch this package and potentially others. But by using fs-perms, they can continue to rely on their special knowledge of '/lib', and the system will fix it up before packaging to where the distribution actually wants the files.

Quoted 100 lines · Mark Hatle

Quoted 51 lines · Matthias Schiffer

Quoted 50 lines · Mark Hatle

Quoted 29 lines · Matthias Schiffer

Quoted 19 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one of the variables.

Quoted 8 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where /lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Quoted 6 lines

There seem to be some intresting ideas going around about what can or should be done via fs-perms.txt... AFAICT, fs-perms.txt can't move around files, so moving files form /lib to /usr/lib must be done in the package recipes themselves. (In my opinion, fs-perms.txt is a bad hack for broken recipes that shouldn't exist anyways, but that's another discussion)

Since I wrote fs-perms.txt, I'll explain the purpose. Individual packages don't know if something is a directory, symbolic link, or what owner/group/permissions a system level directory should be set to.

The entire purpose of it is to declare a common set of -system- directories. (Packages/layers can amend and override this as necessary to add their own system directories.)

FYI System directories are things like /usr/bin. Having every package in the system need to define /usr/bin as a directory with an owner/group of root:root and permission of 0755 is a REALLY bad practice.. but putting this knowledge into a single file that synchronizes everything is very practical.

Quoted 7 lines

Quoted 3 lines

When the system level directories are mapped to symlinks.. the case where everyone is trying to folks /usr -> / or /bin -> /usr/bin.. then it can AUTOMATICALLY map and move the files in these places..

Thanks for the explanation, I asked a similar question a few weeks ago and didn't get a satisfactory answer about what fs-perms can do. I'll rethink my approach for the merged-usr patchset.

(I've been out since near the beginning of December, but I'm back now..)

Quoted 3 lines

So the remaining issue is how to conditionalize this. I'd like to find a solution which doesn't require distros to define their own fs-perms when they just want to use a merged /usr dir.

You conditionalize it by providing different default fs-perms files -- OR since more then one file can be loaded -- additional fs-perms with the permutations you desire.

I could easily see having an files/fs-perms-usr-only.txt and an files/fs-perms-no-usr.txt

Where the first would be something like:

# Define symlinks from base directories to their prefix versions
/bin link ${bindir}
/sbin link ${sbindir}
...

fs-perms-no-user would be:

# Define that $prefix as simply pointing back to root for compatibility
/usr link /

Then in your distro.conf file:

FILESYSTEM_PERMS_TABLES = "files/fs-perms.txt" FILESYSTEM_PERMS_TABLES_append_usronly = " files/fs-perms-usr-only.txt" FILESYSTEM_PERMS_TABLES_append_nousr = " files/fs-perms-no-user.txt"

The above will cause packages to produce the symlinks ONLY if the package itself creates a matching directory. If the package is moving all of it's files to the new configured location (from the distro.conf file) then no link is present.

Assuming you want the links, I would add some kind of a change or configuration to the base-files to do this.

Quoted 4 lines

(I did argue in the past, unsuccessfully, that base-files or something it calls should process the fs-perms files and simply generate a base configuration based on this setup. Ensuring symlinks and final directories were always created.. but that was rejected at the time..)

I see. Do you think it would make sense to provide a fs-perms snippet in oe-core for the Fedora-style merged-usr setup?

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

I think if a distro config changes any of the base paths ({nonarch_,}base_libdir, base_{,s}bindir), *all* packages should respect this. It's the distro's reponsiblity to create symlinks so everything is found again at the expected paths (other examples for such hardcoded
paths: /bin/sh; the dynamic linker). See also my patchset I submitted to this mailing list, which introduces a distro feature to have such symlinks created by base-files.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

When this was written it was heavily argued against this knowledge being in base-files or base-dirs (suggested at the time) packages.

Is that discussion archived somewhere? I'm interested in the argumentation. Do any non-OE distros have a similar feature?

This would have been discussed back in the original oe-core days. Likely around 2010 on the oe-core list....

Okay, I've found at least some fragments of the discussion.

Quoted 16 lines

Quoted 16 lines

Quoted 15 lines

Defining a base setup, and then using a synchronization pass using the fs-perms.txt was the way to go.

Note, fs-perms process is absolutely supposed to move files around -if- a symlink is generated.. i.e.:

/lib -> /usr/lib

if you write to /lib/firmware, the code is supposed to see the directory of '/lib', create a new /usr/lib (set perms properly) and move the contents if /lib to /usr/lib, then replace the directory with a symbolic link.

If it's NOT doing that, lets fix it.

Quoted 9 lines

Quoted 1 line

I didn't try yet as I didn't now that it is supposed to to that.

This is a good example where the upstream software is always going to use '/lib', but you want it to go somewhere else. Without this fs-perms mechanism, you would need to patch this package and potentially others. But by using fs-perms, they can continue to rely on their special knowledge of '/lib', and the system will fix it up before packaging to where the distribution actually wants the files.

I guess I'm fine with fs-perms.txt fixing permissions and owners, but moving around files goes a bit too far in my opinion. Doing so will potentially break relative symlinks and other relative paths used by packages. I'd have implemented the "link" feature as a QA-only thing:
make the build fail if there are any files where a symlink is supposed to go, but don't try to fix it up.

Another (more easily fixable) issue is that the automagic renaming doesn't work if the target dir already exists (if I understand the code correctly). So fixing up a package containing both files in /bin and /usr/bin, where /bin is supposed to link to /usr/bin, will fail.

Matthias

Quoted 23 lines · Mark Hatle

Quoted 22 lines · Matthias Schiffer

Quoted 21 lines · Mark Hatle

Quoted 20 lines · Matthias Schiffer

Quoted 19 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one of the variables.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where /lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Quoted 7 lines

You conditionalize it by providing different default fs-perms files -- OR since more then one file can be loaded -- additional fs-perms with the permutations you desire.

I could easily see having an files/fs-perms-usr-only.txt and an files/fs-perms-no-usr.txt

Quoted 6 lines

Where the first would be something like:

# Define symlinks from base directories to their prefix versions
/bin link ${bindir}
/sbin link ${sbindir}
...

Quoted 24 lines · Matthias Schiffer

Quoted 23 lines · Mark Hatle

Quoted 22 lines · Matthias Schiffer

Quoted 21 lines · Mark Hatle

Quoted 20 lines · Matthias Schiffer

Quoted 19 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one of the variables.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where /lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Ah, that makes sense.

<snip>

Quoted 7 lines

Quoted 7 lines

You conditionalize it by providing different default fs-perms files -- OR since more then one file can be loaded -- additional fs-perms with the permutations you desire.

I could easily see having an files/fs-perms-usr-only.txt and an files/fs-perms-no-usr.txt

Quoted 6 lines

Quoted 6 lines

Where the first would be something like:

# Define symlinks from base directories to their prefix versions
/bin link ${bindir}
/sbin link ${sbindir}
...

This is very interesting -- thank you.

I tried patching our additional fs-perms file as Mark suggested. This resulted in a build failure during packaging of a recipe. The package directory looks odd:

ls -l /home/ian/myproduct/build/tmp/work/myproduct-linux-gnueabi/ myrecipe/4.3-r0/package
total 4

drwxr-xr-x. 2 ian ian 4096 Jan  5 11:30 boot
drwxr-xr-x. 4 ian ian   56 Jan  5 11:30 etc
lrwxrwxrwx. 1 ian ian    8 Jan  5 11:33 lib -> /usr/lib
drwxr-xr-x. 3 ian ian   16 Jan  5 11:33 usr

This can be fixed by teaching classes/package.bbclass to _not_ use the leading slash when making the link. Not sure if that is unacceptably hacky? But it must be better than leaking to host filesystem.

meta/classes/package.bbclass +0 -0

index a86b680..59ef447 100644

Once past that problem there is a QA problem since do_package now installs to /usr/lib (because of fs-perms) instead of /lib which is mentioned in FILES.

Joshua's patches for merged /usr touched FILES in _some_ cases (for example 73a6fe958f47642d18ba0098cfd45c3520d53560) but linux-firmware would require considerably more effort. Is that an argument in favour of nonarch_base_libdir instead of literal "/lib" or is there some other technique to address this?

Ian

Quoted 58 lines · Ian (GE Healthcare)

Quoted 25 lines · Matthias Schiffer

Quoted 24 lines · Mark Hatle

Quoted 23 lines · Matthias Schiffer

Quoted 22 lines · Mark Hatle

Quoted 21 lines · Matthias Schiffer

Quoted 20 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one
of the variables.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where
/lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Ah, that makes sense.

<snip>

Quoted 7 lines

Quoted 7 lines

You conditionalize it by providing different default fs-perms files -- OR since more then one file can be loaded -- additional fs-perms with the permutations you desire.

I could easily see having an files/fs-perms-usr-only.txt and an files/fs-perms-no-usr.txt

Quoted 6 lines

Quoted 6 lines

Where the first would be something like:

# Define symlinks from base directories to their prefix versions
/bin link ${bindir}
/sbin link ${sbindir}
...

This is very interesting -- thank you.

I tried patching our additional fs-perms file as Mark suggested. This resulted in a build failure during packaging of a recipe. The package directory looks odd:

Quoted 7 lines

ls -l /home/ian/myproduct/build/tmp/work/myproduct-linux-gnueabi/ myrecipe/4.3-r0/package
total 4

drwxr-xr-x. 2 ian ian 4096 Jan  5 11:30 boot
drwxr-xr-x. 4 ian ian   56 Jan  5 11:30 etc
lrwxrwxrwx. 1 ian ian    8 Jan  5 11:33 lib -> /usr/lib
drwxr-xr-x. 3 ian ian   16 Jan  5 11:33 usr

We should look into this further. Pseudo (the fakeroot wrapper) should be changing '/usr/lib' [in the cross build filesystem] to a relative path... It will only be '/usr/lib' in the target version if we're in the pseudo environment or the translation is off. (Maybe I'm remembering incorrectly, but that was my memory.)

Quoted 9 lines

This can be fixed by teaching classes/package.bbclass to _not_ use the leading slash when making the link. Not sure if that is unacceptably hacky? But it must be better than leaking to host filesystem.

meta/classes/package.bbclass +0 -0

index a86b680..59ef447 100644

meta/classes/package.bbclass python fixup_perms () { · line 780

Quoted 7 lines

meta/classes/package.bbclass +1 -0

@@ -774,6 +774,7 @@ python fixup_perms () {
         link = fs_perms_table[dir].link
         if link[0] == "/":
             target = dvar + link
+            link = link[1:]
             ptarget = link
         else:
             target = os.path.join(os.path.dirname(origin), link)

I'm definitely interested in understanding what may or may not be working properly here.

Quoted 2 lines

Once past that problem there is a QA problem since do_package now installs to /usr/lib (because of fs-perms) instead of /lib which is mentioned in FILES.

What was the QA issue? About the absolute path name or something else?

Quoted 4 lines

Joshua's patches for merged /usr touched FILES in _some_ cases (for example 73a6fe958f47642d18ba0098cfd45c3520d53560) but linux-firmware would require considerably more effort. Is that an argument in favour of nonarch_base_libdir instead of literal "/lib" or is there some other technique to address this?

For linux-firmware specifically.. we need to make sure the kernel and modules (which use the firmware) use the same filesystem notion as what the linux-firmware package is doing. In the past when I've looked, pretty much everything was just hardcoded as '/lib/firmware'. So using that in the packaging was correct for all three systems. (I don't believe patching the three to use bitbake variables is necessarily a good idea -- since I'm sure there are more instances of this type of usage that I'm unaware of.)

This is where using the fs-perms mechanism to move the files and setup the symlink is the right approach in my opinion. The key thing is figuring out the permutations of the libdir, nonarch libdir (since they may or may not be the same) and dealing with any QA errors that may occur.

Quoted 93 lines · Mark Hatle

Quoted 58 lines · Ian (GE Healthcare)

Quoted 25 lines · Matthias Schiffer

Quoted 24 lines · Mark Hatle

Quoted 23 lines · Matthias Schiffer

Quoted 22 lines · Mark Hatle

Quoted 21 lines · Matthias Schiffer

Quoted 20 lines · Mark Hatle

Quoted 11 lines · Matthias Schiffer

Quoted 6 lines · Ian Ray

The recipe uses hard-coded paths (specifically /lib) in do_install and in FILES, however on a merged /usr system this directory might not exist. Prefer base_libdir.

Signed-off-by Ian Ray <[email protected]>

---

This should use nonarch_base_libdir, base_libdir defaults to /lib64 on ppc64, which is not where the firmware is expected.

At a minimum, I would agree nonarch_base_libdir, however..

I believe that the kernel loader/modules/tools themselves actually have '/lib' hard coded into them. This is the reason why /lib/firmware was used and not one
of the variables.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

This is one of the cases were /lib is actually correct, since that is what the system is expecting. We can make some kind of accommodation for systems where
/lib -> /usr/lib... but that should be done inside of the filesystem setup processing and not the package itself. (I'm referring to the 'meta/files/fs-perms.txt' file.

--Mark

Ah, that makes sense.

<snip>

Quoted 7 lines

Quoted 7 lines

You conditionalize it by providing different default fs-perms files -- OR since more then one file can be loaded -- additional fs-perms with the permutations you desire.

I could easily see having an files/fs-perms-usr-only.txt and an files/fs-perms-no-usr.txt

Quoted 6 lines

Quoted 6 lines

Where the first would be something like:

# Define symlinks from base directories to their prefix versions
/bin link ${bindir}
/sbin link ${sbindir}
...

This is very interesting -- thank you.

I tried patching our additional fs-perms file as Mark suggested. This resulted in a build failure during packaging of a recipe. The package directory looks odd:

Quoted 7 lines

ls -l /home/ian/myproduct/build/tmp/work/myproduct-linux-gnueabi/ myrecipe/4.3-r0/package
total 4

drwxr-xr-x. 2 ian ian 4096 Jan  5 11:30 boot
drwxr-xr-x. 4 ian ian   56 Jan  5 11:30 etc
lrwxrwxrwx. 1 ian ian    8 Jan  5 11:33 lib -> /usr/lib
drwxr-xr-x. 3 ian ian   16 Jan  5 11:33 usr

We should look into this further. Pseudo (the fakeroot wrapper) should be changing '/usr/lib' [in the cross build filesystem] to a relative path... It will only be '/usr/lib' in the target version if we're in the pseudo environment or the translation is off. (Maybe I'm remembering incorrectly, but that was my memory.)

Quoted 9 lines

This can be fixed by teaching classes/package.bbclass to _not_ use the leading slash when making the link. Not sure if that is unacceptably hacky? But it must be better than leaking to host filesystem.

meta/classes/package.bbclass +0 -0

index a86b680..59ef447 100644

meta/classes/package.bbclass python fixup_perms () { · line 780

Quoted 7 lines

meta/classes/package.bbclass +1 -0

@@ -774,6 +774,7 @@ python fixup_perms () {
         link = fs_perms_table[dir].link
         if link[0] == "/":
             target = dvar + link
+            link = link[1:]
             ptarget = link
         else:
             target = os.path.join(os.path.dirname(origin), link)

I'm definitely interested in understanding what may or may not be working properly here.

Quoted 4 lines

Quoted 2 lines

Once past that problem there is a QA problem since do_package now installs to /usr/lib (because of fs-perms) instead of /lib which is mentioned in FILES.

What was the QA issue? About the absolute path name or something else?

ERROR: QA Issue: linux-advantech: Files/directories were installed but not shipped in any package:
/usr
/usr/lib
/usr/lib/modules
/usr/lib/modules/4.4.0-rc3-next-20151202 /usr/lib/modules/4.4.0-rc3-next-20151202/modules.order /usr/lib/modules/4.4.0-rc3-next-20151202/modules.builtin
...

Quoted 13 lines

Quoted 4 lines

Joshua's patches for merged /usr touched FILES in _some_ cases (for example 73a6fe958f47642d18ba0098cfd45c3520d53560) but linux-firmware would require considerably more effort. Is that an argument in favour of nonarch_base_libdir instead of literal "/lib" or is there some other technique to address this?

For linux-firmware specifically.. we need to make sure the kernel and modules (which use the firmware) use the same filesystem notion as what the linux-firmware package is doing. In the past when I've looked, pretty much everything was just hardcoded as '/lib/firmware'. So using that in the packaging was correct for all three systems. (I don't believe patching the three to use bitbake variables is necessarily a good idea -- since I'm sure there are more instances of this type of usage that I'm unaware of.)

Quoted 4 lines

This is where using the fs-perms mechanism to move the files and setup the symlink is the right approach in my opinion. The key thing is figuring out the permutations of the libdir, nonarch libdir (since they may or may not be the same) and dealing with any QA errors that may occur.

This makes sense, but I'm struggling to understand how FILES is affected, when it lists /lib/foo but we install to /usr/lib/foo.

This is a situation where we've got hard coded '/lib' entries in FILES, but we're trying to move the directory elsewhere.... 'adjusting' FILES automatically is probably more error prone and complicated then desired.. but I'm not sure switching to nonarch_baselib_dir is right either.. argh. (But it may certainly be the less evil in this case.)