package_rpm: Add optional improved directory handling

From: Ronan Le Martret <[email protected]>

During spec generation, ideally directories should not be auto packaged under the %file section of rpm packages but take ownership of specific directories.

* packages only empty directories or explict directory.
See:
- http://www.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html
- "The %dir Directive"

* This will prevent the overlapping of security permission.
For example, in Tizen the directory /etc have smack label 'System::Shared' So Only one package should own and set the label of /etc to prevent the overwriting of the smack label.

Existing behaviour is maintained if DIRFILES is not set. If it is set, the modified behaviour is used.

[RP: Modified to allow optional usage of DIRFILES]

Signed-off-by Ronan Le Martret <[email protected]>

Signed-off-by Richard Purdie <[email protected]>

meta/classes/package_rpm.bbclass +1 -1

index 0a32b3e..eecfcb2 100644
@@ -185,7 +185,7 @@ python write_specfile () {
                  if not len(depends_dict[dep]):
                      array.append("%s: %s" % (tag, dep))
 
-    def walk_files(walkpath, target, conffiles):
+    def walk_files(walkpath, target, conffiles, dirfiles):
          # We can race against the ipk/deb backends which create CONTROL or DEBIAN directories
          # when packaging. We just ignore these files which are created in
          # packages-split/ and not package/

@@ -196,11 +196,24 @@ python write_specfile () {
              path = rootpath.replace(walkpath, "")
              if path.endswith("DEBIAN") or path.endswith("CONTROL"):
                  continue

meta/classes/package_rpm.bbclass +13 -0

+                for dir in dirs:
+                    if dir == "CONTROL" or dir == "DEBIAN":
+                        continue
+                    # All packages own the directories their files are in...
+                    target.append('%dir "' + path + '/' + dir + '"')
+            else:
+                # packages own only empty directories or explict directory.
+                # This will prevent the overlapping of security permission.
+                if path and not files and not dirs:
+                    target.append('%dir "' + path + '"')
+                elif path and path in dirfiles:
+                    target.append('%dir "' + path + '"')
+
              for file in files:
                  if file == "CONTROL" or file == "DEBIAN":
                      continue

meta/classes/package_rpm.bbclass +2 -2

          splitname    = strip_multilib(pkgname, d)
 

@@ -367,7 +383,7 @@ python write_specfile () {
              srcrpostrm     = splitrpostrm
 
              file_list = []
-            walk_files(root, file_list, conffiles)
+            walk_files(root, file_list, conffiles, dirfiles)
              if not file_list and localdata.getVar('ALLOW_EMPTY') != "1":
                  bb.note("Not creating empty RPM package for %s" % splitname)
              else:

@@ -474,7 +490,7 @@ python write_specfile () {
 
          # Now process files
          file_list = []
-        walk_files(root, file_list, conffiles)
+        walk_files(root, file_list, conffiles, dirfiles)
          if not file_list and localdata.getVar('ALLOW_EMPTY') != "1":
              bb.note("Not creating empty RPM package for %s" % splitname)
          else:

Quoted 16 lines · Richard Purdie

From: Ronan Le Martret <[email protected]>

During spec generation, ideally directories should not be auto packaged under the %file section of rpm packages but take ownership of specific directories.

* packages only empty directories or explict directory.
See:
- http://www.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html
- "The %dir Directive"

* This will prevent the overlapping of security permission.
For example, in Tizen the directory /etc have smack label 'System::Shared' So Only one package should own and set the label of /etc to prevent the overwriting of the smack label.

Quoted 4 lines

Existing behaviour is maintained if DIRFILES is not set. If it is set, the modified behaviour is used.

[RP: Modified to allow optional usage of DIRFILES]

I'm not sure I understand what is trying to be accomplished here.

Within RPM (4 and 5) it's completely legal for two packages to contain the same directory and permissions entries. This is preferable for our environment as it allows the necessary directories to be generated as packages are installed that use these directories. (Otherwise we have to have "directory" packages that install common directories -- or base packages that own directories for a group of packages.)

I think it's a bug in the SMACK labeling that it doesn't support packages having multiple directory support.

With that said see below..

Quoted 26 lines

Signed-off-by Ronan Le Martret <[email protected]>

Signed-off-by Richard Purdie <[email protected]>

meta/classes/package_rpm.bbclass +2 -6

index 0a32b3e..eecfcb2 100644
@@ -185,7 +185,7 @@ python write_specfile () {
                  if not len(depends_dict[dep]):
                      array.append("%s: %s" % (tag, dep))
 
-    def walk_files(walkpath, target, conffiles):
+    def walk_files(walkpath, target, conffiles, dirfiles):
          # We can race against the ipk/deb backends which create CONTROL or DEBIAN directories
          # when packaging. We just ignore these files which are created in
          # packages-split/ and not package/

@@ -196,11 +196,24 @@ python write_specfile () {
              path = rootpath.replace(walkpath, "")
              if path.endswith("DEBIAN") or path.endswith("CONTROL"):
                  continue
-            for dir in dirs:
-                if dir == "CONTROL" or dir == "DEBIAN":
-                    continue
-                # All packages own the directories their files are in...
-                target.append('%dir "' + path + '/' + dir + '"')
+

meta/classes/package_rpm.bbclass python write_specfile () { · line 203

Quoted 4 lines

meta/classes/package_rpm.bbclass +4 -0

+            # Directory handling can happen in two ways, either DIRFILES is not set at all
+            # in which case we fall back to the older behaviour of packages owning all their
+            # directories
+            if dirfiles is None:

So if dirfiles is none (the normal case) it -will- package directories, correct?
So what happens when you have a recipe that you don't want to package any
directories?

Having to define dirfiles seems to be backwards to me. A blacklist behavior seems more sane, as a way to say "I use this, but I don't own it."

Along those lines (blacklist vs include list), the system directories should already be defined in the meta/files/fs-perms.txt (or related perms files configured by "FILESYSTEM_PERMS_TABLES".

The purpose is to capture system directories that are shared between different recipes to ensure they have the same permissions.

It should be possible to (safely) extend this to include additional common permissions [might violate SMACK security] or directories that should not be packaged because they are 'system' directories.

(Note in the later case, we definitely need at least one 'directory owner' package to handle those -- or they will be generated on demand with default system permissions of 0755 root,root -- which is not always what is desired!)

Quoted 19 lines

meta/classes/package_rpm.bbclass +13 -0

+                for dir in dirs:
+                    if dir == "CONTROL" or dir == "DEBIAN":
+                        continue
+                    # All packages own the directories their files are in...
+                    target.append('%dir "' + path + '/' + dir + '"')
+            else:
+                # packages own only empty directories or explict directory.
+                # This will prevent the overlapping of security permission.
+                if path and not files and not dirs:
+                    target.append('%dir "' + path + '"')
+                elif path and path in dirfiles:
+                    target.append('%dir "' + path + '"')
+
              for file in files:
                  if file == "CONTROL" or file == "DEBIAN":
                      continue

@@ -311,6 +324,9 @@ python write_specfile () {
          bb.data.update_data(localdata)
 

meta/classes/package_rpm.bbclass python write_specfile () { · line 329

Quoted 4 lines

meta/classes/package_rpm.bbclass +3 -0

          conffiles = (localdata.getVar('CONFFILES', True) or "").split()
+        dirfiles = localdata.getVar('DIRFILES', True)
+        if dirfiles is not None:
+            dirfiles = dirfiles.split()

(tabs and spaces mismatch? or just a mailer issue mangling spacing?)

Quoted 31 lines · Mark Hatle

Quoted 16 lines · Richard Purdie

From: Ronan Le Martret <[email protected]>

During spec generation, ideally directories should not be auto packaged under the %file section of rpm packages but take ownership of specific directories.

* packages only empty directories or explict directory.
See:
- http://www.rpm.org/max-rpm/s1-rpm-inside-files-list-directives.html
- "The %dir Directive"

* This will prevent the overlapping of security permission.
For example, in Tizen the directory /etc have smack label 'System::Shared' So Only one package should own and set the label of /etc to prevent the overwriting of the smack label.

Quoted 4 lines

Existing behaviour is maintained if DIRFILES is not set. If it is set, the modified behaviour is used.

[RP: Modified to allow optional usage of DIRFILES]

I'm not sure I understand what is trying to be accomplished here.

Within RPM (4 and 5) it's completely legal for two packages to contain the same directory and permissions entries. This is preferable for our environment as it allows the necessary directories to be generated as packages are installed that use these directories. (Otherwise we have to have "directory" packages that install common directories -- or base packages that own directories for a group of packages.)

Quoted 2 lines

I think it's a bug in the SMACK labeling that it doesn't support packages having multiple directory support.

Whether we agree with it or not, this is something we're being asked about...

Quoted 28 lines

With that said see below..

Quoted 26 lines

Signed-off-by Ronan Le Martret <[email protected]>

Signed-off-by Richard Purdie <[email protected]>

meta/classes/package_rpm.bbclass +2 -6

index 0a32b3e..eecfcb2 100644
@@ -185,7 +185,7 @@ python write_specfile () {
                   if not len(depends_dict[dep]):
                       array.append("%s: %s" % (tag, dep))
 
-    def walk_files(walkpath, target, conffiles):
+    def walk_files(walkpath, target, conffiles, dirfiles):
           # We can race against the ipk/deb backends which create CONTROL or DEBIAN directories
           # when packaging. We just ignore these files which are created in
           # packages-split/ and not package/

@@ -196,11 +196,24 @@ python write_specfile () {
               path = rootpath.replace(walkpath, "")
               if path.endswith("DEBIAN") or path.endswith("CONTROL"):
                   continue
-            for dir in dirs:
-                if dir == "CONTROL" or dir == "DEBIAN":
-                    continue
-                # All packages own the directories their files are in...
-                target.append('%dir "' + path + '/' + dir + '"')
+

Quoted 8 lines

meta/classes/package_rpm.bbclass python write_specfile () { · line 203

Quoted 4 lines

meta/classes/package_rpm.bbclass +4 -0

+            # Directory handling can happen in two ways, either DIRFILES is not set at all
+            # in which case we fall back to the older behaviour of packages owning all their
+            # directories
+            if dirfiles is None:

So if dirfiles is none (the normal case) it -will- package directories, correct?
So what happens when you have a recipe that you don't want to package any
directories?

If DIRFILES is not set, you get the current behaviour, importantly, this means existing code continues to work as is. My biggest concern with some other proposals out there is that they need a flag day complete with breakage :/.

Quoted 2 lines

Having to define dirfiles seems to be backwards to me. A blacklist behavior seems more sane, as a way to say "I use this, but I don't own it."

This is the explicit verses implicit argument. With implicit the risk is things change and you don't notice so there is an argument for this working the other way for that reason. I have to admit I don't have a strong preference, I do know we have people already using it as defined by the code above though.

Quoted 7 lines

Along those lines (blacklist vs include list), the system directories should already be defined in the meta/files/fs-perms.txt (or related perms files configured by "FILESYSTEM_PERMS_TABLES".

The purpose is to capture system directories that are shared between different recipes to ensure they have the same permissions.

Quoted 7 lines

It should be possible to (safely) extend this to include additional common permissions [might violate SMACK security] or directories that should not be packaged because they are 'system' directories.

(Note in the later case, we definitely need at least one 'directory owner' package to handle those -- or they will be generated on demand with default system permissions of 0755 root,root -- which is not always what is desired!)

Going back in time, I remember us specifically talking about directory ownership and how we likely should try and reach a point where the common system directories do become owned by specific packages. With this kind of DIRFILES support we could move in the direction. The perms tables obviously help to a point ensuring consistent permissions but they don't help the ownership problem. Or is this less of an issue since we last discussed it (which admittedly was a while ago)?

Quoted 19 lines

Quoted 19 lines

meta/classes/package_rpm.bbclass +13 -0

+                for dir in dirs:
+                    if dir == "CONTROL" or dir == "DEBIAN":
+                        continue
+                    # All packages own the directories their files are in...
+                    target.append('%dir "' + path + '/' + dir + '"')
+            else:
+                # packages own only empty directories or explict directory.
+                # This will prevent the overlapping of security permission.
+                if path and not files and not dirs:
+                    target.append('%dir "' + path + '"')
+                elif path and path in dirfiles:
+                    target.append('%dir "' + path + '"')
+
               for file in files:
                   if file == "CONTROL" or file == "DEBIAN":
                       continue

@@ -311,6 +324,9 @@ python write_specfile () {
           bb.data.update_data(localdata)
 

Quoted 6 lines

meta/classes/package_rpm.bbclass python write_specfile () { · line 329

Quoted 4 lines

meta/classes/package_rpm.bbclass +3 -0

           conffiles = (localdata.getVar('CONFFILES', True) or "").split()
+        dirfiles = localdata.getVar('DIRFILES', True)
+        if dirfiles is not None:
+            dirfiles = dirfiles.split()

(tabs and spaces mismatch? or just a mailer issue mangling spacing?)

I probably mangled the patch somehow, will check...

Cheers,

Richard

Quoted 8 lines · Richard Purdie

On Fri, 2014-08-29 at 13:36 -0500, Mark Hatle wrote:
Going back in time, I remember us specifically talking about directory ownership and how we likely should try and reach a point where the common system directories do become owned by specific packages. With this kind of DIRFILES support we could move in the direction. The perms tables obviously help to a point ensuring consistent permissions but they don't help the ownership problem. Or is this less of an issue since we last discussed it (which admittedly was a while ago)?

No there is currently nothing that says I exclusively own a directory (or link).
The fs-perms.txt could be extended to do this (in a transparent way).

My concern with the DIRFILES as it appears to be implemented can be shown in the existing example:

I create a new recipe that writes:

/etc/foo.conf
/usr/bin/foo

(that's it)

In the SMACK case, the /etc and /usr/bin directories shouldn't be included.. so how do we define DIRFILES? If it's blank, they'll be included.. but we don't have any directories to set it to... so do we need to do:
DIRFILES = "something_random_so_it_works"

That seems very counter intuitive to me.

This is why I'm suggesting an inverse relationship.. We include everything other then explicitly listed directories. That way the user can globally define /etc, /usr/bin, ... and individual recipes can augment this with their own custom values if appropriate.

and in the default (oe-core) case no change means the directories will continue to be included -- no flag days required.

Quoted 31 lines · Mark Hatle

Quoted 8 lines · Richard Purdie

On Fri, 2014-08-29 at 13:36 -0500, Mark Hatle wrote:
Going back in time, I remember us specifically talking about directory ownership and how we likely should try and reach a point where the common system directories do become owned by specific packages. With this kind of DIRFILES support we could move in the direction. The perms tables obviously help to a point ensuring consistent permissions but they don't help the ownership problem. Or is this less of an issue since we last discussed it (which admittedly was a while ago)?

No there is currently nothing that says I exclusively own a directory (or link).
The fs-perms.txt could be extended to do this (in a transparent way).

My concern with the DIRFILES as it appears to be implemented can be shown in the existing example:

I create a new recipe that writes:

/etc/foo.conf
/usr/bin/foo

(that's it)

In the SMACK case, the /etc and /usr/bin directories shouldn't be included.. so how do we define DIRFILES? If it's blank, they'll be included.. but we don't have any directories to set it to... so do we need to do:
DIRFILES = "something_random_so_it_works"

That seems very counter intuitive to me.

Quoted 7 lines

This is why I'm suggesting an inverse relationship.. We include everything other then explicitly listed directories. That way the user can globally define /etc, /usr/bin, ... and individual recipes can augment this with their own custom values if appropriate.

and in the default (oe-core) case no change means the directories will continue to be included -- no flag days required.

I'm more thinking that when we reach this stage, the core would end up setting:

DIRFILES = ""

as the default (think a core class or conf file), then recipes can override as needed. You don't need something_random_so_it_works, I had the empty value specifically in mind to trigger this from the core (as opposed to None where the variable isn't set at all).

Cheers,

Richard