Skip to contentSkip to conversation

package_rpm.bbclass: support packaging of symlinks to directories

PATCH Superseded Patrick Ohly 4 participants 13 messages Feb 17, 2015 – Feb 25, 2015

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package_rpm.bbclass: support packaging of symlinks to directories

os.walk() returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.

The fix is simple: find such special directory entries and move them to the "files" list.

---
 meta/classes/package_rpm.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)
meta/classes/package_rpm.bbclass +7 -0
index 92ddf7a..6483e96 100644
@@ -197,6 +197,13 @@ python write_specfile () {
             if path.endswith("DEBIAN") or path.endswith("CONTROL"):
                 continue
 
+            # Treat all symlinks to directories as normal files.
+            # os.walk() lists them as directories.
+            for i, entry in enumerate(dirs):
+                if os.path.islink(os.path.join(rootpath, entry)):
+                    del dirs[i]
+                    files.append(entry)
+
             # Directory handling can happen in two ways, either DIRFILES is not set at all
             # in which case we fall back to the older behaviour of packages owning all their
             # directories
Signature · 1 line
1.8.4.5
#1 lore raw

meta/classes/package_rpm.bbclass python write_specfile () { · line 206

Quoted 8 lines · Patrick Ohly
meta/classes/package_rpm.bbclass +7 -0
+            # Treat all symlinks to directories as normal files.
+            # os.walk() lists them as directories.
+            for i, entry in enumerate(dirs):
+                if os.path.islink(os.path.join(rootpath, entry)):
+                    del dirs[i]
+                    files.append(entry)
+

You're deleting elements of a list while you're iterating over it. I'm fairly certain that will lead to pain, unless you explicitly ensure you're operating against a copy: for i, entry in enumerate(list(dirs)):

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#2 lore raw
Quoted 13 lines · Christopher Larson

On Tue, Feb 17, 2015 at 6:42 AM, Patrick Ohly <[email protected]> wrote:
+ # Treat all symlinks to directories as normal
files.
+ # os.walk() lists them as directories.
+ for i, entry in enumerate(dirs):
+ if os.path.islink(os.path.join(rootpath,
entry)):
+ del dirs[i]
+ files.append(entry)
+

Quoted 4 lines

You're deleting elements of a list while you're iterating over it. I'm fairly certain that will lead to pain, unless you explicitly ensure you're operating against a copy: for i, entry in enumerate(list(dirs)):

I was wondering about that myself, but couldn't find any definite statement about whether it's okay or not for enumerate(). It works in practice, but of course that doesn't guarantee that it is okay.

Iterating backwards will be more obviously correct, I'll send a patch update using that.

I'm still curious, though, whether the code above is really broken or just dubious because one cannot be certain.

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#3 lore raw
Quoted 14 lines · Patrick Ohly
Quoted 13 lines · Christopher Larson

On Tue, Feb 17, 2015 at 6:42 AM, Patrick Ohly <[email protected]> wrote:
+ # Treat all symlinks to directories as normal
files.
+ # os.walk() lists them as directories.
+ for i, entry in enumerate(dirs):
+ if os.path.islink(os.path.join(rootpath,
entry)):
+ del dirs[i]
+ files.append(entry)
+

Quoted 8 lines
Quoted 4 lines

You're deleting elements of a list while you're iterating over it. I'm fairly certain that will lead to pain, unless you explicitly ensure you're operating against a copy: for i, entry in enumerate(list(dirs)):

I was wondering about that myself, but couldn't find any definite statement about whether it's okay or not for enumerate(). It works in practice, but of course that doesn't guarantee that it is okay.

The enumerate() documentation says that it is equivalent to a "for in" loop, and documentation for that says "it is *recommended* that you first make a copy" (emphasis mine). IMHO it means the behavior is simply undefined.

Quoted 2 lines

Iterating backwards will be more obviously correct, I'll send a patch update using that.

The recommended approach is using a slice copy
(https://docs.python.org/2/tutorial/controlflow.html#for-statements), so how about this:

# Avoid modifying the list we iterate over, iterate over slice copy

# instead.
for i, entry in enumerate(dirs[:]):
    if os.path.islink(os.path.join(rootpath, entry)):
         del dirs[i]
         files.append(entry)
Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#4 lore raw
In reply to Patrick Ohly
Quoted 8 lines · Patrick Ohly

os.walk() returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.

The fix is simple: find such special directory entries and move them to the "files" list.

I'm curious, what is the problem that is being fixed with this.

Normally you don't want to walk into symlinks to directories, as either you want to capture the symlink itself -- or the symlink is pointing to something you don't have access to when packaging (i.e. /proc).

--Mark

Quoted 24 lines
---
 meta/classes/package_rpm.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)
meta/classes/package_rpm.bbclass +7 -0
index 92ddf7a..6483e96 100644
@@ -197,6 +197,13 @@ python write_specfile () {
             if path.endswith("DEBIAN") or path.endswith("CONTROL"):
                 continue
 
+            # Treat all symlinks to directories as normal files.
+            # os.walk() lists them as directories.
+            for i, entry in enumerate(dirs):
+                if os.path.islink(os.path.join(rootpath, entry)):
+                    del dirs[i]
+                    files.append(entry)
+
             # Directory handling can happen in two ways, either DIRFILES is not set at all
             # in which case we fall back to the older behaviour of packages owning all their
             # directories
#5 lore raw
Quoted 10 lines · Mark Hatle
Quoted 8 lines · Patrick Ohly

os.walk() returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.

The fix is simple: find such special directory entries and move them to the "files" list.

Quoted 7 lines

I'm curious, what is the problem that is being fixed with this.

Normally you don't want to walk into symlinks to directories, as either you want
to capture the symlink itself -- or the symlink is pointing to something you
don't have access to when packaging (i.e. /proc).

This patch doesn't make it walk into the directory symlinks, it moves them into the files list rather than the dirs list so they'll be captured themselves, as far as I can tell from reading it.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#6 lore raw
Quoted 16 lines · Christopher Larson

On Tue, Feb 17, 2015 at 9:54 AM, Mark Hatle <[email protected] <mailto:[email protected]>> wrote:

On 2/17/15 7:42 AM, Patrick Ohly wrote:
> os.walk() returns symlinks to directories in the "dirs" lists, > but then never enters them by default. As a result, the old > code applied neither the directory handling (because that > is active once a directory gets entered) nor the file handling, > and thus never packaged such symlinks.
>
> The fix is simple: find such special directory entries and move > them to the "files" list.

I'm curious, what is the problem that is being fixed with this.

Quoted 8 lines

Normally you don't want to walk into symlinks to directories, as either you want to capture the symlink itself -- or the symlink is pointing to something you don't have access to when packaging (i.e. /proc).

This patch doesn't make it walk into the directory symlinks, it moves them into the files list rather than the dirs list so they'll be captured themselves, as far as I can tell from reading it.

Ok.. I missed that bit.. I'm still curious what this fixes.

--Mark

#7 lore raw
Quoted 13 lines · Mark Hatle
Quoted 11 lines · Christopher Larson

On Tue, Feb 17, 2015 at 9:54 AM, Mark Hatle <[email protected] <mailto:[email protected]>> wrote:

On 2/17/15 7:42 AM, Patrick Ohly wrote:
> os.walk() returns symlinks to directories in the "dirs" lists, > but then never enters them by default. As a result, the old > code applied neither the directory handling (because that > is active once a directory gets entered) nor the file handling, > and thus never packaged such symlinks.
>

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

The fix is simple: find such special directory entries and move them to the "files" list.

I'm curious, what is the problem that is being fixed with this.

Normally you don't want to walk into symlinks to directories, as

either you want

Quoted 1 line
Quoted 1 line

to capture the symlink itself -- or the symlink is pointing to

something you

Quoted 4 lines
Quoted 4 lines

don't have access to when packaging (i.e. /proc).

This patch doesn't make it walk into the directory symlinks, it moves

them into

Quoted 1 line
Quoted 1 line

the files list rather than the dirs list so they'll be captured

themselves, as

Quoted 4 lines
Quoted 1 line

far as I can tell from reading it.

Ok.. I missed that bit.. I'm still curious what this fixes.

I haven't tried myself, but I expect it helps with the /usr merge.

Quoted 7 lines

--Mark

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#8 lore raw
Quoted 12 lines · Dan McGregor
Quoted 4 lines · Mark Hatle
Quoted 2 lines · Christopher Larson

On Tue, Feb 17, 2015 at 9:54 AM, Mark Hatle

<[email protected]

Quoted 5 lines
Quoted 4 lines

<mailto:[email protected]>> wrote:
This patch doesn't make it walk into the directory symlinks, it moves them into the files list rather than the dirs list so they'll be captured themselves, as far as I can tell from reading it.

Quoted 4 lines
Quoted 2 lines

Ok.. I missed that bit.. I'm still curious what this fixes.

I haven't tried myself, but I expect it helps with the /usr merge.

Correct. I was working on a distro with a modified basefiles recipe which only had /usr/[lib|bin|sbin] and symlinks to those in /. Without this patch it was not possible to get these symlinks packaged in the basefiles rpm.

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#9 lore raw
Quoted 16 lines · Patrick Ohly
Quoted 15 lines · Dan McGregor
Quoted 4 lines · Mark Hatle
Quoted 2 lines · Christopher Larson

On Tue, Feb 17, 2015 at 9:54 AM, Mark Hatle

<[email protected]

Quoted 7 lines
Quoted 4 lines

<mailto:[email protected]>> wrote:
This patch doesn't make it walk into the directory symlinks, it moves them into the files list rather than the dirs list so they'll be captured themselves, as far as I can tell from reading it.

Ok.. I missed that bit.. I'm still curious what this fixes.

Quoted 7 lines
Quoted 1 line

I haven't tried myself, but I expect it helps with the /usr merge.

Correct. I was working on a distro with a modified basefiles recipe which only had /usr/[lib|bin|sbin] and symlinks to those in /. Without this patch it was not possible to get these symlinks packaged in the basefiles rpm.

Thanks, I wasn't aware that the symlinks were not working in the basefiles.

--Mark

#10 lore raw

PATCH v2 1/1 package_rpm.bbclass: support packaging of symlinks to directories

In reply to Patrick Ohly

os.walk() returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.

The fix is simple: find such special directory entries and move them to the "files" list. However, one has to be careful about the undefined behavior of modifying a list while iterating over it.

This fix was required for packaging a modified base-files that created symlinks into /usr for /sbin /lib and /sbin.

---
 meta/classes/package_rpm.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
meta/classes/package_rpm.bbclass +7 -4
index 4f9f813..e305e8b 100644
@@ -199,10 +199,13 @@ python write_specfile () {
 
             # Treat all symlinks to directories as normal files.
             # os.walk() lists them as directories.
-            for i, entry in enumerate(dirs):
-                if os.path.islink(os.path.join(rootpath, entry)):
-                    del dirs[i]
-                    files.append(entry)
+            def move_to_files(dir):
+                if os.path.islink(os.path.join(rootpath, dir)):
+                    files.append(dir)
+                    return True
+                else:
+                    return False
+            dirs[:] = [dir for dir in dirs if not move_to_files(dir)]
 
             # Directory handling can happen in two ways, either DIRFILES is not set at all
             # in which case we fall back to the older behaviour of packages owning all their
Signature · 1 line
1.8.4.5
#11 lore raw
Quoted 12 lines · Patrick Ohly
meta/classes/package_rpm.bbclass +0 -4
index 4f9f813..e305e8b 100644
@@ -199,10 +199,13 @@ python write_specfile () {
 
             # Treat all symlinks to directories as normal files.
             # os.walk() lists them as directories.
-            for i, entry in enumerate(dirs):
-                if os.path.islink(os.path.join(rootpath, entry)):
-                    del dirs[i]
-                    files.append(entry)

meta/classes/package_rpm.bbclass python write_specfile () { · line 211

Quoted 10 lines
meta/classes/package_rpm.bbclass +7 -0
+            def move_to_files(dir):
+                if os.path.islink(os.path.join(rootpath, dir)):
+                    files.append(dir)
+                    return True
+                else:
+                    return False
+            dirs[:] = [dir for dir in dirs if not move_to_files(dir)]
 
             # Directory handling can happen in two ways, either DIRFILES is not set at all
             # in which case we fall back to the older behaviour of packages owning all their

Please ignore. I meant to squash the original patch and the list fix, but was interrupted and ended up committing the fix separately with the updated commit message. v3 is the patch that could be merged into master.

Signature · 6 lines
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
#12 lore raw

PATCH v3 1/1 package_rpm.bbclass: support packaging of symlinks to directories

In reply to Patrick Ohly

os.walk() returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.

The fix is simple: find such special directory entries and move them to the "files" list. However, one has to be careful about the undefined behavior of modifying a list while iterating over it.

This fix was required for packaging a modified base-files that created symlinks into /usr for /sbin /lib and /sbin.

---
 meta/classes/package_rpm.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)
meta/classes/package_rpm.bbclass +10 -0
index b87e634..e305e8b 100644
@@ -197,6 +197,16 @@ python write_specfile () {
             if path.endswith("DEBIAN") or path.endswith("CONTROL"):
                 continue
 
+            # Treat all symlinks to directories as normal files.
+            # os.walk() lists them as directories.
+            def move_to_files(dir):
+                if os.path.islink(os.path.join(rootpath, dir)):
+                    files.append(dir)
+                    return True
+                else:
+                    return False
+            dirs[:] = [dir for dir in dirs if not move_to_files(dir)]
+
             # Directory handling can happen in two ways, either DIRFILES is not set at all
             # in which case we fall back to the older behaviour of packages owning all their
             # directories
Signature · 1 line
1.8.4.5