package_rpm.bbclass: support packaging of symlinks to directories
PATCH 1/1 package_rpm.bbclass: support packaging of symlinks to directories
os.walk() returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.
The fix is simple: find such special directory entries and move them to the "files" list.
--- meta/classes/package_rpm.bbclass | 7 +++++++ 1 file changed, 7 insertions(+)
meta/classes/package_rpm.bbclass +7 -0
@@ -197,6 +197,13 @@ python write_specfile () { if path.endswith("DEBIAN") or path.endswith("CONTROL"): continue + # Treat all symlinks to directories as normal files. + # os.walk() lists them as directories. + for i, entry in enumerate(dirs): + if os.path.islink(os.path.join(rootpath, entry)): + del dirs[i] + files.append(entry) + # Directory handling can happen in two ways, either DIRFILES is not set at all # in which case we fall back to the older behaviour of packages owning all their # directories
Signature · 1 line
1.8.4.5
meta/classes/package_rpm.bbclass python write_specfile () { · line 206
Quoted 8 lines · Patrick Ohly
meta/classes/package_rpm.bbclass +7
-0+ # Treat all symlinks to directories as normal files. + # os.walk() lists them as directories. + for i, entry in enumerate(dirs): + if os.path.islink(os.path.join(rootpath, entry)): + del dirs[i] + files.append(entry) +
You're deleting elements of a list while you're iterating over it. I'm fairly certain that will lead to pain, unless you explicitly ensure you're operating against a copy: for i, entry in enumerate(list(dirs)):
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 13 lines · Christopher Larson
On Tue, Feb 17, 2015 at 6:42 AM, Patrick Ohly <[email protected]> wrote:
+ # Treat all symlinks to directories as normal
files.
+ #os.walk()lists them as directories.
+ for i, entry in enumerate(dirs):
+ if os.path.islink(os.path.join(rootpath,
entry)):
+ del dirs[i]
+files.append(entry)
+
Quoted 4 lines
You're deleting elements of a list while you're iterating over it. I'm fairly certain that will lead to pain, unless you explicitly ensure you're operating against a copy: for i, entry in enumerate(list(dirs)):
I was wondering about that myself, but couldn't find any definite statement about whether it's okay or not for enumerate(). It works in practice, but of course that doesn't guarantee that it is okay.
Iterating backwards will be more obviously correct, I'll send a patch update using that.
I'm still curious, though, whether the code above is really broken or just dubious because one cannot be certain.
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Quoted 14 lines · Patrick Ohly
Quoted 13 lines · Christopher Larson
On Tue, Feb 17, 2015 at 6:42 AM, Patrick Ohly <[email protected]> wrote:
+ # Treat all symlinks to directories as normal
files.
+ #os.walk()lists them as directories.
+ for i, entry in enumerate(dirs):
+ if os.path.islink(os.path.join(rootpath,
entry)):
+ del dirs[i]
+files.append(entry)
+
Quoted 8 lines
Quoted 4 lines
You're deleting elements of a list while you're iterating over it. I'm fairly certain that will lead to pain, unless you explicitly ensure you're operating against a copy: for i, entry in enumerate(list(dirs)):
I was wondering about that myself, but couldn't find any definite statement about whether it's okay or not for
enumerate(). It works in practice, but of course that doesn't guarantee that it is okay.
The enumerate() documentation says that it is equivalent to a "for in" loop, and documentation for that says "it is *recommended* that you first make a copy" (emphasis mine). IMHO it means the behavior is simply undefined.
Quoted 2 lines
Iterating backwards will be more obviously correct, I'll send a patch update using that.
The recommended approach is using a slice copy
(https://docs.python.org/2/tutorial/controlflow.html#for-statements), so how about this:
# Avoid modifying the list we iterate over, iterate over slice copy
# instead.
for i, entry in enumerate(dirs[:]):
if os.path.islink(os.path.join(rootpath, entry)):
del dirs[i]
files.append(entry)Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Quoted 8 lines · Patrick Ohly
os.walk()returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.The fix is simple: find such special directory entries and move them to the "files" list.
I'm curious, what is the problem that is being fixed with this.
Normally you don't want to walk into symlinks to directories, as either you want to capture the symlink itself -- or the symlink is pointing to something you don't have access to when packaging (i.e. /proc).
--Mark
Quoted 24 lines
Signed-off-by Patrick Ohly <[email protected]>--- meta/classes/package_rpm.bbclass | 7 +++++++ 1 file changed, 7 insertions(+)meta/classes/package_rpm.bbclass +7
-0@@ -197,6 +197,13 @@ python write_specfile () { if path.endswith("DEBIAN") or path.endswith("CONTROL"): continue + # Treat all symlinks to directories as normal files. + # os.walk() lists them as directories. + for i, entry in enumerate(dirs): + if os.path.islink(os.path.join(rootpath, entry)): + del dirs[i] + files.append(entry) + # Directory handling can happen in two ways, either DIRFILES is not set at all # in which case we fall back to the older behaviour of packages owning all their # directories
Quoted 10 lines · Mark Hatle
Quoted 8 lines · Patrick Ohly
os.walk()returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.The fix is simple: find such special directory entries and move them to the "files" list.
Quoted 7 lines
I'm curious, what is the problem that is being fixed with this.
Normally you don't want to walk into symlinks to directories, as either you want
to capture the symlink itself -- or the symlink is pointing to something you
don't have access to when packaging (i.e. /proc).
This patch doesn't make it walk into the directory symlinks, it moves them into the files list rather than the dirs list so they'll be captured themselves, as far as I can tell from reading it.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 16 lines · Christopher Larson
On Tue, Feb 17, 2015 at 9:54 AM, Mark Hatle <[email protected] <mailto:[email protected]>> wrote:
On 2/17/15 7:42 AM, Patrick Ohly wrote:
>os.walk()returns symlinks to directories in the "dirs" lists, > but then never enters them by default. As a result, the old > code applied neither the directory handling (because that > is active once a directory gets entered) nor the file handling, > and thus never packaged such symlinks.
>
> The fix is simple: find such special directory entries and move > them to the "files" list.I'm curious, what is the problem that is being fixed with this.
Quoted 8 lines
Normally you don't want to walk into symlinks to directories, as either you want to capture the symlink itself -- or the symlink is pointing to something you don't have access to when packaging (i.e. /proc).
This patch doesn't make it walk into the directory symlinks, it moves them into the files list rather than the dirs list so they'll be captured themselves, as far as I can tell from reading it.
Ok.. I missed that bit.. I'm still curious what this fixes.
--Mark
Quoted 13 lines · Mark Hatle
Quoted 11 lines · Christopher Larson
On Tue, Feb 17, 2015 at 9:54 AM, Mark Hatle <[email protected] <mailto:[email protected]>> wrote:
On 2/17/15 7:42 AM, Patrick Ohly wrote:
>os.walk()returns symlinks to directories in the "dirs" lists, > but then never enters them by default. As a result, the old > code applied neither the directory handling (because that > is active once a directory gets entered) nor the file handling, > and thus never packaged such symlinks.
>
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
The fix is simple: find such special directory entries and move them to the "files" list.
I'm curious, what is the problem that is being fixed with this.
Normally you don't want to walk into symlinks to directories, as
either you want
Quoted 1 line
Quoted 1 line
to capture the symlink itself -- or the symlink is pointing to
something you
Quoted 4 lines
Quoted 4 lines
don't have access to when packaging (i.e. /proc).
This patch doesn't make it walk into the directory symlinks, it moves
them into
Quoted 1 line
Quoted 1 line
the files list rather than the dirs list so they'll be captured
themselves, as
Quoted 4 lines
Quoted 1 line
far as I can tell from reading it.
Ok.. I missed that bit.. I'm still curious what this fixes.
I haven't tried myself, but I expect it helps with the /usr merge.
Quoted 7 lines
--Mark
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Quoted 12 lines · Dan McGregor
Quoted 4 lines · Mark Hatle
Quoted 2 lines · Christopher Larson
On Tue, Feb 17, 2015 at 9:54 AM, Mark Hatle
Quoted 5 lines
Quoted 4 lines
<mailto:[email protected]>> wrote:
This patch doesn't make it walk into the directory symlinks, it moves them into the files list rather than the dirs list so they'll be captured themselves, as far as I can tell from reading it.
Quoted 4 lines
Quoted 2 lines
Ok.. I missed that bit.. I'm still curious what this fixes.
I haven't tried myself, but I expect it helps with the /usr merge.
Correct. I was working on a distro with a modified basefiles recipe which only had /usr/[lib|bin|sbin] and symlinks to those in /. Without this patch it was not possible to get these symlinks packaged in the basefiles rpm.
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
Quoted 16 lines · Patrick Ohly
Quoted 15 lines · Dan McGregor
Quoted 4 lines · Mark Hatle
Quoted 2 lines · Christopher Larson
On Tue, Feb 17, 2015 at 9:54 AM, Mark Hatle
Quoted 7 lines
Quoted 4 lines
<mailto:[email protected]>> wrote:
This patch doesn't make it walk into the directory symlinks, it moves them into the files list rather than the dirs list so they'll be captured themselves, as far as I can tell from reading it.Ok.. I missed that bit.. I'm still curious what this fixes.
Quoted 7 lines
Quoted 1 line
I haven't tried myself, but I expect it helps with the /usr merge.
Correct. I was working on a distro with a modified basefiles recipe which only had /usr/[lib|bin|sbin] and symlinks to those in /. Without this patch it was not possible to get these symlinks packaged in the basefiles rpm.
Thanks, I wasn't aware that the symlinks were not working in the basefiles.
--Mark
PATCH v2 1/1 package_rpm.bbclass: support packaging of symlinks to directories
In reply to Patrick Ohlyos.walk() returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.
The fix is simple: find such special directory entries and move them to the "files" list. However, one has to be careful about the undefined behavior of modifying a list while iterating over it.
This fix was required for packaging a modified base-files that created symlinks into /usr for /sbin /lib and /sbin.
--- meta/classes/package_rpm.bbclass | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
meta/classes/package_rpm.bbclass +7 -4
@@ -199,10 +199,13 @@ python write_specfile () { # Treat all symlinks to directories as normal files. # os.walk() lists them as directories. - for i, entry in enumerate(dirs): - if os.path.islink(os.path.join(rootpath, entry)): - del dirs[i] - files.append(entry) + def move_to_files(dir): + if os.path.islink(os.path.join(rootpath, dir)): + files.append(dir) + return True + else: + return False + dirs[:] = [dir for dir in dirs if not move_to_files(dir)] # Directory handling can happen in two ways, either DIRFILES is not set at all # in which case we fall back to the older behaviour of packages owning all their
Signature · 1 line
1.8.4.5
Quoted 12 lines · Patrick Ohly
meta/classes/package_rpm.bbclass +0
-4@@ -199,10 +199,13 @@ python write_specfile () { # Treat all symlinks to directories as normal files. # os.walk() lists them as directories. - for i, entry in enumerate(dirs): - if os.path.islink(os.path.join(rootpath, entry)): - del dirs[i] - files.append(entry)
meta/classes/package_rpm.bbclass python write_specfile () { · line 211
Quoted 10 lines
meta/classes/package_rpm.bbclass +7
-0+ def move_to_files(dir): + if os.path.islink(os.path.join(rootpath, dir)): + files.append(dir) + return True + else: + return False + dirs[:] = [dir for dir in dirs if not move_to_files(dir)] # Directory handling can happen in two ways, either DIRFILES is not set at all # in which case we fall back to the older behaviour of packages owning all their
Please ignore. I meant to squash the original patch and the list fix, but was interrupted and ended up committing the fix separately with the updated commit message. v3 is the patch that could be merged into master.
Signature · 6 lines
Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.
PATCH v3 1/1 package_rpm.bbclass: support packaging of symlinks to directories
In reply to Patrick Ohlyos.walk() returns symlinks to directories in the "dirs" lists, but then never enters them by default. As a result, the old code applied neither the directory handling (because that is active once a directory gets entered) nor the file handling, and thus never packaged such symlinks.
The fix is simple: find such special directory entries and move them to the "files" list. However, one has to be careful about the undefined behavior of modifying a list while iterating over it.
This fix was required for packaging a modified base-files that created symlinks into /usr for /sbin /lib and /sbin.
--- meta/classes/package_rpm.bbclass | 10 ++++++++++ 1 file changed, 10 insertions(+)
meta/classes/package_rpm.bbclass +10 -0
@@ -197,6 +197,16 @@ python write_specfile () { if path.endswith("DEBIAN") or path.endswith("CONTROL"): continue + # Treat all symlinks to directories as normal files. + # os.walk() lists them as directories. + def move_to_files(dir): + if os.path.islink(os.path.join(rootpath, dir)): + files.append(dir) + return True + else: + return False + dirs[:] = [dir for dir in dirs if not move_to_files(dir)] + # Directory handling can happen in two ways, either DIRFILES is not set at all # in which case we fall back to the older behaviour of packages owning all their # directories
Signature · 1 line
1.8.4.5