package_manager.py: fix for "Argument list too long"

PATCH Superseded Robert Yang 3 participants 7 messages Oct 13, 2016 – Oct 17, 2016

lore

Share & tools

lore

Robert Yang

author

2016-10-13 09:45 UTC #0 lore raw

cover package_manager.py: fix for "Argument list too long"

The following changes since commit 1a2311c8fa9a3703568cd390c44639fe3335023e:

build-appliance-image: Update to master head revision (2016-10-11 23:43:20 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/long
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long

Robert Yang (1):
package_manager.py: fix for "Argument list too long"

 meta/lib/oe/package_manager.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Signature · 1 line

2.9.0

Robert Yang

author

2016-10-13 09:45 UTC #1 lore raw

PATCH 1/1 package_manager.py: fix for "Argument list too long"

Fixed when len(TMPDIR) = 410:

$ bitbake core-image-sato-sdk
[snip]
Subprocess output:
/bin/sh: /bin/cp: Argument list too long

ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs [snip]

This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does:
"cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", change cwd to deploy_arch_dir can avoid the error.

Signed-off-by Robert Yang <[email protected]>

---
 meta/lib/oe/package_manager.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

meta/lib/oe/package_manager.py +5 -1

index 3cee973..e0e203b 100644
@@ -1024,7 +1024,11 @@ class RpmPM(PackageManager):
 
             lockfilename = self.d.getVar('DEPLOY_DIR_RPM', True) + "/rpm.lock"
             lf = bb.utils.lockfile(lockfilename, False)
-            oe.path.copyhardlinktree(deploy_arch_dir, arch_channel)
+            # chdir() to deploy_arch_dir to avoid "Argument list too long" error
+            oldcwd = os.getcwd()
+            os.chdir(deploy_arch_dir)
+            oe.path.copyhardlinktree('.', arch_channel)
+            os.chdir(oldcwd)
             bb.utils.unlockfile(lf)
 
             if not arch in ch_already_added:

Signature · 1 line

2.9.0

Burton, Ross

2016-10-13 10:39 UTC #2 lore raw

Quoted 6 lines · Robert Yang

This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does:
"cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", change cwd to deploy_arch_dir can avoid the error.

Would it be better to change the implementation of copyhardlinktree so that it *can't* have this problem?

Ross

Robert Yang

author

2016-10-14 09:58 UTC #3 lore raw

Quoted 11 lines · Ross

On 13 October 2016 at 10:45, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does:
"cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", change cwd to deploy_arch_dir can avoid the error.

Quoted 2 lines

Would it be better to change the implementation of copyhardlinktree so that it *can't* have this problem?

Good idea, thanks, updated in the repo:

git://git.openembedded.org/openembedded-core-contrib rbt/long
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long

Author: Robert Yang <[email protected]>
Date:   Thu Oct 13 01:28:41 2016 -0700

oe/path.py: fix for "Argument list too long"

Fixed when len(TMPDIR) = 410:
$ bitbake core-image-sato-sdk
[snip]
Subprocess output:
/bin/sh: /bin/cp: Argument list too long

ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs [snip]

This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst", while src/* is expanded to "src/file1 src/file2, src/file3..." which causes the "Argument list too long", change cwd to src can fix the problem.

Signed-off-by Robert Yang <[email protected]>

meta/lib/oe/path.py +7 -4

index 06a5af2..13821b3 100644
@@ -80,14 +80,18 @@ def copyhardlinktree(src, dst):
         cmd = "cd %s; find . -type d -print | tar --xattrs 
--xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         source = ''
+        # chdir() to src to avoid "Argument list too long" error
+        oldcwd = os.getcwd()
         if os.path.isdir(src):
+            os.chdir(src)
             import glob
-            if len(glob.glob('%s/.??*' % src)) > 0:
-                source = '%s/.??* ' % src
-            source = source + '%s/*' % src
+            if len(glob.glob('.??*')) > 0:
+                source = '.??* '
+            source += '*'
         else:
             source = src
         cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
+        os.chdir(oldcwd)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:

copytree(src, dst)

// Robert

Quoted 2 lines

Ross

Robert Yang

author

2016-10-14 10:18 UTC #4 lore raw

Quoted 14 lines · Robert Yang

Quoted 11 lines · Ross
On 13 October 2016 at 10:45, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
This is because "copyhardlinktree(deploy_arch_dir, arch_channel)" does:
"cp -afl deploy_arch_dir/* arch_channel", while the deploy_arch_dir/* is expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", change cwd to deploy_arch_dir can avoid the error.

Quoted 8 lines

Quoted 2 lines
Would it be better to change the implementation of copyhardlinktree so that it *can't* have this problem?
Good idea, thanks, updated in the repo:
git://git.openembedded.org/openembedded-core-contrib rbt/long
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long

Sorry, this patch is not what I wanted to paste, it should be:

meta/lib/oe/path.py +8 -4

index 06a5af2..1d9cca5 100644
@@ -80,15 +80,20 @@ def copyhardlinktree(src, dst):
         cmd = "cd %s; find . -type d -print | tar --xattrs 
--xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         source = ''
+        oldcwd = os.getcwd()
         if os.path.isdir(src):
+            dst = os.path.realpath(dst)
+            # chdir() to src to avoid "Argument list too long" error
+            os.chdir(src)
             import glob
-            if len(glob.glob('%s/.??*' % src)) > 0:
-                source = '%s/.??* ' % src
-            source = source + '%s/*' % src
+            if len(glob.glob('./.??*')) > 0:
+                source = './.??* '
+            source += './*'
         else:
             source = src
         cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+        os.chdir(oldcwd)
     else:
         copytree(src, dst)

// Robert

Quoted 56 lines

Author: Robert Yang <[email protected]>
Date:   Thu Oct 13 01:28:41 2016 -0700

oe/path.py: fix for "Argument list too long"

Fixed when len(TMPDIR) = 410:
$ bitbake core-image-sato-sdk
[snip]
Subprocess output:
/bin/sh: /bin/cp: Argument list too long

ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs [snip]

Signed-off-by Robert Yang <[email protected]>

meta/lib/oe/path.py +7 -5

index 06a5af2..13821b3 100644
@@ -80,14 +80,18 @@ def copyhardlinktree(src, dst):
         cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*'
-cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*'
-xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         source = ''
+        # chdir() to src to avoid "Argument list too long" error
+        oldcwd = os.getcwd()
         if os.path.isdir(src):
+            os.chdir(src)
             import glob
-            if len(glob.glob('%s/.??*' % src)) > 0:
-                source = '%s/.??* ' % src
-            source = source + '%s/*' % src
+            if len(glob.glob('.??*')) > 0:
+                source = '.??* '
+            source += '*'
         else:
             source = src
         cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
+        os.chdir(oldcwd)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
         copytree(src, dst)

// Robert

Quoted 2 lines

Ross

Christopher Larson

2016-10-16 04:27 UTC #5 lore raw

Quoted 50 lines · Robert Yang

Quoted 22 lines · Robert Yang
Quoted 13 lines · Ross
On 13 October 2016 at 10:45, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
This is because "copyhardlinktree(deploy_arch_dir, arch_channel)"
does:
"cp -afl deploy_arch_dir/* arch_channel", while the
deploy_arch_dir/* is
expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", change cwd to deploy_arch_dir can avoid the error.
Quoted 4 lines
Would it be better to change the implementation of copyhardlinktree so that it
*can't* have this problem?
Quoted 7 lines
Good idea, thanks, updated in the repo:
git://git.openembedded.org/openembedded-core-contrib rbt/long
http://cgit.openembedded.org/cgit.cgi/openembedded-core-cont
rib/log/?h=rbt/long
Sorry, this patch is not what I wanted to paste, it should be:
meta/lib/oe/path.py +4 -3
index 06a5af2..1d9cca5 100644
@@ -80,15 +80,20 @@ def copyhardlinktree(src, dst):
         cmd = "cd %s; find . -type d -print | tar --xattrs
--xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar
--xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
         source = ''
+        oldcwd = os.getcwd()
         if os.path.isdir(src):
+            dst = os.path.realpath(dst)
+            # chdir() to src to avoid "Argument list too long" error
+            os.chdir(src)
             import glob
-            if len(glob.glob('%s/.??*' % src)) > 0:

meta/lib/oe/path.py def copyhardlinktree(src, dst): · line 96

Quoted 10 lines

meta/lib/oe/path.py +4 -2

-                source = '%s/.??* ' % src
-            source = source + '%s/*' % src
+            if len(glob.glob('./.??*')) > 0:
+                source = './.??* '
+            source += './*'
         else:
             source = src
         cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+        os.chdir(oldcwd)

Question, why not just pass cwd=src in the check_output() call, rather than changing it in the parent python process?

Signature · 5 lines

Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Robert Yang

author

2016-10-17 03:45 UTC #6 lore raw

Hi Christopher,

Quoted 70 lines · Christopher Larson

On Fri, Oct 14, 2016 at 3:18 AM, Robert Yang <[email protected] <mailto:[email protected]>> wrote:
On 10/14/2016 05:58 PM, Robert Yang wrote:
On 10/13/2016 06:39 PM, Burton, Ross wrote:
On 13 October 2016 at 10:45, Robert Yang <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote:
This is because "copyhardlinktree(deploy_arch_dir,
arch_channel)" does:
"cp -afl deploy_arch_dir/* arch_channel", while the
deploy_arch_dir/* is
expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2 deploy_arch_dir/pkg3 ..." which causes the "Argument list too long", change cwd to deploy_arch_dir can avoid the error.
Would it be better to change the implementation of copyhardlinktree so that it
*can't* have this problem?
Good idea, thanks, updated in the repo:
git://git.openembedded.org/openembedded-core-contrib
<http://git.openembedded.org/openembedded-core-contrib> rbt/long
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long
<http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long>
Sorry, this patch is not what I wanted to paste, it should be:
diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 06a5af2..1d9cca5 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -80,15 +80,20 @@ def copyhardlinktree(src, dst):
cmd = "cd %s; find . -type d -print | tar --xattrs
--xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) source = ''
+ oldcwd = os.getcwd()
if os.path.isdir(src):
+ dst = os.path.realpath(dst)
+ # chdir() to src to avoid "Argument list too long" error
+ os.chdir(src)
import glob
- if len(glob.glob('%s/.??*' % src)) > 0:
- source = '%s/.??* ' % src
- source = source + '%s/*' % src
+ if len(glob.glob('./.??*')) > 0:
+ source = './.??* '
+ source += './*'
else:
source = src
cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst) subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+ os.chdir(oldcwd)

Quoted 2 lines

Question, why not just pass cwd=src in the check_output() call, rather than changing it in the parent python process?

src can be a file or directory, so we can't simply use cwd=src, I will use os.getcwd() as cwd when it is not a directory. Updated patch in the repo.

git://git.openembedded.org/openembedded-core-contrib rbt/long

Author: Robert Yang <[email protected]>
Date:   Thu Oct 13 01:28:41 2016 -0700

oe/path.py: fix for "Argument list too long"

Fixed when len(TMPDIR) = 410:
$ bitbake core-image-sato-sdk
[snip]
Subprocess output:
/bin/sh: /bin/cp: Argument list too long

ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs [snip]

Signed-off-by Robert Yang <[email protected]>

meta/lib/oe/path.py +6 -4

index 06a5af2..ed7fd1e 100644
@@ -83,12 +83,14 @@ def copyhardlinktree(src, dst):
         if os.path.isdir(src):
             import glob
             if len(glob.glob('%s/.??*' % src)) > 0:
-                source = '%s/.??* ' % src
-            source = source + '%s/*' % src
+                source = './.??* '
+            source += './*'
+            s_dir = src
         else:
             source = src
-        cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
-        subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+            s_dir = os.getcwd()
+        cmd = 'cp -afl --preserve=xattr %s %s' % (source, os.path.realpath(dst))
+        subprocess.check_output(cmd, shell=True, cwd=s_dir, stderr=subprocess.STDOUT)
     else:
         copytree(src, dst)

// Robert

Quoted 6 lines

--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics