Partially prepare for python3 and fix old Packages file reuse
cover Partially prepare for python3 and fix old Packages file reuse
First 5 patches are taken from oe-core.
Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls.
Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer.
And in the end I've found the cause.. printing long fields (e. g. description) with opkg.py was producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk.
And this also fixes Packages.filelist generation.
I don't use python very much, so please review.
And the speedup? Almost 60 times :)
before:
real 20m10.484s user 18m32.335s sys 0m11.885s
after
real 0m24.492s user 0m23.748s sys 0m0.565s
The following changes since commit 002d29bc605d7c2d02e4cf20a43c5277c15f5597:
[opkg-utils] fix install fail problem Thanks for khorben's patch :-) https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +0000)
are available in the git repository at:
git://github.com/shr-project/opkg-utils jansa/pull
https://github.com/shr-project/opkg-utils/tree/jansa/pull
Christopher Larson (1):
Use python via the PATH, rather than hardcoding /usr/bin/python
Enrico Scholz (1):
opkg-make-index: convert mtime to int before comparing it
Khem Raj (1):
opkg.py: Add knowledge about License field in ipk headers
Martin Jansa (17):
opkg.py: use hashlib instead of old md5 module
opkg.py, arfile.py: report which file has wrong format and use the
same test
arfile: decode read lines as ascii string before getting fields from
it
2to3: use subprocess instead of commands
2to3: print fixes
2to3: dictionary fixes
2to3: exception handling fixes
opkg-make-index: show OSError/IOError
opkg.py: use string funtcions directly on string variable opkg.py: catch TypeError when reading control file, to show which one
is failing
arfile: fix test
opkg-make-index: don't use stdout for Packages output and use __str__
instead of __repr__
opkg.py: cast lines from controlfile as string
opkg.py: computeFileMD5 only when we have fn, otherwise fails to read
None file
opkg.py: fix write_package when called from main test opkg.py: use textwrap for description writing
opkg.py: improve test so it prints temporary control file with long
description and then reads it back
Richard Purdie (1):
opkg-make-index: don't error out when some package disappears
Scott Anderson (1):
arfile.py: handle six digit UIDs
arfile.py | 55 +++++++++++++--------- makePackage | 2 +- opkg-compare-indexes | 22 ++++---- opkg-list-fields | 4 +- opkg-make-index | 74 +++++++++++++++++++---------- opkg-show-deps | 28 +++++------ opkg-unbuild | 4 +- opkg-update-index | 5 +- opkg.py | 125 +++++++++++++++++++++++++++++-------------------- 9 files changed, 186 insertions(+), 133 deletions(-)
Signature · 1 line
1.7.8.5
PATCH 1/22 opkg-make-index: don't error out when some package disappears
From: Richard Purdie <[email protected]>
* If we're building an image and some package rebuilds while this is happening some package can be removed/added to the ipk deploy directory. The image will not depend on this package so we can safely ignore these cases rather than error out.
--- opkg-make-index | 25 ++++++++++++++++++++++++- 1 files changed, 24 insertions(+), 1 deletions(-)
opkg-make-index +24 -1
@@ -96,6 +96,7 @@ if (verbose): files=glob(pkg_dir + '/*.opk') + glob(pkg_dir + '/*.deb') + glob(pkg_dir + '/*.ipk') files.sort() for filename in files: + try: basename = os.path.basename(filename) pkg = None fnameStat = os.stat(filename)
@@ -130,6 +131,12 @@ for filename in files: to_morgue(basename) if opt_s: print filename + except OSError: + sys.stderr.write("Package %s disappeared on us!\n" % (filename)) + continue + except IOError: + sys.stderr.write("Package %s disappeared on us!\n" % (filename)) + continue pkgsStampsFile = open(stamplist_filename, "w") for f in pkgsStamps.keys():
@@ -148,6 +155,7 @@ if packages_filename: names = packages.packages.keys() names.sort() for name in names: + try: pkg = packages.packages[name] if locales_dir and pkg.depends: depends = string.split(pkg.depends, ',')
@@ -165,6 +173,13 @@ for name in names: if (verbose): sys.stderr.write("Writing info for package %s\n" % (pkg.package,)) print pkg + except OSError: + sys.stderr.write("Package %s disappeared on us!\n" % (name)) + continue + except IOError: + sys.stderr.write("Package %s disappeared on us!\n" % (name)) + continue + if packages_filename: sys.stdout.close() sys.stdout = old_stdout
@@ -182,7 +197,15 @@ files = {} names = packages.packages.keys() names.sort() for name in names: - for fn in packages[name].get_file_list(): + try: + fnlist = packages[name].get_file_list() + except OSError, e: + sys.stderr.write("Package %s disappeared on us!\n" % (name)) + continue + except IOError, e: + sys.stderr.write("Package %s disappeared on us!\n" % (name)) + continue + for fn in fnlist: (h,t) = os.path.split(fn) if not t: continue if not files.has_key(t): files[t] = name+':'+fn
Signature · 1 line
1.7.8.5
PATCH 2/22 opkg-make-index: convert mtime to int before comparing it
From: Enrico Scholz <[email protected]>
* The st_mtime attribute (which is a float) is compared against a value from the timestamp database, which was stored as an integer there.
* When working on a filesystem with precise timestamps the comparision will fail nearly everytime hence.
* Although it might be possible to enhance the database to store the fractional part too, this will complicate things more than we would gain by this change.
--- opkg-make-index | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
opkg-make-index +1 -1
@@ -101,7 +101,7 @@ for filename in files: pkg = None fnameStat = os.stat(filename) if old_pkg_hash.has_key(basename): - if pkgsStamps.has_key(basename) and fnameStat.st_mtime == pkgsStamps[basename]: + if pkgsStamps.has_key(basename) and int(fnameStat.st_mtime) == pkgsStamps[basename]: if (verbose): sys.stderr.write("Found %s in Packages\n" % (filename,)) pkg = old_pkg_hash[basename]
Signature · 1 line
1.7.8.5
PATCH 3/22 opkg.py: Add knowledge about License field in ipk headers
In reply to Martin JansaFrom: Khem Raj <[email protected]>
--- opkg.py | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
opkg.py +8 -0
@@ -145,6 +145,7 @@ class Package: self.priority = None self.tags = None self.fn = fn + self.license = None if fn: # see if it is deb format
@@ -319,6 +320,12 @@ class Package: def get_section(self, section): return self.section + def set_license(self, license): + self.license = license + + def get_license(self, license): + return self.license + def get_file_list(self): if not self.fn: return []
@@ -425,6 +432,7 @@ class Package: if self.description: out = out + "Description: %s\n" % (self.description) if self.oe: out = out + "OE: %s\n" % (self.oe) if self.homepage: out = out + "HomePage: %s\n" % (self.homepage) + if self.license: out = out + "License: %s\n" % (self.license) if self.priority: out = out + "Priority: %s\n" % (self.priority) if self.tags: out = out + "Tags: %s\n" % (self.tags) out = out + "\n"
Signature · 1 line
1.7.8.5
PATCH 4/22 arfile.py: handle six digit UIDs
In reply to Martin JansaFrom: Scott Anderson <[email protected]>
* Essentially, the problem is that arfile.py is splitting the ar header with white-space instead of fixed-width fields, so two fields would get treated as a single field. This makes things better than before as it now honors the fixed field widths.
--- arfile.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)
arfile.py +6 -1
@@ -75,7 +75,12 @@ class ArFile: l = self.f.readline() if not l: break l = l.replace('`', '') - descriptor = l.split() + # Field lengths from /usr/include/ar.h: + ar_field_lens = [ 16, 12, 6, 6, 8, 10, 2 ] + descriptor = [] + for field_len in ar_field_lens: + descriptor.append(l[:field_len].strip()) + l = l[field_len:] # print descriptor size = int(descriptor[5]) memberName = descriptor[0][:-1]
Signature · 1 line
1.7.8.5
PATCH 5/22 Use python via the PATH, rather than hardcoding /usr/bin/python
In reply to Martin JansaFrom: Christopher Larson <[email protected]>
--- opkg-list-fields | 2 +- opkg-make-index | 2 +- opkg-show-deps | 2 +- opkg-unbuild | 2 +- opkg-update-index | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)
opkg-list-fields +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python import sys, opkg
opkg-make-index +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python import sys, os, posixpath from glob import glob
opkg-show-deps +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python import sys, os, posixpath from glob import glob
opkg-unbuild +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python import sys, os, re
opkg-update-index +1 -1
@@ -1,4 +1,4 @@ -#!/usr/bin/env python2.1 +#!/usr/bin/env python import sys, os from glob import glob
Signature · 1 line
1.7.8.5
PATCH 6/22 opkg.py: use hashlib instead of old md5 module
In reply to Martin Jansa--- opkg.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
opkg.py +3 -3
@@ -36,7 +36,7 @@ import tempfile import os import sys import glob -import md5 +import hashlib import re import string import commands
@@ -188,8 +188,8 @@ class Package: def _computeFileMD5(self): # compute the MD5. f = open(self.fn, "rb") - sum = md5.new() - while 1: + sum = hashlib.md5() + while True: data = f.read(1024) if not data: break sum.update(data)
Signature · 1 line
1.7.8.5
PATCH 7/22 opkg.py, arfile.py: report which file has wrong format and use the same test
In reply to Martin Jansa--- arfile.py | 16 +++++++++------- opkg.py | 11 ++--------- 2 files changed, 11 insertions(+), 16 deletions(-)
arfile.py +9 -7
@@ -40,13 +40,13 @@ class FileSection: class ArFile: - def __init__(self, f): + def __init__(self, f, fn): self.f = f self.directory = {} self.directoryRead = False signature = self.f.readline() - assert signature == "!<arch>\n" + assert signature == "!<arch>\n" or signature == b"!<arch>\n", "Old ipk format (non-deb) is unsupported, file: %s, magic: %s, expected %s" % (fn, signature, "!<arch>") self.directoryOffset = self.f.tell() def open(self, fname):
@@ -100,9 +100,10 @@ class ArFile: if __name__ == "__main__": if None: - f = open(sys.argv[1], "rb") + fn = sys.argv[1] + f = open(fn, "rb") - ar = ArFile(f) + ar = ArFile(f, fn) tarStream = ar.open("data.tar.gz") print "--------" tarStream = ar.open("data.tar.gz")
@@ -120,10 +121,11 @@ if __name__ == "__main__": for f in os.listdir(dir): if not f.endswith(".opk") and not f.endswith(".ipk"): continue - print "=== %s ===" % f - f = open(dir + "/" + f, "rb") + print("=== %s ===" % f) + fn = "%s/%s" % (dir, f) + f = open(fn, "rb") - ar = ArFile(f) + ar = ArFile(f, fn) tarStream = ar.open("control.tar.gz") tarf = tarfile.open("control.tar.gz", "r", tarStream) #tarf.list()
opkg.py +2 -9
@@ -138,7 +138,6 @@ class Package: #self.size = None self.installed_size = None self.filename = None - self.isdeb = 0 self.file_ext_opk = "ipk" self.homepage = None self.oe = None
@@ -150,18 +149,12 @@ class Package: if fn: # see if it is deb format f = open(fn, "rb") - magic = f.read(4) - f.seek(0, 0) - if (magic == "!<ar"): - self.isdeb = 1 - self.filename = os.path.basename(fn) - assert self.isdeb == 1, "Old ipk format (non-deb) is unsupported" ## sys.stderr.write(" extracting control.tar.gz from %s\n"% (fn,)) - ar = arfile.ArFile(f) + ar = arfile.ArFile(f, fn) tarStream = ar.open("control.tar.gz") tarf = tarfile.open("control.tar.gz", "r", tarStream)
@@ -330,7 +323,7 @@ class Package: if not self.fn: return [] f = open(self.fn, "rb") - ar = arfile.ArFile(f) + ar = arfile.ArFile(f, self.fn) tarStream = ar.open("data.tar.gz") tarf = tarfile.open("data.tar.gz", "r", tarStream) self.file_list = tarf.getnames()
Signature · 1 line
1.7.8.5
PATCH 8/22 arfile: decode read lines as ascii string before getting fields from it
In reply to Martin Jansa* python3 returns them as byte sequence
--- arfile.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
arfile.py +1 -0
@@ -74,6 +74,7 @@ class ArFile: if l == "\n": l = self.f.readline() if not l: break + l = l.decode('ascii') l = l.replace('`', '') # Field lengths from /usr/include/ar.h: ar_field_lens = [ 16, 12, 6, 6, 8, 10, 2 ]
Signature · 1 line
1.7.8.5
PATCH 9/22 2to3: use subprocess instead of commands
In reply to Martin Jansa--- opkg-compare-indexes | 6 +++--- opkg-make-index | 5 ++--- opkg-show-deps | 1 - opkg-update-index | 1 - opkg.py | 2 +- 5 files changed, 6 insertions(+), 9 deletions(-)
opkg-compare-indexes +3 -3
@@ -2,7 +2,7 @@ import sys, os from glob import glob -import commands +import subprocess import opkg pkg_dir1 = sys.argv[1]
@@ -40,8 +40,8 @@ for name in names: if pkg1 and pkg2 and pkg1.version != pkg2.version: print "CHANGED: %s from version %s to %s (%s)" % (pkg1.package, pkg1.version, pkg2.version, pkg2.maintainer) cmd = "opkg-diff %s %s > %s " % ((pkg_dir1 + pkg1.filename), (pkg_dir2 + pkg2.filename), (pkg1.package + '.diff')) - print cmd - commands.getstatusoutput(cmd) + print(cmd) + subprocess.call(cmd) if not pkg1: print "NEW: %s version %s (%s)"% (pkg2.package, pkg2.version, pkg2.maintainer) if not pkg2:
opkg-make-index +2 -3
@@ -2,7 +2,7 @@ import sys, os, posixpath from glob import glob -import commands +import subprocess import opkg import getopt import string
@@ -186,8 +186,7 @@ if packages_filename: gzip_filename = ("%s.gz" % packages_filename) tmp_gzip_filename = ("%s.%d" % (gzip_filename, os.getpid())) gzip_cmd = "gzip -9c < %s > %s" % (tmp_packages_filename, tmp_gzip_filename) - (rc, outtext) = commands.getstatusoutput(gzip_cmd) - print outtext + rc = subprocess.check_output(gzip_cmd, shell=True) os.rename(tmp_packages_filename, packages_filename) os.rename(tmp_gzip_filename, gzip_filename)
opkg-show-deps +0 -1
@@ -2,7 +2,6 @@ import sys, os, posixpath from glob import glob -import commands import opkg import getopt import string
opkg-update-index +0 -1
@@ -2,7 +2,6 @@ import sys, os from glob import glob -import commands import opkg pkg_dir=sys.argv[1]
opkg.py +1 -1
@@ -39,7 +39,7 @@ import glob import hashlib import re import string -import commands +import subprocess from stat import ST_SIZE import arfile import tarfile
Signature · 1 line
1.7.8.5
PATCH 10/22 2to3: print fixes
In reply to Martin Jansa--- arfile.py | 20 ++++++++++---------- makePackage | 2 +- opkg-compare-indexes | 6 +++--- opkg-list-fields | 2 +- opkg-make-index | 8 ++++---- opkg-show-deps | 2 +- opkg-unbuild | 2 +- opkg.py | 24 ++++++++++++------------ 8 files changed, 33 insertions(+), 33 deletions(-)
arfile.py +10 -10
@@ -20,7 +20,7 @@ class FileSection: self.seek(0, 0) def seek(self, offset, whence = 0): -# print "seek(%x, %d)" % (offset, whence) +# print("seek(%x, %d)" % (offset, whence)) if whence == 0: return self.f.seek(offset + self.offset, whence) elif whence == 1:
@@ -31,11 +31,11 @@ class FileSection: assert False def tell(self): -# print "tell()" +# print("tell()") return self.f.tell() - self.offset def read(self, size = -1): -# print "read(%d)" % size +# print("read(%d)" % size) return self.f.read(size) class ArFile:
@@ -82,11 +82,11 @@ class ArFile: for field_len in ar_field_lens: descriptor.append(l[:field_len].strip()) l = l[field_len:] -# print descriptor +# print(descriptor) size = int(descriptor[5]) memberName = descriptor[0][:-1] self.directory[memberName] = descriptor + [self.f.tell()] -# print "read:", memberName +# print(("read:", memberName)) if memberName == fname: # Record directory offset to start from next time self.directoryOffset = self.f.tell() + size
@@ -96,7 +96,7 @@ class ArFile: if size % 2: size = size + 1 data = self.f.seek(size, 1) -# print hex(f.tell()) +# print(hex(self.f.tell())) if __name__ == "__main__":
@@ -106,11 +106,11 @@ if __name__ == "__main__": ar = ArFile(f, fn) tarStream = ar.open("data.tar.gz") - print "--------" + print("--------") tarStream = ar.open("data.tar.gz") - print "--------" + print("--------") tarStream = ar.open("control.tar.gz") - print "--------" + print("--------") tarStream = ar.open("control.tar.gz2") sys.exit(0)
@@ -132,4 +132,4 @@ if __name__ == "__main__": #tarf.list() f2 = tarf.extractfile("control") - print f2.read() + print(f2.read())
makePackage +1 -1
@@ -11,4 +11,4 @@ import opkg fn = sys.argv[1] pkg = opkg.Package(fn) -print pkg +print(pkg)
opkg-compare-indexes +3 -3
@@ -38,12 +38,12 @@ for name in names: if pkgs2.packages.has_key(name): pkg2 = pkgs2.packages[name] if pkg1 and pkg2 and pkg1.version != pkg2.version: - print "CHANGED: %s from version %s to %s (%s)" % (pkg1.package, pkg1.version, pkg2.version, pkg2.maintainer) + print("CHANGED: %s from version %s to %s (%s)" % (pkg1.package, pkg1.version, pkg2.version, pkg2.maintainer)) cmd = "opkg-diff %s %s > %s " % ((pkg_dir1 + pkg1.filename), (pkg_dir2 + pkg2.filename), (pkg1.package + '.diff')) print(cmd) subprocess.call(cmd) if not pkg1: - print "NEW: %s version %s (%s)"% (pkg2.package, pkg2.version, pkg2.maintainer) + print("NEW: %s version %s (%s)"% (pkg2.package, pkg2.version, pkg2.maintainer)) if not pkg2: - print "DELETE: %s version %s (%s)"% (pkg1.package, pkg1.version, pkg1.maintainer) + print("DELETE: %s version %s (%s)"% (pkg1.package, pkg1.version, pkg1.maintainer))
opkg-list-fields +1 -1
@@ -9,5 +9,5 @@ def usage(): if (len(sys.argv) < 2): usage() -print opkg.Package(sys.argv[1]) +print(opkg.Package(sys.argv[1]))
opkg-make-index +4 -4
@@ -125,12 +125,12 @@ for filename in files: if opt_m: to_morgue(old_filename) if opt_s: - print pkg_dir + "/" + old_filename + print(("%s/%s" % (pkg_dir, old_filename))) else: if opt_m: to_morgue(basename) if opt_s: - print filename + print(filename) except OSError: sys.stderr.write("Package %s disappeared on us!\n" % (filename)) continue
@@ -172,7 +172,7 @@ for name in names: continue if (verbose): sys.stderr.write("Writing info for package %s\n" % (pkg.package,)) - print pkg + print(pkg) except OSError: sys.stderr.write("Package %s disappeared on us!\n" % (name)) continue
@@ -216,7 +216,7 @@ if filelist_filename: names = files.keys() names.sort() for name in names: - print name,files[name] + print((name,files[name])) sys.stdout.close() if posixpath.exists(filelist_filename): os.unlink(filelist_filename)
opkg-show-deps +1 -1
@@ -83,5 +83,5 @@ for root in remaining_args: recurse(p) for pkg in required.keys(): - print pkg + print(pkg)
opkg-unbuild +1 -1
@@ -3,7 +3,7 @@ import sys, os, re if (len(sys.argv) == 0): - print 'usage: %s: package.opk' % sys.argv[0] + print('usage: %s: package.opk' % sys.argv[0]) sys.exit(1) for filename in sys.argv[1:]:
opkg.py +12 -12
@@ -56,10 +56,10 @@ class Version: while 1: ## first look for non-numeric version component selfm = re.match('([^0-9]*)(.*)', selfversion) - #print 'selfm', selfm.groups() + #print(('selfm', selfm.groups())) (selfalpha, selfversion) = selfm.groups() refm = re.match('([^0-9]*)(.*)', refversion) - #print 'refm', refm.groups() + #print(('refm', refm.groups()) (refalpha, refversion) = refm.groups() if (selfalpha > refalpha): return 1
@@ -68,8 +68,8 @@ class Version: ## now look for numeric version component (selfnum, selfversion) = re.match('([0-9]*)(.*)', selfversion).groups() (refnum, refversion) = re.match('([0-9]*)(.*)', refversion).groups() - #print 'selfnum', selfnum, selfversion - #print 'refnum', refnum, refversion + #print(('selfnum', selfnum, selfversion) + #print(('refnum', refnum, refversion) if (selfnum != ''): selfnum = int(selfnum) else:
@@ -93,12 +93,12 @@ class Version: else: self_ver_comps = re.match(r"(.+?)(-r.+)?$", self.version) ref_ver_comps = re.match(r"(.+?)(-r.+)?$", ref.version) - #print (self_ver_comps.group(1), self_ver_comps.group(2)) - #print (ref_ver_comps.group(1), ref_ver_comps.group(2)) + #print((self_ver_comps.group(1), self_ver_comps.group(2))) + #print((ref_ver_comps.group(1), ref_ver_comps.group(2))) r = self._versioncompare(self_ver_comps.group(1), ref_ver_comps.group(1)) if r == 0: r = self._versioncompare(self_ver_comps.group(2), ref_ver_comps.group(2)) - #print "compare: %s vs %s = %d" % (self, ref, r) + #print("compare: %s vs %s = %d" % (self, ref, r)) return r def __str__(self):
@@ -220,7 +220,7 @@ class Package: elif self.__dict__.has_key(name): self.__dict__[name] = value else: - print "Lost field %s, %s" % (name,value) + print("Lost field %s, %s" % (name,value)) pass if line and line[0] == '\n':
@@ -391,9 +391,9 @@ class Package: def compare_version(self, ref): """Compare package versions of self and ref""" if not self.version: - print 'No version for package %s' % self.package + print('No version for package %s' % self.package) if not ref.version: - print 'No version for package %s' % ref.package + print('No version for package %s' % ref.package) if not self.parsed_version: self.parsed_version = parse_version(self.version) if not ref.parsed_version:
@@ -499,9 +499,9 @@ if __name__ == "__main__": package.set_depends("libc") package.set_description("A test of the APIs.") - print "<" + print("<") sys.stdout.write(package) - print ">" + print(">") package.write_package("/tmp")
Signature · 1 line
1.7.8.5
PATCH 11/22 2to3: dictionary fixes
In reply to Martin Jansa--- arfile.py | 2 +- opkg-compare-indexes | 10 +++++----- opkg-make-index | 18 +++++++++--------- opkg-show-deps | 12 ++++++------ opkg-update-index | 2 +- opkg.py | 8 ++++---- 6 files changed, 26 insertions(+), 26 deletions(-)
arfile.py +1 -1
@@ -50,7 +50,7 @@ class ArFile: self.directoryOffset = self.f.tell() def open(self, fname): - if self.directory.has_key(fname): + if fname in self.directory: return FileSection(self.f, self.directory[fname][-1], int(self.directory[fname][5])) if self.directoryRead:
opkg-compare-indexes +5 -5
@@ -18,8 +18,8 @@ pkgs1.read_packages_file(pkg_dir1 + '/Packages') pkgs2 = opkg.Packages() pkgs2.read_packages_file(pkg_dir2 + '/Packages') -names1 = pkgs1.packages.keys() -names2 = pkgs2.packages.keys() +names1 = list(pkgs1.packages.keys()) +names2 = list(pkgs2.packages.keys()) ## union of the two names lists pkgs = {}
@@ -28,14 +28,14 @@ for name in names1: for name in names2: pkgs[name] = pkgs2.packages[name] -names = pkgs.keys() +names = list(pkgs.keys()) names.sort() for name in names: pkg1 = None pkg2 = None - if pkgs1.packages.has_key(name): + if name in pkgs1.packages: pkg1 = pkgs1.packages[name] - if pkgs2.packages.has_key(name): + if name in pkgs2.packages: pkg2 = pkgs2.packages[name] if pkg1 and pkg2 and pkg1.version != pkg2.version: print("CHANGED: %s from version %s to %s (%s)" % (pkg1.package, pkg1.version, pkg2.version, pkg2.maintainer))
opkg-make-index +9 -9
@@ -79,7 +79,7 @@ if old_filename: sys.stderr.write("Reading package list from " + old_filename + "\n") old_packages = opkg.Packages() old_packages.read_packages_file(old_filename) - for k in old_packages.packages.keys(): + for k in list(old_packages.packages.keys()): p = old_packages.packages[k] old_pkg_hash[p.filename] = p try:
@@ -100,8 +100,8 @@ for filename in files: basename = os.path.basename(filename) pkg = None fnameStat = os.stat(filename) - if old_pkg_hash.has_key(basename): - if pkgsStamps.has_key(basename) and int(fnameStat.st_mtime) == pkgsStamps[basename]: + if basename in old_pkg_hash: + if basename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[basename]: if (verbose): sys.stderr.write("Found %s in Packages\n" % (filename,)) pkg = old_pkg_hash[basename]
@@ -113,7 +113,7 @@ for filename in files: sys.stderr.write("Reading info for package %s\n" % (filename,)) pkg = opkg.Package(filename) pkg_key = ("%s:%s" % (pkg.package, pkg.architecture)) - if (packages.packages.has_key(pkg_key)): + if (pkg_key in packages.packages): old_filename = packages.packages[pkg_key].filename else: old_filename = ""
@@ -139,7 +139,7 @@ for filename in files: continue pkgsStampsFile = open(stamplist_filename, "w") -for f in pkgsStamps.keys(): +for f in list(pkgsStamps.keys()): pkgsStampsFile.write("%d %s\n" % (pkgsStamps[f], f)) pkgsStampsFile.close()
@@ -152,7 +152,7 @@ if packages_filename: old_stdout = sys.stdout tmp_packages_filename = ("%s.%d" % (packages_filename, os.getpid())) sys.stdout = open(tmp_packages_filename, "w") -names = packages.packages.keys() +names = list(packages.packages.keys()) names.sort() for name in names: try:
@@ -193,7 +193,7 @@ if packages_filename: if verbose: sys.stderr.write("Generate Packages.filelist file\n") files = {} -names = packages.packages.keys() +names = list(packages.packages.keys()) names.sort() for name in names: try:
@@ -207,13 +207,13 @@ for name in names: for fn in fnlist: (h,t) = os.path.split(fn) if not t: continue - if not files.has_key(t): files[t] = name+':'+fn + if t not in files: files[t] = name+':'+fn else: files[t] = files[t] + ',' + name+':'+fn if filelist_filename: tmp_filelist_filename = ("%s.%d" % (filelist_filename, os.getpid())) sys.stdout = open(tmp_filelist_filename, "w") - names = files.keys() + names = list(files.keys()) names.sort() for name in names: print((name,files[name]))
opkg-show-deps +6 -6
@@ -40,21 +40,21 @@ def split_list(str): r.append(ii) return r -for i in packages.packages.keys(): +for i in list(packages.packages.keys()): p = packages.packages[i] - if not provider_hash.has_key(p.package): + if p.package not in provider_hash: provider_hash[p.package] = [] provider_hash[p.package].append(p) if p.provides: provides = string.split(p.provides, ",") for prov in provides: prov = string.strip(prov) - if not provider_hash.has_key(prov): + if prov not in provider_hash: provider_hash[prov] = [] provider_hash[prov].append(p) def find_package(name): - if provider_hash.has_key(name): + if name in provider_hash: return provider_hash[name] return None
@@ -68,7 +68,7 @@ def recurse(pkg): newpkgs = find_package(dep) if newpkgs: for newpkg in newpkgs: - if required.has_key(newpkg.package): + if newpkg.package in required: return recurse(newpkgs[0]) else:
@@ -82,6 +82,6 @@ for root in remaining_args: for p in pkgs: recurse(p) -for pkg in required.keys(): +for pkg in list(required.keys()): print(pkg)
opkg-update-index +1 -1
@@ -15,7 +15,7 @@ packages = opkg.Packages() packages.read_packages_file(pkg_dir + '/Packages') -names = packages.packages.keys() +names = list(packages.packages.keys()) packages.add_package(opkg.Package(pkg_filename))
opkg.py +4 -4
@@ -217,7 +217,7 @@ class Package: self.size = int(value) elif name == 'md5sum': self.md5 = value - elif self.__dict__.has_key(name): + elif name in self.__dict__: self.__dict__[name] = value else: print("Lost field %s, %s" % (name,value))
@@ -447,7 +447,7 @@ class Packages: package = pkg.package arch = pkg.architecture name = ("%s:%s" % (package, arch)) - if (not self.packages.has_key(name)): + if (name not in self.packages): self.packages[name] = pkg if pkg.compare_version(self.packages[name]) >= 0:
@@ -470,14 +470,14 @@ class Packages: def write_packages_file(self, fn): f = open(fn, "w") - names = self.packages.keys() + names = list(self.packages.keys()) names.sort() for name in names: f.write(self.packages[name].__repr__()) return def keys(self): - return self.packages.keys() + return list(self.packages.keys()) def __getitem__(self, key): return self.packages[key]
Signature · 1 line
1.7.8.5
PATCH 12/22 2to3: exception handling fixes
In reply to Martin Jansa--- arfile.py | 4 ++-- opkg-make-index | 4 ++-- opkg.py | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-)
arfile.py +2 -2
@@ -54,11 +54,11 @@ class ArFile: return FileSection(self.f, self.directory[fname][-1], int(self.directory[fname][5])) if self.directoryRead: - raise IOError, (2, "AR member not found: " + fname) + raise IOError("AR member not found: " + fname) f = self._scan(fname) if f == None: - raise IOError, (2, "AR member not found: " + fname) + raise IOError("AR member not found: " + fname) return f
opkg-make-index +2 -2
@@ -198,10 +198,10 @@ names.sort() for name in names: try: fnlist = packages[name].get_file_list() - except OSError, e: + except OSError as e: sys.stderr.write("Package %s disappeared on us!\n" % (name)) continue - except IOError, e: + except IOError as e: sys.stderr.write("Package %s disappeared on us!\n" % (name)) continue for fn in fnlist:
opkg.py +1 -1
@@ -176,7 +176,7 @@ class Package: elif name == 'size': return self._get_file_size() else: - raise AttributeError, name + raise AttributeError(name) def _computeFileMD5(self): # compute the MD5.
Signature · 1 line
1.7.8.5
PATCH 13/22 opkg-make-index: show OSError/IOError
In reply to Martin Jansa--- opkg-make-index | 20 ++++++++++---------- 1 files changed, 10 insertions(+), 10 deletions(-)
opkg-make-index +10 -10
@@ -131,11 +131,11 @@ for filename in files: to_morgue(basename) if opt_s: print(filename) - except OSError: - sys.stderr.write("Package %s disappeared on us!\n" % (filename)) + except OSError as e: + sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (filename, e)) continue - except IOError: - sys.stderr.write("Package %s disappeared on us!\n" % (filename)) + except IOError as e: + sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (filename, e)) continue pkgsStampsFile = open(stamplist_filename, "w")
@@ -173,11 +173,11 @@ for name in names: if (verbose): sys.stderr.write("Writing info for package %s\n" % (pkg.package,)) print(pkg) - except OSError: - sys.stderr.write("Package %s disappeared on us!\n" % (name)) + except OSError as e: + sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) continue - except IOError: - sys.stderr.write("Package %s disappeared on us!\n" % (name)) + except IOError as e: + sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) continue if packages_filename:
@@ -199,10 +199,10 @@ for name in names: try: fnlist = packages[name].get_file_list() except OSError as e: - sys.stderr.write("Package %s disappeared on us!\n" % (name)) + sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) continue except IOError as e: - sys.stderr.write("Package %s disappeared on us!\n" % (name)) + sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) continue for fn in fnlist: (h,t) = os.path.split(fn)
Signature · 1 line
1.7.8.5
PATCH 14/22 opkg.py: use string funtcions directly on string variable
In reply to Martin Jansa--- opkg-make-index | 3 +-- opkg-show-deps | 11 +++++------ opkg.py | 5 ++--- 3 files changed, 8 insertions(+), 11 deletions(-)
opkg-make-index +1 -2
@@ -5,7 +5,6 @@ from glob import glob import subprocess import opkg import getopt -import string import re verbose = 0
@@ -158,7 +157,7 @@ for name in names: try: pkg = packages.packages[name] if locales_dir and pkg.depends: - depends = string.split(pkg.depends, ',') + depends = pkg.depends.split(',') locale = None for d in depends: m = re.match('.*virtual-locale-([a-zA-Z]+).*', d)
opkg-show-deps +5 -6
@@ -4,7 +4,6 @@ import sys, os, posixpath from glob import glob import opkg import getopt -import string import re verbose = 0
@@ -32,11 +31,11 @@ provider_hash = {} def split_list(str): r = [] - l = string.split(str, ",") + l = str.split(",") for i in l: - ll = string.split(i, "|") + ll = i.split("|") for ii in ll: - ii = string.strip(ii) + ii = ii.strip() r.append(ii) return r
@@ -46,9 +45,9 @@ for i in list(packages.packages.keys()): provider_hash[p.package] = [] provider_hash[p.package].append(p) if p.provides: - provides = string.split(p.provides, ",") + provides = p.provides.split(",") for prov in provides: - prov = string.strip(prov) + prov = prov.strip() if prov not in provider_hash: provider_hash[prov] = [] provider_hash[prov].append(p)
opkg.py +2 -3
@@ -38,7 +38,6 @@ import sys import glob import hashlib import re -import string import subprocess from stat import ST_SIZE import arfile
@@ -203,10 +202,10 @@ class Package: line = control.readline() while 1: if not line: break - line = string.rstrip(line) + line = line.rstrip() lineparts = re.match(r'([\w-]*?):\s*(.*)', line) if lineparts: - name = string.lower(lineparts.group(1)) + name = lineparts.group(1).lower() value = lineparts.group(2) while 1: line = control.readline()
Signature · 1 line
1.7.8.5
PATCH 15/22 opkg.py: catch TypeError when reading control file, to show which one is failing
In reply to Martin Jansa--- opkg.py | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-)
opkg.py +10 -3
@@ -161,7 +161,10 @@ class Package: control = tarf.extractfile("control") except KeyError: control = tarf.extractfile("./control") - self.read_control(control) + try: + self.read_control(control) + except TypeError as e: + sys.stderr.write("Cannot read control file '%s' - %s\n" % (fn, e)) control.close() self.scratch_dir = None
@@ -457,9 +460,13 @@ class Packages: def read_packages_file(self, fn): f = open(fn, "r") - while 1: + while True: pkg = Package() - pkg.read_control(f) + try: + pkg.read_control(f) + except TypeError as e: + sys.stderr.write("Cannot read control file '%s' - %s\n" % (fn, e)) + continue if pkg.get_package(): self.add_package(pkg) else:
Signature · 1 line
1.7.8.5
PATCH 16/22 arfile: fix test
In reply to Martin Jansa--- arfile.py | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
arfile.py +4 -1
@@ -131,5 +131,8 @@ if __name__ == "__main__": tarf = tarfile.open("control.tar.gz", "r", tarStream) #tarf.list() - f2 = tarf.extractfile("control") + try: + f2 = tarf.extractfile("control") + except KeyError: + f2 = tarf.extractfile("./control") print(f2.read())
Signature · 1 line
1.7.8.5
PATCH 17/22 opkg-make-index: don't use stdout for Packages output and use __str__ instead of __repr__
In reply to Martin Jansa--- opkg-make-index | 17 +++++++++-------- opkg.py | 4 ++-- 2 files changed, 11 insertions(+), 10 deletions(-)
opkg-make-index +9 -8
@@ -148,9 +148,8 @@ if opt_s: if verbose: sys.stderr.write("Generating Packages file\n") if packages_filename: - old_stdout = sys.stdout tmp_packages_filename = ("%s.%d" % (packages_filename, os.getpid())) - sys.stdout = open(tmp_packages_filename, "w") + pkgsFile = open(tmp_packages_filename, "w") names = list(packages.packages.keys()) names.sort() for name in names:
@@ -171,7 +170,10 @@ for name in names: continue if (verbose): sys.stderr.write("Writing info for package %s\n" % (pkg.package,)) - print(pkg) + if packages_filename: + pkgsFile.write(str(pkg)) + else: + print(pkg) except OSError as e: sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) continue
@@ -180,8 +182,7 @@ for name in names: continue if packages_filename: - sys.stdout.close() - sys.stdout = old_stdout + pkgsFile.close() gzip_filename = ("%s.gz" % packages_filename) tmp_gzip_filename = ("%s.%d" % (gzip_filename, os.getpid())) gzip_cmd = "gzip -9c < %s > %s" % (tmp_packages_filename, tmp_gzip_filename)
@@ -211,12 +212,12 @@ for name in names: if filelist_filename: tmp_filelist_filename = ("%s.%d" % (filelist_filename, os.getpid())) - sys.stdout = open(tmp_filelist_filename, "w") + f = open(tmp_filelist_filename, "w") names = list(files.keys()) names.sort() for name in names: - print((name,files[name])) - sys.stdout.close() + f.write("%s %s\n" % (name, files[name])) + f.close() if posixpath.exists(filelist_filename): os.unlink(filelist_filename) os.rename(tmp_filelist_filename, filelist_filename)
opkg.py +2 -2
@@ -402,7 +402,7 @@ class Package: ref.parsed_version = parse_version(ref.version) return self.parsed_version.compare(ref.parsed_version) - def __repr__(self): + def __str__(self): out = "" # XXX - Some checks need to be made, and some exceptions
@@ -506,7 +506,7 @@ if __name__ == "__main__": package.set_description("A test of the APIs.") print("<") - sys.stdout.write(package) + sys.stdout.write(str(package)) print(">") package.write_package("/tmp")
Signature · 1 line
1.7.8.5
PATCH 18/22 opkg.py: cast lines from controlfile as string
In reply to Martin Jansa--- opkg.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
opkg.py +1 -1
@@ -206,7 +206,7 @@ class Package: while 1: if not line: break line = line.rstrip() - lineparts = re.match(r'([\w-]*?):\s*(.*)', line) + lineparts = re.match(r'([\w-]*?):\s*(.*)', str(line)) if lineparts: name = lineparts.group(1).lower() value = lineparts.group(2)
Signature · 1 line
1.7.8.5
PATCH 19/22 opkg.py: computeFileMD5 only when we have fn, otherwise fails to read None file
In reply to Martin Jansa--- opkg.py | 19 +++++++++++-------- 1 files changed, 11 insertions(+), 8 deletions(-)
opkg.py +11 -8
@@ -182,14 +182,17 @@ class Package: def _computeFileMD5(self): # compute the MD5. - f = open(self.fn, "rb") - sum = hashlib.md5() - while True: - data = f.read(1024) - if not data: break - sum.update(data) - f.close() - self.md5 = sum.hexdigest() + if not self.fn: + self.md5 = 'Unknown' + else: + f = open(self.fn, "rb") + sum = hashlib.md5() + while True: + data = f.read(1024) + if not data: break + sum.update(data) + f.close() + self.md5 = sum.hexdigest() def _get_file_size(self): if not self.fn:
Signature · 1 line
1.7.8.5
PATCH 20/22 opkg.py: fix write_package when called from main test
In reply to Martin Jansa* there is no self.meta_dir before self._setup_scratch_area
* control file needs to be closed before packing it with tar, otherwise it could be empty
--- opkg.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
opkg.py +3 -3
@@ -344,11 +344,11 @@ class Package: return self.file_ext_opk def write_package(self, dirname): - buf = self.render_control() + self._setup_scratch_area() file = open("%s/control" % self.meta_dir, 'w') - file.write(buf) + file.write(str(self)) + file.close() - self._setup_scratch_area() cmd = "cd %s ; tar cvfz %s/control.tar.gz control" % (self.meta_dir, self.scratch_dir)
Signature · 1 line
1.7.8.5
PATCH 21/22 opkg.py: use textwrap for description writing
In reply to Martin Jansa* description field in oe-core sometimes has extra line-feeds and is long
* extra line-feeds breaks read_control, because empty line means next Package
* long descriptions should be wrapped and properly indented, so they are parsed back properly
--- opkg.py | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
opkg.py +4 -1
@@ -42,6 +42,7 @@ import subprocess from stat import ST_SIZE import arfile import tarfile +import textwrap class Version: """A class for holding parsed package version information."""
@@ -427,7 +428,9 @@ class Package: if self.installed_size: out = out + "InstalledSize: %d\n" % int(self.installed_size) if self.filename: out = out + "Filename: %s\n" % (self.filename) if self.source: out = out + "Source: %s\n" % (self.source) - if self.description: out = out + "Description: %s\n" % (self.description) + if self.description: + printable_description = textwrap.dedent(self.description).strip() + out = out + "Description: %s\n" % textwrap.fill(printable_description, width=74, initial_indent=' ', subsequent_indent=' ') if self.oe: out = out + "OE: %s\n" % (self.oe) if self.homepage: out = out + "HomePage: %s\n" % (self.homepage) if self.license: out = out + "License: %s\n" % (self.license)
Signature · 1 line
1.7.8.5
PATCH 22/22 opkg.py: improve test so it prints temporary control file with long description and then reads it back
In reply to Martin Jansa--- opkg.py | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-)
opkg.py +11 -1
@@ -509,11 +509,21 @@ if __name__ == "__main__": package.set_architecture("arm") package.set_maintainer("Testing <[email protected]>") package.set_depends("libc") - package.set_description("A test of the APIs.") + package.set_description("A test of the APIs. And very long descriptions so often used in oe-core\nfoo\n\n\nbar") print("<") sys.stdout.write(str(package)) print(">") + f = open("/tmp/control", "w") + f.write(str(package)) + f.close() + + f = open("/tmp/control", "r") + package2 = Package() + package2.read_control(f) + print("<") + sys.stdout.write(str(package2)) + print(">") package.write_package("/tmp")
Signature · 1 line
1.7.8.5
PATCH 23/23 opkg-make-index: generate complete filelist
In reply to Martin JansaFrom: Martin Jansa <[email protected]>
* when '-l Packages.filelist' option is used together with '-r Packages.old', then only 'new' packages are processed to create Packages.filelist packages found in Packages.old doesn't have fn set so get_file_list() was returning empty
* now added get_file_list_dir() looks for filename in pkg_dir and if it finds correct .ipk file it will use that to read its filelist, but it also means that it will always unpack *all* Packages - very slow.
* it would be nice to add new param for Packages.filelist.old and then filter filelist for packages used from Packages.old and merge it together with new Packages.filelist, but that's more difficult because of files structure.
--- opkg-make-index | 64 +++++++++++++++++++++++++++++------------------------- opkg.py | 16 +++++++++++++ 2 files changed, 50 insertions(+), 30 deletions(-)
opkg-make-index +34 -30
@@ -190,34 +190,38 @@ if packages_filename: os.rename(tmp_packages_filename, packages_filename) os.rename(tmp_gzip_filename, gzip_filename) -if verbose: - sys.stderr.write("Generate Packages.filelist file\n") -files = {} -names = list(packages.packages.keys()) -names.sort() -for name in names: - try: - fnlist = packages[name].get_file_list() - except OSError as e: - sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) - continue - except IOError as e: - sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) - continue - for fn in fnlist: - (h,t) = os.path.split(fn) - if not t: continue - if t not in files: files[t] = name+':'+fn - else: files[t] = files[t] + ',' + name+':'+fn - if filelist_filename: - tmp_filelist_filename = ("%s.%d" % (filelist_filename, os.getpid())) - f = open(tmp_filelist_filename, "w") - names = list(files.keys()) - names.sort() - for name in names: - f.write("%s %s\n" % (name, files[name])) - f.close() - if posixpath.exists(filelist_filename): - os.unlink(filelist_filename) - os.rename(tmp_filelist_filename, filelist_filename) + if verbose: + sys.stderr.write("Generate Packages.filelist file\n") + files = {} + names = list(packages.packages.keys()) + names.sort() + for name in names: + try: + if verbose: + sys.stderr.write("Reading filelist for package '%s'\n" % name) +# sys.stderr.write("Package for name '%s':\n'%s'\n" % (name, packages[name])) + fnlist = packages[name].get_file_list_dir(pkg_dir) +# sys.stderr.write("Filelist for package '%s': '%s'\n" % (name, fnlist)) + except OSError as e: + sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) + continue + except IOError as e: + sys.stderr.write("Package %s disappeared on us!\n(%s)\n" % (name, e)) + continue + for fn in fnlist: + (h,t) = os.path.split(fn) + if not t: continue + if t not in files: files[t] = name+':'+fn + else: files[t] = files[t] + ',' + name+':'+fn + + tmp_filelist_filename = ("%s.%d" % (filelist_filename, os.getpid())) + f = open(tmp_filelist_filename, "w") + names = list(files.keys()) + names.sort() + for name in names: + f.write("%s %s\n" % (name, files[name])) + f.close() + if posixpath.exists(filelist_filename): + os.unlink(filelist_filename) + os.rename(tmp_filelist_filename, filelist_filename)
opkg.py +16 -0
@@ -325,8 +325,24 @@ class Package: def get_license(self, license): return self.license + def get_file_list_dir(self, directory): + if not self.fn: + try: + cmd = "find %s -name %s | head -n 1" % (directory, self.filename) + rc = subprocess.check_output(cmd, shell=True) + newfn = str(rc).split()[0] +# sys.stderr.write("Package '%s' with empty fn and filename is '%s' was found in '%s', updating fn\n" % (self.package, self.filename, newfn)) + self.fn = newfn + except OSError as e: + sys.stderr.write("Cannot find current fn for package '%s' filename '%s' in dir '%s'\n(%s)\n" % (self.package, self.filename, directory, e)) + except IOError as e: + sys.stderr.write("Cannot find current fn for package '%s' filename '%s' in dir '%s'\n(%s)\n" % (self.package, self.filename, directory, e)) + return self.get_file_list() + + def get_file_list(self): if not self.fn: + sys.stderr.write("Package '%s' has empty fn returning empty filelist\n" % (self.package)) return [] f = open(self.fn, "rb") ar = arfile.ArFile(f, self.fn)
Signature · 1 line
1.7.8.5
Martin,
Just wanted to let you know that this was received, I am holding off doing anything about it right now until we get through the current release process, this will be build and tested with early in the 1.3 as appropriate.
Thanks
Sau!
Quoted 87 lines · Martin Jansa
First 5 patches are taken from oe-core.
Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls.
Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer.
And in the end I've found the cause.. printing long fields (e. g. description) with
opkg.pywas producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk.And this also fixes Packages.filelist generation.
I don't use python very much, so please review.
And the speedup? Almost 60 times :)
before:real 20m10.484s user 18m32.335s sys 0m11.885safter
real 0m24.492s user 0m23.748s sys 0m0.565sThe following changes since commit
002d29bc605d7c2d02e4cf20a43c5277c15f5597:[opkg-utils] fix install fail problem Thanks for khorben's patch :-) https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +0000)are available in the git repository at:
git://github.com/shr-project/opkg-utilsjansa/pull
https://github.com/shr-project/opkg-utils/tree/jansa/pullChristopher Larson (1):
Use python via the PATH, rather than hardcoding/usr/bin/pythonEnrico Scholz (1):
opkg-make-index: convert mtime to int before comparing itKhem Raj (1):
opkg.py: Add knowledge about License field in ipk headersMartin Jansa (17):
opkg.py: use hashlib instead of old md5 module
opkg.py,arfile.py: report which file has wrong format and use the
same test
arfile: decode read lines as ascii string before getting fields from
it
2to3: use subprocess instead of commands
2to3: print fixes
2to3: dictionary fixes
2to3: exception handling fixes
opkg-make-index: show OSError/IOError
opkg.py: use string funtcions directly on string variableopkg.py: catch TypeError when reading control file, to show which one
is failing
arfile: fix test
opkg-make-index: don't use stdout for Packages output and use__str__
instead of__repr__
opkg.py: cast lines from controlfile as string
opkg.py: computeFileMD5 only when we have fn, otherwise fails to read
None file
opkg.py: fixwrite_packagewhen called from main testopkg.py: use textwrap for description writing
opkg.py: improve test so it prints temporary control file with long
description and then reads it backRichard Purdie (1):
opkg-make-index: don't error out when some package disappearsScott Anderson (1):
arfile.py: handle six digit UIDsarfile.py | 55 +++++++++++++--------- makePackage | 2 +- opkg-compare-indexes | 22 ++++---- opkg-list-fields | 4 +- opkg-make-index | 74 +++++++++++++++++++---------- opkg-show-deps | 28 +++++------ opkg-unbuild | 4 +- opkg-update-index | 5 +- opkg.py | 125 +++++++++++++++++++++++++++++-------------------- 9 files changed, 186 insertions(+), 133 deletions(-)
Quoted 7 lines · Saul Wold
Martin,
Just wanted to let you know that this was received, I am holding off doing anything about it right now until we get through the current release process, this will be build and tested with early in the 1.3 as appropriate.
Hi,
can you please apply at least first patchset (just import of patches we already had), someone wanted to switch opkg-utils in oe-classic from svn to this git repo, so it would be nice to just use correct SRCREV instead of importing those local patches.
Or import everything as it won't be used until we change SRCREV in recipe and we'll change SRCREV in 1.3 cycle.
Thanks!
Quoted 93 lines
Thanks
Sau!Quoted 87 lines · Martin Jansa
First 5 patches are taken from oe-core.
Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls.
Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer.
And in the end I've found the cause.. printing long fields (e. g. description) with
opkg.pywas producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk.And this also fixes Packages.filelist generation.
I don't use python very much, so please review.
And the speedup? Almost 60 times :)
before:real 20m10.484s user 18m32.335s sys 0m11.885safter
real 0m24.492s user 0m23.748s sys 0m0.565sThe following changes since commit
002d29bc605d7c2d02e4cf20a43c5277c15f5597:[opkg-utils] fix install fail problem Thanks for khorben's patch :-) https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +0000)are available in the git repository at:
git://github.com/shr-project/opkg-utilsjansa/pull
https://github.com/shr-project/opkg-utils/tree/jansa/pullChristopher Larson (1):
Use python via the PATH, rather than hardcoding/usr/bin/pythonEnrico Scholz (1):
opkg-make-index: convert mtime to int before comparing itKhem Raj (1):
opkg.py: Add knowledge about License field in ipk headersMartin Jansa (17):
opkg.py: use hashlib instead of old md5 module
opkg.py,arfile.py: report which file has wrong format and use the
same test
arfile: decode read lines as ascii string before getting fields from
it
2to3: use subprocess instead of commands
2to3: print fixes
2to3: dictionary fixes
2to3: exception handling fixes
opkg-make-index: show OSError/IOError
opkg.py: use string funtcions directly on string variableopkg.py: catch TypeError when reading control file, to show which one
is failing
arfile: fix test
opkg-make-index: don't use stdout for Packages output and use__str__
instead of__repr__
opkg.py: cast lines from controlfile as string
opkg.py: computeFileMD5 only when we have fn, otherwise fails to read
None file
opkg.py: fixwrite_packagewhen called from main testopkg.py: use textwrap for description writing
opkg.py: improve test so it prints temporary control file with long
description and then reads it backRichard Purdie (1):
opkg-make-index: don't error out when some package disappearsScott Anderson (1):
arfile.py: handle six digit UIDsarfile.py | 55 +++++++++++++--------- makePackage | 2 +- opkg-compare-indexes | 22 ++++---- opkg-list-fields | 4 +- opkg-make-index | 74 +++++++++++++++++++---------- opkg-show-deps | 28 +++++------ opkg-unbuild | 4 +- opkg-update-index | 5 +- opkg.py | 125 +++++++++++++++++++++++++++++-------------------- 9 files changed, 186 insertions(+), 133 deletions(-)
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 70 lines · Martin Jansa
First 5 patches are taken from oe-core.
Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls.
Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer.
And in the end I've found the cause.. printing long fields (e. g. description) with
opkg.pywas producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk.And this also fixes Packages.filelist generation.
I don't use python very much, so please review.
And the speedup? Almost 60 times :)
before:real 20m10.484s user 18m32.335s sys 0m11.885safter
real 0m24.492s user 0m23.748s sys 0m0.565sThe following changes since commit
002d29bc605d7c2d02e4cf20a43c5277c15f5597:[opkg-utils] fix install fail problem Thanks for khorben's patch :-) https://docs.openmoko.org/trac/attachment/ticket/2072/patch-opkg-utils_Makefile_install_path.diff (2008-11-03 03:59:59 +0000)
are available in the git repository at:
git://github.com/shr-project/opkg-utilsjansa/pull
https://github.com/shr-project/opkg-utils/tree/jansa/pullChristopher Larson (1):
Use python via the PATH, rather than hardcoding/usr/bin/pythonEnrico Scholz (1):
opkg-make-index: convert mtime to int before comparing itKhem Raj (1):
opkg.py: Add knowledge about License field in ipk headersMartin Jansa (17):
opkg.py: use hashlib instead of old md5 module
opkg.py,arfile.py: report which file has wrong format and use the
same test
arfile: decode read lines as ascii string before getting fields from
it
2to3: use subprocess instead of commands
2to3: print fixes
2to3: dictionary fixes
2to3: exception handling fixes
opkg-make-index: show OSError/IOError
opkg.py: use string funtcions directly on string variableopkg.py: catch TypeError when reading control file, to show which one
is failing
arfile: fix test
opkg-make-index: don't use stdout for Packages output and use__str__
instead of__repr__
opkg.py: cast lines from controlfile as string
opkg.py: computeFileMD5 only when we have fn, otherwise fails to read
None file
opkg.py: fixwrite_packagewhen called from main testopkg.py: use textwrap for description writing
opkg.py: improve test so it prints temporary control file with long
description and then reads it back
Quoted 5 lines
Richard Purdie (1):
opkg-make-index: don't error out when some package disappearsScott Anderson (1):
arfile.py: handle six digit UIDs
I reviewed these and they all look like good improvements. I've merged it into master, thanks!
Richard
Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven:
Quoted 19 lines · Martin Jansa
Quoted 18 lines
First 5 patches are taken from oe-core.
Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls.
Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer.
And in the end I've found the cause.. printing long fields (e. g. description) with
opkg.pywas producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk.And this also fixes Packages.filelist generation.
I don't use python very much, so please review.
And the speedup? Almost 60 times :)
[..]
Quoted 2 lines
I reviewed these and they all look like good improvements. I've merged it into master, thanks!
After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch.
regards,
Koen
Quoted 21 lines · Koen Kooi
Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven:
Quoted 18 lines · Martin Jansa
Quoted 17 lines
First 5 patches are taken from oe-core.
Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls.
Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer.
And in the end I've found the cause.. printing long fields (e. g. description) with
opkg.pywas producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk.And this also fixes Packages.filelist generation.
I don't use python very much, so please review.
Quoted 8 lines
Quoted 1 line
Quoted 1 line
And the speedup? Almost 60 times :)
[..]
Quoted 2 lines
I reviewed these and they all look like good improvements. I've merged it into master, thanks!
After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch.
Agreed, saves about 4 hours per day (13 images each with own package-index run and extra run for feed if built after iamges) on my buildhost for last month or so :)
Cheers,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Op 26 apr. 2012, om 20:52 heeft Martin Jansa het volgende geschreven:
Quoted 26 lines · Koen Kooi
Quoted 25 lines
Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven:
Quoted 19 lines · Martin Jansa
Quoted 18 lines
First 5 patches are taken from oe-core.
Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls.
Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer.
And in the end I've found the cause.. printing long fields (e. g. description) with
opkg.pywas producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk.And this also fixes Packages.filelist generation.
I don't use python very much, so please review.
And the speedup? Almost 60 times :)
[..]
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
I reviewed these and they all look like good improvements. I've merged it into master, thanks!
After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch.
Agreed, saves about 4 hours per day (13 images each with own package-index run and extra run for feed if built after iamges) on my buildhost for last month or so :)
I merged in denzil-next locally and grabbed 26465 from patchwork. Let's see how much time it saves on my machine :)
Op 27 apr. 2012, om 08:39 heeft Koen Kooi het volgende geschreven:
Quoted 32 lines
Op 26 apr. 2012, om 20:52 heeft Martin Jansa het volgende geschreven:
Quoted 29 lines · Koen Kooi
Quoted 28 lines
Op 26 apr. 2012, om 12:43 heeft Richard Purdie het volgende geschreven:
Quoted 19 lines · Martin Jansa
Quoted 18 lines
First 5 patches are taken from oe-core.
Then there are some cleanups and fixes for issues I've found while looking for cause of very long package-index calls.
Some patches uses compatible changes from 2to3 (tested with python2.7), but python3 support is not 100% complete, but were closer.
And in the end I've found the cause.. printing long fields (e. g. description) with
opkg.pywas producing empty lines in Packages file (which are Packages entry separators) so only first entries till first empty line were processed from old Packages file and the rest was always extracted from control files in packaged .ipk.And this also fixes Packages.filelist generation.
I don't use python very much, so please review.
And the speedup? Almost 60 times :)
[..]
Quoted 2 lines
I reviewed these and they all look like good improvements. I've merged it into master, thanks!
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
After some more time in oe-core master for testing we should seriously consider cherry-picking the SRCREV bump to the denzil branch.
Agreed, saves about 4 hours per day (13 images each with own package-index run and extra run for feed if built after iamges) on my buildhost for last month or so :)
I merged in denzil-next locally and grabbed 26465 from patchwork. Let's see how much time it saves on my machine :)
With a relatively small deploy area: 7 minutes saved per image build!