Check invalid characters on some recipe metadata
v2 cover Check invalid characters on some recipe metadata
From: Leonardo Sandoval <[email protected]>
* insane.bbclass: QA check for invalid chars
* package_deb.bbclass: Opens the control file allowing UTF-8 chars and handles a string decoding exception, reported on [1]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=6693
The following changes since commit a533776d6ff83b6e3e830137455b8382d002768b:
perf: fix build breakage on kernels after 4.1 (2015-08-12 11:31:49 -0700)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib lsandov1/package_deb-handle-non-ascii-6693-v2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/package_deb-handle-non-ascii-6693-v2
Leonardo Sandoval (2):
insane.bbclass: Check invalid characters (non UTF8) on recipe metadata package_deb.bbclass: Allow UTF-8 characters on control files
meta/classes/insane.bbclass | 24 +++++++++++++++++++++++- meta/classes/package_deb.bbclass | 18 +++++++++++------- 2 files changed, 34 insertions(+), 8 deletions(-)
Signature · 1 line
1.8.4.5
PATCH v2 1/2 insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata
From: Leonardo Sandoval <[email protected]>
Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
--- meta/classes/insane.bbclass | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +23 -1
@@ -11,6 +11,7 @@ # -Check if packages contains .debug directories or .so files # where they should be in -dev or -dbg # -Check if config.log contains traces to broken autoconf tests +# -Check invalid characters (non-utf8) on some package metadata # -Ensure that binaries in base_[bindir|sbindir|libdir] do not link # into exec_prefix # -Check that scripts in base_[bindir|sbindir|libdir] do not reference
@@ -36,7 +37,7 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \ ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \ perms dep-cmp pkgvarcheck perm-config perm-line perm-link \ split-strip packages-list pkgv-undefined var-undefined \ - version-going-backwards expanded-d \ + version-going-backwards expanded-d invalid-chars \ " ALL_QA = "${WARN_QA} ${ERROR_QA}"
@@ -947,6 +948,24 @@ def package_qa_check_expanded_d(path,name,d,elf,messages): sane = False return sane +def package_qa_check_encoding(keys, encode, d): + def check_encoding(key,enc): + sane = True + value = d.getVar(key, True) + if value: + try: + s = unicode(value, enc) + except UnicodeDecodeError as e: + error_msg = "%s has non %s characters" % (key,enc) + sane = False + package_qa_handle_error("invalid-chars", error_msg, d) + return sane + + for key in keys: + sane = check_encoding(key, encode) + if not sane: + break + # The PACKAGE FUNC to scan each package python do_package_qa () { import subprocess
@@ -956,6 +975,9 @@ python do_package_qa () { bb.build.exec_func("read_subpackage_metadata", d) + # Check non UTF-8 characters on recipe's metadata + package_qa_check_encoding(['DESCRIPTION', 'SUMMARY', 'LICENSE', 'SECTION'], 'utf-8', d) + logdir = d.getVar('T', True) pkg = d.getVar('PN', True)
Signature · 1 line
1.8.4.5
Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata
Quoted 8 lines · On Mon, Aug 17, 2015 at 12:10 AM, < [email protected]>…
From: Leonardo Sandoval <[email protected]>
Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
Signed-off-by: Leonardo Sandoval <
[email protected]>
This commit message doesn't define what "invalid" is.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata
Quoted 5 lines · Christopher Larson
Quoted 2 lines · On Mon, Aug 17, 2015 at 12:10 AM, < [email protected]>…
From: Leonardo Sandoval <[email protected]>
Quoted 8 lines
Quoted 6 lines
Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
Signed-off-by: Leonardo Sandoval <
[email protected]>This commit message doesn't define what "invalid" is.
I will send a v3 to include what invalid means on this context. Basically all non-UTF8 characters are considered invalid.
Quoted 1 line
Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata
In reply to Christopher Larsonhi Chris,
Quoted 5 lines · Christopher Larson
Quoted 2 lines · On Mon, Aug 17, 2015 at 12:10 AM, < [email protected]>…
From: Leonardo Sandoval <[email protected]>
Quoted 8 lines
Quoted 6 lines
Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
Signed-off-by: Leonardo Sandoval <
[email protected]>This commit message doesn't define what "invalid" is.
Looking it closely, the commit message does include in parenthesis what is meant by 'invalid'.
Any concert about this definition?
Quoted 1 line
Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata
Quoted 17 lines · Leonardo Sandoval
hi Chris,
Quoted 13 lines · Christopher Larson
On Mon, Aug 17, 2015 at 12:10 AM, < [email protected]> wrote:
From: Leonardo Sandoval <[email protected]>
Quoted 9 lines
Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
Signed-off-by: Leonardo Sandoval <
[email protected]>
Quoted 8 lines
Quoted 2 lines
This commit message doesn't define what "invalid" is.
Looking it closely, the commit message does include in parenthesis what is meant by 'invalid'.
Any concert about this definition?
Ah, the details were in the summary, but not the description. Generally that should be the other way around, but at least the info is there, so no objections.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
PATCH v2 2/2 package_deb.bbclass: Allow UTF-8 characters on control files
In reply to [email protected]From: Leonardo Sandoval <[email protected]>
Allow UTF-8 characters on control files. Also handle an expection in case of invalid characters (non UTF-8).
[YOCTO #6693]
--- meta/classes/package_deb.bbclass | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
meta/classes/package_deb.bbclass +11 -7
@@ -96,9 +96,8 @@ python do_package_deb () { bb.utils.mkdirhier(controldir) os.chmod(controldir, 0755) try: - ctrlfile = open(os.path.join(controldir, 'control'), 'w') - # import codecs - # ctrlfile = codecs.open("someFile", "w", "utf-8") + import codecs + ctrlfile = codecs.open(os.path.join(controldir, 'control'), 'w', 'utf-8') except OSError: bb.utils.unlockfile(lf) raise bb.build.FuncFailed("unable to open control file for writing.")
@@ -149,7 +148,7 @@ python do_package_deb () { # Special behavior for description... if 'DESCRIPTION' in fs: summary = localdata.getVar('SUMMARY', True) or localdata.getVar('DESCRIPTION', True) or "." - ctrlfile.write('Description: %s\n' % unicode(summary)) + ctrlfile.write('Description: %s\n' % unicode(summary,'utf-8')) description = localdata.getVar('DESCRIPTION', True) or "." description = textwrap.dedent(description).strip() if '\\n' in description:
@@ -158,19 +157,24 @@ python do_package_deb () { # We don't limit the width when manually indent, but we do # need the textwrap.fill() to set the initial_indent and # subsequent_indent, so set a large width - ctrlfile.write('%s\n' % unicode(textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '))) + ctrlfile.write('%s\n' % unicode(textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '),'utf-8')) else: # Auto indent - ctrlfile.write('%s\n' % unicode(textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' '))) + ctrlfile.write('%s\n' % unicode(textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' '),'utf-8')) else: - ctrlfile.write(unicode(c % tuple(pullData(fs, localdata)))) + ctrlfile.write(unicode(c % tuple(pullData(fs, localdata)),'utf-8')) except KeyError: import sys (type, value, traceback) = sys.exc_info() bb.utils.unlockfile(lf) ctrlfile.close() raise bb.build.FuncFailed("Missing field for deb generation: %s" % value) + except UnicodeDecodeError: + bb.utils.unlockfile(lf) + ctrlfile.close() + raise bb.build.FuncFailed("Non UTF-8 characters found in one of the fields") + # more fields custom_fields_chunk = get_package_additional_metadata("deb", localdata)
Signature · 1 line
1.8.4.5