Check non-ASCII characters on recipe metadata
cover Check non-ASCII characters on recipe metadata
From: Leonardo Sandoval <[email protected]>
* First patch creates a QA check on insane.bbclass
* Second patch handles a string encode exception, reported on [1]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=6693
The following changes since commit 0f7df92e3d2dbbb4c94299171d5d0287887e0d28:
clutter-gst: update to 3.0.8 (2015-08-11 09:28:52 -0700)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib lsandov1/package_deb-handle-non-ascii-6693
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/package_deb-handle-non-ascii-6693
Leonardo Sandoval (2):
insane.bbclass: Check non-ASCII characters on metadata package_deb.bbclass: Handle exception when encoding non-ASCII
characters
meta/classes/insane.bbclass | 24 +++++++++++++++++++++++- meta/classes/package_deb.bbclass | 4 ++++ 2 files changed, 27 insertions(+), 1 deletion(-)
Signature · 1 line
1.8.4.5
PATCH 1/2 insane.bbclass: Check non-ASCII characters on metadata
From: Leonardo Sandoval <[email protected]>
Check if non-ASCII characters are present on recipe's metadata. Fields taken into account: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
--- meta/classes/insane.bbclass | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +23 -1
@@ -11,6 +11,7 @@ # -Check if packages contains .debug directories or .so files # where they should be in -dev or -dbg # -Check if config.log contains traces to broken autoconf tests +# -Check non-ascii characters on some package metadata # -Ensure that binaries in base_[bindir|sbindir|libdir] do not link # into exec_prefix # -Check that scripts in base_[bindir|sbindir|libdir] do not reference
@@ -36,7 +37,7 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \ ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \ perms dep-cmp pkgvarcheck perm-config perm-line perm-link \ split-strip packages-list pkgv-undefined var-undefined \ - version-going-backwards expanded-d \ + version-going-backwards expanded-d non-ascii \ " ALL_QA = "${WARN_QA} ${ERROR_QA}"
@@ -947,6 +948,24 @@ def package_qa_check_expanded_d(path,name,d,elf,messages): sane = False return sane +def package_qa_check_ascii_encoding(keys, d): + def check_ascii_encoding(key): + sane = True + value = d.getVar(key, True) + if value: + try: + s = unicode(value) + except UnicodeDecodeError as e: + error_msg = "%s has non-ASCII characters" % key + sane = False + package_qa_handle_error("non-ascii", error_msg, d) + return sane + + for key in keys: + sane = check_ascii_encoding(key) + if not sane: + break + # The PACKAGE FUNC to scan each package python do_package_qa () { import subprocess
@@ -956,6 +975,9 @@ python do_package_qa () { bb.build.exec_func("read_subpackage_metadata", d) + # Check non-ascii characters on recipe's metadata + package_qa_check_ascii_encoding(['DESCRIPTION', 'SUMMARY', 'LICENSE', 'SECTION'], d) + logdir = d.getVar('T', True) pkg = d.getVar('PN', True)
Signature · 1 line
1.8.4.5
PATCH 2/2 package_deb.bbclass: Handle exception when encoding non-ASCII characters
In reply to [email protected]From: Leonardo Sandoval <[email protected]>
On package creation, handle exception when encoding non-ASCII characteres.
[YOCTO #6693]
--- meta/classes/package_deb.bbclass | 4 ++++ 1 file changed, 4 insertions(+)
meta/classes/package_deb.bbclass +4 -0
@@ -171,6 +171,10 @@ python do_package_deb () { bb.utils.unlockfile(lf) ctrlfile.close() raise bb.build.FuncFailed("Missing field for deb generation: %s" % value) + except UnicodeDecodeError: + bb.utils.unlockfile(lf) + ctrlfile.close() + raise bb.build.FuncFailed("Non-ASCII characters found in one of the fields") # more fields custom_fields_chunk = get_package_additional_metadata("deb", localdata)
Signature · 1 line
1.8.4.5
Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters
Quoted 2 lines · On 13 August 2015 at 09:58, <[email protected]> wrote:
On package creation, handle exception when encoding non-ASCII characteres.
Debian control files are defined to be UTF-8, so the use of an ASCII encoding method is wrong (
https://www.debian.org/doc/debian-policy/ch-controlfields.html).
(RPM appears to assume UTF-8 too)
Ross
Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters
Quoted 5 lines · Ross
Debian control files are defined to be UTF-8, so the use of an ASCII encoding method is wrong
(https://www.debian.org/doc/debian-policy/ch-controlfields.html).(RPM appears to assume UTF-8 too)
but check it still fine isnt it.
Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters
Quoted 9 lines · Khem Raj
Quoted 5 lines · Ross
Debian control files are defined to be UTF-8, so the use of an ASCII encoding method is wrong
(https://www.debian.org/doc/debian-policy/ch-controlfields.html).(RPM appears to assume UTF-8 too)
but check it still fine isnt it.
RPM isn't really utf-8.. it's more single 8-bit characters... UTF-8 (1 byte characters) work fine.. multibyte are not promised to work.
If you need special encoding (more then 8-bit characters) then you should be using 'po' style files to translate the 8-bit chars to localized versions...
So checking (and fixing) the strings for various things is a good idea for RPM at least.
--Mark
Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters
Quoted 4 lines · Mark Hatle
RPM isn't really utf-8.. it's more single 8-bit characters... UTF-8 (1 byte
characters) work fine.. multibyte are not promised to work.
Quoted 9 lines
If you need special encoding (more then 8-bit characters) then you should be
using 'po' style files to translate the 8-bit chars to localized versions...So checking (and fixing) the strings for various things is a good idea for RPM
at least.
In the common case of "maintainer's name isn't ASCII" you don't want it translated, and assuming UTF-8 everywhere mostly works.
Ross
Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters
In reply to Khem RajQuoted 2 lines · Khem Raj
but check it still fine isnt it.
Of course, gracefully handling encoding failures is still sensible.
Ross
Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters
Ross/Raj, thanks for your comments. I will send a v2 patches, this time with the correct checks (invalid characters are those non-UTF8) and patch titles.
Quoted 9 lines · Ross
Quoted 2 lines · Khem Raj
but check it still fine isnt it.
Of course, gracefully handling encoding failures is still sensible.
Ross