Skip to contentSkip to conversation

Check non-ASCII characters on recipe metadata

PATCH In review [email protected] 4 participants 9 messages Aug 13, 2015 – Aug 17, 2015

lore
Share & tools
lore
#0 lore raw

cover Check non-ASCII characters on recipe metadata

From: Leonardo Sandoval <[email protected]>

* First patch creates a QA check on insane.bbclass
* Second patch handles a string encode exception, reported on [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=6693

The following changes since commit 0f7df92e3d2dbbb4c94299171d5d0287887e0d28:

clutter-gst: update to 3.0.8 (2015-08-11 09:28:52 -0700)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib lsandov1/package_deb-handle-non-ascii-6693
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/package_deb-handle-non-ascii-6693

Leonardo Sandoval (2):
insane.bbclass: Check non-ASCII characters on metadata package_deb.bbclass: Handle exception when encoding non-ASCII
characters

 meta/classes/insane.bbclass      | 24 +++++++++++++++++++++++-
 meta/classes/package_deb.bbclass |  4 ++++
 2 files changed, 27 insertions(+), 1 deletion(-)
Signature · 1 line
1.8.4.5
#1 lore raw

PATCH 1/2 insane.bbclass: Check non-ASCII characters on metadata

From: Leonardo Sandoval <[email protected]>

Check if non-ASCII characters are present on recipe's metadata. Fields taken into account: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.

---
 meta/classes/insane.bbclass | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +23 -1
index 9c05c86..d560eed 100644
@@ -11,6 +11,7 @@
 #  -Check if packages contains .debug directories or .so files
 #   where they should be in -dev or -dbg
 #  -Check if config.log contains traces to broken autoconf tests
+#  -Check non-ascii characters on some package metadata
 #  -Ensure that binaries in base_[bindir|sbindir|libdir] do not link
 #   into exec_prefix
 #  -Check that scripts in base_[bindir|sbindir|libdir] do not reference
@@ -36,7 +37,7 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
             split-strip packages-list pkgv-undefined var-undefined \
-            version-going-backwards expanded-d \
+            version-going-backwards expanded-d non-ascii \
             "
 
 ALL_QA = "${WARN_QA} ${ERROR_QA}"
@@ -947,6 +948,24 @@ def package_qa_check_expanded_d(path,name,d,elf,messages):
                         sane = False
     return sane
 
+def package_qa_check_ascii_encoding(keys, d):
+    def check_ascii_encoding(key):
+        sane = True
+        value = d.getVar(key, True)
+        if value:
+            try:
+                s = unicode(value)
+            except UnicodeDecodeError as e:
+                error_msg = "%s has non-ASCII characters" % key
+                sane = False
+                package_qa_handle_error("non-ascii", error_msg, d)
+        return sane
+
+    for key in keys:
+        sane = check_ascii_encoding(key)
+        if not sane:
+            break
+
 # The PACKAGE FUNC to scan each package
 python do_package_qa () {
     import subprocess
@@ -956,6 +975,9 @@ python do_package_qa () {
 
     bb.build.exec_func("read_subpackage_metadata", d)
 
+    # Check non-ascii characters on recipe's metadata
+    package_qa_check_ascii_encoding(['DESCRIPTION', 'SUMMARY', 'LICENSE', 'SECTION'], d)
+
     logdir = d.getVar('T', True)
     pkg = d.getVar('PN', True)
 
Signature · 1 line
1.8.4.5
#2 lore raw

PATCH 2/2 package_deb.bbclass: Handle exception when encoding non-ASCII characters

In reply to [email protected]

From: Leonardo Sandoval <[email protected]>

On package creation, handle exception when encoding non-ASCII characteres.

[YOCTO #6693]

---
 meta/classes/package_deb.bbclass | 4 ++++
 1 file changed, 4 insertions(+)
meta/classes/package_deb.bbclass +4 -0
index 9e1ed28..374adb6 100644
@@ -171,6 +171,10 @@ python do_package_deb () {
             bb.utils.unlockfile(lf)
             ctrlfile.close()
             raise bb.build.FuncFailed("Missing field for deb generation: %s" % value)
+        except UnicodeDecodeError:
+            bb.utils.unlockfile(lf)
+            ctrlfile.close()
+            raise bb.build.FuncFailed("Non-ASCII characters found in one of the fields")
         # more fields
 
         custom_fields_chunk = get_package_additional_metadata("deb", localdata)
Signature · 1 line
1.8.4.5
#5 lore raw

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

Quoted 9 lines · Khem Raj
Quoted 5 lines · Ross

Debian control files are defined to be UTF-8, so the use of an ASCII encoding method is wrong
(https://www.debian.org/doc/debian-policy/ch-controlfields.html).

(RPM appears to assume UTF-8 too)

but check it still fine isnt it.

RPM isn't really utf-8.. it's more single 8-bit characters... UTF-8 (1 byte characters) work fine.. multibyte are not promised to work.

If you need special encoding (more then 8-bit characters) then you should be using 'po' style files to translate the 8-bit chars to localized versions...

So checking (and fixing) the strings for various things is a good idea for RPM at least.

--Mark

#6 lore raw

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

Quoted 4 lines · Mark Hatle

RPM isn't really utf-8.. it's more single 8-bit characters... UTF-8 (1 byte
characters) work fine.. multibyte are not promised to work.

Quoted 9 lines

If you need special encoding (more then 8-bit characters) then you should be
using 'po' style files to translate the 8-bit chars to localized versions...

So checking (and fixing) the strings for various things is a good idea for RPM
at least.

In the common case of "maintainer's name isn't ASCII" you don't want it translated, and assuming UTF-8 everywhere mostly works.

Ross

#7 lore raw

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

In reply to Khem Raj
Quoted 2 lines · Khem Raj

but check it still fine isnt it.

Of course, gracefully handling encoding failures is still sensible.

Ross

#8 lore raw

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

Ross/Raj, thanks for your comments. I will send a v2 patches, this time with the correct checks (invalid characters are those non-UTF8) and patch titles.

Quoted 9 lines · Ross
Quoted 2 lines · Khem Raj

but check it still fine isnt it.

Of course, gracefully handling encoding failures is still sensible.

Ross