Skip to contentSkip to conversation

Check invalid characters on some recipe metadata

PATCH In review [email protected] 2 participants 7 messages Aug 17, 2015 – Aug 26, 2015

lore
Share & tools
lore
#0 lore raw

v2 cover Check invalid characters on some recipe metadata

From: Leonardo Sandoval <[email protected]>

* insane.bbclass: QA check for invalid chars
* package_deb.bbclass: Opens the control file allowing UTF-8 chars and handles a string decoding exception, reported on [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=6693

The following changes since commit a533776d6ff83b6e3e830137455b8382d002768b:

perf: fix build breakage on kernels after 4.1 (2015-08-12 11:31:49 -0700)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib lsandov1/package_deb-handle-non-ascii-6693-v2
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/package_deb-handle-non-ascii-6693-v2

Leonardo Sandoval (2):
insane.bbclass: Check invalid characters (non UTF8) on recipe metadata package_deb.bbclass: Allow UTF-8 characters on control files

 meta/classes/insane.bbclass      | 24 +++++++++++++++++++++++-
 meta/classes/package_deb.bbclass | 18 +++++++++++-------
 2 files changed, 34 insertions(+), 8 deletions(-)
Signature · 1 line
1.8.4.5
#1 lore raw

PATCH v2 1/2 insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

From: Leonardo Sandoval <[email protected]>

Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.

---
 meta/classes/insane.bbclass | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
meta/classes/insane.bbclass +23 -1
index 9c05c86..fc16540 100644
@@ -11,6 +11,7 @@
 #  -Check if packages contains .debug directories or .so files
 #   where they should be in -dev or -dbg
 #  -Check if config.log contains traces to broken autoconf tests
+#  -Check invalid characters (non-utf8) on some package metadata
 #  -Ensure that binaries in base_[bindir|sbindir|libdir] do not link
 #   into exec_prefix
 #  -Check that scripts in base_[bindir|sbindir|libdir] do not reference
@@ -36,7 +37,7 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
             split-strip packages-list pkgv-undefined var-undefined \
-            version-going-backwards expanded-d \
+            version-going-backwards expanded-d invalid-chars \
             "
 
 ALL_QA = "${WARN_QA} ${ERROR_QA}"
@@ -947,6 +948,24 @@ def package_qa_check_expanded_d(path,name,d,elf,messages):
                         sane = False
     return sane
 
+def package_qa_check_encoding(keys, encode, d):
+    def check_encoding(key,enc):
+        sane = True
+        value = d.getVar(key, True)
+        if value:
+            try:
+                s = unicode(value, enc)
+            except UnicodeDecodeError as e:
+                error_msg = "%s has non %s characters" % (key,enc)
+                sane = False
+                package_qa_handle_error("invalid-chars", error_msg, d)
+        return sane
+
+    for key in keys:
+        sane = check_encoding(key, encode)
+        if not sane:
+            break
+
 # The PACKAGE FUNC to scan each package
 python do_package_qa () {
     import subprocess
@@ -956,6 +975,9 @@ python do_package_qa () {
 
     bb.build.exec_func("read_subpackage_metadata", d)
 
+    # Check non UTF-8 characters on recipe's metadata
+    package_qa_check_encoding(['DESCRIPTION', 'SUMMARY', 'LICENSE', 'SECTION'], 'utf-8', d)
+
     logdir = d.getVar('T', True)
     pkg = d.getVar('PN', True)
 
Signature · 1 line
1.8.4.5
#2 lore raw

Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

Quoted 8 lines · On Mon, Aug 17, 2015 at 12:10 AM, < [email protected]>…

From: Leonardo Sandoval <[email protected]>

Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.

Signed-off-by: Leonardo Sandoval <
[email protected]>

This commit message doesn't define what "invalid" is.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#3 lore raw

Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

Quoted 5 lines · Christopher Larson
Quoted 2 lines · On Mon, Aug 17, 2015 at 12:10 AM, < [email protected]>…

From: Leonardo Sandoval <[email protected]>

Quoted 8 lines
Quoted 6 lines

Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.

Signed-off-by: Leonardo Sandoval <
[email protected]>

This commit message doesn't define what "invalid" is.

I will send a v3 to include what invalid means on this context. Basically all non-UTF8 characters are considered invalid.

Quoted 1 line
#4 lore raw

Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

In reply to Christopher Larson

hi Chris,

Quoted 5 lines · Christopher Larson
Quoted 2 lines · On Mon, Aug 17, 2015 at 12:10 AM, < [email protected]>…

From: Leonardo Sandoval <[email protected]>

Quoted 8 lines
Quoted 6 lines

Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.

Signed-off-by: Leonardo Sandoval <
[email protected]>

This commit message doesn't define what "invalid" is.

Looking it closely, the commit message does include in parenthesis what is meant by 'invalid'.

Any concert about this definition?

Quoted 1 line
#5 lore raw

Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

Quoted 17 lines · Leonardo Sandoval

hi Chris,

Quoted 13 lines · Christopher Larson

On Mon, Aug 17, 2015 at 12:10 AM, < [email protected]> wrote:

From: Leonardo Sandoval <[email protected]>

Quoted 9 lines

Check if invalid characters are present on recipe's metadata. Fields taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.

Signed-off-by: Leonardo Sandoval <
[email protected]>

Quoted 8 lines
Quoted 2 lines

This commit message doesn't define what "invalid" is.

Looking it closely, the commit message does include in parenthesis what is meant by 'invalid'.

Any concert about this definition?

Ah, the details were in the summary, but not the description. Generally that should be the other way around, but at least the info is there, so no objections.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#6 lore raw

PATCH v2 2/2 package_deb.bbclass: Allow UTF-8 characters on control files

In reply to [email protected]

From: Leonardo Sandoval <[email protected]>

Allow UTF-8 characters on control files. Also handle an expection in case of invalid characters (non UTF-8).

[YOCTO #6693]

---
 meta/classes/package_deb.bbclass | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
meta/classes/package_deb.bbclass +11 -7
index 9e1ed28..398ceee 100644
@@ -96,9 +96,8 @@ python do_package_deb () {
         bb.utils.mkdirhier(controldir)
         os.chmod(controldir, 0755)
         try:
-            ctrlfile = open(os.path.join(controldir, 'control'), 'w')
-            # import codecs
-            # ctrlfile = codecs.open("someFile", "w", "utf-8")
+            import codecs
+            ctrlfile = codecs.open(os.path.join(controldir, 'control'), 'w', 'utf-8')
         except OSError:
             bb.utils.unlockfile(lf)
             raise bb.build.FuncFailed("unable to open control file for writing.")
@@ -149,7 +148,7 @@ python do_package_deb () {
                 # Special behavior for description...
                 if 'DESCRIPTION' in fs:
                      summary = localdata.getVar('SUMMARY', True) or localdata.getVar('DESCRIPTION', True) or "."
-                     ctrlfile.write('Description: %s\n' % unicode(summary))
+                     ctrlfile.write('Description: %s\n' % unicode(summary,'utf-8'))
                      description = localdata.getVar('DESCRIPTION', True) or "."
                      description = textwrap.dedent(description).strip()
                      if '\\n' in description:
@@ -158,19 +157,24 @@ python do_package_deb () {
                              # We don't limit the width when manually indent, but we do
                              # need the textwrap.fill() to set the initial_indent and
                              # subsequent_indent, so set a large width
-                             ctrlfile.write('%s\n' % unicode(textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' ')))
+                             ctrlfile.write('%s\n' % unicode(textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '),'utf-8'))
                      else:
                          # Auto indent
-                         ctrlfile.write('%s\n' % unicode(textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' ')))
+                         ctrlfile.write('%s\n' % unicode(textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' '),'utf-8'))
 
                 else:
-                     ctrlfile.write(unicode(c % tuple(pullData(fs, localdata))))
+                     ctrlfile.write(unicode(c % tuple(pullData(fs, localdata)),'utf-8'))
         except KeyError:
             import sys
             (type, value, traceback) = sys.exc_info()
             bb.utils.unlockfile(lf)
             ctrlfile.close()
             raise bb.build.FuncFailed("Missing field for deb generation: %s" % value)
+        except UnicodeDecodeError:
+            bb.utils.unlockfile(lf)
+            ctrlfile.close()
+            raise bb.build.FuncFailed("Non UTF-8 characters found in one of the fields")
+
         # more fields
 
         custom_fields_chunk = get_package_additional_metadata("deb", localdata)
Signature · 1 line
1.8.4.5