Skip to contentSkip to conversation

buildhistory.bbclass: Specify lang in decoding strings

PATCH In review Khem Raj 2 participants 4 messages Dec 7, 2013 – Jan 23, 2014

lore
Share & tools
lore

Khem Raj

author
#0 lore raw

PATCH 1/1 buildhistory.bbclass: Specify lang in decoding strings

On systems where default locale is utf-8 we get errors like

File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory 0034: if pkginfo.rconflicts:
0035: f.write("RCONFLICTS = %s\n" % pkginfo.rconflicts) 0036: f.write("PKGSIZE = %d\n" % pkginfo.size)
0037: f.write("FILES = %s\n" % pkginfo.files)
*** 0038: f.write("FILELIST = %s\n" % pkginfo.filelist) 0039:
0040: for filevar in pkginfo.filevars:
0041: filevarpath = os.path.join(pkgpath, "latest.%s" % filevar) 0042: val = pkginfo.filevars[filevar]
Exception: UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 337: ordinal not in range(128)

This patch specifies decode to use utf-8 so ascii and utf-8 based locales both work

---
 meta/classes/buildhistory.bbclass |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/buildhistory.bbclass +1 -1
index 1e6d968..4ff39a0 100644
@@ -190,7 +190,7 @@ python buildhistory_emit_pkghistory() {
                 key = item[0]
                 if key.endswith('_' + pkg):
                     key = key[:-len(pkg)-1]
-                pkgdata[key] = item[1].decode('string_escape')
+                pkgdata[key] = item[1].decode('utf-8', 'string_escape')
 
         pkge = pkgdata.get('PKGE', '0')
         pkgv = pkgdata['PKGV']
Signature · 1 line
1.7.10.4
#1 lore raw
Quoted 28 lines · On Friday 06 December 2013 16:11:54 Khem Raj wrote:

On systems where default locale is utf-8 we get errors like

File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory 0034: if pkginfo.rconflicts:
0035: f.write("RCONFLICTS = %s\n" % pkginfo.rconflicts) 0036: f.write("PKGSIZE = %d\n" % pkginfo.size)
0037: f.write("FILES = %s\n" % pkginfo.files)
*** 0038: f.write("FILELIST = %s\n" % pkginfo.filelist) 0039:
0040: for filevar in pkginfo.filevars:
0041: filevarpath = os.path.join(pkgpath, "latest.%s" % filevar) 0042: val = pkginfo.filevars[filevar]
Exception: UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 337: ordinal not in range(128)

This patch specifies decode to use utf-8 so ascii and utf-8 based locales both work

---
 meta/classes/buildhistory.bbclass |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/meta/classes/buildhistory.bbclass index 1e6d968..4ff39a0 100644

meta/classes/buildhistory.bbclass python buildhistory_emit_pkghistory() { · line 196

Quoted 8 lines
meta/classes/buildhistory.bbclass +1 -1
@@ -190,7 +190,7 @@ python buildhistory_emit_pkghistory() {
                 key = item[0]
                 if key.endswith('_' + pkg):
                     key = key[:-len(pkg)-1]
-                pkgdata[key] = item[1].decode('string_escape')
+                pkgdata[key] = item[1].decode('utf-8', 'string_escape')
 
         pkge = pkgdata.get('PKGE', '0')
         pkgv = pkgdata['PKGV']

Khem, did you test that this actually works? Here it does not - I get strings with \n \t in them; reverting this change makes it interpret these as it should.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#2 lore raw
Quoted 39 lines · On Saturday 28 December 2013 22:52:18 Paul Eggleton wrote:
Quoted 38 lines · On Friday 06 December 2013 16:11:54 Khem Raj wrote:

On systems where default locale is utf-8 we get errors like

File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory 0034: if pkginfo.rconflicts:
0035: f.write("RCONFLICTS = %s\n" % pkginfo.rconflicts) 0036: f.write("PKGSIZE = %d\n" % pkginfo.size)
0037: f.write("FILES = %s\n" % pkginfo.files)
*** 0038: f.write("FILELIST = %s\n" % pkginfo.filelist) 0039:
0040: for filevar in pkginfo.filevars:
0041: filevarpath = os.path.join(pkgpath, "latest.%s" % filevar) 0042: val = pkginfo.filevars[filevar]
Exception: UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 337: ordinal not in range(128)

This patch specifies decode to use utf-8 so ascii and utf-8 based locales both work

---

 meta/classes/buildhistory.bbclass |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/meta/classes/buildhistory.bbclass index 1e6d968..4ff39a0 100644

meta/classes/buildhistory.bbclass +1 -1
@@ -190,7 +190,7 @@ python buildhistory_emit_pkghistory() {
 
                 key = item[0]
 
                 if key.endswith('_' + pkg):
                     key = key[:-len(pkg)-1]
 
-                pkgdata[key] = item[1].decode('string_escape')
+                pkgdata[key] = item[1].decode('utf-8', 'string_escape')
Quoted 6 lines
Quoted 2 lines

pkge = pkgdata.get('PKGE', '0')
pkgv = pkgdata['PKGV']

Khem, did you test that this actually works? Here it does not - I get strings with \n \t in them; reverting this change makes it interpret these as it should.

Unless I'm misunderstanding, I think the second parameter here is wrong, since according to the Python docs it's supposed to specify what to do when an error occurs, not specify another encoding to decode. Should it be .decode('utf-8').decode('string_escape') instead?

The problem that I have is I can't reproduce the failure that you observed, so I need you to help me to fix this properly.

Thanks,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
author
#3 lore raw
Quoted 47 lines · Paul Eggleton
Quoted 39 lines · On Saturday 28 December 2013 22:52:18 Paul Eggleton wrote:
Quoted 38 lines · On Friday 06 December 2013 16:11:54 Khem Raj wrote:

On systems where default locale is utf-8 we get errors like

File: 'buildhistory.bbclass', lineno: 38, function: write_pkghistory 0034: if pkginfo.rconflicts:
0035: f.write("RCONFLICTS = %s\n" % pkginfo.rconflicts) 0036: f.write("PKGSIZE = %d\n" % pkginfo.size)
0037: f.write("FILES = %s\n" % pkginfo.files)
*** 0038: f.write("FILELIST = %s\n" % pkginfo.filelist) 0039:
0040: for filevar in pkginfo.filevars:
0041: filevarpath = os.path.join(pkgpath, "latest.%s" % filevar) 0042: val = pkginfo.filevars[filevar]
Exception: UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 337: ordinal not in range(128)

This patch specifies decode to use utf-8 so ascii and utf-8 based locales both work

---

 meta/classes/buildhistory.bbclass |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/meta/classes/buildhistory.bbclass index 1e6d968..4ff39a0 100644

meta/classes/buildhistory.bbclass +1 -1
@@ -190,7 +190,7 @@ python buildhistory_emit_pkghistory() {
 
                 key = item[0]
 
                 if key.endswith('_' + pkg):
                     key = key[:-len(pkg)-1]
 
-                pkgdata[key] = item[1].decode('string_escape')
+                pkgdata[key] = item[1].decode('utf-8', 'string_escape')
Quoted 6 lines
Quoted 2 lines

pkge = pkgdata.get('PKGE', '0')
pkgv = pkgdata['PKGV']

Khem, did you test that this actually works? Here it does not - I get strings with \n \t in them; reverting this change makes it interpret these as it should.

Quoted 5 lines

Unless I'm misunderstanding, I think the second parameter here is wrong, since according to the Python docs it's supposed to specify what to do when an error occurs, not specify another encoding to decode. Should it be .decode('utf-8').decode('string_escape') instead?

Yes that seems more likely the right thing to do.

Quoted 2 lines

The problem that I have is I can't reproduce the failure that you observed, so I need you to help me to fix this properly.

I see the failure on a newly installed debian wheezy with no locale's installed.

Quoted 8 lines

Thanks,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre