Skip to contentSkip to conversation

Fix UTF Encoding issue

PATCH Benjamin Close 6 participants 24 messages Jun 15, 2016

lore
Share & tools
lore

Benjamin Close

author
#0 lore raw
Quoted 1 line

From 83042abf3967b455953cddeab43e33c1d59c6f03 Mon Sep 17 00:00:00 2001

From: Benjamin Close <[email protected]>
Date: Sun, 2 Dec 2007 15:09:00 -0800
Subject: [PATCH] Gitweb: Fix encoding to always translate rather than sometimes fail

When performing the utf translation don't test if $res is defined. It appears that it is defined even when the conversion fails. This causes failures on the writing of the output stream which is expecting UTF.

Instead, immediately return if conversion is successful else force the translation to the fallback encoding

---
 gitweb/gitweb.perl |    8 ++------
 1 files changed, 2 insertions(+), 6 deletions(-)
gitweb/gitweb.perl +2 -6
index 491a3f4..00bbcdf 100755
@@ -696,12 +696,8 @@ sub validate_refname {
 sub to_utf8 {
 	my $str = shift;
 	my $res;
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }
 
 # quote unsafe chars, but keep the slash, even when it's not
Signature · 1 line
1.5.3.6
#1 lore raw
Quoted 12 lines · Benjamin Close
Quoted 1 line

From 83042abf3967b455953cddeab43e33c1d59c6f03 Mon Sep 17 00:00:00 2001

From: Benjamin Close <[email protected]>
Date: Sun, 2 Dec 2007 15:09:00 -0800
Subject: [PATCH] Gitweb: Fix encoding to always translate rather than sometimes fail

When performing the utf translation don't test if $res is defined. It appears that it is defined even when the conversion fails. This causes failures on the writing of the output stream which is expecting UTF.

gitweb/gitweb.perl +0 -0
@@ -696,12 +696,8 @@ sub validate_refname {
 sub to_utf8 {
 	my $str = shift;

gitweb/gitweb.perl sub validate_refname { · line 701

Quoted 10 lines
gitweb/gitweb.perl +2 -6
 	my $res;
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }

This is funny.

I thought the standard catch ... throw idiom in Perl was to do the above like this:

	my $res;
        eval { $res = decode_utf8($str, Encode::FB_CROAK); };
        if ($@) {
        	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
	}
	return $res;

(alternatively, you can assign return value of eval {} to $res).

#2 lore raw

Monday 03 December 2007 Tarihinde 12:14:43 yazmıştı:

Quoted 30 lines
Quoted 13 lines · Benjamin Close
Quoted 1 line

From 83042abf3967b455953cddeab43e33c1d59c6f03 Mon Sep 17 00:00:00 2001

From: Benjamin Close <[email protected]>
Date: Sun, 2 Dec 2007 15:09:00 -0800
Subject: [PATCH] Gitweb: Fix encoding to always translate rather than sometimes fail

When performing the utf translation don't test if $res is defined. It appears that it is defined even when the conversion fails. This causes failures on the writing of the output stream which is expecting UTF.

gitweb/gitweb.perl +0 -0
@@ -696,12 +696,8 @@ sub validate_refname {
 sub to_utf8 {
 	my $str = shift;

gitweb/gitweb.perl sub validate_refname { · line 701

Quoted 10 lines
gitweb/gitweb.perl +2 -6
 	my $res;
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }

This is funny.

I thought the standard catch ... throw idiom in Perl was to do the above like this:

Quoted 6 lines
	my $res;
        eval { $res = decode_utf8($str, Encode::FB_CROAK); };
        if ($@) {
        	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
	}
	return $res;

I think this is correct, but the current code in gitweb doesn't look correct since it checks for $res and not $@.

Regards,
ismail

Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
#3 lore raw
Quoted 23 lines · Ismail Dönmez

Monday 03 December 2007 Tarihinde 12:14:43 yazmıştı:

Quoted 15 lines

gitweb/gitweb.perl sub validate_refname { · line 701

Quoted 9 lines · Benjamin Close
gitweb/gitweb.perl +2 -6
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }

I thought the standard catch ... throw idiom in Perl was to do the above like this:

Quoted 6 lines
	my $res;
        eval { $res = decode_utf8($str, Encode::FB_CROAK); };
        if ($@) {
        	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
	}
	return $res;
Quoted 2 lines

I think this is correct, but the current code in gitweb doesn't look correct since it checks for $res and not $@.

First version of the patch was created by Martin Koegler. I have participated in creating the version which is now in gitweb, but I have to say that I wrote it based on decode_utf8 documentation... which doesn't necessarily agree with facts :-(

I'm all for the "throw idion" version. Ack.

Signature · 1 line
Jakub Narebski
#4 lore raw
Quoted 12 lines · Jakub Narebski
Quoted 11 lines · Ismail Dönmez

Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?:

Quoted 10 lines

gitweb/gitweb.perl sub validate_refname { · line 701

Quoted 9 lines · Benjamin Close
gitweb/gitweb.perl +2 -6
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }

This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1 encoding of the same character sequence yields to different results.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

I thought the standard catch ... throw idiom in Perl was to do the above like this:

	my $res;
        eval { $res = decode_utf8($str, Encode::FB_CROAK); };
        if ($@) {
        	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
	}
	return $res;
Quoted 7 lines
Quoted 2 lines

I think this is correct, but the current code in gitweb doesn't look correct since it checks for $res and not $@.

First version of the patch was created by Martin Koegler. I have participated in creating the version which is now in gitweb, but I have to say that I wrote it based on decode_utf8 documentation... which doesn't necessarily agree with facts :-(

eval { $res = decode_utf8(...); }
if ($@)

     return decode(...);

return $res

or

eval { $res = decode_utf8(...); }
if (defined $res)

      return $res;
else
    return decode(...);

show the same (wrong) behaviour on Debian sarge. They do not always decode non UTF-8 characters correctly, eg.
#öäü does not work
#äöüä does work

On Debian etch, both versions are working.

Quoted 1 line

I'm all for the "throw idion" version. Ack.

mfg Martin Kögler

#5 lore raw
Quoted 16 lines · Martin Koegler
Quoted 12 lines · Jakub Narebski
Quoted 11 lines · Ismail Dönmez

Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?:

Quoted 10 lines

gitweb/gitweb.perl sub validate_refname { · line 701

Quoted 9 lines · Benjamin Close
gitweb/gitweb.perl +2 -6
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }

This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1 encoding of the same character sequence yields to different results.

[...]

Quoted 13 lines

eval { $res = decode_utf8(...); }
if ($@)

     return decode(...);

return $res

or

eval { $res = decode_utf8(...); }
if (defined $res)

      return $res;
else
    return decode(...);
Quoted 6 lines

show the same (wrong) behaviour on Debian sarge. They do not always decode non UTF-8 characters correctly, eg.
#öäü does not work
#äöüä does work

On Debian etch, both versions are working.

I don't know enough Perl to decide if it is a bug in gitweb usage of decode_utf8, if it is a bug in your version of Encode, or if it is bug in Encode.

Send copy of this mail to maintainers of Encode perl module.

Signature · 2 lines
Jakub Narebski
Poland
author
#6 lore raw
Quoted 10 lines · Jakub Narebski
Quoted 8 lines · Martin Koegler
Quoted 6 lines · Jakub Narebski
Quoted 4 lines · Ismail Dönmez

Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?:

Quoted 2 lines

Benjamin Close <[email protected]> writes:

Quoted 15 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
gitweb/gitweb.perl +0 -4
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

gitweb/gitweb.perl sub validate_refname { · line 701

Quoted 6 lines
gitweb/gitweb.perl +2 -2
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }

This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1 encoding of the same character sequence yields to different results.

For the record, this was on a debian sid machine.

#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi

and the result of not using the original patch was:

<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at /usr/lib/perl/5.8/Encode.pm line 166. </pre>

I haven't tried the other solutions tested here.

Quoted 18 lines
Quoted 18 lines

eval { $res = decode_utf8(...); }
if ($@)

     return decode(...);

return $res

or

eval { $res = decode_utf8(...); }
if (defined $res)

      return $res;
else
    return decode(...);

show the same (wrong) behaviour on Debian sarge. They do not always decode non UTF-8 characters correctly, eg.
#öäü does not work
#äöüä does work

Quoted 9 lines
Quoted 2 lines

On Debian etch, both versions are working.

I don't know enough Perl to decide if it is a bug in gitweb usage of decode_utf8, if it is a bug in your version of Encode, or if it is bug in Encode.

Send copy of this mail to maintainers of Encode perl module.

Ismail do you know if sid was also broken?

#7 lore raw

Monday 03 December 2007 Tarihinde 23:46:24 yazmıştı:

Quoted 22 lines
Quoted 5 lines · Jakub Narebski
Quoted 4 lines · Martin Koegler
Quoted 3 lines · Jakub Narebski
Quoted 2 lines · Ismail Dönmez

Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?:

Quoted 1 line

Benjamin Close <[email protected]> writes:

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +0 -5
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +2 -1
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }

This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1 encoding of the same character sequence yields to different results.

For the record, this was on a debian sid machine.

Quoted 8 lines

#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi

and the result of not using the original patch was:

<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at
/usr/lib/perl/5.8/Encode.pm line 166. </pre>

Can you try the attached patch?

Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
gitweb.perl +1 -1
@@ -2159,7 +2159,7 @@
 	}
 	my $owner = $gcos;
 	$owner =~ s/[,;].*$//;
-	return to_utf8($owner);
+	return $owner;
 }
 
 ## ......................................................................
author
#8 lore raw
Quoted 26 lines · Ismail D??nmez

Monday 03 December 2007 Tarihinde 23:46:24 yazm????t??:

Quoted 25 lines
Quoted 5 lines · Jakub Narebski
Quoted 4 lines · Martin Koegler
Quoted 3 lines · Jakub Narebski
Quoted 2 lines · Ismail D??nmez

Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?:

Quoted 1 line

Benjamin Close <[email protected]> writes:

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +0 -5
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +2 -1
-	}
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); };
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 }

This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1 encoding of the same character sequence yields to different results.

For the record, this was on a debian sid machine.

#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi

Quoted 7 lines
Quoted 5 lines

and the result of not using the original patch was:

<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at
/usr/lib/perl/5.8/Encode.pm line 166. </pre>

Can you try the attached patch?

I confirm that the patch corrects the problem.

Without it I get the Cannot decode string error. With it gitweb displays correctly.

Cheers,
Benjamin

#9 lore raw
Quoted 8 lines · Benjamin Close
Quoted 2 lines · Ismail Donmez

Can you try the attached patch?

I confirm that the patch corrects the problem.

Without it I get the Cannot decode string error. With it gitweb displays correctly.

But the patch _avoids_ issue (des not convert owner to utf8), rather than solving it, if I understand it correctly. What if gecos is in utf-8?

Signature · 2 lines
Jakub Narebski
Poland
#10 lore raw

Tuesday 04 December 2007 Tarihinde 01:37:35 yazmıştı:

Quoted 12 lines · Benjamin Close
Quoted 7 lines · Ismail Donmez
Quoted 1 line

Can you try the attached patch?

I confirm that the patch corrects the problem.

Without it I get the Cannot decode string error. With it gitweb displays correctly.

But the patch _avoids_ issue (des not convert owner to utf8), rather than solving it, if I understand it correctly. What if gecos is in utf-8?

Indeed its a workaround but UTF-8 username is correctly displayed in gitweb so my understanding was gecos field is already UTF-8.

Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
#11 lore raw
In reply to Benjamin Close
Quoted 29 lines · Benjamin Close
Quoted 5 lines · Jakub Narebski
Quoted 4 lines · Martin Koegler
Quoted 3 lines · Jakub Narebski
Quoted 2 lines · Ismail Dönmez

Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?:

Quoted 1 line

Benjamin Close <[email protected]> writes:

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
(quoted diff) +0 -5
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str,

Encode::FB_DEFAULT);

(quoted diff) +0 -1
-	}
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +1 -0
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK));

};

(quoted diff) +1 -0
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);

}

This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1
encoding of the same character sequence yields to different results.

For the record, this was on a debian sid machine.

#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi

and the result of not using the original patch was:

Quoted 7 lines

<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at
/usr/lib/perl/5.8/Encode.pm line 166.
</pre>

I haven't tried the other solutions tested here.

Debian etch also has v5.8.8.

My main question is, why is the error not catched?

I'm not a perl programmer, but in your patch the first line is a NOP. The return in eval seems to only returns from the eval block, so any text is decoded as latin1 with the second statement.

In the original version, decode($fallback_encoding, $str, Encode::FB_DEFAULT) can not emit an error, else it would in your version too.

In your version, eval is able to surpress the error of decode_utf8($str, Encode::FB_CROAK);, but not in the original version.

Strange.

mfg Martin Kögler

#12 lore raw

Tuesday 04 December 2007 10:04:07 Martin Koegler yazmıştı:

Quoted 46 lines · Benjamin Close
Quoted 29 lines
Quoted 5 lines · Jakub Narebski
Quoted 4 lines · Martin Koegler
Quoted 3 lines · Jakub Narebski
Quoted 2 lines · Ismail Dönmez

Monday 03 December 2007 Tarihinde 12:14:43 yazm??t?:

Quoted 1 line

Benjamin Close <[email protected]> writes:

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
(quoted diff) +0 -5
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
-	} else {
-		return decode($fallback_encoding, $str,

Encode::FB_DEFAULT);

(quoted diff) +0 -1
-	}
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +1 -0
+	eval { return ($res = decode_utf8($str, Encode::FB_CROAK));

};

(quoted diff) +1 -0
+	return decode($fallback_encoding, $str, Encode::FB_DEFAULT);

}

This version is broken on Debian sarge and etch. Feeding a UTF-8 and a latin1
encoding of the same character sequence yields to different results.

For the record, this was on a debian sid machine.

#perl --version
This is perl, v5.8.8 built for x86_64-linux-gnu-thread-multi

and the result of not using the original patch was:

Quoted 7 lines

<h1>Software error:</h1>
<pre>Cannot decode string with wide characters at
/usr/lib/perl/5.8/Encode.pm line 166.
</pre>

I haven't tried the other solutions tested here.

Debian etch also has v5.8.8.

My main question is, why is the error not catched?

I'm not a perl programmer, but in your patch the first line is a NOP. The return in eval seems to only returns from the eval block, so any text is decoded as latin1 with the second statement.

Quoted 6 lines

In the original version, decode($fallback_encoding, $str, Encode::FB_DEFAULT) can not emit an error, else it would in your version too.

In your version, eval is able to surpress the error of decode_utf8($str, Encode::FB_CROAK);, but not in the original version.

I think just a better method is to use (not tested):

if( is_utf8($str) )

{
	return decode_utf8($str);
}
else {
	return decode($str);
}

Regards,
ismail

Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
#13 lore raw
Quoted 9 lines · Ismail Dönmez

I think just a better method is to use (not tested):

if( is_utf8($str) )

{
	return decode_utf8($str);
}
else {
	return decode($str);
}

I already tried this function. It does not test, if a string is really UTF-8. It seems to be to intended to check, if perl stores the string internally in a multi byte encoding.

mfg Martin Kögler.

#14 lore raw
In reply to Jakub Narebski
Quoted 19 lines · Jakub Narebski
Quoted 18 lines · Martin Koegler

eval { $res = decode_utf8(...); }
if ($@)

     return decode(...);

return $res

or

eval { $res = decode_utf8(...); }
if (defined $res)

      return $res;
else
    return decode(...);

show the same (wrong) behaviour on Debian sarge. They do not always decode non UTF-8 characters correctly, eg.
#öäü does not work
#äöüä does work

Quoted 7 lines
Quoted 1 line

On Debian etch, both versions are working.

I don't know enough Perl to decide if it is a bug in gitweb usage of decode_utf8, if it is a bug in your version of Encode, or if it is bug in Encode.

Send copy of this mail to maintainers of Encode perl module.

The bug affects old versions of perl (Debian sarge = oldstable). As it works on the newer Debian etch, do you really think, that it is a good idea to report issue?

How would you handle a bug report, which reports a bug for gitweb in GIT 1.4, and tells you, that a newer versions works?

As Debian sarge has reached its end of life, the distribution will probable also issue no update.

mfg Martin Kögler

#15 lore raw

Tuesday 04 December 2007 Tarihinde 09:50:28 yazmıştı:

Quoted 3 lines

The bug affects old versions of perl (Debian sarge = oldstable). As it works on the newer Debian etch, do you really think, that it is a good idea to report issue?

Same problem here with v5.8.8 which is latest stable perl5 release.

Regards,
ismail

Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
#16 lore raw
Quoted 6 lines · Ismail Dönmez

Tuesday 04 December 2007 Tarihinde 09:50:28 yazm????t??:

Quoted 3 lines

The bug affects old versions of perl (Debian sarge = oldstable). As it works on the newer Debian etch, do you really think, that it is a good idea to report issue?

Same problem here with v5.8.8 which is latest stable perl5 release.

I have put together a small perl script, which tests the various ways of decoding, which have been posted on the list. The first test is wrong by design. A working decoding method should result in "#öäü#äöü".

Debian sarge:
#öäü#ÀöÌ
##äöü
##äöü
##äöü

Debian etch, OpenSuSE 10.2, Fedora 7:
#öäü#ÀöÌ
#öäü#äöü
#öäü#äöü
#öäü#äöü

mfg Martin Kögler

#!/usr/bin/perl
use Encode;

sub t {
my $str = shift;
my ($res);
eval { return ($res = decode_utf8($str, Encode::FB_CROAK)); }; return decode("latin1", $str, Encode::FB_DEFAULT);
}
sub t1 {
my $str = shift;
my ($res);
eval { ($res = decode_utf8($str, Encode::FB_CROAK)); }; if ($@) {
return decode("latin1", $str, Encode::FB_DEFAULT); } else
{ return $res; }
}

sub t2 {
my $str = shift;
my ($res);

eval { $res = decode_utf8($str, Encode::FB_CROAK); };

 if (defined $res) {
        return $res;
} else {
        return decode("latin1", $str, Encode::FB_DEFAULT);
}

}

sub t3 {
	my $str = shift;
	my $res;
	eval { $res = decode_utf8 ($str, 1); };
	return $res || decode('latin1', $str);
}

print t("#öäü");
print t("#ÀöÌ");
print "\n";
print t1("#öäü");
print t1("#ÀöÌ");
print "\n";
print t2("#öäü");
print t2("#ÀöÌ");
print "\n";
print t3("#öäü");
print t3("#ÀöÌ");
print "\n";

#17 lore raw

Tuesday 04 December 2007 10:16:34 Martin Koegler yazmıştı:
[...]

Quoted 3 lines

print t("#öäü");
print t("#ÀöÌ");
print "\n";

How about this one, doesn't even use Encode, uses just built-in utf8 function :

[~]> cat test.pl
binmode STDOUT, ':utf8';

my $str = "#öäü";

if (utf8::valid($str))

{
    utf8::decode($str);
}

print $str."\n";

[~]> perl test.pl
#öäü

Regards,
ismail

Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
#18 lore raw

Tuesday 04 December 2007 10:28:59 Ismail Dönmez yazmıştı:

Quoted 16 lines

Tuesday 04 December 2007 10:16:34 Martin Koegler yazmıştı:
[...]

Quoted 3 lines

print t("#öäü");
print t("#ÀöÌ");
print "\n";

How about this one, doesn't even use Encode, uses just built-in utf8 function :

[~]> cat test.pl
binmode STDOUT, ':utf8';

my $str = "#öäü";

if (utf8::valid($str))

Quoted 8 lines
{
    utf8::decode($str);
}

print $str."\n";

[~]> perl test.pl
#öäü

Following to_utf8 function works for me :

sub to_utf8 {
·   my $str = shift;

    if(utf8::valid($str))
    {
        utf8::decode($str);
    }
·
    return $str;
}

Regards,
ismail

Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
#19 lore raw
Quoted 1 line · Ismail Dönmez

Following to_utf8 function works for me :

For me too (Debian sarge+etch).

Quoted 9 lines

sub to_utf8 {
·   my $str = shift;

    if(utf8::valid($str))
    {
        utf8::decode($str);
    }
·
    return $str;

In the original thread, there was some discussion, that some people might want a different fallback endcoding. So mayme you should keep the second call to decode for the fallback encoding.

Quoted 1 line

}

mfg Martin Kögler

#20 lore raw

Tuesday 04 December 2007 10:44:12 Martin Koegler yazmıştı:

Quoted 4 lines · Ismail Dönmez
Quoted 1 line

Following to_utf8 function works for me :

For me too (Debian sarge+etch).

Thanks for testing.

Quoted 10 lines
Quoted 9 lines

sub to_utf8 {
·   my $str = shift;

    if(utf8::valid($str))
    {
        utf8::decode($str);
    }
·
    return $str;

Quoted 3 lines

In the original thread, there was some discussion, that some people might want a different fallback endcoding. So mayme you should keep the second call to decode for the fallback encoding.

Probably, I just wanted to fix this damn UTF-8 bug surfacing over and over =)

Regards,
ismail

Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
#21 lore raw

Tuesday 04 December 2007 10:47:39 Ismail Dönmez yazmıştı:

Quoted 7 lines

Tuesday 04 December 2007 10:44:12 Martin Koegler yazmıştı:

Quoted 4 lines · Ismail Dönmez
Quoted 1 line

Following to_utf8 function works for me :

For me too (Debian sarge+etch).

Thanks for testing.

Use Perl built-in utf8 function for UTF-8 decoding.

Signed-off-by İsmail Dönmez <[email protected]>
gitweb/gitweb.perl +3 -4
index ff5daa7..db255c1 100755
@@ -695,10 +695,9 @@ sub validate_refname {
 # in utf-8 thanks to "binmode STDOUT, ':utf8'" at beginning
 sub to_utf8 {
 	my $str = shift;
-	my $res;
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
+        if (utf8::valid($str)) {
+                utf8::decode($str);
+                return $str;
 	} else {
 		return decode($fallback_encoding, $str, Encode::FB_DEFAULT);
 	}
Signature · 1 line
Never learn by your mistakes, if you do you may never dare to try again.
#23 lore raw
In reply to Ismail Dönmez
Quoted 18 lines · Ismail Dönmez

Tuesday 04 December 2007 10:47:39 Ismail Dönmez yazmıştı:

Quoted 7 lines

Tuesday 04 December 2007 10:44:12 Martin Koegler yazmıştı:

Quoted 4 lines · Ismail Dönmez
Quoted 1 line

Following to_utf8 function works for me :

For me too (Debian sarge+etch).

Thanks for testing.

Use Perl built-in utf8 function for UTF-8 decoding.

Signed-off-by İsmail Dönmez <[email protected]>
gitweb/gitweb.perl +0 -0
index ff5daa7..db255c1 100755

gitweb/gitweb.perl sub validate_refname { · line 700

Quoted 10 lines
gitweb/gitweb.perl +3 -4
@@ -695,10 +695,9 @@ sub validate_refname {
 # in utf-8 thanks to "binmode STDOUT, ':utf8'" at beginning
 sub to_utf8 {
 	my $str = shift;
-	my $res;
-	eval { $res = decode_utf8($str, Encode::FB_CROAK); };
-	if (defined $res) {
-		return $res;
+        if (utf8::valid($str)) {
+                utf8::decode($str);
+                return $str;

This is good as it fixes another problem which some may have encountered. On at least one distro that I use (Red Hat Enterprise Linux 3) the Encode module is very old (it's 1.83; the latest release is 2.23), and so gitweb won't even run, dying during compilation with this:

Too many arguments for Encode::decode_utf8 at gitweb.cgi line 686,
near "Encode::FB_CROAK)"

Of course, the workaround is to install a newer version of the module, but this patch eliminates that dependency which is IMO a good thing.

Cheers,
Wincent