Skip to contentSkip to conversation

t/perf: correctly align non-ASCII descriptions in output

PATCH In review Ævar Arnfjörð Bjarmason 2 participants 6 messages Apr 21, 2017

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 t/perf: correctly align non-ASCII descriptions in output

Change the test descriptions from being treated as binary blobs by perl to being treated as UTF-8. This ensures that e.g. a test description like "æ" is counted as 1 character, not 2.

I have WIP performance tests for non-ASCII grep patterns on another topic that are affected by this.

Now instead of:

    $ ./run p0000-perf-lib-sanity.sh
    [...]
    0000.4: export a weird var                                    0.00(0.00+0.00)
    0000.5: éḿíẗ ńöń-ÁŚĆÍÍ ćḧáŕáćẗéŕś   0.00(0.00+0.00)
    0000.7: important variables available in subshells            0.00(0.00+0.00)
    [...]

We emit:

    [...]
    0000.4: export a weird var                                 0.00(0.00+0.00)
    0000.5: éḿíẗ ńöń-ÁŚĆÍÍ ćḧáŕáćẗéŕś                          0.00(0.00+0.00)
    0000.7: important variables available in subshells         0.00(0.00+0.00)
    [...]

Fixes code originally added in 342e9ef2d9 ("Introduce a performance testing framework", 2012-02-17).

---
 t/perf/aggregate.perl           | 3 +++
 t/perf/p0000-perf-lib-sanity.sh | 2 ++
 2 files changed, 5 insertions(+)
t/perf/aggregate.perl +3 -0
index 924b19dab4..1dbc85b214 100755
@@ -88,6 +88,7 @@ for my $t (@tests) {
 sub read_descr {
 	my $name = shift;
 	open my $fh, "<", $name or return "<error reading description>";
+	binmode $fh, ":utf8" or die "PANIC on binmode: $!";
 	my $line = <$fh>;
 	close $fh or die "cannot close $name";
 	chomp $line;
@@ -147,6 +148,8 @@ for my $t (@subtests) {
 my $totalwidth = 3*@dirs+$descrlen;
 $totalwidth += $_ for (@colwidth);
 
+binmode STDOUT, ":utf8" or die "PANIC on binmode: $!";
+
 printf "%-${descrlen}s", "Test";
 for my $i (0..$#dirs) {
 	my $d = $dirs[$i];
t/perf/p0000-perf-lib-sanity.sh +2 -0
index cf8e1efce7..002c21e52a 100755
@@ -33,6 +33,8 @@ test_perf 'export a weird var' '
 	test_export bar
 '
 
+test_perf 'éḿíẗ ńöń-ÁŚĆÍÍ ćḧáŕáćẗéŕś' 'true'
+
 test_expect_success 'test_export works with weird vars' '
 	echo "$bar" &&
 	test "$bar" = "weird # variable"
Signature · 1 line
2.11.0
#1 lore raw
Quoted 6 lines · Ævar Arnfjörð Bjarmason

Change the test descriptions from being treated as binary blobs by perl to being treated as UTF-8. This ensures that e.g. a test description like "æ" is counted as 1 character, not 2.

I have WIP performance tests for non-ASCII grep patterns on another topic that are affected by this.

Makes sense. As this is purely about test titles in our project, choosing utf8 as the only encoding is quite sensible.

t/perf/aggregate.perl for my $t (@tests) { · line 91

Quoted 9 lines
t/perf/aggregate.perl +1 -0
index 924b19dab4..1dbc85b214 100755
@@ -88,6 +88,7 @@ for my $t (@tests) {
 sub read_descr {
 	my $name = shift;
 	open my $fh, "<", $name or return "<error reading description>";
+	binmode $fh, ":utf8" or die "PANIC on binmode: $!";

I thought there was some "use" flag we could set to just make all of our handles utf8. But all I could come up with was stuff like PERLIO and "perl -C". Using binmode isn't too bad, though (I think you could just do it as part of the open, too, but I'm not sure if antique versions of perl support that).

-Peff

#2 lore raw
Quoted 11 lines · Jeff King
Quoted 6 lines · Ævar Arnfjörð Bjarmason

Change the test descriptions from being treated as binary blobs by perl to being treated as UTF-8. This ensures that e.g. a test description like "æ" is counted as 1 character, not 2.

I have WIP performance tests for non-ASCII grep patterns on another topic that are affected by this.

Makes sense. As this is purely about test titles in our project, choosing utf8 as the only encoding is quite sensible.

*Nod*

Quoted 7 lines
Quoted 7 lines
t/perf/aggregate.perl +0 -0
index 924b19dab4..1dbc85b214 100755
@@ -88,6 +88,7 @@ for my $t (@tests) {
 sub read_descr {
      my $name = shift;
Quoted 8 lines

t/perf/aggregate.perl for my $t (@tests) { · line 91

Quoted 2 lines
t/perf/aggregate.perl +1 -0
      open my $fh, "<", $name or return "<error reading description>";
+     binmode $fh, ":utf8" or die "PANIC on binmode: $!";

I thought there was some "use" flag we could set to just make all of our handles utf8. But all I could come up with was stuff like PERLIO and "perl -C". Using binmode isn't too bad, though (I think you could just do it as part of the open, too, but I'm not sure if antique versions of perl support that).

[Debugging perl encoding issues is one of the many perks of my dayjob]

Using binmode like this is about as straightforward as you can get, the former occurrence could be equivalently replaced by:

    utf8::decode(my $line = <$fh>);

But better just to mark the handle as utf8. There's a fancier way to do it as part of the three-arg-open syntax, but I couldn't remember whether all the perl versions we support have it.

About the "use" flag, you're probably thinking of the confusingly named "use utf8", but that's to set your source code to utf8, not your handles, e.g.:

$ perl -CA -MDevel::Peek -wE 'use utf8; my $日本語 = shift; Dump $日本語' æ
SV = PV(0x12cc090) at 0x12cded8
  REFCNT = 1
  FLAGS = (PADMY,POK,pPOK,UTF8)
  PV = 0x12de460 "\303\246"\0 [UTF8 "\x{e6}"]
  CUR = 2
  LEN = 16

As you can see people got a bit overexcited about Unicode in the 90s.

#3 lore raw
Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 5 lines

I thought there was some "use" flag we could set to just make all of our handles utf8. But all I could come up with was stuff like PERLIO and "perl -C". Using binmode isn't too bad, though (I think you could just do it as part of the open, too, but I'm not sure if antique versions of perl support that).

[Debugging perl encoding issues is one of the many perks of my dayjob]

Quoted 8 lines

Using binmode like this is about as straightforward as you can get, the former occurrence could be equivalently replaced by:

    utf8::decode(my $line = <$fh>);

But better just to mark the handle as utf8. There's a fancier way to do it as part of the three-arg-open syntax, but I couldn't remember whether all the perl versions we support have it.

Yeah, I agree marking the handle is better. binmode is pretty straightforward, but we'd have to remember to manually set it if we add any other handles. That's probably not a big deal in this particular script, though, which is pretty short.

Quoted 12 lines

About the "use" flag, you're probably thinking of the confusingly named "use utf8", but that's to set your source code to utf8, not your handles, e.g.:

$ perl -CA -MDevel::Peek -wE 'use utf8; my $日本語 = shift; Dump $日本語' æ
SV = PV(0x12cc090) at 0x12cded8
  REFCNT = 1
  FLAGS = (PADMY,POK,pPOK,UTF8)
  PV = 0x12de460 "\303\246"\0 [UTF8 "\x{e6}"]
  CUR = 2
  LEN = 16
Quoted 1 line

As you can see people got a bit overexcited about Unicode in the 90s.

Yeah, I know "use utf8" doesn't work for that, but I was thinking there was some other trick. Digging...ah, here it is:

use open ':encoding(utf8)'

No clue how portable that is. For such a small script it may be better to just stick with vanilla binmode().

-Peff

#4 lore raw
Quoted 38 lines · Jeff King
Quoted 8 lines · Ævar Arnfjörð Bjarmason
Quoted 5 lines

I thought there was some "use" flag we could set to just make all of our handles utf8. But all I could come up with was stuff like PERLIO and "perl -C". Using binmode isn't too bad, though (I think you could just do it as part of the open, too, but I'm not sure if antique versions of perl support that).

[Debugging perl encoding issues is one of the many perks of my dayjob]

Quoted 8 lines

Using binmode like this is about as straightforward as you can get, the former occurrence could be equivalently replaced by:

    utf8::decode(my $line = <$fh>);

But better just to mark the handle as utf8. There's a fancier way to do it as part of the three-arg-open syntax, but I couldn't remember whether all the perl versions we support have it.

Yeah, I agree marking the handle is better. binmode is pretty straightforward, but we'd have to remember to manually set it if we add any other handles. That's probably not a big deal in this particular script, though, which is pretty short.

Quoted 12 lines

About the "use" flag, you're probably thinking of the confusingly named "use utf8", but that's to set your source code to utf8, not your handles, e.g.:

$ perl -CA -MDevel::Peek -wE 'use utf8; my $日本語 = shift; Dump $日本語' æ
SV = PV(0x12cc090) at 0x12cded8
  REFCNT = 1
  FLAGS = (PADMY,POK,pPOK,UTF8)
  PV = 0x12de460 "\303\246"\0 [UTF8 "\x{e6}"]
  CUR = 2
  LEN = 16
Quoted 1 line

As you can see people got a bit overexcited about Unicode in the 90s.

Quoted 7 lines

Yeah, I know "use utf8" doesn't work for that, but I was thinking there was some other trick. Digging...ah, here it is:

use open ':encoding(utf8)'

No clue how portable that is. For such a small script it may be better to just stick with vanilla binmode().

Yeah that would work, but doesn't work on 5.8.0, which is the lowest version we support.

#5 lore raw
Quoted 10 lines · Ævar Arnfjörð Bjarmason
Quoted 7 lines

Yeah, I know "use utf8" doesn't work for that, but I was thinking there was some other trick. Digging...ah, here it is:

use open ':encoding(utf8)'

No clue how portable that is. For such a small script it may be better to just stick with vanilla binmode().

Yeah that would work, but doesn't work on 5.8.0, which is the lowest version we support.

Thanks for looking that up.

For the benefit of the maintainer, I'll summarize: Ævar's original patch is fine as-is.

-Peff