Make it possible to update git_wcwidth()
PATCH 2/2 Make it possible to update git_wcwidth()
The function git_wcwidth() returns for a given unicode code point the width on the display:
-1 for control characters,
0 for combining or other non-visible code points
1 for e.g. ASCII
2 for double-width code points.
This table had been originally been extracted for one Unicode version, probably 3.2.
Make it possible to update the to a later version of Unicode:
Factor out the table from utf8.c into unicode_width.h and add the script update_unicode.sh to update the table based on the latest Unicode specification files.
Thanks to
Peter Krefting <[email protected]> and
Kevin Bracey <[email protected]>
for helping with their Unicode knowledge
--- .gitignore | 1 + Makefile | 1 + unicode_width.h | 288 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ update_unicode.sh | 37 +++++++ utf8.c | 61 +----------- 5 files changed, 329 insertions(+), 59 deletions(-) create mode 100644 unicode_width.h create mode 100755 update_unicode.sh
.gitignore +1 -0
@@ -226,6 +226,7 @@ /config.mak.autogen /config.mak.append /configure +/unicode /tags /TAGS /cscope*
Makefile +1 -0
@@ -730,6 +730,7 @@ LIB_H += transport.h LIB_H += tree-walk.h LIB_H += tree.h LIB_H += unpack-trees.h +LIB_H += unicode_width.h LIB_H += url.h LIB_H += urlmatch.h LIB_H += userdiff.h
unicode_width.h new file +288 -0
288 lines not on this page · raw message
update_unicode.sh new file +37 -0
@@ -0,0 +1,37 @@ +#!/bin/sh +#See http://www.unicode.org/reports/tr44/ +# +#Me Enclosing_Mark an enclosing combining mark +#Mn Nonspacing_Mark a nonspacing combining mark (zero advance width) +#Cf Format a format control character +# +UNICODEWIDTH_H=../unicode_width.h +if ! test -d unicode; then + mkdir unicode +fi && +( cd unicode && + if ! test -f UnicodeData.txt; then + wget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt + fi && + if ! test -f EastAsianWidth.txt; then + wget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt + fi && + if ! test -d uniset; then + git clone https://github.com/depp/uniset.git + fi && + ( + cd uniset && + if ! test -x uniset; then + autoreconf -i && + ./configure --enable-warnings=-Werror CFLAGS='-O0 -ggdb' + fi && + make + ) && + echo "static const struct interval zero_width[] = {" >$UNICODEWIDTH_H && + UNICODE_DIR=. ./uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD | + grep -v plane >>$UNICODEWIDTH_H && + echo "};" >>$UNICODEWIDTH_H && + echo "static const struct interval double_width[] = {" >>$UNICODEWIDTH_H && + UNICODE_DIR=. ./uniset/uniset --32 eaw:F,W >>$UNICODEWIDTH_H && + echo "};" >>$UNICODEWIDTH_H +)
utf8.c +2 -59
74 lines not on this page · raw message
Signature · 1 line
1.9.2.691.g8d8dc6d
Torsten Bögershausen:
Quoted 6 lines
The function
git_wcwidth()returns for a given unicode code point the width on the display:
-1 for control characters,
0 for combining or other non-visible code points
1 for e.g. ASCII
2 for double-width code points.
This all looks sane, but the problem is that this is also context-dependent since there are a lot of characters with "ambiguous" widths, i.e., characters that are "double" width for CJK locales (and fonts), but "single" width for others. This includes Greek and Cyrillic characters, which are encoded using the double-byte parts of the CJK DBCS encodings.
I'm not quite sure how much impact this would have on day-to-day Git operation in a CJK locale, however, as I guess they would mostly encounter texts in their own language (which would mostly be "double" width) or English (which would be unambiguously "single" width).
Anyone on the list running Git in a CJK locale that would like to weigh in here?
Signature · 1 line
\\// Peter - http://www.softwolves.pp.se/
Quoted 16 lines · Peter Krefting
Torsten Bögershausen:
Quoted 6 lines
The function
git_wcwidth()returns for a given unicode code point the width on the display:
-1 for control characters,
0 for combining or other non-visible code points
1 for e.g. ASCII
2 for double-width code points.This all looks sane, but the problem is that this is also context-dependent since there are a lot of characters with "ambiguous" widths, i.e., characters that are "double" width for CJK locales (and fonts), but "single" width for others. This includes Greek and Cyrillic characters, which are encoded using the double-byte parts of the CJK DBCS encodings.
Quoted 7 lines
I'm not quite sure how much impact this would have on day-to-day Git operation in a CJK locale, however, as I guess they would mostly encounter texts in their own language (which would mostly be "double" width) or English (which would be unambiguously "single" width).
Anyone on the list running Git in a CJK locale that would like to weigh in here?
The issue does appear in the real life. A solution I've seen used in a terminaul emulator program was to give the user a choice to say "I want ambiguous ones to be treated as double (or single)". As a J-locale user, I naturally set the configuration to double while using that program (I no longer do).
Quoted 16 lines · Torsten Bögershausen
The function
git_wcwidth()returns for a given unicode code point the width on the display:
-1 for control characters,
0 for combining or other non-visible code points
1 for e.g. ASCII
2 for double-width code points.This table had been originally been extracted for one Unicode version, probably 3.2.
Make it possible to update the to a later version of Unicode:
Factor out the table from
utf8.cintounicode_width.hand add the scriptupdate_unicode.shto update the table based on the latest Unicode specification files.
Quoted 7 lines
Thanks to
Peter Krefting <[email protected]> and
Kevin Bracey <[email protected]>
for helping with their Unicode knowledgeSigned-off-by Torsten Bögershausen <[email protected]>---
I would say this is a good idea, but a few nitpicks.
unicode_width.h
Quoted 6 lines
unicode_width.h new file +0
-0@@ -0,0 +1,288 @@
Please update the script (and the resulting file) to caution against misuse/mismanagement of this file by adding a comment to at least state:
- This is a generated file and you are not supposed to modify it; and
- This is to be included only once from one place in the code and that is why this does not use the usual #ifndef X/#define X/#endif double-inclusion guards.
An obvious and viable alternative to the second would be to do the usual double-inclusion guard. I do not have much preference either way.
Quoted 12 lines
unicode_width.h new file +4
-0+static const struct interval zero_width[] = { +}; +static const struct interval double_width[] = { +};update_unicode.sh new file +0
-0
update_unicode.sh line 6
Quoted 6 lines
update_unicode.sh new file +6
-0@@ -0,0 +1,37 @@ +#!/bin/sh +#See http://www.unicode.org/reports/tr44/ +# +#Me Enclosing_Mark an enclosing combining mark +#Mn Nonspacing_Mark a nonspacing combining mark (zero advance width) +#Cf Format a format control character
Please have a SP after # in these comments to make them readable?
update_unicode.sh line 11
Quoted 5 lines
update_unicode.sh new file +5
-0+# +UNICODEWIDTH_H=../unicode_width.h +if ! test -d unicode; then + mkdir unicode +fi &&
Style:
if ! test -d unicode
then
...
fiQuoted 5 lines
update_unicode.sh new file +5
-0+( cd unicode && + if ! test -f UnicodeData.txt; then + wget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt + fi && + if ! test -f EastAsianWidth.txt; then
update_unicode.sh line 26
Quoted 10 lines
update_unicode.sh new file +10
-0+ wget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt + fi && + if ! test -d uniset; then + git clone https://github.com/depp/uniset.git + fi && + ( + cd uniset && + if ! test -x uniset; then + autoreconf -i && + ./configure --enable-warnings=-Werror CFLAGS='-O0 -ggdb'
What are these "-O0 -ggdb" about?
Quoted 12 lines
update_unicode.sh new file +11
-0+ fi && + make + ) && + echo "static const struct interval zero_width[] = {" >$UNICODEWIDTH_H && + UNICODE_DIR=. ./uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD | + grep -v plane >>$UNICODEWIDTH_H && + echo "};" >>$UNICODEWIDTH_H && + echo "static const struct interval double_width[] = {" >>$UNICODEWIDTH_H && + UNICODE_DIR=. ./uniset/uniset --32 eaw:F,W >>$UNICODEWIDTH_H && + echo "};" >>$UNICODEWIDTH_H +)
update_unicode.sh static int git_wcwidth(ucs_char_t ch) · line 94
Quoted 6 lines
update_unicode.sh new file +1
-1@@ -147,7 +90,7 @@ static int git_wcwidth(ucs_char_t ch) return -1; /* binary search in table of non-spacing characters */ - if (bisearch(ch, combining, sizeof(combining) + if (bisearch(ch, zero_width, sizeof(zero_width) / sizeof(struct interval) - 1))
To my eyes, that line looks folded at a funny place. I think it is more conventional to fold after the operator, i.e.
if (bisearch(ch, zero_width, sizeof(zero_width) /
sizeof(struct interval) - 1))
but
if (bisearch(ch, zero_width,
sizeof(zero_width) / sizeof(struct interval) - 1))
may probably be a lot more logical and readable. Maybe it is just me.