Unicode: update of combining code points
PATCH 1/1 Unicode: update of combining code points
Unicode 6.3 defines the following code as combining or accents, git_wcwidth() should return 0.
Earlier unicode standards had defined these code point as "reserved":
358 COMBINING DOT ABOVE RIGHT
359 COMBINING ASTERISK BELOW
35A COMBINING DOUBLE RING BELOW
35B COMBINING ZIGZAG ABOVE
35C COMBINING DOUBLE BREVE BELOW
487 COMBINING CYRILLIC POKRYTIE
5A2 HEBREW ACCENT ATNAH HAFUKH,
5BA HEBREW POINT HOLAM HASER FOR VAV
5C5 HEBREW MARK LOWER DOT
5C7 HEBREW POINT QAMATS QATAN
604 ARABIC SIGN SAMVAT
616 ARABIC SMALL HIGH LIGATURE ALEF WITH LAM WITH YEH 617 ARABIC SMALL HIGH ZAIN
618 ARABIC SMALL FATHA
619 ARABIC SMALL DAMMA
61A ARABIC SMALL KASRA
659 ARABIC ZWARAKAY
65A ARABIC VOWEL SIGN SMALL V ABOVE
65B ARABIC VOWEL SIGN INVERTED SMALL V ABOVE 65C ARABIC VOWEL SIGN DOT BELOW
65D ARABIC REVERSED DAMMA
65E ARABIC FATHA WITH TWO DOTS
65F ARABIC WAVY HAMZA BELOW
This commit touches only the range 300-6FF, there may be more to be updated.
--- utf8.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
utf8.c +4 -5
@@ -84,11 +84,10 @@ static int git_wcwidth(ucs_char_t ch)* "uniset +cat=Me +cat=Mn +cat=Cf -00AD +1160-11FF +200B c". */ static const struct interval combining[] = { - {0x0300 ,0x0357 }, {0x035D ,0x036F }, {0x0483 ,0x0486 }, - {0x0488 ,0x0489 }, {0x0591 ,0x05A1 }, {0x05A3 ,0x05B9 }, - {0x05BB ,0x05BD }, {0x05BF ,0x05BF }, {0x05C1 ,0x05C2 }, - {0x05C4 ,0x05C4 }, {0x0600 ,0x0603 }, {0x0610 ,0x0615 }, - {0x064B ,0x0658 }, {0x0670 ,0x0670 }, {0x06D6 ,0x06E4 }, + {0x0300 ,0x036F }, {0x0483 ,0x0489 }, {0x0591 ,0x05BD }, + {0x05BF ,0x05BF }, {0x05C1 ,0x05C2 }, {0x05C4 ,0x05C5 }, + {0x05C7 ,0x05C7 }, {0x0600 ,0x0604 }, {0x0610 ,0x061A }, + {0x064B ,0x065F }, {0x0670 ,0x0670 }, {0x06D6 ,0x06E4 }, {0x06E7 ,0x06E8 }, {0x06EA ,0x06ED }, {0x070F ,0x070F }, {0x0711 ,0x0711 }, {0x0730 ,0x074A }, {0x07A6 ,0x07B0 }, {0x0901 ,0x0902 }, {0x093C ,0x093C }, {0x0941 ,0x0948 },
Signature · 1 line
1.9.0
Torsten Bögershausen:
utf8.c
Quoted 4 lines
utf8.c +0
-0
Is there a script that generates this code from the Unicode database files, or did you hand-update it?
Signature · 1 line
\\// Peter - http://www.softwolves.pp.se/
Quoted 9 lines · Peter Krefting
Torsten Bögershausen:
utf8.c
Quoted 4 lines
utf8.c +0
-0Is there a script that generates this code from the Unicode database files, or did you hand-update it?
Some of the code points which have "0 length on the display" are called "combining", others are called "vowels" or "accents".
E.g. 5BF is not marked any of them, but if you look at the glyph, it should be combining (please correct me if that is wrong).
If I could have found a file which indicates for each code point, what it is, I could write a script.
So yes, it is updated by hand.
Quoted 9 lines · Torsten Bögershausen
Quoted 8 lines · Peter Krefting
Torsten Bögershausen:
utf8.c
Quoted 4 lines
utf8.c +0
-0Is there a script that generates this code from the Unicode database files, or did you hand-update it?
Quoted 4 lines
Some of the code points which have "0 length on the display" are called "combining", others are called "vowels" or "accents".
E.g. 5BF is not marked any of them, but if you look at the glyph, it should be combining (please correct me if that is wrong).
Indeed it is combining (more specifically it has General Category "Nonspacing_Mark" = "Mn").
Quoted 4 lines
If I could have found a file which indicates for each code point, what it is, I could write a script.
The most complete and machine-readable data are in these files:
http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt
http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt
The general categories can also be seen more legibly in:
http://www.unicode.org/Public/UCD/latest/ucd/extracted/DerivedGeneralCategory.txt
For docs, see:
http://www.unicode.org/reports/tr44/
http://www.unicode.org/reports/tr11/
http://www.unicode.org/ucd/
The existing utf8.c comments describe the attributes being selected from the tables (general categories "Cf","Mn","Me", East Asian Width "W", "F"). And they suggest that the combining character table was originally auto-generated from UnicodeData.txt with a "uniset" tool. Presumably this?
https://github.com/depp/uniset
The fullwidth-checking code looks like it was done by hand, although apparently uniset can process EastAsianWidth.txt.
Kevin
Quoted 37 lines · Kevin Bracey
Quoted 9 lines · Torsten Bögershausen
Quoted 8 lines · Peter Krefting
Torsten Bögershausen:
utf8.c
Quoted 4 lines
utf8.c +0
-0Is there a script that generates this code from the Unicode database files, or did you hand-update it?
Quoted 4 lines
Some of the code points which have "0 length on the display" are called "combining", others are called "vowels" or "accents".
E.g. 5BF is not marked any of them, but if you look at the glyph, it should be combining (please correct me if that is wrong).Indeed it is combining (more specifically it has General Category "
Nonspacing_Mark" = "Mn").Quoted 4 lines
If I could have found a file which indicates for each code point, what it is, I could write a script.
The most complete and machine-readable data are in these files:
http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt
http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txtThe general categories can also be seen more legibly in:
http://www.unicode.org/Public/UCD/latest/ucd/extracted/DerivedGeneralCategory.txt
For docs, see:
http://www.unicode.org/reports/tr44/
http://www.unicode.org/reports/tr11/
http://www.unicode.org/ucd/
Quoted 7 lines
The existing
utf8.ccomments describe the attributes being selected from the tables (general categories "Cf","Mn","Me", East Asian Width "W", "F"). And they suggest that the combining character table was originally auto-generated from UnicodeData.txt with a "uniset" tool. Presumably this?https://github.com/depp/uniset
The fullwidth-checking code looks like it was done by hand, although apparently uniset can process EastAsianWidth.txt.
Kevin
Excellent, thanks for the pointers.
Running the script below shows that
"0X00AD SOFT HYPHEN" should have zero length (and some others too). I wonder if that is really the case, and which one of the last 2 lines in the script is the right one.
What does this mean for us:
"Cf Format a format control character"
#!/bin/sh
if ! test -f UnicodeData.txt; then
wget http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt
fi &&
if ! test -f EastAsianWidth.txt; then
wget http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt
fi
if ! test -f DerivedGeneralCategory.txt; then
wget http://www.unicode.org/Public/UCD/latest/ucd/extracted/DerivedGeneralCategory.txt
fi &&
if ! test -d uniset; then
git clone https://github.com/tboegi/uniset.git
fi &&
(
cd uniset &&
if ! test -x uniset; then
autoreconf -i &&
./configure --enable-warnings=-Werror CFLAGS='-O0 -ggdb' fi &&
make
) &&
UNICODE_DIR=. ./uniset/uniset --32 cat:Me,Mn,Cf
#UNICODE_DIR=. ./uniset/uniset --32 cat:Me,Mn
Quoted 5 lines
Signature · 4 lines
-- To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Torsten Bögershausen
Excellent, thanks for the pointers.
Running the script below shows that
"0X00ADSOFT HYPHEN" should have zero length (and some others too). I wonder if that is really the case, and which one of the last 2 lines in the script is the right one.What does this mean for us:
"Cf Format a format control character"
Maybe dig back through the Git logs to check the original logic, but the comments suggest that "Cf" characters have been viewed as zero-width. That makes sense - they're usually markers indicating things like bidirectional text flow, so won't be taking space. (Although they may be causing even more extreme layout effects...)
Soft-hyphen is noted as an explicit exception to the rule in the utf8.c comments. As of Unicode 4.0, it's supposed to be a character indicating a point where a hyphen could be placed if a line-wrap occurs, and if that wrap happens, then it can actually take up 1 space, otherwise not. So its width could be either 0 or 1, depending. Or, quite likely, the terminal doesn't treat it specially, and it always just looks like a hyphen... Thus we err on the safe side and give it width 1.
See http://en.wikipedia.org/wiki/Soft_hyphen for background.
The comments suggest adding "-00AD +1160-11FF" to the uniset command line for that tweak and for composing Hangul. (The +200B tweak isn't necessary any more - Zero-Width Space U+200B became Cf officially in Unicode 4.0.1:
http://en.wikipedia.org/wiki/Zero-width_space
http://www.unicode.org/review/resolved-pri.html#pri21
)
All of this is only really an approximation - a best-effort attempt to figure out the width of a string without any actual communication with the display device. So it'll never be perfect. The choice between double and single width in particular will often be unpredictable, unless you had deeper locale knowledge.
Actually, while doing this, I've realised that this was originally Markus Kuhn's implementation, and that is acknowledged at the top of the file:
http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
Good, because he knows what he's doing.
Kevin
Torsten Bögershausen:
Quoted 4 lines
Some of the code points which have "0 length on the display" are called "combining", others are called "vowels" or "accents".
E.g. 5BF is not marked any of them, but if you look at the glyph, it should be combining (please correct me if that is wrong).
All combining characters has a non-zero combining class in http://www.unicode.org/Public/UNIDATA/UnicodeData.txt (fourth field, called Canonical_Combining_Class in http://www.unicode.org/reports/tr44/ ). For instance, the aforementioned U+05BF is defined as follows:
05BF;HEBREW POINT RAFE;Mn;23;NSM;;;;;N;;;;;
The combining class is 23, so this is a combining character.
There is a difference between non-spacing combining marks ("Mn" in the third column (General_Category)) and others ("Mc" for spacing marks and "Me" for enclosing marks), so they might need specifial handling. Additionally, you have the "zero-width" characters, such as U+200B Zero Width Space. These have the "Cf" class, although it also contains visible characters IIRC.
Signature · 1 line
\\// Peter - http://www.softwolves.pp.se/