update_unicode.sh: update the uniset repo if it exists
PATCH 2/3 update_unicode.sh: remove the plane filters
In reply to Beat BolliThe uniset upstream has accepted my patches that eliminate the Unicode plane offsets from the output in '--32' mode.
Remove the corresponding filter in update_unicode.sh.
--- update_unicode.sh | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
update_unicode.sh +2 -3
@@ -31,11 +31,10 @@ fi && UNICODE_DIR=. && export UNICODE_DIR && cat >$UNICODEWIDTH_H <<-EOF static const struct interval zero_width[] = { - $(uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD | - grep -v plane) + $(uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD) }; static const struct interval double_width[] = { - $(uniset/uniset --32 eaw:F,W | grep -v plane) + $(uniset/uniset --32 eaw:F,W) }; EOF )
Signature · 1 line
2.7.2
PATCH 3/3 update_unicode.sh: restore hexadecimal output
In reply to Beat BolliThe uniset upstream has decided that decimal numbers are The True Way, so let's convert them back to the usual format that's closer to the U+nnnn standard.
The generated unicode_widths.h file again looks exactly the same as two commits ago.
--- update_unicode.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
update_unicode.sh +7 -1
@@ -5,6 +5,12 @@ #Mn Nonspacing_Mark a nonspacing combining mark (zero advance width) #Cf Format a format control character # + +dec_to_hex() { + # convert any decimal numbers to 4-digit hex + perl -pe 's/(\d+)/sprintf("0x%04X", $1)/ge' +} + UNICODEWIDTH_H=../unicode_width.h if ! test -d unicode; then mkdir unicode
@@ -29,7 +35,7 @@ fi && make ) && UNICODE_DIR=. && export UNICODE_DIR && - cat >$UNICODEWIDTH_H <<-EOF + dec_to_hex >$UNICODEWIDTH_H <<-EOF static const struct interval zero_width[] = { $(uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD) };
Signature · 1 line
2.7.2
PATCH 1/3 update_unicode.sh: update the uniset repo if it exists
We need to track the new commits in uniset, otherwise their and our code get out of sync.
---
Junio, these go on top of my bb/unicode-9.0 branch, please.
Thanks!
update_unicode.sh | 5 +++++ 1 file changed, 5 insertions(+)
update_unicode.sh +5 -0
@@ -14,6 +14,11 @@ fi && http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt && if ! test -d uniset; then git clone https://github.com/depp/uniset.git + else + ( + cd uniset && + git pull + ) fi && ( cd uniset &&
Signature · 1 line
2.7.2
Quoted 18 lines · Beat Bolli
We need to track the new commits in uniset, otherwise their and our code get out of sync.
Signed-off-by Beat Bolli <[email protected]>---
Junio, these go on top of my bb/unicode-9.0 branch, please.
Thanks!
update_unicode.sh | 5 +++++ 1 file changed, 5 insertions(+)update_unicode.sh +0
-0
update_unicode.sh fi && · line 20
Quoted 7 lines
update_unicode.sh +4
-0@@ -14,6 +14,11 @@ fi && http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt && if ! test -d uniset; then git clone https://github.com/depp/uniset.git + else + ( + cd uniset && + git pull
If upstream has accepted your patches, that's nice.
Minor question, especially to the next commit:
Should we make sure to checkout the exact version, which has been tested? In this case cb97792880625e24a9f581412d03659091a0e54f
And this is for both a fresh clone and the git pull needs to be replaced by
git fetch && git checkout cb97792880625e24a9f581412d03659091a0e54f
(Which of course is a shell variable
Quoted 32 lines · Torsten Bögershausen
Quoted 18 lines · Beat Bolli
We need to track the new commits in uniset, otherwise their and our code get out of sync.
Signed-off-by Beat Bolli <[email protected]>---
Junio, these go on top of my bb/unicode-9.0 branch, please.
Thanks!
update_unicode.sh | 5 +++++ 1 file changed, 5 insertions(+)update_unicode.sh +0
-0update_unicode.sh fi && · line 20
Quoted 7 lines
update_unicode.sh +4
-0@@ -14,6 +14,11 @@ fi && http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt && if ! test -d uniset; then git clone https://github.com/depp/uniset.git + else + ( + cd uniset && + git pullIf upstream has accepted your patches, that's nice.
Minor question, especially to the next commit:
Should we make sure to checkout the exact version, which has been tested? In this casecb97792880625e24a9f581412d03659091a0e54f
Quoted 6 lines
And this is for both a fresh clone and the git pull needs to be replaced by
git fetch && git checkoutcb97792880625e24a9f581412d03659091a0e54f(Which of course is a shell variable)
I was actually wondering what the policy was for adding submodules to the Git repo, but then decided against it. Another option would be to fork uniset on GitHub and just let it stay on a working commit.
Junio, what's your stance on this?
Beat
Quoted 12 lines
Quoted 11 lines
Minor question, especially to the next commit:
Should we make sure to checkout the exact version, which has been tested? In this casecb97792880625e24a9f581412d03659091a0e54fAnd this is for both a fresh clone and the git pull needs to be replaced by
git fetch && git checkoutcb97792880625e24a9f581412d03659091a0e54f(Which of course is a shell variable)
Quoted 7 lines
I was actually wondering what the policy was for adding submodules to the Git repo, but then decided against it. Another option would be to fork uniset on GitHub and just let it stay on a working commit.
Junio, what's your stance on this?
Beat
If I run ./update_unicode.sh on the latest master of https://github.com/depp/uniset.git ,
commit a5fac4a091857dd5429cc2d, I get a diff in unicode_width.h like this:
-{ 0x0300, 0x036F },
+{ 768, 879 },
IOW, all hex values are printed as decimal values. Not a problem for the compiler, but for the human to check the unicode tables.
So I think we should "pin" the version of uniset.
Quoted 9 lines · Torsten Bögershausen
If I run ./update_unicode.sh on the latest master of
https://github.com/depp/uniset.git , commita5fac4a091857dd5429cc2d, I get a diff inunicode_width.hlike this:(quoted diff) +1
-1-{ 0x0300, 0x036F }, +{ 768, 879 },
Quoted 5 lines
IOW, all hex values are printed as decimal values. Not a problem for the compiler, but for the human to check the unicode tables.
So I think we should "pin" the version of uniset.
Sure, and I'd rather see the update-unicode.sh script moved somewhere in contrib/ while at it. Those who are interested in keeping up with the unicode standard are tiny minority of the developer population, and most of us would treat the built width table as the source (after all, that is what we ship).
To be bluntly honest, I'd rather not to see "update-unicode.sh" download and build uniset at all. It's as if po/ hierarchy shipping with its own script to download and build msgmerge--that's madness. Needless to say, shipping the sources for uniset embedded in our project tree (either as a snapshot-fork or as a submodule) is even worse. Those who want to muck with po/ are expected to have msgmerge and friends. Why not expect the same for those who want to update the unicode width table?
I'd rather see a written instruction telling which snapshot to get and from where to build and place on their $PATH in the README file, sitting next to the update-unicode.sh script in contrib/uniwidth/ directory, for those who are interested in building the width table "from the source", and the update-unicode.sh script to assume that uniset is available.
Quoted 27 lines · Torsten Bögershausen
Quoted 12 lines
Quoted 11 lines
Minor question, especially to the next commit:
Should we make sure to checkout the exact version, which has been tested? In this casecb97792880625e24a9f581412d03659091a0e54fAnd this is for both a fresh clone and the git pull needs to be replaced by
git fetch && git checkoutcb97792880625e24a9f581412d03659091a0e54f(Which of course is a shell variable)
Quoted 7 lines
I was actually wondering what the policy was for adding submodules to the Git repo, but then decided against it. Another option would be to fork uniset on GitHub and just let it stay on a working commit.
Junio, what's your stance on this?
Beat
If I run ./update_unicode.sh on the latest master of https://github.com/depp/uniset.git ,
commita5fac4a091857dd5429cc2d, I get a diff inunicode_width.hlike this:(quoted diff) +1
-1-{ 0x0300, 0x036F }, +{ 768, 879 },
Quoted 5 lines
IOW, all hex values are printed as decimal values. Not a problem for the compiler, but for the human to check the unicode tables.
So I think we should "pin" the version of uniset.
That's what patch 3/3 fixes.
Quoted 32 lines · Junio C Hamano
Quoted 9 lines · Torsten Bögershausen
If I run ./update_unicode.sh on the latest master of
https://github.com/depp/uniset.git , commita5fac4a091857dd5429cc2d, I get a diff inunicode_width.hlike this:(quoted diff) +1
-1-{ 0x0300, 0x036F }, +{ 768, 879 },Quoted 5 lines
IOW, all hex values are printed as decimal values. Not a problem for the compiler, but for the human to check the unicode tables.
So I think we should "pin" the version of uniset.
Sure, and I'd rather see the
update-unicode.shscript moved somewhere in contrib/ while at it. Those who are interested in keeping up with the unicode standard are tiny minority of the developer population, and most of us would treat the built width table as the source (after all, that is what we ship).To be bluntly honest, I'd rather not to see "
update-unicode.sh" download and build uniset at all. It's as if po/ hierarchy shipping with its own script to download and build msgmerge--that's madness. Needless to say, shipping the sources for uniset embedded in our project tree (either as a snapshot-fork or as a submodule) is even worse. Those who want to muck with po/ are expected to have msgmerge and friends. Why not expect the same for those who want to update the unicode width table?
Quoted 7 lines
I'd rather see a written instruction telling which snapshot to get and from where to build and place on their $PATH in the README file, sitting next to the
update-unicode.shscript in contrib/uniwidth/ directory, for those who are interested in building the width table "from the source", and theupdate-unicode.shscript to assume that uniset is available.
OK. So please don't merge bb/unicode-9.0 to next yet; I'll prepare a reroll following your description.
Torsten, is this alright with you?
Cheers, Beat
Quoted 16 lines
Quoted 16 lines
Sure, and I'd rather see the
update-unicode.shscript moved somewhere in contrib/ while at it. Those who are interested in keeping up with the unicode standard are tiny minority of the developer population, and most of us would treat the built width table as the source (after all, that is what we ship).To be bluntly honest, I'd rather not to see "
update-unicode.sh" download and build uniset at all. It's as if po/ hierarchy shipping with its own script to download and build msgmerge--that's madness. Needless to say, shipping the sources for uniset embedded in our project tree (either as a snapshot-fork or as a submodule) is even worse. Those who want to muck with po/ are expected to have msgmerge and friends. Why not expect the same for those who want to update the unicode width table?
Quoted 6 lines
Quoted 6 lines
I'd rather see a written instruction telling which snapshot to get and from where to build and place on their $PATH in the README file, sitting next to the
update-unicode.shscript in contrib/uniwidth/ directory, for those who are interested in building the width table "from the source", and theupdate-unicode.shscript to assume that uniset is available.
OK with the contrib - that's an improvement.
About the instructions how to download and compile:
(we don't need to change the $PATH, do we ?)
I don't know.
The typical instructions I have seen are a sequence of shell commands to be executed, which hopefully simply work by doing "copy-and-paste". I find this error-prone, as you you may loose the last character while moving the mouse, or don't check the error message or return codes. Having a pre-baked shell script, which does use "&&" is in that way more attractive, and the README can be as simple as run "update-unicode.sh" and that's it.
uniset is a small project and where should we put it ?
a) inside the Git tree?
b) /tmp ?
c) into the $HOME directory ?
d) /usr/local
a) is quick and dirty
b) probably OK
c) Not sure about tha
d) Needs super user rights
Can we try to find a good place ?
"contrib/uniwidth/" may be different to find, how about contrib/update-unicode ?
Quoted 4 lines
OK. So please don't merge bb/unicode-9.0 to next yet; I'll prepare a reroll following your description.
Torsten, is this alright with you?
sure
Quoted 1 line
Cheers, Beat
Quoted 7 lines · Torsten Bögershausen
The typical instructions I have seen are a sequence of shell commands to be executed, which hopefully simply work by doing "copy-and-paste". I find this error-prone, as you you may loose the last character while moving the mouse, or don't check the error message or return codes. Having a pre-baked shell script, which does use "&&" is in that way more attractive,
and the README can be as simple as run "update-unicode.sh" and that's it.
That's OK as well.
Quoted 1 line
"contrib/uniwidth/" may be different to find, how about contrib/update-unicode ?
This, too. And as long as .gitignore pattern is set up correctly there, I do not think we terribly mind "git clone ..from..there.." into it, either.