grep: fix multibyte regex handling under macOS
A message that is not in the archive (sent off-list or not yet received): pull.1313.v2.git.git.1661289990205.gitgitgadget () gmail ! com
PATCH v3 1/1 grep: fix multibyte regex handling under macOS
The commit 29de20504e (Makefile: fix default regex settings on Darwin, 2013-05-11) fixed t0070-fundamental.sh under Darwin (macOS) by adopting Git's regex library. However, this library is compiled with NO_MBSUPPORT, which causes git-grep to work incorrectly on multibyte (e.g. UTF-8) files. Current macOS versions pass t0070-fundamental.sh with the native macOS regex library, which also supports multibyte characters.
Adjust the Makefile to use the native regex library, and call setlocale(3) to set CTYPE according to the user's preference. The setlocale call is required on all platforms, but in platforms supporting gettext(3), setlocale was called as a side-effect of initializing gettext. Therefore, move the CTYPE setlocale call from gettext.c to common-main.c and the corresponding locale.h include into git-compat-util.h.
To avoid running the new tests on the few platforms still using the compatibility library, which is compiled without multibyte support, store the corresponding NO_REGEX setting in the GIT-BUILD-OPTIONS file. This makes it available to the test scripts. Thanks to the global initialization of CTYPE setlocale, the test-tool regex command now works correctly with multibyte regexes and is used to further test a platform's support for them.
---
- Move added setlocale from grep.c and test-regex.c to common-main.c
- Move added locale.h include from the above files to git-compat-util.h
- Correct use of test_expect_code 1
- Have old commit reference in the commit message follow commit referencing conventions
Makefile | 2 +- common-main.c | 1 + gettext.c | 2 -- git-compat-util.h | 1 + t/t7818-grep-multibyte.sh | 34 ++++++++++++++++++++++++++++++++++ 5 files changed, 37 insertions(+), 3 deletions(-) create mode 100755 t/t7818-grep-multibyte.sh
Makefile +1 -1
@@ -1427,7 +1427,6 @@ ifeq ($(uname_S),Darwin) APPLE_COMMON_CRYPTO = YesPlease COMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO endif - NO_REGEX = YesPlease PTHREAD_LIBS = endif
@@ -2970,6 +2969,7 @@ GIT-BUILD-OPTIONS: FORCE @echo NO_PERL=\''$(subst ','\'',$(subst ','\'',$(NO_PERL)))'\' >>$@+ @echo NO_PTHREADS=\''$(subst ','\'',$(subst ','\'',$(NO_PTHREADS)))'\' >>$@+ @echo NO_PYTHON=\''$(subst ','\'',$(subst ','\'',$(NO_PYTHON)))'\' >>$@+ + @echo NO_REGEX=\''$(subst ','\'',$(subst ','\'',$(NO_REGEX)))'\' >>$@+ @echo NO_UNIX_SOCKETS=\''$(subst ','\'',$(subst ','\'',$(NO_UNIX_SOCKETS)))'\' >>$@+ @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+
common-main.c +1 -0
@@ -40,6 +40,7 @@ int main(int argc, const char **argv)git_resolve_executable_dir (argv[0 ]); +setlocale (LC_CTYPE ,"" );git_setup_gettext ();initialize_the_repository ();
gettext.c +0 -2
@@ -10,7 +10,6 @@#include "config.h" #ifndef NO_GETTEXT -# include <locale.h> # include <libintl.h> # ifdef GIT_WINDOWS_NATIVE
@@ -80,7 +79,6 @@ static int test_vsnprintf(const char *fmt, ...)static void init_gettext_charset (const char *domain) { -setlocale (LC_CTYPE ,"" ); charset =locale_charset ();bind_textdomain_codeset (domain, charset);
git-compat-util.h +1 -0
@@ -212,6 +212,7 @@#endif #include <errno.h> #include <limits.h> +#include <locale.h> #ifdef NEEDS_SYS_PARAM_H #include <sys/param.h> #endif
t/t7818-grep-multibyte.sh new file +34 -0
@@ -0,0 +1,34 @@ +#!/bin/sh + +test_description='grep multibyte characters' + +. ./test-lib.sh + +# Multibyte regex search is only supported with a native regex library +# that supports it. +# (The supplied compatibility library is compiled with NO_MBSUPPORT.) +test -z "$NO_REGEX" && + LC_ALL=en_US.UTF-8 test-tool regex '^.$' '¿' && + test_set_prereq MB_REGEX + +if ! test_have_prereq MB_REGEX +then + skip_all='multibyte grep tests; Git compiled with NO_REGEX, NO_MBSUPPORT' + test_done +fi + +test_expect_success 'setup' ' + test_write_lines "¿" >file && + git add file && + LC_ALL="en_US.UTF-8" && + export LC_ALL +' +test_expect_success 'grep exactly one char in single-char multibyte file' ' + git grep "^.$" +' + +test_expect_success 'grep two chars in single-char multibyte file' ' + test_expect_code 1 git grep ".." +' + +test_done
Signature · 1 line
2.30.2
Quoted 13 lines · Diomidis Spinellis
Makefile +0
-1@@ -1427,7 +1427,6 @@ ifeq ($(uname_S),Darwin) APPLE_COMMON_CRYPTO = YesPlease COMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO endif - NO_REGEX = YesPlease PTHREAD_LIBS = endif
Makefile GIT-BUILD-OPTIONS: FORCE · line 2975
Quoted 7 lines
Makefile +1
-0@@ -2970,6 +2969,7 @@ GIT-BUILD-OPTIONS: FORCE @echo NO_PERL=\''$(subst ','\'',$(subst ','\'',$(NO_PERL)))'\' >>$@+ @echo NO_PTHREADS=\''$(subst ','\'',$(subst ','\'',$(NO_PTHREADS)))'\' >>$@+ @echo NO_PYTHON=\''$(subst ','\'',$(subst ','\'',$(NO_PYTHON)))'\' >>$@+ + @echo NO_REGEX=\''$(subst ','\'',$(subst ','\'',$(NO_REGEX)))'\' >>$@+ @echo NO_UNIX_SOCKETS=\''$(subst ','\'',$(subst ','\'',$(NO_UNIX_SOCKETS)))'\' >>$@+ @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+
Build part looks good to me.
Quoted 37 lines
common-main.c +1
-0@@ -40,6 +40,7 @@ int main(int argc, const char **argv)git_resolve_executable_dir (argv[0 ]); +setlocale (LC_CTYPE ,"" );git_setup_gettext ();initialize_the_repository ();gettext.c +0
-2@@ -10,7 +10,6 @@#include "config.h" #ifndef NO_GETTEXT -# include <locale.h> # include <libintl.h> # ifdef GIT_WINDOWS_NATIVE @@ -80,7 +79,6 @@ static int test_vsnprintf(const char *fmt, ...)static void init_gettext_charset (const char *domain) { -setlocale (LC_CTYPE ,"" ); charset =locale_charset ();bind_textdomain_codeset (domain, charset);git-compat-util.h +0
-0
git-compat-util.h line 218
Quoted 7 lines
git-compat-util.h +1
-0@@ -212,6 +212,7 @@#endif #include <errno.h> #include <limits.h> +#include <locale.h> #ifdef NEEDS_SYS_PARAM_H #include <sys/param.h> #endif
I'll let others more familiar with the locale support to comment on these changes. We are unconditionally including <locale.h> now; before platforms that lack locale.h can set NO_GETTEXT but that will no longer work as a "workaround" for them. I do not know if thta is a practical downside to anybody, but it could be a problem.
t/t7818-grep-multibyte.sh
Quoted 5 lines
t/t7818-grep-multibyte.sh new file +0
-0
Do we need a new test script for this?
t/t7818-grep-multibyte.sh line 9
Quoted 10 lines
t/t7818-grep-multibyte.sh new file +9
-0@@ -0,0 +1,34 @@ +#!/bin/sh + +test_description='grep multibyte characters' + +. ./test-lib.sh + +# Multibyte regex search is only supported with a native regex library +# that supports it. +# (The supplied compatibility library is compiled with NO_MBSUPPORT.)
This file is not specific to Darwin; "... with a native regex library" etc. is not something we want to see here.
t/t7818-grep-multibyte.sh line 12
Quoted 3 lines
t/t7818-grep-multibyte.sh new file +3
-0+test -z "$NO_REGEX" && + LC_ALL=en_US.UTF-8 test-tool regex '^.$' '¿' && + test_set_prereq MB_REGEX
We can safely drop 'test -z "$NO_REGEX" &&' part here, no? Even if we omit it, those who built with NO_REGEX would fail "test-tool regex" step above. And by omitting $NO_REGEX check, we do not have to look for and update the condition when the fallback regex engine we use starts supporting MB_REGEX.
t/t7818-grep-multibyte.sh line 18
Quoted 5 lines
t/t7818-grep-multibyte.sh new file +5
-0+if ! test_have_prereq MB_REGEX +then + skip_all='multibyte grep tests; Git compiled with NO_REGEX, NO_MBSUPPORT' + test_done +fi
I do not think if we need a single use prereq here. We can just use whatever condition that is used to set MB_REGEX above and do the skip-all thing here.
t/t7818-grep-multibyte.sh line 25
Quoted 6 lines
t/t7818-grep-multibyte.sh new file +6
-0+test_expect_success 'setup' ' + test_write_lines "¿" >file && + git add file && + LC_ALL="en_US.UTF-8" && + export LC_ALL +'
Missing inter-test blank line.
t/t7818-grep-multibyte.sh line 28
Quoted 3 lines
t/t7818-grep-multibyte.sh new file +3
-0+test_expect_success 'grep exactly one char in single-char multibyte file' ' + git grep "^.$" +'
I am not sure how much value we are getting out of this test, which is identical to what we already tested earlier above with "test-tool regex".
Quoted 5 lines
t/t7818-grep-multibyte.sh new file +5
-0+test_expect_success 'grep two chars in single-char multibyte file' ' + test_expect_code 1 git grep ".." +' + +test_done
PATCH v4 1/1 grep: fix multibyte regex handling under macOS
In reply to a message not in the archiveThe commit 29de20504e (Makefile: fix default regex settings on Darwin, 2013-05-11) fixed t0070-fundamental.sh under Darwin (macOS) by adopting Git's regex library. However, this library is compiled with NO_MBSUPPORT, which causes git-grep to work incorrectly on multibyte (e.g. UTF-8) files. Current macOS versions pass t0070-fundamental.sh with the native macOS regex library, which also supports multibyte characters.
Adjust the Makefile to use the native regex library, and call setlocale(3) to set CTYPE according to the user's preference. The setlocale call is required on all platforms, but in platforms supporting gettext(3), setlocale was called as a side-effect of initializing gettext. Therefore, move the CTYPE setlocale call from gettext.c to common-main.c and the corresponding locale.h include into git-compat-util.h.
Thanks to the global initialization of CTYPE setlocale, the test-tool regex command now works correctly with supported multibyte regexes, and is used to set the MB_REGEX test prerequisite by assessing a platform's support for them.
---
- Fold test script into t7810-grep.sh
- Simplify the test prerequisite evaluation
Makefile | 2 +- common-main.c | 1 + gettext.c | 2 -- git-compat-util.h | 1 + t/t7810-grep.sh | 15 +++++++++++++++ 5 files changed, 18 insertions(+), 3 deletions(-)
Makefile +1 -1
@@ -1427,7 +1427,6 @@ ifeq ($(uname_S),Darwin) APPLE_COMMON_CRYPTO = YesPlease COMPAT_CFLAGS += -DAPPLE_COMMON_CRYPTO endif - NO_REGEX = YesPlease PTHREAD_LIBS = endif
@@ -2970,6 +2969,7 @@ GIT-BUILD-OPTIONS: FORCE @echo NO_PERL=\''$(subst ','\'',$(subst ','\'',$(NO_PERL)))'\' >>$@+ @echo NO_PTHREADS=\''$(subst ','\'',$(subst ','\'',$(NO_PTHREADS)))'\' >>$@+ @echo NO_PYTHON=\''$(subst ','\'',$(subst ','\'',$(NO_PYTHON)))'\' >>$@+ + @echo NO_REGEX=\''$(subst ','\'',$(subst ','\'',$(NO_REGEX)))'\' >>$@+ @echo NO_UNIX_SOCKETS=\''$(subst ','\'',$(subst ','\'',$(NO_UNIX_SOCKETS)))'\' >>$@+ @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+
common-main.c +1 -0
@@ -40,6 +40,7 @@ int main(int argc, const char **argv)git_resolve_executable_dir (argv[0 ]); +setlocale (LC_CTYPE ,"" );git_setup_gettext ();initialize_the_repository ();
gettext.c +0 -2
@@ -10,7 +10,6 @@#include "config.h" #ifndef NO_GETTEXT -# include <locale.h> # include <libintl.h> # ifdef GIT_WINDOWS_NATIVE
@@ -80,7 +79,6 @@ static int test_vsnprintf(const char *fmt, ...)static void init_gettext_charset (const char *domain) { -setlocale (LC_CTYPE ,"" ); charset =locale_charset ();bind_textdomain_codeset (domain, charset);
git-compat-util.h +1 -0
@@ -212,6 +212,7 @@#endif #include <errno.h> #include <limits.h> +#include <locale.h> #ifdef NEEDS_SYS_PARAM_H #include <sys/param.h> #endif
t/t7810-grep.sh +15 -0
@@ -18,6 +18,9 @@ test_invalid_grep_expression() { ' } +LC_ALL=en_US.UTF-8 test-tool regex '^.$' '¿' && + test_set_prereq MB_REGEX + cat >hello.c <<EOF #include <assert.h> #include <stdio.h>
@@ -87,6 +90,10 @@ test_expect_success setup ' echo unusual >"\"unusual\" pathname" && echo unusual >"t/nested \"unusual\" pathname" fi && + if test_have_prereq MB_REGEX + then + echo "¿" >reverse-question-mark + fi && git add . && test_tick && git commit -m initial
@@ -568,6 +575,14 @@ do ' done +test_expect_success MB_REGEX 'grep exactly one char in single-char multibyte file' ' + LC_ALL=en_US.UTF-8 git grep "^.$" reverse-question-mark +' + +test_expect_success MB_REGEX 'grep two chars in single-char multibyte file' ' + LC_ALL=en_US.UTF-8 test_expect_code 1 git grep ".." reverse-question-mark +' + cat >expected <<EOF file EOF
Signature · 1 line
2.30.2
Quoted 2 lines · Diomidis Spinellis
- Fold test script into
t7810-grep.sh
- Simplify the test prerequisite evaluation
The tests certainly look better, and as v3, the build procedure sounds OK.
Quoted 18 lines
common-main.c +1
-0@@ -40,6 +40,7 @@ int main(int argc, const char **argv)git_resolve_executable_dir (argv[0 ]); +setlocale (LC_CTYPE ,"" );git_setup_gettext ();initialize_the_repository ();git-compat-util.h +0
-0
git-compat-util.h line 218
Quoted 7 lines
git-compat-util.h +1
-0@@ -212,6 +212,7 @@#endif #include <errno.h> #include <limits.h> +#include <locale.h> #ifdef NEEDS_SYS_PARAM_H #include <sys/param.h> #endif
I'll let others more familiar with the locale support to comment on these changes. We are unconditionally including <locale.h> now; platforms that lack <locale.h> could set NO_GETTEXT to work it around before this change, but that will no longer work.
I do not know if thta is a practical downside to anybody, but it could be a problem.
Perhaps cook this in 'next' and see if anybody screams?
Thanks.
Quoted 10 lines · Junio C Hamano
Quoted 4 lines
+#include <
locale.h>
#ifdefNEEDS_SYS_PARAM_H
#include <sys/param.h>
#endifI'll let others more familiar with the locale support to comment on these changes. We are unconditionally including <
locale.h> now; platforms that lack <locale.h> could setNO_GETTEXTto work it around before this change, but that will no longer work.
Quoted 4 lines
I do not know if thta is a practical downside to anybody, but it could be a problem.
Perhaps cook this in 'next' and see if anybody screams?
I was programming in C before locale.h was introduced, so its availability was something that worried me as well. I looked up whether we could/should use a GNU autoconf macro to test for the locale.h availability. According to autoconf's documentation "All hosted environments that are still of interest for portable code provide all of the headers specified in ISO C90 (as amended in 1995): assert.h, [...] locale.h [...], and wctype.h. Most programs can safely include these headers unconditionally." [1]
[1] https://www.gnu.org/software/autoconf/manual/autoconf-2.70/autoconf.html
Quoted 1 line · Diomidis Spinellis
Quoted 1 line
Perhaps cook this in 'next' and see if anybody screams?
Let's do this.
Quoted 8 lines
I was programming in C before
locale.hwas introduced, so its availability was something that worried me as well. I looked up whether we could/should use a GNU autoconf macro to test for thelocale.havailability. According to autoconf's documentation "All hosted environments that are still of interest for portable code provide all of the headers specified in ISO C90 (as amended in 1995):assert.h, [...]locale.h[...], andwctype.h. Most programs can safely include these headers unconditionally." [1]
Yeah, but there are folks on minor platforms where full POSIX compliance is iffy, so whatever autoconf's documentation says unfortunately does not weigh as much as we wish it does.