Localise error headers

Currently, the headers "error: ", "warning: " etc. - generated by die(), warning() etc. - are not localized, but we feed many localized error messages into these functions so that we produce error messages with mixed localisation.

This series introduces variants of die() etc. that use localised variants of the headers, i.e. _("error: ") etc., and are to be fed localized messages. So, instead of die(_("not workee")), which would produce a mixed localisation (such as "error: geht ned"), one should use die_(_("not workee")) (resulting in "Fehler: geht ned").

In this implementation, the gettext call for the header and the body are done in different places (error function vs. caller) but this call pattern seems to be the easiest variant for the caller, because the message body has to be marked for localisation in any case, and N_() requires more letters than _(), an extra argument to die() etc. even more than the extra "_" in the function name.

1/5 prepares the error machinery
2/5 provides new variants error_() etc.
3/5 has coccinelli rules error(_(E)) -> error_(_(E)) etc. 4/5 applies the coccinelli patches

5/5 is not to be applied to the main tree, but helps you try out the feature:
it has changes to de.po and git.pot so that e.g. "git branch" has fully localised error messages (see the recipe in the commit message).

Michael J Gruber (5):
error/warn framework: prepare for l10n error/warn framework: provide localized variants error/warn framework framework: coccinelli rules error/warn framework: localize
WIP: Feature demonstration

 advice.c                           |   16 +-
 apply.c                            |   10 +-
 apply.h                            |    4 +-
 archive.c                          |    6 +-
 attr.c                             |    3 +-
 bisect.c                           |    2 +-
 branch.c                           |    4 +-
 builtin/add.c                      |   20 +-
 builtin/am.c                       |   27 +-
 builtin/archive.c                  |   10 +-
 builtin/blame.c                    |   12 +-
 builtin/branch.c                   |   45 +-
 builtin/bundle.c                   |    4 +-
 builtin/check-ignore.c             |   14 +-
 builtin/check-mailmap.c            |    2 +-
 builtin/checkout-index.c           |    2 +-
 builtin/checkout.c                 |   27 +-
 builtin/clean.c                    |   10 +-
 builtin/clone.c                    |   39 +-
 builtin/column.c                   |    2 +-
 builtin/commit.c                   |   87 +-
 builtin/config.c                   |    2 +-
 builtin/describe.c                 |    6 +-
 builtin/diff.c                     |    2 +-
 builtin/fetch.c                    |   21 +-
 builtin/gc.c                       |    3 +-
 builtin/grep.c                     |   14 +-
 builtin/help.c                     |    4 +-
 builtin/index-pack.c               |   42 +-
 builtin/interpret-trailers.c       |    2 +-
 builtin/log.c                      |   48 +-
 builtin/merge-recursive.c          |    2 +-
 builtin/merge.c                    |   53 +-
 builtin/mv.c                       |    6 +-
 builtin/notes.c                    |   40 +-
 builtin/pack-objects.c             |    4 +-
 builtin/prune.c                    |    2 +-
 builtin/pull.c                     |   10 +-
 builtin/push.c                     |   22 +-
 builtin/remote.c                   |   10 +-
 builtin/repack.c                   |    4 +-
 builtin/replace.c                  |    2 +-
 builtin/reset.c                    |   10 +-
 builtin/rev-list.c                 |    2 +-
 builtin/rev-parse.c                |    2 +-
 builtin/revert.c                   |    4 +-
 builtin/rm.c                       |    6 +-
 builtin/shortlog.c                 |    2 +-
 builtin/show-branch.c              |    7 +-
 builtin/submodule--helper.c        |    9 +-
 builtin/tag.c                      |   20 +-
 builtin/unpack-objects.c           |    2 +-
 builtin/update-index.c             |    8 +-
 builtin/worktree.c                 |    6 +-
 bundle.c                           |    4 +-
 config.c                           |    4 +-
 connect.c                          |    6 +-
 connected.c                        |    2 +-
 contrib/coccinelle/errorl10n.cocci |   47 +
 daemon.c                           |    3 +-
 diff.c                             |    8 +-
 dir.c                              |    4 +-
 fast-import.c                      |    4 +-
 fetch-pack.c                       |   26 +-
 git-compat-util.h                  |   18 +-
 help.c                             |    2 +-
 http-backend.c                     |    4 +-
 http.c                             |    4 +-
 merge.c                            |    2 +-
 notes-utils.c                      |    2 +-
 pathspec.c                         |   13 +-
 po/de.po                           | 2922 ++++++++++++++++++++++++------------
 po/git.pot                         | 2781 ++++++++++++++++++++++------------
 pretty.c                           |    2 +-
 ref-filter.c                       |   20 +-
 remote.c                           |    2 +-
 revision.c                         |    4 +-
 run-command.c                      |    6 +-
 send-pack.c                        |   12 +-
 sequencer.c                        |    7 +-
 setup.c                            |    6 +-
 sha1_file.c                        |    2 +-
 submodule.c                        |    8 +-
 trailer.c                          |    4 +-
 transport.c                        |    2 +-
 tree-walk.c                        |    2 +-
 usage.c                            |  122 +-
 wrapper.c                          |    2 +-
 wt-status.c                        |    2 +-
 89 files changed, 4408 insertions(+), 2374 deletions(-)
 create mode 100644 contrib/coccinelle/errorl10n.cocci

2.11.0.372.g2fcea0e476

Currently, errors, warnings etc. are output with a fixed prefix "error: " etc. that is not subject to l10n.

Change the call signatures of error_routine() etc. so that they receive the prefix as first argument.

---
 apply.c           |  2 +-
 apply.h           |  4 ++--
 daemon.c          |  3 ++-
 fast-import.c     |  4 ++--
 git-compat-util.h | 10 +++++-----
 http-backend.c    |  4 ++--
 run-command.c     |  4 ++--
 usage.c           | 48 ++++++++++++++++++++++++------------------------
 8 files changed, 40 insertions(+), 39 deletions(-)

index 2ed808d429..0f93792e1c 100644
@@ -112,7 +112,7 @@ void clear_apply_state(struct apply_state *state)
 	/* &state->fn_table is cleared at the end of apply_patch() */
 }
 
-static void mute_routine(const char *msg, va_list params)
+static void mute_routine(const char *prefix, const char *msg, va_list params)
 {
 	/* do nothing */
 }

index b3d6783d55..56b5622868 100644
@@ -100,8 +100,8 @@ struct apply_state {
 	 * set_error_routine() or set_warn_routine() to install muting
 	 * routines when in verbosity_silent mode.
 	 */
-	void (*saved_error_routine)(const char *err, va_list params);
-	void (*saved_warn_routine)(const char *warn, va_list params);
+	void (*saved_error_routine)(const char *prefix, const char *err, va_list params);
+	void (*saved_warn_routine)(const char *prefix, const char *warn, va_list params);
 
 	/* These control whitespace errors */
 	enum apply_ws_error_action ws_error_action;

index 473e6b6b63..cd52a25001 100644
@@ -114,8 +114,9 @@ static void loginfo(const char *err, ...)
 	va_end(params);
 }
 
-static void NORETURN daemon_die(const char *err, va_list params)
+static void NORETURN daemon_die(const char *prefix, const char *err, va_list params)
 {
+	/* no need to pass down prefix */
 	logreport(LOG_ERR, err, params);
 	exit(1);
 }

index f561ba833b..4576f12163 100644
@@ -496,13 +496,13 @@ static void end_packfile(void);
 static void unkeep_all_packs(void);
 static void dump_marks(void);
 
-static NORETURN void die_nicely(const char *err, va_list params)
+static NORETURN void die_nicely(const char *prefix, const char *err, va_list params)
 {
 	static int zombie;
 	char message[2 * PATH_MAX];
 
 	vsnprintf(message, sizeof(message), err, params);
-	fputs("fatal: ", stderr);
+	fputs(prefix, stderr);
 	fputs(message, stderr);
 	fputc('\n', stderr);
 

index 87237b092b..f1bf0a6749 100644
@@ -439,11 +439,11 @@ static inline int const_error(void)
 #define error_errno(...) (error_errno(__VA_ARGS__), const_error())
 #endif
 
-extern void set_die_routine(NORETURN_PTR void (*routine)(const char *err, va_list params));
-extern void set_error_routine(void (*routine)(const char *err, va_list params));
-extern void (*get_error_routine(void))(const char *err, va_list params);
-extern void set_warn_routine(void (*routine)(const char *warn, va_list params));
-extern void (*get_warn_routine(void))(const char *warn, va_list params);
+extern void set_die_routine(NORETURN_PTR void (*routine)(const char *prefix, const char *err, va_list params));
+extern void set_error_routine(void (*routine)(const char *prefix, const char *err, va_list params));
+extern void (*get_error_routine(void))(const char *prefix, const char *err, va_list params);
+extern void set_warn_routine(void (*routine)(const char *prefix, const char *warn, va_list params));
+extern void (*get_warn_routine(void))(const char *prefix, const char *warn, va_list params);
 extern void set_die_is_recursing_routine(int (*routine)(void));
 extern void set_error_handle(FILE *);
 

index eef0a361f4..5c235e8b52 100644
@@ -577,12 +577,12 @@ static void service_rpc(struct strbuf *hdr, char *service_name)
 }
 
 static int dead;
-static NORETURN void die_webcgi(const char *err, va_list params)
+static NORETURN void die_webcgi(const char *prefix, const char *err, va_list params)
 {
 	if (dead <= 1) {
 		struct strbuf hdr = STRBUF_INIT;
 
-		vreportf("fatal: ", err, params);
+		vreportf(prefix, err, params);
 
 		http_status(&hdr, 500, "Internal Server Error");
 		hdr_nocache(&hdr);

index ca905a9e80..3133bf5320 100644
@@ -618,9 +618,9 @@ static void *run_thread(void *data)
 	return (void *)ret;
 }
 
-static NORETURN void die_async(const char *err, va_list params)
+static NORETURN void die_async(const char *prefix, const char *err, va_list params)
 {
-	vreportf("fatal: ", err, params);
+	vreportf(prefix, err, params);
 
 	if (in_async()) {
 		struct async *async = pthread_getspecific(async_key);

2.11.0.372.g2fcea0e476

Provide localized variants of error(), warning(), die() etc. to go along with localized messages.

---
 git-compat-util.h |  8 ++++++
 usage.c           | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+)

index f1bf0a6749..48209a6c67 100644
@@ -405,13 +405,21 @@ struct strbuf;
 /* General helper functions */
 extern void vreportf(const char *prefix, const char *err, va_list params);
 extern NORETURN void usage(const char *err);
+extern NORETURN void usage_(const char *err);
 extern NORETURN void usagef(const char *err, ...) __attribute__((format (printf, 1, 2)));
+extern NORETURN void usagef_(const char *err, ...) __attribute__((format (printf, 1, 2)));
 extern NORETURN void die(const char *err, ...) __attribute__((format (printf, 1, 2)));
+extern NORETURN void die_(const char *err, ...) __attribute__((format (printf, 1, 2)));
 extern NORETURN void die_errno(const char *err, ...) __attribute__((format (printf, 1, 2)));
+extern NORETURN void die_errno_(const char *err, ...) __attribute__((format (printf, 1, 2)));
 extern int error(const char *err, ...) __attribute__((format (printf, 1, 2)));
+extern int error_(const char *err, ...) __attribute__((format (printf, 1, 2)));
 extern int error_errno(const char *err, ...) __attribute__((format (printf, 1, 2)));
+extern int error_errno_(const char *err, ...) __attribute__((format (printf, 1, 2)));
 extern void warning(const char *err, ...) __attribute__((format (printf, 1, 2)));
+extern void warning_(const char *err, ...) __attribute__((format (printf, 1, 2)));
 extern void warning_errno(const char *err, ...) __attribute__((format (printf, 1, 2)));
+extern void warning_errno_(const char *err, ...) __attribute__((format (printf, 1, 2)));
 
 #ifndef NO_OPENSSL
 #ifdef APPLE_COMMON_CRYPTO

2.11.0.372.g2fcea0e476

Provide coccinelli rules which check for error(), warning() etc. with localised argument and create a patch to replace them with error_(), warning_() etc. in order to fully localize them.

---
 contrib/coccinelle/errorl10n.cocci | 47 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100644 contrib/coccinelle/errorl10n.cocci

new file mode 100644
index 0000000000..d62a440644
@@ -0,0 +1,47 @@
+@@
+expression E;
+@@
+- usage(_(E));
++ usage_(_(E));
+
+@@
+expression E;
+@@
+- usagef(_(E));
++ usagef_(_(E));
+
+@@
+expression E;
+@@
+- die(_(E));
++ die_(_(E));
+
+@@
+expression E;
+@@
+- error(_(E));
++ error_(_(E));
+
+@@
+expression E;
+@@
+- warning(_(E));
++ warning_(_(E));
+
+@@
+expression E;
+@@
+- die_errno(_(E));
++ die_errno_(_(E));
+
+@@
+expression E;
+@@
+- error_errno(_(E));
++ error_errno_(_(E));
+
+@@
+expression E;
+@@
+- warning_errno(_(E));
++ warning_errno_(_(E));

2.11.0.372.g2fcea0e476

Another option, not as thorough, but less effort, is changing die/err/warn default routines to the "porcelain" versions where we do _("fatal:") internally _with_ die(), not die_(). We can set this for porcelain commands that we know can be fully i18n-ized. Then maybe die_() will fill in the gap if there's still need for it.

Duy

Have you considered

die_("non localized error here")

to produce:

"Fehler: trotzdem uebersetzt hier"
("error: here it is translated")

I see. We have to markup the strings to be translatable such that the .po files are complete. It would be really handy if there was a way to say "anything that is fed to this function (die_) needs to be marked for translation.

Looking through
https://www.gnu.org/software/gettext/manual/html_node/xgettext-Invocation.html
such a thing doesn't seem to exist.

So in that case die_(_(...)) seems to be the easiest way forward.

Thanks,
Stefan

I think --keyword is exactly for that purpose: marking more text for translations besides standard markers like _() or N_(). Yes we need to call gettext() explicitly in die_() later on. We already do that for parse-options. We just need to N_() the strings, without actually spelling it out.

I still prefer changing the die_routine though since die() in many cases could be used in both plumbing and porcelain contexts. And we have tried to keep plumbing output (and behavior) as stable as possible. The approach has some similarity to unpack_trees() which shares the same porcelain/plumbing problem.

Duy

On the other hand, making die(), not die_(), translatable means the translators will have to translate them _all_ even if only some will end up being displayed. That's 2000+ strings according to git-grep. And some of them, like die("BUG:..") should definitely not be translated. So +1 to die_(), unless we decide all strings are safe to translate.

Duy

I can't say I'm excited about having matching "_" variants for each function. Are we sure that they are necessary? I.e., would it be acceptable to just translate them always?

Your patches 4 and 5 don't seem to have made it to the list. Judging from the diffstat, I'd guess they broke the 100K limit.

-Peff

Jeff King venit, vidit, dixit 04.01.2017 08:05:

We would still need to mark the strings, e.g.

die(N_("oopsie"));

and would not be able to opt out of translating in the code (only in the po file, by not providing a translation).

Hmmpf, I didn't know about the limit. In any case, they were simple results of applying the "make cocci" patches (4/5) resp. providing some "de" strings to try this out.

In any case, the question is whether we want to tell the user

A: B

where A is in English and B is localised, or rather localise both A and B (for A in "error", "fatal", "warning"...).

For localising A and B, we'd need this series or something similar. For keeping the mix, we don't need to do anything ;)

Michael

I meant more along the lines of: would it be OK to just always translate the prefix, even if the message itself is not translated? I.e.,

index 82ff13163..8e5400f57 100644
@@ -32,7 +32,7 @@ static NORETURN void usage_builtin(const char *err, va_list params)
 
 static NORETURN void die_builtin(const char *err, va_list params)
 {
-	vreportf("fatal: ", err, params);
+	vreportf(_("fatal: "), err, params);
 	exit(128);
 }
 

What I wrote above would keep the mix, but switch it in the other direction.

And then presumably that mix would gradually move to 100% consistency as more messages are translated. But the implicit question is: are there die() messages that should never be translated? I'm not sure.

-Peff

I would assume any plumbing command is not localizing? Because in plumbing land, (easily scriptable) you may find a grep on the output/stderr for a certain condition?

To find a good example, "git grep die" giving me some food of though:

die_errno(..) should always take a string marked up for translation, because the errno string is translated?
(-> we'd have to fix up any occurrence of git grep "die_errno(\"")

    apply.c:                die(_("internal error"));

That is funny, too. I think we should substitute that with

die("BUG: untranslated, but what went wrong instead")

That's the assumption I'm challenging. Certainly the behavior and certain aspects of the output of a plumbing command should remain the same over time. But error messages to stderr?

It seems like they should be translated, because plumbing invoked on behalf of porcelain scripts is going to send its stderr directly to the user.

Yes, I would think die_errno() is a no-brainer for translation, since the strerror() will be translated.

Yep. We did not consistently use "BUG:" in the early days. I would say that "BUG" lines do not need to be translated. The point is that nobody should ever see them, so it seems like there is little point in giving extra work to translators.

-Peff

In an ideal world that assumption would hold true and any machine readable information from the plumbing commands are on stdout and nobody in their sane mind would ever try to parse stderr.

There is no easy way to check though except auditing all the code. Having pointed out some string handling mistakes in the prior email, my confidence is not high that plumbing commands are that strict about separating useful machine output and errors for human consumption.

Well that could be solved, c.f. unpack-trees.c lines 15-55.

As another data point (coming from that area of strings):
If you grep for these plumbing strings in the project, i.e.

$ git grep "would be overwritten by merge. Cannot merge"
$ git grep "not uptodate. Cannot merge."
...

you only find the occurrence in the unpack-trees.c lines 15-55, which means our test suite at least doesn't grep for error messages but relies on the exit code of plumbing commands(?!)

I think it's impossible to audit all the code, because "all the code" includes things not in git itself. It would really take a willingness to declare "if you are parsing stderr you're doing it wrong".

I suppose the unpack-trees example speaks against that. I'm not sure that is sane overall, but it is something we tried to account for in the past.

If we are just talking about translation, though, it seems like the right action for a script that really wants to parse stderr is to run the sub-command with LC_MESSAGES=C. I know that's a breaking change, but I wonder if it's a reasonable path forward.

-Peff

In addition, "BUG: " is relatively recent introduction to our codebase. Perhaps having a separate BUG(<string>) function help the distinction further?

I was going to write the same thing. On top of that I wonder if have enough "if (something) die("BUG:")" to justify stealing BUG_ON() from kernel (better than assert since the condition will always be evaluated).

Duy

I don't mind BUG_ON() as an improvement over assert(). However, one of the things I really like about our current die("BUG") is that you actually write a human-readable message. Quite often that serves as a comment to the reader of the code as to what is going on, or you can give additional context in the error message (e.g., which sha1 triggered the bug).

Still, there are many cases where there's not much to say in the message. Doing:

if (!foo)

	BUG("foo is NULL in this function");

is just wasting everybody's time. Something like:

  #define BUG_ON(code) do {
	if (code)
		BUG("%s", #code);

while(0)

is probably fine. One complication is that BUG() would ideally show the file/line number. That has to happen in a macro, but we don't assume variadic macros work everywhere, which we need for the printf-like behavior. So either we have to loosen that restriction[1], or we end up with two "tiers": a BUG(fmt, ...) for human-readable messages, and a more boring version for BUG_ON() which just prints "file:line: BUG:
<code>".

-Peff

[1] I think avoiding variadic macros was reasonable in 2005; C99 was only 6 years old then. Now it's turning 18. Maybe it's time?

Yes, I think so. I have often been tempted to dump core on BUGs for further analysis. You can do that by string-matching "BUG:" from the beginning of a die message, but it's kind of gross. :)

-Peff

Jeff King venit, vidit, dixit 21.01.2017 15:20:

I read back the whole thread, and I'm still not sure if there's consensus and how to go forward. Should we let the topic die? I don't care too much personally, I just thought the mixed tranlations look "unprofessional".

Michael

I don't have a strong preference either way. I also don't care personally about the output (as I do not localize at all). My main concern was keeping the code simple for developers. But if consistent translation is important for people in other languages, I'm OK with whatever we need to do.

-Peff

(0) I do not mind the status quo, and I do not mind telling an end user who comes here with a translated message from die() to try running it again in C locale, either.

(1) I do not think messages (including prefix) from die(), warning(), and error() are sacred, even when they are given by plumbing commands. If the list concensus is that we want to see all translated in the ideal endgame, I think it is OK not to special case the plumbing.

(2) I found your

vreportf(_("fatal: "), err, params);

a reasonable approach, if the we all agree with (1) as our goal, and want a way to gradually get us closer to the "everything translated" endgame.

I do not know what is professional and what is not in this context, though ;-).

Michael, thanks for pinging the thread.

I agree. And the main (*) changes are relative simple too. I've tested it a bit. Seems to work ok.

-- 8< --

index d861bd9985..6f88c6cac5 100644
@@ -2102,7 +2102,7 @@ XGETTEXT_FLAGS = \
 	--msgid-bugs-address="Git Mailing List <[email protected]>" \
 	--from-code=UTF-8
 XGETTEXT_FLAGS_C = $(XGETTEXT_FLAGS) --language=C \
-	--keyword=_ --keyword=N_ --keyword="Q_:1,2"
+	--keyword=_ --keyword=N_ --keyword="Q_:1,2" --keyword=die_errno
 XGETTEXT_FLAGS_SH = $(XGETTEXT_FLAGS) --language=Shell \
 	--keyword=gettextln --keyword=eval_gettextln
 XGETTEXT_FLAGS_PERL = $(XGETTEXT_FLAGS) --language=Perl \

index 17f52c1b5c..c022726f9c 100644
@@ -159,7 +159,7 @@ void NORETURN die_errno(const char *fmt, ...)
 	}
 
 	va_start(params, fmt);
-	die_routine(fmt_with_err(buf, sizeof(buf), fmt), params);
+	die_routine(fmt_with_err(buf, sizeof(buf), _(fmt)), params);
 	va_end(params);
 }
 

-- 8< --

(*) We would need another patch to remove _() from die_errno(_(..)). But that's something I expect coccinelle to be excel at. --
Duy

Oops. Didn't notice that the "fatal:" bit is still untranslated but Micheal's patch can handle that.

Duy