Skip to contentSkip to conversation

git-filter-branch : LANG / LC_ALL = C breaks UTF-8 author names

DISCUSSION Richard MICHAEL 2 participants 4 messages Jun 15, 2016 – Aug 11, 2016

lore
Share & tools
lore
#0 lore raw

Hello all,

I am filtering our repo with git-filter-branch, but as the sed script runs with LANG=C LC_ALL=C (7 bit US ASCII), it dies on commits authored by our team members with accented names. Why are the locales "C"? For compatibility with older sed? I've changed to LANG=en_US.UTF-8, will my change will cause other git-breakage?

git-filter-branch

95: LANG=C LC_ALL=C sed -ne "$pick_id_script"

95: LANG=en_US.UTF-8 sed -ne "$pick_id_script"

Regards,
Richard

#1 lore raw
Quoted 3 lines · Richard MICHAEL

I am filtering our repo with git-filter-branch, but as the sed script runs with LANG=C LC_ALL=C (7 bit US ASCII), it dies on commits authored by our team members with accented names.

Yep, someone else recently sent a report about such a sed version, too. It is breaking our fragile minds; we ought to find some way to deal with it, but we haven't yet.

Jonathan

#2 lore raw
Quoted 10 lines · Jonathan Nieder
Quoted 3 lines · Richard MICHAEL

I am filtering our repo with git-filter-branch, but as the sed script runs with LANG=C LC_ALL=C (7 bit US ASCII), it dies on commits authored by our team members with accented names.

Yep, someone else recently sent a report about such a sed version, too. It is breaking our fragile minds; we ought to find some way to deal with it, but we haven't yet.

Jonathan

Jonathan, thanks for your reply.

What about special casing the bad sed (or whitelisting good sed)? Surely a hack, but would those of us with GNU or BSD would be happy. Which was the troublesome sed?

That opposed to figuring out the problem, reading about unicode, and re-cloning and re-filtering 5,000 commits. :-) Unfortunately, it doesn't "die" well either; the 'export' shell var fails but it keeps processing commits. (If I hadn't investigated and changed the LANG, would I have lost those commits?)

Regards,
Richard

#3 lore raw

Hi Richard,

Quoted 5 lines · Richard MICHAEL
Quoted 5 lines
Quoted 3 lines · Richard MICHAEL

I am filtering our repo with git-filter-branch, but as the sed script runs with LANG=C LC_ALL=C (7 bit US ASCII), it dies on commits authored by our team members with accented names.

[...]

Quoted 3 lines

What about special casing the bad sed (or whitelisting good sed)? Surely a hack, but would those of us with GNU or BSD would be happy. Which was the troublesome sed?

Sorry for the slow response. The problematic sed is GNU sed from MacPorts (I think). Even with LC_ALL=C, .* no longer matches arbitrary sequences of bytes with such sed: you can check yours with

 $ echo 'étale' | LC_ALL=C sed 's/.*//'

Unfortunately I have not been able to reproduce it on Linux. Debian sed 4.2.1-7 and GNU sed v4.2.1-21-gc6d32f0 both produce the expected result:

 $ echo 'étale' | LC_ALL=C sed 's/.*//'
 $
Quoted 3 lines

Unfortunately, it
doesn't "die" well either; the 'export' shell var fails but it keeps processing commits.

Hmm, that sounds like a bug indeed. Here is what the start to a fix might look like, but I stopped early because it there's quite a lot of sed usage in git that expects to be able to process arbitrary data with short, newline-terminated lines (regardless of encoding).

git-filter-branch.sh +8 -6
index 962a93b..34a5fa3 100755
@@ -68,8 +68,8 @@ eval "$functions"
 # "author" or "committer
 
 set_ident () {
-	lid="$(echo "$1" | tr "[A-Z]" "[a-z]")"
-	uid="$(echo "$1" | tr "[a-z]" "[A-Z]")"
+	lid="$(echo "$1" | tr "[A-Z]" "[a-z]")" &&
+	uid="$(echo "$1" | tr "[a-z]" "[A-Z]")" &&
 	pick_id_script='
 		/^'$lid' /{
 			s/'\''/'\''\\'\'\''/g
@@ -90,9 +90,9 @@ set_ident () {
 
 			q
 		}
-	'
+	' &&
 
-	LANG=C LC_ALL=C sed -ne "$pick_id_script"
+	LANG=C LC_ALL=C sed -ne "$pick_id_script" &&
 	# Ensure non-empty id name.
 	echo "case \"\$GIT_${uid}_NAME\" in \"\") GIT_${uid}_NAME=\"\${GIT_${uid}_EMAIL%%@*}\" && export GIT_${uid}_NAME;; esac"
 }
@@ -322,9 +322,11 @@ while read commit parents; do
 	git cat-file commit "$commit" >../commit ||
 		die "Cannot read commit $commit"
 
-	eval "$(set_ident AUTHOR <../commit)" ||
+	set_author=$(set_ident AUTHOR <../commit) &&
+	eval "$set_author" ||
 		die "setting author failed for commit $commit"
-	eval "$(set_ident COMMITTER <../commit)" ||
+	set_committer=$(set_ident COMMITTER <../commit) &&
+	eval "$set_committer" ||
 		die "setting committer failed for commit $commit"
 	eval "$filter_env" < /dev/null ||
 		die "env filter failed: $filter_env"