diff: improve scaling of filenames in diffstat to handle UTF-8 chars
PATCH 1/1 diff: improve scaling of filenames in diffstat to handle UTF-8 chars
The show_stats() function tries to scale the filenames in the diffstat to ensure they don't exceed the given name-width. It does so by calculating the "display width" of the characters to be dropped, but then advances the filename pointer by that number of bytes.
However, the "display width" of a character is not always equal to its byte count. The result is that sometimes, when displaying UTF-8 characters, filenames exceed the given name-width, and frequently the bytes of the UTF-8 characters are truncated.
The following is an example of the issue, where the 2 files are "HelloHi" and "Hello你好", and name-width=6:
...oHi | 0
...<BD><A0>好 | 0Make the filename pointer move by the actual number of bytes of the characters to drop from the filename, rather than their display width, using the utf8_width() function.
--- diff.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)
diff.c +4 -11
@@ -2859,17 +2859,10 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)char *slash; prefix ="..." ; len -=3 ; -/* -* NEEDSWORK: (name_len - len) counts the display -* width, which would be shorter than the byte -* length of the corresponding substring. -* Advancing "name" by that number of bytes does -* *NOT* skip over that many columns, so it is -* very likely that chomping the pathname at the -* slash we will find starting from "name" will -* leave the resulting string still too long. -*/ - name += name_len - len; + +while (name_len > len) + name_len -=utf8_width ((const char **)&name,NULL ); + slash =strchr (name,'/' );if (slash) name = slash;
Signature · 1 line
2.43.0
Quoted 16 lines · LorenzoPegorari
The
show_stats()function tries to scale the filenames in the diffstat to ensure they don't exceed the givenname-width. It does so by calculating the "display width" of the characters to be dropped, but then advances the filename pointer by that number of bytes.However, the "display width" of a character is not always equal to its byte count. The result is that sometimes, when displaying UTF-8 characters, filenames exceed the given
name-width, and frequently the bytes of the UTF-8 characters are truncated.The following is an example of the issue, where the 2 files are "HelloHi" and "Hello你好", and
name-width=6:...oHi | 0 ...<BD><A0>好 | 0
Quoted 8 lines
Make the filename pointer move by the actual number of bytes of the characters to drop from the filename, rather than their display width, using the
utf8_width()function.Signed-off-by LorenzoPegorari <[email protected]>--- diff.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)
Two comments and a half.
* The change needed for this is surprisingly simple.
* You already know about samples that may exhibit the issue you are addressing. Can we add it as a test case somewhere in t/ directory?
* The NEEDSWORK item addressed by this patch is one of the two NEEDSWORK items added by ce8529b2 (diff: leave NEEDWORK notes in show_stats() function, 2022-10-21). Makes me wonder how involved the changes would need to be to solve the other one?
Thanks.
Quoted 26 lines
diff.c +4
-11@@ -2859,17 +2859,10 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)char *slash; prefix ="..." ; len -=3 ; -/* -* NEEDSWORK: (name_len - len) counts the display -* width, which would be shorter than the byte -* length of the corresponding substring. -* Advancing "name" by that number of bytes does -* *NOT* skip over that many columns, so it is -* very likely that chomping the pathname at the -* slash we will find starting from "name" will -* leave the resulting string still too long. -*/ - name += name_len - len; + +while (name_len > len) + name_len -=utf8_width ((const char **)&name,NULL ); + slash =strchr (name,'/' );if (slash) name = slash;
Quoted 22 lines · Junio C Hamano
Quoted 20 lines · LorenzoPegorari
The
show_stats()function tries to scale the filenames in the diffstat to ensure they don't exceed the givenname-width. It does so by calculating the "display width" of the characters to be dropped, but then advances the filename pointer by that number of bytes.However, the "display width" of a character is not always equal to its byte count. The result is that sometimes, when displaying UTF-8 characters, filenames exceed the given
name-width, and frequently the bytes of the UTF-8 characters are truncated.The following is an example of the issue, where the 2 files are "HelloHi" and "Hello你好", and
name-width=6:...oHi | 0 ...<BD><A0>好 | 0Make the filename pointer move by the actual number of bytes of the characters to drop from the filename, rather than their display width, using the
utf8_width()function.
Quoted 8 lines
Quoted 4 lines
Signed-off-by LorenzoPegorari <[email protected]>--- diff.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)Two comments and a half.
* The change needed for this is surprisingly simple.
It is indeed surprisingly simple, I agree!
Quoted 3 lines
* You already know about samples that may exhibit the issue you are addressing. Can we add it as a test case somewhere in t/ directory?
Yeah, we should add a test case. I will do it in the next reroll.
Quoted 4 lines
* The NEEDSWORK item addressed by this patch is one of the two NEEDSWORK items added by
ce8529b2(diff: leave NEEDWORK notes inshow_stats()function, 2022-10-21). Makes me wonder how involved the changes would need to be to solve the other one?
Mmh, I see. I'll take a closer look, but at a first glance it doesn't seem too involved.
Quoted 3 lines
Thanks.
Thank you!
Quoted 27 lines
Quoted 27 lines
diff.c +4
-11@@ -2859,17 +2859,10 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)char *slash; prefix ="..." ; len -=3 ; -/* -* NEEDSWORK: (name_len - len) counts the display -* width, which would be shorter than the byte -* length of the corresponding substring. -* Advancing "name" by that number of bytes does -* *NOT* skip over that many columns, so it is -* very likely that chomping the pathname at the -* slash we will find starting from "name" will -* leave the resulting string still too long. -*/ - name += name_len - len; + +while (name_len > len) + name_len -=utf8_width ((const char **)&name,NULL ); + slash =strchr (name,'/' );if (slash) name = slash;
v2 cover diff: improve scaling of filenames in diffstat to handle UTF-8 chars
In reply to LorenzoPegorariAdded a test (as Junio Hamano suggested) to check how the generated diffstat handles UTF-8 characters when given various name-widths.
This allowed me to notice a bug where, if the given name-width was 2 or less, the len variable would become negative, entering an infinite loop. So I also fixed this bug.
LorenzoPegorari (2):
diff: improve scaling of filenames in diffstat to handle UTF-8 chars
t4073: add test for diffstat paths length when containing UTF-8 chars
diff.c | 17 ++++----- t/meson.build | 1 + t/t4073-diff-stat-name-width.sh | 61 +++++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+), 11 deletions(-) create mode 100755 t/t4073-diff-stat-name-width.sh
Range-diff against v1:
1: 63e73122d1 ! 1: abeb8d3439 diff: improve scaling of filenames in diffstat to handle UTF-8 chars
@@ Commit message
characters to drop from the filename, rather than their display width, using
the `utf8_width()` function.
+ Force `len` to not be less than 0 (this happens if the given `name-width` is
+ 2 or less), otherwise an infinite loop is entered.
+
Signed-off-by: LorenzoPegorari <[email protected]>
## diff.c ##
@@ diff.c: static void show_stats(struct diffstat_t *data, struct diff_options *opt
- * leave the resulting string still too long.
- */
- name += name_len - len;
++ if (len < 0)
++ len = 0;
+
+ while (name_len > len)
+ name_len -= utf8_width((const char**)&name, NULL);-: ---------- > 2: ee088ea6ef t4073: add test for diffstat paths length when containing UTF-8 chars
Signature · 1 line
2.43.0
PATCH v2 1/2 diff: improve scaling of filenames in diffstat to handle UTF-8 chars
The show_stats() function tries to scale the filenames in the diffstat to ensure they don't exceed the given name-width. It does so by calculating the "display width" of the characters to be dropped, but then advances the filename pointer by that number of bytes.
However, the "display width" of a character is not always equal to its byte count. The result is that sometimes, when displaying UTF-8 characters, filenames exceed the given name-width, and frequently the bytes of the UTF-8 characters are truncated.
The following is an example of the issue, where the 2 files are "HelloHi" and "Hello你好", and name-width=6:
...oHi | 0
...<BD><A0>好 | 0Make the filename pointer move by the actual number of bytes of the characters to drop from the filename, rather than their display width, using the utf8_width() function.
Force len to not be less than 0 (this happens if the given name-width is 2 or less), otherwise an infinite loop is entered.
--- diff.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
diff.c +6 -11
@@ -2859,17 +2859,12 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)char *slash; prefix ="..." ; len -=3 ; -/* -* NEEDSWORK: (name_len - len) counts the display -* width, which would be shorter than the byte -* length of the corresponding substring. -* Advancing "name" by that number of bytes does -* *NOT* skip over that many columns, so it is -* very likely that chomping the pathname at the -* slash we will find starting from "name" will -* leave the resulting string still too long. -*/ - name += name_len - len; +if (len <0 ) + len =0 ; + +while (name_len > len) + name_len -=utf8_width ((const char **)&name,NULL ); + slash =strchr (name,'/' );if (slash) name = slash;
Signature · 1 line
2.43.0
PATCH v2 2/2 t4073: add test for diffstat paths length when containing UTF-8 chars
In reply to LorenzoPegorariAdd test checking the length of filepaths containing UTF-8 chars when generating a diffstat with various name-widths.
--- t/meson.build | 1 + t/t4073-diff-stat-name-width.sh | 61 +++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100755 t/t4073-diff-stat-name-width.sh
t/meson.build +1 -0
@@ -498,6 +498,7 @@ integration_tests = [ 't4070-diff-pairs.sh', 't4071-diff-minimal.sh', 't4072-diff-max-depth.sh', + 't4073-diff-stat.sh', 't4100-apply-stat.sh', 't4101-apply-nonl.sh', 't4102-apply-rename.sh',
t/t4073-diff-stat-name-width.sh new file +61 -0
61 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [GSoC PATCH v2 2/2] t4073: add test for diffstat paths length when containing UTF-8 chars
t/meson.build integration_tests = [ · line 501
Quoted 9 lines · LorenzoPegorari
t/meson.build +1
-0@@ -498,6 +498,7 @@ integration_tests = [ 't4070-diff-pairs.sh', 't4071-diff-minimal.sh', 't4072-diff-max-depth.sh', + 't4073-diff-stat.sh',
This name ...
t/t4073-diff-stat-name-width.sh
Quoted 5 lines
t/meson.build +0
-0't4100-apply-stat.sh', 't4101-apply-nonl.sh', 't4102-apply-rename.sh',t/t4073-diff-stat-name-width.sh new file +0
-0
... must match this one. I already locally updated the former to match, so no need to resend, but if you need to reroll the patch in the future, please make sure to correct this part. Thanks.