Skip to contentSkip to conversation

diff: improve scaling of filenames in diffstat to handle UTF-8 chars

PATCH Superseded LorenzoPegorari 2 participants 7 messages Jan 14 – Jan 17 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 diff: improve scaling of filenames in diffstat to handle UTF-8 chars

The show_stats() function tries to scale the filenames in the diffstat to ensure they don't exceed the given name-width. It does so by calculating the "display width" of the characters to be dropped, but then advances the filename pointer by that number of bytes.

However, the "display width" of a character is not always equal to its byte count. The result is that sometimes, when displaying UTF-8 characters, filenames exceed the given name-width, and frequently the bytes of the UTF-8 characters are truncated.

The following is an example of the issue, where the 2 files are "HelloHi" and "Hello你好", and name-width=6:

    ...oHi | 0
    ...<BD><A0>好 | 0

Make the filename pointer move by the actual number of bytes of the characters to drop from the filename, rather than their display width, using the utf8_width() function.

---
 diff.c | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)
diff.c +4 -11
index a68ddd2168..271ace5728 100644
@@ -2859,17 +2859,10 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)
 			char *slash;
 			prefix = "...";
 			len -= 3;
-			/*
-			 * NEEDSWORK: (name_len - len) counts the display
-			 * width, which would be shorter than the byte
-			 * length of the corresponding substring.
-			 * Advancing "name" by that number of bytes does
-			 * *NOT* skip over that many columns, so it is
-			 * very likely that chomping the pathname at the
-			 * slash we will find starting from "name" will
-			 * leave the resulting string still too long.
-			 */
-			name += name_len - len;
+
+			while (name_len > len)
+				name_len -= utf8_width((const char**)&name, NULL);
+
 			slash = strchr(name, '/');
 			if (slash)
 				name = slash;
Signature · 1 line
2.43.0
#1 lore raw
Quoted 16 lines · LorenzoPegorari

The show_stats() function tries to scale the filenames in the diffstat to ensure they don't exceed the given name-width. It does so by calculating the "display width" of the characters to be dropped, but then advances the filename pointer by that number of bytes.

However, the "display width" of a character is not always equal to its byte count. The result is that sometimes, when displaying UTF-8 characters, filenames exceed the given name-width, and frequently the bytes of the UTF-8 characters are truncated.

The following is an example of the issue, where the 2 files are "HelloHi" and "Hello你好", and name-width=6:

    ...oHi | 0
    ...<BD><A0>好 | 0
Quoted 8 lines

Make the filename pointer move by the actual number of bytes of the characters to drop from the filename, rather than their display width, using the utf8_width() function.

---
 diff.c | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

Two comments and a half.

* The change needed for this is surprisingly simple.

* You already know about samples that may exhibit the issue you are addressing. Can we add it as a test case somewhere in t/ directory?

* The NEEDSWORK item addressed by this patch is one of the two NEEDSWORK items added by ce8529b2 (diff: leave NEEDWORK notes in show_stats() function, 2022-10-21). Makes me wonder how involved the changes would need to be to solve the other one?

Thanks.

Quoted 26 lines
diff.c +4 -11
index a68ddd2168..271ace5728 100644
@@ -2859,17 +2859,10 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)
 			char *slash;
 			prefix = "...";
 			len -= 3;
-			/*
-			 * NEEDSWORK: (name_len - len) counts the display
-			 * width, which would be shorter than the byte
-			 * length of the corresponding substring.
-			 * Advancing "name" by that number of bytes does
-			 * *NOT* skip over that many columns, so it is
-			 * very likely that chomping the pathname at the
-			 * slash we will find starting from "name" will
-			 * leave the resulting string still too long.
-			 */
-			name += name_len - len;
+
+			while (name_len > len)
+				name_len -= utf8_width((const char**)&name, NULL);
+
 			slash = strchr(name, '/');
 			if (slash)
 				name = slash;
#2 lore raw
Quoted 22 lines · Junio C Hamano
Quoted 20 lines · LorenzoPegorari

The show_stats() function tries to scale the filenames in the diffstat to ensure they don't exceed the given name-width. It does so by calculating the "display width" of the characters to be dropped, but then advances the filename pointer by that number of bytes.

However, the "display width" of a character is not always equal to its byte count. The result is that sometimes, when displaying UTF-8 characters, filenames exceed the given name-width, and frequently the bytes of the UTF-8 characters are truncated.

The following is an example of the issue, where the 2 files are "HelloHi" and "Hello你好", and name-width=6:

    ...oHi | 0
    ...<BD><A0>好 | 0

Make the filename pointer move by the actual number of bytes of the characters to drop from the filename, rather than their display width, using the utf8_width() function.

Quoted 8 lines
Quoted 4 lines
---
 diff.c | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

Two comments and a half.

* The change needed for this is surprisingly simple.

It is indeed surprisingly simple, I agree!

Quoted 3 lines

* You already know about samples that may exhibit the issue you are addressing. Can we add it as a test case somewhere in t/ directory?

Yeah, we should add a test case. I will do it in the next reroll.

Quoted 4 lines

* The NEEDSWORK item addressed by this patch is one of the two NEEDSWORK items added by ce8529b2 (diff: leave NEEDWORK notes in show_stats() function, 2022-10-21). Makes me wonder how involved the changes would need to be to solve the other one?

Mmh, I see. I'll take a closer look, but at a first glance it doesn't seem too involved.

Quoted 3 lines

Thanks.

Thank you!

Quoted 27 lines
Quoted 27 lines
diff.c +4 -11
index a68ddd2168..271ace5728 100644
@@ -2859,17 +2859,10 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)
 			char *slash;
 			prefix = "...";
 			len -= 3;
-			/*
-			 * NEEDSWORK: (name_len - len) counts the display
-			 * width, which would be shorter than the byte
-			 * length of the corresponding substring.
-			 * Advancing "name" by that number of bytes does
-			 * *NOT* skip over that many columns, so it is
-			 * very likely that chomping the pathname at the
-			 * slash we will find starting from "name" will
-			 * leave the resulting string still too long.
-			 */
-			name += name_len - len;
+
+			while (name_len > len)
+				name_len -= utf8_width((const char**)&name, NULL);
+
 			slash = strchr(name, '/');
 			if (slash)
 				name = slash;
#3 lore raw

v2 cover diff: improve scaling of filenames in diffstat to handle UTF-8 chars

In reply to LorenzoPegorari

Added a test (as Junio Hamano suggested) to check how the generated diffstat handles UTF-8 characters when given various name-widths.

This allowed me to notice a bug where, if the given name-width was 2 or less, the len variable would become negative, entering an infinite loop. So I also fixed this bug.

LorenzoPegorari (2):
diff: improve scaling of filenames in diffstat to handle UTF-8 chars
t4073: add test for diffstat paths length when containing UTF-8 chars

 diff.c                          | 17 ++++-----
 t/meson.build                   |  1 +
 t/t4073-diff-stat-name-width.sh | 61 +++++++++++++++++++++++++++++++++
 3 files changed, 68 insertions(+), 11 deletions(-)
 create mode 100755 t/t4073-diff-stat-name-width.sh

Range-diff against v1:
1: 63e73122d1 ! 1: abeb8d3439 diff: improve scaling of filenames in diffstat to handle UTF-8 chars

    @@ Commit message
         characters to drop from the filename, rather than their display width, using
         the `utf8_width()` function.
     
    +    Force `len` to not be less than 0 (this happens if the given `name-width` is
    +    2 or less), otherwise an infinite loop is entered.
    +
         Signed-off-by: LorenzoPegorari <[email protected]>
     
      ## diff.c ##
    @@ diff.c: static void show_stats(struct diffstat_t *data, struct diff_options *opt
     -			 * leave the resulting string still too long.
     -			 */
     -			name += name_len - len;
    ++			if (len < 0)
    ++				len = 0;
     +
     +			while (name_len > len)
     +				name_len -= utf8_width((const char**)&name, NULL);

-: ---------- > 2: ee088ea6ef t4073: add test for diffstat paths length when containing UTF-8 chars

Signature · 1 line
2.43.0
#4 lore raw

PATCH v2 1/2 diff: improve scaling of filenames in diffstat to handle UTF-8 chars

The show_stats() function tries to scale the filenames in the diffstat to ensure they don't exceed the given name-width. It does so by calculating the "display width" of the characters to be dropped, but then advances the filename pointer by that number of bytes.

However, the "display width" of a character is not always equal to its byte count. The result is that sometimes, when displaying UTF-8 characters, filenames exceed the given name-width, and frequently the bytes of the UTF-8 characters are truncated.

The following is an example of the issue, where the 2 files are "HelloHi" and "Hello你好", and name-width=6:

    ...oHi | 0
    ...<BD><A0>好 | 0

Make the filename pointer move by the actual number of bytes of the characters to drop from the filename, rather than their display width, using the utf8_width() function.

Force len to not be less than 0 (this happens if the given name-width is 2 or less), otherwise an infinite loop is entered.

---
 diff.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)
diff.c +6 -11
index a68ddd2168..452fc69775 100644
@@ -2859,17 +2859,12 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)
 			char *slash;
 			prefix = "...";
 			len -= 3;
-			/*
-			 * NEEDSWORK: (name_len - len) counts the display
-			 * width, which would be shorter than the byte
-			 * length of the corresponding substring.
-			 * Advancing "name" by that number of bytes does
-			 * *NOT* skip over that many columns, so it is
-			 * very likely that chomping the pathname at the
-			 * slash we will find starting from "name" will
-			 * leave the resulting string still too long.
-			 */
-			name += name_len - len;
+			if (len < 0)
+				len = 0;
+
+			while (name_len > len)
+				name_len -= utf8_width((const char**)&name, NULL);
+
 			slash = strchr(name, '/');
 			if (slash)
 				name = slash;
Signature · 1 line
2.43.0
#5 lore raw

PATCH v2 2/2 t4073: add test for diffstat paths length when containing UTF-8 chars

In reply to LorenzoPegorari

Add test checking the length of filepaths containing UTF-8 chars when generating a diffstat with various name-widths.

---
 t/meson.build                   |  1 +
 t/t4073-diff-stat-name-width.sh | 61 +++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)
 create mode 100755 t/t4073-diff-stat-name-width.sh
t/meson.build +1 -0
index 459c52a489..f2ad6d2f12 100644
@@ -498,6 +498,7 @@ integration_tests = [
   't4070-diff-pairs.sh',
   't4071-diff-minimal.sh',
   't4072-diff-max-depth.sh',
+  't4073-diff-stat.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
t/t4073-diff-stat-name-width.sh new file +61 -0

61 lines not on this page · raw message

Signature · 1 line
2.43.0
#6 lore raw

Re: [GSoC PATCH v2 2/2] t4073: add test for diffstat paths length when containing UTF-8 chars

t/meson.build integration_tests = [ · line 501

Quoted 9 lines · LorenzoPegorari
t/meson.build +1 -0
index 459c52a489..f2ad6d2f12 100644
@@ -498,6 +498,7 @@ integration_tests = [
   't4070-diff-pairs.sh',
   't4071-diff-minimal.sh',
   't4072-diff-max-depth.sh',
+  't4073-diff-stat.sh',

This name ...

t/t4073-diff-stat-name-width.sh

Quoted 5 lines
t/meson.build +0 -0
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
t/t4073-diff-stat-name-width.sh new file +0 -0
new file mode 100755

... must match this one. I already locally updated the former to match, so no need to resend, but if you need to reroll the patch in the future, please make sure to correct this part. Thanks.