Skip to contentSkip to conversation

vt: make VS16 emoji widening controllable

PATCH New Nicolas Pitre 2 participants 7 messages Oct 19

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover vt: make VS16 emoji widening controllable

The VT turns a single-width character followed by VS16 (U+FE0F) into a double-width one, as many terminals do for emoji presentation sequences such as ‼️. Applications that compute widths with wcwidth() consider VS16 zero-width though, and those tracking the cursor position themselves get out of sync. With mosh, a single subject line in an alpine message index was enough to leave the whole screen scrolled by one line.

Terminals are split roughly evenly on this, so this series keeps the current behavior by default and provides a way out:

Patch 1 fixes VS16 widening only taking effect after a double-width character had been displayed, because the Unicode screen memory it relies on is allocated on demand.

Patch 2 lets applications control VS16 widening with DEC private mode 2027 (from the Contour terminal's Unicode core proposal), with vt.default_vs16_wide setting its initial and reset state.

Patch 3 adds DECRQM so applications can query mode states, including 2027, rather than waiting for a reply that never comes.

Tested with a program checking the DECRQM replies for all supported modes, the VS16 width with the mode set, reset and after RIS, with and without vt.default_vs16_wide=0, and that other CSI ... $ sequences are still ignored.

diffstat:

 .../admin-guide/kernel-parameters.txt         |  10 ++
 drivers/tty/vt/vt.c                           | 114 +++++++++++++++++-
 include/linux/console_struct.h                |   1 +
 3 files changed, 124 insertions(+), 1 deletion(-)
Signature · 1 line
2.55.0
#1 lore raw

PATCH 1/3 vt: don't make VS16 widening depend on earlier screen contents

From: Nicolas Pitre <[email protected]>

VS16 (U+FE0F) widens the preceding character only when that character can be found in the Unicode screen memory. But that memory is allocated on demand, when the first double-width character is displayed. Until then vc_uniscr_getc() returns 0 and the VS16 is ignored.

The same sequence, e.g. U+203C U+FE0F, therefore takes one column on a console that hasn't displayed any double-width character yet, and two columns afterwards.

Allocate the Unicode screen memory when a VS16 shows up so the preceding character is always available. vc_uniscr_check() populates it from the current screen content, which includes that character.

Fixes 95b05de0a566 ("vt: properly support zero-width Unicode code points")
Assisted-by LLM
---
 drivers/tty/vt/vt.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/tty/vt/vt.c +9 -0
index 49a140b49eff..335253aec549 100644
@@ -3128,6 +3128,15 @@ static int vc_process_ucs(struct vc_data *vc, int *c, int *tc)
 	}
 
 	/* From here the preceding character, if any, must be single-width. */
+
+	/*
+	 * The Unicode screen memory is allocated only when required. VS16
+	 * needs it to find the character it applies to, so it can't depend
+	 * on whether a double-width character was displayed earlier.
+	 */
+	if (curr_c == UCS_VS16)
+		vc_uniscr_check(vc);
+
 	prev_c = vc_uniscr_getc(vc, -1);
 
 	if (curr_c == UCS_VS16 && prev_c != 0) {
Signature · 1 line
2.55.0
#2 lore raw

Re: [PATCH 1/3] vt: don't make VS16 widening depend on earlier screen contents

Sashiko has reviewed this patch and found no issues.
Quoted 11 lines

From: Nicolas Pitre <[email protected]>

VS16 (U+FE0F) widens the preceding character only when that character can be found in the Unicode screen memory. But that memory is allocated on demand, when the first double-width character is displayed. Until then vc_uniscr_getc() returns 0 and the VS16 is ignored.

The same sequence, e.g. U+203C U+FE0F, therefore takes one column on a console that hasn't displayed any double-width character yet, and two columns afterwards.

Quoted 7 lines

Allocate the Unicode screen memory when a VS16 shows up so the preceding character is always available. vc_uniscr_check() populates it from the current screen content, which includes that character.

Fixes 95b05de0a566 ("vt: properly support zero-width Unicode code points")
Assisted-by LLM

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

PATCH 2/3 vt: add DEC private mode 2027 to control VS16 widening

In reply to Nicolas Pitre

From: Nicolas Pitre <[email protected]>

VS16 (U+FE0F) after a single-width character currently turns it into a double-width one, giving emoji presentation sequences such as U+203C U+FE0F (‼️) a square aspect.

There is no consensus on this among terminals. According to the ucs-detect survey [1], these widen such sequences to 2 cells, as the Linux console does:

kitty, foot, Ghostty, Contour, Konsole, iTerm2, Windows Terminal, mintty, Rio, tmux (3.4)

while these keep them at 1 cell:

xterm, VTE based terminals (GNOME Terminal, XFCE Terminal, ...), Alacritty, st, urxvt, Apple Terminal, GNU Screen, libvterm, xterm.js (WezTerm only partially widens)

On the application side, glibc's wcwidth() is per code point and returns 0 for VS16, and so do the tools relying on it. Even vim, which has its own emoji width table, counts U+203C U+FE0F as 1 column.

Applications that track the cursor position themselves therefore get out of sync with a widening terminal. For example, mosh writes full width rows and moves with relative CR/LF. A row containing ‼️ is one column wider on the console than in mosh's model, so its last cell wraps to the next line and the whole screen ends up scrolled by one. Others are reconsidering widening by default too: tmux 3.6 added the "variation-selector-always-wide" option to turn it off, and xterm's emoji width handling is configurable.

DEC private mode 2027, from the Contour terminal's Unicode core proposal [2], is meant to give applications this control. With it set, VS16 promotes a grapheme cluster to emoji presentation and a width of 2 cells. With it reset, the behavior is left to the terminal. Contour, Ghostty, foot and WezTerm implement it.

Make VS16 widening follow that mode. Applications, or the user before starting them, can opt out with CSI ? 2027 l, in which case VS16 is zero-width as wcwidth() says. Only the VS16 part of the proposal is implemented here.

The initial and reset state of the mode comes from the new vt.default_vs16_wide parameter, similar to vt.default_utf8. It defaults to 1, preserving the existing behavior, and can be set to 0 on systems where applications relying on wcwidth() are the norm.

[1] https://ucs-detect.readthedocs.io/results.html
[2] https://github.com/contour-terminal/terminal-unicode-core

Assisted-by LLM
---
 Documentation/admin-guide/kernel-parameters.txt | 10 ++++++++++
 drivers/tty/vt/vt.c                             | 15 ++++++++++++++-
 include/linux/console_struct.h                  |  1 +
 3 files changed, 25 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +10 -0
index e75344f4e0cd..983afb6c531e 100644
@@ -8451,6 +8451,16 @@ Kernel parameters
 			Default is 1, i.e. UTF-8 mode is enabled for all
 			newly opened terminals.
 
+	vt.default_vs16_wide=
+			[VT]
+			Format=<0|1>
+			Set system-wide default for whether VS16 (U+FE0F)
+			turns the preceding single-width character into a
+			double-width one, as in emoji presentation sequences.
+			This is the initial and reset state of DEC private
+			mode 2027 on all tty's, which applications can also
+			change with CSI ? 2027 h/l. Default is 1.
+
 	vt.global_cursor_default=
 			[VT]
 			Format=<-1|0|1>
drivers/tty/vt/vt.c +14 -1
index 335253aec549..4d5c9cabd058 100644
@@ -166,6 +166,9 @@ EXPORT_SYMBOL(global_cursor_default);
 static int cur_default = CUR_UNDERLINE;
 module_param(cur_default, int, S_IRUGO | S_IWUSR);
 
+static bool default_vs16_wide = true;
+module_param(default_vs16_wide, bool, 0644);
+
 /*
  * ignore_poke: don't unblank the screen when things are typed.  This is
  * mainly for the privacy of braille terminal users.
@@ -1984,6 +1987,7 @@ enum {
 	CSI_DEC_hl_MOUSE_VT200	= 1000,
 	CSI_DEC_hl_ALT_SCREEN	= 1049,
 	CSI_DEC_hl_BRACKETED_PASTE = 2004,
+	CSI_DEC_hl_GRAPHEME_WIDTH = 2027,	/* VS16 widening */
 };
 
 /* console_lock is held */
@@ -2039,6 +2043,9 @@ static void csi_DEC_hl(struct vc_data *vc, bool on_off)
 		case CSI_DEC_hl_BRACKETED_PASTE:
 			vc->vc_bracketed_paste = on_off;
 			break;
+		case CSI_DEC_hl_GRAPHEME_WIDTH:
+			vc->vc_vs16_wide = on_off;
+			break;
 		case CSI_DEC_hl_ALT_SCREEN:
 			if (on_off)
 				enter_alt_screen(vc);
@@ -2284,6 +2291,7 @@ static void reset_terminal(struct vc_data *vc, int do_clear)
 	vc->vc_decawm		= 1;
 	vc->vc_deccm		= global_cursor_default;
 	vc->vc_decim		= 0;
+	vc->vc_vs16_wide	= default_vs16_wide;
 
 	if (vc->vc_saved_screen != NULL) {
 		kfree(vc->vc_saved_screen);
@@ -3139,12 +3147,17 @@ static int vc_process_ucs(struct vc_data *vc, int *c, int *tc)
 
 	prev_c = vc_uniscr_getc(vc, -1);
 
-	if (curr_c == UCS_VS16 && prev_c != 0) {
+	if (curr_c == UCS_VS16 && prev_c != 0 && vc->vc_vs16_wide) {
 		/*
 		 * VS16 (U+FE0F) is special. It typically turns the preceding
 		 * single-width character into a double-width one. Let it
 		 * have a width of 1 effectively making the combination with
 		 * the preceding character double-width.
+		 *
+		 * This can be disabled with DEC private mode 2027 for
+		 * applications that compute widths per code point with
+		 * wcwidth(), where VS16 is always zero-width. Otherwise
+		 * they lose track of the cursor position.
 		 */
 		*tc = ' ';
 		return 1;
include/linux/console_struct.h +1 -0
index fe915afdece5..8a1031cd097e 100644
@@ -167,6 +167,7 @@ struct vc_data {
 	unsigned int	vc_decawm	: 1;	/* Autowrap Mode */
 	unsigned int	vc_deccm	: 1;	/* Cursor Visible */
 	unsigned int	vc_decim	: 1;	/* Insert Mode */
+	unsigned int	vc_vs16_wide	: 1;	/* VS16 makes preceding char wide */
 	/* misc */
 	unsigned int	vc_priv		: 3;
 	unsigned int	vc_need_wrap	: 1;
Signature · 1 line
2.55.0
#4 lore raw

Re: [PATCH 2/3] vt: add DEC private mode 2027 to control VS16 widening

Sashiko has reviewed this patch and found no issues.
Quoted 13 lines

From: Nicolas Pitre <[email protected]>

VS16 (U+FE0F) after a single-width character currently turns it into a double-width one, giving emoji presentation sequences such as U+203C U+FE0F (‼️) a square aspect.

There is no consensus on this among terminals. According to the ucs-detect survey [1], these widen such sequences to 2 cells, as the Linux console does:

kitty, foot, Ghostty, Contour, Konsole, iTerm2, Windows Terminal, mintty, Rio, tmux (3.4)

Quoted 8 lines

while these keep them at 1 cell:

[ ... ]
[1] https://ucs-detect.readthedocs.io/results.html
[2] https://github.com/contour-terminal/terminal-unicode-core

Assisted-by LLM

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#5 lore raw

PATCH 3/3 vt: add DECRQM support to report mode states

In reply to Nicolas Pitre

From: Nicolas Pitre <[email protected]>

Applications currently have no way to find out whether the VT supports a given mode, nor what its current state is. DECRQM, "CSI Ps $ p" for ANSI modes and "CSI ? Ps $ p" for DEC private modes, is the standard query for this. But the '$' intermediate makes the parser ignore the whole sequence, leaving applications waiting for a reply that never comes.

This matters especially for mode 2027, whose specification relies on DECRQM for applications to detect support before enabling it.

Add an ESdollar parser state for "CSI parameters $", and answer a final 'p' with DECRPM, "CSI [?] Ps ; Pm $ y", where Pm is 1 (set), 2 (reset), 0 (not recognized) or 4 (permanently reset). All modes handled by csi_DEC_hl() and csi_hl() are reported, with 132-column mode reported as permanently reset since it isn't implemented. Other sequences using the '$' intermediate are still ignored.

Assisted-by LLM
---
 drivers/tty/vt/vt.c | 90 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)
drivers/tty/vt/vt.c +90 -0
index 4d5c9cabd058..22dfab8de7bc 100644
@@ -2083,6 +2083,82 @@ static void csi_hl(struct vc_data *vc, bool on_off)
 		}
 }
 
+/* DECRPM mode states */
+enum {
+	DECRPM_NOT_RECOGNIZED	= 0,
+	DECRPM_SET		= 1,
+	DECRPM_RESET		= 2,
+	DECRPM_PERMANENTLY_RESET = 4,
+};
+
+static int decrpm_state(bool set)
+{
+	return set ? DECRPM_SET : DECRPM_RESET;
+}
+
+/* State of a DEC private mode as handled by csi_DEC_hl() */
+static int csi_DEC_mode(struct vc_data *vc, unsigned int mode)
+{
+	switch (mode) {
+	case CSI_DEC_hl_CURSOR_KEYS:
+		return decrpm_state(is_kbd(vc, decckm));
+	case CSI_DEC_hl_132_COLUMNS:
+		return DECRPM_PERMANENTLY_RESET;
+	case CSI_DEC_hl_REVERSE_VIDEO:
+		return decrpm_state(vc->vc_decscnm);
+	case CSI_DEC_hl_ORIGIN_MODE:
+		return decrpm_state(vc->vc_decom);
+	case CSI_DEC_hl_AUTOWRAP:
+		return decrpm_state(vc->vc_decawm);
+	case CSI_DEC_hl_AUTOREPEAT:
+		return decrpm_state(is_kbd(vc, decarm));
+	case CSI_DEC_hl_MOUSE_X10:
+		return decrpm_state(vc->vc_report_mouse == 1);
+	case CSI_DEC_hl_SHOW_CURSOR:
+		return decrpm_state(vc->vc_deccm);
+	case CSI_DEC_hl_MOUSE_VT200:
+		return decrpm_state(vc->vc_report_mouse == 2);
+	case CSI_DEC_hl_ALT_SCREEN:
+		return decrpm_state(vc->vc_saved_screen != NULL);
+	case CSI_DEC_hl_BRACKETED_PASTE:
+		return decrpm_state(vc->vc_bracketed_paste);
+	case CSI_DEC_hl_GRAPHEME_WIDTH:
+		return decrpm_state(vc->vc_vs16_wide);
+	}
+	return DECRPM_NOT_RECOGNIZED;
+}
+
+/* State of an ANSI mode as handled by csi_hl() */
+static int csi_ANSI_mode(struct vc_data *vc, unsigned int mode)
+{
+	switch (mode) {
+	case CSI_hl_DISPLAY_CTRL:
+		return decrpm_state(vc->vc_disp_ctrl);
+	case CSI_hl_INSERT:
+		return decrpm_state(vc->vc_decim);
+	case CSI_hl_AUTO_NL:
+		return decrpm_state(is_kbd(vc, lnm));
+	}
+	return DECRPM_NOT_RECOGNIZED;
+}
+
+/*
+ * DECRQM: request mode, "CSI Ps $ p" for ANSI modes and "CSI ? Ps $ p" for
+ * DEC private modes. Reply with DECRPM: "CSI [?] Ps ; Pm $ y".
+ */
+/* console_lock is held */
+static void csi_RQM(struct tty_struct *tty, struct vc_data *vc)
+{
+	unsigned int mode = vc->vc_par[0];
+	bool dec = vc->vc_priv == EPdec;
+	char buf[32];
+	int len;
+
+	len = snprintf(buf, sizeof(buf), "\033[%s%u;%d$y", dec ? "?" : "",
+		       mode, dec ? csi_DEC_mode(vc, mode) : csi_ANSI_mode(vc, mode));
+	respond_string(buf, len, tty->port);
+}
+
 enum CSI_right_square_bracket {
 	CSI_RSB_COLOR_FOR_UNDERLINE		= 1,
 	CSI_RSB_COLOR_FOR_HALF_BRIGHT		= 2,
@@ -2233,6 +2309,7 @@ static void restore_cur(struct vc_data *vc)
  * @ESsetG1:		ESC ) parsed
  * @ESpercent:		ESC % parsed
  * @EScsiignore:	CSI [0x20-0x3f] parsed
+ * @ESdollar:		CSI parameters $ parsed -- final char expected
  * @ESnonstd:		OSC parsed
  * @ESpalette:		OSC P parsed
  * @ESosc:		OSC [0-9] parsed
@@ -2254,6 +2331,7 @@ enum vc_ctl_state {
 	ESsetG1,
 	ESpercent,
 	EScsiignore,
+	ESdollar,
 	ESnonstd,
 	ESpalette,
 	ESosc,
@@ -2788,6 +2866,9 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, u8 c)
 			vc->vc_par[vc->vc_npar] *= 10;
 			vc->vc_par[vc->vc_npar] += c - '0';
 			return;
+		case '$':
+			vc->vc_state = ESdollar;
+			return;
 		}
 		if (c >= ASCII_CSI_IGNORE_FIRST && c <= ASCII_CSI_IGNORE_LAST) {
 			vc->vc_state = EScsiignore;
@@ -2846,6 +2927,15 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, u8 c)
 			return;
 		vc->vc_state = ESnormal;
 		return;
+	case ESdollar:	/* CSI parameters $, final char expected */
+		if (c >= ASCII_CSI_IGNORE_FIRST && c <= ASCII_CSI_IGNORE_LAST) {
+			vc->vc_state = EScsiignore;
+			return;
+		}
+		vc->vc_state = ESnormal;
+		if (c == 'p' && (vc->vc_priv == EPdec || vc->vc_priv == EPecma))
+			csi_RQM(tty, vc);
+		return;
 	case ESpercent:	/* ESC % */
 		vc->vc_state = ESnormal;
 		switch (c) {
Signature · 1 line
2.55.0
#6 lore raw

Re: [PATCH 3/3] vt: add DECRQM support to report mode states

Sashiko has reviewed this patch and found no issues.
Quoted 19 lines

From: Nicolas Pitre <[email protected]>

Applications currently have no way to find out whether the VT supports a given mode, nor what its current state is. DECRQM, "CSI Ps $ p" for ANSI modes and "CSI ? Ps $ p" for DEC private modes, is the standard query for this. But the '$' intermediate makes the parser ignore the whole sequence, leaving applications waiting for a reply that never comes.

This matters especially for mode 2027, whose specification relies on DECRQM for applications to detect support before enabling it.

Add an ESdollar parser state for "CSI parameters $", and answer a final 'p' with DECRPM, "CSI [?] Ps ; Pm $ y", where Pm is 1 (set), 2 (reset), 0 (not recognized) or 4 (permanently reset). All modes handled by csi_DEC_hl() and csi_hl() are reported, with 132-column mode reported as permanently reset since it isn't implemented. Other sequences using the '$' intermediate are still ignored.

Quoted 2 lines
Assisted-by LLM

Sashiko has reviewed this patch and found no issues.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3