vt: make VS16 emoji widening controllable
Subsystems (3)
cover vt: make VS16 emoji widening controllable
The VT turns a single-width character followed by VS16 (U+FE0F) into a double-width one, as many terminals do for emoji presentation sequences such as ‼️. Applications that compute widths with wcwidth() consider VS16 zero-width though, and those tracking the cursor position themselves get out of sync. With mosh, a single subject line in an alpine message index was enough to leave the whole screen scrolled by one line.
Terminals are split roughly evenly on this, so this series keeps the current behavior by default and provides a way out:
Patch 1 fixes VS16 widening only taking effect after a double-width character had been displayed, because the Unicode screen memory it relies on is allocated on demand.
Patch 2 lets applications control VS16 widening with DEC private mode 2027 (from the Contour terminal's Unicode core proposal), with vt.default_vs16_wide setting its initial and reset state.
Patch 3 adds DECRQM so applications can query mode states, including 2027, rather than waiting for a reply that never comes.
Tested with a program checking the DECRQM replies for all supported modes, the VS16 width with the mode set, reset and after RIS, with and without vt.default_vs16_wide=0, and that other CSI ... $ sequences are still ignored.
diffstat:
.../admin-guide/kernel-parameters.txt | 10 ++ drivers/tty/vt/vt.c | 114 +++++++++++++++++- include/linux/console_struct.h | 1 + 3 files changed, 124 insertions(+), 1 deletion(-)
Signature · 1 line
2.55.0
PATCH 1/3 vt: don't make VS16 widening depend on earlier screen contents
From: Nicolas Pitre <[email protected]>
VS16 (U+FE0F) widens the preceding character only when that character can be found in the Unicode screen memory. But that memory is allocated on demand, when the first double-width character is displayed. Until then vc_uniscr_getc() returns 0 and the VS16 is ignored.
The same sequence, e.g. U+203C U+FE0F, therefore takes one column on a console that hasn't displayed any double-width character yet, and two columns afterwards.
Allocate the Unicode screen memory when a VS16 shows up so the preceding character is always available. vc_uniscr_check() populates it from the current screen content, which includes that character.
--- drivers/tty/vt/vt.c | 9 +++++++++ 1 file changed, 9 insertions(+)
drivers/tty/vt/vt.c +9 -0
@@ -3128,6 +3128,15 @@ static int vc_process_ucs(struct vc_data *vc, int *c, int *tc) }/* From here the preceding character, if any, must be single-width. */ + +/* +* The Unicode screen memory is allocated only when required. VS16 +* needs it to find the character it applies to, so it can't depend +* on whether a double-width character was displayed earlier. +*/ +if (curr_c ==UCS_VS16 ) +vc_uniscr_check (vc); + prev_c =vc_uniscr_getc (vc,-1 );if (curr_c ==UCS_VS16 && prev_c !=0 ) {
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 1/3] vt: don't make VS16 widening depend on earlier screen contents
Sashiko has reviewed this patch and found no issues.
Quoted 11 lines
From: Nicolas Pitre <[email protected]>
VS16 (U+FE0F) widens the preceding character only when that character can be found in the Unicode screen memory. But that memory is allocated on demand, when the first double-width character is displayed. Until then
vc_uniscr_getc()returns 0 and the VS16 is ignored.The same sequence, e.g. U+203C U+FE0F, therefore takes one column on a console that hasn't displayed any double-width character yet, and two columns afterwards.
Quoted 7 lines
Allocate the Unicode screen memory when a VS16 shows up so the preceding character is always available.
vc_uniscr_check()populates it from the current screen content, which includes that character.Fixes 95b05de0a566 ("vt: properly support zero-width Unicode code points")Assisted-by LLMSigned-off-by Nicolas Pitre <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
PATCH 2/3 vt: add DEC private mode 2027 to control VS16 widening
In reply to Nicolas PitreFrom: Nicolas Pitre <[email protected]>
VS16 (U+FE0F) after a single-width character currently turns it into a double-width one, giving emoji presentation sequences such as U+203C U+FE0F (‼️) a square aspect.
There is no consensus on this among terminals. According to the ucs-detect survey [1], these widen such sequences to 2 cells, as the Linux console does:
kitty, foot, Ghostty, Contour, Konsole, iTerm2, Windows Terminal, mintty, Rio, tmux (3.4)
while these keep them at 1 cell:
xterm, VTE based terminals (GNOME Terminal, XFCE Terminal, ...), Alacritty, st, urxvt, Apple Terminal, GNU Screen, libvterm, xterm.js (WezTerm only partially widens)
On the application side, glibc's wcwidth() is per code point and returns 0 for VS16, and so do the tools relying on it. Even vim, which has its own emoji width table, counts U+203C U+FE0F as 1 column.
Applications that track the cursor position themselves therefore get out of sync with a widening terminal. For example, mosh writes full width rows and moves with relative CR/LF. A row containing ‼️ is one column wider on the console than in mosh's model, so its last cell wraps to the next line and the whole screen ends up scrolled by one. Others are reconsidering widening by default too: tmux 3.6 added the "variation-selector-always-wide" option to turn it off, and xterm's emoji width handling is configurable.
DEC private mode 2027, from the Contour terminal's Unicode core proposal [2], is meant to give applications this control. With it set, VS16 promotes a grapheme cluster to emoji presentation and a width of 2 cells. With it reset, the behavior is left to the terminal. Contour, Ghostty, foot and WezTerm implement it.
Make VS16 widening follow that mode. Applications, or the user before starting them, can opt out with CSI ? 2027 l, in which case VS16 is zero-width as wcwidth() says. Only the VS16 part of the proposal is implemented here.
The initial and reset state of the mode comes from the new vt.default_vs16_wide parameter, similar to vt.default_utf8. It defaults to 1, preserving the existing behavior, and can be set to 0 on systems where applications relying on wcwidth() are the norm.
[1] https://ucs-detect.readthedocs.io/results.html
[2] https://github.com/contour-terminal/terminal-unicode-core
--- Documentation/admin-guide/kernel-parameters.txt | 10 ++++++++++ drivers/tty/vt/vt.c | 15 ++++++++++++++- include/linux/console_struct.h | 1 + 3 files changed, 25 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +10 -0
@@ -8451,6 +8451,16 @@ Kernel parameters Default is 1, i.e. UTF-8 mode is enabled for all newly opened terminals. + vt.default_vs16_wide= + [VT] + Format=<0|1> + Set system-wide default for whether VS16 (U+FE0F) + turns the preceding single-width character into a + double-width one, as in emoji presentation sequences. + This is the initial and reset state of DEC private + mode 2027 on all tty's, which applications can also + change with CSI ? 2027 h/l. Default is 1. + vt.global_cursor_default= [VT] Format=<-1|0|1>
drivers/tty/vt/vt.c +14 -1
@@ -166,6 +166,9 @@ EXPORT_SYMBOL(global_cursor_default);static int cur_default =CUR_UNDERLINE ;module_param (cur_default, int,S_IRUGO |S_IWUSR ); +static bool default_vs16_wide = true; +module_param (default_vs16_wide, bool,0644 ); +/* * ignore_poke: don't unblank the screen when things are typed. This is * mainly for the privacy of braille terminal users.
@@ -1984,6 +1987,7 @@ enum { CSI_DEC_hl_MOUSE_VT200 =1000 , CSI_DEC_hl_ALT_SCREEN =1049 , CSI_DEC_hl_BRACKETED_PASTE =2004 , + CSI_DEC_hl_GRAPHEME_WIDTH = 2027, /* VS16 widening */ };/* console_lock is held */
@@ -2039,6 +2043,9 @@ static void csi_DEC_hl(struct vc_data *vc, bool on_off)case CSI_DEC_hl_BRACKETED_PASTE: vc->vc_bracketed_paste = on_off;break ; +case CSI_DEC_hl_GRAPHEME_WIDTH: + vc->vc_vs16_wide = on_off; +break ;case CSI_DEC_hl_ALT_SCREEN:if (on_off)enter_alt_screen (vc);
@@ -2284,6 +2291,7 @@ static void reset_terminal(struct vc_data *vc, int do_clear) vc->vc_decawm =1 ; vc->vc_deccm = global_cursor_default; vc->vc_decim =0 ; + vc->vc_vs16_wide = default_vs16_wide;if (vc->vc_saved_screen !=NULL ) {kfree (vc->vc_saved_screen);
@@ -3139,12 +3147,17 @@ static int vc_process_ucs(struct vc_data *vc, int *c, int *tc) prev_c =vc_uniscr_getc (vc,-1 ); -if (curr_c ==UCS_VS16 && prev_c !=0 ) { +if (curr_c ==UCS_VS16 && prev_c !=0 && vc->vc_vs16_wide) {/* * VS16 (U+FE0F) is special. It typically turns the preceding * single-width character into a double-width one. Let it * have a width of 1 effectively making the combination with * the preceding character double-width. +* +* This can be disabled with DEC private mode 2027 for +* applications that compute widths per code point with +* wcwidth(), where VS16 is always zero-width. Otherwise +* they lose track of the cursor position. */ *tc =' ' ;return 1 ;
include/linux/console_struct.h +1 -0
@@ -167,6 +167,7 @@ struct vc_data { unsigned int vc_decawm : 1; /* Autowrap Mode */ unsigned int vc_deccm : 1; /* Cursor Visible */ unsigned int vc_decim : 1; /* Insert Mode */ + unsigned int vc_vs16_wide : 1; /* VS16 makes preceding char wide *//* misc */ unsigned int vc_priv : 3; unsigned int vc_need_wrap : 1;
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 2/3] vt: add DEC private mode 2027 to control VS16 widening
Sashiko has reviewed this patch and found no issues.
Quoted 13 lines
From: Nicolas Pitre <[email protected]>
VS16 (U+FE0F) after a single-width character currently turns it into a double-width one, giving emoji presentation sequences such as U+203C U+FE0F (‼️) a square aspect.
There is no consensus on this among terminals. According to the ucs-detect survey [1], these widen such sequences to 2 cells, as the Linux console does:
kitty, foot, Ghostty, Contour, Konsole, iTerm2, Windows Terminal, mintty, Rio, tmux (3.4)
Quoted 8 lines
while these keep them at 1 cell:
[ ... ]
[1] https://ucs-detect.readthedocs.io/results.html
[2] https://github.com/contour-terminal/terminal-unicode-coreAssisted-by LLMSigned-off-by Nicolas Pitre <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH 3/3 vt: add DECRQM support to report mode states
In reply to Nicolas PitreFrom: Nicolas Pitre <[email protected]>
Applications currently have no way to find out whether the VT supports a given mode, nor what its current state is. DECRQM, "CSI Ps $ p" for ANSI modes and "CSI ? Ps $ p" for DEC private modes, is the standard query for this. But the '$' intermediate makes the parser ignore the whole sequence, leaving applications waiting for a reply that never comes.
This matters especially for mode 2027, whose specification relies on DECRQM for applications to detect support before enabling it.
Add an ESdollar parser state for "CSI parameters $", and answer a final 'p' with DECRPM, "CSI [?] Ps ; Pm $ y", where Pm is 1 (set), 2 (reset), 0 (not recognized) or 4 (permanently reset). All modes handled by csi_DEC_hl() and csi_hl() are reported, with 132-column mode reported as permanently reset since it isn't implemented. Other sequences using the '$' intermediate are still ignored.
--- drivers/tty/vt/vt.c | 90 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+)
drivers/tty/vt/vt.c +90 -0
@@ -2083,6 +2083,82 @@ static void csi_hl(struct vc_data *vc, bool on_off) } } +/* DECRPM mode states */ +enum { +DECRPM_NOT_RECOGNIZED =0 , +DECRPM_SET =1 , +DECRPM_RESET =2 , +DECRPM_PERMANENTLY_RESET =4 , +}; + +static int decrpm_state (bool set) +{ +return set ?DECRPM_SET :DECRPM_RESET ; +} + +/* State of a DEC private mode as handled by csi_DEC_hl() */ +static int csi_DEC_mode (struct vc_data *vc,unsigned int mode) +{ +switch (mode) { +case CSI_DEC_hl_CURSOR_KEYS: +return decrpm_state (is_kbd (vc, decckm)); +case CSI_DEC_hl_132_COLUMNS: +return DECRPM_PERMANENTLY_RESET ; +case CSI_DEC_hl_REVERSE_VIDEO: +return decrpm_state (vc->vc_decscnm); +case CSI_DEC_hl_ORIGIN_MODE: +return decrpm_state (vc->vc_decom); +case CSI_DEC_hl_AUTOWRAP: +return decrpm_state (vc->vc_decawm); +case CSI_DEC_hl_AUTOREPEAT: +return decrpm_state (is_kbd (vc, decarm)); +case CSI_DEC_hl_MOUSE_X10: +return decrpm_state (vc->vc_report_mouse ==1 ); +case CSI_DEC_hl_SHOW_CURSOR: +return decrpm_state (vc->vc_deccm); +case CSI_DEC_hl_MOUSE_VT200: +return decrpm_state (vc->vc_report_mouse ==2 ); +case CSI_DEC_hl_ALT_SCREEN: +return decrpm_state (vc->vc_saved_screen !=NULL ); +case CSI_DEC_hl_BRACKETED_PASTE: +return decrpm_state (vc->vc_bracketed_paste); +case CSI_DEC_hl_GRAPHEME_WIDTH: +return decrpm_state (vc->vc_vs16_wide); + } +return DECRPM_NOT_RECOGNIZED ; +} + +/* State of an ANSI mode as handled by csi_hl() */ +static int csi_ANSI_mode (struct vc_data *vc,unsigned int mode) +{ +switch (mode) { +case CSI_hl_DISPLAY_CTRL: +return decrpm_state (vc->vc_disp_ctrl); +case CSI_hl_INSERT: +return decrpm_state (vc->vc_decim); +case CSI_hl_AUTO_NL: +return decrpm_state (is_kbd (vc, lnm)); + } +return DECRPM_NOT_RECOGNIZED ; +} + +/* +* DECRQM: request mode, "CSI Ps $ p" for ANSI modes and "CSI ? Ps $ p" for +* DEC private modes. Reply with DECRPM: "CSI [?] Ps ; Pm $ y". +*/ +/* console_lock is held */ +static void csi_RQM (struct tty_struct *tty,struct vc_data *vc) +{ +unsigned int mode = vc->vc_par[0 ]; +bool dec = vc->vc_priv == EPdec; +char buf[32 ]; +int len; + + len =snprintf (buf,sizeof (buf),"\033[%s%u;%d$y" , dec ?"?" :"" , + mode, dec ?csi_DEC_mode (vc, mode) :csi_ANSI_mode (vc, mode)); +respond_string (buf, len, tty->port); +} +enum CSI_right_square_bracket {CSI_RSB_COLOR_FOR_UNDERLINE =1 ,CSI_RSB_COLOR_FOR_HALF_BRIGHT =2 ,
@@ -2233,6 +2309,7 @@ static void restore_cur(struct vc_data *vc)* @ESsetG1: ESC ) parsed * @ESpercent: ESC % parsed * @EScsiignore: CSI [0x20-0x3f] parsed +* @ESdollar: CSI parameters $ parsed -- final char expected * @ESnonstd: OSC parsed * @ESpalette: OSC P parsed * @ESosc: OSC [0-9] parsed
@@ -2254,6 +2331,7 @@ enum vc_ctl_state { ESsetG1, ESpercent, EScsiignore, + ESdollar, ESnonstd, ESpalette, ESosc,
@@ -2788,6 +2866,9 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, u8 c) vc->vc_par[vc->vc_npar] *=10 ; vc->vc_par[vc->vc_npar] += c -'0' ;return ; +case '$' : + vc->vc_state = ESdollar; +return ; }if (c >=ASCII_CSI_IGNORE_FIRST && c <=ASCII_CSI_IGNORE_LAST ) { vc->vc_state = EScsiignore;
@@ -2846,6 +2927,15 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, u8 c)return ; vc->vc_state = ESnormal;return ; +case ESdollar:/* CSI parameters $, final char expected */ +if (c >=ASCII_CSI_IGNORE_FIRST && c <=ASCII_CSI_IGNORE_LAST ) { + vc->vc_state = EScsiignore; +return ; + } + vc->vc_state = ESnormal; +if (c =='p' && (vc->vc_priv == EPdec || vc->vc_priv == EPecma)) +csi_RQM (tty, vc); +return ;case ESpercent:/* ESC % */ vc->vc_state = ESnormal;switch (c) {
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH 3/3] vt: add DECRQM support to report mode states
Sashiko has reviewed this patch and found no issues.
Quoted 19 lines
From: Nicolas Pitre <[email protected]>
Applications currently have no way to find out whether the VT supports a given mode, nor what its current state is. DECRQM, "CSI Ps $ p" for ANSI modes and "CSI ? Ps $ p" for DEC private modes, is the standard query for this. But the '$' intermediate makes the parser ignore the whole sequence, leaving applications waiting for a reply that never comes.
This matters especially for mode 2027, whose specification relies on DECRQM for applications to detect support before enabling it.
Add an ESdollar parser state for "CSI parameters $", and answer a final 'p' with DECRPM, "CSI [?] Ps ; Pm $ y", where Pm is 1 (set), 2 (reset), 0 (not recognized) or 4 (permanently reset). All modes handled by
csi_DEC_hl()andcsi_hl()are reported, with 132-column mode reported as permanently reset since it isn't implemented. Other sequences using the '$' intermediate are still ignored.
Quoted 2 lines
Assisted-by LLMSigned-off-by Nicolas Pitre <[email protected]>
Sashiko has reviewed this patch and found no issues.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3