iconv.3: Clarify the behavior when input is untranslatable
PATCH 1/1 iconv.3: Clarify the behavior when input is untranslatable
From: Reuben Thomas <[email protected]>
The manual page does not fully reflect the behaviour of glibc's iconv(3). The manual page says:
The conversion can stop for four reasons:
- An invalid multibyte sequence is encountered in the input. In this case, it sets errno to EILSEQ and returns (size_t) -1. *inbuf is left pointing to the beginning of the invalid multibyte sequence.
[...]
The phrase "An invalid multibyte sequence is encountered in the input" is confusing, because it suggests that it refers only to the validity of the input per se (e.g. a non-UTF-8 sequence in input purporting to be UTF-8).
However, according to the original author of the manual page, Bruno Haible[1], it also refers to input that cannot be translated to the desired output encoding; and indeed, glibc's iconv returns EILSEQ when the input cannot be translated, even though it is valid.
This patch adds language that reflects the actual behavior, by adding an explicit bullet that distinguishes this case.
f
--- man3/iconv.3 | 8 ++++++++ 1 file changed, 8 insertions(+)
man3/iconv.3 +8 -0
@@ -80,6 +80,14 @@ .SH DESCRIPTION \fI*inbuf\fP is left pointing to the beginning of the invalid multibyte sequence. .IP \[bu] +An multibyte sequence is encountered in the input which +cannot be translated to the character encoding of the output. +In this case, +it sets \fIerrno\fP to \fBEILSEQ\fP and returns +.IR (size_t)\ \-1 . +\fI*inbuf\fP +is left pointing to the beginning of the invalid multibyte sequence. +.IP \[bu] The input byte sequence has been entirely converted, that is, \fI*inbytesleft\fP has gone down to 0. In this case,
Signature · 1 line
2.40.1
Quoted 2 lines · Alejandro Colomar
This patch adds language that reflects the actual behavior, by adding an explicit bullet that distinguishes this case.
That is the right approach. Thanks for taking the initiative.
But I think that more details should be added, so that programmers are not surprised if their program behaves differently on, say, musl libc or FreeBSD than on glibc.
Find attached my take to describe the condition appropriately.
Bruno
From bc3102bd88b2c481d49cdb3433d8520d1289271b Mon Sep 17 00:00:00 2001
From: Bruno Haible <[email protected]>
Date: Sun, 21 May 2023 13:05:29 +0200
Subject: [PATCH] List a fifth conditions when iconv(3) may stop.
--- man3/iconv.3 | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
man3/iconv.3 +29 -1
@@ -71,7 +71,7 @@ If the character encoding of the input is stateful, the function can also convert a sequence of input bytes to an update to the conversion state without producing any output bytes; such input is called a \fIshift sequence\fP. -The conversion can stop for four reasons: +The conversion can stop for five reasons: .IP \[bu] 3 An invalid multibyte sequence is encountered in the input. In this case,
@@ -80,6 +80,34 @@ it sets \fIerrno\fP to \fBEILSEQ\fP and returns \fI*inbuf\fP is left pointing to the beginning of the invalid multibyte sequence. .IP \[bu] +A multibyte sequence is encountered that is valid but that cannot be +translated to the character encoding of the output. This condition +depends on the implementation and on the conversion descriptor. +In the GNU C library and GNU libiconv, if +.I cd +was created without the suffix +.B //TRANSLIT +or +.BR //IGNORE , +the conversion is strict: lossy conversions produce this condition. +If the suffix +.B //TRANSLIT +was specified, transliteration can avoid this condition in some cases. +In the musl C library, this condition cannot occur because a conversion to +.B '*' +is used as a fallback. +In the FreeBSD, NetBSD, and Solaris implementations of +.BR iconv , +this condition cannot occur either, because a conversion to +.B '?' +is used as a fallback. +When this condition is met, +.B iconv +sets \fIerrno\fP to \fBEILSEQ\fP and returns +.IR (size_t)\ \-1 . +\fI*inbuf\fP +is left pointing to the beginning of the invalid multibyte sequence. +.IP \[bu] The input byte sequence has been entirely converted, that is, \fI*inbytesleft\fP has gone down to 0. In this case,
Signature · 1 line
2.34.1
Hi Bruno
Quoted 11 lines · Bruno Haible
Quoted 2 lines · Alejandro Colomar
This patch adds language that reflects the actual behavior, by adding an explicit bullet that distinguishes this case.
That is the right approach. Thanks for taking the initiative.
But I think that more details should be added, so that programmers are not surprised if their program behaves differently on, say, musl libc or FreeBSD than on glibc.
Find attached my take to describe the condition appropriately.
Thanks!
man3/iconv.3 .SH DESCRIPTION · line 84
Quoted 10 lines
Bruno
man3/iconv.3 +2
-0@@ -80,6 +80,34 @@ .SH DESCRIPTION \fI*inbuf\fP is left pointing to the beginning of the invalid multibyte sequence. .IP \[bu] +A multibyte sequence is encountered that is valid but that cannot be +translated to the character encoding of the output. This condition
Please use semantic newlines. See man-pages(7):
Use semantic newlines
In the source of a manual page, new sentences should be started on new lines, long sentences should be split into lines at clause breaks (commas, semicolons, colons, and so on), and long clauses should be split at phrase boundaries. This convention, sometimes known as "semantic newlines", makes it easier to see the effect of patches, which often operate at the level of in‐ dividual sentences, clauses, or phrases.
man3/iconv.3 it sets \fIerrno\fP to \fBEILSEQ\fP and returns · line 95
Quoted 11 lines
man3/iconv.3 +11
-0+depends on the implementation and on the conversion descriptor. +In the GNU C library and GNU libiconv, if +.I cd +was created without the suffix +.B //TRANSLIT +or +.BR //IGNORE , +the conversion is strict: lossy conversions produce this condition. +If the suffix +.B //TRANSLIT +was specified, transliteration can avoid this condition in some cases.
What do you mean by "can" and "some cases"?
man3/iconv.3 it sets \fIerrno\fP to \fBEILSEQ\fP and returns · line 97
Quoted 2 lines
man3/iconv.3 +2
-0+In the musl C library, this condition cannot occur because a conversion to +.B '*'
I recommend either using \[aq]*\[aq] for producing valid C code, or just having an unquoted *.
man3/iconv.3 it sets \fIerrno\fP to \fBEILSEQ\fP and returns · line 100
Quoted 3 lines
man3/iconv.3 +3
-0+is used as a fallback. +In the FreeBSD, NetBSD, and Solaris implementations of +.BR iconv ,
.BR iconv () ,
man3/iconv.3 it sets \fIerrno\fP to \fBEILSEQ\fP and returns · line 102
Quoted 2 lines
man3/iconv.3 +2
-0+this condition cannot occur either, because a conversion to +.B '?'
Similar stuff here.
man3/iconv.3 it sets \fIerrno\fP to \fBEILSEQ\fP and returns · line 105
Quoted 3 lines
man3/iconv.3 +3
-0+is used as a fallback. +When this condition is met, +.B iconv
And here.
man3/iconv.3 it sets \fIerrno\fP to \fBEILSEQ\fP and returns · line 106
Quoted 1 line
man3/iconv.3 +1
-0+sets \fIerrno\fP to \fBEILSEQ\fP and returns
.I errno
.B EILSEQ
I know in other places in the page we use \f, but I'll fix that at some point. Please use macros for new code.
Cheers,
Alex
Quoted 7 lines
man3/iconv.3 +4
-0+.IR (size_t)\ \-1 . +\fI*inbuf\fP +is left pointing to the beginning of the invalid multibyte sequence. +.IP \[bu] The input byte sequence has been entirely converted, that is, \fI*inbytesleft\fP has gone down to 0. In this case,
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
Hi Alejandro,
Quoted 1 line
Please use semantic newlines. See man-pages(7):
Thanks for explaining. I wondered whether I should use one space or two spaces after the end of a sentence, but found no precedent for either style. This explains it :)
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4
-0+In the GNU C library and GNU libiconv, if +.I cd +was created without the suffix +.B //TRANSLIT
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+or +.BR //IGNORE , +the conversion is strict: lossy conversions produce this condition. +If the suffix +.B //TRANSLIT +was specified, transliteration can avoid this condition in some cases.What do you mean by "can" and "some cases"?
GNU libc and GNU libiconv support transliteration, for example, of "½" to "1/2", or of "å" to "aa" in a Danish locale. Here I want to give a hint at the transliteration facility, but without going into too much detail. "transliteration can avoid this condition if there is a transliteration rule for the multibyte character and it fits the character encoding of the output" is too detailed, IMO.
Do you have a better wording than "can ... in some cases"?
Quoted 2 lines
I recommend either using \[aq]*\[aq] for producing valid C code, or just having an unquoted *.
I made the requested style changes.
New patch is attached.
From caa04c49e89e64d7e8b52ab878c6dc2cd0cef5b9 Mon Sep 17 00:00:00 2001
From: Bruno Haible <[email protected]>
Date: Sun, 21 May 2023 13:05:29 +0200
Subject: [PATCH] List a fifth conditions when iconv(3) may stop.
--- man3/iconv.3 | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-)
man3/iconv.3 +34 -1
@@ -71,7 +71,7 @@ If the character encoding of the input is stateful, the function can also convert a sequence of input bytes to an update to the conversion state without producing any output bytes; such input is called a \fIshift sequence\fP. -The conversion can stop for four reasons: +The conversion can stop for five reasons: .IP \[bu] 3 An invalid multibyte sequence is encountered in the input. In this case,
@@ -80,6 +80,39 @@ it sets \fIerrno\fP to \fBEILSEQ\fP and returns \fI*inbuf\fP is left pointing to the beginning of the invalid multibyte sequence. .IP \[bu] +A multibyte sequence is encountered that is valid but that cannot be +translated to the character encoding of the output. +This condition depends on the implementation and on the conversion +descriptor. +In the GNU C library and GNU libiconv, if +.I cd +was created without the suffix +.B //TRANSLIT +or +.BR //IGNORE , +the conversion is strict: lossy conversions produce this condition. +If the suffix +.B //TRANSLIT +was specified, transliteration can avoid this condition in some cases. +In the musl C library, this condition cannot occur because a conversion to +.B \[aq]*\[aq] +is used as a fallback. +In the FreeBSD, NetBSD, and Solaris implementations of +.BR iconv (), +this condition cannot occur either, because a conversion to +.B \[aq]?\[aq] +is used as a fallback. +When this condition is met, +.BR iconv () +sets +.I errno +to +.B EILSEQ +and returns +.IR (size_t)\ \-1 . +.I *inbuf +is left pointing to the beginning of the unconvertible multibyte sequence. +.IP \[bu] The input byte sequence has been entirely converted, that is, \fI*inbytesleft\fP has gone down to 0. In this case,
Signature · 1 line
2.34.1
2 spaces after the end of a sentence is the _right_ amount (was: [PATCH] iconv.3: Clarify the behavior when input is untranslatable)
Hi Bruno,
Quoted 6 lines · Bruno Haible
Hi Alejandro,
Quoted 1 line
Please use semantic newlines. See man-pages(7):
Thanks for explaining. I wondered whether I should use one space or two spaces after the end of a sentence,
That one's easy: one space is always wrong. :-)
<https://web.archive.org/web/20171107164742/http://www.heracliteanriver.com/?p=324>
<https://lore.kernel.org/linux-man/[email protected]/T/#u>
<https://lists.gnu.org/archive/html/groff/2020-11/msg00076.html>
Quoted 2 lines
but found no precedent for either style. This
explains it :)
:)
Cheers,
Alex
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
Hi Bruno,
Quoted 21 lines · Bruno Haible
Hi Alejandro,
Quoted 1 line
Please use semantic newlines. See man-pages(7):
Thanks for explaining. I wondered whether I should use one space or two spaces after the end of a sentence, but found no precedent for either style. This explains it :)
Quoted 4 lines
Quoted 4 lines
man3/iconv.3 +4
-0+In the GNU C library and GNU libiconv, if +.I cd +was created without the suffix +.B //TRANSLITQuoted 8 lines
man3/iconv.3 it sets \fIerrno\fP to \fBEILSEQ\fP and returns · line 96
Quoted 6 lines
man3/iconv.3 +6
-0+or +.BR //IGNORE , +the conversion is strict: lossy conversions produce this condition. +If the suffix +.B //TRANSLIT +was specified, transliteration can avoid this condition in some cases.What do you mean by "can" and "some cases"?
Quoted 7 lines
GNU libc and GNU libiconv support transliteration, for example, of "½" to "1/2", or of "å" to "aa" in a Danish locale. Here I want to give a hint at the transliteration facility, but without going into too much detail. "transliteration can avoid this condition if there is a transliteration rule for the multibyte character and it fits the character encoding of the output" is too detailed, IMO.
Do you have a better wording than "can ... in some cases"?
If you include the full version in the commit log, to be able to understand it in the future, I'm fine with it.
Quoted 5 lines
Quoted 3 lines
I recommend either using \[aq]*\[aq] for producing valid C code, or just having an unquoted *.
I made the requested style changes.
Thanks,
Alex
Quoted 3 lines
New patch is attached.
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
Quoted 4 lines · Alejandro Colomar
Quoted 1 line
Do you have a better wording than "can ... in some cases"?
If you include the full version in the commit log, to be able to understand it in the future, I'm fine with it.
OK. Here is a patch with the details included in the commit message.
From 4cc4ad011b3ffa30159d3a67e262a46da4600cba Mon Sep 17 00:00:00 2001
From: Bruno Haible <[email protected]>
Date: Sun, 21 May 2023 13:05:29 +0200
Subject: [PATCH] List a fifth condition when iconv(3) may stop.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The wording regarding transliteration is vague, because this man page is not the right place for going into the details of the transliteration. Here are the details:
GNU libc and GNU libiconv support transliteration, for example, of "½" to "1/2", or of "å" to "aa" in a Danish locale. The transliteration maps a multibyte character of the input encoding to zero or more characters in the output. There are two kinds of transliteration rules:
- Those that are valid regardless of locale. Typically this means that the original and the transliterated character have similar glyphs, such as in the case "½" to "1/2".
In GNU libc, these are collected in the files glibc/localedata/locales/translit_*.
- Those that are valid in a single locale only. Often such a rule reflects similar pronounciation of the original and the transliterated characters. Some locales have script-based transliteration, for example from the Cyrillic script to the Latin script.
In GNU libc, these are collected in the file glibc/localedata/locales/<locale>.
In GNU libiconv, transliterations of this kind are not supported.
--- man3/iconv.3 | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-)
man3/iconv.3 +34 -1
@@ -71,7 +71,7 @@ If the character encoding of the input is stateful, the function can also convert a sequence of input bytes to an update to the conversion state without producing any output bytes; such input is called a \fIshift sequence\fP. -The conversion can stop for four reasons: +The conversion can stop for five reasons: .IP \[bu] 3 An invalid multibyte sequence is encountered in the input. In this case,
@@ -80,6 +80,39 @@ it sets \fIerrno\fP to \fBEILSEQ\fP and returns \fI*inbuf\fP is left pointing to the beginning of the invalid multibyte sequence. .IP \[bu] +A multibyte sequence is encountered that is valid but that cannot be +translated to the character encoding of the output. +This condition depends on the implementation and on the conversion +descriptor. +In the GNU C library and GNU libiconv, if +.I cd +was created without the suffix +.B //TRANSLIT +or +.BR //IGNORE , +the conversion is strict: lossy conversions produce this condition. +If the suffix +.B //TRANSLIT +was specified, transliteration can avoid this condition in some cases. +In the musl C library, this condition cannot occur because a conversion to +.B \[aq]*\[aq] +is used as a fallback. +In the FreeBSD, NetBSD, and Solaris implementations of +.BR iconv (), +this condition cannot occur either, because a conversion to +.B \[aq]?\[aq] +is used as a fallback. +When this condition is met, +.BR iconv () +sets +.I errno +to +.B EILSEQ +and returns +.IR (size_t)\ \-1 . +.I *inbuf +is left pointing to the beginning of the unconvertible multibyte sequence. +.IP \[bu] The input byte sequence has been entirely converted, that is, \fI*inbytesleft\fP has gone down to 0. In this case,
Signature · 1 line
2.34.1
Hi Bruno,
Quoted 8 lines · Bruno Haible
Quoted 4 lines · Alejandro Colomar
Quoted 1 line
Do you have a better wording than "can ... in some cases"?
If you include the full version in the commit log, to be able to understand it in the future, I'm fine with it.
OK. Here is a patch with the details included in the commit message.
Thanks! Patch applied.
Cheers,
Alex
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
Sorry, ignore this patch. I forgot to remove Reuben's authorship when I modified it. I also forgot to specify v2.
Quoted 59 lines · Alejandro Colomar
From: Reuben Thomas <[email protected]>
The manual page does not fully reflect the behaviour of glibc's iconv(3). The manual page says:
The conversion can stop for four reasons:
- An invalid multibyte sequence is encountered in the input. In this case, it sets errno to EILSEQ and returns (
size_t) -1. *inbuf is left pointing to the beginning of the invalid multibyte sequence.[...]
The phrase "An invalid multibyte sequence is encountered in the input" is confusing, because it suggests that it refers only to the validity of the input per se (e.g. a non-UTF-8 sequence in input purporting to be UTF-8).
However, according to the original author of the manual page, Bruno Haible[1], it also refers to input that cannot be translated to the desired output encoding; and indeed, glibc's iconv returns EILSEQ when the input cannot be translated, even though it is valid.
This patch adds language that reflects the actual behavior, by adding an explicit bullet that distinguishes this case.
Reported-by Reuben Thomas <[email protected]>Cc Steffen Nurpmeso <[email protected]>Cc Bruno Haible <[email protected]>Cc Martin Sebor <[email protected]>Signed-off-by Alejandro Colomar <[email protected]>f
--- man3/iconv.3 | 8 ++++++++ 1 file changed, 8 insertions(+)man3/iconv.3 +8
-0@@ -80,6 +80,14 @@ .SH DESCRIPTION \fI*inbuf\fP is left pointing to the beginning of the invalid multibyte sequence. .IP \[bu] +An multibyte sequence is encountered in the input which +cannot be translated to the character encoding of the output. +In this case, +it sets \fIerrno\fP to \fBEILSEQ\fP and returns +.IR (size_t)\ \-1 . +\fI*inbuf\fP +is left pointing to the beginning of the invalid multibyte sequence. +.IP \[bu] The input byte sequence has been entirely converted, that is, \fI*inbytesleft\fP has gone down to 0. In this case,
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5