Skip to contentSkip to conversation

isalpha.3: behavior is undefined if c is out-of-range

PATCH In review Yedidyah Bar David 4 participants 12 messages Jun 5, 2023 – Jul 8, 2023 Man-pages: manual pages for Linux -- Sections 2, 3, 4, 5, and 7

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 isalpha.3: behavior is undefined if c is out-of-range

Clarify that the behavior of these functions is undefined if c is neither in the unsigned char range nor EOF.

I copied the added text from toupper.3.

In practice, calling them on out-of-range values - tested with recent glibc/gcc - is simply reading from random process memory - meaning, you either get some garbage, if the memory was readable, or a segmentation fault. See also:

https://stackoverflow.com/questions/65514890/glibcs-isalpha-function-and-the-en-us-utf-8-locale

---
 man3/isalpha.3 | 9 +++++++++
 1 file changed, 9 insertions(+)
man3/isalpha.3 +9 -0
index 443c2aa09..8ad997c29 100644
@@ -145,6 +145,15 @@ is the special locale object
 .BR duplocale (3))
 or is not a valid locale object handle.
 .PP
+If
+.I c
+is neither an
+.I "unsigned char"
+value nor
+.BR EOF ,
+the behavior of these functions
+is undefined.
+.PP
 The list below explains the operation of the functions without
 the "_l" suffix;
 the functions with the "_l" suffix differ only in using the locale object

--
2.31.1

Best regards,
--
Didi

#1 lore raw

Hi Yedidyah,

Quoted 5 lines · Yedidyah Bar David

Clarify that the behavior of these functions is undefined if c is neither in the unsigned char range nor EOF.

I copied the added text from toupper.3.

Quoted 8 lines

In practice, calling them on out-of-range values - tested with recent glibc/gcc - is simply reading from random process memory - meaning, you either get some garbage, if the memory was readable, or a segmentation fault. See also:

https://stackoverflow.com/questions/65514890/glibcs-isalpha-function-and-the-en-us-utf-8-locale

This is already covered by the NOTES section, isn't it? BTW, I'll rename that section to CAVEATS.

Thanks,
Alex

Quoted 31 lines
---
 man3/isalpha.3 | 9 +++++++++
 1 file changed, 9 insertions(+)
man3/isalpha.3 +9 -1
index 443c2aa09..8ad997c29 100644
@@ -145,6 +145,15 @@ is the special locale object
 .BR duplocale (3))
 or is not a valid locale object handle.
 .PP
+If
+.I c
+is neither an
+.I "unsigned char"
+value nor
+.BR EOF ,
+the behavior of these functions
+is undefined.
+.PP
 The list below explains the operation of the functions without
 the "_l" suffix;
 the functions with the "_l" suffix differ only in using the locale object
--

2.31.1

Best regards,
--
Didi

Signature · 2 lines
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
#2 lore raw

PATCH 1/1 isalpha.3: Move NOTES to CAVEATS, and clarify some Undefined Behavior

---

I suggest this instead.

 man3/isalpha.3 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
man3/isalpha.3 +4 -2
index 000e816d7..721ca8972 100644
@@ -345,13 +345,15 @@ .SH HISTORY
 .TP
 .BR isascii_l ()
 glibc 2.3.
-.SH NOTES
+.SH CAVEATS
 The standards require that the argument
 .I c
 for these functions is either
 .B EOF
 or a value that is representable in the type
-.IR "unsigned char" .
+.IR "unsigned char" ;
+otherwise,
+the behavior is undefined.
 If the argument
 .I c
 is of type
Signature · 1 line
2.40.1
#5 lore raw

PATCH 1/1 isalpha.3: Move NOTES to CAVEATS, and clarify some Undefined Behavior

In reply to Alejandro Colomar

---

I'm just resending (now with Acked-by from Sam) to test a change in my git-send-email(1) configuration.

Cheers,
Alex

 man3/isalpha.3 | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
man3/isalpha.3 +4 -2
index 000e816d7..721ca8972 100644
@@ -345,13 +345,15 @@ .SH HISTORY
 .TP
 .BR isascii_l ()
 glibc 2.3.
-.SH NOTES
+.SH CAVEATS
 The standards require that the argument
 .I c
 for these functions is either
 .B EOF
 or a value that is representable in the type
-.IR "unsigned char" .
+.IR "unsigned char" ;
+otherwise,
+the behavior is undefined.
 If the argument
 .I c
 is of type
Signature · 1 line
2.40.1
#6 lore raw
In reply to Alejandro Colomar

Hi,

Quoted 14 lines · Alejandro Colomar

Hi Yedidyah,

Quoted 10 lines · Yedidyah Bar David

Clarify that the behavior of these functions is undefined if c is neither in the unsigned char range nor EOF.

I copied the added text from toupper.3.

In practice, calling them on out-of-range values - tested with recent glibc/gcc - is simply reading from random process memory - meaning, you either get some garbage, if the memory was readable, or a segmentation fault. See also:

Quoted 5 lines
Quoted 3 lines

https://stackoverflow.com/questions/65514890/glibcs-isalpha-function-and-the-en-us-utf-8-locale

This is already covered by the NOTES section, isn't it?

It's _mentioned_ there, correct - but not sure it's covered.

It's also mentioned in toupper.3's NOTES.

I think it's helpful to explicitly say that behavior is undefined in this case. If you feel like doing this inside NOTES, one way or another, ok for me.

Right now, NOTES says what you must do, but not what happens if you don't do that.

It also says that for the common case of using them on signed char, you should explicitly cast to unsigned char, first. It also tries to explain why this is necessary. The explanation explains why it's necessary for compliance with the standard, but not why it's a good thing more generally - latter is not explained,
and AFAICT from reading glibc sources, is not necessary - see e.g. this comment from ctype.h:

These point into arrays of 384, so they can be indexed by any unsigned char' value [0,255]; by EOF (-1); or by any signed char' value [-128,-1). ISO C requires that the ctype functions work for unsigned char' values and for EOF; we also support negative signed char' values for broken old programs.

The real reason why you should not call them on negative values other than EOF - casted to unsigned char or not - is simply that most likely this isn't what you meant to do. But that's not about compliance with the standard...

In my patch, I felt like "something should be done", but when deciding what actually should be done, I decided to simply go with what's in toupper.3, which seems good enough for me.

So, what do you think?

Quoted 2 lines

BTW, I'll
rename that section to CAVEATS.

Not sure all of its current content is a caveat.

Best regards,

Quoted 39 lines

Thanks,
Alex

Quoted 27 lines
---
 man3/isalpha.3 | 9 +++++++++
 1 file changed, 9 insertions(+)
man3/isalpha.3 +9 -1
index 443c2aa09..8ad997c29 100644
@@ -145,6 +145,15 @@ is the special locale object
 .BR duplocale (3))
 or is not a valid locale object handle.
 .PP
+If
+.I c
+is neither an
+.I "unsigned char"
+value nor
+.BR EOF ,
+the behavior of these functions
+is undefined.
+.PP
 The list below explains the operation of the functions without
 the "_l" suffix;
 the functions with the "_l" suffix differ only in using the locale object
--

2.31.1

Quoted 4 lines

Best regards,
--
Didi

--
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5

Signature · 1 line
Didi
#7 lore raw

Hi,

Quoted 17 lines · Yedidyah Bar David

Hi,

Quoted 14 lines · Alejandro Colomar

Hi Yedidyah,

Quoted 10 lines · Yedidyah Bar David

Clarify that the behavior of these functions is undefined if c is neither in the unsigned char range nor EOF.

I copied the added text from toupper.3.

In practice, calling them on out-of-range values - tested with recent glibc/gcc - is simply reading from random process memory - meaning, you either get some garbage, if the memory was readable, or a segmentation fault. See also:

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

https://stackoverflow.com/questions/65514890/glibcs-isalpha-function-and-the-en-us-utf-8-locale

This is already covered by the NOTES section, isn't it?

It's _mentioned_ there, correct - but not sure it's covered.

You're right. That's why I've sent the patch mentioning UB. What do you think about that one? (I now see that you like it).

Quoted 2 lines

It's also mentioned in toupper.3's NOTES.

I'll check that page to see if it needs some simplifying.

Quoted 2 lines

I think it's helpful to explicitly say that behavior is undefined in this case.

Yep.

Quoted 13 lines

If you feel like doing this inside NOTES, one way or another, ok for me.

Right now, NOTES says what you must do, but not what happens if you don't do that.

It also says that for the common case of using them on signed char, you should explicitly cast to unsigned char, first. It also tries to explain why this is necessary. The explanation explains why it's necessary for compliance with the standard, but not why it's a good thing more generally - latter is not explained,
and AFAICT from reading glibc sources, is not necessary - see e.g. this comment from ctype.h:

Quoted 5 lines

These point into arrays of 384, so they can be indexed by any unsigned char' value [0,255]; by EOF (-1); or by any signed char' value [-128,-1). ISO C requires that the ctype functions work for unsigned char' values and for EOF; we also support negative signed char' values for broken old programs.

Consider what happens with character 0xFF. If char is signed, it will be interpreted as -1 (i.e., EOF). We're lucky, because 0xFF is not a meaningful char, so probably all isXXX() functions return false for it, but it's slightly different from EOF semantically. If no locales give a meaning for 0xFF, maybe the cast can be removed from ISO C. I do something different: use -funsigned-char when compiling, so char is effectively unsigned char (except that pointers do not convert automatically).

Quoted 4 lines

The real reason why you should not call them on negative values other than EOF - casted to unsigned char or not - is simply that most likely this isn't what you meant to do. But that's not about compliance with the standard...

I guess the standard was cautious to not make 0xFF a useless char. If that's not an issue, I agree, and these functions could do the conversion internally.

Cheers,
Alex

Quoted 56 lines

In my patch, I felt like "something should be done", but when deciding what actually should be done, I decided to simply go with what's in toupper.3, which seems good enough for me.

So, what do you think?

Quoted 2 lines

BTW, I'll
rename that section to CAVEATS.

Not sure all of its current content is a caveat.

Best regards,

Quoted 39 lines

Thanks,
Alex

Quoted 27 lines
---
 man3/isalpha.3 | 9 +++++++++
 1 file changed, 9 insertions(+)
man3/isalpha.3 +9 -1
index 443c2aa09..8ad997c29 100644
@@ -145,6 +145,15 @@ is the special locale object
 .BR duplocale (3))
 or is not a valid locale object handle.
 .PP
+If
+.I c
+is neither an
+.I "unsigned char"
+value nor
+.BR EOF ,
+the behavior of these functions
+is undefined.
+.PP
 The list below explains the operation of the functions without
 the "_l" suffix;
 the functions with the "_l" suffix differ only in using the locale object
--

2.31.1

Quoted 4 lines

Best regards,
--
Didi

--
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5

Signature · 2 lines
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
#8 lore raw
Quoted 69 lines · Alejandro Colomar

Hi,

Quoted 17 lines · Yedidyah Bar David

Hi,

Quoted 14 lines · Alejandro Colomar

Hi Yedidyah,

Quoted 10 lines · Yedidyah Bar David

Clarify that the behavior of these functions is undefined if c is neither in the unsigned char range nor EOF.

I copied the added text from toupper.3.

In practice, calling them on out-of-range values - tested with recent glibc/gcc - is simply reading from random process memory - meaning, you either get some garbage, if the memory was readable, or a segmentation fault. See also:

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

https://stackoverflow.com/questions/65514890/glibcs-isalpha-function-and-the-en-us-utf-8-locale

This is already covered by the NOTES section, isn't it?

It's _mentioned_ there, correct - but not sure it's covered.

You're right. That's why I've sent the patch mentioning UB. What do you think about that one? (I now see that you like it).

Quoted 2 lines

It's also mentioned in toupper.3's NOTES.

I'll check that page to see if it needs some simplifying.

Quoted 2 lines

I think it's helpful to explicitly say that behavior is undefined in this case.

Yep.

Quoted 13 lines

If you feel like doing this inside NOTES, one way or another, ok for me.

Right now, NOTES says what you must do, but not what happens if you don't do that.

It also says that for the common case of using them on signed char, you should explicitly cast to unsigned char, first. It also tries to explain why this is necessary. The explanation explains why it's necessary for compliance with the standard, but not why it's a good thing more generally - latter is not explained,
and AFAICT from reading glibc sources, is not necessary - see e.g. this comment from ctype.h:

Quoted 5 lines

These point into arrays of 384, so they can be indexed by any unsigned char' value [0,255]; by EOF (-1); or by any signed char' value [-128,-1). ISO C requires that the ctype functions work for unsigned char' values and for EOF; we also support negative signed char' values for broken old programs.

Consider what happens with character 0xFF. If char is signed, it will be interpreted as -1 (i.e., EOF). We're lucky, because 0xFF is not a meaningful char, so probably all isXXX() functions return false for it, but it's slightly different from EOF semantically. If no locales give a meaning for 0xFF, maybe the cast can be removed from ISO C. I do something different: use -funsigned-char when compiling, so char is effectively unsigned char (except that pointers do not convert automatically).

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

The real reason why you should not call them on negative values other than EOF - casted to unsigned char or not - is simply that most likely this isn't what you meant to do. But that's not about compliance with the standard...

I guess the standard was cautious to not make 0xFF a useless char. If that's not an issue, I agree, and these functions could do the conversion internally.

To be clear, I'm talking about this:

$ cat iscntrl.c 
#include <ctype.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>

static const char str_bool[2][8] = { "false", "true" };

static inline const char *
strbool(bool x)

{
	return str_bool[!!x];
}

int
main(void)

{
	signed char    s = 0xFF;
	unsigned char  u = 0xFF;

	printf("iscntrl(-1):  %s\n", strbool(iscntrl(s)));
	printf("iscntrl(255): %s\n", strbool(iscntrl(u)));

	exit(EXIT_SUCCESS);
}
$ ./a.out 
iscntrl(-1):  false
iscntrl(255): false
Quoted 3 lines

Cheers,
Alex

Signature · 2 lines
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
#9 lore raw
In reply to Alejandro Colomar
Quoted 23 lines · Alejandro Colomar

Hi,

Quoted 19 lines · Yedidyah Bar David

Hi,

Quoted 16 lines · Alejandro Colomar

Hi Yedidyah,

Quoted 12 lines · Yedidyah Bar David

Clarify that the behavior of these functions is undefined if c is neither in the unsigned char range nor EOF.

I copied the added text from toupper.3.

In practice, calling them on out-of-range values - tested with recent glibc/gcc - is simply reading from random process memory - meaning, you either get some garbage, if the memory was readable, or a segmentation fault. See also:

https://stackoverflow.com/questions/65514890/glibcs-isalpha-function-and-the-en-us-utf-8-locale

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

This is already covered by the NOTES section, isn't it?

It's _mentioned_ there, correct - but not sure it's covered.

You're right. That's why I've sent the patch mentioning UB. What do you think about that one? (I now see that you like it).

Yes, I like it. Thanks!

Thanks also for the explanations/example.

Best regards,

Signature · 1 line
Didi
#11 lore raw

Hi Yedidyah,

Quoted 3 lines · Yedidyah Bar David

Hi,

Any update on this? Thanks!

Thanks for the ping. I've applied it now.

Cheers,
Alex

Quoted 2 lines

Best regards,

Signature · 2 lines
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5