Remove 'restrict' from 'nptr' in strtol(3)-like functions

Quoted 8 lines · Alejandro Colomar

ISO C specifies these APIs as accepting a restricted pointer in their first parameter:

$ stdc c99 strtol
long int strtol(const char *restrict nptr, char **restrict endptr, int base);
$ stdc c11 strtol
long int strtol(const char *restrict nptr, char **restrict endptr, int base);

Quoted 8 lines

However, it should be considered a defect in ISO C.  It's common to see code that aliases it:

	char str[] = "10 20";

	p = str;
	a = strtol(p, &p, 0);  // Let's ignore error handling for
	b = strtol(p, &p, 0);  // simplicity.

Why this is wrong?

During the execution of strtol() the only expression accessing the object "p" is *endptr. When the body of strtol() refers "nptr" it accesses a different object, not "p".

And if this is really wrong you should report it to WG14 before changing glibc.

[CC += linux-man@, since we're discussing an API documented there, and
the manual page would also need to be updated]

Hi Xi, Jakub,

Quoted 18 lines · Xi Ruoyao

Quoted 8 lines · Alejandro Colomar

ISO C specifies these APIs as accepting a restricted pointer in their first parameter:

$ stdc c99 strtol
long int strtol(const char *restrict nptr, char **restrict endptr, int base);
$ stdc c11 strtol
long int strtol(const char *restrict nptr, char **restrict endptr, int base);

Quoted 8 lines

However, it should be considered a defect in ISO C.  It's common to see code that aliases it:

	char str[] = "10 20";

	p = str;
	a = strtol(p, &p, 0);  // Let's ignore error handling for
	b = strtol(p, &p, 0);  // simplicity.

Quoted 5 lines

Why this is wrong?

During the execution of strtol() the only expression accessing the object "p" is *endptr. When the body of strtol() refers "nptr" it accesses a different object, not "p".

<http://port70.net/~nsz/c/c11/n1570.html#6.7.3p8>

Theoretically, 'restrict' is defined in terms of accesses, not just references, so it's fine for strtol(3) to hold two references of p in restrict pointers. That is, the following code is valid:

	int
	dumb(int *restrict a, int *restrict also_a)
	{
		// We don't access the objects
		return a == also_a;
	}

	int
	main(void)
	{
		int x = 3;

		return dumb(&x, &x);
	}

However, in practice that's dumb. The caller cannot know that the function doesn't access the object, so it must be cautious and enable -Wrestrict, which should be paranoid and do not allow passing references to the same object in different arguments, just in case the function decides to access to objects. Of course, GCC reports a diagnostic for the previous code:

	$ cc -Wall -Wextra dumb.c 
	dumb.c: In function ‘main’:
	dumb.c:13:21: warning: passing argument 1 to ‘restrict’-qualified parameter aliases with argument 2 [-Wrestrict]
	   13 |         return dumb(&x, &x);
	      |                     ^~  ~~

... even when there's no UB, since the object is not being accessed.

But when the thing gets non-trivial, as in strtol(3), GCC misses the -Wrestrict diagnostic, as reported in
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112833>.

Let's write a reproducer by altering the dumb.c program from above, with just another reference:

	int
	dumb2(int *restrict a, int *restrict *restrict ap)
	{
		// We don't access the objects
		return a == *ap;
	}

	int
	main(void)
	{
		int x = 3;
		int *xp = &x;

		return dumb2(&x, &xp);
	}

GCC doesn't report anything bad here, even though it's basically the same as the program from above:

	$ cc -Wall -Wextra dumb2.c
	$

Again, there's no UB, but we really want to be cautious and get a diagnostic as callers, just in case the callee decides to access the object; we never know.

So, GCC should be patched to report a warning in the program above. That will also cause strtol(3) to start issuing warnings in use cases like the one I showed.

Even further, let's try something really weird: inequality comparison, which is only defined for pointers to the same array object:

	int
	dumb3(int *restrict a, int *restrict *restrict ap)
	{
		// We don't access the objects
		return a > *ap;
	}

	int
	main(void)
	{
		int x = 3;
		int *xp = &x;

		return dumb3(&x, &xp);
	}

The behavior is still defined, since the obnjects are not accessed, but the compiler should really warn, on both sides:

(quoted diff) +0 -2

-  The caller is passing references to the same object in restricted
   parameters, which is a red flag.
 
-  The callee is comparing for inequality pointers that should, under
   normal circumstances, cause Undefined Behavior.

Quoted 2 lines

And if this is really wrong you should report it to WG14 before changing glibc.

Well, I don't know how to report that defect to WG14. If you help me, I'll be pleased to do so. Do they have a public mailing list or anything like that?

Quoted 26 lines

But when the thing gets non-trivial, as in strtol(3), GCC misses the -Wrestrict diagnostic, as reported in
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112833>.

Let's write a reproducer by altering the dumb.c program from above, with just another reference:

	int
	dumb2(int *restrict a, int *restrict *restrict ap)
	{
		// We don't access the objects
		return a == *ap;
	}

	int
	main(void)
	{
		int x = 3;
		int *xp = &x;

		return dumb2(&x, &xp);
	}

GCC doesn't report anything bad here, even though it's basically the same as the program from above:

Quoted 2 lines

	$ cc -Wall -Wextra dumb2.c
	$

strtol does have a "char * restrict * restrict" though, so the
situation is different. A "char **" and a "const char *"
shouldn't alias anyway.

Quoted 3 lines

Well, I don't know how to report that defect to WG14. If you help me, I'll be pleased to do so. Do they have a public mailing list or anything like that?

One can submit clarification or change requests:

https://www.open-std.org/jtc1/sc22/wg14/www/contributing.html

Hi Martin,

Quoted 26 lines · Martin Uecker

Quoted 26 lines

But when the thing gets non-trivial, as in strtol(3), GCC misses the -Wrestrict diagnostic, as reported in
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112833>.

Let's write a reproducer by altering the dumb.c program from above, with just another reference:

	int
	dumb2(int *restrict a, int *restrict *restrict ap)
	{
		// We don't access the objects
		return a == *ap;
	}

	int
	main(void)
	{
		int x = 3;
		int *xp = &x;

		return dumb2(&x, &xp);
	}

GCC doesn't report anything bad here, even though it's basically the same as the program from above:

Quoted 6 lines

Quoted 2 lines

	$ cc -Wall -Wextra dumb2.c
	$

strtol does have a "char * restrict * restrict" though, so the
situation is different. A "char **" and a "const char *"
shouldn't alias anyway.

Pedantically, it is actually declared as 'char **restrict' (the inner one is not declared as restrict, even though it will be restricted, since there are no other unrestricted pointers).

I've written functions that more closely resemble strtol(3), to show that in the end they all share the same issue regarding const-ness:

	$ cat d.c 
	int d(const char *restrict ca, char *restrict a)
	{
		return ca > a;
	}

	int main(void)
	{
		char x = 3;
		char *xp = &x;
		d(xp, xp);
	}
	$ cc -Wall -Wextra d.c 
	d.c: In function ‘main’:
	d.c:10:9: warning: passing argument 2 to ‘restrict’-qualified parameter aliases with argument 1 [-Wrestrict]
	   10 |         d(xp, xp);
	      |         ^

This trivial program causes a diagnostic. (Although I think the '>' should also cause a diagnostic!!)

Let's add a reference, to resemble strtol(3):

	$ cat d2.c 
	int d2(const char *restrict ca, char *restrict *restrict ap)
	{
		return ca > *ap;
	}

	int main(void)
	{
		char x = 3;
		char *xp = &x;
		d2(xp, &xp);
	}
	$ cc -Wall -Wextra d2.c 
	$ 

Why does this not cause a -Wrestrict diagnostic, while d.c does? How are these programs any different regarding pointer restrict-ness?

Quoted 8 lines

Quoted 4 lines

Well, I don't know how to report that defect to WG14. If you help me, I'll be pleased to do so. Do they have a public mailing list or anything like that?

One can submit clarification or change requests:

https://www.open-std.org/jtc1/sc22/wg14/www/contributing.html

Thanks! Will do. Anyway, I think this should be discussed in glibc/gcc in parallel, since it's clearly a missed diagnostic, and possibly a dangerous use of restrict if the compiler does any assumptions that shouldn't be done.

Have a lovely day!
Alex

Quoted 2 lines

I've written functions that more closely resemble strtol(3), to show that in the end they all share the same issue regarding const-ness:

Quoted 39 lines

	$ cat d.c 
	int d(const char *restrict ca, char *restrict a)
	{
		return ca > a;
	}

	int main(void)
	{
		char x = 3;
		char *xp = &x;
		d(xp, xp);
	}
	$ cc -Wall -Wextra d.c 
	d.c: In function ‘main’:
	d.c:10:9: warning: passing argument 2 to ‘restrict’-qualified parameter aliases with argument 1 [-Wrestrict]
	   10 |         d(xp, xp);
	      |         ^

This trivial program causes a diagnostic. (Although I think the '>' should also cause a diagnostic!!)

Let's add a reference, to resemble strtol(3):

	$ cat d2.c 
	int d2(const char *restrict ca, char *restrict *restrict ap)
	{
		return ca > *ap;
	}

	int main(void)
	{
		char x = 3;
		char *xp = &x;
		d2(xp, &xp);
	}
	$ cc -Wall -Wextra d2.c 
	$ 

Quoted 2 lines

Why does this not cause a -Wrestrict diagnostic, while d.c does? How are these programs any different regarding pointer restrict-ness?

It would require data flow anaylsis to produce the diagnostic while the first can simply be diagnosed by comparing arguments.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

Well, I don't know how to report that defect to WG14. If you help me, I'll be pleased to do so. Do they have a public mailing list or anything like that?

One can submit clarification or change requests:

https://www.open-std.org/jtc1/sc22/wg14/www/contributing.html

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

Quoted 4 lines · Alejandro Colomar

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

It **shouldn't**.  strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

See my previous reply.

Quoted 4 lines · Alejandro Colomar

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

It **shouldn't**. strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

See my previous reply.

Quoted 11 lines · Alejandro Colomar

Quoted 4 lines

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

It **shouldn't**. strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

Quoted 1 line

See my previous reply.

Right, is there a valid use of strtol where a warning would be justified?

Showing that you can contrive a case where a const char* restrict and char** restrict can alias doesn't mean there's a problem with strtol.

Am Freitag, dem 05.07.2024 um 17:24 +0100 schrieb Jonathan Wakely:

Quoted 13 lines · Xi Ruoyao via Gcc

Quoted 12 lines · Alejandro Colomar

Quoted 4 lines

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

It **shouldn't**. strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

See my previous reply.

Quoted 2 lines

Right, is there a valid use of strtol where a warning would be justified?

Quoted 2 lines

Showing that you can contrive a case where a const char* restrict and char** restrict can alias doesn't mean there's a problem with strtol.

I think his point is that a const char* restrict and something which is stored in a char* whose address is then passed can alias and there a warning would make sense in other situations.

But I am also not convinced removing restrict would be an improvement. It would make more sense to have an annotation that indicates that endptr is only used as output.

Hi,

Quoted 13 lines · Martin Uecker

Am Freitag, dem 05.07.2024 um 17:24 +0100 schrieb Jonathan Wakely:

Quoted 12 lines · Xi Ruoyao via Gcc

Quoted 11 lines · Alejandro Colomar

Quoted 4 lines

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

It **shouldn't**. strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

Quoted 1 line

Quoted 1 line

Quoted 1 line

See my previous reply.

That's not right. See my reply to yours, Xi. The restrict in

char **endptr

already prevents calls such as strtol(x, x, 0).

Quoted 19 lines · Alejandro Colomar

Hi,

Quoted 14 lines · Martin Uecker

Am Freitag, dem 05.07.2024 um 17:24 +0100 schrieb Jonathan Wakely:

Quoted 13 lines · Xi Ruoyao via Gcc

Quoted 12 lines · Alejandro Colomar

Quoted 4 lines

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

It **shouldn't**. strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

See my previous reply.

Quoted 5 lines

That's not right. See my reply to yours, Xi. The restrict in

char **endptr

already prevents calls such as strtol(x, x, 0).

That seems to contradict footnote 153 in C23.

Hi Jonathan,

Quoted 22 lines · Jonathan Wakely

Quoted 21 lines · Alejandro Colomar

Hi,

Quoted 14 lines · Martin Uecker

Am Freitag, dem 05.07.2024 um 17:24 +0100 schrieb Jonathan Wakely:

Quoted 13 lines · Xi Ruoyao via Gcc

Quoted 12 lines · Alejandro Colomar

Quoted 4 lines

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

It **shouldn't**. strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

See my previous reply.

That's not right. See my reply to yours, Xi. The restrict in

Quoted 5 lines

Quoted 3 lines

char **endptr

already prevents calls such as strtol(x, x, 0).

That seems to contradict footnote 153 in C23.

Did you mean a different footnote number?

Here's 153 in N3047:

153) An implementation can delay the choice of which integer type until all enumeration constants have been seen.

which seems completely unrelated.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

It **shouldn't**. strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

See my previous reply.

That's not right. See my reply to yours, Xi. The restrict in

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

char **endptr

already prevents calls such as strtol(x, x, 0).

That seems to contradict footnote 153 in C23.

Did you mean a different footnote number?

No.

Quoted 2 lines

Here's 153 in N3047:

That draft is nearly two years old.

Quoted 1 line

Quoted 4 lines

153) An implementation can delay the choice of which integer type until all enumeration constants have been seen.

which seems completely unrelated.

Because you're looking at a draft from nearly two years ago. Try N3220.

Quoted 2 lines · Alejandro Colomar

Here's 153 in N3047:

That draft is nearly two years old.

Quoted 1 line

Quoted 4 lines

153) An implementation can delay the choice of which integer type until all enumeration constants have been seen.

which seems completely unrelated.

Because you're looking at a draft from nearly two years ago. Try N3220.

That is 6.7.3.1p3:

In what follows, a pointer expression E is said to be based on object P
if (at some sequence point in the execution of B prior to the evaluation of E) modifying P to point to a copy of the array object into which it formerly pointed would change the value of E.153) Note that "based" is defined only for expressions with pointer types.

Footnote 153) In other words, E depends on the value of P itself rather
than on the value of an object referenced indirectly through P. For example, if identifier p has type (int **restrict), then the pointer expressions p and p+1 are based on the restricted pointer object designated by p, but the pointer expressions *p and p[1] are not.

Which would be the same paragraph of the same section on N3047, but footnote number 168.

Quoted 7 lines

Quoted 3 lines

Showing that you can contrive a case where a const char* restrict and char** restrict can alias doesn't mean there's a problem with strtol.

I think his point is that a const char* restrict and something which is stored in a char* whose address is then passed can alias and there a warning would make sense in other situations.

Indeed.

Quoted 3 lines

But I am also not convinced removing restrict would be an improvement. It would make more sense to have an annotation that indicates that endptr is only used as output.

What is the benefit of keeping restrict there? It doesn't provide any benefits, AFAICS.

Am Freitag, dem 05.07.2024 um 21:28 +0200 schrieb Alejandro Colomar:

...

Quoted 8 lines

Quoted 7 lines

Quoted 3 lines

Showing that you can contrive a case where a const char* restrict and char** restrict can alias doesn't mean there's a problem with strtol.

I think his point is that a const char* restrict and something which is stored in a char* whose address is then passed can alias and there a warning would make sense in other situations.

Quoted 8 lines

Indeed.

Quoted 3 lines

But I am also not convinced removing restrict would be an improvement. It would make more sense to have an annotation that indicates that endptr is only used as output.

What is the benefit of keeping restrict there? It doesn't provide any benefits, AFAICS.

Not really I think. I am generally not a fan of restrict. IMHO it is misdesigned and I would like to see it replaced with something better. But I also not convinced it really helps to remove it here.

If we marked endptr as "write_only" (which it might already be) then a future warning mechanism for -Wrestrict could ignore the content of *endptr.

Quoted 26 lines · Martin Uecker

Am Freitag, dem 05.07.2024 um 21:28 +0200 schrieb Alejandro Colomar:

...

Quoted 8 lines

Quoted 7 lines

Quoted 3 lines

Showing that you can contrive a case where a const char* restrict and char** restrict can alias doesn't mean there's a problem with strtol.

I think his point is that a const char* restrict and something which is stored in a char* whose address is then passed can alias and there a warning would make sense in other situations.

Quoted 8 lines

Indeed.

Quoted 3 lines

But I am also not convinced removing restrict would be an improvement. It would make more sense to have an annotation that indicates that endptr is only used as output.

What is the benefit of keeping restrict there? It doesn't provide any benefits, AFAICS.

Not really I think. I am generally not a fan of restrict. IMHO it is misdesigned and I would like to see it replaced with something better. But I also not convinced it really helps to remove it here.

Quoted 3 lines

If we marked endptr as "write_only" (which it might already be) then a future warning mechanism for -Wrestrict could ignore the content of *endptr.

That seems more useful. Add semantic information instead of taking it away.

If the concern is a hypothetical future compiler warning that would give false positives for perfectly valid uses of strtol, then the problem is the compiler warning, not strtol. If additional information can be added to avoid the false positives (and also possibly optimize the code better), then that wouldn't require a change to the standard motivated by a hypothetical compiler warning.

Quoted 3 lines

If we marked endptr as "write_only" (which it might already be) then a future warning mechanism for -Wrestrict could ignore the content of *endptr.

That seems more useful. Add semantic information instead of taking it away. If the concern is a hypothetical future compiler warning that would give false positives for perfectly valid uses of strtol, then the problem is the compiler warning, not strtol. If additional information can be added to avoid the false positives (and also possibly optimize the code better), then that wouldn't require a change to the standard motivated by a hypothetical compiler warning.

Quoted 5 lines · Jonathan Wakely

Quoted 3 lines

If we marked endptr as "write_only" (which it might already be) then a future warning mechanism for -Wrestrict could ignore the content of *endptr.

Quoted 7 lines

That seems more useful. Add semantic information instead of taking it away. If the concern is a hypothetical future compiler warning that would give false positives for perfectly valid uses of strtol, then the problem is the compiler warning, not strtol. If additional information can be added to avoid the false positives (and also possibly optimize the code better), then that wouldn't require a change to the standard motivated by a hypothetical compiler warning.

Let me be a little bit sarcastic.

If so, let's take down -Wrestrict at all, because it triggers false positives at the same rate. How is it even in -Wall and not just -Wextra?

Here's a false positive:

        $ cat d.c
        int is_same_pointer(const char *restrict ca, char *restrict a)
        {
                return ca == a;

        }

        int main(void)
        {
                char x = 3;
                char *xp = &x;
                is_same_pointer(xp, xp);
        }
        $ cc -Wall d.c
        d.c: In function ‘main’:
        d.c:10:9: warning: passing argument 2 to ‘restrict’-qualified parameter aliases with argument 1 [-Wrestrict]
           10 |         d(xp, xp);
              |         ^~~~~~~~~~~~~~~

It's impossible to know if a use of restrict causes UB without reading both the source code of the caller and the callee, so except for -fanalyzer, it's impossible to diagnose something with certainty.

So, it's certainly not something we want in -Wall.

Or should I remove the 'restrict' qualifier from that function, loosing "precious" semantic information, just because the compiler doesn't like it?

Cheers,
Alex

--
<https://www.alejandro-colomar.es/>

Quoted 21 lines · Alejandro Colomar

Quoted 5 lines · Jonathan Wakely

Quoted 3 lines

If we marked endptr as "write_only" (which it might already be) then a future warning mechanism for -Wrestrict could ignore the content of *endptr.

Quoted 7 lines

That seems more useful. Add semantic information instead of taking it away. If the concern is a hypothetical future compiler warning that would give false positives for perfectly valid uses of strtol, then the problem is the compiler warning, not strtol. If additional information can be added to avoid the false positives (and also possibly optimize the code better), then that wouldn't require a change to the standard motivated by a hypothetical compiler warning.

Let me be a little bit sarcastic.

If so, let's take down -Wrestrict at all, because it triggers false positives at the same rate. How is it even in -Wall and not just -Wextra?

Quoted 6 lines

Here's a false positive:

        $ cat d.c
        int is_same_pointer(const char *restrict ca, char *restrict a)
        {
                return ca == a;

This is a strawman argument, because all your example functions have been pretty useless and/or not good uses of restrict.

Yes, if you put restrict on functions where you don't ever access any objects through the pointers, the restrict qualifiers are misleading

and so compilers might give bad warnings for your bad API.

Hi Jonathan,

Quoted 29 lines · Jonathan Wakely

Quoted 21 lines · Alejandro Colomar

Quoted 5 lines · Jonathan Wakely

Quoted 3 lines

If we marked endptr as "write_only" (which it might already be) then a future warning mechanism for -Wrestrict could ignore the content of *endptr.

Quoted 7 lines

That seems more useful. Add semantic information instead of taking it away. If the concern is a hypothetical future compiler warning that would give false positives for perfectly valid uses of strtol, then the problem is the compiler warning, not strtol. If additional information can be added to avoid the false positives (and also possibly optimize the code better), then that wouldn't require a change to the standard motivated by a hypothetical compiler warning.

Let me be a little bit sarcastic.

If so, let's take down -Wrestrict at all, because it triggers false positives at the same rate. How is it even in -Wall and not just -Wextra?

Quoted 6 lines

Here's a false positive:

        $ cat d.c
        int is_same_pointer(const char *restrict ca, char *restrict a)
        {
                return ca == a;

Quoted 5 lines

This is a strawman argument, because all your example functions have been pretty useless and/or not good uses of restrict.

Yes, if you put restrict on functions where you don't ever access any objects through the pointers, the restrict qualifiers are misleading

That's precisely the case with strtol(3): it doesn't access any objects through *endptr, and so that pointer need not be restrict.

Then, nptr is a read-only pointer, so is doesn't matter either if it's accessed or not.

Let's say we add as many attributes as possible to strtol(3):

[[gnu::access(read_only, 1)]]
[[gnu::access(write_only, 1)]]

[[gnu::leaf]]
[[gnu::nothrow]]
[[gnu::null_terminated_string_arg(1)]]
// [[gnu::access(none, *1)]]

long
alx_strtol(const char *nptr, char **_Nullable restrict endp, int base);

Let's say we could mark *endptr as a 'access(none)' pointer, since it's not accessed. Let's say we do that with [[gnu::access(none, *1)]].

Then, do you think the information of that prototype is any different than a prototype with restrict on the remaining pointers?

[[gnu::access(read_only, 1)]]
[[gnu::access(write_only, 1)]]

[[gnu::leaf]]
[[gnu::nothrow]]
[[gnu::null_terminated_string_arg(1)]]
// [[gnu::access(none, *1)]]

long
alx_strtol(const char *restrict nptr,

           char *restrict *_Nullable restrict endp, int base);

I don't think so. Since *endptr is access(none), it certainly cannot access nptr, and thus the qualifier on nptr is superfluous.

And even without the hypothetical [[gnu::access(none, *1)]]:

(quoted diff) +0 -2

-  The callee doesn't care about restrict, because it doesn't access
   any objects via *endptr, so it certainly knows that nptr can be read
   without any concerns about optimization.
 
-  The caller can't know if strtol(3) accesses *endptr, or nptr, and so
   it can't optimize.  Unless it passes an uninitialized value in
   *endptr, which means the caller knows for sure that nptr won't be
   written, regardless of restrict on it or not.

Please, describe what's the information you think is being added by having restrict on nptr, on how it would be lost if we remove it.

Cheers,
Alex

Quoted 1 line

and so compilers might give bad warnings for your bad API.

Signature · 1 line

<https://www.alejandro-colomar.es/>

That's precisely the case with strtol(3): it doesn't access any objects through *endptr, and so that pointer need not be restrict.

Then, nptr is a read-only pointer, so is doesn't matter either if it's accessed or not.

Quoted 3 lines · Alejandro Colomar

That's precisely the case with strtol(3): it doesn't access any objects through *endptr, and so that pointer need not be restrict.

Quoted 2 lines

Then, nptr is a read-only pointer, so is doesn't matter either if it's accessed or not.

Restrict allows to reorder any writes to other objects with an read from nptr then. In strtol at least errno can be written, and depending on the implementation of locale things there may be more.

TBAA does not help here because char aliases with anything.

Quoted 5 lines

[[gnu::access(read_only, 1)]]
[[gnu::access(write_only, 2)]]
[[gnu::leaf]]
[[gnu::nothrow]]
[[gnu::null_terminated_string_arg(1)]]

BTW among your list:

Quoted 5 lines

Quoted 5 lines

[[gnu::access(read_only, 1)]]
[[gnu::access(write_only, 2)]]
[[gnu::leaf]]
[[gnu::nothrow]]
[[gnu::null_terminated_string_arg(1)]]

IMO we should add these access attributes, they'll definitely help the optimization (like, optimize away the initialization of a pointer).

We already have __THROW which expands to nothrow and leaf.

I'm not sure if null_terminated_string_arg is correct: is the following invalid or not?

char p[] = {'1', ')'};
char *q;
strtol(p, &q, 10);
assert(q == &p[1]);

If this is invalid we should have null_terminated_string_arg so at least we'll get a warning against this.

Quoted 5 lines · Alejandro Colomar

That's precisely the case with strtol(3): it doesn't access any objects through *endptr, and so that pointer need not be restrict.

Then, nptr is a read-only pointer, so is doesn't matter either if it's accessed or not.

Restrict allows to reorder any writes to other objects with an read from nptr then. In strtol at least errno can be written, and depending on the implementation of locale things there may be more.

TBAA does not help here because char aliases with anything.

Hi Xi,

Quoted 10 lines · Xi Ruoyao

Quoted 5 lines · Alejandro Colomar

That's precisely the case with strtol(3): it doesn't access any objects through *endptr, and so that pointer need not be restrict.

Then, nptr is a read-only pointer, so is doesn't matter either if it's accessed or not.

Restrict allows to reorder any writes to other objects with an read from nptr then. In strtol at least errno can be written, and depending on the implementation of locale things there may be more.

This does not apply here, I think. Let's include errno in the list of objects that strtol(3) takes, and list their access modes:

- errno access(read_write)

In the callee:
~~~~~~~~~~~~~~

The access modes are known by the callee, because of course it knows what it does, so even without the attributes, it knows that.

strtol(3) cannot write to errno until it has parsed the string. And once it knows it has failed (so wants to set errno), it has no reasons to read nptr again. Thus, even without knowing if 'errno' and 'nptr' are the same thing, there's nothing that could be optimized.

*endptr is access(none), so it is implicitly restricted even without specifying restrict on it; the callee couldn't care less about it.

endptr is 'restrict', so it can be treated separately from the rest.

In the caller:
~~~~~~~~~~~~~~

We can't specify the access mode of *endptr nor errno, so the caller must assume they are read_write.

endptr is 'restrict', but this is only useful for having warnings against dumb things such as strtol(x, x, 0). Other than that, the caller knows what it has passed in endptr, so it knows it's a different thing.

The caller also knows that it hasn't passed errno as nptr.

Then we must make a distinction in what the caller passes in *endptr:

*endptr is uninitialized:
~~~~~~~~~~~~~~~~~~~~~~~~~

The caller knows that nptr is restricted even without the qualifier, since all other objects are either restricted, or known to be different.

*endptr points to the same thing as nptr:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Regardless of the 'restrict' qualifier being specified or not, the caller has no way to determine if the callee accesses the object via nptr or via *endptr, so it must assume the worst case: *endptr; and so it must assume it could have written to it (since *endptr is non-const --and even if it were const, as you said, it means nothing--).

So, even considering errno in the game, I don't see any difference if we specify nptr to be restrict or not.

Thanks for the feedback! I'll incorporate the discussion about errno in the paper for WG14.

Have a lovely day!
Alex

Quoted 2 lines

TBAA does not help here because char aliases with anything.

Signature · 1 line

<https://www.alejandro-colomar.es/>

Quoted 11 lines · Xi Ruoyao via Gcc

Quoted 4 lines · Alejandro Colomar

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

It **shouldn't**. strtol will only violate restrict if it's wrongly implemented, or something dumb is done like "strtol((const char*) &p, &p, 0)".

Quoted 1 line

See my previous reply.

Right, is there a valid use of strtol where a warning would be justified?

Showing that you can contrive a case where a const char* restrict and char** restrict can alias doesn't mean there's a problem with strtol.

Hi Martin,

Quoted 2 lines · Martin Uecker

Quoted 2 lines

I've written functions that more closely resemble strtol(3), to show that in the end they all share the same issue regarding const-ness:

(Above I meant s/const/restrict/)

Quoted 39 lines

Quoted 39 lines

	$ cat d.c 
	int d(const char *restrict ca, char *restrict a)
	{
		return ca > a;
	}

	int main(void)
	{
		char x = 3;
		char *xp = &x;
		d(xp, xp);
	}
	$ cc -Wall -Wextra d.c 
	d.c: In function ‘main’:
	d.c:10:9: warning: passing argument 2 to ‘restrict’-qualified parameter aliases with argument 1 [-Wrestrict]
	   10 |         d(xp, xp);
	      |         ^

This trivial program causes a diagnostic. (Although I think the '>' should also cause a diagnostic!!)

Let's add a reference, to resemble strtol(3):

	$ cat d2.c 
	int d2(const char *restrict ca, char *restrict *restrict ap)
	{
		return ca > *ap;
	}

	int main(void)
	{
		char x = 3;
		char *xp = &x;
		d2(xp, &xp);
	}
	$ cc -Wall -Wextra d2.c 
	$ 

Quoted 5 lines

Quoted 2 lines

Why does this not cause a -Wrestrict diagnostic, while d.c does? How are these programs any different regarding pointer restrict-ness?

It would require data flow anaylsis to produce the diagnostic while the first can simply be diagnosed by comparing arguments.

Agree. It seems like a task for -fanalyzer.

	$ cc -Wall -Wextra -fanalyzer -fuse-linker-plugin -flto d2.c
	$

I'm unable to trigger that at all. It's probably not implemented, I guess. I've updated the bug report
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112833> to change the component to 'analyzer'.

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

Well, I don't know how to report that defect to WG14. If you help me, I'll be pleased to do so. Do they have a public mailing list or anything like that?

One can submit clarification or change requests:

https://www.open-std.org/jtc1/sc22/wg14/www/contributing.html

P.S.:

I've sent a mail to UNE (the Spanish National Body for ISO), and asked them about joining WG14. Let's see what they say.

P.S. 2:

I'm also preparing a paper; would you mind championing it if I'm not yet able to do it when it's ready?

P.S. 3:

Do you know of any Spanish member of WG14? Maybe I can talk with them to have more information about how they work.

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

Quoted 4 lines · Alejandro Colomar via Gcc

At least, I hope there's consensus that while current GCC doesn't warn about this, ideally it should, which means it should warn for valid uses of strtol(3), which means strtol(3) should be fixed, in all of ISO, POSIX, and glibc.

I'm not convinced. It doesn't look like anybody else is convinced. I wouldn't call that consensus.

Hi Martin,

Quoted 2 lines · Martin Uecker

Quoted 2 lines

But when the thing gets non-trivial, as in strtol(3), GCC misses the -Wrestrict diagnostic, as reported in

Quoted 3 lines

strtol does have  a "char * restrict * restrict" though, so the situation is different.   A "char **" and a "const char *" shouldn't alias anyway.

Pedantically, it is actually declared as 'char **restrict' (the inner one is not declared as restrict, even though it will be restricted, since there are no other unrestricted pointers).

Quoted 7 lines · Alejandro Colomar

Quoted 3 lines

strtol does have  a "char * restrict * restrict" though, so the situation is different.   A "char **" and a "const char *" shouldn't alias anyway.

Pedantically, it is actually declared as 'char **restrict' (the inner one is not declared as restrict, even though it will be restricted, since there are no other unrestricted pointers).

So how's the following implementation of strtol (10-based, no negative number handling, no overflow handling, ASCII-only) wrong?

long int my_strtol(const char *restrict nptr, char **restrict endptr) {
long ret = 0;

  while (isdigit(*nptr))
    ret = ret * 10 + (*nptr++ - '0');

*endptr = (char *)nptr;
return ret;
}

There's no dumb thing, there's no restrict violation (unless it's called in a stupid way, see below), and there **shouldn't** be a -Wrestrict warning.

If you do

char *x = NULL;
strtol((char *)&x, &x, 10);

it'll violate restrict. Nobody sane should write this, and it's warned anyway:

t.c: In function 'main':
t.c:6:28: warning: passing argument 2 to 'restrict'-qualified parameter aliases with argument 1 [-Wrestrict]

    6 |         strtol((char *)&x, &x, 10);
      |                ~~~~~~~~~~  ^~

Signature · 2 lines

Xi Ruoyao <[email protected]>
School of Aerospace Science and Technology, Xidian University

Hi Xi,

Quoted 23 lines · Xi Ruoyao

Quoted 7 lines · Alejandro Colomar

Quoted 3 lines

strtol does have  a "char * restrict * restrict" though, so the situation is different.   A "char **" and a "const char *" shouldn't alias anyway.

Pedantically, it is actually declared as 'char **restrict' (the inner one is not declared as restrict, even though it will be restricted, since there are no other unrestricted pointers).

So how's the following implementation of strtol (10-based, no negative number handling, no overflow handling, ASCII-only) wrong?

long int my_strtol(const char *restrict nptr, char **restrict endptr) {
long ret = 0;

  while (isdigit(*nptr))
    ret = ret * 10 + (*nptr++ - '0');

*endptr = (char *)nptr;
return ret;
}

Quoted 8 lines

There's no dumb thing, there's no restrict violation (unless it's called in a stupid way, see below), and there **shouldn't** be a -Wrestrict warning.

If you do

char *x = NULL;
strtol((char *)&x, &x, 10);

The restrict in char **restrict endptr already protects you from this. You don't need to make the first parameter also restricted. See:

	$ cat r.c 
	long alx_strtol(const char *nptr, char **restrict endp);

	int main(void)
	{
		char x = 3;
		char *xp = &x;

		alx_strtol(xp, &xp);  // Fine.
		alx_strtol(xp, (char **) xp);  // Bug.
		alx_strtol((char *) &xp, &xp);  // Bug.
	}
	$ cc -Wall -Wextra -S r.c 
	r.c: In function ‘main’:
	r.c:9:24: warning: passing argument 2 to ‘restrict’-qualified parameter aliases with argument 1 [-Wrestrict]
	    9 |         alx_strtol(xp, (char **) xp);  // Bug.
	      |                        ^~~~~~~~~~~~
	r.c:10:34: warning: passing argument 2 to ‘restrict’-qualified parameter aliases with argument 1 [-Wrestrict]
	   10 |         alx_strtol((char *) &xp, &xp);  // Bug.
	      |                    ~~~~~~~~~~~~  ^~~

Using my proposed prototype wouldn't case any warnings with a powerful -fanalyzer that would be able to emit diagnostics with the current prototype.

In strtol(3), there are 3 pointers:

(quoted diff) +0 -3

-  nptr
-  *endptr
-  endptr

The first two should be allowed to alias each other (and at the end of the call, they definitely alias each other). It is only the third one which must not alias any of the other, which is why my patch (v2) keeps

that restrict, but removes the other one.

Does that make sense?

Cheers,
Alex

Quoted 13 lines

it'll violate restrict. Nobody sane should write this, and it's warned anyway:

t.c: In function 'main':
t.c:6:28: warning: passing argument 2 to 'restrict'-qualified parameter aliases with argument 1 [-Wrestrict]

    6 |         strtol((char *)&x, &x, 10);
      |                ~~~~~~~~~~  ^~

Signature · 2 lines

Xi Ruoyao <[email protected]>
School of Aerospace Science and Technology, Xidian University

Signature · 1 line

<https://www.alejandro-colomar.es/>

Alex,

Your document number is below:

n3294 - strtol(3) et al. shouldn't have a restricted first parameter

Please return the updated document with this number

Best regards,

Dan

Technical Director - Enabling Mission Capability at Scale Principal Member of the Technical Staff
Software Engineering Institute
Carnegie Mellon University
4500 Fifth Avenue
Pittsburgh, PA 15213
WORK: 412-268-7197
CELL: 412-427-4606

-----Original Message-----
From: Alejandro Colomar <[email protected]>
Sent: Friday, July 5, 2024 3:42 PM
To: [email protected]
Cc: Martin Uecker <[email protected]>; Jonathan Wakely <[email protected]>; Xi Ruoyao <[email protected]>; Jakub Jelinek <[email protected]>; [email protected]; [email protected]; Paul Eggert <[email protected]>; [email protected]; LIU Hao <[email protected]>; Richard Earnshaw <[email protected]>; Sam James <[email protected]>
Subject: [WG14] Request for document number; strtol restrictness

Hi,

I have a paper for removing restrict from the first parameter of strtol(3) et al. The title is

strtol(3) et al. should’t have a restricted first parameter.

If it helps, I already have a draft of the paper, which I attach (both the PDF, and the man(7) source).

Cheers,
Alex

--
<https://www.alejandro-colomar.es/>

Alex,

Your document number is below:

n3294 - strtol(3) et al. shouldn't have a restricted first parameter

Please return the updated document with this number

Best regards,

Dan

Technical Director - Enabling Mission Capability at Scale Principal Member of the Technical Staff Software Engineering Institute Carnegie Mellon University
4500 Fifth Avenue
Pittsburgh, PA 15213
WORK: 412-268-7197
CELL: 412-427-4606

-----Original Message-----
From: Alejandro Colomar <[email protected]>
Sent: Friday, July 5, 2024 3:42 PM
To: [email protected]
Cc: Martin Uecker <[email protected]>; Jonathan Wakely <[email protected]>; Xi Ruoyao <[email protected]>; Jakub Jelinek <[email protected]>; [email protected]; [email protected]; Paul Eggert <[email protected]>; [email protected]; LIU Hao <[email protected]>; Richard Earnshaw <[email protected]>; Sam James <[email protected]>
Subject: [WG14] Request for document number; strtol restrictness

Hi,

I have a paper for removing restrict from the first parameter of strtol(3) et al. The title is

strtol(3) et al. should’t have a restricted first parameter.

If it helps, I already have a draft of the paper, which I attach (both the PDF, and the man(7) source).

Cheers,
Alex

--
<https://www.alejandro-colomar.es/>

Hi,

I've incorporated feedback, and here's a new revision, let's call it v0.2, of the draft for a WG14 paper. I've attached the man(7) source, and the generated PDF.

Cheers,
Alex

Hi Alejandro,

if in caller it is known that endptr has access mode "write_only" then it can conclude that the content of *endptr has access mode "none", couldn't it?

You also need to discuss backwards compatibility. Changing the type of those functions can break valid programs.

You would
need to make a case that this is unlikely to affect any real world program.

Hi Martin,

Quoted 6 lines · Martin Uecker

Hi Alejandro,

if in caller it is known that endptr has access mode "write_only" then it can conclude that the content of *endptr has access mode "none", couldn't it?

Hmmmm. I think you're correct. I'll incorporate that and see how it affects the caller.

At first glance, I think it would result in

nptr access(read_only) alias *endptr
endptr access(write_only) unique
errno access(read_write) unique
*endptr access(none) alias nptr

Which is actually having perfect information, regardless of 'restrict' on nptr. :-)

Quoted 3 lines

You also need to discuss backwards compatibility. Changing the type of those functions can break valid programs.

I might be forgetting about other possibilities, but the only one I had in mind that could break API would be function pointers. However, a small experiment seems to say it doesn't:

	$ cat strtolp.c 
	#include <stdlib.h>

	long
	alx_strtol(const char *nptr, char **restrict endptr, int base)
	{
		return strtol(nptr, endptr, base);
	}

	typedef long (*strtolp_t)(const char *restrict nptr,
				  char **restrict endptr, int base);
	typedef long (*strtolpnr_t)(const char *nptr,
				   char **restrict endptr, int base);

	int
	main(void)
	{
		[[maybe_unused]] strtolp_t    a = &strtol;
		[[maybe_unused]] strtolpnr_t  b = &strtol;
		[[maybe_unused]] strtolp_t    c = &alx_strtol;
		[[maybe_unused]] strtolpnr_t  d = &alx_strtol;
	}

	$ cc -Wall -Wextra strtolp.c 
	$

Anyway, I'll say that it doesn't seem to break API.

Quoted 3 lines

You would
need to make a case that this is unlikely to affect any real world program.

If you have something else in mind that could break API, please let me know, and I'll add it to the experiments.

Thanks!

Have a lovely day!
Alex

Am Sonntag, dem 07.07.2024 um 13:07 +0200 schrieb Alejandro Colomar via Gcc:

Quoted 2 lines

Which is actually having perfect information, regardless of 'restrict' on nptr. :-)

Yes, but my point is that even with "restrict" a smarter compiler could then also be smart enough not to warn even when *endptr aliases nptr.

Quoted 6 lines

Quoted 2 lines

You also need to discuss backwards compatibility. Changing the type of those functions can break valid programs.

I might be forgetting about other possibilities, but the only one I had in mind that could break API would be function pointers. However, a small experiment seems to say it doesn't:

Right, the outermost qualifiers are ignored, so this is not a compatibility problem. So I think this is not an issue, but it is worth pointing it out.

Martin

I've incorporated feedback, and here's a new revision, let's call it v0.2, of the draft for a WG14 paper.

Quoted 2 lines · Alejandro Colomar

I've incorporated feedback, and here's a new revision, let's call it v0.2, of the draft for a WG14 paper.

Although I have not followed the email discussion closely, I read v0.2 and think that as stated there is little chance that its proposal will be accepted.

Fundamentally the proposal is trying to say that there are two styles X and Y for declaring strtol and similar functions, and that although both styles are correct in some sense, style Y is better than style X. However, the advantages of Y are not clearly stated and the advantages of style X over Y are not admitted, so the proposal is not making the case clearly and fairly.

One other thing: to maximize the chance of a proposal being accepted, please tailor it for its expected readership. The C committee is expert on ‘restrict’, so don’t try to define ‘restrict’ in your own way. Unless merely repeating the language of the standard, any definition given for ‘restrict’ is likely to cause the committee to quibble with the restatement of the standard wording. (It is OK to mention some corollaries of the standard definition, so long as the corollaries are not immediately obvious.)

Here are some comments about the proposal. At the start these comments are detailed; towards the end, as I could see the direction the proposal was headed and was convinced it wouldn’t be accepted as stated, the comments are less detailed.

"The API may copy"

One normally doesn’t think of the application programming interface as copying. Please replace the phrase “the API” with “the caller” or “the callee” as appropriate. (Although ‘restrict’ can be used in places other than function parameters, I don’t think the proposal is concerned about those cases and so it doesn’t need to go into that.)

"To avoid violations of for example C11::6.5.16.1p3,"

Code that violates C11::6.5.16.1p3 will do so regardless of whether ‘restrict’ is present. I would not mention C11::6.5.16.1p3 as it’s a red herring. Fundamentally, ‘restrict’ is not about the consequences of caching when one does overlapping moves; it’s about caching in a more general sense.

“As long as an object is only accessed via one restricted pointer, other restricted pointers are allowed to point to the same object.”

“only accessed” → “accessed only”

“This is less strict than I think it should be, but this proposal doesn’t attempt to change that definition.”

I would omit this sentence and all similar sentences. Don’t distract the reader with other potential proposals. The proposal as it stands is complicated enough.

“return ca > a;”
“return ca > *ap;”

I fail to understand why these examples are present. It’s not simply that nobody writes code like that: the examples are not on point. I would remove the entire programs containing them, along with the sections that discuss them. When writing to the C committee one can assume the reader is expert in ‘restrict’, there is no need for examples such as these.

“strtol(3) accepts 4 objects via pointer parameters and global variables.”

Omit the “(3)”, here and elsewhere, as the audience is the C standard committee.

“accepts” is a strange word to use here: normally one says “accepts” to talk about parameters, not global variables.

Also, “global variables” is not
right here. The C standard allows strtol, for example, to read and write an internal static cache. (Yes, that would be weird, but it’s allowed.)

I
suggest rephrasing this sentence to talk about accessing, not accepting.

“endptr access(write_only) ... *endptr access(none)”

This is true for glibc, but it’s not necessarily true for all conforming strtol implementations. If endptr is non-null, a conforming strtol implementation can both read and write *endptr;

it can also both read and
write **endptr. (Although it would need to write before reading, reading is allowed.)

“This qualifier helps catch obvious bugs such as strtol(p, p, 0) and strtol(&p, &p, 0) .”

No it doesn’t. Ordinary type checking catches those obvious bugs, and ‘restrict’ provides no additional help there. Please complicate the examples to make the point more clearly.

“The caller knows that errno doesn’t alias any of the function arguments.”

Only because all args are declared with ‘restrict’. So if the proposal is accepted, the caller doesn’t necessarily know that.

“The callee knows that *endptr is not accessed.”

This is true for glibc, but not necessarily true for every conforming strtol implementation.

“It might seem that it’s a problem that the callee doesn’t know if nptr can alias errno or not. However, the callee will not write to the latter directly until it knows it has failed,”

Again this is true for glibc, but not necessarily true for every conforming strtol implementation.

To my mind this is the most serious objection. The current standard prohibits calls like strtol((char *) &errno, 0, 0). The proposal would relax the standard to allow such calls. In other words, the proposal would constrain implementations to support such calls.

Why is this change worth
making? Real-world programs do not make calls like that.

“But nothing prohibits those internal helper functions to specify that nptr is restrict and thus distinct from errno.”

Although true, it’s also the case that the C standard does not *require* internal helper functions to use ‘restrict’. All that matters is the accesses. So I’m not sure what the point of this statement is.

“m = strtol(p, &p, 0); An analyzer more powerful than the current ones could extend the current -Wrestrict diagnostic to also diagnose this case.”

Why would an analyzer want to do that? This case is a perfectly normal thing to do and it has well-defined behavior.

“To prevent triggering diagnostics in a powerful analyzer that would be smart enough to diagnose the example function g(), the prototype of strtol(3) should be changed to ‘long int strtol(const char *nptr, char **restrict endptr, int base);’”

Sorry, but the case has not been made to make any such change to strtol’s prototype. On the contrary, what I’m mostly gathering from the discussion is that ‘restrict’ can be confusing, which is not news.

n3220 §6.7.4.2 examples 5 through 7 demonstrate that the C committee has thought through the points you’re making. (These examples were not present in C11.) This may help to explain why the standard specifies strtol with ‘restrict’ on both arguments.

Quoted 3 lines · Paul Eggert

Also, “global variables” is not
right here. The C standard allows strtol, for example, to read and write an internal static cache. (Yes, that would be weird, but it’s allowed.)

That's not part of the API. A user must not access internal static cache

Quoted 5 lines

“endptr access(write_only) ... *endptr access(none)”

This is true for glibc, but it’s not necessarily true for all conforming strtol implementations. If endptr is non-null, a conforming strtol implementation can both read and write *endptr;

It can't, I think. It's perfectly valid to pass an uninitialized endptr, which means the callee must not read the original value.

Here, we need to consider two separate objects. The object pointed-to by *endptr _before_ the object pointed to by endptr is written to, and the object pointed-to by *endptr _after_ the object pointed to by endptr is written to.

Quoted 4 lines

“The caller knows that errno doesn’t alias any of the function arguments.”

Only because all args are declared with ‘restrict’. So if the proposal is accepted, the caller doesn’t necessarily know that.

Not really. The caller has created the string (or has received it via a restricted pointer)

Quoted 4 lines

“The callee knows that *endptr is not accessed.”

This is true for glibc, but not necessarily true for every conforming strtol implementation.

The original *endptr may be uninitialized, and so must not be accessed.

Quoted 6 lines

“It might seem that it’s a problem that the callee doesn’t know if nptr can alias errno or not. However, the callee will not write to the latter directly until it knows it has failed,”

Again this is true for glibc, but not necessarily true for every conforming strtol implementation.

An implementation is free to set errno = EDEADLK in the middle of it, as long as it later removes that. However, I don't see how it would make any sense.

Let's find
an ISO C function that accepts a non-restrict string:

	int system(const char *string);

Does ISO C constrain implementations to support system((char *)&errno)? I don't think so. Maybe it does implicitly because of a defect in the wording, but even then it's widely understood that it doesn't.

Quoted 2 lines

Why is this change worth
making? Real-world programs do not make calls like that.

Because it makes analysis of 'restrict' more consistent. The obvious improvement of GCC's analyzer to catch restrict violations will trigger false positives in normal uses of strtol(3).

Quoted 5 lines

“m = strtol(p, &p, 0); An analyzer more powerful than the current ones could extend the current -Wrestrict diagnostic to also diagnose this case.”

Why would an analyzer want to do that? This case is a perfectly normal thing to do and it has well-defined behavior.

Because without an analyzer, restrict cannot emit many useful diagnostics. It's a qualifier that's all about data flow analysis, and normal diagnostics aren't able to do that.

A qualifier that enables optimizations but doesn't enable diagnostics is quite dangerous, and probably better not used. If however, the analyzer emits advanced diagnostics for misuses of it, then it's a good qualifier.

Quoted 7 lines · Alejandro Colomar

Quoted 3 lines · Paul Eggert

Also, “global variables” is not
right here. The C standard allows strtol, for example, to read and write an internal static cache. (Yes, that would be weird, but it’s allowed.)

That's not part of the API. A user must not access internal static cache

Although true in the normal (sane) case, as an extension the implementation can make such a static cache visible to the user, and in this case the caller must not pass cache addresses as arguments to strtol.

For other functions this point is not purely academic. For example, the C standard specifies the signature "FILE *fopen(const char *restrict, const char *restrict);". If I understand your argument correctly, it says that the "restrict"s can be omitted there without changing the set of valid programs.

But that can't be right, as omitting the "restrict"s would make the following code be valid in any platform where sizeof(int)>1:

   char *p = (char *) &errno;
   p[0] = 'r';
   p[1] = 0;
   FILE *f = fopen (p, p);

even though the current standard says this code is invalid.

Quoted 6 lines

Quoted 5 lines

“endptr access(write_only) ... *endptr access(none)”

This is true for glibc, but it’s not necessarily true for all conforming strtol implementations. If endptr is non-null, a conforming strtol implementation can both read and write *endptr;

Quoted 2 lines

It can't, I think. It's perfectly valid to pass an uninitialized endptr, which means the callee must not read the original value.

Sure, but the callee can do something silly like "*endptr = p + 1; *endptr = *endptr - 1;". That is, it can read *endptr after writing it, without any undefined behavior. (And if the callee is written in assembly language it can read *endptr even before writing it - but I digress.)

The point is that it is not correct to say that *endptr cannot be read from; it can. Similarly for **endptr.

Quoted 4 lines

Here, we need to consider two separate objects. The object pointed-to by *endptr _before_ the object pointed to by endptr is written to, and the object pointed-to by *endptr _after_ the object pointed to by endptr is written to.

Those are not the only possibilities. The C standard also permits strtol to set *endptr to some other pointer value, not pointing anywhere into the string being scanned, so long as it sets *endptr correctly before it returns.

Quoted 7 lines

Quoted 4 lines

“The caller knows that errno doesn’t alias any of the function arguments.”

Only because all args are declared with ‘restrict’. So if the proposal is accepted, the caller doesn’t necessarily know that.

Not really. The caller has created the string (or has received it via a restricted pointer)

v0.2 doesn't state the assumption that the caller either created the string or received it via a restricted pointer. If this assumption were stated clearly, that would address the objection here.

Quoted 6 lines

Quoted 4 lines

“The callee knows that *endptr is not accessed.”

This is true for glibc, but not necessarily true for every conforming strtol implementation.

The original *endptr may be uninitialized, and so must not be accessed.

**endptr can be read once the callee sets *endptr. **endptr can even be written, if the callee temporarily sets *endptr to point to a writable buffer; admittedly this would be weird but it's allowed.

Quoted 10 lines

Quoted 6 lines

“It might seem that it’s a problem that the callee doesn’t know if nptr can alias errno or not. However, the callee will not write to the latter directly until it knows it has failed,”

Again this is true for glibc, but not necessarily true for every conforming strtol implementation.

An implementation is free to set errno = EDEADLK in the middle of it, as long as it later removes that. However, I don't see how it would make any sense.

It could make sense in some cases. Here the spec is a bit tricky, but an implementation is allowed to set errno = EINVAL first thing, and then set errno to some other nonzero value if it determines that the arguments are valid. I wouldn't implement strtol that way, but I can see where someone else might do that.

Quoted 8 lines

Let's find
an ISO C function that accepts a non-restrict string:

	int system(const char *string);

Does ISO C constrain implementations to support system((char *)&errno)? I don't think so. Maybe it does implicitly because of a defect in the wording, but even then it's widely understood that it doesn't.

'system' is a special case since the C standard says 'system' can do pretty much anything it likes. That being said, I agree that implementations shouldn't need to support calls like atol((char *) &errno). Certainly the C standard's description of atol, which defines atol's behavior in terms of a call to strtol, means that atol's argument in practice must follow the 'restrict' rules.

Perhaps we should report this sort of thing as a defect in the standard. It is odd, for example, that fopen's two arguments are both const char *restrict, but system's argument lacks the "restrict".

Quoted 6 lines

Quoted 2 lines

Why is this change worth
making? Real-world programs do not make calls like that.

Because it makes analysis of 'restrict' more consistent. The obvious improvement of GCC's analyzer to catch restrict violations will trigger false positives in normal uses of strtol(3).

v0.2 does not support this line of reasoning. On the contrary, v0.2 suggests that a compiler should diagnose calls like "strtol(p, &p, 0)", which would be wrong as that call is perfectly reasonable.

Another way to put it: v0.2 does not clearly state the advantages of the proposed change, and in at least one area what it states as an advantage would actually be a disadvantage.

Quoted 6 lines

Quoted 5 lines

“m = strtol(p, &p, 0); An analyzer more powerful than the current ones could extend the current -Wrestrict diagnostic to also diagnose this case.”

Why would an analyzer want to do that? This case is a perfectly normal thing to do and it has well-defined behavior.

Quoted 8 lines

Because without an analyzer, restrict cannot emit many useful diagnostics. It's a qualifier that's all about data flow analysis, and normal diagnostics aren't able to do that.

A qualifier that enables optimizations but doesn't enable diagnostics is quite dangerous, and probably better not used. If however, the analyzer emits advanced diagnostics for misuses of it, then it's a good qualifier.

Sorry, but I don't understand what you're trying to say here. Really, I can't make heads or tails of it. As-is, 'restrict' can be useful both for optimization and for generating diagnostics, and GCC does both of these things right now even if you don't use -fanalyzer.

Perhaps adding an example or two would help explain your point. But they'd need to be better examples than what's in v0.2 because v0.2 is unclear about this quality-of-diagnostics issue, as it relates to strtol.

a small set of functions
accept pointers that alias each other, but one of them is never accessed; in those few cases, restrict was added to the parameters in ISO C, but I claim it would be better removed.

the small set of functions where this happens don't seem to use any state, so we don't need to care about implementations using internal buffers that are passed somehow to the user.

Maybe I should use abstract names for the objects, to avoid confusing them with the pointer variables that are used to pass them?

this formal
definition is quite unreadable, though. The more I read it, the less sure I am about it.

If L is used to access the value of the object X that it designates, and X is also modified (by any means), then the following requirements apply: T shall not be const-qualified

This reads to me as "const variables are not writable when they are accessed via a restricted pointer; casting away is not enough". Am I reading this correctly?

Quoted 4 lines

an implementation is allowed to set errno = EINVAL first thing, and then set errno to some other nonzero value if it determines that the arguments are valid. I wouldn't implement strtol that way, but I can see where someone else might do that.

In any case an implementation is not obliged to pessimize strtol(3). It is only allowed to. Should we not allow them to do so?

Let's take a simpler one: rename(2). Is it allowed to receive &errno? Hopefully not.

Quoted 7 lines

Quoted 6 lines

Quoted 2 lines

Why is this change worth
making? Real-world programs do not make calls like that.

Because it makes analysis of 'restrict' more consistent. The obvious improvement of GCC's analyzer to catch restrict violations will trigger false positives in normal uses of strtol(3).

Quoted 3 lines

v0.2 does not support this line of reasoning. On the contrary, v0.2 suggests that a compiler should diagnose calls like "strtol(p, &p, 0)", which would be wrong as that call is perfectly reasonable.

That call is perfectly, reasonable, which is why I suggest that the standard should modify the prototype so that strtol(p, &p, 0), which is a reasonable call, should not be warned by a compiler that would diagnose such calls.

That is, just by reading the prototypes:

	void foo(int *restrict x, int **restrict p);

and

	void bar(int *x, int **restrict endp);

one should be able to determine that

	foo(p, &p);

is probably causing UB (and thus trigger a warning) but

	bar(p, &p);

is fine.

Quoted 3 lines

Another way to put it: v0.2 does not clearly state the advantages of the proposed change, and in at least one area what it states as an advantage would actually be a disadvantage.

The advantage is having more information in the caller. As a caller, I want to distinguish calls where it's ok to pass pointers that alias, and where not. And I want my compiler to be able to help me there.

I'd rather have a simple analyzer, which will provide for less false positives and negatives.

GCC can only catch the most obvious violations of restrict.

	#include <string.h>

	typedef struct {
		int x;
	} T;

	[[gnu::access(read_only, 1)]]
	[[gnu::access(read_only, 2)]]
	void
	replace(T *restrict *restrict ls, const T *restrict new, size_t pos)
	{
		memcpy(ls[pos], new, sizeof(T));
	}

	void
	f(T *restrict *restrict ls)
	{
		replace(ls, ls[0], 1);
	}

	$ gcc-14 -Wall -Wextra -fanalyzer replace.c -S
	$

The above program causes UB,

writing an analyzer that triggers on this code, it will trigger on strtol(3) too.

Quoted 4 lines · Alejandro Colomar

a small set of functions
accept pointers that alias each other, but one of them is never accessed; in those few cases, restrict was added to the parameters in ISO C, but I claim it would be better removed.

Are these aliasing pointers the nptr and initial *endptr of strtol?

That is,
are you saying that last line in the following example, which is currently invalid, should become valid and should be implementable as ‘end = s; long l = 0;’?

   char *end;
   char *s = (char *) &end;
   *s = '\0';
   long l = strtol (s, &end, 0);

If so, I fail to see the motivation for the proposed change, as nobody writes (or should write) code like that. And if not, evidently I misunderstand the proposal.

Quoted 3 lines

Maybe I should use abstract names for the objects, to avoid confusing them with the pointer variables that are used to pass them?

That might help, yes, since v0.2 is unclear on this point.

Quoted 3 lines

this formal
definition is quite unreadable, though. The more I read it, the less sure I am about it.

Yes, it’s lovely isn’t it? One must understand what the C committee intended in order to read and understand that part of the standard.

Quoted 4 lines

If L is used to access the value of the object X that it designates, and X is also modified (by any means), then the following requirements apply: T shall not be const-qualified

Quoted 3 lines

This reads to me as "const variables are not writable when they are accessed via a restricted pointer; casting away is not enough". Am I reading this correctly?

In that quoted statement, the restricted pointer is not allowed to be pointer-to-const. However, I’m not quite sure what your question means, as the phrase “const variables” does not appear in the standard. Perhaps give an example to clarify the question?

Quoted 1 line

Quoted 1 line

an implementation is allowed to set errno = EINVAL first thing, and then

set

Quoted 4 lines

Quoted 3 lines

errno to some other nonzero value if it determines that the arguments are valid. I wouldn't implement strtol that way, but I can see where someone else might do that.

Quoted 2 lines

In any case an implementation is not obliged to pessimize strtol(3). It is only allowed to. Should we not allow them to do so?

Of course the standard should allow suboptimal implementations. However, I’m not sure what the point of the question is. The “errno = EINVAL first thing” comment says that removing ‘restrict’ obliges the implementation to support obviously-bogus calls like strtol(&errno, ...), which might make the implementation less efficient.

I don’t see how the question is relevant to
that comment.

Quoted 2 lines

Let's take a simpler one: rename(2). Is it allowed to receive &errno? Hopefully not.

I agree with that hope, but the current C standard seems to allow it. I think we both agree this is a defect in the standard.

Quoted 8 lines

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

Why is this change worth
making? Real-world programs do not make calls like that.

Because it makes analysis of 'restrict' more consistent. The obvious improvement of GCC's analyzer to catch restrict violations will trigger false positives in normal uses of strtol(3).

v0.2 does not support this line of reasoning. On the contrary, v0.2

suggests

Quoted 1 line

Quoted 1 line

that a compiler should diagnose calls like "strtol(p, &p, 0)", which

would

Quoted 6 lines

Quoted 1 line

be wrong as that call is perfectly reasonable.

That call is perfectly, reasonable, which is why I suggest that the standard should modify the prototype so that strtol(p, &p, 0), which is a reasonable call, should not be warned by a compiler that would diagnose such calls.

Of course they shouldn’t warn. But where are these compilers?

v0.2 asserts that “An analyzer more powerful than the current ones could extend the current -Wrestrict diagnostic to also diagnose this case.” But why would an analyzer want to do that? v0.2 doesn’t say.

The proposal merely asks to change prototypes for the C standard functions strtol, strtoul, etc. But if that is the only change needed then why bother? C compilers already do special-case analysis for functions defined by the C standard, and they can suppress undesirable diagnostics for these special cases.

If you’ve identified a more general problem with ‘restrict’ then welcome to the club! The experts already know it’s confusing and limited, and are discussing about whether and how to improve things in the next C standard. I am sure you’d be welcome to those discussions.

Quoted 14 lines

That is, just by reading the prototypes:

	void foo(int *restrict x, int **restrict p);

and

	void bar(int *x, int **restrict endp);

one should be able to determine that

	foo(p, &p);

is probably causing UB (and thus trigger a warning) but

Quoted 3 lines

	bar(p, &p);

is fine.

Sure, but this is a discussion we should be having with the compiler writers, no?

Is this the main motivation for the proposal?

If so, how would weakening the
spec for strtol etc. affect that discussion with the compiler writers? v0.2 does not make this clear.

Quoted 4 lines

Quoted 3 lines

Another way to put it: v0.2 does not clearly state the advantages of the proposed change, and in at least one area what it states as an advantage would actually be a disadvantage.

Quoted 3 lines

The advantage is having more information in the caller. As a caller, I want to distinguish calls where it's ok to pass pointers that alias, and where not. And I want my compiler to be able to help me there.

I’m still not understanding. Removing ‘restrict’ from strtol’s first arg gives the caller less information, not more.

Quoted 2 lines

I'd rather have a simple analyzer, which will provide for less false positives and negatives.

The C committee appears to have the opposite opinion, as when they were asked about this matter they added Examples 5 through 7 to what is now §6.7.4.2 (Formal definition of restrict). These examples say that Example 2 (which uses ‘restrict’ on all arguments) is the simplest and most effective way to use ‘restrict’, even though a smarter compiler can still make some good inferences when some pointer args are ‘restrict’ and others are merely pointers to const.

If the proposal is disagreeing with Examples 5 through 7, this point needs to be thoroughly discussed in the proposal.

Quoted 1 line

GCC can only catch the most obvious violations of restrict.

Yes, but I fail to see how changing the API for strtol etc. would improve that situation.

Quoted 20 lines

	#include <string.h>

	typedef struct {
		int x;
	} T;

	[[gnu::access(read_only, 1)]]
	[[gnu::access(read_only, 2)]]
	void
	replace(T *restrict *restrict ls, const T *restrict new, size_t pos)
	{
		memcpy(ls[pos], new, sizeof(T));
	}

	void
	f(T *restrict *restrict ls)
	{
		replace(ls, ls[0], 1);
	}

Quoted 4 lines

	$ gcc-14 -Wall -Wextra -fanalyzer replace.c -S
	$

The above program causes UB,

It’s not a complete program and I don’t see the undefined behavior.

If
behavior is undefined because it violates the [[gnu::access(...)]] restrictions,

that is not the sort of example that would convince the C standardization committee; they’d want to see a standard C program.

I tried to write a standard C program to illustrate the issue, and came up with the following.

Hi Paul,

Quoted 10 lines · Paul Eggert

Quoted 3 lines · Alejandro Colomar

I've incorporated feedback, and here's a new revision, let's call it
v0.2, of the draft for a WG14 paper.

Although I have not followed the email discussion closely, I read v0.2 and
think that as stated there is little chance that its proposal will be
accepted.

Thanks for reading thoroughly, and the feedback!

Quoted 36 lines

Fundamentally the proposal is trying to say that there are two styles X and
Y for declaring strtol and similar functions, and that although both styles
are correct in some sense, style Y is better than style X. However, the
advantages of Y are not clearly stated and the advantages of style X over Y
are not admitted, so the proposal is not making the case clearly and fairly.

One other thing: to maximize the chance of a proposal being accepted, please
tailor it for its expected readership. The C committee is expert on ‘restrict’, so don’t try to define ‘restrict’ in your own way. Unless merely
repeating the language of the standard, any definition given for ‘restrict’
is likely to cause the committee to quibble with the restatement of the
standard wording. (It is OK to mention some corollaries of the standard
definition, so long as the corollaries are not immediately obvious.)

Here are some comments about the proposal. At the start these comments are
detailed; towards the end, as I could see the direction the proposal was
headed and was convinced it wouldn’t be accepted as stated, the comments are
less detailed.

"The API may copy"

Quoted 9 lines

One normally doesn’t think of the application programming interface as
copying. Please replace the phrase “the API” with “the caller” or “the
callee” as appropriate. (Although ‘restrict’ can be used in places other
than function parameters, I don’t think the proposal is concerned about
those cases and so it doesn’t need to go into that.)

Ok.

Quoted 9 lines

"To avoid violations of for example C11::6.5.16.1p3,"

Code that violates C11::6.5.16.1p3 will do so regardless of whether ‘restrict’ is present. I would not mention C11::6.5.16.1p3 as it’s a red
herring. Fundamentally, ‘restrict’ is not about the consequences of caching
when one does overlapping moves; it’s about caching in a more general sense.

The violations are UB regardless of restrict, but consistent use of restrict allows the caller to have a rough model of what the callee will
do with the objects, and prevent those violations via compiler diagnostics.  I've reworded that part to make it more clear why I'm mentioning that.

Quoted 5 lines

“As long as an object is only accessed via one restricted pointer, other
restricted pointers are allowed to point to the same object.”

“only accessed” → “accessed only”

Ok.

Quoted 8 lines

“This is less strict than I think it should be, but this proposal doesn’t
attempt to change that definition.”

I would omit this sentence and all similar sentences. Don’t distract the
reader with other potential proposals. The proposal as it stands is complicated enough.

Ok.

Quoted 12 lines

“return ca > a;”
“return ca > *ap;”

I fail to understand why these examples are present. It’s not simply that
nobody writes code like that: the examples are not on point. I would remove
the entire programs containing them, along with the sections that discuss
them. When writing to the C committee one can assume the reader is expert in
‘restrict’, there is no need for examples such as these.

Those are examples of how consistent use of restrict can --or could, in
the case of g()-- detect, via compiler diagnostics, (likely) violations
of seemingly unrelated parts of the standard, such as the referenced C11::6.5.16.1p3, or in this case, C11::6.5.8p5.

Quoted 6 lines

“strtol(3) accepts 4 objects via pointer parameters and global variables.”

Omit the “(3)”, here and elsewhere, as the audience is the C standard
committee.

The C standard committee doesn't know about historic use of (3)? That
predates the standard, and they built on top of that (C originated in Unix).  While they probably don't care about it anymore, I expect my paper to be read by other audience, including GCC and glibc, and I prefer to keep it readable for that audience.  I expect the standard committee to at least have a rough idea of the existence of this syntax,
and respect it, even if they don't use it or like it.

Quoted 3 lines

“accepts” is a strange word to use here: normally one says “accepts” to talk
about parameters, not global variables.

The thing is, strtol(3) does not actually access *endptr.  I thought that might cause more confusion than using "accepts".

Quoted 5 lines

Also, “global variables” is not
right here. The C standard allows strtol, for example, to read and write an
internal static cache. (Yes, that would be weird, but it’s allowed.)

That's not part of the API.  A user must not access internal static cache, and so the implementation is free to assume that it doesn't, regardless of the use of restrict in the API, so it is not relevant for
the purpose of this discussion, I think.

Quoted 3 lines

I
suggest rephrasing this sentence to talk about accessing, not accepting.

I don't want to use accessing, for it would be inconsistent with later
saying that *endptr is not accessed.  However, I'm open to other suggested terms that might be more appropriate than both.

Quoted 6 lines

“endptr access(write_only) ... *endptr access(none)”

This is true for glibc, but it’s not necessarily true for all conforming
strtol implementations. If endptr is non-null, a conforming strtol implementation can both read and write *endptr;

It can't, I think.  It's perfectly valid to pass an uninitialized endptr, which means the callee must not read the original value.

        char *end;
        strtol("0", &end, 0);

If strtol(3) would be allowed to read it, the user would need to initialize it.

Quoted 4 lines

it can also both read and
write **endptr. (Although it would need to write before reading, reading is
allowed.)

Here, we need to consider two separate objects.  The object pointed- to
by *endptr _before_ the object pointed to by endptr is written to, and
the object pointed-to by *endptr _after_ the object pointed to by endptr
is written to.

For the former (the original *endptr):

        Since *endptr might be uninitialized, strtol(3) must NOT access
        the object pointed to by an uninitialized pointer.

For the latter (the final *endptr):

        The callee cannot write to it, since the specification of the function is that the string will not be modified.  And in any case, such an access is ultimately derived from nptr, not from
        *endptr, so it does not matter for the discussion of *endptr.

Of course, that's derived from the specification of the function, and not from its prototype, since ISO C doesn't provide such detailed prototypes (since it doesn't have the [[gnu::access()]] attribute). But
the standard must abide by its own specification of functions, anyway.

Quoted 9 lines

“This qualifier helps catch obvious bugs such as strtol(p, p, 0) and
strtol(&p, &p, 0) .”

No it doesn’t. Ordinary type checking catches those obvious bugs, and
‘restrict’ provides no additional help there. Please complicate the examples
to make the point more clearly.

To be pedantic, I didn't specify the type of p, so it might be (void *),
and thus avoid type checking at all.  However, to avoid second guessing
from the standards committee, I'll add casts, to make it more obvious that restrict is catching those.

Quoted 6 lines

“The caller knows that errno doesn’t alias any of the function arguments.”

Only because all args are declared with ‘restrict’. So if the proposal is
accepted, the caller doesn’t necessarily know that.

Not really.  The caller has created the string (or has received it via a
restricted pointer), and so it knows it's not derived from errno.

        char buf[LINE_MAX + 1];

        fgets(...);
        n = strtol(buf, ...);

This caller knows with certainty that errno does not alias buf.  Of course, in some complex cases, it might not know, but I ommitted that for simplicity.  And in any case, I don't think any optimizations are affected by that in the caller.

Quoted 7 lines

“The callee knows that *endptr is not accessed.”

This is true for glibc, but not necessarily true for every conforming strtol
implementation.

The original *endptr may be uninitialized, and so must not be accessed.

Quoted 9 lines

“It might seem that it’s a problem that the callee doesn’t know if nptr can
alias errno or not. However, the callee will not write to the latter
directly until it knows it has failed,”

Again this is true for glibc, but not necessarily true for every conforming
strtol implementation.

An implementation is free to set errno = EDEADLK in the middle of it, as
long as it later removes that.  However, I don't see how it would make
any sense.

If that's done, it's probably done via a helper internal function, which
as said below, can use restrict for nptr, and thus know with certainty
that nptr is distinct from errno.

If that's done directly in the body of strtol(3) (the only place where
it's not known that nptr is distinct from errno) we can probably agree
that the implementation is doing that just for fun, and doesn't care about optimization, and thus we can safely ignore it.

Quoted 6 lines

To my mind this is the most serious objection. The current standard prohibits calls like strtol((char *) &errno, 0, 0). The proposal would relax
the standard to allow such calls. In other words, the proposal would
constrain implementations to support such calls.

I don't think it does.  ISO C specifies that strtol(3) takes a string as
its first parameter, and errno is not (unless you do this:).

        (char *)&errno = "111";

Okay, let's assume you're allowed to do that, since a char* can alias anything.

I still don't think ISO C constrains implementations to allow passing (char *)&errno as a char*, just because it's not restrict.  Let's find
an ISO C function that accepts a non-restrict string:

        int system(const char *string);

Does ISO C constrain implementations to support system((char *)&errno)?
I don't think so.  Maybe it does implicitly because of a defect in the
wording, but even then it's widely understood that it doesn't.

Quoted 2 lines

Why is this change worth
making? Real-world programs do not make calls like that.

Because it makes analysis of 'restrict' more consistent.  The obvious improvement of GCC's analyzer to catch restrict violations will trigger
false positives in normal uses of strtol(3).

Quoted 10 lines

“But nothing prohibits those internal helper functions to specify that nptr
is restrict and thus distinct from errno.”

Although true, it’s also the case that the C standard does not *require*
internal helper functions to use ‘restrict’. All that matters is the
accesses. So I’m not sure what the point of this statement is.

If an implementation wants to optimize, it should be careful and use restrict.  If it doesn't, then it can't complain that ISO C doesn't allow it to.  It's actually allowed to optimize, but it has to do some
work for it.

Quoted 8 lines

“m = strtol(p, &p, 0); An analyzer more powerful than the current ones
could extend the current -Wrestrict diagnostic to also diagnose this case.”

Why would an analyzer want to do that? This case is a perfectly normal thing
to do and it has well-defined behavior.

Because without an analyzer, restrict cannot emit many useful diagnostics.  It's a qualifier that's all about data flow analysis, and
normal diagnostics aren't able to do that.

A qualifier that enables optimizations but doesn't enable diagnostics is
quite dangerous, and probably better not used.  If however, the analyzer
emits advanced diagnostics for misuses of it, then it's a good qualifier.

Have a lovely day!
Alex

Quoted 22 lines

“To prevent triggering diagnostics in a powerful analyzer that would be
smart enough to diagnose the example function g(), the prototype of strtol(3) should be changed to ‘long int strtol(const char *nptr, char
**restrict endptr, int base);’”

Sorry, but the case has not been made to make any such change to strtol’s
prototype. On the contrary, what I’m mostly gathering from the discussion is
that ‘restrict’ can be confusing, which is not news.

n3220 §6.7.4.2 examples 5 through 7 demonstrate that the C committee has
thought through the points you’re making. (These examples were not present
in C11.) This may help to explain why the standard specifies strtol with
‘restrict’ on both arguments.

Quoted 7 lines

Quoted 2 lines

Why is this change worth
making? Real-world programs do not make calls like that.

Because it makes analysis of 'restrict' more consistent.  The obvious improvement of GCC's analyzer to catch restrict violations will trigger
false positives in normal uses of strtol(3).

Hi Alejandro

I'm author/maintainer of GCC's -fanalyzer option, which is presumably why you CCed me on this.

One of my GSoC 2022 students (Tim Lange)
looked at making use of 'restrict' in -fanalyzer, see e.g.
https://lists.gnu.org/archive/html/bug-gnulib/2022-07/msg00062.html

Based on Paul's comment here:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99860#c2 (and its references) I came to the conclusion at the time that we should work on something else, as the meaning of 'restrict' is too ambiguous.

Later, I added a new -Wanalyzer-overlapping-buffers warning in GCC 14, which simply has a hardcoded set of standard library functions that it "knows" to warn about.

Has the C standard clarified the meaning of 'restrict' since that discussion? Without that, I wasn't planning to touch 'restrict' in GCC's -fanalyzer.

Sorry if I'm missing anything here; I confess I've skimmed through parts of this thread.

Dave

And then have it mean something strict, such as: The object pointed to by the pointer is not pointed to by any other pointer; period.

This definition is already what -Wrestrict seems to understand.

Quoted 4 lines

Later, I added a new -Wanalyzer-overlapping-buffers warning in GCC 14, which simply has a hardcoded set of standard library functions that it "knows" to warn about.

Hmmm, so it doesn't help at all for anything other than libc. Ok.

Quoted 3 lines

Has the C standard clarified the meaning of 'restrict' since that discussion? Without that, I wasn't planning to touch 'restrict' in GCC's -fanalyzer.

Meh; no they didn't.

I understand. That's why I don't like innovations
in ISO C, and prefer that implementations innovate with real stuff.

Am Montag, dem 08.07.2024 um 17:01 +0200 schrieb Alejandro Colomar:

Quoted 1 line · David Malcolm

...

Quoted 3 lines

And then have it mean something strict, such as: The object pointed to by the pointer is not pointed to by any other pointer; period.

Quoted 1 line

This definition is already what -Wrestrict seems to understand.

One of the main uses of restrict is scientific computing. In this context such a definition of "restrict" would not work for many important use cases. But I agree that for warning purposes the definition of "restrict" in ISO C is not helpful.

Quoted 5 lines

Quoted 3 lines

Has the C standard clarified the meaning of 'restrict' since that discussion? Without that, I wasn't planning to touch 'restrict' in GCC's -fanalyzer.

Meh; no they didn't.

There were examples added in C23 and there are now several papers being under discussion.

Hi Martin,

Quoted 9 lines · Martin Uecker

Am Montag, dem 08.07.2024 um 17:01 +0200 schrieb Alejandro Colomar:

Quoted 1 line · David Malcolm

...

Quoted 4 lines

And then have it mean something strict, such as: The object pointed to by the pointer is not pointed to by any other pointer; period.

This definition is already what -Wrestrict seems to understand.

Quoted 4 lines

One of the main uses of restrict is scientific computing. In this context such a definition of "restrict" would not work for many important use cases. But I agree that for warning purposes the definition of "restrict" in ISO C is not helpful.

Do you have some examples of functions where this matters and is important? I'm curious to see them. Maybe we find some alternative.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Has the C standard clarified the meaning of 'restrict' since that discussion? Without that, I wasn't planning to touch 'restrict' in GCC's -fanalyzer.

Quoted 4 lines

Quoted 1 line

Meh; no they didn't.

There were examples added in C23 and there are now several papers being under discussion.

Hmm, yeah, the examples help with the formal definition. I was thinking of the definition itself, which I still find quite confusing. :-)

Have a lovely night!
Alex

Am Montag, dem 08.07.2024 um 22:17 +0200 schrieb Alejandro Colomar:

Quoted 12 lines

Hi Martin,

Quoted 9 lines · Martin Uecker

Am Montag, dem 08.07.2024 um 17:01 +0200 schrieb Alejandro Colomar:

Quoted 1 line · David Malcolm

...

Quoted 4 lines

And then have it mean something strict, such as: The object pointed to by the pointer is not pointed to by any other pointer; period.

This definition is already what -Wrestrict seems to understand.

Quoted 7 lines

Quoted 4 lines

One of the main uses of restrict is scientific computing. In this context such a definition of "restrict" would not work for many important use cases. But I agree that for warning purposes the definition of "restrict" in ISO C is not helpful.

Do you have some examples of functions where this matters and is important? I'm curious to see them. Maybe we find some alternative.

In many numerical algorithms you want to operate on different parts of the same array object. E.g. for matrix decompositions you want to take a row / column and add it to another. Other examples are algorithms that decompose some input (.e.g. high and low band in a wavelet transform) and store it into the same output array, etc.

Without new notation for strided array slicing, one

fundamentally needs the flexibility of restrict that only guarantuees that actual accesses do not conflict.

But this then implies that one can not use restrict as a contract specification on function prototypes, but has to analyze the implementation of a function to see if it is used correctly. But I would not see it as a design problem of restrict. It was simply not the intended use case when originally designed.

Quoted 10 lines · David Malcolm

Quoted 8 lines

Quoted 2 lines

Why is this change worth
making? Real-world programs do not make calls like that.

Because it makes analysis of 'restrict' more consistent.  The obvious
improvement of GCC's analyzer to catch restrict violations will trigger
false positives in normal uses of strtol(3).

Hi Alejandro

Hi Dave,

Quoted 3 lines

I'm author/maintainer of GCC's -fanalyzer option, which is presumably
why you CCed me on this.

Yup.

Quoted 4 lines

One of my GSoC 2022 students (Tim Lange)
looked at making use of 'restrict' in -fanalyzer, see e.g.
https://lists.gnu.org/archive/html/bug-gnulib/2022-07/msg00062.html

Quoted 5 lines

Based on Paul's comment here:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99860#c2 (and its references) I came to the conclusion at the time that we should work on
something else, as the meaning of 'restrict' is too ambiguous.

restrict, as of the formal definition of ISO C is useless crap.  The more I read it, the more I agree.

restrict, as of what -Wrestrict warns about, seems a reasonable thing.

How about a [[gnu::restrict()]] attribute, similar to [[gnu::access()]],
which is simpler than the qualifier?  Since restrict is only meaningful
in function boundaries, it would make sense to have a function attribute.  We don't want a qualifier that must follow discarding rules.

Quoted 2 lines

restrict, as of the formal definition of ISO C is useless crap.  The more I read it, the more I agree.

Please note that "useless crap" was your wording, not mine.

Quoted 5 lines

restrict, as of what -Wrestrict warns about, seems a reasonable thing.

Quoted 7 lines

How about a [[gnu::restrict()]] attribute, similar to [[gnu::access()]],
which is simpler than the qualifier?  Since restrict is only meaningful
in function boundaries, it would make sense to have a function attribute.  We don't want a qualifier that must follow discarding rules.

If it doesn't have the same meaning as "restrict" then perhaps call the proposed attribute something other than "restrict"?

That said, I don't have strong opinions on any of this, except to note that I have more than enough *other* work on improvements to GCC's static analyzer and usability to keep me busy, so getting sucked into discussion/implementation on 'restrict' is something I want to avoid, and -Wanalyzer-overlapping-buffers is getting the job done for me at the moment.

[...snip...]

Hope this is constructive; sorry again if I missed anything due to only skimming the thread

Dave

Quoted 11 lines

Quoted 10 lines

restrict, as of what -Wrestrict warns about, seems a reasonable thing.

How about a [[gnu::restrict()]] attribute, similar to [[gnu::access()]],
which is simpler than the qualifier?  Since restrict is only meaningful
in function boundaries, it would make sense to have a function attribute.  We don't want a qualifier that must follow discarding rules.

Quoted 2 lines

If it doesn't have the same meaning as "restrict" then perhaps call the proposed attribute something other than "restrict"?

Yup, I was thinking that maybe noalias is a better name.

Quoted 1 line · Alejandro Colomar

Yup, I was thinking that maybe noalias is a better name.

Name is one thing, but you'd also need to clearly define what it means. When restrict is access based, it is clear what it means.

If you want something else which is not based on accesses and which should allow warnings in the callers, I suppose you need to specify not just the pointer but the extent as well (and maybe stride) or that it is an '\0'

terminated string, because if you want to say that for
void foo (char *, const char *, int);
the 2 pointers don't really alias, the size information is missing. So, shall the new warning warn on
struct S { char a[1024]; char b[1024]; } s;
foo (s.a, s.b, 512);

or not? Or foo (s.a, s.a + 512, 512);

Jakub

Hi Jakub,

Quoted 9 lines · Jakub Jelinek

Quoted 1 line · Alejandro Colomar

Yup, I was thinking that maybe noalias is a better name.

Name is one thing, but you'd also need to clearly define what it means. When restrict is access based, it is clear what it means.

If you want something else which is not based on accesses and which should allow warnings in the callers, I suppose you need to specify not just the pointer but the extent as well (and maybe stride) or that it is an '\0'

Agree. Here's how I'd define it as an attribute:

noalias

The noalias function attribute specifies that the pointer to which it applies is the only reference to the array object that it points to (except that a pointer to one past the last element may overlap another object).

If the number of elements is specified with array notation, the array object to be considered is a subobject of the original array object, which is limited to the number of elements specified in the function prototype.

Example:

[[alx::noalias(1)]] [[alx::noalias(2)]] [[gnu::access(read_write, 1)]] [[gnu::access(read_only, 2)]] void add_inplace(int a[n], const int b[n], size_t n);

char arr[100] = ...;

add_inplace(arr, arr + 50, 50);

In the example above, the parameters a and b don't alias inside the function, since the subobjects of 50 elements do not overlap eachother, even though they are one single array object to the outer function.

It may need some adjustment, to avoid conflicts with other parts of ISO C, but this is the idea I have in mind.

Quoted 6 lines

terminated string, because if you want to say that for
void foo (char *, const char *, int);
the 2 pointers don't really alias, the size information is missing. So, shall the new warning warn on
struct S { char a[1024]; char b[1024]; } s;
foo (s.a, s.b, 512);

This does not need clarification of bounds. You're passing separate objects, and thus cannot alias (except that maybe you're able to cast to the struct type, and then access s.b from a pointer derived from s.a; I never know that rule too well).

Quoted 1 line

or not? Or foo (s.a, s.a + 512, 512);

According to the definition I provide in this email, the above is just fine.

Thanks!

Have a lovely day!
Alex

Quoted 4 lines

Jakub

Signature · 1 line

<https://www.alejandro-colomar.es/>

6.7.13.x The restrict function attribute
Constraints
The restrict attribute shall be applied to a function.

A 1‐based index can be specified in an attribute argument clause, to associate the attribute with the corresponding parameter of the function, which must be of a pointer type.

(Optional.) The argument attribute clause may be omitted,
which is equivalent to specifying the attribute once for each parameter that is a pointer.

If the number of elements is specified with array notation (or a compiler‐specific attribute), the array object to be considered for aliasing is a sub‐object of the original ar‐ ray object, limited by the number of elements specifiedr [1].

     [1]  For the following prototype:

                 [[restrict(1)]] [[restrict(2)]]
                 void f(size_t n, int a[n], const int b[n]);

I know that "noalias" was included in some C89 drafts but removed from the final standard after objections. Maybe someone who was around then could explain what "noalias" was, what the problems with it were

and how it differs from "restrict",

To comprehensively address this demands so we can make sure that any new proposals in this area don't suffer from whatever the perceived deficiencies of "noalias" were?

Quoted 3 lines · Joseph Myers

I know that "noalias" was included in some C89 drafts but removed from the final standard after objections. Maybe someone who was around then could explain what "noalias" was, what the problems with it were

For this part, I think the source most often cited is Dennis Ritchie's thunderbolt aimed directly at "noalias".

https://www.lysator.liu.se/c/dmr-on-noalias.html

Quoted 2 lines

and how it differs from "restrict",

I can only disqualify myself as an authority here.

Quoted 3 lines

To comprehensively address this demands so we can make sure that any new proposals in this area don't suffer from whatever the perceived deficiencies of "noalias" were?

I think it would be valuable to get such a discussion into the rationale of the next C standard.

Regards,
Branden

Maybe someone who was around then could
explain what "noalias" was, what the problems with it were and how it differs from "restrict"

Quoted 4 lines · Alejandro Colomar via Gcc

6.7.13.x The restrict function attribute
Constraints
The restrict attribute shall be applied to a function.

Quoted 3 lines

A 1‐based index can be specified in an attribute argument clause, to associate the attribute with the corresponding parameter of the function, which must be of a pointer type.

It's more appropriate to say "shall", and you need a requirement for the pointer to be a pointer to a complete object type (it makes no sense with function pointers, or void).

That is, something like "If an attribute
argument clause is present, it shall have the form:

( constant-expression )

The constant expression shall be an integer constant expression with positive value. It shall be the index, counting starting from 1, of a function parameter whose type is a pointer to a complete object type.".

(That might not quite be sufficient - there are the usual questions of exactly *when* the type needs to be complete, if it's completed part way through the function definition, but the standard already doesn't tend to specify such things very precisely.)

Quoted 3 lines

(Optional.) The argument attribute clause may be omitted,
which is equivalent to specifying the attribute once for each parameter that is a pointer.

For each parameter that is a pointer to a complete object type, or should there be a constraint violation in this case if some are pointers to such types and some are pointers to other types?

Quoted 5 lines

If the number of elements is specified with array notation (or a compiler‐specific attribute), the array object to be considered for aliasing is a sub‐object of the original ar‐ ray object, limited by the number of elements specifiedr [1].

This is semantically problematic in the absence of something like N2906 (different declarations could use different numbers of elements),

and even
N2906 wouldn't help for the VLA case.

Quoted 4 lines

     [1]  For the following prototype:

                 [[restrict(1)]] [[restrict(2)]]
                 void f(size_t n, int a[n], const int b[n]);

That declaration currently means

void f(size_t n, int a[*], const int b[*]);

(that is, the expression giving a VLA size is ignored). It's equivalent to e.g.

  void f(size_t n, int a[n + foo()], const int b[n + bar()]);

where because the size expressions are never evaluated and there's no time defined for evaluation, it's far from clear what anything talking about them giving an array size would even mean.

I know that "noalias" was included in some C89 drafts but removed from the final standard after objections. Maybe someone who was around then could explain what "noalias" was, what the problems with it were and how it differs from "restrict", so we can make sure that any new proposals in this area don't suffer from whatever the perceived deficiencies of "noalias" were?

Signature · 3 lines

-- 
Joseph S. Myers
[email protected]

I don't see why it should not apply to void*. memcpy(3) should get this attribute:

	[[alx::restrict(1)]]
	[[alx::restrict(2)]]
	void *memcpy(void *dst, const void *src, size_t n);

BTW, the author of n2529 didn't follow up, right? I'd like that in, so I'll prepare something after n2906 is merged. Martin, would you mind pinging me about it?

Quoted 3 lines · Alejandro Colomar via Gcc

I don't see why it should not apply to void*. memcpy(3) should get this attribute:

Quoted 3 lines

	[[alx::restrict(1)]]
	[[alx::restrict(2)]]
	void *memcpy(void *dst, const void *src, size_t n);

That would disallow copying between disjoint subarrays within the same toplevel object (and there's no way to specify an array size for void *), which hardly seems right.

Quoted 3 lines

BTW, the author of n2529 didn't follow up, right? I'd like that in, so I'll prepare something after n2906 is merged. Martin, would you mind pinging me about it?

See reflector message SC22WG14.18575, 17 Nov 2020 (the former convenor replying when I asked about just that paper).

As far as I know the author
has not yet provided an updated version / asked for it to be added to a meeting agenda.

Signature · 3 lines

-- 
Joseph S. Myers
[email protected]

Quoted 2 lines

See reflector message SC22WG14.18575, 17 Nov 2020 (the former convenor replying when I asked about just that paper).

Where can I find reflector messages?

And another one to propose that [n] means the same as [static n] except for the nonnull property of static.

Quoted 4 lines · Alejandro Colomar via Gcc

Quoted 2 lines

See reflector message SC22WG14.18575, 17 Nov 2020 (the former convenor replying when I asked about just that paper).

Where can I find reflector messages?

https://www.open-std.org/jtc1/sc22/wg14/18575

Quoted 3 lines

And another one to propose that [n] means the same as [static n] except for the nonnull property of static.

I'm not convinced that introducing extra undefined behavior for things that have been valid since C89 (which would be the effect of such a change for any code that passes a smaller array) is a good idea - the general mood is to *reduce* undefined behavior.

Signature · 3 lines

-- 
Joseph S. Myers
[email protected]

Quoted 8 lines · Alejandro Colomar via Gcc

Quoted 5 lines

Quoted 3 lines

See reflector message SC22WG14.18575, 17 Nov 2020 (the former convenor replying when I asked about just that paper).

Where can I find reflector messages?

https://www.open-std.org/jtc1/sc22/wg14/18575

Thanks!

Quoted 8 lines

Quoted 3 lines

And another one to propose that [n] means the same as [static n] except for the nonnull property of static.

I'm not convinced that introducing extra undefined behavior for things that have been valid since C89 (which would be the effect of such a change for any code that passes a smaller array) is a good idea - the general mood is to *reduce* undefined behavior.

While [n] has always _officially_ meant the same as [], it has never made any sense to write code like that. Unofficially, it has always meant the obvious thing.

Maybe if GNU C compilers (GCC and Clang) add it first as an extension, adding diagnostics, it would help.

Does anyone know of any existing code that uses [n] for meaning anything other than "n elements are available to the function"?

Functions that specify [n] most likely (definitely?) already mean that n elements are accessed, and thus passing something different than n elements results in UB one way or another. Having the compiler enforce that via diagnostics and UB is probably an improvement.

Cheers,
Alex

Quoted 5 lines

Signature · 3 lines

-- 
Joseph S. Myers
[email protected]

Quoted 2 lines

Maybe if GNU C compilers (GCC and Clang) add it first as an extension, adding diagnostics, it would help.

Both GCC and Clang already have such diagnostics and/or run-time checks:

https://godbolt.org/z/MPnxqb9h7

Quoted 5 lines

Quoted 2 lines

Maybe if GNU C compilers (GCC and Clang) add it first as an extension, adding diagnostics, it would help.

Both GCC and Clang already have such diagnostics and/or run-time checks:

Quoted 1 line

https://godbolt.org/z/MPnxqb9h7

Hi Martin,

I guess that's prior art enough to make this UB in ISO C. Is there any paper for this already? Does any of your paper cover that? Should I prepare one?

Am Samstag, dem 27.07.2024 um 00:26 +0200 schrieb Alejandro Colomar:

Quoted 8 lines · Martin Uecker

Quoted 6 lines

Quoted 2 lines

Maybe if GNU C compilers (GCC and Clang) add it first as an extension, adding diagnostics, it would help.

Both GCC and Clang already have such diagnostics and/or run-time checks:

https://godbolt.org/z/MPnxqb9h7

Quoted 6 lines

Hi Martin,

I guess that's prior art enough to make this UB in ISO C. Is there any paper for this already? Does any of your paper cover that? Should I prepare one?

What do you mean by "this"?

Adding UB would likely see a lot
of opposition,

even where this could enable run-time checks.

N2906 would make

int foo(char f[4]);
int foo(char f[5]);

a constraint violation (although having those types be incompatible could also cause UB indirectly, this would not be its main effect).

So I think brining a new version of this paper forward would be a possible next step, addressing the issues raised in the past.

Martin