new Page: isalpha__3(3)

Hi List,
The ctype macros like isalpha(3) have a locale specific counterpart. This page was missing.

re,
wh

.\" Copyright (c) 2013 by Walter Harms
.\"
.\" %%%LICENSE_START(VERBATIM)
.\" Permission is granted to make and distribute verbatim copies of this .\" manual provided the copyright notice and this permission notice are .\" preserved on all copies.
.\"
.\" Permission is granted to copy and distribute modified versions of this .\" manual under the conditions for verbatim copying, provided that the .\" entire resulting derived work is distributed under the terms of a .\" permission notice identical to this one.
.\"
.\" Since the Linux kernel and libraries are constantly changing, this .\" manual page may be incorrect or out-of-date. The author(s) assume no .\" responsibility for errors or omissions, or for damages resulting from .\" the use of the information contained herein. The author(s) may not .\" have taken the same level of care in the production of this manual, .\" which is licensed free of charge, as they might when working .\" professionally.
.\"
.\" Formatted or processed versions of this manual, if unaccompanied by .\" the source, must acknowledge the copyright and authors of this work. .\" %%%LICENSE_END
.\"
.\"
.TH ISALPHA_L 3 2013-09-20 "GNU" "Linux Programmer's Manual"
.SH NAME
isalnum_l, isalpha_l, isblank_l, iscntrl_l, isdigit_l, isgraph_l, islower_l, isprint_l, ispunct_l, isspace_l, isupper_l, isxdigit_l \- character classification routines
.SH SYNOPSIS
.nf
.B #include <ctype.h>
.sp
.BI "int isalnum_l(int " "c" ", locale_t " loc );
.br
.BI "int isalpha_l(int " "c" ", locale_t " loc );
.br
.BI "int isascii_l(int " "c" ", locale_t " loc );
.br
.BI "int isblank_l(int " "c" ", locale_t " loc );
.br
.BI "int iscntrl_l(int " "c" ", locale_t " loc );
.br
.BI "int isdigit_l(int " "c" ", locale_t " loc );
.br
.BI "int isgraph_l(int " "c" ", locale_t " loc );
.br
.BI "int islower_l(int " "c" ", locale_t " loc );
.br
.BI "int isprint_l(int " "c" ", locale_t " loc );
.br
.BI "int ispunct_l(int " "c" ", locale_t " loc );
.br
.BI "int isspace_l(int " "c" ", locale_t " loc );
.br
.BI "int isupper_l(int " "c" ", locale_t " loc );
.br
.BI "int isxdigit_l(int " "c" ", locale_t " loc );
.fi
.sp
.SH DESCRIPTION
These functions check whether
.IR c ,
which must have the value of an
.I unsigned char
or
.BR EOF ,
falls into a certain character class according to the current locale. .TP
.BR isalnum_l ()
checks for an alphanumeric character; it is equivalent to
.BI "(isalpha(" c ") || isdigit(" c "))" \fR.
.TP
.BR isalpha_l ()
checks for an alphabetic character; in the standard \fB"C"\fP locale, it is equivalent to
.BI "(isupper_l(" c ") || islower_l(" c "))" \fR.
In some locales, there may be additional characters for which .BR isalpha ()
is true\(emletters which are neither upper case nor lower
case.
.TP
.BR isascii_l ()
checks whether \fIc\fP is a 7-bit
.I unsigned char
value that fits into
the ASCII character set.
.TP
.BR isblank_l ()
checks for a blank character; that is, a space or a tab.
.TP
.BR iscntrl_l ()
checks for a control character.
.TP
.BR isdigit_l ()
checks for a digit (0 through 9).
.TP
.BR isgraph_l ()
checks for any printable character except space.
.TP
.BR islower_l ()
checks for a lower-case character.
.TP
.BR isprint_l ()
checks for any printable character including space.
.TP
.BR ispunct_l ()
checks for any printable character which is not a space or an alphanumeric character.
.TP
.BR isspace_l ()
checks for white-space characters.
In the
.B """C"""
and
.B """POSIX"""
locales, these are: space, form-feed
.RB ( \(aq\ef\(aq ),
newline
.RB ( \(aq\en\(aq ),
carriage return
.RB ( \(aq\er\(aq ),
horizontal tab
.RB ( \(aq\et\(aq ),
and vertical tab
.RB ( \(aq\ev\(aq ).
.TP
.BR isupper_l ()
checks for an uppercase letter.
.TP
.BR isxdigit_l ()
checks for a hexadecimal digits, that is, one of
.br
.BR "0 1 2 3 4 5 6 7 8 9 a b c d e f A B C D E F" .
.SH RETURN VALUE
The values returned are nonzero if the character
.I c
falls into the tested class, and a zero value
if not.
.SH CONFORMING TO
POSIX.1-2008 specifies all of these functions.
.SH NOTES
The details of what characters belong into which class depend on the current locale.
.sp
from
.IR locale.h :
The concept of one static locale per category is not very well thought out. Many applications will need to process its data using information from several different locales. Another application is the implementation of the internationalization handling in the upcoming ISO C++ standard library. To support this another set of the functions using locale data exist which have an additional argument.

For example,
.BR isupper ()
will not recognize an A-umlaut (\(:A) as an uppercase letter in the default .B "C"
locale.
.SH EXAMPLE
The following example takes a locale abbreviation like "de_DE" as argument. Is no argument is supplied it will use "C". With "de_DE" the code will identify the O-Umlaut correctly as alphanumeric character but not with "C". In contrast the punctuation will perform as before.
.nf

#include <stdio.h>
#include <locale.h>

int main(int argc,char *argv[])
{
char *str="c2p\(:O.,";
int i;

  locale_t loc;
    if (argc > 1 )
      loc = newlocale (LC_ALL_MASK, argv[1], NULL);
    else
      loc = newlocale (LC_ALL_MASK, "C", NULL);

  for(i=0;str[i]!=0;i++) {
    if (isalnum_l(str[i],loc))
	printf("The character %c is alphanumeric.\\n",str[i]);
    if ( ispunct_l(str[i],loc) )
	printf ("The character %c is punctuation.\\n",str[i]);

}
return 0;
}

.fi
.SH SEE ALSO
.BR iswalnum (3),
.BR iswalpha (3),
.BR iswblank (3),
.BR iswcntrl (3),
.BR iswdigit (3),
.BR iswgraph (3),
.BR iswlower (3),
.BR iswprint (3),
.BR iswpunct (3),
.BR iswspace (3),
.BR iswupper (3),
.BR iswxdigit (3),
.BR setlocale (3),
.BR toascii (3),
.BR tolower (3),
.BR toupper (3),
.BR ascii (7),
.BR locale (7)
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

[CC += Bruno;

Bruno, if you have *two* moments to spare, maybe you could confirm that I am not saying stupid things about UTF-8, MBS, and WCS. If you have only *one* moment to spare, I'd appreciate if you look at the program I provide below as an example of the use of mbstowcs(); I plan to add it to the mbstowcs() man page that you wrote, and I wonder if you might check it to see that everything is in order.Thanks! To save time, just grep for the instance of your name below...]

Hello Walter,

You've submitted a number of pages over the past months that I have not found the energy to review. I respond to this submission, with the goal of explaining why, since it shows many of the problems that I see in the other submissions (and, in several cases, problems that I have commented on in your past submissions).

Problem: much of the text below is a straight copy from isalpha.3, written by Thomas Koenig. You can't do this without attribution. (Simplest would have been to retain Thomas's copyright line and license; I see that you did the latter.) To be clear, I do not believe you've done this maliciously; rather, you've done it in ignorance of what the requirements of copyright law are. But, I feel sure I've mentioned this issue to you in the past...

Problem: notwithstanding the copyright issue, the duplication of text is done without good reason. A more sensible approach in this case would be to perform some integration/reworking in the existing ispalpha.3 page, or simply to defer to that page inside this page.

Problem: no description of the feature test macro requirements for these pages. See man-pages(7).

Problem: this patch includes no "link" pages (containing just ".so man3/isalpha_l.3" for the dozen or so other functions documented on this page.)

The above sentence applies for the functions documented in isalpha.3, but is meaningless for the functions listed in the SYNOPSIS of this page, where the conversion is dependent on the locale 'loc', not the current locale.

Problem: all of the rest of the DESCRIPTION contains nothing new that isn't already in isalpha.3.

Problem: there is no mention of the requirements on 'loc'. (It must be a valid locale handle and must not be LC_GLOBAL_LOCALE.)

(Minor) problem: no VERSIONS section.

Simply quoting this text from locale.h without explanation does not really add much to the description. The point is that the *_l pages are designed to address the limitation that the traditional locale APIs do not mix well with multi-threaded applications and with applications that must deal with multiple locales. A general statement to that effect needs to appear somewhere, though probably not on this page. (I'll add something to locale(7).)

Problem: The sentence above relates to a function not even in the SYNOPSIS of this page.

The example program above violates multiple guidelines from man-pages(7), including, for example:

* 4-space indent levels
* spacing around operators and parentheses does not follow K&R norms ("indent -kr" fixes most of this) or the norms demonstrated in numerous existing pages.
* The program does not compile without warnings when using 'cc -Wall" (in particular, _XOPEN_SOURCE needs to be defined as 700).
* Various header files that should be included, are not.
* The program does not do error checking of function calls.

But, more to the point, the program appears to be broken, if you are operating in a UTF-8 locale, which I assume you are. I suppose the program does work if you are operating in an iso-8859-1 locale (though that seems an unlikely set-up these days), but that point would need some careful explanation in the man page, or some clarification in a shell session log that shows a run of the program, otherwise the program would cause much confusion for people on UTF-8 systems.

Functions such as isalnum_l() can't be applied to UTF-8 characters. POSIX seems clear:

The c argument is an int, the value of which the application shall ensure is representable as an unsigned char or equal to the value of the macro EOF. If the argument has any other value, the behavior is undefined.

(It would have been useful to see sample output from your program as part of the man page, as a help to the reader, but also as a check of what you believe is happening, and what locale you are working with.)

Instead, a conversion to wide characters is needed (mbstowcs(3)), and then the use of the isw*_l() functions. See my example, further down.

A modified version of your program illustrates the problem:

#define _XOPEN_SOURCE 700
#include <locale.h>
#include <string.h>
#include <stdio.h>
#include <locale.h>
#include <ctype.h>

int main(int argc, char *argv[])

{
    char *str = "c2pÖ.,";
    int i;
    locale_t loc;

    printf("string length = %ld\n", (long) strlen(str));
    if (argc > 1)
        loc = newlocale(LC_ALL_MASK, argv[1], NULL);
    else
        loc = newlocale(LC_ALL_MASK, "C", NULL);

    for (i = 0; str[i] != 0; i++) {
        printf("%d: %x %c: ", i, str[i] & 0xff, str[i] & 0xff);

        printf("%salphanumeric ", isalnum_l(str[i], loc) ? "" : "!");
        printf("%spunctuation ", ispunct_l(str[i], loc) ? "" : "!");
        printf("\n");
    }
    return 0;
}

See what happens when we run this:

$ ./a.out de_DE
string length = 7
0: 63 c: alphanumeric !punctuation 
1: 32 2: alphanumeric !punctuation 
2: 70 p: alphanumeric !punctuation 
3: c3 �: alphanumeric !punctuation   <====
4: 96 �: !alphanumeric !punctuation  <====
5: 2e .: !alphanumeric punctuation 
6: 2c ,: !alphanumeric punctuation 

Assuming you are operating in a UTF-8 system, the fourth glyph in your string (O-umlaut) is actually treated as two bytes, not one character by the program, but the construction of your program hides this, because it does not print information about all of the bytes!

Now, I am no expert (and so I may yet end up embarrassed), but it appears to me that you have misunderstood some of the fundamentals of how multibyte characters, UTF-8, and wide character strings work, and you didn't gain that understanding while writing the page. (Good reading:
http://www.joelonsoftware.com/articles/Unicode.html
http://www.cprogramming.com/tutorial/unicode.html
And I admit I learned a whole lot as I pulled your program apart.)

Repeating the SEE ALSO from isalpha.3 isn't really correct (or useful) here.

(Minor) problem: probably, other pages (at least, locale(7)) should add SEE ALSO references to this page. That change is not in this patch.

==

Now, to be clear: many page submissions that I receive fail on some of the points mentioned above, but this page fails on multiple counts.

In summary... Walter, you are often good at finding things that need to be documented, and I know your work is well intended. However, the pages you submit often require so much review/repair effort (in some cases, initial drafts of pages appear not to even have been run through a spell checker, though this page seems okay), that it is often faster to write the pages myself. And some of the same problems that I comment on in earlier submissions turn up again in new submissions. Thus, it is hard for me to find the enthusiasm to review these pages myself (and I have in any case very limited bandwidth) and help them get repaired (and it's rare that others step in, though I noticed that Stefan Puiu did take a shot on one of your submissions). I do not know what the solution is here, but this mail explains the problems from my side, and why I'm often unresponsive / slow to respond to your submissions (and am likely to remain so, unless something changes).

And here's how I think the kind of thing you intended to do in your example program actually needs to be done. (Bruno, perhaps you can confirm that this code is okay, as I plan to place this example in the wcstombs(3) man page.)

---8x------8x------8x------8x------8x------8x------8x------8x------8x---

#include <locale.h>
#include <wchar.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int
main(int argc, char *argv[])

{
    size_t mbslen;	/* Number of multibyte characters in source */
    wchar_t *wcs;	/* Pointer to converted wide character string */
    wchar_t *wp;

    if (argc < 3) {
        fprintf(stderr, "Usage: %s <locale> <string>\n", argv[0]);
        exit(EXIT_FAILURE);
    }

    /* Apply the specified locale */

    if (setlocale(LC_ALL, argv[1]) == NULL) {
        perror("setlocale");
        exit(EXIT_FAILURE);
    }

    /* Calculate the length required to hold argv[2] converted to
       a wide character string */
       
    mbslen = mbstowcs(NULL, argv[2], 0);
    if (mbslen == -1) {
        perror("mbstowcs");
        exit(EXIT_FAILURE);
    }

    /* Describe the source string to the user */

    printf("Length of source string (excluding terminator):\n");
    printf("    %ld bytes\n", (long) strlen(argv[2]));
    printf("    %ld multibyte characters\n\n", (long) mbslen);

    /* Allocate wide character string of the desired size.  Add 1
       to allow for terminating null wide character (L'\0'). */

    wcs = calloc(mbslen + 1, sizeof(wchar_t));
    if (wcs == NULL) {
        perror("calloc");
        exit(EXIT_FAILURE);
    }
    
    /* Convert the multibyte character string in argv[2] to a 
       wide character string */

    if (mbstowcs(wcs, argv[2], mbslen + 1) == -1) {
        perror("mbstowcs");
        exit(EXIT_FAILURE);
    }

    printf("Wide character string is: %ls (%ld characters)\n",
            wcs, (long) mbslen);

    /* Now do some inspection of the classes of the characters in
       the wide character string */
    
    for (wp = wcs; *wp != 0; wp++) {
        printf("    %lc ", (wint_t) *wp);

        if (!iswalpha(*wp))
            printf("!");
        printf("alpha ");

        if (iswalpha(*wp)) {
            if (iswupper(*wp))
                printf("upper ");

            if (iswlower(*wp))
                printf("lower ");
        }

        putchar('\n');
    }

    exit(EXIT_SUCCESS);
}

---8x------8x------8x------8x------8x------8x------8x------8x------8x---

And here's an example of what we see when running the program:

$ ./a.out de_DE.UTF-8 "Grüße!"
Length of source string (excluding terminator):
    8 bytes
    6 multibyte characters

Wide character string is: Grüße! (6 characters)
G alpha upper
r alpha lower
ü alpha lower
ß alpha lower
e alpha lower
! !alpha

With kind regards,

Michael

PS I'm working on adding the *_l functions to the isalpha.3 page, and will send a draft to the list.

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Am 10.03.2014 14:24, schrieb Michael Kerrisk (man-pages):

My major problem was i got no reply at all and was wondering if the page arrived in the ml at all.
I shortened the mail a bit.
Since you made a new page, most stuff is now obsolete but i would like to explain a few things i did.

For me that was very clear at that time since i had exactly that problem, i needed two locale internally for parsing, no need for threads. For me that illustrated the intention of the original author, therefore it went into notes.

yes, bad example, the basic idea was to describe that some function depend on a certain locale.

I use 8859-1 and i guess that is the reason i did not see any problems. But i am really wondering why i missed the -Wall point.

I found isw*_l() also while figuring out what to do with isalpha_l and friends.

I admit that i underestimated the complexity of the these locale stuff. I can only say i tried to document what was not documented and add an example basically showing how i came to that conclusion and how it works.

Sorry about that that i always try to get better, the wired thing is i can not find anything in my mailarchive. I found i send some pages years ago what i can not find is any reply, with one exception a typo in sem_wait.3.
Serious, i can not remember what happened, did i ever send a reply ?

And important, i guess the ISWALPHA_L page has the same defects as it is a direct derivative of ISALPHA_L.

NTL it seems there are a lot more _l functions, i did not check if there is a page but you may like to add some of them to undocumented(3):
see: http://www.freebsd.org/cgi/man.cgi?query=xlocale&sektion=3

sorry for the trouble,
wh

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

[CC restored, since I think this is a point that others may have comments on]

Hi Stefan,

The 'z' specifier is a C99 invention as I recall, and it took a few years before it became widespread. For example, it wasn't on Sun's libc for Solaris 8, and reading some man pages suggests that it wasn't there on Solaris 9 (released in 2002), though it was there by Solaris 10 (2005). Likewise, FreeBSD seems to have since it version 5 (2003).

So although glibc has had 'z' for a long time (at least as far back as glibc 2.1 at the start of 1999, though I suspect a little earlier as well), there were even a few years ago a lot of installed non-Linux systems that didn't support it.

So, for portable code, I've tended to almost reflexively use %ld+(long). Probably, the need to do that is less pressing now. There are many fewer of those installed legacy systems these days. I'll change that code for the mbstowcs() page to use %zu.

Thanks,

Michael

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html