New manpage for betoh64() and friends

Hello,

A few months ago Ulrich Drepper added a family of byteorder conversion macros to glibc.

I've written a rough manpage for them; attached.

The colophon may have the wrong version, I just guessed it.

The endian.3 file needs an big bundle of links to it, listed below:
betoh64
betoh32
betoh16
htobe64
htobe32
htobe16
letoh64
letoh32
letoh16
htole64
htole32
htole16

Also, the existing byteorder.3 manpage should get a "SEE ALSO" reference to the new family:
.BR endian (3)

Regards,
Nanno

PS. Please CC me explicitly on any email, I'm not on the 'vger' list.

.\" Written by Nanno Langstraat, released into the Public Domain .\"
.\" Since the Linux kernel and libraries are constantly changing, this .\" manual page may be incorrect or out-of-date. The author(s) assume no .\" responsibility for errors or omissions, or for damages resulting from .\" the use of the information contained herein. The author(s) may not .\" have taken the same level of care in the production of this manual, .\" which is licensed free of charge, as they might when working .\" professionally.
.\"
.TH ENDIAN 3 2008-07-08 "GNU" "Linux Programmer's Manual" .SH NAME
htobeNN, htoleNN, betohNN, letohNN \- convert values between host and big/little-endian byte order
.SH SYNOPSIS
.nf
.B #define __USE_BSD
.B #include <endian.h>
.sp
.BI "uint64_t htobe64(uint64_t " host_64bits );
.sp
.BI "uint16_t letoh16(uint16_t " little_endian_16bits );
.sp
.BI "uint32_t betoh32(uint32_t " big_endian_32bits );
.sp
...
etc.
.fi
.SH DESCRIPTION
These functions convert the byte encoding of integer values from the byte order that the current CPU (the "host") happens to use, to the standard little-endian / big-endian byte orders, and back again.

The functions all follow the following naming pattern:
.BR {SRC} to {DST}{BITS} ().

.BI {SRC}
is the byte order of the input parameter,
.BI {DST}
is the byte order of the desired return value.
These can be
.BI be
for "big endian",
.BI le
for "little endian", or
.BI h
for "host".

.BI {BITS}
is the number of bits of the value. It can currently be .BI 16
,
.BI 32
or
.BI 64 .

.SH VERSIONS
These function were added to GNU glibc version 2.8.90.

.SH "CONFORMING TO"
OpenBSD contains these functions, but requires <sys/endian.h> instead of <endian.h>.

FreeBSD has not (as of 2008) adopted these functions.

.SH NOTES
These
.BR endian (3)
functions are similar to the older
.BR byteorder (3)
family of functions.

For example,
.BR betoh32 ()
is identical to
.BR ntohl () .

The advantage of the
.BR byteorder (3)
functions is that they are available on most operating systems. Their drawback is that they were designed only for use in the context of TCP/IP. They therefore lack 64-bit variants and little-endian variants.

.SH EXAMPLE
Given the following code:

    uint8_t   file_input[4] = { 0x12, 0x34, 0x56, 0x78 };
    uint32_t  out = betoh32( (uint32_t*)file_input );

Variable "out" will equal 0x12345678.

(Unrelated side note: this short code example is not alignment-safe in the (uint32_t*) pointer cast)

.SH "SEE ALSO"
.BR byteorder (3)
.SH COLOPHON
This page is part of release 3.04 of the Linux
.I man-pages
project.
A description of the project,
and information about reporting bugs,
can be found at
http://www.kernel.org/doc/man-pages/.

Hi Nanno,

What version of glibc do these appear in? I can't find them in the headers or source for glibc 2.8.

Cheers,

Michael

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

In the manpage, I describe it as:

which I guesstimated from the CVS tags.

Here's the CVS log:

http://sources.redhat.com/cgi-bin/cvsweb.cgi/libc/string/endian.h?cvsroot=glibc

See revisions 1.9 and 1.10.

Regards,
Nanno

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Oops, I missed the "fedora-branch" versus "MAIN" branch difference.

Does that mean that the next "release" of glibc will be called 2.9 ? I'm not sure, myself: glibc doesn't do tarball releases anymore, you're supposed to go by the CVS tags.

Regards,
Nanno

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Nanno,

These functions don't seem to have made it into glibc 2.9. Do you know any more about this?

Cheers,

Michael

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

No, I have not had any other communications about those macros in the intervening time.

However, the following CVSweb page for file 'string/endian.h' with tag 'glibc-2_9' implies that it should be in glibc 2.9:
http://sources.redhat.com/cgi-bin/cvsweb.cgi/libc/string/endian.h?cvsroot=glibc&only_with_tag=glibc-2_9

Is there a difference between this CVSweb tag 'glibc-2_9' and the real glibc 2.9?

Note that the macros are only activated #ifdef __USE_BSD.

Regards,
Nanno

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Nanno,

I spoke too soon. I'd gone back and looked at your initial mail, seen mention of "betoh64" as one of the required link names, grepped for that in the glibc headers, and hadn't found it. But that name is wrong, I think you meant "be64toh". And indeed these interfaces are present in glibc 2.9.

Cheers,

Michael

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Ah, I see.

I'm very sorry for all the confusion that this one manpage has cost you.

My simple aim was: to get a standard set of htobe64() / betoh64() etc. macros on all platforms
(OpenBSD, Linux, FreeBSD; and hopefully outward from there).

End result: complete failure.

* OpenBSD wasn't willing to adopt <endian.h> instead of <sys/endian.h> for userspace apps.

(would accomodate glibc: glibc makes a sharper distinction between kernel API vs. userspace C library than the *BSDs do)

* glibc's Ulrich Drepper evidently chose to use his own naming scheme (be64toh), instead of adopting the original OpenBSD scheme (betoh64).

* FreeBSD didn't have much interest altogether.

So, I clearly didn't hit the right notes to interest anybody in adopting a standard.

I hope that this at least explains the situation & our grep confusion.

With regards,
Nanno

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Nanno,

No problem.

I assume OpenBSD has had these macros for a while. Do you know how long?

That's very unfortunate.

Not sure what you mean with this last sentence? FreeBSD seems to have these functions also, with names the same as OpenBSD.

:-(

Yes, that clarifies things, and also means that I have to fix things in the man page, since there I say that the same functions exist on the BSDs. However, the names of some of them are different, as you point out (I hadn't noticed that).

Cheers,

Michael

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hello Nanno

Thanks for that! I used some pieces from this, but rewrote much of the page.

That piece is automatically added by scripts.

The above names aren't all correct. e.g., betoh64 should be be64toh, etc. (Were the names different in the initial implementation?)

Thanks for thinking of that. Done.

The version of the page that is currently committed for release in man-pages-3.17 is shown below. Let me know if you see anything that needs fixing.

Thanks,

Michael

.\" Copyright (C) 2009, Linux Foundation, written by Michael Kerrisk
.\" <[email protected]>
.\" a few pieces remain from an earlier version
.\" Copyright (C) 2008, Nanno Langstraat <[email protected]> .\"
.\" Permission is granted to make and distribute verbatim copies of this .\" manual provided the copyright notice and this permission notice are .\" preserved on all copies.
.\"
.\" Permission is granted to copy and distribute modified versions of this .\" manual under the conditions for verbatim copying, provided that the .\" entire resulting derived work is distributed under the terms of a .\" permission notice identical to this one.
.\"
.\" Since the Linux kernel and libraries are constantly changing, this .\" manual page may be incorrect or out-of-date. The author(s) assume no .\" responsibility for errors or omissions, or for damages resulting from .\" the use of the information contained herein. The author(s) may not .\" have taken the same level of care in the production of this manual, .\" which is licensed free of charge, as they might when working .\" professionally.
.\"
.\" Formatted or processed versions of this manual, if unaccompanied by .\" the source, must acknowledge the copyright and authors of this work. .\"
.TH ENDIAN 3 2009-01-15 "GNU" "Linux Programmer's Manual"
.SH NAME
htobe16, htole16, be16toh, le16toh, htobe32, htole32, be32toh, le32toh, htobe64, htole64, be64toh, le64toh \-
convert values between host and big-/little-endian byte order .SH SYNOPSIS
.nf
.B #define _BSD_SOURCE
.B #include <endian.h>

.BI "uint16_t htobe16(uint16_t " host_16bits ); .BI "uint16_t htole16(uint16_t " host_16bits ); .BI "uint16_t be16toh(uint16_t " big_endian_16bits ); .BI "uint16_t le16toh(uint16_t " little_endian_16bits );

.BI "uint32_t htobe32(uint32_t " host_32bits ); .BI "uint32_t htole32(uint32_t " host_32bits ); .BI "uint32_t be32toh(uint32_t " big_endian_32bits ); .BI "uint32_t le32toh(uint32_t " little_endian_32bits );

.BI "uint64_t htobe64(uint64_t " host_64bits );
.BI "uint64_t htole64(uint64_t " host_64bits );
.BI "uint64_t be64toh(uint64_t " big_endian_64bits ); .BI "uint64_t le64toh(uint64_t " little_endian_64bits ); .fi
.SH DESCRIPTION
These functions convert the byte encoding of integer values from the byte order that the current CPU (the "host") uses, to and from little-endian and big-endian byte order.

The number,
.IR nn ,
in the name of each function indicates the size of integer handled by the function, either 16, 32, or 64 bits.

The functions with names of the form "htobe\fInn\fP" convert from host byte order to big-endian order.

The functions with names of the form "htole\fInn\fP" convert from host byte order to little-endian order.

The functions with names of the form "be\fInn\fPtoh" convert from big-endian order to host byte order.

The functions with names of the form "le\fInn\fPtoh" convert from little-endian order to host byte order.
.SH VERSIONS
These function were added to glibc in version 2.9. .SH "CONFORMING TO"
These functions are non-standard, but present on the BSDs, where the required header file is
.I <sys/endian.h>
instead of
.IR <endian.h> .
.SH NOTES
These functions are similar to the older
.BR byteorder (3)
family of functions.
For example,
.BR be32toh ()
is identical to
.BR ntohl () .

The advantage of the
.BR byteorder (3)
functions is that they are standard functions available
on all Unix systems.
On the other hand, the fact that they were designed
for use in the context of TCP/IP means that
they lack the 64-bit and little-endian variants described in this page. .SH EXAMPLE
The program below display the results of converting an integer from host byte order to both little-endian and big-endian byte order. Since host byte order is either little-endian or big-endian, only one of these conversions will have an effect.
When we run this program on a little-endian system such as x86-32, we see the following:
.in +4n
.nf

$ \fB./a.out\fP
x.u32 = 0x44332211
htole32(x.u32) = 0x44332211
htobe32(x.u32) = 0x11223344
.fi
.in
.SS Program source
\&
.nf
#include <endian.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>

int
main(int argc, char *argv[])

{
    union {
        uint32_t u32;
        uint8_t arr[4];
    } x;

    x.arr[0] = 0x11;    /* Lowest-address byte */
    x.arr[1] = 0x22;
    x.arr[2] = 0x33;
    x.arr[3] = 0x44;    /* Highest-address byte */

    printf("x.u32 = 0x%x\\n", x.u32);
    printf("htole32(x.u32) = 0x%x\\n", htole32(x.u32));
    printf("htobe32(x.u32) = 0x%x\\n", htobe32(x.u32));

    exit(EXIT_SUCCESS);
}

.fi
.SH "SEE ALSO"
.BR byteorder (3)

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Alright, so it turns out that this is not accurate because of Ulrich Drepper's beNtoh() versus OpenBSD's betohN() deviation.

(and FreeBSD into the mix is even more muddled: they got an earful from Theo de Raadt over this issue)

Suggestion:

.SH "CONFORMING TO"
These functions are non-standard. The BSDs contain similar but not wholly identical functions in header file .I <sys/endian.h>
(instead of glibc's
.IR <endian.h> ).

It grates that this mealy-mouthed text is longer than the source changes required to fix the differences, but hey.

Nanno

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Nanno,

Yes, see my previous mail.

Muddled in what sense? (See my comparisons of the names below.)

Unless I've missed something, the situtation seems to be this:
* FreeBSD, NetBSD, and glibc all use *exactly) the same naming convention, so that they have, e.g., le16toh.
* OpenBSD uses a different convention, so that they have, e.g., letoh16. (This naming convention seems to me a little more sensible.)

I'm guessing that the order of events went like this:

1) OpenBSD provided the first implementation.
2) the other BSDs added the interfaces but changed the naming convention.
3) Ulrich Drepper opted to follow FreeBSD's naming convention (which seems a reasonable thing to do, given that that is the dominat BSD).

Do you know if that is the order in which things happened?

Cheers,

Michael

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Oky -- so I found
http://thread.gmane.org/gmane.os.openbsd.tech/15161/focus=15179
and
http://thread.gmane.org/gmane.os.freebsd.devel.hackers/32852
which answer all my questions, I think.

Cheers,

Michael

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Yeah...

What I did: "Hello good fellows on mailing list X, let us discuss setting a standard."

What might have worked better: "Hey mailing list X, here's a patch to get an identical API on X, Y and Z. Please apply."

Cheers,
Nanno

--
To unsubscribe from this list: send the line "unsubscribe linux-man" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

On Mon, Jan 19, 2009 at 1:09 PM, Nanno Langstraat <[email protected]> wrote:

[...]

I've added some text along these lines to the man page.

Cheers,

Michael

Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html